2024-01-11

Design, what design?

The snag, of course, is that doing good experiments is difficult.

— David COLQUHOUN, 1971 (Lectures in Biostatistics)

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.

— Sir Ronald FISHER, 1938 (Presidential Address to the First Indian Statistical Congress)

  • I have done post-mortems on many ‘un-analysable’ studies (experimental and observational) — and not just in PhD theses… in research seminars and journal manuscripts.

  • Design your studies fully before you start them.
    What does this mean?

    • Plan the statistical analysis before you gather any data!
    • Translate your questions into formal hypotheses \((H_0, H_a)\).
    • Identify appropriate tests for your \(H_0\).
    • The analysis must match the study design.

  • This week is specifically about desigining experiments (rather than observational studies).

Three fundamental principles

  1. Replication

  2. Randomisation

  3. Blocking

All three are somewhat related. We’ll visit them in more detail in turn.

Replication

Most people need all the help they can get to prevent them making fools of themselves by claiming that their favourite theory is substantiated by observations that do nothing of the sort.

— David COLQUHOUN, 1971 (Lectures in Biostatistics)

You need evidence!

Your evidence comes from observations. You need sufficient \(N\) of observations to have sufficient evidence for your claims.

  • Any old observations? No: replicates — measurements of exactly the same thing.
  • Any old replicates? No: they need to be independent.

Where replicates are not independent: Pseudoreplication.

Assume you want to compare two fertilisers, A and B.
Here is what does not work:

  • Apply fertiliser A and B on one plot each.
  • Measure 1000 plants from Plot 1 and 1000 plants from Plot 2. \(N=2\times 1000\)? — No: more like \(N=2\times 1\).

Pseudoreplication & Independence

What is the problem with the two-plot fertiliser design?

 Plot 1  Plot 2
+-------+-------+
¦       ¦       ¦
¦   A   ¦   B   ¦
¦       ¦       ¦
+-------+-------+
  • The two plots will not be identical (drainage, shade…).
  • Assume the two fertilisers are no different: you gather a lot of evidence \((2\times N=1000)\) about… — the two plots!
  • Impossible to tell fertiliser (A vs B) from plot (1 vs 2) effects.
  • All measurements from one plot share information about growth in that plot: not independent.

Other examples of pseudoreplication

  • Same individual measured repeatedly over time (regardless of interval).
  • Measurements taken in the same place.

These are common issues in field ecology (behaviour of starlings in two flocks in two places), but also in the lab (Petri dishes…).

  • We need more plots / flocks / Petri dishes.
Not what $N=1000$ looks like!

Not what \(N=1000\) looks like!

Formal perspective

In ANOVA variance partitioning, \(N\) of replicates determines Error DoF.

True replication

  • Increases Error DoF and thus \(F\)-ratio (Explained MS / Error MS).
  • Makes \(s\) a better estimate of \(\sigma\).
  • \(s\) and \(N\) feed into SEM — in simple one-sample case, we have:

\[\mathrm{SEM} = \frac{s}{\sqrt{N}}\]

True replication thus improves precision of estimates \((\hat\mu, s)\).

Pseudoreplication

  • Inflates Error DoF and \(F\)-ratio
  • SEM too low (false precision) and \(P\)-value too small (false significance).

How many is enough? — at the very least, enough for 10 error DoFs.

Randomisation

Taking precautions against making a fool of yourself.

Avoid three cardinal sins:

  1. Systematic Designs — next slide
  2. Unconscious Bias
    • Example: picking 3 \(\times\) 20 crickets from a cage in sequence: the first 20 will be the slowest / sickest / biggest / prettiest…
    • Very hard to predict direction of bias.
  3. Undesirable Subjectivity:
    • almost impossible for humans to do anything truly random.
    • avoiding a pattern creates (non-random) ‘anti-pattern’

Bad: Systematic Designs

What’s wrong with this?

+-------+-------+-------+-------+
¦       ¦       ¦       ¦       ¦
¦   A   ¦   A   ¦   C   ¦   C   ¦
¦       ¦       ¦       ¦       ¦
+-------+-------+-------+-------+
¦       ¦       ¦       ¦       ¦
¦   A   ¦   A   ¦   C   ¦   C   ¦
¦       ¦       ¦       ¦       ¦
+-------+-------+-------+-------+
¦       ¦       ¦       ¦       ¦
¦   B   ¦   B   ¦   D   ¦   D   ¦
¦       ¦       ¦       ¦       ¦
+-------+-------+-------+-------+
¦       ¦       ¦       ¦       ¦
¦   B   ¦   B   ¦   D   ¦   D   ¦
¦       ¦       ¦       ¦       ¦
+-------+-------+-------+-------+

It’s essentially the same as:

+-------+-------+
¦ A     ¦ C     ¦
¦ A   A ¦ C   C ¦
¦     A ¦     C ¦
+-------+-------+
¦ B     ¦ D     ¦
¦ B   B ¦ D   D ¦
¦     B ¦     D ¦
+-------+-------+

All A treatment plots are adjacent: they are likely more similar to one another! And so are all B, C,…

  • Underestimates within-treatment variance;
  • Overestimates between-treatment variance = treatment effect;
  • Inflates \(F\), \(P\) too low…

Pseudoreplication all over again!

True randomisation

How not to fall prey to selection biases and unconscious patterns?

1.) Write out list of all experimental units (plots, mice…). E.g, twelve mice:

mouse.id <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
# or, smarter,
mouse.id <- seq(1, 12)

2.) Write out list of treatments to be assigned. E.g., three treatments (A, B, C):

treatment <- c("A", "A", "A", "A", "B", "B", "B", "B", "C", "C", "C", "C")
# or, smarter:
treatment <- rep(c("A", "B", "C"), each = 4)

3.) Randomise and pair up:

treat.assigned <- sample(treatment, size = 12, replace = FALSE)
Randomised.Design <- data.frame(mouse.id, treat.assigned)

Result of Randomisation

Randomised.Design
##    mouse.id treat.assigned
## 1         1              A
## 2         2              C
## 3         3              B
## 4         4              B
## 5         5              A
## 6         6              B
## 7         7              C
## 8         8              C
## 9         9              A
## 10       10              C
## 11       11              B
## 12       12              A
with(Randomised.Design, table(treat.assigned))
## treat.assigned
## A B C 
## 4 4 4