Experimental Design

Goal: Understand variation

We want to distinguish between variation of interest and variation from other sources (again, increase signal-to-noise ratio).

“Whenever we carry out an experiment or observational study, we are either interested in measuring random variation, or (more often) trying to find ways to remove or reduce the effects of random variation, so that the effects that we care about can be seen more clearly.”

Definition: Replication involves making the same manipulations to and taking the same measurements on a number of different, independent experimental subjects.

We are essentially talking about sample size here, but there is more to it than that due to the independence issue.

http://www.zoology.ubc.ca/~whitlock/kingfisher/SamplingNormal.htm

Example: Does sex have an effect on human height?

Measure height in 10 married couples of the opposite sex. Are we safe in restricting our sample to married couples?

Definition: Pseudoreplicates are dependent measures.

Definition: Replicate measures must be independent of each other, i.e. a measurement made on one individual should not provide any useful information about that factor on another individual.

Definition: Pseudoreplication occurs if we analyze pseudoreplicates as if they were replicates.

When we pseudoreplicate, we are making a false claim about the amount of replication.

Both accuracy and precision are affected by pseudoreplication!

Accuracy: Pseudoreplication changes our question from general and interesting to more specific and less interesting.

Precision: Pseudoreplication underestimates the precision due to dependence of measures of interest.

Question: Do blue tit nestlings raised in nest boxes suffer more from external parasites than those raised in natural cavities?

Experimental Design: Investigate the four nestlings in a particular nest box and count the number of parasites on each.

Discuss: Why are these pseudoreplicates? Explain how they affect precision and accuracy.

Answer #1: This design gives you information on parasite load only for birds in this particular nest box.

Answer #2: Nestlings will be similar in many ways due to sharing nest box, and thus variation of parasite load between nestlings in this box will be smaller than between all nestlings.

Pseudoreplication is a biological and experimental design issue, not a statistical issue. Data doesn't look pseudoreplicated.

Common sources of pseudoreplication:

• A shared enclosure
  • Environment affects all individuals similarly
  • Individuals can affect other individuals
• The common environment
• Relatedness
• Pseudoreplicated stimulus
• Measurements over time
• Species comparisons

Make sure you are…

1. accounting for all possible variation.
2. controlling for possible confounding variables.

Experimental study: Make sure individuals differ systematically only in the explanatory variable(s) of interest.

Observational studies: Be aware of confounding variables.

Random sampling, or randomization, can solve many of these problems.

Blocking is another technique to address pseudoreplication in experimental studies.

Matching is analogous to blocking for observational studies.

Birdsongs and attractiveness

Question: How do we measure relationship between male birdsongs and attractiveness to females?

Experimental Design: Record the complex song of one male and the simple song of another male, and then play these same two songs to each of 40 different females. Compute a confidence interval for the mean attractiveness of the two male songs.

Discuss: What is wrong with this design so far?

Answer: Each measure of female choice is a pseudoreplicate (\( n=40 \)).

Discuss: What is wrong with this design so far?

Answer: Each measure of female choice is a pseudoreplicate (\( n=40 \)).

Discuss: What can we do to correct for this pseudoreplication?

Answer: Record songs of 40 males with complex songs, and 40 separate males with simple songs. Each female should listen to a unique pair of songs, one simple and one complex. Design can get even more complicated than this.

Discuss: What are examples of confounding variables in the pseudoreplicated case?

Blood sugar levels

Experimental Design: Phlebotomist takes 15 samples from each of 10 patients, yielding a total of 150 measurements.

Discuss: What is the replicate and sample size in this situation? Why?

Antibiotics and bacterial growth rates

Experimental Design: Two agar plates: one with antibiotic, one without. Spread bacteria on both plates, let them grow for 24 hours, then measure diameter of 100 colonies on each plate?

Discuss: What is the replicate and sample size in this situation? Why?

Three things:

• Plan for precision (estimation)
• Plan for power (hypothesis testing)
• Plan for data loss

We'll use a two-sample \( t \)-test as the example in this section.

We would like to compute a 95% confidence interval for \( \mu_{1}-\mu_{2} \).

\[ \bar{Y}_{1}-\bar{Y}_{2} \pm \mathrm{margin \ of \ error}, \]

where “margin of error” is the half-width of the 95% confidence interval.

In this case, the following formula is an approximation to the number of samples needed to achieve the desired margin of error (assuming balanced design, i.e. \( n_{1}=n_{2}=n \)):

\[ n \approx 8\left(\frac{\mathrm{margin \ of \ error}}{\sigma}\right)^{-2} \]

Two-sample \( t \)-test:

\[ H_{0}: \mu_{1} - \mu_{2} = 0. \] \[ H_{A}: \mu_{1} - \mu_{2} \neq 0. \]

A conventional power to aim for is 0.80, i.e. we aim to prove \( H_{0} \) is false in 80% of experiments.

Assuming a significance level of 0.05, a quick approximation to the planned sample size \( n \) in each of two groups is

\[ n \approx 16\left(\frac{D}{\sigma}\right)^{-2}, \]

where \( D = |\mu_{1}-\mu_{2}| \) is the effect size.

library(pwr)

Two-sample \( t \)-test with significance level 0.05, 80% power, and relative effect size \( d = \frac{|\mu_{1}-\mu_{2}|}{\sigma} = 0.3 \).

pwr.t.test(d=0.3, power=0.8, type="two.sample")

     Two-sample t test power calculation 

              n = 175.3847
              d = 0.3
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

Definition: Proper randomization means that any individual experimental subject has the same chance as any other individual of finding itself in each experimental group, as well as prepared, setup, or measured in the same way.

• Improper randomization can lead to the introduction of confounding variables in the experimental protocol itself (experimental artifacts).
• Randomization breaks the association between possible confounding variables and the explanatory variable.
• Randomization allows the causal relationship between the explanatory and response variables to be assessed.

Definition: Proper randomization means that any individual experimental subject has the same chance as any other individual of finding itself in each experimental group, as well as prepared, setup, or measured in the same way.

• Randomization does not eliminate variation by confounding variables, only their correlation with treatment.
• Randomization ensures that variation by confounding variables is similar between treatment groups and occurs by chance alone.

Question: Does a specific genetic modification to a tomato plant affect its growth rate?

Experimental Design: Place 50 genetically modified plants, and 50 unmodified plants, into individual pots with compost, and then put them all into a growth chamber.

Discuss: Where can improper randomization appear in this example?

Answer: For example:
-      Difference in compost quality.
-      Difference in temperature across chamber.

Let's look at temperature as a possible confounding variable:

The above randomization would not remove temperature difference across chamber, but simply remove correlation with treatment.

What if we would like to reduce the variation from temperature? We can try blocking.

Our attempt to control for temperature:

Discuss: What’s right and wrong with this particular design?

This particular blocking design is properly replicated and randomized.

The variation due to temperature in each chamber has been reduced, so that the difference between treatments becomes more apparent.

There was a systematic difference of temperature across the original chamber. We have now adjusted the design to systematically account for this difference.

What if you can't do experiments? Randomization does not apply here.

Two strategies are used to limit effects of confounding variables on a difference between treatments in a controlled observational study.

Definition: With matching, every individual in the treatment group is paired with a control individual having the same of closely similar values for the suspected confounding variable.

Definition: With adjustment, use a statistical method, such as analysis of covariance, to correct for differences between treatment and control groups in suspected confounding variables.

Assigning treatments to subjects (one possibility):

1. List all \( n \) subjects, one per row, in a spreadsheet.
2. Use the computer to give each subject a random number.
3. Assign treatment A or B to those subjects receiving the lowest or highest numbers, respectively.

Remember, randomization is important in all processes of the experiment, including preparation, setup, and measurement.

Randomize measurement of replicates in time:

• Watching 50 hours of great tit courtship behaviour on video increases your ability to observe
• After 10 hours of counting through a microscope, tiredeness kicks in
• Aging equipment

This shows time of measurement could be a confounding factor.

• Whitlock & Schluter, Interleaf 2: Pseudoreplication (pp. 115-116)
• Whitlock & Schluter, Chapter 14: Designing experiments