EXPERIMENTAL DESIGN

One one of the central concepts of statistics states: “Correlation does not imply causation”. This is a fundamental notion to not only understanding regression but also for experimental design. While regression analysis (linear, logistic, multivariate, etc.) can be extremely useful for analyzing data and identifying potential trends, experiments can provide reliable evidence that supports a causal relationship between two variables. The basic outline of experimental design includes an explanatory response variable and response variable, randomization, control, and replication. When properly implemented, these principles increase the reliability and validity of the results and allows the experimenter to exclude confounding or lurking variables as an alternative explanation.

Studies that do not follow experimental design principles are labeled observational studies where researchers observe the effects of a risk factor or treatment without intervention. This is particularly are useful in fields such as epidemiology and psychology. Other studies that fall in between the two are termed quasi-experimental designs, which depends on the type of research question of interest, but these kinds of studies can also be useful for validating treatment methods or investigating potential associations on a more exploratory basis.

The aim of an experiment is to study the relationship between two variables. Following the scientific method, experiments are designed to test hypotheses formulated from theories and observations of the natural world. To infer a causal relationship between two variables, sound experimental design is crucial to producing accurate, reliable data. Of the two, the independent/explanatory variable serves to generate a change in the affected dependent/response variable.

DESIGN PRINCIPLES

In order to determine whether experimental results are valid, researchers must follow several design principles: control, randomization, and replication.

CONTROLS AND BIASES

Think back to the mention of confounding variables. How do we reduce or eliminate their effect? By controlling for the effects that lurking variables can have on the DV. The issue with not controlling properly can lead to several biases. The most prominent bias is experimental bias which occurs when the researcher unconsciously affects results or data to favor certain outcomes based on personal influence. Another kind of bias that occurs in medical settings is the placebo effect which occurs when patients react to a treatment that they believe will have a positive effect even when in actuality no such method has been provided.

In addition, the reversal may affect the results; if participants suspect that they received no actual treatment, then that could change their behavior and the efficacy of the treatment could not be accurately evaluated. In this case, double-blinding is a common and recommended method in which neither the researchers nor participants are aware of the patients’ group status.

RANDOMIZATION

The practice of eliminating one’s own subjective bias with personal judgment is not a reliable method. Thus, randomizing experiments is one of the key features of receiving approval from the standards of academic journals. In addition, it is crucial to practice random sampling when selecting subjects for an experiment. Failure to do so means that any conclusions cannot be attributed to the general populous since the sample does not accurately reflect the overall population. For instance, if only participants under the age of 18 are used to test a medication, then the results of the experiment could not be attributed to the adult population - especially in this case in which children are known to have different reactions to dosages of medication due to either differences in hormone levels, age, or weight.

Two common randomized experimental designs are detailed below:

1. Completely Randomized Design

Items or participants are assigned to groups completely at random using objective methods or random number generators. One common practice is to label each subjects and then assign treatments using a table or random numbers, or running subjects through a computer program to select a random sample. Imagine a simple medical experiment: a company wishes to test the efficacy of a new vaccine and randomly selects 1000 people from across the US to test this vaccine. Subjects are randomly assigned to either the placebo or vaccine treatment. A table of this design is shown on the next tab.

Note: While the participants are randomly distributed between the two treatments, observe that the number of participants per treatment is equal. In experiments, sample size can influence the results and variability, so it is important to keep the treatment sizes generally the same. In statistics, as the sample size increases the amount of variability lessens due to the effect of averaging all participant results. Essentially, it is less likely that the outcome is due to random chance.

2. Randomized Block Design

In this design, the experimenter distributes subjects into subgroups labeled blocks in a way that reduces the variability within the blocks more so than that between blocks. Then the subjects within each block are randomly assigned to treatment conditions. For example, a sample population of 1000 participants are part of an experiment testing the efficacy of a vaccine. The experimenters decide to split the population into two subgroups: male and female. The participants are assigned to blocks based on gender, and within each block the 500 participants are randomly split between treatments (placebo or vaccine). A table of this design is shown on the next tab.

Note: Gender is commonly used to split participants into blocks since it has been repeatedly observed that men and women have differences in physiology as well as reaction to medication. This particular design explicitly removes gender as a source of variability and as a confounder.

REPLICATION

After an experiment is designed following the principles above and the results are analyzed, we must still remain cautious of drawing strong conclusions from a single experiment no matter the statistical significance shown. For instance, the “success” of one experiment may still be due to random chance or bias or confounders, even when controlling for other variables. Thus, replication will show if a treatment is actually effective over time due to the long-term averaging effect of multiple experiments, which also reduces experimental bias since the original researchers would not be involved int he following experiments.

TABLES

QUIZ

ETHICS

MISLEADING GRAPHS