• Sometimes a research question pertains to group differences

• In this course, we will be focusing on the case of 3 or more means, in which case you would use a test called Analysis of Variance (ANOVA).

• Don't let the name fool you- this is a mean-comparison test.

• test for between group differences when there is one grouping variable and one continuous numeric outcome
• The One-Way ANOVA method helps understand group differences respective to the expected amount of inter-individual variation for the populations

The test-statistic for ANOVA distributed on the F distribution and is calculated using the formula:

\(F = \frac{MS_{between}}{MS_{within}}\)

in which \(MS_{between}\) is a measure of between-group variance and \(MS_{within}\) is a measure of within-group (inter-individual) variance

\(F = \frac{MS_{between}}{MS_{within}}\)

• What happens if there is more variance between groups (numerator) than there is variance within groups?

• What happens if both are the same?

• The F ratio is larger than 1
• The F ratio is exactly 1

The larger the F-statistic, the more likely we will be able to reject the null hypthesis that there are no group differences (between any two groups)

Note: Sometimes a visual can be helpful in understanding what seems like an abstract concept

The One-way ANOVA does have some assumptions, violations of which may result in incorrect and inconsistent inferences. The assumptions are:

• Each sample (representing each group) is drawn from a population that is Normally distributed on the outcome
• Each of the aforementioned populations have equal variance (i.e. homeogeneity of variance)

The ANOVA is an omnibus test!

• If you find that your F-value is statistically significant, you may choose to run post-hoc tests to determine which means are significantly different from one another. These tests are just traditional pair-wise comparisons (like a t-test) for each pair of means.
• BUT proceed with caution…

When you select an alpha level for your test such as a=.05:

• if your study outcome is extreme/rare enough that it is less than %5 likely to occur under the null, you will reject the null hypothesis.
• However, these extreme/rare values can and do occur under the null, with exactly 5% probability.
• Thus, you accept that there is a 5% chance of making a Type I error, or rejecting the null when in reality the null is True.

Note: You can make the alpha level smaller (e.g. push the critical value further into the tail of the comparison distribution) to be more conservative and protect against Type I errors, but in doing so you increase the chance of a Type II error (failure to reject the null when, in the real world, the null is False)

Running multiple comparisons means multiple tests…

• Running all those individual tests leads to inflated family-wise Type I error rates. * One common solution is to use adjusted p-values such as in a Tukey's HSD.
• Alternatively, only test specific pairwise comparisons that you choose ahead of time. These decisions are called apriori and allow you to reduce the number of tests, and therefore the overall probability of making a Type I error in your study.
• Whether apriori or post-hoc tests are run, a Bonferroni correction is typically applied to the tests. This correction splits the desired alpha level easily across all the tests. For each individual test it will be harder to reject the null, but your study is protected against inflated Type I error rates.
• Finally, contrast-coding of the outcome can help avoid this problem

The one-way ANOVA is technically another special case of the General Linear Model with:

• single numeric outcome variable (the variable in which you are wanting to know about mean differences) and
• a single grouping variable.

Consider a study on ODD.

Grouping variable with 3 treatment groups: * 1=standard best practice * 2=new therapeutic treatment * 3=control

Outcome: Behavior score (scale score, obtained by surveying parent)

ANOVA frame: "Do these three groups have significantly different mean behavior scores?"

Linear Model frame: "Does group membership predict behavior levels?"

In both cases, the goal is to get an estimate of behavior levels for each of the three groups, and to use a statistical test to determine whether the mean levels of behavior for each of the three groups are statistically different from one another.

Informal Model:

Behavior ~ Intercept + Group2membership + Group1membership

if group membership is recorded as "1" for those in the group and 0 for those not in the group

Formal model:

\(Y_{i} = \beta_{0} + \beta_{1}G_{i1} ... + \beta_{k}G_{ik} + e_{i}\)

where \(G_{i1}\) =1 for individuals in Group 1, and is 0 otherwise. Similarly, \(G_{i2}\) =1 for individuals in Group 2, and is 0 otherwise, and so on.

These G variables are called dummy variables and are typically created by the researcher or analyst post data collection.

For the sake of clarity, consider a person who is in Group 2 (the group receiving new theapeutic treatment). Their expected behavior would be determined by the equation:

BehaviorScore ~ B0 + B1*(1) + B2*(0)

or just BehaviorScore ~ B0 + B1*(1)

Note: as long as individuals are in the same group, this equation will produce the same expected value, which makes sense because with ANOVA we are only interested in group differences

Multivariate ANOVA, or MANOVA is an extension of ANOVA that allows for more than 1 outcome variable.

Example: we have another outcome in our ODD study: the total cumulative number of derogatory remarks in the classroom since the implementation of treatment, collected from the teacher/schools. We are interested in whether the children in different treatment groups will differ on their total behavior score and/or on their number of derogatory remarks in the classroom.

Recall that the F- statistic used in ANOVA to test for significant between-group differences is calculated as a ratio of between-group variance and within-group variance on the outcome of interest, or

F formula

This statistic is limited by the fact that can only account for variance on a single outcome.

Thus, in the case of MANOVA, we will use a different statistic that follows an approximate F-distribution.

• Wilk's Lambda
• Hotelling-Lawley statistic.

represent the amount of variation in the outcome that is not explained by the grouping variable. Thus, lower values are supportive of strong grouping effects, more often leading you to reject the null. (H0: there are no significant differences in group means on either of the outcome variables.)

• Pillai's Criterion

ranges from 0 to 1, represents how well the grouping variable distinguishes individuals on the outcomes. Higher values indicate that the grouping is important, and are evidence for rejecting the null hypothesis.