What We Know So Far

Correlations

1. Understanding Correlation Values

Correlation Coefficient (r): Always between -1 and +1

Direction:

- Positive values (+): As one variable increases, the other increases

- Negative values (-): As one variable increases, the other decreases

Strength interpretation:

- ±0.00 to ±0.19: Very weak

- ±0.20 to ±0.39: Weak

- ±0.40 to ±0.59: Moderate

- ±0.60 to ±0.79: Strong

- ±0.80 to ±1.00: Very strong

2. Reading Correlation Output

When looking at statistical output, focus on:

1. Correlation coefficient (r)
   • Example: r = 0.72
   • Interpretation: “There is a strong positive correlation”
2. P-value
   • If p < .05: “statistically significant”
   • If p > .05: “not statistically significant”
   • Example: “r(58) = 0.72, p < .001”
   • The number in parentheses (58) is degrees of freedom (N-2)
3. Sample size (N)
   • Larger samples = more reliable correlations
   • Small samples may show strong correlations by chance

3. Interpreting Scatterplots

Look for these patterns:

1. Direction

   Positive /          Negative \          No relationship •

2. Strength

   • Tight cluster = strong correlation
   • Scattered points = weak correlation
   • Random cloud = no correlation

3. Shape

   • Linear (straight line pattern)
   • Curved (might need different analysis)
   • Other patterns (might violate assumptions)

4. Outliers

   • Look for points far from the pattern
   • Note if they might affect the correlation

4. Writing Correlation Results

Template for writing results: “There was a [strength] [direction] correlation between [variable 1] and [variable 2], r(df) = [value], p = [value]”

Examples: - “There was a strong positive correlation between height and weight, r(28) = 0.75, p < .001” - “Sleep and exam performance showed a moderate positive correlation, r(98) = 0.45, p = .003”

5. Common Mistakes to Avoid

1. Causation Claims
   • ❌ “X causes Y”
   • “X and Y are related” or “X and Y are associated”
2. Direction Confusion
   • ❌ “r = -0.75 shows a strong positive relationship”
   • “r = -0.75 shows a strong negative relationship”
3. Significance vs. Strength
   • Remember: A significant correlation isn’t necessarily strong
   • A strong correlation might not be significant with small samples

6. Red Flags to Watch For

• Perfect correlations (r = 1.0 or -1.0) are rare in real data
• Very strong correlations (>0.90) might indicate:
  • Same variable measured twice
  • Part-whole relationships
  • Potential data issues

7. Quick Checklist for Exam Questions

When interpreting correlations, always check:

[ ] Direction (positive/negative)

[ ] Strength (weak/moderate/strong)

[ ] Statistical significance (p-value)

[ ] Sample size

[ ] Any obvious outliers or patterns

[ ] Whether assumptions are met (linear relationship)

8. Key Terms to Know

• Zero correlation: r = 0 (no linear relationship)
• Perfect correlation: r = 1 or -1
• Spurious correlation: When two variables appear related but aren’t meaningfully connected
• Coefficient of determination: r² (proportion of shared variance)

Remember: Correlation only measures linear relationships. Two variables might be strongly related in other ways but show low correlation if the relationship isn’t linear.

Bivariate Regression

1. Basic Formula Notation

The standard equation for a bivariate regression model is:

\[Y = \beta_0 + \beta_1X + \varepsilon\]

Where:
- \(Y\) denotes the dependent (outcome) variable.
- \(X\) denotes the independent (predictor) variable.
- \(\beta_0\) is the intercept (constant term), representing the predicted value of \(Y\) when \(X = 0\).
- \(\beta_1\) is the slope coefficient, representing the amount of change in \(Y\) for a one-unit change in \(X\).
- \(\varepsilon\) represents the residuals (or error term): the difference between the observed and predicted values of \(Y\).

2. Interpreting the Coefficients

• Intercept (\(\beta_0\)):
  This is the expected value of \(Y\) when \(X = 0\). For instance, if \(\beta_0 = 5\), the model predicts that \(Y\) is 5 when there is no contribution from \(X\).

• Slope (\(\beta_1\)):
  This indicates how much \(Y\) is expected to change for each one-unit increase in \(X\). For example, if \(\beta_1 = 2.5\), then for every additional unit of \(X\), the predicted \(Y\) increases by 2.5 units.

3. Residuals

Residuals are the differences between the actual observed values and the values predicted by the model:

\[{Residual} = Y_{observed} - Y_{predicted}\]

• They provide an estimate of the error in the prediction.
  
• Analyzing residuals can help check for model adequacy; ideally, residuals should be:
  • Randomly scattered (no clear pattern).
    
  • Approximately normally distributed.
    
  • Equal in variance (homoscedasticity).

4. Interpreting Bivariate Regression Output

When you are given output from a bivariate regression analysis (such as an output table), here’s what to look for:

• Coefficient Estimates:
  • The intercept (\(\beta_0\)) and slope (\(\beta_1\)).
    
  • Their standard errors, which reflect the precision of these estimates.
• t-values and p-values:
  • Statistical tests (t-tests) are used to determine if the coefficients are significantly different from zero.
    
  • If the p-value for the slope is less than .05, you can conclude that there is a statistically significant linear relationship between \(X\) and \(Y\).
• R-squared (\(R^2\)):
  • Indicates the proportion of variance in the dependent variable that is explained by the independent variable.
    
  • For example, an \(R^2\) of 0.60 means that 60% of the variability in \(Y\) is explained by \(X\).
• Residual Standard Error (RSE):
  • Provides a measure of the typical size of the residuals (the prediction error).

5. Solving the Equation with Given Values

If you are provided with a regression equation and specific values of \(X\), you can predict \(Y\) by plugging these values into the equation.

Example:

Suppose the estimated regression equation is:

\[\hat{Y} = 10 + 3X\]

• To predict \(Y\) when \(X = 4\):

  \[ \hat{Y} = 10 + 3(4) = 10 + 12 = 22 \]

• This means when \(X=4\), the predicted value of \(Y\) is 22.

Remember:
- Always check the context to determine if the intercept is meaningful. Sometimes, \(X=0\) may be outside the range of the data.
- Consider the error (residual) as a measure of uncertainty in your prediction.

6. Tips for Exam Success

• Understand the Story:
  Be clear on what the intercept and slope mean in the given context. For instance, if you’re predicting exam scores from study hours, explain what each unit change in study hours implies for exam scores.

• Check Assumptions:
  Ensure linearity, normality, and homoscedasticity by examining residual plots (even though you don’t have to generate them in the exam, be ready to critique output).

• Report Clearly:
  Use a template such as:
  “The regression analysis revealed that for every one-unit increase in \(X\), the predicted value of \(Y\) increases/decreases by \(\beta_1\) units (p < .05). The intercept, \(\beta_0\), represents the predicted value of \(Y\) when \(X = 0\). This model explained \(R^2\) of the variance in \(Y\).”

• Practice Interpretation:
  Look at multiple examples of regression output to get comfortable with the language and interpretation before the exam.

Multiple Regression

1. The Formula

Basic multiple regression equation:

\[ Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + ... + \beta_kX_k + \varepsilon \]

Where:

• \(Y\) = Dependent/outcome variable

• \(beta_0\) = Intercept (Y-value when all Xs = 0)

• \(beta_1, \beta_2, ..., \beta_k\) = Regression coefficients (slopes) for each predictor

• \(X_1, X_2, ..., X_k\) = Independent/predictor variables

• \(\varepsilon\) = Error term (residuals)

• \(k\) = Number of predictors

2. Key Concepts

Unique Effects:

• Each coefficient (\(\beta\)) represents the unique effect of that predictor

• “Holding all other variables constant” or “controlling for other variables”

• Different from bivariate correlations because other variables are controlled

Multiple R²:

• Proportion of variance in Y explained by all predictors together

• Range: 0 to 1 (often reported as percentage)

• Larger = better overall model fit

Adjusted R²:

• Corrected R² that accounts for number of predictors

• Always smaller than Multiple R²

• Use this when comparing models with different numbers of predictors

3. Interpreting Output

For Each Predictor (X):

1. Look at coefficient (β):

   • Positive = As X increases, Y increases

   • Negative = As X increases, Y decreases

   • Size = Change in Y for one unit increase in X

2. Check p-value:

   • p < .05 = “statistically significant”

   • p > .05 = “not statistically significant”

3. Check standard error:

   • Smaller = More precise estimate
   • Larger = Less precise estimate

For Overall Model:

1. F-test and its p-value:

   • Tests if model explains significant variance

   • If p < .05, model is significant

2. R² and Adjusted R²:

   • How much variance is explained
   • Example: R² = .25 means model explains 25% of variance in Y

4. Writing Results

Template: “The multiple regression model was significant, F(df1, df2) = [value], p = [value], R² = [value]. [Significant predictor] was a significant predictor (β = [value], p = [value]), indicating that for each unit increase in [predictor], [outcome] increased/decreased by [value] units, holding other variables constant.”

Example: “The multiple regression model was significant, F(3, 96) = 15.42, p < .001, R² = .32. Study time was a significant predictor (β = 2.5, p = .003), indicating that for each additional hour of study, test scores increased by 2.5 points, holding other variables constant.”

5. Solving the Equation

Given equation: \[ Y = 10 + 2X_1 + 3X_2 - 1X_3 \]

To solve:

1. Plug in values for each X

2. Calculate Y

Example: If X₁ = 5, X₂ = 3, X₃ = 2:

\[ Y = 10 + 2(5) + 3(3) - 1(2) \]

\[ Y = 10 + 10 + 9 - 2 \] \[ Y = 27 \]

6. Common Exam Questions

1. Interpretation Questions:
   • “What happens to Y when X₁ increases by one unit?”
   • “Which predictor has the strongest effect?”
   • “How much variance is explained by the model?”
2. Calculation Questions:
   • “What is the predicted Y when X₁=4, X₂=3?”
   • “How much does Y change when X₁ increases by 2?”

7. Quick Tips

• Coefficients show unique effects (not total effects)
• Larger absolute coefficient ≠ more important predictor
• Always check both individual predictors AND overall model
• R² cannot decrease when adding predictors
• Adjusted R² can decrease with irrelevant predictors
• Remember: all effects assume other variables are held constant

8. Red Flags

Watch out for:

• Very high R² (>.90) might indicate multicollinearity

• Very different results from bivariate correlations

• Non-significant F-test but significant predictors

• Coefficients that don’t make theoretical sense

9. Assumptions

Key assumptions to remember:

• Linearity - Independence of observations

• Homoscedasticity (equal variance)

• Normality of residuals

• No perfect multicollinearity

Remember: In the exam, you might be asked to interpret output rather than check assumptions, but knowing them helps understand the limitations of your interpretations.

Categorical Coding: Dummy Coding

1. What is Dummy Coding?

• Definition: Dummy coding is a method to include categorical variables in regression analyses by converting each category (except one reference category) into a binary (0/1) variable.
  
• Purpose: Allows the model to estimate unique effects for each category relative to a baseline (reference) group.

2. How R Codes Categorical Variables (by Default)

• Default Behavior:
  • R orders levels alphabetically.
    
  • The first level in alphabetical order becomes the reference group.
    
• Example:
  • For a categorical variable teaching_method with levels interactive and traditional, R would set interactive as the reference (if “i” comes before “t”). The dummy variable is then created for traditional (1 = traditional, 0 = interactive).

3. Impact on the Intercept

• Intercept (\(\beta_0\)):
  • Represents the predicted value of the outcome when all predictors are 0.
    
  • For dummy coded variables, if the dummy variable is 0 the observation is in the reference group.
    
• Example Without Continuous Predictors:
  • Regression equation:
    \[ Y = \beta_0 + \beta_1 D \]
    
  • If D is the dummy for the non-reference group, then:
    • Intercept \(\beta_0\): Predicted outcome for the reference group.
      
    • Coefficient \(\beta_1\): The difference in the predicted outcome between the non-reference and reference groups.

4. Interpreting Regression with Two Categorical Dummy Coded Main Effects

• Model Example:
  Suppose we have two categorical predictors:

  • teaching_method (levels: interactive [reference] and traditional)
    
  • assessment_style (levels: objective [reference] and subjective)

  The regression model might look like:
  \[ Y = \beta_0 + \beta_1 D_{teaching} + \beta_2 D_{assessment} + \varepsilon \]

• Interpretation:

  • Intercept (\(\beta_0\)): The predicted outcome for the group that is in both reference categories (i.e., interactive teaching and objective assessment).
    
  • Coefficient (\(\beta_1\)) for teaching_method: The difference in the outcome between traditional and interactive teaching, holding assessment style constant.
    
  • Coefficient (\(\beta_2\)) for assessment_style: The difference in the outcome between subjective and objective assessments, holding teaching method constant.

5. Adding a Continuous Main Effect

• Extended Model:
  Including a continuous predictor, e.g., study_hours, modifies the model:
  \[ Y = \beta_0 + \beta_1 D_{teaching} + \beta_2 D_{assessment} + \beta_3 X_{study} + \varepsilon \]

• Interpretation Changes:

  • Intercept (\(\beta_0\)): Now represents the predicted outcome for the reference group (interactive teaching and objective assessment) when study_hours = 0. (Note: Check if zero is meaningful.)
    
  • Continuous Predictor (\(\beta_3\)): Represents the change in the outcome for each one-unit increase in study_hours, assuming other predictors (including dummy variables) remain constant.
    
  • Dummy Coefficients (\(\beta_1, \beta_2\)): Represent the difference in outcomes between groups, controlling for study_hours.

6. Example: Interpreting Dummy Code Output

Output Example:

Predictor	Estimate	Std. Error	t value	Pr(>
(Intercept)	50.0	2.0	25.0	< .001
teaching_methodtraditional	-8.0	2.5	-3.2	0.002
assessment_stylesubjective	5.0	2.3	2.2	0.03
study_hours	3.0	0.5	6.0	< .001

• Intercept (50.0):
  The predicted outcome for the reference group, i.e., students in the interactive teaching method with an objective assessment when study_hours = 0.

• teaching_methodtraditional (-8.0):
  When switching from interactive to traditional teaching, the outcome is expected to decrease by 8 units, holding assessment style and study hours constant.

• assessment_stylesubjective (5.0):
  Changing from objective to subjective assessment is associated with a 5-unit increase in the outcome, holding other predictors constant.

• study_hours (3.0):
  For every additional hour studied, the outcome is predicted to increase by 3 units, controlling for teaching method and assessment style.

7. Key Points to Remember

• Reference Group Matters:
  The intercept reflects the reference group defined by your dummy coding.

• Coefficients are Relative:
  Dummy coefficients show differences relative to the reference category, not the absolute effect of being in a given group.

• Model Interpretation with Continuous Variables:
  Adding a continuous predictor shifts the meaning of the intercept to the predicted outcome when that continuous variable is zero, so ensure zero is a meaningful value or consider centering.

• Always Check Output in Context:
  Interpret coefficients in light of your specific research question and data context.

Zero-Sum coding/contrasts

Effect-coding -1 and 1

1. What is Effect Coding?

Effect coding is a contrast method where category weights sum to zero (zero-sum contrasts). Unlike dummy coding (which typically uses 0/1), effect coding uses values like -1 and 1. This approach allows for comparing each category’s effect relative to the overall mean rather than a single reference group.

2. The Basic Formula with Effect Coding

For a categorical variable with two levels (e.g., A and B), effect coding can be represented as:

\[Y = \beta_0 + \beta_1X + \varepsilon\]

However, under effect coding:
- The two levels are coded as:
- Level A: -1
- Level B: 1

• Intercept (\(\beta_0\)):
  Represents the grand mean (i.e., the mean of all groups), because the contrast weights sum to zero.

• Coefficient (\(\beta_1\)):
  Represents the deviation from the grand mean for the effect coded variable. A positive \(\beta_1\) indicates that the group coded 1 is above the overall mean, while the group coded -1 is below it (by the same magnitude).

3. Effects with Two Effect-Coded Variables

Assume you have two categorical predictors that are effect coded. For example

• Predictor 1: Teaching method with two levels (interactive = -1, traditional = 1)

• Predictor 2: Assessment style with two levels (objective = -1, subjective = 1)

The multiple regression model becomes:

\[Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \varepsilon\]

Interpretation:
- Intercept (\(\beta_0\)):
The overall (grand) mean of the outcome across all groups.
- \(\beta_1\) (Teaching Method):
Half the difference between traditional and interactive. Specifically, if \(\beta_1 = 4\) then the mean for traditional is \(\beta_0 + 4\) and for interactive is \(\beta_0 - 4\).
- \(\beta_2\) (Assessment Style):
Similarly, it represents half the difference between subjective and objective assessment groups.

4. Adding a Continuous Predictor

When a continuous variable is added to an effect coded model, it looks like:

\[Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \beta_3Z + \varepsilon\]

Where:
- \(Z\) is the continuous predictor (e.g., study hours).

Interpretation:
- Intercept (\(\beta_0\)):
Represents the predicted outcome when both effect coded predictors are at their midpoint (their contributions cancel out) and the continuous variable is 0.
Note: If 0 is not a meaningful value for \(Z\), consider centering it.
- Coefficients for Effect-Coded Variables (\(\beta_1, \beta_2\)):
Indicate the effect each categorical variable has relative to the grand mean, holding the continuous predictor constant.
- Coefficient for the Continuous Variable (\(\beta_3\)):
Represents the change in the outcome for each one-unit change in \(Z\), holding the categorical effects constant.

5. Examples of Interpreting Effect Coding Output

Imagine the regression output is:

Predictor	Estimate	Std. Error	t value	Pr(>
(Intercept)	70.0	3.0	23.33	< .001
teaching_method (interactive = -1, traditional = 1)	6.0	2.0	3.00	0.004
assessment_style (objective = -1, subjective = 1)	-4.0	2.5	-1.60	0.115
study_hours	2.5	0.75	3.33	0.001

Interpretation:

• Intercept (70.0):
  This is the grand mean of the outcome when the effect coded variables are at their midpoint (i.e., the average of the two categories) and study_hours = 0 (if not centered, interpret with caution).

• Teaching Method (6.0):
  Indicates that the traditional method (coded as 1) has a predicted outcome 6 points above the grand mean, while the interactive method (coded as -1) is 6 points below the grand mean.

  • Traditional mean = \(70.0 + 6.0 = 76.0\)
    
  • Interactive mean = \(70.0 - 6.0 = 64.0\)

• Assessment Style (-4.0):
  Suggests that the subjective assessment groups (coded as 1) have a predicted outcome 4 points below the grand mean, while the objective assessment (coded as -1) is 4 points above the grand mean.

  • Objective mean = \(70.0 + 4.0 = 74.0\)
    
  • Subjective mean = \(70.0 - 4.0 = 66.0\)
    (Note: This predictor is non-significant in the provided context, as indicated by the p-value, but the interpretation approach remains the same.)

• Study Hours (2.5):
  For each additional hour of study, the outcome increases by 2.5 units, irrespective of the categorical groupings.

6. Key Takeaways

• Zero-Sum Property:
  Effect coding ensures that the contrast weights sum to zero, which shifts the interpretation of the intercept to the grand mean rather than a specific reference category.

• Comparisons are Relative:
  Each effect coded coefficient represents half the difference between its two groups relative to the grand mean.

• Inclusion of Continuous Variables:
  When adding continuous predictors, the interpretation for categorical variables remains relative to the overall mean, holding the continuous variable constant.

• Use When Appropriate:
  Effect coding is particularly useful when no single category serves as a natural reference or when the focus is on comparing group effects against an overall mean.

Manual Orthogonal Contrasts

Orthogonal contrasts compare group means using sets of weights that sum to zero and are independent (i.e., uncorrelated). Two contrasts, say \(c\) and \(d\), are orthogonal if: \[ \sum_{i=1}^{k} w_{c,i} \times w_{d,i} = 0 \] where \(k\) represents the number of groups.

Example Setup: Three Groups

Assume we have three groups with means \(\mu_1\), \(\mu_2\), and \(\mu_3\).

Contrast 1: Compare Group 1 vs. the Average of Groups 2 and 3

Using the fractional style:

• Group 1: \(1\)

• Group 2: \(-\frac{1}{2}\)

• Group 3: \(\frac{1}{2}\)

Table: Contrast 1 (c₁)

Group	Weight
\(\mu_1\)	\(1\)
\(\mu_2\)	\(-\frac{1}{2}\)
\(\mu_3\)	\(\frac{1}{2}\)

Interpretation:
A positive regression coefficient for \(c₁\) indicates that Group 1’s mean exceeds the average of Groups 2 and 3 by the value of the coefficient.

Contrast 2: Compare Group 2 vs. Group 3

Using fractional weights:

• Group 1: \(0\)

• Group 2: \(\frac{1}{2}\)

• Group 3: \(-\frac{1}{2}\)

Table: Contrast 2 (c₂)

Group	Weight
\(\mu_1\)	\(0\)
\(\mu_2\)	\(\frac{1}{2}\)
\(\mu_3\)	\(-\frac{1}{2}\)

Interpretation:
A positive coefficient for \(c₂\) means that Group 2’s mean is higher than Group 3’s mean.

Checking Orthogonality for Three-Group Contrasts

Compute the dot product of \((c₁\)) and \(c\): \[ (1 \times 0) + \left(-\frac{1}{2} \times \frac{1}{2}\right) + \left(-\frac{1}{2} \times -\frac{1}{2}\right) = 0 - \frac{1}{4} + \frac{1}{4} = 0. \] Since the sum equals 0, \(c₁\) and \(c₂\) are orthogonal.

Extended Example: Four Groups with Three Orthogonal Contrasts

Now suppose we have four groups with means \(\mu_1\), \(\mu_2\), \(\mu_3\), and \(\mu_4\). Here is one set of orthogonal contrasts using fractional weights:

Contrast A: Compare Group 1 vs. the Average of Groups 2, 3, and 4

Suggested weights:

• Group 1: \(-\frac{3}{4}\)

• Group 2: \(\frac{1}{4}\)

• Group 3: \(\frac{1}{4}\)

• Group 4: \(\frac{1}{4}\)

Table: Contrast A

Group	Weight
\(\mu_1\)	\(-\frac{3}{4}\)
\(\mu_2\)	\(\frac{1}{4}\)
\(\mu_3\)	\(\frac{1}{4}\)
\(\mu_4\)	\(\frac{1}{4}\)

Contrast B: Compare Group 2 vs. Group 3

Suggested weights:

• Group 1: \(0\)

• Group 2: \(-\frac{1}{2}\)

• Group 3: \(\frac{1}{2}\)

• Group 4: \(0\)

Table: Contrast B

Group	Weight
\(\mu_1\)	\(0\)
\(\mu_2\)	\(-\frac{1}{2}\)
\(\mu_3\)	\(\frac{1}{2}\)
\(\mu_4\)	\(0\)

Contrast C: Compare Group 4 vs. Group 2 and Group 3 (Residual Contrast)

One possible set of weights:

• Group 1: \(0\)

• Group 2: \(\frac{1}{4}\)

• Group 3: \(\frac{1}{4}\)

• Group 4: \(-\frac{1}{2}\)

Table: Contrast C

Group	Weight
\(\mu_1\)	\(0\)
\(\mu_2\)	\(\frac{1}{4}\)
\(\mu_3\)	\(\frac{1}{4}\)
\(\mu_4\)	\(-\frac{1}{2}\)

Checking Orthogonality for Four-Group Contrasts

1. Contrast A \(\cdot\) Contrast B: \[ \left(-\frac{3}{4}\times 0\right) + \left(\frac{1}{4} \times -\frac{1}{2}\right) + \left(\frac{1}{4} \times \frac{1}{2}\right) + \left(\frac{1}{4} \times 0\right) = 0 - \frac{1}{8} + \frac{1}{8} + 0 = 0. \]

2. Contrast A\(\cdot\) Contrast C:
   \[ \left(-\frac{3}{4}\times 0\right) + \left(\frac{1}{4} \times \frac{1}{4}\right) + \left(\frac{1}{4} \times \frac{1}{4}\right) + \left(\frac{1}{4} \times -\frac{1}{2}\right) = 0 + \frac{1}{16} + \frac{1}{16} - \frac{1}{8} = 0 \]

3. Contrast B \(\cdot\) Contrast C:
   \[ (0 \times 0) + \left(-\frac{1}{2} \times \frac{1}{4}\right) + \left(\frac{1}{2} \times \frac{1}{4}\right) + (0 \times -\frac{1}{2}) = 0 - \frac{1}{8} + \frac{1}{8} + 0 = 0 \]

Summary Table of All Four-Group Contrasts

Group	Contrast A	Contrast B	Contrast C	Interpretation
\[\mu_1\]	\[-\frac{3}{4}\]	\[0\]	\[0\]	A: Group 1 vs. average of Groups 2,3,4
\[\mu_2\]	\[\frac{1}{4}\]	\[-\frac{1}{2}\]	\[\frac{1}{4}\]	B: Group 2 vs. Group 3
\[\mu_3\]	\[\frac{1}{4}\]	\[\frac{1}{2}\]	\[\frac{1}{4}\]	C: Group 4 vs. average of Groups 2,3
\[\mu_4\]	\[\frac{1}{4}\]	\[0\]	\[-\frac{1}{2}\]
Sum	0	0	0	All contrasts sum to zero

Interpretation of Coefficients

For a regression model using these contrasts:
\[Y = \beta_0 + \beta_A(\text{Contrast A}) + \beta_B(\text{Contrast B}) + \beta_C(\text{Contrast C}) + \varepsilon\]

• \(\beta_A\): Difference between Group 1 and the average of Groups 2, 3, and 4
  
• \(\beta_B\): Difference between Groups 2 and 3
  
• \(\beta_C\): Difference between Group 4 and the average of Groups 2 and 3

Quick Tips for Creating Orthogonal Contrasts

1. Start with the Most Important Comparison:
   • Begin with the contrast that tests your primary hypothesis
     
   • Use fractions that sum to zero
2. Create Second Contrast:
   • Choose weights that sum to zero
     
   • Ensure dot product with first contrast equals zero
     
   • Use zeros for groups not involved in the comparison
3. Create Additional Contrasts:
   • Each new contrast must be orthogonal to ALL previous contrasts
     
   • Check orthogonality using dot products
     
   • Use fractions that make mathematical sense (e.g., \(\frac{1}{2}\), \(\frac{1}{3}\), \(\frac{1}{4}\))
4. Verify All Properties:
   • All contrast weights sum to zero
     
   • All pairs of contrasts are orthogonal (dot product = 0)
     
   • No weight is larger than 1
     
   • Denominators reflect group sizes in comparisons

Common Patterns for Contrast Weights

Comparison Type	Example Weights	Notes
One vs. One	\[(0, \frac{1}{2}, -\frac{1}{2}, 0)\]	Compare two specific groups
One vs. Rest	\[-\frac{3}{4}, \frac{1}{4}, \frac{1}{4}, \frac{1}{4}\]	Compare one group to average of others
Subset vs. Subset	\[0, \frac{1}{4}, \frac{1}{4}, -\frac{1}{2}\]	Compare average of some groups to others

Assumptions

Regression analysis relies on several assumptions that must be checked to ensure valid inferences. This section summarizes the main assumptions, the relevant diagnostic plots, and what to look for in each.

1. Key Assumptions

• Linearity:
  The relationship between predictors and the outcome is linear.
  Diagnostic: Look for a straight-line pattern in scatterplots and partially in residual plots.

• Independence of Errors:
  Observations and their errors are independent.

• Homoscedasticity:
  The variance of the residuals is constant across all levels of the predictors.
  Diagnostic: Residual versus fitted plots should display a random scatter with no clear funneling or pattern.

• Normality of Residuals:
  The residuals are normally distributed.
  Diagnostic: Q-Q plots (quantile–quantile plots) should show points roughly along a 45° line.

• No Influential Outliers/High Leverage Points:
  There are no extreme observations that unduly influence the model fit.
  Diagnostic: Leverage statistics and Cook’s distance help identify points that might be problematic.

2. Residuals and Diagnostic Plots

Residual Plot

• Purpose: Check for non-linearity and heteroscedasticity.
  
• How to Interpret:
  • Ideal: Residuals are randomly scattered around 0 without a discernible pattern.
  • Red Flags:
    • A funnel or cone shape indicates heteroscedasticity (non-constant variance).
    • Curved patterns suggest non-linear relationships.

Q-Q Plot (Quantile-Quantile Plot)

• Purpose: Assess the normality of residuals.
• How to Interpret:
  • Ideal: Points fall approximately on the 45° (diagonal) line.
  • Red Flags:
    • Systematic deviations (e.g., S-shaped or curved patterns) suggest that residuals depart from normality.

Leverage and Influential Points

• Leverage:
  Measures how far an observation’s predictor values are from those of other observations.
  • High Leverage Point: An observation with extreme predictor values can have a disproportionate effect on the model.
• Cook’s Distance:
  Combines information on leverage and the residual size to indicate overall influence.
  • Red Flags: Points with unusually high leverage or Cook’s distance should be examined as potential outliers that might distort model estimates.

3. Heteroscedasticity vs. Homoscedasticity

• Homoscedasticity:
  • Residuals have roughly equal spread across predicted values.
  • Plot: Residual vs. Fitted plot shows a random cloud with equal spread.
• Heteroscedasticity:
  • Variance of residuals increases or decreases with fitted values (non-constant error spread).
  • Plot: Residual vs. Fitted plot shows a pattern (e.g., a funnel shape, where the spread widens as fitted values increase).

4. Practical Tips

• Evaluate Plots Together:
  No single plot tells the whole story. Use the residual plot, Q-Q plot, and leverage statistics in conjunction.

• Transformations:
  If assumptions are violated (e.g., heteroscedasticity or non-normality), consider data transformations (like logarithmic transformations) or robust regression methods.

• Document Findings:
  In exam responses or reports, clearly state which assumptions were checked, what the plots showed, and if any issues were identified (and how you might address them).

• Red Flag Examples:

  • Residual Plot: A visible funnel shape suggests heteroscedasticity.
  • Q-Q Plot: Deviations from the line in the tails signal non-normal residuals.
  • Leverage: An observation standing out on a leverage plot should be scrutinized for undue influence.

These plots effectively illustrate the red flag examples of diagnostic issues:

• Heteroscedasticity: Unequal variance in residuals.

• Non-linearity: Curved residual patterns.

• Outliers/High Leverage: Extreme observations that could unduly influence the estimates.

R^2 and F for model evaluation

Model Comparison

Bootstrapping

What is Bootstrapping?

• A resampling method that creates multiple datasets by sampling with replacement from original data
• Provides empirical estimates of standard errors and confidence intervals
• Useful when:
  • Sample size is small
  • Distribution assumptions are violated
  • Traditional inference methods may be unreliable
  • Want to estimate sampling distribution empirically

Key Formula

Number of possible bootstrap samples: \[ \text{Number of possible samples} = \binom{n+n-1}{n} = \frac{(2n-1)!}{n!(n-1)!} \] where n is the sample size

2. Types of Bootstrap Methods

Non-Parametric Bootstrap

1. Process:
   • Resample directly from observed data points
   • Sample n observations with replacement
   • Fit model to resampled data
   • Store parameter estimates
   • Repeat B times (typically 1000-10000)
2. Advantages:
   • No distributional assumptions
   • Maintains original data structure
   • Captures true data variability

Parametric Bootstrap

1. Process:
   • Fit initial model to data
   • Generate new Y values from fitted model
   • Resample from generated values
   • Refit model and store parameters
2. When to Use:
   • Known underlying distribution
   • Want to incorporate model assumptions
   • More efficient if assumptions are correct

3. Bootstrap Confidence Intervals

Methods for CI Construction

1. Percentile Method
   • Sort bootstrap estimates
   • Take 2.5th and 97.5th percentiles for 95% CI \[ CI_{95\%} = [\hat{\theta}_{(0.025B)}, \hat{\theta}_{(0.975B)}] \]
2. BCa (Bias-Corrected and Accelerated)
   • Adjusts for bias and skewness
   • More accurate than percentile method
   • Uses two correction factors:
     • z₀: bias correction
     • a: acceleration factor
3. Normal Theory Method
   • Uses standard error of bootstrap distribution \[ CI_{95\%} = \hat{\theta} \pm 1.96 \times SE_{boot} \]

4. Key Formulas and Calculations

Standard Error

\[ SE_{boot} = \sqrt{\frac{\sum_{i=1}^B (\hat{\theta}_i - \bar{\theta})^2}{B-1}} \]

Bootstrap Mean

\[ \bar{\theta} = \frac{\sum_{i=1}^B \hat{\theta}_i}{B} \]

Bias Estimate

\[ \text{Bias} = \bar{\theta} - \hat{\theta}_{original} \]

5. Rules of Thumb

Number of Bootstrap Replicates (B)

• Minimum: 1000 for standard errors
• Recommended: 2000-5000 for confidence intervals
• BCa intervals: 5000-10000

When to Use Bootstrap

1. Sample Size Considerations:
   • Especially useful when n < 30
   • Valuable for non-normal data at any sample size
2. Model Diagnostics:
   • Heteroscedasticity present
   • Non-normal residuals
   • Influential observations
3. Warning Signs:
   • Large bias (bootstrap mean far from original estimate)
   • Extremely wide confidence intervals
   • Highly asymmetric confidence intervals

Practice Questions

Question 1: Bootstrap Process

Q: Outline the steps for performing a non-parametric bootstrap for a linear regression coefficient.

A:

1. From original dataset (n observations):

   • Sample n observations with replacement

   • Fit regression model

   • Store coefficient estimate

2. Repeat process B times (e.g., 2000)

3. Calculate: Bootstrap SE using stored estimates

   • 95% CI using percentiles - Check distribution shape

Question 2: Interpreting Results

Q: Given these bootstrap results for a slope coefficient: - Original estimate: 0.35

Bootstrap mean: 0.34

Bootstrap SE: 0.12

95% CI: [0.11, 0.58]

What can you conclude?

A:

1. Minimal bias (original 0.35 vs bootstrap 0.34)

2. Significant effect (CI excludes 0)

3. Moderate uncertainty (SE = 0.12)

4. Relatively symmetric CI suggests normality

Question 3: Choosing Methods

Q: When would you choose BCa intervals over percentile intervals?

A:

1. When there’s evidence of bias in estimates

2. Small sample sizes (n < 50)

3. When distribution of bootstrap estimates is skewed

4. When more precise confidence intervals are needed

5. When computational resources allow for larger B

Common Exam Pitfalls to Avoid

1. Confusing sampling with vs. without replacement
2. Forgetting to check for bootstrap bias
3. Using too few bootstrap replicates
4. Not considering computational efficiency
5. Misinterpreting bootstrap confidence intervals

Remember: Bootstrap is a tool for understanding uncertainty in estimates, not a way to improve the estimates themselves!

Logistic Regression

Logistic regression is ideal for modeling binary outcomes. Below are the key concepts, interpretation guidelines, and exam questions to help you understand logistic regression.

1. Model Formulation

Logistic Function

• Probability Model:
  The model estimates the probability that an event occurs:
  \[ p = \Pr(Y=1 | X) = \frac{e^{\eta}}{1 + e^{\eta}} \]
  
• Linear Predictor:
  The log-odds (logit) is modeled as:
  \[ \eta = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_k X_k \]

Logit Transformation

• Definition:
  The logit (log-odds) is given by:
  \[ \text{logit}(p) = \log\left(\frac{p}{1-p}\right) = \eta \]
  
• This transformation allows you to interpret coefficients directly on the log-odds scale.

2. Interpretation of Coefficients

Intercept (\(\beta_0\))

• Meaning:
  The intercept is the log-odds of the outcome when all predictors equal zero.
  
• Conversion:
  Convert the intercept to probability using:
  \[ p = \frac{e^{\beta_0}}{1 + e^{\beta_0}} \]

Slope Coefficients (\(\beta_i\))

• Log-Odds Change:
  A one-unit increase in $ X_i $ changes the log-odds of the outcome by \(\beta_i\).
  
• Odds Ratio (OR):
  The odds ratio is calculated as:
  $ = e^{_i} $
  • When \(\beta_i > 0\), the odds of the outcome increase.
    
  • When \(\beta_i < 0\), the odds of the outcome decrease.

Confidence Intervals for Coefficients

• 95% Confidence Interval (CI):
  A 95% CI for a coefficient \(\beta_i\) provides a range in which the true log-odds change is likely to lie. After exponentiating, the CI gives an interval for the odds ratio.
  
• Example:
  If \(\beta_1 = 0.8\) with a 95% CI of \((0.3, 1.3)\), then:
  • Odds Ratio \(= e^{0.8} \approx 2.23\)
    
  • CI for OR: \((e^{0.3}, e^{1.3}) \approx (1.35, 3.67)\)

3. Example Output Interpretation

Consider the following output table from a logistic regression:

Predictor	Coefficient (\(\beta\))	95% CI (log-odds)	Odds Ratio (OR)	95% CI (OR)	p-value
(Intercept)	-1.2	(-1.8, -0.6)	0.30	(0.17, 0.55)	<0.001
Age	0.05	(0.01, 0.09)	1.05	(1.01, 1.09)	0.02
Income	0.002	(0.0005, 0.0035)	1.002	(1.0005, 1.0035)	0.01
Smoker (1=Yes)	0.7	(0.3, 1.1)	2.01	(1.35, 3.00)	0.001

How to Interpret This Output:

• Intercept:
  The log-odds of the outcome when Age = 0, Income = 0, and the subject is not a smoker is -1.2. The corresponding probability is:
  \[ p = \frac{e^{-1.2}}{1+e^{-1.2}} \approx 0.23. \]
  

• Age:
  For every one-year increase in age, the log-odds increase by 0.05. The odds increase by about 5% (OR = 1.05).
  

• Income:
  Each unit increase in income (appropriately scaled) increases the log-odds by 0.002. This is a very small effect on the odds.
  

• Smoker:
  Smokers have log-odds that are 0.7 higher than non-smokers. The odds of the outcome are about twice as high for smokers (OR ≈ 2.01).

• Confidence Intervals:
  The 95% CI for each predictor indicates the range in which we are 95% confident that the true coefficient (or OR) lies. For example, the OR for smoker falls between 1.35 and 3.00, showing a robust increase in odds.

4. Model Assumptions and Diagnostics

Key Assumptions:

• Independence of Observations:
  Each observation should be independent.
  
• Correct Specification:
  The logit is linearly related to the predictors.
  
• Absence of Multicollinearity:
  Predictors should not be highly correlated.

Diagnostics:

• Use likelihood ratio tests or Wald tests for overall model significance.
  
• Assess goodness-of-fit with Hosmer-Lemeshow or similar tests.
  
• Examine classification tables and the ROC curve for predictive performance.

5. Potential Exam Questions (Practice)

1. Coefficient Interpretation:
   • What is the interpretation of a slope coefficient of 0.05 in terms of odds in a logistic regression model?
     
   • How would you convert a log-odds coefficient into an odds ratio?

Show Answer

Q: What is the interpretation of a slope coefficient of 0.05 in terms of odds?
Answer:
1. First, calculate the odds ratio:
\[e^{0.05} = 1.051\]
2. Interpretation:
- For a one-unit increase in the predictor, the odds of the outcome multiply by 1.051
- This represents a 5.1% increase in the odds
- Since the coefficient is small, we can also say approximately a 5% increase in odds

Q: How would you convert a log-odds coefficient into an odds ratio?
Answer:
1. Take the exponential (e^x) of the coefficient
2. Example steps:
- If coefficient β = 0.7
- Odds ratio = e^0.7 = 2.014
- The odds are approximately doubled for a one-unit increase in the predictor

2. Confidence Intervals:
   • Given a coefficient of 0.7 for a binary predictor with a 95% CI of (0.3, 1.1), interpret the meaning of this confidence interval.
     
   • How does exponentiating a confidence interval help in interpreting odds ratios?

Show Answer

Q: Given a coefficient of 0.7 for a binary predictor with a 95% CI of (0.3, 1.1), interpret this confidence interval.
Answer:
1. Convert to odds ratios:
- Lower bound: e^0.3 = 1.35
- Coefficient: e^0.7 = 2.01
- Upper bound: e^1.1 = 3.00

2. Interpretation:
   • We are 95% confident that the true odds ratio lies between 1.35 and 3.00
     
   • Since the CI doesn’t include 1, this is a statistically significant effect
     
   • At minimum, the odds increase by 35% (lower bound)
     
   • At maximum, the odds triple (upper bound)

Q: How does exponentiating a confidence interval help in interpreting odds ratios?
Answer:
1. Exponentiating transforms log-odds to odds ratios, which are more intuitive
2. Key benefits:
- Odds ratios are always positive
- Value of 1 indicates no effect
- Values > 1 indicate increased odds
- Values < 1 indicate decreased odds
3. Example: CI of (-0.5, 0.3) becomes (e^-0.5, e^0.3) = (0.61, 1.35)
- Interpretation: We’re 95% confident the true odds ratio is between 0.61 and 1.35
- Since this CI includes 1, the effect is not statistically significant

3. Probability vs. Odds:
   • Explain why logistic regression models the log-odds instead of directly modeling probabilities.
     
   • If the intercept in a model is -1.2, what is the corresponding predicted probability of the outcome when all predictors equal zero?

Show Answer

Q: Explain why logistic regression models the log-odds instead of directly modeling probabilities.
Answer:
1. Key reasons:
- Log-odds can take any real value (-∞ to +∞)
- Probabilities are constrained between 0 and 1
- Log-odds transformation creates a linear relationship with predictors
- Coefficients can be interpreted as additive effects on log-odds scale

2. Mathematical advantages:
   • Linear relationship makes estimation simpler
     
   • Avoids predicted probabilities outside [0,1]
     
   • Allows for multiplicative effects on odds scale

Q: If the intercept in a model is -1.2, what is the corresponding predicted probability when all predictors equal zero?
Answer:
1. Steps to convert log-odds to probability:
\[p = \frac{e^{-1.2}}{1 + e^{-1.2}}\]

2. Calculation:
   • e^-1.2 = 0.301
     
   • p = 0.301/(1 + 0.301) = 0.231
3. Interpretation:
   • When all predictors are zero, the predicted probability is 0.231 or 23.1%

4. Model Diagnostics:
   • List and explain at least two diagnostic methods to assess the goodness-of-fit in logistic regression.
     
   • How would you interpret an ROC curve in the context of logistic regression?

Show Answer

Q: List and explain at least two diagnostic methods to assess goodness-of-fit.
Answer:
1. Hosmer-Lemeshow Test:
- Groups observations by predicted probabilities
- Compares observed vs. expected frequencies
- Non-significant p-value indicates good fit
- Look for p > 0.05 to suggest adequate fit

2. Classification Table:
   • Shows predicted vs. actual outcomes
     
   • Calculates sensitivity and specificity
     
   • Overall accuracy = (True Positives + True Negatives)/Total
     
   • Good for assessing predictive performance

Q: How would you interpret an ROC curve?
Answer:
1. Key aspects to examine:
- Area Under Curve (AUC) ranges from 0.5 to 1.0
- AUC = 0.5 suggests no discrimination
- AUC > 0.7 indicates acceptable discrimination
- AUC > 0.8 indicates excellent discrimination

2. Interpretation guidelines:
   • Plot shows trade-off between sensitivity and specificity
     
   • Curve closer to top-left corner indicates better model
     
   • Can use to select optimal probability cutoff

5. Application-based Question:
   • Given a logistic regression output table, practice explaining the output in plain language and relate how changes in predictors affect the outcome probability.

Show Answer

Answer:
1. Intercept (-2.300):
- Log-odds when all predictors are zero
- Probability = e^-2.3/(1 + e^-2.3) = 0.091
- 9.1% probability for non-smokers at age zero

2. Age (0.045):
   • For each year increase in age:
     
   • Odds ratio = e^0.045 = 1.046
     
   • Odds increase by 4.6%
3. Smoking (0.693):
   • Being a smoker vs. non-smoker:
     
   • Odds ratio = e^0.693 = 2.000
     
   • Odds double for smokers compared to non-smokers
4. Overall interpretation:
   • Both age and smoking significantly predict the outcome
     
   • Strongest effect is smoking (doubles odds)
     
   • Age has a smaller but significant cumulative effect

Interpreting Odds Ratios

Basic Formula for Percentage Change

For any odds ratio (OR):
- Percentage change = (OR - 1) × 100%
- For ORs < 1: Percentage change = (1 - OR) × 100% decrease

1. When OR > 1 (Positive Association)

Formula

\[ \text{Percentage increase} = (OR - 1) \times 100\% \]

Examples

# Example output  
Predictor   OR    95% CI  
Age         1.25  (1.10, 1.42)  
SmokingYes  2.50  (1.80, 3.47)  
Exercise    1.75  (1.25, 2.45) 

Interpretations:

Age: OR = 1.25 (1.25 - 1) × 100% = 25% increase in odds Smoking: OR = 2.50 (2.50 - 1) × 100% = 150% increase in odds Exercise: OR = 1.75 (1.75 - 1) × 100% = 75% increase in odds

Titanic

Example

library(performance)
titanic <- carData::TitanicSurvival
titanic$survived_binary <- as.numeric(titanic$survived) - 1

mod.3 <- lm(survived_binary ~ sex, titanic)
summary(mod.3)

## 
## Call:
## lm(formula = survived_binary ~ sex, data = titanic)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -0.7275 -0.1910 -0.1910  0.2725  0.8090 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  0.72747    0.01912   38.05   <2e-16 ***
## sexmale     -0.53648    0.02382  -22.52   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.4127 on 1307 degrees of freedom
## Multiple R-squared:  0.2795, Adjusted R-squared:  0.279 
## F-statistic: 507.1 on 1 and 1307 DF,  p-value: < 2.2e-16

check_model(mod.3)

mod.4 <- glm(survived_binary ~ sex + age, na.omit(titanic), 
             family = "binomial")
summary(mod.4)

## 
## Call:
## glm(formula = survived_binary ~ sex + age, family = "binomial", 
##     data = na.omit(titanic))
## 
## Coefficients:
##              Estimate Std. Error z value Pr(>|z|)    
## (Intercept)  1.235414   0.192032   6.433 1.25e-10 ***
## sexmale     -2.460689   0.152315 -16.155  < 2e-16 ***
## age         -0.004254   0.005207  -0.817    0.414    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 1414.6  on 1045  degrees of freedom
## Residual deviance: 1101.3  on 1043  degrees of freedom
## AIC: 1107.3
## 
## Number of Fisher Scoring iterations: 4

m4_null <- glm(survived_binary ~ 1, family = "binomial", na.omit(titanic))
anova(m4_null, mod.4, test= "Chisq")

## Analysis of Deviance Table
## 
## Model 1: survived_binary ~ 1
## Model 2: survived_binary ~ sex + age
##   Resid. Df Resid. Dev Df Deviance  Pr(>Chi)    
## 1      1045     1414.6                          
## 2      1043     1101.3  2   313.28 < 2.2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Odds Ratios

exp(coef(mod.4))

## (Intercept)     sexmale         age 
##  3.43980286  0.08537609  0.99575479

(0.08537609 - 1) * 100

## [1] -91.46239

(0.99575479  - 1) * 100

## [1] -0.424521

For sex, being Male leads to 91% decreased odds of survival.

For every one unit increase in age, the odds of survival decrease by 0.43%. (nbn-significant)

exp(confint(mod.4))

## Waiting for profiling to be done...

##                  2.5 %    97.5 %
## (Intercept) 2.37427561 5.0440368
## sexmale     0.06307172 0.1146265
## age         0.98559653 1.0059398

m4_null <- glm(survived_binary ~ 1, family = "binomial", na.omit(titanic))
m4_sex <- glm(survived_binary ~ sex, family = "binomial", na.omit(titanic))
m4_full <- glm(survived_binary ~ sex + age, family = "binomial", na.omit(titanic))

AIC(m4_null, m4_sex, m4_full)

##         df      AIC
## m4_null  1 1416.620
## m4_sex   2 1106.008
## m4_full  3 1107.339

BIC(m4_null, m4_sex, m4_full)

##         df      BIC
## m4_null  1 1421.573
## m4_sex   2 1115.914
## m4_full  3 1122.197

Power

What is Statistical Power?

• The probability of correctly rejecting a false null hypothesis
• Power = P(Reject H₀ | H₀ is false)
• Power = 1 - β (where β is Type II error rate)
• Typically aim for power of 0.80 (80%) or higher

Key Components

1. Effect Size (d): Magnitude of the difference/relationship
2. Sample Size (n): Number of observations
3. Alpha (α): Type I error rate (usually 0.05)
4. Power (1-β): Probability of detecting true effect

Relationship Formula

\[ \text{Power} = f(\text{Effect Size}, \text{Sample Size}, \alpha) \]

Types of Effect Sizes

For Mean Differences

1. Cohen’s d: \[ d = \frac{\mu_1 - \mu_2}{\sigma_{pooled}} \] where σ_pooled is the pooled standard deviation

2. Standardized Guidelines:

   • Small effect: d = 0.2
   • Medium effect: d = 0.5
   • Large effect: d = 0.8

For Correlations

1. Pearson’s r:
   • Small effect: r = 0.1
   • Medium effect: r = 0.3
   • Large effect: r = 0.5

For ANOVA

1. Eta-squared (η²): \[ \eta^2 = \frac{SS_{effect}}{SS_{total}} \]
   • Small effect: η² = 0.01
   • Medium effect: η² = 0.06
   • Large effect: η² = 0.14

Power Analysis Calculations

Sample Size Determination

1. Required inputs:
   • Desired power (typically 0.80)
   • Expected effect size
   • Alpha level (typically 0.05)
   • Test type (one/two-tailed)
2. Formula for two independent means: \[ n = 2\left(\frac{(z_α + z_β)}{d}\right)^2 \] where:
   • z_α is the critical value for α
   • z_β is the critical value for β
   • d is Cohen’s d

Power Curves

• Show relationship between power and sample size
• Y-axis: Power (0 to 1)
• X-axis: Sample size
• Different curves for different effect sizes

Common Research Scenarios

Independent Samples t-test

1. Required sample size per group: \[ n_{per group} = 2\left(\frac{(z_α + z_β)}{d}\right)^2 \]

2. Total sample size: \[ N_{total} = 2n_{per group} \]

Paired Samples t-test

• Generally requires smaller sample size than independent
• Accounts for correlation between measurements
• Power increases as correlation increases

ANOVA

1. Required inputs:
   • Number of groups
   • Expected effect size (η²)
   • Desired power
   • Alpha level
2. Considerations:
   • Balance between groups
   • Post-hoc comparisons
   • Multiple testing corrections

Practical Guidelines

Planning Studies

1. A Priori Power Analysis:
   • Conduct before data collection
   • Determine required sample size
   • Consider feasibility constraints
2. Post Hoc Power Analysis:
   • Calculate achieved power
   • Interpret non-significant results
   • Limited utility - use with caution

Common Mistakes to Avoid

1. Sample Size:
   • Not accounting for attrition
   • Ignoring practical constraints
   • Using rules of thumb without power analysis
2. Effect Size:
   • Unrealistic effect size estimates
   • Not considering previous research
   • Ignoring practical significance

Practice Questions

Question 1: Power Calculation

Q: A researcher wants to detect a medium effect size (d = 0.5) using an independent samples t-test. What sample size per group is needed for 80% power with α = 0.05?

A: 1. Using the formula: \[ n_{per group} = 2\left(\frac{(1.96 + 0.84)}{0.5}\right)^2 \] 2. n ≈ 64 per group 3. Total N needed = 128

Question 2: Effect Size Interpretation

Q: Given these values from a study:

- Mean Group 1 = 75

- Mean Group 2 = 65

- Pooled SD = 20

Calculate and interpret Cohen’s d.

A: 1. d = (75 - 65)/20 = 0.5

2. This represents a medium effect size

3. Means differ by half a standard deviation

Question 3: Power Analysis Scenario

Q: A study found no significant difference between groups (p = 0.08). Sample size was 20 per group. What issues might exist regarding statistical power?

A:

1. Likely underpowered study

2. Small sample size increases Type II error risk

3. Should calculate achieved power

4. Consider effect size and practical significance

5. Results might be inconclusive rather than “no effect”

Key Formulas Summary

Power Calculations

\[ \text{Power} = 1 - \beta \] \[ \text{Required N} = f(\alpha, \text{power}, \text{effect size}) \]

Effect Sizes

\[ d = \frac{\mu_1 - \mu_2}{\sigma_{pooled}} \] \[ \eta^2 = \frac{SS_{effect}}{SS_{total}} \]

Critical Values

• z_α (one-tailed, α = 0.05) = 1.645
• z_α (two-tailed, α = 0.05) = 1.96
• z_β (power = 0.80) = 0.84

Remember:

- Always conduct power analysis before data collection

- Consider practical and statistical significance

- Report effect sizes alongside p-values

- Use appropriate effect size measures for your analysis

What we have left to cover:

• cat x cat Interactions

• Power

• Aff R2/F/Model comparison to document