AI Experiment Analysis

Loading Libraries

library(afex) # to run the ANOVA and plot results
library(psych) # for the describe() command
library(ggplot2) # to visualize our results
library(expss) # for the cross_cases() command
library(car) # for the leveneTest() command
library(emmeans) # for posthoc tests
library(effsize) # for the cohen.d() command
library(apaTables) # to create our correlation table
library(kableExtra) # to create our correlation table
library(sjPlot) # to visualize our results

Importing Data

# import your AI results dataset
d <- read.csv(file="Data/resultsfinal.csv", header=T)

State Your Hypotheses & Chosen Tests

Check Your Variables

This is just basic variable checking that is used across all HW assignments.

# # to view stats for all variables
 describe(d)
           vars   n  mean    sd median trimmed   mad  min   max range  skew
id            1 100 50.50 29.01  50.50   50.50 37.06  1.0 100.0  99.0  0.00
identity*     2 100 50.50 29.01  50.50   50.50 37.06  1.0 100.0  99.0  0.00
consent*      3 100  1.97  0.95   2.00    1.96  1.48  1.0   3.0   2.0  0.06
age           4 100 41.17 14.10  35.00   39.88  9.64 17.0  80.0  63.0  0.90
race          5 100  4.93  1.46   6.00    5.01  0.00  2.0   7.0   5.0 -0.50
gender        6 100  1.98  0.57   2.00    2.00  0.00  1.0   7.0   6.0  6.54
manip_out*    7 100 50.50 29.01  50.50   50.50 37.06  1.0 100.0  99.0  0.00
survey1       8 100  3.39  0.22   3.50    3.43  0.00  2.5   3.8   1.3 -1.68
survey2       9 100  4.47  0.32   4.33    4.48  0.49  3.0   5.0   2.0 -1.19
ai_manip*    10 100 50.50 29.01  50.50   50.50 37.06  1.0 100.0  99.0  0.00
Condition    11 100  1.50  0.50   1.50    1.50  0.74  1.0   2.0   1.0  0.00
           kurtosis   se
id            -1.24 2.90
identity*     -1.24 2.90
consent*      -1.90 0.09
age            0.34 1.41
race          -1.48 0.15
gender        58.56 0.06
manip_out*    -1.24 2.90
survey1        2.41 0.02
survey2        4.20 0.03
ai_manip*     -1.24 2.90
Condition     -2.02 0.05
# 
# # we'll use the describeBy() command to view skew and kurtosis across our IVs
# describeBy (d, group = Condition)
# 
# # also use histograms and scatterplots to examine your continuous variables
 hist(d$survey1)

 plot(d$survey1, d$survey2)

# 
# # and table() and cross_cases() to examine your categorical variables
# # you may not need the cross_cases code
 table(d$survey1)

 2.5 2.73    3 3.06  3.1 3.13 3.14  3.2  3.4 3.47  3.5  3.6  3.8 
   1    1   12    1    1    1    1    5    2    1   72    1    1 
 cross_cases(d, survey1, survey2)
 survey2 
 3   3.33   4   4.33   4.66   4.67   5 
 survey1 
   2.5  1
   2.73  1
   3  1 5 1 3 2
   3.06  1
   3.1  1
   3.13  1
   3.14  1
   3.2  1 2 2
   3.4  2
   3.47  1
   3.5  1 1 6 30 1 26 7
   3.6  1
   3.8  1
   #Total cases  1 1 8 44 2 35 9 
# 
# # and boxplot to examine any categorical variables with continuous variables
 boxplot(d$survey1~d$Condition)

# 
# #convert any categorical variables to factors
# d$variable <- as.factor(d$variable)

Check Your Assumptions

t-Test Assumptions

  • Data values must be independent (independent t-test only) (confirmed by data report)
  • Data obtained via a random sample (confirmed by data report)
  • IV must have two levels (will check below)
  • Dependent variable must be normally distributed (will check below. if issues, note and proceed)
  • Variances of the two groups must be approximately equal, aka ‘homogeneity of variance’. Lacking this makes our results inaccurate (will check below - this really only applies to Student’s t-test, but we’ll check it anyway)

Checking IV levels

# # preview the levels and counts for your IV
# table(d$iv, useNA = "always")
# 
# # note that the table() output shows you exactly how the levels of your variable are written. when recoding, make sure you are spelling them exactly as they appear
# 
# # to drop levels from your variable
# # this subsets the data and says that any participant who is coded as 'BAD' should be removed
# d <- subset(d, IV != "BAD")
# 
# table(d$iv, useNA = "always")
# 
# # to combine levels
# # this says that where any participant is coded as 'BAD' it should be replaced by 'GOOD'
# d$iv_rc[d$iv == "BAD"] <- "GOOD"
# 
# table(d$iv, useNA = "always")
# 
# # check your variable types
# str(d)
# 
# # make sure that your IV is recognized as a factor by R
# # if you created a new _rc variable make sure to use that one instead
# d$iv <- as.factor(d$iv)

Testing Homogeneity of Variance with Levene’s Test

We can test whether the variances of our two groups are equal using Levene’s test. The null hypothesis is that the variance between the two groups is equal, which is the result we want. So when running Levene’s test we’re hoping for a non-significant result!

# # use the leveneTest() command from the car package to test homogeneity of variance
# # uses the same 'formula' setup that we'll use for our t-test: formula is y~x, where y is our DV and x is our IV
# leveneTest(dv~iv, data = d)
# # Cook's distance
# plot(reg_model, 4)
# 
# # Residuals vs Leverage
# plot(reg_model, 5)

Check homogeneity of variance in a Scale-Location plot

You can check out this page for some other examples of this type of plot. (Notice that the Scale-Location plot is the third in the grids.)

For your homework, you’ll simply need to generate this plot and talk about how your plot compares to the ones pictured. Is it closer to the ‘good’ plots or one of the ‘bad’ plots? Again, this is a judgement call! It’s okay if feel uncertain, and you won’t be penalized for that.

# plot(reg_model, 3)

Multiple Linear Regression Assumptions

  • Observations should be independent (confirmed by data report)
  • Number of cases should be adequate (N ≥ 80 + 8m, where m is the number of IVs). If you don’t have enough, it won’t run. (will check this below)
  • Independent variables should not be too correlated (aka multicollinearity). (will check this below)
  • Relationship between the variables should be linear. (will check this below)
  • Outliers should be identified and removed. (will check this below)
  • Residuals should be normally distributed and have constant variance. (will check this below)

Check Number of Cases

For your homework, if you don’t have the required number of cases you’ll need to drop one of your independent variables. Reach out to me if this happens and we can figure out the best way to proceed!

# needed <- 80 + 8*3
# nrow(d) >= needed

Check multicollinearity

  • Higher values indicate more multicollinearity. This usually requires dropping a variable. For your homework, you will need to discuss multicollinearity and any high values, but you don’t have to drop any variables.
  • Cutoff is usually 5
# vif(reg_model)

Check linearity with Residuals vs Fitted plot

For some examples of good Residuals vs Fitted plot and ones that show serious errors, check out this page.

For your homework, you’ll simply need to generate this plot and talk about how your plot compares to the good and problematic plots linked to above. Is it closer to the ‘good’ plots or one of the ‘bad’ plots? This is going to be a judgement call, and that’s okay! In practice, you’ll always be making these judgement calls as part of a team, so this assignment is just about getting experience with it, not making the perfect call.

# plot(reg_model, 1)

Check for outliers using Cook’s distance and a Residuals vs Leverage plot

For your homework, you’ll simply need to generate these plots, assess Cook’s distance in your dataset, and then identify any potential cases that are prominent outliers. Since we have some cutoffs, that makes this process is a bit less subjective than some of the other assessments we’ve done here, which is a nice change!

# # Cook's distance
# plot(reg_model, 4)
# 
# # Residuals vs Leverage
# plot(reg_model, 5)

Check homogeneity of variance in a Scale-Location plot

You can check out this page for some other examples of this type of plot. (Notice that the Scale-Location plot is the third in the grids.)

For your homework, you’ll simply need to generate this plot and talk about how your plot compares to the ones pictured. Is it closer to the ‘good’ plots or one of the ‘bad’ plots? Again, this is a judgement call! It’s okay if feel uncertain, and you won’t be penalized for that.

# plot(reg_model, 3)

Check normality of residuals with a Q-Q plot

This page shows how different types of non-normality appear on a Q-Q plot. It’s normal for Q-Q plots show a bit of deviation at the ends.

This page also shows some examples that help us put our Q-Q plot into context. Although it isn’t perfect, we don’t have any serious issues and are okay to proceed.

For your homework, you’ll simply need to generate this plot and talk about how your plot compares to the ones pictured. Does it seem like any skew or kurtosis is indicated by your plot? Is it closer to the ‘good’/‘bad’ plots from the second link?

# plot(reg_model, 2)

Issues with My Data

Describe any issues and why they’re problematic here.

Run Your Analysis

Run a t-Test

# # very simple! we specify the dataframe alongside the variables instead of having a separate argument for the dataframe like we did for leveneTest()
 t_output <- t.test(d$survey1~d$Condition)
t_output2 <- t.test(d$survey2~d$Condition)

View Test Output

 t_output

    Welch Two Sample t-test

data:  d$survey1 by d$Condition
t = 0.52895, df = 94.885, p-value = 0.5981
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
 -0.06332412  0.10932412
sample estimates:
mean in group 1 mean in group 2 
         3.4048          3.3818 
 t_output2

    Welch Two Sample t-test

data:  d$survey2 by d$Condition
t = -0.21198, df = 85.656, p-value = 0.8326
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
 -0.1411462  0.1139462
sample estimates:
mean in group 1 mean in group 2 
         4.4594          4.4730

Calculate Cohen’s d

# # once again, we use our formula to calculate cohen's d
 d_output <- cohen.d(d$survey1~d$Condition)
Warning in cohen.d.formula(d$survey1 ~ d$Condition): Cohercing rhs of formula
to factor
d_output2 <- cohen.d(d$survey2~d$Condition)
Warning in cohen.d.formula(d$survey2 ~ d$Condition): Cohercing rhs of formula
to factor

View Effect Size

  • Trivial: < .2
  • Small: between .2 and .5
  • Medium: between .5 and .8
  • Large: > .8
 d_output

Cohen's d

d estimate: 0.1057908 (negligible)
95 percent confidence interval:
     lower      upper 
-0.2913802  0.5029618 
 d_output2

Cohen's d

d estimate: -0.04239634 (negligible)
95 percent confidence interval:
     lower      upper 
-0.4393344  0.3545417 
# aov_model <- aov_ez(data = d,
#                     id = "X",
#                     between = c("IV1"),
#                     dv = "pss",
#                     anova_table = list(es = "pes"))
# 
# aov_model2 <- aov_ez(data = d2,
#                     id = "X",
#                     between = c("IV1","IV2"),
#                     dv = "pss",
#                     anova_table = list(es = "pes"))

View Output

Effect size cutoffs from Cohen (1988):

  • η2 = 0.01 indicates a small effect
  • η2 = 0.06 indicates a medium effect
  • η2 = 0.14 indicates a large effect
# nice(aov_model)
# 
# nice(aov_model2)

Visualize Results

# afex_plot(aov_model, x = "IV1")
# 
# afex_plot(aov_model2, x = "IV1", trace = "IV2")
# afex_plot(aov_model2, x = "IV2", trace = "IV1")

Run Posthoc Tests (One-Way)

Only run posthocs if the test is significant! E.g., only run the posthoc tests on gender if there is a main effect for gender.

# emmeans(aov_model, specs="IV1", adjust="tukey")
# pairs(emmeans(aov_model, specs="IV1", adjust="tukey"))

Run Posthoc Tests (Two-Way)

Only run posthocs if the test is significant! E.g., only run the posthoc tests on gender if there is a main effect for gender.

# emmeans(aov_model, specs="IV1", adjust="tukey")
# pairs(emmeans(aov_model, specs="IV1", adjust="tukey"))
#  
# emmeans(aov_model2, specs="IV2", adjust="tukey")
# pairs(emmeans(aov_model2, specs="IV2", adjust="tukey"))
#  
# emmeans(aov_model2, specs="IV1", by="IV2", adjust="sidak")
# pairs(emmeans(aov_model2, specs="IV2", by="IV1", adjust="sidak"))
# 
# emmeans(aov_model2, specs="IV2", by="IV1", adjust="sidak")
# pairs(emmeans(aov_model2, specs="IV2", by="IV1", adjust="sidak"))

View Test Output

Effect size cutoffs from Cohen (1988): * Trivial: < .1 * Small: between .1 and .3 * Medium: between .3 and .5 * Large: > .5

# summary(reg_model)

Write Up Results

t-Test

Write-up of your results goes here. Check past labs/HWs for template.

Correlation Test

Write-up of your results goes here. Check past labs/HWs for template. Depending on how many variables you have here, I may need to help you tweak your table output.

One-Way ANOVA

Write-up of your results goes here. Check past labs/HWs for template.

Two-Way ANOVA

Write-up of your results goes here. Check past labs/HWs for template.

Multiple Linear Regression

Write-up of your results goes here. Check past labs/HWs for template.

References

Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic.