AI Experiment Analysis

Loading Libraries

library(afex) # to run the ANOVA and plot results
library(psych) # for the describe() command
library(ggplot2) # to visualize our results
library(expss) # for the cross_cases() command
library(car) # for the leveneTest() command
library(emmeans) # for posthoc tests
library(effsize) # for the cohen.d() command
library(apaTables) # to create our correlation table
library(kableExtra) # to create our correlation table
library(sjPlot) # to visualize our results

Importing Data

# # import your AI results dataset
d <- read.csv(file="Data/FinalResultsDatasetCombined.csv", header=T)

State Your Hypotheses & Chosen Tests

H1: I predict that participants in the equal society condition will report higher life satisfaction (SWB) than those in the unequal society condition.

H2: I predict that participants who score lower on the mindful attention awareness scale (more mindful individuals) will experience higher life satisfactions (SWB).

Check Your Variables

This is just basic variable checking that is used across all HW assignments.

# # to view stats for all variables
describe(d)
           vars   n  mean    sd median trimmed   mad   min    max range  skew
id            1 100 50.50 29.01  50.50   50.50 37.06  1.00 100.00 99.00  0.00
identity*     2 100 50.50 29.01  50.50   50.50 37.06  1.00 100.00 99.00  0.00
consent*      3 100  1.78  0.42   2.00    1.85  0.00  1.00   2.00  1.00 -1.33
age           4 100 41.81 15.49  37.00   40.91 10.38 15.00  94.00 79.00  0.73
race*         5 100  5.07  2.02   5.00    5.12  2.97  1.00   8.00  7.00 -0.15
gender        6 100  1.94  0.42   2.00    2.00  0.00  1.00   5.00  4.00  2.82
manip_out*    7 100 50.50 29.01  50.50   50.50 37.06  1.00 100.00 99.00  0.00
survey1       8 100  4.88  0.68   4.97    4.93  0.64  2.93   6.13  3.20 -0.66
survey2       9 100  3.43  0.38   3.33    3.42  0.25  2.50   4.17  1.67  0.26
ai_manip*    10 100 50.50 29.01  50.50   50.50 37.06  1.00 100.00 99.00  0.00
condition    11 100  1.50  0.50   1.50    1.50  0.74  1.00   2.00  1.00  0.00
           kurtosis   se
id            -1.24 2.90
identity*     -1.24 2.90
consent*      -0.23 0.04
age            0.24 1.55
race*         -1.72 0.20
gender        26.83 0.04
manip_out*    -1.24 2.90
survey1        0.04 0.07
survey2       -0.88 0.04
ai_manip*     -1.24 2.90
condition     -2.02 0.05
# we'll use the describeBy() command to view skew and kurtosis across our IVs
describeBy(d, group = "condition")

 Descriptive statistics by group 
condition: 1
          vars  n  mean    sd median trimmed   mad   min    max range  skew
id           1 50 25.50 14.58  25.50   25.50 18.53  1.00  50.00 49.00  0.00
identity     2 50 47.42 31.27  47.50   46.83 42.25  1.00 100.00 99.00  0.07
consent      3 50  1.80  0.40   2.00    1.88  0.00  1.00   2.00  1.00 -1.46
age          4 50 39.60 16.04  35.50   38.48 14.08 15.00  80.00 65.00  0.54
race         5 50  5.22  2.03   6.50    5.30  2.22  1.00   8.00  7.00 -0.29
gender       6 50  2.00  0.49   2.00    2.00  0.00  1.00   5.00  4.00  3.96
manip_out    7 50 49.70 27.80  48.50   49.80 32.62  1.00  97.00 96.00  0.00
survey1      8 50  4.80  0.73   4.80    4.83  0.79  2.93   6.13  3.20 -0.33
survey2      9 50  3.42  0.41   3.25    3.40  0.37  2.50   4.17  1.67  0.27
ai_manip    10 50 50.74 28.55  48.50   50.70 36.32  3.00 100.00 97.00  0.03
condition   11 50  1.00  0.00   1.00    1.00  0.00  1.00   1.00  0.00   NaN
          kurtosis   se
id           -1.27 2.06
identity     -1.34 4.42
consent       0.12 0.06
age          -0.48 2.27
race         -1.61 0.29
gender       25.01 0.07
manip_out    -1.15 3.93
survey1      -0.57 0.10
survey2      -0.93 0.06
ai_manip     -1.32 4.04
condition      NaN 0.00
------------------------------------------------------------ 
condition: 2
          vars  n  mean    sd median trimmed   mad   min    max range  skew
id           1 50 75.50 14.58  75.50   75.50 18.53 51.00 100.00 49.00  0.00
identity     2 50 53.58 26.52  54.50   53.52 36.32 11.00  98.00 87.00  0.02
consent      3 50  1.76  0.43   2.00    1.82  0.00  1.00   2.00  1.00 -1.18
age          4 50 44.02 14.76  38.00   42.38  8.90 19.00  94.00 75.00  1.08
race         5 50  4.92  2.02   5.00    4.95  2.97  2.00   8.00  6.00  0.00
gender       6 50  1.88  0.33   2.00    1.98  0.00  1.00   2.00  1.00 -2.27
manip_out    7 50 51.30 30.44  53.00   51.23 39.29  3.00 100.00 97.00 -0.01
survey1      8 50  4.96  0.61   5.03    5.03  0.54  3.00   5.87  2.87 -1.05
survey2      9 50  3.45  0.35   3.42    3.44  0.37  2.83   4.17  1.33  0.29
ai_manip    10 50 50.26 29.75  52.00   50.23 37.06  1.00  99.00 98.00 -0.03
condition   11 50  2.00  0.00   2.00    2.00  0.00  2.00   2.00  0.00   NaN
          kurtosis   se
id           -1.27 2.06
identity     -1.30 3.75
consent      -0.62 0.06
age           0.84 2.09
race         -1.83 0.29
gender        3.21 0.05
manip_out    -1.38 4.30
survey1       1.09 0.09
survey2      -1.02 0.05
ai_manip     -1.23 4.21
condition      NaN 0.00
# also use histograms and scatterplots to examine your continuous variables
hist(d$survey1)

hist(d$survey2)

plot(d$survey1, d$survey2)

# and table() and cross_cases() to examine your categorical variables
# you may not need the cross_cases code
table(d$condition)

 1  2 
50 50 
# cross_cases(d, survey1, survey2)

# and boxplot to examine any categorical variables with continuous variables
boxplot(d$survey2~d$condition)

boxplot(d$survey1~d$condition)

# convert any categorical variables to factors
# d$variable <- as.factor(d$variable)

Check Your Assumptions

t-Test Assumptions

  • Data values must be independent (independent t-test only) (confirmed by data report)
  • Data obtained via a random sample (confirmed by data report)
  • IV must have two levels (will check below)
  • Dependent variable must be normally distributed (will check below. if issues, note and proceed)
  • Variances of the two groups must be approximately equal, aka ‘homogeneity of variance’. Lacking this makes our results inaccurate (will check below - this really only applies to Student’s t-test, but we’ll check it anyway)

Checking IV levels

# preview the levels and counts for your IV
table(d$condition, useNA = "always")

   1    2 <NA> 
  50   50    0 
# note that the table() output shows you exactly how the levels of your variable are written. when recoding, make sure you are spelling them exactly as they appear

# to drop levels from your variable
# this subsets the data and says that any participant who is coded as 'BAD' should be removed
# d <- subset(d, condition != "BAD")

table(d$condition, useNA = "always")

   1    2 <NA> 
  50   50    0 
# to combine levels
# this says that where any participant is coded as 'BAD' it should be replaced by 'GOOD'
# d$iv_rc[d$condition == "BAD"] <- "GOOD"

table(d$condition, useNA = "always")

   1    2 <NA> 
  50   50    0 
# check your variable types
str(d)
'data.frame':   100 obs. of  11 variables:
 $ id       : int  1 2 3 4 5 6 7 8 9 10 ...
 $ identity : chr  "I'm a 37-year-old Black woman living in Atlanta. I work as a nurse, making about $65,000 a year. While I love h"| __truncated__ "I'm 30, a White woman from Ohio. I work as a marketing manager, earning around $75,000 a year. I often feel iso"| __truncated__ "I'm 53, a Black woman working as a nurse in a bustling city. My income is decent, but rising costs make it hard"| __truncated__ "I'm a 39-year-old Black woman living in Atlanta. I work as a school counselor, earning about $55,000 a year. I "| __truncated__ ...
 $ consent  : chr  "I understand these instructions." "I understand these instructions." "I understand these instructions." "I understand the instructions." ...
 $ age      : int  37 30 53 39 26 43 43 57 21 36 ...
 $ race     : chr  "3" "6" "3" "3" ...
 $ gender   : int  2 2 2 2 2 2 2 2 2 2 ...
 $ manip_out: chr  "Day 1: Daily Allowance - $10  \nSince I want to start the week on a positive note, I decide to focus on self-ca"| __truncated__ "Daily Allowance: $20\n\nDay 1: I decided to start the week with a focus on self-care. I spent $5 on a yoga clas"| __truncated__ "Day 1: Daily Allowance - $10  \nToday, I decided to spend my allowance wisely. I had a long shift at the hospit"| __truncated__ "Day 1:  \nDaily Allowance: $10  \nToday, I focused on self-care after a long week at work. I decided to go for "| __truncated__ ...
 $ survey1  : num  4.93 3.33 4.47 5.13 4.6 ...
 $ survey2  : num  4.17 3.17 4.17 3.5 2.83 ...
 $ ai_manip : chr  "During the simulation, I navigated my daily life by focusing on self-care, socializing, and balancing work with"| __truncated__ "1) During the simulation, I focused on self-care and socializing. I spent time on yoga and meditation to addres"| __truncated__ "During the simulation, I navigated my daily life by making intentional choices that prioritized my well-being a"| __truncated__ "During the simulation, I navigated my daily life by balancing work, self-care, and socializing. I often focused"| __truncated__ ...
 $ condition: int  1 1 1 1 1 1 1 1 1 1 ...
# make sure that your IV is recognized as a factor by R
# if you created a new _rc variable make sure to use that one instead
d$condition <- as.factor(d$condition)

Testing Homogeneity of Variance with Levene’s Test

We can test whether the variances of our two groups are equal using Levene’s test. The null hypothesis is that the variance between the two groups is equal, which is the result we want. So when running Levene’s test we’re hoping for a non-significant result!

# use the leveneTest() command from the car package to test homogeneity of variance
# uses the same 'formula' setup that we'll use for our t-test: formula is y~x, where y is our DV and x is our IV
leveneTest(survey2~condition, data = d)
Levene's Test for Homogeneity of Variance (center = median)
      Df F value Pr(>F)
group  1  0.9983 0.3202
      98

Pearson’s Correlation Coefficient Assumptions

  • Should have two measurements for each participant for each variable (confirmed by earlier procedures – we dropped any participants with missing data)
  • Variables should be continuous and normally distributed, or assessments of the relationship may be inaccurate (will do below)
  • Outliers should be identified and removed, or results will be inaccurate (will do below)
  • Relationship between the variables should be linear, or they will not be detected (will do below)

Run a Multiple Linear Regression

To check the assumptions for Pearson’s correlation coefficient, we run our regression and then check our diagnostic plots.

# use the lm() command to run the regression
# dependent/outcome variable on the left, independent/predictor variables on the right
reg_model <- lm(survey2 ~ survey1 + condition, data = d)

Check linearity with Residuals vs Fitted plot

For some examples of good Residuals vs Fitted plot and ones that show serious errors, check out this page.

For your homework, you’ll simply need to generate this plot and talk about how your plot compares to the good and problematic plots linked to above. Is it closer to the ‘good’ plots or one of the ‘bad’ plots? This is going to be a judgement call, and that’s okay! In practice, you’ll always be making these judgement calls as part of a team, so this assignment is just about getting experience with it, not making the perfect call.

plot(reg_model, 1)

Check for outliers using Cook’s distance and a Residuals vs Leverage plot

For your homework, you’ll simply need to generate these plots, assess Cook’s distance in your dataset, and then identify any potential cases that are prominent outliers.

# Cook's distance
plot(reg_model, 4)

# Residuals vs Leverage
plot(reg_model, 5)

Multiple Linear Regression Assumptions

  • Observations should be independent (confirmed by data report)
  • Number of cases should be adequate (N ≥ 80 + 8m, where m is the number of IVs). If you don’t have enough, it won’t run. (will check this below)
  • Independent variables should not be too correlated (aka multicollinearity). (will check this below)
  • Relationship between the variables should be linear. (will check this below)
  • Outliers should be identified and removed. (will check this below)
  • Residuals should be normally distributed and have constant variance. (will check this below)

Check Number of Cases

For your homework, if you don’t have the required number of cases you’ll need to drop one of your independent variables. Reach out to me if this happens and we can figure out the best way to proceed!

needed <- 80 + 8*3
nrow(d) >= needed
[1] FALSE

Run a Multiple Linear Regression

To check the assumptions for an MLR, we run our regression and then check some output and diagnostic plots BEFORE looking at our results.

# use the lm() command to run the regression
# dependent/outcome variable on the left, independent/predictor variables on the right
reg_model <- lm(survey1 ~ survey2 + condition, data = d)

Check multicollinearity

  • Higher values indicate more multicollinearity. This usually requires dropping a variable. For your homework, you will need to discuss multicollinearity and any high values, but you don’t have to drop any variables.
  • Cutoff is usually 5
vif(reg_model)
  survey2 condition 
 1.001957  1.001957

Check linearity with Residuals vs Fitted plot

For some examples of good Residuals vs Fitted plot and ones that show serious errors, check out this page.

For your homework, you’ll simply need to generate this plot and talk about how your plot compares to the good and problematic plots linked to above. Is it closer to the ‘good’ plots or one of the ‘bad’ plots? This is going to be a judgement call, and that’s okay! In practice, you’ll always be making these judgement calls as part of a team, so this assignment is just about getting experience with it, not making the perfect call.

plot(reg_model, 1)

Check for outliers using Cook’s distance and a Residuals vs Leverage plot

For your homework, you’ll simply need to generate these plots, assess Cook’s distance in your dataset, and then identify any potential cases that are prominent outliers. Since we have some cutoffs, that makes this process is a bit less subjective than some of the other assessments we’ve done here, which is a nice change!

# Cook's distance
plot(reg_model, 4)

# Residuals vs Leverage
plot(reg_model, 5)

Check homogeneity of variance in a Scale-Location plot

You can check out this page for some other examples of this type of plot. (Notice that the Scale-Location plot is the third in the grids.)

For your homework, you’ll simply need to generate this plot and talk about how your plot compares to the ones pictured. Is it closer to the ‘good’ plots or one of the ‘bad’ plots? Again, this is a judgement call! It’s okay if feel uncertain, and you won’t be penalized for that.

plot(reg_model, 3)

Check normality of residuals with a Q-Q plot

This page shows how different types of non-normality appear on a Q-Q plot. It’s normal for Q-Q plots show a bit of deviation at the ends.

This page also shows some examples that help us put our Q-Q plot into context. Although it isn’t perfect, we don’t have any serious issues and are okay to proceed.

For your homework, you’ll simply need to generate this plot and talk about how your plot compares to the ones pictured. Does it seem like any skew or kurtosis is indicated by your plot? Is it closer to the ‘good’/‘bad’ plots from the second link?

plot(reg_model, 2)

Issues with My Data

While this study provides useful insights, there are a few important constraints to consider. First, there is a possibility that the condition where I assign the participant in shows a weak simulation environment. There could also be potential race and gender biases from within the experimental data as many of our racial group to be assigned to the black and white category, our gender is primarily cis-women, and the participant pool derived of maily middle aged people as well. Additionally, averaged MAAS and SWB data generated from AI could potentially be miscalculated. Our kurtosis for gender is high for both condition 1 and condition 2. Other than that, our residual vs leverage, residual vs fitted, cook’s distance, box-plot, histogram, and Q-Q residual graphs/tables showed no concerning patterns or extreme outliers. Multicollinearity also appear to be below 5.

Run Your Analysis

Run a t-Test

# very simple! we specify the dataframe alongside the variables instead of having a separate argument for the dataframe like we did for leveneTest()
t_output <- t.test(d$survey2~d$condition)

View Test Output

t_output

    Welch Two Sample t-test

data:  d$survey2 by d$condition
t = -0.43793, df = 95.54, p-value = 0.6624
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
 -0.1844318  0.1177651
sample estimates:
mean in group 1 mean in group 2 
       3.416667        3.450000

Calculate Cohen’s d

# once again, we use our formula to calculate cohen's d
d_output <- cohen.d(d$survey2~d$condition)

View Effect Size

  • Trivial: < .2
  • Small: between .2 and .5
  • Medium: between .5 and .8
  • Large: > .8
d_output

Cohen's d

d estimate: -0.08758557 (negligible)
95 percent confidence interval:
     lower      upper 
-0.4846693  0.3094982

Run a Correlation Test

Create a Correlation Matrix

d2 <- subset(d, select= c(survey1, survey2))
corr_output_m <- corr.test(d2)

View Test Output

  • Strong effect: Between |0.50| and |1|
  • Moderate effect: Between |0.30| and |0.49|
  • Weak effect: Between |0.10| and |0.29|
  • Trivial effect: Less than |0.09|
corr_output_m
Call:corr.test(x = d2)
Correlation matrix 
        survey1 survey2
survey1    1.00    0.05
survey2    0.05    1.00
Sample Size 
[1] 100
Probability values (Entries above the diagonal are adjusted for multiple tests.) 
        survey1 survey2
survey1    0.00    0.63
survey2    0.63    0.00

 To see confidence intervals of the correlations, print with the short=FALSE option

Run a Multiple Linear Regression

You already ran this!

View Test Output

Effect size cutoffs from Cohen (1988): * Trivial: < .1 * Small: between .1 and .3 * Medium: between .3 and .5 * Large: > .5

summary(reg_model)

Call:
lm(formula = survey1 ~ survey2 + condition, data = d)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.96524 -0.38417  0.04679  0.51770  1.28643 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  4.53442    0.62095   7.302  7.9e-11 ***
survey2      0.07812    0.17957   0.435    0.664    
condition2   0.15740    0.13544   1.162    0.248    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.6765 on 97 degrees of freedom
Multiple R-squared:  0.0161,    Adjusted R-squared:  -0.004184 
F-statistic: 0.7938 on 2 and 97 DF,  p-value: 0.4551

Write Up Results

t-Test

A Welch two-sample t-test was conducted to compare life satisfaction scores between participants in the equal society (n = 50, M = 3.42, SD = 0.41) and unequal society conditions (n = 50, M = 3.45, SD = 0.35). The analysis revealed no statistically significant difference between groups, t(95.54) = -0.44, p = .662, 95% CI [-0.18, 0.12]. The effect size was negligible and trivial (Cohen’s d = -0.09), indicating that the economic inequality manipulation had minimal observable impact on participants’ reported life satisfaction.

Correlation Test

A Pearson correlation revealed no significant relationship between mindfulness (M = 3.43, SD = 0.38) and life satisfaction (M = 3.43, SD = 0.68), r(98) = .05, p = .63, N = 100. The effect was negligible, with the 95% confidence interval spanning from -0.15 to 0.25, indicating trivial to small, that even if a true correlation exists, it is likely minimal at best.

Multiple Linear Regression

A multiple linear regression tested whether life satisfaction (SWB) and experimental condition (equal vs. unequal society) predicted mindfulness scores (MAAS). The overall model was not statistically significant, F(2, 97) = 0.79, p = .455, (R² = .02). The non-significant coefficients for SWB (b = 0.08, SE = 0.18, p = .664, 95% CI [-0.28, 0.44]) and condition (b = 0.16, SE = 0.14, p = .248, 95% CI [-0.11, 0.42]) suggest that neither variable meaningfully influenced mindfulness during the simulation. The significant intercept (b = 4.53, p < .001) indicates average mindfulness levels fell near the scale midpoint regardless of other factors. These null findings align with prior non-significant results in this data-set, collectively implying the simulated societal structures and SWB had limited impact on participants’ moment-to-moment awareness.

References

Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic.