AI Experiment Analysis

Loading Libraries

library(afex) # to run the ANOVA and plot results
library(psych) # for the describe() command
library(ggplot2) # to visualize our results
library(expss) # for the cross_cases() command
library(car) # for the leveneTest() command
library(emmeans) # for posthoc tests
library(effsize) # for the cohen.d() command
library(apaTables) # to create our correlation table
library(kableExtra) # to create our correlation table
library(sjPlot) # to visualize our results

Importing Data

# import your AI results dataset
d <- read.csv(file="Data/final_results(in).csv", header=T)

State Your Hypotheses & Chosen Tests

H1: Participants exposed to affirming messages will report higher self-esteem than participants exposed to non-affirming messages.

H2: Perceived social support will predict self-esteem, and the relationship will be positive (higher perceived social support will be associated with higher self-esteem).

Test 1: T-test

Test 2: Correlation test

Check Your Variables

This is just basic variable checking that is used across all HW assignments.

# to view stats for all variables
describe(d)

           vars   n  mean    sd median trimmed   mad  min   max range  skew
id            1 100 50.50 29.01   50.5   50.50 37.06  1.0 100.0    99  0.00
identity*     2 100 50.50 29.01   50.5   50.50 37.06  1.0 100.0    99  0.00
consent*      3 100 44.41 25.08   42.5   44.51 33.36  1.0  87.0    86 -0.01
age           4 100 39.90 13.16   34.0   38.21  5.93 18.0  85.0    67  1.31
race          5 100  4.73  1.54    6.0    4.82  1.48  1.0   7.0     6 -0.37
gender        6 100  1.96  0.40    2.0    2.00  0.00  1.0   5.0     4  3.42
manip_out*    7 100 50.50 29.01   50.5   50.50 37.06  1.0 100.0    99  0.00
survey1       8 100  2.48  0.20    2.5    2.50  0.15  1.9   2.9     1 -0.98
survey2       9 100  1.80  0.39    2.0    1.85  0.00  0.0   3.0     3 -1.29
ai_manip*    10 100 50.50 29.01   50.5   50.50 37.06  1.0 100.0    99  0.00
condition    11 100  1.50  0.50    1.5    1.50  0.74  1.0   2.0     1  0.00
           kurtosis   se
id            -1.24 2.90
identity*     -1.24 2.90
consent*      -1.26 2.51
age            1.60 1.32
race          -1.25 0.15
gender        32.68 0.04
manip_out*    -1.24 2.90
survey1        1.39 0.02
survey2        4.06 0.04
ai_manip*     -1.24 2.90
condition     -2.02 0.05

# we'll use the describeBy() command to view skew and kurtosis across our IVs
describeBy(d, group = d$condition)


 Descriptive statistics by group 
group: 1
          vars  n  mean    sd median trimmed   mad  min   max range  skew
id           1 50 25.50 14.58  25.50   25.50 18.53  1.0  50.0  49.0  0.00
identity     2 50 51.78 28.46  49.50   52.17 34.84  3.0 100.0  97.0 -0.09
consent      3 50 45.88 24.64  44.00   46.17 31.88  2.0  87.0  85.0 -0.05
age          4 50 39.24 11.98  34.00   37.92  5.93 18.0  85.0  67.0  1.46
race         5 50  4.74  1.48   6.00    4.88  0.74  1.0   7.0   6.0 -0.52
gender       6 50  1.96  0.20   2.00    2.00  0.00  1.0   2.0   1.0 -4.55
manip_out    7 50 45.50 23.57  43.50   45.60 22.24  1.0  88.0  87.0  0.02
survey1      8 50  2.52  0.18   2.50    2.54  0.15  1.9   2.8   0.9 -1.19
survey2      9 50  1.75  0.46   1.83    1.79  0.25  0.0   3.0   3.0 -1.08
ai_manip    10 50 48.66 29.48  44.50   47.80 33.36  2.0 100.0  98.0  0.27
condition   11 50  1.00  0.00   1.00    1.00  0.00  1.0   1.0   0.0   NaN
          kurtosis   se
id           -1.27 2.06
identity     -1.18 4.03
consent      -1.08 3.49
age           3.09 1.69
race         -1.00 0.21
gender       19.13 0.03
manip_out    -0.66 3.33
survey1       2.25 0.03
survey2       3.21 0.07
ai_manip     -1.20 4.17
condition      NaN 0.00
------------------------------------------------------------ 
group: 2
          vars  n  mean    sd median trimmed   mad  min   max range  skew
id           1 50 75.50 14.58   75.5   75.50 18.53 51.0 100.0  49.0  0.00
identity     2 50 49.22 29.78   51.5   48.80 42.25  1.0  99.0  98.0  0.09
consent      3 50 42.94 25.68   42.0   42.90 34.10  1.0  84.0  83.0  0.04
age          4 50 40.56 14.33   34.5   38.88  3.71 19.0  79.0  60.0  1.13
race         5 50  4.72  1.60    6.0    4.78  1.48  2.0   7.0   5.0 -0.25
gender       6 50  1.96  0.53    2.0    2.00  0.00  1.0   5.0   4.0  3.13
manip_out    7 50 55.50 33.07   68.5   56.20 37.06  3.0 100.0  97.0 -0.23
survey1      8 50  2.43  0.21    2.5    2.45  0.15  1.9   2.9   1.0 -0.78
survey2      9 50  1.85  0.30    2.0    1.89  0.00  1.0   2.5   1.5 -1.12
ai_manip    10 50 52.34 28.72   54.5   53.30 34.84  1.0  96.0  95.0 -0.29
condition   11 50  2.00  0.00    2.0    2.00  0.00  2.0   2.0   0.0   NaN
          kurtosis   se
id           -1.27 2.06
identity     -1.33 4.21
consent      -1.47 3.63
age           0.49 2.03
race         -1.51 0.23
gender       19.22 0.08
manip_out    -1.58 4.68
survey1       0.86 0.03
survey2       0.90 0.04
ai_manip     -1.22 4.06
condition      NaN 0.00

# also use histograms and scatterplots to examine your continuous variables
hist(d$survey1)

plot(d$survey1, d$survey2)

# and table() and cross_cases() to examine your categorical variables
# you may not need the cross_cases code
table(d$condition)


 1  2 
50 50

cross_cases(d, d$condition, d$survey2)

	d$survey2
	0	1	1.166666667	1.333333333	1.4	1.5	1.6	1.666666667	1.75	1.8	1.833333333	2	2.166666667	2.2	2.4	2.5	3
d$condition
1	1	6			2	2	2	4	4	1	4	21		1	1		1
2		2	1	2		5		5			1	31	2			1
#Total cases	1	8	1	2	2	7	2	9	4	1	5	52	2	1	1	1	1

# and boxplot to examine any categorical variables with continuous variables
boxplot(d$survey1~d$condition)

#convert any categorical variables to factors
# d$survey2 <- as.factor(d$survey2)

Check Your Assumptions

t-Test Assumptions

Data values must be independent (independent t-test only) (confirmed by data report)
Data obtained via a random sample (confirmed by data report)
IV must have two levels (will check below)
Dependent variable must be normally distributed (will check below. if issues, note and proceed)
Variances of the two groups must be approximately equal, aka ‘homogeneity of variance’. Lacking this makes our results inaccurate (will check below - this really only applies to Student’s t-test, but we’ll check it anyway)

Checking IV levels

# preview the levels and counts for your IV
table(d$condition, useNA = "always")


   1    2 <NA> 
  50   50    0

# check your variable types
str(d)

'data.frame':   100 obs. of  11 variables:
 $ id       : int  1 2 3 4 5 6 7 8 9 10 ...
 $ identity : chr  "I’m a 32-year-old Asian woman living in Seattle. I’m passionate about tech, but I often feel isolated in a pred"| __truncated__ "I'm a 38-year-old multiracial woman, navigating the complexities of my identity daily. I often feel caught betw"| __truncated__ "I’m 32, Asian American, working as a software engineer in California. I wrestle with feeling disconnected from "| __truncated__ "I’m 51, a Black woman from Atlanta. I’m passionate about social justice but often feel overwhelmed by the weigh"| __truncated__ ...
 $ consent  : chr  "I understand the instructions. I'm ready to view the message and complete the reflection and survey as required." "I understand the instructions. I will view the brief message and then respond to the questions, complete the re"| __truncated__ "I understand the instructions. I'm ready to view the brief message and respond to the questions and reflection "| __truncated__ "I understand the instructions. Please proceed with the message for the study." ...
 $ age      : int  32 38 32 51 45 46 49 45 34 32 ...
 $ race     : int  2 7 2 3 4 6 6 4 6 4 ...
 $ gender   : int  2 2 1 2 2 2 2 2 2 2 ...
 $ manip_out: chr  "**Positive Motivational Message:**\n\n\"Every small step you take today is a step towards your dreams. Embrace "| __truncated__ "**Positive Motivational Message:**  \n\"Embrace your uniqueness—it's your greatest strength! Every story, every"| __truncated__ "**Message:** \"Every challenge we face is an opportunity for growth. Embrace your journey, celebrate small vict"| __truncated__ "\"Every small step you take towards change matters. Your voice is powerful, and together, we can create a world"| __truncated__ ...
 $ survey1  : num  2.5 2.5 2.5 2.5 2.5 2.6 2.3 2.5 2.5 2.5 ...
 $ survey2  : num  2 2.2 2 2 1.8 ...
 $ ai_manip : chr  "I answered based on my feelings of isolation and the need for affirmation. The motivational message resonated w"| __truncated__ "I answered based on my experiences of identity and belonging, feeling validated by affirming messages that cele"| __truncated__ "I answered the questions based on my personal experiences and feelings of disconnection, pressure, and longing "| __truncated__ "I answered the questions based on my experiences and feelings toward social justice. The empowering message res"| __truncated__ ...
 $ condition: int  1 1 1 1 1 1 1 1 1 1 ...

# make sure that your IV is recognized as a factor by R
# if you created a new _rc variable make sure to use that one instead
d$condition <- as.factor(d$condition)

Testing Homogeneity of Variance with Levene’s Test

We can test whether the variances of our two groups are equal using Levene’s test. The null hypothesis is that the variance between the two groups is equal, which is the result we want. So when running Levene’s test we’re hoping for a non-significant result!

# use the leveneTest() command from the car package to test homogeneity of variance
# uses the same 'formula' setup that we'll use for our t-test: formula is y~x, where y is our DV and x is our IV
leveneTest(d$survey1~d$condition, data = d)

Levene's Test for Homogeneity of Variance (center = median)
      Df F value Pr(>F)
group  1  0.4224 0.5173
      98

Pearson’s Correlation Coefficient Assumptions

Should have two measurements for each participant for each variable (confirmed by earlier procedures – we dropped any participants with missing data)
Variables should be continuous and normally distributed, or assessments of the relationship may be inaccurate (will do below)
Outliers should be identified and removed, or results will be inaccurate (will do below)
Relationship between the variables should be linear, or they will not be detected (will do below)

Run a Multiple Linear Regression

To check the assumptions for Pearson’s correlation coefficient, we run our regression and then check our diagnostic plots.

# use the lm() command to run the regression
# dependent/outcome variable on the left, independent/predictor variables on the right
reg_model <- lm(d$survey1 ~ d$survey2, data = d)

Check linearity with Residuals vs Fitted plot

For some examples of good Residuals vs Fitted plot and ones that show serious errors, check out this page.

For your homework, you’ll simply need to generate this plot and talk about how your plot compares to the good and problematic plots linked to above. Is it closer to the ‘good’ plots or one of the ‘bad’ plots? This is going to be a judgement call, and that’s okay! In practice, you’ll always be making these judgement calls as part of a team, so this assignment is just about getting experience with it, not making the perfect call.

plot(reg_model, 1)

Check for outliers using Cook’s distance and a Residuals vs Leverage plot

For your homework, you’ll simply need to generate these plots, assess Cook’s distance in your dataset, and then identify any potential cases that are prominent outliers.

# Cook's distance
plot(reg_model, 4)

# Residuals vs Leverage
plot(reg_model, 5)

Issues with My Data

There were a few minor issues with the data that should be noted. The assumptions of independence, random sampling, equal group sizes, and homogeneity of variance were met. However, our dependent variable was outside the normal range with a skew of −0.98 and kurtosis of 1.39. The skew of condition and survey 2 were even more outside of that range, with a skew of −1.29 and kurtosis of 4.06, along with a histogram that suggested uneven distribution.

Run Your Analysis

Run a t-Test

# very simple! we specify the dataframe alongside the variables instead of having a separate argument for the dataframe like we did for leveneTest()
t_output <- t.test(d$survey1~d$condition)

View Test Output

t_output


    Welch Two Sample t-test

data:  d$survey1 by d$condition
t = 2.2437, df = 96.108, p-value = 0.02715
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
 0.01014807 0.16585193
sample estimates:
mean in group 1 mean in group 2 
          2.520           2.432

Calculate Cohen’s d

# once again, we use our formula to calculate cohen's d
d_output <- cohen.d(d$survey1~d$condition)

View Effect Size

Trivial: < .2
Small: between .2 and .5
Medium: between .5 and .8
Large: > .8

d_output


Cohen's d

d estimate: 0.4487394 (small)
95 percent confidence interval:
     lower      upper 
0.04688183 0.85059689

Run a Correlation Test

Create a Correlation Matrix

d$survey2_numeric <- as.numeric(as.character(d$survey2))
d2 <- subset(d, select = c(survey1, survey2_numeric))
corr_output_m <- corr.test(d2)

View Test Output

Strong effect: Between |0.50| and |1|
Moderate effect: Between |0.30| and |0.49|
Weak effect: Between |0.10| and |0.29|
Trivial effect: Less than |0.09|

corr_output_m

Call:corr.test(x = d2)
Correlation matrix 
                survey1 survey2_numeric
survey1            1.00            0.02
survey2_numeric    0.02            1.00
Sample Size 
[1] 100
Probability values (Entries above the diagonal are adjusted for multiple tests.) 
                survey1 survey2_numeric
survey1            0.00            0.86
survey2_numeric    0.86            0.00

 To see confidence intervals of the correlations, print with the short=FALSE option

Write Up Results

t-Test

We tested our hypothesis that participants exposed to affirming messages would report significantly higher self-esteem than those exposed to non-affirming messages using an independent samples t-test.The data met all the assumptions for a t-test, and we found a significant difference in self-esteem scores between conditions, t(96.11) = 2.24, p = .027, d = 0.45, 95% CI [0.01, 0.17] (refer to Figure 1).

Our effect size was small according to Cohen (1988).

Correlation Test

We hypothesized that higher perceived social support would be positively associated with higher self-esteem.

Contrary to expectations, self-esteem and perceived social support were not significantly correlated, r(98) = .02, p = .86. The effect size was trivial (Cohen, 1988) which suggests no meaningful association in the population. These findings suggest that, in this sample, self-esteem and perceived social support are unrelated. See Table 1 for results.

Variable	M	SD	1
Self-Esteem Score	2.48	0.20

Percieved Social Support	1.80	0.39	.02
			[-.18, .21]

Note:
M and SD are used to represent mean and standard deviation, respectively. Values in square brackets indicate the 95% confidence interval. The confidence interval is a plausible range of population correlations that could have caused the sample correlation.
^* indicates p < .05
^** indicates p < .01.

References

Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic.