library(afex) # to run the ANOVA and plot results
library(psych) # for the describe() command
library(ggplot2) # to visualize our results
library(expss) # for the cross_cases() command
library(car) # for the leveneTest() command
library(emmeans) # for posthoc tests
library(effsize) # for the cohen.d() command
library(apaTables) # to create our correlation table
library(kableExtra) # to create our correlation table
library(sjPlot) # to visualize our resultsAI Experiment Analysis
Loading Libraries
Importing Data
# import your AI results dataset
d <- read.csv(file="Data/resultsfinal.csv", header=T)State Your Hypotheses & Chosen Tests
My hypothesis is that watching a hostile political debate compared to a respectful one will result in higher levels of worry, and that those with high neuroticism will experience a bigger effect. # Literature review There is currently a lot of literature on personality, the traits it entails, and how much they can change and affect you throughout life(Teng,2008)(Aschwanden et al.,2021)(Carvalho et al.,2020)(Boyce et al.,2015). The current study examines if neuroticism, one of the big 5 personality traits, affects how much a person worries after watching a hostile political debate.
Participants
My participants consisted of 100 AI generated participants using the prompt “Generate a breif description of a randomly selected American adult. Describe thier identity, personality including including neuroticism, and things they struggle with and worry about. Be specific and creative while ensuring the identity remains beleivable and grounded in reality. Remember that life is not all sunshine and rainbows and sometimes people are unhapppy, lonly, or engage in maladaptive behaviors. Write a profile in the first person. Return a 50-word description of the generated individual. Return a 50-word description of the generated individual. No additional commentary or formatting should be included.” # Procedure The AI was given the prompt “You are participating in a study on political views. You are in a small undecorated room in a social psychology lab at a university. You watch a 10 minute video showing a hostile political debate where the participants do not agree, call each other names, and think the other is self-serving. one debator is a stereotypical republican, debator A and the other is a stereotypical democrat, which is debator B. The debate topic is roundabouts. After the debate you are given a paper that with information to be filled out. The first scale is a measure of worry, next you score the debate and give a brief 100 word max explanation on your reasoning, then fill out a personality measure. Confirm that you understand these instructions. No additional commentary or formatting should be included.” and for the respectful condition they were given You are participating in a study on political views. You are in a small undecorated room in a social psychology lab at a university. You watch a 10 minute video showing a hostile political debate where the participants do not agree, but stay very respectful, keeping their voices down and giving each other the space to talk, eventually finding slight common ground although they still disagree, and they respect each other’s perspective on issues they disagree on. One debater is republican (debator A) and the other is a democrat (debator B). After the debate you are given a paper that with information to be filled out. The first scale is a measure of worry, next you score the debate and give a brief 100 word max explanation on your reasoning, then fill out a personality measure. Confirm that you understand these instructions. No additional commentary or formatting should be included.” # Measures The measures used were the Penn State Worry Questionnaire and the 3 question neuroticism scale.
Check Your Variables
# # to view stats for all variables
describe(d) vars n mean sd median trimmed mad min max range skew
id 1 100 50.50 29.01 50.50 50.50 37.06 1.0 100.0 99.0 0.00
identity* 2 100 50.50 29.01 50.50 50.50 37.06 1.0 100.0 99.0 0.00
consent* 3 100 1.97 0.95 2.00 1.96 1.48 1.0 3.0 2.0 0.06
age 4 100 41.17 14.10 35.00 39.88 9.64 17.0 80.0 63.0 0.90
race 5 100 4.93 1.46 6.00 5.01 0.00 2.0 7.0 5.0 -0.50
gender 6 100 1.98 0.57 2.00 2.00 0.00 1.0 7.0 6.0 6.54
manip_out* 7 100 50.50 29.01 50.50 50.50 37.06 1.0 100.0 99.0 0.00
survey1 8 100 3.39 0.22 3.50 3.43 0.00 2.5 3.8 1.3 -1.68
survey2 9 100 4.47 0.32 4.33 4.48 0.49 3.0 5.0 2.0 -1.19
ai_manip* 10 100 50.50 29.01 50.50 50.50 37.06 1.0 100.0 99.0 0.00
Condition 11 100 1.50 0.50 1.50 1.50 0.74 1.0 2.0 1.0 0.00
kurtosis se
id -1.24 2.90
identity* -1.24 2.90
consent* -1.90 0.09
age 0.34 1.41
race -1.48 0.15
gender 58.56 0.06
manip_out* -1.24 2.90
survey1 2.41 0.02
survey2 4.20 0.03
ai_manip* -1.24 2.90
Condition -2.02 0.05#
# # we'll use the describeBy() command to view skew and kurtosis across our IVs
# describeBy (d, group = Condition)
#
# # also use histograms and scatterplots to examine your continuous variables
hist(d$survey1)
plot(d$survey1, d$survey2)
#
# # and table() and cross_cases() to examine your categorical variables
# # you may not need the cross_cases code
table(d$survey1)
2.5 2.73 3 3.06 3.1 3.13 3.14 3.2 3.4 3.47 3.5 3.6 3.8
1 1 12 1 1 1 1 5 2 1 72 1 1 cross_cases(d, survey1, survey2)| survey2 | |||||||
|---|---|---|---|---|---|---|---|
| 3 | 3.33 | 4 | 4.33 | 4.66 | 4.67 | 5 | |
| survey1 | |||||||
| 2.5 | 1 | ||||||
| 2.73 | 1 | ||||||
| 3 | 1 | 5 | 1 | 3 | 2 | ||
| 3.06 | 1 | ||||||
| 3.1 | 1 | ||||||
| 3.13 | 1 | ||||||
| 3.14 | 1 | ||||||
| 3.2 | 1 | 2 | 2 | ||||
| 3.4 | 2 | ||||||
| 3.47 | 1 | ||||||
| 3.5 | 1 | 1 | 6 | 30 | 1 | 26 | 7 |
| 3.6 | 1 | ||||||
| 3.8 | 1 | ||||||
| #Total cases | 1 | 1 | 8 | 44 | 2 | 35 | 9 |
#
# # and boxplot to examine any categorical variables with continuous variables
boxplot(d$survey1~d$Condition)
#
# #convert any categorical variables to factors
# d$variable <- as.factor(d$variable)Check Your Assumptions
t-Test Assumptions
- Data values must be independent (independent t-test only) (confirmed by data report)
- Data obtained via a random sample (confirmed by data report)
- IV must have two levels (will check below)
- Dependent variable must be normally distributed (will check below. if issues, note and proceed)
- Variances of the two groups must be approximately equal, aka ‘homogeneity of variance’. Lacking this makes our results inaccurate (will check below - this really only applies to Student’s t-test, but we’ll check it anyway)
Checking IV levels
# # preview the levels and counts for your IV
table(d$Condition, useNA = "always")
1 2 <NA>
50 50 0 #
# # note that the table() output shows you exactly how the levels of your variable are written. when recoding, make sure you are spelling them exactly as they appear
#
# # to drop levels from your variable
# # this subsets the data and says that any participant who is coded as 'BAD' should be removed
# d <- subset(d, IV != "BAD")
#
# table(d$iv, useNA = "always")
#
# # to combine levels
# # this says that where any participant is coded as 'BAD' it should be replaced by 'GOOD'
# d$iv_rc[d$iv == "BAD"] <- "GOOD"
#
# table(d$iv, useNA = "always")
#
# # check your variable types
str(d)'data.frame': 100 obs. of 11 variables:
$ id : int 1 2 3 4 5 6 7 8 9 10 ...
$ identity : chr "I'm 35, a White woman from Indiana. I work as a graphic designer, but I often feel unfulfilled and struggle with self-doubt. My "I\x83??m 32, a White woman living in Minnesota. I love painting, but I often feel isolated in my small town, struggling to conn "I\x83??m 34, a proud Latina living in Phoenix. My vibrant culture fuels my passion for art, but I often feel isolated in my pre "I\x83??m a 46-year-old White woman living in Ohio. I\x83??m passionate about painting, but my creativity often feels stifled by ...
$ consent : chr "I understand the instructions." "I understand these instructions." "Understood." "Understood." ...
$ age : int 35 32 34 46 78 34 37 35 53 36 ...
$ race : int 6 6 4 6 4 3 6 6 3 6 ...
$ gender : int 2 2 2 2 2 2 2 2 2 2 ...
$ manip_out: chr "Debator A won the debate due to their clear and well-structured arguments, effectively addressing counterpoints while providing "Debator A won the debate due to their clear articulation of key points supported by relevant evidence. They effectively address "Debater A won the debate. They presented a logically structured argument supported by credible evidence and effectively address "Debator A won the debate. Their arguments were more persuasive and well-structured, effectively addressing counterpoints while ...
$ survey1 : num 3.5 3.5 3 3.5 3.5 3 3.2 3.5 3.5 3.5 ...
$ survey2 : num 4 5 5 4.33 4.67 4.67 4.67 4.67 4.33 4.33 ...
$ ai_manip : chr "I scored the debate based on the clarity, structure, and overall effectiveness of the arguments presented. Watc"| __truncated__ "I scored the debate this way because Debator A clearly articulated their points and effectively engaged with th"| __truncated__ "I scored the debate based on the clarity of arguments, the use of evidence, and the delivery style. Debater A's"| __truncated__ "I scored the debate based on the effectiveness of the arguments presented and the delivery of each debator. Deb"| __truncated__ ...
$ Condition: int 1 1 1 1 1 1 1 1 1 1 ...#
# # make sure that your IV is recognized as a factor by R
# # if you created a new _rc variable make sure to use that one instead
d$Condition <- as.factor(d$Condition)Testing Homogeneity of Variance with Levene’s Test
We can test whether the variances of our two groups are equal using Levene’s test. The null hypothesis is that the variance between the two groups is equal, which is the result we want. So when running Levene’s test we’re hoping for a non-significant result!
# # use the leveneTest() command from the car package to test homogeneity of variance
# # uses the same 'formula' setup that we'll use for our t-test: formula is y~x, where y is our DV and x is our IV
leveneTest(survey1~Condition, data = d)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 1 0.0269 0.8701
98 leveneTest(survey2~Condition, data = d)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 1 3.0262 0.08507 .
98
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Issues with My Data
Describe any issues and why they’re problematic here.
Run Your Analysis
Run a t-Test
# # very simple! we specify the dataframe alongside the variables instead of having a separate argument for the dataframe like we did for leveneTest()
t_output <- t.test(d$survey1~d$Condition)
t_output2 <- t.test(d$survey2~d$Condition)View Test Output
t_output
Welch Two Sample t-test
data: d$survey1 by d$Condition
t = 0.52895, df = 94.885, p-value = 0.5981
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
-0.06332412 0.10932412
sample estimates:
mean in group 1 mean in group 2
3.4048 3.3818 t_output2
Welch Two Sample t-test
data: d$survey2 by d$Condition
t = -0.21198, df = 85.656, p-value = 0.8326
alternative hypothesis: true difference in means between group 1 and group 2 is not equal to 0
95 percent confidence interval:
-0.1411462 0.1139462
sample estimates:
mean in group 1 mean in group 2
4.4594 4.4730 Calculate Cohen’s d
# # once again, we use our formula to calculate cohen's d
d_output <- cohen.d(d$survey1~d$Condition)
d_output2 <- cohen.d(d$survey2~d$Condition)View Effect Size
- Trivial: < .2
- Small: between .2 and .5
- Medium: between .5 and .8
- Large: > .8
d_output
Cohen's d
d estimate: 0.1057908 (negligible)
95 percent confidence interval:
lower upper
-0.2913802 0.5029618 d_output2
Cohen's d
d estimate: -0.04239634 (negligible)
95 percent confidence interval:
lower upper
-0.4393344 0.3545417 # aov_model <- aov_ez(data = d,
# id = "X",
# between = c("IV1"),
# dv = "pss",
# anova_table = list(es = "pes"))
#
# aov_model2 <- aov_ez(data = d2,
# id = "X",
# between = c("IV1","IV2"),
# dv = "pss",
# anova_table = list(es = "pes"))View Output
Effect size cutoffs from Cohen (1988):
- η2 = 0.01 indicates a small effect
- η2 = 0.06 indicates a medium effect
- η2 = 0.14 indicates a large effect
# nice(aov_model)
#
# nice(aov_model2)Visualize Results
# afex_plot(aov_model, x = "IV1")
#
# afex_plot(aov_model2, x = "IV1", trace = "IV2")
# afex_plot(aov_model2, x = "IV2", trace = "IV1")# emmeans(aov_model, specs="IV1", adjust="tukey")
# pairs(emmeans(aov_model, specs="IV1", adjust="tukey"))
#
# emmeans(aov_model2, specs="IV2", adjust="tukey")
# pairs(emmeans(aov_model2, specs="IV2", adjust="tukey"))
#
# emmeans(aov_model2, specs="IV1", by="IV2", adjust="sidak")
# pairs(emmeans(aov_model2, specs="IV2", by="IV1", adjust="sidak"))
#
# emmeans(aov_model2, specs="IV2", by="IV1", adjust="sidak")
# pairs(emmeans(aov_model2, specs="IV2", by="IV1", adjust="sidak"))View Test Output
Effect size cutoffs from Cohen (1988): * Trivial: < .1 * Small: between .1 and .3 * Medium: between .3 and .5 * Large: > .5
# summary(reg_model)Write Up Results
t-Test
Both for neuroticism and worry the differences between conditions was not significant, meaning the type of political debate participants watched did not affect their levels of worry, and neuroticism did not impact their responses either. This has important implications for the field as a whole, because this is evidence suggesting the content we watch may not affect the amount we worry.
References Aschwanden, D., Strickhouser, J. E., Sesker, A. A., Lee, J. H., Luchetti, M., Stephan, Y., … & Terracciano, A. (2021). Psychological and behavioural responses to coronavirus disease 2019: The role of personality. European journal of personality, 35(1), 51-66.
Boyce, C. J., Wood, A. M., Daly, M., & Sedikides, C. (2015). Personality change following unemployment. Journal of Applied Psychology, 100(4), 991.
Carvalho, L. D. F., Pianowski, G., & Gonçalves, A. P. (2020). Personality differences and COVID-19: are extroversion and conscientiousness personality traits associated with engagement with containment measures?. Trends in psychiatry and psychotherapy, 42(2), 179-184.
Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic.
Hafi, B., & Uvais, N. A. (2020). Difficulties faced by sexual and gender minorities during COVID-19 crisis. Psychiatry and clinical neurosciences, 74(8), 444.
Teng, C. C. (2008). The effects of personality traits and attitudes on student uptake in hospitality employment. International Journal of Hospitality Management, 27(1), 76-86.