here() starts at /Users/sarawu/Git_Projects/idea_generation
Attaching package: 'renv'
The following objects are masked from 'package:stats':
embed, update
The following objects are masked from 'package:utils':
history, upgrade
The following objects are masked from 'package:base':
autoload, load, remove, use
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.4 ✔ readr 2.1.5
✔ forcats 1.0.0 ✔ stringr 1.5.1
✔ ggplot2 3.5.1 ✔ tibble 3.2.1
✔ lubridate 1.9.3 ✔ tidyr 1.3.1
✔ purrr 1.0.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
✖ purrr::modify() masks renv::modify()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
[0;34mThis is text (version 1.7.0).
[0m[0;32mNewer versions may have improved functions and updated defaults to reflect current understandings of the state-of-the-art.
[0m[0;35mMacOS detected: Setting OpenMP environment variables to avoid potential crash due to libomp conflicts.
[0m[0;35mWhen using the L-BAM library, be aware that models may be downloaded from external sources. Using models may carry security risks, including the possibility of malicious code in RDS files. Always review and trust the source of any model you load.
[0m[0;35mThe text package is provided 'as is' without any warranty of any kind.
[0m[0;32m
For more information about the package see www.r-text.org and www.r-topics.org.[0m
Attaching package: 'scales'
The following object is masked from 'package:purrr':
discard
The following object is masked from 'package:readr':
col_factor
Loading required package: Matrix
Attaching package: 'Matrix'
The following objects are masked from 'package:tidyr':
expand, pack, unpack
Attaching package: 'lmerTest'
The following object is masked from 'package:lme4':
lmer
The following object is masked from 'package:stats':
step
Attaching package: 'simr'
The following object is masked from 'package:lme4':
getData
The following object is masked from 'package:stringr':
fixed
Attaching package: 'kableExtra'
The following object is masked from 'package:dplyr':
group_rows
Loading required package: tinylabels
Welcome to emmeans.
Caution: You lose important information if you filter this package's results.
See '? untidy'
This is lavaan 0.6-20
lavaan is FREE software! Please report any bugs.
Attaching package: 'sjstats'
The following objects are masked from 'package:effectsize':
cohens_f, cramers_v, phi
The following object is masked from 'package:papaja':
se
The following object is masked from 'package:broom':
bootstrap
Loading required package: MASS
Attaching package: 'MASS'
The following object is masked from 'package:gtsummary':
select
The following object is masked from 'package:patchwork':
area
The following object is masked from 'package:dplyr':
select
Loading required package: msm
Loading required package: polycor
wCorr v1.9.8
Attaching package: 'rstatix'
The following object is masked from 'package:MASS':
select
The following object is masked from 'package:sjstats':
t_test
The following objects are masked from 'package:effectsize':
cohens_d, eta_squared
The following object is masked from 'package:stats':
filter
Loading required package: mvtnorm
Attaching package: 'mvtnorm'
The following object is masked from 'package:effectsize':
standardize
Loading required package: sandwich
mediation: Causal Mediation Analysis
Version: 4.5.1
Loading required package: boot
Attaching package: 'boot'
The following object is masked from 'package:msm':
cav
Attaching package: 'QuantPsyc'
The following object is masked from 'package:Matrix':
norm
The following object is masked from 'package:base':
norm
# How unique did participants think their ideas were?perceived_uniqueness <- iriss_self_perceptions %>%drop_na(unique) %>%group_by(condition) %>%count(unique) %>%mutate(prop=n/sum(n)) %>%ggplot(aes(x=unique, y=prop, fill=condition)) +geom_col(position ="dodge") +scale_x_discrete(labels =c("1"="No one","2"="Some people","3"="Most people","4"="Everyone" )) +labs(x="How many people came up with this idea?",y="Proportion of responses") +theme_apa()ggsave("figures/uniqueness.png", plot=perceived_uniqueness)
Saving 7 x 5 in image
# How does mean originality compare across conditions?iriss_self_perceptions %>%drop_na(originality) %>%group_by(submitter_id, condition) %>%summarize(originality =mean(originality)) %>%ggplot(aes(x=condition, y=originality)) +geom_boxplot() +stat_summary(fun="mean")
`summarise()` has grouped output by 'submitter_id'. You can override using the
`.groups` argument.
Warning: Removed 3 rows containing missing values or values outside the scale range
(`geom_segment()`).
# Make sure the reference group is the Control group!iriss_self_originality_mod <-lmer( originality ~ condition + (1|object) + (1|submitter_id), data= iriss_self_perceptions)summary(iriss_self_originality_mod)
Linear mixed model fit by REML. t-tests use Satterthwaite's method [
lmerModLmerTest]
Formula: originality ~ condition + (1 | object) + (1 | submitter_id)
Data: iriss_self_perceptions
REML criterion at convergence: 1407.9
Scaled residuals:
Min 1Q Median 3Q Max
-1.97423 -0.66248 0.04491 0.68796 2.21822
Random effects:
Groups Name Variance Std.Dev.
submitter_id (Intercept) 0.37877 0.6154
object (Intercept) 0.01557 0.1248
Residual 1.11078 1.0539
Number of obs: 441, groups: submitter_id, 149; object, 3
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 2.74988 0.14353 13.54053 19.158 3.36e-11 ***
conditionHigh-Agency 0.11784 0.17477 145.99395 0.674 0.501
conditionLow-Agency 0.06594 0.17476 145.99142 0.377 0.706
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr) cndH-A
cndtnHgh-Ag -0.614
cndtnLw-Agn -0.614 0.505
iriss_manual_ratings <-read_csv("data/iriss_manual_ratings.csv") %>%filter(DistributionChannel=="anonymous") %>%filter(attn_check =="4"& attn_check_4_TEXT =="8") %>%drop_na(frisbee.0.condition) %>%# Drop rows with missing data (experimenter error) mutate(age =as.numeric(age))
Rows: 160 Columns: 112
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (112): StartDate, EndDate, Status, Progress, Duration (in seconds), Fini...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Warning: There was 1 warning in `mutate()`.
ℹ In argument: `condition = fct_relevel(...)`.
Caused by warning:
! 2 unknown levels in `f`: low-agency and high-agency
`summarise()` has grouped output by 'originality'. You can override using the
`.groups` argument.
Warning: The `legend.text.align` argument of `theme()` is deprecated as of ggplot2
3.5.0.
ℹ Please use theme(legend.text = element_text(hjust)) instead.
Linear mixed model fit by REML. t-tests use Satterthwaite's method [
lmerModLmerTest]
Formula: originality ~ condition + (1 | object) + (1 | submitter_id)
Data: iriss_manual_ratings_metrics
REML criterion at convergence: 6856.7
Scaled residuals:
Min 1Q Median 3Q Max
-1.9786 -0.8738 -0.1548 0.7059 2.6612
Random effects:
Groups Name Variance Std.Dev.
submitter_id (Intercept) 0.12356 0.3515
object (Intercept) 0.04517 0.2125
Residual 1.37521 1.1727
Number of obs: 2130, groups: submitter_id, 141; object, 3
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 2.436143 0.140133 3.113507 17.384 0.000333 ***
conditionHigh-Agency -0.007137 0.096005 135.720987 -0.074 0.940847
conditionLow-Agency 0.076654 0.096377 138.589138 0.795 0.427767
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Correlation of Fixed Effects:
(Intr) cndH-A
cndtnHgh-Ag -0.341
cndtnLw-Agn -0.339 0.495
anova(iriss_originality_mod)
Type III Analysis of Variance Table with Satterthwaite's method
Sum Sq Mean Sq NumDF DenDF F value Pr(>F)
condition 1.2697 0.63485 2 137.95 0.4616 0.6312
Rows: 4500 Columns: 4
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): condition, object, submitter_id
dbl (1): similarity
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
According to the linear mixed model, participants in the low-agency condition produce significantly more similar ideas than the Control group. ### Noncompliance in AI Access conditions
This is JUST on the submission side. Here, we identify the trials on which high-agency participants and low-agency participants did not use AI, regardless of whether ideas from those conversations got evaluated.
# Some of these were not evaluatedno_chats_high_agency <-read_csv("data/pilot_no_first_message.csv")
Rows: 57 Columns: 36
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (6): participant_id, condition, object, aggregated_ideas, process_open-...
lgl (30): user_message_1, ai_response_1, user_message_2, ai_response_2, user...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 36 Columns: 36
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (6): participant_id, condition, object, aggregated_ideas, process_open-...
lgl (30): user_message_1, ai_response_1, user_message_2, ai_response_2, user...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
noncompliant_conversations <-bind_rows(no_chats_high_agency, no_chats_low_agency) %>%rename(submitter_id = participant_id) %>%mutate(condition=case_when(condition=="Passenger"~"Low-Agency", condition=="Pilot"~"High-Agency", condition=="Control"~"Control"))# Number of noncompliant conversationsnoncompliant_conversations %>%group_by(condition) %>%summarize(n_distinct(submitter_id))
After exclusions, there were 123 participants in total
# Exclude the noncompliant conversations from the analysiscompliant_conversations %>%group_by(condition) %>%summarize(unique_people =n_distinct(submitter_id))
# A tibble: 3 × 2
condition unique_people
<chr> <int>
1 Control 47
2 High-Agency 36
3 Low-Agency 40
write_csv(compliant_conversations, "data/evaluated_compliant_conversations.csv")compliant_conversations_process <- compliant_conversations %>%left_join(process_vars, by=c("submitter_id", "object")) %>%mutate(across(conv_with_ai_1:intrinsic_motivation_4, as.numeric),intrinsic_motivation_mean =rowMeans(across(intrinsic_motivation_1:intrinsic_motivation_4),na.rm =TRUE )) %>%mutate(conv_with_ai_mean =rowMeans(across(conv_with_ai_1:conv_with_ai_3),na.rm =TRUE )) %>%mutate(consideration_aid_mean =rowMeans(across(consideration_aid_1:consideration_aid_3),na.rm =TRUE ))# compliant_trials %>% # distinct(condition, submitter_id, object) %>% # filter(condition != "Control") %>% # count(condition)# # compliant_trials %>% # distinct(condition, submitter_id) %>% # count(condition)# Which participants were completely excluded because all of their trials were noncompliant?# excluded_participants <- noncompliant %>%# count(submitter_id, condition) %>%# filter(n==3) %>% # arrange(condition)# # write_csv(excluded_participants, "data/fully_excluded_participants.csv")# # excluded_participants %>% # count(condition)# # # Which participants had AT LEAST ONE noncompliant trial?# flagged_participants <- noncompliant %>%# distinct(submitter_id, condition)compliant_ideas <- compliant_conversations %>%separate_rows(aggregated_uses, sep ="; ") %>%rename(use="aggregated_uses") %>%distinct(submitter_id, condition, object, use)
Rows: 1435 Columns: 4
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): condition, object, submitter_id
dbl (1): similarity
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Age and gender covariates
# Add age and gender covariatesdemographics_recoded <- submitter_demographics %>%mutate(gender=if_else(gender=="Female",1, 0),race=if_else(race=="White or Caucasian",1,0))# # similarity_demographics <- left_join(compliant_similarities, demographics_recoded, by="submitter_id")
# Effect Size for ANOVA (Type III)
Parameter | Eta2 (partial) | 95% CI
-----------------------------------------
condition | 0.07 | [0.01, 1.00]
- One-sided CIs: upper bound fixed at [1.00].
# Get estimated marginal meansemm <-emmeans(compliant_semantic_similarity_mod, ~ condition, lmerTest.limit=3380)# With adjustment for multiple comparisons (Tukey is default)pairs(emm, adjust ="tukey", reverse =TRUE)
contrast estimate SE df t.ratio p.value
(High-Agency) - Control 0.019 0.0257 121 0.739 0.7408
(Low-Agency) - Control 0.074 0.0246 116 3.007 0.0090
(Low-Agency) - (High-Agency) 0.055 0.0268 123 2.056 0.1035
Degrees-of-freedom method: kenward-roger
P value adjustment: tukey method for comparing a family of 3 estimates
Saving 7 x 5 in image
Picking joint bandwidth of 0.0433
Around 24% of the low-agency conversations did not use AI and around 38% of the high-agency conversations did not use AI. Excluding the noncompliant trials did not change the results.
duration <- iriss %>%select(submitter_id, condition, contains("Duration")) %>%mutate(across(c(3:6), as.numeric)) %>%rename(duration_sec ="Duration (in seconds)") %>%mutate(duration_min = duration_sec/60) %>%relocate(duration_min, .after=duration_sec)duration %>%get_summary_stats(duration_min, type ="five_number")
# A tibble: 1 × 7
variable n min max q1 median q3
<fct> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 duration_min 150 5.73 10003. 16.9 21.9 31.3
# Timing variables refer to the amount of time it took the participant to read their condition's instructionstiming <- iriss %>%select(submitter_id, condition, contains("Page Submit")) %>%mutate(across(c(3:5), as.numeric)) %>%# Make all timer columns numericpivot_longer(cols=starts_with("timer"), names_to ="trial", values_to ="duration")
# Perceptions of uniqueness in the evaluated, compliant sampleuniqueness <- iriss_self_perceptions %>%right_join(compliant_conversations, by=c("submitter_id", "object")) %>%mutate(unique=as.numeric(unique))unique_lm <-lm(unique~condition.x, data=uniqueness)summary(unique_lm)
Call:
lm(formula = unique ~ condition.x, data = uniqueness)
Residuals:
Min 1Q Median 3Q Max
-1.4861 -0.4861 -0.2963 0.6630 1.7037
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.33696 0.08875 26.332 <2e-16 ***
condition.xHigh-Agency -0.04066 0.14593 -0.279 0.781
condition.xLow-Agency 0.14915 0.13394 1.114 0.267
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.8513 on 215 degrees of freedom
(119 observations deleted due to missingness)
Multiple R-squared: 0.00863, Adjusted R-squared: -0.0005921
F-statistic: 0.9358 on 2 and 215 DF, p-value: 0.3939
Rows: 207 Columns: 9
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (4): participant_id, condition, last_user_message, reasoning
dbl (5): last_turn, asking, doing, expressing, conversation_length
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 76 Columns: 8
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (2): ResponseId, Condition
dbl (6): total_count, direct_ask_count, constraining_count, backgrounding_co...
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Two Sample t-test
data: direct_ask_proportion by condition
t = -3.6733, df = 74, p-value = 0.00045
alternative hypothesis: true difference in means between group High-Agency and group Low-Agency is not equal to 0
95 percent confidence interval:
-0.5319985 -0.1578163
sample estimates:
mean in group High-Agency mean in group Low-Agency
0.3842593 0.7291667
# Does the proportion of direct asks correlate with person-level homogeneity?participant_compliant_similarity <- compliant_similarities %>%filter(condition %in%c("Low-Agency", "High-Agency")) %>%group_by(submitter_id, condition) %>%summarize(mean_similarity =mean(similarity))
`summarise()` has grouped output by 'submitter_id'. You can override using the
`.groups` argument.
direct_ask_similarity <-full_join(direct_asks, participant_compliant_similarity, by=c("submitter_id", "condition")) %>%mutate(direct_ask_binary =if_else(direct_ask_proportion==0, 0, 1)) # If no direct asks, they get coded as Guided Ideationggplot(direct_ask_similarity, aes(x=direct_ask_proportion,y=mean_similarity)) +geom_point() +stat_smooth()
`geom_smooth()` using method = 'loess' and formula = 'y ~ x'
Warning in cor.test.default(direct_ask_similarity$direct_ask_proportion, :
Cannot compute exact p-value with ties
Spearman's rank correlation rho
data: direct_ask_similarity$direct_ask_proportion and direct_ask_similarity$mean_similarity
S = 68626, p-value = 0.5956
alternative hypothesis: true rho is not equal to 0
sample estimates:
rho
0.06184404
direct_ask_sim_model <-lm(mean_similarity ~ direct_ask_proportion*condition, data = direct_ask_similarity)summary(direct_ask_sim_model)
Call:
lm(formula = mean_similarity ~ direct_ask_proportion * condition,
data = direct_ask_similarity)
Residuals:
Min 1Q Median 3Q Max
-0.19326 -0.10037 -0.02794 0.06258 0.45113
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.25063 0.03026 8.283 4.54e-12
direct_ask_proportion -0.01202 0.05051 -0.238 0.813
conditionLow-Agency 0.02967 0.05972 0.497 0.621
direct_ask_proportion:conditionLow-Agency 0.03012 0.08140 0.370 0.712
(Intercept) ***
direct_ask_proportion
conditionLow-Agency
direct_ask_proportion:conditionLow-Agency
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.1393 on 72 degrees of freedom
Multiple R-squared: 0.03148, Adjusted R-squared: -0.008876
F-statistic: 0.7801 on 3 and 72 DF, p-value: 0.5089
# Logistic regression predicting direct ask from conditiondirect_ask.log <-glm(direct_ask_binary ~ condition, family ="binomial", data=direct_ask_similarity)summary(direct_ask.log)
Call:
glm(formula = direct_ask_binary ~ condition, family = "binomial",
data = direct_ask_similarity)
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.2231 0.3354 -0.665 0.505868
conditionLow-Agency 2.4204 0.6247 3.875 0.000107 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 94.796 on 75 degrees of freedom
Residual deviance: 75.468 on 74 degrees of freedom
AIC: 79.468
Number of Fisher Scoring iterations: 4
exp(coef(direct_ask.log))
(Intercept) conditionLow-Agency
0.80 11.25
# Does direct ask binary predict mean similarity?direct_ask.lm <-lm(mean_similarity ~ direct_ask_binary*condition, data=direct_ask_similarity)summary(direct_ask.lm)
Call:
lm(formula = mean_similarity ~ direct_ask_binary * condition,
data = direct_ask_similarity)
Residuals:
Min 1Q Median 3Q Max
-0.19925 -0.10360 -0.02861 0.06717 0.43308
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.25679 0.03063 8.383 2.95e-12 ***
direct_ask_binary -0.02426 0.04595 -0.528 0.599
conditionLow-Agency -0.06132 0.07503 -0.817 0.416
direct_ask_binary:conditionLow-Agency 0.13318 0.08558 1.556 0.124
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.137 on 72 degrees of freedom
Multiple R-squared: 0.06288, Adjusted R-squared: 0.02384
F-statistic: 1.61 on 3 and 72 DF, p-value: 0.1944
