Frequency Effect on Word Choice
Main Model
In our main analysis, we considered participants’ likelihood of choosing the word for the nearest compass direction, dependent on whether that compass direction was a high- or a low-frequency word, while controlling for the distance from the nearest learned compass direction. We focused specifically on low-frequency/high-frequency trials, in which a compass direction was tested in between a low-frequency and a high-frequency trained direction.
As a conservative test, we retained only trials in which participants chose one of the two principal direction words within 45° of the stimulus direction (94.41% of all low-frequency/high-frequency trials).
#just trials with a left or right angle choice
#model
m <- glmer(matchChoice~hfTrial+angleDiffFromMatchC+(1+hfTrial+angleDiffFromMatchC|subjCode)+(1|targetLabel),data=subset(d, listChoice==1),family=binomial,glmerControl(optimizer="bobyqa"))
summary(m)
## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: matchChoice ~ hfTrial + angleDiffFromMatchC + (1 + hfTrial +
## angleDiffFromMatchC | subjCode) + (1 | targetLabel)
## Data: subset(d, listChoice == 1)
## Control: glmerControl(optimizer = "bobyqa")
##
## AIC BIC logLik deviance df.resid
## 2453.4 2514.6 -1216.7 2433.4 3348
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -8.9254 0.0982 0.2040 0.4370 2.2482
##
## Random effects:
## Groups Name Variance Std.Dev. Corr
## subjCode (Intercept) 0.240227 0.49013
## hfTrial 1.211711 1.10078 -0.26
## angleDiffFromMatchC 0.001383 0.03719 -0.98 0.06
## targetLabel (Intercept) 0.020018 0.14149
## Number of obs: 3358, groups: subjCode, 39; targetLabel, 18
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 2.34236 0.12216 19.174 < 2e-16 ***
## hfTrial 0.70564 0.20946 3.369 0.000755 ***
## angleDiffFromMatchC -0.22760 0.01347 -16.892 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) hfTril
## hfTrial -0.145
## anglDffFrMC -0.778 0.039
## convergence code: 0
## boundary (singular) fit: see ?isSingular
confint(m,method="Wald")[8:10,]
## 2.5 % 97.5 %
## (Intercept) 2.1029300 2.5817911
## hfTrial 0.2951182 1.1161689
## angleDiffFromMatchC -0.2540037 -0.2011891
#calculate shift in x-axis units (degrees of angle)
shift_x <- -(summary(m)$coefficients[1,1]+0.5*summary(m)$coefficients[2,1])/summary(m)$coefficients[3,1] + (summary(m)$coefficients[1,1]-0.5*summary(m)$coefficients[2,1])/summary(m)$coefficients[3,1]
#low 95% CI
shift_x_lower <- -(summary(m)$coefficients[1,1]+0.5*confint(m,method="Wald")[8:10,][2,1])/summary(m)$coefficients[3,1] + (summary(m)$coefficients[1,1]-0.5*confint(m,method="Wald")[8:10,][2,1])/summary(m)$coefficients[3,1]
#high 95% CI
shift_x_upper <- -(summary(m)$coefficients[1,1]+0.5*confint(m,method="Wald")[8:10,][2,2])/summary(m)$coefficients[3,1] + (summary(m)$coefficients[1,1]-0.5*confint(m,method="Wald")[8:10,][2,2])/summary(m)$coefficients[3,1]
## Main model has a singular fit warning - since this fit does not appear to impact fit, we retained the more complex random effects structure.
## However, we also fit a simplified model with the random slope for (the less theoretically important predictor) angleDiffFromMatchC removed, to ensure that the results are similar across different random effects structures and to alleviate concerns about a the boundary fit.
## This model yields very similar results (uncomment model below to view)
# m <- glmer(matchChoice~hfTrial+angleDiffFromMatchC+(1+hfTrial|subjCode)+(1|targetLabel),data=subset(d, listChoice==1),family=binomial,glmerControl(optimizer="bobyqa"))
# summary(m)
This effect corresponded to an estimated 3.1° shift (95% CI = [1.3°, 4.9°]) in participants’ decision boundary for high-frequency words as compared to low-frequency words.
Robustness Checks
Controlling for final retention accuracy of labels on each trial
To ensure that the frequency effect is not an artifact of participants’ being slightly more likely to forget the low-frequency labels, we first re-fit the model while controlling for participants’ accuracy during the Untimed Retention Test for the two (nearby) compass directions involved in each trial.
#controlling for accuracy for nearby labels
m=glmer(matchChoice~hfTrial+angleDiffFromMatchC+finalAccuracyNearbyLabels+(1+hfTrial+angleDiffFromMatchC+finalAccuracyNearbyLabels|subjCode)+(1|targetLabel),data=subset(d, listChoice==1),family=binomial,glmerControl(optimizer="bobyqa"))
summary(m)
## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula:
## matchChoice ~ hfTrial + angleDiffFromMatchC + finalAccuracyNearbyLabels +
## (1 + hfTrial + angleDiffFromMatchC + finalAccuracyNearbyLabels |
## subjCode) + (1 | targetLabel)
## Data: subset(d, listChoice == 1)
## Control: glmerControl(optimizer = "bobyqa")
##
## AIC BIC logLik deviance df.resid
## 2461.2 2553.0 -1215.6 2431.2 3343
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -9.3204 0.0982 0.2044 0.4364 2.2383
##
## Random effects:
## Groups Name Variance Std.Dev. Corr
## subjCode (Intercept) 0.210633 0.45895
## hfTrial 1.219664 1.10438 -0.83
## angleDiffFromMatchC 0.001256 0.03544 0.49 0.08
## finalAccuracyNearbyLabels 0.637278 0.79830 -0.81 0.35 -0.91
## targetLabel (Intercept) 0.017727 0.13314
## Number of obs: 3358, groups: subjCode, 39; targetLabel, 18
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 1.97635 0.94416 2.093 0.036329 *
## hfTrial 0.71757 0.21023 3.413 0.000642 ***
## angleDiffFromMatchC -0.22721 0.01337 -16.997 < 2e-16 ***
## finalAccuracyNearbyLabels 0.36827 0.95035 0.388 0.698380
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) hfTril anDFMC
## hfTrial -0.054
## anglDffFrMC -0.054 0.047
## fnlAccrcyNL -0.991 0.038 -0.046
## convergence code: 1
## boundary (singular) fit: see ?isSingular
confint(m,method="Wald")[12:15,]
## 2.5 % 97.5 %
## (Intercept) 0.1258202 3.8268763
## hfTrial 0.3055331 1.1296137
## angleDiffFromMatchC -0.2534079 -0.2010068
## finalAccuracyNearbyLabels -1.4943857 2.2309223
Including only participants with perfect recall for all compass directions at the end of the experiment
To further ensure that the frequency effect is not an artifact of participants’ being slightly more likely to forget the low-frequency labels, we next re-fit the same model using a stricter inclusion criterion, including only participants who recalled all items correctly during the Untimed Retetion test.
final_accuracy <- d %>%
filter(trialType=="finalName") %>%
group_by(subjCode,trialType) %>%
summarize(N=n(),accuracy=mean(isRight)) %>%
ungroup()
#select only participants with perfect recall on the final test block
perfect_final_accuracy_subjects <- as.character(filter(final_accuracy,accuracy==1)$subjCode)
m=glmer(matchChoice~hfTrial+angleDiffFromMatchC+(1+hfTrial+angleDiffFromMatchC|subjCode)+(1|targetLabel),data=subset(d, listChoice==1&subjCode %in% perfect_final_accuracy_subjects),family=binomial,glmerControl(optimizer="bobyqa"))
summary(m)
## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: matchChoice ~ hfTrial + angleDiffFromMatchC + (1 + hfTrial +
## angleDiffFromMatchC | subjCode) + (1 | targetLabel)
## Data:
## subset(d, listChoice == 1 & subjCode %in% perfect_final_accuracy_subjects)
## Control: glmerControl(optimizer = "bobyqa")
##
## AIC BIC logLik deviance df.resid
## 2108.7 2168.6 -1044.3 2088.7 2946
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -8.7064 0.0947 0.1974 0.4321 2.1530
##
## Random effects:
## Groups Name Variance Std.Dev. Corr
## subjCode (Intercept) 0.187489 0.43300
## hfTrial 0.785634 0.88636 -0.17
## angleDiffFromMatchC 0.001241 0.03523 -0.96 -0.11
## targetLabel (Intercept) 0.074566 0.27307
## Number of obs: 2956, groups: subjCode, 34; targetLabel, 18
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 2.46765 0.13898 17.755 < 2e-16 ***
## hfTrial 0.54033 0.19591 2.758 0.00582 **
## angleDiffFromMatchC -0.23838 0.01465 -16.268 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) hfTril
## hfTrial -0.085
## anglDffFrMC -0.722 -0.001
## convergence code: 0
## boundary (singular) fit: see ?isSingular
confint(m,method="Wald")[8:10,]
## 2.5 % 97.5 %
## (Intercept) 2.1952530 2.7400423
## hfTrial 0.1563494 0.9243204
## angleDiffFromMatchC -0.2671023 -0.2096603
## maximal model yields a singular fit
## model with simpler random-effects structure (random slope for angleDiffFromMatchC removed) yields similar results without singular fit
# m=glmer(matchChoice~hfTrial+angleDiffFromMatchC+(1+hfTrial|subjCode)+(1|targetLabel),data=subset(d, listChoice==1&subjCode %in% perfect_final_accuracy_subjects),family=binomial,glmerControl(optimizer="bobyqa"))
# summary(m)
Controlling for compass direction character length
The words given to each of the compass directions varied in character length (and therefore perhaps in how easy they are to produce/ type). Beyond randomly assigning compass directions and counterbalancing their roles across participants, we also fit all models with by-item random effects to ensure that the effect of frequency generalizes across items. In the following model, we also explicitly control for character length to ensure that the effects hold even after accounting for character length of the nearest/ target compass direction.
m <- glmer(matchChoice~hfTrial+angleDiffFromMatchC+nearestLabel_length+(1+hfTrial+angleDiffFromMatchC+nearestLabel_length|subjCode)+(1|targetLabel),data=subset(d, listChoice==1),family=binomial,glmerControl(optimizer="bobyqa"))
summary(m)
## Generalized linear mixed model fit by maximum likelihood (Laplace
## Approximation) [glmerMod]
## Family: binomial ( logit )
## Formula: matchChoice ~ hfTrial + angleDiffFromMatchC + nearestLabel_length +
## (1 + hfTrial + angleDiffFromMatchC + nearestLabel_length |
## subjCode) + (1 | targetLabel)
## Data: subset(d, listChoice == 1)
## Control: glmerControl(optimizer = "bobyqa")
##
## AIC BIC logLik deviance df.resid
## 2444.1 2535.9 -1207.1 2414.1 3343
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -8.0225 0.0931 0.2010 0.4242 2.3324
##
## Random effects:
## Groups Name Variance Std.Dev. Corr
## subjCode (Intercept) 2.899e+00 1.703e+00
## hfTrial 1.359e+00 1.166e+00 -0.04
## angleDiffFromMatchC 1.458e-03 3.819e-02 0.22 0.00
## nearestLabel_length 1.869e-01 4.323e-01 -0.96 -0.03 -0.46
## targetLabel (Intercept) 8.176e-09 9.042e-05
## Number of obs: 3358, groups: subjCode, 39; targetLabel, 18
##
## Fixed effects:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 2.57496 0.39227 6.564 5.23e-11 ***
## hfTrial 0.68557 0.21902 3.130 0.00175 **
## angleDiffFromMatchC -0.23066 0.01371 -16.829 < 2e-16 ***
## nearestLabel_length -0.04607 0.09452 -0.487 0.62595
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
## (Intr) hfTril anDFMC
## hfTrial -0.040
## anglDffFrMC -0.096 0.021
## nrstLbl_lng -0.955 -0.005 -0.144
## convergence code: 0
## boundary (singular) fit: see ?isSingular
confint(m,method="Wald")[8:10,]
## 2.5 % 97.5 %
## .sig08 NA NA
## .sig09 NA NA
## .sig10 NA NA
## maximal model yields a singular fit
## model with simpler random-effects structure (random slopes for angleDiffFromMatchC and nearestLabel_length removed) yields similar results without singular fit
# m <- glmer(matchChoice~hfTrial+angleDiffFromMatchC+nearestLabel_length+(1+hfTrial|subjCode)+(1|targetLabel),data=subset(d, listChoice==1),family=binomial,glmerControl(optimizer="bobyqa"))
# summary(m)
Plot
Plot the effect of training frequency on word/ compass direction choice.
#refit model without centering angle for simpler plotting (coefficients essentially equivalent)
m <- glmer(matchChoice~hfTrial+angleDiffFromMatch+(1+hfTrial+angleDiffFromMatch|subjCode)+(1|targetLabel),data=subset(d, listChoice==1),family=binomial,glmerControl(optimizer="bobyqa"))
pX <- expand.grid(angleDiffFromMatch=seq(0,22.5,by=0.1),hfTrial=c(-0.5,0.5))
predictions <- predictSE(m,pX,re.form=NA, type="response")
pX$fit <- predictions$fit
pX$se.fit <- predictions$se.fit
### Three different plotting designs - same effect ###
# Plot 1
q <- ggplot(subset(d, trialType=="test"&!is.na(matchChoice)),aes(angleDiffFromMatch,matchChoice,color=as.character(hfTrial)))+
geom_jitter(width=0.5,height=0.03,alpha=0.2)+
#geom_violinh(aes(y=as.factor(matchChoice),fill=as.character(hfTrial)),scale="count",width=0.3,alpha=0.3,color=NA)+
#geom_violinhalf(aes(y=as.factor(matchChoice),fill=as.character(hfTrial)),scale="count",width=0.3,alpha=0.3,color=NA,orientation="y")+
geom_smooth(data=pX,aes(y=fit,ymax=fit+se.fit,ymin=fit-se.fit,fill=as.character(hfTrial)),stat="identity")+
theme_classic(base_size=18)+
ylab("Probability of choosing\nnearest compass direction")+
# scale_color_brewer(palette="Set1",name="Frequency of Nearest Word",
# breaks=c(0.5,-0.5),
# labels=c("High-Frequency","Low-Frequency"),direction=-1)+
scale_color_manual(values = c("#E41A1C","#377EB8"),
name="Frequency of Compass Direction",
breaks=c(0.5,-0.5),
labels=c("High-Frequency","Low-Frequency"))+
# scale_fill_brewer(palette="Set1",name="Frequency of Nearest Compass Direction",
# breaks=c(0.5,-0.5),
# labels=c("High-Frequency","Low-Frequency"),direction=-1)+
scale_fill_manual(values = c("#E41A1C","#BDE0ED"),
name="Frequency of Compass Direction",
breaks=c(0.5,-0.5),
labels=c("High-Frequency","Low-Frequency"))+
xlab("Distance from nearest compass direction")+
geom_vline(xintercept=22.5,linetype="dashed")+
theme(legend.position=c(0.4,0.4))#+
#ylim(-0.05,1.05)
ggsave(here::here("plots","exp1_frequencyEffect_old.jpg"), width=9, height=6)
#Plot 2
p_freq1 <- ggplot(subset(d, trialType=="test"&!is.na(matchChoice)),aes(angleDiffFromMatch,as.factor(matchChoice),color=as.character(hfTrial)))+
geom_point(size = 0.5, alpha=0.3,shape=19,position = position_jitterdodge(jitter.width = 0.05,jitter.height = 0.5,
dodge.width = 0.2,
seed = 1
))+
geom_violinh(data=subset(d, trialType=="test"&!is.na(matchChoice)&hfTrial==0.5),aes(fill=as.character(hfTrial)),position = position_nudge(x = 0, y = .3 ),scale="count",width=0.4,alpha=0.5,color=NA)+
geom_violinh(data=subset(d, trialType=="test"&!is.na(matchChoice)&hfTrial==-0.5),aes(fill=as.character(hfTrial)),position = position_nudge(x = 0, y = -.3 ),scale="count",width=0.4,alpha=0.5,color=NA)+
geom_smooth(data=pX,aes(y=fit*4+1,ymax=(fit+se.fit)*4+1,ymin=(fit-se.fit)*4+1,fill=as.character(hfTrial)),stat="identity")+
theme_classic(base_size=18)+
ylab("Probability of choosing\nnearest compass direction")+
# scale_color_brewer(palette="Set1",name="Frequency of Nearest Word",
# breaks=c(0.5,-0.5),
# labels=c("High-Frequency","Low-Frequency"),direction=-1)+
scale_color_manual(values = c("#E41A1C","#377EB8"),
name="Frequency of Compass Direction",
breaks=c(0.5,-0.5),
labels=c("High-Frequency","Low-Frequency"))+
# scale_fill_brewer(palette="Set1",name="Frequency of Nearest Compass Direction",
# breaks=c(0.5,-0.5),
# labels=c("High-Frequency","Low-Frequency"),direction=-1)+
scale_fill_manual(values = c("#E41A1C","#BDE0ED"),
name="Frequency of Compass Direction",
breaks=c(0.5,-0.5),
labels=c("High-Frequency","Low-Frequency"))+
scale_y_discrete(limits=c("0","0.25","0.5","0.75","1"))+
xlab("Distance from nearest compass direction")+
geom_vline(xintercept=22.5,linetype="dashed")+
theme(legend.position=c(0.4,0.4))
p_freq1
ggsave(here::here("plots","exp1_frequencyEffect.jpg"), width=9, height=6)
# Plot 3
#alternate
p <- ggplot(subset(d, trialType=="test"&!is.na(matchChoice)),aes(angleDiffFromMatch,as.factor(matchChoice),color=as.character(hfTrial)))+
geom_point(size = 0.5, shape=19,alpha=0.2,position = position_jitterdodge(jitter.width = 0.05,jitter.height = 0.5,
dodge.width = 1.2,
seed = 1
))+
geom_violinh(data=subset(d, trialType=="test"&!is.na(matchChoice)&hfTrial==0.5),aes(fill=as.character(hfTrial)),position = position_nudge(x = 0, y = .3 ),scale="count",width=0.75,alpha=0.4, color=NA)+
geom_violinh(data=subset(d, trialType=="test"&!is.na(matchChoice)&hfTrial==-0.5),aes(fill=as.character(hfTrial)),position = position_nudge(x = 0, y = -.3 ),scale="count",width=0.75,alpha=0.4,color=NA)+
geom_smooth(data=pX,aes(y=fit*4+1,ymax=(fit+se.fit)*4+1,ymin=(fit-se.fit)*4+1,fill=as.character(hfTrial)),stat="identity")+
theme_classic(base_size=18)+
ylab("Probability of choosing\nnearest compass direction")+
# scale_color_brewer(palette="Set1",name="Frequency of Nearest Word",
# breaks=c(0.5,-0.5),
# labels=c("High-Frequency","Low-Frequency"),direction=-1)+
scale_color_manual(values = c("#E41A1C","#377EB8"),
name="Frequency of Compass Direction",
breaks=c(0.5,-0.5),
labels=c("High-Frequency","Low-Frequency"))+
# scale_fill_brewer(palette="Set1",name="Frequency of Nearest Compass Direction",
# breaks=c(0.5,-0.5),
# labels=c("High-Frequency","Low-Frequency"),direction=-1)+
scale_fill_manual(values = c("#E41A1C","#BDE0ED"),
name="Frequency of Compass Direction",
breaks=c(0.5,-0.5),
labels=c("High-Frequency","Low-Frequency"))+
scale_y_discrete(limits=c("0","0.25","0.5","0.75","1"))+
xlab("Distance from nearest compass direction")+
geom_vline(xintercept=22.5,linetype="dashed")+
theme(legend.position=c(0.4,0.4))
ggsave(here::here("plots","exp1_frequencyEffect_alternate.jpg"), width=9, height=6)
