Replication of the Behavioral Experiment by Daffner et al (2000, Journal of Cognitive Neuroscience)
Anjie Cao (anjiecao@stanford.edu)
November 20, 2020
Link to github
Link to experiment
Proposal
This particular experiment is chosen because I wanted to test the plausibility of using a subject-controlled visual oddball paradigm (VOP) in a web-based experiment, in the hope of using it as an avenue to explore factors influencing people’s looking time. In the infant literature, looking time is an extremely valuable measurement due to infants’ limited behavioral repertoire. Although it has been taken as indices for many perceptual and cognitive constructs, the field has not agreed on the best interpretations for looking time. The same results are often open to different interpretations, some even leading to opposite conclusions. This contention is due to the under-explored underlying mechanisms responsible for infants’ looking duration toward different stimuli. But due to the difficulty of recruiting infants participants, it is often quite challenging, if not impossible, to reach a sufficient sample size for well-powered analysis. In my project, I am hoping to first develop an analog paradigm in adults to explore the different factors responsible for looking time, and eventually import the insights back to the infant literature. VOP is one promising candidate for such analog. In Daffner et al. (2000), the researchers used this paradigm to show that adult viewing time was influenced by the complexity of the visual stimuli and the amount of exposure they received. These two factors were also considered to be highly relevant to infants’ looking time (Hunter & Ames, 1988). As a result, this paper was chosen.
The stimuli were a set of unique figures. They either came from a set of simple geometric shapes (simple), or a collection of unusual/unfamiliar figures (unusual). Each participant was tested in three tasks, the order of which was counterbalanced across participants. The three tasks were identical in the procedure but differed in the compositions of the stimuli. Each task contains 250 figures, presented in five blocks of 50 figures. Blocks were separated by a brief break, and tasks were separated by a longer break. In a given task, there are three types of stimuli in the 250 figures: background (175 out of 250, 70% frequency), target (37~ out of 250, about 15% frequency), and deviant (38 out of 250, about 15% frequency). The three tasks are All Simple Task, All Unusual Task, and the Mixed Stimuli Task. In the All Simple Task, background, target, and deviant stimuli all came from the simple geometric shapes set. In the All Unusual Task, all came from the unusual figure set. In the Mixed Stimuli Task, the background and target came from the unusual setting and the deviant came from the geometric set. In all three tasks, participants were instructed to view the figures on the screen. They were told that they can “look at each picture for however long or short they liked”(Daffner et al., 2000, pg 402). When they were done viewing a picture, they can push a button and triggered the onset of the next stimulus. Each stimulus was presented for a minimum of 600 msec. Two stimuli were separated by an interval ranged between 800 msec and 1300 msec. In the original study, they were also told to press a foot pedal when seeing the target stimuli because they served as “Sequence Markers” to help participants keep track of the progress of the experiments. In this replication, pressing the foot pedal would be substituted with a different key on the keyboard.
There are two potential challenges for replicating this study. First, the accuracy of the results. The web-based experiment can be noisier than a laboratory experiment when other things are being equal. The nature of this experiment calls for recording accurate response time (the average looking time is on the scale of milliseconds). In response to this challenge, I am hoping to use the “jspsych-psychophysics” plugin, which can deliver stimuli with more accurate onset time (Kuroki, 2000). Second, the engagement of participants. The original study tested participants in the laboratory setting with EEG caps. Naturally, they were less likely to be distracted and more likely to remain engaged throughout the experiment. It is unclear whether the participants in the online experiment would remain at the same level of engagement throughout. Web-version may need additional incentives in-between block or task to keep participants motivated.
Link to the repo
Link to the paper
Introduction
In Daffner et al., (2000), the authors used a subject-controlled visual oddball paradigm (VOP) to test healthy adult participants’ viewing time toward stimuli. Participants were instructed to look at pictures for as long as they wanted. They were also told to press a foot pedal when a “target stimulus” occurred on the screen. With a within-subject design, the experiment manipulated the visual complexity and occurring frequencies of the stimuli. There were three tasks for each participant. In each task, the participants viewed 250 pictures across five blocks. Of the 250 pictures, 70% of them were the background stimulus, which was one selected figure repeatedly appeared. 15% of the pictures were the target stimulus. The target stimulus had a similar level of visual complexity as the background stimuli. Like background stimuli, it also repeatedly appear. the participants were told to press a foot pedal upon seeing them. The remaining 15% of the pictures were the deviant stimuli, which were a unique set of figures, each appeared only once on the screen. The stimuli were drawn from two groups of stimuli, one with high visual complexity (unusual) and the other with low visual complexity (simple). The three tasks differed in the stimuli groups for the background, the target, and the deviant stimuli. In the “All simple task” and “All unusual task”, the three categories of stimuli were all drawn from the simple group or the unusual group. The “Mixed stimuli task” drawn the background and the target stimuli from the complex group, and the deviant stimuli from the simple group. The authors found: a) an interaction effect between task and stimulus type. Participants spent longer time looking at the unusual deviant stimuli in the All Unusual task than the simple deviant stimuli in the All simple task and the Mixed Stimuli task. b) a main effect of stimulus type. Across all three tasks, the participants looked longer at the deviant stimuli than the background stimuli. and c) a main effect of task type. In the All simple task, the participants looked shorter at the background stimuli comparing to when they were looking at the background stimuli in the All unusual task and the Mixed stimuli task.
These three findings will also be the target findings for the current replication. Note that there would be one crucial design difference in the current replication, in addition to the format difference: the current replication includes an additional task. The three tasks of the original design belong to a two (target and background versus deviant) by two (simple versus unusual) design. Since the goal of the current replication is to test the feasibility of this testing paradigm, I decided to include the omitted design from the two by two matrix. The fourth, “Mixed stimuli with the complex deviant task”, mirrored the components of the “Mixed task”, such that I will draw the background and target stimuli from the simple stimuli group and the deviant from the unusual group. This addition of new task leads to the predictions as following: a) in the interaction effect, we should see participants in the mixed stimuli with complex deviant tasks look similarly long at the deviant stimuli as in the All Complex task. The deviant stimuli in both tasks should have longer looking time than the deviant stimuli in the All Simple and Mixed tasks. b) The main effect applies to the new task as well, such that participants also looked longer at the deviant stimuli than the background stimuli. c) the participants in the new task should show a similar looking pattern like the ones in the All simple task, such that they looked shorter at the background stimuli comparing to when they were looking at the rest of the two tasks.
Methods
Power Analysis
I ran a simulation for the anova test. The code and results can be found here. Using the original ANOVA test, 25 participants is enough to reach power = 0.8, and about 40 participants can reach above 0.95 power. Due to design differences and the unaccounted for covariances, I plan to recruit 40 participants.
Planned Sample
The planned sample size is 40 They should be healthy adults without any known history of neurological problems or learning disabilities. Participants will be excluded following the Data exclusion rules specified in the Analysis Plan section.
Materials
“All stimuli either came from a set of simple geometric figures or a set of unusual/unfamiliar figures (e.g., fragmented or ‘‘impossible’’ objects), many of which were from the collection of drawings that have been used by Kosslyn et al. (1994) and Kroll and Potter (1984). The particular stimuli assigned to each task were randomly chosen from the two sets of figures noted above.” (Daffner et al., 2000, pg 402)
Note: - The stimuli used in the current replication is from the set of novel objects stimuli normed by Lewis & Frank (2016). The simple geometric figures were substituted with novel objects with complexity score in the lowest 1.5 quintile. The “unusual figures” were substituted with the novel objects in the highest 1.5 quintiles.
Procedure
“Subjects were introduced to the laboratory setting and, following electrode placement (see below), were seated in front of a computer monitor in a sound attenuated room. Subjects were told that the experiment involved the study of brain wave responses as they looked at different kinds of pictures. They were informed that they would be viewing a set of pictures and that they could look at each picture for however long or short they liked. They controlled the viewing duration by a button press that triggered the onset of the next stimulus. They were explicitly informed that they would not be asked questions about the pictures at the conclusion of the experiment. Subjects also were asked to respond to the designated target stimulus by pressing a foot pedal (ipsilateral to the button press). The targets were called ‘‘Sequence Markers’’ and subjects were told that they were included in the task to help the experimenters keep track of where they were in the sequence of drawings. As with the other stimuli, subjects controlled how long they looked at each of the targets. The 250 stimuli for each task were presented in five blocks of 50 stimuli each, with a brief break between each block. Longer breaks were taken between each task. The laterality of the responding hand and foot was counter- balanced across subjects. Although viewing durations were calculated by subtracting the stimulus onset time from the button press time, all stimuli were displayed for a minimum duration of 600 msec. The interval between the offset of one stimulus and the onset of the next stimulus ranged between 800 and 1300 msec.” (Daffner et al., 2000, pg 402 - 403)
Note: Because the current replication will be done online, I made modifications as following
- The current replication includes four tasks instead of three for reasons explained in the introduction section.
- The length of each task is shortened to 49 trials. This is to make sure that participants can go through the entire study via the online format.
- A practice phase is added to familiarize the participants with the test-setups.
- Also to adapt to the online format, the minimum viewing duration is reduced from 600 msec to 200 msec.
- Instead of pressing the foot pedal, the participants were prompted to press the space bar when seeing the target stimuli. When the participants press the space bar in the target trial, the fixation window’s border will flash red to indicate the participant has reacted. Participants can press the space bar in other trials as well but they will not see the visual feedback.
Analysis Plan
- Data cleaning rules: the script for extracting and aggregating relevant data from the raw CSV will be available in the SI.
- Data exclusion rules: In Daffner et al.(2000) the exclusion rule is based on criteria related to the EEG measurement, which is unobtainable with the online format. The current replication experiment has both participant-level exclusion criterion and the trial-level exclusion criterion. On the participant level, a participant is excluded from the final analysis if a) one presses the space bar for more than non-target 10% trials or b) one does not press the space bar for more than 10% of the target trial or c) indicate one has a history of neurological disorder or learning disabilities or d) missing data from more than 15% trials after applying the trial-based exclusion rule. On the trial-level, a trial is excluded if the RT is more than 3 standard deviation away from the log-transformed scale.
Clarify key analysis of interest here You can also pre-specify additional analyses you plan to do.
- 2 stimuli type x 4 task type repeated measures ANOVA
- Mixed effects linear regression: log(trial_looking_time) ~ trial_stimulus_type * block + (1|subject)
- Exploratory analysis:
- Demographics: log(trial_looking_time) ~ demog_age + (1|subject/block_trial_num); log(RT) ~ log(trial_looking_time) ~ demog_age + trial_stimulus_type * block + (1|subject/block_trial_num); log(trial_looking_time) ~ demog_education + (1 | subject/block_trial_num); log(trial_looking_time) ~ demog_education + trial_stimulus_type * block + (1 | subject/block_trial_num)
- Target reaction time: trial_space_bar_rt ~ trial_stimulus_complexity * block + (1 |subject/block_trial_num)
- Number of background stimuli: log(trial_looking_time) ~ bkgd_rep * block + (1 | subject/block_trial_num);
- Length of intervals between targets: log(trial_space_bar_rt) ~ target_interval * block + (1 | subject/block_trial_num)
- Effect of stimuli repetition: log(trial_looking_time) ~ trial_stimulus_type * block + block_trial_num * block_number + block_trial_num * trial_stimulus_type * num_times_stimulus_seen + (1 | subject/block_trial_num)
Differences from Original Study
Explicitly describe known differences in sample, setting, procedure, and analysis plan from the original study. The goal, of course, is to minimize those differences, but differences will inevitably occur. Also, note whether such differences are anticipated to make a difference based on claims in the original article or subsequent published research on the conditions for obtaining the effect.
- Sample differences: the original study tested 24 right handed undergraduate students with a mean age 20.54 years. I anticipated the current sample will have a higher mean age and a wider range of age. Because of the within-subject design, the age should not alter the claimed effects.
- Setting differences: the original study was conducted in the lab with the participants wearing EEG caps. The current setting is an online browser-based experiment. The setting difference can lead current participants to be less motivated and engaged. It can increase the noise in the data.
- Procedure differences: There are three differences. First, the original study asks the participant to press a foot pedal as a response to the target trial, and the current study changes it to be the space bar and also add visual feedback when the response is provided. Second, the original study includes three tasks and the current includes four. Neither of the change should make a difference in the effects. Third, the length of the task is cut to about the quarter of the original length.
- Analysis differences: for the confirmatory analysis, there is no difference. The exclusion rule is changed because the current replication effort will not collect EEG data.
Methods Addendum (Post Data Collection)
Actual Sample
Final sample includes 42 participants. 25 participants were tested but excluded in the final analysis based on the preregistered exclusion criteria (Due to not following the instruction for pressing space bar: N = 24; Due to lack of variation in reaction time or having too many outliers in reaction time: N = 25). The mean age of the final sample is 37(SD = 13) years old .
Differences from pre-data collection methods plan
Data collection does not have any difference from what was described as the original plan.
Results
Data preparation
Data preparation following the analysis plan. Scripts for preprocessing data can be seen here. And the preregistration can be seen here
library(tidyverse)
library(here)
library(rstatix)
library(lme4)
library(DT)
library(jsonlite)
library(tidyverse)
library(kableExtra)
source(here("writeup/daffner2000_analysis/script/00_read_raw_data.R"))
source(here("writeup/daffner2000_analysis/script/01_apply_exclusion_criteria.R"))
source(here("writeup/daffner2000_analysis/script/02_tidy_data.R"))
RAW_DATA_DIR <- here("writeup/daffner2000_analysis/raw_data/")
AGGREGATED_DATA_PATH <- here("writeup/daffner2000_analysis/processed_data/aggregated_data.csv")
PROCESSED_DATA_PATH <- here("writeup/daffner2000_analysis/processed_data/processed_data.csv")
####Aggregate across participants, clean up demographic fields
aggregate_data(RAW_DATA_DIR, AGGREGATED_DATA_PATH)
####Import data
data_with_demog <- read_csv(AGGREGATED_DATA_PATH)
#### Data exclusion / filtering
# participants level exclusion
filtered_data <- filter_demog(data_with_demog)
filtered_data <- filter_spacebar(filtered_data)
filtered_data <- filter_trial_variation(filtered_data)
filtered_data <- filter_too_many_outliers(filtered_data)
#trial level exclusion
filtered_data <- filter_outlier_trial(filtered_data)
#### Prepare data for analysis - create columns etc.
d <- tidy_data(filtered_data,PROCESSED_DATA_PATH) # this also writesConfirmatory analysis
preview data
d %>% datatable()# just look at the test presented in the results
anova_res <- anova_test(data = filter(d, block != "mixed_complex_deviant"),
formula = trial_looking_time ~ trial_stimulus_type * block,
dv = trial_looking_time)
anova_res## ANOVA Table (type II tests)
##
## Effect DFn DFd F p p<.05 ges
## 1 trial_stimulus_type 2 5797 449.163 4.42e-182 * 0.134
## 2 block 2 5797 5.450 4.00e-03 * 0.002
## 3 trial_stimulus_type:block 4 5797 2.729 2.80e-02 * 0.002d %>%
filter(trial_stimulus_type != "target", block != "mixed_complex_deviant") %>%
group_by(trial_stimulus_type, block) %>%
summarize(
mean = mean(as.numeric(trial_looking_time), na.rm = TRUE),
sd = sd(as.numeric(trial_looking_time), na.rm = TRUE),
n = n(),
ci_range_95 = qt(1 - (0.05 / 2), n - 1) * (sd/sqrt(n)),
ci_lower = mean - ci_range_95,
ci_upper = mean + ci_range_95
) %>%
mutate(
block_print = case_when(
block == "mixed_complex_deviant" ~ "Complex Deviant Block",
block == "mixed_simple_deviant" ~ "Simple Deviant Block",
block == "all_simple" ~ "All Simple Block",
block == "all_complex" ~ "All Complex Block"
),
stimulus_type_print = case_when(
trial_stimulus_type == "background" ~ "Background",
trial_stimulus_type == "deviant" ~ "Deviant"
)
) %>%
ggplot(aes(x = block_print, y = mean )) +
geom_col(aes(fill = block_print), colour = "black") +
scale_x_discrete(limits = c( "All Simple Block", "All Complex Block",
"Simple Deviant Block")) +
scale_fill_manual(values = c("gray","white", "black")) +
guides(fill = FALSE) +
geom_errorbar(aes(ymin = ci_lower, ymax = ci_upper), width = .2) +
facet_wrap(~ fct_rev(stimulus_type_print)) +
xlab("") +
ylab("Mean Looking Time") +
theme_classic() +
theme(axis.text.x = element_text(angle = 45, vjust = 1, hjust=1))## `summarise()` regrouping output by 'trial_stimulus_type' (override with `.groups` argument)
› Original Plot: 
Exploratory analyses
Mixed effect linear regression model
basic_m <- lmer(log(trial_looking_time) ~ trial_stimulus_type * block + (1|subject),
data = d)
summary(basic_m)## Linear mixed model fit by REML ['lmerMod']
## Formula: log(trial_looking_time) ~ trial_stimulus_type * block + (1 |
## subject)
## Data: d
##
## REML criterion at convergence: 10616.3
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.3471 -0.5241 -0.1523 0.3587 5.4042
##
## Random effects:
## Groups Name Variance Std.Dev.
## subject (Intercept) 0.1001 0.3164
## Residual 0.2227 0.4719
## Number of obs: 7769, groups: subject, 42
##
## Fixed effects:
## Estimate Std. Error
## (Intercept) 6.218527 0.050443
## trial_stimulus_typedeviant 0.219409 0.030670
## trial_stimulus_typetarget 0.613224 0.031048
## blockall_simple 0.006138 0.017736
## blockmixed_complex_deviant 0.027101 0.017838
## blockmixed_simple_deviant 0.013656 0.018082
## trial_stimulus_typedeviant:blockall_simple 0.012400 0.043358
## trial_stimulus_typetarget:blockall_simple 0.042841 0.043881
## trial_stimulus_typedeviant:blockmixed_complex_deviant 0.015950 0.043425
## trial_stimulus_typetarget:blockmixed_complex_deviant 0.110322 0.044058
## trial_stimulus_typedeviant:blockmixed_simple_deviant 0.040369 0.044142
## trial_stimulus_typetarget:blockmixed_simple_deviant 0.166104 0.044662
## t value
## (Intercept) 123.277
## trial_stimulus_typedeviant 7.154
## trial_stimulus_typetarget 19.751
## blockall_simple 0.346
## blockmixed_complex_deviant 1.519
## blockmixed_simple_deviant 0.755
## trial_stimulus_typedeviant:blockall_simple 0.286
## trial_stimulus_typetarget:blockall_simple 0.976
## trial_stimulus_typedeviant:blockmixed_complex_deviant 0.367
## trial_stimulus_typetarget:blockmixed_complex_deviant 2.504
## trial_stimulus_typedeviant:blockmixed_simple_deviant 0.915
## trial_stimulus_typetarget:blockmixed_simple_deviant 3.719
##
## Correlation of Fixed Effects:
## (Intr) trl_stmls_typd trl_stmls_typt blckl_
## trl_stmls_typd -0.101
## trl_stmls_typt -0.100 0.164
## blckll_smpl -0.175 0.288 0.284
## blckmxd_cm_ -0.177 0.286 0.283 0.495
## blckmxd_sm_ -0.172 0.282 0.279 0.489
## trl_stmls_typd:_ 0.072 -0.707 -0.116 -0.409
## trl_stmls_typt:_ 0.071 -0.116 -0.707 -0.404
## trl_stmls_typdvnt:blckmxd_c_ 0.071 -0.706 -0.116 -0.203
## trl_stmls_typtrgt:blckmxd_c_ 0.071 -0.116 -0.705 -0.200
## trl_stmls_typdvnt:blckmxd_s_ 0.070 -0.695 -0.114 -0.200
## trl_stmls_typtrgt:blckmxd_s_ 0.070 -0.114 -0.695 -0.198
## blckmxd_c_ blckmxd_s_ trl_stmls_typd:_
## trl_stmls_typd
## trl_stmls_typt
## blckll_smpl
## blckmxd_cm_
## blckmxd_sm_ 0.490
## trl_stmls_typd:_ -0.203 -0.200
## trl_stmls_typt:_ -0.200 -0.197 0.165
## trl_stmls_typdvnt:blckmxd_c_ -0.406 -0.200 0.500
## trl_stmls_typtrgt:blckmxd_c_ -0.402 -0.197 0.082
## trl_stmls_typdvnt:blckmxd_s_ -0.199 -0.406 0.491
## trl_stmls_typtrgt:blckmxd_s_ -0.197 -0.401 0.081
## trl_stmls_typt:_ trl_stmls_typdvnt:blckmxd_c_
## trl_stmls_typd
## trl_stmls_typt
## blckll_smpl
## blckmxd_cm_
## blckmxd_sm_
## trl_stmls_typd:_
## trl_stmls_typt:_
## trl_stmls_typdvnt:blckmxd_c_ 0.082
## trl_stmls_typtrgt:blckmxd_c_ 0.499 0.164
## trl_stmls_typdvnt:blckmxd_s_ 0.081 0.491
## trl_stmls_typtrgt:blckmxd_s_ 0.492 0.081
## trl_stmls_typtrgt:blckmxd_c_
## trl_stmls_typd
## trl_stmls_typt
## blckll_smpl
## blckmxd_cm_
## blckmxd_sm_
## trl_stmls_typd:_
## trl_stmls_typt:_
## trl_stmls_typdvnt:blckmxd_c_
## trl_stmls_typtrgt:blckmxd_c_
## trl_stmls_typdvnt:blckmxd_s_ 0.080
## trl_stmls_typtrgt:blckmxd_s_ 0.490
## trl_stmls_typdvnt:blckmxd_s_
## trl_stmls_typd
## trl_stmls_typt
## blckll_smpl
## blckmxd_cm_
## blckmxd_sm_
## trl_stmls_typd:_
## trl_stmls_typt:_
## trl_stmls_typdvnt:blckmxd_c_
## trl_stmls_typtrgt:blckmxd_c_
## trl_stmls_typdvnt:blckmxd_s_
## trl_stmls_typtrgt:blckmxd_s_ 0.164Effect of age
age_only_m <- lmer(log(trial_looking_time) ~ demog_age + (1|subject/block_trial_num), data = d)
age_with_task_m <- lmer(log(trial_looking_time) ~ demog_age + trial_stimulus_type * block + (1|subject/block_trial_num),
data = d)
summary(age_only_m)## Linear mixed model fit by REML ['lmerMod']
## Formula: log(trial_looking_time) ~ demog_age + (1 | subject/block_trial_num)
## Data: d
##
## REML criterion at convergence: 11616.2
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -3.0464 -0.6208 -0.2932 0.3287 4.4873
##
## Random effects:
## Groups Name Variance Std.Dev.
## block_trial_num:subject (Intercept) 0.008916 0.09442
## subject (Intercept) 0.101602 0.31875
## Residual 0.269307 0.51895
## Number of obs: 7351, groups: block_trial_num:subject, 1858; subject, 38
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 6.225350 0.158696 39.228
## demog_age 0.003795 0.004048 0.937
##
## Correlation of Fixed Effects:
## (Intr)
## demog_age -0.945summary(age_with_task_m)## Linear mixed model fit by REML ['lmerMod']
## Formula: log(trial_looking_time) ~ demog_age + trial_stimulus_type * block +
## (1 | subject/block_trial_num)
## Data: d
##
## REML criterion at convergence: 9750.6
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.4608 -0.5059 -0.1394 0.3594 5.4264
##
## Random effects:
## Groups Name Variance Std.Dev.
## block_trial_num:subject (Intercept) 0.005566 0.07461
## subject (Intercept) 0.103177 0.32121
## Residual 0.208398 0.45651
## Number of obs: 7351, groups: block_trial_num:subject, 1858; subject, 38
##
## Fixed effects:
## Estimate Std. Error
## (Intercept) 6.082163 0.160019
## demog_age 0.003713 0.004072
## trial_stimulus_typedeviant 0.243242 0.031208
## trial_stimulus_typetarget 0.656823 0.031571
## blockall_simple 0.002627 0.017861
## blockmixed_complex_deviant 0.020230 0.017839
## blockmixed_simple_deviant 0.018042 0.017837
## trial_stimulus_typedeviant:blockall_simple -0.010993 0.044107
## trial_stimulus_typetarget:blockall_simple 0.051140 0.044572
## trial_stimulus_typedeviant:blockmixed_complex_deviant -0.003821 0.044083
## trial_stimulus_typetarget:blockmixed_complex_deviant 0.099901 0.044643
## trial_stimulus_typedeviant:blockmixed_simple_deviant 0.010628 0.044131
## trial_stimulus_typetarget:blockmixed_simple_deviant 0.124567 0.044573
## t value
## (Intercept) 38.009
## demog_age 0.912
## trial_stimulus_typedeviant 7.794
## trial_stimulus_typetarget 20.805
## blockall_simple 0.147
## blockmixed_complex_deviant 1.134
## blockmixed_simple_deviant 1.012
## trial_stimulus_typedeviant:blockall_simple -0.249
## trial_stimulus_typetarget:blockall_simple 1.147
## trial_stimulus_typedeviant:blockmixed_complex_deviant -0.087
## trial_stimulus_typetarget:blockmixed_complex_deviant 2.238
## trial_stimulus_typedeviant:blockmixed_simple_deviant 0.241
## trial_stimulus_typetarget:blockmixed_simple_deviant 2.795Effect of education
edu_only_m <- lmer(log(trial_looking_time) ~ demog_education + (1|subject/block_trial_num), data = d)
edu_with_task_m <- lmer(log(trial_looking_time) ~ demog_education + trial_stimulus_type * block + (1|subject/block_trial_num), data = d)
summary(edu_only_m)## Linear mixed model fit by REML ['lmerMod']
## Formula:
## log(trial_looking_time) ~ demog_education + (1 | subject/block_trial_num)
## Data: d
##
## REML criterion at convergence: 11609.1
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -3.0468 -0.6197 -0.2929 0.3274 4.4862
##
## Random effects:
## Groups Name Variance Std.Dev.
## block_trial_num:subject (Intercept) 0.008915 0.09442
## subject (Intercept) 0.097611 0.31243
## Residual 0.269308 0.51895
## Number of obs: 7351, groups: block_trial_num:subject, 1858; subject, 38
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 5.95051 0.27490 21.646
## demog_education 0.10737 0.06983 1.538
##
## Correlation of Fixed Effects:
## (Intr)
## demog_edctn -0.983summary(edu_with_task_m)## Linear mixed model fit by REML ['lmerMod']
## Formula: log(trial_looking_time) ~ demog_education + trial_stimulus_type *
## block + (1 | subject/block_trial_num)
## Data: d
##
## REML criterion at convergence: 9743.4
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.4624 -0.5053 -0.1399 0.3592 5.4276
##
## Random effects:
## Groups Name Variance Std.Dev.
## block_trial_num:subject (Intercept) 0.005564 0.07459
## subject (Intercept) 0.099041 0.31471
## Residual 0.208401 0.45651
## Number of obs: 7351, groups: block_trial_num:subject, 1858; subject, 38
##
## Fixed effects:
## Estimate Std. Error
## (Intercept) 5.802903 0.276588
## demog_education 0.107728 0.070194
## trial_stimulus_typedeviant 0.243244 0.031208
## trial_stimulus_typetarget 0.656832 0.031571
## blockall_simple 0.002628 0.017861
## blockmixed_complex_deviant 0.020230 0.017840
## blockmixed_simple_deviant 0.018043 0.017837
## trial_stimulus_typedeviant:blockall_simple -0.011006 0.044107
## trial_stimulus_typetarget:blockall_simple 0.051136 0.044572
## trial_stimulus_typedeviant:blockmixed_complex_deviant -0.003821 0.044083
## trial_stimulus_typetarget:blockmixed_complex_deviant 0.099901 0.044643
## trial_stimulus_typedeviant:blockmixed_simple_deviant 0.010618 0.044131
## trial_stimulus_typetarget:blockmixed_simple_deviant 0.124563 0.044573
## t value
## (Intercept) 20.980
## demog_education 1.535
## trial_stimulus_typedeviant 7.794
## trial_stimulus_typetarget 20.805
## blockall_simple 0.147
## blockmixed_complex_deviant 1.134
## blockmixed_simple_deviant 1.012
## trial_stimulus_typedeviant:blockall_simple -0.250
## trial_stimulus_typetarget:blockall_simple 1.147
## trial_stimulus_typedeviant:blockmixed_complex_deviant -0.087
## trial_stimulus_typetarget:blockmixed_complex_deviant 2.238
## trial_stimulus_typedeviant:blockmixed_simple_deviant 0.241
## trial_stimulus_typetarget:blockmixed_simple_deviant 2.795##
## Correlation matrix not shown by default, as p = 13 > 12.
## Use print(x, correlation=TRUE) or
## vcov(x) if you need itTask effect on target reaction time
target_rt_m <- lmer(trial_space_bar_rt ~ trial_stimulus_complexity * block + (1|subject/block_trial_num),
data = filter(d, trial_stimulus_type == "target"))## fixed-effect model matrix is rank deficient so dropping 4 columns / coefficients## boundary (singular) fit: see ?isSingularsummary(target_rt_m)## Linear mixed model fit by REML ['lmerMod']
## Formula: trial_space_bar_rt ~ trial_stimulus_complexity * block + (1 |
## subject/block_trial_num)
## Data: filter(d, trial_stimulus_type == "target")
##
## REML criterion at convergence: 9850.6
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -3.5852 -0.4457 -0.1665 0.1590 8.1101
##
## Random effects:
## Groups Name Variance Std.Dev.
## block_trial_num:subject (Intercept) 0 0.0
## subject (Intercept) 52763 229.7
## Residual 100759 317.4
## Number of obs: 683, groups: block_trial_num:subject, 579; subject, 37
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 546.021 46.296 11.794
## trial_stimulus_complexitysimple 3.351 34.858 0.096
## blockall_simple -8.867 35.090 -0.253
## blockmixed_simple_deviant -20.540 34.781 -0.591
##
## Correlation of Fixed Effects:
## (Intr) trl_s_ blckl_
## trl_stmls_c -0.403
## blckll_smpl 0.001 -0.459
## blckmxd_sm_ -0.396 0.534 -0.002
## fit warnings:
## fixed-effect model matrix is rank deficient so dropping 4 columns / coefficients
## convergence code: 0
## boundary (singular) fit: see ?isSingularNumber of background stimuli
no_target <- d %>%
filter(trial_stimulus_type != "target")
no_target <- no_target %>%
mutate(
temp_id = paste0(subject, block_number)
)
no_target$block_trial_num <- sequence(rle(no_target$temp_id)$lengths)
no_target <- no_target %>% select(-temp_id)
repetition_bckgd <- no_target %>%
group_by(subject, block) %>%
filter(trial_stimulus_type == "deviant") %>%
mutate(
bkgd_rep = block_trial_num - (lag(block_trial_num, default = block_trial_num[1])) - 1
)
num_bkgd <- lmer(log(trial_looking_time) ~ bkgd_rep * block + (1 | subject / block_trial_num),
data = repetition_bckgd)
summary(num_bkgd)## Linear mixed model fit by REML ['lmerMod']
## Formula:
## log(trial_looking_time) ~ bkgd_rep * block + (1 | subject/block_trial_num)
## Data: repetition_bckgd
##
## REML criterion at convergence: 1551.9
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -3.5420 -0.5967 -0.0741 0.4283 4.7766
##
## Random effects:
## Groups Name Variance Std.Dev.
## block_trial_num:subject (Intercept) 0.006006 0.0775
## subject (Intercept) 0.130025 0.3606
## Residual 0.197336 0.4442
## Number of obs: 1117, groups: block_trial_num:subject, 864; subject, 42
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 6.481405 0.070200 92.327
## bkgd_rep -0.011598 0.008509 -1.363
## blockall_simple -0.009373 0.058746 -0.160
## blockmixed_complex_deviant 0.110241 0.060121 1.834
## blockmixed_simple_deviant 0.017269 0.060669 0.285
## bkgd_rep:blockall_simple 0.007203 0.011729 0.614
## bkgd_rep:blockmixed_complex_deviant -0.018252 0.012194 -1.497
## bkgd_rep:blockmixed_simple_deviant 0.009273 0.011975 0.774
##
## Correlation of Fixed Effects:
## (Intr) bkgd_r blckl_ blckmxd_c_ blckmxd_s_ bkg_:_
## bkgd_rep -0.472
## blckll_smpl -0.438 0.562
## blckmxd_cm_ -0.432 0.550 0.512
## blckmxd_sm_ -0.424 0.544 0.507 0.499
## bkgd_rp:bl_ 0.341 -0.723 -0.767 -0.399 -0.395
## bkgd_rp:blckmxd_c_ 0.329 -0.696 -0.392 -0.775 -0.381 0.504
## bkgd_rp:blckmxd_s_ 0.334 -0.709 -0.399 -0.392 -0.771 0.514
## bkgd_rp:blckmxd_c_
## bkgd_rep
## blckll_smpl
## blckmxd_cm_
## blckmxd_sm_
## bkgd_rp:bl_
## bkgd_rp:blckmxd_c_
## bkgd_rp:blckmxd_s_ 0.495
## convergence code: 0
## Model failed to converge with max|grad| = 0.00460006 (tol = 0.002, component 1)repetition_bckgd %>%
ggplot(aes(y = log(trial_looking_time),
x = bkgd_rep,
color = trial_stimulus_complexity)) +
geom_point() +
geom_smooth(method = "lm") +
labs(title = "looking time at deviant stimuli w/ number of background stimuli preceding it ")## `geom_smooth()` using formula 'y ~ x'
repetition_bckgd %>%
ggplot(aes(y = log(trial_looking_time),
x = bkgd_rep,
color = trial_stimulus_complexity)) +
geom_violin() +
labs(title = "looking time at deviant stimuli w/ number of background stimuli preceding it ")## Warning: position_dodge requires non-overlapping x intervals
repetition_bckgd %>%
mutate(
complexity_combination = case_when(
grepl("all", block) ~ "same",
grepl("mixed", block) ~ "different"
)
) %>%
ggplot(aes(y = log(trial_looking_time),
x = bkgd_rep,
color = complexity_combination)) +
geom_point() +
geom_smooth(method = "lm") +
labs(title = "looking time at deviant stimuli w/ number of background stimuli preceding it ")## `geom_smooth()` using formula 'y ~ x'
repetition_bckgd %>%
mutate(
complexity_combination = case_when(
grepl("all", block) ~ "same",
grepl("mixed", block) ~ "different"
)
) %>%
ggplot(aes(y = log(trial_looking_time),
x = bkgd_rep,
color = complexity_combination)) +
geom_violin() +
labs(title = "looking time at deviant stimuli w/ number of background stimuli preceding it ")## Warning: position_dodge requires non-overlapping x intervals
Length of intervals between targets
target_interval <- d %>%
group_by(subject, block) %>%
filter(trial_stimulus_type == "target") %>%
mutate(
target_interval = block_trial_num - (lag(block_trial_num, default = block_trial_num[1])) - 1
)
interval_length_m <- lmer(log(trial_space_bar_rt) ~ target_interval * block + (1 | subject / block_trial_num),
data = target_interval)
summary(interval_length_m)## Linear mixed model fit by REML ['lmerMod']
## Formula:
## log(trial_space_bar_rt) ~ target_interval * block + (1 | subject/block_trial_num)
## Data: target_interval
##
## REML criterion at convergence: 1062.3
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.9795 -0.4867 -0.0424 0.3761 5.0723
##
## Random effects:
## Groups Name Variance Std.Dev.
## block_trial_num:subject (Intercept) 0.0000 0.0000
## subject (Intercept) 0.2955 0.5436
## Residual 0.2211 0.4702
## Number of obs: 683, groups: block_trial_num:subject, 579; subject, 37
##
## Fixed effects:
## Estimate Std. Error t value
## (Intercept) 6.0219215 0.1035899 58.132
## target_interval -0.0017043 0.0071273 -0.239
## blockall_simple -0.0501688 0.0707993 -0.709
## blockmixed_complex_deviant 0.0086013 0.0684352 0.126
## blockmixed_simple_deviant -0.0479775 0.0680868 -0.705
## target_interval:blockall_simple 0.0036876 0.0098542 0.374
## target_interval:blockmixed_complex_deviant -0.0008751 0.0095306 -0.092
## target_interval:blockmixed_simple_deviant 0.0063034 0.0102954 0.612
##
## Correlation of Fixed Effects:
## (Intr) trgt_n blckl_ blckmxd_c_ blckmxd_s_ trg_:_
## targt_ntrvl -0.319
## blckll_smpl -0.344 0.471
## blckmxd_cm_ -0.359 0.487 0.530
## blckmxd_sm_ -0.356 0.494 0.526 0.550
## trgt_ntrv:_ 0.231 -0.726 -0.647 -0.355 -0.357
## trgt_ntrvl:blckmxd_c_ 0.241 -0.747 -0.357 -0.654 -0.371 0.546
## trgt_ntrvl:blckmxd_s_ 0.224 -0.696 -0.329 -0.342 -0.650 0.507
## trgt_ntrvl:blckmxd_c_
## targt_ntrvl
## blckll_smpl
## blckmxd_cm_
## blckmxd_sm_
## trgt_ntrv:_
## trgt_ntrvl:blckmxd_c_
## trgt_ntrvl:blckmxd_s_ 0.521
## convergence code: 0
## boundary (singular) fit: see ?isSingulareffect of stimuli repetition
count_num_seen <- d %>%
mutate(
number = 1) %>%
group_by(
subject, block, trial_stimulus_type
) %>%
# calculate how many times the stimuli has been shown consecutively
mutate(num_times_stimulus_seen = cumsum(number))
stimuli_rep_m <- lmer(log(trial_looking_time) ~ trial_stimulus_type * block + block_trial_num * block_number + block_trial_num * trial_stimulus_type * num_times_stimulus_seen + (1 | subject / block_trial_num),
data = count_num_seen)
summary(stimuli_rep_m)## Linear mixed model fit by REML ['lmerMod']
## Formula:
## log(trial_looking_time) ~ trial_stimulus_type * block + block_trial_num *
## block_number + block_trial_num * trial_stimulus_type * num_times_stimulus_seen +
## (1 | subject/block_trial_num)
## Data: count_num_seen
##
## REML criterion at convergence: 10500.2
##
## Scaled residuals:
## Min 1Q Median 3Q Max
## -4.3106 -0.5312 -0.1265 0.3749 5.3908
##
## Random effects:
## Groups Name Variance Std.Dev.
## block_trial_num:subject (Intercept) 0.00273 0.05225
## subject (Intercept) 0.10040 0.31686
## Residual 0.21348 0.46204
## Number of obs: 7769, groups: block_trial_num:subject, 2035; subject, 42
##
## Fixed effects:
## Estimate
## (Intercept) 6.517e+00
## trial_stimulus_typedeviant 2.064e-01
## trial_stimulus_typetarget 6.974e-01
## blockall_simple 6.736e-03
## blockmixed_complex_deviant 2.343e-02
## blockmixed_simple_deviant 2.091e-02
## block_trial_num 3.014e-03
## block_number -6.808e-02
## num_times_stimulus_seen -3.134e-02
## trial_stimulus_typedeviant:blockall_simple 1.492e-02
## trial_stimulus_typetarget:blockall_simple 3.508e-02
## trial_stimulus_typedeviant:blockmixed_complex_deviant 1.351e-02
## trial_stimulus_typetarget:blockmixed_complex_deviant 1.043e-01
## trial_stimulus_typedeviant:blockmixed_simple_deviant 3.663e-02
## trial_stimulus_typetarget:blockmixed_simple_deviant 1.584e-01
## block_trial_num:block_number 1.504e-03
## trial_stimulus_typedeviant:block_trial_num -1.551e-02
## trial_stimulus_typetarget:block_trial_num -9.064e-03
## block_trial_num:num_times_stimulus_seen 3.865e-04
## trial_stimulus_typedeviant:num_times_stimulus_seen -1.426e-02
## trial_stimulus_typetarget:num_times_stimulus_seen -7.278e-02
## trial_stimulus_typedeviant:block_trial_num:num_times_stimulus_seen 1.557e-03
## trial_stimulus_typetarget:block_trial_num:num_times_stimulus_seen 1.444e-03
## Std. Error
## (Intercept) 5.570e-02
## trial_stimulus_typedeviant 6.044e-02
## trial_stimulus_typetarget 6.456e-02
## blockall_simple 1.745e-02
## blockmixed_complex_deviant 1.751e-02
## blockmixed_simple_deviant 1.782e-02
## block_trial_num 4.388e-03
## block_number 9.581e-03
## num_times_stimulus_seen 6.147e-03
## trial_stimulus_typedeviant:blockall_simple 4.297e-02
## trial_stimulus_typetarget:blockall_simple 4.327e-02
## trial_stimulus_typedeviant:blockmixed_complex_deviant 4.278e-02
## trial_stimulus_typetarget:blockmixed_complex_deviant 4.342e-02
## trial_stimulus_typedeviant:blockmixed_simple_deviant 4.355e-02
## trial_stimulus_typetarget:blockmixed_simple_deviant 4.415e-02
## block_trial_num:block_number 3.386e-04
## trial_stimulus_typedeviant:block_trial_num 6.195e-03
## trial_stimulus_typetarget:block_trial_num 5.422e-03
## block_trial_num:num_times_stimulus_seen 5.032e-05
## trial_stimulus_typedeviant:num_times_stimulus_seen 2.860e-02
## trial_stimulus_typetarget:num_times_stimulus_seen 2.286e-02
## trial_stimulus_typedeviant:block_trial_num:num_times_stimulus_seen 5.984e-04
## trial_stimulus_typetarget:block_trial_num:num_times_stimulus_seen 5.990e-04
## t value
## (Intercept) 116.998
## trial_stimulus_typedeviant 3.415
## trial_stimulus_typetarget 10.801
## blockall_simple 0.386
## blockmixed_complex_deviant 1.338
## blockmixed_simple_deviant 1.173
## block_trial_num 0.687
## block_number -7.106
## num_times_stimulus_seen -5.099
## trial_stimulus_typedeviant:blockall_simple 0.347
## trial_stimulus_typetarget:blockall_simple 0.811
## trial_stimulus_typedeviant:blockmixed_complex_deviant 0.316
## trial_stimulus_typetarget:blockmixed_complex_deviant 2.403
## trial_stimulus_typedeviant:blockmixed_simple_deviant 0.841
## trial_stimulus_typetarget:blockmixed_simple_deviant 3.588
## block_trial_num:block_number 4.441
## trial_stimulus_typedeviant:block_trial_num -2.504
## trial_stimulus_typetarget:block_trial_num -1.672
## block_trial_num:num_times_stimulus_seen 7.682
## trial_stimulus_typedeviant:num_times_stimulus_seen -0.499
## trial_stimulus_typetarget:num_times_stimulus_seen -3.184
## trial_stimulus_typedeviant:block_trial_num:num_times_stimulus_seen 2.602
## trial_stimulus_typetarget:block_trial_num:num_times_stimulus_seen 2.410##
## Correlation matrix not shown by default, as p = 23 > 12.
## Use print(x, correlation=TRUE) or
## vcov(x) if you need itcount_num_seen %>%
ggplot(
aes(y = log(trial_looking_time),
x = num_times_stimulus_seen,
color = trial_stimulus_complexity)
) +
geom_point(aes(alpha = 0.2)) +
guides(alpha = FALSE) +
labs(color = "Stimulus Complexity") +
ylab("Mean Looking Time (log)") +
xlab("Number of Stimulus Repetitions") +
geom_smooth(method = "lm") +
theme(axis.text = element_text(size = 10))## `geom_smooth()` using formula 'y ~ x'
Discussion
Summary of Replication Attempt
The confirmatory analysis was a 2 (stimulus type) x 3 (block) repeated measure ANOVA. The task by stimulus type interaction was significant (F(4, 5797) = 2.729, p < 0.05), but as the figure showed the direction of the interaction was visibly inconsistent with the original. The replication attempt was not successful.
Commentary
I did not conduct further post-hoc analysis based on the ANOVA results because there are reasons to believe that the looking time data distribution does not satisfy ANOVA’s assumption. As the figure below shows, after transformed to the log-space, the looking time distribution showed a skewed bi-modal distribution across all blocks.
d %>%
ggplot(aes(x = trial_looking_time)) +
geom_histogram() +
scale_x_log10() +
facet_wrap(~block)
It is unclear what factors gave rise to the second mode. One possible explanation is the presence of “long lookers”, i.e. participants whose looking time are above certain criteria. Based on the plot above, I tentatively grouped the participants into “long lookers” and “short lookers”. Participants who were long lookers have average looking time above 800 ms, the rest are short lookers. The plot below provides some evidence that the long lookers were mainly responsible for the second mode.
lower_mean_sbj <- d %>%
group_by(subject) %>%
summarise(
mean_LT = mean(trial_looking_time)
) %>%
mutate(
group = case_when(
mean_LT < 800 ~ "lower_mean",
TRUE ~ "higher_mean"
)
) %>%
filter(group == "lower_mean") %>%
select(subject) %>%
pull()
higher_mean_sbj <- d %>%
group_by(subject) %>%
summarise(
mean_LT = mean(trial_looking_time)
) %>%
mutate(
group = case_when(
mean_LT < 800 ~ "lower_mean",
TRUE ~ "higher_mean"
)
) %>%
filter(group == "higher_mean") %>%
select(subject) %>%
pull()
d %>%
mutate(
group = case_when(
subject %in% lower_mean_sbj ~ "lower_mean_sbj",
TRUE ~ "higher_mean_sbj"
)
) %>%
ggplot(aes(x = trial_looking_time)) +
geom_histogram()+
scale_x_log10()+
facet_wrap(~group)
However, I was not able to pin down what common characteristics the long lookers shared. I looked at their demographic information and none of them seems to be relevant. The takeaway message here is that in the future more stringent exclusion criteria might be needed.
The failure to replicate the main findings might due to the implementational differences. Two aspects are particularly relevant.
First, the differences in the experiment setting. The original study was a in-lab EEG study where the participants were required to wear EEG cap and were constantly monitored by the experimenters. It is likely that they were more focused and motivated in the task than the participants in the current online version of the task. The difference in the overall looking time provides some evidence: the scale of the y-axis on my plot was significantly smaller than the scale of the y-axis on the original plot.
Second, the differences in the stimuli used. Because I was not able to obtain the original stimuli set from the authors, I used the novel object complexity norm developed by Lewis & Frank (2016). In the current study, “complex stimuli” were the 18 novel objects rated as most complex by the participants in the Lewis & Frank (2016) study, and the “simple stimuli” were the 18 ones rated as being least complex. However, because the span of the complexity norm was 60 novel objects, it is possible that the amount of complexity differences across two groups were not sufficient to elicit the complexity effect. Additionally, all of the objects are novel objects rarely encountered in the real life. The novelty itself might mask the simplicity of the “simple stimuli”, and thus leading to the lack of complexity effect.