DiD Calculations
Lucy Jackson
2026-03-20
Calculating Difference-in-Differences.
Summary statements
Reaction Time: The DiD approach computes the difference-in-differences (e.g., difference in Post-Pre Change between Stereo/CounterS Media vs. Control) from the model-derived estimated marginal means.
Gaze Data: The DiD approach computes the difference-in-differences from the model-predicted trajectories.
This approach was adopted from research in economics/public health policy/user research. Where often they want to look at causal effects of a certain treatment. At the bottom of this html I go into detail about this.
I will start with Rachel’s question: Confusion about where the z-ratio and p-values came from when looking at the resulting comparisons in Post-Pre Media Changes.
The z-ratio and p-values appear due to the contrasts/comparisons that have been carried out within the steps. Like the pairwise comparisons in Phase 1 on the emmeans differences (Accent x Condition combination), Phase 2 also carries out the same contrasts on the emmeans, which automatically produce the significance values.
Below I will demonstrate the steps for Isolating the Media for Reaction Time, and demonstrate where and how these values arise.
Step 1: Generating the Estimated Marginal Means from the model.
- The estimated marginal means of the RTs from the model is calculated across all factor-level combinations of Condition * Pre-Post * Media combinations.
# This calculates the emmeans across all Condition * Pre-Post * Media combinations.
p2_means <- emmeans(P2_RT_OCC, ~ condition | pre_post * media)Click to View p2_means output
## pre_post = pre, media = Control:
## condition emmean SE df asymp.LCL asymp.UCL
## congruent 7.64 0.0597 Inf 7.53 7.76
## incongruent 7.68 0.0597 Inf 7.56 7.79
##
## pre_post = post, media = Control:
## condition emmean SE df asymp.LCL asymp.UCL
## congruent 7.39 0.0598 Inf 7.27 7.50
## incongruent 7.46 0.0597 Inf 7.35 7.58
##
## pre_post = pre, media = Stereo:
## condition emmean SE df asymp.LCL asymp.UCL
## congruent 7.60 0.0596 Inf 7.49 7.72
## incongruent 7.58 0.0597 Inf 7.46 7.69
##
## pre_post = post, media = Stereo:
## condition emmean SE df asymp.LCL asymp.UCL
## congruent 7.33 0.0596 Inf 7.21 7.45
## incongruent 7.43 0.0597 Inf 7.31 7.54
##
## pre_post = pre, media = CounterS:
## condition emmean SE df asymp.LCL asymp.UCL
## congruent 7.52 0.0592 Inf 7.40 7.64
## incongruent 7.68 0.0594 Inf 7.57 7.80
##
## pre_post = post, media = CounterS:
## condition emmean SE df asymp.LCL asymp.UCL
## congruent 7.37 0.0593 Inf 7.25 7.48
## incongruent 7.41 0.0591 Inf 7.29 7.52
##
## Degrees-of-freedom method: asymptotic
## Confidence level used: 0.95Step 2: Calculate the within-group Media Post-Pre Change in emmeans.
Taking the emmeans (p2_means) and subtracting the emmeans value for each Pre-Media:Condition group, from their respective Post-Media Condition group.
This step takes the estimated marginal means for the RTs (p2_means), and subtracts the corresponding Pre-value for each Media:Condition, from its Post-Media:Condition value:
\[\text{Post-Pre Change} = \text{Post-test Mean} - \text{Pre-test Mean} \]
It is this contrast of the estimated marginal means, that automatically compute the z-ratio and p-values to determine its significant variation.
For example, the estimated marginal means for the Pre-Control-Congruent (7.64) is subtracted from the estimated marginal means for the Post-Control-Congruent (7.39). This negative Post-Pre change (-0.25)
\[ \text{Media Effect}_{\text{Control, Congruent}} = 7.39 - 7.64 = -0.25 \]
The below contrast computes this and determines the significance of the Post-Pre contrast, with the resulting z-ratio and p-values.
# Taking the emmeans (p2_means) and subtracting the emmeans value for each Pre-Media:Condition group, from their
# respective Post-Media Condition group.
change_over_time <- contrast(
p2_means,
method = list("Post - Pre" = c(-1, 1)),
by = c("media", "condition")
)- Note that no adjustment is applied in the contrasts as it is a within-group comparison.
Click to View Post-Pre (change_over_time) output
## media = Control, condition = congruent:
## contrast estimate SE df z.ratio p.value
## Post - Pre -0.256 0.0316 Inf -8.099 <.0001
##
## media = Stereo, condition = congruent:
## contrast estimate SE df z.ratio p.value
## Post - Pre -0.274 0.0309 Inf -8.866 <.0001
##
## media = CounterS, condition = congruent:
## contrast estimate SE df z.ratio p.value
## Post - Pre -0.152 0.0322 Inf -4.733 <.0001
##
## media = Control, condition = incongruent:
## contrast estimate SE df z.ratio p.value
## Post - Pre -0.211 0.0312 Inf -6.771 <.0001
##
## media = Stereo, condition = incongruent:
## contrast estimate SE df z.ratio p.value
## Post - Pre -0.151 0.0314 Inf -4.807 <.0001
##
## media = CounterS, condition = incongruent:
## contrast estimate SE df z.ratio p.value
## Post - Pre -0.275 0.0322 Inf -8.561 <.0001
##
## Degrees-of-freedom method: asymptoticStep 3: Isolating the Media Effect compared to the Control Baseline
This step takes the computed Post-Pre differences, and now computes the final step of the difference-in-differences approach.
The Post-Pre Change for the Control group is subtracted from the Post-Pre Change observed for both the Stereotypical and Counter-stereotypical media.
\[\text{Media Effect (DiD)} = \text{Post-Pre}_{\text{Media}} - \text{Post-Pre}_{\text{Control}} \] This difference-in-differences (DiD) approach isolates the media effect by controlling for the neutral Control baseline. By considering the change from Pre- to Post-exposure, rather than simply examining Post-condition values, this method accounts for between-subject variability in baseline performance across the three media groups.
In other words, it adjusts for the different starting points of participants prior to media exposure. Consequently, this approach provides a precise estimate of the magnitude of the change following media exposure, by assessing both the Pre- and Post-exposure measurements.
media_vs_control_change <- contrast(
change_over_time,
method = list(
"Stereo - Control" = c(-1, 1, 0),
"CounterS - Control" = c(-1, 0, 1)
),
by = "condition",
adjust = "bonferroni" # <--- I added this here
)Click to View DiD Media vs. Control (media_vs_control_change) output
## condition = congruent:
## contrast estimate SE df z.ratio p.value
## Stereo - Control -0.0180 0.00852 Inf -2.106 0.0703
## CounterS - Control 0.1035 0.00852 Inf 12.152 <.0001
##
## condition = incongruent:
## contrast estimate SE df z.ratio p.value
## Stereo - Control 0.0600 0.00850 Inf 7.055 <.0001
## CounterS - Control -0.0642 0.00856 Inf -7.504 <.0001
##
## Degrees-of-freedom method: asymptotic
## P value adjustment: bonferroni method for 2 testsBonferonni adjustements were applied here for the multiple, complex interactions.
Rationale: Using a Tukey adjustment (like in Phase 1 where I wanted to view all combinations of Accent x Condition) would not be appropriate here. As I am just interested in comparison of Stereo/CounterS Media groups to the Control, using a Tukey approach would penalise the p-value for every possible comparison I could make, and not just my Control vs. focus.
This same approach is applied to the gaze data, on the model-predicted trajectories. Where rather than the emmean values being the outcome variable, it is elogBiasAdvantage. I have inputted my code for gaze traces at the very end for you to view (and will be in full detail in methods with a working case example), but now will focus on how/why this approach what decided.
My application of difference-in-difference approaches was guided by conceptual frameworks found in public health/economic and user research (priming) studies (e.g., A/B testing of a certain product). I first found this in Wing et al (2018) who demonstrated best practice in isolating a causal effect in public health (e.g., effect of policy on vaccine uptake/reduction in drug abuse). Here they refer to a ‘treatment effect’ as the difference in outcome for a participant between two conditions (the treatment vs the control condition).
I found this further extended to economic research, with the works of Card & Krueger’s study as a common example in isolating effects. Here, they wanted to look at the causal effects of a minimum wage increase on employment in fast-food sectors.
For each store, they measured the full-time-equivalent (FTE) employment before and after the wage increase both in the states of New Jersey and Pennsylvania. This Post-Pre Wage Change (analogous to my Post-Pre Media Change) classed as the ’simple difference/simple change’.
Simple changes for each state:
\[ \text{Change}_{NJ} = \text{Employment}_{Post} - \text{Employment}_{Pre} \]
\[ \text{Change}_{PA} = \text{Employment}_{Post} - \text{Employment}_{Pre} \]
New Jersey was treated as the ‘treatment’ state as they experienced the policy change in minimum wage increase. However, Pennsylvania served as the ‘Control’ state as they did not experience the policy change. So, to isolate the causal effect of minimum wage increase, they then deducted the Post-Pre Change of the Control, from the Treatment state. This is now the step classed as the ‘difference-in-differences’:
\[ \text{DiD: Effect of Minimum Wage Policy} = \text{Change}_{NJ} - \text{Change}_{PA} \]
I adapted this method directly to my research, with the ‘Control’ (Neutral wildlife) media, and the Treatment (Stereotypical/Counter-stereotypical) media. Where my simple difference is Post-Pre across the Medias, and the difference-in-differences is the Control deducted from the Treatment groups
One point to make is that unlike Card & Krueger, who compute raw differences first and then model with covariates, I first fit a mixed-effects model accounting for random effects of Subject and Item, as well as fixed effects for Condition, Pre/Post, and Media. Then, the estimated marginal means derived from the model already incorporate these adjustments. I would argue that although I conduct difference-in-differences on the modelled output, rather than modelling on the differences, my approach is still fine.
Computing the Post–Pre contrasts and the DiD estimate from these model-derived EMMs ensures that the differences are already adjusted for participant-level variability and other covariates, arguably providing a cleaner and more precise estimate of the media effect. We see this logic of observing causal effects in User Research too (basically anything where we are eliciting a treatment/exposure effect), where one aims to isolate the treatment effect by subtracting the baseline effect.
A ‘meaty’ overview of difference-in-difference estimates can be found here, which I will discuss more in my stats section.
Gaze data (DiD Approach)
I have just put this in here for you to see, but follows same logic.
Difference-in-differences approach: Post-pre change in model-predicted trajectories of elogit Bias Advantage calculated for each Media Condition.
From calculating the difference-in-differences, and the isolated media ‘treatment’ effect, compared to the Control. I am able to ascertain pointwise significance in where the treatment media (Stereo/CounterS) significantly differs from the Control baseline now.
Click to View Phase 2: Isolated Media Effect
#######################################
# GAZE DATA – Difference-in-Differences
########################################
# --- Set up conditions ---
conditions <- c("congruent", "incongruent")
pre_posts <- c("pre", "post")
medias <- c("Control", "Stereo", "CounterS")
# Model predicted trajectories
new_data_p2 <- expand.grid(
Time = seq(-400, 600, length.out = 400),
condition = conditions,
pre_post = pre_posts,
media = medias,
Subject = "51_a_nc",
Item = "19",
TRIAL_INDEX = 55
) %>%
mutate(
condition_prepost_media = paste(condition, pre_post, media, sep = "."),
condition_prepost = paste(condition, pre_post, sep = ".")
)
predicted_data_p2 <- predict(
P2.nd,
newdata = new_data_p2,
type = "link",
se.fit = TRUE,
exclude = c("s(Time,Item)", "s(Time,Subject)")
)
plot_data_p2 <- new_data_p2 %>%
mutate(
fit = predicted_data_p2$fit,
se = predicted_data_p2$se.fit
)
# Compute Post–Pre Change across Media x Condition
delta_cond_media <- plot_data_p2 %>%
select(Time, media, condition, pre_post, fit, se) %>%
pivot_wider(
names_from = pre_post,
values_from = c(fit, se)
) %>%
mutate(
delta = fit_post - fit_pre,
delta_se = sqrt(se_post^2 + se_pre^2)
)
# Difference-in-Differences (Media – Control) -- Pre-Post Change Baseline Subtraction
did_cond_media <- delta_cond_media %>%
select(Time, media, condition, delta, delta_se) %>%
pivot_wider(
names_from = media,
values_from = c(delta, delta_se)
) %>%
mutate(
did_Stereo = delta_Stereo - delta_Control,
did_se_Stereo = sqrt(delta_se_Stereo^2 + delta_se_Control^2),
lower_Stereo = did_Stereo - 1.96 * did_se_Stereo,
upper_Stereo = did_Stereo + 1.96 * did_se_Stereo,
did_CounterS = delta_CounterS - delta_Control,
did_se_CounterS = sqrt(delta_se_CounterS^2 + delta_se_Control^2),
lower_CounterS = did_CounterS - 1.96 * did_se_CounterS,
upper_CounterS = did_CounterS + 1.96 * did_se_CounterS
) %>%
# Pivot back to long for plotting
pivot_longer(
cols = starts_with("did_") | starts_with("lower_") | starts_with("upper_"),
names_to = c(".value", "media"),
names_pattern = "(.*)_(Stereo|CounterS)"
) %>%
mutate(
is_sig = (lower > 0 | upper < 0)
)
# Pointwise Significance
did_cond_media <- did_cond_media %>%
mutate(
is_sig = (lower > 0 | upper < 0),
sig_y = -7.5
)
########################################
# Visualisation of Isolated Media Effect
########################################
# Forcing Stereotypical First
did_cond_media <- did_cond_media %>%
mutate(media = factor(media, levels = c("Stereo", "CounterS")))
# Define the labels for the facets
media_labels <- c(
Stereo = "Stereotypical Media",
CounterS = "Counter-Stereotypical Media"
)
ggplot(did_cond_media,
aes(x = Time, y = did, colour = condition, fill = condition)) +
geom_tile(data = subset(did_cond_media, is_sig),aes(y = sig_y, height = 0.6),
alpha = 0.9, colour = NA ) +
geom_ribbon( aes(ymin = lower, ymax = upper), alpha = 0.15, colour = NA ) +
geom_line(aes(linetype = condition), linewidth = 1.2) +
geom_hline( yintercept = 0, linetype = "solid", linewidth = 0.8,
colour = "black") +
#annotate( "text", x = 580, y = 0.5, label = "Control Baseline", size = 3.5, fontface = "italic", hjust = 1) +
geom_vline( xintercept = 0, linetype = "dotted", colour = "grey40" ) +
facet_wrap(~media, labeller = as_labeller(media_labels)) +
coord_cartesian(ylim = c(-8, 8)) +
scale_y_continuous(breaks = seq(-8, 8, by = 2)) +
scale_x_continuous(
limits = c(-400, 600),
breaks = seq(-400, 600, 200) ) +
scale_colour_manual( name = "Trial Type", values = c(congruent = "darkred", incongruent = "#0072B2") ) +
scale_fill_manual( name = "Trial Type", values = c(congruent = "darkred", incongruent = "#0072B2") ) +
scale_linetype_manual( values = c(congruent = "solid", incongruent = "longdash")) +
labs(
y = "Post-Pre Change in Bias Advantage (elogit)",
x = "Time (ms) relative to Target Onset",
title = "Occupation: Isolated Media Effect",
subtitle = "Difference-in-Differences relative to Control baseline",
caption = str_wrap("Coloured horizontal indicators denote time windows of significant pointwise deviation from the Control baseline (p < .05). Positive values indicate an increase in Bias Advantage relative to Control.", width = 95)) +
theme_bw(base_size = 18) +
theme(
plot.title = element_text(face = "bold", hjust = 0.5, size = 22),
plot.subtitle = element_text(hjust = 0.5, size = 16),
plot.caption = element_text(size = 16, hjust = 0, margin = margin(t = 20), lineheight = 1.1),
legend.position = "bottom",
legend.text = element_text(size = 16),
legend.title = element_text(size = 18, face = "bold"),
strip.background = element_rect(fill = "grey95"),
strip.text = element_text(face = "bold", size = 18),
axis.title = element_text(size = 18),
axis.text = element_text(size = 14)
) +
guides(
colour = guide_legend(override.aes = list(alpha = 1, linewidth = 2.5, fill = NA)),
fill = "none",
linetype = "none"
)
DiD over difference smooths for Gaze data
As discussed, this option is much better than difference smooth approaches. Difference smoths can only show comparisons across two levels (e.g., Accent vs Condition). However, because we have Condition x Pre-Post x Media, difference smooths are limited in that they cannot display the extent of Media Change.
Here we can compute the Control vs Treatment Media effect (congruent trials), in both the Pre and Post phases. However, we are not accounting for the different starting points of the media, and not actually quantifying the extent of the media change from Pre to Post. So arguably, it’s very limited in actually accounting for media change.
Plus, you need a heck of a lot of difference smooths to even get to this point. My approach of the DiD smooths already captures the 1) baseline differences in Bias Advantage (pre-phase) across the media groups and then 2) directly quantifies the difference before and after the media (Post-Pre Change), then finally 3) isolates the causal effect of Stereo/CounterS by subtracting the Control neutral media Here, we then are able to use the plotted DiDs I have created to see any pointwise significance. That is, significantly different Post-Pre changes in Bias Advantage compared to the Control baseline.
