My Goals This Week

  1. Finalise and record our slides and group presentation!
  2. Attend Q&A and start thinking about exploratory analysis for Haigh’s paper.
  3. For this week’s learning log, attempt at least one potential exploratory analysis.

How I Made Progress

Exploratory analysis Q1: Is there a relationship between participants’ recall of headlines and their feelings of mistrust in science?

The headline recall test (for both experiments) were used as an attention check. However, it is also possible that participants’ implicit biases may play a role in affecting their attention to scientific headlines - and thus affecting their recall score. Those who have more trust in science may unintentionally pay more attention to science-related headlines. This is consistent with a study by Enzenbach et al. (2019) which found participants’ interest in research topic can result in selection bias. Therefore, I am interested to see whether there is a relationship between participants’ recall and their feelings of mistrust in science.

I am loading packages needed. I’m also setting the theme for all future plots using theme_set() and the argument theme_bw to specify the theme.

library(tidyverse) 
library(ggeasy) # needed to format plots
library(gt) # needed for tables
library(rstatix) # needed for calculating effect size

theme_set(theme_bw()) # sets theme at the start for all plots' theme later on

Step 1: Descriptive Statistics

To start, I am going to count the number for each recall_score variable using count(). Since the study excluded participants that scored 3 and below, data displayed is from 4-7. From this, we can see that the majority of participants scored 7, and only 7 participants scored a recall score of 4.

exptwofinaldata %>% 
  count(recall_score)
## # A tibble: 4 x 2
##   recall_score     n
##   <chr>        <int>
## 1 4                7
## 2 5               14
## 3 6               61
## 4 7              318

For the purpose of this analysis, I am going to exclude participants who scored 6 from subsequent analyses to allow maximal differentiation between low vs. high recall groups. Next, I am going to mutate() a new variable called recallers within the data. Using the function case_when() from dplyr package within mutate() allows me to specify recall_score 4 - 5 as “Low” and recall_score 7 as “High”. I also use na.omit() which allows me to remove NAs (since those that have recall_score 6 will result in NA under the new variable recallers).

exptwofinaldata <- exptwofinaldata %>% 
  mutate(
    recallers = case_when(
      recall_score == 4 ~ "Low",
      recall_score == 5 ~ "Low",
      recall_score == 7 ~ "High"
    )
  ) %>% 
  na.omit() #removes NA from tibble

exptwofinaldata %>% 
  count(recallers)
## # A tibble: 2 x 2
##   recallers     n
##   <chr>     <int>
## 1 High        318
## 2 Low          21

I can then find the means, SE, and SD for mistrust grouped by recallers, using summarise() from dplyr package. Using gt() from the gt package we can display the data in a table, and fmt_number() further allows us to format the numeric values within the table in 2 decimal places for mean, SD, SE.

byrecall <- exptwofinaldata %>% 
  filter(
    recallers == "Low" | recallers == "High"
    ) %>% 
  group_by(recallers) %>% 
  summarise(mean = mean(mistrust),
            sd = sd(mistrust),
            n = n(),
            se = sd/sqrt(n))  

byrecall %>% 
  gt() %>% 
  fmt_number(
    columns = vars(mean, sd, se),
    decimals = 2
  ) %>% 
  tab_header (
    title = "Perceived mistrust in high vs. low recall groups"
  )
Perceived mistrust in high vs. low recall groups
recallers mean sd n se
High 1.92 0.64 318 0.04
Low 2.37 0.63 21 0.14

Looks like low recallers have on average higher mistrust of expertise compared to high recallers. This seems to suggest that participants who score higher on recall (perhaps as a result of paying more attention towards headlines) also have on average more trust in science and expert opinion.

Step 2: Data visualisation

Using ggplot, geom_jitter() and geom_boxplot, we can visualise the data nicely. A few aesthetic changes previously explained in the verification report are also used here.

recallplot <- ggplot(
  data = exptwofinaldata,
  aes(
    x = conflict,
    y = mistrust,
    fill = recallers
  ) 
) +
  geom_jitter() +
  geom_boxplot() +
  ggtitle(
    label = "Mistrust of Expertise in Low vs. High Recall"
  ) +
  facet_wrap(
    vars(format),
    strip.position = "bottom" #moves strip to bottom
  ) +
  easy_center_title() +
  easy_remove_x_axis(what = c("title")) +
  easy_add_legend_title("Recall Score") +
  scale_y_continuous(
    name = "Mistrust of expertise"
  )

plot(recallplot)

Visually inspecting these plots, it looks like those who score low on recall also have on average higher mistrust, when compared to high recallers.

We can run a t-test to find out if there is a statistically significant difference. Since high/low recallers are a categorical variable and mistrust of expertise is a continuous variable - I am going to use a t-test.

Step 3: Statistics using t-test

lowrecall <- exptwofinaldata %>% 
  filter (
    recallers == "Low"
    )

highrecall <- exptwofinaldata %>% 
  filter (
    recallers == "High"
  ) 

t.test(lowrecall$mistrust, highrecall$mistrust)
## 
##  Welch Two Sample t-test
## 
## data:  lowrecall$mistrust and highrecall$mistrust
## t = 3.1594, df = 22.797, p-value = 0.004415
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.15521 0.74476
## sample estimates:
## mean of x mean of y 
##  2.365079  1.915094

p = 0.004415. Based on a=0.05, we reject H0 and conclude that there is a statistically significant difference between low vs. high recallers’ perceived mistrust.

Step 4: Calculate effect size

I am going to use cohens_d() from the rstatix package to calculate effect size above.

exptwofinaldata %>% 
  cohens_d(mistrust ~ recallers, var.equal = TRUE)
## # A tibble: 1 x 7
##   .y.      group1 group2 effsize    n1    n2 magnitude
## * <chr>    <chr>  <chr>    <dbl> <int> <int> <ord>    
## 1 mistrust High   Low     -0.704   318    21 moderate

This effect size of is moderate (d > 0.5), indicating that the means between high vs. low recall groups differ by approximately 0.7 SDs.

Therefore, we can say that participants who have high headline recall scores show on average greater trust in sceince and expert opinion. Conversly, participants who have low headline recall scores show on average greater mistrust and skepticism. This effect is at least moderate in size.

Challenges

  • I am unsure whether this is the best way to convey the data above - which is really pushing me out of my comfort zone. Contrasting this back to verification report where we didn’t even have to consider this, and just focused on replicating whatever the authors had created!

  • I really struggled with exploratory analysis, specifically which questions to ask that would be interesting but also reveal a potentially significantly different effect. I had to play around with the data a lot to find this question (which also reminded me of the dangers of lack of transparency when it comes to exploratory analysis)! Afterall, we had learnt in Year 2’s Stats - run enough analyses and you are bound to find something statistically significant to report!

Wins

  • I am pretty proud that I was able to use gt(), t.test() and cohens_d which I have never used prior to this. The last one required a bit of Googling and even reading up about effect sizes to refresh my memory - however thankfully I had attended the Q&A this week which really helped with everything else.

Next Steps

  • I am still not 100% confident with rubber ducking as I go (am I including enough information? is this redundant? is something important that I am unintentionally excluding?) To combat this, I am going to read up on Jenny’s Roaches example to wrap my head around how to rubber duck and comment as I go!