Introduction

I am interested in understanding and correcting biases, especially in the context of social groups. This paper by Lloyd, Hugenberg, McConnel, Kunstman, & Deska (2017) examines the truth bias by understanding how group membership influences perceivers decisions on whether an individual is seen as deceiving or truthful. I plan to replication Study 1c, which found that White American participants used the truth response more for Black targets than for White targets. The paper also used the Open Science Framework, making it relatively easy to replicate.

Participants will be viewing one of two sets of 40 deception-judgement videos that are provided by the author. Each video will show an individual telling a story, and participants will determine whether the individual is lying or not. They will also be completing Plant and Devine’s (1998) Internal and External Motivation to Respond Without Prejudice measure, Glaser and Knowles’s (2008) Implicit Motivation to Control Prejudice task, and Kunstman et al. (2013) 17-item intergroup contact measure. Because I am not very familiar with amazon mechanical turk, I am expecting to have a some challenges within making sure that the stimuli will be represented correctly, without errors within the survey flow. Because I have not done statistical analysis for a while, I also expect some challenges within the data analysis bit. Fortunately, because the study is on OSF, I hope that it would be relatively easier to learn.

Methods

Power Analysis

According to Lloyd et al.(2017), an a priori power analysis indicated that 67 participants are needed to achieve 80% power for the analysis.

Using their data in their achieved effect size of 1.09. I require 9 participants. I divided the effect size in half and using G*power indicated that I would need 27 participants.

Planned Sample

27 White Americans

Materials

  • MU3D Videos
  • Plant and Devine’s (1998) Internal and External Motivation to Respond Without Prejudice measure
  • Glaser and Knowles’s (2008) Implicit Motivation to Control Prejudice task
  • Kunstman et al. (2013) 17-item intergroup contact measure.

Procedure

Participants were randomly assigned to view one of two sets of 40 deception-judgment videos (Lloyd et al., 2017; for details on the development of the stimuli, see the Supplemental Material). Videos featured Black and White college-age individuals describing acquaintances; each set of videos featured 20 positive descriptions and 20 negative descriptions, and valence was balanced across race. Participants saw each target person twice, once when the target was lying and once when the target was telling the truth. Within each set, videos were presented in random order. After each video, participants provided a truth/lie judgment, and they had an unlimited amount of time to respond. Participants were instructed that they might see the same individual multiple times but that they should base their judgments only on the current video because some individuals might tell only lies, some might tell only truths, and some might tell a combination of truths and lies. Participants completed the deception judgment task in addition to the IMS (M = 7.59, SD = 1.42, a = .86) and EMS (M = 4.77, SD = 1.60, a = .77). The only modification in the current study was that participants completed a 10-item measure assessing interpersonal contact both with Whites and with Blacks. Participants responded to each item on a 10-point scale with anchors of 0 and 9 or more (e.g., “How many Black [White] friends do you have in college?”; mean number of White contacts = 8.06, SD = 2.15, a= .85; mean number of Black contacts = 3.43, SD = 1.88, a = .76; adapted from Kunstman et al., 2013). Number of Black contacts, number of White contacts, and the difference between them (i.e., White contacts minus Black contacts) did not moderate the findings reported (ps > .130).

Analysis Plan

Paired T-test between measurement of truth biases for black targets and white targets

Differences from Original Study

Procedures were followed exactly as researchers reported.

Methods Addendum (Post Data Collection)

Actual Sample

27 White American Participants

Differences from pre-data collection methods plan

none

Results

Data preparation

Data preparation following the analysis plan.

library(tidyverse)
## Warning: package 'tidyverse' was built under R version 3.4.4
## -- Attaching packages ----------------------------------- tidyverse 1.2.1 --
## v ggplot2 3.0.0     v purrr   0.2.5
## v tibble  1.4.2     v dplyr   0.7.6
## v tidyr   0.8.1     v stringr 1.3.1
## v readr   1.1.1     v forcats 0.3.0
## Warning: package 'ggplot2' was built under R version 3.4.4
## Warning: package 'tibble' was built under R version 3.4.4
## Warning: package 'tidyr' was built under R version 3.4.4
## Warning: package 'readr' was built under R version 3.4.4
## Warning: package 'purrr' was built under R version 3.4.4
## Warning: package 'dplyr' was built under R version 3.4.4
## Warning: package 'stringr' was built under R version 3.4.4
## Warning: package 'forcats' was built under R version 3.4.4
## -- Conflicts -------------------------------------- tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()
library(readxl)
## Warning: package 'readxl' was built under R version 3.4.4
library(effsize)
## Warning: package 'effsize' was built under R version 3.4.4
library(dplyr)
library(ggpubr)
## Warning: package 'ggpubr' was built under R version 3.4.4
## Loading required package: magrittr
## Warning: package 'magrittr' was built under R version 3.4.4
## 
## Attaching package: 'magrittr'
## The following object is masked from 'package:purrr':
## 
##     set_names
## The following object is masked from 'package:tidyr':
## 
##     extract
#Used to clean up data and take out unnecessary data for study 1c
lloyd1c_wide <- read.csv("lloyd2017_data_modified.csv") %>%
  mutate(subId = rownames(.)) %>%
  select(starts_with("WM"), starts_with("BM"), subId, AttCheck1, AttnCheck2)
## Warning: package 'bindrcpp' was built under R version 3.4.4
#Used to tidy data for analysis
lloyd1c_long <- lloyd1c_wide %>%
  gather(question, response, -subId, -AttCheck1, -AttnCheck2) %>%
  filter(!is.na(response), AttCheck1 == 5) %>%
  separate(question, into = c("race", "gender", "valence", "lie", "qnumber"), sep=1:4) 

#Data prepped for t-test
lloyd1c_final <- lloyd1c_long %>%
  group_by(subId, race) %>%
  summarise(meanResponse = mean(response)) %>%
  spread(race, meanResponse)

Confirmatory analysis

#T-test on means of truth responses to each race  
t.test(lloyd1c_final$B, lloyd1c_final$W, paired = TRUE)
## 
##  Paired t-test
## 
## data:  lloyd1c_final$B and lloyd1c_final$W
## t = 3.6168, df = 24, p-value = 0.001379
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.09445795 0.34554205
## sample estimates:
## mean of the differences 
##                    0.22
#Preparing data for box plot
lloyd1c_plot <- lloyd1c_final%>%
  gather(race, score, -subId)

#Boxplot showing range of responses from participants. 
#Score plotted are means of participants responses. Scores range from 1 to -1. Higher scores indicate higher tendency to use truth response. 
ggboxplot(lloyd1c_plot, x = "race", y = "score", 
          color = "race", palette = c("#00AFBB", "#E7B800"),
          order = c("B", "W"),
          ylab = "score", xlab = "race")

Discussion

Summary of Replication Attempt

With 25 participants for the final analysis (2 excluded for attention check failure), a paired t-test was ran. The result was significant with p=.001, suggesting that participants were more likely to favor truth responses for black targets rather than white targets. This finding replicates the Lloyd et al. (2016) study 1c result.

Commentary

I believe that this study was accurate replication of Lloyd et al. (2016) study 1c findings. With the help of the original researchers, I was able to acquire materials and procedures necessary for a successful replication. Through the presentation and discussion of this replication, I came across an interesting question on the content of the MU3D video. I think these videos were controlled very well for length, valence, and lie/truth content as described in the supplemental material. However, I did not find information on how target attractiveness was controlled. One possibility is that participants may have been using physical attractiveness as a cue for lie/truth detection. It would be interesting to see if the black or white target videos could also be indexed based on perceived attractiveness as previous research within the paper indicates that attractiveness can influence trustworthiness judgements. Having the MU3D videos being controlled for attractiveness could make this finding more powerful.