Background
The purpose of this study is to explore potential effects of AI and
the medium in which it is being engage with on trust and ensuing
adoption behaviors.
Procedure
In a 2 (AI vs. human) X 2 (chat vs. no chat) design, participants
were asked to imagine a scenario in which they are a hiring manager who
needs to read through ~100 applications for an open role. They then
watch one of four screen recordings that simulate a request from the
hiring manager to HR or an AI tool via a chat or a form, depending on
condition. After the interaction, they are told that HR/AI tool got back
to them with a recommended list of five applicants.
Then, participants are asked about their trust in the recommendation, as
well as the likelihood that they will engage in a number of adoption
behaviors, in random order.
Finally, they answer a demographic questionnaire.
Analysis plan
This study is extremely under-powered. The purpose of it is just to
get a sense of some directional effects. Therefore, we will mostly look
at the means and effect sizes, rather than p-values.
That said, we will conduct a 2X2 between-subjects ANOVA, testing the
effects of condition on each of the five adoption behaviors.
Additionally, we will conduct mediation analyses with each of the
mechanism measures as mediators, and the strongest DV’s from the ANOVA’s
as outcome variables.
Demographics
Race
|
race
|
N
|
Perc
|
|
American Indian or Alaska Native
|
2
|
1.00
|
|
Asian
|
19
|
9.45
|
|
Black or African American
|
17
|
8.46
|
|
Hispanic, Latino, or Spanish origin
|
6
|
2.99
|
|
Other (please specify)
|
1
|
0.50
|
|
White
|
138
|
68.66
|
|
multiracial
|
17
|
8.46
|
|
NA
|
1
|
0.50
|
Gender
|
gender
|
N
|
Perc
|
|
man
|
122
|
60.70
|
|
other
|
3
|
1.49
|
|
woman
|
76
|
37.81
|
Age
|
age_mean
|
age_sd
|
|
38.14
|
9.7
|
Education
|
edu
|
N
|
Perc
|
|
noHS
|
2
|
1.00
|
|
GED
|
33
|
16.42
|
|
2yearColl
|
25
|
12.44
|
|
4yearColl
|
103
|
51.24
|
|
MA
|
27
|
13.43
|
|
PHD
|
10
|
4.98
|
|
NA
|
1
|
0.50
|
Income
Employment
|
employment
|
N
|
Perc
|
|
Full-time
|
169
|
84.08
|
|
Full-time, Homemaker
|
1
|
0.50
|
|
Other
|
2
|
1.00
|
|
Part-time
|
25
|
12.44
|
|
Part-time, Homemaker
|
1
|
0.50
|
|
Part-time, Other
|
1
|
0.50
|
|
Permanently disabled
|
1
|
0.50
|
|
Unemployed
|
1
|
0.50
|
Work Experience
|
workex
|
N
|
Perc
|
|
Less than 5 years
|
19
|
9.45
|
|
5 - 10 years
|
46
|
22.89
|
|
10 - 20 years
|
67
|
33.33
|
|
More than 20 years
|
69
|
34.33
|
Recruiting Experience
|
recruitex
|
N
|
Perc
|
|
No experience or very limited
|
82
|
40.80
|
|
Some experience
|
101
|
50.25
|
|
Significant experience
|
18
|
8.96
|
Measures
Adoption
Please indicate the likelihood that you do each of the following (1 =
Extremely Unlikely to 5 = Extremely Likely):
1. Accept the recommendation
2. Skim these five applications before proceeding
3. Ignore the recommendation
4. Select, at random, five other applications and read all ten to
compare
5. Skim through all applications but read the five recommended
applications more closely
Mechanisms
trust_qual: To what extent do you trust the [HR team
/ AI tool] to come up with the best candidates for this job?
trust_bias: To what extent do you trust the [HR team /
AI tool] to judge the applications without bias?
trust_person: To what extent do you trust that the [HR
team / AI tool]’s recommendations are personalized to your needs?
Analysis
DV1: Accept the recommendation
Descriptives
|
ai
|
chat
|
adopt_1_M
|
adopt_1_SD
|
|
0
|
0
|
3.77
|
0.89
|
|
0
|
1
|
4.17
|
0.56
|
|
1
|
0
|
3.39
|
1.20
|
|
1
|
1
|
3.45
|
0.98
|
Two-way ANOVA
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
|
ai
|
1
|
197
|
17.193
|
5.01e-05
|
|
0.080
|
|
chat
|
1
|
197
|
3.016
|
8.40e-02
|
|
0.015
|
|
ai:chat
|
1
|
197
|
1.585
|
2.09e-01
|
|
0.008
|
Bonferroni-corrected post-hoc comparisons: Chat
|
ai
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
chat
|
1
|
98
|
6.867
|
0.010
|
|
0.065000
|
0.02
|
|
1
|
chat
|
1
|
99
|
0.085
|
0.771
|
|
0.000859
|
1.00
|
Bonferroni-corrected post-hoc comparisons: AI
|
chat
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
ai
|
1
|
97
|
3.286
|
7.30e-02
|
|
0.033
|
1.46e-01
|
|
1
|
ai
|
1
|
100
|
19.586
|
2.46e-05
|
|
0.164
|
4.92e-05
|
Plot
DV2: Skim these five applications before proceeding
Descriptives
|
ai
|
chat
|
adopt_2_M
|
adopt_2_SD
|
|
0
|
0
|
3.91
|
1.23
|
|
0
|
1
|
3.91
|
1.30
|
|
1
|
0
|
3.61
|
1.29
|
|
1
|
1
|
3.78
|
1.18
|
Two-way ANOVA
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
|
ai
|
1
|
197
|
1.484
|
0.225
|
|
0.007
|
|
chat
|
1
|
197
|
0.267
|
0.606
|
|
0.001
|
|
ai:chat
|
1
|
197
|
0.216
|
0.643
|
|
0.001
|
Bonferroni-corrected post-hoc comparisons: Chat
|
ai
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
chat
|
1
|
98
|
0.001
|
0.971
|
|
1.36e-05
|
1.000
|
|
1
|
chat
|
1
|
99
|
0.494
|
0.484
|
|
5.00e-03
|
0.968
|
Bonferroni-corrected post-hoc comparisons: AI
|
chat
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
ai
|
1
|
97
|
1.373
|
0.244
|
|
0.014
|
0.488
|
|
1
|
ai
|
1
|
100
|
0.293
|
0.589
|
|
0.003
|
1.000
|
Plot
DV3: Ignore the recommendation
Descriptives
|
ai
|
chat
|
adopt_3_M
|
adopt_3_SD
|
|
0
|
0
|
1.83
|
0.87
|
|
0
|
1
|
1.68
|
0.91
|
|
1
|
0
|
2.24
|
1.08
|
|
1
|
1
|
2.29
|
1.10
|
Two-way ANOVA
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
|
ai
|
1
|
197
|
13.075
|
0.00038
|
|
0.062000
|
|
chat
|
1
|
197
|
0.120
|
0.73000
|
|
0.000608
|
|
ai:chat
|
1
|
197
|
0.509
|
0.47600
|
|
0.003000
|
Bonferroni-corrected post-hoc comparisons: Chat
|
ai
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
chat
|
1
|
98
|
0.701
|
0.405
|
|
0.00700
|
0.81
|
|
1
|
chat
|
1
|
99
|
0.056
|
0.813
|
|
0.00057
|
1.00
|
Bonferroni-corrected post-hoc comparisons: AI
|
chat
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
ai
|
1
|
97
|
4.350
|
0.040
|
|
0.043
|
0.080
|
|
1
|
ai
|
1
|
100
|
9.108
|
0.003
|
|
0.083
|
0.006
|
Plot
DV4: Select, at random, five other applications and read all ten to
compare
Descriptives
|
ai
|
chat
|
adopt_4_M
|
adopt_4_SD
|
|
0
|
0
|
2.60
|
1.32
|
|
0
|
1
|
2.21
|
1.20
|
|
1
|
0
|
3.04
|
1.37
|
|
1
|
1
|
2.91
|
1.39
|
Two-way ANOVA
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
|
ai
|
1
|
197
|
9.204
|
0.003
|
|
0.045
|
|
chat
|
1
|
197
|
1.969
|
0.162
|
|
0.010
|
|
ai:chat
|
1
|
197
|
0.470
|
0.494
|
|
0.002
|
Bonferroni-corrected post-hoc comparisons: Chat
|
ai
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
chat
|
1
|
98
|
2.384
|
0.126
|
|
0.024
|
0.252
|
|
1
|
chat
|
1
|
99
|
0.238
|
0.627
|
|
0.002
|
1.000
|
Bonferroni-corrected post-hoc comparisons: AI
|
chat
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
ai
|
1
|
97
|
2.645
|
0.107
|
|
0.027
|
0.214
|
|
1
|
ai
|
1
|
100
|
7.211
|
0.008
|
|
0.067
|
0.016
|
Plot
DV5: Skim through all applications but read the five recommended
applications more closely
Descriptives
|
ai
|
chat
|
adopt_5_M
|
adopt_5_SD
|
|
0
|
0
|
3.79
|
1.12
|
|
0
|
1
|
3.96
|
1.02
|
|
1
|
0
|
3.70
|
1.15
|
|
1
|
1
|
3.98
|
1.13
|
Two-way ANOVA
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
|
ai
|
1
|
197
|
0.053
|
0.817
|
|
0.000271
|
|
chat
|
1
|
197
|
2.075
|
0.151
|
|
0.010000
|
|
ai:chat
|
1
|
197
|
0.150
|
0.699
|
|
0.000759
|
Bonferroni-corrected post-hoc comparisons: Chat
|
ai
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
chat
|
1
|
98
|
0.590
|
0.444
|
|
0.006
|
0.888
|
|
1
|
chat
|
1
|
99
|
1.578
|
0.212
|
|
0.016
|
0.424
|
Bonferroni-corrected post-hoc comparisons: AI
|
chat
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
ai
|
1
|
97
|
0.180
|
0.672
|
|
0.002000
|
1
|
|
1
|
ai
|
1
|
100
|
0.013
|
0.910
|
|
0.000129
|
1
|
Plot
MECH1: trust_qual
To what extent do you trust the [HR team / AI tool] to come up with
the best candidates for this job?
Descriptives
|
ai
|
chat
|
trust_qual_M
|
trust_qual_SD
|
|
0
|
0
|
3.81
|
0.88
|
|
0
|
1
|
4.19
|
0.65
|
|
1
|
0
|
3.17
|
1.20
|
|
1
|
1
|
3.38
|
1.06
|
Two-way ANOVA
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
|
ai
|
1
|
197
|
27.854
|
3.00e-07
|
|
0.124
|
|
chat
|
1
|
197
|
4.600
|
3.30e-02
|
|
0.023
|
|
ai:chat
|
1
|
197
|
0.395
|
5.31e-01
|
|
0.002
|
Bonferroni-corrected post-hoc comparisons: Chat
|
ai
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
chat
|
1
|
98
|
5.941
|
0.017
|
|
0.057
|
0.034
|
|
1
|
chat
|
1
|
99
|
0.854
|
0.358
|
|
0.009
|
0.716
|
Bonferroni-corrected post-hoc comparisons: AI
|
chat
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
ai
|
1
|
97
|
9.268
|
3.00e-03
|
|
0.087
|
6.00e-03
|
|
1
|
ai
|
1
|
100
|
20.701
|
1.51e-05
|
|
0.172
|
3.02e-05
|
Plot
MECH2: trust_bias
To what extent do you trust the [HR team / AI tool] to judge the
applications without bias?
Descriptives
|
ai
|
chat
|
trust_bias_M
|
trust_bias_SD
|
|
0
|
0
|
3.70
|
1.03
|
|
0
|
1
|
4.21
|
0.72
|
|
1
|
0
|
3.65
|
1.08
|
|
1
|
1
|
3.64
|
1.19
|
Two-way ANOVA
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
|
ai
|
1
|
197
|
4.577
|
0.034
|
|
0.023
|
|
chat
|
1
|
197
|
2.941
|
0.088
|
|
0.015
|
|
ai:chat
|
1
|
197
|
3.325
|
0.070
|
|
0.017
|
Bonferroni-corrected post-hoc comparisons: Chat
|
ai
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
chat
|
1
|
98
|
8.180
|
0.005
|
|
7.70e-02
|
0.01
|
|
1
|
chat
|
1
|
99
|
0.005
|
0.945
|
|
4.85e-05
|
1.00
|
Bonferroni-corrected post-hoc comparisons: AI
|
chat
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
ai
|
1
|
97
|
0.047
|
0.829
|
|
0.000483
|
1.00
|
|
1
|
ai
|
1
|
100
|
8.370
|
0.005
|
|
0.077000
|
0.01
|
Plot
MECH3: trust_person
To what extent do you trust that the [HR team / AI tool]’s
recommendations are personalized to your needs?
Descriptives
|
ai
|
chat
|
trust_person_M
|
trust_person_SD
|
|
0
|
0
|
3.70
|
0.93
|
|
0
|
1
|
4.23
|
0.73
|
|
1
|
0
|
3.39
|
1.08
|
|
1
|
1
|
3.44
|
1.12
|
Two-way ANOVA
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
|
ai
|
1
|
197
|
15.791
|
9.92e-05
|
|
0.074
|
|
chat
|
1
|
197
|
4.369
|
3.80e-02
|
|
0.022
|
|
ai:chat
|
1
|
197
|
3.119
|
7.90e-02
|
|
0.016
|
Bonferroni-corrected post-hoc comparisons: Chat
|
ai
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
chat
|
1
|
98
|
10.075
|
0.002
|
|
0.093000
|
0.004
|
|
1
|
chat
|
1
|
99
|
0.042
|
0.838
|
|
0.000422
|
1.000
|
Bonferroni-corrected post-hoc comparisons: AI
|
chat
|
Effect
|
DFn
|
DFd
|
F
|
p
|
p<.05
|
ges
|
p.adj
|
|
0
|
ai
|
1
|
97
|
2.292
|
1.33e-01
|
|
0.023
|
0.2660000
|
|
1
|
ai
|
1
|
100
|
17.537
|
6.08e-05
|
|
0.149
|
0.0001216
|
Plot
