Descriptive statistics

Observations:

  • Much higher attrition rate in the “within” condition
  • Gender imbalance
Demographics (all, including unfinished surveys)
between
within
N Percent N Percent N Percent
gender f 593 24.04 419 16.98 174 7.05
m 1780 72.15 1170 47.43 610 24.73
d 11 0.45 7 0.28 4 0.16
no 0 0.00 0 0.00 0 0.00
exam_year no exam 655 26.55 446 18.08 209 8.47
2015- 992 40.21 650 26.35 342 13.86
2010-2014 234 9.49 161 6.53 73 2.96
2005-2009 169 6.85 118 4.78 51 2.07
2000-2004 141 5.72 97 3.93 44 1.78
1995-1999 90 3.65 54 2.19 36 1.46
1990-1994 69 2.80 51 2.07 18 0.73
1985-1989 56 2.27 39 1.58 17 0.69
1980-1984 24 0.97 18 0.73 6 0.24
-1980 23 0.93 15 0.61 8 0.32
occupation judge 170 6.89 114 4.62 56 2.27
prosec 88 3.57 60 2.43 28 1.13
lawyer 540 21.89 347 14.07 193 7.82
in-house 209 8.47 145 5.88 64 2.59
admin 188 7.62 133 5.39 55 2.23
professor 24 0.97 15 0.61 9 0.36
scholar 0 0.00 0 0.00 0 0.00
doctoral candidate 39 1.58 28 1.13 11 0.45
trainee 367 14.88 244 9.89 123 4.99
student 574 23.27 388 15.73 186 7.54
other 175 7.09 122 4.95 53 2.15
field_law private 1103 44.71 738 29.91 365 14.80
criminal 535 21.69 363 14.71 172 6.97
public 514 20.84 349 14.15 165 6.69
none 308 12.48 205 8.31 103 4.18
Demographics (only finished surveys)
between
within
N Percent N Percent N Percent
gender f 593 24.04 419 16.98 174 7.05
m 1780 72.15 1170 47.43 610 24.73
d 11 0.45 7 0.28 4 0.16
no 0 0.00 0 0.00 0 0.00
exam_year no exam 655 26.55 446 18.08 209 8.47
2015- 992 40.21 650 26.35 342 13.86
2010-2014 234 9.49 161 6.53 73 2.96
2005-2009 169 6.85 118 4.78 51 2.07
2000-2004 141 5.72 97 3.93 44 1.78
1995-1999 90 3.65 54 2.19 36 1.46
1990-1994 69 2.80 51 2.07 18 0.73
1985-1989 56 2.27 39 1.58 17 0.69
1980-1984 24 0.97 18 0.73 6 0.24
-1980 23 0.93 15 0.61 8 0.32
occupation judge 170 6.89 114 4.62 56 2.27
prosec 88 3.57 60 2.43 28 1.13
lawyer 540 21.89 347 14.07 193 7.82
in-house 209 8.47 145 5.88 64 2.59
admin 188 7.62 133 5.39 55 2.23
professor 24 0.97 15 0.61 9 0.36
scholar 0 0.00 0 0.00 0 0.00
doctoral candidate 39 1.58 28 1.13 11 0.45
trainee 367 14.88 244 9.89 123 4.99
student 574 23.27 388 15.73 186 7.54
other 175 7.09 122 4.95 53 2.15
field_law private 1103 44.71 738 29.91 365 14.80
criminal 535 21.69 363 14.71 172 6.97
public 514 20.84 349 14.15 165 6.69
none 308 12.48 205 8.31 103 4.18
Demographics (only finished surveys, has state exam)
between
within
N Percent N Percent N Percent
gender f 474 26.36 333 18.52 141 7.84
m 1270 70.63 829 46.11 441 24.53
d 6 0.33 4 0.22 2 0.11
no 0 0.00 0 0.00 0 0.00
exam_year no exam 0 0.00 0 0.00 0 0.00
2015- 992 55.17 650 36.15 342 19.02
2010-2014 234 13.01 161 8.95 73 4.06
2005-2009 169 9.40 118 6.56 51 2.84
2000-2004 141 7.84 97 5.39 44 2.45
1995-1999 90 5.01 54 3.00 36 2.00
1990-1994 69 3.84 51 2.84 18 1.00
1985-1989 56 3.11 39 2.17 17 0.95
1980-1984 24 1.33 18 1.00 6 0.33
-1980 23 1.28 15 0.83 8 0.44
occupation judge 170 9.45 114 6.34 56 3.11
prosec 88 4.89 60 3.34 28 1.56
lawyer 539 29.98 347 19.30 192 10.68
in-house 195 10.85 138 7.68 57 3.17
admin 184 10.23 130 7.23 54 3.00
professor 22 1.22 13 0.72 9 0.50
scholar 0 0.00 0 0.00 0 0.00
doctoral candidate 36 2.00 25 1.39 11 0.61
trainee 366 20.36 243 13.52 123 6.84
student 55 3.06 35 1.95 20 1.11
other 57 3.17 38 2.11 19 1.06
field_law private 895 49.78 594 33.04 301 16.74
criminal 397 22.08 272 15.13 125 6.95
public 377 20.97 253 14.07 124 6.90
none 127 7.06 82 4.56 45 2.50

Treatment results

Unless otherwise indicated, we use only observations after start of data collection where participants have passed the first state exam and have fully completed the survey.

Between subjects

treatment N Mean SD P0  P25  P50  P75  P100
intent harm low 622 50.20 28.82 0.00 26.00 50.00 74.00 100.00
harm high 577 49.75 29.63 0.00 25.00 49.00 75.00 100.00
benefit low 597 45.98 32.32 0.00 17.00 42.00 74.00 100.00
benefit high 601 45.05 32.22 0.00 17.00 41.00 72.00 100.00
prob low 580 57.09 30.29 0.00 33.00 63.00 83.00 100.00
prob high 619 61.15 28.47 0.00 40.00 67.00 83.00 100.00 

## [1] "G1 =  harm low , G2 =  harm high"
## Warning in wilcox.test.default(g1, g2, exact = TRUE): cannot compute exact
## p-value with ties
## 
##  Wilcoxon rank sum test with continuity correction
## 
## data:  g1 and g2
## W = 181232, p-value = 0.7658
## alternative hypothesis: true location shift is not equal to 0
## 
## [1] "G1 =  benefit low , G2 =  benefit high"
## Warning in wilcox.test.default(g1, g2, exact = TRUE): cannot compute exact
## p-value with ties
## 
##  Wilcoxon rank sum test with continuity correction
## 
## data:  g1 and g2
## W = 182284, p-value = 0.6297
## alternative hypothesis: true location shift is not equal to 0
## 
## [1] "G1 =  prob low , G2 =  prob high"
## Warning in wilcox.test.default(g1, g2, exact = TRUE): cannot compute exact
## p-value with ties
## 
##  Wilcoxon rank sum test with continuity correction
## 
## data:  g1 and g2
## W = 166303, p-value = 0.02743
## alternative hypothesis: true location shift is not equal to 0

Within subjects

treatment N Mean SD P0  P25  P50  P75  P100
intent harm low 593 49.14 33.19 0.00 21.00 50.00 76.00 100.00
harm high 595 48.66 33.27 0.00 20.00 46.00 76.00 100.00
benefit low 595 50.09 35.74 0.00 17.00 51.00 84.00 100.00
benefit high 595 55.75 35.21 0.00 24.00 62.00 87.50 100.00
prob low/harm low 594 48.85 32.16 0.00 22.00 43.00 75.00 100.00
prob high/harm low 594 68.40 27.27 0.00 53.00 72.50 92.00 100.00
prob low/harm high 592 55.86 31.36 0.00 33.00 58.00 83.25 100.00
prob high/harm high 592 72.02 28.14 0.00 58.00 80.50 97.00 100.00 

## [1] "G1 =  harm low , G2 =  harm high"
## Warning in wilcox.test.default(g1, g2, exact = TRUE, paired = TRUE): cannot
## compute exact p-value with ties
## Warning in wilcox.test.default(g1, g2, exact = TRUE, paired = TRUE): cannot
## compute exact p-value with zeroes
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  g1 and g2
## V = 36823, p-value = 0.3114
## alternative hypothesis: true location shift is not equal to 0
## [1] "G1 =  benefit low , G2 =  benefit high"
## Warning in wilcox.test.default(g1, g2, exact = TRUE, paired = TRUE): cannot
## compute exact p-value with ties

## Warning in wilcox.test.default(g1, g2, exact = TRUE, paired = TRUE): cannot
## compute exact p-value with zeroes
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  g1 and g2
## V = 20060, p-value < 2.2e-16
## alternative hypothesis: true location shift is not equal to 0
## [1] "G1 =  prob high/harm high , G2 =  prob high/harm low"
## Warning in wilcox.test.default(g1, g2, exact = TRUE, paired = TRUE): cannot
## compute exact p-value with ties

## Warning in wilcox.test.default(g1, g2, exact = TRUE, paired = TRUE): cannot
## compute exact p-value with zeroes
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  g1 and g2
## V = 70793, p-value = 6.561e-14
## alternative hypothesis: true location shift is not equal to 0

Doctrinal views

Doctrinal view: objective vs. subjective (within and between combined)
general_obj_subj N %
subjective 976 54.3
objective 796 44.3
Doctrinal view: criteria (within and between combined)
N Mean SD  P10  P25 Median  P75  P90
general_interest 1787 73.23 20.54 50.00 53.00 75.00 91.00 100.00
general_probability 1793 87.70 15.88 68.00 82.00 93.00 100.00 100.00
general_harm 1789 52.50 21.75 25.00 49.00 50.00 63.00 84.00
general_negligence 1791 69.21 22.21 43.00 54.00 70.00 85.00 100.00
general_effort 1793 26.68 23.85 0.00 6.00 20.00 42.00 52.00
general_accustomed 1792 42.26 22.90 13.00 27.00 47.00 51.00 73.00
Doctrinal view: criteria (within and between combined)
N
Mean
SD
Median
subjective objective subjective objective subjective objective subjective objective
general_interest 971 791 72.02 74.66 20.80 20.11 74.00 77.00
general_probability 974 794 87.49 88.05 15.59 16.26 92.00 94.00
general_harm 971 793 51.51 53.92 21.34 22.11 50.00 50.00
general_negligence 973 793 68.01 70.91 22.07 22.18 68.00 73.00
general_effort 973 795 26.14 26.98 24.08 23.21 19.00 22.00
general_accustomed 972 795 41.16 43.33 22.46 23.25 44.00 49.00

Doctrinal views and case assessment

Plots

Regression

## 
## Call:
## lm(formula = intent ~ treatment * general_harm, data = .)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -59.33 -28.22  -1.24  26.93  62.80 
## 
## Coefficients:
##                          Estimate Std. Error t value Pr(>|t|)    
## (Intercept)              42.54030    2.44785  17.379  < 2e-16 ***
## treatment.L              -7.55969    3.46178  -2.184  0.02918 *  
## general_harm              0.12359    0.04313   2.866  0.00424 ** 
## treatment.L:general_harm  0.13821    0.06099   2.266  0.02363 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 33.07 on 1178 degrees of freedom
##   (8 observations deleted due to missingness)
## Multiple R-squared:  0.01126,    Adjusted R-squared:  0.008739 
## F-statistic: 4.471 on 3 and 1178 DF,  p-value: 0.003952
## 
## Call:
## lm(formula = intent ~ treatment + general_harm + occupation, 
##     data = .)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -62.366 -28.059  -2.312  27.902  70.968 
## 
## Coefficients:
##                               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                   40.73388    3.92154  10.387  < 2e-16 ***
## treatment.L                   -0.17218    1.38718  -0.124  0.90124    
## general_harm                   0.14489    0.04474   3.239  0.00124 ** 
## occupationprosec               9.19496    5.39715   1.704  0.08872 .  
## occupationlawyer               0.80999    3.54684   0.228  0.81940    
## occupationin-house             6.55540    4.40619   1.488  0.13709    
## occupationadmin               -2.40111    4.46898  -0.537  0.59118    
## occupationprofessor          -17.52075    8.37337  -2.092  0.03662 *  
## occupationdoctoral candidate   2.93171    7.72427   0.380  0.70436    
## occupationtrainee             -3.01574    3.75867  -0.802  0.42253    
## occupationstudent              6.40042    6.07587   1.053  0.29238    
## occupationother                0.52904    6.19403   0.085  0.93195    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 32.97 on 1118 degrees of freedom
##   (60 observations deleted due to missingness)
## Multiple R-squared:  0.02547,    Adjusted R-squared:  0.01588 
## F-statistic: 2.656 on 11 and 1118 DF,  p-value: 0.002303

## 
## Call:
## lm(formula = intent ~ treatment * general_harm, data = .)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -51.504 -24.285  -0.601  24.401  50.935 
## 
## Coefficients:
##                           Estimate Std. Error t value Pr(>|t|)    
## (Intercept)              49.692418   2.245654  22.128   <2e-16 ***
## treatment.L               0.886782   3.175835   0.279    0.780    
## general_harm              0.005977   0.039465   0.151    0.880    
## treatment.L:general_harm -0.026031   0.055812  -0.466    0.641    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 29.24 on 1190 degrees of freedom
##   (9 observations deleted due to missingness)
## Multiple R-squared:  0.0003314,  Adjusted R-squared:  -0.002189 
## F-statistic: 0.1315 on 3 and 1190 DF,  p-value: 0.9414
## 
## Call:
## lm(formula = intent ~ treatment + general_harm + occupation, 
##     data = .)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -62.243 -22.794  -0.926  23.617  53.140 
## 
## Coefficients:
##                               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                  48.051663   3.495441  13.747   <2e-16 ***
## treatment.L                  -0.495288   1.232839  -0.402   0.6879    
## general_harm                  0.005185   0.040807   0.127   0.8989    
## occupationprosec              6.741414   4.696220   1.435   0.1514    
## occupationlawyer             -0.867285   3.182682  -0.273   0.7853    
## occupationin-house            2.936102   3.724089   0.788   0.4306    
## occupationadmin               6.145144   3.767495   1.631   0.1032    
## occupationprofessor           6.964617   8.560717   0.814   0.4161    
## occupationdoctoral candidate  5.847877   6.454934   0.906   0.3652    
## occupationtrainee            -0.229076   3.343332  -0.069   0.9454    
## occupationstudent            13.488241   5.657858   2.384   0.0173 *  
## occupationother               3.924437   5.614987   0.699   0.4847    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 29.14 on 1122 degrees of freedom
##   (69 observations deleted due to missingness)
## Multiple R-squared:  0.01426,    Adjusted R-squared:  0.004597 
## F-statistic: 1.476 on 11 and 1122 DF,  p-value: 0.1346

## 
## Call:
## lm(formula = intent ~ treatment * general_interest, data = .)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -58.532 -32.839   4.967  32.551  53.770 
## 
## Coefficients:
##                              Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                  46.99847    3.78560  12.415   <2e-16 ***
## treatment.L                   1.31740    5.35364   0.246    0.806    
## general_interest              0.08015    0.04934   1.625    0.104    
## treatment.L:general_interest  0.03658    0.06977   0.524    0.600    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 35.43 on 1182 degrees of freedom
##   (4 observations deleted due to missingness)
## Multiple R-squared:  0.008842,   Adjusted R-squared:  0.006326 
## F-statistic: 3.515 on 3 and 1182 DF,  p-value: 0.01475

Additional analyses (ideas collection)

Evaluate free text comments at the end of survey

Scale issues

  • Comparing means is problematic because of end-of-scale effects (which deflates means because extremists can’t fully express their views)

Treatment effects

  • Interaction effects in Heating: curiously, there seems to be a negative interaction effect (but mind the scaling issue)
  • Are there treatment effects on variance of judgments?

Effects of b/t and w/i

  • Level effects from w/i
    • No for DELAY
    • Yes for REFERENCE: 50%/55% vs. 45% in b/t
    • No for HEATING: 56%/72% vs. 57%/61% in b/t
  • Treatment effects from w/i (i.e., interaction w/i × treatment)
    • No for DELAY
    • Yes for REFERENCE
    • Yes for HEATING
    • If w/i were the correct effect size, would we have power to find it in b/t?

Doctrinal views

  • Does w/i have an effect on doctrinal views (because it has made certain aspects more salient)

Doctrinal views and intent assessment

  • Figure out if lack of finding (and the unexpected effect direction) in “b/t” for “delay” and “reference” masks individual heterogeneity.
  • Do general views predict case judgments?
  • Our original idea was that “b/t” could elicit intuitions that are not in line with widespread doctrine. Perhaps one can still find this effect when one controls for participants’ doctrinal/general views (prediction: in “b/t”, general views should have less predictive power).

Is there an effect of “w/i” vs. “b/t” on the level of “intent”? That is, does a salient treatment shift the mean/intercept?

Do doctrinal views cluster?