1 Summary

Here I show how we calculated the averaging gains in the following study

Herzog, S. M., & Hertwig, R. (2009). The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping. Psychological Science, 20, 231–237. doi:[10.1111/j.1467-9280.2009.02271.x](http://dx.doi.org/10.1111/j.1467-9280.2009.02271.x)

2 Some illustrative data

Here I create some synthetic data for 5 participants and 5 questions each. The correct answers for the questions are: 80, 90, 100, 110, 120. I assume that

To keep the numbers simple, the judgments are rounded to the nearest integer.

As in the paper (footnote 3, p. 234), the average between the first and second estimate is also rounded:

All averages of two estimates were rounded, so any superiority of the averages cannot be explained by their being more fine grained than the raw estimates.

Below I create this data and calculate several measures.

2.1 Absolute errors of first and averaged judgments

We calculate for each participant i and question j the absolute error \({ {\rm{AE}}_{ij} }\) between the estimate \({{J}_{ij}}\) its true value \({ {\rm{truth}}_{j} }\)—for first and averaged estimates (1 or “avg” in the superscript):

\[{ {\rm{AE}}_{ij}^1 = \left| {J_{ij}^1 - {\rm{truth}_j}} \right| }\] \[{ {\rm{AE}}_{ij}^\rm{avg} = \left| {J_{ij}^\rm{avg} - {\rm{truth}_j}} \right| }\]

2.2 Accuracy gains

We defined accuracy gains from using the averaged judgment instead of the first judgment \({ {\rm{AG}}_{ij}}^{{\rm{J1} \to {\rm{avg}}} }\) as follows (p. 234; my emphasis)

The accuracy gain for a participant was defined as the median decrease in error of the average of the two estimates relative to the first estimate, across items.

\[{\rm{AG}}_{ij}^{{\rm{J1}} \to {\rm{avg}}} = {{{\rm{AE}}_{ij}^1 - {\rm{AE}}_{ij}^{avg}} \over {{\rm{AE}}_{ij}^1}}\]

This is the proportion of absolute error reduced when using the average instead of the first judgment.

  • If the average and the first judgment are equally accurate (i.e., \({{\rm{AE}}_{ij}^1 = {\rm{AE}}_{ij}^{avg}}\)) then \({ {\rm{AG}}_{ij}^{{\rm{J1}} \to {\rm{avg}}} = 0 }\) (i.e., zero percent of the error has been reduced).

  • If the average is more accurate than the first estimate (i.e., \({{\rm{AE}}_{ij}^1 > {\rm{AE}}_{ij}^{avg}}\)) then \({ {\rm{AG}}_{ij}^{{\rm{J1}} \to {\rm{avg}}} > 0 }\) (i.e., a positive percent of the error has been reduced).

  • If the average is less accurate than the first estimate (i.e., \({{\rm{AE}}_{ij}^1 < {\rm{AE}}_{ij}^{avg}}\)) then \({ {\rm{AG}}_{ij}^{{\rm{J1}} \to {\rm{avg}}} < 0 }\) (i.e., a negative percent of the error has been reduced, that is, the error actually increased).

Table 1: Illustrative, synthetic data set
p_i p_i_error_sd q_j_truth J1 J2 avg AE_J1 AE_avg AG_J1_to_avg
1 10 80 97 52 74 17 6 0.65
1 10 90 129 79 104 39 14 0.64
1 10 100 112 90 101 12 1 0.92
1 10 110 136 78 107 26 3 0.88
1 10 120 162 86 124 42 4 0.90
2 15 80 96 60 78 16 2 0.88
2 15 90 108 82 95 18 5 0.72
2 15 100 115 108 112 15 12 0.20
2 15 110 131 103 117 21 7 0.67
2 15 120 147 117 132 27 12 0.56
3 20 80 130 74 102 50 22 0.56
3 20 90 99 45 72 9 18 -1.00
3 20 100 103 35 69 3 31 -9.33
3 20 110 132 50 91 22 19 0.14
3 20 120 124 92 108 4 12 -2.00
4 25 80 102 94 98 22 18 0.18
4 25 90 100 45 72 10 18 -0.80
4 25 100 70 73 72 30 28 0.07
4 25 110 141 132 136 31 26 0.16
4 25 120 139 84 112 19 8 0.58
5 30 80 132 72 102 52 22 0.58
5 30 90 143 38 90 53 0 1.00
5 30 100 169 61 115 69 15 0.78
5 30 110 156 63 110 46 0 1.00
5 30 120 170 93 132 50 12 0.76

3 Quantifying averaging gains per participant

To summarize the averaging gains for a participant, we calculated the median averaging gain per participant (p. 234):

The accuracy gain for a participant was defined as the median decrease in error of the average of the two estimates relative to the first estimate, across items.

Table 2: Median averaging gain per participant
p_i median_AG_J1_to_avg
1 0.88
2 0.67
3 -1.00
4 0.16
5 0.78