New NSM Analysis for DB and Confidence Manuscript

Simulation Goals

The goal of these simulations is to create a ‘bare-bones’ example of how confidence can be related to accuracy within one mind. Here is the basic framework: Participants are presented with two cues represented by Normal distributions with some mean and variance. Cue A is defined as a Normal with mu = 0 and sd = 1. Cue B is defined as a Normal with a positive bias (mu > 0), and can take on several different standard deviations (e.g.; low = .5, medium = 1, high = 2). In Phase 1, the agent chooses a cue at random and draws N (in this case 4) random samples. It then generate best estimates and confidence intervals (using one of two definitions for each). In Phase 2, it is assigned to either a control or dialectical condition. In the control condition, it selects the same cue and in the dialectical condition it selects the other cue. It then draws N (again, 4) new random samples from the selected cue.

Estimate formulas

In each estimation phase, agents select a probe cue, then draw \(wm_i\) samples from the corresponding cue distribution, where \(wm_i\) is the working memory capacity of the agent. To keep things simple, I’ll set \(wm_i\) to 4. These samples define the Subjective Sample Distribution (SSD). Based on the SSD, the agent gives responses as follows:

1. Best Estimates

   • Median = median(SSD)
   • Mean = mean(SSD)

2. Confidence Intervals

   • Range = range(SSD)
   • Juslin (see Juslin et al., 2009) = \(2 \times \sigma_{SSD} \times qz_{.9}\)

Simulation Code

(hidded but included in markdown)

Results

Does confidence correlate with accuracy in first estimates? I’ll test this in three ways

1. Within an unbiased cue (mu = 0, sd = 1) (green line)
2. Within a biased cue (mu = .5, sd = 2) (blue line)
3. Across both cues (orange line)

Notes

1. The green points are for the unbiased cue (mu = 0), the blue points are for the biased cue (mu = .5).
2. The blue and green regression lines regress best estimate absolute deviation on confidence within a cue.
3. The orange regression line does the same regression but across both cues.

Key Results

1. The operational definition of best estimates (mean vs. median), and imprecision (SSD range vs. Juslin) does not affect any of our main conclusions (all four plots look virtually identical)

2. Within a cue, imprecision is virtually uncorrelated with estimate error (!). You can see this by the very small r values within each cue. This is true for both the unbiased cue and the biased cue. In other words, if you use a single cue, then confidence should be unrelated to accuracy of point estimates.

3. Across cues, imprecision is positively correlated with estimate error. You can see this by the positive r values for the orange regression lines. In other words, the more confident and more precise you are, the more accurate your best estimate is.

Conclusion

Positive resolution in repeated judgments is due to differential cue use, and not to repeated estimation from the same cue.

Comparing estimate accuracy: First vs. Average (blend) vs. High Confidence

Criterion Definitions

Next I explore the accuracy of different aggregation strategies: First (take the first best estimate), Blend (average the two best estimates), and High Confidence (take the high confidence estiamte).

1. First - Blend: Mean difference in absolute deviation between first estimates and blended estimates. Higher values indicate higher accuracy for blend.

2. First - HConf: Mean difference in absolute deviation between first estimates and High confidence estimates. Higher values indicate higher accuracy for High confidence estimates.

3. Blend - HConf: Mean difference in absolute deviation between blended estimates and High confidence estimates. Higher values indicate higher accuracy for High confidence estimates.

Best est = Mean, Confidence = Range

Best est = Median, Confidence = Range

Best Est = Mean, Confidence = Juslin

Best est = Median, Confidence = Juslin

Results

1. No systematic difference between using mean or median as best estimate.

2. No systematic difference between using SSD range or Juslin’s approach

3. Dialectical instructions tend to increase accuracy in all conditions EXCEPT when cue B is biased (mean > 0) and has an smaller variance than cue A (i.e.; less than 1).

4. High confidence choosing only outperforms averaging when cue B is both biased AND has a higher variance than cue A (bottom right plot).

Conclusion

High confidence choosing will benefit you (relative to first estimates) in all cases EXCEPT when one (evil) cue has a smaller variance AND a higher bias than the other cue. However, high-confidence choosing only beats averaging when one cue has a higher variance and a higher bias, and provided that you use both cues (i.e.; dialectical bootstrapping).