NIP Budget Estimation Survey

This project examines whether collective intelligence can reveal hidden truths about government spending. Using a “wisdom of the crowds” paradigm, we ask both laypeople and political science experts to estimate details of the classified NIP budget, allowing us to compare the accuracy of general and expert consensus. 

Sample

Study 1: Participants will be N = 1000 English-speaking US residents

Study 2: Participants will be a convenience sample of academic political scientists, which we are characterizing as ‘experts’ in that they are more likely to be familiar with the intelligence community and their objectives. Collected via poli sci listserv

Power

We have characterized our approach as a “wisdom of the crowds” problem. In “wisdom of the crowds” paradigms, the collective error is a function of the diversity of the group and individual error, where increasing diversity of the group reduces collective error (see Centola 2022). In other words, the larger N the better the collective estimate – in theory. These all rest on the assumption that estimating the NIP budget is a true estimation problem that conforms to the wisdom of the crowds formula.  

Bias 

There is no clear reason to expect ordinary members of the public to have systematic bias in their estimates of the NIP budget when the task is to estimate the breakdown by agency.

Systematic bias may be introduced when people are estimating based on objective of each agency. For example, members of the public might be more aware of developments in AI than they are aware of cybersecurity threats posed by China, which would systematically bias their estimates upwards on AI budget.

Ideally, a sample of the general US population that is reasonably well-versed in current events should exhibit a good distribution of error.

In a simple estimation problem with no systematic bias and well-varied error (I.e. a diversified sample), sample sizes as small as N = 200 can be expected to achieve very good estimates.

Study 1: General population 

First, participants are screened for clearance level. Anyone that reports having a clearance (including Public Trust) is routed away and does not encounter the NIP budget estimation task.

The remaining participants next read a brief introduction about the NIP budget and that they will be estimating how much money the NIP allocates to each of its agencies in FY 2024. Throughout the estimation task instructions, we encourage participants to take their best guess and emphasize that we are interested in capturing their intuitive guesses.

After reading about the NIP budget, participants estimate the top line figure for FY 2024

After reporting their estimates for the top line figure, participants were randomly assigned to one of two conditions: blind to the NIP budget topline or not. In the unblind condition, we immediately reveal the actual NIP budget for FY 2024, then participants allocate that budget to each agency in the IC. In the blind condition, participants allocate their estimated budget to each agency in the IC, and then we reveal to them what the actual budget for FY 2024 was. In both conditions, after revealing the top line figure, participants report whether they think the budget is too low or too high.

Finally, participants allocate each agency’s budget across 8 objectives, shown below.

clearance

Do you now or have you ever held a clearance with the federal government?

  1. Yes, public trust or equivalent
  2. Yes, confidential clearance
  3. Yes, secret clearance
  4. Yes, Top Secret clearance
  5. Yes, Top Secret/SCI clearance
  6. Yes, another type of clearance or clearance in progress

estimate_top_number and estimate_topline_magnitude

type a number in the box and select “million” or “billion” from the dropdown menu

The National Intelligence Program budget for FY 2024 was… [number] [magnitude; 1 = million, 2 = billion]

estimate_byAgency_x

Next, estimate - by percentage - how the budget was allocated to each of the agencies and groups. You are estimating by percentage, so your total must add up to 100%, and you will not be able to allocate more than 100%. [0 - 100]

topline_eval

Would you say that the NIP Budget of $76.5 Billion is too high, too low, or just right?

  1. Much too high - we should spend much less on the National Intelligence Program
  2. Too high - we should spend less on the NIP
  3. Just right - we are spending the right amount on the National Intelligence Program
  4. Too low - we should spend more on the National Intelligence Program
  5. Much too low - we should spend much more on the National Intelligence Program

x_estimate_byObjective_x

Now please estimate - by percentage - how much of their budget the ${lm://Field/2} allocated for each of these domains, relevant for FY 2024.

As a reminder, you indicated that the ${lm://Field/2} received ${lm://Field/3}% of the NIP budget, which translates to $$e{ lm://Field/3 * 765} million.

[0 - 100]

optout

We aim to use only high-quality responses. If you feel you were not paying attention during this survey, were distracted, or have another reason to believe your responses are not high quality, let us know here.

Would you like your responses removed so they are not used in analysis?

  1. Yes, delete my data
  2. No, keep my data

Participants & exclusions

We collected a total of 986 complete responses after excluding those who opted out at the end of the survey. A further 129 did not respond to the NIP budget estimation task because they reported having some level of clearance.

Participants by clearance level

clearance

n

none

857

publicTrust

62

confidential

28

secret

22

topSecret

9

topSecretSCI

6

other

2

Some participants wrote in unusual figures for their top line estimates. To address this subset of participants, we manually reviewed each of the 9 submissions that reported numbers greater than 1 trillion. Decisions on the data points are detailed below.

Reminder: add note about robustness verification w/ and w/o exclusions

  PID estimate_topline_number estimate_topline_magnitude
1  24                    1200                          2
2  78                  250000                          2
3 312               100000000                          2
4 330                10000000                          2
5 603               100000000                          1
6 654                 1000000                          2
7 830                12400000                          1
8 872                50000000                          2
9 887             40000000000                          2
  • PID 24: kept. reasonable estimate at 1.2 trillion
  • PID 78: dropped. not obvious how to interpret 250,000 * billion
  • PID 312: dropped. not obvious how to interpret
  • PID 330: dropped. not obvious how to interpret
  • PID 603: kept, changed to 100 million
  • PID 654: dropped, not obvious how to interpret
  • PID 830: kept, changed to 12.4 million
  • PID 872: dropped. not obvious how to interpret
  • PID 887: kept, changed to 40 billion

Results

The aggregate estimate for the top line of the NIP budget FY2024 was $83479002347.4178

This evidence suggests that estimating the black budget’s top-line figure can be reasonably characterized as a wisdom-of-the-crowds estimation problem. Estimates of the NIP budget drawn from a reasonably diversified sample of the general public exhibit sufficiently distributed variance around the true value such that their aggregate converges on the true mean.

Estimates by agency: overall

Overall, the aggregate estimate of the top line figure of the NIP budget (83.5 billion) is very close to the true value (76.5 billion). When estimating by agency, on average, people estimate that the CIA has the largest portion of the NIP budget by far. They estimate that ODNI, NRO, and NGA have the smallest portions of the NIP budget. We found no evidence that being blind to the top line figure (vs being told the true top line figure) had an effect on the agency estimates.

The delta between the per-agency estimates and the agencies’ true budgets is wider than the delta between the top line estimate and the true estimate. Accordingly, there is reason to expect that the per-agency estimation task does not constitute a wisdom-of-the-crowds type of estimation problem as well as estimation of the top line figure does. This is likely because systematic bias is introduced to the per-agency estimates.

Readers will note that the per-agency budget estimates reflect a hierarchy based on name recognition, where people estimated that the more recognizable agencies - CIA, NSA, FBI - received more funding than the less recognizable agencies - NGA, NRO, ODNI. This pattern suggests that people use a rule of thumb where the bigger the agencies are in people’s minds - which could be because of some combination of popular culture, media, current events, and historical prominence - the bigger the slice of the NIP pie. This pattern suggests that the per-agency estimation task does not share the properties that constitute a wisdom-of-the-crowds estimation problem because systematic bias is introduced that biases people to overestimate the budgets of agencies based on name recognition - a feature that does not reliably map onto true budgetary allocations.

Estimates by objective

Exploratory measure

Immediately after revealing the true top line NIP figure for FY 2024, participants reported whether they felt the budget was too high or too low. Participants responded on 5 point scale: 1 = “much too high”; 3 = “just right”; 5 = “much too low”


Call:
lm(formula = estimate_topline ~ topline_eval, data = budgetEstimationData)

Residuals:
          Min            1Q        Median            3Q           Max 
-137663549561  -69950306903  -36589685575   -5130617568 1095689071768 

Coefficients:
                Estimate  Std. Error t value     Pr(>|t|)    
(Intercept)   4229064246 14946107791   0.283        0.777    
topline_eval 33360621329  5884461393   5.669 0.0000000197 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 154100000000 on 849 degrees of freedom
  (1 observation deleted due to missingness)
Multiple R-squared:  0.03648,   Adjusted R-squared:  0.03534 
F-statistic: 32.14 on 1 and 849 DF,  p-value: 0.00000001965

There is a relationship between evaluations of the NIP budget and participants’ estimated top line figure, where the higher their estimate of the top line, the more they think the government should spend on the NIP.