Intervention Program Research Requirements in Mississippi: Registration and Reproducibility

We're here today to give you guidance on setting up a study to evaluate your intervention program.

Mainly, we're here to cover making your research:

• Pre-registered
• Reproducible

But first, we'll talk about…

Performance-based budgeting may feel like it involves unnecessary hoops.

But its requirements:

• Aren't arbitrary
• Aren't idiosyncratic to you in this room
• Aren't separate from what you already do

You in this room aren't being made to do something extra; you're being given an extra chance!

MS Code § 27-103-159 (2014) says:

“Evidence-based program” shall mean a program or practice that has had multiple site random controlled trials across heterogeneous populations demonstrating that the program or practice is effective for the population.

Why is the law so specific?

Because programs meeting that standard answer certain questions:

• Multiple-site, heterogeneous populations: Is the effect generalizable?
• Randomized controlled: Is the effect because of the program?
• Trials: Is the effect real?
• Demonstrating effectiveness: Is the effect important?

(We also want to know whether a program is cost-effective, but that's another story)

Notice:

• These questions aren't unusual or hyperspecialized; they're things we always want to know about programs.
• You can't dependably answer these questions with anecdote or simple numeric comparison.

If you're here, you don't have evidence for your program that meets the legal standard!

Which is another way of saying that you don't have definitive answers to the above questions.

We're here to start you on the road to answering those questions that you, and the state of MS, already care about.

If you read science journalism these days – or just pay attention to what the media say about what foods are good and bad for us – you already know something very important about contemporary research:

There is a crisis of reproducibility in science, especially social science!

Some have gone so far as to suggest that most published research is false.

This isn't the fault of “science.” It's the fault of individual studies!

There are several ways.

• P-hacking
• HARKing
• Reasonable-seeming choices in response to the data

And this isn't even an exhaustive list! These sorts of problems can make even crazy results seem scientifically justfied.

Here's where pre-registration and reproducibility come in.

A simplified overview of the process:

• You complete a research plan
• You submit the plan to the state (pre-registration)
• You do your research
• You write up your research and provide the state with documentation (reproducibility)

Note that these steps are not sufficient to guarantee funding… but each step is necessary.

An excellent conceptual overview of the process is here.

It's strongly recommended reading even if you skip the articles already mentioned!

The report that you pre-register should conform to one of two existing, internationally accepted standards: CONSORT or TREND.

CONSORT is for randomized trials.

CONSORT materials from your handout packet:

• The statement
• The checklist
• The flow diagram
• Explanation and elaboration

The CONSORT website is extremely helpful, and has other checklists and documents that may be useful to you!

TREND is for non-randomized trials.

TREND materials from your handout packet:

• The statement
• The checklist

TREND is designed to work with CONSORT, so the earlier website will be helpful even here.

You will write up a report that includes every item on your checklist except those under 'results' and 'discussion' at the initial submission phase!

After the research is done, you will finish your report, including 'results' and 'discussion', and resubmit!

Note that adequate answers to many of the checklist elements will require fairly technical decisions. This checklist is not a substitute for skilled research staff – it just makes their jobs easier and their results more trustworthy!

A fully randomized trial is highly preferable.

• Easier math (in control and testing)
• Better causal generalizations

Even partial randomization is far inferior, and requires some extra decisions.

• CONSORT does have alternative checklists for this purpose on their site

Nonrandom (quasi-experimental) trials have even more issues!

• In particular, you've got to think hard about controlling for confounding variables.

You are already doing limited trials, so a lottery system is natural.

You may worry about sample size and attrition, but people have developed ways of coping!

When submitting your final research paper:

• Submit all raw data
  • Raw data should be time-stamped from original entry and unaltered
  • Analytic spreadsheets (if any) and data storage spreadsheets should be distinct
• Submit all analytic code
  • As much analysis as possible should be done via code
  • Code should be literate insofar as possible
  • Anything and everything not achieved through code should be written up exhaustively

The goal: The reader should be able to take

• Your raw data (which is unaltered from original entry)

Apply

• Your code (which is understandable to a non-coding English speaker)

And get

• Your results

With no further manipulation necessary!

This is a golden opportunity for you: You are getting paid to do something you already need to do for your own program!

If you do a good job following the standards set out here, by the end of this process you will have cutting-edge, publication-quality research. This will:

• Improve the intellectual standing of your program
• Improve your future funding prospects
• Improve your likelihood of helping clients

Questions?