PBB presentation

PEER staff
2/16/16

What am I going to talk about?

First, a discussion of program inventory.

  • Why we care about it
  • Inventory webapp demonstration

General outline continued

Second, a discussion of program evaluation.

  • What it is and why it matters
  • Results first ROI modeling
  • The 7 elements of quality program design

Program inventory

Agencies involved in the inventory pilot:

  • Corrections
  • Education
  • Health
  • Transportation

Program inventory

The project seems simple, but is surprisingly complex!

It's also necessary for several reasons:

  • Accountability for money spent
  • Accountability for performance
  • Transparency for all stakeholders

Program inventory

It's not enough to have an inventory; we also have to make that inventory available!

But here we face a problem: Balancing the needs of public users and power users.

Program inventory

A possible solution

Program inventory

Where do we go now?

  • Improve the current product
  • Expand the inventory
  • Change the culture
    • At the agency level
    • At the state level

Program inventory

Questions?

Program evaluation

In the current context, we're mostly talking about evaluating intervention programs.

Intervention programs defined by opposition:

  • Programs we need for their own sake
  • Programs that do something for us

The distinction can get fuzzy, but not insolubly so.

Program evaluation

Five questions to ask about an intervention program:

  • Is the effect real?
  • Is the effect important?
  • Is the effect because of the program?
  • Is the effect generalizable?
  • Is the program cost-effective?

Program evaluation

A quick digression on reality, with pictures!

Imagine two programs doing the same thing.

  • One of them achieves 40 units of effect. Boo.
  • The other achieves 80 units of effect. Yay!

Here's what we probably imagine...

plot of chunk unnamed-chunk-1

But here's what could be happening!

plot of chunk unnamed-chunk-2

Program evaluation

Back to the five questions:

  • Is the effect real?
  • Is the effect important?
  • Is the effect because of the program?
  • Is the effect generalizable?
  • Is the program cost-effective?

Program evaluation

A takeaway point worth its own slide:

Answering these questions can sometimes get technical, but asking them is just common sense! We already want to know this stuff.

Program evaluation

How do these questions relate to what we actually do?

  • Looking forward, we should:
    • Choose evidence-based programming
    • Calculate ROI rigorously
  • Looking backward, we should:
    • Evaluate our own programs carefully
      • Plan ahead
      • Do statistical analysis
      • Use control groups
      • Randomize if possible

Program evaluation

Finally, the Results First ROI analysis!

3 steps:

  • Meta-analyze (the research)
  • Monetize (the effect size)
  • Monte Carlo (the result from step 2)

Program evaluation

The 7 elements of quality program design

  • Premise of new activity
  • Needs assessment
  • Description of new activity
  • Research and evidence filter
  • Implementation plan
  • Fidelity plan
  • Measurement and evaluation

That's all, folks!

Questions?