Introduction

• Very much a work in progress
• Will take any feedback, would love suggestions

Monitoring & Performance

High Level RQ: Does collecting data as part of project M&E improve environmental project performance?

Quote from Medium-Size Project terminal evaluation:

“At the mid-point of a project, the accumulation of implementation issues, including a Project Steering Committee that remains to be established, multiple delays in setting up the project management unit (including the recruitment of WCS as Chief Technical Advisor), in recruiting experts and in starting-up activities - significant changes in the implementation arrangements - repeated ratings in the unsatisfactory range for the annual project implementation reviews (PIRs) and the poor quality of these reports, should have triggered some sort of warning to require that a midterm review is conducted to identify issues and solutions to increase the chances of meeting intended outcomes or of making progress by the end of its implementation cycle.”

Monitoring & Performance

High Level RQ: Does collecting data as part of project M&E improve environmental project performance?

Goal: Identify effect of midterm performance report on GEF project outcomes

Relevant Literature

• Project-level characteristics greater determinant of performance than country level (Dollar & Svensson 2000; Denizer et al. 2013; Bulman et al. 2017)
• Monitoring and evaluation may be correlated with performance (Chauvet et al. 2018; Kilby 2000; Bourgiugnon & Platteau 2024)

Gaps in Literature

• No specific focus on M&E in Environmental projects
• Most data is from WB projects
• Lots of concerns about project interim reports being biased
• Only binary performance scores used (Successful/Unsuccessful)
• Research is generally more concerned with the role of monitoring as supervision as less as a strategy for learning

Data & Methods

• Setting: GEF Projects (all)
• Data: GEF-4 Projects (Maybe GEF-5, GEF-6)

Why?

• GEF policies designed so that projects receiving more than 1M in funding from the GEF trust fund (Full-Size Projects) are required to undergo a thorough midterm review by an external evaluator, with results of the review sent to GEFIEO
• Projects receiving less than 1M (Medium-Size Projects) do not undergo a midterm review, only a terminal evaluation

Identification Strategy

RD using cutoff for Medium-Size Projects (MSPs) and Full-Size Projects (FSPs)

Identification Strategy

Note: Total budget varies across project, cutoff only applies to amount of GEF Grant funding received

Identification Strategy

Concerns/Issues

• Using a modified project outcome score based on Ordered Probit results – is this valid?
  
• Lack of balance above/below cutoff
• Bunching around cutoff
• Ken G suggested doing a bunching analysis (instead?), but not sure how/what to do with bunching analysis

RD Analysis

*Using GEF-4 data for all following results

• Outcome variable: Modified Outcome Score (Probit Results)

Brief segue into the outcome score analysis…

• Projects are “rated” by an external evaluator at the conclusion of the project.
• Projects are assigned four intermediate ratings, then one “comprehensive” outcome score

Intermediate Ratings:

• M&E Design (1-6)
  
• M&E Implementation (1-6)
  
• Execution (1-6)
  
• Likelihood of Sustainability (1-4)

Final Outcome Rating (1-6)

• (1 = Highly Unsatisfactory -> 6 = Highly Satisfactory)

Modified Outcome Score

• No actual formula for how the final outcome score is calculated
• Most Projects are rated as 4 or 5, despite wider spread in intermediate ratings

Ordered Probit Results

Modified Outcome Score

Using the probit results, I generate an adjusted outcome score for each project:

\[ Outcome_{adjusted} = (\text{M&E Imp.}) + 3* (\text{Execution}) + 2* (\text{Sustainability}) \] —

##        left     right
## h  616041.1  616041.1
## b 1233724.6 1233724.6

End of segue!

Back to our RDD…

RDD Results

Bandwidths (Million USD)

• Uncorrected (h): \(\pm\) 0.62 \(\implies\) (0.38, 1.62)
• Bias-Corrected (b): \(\pm\) 1.23 \(\implies\) (-0.77, 2.23)

*Using bias-corrected bandwidth estimates (rddrobust package) because of imbalance of data near cutoff

RDD Results

Balance Table

RD Results

Where do we go from here?

RD Analysis:

• Should I narrow the sample?
• I have GEF-5 data, but cutoff is different
• Bunching Analysis?

GEF - 4 Histogram (and evidence of Bunching)

Bunching Analysis

• Seems to have very different purpose
• Can show that there is bunching around the cutoff, but not sure what to do with this
• Would it make sense to do a DWL calculation if I used CO2 emissions as an outcome variable?
• Could also look at whether the bunching follows expected patterns (more risky project means manipulation of grant size) (this type of analysis is done by Chauvet et al. 2018)

GEF - 5 Histogram (and evidence of Bunching)

Start of Bunching Analysis à la Ito & Sallee

• Method: Count how many projects are in each funding bin, then remove the projects that are exactly 1M, because this is evidence of funding manipulation
• Why? Theoretically allows for a counterfactual estimate of projects that should have been made FSP, but weren’t

Circling Back

Quote from Medium-Size Project terminal evaluation:

“At the mid-point of a project, the accumulation of implementation issues, including a Project Steering Committee that remains to be established, multiple delays in setting up the project management unit (including the recruitment of WCS as Chief Technical Advisor), in recruiting experts and in starting-up activities - significant changes in the implementation arrangements - repeated ratings in the unsatisfactory range for the annual project implementation reviews (PIRs) and the poor quality of these reports, should have triggered some sort of warning to require that a midterm review is conducted to identify issues and solutions to increase the chances of meeting intended outcomes or of making progress by the end of its implementation cycle.”

Conclusion

Thanks for listening! Super grateful for this group ❤️

Appendix