Reviews of Secondary Research:
A Growing Frontier for Evidence Synthesis

Dr. Damien Beillouin
Researcher, CIRAD & Trainer, FRB Training Course UPR HortSys, PerSyst – CIRAD

2025-06-17

Act I — The Data Deluge: Food Systems on a Burning Planet 🌍🔥

“We are the first generation to feel the effects of climate change — and the last that can do something about it.”
— Barack Obama - 2015

  • Food systems are the main driver behind 5 of the 6 planetary boundary breaches

  • The food system alone is responsible for:

    • 30% of GHG emissions,
    • 70% of freshwater use,
    • 80% of deforestation,
    • Massive biodiversity loss.
    • Food system change is vital to achieve 1.5°C, beyond fossil fuel shifts.
    • Diets change could prevent up to 15 million deaths per year.


Planetary Boundaries

Campbell, B. M., et al. 2017. Ecology and Society
doi.org/10.5751/ES-09595-220408

“Science is built up of facts, as a house is built of stones; but a collection of facts is no more a science than a heap of stones is a house.”
Henri Poincaré

The volume of scientific publications is staggering:

  • Over 2.5 million new peer-reviewed articles annually.
  • Approximately 7,000 new publications every single day.
  • Our cumulative scientific knowledge effectively doubles every 9-10 years.


  • Over 10,000 systematic reviews are published annually in health sciences alone.
  • In environmental and agricultural sciences: from a few hundred in early 2000s to ~1,500-2,000 per year

➤ An Avalanche of Knowledge.

Planetary Boundaries

Ouvrir la Science clic here

NB_MA

Bonfanti et al., 2023, Scientific Data https://doi.org/10.1016/j.dib.2023.109555

:::

Pyramide

Meta-of-meta: connecting the dots across hundreds of studies.

:::

Act II — Umbrella Reviews Illustrated Through Concrete Examples 🛠️⚖️

— 🌾 Crop Diversification and Planetary Boundaries: Outdated Tradition or Future Solution? —

image_crop_diver

Diversified system.

monoculture

Monocrop field

The Challenge: Navigating a Sea of Solutions 🌊


The critical challenge in agricultural science isn’t a lack of ideas, but a lack of clarity amidst their abundance:

  • Overwhelming Solutions: We’re faced with a multitude of proposed solutions: regenerative agriculture, agroforestry, organic farming, GMOs, intercropping, cover crops, and more. This sheer volume can lead to decision paralysis.

  • Contextual Variability: The effectiveness of these solutions varies dramatically. Results are highly dependent on specific factors like soil type, local climate, crop species, and precise management practices. What works in one field may fail in another.

  • The Decision-Maker’s Dilemma: Policymakers and practitioners grapple with a crucial question: How can they identify high-quality, non-redundant evidence syntheses that offer reliable, context-aware guidance for effective action?

Complexity

Kremen et al. in prep, https://doi.org/10.XXXX

🌾 Diversification Works — And the Numbers Prove It

Diversified cropping systems often outperform simplified ones — for nature, for climate, and for food production

GCB_Results

Beillouin et al., 2021. Global Change Biology.
doi.org/10.1111/gcb.15747

🌿 How Does Crop Diversification Compare to Other Farming Strategies? - CARBON

Not all agricultural practices perform equally.
Crop diversification — including agroforestry — stands out as one of the most effective strategies
for improving ecosystem services and long-term sustainability.

MA_Carbon

Beillouin et al., 2021. Global Change Biology.
doi.org/10.1111/gcb.15747

🌿 How Does Crop Diversification Compare to Other Farming Strategies? - BIODIVERSITY

Not all agricultural practices perform equally.
Crop diversification — including agroforestry — stands out as one of the most effective strategies
for improving ecosystem services and long-term sustainability.

MA_Biodiv

Beillouin et al., in prep. .
doi.org/XXX

Still Emerging Evidence

K_Gap

Beillouin et al., 2019. Data in brief.
doi.org/10.1016/j.dib.2019.103898

What Species Are We Talking About?

Species_Gap

Beillouin et al., 2019. Env. Res. let..
doi.org/10.1088/1748-9326/ab4449

Blind Spots on the Global Map

Map_divers

Beillouin et al., 2021. Glob. chane Biology.
doi.org/10.1111/gcb.15747

Act III — Methodology and Challenges: The Core of Umbrella Reviews 🛠️⚖️

Some Advantages of high level Data integration


Embracing Umbrella Reviews is a strategic advancement.

  • Synthesizing Complex Phenomena: Ecological and evolutionary questions are vast; umbrella reviews aggregate diverse evidence for a holistic picture.

  • Enhancing Trustworthiness: By appraising and integrating multiple syntheses, they identify consensus and conflicts, increasing confidence in conclusions.

  • Optimizing Research Agendas: They pinpoint redundancies and critical knowledge gaps, precisely guiding future research priorities.

Challenges of High-Level Data Integration ⚠️

Focus on the quality of meta-analyses

  • Quality Propagation: The overall quality of an umbrella review is inherently limited by the methodological rigor and reporting of the individual syntheses it includes. Flaws can propagate upwards.

Quality

Focus on the quality of meta-analyses

Doi_quality

Focus on the overlap of primary studies

  • Overlap Bias: Multiple included reviews often share the same primary studies. This can artificially inflate the precision of findings if not properly accounted for.

  • Quantification: Corrected Covered Area (CCA) (see Kirvalidze et al., 2022)

\[ \text{CCA} = (N - r) \frac{1}{(r . c) - r} \]

Where:
* \(N\) is the total number of times primary publications appeared in reviews (inclusive of double-counting).
* \(r\) is the number of unique primary publications.
* \(c\) is the number of systematic reviews included in the umbrella review

Redundancy

Focus on the publication bias

Selective Reporting & Publication Bias (at the Synthesis Level):

Challenge: Syntheses with “positive” findings are more likely to be published, distorting the overall picture.

Detection: Funnel plots and regression tests (e.g., Egger’s test) can be adapted at the meta-level

Focus on the black box problem

  • “Black Box” Problem: Umbrella reviews typically aggregate summary data from syntheses, not raw primary study data. This can limit the ability to conduct very granular, novel subgroup analyses.

-> What Shapes the Success of Diversified Cropping Systems?

Effect of legume-based rotations on crop yield (global scale)

Effect of agroforestry on SOC (LAC countries)

detailled study

Zhao et al., 2022. Nat. Comm.
doi.org/10.1038/s41467-022-32464-0

Beillouin et al., XXX. in Prep
doi.org/XXXX

Time Lag in Rapidly Evolving Fields: A Persistent Challenge ⏳

Even with advanced synthesis methods, staying current in dynamic fields remains a significant hurdle:

  • Inherent Delay: A rigorous umbrella review, designed for comprehensiveness and quality, typically takes 6-12 months to complete.

SOMA

SOMA: second order MA REMA: First order MA MAMA: Select the MA with lower uncertainty COMA: Vote counting of MA

Act IV — Conclusion: The Future of Evidence-Based Decision-Making – Also Implies Umbrella Reviews 🌿✨

The Umbrella Review Revolution

The explosion of knowledge demands an evolution in how we synthesize and apply evidence.

  • Umbrella Reviews are a critical methodological advancement.
  • They enable us to navigate the information deluge and drive real scientific progress.

These high-level syntheses offer distinct advantages:

  • Aggregating evidence beyond single systematic reviews for macroscopic, integrated understanding.
  • Exposing and quantifying inherent biases (e.g., overlap, quality propagation), enhancing trustworthiness.
  • Guiding future research by precisely identifying robust evidence, inconsistencies, and knowledge gaps.
  • Facilitating knowledge translation to interdisciplinary researchers and policymakers.

…. BUT not only

  • Also subject to their own biases (e.g., selection of reviews, synthesis methods), which need careful consideration… (same problems as first order MA + some others!)

  • Current syntheses often fall short of answering essential, complex questions, such as the interaction effects between different practices (e.g., how regenerative agriculture combines with agroforestry).

Towards a More Agile & Comprehensive Synthesis 🚀

The future of evidence synthesis demands continuous innovation to truly inform policy and practice:

  • Living Reviews: To overcome time lag, living systematic reviews and living umbrella reviews are essential. These dynamic, continuously updated syntheses ensure findings remain current in fast-evolving fields. see : -> https://www.impact4soil.com/


  • Open Data & Infrastructure: The true power of advanced synthesis hinges on the widespread availability of high-quality data and the development of robust, accessible databases. Without this, comprehensive analysis is hindered. see current ongoing project at the joint research center. -> see: https://data.jrc.ec.europa.eu/collection/id-00399


Thank You! 🙏

Questions & Discussion



Dr. Damien Beillouin\ Researcher, CIRAD\ damien.beillouin@cirad.fr

Damien Beillouin – CIRAD | Research Synthesis