Overviews in Education Research: A Systematic Review and Analysis

Joshau R. Polanin, Brandy R. Maynard, Nathaniel Dell
July 9, 2015

What is an overview?

Structure of overviews

hierarchy

  • Synthesis of syntheses
    • Terms: Overview (Cochrane terminology); review of reviews; meta-meta-analysis; meta-synthesis; umbrella reviews; many others
    • Attempt to combine, quantitative or otherwise, multiple reviews (systematic or otherwise) of primary research

Are overviews needed?

bastian

  • Growing systematic review enterprise
  • Overviews are well-cited
SRSM 2015, Polanin et al. [3]

Overviews in brief

  • Reasons for conducting an overview
    1. Systematic review industry is booming
    2. Comprehensive structure for defining and disseminating large bodies of research
    3. Reviews may derive different conclusions
    4. Network meta-analysis
  • Reasons against conducting an overview
    1. Messy
    2. Level of abstraction
    3. Ill-defined (at least in the education/social sciences) conduct and reporting guidelines
SRSM 2015, Polanin et al. [4]

Overviews in the literature

Cochrane handbook

cochrane

SRSM 2015, Polanin et al. [5]

Overviews in the literature

Thomson et al. (2014)

thomson

SRSM 2015, Polanin et al. [6]

Overviews in the literature

Cooper & Koenka (2012)

cooper

SRSM 2015, Polanin et al. [7]

The purpose of the project

How are overviews created and used in education?

Research questions

Question 1 To what extent are overviews being conducted in the area of education related research with pre-school to post-secondary student populations?

Question 2 To what extent are methodological characteristics being reported in overviews of education?

Question 3 What methods are overview authors using to conduct overviews?

SRSM 2015, Polanin et al. [9]

Methods

Methods

  • Inclusion criteria
    • Studies where the goal was to synthesize more than one review
    • More than 50% education-related
    • Published/unpublished; no date restrictions; English only
  • Search
    • Nine international databases and Google Scholar
    • Reference harvested; authors contacted; websites searched
SRSM 2015, Polanin et al. [11]

Methods

  • Screen
    • One person abstract screened
    • Two people independently double coded full-text
  • Code
    • 1) bibliographic information; 2) overview characteristics and methods; 3) data related to information overview authors provided about included reviews; 4) overlap, quality, and up-to-dateness of included reviews; and 5) overview synthesis methods; 6) Google Scholar citations as of March 2015.
    • Double coded (92.5% agreement)
    • Filemaker Pro
  • Analysis
    • Descriptive and groupings; timing analysis
SRSM 2015, Polanin et al. [12]

Results

Results

Screening process

screen

SRSM 2015, Polanin et al. [14]

Results

Sample characteristics (k = 25)

sample

SRSM 2015, Polanin et al. [15]

Results

Methodological characteristics

characteristics

Results

Synthesis methods

plot of chunk unnamed-chunk-1

SRSM 2015, Polanin et al. [17]

Results

Synthesis methods: Quantitative synthesis only

plot of chunk unnamed-chunk-2

SRSM 2015, Polanin et al. [18]

Results

Synthesis methods: Quantitative synthesis only

plot of chunk unnamed-chunk-3

SRSM 2015, Polanin et al. [19]

Results

Overlap of primary studies across reviews?

  • Only 32% mentioned or accounted for overlap
  • 3 studies “removed” highly overlapping studies
  • 1 study provided a matrix of all overlapping studies
SRSM 2015, Polanin et al. [20]

Results

Review characteristics reported

review

Conclusions

Conclusions

Three major concerns

  • Lack of reporting of methods used and characteristics of included reviews and primary studies
  • Sparse attention to overlap across reviews

  • Underreporting of procedures used to synthesize reviews

  • Conclusion: Major room for improvement!

SRSM 2015, Polanin et al. [23]

Thank you!

  • Contact
    • Brandy Maynard, MSW, Ph.D.
    • Assistant Professor
    • Saint Louis University
    • bmaynar1@slu.edu