Reminders and Tips

• You can use control + F or command + F to search this lesson quickly.
• Send any comments or questions to Anshul Kumar (akumar@mghihp.edu). Also copy your course instructor.

Learning goals

This week we will learn about quantitative meta-analysis.

The learning goals for this week are as follows:

1. Explain what a meta-analysis adds to a systematic review.
2. Compare and explain differences between the different types of data and effect measures, including distinguishing between fixed effect and random effects meta-analysis.
3. Describe how to best review and interpret a published meta-analysis.
4. Optional: Practice using meta-analytic models.

Your to-do list

Here is what you need to do to complete this week/lesson:

1. Read this document.
2. Read and respond to two articles: Serghiou & Goodman 2019 and Murad et al 2010. Respond in the discussion forum.
3. Optional: Complete the meta-analysis practice exercise using Microsoft Excel.

Acknowledgments

Many portions of this document were taken and/or adopted from the resources created by Dr. Annie Fox for the MGHIHP course HE802 taught in 2018.

Scenario 1: Perfect conditions

We decide to read each of the studies very carefully. We learn that, coincidentally enough, all of them featured the exact same research methods. All of them were conducted at the same time of year, in the same lab, over the same duration of time, using the same amount of fertilizer, using the same number of plants in the treatment and control group, using a single and identical type of plant, having the same levels of potential error as far as anybody can think, and so on.

In this extremely unlikely scenario, it would be satisfactory to conduct a quantitative meta-analysis by simply taking the average of the effect sizes found in the studies.

Scenario 2: Homogeneous study populations (fixed effect)

Now let’s consider a slightly more realistic situation. We read all four studies and we see that all of them were conducted on the same type and subtype of plant. They also all use similar experimental methods, including the duration of the experiment; the timings and quantitites of water, light, and fertilizer given to each plant; etc. However, the studies each have different sample sizes and therefore different variances for the estimates of their effect sizes. In other words, the studies with large sample sizes have lower variance and therefore smaller confidence intervals about the effect of fertilizer on plants. And the studies with smaller sample sizes have larger variances and larger confidence intervals.

In this scenario, we could not just take the simple average of the four studies and report that as our meta-analysis result. Instead, we need to do what is called a fixed effect meta-analysis, which is the right type of meta-analysis to conduct if the population studied in all of the studies that you want to meta-analyze is the same (in this case, the same type of plant), but the sample sizes are different. To do a fixed effect meta-analysis, all observations from all studies should be drawn from the same population. To account for this difference in sample sizes, we will use the help of computer software to over-weight the studies that have larger sample sizes.

When we decide to do a fixed effect analysis, we are making the assumption that there is a single true answer to our research question. In other words, we are assuming that there is a single true difference in growth between plants (of a particular type) that receive fertilizer compared to those that do not.

Scenario 3: Heterogeneous study populations (random effects)

Even more realistically, if you look across all of the studies that examine how fertilizer affects plant growth, the actual type of plant used in the experiment may vary. Let’s imagine that’s what we found when we looked closely at these four studies. The research question was the same for each study, but the type of plant used in each study was different.

In this scenario, where the observations (the plants) are not drawn from a single, homogeneous population, we need to do what is called a random effects meta-analysis, since the observations included in the various studies have different characteristics (in this case, they are different types of plants). Unlike a fixed effect analysis, in a random effects analysis, we are not making the assumption that there is a single true effect size that all of the studies are trying to estimate. Instead, each study corresponds to its own true effect size and our goal is to simply come up with a summary measure.

Scenario 4: Non-comparable experimental conditions

Finally, in another plausible scenario, we read these four studies carefully and decide that they simply cannot be compared to each other in any way. Doing so would be irresponsible of us as researchers. In this scenario, it turns out that the studies not only use different types of plants but they also use different fertilizer; give the plants different amounts of light, water, and soil; and report plant growth over different lengths of time.

It is our responsibility as a research team to make sure that we are answering the initial research question appropriately and with the correct methods. If the four studies that we hope to meta-analyze are too different from each other, it would not be appropriate to put them together into a quantitative meta-analysis. In this scenario, the four studies are considered to be non-comparable. This determination of comparability cannot be made using a statistical test. The researchers must determine this using their own judgment.

Continuing with this scenario, since our research team still does want to report all four of these studies (even though it is not appropriate to analyze them quantitatively), we will instead choose to simply report the findings from all four studies in a table, without calculating any quantitative summary measure.

Remember that it’s up to you and your team to practice meta-analysis the right way. Even these two agree… ^{1 2}

Serghiou & Goodman 2019

• Serghiou, S., & Goodman, S. N. (2019). Random-Effects Meta-analysis Summarizing Evidence With Caveats. JAMA, 321(3), 301-302. http://doi.org/10.1001/jama.2018.19684. https://www.aub.edu.lb/SHARP/PublishingImages/Pages/publications/jama_serghiou_2018.pdf. (Also available in D2L)

Try to answer the following questions as you read:

1. Do you agree with the decision to use a random effects model rather than a fixed effect model for the opioid meta-analysis described? Why or why not?
2. Under what circumstances would a fixed effect model have been appropriate?
3. Under what circumstances would no type of meta analysis have been appropriate? (Think of a modification to the current circumstances that would render random effects meta-analysis unusable).

Have a look at this guide to understanding heterogeneity in meta-analysis:

• Identifying and measuring heterogeneity. Cochrane Handbook for Systematic Reviews of Interventions. https://handbook-5-1.cochrane.org/chapter_9/9_5_2_identifying_and_measuring_heterogeneity.htm.
• Optional: Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327(7414), 557-560. https://doi.org/10.1136/bmj.327.7414.557.

Now answer the following questions:

1. Why do we care about the level of heterogeneity in a quantitative meta-analysis?
2. In the opioid meta-analysis featured in the Serghiou and Goodman article, how should we interpret the results, given the reported levels of heterogeneity?

Murad et al 2010

Read the following article and then respond to the discussion questions. Please submit your responses in the Week 12 discussion forum in D2L.

• Murad, M. H., Coto-Yglesias, F., Varkey, P., Prokop, L. J., & Murad, A. L. (2010). The effectiveness of self-directed learning in health professions education: a systematic review. Medical education, 44(11), 1057-1068. https://doi.org/10.1111/j.1365-2923.2010.03750.x. https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1365-2923.2010.03750.x. (Also in D2L).

Try to answer the following questions as you read:

1. What research question is this article trying to answer?
2. According to you, did the authors choose a good statistical method to use given the studies they wanted to meta-analyze? Do you have any other feedback about the methodology they used?
3. What do you consider to be the most important quantitative finding of this study?
4. Forest plots are usually used to display the results of a quantitative meta-analysis. What can you glean about the studies in this meta-analysis just by looking at the forest plots?
5. What is/are the answer(s) to this article’s research questions? Address if and how these answers are taken directly from the quantitative meta analysis.
6. How high or low is the heterogeneity in this study? How do you know?

Prepare the data and file

1. Download the Excel file called “Meta-Analysis Tool.xlsm” in D2L. You may need to click “Enable Content” within the file after opening it.
2. Go to the sheet called “Original two groups”, which will help you do a meta analysis of your own. It has been populated with the data from Part 1 of the example. To make things easy, we assume that the mean growth in the control group is always 0. If you look a the “Difference between means” column, you will see the same effect sizes that are listed above in Part 1. To start, all groups have been given a standard deviation of 0.5 units³, with a group size of 60 plants. This initial data is a copy of Scenario 1 from above. All study details are the same across the four studies, except for the effect sizes found.

Analyze scenario 1

1. You should select “Fixed effect” as the model type in the bottom left corner. What overall effect size do you find? How does this compare to the simple average of the four individual effect sizes?
2. What level of heterogeneity do you find? Report both Q and I². Is this high or low?
3. Change the standard deviation in all groups to 0.25. How does this change the results? Make sure you pay attention to the confidence intervals all the way to the right. How does this change the shape of the diamond at the bottom of the forest plot?
4. Change the sample size (number of plants) in all groups from 60 to 10. How does this change the results?

Analyze scenario 2

1. In scenario 1, all of the group characteristics are identical to each other. In scenario 2, the experimental designs of the four studies are all identical replications of each other (so keep the model type on “Fixed effect”), but the sample sizes and standard deviations vary.
2. What happens when you vary the sample sizes of the groups across the four studies? Also pay attention to the heterogeneity measures at the bottom as you make these changes.
3. Now vary the standard deviations and report what happens.
4. What causes the shape of the diamond at the bottom and the size of the overall confidence interval to vary?

Analyze scenario 3

1. Now switch the model type to “Random effects.” What changes do you see right away? Make sure you’re still looking at those heterogeneity statistics!
2. What assumptions are we making about the four studies in scenario 3 that we were not making before?
3. Like before, vary the sample sizes and standard deviations. What did you learn about random effects meta-analysis by doing this?

Analyze scenario 4

Just kidding. Remember: scenario 4 is the one in which we decided that the four studies were not suitably enough comparable to each other to do a meta-analysis.

Reading

• Yu Y, Fu R, Wagner J, Ahmed A, Smith C, Chou R. Developing and Piloting a Tool To Create Dot Plots To Summarize Pooled Data for Multiple Outcomes in Systematic Reviews. Methods Research Report. (Prepared by the Pacific Northwest Evidence-based Practice Center under Contract No. 75Q80120D00006. AHRQ Publication No. 22-EHC001. Rockville, MD: Agency for Healthcare Research and Quality. October 2021. doi: 10.23970/AHRQEPCMETHODSDOTPLOTS
• Common mistakes in Meta-Analysis and How to Avoid Them Fixed-effect vs. Random-effects. https://www.meta-analysis-workshops.com/download/common-mistakes2.pdf.
• Sections 16.5–16.8 in: The Pennsylvania State University. 2018. Stat 509: Design and Analysis of Clinical Trials. https://newonlinecourses.science.psu.edu/stat509/node/142/.
• Sections 4.1 and 4.2 in: Harrer, M., Cuijpers, P., Furukawa, T.A, & Ebert, D. D. 2019. Doing Meta-Analysis in R: A Hand-on Guide. https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/. Ignore the calculations and coding parts; just focus on the conceptual portions for now.
• Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327(7414), 557-560. https://doi.org/10.1136/bmj.327.7414.557.
• Cook, D. A., et al. (2011). Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. Jama, 306(9), 978-988. http://doi.org/10.1001/jama.2011.1234. https://med.virginia.edu/medical-simulation-center/wp-content/uploads/sites/254/2016/01/2011JAMA_Cook_Meta-analysis.pdf.
• Herbison, P., Hay-Smith, J., & Gillespie, W. J. (2006). Adjustment of meta-analyses on the basis of quality scores should be abandoned. Journal of clinical epidemiology, 59(12), 1249-e1. https://doi.org/10.1016/j.jclinepi.2006.03.008.

Videos

• Lecture 8A: Fixed Effect Model. Introduction to Systematic Review and Meta-Analysis. Coursera. https://www.coursera.org/lecture/systematic-review/lecture-8a-fixed-effect-model-sXL9J.
• Lecture 8B: Random Effects Model. Introduction to Systematic Review and Meta-Analysis. Coursera. https://www.coursera.org/lecture/systematic-review/lecture-8b-random-effects-model-uqNFi.

Types of reviews

• Narrative review: descriptive, not systematic; describes a set of articles selected by the author; author describes the findings of different studies, but may not capture the full scope of the literature
• Systematic review: Comprehensive, detailed, a priori search strategy developed; requires a team of individuals (minimum of 2, ideally more than that); synthesizes the findings from similar studies Systematic reviews can stand on their own, or they may be presented in conjunction with a meta-analysis.
• Meta-analysis: Comprehensive, detailed, a priori search strategy developed, uses quantitative methods to statistically summarize research; requires a team of individuals (minimum of 2, ideally more than that); Meta-analyses synthesize the findings of multiple studies into an overall effect size.
• A note about terminology – in medicine, systematic reviews (usually) contain meta-analytic findings, but are only referred to as “systematic reviews.” This is not the case in other fields (for example, in Psychology, if an article is called a systematic review, it wouldn’t necessarily contain meta-analytic findings).

Meta-analysis data analysis and presentation

• Fixed Effect Model: used when the studies that are being compared are homogenous—they are essentially replications, and use similar methodology and similar variables. They likely have overlapping confidence intervals, and the assumption is that all studies are estimating the same population parameter.
• Random Effects Model: Used when the studies that are being compared are heterogeneous. For example, the studies may use different samples of participants—they may vary on age, gender, race, etc. Thus, the effect may not be the same across all of these different factors. The Random Effects model It assumes that the studies are estimating somewhat different population parameters, and accounts for this by using weighing the studies accordingly in the calculation of the overall effect size/confidence interval.
• Forest plots are used to visually display the confidence intervals for comparable studies included in a meta-analysis. The bottom of the forest plot contains the results of the meta-analysis: a confidence interval based on the combined findings of the studies that have been included.
• Effect size: The predicted effect of the independent variable on the dependent variable. Also the difference between the treatment and control groups in the measured outcome of interest (dependent variable).