Strong publication bias in early intervention studies
About
Publication bias is when scientists for whatever reason don’t publish all findings they get. Usually, this is towards larger effect, but in a few cases scientists prefer to find smaller effects. It depends on the political climate and their own beliefs. Since social scientists are overwhelmingly left-wing, we can thus expect very strong publication bias effects left-wing people like, such as early intervention studies. Here this question is analyzed.
Data are from: Duncan and Magnuson’s Investing in Preschool Programs
Note that for some reason, the authors winsorised the data, so the results would probably be more extreme than those seen here.
Initialize
#pkgs
library(pacman)
p_load(kirkegaard, readr, metafor)
#data
d = read.csv("data.csv", encoding="UTF-8")
colnames(d) = c("Name", "Year", "d", "reciproc.var", "Headstart")Recode a bit
#recode headstart
d$Headstart = plyr::mapvalues(d$Headstart, from = c(1,0), to = c("Headstart", "Other"))
#names of specific studies
d$Names2 = rep("", nrow(d))
d[42,"Names2"] = "Karnes"
d[39,"Names2"] = "Perry"
d[53,"Names2"] = "Abecedarian"
#variance
d$var = d$reciproc.var^-1Analyze
#meta
RE_main = rma(d$d, vi = d$var)
RE_main##
## Random-Effects Model (k = 84; tau^2 estimator: REML)
##
## tau^2 (estimated amount of total heterogeneity): 0.0179 (SE = 0.0075)
## tau (square root of estimated tau^2 value): 0.1337
## I^2 (total heterogeneity / total variability): 38.32%
## H^2 (total variability / sampling variability): 1.62
##
## Test for Heterogeneity:
## Q(df = 83) = 137.9876, p-val = 0.0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## 0.2510 0.0263 9.5456 <.0001 0.1995 0.3026 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#forest
GG_forest(RE_main)
ggsave("figures/forest.png")## Saving 7 x 5 in image#funnel
GG_funnel(RE_main)
ggsave("figures/funnel.png")## Saving 7 x 5 in image#trim and fill
trimfill(RE_main, estimator = "L0")##
## Estimated number of missing studies on the left side: 21 (SE = 5.9875)
##
## Random-Effects Model (k = 105; tau^2 estimator: REML)
##
## tau^2 (estimated amount of total heterogeneity): 0.0514 (SE = 0.0136)
## tau (square root of estimated tau^2 value): 0.2267
## I^2 (total heterogeneity / total variability): 60.75%
## H^2 (total variability / sampling variability): 2.55
##
## Test for Heterogeneity:
## Q(df = 104) = 233.9445, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## 0.1764 0.0322 5.4749 <.0001 0.1133 0.2396 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1trimfill(RE_main, estimator = "R0")##
## Estimated number of missing studies on the left side: 9 (SE = 4.4721)
## Test of H0: no missing studies on the left side: p-val = 0.0010
##
## Random-Effects Model (k = 93; tau^2 estimator: REML)
##
## tau^2 (estimated amount of total heterogeneity): 0.0272 (SE = 0.0094)
## tau (square root of estimated tau^2 value): 0.1650
## I^2 (total heterogeneity / total variability): 46.65%
## H^2 (total variability / sampling variability): 1.87
##
## Test for Heterogeneity:
## Q(df = 92) = 182.9974, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## 0.2241 0.0285 7.8708 <.0001 0.1683 0.2799 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1trimfill(RE_main, estimator = "Q0")##
## Estimated number of missing studies on the left side: 30 (SE = 8.8857)
##
## Random-Effects Model (k = 114; tau^2 estimator: REML)
##
## tau^2 (estimated amount of total heterogeneity): 0.0625 (SE = 0.0150)
## tau (square root of estimated tau^2 value): 0.2499
## I^2 (total heterogeneity / total variability): 64.93%
## H^2 (total variability / sampling variability): 2.85
##
## Test for Heterogeneity:
## Q(df = 113) = 271.3970, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## 0.1430 0.0328 4.3603 <.0001 0.0787 0.2073 ***
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1#scatter
GG_scatter(d, "reciproc.var", "d", case_names_vector = "Names2", check_overlap = F) +
ylab("Effect size (d)") +
xlab("Precision (1/var; with cutoff)")
ggsave("figures/scatter.png")## Saving 7 x 5 in imagePrint dataset
d## Name
## 1 Evaluation of the effect of Head Start program on cognitive growth of disadvantaged children
## 2 National Head Start and Summer Head Start program, 1965-1968
## 3 Summer Head Start in Kearney, NE
## 4 Summer Head Start in San Jose, CA
## 5 Summer Head Start in Cambridge, MA
## 6 Head Start in Miami, FL
## 7 Summer Head Start in Duluth, MN
## 8 Impact of 1965 Summer Head Start on children's concept attainment by Allerhand
## 9 Summer Head Start in Lincoln, NE, matched pairs
## 10 Summer Head Start in Lincoln, NE, unmatched pairs
## 11 Bicultural preschool programmed Head Start
## 12 Summer Head Start in Washington, DC
## 13 Head Start effects on children's behavior and cognitive functioning one year later by Nummedal and Stern
## 14 Experiments in Head Start and early education: Direct instruction in Head Start vs. No preschool
## 15 Experiments in Head Start and early education: Enrichment in Head Start vs. No preschool
## 16 Head Start in rural areas, MN
## 17 Comparison of children enrolled in Head start or no preschool in two cities
## 18 Planned Variation in Head Start, Head Start with curricula vs. no Head Start
## 19 Planned Variation in Head Start, standard Head Start vs. no Head Start
## 20 Head Start in New Haven, CT
## 21 Head Start in New Haven, CT
## 22 Head Start Bilingual Bicultural Development Project, bilingual Head Start vs. Stay at home
## 23 Head Start Bilingual Bicultural Development Project, standard Head Start vs. Stay at home
## 24 Head Start in New Haven, CT
## 25 Head Start in Guam
## 26 Early Head Start Research and Evaluation Project
## 27 ECLS-K Head Start Study, white
## 28 ECLS-K Head Start Study, African-American
## 29 ECLS-K Head Start Study, Hispanic
## 30 Southeastern Head Start program of high quality
## 31 National Head Start Impact Study First Year, 3-year-old cohort
## 32 National Head Start Impact Study First Year, 4-year-old cohort
## 33 NYC Early Head Start, 12 months of younger
## 34 NYC Early Head Start, older than 12 months
## 35 Early Training Project
## 36 Preschool intervention with disadvantaged children in early 1960s Cambridge, Pre-k & responsive environments
## 37 Preschool intervention with disadvantaged children in early 1960s Cambridge, Pre-k
## 38 Preschool intervention with disadvantaged children in early 1960s Cambridge, responsive environments
## 39 The Perry Preschool
## 40 Howard University Preschool Program
## 41 6-month Full-Day Preschool for 3- to 4-year-olds with curriculum emphasis on language, social skills, & health
## 42 Karnes Ameliorative Curriculum Pre-K
## 43 NY Pre-K for disadvantaged children
## 44 Preschool in Fremont, CA
## 45 Urbana Infant Home Tutoring Program
## 46 Greensboro Demonstration Nursery Center
## 47 Project Know-How
## 48 Yale Child Welfare Research Program
## 49 Home Oriented Preschool Education, TV & Home visit & Mobile classroom vs. control
## 50 Home Oriented Preschool Education, TV & Home visit & Early childhood classroom vs. control, 1969
## 51 Home Oriented Preschool Education, TV & Home visit & Early childhood classroom vs. control, 1970
## 52 Syracuse University Family Development & Research
## 53 Abecedarian Project
## 54 Pittsfield, MA Parent-Child Home Program, 1978
## 55 Pittsfield, MA Parent-Child Home Program, 1983
## 56 Infants in community-based day care
## 57 Project CARE
## 58 Field Teacher Mother Parent Training
## 59 BYU Pre-school program effects on Low-Risk Children
## 60 Child care from infancy to 3 years of age
## 61 Children of Poverty: A multi-level analysis of the determinants of intellectual development
## 62 Infant Health and Development Program
## 63 Chicago Parent Center
## 64 Even Start National Evaluation
## 65 North Carolina Pre-K Evaluation, Chapter 1 preschool
## 66 North Carolina Pre-K Evaluation, Family-focused preschool
## 67 North Carolina Pre-K Evaluation, Community daycare
## 68 Comprehensive Child Development Program
## 69 The NICHD Study of Early Child Care and Youth Development
## 70 Kentucky Educational Reform Act (KERA) Preschool Program, cohort 1
## 71 Kentucky Educational Reform Act (KERA) Preschool Program, cohort 3
## 72 Georgia Pre-K Program
## 73 Michigan School Readiness Program Longitudinal Evaluation
## 74 Charlotte Bright Beginnings Pre-K Evaluation
## 75 Center-based language intervention program
## 76 Parent-child intervention for depressed mothers
## 77 Literature-Math-Scaffolding Intervention
## 78 Third Even Start evaluation
## 79 Utah Pre-K Kindergarten Readiness Program
## 80 Five State Pre-K RD Evaluation, Michigan
## 81 Five State Pre-K RD Evaluation, New Jersey
## 82 Five State Pre-K RD Evaluation, Kolahoma
## 83 Five State Pre-K RD Evaluation, South Carolina
## 84 Five State Pre-K RD Evaluation, West Virginia
## Year d reciproc.var Headstart Names2 var
## 1 1963 0.54 15.13 Headstart 0.06609385
## 2 1965 0.10 53.46 Headstart 0.01870557
## 3 1965 0.08 6.08 Headstart 0.16447368
## 4 1965 0.42 14.34 Headstart 0.06973501
## 5 1965 0.14 11.03 Headstart 0.09066183
## 6 1965 0.79 9.63 Headstart 0.10384216
## 7 1965 0.38 14.93 Headstart 0.06697924
## 8 1965 0.46 13.28 Headstart 0.07530120
## 9 1965 0.03 71.07 Headstart 0.01407063
## 10 1965 -0.06 3.04 Headstart 0.32894737
## 11 1966 0.74 6.29 Headstart 0.15898251
## 12 1966 -0.07 35.40 Headstart 0.02824859
## 13 1967 -0.13 13.50 Headstart 0.07407407
## 14 1968 0.47 12.14 Headstart 0.08237232
## 15 1968 0.40 13.25 Headstart 0.07547170
## 16 1968 0.70 15.93 Headstart 0.06277464
## 17 1969 0.33 93.45 Headstart 0.01070091
## 18 1969 0.32 10.13 Headstart 0.09871668
## 19 1969 0.27 23.96 Headstart 0.04173623
## 20 1971 0.55 12.91 Headstart 0.07745933
## 21 1971 0.39 25.94 Headstart 0.03855050
## 22 1979 0.29 16.18 Headstart 0.06180470
## 23 1979 0.33 6.99 Headstart 0.14306152
## 24 1979 0.55 17.33 Headstart 0.05770340
## 25 1985 0.02 19.95 Headstart 0.05012531
## 26 1996 0.13 100.00 Headstart 0.01000000
## 27 1997 -0.06 100.00 Headstart 0.01000000
## 28 1997 0.05 100.00 Headstart 0.01000000
## 29 1997 -0.04 39.42 Headstart 0.02536783
## 30 1998 0.35 24.43 Headstart 0.04093328
## 31 2002 0.26 47.99 Headstart 0.02083767
## 32 2002 0.25 100.00 Headstart 0.01000000
## 33 2002 -0.34 1.30 Headstart 0.76923077
## 34 2002 0.23 0.61 Headstart 1.63934426
## 35 1962 1.04 8.47 Other 0.11806375
## 36 1962 -0.01 11.42 Other 0.08756567
## 37 1962 0.14 9.36 Other 0.10683761
## 38 1962 0.00 8.72 Other 0.11467890
## 39 1962 1.10 9.23 Other Perry 0.10834236
## 40 1964 0.47 20.41 Other 0.04899559
## 41 1964 0.74 10.38 Other 0.09633911
## 42 1965 1.60 1.35 Other Karnes 0.74074074
## 43 1965 0.08 100.00 Other 0.01000000
## 44 1966 -0.19 7.29 Other 0.13717421
## 45 1966 1.10 6.40 Other 0.15625000
## 46 1967 0.81 7.29 Other 0.13717421
## 47 1967 0.65 13.16 Other 0.07598784
## 48 1968 1.22 7.88 Other 0.12690355
## 49 1968 0.15 2.88 Other 0.34722222
## 50 1968 0.38 6.20 Other 0.16129032
## 51 1968 0.67 5.53 Other 0.18083183
## 52 1969 0.46 100.00 Other 0.01000000
## 53 1972 0.90 16.47 Other Abecedarian 0.06071645
## 54 1974 0.36 16.41 Other 0.06093845
## 55 1974 0.86 8.35 Other 0.11976048
## 56 1976 0.39 1.34 Other 0.74626866
## 57 1978 0.44 4.27 Other 0.23419204
## 58 1979 0.62 17.29 Other 0.05783690
## 59 1980 0.48 18.09 Other 0.05527916
## 60 1981 0.01 5.71 Other 0.17513135
## 61 1983 0.74 29.98 Other 0.03335557
## 62 1984 0.26 76.74 Other 0.01303101
## 63 1985 0.17 93.85 Other 0.01065530
## 64 1991 0.04 27.13 Other 0.03685957
## 65 1991 0.10 47.52 Other 0.02104377
## 66 1991 0.13 18.29 Other 0.05467469
## 67 1991 0.49 20.06 Other 0.04985045
## 68 1991 0.03 100.00 Other 0.01000000
## 69 1991 0.20 100.00 Other 0.01000000
## 70 1992 0.22 79.58 Other 0.01256597
## 71 1992 0.11 63.45 Other 0.01576044
## 72 1993 0.23 53.98 Other 0.01852538
## 73 1995 0.25 52.17 Other 0.01916810
## 74 1997 0.29 100.00 Other 0.01000000
## 75 1997 1.09 2.99 Other 0.33444816
## 76 1997 0.43 20.57 Other 0.04861449
## 77 1998 0.26 14.83 Other 0.06743088
## 78 1999 -0.19 38.91 Other 0.02570033
## 79 2003 0.16 43.92 Other 0.02276867
## 80 2004 0.13 100.00 Other 0.01000000
## 81 2004 0.09 100.00 Other 0.01000000
## 82 2004 0.14 100.00 Other 0.01000000
## 83 2004 0.14 100.00 Other 0.01000000
## 84 2004 0.15 100.00 Other 0.01000000