Introduction.

Data Overview.

This dataset include 21 studies which examine whether the chances of receiving a grant or fellowship award differs for men and women. Source: Bornmann, L., Mutz, R., & Daniel, H. (2007). Gender differences in grant peer review: A metaanalysis. Journal of Informetrics, 1(3), 226–238. https://doi.org/10.1016/j.joi.2007.03.001.

The data consists of 13 variables:

  • study (character) study reference

  • obs (numeric) observation within study

  • doctype (character) document type

  • gender (character) gender of the study authors

  • year (numeric) (average) cohort year

  • org (character) funding organization / program

  • country (character) country of the funding organization / program

  • type (character) fellowship or grant application

  • discipline (character) discipline / field

  • waward (numeric) number of women who received a grant/fellowship award

  • wtotal (numeric) number of women who applied for an award

  • maward (numeric) number of men who received a grant/fellowship award

  • mtotal (numeric) number of men who applied for an award

Methods

Many studies provide multiple comparisons (e.g., for different years / cohorts / disciplines), so a multilevel meta-analytic model is be used to account for the multilevel structure in these data. Metafor pacakge was used to conduct the analysis.

The Analysis

Import data

The data is publically available at metafor package.

library(metafor)
dat <- dat.bornmann2007
head(dat)
##           study obs doctype gender year  org country       type
## 1 Ackers (2000)   1    Grey    M&F 1996 MSCA  Europe Fellowship
## 2 Ackers (2000)   2    Grey    M&F 1996 MSCA  Europe Fellowship
## 3 Ackers (2000)   3    Grey    M&F 1996 MSCA  Europe Fellowship
## 4 Ackers (2000)   4    Grey    M&F 1996 MSCA  Europe Fellowship
## 5 Ackers (2000)   5    Grey    M&F 1996 MSCA  Europe Fellowship
## 6 Ackers (2000)   6    Grey    M&F 1996 MSCA  Europe Fellowship
##                  discipline waward wtotal maward mtotal
## 1         Physical Sciences    139    711    274   1029
## 2         Physical Sciences     45    258    166    908
## 3         Physical Sciences     44    236    219    928
## 4         Physical Sciences     63    251     96    507
## 5 Social Sciences / Biology    157    910    252   1118
## 6         Physical Sciences    114    589    460   2244

Pooling the effect size

The effect size is calculated as log odds ratio.

dat <- escalc(measure="OR", ai=waward, n1i=wtotal, ci=maward, n2i=mtotal, data=dat)
dat <- as.data.frame(dat)
head(dat)
##           study obs doctype gender year  org country       type
## 1 Ackers (2000)   1    Grey    M&F 1996 MSCA  Europe Fellowship
## 2 Ackers (2000)   2    Grey    M&F 1996 MSCA  Europe Fellowship
## 3 Ackers (2000)   3    Grey    M&F 1996 MSCA  Europe Fellowship
## 4 Ackers (2000)   4    Grey    M&F 1996 MSCA  Europe Fellowship
## 5 Ackers (2000)   5    Grey    M&F 1996 MSCA  Europe Fellowship
## 6 Ackers (2000)   6    Grey    M&F 1996 MSCA  Europe Fellowship
##                  discipline waward wtotal maward mtotal          yi         vi
## 1         Physical Sciences    139    711    274   1029 -0.40107542 0.01391663
## 2         Physical Sciences     45    258    166    908 -0.05726822 0.03428886
## 3         Physical Sciences     44    236    219    928 -0.29852194 0.03391225
## 4         Physical Sciences     63    251     96    507  0.36093779 0.03404192
## 5 Social Sciences / Biology    157    910    252   1118 -0.33336360 0.01282044
## 6         Physical Sciences    114    589    460   2244 -0.07172953 0.01361164

3-leval meta-analysis model fitting.

To fit a multi-level model, we have to specify the following parameters.

  • yi The name of the column in our data set which contains the calculated effect sizes.

  • Vi The name of the column in our data set which contains the variance of the calculated effect sizes.

  • slab The name of the column in our data set which contains the study labels.

  • data The name of the data set.

  • test The test we want to apply for our regression coefficients. We can choose from "z" (default) and "t" (recommended; uses a test similar to the Knapp-Hartung method).

  • method The method used to estimate the model parameters. Both "REML" (recommended; restricted maximum-likelihood) and "ML" (maximum likelihood) are possible.

res <- rma.mv(yi, vi, random = ~ 1 | study/obs, data=dat, test = "t")
res
## 
## Multivariate Meta-Analysis Model (k = 66; method: REML)
## 
## Variance Components:
## 
##             estim    sqrt  nlvls  fixed     factor 
## sigma^2.1  0.0161  0.1268     21     no      study 
## sigma^2.2  0.0038  0.0613     66     no  study/obs 
## 
## Test for Heterogeneity:
## Q(df = 65) = 221.2850, p-val < .0001
## 
## Model Results:
## 
## estimate      se     tval  df    pval    ci.lb    ci.ub 
##  -0.1010  0.0417  -2.4196  65  0.0183  -0.1843  -0.0176  * 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The model results consist of 3 parts:

First, the variance components in which the random-effects variances calculated for each level of our model. Sigma^2.1 shows variance at level 3 or betweeen-study variance while sigma^2.2 shows variance at level 2 or within-studies variance. The number of studies is 21 with 66 effect sizes.

Second, The test of heterogeneity part. This part shows the overall heterogeneity, however this is informative as we need to quantify the heterogeneity at the 3 levels. We can claculate it using dmetar package

library(dmetar)
## Extensive documentation for the dmetar package can be found at: 
##  www.bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/
i2 <- var.comp(res)
summary(i2)
##         % of total variance    I2
## Level 1            24.03213   ---
## Level 2            14.36985 14.37
## Level 3            61.59802  61.6
## Total I2: 75.97%

The output shows that the heterogeneity variance at level 3 = 61.6 and at level 2 = 14.37. We also can plot the distribution of the total variance.

plot(i2)

Third, the model results shows the effect size estimate with confidence intervals. The mean log odds ratio = - 0.1, P value = 0.016. That means that the odds women to receive a grant or felloship is lesser than men.

Subgroup multi-level analysis

Let's test for a difference between fellowship and grant applications as well as difference between countries

res2 <- rma.mv(yi, vi, mods = ~ type + country, random = ~ 1 | study/obs, data=dat, test = "t")
res2
## 
## Multivariate Meta-Analysis Model (k = 66; method: REML)
## 
## Variance Components:
## 
##             estim    sqrt  nlvls  fixed     factor 
## sigma^2.1  0.0023  0.0479     21     no      study 
## sigma^2.2  0.0031  0.0559     66     no  study/obs 
## 
## Test for Residual Heterogeneity:
## QE(df = 60) = 106.6330, p-val = 0.0002
## 
## Test of Moderators (coefficients 2:6):
## F(df1 = 5, df2 = 60) = 5.1969, p-val = 0.0005
## 
## Model Results:
## 
##                        estimate      se     tval  df    pval    ci.lb   ci.ub 
## intrcpt                 -0.1416  0.0998  -1.4189  60  0.1611  -0.3412  0.0580 
## typeGrant                0.1390  0.0541   2.5699  60  0.0127   0.0308  0.2471 
## countryCanada           -0.0774  0.1258  -0.6152  60  0.5408  -0.3290  0.1742 
## countryEurope           -0.1125  0.1001  -1.1235  60  0.2657  -0.3127  0.0878 
## countryUnited Kingdom    0.0789  0.1149   0.6863  60  0.4951  -0.1510  0.3087 
## countryUnited States     0.0581  0.0992   0.5859  60  0.5601  -0.1403  0.2565 
##  
## intrcpt 
## typeGrant              * 
## countryCanada 
## countryEurope 
## countryUnited Kingdom 
## countryUnited States 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The test of moderators shows that that F5 = 25.98 , with p < 0.001.This means that there is significant difference between the subgroups. The estimates table shows only significant with application type with the effect in grant type is -0.2806.

Publication Bias.

1- Funnel plot

The funnel plot shows that studies are slightly asymmetrically distributed, so publication bias could exist.

funnel.rma(res, label = F)

2- Egger's regression test.

There is no direct function to conduct Egger's test for multi-level model. Alternatively, we calculate it by using the standard errors of the effect size estimates as a predictor in the meta-regression.

res3 <- rma.mv(yi, vi, mods = ~ sqrt(vi) , random = ~ 1 | study/obs, data=dat, test = "t")
res3
## 
## Multivariate Meta-Analysis Model (k = 66; method: REML)
## 
## Variance Components:
## 
##             estim    sqrt  nlvls  fixed     factor 
## sigma^2.1  0.0165  0.1283     21     no      study 
## sigma^2.2  0.0039  0.0626     66     no  study/obs 
## 
## Test for Residual Heterogeneity:
## QE(df = 64) = 186.5724, p-val < .0001
## 
## Test of Moderators (coefficient 2):
## F(df1 = 1, df2 = 64) = 0.6776, p-val = 0.4135
## 
## Model Results:
## 
##           estimate      se     tval  df    pval    ci.lb   ci.ub 
## intrcpt    -0.0661  0.0597  -1.1087  64  0.2717  -0.1853  0.0530    
## sqrt(vi)   -0.2151  0.2613  -0.8232  64  0.4135  -0.7372  0.3070    
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The coefficient of the standard error was insignificant (r=-0.2, P= 0.41). The results mean that studies are symmetrically distributed, and publication bias are absent which contraindicates the funnel plot.