#Packages
library(pacman); p_load(DT)
datatable(data, extensions = c("Buttons", "FixedColumns"), options = list(dom = 'Bfrtip', buttons = c('copy', 'csv', 'print'), scrollX = T, fixedColumns = list(leftColumns = 3)))Harrison et al. (2021) evaluated the Greater Male Variability Hypothesis (GMVH) in a meta-analysis of 220 different animal species. The idea behind their analysis was to provide evidence regarding the GMVH. They ultimately yielded inferences regarding humans from their results. Some of the statements made by the lead author can be read here: https://www.anu.edu.au/news/all-news/sexist-%E2%80%9Csexplanation%E2%80%9D-for-men%E2%80%99s-brilliance-debunked.
Unfortunately, what they did was not appropriate to draw even mild negative inferences. I previously provided data relevant to this issue for human differences in the variances of cognitive ability (https://rpubs.com/JLLJ/SDVR). At the end of that page, I provided a simulation based on the empirical variances referenced on that page. The simulation showed that the confidence intervals for variance ratios were very large. In other words, large samples are required to detect differences of the magnitudes observed in humans. In the case of humans, where the variance ratio was around 1.16 (higher = greater variance for males), samples numbering over a thousand for each sex would have been required.
When it comes to using animal samples to draw negative inferences regarding human samples, there are a number of issues. One of these is a conceptual one that cannot be overcome without an amount of data and analysis that is not feasible while the other regards power.
Regarding the conceptual issue, one cannot draw negative inferences regarding human differences from animal data; one can support human differences with animal data because that supports biological generality, whereas biological generality is not required for humans to differ. The presence of a phenomenon in humans can be supported by animal data but the absence of the same phenomenon in animals does not provide evidence of its absence in humans. Humans have rockets, ants do not. One frequently empirically examinde arena in which this could not be more clear is pharmacology, wherein medicines that work very well in humans often do not work at all in mice and, sometimes, vice-versa. Sometimes, however, the mechanism for something that works in humans can be verified in animal models.
The second issue - power - is what I will address here. As previously mentioned, samples of greater than 1000 per group are needed to address human sex differences in variances. Imbalanced samples and violations of measurement invariance can also have an impact in this regard. In Harrison et al.’s dataset, their largest sample had a male n of 1611 and a female n of 1503; the second-largest had 761 and 872. That is, only one sample had a sufficient n to evaluate differences on the scale of human sex differences in cognitive ability. For the total dataset, the mean male n was 29.72 with a median of 20 and an SD of 52.20 versus female values of 29.01, 19, and 52.37. Male and female n’s correlated at r = 0.966. All-in-all, there was virtually no power to provide any information with respect to human-sized differences.
It could be argued that meta-analysis overcomes this issue by aggregating a great deal of small data. This argument is faulty; it involves the assumption of a lack of meaningful publication bias and that the individual estimates are centered around a true mean (or mean distribution). Moreover, we are supposed to take for granted that the aggregation of many unreliable or useless estimates is something meaningful. In a recent analysis by Knarven, Stroemland & Johannesson (2020), they observed that meta-analytic effects usually greatly overestimated effect sizes when compared with preregistered replication projects. Garbage in, garbage out.
Regarding the second assumption, it can hardly be considered meaningful. I am not aware of anyone who believes in a universal pattern of sex-related variability. There are numerous theories explaining particular differences with reference to, for example, mammal-specific hormones, which would fail to explain any species’ pattern among invertebrates. Male and female denote relative gamete sizes - anisogamy - but what is typically proffered as a driver of greater male variability is the role that comes with smaller gametes, which is to say, mobility during gestation. Alterations to expectations can occur for many reasons like differences in mating structure (which the authors included a variable for), but the same mating system might be proposed to have different effects in different species. For example, in lekking birds, we might expect relatively GMV compared to closely-related species with monogamy, but not the same for multiple pairing when it occurs for reasons related to the harshness of extant ecology. Why in the world would an analysis of many diverse species provide inferential value for the GMVH when it is not a hypothesis about all species and the selection of species was not based on the idea of the GMVH being relevant - and, importantly, this must also be directionally relevant or the data ought to be analyzed with that taken into account! - for all of the included species?
There is another major problem when it comes to the analysis of animal behavior: the reliability of animal measurements is amazingly poor. Take chimpanzee tests, since these have some of the highest validities of animal cognitive assessments, at test-retest reliabilities around 0.5. This would be classified as “extremely poor” for humans. Moreover, that is for a test battery; as a stylized fact, questionnaires and instruments that afford the opportunity for consistent criteria for measurement are far more reliable than behavioral measurements. If we are generous and say that the included species in toto - virtually all of whom have vastly less reliable measurements than those for chimpanzees - have measurement reliabilities of 0.5, and we are also generous to the questionnaire-behavior difference and say reliability is 85% as large as an actual test, we might say that the reliability is around 0.425. Using this wildly inaccurate estimate, the variances for estimates will be considerably larger, which means much more chance to capitalize on.
To illustrate some of the unreliability issue with Harrison et al.’s analysis for inferences regarding humans and even for inferences regarding their own samples, I have provided Bayes Factors (BFs) for the variances of their largest ten analyses, smallest ten analyses, and the ten analyses around the median male N and the median itself. Boing-Messing & Mulder (2018) provided the method to compute these. As has been noticed by many others, BFs are more related to N’s than p values. We are lucky that, in this study - with its typically very small N’s and excellent balance by sex - there is at least publicly available data. We are unlucky that the original authors did not provide BFs, but given the extremely probable unreliability of their estimates, individual of these BFs will not be useful, since it is trivial to generate a large gap with unreliable estimates, and these will yield considerable BFs.
I utilized Jeffreys’ range for the evidential values of BFs as found on page 105 (Table 7.1) of Bayesian Cognitive Modeling (Lee & Wagenmakers, 2013).
#Variances for the samples sorted by N; order is males then females
## Largest
s21l <- c(1.244, 1.163)^2
s22l <- c(0.993, 1.152)^2
s23l <- c(0.412, 0.427)^2
s24l <- c(17.23, 11.865)^2
s25l <- c(18.1, 21.977)^2
s26l <- c(26.116, 27.766)^2
s27l <- c(0.04, 0.04)^2
s28l <- c(3.898, 3.655)^2
s29l <- c(6.287, 5.387)^2
s210l <- c(0.334, 0.258)^2
## Smallest
s21s <- c(0.767, 0.914)^2
s22s <- c(0.099, 0.085)^2
s23s <- c(0.071, 0.127)^2
s24s <- c(0.014, 0.071)^2
s25s <- c(0.071, 0.099)^2
s26s <- c(0.085, 0.141)^2
s27s <- c(0.042, 0.071)^2
s28s <- c(0.071, 0.085)^2
s29s <- c(0.113, 0.085)^2
s210s <- c(0.071, 0.071)^2
## Median
s21m <- c(1.438, 0.941)^2
s22m <- c(1.394, 1.048)^2
s23m <- c(1.393, 1.463)^2
s24m <- c(0.769, 0.758)^2
s25m <- c(1.003, 0.839)^2
s26m <- c(1.376, 0.860)^2
s27m <- c(1.257, 0.735)^2
s28m <- c(0.849, 0.816)^2
s29m <- c(1.218, 0.805)^2
s210m <- c(1.604, 1.465)^2
s211m <- c(1.212, 1.148)^2
#N's for the samples; same order
## Largest
nl1 <- c(1611, 1503)
nl2 <- c(761, 872)
nl3 <- c(525, 505)
nl4 <- c(525, 505)
nl5 <- c(363, 392)
nl6 <- c(353, 377)
nl7 <- c(335, 335)
nl8 <- c(314, 139)
nl9 <- c(305, 126)
nl10 <- c(253, 260)
## Smallest
ns1 <- c(2, 6)
ns2 <- c(2, 2)
## Median
nm1 <- c(20, 19)
b <- "default"
#Hypothesis 1: Male = Female; 2: Male less than Female; 3: Not male less than female
Hypotheses <- c("1=2", "1<2", "not 1<2")
#Number of Draws
SimN <- 1e5
#Prior probabilities specified such that all outcomes are equal, to entertain SD_M = SD_F.
prior.probabilities <- "default"
set.seed(1)
## Largest
lml1l <- log_marginal_likelihoods(s21l, nl1, b, Hypotheses, SimN)
lml2l <- log_marginal_likelihoods(s22l, nl2, b, Hypotheses, SimN)
lml3l <- log_marginal_likelihoods(s23l, nl3, b, Hypotheses, SimN)
lml4l <- log_marginal_likelihoods(s24l, nl4, b, Hypotheses, SimN)
lml5l <- log_marginal_likelihoods(s25l, nl5, b, Hypotheses, SimN)
lml6l <- log_marginal_likelihoods(s26l, nl6, b, Hypotheses, SimN)
lml7l <- log_marginal_likelihoods(s27l, nl7, b, Hypotheses, SimN)
lml8l <- log_marginal_likelihoods(s28l, nl8, b, Hypotheses, SimN)
lml9l <- log_marginal_likelihoods(s29l, nl9, b, Hypotheses, SimN)
lml10l <- log_marginal_likelihoods(s210l, nl10, b, Hypotheses, SimN)
## Smallest
lml1s <- log_marginal_likelihoods(s21s, ns1, b, Hypotheses, SimN)
lml2s <- log_marginal_likelihoods(s22s, ns2, b, Hypotheses, SimN)
lml3s <- log_marginal_likelihoods(s23s, ns2, b, Hypotheses, SimN)
lml4s <- log_marginal_likelihoods(s24s, ns2, b, Hypotheses, SimN)
lml5s <- log_marginal_likelihoods(s25s, ns2, b, Hypotheses, SimN)
lml6s <- log_marginal_likelihoods(s26s, ns2, b, Hypotheses, SimN)
lml7s <- log_marginal_likelihoods(s27s, ns2, b, Hypotheses, SimN)
lml8s <- log_marginal_likelihoods(s28s, ns2, b, Hypotheses, SimN)
lml9s <- log_marginal_likelihoods(s29s, ns2, b, Hypotheses, SimN)
lml10s <- log_marginal_likelihoods(s210s, ns2, b, Hypotheses, SimN)
## Median
lml1m <- log_marginal_likelihoods(s21m, nm1, b, Hypotheses, SimN)
lml2m <- log_marginal_likelihoods(s22m, nm1, b, Hypotheses, SimN)
lml3m <- log_marginal_likelihoods(s23m, nm1, b, Hypotheses, SimN)
lml4m <- log_marginal_likelihoods(s24m, nm1, b, Hypotheses, SimN)
lml5m <- log_marginal_likelihoods(s25m, nm1, b, Hypotheses, SimN)
lml6m <- log_marginal_likelihoods(s26m, nm1, b, Hypotheses, SimN)
lml7m <- log_marginal_likelihoods(s27m, nm1, b, Hypotheses, SimN)
lml8m <- log_marginal_likelihoods(s28m, nm1, b, Hypotheses, SimN)
lml9m <- log_marginal_likelihoods(s29m, nm1, b, Hypotheses, SimN)
lml10m <- log_marginal_likelihoods(s210m, nm1, b, Hypotheses, SimN)
lml11m <- log_marginal_likelihoods(s211m, nm1, b, Hypotheses, SimN)Bayes Factors!
## Largest
bayes_factors(lml1l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 201.6922 0.771727162
## [2,] 0.00495805 1.0000 0.003826262
## [3,] 1.29579474 261.3517 1.000000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.000000000 192.1439 0.770325005
## [2,] 0.005204433 1.0000 0.004009105
## [3,] 1.298153368 249.4322 1.000000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.000000000 200.2291 0.73992873
## [2,] 0.004994278 1.0000 0.00369541
## [3,] 1.351481508 270.6060 1.00000000bayes_factors(lml2l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 0.0027826829 92.75517
## [2,] 359.36541331 1.0000000000 33333.00000
## [3,] 0.01078107 0.0000300003 1.00000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 2.791440e-03 77.02917
## [2,] 358.23804484 1.000000e+00 27594.78093
## [3,] 0.01298209 3.623874e-05 1.00000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 0.0025592167 85.30552
## [2,] 390.74456503 1.0000000000 33332.66667
## [3,] 0.01172257 0.0000300006 1.00000bayes_factors(lml3l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 13.3333379 50.167057
## [2,] 0.07499997 1.0000000 3.762528
## [3,] 0.01993340 0.2657788 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 13.2029241 48.296913
## [2,] 0.07574080 1.0000000 3.658047
## [3,] 0.02070526 0.2733699 1.000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 12.9306055 49.437940
## [2,] 0.07733590 1.0000000 3.823328
## [3,] 0.02022738 0.2615523 1.000000bayes_factors(lml4l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.000000e+00 Inf 1.645979e-14
## [2,] 0.000000e+00 1 0.000000e+00
## [3,] 6.075411e+13 Inf 1.000000e+00
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.000000e+00 Inf 1.460917e-14
## [2,] 0.000000e+00 1 0.000000e+00
## [3,] 6.845015e+13 Inf 1.000000e+00
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.000000e+00 Inf 1.180787e-14
## [2,] 0.000000e+00 1 0.000000e+00
## [3,] 8.468927e+13 Inf 1.000000e+00bayes_factors(lml5l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 0.0125684075 78.54155
## [2,] 79.56457484 1.0000000000 6249.12500
## [3,] 0.01273211 0.0001600224 1.00000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.000000000 1.267587e-02 164.3384
## [2,] 78.890027818 1.000000e+00 12964.6583
## [3,] 0.006085006 7.713277e-05 1.0000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.000000000 0.0112666173 187.6437
## [2,] 88.757785690 1.0000000000 16654.8407
## [3,] 0.005329249 0.0000600426 1.0000bayes_factors(lml6l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 7.1641859 50.966321
## [2,] 0.1395832 1.0000000 7.114042
## [3,] 0.0196208 0.1405671 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 7.1110803 47.585615
## [2,] 0.14062561 1.0000000 6.691756
## [3,] 0.02101475 0.1494376 1.000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 6.892318 49.711269
## [2,] 0.14508907 1.000000 7.212562
## [3,] 0.02011616 0.138647 1.000000bayes_factors(lml7l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 23.4067359 23.453447
## [2,] 0.04272274 1.0000000 1.001996
## [3,] 0.04263766 0.9980084 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 22.952915 22.8518331
## [2,] 0.04356745 1.000000 0.9955961
## [3,] 0.04376017 1.004423 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 23.014594 22.8990051
## [2,] 0.04345069 1.000000 0.9949776
## [3,] 0.04367002 1.005048 1.0000000bayes_factors(lml8l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 29.348224 7.882521
## [2,] 0.03407361 1.000000 0.268586
## [3,] 0.12686297 3.723203 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 29.293501 7.5789816
## [2,] 0.03413726 1.000000 0.2587257
## [3,] 0.13194385 3.865097 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 30.60908 7.2796427
## [2,] 0.03267004 1.00000 0.2378263
## [3,] 0.13736938 4.20475 1.0000000bayes_factors(lml9l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 43.99089 1.30393015
## [2,] 0.02273198 1.00000 0.02964091
## [3,] 0.76691224 33.73715 1.00000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 43.63246 1.25974992
## [2,] 0.02291872 1.00000 0.02887185
## [3,] 0.79380835 34.63581 1.00000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 48.33229 1.14464447
## [2,] 0.0206901 1.00000 0.02368281
## [3,] 0.8736337 42.22472 1.00000000bayes_factors(lml10l)## $BBF
## [,1] [,2] [,3]
## [1,] 1.000000e+00 130.5653 0.0026113844
## [2,] 7.659003e-03 1.0000 0.0000200006
## [3,] 3.829386e+02 49998.5000 1.0000000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.000000e+00 186.6003 2.605411e-03
## [2,] 5.359049e-03 1.0000 1.396252e-05
## [3,] 3.838167e+02 71620.3013 1.000000e+00
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.000000e+00 222.7512 2.232916e-03
## [2,] 4.489314e-03 1.0000 1.002426e-05
## [3,] 4.478450e+02 99758.0000 1.000000e+00## Smallest
bayes_factors(lml1s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 1.902951 1.224402
## [2,] 0.5254996 1.000000 0.643423
## [3,] 0.8167250 1.554188 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 2.122894 0.9667934
## [2,] 0.4710551 1.000000 0.4554130
## [3,] 1.0343472 2.195809 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 1.678655 1.3161889
## [2,] 0.5957151 1.000000 0.7840737
## [3,] 0.7597694 1.275390 1.0000000bayes_factors(lml2s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 1.303366 1.1508924
## [2,] 0.7672443 1.000000 0.8830157
## [3,] 0.8688909 1.132483 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 1.005594 1.0013149
## [2,] 0.9944371 1.000000 0.9957446
## [3,] 0.9986869 1.004274 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000 1.109927 0.9130919
## [2,] 0.90096 1.000000 0.8226593
## [3,] 1.09518 1.215570 1.0000000bayes_factors(lml3s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 0.9680482 1.481416
## [2,] 1.0330065 1.0000000 1.530313
## [3,] 0.6750297 0.6534613 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 0.9990510 1.004137
## [2,] 1.0009499 1.0000000 1.005091
## [3,] 0.9958796 0.9949345 1.000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000 0.7414108 1.543829
## [2,] 1.34878 1.0000000 2.082286
## [3,] 0.64774 0.4802414 1.000000bayes_factors(lml4s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.000000 0.7705316 1.679574
## [2,] 1.297805 1.0000000 2.179760
## [3,] 0.595389 0.4587660 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.000000 0.9971876 0.9915308
## [2,] 1.002820 1.0000000 0.9943273
## [3,] 1.008541 1.0057051 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000 0.5693140 3.994887
## [2,] 1.75650 1.0000000 7.017018
## [3,] 0.25032 0.1425107 1.000000bayes_factors(lml5s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 1.0697938 1.38089
## [2,] 0.9347596 1.0000000 1.29080
## [3,] 0.7241708 0.7747135 1.00000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.000000 0.9919291 1.004385
## [2,] 1.008137 1.0000000 1.012557
## [3,] 0.995634 0.9875983 1.000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000 0.8235604 1.260112
## [2,] 1.21424 1.0000000 1.530079
## [3,] 0.79358 0.6535611 1.000000bayes_factors(lml6s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 0.992606 1.439623
## [2,] 1.0074491 1.000000 1.450347
## [3,] 0.6946261 0.689490 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.000000 0.9948111 0.9930116
## [2,] 1.005216 1.0000000 0.9981912
## [3,] 1.007038 1.0018121 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000 0.7630792 1.437939
## [2,] 1.31048 1.0000000 1.884390
## [3,] 0.69544 0.5306758 1.000000bayes_factors(lml7s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 0.9861769 1.452090
## [2,] 1.0140169 1.0000000 1.472443
## [3,] 0.6886628 0.6791433 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 0.9992888 1.002653
## [2,] 1.0007117 1.0000000 1.003367
## [3,] 0.9973536 0.9966443 1.000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000 0.7566242 1.474100
## [2,] 1.32166 1.0000000 1.948259
## [3,] 0.67838 0.5132788 1.000000bayes_factors(lml8s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 1.1297978 1.307754
## [2,] 0.8851142 1.0000000 1.157512
## [3,] 0.7646698 0.8639222 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 1.002639 1.0006333
## [2,] 0.9973678 1.000000 0.9979994
## [3,] 0.9993671 1.002005 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000 0.8976822 1.130864
## [2,] 1.11398 1.0000000 1.259759
## [3,] 0.88428 0.7938024 1.000000bayes_factors(lml9s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 1.354293 1.0870626
## [2,] 0.7383928 1.000000 0.8026791
## [3,] 0.9199102 1.245828 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 0.997705 1.003852
## [2,] 1.0023003 1.000000 1.006161
## [3,] 0.9961633 0.993877 1.000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000 1.220763 0.8483491
## [2,] 0.81916 1.000000 0.6949337
## [3,] 1.17876 1.438986 1.0000000bayes_factors(lml10s)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 1.228154 1.2209518
## [2,] 0.8142300 1.000000 0.9941356
## [3,] 0.8190332 1.005899 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 0.9973168 1.001140
## [2,] 1.0026904 1.0000000 1.003834
## [3,] 0.9988612 0.9961810 1.000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000 1.001201 1.0001200
## [2,] 0.99880 1.000000 0.9989199
## [3,] 0.99988 1.001081 1.0000000## Median
bayes_factors(lml1m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 12.35758 0.79415258
## [2,] 0.08092201 1.00000 0.06426442
## [3,] 1.25920386 15.56071 1.00000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 11.19719 0.76338928
## [2,] 0.08930815 1.00000 0.06817688
## [3,] 1.30994765 14.66773 1.00000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 15.07430 0.60754249
## [2,] 0.06633809 1.00000 0.04030321
## [3,] 1.64597542 24.81192 1.00000000bayes_factors(lml2m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 9.729568 1.696477
## [2,] 0.1027795 1.000000 0.174363
## [3,] 0.5894569 5.735161 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 9.329516 1.7615079
## [2,] 0.1071867 1.000000 0.1888102
## [3,] 0.5676954 5.296324 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 11.310698 1.4963286
## [2,] 0.08841187 1.000000 0.1322932
## [3,] 0.66830241 7.558967 1.0000000bayes_factors(lml3m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 4.3581370 5.945627
## [2,] 0.2294558 1.0000000 1.364259
## [3,] 0.1681908 0.7329987 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 4.5896561 6.054574
## [2,] 0.2178812 1.0000000 1.319178
## [3,] 0.1651644 0.7580477 1.000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 4.433594 6.260546
## [2,] 0.2255507 1.000000 1.412070
## [3,] 0.1597305 0.708180 1.000000bayes_factors(lml4m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 5.256661 4.9165642
## [2,] 0.1902348 1.000000 0.9353018
## [3,] 0.2033941 1.069174 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 5.533609 5.1521544
## [2,] 0.1807139 1.000000 0.9310658
## [3,] 0.1940936 1.074038 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 5.574302 5.0752141
## [2,] 0.1793947 1.000000 0.9104663
## [3,] 0.1970360 1.098338 1.0000000bayes_factors(lml5m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 8.009439 2.7256917
## [2,] 0.1248527 1.000000 0.3403099
## [3,] 0.3668793 2.938498 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 7.892001 2.9058453
## [2,] 0.1267106 1.000000 0.3682013
## [3,] 0.3441339 2.715905 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 8.914695 2.6083382
## [2,] 0.1121743 1.000000 0.2925886
## [3,] 0.3833859 3.417768 1.0000000bayes_factors(lml6m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 12.76033 0.59972153
## [2,] 0.07836786 1.00000 0.04699889
## [3,] 1.66744054 21.27710 1.00000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 11.55988 0.55674196
## [2,] 0.08650606 1.00000 0.04816155
## [3,] 1.79616423 20.76345 1.00000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 16.31692 0.43007598
## [2,] 0.06128608 1.00000 0.02635767
## [3,] 2.32517052 37.93962 1.00000000bayes_factors(lml7m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 14.12483 0.38872609
## [2,] 0.07079732 1.00000 0.02752077
## [3,] 2.57250549 36.33620 1.00000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 12.53612 0.33081691
## [2,] 0.07976948 1.00000 0.02638909
## [3,] 3.02282013 37.89445 1.00000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 18.67440 0.24883555
## [2,] 0.05354925 1.00000 0.01332496
## [3,] 4.01871845 75.04715 1.00000000bayes_factors(lml8m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 5.694683 4.5614998
## [2,] 0.1756024 1.000000 0.8010103
## [3,] 0.2192261 1.248423 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 5.935347 4.7570303
## [2,] 0.1684821 1.000000 0.8014747
## [3,] 0.2102152 1.247700 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 6.056477 4.6501694
## [2,] 0.1651125 1.000000 0.7678011
## [3,] 0.2150459 1.302421 1.0000000bayes_factors(lml9m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 11.92603 0.8429244
## [2,] 0.08385022 1.00000 0.0706794
## [3,] 1.18634598 14.14839 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 10.95033 0.81978568
## [2,] 0.09132144 1.00000 0.07486401
## [3,] 1.21983102 13.35755 1.00000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 14.78256 0.65318445
## [2,] 0.06764727 1.00000 0.04418614
## [3,] 1.53096111 22.63153 1.00000000bayes_factors(lml10m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 6.488392 3.845559
## [2,] 0.1541214 1.000000 0.592683
## [3,] 0.2600402 1.687243 1.000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 6.628005 4.0483708
## [2,] 0.1508750 1.000000 0.6107978
## [3,] 0.2470129 1.637203 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 7.017475 3.8273780
## [2,] 0.1425014 1.000000 0.5454067
## [3,] 0.2612755 1.833494 1.0000000bayes_factors(lml11m)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 5.918210 4.3360445
## [2,] 0.1689700 1.000000 0.7326614
## [3,] 0.2306249 1.364887 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 6.142678 4.5454187
## [2,] 0.1627954 1.000000 0.7399734
## [3,] 0.2200017 1.351400 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 6.354276 4.4231625
## [2,] 0.1573743 1.000000 0.6960923
## [3,] 0.2260826 1.436591 1.0000000Posterior Probabilities!
## Largest
posterior_probabilities(lml1l, prior.probabilities)## BBF GFBF AFBF
## H1 0.434640350 0.434148789 0.424362519
## H2 0.002154968 0.002259498 0.002119384
## H3 0.563204681 0.563591713 0.573518097posterior_probabilities(lml2l, prior.probabilities)## BBF GFBF AFBF
## H1 2.774878e-03 2.783569e-03 2.552607e-03
## H2 9.971952e-01 9.971803e-01 9.974175e-01
## H3 2.991616e-05 3.613655e-05 2.992312e-05posterior_probabilities(lml3l, prior.probabilities)## BBF GFBF AFBF
## H1 0.91329758 0.91203757 0.91110920
## H2 0.06849729 0.06907845 0.07046145
## H3 0.01820513 0.01888397 0.01842935posterior_probabilities(lml4l, prior.probabilities)## BBF GFBF AFBF
## H1 1.645979e-14 1.460917e-14 1.180787e-14
## H2 0.000000e+00 0.000000e+00 0.000000e+00
## H3 1.000000e+00 1.000000e+00 1.000000e+00posterior_probabilities(lml5l, prior.probabilities)## BBF GFBF AFBF
## H1 0.0124104421 1.251625e-02 1.114043e-02
## H2 0.9874315468 9.874076e-01 9.888002e-01
## H3 0.0001580112 7.616148e-05 5.937014e-05posterior_probabilities(lml6l, prior.probabilities)## BBF GFBF AFBF
## H1 0.8626609 0.86085163 0.85821791
## H2 0.1204130 0.12105778 0.12451804
## H3 0.0169261 0.01809059 0.01726405posterior_probabilities(lml7l, prior.probabilities)## BBF GFBF AFBF
## H1 0.92135294 0.91968601 0.91986105
## H2 0.03936273 0.04006838 0.03996860
## H3 0.03928433 0.04024561 0.04017035posterior_probabilities(lml8l, prior.probabilities)## BBF GFBF AFBF
## H1 0.86137349 0.85757328 0.85467206
## H2 0.02935011 0.02927521 0.02792217
## H3 0.10927640 0.11315152 0.11740577posterior_probabilities(lml9l, prior.probabilities)## BBF GFBF AFBF
## H1 0.55877028 0.55044041 0.52789285
## H2 0.01270195 0.01261539 0.01092216
## H3 0.42852777 0.43694420 0.46118499posterior_probabilities(lml10l, prior.probabilities)## BBF GFBF AFBF
## H1 2.604531e-03 2.598604e-03 2.227918e-03
## H2 1.994811e-05 1.392604e-05 1.000183e-05
## H3 9.973755e-01 9.973875e-01 9.977621e-01## Smallest
posterior_probabilities(lml1s, prior.probabilities)## BBF GFBF AFBF
## H1 0.4269445 0.3991375 0.4245411
## H2 0.2243592 0.1880158 0.2529056
## H3 0.3486963 0.4128467 0.3225533posterior_probabilities(lml2s, prior.probabilities)## BBF GFBF AFBF
## H1 0.3793432 0.3340991 0.3337628
## H2 0.2910489 0.3322405 0.3007069
## H3 0.3296079 0.3336604 0.3655303posterior_probabilities(lml3s, prior.probabilities)## BBF GFBF AFBF
## H1 0.3692713 0.3336860 0.3337204
## H2 0.3814596 0.3340030 0.4501155
## H3 0.2492691 0.3323111 0.2161641posterior_probabilities(lml4s, prior.probabilities)## BBF GFBF AFBF
## H1 0.3456387 0.3320757 0.33257727
## H2 0.4485718 0.3330122 0.58417198
## H3 0.2057895 0.3349121 0.08325074posterior_probabilities(lml5s, prior.probabilities)## BBF GFBF AFBF
## H1 0.3760911 0.3329149 0.3324667
## H2 0.3515547 0.3356237 0.4036944
## H3 0.2723542 0.3314614 0.2638389posterior_probabilities(lml6s, prior.probabilities)## BBF GFBF AFBF
## H1 0.3700859 0.3319774 0.3326769
## H2 0.3728427 0.3337090 0.4359664
## H3 0.2570714 0.3343137 0.2313568posterior_probabilities(lml7s, prior.probabilities)## BBF GFBF AFBF
## H1 0.3700032 0.3335484 0.3333289
## H2 0.3751894 0.3337858 0.4405475
## H3 0.2548074 0.3326657 0.2261237posterior_probabilities(lml8s, prior.probabilities)## BBF GFBF AFBF
## H1 0.3773893 0.3336965 0.3335268
## H2 0.3340326 0.3328182 0.3715422
## H3 0.2885782 0.3334853 0.2949311posterior_probabilities(lml9s, prior.probabilities)## BBF GFBF AFBF
## H1 0.3761798 0.3335041 0.3335646
## H2 0.2777685 0.3342713 0.2732428
## H3 0.3460517 0.3322246 0.3931926posterior_probabilities(lml10s, prior.probabilities)## BBF GFBF AFBF
## H1 0.3797569 0.3331610 0.3334801
## H2 0.3092095 0.3340574 0.3330799
## H3 0.3110335 0.3327816 0.3334400## Median
posterior_probabilities(lml1m, prior.probabilities)## BBF GFBF AFBF
## H1 0.4273274 0.41679591 0.36868894
## H2 0.0345802 0.03722327 0.02445812
## H3 0.5380924 0.54598082 0.60685294posterior_probabilities(lml2m, prior.probabilities)## BBF GFBF AFBF
## H1 0.59093398 0.59705694 0.56924453
## H2 0.06073589 0.06399656 0.05032797
## H3 0.34833013 0.33894650 0.38042749posterior_probabilities(lml3m, prior.probabilities)## BBF GFBF AFBF
## H1 0.7154884 0.7230419 0.7218751
## H2 0.1641730 0.1575373 0.1628194
## H3 0.1203386 0.1194208 0.1153055posterior_probabilities(lml4m, prior.probabilities)## BBF GFBF AFBF
## H1 0.7175511 0.7273746 0.7265168
## H2 0.1365032 0.1314467 0.1303332
## H3 0.1459456 0.1411787 0.1431500posterior_probabilities(lml5m, prior.probabilities)## BBF GFBF AFBF
## H1 0.67036169 0.67988151 0.6686458
## H2 0.08369646 0.08614818 0.0750049
## H3 0.24594186 0.23397031 0.2563493posterior_probabilities(lml6m, prior.probabilities)## BBF GFBF AFBF
## H1 0.3641915 0.3469006 0.29529391
## H2 0.0285409 0.0300090 0.01809741
## H3 0.6072676 0.6230904 0.68660869posterior_probabilities(lml7m, prior.probabilities)## BBF GFBF AFBF
## H1 0.27447622 0.24374848 0.19715048
## H2 0.01943218 0.01944369 0.01055726
## H3 0.70609159 0.73680783 0.79229226posterior_probabilities(lml8m, prior.probabilities)## BBF GFBF AFBF
## H1 0.7169340 0.7253224 0.7245545
## H2 0.1258953 0.1222039 0.1196330
## H3 0.1571707 0.1524738 0.1558125posterior_probabilities(lml9m, prior.probabilities)## BBF GFBF AFBF
## H1 0.44049056 0.43268457 0.38482136
## H2 0.03693523 0.03951338 0.02603211
## H3 0.52257421 0.52780206 0.58914653posterior_probabilities(lml10m, prior.probabilities)## BBF GFBF AFBF
## H1 0.7071328 0.7153649 0.7123639
## H2 0.1089843 0.1079307 0.1015129
## H3 0.1838829 0.1767044 0.1861232posterior_probabilities(lml11m, prior.probabilities)## BBF GFBF AFBF
## H1 0.7144924 0.7231719 0.7228270
## H2 0.1207278 0.1177291 0.1137544
## H3 0.1647798 0.1590991 0.1634186Various samples appear to favor one or another hypothesis. But, outside of the large samples, the evidential value does not - a single time - exceed the level of anecdote.
Given the dependence of BFs on N, we could take another exemplary look at the data. Take their first data point after which the N is not at least a balanced 200 (i.e., from largest to smallest, this would be their 99th datapoint out of 2100). If we compute the BFs and posteriors for this, what do they look like?
K100N <- c(98, 77)
K100Var <- c(2.673, 3.159)^2
K100LML <- log_marginal_likelihoods(K100Var, K100N, b, Hypotheses, SimN)
bayes_factors(K100LML); posterior_probabilities(K100LML, prior.probabilities)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 1.9669939 28.93366
## [2,] 0.50838997 1.0000000 14.70958
## [3,] 0.03456182 0.0679829 1.00000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 2.05763677 26.18517
## [2,] 0.48599443 1.00000000 12.72585
## [3,] 0.03818956 0.07858024 1.00000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 1.88041901 29.19259
## [2,] 0.53179637 1.00000000 15.52451
## [3,] 0.03425527 0.06441426 1.00000## BBF GFBF AFBF
## H1 0.64810839 0.65608877 0.63854855
## H2 0.32949181 0.31885549 0.33957780
## H3 0.02239981 0.02505574 0.02187365Apparently, they look terrible: the evidential quality is “strong”. The problem is that this makes all the subsequent datapoints appear that much worse. 99/2162 = 4.6% of the data. Or, in other words, 95.4% of the data is worse (based on N). Even worse, this datapoint had a staggering variance ratio favoring women, where higher equals more variable for women, the variance ratio was 1.397, or approximately 20% greater than the male-female difference in human cognitive abilities. With a staggering variance such as this, one might think the “strong” evidence would favor greater female variability, but in this data, they favor no difference! The quality of individual data points in this meta-analysis was low.
How do these results compare to human-like data?
1 / 0.9284^2 #Human VR for Cognitive Ability## [1] 1.160192HumanVars <- c(1, 0.9284)^2
NSmall <- c(2, 6)
NMedium <- c(20, 19)
NLarge <- c(535, 502)
NRealistic <- c(1500, 1500)
HumanLMLS <- log_marginal_likelihoods(HumanVars, NSmall, b, Hypotheses, SimN)
HumanLMLM <- log_marginal_likelihoods(HumanVars, NMedium, b, Hypotheses, SimN)
HumanLMLL <- log_marginal_likelihoods(HumanVars, NLarge, b, Hypotheses, SimN)
HumanLMLR <- log_marginal_likelihoods(HumanVars, NRealistic, b, Hypotheses, SimN)
bayes_factors(HumanLMLS)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 2.292959 1.0793591
## [2,] 0.4361178 1.000000 0.4707277
## [3,] 0.9264757 2.124370 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 2.335018 0.9194916
## [2,] 0.4282622 1.000000 0.3937835
## [3,] 1.0875575 2.539467 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 2.100895 1.0446632
## [2,] 0.4759877 1.000000 0.4972468
## [3,] 0.9572463 2.011074 1.0000000bayes_factors(HumanLMLM)## $BBF
## [,1] [,2] [,3]
## [1,] 1.0000000 6.235668 4.0601977
## [2,] 0.1603677 1.000000 0.6511248
## [3,] 0.2462934 1.535804 1.0000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 6.422554 4.2629687
## [2,] 0.1557013 1.000000 0.6637498
## [3,] 0.2345783 1.506592 1.0000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.0000000 6.762861 4.0735487
## [2,] 0.1478664 1.000000 0.6023411
## [3,] 0.2454862 1.660189 1.0000000bayes_factors(HumanLMLL)## $BBF
## [,1] [,2] [,3]
## [1,] 1.00000000 75.44537 3.76462846
## [2,] 0.01325462 1.00000 0.04989873
## [3,] 0.26563046 20.04059 1.00000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 70.12194 3.77921868
## [2,] 0.01426087 1.00000 0.05389496
## [3,] 0.26460496 18.55461 1.00000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 73.81101 3.60846607
## [2,] 0.01354811 1.00000 0.04888791
## [3,] 0.27712606 20.45495 1.00000000bayes_factors(HumanLMLR)## $BBF
## [,1] [,2] [,3]
## [1,] 1.000000000 188.2538 0.407406441
## [2,] 0.005311977 1.0000 0.002164133
## [3,] 2.454551278 462.0787 1.000000000
##
## $GFBF
## [,1] [,2] [,3]
## [1,] 1.00000000 186.6305 0.409436522
## [2,] 0.00535818 1.0000 0.002193835
## [3,] 2.44238104 455.8229 1.000000000
##
## $AFBF
## [,1] [,2] [,3]
## [1,] 1.000000000 196.9965 0.390795574
## [2,] 0.005076232 1.0000 0.001983769
## [3,] 2.558882617 504.0909 1.000000000posterior_probabilities(HumanLMLS, prior.probabilities)## BBF GFBF AFBF
## H1 0.4232637 0.3974848 0.4109757
## H2 0.1845928 0.1702277 0.1956194
## H3 0.3921435 0.4322875 0.3934050posterior_probabilities(HumanLMLM, prior.probabilities)## BBF GFBF AFBF
## H1 0.7109033 0.7192798 0.7176934
## H2 0.1140060 0.1119928 0.1061228
## H3 0.1750908 0.1687274 0.1761838posterior_probabilities(HumanLMLL, prior.probabilities)## BBF GFBF AFBF
## H1 0.7819311 0.78194286 0.77478888
## H2 0.0103642 0.01115119 0.01049693
## H3 0.2077047 0.20690596 0.21471419posterior_probabilities(HumanLMLR, prior.probabilities)## BBF GFBF AFBF
## H1 0.289028764 0.290045139 0.280586854
## H2 0.001535314 0.001554114 0.001424324
## H3 0.709435922 0.708400747 0.717988822At the small sample sizes typical of the meta-analysis, evidence was basically random and valueless. At the median sample size, no difference was favored. At a large sample size for this meta-analysis, no difference was still favored. At a more realistic sample size to detect such a difference, of 1500 each, the evidence favors the correct hypothesis for those data - GMV - but it only does so strongly when compared to an errant hypothesis of GFV and it barely beats no difference! This large of a sample size was only observed once for this data and it was 1.907 times as large as the next-largest!
Unpacking everything above in a simple bulleted list
Because of the sampling, reliability, and theoretical issues, the aggregation of these variances into a Bayesian meta-analysis would be misleading at best. We are left without conclusion.
Harrison, L. M., Noble, D. W. A., & Jennions, M. D. (2021). A meta-analysis of sex differences in animal personality: No evidence for the greater male variability hypothesis. Biological Reviews. https://doi.org/10.1111/brv.12818
Boing-Messing, F., & Mulder, J. (2018). Automatic Bayes Factors for Testing Equality- and Inequality-Constrained Hypotheses on Variances. Psychometrika, 83(3), 586–617. https://doi.org/10.1007/s11336-018-9615-z
Kvarven, A., Stroemland, E., & Johannesson, M. (2019). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour, 1–12. https://doi.org/10.1038/s41562-019-0787-z
Lee, M. D., & Wagenmakers, E.-J. (2013). Bayesian Cognitive Modeling: A Practical Course. Cambridge University Press. https://doi.org/10.1017/CBO9781139087759
sessionInfo()## R version 4.1.2 (2021-11-01)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 19042)
##
## Matrix products: default
##
## locale:
## [1] LC_COLLATE=English_United States.1252
## [2] LC_CTYPE=English_United States.1252
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United States.1252
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] DT_0.20 pacman_0.5.1
##
## loaded via a namespace (and not attached):
## [1] digest_0.6.29 R6_2.5.1 jsonlite_1.7.2 magrittr_2.0.1
## [5] evaluate_0.14 rlang_0.4.12 stringi_1.7.6 jquerylib_0.1.4
## [9] bslib_0.3.1 rmarkdown_2.11 tools_4.1.2 stringr_1.4.0
## [13] htmlwidgets_1.5.4 crosstalk_1.2.0 xfun_0.29 yaml_2.2.1
## [17] fastmap_1.1.0 compiler_4.1.2 htmltools_0.5.2 knitr_1.37
## [21] sass_0.4.0