• GFM in Research on Scientific Thinking
  • Review
  • Simulations
  • Discussion
• GFM in Broader Psychology
  • Intelligence
  • Personality
  • Clinical
• GFM Issues
  • Statistical
  • Theoretical
  • Indeterminacy

“a general factor (GF) is assumed to influence all observed variables”. “a general second-order factor (GF) having an influence on all first-order factors”. (Eid, Heene, & , 2016)

General factor GF: A latent random variable that directly or indirectly predicts all observed variables that belong to the represented construct.

• Latent variable (Borsboom, 2008):
  • Inference from data structure to variable structure prone to error (epistemic accessability).

• Observed variable:
  • relation between variable structure and data structure deterministic, causally isolated, and of equivalent cardinality.

• Psychological variables: Latent.

• Long interest in education, development, assessment (Blair, 1940; Nerring, 1918; Piaget & Inhelder, 1958).

• Two waves of test development (Opitz, Fischer, & Heene, submitted)
  • wave I: 1970-1990.
  • wave II: since 2000s. Following PISA etc: Rasch Model Application. GFM.

• Why is this GFM?
  • Random Effect: All systematic answer differences depend on a single person characteristic.
  • GLMM: Crossed random effects linear model with logistic link function.

• Why Rasch?
  • Sufficient statistics: All information is in the sum score.
  • Specific objectivity: Same information for any persons and items.
  • Invariant measurement: Linking of items and tests in large-scale and longitudinal studies.
  • 2pl: No measurement and exchangeability; everyone wants Rasch!

• Unidimensionality (GFM)
  • One coherent psychological trait

• Local (stochastic) independence
  • No additional systematic influences on answers

• Homogenous item discriminations
  • Parallel Item Characteristic Curves (equal factor loadings)
    

• 7 instruments based on Rasch model, mostly since 2010 (Review: Opitz, Fischer, & Heene, submitted)
• Typical approach is to fit unidimensional Rasch model in ConQuest software, remove items based on infit statistics

• Based on the individual response residuals
  \[ \begin{align} R_{pi} &= X_{pi} - E_{pi}\\ Z_{pi}^2 &= \frac{R_{pi}^2}{VAR(X_{pi})}\\ \end{align} \]

• \[ \begin{equation} INFIT_i = \frac{\sum^n_{p=1}R^2_{pi}}{\sum^n_{p=1}VAR(X_{pi})} \end{equation} \]

• Degree of item-level deviation from expected scores given the Rasch model

• Dichotomous data
  • used R package eRm
  • Item size: 12, 24, 60, 120
  • Sample Size: 150, 300, 500, 1000
  • 1000 data sets per combination
  • Code, slides, comments at https://github.com/dostodabsi/simrasch

• Sim1: Unequal item discriminations
  • \(d_i \sim \text{lognormal}(0, \sigma)\)
  • \(\sigma\) either .1, .3, .5
• Sim2: Typical theoretical model
  • 3-5 correlated facets of Scientific Reasoning assumed
  • Simulated factor structure:
  • \(\left( \begin{array}{ccc} 1.0 & 0.6 & 0.5 & 0.4\\ 0.6 & 1.0 & 0.7 & 0.4\\ 0.5 & 0.7 & 1.0 & 0.3\\ 0.4 & 0.4 & 0.3 & 1.0 \end{array} \right)\)
  • How do infit statistics respond to such data?

• Conclusions in reviewed articles:
  • scale measures unitary construct.
  • support for unidimensional theoretical model.
  • instrument possesses adequate reliability and validity.

• Inconsistent assumptions, models, testing, and inferences?
  • Different paradigms (Andersen, 2010; Linacre, 2010)
• Two traditional perspectives on Rasch: Utility vs. hypothesis testing (cf. Linacre, 2010).

• Researchers apply practices of the utility/Rasch school. Potential reasons?
  • Following large scale programs
  • Limiting software (ConQuest)
  • Lack of knowledge about Rasch foundations and model tests

• Testing fit of data to model.
  • Rasch makes measurement from psychological data.
  • Sufficient fit: Valid measurement.
  • Choose your grain size.

• Itemfit strongly suggested, to test affordances/predictions of the model (Wu, Wilson, Wang, Boone)
  • Points towards measurement problems.
  • Central criterion for fit of data.

• Testing of fit of model to the data.
  • Rasch tests whether measurement took place.
  • Absolute fit: Valid measurement.
  • Examine the grain size.

• Itemfit strongly critisized (Christensen & Kreiner, 2013; Heene et al., 2014; Smith, 1995; Smith, 1996; Smith et al., 1998)
  • Underpowered.
  • Apply only after inspection of global model fit.
  • Item removal can decrease construct validity.
  • Itemfit statistics are in general no adequate means for testing the Rasch model.
  • …if Rasch does not fit:

• Testing the Rasch model
  • Andersen test (Person homogeneity; Andersen, 1973)
  • Martin Löf test (Item homogeneity; Christensen et al., 2002)
  • Recursive Partitioning (continuous DIF - Rasch Trees; Strobl et al., 2013)
  • M2 statistic (Maydeu-Olivares, 2013)
  • nonparametric Statistics (t1, t10; Koller et al., 2015)
  • posterior predictive checking (Bayesianisch; Fox, 2010; Sinharay, 2006)
  • SEM-like statistics (RMSEA, CFI, SRMR; Maydeu-Olivares, 2014)
  • ! Battle-testing !

• Match of theoretical and statistical model
  • IRT: Multidimensional Rasch model, 2PL, Mixtures (Fischer & Molenaar, 1995)
  • Nonparametric IRT: Mokken Scaling (van der Graaf et al., 2015)
  • Applied IRT: cognitive diagnosis modeling (de la Torre & Minchen, 2014)
  • R-Software

• Psychological: What is ST?
  • Dynamics/Intercorrelations, or a psychological trait?

• Pedagogical: How to intervene?
  • Differential validities?

Charles Spearman (1904)

Any alternative after 100 years?

• Thomson (1916): Bonds:
  • elements of activity at the lower level are entirely specific, but those at the higher level are such that they may come into play in different activities. Any activity is a sample of these elements.

• van der Maas et al., 2006: Mutualism.
  • 

• Consensus still on G (Stern & Neubauer, 2016).

Musek (2007)

Heavily criticized (e.g., Prinz, 2014)

• "The p factor explains why it is challenging to find causes, consequences, biomarkers, and treatments with specificity to individual mental disorders"

• "…researchers should no longer assume a specific relation between the disorder they study and a biomarker/ cause/consequence/treatment without empirical verification. Rather, our finding suggests the default assumption must be that biomarkers/causes/consequences/ treatments relate first to p." (Caspi et al., 2014, p. 137).

• Else? (Musek, 2007).

• emotionality (Diener, Smith, & Fujita, 1995; Larsen & Diener, 1992; Tellegen, 1985; Watson & Clark, 1993)

• motivation (Cattell, 1957; Cattell, Radcliffe, & Sweney, 1963; Elliot & Thrash, 2002)

• self-concept (Marsh, 1990; Marsh & Shavelson, 1985; Marsh, Byrne, & Shavelson, 1988)

• values.

• Model equivalence (MacCallum, 2000; Raykov & Penev, 1999; Raykov & Marcoulides, 2001).

• Bias (Murray & Johnson, 2013).

• Sampling (Eid & Koch, 2014; Eid, Geiser, Koch, & Heene, 2016).

• Model equivalence (MacCallum, 2000; Raykov & Penev, 1999; Raykov & Marcoulides, 2001). We compare only few models. Confirmation bias (MacCallum, 2000)

• "An especially interesting phenomenon in the context of alternative models is the existence of equivalent models, which are alternative models that fit any data to the same degree. Such models can be distinguished only in terms of substantive meaning." (MacCallum, 2000)

• Raykov & Marcoulides, 2001:
  • 

• Good model fit does not indicate the correctness of a model. (See also Roberts & Pashler, 2000)

• Bias (Murray & Johnson, 2013). Bi-factor GFM is biased in model comparisons due to unmodelled complexity.

• Sampling (Eid & Koch, 2014; Eid, Geiser, Koch, & Heene, 2016). "domains have to be randomly chosen from a set of possible domains. they are considered levels of a random GT-facet"

• Quantity (Michell, 1997, 2008)

• Interindividual vs. intra-individual structure (Borsboom, 2005; Molenaar, 2004, 2008; Molenaar & Campbell, 2009)

"a discrete quantitative difference need not be caused by a quantitative factor at all, let alone one that is a continuous quantity." (Michell, 2013)

"when the ideological support structures of a science sustain serious blind spots like this, then that science is in the grip of some kind of thought disorder." (Michell, 1997)

Scientific measurement? See also Inventing Temperature (Chang, 2007)

there is no single piece of evidence more important to a construct's definition than the causal relationship between the construct and its indicators

• Interindividual vs. intra-individual structure (Borsboom, 2005; Molenaar, 2004, 2008; Molenaar & Campbell, 2009)

• We are used to modelling inter-individual variance for 110 years.

• What between-subjects latent variables models do is specify sources of between-subjects differences, but because they are silent with respect to the question of how individual scores are produced, they cannot be interpreted as posing intelligence as a causal force within Einstein. (Borsboom, 2003)

• variations in height have demonstrably the same effect within and between subjects. But it remains to be shown that the same holds true for psychological variables like intelligence. (See Borkenau & Ostendorf, 1998: Big Five)

Argument 1: Statistically, a General Factor represents covariances. It is a behaviorist construct based on logical positivism and therefore formative. Thus, it cannot inform cognitivist theories, which the assumption of the "existence" of something going beyond covariance dynamics is.

GFM is applied behaviorist

GFM is interpreted cognitivist

Argument 2: Correlational data cannot be used to test the assumption of a reflective latent construct because it is a causal assumption. Thus, GFM cannot be used to test the assumption of a GF. This needs experimental or longitudinal data. This renders degrees of freedom meaningless.

GFM is applied correlationally.

GFM is interpreted causally.

van der Maas et al. (2016): If we just count parameters, the g-factor model seems simpler than an unconstrained mutualism model. This might not be true for constrained versions of the mutualism model. But more importantly, what are the costs of introducing a mysterious latent variable, as the common cause of ‘everything’ in the g-factor model?

Underestimation of the flexibility and thereby the complexity of GF-theories.

Degrees of freedom are not a valid means of model comparison in GFM.

Alternative routes:

• Experimentation (e.g., multinomial modeling)

• Longitudinal Data (e.g., intra-individual)

• Complexity Modeling (e.g., [dynamic] network modeling)

Theoretical Research into model complexity.

• GFM has statistical problems.

• GFM has theoretical problems.

• GFM does not work.

• We have been engaging in behaviorist modeling of cognitive constructs for more than 110 years.

• Psychometrics is a mess.