Reliability
Deon de Bruin
2023-04-24
Department of Industrial Psychology
Stellenbosch University
South Africa
Dimensionality and reliability
This document shows how several reliability coefficients can be obtained with the psych package (Revelle, 2023) in R. The data in the examples are the responses of 1377 persons to the items of the General Work Stress Scale (de Bruin & Taylor, 2005; de Bruin, 2006). Revelle and Condon (2019) give a comprehensive tutorial on reliability.
The dimensionality of a scale should be established before the estimation of its reliability. The simplest case is a unidimensional scale in which the reliability coefficient has a clear meaning: The proportion of observed variance that is true variance (i.e. variance that is due to the single common factor that is measured by the items). In general, this proportion is given by coefficient \(\omega\).
In the presence of multidimensionality the reliability coefficient of a total score reflects the proportion of variance that is due to all common factors that is measured by the items. Following McDonald (1999), I view the calculation of a reliability coefficient in such situations as not well motivated (unless there is evidence of a very strong general factor).
Three reliability coefficients appear to be particularly useful, namely \(\omega_{total}\), \(\omega_{hierarchical}\), and \(\beta\). Coefficient \(\omega_{total}\) reflects the proportion of variance in the total score that is due to variance explained by the common factor(s) that are measured by the items. Coefficient \(\omega_{hierarchical}\) reflects the proportion of variance in the total score that is explained by a general factor. Coefficient \(\beta\) reflects the worst possible split-half reliability and also is a measure of the proportion of variance of the total score that is explained by a general factor.
Coefficient \(\alpha\) is the most popular indicator of test reliability. However, \(\alpha\) reflects the reliability of a test under rather strict assumptions: (a) the test is unidimensional, (b) the unique variances of the items are uncorrelated, and (c) the items are essentially tau-equivalent (i.e. they have equivalent unstandardized factor loadings in a unidimensional solution). Violations of these assumptions could lead to over or under estimation of test reliability. Coefficient \(\omega\) relaxes the requirement of tau-equivalent items and is more generally applicable. One might expect \(\omega\) of a unidimensional scale to be somewhat higher than the corresponding \(\alpha\).
Revelle (2023) describes some experimental measures of unidimensionality. The first, r.fit, examines the degree to which the average inter-item correlation reproduces the observed correlation matrix. The second, fa.fit, examines the degree to which a single factor model reproduces the observed correlation matrix. The unidimensionality index, u is the product of r.fit and fa.fit. Values close to unity suggest unidimensionality. The function automatically reverses negatively scored items.
mydata <- read.csv("~/GWScountry.csv")
mydata <- mydata[2:10]
library(psych)
scree(mydata)
fa.parallel(mydata)
## Parallel analysis suggests that the number of factors = 4 and the number of components = 1nfactors(mydata)
##
## Number of factors
## Call: vss(x = x, n = n, rotate = rotate, diagonal = diagonal, fm = fm,
## n.obs = n.obs, plot = FALSE, title = title, use = use, cor = cor)
## VSS complexity 1 achieves a maximimum of 0.91 with 1 factors
## VSS complexity 2 achieves a maximimum of 0.93 with 2 factors
## The Velicer MAP achieves a minimum of 0.04 with 1 factors
## Empirical BIC achieves a minimum of -62.49 with 3 factors
## Sample Size adjusted BIC achieves a minimum of -13.16 with 4 factors
##
## Statistics by number of factors
## vss1 vss2 map dof chisq prob sqresid fit RMSEA BIC SABIC complex
## 1 0.91 0.00 0.038 27 8.1e+02 1.1e-153 2.5 0.91 0.145 617.0 702.8 1.0
## 2 0.57 0.93 0.047 19 2.6e+02 2.8e-44 1.9 0.93 0.096 122.7 183.1 1.5
## 3 0.43 0.77 0.070 12 6.0e+01 2.5e-08 1.5 0.95 0.054 -26.9 11.2 2.0
## 4 0.37 0.68 0.109 6 1.1e+01 8.4e-02 1.2 0.96 0.025 -32.2 -13.2 2.3
## 5 0.37 0.71 0.178 1 3.0e-01 5.8e-01 1.1 0.96 0.000 -6.9 -3.8 2.3
## 6 0.39 0.65 0.272 -3 5.1e-07 NA 1.1 0.96 NA NA NA 2.3
## 7 0.37 0.65 0.451 -6 2.7e-07 NA 1.1 0.96 NA NA NA 2.3
## 8 0.37 0.65 1.000 -8 1.1e-09 NA 1.1 0.96 NA NA NA 2.3
## 9 0.37 0.65 NA -9 1.1e-09 NA 1.1 0.96 NA NA NA 2.3
## eChisq SRMR eCRMS eBIC
## 1 3.8e+02 6.2e-02 0.0717 187.5
## 2 1.2e+02 3.5e-02 0.0486 -13.8
## 3 2.4e+01 1.6e-02 0.0271 -62.5
## 4 2.7e+00 5.3e-03 0.0129 -40.6
## 5 8.2e-02 9.1e-04 0.0055 -7.1
## 6 8.3e-08 9.1e-07 NA NA
## 7 5.4e-08 7.4e-07 NA NA
## 8 2.0e-10 4.5e-08 NA NA
## 9 2.0e-10 4.5e-08 NA NAunidim(mydata)##
## A measure of unidimensionality
## Call: unidim(x = mydata)
##
## Unidimensionality index =
## u av.r fit fa.fit alpha av.r median.r Unidim.A
## 0.96 0.98 0.99 0.90 0.51 0.50 1.00
##
## unidim adjusted index reverses negatively scored items.
## alpha Based upon reverse scoring some items.
## average and median correlations are based upon reversed scored itemsTreating the GWSS as undimensional
The plot displays the distribution of the split-half reliabilities of
the test as well as \(\omega_{total}\),
\(\omega_{hierarchical}\), \(\alpha\) and a measure of unidimensionality
(Revelle, 2023). The function also yields the maximum split-half
reliability (Guttman’s \(\lambda_4\))
and the minimum split-half reliability (Revelle’s \(\beta\)). The reliability()
function finds the minimum and maximum split-half reliabilities by
“brute force” (i.e. empirically) (Revelle, 2023).
rel <-reliability(mydata, nfactors = 1)
rel## Measures of reliability
## reliability(keys = mydata, nfactors = 1)
## omega_h alpha omega.tot Uni r.fit fa.fit max.split min.split mean.r
## All_items 0.9 0.9 0.9 0.96 0.98 0.99 0.92 0.84 0.51
## med.r n.items
## All_items 0.5 9plot(rel)
splitHalf(mydata)## Split half reliabilities
## Call: splitHalf(r = mydata)
##
## Maximum split half reliability (lambda 4) = 0.92
## Guttman lambda 6 = 0.91
## Average split half reliability = 0.89
## Guttman lambda 3 (alpha) = 0.9
## Guttman lambda 2 = 0.9
## Minimum split half reliability (beta) = 0.84
## Average interitem r = 0.51 with median = 0.5guttman(mydata)## Call: guttman(r = mydata)
##
## Alternative estimates of reliability
##
## Guttman bounds
## L1 = 0.8
## L2 = 0.9
## L3 (alpha) = 0.9
## L4 (max) = 0.92
## L5 = 0.87
## L6 (smc) = 0.91
## TenBerge bounds
## mu0 = 0.9 mu1 = 0.9 mu2 = 0.9 mu3 = 0.9
##
## alpha of first PC = 0.9
## estimated greatest lower bound based upon communalities= 0.93
##
## beta found by splitHalf = 0.84tenberge(mydata)## $mu0
## [1] 0.9022746
##
## $mu1
## [1] 0.9034413
##
## $mu2
## [1] 0.903741
##
## $mu3
## [1] 0.9038312alpha(mydata)##
## Reliability analysis
## Call: alpha(x = mydata)
##
## raw_alpha std.alpha G6(smc) average_r S/N ase mean sd median_r
## 0.9 0.9 0.91 0.51 9.2 0.004 2.2 0.76 0.5
##
## 95% confidence boundaries
## lower alpha upper
## Feldt 0.89 0.9 0.91
## Duhachek 0.89 0.9 0.91
##
## Reliability if an item is dropped:
## raw_alpha std.alpha G6(smc) average_r S/N alpha se var.r med.r
## i1 0.89 0.89 0.89 0.50 8.1 0.0045 0.0043 0.50
## i2 0.88 0.89 0.88 0.49 7.7 0.0047 0.0038 0.49
## i3 0.89 0.89 0.89 0.50 8.1 0.0045 0.0064 0.50
## i4 0.89 0.89 0.89 0.51 8.4 0.0044 0.0071 0.51
## i5 0.89 0.89 0.89 0.51 8.4 0.0043 0.0065 0.51
## i6 0.89 0.89 0.89 0.50 7.9 0.0046 0.0068 0.47
## i7 0.90 0.90 0.90 0.52 8.6 0.0042 0.0061 0.51
## i8 0.89 0.89 0.89 0.50 8.1 0.0045 0.0076 0.48
## i9 0.89 0.90 0.90 0.52 8.6 0.0043 0.0068 0.51
##
## Item statistics
## n raw.r std.r r.cor r.drop mean sd
## i1 1377 0.77 0.76 0.74 0.69 2.6 1.06
## i2 1377 0.82 0.81 0.80 0.75 2.1 1.11
## i3 1377 0.77 0.76 0.72 0.69 2.2 1.06
## i4 1377 0.74 0.73 0.69 0.65 2.2 1.09
## i5 1377 0.71 0.72 0.68 0.63 2.0 0.93
## i6 1377 0.78 0.79 0.76 0.72 2.1 0.90
## i7 1377 0.70 0.70 0.65 0.61 2.7 1.05
## i8 1377 0.76 0.76 0.72 0.69 2.0 0.94
## i9 1377 0.70 0.70 0.65 0.61 2.0 0.96
##
## Non missing response frequency for each item
## 1 2 3 4 5 miss
## i1 0.17 0.27 0.39 0.11 0.05 0
## i2 0.36 0.30 0.23 0.08 0.04 0
## i3 0.28 0.36 0.24 0.08 0.04 0
## i4 0.31 0.30 0.26 0.10 0.03 0
## i5 0.32 0.40 0.22 0.05 0.01 0
## i6 0.28 0.43 0.22 0.05 0.01 0
## i7 0.14 0.31 0.32 0.19 0.04 0
## i8 0.36 0.38 0.20 0.04 0.02 0
## i9 0.35 0.39 0.19 0.04 0.02 0Treating the GWSS as multidimensional
One might decide to treat the GWSS as multidimensional with three correlated factors. In such situations the focus might shift to finding the proportion of variance in the total score that is explained by a general factor common to all the items, i.e. coefficient \(\omega_{hierarchical}\). Both coefficients \(\omega_{hierarchical}\) and \(\beta\) are estimates of the general factor saturation of a scale. If these values are high it might be justified to treat the scale as if it is unidimensional.
rel2 <- reliability(mydata, nfactors = 3)
rel2## Measures of reliability
## reliability(keys = mydata, nfactors = 3)
## omega_h alpha omega.tot Uni r.fit fa.fit max.split min.split mean.r
## All_items 0.8 0.9 0.92 0.96 0.98 0.99 0.92 0.84 0.51
## med.r n.items
## All_items 0.5 9plot(rel2)
omega(mydata, nfactors = 3)
## Omega
## Call: omegah(m = m, nfactors = nfactors, fm = fm, key = key, flip = flip,
## digits = digits, title = title, sl = sl, labels = labels,
## plot = plot, n.obs = n.obs, rotate = rotate, Phi = Phi, option = option,
## covar = covar)
## Alpha: 0.9
## G.6: 0.91
## Omega Hierarchical: 0.8
## Omega H asymptotic: 0.87
## Omega Total 0.92
##
## Schmid Leiman Factor loadings greater than 0.2
## g F1* F2* F3* h2 u2 p2
## i1 0.66 0.51 0.69 0.31 0.63
## i2 0.72 0.60 0.87 0.13 0.59
## i3 0.65 0.32 0.54 0.46 0.78
## i4 0.66 0.46 0.65 0.35 0.67
## i5 0.67 0.33 0.56 0.44 0.81
## i6 0.77 0.38 0.74 0.26 0.80
## i7 0.63 0.40 0.56 0.44 0.71
## i8 0.68 0.52 0.48 0.88
## i9 0.61 0.42 0.58 0.87
##
## With Sums of squares of:
## g F1* F2* F3*
## 4.08 0.76 0.31 0.42
##
## general/max 5.4 max/min = 2.46
## mean percent general = 0.75 with sd = 0.1 and cv of 0.14
## Explained Common Variance of the general factor = 0.73
##
## The degrees of freedom are 12 and the fit is 0.04
## The number of observations was 1377 with Chi Square = 59.79 with prob < 2.5e-08
## The root mean square of the residuals is 0.02
## The df corrected root mean square of the residuals is 0.03
## RMSEA index = 0.054 and the 10 % confidence intervals are 0.041 0.068
## BIC = -26.94
##
## Compare this with the adequacy of just a general factor and no group factors
## The degrees of freedom for just the general factor are 27 and the fit is 0.69
## The number of observations was 1377 with Chi Square = 945.13 with prob < 9.5e-182
## The root mean square of the residuals is 0.09
## The df corrected root mean square of the residuals is 0.1
##
## RMSEA index = 0.157 and the 10 % confidence intervals are 0.149 0.166
## BIC = 749.98
##
## Measures of factor score adequacy
## g F1* F2* F3*
## Correlation of scores with factors 0.90 0.78 0.54 0.63
## Multiple R square of scores with factors 0.82 0.61 0.29 0.40
## Minimum correlation of factor score estimates 0.64 0.21 -0.42 -0.20
##
## Total, General and Subset omega for each subset
## g F1* F2* F3*
## Omega total for total scores and subscales 0.92 0.87 0.80 0.79
## Omega general for total scores and subscales 0.80 0.58 0.67 0.62
## Omega group for total scores and subscales 0.09 0.29 0.13 0.17iclust(mydata)
## ICLUST (Item Cluster Analysis)
## Call: iclust(r.mat = mydata)
##
## Purified Alpha:
## [1] 0.9
##
## G6* reliability:
## [1] 1
##
## Original Beta:
## [1] 0.82
##
## Cluster size:
## [1] 9
##
## Item by Cluster Structure matrix:
## [,1]
## i1 0.74
## i2 0.80
## i3 0.72
## i4 0.69
## i5 0.68
## i6 0.76
## i7 0.65
## i8 0.72
## i9 0.65
##
## With eigenvalues of:
## [1] 4.6
##
## Purified scale intercorrelations
## reliabilities on diagonal
## correlations corrected for attenuation above diagonal:
## [,1]
## [1,] 0.9
##
## Cluster fit = 0.91 Pattern fit = 0.99 RMSR = 0.06References
de Bruin, G. P. & Taylor, N. (2005). Development of the Sources of Work Stress Inventory. South African Journal of Psychology, 35, 748-765.
de Bruin, G. P. (2006). Dimensionality of the General Work Stress Scale. SA Journal of Industrial Psychology, 32(4), 68-75.
Revelle, W. (2023). psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University, Evanston, Illinois. R package version 2.3.3, https://CRAN.R-project.org/package=psych.
Revelle, W., & Condon, D. (2019). Reliability from \(\alpha\) to \(\omega\). Psychological Assessment, 31, 1395-1411.