library(tidyverse)
library(psych)Reliability
Getting Started
We’ll start off by loading two packages:
We’re going to use the listening placement test data from the last homework assignment.
lpt <- read_csv("listening_placement.csv")Calculate Reliability and Item Statistics
Now we’ll use psych() to calculate Cronbach’s alpha and item statistics.
lpt_rel <- alpha(select(lpt, 2:36))Warning in cor.smooth(r): Matrix was not positive definite, smoothing was doneWarning in alpha(select(lpt, 2:36)): Some items were negatively correlated with the first principal component and probably
should be reversed.
To do this, run the function again with the 'check.keys=TRUE' optionSome items ( q9_list_det q13_list_prag q20_list_mi q35_list_inf_an ) were negatively correlated with the first principal component and
probably should be reversed.
To do this, run the function again with the 'check.keys=TRUE' optionLet’s get the reliability first:
lpt_rel$total raw_alpha std.alpha G6(smc) average_r S/N ase mean
0.7859081 0.784249 0.9356328 0.09408505 3.634973 0.05481995 0.6019048
sd median_r
0.1593324 0.1020621The Cronbach’s alpha value is roughly .79. Is this acceptable for a mid- to high-stakes test?
Not really. So what can we do? Although longer tests are generally more reliable, bad items have a negative impact on reliability. In some cases, removing a bad item can lead to a shorter but nonetheless more reliable test.
Let’s look first at the item stats again:
lpt_rel$item.stats n raw.r std.r r.cor r.drop mean
q1_list_mi 30 0.20797645 0.18365503 0.14371999 0.118755627 0.4666667
q2_list_det 30 0.29485598 0.30193046 0.29811527 0.226206840 0.8000000
q3_list_det 30 0.30776148 0.32399630 0.30867892 0.223733955 0.6000000
q4_list_det 30 0.37852269 0.37523222 0.35898943 0.306342148 0.7333333
q5_list_det 30 0.55846137 0.56781622 0.55951885 0.493412287 0.3666667
q6_list_det 30 0.59495220 0.59932922 0.59427788 0.552625615 0.8666667
q7_list_det 30 0.26445846 0.26675782 0.25486679 0.194771267 0.8000000
q8_list_det 30 0.45080617 0.45641331 0.45545984 0.376633712 0.6333333
q9_list_det 30 0.03612594 0.07429482 0.04450912 0.003389000 0.9666667
q10_list_mi 30 0.29603591 0.31257640 0.30188869 0.219966497 0.7333333
q11_list_det 30 0.26749821 0.29026357 0.28568148 0.215634846 0.9000000
q12_list_mi 30 0.32293503 0.34249704 0.34281964 0.251370672 0.7666667
q13_list_prag 30 0.10175984 0.10868202 0.08208744 0.012473522 0.4000000
q14_list_det 30 0.30031564 0.29828965 0.28986855 0.215955954 0.4000000
q15_list_det 30 0.47228979 0.47579714 0.47492163 0.403499745 0.7000000
q16_list_det 30 0.58573960 0.57825952 0.57365422 0.522280450 0.4000000
q17_list_inf 30 0.15806712 0.16395500 0.14497846 0.085878258 0.8000000
q18_list_inf 30 0.22729692 0.21276219 0.19428614 0.148905839 0.7333333
q19_list_det 30 0.63041465 0.64045270 0.64400855 0.571818154 0.6000000
q20_list_mi 30 0.06174301 0.03973088 0.02361974 -0.029290023 0.4666667
q21_list_det 30 0.40124732 0.41623949 0.40697720 0.337394581 0.8000000
q22_list_prag 30 0.35258507 0.32776376 0.30452946 0.269226252 0.5333333
q23_list_mi 30 0.50726537 0.49786388 0.49185851 0.439183778 0.3333333
q24_list_det 30 0.42732661 0.41026561 0.39472239 0.348668700 0.4666667
q25_list_prag 30 0.09457490 0.10574395 0.08405150 0.008636107 0.3333333
q26_list_det_an 30 0.35344884 0.37564601 0.35902863 0.291886340 0.1666667
q27_list_det_an 30 0.34515527 0.34188016 0.32234885 0.261978260 0.4333333
q28_list_det_an 30 0.51156423 0.49236011 0.47918192 0.443843560 0.6666667
q29_list_prag_an 30 0.24700686 0.24950678 0.23497975 0.159566034 0.4333333
q30_list_mi_an 30 0.53735738 0.53571139 0.54191025 0.471882709 0.6666667
q31_list_mi_an 30 0.35168296 0.34979864 0.35088694 0.281424239 0.7666667
q32_list_det_an 30 0.70760737 0.69599909 0.70042705 0.657500038 0.4666667
q33_list_det 30 0.19980132 0.18208798 0.17441587 0.120710295 0.7333333
q34_list_prag_an 30 0.59168889 0.57009554 0.57421778 0.533206440 0.7000000
q35_list_inf_an 30 -0.04743840 -0.04089793 -0.06937374 -0.136677291 0.4333333
sd
q1_list_mi 0.5074163
q2_list_det 0.4068381
q3_list_det 0.4982729
q4_list_det 0.4497764
q5_list_det 0.4901325
q6_list_det 0.3457459
q7_list_det 0.4068381
q8_list_det 0.4901325
q9_list_det 0.1825742
q10_list_mi 0.4497764
q11_list_det 0.3051286
q12_list_mi 0.4301831
q13_list_prag 0.4982729
q14_list_det 0.4982729
q15_list_det 0.4660916
q16_list_det 0.4982729
q17_list_inf 0.4068381
q18_list_inf 0.4497764
q19_list_det 0.4982729
q20_list_mi 0.5074163
q21_list_det 0.4068381
q22_list_prag 0.5074163
q23_list_mi 0.4794633
q24_list_det 0.5074163
q25_list_prag 0.4794633
q26_list_det_an 0.3790490
q27_list_det_an 0.5040069
q28_list_det_an 0.4794633
q29_list_prag_an 0.5040069
q30_list_mi_an 0.4794633
q31_list_mi_an 0.4301831
q32_list_det_an 0.5074163
q33_list_det 0.4497764
q34_list_prag_an 0.4660916
q35_list_inf_an 0.5040069q9, q20, q25, and q35 had low or even negative item discrimination. Removing one or more of these items might improve the reliability of the test.
We can check by looking at alpha.drop statistics. In this table, the raw_alpha column tells us what the new reliability of the test will be if we drop just this one item.
lpt_rel$alpha.drop raw_alpha std.alpha G6(smc) average_r S/N var.r
q1_list_mi 0.7879527 0.7862037 0.9429635 0.09760107 3.677350 0.03850510
q2_list_det 0.7827391 0.7812266 0.9481685 0.09504523 3.570939 0.03813537
q3_list_det 0.7831965 0.7802755 0.9433799 0.09456841 3.551153 0.03809109
q4_list_det 0.7796148 0.7780388 0.9394387 0.09346124 3.505291 0.03650711
q5_list_det 0.7711049 0.7692612 0.9425789 0.08929966 3.333905 0.03822051
q6_list_det 0.7725226 0.7677664 0.9401948 0.08861869 3.306010 0.03830217
q7_list_det 0.7838748 0.7827277 0.9389941 0.09580529 3.602520 0.03795160
q8_list_det 0.7763877 0.7744117 0.9442337 0.09170699 3.432854 0.03704415
q9_list_det 0.7873786 0.7906319 0.9462978 0.09996426 3.776278 0.03802587
q10_list_mi 0.7830939 0.7807686 0.9442554 0.09481518 3.561390 0.03773679
q11_list_det 0.7831789 0.7817265 0.9482149 0.09529735 3.581409 0.03788143
q12_list_mi 0.7818179 0.7794724 0.9552254 0.09416862 3.534580 0.03817367
q13_list_prag 0.7923869 0.7892569 0.9454741 0.09922118 3.745115 0.03738664
q14_list_det 0.7835411 0.7813828 0.9414773 0.09512391 3.574205 0.03786801
q15_list_det 0.7754457 0.7735302 0.9423376 0.09128812 3.415600 0.03853280
q16_list_det 0.7696077 0.7687677 0.9382661 0.08907399 3.324656 0.03765668
q17_list_inf 0.7877693 0.7870134 0.9418943 0.09802677 3.695132 0.03668128
q18_list_inf 0.7859202 0.7849974 0.9416867 0.09697209 3.651106 0.03837528
q19_list_det 0.7672831 0.7657898 0.9389807 0.08773004 3.269670 0.03746363
q20_list_mi 0.7944211 0.7919983 0.9464820 0.10071116 3.807652 0.03773105
q21_list_det 0.7786800 0.7762196 0.9478727 0.09257511 3.468666 0.03825435
q22_list_prag 0.7811975 0.7801123 0.9402373 0.09448700 3.547777 0.03794513
q23_list_mi 0.7737205 0.7725192 0.9425494 0.09081127 3.395976 0.03729704
q24_list_det 0.7775563 0.7764862 0.9381268 0.09270420 3.473997 0.03748339
q25_list_prag 0.7920916 0.7893750 0.9466182 0.09928467 3.747775 0.03821918
q26_list_det_an 0.7805000 0.7780205 0.9443817 0.09345230 3.504921 0.03881052
q27_list_det_an 0.7815148 0.7794993 0.9476669 0.09418195 3.535132 0.03865731
q28_list_det_an 0.7735142 0.7727721 0.9400151 0.09093020 3.400869 0.03765806
q29_list_prag_an 0.7860956 0.7834574 0.9489186 0.09617807 3.618029 0.03784971
q30_list_mi_an 0.7722695 0.7707667 0.9504849 0.08999342 3.362367 0.03817146
q31_list_mi_an 0.7806599 0.7791540 0.9433129 0.09401084 3.528043 0.03773237
q32_list_det_an 0.7628988 0.7630723 0.9374848 0.08652973 3.220697 0.03669790
q33_list_det 0.7870327 0.7862683 0.9462955 0.09763493 3.678764 0.03755418
q34_list_prag_an 0.7698442 0.7691537 0.9374268 0.08925040 3.331886 0.03675046
q35_list_inf_an 0.7988790 0.7951258 0.9455155 0.10245348 3.881045 0.03638508
med.r
q1_list_mi 0.10984701
q2_list_det 0.10090092
q3_list_det 0.10206207
q4_list_det 0.10566113
q5_list_det 0.09869275
q6_list_det 0.09869275
q7_list_det 0.10206207
q8_list_det 0.10090092
q9_list_det 0.10701843
q10_list_mi 0.10566113
q11_list_det 0.10660036
q12_list_mi 0.10206207
q13_list_prag 0.10984701
q14_list_det 0.10206207
q15_list_det 0.10090092
q16_list_det 0.10000000
q17_list_inf 0.10701843
q18_list_inf 0.10984701
q19_list_det 0.09869275
q20_list_mi 0.11080066
q21_list_det 0.10090092
q22_list_prag 0.10206207
q23_list_mi 0.10090092
q24_list_det 0.10090092
q25_list_prag 0.11080066
q26_list_det_an 0.10090092
q27_list_det_an 0.10090092
q28_list_det_an 0.10090092
q29_list_prag_an 0.10206207
q30_list_mi_an 0.10090092
q31_list_mi_an 0.10206207
q32_list_det_an 0.09869275
q33_list_det 0.10566113
q34_list_prag_an 0.10090092
q35_list_inf_an 0.11080066So dropping q9 doesn’t look like it will have much of an impact on reliability. However, if we drop q35, we should see a very slight increase in reliability - going up to .80.
Revising a Test for Higher Reliability
Let’s ‘create’ a revised version of the test without q35.
lpt_revised <- select(lpt, 2:35)And now the reliability…
lpt_revised_reliability <- alpha(lpt_revised)Warning in cor.smooth(r): Matrix was not positive definite, smoothing was doneWarning in alpha(lpt_revised): Some items were negatively correlated with the first principal component and probably
should be reversed.
To do this, run the function again with the 'check.keys=TRUE' optionSome items ( q9_list_det q13_list_prag q20_list_mi ) were negatively correlated with the first principal component and
probably should be reversed.
To do this, run the function again with the 'check.keys=TRUE' optionlpt_revised_reliability$total raw_alpha std.alpha G6(smc) average_r S/N ase mean sd
0.798879 0.7951258 0.9455155 0.1024535 3.881045 0.05167933 0.6068627 0.165386
median_r
0.1108007Indeed, we have a very slight improvement to reliability! We can also check and see if there is any further room for improvement.
lpt_revised_reliability$alpha.drop raw_alpha std.alpha G6(smc) average_r S/N var.r
q1_list_mi 0.8007717 0.7968949 0.9520947 0.10626169 3.923560 0.03704892
q2_list_det 0.7960868 0.7924305 0.9589464 0.10369111 3.817664 0.03664196
q3_list_det 0.7964436 0.7913866 0.9530756 0.10310384 3.793557 0.03663465
q4_list_det 0.7931594 0.7893667 0.9491756 0.10198187 3.747588 0.03495000
q5_list_det 0.7845533 0.7804533 0.9513383 0.09724677 3.554840 0.03700912
q6_list_det 0.7865391 0.7797558 0.9503175 0.09689036 3.540414 0.03692613
q7_list_det 0.7960868 0.7927320 0.9498190 0.10386166 3.824672 0.03686615
q8_list_det 0.7920673 0.7875765 0.9567317 0.10100305 3.707577 0.03564447
q9_list_det 0.8006250 0.8020467 0.9555042 0.10935252 4.051696 0.03637978
q10_list_mi 0.7964109 0.7919505 0.9532531 0.10342043 3.806549 0.03623199
q11_list_det 0.7967422 0.7933520 0.9565803 0.10421378 3.839147 0.03631434
q12_list_mi 0.7960293 0.7915553 0.9631496 0.10319837 3.797435 0.03665090
q13_list_prag 0.8068296 0.8014437 0.9576241 0.10898360 4.036355 0.03585411
q14_list_det 0.7965512 0.7922515 0.9554882 0.10359004 3.813513 0.03643771
q15_list_det 0.7895382 0.7853373 0.9515165 0.09979877 3.658470 0.03710441
q16_list_det 0.7846944 0.7813001 0.9495497 0.09768210 3.572476 0.03618961
q17_list_inf 0.7997606 0.7968272 0.9519475 0.10622199 3.921920 0.03547785
q18_list_inf 0.7995748 0.7964485 0.9512372 0.10600023 3.912761 0.03680661
q19_list_det 0.7815710 0.7776734 0.9545031 0.09583807 3.497887 0.03608906
q20_list_mi 0.8073268 0.8028087 0.9584350 0.10982152 4.071218 0.03607319
q21_list_det 0.7933905 0.7888605 0.9566387 0.10170360 3.736204 0.03681283
q22_list_prag 0.7939681 0.7907204 0.9500489 0.10273167 3.778296 0.03664597
q23_list_mi 0.7878198 0.7842650 0.9515328 0.09922984 3.635316 0.03581617
q24_list_det 0.7910351 0.7876791 0.9483866 0.10105877 3.709852 0.03605193
q25_list_prag 0.8049722 0.8002068 0.9560389 0.10823282 4.005174 0.03663941
q26_list_det_an 0.7946222 0.7901431 0.9574112 0.10241089 3.765152 0.03733845
q27_list_det_an 0.7949714 0.7907325 0.9681151 0.10273845 3.778573 0.03722516
q28_list_det_an 0.7868549 0.7838213 0.9492389 0.09899586 3.625803 0.03637231
q29_list_prag_an 0.7987270 0.7940322 0.9573850 0.10460222 3.855128 0.03646357
q30_list_mi_an 0.7862723 0.7824362 0.9593204 0.09827078 3.596352 0.03679058
q31_list_mi_an 0.7954932 0.7917932 0.9520462 0.10333194 3.802917 0.03626991
q32_list_det_an 0.7779568 0.7754913 0.9468352 0.09475376 3.454170 0.03523702
q33_list_det 0.7990534 0.7961744 0.9557791 0.10584022 3.906156 0.03637759
q34_list_prag_an 0.7837269 0.7806657 0.9489006 0.09735569 3.559251 0.03535049
med.r
q1_list_mi 0.1111111
q2_list_det 0.1108007
q3_list_det 0.1108007
q4_list_det 0.1108007
q5_list_det 0.1020621
q6_list_det 0.1020621
q7_list_det 0.1108007
q8_list_det 0.1108007
q9_list_det 0.1111111
q10_list_mi 0.1109559
q11_list_det 0.1108007
q12_list_mi 0.1108007
q13_list_prag 0.1111111
q14_list_det 0.1108007
q15_list_det 0.1098470
q16_list_det 0.1061307
q17_list_inf 0.1109559
q18_list_inf 0.1111111
q19_list_det 0.1009009
q20_list_mi 0.1136514
q21_list_det 0.1108007
q22_list_prag 0.1108007
q23_list_mi 0.1103238
q24_list_det 0.1098470
q25_list_prag 0.1112834
q26_list_det_an 0.1108007
q27_list_det_an 0.1108007
q28_list_det_an 0.1103238
q29_list_prag_an 0.1108007
q30_list_mi_an 0.1098470
q31_list_mi_an 0.1108007
q32_list_det_an 0.1020621
q33_list_det 0.1111111
q34_list_prag_an 0.1103238Looking at the alpha.drop stats and thinking about which items have weak discrimination, it looks like we could see another slight increase from dropping q20 or q25.
Of course, in the real world, we should be thinking about item content, too. But in general, if you can shorten a test and maintain adequate content/construct coverage and maintain/improve reliability of total scores, it’s a good idea.