1 Accuracy

1.1 Descriptive data

Average accuracy for each PGY:

##   Group.1         x
## 1       1 0.9262610
## 2       2 0.8886709
## 3       3 0.8430727
## 4       4 0.8304636

Average TLS rate for each PGY:

##   Group.1          x
## 1       1 0.05584216
## 2       2 0.08635816
## 3       3 0.12799056
## 4       4 0.14164259

1.2 OLS without intercept

Accuracy vs PGY; OLS without intercept:

ols1 <-lm(Accuracy. ~ PGY.level - 1, data=drh)
summary(ols1)
## 
## Call:
## lm(formula = Accuracy. ~ PGY.level - 1, data = drh)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -0.6131 -0.1460  0.1412  0.4219  0.7144 
## 
## Coefficients:
##           Estimate Std. Error t value Pr(>|t|)    
## PGY.level 0.285621   0.007674   37.22   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.3956 on 354 degrees of freedom
## Multiple R-squared:  0.7965, Adjusted R-squared:  0.7959 
## F-statistic:  1385 on 1 and 354 DF,  p-value: < 2.2e-16

Accuracy vs PGY; OLS with intercept:

ols2 <-lm(Accuracy. ~ PGY.level, data=drh)
summary(ols2)
## 
## Call:
## lm(formula = Accuracy. ~ PGY.level, data = drh)
## 
## Residuals:
##       Min        1Q    Median        3Q       Max 
## -0.292700 -0.034981  0.009571  0.046318  0.177888 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  0.955364   0.008900  107.34   <2e-16 ***
## PGY.level   -0.033313   0.003253  -10.24   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.0683 on 353 degrees of freedom
## Multiple R-squared:  0.229,  Adjusted R-squared:  0.2268 
## F-statistic: 104.9 on 1 and 353 DF,  p-value: < 2.2e-16

1.3 Fixed Effects Regressions

The procedures used here on March 20, 2019 were guided by: https://www.princeton.edu/~otorres/Panel101R.pdf.

1.3.1 Fixed effects using least squares dummy variable model

Accuracy vs PGY with individual fixed effect:

fixed2 <-lm(Accuracy. ~ PGY.level + factor(Study.No) - 1, data=drh)
summary(fixed2)
## 
## Call:
## lm(formula = Accuracy. ~ PGY.level + factor(Study.No) - 1, data = drh)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.17850 -0.02459  0.00000  0.02604  0.16348 
## 
## Coefficients:
##                      Estimate Std. Error t value Pr(>|t|)    
## PGY.level           -0.031765   0.003631  -8.747 6.27e-16 ***
## factor(Study.No)1    0.979597   0.044709  21.910  < 2e-16 ***
## factor(Study.No)2    0.975778   0.044709  21.825  < 2e-16 ***
## factor(Study.No)3    0.934820   0.044709  20.909  < 2e-16 ***
## factor(Study.No)4    0.874404   0.044709  19.557  < 2e-16 ***
## factor(Study.No)5    0.971809   0.044709  21.736  < 2e-16 ***
## factor(Study.No)6    0.865297   0.044709  19.354  < 2e-16 ***
## factor(Study.No)7    1.008757   0.044709  22.562  < 2e-16 ***
## factor(Study.No)8    0.931007   0.044709  20.823  < 2e-16 ***
## factor(Study.No)9    0.943599   0.044709  21.105  < 2e-16 ***
## factor(Study.No)10   0.817549   0.044709  18.286  < 2e-16 ***
## factor(Study.No)11   0.970498   0.044709  21.707  < 2e-16 ***
## factor(Study.No)12   0.944320   0.031640  29.845  < 2e-16 ***
## factor(Study.No)13   0.977299   0.031640  30.888  < 2e-16 ***
## factor(Study.No)14   0.960143   0.031640  30.345  < 2e-16 ***
## factor(Study.No)15   0.981843   0.031640  31.031  < 2e-16 ***
## factor(Study.No)16   0.994779   0.031640  31.440  < 2e-16 ***
## factor(Study.No)17   0.987659   0.031640  31.215  < 2e-16 ***
## factor(Study.No)18   0.999508   0.031640  31.590  < 2e-16 ***
## factor(Study.No)19   0.862595   0.031640  27.262  < 2e-16 ***
## factor(Study.No)20   0.949843   0.031640  30.020  < 2e-16 ***
## factor(Study.No)21   0.951040   0.031640  30.058  < 2e-16 ***
## factor(Study.No)22   0.994018   0.031640  31.416  < 2e-16 ***
## factor(Study.No)23   1.003543   0.031640  31.717  < 2e-16 ***
## factor(Study.No)24   0.955257   0.035332  27.036  < 2e-16 ***
## factor(Study.No)25   0.917176   0.036655  25.022  < 2e-16 ***
## factor(Study.No)26   0.987149   0.036655  26.931  < 2e-16 ***
## factor(Study.No)27   0.932729   0.036655  25.446  < 2e-16 ***
## factor(Study.No)28   0.858707   0.036655  23.427  < 2e-16 ***
## factor(Study.No)29   0.859442   0.036655  23.447  < 2e-16 ***
## factor(Study.No)30   0.995020   0.036655  27.145  < 2e-16 ***
## factor(Study.No)31   1.017516   0.036655  27.759  < 2e-16 ***
## factor(Study.No)32   0.915948   0.036655  24.988  < 2e-16 ***
## factor(Study.No)33   0.974962   0.036655  26.598  < 2e-16 ***
## factor(Study.No)34   1.019053   0.036655  27.801  < 2e-16 ***
## factor(Study.No)35   0.991256   0.036655  27.043  < 2e-16 ***
## factor(Study.No)36   0.915634   0.036655  24.980  < 2e-16 ***
## factor(Study.No)37   1.003495   0.036655  27.377  < 2e-16 ***
## factor(Study.No)38   0.990974   0.036655  27.035  < 2e-16 ***
## factor(Study.No)39   1.004534   0.036655  27.405  < 2e-16 ***
## factor(Study.No)40   1.021547   0.036655  27.869  < 2e-16 ***
## factor(Study.No)41   1.050386   0.036655  28.656  < 2e-16 ***
## factor(Study.No)42   0.953162   0.031640  30.125  < 2e-16 ***
## factor(Study.No)43   0.924465   0.031640  29.218  < 2e-16 ***
## factor(Study.No)44   0.976293   0.031640  30.856  < 2e-16 ***
## factor(Study.No)45   0.918311   0.031640  29.023  < 2e-16 ***
## factor(Study.No)46   0.980018   0.031640  30.974  < 2e-16 ***
## factor(Study.No)47   0.965469   0.031640  30.514  < 2e-16 ***
## factor(Study.No)48   1.002507   0.031640  31.684  < 2e-16 ***
## factor(Study.No)49   0.923691   0.031640  29.193  < 2e-16 ***
## factor(Study.No)50   0.942006   0.031640  29.772  < 2e-16 ***
## factor(Study.No)51   0.994619   0.031640  31.435  < 2e-16 ***
## factor(Study.No)52   0.945342   0.031640  29.878  < 2e-16 ***
## factor(Study.No)53   1.008687   0.031640  31.880  < 2e-16 ***
## factor(Study.No)54   0.975300   0.031640  30.824  < 2e-16 ***
## factor(Study.No)55   0.922575   0.031640  29.158  < 2e-16 ***
## factor(Study.No)56   0.934751   0.062336  14.995  < 2e-16 ***
## factor(Study.No)57   0.851197   0.062336  13.655  < 2e-16 ***
## factor(Study.No)58   0.888963   0.062336  14.261  < 2e-16 ***
## factor(Study.No)59   0.912773   0.062336  14.643  < 2e-16 ***
## factor(Study.No)60   0.822711   0.062336  13.198  < 2e-16 ***
## factor(Study.No)61   0.656470   0.062336  10.531  < 2e-16 ***
## factor(Study.No)62   0.907547   0.062336  14.559  < 2e-16 ***
## factor(Study.No)63   0.870648   0.062336  13.967  < 2e-16 ***
## factor(Study.No)64   1.012773   0.062336  16.247  < 2e-16 ***
## factor(Study.No)65   0.993725   0.062336  15.941  < 2e-16 ***
## factor(Study.No)66   1.127059   0.062336  18.080  < 2e-16 ***
## factor(Study.No)67   0.854331   0.062336  13.705  < 2e-16 ***
## factor(Study.No)68   0.927059   0.062336  14.872  < 2e-16 ***
## factor(Study.No)69   0.903529   0.062336  14.494  < 2e-16 ***
## factor(Study.No)70   0.862353   0.062336  13.834  < 2e-16 ***
## factor(Study.No)71   0.979598   0.031640  30.960  < 2e-16 ***
## factor(Study.No)72   1.012805   0.031640  32.010  < 2e-16 ***
## factor(Study.No)73   0.954081   0.031640  30.154  < 2e-16 ***
## factor(Study.No)74   0.956311   0.031640  30.224  < 2e-16 ***
## factor(Study.No)75   0.929836   0.031640  29.388  < 2e-16 ***
## factor(Study.No)76   0.963468   0.031640  30.451  < 2e-16 ***
## factor(Study.No)77   1.014856   0.031640  32.075  < 2e-16 ***
## factor(Study.No)78   0.966582   0.031640  30.549  < 2e-16 ***
## factor(Study.No)79   0.954435   0.031640  30.165  < 2e-16 ***
## factor(Study.No)80   0.930560   0.043210  21.536  < 2e-16 ***
## factor(Study.No)81   0.781765   0.060729  12.873  < 2e-16 ***
## factor(Study.No)82   0.975161   0.060729  16.058  < 2e-16 ***
## factor(Study.No)83   0.951282   0.031640  30.065  < 2e-16 ***
## factor(Study.No)84   0.895109   0.031640  28.290  < 2e-16 ***
## factor(Study.No)85   0.850515   0.060729  14.005  < 2e-16 ***
## factor(Study.No)86   0.902732   0.060729  14.865  < 2e-16 ***
## factor(Study.No)87   0.940856   0.060729  15.493  < 2e-16 ***
## factor(Study.No)88   0.984890   0.060729  16.218  < 2e-16 ***
## factor(Study.No)89   0.986749   0.060729  16.248  < 2e-16 ***
## factor(Study.No)90   0.986567   0.060729  16.245  < 2e-16 ***
## factor(Study.No)91   0.927991   0.060729  15.281  < 2e-16 ***
## factor(Study.No)92   0.965098   0.060729  15.892  < 2e-16 ***
## factor(Study.No)93   0.965775   0.060729  15.903  < 2e-16 ***
## factor(Study.No)94   0.937868   0.060729  15.444  < 2e-16 ***
## factor(Study.No)95   0.919824   0.060729  15.146  < 2e-16 ***
## factor(Study.No)96   1.003694   0.060729  16.528  < 2e-16 ***
## factor(Study.No)97   0.935719   0.060729  15.408  < 2e-16 ***
## factor(Study.No)98   0.985611   0.060729  16.230  < 2e-16 ***
## factor(Study.No)99   0.994879   0.060729  16.382  < 2e-16 ***
## factor(Study.No)100  0.970361   0.060729  15.979  < 2e-16 ***
## factor(Study.No)101  0.968234   0.043210  22.408  < 2e-16 ***
## factor(Study.No)102  0.982069   0.043210  22.728  < 2e-16 ***
## factor(Study.No)103  0.932593   0.043210  21.583  < 2e-16 ***
## factor(Study.No)104  0.913758   0.043210  21.147  < 2e-16 ***
## factor(Study.No)105  0.970264   0.043210  22.455  < 2e-16 ***
## factor(Study.No)106  0.981959   0.043210  22.726  < 2e-16 ***
## factor(Study.No)107  0.890596   0.043210  20.611  < 2e-16 ***
## factor(Study.No)108  1.004860   0.043210  23.256  < 2e-16 ***
## factor(Study.No)109  0.922381   0.043210  21.347  < 2e-16 ***
## factor(Study.No)110  0.963564   0.043210  22.300  < 2e-16 ***
## factor(Study.No)111  0.901375   0.043210  20.861  < 2e-16 ***
## factor(Study.No)112  0.958891   0.043210  22.192  < 2e-16 ***
## factor(Study.No)113  0.938104   0.043210  21.711  < 2e-16 ***
## factor(Study.No)114  0.940350   0.035745  26.307  < 2e-16 ***
## factor(Study.No)115  0.940479   0.035745  26.311  < 2e-16 ***
## factor(Study.No)116  0.934124   0.035745  26.133  < 2e-16 ***
## factor(Study.No)117  0.917253   0.043816  20.934  < 2e-16 ***
## factor(Study.No)118  1.006004   0.060729  16.566  < 2e-16 ***
## factor(Study.No)119  0.975109   0.043816  22.255  < 2e-16 ***
## factor(Study.No)120  0.970014   0.035745  27.137  < 2e-16 ***
## factor(Study.No)121  0.961191   0.035745  26.891  < 2e-16 ***
## factor(Study.No)122  0.945694   0.035745  26.457  < 2e-16 ***
## factor(Study.No)123  0.864994   0.035745  24.199  < 2e-16 ***
## factor(Study.No)124  0.933818   0.035745  26.125  < 2e-16 ***
## factor(Study.No)125  0.913158   0.035745  25.547  < 2e-16 ***
## factor(Study.No)126  0.954559   0.035745  26.705  < 2e-16 ***
## factor(Study.No)127  0.936451   0.035745  26.198  < 2e-16 ***
## factor(Study.No)128  0.893006   0.035745  24.983  < 2e-16 ***
## factor(Study.No)129  0.898595   0.035745  25.139  < 2e-16 ***
## factor(Study.No)130  0.977348   0.035745  27.343  < 2e-16 ***
## factor(Study.No)131  0.968730   0.036655  26.428  < 2e-16 ***
## factor(Study.No)132  0.862032   0.036655  23.517  < 2e-16 ***
## factor(Study.No)133  0.974313   0.036655  26.580  < 2e-16 ***
## factor(Study.No)134  0.789681   0.044709  17.662  < 2e-16 ***
## factor(Study.No)135  0.976863   0.044709  21.849  < 2e-16 ***
## factor(Study.No)136  1.036694   0.044709  23.187  < 2e-16 ***
## factor(Study.No)137  0.935585   0.031640  29.569  < 2e-16 ***
## factor(Study.No)138  0.931558   0.031640  29.442  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.06062 on 216 degrees of freedom
## Multiple R-squared:  0.9971, Adjusted R-squared:  0.9952 
## F-statistic: 531.2 on 139 and 216 DF,  p-value: < 2.2e-16

1.3.2 Fixed effects using plm package

Including year as panel marker:

# fixed3 <- plm(Accuracy. ~ PGY.level, data=drh, index=c("Study.No", "PGY.level"), model="within")
# summary(fixed3)

Without year as panel marker:

fixed4 <- plm(Accuracy. ~ PGY.level, data=drh, index=c("Study.No"), model="within")
summary(fixed4)
## Oneway (individual) effect Within Model
## 
## Call:
## plm(formula = Accuracy. ~ PGY.level, data = drh, model = "within", 
##     index = c("Study.No"))
## 
## Unbalanced Panel: n = 138, T = 1-4, N = 355
## 
## Residuals:
##      Min.   1st Qu.    Median   3rd Qu.      Max. 
## -0.178502 -0.024591  0.000000  0.026042  0.163475 
## 
## Coefficients:
##             Estimate Std. Error t-value  Pr(>|t|)    
## PGY.level -0.0317647  0.0036314 -8.7472 6.268e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Total Sum of Squares:    1.0749
## Residual Sum of Squares: 0.79375
## R-Squared:      0.26157
## Adj. R-Squared: -0.2102
## F-statistic: 76.5141 on 1 and 216 DF, p-value: 6.2678e-16

1.4 Random effects regressions

1.4.1 Random effects with plm package

random1 <- plm(Accuracy. ~ PGY.level, data=drh, index=c("Study.No"), model="within")
summary(random1)
## Oneway (individual) effect Within Model
## 
## Call:
## plm(formula = Accuracy. ~ PGY.level, data = drh, model = "within", 
##     index = c("Study.No"))
## 
## Unbalanced Panel: n = 138, T = 1-4, N = 355
## 
## Residuals:
##      Min.   1st Qu.    Median   3rd Qu.      Max. 
## -0.178502 -0.024591  0.000000  0.026042  0.163475 
## 
## Coefficients:
##             Estimate Std. Error t-value  Pr(>|t|)    
## PGY.level -0.0317647  0.0036314 -8.7472 6.268e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Total Sum of Squares:    1.0749
## Residual Sum of Squares: 0.79375
## R-Squared:      0.26157
## Adj. R-Squared: -0.2102
## F-statistic: 76.5141 on 1 and 216 DF, p-value: 6.2678e-16

1.4.2 Random effects with lme4 package:

random2 <- lmer(Accuracy. ~ 1 + PGY.level + (1 | Study.No), data=drh)
summary(random2)
## Linear mixed model fit by REML ['lmerMod']
## Formula: Accuracy. ~ 1 + PGY.level + (1 | Study.No)
##    Data: drh
## 
## REML criterion at convergence: -888.6
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.8597 -0.4664  0.0637  0.6021  2.4075 
## 
## Random effects:
##  Groups   Name        Variance  Std.Dev.
##  Study.No (Intercept) 0.0008144 0.02854 
##  Residual             0.0038832 0.06231 
## Number of obs: 355, groups:  Study.No, 138
## 
## Fixed effects:
##              Estimate Std. Error t value
## (Intercept)  0.954187   0.008836  107.99
## PGY.level   -0.033454   0.003118  -10.73
## 
## Correlation of Fixed Effects:
##           (Intr)
## PGY.level -0.880
icc(random2)
## 
## Intraclass Correlation Coefficient for Linear mixed model
## 
## Family : gaussian (identity)
## Formula: Accuracy. ~ 1 + PGY.level + (1 | Study.No)
## 
##   ICC (Study.No): 0.1734
tab_model(random2, title="Random effects with lme4 package, nicer table")
## Computing p-values via Wald-statistics approximation (treating t as Wald z).
Random effects with lme4 package, nicer table
  Accuracy
Predictors Estimates CI p
(Intercept) 0.95 0.94 – 0.97 <0.001
PGY level -0.03 -0.04 – -0.03 <0.001
Random Effects
σ2 0.00
τ00 Study.No 0.00
ICC Study.No 0.17
Observations 355
Marginal R2 / Conditional R2 0.229 / 0.362