1 Accuracy

1.1 Descriptive data

Note that this type of data is called panel, repeated measures, or longitudinal data.

Average accuracy for each PGY:

##   Group.1         x
## 1       1 0.9262610
## 2       2 0.8886709
## 3       3 0.8430727
## 4       4 0.8304636

1.2 OLS without intercept

Accuracy vs PGY; OLS without intercept:

ols1 <-lm(Accuracy. ~ PGY.level - 1, data=drh)
summary(ols1)

## 
## Call:
## lm(formula = Accuracy. ~ PGY.level - 1, data = drh)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -0.6131 -0.1460  0.1412  0.4219  0.7144 
## 
## Coefficients:
##           Estimate Std. Error t value Pr(>|t|)    
## PGY.level 0.285621   0.007674   37.22   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.3956 on 354 degrees of freedom
## Multiple R-squared:  0.7965, Adjusted R-squared:  0.7959 
## F-statistic:  1385 on 1 and 354 DF,  p-value: < 2.2e-16

Accuracy vs PGY; OLS with intercept:

ols2 <-lm(Accuracy. ~ PGY.level, data=drh)
summary(ols2)

## 
## Call:
## lm(formula = Accuracy. ~ PGY.level, data = drh)
## 
## Residuals:
##       Min        1Q    Median        3Q       Max 
## -0.292700 -0.034981  0.009571  0.046318  0.177888 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  0.955364   0.008900  107.34   <2e-16 ***
## PGY.level   -0.033313   0.003253  -10.24   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.0683 on 353 degrees of freedom
## Multiple R-squared:  0.229,  Adjusted R-squared:  0.2268 
## F-statistic: 104.9 on 1 and 353 DF,  p-value: < 2.2e-16

1.3 Fixed Effects Regressions

The procedures used here on March 20, 2019 were guided by: https://www.princeton.edu/~otorres/Panel101R.pdf.

1.3.1 Fixed effects using least squares dummy variable model

Accuracy vs PGY with individual fixed effect:

fixed2 <-lm(Accuracy. ~ PGY.level + factor(Study.No) - 1, data=drh)
summary(fixed2)

## 
## Call:
## lm(formula = Accuracy. ~ PGY.level + factor(Study.No) - 1, data = drh)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.17850 -0.02459  0.00000  0.02604  0.16348 
## 
## Coefficients:
##                      Estimate Std. Error t value Pr(>|t|)    
## PGY.level           -0.031765   0.003631  -8.747 6.27e-16 ***
## factor(Study.No)1    0.979597   0.044709  21.910  < 2e-16 ***
## factor(Study.No)2    0.975778   0.044709  21.825  < 2e-16 ***
## factor(Study.No)3    0.934820   0.044709  20.909  < 2e-16 ***
## factor(Study.No)4    0.874404   0.044709  19.557  < 2e-16 ***
## factor(Study.No)5    0.971809   0.044709  21.736  < 2e-16 ***
## factor(Study.No)6    0.865297   0.044709  19.354  < 2e-16 ***
## factor(Study.No)7    1.008757   0.044709  22.562  < 2e-16 ***
## factor(Study.No)8    0.931007   0.044709  20.823  < 2e-16 ***
## factor(Study.No)9    0.943599   0.044709  21.105  < 2e-16 ***
## factor(Study.No)10   0.817549   0.044709  18.286  < 2e-16 ***
## factor(Study.No)11   0.970498   0.044709  21.707  < 2e-16 ***
## factor(Study.No)12   0.944320   0.031640  29.845  < 2e-16 ***
## factor(Study.No)13   0.977299   0.031640  30.888  < 2e-16 ***
## factor(Study.No)14   0.960143   0.031640  30.345  < 2e-16 ***
## factor(Study.No)15   0.981843   0.031640  31.031  < 2e-16 ***
## factor(Study.No)16   0.994779   0.031640  31.440  < 2e-16 ***
## factor(Study.No)17   0.987659   0.031640  31.215  < 2e-16 ***
## factor(Study.No)18   0.999508   0.031640  31.590  < 2e-16 ***
## factor(Study.No)19   0.862595   0.031640  27.262  < 2e-16 ***
## factor(Study.No)20   0.949843   0.031640  30.020  < 2e-16 ***
## factor(Study.No)21   0.951040   0.031640  30.058  < 2e-16 ***
## factor(Study.No)22   0.994018   0.031640  31.416  < 2e-16 ***
## factor(Study.No)23   1.003543   0.031640  31.717  < 2e-16 ***
## factor(Study.No)24   0.955257   0.035332  27.036  < 2e-16 ***
## factor(Study.No)25   0.917176   0.036655  25.022  < 2e-16 ***
## factor(Study.No)26   0.987149   0.036655  26.931  < 2e-16 ***
## factor(Study.No)27   0.932729   0.036655  25.446  < 2e-16 ***
## factor(Study.No)28   0.858707   0.036655  23.427  < 2e-16 ***
## factor(Study.No)29   0.859442   0.036655  23.447  < 2e-16 ***
## factor(Study.No)30   0.995020   0.036655  27.145  < 2e-16 ***
## factor(Study.No)31   1.017516   0.036655  27.759  < 2e-16 ***
## factor(Study.No)32   0.915948   0.036655  24.988  < 2e-16 ***
## factor(Study.No)33   0.974962   0.036655  26.598  < 2e-16 ***
## factor(Study.No)34   1.019053   0.036655  27.801  < 2e-16 ***
## factor(Study.No)35   0.991256   0.036655  27.043  < 2e-16 ***
## factor(Study.No)36   0.915634   0.036655  24.980  < 2e-16 ***
## factor(Study.No)37   1.003495   0.036655  27.377  < 2e-16 ***
## factor(Study.No)38   0.990974   0.036655  27.035  < 2e-16 ***
## factor(Study.No)39   1.004534   0.036655  27.405  < 2e-16 ***
## factor(Study.No)40   1.021547   0.036655  27.869  < 2e-16 ***
## factor(Study.No)41   1.050386   0.036655  28.656  < 2e-16 ***
## factor(Study.No)42   0.953162   0.031640  30.125  < 2e-16 ***
## factor(Study.No)43   0.924465   0.031640  29.218  < 2e-16 ***
## factor(Study.No)44   0.976293   0.031640  30.856  < 2e-16 ***
## factor(Study.No)45   0.918311   0.031640  29.023  < 2e-16 ***
## factor(Study.No)46   0.980018   0.031640  30.974  < 2e-16 ***
## factor(Study.No)47   0.965469   0.031640  30.514  < 2e-16 ***
## factor(Study.No)48   1.002507   0.031640  31.684  < 2e-16 ***
## factor(Study.No)49   0.923691   0.031640  29.193  < 2e-16 ***
## factor(Study.No)50   0.942006   0.031640  29.772  < 2e-16 ***
## factor(Study.No)51   0.994619   0.031640  31.435  < 2e-16 ***
## factor(Study.No)52   0.945342   0.031640  29.878  < 2e-16 ***
## factor(Study.No)53   1.008687   0.031640  31.880  < 2e-16 ***
## factor(Study.No)54   0.975300   0.031640  30.824  < 2e-16 ***
## factor(Study.No)55   0.922575   0.031640  29.158  < 2e-16 ***
## factor(Study.No)56   0.934751   0.062336  14.995  < 2e-16 ***
## factor(Study.No)57   0.851197   0.062336  13.655  < 2e-16 ***
## factor(Study.No)58   0.888963   0.062336  14.261  < 2e-16 ***
## factor(Study.No)59   0.912773   0.062336  14.643  < 2e-16 ***
## factor(Study.No)60   0.822711   0.062336  13.198  < 2e-16 ***
## factor(Study.No)61   0.656470   0.062336  10.531  < 2e-16 ***
## factor(Study.No)62   0.907547   0.062336  14.559  < 2e-16 ***
## factor(Study.No)63   0.870648   0.062336  13.967  < 2e-16 ***
## factor(Study.No)64   1.012773   0.062336  16.247  < 2e-16 ***
## factor(Study.No)65   0.993725   0.062336  15.941  < 2e-16 ***
## factor(Study.No)66   1.127059   0.062336  18.080  < 2e-16 ***
## factor(Study.No)67   0.854331   0.062336  13.705  < 2e-16 ***
## factor(Study.No)68   0.927059   0.062336  14.872  < 2e-16 ***
## factor(Study.No)69   0.903529   0.062336  14.494  < 2e-16 ***
## factor(Study.No)70   0.862353   0.062336  13.834  < 2e-16 ***
## factor(Study.No)71   0.979598   0.031640  30.960  < 2e-16 ***
## factor(Study.No)72   1.012805   0.031640  32.010  < 2e-16 ***
## factor(Study.No)73   0.954081   0.031640  30.154  < 2e-16 ***
## factor(Study.No)74   0.956311   0.031640  30.224  < 2e-16 ***
## factor(Study.No)75   0.929836   0.031640  29.388  < 2e-16 ***
## factor(Study.No)76   0.963468   0.031640  30.451  < 2e-16 ***
## factor(Study.No)77   1.014856   0.031640  32.075  < 2e-16 ***
## factor(Study.No)78   0.966582   0.031640  30.549  < 2e-16 ***
## factor(Study.No)79   0.954435   0.031640  30.165  < 2e-16 ***
## factor(Study.No)80   0.930560   0.043210  21.536  < 2e-16 ***
## factor(Study.No)81   0.781765   0.060729  12.873  < 2e-16 ***
## factor(Study.No)82   0.975161   0.060729  16.058  < 2e-16 ***
## factor(Study.No)83   0.951282   0.031640  30.065  < 2e-16 ***
## factor(Study.No)84   0.895109   0.031640  28.290  < 2e-16 ***
## factor(Study.No)85   0.850515   0.060729  14.005  < 2e-16 ***
## factor(Study.No)86   0.902732   0.060729  14.865  < 2e-16 ***
## factor(Study.No)87   0.940856   0.060729  15.493  < 2e-16 ***
## factor(Study.No)88   0.984890   0.060729  16.218  < 2e-16 ***
## factor(Study.No)89   0.986749   0.060729  16.248  < 2e-16 ***
## factor(Study.No)90   0.986567   0.060729  16.245  < 2e-16 ***
## factor(Study.No)91   0.927991   0.060729  15.281  < 2e-16 ***
## factor(Study.No)92   0.965098   0.060729  15.892  < 2e-16 ***
## factor(Study.No)93   0.965775   0.060729  15.903  < 2e-16 ***
## factor(Study.No)94   0.937868   0.060729  15.444  < 2e-16 ***
## factor(Study.No)95   0.919824   0.060729  15.146  < 2e-16 ***
## factor(Study.No)96   1.003694   0.060729  16.528  < 2e-16 ***
## factor(Study.No)97   0.935719   0.060729  15.408  < 2e-16 ***
## factor(Study.No)98   0.985611   0.060729  16.230  < 2e-16 ***
## factor(Study.No)99   0.994879   0.060729  16.382  < 2e-16 ***
## factor(Study.No)100  0.970361   0.060729  15.979  < 2e-16 ***
## factor(Study.No)101  0.968234   0.043210  22.408  < 2e-16 ***
## factor(Study.No)102  0.982069   0.043210  22.728  < 2e-16 ***
## factor(Study.No)103  0.932593   0.043210  21.583  < 2e-16 ***
## factor(Study.No)104  0.913758   0.043210  21.147  < 2e-16 ***
## factor(Study.No)105  0.970264   0.043210  22.455  < 2e-16 ***
## factor(Study.No)106  0.981959   0.043210  22.726  < 2e-16 ***
## factor(Study.No)107  0.890596   0.043210  20.611  < 2e-16 ***
## factor(Study.No)108  1.004860   0.043210  23.256  < 2e-16 ***
## factor(Study.No)109  0.922381   0.043210  21.347  < 2e-16 ***
## factor(Study.No)110  0.963564   0.043210  22.300  < 2e-16 ***
## factor(Study.No)111  0.901375   0.043210  20.861  < 2e-16 ***
## factor(Study.No)112  0.958891   0.043210  22.192  < 2e-16 ***
## factor(Study.No)113  0.938104   0.043210  21.711  < 2e-16 ***
## factor(Study.No)114  0.940350   0.035745  26.307  < 2e-16 ***
## factor(Study.No)115  0.940479   0.035745  26.311  < 2e-16 ***
## factor(Study.No)116  0.934124   0.035745  26.133  < 2e-16 ***
## factor(Study.No)117  0.917253   0.043816  20.934  < 2e-16 ***
## factor(Study.No)118  1.006004   0.060729  16.566  < 2e-16 ***
## factor(Study.No)119  0.975109   0.043816  22.255  < 2e-16 ***
## factor(Study.No)120  0.970014   0.035745  27.137  < 2e-16 ***
## factor(Study.No)121  0.961191   0.035745  26.891  < 2e-16 ***
## factor(Study.No)122  0.945694   0.035745  26.457  < 2e-16 ***
## factor(Study.No)123  0.864994   0.035745  24.199  < 2e-16 ***
## factor(Study.No)124  0.933818   0.035745  26.125  < 2e-16 ***
## factor(Study.No)125  0.913158   0.035745  25.547  < 2e-16 ***
## factor(Study.No)126  0.954559   0.035745  26.705  < 2e-16 ***
## factor(Study.No)127  0.936451   0.035745  26.198  < 2e-16 ***
## factor(Study.No)128  0.893006   0.035745  24.983  < 2e-16 ***
## factor(Study.No)129  0.898595   0.035745  25.139  < 2e-16 ***
## factor(Study.No)130  0.977348   0.035745  27.343  < 2e-16 ***
## factor(Study.No)131  0.968730   0.036655  26.428  < 2e-16 ***
## factor(Study.No)132  0.862032   0.036655  23.517  < 2e-16 ***
## factor(Study.No)133  0.974313   0.036655  26.580  < 2e-16 ***
## factor(Study.No)134  0.789681   0.044709  17.662  < 2e-16 ***
## factor(Study.No)135  0.976863   0.044709  21.849  < 2e-16 ***
## factor(Study.No)136  1.036694   0.044709  23.187  < 2e-16 ***
## factor(Study.No)137  0.935585   0.031640  29.569  < 2e-16 ***
## factor(Study.No)138  0.931558   0.031640  29.442  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.06062 on 216 degrees of freedom
## Multiple R-squared:  0.9971, Adjusted R-squared:  0.9952 
## F-statistic: 531.2 on 139 and 216 DF,  p-value: < 2.2e-16

1.3.2 Fixed effects using plm package

Including year as panel marker:

# fixed3 <- plm(Accuracy. ~ PGY.level, data=drh, index=c("Study.No", "PGY.level"), model="within")
# summary(fixed3)

Without year as panel marker:

fixed4 <- plm(Accuracy. ~ PGY.level, data=drh, index=c("Study.No"), model="within")
summary(fixed4)

## Oneway (individual) effect Within Model
## 
## Call:
## plm(formula = Accuracy. ~ PGY.level, data = drh, model = "within", 
##     index = c("Study.No"))
## 
## Unbalanced Panel: n = 138, T = 1-4, N = 355
## 
## Residuals:
##      Min.   1st Qu.    Median   3rd Qu.      Max. 
## -0.178502 -0.024591  0.000000  0.026042  0.163475 
## 
## Coefficients:
##             Estimate Std. Error t-value  Pr(>|t|)    
## PGY.level -0.0317647  0.0036314 -8.7472 6.268e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Total Sum of Squares:    1.0749
## Residual Sum of Squares: 0.79375
## R-Squared:      0.26157
## Adj. R-Squared: -0.2102
## F-statistic: 76.5141 on 1 and 216 DF, p-value: 6.2678e-16

1.4 Random effects regressions

1.4.1 Random effects with plm package

(It’s unclear if this is random slopes or random intercepts)

random1 <- plm(Accuracy. ~ PGY.level, data=drh, index=c("Study.No"), model="within")
summary(random1)

## Oneway (individual) effect Within Model
## 
## Call:
## plm(formula = Accuracy. ~ PGY.level, data = drh, model = "within", 
##     index = c("Study.No"))
## 
## Unbalanced Panel: n = 138, T = 1-4, N = 355
## 
## Residuals:
##      Min.   1st Qu.    Median   3rd Qu.      Max. 
## -0.178502 -0.024591  0.000000  0.026042  0.163475 
## 
## Coefficients:
##             Estimate Std. Error t-value  Pr(>|t|)    
## PGY.level -0.0317647  0.0036314 -8.7472 6.268e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Total Sum of Squares:    1.0749
## Residual Sum of Squares: 0.79375
## R-Squared:      0.26157
## Adj. R-Squared: -0.2102
## F-statistic: 76.5141 on 1 and 216 DF, p-value: 6.2678e-16

1.4.2 Random intercepts with lme4 package

random2 <- lmer(Accuracy. ~ 1 + PGY.level + (1 | Study.No), data=drh)
summary(random2)

## Linear mixed model fit by REML ['lmerMod']
## Formula: Accuracy. ~ 1 + PGY.level + (1 | Study.No)
##    Data: drh
## 
## REML criterion at convergence: -888.6
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.8597 -0.4664  0.0637  0.6021  2.4075 
## 
## Random effects:
##  Groups   Name        Variance  Std.Dev.
##  Study.No (Intercept) 0.0008144 0.02854 
##  Residual             0.0038832 0.06231 
## Number of obs: 355, groups:  Study.No, 138
## 
## Fixed effects:
##              Estimate Std. Error t value
## (Intercept)  0.954187   0.008836  107.99
## PGY.level   -0.033454   0.003118  -10.73
## 
## Correlation of Fixed Effects:
##           (Intr)
## PGY.level -0.880

icc(random2)

## 
## Intraclass Correlation Coefficient for Linear mixed model
## 
## Family : gaussian (identity)
## Formula: Accuracy. ~ 1 + PGY.level + (1 | Study.No)
## 
##   ICC (Study.No): 0.1734

tab_model(random2, title="Random intercepts with lme4 package, nicer table")

## Computing p-values via Wald-statistics approximation (treating t as Wald z).

Random intercepts with lme4 package, nicer table
	Accuracy
Predictors	Estimates	CI	p
(Intercept)	0.95	0.94 – 0.97	<0.001
PGY level	-0.03	-0.04 – -0.03	<0.001
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.17
Observations	355
Marginal R² / Conditional R²	0.229 / 0.362

1.4.3 Random slopes with lme4 package:

AccRandom3 <- lmer(Accuracy. ~ 1 + PGY.level + (1 + PGY.level | Study.No), data=drh)

## singular fit

summary(AccRandom3)

## Linear mixed model fit by REML ['lmerMod']
## Formula: Accuracy. ~ 1 + PGY.level + (1 + PGY.level | Study.No)
##    Data: drh
## 
## REML criterion at convergence: -903.7
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -4.1435 -0.4775  0.1319  0.6255  2.4060 
## 
## Random effects:
##  Groups   Name        Variance  Std.Dev. Corr 
##  Study.No (Intercept) 0.0001218 0.01104       
##           PGY.level   0.0002726 0.01651  -1.00
##  Residual             0.0034623 0.05884       
## Number of obs: 355, groups:  Study.No, 138
## 
## Fixed effects:
##              Estimate Std. Error t value
## (Intercept)  0.956272   0.007935  120.52
## PGY.level   -0.034562   0.003333  -10.37
## 
## Correlation of Fixed Effects:
##           (Intr)
## PGY.level -0.865
## convergence code: 0
## singular fit

icc(AccRandom3)

## Caution! ICC for random-slope-intercept models usually not meaningful. Use `adjusted = TRUE` to use the mean random effect variance to calculate the ICC. See 'Note' in `?icc`.

## 
## Intraclass Correlation Coefficient for Linear mixed model
## 
## Family : gaussian (identity)
## Formula: Accuracy. ~ 1 + PGY.level + (1 + PGY.level | Study.No)
## 
##   ICC (Study.No): 0.0340

tab_model(AccRandom3, title="Random slopes with lme4 package, nicer table")

## Computing p-values via Wald-statistics approximation (treating t as Wald z).

## Caution! ICC for random-slope-intercept models usually not meaningful. Use `adjusted = TRUE` to use the mean random effect variance to calculate the ICC. See 'Note' in `?icc`.

Random slopes with lme4 package, nicer table
	Accuracy
Predictors	Estimates	CI	p
(Intercept)	0.96	0.94 – 0.97	<0.001
PGY level	-0.03	-0.04 – -0.03	<0.001
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
τ₁₁ _{Study.No.PGY.level}	0.00
ρ₀₁ _Study.No	-1.00
ICC _Study.No	0.03
Observations	355

2 TLS rate

2.1 Descriptive data

Average TLS rate for each PGY:

##   Group.1          x
## 1       1 0.05584216
## 2       2 0.08635816
## 3       3 0.12799056
## 4       4 0.14164259

2.2 OLS without intercept

TLS rate vs PGY; OLS without intercept:

ols1 <-lm(TLS_rate ~ PGY.level - 1, data=drh)
summary(ols1)

## 
## Call:
## lm(formula = TLS_rate ~ PGY.level - 1, data = drh)
## 
## Residuals:
##       Min        1Q    Median        3Q       Max 
## -0.157306 -0.032457 -0.001784  0.036379  0.272265 
## 
## Coefficients:
##           Estimate Std. Error t value Pr(>|t|)    
## PGY.level  0.03933    0.00122   32.24   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.06287 on 354 degrees of freedom
## Multiple R-squared:  0.746,  Adjusted R-squared:  0.7452 
## F-statistic:  1039 on 1 and 354 DF,  p-value: < 2.2e-16

TLS rate vs PGY; OLS with intercept:

ols2 <-lm(TLS_rate ~ PGY.level, data=drh)
summary(ols2)

## 
## Call:
## lm(formula = TLS_rate ~ PGY.level, data = drh)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.14785 -0.03673 -0.01087  0.02814  0.27231 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 0.028181   0.008067   3.494 0.000537 ***
## PGY.level   0.029919   0.002949  10.147  < 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.0619 on 353 degrees of freedom
## Multiple R-squared:  0.2258, Adjusted R-squared:  0.2236 
## F-statistic:   103 on 1 and 353 DF,  p-value: < 2.2e-16

2.3 Fixed Effects Regressions

The procedures used here on March 20, 2019 were guided by: https://www.princeton.edu/~otorres/Panel101R.pdf.

2.3.1 Fixed effects using least squares dummy variable model

TLS rate vs PGY with individual fixed effect:

fixed2 <-lm(TLS_rate ~ PGY.level + factor(Study.No) - 1, data=drh)
summary(fixed2)

## 
## Call:
## lm(formula = TLS_rate ~ PGY.level + factor(Study.No) - 1, data = drh)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.14821 -0.02300  0.00000  0.02003  0.14075 
## 
## Coefficients:
##                       Estimate Std. Error t value Pr(>|t|)    
## PGY.level            0.0277621  0.0032366   8.578 1.89e-15 ***
## factor(Study.No)1    0.0344115  0.0398488   0.864 0.388792    
## factor(Study.No)2   -0.0061940  0.0398488  -0.155 0.876622    
## factor(Study.No)3    0.0791892  0.0398488   1.987 0.048159 *  
## factor(Study.No)4    0.1306765  0.0398488   3.279 0.001213 ** 
## factor(Study.No)5    0.0378144  0.0398488   0.949 0.343708    
## factor(Study.No)6    0.1053280  0.0398488   2.643 0.008815 ** 
## factor(Study.No)7   -0.0330993  0.0398488  -0.831 0.407104    
## factor(Study.No)8    0.0679642  0.0398488   1.706 0.089529 .  
## factor(Study.No)9    0.0450757  0.0398488   1.131 0.259239    
## factor(Study.No)10   0.1570105  0.0398488   3.940 0.000110 ***
## factor(Study.No)11   0.0207210  0.0398488   0.520 0.603603    
## factor(Study.No)12   0.0263683  0.0282006   0.935 0.350819    
## factor(Study.No)13   0.0109752  0.0282006   0.389 0.697523    
## factor(Study.No)14   0.0125342  0.0282006   0.444 0.657152    
## factor(Study.No)15   0.0077900  0.0282006   0.276 0.782632    
## factor(Study.No)16  -0.0301946  0.0282006  -1.071 0.285496    
## factor(Study.No)17  -0.0003719  0.0282006  -0.013 0.989491    
## factor(Study.No)18  -0.0081873  0.0282006  -0.290 0.771847    
## factor(Study.No)19   0.1287876  0.0282006   4.567 8.32e-06 ***
## factor(Study.No)20   0.0244195  0.0282006   0.866 0.387495    
## factor(Study.No)21   0.0129495  0.0282006   0.459 0.646558    
## factor(Study.No)22   0.0003676  0.0282006   0.013 0.989612    
## factor(Study.No)23   0.0007812  0.0282006   0.028 0.977926    
## factor(Study.No)24   0.0292169  0.0314911   0.928 0.354557    
## factor(Study.No)25   0.0670580  0.0326703   2.053 0.041320 *  
## factor(Study.No)26   0.0065934  0.0326703   0.202 0.840250    
## factor(Study.No)27   0.0535380  0.0326703   1.639 0.102724    
## factor(Study.No)28   0.1208173  0.0326703   3.698 0.000276 ***
## factor(Study.No)29   0.1173769  0.0326703   3.593 0.000405 ***
## factor(Study.No)30   0.0084668  0.0326703   0.259 0.795759    
## factor(Study.No)31  -0.0277975  0.0326703  -0.851 0.395795    
## factor(Study.No)32   0.0608482  0.0326703   1.862 0.063891 .  
## factor(Study.No)33   0.0076362  0.0326703   0.234 0.815413    
## factor(Study.No)34  -0.0220138  0.0326703  -0.674 0.501148    
## factor(Study.No)35   0.0066675  0.0326703   0.204 0.838479    
## factor(Study.No)36   0.0738073  0.0326703   2.259 0.024872 *  
## factor(Study.No)37  -0.0131680  0.0326703  -0.403 0.687305    
## factor(Study.No)38   0.0057973  0.0326703   0.177 0.859321    
## factor(Study.No)39  -0.0045602  0.0326703  -0.140 0.889120    
## factor(Study.No)40  -0.0245286  0.0326703  -0.751 0.453594    
## factor(Study.No)41  -0.0493730  0.0326703  -1.511 0.132186    
## factor(Study.No)42   0.0472780  0.0282006   1.676 0.095089 .  
## factor(Study.No)43   0.0530222  0.0282006   1.880 0.061429 .  
## factor(Study.No)44   0.0217954  0.0282006   0.773 0.440443    
## factor(Study.No)45   0.0543311  0.0282006   1.927 0.055343 .  
## factor(Study.No)46   0.0035405  0.0282006   0.126 0.900207    
## factor(Study.No)47   0.0164414  0.0282006   0.583 0.560490    
## factor(Study.No)48  -0.0091705  0.0282006  -0.325 0.745354    
## factor(Study.No)49   0.0604781  0.0282006   2.145 0.033104 *  
## factor(Study.No)50   0.0331901  0.0282006   1.177 0.240519    
## factor(Study.No)51   0.0006362  0.0282006   0.023 0.982021    
## factor(Study.No)52   0.0207029  0.0282006   0.734 0.463665    
## factor(Study.No)53  -0.0294200  0.0282006  -1.043 0.298003    
## factor(Study.No)54   0.0216749  0.0282006   0.769 0.442971    
## factor(Study.No)55   0.0721605  0.0282006   2.559 0.011186 *  
## factor(Study.No)56   0.0427977  0.0555591   0.770 0.441958    
## factor(Study.No)57   0.1303308  0.0555591   2.346 0.019892 *  
## factor(Study.No)58   0.1032372  0.0555591   1.858 0.064508 .  
## factor(Study.No)59   0.0318087  0.0555591   0.573 0.567566    
## factor(Study.No)60   0.1063428  0.0555591   1.914 0.056936 .  
## factor(Study.No)61   0.3007162  0.0555591   5.413 1.65e-07 ***
## factor(Study.No)62   0.1084637  0.0555591   1.952 0.052204 .  
## factor(Study.No)63   0.0684387  0.0555591   1.232 0.219356    
## factor(Study.No)64  -0.0110485  0.0555591  -0.199 0.842559    
## factor(Study.No)65   0.0222849  0.0555591   0.401 0.688742    
## factor(Study.No)66  -0.1110485  0.0555591  -1.999 0.046890 *  
## factor(Study.No)67   0.1616788  0.0555591   2.910 0.003992 ** 
## factor(Study.No)68  -0.0443818  0.0555591  -0.799 0.425271    
## factor(Study.No)69   0.0536574  0.0555591   0.966 0.335239    
## factor(Study.No)70   0.1536574  0.0555591   2.766 0.006173 ** 
## factor(Study.No)71   0.0065014  0.0282006   0.231 0.817890    
## factor(Study.No)72  -0.0193757  0.0282006  -0.687 0.492777    
## factor(Study.No)73   0.0272970  0.0282006   0.968 0.334148    
## factor(Study.No)74   0.0345886  0.0282006   1.227 0.221339    
## factor(Study.No)75   0.0176216  0.0282006   0.625 0.532718    
## factor(Study.No)76   0.0200157  0.0282006   0.710 0.478618    
## factor(Study.No)77  -0.0134207  0.0282006  -0.476 0.634625    
## factor(Study.No)78   0.0317146  0.0282006   1.125 0.262004    
## factor(Study.No)79   0.0250385  0.0282006   0.888 0.375599    
## factor(Study.No)80   0.0524390  0.0385120   1.362 0.174734    
## factor(Study.No)81   0.1222379  0.0541265   2.258 0.024921 *  
## factor(Study.No)82   0.0288417  0.0541265   0.533 0.594681    
## factor(Study.No)83   0.0389038  0.0282006   1.380 0.169155    
## factor(Study.No)84   0.0910921  0.0282006   3.230 0.001430 ** 
## factor(Study.No)85   0.1034879  0.0541265   1.912 0.057204 .  
## factor(Study.No)86   0.0797648  0.0541265   1.474 0.142026    
## factor(Study.No)87   0.0411084  0.0541265   0.759 0.448389    
## factor(Study.No)88   0.0086962  0.0541265   0.161 0.872508    
## factor(Study.No)89  -0.0020386  0.0541265  -0.038 0.969990    
## factor(Study.No)90   0.0061362  0.0541265   0.113 0.909845    
## factor(Study.No)91   0.0571435  0.0541265   1.056 0.292266    
## factor(Study.No)92   0.0389045  0.0541265   0.719 0.473059    
## factor(Study.No)93   0.0229993  0.0541265   0.425 0.671320    
## factor(Study.No)94   0.0426604  0.0541265   0.788 0.431467    
## factor(Study.No)95   0.0841782  0.0541265   1.555 0.121360    
## factor(Study.No)96  -0.0032007  0.0541265  -0.059 0.952900    
## factor(Study.No)97   0.0626334  0.0541265   1.157 0.248483    
## factor(Study.No)98   0.0081353  0.0541265   0.150 0.880667    
## factor(Study.No)99   0.0009264  0.0541265   0.017 0.986360    
## factor(Study.No)100  0.0248695  0.0541265   0.459 0.646359    
## factor(Study.No)101  0.0089283  0.0385120   0.232 0.816888    
## factor(Study.No)102  0.0127567  0.0385120   0.331 0.740785    
## factor(Study.No)103  0.0284535  0.0385120   0.739 0.460818    
## factor(Study.No)104  0.0745984  0.0385120   1.937 0.054047 .  
## factor(Study.No)105  0.0136669  0.0385120   0.355 0.723030    
## factor(Study.No)106  0.0130160  0.0385120   0.338 0.735712    
## factor(Study.No)107  0.0641261  0.0385120   1.665 0.097344 .  
## factor(Study.No)108 -0.0067618  0.0385120  -0.176 0.860792    
## factor(Study.No)109  0.0709423  0.0385120   1.842 0.066834 .  
## factor(Study.No)110  0.0297817  0.0385120   0.773 0.440185    
## factor(Study.No)111  0.0622155  0.0385120   1.615 0.107666    
## factor(Study.No)112  0.0255817  0.0385120   0.664 0.507237    
## factor(Study.No)113  0.0396740  0.0385120   1.030 0.304081    
## factor(Study.No)114  0.0497759  0.0318586   1.562 0.119657    
## factor(Study.No)115  0.0456783  0.0318586   1.434 0.153080    
## factor(Study.No)116  0.0575494  0.0318586   1.806 0.072248 .  
## factor(Study.No)117  0.0745763  0.0390522   1.910 0.057502 .  
## factor(Study.No)118 -0.0066848  0.0541265  -0.124 0.901823    
## factor(Study.No)119 -0.0080640  0.0390522  -0.206 0.836600    
## factor(Study.No)120  0.0152688  0.0318586   0.479 0.632233    
## factor(Study.No)121  0.0287920  0.0318586   0.904 0.367138    
## factor(Study.No)122  0.0294335  0.0318586   0.924 0.356580    
## factor(Study.No)123  0.1206545  0.0318586   3.787 0.000197 ***
## factor(Study.No)124  0.0497974  0.0318586   1.563 0.119499    
## factor(Study.No)125  0.0522598  0.0318586   1.640 0.102384    
## factor(Study.No)126  0.0060695  0.0318586   0.191 0.849086    
## factor(Study.No)127  0.0392122  0.0318586   1.231 0.219729    
## factor(Study.No)128  0.0775215  0.0318586   2.433 0.015775 *  
## factor(Study.No)129  0.0855111  0.0318586   2.684 0.007837 ** 
## factor(Study.No)130  0.0014657  0.0318586   0.046 0.963347    
## factor(Study.No)131  0.0260689  0.0326703   0.798 0.425782    
## factor(Study.No)132  0.1154947  0.0326703   3.535 0.000499 ***
## factor(Study.No)133  0.0136485  0.0326703   0.418 0.676535    
## factor(Study.No)134  0.1913611  0.0398488   4.802 2.93e-06 ***
## factor(Study.No)135  0.0274054  0.0398488   0.688 0.492358    
## factor(Study.No)136 -0.0599260  0.0398488  -1.504 0.134085    
## factor(Study.No)137  0.0624666  0.0282006   2.215 0.027799 *  
## factor(Study.No)138  0.0447329  0.0282006   1.586 0.114148    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.05403 on 216 degrees of freedom
## Multiple R-squared:  0.8855, Adjusted R-squared:  0.8119 
## F-statistic: 12.02 on 139 and 216 DF,  p-value: < 2.2e-16

2.3.2 Fixed effects using plm package

Including year as panel marker:

# fixed3 <- plm(TLS_rate. ~ PGY.level, data=drh, index=c("Study.No", "PGY.level"), model="within")
# summary(fixed3)

Without year as panel marker:

fixed4 <- plm(TLS_rate ~ PGY.level, data=drh, index=c("Study.No"), model="within")
summary(fixed4)

## Oneway (individual) effect Within Model
## 
## Call:
## plm(formula = TLS_rate ~ PGY.level, data = drh, model = "within", 
##     index = c("Study.No"))
## 
## Unbalanced Panel: n = 138, T = 1-4, N = 355
## 
## Residuals:
##      Min.   1st Qu.    Median   3rd Qu.      Max. 
## -0.148205 -0.022999  0.000000  0.020030  0.140748 
## 
## Coefficients:
##            Estimate Std. Error t-value  Pr(>|t|)    
## PGY.level 0.0277621  0.0032366  8.5775 1.892e-15 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Total Sum of Squares:    0.84533
## Residual Sum of Squares: 0.63055
## R-Squared:      0.25408
## Adj. R-Squared: -0.22248
## F-statistic: 73.5742 on 1 and 216 DF, p-value: 1.8923e-15

2.4 Random effects regressions

2.4.1 Random effects with plm package

(It’s unclear if this is random slopes or random intercepts).

random1 <- plm(TLS_rate ~ PGY.level, data=drh, index=c("Study.No"), model="within")
summary(random1)

## Oneway (individual) effect Within Model
## 
## Call:
## plm(formula = TLS_rate ~ PGY.level, data = drh, model = "within", 
##     index = c("Study.No"))
## 
## Unbalanced Panel: n = 138, T = 1-4, N = 355
## 
## Residuals:
##      Min.   1st Qu.    Median   3rd Qu.      Max. 
## -0.148205 -0.022999  0.000000  0.020030  0.140748 
## 
## Coefficients:
##            Estimate Std. Error t-value  Pr(>|t|)    
## PGY.level 0.0277621  0.0032366  8.5775 1.892e-15 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Total Sum of Squares:    0.84533
## Residual Sum of Squares: 0.63055
## R-Squared:      0.25408
## Adj. R-Squared: -0.22248
## F-statistic: 73.5742 on 1 and 216 DF, p-value: 1.8923e-15

2.4.2 Random intercepts with lme4 package

random2 <- lmer(TLS_rate ~ 1 + PGY.level + (1 | Study.No), data=drh)
summary(random2)

## Linear mixed model fit by REML ['lmerMod']
## Formula: TLS_rate ~ 1 + PGY.level + (1 | Study.No)
##    Data: drh
## 
## REML criterion at convergence: -961.5
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.5360 -0.5508 -0.1165  0.4332  3.9300 
## 
## Random effects:
##  Groups   Name        Variance  Std.Dev.
##  Study.No (Intercept) 0.0008102 0.02846 
##  Residual             0.0030571 0.05529 
## Number of obs: 355, groups:  Study.No, 138
## 
## Fixed effects:
##             Estimate Std. Error t value
## (Intercept) 0.030332   0.007978   3.802
## PGY.level   0.029745   0.002792  10.654
## 
## Correlation of Fixed Effects:
##           (Intr)
## PGY.level -0.872

icc(random2)

## 
## Intraclass Correlation Coefficient for Linear mixed model
## 
## Family : gaussian (identity)
## Formula: TLS_rate ~ 1 + PGY.level + (1 | Study.No)
## 
##   ICC (Study.No): 0.2095

tab_model(random2, title="Random effects with lme4 package, nicer table")

## Computing p-values via Wald-statistics approximation (treating t as Wald z).

Random effects with lme4 package, nicer table
	TLS rate
Predictors	Estimates	CI	p
(Intercept)	0.03	0.01 – 0.05	<0.001
PGY level	0.03	0.02 – 0.04	<0.001
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.21
Observations	355
Marginal R² / Conditional R²	0.222 / 0.385

2.4.3 Random slopes with lme4 package:

# TLSRandom3 <- lmer(TLS_rate ~ 1 + PGY.level + (1 + PGY.level | Study.No), data=drh)
TLSRandom3 <- lmer(TLS_rate ~ 1 + PGY.level + (1 | Study.No), data=drh) # REML = FALSE makes it like stata apparently
summary(TLSRandom3)

## Linear mixed model fit by REML ['lmerMod']
## Formula: TLS_rate ~ 1 + PGY.level + (1 | Study.No)
##    Data: drh
## 
## REML criterion at convergence: -961.5
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.5360 -0.5508 -0.1165  0.4332  3.9300 
## 
## Random effects:
##  Groups   Name        Variance  Std.Dev.
##  Study.No (Intercept) 0.0008102 0.02846 
##  Residual             0.0030571 0.05529 
## Number of obs: 355, groups:  Study.No, 138
## 
## Fixed effects:
##             Estimate Std. Error t value
## (Intercept) 0.030332   0.007978   3.802
## PGY.level   0.029745   0.002792  10.654
## 
## Correlation of Fixed Effects:
##           (Intr)
## PGY.level -0.872

icc(TLSRandom3)

## 
## Intraclass Correlation Coefficient for Linear mixed model
## 
## Family : gaussian (identity)
## Formula: TLS_rate ~ 1 + PGY.level + (1 | Study.No)
## 
##   ICC (Study.No): 0.2095

tab_model(TLSRandom3, title="Random slopes with lme4 package, nicer table")

## Computing p-values via Wald-statistics approximation (treating t as Wald z).

Random slopes with lme4 package, nicer table
	TLS rate
Predictors	Estimates	CI	p
(Intercept)	0.03	0.01 – 0.05	<0.001
PGY level	0.03	0.02 – 0.04	<0.001
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.21
Observations	355
Marginal R² / Conditional R²	0.222 / 0.385

3 Random slopes – 2019-04-25

3.1 Random slopes model - same as before

Random slopes model
	TLS rate
Predictors	Estimates	CI	p
(Intercept)	0.03	0.01 – 0.05	<0.001
PGY level	0.03	0.02 – 0.04	<0.001
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.21
Observations	355
Marginal R² / Conditional R²	0.222 / 0.385

3.2 Random slopes model with squared PGY term

Random slopes model with squared PGY term
	TLS rate
Predictors	Estimates	CI	p
(Intercept)	0.01	-0.03 – 0.04	0.699
PGY level	0.05	0.02 – 0.08	<0.001
PGY level squared	-0.00	-0.01 – 0.00	0.107
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.22
Observations	355
Marginal R² / Conditional R²	0.226 / 0.396

3.3 Random slopes model with start year

Random slopes model with squared PGY term
	TLS rate
Predictors	Estimates	CI	p
(Intercept)	2.09	-5.94 – 10.13	0.610
PGY level	0.03	0.02 – 0.04	<0.001
academic year start	-0.00	-0.01 – 0.00	0.615
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.21
Observations	355
Marginal R² / Conditional R²	0.224 / 0.391

3.4 Random slopes model with PGY as factor variable

The regressions below suggest that there may be a leveling off in the relationship between TLS_rate and PGY.level as PGY.level increases. According to these results, there is no statistically significant difference (at the p = 0.05 level) in TLS_rate when PGY.level = 3 and when PGY.level = 4.

3.4.1 PGY = 1 as reference category

Random slopes model with squared PGY term
	TLS rate
Predictors	Estimates	CI	p
(Intercept)	0.06	0.04 – 0.07	<0.001
as factor(PGY level)2	0.03	0.02 – 0.05	<0.001
as factor(PGY level)3	0.07	0.06 – 0.09	<0.001
as factor(PGY level)4	0.09	0.07 – 0.10	<0.001
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.22
Observations	355
Marginal R² / Conditional R²	0.229 / 0.398

3.4.2 PGY = 2 as reference category

Random slopes model with squared PGY term
	TLS rate
Predictors	Estimates	CI	p
(Intercept)	0.09	0.08 – 0.10	<0.001
factor(as factor(PGY level),levels=c(“2”,“1”,“3”,“4”))1	-0.03	-0.05 – -0.02	<0.001
factor(as factor(PGY level),levels=c(“2”,“1”,“3”,“4”))3	0.04	0.02 – 0.06	<0.001
factor(as factor(PGY level),levels=c(“2”,“1”,“3”,“4”))4	0.05	0.04 – 0.07	<0.001
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.22
Observations	355
Marginal R² / Conditional R²	0.229 / 0.398

3.4.3 PGY = 3 as reference category

Random slopes model with squared PGY term
	TLS rate
Predictors	Estimates	CI	p
(Intercept)	0.13	0.12 – 0.14	<0.001
factor(as factor(PGY level),levels=c(“3”,“1”,“2”,“4”))1	-0.07	-0.09 – -0.06	<0.001
factor(as factor(PGY level),levels=c(“3”,“1”,“2”,“4”))2	-0.04	-0.06 – -0.02	<0.001
factor(as factor(PGY level),levels=c(“3”,“1”,“2”,“4”))4	0.01	-0.00 – 0.03	0.128
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.22
Observations	355
Marginal R² / Conditional R²	0.229 / 0.398

3.4.4 PGY = 4 as reference category

Random slopes model with squared PGY term
	TLS rate
Predictors	Estimates	CI	p
(Intercept)	0.14	0.13 – 0.16	<0.001
factor(as factor(PGY level),levels=c(“4”,“1”,“2”,“3”))1	-0.09	-0.10 – -0.07	<0.001
factor(as factor(PGY level),levels=c(“4”,“1”,“2”,“3”))2	-0.05	-0.07 – -0.04	<0.001
factor(as factor(PGY level),levels=c(“4”,“1”,“2”,“3”))3	-0.01	-0.03 – 0.00	0.128
Random Effects
σ²	0.00
τ₀₀ _Study.No	0.00
ICC _Study.No	0.22
Observations	355
Marginal R² / Conditional R²	0.229 / 0.398

4 Reference materials and resources

https://www.princeton.edu/~otorres/Panel101R.pdf
http://www.polsci.ucsb.edu/faculty/glasgow/ps207/ps207_class1.r
https://mpra.ub.uni-muenchen.de/76869/1/MPRA_paper_76869.pdf
https://rpsychologist.com/r-guide-longitudinal-lme-lmer (covers AR1 structure as well)
https://rpubs.com/rslbliss/r_mlm_ws
https://www.jaredknowles.com/journal/2013/11/25/getting-started-with-mixed-effect-models-in-r
https://rpsychologist.com/r-guide-longitudinal-lme-lmer - Using R and lme/lmer to fit different two- and three-level longitudinal modelsPosted by Kristoffer Magnusson on 2015-04-21 17:30:00+02:00 in R (very very good)

R. Haney Project on Ultrasound Panel Data

Prepared by Rachel with help from Anshul

Version – 25 April 2019