Scenario 1: Null association- 1.5 year followup, no censoring or competing risks

True RR: 1

True RD: 0

Relative risk performance

estimator variance_estimator Qint DetQ bias variance mse bias_se_ratio coverage o.coverage mean_ci_width power
LASSO ic Yes No -0.135 0.015 0.033 -1.100 80.5 80.5 0.473 19.5
LASSO tmle Yes No -0.135 0.015 0.033 -1.100 80.5 80.5 0.475 19.5
GLM tmle Yes No -0.125 0.015 0.031 -1.015 83.5 81.5 0.481 16.5
LASSO ic No No -0.076 0.015 0.021 -0.622 89.5 90.0 0.441 10.5
GLM ic No No -0.048 0.015 0.017 -0.396 91.5 93.0 0.433 8.5
LASSO tmle No No -0.076 0.015 0.021 -0.622 93.0 90.0 0.486 7.0
GLM tmle No No -0.048 0.015 0.017 -0.396 94.5 93.0 0.469 5.5

Risk difference performance

estimator variance_estimator Qint DetQ bias variance mse bias_se_ratio coverage o.coverage mean_ci_width power
LASSO ic Yes No -0.00283 1e-05 1e-05 -1.09626 76.0 80.5 0.00989 24.0
LASSO tmle Yes No -0.00283 1e-05 1e-05 -1.09626 76.0 80.5 0.00992 24.0
GLM tmle Yes No -0.00262 1e-05 1e-05 -1.00219 78.5 83.5 0.01014 21.5
LASSO ic No No -0.00158 1e-05 1e-05 -0.57686 85.0 91.0 0.00974 15.0
GLM ic No No -0.00095 1e-05 1e-05 -0.33884 88.0 94.0 0.00982 12.0
LASSO tmle No No -0.00158 1e-05 1e-05 -0.57679 88.5 91.0 0.01066 11.5
GLM tmle No No -0.00095 1e-05 1e-05 -0.33884 91.5 94.0 0.01058 8.5

Notes

  • True RR: 1
  • True RD: 0
  • Dementia prevalence: 13.5%
  • GLM performs better than LASSO
  • IC still anti-conservative
  • Intercept-only Q prediction doesn’t improve prediction
  • Better performance estimating RR compared to RD

Scenario 2: Null association- 1.5 year followup, censoring and competing risks

Relative risk performance

estimator variance_estimator Qint DetQ bias variance mse bias_se_ratio coverage o.coverage mean_ci_width power
LASSO ic No No -0.206 0.036 0.079 -1.081 69.0 80.0 0.583 31.0
LASSO ic Yes No -0.206 0.036 0.079 -1.081 69.0 80.0 0.583 31.0
LASSO tmle Yes No -0.206 0.036 0.079 -1.081 71.0 80.0 0.605 29.0
LASSO ic No Yes -0.079 0.036 0.042 -0.421 86.0 93.5 0.572 14.0
LASSO NA No Yes NA NA NA NA 89.5 NA NA 10.5
LASSO tmle No No -0.098 0.039 0.049 -0.497 93.0 91.5 0.686 7.0

Risk difference performance

estimator variance_estimator Qint DetQ bias variance mse bias_se_ratio coverage o.coverage mean_ci_width power
LASSO ic No No -0.00336 1e-05 2e-05 -1.10400 63.5 79.0 0.00941 36.5
LASSO ic Yes No -0.00336 1e-05 2e-05 -1.10400 63.5 79.0 0.00941 36.5
LASSO tmle Yes No -0.00336 1e-05 2e-05 -1.10400 65.0 79.0 0.00967 35.0
LASSO ic No Yes -0.00109 1e-05 1e-05 -0.36050 83.5 93.5 0.00931 16.5
LASSO tmle No No -0.00164 1e-05 2e-05 -0.43727 86.0 93.5 0.01296 14.0
LASSO NA No Yes NA NA NA NA 89.5 NA 0.01127 10.5

Notes

  • Higher bias and variance with censoring and competing risks
  • Bootstrap coverage in between IC and TMLE variance estimators
  • Bootstrap with clustered ID’s in CV procedures (also with Deterministic Q)
  • Deterministic Q increases performance

Scenario 3: Simulated outcome- 1.5 year followup

Relative risk performance - uncensored data

estimator variance_estimator Qint DetQ bias variance mse bias_se_ratio coverage o.coverage mean_ci_width power
LASSO ic No No -0.394 2.015 2.170 -0.278 70.0 99.5 1.842 65.0
LASSO ic Yes No -0.494 1.962 2.206 -0.353 72.0 99.5 2.031 65.0
LASSO ic No No 0.110 0.032 0.044 0.613 84.0 93.0 0.596 99.5
LASSO tmle No No 0.110 0.032 0.044 0.613 89.5 93.0 0.686 99.5
LASSO ic Yes No 0.035 0.032 0.034 0.193 93.0 94.5 0.646 100.0
LASSO tmle No No -0.393 2.017 2.172 -0.277 93.5 99.5 Inf 0.0
LASSO tmle Yes No 0.035 0.032 0.034 0.193 96.5 94.5 0.713 100.0
LASSO tmle Yes No -0.494 1.959 2.203 -0.353 97.0 99.5 Inf 0.0

Relative risk performance - censoring and competing risks

estimator variance_estimator Qint DetQ bias variance mse bias_se_ratio coverage o.coverage mean_ci_width power
LASSO ic No Yes 0.060 0.106 0.109 0.183 73.0 94.5 0.764 90.0
LASSO ic No No -0.041 0.111 0.113 -0.123 74.0 94.0 0.701 96.0
LASSO ic Yes No -0.220 0.105 0.154 -0.680 80.0 91.0 0.904 99.0
LASSO bootstrap-no replacement No Yes NA NA NA NA 84.5 NA NA 92.0
LASSO bootstrap-clustered ID No No NA NA NA NA 89.5 NA NA 87.0
LASSO bootstrap-clustered ID No Yes NA NA NA NA 90.5 NA NA 79.0
LASSO tmle No No -0.041 0.111 0.112 -0.122 94.0 94.0 1.219 89.0
LASSO tmle Yes No -0.220 0.105 0.154 -0.680 99.5 91.0 1.427 97.5
  • True RR: 0.43

Scenario 4: Simulated outcome- 5 year followup

Relative risk performance

estimator variance_estimator Qint DetQ censoring_in_data bias variance mse bias_se_ratio coverage o.coverage mean_ci_width power
GLM ic No No Censored+Competing risks 0.355 0.123 0.249 1.012 16.5 81.0 0.185 90
LASSO ic No No Censored+Competing risks 0.027 0.096 0.097 0.088 37.0 97.5 0.308 100
GLM tmle No No Censored+Competing risks 0.355 0.123 0.249 1.012 47.5 81.0 0.777 60
LASSO NA No Yes Censored+Competing risks NA NA NA NA 86.5 NA NA 58
LASSO tmle No No Censored+Competing risks 0.027 0.095 0.096 0.086 94.5 97.5 1.359 75
LASSO tmle Yes No Censored+Competing risks -0.266 0.088 0.159 -0.897 100.0 84.0 1.477 100
  • True RR: 0.43

Scenario 4: Old simulated data

Relative risk performance

##  [1] "GLM Q-intercept LASSO prescreen -TMLE var"
##  [2] "LASSO IC - common outcome"                
##  [3] "LASSO no DetQ Qint tmle"                  
##  [4] "LASSO no DetQ Qint tmle- interactions"    
##  [5] "LASSO no DetQ Qint tmle - common outcome" 
##  [6] "LASSO no DetQ Qint tmle updated estimator"
##  [7] "LASSO no DetQ tmle"                       
##  [8] "LASSO no DetQ tmle updated estimator"     
##  [9] "LASSO Qint tmle AUC"                      
## [10] "LASSO tmle - common outcome"              
## [11] "Ridge no DetQ ic - est update"            
## [12] "Ridge no DetQ Qint iptw"                  
## [13] "Ridge no DetQ Qint iptw 1se"              
## [14] "Ridge no DetQ Qint tmle"                  
## [15] "Ridge no DetQ tmle - est update"          
## [16] "unadj, no detQ, ic"                       
## [17] "unadj, no detQ, tmle"                     
## [18] "unadj, Q-intercept, no detQ, ic"          
## [19] "unadj, Q-intercept, no detQ, tmle"        
## [20] "Bootstrap - clustered ID"
bias_logRR variance_logRR mse bias_se_ratio coverage o.coverage o.coverage_logRR mean_ci_width_logRR power o.power mean_ci_width
NA NA NA NA 87.500 NA NA NA 58.000 NA 1.772
-0.271 0.282 0.355 -0.509 98.551 95.652 92.754 3.348 5.797 42.029 NA

–>

–> –> –> –>