This document generally explains the fundamental concepts of the rmda R package and replicates (with some modifications) some of the analyses from https://mdbrown.github.io/rmda/.

The risk models discussed in this package stem from clinical decision-making and they pertain to medical patients.

R package rmda

The package was created by Marshall Brown in 2018. Before loading the package, install it with install.packages('rmda')

library('rmda')
# The dataset "dcaData" is included in the `rmda` package and contains simulated data
data(dcaData)
str(dcaData)
## tibble [500 × 6] (S3: tbl_df/tbl/data.frame)
##  $ Age    : int [1:500] 33 29 28 27 23 35 34 29 35 27 ...
##  $ Female : num [1:500] 1 1 1 0 1 1 1 1 1 1 ...
##  $ Smokes : logi [1:500] FALSE FALSE FALSE FALSE FALSE FALSE ...
##  $ Marker1: num [1:500] 0.245 0.943 0.774 0.406 0.508 ...
##  $ Marker2: num [1:500] 1.02108 -0.25576 0.33184 -0.00569 0.20753 ...
##  $ Cancer : int [1:500] 0 0 0 0 0 0 0 0 0 0 ...
head(dcaData)
## # A tibble: 6 x 6
##     Age Female Smokes Marker1  Marker2 Cancer
##   <int>  <dbl> <lgl>    <dbl>    <dbl>  <int>
## 1    33      1 FALSE    0.245  1.02         0
## 2    29      1 FALSE    0.943 -0.256        0
## 3    28      1 FALSE    0.774  0.332        0
## 4    27      0 FALSE    0.406 -0.00569      0
## 5    23      1 FALSE    0.508  0.208        0
## 6    35      1 FALSE    0.186  1.41         0
summary(dcaData)
##       Age            Female       Smokes           Marker1        
##  Min.   :20.00   Min.   :0.00   Mode :logical   Min.   :0.003914  
##  1st Qu.:23.00   1st Qu.:0.00   FALSE:413       1st Qu.:0.269323  
##  Median :28.00   Median :1.00   TRUE :87        Median :0.515159  
##  Mean   :27.75   Mean   :0.57                   Mean   :0.510257  
##  3rd Qu.:32.00   3rd Qu.:1.00                   3rd Qu.:0.757597  
##  Max.   :35.00   Max.   :1.00                   Max.   :0.993512  
##     Marker2             Cancer    
##  Min.   :-2.93853   Min.   :0.00  
##  1st Qu.:-0.60150   1st Qu.:0.00  
##  Median : 0.04613   Median :0.00  
##  Mean   : 0.05221   Mean   :0.12  
##  3rd Qu.: 0.70680   3rd Qu.:0.00  
##  Max.   : 2.87567   Max.   :1.00

There are 500 observations and six variables in the dataset: Age, Female, Smoking status, Marker 1, Marker 2 and a binary variable Cancer.

Main functions in the package

  • decision_curve => Estimate (standardized) net benefit curves with bootstrap confidence intervals.
  • plot_decision_curve => Plot a decision curve or multiple curves.
  • plot_clinical_impact and plot_roc_components => Alternative plots for the output of decision_curve showing measures of clinical impact or the components of the ROC curve (true/false positive rates) across a range of risk thresholds.
  • cv_decision_curve => Calculate k-fold cross-validated estimates of a decision curve and its components.

Decision curve

The decision_curve function is used to create a decision curve object for a logistic model. This allows one to predict cancer status using features such as age, gender, and smoking status.

The “opt-in” policy, which is a default in the decision_curve function, is a risk model where patients are categorized as “high-risk”, with a recommendation to treat these high-risk patients with some intervention. Meaning, that treatment is assigned only to a particular part of the population estimated to need the intervention. The relative costs and benefits of such a policy (=net benefit) are specified as the ‘cost’ of treating an individual without the event (a control) compared to the ‘benefit’ of treating an observation with the event (a case).

Then, the decision curve is plotted using the plot_decision_curve function.

# The author sets seed at 123; I do the same throughout this document to replicate the results
set.seed(123)
  
# the below model is run with default settings and 50 bootstraps
baseline.model <- decision_curve(Cancer~ Age + Female + Smokes, #fitting a logistic model
                                 data = dcaData,
                                 study.design = "cohort",
                                 policy = "opt-in", #default
                                 bootstraps = 50)
## Note:  The data provided is used to both fit a prediction model and to estimate the respective decision curve. This may cause bias in decision curve estimates leading to over-confidence in model performance.
#plotting the decision curve
plot1 <- plot_decision_curve(baseline.model, curve.names="baseline model")

The baseline model was without two variables describing markers. Next, a model with both markers is built on top of the baseline model.

set.seed(123)
full.model <- decision_curve(Cancer~ Age + Female + Smokes + Marker1 + Marker2,
                                 data = dcaData,
                                 bootstraps = 50)
## Note:  The data provided is used to both fit a prediction model and to estimate the respective decision curve. This may cause bias in decision curve estimates leading to over-confidence in model performance.

The plot_decision_curve function can be fed with a list of both models to plot them in one graph.

plot2 <- plot_decision_curve( list(baseline.model, full.model), 
                   curve.names = c("Baseline model", "Full model"), xlim = c(0, 1), legend.position = "bottomright") 
## Note: When multiple decision curves are plotted, decision curves for 'All' are calculated using the prevalence from the first DecisionCurve object in the list provided.

Alternative plotting functions

The authors offer two more useful plotting functions:

  • plot_roc_components plots the components of the ROC curve. The ROC curve (receiver operating characteristic curve) illustrates a binary classifier system’s diagnostic ability as its discrimination threshold is varied. It is created by plotting true positive rate against false positive rate over a range of high-risk thresholds. It can also be thought of as a plot of the power as a function of the Type I Error of the decision rule.
  • plot_clinical_impact function plots the number classified as high risk and the number with the outcome classified as high risk for a given high risk threshold (if we were to use the specified model to classify 1,000 hypothetical subjects).
plot_roc_components(full.model,  xlim = c(0, 0.4), 
                    col = c("black", "red"))

plot_clinical_impact(full.model, xlim = c(0, .4), 
                     col = c("black", "blue"))

Changing the deafult policy from “opt-in” to “opt-out”

In the above baseline and full model, the default policy of “opt-in” was implemented. It is possible to change the policy to “opt-out”. This may be useful when treatment is recommended to an entire patient population. A risk model’s potential use in this setting is to identify ‘low-risk’ patients and recommend that those patients ‘opt-out’ of treatment.

#to calculate the net benefit for an ‘opt-out’ policy, set policy = 'opt-out' when using the decision_curve function
set.seed(123)
opt.out.dc <- decision_curve(Cancer~Age + Female + Smokes + Marker1 + Marker2,
              data = dcaData, 
              bootstraps = 50, 
              policy = 'opt-out')
## Note:  The data provided is used to both fit a prediction model and to estimate the respective decision curve. This may cause bias in decision curve estimates leading to over-confidence in model performance.
plot_decision_curve( opt.out.dc,  xlim = c(0, 1), ylim = c(-.2,2), 
                     standardize = FALSE, 
                     curve.names = "model", legend = 'bottomright')

When policy is set to ‘opt-out’, the clinical impact plot displays the number considered low risk at each threshold instead of the number high risk.

plot_clinical_impact( opt.out.dc, col = c("black", "blue"), legend = 'bottomright')

Getting model estimates

To get the point estimated, one uses the summary function. Measures that can be printed out are “sNB”, which is the net benefit for a model with the default option “opt-in”, “NB” which is the net benefit for the out-out policy, “TPR” for true positive rates, and “FPR” for false-positive rates. One can also specify how many decimal places to print, with an ‘nround’ option.

#getting standardized net benefit with confidence intervals
summary(full.model, measure="sNB")
## 
## Standardized Net Benefit (95% Confidence Intervals):
## --------------------------------------------------------------------------------------------------------
##    risk      cost:benefit     percent              All            Cancer ~ Age + Female + Smokes   None 
##  threshold      ratio        high risk                                 + Marker1 + Marker2              
## ----------- -------------- -------------- ---------------------- -------------------------------- ------
##      0           0:1            100                 1                           1                   0   
##                              (100, 100)           (1, 1)                      (1, 1)                    
## 
##    0.01          1:99           75.2              0.926                       0.913                 0   
##                             (56.2, 83.4)      (0.898, 0.939)              (0.855, 0.953)                
## 
##    0.02          1:49           64.8               0.85                       0.876                 0   
##                             (45.6, 71.4)      (0.794, 0.877)              (0.791, 0.916)                
## 
##    0.03          3:97            58               0.773                       0.813                 0   
##                              (40.2, 65)       (0.687, 0.813)              (0.726, 0.879)                
## 
##    0.04          1:24           51.4              0.694                       0.759                 0   
##                             (35.6, 60.4)      (0.579, 0.748)              (0.681, 0.851)                
## 
##    0.05          1:19           45.6              0.614                        0.73                 0   
##                              (32, 54.2)       (0.468, 0.682)              (0.652, 0.83)                 
## 
##    0.06          3:47           39.8              0.532                        0.71                 0   
##                             (28.4, 48.4)      (0.355, 0.614)              (0.64, 0.819)                 
## 
##    0.07          7:93           36.4              0.448                       0.704                 0   
##                             (27.4, 44.2)      (0.239, 0.545)              (0.596, 0.808)                
## 
##    0.08          2:23           34.2              0.362                       0.676                 0   
##                             (25.6, 40.8)      (0.121, 0.475)              (0.579, 0.793)                
## 
##    0.09          9:91           32.6              0.275                       0.665                 0   
##                             (24.6, 39.2)        (0, 0.402)                (0.56, 0.783)                 
## 
##     0.1          1:9             31               0.185                       0.639                 0   
##                             (23.4, 36.8)     (-0.123, 0.329)              (0.542, 0.765)                
## 
##    0.11         11:89           29.8              0.094                       0.629                 0   
##                             (22.8, 34.4)      (-0.25, 0.253)              (0.535, 0.757)                
## 
##    0.12          3:22           28.4                0                         0.624                 0   
##                              (22.2, 33)      (-0.379, 0.176)              (0.525, 0.754)                
## 
##    0.13         13:87           27.2              -0.096                      0.619                 0   
##                             (21.8, 31.2)     (-0.511, 0.097)              (0.51, 0.748)                 
## 
##    0.14          7:43            26               -0.194                      0.616                 0   
##                             (20.2, 30.8)     (-0.646, 0.016)              (0.491, 0.735)                
## 
##    0.15          3:17           25.6              -0.294                      0.604                 0   
##                             (19.8, 29.2)     (-0.784, -0.066)             (0.491, 0.731)                
## 
##    0.16          4:21           24.4              -0.397                      0.585                 0   
##                               (19, 29)       (-0.926, -0.151)             (0.425, 0.702)                
## 
##    0.17         17:83            23               -0.502                      0.591                 0   
##                             (18.2, 28.6)     (-1.071, -0.238)             (0.417, 0.696)                
## 
##    0.18          9:41           22.8              -0.61                       0.559                 0   
##                             (17.8, 26.4)     (-1.22, -0.326)              (0.437, 0.708)                
## 
##    0.19         19:81           22.4              -0.72                       0.529                 0   
##                              (17, 25.2)      (-1.372, -0.417)             (0.38, 0.705)                 
## 
##     0.2          1:4             21               -0.833                      0.521                 0   
##                              (16.2, 25)      (-1.528, -0.511)             (0.375, 0.669)                
## 
##    0.21         21:79           20.4              -0.949                      0.497                 0   
##                             (13.8, 23.8)     (-1.688, -0.606)             (0.311, 0.66)                 
## 
##    0.22         11:39            19               -1.068                      0.451                 0   
##                             (13.4, 23.2)     (-1.852, -0.704)             (0.281, 0.659)                
## 
##    0.23         23:77           17.8              -1.19                       0.444                 0   
##                             (12.4, 22.6)     (-2.02, -0.805)              (0.281, 0.622)                
## 
##    0.24          6:19           17.2              -1.316                      0.425                 0   
##                              (12, 22.2)      (-2.193, -0.908)             (0.235, 0.616)                
## 
##    0.25          1:3            16.4              -1.444                      0.389                 0   
##                              (11.6, 21)      (-2.37, -1.014)              (0.248, 0.602)                
## 
##    0.26         13:37            16               -1.577                      0.342                 0   
##                              (10.8, 21)      (-2.553, -1.123)             (0.224, 0.57)                 
## 
##    0.27         27:73           15.6              -1.712                      0.341                 0   
##                              (10.2, 21)      (-2.74, -1.235)              (0.21, 0.556)                 
## 
##    0.28          7:18            15               -1.852                      0.347                 0   
##                             (9.6, 19.8)      (-2.932, -1.35)              (0.209, 0.523)                
## 
##    0.29         29:71           14.6              -1.995                      0.325                 0   
##                             (9.2, 19.2)      (-3.13, -1.468)              (0.205, 0.531)                
## 
##     0.3          3:7            14.2              -2.143                      0.302                 0   
##                              (8.4, 19)       (-3.333, -1.59)              (0.205, 0.521)                
## 
##    0.31         31:69           13.8              -2.295                       0.28                 0   
##                              (8.2, 19)       (-3.543, -1.715)             (0.199, 0.504)                
## 
##    0.32          8:17           13.2              -2.451                      0.267                 0   
##                              (7.8, 17)       (-3.758, -1.843)             (0.113, 0.506)                
## 
##    0.33         33:67           12.8              -2.612                      0.221                 0   
##                             (7.2, 16.6)      (-3.98, -1.976)              (0.145, 0.503)                
## 
##    0.34         17:33           12.4              -2.778                      0.225                 0   
##                             (7.2, 15.8)      (-4.209, -2.113)             (0.112, 0.513)                
## 
##    0.35          7:13           11.4              -2.949                      0.258                 0   
##                             (6.6, 14.8)      (-4.444, -2.254)             (0.134, 0.488)                
## 
##    0.36          9:16            11               -3.125                       0.24                 0   
##                              (6.4, 14)       (-4.688, -2.399)             (0.137, 0.481)                
## 
##    0.37         37:63            10               -3.307                      0.251                 0   
##                              (6.2, 14)       (-4.938, -2.549)             (0.122, 0.514)                
## 
##    0.38         19:31           9.6               -3.495                      0.262                 0   
##                              (5.4, 14)       (-5.197, -2.703)             (0.118, 0.492)                
## 
##    0.39         39:61            9                -3.689                      0.258                 0   
##                             (5.2, 13.4)      (-5.464, -2.863)             (0.143, 0.485)                
## 
##     0.4          2:3            8.2               -3.889                      0.294                 0   
##                             (5.2, 12.8)      (-5.741, -3.028)             (0.134, 0.466)                
## 
##    0.41         41:59           7.6               -4.096                      0.323                 0   
##                             (5.2, 12.4)      (-6.026, -3.199)             (0.118, 0.477)                
## 
##    0.42         21:29           7.4               -4.31                       0.329                 0   
##                             (5.2, 12.4)      (-6.322, -3.375)             (0.099, 0.454)                
## 
##    0.43         43:57           7.2               -4.532                      0.308                 0   
##                               (5, 12)        (-6.628, -3.558)             (0.098, 0.447)                
## 
##    0.44         11:14            7                -4.762                      0.286                 0   
##                             (4.6, 11.8)      (-6.944, -3.747)             (0.125, 0.418)                
## 
##    0.45          9:11           6.4                 -5                         0.23                 0   
##                             (4.4, 11.6)      (-7.273, -3.944)             (0.098, 0.428)                
## 
##    0.46         23:27           6.2               -5.247                      0.208                 0   
##                              (4, 10.8)       (-7.613, -4.147)             (0.081, 0.422)                
## 
##    0.47         47:53            6                -5.503                      0.186                 0   
##                             (3.6, 10.4)      (-7.966, -4.358)             (0.041, 0.385)                
## 
##    0.48         12:13           5.8               -5.769                      0.195                 0   
##                              (3, 10.2)       (-8.333, -4.577)             (0.05, 0.378)                 
## 
##    0.49         49:51           5.6               -6.046                      0.173                 0   
##                               (3, 10)        (-8.715, -4.805)             (0.054, 0.353)                
## 
##     0.5          1:1            5.2               -6.333                      0.167                 0   
##                              (2.8, 9.8)      (-9.111, -5.042)             (0.053, 0.352)                
## 
##    0.51         51:49            5                -6.633                      0.179                 0   
##                              (2.8, 9.8)      (-9.524, -5.289)             (0.059, 0.348)                
## 
##    0.52         13:12           4.8               -6.944                      0.157                 0   
##                              (2.8, 9.8)      (-9.954, -5.546)             (0.054, 0.364)                
## 
##    0.53         53:47           4.2               -7.27                       0.137                 0   
##                              (2.4, 9.4)     (-10.402, -5.814)             (0.049, 0.328)                
## 
##    0.54         27:23           4.2               -7.609                      0.133                 0   
##                              (1.6, 8.2)      (-10.87, -6.093)             (0.041, 0.325)                
## 
##    0.55          11:9           3.8               -7.963                      0.131                 0   
##                              (1.6, 8.2)     (-11.358, -6.385)             (0.033, 0.347)                
## 
##    0.56         14:11           3.8               -8.333                      0.127                 0   
##                               (1.6, 8)       (-11.869, -6.69)             (0.026, 0.317)                
## 
##    0.57         57:43           3.8               -8.721                      0.123                 0   
##                               (1.6, 8)      (-12.403, -7.009)             (0.02, 0.309)                 
## 
##    0.58         29:21           3.6               -9.127                      0.141                 0   
##                              (1.6, 7.4)     (-12.963, -7.344)             (0.012, 0.305)                
## 
##    0.59         59:41           3.4               -9.553                      0.161                 0   
##                              (1.2, 7.2)      (-13.55, -7.695)             (0.037, 0.296)                
## 
##     0.6          3:2            3.4                -10                        0.158                 0   
##                              (1.2, 6.8)     (-14.167, -8.063)             (0.01, 0.344)                 
## 
##    0.61         61:39           3.2               -10.47                      0.181                 0   
##                              (1.2, 6.6)     (-14.815, -8.451)             (0.006, 0.322)                
## 
##    0.62         31:19            3               -10.965                      0.162                 0   
##                              (1.2, 6.4)     (-15.497, -8.858)             (0.002, 0.29)                 
## 
##    0.63         63:37            3               -11.486                       0.16                 0   
##                              (1.2, 5.8)     (-16.216, -9.288)             (0.026, 0.288)                
## 
##    0.64          16:9           2.8              -12.037                      0.141                 0   
##                              (1.2, 5.6)     (-16.975, -9.742)             (0.02, 0.286)                 
## 
##    0.65          13:7           2.8              -12.619                      0.138                 0   
##                               (1.2, 5)      (-17.778, -10.221)            (0.026, 0.286)                
## 
##    0.66         33:17           2.8              -13.235                      0.135                 0   
##                                (1, 5)       (-18.627, -10.729)            (0.04, 0.269)                 
## 
##    0.67         67:33           2.8              -13.889                      0.132                 0   
##                               (1, 4.8)      (-19.529, -11.268)            (0.035, 0.264)                
## 
##    0.68          17:8           2.8              -14.583                      0.129                 0   
##                               (1, 4.8)      (-20.486, -11.84)             (0.027, 0.258)                
## 
##    0.69         69:31           2.8              -15.323                      0.126                 0   
##                              (0.8, 4.6)     (-21.505, -12.449)            (0.021, 0.262)                
## 
##     0.7          7:3            2.6              -16.111                      0.161                 0   
##                              (0.8, 4.6)     (-22.593, -13.099)           (-0.016, 0.262)                
## 
##    0.71         71:29           2.2              -16.954                      0.126                 0   
##                              (0.8, 4.4)     (-23.755, -13.793)           (-0.022, 0.262)                
## 
##    0.72          18:7           1.8              -17.857                       0.09                 0   
##                              (0.8, 4.2)       (-25, -14.537)             (-0.028, 0.249)                
## 
##    0.73         73:27           1.6              -18.827                      0.072                 0   
##                               (0.8, 4)      (-26.337, -15.336)           (-0.051, 0.246)                
## 
##    0.74         37:13           1.6              -19.872                      0.069                 0   
##                               (0.6, 4)      (-27.778, -16.197)           (-0.078, 0.244)                
## 
##    0.75          3:1            1.6                -21                        0.067                 0   
##                               (0.4, 4)      (-29.333, -17.127)           (-0.086, 0.246)                
## 
##    0.76          19:6           1.6              -22.222                      0.064                 0   
##                              (0.4, 3.6)     (-31.019, -18.134)           (-0.074, 0.246)                
## 
##    0.77         77:23           1.6              -23.551                      0.061                 0   
##                              (0.2, 3.6)     (-32.85, -19.228)            (-0.083, 0.218)                
## 
##    0.78         39:11           1.2                -25                         0.1                  0   
##                              (0.2, 3.6)     (-34.848, -20.423)           (-0.092, 0.206)                
## 
##    0.79         79:21           1.2              -26.587                       0.1                  0   
##                              (0.2, 3.6)     (-37.037, -21.73)            (-0.023, 0.186)                
## 
##     0.8          4:1            1.2              -28.333                       0.1                  0   
##                               (0.2, 3)      (-39.444, -23.169)            (-0.03, 0.186)                
## 
##    0.81         81:19           1.2              -30.263                       0.1                  0   
##                              (0.2, 2.8)     (-42.105, -24.759)            (-0.038, 0.18)                
## 
##    0.82          41:9           1.2              -32.407                       0.1                  0   
##                               (0, 2.6)      (-45.062, -26.526)           (-0.046, 0.194)                
## 
##    0.83         83:17           1.2              -34.804                       0.1                  0   
##                               (0, 2.6)       (-48.366, -28.5)            (-0.048, 0.194)                
## 
##    0.84          21:4            1                -37.5                       0.083                 0   
##                               (0, 2.6)      (-52.083, -30.722)           (-0.054, 0.177)                
## 
##    0.85          17:3           0.8              -40.556                      0.067                 0   
##                               (0, 2.6)      (-56.296, -33.239)           (-0.061, 0.177)                
## 
##    0.86          43:7           0.8              -44.048                      0.067                 0   
##                               (0, 2.2)      (-61.111, -36.117)              (0, 0.172)                  
## 
##    0.87         87:13           0.4              -48.077                      0.033                 0   
##                               (0, 2.2)      (-66.667, -39.437)              (0, 0.172)                  
## 
##    0.88          22:3           0.2              -52.778                      0.017                 0   
##                               (0, 2.2)      (-73.148, -43.31)               (0, 0.172)                  
## 
##    0.89         89:11            0               -58.333                        0                   0   
##                               (0, 1.8)      (-80.808, -47.887)              (0, 0.125)                  
## 
##     0.9          9:1             0                 -65                          0                   0   
##                               (0, 1.6)        (-90, -53.38)                 (0, 0.125)                  
## 
##    0.91          91:9            0               -73.148                        0                   0   
##                               (0, 1.4)     (-101.235, -60.094)              (0, 0.121)                  
## 
##    0.92          23:2            0               -83.333                        0                   0   
##                               (0, 1.2)     (-115.278, -68.486)              (0, 0.097)                  
## 
##    0.93          93:7            0               -96.429                        0                   0   
##                                (0, 1)      (-133.333, -79.276)              (0, 0.081)                  
## 
##    0.94          47:3            0               -113.889                       0                   0   
##                                (0, 1)      (-157.407, -93.662)              (0, 0.081)                  
## 
##    0.95          19:1            0               -138.333                       0                   0   
##                               (0, 0.8)     (-191.111, -113.803)             (0, 0.065)                  
## 
##    0.96          24:1            0                 -175                         0                   0   
##                               (0, 0.4)     (-241.667, -144.014)             (0, 0.04)                   
## 
##    0.97          97:3            0               -236.111                       0                   0   
##                               (0, 0.4)     (-325.926, -194.366)             (0, 0.031)                  
## 
##    0.98          49:1            0               -358.333                       0                   0   
##                                (0, 0)      (-494.444, -295.07)                (0, 0)                    
## 
##    0.99          99:1            0                 -725                         0                   0   
##                                (0, 0)       (-1000, -597.183)                 (0, 0)                    
## 
##      1          Inf:1            0                  NA                          NA                  NA  
##                                (0, 0)            (NA, NA)                    (NA, NA)                   
## --------------------------------------------------------------------------------------------------------
#getting true positive rates with two decimal places
summary(full.model, nround = 2, measure = "TPR")
## 
## Sensitivity (TPR) (95% Confidence Intervals):
## ------------------------------------------------------------------------------------------
##    risk      cost:benefit     percent       All     Cancer ~ Age + Female + Smokes   None 
##  threshold      ratio        high risk                   + Marker1 + Marker2              
## ----------- -------------- -------------- -------- -------------------------------- ------
##      0           0:1            100          1                    1                   0   
##                              (100, 100)    (1, 1)               (1, 1)                    
## 
##    0.01          1:99           75.2         1                   0.97                 0   
##                             (56.2, 83.4)   (1, 1)             (0.93, 1)                   
## 
##    0.02          1:49           64.8         1                   0.97                 0   
##                             (45.6, 71.4)   (1, 1)             (0.89, 1)                   
## 
##    0.03          3:97            58          1                   0.93                 0   
##                              (40.2, 65)    (1, 1)            (0.87, 0.99)                 
## 
##    0.04          1:24           51.4         1                   0.9                  0   
##                             (35.6, 60.4)   (1, 1)            (0.84, 0.97)                 
## 
##    0.05          1:19           45.6         1                   0.88                 0   
##                              (32, 54.2)    (1, 1)            (0.84, 0.94)                 
## 
##    0.06          3:47           39.8         1                   0.87                 0   
##                             (28.4, 48.4)   (1, 1)            (0.82, 0.93)                 
## 
##    0.07          7:93           36.4         1                   0.87                 0   
##                             (27.4, 44.2)   (1, 1)            (0.8, 0.93)                  
## 
##    0.08          2:23           34.2         1                   0.85                 0   
##                             (25.6, 40.8)   (1, 1)            (0.79, 0.92)                 
## 
##    0.09          9:91           32.6         1                   0.85                 0   
##                             (24.6, 39.2)   (1, 1)            (0.78, 0.92)                 
## 
##     0.1          1:9             31          1                   0.83                 0   
##                             (23.4, 36.8)   (1, 1)            (0.78, 0.92)                 
## 
##    0.11         11:89           29.8         1                   0.83                 0   
##                             (22.8, 34.4)   (1, 1)            (0.78, 0.92)                 
## 
##    0.12          3:22           28.4         1                   0.83                 0   
##                              (22.2, 33)    (1, 1)            (0.78, 0.92)                 
## 
##    0.13         13:87           27.2         1                   0.83                 0   
##                             (21.8, 31.2)   (1, 1)            (0.76, 0.92)                 
## 
##    0.14          7:43            26          1                   0.83                 0   
##                             (20.2, 30.8)   (1, 1)            (0.75, 0.9)                  
## 
##    0.15          3:17           25.6         1                   0.83                 0   
##                             (19.8, 29.2)   (1, 1)            (0.72, 0.9)                  
## 
##    0.16          4:21           24.4         1                   0.82                 0   
##                               (19, 29)     (1, 1)            (0.69, 0.89)                 
## 
##    0.17         17:83            23          1                   0.82                 0   
##                             (18.2, 28.6)   (1, 1)            (0.69, 0.88)                 
## 
##    0.18          9:41           22.8         1                   0.8                  0   
##                             (17.8, 26.4)   (1, 1)            (0.69, 0.88)                 
## 
##    0.19         19:81           22.4         1                   0.78                 0   
##                              (17, 25.2)    (1, 1)            (0.65, 0.87)                 
## 
##     0.2          1:4             21          1                   0.77                 0   
##                              (16.2, 25)    (1, 1)            (0.64, 0.87)                 
## 
##    0.21         21:79           20.4         1                   0.75                 0   
##                             (13.8, 23.8)   (1, 1)            (0.59, 0.87)                 
## 
##    0.22         11:39            19          1                   0.7                  0   
##                             (13.4, 23.2)   (1, 1)            (0.57, 0.86)                 
## 
##    0.23         23:77           17.8         1                   0.68                 0   
##                             (12.4, 22.6)   (1, 1)            (0.53, 0.84)                 
## 
##    0.24          6:19           17.2         1                   0.67                 0   
##                              (12, 22.2)    (1, 1)            (0.53, 0.83)                 
## 
##    0.25          1:3            16.4         1                   0.63                 0   
##                              (11.6, 21)    (1, 1)            (0.53, 0.82)                 
## 
##    0.26         13:37            16          1                   0.6                  0   
##                              (10.8, 21)    (1, 1)            (0.47, 0.82)                 
## 
##    0.27         27:73           15.6         1                   0.6                  0   
##                              (10.2, 21)    (1, 1)            (0.45, 0.8)                  
## 
##    0.28          7:18            15          1                   0.6                  0   
##                             (9.6, 19.8)    (1, 1)            (0.43, 0.75)                 
## 
##    0.29         29:71           14.6         1                   0.58                 0   
##                             (9.2, 19.2)    (1, 1)            (0.41, 0.75)                 
## 
##     0.3          3:7            14.2         1                   0.57                 0   
##                              (8.4, 19)     (1, 1)            (0.41, 0.75)                 
## 
##    0.31         31:69           13.8         1                   0.55                 0   
##                              (8.2, 19)     (1, 1)            (0.39, 0.75)                 
## 
##    0.32          8:17           13.2         1                   0.53                 0   
##                              (7.8, 17)     (1, 1)            (0.36, 0.73)                 
## 
##    0.33         33:67           12.8         1                   0.5                  0   
##                             (7.2, 16.6)    (1, 1)            (0.35, 0.7)                  
## 
##    0.34         17:33           12.4         1                   0.5                  0   
##                             (7.2, 15.8)    (1, 1)            (0.31, 0.7)                  
## 
##    0.35          7:13           11.4         1                   0.5                  0   
##                             (6.6, 14.8)    (1, 1)            (0.31, 0.7)                  
## 
##    0.36          9:16            11          1                   0.48                 0   
##                              (6.4, 14)     (1, 1)            (0.31, 0.7)                  
## 
##    0.37         37:63            10          1                   0.47                 0   
##                              (6.2, 14)     (1, 1)            (0.31, 0.7)                  
## 
##    0.38         19:31           9.6          1                   0.47                 0   
##                              (5.4, 14)     (1, 1)            (0.31, 0.66)                 
## 
##    0.39         39:61            9           1                   0.45                 0   
##                             (5.2, 13.4)    (1, 1)            (0.29, 0.66)                 
## 
##     0.4          2:3            8.2          1                   0.45                 0   
##                             (5.2, 12.8)    (1, 1)            (0.29, 0.62)                 
## 
##    0.41         41:59           7.6          1                   0.45                 0   
##                             (5.2, 12.4)    (1, 1)            (0.29, 0.62)                 
## 
##    0.42         21:29           7.4          1                   0.45                 0   
##                             (5.2, 12.4)    (1, 1)            (0.29, 0.61)                 
## 
##    0.43         43:57           7.2          1                   0.43                 0   
##                               (5, 12)      (1, 1)            (0.29, 0.6)                  
## 
##    0.44         11:14            7           1                   0.42                 0   
##                             (4.6, 11.8)    (1, 1)            (0.26, 0.58)                 
## 
##    0.45          9:11           6.4          1                   0.37                 0   
##                             (4.4, 11.6)    (1, 1)            (0.26, 0.57)                 
## 
##    0.46         23:27           6.2          1                   0.35                 0   
##                              (4, 10.8)     (1, 1)            (0.25, 0.57)                 
## 
##    0.47         47:53            6           1                   0.33                 0   
##                             (3.6, 10.4)    (1, 1)            (0.21, 0.57)                 
## 
##    0.48         12:13           5.8          1                   0.33                 0   
##                              (3, 10.2)     (1, 1)            (0.21, 0.57)                 
## 
##    0.49         49:51           5.6          1                   0.32                 0   
##                               (3, 10)      (1, 1)            (0.21, 0.55)                 
## 
##     0.5          1:1            5.2          1                   0.3                  0   
##                              (2.8, 9.8)    (1, 1)            (0.2, 0.55)                  
## 
##    0.51         51:49            5           1                   0.3                  0   
##                              (2.8, 9.8)    (1, 1)            (0.17, 0.52)                 
## 
##    0.52         13:12           4.8          1                   0.28                 0   
##                              (2.8, 9.8)    (1, 1)            (0.16, 0.48)                 
## 
##    0.53         53:47           4.2          1                   0.25                 0   
##                              (2.4, 9.4)    (1, 1)            (0.16, 0.43)                 
## 
##    0.54         27:23           4.2          1                   0.25                 0   
##                              (1.6, 8.2)    (1, 1)            (0.11, 0.43)                 
## 
##    0.55          11:9           3.8          1                   0.23                 0   
##                              (1.6, 8.2)    (1, 1)            (0.11, 0.42)                 
## 
##    0.56         14:11           3.8          1                   0.23                 0   
##                               (1.6, 8)     (1, 1)            (0.11, 0.42)                 
## 
##    0.57         57:43           3.8          1                   0.23                 0   
##                               (1.6, 8)     (1, 1)            (0.11, 0.41)                 
## 
##    0.58         29:21           3.6          1                   0.23                 0   
##                              (1.6, 7.4)    (1, 1)            (0.1, 0.41)                  
## 
##    0.59         59:41           3.4          1                   0.23                 0   
##                              (1.2, 7.2)    (1, 1)             (0.1, 0.4)                  
## 
##     0.6          3:2            3.4          1                   0.23                 0   
##                              (1.2, 6.8)    (1, 1)             (0.1, 0.4)                  
## 
##    0.61         61:39           3.2          1                   0.23                 0   
##                              (1.2, 6.6)    (1, 1)             (0.1, 0.4)                  
## 
##    0.62         31:19            3           1                   0.22                 0   
##                              (1.2, 6.4)    (1, 1)             (0.1, 0.4)                  
## 
##    0.63         63:37            3           1                   0.22                 0   
##                              (1.2, 5.8)    (1, 1)            (0.1, 0.38)                  
## 
##    0.64          16:9           2.8          1                   0.2                  0   
##                              (1.2, 5.6)    (1, 1)            (0.1, 0.37)                  
## 
##    0.65          13:7           2.8          1                   0.2                  0   
##                               (1.2, 5)     (1, 1)            (0.1, 0.34)                  
## 
##    0.66         33:17           2.8          1                   0.2                  0   
##                                (1, 5)      (1, 1)            (0.1, 0.34)                  
## 
##    0.67         67:33           2.8          1                   0.2                  0   
##                               (1, 4.8)     (1, 1)            (0.08, 0.32)                 
## 
##    0.68          17:8           2.8          1                   0.2                  0   
##                               (1, 4.8)     (1, 1)            (0.08, 0.32)                 
## 
##    0.69         69:31           2.8          1                   0.2                  0   
##                              (0.8, 4.6)    (1, 1)            (0.08, 0.32)                 
## 
##     0.7          7:3            2.6          1                   0.2                  0   
##                              (0.8, 4.6)    (1, 1)            (0.06, 0.31)                 
## 
##    0.71         71:29           2.2          1                   0.17                 0   
##                              (0.8, 4.4)    (1, 1)            (0.06, 0.31)                 
## 
##    0.72          18:7           1.8          1                   0.13                 0   
##                              (0.8, 4.2)    (1, 1)            (0.06, 0.31)                 
## 
##    0.73         73:27           1.6          1                   0.12                 0   
##                               (0.8, 4)     (1, 1)            (0.06, 0.29)                 
## 
##    0.74         37:13           1.6          1                   0.12                 0   
##                               (0.6, 4)     (1, 1)            (0.05, 0.29)                 
## 
##    0.75          3:1            1.6          1                   0.12                 0   
##                               (0.4, 4)     (1, 1)            (0.03, 0.29)                 
## 
##    0.76          19:6           1.6          1                   0.12                 0   
##                              (0.4, 3.6)    (1, 1)            (0.03, 0.28)                 
## 
##    0.77         77:23           1.6          1                   0.12                 0   
##                              (0.2, 3.6)    (1, 1)            (0.01, 0.27)                 
## 
##    0.78         39:11           1.2          1                   0.1                  0   
##                              (0.2, 3.6)    (1, 1)            (0.01, 0.26)                 
## 
##    0.79         79:21           1.2          1                   0.1                  0   
##                              (0.2, 3.6)    (1, 1)            (0.01, 0.25)                 
## 
##     0.8          4:1            1.2          1                   0.1                  0   
##                               (0.2, 3)     (1, 1)            (0.01, 0.23)                 
## 
##    0.81         81:19           1.2          1                   0.1                  0   
##                              (0.2, 2.8)    (1, 1)            (0.01, 0.21)                 
## 
##    0.82          41:9           1.2          1                   0.1                  0   
##                               (0, 2.6)     (1, 1)              (0, 0.2)                   
## 
##    0.83         83:17           1.2          1                   0.1                  0   
##                               (0, 2.6)     (1, 1)              (0, 0.2)                   
## 
##    0.84          21:4            1           1                   0.08                 0   
##                               (0, 2.6)     (1, 1)              (0, 0.2)                   
## 
##    0.85          17:3           0.8          1                   0.07                 0   
##                               (0, 2.6)     (1, 1)              (0, 0.2)                   
## 
##    0.86          43:7           0.8          1                   0.07                 0   
##                               (0, 2.2)     (1, 1)             (0, 0.18)                   
## 
##    0.87         87:13           0.4          1                   0.03                 0   
##                               (0, 2.2)     (1, 1)             (0, 0.18)                   
## 
##    0.88          22:3           0.2          1                   0.02                 0   
##                               (0, 2.2)     (1, 1)             (0, 0.18)                   
## 
##    0.89         89:11            0           1                    0                   0   
##                               (0, 1.8)     (1, 1)             (0, 0.15)                   
## 
##     0.9          9:1             0           1                    0                   0   
##                               (0, 1.6)     (1, 1)             (0, 0.12)                   
## 
##    0.91          91:9            0           1                    0                   0   
##                               (0, 1.4)     (1, 1)             (0, 0.12)                   
## 
##    0.92          23:2            0           1                    0                   0   
##                               (0, 1.2)     (1, 1)              (0, 0.1)                   
## 
##    0.93          93:7            0           1                    0                   0   
##                                (0, 1)      (1, 1)             (0, 0.08)                   
## 
##    0.94          47:3            0           1                    0                   0   
##                                (0, 1)      (1, 1)             (0, 0.08)                   
## 
##    0.95          19:1            0           1                    0                   0   
##                               (0, 0.8)     (1, 1)             (0, 0.06)                   
## 
##    0.96          24:1            0           1                    0                   0   
##                               (0, 0.4)     (1, 1)             (0, 0.04)                   
## 
##    0.97          97:3            0           1                    0                   0   
##                               (0, 0.4)     (1, 1)             (0, 0.03)                   
## 
##    0.98          49:1            0           1                    0                   0   
##                                (0, 0)      (1, 1)               (0, 0)                    
## 
##    0.99          99:1            0           1                    0                   0   
##                                (0, 0)      (1, 1)               (0, 0)                    
## 
##      1          Inf:1            0           0                    0                   0   
##                                (0, 0)      (0, 0)               (0, 0)                    
## ------------------------------------------------------------------------------------------

Cross-validation

To obtain bias corrected decision curves, one can perform k-fold cross-validation with a cv_decision_curve function.

full.model_cv <- cv_decision_curve(Cancer~Age + Female + Smokes + Marker1 + Marker2,
                                    data = dcaData)
## Note:  The data provided is used to both fit a prediction model and to estimate the respective decision curve. This may cause bias in decision curve estimates leading to over-confidence in model performance.
plot_decision_curve(list(full.model, full.model_cv), 
                   curve.names = c("Initial curve", "Cross-validated curve"),
                   col = c("red", "blue"), 
                   lty = c(2,1), 
                   lwd = c(3,2, 2, 1), 
                   legend.position = "bottomright") 
## Note: When multiple decision curves are plotted, decision curves for 'All' are calculated using the prevalence from the first DecisionCurve object in the list provided.

#plotting cross-validated clinical impact curve
plot_clinical_impact(full.model_cv, legend.position = "bottomright")

In this document, the key concepts of the rmda R package for risk models were generally explained, and some analyses replicated from https://mdbrown.github.io/rmda/.