Validation of ICD-9-CM/ICD-10-CM Codes for Automated Electronic Scoring of APACHE II, APACHE III, and SAPS II

Supplemental Material

Eric Kao, Brian Gulbis

November 30, 2016

Patient Selection

There were 16,242 patient encounters identified which met inclusion criteria. Patients were then excluded for the following reasons:

This left a total of 3,390 eligible patients remaining. From these, a random sample of 200 patients were selected for inclusion in the study.

Comorbidity Definitions

The following tables contain the list of ICD-9-CM and ICD-10-CM codes which were included in the author-defined comorbidity set.1 Any child-codes for the codes listed were also included (i.e., 196 includes 196.0, 196.1, etc.)

ICD-9-CM Codes Used in Author-Defined Comorbidity Set for APACHE II

Group Comorbidity Criteria ICD-9-CM Code
Liver
Biopsy proven cirrhosis and documented portal hypertension 456, 567.23, 571.2, 571.5, 572.2, 572.4, 789.59
Plus 572.3
Past upper GI bleed attributed to portal hypertension 530.82, 578.9
Plus 572.3
Prior hepatic failure, encephalopathy, or coma 070.2, 070.3, 070.41, 070.44, 070.51, 070.54, 070.7, 155, 275, 571, 573.3
Or 572.2
Cardiovascular
Heart failure 428
Respiratory
Chronic restrictive, obstructive, or vascular disease 490, 491, 492.0, 492.1, 492.2, 492.3, 492.4, 492.5, 492.6, 492.7, 492.8, 493.0, 493.1, 493.2, 493.3, 493.4, 493.5, 493.6, 493.7, 493.8, 493.90, 493.91, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506.4
Or 440, 441.2, 441.4, 441.7, 441.9, 443.1, 443.2, 443.3, 443.4, 443.5, 443.6, 443.7, 443.8, 443.9, 447.1, 557.1, 557.9, V43.4
Chronic hypoxia 518.83, 799.02
Chronic hypercapnia 786.09
Secondary polycythemia 289.0
Severe pulmonary hypertension 416
Respirator dependence V46.11
Renal
Receiving chronic dialysis 585.6
Immuno-compromised
Immuno-suppression therapy V87.46
Chemotherapy V58.1, V87.41, E933.1
Radiation E926
Long-term or high-dose steroids V58.65
Leukemia 204, 205, 206, 207, 208
Lymphoma 200, 201, 202.0, 202.1, 202.2, 202.30, 202.31, 202.32, 202.33, 202.34, 202.35, 202.36, 202.37, 202.38, 202.5, 202.6, 202.7, 202.8, 202.9, 203.00, 203.01, 203.80, 203.81, 238.6, 273.3, V10.71, V10.72, V10.79
AIDS 042
Plus 176
Or 136.3
Or 130
Or 010, 011, 012, 013, 014, 015, 016, 017, 018

ICD-10-CM Codes Used in Author-Defined Comorbidity Set for APACHE II

Group Comorbidity Criteria ICD-10-CM Code
Liver
Biopsy proven cirrhosis and documented portal hypertension I85.00, I85.01, I85.10, I85.11, K65.2, K70.30, K72.90, K72.91, K74.0, K74.60, K74.69, K76.7, R18.8
Plus K76.6
Past upper GI bleed attributed to portal hypertension K22.8, K92.2
Plus K76.6
Prior hepatic failure, encephalopathy, or coma B16.0, B16.1, B16.2, B16.9, B17.10, B17.11, B18.0, B18.0, B18.1, B18.1, B18.2, B18.2, B19.10, B19.11, B19.20, B19.21, C22.0, C22.1, C22.2, C22.7, C22.8, K70.0, K70.10, K70.30, K70.9, K71.6, K73.0, K73.2, K73.8, K73.9, K74.0, K74.1, K74.3, K74.4, K74.5, K74.60, K74.69, K75.4, K75.9, K76.0, K76.89, K76.9
Or K72.90, K72.91
Cardiovascular
Heart failure I50.1, I50.20, I50.21, I50.22, I50.23, I50.30, I50.31, I50.32, I50.33, I50.40, I50.41, I50.42, I50.43, I50.9, I50.9
Respiratory
Chronic restrictive, obstructive, or vascular disease I27.8, I27.9, J40, J41, J42, J43, J44, J45, J46, J47, J60, J61, J62, J63, J64, J65, J66, J67, J68.4, J70.1, J70.3
Or I70, I71, I73.1, I73.8, I73.9, I77.1, I79.0, I79.2, K55.1, K55.8, K55.9, Z95.8, Z95.9
Chronic hypoxia J96.10, R09.02
Chronic hypercapnia R06.00, R06.09, R06.3, R06.83, R06.89
Secondary polycythemia D75.1
Severe pulmonary hypertension I27.0
Respirator dependence Z99.11
Renal
Receiving chronic dialysis N18.6
Immuno-compromised
Immuno-suppression therapy Z92.25
Chemotherapy Z51.11, Z51.12, Z92.21
Radiation W90.0X
Long-term or high-dose steroids Z79.51, Z79.52
Leukemia C95.00, C95.01, C95.02, C95.10, C95.11, C95.12, C95.90, C95.90, C95.90, C95.91, C95.91, C95.91, C95.92, C95.92, C95.92
Lymphoma C81, C82, C83, C84, C85, C88, C90.0, C90.2, C96
AIDS B20
Plus C46.0
Or B59
Or B58.01, B58.09, B58.1, B58.2, B58.3, B58.81, B58.89, B58.89, B58.9
Or A15.0, A15.4, A15.5, A15.6, A15.7, A15.8, A17.0, A17.1, A17.81, A17.82, A17.89, A17.9, A18.01, A18.02, A18.03, A18.10, A18.11, A18.12, A18.13, A18.14, A18.15, A18.16, A18.17, A18.18, A18.2, A18.31, A18.32, A18.39, A18.4, A18.50, A18.51, A18.52, A18.53, A18.54, A18.59, A18.6, A18.7, A18.81, A18.84, A18.85, A18.89, A19.2, A19.8, A19.9

ICD-9-CM Codes Used in Author-Defined Comorbidity Set for APACHE III

Comorbidity Criteria ICD-9-CM Code
AIDS 042
Plus 176
Or 136.3
Or 130
Or 010, 011, 012, 013, 014, 015, 016, 017, 018
Hepatic failure 070.2, 070.3, 070.41, 070.44, 070.51, 070.54, 070.7, 155, 275, 571, 573.3
Lymphoma 200, 201, 202.0, 202.1, 202.2, 202.30, 202.31, 202.32, 202.33, 202.34, 202.35, 202.36, 202.37, 202.38, 202.5, 202.6, 202.7, 202.8, 202.9, 203.00, 203.01, 203.80, 203.81, 238.6, 273.3, V10.71, V10.72, V10.79
Metastatic cancer 196, 197, 198, 199.0, 199.1
Leukemia or multiple myeloma 204, 205, 206, 207, 208
Or 203
Immuno-suppression therapy V87.46
Cirrhosis 456, 567.23, 571.2, 571.5, 572.2, 572.4, 789.59

ICD-10-CM Codes Used in Author-Defined Comorbidity Set for APACHE III

Comorbidity Criteria ICD-10-CM Code
AIDS B20
Plus C46.0
Or B59
Or B58.01, B58.09, B58.1, B58.2, B58.3, B58.81, B58.89, B58.89, B58.9
Or A15.0, A15.4, A15.5, A15.6, A15.7, A15.8, A17.0, A17.1, A17.81, A17.82, A17.89, A17.9, A18.01, A18.02, A18.03, A18.10, A18.11, A18.12, A18.13, A18.14, A18.15, A18.16, A18.17, A18.18, A18.2, A18.31, A18.32, A18.39, A18.4, A18.50, A18.51, A18.52, A18.53, A18.54, A18.59, A18.6, A18.7, A18.81, A18.84, A18.85, A18.89, A19.2, A19.8, A19.9
Hepatic failure B16.0, B16.1, B16.2, B16.9, B17.10, B17.11, B18.0, B18.0, B18.1, B18.1, B18.2, B18.2, B19.10, B19.11, B19.20, B19.21, C22.0, C22.1, C22.2, C22.7, C22.8, K70.0, K70.10, K70.30, K70.9, K71.6, K73.0, K73.2, K73.8, K73.9, K74.0, K74.1, K74.3, K74.4, K74.5, K74.60, K74.69, K75.4, K75.9, K76.0, K76.89, K76.9
Lymphoma C81, C82, C83, C84, C85, C88, C90.0, C90.2, C96
Metastatic cancer C77, C78, C79, C80
Leukemia or multiple myeloma C95.00, C95.01, C95.02, C95.10, C95.11, C95.12, C95.90, C95.90, C95.90, C95.91, C95.91, C95.91, C95.92, C95.92, C95.92
Or C90.00, C90.00, C90.01, C90.02
Immuno-suppression therapy Z92.25
Cirrhosis I85.00, I85.01, I85.10, I85.11, K65.2, K70.30, K72.90, K72.91, K74.0, K74.60, K74.69, K76.7, R18.8

ICD-9-CM Codes Used in Author-Defined Comorbidity Set for SAPS II

Comorbidity Criteria ICD-9-CM Code
AIDS 042
Plus 176
Or 136.3
Or 130
Or 010, 011, 012, 013, 014, 015, 016, 017, 018
Metastatic cancer 196, 197, 198, 199.0, 199.1
Hematologic malignancy 204, 205, 206, 207, 208
Or 200, 201, 202.0, 202.1, 202.2, 202.30, 202.31, 202.32, 202.33, 202.34, 202.35, 202.36, 202.37, 202.38, 202.5, 202.6, 202.7, 202.8, 202.9, 203.00, 203.01, 203.80, 203.81, 238.6, 273.3, V10.71, V10.72, V10.79
Or 203

ICD-10-CM Codes Used in Author-Defined Comorbidity Set for SAPS II

Comorbidity Criteria ICD-10-CM Code
AIDS B20
Plus C46.0
Or B59
Or B58.01, B58.09, B58.1, B58.2, B58.3, B58.81, B58.89, B58.89, B58.9
Or A15.0, A15.4, A15.5, A15.6, A15.7, A15.8, A17.0, A17.1, A17.81, A17.82, A17.89, A17.9, A18.01, A18.02, A18.03, A18.10, A18.11, A18.12, A18.13, A18.14, A18.15, A18.16, A18.17, A18.18, A18.2, A18.31, A18.32, A18.39, A18.4, A18.50, A18.51, A18.52, A18.53, A18.54, A18.59, A18.6, A18.7, A18.81, A18.84, A18.85, A18.89, A19.2, A19.8, A19.9
Metastatic cancer C77, C78, C79, C80
Hematologic malignancy C95.00, C95.01, C95.02, C95.10, C95.11, C95.12, C95.90, C95.90, C95.90, C95.91, C95.91, C95.91, C95.92, C95.92, C95.92
Or C81, C82, C83, C84, C85, C88, C90.0, C90.2, C96
Or C90.00, C90.00, C90.01, C90.02

Statistical Methods

Data processing, risk score calculations, and statistical analysis were performed using R version 3.3.2 (2016-10-31) (R Core Team (2016R Core Team. 2016. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.)). The icd package (Wasey (2016Wasey, Jack O. 2016. Icd: Tools for Working with Icd-9 and Icd-10 Codes, and Finding Comorbidities. https://CRAN.R-project.org/package=icd.)) was used to cross-reference patient diagnosis codes with comorbidity set diagnosis codes.

Continuous data were analyzed using a paired t-test and Wilcoxon signed rank test. The p-value was then adjusted using the Benjamini and Hochberg method to account for performing multiple comparisons.

A univarite linear regression model was fit using the risk score from each comorbidity set to predict the risk score for the manual review set.

Supplemental Results

The following tables contain the mean and median risk score calculated using each of the eight comorbidity sets, as well as the manual review set. A comparison of the means and medians was performed.2 The adjusted p-value and 95% confidence intervals (CI) are reported for the t-test and Wilcoxon rank sum (denoted by the *)

Primary Endpoint

APACHE II Score

Tool Mean SD P-value 95% CI Median IQR P-value* 95% CI*
AHRQ 20.20 6.858 0.131 0.014 - 0.646 20 15 - 24 0.168 0 - 2
AHRQ+DRG 19.79 6.832 0.619 -0.397 - 0.237 19 15 - 24 0.518 -1.5 - 0
Authors 20.05 6.910 0.425 -0.138 - 0.498 20 15 - 24 0.504 0 - 2
Authors+DRG 19.65 6.867 0.341 -0.535 - 0.095 19 15 - 24 0.271 -1.5 - 0
Elixhauser 20.18 6.927 0.131 0.001 - 0.619 20 15 - 24 0.168 0 - 2
Elixhauser+DRG 19.77 6.900 0.619 -0.409 - 0.209 19 15 - 24 0.518 -1.5 - 0
Manual 19.87 7.021 NA NA 19 15 - 24.25 NA NA
Quan 20.18 6.927 0.131 0.001 - 0.619 20 15 - 24 0.168 0 - 2
Quan+DRG 19.78 6.892 0.619 -0.399 - 0.219 19 15 - 24 0.518 -1.5 - 0

APACHE III Score

Tool Mean SD P-value 95% CI Median IQR P-value* 95% CI*
AHRQ 85.715 29.300 0.007 0.302 - 1.518 82.0 65.8 - 103.2 0.012 3 - 13
AHRQ+DRG 85.555 29.259 0.012 0.169 - 1.331 82.0 65 - 103.2 0.018 1.5 - 11.5
Authors 85.715 29.300 0.007 0.302 - 1.518 82.0 65.8 - 103.2 0.012 3 - 13
Authors+DRG 85.555 29.259 0.012 0.169 - 1.331 82.0 65 - 103.2 0.018 1.5 - 11.5
Elixhauser 85.715 29.300 0.007 0.302 - 1.518 82.0 65.8 - 103.2 0.012 3 - 13
Elixhauser+DRG 85.555 29.259 0.012 0.169 - 1.331 82.0 65 - 103.2 0.018 1.5 - 11.5
Manual 84.805 28.998 NA NA 81.5 63.8 - 103 NA NA
Quan 85.715 29.300 0.007 0.302 - 1.518 82.0 65.8 - 103.2 0.012 3 - 13
Quan+DRG 85.555 29.259 0.012 0.169 - 1.331 82.0 65 - 103.2 0.018 1.5 - 11.5

SAPS II Score

Tool Mean SD P-value 95% CI Median IQR P-value* 95% CI*
AHRQ 37.12 14.564 0.182 -0.085 - 0.445 36.0 26 - 47 0.518 -0.5 - 9.5
AHRQ+DRG 37.12 14.564 0.182 -0.085 - 0.445 36.0 26 - 47 0.518 -0.5 - 9.5
Authors 37.12 14.564 0.182 -0.085 - 0.445 36.0 26 - 47 0.518 -0.5 - 9.5
Authors+DRG 37.12 14.564 0.182 -0.085 - 0.445 36.0 26 - 47 0.518 -0.5 - 9.5
Elixhauser 37.12 14.564 0.182 -0.085 - 0.445 36.0 26 - 47 0.518 -0.5 - 9.5
Elixhauser+DRG 37.12 14.564 0.182 -0.085 - 0.445 36.0 26 - 47 0.518 -0.5 - 9.5
Manual 36.94 14.449 NA NA 34.5 26 - 46.2 NA NA
Quan 37.12 14.564 0.182 -0.085 - 0.445 36.0 26 - 47 0.518 -0.5 - 9.5
Quan+DRG 37.12 14.564 0.182 -0.085 - 0.445 36.0 26 - 47 0.518 -0.5 - 9.5

Secondary Endpoints

Predicted Mortality using SAPS II

Tool Mean SD P-value 95% CI Median IQR P-value* 95% CI*
AHRQ 0.251 0.230 0.069 0 - 0.008 0.181 0.1 - 0.4 0.08 -0.009 - 0.186
AHRQ+DRG 0.251 0.230 0.069 0 - 0.008 0.181 0.1 - 0.4 0.08 -0.009 - 0.186
Authors 0.251 0.230 0.069 0 - 0.008 0.181 0.1 - 0.4 0.08 -0.009 - 0.186
Authors+DRG 0.251 0.230 0.069 0 - 0.008 0.181 0.1 - 0.4 0.08 -0.009 - 0.186
Elixhauser 0.251 0.230 0.069 0 - 0.008 0.181 0.1 - 0.4 0.08 -0.009 - 0.186
Elixhauser+DRG 0.251 0.230 0.069 0 - 0.008 0.181 0.1 - 0.4 0.08 -0.009 - 0.186
Manual 0.248 0.228 NA NA 0.160 0.1 - 0.4 NA NA
Quan 0.251 0.230 0.069 0 - 0.008 0.181 0.1 - 0.4 0.08 -0.009 - 0.186
Quan+DRG 0.251 0.230 0.069 0 - 0.008 0.181 0.1 - 0.4 0.08 -0.009 - 0.186

Number of Comorbidities per Patient using APACHE II

Tool Mean SD P-value 95% CI Median IQR P-value* 95% CI*
AHRQ 0.760 0.731 0.051 0.037 - 0.273 1 0 - 1 0.052 0 - 1
AHRQ+DRG 0.595 0.673 0.862 -0.124 - 0.104 0 0 - 1 0.912 0 - 0
Authors 0.690 0.719 0.300 -0.033 - 0.203 1 0 - 1 0.255 0 - 0.5
Authors+DRG 0.530 0.641 0.300 -0.187 - 0.037 0 0 - 1 0.329 -0.5 - 0
Elixhauser 0.745 0.737 0.051 0.023 - 0.257 1 0 - 1 0.052 0 - 0.5
Elixhauser+DRG 0.575 0.661 0.792 -0.141 - 0.081 0 0 - 1 0.784 0 - 0
Manual 0.605 0.722 NA NA 0 0 - 1 NA NA
Quan 0.750 0.742 0.051 0.028 - 0.262 1 0 - 1 0.052 0 - 0.5
Quan+DRG 0.585 0.667 0.828 -0.132 - 0.092 0 0 - 1 0.815 0 - 0

The following table contains the number of patients with different comorbidities identified by the comorbidity sets when compared with manual review. For APACHE II, this was counted as the number of patients where a comorbidity was identified by ICD or by manual, but not both. For APACHE III and SAPS II, this was counted as the number of patients where the primary comorbidity identified by ICD coding varied from the primary comorbidity identified by manual review.

Proportion of Patients with Differing Comorbidities

Tool APACHE II, n (%) APACHE III, n (%) SAPS II, n (%)
AHRQ 72 (36) 32 (16) 8 (4)
AHRQ+DRG 68 (34) 31 (15.5) 8 (4)
Authors 71 (35.5) 32 (16) 8 (4)
Authors+DRG 68 (34) 31 (15.5) 8 (4)
Elixhauser 70 (35) 32 (16) 8 (4)
Elixhauser+DRG 66 (33) 31 (15.5) 8 (4)
Quan 70 (35) 32 (16) 8 (4)
Quan+DRG 67 (33.5) 31 (15.5) 8 (4)

Linear Regression Models

Linear Regression Models for APACHE II

Tool r.squared adj.r.squared sigma statistic p.value df logLik AIC BIC deviance df.residual
AHRQ 0.897 0.896 2.262 1718.366 0 2 -446.066 898.132 908.026 1013.432 198
AHRQ+DRG 0.896 0.896 2.269 1706.410 0 2 -446.692 899.383 909.278 1019.794 198
Authors 0.896 0.896 2.269 1706.705 0 2 -446.676 899.352 909.247 1019.636 198
Authors+DRG 0.897 0.897 2.257 1726.889 0 2 -445.622 897.244 907.139 1008.945 198
Elixhauser 0.902 0.901 2.203 1822.143 0 2 -440.792 887.584 897.479 961.371 198
Elixhauser+DRG 0.902 0.901 2.208 1814.341 0 2 -441.179 888.358 898.253 965.098 198
Quan 0.902 0.901 2.203 1822.143 0 2 -440.792 887.584 897.479 961.371 198
Quan+DRG 0.901 0.901 2.213 1804.009 0 2 -441.694 889.387 899.282 970.079 198

Fitted Values vs. Residuals for APACHE II Linear Regression Models

Fitted Values vs. Residuals for APACHE II Linear Regression Models

Linear Regression Models for APACHE III

Tool r.squared adj.r.squared sigma statistic p.value df logLik AIC BIC deviance df.residual
AHRQ 0.978 0.978 4.328 8735.812 0 2 -575.802 1157.604 1167.499 3708.786 198
AHRQ+DRG 0.980 0.980 4.140 9564.716 0 2 -566.929 1139.858 1149.753 3393.891 198
Authors 0.978 0.978 4.328 8735.812 0 2 -575.802 1157.604 1167.499 3708.786 198
Authors+DRG 0.980 0.980 4.140 9564.716 0 2 -566.929 1139.858 1149.753 3393.891 198
Elixhauser 0.978 0.978 4.328 8735.812 0 2 -575.802 1157.604 1167.499 3708.786 198
Elixhauser+DRG 0.980 0.980 4.140 9564.716 0 2 -566.929 1139.858 1149.753 3393.891 198
Quan 0.978 0.978 4.328 8735.812 0 2 -575.802 1157.604 1167.499 3708.786 198
Quan+DRG 0.980 0.980 4.140 9564.716 0 2 -566.929 1139.858 1149.753 3393.891 198

Fitted Values vs. Residuals for APACHE III Linear Regression Models

Fitted Values vs. Residuals for APACHE III Linear Regression Models

Linear Regression Models for SAPS II

Tool r.squared adj.r.squared sigma statistic p.value df logLik AIC BIC deviance df.residual
AHRQ 0.983 0.983 1.889 11449.91 0 2 -409.948 825.897 835.792 706.218 198
AHRQ+DRG 0.983 0.983 1.889 11449.91 0 2 -409.948 825.897 835.792 706.218 198
Authors 0.983 0.983 1.889 11449.91 0 2 -409.948 825.897 835.792 706.218 198
Authors+DRG 0.983 0.983 1.889 11449.91 0 2 -409.948 825.897 835.792 706.218 198
Elixhauser 0.983 0.983 1.889 11449.91 0 2 -409.948 825.897 835.792 706.218 198
Elixhauser+DRG 0.983 0.983 1.889 11449.91 0 2 -409.948 825.897 835.792 706.218 198
Quan 0.983 0.983 1.889 11449.91 0 2 -409.948 825.897 835.792 706.218 198
Quan+DRG 0.983 0.983 1.889 11449.91 0 2 -409.948 825.897 835.792 706.218 198

Fitted Values vs. Residuals for SAPS II Linear Regression Models

Fitted Values vs. Residuals for SAPS II Linear Regression Models