1 Setup

1.1 Load packages

1.2 Import data

1.3 Clean and recode

## Total encounters: 47497
## Deterioration events: 1324
## Event rate: 2.79 %
## Analytic n: 47497
## Training set: n = 33247 | events = 921
## Test set:     n = 14250 | events = 403
## # A tibble: 2 × 2
##   separation   Encounters
##   <fct>             <int>
## 1 Admitted          14447
## 2 Not admitted      33050
## [1] 47497

2 Descriptive Statistics

2.1 Table 1

Baseline characteristics stratified by clinical deterioration

Variable

Overall
n = 474971

No deterioration
n = 461731

Deterioration
n = 13241

p-value2

Age group

<0.001

16-19

2,809 (5.9%)

2,785 (6.0%)

24 (1.8%)

20-39

18,079 (38.1%)

17,933 (38.8%)

146 (11.0%)

40-59

13,175 (27.7%)

12,933 (28.0%)

242 (18.3%)

60-79

9,640 (20.3%)

9,127 (19.8%)

513 (38.7%)

80+

3,794 (8.0%)

3,395 (7.4%)

399 (30.1%)

Sex

0.006

Male

22,883 (48.2%)

22,195 (48.1%)

688 (52.0%)

Female

24,611 (51.8%)

23,975 (51.9%)

636 (48.0%)

Not recorded

3

3

0

Mode of arrival

<0.001

Ambulance

9,720 (20.5%)

9,021 (19.5%)

699 (52.8%)

Public transport

220 (0.5%)

219 (0.5%)

1 (0.1%)

Private vehicle

36,410 (76.7%)

35,830 (77.6%)

580 (43.8%)

Helicopter

9 (0.0%)

9 (0.0%)

0 (0.0%)

Air ambulance

2 (0.0%)

2 (0.0%)

0 (0.0%)

Internal ambulance

154 (0.3%)

134 (0.3%)

20 (1.5%)

Police

486 (1.0%)

479 (1.0%)

7 (0.5%)

Walked in

250 (0.5%)

248 (0.5%)

2 (0.2%)

Retrieval team

3 (0.0%)

3 (0.0%)

0 (0.0%)

Internal bed

243 (0.5%)

228 (0.5%)

15 (1.1%)

Triage category (ATS)

<0.001

3 - Urgent

27,318 (57.5%)

26,242 (56.8%)

1,076 (81.3%)

4 - Semi-Urgent

16,490 (34.7%)

16,259 (35.2%)

231 (17.4%)

5 - Non-Urgent

3,689 (7.8%)

3,672 (8.0%)

17 (1.3%)

Time of arrival

<0.001

0800-1759

28,281 (59.5%)

27,426 (59.4%)

855 (64.6%)

1800-2259

10,507 (22.1%)

10,253 (22.2%)

254 (19.2%)

2300-0759

8,709 (18.3%)

8,494 (18.4%)

215 (16.2%)

Prior hospital admission (30 days)

2,608 (5.5%)

2,449 (5.3%)

159 (12.0%)

<0.001

Presenting problem category

<0.001

Abdominal/GI

9,323 (19.6%)

9,015 (19.5%)

308 (23.3%)

Cardiovascular

1,110 (2.3%)

1,080 (2.3%)

30 (2.3%)

General symptoms

5,331 (11.2%)

5,013 (10.9%)

318 (24.0%)

Febrile illness

1,239 (2.6%)

1,173 (2.5%)

66 (5.0%)

Injury

6,359 (13.4%)

6,297 (13.6%)

62 (4.7%)

Respiratory

2,112 (4.4%)

1,954 (4.2%)

158 (11.9%)

Musculoskeletal

3,981 (8.4%)

3,898 (8.4%)

83 (6.3%)

Neurological

3,227 (6.8%)

3,148 (6.8%)

79 (6.0%)

Mental health

1,940 (4.1%)

1,909 (4.1%)

31 (2.3%)

Toxicological

253 (0.5%)

250 (0.5%)

3 (0.2%)

ENT/eye/head & neck

4,700 (9.9%)

4,657 (10.1%)

43 (3.2%)

Administrative

1,568 (3.3%)

1,525 (3.3%)

43 (3.2%)

Genitourinary

1,977 (4.2%)

1,940 (4.2%)

37 (2.8%)

Endocrine

110 (0.2%)

101 (0.2%)

9 (0.7%)

Obstetrics/Gynaecology

1,734 (3.7%)

1,723 (3.7%)

11 (0.8%)

Skin/allergy

2,521 (5.3%)

2,479 (5.4%)

42 (3.2%)

Other medical

12 (0.0%)

11 (0.0%)

1 (0.1%)

START score

11.3 (6.7)

11.1 (6.6)

18.3 (5.4)

<0.001

1n (%); Mean (SD)

2Pearson's Chi-squared test; Welch Two Sample t-test

2.2 Outcome component frequencies

Composite outcome component frequencies
Outcome n Rate (%)
RRC activation 1056 2.22
Code Blue 385 0.81
ICU admission 115 0.24
In-hospital death 217 0.46
Composite (any) 1324 2.79

3 Model A: START Score Performance

3.1 ROC curve and AUC

## START score AUC: 0.7997
## 95% CI (DeLong): 0.7885 - 0.811

3.2 Training-derived START thresholds and held-out validation

3.2.1 Threshold derivation in the training cohort

3.2.2 Fixed-threshold validation in the test cohort

3.2.3 Training-set threshold performance

Derivation of integer START thresholds in the training cohort (n = 33,247; 921 deterioration events).

Criterion

Rule

Training sensitivity (%)

Training specificity (%)

Youden

START >= 16

71.1

74.6

Sensitivity >=85%

START >= 13

85.1

56.2

Sensitivity >=90%

START >= 11

91.4

45.6

3.2.4 Held-out test-set threshold performance

Held-out validation of training-derived START thresholds (n = 14,250; 403 deterioration events).

Training
criterion

Fixed
rule

Se (%)
95% CI

Sp (%)
95% CI

PPV (%)
95% CI

NPV (%)
95% CI

Flagged
n (%)

Missed
n / events (%)

Youden

START >= 16

69.5
(64.7-73.9)

75.1
(74.4-75.8)

7.5
(6.7-8.4)

98.8
(98.6-99.0)

3,726
(26.1%)

123 / 403
(30.5%)

Sensitivity >=85%

START >= 13

85.4
(81.5-88.7)

57.0
(56.1-57.8)

5.5
(4.9-6.0)

99.3
(99.0-99.4)

6,302
(44.2%)

59 / 403
(14.6%)

Sensitivity >=90%

START >= 11

92.1
(89.0-94.5)

46.7
(45.8-47.5)

4.8
(4.3-5.3)

99.5
(99.3-99.7)

7,757
(54.4%)

32 / 403
(7.9%)

All thresholds were selected using training data only and applied unchanged to the test cohort.

Se: sensitivity; Sp: specificity; PPV/NPV: positive/negative predictive value. CI: exact binomial 95% confidence interval; NE: not estimable.

Flagged percentages use all test encounters; missed percentages use test deterioration events. Sensitivity targets apply to training estimates, not guaranteed test performance.

3.3 Deterioration by score

Figure X. Observed deterioration rate across the START score range. Shaded band shows 95% confidence interval. Dashed line indicates optimal cutpoint (score = 14) by Youden index. Histogram shows score distribution.

Figure X. Observed deterioration rate across the START score range. Shaded band shows 95% confidence interval. Dashed line indicates optimal cutpoint (score = 14) by Youden index. Histogram shows score distribution.


4 Component Analysis (Univariable Logistic Regression)

Each START component variable entered separately.

Univariable logistic regression: association of each variable with clinical deterioration

Characteristic

OR

95% CI

p-value

Age group

16-19

—

—

20-39

0.94

0.62, 1.49

0.8

40-59

2.17

1.46, 3.40

<0.001

60-79

6.52

4.42, 10.1

<0.001

80+

13.6

9.22, 21.2

<0.001

Ambulance arrival

No

—

—

Yes

4.80

4.29, 5.36

<0.001

Triage category (ATS)

3 - Urgent

—

—

4 - Semi-Urgent

0.35

0.30, 0.40

<0.001

5 - Non-Urgent

0.11

0.07, 0.18

<0.001

Time of arrival

0800-1759

—

—

1800-2259

0.79

0.69, 0.91

0.001

2300-0759

0.81

0.70, 0.94

0.007

Prior hospital admission (30 days)

No

—

—

Yes

2.44

2.05, 2.88

<0.001

Presenting problem category

Abdominal/GI

—

—

Cardiovascular

0.81

0.54, 1.17

0.3

General symptoms

1.86

1.58, 2.18

<0.001

Febrile illness

1.65

1.24, 2.15

<0.001

Injury

0.29

0.22, 0.38

<0.001

Respiratory

2.37

1.94, 2.88

<0.001

Musculoskeletal

0.62

0.48, 0.79

<0.001

Neurological

0.73

0.57, 0.94

0.016

Mental health

0.48

0.32, 0.68

<0.001

Toxicological

0.35

0.09, 0.93

0.073

ENT/eye/head & neck

0.27

0.19, 0.37

<0.001

Administrative

0.83

0.59, 1.13

0.2

Genitourinary

0.56

0.39, 0.78

<0.001

Endocrine

2.61

1.22, 4.92

0.007

Obstetrics/Gynaecology

0.19

0.10, 0.32

<0.001

Skin/allergy

0.50

0.35, 0.68

<0.001

Other medical

2.66

0.15, 13.7

0.3

Abbreviations: CI = Confidence Interval, OR = Odds Ratio


5 Model B: Exploratory analysis: Firth START-predictor model

5.1 Fitting the model

## START-predictor model: Firth mean bias-reduced logistic regression
## deterioration ~ age + ambulance + triage + time_code + prior_adm + 
##     destiny
## Training observations: 33247
## 
## Rare-category coefficient and standard error:
##                       Estimate Std. Error
## destinyOther medical 0.7621785   1.674997

5.2 MV regression results

Firth mean bias-reduced logistic regression using START predictors

Characteristic

Adjusted OR

95% CI

p-value

Age group

16-19

—

—

20-39

0.67

0.41, 1.08

0.10

40-59

1.33

0.84, 2.11

0.2

60-79

3.14

2.00, 4.94

<0.001

80+

4.49

2.83, 7.13

<0.001

Ambulance arrival

No

—

—

Yes

2.63

2.27, 3.06

<0.001

Triage category (ATS)

3 - Urgent

—

—

4 - Semi-Urgent

0.53

0.44, 0.63

<0.001

5 - Non-Urgent

0.29

0.17, 0.48

<0.001

Time of arrival

0800-1759

—

—

1800-2259

0.84

0.70, 1.00

0.044

2300-0759

0.77

0.63, 0.93

0.006

Prior hospital admission (30 days)

No

—

—

Yes

1.39

1.13, 1.73

0.002

Presenting problem category

Abdominal/GI

—

—

Cardiovascular

0.45

0.27, 0.74

0.002

General symptoms

1.51

1.24, 1.85

<0.001

Febrile illness

1.53

1.09, 2.15

0.013

Injury

0.38

0.27, 0.53

<0.001

Respiratory

1.38

1.08, 1.76

0.010

Musculoskeletal

0.70

0.52, 0.95

0.024

Neurological

0.55

0.41, 0.75

<0.001

Mental health

0.56

0.36, 0.87

0.010

Toxicological

0.52

0.18, 1.54

0.2

ENT/eye/head & neck

0.44

0.30, 0.65

<0.001

Administrative

0.93

0.62, 1.39

0.7

Genitourinary

0.51

0.34, 0.76

<0.001

Endocrine

0.79

0.26, 2.38

0.7

Obstetrics/Gynaecology

0.54

0.28, 1.05

0.069

Skin/allergy

0.92

0.62, 1.37

0.7

Other medical

2.14

0.08, 57.1

0.6

Abbreviations: CI = Confidence Interval, OR = Odds Ratio

5.3 Test-set AUROC

## Firth START-predictor MV model AUC (test set): 0.8178
## 95% CI (DeLong): 0.7991 - 0.8365

5.4 Sensitivity analysis: ordinary versus Firth logistic regression

Sensitivity analysis: ordinary versus Firth logistic regression.

Measure

Ordinary logistic regression

Firth logistic regression

Mean predicted training risk (%)

2.770

2.806

Mean predicted test risk (%)

2.723

2.758

Test AUROC (95% CI)

0.8162
(0.7971 to 0.8353)

0.8178
(0.7991 to 0.8365)

Test calibration-in-the-large (95% CI; ideal 0)

0.0416
(-0.0608 to 0.1440)

0.0276
(-0.0747 to 0.1299)

Test calibration slope (95% CI; ideal 1)

0.9343
(0.8489 to 1.0197)

0.9600
(0.8735 to 1.0465)

Numerical convergence

Yes

Yes

Other medical: coefficient (standard error)

-9.690 (168.329)

0.762 (1.675)

Training: n = 33,247, observed event rate 2.770%. Test: n = 14,250, observed event rate 2.828%.

Both models use identical START predictor coding and training/test partitions. AUROC CIs use DeLong; calibration CIs use Wald intervals.

Numerical convergence does not establish a finite MLE under separation. The Other medical training category has nine encounters and no events; its MLE estimate is unstable.

Calibration regressions assess the original predictions only. They do not recalibrate either model.

5.5 HL+Calibration plot

## Firth START-predictor MV model calibration
## Hosmer-Lemeshow: X² = 15.202 | df = 8 | p = 0.0553

## START MV vs. START analogue

Original START versus Firth START-predictor MV model. Paired DeLong p = 0.000123. Difference in AUROC (MV - START) = 0.0196
Model AUC 95% CI
Original START score 0.7982 0.779 - 0.818
Firth START-predictor MV model 0.8178 0.799 - 0.837


6 Model C: Exploratory integer simplification

6.1 Training coefficients and integer weights

## Model C scale: 0.07330840 log-odds per point.
## Intercept retained for approximate probabilities: -4.03929 ; excluded from the point total.
## Reference categories contribute zero points; negative weights are retained.
Exploratory Model C: training coefficients and integer weights.
Term Coefficient Points Rounded coefficient
age20-39 -0.40529 -6 -0.43985
age40-59 0.28548 4 0.29323
age60-79 1.14558 16 1.17293
age80+ 1.50230 20 1.46617
ambulanceYes 0.96835 13 0.95301
triage4 - Semi-Urgent -0.64084 -9 -0.65978
triage5 - Non-Urgent -1.24892 -17 -1.24624
time_code1800-2259 -0.17847 -2 -0.14662
time_code2300-0759 -0.26562 -4 -0.29323
prior_admYes 0.33239 5 0.36654
destinyCardiovascular -0.80103 -11 -0.80639
destinyGeneral symptoms 0.41468 6 0.43985
destinyFebrile illness 0.42686 6 0.43985
destinyInjury -0.97479 -13 -0.95301
destinyRespiratory 0.32164 4 0.29323
destinyMusculoskeletal -0.35065 -5 -0.36654
destinyNeurological -0.59050 -8 -0.58647
destinyMental health -0.57472 -8 -0.58647
destinyToxicological -0.64744 -9 -0.65978
destinyENT/eye/head & neck -0.82436 -11 -0.80639
destinyAdministrative -0.07331 -1 -0.07331
destinyGenitourinary -0.66991 -9 -0.65978
destinyEndocrine -0.24196 -3 -0.21993
destinyObstetrics/Gynaecology -0.61519 -8 -0.58647
destinySkin/allergy -0.08269 -1 -0.07331
destinyOther medical 0.76218 10 0.73308

6.2 Test-set predictions

## Common test sample: n = 14250 | events = 403 | excluded = 0

6.3 Paired DeLong comparison

Exploratory three-model comparison on the common held-out test sample.
Model AUROC 95% CI
A. Original START score 0.7982 0.779 - 0.818
B. Firth START-predictor MV model 0.8178 0.799 - 0.837
C. Integer simplification of B 0.8174 0.799 - 0.836
## Paired DeLong: AUROC difference (B - C) = 0.0004 | 95% CI: -0.0003 to 0.0011 | p = 0.222

6.4 ROC comparison


6.5 Model C versus original START and threshold validation

Exploratory held-out comparison of Model C with original START.

Model

AUROC

95% CI (DeLong)

A: Original START

0.7982

0.7788 to 0.8177

C: Integer simplification

0.8174

0.7986 to 0.8362

Common test sample: n = 14,250, events = 403. Paired DeLong difference (C - A) = 0.0192; 95% CI 0.0091 to 0.0292; p = 0.000189.

Training derivation of integer thresholds for Model C.

Criterion

Rule

Training sensitivity (%)

Training specificity (%)

Youden

C score >= 8

75.8

76.3

Sensitivity >=85%

C score >= 0

86.5

61.2

Sensitivity >=90%

C score >= -3

90.1

56.3

Training cohort only: n = 33,247, events = 921. All rules use the existing signed Model C integer total; negative cutpoints are valid.

Youden ties favour higher specificity, then a higher cutpoint. Sensitivity targets select the highest integer cutpoint meeting the target in training.

Held-out test performance of training-derived Model C thresholds.

Training
criterion

Fixed
rule

Se (%)
95% CI

Sp (%)
95% CI

PPV (%)
95% CI

NPV (%)
95% CI

Flagged
n (%)

Missed
n / events (%)

Youden

C score >= 8

72.0
(67.3-76.3)

76.9
(76.2-77.6)

8.3
(7.4-9.3)

99.0
(98.7-99.1)

3,485
(24.5%)

113 / 403
(28.0%)

Sensitivity >=85%

C score >= 0

84.6
(80.7-88.0)

61.9
(61.1-62.7)

6.1
(5.5-6.7)

99.3
(99.1-99.4)

5,612
(39.4%)

62 / 403
(15.4%)

Sensitivity >=90%

C score >= -3

88.8
(85.3-91.7)

57.0
(56.2-57.8)

5.7
(5.1-6.3)

99.4
(99.2-99.6)

6,314
(44.3%)

45 / 403
(11.2%)

Test cohort: n = 14,250, events = 403. Thresholds were derived in the overall training cohort and applied unchanged; no test-set optimisation or model refitting.

Se: sensitivity; Sp: specificity; PPV/NPV: positive/negative predictive value. CIs are exact binomial. Flagged percentages use all test encounters; missed percentages use test deterioration events.

The 85% and 90% targets concern training point estimates and are not guaranteed in the test cohort. These are Model C thresholds, distinct from published START thresholds. NE: not estimable.


7 Exploratory subgroup analysis by admission status

Exploratory model discrimination by admission status in the test cohort.

Subgroup

N

Events

Event rate (%)
95% CI

A: START
AUROC (95% CI)

B: Firth
AUROC (95% CI)

C: Integer
AUROC (95% CI)

Admitted

4,273

391

9.15
(8.30-10.05)

0.6652
(0.6390-0.6915)

0.6931
(0.6680-0.7183)

0.6928
(0.6676-0.7180)

Not admitted

9,977

12

0.12
(0.06-0.21)

0.8283
(0.7198-0.9369)

0.8554
(0.7666-0.9442)

0.8559
(0.7685-0.9433)

Existing held-out test cohort only. A, B and C use identical encounters within each subgroup. AUROC CIs use DeLong; event-rate CIs are exact binomial.

Admission status describes final disposition. These analyses are exploratory; the non-admitted subgroup has few events. Differences between subgroup AUROCs do not establish an interaction. NE: not estimable.

Exploratory subgroup performance of fixed original START thresholds.

Subgroup

Training criterion
and fixed rule

Se (%)
95% CI

Sp (%)
95% CI

PPV (%)
95% CI

NPV (%)
95% CI

Flagged
n (%)

Missed
n / events

Admitted

Youden
START >= 16

69.8
(65.0-74.3)

55.2
(53.6-56.8)

13.57
(12.10-15.14)

94.78
(93.78-95.66)

2,012
(47.1%)

118 / 391

Admitted

Sensitivity >=85%
START >= 13

85.4
(81.5-88.8)

34.1
(32.6-35.6)

11.54
(10.40-12.76)

95.87
(94.68-96.85)

2,894
(67.7%)

57 / 391

Admitted

Sensitivity >=90%
START >= 11

92.3
(89.2-94.8)

25.1
(23.7-26.5)

11.04
(9.99-12.17)

97.01
(95.76-97.98)

3,269
(76.5%)

30 / 391

Not admitted

Youden
START >= 16

58.3
(27.7-84.8)

82.9
(82.1-83.6)

0.41
(0.16-0.84)

99.94
(99.86-99.98)

1,714
(17.2%)

5 / 12

Not admitted

Sensitivity >=85%
START >= 13

83.3
(51.6-97.9)

65.9
(65.0-66.8)

0.29
(0.14-0.54)

99.97
(99.89-100.00)

3,408
(34.2%)

2 / 12

Not admitted

Sensitivity >=90%
START >= 11

83.3
(51.6-97.9)

55.1
(54.1-56.0)

0.22
(0.11-0.41)

99.96
(99.87-100.00)

4,488
(45.0%)

2 / 12

Model A (published START) thresholds were selected in the overall training cohort and applied unchanged within each test subgroup.

Se: sensitivity; Sp: specificity; PPV/NPV: positive/negative predictive value. Metric CIs are exact binomial. Flagged percentages use the subgroup denominator; missed counts use subgroup deterioration events.

Sensitivity targets concern the overall training cohort and are not guaranteed within subgroups. These thresholds apply to original START, not Model C.


8 Additional exploratory analyses

8.1 ATS category and age versus START models

Exploratory held-out discrimination: ATS, ATS plus age, and START models.

Model

AUROC

95% CI (DeLong)

ATS category alone

0.6396

0.6219 to 0.6573

ATS category + age band

0.7709

0.7492 to 0.7926

Original START

0.7982

0.7788 to 0.8177

Firth START-predictor model

0.8178

0.7991 to 0.8365

Training: n = 33,247, events = 921. Evaluation: identical 14,250 held-out encounters, including 403 events, for every model.

Reduced models use Firth regression with categorical ATS and the existing age bands (16-19, 20-39, 40-59, 60-79, 80+), without interaction or variable selection. ATS-only AUROC matches raw ATS ranking.

Exploratory paired AUROC comparisons against the simpler predictors.

Contrast

AUROC
difference

95% CI

Raw p

Holm p

Original START minus ATS category alone

0.1587

0.1413 to 0.1760

< 1e-16

< 1e-16

Firth START-predictor model minus ATS category alone

0.1783

0.1563 to 0.2003

< 1e-16

< 1e-16

Original START minus ATS category + age band

0.0273

0.0127 to 0.0420

0.000257

0.000257

Firth START-predictor model minus ATS category + age band

0.0469

0.0337 to 0.0602

3.7e-12

7.4e-12

Paired DeLong contrasts use identical test encounters. Positive differences favour the START benchmark. Holm adjustment covers these four exploratory comparisons; confidence intervals are unadjusted.

This assesses discrimination. A nonsignificant difference would not establish equivalence or comparable clinical utility.

8.2 Recorded deterioration components in non-admitted encounters

Recorded deterioration components in the 12 non-admitted test encounters.

Component

Encounters

% of 12 cases

RRC

0

0.0

Code Blue

0

0.0

ICU admission

0

0.0

Mortality

12

100.0

Existing held-out test cohort only. Component counts are inclusive: an encounter can have more than one component.

These are recorded binary flags, not adjudicated causes or timings. In particular, the Mortality flag does not establish the time, place or circumstances of death.

Mutually exclusive flag combinations among non-admitted deterioration cases.

Recorded component combination

Encounters

Mortality only

12

Combinations are mutually exclusive and sum to 12 encounters.

An encounter-level audit, including the source identifier, is retained in na12_audit; identifiers are not printed in this report.


9 Session Info