1 Overview

This tutorial develops a complete clinical prediction-modeling workflow using the NHANES R package and diabetes as the binary outcome.

The workflow covers:

  1. Research question and target population
  2. Data import and data dictionary
  3. Duplicate, type, coding, range, and logical checks
  4. Missing-data assessment
  5. Outlier and influential-observation assessment
  6. Exploratory data analysis (EDA)
  7. Continuous-variable handling and nonlinear effects
  8. Multicollinearity assessment
  9. Multiple imputation
  10. Logistic regression model development
  11. Interaction assessment
  12. Model-selection approaches
  13. Penalized regression with LASSO
  14. Apparent model performance
  15. Bootstrap internal validation
  16. Discrimination
  17. Calibration
  18. Brier score
  19. Decision-curve analysis
  20. Threshold-based classification performance
  21. Sensitivity analyses
  22. Subgroup performance
  23. Temporal validation using NHANES survey cycles
  24. Recalibration
  25. Individual prediction
  26. Nomogram
  27. Recommended reporting framework

Important: NHANES::NHANES is a teaching dataset derived from NHANES. This example predicts prevalent diabetes at the time of survey, not future incident diabetes. For a publication-quality NHANES analysis, use the original CDC NHANES files and account for the complex survey design, weights, strata, and PSU variables.

2 1. Setup

2.1 1.1 Install required packages

The following chunk installs any missing packages. It only needs internet access the first time a package is installed.

required_packages <- c(
  "NHANES",
  "tidyverse",
  "gtsummary",
  "naniar",
  "mice",
  "car",
  "rms",
  "pROC",
  "glmnet",
  "broom"
)

missing_packages <- required_packages[
  !required_packages %in% rownames(installed.packages())
]

if (length(missing_packages) > 0) {
  install.packages(missing_packages)
}

2.2 1.2 Load packages

library(NHANES)
library(tidyverse)
library(gtsummary)
library(naniar)
library(mice)
library(car)
library(rms)
library(pROC)
library(glmnet)
library(broom)

2.3 1.3 Reproducibility settings

set.seed(2026)
options(stringsAsFactors = FALSE)

3 2. Research Question

We want to predict whether an adult participant has diabetes using demographic, anthropometric, blood-pressure, lipid, and lifestyle information.

The binary outcome is

\[ Y_i = \begin{cases} 1, & \text{Diabetes = Yes} \\ 0, & \text{Diabetes = No} \end{cases} \]

The candidate predictor set is

\[ \text{Age} + \text{BMI} + \text{BPSysAve} + \text{TotChol} + \text{Gender} + \text{Race1} + \text{PhysActive} + \text{Smoke100}. \]

We deliberately do not use DiabetesAge, because it is downstream of diabetes diagnosis and would create outcome leakage.

4 3. Load and Inspect the NHANES Data

data("NHANES")

cat("Rows:", nrow(NHANES), "\n")
## Rows: 10000
cat("Columns:", ncol(NHANES), "\n")
## Columns: 76
glimpse(NHANES)
## Rows: 10,000
## Columns: 76
## $ ID               <int> 51624, 51624, 51624, 51625, 51630, 51638, 51646, 5164…
## $ SurveyYr         <fct> 2009_10, 2009_10, 2009_10, 2009_10, 2009_10, 2009_10,…
## $ Gender           <fct> male, male, male, male, female, male, male, female, f…
## $ Age              <int> 34, 34, 34, 4, 49, 9, 8, 45, 45, 45, 66, 58, 54, 10, …
## $ AgeDecade        <fct>  30-39,  30-39,  30-39,  0-9,  40-49,  0-9,  0-9,  40…
## $ AgeMonths        <int> 409, 409, 409, 49, 596, 115, 101, 541, 541, 541, 795,…
## $ Race1            <fct> White, White, White, Other, White, White, White, Whit…
## $ Race3            <fct> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ Education        <fct> High School, High School, High School, NA, Some Colle…
## $ MaritalStatus    <fct> Married, Married, Married, NA, LivePartner, NA, NA, M…
## $ HHIncome         <fct> 25000-34999, 25000-34999, 25000-34999, 20000-24999, 3…
## $ HHIncomeMid      <int> 30000, 30000, 30000, 22500, 40000, 87500, 60000, 8750…
## $ Poverty          <dbl> 1.36, 1.36, 1.36, 1.07, 1.91, 1.84, 2.33, 5.00, 5.00,…
## $ HomeRooms        <int> 6, 6, 6, 9, 5, 6, 7, 6, 6, 6, 5, 10, 6, 10, 10, 4, 3,…
## $ HomeOwn          <fct> Own, Own, Own, Own, Rent, Rent, Own, Own, Own, Own, O…
## $ Work             <fct> NotWorking, NotWorking, NotWorking, NA, NotWorking, N…
## $ Weight           <dbl> 87.4, 87.4, 87.4, 17.0, 86.7, 29.8, 35.2, 75.7, 75.7,…
## $ Length           <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ HeadCirc         <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ Height           <dbl> 164.7, 164.7, 164.7, 105.4, 168.4, 133.1, 130.6, 166.…
## $ BMI              <dbl> 32.22, 32.22, 32.22, 15.30, 30.57, 16.82, 20.64, 27.2…
## $ BMICatUnder20yrs <fct> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ BMI_WHO          <fct> 30.0_plus, 30.0_plus, 30.0_plus, 12.0_18.5, 30.0_plus…
## $ Pulse            <int> 70, 70, 70, NA, 86, 82, 72, 62, 62, 62, 60, 62, 76, 8…
## $ BPSysAve         <int> 113, 113, 113, NA, 112, 86, 107, 118, 118, 118, 111, …
## $ BPDiaAve         <int> 85, 85, 85, NA, 75, 47, 37, 64, 64, 64, 63, 74, 85, 6…
## $ BPSys1           <int> 114, 114, 114, NA, 118, 84, 114, 106, 106, 106, 124, …
## $ BPDia1           <int> 88, 88, 88, NA, 82, 50, 46, 62, 62, 62, 64, 76, 86, 6…
## $ BPSys2           <int> 114, 114, 114, NA, 108, 84, 108, 118, 118, 118, 108, …
## $ BPDia2           <int> 88, 88, 88, NA, 74, 50, 36, 68, 68, 68, 62, 72, 88, 6…
## $ BPSys3           <int> 112, 112, 112, NA, 116, 88, 106, 118, 118, 118, 114, …
## $ BPDia3           <int> 82, 82, 82, NA, 76, 44, 38, 60, 60, 60, 64, 76, 82, 7…
## $ Testosterone     <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ DirectChol       <dbl> 1.29, 1.29, 1.29, NA, 1.16, 1.34, 1.55, 2.12, 2.12, 2…
## $ TotChol          <dbl> 3.49, 3.49, 3.49, NA, 6.70, 4.86, 4.09, 5.82, 5.82, 5…
## $ UrineVol1        <int> 352, 352, 352, NA, 77, 123, 238, 106, 106, 106, 113, …
## $ UrineFlow1       <dbl> NA, NA, NA, NA, 0.094, 1.538, 1.322, 1.116, 1.116, 1.…
## $ UrineVol2        <int> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ UrineFlow2       <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ Diabetes         <fct> No, No, No, No, No, No, No, No, No, No, No, No, No, N…
## $ DiabetesAge      <int> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ HealthGen        <fct> Good, Good, Good, NA, Good, NA, NA, Vgood, Vgood, Vgo…
## $ DaysPhysHlthBad  <int> 0, 0, 0, NA, 0, NA, NA, 0, 0, 0, 10, 0, 4, NA, NA, 0,…
## $ DaysMentHlthBad  <int> 15, 15, 15, NA, 10, NA, NA, 3, 3, 3, 0, 0, 0, NA, NA,…
## $ LittleInterest   <fct> Most, Most, Most, NA, Several, NA, NA, None, None, No…
## $ Depressed        <fct> Several, Several, Several, NA, Several, NA, NA, None,…
## $ nPregnancies     <int> NA, NA, NA, NA, 2, NA, NA, 1, 1, 1, NA, NA, NA, NA, N…
## $ nBabies          <int> NA, NA, NA, NA, 2, NA, NA, NA, NA, NA, NA, NA, NA, NA…
## $ Age1stBaby       <int> NA, NA, NA, NA, 27, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ SleepHrsNight    <int> 4, 4, 4, NA, 8, NA, NA, 8, 8, 8, 7, 5, 4, NA, 5, 7, N…
## $ SleepTrouble     <fct> Yes, Yes, Yes, NA, Yes, NA, NA, No, No, No, No, No, Y…
## $ PhysActive       <fct> No, No, No, NA, No, NA, NA, Yes, Yes, Yes, Yes, Yes, …
## $ PhysActiveDays   <int> NA, NA, NA, NA, NA, NA, NA, 5, 5, 5, 7, 5, 1, NA, 2, …
## $ TVHrsDay         <fct> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ CompHrsDay       <fct> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ TVHrsDayChild    <int> NA, NA, NA, 4, NA, 5, 1, NA, NA, NA, NA, NA, NA, 4, N…
## $ CompHrsDayChild  <int> NA, NA, NA, 1, NA, 0, 6, NA, NA, NA, NA, NA, NA, 3, N…
## $ Alcohol12PlusYr  <fct> Yes, Yes, Yes, NA, Yes, NA, NA, Yes, Yes, Yes, Yes, Y…
## $ AlcoholDay       <int> NA, NA, NA, NA, 2, NA, NA, 3, 3, 3, 1, 2, 6, NA, NA, …
## $ AlcoholYear      <int> 0, 0, 0, NA, 20, NA, NA, 52, 52, 52, 100, 104, 364, N…
## $ SmokeNow         <fct> No, No, No, NA, Yes, NA, NA, NA, NA, NA, No, NA, NA, …
## $ Smoke100         <fct> Yes, Yes, Yes, NA, Yes, NA, NA, No, No, No, Yes, No, …
## $ Smoke100n        <fct> Smoker, Smoker, Smoker, NA, Smoker, NA, NA, Non-Smoke…
## $ SmokeAge         <int> 18, 18, 18, NA, 38, NA, NA, NA, NA, NA, 13, NA, NA, N…
## $ Marijuana        <fct> Yes, Yes, Yes, NA, Yes, NA, NA, Yes, Yes, Yes, NA, Ye…
## $ AgeFirstMarij    <int> 17, 17, 17, NA, 18, NA, NA, 13, 13, 13, NA, 19, 15, N…
## $ RegularMarij     <fct> No, No, No, NA, No, NA, NA, No, No, No, NA, Yes, Yes,…
## $ AgeRegMarij      <int> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 20, 15, N…
## $ HardDrugs        <fct> Yes, Yes, Yes, NA, Yes, NA, NA, No, No, No, No, Yes, …
## $ SexEver          <fct> Yes, Yes, Yes, NA, Yes, NA, NA, Yes, Yes, Yes, Yes, Y…
## $ SexAge           <int> 16, 16, 16, NA, 12, NA, NA, 13, 13, 13, 17, 22, 12, N…
## $ SexNumPartnLife  <int> 8, 8, 8, NA, 10, NA, NA, 20, 20, 20, 15, 7, 100, NA, …
## $ SexNumPartYear   <int> 1, 1, 1, NA, 1, NA, NA, 0, 0, 0, NA, 1, 1, NA, NA, 1,…
## $ SameSex          <fct> No, No, No, NA, Yes, NA, NA, Yes, Yes, Yes, No, No, N…
## $ SexOrientation   <fct> Heterosexual, Heterosexual, Heterosexual, NA, Heteros…
## $ PregnantNow      <fct> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…

Check the variables that will be used:

required_vars <- c(
  "ID", "SurveyYr", "Diabetes", "Age", "Gender", "Race1",
  "BMI", "BPSysAve", "BPDiaAve", "TotChol", "DirectChol",
  "PhysActive", "Smoke100", "Education", "Poverty", "SleepHrsNight"
)

missing_vars <- setdiff(required_vars, names(NHANES))

if (length(missing_vars) > 0) {
  stop(
    "The following expected NHANES variables are missing: ",
    paste(missing_vars, collapse = ", ")
  )
}

5 4. Create the Initial Analysis Dataset

We restrict the example to adults aged 20 years or older and participants with a defined diabetes outcome.

nhanes0 <- NHANES %>%
  select(all_of(required_vars))

nhanes1 <- nhanes0 %>%
  filter(
    Age >= 20,
    Diabetes %in% c("Yes", "No")
  ) %>%
  mutate(
    Diabetes_num = if_else(Diabetes == "Yes", 1L, 0L)
  )

cat("Adult analytical cohort N =", nrow(nhanes1), "\n")
## Adult analytical cohort N = 7233

Check outcome coding:

table(nhanes1$Diabetes, nhanes1$Diabetes_num, useNA = "ifany")
##      
##          0    1
##   No  6490    0
##   Yes    0  743
prop.table(table(nhanes1$Diabetes))
## 
##        No       Yes 
## 0.8972764 0.1027236

6 5. Basic Data-Quality Checks

6.1 5.1 Duplicate subject IDs

duplicate_ids <- nhanes1 %>%
  count(ID) %>%
  filter(n > 1)

duplicate_ids

A duplicate ID should never be deleted automatically. It may represent a true duplicate, repeated measurement, merge error, or multiple records per participant.

6.2 5.2 Variable classes

sapply(nhanes1, class)
##            ID      SurveyYr      Diabetes           Age        Gender 
##     "integer"      "factor"      "factor"     "integer"      "factor" 
##         Race1           BMI      BPSysAve      BPDiaAve       TotChol 
##      "factor"     "numeric"     "integer"     "integer"     "numeric" 
##    DirectChol    PhysActive      Smoke100     Education       Poverty 
##     "numeric"      "factor"      "factor"      "factor"     "numeric" 
## SleepHrsNight  Diabetes_num 
##     "integer"     "integer"

6.3 5.3 Categorical coding

list(
  Diabetes = levels(nhanes1$Diabetes),
  SurveyYr = levels(nhanes1$SurveyYr),
  Gender = levels(nhanes1$Gender),
  Race1 = levels(nhanes1$Race1),
  PhysActive = levels(nhanes1$PhysActive),
  Smoke100 = levels(nhanes1$Smoke100),
  Education = levels(nhanes1$Education)
)
## $Diabetes
## [1] "No"  "Yes"
## 
## $SurveyYr
## [1] "2009_10" "2011_12"
## 
## $Gender
## [1] "female" "male"  
## 
## $Race1
## [1] "Black"    "Hispanic" "Mexican"  "White"    "Other"   
## 
## $PhysActive
## [1] "No"  "Yes"
## 
## $Smoke100
## [1] "No"  "Yes"
## 
## $Education
## [1] "8th Grade"      "9 - 11th Grade" "High School"    "Some College"  
## [5] "College Grad"

6.4 5.4 Continuous-variable ranges

range_summary <- nhanes1 %>%
  summarise(
    across(
      c(Age, BMI, BPSysAve, BPDiaAve, TotChol, DirectChol, Poverty, SleepHrsNight),
      list(
        min = ~ min(.x, na.rm = TRUE),
        max = ~ max(.x, na.rm = TRUE)
      )
    )
  )

range_summary

6.5 5.5 Flag potentially impossible values

The following rules are screening rules, not automatic deletion rules.

flagged_values <- nhanes1 %>%
  filter(
    Age < 20 | Age > 100 |
      (!is.na(BMI) & BMI <= 0) |
      (!is.na(BPSysAve) & BPSysAve <= 0) |
      (!is.na(BPDiaAve) & BPDiaAve < 0) |
      (!is.na(TotChol) & TotChol <= 0)
  )

flagged_values %>%
  select(ID, Age, BMI, BPSysAve, BPDiaAve, TotChol) %>%
  head(20)

7 6. Missing-Data Assessment

7.1 6.1 Missing percentage by variable

missing_summary <- nhanes1 %>%
  summarise(
    across(
      everything(),
      ~ mean(is.na(.x)) * 100
    )
  ) %>%
  pivot_longer(
    cols = everything(),
    names_to = "Variable",
    values_to = "Missing_Percent"
  ) %>%
  arrange(desc(Missing_Percent))

missing_summary

7.2 6.2 Missingness visualization

vis_miss(nhanes1)

gg_miss_var(nhanes1)

7.3 6.3 Missing-data pattern for modeling variables

model_vars_for_missing <- nhanes1 %>%
  select(
    Diabetes_num,
    Age,
    BMI,
    BPSysAve,
    TotChol,
    Gender,
    Race1,
    PhysActive,
    Smoke100
  )

md.pattern(model_vars_for_missing)

##      Diabetes_num Age Gender Race1 PhysActive Smoke100 BMI BPSysAve TotChol    
## 6575            1   1      1     1          1        1   1        1       1   0
## 342             1   1      1     1          1        1   1        1       0   1
## 214             1   1      1     1          1        1   1        0       1   1
## 39              1   1      1     1          1        1   1        0       0   2
## 45              1   1      1     1          1        1   0        1       1   1
## 7               1   1      1     1          1        1   0        1       0   2
## 9               1   1      1     1          1        1   0        0       1   2
## 2               1   1      1     1          1        1   0        0       0   3
##                 0   0      0     0          0        0  63      264     390 717

8 7. Descriptive Analysis and EDA

8.1 7.1 Table 1 by diabetes status

nhanes1 %>%
  select(
    Diabetes,
    Age,
    Gender,
    Race1,
    BMI,
    BPSysAve,
    TotChol,
    PhysActive,
    Smoke100
  ) %>%
  tbl_summary(
    by = Diabetes,
    statistic = list(
      all_continuous() ~ "{mean} ({sd}); median {median} [{p25}, {p75}]",
      all_categorical() ~ "{n} ({p}%)"
    ),
    missing = "ifany"
  )
Characteristic No
N = 6,4901		 Yes
N = 7431
Age 46 (17); median 44 [31, 57] 60 (13); median 61 [52, 70]
Gender

    female 3,337 (51%) 344 (46%)
    male 3,153 (49%) 399 (54%)
Race1

    Black 691 (11%) 130 (17%)
    Hispanic 372 (5.7%) 44 (5.9%)
    Mexican 537 (8.3%) 64 (8.6%)
    White 4,408 (68%) 439 (59%)
    Other 482 (7.4%) 66 (8.9%)
BMI 28 (6); median 27 [24, 32] 33 (8); median 32 [27, 37]
    Unknown 52 11
BPSysAve 120 (17); median 118 [109, 128] 128 (19); median 127 [115, 140]
    Unknown 248 16
TotChol 5.10 (1.04); median 5.02 [4.34, 5.72] 4.79 (1.16); median 4.68 [3.98, 5.43]
    Unknown 331 59
PhysActive 3,525 (54%) 271 (36%)
Smoke100 2,836 (44%) 373 (50%)
1 Mean (SD); median Median [Q1, Q3]; n (%)

For prediction modeling, a variable should not be excluded simply because its univariable P-value is greater than 0.05.

8.2 7.2 Distribution of age

ggplot(nhanes1, aes(x = Age)) +
  geom_histogram(bins = 30) +
  labs(
    title = "Distribution of Age",
    x = "Age (years)",
    y = "Count"
  )

8.3 7.3 Distribution of BMI

ggplot(nhanes1, aes(x = BMI)) +
  geom_histogram(bins = 30) +
  labs(
    title = "Distribution of BMI",
    x = "BMI",
    y = "Count"
  )
## Warning: Removed 63 rows containing non-finite outside the scale range
## (`stat_bin()`).

8.4 7.4 Distribution of systolic blood pressure

ggplot(nhanes1, aes(x = BPSysAve)) +
  geom_histogram(bins = 30) +
  labs(
    title = "Distribution of Average Systolic Blood Pressure",
    x = "Average systolic BP",
    y = "Count"
  )
## Warning: Removed 264 rows containing non-finite outside the scale range
## (`stat_bin()`).

8.5 7.5 Diabetes versus BMI

ggplot(nhanes1, aes(x = Diabetes, y = BMI)) +
  geom_boxplot() +
  labs(
    title = "BMI by Diabetes Status",
    x = "Diabetes",
    y = "BMI"
  )
## Warning: Removed 63 rows containing non-finite outside the scale range
## (`stat_boxplot()`).

8.6 7.6 Diabetes versus age

ggplot(nhanes1, aes(x = Diabetes, y = Age)) +
  geom_boxplot() +
  labs(
    title = "Age by Diabetes Status",
    x = "Diabetes",
    y = "Age"
  )

9 8. Outlier Assessment

9.1 8.1 IQR flagging for BMI

q1_bmi <- quantile(nhanes1$BMI, 0.25, na.rm = TRUE)
q3_bmi <- quantile(nhanes1$BMI, 0.75, na.rm = TRUE)
iqr_bmi <- q3_bmi - q1_bmi

bmi_lower <- q1_bmi - 1.5 * iqr_bmi
bmi_upper <- q3_bmi + 1.5 * iqr_bmi

bmi_outliers <- nhanes1 %>%
  filter(
    !is.na(BMI),
    BMI < bmi_lower | BMI > bmi_upper
  ) %>%
  select(ID, Age, Gender, BMI, Diabetes)

head(bmi_outliers, 20)

A statistical outlier is not automatically a data error. Clinically plausible extreme values should generally be retained unless there is evidence of measurement or data-entry error.

10 9. Correlation and Multicollinearity Screening

10.1 9.1 Correlation matrix among continuous variables

cor_data <- nhanes1 %>%
  select(
    Age,
    BMI,
    BPSysAve,
    BPDiaAve,
    TotChol,
    DirectChol
  )

round(
  cor(cor_data, use = "pairwise.complete.obs"),
  2
)
##              Age   BMI BPSysAve BPDiaAve TotChol DirectChol
## Age         1.00  0.04     0.41    -0.06    0.13       0.11
## BMI         0.04  1.00     0.13     0.12    0.00      -0.34
## BPSysAve    0.41  0.13     1.00     0.33    0.13       0.00
## BPDiaAve   -0.06  0.12     0.33     1.00    0.17      -0.04
## TotChol     0.13  0.00     0.13     0.17    1.00       0.22
## DirectChol  0.11 -0.34     0.00    -0.04    0.22       1.00

11 10. Construct the Modeling Dataset

model_data <- nhanes1 %>%
  select(
    ID,
    SurveyYr,
    Diabetes,
    Diabetes_num,
    Age,
    BMI,
    BPSysAve,
    TotChol,
    Gender,
    Race1,
    PhysActive,
    Smoke100
  )

12 11. Complete-Case Dataset for Sensitivity Analysis

model_cc <- model_data %>%
  drop_na(
    Age,
    BMI,
    BPSysAve,
    TotChol,
    Gender,
    Race1,
    PhysActive,
    Smoke100
  )

cat("Original modeling N =", nrow(model_data), "\n")
## Original modeling N = 7233
cat("Complete-case N =", nrow(model_cc), "\n")
## Complete-case N = 6575
cat(
  "Complete-case retention =",
  round(100 * nrow(model_cc) / nrow(model_data), 1),
  "%\n"
)
## Complete-case retention = 90.9 %

12.1 11.1 Compare complete versus incomplete cases

model_data <- model_data %>%
  mutate(
    CompleteCase = if_else(
      complete.cases(
        Age,
        BMI,
        BPSysAve,
        TotChol,
        Gender,
        Race1,
        PhysActive,
        Smoke100
      ),
      "Complete",
      "Missing"
    )
  )

model_data %>%
  group_by(CompleteCase) %>%
  summarise(
    N = n(),
    Mean_Age = mean(Age, na.rm = TRUE),
    Mean_BMI = mean(BMI, na.rm = TRUE),
    Diabetes_Rate = mean(Diabetes_num, na.rm = TRUE),
    .groups = "drop"
  )

13 12. Multiple Imputation with MICE

13.1 12.1 Prepare the imputation dataset

The outcome is included in the imputation model because it can help predict missing predictor values. The outcome itself is not imputed here.

mi_data <- model_data %>%
  select(
    Diabetes_num,
    Age,
    BMI,
    BPSysAve,
    TotChol,
    Gender,
    Race1,
    PhysActive,
    Smoke100
  )

13.2 12.2 Inspect default MICE methods

ini <- mice(
  mi_data,
  maxit = 0,
  printFlag = FALSE
)

method <- ini$method
predictor_matrix <- ini$predictorMatrix

method
## Diabetes_num          Age          BMI     BPSysAve      TotChol       Gender 
##           ""           ""        "pmm"        "pmm"        "pmm"           "" 
##        Race1   PhysActive     Smoke100 
##           ""           ""           ""

Prevent imputation of the observed outcome:

method["Diabetes_num"] <- ""
predictor_matrix[, "Diabetes_num"] <- 1
predictor_matrix["Diabetes_num", ] <- 0

method
## Diabetes_num          Age          BMI     BPSysAve      TotChol       Gender 
##           ""           ""        "pmm"        "pmm"        "pmm"           "" 
##        Race1   PhysActive     Smoke100 
##           ""           ""           ""

13.3 12.3 Run multiple imputation

set.seed(2026)

imp <- mice(
  mi_data,
  m = 20,
  maxit = 20,
  method = method,
  predictorMatrix = predictor_matrix,
  seed = 2026,
  printFlag = FALSE
)

imp
## Class: mids
## Number of multiple imputations:  20 
## Imputation methods:
## Diabetes_num          Age          BMI     BPSysAve      TotChol       Gender 
##           ""           ""        "pmm"        "pmm"        "pmm"           "" 
##        Race1   PhysActive     Smoke100 
##           ""           ""           "" 
## PredictorMatrix:
##              Diabetes_num Age BMI BPSysAve TotChol Gender Race1 PhysActive
## Diabetes_num            0   0   0        0       0      0     0          0
## Age                     1   0   1        1       1      1     1          1
## BMI                     1   1   0        1       1      1     1          1
## BPSysAve                1   1   1        0       1      1     1          1
## TotChol                 1   1   1        1       0      1     1          1
## Gender                  1   1   1        1       1      0     1          1
##              Smoke100
## Diabetes_num        0
## Age                 1
## BMI                 1
## BPSysAve            1
## TotChol             1
## Gender              1

13.4 12.4 Inspect convergence

plot(imp)

13.5 12.5 Compare observed and imputed BMI distributions

densityplot(imp, ~ BMI)

14 13. Fit the Prespecified Logistic Model Across Imputed Datasets

fit_mi <- with(
  imp,
  glm(
    Diabetes_num ~
      Age +
      BMI +
      BPSysAve +
      TotChol +
      Gender +
      Race1 +
      PhysActive +
      Smoke100,
    family = binomial()
  )
)

pooled_fit <- pool(fit_mi)

Pooled odds ratios:

summary(
  pooled_fit,
  conf.int = TRUE,
  exponentiate = TRUE
)

Important statistical distinction: The pooled coefficients are useful for regression inference. A fully rigorous prediction-validation workflow with missing data should repeat imputation inside each resampling iteration. The remainder of this tutorial uses one completed dataset for transparent teaching of model diagnostics and validation. That simplification should not be mistaken for the strictest publication-level MI + bootstrap procedure.

15 14. Create a Completed Dataset for the Remaining Demonstration

model_complete <- complete(imp, action = 1)

summary(model_complete)
##   Diabetes_num         Age             BMI           BPSysAve    
##  Min.   :0.0000   Min.   :20.00   Min.   :15.02   Min.   : 78.0  
##  1st Qu.:0.0000   1st Qu.:33.00   1st Qu.:24.10   1st Qu.:109.0  
##  Median :0.0000   Median :46.00   Median :27.75   Median :119.0  
##  Mean   :0.1027   Mean   :47.17   Mean   :28.76   Mean   :120.9  
##  3rd Qu.:0.0000   3rd Qu.:60.00   3rd Qu.:32.20   3rd Qu.:130.0  
##  Max.   :1.0000   Max.   :80.00   Max.   :81.25   Max.   :226.0  
##     TotChol          Gender          Race1      PhysActive Smoke100  
##  Min.   : 1.530   female:3681   Black   : 821   No :3437   No :4024  
##  1st Qu.: 4.290   male  :3552   Hispanic: 416   Yes:3796   Yes:3209  
##  Median : 4.990                 Mexican : 601                        
##  Mean   : 5.069                 White   :4847                        
##  3rd Qu.: 5.690                 Other   : 548                        
##  Max.   :13.650

16 15. Fit a Full Logistic Regression Model

fit_full <- glm(
  Diabetes_num ~
    Age +
    BMI +
    BPSysAve +
    TotChol +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = model_complete,
  family = binomial()
)

summary(fit_full)
## 
## Call:
## glm(formula = Diabetes_num ~ Age + BMI + BPSysAve + TotChol + 
##     Gender + Race1 + PhysActive + Smoke100, family = binomial(), 
##     data = model_complete)
## 
## Coefficients:
##                Estimate Std. Error z value Pr(>|z|)    
## (Intercept)   -7.112020   0.445556 -15.962  < 2e-16 ***
## Age            0.060354   0.003116  19.369  < 2e-16 ***
## BMI            0.094418   0.006056  15.591  < 2e-16 ***
## BPSysAve       0.004711   0.002364   1.993  0.04625 *  
## TotChol       -0.287436   0.043076  -6.673 2.51e-11 ***
## Gendermale     0.323390   0.088541   3.652  0.00026 ***
## Race1Hispanic -0.114461   0.205495  -0.557  0.57753    
## Race1Mexican  -0.025664   0.181059  -0.142  0.88728    
## Race1White    -0.747630   0.123100  -6.073 1.25e-09 ***
## Race1Other     0.222607   0.183607   1.212  0.22535    
## PhysActiveYes -0.170460   0.089499  -1.905  0.05683 .  
## Smoke100Yes    0.163988   0.087292   1.879  0.06030 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 4788.6  on 7232  degrees of freedom
## Residual deviance: 3837.1  on 7221  degrees of freedom
## AIC: 3861.1
## 
## Number of Fisher Scoring iterations: 6

Odds ratios and confidence intervals:

broom::tidy(
  fit_full,
  exponentiate = TRUE,
  conf.int = TRUE
)

For a prediction model, statistical significance is not the main criterion for retaining a prespecified predictor.

17 16. Multicollinearity Assessment

vif_result <- car::vif(fit_full)
vif_result
##                GVIF Df GVIF^(1/(2*Df))
## Age        1.351138  1        1.162385
## BMI        1.122581  1        1.059519
## BPSysAve   1.156118  1        1.075229
## TotChol    1.045168  1        1.022335
## Gender     1.086153  1        1.042187
## Race1      1.160336  4        1.018763
## PhysActive 1.065883  1        1.032416
## Smoke100   1.060550  1        1.029830

If vif() returns generalized VIF values for multi-level factors, the commonly reviewed quantity is

\[ GVIF^{1/(2df)}. \]

18 17. Assess Nonlinear Predictor Effects with Restricted Cubic Splines

18.1 17.1 Prepare the rms environment

dd <- datadist(model_complete)
options(datadist = "dd")

18.2 17.2 Linear logistic model using lrm()

fit_linear <- lrm(
  Diabetes_num ~
    Age +
    BMI +
    BPSysAve +
    TotChol +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = model_complete,
  x = TRUE,
  y = TRUE
)

fit_linear
## Logistic Regression Model
## 
## lrm(formula = Diabetes_num ~ Age + BMI + BPSysAve + TotChol + 
##     Gender + Race1 + PhysActive + Smoke100, data = model_complete, 
##     x = TRUE, y = TRUE)
## 
##                        Model Likelihood      Discrimination    Rank Discrim.    
##                              Ratio Test             Indexes          Indexes    
## Obs          7233    LR chi2     951.51      R2       0.255    C       0.820    
##  0           6490    d.f.            11    R2(11,7233)0.122    Dxy     0.639    
##  1            743    Pr(> chi2) <0.0001    R2(11,2000)0.375    gamma   0.639    
## max |deriv| 5e-05                            Brier    0.078    tau-a   0.118    
## 
##                Coef    S.E.   Wald Z Pr(>|Z|)
## Intercept      -7.1120 0.4456 -15.96 <0.0001 
## Age             0.0604 0.0031  19.37 <0.0001 
## BMI             0.0944 0.0061  15.59 <0.0001 
## BPSysAve        0.0047 0.0024   1.99 0.0463  
## TotChol        -0.2874 0.0431  -6.67 <0.0001 
## Gender=male     0.3234 0.0885   3.65 0.0003  
## Race1=Hispanic -0.1145 0.2055  -0.56 0.5775  
## Race1=Mexican  -0.0257 0.1811  -0.14 0.8873  
## Race1=White    -0.7476 0.1231  -6.07 <0.0001 
## Race1=Other     0.2226 0.1836   1.21 0.2254  
## PhysActive=Yes -0.1705 0.0895  -1.90 0.0568  
## Smoke100=Yes    0.1640 0.0873   1.88 0.0603

18.3 17.3 Restricted cubic spline model

fit_spline <- lrm(
  Diabetes_num ~
    rcs(Age, 4) +
    rcs(BMI, 4) +
    rcs(BPSysAve, 4) +
    rcs(TotChol, 4) +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = model_complete,
  x = TRUE,
  y = TRUE
)

fit_spline
## Logistic Regression Model
## 
## lrm(formula = Diabetes_num ~ rcs(Age, 4) + rcs(BMI, 4) + rcs(BPSysAve, 
##     4) + rcs(TotChol, 4) + Gender + Race1 + PhysActive + Smoke100, 
##     data = model_complete, x = TRUE, y = TRUE)
## 
##                         Model Likelihood      Discrimination    Rank Discrim.    
##                               Ratio Test             Indexes          Indexes    
## Obs           7233    LR chi2    1071.54      R2       0.284    C       0.833    
##  0            6490    d.f.            19    R2(19,7233)0.135    Dxy     0.666    
##  1             743    Pr(> chi2) <0.0001    R2(19,2000)0.409    gamma   0.666    
## max |deriv| 0.0002                            Brier    0.076    tau-a   0.123    
## 
##                Coef    S.E.   Wald Z Pr(>|Z|)
## Intercept      -3.6721 1.9681 -1.87  0.0621  
## Age             0.0956 0.0263  3.64  0.0003  
## Age'            0.0529 0.0769  0.69  0.4915  
## Age''          -0.3438 0.1992 -1.73  0.0844  
## BMI             0.0588 0.0466  1.26  0.2072  
## BMI'            0.1883 0.1846  1.02  0.3077  
## BMI''          -0.5119 0.4603 -1.11  0.2661  
## BPSysAve       -0.0251 0.0131 -1.92  0.0544  
## BPSysAve'       0.1395 0.0521  2.68  0.0074  
## BPSysAve''     -0.3822 0.1423 -2.69  0.0072  
## TotChol        -0.5670 0.1454 -3.90  <0.0001 
## TotChol'       -0.2385 0.5390 -0.44  0.6582  
## TotChol''       2.4817 1.7286  1.44  0.1511  
## Gender=male     0.2257 0.0908  2.48  0.0130  
## Race1=Hispanic -0.0573 0.2106 -0.27  0.7856  
## Race1=Mexican   0.0191 0.1875  0.10  0.9189  
## Race1=White    -0.7037 0.1246 -5.65  <0.0001 
## Race1=Other     0.2709 0.1871  1.45  0.1477  
## PhysActive=Yes -0.1714 0.0903 -1.90  0.0576  
## Smoke100=Yes    0.1400 0.0884  1.58  0.1133

18.4 17.4 Test overall and nonlinear effects

anova(fit_spline)
##                 Wald Statistics          Response: Diabetes_num 
## 
##  Factor          Chi-Square d.f. P     
##  Age             318.48      3   <.0001
##   Nonlinear       70.41      2   <.0001
##  BMI             212.90      3   <.0001
##   Nonlinear        2.02      2   0.3645
##  BPSysAve         13.93      3   0.0030
##   Nonlinear        7.22      2   0.0270
##  TotChol          96.56      3   <.0001
##   Nonlinear       37.37      2   <.0001
##  Gender            6.17      1   0.0130
##  Race1            70.57      4   <.0001
##  PhysActive        3.61      1   0.0576
##  Smoke100          2.51      1   0.1133
##  TOTAL NONLINEAR 117.10      8   <.0001
##  TOTAL           709.38     19   <.0001

The Nonlinear rows test whether a purely linear representation is inadequate for the corresponding continuous predictor.

18.5 17.5 Plot age effect

plot(
  Predict(
    fit_spline,
    Age,
    fun = plogis
  ),
  xlab = "Age",
  ylab = "Predicted probability"
)

18.6 17.6 Plot BMI effect

plot(
  Predict(
    fit_spline,
    BMI,
    fun = plogis
  ),
  xlab = "BMI",
  ylab = "Predicted probability"
)

18.7 17.7 Compare model AIC

AIC(fit_linear, fit_spline)
## [1] 3861.117

A lower AIC indicates a better tradeoff between likelihood fit and model complexity, but AIC should not be the only model-selection criterion.

19 18. Example Interaction Assessment

Suppose there is a clinically plausible question about whether the BMI association differs with age.

fit_interaction <- glm(
  Diabetes_num ~
    Age * BMI +
    BPSysAve +
    TotChol +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = model_complete,
  family = binomial()
)

Likelihood-ratio comparison:

anova(
  fit_full,
  fit_interaction,
  test = "LRT"
)

Interactions should ideally be prespecified or clinically motivated rather than discovered by indiscriminately testing every possible pair.

20 19. Traditional Stepwise Selection: Demonstration Only

fit_step <- step(
  fit_full,
  direction = "backward",
  trace = FALSE
)

formula(fit_step)
## Diabetes_num ~ Age + BMI + BPSysAve + TotChol + Gender + Race1 + 
##     PhysActive + Smoke100
summary(fit_step)
## 
## Call:
## glm(formula = Diabetes_num ~ Age + BMI + BPSysAve + TotChol + 
##     Gender + Race1 + PhysActive + Smoke100, family = binomial(), 
##     data = model_complete)
## 
## Coefficients:
##                Estimate Std. Error z value Pr(>|z|)    
## (Intercept)   -7.112020   0.445556 -15.962  < 2e-16 ***
## Age            0.060354   0.003116  19.369  < 2e-16 ***
## BMI            0.094418   0.006056  15.591  < 2e-16 ***
## BPSysAve       0.004711   0.002364   1.993  0.04625 *  
## TotChol       -0.287436   0.043076  -6.673 2.51e-11 ***
## Gendermale     0.323390   0.088541   3.652  0.00026 ***
## Race1Hispanic -0.114461   0.205495  -0.557  0.57753    
## Race1Mexican  -0.025664   0.181059  -0.142  0.88728    
## Race1White    -0.747630   0.123100  -6.073 1.25e-09 ***
## Race1Other     0.222607   0.183607   1.212  0.22535    
## PhysActiveYes -0.170460   0.089499  -1.905  0.05683 .  
## Smoke100Yes    0.163988   0.087292   1.879  0.06030 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 4788.6  on 7232  degrees of freedom
## Residual deviance: 3837.1  on 7221  degrees of freedom
## AIC: 3861.1
## 
## Number of Fisher Scoring iterations: 6

Stepwise selection is shown because it is common in practice, but it is generally not preferred for modern clinical prediction modeling because it can produce unstable variable selection, biased coefficients, and optimistic performance.

21 20. Penalized Regression with LASSO

21.1 20.1 Create model matrix

x <- model.matrix(
  Diabetes_num ~
    Age +
    BMI +
    BPSysAve +
    TotChol +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = model_complete
)[, -1, drop = FALSE]

y <- model_complete$Diabetes_num

21.2 20.2 Cross-validated LASSO

set.seed(2026)

cv_lasso <- cv.glmnet(
  x = x,
  y = y,
  family = "binomial",
  alpha = 1,
  nfolds = 10,
  type.measure = "deviance"
)

cv_lasso$lambda.min
## [1] 0.001002943
cv_lasso$lambda.1se
## [1] 0.0112663
plot(cv_lasso)

21.3 20.3 LASSO coefficients

coef(cv_lasso, s = "lambda.min")
## 12 x 1 sparse Matrix of class "dgCMatrix"
##                 lambda.min
## (Intercept)   -6.996878647
## Age            0.059204398
## BMI            0.092136300
## BPSysAve       0.004389612
## TotChol       -0.277855073
## Gendermale     0.294309734
## Race1Hispanic -0.036331184
## Race1Mexican   .          
## Race1White    -0.697458821
## Race1Other     0.201671224
## PhysActiveYes -0.151480083
## Smoke100Yes    0.143180777
coef(cv_lasso, s = "lambda.1se")
## 12 x 1 sparse Matrix of class "dgCMatrix"
##                 lambda.1se
## (Intercept)   -5.896861884
## Age            0.049006896
## BMI            0.072574863
## BPSysAve       0.001079893
## TotChol       -0.180075353
## Gendermale     0.034040666
## Race1Hispanic  .          
## Race1Mexican   .          
## Race1White    -0.399892727
## Race1Other     .          
## PhysActiveYes  .          
## Smoke100Yes    .

LASSO performs shrinkage and may set some coefficients exactly to zero.

22 21. Apparent Discrimination

For the remaining primary demonstration, we use the spline model.

model_complete$pred_prob <- as.numeric(
  predict(
    fit_spline,
    type = "fitted"
  )
)

22.1 21.1 ROC curve and AUC

roc_obj <- roc(
  response = model_complete$Diabetes_num,
  predictor = model_complete$pred_prob,
  quiet = TRUE
)

roc_obj
## 
## Call:
## roc.default(response = model_complete$Diabetes_num, predictor = model_complete$pred_prob,     quiet = TRUE)
## 
## Data: model_complete$pred_prob in 6490 controls (model_complete$Diabetes_num 0) < 743 cases (model_complete$Diabetes_num 1).
## Area under the curve: 0.8328
auc(roc_obj)
## Area under the curve: 0.8328
ci.auc(roc_obj)
## 95% CI: 0.8184-0.8472 (DeLong)
plot(
  roc_obj,
  print.auc = TRUE,
  main = "ROC Curve: Apparent Performance"
)

The AUC measures discrimination: how well the model ranks participants with diabetes above participants without diabetes.

23 22. Bootstrap Internal Validation

Bootstrap validation estimates model optimism by repeatedly rebuilding the model in bootstrap samples and testing performance in the original data.

set.seed(2026)

val_boot <- validate(
  fit_spline,
  method = "boot",
  B = 500
)

val_boot
##           index.orig training    test optimism index.corrected   Lower  Upper
## Dxy           0.6657   0.6710  0.6617   0.0093          0.6563  0.6263 0.6858
## R2            0.2844   0.2902  0.2800   0.0102          0.2742  0.2440 0.3029
## Intercept     0.0000   0.0000 -0.0469   0.0469         -0.0469 -0.2319 0.1421
## Slope         1.0000   1.0000  0.9730   0.0270          0.9730  0.8780 1.0676
## Emax          0.0000   0.0000  0.0228  -0.0228          0.0228 -0.0072 0.0719
## D             0.1480   0.1516  0.1456   0.0061          0.1419  0.1227 0.1601
## U            -0.0003  -0.0003  0.0001  -0.0004          0.0001 -0.0005 0.0016
## Q             0.1483   0.1519  0.1455   0.0065          0.1418  0.1215 0.1600
## B             0.0756   0.0754  0.0760  -0.0006          0.0762  0.0719 0.0807
## g             1.8068   1.8416  1.7897   0.0519          1.7549  1.6042 1.9018
## gp            0.1225   0.1239  0.1216   0.0023          0.1203  0.1108 0.1296
##             n
## Dxy       500
## R2        500
## Intercept 500
## Slope     500
## Emax      500
## D         500
## U         500
## Q         500
## B         500
## g         500
## gp        500

The most useful columns are typically:

For a final analysis, increasing B to 1000 or more is common when computationally feasible.

24 23. Bootstrap Calibration

set.seed(2026)

cal_boot <- calibrate(
  fit_spline,
  method = "boot",
  B = 500
)

cal_boot
##              predy calibrated.orig calibrated.corrected    index.orig
##  [1,] 0.0008740645     0.003503405          0.004389022  0.0026293404
##  [2,] 0.0179701314     0.017495185          0.018255665 -0.0004749465
##  [3,] 0.0350661982     0.032251078          0.033482756 -0.0028151199
##  [4,] 0.0521622651     0.048175553          0.049763262 -0.0039867124
##  [5,] 0.0692583319     0.064897931          0.066295135 -0.0043604013
##  [6,] 0.0863543988     0.082184954          0.083726698 -0.0041694451
##  [7,] 0.1034504656     0.104859703          0.106572743  0.0014092371
##  [8,] 0.1205465325     0.127396234          0.128842993  0.0068497011
##  [9,] 0.1376425994     0.144538212          0.145571183  0.0068956127
## [10,] 0.1547386662     0.162371847          0.162968787  0.0076331811
## [11,] 0.1718347331     0.182284625          0.182325150  0.0104498917
## [12,] 0.1889307999     0.200513594          0.199763619  0.0115827944
## [13,] 0.2060268668     0.215106080          0.213528258  0.0090792130
## [14,] 0.2231229337     0.229356138          0.227062738  0.0062332045
## [15,] 0.2402190005     0.244710764          0.241820400  0.0044917632
## [16,] 0.2573150674     0.259927244          0.256472316  0.0026121769
## [17,] 0.2744111342     0.274667473          0.270632194  0.0002563392
## [18,] 0.2915072011     0.289100262          0.284462728 -0.0024069389
## [19,] 0.3086032680     0.303822359          0.298570545 -0.0047809092
## [20,] 0.3256993348     0.318808065          0.312932143 -0.0068912702
## [21,] 0.3427954017     0.333831082          0.327326785 -0.0089643199
## [22,] 0.3598914685     0.349200941          0.342089215 -0.0106905274
## [23,] 0.3769875354     0.365180778          0.357500029 -0.0118067569
## [24,] 0.3940836022     0.381932592          0.373719289 -0.0121510100
## [25,] 0.4111796691     0.399121818          0.390393768 -0.0120578510
## [26,] 0.4282757360     0.416429080          0.407184202 -0.0118466558
## [27,] 0.4453718028     0.433731524          0.423953442 -0.0116402788
## [28,] 0.4624678697     0.450987442          0.440655712 -0.0114804279
## [29,] 0.4795639365     0.468186677          0.457284027 -0.0113772592
## [30,] 0.4966600034     0.485359787          0.473876984 -0.0113002160
## [31,] 0.5137560703     0.502553922          0.490493507 -0.0112021481
## [32,] 0.5308521371     0.519785139          0.507158992 -0.0110669986
## [33,] 0.5479482040     0.537051469          0.523874150 -0.0108967351
## [34,] 0.5650442708     0.554347536          0.540632497 -0.0106967353
## [35,] 0.5821403377     0.571665030          0.557423424 -0.0104753073
## [36,] 0.5992364045     0.589002215          0.574242812 -0.0102341897
## [37,] 0.6163324714     0.606363344          0.591092690 -0.0099691279
## [38,] 0.6334285383     0.623757401          0.607980773 -0.0096711372
## [39,] 0.6505246051     0.641194534          0.624915951 -0.0093300710
## [40,] 0.6676206720     0.658678355          0.641900906 -0.0089423175
## [41,] 0.6847167388     0.676206823          0.658932202 -0.0085099158
## [42,] 0.7018128057     0.693788090          0.676018492 -0.0080247159
## [43,] 0.7189088726     0.711419143          0.693155550 -0.0074897292
## [44,] 0.7360049394     0.729093823          0.710335347 -0.0069111166
## [45,] 0.7531010063     0.746809825          0.727429616 -0.0062911814
## [46,] 0.7701970731     0.764560824          0.744675831 -0.0056362488
## [47,] 0.7872931400     0.782354219          0.761926344 -0.0049389211
## [48,] 0.8043892069     0.800208786          0.779178736 -0.0041804204
## [49,] 0.8214852737     0.818083266          0.796551706 -0.0034020073
## [50,] 0.8385813406     0.835957746          0.813565873 -0.0026235943
##            training          test      optimism index.corrected        Lower
##  [1,]  0.0026853398  0.0035709566 -8.856168e-04    0.0035149572           NA
##  [2,] -0.0004664004  0.0002940796 -7.604800e-04    0.0002855335 -0.004156668
##  [3,] -0.0028072088 -0.0015755312 -1.231678e-03   -0.0015834422 -0.008012369
##  [4,] -0.0041038711 -0.0025161616 -1.587710e-03   -0.0023990028 -0.012153892
##  [5,] -0.0038223561 -0.0024251514 -1.397205e-03   -0.0029631966 -0.014963211
##  [6,] -0.0028562934 -0.0013145487 -1.541745e-03   -0.0026277004 -0.016109663
##  [7,]  0.0009748609  0.0026879017 -1.713041e-03    0.0031222778 -0.011944247
##  [8,]  0.0042697897  0.0057165494 -1.446760e-03    0.0082964608 -0.008051966
##  [9,]  0.0058705209  0.0069034921 -1.032971e-03    0.0079285839 -0.009638672
## [10,]  0.0071243964  0.0077213357 -5.969393e-04    0.0082301204 -0.009837520
## [11,]  0.0085085699  0.0085490953 -4.052538e-05    0.0104904171 -0.008147028
## [12,]  0.0091916227  0.0084416471  7.499756e-04    0.0108328188 -0.009076676
## [13,]  0.0085028587  0.0069250369  1.577822e-03    0.0075013911 -0.012884590
## [14,]  0.0069801225  0.0046867226  2.293400e-03    0.0039398046 -0.017042771
## [15,]  0.0052219696  0.0023316056  2.890364e-03    0.0016013992 -0.021124085
## [16,]  0.0032959926 -0.0001589358  3.454928e-03   -0.0008427515 -0.025027794
## [17,]  0.0010876792 -0.0029476005  4.035280e-03   -0.0037789405 -0.029352329
## [18,] -0.0012814332 -0.0059189674  4.637534e-03   -0.0070444731 -0.034096437
## [19,] -0.0035252927 -0.0087771060  5.251813e-03   -0.0100327225 -0.038926153
## [20,] -0.0054706815 -0.0113466031  5.875922e-03   -0.0127671917 -0.043068976
## [21,] -0.0071110099 -0.0136153068  6.504297e-03   -0.0154686169 -0.047177229
## [22,] -0.0084398849 -0.0155516111  7.111726e-03   -0.0178022536 -0.051382974
## [23,] -0.0093992164 -0.0170799662  7.680750e-03   -0.0194875068 -0.055124509
## [24,] -0.0099676491 -0.0181809521  8.213303e-03   -0.0203643130 -0.058224796
## [25,] -0.0101875436 -0.0189155933  8.728050e-03   -0.0207859007 -0.060047278
## [26,] -0.0101744189 -0.0194192974  9.244878e-03   -0.0210915343 -0.061759675
## [27,] -0.0100675588 -0.0198456409  9.778082e-03   -0.0214183609 -0.063952575
## [28,] -0.0099585910 -0.0202903211  1.033173e-02   -0.0218121580 -0.066118565
## [29,] -0.0098762804 -0.0207789312  1.090265e-02   -0.0222799100 -0.068657443
## [30,] -0.0098047423 -0.0212875452  1.148280e-02   -0.0227830190 -0.071279835
## [31,] -0.0097174984 -0.0217779133  1.206041e-02   -0.0232625631 -0.073494003
## [32,] -0.0096048458 -0.0222309920  1.262615e-02   -0.0236931448 -0.076105162
## [33,] -0.0094691211 -0.0226464405  1.317732e-02   -0.0240740545 -0.078937714
## [34,] -0.0093146276 -0.0230296665  1.371504e-02   -0.0244117741 -0.081691777
## [35,] -0.0091437444 -0.0233853508  1.424161e-02   -0.0247169137 -0.084713775
## [36,] -0.0089565557 -0.0237159590  1.475940e-02   -0.0249935930 -0.087304054
## [37,] -0.0087502762 -0.0240209292  1.527065e-02   -0.0252397809 -0.090165224
## [38,] -0.0085208697 -0.0242974973  1.577663e-02   -0.0254477648 -0.092789722
## [39,] -0.0082635681 -0.0245421514  1.627858e-02   -0.0256086543 -0.095548761
## [40,] -0.0079740395 -0.0247514883  1.677745e-02   -0.0257197662 -0.098454911
## [41,] -0.0076472863 -0.0249219075  1.727462e-02   -0.0257845370 -0.101526595
## [42,] -0.0072823266 -0.0250519242  1.776960e-02   -0.0257943134 -0.104305237
## [43,] -0.0068761872 -0.0251397805  1.826359e-02   -0.0257533224 -0.107240929
## [44,] -0.0064274853 -0.0251859613  1.875848e-02   -0.0256695926 -0.110526259
## [45,] -0.0058751741 -0.0252553829  1.938021e-02   -0.0256713902           NA
## [46,] -0.0053319534 -0.0252169464  1.988499e-02   -0.0255212418           NA
## [47,] -0.0047632406 -0.0251911153  2.042787e-02   -0.0253667958           NA
## [48,] -0.0042054104 -0.0252354605  2.103005e-02   -0.0252104706           NA
## [49,] -0.0036552396 -0.0251868004  2.153156e-02   -0.0249335681           NA
## [50,] -0.0030717871 -0.0254636608  2.239187e-02   -0.0250154679           NA
##             Upper   n
##  [1,]          NA 312
##  [2,] 0.005087578 500
##  [3,] 0.006106309 500
##  [4,] 0.008109391 500
##  [5,] 0.009117669 500
##  [6,] 0.011537493 500
##  [7,] 0.018410024 500
##  [8,] 0.024888536 500
##  [9,] 0.026059330 500
## [10,] 0.028080623 500
## [11,] 0.031439066 500
## [12,] 0.032053113 500
## [13,] 0.029898070 500
## [14,] 0.027769281 500
## [15,] 0.025934529 500
## [16,] 0.024736779 500
## [17,] 0.023399123 500
## [18,] 0.021991199 500
## [19,] 0.020721715 500
## [20,] 0.020283906 500
## [21,] 0.019944784 500
## [22,] 0.019427573 500
## [23,] 0.019400622 500
## [24,] 0.020058472 500
## [25,] 0.022033922 500
## [26,] 0.024056934 500
## [27,] 0.025548586 500
## [28,] 0.027293459 500
## [29,] 0.028904629 500
## [30,] 0.030694038 500
## [31,] 0.033153496 500
## [32,] 0.035255094 500
## [33,] 0.037182465 500
## [34,] 0.039301275 500
## [35,] 0.041210235 500
## [36,] 0.043723834 500
## [37,] 0.046029474 500
## [38,] 0.048711201 500
## [39,] 0.051334215 500
## [40,] 0.053886442 500
## [41,] 0.056344331 500
## [42,] 0.059220330 500
## [43,] 0.061999723 500
## [44,] 0.064477397 500
## [45,]          NA 499
## [46,]          NA 499
## [47,]          NA 498
## [48,]          NA 496
## [49,]          NA 494
## [50,]          NA 491
## attr(,"class")
## [1] "calibrate.default"
## attr(,"call")
## calibrate.default(fit = fit_spline, method = "boot", B = 500)
## attr(,"kint")
## [1] 1
## attr(,"model")
## [1] "lr"
## attr(,"lev.name")
## [1] "1"
## attr(,"yvar.name")
## [1] "Diabetes_num"
## attr(,"n")
## [1] 7233
## attr(,"freq")
##    0    1 
## 6490  743 
## attr(,"non.slopes")
## [1] 1
## attr(,"B")
## [1] 500
## attr(,"method")
## [1] "boot"
## attr(,"predicted")
##    [1] 0.0441182752 0.0441182752 0.0441182752 0.0491437658 0.0163394264
##    [6] 0.0163394264 0.0163394264 0.0762464798 0.0881837421 0.0581427919
##   [11] 0.1689632888 0.0528147513 0.0116382075 0.1096401680 0.0300342754
##   [16] 0.0300342754 0.0443655164 0.3361993836 0.3361993836 0.0623880305
##   [21] 0.0081312791 0.0580411428 0.0580411428 0.3273302720 0.0012946986
##   [26] 0.3199213677 0.3199213677 0.1236380069 0.1412074130 0.1412074130
##   [31] 0.0102144959 0.0165710671 0.1209914410 0.0094000406 0.0032348333
##   [36] 0.0237414481 0.0237414481 0.0420925039 0.2261539777 0.2518390392
##   [41] 0.2651252431 0.2651252431 0.2651252431 0.0116258166 0.0220537528
##   [46] 0.0145663561 0.0069237961 0.0069237961 0.0069237961 0.0761434102
##   [51] 0.0761434102 0.0453561636 0.0052290416 0.0052290416 0.0052290416
##   [56] 0.0373427632 0.0303619507 0.0303619507 0.0303619507 0.0078656283
##   [61] 0.0078656283 0.0078656283 0.0078656283 0.1204532372 0.1204532372
##   [66] 0.1508447593 0.0062859215 0.2394853107 0.1019890488 0.0086563122
##   [71] 0.1985521496 0.2223965473 0.4416158136 0.0127289993 0.0127289993
##   [76] 0.0127289993 0.0458980514 0.0458980514 0.0458980514 0.2605037986
##   [81] 0.0750070371 0.0750070371 0.0959437110 0.0359848106 0.2106029499
##   [86] 0.0873609396 0.0873609396 0.0873609396 0.0873609396 0.2670399649
##   [91] 0.2686794156 0.0114282961 0.3886882493 0.1141716313 0.2439924856
##   [96] 0.6221556793 0.0437853010 0.0049180182 0.0049180182 0.0049180182
##  [101] 0.2344008275 0.2344008275 0.1632264865 0.0170007883 0.0282941688
##  [106] 0.0282941688 0.0147171268 0.1719664210 0.1371654296 0.1982187580
##  [111] 0.0296923912 0.0296923912 0.0284087362 0.0284087362 0.1655477583
##  [116] 0.2053318454 0.2053318454 0.0278375236 0.0101857976 0.0101857976
##  [121] 0.0101857976 0.0101857976 0.0237403313 0.0146445340 0.0291908172
##  [126] 0.0291908172 0.1156685051 0.0038524490 0.0339374137 0.0518878557
##  [131] 0.0518878557 0.1397548201 0.1265352532 0.2044654201 0.2044654201
##  [136] 0.0751491152 0.0751491152 0.0029735577 0.1032428542 0.1032428542
##  [141] 0.0121127883 0.0129081936 0.0129081936 0.0954297811 0.0954297811
##  [146] 0.0212567719 0.0212567719 0.0620639882 0.0620639882 0.0299055419
##  [151] 0.0299055419 0.0140338087 0.0227707393 0.2254748209 0.2254748209
##  [156] 0.1259682622 0.2097270692 0.0906997309 0.0041974405 0.0646391739
##  [161] 0.7749016379 0.0243117718 0.2295473072 0.1066526496 0.1066526496
##  [166] 0.0257042564 0.0028875202 0.0028875202 0.1743795202 0.1743795202
##  [171] 0.0070065504 0.0070065504 0.0570139703 0.1133573488 0.1652417600
##  [176] 0.1652417600 0.5193044954 0.1634717559 0.1634717559 0.2819429124
##  [181] 0.0266697361 0.2443684773 0.2443684773 0.0098713676 0.0098713676
##  [186] 0.0098713676 0.0634623261 0.0634623261 0.0097201466 0.0097201466
##  [191] 0.2715175163 0.0117339429 0.7071651212 0.7071651212 0.7071651212
##  [196] 0.2052151753 0.0080541384 0.0421866071 0.0421866071 0.1407308366
##  [201] 0.0493745569 0.1033923710 0.3597848480 0.0929046912 0.0929046912
##  [206] 0.0220382423 0.4595480165 0.1721930855 0.0277959237 0.0277959237
##  [211] 0.0566816960 0.0430700988 0.4364552602 0.1859282545 0.1465401788
##  [216] 0.1465401788 0.0650532549 0.0016481463 0.0121474767 0.0121474767
##  [221] 0.0521703868 0.2661203524 0.2661203524 0.2661203524 0.0481820060
##  [226] 0.0481820060 0.0481820060 0.0314527216 0.0314527216 0.0070094538
##  [231] 0.1671590400 0.0920924188 0.3478348335 0.3478348335 0.3478348335
##  [236] 0.3478348335 0.0215175653 0.1492224323 0.1492224323 0.0896636146
##  [241] 0.0495453779 0.0495453779 0.2559938483 0.4479687817 0.0922976087
##  [246] 0.0140976563 0.0140976563 0.0896374717 0.0896374717 0.0039061477
##  [251] 0.0037265720 0.0893127070 0.0046092219 0.0046092219 0.0313013473
##  [256] 0.0051224409 0.0481577684 0.0611623169 0.0611623169 0.0225749486
##  [261] 0.0939783540 0.0035731769 0.0432750634 0.0432750634 0.0432750634
##  [266] 0.0127430103 0.0718663791 0.0025881583 0.9381325781 0.9381325781
##  [271] 0.0092286366 0.0473694595 0.0895678254 0.0354532635 0.3480558763
##  [276] 0.1152382320 0.1207865561 0.0066042225 0.6273953995 0.0498626546
##  [281] 0.0498626546 0.0024566434 0.1261584307 0.0706779905 0.0173745582
##  [286] 0.1410215751 0.0072543494 0.0209611454 0.0676974581 0.0676974581
##  [291] 0.1386838660 0.3487463504 0.0476430661 0.0366490162 0.0642404078
##  [296] 0.1048342911 0.1629570325 0.1629570325 0.1629570325 0.0116647892
##  [301] 0.5270196377 0.0051687162 0.0342335884 0.0342335884 0.0291315234
##  [306] 0.1250632330 0.2751315309 0.2751315309 0.3197403531 0.0281001181
##  [311] 0.0281001181 0.0281001181 0.2611734080 0.0098017587 0.0098017587
##  [316] 0.1569575487 0.1569575487 0.1569575487 0.0059874103 0.2420489051
##  [321] 0.2420489051 0.3136818026 0.1676587184 0.1676587184 0.0480356794
##  [326] 0.0480356794 0.0480356794 0.0037312728 0.0037312728 0.0044666087
##  [331] 0.0802899607 0.1365926050 0.1365926050 0.1365926050 0.0471858984
##  [336] 0.0471858984 0.0118434498 0.0979179937 0.0979179937 0.0979179937
##  [341] 0.0979179937 0.1610710990 0.0386160898 0.0392160144 0.0402809874
##  [346] 0.0402809874 0.0127208322 0.0127208322 0.1577621756 0.1577621756
##  [351] 0.0241417987 0.2066977717 0.0663537438 0.0663537438 0.1655690142
##  [356] 0.0132581187 0.0683342427 0.0683342427 0.0683342427 0.0083897086
##  [361] 0.0183490948 0.0163557219 0.0055337197 0.0093299443 0.0222225007
##  [366] 0.0222225007 0.4373367550 0.1270477795 0.1270477795 0.0663901508
##  [371] 0.0104112931 0.0104112931 0.0065028553 0.0065028553 0.0776947468
##  [376] 0.0776947468 0.1572474851 0.1320563423 0.0090505665 0.2534132943
##  [381] 0.4100147141 0.0678225519 0.0678225519 0.0045936744 0.0045936744
##  [386] 0.0404666691 0.0404666691 0.5009589388 0.4342723398 0.0229773166
##  [391] 0.0346799990 0.0346799990 0.0346799990 0.0346799990 0.2643931283
##  [396] 0.0866144780 0.0511725932 0.1275853856 0.2334846976 0.5327765747
##  [401] 0.0064224187 0.0762473389 0.4449007627 0.0292800809 0.2620832301
##  [406] 0.0688413506 0.0044944077 0.1227460970 0.0206990629 0.3011519713
##  [411] 0.0122503564 0.0122503564 0.2386779630 0.0151335040 0.2880107010
##  [416] 0.2880107010 0.3186079812 0.1933239694 0.1933239694 0.2375378416
##  [421] 0.4110146445 0.1028402931 0.0154302319 0.0292069922 0.0164171528
##  [426] 0.0164171528 0.0221129370 0.0111514764 0.1489719456 0.0600022557
##  [431] 0.0408448496 0.0408448496 0.0738054443 0.0435596225 0.0284664238
##  [436] 0.1975808132 0.1975808132 0.1975808132 0.2201323691 0.2201323691
##  [441] 0.2201323691 0.0297304091 0.0297304091 0.0121869476 0.1920687924
##  [446] 0.0116421618 0.0028669260 0.1901509040 0.1273364421 0.1273364421
##  [451] 0.0175474855 0.0048291293 0.0077719238 0.0254169146 0.0789038230
##  [456] 0.0789038230 0.0789038230 0.1803873182 0.1803873182 0.1911349199
##  [461] 0.0038287407 0.0038287407 0.1396386054 0.1396386054 0.5564522380
##  [466] 0.1715518343 0.1715518343 0.1093205910 0.1322632563 0.0834424475
##  [471] 0.0066964601 0.0478941716 0.0930201892 0.1218570446 0.1218570446
##  [476] 0.3472309546 0.0081595695 0.1188958899 0.3161153865 0.0204277652
##  [481] 0.0204277652 0.0261143824 0.0261143824 0.3633273411 0.1155027174
##  [486] 0.1155027174 0.0060995195 0.3864233527 0.0282290599 0.0282290599
##  [491] 0.0320006064 0.0142624706 0.0142624706 0.0142624706 0.0015962161
##  [496] 0.0141130967 0.0751589023 0.0751589023 0.0751589023 0.1545424508
##  [501] 0.0627197950 0.0627197950 0.0094720202 0.0094720202 0.0094720202
##  [506] 0.0094720202 0.1414279290 0.0234344845 0.0273925253 0.0398019105
##  [511] 0.0030032756 0.0117379210 0.0117379210 0.2839615905 0.0607433174
##  [516] 0.0510520447 0.0510520447 0.0249943515 0.0249943515 0.0211322789
##  [521] 0.0712727693 0.1126850390 0.0228371908 0.0228371908 0.1250587688
##  [526] 0.1250587688 0.1142668203 0.0752973367 0.1144609829 0.0054326366
##  [531] 0.1833636369 0.0115744294 0.5014830582 0.0801059776 0.0801059776
##  [536] 0.0060538961 0.0780514494 0.3765510117 0.1903865346 0.0437358156
##  [541] 0.0437358156 0.0437358156 0.0437358156 0.2973334892 0.3436662210
##  [546] 0.1414708360 0.0024633090 0.0024633090 0.0504551657 0.0050777564
##  [551] 0.4752854238 0.0914184877 0.0040371740 0.0040371740 0.0549792289
##  [556] 0.0098358096 0.0253466178 0.0053632134 0.0053632134 0.0612565183
##  [561] 0.0012780064 0.0012780064 0.0046790806 0.1793171362 0.1766479348
##  [566] 0.5203874620 0.2597046953 0.0121005148 0.0817106075 0.0172000767
##  [571] 0.0753565609 0.1001108082 0.0051163359 0.4209632085 0.3947422421
##  [576] 0.4487515273 0.4536685390 0.2172612839 0.1058365966 0.0207429884
##  [581] 0.0207429884 0.3951355624 0.0054288840 0.2592289966 0.2592289966
##  [586] 0.0611569501 0.1442072180 0.0415866978 0.0415866978 0.0097775803
##  [591] 0.1388132596 0.0517104020 0.0517104020 0.5544809784 0.5544809784
##  [596] 0.2559922388 0.0116491615 0.0043799970 0.0101052742 0.0112517983
##  [601] 0.0649396793 0.2831124623 0.0111600045 0.3066993059 0.3066993059
##  [606] 0.3066993059 0.3066993059 0.3266949476 0.0390063213 0.0076650947
##  [611] 0.0076650947 0.0076650947 0.0352438332 0.3518866430 0.0132736976
##  [616] 0.0132736976 0.0199506224 0.0199506224 0.0022415530 0.3931492935
##  [621] 0.3931492935 0.3931492935 0.0290295383 0.0290295383 0.0173443593
##  [626] 0.0090556653 0.0095295441 0.0117052325 0.0117052325 0.0117052325
##  [631] 0.0117052325 0.0117052325 0.1866184292 0.0101232526 0.0140461114
##  [636] 0.0639761649 0.1345038923 0.2348888243 0.0023768230 0.0023768230
##  [641] 0.0392579925 0.0120850584 0.0300456538 0.0300456538 0.0015295384
##  [646] 0.0050328385 0.1113122391 0.0151682175 0.0151682175 0.0518010623
##  [651] 0.0518010623 0.0518010623 0.0861938195 0.0994368506 0.0833702059
##  [656] 0.0833702059 0.1641955327 0.0146191448 0.0146191448 0.0146191448
##  [661] 0.1211077289 0.2502260937 0.0559203292 0.2446864764 0.0143278683
##  [666] 0.0143278683 0.0066654168 0.0066654168 0.0974380678 0.0166946369
##  [671] 0.0042645212 0.0511800402 0.0119424038 0.0119424038 0.1465013220
##  [676] 0.1465013220 0.0119474836 0.0316907372 0.1186148913 0.1186148913
##  [681] 0.1186148913 0.1186148913 0.0364659958 0.0068930675 0.3453046620
##  [686] 0.0078619845 0.0078619845 0.0027089843 0.0561924359 0.0561924359
##  [691] 0.0034972788 0.0440368616 0.0440368616 0.0440368616 0.0440368616
##  [696] 0.3131830745 0.0047029511 0.0146056029 0.1266812925 0.1266812925
##  [701] 0.0474936226 0.0474936226 0.0474936226 0.1339764737 0.0692651931
##  [706] 0.0692651931 0.0595064284 0.0995591129 0.0995591129 0.0950512066
##  [711] 0.0158728189 0.0158728189 0.4366577159 0.0050885850 0.0794846350
##  [716] 0.1608433490 0.0169383355 0.1789930395 0.0119402930 0.0841916625
##  [721] 0.0841916625 0.0408605945 0.0408605945 0.0408605945 0.0185480587
##  [726] 0.0817376264 0.0049435531 0.0322591613 0.0598234350 0.1760570364
##  [731] 0.1760570364 0.0837909210 0.3066506023 0.0113895113 0.0288613395
##  [736] 0.0039501383 0.0559662710 0.1947190447 0.1947190447 0.0066407911
##  [741] 0.0677615012 0.0677615012 0.0677615012 0.0677615012 0.0677615012
##  [746] 0.0758484786 0.0758484786 0.0758484786 0.0719785489 0.0719785489
##  [751] 0.0345780732 0.0102037507 0.0128311023 0.0236624927 0.0127635088
##  [756] 0.0127635088 0.0127635088 0.0127635088 0.0034373201 0.7422457440
##  [761] 0.1035096408 0.0236495458 0.5504995310 0.0106552200 0.0117464412
##  [766] 0.0237524480 0.0966140447 0.0966140447 0.0125106767 0.0125106767
##  [771] 0.0063009065 0.0133943151 0.0145891122 0.4876487212 0.1800625573
##  [776] 0.0958419277 0.0491869252 0.1963497671 0.4028502585 0.3307194742
##  [781] 0.1499046659 0.0169029452 0.0169029452 0.0723708418 0.0963556300
##  [786] 0.0963556300 0.0205154534 0.0205154534 0.0131245989 0.3137736891
##  [791] 0.0152562962 0.0152562962 0.0472483386 0.0179504844 0.0179504844
##  [796] 0.0835105112 0.9323288478 0.1796770488 0.0577749396 0.0460132639
##  [801] 0.1043490150 0.1043490150 0.0502688247 0.0502688247 0.1970730479
##  [806] 0.0416368132 0.0416368132 0.0039136586 0.0078487668 0.0078487668
##  [811] 0.0165666323 0.0165666323 0.0590780094 0.2655120140 0.0063620673
##  [816] 0.0063620673 0.0921895588 0.0039158220 0.0059109339 0.0039158220
##  [821] 0.0703375135 0.0951732772 0.0178438529 0.0132604980 0.0032009392
##  [826] 0.0050252345 0.0096408550 0.1187338496 0.0290140502 0.0069533811
##  [831] 0.0204050042 0.1738272735 0.1738272735 0.1738272735 0.0086463642
##  [836] 0.0086463642 0.0408913685 0.0134089517 0.0134089517 0.1817155572
##  [841] 0.2236155659 0.0158925348 0.0317643218 0.0317643218 0.0144676678
##  [846] 0.0218040950 0.0218040950 0.0218040950 0.0532053565 0.0532053565
##  [851] 0.1449352205 0.0507439128 0.0016612547 0.0016612547 0.0016612547
##  [856] 0.5648825884 0.0035803879 0.4690367407 0.0942797866 0.0016786526
##  [861] 0.0016786526 0.0659725920 0.0659725920 0.0447070884 0.0447070884
##  [866] 0.0408349345 0.1977480140 0.0868883874 0.0313897193 0.0108171707
##  [871] 0.0153417383 0.1584683276 0.0172159719 0.0554241723 0.0135791322
##  [876] 0.0182537161 0.0191574029 0.0313403310 0.0689293730 0.0054907610
##  [881] 0.3360347985 0.0113101192 0.0326618218 0.0326618218 0.0835441723
##  [886] 0.2584173085 0.0920425119 0.1963509040 0.3997499030 0.3997499030
##  [891] 0.1209131086 0.0275860329 0.0101798710 0.0453169140 0.0120582635
##  [896] 0.0028910504 0.0028910504 0.0155993036 0.0129636079 0.0966663994
##  [901] 0.0966663994 0.0966663994 0.2391356755 0.0976364443 0.3626238952
##  [906] 0.1353397009 0.2846813368 0.2846813368 0.0203188862 0.5162120065
##  [911] 0.0093059873 0.0093059873 0.0148378873 0.1072517125 0.1072517125
##  [916] 0.0530142650 0.0451365363 0.0152880394 0.0317301462 0.1154721745
##  [921] 0.1154721745 0.1154721745 0.1154721745 0.1154721745 0.0676567949
##  [926] 0.0676567949 0.0676567949 0.0676567949 0.0692414523 0.0692414523
##  [931] 0.0096696013 0.0962693291 0.1207396624 0.0444210382 0.0850734624
##  [936] 0.0850734624 0.0620672225 0.0620672225 0.3205811300 0.3205811300
##  [941] 0.3205811300 0.0034619932 0.0259252046 0.0244679522 0.0259252046
##  [946] 0.0957793133 0.1088050900 0.2365200061 0.2365200061 0.0827477732
##  [951] 0.0027868360 0.0822074872 0.1228562010 0.3416681082 0.0623870193
##  [956] 0.0623870193 0.0784703366 0.0862447238 0.0098902430 0.0029406399
##  [961] 0.0541253073 0.0541253073 0.0541253073 0.0245132671 0.1010330609
##  [966] 0.0052466925 0.3843609592 0.3843609592 0.0139651388 0.0030122843
##  [971] 0.4003126568 0.0160983148 0.0160983148 0.0160983148 0.1610224548
##  [976] 0.0028871391 0.0270065720 0.0270065720 0.0270065720 0.0270065720
##  [981] 0.0975285166 0.0844522829 0.0015354832 0.0015354832 0.0015354832
##  [986] 0.4699849864 0.0651689168 0.0837920777 0.3216691695 0.0089233712
##  [991] 0.0300425668 0.0300425668 0.0300425668 0.0788470480 0.0180812630
##  [996] 0.2843099632 0.0236827817 0.0236827817 0.0236827817 0.0386329416
## [1001] 0.7638946518 0.1787503484 0.1787503484 0.1787503484 0.0502966957
## [1006] 0.0047520179 0.0047520179 0.0047520179 0.0066948394 0.4265655860
## [1011] 0.0259008338 0.2789670914 0.0259328943 0.0365223100 0.0476461926
## [1016] 0.0476461926 0.0743733770 0.0743733770 0.0670973218 0.0670973218
## [1021] 0.0043841900 0.0043841900 0.6760432471 0.0078306856 0.1005029751
## [1026] 0.0173530850 0.0322162360 0.4551595049 0.4551595049 0.1289872058
## [1031] 0.1289872058 0.0057744590 0.0057744590 0.0057744590 0.0057886684
## [1036] 0.0897219504 0.0897219504 0.4295348994 0.4295348994 0.0044916761
## [1041] 0.0152166961 0.0783488417 0.0783488417 0.1071529646 0.1789458675
## [1046] 0.4003894822 0.0560399417 0.0041175213 0.0760484605 0.0760484605
## [1051] 0.1865578405 0.1445339986 0.0173475522 0.0064385451 0.0064385451
## [1056] 0.0064385451 0.0117514053 0.1512648650 0.0272756201 0.0061096066
## [1061] 0.0040458995 0.0283256234 0.0177755126 0.0177755126 0.0299212556
## [1066] 0.5809697490 0.0043479536 0.0580974899 0.2303470429 0.1564206178
## [1071] 0.0121644533 0.0786338787 0.0044616112 0.0050449129 0.0089787249
## [1076] 0.0310717261 0.0273274194 0.0273274194 0.0335201808 0.1503996863
## [1081] 0.1503996863 0.0124394057 0.0442564716 0.0045632336 0.0045632336
## [1086] 0.0149334255 0.0548603726 0.0548603726 0.0267878900 0.0267878900
## [1091] 0.0258485287 0.0930165930 0.0403414153 0.0016451726 0.1026868829
## [1096] 0.1026868829 0.2715976254 0.0633697049 0.2716745687 0.2460146950
## [1101] 0.0253406939 0.0253406939 0.0533129343 0.5903570393 0.1406536627
## [1106] 0.0351274384 0.0755651700 0.3282981804 0.0215864062 0.0174561833
## [1111] 0.0174561833 0.5355049454 0.0417929733 0.2355399095 0.0178089383
## [1116] 0.0178089383 0.0178089383 0.0178089383 0.4123519906 0.4123519906
## [1121] 0.2129661154 0.2341715125 0.0108085606 0.0981346274 0.1721880046
## [1126] 0.1721880046 0.1680166812 0.0698291378 0.0698291378 0.0638873135
## [1131] 0.0140350863 0.0140350863 0.0024126521 0.0039261437 0.0039261437
## [1136] 0.5358291989 0.2217194695 0.2217194695 0.0199503619 0.2263621630
## [1141] 0.0192799432 0.1068010791 0.1980461296 0.1068010791 0.0214194293
## [1146] 0.1055001381 0.5610228842 0.1033809288 0.3575075582 0.3575075582
## [1151] 0.3575075582 0.0496316073 0.2023769313 0.0236464403 0.0027712095
## [1156] 0.0027712095 0.5358627714 0.1538061826 0.1538061826 0.0022244245
## [1161] 0.0051823865 0.0451270859 0.0496922887 0.0201866845 0.6108383626
## [1166] 0.7051836778 0.0848655835 0.0848655835 0.0848655835 0.0052145910
## [1171] 0.0770473432 0.0770473432 0.0770473432 0.0028957788 0.0223209299
## [1176] 0.5002957746 0.0994271454 0.0152309370 0.0168271660 0.0168271660
## [1181] 0.1144174932 0.1144174932 0.1144174932 0.0054156292 0.0329800861
## [1186] 0.0022570472 0.0022570472 0.1525623097 0.0541293525 0.0023867871
## [1191] 0.0023867871 0.0023867871 0.0023867871 0.1109619157 0.1109619157
## [1196] 0.5581625523 0.0676133409 0.0676133409 0.0676133409 0.1851913739
## [1201] 0.0690145504 0.0125011517 0.0066617165 0.0066617165 0.1085055319
## [1206] 0.1085055319 0.1085055319 0.0035127251 0.2628173811 0.0152621500
## [1211] 0.0152621500 0.0152621500 0.2802496778 0.0337950473 0.0078067241
## [1216] 0.0078067241 0.2728938031 0.0062639077 0.3481821148 0.0123266008
## [1221] 0.0123266008 0.0031496592 0.0061330276 0.0061330276 0.4362836811
## [1226] 0.4362836811 0.0059361748 0.1832090244 0.0059761170 0.0059761170
## [1231] 0.0079164832 0.0014290700 0.0014290700 0.1048233154 0.0409892332
## [1236] 0.0543144567 0.1006474300 0.1006474300 0.0204687155 0.0245313117
## [1241] 0.1027781368 0.1027781368 0.0089267497 0.2559801409 0.0020563814
## [1246] 0.0020563814 0.0145537432 0.4321299724 0.0168059441 0.0168059441
## [1251] 0.0154019284 0.0154019284 0.0154019284 0.0154019284 0.0066482763
## [1256] 0.0249841631 0.0249841631 0.0027305505 0.0027305505 0.0027305505
## [1261] 0.0027305505 0.0762905032 0.0323900676 0.0323900676 0.0453379784
## [1266] 0.5266667942 0.0231508492 0.0231508492 0.0709175997 0.0709175997
## [1271] 0.0044369375 0.1384091036 0.0118858452 0.0118858452 0.1610924192
## [1276] 0.3232013125 0.0101163890 0.0101163890 0.0058932395 0.0058932395
## [1281] 0.0163379506 0.0163379506 0.0163379506 0.2886965228 0.2886965228
## [1286] 0.2886965228 0.0441402458 0.0351895136 0.4377327818 0.0367282424
## [1291] 0.7420397593 0.0544577561 0.2014923112 0.1373516806 0.1942728543
## [1296] 0.1064844254 0.4869048725 0.0508806113 0.0508806113 0.0823463181
## [1301] 0.3880439428 0.0660251124 0.0660251124 0.0966028428 0.0966028428
## [1306] 0.1195506947 0.1195506947 0.1704724610 0.1704724610 0.1704724610
## [1311] 0.0147798820 0.0274612677 0.0274612677 0.0274612677 0.1264828265
## [1316] 0.1264828265 0.1264828265 0.0366823742 0.0665247288 0.0665247288
## [1321] 0.0206184487 0.3537385437 0.0764911876 0.0283355051 0.0283355051
## [1326] 0.0283355051 0.0031804053 0.0128974981 0.0461469288 0.0461469288
## [1331] 0.0174709287 0.0174709287 0.2114403498 0.0294251196 0.0294251196
## [1336] 0.4899530414 0.4899530414 0.3789364527 0.0040883421 0.0262650215
## [1341] 0.0226889378 0.3655523012 0.0847844996 0.0343847468 0.2274009734
## [1346] 0.0094760831 0.0043119001 0.0152191464 0.0152191464 0.0191007866
## [1351] 0.0028309793 0.0028309793 0.3584736396 0.0495831787 0.1436513716
## [1356] 0.1996832076 0.2048948993 0.0109158465 0.0082399284 0.0079678021
## [1361] 0.1597144110 0.0364275701 0.4723986927 0.0889505095 0.0889505095
## [1366] 0.0209456633 0.0237087567 0.0735234690 0.0885085112 0.0131067182
## [1371] 0.0031419242 0.0431743112 0.0070188687 0.0070188687 0.0070188687
## [1376] 0.0070188687 0.0609550368 0.0069106741 0.0142740180 0.0332439181
## [1381] 0.0118739276 0.0239748255 0.0239748255 0.7302332219 0.0065344328
## [1386] 0.0065344328 0.0065344328 0.0476379590 0.0476379590 0.0476379590
## [1391] 0.0055467856 0.0055467856 0.0055467856 0.1021003521 0.0092166066
## [1396] 0.0137409335 0.0135930200 0.0129100103 0.0200516140 0.0417948723
## [1401] 0.0417948723 0.0756135120 0.0337483851 0.0337483851 0.0093615773
## [1406] 0.1472779675 0.1314546689 0.0434905040 0.0101087602 0.1986757021
## [1411] 0.1986757021 0.1478728065 0.0150268477 0.0071558623 0.0071558623
## [1416] 0.0071558623 0.0160885424 0.0032703871 0.0032703871 0.1720359580
## [1421] 0.1369781451 0.1369781451 0.3713210227 0.1699793692 0.5514449507
## [1426] 0.6815010141 0.2101949704 0.1798891229 0.0087807137 0.0087807137
## [1431] 0.0087807137 0.0087807137 0.1853479722 0.0758610716 0.1739832564
## [1436] 0.1739832564 0.1739832564 0.2417431435 0.0248645322 0.0248645322
## [1441] 0.1263571052 0.0277307655 0.0079590206 0.0079590206 0.0079590206
## [1446] 0.0600392895 0.0058491636 0.0058491636 0.0058491636 0.5041687510
## [1451] 0.1803398586 0.0105476472 0.0105476472 0.0105476472 0.1835987409
## [1456] 0.0373544251 0.0063501577 0.0063501577 0.0063501577 0.0063501577
## [1461] 0.0063501577 0.0033328285 0.0113381988 0.0113381988 0.2840665506
## [1466] 0.2840665506 0.0651768475 0.1589488268 0.0204130912 0.0204130912
## [1471] 0.0204130912 0.0594396818 0.0046752512 0.0955462450 0.0128645596
## [1476] 0.0188505854 0.2093131745 0.2093131745 0.0200868810 0.1104147766
## [1481] 0.1104147766 0.1104147766 0.0027658895 0.4207149977 0.0024683609
## [1486] 0.2124404462 0.0048391291 0.1336035464 0.1867318007 0.1867318007
## [1491] 0.0018537505 0.0018537505 0.0018537505 0.0647584140 0.0647584140
## [1496] 0.1193600913 0.1896575919 0.0605416936 0.1539494840 0.0158453992
## [1501] 0.0158453992 0.1029540692 0.1351257310 0.1351257310 0.0181747053
## [1506] 0.3422237316 0.2115467856 0.0373392368 0.0373392368 0.0136966346
## [1511] 0.1268652413 0.0317187163 0.1515546401 0.1176066886 0.1176066886
## [1516] 0.1206765129 0.0270754118 0.0270754118 0.0270754118 0.0024564053
## [1521] 0.0024564053 0.0024564053 0.0024564053 0.0887265041 0.1445204721
## [1526] 0.0035731027 0.0035731027 0.0959710524 0.0058393815 0.0513739095
## [1531] 0.0513739095 0.0215101520 0.0296762920 0.0296762920 0.1555892508
## [1536] 0.0044639112 0.0044639112 0.0044639112 0.0044639112 0.1160393081
## [1541] 0.1160393081 0.1319969938 0.0086856028 0.0188814011 0.0188814011
## [1546] 0.0210054996 0.0210054996 0.1723436633 0.0038774046 0.0038774046
## [1551] 0.0038774046 0.0038774046 0.1073281149 0.0050849814 0.0050849814
## [1556] 0.2730552878 0.4708462492 0.3081739605 0.0307832168 0.0307832168
## [1561] 0.2870317577 0.0294645140 0.0083660356 0.0106856969 0.0384181484
## [1566] 0.0384181484 0.0258472751 0.0364519428 0.0238603686 0.1151570805
## [1571] 0.6894067804 0.0429816281 0.0021367764 0.0021367764 0.0021367764
## [1576] 0.0739802298 0.0074758549 0.0074758549 0.0593764504 0.0173732645
## [1581] 0.0414030673 0.5171914126 0.4524870908 0.2051472954 0.1150411892
## [1586] 0.0135498706 0.0835123429 0.0089945692 0.3673421811 0.0072235975
## [1591] 0.0649465482 0.0258729701 0.3499296115 0.3499296115 0.0186698875
## [1596] 0.2251722344 0.0253457648 0.0858581379 0.0600290794 0.0562139645
## [1601] 0.0571332189 0.0567071251 0.0125201307 0.0272347363 0.0155230551
## [1606] 0.0775860346 0.2847420990 0.0899037314 0.0051865472 0.0983304875
## [1611] 0.0466972955 0.2115942320 0.2018100277 0.0197956200 0.3094756576
## [1616] 0.3094756576 0.3094756576 0.4527053618 0.0128111296 0.0117164871
## [1621] 0.0076815624 0.0076815624 0.1457524813 0.1488903997 0.4420701091
## [1626] 0.0485767213 0.0485767213 0.0277625998 0.0277625998 0.0685524698
## [1631] 0.0685524698 0.2008764393 0.2008764393 0.2008764393 0.0159196012
## [1636] 0.0971778804 0.0438342789 0.0438342789 0.0176524044 0.2741306599
## [1641] 0.0191086257 0.0191086257 0.0191086257 0.0031892105 0.0322255773
## [1646] 0.0485690761 0.0485690761 0.0485690761 0.0485690761 0.1409438470
## [1651] 0.0034177361 0.0444101901 0.0334647954 0.0612413698 0.0612413698
## [1656] 0.0224408975 0.0104657212 0.0104657212 0.0236624704 0.0642911367
## [1661] 0.0286156682 0.0286156682 0.0091701666 0.0091701666 0.0182489872
## [1666] 0.0592766275 0.0057269177 0.0057269177 0.0057269177 0.0025570074
## [1671] 0.0011717641 0.3064124379 0.0175718067 0.3344907692 0.3344907692
## [1676] 0.3804710693 0.0419100655 0.0697055952 0.0497294038 0.3220835521
## [1681] 0.3728468354 0.1520269895 0.1520269895 0.0265551769 0.1019168022
## [1686] 0.0591276271 0.0591276271 0.0786605894 0.0211882573 0.0068950976
## [1691] 0.0271984408 0.0665529285 0.0987303307 0.0987303307 0.0987303307
## [1696] 0.0305548432 0.0402132047 0.0402132047 0.0402132047 0.2424753333
## [1701] 0.2424753333 0.0029860391 0.0029860391 0.0029860391 0.0026154181
## [1706] 0.0026154181 0.5027154868 0.0843539438 0.0024560603 0.3129683519
## [1711] 0.0056840563 0.5322187661 0.0887014714 0.1221591670 0.1192331277
## [1716] 0.1221591670 0.0083792817 0.1374703889 0.0246300468 0.0246300468
## [1721] 0.0082738003 0.0492631486 0.0492631486 0.5509623882 0.3055366125
## [1726] 0.3055366125 0.0189008928 0.0615452282 0.3897189772 0.0182055204
## [1731] 0.0327556269 0.1413888014 0.3825418525 0.1586068064 0.0438319492
## [1736] 0.0438319492 0.0438319492 0.0247573074 0.0179932376 0.0179932376
## [1741] 0.3446910446 0.0121016596 0.0807348688 0.1102697602 0.0055997621
## [1746] 0.0316272765 0.0316272765 0.0316272765 0.0061491959 0.0455902273
## [1751] 0.1494265818 0.0370576364 0.0265381268 0.0951733376 0.0951733376
## [1756] 0.0951733376 0.0416486579 0.0416486579 0.0042216463 0.0973768951
## [1761] 0.2876349907 0.2876349907 0.3770980050 0.0008685074 0.0008740645
## [1766] 0.0022144916 0.0107269029 0.0107269029 0.0107269029 0.3734062876
## [1771] 0.0333485171 0.0565405363 0.0173248597 0.0173248597 0.0411444998
## [1776] 0.0079898540 0.6157231109 0.0592856193 0.0592856193 0.1443750149
## [1781] 0.2685813900 0.2685813900 0.1111615346 0.1079687274 0.3439338342
## [1786] 0.3712337216 0.1900912450 0.0101266589 0.0101266589 0.0124905880
## [1791] 0.0251176272 0.0110283534 0.0877380592 0.3270242958 0.1711671141
## [1796] 0.0431965597 0.2072528511 0.0625291973 0.1139824616 0.0146511928
## [1801] 0.0426239189 0.0426239189 0.0197518875 0.0197518875 0.0197518875
## [1806] 0.0080644836 0.0051079489 0.0133807666 0.0133807666 0.0033412170
## [1811] 0.3425131057 0.1277246481 0.1175862924 0.1175862924 0.1175862924
## [1816] 0.0509976068 0.0613132060 0.0613132060 0.2593442835 0.2593442835
## [1821] 0.2593442835 0.0762821302 0.0369887338 0.0454125101 0.1639020516
## [1826] 0.1444026016 0.0065942038 0.0938363208 0.0938363208 0.2159859481
## [1831] 0.0035452915 0.0035452915 0.4293018743 0.2381727680 0.0419552777
## [1836] 0.3571439500 0.1763947821 0.0026066975 0.1310483716 0.0314308900
## [1841] 0.0190713860 0.1000397210 0.2875431884 0.0038548542 0.4757448859
## [1846] 0.4757448859 0.1597936068 0.1597936068 0.1597936068 0.1607515830
## [1851] 0.0086802500 0.0173796992 0.0905338586 0.0905338586 0.0148662596
## [1856] 0.0831493342 0.4812560907 0.0842090786 0.0482189217 0.0213311648
## [1861] 0.0225434196 0.0116861103 0.0305388274 0.0238763047 0.5044588822
## [1866] 0.5290208419 0.0947696617 0.0947696617 0.1363888488 0.0183908562
## [1871] 0.0070191577 0.0071071067 0.0071071067 0.2013960823 0.0124317160
## [1876] 0.1245325367 0.2008012633 0.2008012633 0.2008012633 0.2008012633
## [1881] 0.1586952885 0.0044209686 0.1601699180 0.0154426212 0.0154426212
## [1886] 0.1564481593 0.1346553158 0.0143638838 0.0176396219 0.1799641903
## [1891] 0.0175054399 0.0018787886 0.1145944405 0.1145944405 0.1145944405
## [1896] 0.1899407766 0.0365440227 0.0365440227 0.0058355582 0.0195088801
## [1901] 0.0195088801 0.1616040932 0.1616040932 0.0203710753 0.0049211153
## [1906] 0.0566270194 0.1497911677 0.0127611917 0.0127611917 0.0127611917
## [1911] 0.0127611917 0.4857948756 0.0170960113 0.0170960113 0.0024476276
## [1916] 0.2396299219 0.2396299219 0.2396299219 0.2396299219 0.2396299219
## [1921] 0.0187098825 0.0163033450 0.0475560554 0.0475560554 0.0335959569
## [1926] 0.3091281940 0.0058711492 0.0667493958 0.1208971058 0.3627057317
## [1931] 0.1653085009 0.0067356503 0.0067356503 0.0067356503 0.1234972079
## [1936] 0.1234972079 0.0589222909 0.0589222909 0.0051741071 0.0028880391
## [1941] 0.0028880391 0.0028880391 0.0132149263 0.0624819975 0.0111241549
## [1946] 0.0111241549 0.0111241549 0.0202900572 0.1715727289 0.1330236153
## [1951] 0.1330236153 0.0134009763 0.0134009763 0.0935873005 0.5591278638
## [1956] 0.4237987921 0.4237987921 0.4237987921 0.4237987921 0.0041207130
## [1961] 0.0041207130 0.1510902921 0.0755481075 0.0061007762 0.0218335356
## [1966] 0.3916315381 0.0769890930 0.6565501583 0.1888967813 0.0199188312
## [1971] 0.0199188312 0.1049262853 0.0135831297 0.1309419873 0.0046095856
## [1976] 0.0046095856 0.0046095856 0.0046095856 0.0046095856 0.0714740167
## [1981] 0.0714740167 0.0117843774 0.0117843774 0.0038633656 0.0502086046
## [1986] 0.0322851728 0.0322851728 0.0625545869 0.0272612804 0.0272612804
## [1991] 0.0272612804 0.0310742431 0.0854976192 0.0772636452 0.0772636452
## [1996] 0.0772636452 0.0105860867 0.0606536986 0.3014877649 0.0174043849
## [2001] 0.0137354665 0.0255771036 0.0255771036 0.0255771036 0.0921514456
## [2006] 0.1138484850 0.0021191348 0.0021191348 0.1798548143 0.1214407834
## [2011] 0.1214407834 0.0012934202 0.0513789088 0.0806306867 0.1322421274
## [2016] 0.0936279224 0.0095512009 0.0095512009 0.1988178649 0.1098923165
## [2021] 0.0197927217 0.0062450908 0.0063385821 0.0063385821 0.0063385821
## [2026] 0.0063385821 0.1832560963 0.0209531450 0.0110682345 0.0960208386
## [2031] 0.0260368917 0.0260368917 0.0260368917 0.0136824178 0.0136824178
## [2036] 0.0136824178 0.2297696134 0.2297696134 0.2297696134 0.2632176438
## [2041] 0.0520389649 0.1904285626 0.0710203045 0.0710203045 0.0250484080
## [2046] 0.0994914126 0.2132733543 0.0265774327 0.0265774327 0.4080365203
## [2051] 0.0842042071 0.0652709965 0.1437341782 0.1437341782 0.1437341782
## [2056] 0.0086002753 0.0086002753 0.0086002753 0.0086002753 0.0414735844
## [2061] 0.0023788998 0.0061147351 0.0310546025 0.0310546025 0.0310546025
## [2066] 0.0033421061 0.0033421061 0.0033421061 0.3925720648 0.0154959517
## [2071] 0.0154959517 0.0942969958 0.0942969958 0.0130346265 0.0058859661
## [2076] 0.0058859661 0.0151396521 0.3268257542 0.2367653445 0.0239465794
## [2081] 0.0239465794 0.0025799521 0.0233473889 0.0233473889 0.0762951726
## [2086] 0.1534959247 0.1534959247 0.0557763157 0.0573649703 0.0573649703
## [2091] 0.0041109586 0.0822392545 0.1150780423 0.1431008751 0.1007523767
## [2096] 0.0829271056 0.0407680341 0.0061520362 0.0204085901 0.3253899505
## [2101] 0.0177025813 0.0177025813 0.1021650102 0.0258570783 0.0258570783
## [2106] 0.1794899302 0.1794899302 0.0385916630 0.0693730510 0.0693730510
## [2111] 0.2694610246 0.0139254058 0.0139254058 0.0139254058 0.0169043302
## [2116] 0.1475495943 0.0134758013 0.0046206073 0.0859986977 0.0859986977
## [2121] 0.3700587547 0.3700587547 0.3700587547 0.3700587547 0.3700587547
## [2126] 0.0548145021 0.4271034411 0.2462181207 0.2462181207 0.0845638425
## [2131] 0.0845638425 0.0845638425 0.0290640582 0.0290640582 0.0290640582
## [2136] 0.0290640582 0.0106745191 0.0050561379 0.1165954095 0.1165954095
## [2141] 0.0060884088 0.0059115903 0.0229479785 0.0229479785 0.0575327920
## [2146] 0.0575327920 0.0079643359 0.0188215475 0.0256462796 0.1443584929
## [2151] 0.0443847115 0.0443847115 0.1932624883 0.1932624883 0.0121077405
## [2156] 0.1981465737 0.0323229472 0.0323229472 0.0323229472 0.0136094041
## [2161] 0.0136094041 0.4225007524 0.2077509216 0.2077509216 0.0053562041
## [2166] 0.0053562041 0.3581361574 0.0028554195 0.0093708663 0.0093708663
## [2171] 0.0093708663 0.0985216866 0.1028059665 0.1095416632 0.3896479079
## [2176] 0.0553556108 0.0620410425 0.0264681521 0.0264681521 0.0034199640
## [2181] 0.0034199640 0.0289168497 0.1424117642 0.0051829381 0.1307644231
## [2186] 0.0545358154 0.0545358154 0.0545358154 0.0283277646 0.0034046932
## [2191] 0.0446211377 0.0624433280 0.1116849549 0.0361421747 0.0361421747
## [2196] 0.0361421747 0.0096253236 0.1157156852 0.0058445589 0.0058445589
## [2201] 0.3793957112 0.3793957112 0.0042669932 0.0042669932 0.0078200717
## [2206] 0.0193134821 0.0213363325 0.0066629196 0.0058116944 0.0981000354
## [2211] 0.0981000354 0.0824924631 0.0824924631 0.0801475950 0.0801475950
## [2216] 0.0801475950 0.1068418252 0.0292929426 0.0414678416 0.0414678416
## [2221] 0.0468222360 0.0468222360 0.0027917446 0.0066291236 0.0111779782
## [2226] 0.2088082393 0.0206321199 0.0795226270 0.0168889924 0.0792782076
## [2231] 0.0466197190 0.0923138790 0.0923138790 0.0923138790 0.0923138790
## [2236] 0.6116986957 0.0178867754 0.0358092160 0.1209217620 0.1209217620
## [2241] 0.0167958061 0.0167958061 0.0169700259 0.0490851213 0.0490851213
## [2246] 0.0032729565 0.0032729565 0.0832578382 0.0384158526 0.2105591386
## [2251] 0.2004828837 0.0240099722 0.0994750987 0.0457511634 0.0479073149
## [2256] 0.0479073149 0.0479073149 0.0479073149 0.1520373544 0.0016934730
## [2261] 0.0016803404 0.0016803404 0.0038042399 0.0041692051 0.0082207833
## [2266] 0.0370008366 0.0153371380 0.0146205579 0.0186503073 0.0186503073
## [2271] 0.1117003598 0.1761346286 0.1556571329 0.3520523019 0.3520523019
## [2276] 0.1401417049 0.1401417049 0.0060546527 0.1993439606 0.0946944465
## [2281] 0.2963358981 0.0435904561 0.0435904561 0.2473319873 0.0497102947
## [2286] 0.0497102947 0.0497102947 0.1315916925 0.0436906379 0.0436906379
## [2291] 0.5234761931 0.0391605794 0.0391605794 0.0862870212 0.0862870212
## [2296] 0.0299702484 0.0299702484 0.0299702484 0.0212220052 0.0213822798
## [2301] 0.0190406036 0.0190406036 0.0190406036 0.0125147015 0.0338219560
## [2306] 0.2993042469 0.0023359949 0.0023359949 0.0881711532 0.0684060217
## [2311] 0.1032873452 0.0711844867 0.3581770472 0.0689272106 0.0689272106
## [2316] 0.0036342610 0.2755252628 0.1114239606 0.1114239606 0.0683406843
## [2321] 0.0255747227 0.0076427050 0.2542225420 0.0731335735 0.0116750341
## [2326] 0.0167244665 0.0368276019 0.0180021168 0.0180021168 0.3767581548
## [2331] 0.0659794094 0.0068752345 0.0410523993 0.0064012823 0.0194286125
## [2336] 0.0194286125 0.0194286125 0.0194286125 0.0159204687 0.0043213259
## [2341] 0.1782647781 0.1782647781 0.1066292912 0.1066292912 0.0653890255
## [2346] 0.1739725452 0.1038242934 0.1150852813 0.1150852813 0.0100579294
## [2351] 0.0100579294 0.0064802085 0.0254201013 0.0407964584 0.0407964584
## [2356] 0.0237431609 0.3118388335 0.3118388335 0.1195876238 0.0784254007
## [2361] 0.0988067338 0.0988067338 0.0215055431 0.2961326568 0.4179894153
## [2366] 0.0183665664 0.1818689458 0.0416863839 0.0416863839 0.0416863839
## [2371] 0.0416863839 0.1150579721 0.0914266819 0.2850516572 0.1038071475
## [2376] 0.1552986496 0.0130489115 0.0482916255 0.0184165311 0.0184165311
## [2381] 0.0909881834 0.1801003435 0.0088241706 0.0923117214 0.6441833385
## [2386] 0.0767338883 0.0136841232 0.0227813767 0.0692502597 0.0161359584
## [2391] 0.0603398723 0.1521101021 0.0984023779 0.0265554417 0.0648557613
## [2396] 0.0789182070 0.0049794301 0.0049794301 0.0127875213 0.1033436000
## [2401] 0.0369229434 0.0369229434 0.0369229434 0.1679891663 0.0396235001
## [2406] 0.0396235001 0.0396235001 0.0396235001 0.0943813928 0.0621723408
## [2411] 0.4099117411 0.5835881214 0.0218863616 0.0208282698 0.5028091510
## [2416] 0.3330225193 0.0650073737 0.3116815467 0.0317820789 0.1239856388
## [2421] 0.2854564412 0.1193934968 0.1526870008 0.1526870008 0.6127225571
## [2426] 0.6130982696 0.1352418875 0.1887997973 0.1728459971 0.1728459971
## [2431] 0.0988246853 0.1189394488 0.1189394488 0.1189394488 0.0952634415
## [2436] 0.0092908033 0.2780992740 0.1048344998 0.1048344998 0.0843323149
## [2441] 0.0974395055 0.1027594281 0.0042025819 0.0221524205 0.0311882693
## [2446] 0.0251371955 0.0340104730 0.1283601106 0.1283601106 0.2217907207
## [2451] 0.0936388252 0.1449895548 0.0482090383 0.0355111074 0.0975178105
## [2456] 0.1586878187 0.0703683317 0.0312109480 0.1656942441 0.2818919031
## [2461] 0.0037184242 0.0374658233 0.2657774360 0.1766572137 0.1962291070
## [2466] 0.0241364635 0.0241364635 0.4761745309 0.0079685213 0.0079685213
## [2471] 0.2391033050 0.2391033050 0.2391033050 0.0370842928 0.0370842928
## [2476] 0.1006684262 0.1006684262 0.0245748777 0.0027863438 0.2312643347
## [2481] 0.6862081532 0.0032229082 0.0032229082 0.0330002290 0.0133337310
## [2486] 0.0133337310 0.0375387177 0.1613656717 0.1203989538 0.1203989538
## [2491] 0.0084879621 0.0064171833 0.3949631040 0.5806994526 0.0452548095
## [2496] 0.2013168108 0.0976761071 0.0976761071 0.0538388287 0.1609398579
## [2501] 0.1609398579 0.1690196826 0.1690196826 0.1690196826 0.1285910754
## [2506] 0.0031304587 0.2478305340 0.1735468455 0.0591217523 0.0279043801
## [2511] 0.0190985915 0.0771216891 0.0110859774 0.0167892108 0.0761978843
## [2516] 0.2724187332 0.0807654767 0.1452936055 0.0366853839 0.1897801938
## [2521] 0.2134323450 0.2134323450 0.2134323450 0.0623573288 0.0623573288
## [2526] 0.1998198184 0.0232200700 0.1766231196 0.0068169811 0.1153126458
## [2531] 0.2358658171 0.0499995280 0.0499995280 0.0499995280 0.0499995280
## [2536] 0.0760821926 0.0760821926 0.2505008004 0.3060485296 0.3060485296
## [2541] 0.0104528868 0.0766291567 0.0134525940 0.1305888450 0.1305888450
## [2546] 0.1305888450 0.0298260407 0.3334280850 0.3334280850 0.0344479222
## [2551] 0.0119816112 0.0106346511 0.0045359695 0.1096314118 0.0548951505
## [2556] 0.0545228281 0.0619139379 0.0619139379 0.0761502902 0.0761502902
## [2561] 0.0608753876 0.1172004500 0.0059499569 0.0059499569 0.0660343888
## [2566] 0.0660343888 0.2000962170 0.0879584500 0.3293418877 0.3809454143
## [2571] 0.0820296621 0.0820296621 0.3444113927 0.0097411776 0.0168528006
## [2576] 0.0570414375 0.0092281943 0.0092281943 0.0092281943 0.0723671343
## [2581] 0.1140144846 0.0239857854 0.0239857854 0.0239857854 0.2451655142
## [2586] 0.2451655142 0.0986266910 0.2361263667 0.2361263667 0.1048882646
## [2591] 0.0130826664 0.0200228976 0.0066203127 0.0066203127 0.0333623180
## [2596] 0.0044438095 0.0044438095 0.0044438095 0.0044438095 0.0044438095
## [2601] 0.0189882128 0.0189882128 0.0189882128 0.0019489954 0.2489535382
## [2606] 0.0327522181 0.0327522181 0.1229779358 0.0813682192 0.0154236580
## [2611] 0.0039281356 0.0154236580 0.0351936016 0.0314752864 0.0576641384
## [2616] 0.0576641384 0.0663020224 0.0663020224 0.1051530401 0.1051530401
## [2621] 0.0604979021 0.0604979021 0.1507986837 0.1215081580 0.1335720416
## [2626] 0.1276769124 0.1276769124 0.0034923126 0.0034923126 0.0419953570
## [2631] 0.3588767326 0.0084569633 0.3111688201 0.1130795457 0.1181613201
## [2636] 0.1181613201 0.1181613201 0.0147923554 0.0147923554 0.0779552844
## [2641] 0.2900782233 0.2900782233 0.0623525748 0.0623525748 0.0305287179
## [2646] 0.2923033790 0.0043461009 0.0583825491 0.0990420723 0.0115908134
## [2651] 0.0089713487 0.3121993923 0.0159412332 0.0159412332 0.1160193277
## [2656] 0.1021052906 0.1021052906 0.3258254985 0.3258254985 0.3258254985
## [2661] 0.3258254985 0.0077262741 0.0077262741 0.1611030313 0.3152805656
## [2666] 0.0090613145 0.2960307607 0.0070237356 0.0070237356 0.0082661588
## [2671] 0.0082661588 0.0323744511 0.0050295705 0.0050295705 0.0095242537
## [2676] 0.1111646840 0.1111646840 0.1111646840 0.0165589005 0.0165589005
## [2681] 0.0040337381 0.1451669936 0.0107820656 0.1429869600 0.0043827455
## [2686] 0.0046243316 0.1503340628 0.3771424081 0.1958583579 0.0047841445
## [2691] 0.0047841445 0.0081061928 0.0081061928 0.0081061928 0.1865207341
## [2696] 0.4613724546 0.1491181495 0.1946436052 0.0668503394 0.0767547807
## [2701] 0.3170244495 0.0370319067 0.0305890954 0.0305890954 0.0305890954
## [2706] 0.0305890954 0.0131952231 0.0679314430 0.0047115356 0.0811897559
## [2711] 0.0131631178 0.0131631178 0.0131631178 0.2013452895 0.0293985756
## [2716] 0.0293985756 0.0293985756 0.1927306261 0.2551569974 0.4303068911
## [2721] 0.0188339346 0.1247444701 0.2001170647 0.0574422949 0.0988906416
## [2726] 0.0021684257 0.0434134094 0.0434134094 0.0418126804 0.0406123838
## [2731] 0.0321370649 0.0087765319 0.0087765319 0.0087765319 0.0741515929
## [2736] 0.0741515929 0.0741515929 0.0741515929 0.1695498571 0.1695498571
## [2741] 0.1695498571 0.1695498571 0.0109932724 0.0109932724 0.1677231480
## [2746] 0.0108974885 0.0058327972 0.2586166304 0.0207952056 0.0207952056
## [2751] 0.0163795974 0.0142789881 0.0113036242 0.0046339982 0.6336086949
## [2756] 0.6336086949 0.0552855275 0.0719250334 0.0719250334 0.0719250334
## [2761] 0.1214135890 0.0027195864 0.0167756048 0.1075973315 0.1075973315
## [2766] 0.1075973315 0.1075973315 0.5489868769 0.0104439392 0.0031128318
## [2771] 0.0050807875 0.0599171487 0.0599171487 0.0556415565 0.0059638799
## [2776] 0.0040922316 0.0379170140 0.0304935506 0.0304935506 0.2652066031
## [2781] 0.0134708871 0.3859930894 0.4151714041 0.0940847450 0.0014082045
## [2786] 0.0014082045 0.0193511673 0.0192048503 0.0630307548 0.0630307548
## [2791] 0.0630307548 0.0062519152 0.0062519152 0.1101394896 0.0076274690
## [2796] 0.3867663401 0.2971726189 0.2971726189 0.2971726189 0.0030307056
## [2801] 0.0030307056 0.0638650762 0.0638650762 0.0638650762 0.4632091311
## [2806] 0.4632091311 0.0344735429 0.0344735429 0.0344735429 0.0612571206
## [2811] 0.3456281985 0.3456281985 0.0033708989 0.0012809130 0.1616301845
## [2816] 0.1616301845 0.1616301845 0.2822109533 0.0999262427 0.0999262427
## [2821] 0.1261224788 0.1261224788 0.1261224788 0.1261224788 0.1261224788
## [2826] 0.0842199336 0.0842199336 0.0140134136 0.0140134136 0.0140134136
## [2831] 0.0140134136 0.3279551638 0.2191967502 0.0409411750 0.0161109479
## [2836] 0.0161109479 0.0176182194 0.0203529269 0.0062384555 0.4489780975
## [2841] 0.3555166699 0.3293467718 0.0112666209 0.1823693075 0.0769476832
## [2846] 0.0218845115 0.0809098543 0.0809098543 0.3584682758 0.3584682758
## [2851] 0.3586252681 0.3586252681 0.0085978651 0.0085978651 0.0165565176
## [2856] 0.0165565176 0.0165565176 0.0197666844 0.0197666844 0.0376149148
## [2861] 0.0376149148 0.1804504713 0.1804504713 0.1018928961 0.1499672774
## [2866] 0.0458836919 0.0111010853 0.0111010853 0.0050017510 0.0084820103
## [2871] 0.0277325877 0.0277325877 0.0737646172 0.0737646172 0.1515436708
## [2876] 0.2507223253 0.2507223253 0.0264656281 0.0088279246 0.0095951683
## [2881] 0.0393469446 0.0393469446 0.0393469446 0.0133522466 0.1397145712
## [2886] 0.1397145712 0.0344686881 0.1610149956 0.1061387077 0.0909930244
## [2891] 0.0282908611 0.0282908611 0.0282908611 0.0282908611 0.0449836956
## [2896] 0.0449836956 0.0106918077 0.2982494305 0.0802591276 0.0802591276
## [2901] 0.0038805840 0.0978552736 0.1408107403 0.0019723610 0.0019723610
## [2906] 0.0019723610 0.0529766024 0.0059307983 0.2240487122 0.0662008923
## [2911] 0.0305403390 0.0726290957 0.0815778441 0.0800662120 0.0598322611
## [2916] 0.0598322611 0.0598322611 0.0598322611 0.0424722139 0.2031825721
## [2921] 0.0471376097 0.1807431533 0.1807431533 0.0012486076 0.0155406228
## [2926] 0.0173810637 0.1290745104 0.0883907270 0.0722314540 0.0456984726
## [2931] 0.0456984726 0.0055565812 0.0984151993 0.0984151993 0.3249393936
## [2936] 0.3249393936 0.3249393936 0.1070247777 0.0066486660 0.0280552975
## [2941] 0.0246121903 0.0246121903 0.0246121903 0.0052526833 0.1531546969
## [2946] 0.2150327543 0.3593807512 0.2985526378 0.0581234831 0.0581234831
## [2951] 0.1058725345 0.1058725345 0.4938452305 0.4938452305 0.0336011025
## [2956] 0.0336011025 0.2011417979 0.2011417979 0.0023369563 0.0210218606
## [2961] 0.0210218606 0.0210218606 0.3833294317 0.0862540540 0.0862540540
## [2966] 0.0220151546 0.0220151546 0.0623198864 0.0623198864 0.0623198864
## [2971] 0.0623198864 0.0604863655 0.0604863655 0.0103702131 0.1926111287
## [2976] 0.1926111287 0.0326127621 0.4429817550 0.0507832728 0.0507832728
## [2981] 0.0473008736 0.0473008736 0.0771861024 0.4163428350 0.0483217464
## [2986] 0.0107374165 0.4269430497 0.0035721139 0.0813719720 0.0813719720
## [2991] 0.5040342559 0.0088414977 0.0116783005 0.0662981116 0.3003573720
## [2996] 0.0636821522 0.0636821522 0.0636821522 0.1087507717 0.1087507717
## [3001] 0.1817943978 0.2024722237 0.0034947826 0.0042912709 0.1483026576
## [3006] 0.2329310998 0.0160611371 0.0160611371 0.0290662045 0.0290662045
## [3011] 0.1445346606 0.2795806995 0.0750930278 0.3151782558 0.3151782558
## [3016] 0.0341848084 0.0341848084 0.0049023727 0.0049023727 0.0049023727
## [3021] 0.0137301549 0.0137301549 0.0137301549 0.0137301549 0.0240167711
## [3026] 0.1666304300 0.0677912010 0.0677912010 0.2539872470 0.3928952869
## [3031] 0.3928952869 0.0026596954 0.1511061605 0.1511061605 0.1511061605
## [3036] 0.0364519427 0.0364519427 0.0364519427 0.1682156273 0.1682156273
## [3041] 0.1682156273 0.0381480418 0.0381480418 0.0381480418 0.0178274185
## [3046] 0.0293679420 0.0177540823 0.0342021410 0.0342021410 0.0342021410
## [3051] 0.0075781880 0.0888034349 0.5380378996 0.0125236082 0.0261779771
## [3056] 0.0261779771 0.0261779771 0.0421554752 0.0421554752 0.4852644097
## [3061] 0.4852644097 0.4852644097 0.0402009320 0.0047449323 0.3756474519
## [3066] 0.2132105715 0.0535226472 0.0349908269 0.0221133478 0.3592219474
## [3071] 0.0286508902 0.0039395040 0.0629862233 0.0077057718 0.0137739596
## [3076] 0.0216506718 0.0103410148 0.0065149197 0.0717246199 0.0163188696
## [3081] 0.0163188696 0.0163188696 0.0280988178 0.0280988178 0.2060514136
## [3086] 0.0036546768 0.1535874838 0.3283934177 0.3283934177 0.0302047783
## [3091] 0.1533181553 0.0452585913 0.0022981455 0.0117849364 0.1154821829
## [3096] 0.1154821829 0.0060890550 0.1061843132 0.0733562578 0.0058646925
## [3101] 0.0058646925 0.0844327536 0.0238988380 0.4870437707 0.0210848557
## [3106] 0.0078392847 0.3765159058 0.3765159058 0.0251341224 0.1012354509
## [3111] 0.0066751973 0.0066751973 0.0066751973 0.0066751973 0.0066751973
## [3116] 0.0066751973 0.0345185276 0.0317248064 0.0081873887 0.1991783517
## [3121] 0.0041635402 0.0057969523 0.0057969523 0.0362922433 0.0362922433
## [3126] 0.0362922433 0.0090990887 0.0090990887 0.5319585499 0.1140758070
## [3131] 0.0042708934 0.0042708934 0.1117818131 0.0142150824 0.0142150824
## [3136] 0.0034088850 0.0745662597 0.3756685310 0.1112150296 0.1112150296
## [3141] 0.0234242719 0.0511789579 0.1131401296 0.1131401296 0.1131401296
## [3146] 0.0677957587 0.0542583673 0.0065816200 0.0118290618 0.0641510921
## [3151] 0.0232860660 0.0232860660 0.0232860660 0.0385686098 0.0037439132
## [3156] 0.0024818893 0.0024818893 0.1826274548 0.0029188691 0.0029188691
## [3161] 0.0339276771 0.0045778415 0.6822757295 0.6822757295 0.0040665433
## [3166] 0.0040665433 0.3617409664 0.1011607636 0.1174120261 0.1174120261
## [3171] 0.0117370054 0.0117370054 0.0096236398 0.0096236398 0.0105574369
## [3176] 0.0196968516 0.0196968516 0.0196968516 0.0754612345 0.0030508250
## [3181] 0.0030508250 0.0030508250 0.0359362125 0.6621364288 0.1135253540
## [3186] 0.0445160875 0.0445160875 0.1780951066 0.0143006447 0.0161930855
## [3191] 0.0161930855 0.2402886932 0.2402886932 0.0137721333 0.0137721333
## [3196] 0.0137721333 0.0137721333 0.0742834096 0.0623326826 0.0623326826
## [3201] 0.0193734168 0.0193734168 0.0308194015 0.0852946123 0.3070173945
## [3206] 0.0115201856 0.2696457996 0.2696457996 0.2696457996 0.0312235779
## [3211] 0.0312235779 0.0468220905 0.0081289792 0.0055075721 0.0055075721
## [3216] 0.0055075721 0.1087709939 0.0790459144 0.0790459144 0.1121760904
## [3221] 0.1786961511 0.4086496635 0.0031241497 0.0031241497 0.0206583281
## [3226] 0.0345406281 0.0865841100 0.1100406076 0.1100406076 0.1100406076
## [3231] 0.5249132739 0.3734847353 0.0211240230 0.0042300198 0.3418187764
## [3236] 0.3418187764 0.1226489385 0.2407447299 0.0016795303 0.0723665686
## [3241] 0.1484122407 0.1484122407 0.1484122407 0.0838168349 0.0699162333
## [3246] 0.0356072841 0.0356072841 0.3762885608 0.0575921951 0.1672858478
## [3251] 0.2398552510 0.0171541930 0.2214964645 0.2214964645 0.2656181291
## [3256] 0.0050075738 0.0930927053 0.0930927053 0.0455439261 0.0067071088
## [3261] 0.2903080921 0.0557964395 0.0121556191 0.0121556191 0.0121556191
## [3266] 0.0242481720 0.0129309966 0.0129309966 0.0129309966 0.1072820614
## [3271] 0.0121037025 0.2111939697 0.2111939697 0.0286616920 0.0286616920
## [3276] 0.1778920229 0.3263349751 0.1523055147 0.1321040014 0.0409388791
## [3281] 0.0409388791 0.0409388791 0.0409388791 0.0466958230 0.0466958230
## [3286] 0.0222844328 0.1793997802 0.0971919871 0.0256965764 0.0256965764
## [3291] 0.0598520123 0.0050714002 0.2714166612 0.0085615253 0.0028295351
## [3296] 0.0138370028 0.2024058696 0.2024058696 0.0377953279 0.0236538494
## [3301] 0.0705978439 0.0306526669 0.0306526669 0.0306526669 0.0520098608
## [3306] 0.0520098608 0.2348379080 0.0132931142 0.2597786076 0.2597786076
## [3311] 0.1688007879 0.2501878816 0.3035451211 0.2260048638 0.3851222196
## [3316] 0.0193961094 0.0062765215 0.0062765215 0.0062765215 0.0062765215
## [3321] 0.0233158546 0.1999758617 0.0078379698 0.1776738631 0.0185566574
## [3326] 0.0185566574 0.0185566574 0.1990478224 0.0277761153 0.1151038012
## [3331] 0.2099086020 0.1017512267 0.0331639994 0.0643947402 0.0643947402
## [3336] 0.2713099484 0.0292364970 0.0292364970 0.0292364970 0.0232326478
## [3341] 0.0017656642 0.0017656642 0.0109495789 0.0109495789 0.0389913068
## [3346] 0.0389913068 0.0389913068 0.0389913068 0.0162595968 0.0510580037
## [3351] 0.0510580037 0.0510580037 0.0160742410 0.0160742410 0.0048550156
## [3356] 0.0554212720 0.2953769426 0.0581663034 0.0581663034 0.0112043146
## [3361] 0.0325088020 0.0325088020 0.0325088020 0.1544960682 0.0104494293
## [3366] 0.0919744442 0.0089292750 0.0552269393 0.0433228670 0.3162701793
## [3371] 0.0896381067 0.0896381067 0.0896381067 0.0896381067 0.0224443840
## [3376] 0.0171643539 0.0171643539 0.3654949056 0.2251425144 0.1814029098
## [3381] 0.0066542543 0.0066542543 0.0066542543 0.0066542543 0.1949551956
## [3386] 0.1949551956 0.0071647767 0.0218611842 0.0145962394 0.0876944112
## [3391] 0.0475422235 0.0475422235 0.0475422235 0.0475422235 0.0475422235
## [3396] 0.0475422235 0.0090881392 0.0186295853 0.0186295853 0.0665436855
## [3401] 0.2839534884 0.2839534884 0.0983994813 0.0983994813 0.0692601019
## [3406] 0.0692601019 0.0245850519 0.2246420538 0.2246420538 0.0688664566
## [3411] 0.5102424968 0.1067316285 0.1067316285 0.0240200918 0.0240200918
## [3416] 0.0122932359 0.1452416828 0.0400845964 0.0400845964 0.0118657662
## [3421] 0.0118657662 0.0601265282 0.0395646860 0.0395646860 0.0183640477
## [3426] 0.0847624218 0.0847624218 0.0115950454 0.0023087324 0.0434214520
## [3431] 0.0066018435 0.0066018435 0.0066018435 0.0561275680 0.0144883323
## [3436] 0.2525319694 0.1309320620 0.0055584884 0.0807642249 0.1511100966
## [3441] 0.0816665474 0.0816665474 0.0069048205 0.0194918727 0.1689036069
## [3446] 0.1689036069 0.3217059077 0.0277469508 0.0560156855 0.0560156855
## [3451] 0.2085505859 0.0442598941 0.0134891039 0.0399971592 0.0214895960
## [3456] 0.0214895960 0.0078713994 0.0166639819 0.0044971772 0.0044971772
## [3461] 0.4315961858 0.0155549852 0.0033033087 0.0033033087 0.0100476533
## [3466] 0.0100476533 0.3836361149 0.0521659040 0.0521659040 0.2398676703
## [3471] 0.0236992254 0.0236992254 0.0236992254 0.0182130230 0.0997046781
## [3476] 0.0024019980 0.0024019980 0.0596069826 0.0596069826 0.0596069826
## [3481] 0.0596069826 0.3273581299 0.3273581299 0.3120061162 0.1591875695
## [3486] 0.0048840862 0.0048840862 0.0267227459 0.0928928552 0.0037062772
## [3491] 0.0267962999 0.0267962999 0.0267962999 0.0147080129 0.0147080129
## [3496] 0.0147080129 0.0147080129 0.0065347530 0.0994249195 0.0046716801
## [3501] 0.0046716801 0.0913094060 0.1237002774 0.1237002774 0.1649006697
## [3506] 0.2098263431 0.0020617342 0.0533978917 0.0533978917 0.0533978917
## [3511] 0.1967461747 0.1967461747 0.1967461747 0.1967461747 0.1967461747
## [3516] 0.2892980444 0.0031167052 0.0031167052 0.0276951494 0.0276951494
## [3521] 0.2465333489 0.0465902214 0.0465902214 0.0157492333 0.0042542926
## [3526] 0.0042542926 0.0386271020 0.0210608647 0.0210608647 0.0210608647
## [3531] 0.0210608647 0.0169240191 0.0046760130 0.0103792008 0.0060010001
## [3536] 0.0060010001 0.0377041626 0.0447642237 0.0447642237 0.0122247527
## [3541] 0.2587972309 0.0481297260 0.0101053954 0.1538782290 0.0286250377
## [3546] 0.0286250377 0.0230450799 0.0230450799 0.0034465002 0.2581024457
## [3551] 0.0599612877 0.0427697755 0.0427697755 0.1103645545 0.1103645545
## [3556] 0.1103645545 0.1103645545 0.5913990244 0.1537789817 0.1286815386
## [3561] 0.0053659738 0.0053659738 0.0098234521 0.0325133274 0.0145867541
## [3566] 0.0049923368 0.0049923368 0.0355742177 0.0603035768 0.0546304348
## [3571] 0.0042495326 0.1214916640 0.0622035470 0.1099184840 0.5122455065
## [3576] 0.0166212090 0.0166212090 0.0166212090 0.0166212090 0.0087498270
## [3581] 0.0067382780 0.0067382780 0.3162502361 0.1529653452 0.1529653452
## [3586] 0.1529653452 0.0622389769 0.0610229918 0.4060373455 0.0577156474
## [3591] 0.0577156474 0.0577156474 0.3723421277 0.0708711756 0.2132844193
## [3596] 0.0279484277 0.0279484277 0.0322911324 0.0095588109 0.0699586148
## [3601] 0.0347105064 0.0355355310 0.0201343262 0.0201343262 0.0042334824
## [3606] 0.0016300637 0.0016300637 0.1533462482 0.1533462482 0.0414782999
## [3611] 0.0458565092 0.3503552362 0.1858207644 0.0121395446 0.0067012003
## [3616] 0.0091934807 0.0091934807 0.0028603448 0.2892734973 0.0744669789
## [3621] 0.0744669789 0.0071850686 0.4757425461 0.3224359486 0.3224359486
## [3626] 0.0273962755 0.0577632701 0.0247383186 0.0247383186 0.6872220423
## [3631] 0.6872220423 0.0771733830 0.0771733830 0.0074858897 0.0074858897
## [3636] 0.3006246921 0.0985244080 0.0052492449 0.0066081641 0.0669385989
## [3641] 0.0669385989 0.1813790288 0.3741099568 0.0757620265 0.0775724240
## [3646] 0.0653698200 0.1265930526 0.2173416459 0.2173416459 0.0063059028
## [3651] 0.1276065207 0.0242764317 0.1179709814 0.1179709814 0.1179709814
## [3656] 0.1179709814 0.0049335274 0.0049335274 0.0049335274 0.0049335274
## [3661] 0.0102918859 0.0132084707 0.1787870585 0.0339367294 0.0050214288
## [3666] 0.0050214288 0.0050214288 0.0050214288 0.0820152864 0.1870723563
## [3671] 0.1870723563 0.1870723563 0.0171817037 0.1393864292 0.0247848901
## [3676] 0.7920111036 0.2473361159 0.2473361159 0.0076251506 0.0076251506
## [3681] 0.2843871038 0.0917916881 0.7396021199 0.0425697190 0.3478346595
## [3686] 0.3478346595 0.0787687478 0.0787687478 0.4112527031 0.4112527031
## [3691] 0.2893622090 0.0646176043 0.0646176043 0.0169615647 0.0169615647
## [3696] 0.0738300950 0.0448524901 0.0448524901 0.2109627196 0.0721990294
## [3701] 0.0721990294 0.0351644361 0.2057592593 0.0398672825 0.0398672825
## [3706] 0.0398672825 0.0398672825 0.0161533934 0.0161533934 0.0161533934
## [3711] 0.0257003240 0.0235920004 0.0791840119 0.0171677063 0.0171677063
## [3716] 0.0172218428 0.0172218428 0.0172218428 0.0079471579 0.0079471579
## [3721] 0.0079471579 0.0165083795 0.2430807045 0.1306806042 0.0062126277
## [3726] 0.5785017954 0.0023739746 0.6218854912 0.6218854912 0.5732115692
## [3731] 0.1280686396 0.6222331065 0.6222331065 0.6222331065 0.6222331065
## [3736] 0.0037941359 0.0037941359 0.0037941359 0.0037941359 0.0132063582
## [3741] 0.1806439833 0.1806439833 0.1806439833 0.1806439833 0.2453001880
## [3746] 0.0904404615 0.0904404615 0.0067879803 0.0043294751 0.0117713254
## [3751] 0.0772233912 0.0332760642 0.4972465228 0.0052775944 0.0128812306
## [3756] 0.0128812306 0.3401515903 0.0196368038 0.0196368038 0.0635986746
## [3761] 0.0635986746 0.0635986746 0.0635986746 0.2688498483 0.2688498483
## [3766] 0.0170662233 0.0651486037 0.0307887710 0.0643297398 0.0643297398
## [3771] 0.0198474478 0.1878206395 0.1490877572 0.0450690814 0.0450690814
## [3776] 0.0089740667 0.0089740667 0.0365134755 0.0365134755 0.2292302854
## [3781] 0.2292302854 0.3091447861 0.3091447861 0.0603229168 0.0551989740
## [3786] 0.0225217736 0.0027872978 0.1442049747 0.1364802996 0.1364802996
## [3791] 0.0460100122 0.4242957131 0.1012060221 0.0973879877 0.0973879877
## [3796] 0.0105854243 0.0105854243 0.0105854243 0.0105854243 0.0207545292
## [3801] 0.0207545292 0.0207545292 0.1134755309 0.0093211060 0.0093211060
## [3806] 0.0093211060 0.0019738196 0.0019738196 0.0019738196 0.0019738196
## [3811] 0.3117524878 0.0021273254 0.3168601609 0.3168601609 0.3168601609
## [3816] 0.0046759993 0.0046759993 0.0046759993 0.0046759993 0.0046759993
## [3821] 0.2003300601 0.0068711820 0.0912726972 0.0912726972 0.0912726972
## [3826] 0.2112515643 0.2112515643 0.0319084011 0.0256735610 0.2492299488
## [3831] 0.1169703339 0.0993570613 0.2211741338 0.2211741338 0.2211741338
## [3836] 0.0048717863 0.0048717863 0.0767111822 0.2134106516 0.2134106516
## [3841] 0.7605558482 0.7605558482 0.7605558482 0.4219033462 0.2539948918
## [3846] 0.0055370488 0.4142160023 0.0033284110 0.0033284110 0.0121515802
## [3851] 0.0121515802 0.0309423438 0.2983550432 0.1550389520 0.1550389520
## [3856] 0.4840473354 0.0547884240 0.2075853488 0.0572590946 0.0804536257
## [3861] 0.0804536257 0.0804536257 0.0804536257 0.0804536257 0.0420488130
## [3866] 0.0420488130 0.1767104911 0.1767104911 0.0141202462 0.0141202462
## [3871] 0.1025603643 0.2255595751 0.1424939228 0.1501890356 0.0091854424
## [3876] 0.3992208235 0.2090854645 0.0094439813 0.0163414605 0.0337576087
## [3881] 0.0124130311 0.0233907995 0.2500204546 0.2500204546 0.2500204546
## [3886] 0.2500204546 0.0551589734 0.0817825026 0.0083667149 0.0083667149
## [3891] 0.0083667149 0.3150383277 0.3150383277 0.3150383277 0.3150383277
## [3896] 0.3150383277 0.1999755777 0.1654600095 0.0235353370 0.1624919723
## [3901] 0.1858996329 0.0356777241 0.0085314741 0.0558982181 0.0063024089
## [3906] 0.5526201306 0.2592934200 0.0039606362 0.1499464035 0.1499464035
## [3911] 0.1499464035 0.1499464035 0.0731494350 0.0731494350 0.1004638997
## [3916] 0.0483940272 0.1465918906 0.0637644330 0.0637644330 0.0637644330
## [3921] 0.0683813610 0.0683813610 0.0683813610 0.1437220732 0.0110247644
## [3926] 0.0110247644 0.0110247644 0.0147436434 0.0411303684 0.0382062427
## [3931] 0.0382062427 0.0107762082 0.0107762082 0.0148277541 0.0080185056
## [3936] 0.0080185056 0.0080185056 0.2998440320 0.2998440320 0.2998440320
## [3941] 0.2998440320 0.2998440320 0.2998440320 0.2998440320 0.2152828464
## [3946] 0.1744036393 0.4534166226 0.0968169014 0.0281436671 0.2304937183
## [3951] 0.0049945363 0.0049945363 0.0800135178 0.2742617773 0.2240287095
## [3956] 0.0039067149 0.0039067149 0.0039067149 0.0080467148 0.2979195809
## [3961] 0.3891475124 0.0486964371 0.0486964371 0.0486964371 0.2126604456
## [3966] 0.1336380934 0.1336380934 0.1336380934 0.2429033997 0.1920960285
## [3971] 0.3951730221 0.2050202803 0.0605565251 0.0605565251 0.0605565251
## [3976] 0.0107663011 0.0107663011 0.0066800444 0.0289147129 0.0461802439
## [3981] 0.0204420009 0.0204420009 0.1889194561 0.0075007598 0.0097788386
## [3986] 0.1910118380 0.0182611685 0.0182611685 0.0049130376 0.0705272702
## [3991] 0.0705272702 0.0705272702 0.1882896315 0.1047029741 0.1047029741
## [3996] 0.0731742032 0.0731742032 0.0731742032 0.0731742032 0.2479236876
## [4001] 0.0829231405 0.0829231405 0.0729004023 0.0729004023 0.0366248965
## [4006] 0.0328576423 0.0269807469 0.0241917189 0.0152795391 0.0152795391
## [4011] 0.3927358093 0.0117041202 0.3150047125 0.0090340763 0.0104769060
## [4016] 0.1496496533 0.0287785082 0.0192553519 0.0192553519 0.0192553519
## [4021] 0.0192553519 0.0192553519 0.1300669893 0.0340724778 0.0340724778
## [4026] 0.1851985107 0.1838801315 0.4363555347 0.4415993223 0.0950279041
## [4031] 0.0950279041 0.0495399016 0.0950279041 0.0950279041 0.0495399016
## [4036] 0.0048041871 0.1826661353 0.6529689876 0.1073467082 0.0033678306
## [4041] 0.0198083754 0.0113735852 0.4103530598 0.0768870832 0.0768870832
## [4046] 0.0924270815 0.0513083156 0.0513083156 0.1528213585 0.1528213585
## [4051] 0.8972893160 0.0366950030 0.0171469414 0.1842374529 0.1842374529
## [4056] 0.0183043121 0.0183043121 0.1061053135 0.1061053135 0.0167345952
## [4061] 0.0246930188 0.0246930188 0.2438669821 0.1159405805 0.0639843298
## [4066] 0.1617485393 0.1617485393 0.0312962214 0.1639535918 0.1161156126
## [4071] 0.7168679627 0.7168679627 0.0027186994 0.0027186994 0.0344357918
## [4076] 0.0344357918 0.2991614031 0.2991614031 0.1296599661 0.3553075397
## [4081] 0.2017444172 0.0804074043 0.1587139912 0.1587139912 0.1587139912
## [4086] 0.1587139912 0.1587139912 0.1587139912 0.1587139912 0.0025285717
## [4091] 0.0025285717 0.0025285717 0.0150313173 0.0139013512 0.2025592151
## [4096] 0.1611466243 0.1611466243 0.0269934612 0.0134082097 0.0830070842
## [4101] 0.0830070842 0.0830070842 0.0830070842 0.0830070842 0.0830070842
## [4106] 0.3509948644 0.0428750199 0.2071675393 0.0131596920 0.3279489029
## [4111] 0.0215891370 0.1026795550 0.0039736258 0.1858843383 0.2563971789
## [4116] 0.0268903935 0.1028284018 0.1127440211 0.1127440211 0.1127440211
## [4121] 0.1127440211 0.1127440211 0.1127440211 0.4539680035 0.1014472374
## [4126] 0.0918057738 0.3006191467 0.3006191467 0.0169449764 0.0169449764
## [4131] 0.0169449764 0.2512862711 0.1342830249 0.1342830249 0.1342830249
## [4136] 0.0037136846 0.0037136846 0.0649376762 0.1424694561 0.2134285195
## [4141] 0.0679546196 0.0019326627 0.0022604357 0.0013198974 0.0019326627
## [4146] 0.0019326627 0.0058194101 0.0586721743 0.0586721743 0.0188490990
## [4151] 0.2727129160 0.2727129160 0.0244812911 0.0891739611 0.0891739611
## [4156] 0.0891739611 0.0891739611 0.0891739611 0.0095174285 0.0095174285
## [4161] 0.0095174285 0.0095174285 0.2367099938 0.0163044116 0.0163044116
## [4166] 0.0643245820 0.0803188543 0.2666613082 0.0364312874 0.0364312874
## [4171] 0.1561828096 0.1561828096 0.1561828096 0.0740975775 0.4414410018
## [4176] 0.0029303212 0.0018458402 0.0250127759 0.0250127759 0.0250127759
## [4181] 0.4379569565 0.0451512767 0.0032573837 0.1691834386 0.1691834386
## [4186] 0.1691834386 0.0375323951 0.0437829501 0.1829722706 0.0735232914
## [4191] 0.0735232914 0.1765101084 0.1765101084 0.1765101084 0.1765101084
## [4196] 0.1323085237 0.0525628412 0.0525628412 0.0074417436 0.0715949997
## [4201] 0.0318910902 0.0318910902 0.0244709424 0.0667389786 0.0667389786
## [4206] 0.0092218698 0.1765637618 0.3334594788 0.0301287901 0.2999292831
## [4211] 0.4448815520 0.0144998356 0.0045097048 0.0045097048 0.0075515718
## [4216] 0.0075515718 0.0569699778 0.0569699778 0.0569699778 0.1474775177
## [4221] 0.0039130913 0.0039130913 0.3760886923 0.1706645933 0.1591881015
## [4226] 0.1591881015 0.1591881015 0.0100488323 0.2614802681 0.0058405044
## [4231] 0.0058405044 0.0180167717 0.3116315702 0.2225025072 0.1866570476
## [4236] 0.1866570476 0.0224278090 0.0721604524 0.0053353357 0.1150947773
## [4241] 0.1150947773 0.0712073668 0.0712073668 0.0124057428 0.1658658719
## [4246] 0.1658658719 0.1658658719 0.0873760918 0.0606946913 0.0606946913
## [4251] 0.0606946913 0.2190151398 0.0381616141 0.0389368815 0.0389368815
## [4256] 0.1444259997 0.1444259997 0.1444259997 0.1444259997 0.0029862887
## [4261] 0.0025398869 0.0025398869 0.0247048126 0.0912762198 0.1108917600
## [4266] 0.0665820378 0.0442261984 0.0442261984 0.0442261984 0.0442261984
## [4271] 0.0442261984 0.0442261984 0.0684843205 0.0029737374 0.0079103379
## [4276] 0.0079103379 0.0608741657 0.0608741657 0.1277390105 0.0195660159
## [4281] 0.0075744337 0.0075744337 0.0075744337 0.0075744337 0.0035042318
## [4286] 0.0035042318 0.1160375520 0.1160375520 0.1160375520 0.1160375520
## [4291] 0.1160375520 0.0150012421 0.4388000229 0.0236535628 0.0327204746
## [4296] 0.0921545064 0.0921545064 0.0921545064 0.1303012952 0.0541287799
## [4301] 0.0541287799 0.2613606114 0.0999403482 0.0999403482 0.0999403482
## [4306] 0.0283017259 0.0283017259 0.0283017259 0.0425530571 0.1420559242
## [4311] 0.1420559242 0.1420559242 0.0646900803 0.0384221847 0.0384221847
## [4316] 0.0384221847 0.0819512776 0.0819512776 0.1052504809 0.0029908468
## [4321] 0.0029908468 0.0572114911 0.0011927954 0.0011927954 0.3844176057
## [4326] 0.3132185315 0.2327202409 0.2327202409 0.0129995117 0.0174072384
## [4331] 0.1844059901 0.1651991954 0.1651991954 0.0532438329 0.0532438329
## [4336] 0.0532438329 0.0532438329 0.0604691606 0.0324257486 0.0340761289
## [4341] 0.0136108487 0.0136108487 0.0136108487 0.0136108487 0.0136108487
## [4346] 0.0126827189 0.0106005468 0.0873712924 0.0873712924 0.0356721944
## [4351] 0.0053749355 0.1132048448 0.0873531822 0.1010900996 0.1189330623
## [4356] 0.1095987350 0.0107708821 0.3204371518 0.0070075037 0.0536925399
## [4361] 0.0766034952 0.0766034952 0.1064365459 0.0386862414 0.0386862414
## [4366] 0.0040849728 0.0040849728 0.3142977511 0.0088094169 0.0088094169
## [4371] 0.3795937285 0.0618930316 0.0182129888 0.0182129888 0.0182129888
## [4376] 0.0663273389 0.0576251970 0.0576251970 0.0094347242 0.0094347242
## [4381] 0.0382808688 0.0424266597 0.0424266597 0.0424266597 0.0231272863
## [4386] 0.0093861681 0.1772306793 0.1772306793 0.1772306793 0.0436662866
## [4391] 0.0139398133 0.1270419689 0.1270419689 0.1083548612 0.0362984191
## [4396] 0.5278967375 0.0088870677 0.0798545858 0.0383231924 0.4198102175
## [4401] 0.4198102175 0.0248727989 0.0248727989 0.0029339331 0.0472149677
## [4406] 0.0472149677 0.0472149677 0.0472149677 0.0074990629 0.0311933884
## [4411] 0.0208931618 0.0120912875 0.1537120442 0.1999545775 0.2453881413
## [4416] 0.2453881413 0.0564057924 0.0269649590 0.1202578391 0.0082953406
## [4421] 0.0082953406 0.0937098416 0.0031579529 0.0031579529 0.0100331603
## [4426] 0.0100331603 0.0100331603 0.0100331603 0.0100331603 0.0813966510
## [4431] 0.0813966510 0.0813966510 0.0813966510 0.0813966510 0.2376309400
## [4436] 0.0164687655 0.1015612050 0.1015612050 0.1143947548 0.1143947548
## [4441] 0.0105838009 0.1466821640 0.1466821640 0.0063510565 0.2638428310
## [4446] 0.0017553107 0.1196248986 0.1196248986 0.1449130660 0.1515630864
## [4451] 0.1515630864 0.2097077144 0.0565685855 0.3483052503 0.1117381060
## [4456] 0.2352964689 0.2352964689 0.0158200788 0.2103970375 0.0385155660
## [4461] 0.0385155660 0.2292058535 0.0350821274 0.0350821274 0.0350821274
## [4466] 0.0350821274 0.0062983668 0.0062983668 0.0854870292 0.0020677273
## [4471] 0.0282937338 0.0282937338 0.0057996465 0.3843763529 0.3016304270
## [4476] 0.0132794082 0.0039666517 0.0039666517 0.3841764721 0.0316248005
## [4481] 0.0808367043 0.0155032213 0.5010075499 0.0849706800 0.1399565563
## [4486] 0.0012941101 0.0012675471 0.0776021566 0.0191432813 0.0191432813
## [4491] 0.0940775919 0.0940775919 0.0940775919 0.1991977961 0.0155736338
## [4496] 0.1359820106 0.1359820106 0.1359820106 0.0292720904 0.0292720904
## [4501] 0.0094872812 0.0094872812 0.0094872812 0.0018526106 0.0584026073
## [4506] 0.0584026073 0.0584026073 0.0584026073 0.4435835665 0.0684946199
## [4511] 0.0684946199 0.0451309363 0.0337377483 0.0153110981 0.0062358883
## [4516] 0.0131938381 0.0713954016 0.0082467552 0.0081348323 0.0724905674
## [4521] 0.0062979107 0.1263300409 0.0293163943 0.0044409691 0.0044409691
## [4526] 0.0044409691 0.0044409691 0.3932398710 0.3932398710 0.1924453268
## [4531] 0.0032273322 0.0032273322 0.1890454748 0.1510781368 0.2948398170
## [4536] 0.1945398955 0.0083331715 0.0083331715 0.0173430645 0.0010511971
## [4541] 0.1078134981 0.1078134981 0.1078134981 0.1078134981 0.2352338672
## [4546] 0.2352338672 0.2352338672 0.2352338672 0.2352338672 0.2352338672
## [4551] 0.2041700764 0.2041700764 0.2210541381 0.2029937035 0.1084641994
## [4556] 0.5412128562 0.5412128562 0.5412128562 0.1436643187 0.0183519899
## [4561] 0.0183519899 0.0183519899 0.0183519899 0.4621966879 0.4621966879
## [4566] 0.0507642677 0.1241920780 0.0352268142 0.0352268142 0.1221171695
## [4571] 0.1221171695 0.0223326559 0.1043864089 0.1043864089 0.1043864089
## [4576] 0.0336103839 0.1289420958 0.1289420958 0.1737390861 0.0021898759
## [4581] 0.0771773896 0.0771773896 0.0771773896 0.4449981696 0.4449981696
## [4586] 0.0124686927 0.0349173652 0.0359522266 0.6301282737 0.6335672372
## [4591] 0.0071269871 0.0071269871 0.0348713200 0.0348713200 0.0348713200
## [4596] 0.0348713200 0.0121007517 0.0121007517 0.0610143755 0.0777020094
## [4601] 0.0362715583 0.0362715583 0.0362715583 0.0362715583 0.0362715583
## [4606] 0.0362715583 0.1215990430 0.0339343653 0.0339343653 0.0339343653
## [4611] 0.0339343653 0.0339343653 0.1858044542 0.0094499507 0.0098596251
## [4616] 0.0244488503 0.0244488503 0.2858830699 0.2858830699 0.0082444360
## [4621] 0.0196669065 0.0196669065 0.0037334042 0.0318365152 0.1139813079
## [4626] 0.1139813079 0.1139813079 0.0124394034 0.0124394034 0.0146824651
## [4631] 0.0146824651 0.0146824651 0.0410126726 0.0943870454 0.0943870454
## [4636] 0.0943870454 0.0214454607 0.0139641700 0.0241623105 0.0241623105
## [4641] 0.1499648332 0.1499648332 0.0066097853 0.2237727326 0.2237727326
## [4646] 0.0433390280 0.0433390280 0.1398630188 0.0371118161 0.2653190389
## [4651] 0.2653190389 0.2653190389 0.1218134810 0.0224302113 0.0224302113
## [4656] 0.0309148141 0.0309148141 0.0143472731 0.2516794955 0.4160322429
## [4661] 0.1644962211 0.0342118130 0.0342118130 0.0342118130 0.0342118130
## [4666] 0.0342118130 0.0146479006 0.0055391105 0.0795386102 0.0980514323
## [4671] 0.0980514323 0.0980514323 0.0980514323 0.0980514323 0.3601826079
## [4676] 0.0021152355 0.0021152355 0.0021152355 0.0021152355 0.1179490665
## [4681] 0.1482145651 0.1482145651 0.0243782722 0.0075599601 0.0103427255
## [4686] 0.0103427255 0.2326103355 0.2326103355 0.2326103355 0.0119255618
## [4691] 0.0119255618 0.0807045235 0.0281123120 0.3996616735 0.0022059728
## [4696] 0.2617268966 0.2617268966 0.2617268966 0.2617268966 0.0293827034
## [4701] 0.1378594347 0.0090608878 0.1065893833 0.0740386378 0.0074498998
## [4706] 0.1791401033 0.1772769782 0.2505415208 0.0178435101 0.0994916659
## [4711] 0.0994916659 0.0120982868 0.0120982868 0.0951622116 0.0062457878
## [4716] 0.0062457878 0.0062457878 0.0205296042 0.0205296042 0.0205296042
## [4721] 0.0205296042 0.1022406544 0.0179160222 0.0179160222 0.0179160222
## [4726] 0.0179160222 0.0063191384 0.2922479110 0.0058886400 0.0109424730
## [4731] 0.0109424730 0.0109424730 0.0109424730 0.0109424730 0.0223116926
## [4736] 0.1578857728 0.0812797360 0.0812797360 0.0525673237 0.1709068748
## [4741] 0.1317525726 0.0295040411 0.0295040411 0.0031999863 0.2210422824
## [4746] 0.2210422824 0.0318126264 0.0318126264 0.0318126264 0.3226399009
## [4751] 0.0064028040 0.0064028040 0.0114593548 0.0114593548 0.0114593548
## [4756] 0.2807445569 0.2807445569 0.0028708150 0.0028708150 0.0028708150
## [4761] 0.0028708150 0.1571179044 0.0134478345 0.0591330738 0.0591330738
## [4766] 0.0464510922 0.1463446413 0.1463446413 0.0248745554 0.0139442936
## [4771] 0.0031552336 0.2170942822 0.0138928984 0.0129056676 0.0129056676
## [4776] 0.1893123035 0.0041000565 0.0041000565 0.4186324371 0.2931974531
## [4781] 0.2931974531 0.0010971887 0.1257703764 0.2069180947 0.2069180947
## [4786] 0.0059949277 0.0049388677 0.0198092497 0.0198092497 0.0066065529
## [4791] 0.2722009896 0.2722009896 0.4319039633 0.0840609064 0.0099129655
## [4796] 0.0044410194 0.1332630874 0.2311063276 0.2311063276 0.2735433974
## [4801] 0.1390545178 0.1390545178 0.0928593877 0.0928593877 0.0928593877
## [4806] 0.0016987917 0.0016987917 0.0016987917 0.2788215561 0.0018562599
## [4811] 0.0018562599 0.0098185951 0.0609309272 0.0609309272 0.0330729761
## [4816] 0.0330729761 0.0330729761 0.0330729761 0.0947005617 0.0456525866
## [4821] 0.0456525866 0.1091323697 0.0067215635 0.0208157492 0.0180598997
## [4826] 0.0492987330 0.0492987330 0.0492987330 0.0492987330 0.2151380217
## [4831] 0.0585295328 0.1971867519 0.2066030304 0.0738768049 0.0046820174
## [4836] 0.0046820174 0.0046820174 0.0083925017 0.0011276952 0.0477035176
## [4841] 0.1290699394 0.0422279242 0.5308892831 0.2328686688 0.0252645910
## [4846] 0.4052943831 0.0159428626 0.1712725470 0.1561185948 0.0238945966
## [4851] 0.0238945966 0.0238945966 0.0238945966 0.0238945966 0.4501302773
## [4856] 0.3970814320 0.2313492348 0.0198353931 0.0151396950 0.0058816213
## [4861] 0.2015614746 0.0514867425 0.0251594298 0.0838965851 0.0082851112
## [4866] 0.0648580322 0.2194222977 0.0229726998 0.0313545240 0.0593529885
## [4871] 0.0313545240 0.0015879595 0.0056900776 0.0064117547 0.0678420849
## [4876] 0.0678420849 0.0399670408 0.0060057846 0.0060057846 0.0060057846
## [4881] 0.0060057846 0.0295655145 0.0295655145 0.2728278937 0.5684461030
## [4886] 0.3882865017 0.0565241848 0.1192197113 0.0334728239 0.0073213721
## [4891] 0.0038499470 0.0038499470 0.0038499470 0.0038499470 0.0038499470
## [4896] 0.0033387471 0.0417770892 0.0417770892 0.4885614086 0.3080640625
## [4901] 0.0554360984 0.0699133325 0.1667488899 0.0225296045 0.0126076284
## [4906] 0.0126076284 0.0126076284 0.0222643052 0.0222643052 0.0222643052
## [4911] 0.0222643052 0.0233211841 0.0233211841 0.0233211841 0.0899933388
## [4916] 0.0147935635 0.1063163724 0.2819975709 0.0085752447 0.0048719208
## [4921] 0.0571283914 0.0571283914 0.0077427873 0.0400207924 0.0400207924
## [4926] 0.0400207924 0.0400207924 0.0400207924 0.0525881119 0.0525881119
## [4931] 0.0525881119 0.0525881119 0.0525881119 0.0059098716 0.5249364342
## [4936] 0.0718139891 0.0718139891 0.0165163525 0.0047803345 0.3569390364
## [4941] 0.0623777826 0.1999533632 0.1999533632 0.1999533632 0.0278686805
## [4946] 0.0098100070 0.0076957899 0.0076957899 0.1432240427 0.1432240427
## [4951] 0.0343528631 0.3064758920 0.3064758920 0.0303747408 0.0068964473
## [4956] 0.2986787568 0.0017667263 0.0287342032 0.1146418993 0.0039219363
## [4961] 0.0039219363 0.0203639049 0.0203639049 0.0286066173 0.0163931155
## [4966] 0.0133807746 0.1815937520 0.1815937520 0.1815937520 0.1877317178
## [4971] 0.1877317178 0.1877317178 0.2666895287 0.0134838657 0.0134838657
## [4976] 0.0134838657 0.1541352408 0.0603786890 0.3394656297 0.1479167361
## [4981] 0.1479167361 0.1807005271 0.1807005271 0.0450773321 0.0450773321
## [4986] 0.0706324490 0.1071828084 0.1422892437 0.1422892437 0.1422892437
## [4991] 0.1422892437 0.0120909472 0.0120909472 0.0209420284 0.0704857162
## [4996] 0.0704857162 0.0704857162 0.0704857162 0.1344021953 0.1344021953
## [5001] 0.0874817815 0.0045861963 0.0247804819 0.1003699603 0.1003699603
## [5006] 0.1003699603 0.1569374443 0.1569374443 0.1569374443 0.2632540257
## [5011] 0.2632540257 0.0115091549 0.1234493228 0.0097268704 0.0110703705
## [5016] 0.0042093040 0.0042093040 0.0846311157 0.2078349416 0.2078349416
## [5021] 0.2078349416 0.2078349416 0.2078349416 0.0509032637 0.0256306167
## [5026] 0.0256306167 0.0029212175 0.2703408713 0.0268103488 0.0070017234
## [5031] 0.0070017234 0.2414150072 0.0308372241 0.0193888471 0.0193888471
## [5036] 0.0193888471 0.0193888471 0.0174476022 0.0174185119 0.0174185119
## [5041] 0.0174185119 0.0481289089 0.0481289089 0.0481289089 0.0481289089
## [5046] 0.0043053433 0.1413366139 0.0955835301 0.0955835301 0.0955835301
## [5051] 0.0955835301 0.1094122296 0.1094122296 0.1094122296 0.1078189076
## [5056] 0.1348967322 0.0130760671 0.2373661775 0.1420236051 0.4085542563
## [5061] 0.6020993856 0.0064717092 0.1082027431 0.2112321345 0.0104831353
## [5066] 0.0020638942 0.0089480428 0.0089480428 0.1615432145 0.2317643506
## [5071] 0.0793511505 0.0793511505 0.0793511505 0.0793511505 0.0793511505
## [5076] 0.0793511505 0.0541816035 0.0245045231 0.0245045231 0.0266794208
## [5081] 0.0026412884 0.0026412884 0.0026412884 0.0026412884 0.0661772890
## [5086] 0.0296142280 0.2402823107 0.0696583789 0.2075221143 0.0201679281
## [5091] 0.0125240134 0.1651680160 0.0068261249 0.0068261249 0.0270170975
## [5096] 0.0036712018 0.0483276785 0.1210422067 0.0048466567 0.0728888328
## [5101] 0.2954853271 0.1762693311 0.0297416195 0.0297416195 0.0297416195
## [5106] 0.0120036039 0.0120036039 0.0120036039 0.1027352592 0.1361421051
## [5111] 0.0205473362 0.1538913671 0.2590321353 0.0673233709 0.0673233709
## [5116] 0.0781122671 0.0781122671 0.0034830469 0.0034830469 0.0034830469
## [5121] 0.0082069658 0.0082069658 0.2318819723 0.0963425009 0.0226470056
## [5126] 0.0442706531 0.0442706531 0.0442706531 0.0442706531 0.0028198999
## [5131] 0.0209702992 0.0146865093 0.1017125266 0.1017125266 0.0115879326
## [5136] 0.0115879326 0.0955864879 0.0955864879 0.0324396074 0.0363549922
## [5141] 0.1048837999 0.0864256299 0.0864256299 0.0864256299 0.0603337836
## [5146] 0.0603337836 0.0067496870 0.2506579096 0.1155834089 0.1155834089
## [5151] 0.0266973818 0.0266973818 0.2167619818 0.0263437962 0.0263437962
## [5156] 0.0068857118 0.0068857118 0.0109608865 0.0109608865 0.0109608865
## [5161] 0.0406151460 0.0218783659 0.0218783659 0.0218783659 0.0270406553
## [5166] 0.1623517539 0.1623517539 0.1623517539 0.1623517539 0.0060614760
## [5171] 0.3554995622 0.3554995622 0.0523647388 0.0103430964 0.0962923346
## [5176] 0.0962923346 0.0962923346 0.1132489816 0.1561060055 0.0368647417
## [5181] 0.2350241244 0.2350241244 0.2812435201 0.0051543836 0.0154691272
## [5186] 0.1147012816 0.2276255403 0.0891473784 0.2462032297 0.2891734843
## [5191] 0.3319608135 0.3319608135 0.0080802190 0.1100324836 0.0364551218
## [5196] 0.0364551218 0.0364551218 0.0364551218 0.0364551218 0.0364551218
## [5201] 0.1115666528 0.0014462073 0.0723473501 0.2060727852 0.2060727852
## [5206] 0.2060727852 0.2060727852 0.2416687921 0.0223115686 0.0660228915
## [5211] 0.0681227986 0.0102271742 0.0069913861 0.5475586533 0.5475586533
## [5216] 0.5475586533 0.0610547315 0.0610547315 0.0610547315 0.0354896975
## [5221] 0.1687570463 0.0705045197 0.0040556009 0.3065497438 0.0032931812
## [5226] 0.0014470140 0.4993407764 0.0102807729 0.0018412661 0.1719094085
## [5231] 0.0395105129 0.0395105129 0.0395105129 0.1042994163 0.1042994163
## [5236] 0.1042994163 0.1042994163 0.2896374424 0.0907563842 0.0907563842
## [5241] 0.0907563842 0.0907563842 0.0907563842 0.1856779020 0.1432389749
## [5246] 0.1432389749 0.2166437982 0.2166437982 0.2166437982 0.0585113733
## [5251] 0.0585113733 0.0027810779 0.0027810779 0.0027810779 0.0027810779
## [5256] 0.0592917618 0.0592917618 0.1649365795 0.1649365795 0.1019574926
## [5261] 0.1019574926 0.1019574926 0.1019574926 0.0154973003 0.0105666483
## [5266] 0.0105666483 0.0105666483 0.0105666483 0.0105666483 0.0115921338
## [5271] 0.0526261455 0.0481305440 0.0122508684 0.0122508684 0.0369158349
## [5276] 0.0369158349 0.3643175746 0.0140167555 0.0140167555 0.2914226542
## [5281] 0.0670159030 0.0138370734 0.0138370734 0.0138370734 0.0138370734
## [5286] 0.0997962518 0.0994232035 0.0994232035 0.0374154316 0.0374154316
## [5291] 0.2951527254 0.0896640568 0.0896640568 0.0809339448 0.0244156981
## [5296] 0.0244156981 0.1864336130 0.0023846102 0.1378665854 0.1378665854
## [5301] 0.1378665854 0.2830254982 0.2421236643 0.0030953466 0.0030953466
## [5306] 0.0030953466 0.0030953466 0.0225466590 0.3002547344 0.0361557241
## [5311] 0.0361557241 0.0361557241 0.0747249987 0.0033804634 0.0143363930
## [5316] 0.0143363930 0.2911918707 0.0740122736 0.3454879021 0.0663412600
## [5321] 0.0411215447 0.0685438616 0.0739167324 0.0739167324 0.0739167324
## [5326] 0.3754951256 0.4700935117 0.3024072547 0.3662605238 0.2016832302
## [5331] 0.0133899773 0.0198891149 0.1745573444 0.0122298992 0.0132243078
## [5336] 0.0046010279 0.1705079771 0.1904500646 0.1904500646 0.0164984685
## [5341] 0.0186041753 0.2785552867 0.2785552867 0.2785552867 0.0389320509
## [5346] 0.0071953815 0.0071953815 0.0071953815 0.0071953815 0.0372831115
## [5351] 0.0372831115 0.0038823082 0.0321406022 0.0321406022 0.0321406022
## [5356] 0.3966443057 0.0336715145 0.0052167119 0.5315307452 0.5315307452
## [5361] 0.0047781558 0.3125424686 0.3125424686 0.0542036483 0.3162003746
## [5366] 0.0327161307 0.0164350263 0.0173152273 0.0034893065 0.0034893065
## [5371] 0.0034893065 0.2469235975 0.6591103545 0.6591103545 0.6591103545
## [5376] 0.0284588132 0.0069540675 0.1158369383 0.1158369383 0.1158369383
## [5381] 0.0279842804 0.0012740160 0.0998793473 0.0998793473 0.0137087110
## [5386] 0.0334918635 0.1906392520 0.1906392520 0.1833310821 0.1833310821
## [5391] 0.0569395321 0.0569395321 0.1408896733 0.1408896733 0.0957150185
## [5396] 0.0347799346 0.0442867503 0.0028158226 0.0028158226 0.0031753631
## [5401] 0.0031753631 0.3812845173 0.3812845173 0.3812845173 0.3812845173
## [5406] 0.3812845173 0.1855744209 0.0719820562 0.0018725590 0.0018725590
## [5411] 0.0018725590 0.0018725590 0.0565019335 0.5385385099 0.5385385099
## [5416] 0.5385385099 0.1754137223 0.0613169040 0.0613169040 0.0164528280
## [5421] 0.0164528280 0.0096030149 0.0978049628 0.1305346886 0.1305346886
## [5426] 0.0352830953 0.0352830953 0.0352830953 0.2796181981 0.2544955209
## [5431] 0.7593649923 0.0060286670 0.0060286670 0.0060286670 0.1873629381
## [5436] 0.1873629381 0.1037541923 0.1037541923 0.2275437396 0.0077291563
## [5441] 0.0312489935 0.0312489935 0.0135240259 0.0135240259 0.0090337418
## [5446] 0.0157036582 0.0157036582 0.0157036582 0.0127878045 0.1138333955
## [5451] 0.0209895345 0.0209895345 0.1361462757 0.1361462757 0.0252807708
## [5456] 0.3539969019 0.0196582175 0.0196582175 0.0174255280 0.0174255280
## [5461] 0.0174255280 0.0174255280 0.0174255280 0.0362533022 0.0362533022
## [5466] 0.0630215789 0.0179918357 0.0073814732 0.0141206260 0.0141206260
## [5471] 0.0063018479 0.0063018479 0.0542488954 0.0542488954 0.0542488954
## [5476] 0.0112506380 0.0084107946 0.0026423494 0.0738256881 0.1143565177
## [5481] 0.0242504311 0.0242504311 0.0057686095 0.0853045987 0.0853045987
## [5486] 0.0225155892 0.1378780664 0.1347611552 0.0048046252 0.0048046252
## [5491] 0.0114979063 0.0130802555 0.2191388276 0.0051335473 0.0061906677
## [5496] 0.0061906677 0.0273638548 0.2470551647 0.2979227760 0.2260012898
## [5501] 0.2260012898 0.0128779744 0.4424731999 0.0609278501 0.0931341958
## [5506] 0.0618299865 0.0618299865 0.0448076257 0.4160042226 0.4160042226
## [5511] 0.0146112257 0.0074051981 0.0501841414 0.0501841414 0.0213334436
## [5516] 0.3614159900 0.0064877259 0.2572427067 0.0289577173 0.0289577173
## [5521] 0.0289577173 0.0289577173 0.0289577173 0.0289577173 0.0773488549
## [5526] 0.0557004203 0.3437740592 0.1848392846 0.0396275142 0.0396275142
## [5531] 0.0935891914 0.1323574631 0.4220899010 0.2154282077 0.5646300142
## [5536] 0.0127692228 0.0127692228 0.2659474254 0.2659474254 0.2021827586
## [5541] 0.1573138060 0.3103188956 0.6743198371 0.0082057316 0.0080640326
## [5546] 0.0080640326 0.5061254893 0.0585741911 0.2383586286 0.0484365952
## [5551] 0.0484365952 0.0666840077 0.0666840077 0.0666840077 0.1070530545
## [5556] 0.1070530545 0.1070530545 0.0166998427 0.0166998427 0.0926599147
## [5561] 0.3604969862 0.3604969862 0.1741236565 0.1280757350 0.0050790296
## [5566] 0.0050790296 0.3258643476 0.0487783884 0.0356588094 0.0811998691
## [5571] 0.1184319835 0.0163486255 0.0263810887 0.0578109003 0.0578109003
## [5576] 0.0578109003 0.0530011817 0.1702314412 0.1702314412 0.1702314412
## [5581] 0.1702314412 0.1702314412 0.2713666225 0.2713666225 0.0675890259
## [5586] 0.1206474564 0.1206474564 0.0592280476 0.0592280476 0.0262687487
## [5591] 0.0262687487 0.0262687487 0.1367728904 0.0210315953 0.0210315953
## [5596] 0.0210315953 0.2378403486 0.2375118074 0.2429454598 0.0191631054
## [5601] 0.2743806907 0.2743806907 0.4544148866 0.1029334369 0.1029334369
## [5606] 0.1029334369 0.1029334369 0.1029334369 0.0686839763 0.0686839763
## [5611] 0.0686839763 0.2314610924 0.1887077046 0.1812206264 0.0941441851
## [5616] 0.0714468155 0.4086310843 0.4086310843 0.4086310843 0.0096772059
## [5621] 0.0096772059 0.2114244963 0.1341244395 0.1341244395 0.0391082612
## [5626] 0.0103400168 0.0103400168 0.0067209553 0.0197085241 0.0197085241
## [5631] 0.0197085241 0.2159576306 0.0037386912 0.0037386912 0.3322720678
## [5636] 0.3255842995 0.2275539430 0.1461446926 0.0368369013 0.1320220321
## [5641] 0.0815231798 0.0412041117 0.0216127472 0.0096702584 0.0096702584
## [5646] 0.1272854984 0.1272854984 0.1272854984 0.2172670924 0.2172670924
## [5651] 0.2172670924 0.2740779317 0.0171163931 0.0043238759 0.0043238759
## [5656] 0.0043238759 0.0247748850 0.0770710938 0.3235054215 0.3235054215
## [5661] 0.4331549397 0.4331549397 0.0702401553 0.0156589253 0.0156589253
## [5666] 0.3862562260 0.3013385240 0.0317730977 0.0135918455 0.0504932712
## [5671] 0.0410069000 0.3205394915 0.3205394915 0.0458626990 0.3401939720
## [5676] 0.2617855435 0.7421049020 0.7421049020 0.0027021797 0.1184458558
## [5681] 0.0054171722 0.0023388185 0.0023388185 0.0023388185 0.3405404784
## [5686] 0.0086713989 0.0282808142 0.0585303212 0.0111232060 0.0172367204
## [5691] 0.1245264400 0.0020504892 0.0020504892 0.0315336070 0.0315336070
## [5696] 0.0315336070 0.0315336070 0.0315336070 0.0463916928 0.0308261635
## [5701] 0.0626355625 0.0626355625 0.0239626579 0.0239626579 0.1217165917
## [5706] 0.1217165917 0.3664388407 0.3664388407 0.1455995430 0.4077945162
## [5711] 0.4077945162 0.2541027845 0.0103149305 0.0103149305 0.0022421099
## [5716] 0.0022421099 0.1363718678 0.0100106658 0.0100106658 0.0100106658
## [5721] 0.0428442854 0.0428442854 0.1697701651 0.1697701651 0.1697701651
## [5726] 0.1697701651 0.0328765716 0.0328765716 0.0328765716 0.2091616815
## [5731] 0.0928155442 0.1004735309 0.0136252821 0.0136252821 0.2109620183
## [5736] 0.0263649050 0.0096981879 0.0130536393 0.0130536393 0.5645075863
## [5741] 0.0290596699 0.4367873454 0.0060017158 0.0298241567 0.0298241567
## [5746] 0.1653543228 0.0991130414 0.0192705969 0.0821727205 0.0821727205
## [5751] 0.2193015875 0.0221336723 0.6064156191 0.0793025524 0.0049547189
## [5756] 0.0917424567 0.0777466609 0.0398613272 0.1142194341 0.1142194341
## [5761] 0.0612438370 0.0612438370 0.0612438370 0.0206419611 0.0206419611
## [5766] 0.0206419611 0.0273574909 0.0273574909 0.0273574909 0.0015146294
## [5771] 0.0015146294 0.0034101549 0.0040459737 0.0040459737 0.5034040472
## [5776] 0.5034040472 0.4256662839 0.0108208010 0.0198172576 0.4042173828
## [5781] 0.1719288212 0.1719288212 0.2396040769 0.0103712064 0.0103712064
## [5786] 0.0112680495 0.0040749208 0.0085472672 0.1487932530 0.1487932530
## [5791] 0.1487932530 0.1487932530 0.1487932530 0.1487932530 0.0072426111
## [5796] 0.0037054563 0.0037054563 0.2612219566 0.0977160100 0.0856509316
## [5801] 0.0977160100 0.0856509316 0.0132868650 0.0167239832 0.0013840608
## [5806] 0.0013840608 0.0800739183 0.0800739183 0.0800739183 0.0763784931
## [5811] 0.0159105706 0.0110083648 0.1525970075 0.0090418936 0.0722643233
## [5816] 0.1142617184 0.1407682160 0.0581594755 0.1023456563 0.0399971086
## [5821] 0.1721248629 0.1721248629 0.0616193875 0.1577924576 0.1037984588
## [5826] 0.0201296053 0.0201296053 0.2503820653 0.2503820653 0.1438228669
## [5831] 0.6941475682 0.1785512673 0.1785512673 0.0333968801 0.0026078265
## [5836] 0.0026078265 0.0320871494 0.0320871494 0.0320871494 0.1785838189
## [5841] 0.0088033672 0.0691079531 0.0691079531 0.3535364915 0.3535364915
## [5846] 0.0397027974 0.0955485348 0.2118028129 0.0304527236 0.0304527236
## [5851] 0.0013388890 0.1858039663 0.0556296216 0.0202670010 0.1296602623
## [5856] 0.1296602623 0.0816743546 0.0816743546 0.0816743546 0.0816743546
## [5861] 0.5724898631 0.0666756774 0.0666756774 0.0840491529 0.0840491529
## [5866] 0.0840491529 0.0150011346 0.0150011346 0.0150011346 0.0150011346
## [5871] 0.0051338624 0.0098083418 0.1345395502 0.1345395502 0.1345395502
## [5876] 0.0275591071 0.4161318820 0.1788722055 0.1788722055 0.1788722055
## [5881] 0.2455547952 0.2396323660 0.2396323660 0.2396323660 0.2396323660
## [5886] 0.2396323660 0.0286288629 0.3582702892 0.0056870553 0.2963796760
## [5891] 0.0888451528 0.0261415388 0.0407153281 0.0037567325 0.0034995358
## [5896] 0.2279153562 0.2279153562 0.2279153562 0.0097573998 0.0405078500
## [5901] 0.1787518712 0.1053753358 0.0691218285 0.4229871084 0.0369996522
## [5906] 0.2938093620 0.2129781056 0.2305032996 0.0385814062 0.0815415095
## [5911] 0.1247204807 0.0122230604 0.0122230604 0.0122230604 0.0092912016
## [5916] 0.0050899389 0.0050899389 0.0050899389 0.0050899389 0.0452088771
## [5921] 0.0452088771 0.0279238237 0.0279238237 0.0279238237 0.0279238237
## [5926] 0.0279238237 0.2734811391 0.2734811391 0.2734811391 0.0122512964
## [5931] 0.0103604374 0.0734805047 0.1464474283 0.0499319763 0.0499319763
## [5936] 0.0326268636 0.0326268636 0.0326268636 0.0068243794 0.0068243794
## [5941] 0.0022482557 0.0022482557 0.0245688074 0.0245688074 0.0245688074
## [5946] 0.3775999761 0.4423630245 0.4423630245 0.0202307002 0.0074568294
## [5951] 0.0074568294 0.0074568294 0.2513959767 0.0630157776 0.0630157776
## [5956] 0.0630157776 0.0630157776 0.0630157776 0.0630157776 0.0638865746
## [5961] 0.0638865746 0.0638865746 0.1984997150 0.1984997150 0.1984997150
## [5966] 0.0700106268 0.0373689624 0.0970788246 0.0366488915 0.0108720190
## [5971] 0.0108720190 0.0108720190 0.5344759568 0.5344759568 0.0506742994
## [5976] 0.0012184100 0.2704544428 0.1878505311 0.1878505311 0.1878505311
## [5981] 0.1878505311 0.1312741507 0.2688384745 0.0330591541 0.0191426483
## [5986] 0.0292761284 0.0972878038 0.0192000114 0.0192000114 0.0445695023
## [5991] 0.1816003609 0.1816003609 0.0716401745 0.3689712924 0.3689712924
## [5996] 0.0184297686 0.1270941296 0.2033839345 0.2298964882 0.1782521690
## [6001] 0.1209187006 0.1209187006 0.1209187006 0.1209187006 0.1209187006
## [6006] 0.2984907871 0.0079626729 0.0782800449 0.0782800449 0.0071573662
## [6011] 0.0071573662 0.2743738685 0.0692544632 0.0692544632 0.0086324969
## [6016] 0.0086324969 0.1150058887 0.1150058887 0.0327017476 0.1201985144
## [6021] 0.1201985144 0.1201985144 0.0031645177 0.0031645177 0.0045288318
## [6026] 0.0045288318 0.2626161786 0.0727516977 0.0727516977 0.1242841528
## [6031] 0.1242841528 0.1823248256 0.1308920679 0.1308920679 0.1308920679
## [6036] 0.1308920679 0.1308920679 0.0049384929 0.1630909563 0.0597640977
## [6041] 0.0070149286 0.0070149286 0.0070149286 0.0070149286 0.0044185594
## [6046] 0.4756964022 0.4756964022 0.4756964022 0.4756964022 0.4756964022
## [6051] 0.1031366941 0.0395992169 0.0752262818 0.0752262818 0.0752262818
## [6056] 0.4827775423 0.0027444738 0.0027444738 0.0027444738 0.0027444738
## [6061] 0.3370415575 0.0657182492 0.0504319502 0.2403400526 0.1872325185
## [6066] 0.0129997718 0.0129997718 0.0129997718 0.0129997718 0.0129997718
## [6071] 0.0619789371 0.0619789371 0.0619789371 0.0619789371 0.0619789371
## [6076] 0.0619789371 0.0065353531 0.0042891376 0.0134351538 0.0134351538
## [6081] 0.0050767319 0.0020271408 0.1907780427 0.1907780427 0.1907780427
## [6086] 0.0657856228 0.0715088285 0.5788938440 0.1134487939 0.5936484385
## [6091] 0.1173387130 0.1415441210 0.1415441210 0.0686520231 0.2182789729
## [6096] 0.2182789729 0.2182789729 0.2185753933 0.0119484489 0.0119484489
## [6101] 0.0641544606 0.0168073349 0.0604322507 0.0604322507 0.0604322507
## [6106] 0.0604322507 0.0136760774 0.0136760774 0.0305172819 0.1253684890
## [6111] 0.1253684890 0.1253684890 0.1318649375 0.1318649375 0.1841346934
## [6116] 0.0191703486 0.0748935033 0.0748935033 0.0713503250 0.0252819490
## [6121] 0.0252819490 0.0252819490 0.0065263601 0.0065263601 0.0065263601
## [6126] 0.0065263601 0.2600735293 0.0099644158 0.1696245030 0.0246420102
## [6131] 0.0246420102 0.1117828776 0.1818202518 0.1418277224 0.0429590305
## [6136] 0.2519544154 0.0067751607 0.0166415241 0.0718693775 0.0087631487
## [6141] 0.0087631487 0.0569056495 0.0569056495 0.0569056495 0.0569056495
## [6146] 0.3088472534 0.0075264974 0.0517163324 0.0517163324 0.0517163324
## [6151] 0.0517163324 0.0517163324 0.0822531350 0.1020214621 0.0590801554
## [6156] 0.1371410993 0.1371410993 0.1371410993 0.0277871907 0.1199997519
## [6161] 0.1199997519 0.4854047783 0.0495463568 0.0488830973 0.0488830973
## [6166] 0.0829569811 0.3427290332 0.0142320187 0.1990209090 0.1990209090
## [6171] 0.6208565803 0.0033947901 0.0033947901 0.0226235816 0.0216737111
## [6176] 0.0216737111 0.2982717205 0.0008283317 0.0008283317 0.0008283317
## [6181] 0.1150306313 0.0060602463 0.0867841661 0.0113653832 0.1676721886
## [6186] 0.1676721886 0.1695564748 0.0359785987 0.0398579557 0.0398579557
## [6191] 0.0893381377 0.0893381377 0.0174301955 0.1712084785 0.1726055596
## [6196] 0.1726055596 0.0703973349 0.4014270779 0.1156271741 0.1156271741
## [6201] 0.1156271741 0.1156271741 0.0096184178 0.0044359726 0.1181031656
## [6206] 0.4902546943 0.1421011582 0.1833465485 0.0129505327 0.0057872824
## [6211] 0.0057872824 0.3163000057 0.0537493449 0.0032586364 0.1388616123
## [6216] 0.1186886251 0.0259099539 0.0259099539 0.0259099539 0.0259099539
## [6221] 0.0580497248 0.0580497248 0.1804209249 0.0076586967 0.0068868438
## [6226] 0.1081739155 0.1081739155 0.1081739155 0.1081739155 0.2398541514
## [6231] 0.2398541514 0.0021378654 0.0289432181 0.1061029598 0.0656544260
## [6236] 0.1579651529 0.2180641143 0.4797082483 0.0293338606 0.0293338606
## [6241] 0.0735692588 0.0656330825 0.0044634140 0.5050630205 0.7452360857
## [6246] 0.0038364359 0.0043043291 0.0043043291 0.2566485177 0.2566485177
## [6251] 0.2566485177 0.2566485177 0.0185235969 0.1650722467 0.1650722467
## [6256] 0.1650722467 0.1650722467 0.0257523518 0.0019624318 0.0019624318
## [6261] 0.0250977016 0.4119602671 0.0095984401 0.1743677272 0.0249153765
## [6266] 0.0249153765 0.0912434936 0.1156201343 0.1156201343 0.2340360838
## [6271] 0.2340360838 0.0720708642 0.1097248142 0.2867671519 0.0066561142
## [6276] 0.1704899400 0.3878939951 0.2297911574 0.1068532257 0.1068532257
## [6281] 0.1068532257 0.1068532257 0.0343534594 0.1889223389 0.1749066114
## [6286] 0.0824543636 0.0824543636 0.0827984186 0.0132415380 0.0133837316
## [6291] 0.0133837316 0.2786739449 0.2786739449 0.2603427539 0.2603427539
## [6296] 0.1939409908 0.1939409908 0.0957855612 0.1552125391 0.0570208292
## [6301] 0.0341360812 0.0570208292 0.0376550507 0.0376550507 0.0376550507
## [6306] 0.0376550507 0.2848596139 0.0397054289 0.1463303284 0.1463303284
## [6311] 0.1909914317 0.0059125680 0.0059125680 0.0073886506 0.0073886506
## [6316] 0.0544406360 0.0544406360 0.0544406360 0.0544406360 0.0467969122
## [6321] 0.1661305510 0.0294265675 0.1652491181 0.0028611686 0.0028611686
## [6326] 0.0028611686 0.1449082335 0.0032758446 0.3961499435 0.3961499435
## [6331] 0.3961499435 0.3961499435 0.0039114015 0.1150162481 0.0837435580
## [6336] 0.0837435580 0.0837435580 0.0837435580 0.0837435580 0.0043835598
## [6341] 0.0188025977 0.0063228856 0.0070903926 0.0070903926 0.2328624837
## [6346] 0.2328624837 0.0199729572 0.0199729572 0.0199729572 0.0199729572
## [6351] 0.2424971335 0.0093812502 0.0067576041 0.3100068737 0.0494662695
## [6356] 0.0220513409 0.1286246707 0.0057177931 0.0100068939 0.0100068939
## [6361] 0.0020364800 0.1000720813 0.1000720813 0.1000720813 0.0546648085
## [6366] 0.0045616737 0.0045616737 0.2965446802 0.2965446802 0.2965446802
## [6371] 0.2965446802 0.2965446802 0.2965446802 0.2965446802 0.0792231344
## [6376] 0.0792231344 0.1125631631 0.1125631631 0.1125631631 0.1125631631
## [6381] 0.1125631631 0.0128127789 0.0128127789 0.0026191028 0.0022685238
## [6386] 0.0028672813 0.0022685238 0.0022685238 0.0022685238 0.3403046701
## [6391] 0.0776183551 0.0138264726 0.0138264726 0.0916188545 0.0354513523
## [6396] 0.0204269759 0.5614555724 0.0829112128 0.0829112128 0.0829112128
## [6401] 0.8385813406 0.1089119849 0.1089119849 0.6314219116 0.0374758910
## [6406] 0.0374758910 0.0374758910 0.0340373946 0.1489650038 0.0869900036
## [6411] 0.2130510768 0.2130510768 0.0061183702 0.2909984295 0.0115717089
## [6416] 0.0115717089 0.0115717089 0.0115717089 0.0211175848 0.0211175848
## [6421] 0.0080292216 0.0080292216 0.0080292216 0.4744236010 0.4744236010
## [6426] 0.0057435258 0.1330320580 0.0098328952 0.0543111276 0.0543111276
## [6431] 0.0543111276 0.0803918899 0.0061896108 0.0061896108 0.0061896108
## [6436] 0.0061896108 0.0944239897 0.0192042355 0.0442119351 0.0338976655
## [6441] 0.0338976655 0.0338976655 0.0928164500 0.0928164500 0.7084380490
## [6446] 0.7084380490 0.1251794795 0.0581084893 0.0019074001 0.0019074001
## [6451] 0.0889510056 0.2567076649 0.5614730313 0.2372154331 0.1277196342
## [6456] 0.0155293033 0.0155293033 0.0155293033 0.0155293033 0.0155293033
## [6461] 0.0115329260 0.1092979431 0.1092979431 0.1092979431 0.0560462019
## [6466] 0.0104420437 0.0860413552 0.0860413552 0.3639278489 0.5780041544
## [6471] 0.7162746774 0.6996912799 0.6996912799 0.6996912799 0.0085332732
## [6476] 0.0085332732 0.0085332732 0.1820858368 0.0118934545 0.2352228493
## [6481] 0.0174168845 0.2074515074 0.0093303115 0.0320802872 0.0320802872
## [6486] 0.0065601274 0.0104430097 0.0104430097 0.0082405119 0.0371016140
## [6491] 0.0371016140 0.0371016140 0.0371016140 0.0371016140 0.0331250295
## [6496] 0.0023787506 0.0023787506 0.0053123293 0.0130642880 0.0130642880
## [6501] 0.0051022458 0.0051022458 0.5559451516 0.0644173905 0.0644173905
## [6506] 0.0644173905 0.0720313240 0.4275611287 0.0040221664 0.0069649239
## [6511] 0.0584518266 0.0584518266 0.0584518266 0.1102751350 0.0192350655
## [6516] 0.0671987364 0.1025227352 0.0918663574 0.2380322636 0.0055751300
## [6521] 0.6878040731 0.0845973730 0.0184732088 0.0184732088 0.0184732088
## [6526] 0.0068270339 0.0089478698 0.1967877792 0.0333157765 0.0333157765
## [6531] 0.0333157765 0.0333157765 0.3418870781 0.0087524842 0.0087524842
## [6536] 0.0087524842 0.0360802068 0.0360802068 0.4050593384 0.0884887616
## [6541] 0.0884887616 0.0074133292 0.0074133292 0.0587844316 0.0494441346
## [6546] 0.1479922706 0.0260093549 0.0260093549 0.0260093549 0.1299198046
## [6551] 0.0465244947 0.0531842588 0.0191438488 0.0191438488 0.0191438488
## [6556] 0.4871542540 0.1987634151 0.1003789221 0.0619717753 0.0519247019
## [6561] 0.0519247019 0.0519247019 0.0065659936 0.0065659936 0.0065659936
## [6566] 0.0553494228 0.0610772982 0.0610772982 0.1039193600 0.1039193600
## [6571] 0.1039193600 0.1039193600 0.1039193600 0.0182447644 0.1236429825
## [6576] 0.1236429825 0.1236429825 0.0738090609 0.0738090609 0.0836755146
## [6581] 0.0390675322 0.0761535197 0.0761535197 0.0761535197 0.0055404938
## [6586] 0.0055404938 0.0055404938 0.0034422081 0.0035151899 0.0035151899
## [6591] 0.0016961614 0.0016961614 0.1179074522 0.0379694070 0.0379694070
## [6596] 0.0739597944 0.2050302717 0.0311606161 0.0437852300 0.0437852300
## [6601] 0.0437852300 0.0085735649 0.0085735649 0.4531253258 0.0327545387
## [6606] 0.1258198003 0.0275619271 0.0275619271 0.0201142034 0.0201142034
## [6611] 0.0201142034 0.0021025703 0.0838187427 0.0103048666 0.0375323119
## [6616] 0.0628794140 0.0628794140 0.0628794140 0.0067887673 0.0265718697
## [6621] 0.0265718697 0.0265718697 0.0265718697 0.1114708523 0.1114708523
## [6626] 0.0117516729 0.0166770017 0.0296227181 0.1395387935 0.2293366080
## [6631] 0.2293366080 0.0065846009 0.0065846009 0.0812024434 0.0812024434
## [6636] 0.0019539674 0.0231740031 0.1334461294 0.2501870031 0.0719908891
## [6641] 0.0825572365 0.0189071674 0.0101196051 0.0101196051 0.0101196051
## [6646] 0.1314325092 0.1314325092 0.1333696441 0.0112929909 0.0070749953
## [6651] 0.1094791800 0.0359726922 0.0359726922 0.0043501238 0.0043501238
## [6656] 0.0930473064 0.0025337946 0.0018395699 0.0025337946 0.0049657838
## [6661] 0.4641233157 0.2194339151 0.0406313449 0.0406313449 0.0406313449
## [6666] 0.4599464640 0.0310886065 0.0310886065 0.0070251397 0.0070251397
## [6671] 0.0070251397 0.0070251397 0.0070251397 0.0031647258 0.0343575240
## [6676] 0.0343575240 0.0343575240 0.0404734320 0.0404734320 0.0404734320
## [6681] 0.0404734320 0.0146173173 0.0113477637 0.0087893867 0.2040910718
## [6686] 0.0767318014 0.0563450774 0.0766587571 0.0105167090 0.0166455804
## [6691] 0.0227586968 0.0227586968 0.0227586968 0.2179894364 0.2566186678
## [6696] 0.0542676337 0.0542676337 0.0542676337 0.2314024778 0.1467307400
## [6701] 0.0779870749 0.0873538068 0.0193074749 0.0089068288 0.0126125165
## [6706] 0.0161139182 0.0141698209 0.0155280197 0.0206432879 0.0380257311
## [6711] 0.0380257311 0.1372022336 0.1372022336 0.3581677261 0.1521820292
## [6716] 0.0765966630 0.0765966630 0.0765966630 0.0765966630 0.0172356320
## [6721] 0.0054390727 0.1166285907 0.0931413863 0.0931413863 0.0931413863
## [6726] 0.0931413863 0.3010238455 0.2105251587 0.2105251587 0.0739327297
## [6731] 0.1064601695 0.0095467936 0.1932283492 0.2593181009 0.2593181009
## [6736] 0.1047204907 0.0241460564 0.0241460564 0.0231540498 0.0207776987
## [6741] 0.1070677266 0.1070677266 0.1070677266 0.0686990614 0.0686990614
## [6746] 0.0686990614 0.0686990614 0.0686990614 0.0686990614 0.0176636515
## [6751] 0.0176636515 0.0176636515 0.1229812322 0.4371840214 0.0439290636
## [6756] 0.2505866845 0.1118210852 0.0102289464 0.3705225681 0.0752788742
## [6761] 0.1618852840 0.0401851518 0.0249831529 0.0055353852 0.0613650376
## [6766] 0.0613650376 0.0613650376 0.0613650376 0.0613650376 0.0359705208
## [6771] 0.0934926909 0.0934926909 0.0160868495 0.0160868495 0.0120886199
## [6776] 0.2679955003 0.2679955003 0.1566026755 0.0091022316 0.0132100899
## [6781] 0.0080712645 0.0248789623 0.1268271358 0.1268271358 0.1268271358
## [6786] 0.0123050988 0.0046204682 0.0046204682 0.0046204682 0.0203232618
## [6791] 0.1073262292 0.1709020718 0.0085268075 0.2874609963 0.2874609963
## [6796] 0.2077796332 0.2077796332 0.2077796332 0.2374869700 0.2374869700
## [6801] 0.2374869700 0.2486681586 0.2374869700 0.0845861978 0.0845861978
## [6806] 0.1263708982 0.1990601249 0.1263708982 0.0243756285 0.0243756285
## [6811] 0.0243756285 0.0160226435 0.2032657599 0.0045170650 0.0179629989
## [6816] 0.0149548186 0.0149548186 0.1832471162 0.1832471162 0.0396216525
## [6821] 0.0396216525 0.0396216525 0.0396216525 0.0111764787 0.0111764787
## [6826] 0.0111764787 0.0111764787 0.0145162393 0.0122664448 0.0080253201
## [6831] 0.0080253201 0.0080253201 0.0019527928 0.0071551338 0.0071551338
## [6836] 0.0655252027 0.0655252027 0.2224560761 0.0368021095 0.0368021095
## [6841] 0.0368021095 0.0368021095 0.0110540891 0.0335707326 0.0335707326
## [6846] 0.0335707326 0.0445562520 0.0021581996 0.0021581996 0.3125027979
## [6851] 0.0055313884 0.0055313884 0.5572244799 0.5572244799 0.0689691371
## [6856] 0.0689691371 0.0689691371 0.1013094169 0.1013094169 0.1013094169
## [6861] 0.1783964066 0.0296396781 0.0473945219 0.0378222063 0.0378222063
## [6866] 0.0823053501 0.0823053501 0.0134792792 0.0413262837 0.0483540440
## [6871] 0.0802090038 0.0802090038 0.0802090038 0.1572651051 0.0596385873
## [6876] 0.0596385873 0.3016637882 0.0098630084 0.2564921005 0.0028490065
## [6881] 0.0028490065 0.0028490065 0.0028490065 0.0088434815 0.0088434815
## [6886] 0.0088434815 0.0088434815 0.0812990955 0.0096614468 0.0096614468
## [6891] 0.1901787513 0.1901787513 0.0012899560 0.0241447647 0.0241447647
## [6896] 0.0241447647 0.0742612378 0.1375038674 0.0533329721 0.0533329721
## [6901] 0.0032344989 0.0032344989 0.2125628661 0.0054418462 0.0482572270
## [6906] 0.0083659277 0.0083659277 0.1324943247 0.1324943247 0.1324943247
## [6911] 0.1017161564 0.1017993060 0.1017993060 0.1017993060 0.1017993060
## [6916] 0.0790566390 0.0790566390 0.0017697371 0.0017697371 0.2988275317
## [6921] 0.0548253446 0.0548253446 0.0548253446 0.0548253446 0.0090025637
## [6926] 0.0281500385 0.0234041060 0.0399769489 0.1265542882 0.1265542882
## [6931] 0.2538988129 0.0067921395 0.0067921395 0.0067921395 0.2739322087
## [6936] 0.0121200563 0.1672734456 0.0317928144 0.0499135402 0.0499135402
## [6941] 0.0499135402 0.0499135402 0.0499135402 0.0652342158 0.0652342158
## [6946] 0.0652342158 0.0944935701 0.1433532804 0.0012721875 0.0012721875
## [6951] 0.2011865123 0.0230854479 0.1429514955 0.0625109529 0.3673190358
## [6956] 0.0593484041 0.0593484041 0.0593484041 0.0068375524 0.1729470550
## [6961] 0.0717602031 0.0440616546 0.0103263937 0.0103263937 0.0199947507
## [6966] 0.0199947507 0.0938247222 0.0043611111 0.0789309069 0.0789309069
## [6971] 0.0450661428 0.0450661428 0.1845371284 0.0211530123 0.0211530123
## [6976] 0.0211530123 0.0345649059 0.0345649059 0.0363599158 0.1110786931
## [6981] 0.0065842973 0.0065842973 0.0052304883 0.0146421128 0.0533565589
## [6986] 0.0533565589 0.0154311944 0.0061491596 0.0061491596 0.0061491596
## [6991] 0.0570070087 0.0570070087 0.0570070087 0.0171271813 0.0239812003
## [6996] 0.0239812003 0.0726732968 0.0726732968 0.0726732968 0.0726732968
## [7001] 0.0726732968 0.0412669936 0.0917559530 0.0917559530 0.0917559530
## [7006] 0.0469555857 0.0079224209 0.0079224209 0.0079224209 0.0040603720
## [7011] 0.0225690321 0.0038639240 0.0634697814 0.0634697814 0.0634697814
## [7016] 0.0393329529 0.0393329529 0.0393329529 0.0393329529 0.0015354207
## [7021] 0.0015354207 0.0119241888 0.0119241888 0.0119241888 0.0013816689
## [7026] 0.0013816689 0.0949390841 0.0949390841 0.2076817939 0.2076817939
## [7031] 0.2076817939 0.2076817939 0.2076817939 0.5395823281 0.6183323086
## [7036] 0.1166791246 0.1166791246 0.2918208966 0.2918208966 0.2918208966
## [7041] 0.2918208966 0.1775174277 0.0678630680 0.0662803641 0.0557046230
## [7046] 0.2368984237 0.0786860635 0.0786860635 0.0618965143 0.0618965143
## [7051] 0.0060751009 0.1701172609 0.0139582507 0.0155906768 0.0020586961
## [7056] 0.0143835515 0.0143835515 0.1418300656 0.0997981320 0.0285411298
## [7061] 0.0285411298 0.0098282054 0.0905657957 0.0255171797 0.0179524451
## [7066] 0.0179524451 0.0176504702 0.0129908211 0.2604205507 0.0120326433
## [7071] 0.0384308485 0.0384308485 0.0384308485 0.0249430066 0.0076046684
## [7076] 0.0072686109 0.0071850078 0.0039361063 0.0039361063 0.0445214127
## [7081] 0.0445214127 0.0445214127 0.2547937520 0.0909858382 0.1864006664
## [7086] 0.0159688675 0.0509979167 0.2792484623 0.0290340934 0.0290340934
## [7091] 0.0233387889 0.2869045560 0.0067630360 0.0067630360 0.0067630360
## [7096] 0.0067630360 0.0067630360 0.0185611024 0.5278596012 0.0066151984
## [7101] 0.0013674547 0.0013674547 0.1950349413 0.1217284518 0.0143353876
## [7106] 0.0018283984 0.0173139907 0.0173139907 0.0173139907 0.2021883348
## [7111] 0.2855885035 0.0428422525 0.0428422525 0.0428422525 0.0643126311
## [7116] 0.0643126311 0.0080914213 0.0080914213 0.0024338867 0.1432023366
## [7121] 0.0820750735 0.0123165442 0.2003558492 0.3331419101 0.0057732916
## [7126] 0.0710531223 0.0564428909 0.0564428909 0.0564428909 0.0861889793
## [7131] 0.0861889793 0.0121248923 0.3216501633 0.3216501633 0.1620696726
## [7136] 0.1620696726 0.2100959968 0.2100959968 0.2100959968 0.1047662969
## [7141] 0.3215742941 0.3716589360 0.0132667423 0.1783798417 0.1783798417
## [7146] 0.1783798417 0.2259105218 0.0154680440 0.0154680440 0.1628279074
## [7151] 0.2714546766 0.7369112464 0.0073262745 0.0216407701 0.0165944686
## [7156] 0.2393650939 0.4226958299 0.1247976588 0.0220260706 0.0031230677
## [7161] 0.0133838306 0.0133838306 0.0025976153 0.0097058602 0.0932132054
## [7166] 0.0932132054 0.0105755955 0.0105755955 0.0105755955 0.0387343707
## [7171] 0.0387343707 0.0490438103 0.0273382789 0.0273382789 0.0070748723
## [7176] 0.0070748723 0.2932276043 0.4863374046 0.4863374046 0.0013282882
## [7181] 0.6608084173 0.0281084560 0.4563712844 0.0102376631 0.1506647506
## [7186] 0.1506647506 0.0239922091 0.0239922091 0.2429037449 0.2429037449
## [7191] 0.1087563862 0.1087563862 0.1384533468 0.0320592226 0.0453270288
## [7196] 0.0480501142 0.2579502027 0.2579502027 0.2270860984 0.0118686457
## [7201] 0.2364102025 0.0105204454 0.0105204454 0.1664600188 0.2442540780
## [7206] 0.2442540780 0.0167421512 0.0167421512 0.0111344031 0.0869532203
## [7211] 0.0869532203 0.0869532203 0.0869532203 0.4338771247 0.4338771247
## [7216] 0.4338771247 0.0049341286 0.0384210198 0.0049574236 0.2631225968
## [7221] 0.2112067291 0.3955287280 0.1421335212 0.1421335212 0.1421335212
## [7226] 0.1483770157 0.1483770157 0.0311688121 0.0311688121 0.0311688121
## [7231] 0.0165044699 0.1462450707 0.1462450707
## attr(,"smoother")
## [1] "lowess"
plot(
  cal_boot,
  xlab = "Predicted Probability",
  ylab = "Observed Probability"
)

## 
## n=7233   Mean absolute error=0.004   Mean squared error=5e-05
## 0.9 Quantile of absolute error=0.011

A perfectly calibrated model would lie on the 45-degree identity line.

25 24. Calibration Intercept and Calibration Slope

Calibration should preferably be evaluated on validation or resampled predictions. The following calculation on the development data is shown to explain the definitions.

model_complete$lp <- as.numeric(
  predict(
    fit_spline,
    type = "lp"
  )
)

25.1 24.1 Calibration intercept with slope fixed at 1

cal_intercept_model <- glm(
  Diabetes_num ~ 1 + offset(lp),
  data = model_complete,
  family = binomial()
)

calibration_intercept <- unname(coef(cal_intercept_model)[1])
calibration_intercept
## [1] 3.057055e-10

Ideal value:

\[ \text{Calibration intercept}=0. \]

25.2 24.2 Calibration slope

cal_slope_model <- glm(
  Diabetes_num ~ lp,
  data = model_complete,
  family = binomial()
)

calibration_slope <- unname(coef(cal_slope_model)[2])
calibration_slope
## [1] 1

Ideal value:

\[ \text{Calibration slope}=1. \]

A slope less than 1 often indicates overly extreme predictions and overfitting.

26 25. Brier Score

The Brier score is

\[ \text{Brier} = \frac{1}{n}\sum_{i=1}^{n}(\hat p_i-y_i)^2. \]

brier_model <- mean(
  (model_complete$pred_prob - model_complete$Diabetes_num)^2
)

prevalence <- mean(model_complete$Diabetes_num)

brier_null <- mean(
  (prevalence - model_complete$Diabetes_num)^2
)

brier_model
## [1] 0.07563335
brier_null
## [1] 0.09217148

A smaller Brier score indicates lower overall prediction error.

27 26. Decision-Curve Analysis

To avoid dependence on an additional package, this tutorial calculates net benefit directly.

For a threshold probability \(p_t\), net benefit is

\[ NB = \frac{TP}{N} - \frac{FP}{N}\frac{p_t}{1-p_t}. \]

27.1 26.1 Function to calculate model net benefit

net_benefit_model <- function(y, p, threshold) {
  pred_positive <- p >= threshold

  tp <- sum(pred_positive & y == 1)
  fp <- sum(pred_positive & y == 0)
  n <- length(y)

  tp / n - fp / n * threshold / (1 - threshold)
}

net_benefit_all <- function(y, threshold) {
  prevalence <- mean(y)
  prevalence - (1 - prevalence) * threshold / (1 - threshold)
}

27.2 26.2 Calculate DCA across thresholds

thresholds <- seq(0.01, 0.50, by = 0.01)

dca_results <- tibble(
  threshold = thresholds,
  model = map_dbl(
    thresholds,
    ~ net_benefit_model(
      y = model_complete$Diabetes_num,
      p = model_complete$pred_prob,
      threshold = .x
    )
  ),
  treat_all = map_dbl(
    thresholds,
    ~ net_benefit_all(
      y = model_complete$Diabetes_num,
      threshold = .x
    )
  ),
  treat_none = 0
) %>%
  pivot_longer(
    cols = c(model, treat_all, treat_none),
    names_to = "Strategy",
    values_to = "Net_Benefit"
  )

27.3 26.3 Plot decision curves

ggplot(
  dca_results,
  aes(
    x = threshold,
    y = Net_Benefit,
    linetype = Strategy
  )
) +
  geom_line(linewidth = 0.9) +
  labs(
    title = "Decision Curve Analysis",
    x = "Threshold Probability",
    y = "Net Benefit"
  ) +
  theme_minimal()

DCA is used to determine whether the model provides useful clinical net benefit over a clinically relevant threshold range. It is not simply an automatic cutoff-selection procedure.

28 27. Threshold-Based Classification Performance

Suppose, for illustration, a threshold of 20% is clinically relevant.

cutoff <- 0.20

model_complete <- model_complete %>%
  mutate(
    pred_class = if_else(pred_prob >= cutoff, 1L, 0L)
  )

confusion_matrix <- table(
  Predicted = model_complete$pred_class,
  Observed = model_complete$Diabetes_num
)

confusion_matrix
##          Observed
## Predicted    0    1
##         0 5693  342
##         1  797  401

Calculate sensitivity, specificity, PPV, and NPV:

tp <- sum(
  model_complete$pred_class == 1 &
    model_complete$Diabetes_num == 1
)

fp <- sum(
  model_complete$pred_class == 1 &
    model_complete$Diabetes_num == 0
)

tn <- sum(
  model_complete$pred_class == 0 &
    model_complete$Diabetes_num == 0
)

fn <- sum(
  model_complete$pred_class == 0 &
    model_complete$Diabetes_num == 1
)

classification_metrics <- tibble(
  Metric = c("Sensitivity", "Specificity", "PPV", "NPV"),
  Value = c(
    tp / (tp + fn),
    tn / (tn + fp),
    tp / (tp + fp),
    tn / (tn + fn)
  )
)

classification_metrics

28.1 27.1 Youden index for statistical comparison only

coords(
  roc_obj,
  x = "best",
  best.method = "youden",
  ret = c("threshold", "sensitivity", "specificity")
)

A Youden-optimal cutoff maximizes sensitivity + specificity - 1, but it does not directly incorporate clinical consequences, treatment burden, resource use, or the relative cost of false positives and false negatives.

29 28. Influential Observations

For influence diagnostics, use the ordinary logistic model.

cook <- cooks.distance(fit_full)

cook_threshold <- 4 / nrow(model_complete)

influential_indices <- which(cook > cook_threshold)

cat("Number above 4/n threshold:", length(influential_indices), "\n")
## Number above 4/n threshold: 645
plot(
  cook,
  type = "h",
  ylab = "Cook's Distance",
  main = "Influence Diagnostic"
)
abline(h = cook_threshold, lty = 2)

The 4/n rule is only a screening heuristic. Large Cook’s distance does not justify automatic deletion.

30 29. Sensitivity Analysis: Remove Highly Influential Observations

This is a sensitivity analysis, not the primary model.

model_sens <- model_complete[
  cook <= cook_threshold,
  ,
  drop = FALSE
]

fit_sens <- glm(
  Diabetes_num ~
    Age +
    BMI +
    BPSysAve +
    TotChol +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = model_sens,
  family = binomial()
)

coef_comparison <- tibble(
  Term = names(coef(fit_full)),
  Primary = coef(fit_full),
  Sensitivity = coef(fit_sens)[names(coef(fit_full))]
)

coef_comparison

31 30. Sensitivity Analysis: Complete Case versus Multiple Imputation

Complete-case model:

fit_cc <- glm(
  Diabetes_num ~
    Age +
    BMI +
    BPSysAve +
    TotChol +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = model_cc,
  family = binomial()
)

broom::tidy(
  fit_cc,
  exponentiate = TRUE,
  conf.int = TRUE
)

Compare this with the pooled MI estimates reported earlier.

32 31. Sensitivity Analysis: Linear versus Spline Functional Forms

set.seed(2026)

val_linear <- validate(
  fit_linear,
  method = "boot",
  B = 300
)

set.seed(2026)

val_spline <- validate(
  fit_spline,
  method = "boot",
  B = 300
)

val_linear
##           index.orig training    test optimism index.corrected   Lower  Upper
## Dxy           0.6395   0.6424  0.6368   0.0056          0.6338  0.6044 0.6648
## R2            0.2546   0.2579  0.2520   0.0059          0.2487  0.2204 0.2766
## Intercept     0.0000   0.0000 -0.0306   0.0306         -0.0306 -0.2267 0.1722
## Slope         1.0000   1.0000  0.9855   0.0145          0.9855  0.8995 1.0790
## Emax          0.0000   0.0000  0.0211  -0.0211          0.0211 -0.0057 0.0671
## D             0.1314   0.1337  0.1300   0.0037          0.1277  0.1098 0.1451
## U            -0.0003  -0.0003  0.0000  -0.0003          0.0000 -0.0005 0.0013
## Q             0.1317   0.1340  0.1299   0.0040          0.1276  0.1089 0.1449
## B             0.0781   0.0783  0.0784  -0.0001          0.0783  0.0739 0.0829
## g             1.4931   1.5056  1.4829   0.0227          1.4704  1.3604 1.5856
## gp            0.1157   0.1168  0.1152   0.0017          0.1140  0.1046 0.1238
##             n
## Dxy       300
## R2        300
## Intercept 300
## Slope     300
## Emax      300
## D         300
## U         300
## Q         300
## B         300
## g         300
## gp        300
val_spline
##           index.orig training    test optimism index.corrected   Lower  Upper
## Dxy           0.6657   0.6708  0.6617   0.0091          0.6566  0.6274 0.6859
## R2            0.2844   0.2901  0.2800   0.0101          0.2743  0.2443 0.3032
## Intercept     0.0000   0.0000 -0.0476   0.0476         -0.0476 -0.2305 0.1494
## Slope         1.0000   1.0000  0.9738   0.0262          0.9738  0.8776 1.0651
## Emax          0.0000   0.0000  0.0230  -0.0230          0.0230 -0.0073 0.0715
## D             0.1480   0.1518  0.1456   0.0062          0.1418  0.1224 0.1601
## U            -0.0003  -0.0003  0.0001  -0.0004          0.0001 -0.0005 0.0016
## Q             0.1483   0.1520  0.1455   0.0066          0.1417  0.1213 0.1602
## B             0.0756   0.0755  0.0760  -0.0005          0.0761  0.0719 0.0808
## g             1.8068   1.8412  1.7909   0.0503          1.7565  1.6079 1.9012
## gp            0.1225   0.1241  0.1217   0.0024          0.1201  0.1105 0.1298
##             n
## Dxy       300
## R2        300
## Intercept 300
## Slope     300
## Emax      300
## D         300
## U         300
## Q         300
## B         300
## g         300
## gp        300

If a more complex spline model provides little improvement after optimism correction, the simpler linear model may be preferable.

33 32. Subgroup Performance

Subgroup evaluation asks whether the same model performs similarly across clinically important groups. It is different from fitting a new model within each subgroup.

33.1 32.1 AUC by gender

gender_auc <- model_complete %>%
  split(.$Gender) %>%
  imap_dfr(
    function(dat, group_name) {
      if (length(unique(dat$Diabetes_num)) < 2) {
        return(
          tibble(
            Gender = group_name,
            N = nrow(dat),
            AUC = NA_real_
          )
        )
      }

      r <- roc(
        dat$Diabetes_num,
        dat$pred_prob,
        quiet = TRUE
      )

      tibble(
        Gender = group_name,
        N = nrow(dat),
        AUC = as.numeric(auc(r))
      )
    }
  )

gender_auc

33.2 32.2 Mean predicted versus observed risk by race

race_calibration <- model_complete %>%
  group_by(Race1) %>%
  summarise(
    N = n(),
    Observed = mean(Diabetes_num),
    Predicted = mean(pred_prob),
    Difference = Predicted - Observed,
    .groups = "drop"
  )

race_calibration

This is only a simple subgroup calibration summary. More complete subgroup validation would also examine calibration slopes and smooth calibration curves where sample size permits.

34 33. Temporal Validation Using Survey Cycle

The NHANES teaching dataset contains two survey cycles. We use the earlier cycle for development and the later cycle as a temporal validation sample.

Check the actual coding first:

levels(NHANES$SurveyYr)
## [1] "2009_10" "2011_12"
table(NHANES$SurveyYr, useNA = "ifany")
## 
## 2009_10 2011_12 
##    5000    5000

34.1 33.1 Create temporal-validation data

For clarity, this section uses complete cases. In a publication-level analysis, the missing-data strategy should be carefully transported to the validation sample as well.

temporal_data <- NHANES %>%
  filter(
    Age >= 20,
    Diabetes %in% c("Yes", "No")
  ) %>%
  mutate(
    Diabetes_num = if_else(Diabetes == "Yes", 1L, 0L)
  ) %>%
  select(
    SurveyYr,
    Diabetes_num,
    Age,
    BMI,
    BPSysAve,
    TotChol,
    Gender,
    Race1,
    PhysActive,
    Smoke100
  ) %>%
  drop_na()

survey_levels <- levels(droplevels(temporal_data$SurveyYr))

if (length(survey_levels) != 2) {
  stop(
    "Expected exactly two SurveyYr levels, but found: ",
    paste(survey_levels, collapse = ", ")
  )
}

survey_levels
## [1] "2009_10" "2011_12"

Use the first factor level as development and the second as validation:

development_level <- survey_levels[1]
validation_level <- survey_levels[2]

development <- temporal_data %>%
  filter(SurveyYr == development_level) %>%
  droplevels()

validation <- temporal_data %>%
  filter(SurveyYr == validation_level) %>%
  droplevels()

cat("Development cycle:", development_level, "N =", nrow(development), "\n")
## Development cycle: 2009_10 N = 3309
cat("Validation cycle:", validation_level, "N =", nrow(validation), "\n")
## Validation cycle: 2011_12 N = 3266

35 34. Develop the Model Only in the Earlier Survey Cycle

fit_dev <- glm(
  Diabetes_num ~
    Age +
    BMI +
    BPSysAve +
    TotChol +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = development,
  family = binomial()
)

summary(fit_dev)
## 
## Call:
## glm(formula = Diabetes_num ~ Age + BMI + BPSysAve + TotChol + 
##     Gender + Race1 + PhysActive + Smoke100, family = binomial(), 
##     data = development)
## 
## Coefficients:
##                 Estimate Std. Error z value Pr(>|z|)    
## (Intercept)   -6.8850902  0.6860389 -10.036  < 2e-16 ***
## Age            0.0664924  0.0047231  14.078  < 2e-16 ***
## BMI            0.0883082  0.0090216   9.789  < 2e-16 ***
## BPSysAve       0.0006185  0.0035770   0.173  0.86272    
## TotChol       -0.3032185  0.0651632  -4.653 3.27e-06 ***
## Gendermale     0.3103351  0.1300745   2.386  0.01704 *  
## Race1Hispanic  0.1373539  0.3315429   0.414  0.67866    
## Race1Mexican   0.1655300  0.2704597   0.612  0.54052    
## Race1White    -0.6308608  0.1940783  -3.251  0.00115 ** 
## Race1Other     0.4236475  0.2992126   1.416  0.15681    
## PhysActiveYes -0.0059153  0.1328445  -0.045  0.96448    
## Smoke100Yes    0.3063495  0.1285849   2.382  0.01720 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 2182.4  on 3308  degrees of freedom
## Residual deviance: 1743.7  on 3297  degrees of freedom
## AIC: 1767.7
## 
## Number of Fisher Scoring iterations: 6

36 35. Apply the Frozen Model to the Validation Cycle

The validation model must not be refitted before performance is evaluated.

validation$pred <- predict(
  fit_dev,
  newdata = validation,
  type = "response"
)

validation$lp <- predict(
  fit_dev,
  newdata = validation,
  type = "link"
)

37 36. Temporal Validation Discrimination

roc_val <- roc(
  response = validation$Diabetes_num,
  predictor = validation$pred,
  quiet = TRUE
)

validation_auc <- as.numeric(auc(roc_val))
validation_auc_ci <- ci.auc(roc_val)

validation_auc
## [1] 0.8026075
validation_auc_ci
## 95% CI: 0.7787-0.8265 (DeLong)
plot(
  roc_val,
  print.auc = TRUE,
  main = "ROC Curve: Temporal Validation"
)

38 37. Temporal Validation Calibration

38.1 37.1 Calibration-in-the-large

cal_intercept_val_model <- glm(
  Diabetes_num ~ 1 + offset(lp),
  data = validation,
  family = binomial()
)

cal_intercept_val <- unname(
  coef(cal_intercept_val_model)[1]
)

cal_intercept_val
## [1] -0.1317141

Ideal:

\[ \alpha = 0. \]

38.2 37.2 Calibration slope

cal_slope_val_model <- glm(
  Diabetes_num ~ lp,
  data = validation,
  family = binomial()
)

cal_slope_val <- unname(
  coef(cal_slope_val_model)[2]
)

cal_slope_val
## [1] 0.9214491

Ideal:

\[ \gamma = 1. \]

38.3 37.3 Decile calibration table

validation <- validation %>%
  mutate(
    risk_group = ntile(pred, 10)
  )

calibration_table <- validation %>%
  group_by(risk_group) %>%
  summarise(
    N = n(),
    Predicted = mean(pred),
    Observed = mean(Diabetes_num),
    .groups = "drop"
  )

calibration_table
ggplot(
  calibration_table,
  aes(
    x = Predicted,
    y = Observed
  )
) +
  geom_point() +
  geom_line() +
  geom_abline(
    intercept = 0,
    slope = 1,
    linetype = 2
  ) +
  coord_equal() +
  labs(
    title = "Temporal Validation Calibration",
    x = "Mean Predicted Probability",
    y = "Observed Proportion"
  ) +
  theme_minimal()

39 38. Temporal Validation Brier Score

brier_val <- mean(
  (validation$pred - validation$Diabetes_num)^2
)

brier_val
## [1] 0.07749806

40 39. Recalibration

If discrimination remains acceptable but calibration has shifted, a recalibration model can be considered.

recal_model <- glm(
  Diabetes_num ~ lp,
  data = validation,
  family = binomial()
)

coef(recal_model)
## (Intercept)          lp 
##  -0.2579499   0.9214491

The updated linear predictor is

\[ LP_{new} = \alpha + \gamma LP_{old}, \]

where \(\alpha\) is the calibration intercept and \(\gamma\) is the calibration slope.

Predictions from the recalibrated model:

validation$pred_recalibrated <- predict(
  recal_model,
  type = "response"
)

head(
  validation %>%
    select(pred, pred_recalibrated)
)

41 40. Final Logistic Model for Individual Prediction

For an easy-to-deploy risk calculator, use the ordinary logistic model fitted to the completed modeling dataset.

final_fit <- glm(
  Diabetes_num ~
    Age +
    BMI +
    BPSysAve +
    TotChol +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = model_complete,
  family = binomial()
)

coef(final_fit)
##   (Intercept)           Age           BMI      BPSysAve       TotChol 
##  -7.112019578   0.060353831   0.094418073   0.004711273  -0.287435639 
##    Gendermale Race1Hispanic  Race1Mexican    Race1White    Race1Other 
##   0.323389816  -0.114460591  -0.025663962  -0.747630347   0.222607119 
## PhysActiveYes   Smoke100Yes 
##  -0.170460081   0.163988180

The linear predictor is

\[ LP = \beta_0 + \sum_{j=1}^{p}\beta_j X_j, \]

and the predicted probability is

\[ P(\text{Diabetes}) = \frac{e^{LP}}{1+e^{LP}}. \]

42 41. Robust Construction of a New Patient

Factor levels can differ in capitalization across datasets. The following helper selects an available level using case-insensitive matching.

match_factor_level <- function(x, desired) {
  lvls <- levels(x)
  idx <- match(tolower(desired), tolower(lvls))

  if (is.na(idx)) {
    stop(
      "Requested level '", desired,
      "' not found. Available levels: ",
      paste(lvls, collapse = ", ")
    )
  }

  lvls[idx]
}

Inspect available levels:

levels(model_complete$Gender)
## [1] "female" "male"
levels(model_complete$Race1)
## [1] "Black"    "Hispanic" "Mexican"  "White"    "Other"
levels(model_complete$PhysActive)
## [1] "No"  "Yes"
levels(model_complete$Smoke100)
## [1] "No"  "Yes"

Create an illustrative new patient:

new_patient <- tibble(
  Age = 60,
  BMI = 32,
  BPSysAve = 145,
  TotChol = 5.2,
  Gender = factor(
    match_factor_level(model_complete$Gender, "male"),
    levels = levels(model_complete$Gender)
  ),
  Race1 = factor(
    match_factor_level(model_complete$Race1, "white"),
    levels = levels(model_complete$Race1)
  ),
  PhysActive = factor(
    match_factor_level(model_complete$PhysActive, "no"),
    levels = levels(model_complete$PhysActive)
  ),
  Smoke100 = factor(
    match_factor_level(model_complete$Smoke100, "yes"),
    levels = levels(model_complete$Smoke100)
  )
)

new_patient

43 42. Predict Diabetes Probability for the New Patient

new_patient_probability <- predict(
  final_fit,
  newdata = new_patient,
  type = "response"
)

new_patient_probability
##         1 
## 0.1763617

Because the NHANES example is cross-sectional, interpret this as a model-based probability of prevalent diabetes status, not future 5-year risk.

44 43. Nomogram

A nomogram can be created from the ordinary lrm() model.

nom_fit <- lrm(
  Diabetes_num ~
    Age +
    BMI +
    BPSysAve +
    TotChol +
    Gender +
    Race1 +
    PhysActive +
    Smoke100,
  data = model_complete,
  x = TRUE,
  y = TRUE
)
nom <- nomogram(
  nom_fit,
  fun = plogis,
  fun.at = c(0.05, 0.10, 0.20, 0.30, 0.50),
  funlabel = "Probability of Diabetes"
)

plot(nom)

45 44. Summary Performance Table

performance_summary <- tibble(
  Domain = c(
    "Development sample size",
    "Development event rate",
    "Apparent AUC",
    "Apparent Brier score",
    "Temporal validation sample size",
    "Temporal validation AUC",
    "Temporal validation calibration intercept",
    "Temporal validation calibration slope",
    "Temporal validation Brier score"
  ),
  Value = c(
    as.character(nrow(model_complete)),
    sprintf("%.3f", mean(model_complete$Diabetes_num)),
    sprintf("%.3f", as.numeric(auc(roc_obj))),
    sprintf("%.3f", brier_model),
    as.character(nrow(validation)),
    sprintf("%.3f", validation_auc),
    sprintf("%.3f", cal_intercept_val),
    sprintf("%.3f", cal_slope_val),
    sprintf("%.3f", brier_val)
  )
)

performance_summary

47 46. Full Workflow Recap

The complete process can be summarized as:

Research question
        ↓
Define target population and outcome
        ↓
Choose clinically justified candidate predictors
        ↓
Check IDs and duplicates
        ↓
Check coding and variable types
        ↓
Check ranges and clinically impossible values
        ↓
Assess missingness
        ↓
Assess outliers
        ↓
EDA and descriptive statistics
        ↓
Check correlation and multicollinearity
        ↓
Assess continuous-variable functional forms
        ↓
Multiple imputation
        ↓
Fit prespecified/full model
        ↓
Consider nonlinear terms and interactions
        ↓
Consider penalization/shrinkage
        ↓
Estimate apparent performance
        ↓
Bootstrap internal validation
        ↓
Optimism-corrected discrimination
        ↓
Calibration assessment
        ↓
Brier score
        ↓
Decision-curve analysis
        ↓
Clinically justified threshold assessment
        ↓
Sensitivity analyses
        ↓
Subgroup performance
        ↓
Temporal/external validation
        ↓
Recalibration if needed
        ↓
Individual prediction
        ↓
Nomogram / calculator / deployment

48 47. Publication-Level Upgrades

For a real publication, consider the following upgrades beyond this teaching example:

  1. Use the original CDC NHANES data rather than the teaching package dataset.
  2. Account for NHANES sampling weights, strata, and PSU structure when the research question targets population inference.
  3. Prespecify predictors based on clinical knowledge and prior literature.
  4. Determine model sample size using prediction-model sample-size methods rather than relying only on the historical 10 events per variable rule.
  5. Avoid arbitrary categorization of continuous variables.
  6. Use nonlinear modeling when clinically/statistically justified.
  7. Perform multiple imputation within the resampling procedure for rigorous internal validation with missing data.
  8. Preserve the complete model-building algorithm inside every bootstrap resample, including variable selection if variable selection is part of the algorithm.
  9. Report optimism-corrected discrimination and calibration.
  10. Validate the frozen model in independent data before clinical deployment.
  11. Report according to current prediction-model reporting guidance applicable to the study.
  12. Keep data-cleaning, derivation, modeling, validation, and reporting code reproducible and version controlled.

49 48. Session Information

Always save session information so the analysis can be reproduced later.

sessionInfo()
## R version 4.4.3 (2025-02-28)
## Platform: aarch64-apple-darwin20
## Running under: macOS 26.5.2
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## time zone: America/New_York
## tzcode source: internal
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] broom_1.0.10    glmnet_4.1-10   Matrix_1.7-2    pROC_1.19.0.1  
##  [5] rms_8.1-1       Hmisc_5.2-5     car_3.1-3       carData_3.0-5  
##  [9] mice_3.19.0     naniar_1.1.0    gtsummary_2.5.0 lubridate_1.9.4
## [13] forcats_1.0.1   stringr_1.5.2   dplyr_1.2.0     purrr_1.1.0    
## [17] readr_2.1.5     tidyr_1.3.1     tibble_3.2.1    ggplot2_4.0.2  
## [21] tidyverse_2.0.0 NHANES_2.1.0   
## 
## loaded via a namespace (and not attached):
##  [1] Rdpack_2.6.4       gridExtra_2.3      sandwich_3.1-1     rlang_1.1.7       
##  [5] magrittr_2.0.3     multcomp_1.4-30    polspline_1.1.25   compiler_4.4.3    
##  [9] vctrs_0.7.2        quantreg_6.1       pkgconfig_2.0.3    shape_1.4.6.1     
## [13] fastmap_1.2.0      backports_1.5.0    labeling_0.4.3     rmarkdown_2.30    
## [17] markdown_2.0       tzdb_0.5.0         nloptr_2.2.1       visdat_0.6.0      
## [21] MatrixModels_0.5-4 xfun_0.53          jomo_2.7-6         cachem_1.1.0      
## [25] litedown_0.7       jsonlite_2.0.0     pan_1.9            cluster_2.1.8     
## [29] R6_2.6.1           bslib_0.9.0        stringi_1.8.7      RColorBrewer_1.1-3
## [33] boot_1.3-31        rpart_4.1.24       jquerylib_0.1.4    Rcpp_1.1.0        
## [37] iterators_1.0.14   knitr_1.50         zoo_1.8-14         base64enc_0.1-3   
## [41] splines_4.4.3      nnet_7.3-20        timechange_0.3.0   tidyselect_1.2.1  
## [45] rstudioapi_0.17.1  abind_1.4-8        yaml_2.3.10        codetools_0.2-20  
## [49] lattice_0.22-6     withr_3.0.2        S7_0.2.0           evaluate_1.0.5    
## [53] foreign_0.8-88     survival_3.8-3     xml2_1.4.0         pillar_1.10.1     
## [57] checkmate_2.3.4    foreach_1.5.2      reformulas_0.4.1   generics_0.1.4    
## [61] hms_1.1.4          commonmark_2.0.0   scales_1.4.0       minqa_1.2.8       
## [65] glue_1.8.0         tools_4.4.3        data.table_1.17.8  lme4_1.1-37       
## [69] SparseM_1.84-2     fs_1.6.6           mvtnorm_1.3-3      grid_4.4.3        
## [73] rbibutils_2.3      cards_0.7.1        colorspace_2.1-1   nlme_3.1-167      
## [77] htmlTable_2.4.3    Formula_1.2-5      cli_3.6.5          gt_1.3.0          
## [81] gtable_0.3.6       sass_0.4.10        digest_0.6.37      TH.data_1.1-5     
## [85] htmlwidgets_1.6.4  farver_2.1.2       htmltools_0.5.8.1  lifecycle_1.0.5   
## [89] mitml_0.4-5        MASS_7.3-64