Methods in Biostatistics III
Problem Set I
Hrishikesh Sathyamoorthy
load("C:/Users/hrish/OneDrive/Documents/nmes.rdata")
library(ggplot2)
library(splines)
library(splines2)
## Warning: package 'splines2' was built under R version 4.3.2
library(dplyr)
## Warning: package 'dplyr' was built under R version 4.3.2
##
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
##
## filter, lag
## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, union
library(knitr)
## Warning: package 'knitr' was built under R version 4.3.2
library(rgl)
## Warning: package 'rgl' was built under R version 4.3.2
library(scatterplot3d)
library(boot)
library(gridExtra)
##
## Attaching package: 'gridExtra'
## The following object is masked from 'package:dplyr':
##
## combine
library(tidyverse)
## Warning: package 'tidyr' was built under R version 4.3.2
## Warning: package 'readr' was built under R version 4.3.2
## Warning: package 'lubridate' was built under R version 4.3.2
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ forcats 1.0.0 ✔ stringr 1.5.1
## ✔ lubridate 1.9.3 ✔ tibble 3.2.1
## ✔ purrr 1.0.2 ✔ tidyr 1.3.1
## ✔ readr 2.1.5
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ gridExtra::combine() masks dplyr::combine()
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(lmtest)
## Warning: package 'lmtest' was built under R version 4.3.2
## Loading required package: zoo
##
## Attaching package: 'zoo'
##
## The following objects are masked from 'package:base':
##
## as.Date, as.Date.numeric
## Create custom theme
custom_theme <- theme(
panel.background = element_blank(),
panel.margin = unit(c(0,0, 0, 0), "npc"),
title = element_text(size = 10, face = "bold"),
plot.title = element_text(hjust = 0.5),
axis.ticks = element_blank(),
axis.text = element_text(size = 12),
axis.title = element_text(size = 14, face = "bold"),
axis.line = element_line(size = 0.5)
)
## Warning: The `size` argument of `element_line()` is deprecated as of ggplot2 3.4.0.
## ℹ Please use the `linewidth` argument instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.
## Warning: The `panel.margin` argument of `theme()` is deprecated as of ggplot2 2.2.0.
## ℹ Please use the `panel.spacing` argument instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.
Part I.Analysis of Variance
Using data from the NMES, perform an analysis comparing the mean total expenditures for 65 year old ever vs. never smokers using three methods: two-sample t-test, analysis of variance and a simple linear regression model.
Create the sample of 65 year old ever and never smokers from the NMES. HINT: The eversmk variable in the dataset is a factor with levels 0 (never smoker), 1(ever smoker) and . (missing). You will need to create a subset where lastage = 65 and eversmk = 0 or 1. Call this data: d.
Create a numeric variable “ever” which is 1 (ever smoker) or 0 (never smoker).
Conduct the two-sample t-test, analysis of variance and simple linear regression:
Compute the square root of the “mean squared error” from the analysis of variance table and compare this to the “residual standard error” output from the lm function. Are these the same or different?
Compare the (t-statistic)2 and F-statistics with corresponding p-values. Are these the same or different?
data_q2 <- nmes
## 1.c.i) Creating sample of never and ever smokers and removing missing values and
## 1.c.ii) Creating a numeric variable 'ever'
d <- data_q2 |>
select(pidx, totalexp, lastage, eversmk, current, former) |>
mutate(eversmk = as.factor(eversmk)) |>
filter(lastage == 65 & eversmk %in% c(0,1)) |>
mutate(ever = case_when(
eversmk == 0 ~ 0,
eversmk == 1 ~ 1)) |>
filter(!is.na(lastage) & !is.na(eversmk)) |>
ungroup()
## 1.c.iii) Conducting t-test, ANOVA, and linear regression
ttest.r <- t.test(totalexp~ever,data=d,var.equal=TRUE)
aov.r <- summary(aov(totalexp~ever,data=d))
lm.r <- summary(lm(totalexp~ever,data=d))
## 1.c.iv) Comparing square root of MSE from ANOVA and residual SE from linear regression
sqrt(aov.r[[1]]$"Mean Sq")[2]
## [1] 9325.891
lm.r$sigma
## [1] 9325.891
## 1.c.v) Comparing t-statistic squared with the F-statistic of linear regression
(ttest.r$statistic)^2
## t
## 4.383587
lm.r$fstatistic[1]
## value
## 4.383587
The square root of MSE from ANOVA and residual SE from linear regression are similar (9325.891)
The values of t-statistic squared and the F-statistic of linear regression are similar (4.383587)
Using the sample from part 1c, create a new variable X that is 0 = never smoker, 1 = former smoker, 2 = current smoker and remove individuals with missing current/former smoker information.
Conduct the analysis of variance and linear regression model
The linear regression model has an intercept and two slopes. Write out the definition of slopes and intercept with respect to the group means. Show that the null hypotheses of the ANOVA and linear regression are similar.
Using the F-tests, what do you conclude regarding differences in the mean total expenditures for 65 year old current, former and never smokers?
## 2.i) Creating a new categorical smoking variable (0, 1, 2)
d$X = ifelse(d$current=="1", 2, ifelse(d$former=="1", 1,ifelse(d$current=="." & d$former==".",NA,0)))
d = d[!is.na(d$X),]
## 2.ii) Conducting ANOVA and linear regression
summary(aov(totalexp~as.factor(X),data=d))
## Df Sum Sq Mean Sq F value Pr(>F)
## as.factor(X) 2 4.161e+08 208031361 2.351 0.097 .
## Residuals 297 2.628e+10 88474011
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
summary(lm(totalexp~as.factor(X),data=d))
##
## Call:
## lm(formula = totalexp ~ as.factor(X), data = d)
##
## Residuals:
## Min 1Q Median 3Q Max
## -4452 -3663 -1881 -722 108605
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2092.8 789.3 2.651 0.00845 **
## as.factor(X)1 2358.2 1210.6 1.948 0.05237 .
## as.factor(X)2 2359.6 1514.1 1.558 0.12018
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 9406 on 297 degrees of freedom
## Multiple R-squared: 0.01559, Adjusted R-squared: 0.008958
## F-statistic: 2.351 on 2 and 297 DF, p-value: 0.09701
2. iii) β0- This is the intercept term. It represents the mean of the reference group (Group A in this case). μA = 2092.8 β1- This represents the difference in means between Group B and the reference group (Group A). μB - μA = 2358.2 β2- Similarly, this coefficient represents the difference in means between Group C and the reference group (Group A). μC - μA = 2359.6
iv) At a significance level of 0.05, there is no statistically significant difference in mean total expenditures for 65-year-old current, former, and never smokers.
Part II. Advanced Inferences for Linear Regression
Fit a MLR of expenditures on age and smoking status as: expenditure ~ (agem65 + age_sp1 + age_sp2) + ever + ever*(agem65 + age_sp1 + age_sp2)
Write a short, scientific interpretation of each coefficient in the model; use the estimated coefficient with corresponding confidence interval.
## Modifying dataset for the required task
data_q2 <- data_q2 |>
select(pidx, totalexp, lastage, eversmk) |>
mutate(eversmk = as.factor(eversmk)) |>
rename(age = lastage) |>
filter(age >= 65 & eversmk %in% c(0,1)) |>
mutate(ever = case_when(
eversmk == 0 ~ 0,
eversmk == 1 ~ 1))
## 1. Defining the variables and spline terms
data_q2 <- mutate(data_q2,
agem65 = age-65,
age_sp1 = ifelse(age - 75 > 0, age - 75, 0),
age_sp2 = ifelse(age - 85 > 0, age - 85, 0))
## Fitting the MLR model
reg_exp_age_smk_q2 <- lm(totalexp ~ agem65 + age_sp1 + age_sp2 + ever + ever*(agem65 + age_sp1 + age_sp2),
data = data_q2)
summary(reg_exp_age_smk_q2)
##
## Call:
## lm(formula = totalexp ~ agem65 + age_sp1 + age_sp2 + ever + ever *
## (agem65 + age_sp1 + age_sp2), data = data_q2)
##
## Residuals:
## Min 1Q Median 3Q Max
## -9846 -3739 -2838 -882 171074
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2445.18 469.31 5.210 1.97e-07 ***
## agem65 161.72 73.38 2.204 0.0276 *
## age_sp1 -102.24 140.94 -0.725 0.4682
## age_sp2 546.81 257.02 2.127 0.0334 *
## ever 1513.54 624.50 2.424 0.0154 *
## agem65:ever -140.64 100.12 -1.405 0.1602
## age_sp1:ever 261.66 206.39 1.268 0.2049
## age_sp2:ever -964.30 463.21 -2.082 0.0374 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 10040 on 4720 degrees of freedom
## Multiple R-squared: 0.008323, Adjusted R-squared: 0.006852
## F-statistic: 5.659 on 7 and 4720 DF, p-value: 1.591e-06
confint(reg_exp_age_smk_q2)
## 2.5 % 97.5 %
## (Intercept) 1525.10262 3365.25200
## agem65 17.86463 305.56559
## age_sp1 -378.53673 174.06160
## age_sp2 42.93004 1050.68155
## ever 289.22976 2737.85040
## agem65:ever -336.91405 55.63338
## age_sp1:ever -142.96473 666.28276
## age_sp2:ever -1872.40197 -56.19044
Intercept: The estimated medical expenditure for a never smoker at the age of 65 years is around $2445.18 (95% confidence interval: $1525.10 to $3365.25).
Co-efficients: agem65- Amongst never smokers between ages 65 years and 75 years, the change in expenditure comparing two people who differ by 1 year in age is around $161.72 (95% confidence interval: $17.86 to $305.57).
age_sp1- Amongst never smokers between ages 75 and 85 years, the change in expenditure comparing two people who differ by 1 year in age decreases by around $102.24 (95% confidence interval: -$378.54 to $174.06), when compared to the never smokers of ages 65-75 years.
age_sp2- Amongst never smokers over the age of 85 years, the change in expenditure comparing two people who differ by 1 year in age increases by around $546.81 (95% confidence interval: $42.93 to $1050.68), when compared to the never smokers of ages 75-85 years.
ever- The difference in expenditures between ever-smokers and never-smokers at the age of 65 is around $1513.54 (95% confidence interval: $289.23 to $2737.85).
agem65 * ever- When compared to never smokers of age 65 years, the yearly expenditure decreases by about $140.64 (95% confidence interval: -$336.91 to $55.63) for ever smokers, between ages 65 years and 75 years.
age_sp1 * ever- When compared to never smokers of ages 65 to 75 years, the yearly expenditure increases by about $261.66 (95% confidence interval: -$142.96 to $666.28) for ever smokers, amongst ages 75-85 years.
age_sp2 * ever- When compared to never smokers of ages 75 to 85 years, the yearly expenditure decreases by about $964.30 (95% confidence interval: -$1872.40 to -$56.19) for ever smokers, amongst ages older than 85 years.
## Using the model to estimate the predicted values and displaying them against the data
predicted_values <- predict(reg_exp_age_smk_q2, data_q2)
data_q2$predicted_values <- predict(reg_exp_age_smk_q2)
# Creating a new dataframe for line plot
line_data <- data.frame(
Age = data_q2$age,
Ever = as.factor(data_q2$ever),
Predicted = predicted_values
)
plot_data <- data.frame(
Age = data_q2$age,
Expenditures = data_q2$totalexp,
Predicted = predicted_values
)
# Plot using ggplot2
ggplot(plot_data, aes(x = Age, y = Expenditures)) +
geom_point(color = "grey", alpha = 0.5) +
scale_x_continuous(name = "Age", limits = c(65, 94)) +
scale_y_continuous(name = "Expenditures", limits = c(0, 15000)) +
ggtitle("Actual vs Predicted Expenditures") +
custom_theme +
theme(legend.position = c(0.2, 0.9),
legend.title = element_blank(),
legend.text = element_text(size = 10),
legend.key = element_blank(),
plot.title = element_text(size = 14, face = "bold", hjust = 0.5)) +
geom_line(data = line_data, aes(x = Age, y = Predicted, color = Ever), size = 1.5) +
scale_color_manual(breaks = c("0", "1"),
values = c("black", "blue"),
labels = c("never smokers", "ever smokers"))
## Warning: A numeric `legend.position` argument in `theme()` was deprecated in ggplot2
## 3.5.0.
## ℹ Please use the `legend.position.inside` argument of `theme()` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.
## Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
## ℹ Please use `linewidth` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.
## Warning: Removed 316 rows containing missing values or values outside the scale range
## (`geom_point()`).
## Modifying dataset for the required task
data_q2_ne <- data_q2 |>
select(age, ever, predicted_values) |>
group_by(age, ever) |>
slice(1) |>
mutate(ever = case_when(
ever == 0 ~ "never",
ever == 1 ~ "ever")) |>
pivot_wider(names_from = ever, values_from = predicted_values)
# Calculate the difference in mean expenditures between ever and never smokers
data_q2_ne$predicted_values_diff = data_q2_ne$ever - data_q2_ne$never
# Plot the difference in mean expenditures between ever and never smokers as a function of age
ggplot(data_q2_ne, aes(x = age, y = predicted_values_diff)) +
geom_line(size = 1.5, color = "darkgreen") +
geom_hline(yintercept = 0, linetype = "solid", color = "darkblue") +
labs(x = "Age (in years)",
y = "Difference in Mean Medical Expenditure \n (in dollars)",
title = "Difference in Average Medical Expenditures \n Between Ever and Never Smokers Against Age") +
custom_theme +
scale_x_continuous(limits = c(65, 94)) +
scale_y_continuous(limits = c(-6000, 2000)) +
theme(legend.position = c(0.2, 0.9),
legend.title = element_blank(),
legend.text = element_text(size = 10),
legend.key = element_blank(),
plot.title = element_text(size = 14, face = "bold", hjust = 0.5))
## Warning: Removed 1 row containing missing values or values outside the scale range
## (`geom_line()`).
There are a couple of reasons behind why ever smokers among persons over 85 years have lesser average expenditures.
a. There could be the phenomenon of reverse causation. The persons may have had some health conditions that lead to high medical expenditures in the past which might have lead smokers to quit. Thus, among the elderly, individuals who have never smoked might currently have more health-related expenses because they are the ones who are now developing conditions that smokers might have developed earlier in life.b. There could be survivorship bias where smokers who have survived to age 85 may represent a subgroup with particularly better health status than their peers, or they may have quit smoking early enough to resolve some of the long-term health risks associated with smoking.
# Obtain the variance-covariance matrix of the model
vcov_matrix <- vcov(reg_exp_age_smk_q2)
# Define the gradient vector for the difference at age 65
gradient_65 <- c(0, 0, 0, 0, 1, 0, 0, 0)
# Standard error of the difference at age 65
std_error_65 <- sqrt(t(gradient_65) %*% vcov_matrix %*% gradient_65)
# Age- 75 years
dummy <- matrix(c(1,10), nrow = 1, ncol = 2)
mean_75 = dummy %*% matrix(coef(reg_exp_age_smk_q2)[5:6])
std_error_75 = sqrt(dummy %*% matrix(vcov_matrix[5:6,5:6], nrow = 2, ncol = 2) %*% t(dummy))
ci_75 = c(mean_75 - 1.96*std_error_75, mean_75 + 1.96*std_error_75)
# Age- 90 years
dummy <- matrix(c(1,25,15,5), nrow = 1, ncol = 4)
mean_90 = dummy %*% matrix(coef(reg_exp_age_smk_q2)[5:8])
std_error_90 = sqrt(dummy %*% matrix(vcov_matrix[5:8,5:8], nrow = 4, ncol = 4) %*% t(dummy))
ci_90 = c(mean_90 - 1.96*std_error_90, mean_90 + 1.96*std_error_90)
# Standard errors
std_error_65
## [,1]
## [1,] 624.4994
std_error_75
## [,1]
## [1,] 587.3095
std_error_90
## [,1]
## [1,] 1672.993
#Average difference between ever and never smokers for-
# 65 year olds- β4 = 1513.54
# 75 year olds- β4 + (75-65)*β5 = 1513.54 - 1406.4 = 107.14
# 85 year olds- β4 + (90-65)*β5 + (90-75)*β6 + (90-85)*β7 = 1513.54 - 3516 + 3924.9 - 4821.5 = -2899.2
# Confidence interval calculations
1513.54 + c(-1, 1) * 1.96 * std_error_65
## Warning in c(-1, 1) * 1.96 * std_error_65: Recycling array of length 1 in vector-array arithmetic is deprecated.
## Use c() or as.vector() instead.
## [1] 289.5211 2737.5589
ci_75
## [1] -1043.990 1258.263
ci_90
## [1] -6178.130 380.002
For age 65: Estimated difference = β4 = $1513.54 Standard error = 624.50 95% confidence interval = (289.52, 2737.56)
For age 75: **Estimated difference = β4 + (75-65)*β5 = 1513.54 - 1406.4 = $107.14 Standard error = 587.31 95% confidence interval = (-1043.99, 1258.27)**
For age 90: **Estimated difference = β4 + (90-65)β5 + (90-75)β6 + (90-85)*β7 = 1513.54 - 3516 + 3924.9 - 4821.5 = -$2899.2 Standard error = 1673.00 95% confidence interval = (-6178.13, 380.02)**
# Coefficients from the model
beta_0 <- coef(reg_exp_age_smk_q2)["(Intercept)"]
beta_4 <- coef(reg_exp_age_smk_q2)["ever"]
# Ratio at age 65
ratio_65 <- (beta_0 + beta_4) / beta_0
ratio_65
## (Intercept)
## 1.61899
# Derivatives of the ratio with respect to beta_0 and beta_4
g_beta_0 <- -beta_4 / beta_0^2
g_beta_4 <- 1 / beta_0
# Gradient vector
gradient <- c(g_beta_0, 0, 0, 0, g_beta_4, 0, 0, 0)
# Standard error of the ratio using the delta method
std_error_ratio_65 <- sqrt(t(gradient) %*% vcov_matrix %*% gradient)
print(std_error_ratio_65)
## [,1]
## [1,] 0.3534826
# Confidence interval calculations
ratio_65 + c(-1, 1) * 1.96 * std_error_ratio_65
## Warning in c(-1, 1) * 1.96 * std_error_ratio_65: Recycling array of length 1 in vector-array arithmetic is deprecated.
## Use c() or as.vector() instead.
## [1] 0.926164 2.311816
set.seed(653)
# Create the boot function, return the ratio
ratio_boot <- function(data, index) {
resample <- data_q2[index,]
boot_ratio_model = lm(totalexp ~ (agem65 + age_sp1 + age_sp2) + ever + ever*(agem65 + age_sp1 + age_sp2), data = resample)
ratio = (coef(boot_ratio_model)[1] + coef(boot_ratio_model)[5])/coef(boot_ratio_model)[1]
return(ratio)
}
# Run boot function and store boot object
# 500 replicates
mlr_ratio_result <- boot(data_q2, ratio_boot, R = 500)
mlr_ratio_result
##
## ORDINARY NONPARAMETRIC BOOTSTRAP
##
##
## Call:
## boot(data = data_q2, statistic = ratio_boot, R = 500)
##
##
## Bootstrap Statistics :
## original bias std. error
## t1* 1.61899 -0.0005968935 0.3029594
# Calculate the boostrap 95% CI using the percentile method
quantile(mlr_ratio_result$t, c(0.025,0.975))
## 2.5% 97.5%
## 1.080503 2.242679
Estimated ratio = 1.62 Standard error = 0.35 95% confidence interval = (0.93, 2.31)
Bootstrap ratio = 1.62 Standard error = 0.30 95% confidence interval = (1.08, 2.24)
The bootstrap standard error (0.35) is smaller than the model standard error (0.30) calculated using delta method, and thus a narrower confidence interval.
set.seed(653)
# Creating the boot function
reg_exp_age_smk_q2_boot <- function(data, index) {
resample <- data_q2[index,]
boot_model <- lm(totalexp ~ (agem65 + age_sp1 + age_sp2) + ever + ever*(agem65 + age_sp1 + age_sp2), data = resample)
dummy_boot_1 <- matrix(c(1,10), nrow = 1, ncol = 2)
mean_75 = dummy_boot_1 %*% matrix(coef(boot_model)[5:6])
dummy_boot_2 <- matrix(c(1,25,15,5), nrow = 1, ncol = 4)
mean_90 = dummy_boot_2 %*% matrix(coef(boot_model)[5:8])
return(c(coef(boot_model)[5], mean_75, mean_90))
}
# Runing the boot function and storing the results
boot_result <- boot(data_q2, reg_exp_age_smk_q2_boot, R = 500)
boot_result
##
## ORDINARY NONPARAMETRIC BOOTSTRAP
##
##
## Call:
## boot(data = data_q2, statistic = reg_exp_age_smk_q2_boot, R = 500)
##
##
## Bootstrap Statistics :
## original bias std. error
## t1* 1513.5401 -52.10542 601.4837
## t2* 107.1367 39.05466 609.3461
## t3* -2899.0641 36.96127 2218.6220
# Retrieving the bootstrapped 95% CI and point estimates
boot_ci <- sapply(1:3, function(x) boot.ci(boot_result, index = x, type = "norm")$normal[2:3])
boot_ci
## [,1] [,2] [,3]
## [1,] 386.7591 -1126.214 -7284.445
## [2,] 2744.5319 1262.378 1412.394
For age 65: Bootstrap Estimated difference = $1513.54 Bootstrap Standard error = 601.48 Bootstrap 95% confidence interval = (386.76, 2744.53)
For age 75: Bootstrap Estimated difference = $107.14 Bootstrap Standard error = 609.35 Bootstrap 95% confidence interval = (-1126.21, 1262.38)
For age 90: Bootstrap Estimated difference = -$2899.2 Bootstrap Standard error = 2218.62 Bootstrap 95% confidence interval = (-7284.45, 1412.39)
# Table from Q4.
age <- c(65, 75, 90)
estimated_difference <- c(1513.54, 107.14, -2899.2)
linear_model_std_error <- c(624.50, 587.31, 1673.00)
linear_model_95ci <- c("(289.52, 2737.56)", "(-1043.99, 1258.27)", "(-6178.13, 380.02)")
bootstrap_std_error <- c(601.48, 609.35, 2218.62) # Placeholder values
bootstrap_95ci <- c("(386.76, 2744.53)", "(-1126.21, 1262.38)", "(-7284.45, 1412.39)")
# Creating a data frame
results_table <- data.frame(
'Age' = age,
'Estimated Difference' = estimated_difference,
'Model SE' = linear_model_std_error,
'Model 95% CI' = linear_model_95ci,
'Bootstrap SE' = bootstrap_std_error,
'Bootstrap 95% CI' = bootstrap_95ci
)
# Creating the table
kable(results_table, format = "simple", align = "c")
| Age | Estimated.Difference | Model.SE | Model.95..CI | Bootstrap.SE | Bootstrap.95..CI |
|---|---|---|---|---|---|
| 65 | 1513.54 | 624.50 | (289.52, 2737.56) | 601.48 | (386.76, 2744.53) |
| 75 | 107.14 | 587.31 | (-1043.99, 1258.27) | 609.35 | (-1126.21, 1262.38) |
| 90 | -2899.20 | 1673.00 | (-6178.13, 380.02) | 2218.62 | (-7284.45, 1412.39) |
# Null hypothesis with no smoking and interaction terms
reg_null = lm(totalexp ~ (agem65 + age_sp1 + age_sp2), data = data_q2)
# Alternate hypothesis with all covariates
reg_alt = lm(totalexp ~ (agem65 + age_sp1 + age_sp2) + ever + ever*(agem65 + age_sp1 + age_sp2), data = data_q2)
# Likelihood ratio test
lrtest(reg_null, reg_alt)
## Likelihood ratio test
##
## Model 1: totalexp ~ (agem65 + age_sp1 + age_sp2)
## Model 2: totalexp ~ (agem65 + age_sp1 + age_sp2) + ever + ever * (agem65 +
## age_sp1 + age_sp2)
## #Df LogLik Df Chisq Pr(>Chisq)
## 1 5 -50278
## 2 9 -50272 4 12.309 0.0152 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
# F-test
anova(reg_null, reg_alt)
## Analysis of Variance Table
##
## Model 1: totalexp ~ (agem65 + age_sp1 + age_sp2)
## Model 2: totalexp ~ (agem65 + age_sp1 + age_sp2) + ever + ever * (agem65 +
## age_sp1 + age_sp2)
## Res.Df RSS Df Sum of Sq F Pr(>F)
## 1 4724 4.7747e+11
## 2 4720 4.7622e+11 4 1241422709 3.076 0.01532 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
The likelihood ratio test shows statistically significant results at a significance level of 0.05, with the test statistic equal to (chi-squared) 12.309 and a p-value of 0.0152. The F-test has similar results with a test statistic of 3.076 and a p-value of 0.01532. This favours the rejection of the null hypothesis. Thus, on average, ever and never smokers use different quantities of medical services.
Objective To determine whether ever and never smokers of the same age over 65 years have roughly the same average total medical expenditures.
Data The 1987 National Medical Expenditure Survey dataset consists of 13,648 subjects with 22 variables, that includes demographic information, detailed smoking history (smoking status, number of cigarettes smoked, pack years, etc.), health indicators, and socioeconomic factors.
Methods We fit and plot a multiple linear regression model allowing the total medical expenditures to change as a function of age (linear spline with knot at 75 and 85 years) separately for ever smokers and never smokers. We also plot the difference in mean expenditures between ever and never smokers as a function of age. We use both the linear model and bootstrap methods to calculate the estimated differences in expenditures among ever and never smokers along with its 95% confidence intervals at different age points of 65, 75, and 90 years. Additionally, we estimate the ratio of the average expenditures comparing ever to never smokers at age 65 using both the linear regression and bootstrap methods. Finally, we use a likelihood ratio test and an F-test to strengthen our hypothesis of difference in medical expenditures across the groups.
Results Based on the linear regression model, the estimated difference (95% confidence interval) in average expenditures comparing ever to never smokers at ages 65, 75, and 90 are 1513.54 (289.52, 2773.56), 107.14 (-1043.99, 1258.27), and -2899.2 (-6178.27, 379.87) dollars, respectively, which are confirmed by similar confidence intervals obtained from bootstrap. The ratio of the average expenditures comparing ever to never smokers at age 65 years is estimated at 1.62 (95% confidence interval from linear model: 0.93 to 2.31; 95% confidence interval from bootstrap: 1.08 to 2.24). Both the likelihood ratio test and the ANOVA tests were statistically significant result (p-value = 0.015). All of these results indicate that the mean expenditures at any age are different for ever and never smokers.
Discussion The estimated differences in average expenditures comparing ever to never smokers show that at the age of 65, ever smokers have higher average medical expenditures than never smokers. When the age increase to 75, the expenditures become roughly comparable. However, at the age of 90, ever smokers have much lower expenditures than never smokers, probably because of survivorship bias or reverse causation phenomena.
Part III. Estimating rates of change from smooth functions
In a linear spline model, slope(age) represents the rate of change of expected total expenditures with respect to age. The linear spline will have different slopes for different segments of age divided by the knots. Between 65 and 75 years, there is a single slope determined by the linear relationship between age and expenditures. At the first knot at 75 years, there is an “elbow,” a point where the slope can change abruptly, leading to a new slope from 75 to 85 years. Similarly, at 85 years, there is another elbow, and the slope can change again for ages beyond 85. Thus, the function of the slope is piecewise linear with potential discontinuities in the slope at the knot points.
## 2.a) Including only ever smokers, fitting a cubic spline and storing relevant values
rateofchange_cs <- data_q2 |> filter((eversmk == 1))
# Fitting a cubic spline model for ever smokers
fit_bs <- lm(totalexp ~ bs(age, knots = c(75, 85)), data = rateofchange_cs)
summary(fit_bs)
##
## Call:
## lm(formula = totalexp ~ bs(age, knots = c(75, 85)), data = rateofchange_cs)
##
## Residuals:
## Min 1Q Median 3Q Max
## -7135 -3789 -3150 -1028 171322
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 4647.8 654.3 7.104 1.59e-12 ***
## bs(age, knots = c(75, 85))1 -1509.5 1467.3 -1.029 0.304
## bs(age, knots = c(75, 85))2 -611.2 1382.8 -0.442 0.659
## bs(age, knots = c(75, 85))3 2554.6 2331.0 1.096 0.273
## bs(age, knots = c(75, 85))4 -2597.2 3601.5 -0.721 0.471
## bs(age, knots = c(75, 85))5 2487.4 4473.4 0.556 0.578
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 10490 on 2416 degrees of freedom
## Multiple R-squared: 0.003386, Adjusted R-squared: 0.001323
## F-statistic: 1.642 on 5 and 2416 DF, p-value: 0.1456
# Saving the estimated coefficients and variance matrix
fit_coefficients_bs <- coef(fit_bs)
V_bs <- vcov(fit_bs)
## 2. b) Generating the design matrix and its derivative for the cubic spline function
ages <- 65:94
X <- model.matrix(~ bs(ages, knots = c(75, 85)))
delta <- 0.001
X_plus_delta <- model.matrix(~ bs(ages+delta, knots = c(75, 85)))
X_minus_delta <- model.matrix(~ bs(ages-delta, knots = c(75, 85)))
dX <- (X_plus_delta - X_minus_delta)/(2*delta)
# Estimating the expected total expenditures and slope for ages 65 to 94
expected_expenditures <- X %*% fit_coefficients_bs # Mean expenditures for each age
slope <- dX %*% fit_coefficients_bs # Slope for each age
# Computing the variance of the slope
Var_slope <- diag(dX %*% V_bs %*% t(dX))
Var_expenditures <- diag(X %*% V_bs %*% t(X))
# 95% Confidence intervals for the slope
CI_lower <- slope - 1.96 * sqrt(Var_slope)
CI_upper <- slope + 1.96 * sqrt(Var_slope)
## 2.c) Plotting E(total expenditures) vs. age and slope(age) vs. age
plot_data <- data.frame(
age = ages,
expected_expenditures = c(expected_expenditures),
slope = c(slope),
CI_lower = CI_lower,
CI_upper = CI_upper
)
# Plot expected expenditures vs age
p1 <- ggplot(plot_data, aes(x = ages)) +
geom_ribbon(aes(ymin = expected_expenditures - 1.96 * sqrt(Var_expenditures),
ymax = expected_expenditures + 1.96 * sqrt(Var_expenditures)), fill = "grey") +
geom_line(aes(y = expected_expenditures)) +
custom_theme +
labs(title = "Expected Total Expenditures by Age", y = "Expected Total Expenditures", x = "Age")
# Plot slope vs age
p2 <- ggplot(plot_data, aes(x = ages)) +
geom_ribbon(aes(ymin = CI_lower, ymax = CI_upper), fill = "grey") +
geom_line(aes(y = slope)) +
custom_theme +
labs(title = "Slope of Total Expenditures by Age", y = "Slope", x = "Age")
# Combine plots
grid.arrange(p1, p2, ncol = 2)
3. Next, we will utilize the splines2 package in R to help us generate dX(age) to estimate slope(age) for a natural cubic spline model with knots at 75 and 85 years. Install the splines2 package if not already done so and include library(splines2) in your R code.
Fit the natural cubic spline model with knots at 75 and 85 years and save the coefficients and variance of the estimated coefficients.
Generate the design matrix X and derivative of the design matrix, dX(age), evaluated at ages 65 through 94, and the estimated E(total expenditures) and slope(age) for ages 65 through 94.
Compute the Var(slope(age)) and create a two panel figure displaying E(total expenditures) and slope(age) for ages 65 to 94 for ever smokers.
## 3.a) Fit the natural cubic spline model with knots at 75 and 85 years for ever smokers
fit_nsp <- lm(rateofchange_cs$totalexp ~ -1 + nsp(rateofchange_cs$age, knots=c(75,85), intercept=TRUE, data = rateofchange_cs))
fit_coeff_nsp <- fit_nsp$coefficients
V.coeff <- vcov(fit_nsp)
## 3.b) Generate the design matrix X and its derivative dX for the natural cubic spline function
Xage <- model.matrix(~ naturalSpline(ages, knots=c(75,85)))
Xage_plus_delta <- model.matrix(~ naturalSpline(ages+delta, knots = c(75, 85)))
Xage_minus_delta <- model.matrix(~ naturalSpline(ages-delta, knots = c(75, 85)))
dXage <- (Xage_plus_delta - Xage_minus_delta)/(2*delta)
# Estimating the expected total expenditures and slope for ages 65 to 94
expected_expenditures.age <- Xage %*% fit_coeff_nsp # Mean expenditures for each age
slope.age <- dXage %*% fit_coeff_nsp # Slope for each age
## 3.c) Compute the variance of the slope estimates
Var_slope.age <- diag(dXage %*% V.coeff %*% t(dXage))
Var_expenditures.age <- diag(Xage %*% V.coeff %*% t(Xage))
# 95% Confidence intervals for the expenditure
lower_bound_e <- expected_expenditures.age - 1.96 * sqrt(Var_expenditures.age)
upper_bound_e <- expected_expenditures.age + 1.96 * sqrt(Var_expenditures.age)
# 95% Confidence intervals for the slope
lower_bound <- slope.age - 1.96 * sqrt(Var_slope.age)
upper_bound <- slope.age + 1.96 * sqrt(Var_slope.age)
# Prepare the data for plotting
plot_data <- data.frame(
ages = seq(65, 94),
mean_Y = expected_expenditures.age,
slope = slope.age,
lower1 = lower_bound,
upper1 = upper_bound,
lower2 = lower_bound_e,
upper2 = upper_bound_e
)
# Plotting
p1 <- ggplot(plot_data, aes(x=ages)) +
geom_line(aes(y=expected_expenditures.age), color="darkblue") +
geom_ribbon(aes(ymin=lower2, ymax=upper2), alpha=0.1, fill = "cyan") +
custom_theme +
labs(title="Expected Total Expenditures by Age", y="Expected Total Expenditures", x="Age")
p2 <- ggplot(plot_data, aes(x=ages)) +
geom_line(aes(y=slope.age), color="darkgreen") +
geom_ribbon(aes(ymin=lower1, ymax=upper1), alpha=0.1, fill = "lightgreen") +
custom_theme +
labs(title="Slope of Total Expenditures by Age", y="Slope", x="Age")
# Combine plots
grid.arrange(p1, p2, ncol=2)
d. Repeat the process above for never smokers and create a final two panel figure with E(total expenditures) and slope(age) for ages 65 to 94 separately for ever and never smokers. Include confidence intervals for slope(age).
## 3.d) Including only never smokers, fitting a cubic spline and storing relevant values
rateofchange_ns <- data_q2 |> filter((eversmk == 0))
# Fit the natural cubic spline model with knots at 75 and 85 years for never smokers
fit_nsp_never <- lm(rateofchange_ns$totalexp ~ -1 + nsp(rateofchange_ns$age, knots=c(75,85), intercept = TRUE, data = rateofchange_ns))
fit_coeff_nsp_never <- coef(fit_nsp_never)
V.coeff_n <- vcov(fit_nsp_never)
## Generate the design matrix X and its derivative dX for the natural cubic spline function
X_n <- model.matrix(~ naturalSpline(ages, knots=c(75,85)), intercept = TRUE)
X_n_plus_delta <- model.matrix(~ naturalSpline(ages+delta, knots = c(75, 85)))
X_n_minus_delta <- model.matrix(~ naturalSpline(ages-delta, knots = c(75, 85)))
dX_n <- (X_n_plus_delta - X_n_minus_delta)/(2*delta)
# Estimating the expected total expenditures and slope for ages 65 to 94
expected_expenditures.age_n <- X_n %*% fit_coeff_nsp_never # Mean expenditures for each age
slope.age_n <- dX_n %*% fit_coeff_nsp_never # Slope for each age
# Compute the variance of the slope estimates
Var_slope.age_n <- diag(dX_n %*% V.coeff_n %*% t(dX_n))
Var_expenditures.age_n <- diag(X_n %*% V.coeff_n %*% t(X_n))
# 95% Confidence intervals for the expenditure
lower_bound_e <- slope.age_n - 1.96 * sqrt(Var_slope.age_n)
upper_bound_e <- slope.age_n + 1.96 * sqrt(Var_slope.age_n)
# 95% Confidence intervals for the slope
lower_bound <- expected_expenditures.age_n - 1.96 * sqrt(Var_expenditures.age_n)
upper_bound <- expected_expenditures.age_n + 1.96 * sqrt(Var_expenditures.age_n)
# Prepare the data for plotting
plot_data <- data.frame(
age = seq(65, 94),
mean_Y = expected_expenditures.age_n,
slope = slope.age_n,
lower1 = lower_bound,
upper1 = upper_bound,
lower2 = lower_bound_e,
upper2 = upper_bound_e
)
# Plot
p1 <- ggplot(plot_data, aes(x=ages)) +
geom_line(aes(y=expected_expenditures.age_n), color="red") +
geom_ribbon(aes(ymin=lower1, ymax=upper1), alpha=0.1, fill="pink") +
custom_theme +
labs(title="Expected Total Expenditures by Age", y="Expected Total Expenditures", x="Age")
p2 <- ggplot(plot_data, aes(x=ages)) +
geom_line(aes(y=slope.age_n), color="darkblue") +
geom_ribbon(aes(ymin=lower2, ymax=upper2), alpha=0.1, fill="purple") +
custom_theme +
labs(title="Slope of Total Expenditures by Age", y="Slope", x="Age")
# Combine plots
grid.arrange(p1, p2, ncol=2)
Amongst never smokers, the expected total expenditures increases with age, until the age of 80 years, after which there is a sharp increase. On the other hand, the slope increases steadily until the age of 75, followed by a sharp increase with a peak at the age of 85, after which there is a marked decrease.