Assignment 11: Detecting Overfitting and Overfitting Issues

Assignment Objectives

• Enhance understanding the procedure of likelihood-based chi-square hypothesis testing .

• Implement the procedures for detecting overfitting/underfitting issues in practical applications.

Policies of Using AI Tools

Policy on AI Tool Use: Please adhere to the AI tool policy specified in the course syllabus. The direct copying of AI-generated content is strictly prohibited. All submitted work must reflect your own understanding; where external tools are consulted, content must be thoroughly rephrased and synthesized in your own words.

Code Inclusion Requirement: Any code included in your essay must be properly commented to explain the purpose and/or expected output of key code lines. Submitting AI-generated code without meaningful, student-added comments will not be accepted.

Testing Overfitting/Underfitting

In Machine Learning and Statistics, overfitting occurs when a model is too complex and learns noise, leading to poor performance on new data, while underfitting happens when a model is too simple to capture important patterns, resulting in high errors overall; both issues are explained by the Bias–Variance Tradeoff and can cause unreliable predictions in real-world applications.

The probability density function (PDF) of the Weibull distribution is:

\[ f(t; \lambda, \beta) = \frac{\beta}{\lambda} \left( \frac{t}{\lambda} \right)^{\beta-1} \exp\left[ -\left( \frac{t}{\lambda} \right)^\beta \right], \quad t \ge 0 \] where \(\lambda > 0\) is the scale parameter (characteristic life) and \(\beta > 0\) is the shape parameter.

When \(\beta = 1\), the Weibull PDF simplifies to the exponential PDF:

\[ f(t; \lambda) = \frac{1}{\lambda} \exp\left( -\frac{t}{\lambda} \right) \] with constant hazard rate \(h(t) = 1/\lambda\).

This assignment focuses on performing a hypothesis test for the shape parameter (\(\beta\)) of the Weibull distribution within a reliability mode

\[\begin{align} H_0&: \beta = 1 \quad \text{(Exponential model, simpler)} \\ H_1&: \beta \neq 1 \quad \text{(Weibull model, more complex)} \end{align}\]

Question: Reliability Application

A mid-sized manufacturing company producing industrial conveyor systems began experiencing unexpected downtime in one of its distribution facilities, prompting concern about the reliability of a newly sourced batch of ball bearings used in the motor assemblies. These bearings, supplied by a vendor adopting cost-saving production methods, were installed across multiple units operating under continuous load conditions. After several months, maintenance logs revealed a pattern of increasing failures, with components lasting anywhere from a few dozen to over 150 hours before breakdown. To investigate, the engineering team collected time-to-failure data from 50 identical bearings and conducted a Weibull analysis within the framework of Reliability Engineering. The 50 time-to-failure (survival time) are:

12.4, 18.7, 25.3, 30.1, 33.5, 35.2, 38.9, 40.3, 42.7, 45.1, 47.6, 49.8, 52.4, 55.0, 
57.3, 60.2, 62.8, 65.1, 67.9, 70.5, 72.3, 75.6, 78.2, 80.9, 83.4, 85.7, 88.1, 90.6, 
93.2, 95.8, 98.4, 101.0, 104.5, 107.3, 110.6, 113.2, 116.8, 120.1, 123.7, 127.4,
130.9, 134.5, 138.2, 142.0, 146.3, 150.7, 155.2, 160.8, 168.4, 175.9

This assignment focuses on hypothesis \(H_0: \beta = 1\) (exponential) against \(H_1: \beta \neq 1\) (Weibull). This framework detects overfitting (fitting a Weibull when exponential is true) and underfitting (fitting exponential when Weibull with \(\beta \neq 1\) is true).

a). Find the MLE of the Weibull parameters \(\lambda\) (scale) and \(\beta\) (shape), denoted by \(\hat{\lambda}\) and \(\hat{\beta}\), respectively, using the optim() procedure. [Hint: You should provide explicit expressions for the log-likelihood and gradient functions of the Weibull distribution parameters.]

############################################################
# HW: Weibull vs Exponential Model (Parts a–e)
# Data: Bearing time-to-failure (hours)
############################################################

# Input the data directly
t <- c(
  12.4, 18.7, 25.3, 30.1, 33.5, 35.2, 38.9, 40.3, 42.7, 45.1,
  47.6, 49.8, 52.4, 55.0, 57.3, 60.2, 62.8, 65.1, 67.9, 70.5,
  72.3, 75.6, 78.2, 80.9, 83.4, 85.7, 88.1, 90.6, 93.2, 95.8,
  98.4, 101.0, 104.5, 107.3, 110.6, 113.2, 116.8, 120.1, 123.7, 127.4,
  130.9, 134.5, 138.2, 142.0, 146.3, 150.7, 155.2, 160.8, 168.4, 175.9
)

n <- length(t)

############################################################
# (a) Weibull MLE using numerical optimization
############################################################

# Negative log-likelihood for Weibull
weibull_nll <- function(par, t) {
  lambda <- par[1]
  beta   <- par[2]
  
  # Enforce parameter constraints
  if (lambda <= 0 || beta <= 0) return(Inf)
  
  # Log-likelihood
  ll <- sum(log(beta/lambda) +
            (beta - 1)*log(t/lambda) -
            (t/lambda)^beta)
  
  return(-ll)  # minimize negative log-likelihood
}

# Initial parameter guesses
init <- c(lambda = mean(t), beta = 1)

# Run optimization
fit_weibull <- optim(init, weibull_nll, t = t, hessian = TRUE)

# Extract MLEs
lambda_hat <- fit_weibull$par[1]
beta_hat   <- fit_weibull$par[2]

cat("Weibull MLEs:\n")

Weibull MLEs:

cat("lambda_hat =", lambda_hat, "\n")

lambda_hat = 99.02759 

cat("beta_hat   =", beta_hat, "\n\n")

beta_hat   = 2.205876 

b). Find the MLE of the exponential parameter \(\lambda\) (scale), denoted by \(\hat{\lambda}\), using any procedure. [Hint: You should provide explicit expressions for the log-likelihood and gradient functions of the exponential distribution parameters.]

############################################################
# (b) Exponential MLE
############################################################

# MLE for exponential distribution
lambda_exp_hat <- mean(t)

cat("Exponential MLE:\n")

Exponential MLE:

cat("lambda_hat =", lambda_exp_hat, "\n\n")

lambda_hat = 87.61 

c). Perform the likelihood ratio \(\chi^2\) test on \(\beta = 1\). What is the p-value? [Hint: Use the results in a) and b).]

############################################################
# (c) Likelihood Ratio Test
############################################################

# Log-likelihood under Weibull
ll_weibull <- -fit_weibull$value

# Log-likelihood under exponential
ll_exp <- sum(-log(lambda_exp_hat) - t/lambda_exp_hat)

# LRT statistic
LRT_stat <- 2 * (ll_weibull - ll_exp)

# p-value
pval_LRT <- 1 - pchisq(LRT_stat, df = 1)

cat("Likelihood Ratio Test:\n")

Likelihood Ratio Test:

cat("LRT statistic =", LRT_stat, "\n")

LRT statistic = 34.44995 

cat("p-value       =", pval_LRT, "\n\n")

p-value       = 4.373535e-09 

d). Perform the Wald \(\chi^2\) on the same hypothesis \(\beta = 1\). What is the p-value? [Hint: You need to find the observed Fisher information matrix (i.e., the negative Hessian matrix from optim()) based on the Weibull distribution. The inverse of the negative observed Hessian matrix is the covariance matrix.]

############################################################
# (d) Wald Test
############################################################

# Variance-covariance matrix from Hessian
vcov_mat <- solve(fit_weibull$hessian)

# Variance of beta
var_beta <- vcov_mat[2,2]

# Wald statistic
W_stat <- (beta_hat - 1)^2 / var_beta

# p-value
pval_Wald <- 1 - pchisq(W_stat, df = 1)

cat("Wald Test:\n")

Wald Test:

cat("W statistic =", W_stat, "\n")

W statistic = 23.10901 

cat("p-value     =", pval_Wald, "\n\n")

p-value     = 1.530721e-06 

e). Write a summary of the above analyses to address the following:

• Whether the two tests generated the same results.

• Which model is recommended for the data.

• Draw the density curve based on the MLE(s) of the parameter(s) and describe the distribution of the time-to-failure.

############################################################
# (e) Interpretation
############################################################

cat("Summary:\n")

Summary:

if (pval_LRT < 0.05 & pval_Wald < 0.05) {
  cat("Both tests reject H0: beta = 1.\n")
  cat("The Weibull model provides a significantly better fit.\n")
} else {
  cat("Fail to reject H0: exponential model may be adequate.\n")
}

Both tests reject H0: beta = 1.
The Weibull model provides a significantly better fit.

if (beta_hat > 1) {
  cat("beta_hat >", 1, ": increasing failure rate (wear-out behavior).\n")
} else if (beta_hat < 1) {
  cat("beta_hat <", 1, ": decreasing failure rate (early failures).\n")
} else {
  cat("beta_hat =", 1, ": constant failure rate.\n")
}

beta_hat > 1 : increasing failure rate (wear-out behavior).

############################################################
# Density Plot using Weibull MLEs
############################################################

# Create a sequence of x values
x_vals <- seq(min(t), max(t), length.out = 200)

# Weibull density using MLEs
weibull_density <- (beta_hat / lambda_hat) *
                   (x_vals / lambda_hat)^(beta_hat - 1) *
                   exp(-(x_vals / lambda_hat)^beta_hat)

# Plot histogram of data (scaled to density)
hist(t, probability = TRUE,
     main = "Weibull Fit to Time-to-Failure Data",
     xlab = "Time to Failure (hours)",
     border = "black")

# Overlay Weibull density curve
lines(x_vals, weibull_density, lwd = 2)

# Optional: add exponential curve for comparison
exp_density <- (1 / lambda_exp_hat) * exp(-x_vals / lambda_exp_hat)
lines(x_vals, exp_density, lwd = 2, lty = 2)

legend("topright",
       legend = c("Weibull Fit", "Exponential Fit"),
       lwd = 2,
       lty = c(1, 2))