• Definition: Maximum Likelihood Estimation is a technique for estimating the parameters of a statistical model by maximizing the likelihood function.
• Purpose: To find the parameter values that make the observed data most probable.
• Likelihood Function: Represents the probability of the observed data given a set of parameter values.
• MLE: The parameter values that maximize the likelihood function are called maximum likelihood estimates.

• Consider a simple example of a normal distribution:
  • Given data points \(x_1, x_2, \ldots, x_n\) from a normal distribution with unknown mean \(\mu\) and known variance \(\sigma^2\).
  • The likelihood function is: \[ L(\mu) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x_i - \mu)^2}{2\sigma^2}} \]

• To find the MLE of \(\mu\), we maximize the likelihood function or equivalently maximize the log-likelihood: \[ \log L(\mu) = -\frac{n}{2} \log(2\pi\sigma^2) - \frac{1}{2\sigma^2} \sum_{i=1}^{n} (x_i - \mu)^2 \]
• Taking the derivative and setting it to zero gives the MLE: \[ \hat{\mu} = \frac{1}{n} \sum_{i=1}^{n} x_i \]

• Context: Estimating the mean height of a population based on a sample.
• Data: Heights of 10 individuals.

# Sample data
heights <- c(160, 165, 170, 175, 180, 155, 168, 172, 178, 163)

# Calculate MLE for mean
mle_mean <- mean(heights)
mle_mean

## [1] 168.6

# Load ggplot2
library(ggplot2)

# Define a function for likelihood
likelihood_function <- function(mu, data) {
  n <- length(data)
  return(prod(dnorm(data, mean = mu, sd = sd(data))))
}

# Generate values for mu
mu_values <- seq(150, 190, length.out = 100)
likelihood_values <- sapply(mu_values, likelihood_function, data
                            = heights)

ggplot(data.frame(mu = mu_values, likelihood = likelihood_values), 
       aes(mu, likelihood)) +
  geom_line() + labs(title = "Likelihood Function", 
                     x = "Mean Height (mu)",  y = "Likelihood")

• Context: Estimating the rate parameter \(\lambda\) of an exponential distribution based on observed waiting times.

• Data: Waiting times (in minutes) from an experiment.

waiting_times <- c(2.3, 1.5, 3.8, 2.7, 1.9, 4.1, 2.6, 3.3)
mle_lambda <- 1 / mean(waiting_times)
lambda_values <- seq(0, 1, length.out = 100)
likelihood_function_exp <- function(lambda, data) {
  n <- length(data)
  return(lambda^n * exp(-lambda * sum(data)))
}
likelihood_values_exp <- sapply(lambda_values, 
                                likelihood_function_exp, 
                                data = waiting_times)

# Plot the likelihood function using ggplot2
library(ggplot2)

ggplot(data.frame(lambda = lambda_values, likelihood = 
                    likelihood_values_exp), 
       aes(lambda, likelihood)) +
  geom_line() +
  labs(title = "Likelihood Function for Exponential Distribution", 
       x = "Rate Parameter (λ)", 
       y = "Likelihood")

• Context: This example estimates the parameters (mean and standard deviation) of a normal distribution using Maximum Likelihood Estimation (MLE). It demonstrates how well the estimated normal distribution fits simulated data.

• Data: A sample dataset containing:

  • Value: A numeric variable representing data points drawn from a normal distribution with a mean of 5 and a standard deviation of 2. The dataset consists of 100 observations, reflecting random variation typical in real-world data.
  • Analysis: MLE is applied to estimate the mean and standard deviation based on the simulated data.

library(plotly)

## 
## Attaching package: 'plotly'

## The following object is masked from 'package:ggplot2':
## 
##     last_plot

## The following object is masked from 'package:stats':
## 
##     filter

## The following object is masked from 'package:graphics':
## 
##     layout

set.seed(123)
n <- 100
true_mean <- 5
true_sd <- 2
data <- rnorm(n, mean = true_mean, sd = true_sd)

mle_mean <- mean(data)
mle_sd <- sd(data)

x_values <- seq(min(data) - 3, max(data) + 3, length.out = 100)
y_values <- dnorm(x_values, mean = mle_mean, sd = mle_sd)

plot <- plot_ly() %>%
  add_trace(x = x_values, y = y_values, type = 'scatter', 
            mode = 'lines', name = 'MLE Normal Distribution', line = 
              list(color = 'blue')) %>%
  add_trace(x = data, y = rep(0, n), type = 'scatter', 
            mode = 'markers', name = 'Simulated Data', marker = 
              list(color = 'red', size = 5))%>%
  layout(title = 'Maximum Likelihood Estimation of Normal Distribution',
         xaxis = list(title = 'Value'),
         yaxis = list(title = 'Density'))
plot

• Interpretation:
  • The peak of the likelihood function corresponds to the Maximum Likelihood Estimate (MLE) of the parameter.
  • In this case, the peak shows the most probable value for the mean height based on the given sample data.
• Visualization: The 3D surface plot shows the likelihood landscape over two parameters for a bivariate normal distribution, illustrating the concept of likelihood maximization in multiple dimensions.

• Item Response Theory:
  • MLE is used to estimate item parameters (e.g., difficulty, discrimination) and latent traits (abilities) by finding the values that maximize the likelihood of observed responses to test items, based on the IRT model.
• Biology:
  • Estimating population parameters (e.g., growth rates or survival probabilities) based on sample data from ecological or biological experiments.

• Economics:
  • MLE is widely used in econometrics to estimate the parameters of models, such as regression coefficients in time-series analysis or demand forecasting.
• Machine Learning:
  • MLE is the backbone of many probabilistic models, including Naive Bayes classifiers and Hidden Markov Models (HMMs). It is used to fit models to data for prediction, classification, and clustering tasks.

• Summary:
  • Maximum Likelihood Estimation (MLE) is a powerful statistical method for estimating model parameters that make the observed data most probable.
• Why MLE Matters:
  • The flexibility and power of MLE make it a fundamental tool in statistical inference and data science.
  • MLE is a key parameter estimation function that provides greater accuracy in certain models which other parameter estimation functions cannot, for example, item repsonse theory.