ETS Model Selection: Brute Force vs Automatic

Overview

This document demonstrates that brute force model comparison and automatic ETS selection yield identical results, providing confidence in both approaches for time series forecasting.

The 9 Core ETS Models

ETS (Error, Trend, Seasonal) models describe time series using combinations of:

• Error type: Additive (A) or Multiplicative (M)
  
• Trend: None (N), Additive (A), or Multiplicative (M)
  
• Seasonality: None (N), Additive (A), or Multiplicative (M)

This yields 9 common ETS models that we can systematically compare:

Key Takeaways

Main Finding

Both brute force comparison (fitting all 9 models manually) and automatic selection converge on the same optimal model, validating the reliability of automated ETS selection algorithms.

Practical Implications

• Automatic selection is computationally efficient while maintaining accuracy
• Manual comparison provides transparency into the selection process
  
• Both approaches use AICc for model comparison, ensuring consistent results :::{r nine-ets-models}

# Load required libraries
library(fpp3)

# Demonstrate the 9 core ETS models using Australian retail data
aus_retail_total <- aus_retail |>
  summarise(Turnover = sum(Turnover, na.rm = TRUE))

# Fit all 9 core ETS models
nine_ets_models <- aus_retail_total |>
  model(
    # No trend, no seasonality
    ETS_ANN = ETS(Turnover ~ error("A") + trend("N") + season("N")),
    ETS_MNN = ETS(Turnover ~ error("M") + trend("N") + season("N")),
    
    # Additive trend, no seasonality  
    ETS_AAN = ETS(Turnover ~ error("A") + trend("A") + season("N")),
    ETS_MAN = ETS(Turnover ~ error("M") + trend("A") + season("N")),
    
    # Multiplicative trend, no seasonality
    ETS_AMN = ETS(Turnover ~ error("A") + trend("M") + season("N")),
    ETS_MMN = ETS(Turnover ~ error("M") + trend("M") + season("N")),
    
    # Additive trend, additive seasonality
    ETS_AAA = ETS(Turnover ~ error("A") + trend("A") + season("A")),
    ETS_MAA = ETS(Turnover ~ error("M") + trend("A") + season("A")),
    
    # Additive trend, multiplicative seasonality  
    ETS_AAM = ETS(Turnover ~ error("A") + trend("A") + season("M")),
    ETS_MAM = ETS(Turnover ~ error("M") + trend("A") + season("M")),
    
    # Automatic selection
    ETS_AUTO = ETS(Turnover)
  )

# Compare brute force vs automatic - show they give same results
model_comparison <- nine_ets_models |>
  glance() |>
  arrange(AICc) |>
  select(.model, AICc, BIC, sigma2) |>
  mutate(
    Delta_AICc = AICc - min(AICc),
    Delta_AICc = round(Delta_AICc, 2)
  )

cat("Model Comparison (ranked by AICc):\n")

Model Comparison (ranked by AICc):

print(model_comparison, n = Inf)

# A tibble: 11 × 5
   .model    AICc   BIC  sigma2 Delta_AICc
   <chr>    <dbl> <dbl>   <dbl>      <dbl>
 1 ETS_MAM  8047. 8115. 3.98e-4       0   
 2 ETS_AUTO 8047. 8115. 3.98e-4       0   
 3 ETS_AAM  8048. 8116. 1.83e+5       0.91
 4 ETS_AAA  8276. 8345. 3.07e+5     229.  
 5 ETS_MAA  8569. 8637. 1.30e-3     522.  
 6 ETS_MMN  9480. 9500. 1.04e-2    1433.  
 7 ETS_MAN  9482. 9502. 1.06e-2    1435.  
 8 ETS_MNN  9533. 9545. 1.22e-2    1486.  
 9 ETS_AMN  9675. 9696. 7.54e+6    1628.  
10 ETS_AAN  9675. 9696. 7.55e+6    1628.  
11 ETS_ANN  9723. 9735. 8.45e+6    1676.  

# Show what automatic selection chose
auto_model_info <- nine_ets_models |>
  select(ETS_AUTO) |>
  report()

Series: Turnover 
Model: ETS(M,A,M) 
  Smoothing parameters:
    alpha = 0.243116 
    beta  = 0.01110633 
    gamma = 0.2014983 

  Initial states:
     l[0]     b[0]      s[0]     s[-1]    s[-2]    s[-3]    s[-4]     s[-5]
 6309.891 57.47676 0.9708111 0.9002087 0.944641 1.345776 1.044253 0.9972006
     s[-6]     s[-7]     s[-8]     s[-9]   s[-10]    s[-11]
 0.9480892 0.9655833 0.9714195 0.9489153 1.008861 0.9542413

  sigma^2:  4e-04

     AIC     AICc      BIC 
8045.699 8047.146 8115.213 

cat("\ Automatic selection chose:\n")

 Automatic selection chose:

print(auto_model_info)

# A mable: 1 x 1
      ETS_AUTO
       <model>
1 <ETS(M,A,M)>

# Verify brute force vs automatic give same results
best_manual <- model_comparison |>
  slice(1) |>
  pull(.model)

• This matches the automatic selection, confirming both approaches yield identical results.

best_manual

[1] "ETS_MAM"