Part 1: Judgmental Forecasting

Suppose a health system has just gotten a newly approved specialty therapy onto formulary and needs a monthly demand forecast for its first year how many patients start it, how much to stock, how much clinic capacity to hold open. There is no history to fit. The drug did not exist last year, the eligible population is defined by a brand-new label, and uptake depends on things that live in nobody’s spreadsheet: how fast clinicians trust it, whether payers put up prior-authorization friction, what competing therapies do. A seasonal ARIMA has nothing to chew on here. This is judgmental territory.

What makes it judgmental territory specifically is that the useful information is forward-looking and human, not encoded in a past series. Hyndman and Athanasopoulos (2021, ch. 6) frame judgmental forecasting as the appropriate tool precisely when data are unavailable or when a known future event breaks the historical pattern. A product launch is the textbook version of both at once.

I actually ran into the mirror image of this in my midterm assignment. Forecasting beer/wine/liquor retail sales, my statistical models systematically under-forecast right through the 2020 COVID demand surge, because the surge was a structural break that simply was not in the training window. The models were not wrong about the past; they were blind to a regime the past never contained. That is the same failure mode a new-therapy launch would hand a purely statistical approach, the difference is that with a launch you at least know the break is coming, which is exactly the situation where a human can add information a model cannot.

Where judgmental methods actually earn their keep, in my reading: (1) little or no historical data, like the launch above; (2) a dominant, known, upcoming event: a policy change, a promotion, a plant coming online, where the effect is real but unmodeled; and (3) structural breaks you can anticipate.

Combining judgment with statistics. The version I find most defensible is not “human vs. model” but a statistical baseline with disciplined judgmental adjustments layered on top, and for events only. Once even a few months of launch data exist, a regression or ETS baseline can carry the structure, while a structured expert panel adjusts for the known upcoming reimbursement decision. A Delphi process is attractive here because the anonymity and iteration blunt the loudest-voice-in-the-room problem; scenario analysis is useful for the payer question, since the honest answer is a small set of futures with different probabilities rather than one point.

The risks of over-trusting intuition are the part I take most seriously. Anchoring (the first number someone says drags the rest), optimism or wishful bias (the launch team wants uptake to be fast), groupthink, and badly calibrated confidence intervals that are almost always too narrow. Mitigations that follow Hyndman and Athanasopoulos’s (2021) principles: set the forecasting task explicitly and document it; elicit independently before aggregating (the Delphi move); and/or adjust only for signal you can name.

What I would like to know is at what point does enough launch data accumulate that the judgmental adjustments start doing more harm than good? A few months feels too soon to trust a model, but I don’t think there’s a principled rule for when to hand the wheel back.

Part 2: Time Series Regression - Victorian Electricity Demand

For the regression side I used vic_elec from fpp3: half-hourly electricity demand and temperature for Victoria, because it comes with an obvious external predictor (temperature) that has a genuine physical link to the target. I aggregated to daily totals for 2014, using daily maximum temperature, which is the framing the textbook uses.

elec <- vic_elec |>
  filter(year(Time) == 2014) |>
  index_by(Date = as_date(Time)) |>
  summarise(
    Demand      = sum(Demand) / 1e3,   # daily total, GWh
    Temperature = max(Temperature),    # daily maximum, °C
    Holiday     = any(Holiday)
  )

elec |>
  autoplot(Demand, colour = BC_INK, linewidth = 0.4) +
  labs(title = "Daily electricity demand, Victoria (2014)",
       x = NULL, y = "Demand (GWh)", caption = BADGE) +
  theme_bc()

The series has a clear weekly rhythm (weekdays above weekends) and spikes in the Australian summer (December through February). That summer behavior is the tell that temperature belongs in the model.

Model specification

I fit three models so the contribution of the external predictor is visible rather than assumed. All three include a linear trend() and a weekly season() (day-of-week). The difference is temperature: absent, entered linearly, then entered as a quadratic.

fit <- elec |>
  model(
    base = TSLM(Demand ~ trend() + season()),
    lin  = TSLM(Demand ~ trend() + season() + Temperature),
    quad = TSLM(Demand ~ trend() + season() + Temperature + I(Temperature^2))
  )

glance(fit) |> select(.model, r_squared, adj_r_squared, AICc)

Here’s the part that surprised me. Adding temperature linearly moved R² from about 0.353 to 0.356, essentially nothing. My first instinct was that temperature just wasn’t very predictive, which felt wrong physically. The quadratic term is what exposed the problem: R² jumps to about 0.792 (adjusted 0.786). A linear term forces demand to move monotonically with heat, but Victorian demand does the opposite of monotonic, it’s high when it’s cold (heating) and high when it’s hot (air conditioning), with a trough in between. Averaging those two slopes into one line cancels most of the signal.

elec |>
  ggplot(aes(Temperature, Demand)) +
  geom_point(colour = BC_INK, alpha = 0.45, size = 1.3) +
  geom_smooth(method = "lm", formula = y ~ poly(x, 2),
              se = FALSE, colour = BC_INK, linewidth = 0.7) +
  annotate("text", x = 24, y = 172, label = "Lowest demand near 24°C",
           size = 3, colour = "grey40") +
  labs(title = "Daily demand against maximum temperature",
       x = "Maximum temperature (°C)", y = "Demand (GWh)", caption = BADGE) +
  theme_bc()

The U is unmistakable: demand bottoms out around 24 °C and climbs toward both extremes. So the honest lesson from Part 2 isn’t “external predictors improve accuracy”, it’s that they improve accuracy only if you get the functional form right, and getting that form right was a judgment call, not something the data volunteered.

Out-of-sample accuracy

R² on the training data rewards complexity, so I held out November–December 2014 (the summer ramp, roughly the last 17% of the year) and forecast it. Temperature for the test window is supplied to the model, reasonable here because in an operational setting you’d pair this with a weather forecast, though that does push the true uncertainty onto whoever forecasts the weather.

train <- elec |> filter(Date <  as_date("2014-11-01"))
test  <- elec |> filter(Date >= as_date("2014-11-01"))

fit_tr <- train |>
  model(
    base = TSLM(Demand ~ trend() + season()),
    quad = TSLM(Demand ~ trend() + season() + Temperature + I(Temperature^2))
  )

fc  <- fit_tr |> forecast(new_data = test)
acc <- accuracy(fc, elec)
acc |> select(.model, RMSE, MAE)
acc |>
  mutate(.model = recode(.model,
                         base = "Trend + season",
                         quad = "Trend + season + temperature²")) |>
  ggplot(aes(x = RMSE, y = reorder(.model, -RMSE))) +
  geom_segment(aes(x = 0, xend = RMSE, yend = .model),
               colour = "grey85", linewidth = 0.3) +
  geom_point(colour = BC_INK, size = 3) +
  geom_text(aes(label = round(RMSE, 1)), hjust = -0.4, size = 3, colour = "grey40") +
  xlim(0, max(acc$RMSE) * 1.15) +
  labs(title = "Out-of-sample accuracy (Nov–Dec 2014 holdout)",
       x = "Test RMSE (GWh)", y = NULL, caption = BADGE) +
  theme_bc()

The temperature model carries out of sample: test RMSE falls from about 21.8 GWh to about 16.5 GWh, roughly a 24% reduction. So this isn’t just in-sample overfitting — the U-shape is real structure, not a curve chasing noise in the training window.

Residuals and the case for ARIMAX

This is where I’d temper enthusiasm. The quadratic model fits well, but its residuals are badly autocorrelated:

res_acf <- fit |> select(quad) |> augment() |>
  ACF(.innov, lag_max = 21) |>
  as_tibble() |> mutate(lag = as.numeric(lag))

res_acf |>
  ggplot(aes(x = lag, y = acf)) +
  geom_hline(yintercept = 0, colour = "grey80", linewidth = 0.3) +
  geom_segment(aes(xend = lag, yend = 0), colour = BC_INK, linewidth = 0.3) +
  geom_point(colour = BC_INK, size = 1.6) +
  labs(title = "Residual autocorrelation — temperature model",
       x = "Lag (days)", y = "ACF", caption = BADGE) +
  theme_bc()

fit |> augment() |> filter(.model == "quad") |>
  features(.innov, ljung_box, lag = 14)

The lag-1 autocorrelation sits around 0.57 and the Ljung-Box test rejects independence hard (p ≈ 0). Today’s demand error tells you a lot about tomorrow’s, which means the OLS assumption of independent errors is violated. The point estimates are probably fine, but the standard errors and prediction intervals are not trustworthy; they’ll be too narrow, and any significance test on the coefficients is optimistic.

That is the textbook signature that a dynamic regression / ARIMAX model is the more appropriate choice here. Instead of assuming white-noise errors, ARIMAX (ARIMA(Demand ~ Temperature + I(Temperature^2) + ...)) lets the error term carry its own ARIMA structure, absorbing the day-to-day persistence the OLS residuals show. Would it beat this model? Almost certainly on interval, and likely a bit on point accuracy, the tradeoff is that it’s harder to fit and interpret, and forecasting it forward still leans on a future temperature path. For a first pass I’d keep the quadratic TSLM to establish the temperature effect cleanly, then move to ARIMA errors once I care about the intervals, which, for capacity planning, I eventually would.

Part 3: Reflection

Putting the two halves side by side, the thing that struck me is that “judgment” showed up on both sides of this assignment, not just in Part 1. Part 1 is judgment where there’s no data at all, the therapy launch, where the future genuinely isn’t in the past and a person has to supply the missing structure. But Part 2 turned out to be judgment too: the data were sitting right there, and a naïve linear temperature term still learned almost nothing. What rescued the model was a human decision about functional form, recognizing the U-shape before the regression could reward it. More data wouldn’t have fixed the linear model; better judgment about the specification did.

So I’d frame the strengths and weaknesses as complementary rather than competing. Judgmental forecasting is strong exactly where the historical series is silent: new products, known upcoming shocks, regime changes, and weak because it’s bias-prone, doesn’t scale, and produces overconfident intervals unless it’s structured, documented, and scored. Time series regression is strong where the past encodes stable structure you can estimate and validate on a holdout, and weak in two ways I hit directly: it assumes the past regime persists (the COVID under-forecasting from my midterm assignment), and its inference quietly breaks when residuals autocorrelate (the ACF above), which is easy to miss if you only look at R².

The synthesis I’m landing on: use statistics for the structure the data can support, use judgment for the structure it can’t yet and treat the choice of model form itself as a judgmental act that deserves the same scrutiny as any expert opinion. What I still don’t have a clean answer for is how to quantify the judgment in Part 2. I can score an expert panel’s forecasts over time, but how do I audit my own decision to reach for a quadratic, beyond the holdout, without just retrofitting a story the data already suggested?

References

Hyndman, R. J., & Athanasopoulos, G. (2021). Forecasting: Principles and practice (3rd ed.). OTexts. https://otexts.com/fpp3/