metrics_summary   <- read_csv("C:/Users/dbely/OneDrive/Desktop/Data Forecasting/Week 7/metrics_summary.csv", show_col_types = FALSE)
metrics_horizon   <- read_csv("C:/Users/dbely/OneDrive/Desktop/Data Forecasting/Week 7/metrics_by_horizon.csv", show_col_types = FALSE)
loss_curves       <- read_csv("C:/Users/dbely/OneDrive/Desktop/Data Forecasting/Week 7/loss_curves.csv", show_col_types = FALSE)
stitched          <- read_csv("C:/Users/dbely/OneDrive/Desktop/Data Forecasting/Week 7/stitched_test_predictions.csv", show_col_types = FALSE)
recent_eda        <- read_csv("C:/Users/dbely/OneDrive/Desktop/Data Forecasting/Week 7/vic_elec_recent_for_eda.csv", show_col_types = FALSE)
data_summary      <- read_csv("C:/Users/dbely/OneDrive/Desktop/Data Forecasting/Week 7/data_summary.csv", show_col_types = FALSE)

method_levels <- c("TFT", "GRU", "LSTM", "Naive48", "ETS", "SeasonalNaive")
metrics_summary <- metrics_summary %>% mutate(Method = factor(Method, levels = method_levels)) %>% arrange(Method)

Part 1: Exploring PyTorch Models for Time Series Forecasting

Dataset

This week I used vic_elec: half-hourly operational electricity demand for Victoria, Australia, 2011-12-31 to 2014-12-31 (52,608 total half-hour observations). I didn’t have tsibbledata available in the Python environment where the deep learning work happened, so I rebuilt it directly from the same raw AEMO demand/temperature files and BOM holiday list that the package itself is built from. Two data-quality notes before anything else, because they’re the kind of thing that quietly wrecks a pipeline if you don’t check for them:

  • Three clock-change days (one per year) produce duplicate half-hourly timestamps in the raw feed when daylight saving starts. I collapsed these by averaging rather than arbitrarily keeping one.
  • A handful of half-hours don’t exist at all (the other side of the DST transition) and needed interpolation.

After cleanup the series is a clean, fully regular half-hourly grid: 52,608 observations, matching the published vic_elec row count.

ggplot(recent_eda, aes(x = Time, y = Demand)) +
  geom_line(color = navy, linewidth = 0.35) +
  scale_y_continuous(labels = comma) +
  labs(title = "Victorian electricity demand, last 60 days of the sample",
       subtitle = "Daily and weekly cycles are visually obvious before any modeling starts",
       x = NULL, y = "Demand (MW)") +
  theme_bc() +
  bc_badge()

The daily cycle (trough overnight, shoulder peaks morning and evening) and the weekly cycle (weekday vs. weekend amplitude) are both plainly visible without any decomposition, which matters later, because it tells you a model with only single-period seasonality is missing something real.

ggplot(recent_eda, aes(x = Temperature, y = Demand)) +
  geom_point(color = navy, alpha = 0.35, size = 0.8) +
  labs(title = "Demand vs. temperature is not linear",
       subtitle = "Both cold mornings and hot afternoons push demand up, heating and cooling load stacked on one axis",
       x = "Temperature (deg C)", y = "Demand (MW)") +
  theme_bc() +
  bc_badge()

This U-shape is the classic vic_elec finding and it’s worth stating plainly: a model that treats temperature as linearly related to demand is wrong by construction. That’s one of the more concrete “why deep learning here” arguments in this whole assignment, I didn’t have to engineer a heating/cooling-degree transform for the neural models to pick this up, but I would have needed to for a linear regression.

Feature engineering

  • Calendar features: hour-of-day, day-of-week, and month, each as sin/cos pairs so the model sees a continuous cycle rather than a discontinuity between 23:30 and 00:00.
  • Holiday flag, from the official BOM/AEMO holiday list.
  • Lag features: lag_48 (same half-hour, previous day) and lag_336 (same half-hour, previous week). I checked explicitly that these never leak: for a 48-step decoder window, the furthest-forward lag_48 reference is exactly the last encoder timestep, and lag_336 never gets closer than that, both are always drawing on real history relative to the forecast origin, not the future.

Supervised structure and train/val/test split

Encoder length 144 steps (3 days of history), decoder/prediction length 48 steps (next full day), built as a TimeSeriesDataSet in pytorch-forecasting. Split chronologically: last 14 days held out as test, the 14 days before that as validation, everything else as training (50,737 training windows). Both validation and test are evaluated as dense rolling origins, every possible 48-step-ahead window in that block, stride 1 (625 test origins), rather than a single forecast, which matters a lot for what follows.

Model implementation

I trained three deep learning architectures via pytorch-forecasting, all sharing the same TimeSeriesDataSet pipeline above:

  • LSTM and GRU, via RecurrentNetwork (hidden size 32, 2 layers, dropout 0.15)
  • Temporal Fusion Transformer (TFT) (hidden size 16, 2 attention heads, dropout 0.15)

One architectural constraint surprised me: RecurrentNetwork requires the same set of variables in the encoder and decoder (aside from the target). That’s a real limitation, not a config quirk, a plain RNN processes encoder and decoder as one continuous sequence, so it has no mechanism for “this input is only observed in the past.” Temperature genuinely isn’t known in advance in a real deployment (you’d need an actual weather forecast product to feed it), so the options were either to fabricate a leakage by feeding future actual temperature to the model, or drop it. I dropped it, LSTM and GRU never see temperature at all, only calendar features and the two lag features (which, as checked above, are decoder-safe).

TFT doesn’t have this constraint. Its variable-selection network natively supports covariates that are only observed historically, so temperature went in as time_varying_unknown_reals, available to the encoder, correctly withheld from the decoder. This is the first genuine, mechanistic point in favor of TFT over the plain RNNs in this assignment, and it has nothing to do with attention or interpretability, it’s just a more honest way to encode what the model actually knows at forecast time.

A practical-issues note that belongs in this section, not buried in the challenges list below: this ran on CPU only, and partway through the session the environment dropped to a single available core. Batch cost roughly doubled for the RNNs mid-run (LSTM trained at ~0.38s/batch, GRU at ~1.4-1.7s/batch under otherwise similar settings). I ended up training in small, checkpointed batch budgets rather than clean epoch loops, resuming from disk between calls. One side effect: the three models did not get equal training budgets, LSTM completed 6 full epochs, GRU and TFT stopped at roughly 3 epoch-equivalents each. That’s a real asymmetry in what follows, not a design choice, and I’m calling attention to it rather than pretending the comparison is perfectly fair.

loss_long <- loss_curves %>%
  pivot_longer(c(train_loss, val_loss), names_to = "split", values_to = "loss") %>%
  mutate(split = recode(split, train_loss = "Train", val_loss = "Validation"),
         model = factor(model, levels = c("LSTM", "GRU", "TFT")))

ggplot(loss_long, aes(x = epoch, y = loss, color = split)) +
  geom_line(linewidth = 0.6) +
  geom_point(size = 1.6) +
  facet_wrap(~model, scales = "free_y") +
  scale_color_manual(values = c(Train = navy, Validation = "#B08D57")) +
  labs(title = "Training vs. validation loss, all three deep models",
       subtitle = "LSTM and GRU overfit almost immediately; TFT does not, within the epochs it got",
       x = "Epoch (equivalent)", y = "Loss") +
  theme_bc() +
  bc_badge()

This is the central finding of Part 1, and it’s not a subtle one. Both LSTM and GRU show textbook overfitting: training loss falls steadily and substantially (LSTM 256 -> 41 MAE over 6 epochs; GRU 185 -> 44 over 3 epoch-equivalents) while validation loss bottoms out at the very first epoch and then sits flat-to-noisy afterward (LSTM: 200 -> 266 -> 217 -> 206 -> 259 -> 244; GRU: 176 -> 234 -> 178). Past epoch 1, every bit of additional training loss improvement on both RNNs was memorization, not generalization.

TFT did not show this pattern in the epochs it got, training and validation loss fell together (169/101 -> 100/92 -> 93/92). We need to be careful about the causal story here, because there only 3 epoch-equivalents for TFT and that’s not enough to rule out that it simply hadn’t reached the overfitting regime yet. It’s plausible that TFT’s variable-selection gating provides real implicit regularization; it’s equally plausible we’d see the same divergence at epoch 6 if we had the compute budget to get there. I don’t have enough evidence to pick between those two explanations.

Performance evaluation

metrics_summary %>%
  mutate(RMSE = round(RMSE, 1), MAE = round(MAE, 1), MAPE = paste0(round(MAPE, 1), "%")) %>%
  kable(caption = "Test-set accuracy, 625 rolling origins, aggregated across the full 48-step horizon") %>%
  kable_styling(full_width = FALSE, bootstrap_options = c("striped", "condensed"))
Test-set accuracy, 625 rolling origins, aggregated across the full 48-step horizon
Method RMSE MAE MAPE
TFT 319.6 215.7 5.2%
GRU 327.8 228.5 5.5%
LSTM 337.4 255.2 6.2%
Naive48 444.0 318.6 7.6%
ETS 564.7 429.7 11%
SeasonalNaive 608.7 451.4 11.6% 
ggplot(metrics_summary, aes(x = fct_reorder(Method, RMSE), y = RMSE)) +
  geom_segment(aes(xend = Method, y = 0, yend = RMSE), color = "grey75", linewidth = 0.4) +
  geom_point(color = navy, size = 3) +
  coord_flip() +
  scale_y_continuous(labels = comma) +
  labs(title = "RMSE by method, ranked",
       subtitle = "TFT lowest, both statistical baselines highest",
       x = NULL, y = "RMSE (MW)") +
  theme_bc() +
  bc_badge()

TFT comes out on top (RMSE 320, MAPE 5.2%), with GRU close behind and LSTM a bit further back, consistent with TFT’s access to temperature and its attention mechanism, though I’d want more epochs before treating the TFT/GRU gap as settled. What genuinely surprised me: the plain lag-48 naive forecast (444 RMSE) beats both statistical baselines outright, and beats them by a wide margin. That’s worth sitting with rather than rushing past.

A methodology correction I want to be transparent about. My first pass at evaluating ETS produced an RMSE of 956 and a MAPE over 20%, dramatically worse than even the weekly seasonal-naive benchmark. That result was wrong, and it took me a minute to see why: I’d fit ETS once at the start of the test block and then re-sliced that same long-horizon forecast across all 625 rolling origins, which meant “origin 600” was silently pulling a 600-plus-step-ahead extrapolation, not a fresh 48-step forecast. The RNNs and TFT get a freshly re-conditioned encoder window at every origin; my first ETS pass didn’t get the equivalent treatment, so the comparison wasn’t fair to it. I refit ETS at each of the 625 origins on a trailing 120-day window (additive damped trend, additive seasonality at s=48) and re-ran the evaluation. Corrected RMSE: 565, MAPE 11%. Still the weakest method in the table, but the honest weakest – not an artifact of a broken backtest. I’m including this because it’s exactly the kind of practical issue this assignment is asking about, and it’s the kind of bug that’s easy to miss if you don’t sanity-check that your baseline’s predictions actually move with each new origin.

metrics_horizon <- metrics_horizon %>% mutate(Method = factor(Method, levels = method_levels))

ggplot(metrics_horizon, aes(x = horizon, y = RMSE, color = Method)) +
  geom_line(linewidth = 0.55) +
  scale_color_manual(values = setNames(bc_pal, method_levels)) +
  labs(title = "RMSE by forecast horizon step (1 = 30 min ahead, 48 = next day)",
       subtitle = "RNNs are sharpest at short horizons; TFT decays least; ETS has a distinctive hump",
       x = "Steps ahead", y = "RMSE (MW)") +
  theme_bc() +
  bc_badge()

The horizon breakdown explains more than the aggregate table does. LSTM and GRU are excellent at one step ahead (RMSE in the mid-30s, better than everything else in the table) and then degrade sharply, landing around 370-400 by the end of the day. TFT starts worse at h=1 (~172) but degrades much less steeply, which is why it wins on the aggregate metric despite losing badly at short horizons, it’s the more consistent forecaster across the day, not the most accurate at any single point. ETS shows a genuine hump: good at h=1, worst around h=24 (roughly 12 hours ahead, the point furthest from both the fitting window and the next full daily cycle), then partially recovering by h=48 as the seasonal component wraps back to a similar phase. The naive benchmarks are flat by construction, they’re just fixed historical lookups, so there’s nothing to decay.

plot_methods <- c("TFT", "Naive48", "ETS")
stitched_plot <- stitched %>% filter(method %in% plot_methods)
actual_line <- stitched %>% filter(method == "TFT") %>% select(Time, actual)

ggplot() +
  geom_line(data = actual_line, aes(x = Time, y = actual), color = "grey25", linewidth = 0.4) +
  geom_line(data = stitched_plot, aes(x = Time, y = pred, color = method), linewidth = 0.4, alpha = 0.85) +
  scale_color_manual(values = setNames(bc_pal[c(1,4,5)], plot_methods)) +
  labs(title = "Predictions vs. actuals, full 14-day test period",
       subtitle = "Grey = actual demand. Non-overlapping daily forecasts stitched across the test window.",
       x = NULL, y = "Demand (MW)") +
  theme_bc() +
  bc_badge()

Challenges

  • Overfitting: clear and immediate for both RNNs (see loss curves above); not observed for TFT within its smaller epoch budget, with the caveat noted above that I can’t fully separate “TFT regularizes better” from “TFT just didn’t train long enough to overfit.”
  • Long training times: CPU-only, and a mid-session drop to a single available core forced a shift from ordinary epoch loops to checkpointed, batch-budgeted training, directly responsible for the uneven epoch counts across models (LSTM 6, GRU and TFT ~3 each).
  • Hyperparameter tuning: essentially none. Hidden sizes were chosen small up front specifically to stay CPU-feasible, not tuned. I’d treat every model here as a reasonable-first-attempt baseline, not a tuned architecture.
  • Data preparation complexity: DST duplicate/missing timestamps in the raw feed; distinguishing which engineered features are legitimately decoder-safe (lag_48, lag_336) from which are genuinely unknowable at forecast time (Temperature); and the RecurrentNetwork encoder/decoder symmetry constraint that forced temperature out of the RNN models entirely.

Part 2: Real-World Applications and Interpretability

Scenario analysis

Essentially this is the actual use case behind the dataset: a grid operator forecasting next-day demand to schedule generation and reserve capacity. The case for deep learning here isn’t “it’s more accurate” in some abstract sense, Naive48 already showed that a huge chunk of the signal in this series is just “tomorrow looks like today.” The real case is twofold. First, TFT ingests heterogeneous covariates (weather, calendar, holidays) without hand-engineering a separate seasonal or nonlinear transform for each one, the temperature U-shape above got picked up for free. Second, TFT’s quantile output gives an organization a distribution, not a point forecast, which maps directly onto a capacity-planning decision (“what generation do we need in reserve to cover the 90th percentile case”) without a separate uncertainty-modeling step bolted on afterward.

That said, I don’t think it’s honest to frame this as deep learning simply winning. Naive48 costs nothing to build, nothing to explain, and nothing to audit, and it beat both statistical baselines outright in this assignment. Any organization actually making this decision should weigh the accuracy gain of TFT/GRU against the very real costs below, not assume the more sophisticated model is automatically worth deploying.

Interpretability and challenges

The three architectures sit at genuinely different points on the interpretability spectrum, and it’s not just “deep learning bad, stats good”:

  • ETS is fully transparent, level, trend, and seasonal components are literally readable numbers. You can explain a forecast to a non-technical stakeholder in one sentence.
  • LSTM/GRU are close to black boxes. There’s no native mechanism to say which past time steps or which features drove a given forecast.
  • TFT sits in between by design: its variable-selection network produces per-feature importance weights, and its attention layer shows which historical time steps the model actually attended to for a given forecast. That’s a real, architecturally-built-in interpretability advantage over the plain RNNs, not an add-on.

Limitations worth naming plainly: large data requirements (this assignment had three full years of clean half-hourly history to work with, which is more than a lot of real deployments will have); computational resources (even at this toy scale, CPU-only training needed real workarounds, a production deployment training on years of multi-site data would need GPU infrastructure as a baseline requirement, not a nice-to-have); and explaining predictions to stakeholders (a board is going to accept “it’s a weekday afternoon in a heatwave” from ETS far more readily than a median quantile from a 31,000-parameter attention network, even when the network is more accurate).

Proposed solutions: lean on TFT’s native attention/variable-selection outputs first, since they’re already there and don’t require a separate post-hoc step. SHAP is the natural next tool for the RNNs specifically, but it’s worth being honest that SHAP on a sequence model is expensive to compute and its temporal attributions are harder to communicate to a stakeholder than TFT’s attention weights are, it’s not a free upgrade. A practical middle path several organizations actually use: deploy the more accurate model, but publish period-level attention/SHAP summaries rather than trying to explain any single forecast in isolation, since per-forecast explanations at this level of model complexity rarely hold up well to real scrutiny anyway.

I’m still turning over the Naive48-beats-ETS result. Is that a genuine statement about this particular series (strong, stable day-over-day persistence, weak week-over-week persistence), or is it more a statement about single-seasonal ETS being the wrong tool for a series with two seasonal periods stacked on top of each other? I lean toward the latter, but I don’t think this assignment gives me enough to fully separate the two, curious whether anyone else’s series showed the same pattern.