1 Objective

Intraday forecast of WA WEM reference trading price (short horizon) with interpretable drivers.
Primary sources: RTP, Operational Demand/Withdrawal; SCADA as enrichment when available.
Deliverables: metrics vs persistence for next X intervals.

The analysis draws only on AEMO’s public WEM Market Data for the SWIS. The label is the Reference Trading Price (RTP) — the 30-minute trading-interval price defined as the time-weighted average of dispatch-interval market clearing prices and typically published shortly after each interval. My understanding is that it’s the price traders are actually marked to.

On the explanatory side, I use the 5-minute Operational Demand and Operational Withdrawal series (aggregated to 30-minute) as a system-load pulse; 5-minute facility SCADA (summed to system MW) as a real-time supply/response proxy; real-time outages as an availability constraint flag; and FCESS accreditation data for context on regulation/contingency capability that can interact with price formation when the system is tight.

All feeds are the post-reform WEM datasets; dispatch-interval signals are reconciled to trading-interval timestamps in Australia/Perth.

dataset	rows
rtp	672
opdw	3746
scada	4032

1.1 Historical Data

Operational Demand (MW): AEMO’s system-level demand measure at 5-minute dispatch-interval cadence used in operational forecasting/dispatch inputs. “Operational” in AEMO usage refers to electricity consumed by customers as supplied by scheduled / semi-scheduled / significant non-scheduled generation, expressed as power (MW) rather than energy (MWh).AEMO Operational Consumption Defintion
Operational Withdrawal (MW): The system’s withdrawal quantity (load) at the same 5-minute cadence. In the WEM sign convention withdrawal is negative (injection is positive), which is why you’ll see negative values in the feed. (AEMO procedures explicitly define withdrawal forecasts ≤ 0; the OPDW daily JSON also shows negative withdrawal values.) WEM Procedure: Facility Dispatch Process

Operational Demand & Withdrawal — Descriptor (post-reform history)
Category	Metric	Value
Coverage	Rows	200,737
	Start	2023-10-01 08:00
	End	2025-08-28 18:28
	Span (days)	697.4
Operational demand (MW)	Top 10% (P90)	2,741.43
	Median (P50)	1,999.76
	Mean	2,059.83
	Bottom 10% (P10)	1,386.88
	SD	538.92
	IQR	608.56
Operational withdrawal (MW)	Top 10% (P90)	-4.81
	Median (P50)	-33.31
	Mean	-51.48
	Bottom 10% (P10)	-132.51
	SD	63.80
	IQR	54.38
Withdrawal is negative by WEM convention.
Rows = 5-minute observations; units = MW.

1.1.1 Operational Demand

1.1.2 Operational Withdrawal

On first glance demand seems to be as expected, seasonality, drift and peak times are what I would have expected. The spread between years is not as dramatic as withdrawals. There appears to be a negative correlation (absolute value terms) with a flatter and lower trend in the preceeding year than YTD.

Left (month × year heatmap): correlations are weak-to-moderate positive in most months (≈ 0.14 → 0.64), with an outlier high in Oct-2024 (~0.64). 2025 sits mostly 0.27–0.50. Right (r by half-hour slot): overnight (≈00:00–06:30) correlations are clearly negative (down to about −0.6). Around 07:30–08:00 the sign flips and stays near 0 to mildly positive through the day. Evenings show year divergence: 2023 pushes +0.3→+0.5, 2024 hovers 0→+0.2, 2025 trends closer to 0 or slightly negative late evening.

Positive r when demand rises, withdrawals tend to move toward zero (i.e., less negative): consistent with curtailment or load shifting when the system is tighter. My understanding is that Operational withdrawal is a signal of big, visible loads, not the whole system.

1.1.3 Reference Trading Price

RTP — Coverage & Dispersion (post-reform history)
Category	Metric	Value
Coverage	Rows	33435
Coverage	Start	2023-10-01 08:00
Coverage	End	2025-08-28 07:30
Coverage	Span (days)	697.0
Price (AUD/MWh)	Median	87.35
Price (AUD/MWh)	Mean	84.91
Price (AUD/MWh)	SD	118.74
Price (AUD/MWh)	IQR	60.76
Price (AUD/MWh)	P95	216.13
Price (AUD/MWh)	P99	635.49
Price (AUD/MWh)	Spike count (≥P95)	1672.00

Faceting by month was messy. This will suffice for a first glance. 2025 is more stable than preceeding years, so far.

The market allows offers down to an Energy Offer Price Floor of −$1,000/MWh, so when supply exceeds demand—typically around midday with high PV—and inflexible plant would rather stay online than incur shutdown/restart costs, the price can clear below zero. Generators sometimes accept negative prices because the avoidance of cycling costs still makes them whole. Negative prices tell flexible demand and storage to absorb energy and tell generators to curtail. In the sample, 2025 YTD shows no negative medians by time-of-day, but we still observe some negative intervals—concentrated around the midday trough the data.

1.2 Current Price & Demand

1.2.1 Price & Demand Relationship

I am going to run simple OLS regressions on price to better understand the relationship between price and demand.

Regression Results: Price models (HAC Newey-West SEs)
Variable	M1	M2	M3	M4	M5	M6
Intercept	-142.417*** (8.530)	-128.968*** (8.355)	84.890*** (1.711)	-122.472*** (10.530)	-124.527*** (10.437)	19.522*** (2.199)
Demand (MW)	0.110*** (0.004)			0.101*** (0.005)	0.102*** (0.005)
Lagged demand (MW)		0.104*** (0.004)
Delta Demand (MW)			0.191*** (0.009)
Lagged price ($/MWh)						0.709*** (0.012)
Time-slot FE	No	No	No	Yes	Yes	Yes
Day-of-week FE	No	No	No	No	Yes	Yes
Observations	33,434	33,434	33,434	33,434	33,434	33,434
R2	0.250	0.221	0.026	0.261	0.262	0.578
Adj R2	0.250	0.221	0.026	0.260	0.260	0.577
Residual SE	102.8145	104.7633	117.1855	102.1515	102.1002	77.1736
AIC	404680.1	405935.7	413428.6	404294.5	404266.9	385550.7
BIC	404705.4	405961.0	413453.8	404715.3	404738.2	386022.1

There is limited value in the above but much of the results are expected. The variables are significant, R² values tend to be weak. Considering seasonality and daily trends were always going to be the most important factors.

Lagged price: price is sticky. Offers, commitment status, and ramp constraints don’t reset every 30 minutes. The previous term captures that immediate persistence and often takes a big share of the explainable variance.
Δ Demand (half-hour change): markets react to movement as much as level. Ramping up demand tightens headroom and draws marginal units; ramping down does the opposite. The first derivative picks up these near-term pressure effects even when the level variable is flat.
Demand (level): a simple scarcity proxy. Higher absolute load increases the chance the marginal unit is expensive or inflexible; it also helps separate a “busy but stable” system (Δ≈0) from genuine ramps.
Time-of-day structure:
- In the raw model, a quadratic in a 0–1 slot index gives a smooth trough/peak shape without exploding the DoF count.
- In the asinh model, a Fourier pair (sin/cos) provides a smooth daily cycle that’s robust to small timing shifts.
asinh(price/100): prices are heavy-tailed and can be negative. asinh(x/c) acts like log(2x/c) for large |x| but is nearly linear around zero and keeps the sign, so coefficients are easier to interpret than a log transform while avoiding the extreme spikes.

Simple OLS bridge models (HAC Newey–West SEs)
Variable	A: raw + quad(slot)	B: asinh + AR(1) + Δd + level + Fourier(slot)
Intercept	-26.297*** (2.754)	-0.111*** (0.010)
Lagged price ($/MWh)	0.670*** (0.013)
Lagged asinh(price)		0.784*** (0.006)
Δ Demand (MW)	0.185*** (0.006)	0.001*** (0.000)
Demand (MW)	0.031*** (0.002)	0.000*** (0.000)
Time-of-day (0..1)	-61.197*** (5.206)
Time-of-day^2	61.752*** (5.231)
sin(2pislot/48)		0.007** (0.002)
cos(2pislot/48)		0.049*** (0.003)
Observations	33,434	33,434
R2	0.601	0.787
Adj R2	0.601	0.787
Residual SE	75.0165	0.2896
AIC	383606.1	12034.4
BIC	383665.0	12093.3
Cells show estimate with significance (· p<0.10, * p<0.05, p<0.01, * p<0.001); robust SEs in parentheses.

1.2.2 Price & Demand Findings

Results

Lagged price / Lagged asinh(price): Large and tight. ~0.67 on raw price; ~0.78 on the asinh scale.
Δ Demand (MW): Positive and precise in both models. In the raw model it’s ~+0.185 $/MWh per +1 MW ramp; so a +100 MW ramp lifts price by ~$18–19/MWh all else equal. In the asinh model the coefficient (~0.001) maps to ~$0.10/MWh per MW around typical price levels (because asinh is near-linear there with scale 100).
Demand (level): Positive but smaller effect. Raw coefficient ~+0.031 $/MWh per MW (so +100 MW ≈ +$3/MWh).
Time of day:
- Quadratic slot (raw model): negative linear + positive quadratic = the familiar midday trough / evening lift without over-parameterising.
- Fourier pair (asinh model): both sin and cos are significant, giving a smooth daily cycle and slightly better fit.

R² ≈ 0.60 (raw) vs 0.79 (asinh). The transform stabilises variance and increases explanatory power, as expected.

Shortcomings

Endogeneity / simultaneity. Demand and price are jointly determined by dispatch rules and bids. OLS treats Δd and level as exogenous; that’s a stretch. We may need instruments or high-frequency leads/lags as a robustness check.
Omitted supply-side drivers. No explicit controls for renewables / SCADA net supply, forced outages, FCESS activations, fuel costs, or unit constraints. These can move price independently of demand.
Spike mechanics. A linear mean model (even with asinh) underweights tail behaviour. Spikes are not well captured by Gaussian residuals.
Seasonal structure. Quadratic / single Fourier pair is okay for a first pass, but intraday effects vary by season and weekday.
Serial dependence beyond AR(1). Dynamics surely longer memory than one lag; residual ACF/PACF not yet checked.
Comparability of metrics. R² across scales is fine, but AIC/BIC across different response transforms is not. Out-of-sample MAE/DA on a common (raw-price) scale may be more appropriate.

2 Forecast

2.1 Model 1

I build a one-step (next 30-min) forecast for RTP that’s simple, fast, and considers the time order. I transformed price with asinh(price/100) to adjust for spikes but keep sign, then fit a regularised linear model (Elastic Net) using the signals I’ve validated: lagged price, Δ demand, level demand, and time-of-day (sin/cos) plus day-of-week.

I evaluate it with rolling-origin cross validation (expanding window) to avoid look-ahead, pick the best alpha on out-of-sample error, refit on all data, and compare against a persistence baseline (next price = current price).

Metrics are MAE and direction accuracy, reported on the raw price scale (I invert asinh via 100*sinh(.)). Finally, I produce a forecast.

Model	MAE	DirAcc
Persistence	24.093	0.516
ElasticNet(asinh), alpha=0	27.114	0.571

The Elastic Net is better at calling the sign of the next move but worse at level: MAE 27.1 (≈ +12.5% worse), Direction accuracy 0.571 (≈ +5.5 pp better). I benchmark against a “persistence” forecast (next price = last price). It’s a tough short-horizon yardstick in power; if a model can’t beat it, it’s not adding signal. Below I only plot Elastic Net vs actual for clarity, and keep persistence in the metrics table as a reference.

Window metrics for the plotted period
Model	MAE	DirAcc
Elastic Net	39.849	0.537
Persistence	35.913	0.516

The model uses only information available at the start of each trading interval: last price (momentum), the change in price, current demand level and change, plus hour and day-of-week. The model is decent at calling up-moves vs down-moves into the next interval (window DirAcc ≈ 0.54 vs 0.52 for persistence).Perhaps when the model diverges materially from last price (or deviates from its usual error band), that’s a flag to dig into SCADA/outages/bids rather than a trade signal by itself.

Limitations

Price spike mechanics are not modelled. Asinh scaling + ridge shrinkage pulls predictions toward the center, so sharp reversals and scarcity spikes are under-predicted; deep troughs can be over-predicted. Exogenous Features. I’m not using SCADA deltas, outage flags, PV/wind/temperature proxies, or ramp constraints—all of which matter exactly when price becomes nonlinear.

2.2 Model 2

I’ve pulled in data on outages, SCADA supply metrics and a proxy for weather. There was some cleaning and aggregation required from source. I’ll plug in the variables first off then check the model. Then, I’ll conduct some time series tests with a view to making better predictions.

Feature preview (SCADA scale factor = 12, median SCADA/demand = 0.079)
TI	price	demand_mw	withdrawal_mw	scada_mw	d_scada	d2_scada	d_demand	d2_demand	hod	doy	solar_idx	s_doy	c_doy	ref_price_lag1	y_lead1	scada_to_demand	ramp_gap
2025-08-15 08:00:00	134.31	2658.285	-57.962	2635.576					8.0	227	0.500	-0.691	-0.723		126.66	0.991
2025-08-15 08:30:00	126.66	2294.467	-8.565	2320.071	-315.506		-363.818		8.5	227	0.609	-0.691	-0.723	134.31	105.39	1.011	-48.313
2025-08-15 09:00:00	105.39	2043.479	-19.321	2058.517	-261.554	53.952	-250.988	112.830	9.0	227	0.707	-0.691	-0.723	126.66	82.99	1.007	10.566
2025-08-15 09:30:00	82.99	1839.797	-64.050	1798.375	-260.142	1.412	-203.683	47.305	9.5	227	0.793	-0.691	-0.723	105.39	84.61	0.977	56.459
2025-08-15 10:00:00	84.61	1761.666	-189.541	1593.866	-204.509	55.633	-78.131	125.552	10.0	227	0.866	-0.691	-0.723	82.99	84.16	0.905	126.378
2025-08-15 10:30:00	84.16	1623.019	-222.672	1414.309	-179.558	24.951	-138.647	-60.515	10.5	227	0.924	-0.691	-0.723	84.61	81.89	0.871	40.911
2025-08-15 11:00:00	81.89	1601.562	-332.880	1285.706	-128.603	50.955	-21.457	117.189	11.0	227	0.966	-0.691	-0.723	84.16	69.81	0.803	107.146
2025-08-15 11:30:00	69.81	1487.676	-302.790	1191.068	-94.638	33.965	-113.886	-92.429	11.5	227	0.991	-0.691	-0.723	81.89	78.14	0.801	-19.248
2025-08-15 12:00:00	78.14	1463.283	-300.232	1169.407	-21.662	72.976	-24.393	89.494	12.0	227	1.000	-0.691	-0.723	69.81	75.61	0.799	-2.731
2025-08-15 12:30:00	75.61	1494.714	-332.952	1166.331	-3.075	18.586	31.431	55.824	12.5	227	0.991	-0.691	-0.723	78.14	80.40	0.780	34.506
2025-08-15 13:00:00	80.40	1520.002	-353.697	1171.957	5.626	8.701	25.288	-6.143	13.0	227	0.966	-0.691	-0.723	75.61	80.30	0.771	19.663
2025-08-15 13:30:00	80.30	1562.429	-396.332	1169.030	-2.927	-8.553	42.427	17.138	13.5	227	0.924	-0.691	-0.723	80.40	81.06	0.748	45.354
2025-08-15 14:00:00	81.06	1659.784	-453.638	1213.280	44.250	47.177	97.355	54.929	14.0	227	0.866	-0.691	-0.723	80.30	82.05	0.731	53.105
2025-08-15 14:30:00	82.05	1776.850	-484.555	1286.842	73.562	29.312	117.066	19.711	14.5	227	0.793	-0.691	-0.723	81.06	67.50	0.724	43.504
2025-08-15 15:00:00	67.50	1635.953	-224.731	1408.419	121.577	48.015	-140.897	-257.964	15.0	227	0.707	-0.691	-0.723	82.05	79.56	0.861	-262.475
2025-08-15 15:30:00	79.56	1876.377	-259.913	1607.798	199.379	77.801	240.424	381.321	15.5	227	0.609	-0.691	-0.723	67.50	89.44	0.857	41.045
2025-08-15 16:00:00	89.44	2079.263	-217.259	1843.601	235.803	36.424	202.886	-37.538	16.0	227	0.500	-0.691	-0.723	79.56	97.72	0.887	-32.917
2025-08-15 16:30:00	97.72	2263.956	-75.293	2143.345	299.744	63.941	184.693	-18.193	16.5	227	0.383	-0.691	-0.723	89.44	100.34	0.947	-115.052
2025-08-15 17:00:00	100.34	2635.113	-86.840	2479.711	336.366	36.621	371.158	186.465	17.0	227	0.259	-0.691	-0.723	97.72	112.21	0.941	34.792
2025-08-15 17:30:00	112.21	2830.759	-5.131	2766.116	286.405	-49.961	195.646	-175.512	17.5	227	0.131	-0.691	-0.723	100.34	117.45	0.977	-90.759
2025-08-15 18:00:00	117.45	3008.087	-4.849	2947.530	181.414	-104.991	177.328	-18.318	18.0	227	0.000	-0.691	-0.723	112.21	116.63	0.980	-4.086
2025-08-15 18:30:00	116.63	3067.748	-7.591	3015.388	67.858	-113.556	59.661	-117.667	18.5	227	0.000	-0.691	-0.723	117.45	119.10	0.983	-8.198
2025-08-15 19:00:00	119.10	3048.287	-6.561	2997.098	-18.289	-86.148	-19.461	-79.121	19.0	227	0.000	-0.691	-0.723	116.63	115.35	0.983	-1.171
2025-08-15 19:30:00	115.35	3034.949	-5.822	2979.786	-17.313	0.977	-13.338	6.122	19.5	227	0.000	-0.691	-0.723	119.10	161.72	0.982	3.974
2025-08-15 20:00:00	161.72	2987.245	-6.866	2941.594	-38.192	-20.880	-47.705	-34.366	20.0	227	0.000	-0.691	-0.723	115.35	123.08	0.985	-9.512
2025-08-15 20:30:00	123.08	2934.159	-6.455	2886.364	-55.229	-17.037	-53.086	-5.381	20.5	227	0.000	-0.691	-0.723	161.72	113.93	0.984	2.144
2025-08-15 21:00:00	113.93	2890.502	-43.621	2809.188	-77.176	-21.947	-43.657	9.429	21.0	227	0.000	-0.691	-0.723	123.08	97.44	0.972	33.519
2025-08-15 21:30:00	97.44	2760.853	-31.177	2692.138	-117.050	-39.874	-129.649	-85.992	21.5	227	0.000	-0.691	-0.723	113.93	122.13	0.975	-12.599
2025-08-15 22:00:00	122.13	2602.366	-5.253	2560.531	-131.607	-14.557	-158.486	-28.837	22.0	227	0.000	-0.691	-0.723	97.44	97.83	0.984	-26.879
2025-08-15 22:30:00	97.83	2475.789	-5.591	2432.394	-128.137	3.470	-126.577	31.909	22.5	227	0.000	-0.691	-0.723	122.13	88.26	0.982	1.560
2025-08-15 23:00:00	88.26	2355.689	-5.990	2311.343	-121.052	7.086	-120.100	6.477	23.0	227	0.000	-0.691	-0.723	97.83	87.48	0.981	0.951
2025-08-15 23:30:00	87.48	2245.543	-7.040	2200.469	-110.874	10.177	-110.146	9.954	23.5	227	0.000	-0.691	-0.723	88.26	87.78	0.980	0.728
2025-08-16 00:00:00	87.78	2153.012	-6.212	2111.026	-89.443	21.432	-92.531	17.616	0.0	228	0.000	-0.704	-0.711	87.48	85.90	0.980	-3.088
2025-08-16 00:30:00	85.90	2100.603	-4.242	2055.078	-55.948	33.495	-52.409	40.122	0.5	228	0.000	-0.704	-0.711	87.78	85.90	0.978	3.539
2025-08-16 01:00:00	85.90	2055.160	-4.653	2008.223	-46.855	9.093	-45.444	6.965	1.0	228	0.000	-0.704	-0.711	85.90	85.47	0.977	1.412
2025-08-16 01:30:00	85.47	2026.839	-4.201	1980.206	-28.017	18.838	-28.321	17.123	1.5	228	0.000	-0.704	-0.711	85.90	84.20	0.977	-0.304
2025-08-16 02:00:00	84.20	2007.870	-4.040	1958.359	-21.846	6.171	-18.969	9.352	2.0	228	0.000	-0.704	-0.711	85.47	85.30	0.975	2.877
2025-08-16 02:30:00	85.30	1994.408	-6.202	1946.532	-11.827	10.019	-13.462	5.507	2.5	228	0.000	-0.704	-0.711	84.20	83.49	0.976	-1.634
2025-08-16 03:00:00	83.49	1992.032	-7.294	1940.553	-5.979	5.848	-2.376	11.085	3.0	228	0.000	-0.704	-0.711	85.30	83.83	0.974	3.602
2025-08-16 03:30:00	83.83	2011.210	-19.109	1944.082	3.529	9.507	19.178	21.555	3.5	228	0.000	-0.704	-0.711	83.49	88.47	0.967	15.650

Enhanced model — overall metrics vs persistence
Model	MAE	DirAcc
Elastic Net (alpha=0.50)	25.061	0.588
Persistence	24.908	0.523

Elastic Net — top coefficient magnitudes (asinh scale)
Feature	Estimate
s_doy	11.9838
c_doy	11.5570
wday3	0.1894
wday5	0.1633
wday6	0.1459
wday4	0.1378
wday7	0.0941
wday2	0.0526
price_now	0.0023
hod	-0.0007
d_price	-0.0006
d_scada	0.0003

Directional accuracy is improving, it’s usefulness will depend on clearing times for trades and strategy, among other factors. I’ll continue to try to improve the model and return to this at the end.

Granger causality on deltas (p = 4). Tests are skipped when x has no variation or df are insufficient.
Test	Lags	F_stat	Pr_gt_F	Note	Signif
demand -> price	4	18.516	0.000		***
scada -> price	4	21.540	0.000		***
outage -> price	4			x lags have no variation
price -> demand	4	3.130	0.015		*
price -> scada	4	7.000	0.000		***

The PACF & ACF give credibility to recent lags, I will add a rolling average for price on the same time interval. - I hope this will add something taking into account seasonality, drift and time specific factors.

Window metrics for the plotted period (same window)
Model	MAE	DirAcc
Model 1	53.586	0.632
Model 2A	56.240	0.600
Model 2B	51.187	0.600

Model 1 (green): baseline Elastic Net on a stabilized price target using only price & demand features; it establishes whether simple load/price momentum carries signal.
Model 2A (amber): adds supply (SCADA level & ramp) and outage MW, which Granger screens suggested are predictive on deltas; this should improve timing/level on ramps and constraint-driven moves.
Model 2B (purple): removes much of the intraday shape by predicting the next-interval anomaly relative to a rolling same-slot baseline—useful for isolating event-driven deviations (ramps, outages) and improving direction changes.

The residual histograms quickly show bias and dispersion by model. A Traders could use: - Bias direction (hist skew) to calibrate how much to fade or lean on a model intraday.
- Spread to decide when a model adds information vs persistence.

In reality, it’s best of a bad lot - these models are far from perfect. I suspect forecasting drivers and their dynamics would be the best next step but this is as far as I will take it. Backtesting the models show a little better than 50% accuracy in terms of direcion. So a little better than a coin toss, but then again so is persistence.

I don’t understand the mechanics of clearing trades well enough to suggest a strategy at this point. I assume it would depend on the capacity to trade as much as it does the modelling efforts.

Back-test over EVERY observation in the last 7 days (1/3/6-step horizons). Ordered by Model, then Horizon.
Model	Horizon	MAE	RMSE	DirAcc
Model 1	1 step	84.23	288.80	0.550
Model 1	3 steps	60.87	105.13	0.512
Model 1	6 steps	78.15	141.40	0.527
Model 2A	1 step	98.49	368.37	0.561
Model 2A	3 steps	59.27	103.70	0.546
Model 2A	6 steps	78.44	142.14	0.500
Model 2B	1 step	114.39	453.42	0.557
Model 2B	3 steps	63.47	127.37	0.515
Model 2B	6 steps	71.15	124.19	0.531
Persistence	1 step	37.71	76.42	0.509
Persistence	3 steps	56.42	104.29	0.519
Persistence	6 steps	72.82	131.65	0.490

Post-Reform WEM Price Forecast

Balancing Prices using Demand, Supply & Outages

Cormac Harten

2025-08-29