# Current preview: tuned models, September 2026 held-out test

The preview uses real AEMO TOTALDEMAND observations for NSW, QLD, VIC, SA and TAS. Each region is forecast independently; NEM point forecasts sum the five regional predictions. Negative demand is valid for this dispatch series and is retained.

## Data and time split

The experiment covers July 2023–September 2026. Three consecutive five-minute observations form one quarter-hour average. Missing intervals cannot contribute partial NEM totals. AEMO market timestamps use fixed UTC+10; charts show UTC.

- Training starts July 2023 and expands each month.
- Twelve validation months: August 2025–July 2026. Each fold fits only data before that month.
- Settings are selected separately for each region/model family by the equal-month mean of model MAE divided by baseline MAE. All folds must succeed; SARIMA must converge.
- Separate confirmation test: September 2026. August informed the Queensland diagnosis and is not an independent confirmation. Fitted parameters use data through August 2026 (1,158 days). Daily issues predict 96 quarter-hours. Prior observed demand updates inputs/state; there is no refitting or parameter selection during this test.

Historical observations contain final revisions. This experiment does not recreate every publication delay or data revision available to an operator at the time.

## Models

**Weighted seasonal baseline:** 40% prior-year average adjusted for recent level, 20% monthly, 25% weekly and 15% daily references. Only observations before issue time are used.

**LightGBM in NSW, VIC, SA and TAS:** 250 estimators, learning rate 0.04, 31 leaves, maximum depth 6, minimum 80 rows per leaf, L2 regularization 10, `linear_tree=true`, `linear_lambda=10`. Training-only standardization, L2 regression objective, seed 42, one compute thread. Inputs: demand lags 1/4/96/672, preceding 4/96-reading means, sine/cosine of hour and weekday indicators. Each future prediction feeds subsequent recursive steps. Eight configurations tested, including paired ordinary/linear-leaf trees.

**Queensland LightGBM:** direct correction of the weighted seasonal baseline, without recursive forecast feedback. 220 estimators, learning rate 0.035, 15 leaves, depth 5, minimum 120 rows per leaf, L2 regularization 15, absolute-error objective, `linear_tree=false`, full correction weight 1. Features include the baseline and its four components; target-aligned lags of 1/2/3/7/14/28/364 days; origin demand and recent averages; annual level adjustment; lead time, hour, weekday and day of year. Minimum 392-day context. Training origins every six hours, 25 lead offsets per origin; inference covers all 96 quarter-hours. Direct ordinary/linear-tree alternatives and four correction weights were compared on the same 12 monthly folds before September confirmation. September QLD MAE: baseline 204.97 MW, previous model 229.37 MW, revised model 185.51 MW (+9.50% FAV).

**Tuned SARIMA:** order (1,0,0), seasonal order (0,1,1,24), no trend, at most 60 optimizer iterations, stationary/invertible constraints. Fits hourly averages. Historical quarter-hour offsets from the preceding 28 days convert hourly forecasts back to 15-minute values. Four configurations tested.

Archived weather forecasts are excluded from this experiment because the available exact historical runs did not cover its full validation period.

## Metrics and uncertainty

MAE = mean absolute forecast error. FAV = 100 × (1 − model MAE / baseline MAE), using the same issue, target and actual revision. Positive FAV means less error, not monetary savings. Zero baseline error makes FAV undefined.

NEM metrics are computed after summing regional forecasts and actuals. Regional errors can cancel, so aggregate FAV is not an average of regional FAV.

The full month contains 2,880 predictions per model/region. NEM MAE: baseline 915.97 MW; tuned LightGBM 818.35 MW (+10.66% FAV); tuned SARIMA about 960.00 MW (−4.81% FAV). Results are historical evidence, not promised future accuracy.

Prediction bands use July out-of-sample residual quantiles by six-hour lead bucket, before September targets. NEM widths use aggregate residuals. Nominal 90% coverage is evaluated, not guaranteed. The preview also reports RMSE, bias, WAPE, empirical coverage and errors by lead bucket.

## Drift and forecast history

Density plots compare the fitted model's training reference with the final seven observed days of the test month. Demand, seasonally adjusted demand and changes between readings are checked. KS and Cramér–von Mises tests, Holm correction, Wasserstein distance and PSI are descriptive diagnostics; time dependence limits classical p-value interpretation.

The time slicer shows the saved daily test issues. It does not invent hourly historical forecast revisions. Hourly prospective forecasting is a separate deployment path: immutable predictions are saved first, then evaluated when observations arrive.

The preview and its CSV exports contain selected-model test results only. Private tuning runs, reconciliation experiments, fitted artifacts and credentials remain outside the public site.

## Live deployment

Live inference runs hourly in GitHub Actions with compact native model artifacts stored privately in Cloudflare R2. The initial fit uses 116,880 quarter-hour observations per region from July 2023 through October 30, 2026 02:00 UTC. Locked parameters refit monthly; inference incorporates subsequent observations. Initial live ranges use September residuals, which are prior to live targets. These ranges are not an independent coverage test on September.

Each immutable Parquet vintage records issue time, target, region, model version, point estimate and range. Later AEMO readings add evaluation records; revised readings do not rewrite predictions. The read-only API supplies the last seven days of evaluated forecasts and up to 96 recent vintages for the dashboard. All original forecast/evaluation files remain archived. Live FAV starts empty until actuals arrive. GitHub scheduling is best effort; stale data is flagged rather than represented as current.