Homework 3 - Time Series Forecasting
Lewris Mota Sanchez
2024-09-22
Imports
library(fpp3)
library(USgas)
library(gridExtra)
library(ggplot2)
library(cowplot)
options(scipen=10000)Excercise 5.1
Produce forecasts for the following series using whichever of NAIVE(y), SNAIVE(y) or RW(y ~ drift()) is more appropriate in each case:
Australian Population
aus_data = global_economy |> filter(Country == "Australia")
aus_population_train <- aus_data |> select(Population) |> filter_index("1960" ~ "2001")
aus_pop_test <- aus_data |> filter_index("2002" ~ .)|> select(Population)Australia Population Plot
aus_data |> autoplot(Population) 
Fitted Models
aus_fit <- aus_population_train |> model(Naive = NAIVE(Population),
# Snaive =SNAIVE(Population),
Rw = RW(Population ~ drift()))
aus_fc <- aus_fit |> forecast(h= 16)
aus_fc |> autoplot(aus_population_train,level = NULL) + autolayer(aus_pop_test,colour ="black")
accuracy(aus_fc,aus_data)## # A tibble: 2 × 10
## .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Naive Test 2522013. 2966158. 2522013. 11.1 11.1 11.3 13.0 0.820
## 2 Rw Test 627856. 830374. 627856. 2.70 2.70 2.82 3.63 0.827accuracy(aus_fit)## # A tibble: 2 × 10
## .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Naive Training 2.23e+ 5 228865. 222842. 1.54 1.54 1 1 0.231
## 2 Rw Training -4.09e-10 52160. 37258. 0.00366 0.262 0.167 0.228 0.231The Rw model outperforms the Naive model on all key metrics for the test data. The Rw model has a much lower ME (627,856.2) compared to the Naive model’s ME (2,522,013.4), indicating less bias. The Rw model also has significantly smaller errors, with a RMSE of 830,373.6 and MAE of 627,856.2, compared to the Naive model’s RMSE of 2,966,158.0 and MAE of 2,522,013.4. In percentage terms, the MAPE of 2.70% for Rw is far better than the Naive’s 11.05%, showing greater accuracy.
In terms of scaled error metrics, Rw’s MASE (2.82) and RMSSE (3.63) are significantly better than the Naive model’s MASE (11.32) and RMSSE (12.96), indicating much stronger performance relative to a naive forecast. Both models have similar residual autocorrelation (ACF1), but the Rw model’s overall performance is much more robust.
Bricks Production
aus_production_bricks <- aus_production |> select(Bricks) |> filter(!is.na(Bricks)) |> as_tsibble(index=Quarter)
aus_production_tr <- aus_production_bricks |> filter(year(Quarter) <= 1992)
aus_production_test <- aus_production_bricks |> filter(year(Quarter) > 1992)Bricks Production Plot
aus_production_bricks |> autoplot(Bricks) 
Fitted Models
aus_brick_fit <- aus_production_tr |> model(Naive = NAIVE(Bricks),
Snaive =SNAIVE(Bricks ~ lag("year")),
Drift = RW(Bricks ~ drift()))
aus_brick_fc <- aus_brick_fit |> forecast(h= "12 years")
# aus_fc
aus_brick_fc |> autoplot(aus_production_tr,level = NULL) + autolayer(aus_production_test,colour ="black")## Plot variable not specified, automatically selected `.vars = Bricks`
accuracy(aus_brick_fit)## # A tibble: 3 × 10
## .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Naive Training 1.57e+ 0 41.1 33.1 0.0284 8.29 0.917 0.835 -0.0835
## 2 Snaive Training 5.94e+ 0 49.3 36.1 1.32 8.83 1 1 0.807
## 3 Drift Training 1.24e-14 41.1 33.1 -0.392 8.31 0.918 0.834 -0.0835accuracy(aus_brick_fc,aus_production_bricks)## # A tibble: 3 × 10
## .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Drift Test -54.1 74.5 65.9 -14.8 17.3 1.83 1.51 0.747
## 2 Naive Test -15.6 44.7 33.3 -5.05 8.84 0.924 0.908 0.638
## 3 Snaive Test -8.88 39.4 32.8 -3.08 8.39 0.909 0.799 0.757The Snaive model performs the best in the test data, with the lowest RMSE (39.36) and MAE (32.79), indicating smaller overall and absolute prediction errors compared to the Naive and Drift models. It also has the lowest MAPE (8.39%), meaning it is the most accurate in percentage terms. The Snaive model shows lower residual autocorrelation (ACF1 = 0.7566) than the Drift model, though it is slightly higher than the Naive model.
While all models show some level of bias, Snaive has the least bias (ME = -8.88) and MPE = -3.08%, indicating it performs better than both Naive and Drift in terms of under-prediction. Additionally, its MASE (0.91) and RMSSE (0.80) indicate that it outperforms a naive forecast more effectively than the other models.
NSW Lambs
Training Data and Test Data
min_year <- min(year(aus_livestock$Month))
max_year <- max(year(aus_livestock$Month))
date <- min_year + round((max_year - min_year) * 0.7)
lambs_data <- aus_livestock |> filter(Animal == "Lambs", State == "New South Wales") |> select(Month,Count)
lambs_train <- lambs_data |> filter(year(Month) <= date)
lambs_test <- lambs_data |> filter(year(Month) > date)NSW Lambs Plot
lambs_data |> autoplot(Count) 
Fitted Models
hParam <- paste(max_year - date, "years")
lambs_fit <- lambs_train |> model(Naive = NAIVE(Count),
Snaive =SNAIVE(Count),
Drift = RW(Count ~ drift()))
lambs_fc <- lambs_fit |> forecast(h= hParam)
# aus_fc
lambs_fc |> autoplot(lambs_train,level = NULL) + autolayer(lambs_test,colour ="black")## Plot variable not specified, automatically selected `.vars = Count`
accuracy(lambs_fc,lambs_data)## # A tibble: 3 × 10
## .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Drift Test 137921. 154313. 138464. 33.5 33.7 3.06 2.59 0.749
## 2 Naive Test 78945. 91594. 80179. 18.8 19.3 1.77 1.53 0.456
## 3 Snaive Test 60128. 82860. 66749. 14.1 16.1 1.48 1.39 0.604accuracy(lambs_fit)## # A tibble: 3 × 10
## .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Naive Training -6.98e+ 2 51215. 40767. -0.922 10.0 0.902 0.858 -0.343
## 2 Snaive Training -5.88e+ 3 59686. 45208. -2.40 11.1 1 1 0.518
## 3 Drift Training 1.35e-12 51210. 40759. -0.744 10.0 0.902 0.858 -0.343Given the evident seasonality in the data, SNAIVE is the most appropriate method to forecast this time series.
The Snaive model performs best overall, with the lowest RMSE (82,859.50), MAE (66,749.40), and MAPE (16.10%), indicating more accurate predictions and fewer errors compared to the Naive and Drift models. It also outperforms them in scaled errors (MASE = 1.48, RMSSE = 1.39), though there is some residual autocorrelation (ACF1 = 0.60).
The Naive model ranks second, with higher errors (RMSE = 91,593.64, MAPE = 19.27%) but lower autocorrelation (ACF1 = 0.46), making it more reliable than the Drift model, which has the highest errors (RMSE = 154,312.95, MAPE = 33.69%) and residual autocorrelation (ACF1 = 0.75).
Household Wealth
Household wealth Plot
hh_budget |> autoplot(Wealth) + facet_grid(Country~.)
Fitted Models
wealth_fit <- hh_budget |> model(Naive = NAIVE(Wealth),
Rw = RW(Wealth ~ drift()))
wealth_fc <- wealth_fit |> forecast(h= 10)
wealth_fc |> autoplot(hh_budget,level = NULL) 
# + autolayer(lambs_test,colour ="black")accuracy(wealth_fit)## # A tibble: 8 × 11
## Country .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Australia Naive Trai… 5.10e+ 0 23.1 16.9 1.18 4.75 1 1 -0.0741
## 2 Australia Rw Trai… -1.08e-14 22.6 15.6 -0.243 4.44 0.923 0.975 -0.0741
## 3 Canada Naive Trai… 7.76e+ 0 23.9 19.6 1.48 4.28 1 1 0.0485
## 4 Canada Rw Trai… 0 22.6 17.7 -0.199 3.90 0.903 0.946 0.0485
## 5 Japan Naive Trai… 8.95e+ 0 18.6 15.9 1.72 3.03 1 1 0.101
## 6 Japan Rw Trai… -3.25e-14 16.4 12.1 -0.0196 2.28 0.761 0.877 0.101
## 7 USA Naive Trai… 6.52e+ 0 33.4 26.6 1.01 5.10 1 1 0.0565
## 8 USA Rw Trai… 1.35e-14 32.7 24.7 -0.225 4.75 0.927 0.981 0.0565Across all countries (Australia, Canada, Japan, and the USA), the Random Walk (Rw) model consistently outperforms the Naive model, demonstrating lower errors and better performance metrics. Since the time series data is yearly and does not show strong signs of seasonality, Rw is particularly well-suited for this type of trending data, as it effectively captures long-term patterns and trends.
Australia: The Rw model shows better accuracy, with a lower RMSE (22.58) and MAE (15.61) compared to the Naive model’s RMSE (23.15) and MAE (16.91). The MASE (0.92) and RMSSE (0.98) confirm that the Rw model captures the trend more accurately.
Canada: With a lower RMSE (22.57) and MAE (17.73) than the Naive model, Rw also exhibits better accuracy and less error, making it more reliable for this trending dataset.
Japan: The Rw model performs significantly better than Naive, with a much lower RMSE (16.36) and MAE (12.10), highlighting its strength in trending data. The MAPE (2.28%) is also the lowest across all countries, further confirming the model’s accuracy.
USA: Again, the Rw model shows improvement over Naive, with a lower RMSE (32.71) and MAE (24.69), along with better scaled error measures, making it the best choice for tracking trends in this yearly data.
Food Turnover
Australian takeaway food turnover Plot
turnover_data <- aus_retail |> filter(Industry == "Takeaway food services") |> select(Turnover) |> select(-Industry)turnover_data |> autoplot(Turnover) + facet_wrap(State~., scales='free_y', ncol = 2 )+ labs(y="Turnover in millions AUD",x="Months")
turnover_fit <- turnover_data |> model(Naive = NAIVE(Turnover),
Snaive =SNAIVE(Turnover),
Drift = RW(Turnover ~ drift()))
turnover_fc <- turnover_fit |> forecast(h= 10)
# aus_fc
turnover_fc |> autoplot(turnover_data, level = NULL) + facet_wrap(State~., scales='free_y', ncol = 2 )+ labs(y="Turnover in millions AUD",x="Months")
accuracy(turnover_fit)## # A tibble: 24 × 11
## State .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Austra… Naive Trai… 5.89e- 2 0.951 0.666 0.173 5.94 0.473 0.517 -0.241
## 2 Austra… Snaive Trai… 6.31e- 1 1.84 1.41 4.06 12.2 1 1 0.838
## 3 Austra… Drift Trai… 2.39e-16 0.950 0.666 -0.453 5.97 0.474 0.516 -0.241
## 4 New So… Naive Trai… 1.24e+ 0 20.3 13.6 0.207 5.37 0.587 0.652 -0.270
## 5 New So… Snaive Trai… 1.33e+ 1 31.2 23.2 4.53 9.32 1 1 0.901
## 6 New So… Drift Trai… -4.33e-15 20.3 13.6 -0.454 5.37 0.585 0.651 -0.270
## 7 Northe… Naive Trai… 4.43e- 2 1.07 0.707 0.0116 6.88 0.410 0.400 -0.228
## 8 Northe… Snaive Trai… 4.97e- 1 2.68 1.72 3.10 14.7 1 1 0.881
## 9 Northe… Drift Trai… 3.35e-16 1.07 0.710 -0.536 6.94 0.412 0.399 -0.228
## 10 Queens… Naive Trai… 7.55e- 1 12.9 8.69 0.338 5.39 0.689 0.760 -0.322
## # ℹ 14 more rowsThe Drift model emerges as the best performer across the various states, consistently demonstrating the lowest error metrics and effectively handling trends in the training data. In states like the Australian Capital Territory, New South Wales, and Queensland, the Drift model outperforms both the Naive and Snaive models by delivering lower RMSE and MAE, while maintaining no bias, as evidenced by an ME close to zero. For example, in New South Wales, the Drift model achieves an RMSE of 20.29 and an MAE of 13.55, both superior to the other models, making it the most accurate at capturing trends. Similarly, in regions such as Victoria, Tasmania, and South Australia, the Drift model consistently provides better accuracy, with significantly lower error rates than the Naive and Snaive models. The Snaive model, designed to capture seasonality, performs poorly due to the trending nature of the data, while the Naive model, although consistent, fails to account for trends as effectively as the Drift model. Overall, the Drift model’s ability to handle trend-dominated data while maintaining low error rates and minimal bias makes it the most reliable and accurate model for forecasting across all states.
Excercise 5.2
Use the Facebook stock price (data set gafa_stock) to do the following.
Facebook stock Data
fb_stock = gafa_stock |> filter(Symbol == "FB") |> mutate(day = row_number()) |> update_tsibble(index = day,regular = TRUE)
head(fb_stock)## # A tsibble: 6 x 9 [1]
## # Key: Symbol [1]
## Symbol Date Open High Low Close Adj_Close Volume day
## <chr> <date> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <int>
## 1 FB 2014-01-02 54.8 55.2 54.2 54.7 54.7 43195500 1
## 2 FB 2014-01-03 55.0 55.7 54.5 54.6 54.6 38246200 2
## 3 FB 2014-01-06 54.4 57.3 54.0 57.2 57.2 68852600 3
## 4 FB 2014-01-07 57.7 58.5 57.2 57.9 57.9 77207400 4
## 5 FB 2014-01-08 57.6 58.4 57.2 58.2 58.2 56682400 5
## 6 FB 2014-01-09 58.7 59.0 56.7 57.2 57.2 92253300 6Produce a time plot of the series.
fb_stock |> autoplot(Close) 
Produce forecasts using the drift method and plot them.
fb_stock_fit <- fb_stock |> model(Drift= RW(Close ~ drift()))
fb_stock_fc <- fb_stock_fit |> forecast(h = 180)
fb_stock_fc |> autoplot(fb_stock)
Show that the forecasts are identical to extending the line drawn between the first and last observations.
Y <- (fb_stock$Close)[1]
yEnd <- (fb_stock[nrow(fb_stock),])$Close
xEnd <- nrow(fb_stock)
fb_stock_fc |> autoplot(fb_stock,level = NULL) + geom_segment(aes(x = 1, y = Y, xend = xEnd, yend = yEnd),
colour = "blue", linetype = "dashed")
Try using some of the other benchmark functions to forecast the same data set. Which do you think is best? Why?
fb_stock_fit_multi <- fb_stock |> model(Naive = NAIVE(Close),
MEAN(Close),
Drift= RW(Close ~ drift()))
fb_stock_fc_multi <- fb_stock_fit_multi |> forecast(h = 180)
fb_stock_fc_multi |> autoplot(fb_stock,level = FALSE)
Trained model Evaluation
accuracy(fb_stock_fit_multi)## # A tibble: 3 × 11
## Symbol .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 FB Naive Trai… 6.08e- 2 2.41 1.47 5.15e-2 1.26 1 1 -0.0205
## 2 FB MEAN(C… Trai… -1.23e-15 41.3 35.6 -1.36e+1 34.5 24.2 17.1 0.997
## 3 FB Drift Trai… 0 2.41 1.46 -5.71e-3 1.26 0.998 1.00 -0.0205The Drift model is the best performer for FB, showing no bias and the lowest error metrics, with an RMSE of 2.41 and MAE of 1.46, slightly outperforming the Naive model, which has nearly identical values. However, the Drift model is more appropriate for this dataset because it accounts for the underlying trend in the data, making it better suited for time series with clear trends. The Naive model performs well but doesn’t handle trends as effectively. In contrast, the MEAN(Close) model performs poorly, with high errors and significant bias. Overall, Drift is the most accurate and reliable choice, especially given the trending nature of the data.
Excercise 5.3
Apply a seasonal naïve method to the quarterly Australian beer production data from 1992. Check if the residuals look like white noise, and plot the forecasts.
# Extract data of interest
recent_production <- aus_production |> filter(year(Quarter) >= 1992)
recent_production## # A tsibble: 74 x 7 [1Q]
## Quarter Beer Tobacco Bricks Cement Electricity Gas
## <qtr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 1992 Q1 443 5777 383 1289 38332 117
## 2 1992 Q2 410 5853 404 1501 39774 151
## 3 1992 Q3 420 6416 446 1539 42246 175
## 4 1992 Q4 532 5825 420 1568 38498 129
## 5 1993 Q1 433 5724 394 1450 39460 116
## 6 1993 Q2 421 6036 462 1668 41356 149
## 7 1993 Q3 410 6570 475 1648 42949 163
## 8 1993 Q4 512 5675 443 1863 40974 138
## 9 1994 Q1 449 5311 421 1468 40162 127
## 10 1994 Q2 381 5717 475 1755 41199 159
## # ℹ 64 more rows# Define and estimate a model
fit <- recent_production |> model(SNAIVE(Beer))
# Look at the residuals
fit |> gg_tsresiduals()
# Look a some forecasts
fit |> forecast() |> autoplot(recent_production)
What do you conclude?
Based on the plots, the residuals exhibit a bimodal distribution, indicating that the model is likely missing important features or patterns, such as nonlinearity or latent subgroups within the data. Additionally, the autocorrelation function (ACF) shows a single significant value outside the Bartlett bands, with a correlation of around -0.7 to -0.8. This strong negative autocorrelation suggests a meaningful temporal dependency at lag 4, further confirming that the model has not fully captured the underlying structure of the data. Therefore, the model needs to be refined in order to capture all the remaining patterns in the data.
Excercise 5.4
Repeat the previous exercise using the Australian Exports series from global_economy and the Bricks series from aus_production. Use whichever of NAIVE() or SNAIVE() is more appropriate in each case.
Australian Exports
# Extract data of interest
aus_prod <- global_economy |> filter(Code == "AUS")
aus_prod## # A tsibble: 58 x 9 [1Y]
## # Key: Country [1]
## Country Code Year GDP Growth CPI Imports Exports Population
## <fct> <fct> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Australia AUS 1960 18573188487. NA 7.96 14.1 13.0 10276477
## 2 Australia AUS 1961 19648336880. 2.49 8.14 15.0 12.4 10483000
## 3 Australia AUS 1962 19888005376. 1.30 8.12 12.6 13.9 10742000
## 4 Australia AUS 1963 21501847911. 6.21 8.17 13.8 13.0 10950000
## 5 Australia AUS 1964 23758539590. 6.98 8.40 13.8 14.9 11167000
## 6 Australia AUS 1965 25931235301. 5.98 8.69 15.3 13.2 11388000
## 7 Australia AUS 1966 27261731437. 2.38 8.98 15.1 12.9 11651000
## 8 Australia AUS 1967 30389741292. 6.30 9.29 13.9 12.9 11799000
## 9 Australia AUS 1968 32657632434. 5.10 9.52 14.5 12.3 12009000
## 10 Australia AUS 1969 36620002240. 7.04 9.83 13.3 12.0 12263000
## # ℹ 48 more rows# Define and estimate a model
aus_exports_fit <- aus_prod |> model(Naive=NAIVE(Exports))
# Look a some forecasts
aus_exports_fit |> forecast(h=10) |> autoplot(aus_prod)
Naive Residuals
aus_exports_fit |> select(Naive) |> gg_tsresiduals()
What do you conclude?
Based on the plots, the residuals exhibit a normal distribution with a mean around 0 and constant variance, indicating that the model is well-specified. Additionally, the autocorrelation function (ACF) shows a single value that falls outside the Bartlett bands, but with a value of -0.3, it is not strong enough to be considered significant. Therefore, we can conclude that the model has successfully captured the underlying structure of the data, and the residuals resemble white noise, suggesting a good fit.
Australian Production
aus_brick_prod <- aus_production |> filter(!is.na(Bricks))
# Define and estimate a model
aus_brick_fit <- aus_brick_prod |> model(Naive=NAIVE(Bricks),
Snaive =SNAIVE(Bricks ~ lag("year")),
)
# Look a some forecasts
aus_brick_fit |> forecast(h=10) |> autoplot(aus_brick_prod, level=NULL)
Naive Residuals
aus_brick_fit |> select(Naive) |> gg_tsresiduals()
Based on the plots, the residuals exhibit a left-skewed bimodal distribution, suggesting that the model is likely missing important features or patterns. Additionally, the autocorrelation function (ACF) shows multiple values exceeding the Bartlett bands in both directions, occurring at regular intervals of 4 lags, indicating the presence of seasonality. This suggests that the naive model may not be appropriate for the data, and a model like SNAIVE, which accounts for seasonality, would be a better fit.
SNAIVE Residuals
aus_brick_fit |> select(Snaive) |> gg_tsresiduals()
What do you conclude?
Based on the plot for the seasonal naive model, the residuals exhibit a left-skewed distribution, and the autocorrelation function (ACF) shows multiple values exceeding the Bartlett bands in both directions, though at less frequent intervals. The sinusoidal pattern in the ACF suggests the presence of cyclic behavior in the data, which the model has not fully captured.
Overall, while the seasonal naive model is a better fit than the naive model, the residuals still do not resemble white noise, indicating that important patterns, particularly the cyclic nature, have not been exploited. This suggests that a more sophisticated model could be applied to better fit the data.
Excercise 5.7
For your retail time series (from Exercise 7 in Section 2.10).
set.seed(12345678)
myseries <- aus_retail |>
filter(`Series ID` == sample(aus_retail$`Series ID`,1))Create a training dataset consisting of observations before 2011.
myseries_train <- myseries |>
filter(year(Month) < 2011)Check that your data have been split appropriately.
autoplot(myseries, Turnover) +
autolayer(myseries_train, Turnover, colour = "red")
Fit a seasonal naïve model using SNAIVE().
fit <- myseries_train |>
model(SNAIVE(Turnover))Check the residuals.
fit |> gg_tsresiduals()## Warning: Removed 12 rows containing missing values or values outside the scale range
## (`geom_line()`).## Warning: Removed 12 rows containing missing values or values outside the scale range
## (`geom_point()`).## Warning: Removed 12 rows containing non-finite outside the scale range
## (`stat_bin()`).
Do the residuals appear to be uncorrelated and normally distributed?
The ACF plot reveals some autocorrelation in the data. The histogram shows that the residuals are slightly left-skewed and not perfectly centered around zero, though their overall shape closely approximates a normal distribution.
Produce forecasts for the test data.
fc <- fit |>
forecast(new_data = anti_join(myseries, myseries_train))## Joining with `by = join_by(State, Industry, `Series ID`, Month, Turnover)`fc |> autoplot(myseries)
Compare the accuracy of your forecasts against the actual values.
fit |> accuracy()## # A tibble: 1 × 12
## State Industry .model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Norther… Clothin… SNAIV… Trai… 0.439 1.21 0.915 5.23 12.4 1 1 0.768fc |> accuracy(myseries)## # A tibble: 1 × 12
## .model State Industry .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
## <chr> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 SNAIVE(T… Nort… Clothin… Test 0.836 1.55 1.24 5.94 9.06 1.36 1.28 0.601The comparison between training and test data shows that the model struggles to generalize, with increased bias and larger prediction errors on unseen data. The Mean Error (ME) rose from 0.44 to 0.84, indicating greater over-prediction, while both RMSE and MAE increased, showing larger deviations from actual values on the test set. This suggests overfitting, as the model performs worse when applied to new data.
Although MAPE improved from 12.40% to 9.06%, suggesting better percentage accuracy on the test data, the rise in MASE and RMSSE above 1.00 (1.36 and 1.28) shows the model performed worse than the naive forecast. This highlights that the model fails to add meaningful predictive value over a simple baseline.
Finally, while ACF1 dropped from 0.77 to 0.60, indicating slight improvement in capturing time-dependence, significant autocorrelation remains in the residuals. This suggests the model has not fully captured key cyclic or structural patterns, emphasizing the need for a more refined approach to improve performance.
How sensitive are the accuracy measures to the amount of training data used?
The amount of training data used has a significant impact on the accuracy of time series models. With a smaller training dataset, accuracy metrics such as ME, RMSE, and MAE tend to be more volatile, leading to higher errors. The model may struggle to capture meaningful patterns due to the limited data available, resulting in less reliable predictions. In this scenario, the model is more likely to overfit or miss important trends, causing metrics like MAPE to become skewed, especially if there are outliers or small actual values in the data. Additionally, measures such as MASE and RMSSE may show that the model performs no better than a naive forecast, as the model lacks sufficient data to learn beyond simple time-dependent patterns. ACF1 (autocorrelation at lag 1) might indicate a failure to capture the time-dependent structure in the data, as residuals tend to remain autocorrelated when the training set is too small.
In contrast, with a larger training dataset, these accuracy metrics tend to stabilize, showing improved performance. As the model has access to more data, it is better able to learn complex patterns, reducing both errors and overfitting. RMSE, MAE, and MAPE will generally decrease, indicating more accurate predictions and fewer large deviations. MASE and RMSSE are likely to show improved performance relative to the naive forecast, demonstrating that the model has captured important trends and seasonal behaviors. Similarly, ACF1 values will typically decrease, reflecting reduced autocorrelation in residuals as the model successfully learns time-dependent structures. Overall, the more data available, the more robust and accurate the model’s predictions become, with accuracy metrics reflecting these improvements.