R Markdown

Build an auto.arima model as well as an ETS model. Which performed better? Now hold out 6 months of data for a test set and try to forecast using the ETS and the auto.arima. Which performs better on the hold-out set?

Importing the data and getting it ready to use

library(ggplot2)
library(tidyr)
library(forecast)
## Registered S3 method overwritten by 'quantmod':
##   method            from
##   as.zoo.data.frame zoo
setwd('/Users/yeshimonipede/Desktop/BC_Spring2024')
USD_TRY<-read.csv('USD_TRY Historical Data.csv')


dates <- as.Date(USD_TRY[, 1])  # Convert date column to Date class
exchange_rates <- USD_TRY[, 2]  # Extract exchange rate column

#Converting to a time series
USD_TRY_ts<-ts(exchange_rates, start = c(1990,2), frequency = 12)
USD_TRY_ts <- ts(rev(USD_TRY_ts), start = start(USD_TRY_ts), frequency = frequency(USD_TRY_ts))

Creating a train and test set

# Define the start and end dates for training and test sets
start_train <- c(1990, 2)
end_train <- c(2023, 9)  # End of September 2023
start_test <- c(2023, 10)  # Start of October 2023
end_test<-c(2024,3)

# Create training and test sets
USD_TRY_ts_train <- window(USD_TRY_ts, start = start_train, end = end_train)
USD_TRY_ts_test <- window(USD_TRY_ts, start = start_test,end=end_test)

Running the ARIMA model

(fit.arima<-auto.arima(USD_TRY_ts_train))
## Series: USD_TRY_ts_train 
## ARIMA(2,2,3)(0,0,1)[12] 
## 
## Coefficients:
##          ar1      ar2      ma1     ma2     ma3     sma1
##       0.9334  -0.1119  -1.7746  0.6549  0.1444  -0.2196
## s.e.  0.2100   0.2093   0.2095  0.4118  0.2051   0.0834
## 
## sigma^2 = 0.1164:  log likelihood = -137
## AIC=288   AICc=288.29   BIC=315.98

Checking the residuals

checkresiduals(fit.arima)

## 
##  Ljung-Box test
## 
## data:  Residuals from ARIMA(2,2,3)(0,0,1)[12]
## Q* = 97.506, df = 18, p-value = 6.327e-13
## 
## Model df: 6.   Total lags used: 24

The residuals do not look very similar to white noise which could indicate that the model is not capturing the training data well. The p-value for the the ARIMA model is very small (p-value = 6.327e-13) which also may mean that the ARIMA model is not fitting the data well. Also, the residuals do not all fit in the range of the significance bands on the ACF plot. This may be another indication that the data is not being explained well by this model.

Running the ETS model

(fit.ets<-ets(USD_TRY_ts_train))
## ETS(M,A,N) 
## 
## Call:
##  ets(y = USD_TRY_ts_train) 
## 
##   Smoothing parameters:
##     alpha = 0.9999 
##     beta  = 0.1705 
## 
##   Initial states:
##     l = 0.0023 
##     b = 1e-04 
## 
##   sigma:  0.0555
## 
##       AIC      AICc       BIC 
## -197.3376 -197.1868 -177.3305

Checking the residuals

checkresiduals(fit.ets)

## 
##  Ljung-Box test
## 
## data:  Residuals from ETS(M,A,N)
## Q* = 26.372, df = 24, p-value = 0.3346
## 
## Model df: 0.   Total lags used: 24

The residuals look fairly similar to white noise which is a good sign in terms of model fit. The p-value for the the ETS is quite large (p=0.3346) which also supports that the model is a good fit. Also, the residuals all fit in the range of the significance bands on the ACF plot. This also suggests that the data is being captured well by the model.

Generate forecasts from an ARIMA model

USD_TRY_ts_train %>% auto.arima()%>% forecast(h=12)%>%autoplot

Generate forecasts from an ETS model

USD_TRY_ts_train %>% ets()%>% forecast(h=12)%>%autoplot

Evaluation of forecasting performance between the two models between the train and test set

acc_ARIMA <-fit.arima%>%forecast(h=6)%>%accuracy(USD_TRY_ts_test)
acc_ARIMA[,c("RMSE","MAE","MAPE","MASE")]
##                   RMSE       MAE      MAPE      MASE
## Training set 0.3377558 0.1106360 3.4553664 0.1559670
## Test set     0.3462032 0.2949322 0.9587556 0.4157752
acc_ETS<-fit.ets%>%forecast(h=6)%>%accuracy(USD_TRY_ts_test)
acc_ETS[,c("RMSE","MAE","MAPE","MASE")]
##                   RMSE       MAE     MAPE      MASE
## Training set 0.3613771 0.1095107 3.275577 0.1543806
## Test set     0.6389204 0.5692167 1.859810 0.8024425

Both models have similar performance across most metrics for the training set. However, the ETS model has slightly lower values for MAE, MAPE, and MASE compared to the ARIMA model, indicating slightly better performance in terms of the average error, average magnitude of errors, and percentage errors. The ETS model had slightly higher RMSE scores. It is fair to say that generally, the ETS model performed better on the training data based off of these metrics.

However, the ARIMA model fits the test set data substantially better than the ETS model. The RMSE, MAE, MAPE, and MASE scores are almost twice as large for the ARIMA model versus when assessing the ETS model. It is fair to say that the ARIMA model performed considerably better on the test set.

The AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion) values are measures of the model’s fit to the data. Lower values indicate better fit: ARIMA Metrics: AIC=288 BIC=315.98 ETS Metrics: AIC = -197.3376 BIC =-177.3305

The ETS model has lower ETS and BIC scores which suggests that it does a better job fitting the training data than the ARIMA model does. However, it is important to note that these metrics capture how well the models fit the TRAINING data rather than the test data.

In conclusion, evidence gathered from analyzing the residuals and their corresponding plots as well as considering AIC and BIC values, as well as analyzing the RMSE, MAE, MAPE, and MASE scores suggests that the ETS model does a substantially better job fitting the TRAINING DATA. However looking at the RMSE, MAE, MAPE, and MASE scores of the TEST DATA suggest that the ARIMA does a much better job fitting the test data.