naive: a brief introduction

The simplest baseline

naive is the simplest possible model you can imagine to forecast a sequence of values: it selects the most common “patterns” of sequences to propose an empirical extrapolation for the future sequence. The idea behind naive is that of having a minimal benchmark for comparing most sophisticated models (both in term of computation time and testing error of prediction).

A brief overview on the process:

1. Some basic transformation are directly managed in background. Differentiation and integration are automatically managed by naive using the maximal p-value in a recursive F-test for de-trending each time-feature: this allows to easily determine the different dynamic characteristics of each time feature, random walk, trend, exponential (somehow more simple and practical compared to other formal approaches like Augmented Dickey-Fuller or Ljung Box Test). If you have limited missing values in your time features, naive automatically proceeds with the imputation using the Kalman filter method¹. If you prefer to project into the future the smoothed version, you can set smoother = TRUE to use loess² function.

2. After differentiation, each time features is reframed according to sequence length and stride (each time feature is segmented in sequences of seq_len shifted by stride), defining a data set of temporal segments that could be overlapping (when stride is less than seq_len) or not (when stride is equal or greater than seq_len). A distance metric is calculated among each sequence and the location parameter of distance matrix is calculated (mean, median or mode, diagonal excluded): the most recurring patterns are extracted selecting the sequences whom distances are inside a percentile cover built around the location parameter. The selected patterns are used as empirical distribution to simulate the possible future values and calculate the quantile prediction.

3. The test errors are cross-validated through an expanding validation n_windows: the default value is set to 10, meaning that the time features are divided into 10 + 1 segments guaranteeing at least ten validation sets to measure the error on unforeseen data. For each point in the prediction sequence, a thousand samples are collected for the calculation of quantiles, mean, mode, standard deviation, skewness and kurtosis, and other less common measures (see below).

The process flow of naive

For example, let’s look at tech giants’ stocks …

The dataset time features included with naive is a recent take on some Big Techs’ stock prices (source: Yahoo Finance). The data is expected in a dataframe format, where each column represents a different time series (the date information is not mandatory and could be provided separately).

In the first example, we are predicting the close price for Amazon and Google In this example we try to set seq_len = 50 (sequence length), using a cross-validation scheme of 10 n_windows for error measurement.

example1 <- naive(time_features[, 4:5], seq_len = 30, n_windows = 10,  dates = time_features$date)
  time: 89.5 sec elapsed

The result is a list of different components, as you can see below.

names(example1)
  [1] "exploration" "history"     "best_model"  "time_log"
names(example1$best_model)
  [1] "quant_preds" "plots"       "errors"

exploration takes in all models tested during the exploration.history includes selected parameters and error metrics for the explored space during random search (beside prediction score, me, mae, mse, rmsse, mpe, mape, rmae, rrmse, rame, mase, smse, sce, gmrae ³, averaged across features and validation windows). best_model collects a list of information for the best model selected according to the average error metric: you will find the prediction intervals (quant_preds), the visualizations (plots) and the testing error metric for each time feature (errors).

The quant_preds is a list including the predicted results for each time-feature (quantile, min, max, mean, mode, sd, skewness, kurtosis, iqr to range, median range ratio, upside probability, divergence for each time point in the seq_len sequence). The IQR to range is the interquartile range to the min-max range, the median range ratio is the range above median to the range below it, the upside probability is the probability of growth compared to the former point in the time sequence, the divergence is the maximum distance of cumulative normal curve of each point to the former point in the sequence.

For each time features included in the model, you get a plot of the median with the chosen confidence interval (ci default is 0.8). As in other packages⁴, we provide different stats to give a better hint on the different dynamics related to aleatoric and epistemic uncertainty.

  $AMZN.Close

  
  $GOOGL.Close

An exploration of the hyper-parameter space

The hyper-parameter space defined by seq_len, method, location and cover is kind of huge. Now, let’s try a random search for the best parameter settings. The following example shows how to sample 100 different models for a sequence of 30 time steps.

example2 <- naive(time_features[, 4:5], seq_len = 30, n_samp = 100, n_windows = 10, dates = time_features$date)
  time: 316.6 sec elapsed

If we compare the error statistics from the best model in example2 with the model in example1, for Amazon and Google we see consistent improvement. All the relative and scaled error metrics defaults to naive, but you can choose more challenging thresholds (like the deviation of the whole time feature or the average of the whole predicted sequence).

The error statistics from example1 (averaged across 10 expanding validation windows):

example1$best_model$errors
              pred_score       me      mae      mse   rmsse    mpe   mape   rmae
  AMZN.Close      0.1338 169.8016 185.0265 55874.76 53.3262 0.0794 0.0877 2.2428
  GOOGL.Close     0.0992 101.4664 107.1904 16207.96 37.4059 0.0716 0.0751 2.6984
               rrmse    rame    mase     smse      sce  gmrae
  AMZN.Close  2.2683  4.1215 11.3708 3154.579 316.5591 2.0452
  GOOGL.Close 2.6257 15.7802  9.9103 1432.759 283.6309 2.6248

The error statistics from example2 (as above, averaged across 10 expanding validation windows):

example2$best_model$errors
              pred_score       me      mae      mse   rmsse    mpe   mape   rmae
  AMZN.Close      0.1357 170.7348 185.8427 56560.19 53.4571 0.0796 0.0879 2.2483
  GOOGL.Close     0.0995 101.1778 106.9875 16167.70 37.3364 0.0712 0.0748 2.6893
               rrmse    rame    mase     smse      sce  gmrae
  AMZN.Close  2.2717  4.1114 11.3971 3181.336 317.4391 2.0492
  GOOGL.Close 2.6163 15.4933  9.8975 1429.830 282.9264 2.6217

The improvement is clear for both the time features, but we are still using a naive approach to measure scaled and relative errors. Let’s try to shift to deviation as scale, and average as benchmark, that are more challenging evaluation criteria.

example3 <- naive(time_features[, 4:5], seq_len = 30, n_windows = 10, dates = time_features$date, error_scale = "deviation", error_benchmark = "average")
  time: 88.72 sec elapsed

As you can see, the relative and scaled measures change sensibly as we raise the bar of our expectations:

example3$best_model$errors
              pred_score       me      mae      mse   rmsse    mpe   mape   rmae
  AMZN.Close      0.1338 169.8016 185.0265 55874.76 11.9350 0.0794 0.0877 1.0797
  GOOGL.Close     0.0992 101.4664 107.1904 16207.96 10.3578 0.0716 0.0751 1.0544
               rrmse rame   mase     smse     sce  gmrae
  AMZN.Close  1.1810    1 0.6396 165.4390 18.1701 0.8392
  GOOGL.Close 1.1393    1 0.7990 112.7934 23.0830 0.8762

Some useful references