segen: a brief introduction

When is a sequence considered “general”?

Segen is a model for sequence generalization using the “network” of similarities among sequences for the extrapolation of the next sequence. The notion of “network” here is related to the computation of a matrix of distances that is converted to the equivalent of an adjacency matrix using a similarity threshold. The idea behind segen is that the lower triangle of such matrix is enough to predict (to some extent) the behavior of a sequence of sequences.

A brief overview on the process:

1. The test errors are cross-validated through expanding validation windows, where the default value of n_windows is set to 10, meaning that the time features are divided into 10 + 1 segments guaranteeing at least ten validation sets to measure the error on unforeseen data.

2. Some basic transformation are directly managed in background. Differentiation and integration are automatically managed by segen using the maximal p-value in a recursive F-test for de-trending each time-feature: this allows to easily determine the different dynamic characteristics of each time feature, random walk, trend, exponential (somehow more simple and practical compared to other formal approaches like Augmented Dickey-Fuller or Ljung Box Test). If you have limited missing values in your time features, segen automatically proceeds with the imputation using the Kalman filter method¹. If you prefer to project into the future the smoothed version, you can set smoother = TRUE to use loess² function.

3. After differentiation, each time features is reframed according to sequence length (each time feature is segmented in sequences of seq_len). A distance metric is calculated among each sequence and the value are compared to a similarity threshold (the quantile value of distances, where 0 means maximum and 1 minimum difference respectively). Location and scales parameters are calculated for each column and used to simulate weights used to calculate the generalized sequence as weighted average. The weights could optionally be rescaled using min-max normalization so to make the generalization somehow more “fuzzy”.

4. For each point in the prediction sequence, a thousand samples are collected for the calculation of quantiles, mean, mode, standard deviation, skewness and kurtosis, and other less common measures.

That’s all. The process is quite easy and relatively fast (as always, the only bottleneck is represented by the quadratic complexity in the calculation of distance matrix for very large sets of sequences, but that’s another issue).

The process flow of segen

Standing on the shoulders’ of (Tech) giants

The dataset time features included with segen is a recent take on some Big Techs’ stock prices (source: Yahoo Finance). The data is expected in a dataframe format, where each column represents a different time series (the date information is not mandatory and could be provided separately).

In the first example, we are predicting the close price for IBM and Microsoft. In this example we try to set seq_len = 20 (sequence length), using a cross-validation scheme of 5 n_windows for error measurement.

example1 <- segen(time_features, seq_len = 20, n_windows = 5,  n_samp = 10, dates = rownames(time_features))
  time: 163.96 sec elapsed

The result is a list of different components, as you can see below.

names(example1)
  [1] "exploration" "history"     "best_model"  "time_log"
names(example1$best_model)
  [1] "exploration"    "predictions"    "testing_errors" "plots"

exploration includes all models tested during the exploration.history includes selected parameters and error metrics for the explored space during random search (beside prediction score, me, mae, mse, rmsse, mpe, mape, rmae, rrmse, rame, mase, smse, sce, gmrae ³, averaged across features and validation windows). best_model collects a list of information for the best model selected according to the average error metric: you will find the prediction intervals (predictions), the visualizations (plots) and the testing error metric for each time feature (testing_errors).

The predictions is a list including the predicted results for each time-feature (quantile, min, max, mean, mode, sd, skewness, kurtosis, iqr to range, median range ratio, upside probability, divergence for each time point in the seq_len sequence). The IQR to range is the interquartile range to the min-max range, the median range ratio is the range above median to the range below it, the upside probability is the probability of growth compared to the former point in the time sequence, the divergence is the maximum distance of cumulative normal curve of each point to the former point in the sequence.

For each time features included in the model, you get a plot of the median with the chosen confidence interval (ci default is 0.8). As in other packages⁴, we provide different stats to give a better hint on the different dynamics related to aleatoric and epistemic uncertainty.

  $IBM.Close

  
  $MSFT.Close

Wandering around the hyper-parameter space

The hyper-parameter space is defined by seq_len, dist_method, similarity and rescale. Now, let’s try a random search for the best parameter settings. The following example shows how to sample 30 different models for a sequence of 20 time steps.

example2 <- segen(time_features, seq_len = 20, n_samp = 30, n_windows = 5, dates = rownames(time_features))
  time: 360.14 sec elapsed

If we compare the error statistics from the best model in example2 with the model in example1, for IBM and Microsoft we see consistent improvement. All the relative and scaled error metrics defaults to segen, but you can choose more challenging thresholds (like the deviation of the whole time feature or the average of the whole predicted sequence).

The error statistics from example1 (averaged across 10 expanding validation windows):

example1$best_model$testing_errors
                 me    mae     mse  rmsse    mpe   mape   rmae  rrmse   rame
  IBM.Close  3.0814 4.3986 32.5930 4.0150 0.0234 0.0338 1.1202 1.1316 1.2604
  MSFT.Close 1.7566 5.7480 50.7096 4.1506 0.0064 0.0222 0.8200 0.8740 2.3276
               mase    smse     sce  gmrae
  IBM.Close  2.9098 22.1456 41.0728 1.0792
  MSFT.Close 1.9926 17.5462 12.3304 0.6554

The error statistics from example2 (as above, averaged across 10 expanding validation windows):

example2$best_model$testing_errors
                 me   mae     mse  rmsse    mpe   mape   rmae  rrmse   rame
  IBM.Close  3.0196 4.371 31.9238 3.9840 0.0230 0.0336 1.1188 1.1292 1.2572
  MSFT.Close 1.7416 5.758 50.7996 4.1546 0.0064 0.0224 0.8216 0.8748 2.3410
               mase    smse     sce  gmrae
  IBM.Close  2.8904 21.6836 40.2366 1.0872
  MSFT.Close 1.9960 17.5792 12.2248 0.6648

The improvement is somehow unclear between the two examples, but we are still using a naive approach to measure scaled and relative errors. Let’s try to shift to deviation as scale, and average as benchmark, that are more challenging evaluation criteria, and extend to 100 samples.

example3 <- segen(time_features, seq_len = 20, n_windows = 5, dates = rownames(time_features), error_scale = "deviation", error_benchmark = "average", n_samp = 100)
  time: 1282.79 sec elapsed

As you can see, the relative and scaled measures shift sensibly as we change the bar of our expectations:

example3$best_model$testing_errors
                 me    mae     mse  rmsse    mpe   mape   rmae  rrmse rame   mase
  IBM.Close  3.0090 4.4084 32.3898 1.9778 0.0230 0.0338 1.0928 1.0548    1 0.7206
  MSFT.Close 1.3992 5.7070 50.8614 1.5518 0.0052 0.0222 0.9808 0.9970    1 0.2892
               smse     sce  gmrae
  IBM.Close  4.8464 10.4262 1.1220
  MSFT.Close 2.6318  1.4278 0.9256

Some useful references