Seasonal Trend Decomposition

Seasonal Trend Decomposition can be used for anomaly detection. The seasonal trend decomposition using Loess(STL) is an algorithm that was developed to help to divide up a time series into three components namely: the trend, seasonality and remainder.

The use of the stl function can be demonstrated using one of the data sets available within the base R installation. The well used nottem data set (Average Monthly Temperatures at Nottingham, 1920-1939) is a good starting point.

head(nottem)
## [1] 40.6 40.8 44.4 46.7 54.1 58.5
modelStl <- stl(nottem, s.window = "periodic")

plot(modelStl)

The four graphs are the original data, seasonal component, trend component and the remainder and this shows the periodic seasonal pattern extracted out from the original data and the trend that moves around between 47 and 51 degrees Fahrenheit. There is a bar at the right hand side of each graph to allow a relative comparison of the magnitudes of each component. For this data the change in trend is less than the variation doing to the monthly variation.

Twitter’s introduction to practical and robust anomaly detection in a time series

The primary algorithm, Seasonal Hybrid ESD (S-H-ESD), builds upon the Generalized ESD test [3] for detecting anomalies. S-H-ESD can be used to detect both global and local anomalies. This is achieved by employing time series decomposition and using robust statistical metrics, viz., median together with ESD. In addition, for long time series such as 6 months of minutely data, the algorithm employs piecewise approximation. This is rooted to the fact that trend extraction in the presence of anomalies is non-trivial for anomaly detection [4].

#install.packages("devtools")
# devtools::install_github("twitter/AnomalyDetection")
library(AnomalyDetection)
res = AnomalyDetectionTs(raw_data, max_anoms=0.02, direction='both', plot=TRUE)
res$plot

AnomalyDetectionVec(raw_data[,2], max_anoms=0.02, period=1440, direction='both', only_last=FALSE, plot=TRUE)
## $anoms
##     index    anoms
## 1     125  21.3510
## 2    5320 193.1036
## 3    6224 148.1740
## 4    7426  52.7478
## 5    7428  49.6582
## 6    7430  35.6067
## 7    7431  32.5045
## 8    7432  30.0555
## 9    7433  31.2614
## 10   7434  30.2551
## 11   7435  27.3860
## 12   7436  28.9807
## 13   7437  29.0844
## 14   7438  26.9185
## 15   7439  26.4621
## 16   7440  27.2180
## 17   7441  40.4268
## 18   7442  28.8811
## 19   7443  27.1294
## 20   7444  26.9913
## 21   7445  26.7741
## 22   7446  30.6972
## 23   7447  27.6085
## 24   7448  25.2841
## 25   7449  25.2264
## 26   7450  25.1927
## 27   7451  27.6501
## 28   7452  24.8858
## 29   7453  24.8099
## 30   7454  24.2950
## 31   7455  24.5221
## 32   7456  26.7354
## 33   7457  24.7589
## 34   7458  25.6300
## 35   7459  24.6949
## 36   7460  23.6518
## 37   7461  27.6977
## 38   7462  25.8687
## 39   7463  24.6654
## 40   7464  24.6694
## 41   7465  24.4351
## 42   7466  25.7723
## 43   7467  24.2502
## 44   7468  25.0965
## 45   7469  23.9362
## 46   7470  24.4127
## 47   7471  32.8182
## 48   7472  27.4586
## 49   7473  26.3163
## 50   7474  30.3965
## 51   7475  26.9516
## 52   7476  26.8908
## 53   7477  28.9461
## 54   7478  28.9742
## 55   7479  30.0406
## 56   7480  28.8507
## 57   7481  32.7811
## 58   7482  27.7468
## 59   7483  30.8174
## 60   7484  28.5658
## 61   7485  27.6677
## 62   7486  31.0409
## 63   7576 154.7750
## 64   7577 148.8620
## 65   7583 165.7870
## 66   7584 167.5850
## 67   7585 170.3490
## 68   7586 180.8990
## 69   7587 170.5130
## 70   7588 174.6780
## 71   7589 164.7350
## 72   7590 178.8220
## 73   7591 198.3260
## 74   7592 203.9010
## 75   7593 200.3090
## 76   7594 178.4910
## 77   7595 167.7480
## 78   7596 183.0180
## 79   7597 176.7690
## 80   7598 186.8230
## 81   7599 183.6600
## 82   7600 179.2760
## 83   7601 197.2830
## 84   7602 191.0970
## 85   7603 194.6700
## 86   7604 177.3250
## 87   7605 173.7580
## 88   7606 200.8160
## 89   7607 186.2350
## 90   7608 185.4210
## 91   7609 178.9580
## 92   7610 171.7500
## 93   7611 203.2310
## 94   7612 181.3540
## 95   7613 186.7780
## 96   7614 175.5820
## 97   7615 176.1250
## 98   7616 181.5140
## 99   7617 175.2610
## 100  7618 164.7190
## 101  7621 170.7360
## 102  7622 151.5490
## 103  7623 149.4120
## 104  7624 150.5540
## 105  7626 149.5410
## 106 10640 188.2908
## 107 13632  56.4691
## 108 13633  54.9415
## 109 13634  52.0359
## 110 13635  47.7313
## 111 13636  50.5876
## 112 13637  48.2846
## 113 13638  44.6438
## 114 13639  42.3077
## 115 13640  38.8363
## 116 13641  41.0145
## 117 13642  39.5523
## 118 13643  38.9117
## 119 13644  37.3052
## 120 13645  36.1725
## 121 13646  37.5150
## 122 13647  38.1387
## 123 13648  39.5351
## 124 13649  38.1834
## 125 13650  37.5988
## 126 13651  43.6522
## 127 13652  47.9571
## 128 14348 210.0000
## 129 14358  40.0000
## 130 14368 250.0000
## 131 14378  40.0000
## 
## $plot

res = AnomalyDetectionTs(raw_data, max_anoms=0.02, direction='both', only_last="day", plot=TRUE)
res$plot