Anomaly Detection Notebook

Code examples are from https://www.r-bloggers.com/anomaly-detection-in-r-2/ written by the team at Perceptive Analytics: Madhur Modi, Prudhvi Potuganti, Saneesh Veetil and Chaitanya Sagar.

Check if the packages needed are already installed and if not install them. First install the devtools package. Then install the github packages.

The wikipediatrend package contains the API to access wikipedia trends data on any page in Wikipedia. The AnomalyDetection package detects anomalies in seasonal univariate time series.

neededPackages = c("devtools", "Rcpp", "ggplot2")
for (i in 1:length(neededPackages)) {
    ifelse(!neededPackages[i] %in% installed.packages(), install.packages(neededPackages[i]), 
    print(paste(neededPackages[i], "is already installed.")))
}

[1] "devtools is already installed."
[1] "Rcpp is already installed."
[1] "ggplot2 is already installed."

library(devtools)
gitHubPackages = c("wikipediatrend", "AnomalyDetection")
gitHubLoction = c("petermeissner/wikipediatrend", "twitter/AnomalyDetection")
for (i in 1:length(gitHubPackages)) {
    ifelse(!gitHubPackages[i] %in% installed.packages(), install_github(gitHubLoction[i]), 
    print(paste(gitHubPackages[i], "is already installed.")))
}

[1] "wikipediatrend is already installed."
[1] "AnomalyDetection is already installed."

library(Rcpp)
library(wikipediatrend)
library(AnomalyDetection)

Download the trends data from the English Wikipedia webpage for President Michael D. Higgins “Michael_D._Higgins" and save it as the variable higginsWikipedia. View the first 6 rows.

higginsWikipedia = wp_trend("Michael_D._Higgins", from="2018-01-01", lang = "en")
head(higginsWikipedia)

  project   language article            access     agent      granularity date       views
1 wikipedia en       Michael_D._Higgins all-access all-agents daily       2018-01-01 785  
2 wikipedia en       Michael_D._Higgins all-access all-agents daily       2018-01-02 773  
3 wikipedia en       Michael_D._Higgins all-access all-agents daily       2018-01-03 657  
4 wikipedia en       Michael_D._Higgins all-access all-agents daily       2018-01-04 668  
5 wikipedia en       Michael_D._Higgins all-access all-agents daily       2018-01-05 622  
6 wikipedia en       Michael_D._Higgins all-access all-agents daily       2018-01-06 689  

Plot the data using ggplot2. Set the x-axis to the Date and the y-axis to the number of views.

library(ggplot2)
ggplot(higginsWikipedia, aes(x=date, y=views, color=views)) + 
  geom_line()

Prepare the data for the AnomalyDetection package by keeping only the date and page views and discard all other variables. The input must be a series of pairs. Check the structure of the higginsWikipedia dataframe. The date column must be in POSIXct format.

columns_to_keep=c("date","views")
higginsWikipedia=higginsWikipedia[,columns_to_keep]
str(higginsWikipedia)

Classes ‘wp_df’ and 'data.frame':   350 obs. of  2 variables:
 $ date : POSIXct, format: "2018-01-01" "2018-01-02" "2018-01-03" ...
 $ views: num  785 773 657 668 622 ...

AnomalyDetection Method

Apply anomaly detection and plot the results.

AnomalyDetectionHiggins = AnomalyDetectionTs(higginsWikipedia, direction="pos", plot=TRUE, title = "Anomaly Detection")
AnomalyDetectionHiggins$plot

Look at the dates the anomalies occured.

anomaliesHiggins$anoms$timestamp

Anomalize Method

Install anomalize package.

#install.packages('anomalize')
#Update from github
#library(devtools)
#install_github("business-science/anomalize")
#Load the package
library(anomalize)
# We will also use tidyverse package for processing and coindeskr to get bitcoin data
library(tidyverse)

Decompose data using time_decompose() function in anomalize package. We will use stl method which extracts seasonality.

higginsWikipedia_ts = higginsWikipedia %>%
                          as.tibble()
higginsWikipedia_ts %>%
  time_decompose(views, method = "stl",  frequency = "auto", trend = "auto") %>%  
  anomalize(remainder, method = "gesd", alpha = 0.05, max_anoms = 0.1) %>% 
  plot_anomaly_decomposition()

Converting from tbl_df to tbl_time.
Auto-index message: index = date
frequency = 7 days
trend = 90.5 days

Decompose data using time_decompose() function in anomalize package. We will use stl method which extracts seasonality.

higginsWikipedia_ts %>% 
  time_decompose(views, method = "stl", frequency = "auto", trend = "auto") %>%  
  anomalize(remainder, method = "gesd", alpha = 0.05, max_anoms = 0.1) %>% 
  plot_anomaly_decomposition()

Converting from tbl_df to tbl_time.
Auto-index message: index = date
frequency = 7 days
trend = 90.5 days

Plot the data again by recomposing data

higginsWikipedia_ts %>% 
  time_decompose(views) %>% 

Extract the anomalies

anomalizeHiggins = higginsWikipedia_ts %>% 
  time_decompose(views) %>%  
  anomalize(remainder) %>%  
  time_recompose() %>%  
  filter(anomaly == 'Yes')

Converting from tbl_df to tbl_time.
Auto-index message: index = date
frequency = 7 days
trend = 90.5 days

anomalizeHiggins

# A time tibble: 36 x 10
# Index: date
   date                observed season trend remainder remainder_l1 remainder_l2 anomaly recomposed_l1
   <dttm>                 <dbl>  <dbl> <dbl>     <dbl>        <dbl>        <dbl> <chr>           <dbl>
 1 2018-01-15 00:00:00   10627    59.7  851.     9716.       -1288.        1459. Yes             -377.
 2 2018-01-16 00:00:00   12142    39.8  850.    11252.       -1288.        1459. Yes             -398.
 3 2018-01-17 00:00:00    4009   -15.1  850.     3174.       -1288.        1459. Yes             -453.
 4 2018-01-29 00:00:00    3260.   59.7  845.     2356.       -1288.        1459. Yes             -384.
 5 2018-02-10 00:00:00    2773   -46.1  842.     1977.       -1288.        1459. Yes             -492.
 6 2018-02-21 00:00:00    2364   -15.1  830.     1549.       -1288.        1459. Yes             -473.
 7 2018-03-10 00:00:00    3682   -46.1  845.     2883.       -1288.        1459. Yes             -489.
 8 2018-03-17 00:00:00    2307   -46.1  855.     1498.       -1288.        1459. Yes             -479.
 9 2018-05-26 00:00:00    2972   -46.1  793.     2225.       -1288.        1459. Yes             -541.
10 2018-07-10 00:00:00    5597    39.8  831.     4726.       -1288.        1459. Yes             -417.
# ... with 26 more rows, and 1 more variable: recomposed_l2 <dbl>