Demand for Detecting Financial Fraud

Fraud detection is one of the top priorities for banks and financial institutions, which can be addressed using machine learning. According to a report published by Nilson, in 2017 the worldwide losses in card fraud related cases reached 22.8 billion dollars. The problem is forecasted to get worse in the following years, by 2021, the card fraud bill is expected to be 32.96 billion dollars.

For years, fraudsters would simply take numbers from credit or debit cards and print them onto blank plastic cards to use at brick-and-mortar stores. But in 2015, Visa and Mastercard mandated that banks and merchants introduce EMV - chip card technology, which made it possible for merchants to start requesting a PIN for each transaction. Nevertheless, experts predict online credit card fraud to soar to a whopping $32 billion in 2020.

In addition to the implementation of chip card technology, companies have been investing massive amounts in other technologies for detecting fraudulent transactions.

Machine Learning Approach to Detecting Fraud Transaction

In Machine Learning, problems like fraud detection are usually framed as classification problems —predicting a discrete class label output given a data observation. Examples of classification problems that can be thought of are Spam Detectors, Recommender Systems and Loan Default Prediction.

Challenge when Using Machine Learning Approach to Detecting Fraud Transaction

The main challenge when it comes to modeling fraud detection as a classification problem comes from the fact that in real world data, the majority of transactions is not fraudulent. Investment in technology for fraud detection has increased over the years so this shouldn’t be a surprise, but this brings us a problem: imbalanced data.

An example is data set provided by researchers from the Université Libre de Bruxelles in Belgium (for the full work, you can read their paper: Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi, Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015). This data set can be download from https://www.kaggle.com/mlg-ulb/creditcardfraud.

The datasets contain transactions made by credit cards in two days in September 2013 by European cardholders. We have 492 frauds out of 284,807 transactions. In this example, fraud transactions account for 0.172% of all transactions.

A Simple Solution: Searching Optimal Threshold

In this section I will present a simple solution for handling Imbalanced Data when developing a approach for detecting fraud transactions.

# Load packages for data manipulation: 
rm(list = ls())
library(tidyverse)
library(magrittr)

# Load data (downloaded from https://www.kaggle.com/mlg-ulb/creditcardfraud): 
creditcard <- read_csv("C:/creditcardfraud/creditcard.csv")

# Remove Time Column and convert to factor: 
creditcard %<>% 
  select(-Time) %>% 
  mutate(Class = case_when(Class == 1 ~ "Fraud", TRUE ~ "NonFraud") %>% as.factor())

theme_set(theme_minimal())
creditcard %>% 
  filter(Amount < 200) %>% 
  ggplot(aes(Class, Amount, fill = Class, color = Class)) + 
  geom_violin(alpha = 0.2)

# Scale 0-1 for numeric culumns: 
creditcard %<>% mutate_if(is.numeric, function(x) {(x - min(x)) / (max(x) - min(x))})

From above plot we can see that fraudulent transactions are anomalously centered around 100. This might be part of the fraudster’s strategy, instead of having large amounts at regular times, they hide small amounts more or less uniformly in time.

# Activate h2o package for deep learning: 
library(h2o)
h2o.init(nthreads = -1, max_mem_size = "16g")

##  Connection successful!
## 
## R is connected to the H2O cluster: 
##     H2O cluster uptime:         5 hours 37 minutes 
##     H2O cluster timezone:       Asia/Bangkok 
##     H2O data parsing timezone:  UTC 
##     H2O cluster version:        3.20.0.2 
##     H2O cluster version age:    5 months and 24 days !!! 
##     H2O cluster name:           H2O_started_from_R_ADMIN_mef573 
##     H2O cluster total nodes:    1 
##     H2O cluster total memory:   8.83 GB 
##     H2O cluster total cores:    32 
##     H2O cluster allowed cores:  32 
##     H2O cluster healthy:        TRUE 
##     H2O Connection ip:          localhost 
##     H2O Connection port:        54321 
##     H2O Connection proxy:       NA 
##     H2O Internal Security:      FALSE 
##     H2O API Extensions:         Algos, AutoML, Core V3, Core V4 
##     R Version:                  R version 3.5.1 (2018-07-02)

h2o.no_progress()

# Convert to H2o Frame: 
creditcard_hf <- as.h2o(creditcard)

# Split data sets: 
splits <- h2o.splitFrame(creditcard_hf, ratios = c(0.5), seed = 29)
train <- splits[[1]]
test <- splits[[2]]
test_for_cross <- as.data.frame(test)

# Identify inputput and output: 
y <- "Class"
x <- setdiff(names(train), y)

R codes for searching optimal threshold:

pure_nn <- h2o.deeplearning(x = x, y = y, 
                            training_frame = train,
                            reproducible = TRUE, 
                            balance_classes = TRUE,
                            ignore_const_cols = FALSE,
                            seed = 29,
                            hidden = c(30, 30, 30), 
                            epochs = 100,
                            activation = "Tanh")



library(caret)

eval_fun <- function(thre) {
  lapply(1:10, function(x) {
    
    set.seed(x)
    id <- createDataPartition(y = test_for_cross$Class, p = 0.5, list = FALSE)
    test_df <- test_for_cross[id, ]
  
    du_bao_prob <- h2o.predict(pure_nn, test_df %>% as.h2o()) %>% 
      as.data.frame() %>% 
      pull(Fraud)
    
    du_bao <- case_when(du_bao_prob >= thre ~ "Fraud", 
                        du_bao_prob < thre ~ "NonFraud") %>% as.factor()
    cm <- confusionMatrix(du_bao, test_df$Class, positive = "Fraud")
    
    bg_gg <- cm$table %>% 
      as.vector() %>% 
      matrix(ncol = 4) %>% 
      as.data.frame() %>% 
      rename(TP = V1, FN = V2, FP = V3, TN = V4)
  
    
    kq <- c(cm$overall, cm$byClass) 
    ten <- kq %>% as.data.frame() %>% row.names()
    
    kq %>% 
      as.vector() %>% 
      matrix(ncol = 18) %>% 
      as.data.frame() -> all_df
    
    names(all_df) <- ten
    all_df <- bind_cols(all_df, bg_gg)
    return(all_df)
  })
}



# Đánh giá sự biến đổi theo một loạt ngưỡng: 

system.time(so_sanh_list <- lapply(seq(0.05, 0.6, by = 0.05), eval_fun))

##    user  system elapsed 
##  737.65   32.22 1010.64

so_sanh_df <- do.call("bind_rows", so_sanh_list) 

so_sanh_df %<>% 
  mutate(Threshold = lapply(seq(0.05, 0.6, by = 0.05), function(x) {rep(x, 10)}) %>% unlist())


so_sanh_df %>% 
  group_by(Threshold) %>% 
  summarise_each(funs(median), Accuracy, Kappa, Sensitivity, Specificity) %>% 
  gather(Metric, b, -Threshold) %>% 
  ggplot(aes(Threshold, b, color = Metric)) + 
  geom_line() + 
  geom_point(size = 3) + 
  scale_y_continuous(labels = scales::percent) + 
  theme(panel.grid.minor = element_blank()) + 
  scale_x_continuous(breaks = seq(0.05, 0.6, by = 0.05)) + 
  labs(y = "Accuracy Rate", 
       title = "Variation of Classifier's Metrics by Threshold")

Conclusion: Optimal Threshold should be 0.15. Next we can comprehensively evaluate ability to investigate fraud cases corresponding to the selected thresholds:

my_cm_com_dl <- function(thre) {
  du_bao_prob <- h2o.predict(pure_nn, test) %>% as.data.frame() %>% pull(Fraud)
  du_bao <- case_when(du_bao_prob >= thre ~ "Fraud", 
                      du_bao_prob < thre ~ "NonFraud") %>% as.factor()
  cm <- confusionMatrix(du_bao, test_for_cross$Class, positive = "Fraud")
  return(cm)
  
}

my_threshold <- c(0.05, 0.15, 0.25, 0.5)
results_list_dl <- lapply(my_threshold, my_cm_com_dl)


vis_detection_rate_dl <- function(x) {
  
  results_list_dl[[x]]$table %>% as.data.frame() -> m
  rate <- round(100*m$Freq[1] / sum(m$Freq[c(1, 2)]), 2)
  acc <- round(100*sum(m$Freq[c(1, 4)]) / sum(m$Freq), 2)
  acc <- paste0(acc, "%")
  
  m %>% 
    ggplot(aes(Reference, Freq, fill = Prediction)) +
    geom_col(position = "fill") + 
    scale_fill_manual(values = c("#e41a1c", "#377eb8"), name = "") + 
    theme(panel.grid.minor.y = element_blank()) + 
    theme(panel.grid.minor.x = element_blank()) + 
    scale_y_continuous(labels = scales::percent) + 
    labs(x = NULL, y = NULL, 
         title = paste0("Detecting Fraud Cases when Threshold = ", my_threshold[x]), 
         subtitle = paste0("Detecting Rate for Fraud Cases: ", rate, "%", ", ", "Accuracy: ", acc))
  }


gridExtra::grid.arrange(vis_detection_rate_dl(1), 
                        vis_detection_rate_dl(2), 
                        vis_detection_rate_dl(3), 
                        vis_detection_rate_dl(4))

Random Forests

Instead of using Deep Learning Approach, we can train a Random Forests as a tool for detecting fraud transactions:

# Set hyperparameter grid: 
hyper_grid.h2o <- list(ntrees = seq(100, 800, by = 100), 
                       mtries = seq(10, 28, by = 2),
                       sample_rate = c(0.40, 0.55, 0.632, 0.70, 0.80))

# Build grid search: 
system.time(grid <- h2o.grid(algorithm = "randomForest",
                             grid_id = "rf_grid",
                             x = x, y = y, 
                             training_frame = train,
                             hyper_params = hyper_grid.h2o,
                             search_criteria = list(strategy = "Cartesian")))

##    user  system elapsed 
##   50.33    5.95 8429.39

# Collect the results and sort by our model performance metric of choice: 
grid_perf <- h2o.getGrid(grid_id = "rf_grid", 
                         sort_by = "auc", 
                         decreasing = FALSE)

# Best model chosen by validation error: 
best_model_id <- grid_perf@model_ids[[1]]
best_model <- h2o.getModel(best_model_id)


my_cm_com_rf <- function(thre) {
  du_bao_prob <- h2o.predict(best_model, test) %>% as.data.frame() %>% pull(Fraud)
  du_bao <- case_when(du_bao_prob >= thre ~ "Fraud", 
                      du_bao_prob < thre ~ "NonFraud") %>% as.factor()
  cm <- confusionMatrix(du_bao, test_for_cross$Class, positive = "Fraud")
  return(cm)
  
}


results_list_rf <- lapply(my_threshold, my_cm_com_rf)


vis_detection_rate_rf <- function(x) {
  
  results_list_rf[[x]]$table %>% as.data.frame() -> m
  rate <- round(100*m$Freq[1] / sum(m$Freq[c(1, 2)]), 2)
  acc <- round(100*sum(m$Freq[c(1, 4)]) / sum(m$Freq), 2)
  acc <- paste0(acc, "%")
  
  m %>% 
    ggplot(aes(Reference, Freq, fill = Prediction)) +
    geom_col(position = "fill") + 
    scale_fill_manual(values = c("#e41a1c", "#377eb8"), name = "") + 
    theme(panel.grid.minor.y = element_blank()) + 
    theme(panel.grid.minor.x = element_blank()) + 
    scale_y_continuous(labels = scales::percent) + 
    labs(x = NULL, y = NULL, 
         title = paste0("Detecting Fraud Cases when Threshold = ", my_threshold[x]), 
         subtitle = paste0("Detecting Rate for Fraud Cases: ", rate, "%", ", ", "Accuracy: ", acc))
  }


gridExtra::grid.arrange(vis_detection_rate_rf(1), 
                        vis_detection_rate_rf(2), 
                        vis_detection_rate_rf(3), 
                        vis_detection_rate_rf(4))

Because of the combinatorial explosion, each additional hyperparameter that gets added to our grid search has a huge effect on the time to complete. Consequently, h2o provides an additional grid search path called “RandomDiscrete”, which will jump from one random combination to another and stop once a certain level of improvement has been made, certain amount of time has been exceeded, or a certain amount of models have been ran (or a combination of these have been met). Although using a random discrete search path will likely not find the optimal model, it typically does a good job of finding a very good model.

# Set hyperparameter grid2: 
hyper2_grid.h2o <- list(ntrees = seq(100, 800, by = 100),
                        mtries = seq(5, 25, by = 5),
                        max_depth = seq(10, 25, by = 5),
                        min_rows = seq(1, 5, by = 1),
                        nbins = seq(10, 20, by = 2),
                        sample_rate = c(0.4, 0.55, 0.632, 0.75, 0.80))

# Select random grid search criteria: 
search_criteria2 <- list(strategy = "RandomDiscrete",
                         stopping_metric = "AUC",
                         stopping_tolerance = 0.005,
                         stopping_rounds = 10,
                         max_runtime_secs = 30*60)

# Build grid search: 

system.time(random_grid <- h2o.grid(algorithm = "randomForest",
                                    grid_id = "rf_grid2",
                                    x = x, y = y, 
                                    training_frame = train,
                                    hyper_params = hyper2_grid.h2o,
                                    search_criteria = search_criteria2))

##    user  system elapsed 
##    8.33    0.89  712.53

# Collect the results and sort by our model performance metric of choice: 
grid_perf2 <- h2o.getGrid(grid_id = "rf_grid2", 
                          sort_by = "auc", 
                          decreasing = FALSE)

# Best Model: 
best_model2 <- h2o.getModel(grid_perf2@model_ids[[1]])


my_cm_com_rf2 <- function(thre) {
  du_bao_prob <- h2o.predict(best_model2, test) %>% as.data.frame() %>% pull(Fraud)
  du_bao <- case_when(du_bao_prob >= thre ~ "Fraud", 
                      du_bao_prob < thre ~ "NonFraud") %>% as.factor()
  cm <- confusionMatrix(du_bao, test_for_cross$Class, positive = "Fraud")
  return(cm)
  
}


results_list_rf2 <- lapply(my_threshold, my_cm_com_rf2)


vis_detection_rate_rf2 <- function(x) {
  
  results_list_rf2[[x]]$table %>% as.data.frame() -> m
  rate <- round(100*m$Freq[1] / sum(m$Freq[c(1, 2)]), 2)
  acc <- round(100*sum(m$Freq[c(1, 4)]) / sum(m$Freq), 2)
  acc <- paste0(acc, "%")
  
  m %>% 
    ggplot(aes(Reference, Freq, fill = Prediction)) +
    geom_col(position = "fill") + 
    scale_fill_manual(values = c("#e41a1c", "#377eb8"), name = "") + 
    theme(panel.grid.minor.y = element_blank()) + 
    theme(panel.grid.minor.x = element_blank()) + 
    scale_y_continuous(labels = scales::percent) + 
    labs(x = NULL, y = NULL, 
         title = paste0("Detecting Fraud Cases when Threshold = ", my_threshold[x]), 
         subtitle = paste0("Detecting Rate for Fraud Cases: ", rate, "%", ", ", "Accuracy: ", acc))
  }


gridExtra::grid.arrange(vis_detection_rate_rf2(1), 
                        vis_detection_rate_rf2(2), 
                        vis_detection_rate_rf2(3), 
                        vis_detection_rate_rf2(4))

LS0tDQp0aXRsZTogIkRldGVjdGluZyBGaW5hbmNpYWwgRnJhdWQgVXNpbmcgTWFjaGluZSBMZWFybmluZzogV2FyIEFnYWluc3QgSW1iYWxhbmNlZCBEYXRhIiANCnN1YnRpdGxlOiAiUiBmb3IgUGxlYXN1cmUiDQphdXRob3I6ICJOZ3V5ZW4gQ2hpIER1bmciDQpvdXRwdXQ6DQogIGh0bWxfZG9jdW1lbnQ6IA0KICAgIGNvZGVfZG93bmxvYWQ6IHRydWUNCiAgICAjIGNvZGVfZm9sZGluZzogaGlkZQ0KICAgIGhpZ2hsaWdodDogcHlnbWVudHMNCiAgICAjIG51bWJlcl9zZWN0aW9uczogeWVzDQogICAgdGhlbWU6ICJmbGF0bHkiDQogICAgdG9jOiBUUlVFDQogICAgdG9jX2Zsb2F0OiBUUlVFDQotLS0NCg0KYGBge3Igc2V0dXAsaW5jbHVkZT1GQUxTRX0NCmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSwgd2FybmluZyA9IEZBTFNFLCBtZXNzYWdlID0gRkFMU0UpDQpgYGANCg0KDQojIERlbWFuZCBmb3IgRGV0ZWN0aW5nIEZpbmFuY2lhbCBGcmF1ZA0KDQpGcmF1ZCBkZXRlY3Rpb24gaXMgb25lIG9mIHRoZSB0b3AgcHJpb3JpdGllcyBmb3IgYmFua3MgYW5kIGZpbmFuY2lhbCBpbnN0aXR1dGlvbnMsIHdoaWNoIGNhbiBiZSBhZGRyZXNzZWQgdXNpbmcgbWFjaGluZSBsZWFybmluZy4gQWNjb3JkaW5nIHRvIGEgcmVwb3J0IHB1Ymxpc2hlZCBieSBOaWxzb24sIGluIDIwMTcgdGhlIHdvcmxkd2lkZSBsb3NzZXMgaW4gY2FyZCBmcmF1ZCByZWxhdGVkIGNhc2VzIHJlYWNoZWQgMjIuOCBiaWxsaW9uIGRvbGxhcnMuIFRoZSBwcm9ibGVtIGlzIGZvcmVjYXN0ZWQgdG8gZ2V0IHdvcnNlIGluIHRoZSBmb2xsb3dpbmcgeWVhcnMsIGJ5IDIwMjEsIHRoZSBjYXJkIGZyYXVkIGJpbGwgaXMgZXhwZWN0ZWQgdG8gYmUgMzIuOTYgYmlsbGlvbiBkb2xsYXJzLiANCg0KRm9yIHllYXJzLCBmcmF1ZHN0ZXJzIHdvdWxkIHNpbXBseSB0YWtlIG51bWJlcnMgZnJvbSBjcmVkaXQgb3IgZGViaXQgY2FyZHMgYW5kIHByaW50IHRoZW0gb250byBibGFuayBwbGFzdGljIGNhcmRzIHRvIHVzZSBhdCBicmljay1hbmQtbW9ydGFyIHN0b3Jlcy4gQnV0IGluIDIwMTUsIFZpc2EgYW5kIE1hc3RlcmNhcmQgbWFuZGF0ZWQgdGhhdCBiYW5rcyBhbmQgbWVyY2hhbnRzIGludHJvZHVjZSBFTVYgLSBjaGlwIGNhcmQgdGVjaG5vbG9neSwgd2hpY2ggbWFkZSBpdCBwb3NzaWJsZSBmb3IgbWVyY2hhbnRzIHRvIHN0YXJ0IHJlcXVlc3RpbmcgYSBQSU4gZm9yIGVhY2ggdHJhbnNhY3Rpb24uIE5ldmVydGhlbGVzcywgZXhwZXJ0cyBwcmVkaWN0IG9ubGluZSBjcmVkaXQgY2FyZCBmcmF1ZCB0byBzb2FyIHRvIGEgd2hvcHBpbmcgJDMyIGJpbGxpb24gaW4gMjAyMC4NCg0KSW4gYWRkaXRpb24gdG8gdGhlIGltcGxlbWVudGF0aW9uIG9mIGNoaXAgY2FyZCB0ZWNobm9sb2d5LCBjb21wYW5pZXMgaGF2ZSBiZWVuIGludmVzdGluZyBtYXNzaXZlIGFtb3VudHMgaW4gb3RoZXIgdGVjaG5vbG9naWVzIGZvciBkZXRlY3RpbmcgZnJhdWR1bGVudCB0cmFuc2FjdGlvbnMuDQoNCiMgTWFjaGluZSBMZWFybmluZyBBcHByb2FjaCB0byBEZXRlY3RpbmcgRnJhdWQgVHJhbnNhY3Rpb24NCg0KSW4gTWFjaGluZSBMZWFybmluZywgcHJvYmxlbXMgbGlrZSBmcmF1ZCBkZXRlY3Rpb24gYXJlIHVzdWFsbHkgZnJhbWVkIGFzIGNsYXNzaWZpY2F0aW9uIHByb2JsZW1zIOKAlHByZWRpY3RpbmcgYSBkaXNjcmV0ZSBjbGFzcyBsYWJlbCBvdXRwdXQgZ2l2ZW4gYSBkYXRhIG9ic2VydmF0aW9uLiBFeGFtcGxlcyBvZiBjbGFzc2lmaWNhdGlvbiBwcm9ibGVtcyB0aGF0IGNhbiBiZSB0aG91Z2h0IG9mIGFyZSBTcGFtIERldGVjdG9ycywgUmVjb21tZW5kZXIgU3lzdGVtcyBhbmQgTG9hbiBEZWZhdWx0IFByZWRpY3Rpb24uDQoNCiMgQ2hhbGxlbmdlIHdoZW4gVXNpbmcgTWFjaGluZSBMZWFybmluZyBBcHByb2FjaCB0byBEZXRlY3RpbmcgRnJhdWQgVHJhbnNhY3Rpb24NCg0KVGhlIG1haW4gY2hhbGxlbmdlIHdoZW4gaXQgY29tZXMgdG8gbW9kZWxpbmcgZnJhdWQgZGV0ZWN0aW9uIGFzIGEgY2xhc3NpZmljYXRpb24gcHJvYmxlbSBjb21lcyBmcm9tIHRoZSBmYWN0IHRoYXQgaW4gcmVhbCB3b3JsZCBkYXRhLCB0aGUgbWFqb3JpdHkgb2YgdHJhbnNhY3Rpb25zIGlzIG5vdCBmcmF1ZHVsZW50LiBJbnZlc3RtZW50IGluIHRlY2hub2xvZ3kgZm9yIGZyYXVkIGRldGVjdGlvbiBoYXMgaW5jcmVhc2VkIG92ZXIgdGhlIHllYXJzIHNvIHRoaXMgc2hvdWxkbuKAmXQgYmUgYSBzdXJwcmlzZSwgYnV0IHRoaXMgYnJpbmdzIHVzIGEgcHJvYmxlbTogaW1iYWxhbmNlZCBkYXRhLg0KDQpBbiBleGFtcGxlIGlzIGRhdGEgc2V0IHByb3ZpZGVkIGJ5IHJlc2VhcmNoZXJzIGZyb20gdGhlICoqVW5pdmVyc2l0w6kgTGlicmUgZGUgQnJ1eGVsbGVzIGluIEJlbGdpdW0qKiAoZm9yIHRoZSBmdWxsIHdvcmssIHlvdSBjYW4gcmVhZCB0aGVpciBwYXBlcjogQW5kcmVhIERhbCBQb3p6b2xvLCBPbGl2aWVyIENhZWxlbiwgUmVpZCBBLiBKb2huc29uIGFuZCBHaWFubHVjYSBCb250ZW1waSwgQ2FsaWJyYXRpbmcgUHJvYmFiaWxpdHkgd2l0aCBVbmRlcnNhbXBsaW5nIGZvciBVbmJhbGFuY2VkIENsYXNzaWZpY2F0aW9uLiBJbiBTeW1wb3NpdW0gb24gQ29tcHV0YXRpb25hbCBJbnRlbGxpZ2VuY2UgYW5kIERhdGEgTWluaW5nIChDSURNKSwgSUVFRSwgMjAxNSkuIFRoaXMgZGF0YSBzZXQgY2FuIGJlIGRvd25sb2FkIGZyb20gaHR0cHM6Ly93d3cua2FnZ2xlLmNvbS9tbGctdWxiL2NyZWRpdGNhcmRmcmF1ZC4gDQoNClRoZSBkYXRhc2V0cyBjb250YWluIHRyYW5zYWN0aW9ucyBtYWRlIGJ5IGNyZWRpdCBjYXJkcyBpbiB0d28gZGF5cyBpbiBTZXB0ZW1iZXIgMjAxMyBieSBFdXJvcGVhbiBjYXJkaG9sZGVycy4gV2UgaGF2ZSA0OTIgZnJhdWRzIG91dCBvZiAyODQsODA3IHRyYW5zYWN0aW9ucy4gSW4gdGhpcyBleGFtcGxlLCBmcmF1ZCB0cmFuc2FjdGlvbnMgYWNjb3VudCBmb3IgMC4xNzIlIG9mIGFsbCB0cmFuc2FjdGlvbnMuDQoNCiMgQSBTaW1wbGUgU29sdXRpb246IFNlYXJjaGluZyBPcHRpbWFsIFRocmVzaG9sZA0KDQpJbiB0aGlzIHNlY3Rpb24gSSB3aWxsIHByZXNlbnQgYSBzaW1wbGUgc29sdXRpb24gZm9yIGhhbmRsaW5nIEltYmFsYW5jZWQgRGF0YSB3aGVuIGRldmVsb3BpbmcgYSBhcHByb2FjaCBmb3IgZGV0ZWN0aW5nIGZyYXVkIHRyYW5zYWN0aW9ucy4gDQoNCg0KYGBge3J9DQojIExvYWQgcGFja2FnZXMgZm9yIGRhdGEgbWFuaXB1bGF0aW9uOiANCnJtKGxpc3QgPSBscygpKQ0KbGlicmFyeSh0aWR5dmVyc2UpDQpsaWJyYXJ5KG1hZ3JpdHRyKQ0KDQojIExvYWQgZGF0YSAoZG93bmxvYWRlZCBmcm9tIGh0dHBzOi8vd3d3LmthZ2dsZS5jb20vbWxnLXVsYi9jcmVkaXRjYXJkZnJhdWQpOiANCmNyZWRpdGNhcmQgPC0gcmVhZF9jc3YoIkM6L2NyZWRpdGNhcmRmcmF1ZC9jcmVkaXRjYXJkLmNzdiIpDQoNCiMgUmVtb3ZlIFRpbWUgQ29sdW1uIGFuZCBjb252ZXJ0IHRvIGZhY3RvcjogDQpjcmVkaXRjYXJkICU8PiUgDQogIHNlbGVjdCgtVGltZSkgJT4lIA0KICBtdXRhdGUoQ2xhc3MgPSBjYXNlX3doZW4oQ2xhc3MgPT0gMSB+ICJGcmF1ZCIsIFRSVUUgfiAiTm9uRnJhdWQiKSAlPiUgYXMuZmFjdG9yKCkpDQoNCnRoZW1lX3NldCh0aGVtZV9taW5pbWFsKCkpDQpjcmVkaXRjYXJkICU+JSANCiAgZmlsdGVyKEFtb3VudCA8IDIwMCkgJT4lIA0KICBnZ3Bsb3QoYWVzKENsYXNzLCBBbW91bnQsIGZpbGwgPSBDbGFzcywgY29sb3IgPSBDbGFzcykpICsgDQogIGdlb21fdmlvbGluKGFscGhhID0gMC4yKQ0KDQoNCiMgU2NhbGUgMC0xIGZvciBudW1lcmljIGN1bHVtbnM6IA0KY3JlZGl0Y2FyZCAlPD4lIG11dGF0ZV9pZihpcy5udW1lcmljLCBmdW5jdGlvbih4KSB7KHggLSBtaW4oeCkpIC8gKG1heCh4KSAtIG1pbih4KSl9KQ0KYGBgDQoNCkZyb20gYWJvdmUgcGxvdCB3ZSBjYW4gc2VlIHRoYXQgZnJhdWR1bGVudCB0cmFuc2FjdGlvbnMgYXJlIGFub21hbG91c2x5IGNlbnRlcmVkIGFyb3VuZCAxMDAuIFRoaXMgbWlnaHQgYmUgcGFydCBvZiB0aGUgZnJhdWRzdGVyJ3Mgc3RyYXRlZ3ksIGluc3RlYWQgb2YgaGF2aW5nIGxhcmdlIGFtb3VudHMgYXQgcmVndWxhciB0aW1lcywgdGhleSBoaWRlIHNtYWxsIGFtb3VudHMgbW9yZSBvciBsZXNzIHVuaWZvcm1seSBpbiB0aW1lLg0KDQpgYGB7cn0NCiMgQWN0aXZhdGUgaDJvIHBhY2thZ2UgZm9yIGRlZXAgbGVhcm5pbmc6IA0KbGlicmFyeShoMm8pDQpoMm8uaW5pdChudGhyZWFkcyA9IC0xLCBtYXhfbWVtX3NpemUgPSAiMTZnIikNCmgyby5ub19wcm9ncmVzcygpDQoNCiMgQ29udmVydCB0byBIMm8gRnJhbWU6IA0KY3JlZGl0Y2FyZF9oZiA8LSBhcy5oMm8oY3JlZGl0Y2FyZCkNCg0KIyBTcGxpdCBkYXRhIHNldHM6IA0Kc3BsaXRzIDwtIGgyby5zcGxpdEZyYW1lKGNyZWRpdGNhcmRfaGYsIHJhdGlvcyA9IGMoMC41KSwgc2VlZCA9IDI5KQ0KdHJhaW4gPC0gc3BsaXRzW1sxXV0NCnRlc3QgPC0gc3BsaXRzW1syXV0NCnRlc3RfZm9yX2Nyb3NzIDwtIGFzLmRhdGEuZnJhbWUodGVzdCkNCg0KIyBJZGVudGlmeSBpbnB1dHB1dCBhbmQgb3V0cHV0OiANCnkgPC0gIkNsYXNzIg0KeCA8LSBzZXRkaWZmKG5hbWVzKHRyYWluKSwgeSkNCmBgYA0KDQpSIGNvZGVzIGZvciBzZWFyY2hpbmcgb3B0aW1hbCB0aHJlc2hvbGQ6IA0KDQpgYGB7cn0NCg0KcHVyZV9ubiA8LSBoMm8uZGVlcGxlYXJuaW5nKHggPSB4LCB5ID0geSwgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgdHJhaW5pbmdfZnJhbWUgPSB0cmFpbiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICByZXByb2R1Y2libGUgPSBUUlVFLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBiYWxhbmNlX2NsYXNzZXMgPSBUUlVFLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIGlnbm9yZV9jb25zdF9jb2xzID0gRkFMU0UsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgc2VlZCA9IDI5LA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIGhpZGRlbiA9IGMoMzAsIDMwLCAzMCksIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIGVwb2NocyA9IDEwMCwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBhY3RpdmF0aW9uID0gIlRhbmgiKQ0KDQoNCg0KbGlicmFyeShjYXJldCkNCg0KZXZhbF9mdW4gPC0gZnVuY3Rpb24odGhyZSkgew0KICBsYXBwbHkoMToxMCwgZnVuY3Rpb24oeCkgew0KICAgIA0KICAgIHNldC5zZWVkKHgpDQogICAgaWQgPC0gY3JlYXRlRGF0YVBhcnRpdGlvbih5ID0gdGVzdF9mb3JfY3Jvc3MkQ2xhc3MsIHAgPSAwLjUsIGxpc3QgPSBGQUxTRSkNCiAgICB0ZXN0X2RmIDwtIHRlc3RfZm9yX2Nyb3NzW2lkLCBdDQogIA0KICAgIGR1X2Jhb19wcm9iIDwtIGgyby5wcmVkaWN0KHB1cmVfbm4sIHRlc3RfZGYgJT4lIGFzLmgybygpKSAlPiUgDQogICAgICBhcy5kYXRhLmZyYW1lKCkgJT4lIA0KICAgICAgcHVsbChGcmF1ZCkNCiAgICANCiAgICBkdV9iYW8gPC0gY2FzZV93aGVuKGR1X2Jhb19wcm9iID49IHRocmUgfiAiRnJhdWQiLCANCiAgICAgICAgICAgICAgICAgICAgICAgIGR1X2Jhb19wcm9iIDwgdGhyZSB+ICJOb25GcmF1ZCIpICU+JSBhcy5mYWN0b3IoKQ0KICAgIGNtIDwtIGNvbmZ1c2lvbk1hdHJpeChkdV9iYW8sIHRlc3RfZGYkQ2xhc3MsIHBvc2l0aXZlID0gIkZyYXVkIikNCiAgICANCiAgICBiZ19nZyA8LSBjbSR0YWJsZSAlPiUgDQogICAgICBhcy52ZWN0b3IoKSAlPiUgDQogICAgICBtYXRyaXgobmNvbCA9IDQpICU+JSANCiAgICAgIGFzLmRhdGEuZnJhbWUoKSAlPiUgDQogICAgICByZW5hbWUoVFAgPSBWMSwgRk4gPSBWMiwgRlAgPSBWMywgVE4gPSBWNCkNCiAgDQogICAgDQogICAga3EgPC0gYyhjbSRvdmVyYWxsLCBjbSRieUNsYXNzKSANCiAgICB0ZW4gPC0ga3EgJT4lIGFzLmRhdGEuZnJhbWUoKSAlPiUgcm93Lm5hbWVzKCkNCiAgICANCiAgICBrcSAlPiUgDQogICAgICBhcy52ZWN0b3IoKSAlPiUgDQogICAgICBtYXRyaXgobmNvbCA9IDE4KSAlPiUgDQogICAgICBhcy5kYXRhLmZyYW1lKCkgLT4gYWxsX2RmDQogICAgDQogICAgbmFtZXMoYWxsX2RmKSA8LSB0ZW4NCiAgICBhbGxfZGYgPC0gYmluZF9jb2xzKGFsbF9kZiwgYmdfZ2cpDQogICAgcmV0dXJuKGFsbF9kZikNCiAgfSkNCn0NCg0KDQoNCiMgxJDDoW5oIGdpw6Egc+G7sSBiaeG6v24gxJHhu5VpIHRoZW8gbeG7mXQgbG/huqF0IG5nxrDhu6FuZzogDQoNCnN5c3RlbS50aW1lKHNvX3NhbmhfbGlzdCA8LSBsYXBwbHkoc2VxKDAuMDUsIDAuNiwgYnkgPSAwLjA1KSwgZXZhbF9mdW4pKQ0Kc29fc2FuaF9kZiA8LSBkby5jYWxsKCJiaW5kX3Jvd3MiLCBzb19zYW5oX2xpc3QpIA0KDQpzb19zYW5oX2RmICU8PiUgDQogIG11dGF0ZShUaHJlc2hvbGQgPSBsYXBwbHkoc2VxKDAuMDUsIDAuNiwgYnkgPSAwLjA1KSwgZnVuY3Rpb24oeCkge3JlcCh4LCAxMCl9KSAlPiUgdW5saXN0KCkpDQoNCg0Kc29fc2FuaF9kZiAlPiUgDQogIGdyb3VwX2J5KFRocmVzaG9sZCkgJT4lIA0KICBzdW1tYXJpc2VfZWFjaChmdW5zKG1lZGlhbiksIEFjY3VyYWN5LCBLYXBwYSwgU2Vuc2l0aXZpdHksIFNwZWNpZmljaXR5KSAlPiUgDQogIGdhdGhlcihNZXRyaWMsIGIsIC1UaHJlc2hvbGQpICU+JSANCiAgZ2dwbG90KGFlcyhUaHJlc2hvbGQsIGIsIGNvbG9yID0gTWV0cmljKSkgKyANCiAgZ2VvbV9saW5lKCkgKyANCiAgZ2VvbV9wb2ludChzaXplID0gMykgKyANCiAgc2NhbGVfeV9jb250aW51b3VzKGxhYmVscyA9IHNjYWxlczo6cGVyY2VudCkgKyANCiAgdGhlbWUocGFuZWwuZ3JpZC5taW5vciA9IGVsZW1lbnRfYmxhbmsoKSkgKyANCiAgc2NhbGVfeF9jb250aW51b3VzKGJyZWFrcyA9IHNlcSgwLjA1LCAwLjYsIGJ5ID0gMC4wNSkpICsgDQogIGxhYnMoeSA9ICJBY2N1cmFjeSBSYXRlIiwgDQogICAgICAgdGl0bGUgPSAiVmFyaWF0aW9uIG9mIENsYXNzaWZpZXIncyBNZXRyaWNzIGJ5IFRocmVzaG9sZCIpDQpgYGANCg0KQ29uY2x1c2lvbjogT3B0aW1hbCBUaHJlc2hvbGQgc2hvdWxkIGJlIDAuMTUuIE5leHQgd2UgY2FuIGNvbXByZWhlbnNpdmVseSBldmFsdWF0ZSBhYmlsaXR5IHRvIGludmVzdGlnYXRlIGZyYXVkIGNhc2VzIGNvcnJlc3BvbmRpbmcgdG8gdGhlIHNlbGVjdGVkIHRocmVzaG9sZHM6IA0KDQpgYGB7ciwgZmlnLmZ1bGx3aWR0aCA9IFRSVUUsIGZpZy5oZWlnaHQ9MTIsIGZpZy53aWR0aD0xNH0NCg0KbXlfY21fY29tX2RsIDwtIGZ1bmN0aW9uKHRocmUpIHsNCiAgZHVfYmFvX3Byb2IgPC0gaDJvLnByZWRpY3QocHVyZV9ubiwgdGVzdCkgJT4lIGFzLmRhdGEuZnJhbWUoKSAlPiUgcHVsbChGcmF1ZCkNCiAgZHVfYmFvIDwtIGNhc2Vfd2hlbihkdV9iYW9fcHJvYiA+PSB0aHJlIH4gIkZyYXVkIiwgDQogICAgICAgICAgICAgICAgICAgICAgZHVfYmFvX3Byb2IgPCB0aHJlIH4gIk5vbkZyYXVkIikgJT4lIGFzLmZhY3RvcigpDQogIGNtIDwtIGNvbmZ1c2lvbk1hdHJpeChkdV9iYW8sIHRlc3RfZm9yX2Nyb3NzJENsYXNzLCBwb3NpdGl2ZSA9ICJGcmF1ZCIpDQogIHJldHVybihjbSkNCiAgDQp9DQoNCm15X3RocmVzaG9sZCA8LSBjKDAuMDUsIDAuMTUsIDAuMjUsIDAuNSkNCnJlc3VsdHNfbGlzdF9kbCA8LSBsYXBwbHkobXlfdGhyZXNob2xkLCBteV9jbV9jb21fZGwpDQoNCg0KdmlzX2RldGVjdGlvbl9yYXRlX2RsIDwtIGZ1bmN0aW9uKHgpIHsNCiAgDQogIHJlc3VsdHNfbGlzdF9kbFtbeF1dJHRhYmxlICU+JSBhcy5kYXRhLmZyYW1lKCkgLT4gbQ0KICByYXRlIDwtIHJvdW5kKDEwMCptJEZyZXFbMV0gLyBzdW0obSRGcmVxW2MoMSwgMildKSwgMikNCiAgYWNjIDwtIHJvdW5kKDEwMCpzdW0obSRGcmVxW2MoMSwgNCldKSAvIHN1bShtJEZyZXEpLCAyKQ0KICBhY2MgPC0gcGFzdGUwKGFjYywgIiUiKQ0KICANCiAgbSAlPiUgDQogICAgZ2dwbG90KGFlcyhSZWZlcmVuY2UsIEZyZXEsIGZpbGwgPSBQcmVkaWN0aW9uKSkgKw0KICAgIGdlb21fY29sKHBvc2l0aW9uID0gImZpbGwiKSArIA0KICAgIHNjYWxlX2ZpbGxfbWFudWFsKHZhbHVlcyA9IGMoIiNlNDFhMWMiLCAiIzM3N2ViOCIpLCBuYW1lID0gIiIpICsgDQogICAgdGhlbWUocGFuZWwuZ3JpZC5taW5vci55ID0gZWxlbWVudF9ibGFuaygpKSArIA0KICAgIHRoZW1lKHBhbmVsLmdyaWQubWlub3IueCA9IGVsZW1lbnRfYmxhbmsoKSkgKyANCiAgICBzY2FsZV95X2NvbnRpbnVvdXMobGFiZWxzID0gc2NhbGVzOjpwZXJjZW50KSArIA0KICAgIGxhYnMoeCA9IE5VTEwsIHkgPSBOVUxMLCANCiAgICAgICAgIHRpdGxlID0gcGFzdGUwKCJEZXRlY3RpbmcgRnJhdWQgQ2FzZXMgd2hlbiBUaHJlc2hvbGQgPSAiLCBteV90aHJlc2hvbGRbeF0pLCANCiAgICAgICAgIHN1YnRpdGxlID0gcGFzdGUwKCJEZXRlY3RpbmcgUmF0ZSBmb3IgRnJhdWQgQ2FzZXM6ICIsIHJhdGUsICIlIiwgIiwgIiwgIkFjY3VyYWN5OiAiLCBhY2MpKQ0KICB9DQoNCg0KZ3JpZEV4dHJhOjpncmlkLmFycmFuZ2UodmlzX2RldGVjdGlvbl9yYXRlX2RsKDEpLCANCiAgICAgICAgICAgICAgICAgICAgICAgIHZpc19kZXRlY3Rpb25fcmF0ZV9kbCgyKSwgDQogICAgICAgICAgICAgICAgICAgICAgICB2aXNfZGV0ZWN0aW9uX3JhdGVfZGwoMyksIA0KICAgICAgICAgICAgICAgICAgICAgICAgdmlzX2RldGVjdGlvbl9yYXRlX2RsKDQpKQ0KDQoNCg0KDQpgYGANCg0KDQojIFJhbmRvbSBGb3Jlc3RzDQoNCkluc3RlYWQgb2YgdXNpbmcgRGVlcCBMZWFybmluZyBBcHByb2FjaCwgd2UgY2FuIHRyYWluIGEgUmFuZG9tIEZvcmVzdHMgYXMgYSB0b29sIGZvciBkZXRlY3RpbmcgZnJhdWQgdHJhbnNhY3Rpb25zOiANCg0KYGBge3IsIGZpZy5mdWxsd2lkdGggPSBUUlVFLCBmaWcuaGVpZ2h0PTEyLCBmaWcud2lkdGg9MTQsIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9DQojIFNldCBoeXBlcnBhcmFtZXRlciBncmlkOiANCmh5cGVyX2dyaWQuaDJvIDwtIGxpc3QobnRyZWVzID0gc2VxKDEwMCwgODAwLCBieSA9IDEwMCksIA0KICAgICAgICAgICAgICAgICAgICAgICBtdHJpZXMgPSBzZXEoMTAsIDI4LCBieSA9IDIpLA0KICAgICAgICAgICAgICAgICAgICAgICBzYW1wbGVfcmF0ZSA9IGMoMC40MCwgMC41NSwgMC42MzIsIDAuNzAsIDAuODApKQ0KDQojIEJ1aWxkIGdyaWQgc2VhcmNoOiANCnN5c3RlbS50aW1lKGdyaWQgPC0gaDJvLmdyaWQoYWxnb3JpdGhtID0gInJhbmRvbUZvcmVzdCIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdyaWRfaWQgPSAicmZfZ3JpZCIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHggPSB4LCB5ID0geSwgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRyYWluaW5nX2ZyYW1lID0gdHJhaW4sDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgIGh5cGVyX3BhcmFtcyA9IGh5cGVyX2dyaWQuaDJvLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzZWFyY2hfY3JpdGVyaWEgPSBsaXN0KHN0cmF0ZWd5ID0gIkNhcnRlc2lhbiIpKSkNCg0KDQojIENvbGxlY3QgdGhlIHJlc3VsdHMgYW5kIHNvcnQgYnkgb3VyIG1vZGVsIHBlcmZvcm1hbmNlIG1ldHJpYyBvZiBjaG9pY2U6IA0KZ3JpZF9wZXJmIDwtIGgyby5nZXRHcmlkKGdyaWRfaWQgPSAicmZfZ3JpZCIsIA0KICAgICAgICAgICAgICAgICAgICAgICAgIHNvcnRfYnkgPSAiYXVjIiwgDQogICAgICAgICAgICAgICAgICAgICAgICAgZGVjcmVhc2luZyA9IEZBTFNFKQ0KDQojIEJlc3QgbW9kZWwgY2hvc2VuIGJ5IHZhbGlkYXRpb24gZXJyb3I6IA0KYmVzdF9tb2RlbF9pZCA8LSBncmlkX3BlcmZAbW9kZWxfaWRzW1sxXV0NCmJlc3RfbW9kZWwgPC0gaDJvLmdldE1vZGVsKGJlc3RfbW9kZWxfaWQpDQoNCg0KbXlfY21fY29tX3JmIDwtIGZ1bmN0aW9uKHRocmUpIHsNCiAgZHVfYmFvX3Byb2IgPC0gaDJvLnByZWRpY3QoYmVzdF9tb2RlbCwgdGVzdCkgJT4lIGFzLmRhdGEuZnJhbWUoKSAlPiUgcHVsbChGcmF1ZCkNCiAgZHVfYmFvIDwtIGNhc2Vfd2hlbihkdV9iYW9fcHJvYiA+PSB0aHJlIH4gIkZyYXVkIiwgDQogICAgICAgICAgICAgICAgICAgICAgZHVfYmFvX3Byb2IgPCB0aHJlIH4gIk5vbkZyYXVkIikgJT4lIGFzLmZhY3RvcigpDQogIGNtIDwtIGNvbmZ1c2lvbk1hdHJpeChkdV9iYW8sIHRlc3RfZm9yX2Nyb3NzJENsYXNzLCBwb3NpdGl2ZSA9ICJGcmF1ZCIpDQogIHJldHVybihjbSkNCiAgDQp9DQoNCg0KcmVzdWx0c19saXN0X3JmIDwtIGxhcHBseShteV90aHJlc2hvbGQsIG15X2NtX2NvbV9yZikNCg0KDQp2aXNfZGV0ZWN0aW9uX3JhdGVfcmYgPC0gZnVuY3Rpb24oeCkgew0KICANCiAgcmVzdWx0c19saXN0X3JmW1t4XV0kdGFibGUgJT4lIGFzLmRhdGEuZnJhbWUoKSAtPiBtDQogIHJhdGUgPC0gcm91bmQoMTAwKm0kRnJlcVsxXSAvIHN1bShtJEZyZXFbYygxLCAyKV0pLCAyKQ0KICBhY2MgPC0gcm91bmQoMTAwKnN1bShtJEZyZXFbYygxLCA0KV0pIC8gc3VtKG0kRnJlcSksIDIpDQogIGFjYyA8LSBwYXN0ZTAoYWNjLCAiJSIpDQogIA0KICBtICU+JSANCiAgICBnZ3Bsb3QoYWVzKFJlZmVyZW5jZSwgRnJlcSwgZmlsbCA9IFByZWRpY3Rpb24pKSArDQogICAgZ2VvbV9jb2wocG9zaXRpb24gPSAiZmlsbCIpICsgDQogICAgc2NhbGVfZmlsbF9tYW51YWwodmFsdWVzID0gYygiI2U0MWExYyIsICIjMzc3ZWI4IiksIG5hbWUgPSAiIikgKyANCiAgICB0aGVtZShwYW5lbC5ncmlkLm1pbm9yLnkgPSBlbGVtZW50X2JsYW5rKCkpICsgDQogICAgdGhlbWUocGFuZWwuZ3JpZC5taW5vci54ID0gZWxlbWVudF9ibGFuaygpKSArIA0KICAgIHNjYWxlX3lfY29udGludW91cyhsYWJlbHMgPSBzY2FsZXM6OnBlcmNlbnQpICsgDQogICAgbGFicyh4ID0gTlVMTCwgeSA9IE5VTEwsIA0KICAgICAgICAgdGl0bGUgPSBwYXN0ZTAoIkRldGVjdGluZyBGcmF1ZCBDYXNlcyB3aGVuIFRocmVzaG9sZCA9ICIsIG15X3RocmVzaG9sZFt4XSksIA0KICAgICAgICAgc3VidGl0bGUgPSBwYXN0ZTAoIkRldGVjdGluZyBSYXRlIGZvciBGcmF1ZCBDYXNlczogIiwgcmF0ZSwgIiUiLCAiLCAiLCAiQWNjdXJhY3k6ICIsIGFjYykpDQogIH0NCg0KDQpncmlkRXh0cmE6OmdyaWQuYXJyYW5nZSh2aXNfZGV0ZWN0aW9uX3JhdGVfcmYoMSksIA0KICAgICAgICAgICAgICAgICAgICAgICAgdmlzX2RldGVjdGlvbl9yYXRlX3JmKDIpLCANCiAgICAgICAgICAgICAgICAgICAgICAgIHZpc19kZXRlY3Rpb25fcmF0ZV9yZigzKSwgDQogICAgICAgICAgICAgICAgICAgICAgICB2aXNfZGV0ZWN0aW9uX3JhdGVfcmYoNCkpDQoNCg0KYGBgDQoNCg0KQmVjYXVzZSBvZiB0aGUgY29tYmluYXRvcmlhbCBleHBsb3Npb24sIGVhY2ggYWRkaXRpb25hbCBoeXBlcnBhcmFtZXRlciB0aGF0IGdldHMgYWRkZWQgdG8gb3VyIGdyaWQgc2VhcmNoIGhhcyBhIGh1Z2UgZWZmZWN0IG9uIHRoZSB0aW1lIHRvIGNvbXBsZXRlLiBDb25zZXF1ZW50bHksIGgybyBwcm92aWRlcyBhbiBhZGRpdGlvbmFsIGdyaWQgc2VhcmNoIHBhdGggY2FsbGVkIOKAnFJhbmRvbURpc2NyZXRl4oCdLCB3aGljaCB3aWxsIGp1bXAgZnJvbSBvbmUgcmFuZG9tIGNvbWJpbmF0aW9uIHRvIGFub3RoZXIgYW5kIHN0b3Agb25jZSBhIGNlcnRhaW4gbGV2ZWwgb2YgaW1wcm92ZW1lbnQgaGFzIGJlZW4gbWFkZSwgY2VydGFpbiBhbW91bnQgb2YgdGltZSBoYXMgYmVlbiBleGNlZWRlZCwgb3IgYSBjZXJ0YWluIGFtb3VudCBvZiBtb2RlbHMgaGF2ZSBiZWVuIHJhbiAob3IgYSBjb21iaW5hdGlvbiBvZiB0aGVzZSBoYXZlIGJlZW4gbWV0KS4gQWx0aG91Z2ggdXNpbmcgYSByYW5kb20gZGlzY3JldGUgc2VhcmNoIHBhdGggd2lsbCBsaWtlbHkgbm90IGZpbmQgdGhlIG9wdGltYWwgbW9kZWwsIGl0IHR5cGljYWxseSBkb2VzIGEgZ29vZCBqb2Igb2YgZmluZGluZyBhIHZlcnkgZ29vZCBtb2RlbC4NCg0KYGBge3IsIGZpZy5mdWxsd2lkdGggPSBUUlVFLCBmaWcuaGVpZ2h0PTEyLCBmaWcud2lkdGg9MTQsIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9DQojIFNldCBoeXBlcnBhcmFtZXRlciBncmlkMjogDQpoeXBlcjJfZ3JpZC5oMm8gPC0gbGlzdChudHJlZXMgPSBzZXEoMTAwLCA4MDAsIGJ5ID0gMTAwKSwNCiAgICAgICAgICAgICAgICAgICAgICAgIG10cmllcyA9IHNlcSg1LCAyNSwgYnkgPSA1KSwNCiAgICAgICAgICAgICAgICAgICAgICAgIG1heF9kZXB0aCA9IHNlcSgxMCwgMjUsIGJ5ID0gNSksDQogICAgICAgICAgICAgICAgICAgICAgICBtaW5fcm93cyA9IHNlcSgxLCA1LCBieSA9IDEpLA0KICAgICAgICAgICAgICAgICAgICAgICAgbmJpbnMgPSBzZXEoMTAsIDIwLCBieSA9IDIpLA0KICAgICAgICAgICAgICAgICAgICAgICAgc2FtcGxlX3JhdGUgPSBjKDAuNCwgMC41NSwgMC42MzIsIDAuNzUsIDAuODApKQ0KDQojIFNlbGVjdCByYW5kb20gZ3JpZCBzZWFyY2ggY3JpdGVyaWE6IA0Kc2VhcmNoX2NyaXRlcmlhMiA8LSBsaXN0KHN0cmF0ZWd5ID0gIlJhbmRvbURpc2NyZXRlIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICBzdG9wcGluZ19tZXRyaWMgPSAiQVVDIiwNCiAgICAgICAgICAgICAgICAgICAgICAgICBzdG9wcGluZ190b2xlcmFuY2UgPSAwLjAwNSwNCiAgICAgICAgICAgICAgICAgICAgICAgICBzdG9wcGluZ19yb3VuZHMgPSAxMCwNCiAgICAgICAgICAgICAgICAgICAgICAgICBtYXhfcnVudGltZV9zZWNzID0gMzAqNjApDQoNCiMgQnVpbGQgZ3JpZCBzZWFyY2g6IA0KDQpzeXN0ZW0udGltZShyYW5kb21fZ3JpZCA8LSBoMm8uZ3JpZChhbGdvcml0aG0gPSAicmFuZG9tRm9yZXN0IiwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGdyaWRfaWQgPSAicmZfZ3JpZDIiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeCA9IHgsIHkgPSB5LCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRyYWluaW5nX2ZyYW1lID0gdHJhaW4sDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBoeXBlcl9wYXJhbXMgPSBoeXBlcjJfZ3JpZC5oMm8sDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBzZWFyY2hfY3JpdGVyaWEgPSBzZWFyY2hfY3JpdGVyaWEyKSkNCg0KIyBDb2xsZWN0IHRoZSByZXN1bHRzIGFuZCBzb3J0IGJ5IG91ciBtb2RlbCBwZXJmb3JtYW5jZSBtZXRyaWMgb2YgY2hvaWNlOiANCmdyaWRfcGVyZjIgPC0gaDJvLmdldEdyaWQoZ3JpZF9pZCA9ICJyZl9ncmlkMiIsIA0KICAgICAgICAgICAgICAgICAgICAgICAgICBzb3J0X2J5ID0gImF1YyIsIA0KICAgICAgICAgICAgICAgICAgICAgICAgICBkZWNyZWFzaW5nID0gRkFMU0UpDQoNCiMgQmVzdCBNb2RlbDogDQpiZXN0X21vZGVsMiA8LSBoMm8uZ2V0TW9kZWwoZ3JpZF9wZXJmMkBtb2RlbF9pZHNbWzFdXSkNCg0KDQpteV9jbV9jb21fcmYyIDwtIGZ1bmN0aW9uKHRocmUpIHsNCiAgZHVfYmFvX3Byb2IgPC0gaDJvLnByZWRpY3QoYmVzdF9tb2RlbDIsIHRlc3QpICU+JSBhcy5kYXRhLmZyYW1lKCkgJT4lIHB1bGwoRnJhdWQpDQogIGR1X2JhbyA8LSBjYXNlX3doZW4oZHVfYmFvX3Byb2IgPj0gdGhyZSB+ICJGcmF1ZCIsIA0KICAgICAgICAgICAgICAgICAgICAgIGR1X2Jhb19wcm9iIDwgdGhyZSB+ICJOb25GcmF1ZCIpICU+JSBhcy5mYWN0b3IoKQ0KICBjbSA8LSBjb25mdXNpb25NYXRyaXgoZHVfYmFvLCB0ZXN0X2Zvcl9jcm9zcyRDbGFzcywgcG9zaXRpdmUgPSAiRnJhdWQiKQ0KICByZXR1cm4oY20pDQogIA0KfQ0KDQoNCnJlc3VsdHNfbGlzdF9yZjIgPC0gbGFwcGx5KG15X3RocmVzaG9sZCwgbXlfY21fY29tX3JmMikNCg0KDQp2aXNfZGV0ZWN0aW9uX3JhdGVfcmYyIDwtIGZ1bmN0aW9uKHgpIHsNCiAgDQogIHJlc3VsdHNfbGlzdF9yZjJbW3hdXSR0YWJsZSAlPiUgYXMuZGF0YS5mcmFtZSgpIC0+IG0NCiAgcmF0ZSA8LSByb3VuZCgxMDAqbSRGcmVxWzFdIC8gc3VtKG0kRnJlcVtjKDEsIDIpXSksIDIpDQogIGFjYyA8LSByb3VuZCgxMDAqc3VtKG0kRnJlcVtjKDEsIDQpXSkgLyBzdW0obSRGcmVxKSwgMikNCiAgYWNjIDwtIHBhc3RlMChhY2MsICIlIikNCiAgDQogIG0gJT4lIA0KICAgIGdncGxvdChhZXMoUmVmZXJlbmNlLCBGcmVxLCBmaWxsID0gUHJlZGljdGlvbikpICsNCiAgICBnZW9tX2NvbChwb3NpdGlvbiA9ICJmaWxsIikgKyANCiAgICBzY2FsZV9maWxsX21hbnVhbCh2YWx1ZXMgPSBjKCIjZTQxYTFjIiwgIiMzNzdlYjgiKSwgbmFtZSA9ICIiKSArIA0KICAgIHRoZW1lKHBhbmVsLmdyaWQubWlub3IueSA9IGVsZW1lbnRfYmxhbmsoKSkgKyANCiAgICB0aGVtZShwYW5lbC5ncmlkLm1pbm9yLnggPSBlbGVtZW50X2JsYW5rKCkpICsgDQogICAgc2NhbGVfeV9jb250aW51b3VzKGxhYmVscyA9IHNjYWxlczo6cGVyY2VudCkgKyANCiAgICBsYWJzKHggPSBOVUxMLCB5ID0gTlVMTCwgDQogICAgICAgICB0aXRsZSA9IHBhc3RlMCgiRGV0ZWN0aW5nIEZyYXVkIENhc2VzIHdoZW4gVGhyZXNob2xkID0gIiwgbXlfdGhyZXNob2xkW3hdKSwgDQogICAgICAgICBzdWJ0aXRsZSA9IHBhc3RlMCgiRGV0ZWN0aW5nIFJhdGUgZm9yIEZyYXVkIENhc2VzOiAiLCByYXRlLCAiJSIsICIsICIsICJBY2N1cmFjeTogIiwgYWNjKSkNCiAgfQ0KDQoNCmdyaWRFeHRyYTo6Z3JpZC5hcnJhbmdlKHZpc19kZXRlY3Rpb25fcmF0ZV9yZjIoMSksIA0KICAgICAgICAgICAgICAgICAgICAgICAgdmlzX2RldGVjdGlvbl9yYXRlX3JmMigyKSwgDQogICAgICAgICAgICAgICAgICAgICAgICB2aXNfZGV0ZWN0aW9uX3JhdGVfcmYyKDMpLCANCiAgICAgICAgICAgICAgICAgICAgICAgIHZpc19kZXRlY3Rpb25fcmF0ZV9yZjIoNCkpDQoNCg0KYGBgDQoNCg0KDQoNCg==

Detecting Financial Fraud Using Machine Learning: War Against Imbalanced Data

R for Pleasure

Nguyen Chi Dung

Demand for Detecting Financial Fraud

Machine Learning Approach to Detecting Fraud Transaction

Challenge when Using Machine Learning Approach to Detecting Fraud Transaction

A Simple Solution: Searching Optimal Threshold

Random Forests