Incognia Data Analysis Report
Introduction
Statistical analysis is a efficient way to get insights about any data, making anyone able to ask the right question. In addition, using the best technologies is crucial to get better results and faster. Therefore, this analysis used the powerful Language R to load and transform the dataset provided, creating significant visualizations with RStudio tool. Thus, this report has the result of statistical thinking that collected meaningful data to answer the right question with detailed analysis of patterns found in the data, making possible to draw conclusions that go beyond the observed data.
Dataset description
Each event in the dataset analysed was a login to the client’s app and the purpose of the analysis was to find patterns related to accounts and devices that indicate suspicious behavior, witch is possibly associated with fraud. The dataset contains records from July 2021 and it’s a great dataset for evaluating linear regression models.
Initial dataset schema
- event_id: identifier of the event.
- event_timestamp: event datetime in milliseconds.
- account_id: identifier of the account associated with the event.
- device: identifier of the device that performed the operation.
- distance_fl: distance (in meters) from the device, at the time of the event, to one of the frequent locations related to the account.
- device_age_days: days since an account appeared related to a device.
- is_emulator: indicates whether the device is an emulator.
- has_fake_location: indicates whether the device was using false locations at the time of the operation.
- has_root_permissions: indicates whether the device has device administrator permissions.
- app_is_from_official_store: indicates whether or not the app used to perform the operation came from an official store.
Dataset processing
Loading dataset
tb <- read.table("Dados/hugo_incognia_db_for_da_test.csv",
dec = ".",
sep = ",",
h = T,
fileEncoding = "windows-1252")Dataset general information
Dataset size is 444758 rows and 10 columns
Compactly displaying the structure of dataset
## 'data.frame': 444758 obs. of 10 variables:
## $ event_id : chr "59bcca18-c726" "64c883df-2a7b" "527f8468-6a6a" "e062b8d9-6b5a" ...
## $ event_timestamp : num 1.63e+12 1.63e+12 1.63e+12 1.63e+12 1.63e+12 ...
## $ account_id : int 1893155083 2050114867 422735906 1766209306 1860027669 256278006 1171219478 1498658439 1355561417 2086562840 ...
## $ device : int 1801928094 1971517057 1956878295 1208828324 1768952722 1736326652 1079123361 1470762726 550479659 1021842367 ...
## $ distance_to_frequent_location: num 1.85 36.26 226.41 0.11 1.73 ...
## $ device_age_days : int 527 184 92 366 278 524 265 181 1 505 ...
## $ is_emulator : chr "false" "false" "false" "false" ...
## $ has_fake_location : chr "false" "false" "false" "false" ...
## $ has_root_permissions : chr "false" "false" "false" "false" ...
## $ app_is_from_official_store : chr "true" "true" "true" "true" ...Dataset transformation
Removing id column, not necessary for analysis
tb$event_id <- NULLShortening some column names
names(tb)[4] <- "distance_fl"
names(tb)[5] <- "device_age"
names(tb)[7] <- "fake_location"
names(tb)[8] <- "root_permissions"
names(tb)[9] <- "official_store"Converting variable to integer
tb$distance_fl <- as.integer(tb$distance_fl)Converting miliseconds to timestamp
tb$event_timestamp <- as.POSIXct(tb$event_timestamp / 1000,
origin = "1970-01-01",
tz = "UTC")Creating variable Event Hour, Date and Weekday
tb$event_hour <- format(tb$event_timestamp,"%H")
tb$event_hour <- as.numeric(tb$event_hour)
tb$event_date <- as.Date(tb$event_timestamp,format="%Y-%m-%d")
Sys.setlocale("LC_TIME","English")## [1] "English_United States.1252"tb$weekday <- weekdays(tb$event_date)
tb$wday <- as.integer(as.POSIXlt(tb$event_date)$wday)The date range of the dataset is: 2021-07-01 to 2021-07-31
Missing values
sapply(tb, function(x) sum(is.na(x)))## event_timestamp account_id device distance_fl
## 0 0 0 678
## device_age is_emulator fake_location root_permissions
## 0 0 0 0
## official_store event_hour event_date weekday
## 0 0 0 0
## wday
## 0Amelia::missmap(tb, main = "Missing Values")
There are 678 missing values. However, all missing values are in a location variable, which is crucial to determine risk because it’s a login attempt without location information. Solution was replace for -1 instead removing.
tb <- tidyr::replace_na(tb, list(distance_fl = -1))Creating a Device Age Category
tb$age_category <- cut(tb$device_age,
breaks = c(0,1,7,30,365,Inf),
labels = c("Day","Week","Month","Year","Year+"), right = FALSE)Creating a Distance to frequent location Category
tb$distance_group <- cut(tb$distance_fl,
breaks = c(-1,0,1,10,Inf),
labels = c("No Location",
"Freq Location",
"Near FL",
"Far FL"), right = FALSE)Creating Risk Score variable
tb$score_risk = 0
tb$score_risk <- tb$score_risk + ifelse(tb$event_hour>=0 & tb$event_hour<=6,1,0)
tb$score_risk <- tb$score_risk + ifelse(tb$distance_group=="No Location",3,0)
tb$score_risk <- tb$score_risk + ifelse(tb$distance_group=="Far FL",2,0)
tb$score_risk <- tb$score_risk + ifelse(tb$distance_group=="Near FL",1,0)
tb$score_risk <- tb$score_risk + ifelse(tb$age_category=="Day",1,0)
tb$score_risk <- tb$score_risk + ifelse(tb$is_emulator=="true",3,0)
tb$score_risk <- tb$score_risk + ifelse(tb$fake_location=="true",3,0)
tb$score_risk <- tb$score_risk + ifelse(tb$root_permissions=="true",3,0)
tb$score_risk <- tb$score_risk + ifelse(tb$official_store=="false",3,0)Creating Risk Level (category based on score)
tb$risk_level <- cut(tb$score_risk,
breaks = c(0,2,3,Inf),
labels = c("Low","Medium","High"), right = FALSE)Variables added to the dataset schema
- event_hour: hour of the event day.
- event_date: event date (yyyy-mm-dd)
- age_category: variable device_age sliced in a category (“Day”, “Week”, “Month”, “Year”, “Year+”)
- distance_group: variable distance_fl sliced in a category (“No Location”, “Freq Location”, “Near FL”, “Far FL”)
- score_risk: variable that accumulates points whenever another variable indicates some risk
- risk_level: variable score_risk sliced in a category (“Low”, “Medium”, “High”)
Graphical representation of dataset
Heatmap to analyse correlation between variables
M = cor(tb_num)
corrplot::corrplot(M, method = 'color', order = 'alphabet')
In this heatmap it is possible to observe that there is little or no correlation between the numerical variables
Barplot analysis
In the bar charts above, it is possible to observe the distribution of events in the risk_level variable, in which most of the events in this dataset can be considered as low risk.
Regarding distance_to_frequent_location, the information shows that most of the events occurred in places that are not exactly at the frequent location.
About the device age category, most of the events are associated with devices aged between one month and one year.
According to the Hours of Day chart, it is quite visible that few events occurred between midnight and 6am.
Among the Boolean variables, there were very few contradictory cases related to suspicious behavior of the device.
Boxplot analysis - Numeric Variables
In the box plot diagram above, it is possible to analyze the representation of the observed data variation of the numeric variables.
Boxplot analysis - Boolean Variables
The box plots above show the data variation of the Boolean variables in relation to the Device Age in an attempt to identify any correlation between all of them.
Moreover, in the last box plot, there is a relationship between Device Age and Distance to Frequent Location category, showing how most of the devices with no locations available are device in the minimal age.
Scatter Plot analysis
In the scatter plot above, it is easy observe the data and how scattered data is. In addition, there is little relationship between the distance from the frequent location and device age.
Histogram of distribution
This histogram illustrates the distribution of events in relation to the maximum age found for each device, concluding that there are usually many more devices with a lower age.
Area chart analysis
The area chart gives a good perception of the proportionality of the risk level over the days the event occurred. According to the chart, there is no oscillation in high-risk events in relation to the day of the week, while the risk levels low and medium show oscillation on days that are weekends.
Meanwhile, the stacked bar chart below confirms this information showing fewer events on Sundays.
Stacked bar chart analysis
Analysis Coclusion
The dataset analysis found patterns related to the accounts and devices that indicate suspicious behavior, possibly associated with fraud. In this report, it is possible to understand how these patterns occur and how often. Moreover, with the patterns found, it was possible to create a variable that measures risk with a score points. The higher the score, the greater the risk. In addition, another variable was defined to categorize the risk in 3 levels: Low, Medium and High. Thus, Incognia will be able to improve the detection algorithm and increase efficiency in the communication with the financial client in order to avoid fraud.
Machine Learning Model
Objective
Create a machine learning model to predict whether a financial event is a fraud event based on patterns found in a dataset.
Preparing data to the model
Sample random rows in dataframe
Only 3000 rows were selected at random from the dataset. More than this value requires a higher computational level.
Sample random rows in dataframe
df = data.frame(tb)
tb_ml <- df[sample(nrow(df), 3000), ]Suggesting a variable as a possible fraud Risk Score greater than 3
tb_ml$is_fraud <- ifelse(tb_ml$score_risk>=4,1,0)Selecting rows according to variable IS_FRAUD
index <- createDataPartition(tb_ml$is_fraud, p = 0.75, list = FALSE)Setting training data as a subset
data_training <- tb_ml[index,]Percentage comparison between training classes and original dataset
data_comparison <- cbind(prop.table(table(data_training$is_fraud)),
prop.table(table(tb_ml$is_fraud)))
colnames(data_comparison) <- c("Training", "Original")
data_comparison## Training Original
## 0 0.994669036 0.995
## 1 0.005330964 0.005Plot - Training vs original
ggplot(melt_data_comparison, aes(x = X1, y = value)) +
geom_bar( aes(fill = X2), stat = "identity", position = "dodge") +
ggtitle("Training vs original") +
theme(axis.text.x = element_text(angle = 90, hjust = 1))
Building model version 1
model_v1 <- randomForest(is_fraud ~ ., data = data_training)##
## Call:
## randomForest(formula = is_fraud ~ ., data = data_training)
## Type of random forest: classification
## Number of trees: 500
## No. of variables tried at each split: 2
##
## OOB estimate of error rate: 0.4%
## Confusion matrix:
## 0 1 class.error
## 0 2239 0 0.00
## 1 9 3 0.75
Predicting the test dataset
predict_v1 <- predict(model_v1, data_test)Confusion Matrix to calculate a cross-tabulation of observed and predicted classes
cm_v1 <- caret::confusionMatrix(predict_v1, data_test$is_fraud, positive = "1")
cm_v1## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 746 2
## 1 0 1
##
## Accuracy : 0.9973
## 95% CI : (0.9904, 0.9997)
## No Information Rate : 0.996
## P-Value [Acc > NIR] : 0.4227
##
## Kappa : 0.499
##
## Mcnemar's Test P-Value : 0.4795
##
## Sensitivity : 0.333333
## Specificity : 1.000000
## Pos Pred Value : 1.000000
## Neg Pred Value : 0.997326
## Prevalence : 0.004005
## Detection Rate : 0.001335
## Detection Prevalence : 0.001335
## Balanced Accuracy : 0.666667
##
## 'Positive' Class : 1
## Precision, Recall e F1-Score, measures to evaluate predict model
# Precision, Recall e F1-Score, measures to evaluate predict model
y <- data_test$is_fraud
y_pred_v1 <- predict_v1
precision <- posPredValue(y_pred_v1, y)
recall <- sensitivity(y_pred_v1, y)
F1 <- (2 * precision * recall) / (precision + recall)
df2 <- data.frame(precision,recall,F1)
names(df2) <- c("Precision","Recall","F1")
df2## Precision Recall F1
## 1 0.9973262 1 0.9986613SMOTE algorithm for unbalanced classification problems
set.seed(9560)
data_training_bal <- SMOTE(is_fraud ~ ., data = data_training) Building model version 2
model_v2 <- randomForest(is_fraud ~ ., data = data_training_bal)
model_v2##
## Call:
## randomForest(formula = is_fraud ~ ., data = data_training_bal)
## Type of random forest: classification
## Number of trees: 500
## No. of variables tried at each split: 2
##
## OOB estimate of error rate: 8.33%
## Confusion matrix:
## 0 1 class.error
## 0 45 3 0.0625000
## 1 4 32 0.1111111plot(model_v2)
Predicting the test dataset
predict_v2 <- predict(model_v2, data_test)Confusion Matrix
cm_v2 <- caret::confusionMatrix(predict_v2, data_test$is_fraud, positive = "1")
cm_v2## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 704 1
## 1 42 2
##
## Accuracy : 0.9426
## 95% CI : (0.9234, 0.9581)
## No Information Rate : 0.996
## P-Value [Acc > NIR] : 1
##
## Kappa : 0.0782
##
## Mcnemar's Test P-Value : 1.061e-09
##
## Sensitivity : 0.666667
## Specificity : 0.943700
## Pos Pred Value : 0.045455
## Neg Pred Value : 0.998582
## Prevalence : 0.004005
## Detection Rate : 0.002670
## Detection Prevalence : 0.058745
## Balanced Accuracy : 0.805183
##
## 'Positive' Class : 1
## Precision, Recall e F1-Score
y <- data_test$is_fraud
y_pred_v2 <- predict_v2
precision <- posPredValue(y_pred_v2, y)
recall <- sensitivity(y_pred_v2, y)
F1 <- (2 * precision * recall) / (precision + recall)
df3 <- data.frame(precision,recall,F1)
names(df3) <- c("Precision","Recall","F1")
df3## Precision Recall F1
## 1 0.9985816 0.9436997 0.9703653Most important variables to predict
varImpPlot(model_v2)
Ranking of the most important variables
imp_var <- importance(model_v2)
varImportance <- data.frame(Variables = row.names(imp_var),
Importance = round(imp_var[ ,'MeanDecreaseGini'],2))
rankImportance <- varImportance %>%
mutate(Rank = paste0('#', dense_rank(desc(Importance))))
Building model version 3 with the most important variables
model_v3 <- randomForest(is_fraud ~ device_age +
event_hour +
distance_fl +
weekday +
official_store +
root_permissions,
data = data_training_bal)plot(model_v3)
Predicting the test dataset
predict_v3 <- predict(model_v3, data_test)Confusion Matrix
cm_v3 <- caret::confusionMatrix(predict_v3, data_test$is_fraud, positive = "1")
cm_v3## Confusion Matrix and Statistics
##
## Reference
## Prediction 0 1
## 0 703 0
## 1 43 3
##
## Accuracy : 0.9426
## 95% CI : (0.9234, 0.9581)
## No Information Rate : 0.996
## P-Value [Acc > NIR] : 1
##
## Kappa : 0.1158
##
## Mcnemar's Test P-Value : 1.504e-10
##
## Sensitivity : 1.000000
## Specificity : 0.942359
## Pos Pred Value : 0.065217
## Neg Pred Value : 1.000000
## Prevalence : 0.004005
## Detection Rate : 0.004005
## Detection Prevalence : 0.061415
## Balanced Accuracy : 0.971180
##
## 'Positive' Class : 1
## Precision, Recall e F1-Score
y <- data_test$is_fraud
y_pred_v3 <- predict_v3
precision <- posPredValue(y_pred_v3, y)
recall <- sensitivity(y_pred_v3, y)
F1 <- (2 * precision * recall) / (precision + recall)
df4 <- data.frame(precision,recall,F1)
names(df4) <- c("Precision","Recall","F1")
df4## Precision Recall F1
## 1 1 0.9423592 0.9703244Salving model file
rds <- paste("Dados/",
format(Sys.time(), "%y%m%d%H%M%S"),
"_model_v3.rds",sep = "")
saveRDS(model_v3, file = rds)Predicting three random events
# Input
device_age <- c(0, 10, 100)
event_hour <- c("4", "10", "11")
distance_fl <- c(1000, 1, 0)
weekday <- c("Sunday","Sunday","Sunday")
official_store <- c("true","true","true")
root_permissions <- c("false","false","false")Predictions results
pred_new_events <- predict(model_v3, new_events)
pred = data.frame(pred_new_events)
pred$is_fraud <- ifelse(pred_new_events==0,"No","Yes")
pred## pred_new_events is_fraud
## 1 1 Yes
## 2 0 No
## 3 0 NoModel Results
To create a machine learning model it would be important to have a sample with descriptive information determining whether each event is a fraud event. With this information, it would be possible to create a prediction algorithm model to predict which future events will be a fraud event based on the training and testing model.
As the dataset did not contain the information (yes/no fraud), then a risk level was suggested based on the patterns found. An experiment was done based on the hypothesis that a score of 4+ is a fraud event. However, the rules that defined this score make the confidence level of the experiment result in 100%. That is, the model does not make mistakes in the predictions. Therefore, the ideal is to have the information (yes/no fraud) to create a predictive model without being biased.