Data Pre-Proccessing
Identifying the columns with missing values
The is.na function shows the number of nulls per column.
colSums(is.na(data)==TRUE|data=='')
#> Age_Categories Sex_of_driver Educational_level
#> 1548 178 841
#> Vehicle_driver_relation Driving_experience Lanes_or_Medians
#> 593 862 442
#> Types_of_Junction Road_surface_type Light_conditions
#> 1078 172 0
#> Weather_conditions Type_of_collision Vehicle_movement
#> 292 169 396
#> Pedestrian_movement Cause_of_accident Accident_severity
#> 0 25 13
The Number of Missing Values
There are a total of 6582 Null values in the dataset. Based on the line graph below, Age_Categories has the highest Null values of more than 1500 records. While Accident_severity has the lowest Null values with only 13 records.
gg_miss_var(data)
#> Warning: The `guide` argument in `scale_*()` cannot be `FALSE`. This was deprecated in
#> ggplot2 3.3.4.
#> ℹ Please use "none" instead.
#> ℹ The deprecated feature was likely used in the naniar package.
#> Please report the issue at <]8;;https://github.com/njtierney/naniar/issueshttps://github.com/njtierney/naniar/issues]8;;>.
The Precentage of missing values
As the graph below shows, Black color cells indicate percentage of Null values and gray color cells indicate percentage of Non Null values. Age_Categories has the highest percentage of Null values with 12.57% and the lowest percentage of Null values is 0.09%. Overall, this dataset has 3.6% of Null values compared to the total size of the dataset.
vis_miss(data)
#> Warning: `gather_()` was deprecated in tidyr 1.2.0.
#> ℹ Please use `gather()` instead.
#> ℹ The deprecated feature was likely used in the visdat package.
#> Please report the issue at <]8;;https://github.com/ropensci/visdat/issueshttps://github.com/ropensci/visdat/issues]8;;>.
Impute Data
The missing data can create problems for analyzing data. Replacing missing data with substituted values is seen as a way to avoid pitfalls involved with listwise deletion of cases that have missing values. In the “Accident_Severity” column. The null values were replaced by the mean of the feature.
data$Accident_severity[is.na(data$Accident_severity)==TRUE] = round(mean(data$Accident_severity, na.rm = TRUE))The null values in the four other columns including “Age_Categories”,
“Educational_level”, “Driving_experience”, “Lanes_or_Medians”,
“Educational_level”, “Driving_experience”, “Lanes_or_Medians” were
replaced with the mode of the feature as shown in below
respectively.
getmode <- function(a) {
uniqage <- unique(a)
uniqage[which.max(tabulate(match(a, uniqage)))]
}
data$Age_Categories[is.na(data$Age_Categories)==TRUE]= getmode(data$Age_Categories)
print( getmode(data$Age_Categories))
#> [1] "18-30"
data$Educational_level[is.na(data$Educational_level)==TRUE]= getmode(data$Educational_level)
print( getmode(data$Educational_level))
#> [1] "Junior high school"
data$Driving_experience[is.na(data$Driving_experience)==TRUE] = getmode(data$Driving_experience)
print(getmode(data$Driving_experience))
#> [1] "5-10yr"
data$Lanes_or_Medians[is.na(data$Lanes_or_Medians)==TRUE] = getmode(data$Lanes_or_Medians)
print(getmode(data$Lanes_or_Medians))
#> [1] "Two-way (divided with broken lines road marking)"
The missing values in other columns including, “Road_surface_type”,
“Type_of_collision”, “Cause_of_accident”, “Types_of_Junction”,
“Vehicle_movement”, “Vehicle_driver_relation”, “Weather_conditions”,
“Sex_of_driver” were replaced by the “Unknown” keyword because of the
importance and high sensitivity of the columns’ values in the analysis
stage and obviously making a better decision based on them.
data$Road_surface_type[is.na(data$Road_surface_type)==TRUE]= "Unknown"
data$Type_of_collision[is.na(data$Type_of_collision)==TRUE]= "Unknown"
data$Cause_of_accident[is.na(data$Cause_of_accident)==TRUE]= "Unknown"
data$Types_of_Junction [is.na(data$Types_of_Junction )==TRUE]= "Unknown"
data$Vehicle_movement[is.na(data$Vehicle_movement)==TRUE]= "Unknown"
data$Vehicle_driver_relation[is.na(data$Vehicle_driver_relation)==TRUE]= "Unknown"
data$Weather_conditions[is.na(data$Weather_conditions)==TRUE]= "Unknown"
data$Sex_of_driver[is.na(data$Sex_of_driver)==TRUE]= "Unknown"
After imputation the dataset, the vis_miss function was run to recheck the null data situation. Fortunately, There was no missing data anymore.
vis_miss(data)
colSums(is.na(data)==TRUE|data=='')
#> Age_Categories Sex_of_driver Educational_level
#> 0 0 0
#> Vehicle_driver_relation Driving_experience Lanes_or_Medians
#> 0 0 0
#> Types_of_Junction Road_surface_type Light_conditions
#> 0 0 0
#> Weather_conditions Type_of_collision Vehicle_movement
#> 0 0 0
#> Pedestrian_movement Cause_of_accident Accident_severity
#> 0 0 0
