Many real world data is dirty and needs to be cleaned before used in code. The process of cleaning a dataset is called Data Preprocessing.
Preprocessing of data includes below steps : 1. Taking care of missing data 2. Catgorical Data 3. Splitting data into training and test data sets 4. feature scaling
#import the data set first
# data is present in Data.csv file in current folder or working directory(use command setwd("Current location of the Data.csv") to set the working directory to current directory.)
dataset <- read.csv("Data.csv")
#Showing the dataset
dataset## Country Age Salary Purchased
## 1 France 44 72000 No
## 2 Spain 27 48000 Yes
## 3 Germany 30 54000 No
## 4 Spain 38 61000 No
## 5 Germany 40 NA Yes
## 6 France 35 58000 Yes
## 7 Spain NA 52000 No
## 8 France 48 79000 Yes
## 9 Germany 50 83000 No
## 10 France 37 67000 Yes# So missing values present in both Age and Salary Columns
#taking care of missing values
# By replacing it to the average value for non NA entries.
dataset$Age <- ifelse(is.na(dataset$Age),
ave(dataset$Age, FUN = function(x)
mean(x, na.rm = TRUE)),
dataset$Age)
dataset$Salary <- ifelse(is.na(dataset$Salary),
ave(dataset$Salary, FUN = function(x)
mean(x, na.rm = TRUE)),
dataset$Salary)How the ave() function works here ?
read it like this : we are changing Age column of dataset and if the column entry is NA then, take the average of the dataset$Age column where function FUN is function of x which calculates the mean excluding(na.rm = TRUE) the NA values.
else,
take whatever present in dataset$Age
mean() :
#defining x = 1 2 3
x <- 1:3
#introducing missing value
x[1] <- NA
# mean = NA
mean(x)## [1] NA# mean = mean excluding the NA value
mean(x, na.rm = T)## [1] 2.5So finally the dataset looks like this :
dataset## Country Age Salary Purchased
## 1 France 44.00000 72000.00 No
## 2 Spain 27.00000 48000.00 Yes
## 3 Germany 30.00000 54000.00 No
## 4 Spain 38.00000 61000.00 No
## 5 Germany 40.00000 63777.78 Yes
## 6 France 35.00000 58000.00 Yes
## 7 Spain 38.77778 52000.00 No
## 8 France 48.00000 79000.00 Yes
## 9 Germany 50.00000 83000.00 No
## 10 France 37.00000 67000.00 YesNow the missing values are replaced by the average of the respective columns!
Categorical data is non numeric data which belongs to specific set of categories. Like the Country column in dataset
By default read.csv() function in R makes all the string variables as categorical variables(factor) but suppose there is a name column in the dataset in that case we dont need them as categorical variables. Below is the code to make specific variables as factor variables.
# Encoding categorical data
dataset$Country = factor(dataset$Country,
levels = c('France', 'Spain', 'Germany'),
labels = c(1, 2, 3))
dataset$Purchased = factor(dataset$Purchased,
levels = c('No', 'Yes'),
labels = c(0, 1))So we need to split the dataset into training and test, using below code we can do so,
For this we need to install catools,
#install.packages("catTools") #if not present
library(caTools) #adding caTools to the library
set.seed(123) # this is to ensure same output as split is done randomly, you can exclude in real time
split = sample.split(dataset$Purchased,SplitRatio = 0.8)
training_set = subset(dataset,split == TRUE)
test_set = subset(dataset, split == FALSE)SplitRatio is the ratio in which training and test set, its usually set an 80:20 for training and test respectively.
sample.split() methid takes the column and calculates a numeric array with true and false in random locations and with the given split ratio.
subset() method takes the dataset and subset according to the condition
Feature scaling is needed when different features has different ranges, for example Age and Salary Column.
They have very different ranges but when we training a model, which is basically trying to fit some line(in linear regression) then the error is trying to be minimized,
to minimize the error the euclidian distance is minimized using some algorithm(gradient descent )
But if no feature scaling is applied then the training will be highly biased with the feature having large values because the euclidian distance will be large there.
Hence, We need feature scaling which is done in below steps :
#feature scaling
training_set[,2:3] = scale(training_set[,2:3])
test_set[,2:3] = scale(test_set[,2:3])2:3 is for both Age and Salary Now the dataset(training and test both) looks like :
training_set## Country Age Salary Purchased
## 1 1 0.90101716 0.9392746 0
## 2 2 -1.58847494 -1.3371160 1
## 3 3 -1.14915281 -0.7680183 0
## 4 2 0.02237289 -0.1040711 0
## 5 3 0.31525431 0.1594000 1
## 7 2 0.13627122 -0.9577176 0
## 8 1 1.48678000 1.6032218 1
## 10 1 -0.12406783 0.4650265 1test_set## Country Age Salary Purchased
## 6 1 -0.7071068 -0.7071068 1
## 9 3 0.7071068 0.7071068 0Note : Most libraries in R internally take care this feature scaling problem(overfitting) so we might not need to include this always.
Now the Data Preprocessing part is done..!!!