MATH2349 Semester 1, 2020
Assignment 2
Chinmay Pradeep Karangutkar - S3827733
Required packages
library(readr)## Warning: package 'readr' was built under R version 3.6.3library(dplyr)## Warning: package 'dplyr' was built under R version 3.6.3##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionlibrary(tidyr)## Warning: package 'tidyr' was built under R version 3.6.3library(Hmisc)## Warning: package 'Hmisc' was built under R version 3.6.3## Loading required package: lattice## Loading required package: survival## Loading required package: Formula## Loading required package: ggplot2## Warning: package 'ggplot2' was built under R version 3.6.3##
## Attaching package: 'Hmisc'## The following objects are masked from 'package:dplyr':
##
## src, summarize## The following objects are masked from 'package:base':
##
## format.pval, unitslibrary(ggplot2)
library(outliers)
library(forecast)## Warning: package 'forecast' was built under R version 3.6.3## Registered S3 method overwritten by 'quantmod':
## method from
## as.zoo.data.frame zoolibrary(knitr)## Warning: package 'knitr' was built under R version 3.6.3library(stringr)## Warning: package 'stringr' was built under R version 3.6.3Executive Summary
- Thus we imported two data sets (world population and flight traffic) with the help of readr package using read_csv() function and saved those data sets into two new variables.
- Therefore in order to create a common key to merge both we have to create common data for specific variables in both dataset.
- Here we will work on country and year where the names of country where upper case at one place and lower case at another, so we will convert country names with lowercase to uppercase by using toupper() function.Therefore with this we can perform join() operations on both data sets.
- Thus we will perform left join on flight traffic data using population data.
- There were some spelling errors in column names , so by using dplyr’s rename() function we will solve those spelling errors.
Data Information
We choosed two data sets (world population and flight traffic). World population data is exact description of population count of different countries for the year of 1960 - 2016 , which consist variables such as :- Country name, country code ,indicator name and code and population count for each year from (1960-2016).
The second data set is based on flight traffic in India .which consists variables such as :- Country , Quarter,Year, Passengers comming to India, Passengers comming from India, Flights departing from India and flights arriving in India.
Therefore by performing leftjoin() we got a new dataframe named flight where both data got combined on basis of common country name and year, on which we can carry out our further operations.
Data Source :- Kaggle
Link:-
(World population) www.kaggle.com/chadalee/country-wise-population-data
(Flight Traffic) www.kaggle.com/rajanand/international-air-traffic-from-and-to-india?select=Countrywise+Quarterly+International+Air+Traffic+To+And+From+The+Indian+Territory.csv
Tidy & Manipulate Data I :-
- Thus after having a look over data we can say that the world population data set was untidy, the reason for its untidyness was its variables describing the population count for the years (1960 - 2016).So in this case every year resulted into a new column which made our data wider and it resulted into dirty data.
- Therefore in order to make the data tidy we will use the package tidyverse , thus by using tidyr we can implement various functions such as gather() & spread().
- Hence here we will use gather() function which will reduce the wider data. In simple words collects a set of column names and places them into a single “key” column,it also collects the cells of those columns and places them into a single value column.
- Thus a two columns will replace all year columns where one column will only display year and second column will display population count.After making it tidy we joined the data sets of country flight traffic and world population.Even we changed the names of some specific columns by using rename() functions, example :- FREIGHT was replaced by FLIGHT.
world_pop <- read_csv("world_pop.csv")## Parsed with column specification:
## cols(
## .default = col_double(),
## Country = col_character(),
## `Country Code` = col_character(),
## `Indicator Name` = col_character(),
## `Indicator Code` = col_character()
## )## See spec(...) for full column specifications.head(world_pop,10) country<- read_csv("Countrywise Quarterly International Air Traffic To And From The Indian Territory.csv")## Parsed with column specification:
## cols(
## YEAR = col_double(),
## QUARTER = col_character(),
## `COUNTRY NAME` = col_character(),
## `PASSENGERS TO INDIA` = col_double(),
## `PASSENGERS FROM INDIA` = col_double(),
## `FREIGHT TO INDIA` = col_double(),
## `FREIGHT FROM INDIA` = col_double()
## )head(country,10)World<-world_pop %>%gather(year,Population,5:61) #data was untidy hence gathering process done.
head(World,10)class(country$YEAR)## [1] "numeric"country$YEAR<-as.factor(country$YEAR)
World$Country<-toupper(World$Country) #changing country names to uppercase for join function.
flight<-left_join(country,World,by=c("COUNTRY NAME"="Country","YEAR"="year"))
flight<-flight %>% rename(`FLIGHT FROM INDIA`=`FREIGHT FROM INDIA`,`FLIGHT TO INDIA`=`FREIGHT TO INDIA`) # spelling correctiong using dplyr.
head(flight,10)Understand
- The most important phase of data science is understanding data, which includes having an actual idea regarding the data.Thus with the help of dplyr paclage we can perform some operations which will give us some information regarding data.
- Here we performed dim()- which shows the dimesions of data,summary() which gives exact summary of data which includes mean , median, NA ,quartiles of data and data types.Thus we can also check individual data types using typeof() function.
- Thus after getting idea of types we can change the data types where required. Therefore we converted some datatypes into factor as that data should be in categeoric format but were in integer or character.So using as.factor() we converted all required data types to factor.
- Therefore as specified in requirments we will perform leveling and labeling on one of the factor ie(categoric data).Thus by using levels() and labels we performend labeling and leveling on QUARTER, example :- Q1 was labelled as Q-1.
dim(flight) #dimension of data frame## [1] 499 11summary(flight)## YEAR QUARTER COUNTRY NAME PASSENGERS TO INDIA
## Length:499 Length:499 Length:499 Min. : 0
## Class :character Class :character Class :character 1st Qu.: 5921
## Mode :character Mode :character Mode :character Median : 21144
## Mean : 114450
## 3rd Qu.: 108567
## Max. :2384541
##
## PASSENGERS FROM INDIA FLIGHT TO INDIA FLIGHT FROM INDIA Country Code
## Min. : 0 Min. : 0.0 Min. : 0.00 Length:499
## 1st Qu.: 6913 1st Qu.: 4.0 1st Qu.: 51.15 Class :character
## Median : 21846 Median : 255.7 Median : 588.40 Mode :character
## Mean : 119050 Mean : 2576.1 Mean : 4041.46
## 3rd Qu.: 112085 3rd Qu.: 2150.1 3rd Qu.: 3076.30
## Max. :2480816 Max. :35914.1 Max. :108131.40
##
## Indicator Name Indicator Code Population
## Length:499 Length:499 Min. :9.342e+04
## Class :character Class :character 1st Qu.:5.607e+06
## Mode :character Mode :character Median :2.898e+07
## Mean :7.166e+07
## 3rd Qu.:6.886e+07
## Max. :1.379e+09
## NA's :138typeof(flight$QUARTER) #checking type of data.## [1] "character"typeof(flight$`Country Code`)## [1] "character"typeof(flight$`Indicator Code`)## [1] "character"typeof(flight$YEAR)## [1] "character"typeof(flight$Population)## [1] "double"typeof(flight$`PASSENGERS FROM INDIA`)## [1] "double"typeof(flight$`FLIGHT FROM INDIA`)## [1] "double"typeof(flight$`Indicator Name`)## [1] "character"flight$YEAR<-as.factor(flight$YEAR)
flight$QUARTER<-as.factor(flight$QUARTER) # changing data types where required.
flight$`Country Code`<-as.factor(flight$`Country Code`)
flight$`Indicator Name`<-as.factor(flight$`Indicator Name`)
flight$`Indicator Code`<-as.factor(flight$`Indicator Code`)
levels(flight$QUARTER) # labelling or ordering one of the factor variable.## [1] "Q1" "Q2" "Q3" "Q4"flight$QUARTER<- factor(flight$QUARTER,levels = c("Q1", "Q2", "Q3", "Q4"),labels=c("Q-1","Q-2","Q-3","Q-4"))Tidy & Manipulate Data II
- Tidy & Manipulate Data part 1 was done before in initial phases just after importing data because in initial phases there was need to clean the data before combining with another data set.Therefore we will work on creating another variable which will be created using data.
- In simple words we will mutate another variable using existing variables of current data.Thus we will create new column named Difference using flights from India & Flights to India.The variable Difference will be exact diffrence between flights arriving to India and Flights deporting from India.
- Thus with help of mutate() function we can perform this task. The head() function is used to view first occuring records of data, we have used head() above for many purposes.
flight<-mutate(flight,Difference =(flight$`FLIGHT FROM INDIA`-flight$`FLIGHT TO INDIA`))
head(flight,10)Scan I
- The main task in scanning process is to find misiing values . Missing values are basically values that is not stored for a variable in the observation of interest.
- There is huge need to remove or replace such missing values.Thus we will work on detecting missing values in data
- Detection can be done via is.na() method , this will return all mising values present but it may flood your console so you can use sum() which will count exact number of misisng values present in data.
- Thus you can use this duo of sum(is.na()) on every variable to check the misisng values count. Now comes the important step to treat such missing values , so using impute method we can treat the misisng values , here we will replace missing values by mean & mode.Mode is used for categroric variables and Mean is used for numeric variables.
- So after doing all imputing process we will check the status using sum(is.na()) on entire data and we get null count for missing values.
- Special values are mostly INF (“infinity”) & NAN (“not a number”) we can check those using is.infinite and is.nan and we can count them using sum() function.
sum(is.na(flight)) # finding actual sum of missing values in dataframe.## [1] 552flight$`Country Code`<- impute(flight$`Country Code`, fun= mode) # removing null
sum(is.na(flight$`Country Code`)) ## [1] 0flight$`Indicator Name`<- impute(flight$`Indicator Name`, fun= mode)
sum(is.na(flight$`Country Code`))## [1] 0flight$`Indicator Code`<-impute(flight$`Indicator Code`,fun=mode)
sum(is.na(flight$`Indicator Code`))## [1] 0flight$Population<-impute(flight$Population,fun=mean)
sum(is.na(flight$Population))## [1] 0sum(is.na(flight$Difference))## [1] 0sum(is.na(flight)) ## [1] 0flight$QUARTER<-impute(flight$QUARTER,fun=mode)
sum(is.na(flight$QUARTER))## [1] 0sum(is.na(flight))## [1] 0sum(is.infinite(flight$Difference)) # Finding special INF values## [1] 0sum(is.infinite(flight$YEAR))## [1] 0sum(is.infinite(flight$Population))## [1] 0sum(is.infinite(flight$`PASSENGERS TO INDIA`))## [1] 0sum(is.infinite(flight$`COUNTRY NAME`))## [1] 0sum(is.infinite(flight$`Indicator Code`))## [1] 0sum(is.infinite(flight$`FLIGHT TO INDIA`)) #No special values ( INF ...)## [1] 0sum(is.nan(flight$Difference)) # Finding special NAN values## [1] 0sum(is.nan(flight$YEAR))## [1] 0sum(is.nan(flight$Population))## [1] 0sum(is.nan(flight$`PASSENGERS TO INDIA`))## [1] 0sum(is.nan(flight$`COUNTRY NAME`))## [1] 0sum(is.nan(flight$`Indicator Code`))## [1] 0sum(is.nan(flight$`FLIGHT TO INDIA`)) # No special NAN values ## [1] 0Scan II
- Outlier is a point which falls more than 1.5 times the interquartile range above the third quartile or below the first quartile.So in order to check if our data contains outlier we can use the Box-plot method.
- Box-plot shows the entire shape of distribution,For Outlier Detection the variable should be in the range of boxplot.
- The Box-Plot range consist of minimum and maximum , the variable must be inside both ie (range of −1.5×IQR to 1.5×IQR)
- Thus using Box-Plot we can detect Outliers.
- Outliers can be cured using capping method or also knows as winsorising method . In capping method instead of removing outlier it replace the outliers with the nearest neighbours that are not outliers.
- Thus how we deal with outliers and we also do same process in combined way by using sapply function and later we can check the summary of data obtained after capping and compare it with previous normal summary we will observe some change,thats how many outliers are cured.
class(flight$Population)## [1] "impute"flight$Population<-as.numeric(flight$Population)
pindiato<-flight$`PASSENGERS TO INDIA` %>% boxplot(main="box pot",ylab="PASSENGERS TO INDIA",col="yellow",horizontal = TRUE)
fromindia<-flight$`FLIGHT FROM INDIA` %>% boxplot(main="box pot",ylab="FLIGHTS FROM INDIA",col="red",horizontal = TRUE)
pfromindia<-flight$`PASSENGERS FROM INDIA` %>% boxplot(main="box pot",ylab="PASSENGERS FROM INDIA",col="blue",horizontal = TRUE)
indiato<-flight$`FLIGHT TO INDIA`%>% boxplot(main="box pot",ylab="FLIGHTS TO INDIA",col="green",horizontal = TRUE)
popoutlier<-flight$Population %>% boxplot(main="box pot",ylab=" Population",col="pink",horizontal = TRUE)
# detecting outliers in population
sum(popoutlier$out)## [1] 12529875956sum(pindiato$out)## [1] 40112418sum(fromindia$out)## [1] 1503252sum(indiato$out)## [1] 981997.1#excluding outliers by capping method.
cap <- function(x){
quantiles <- quantile( x, c(.05, 0.25, 0.75, .95 ) )
x[ x < quantiles[2] - 1.5*IQR(x) ] <- quantiles[1]
x[ x > quantiles[3] + 1.5*IQR(x) ] <- quantiles[4]
x
}
Diamonds_sub <- flight %>% dplyr::select(Population,`FLIGHT TO INDIA`,`FLIGHT FROM INDIA`,`PASSENGERS TO INDIA`,`PASSENGERS FROM INDIA`)
cappopulation<- Diamonds_sub$Population %>% cap()
cappopulation %>% boxplot(main=" population (cap)",col="pink",horizontal = TRUE)
capflight1<- Diamonds_sub$`FLIGHT TO INDIA` %>% cap()
capflight2<- Diamonds_sub$`FLIGHT FROM INDIA` %>% cap()
cappass1<- Diamonds_sub$`PASSENGERS TO INDIA` %>% cap()
cappass1 %>% boxplot(main="Passengers to India(CAP)",col="yellow",horizontal = TRUE)
cappass2 <- Diamonds_sub$`PASSENGERS FROM INDIA` %>% cap()
cappass2 %>% boxplot(horizontal=TRUE,main="Passengers from India(cap)",col="blue")
datasummary<-sapply(Diamonds_sub,FUN = cap) # we can also do above step using sapply method in combined way.
summary(Diamonds_sub)## Population FLIGHT TO INDIA FLIGHT FROM INDIA PASSENGERS TO INDIA
## Min. :9.342e+04 Min. : 0.0 Min. : 0.00 Min. : 0
## 1st Qu.:8.747e+06 1st Qu.: 4.0 1st Qu.: 51.15 1st Qu.: 5921
## Median :6.060e+07 Median : 255.7 Median : 588.40 Median : 21144
## Mean :7.166e+07 Mean : 2576.1 Mean : 4041.46 Mean : 114450
## 3rd Qu.:7.166e+07 3rd Qu.: 2150.1 3rd Qu.: 3076.30 3rd Qu.: 108567
## Max. :1.379e+09 Max. :35914.1 Max. :108131.40 Max. :2384541
## PASSENGERS FROM INDIA
## Min. : 0
## 1st Qu.: 6913
## Median : 21846
## Mean : 119050
## 3rd Qu.: 112085
## Max. :2480816summary(datasummary) # compare with above summary.## Population FLIGHT TO INDIA FLIGHT FROM INDIA PASSENGERS TO INDIA
## Min. : 93419 Min. : 0.0 Min. : 0.00 Min. : 0
## 1st Qu.: 8747358 1st Qu.: 4.0 1st Qu.: 51.15 1st Qu.: 5921
## Median : 60600590 Median : 255.7 Median : 588.40 Median : 21144
## Mean : 51178561 Mean : 2398.5 Mean : 3161.12 Mean : 86828
## 3rd Qu.: 71660319 3rd Qu.: 2150.1 3rd Qu.: 3076.30 3rd Qu.:108567
## Max. :162951560 Max. :13744.2 Max. :16624.41 Max. :381580
## PASSENGERS FROM INDIA
## Min. : 0
## 1st Qu.: 6913
## Median : 21846
## Mean : 90527
## 3rd Qu.:112085
## Max. :401412Transform
- Data Transformation is an important step for machine learning prospectus. As transformation is required to improve normality of variance. In simple words its used to reduce skewness of variances.
- Thus we will first use Box-Cox Transformation , Boxcox is basically a type of power transformation which is used to transform non-normal data into normalized form.Thus we will compare histograms of normal population and boxcox data of population.
- We can also work for reducing skewness by using logs and powers , For reducing right skewness we can use logs and roots , for reducing left skewness we can use squares and cubes . Later we can compare the skewness of normal population and log or squared population.The selection of log or square method depends upon which gives better result.
- Normalizing is done because some values could ranges from 1 to 100000 and some may range 1-10 and the impact on response variables by the variables with high range will be more as compared to others , this could affect on the prediction accuracy.
- We can perform Normalization in two ways using mean ie(center = TRUE) and by using standard deviation . Thus we can compare both histograms and select wisely.
#box-cox for reducing skewness of population
class(flight$Population)## [1] "numeric"flight$Population %>% hist(main="Population")
boxycoxy<-BoxCox(flight$Population,lambda = "auto")
boxycoxy %>% hist(main="BOX-COX")
#for reducing right skewness
logpop<-log(flight$Population)
hist(logpop)
#for left skewness
square<- function(x) {x*x}
popsquare<-square(flight$Population)
hist(popsquare)
#normalizing
#without centering
popnocenter<-scale(flight$Population,center = FALSE,scale = sd(flight$Population))
hist(popnocenter)
#with centering
popcenter<-scale(flight$Population,center = TRUE,scale = FALSE)
hist(popcenter)
References:
- RMIT Canvas course modules and Data Camp.
- Data Source :- Kaggle
- Link:- (World population) www.kaggle.com/chadalee/country-wise-population-data (Flight Traffic) www.kaggle.com/rajanand/international-air-traffic-from-and-to-india?select=Countrywise+Quarterly+International+Air+Traffic+To+And+From+The+Indian+Territory.csv