MATH2349 Semester 2, 2018

Assignment 3

Required packages

library(readr)
library(dplyr)
library(ggplot2)
library(outliers)
library(knitr)
library(tidyr)
library(Hmisc)
library(lubridate)

Executive Summary

The main objective of this assignment is to utilize the knowledge gained on preprocessing the data throughout this course.To achieve this, diverse preprocessing techniques has been used to pre-process the data.

Initially, the data sets were collected from the internet(open source).In this process two data sets were opted as choice which deal about the “pay stay parking restrictions” and “sign plates located in each pay stay zone”.

Secondly, both the data sets has been imported into the R Studio from the working directory. For better understanding, they were filtered and scraped accordingly. The two data sets were joined respectively by “pay_stay_zone” variable.

The joined data set was assigned to an object called ‘Final_data’ and this object is used for the further operations. On checking the structure of the data, it is clear that the data set is perfectly tidy.

Later, the structure of the variables has been scrutinized for understanding their data types and had performed required conversions which were needed to fit the requirements.The variables that require factor was made into factor and was leveled.

All the errors and missing values were identified and were replaced by the mean value in the case of numeric variables and were replaced by mode value in the case of categorical variables.

Finally, boxplots has been created for the numeric variables to identify the outliers. On performing this, it was identified that there were outliers for the ‘maximum_stay’. These were handled using ‘CAPPING’ approach.

Last but not the least, logarithm transformation is done to change the distribution of ‘maximum_stay’ from right_skewed to normal successfully.

Data

DATA SET-1

SIGN PLATES LOCATED IN EACH PAY STAY ZONE

1. DESCRIPTION: This data set contains data about the sign plates located in each pay stay zone. A sign plate is where the parking restrictions are displayed. There can be multiple restrictions per Pay Stay zone.

2. The data set was fetched from an open source.The refrence is as follows:

   SOURCE: https://data.melbourne.vic.gov.au/Transport-Movement/Sign-plates-located-in-each-Pay-Stay-zone/wwkr-v8s7

3. VARIABLE DESCRIPTION: The variables of this data set can be explained as follows:

   1. pay_stay_zone- A collection of one or more parking bays in which the same restrictions apply.
   2. restriction_and_payment_display- The parking conditions as displayed on the parking sign.
   3. sign_exemptions- Any exemptions displayed on the sign.
   4. days_of_restriction- The days the restrictions applies.
   5. time_restrictions_start- Time the restrictions start.
   6. time_restrictions_finish- Time the restrictions finish.

DATA SET-2

PAY STAY PARKING RESTRICTIONS

1. DESCRIPTION: This dataset contains detailed daily information on parking restrictions in each Pay Stay zone. There can be multiple restrictions per Pay Stay zone per day.

2. The data set was fetched from an open source.The refrence is as follows:

   SOURCE: https://data.melbourne.vic.gov.au/Transport-Movement/Pay-Stay-parking-restrictions/ambt-72qg

3. VARIABLE DESCRIPTION: The variables of this data set can be explained as follows:

   1. pay_stay_zone- A collection of one or more parking bays in which the same restrictions apply.
   2. day_of_week- Day of the week the restriction applies. 1 - Sunday,2- Monday,3- Tuesday,4- Wednesday,5- Thursday,6- Friday 7- Saturday.
   3. start_time- What time the restriction starts.
   4. end_time- What time the restriction ends.
   5. minimum_stay- The minimum stay in minutes.
   6. maximum_stay- The maximum stay in minutes.
   7. cost_per_hour- The cost per hour in cents.

STEPS PERFORMED

1. In this section, both data sets were imported into the R Studio using a readr function called ‘read_csv()’ and their head part is displayed using ‘head()’ function.

2. On successfully fetching the data into the R Studio the two data sets were joined together using a function called ‘inner_join()’. The data sets were joined by the ‘pay_stay_zone’ variable.

3. Now, the newly obtained combined data frame is assigned a new name called ‘Final_data()’ on which the later operations in the assignment were worked. Then the dimensions of the new data frame were observed using ‘dim()’ function which tend to have 12 variables in it.

4. Later, the structure of the new data frame has been observed. It has given a tremondous result showing all the variables were systematically arranged.

Sign_plates_located_in_each_Pay_Stay_zone <- read_csv("Sign_plates_located_in_each_Pay_Stay_zone.csv")

Parsed with column specification:
cols(
  pay_stay_zone = col_integer(),
  restriction_and_payment_display = col_character(),
  sign_exemptions = col_character(),
  days_of_restriction = col_character(),
  time_restrictions_start = col_time(format = ""),
  time_restrictions_finish = col_time(format = "")
)

head(Sign_plates_located_in_each_Pay_Stay_zone)

Pay_Stay_parking_restrictions <- read_csv("Pay_Stay_parking_restrictions.csv")

Parsed with column specification:
cols(
  pay_stay_zone = col_integer(),
  day_of_week = col_integer(),
  start_time = col_time(format = ""),
  end_time = col_time(format = ""),
  minimum_stay = col_integer(),
  maximum_stay = col_integer(),
  cost_per_hour = col_integer()
)

head(Pay_Stay_parking_restrictions)

Final_data <- inner_join(Sign_plates_located_in_each_Pay_Stay_zone,Pay_Stay_parking_restrictions)

Joining, by = "pay_stay_zone"

str(Final_data)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   11300 obs. of  12 variables:
 $ pay_stay_zone                  : int  30001095 30001095 30001095 30001095 30001095 30001095 30001082 30001082 30001082 30001082 ...
 $ restriction_and_payment_display: chr  "1P Ticket" "1P Ticket" "1P Ticket" "1P Ticket" ...
 $ sign_exemptions                : chr  NA NA NA NA ...
 $ days_of_restriction            : chr  "MF" "MF" "MF" "MF" ...
 $ time_restrictions_start        : 'hms' num  07:30:00 07:30:00 07:30:00 07:30:00 ...
  ..- attr(*, "units")= chr "secs"
 $ time_restrictions_finish       : 'hms' num  18:30:00 18:30:00 18:30:00 18:30:00 ...
  ..- attr(*, "units")= chr "secs"
 $ day_of_week                    : int  2 3 4 5 6 7 2 3 4 5 ...
 $ start_time                     : 'hms' num  07:30:00 07:30:00 07:30:00 07:30:00 ...
  ..- attr(*, "units")= chr "secs"
 $ end_time                       : 'hms' num  18:30:00 18:30:00 18:30:00 18:30:00 ...
  ..- attr(*, "units")= chr "secs"
 $ minimum_stay                   : int  0 0 0 0 0 0 0 0 0 0 ...
 $ maximum_stay                   : int  60 60 60 60 60 60 660 660 660 660 ...
 $ cost_per_hour                  : int  320 320 320 320 320 320 80 80 80 80 ...

head(Final_data)

dim(Final_data)

[1] 11300    12

Understand

STEPS PERFORMED

1. In this section, all the types of the varaibles were investigated using ‘typeof()’ function. In prior to that ‘days_of_restriction’ variable is fcatorized and was assigned levels to it. It depicts that the order priority is MF<SA<SS.

2. ‘hms()’ function was applied to the time variables which shows the time in a format of hours-minutes-seconds.This was done for the easy understanding of the future operations that had been performed in the next sections like ‘mutate()’.

3. Then the class of the start_time and end_time was observed as ‘Period’.

4. The ‘class()’ function was used to verify the class of the factored and ordered variable “days_of_restriction” and it was observed to be ordered, factored successfully.

5. Finally, the data types of the variables were summarised using ‘typeof()’ function for all the varibales.

Final_data$days_of_restriction <- factor(Final_data$days_of_restriction, levels = c("MF", "SA","SS"), labels = c("Mon-Fri", "Sat","Sun"),ordered = TRUE)
Final_data$start_time <- hms(Final_data$start_time)
Final_data$end_time <- hms(Final_data$end_time)
class(Final_data$start_time)

[1] "Period"
attr(,"package")
[1] "lubridate"

class(Final_data$end_time)

[1] "Period"
attr(,"package")
[1] "lubridate"

head(Final_data$days_of_restriction)

[1] Mon-Fri Mon-Fri Mon-Fri Mon-Fri Mon-Fri Mon-Fri
Levels: Mon-Fri < Sat < Sun

class(Final_data$days_of_restriction)

[1] "ordered" "factor" 

typeof(Final_data$pay_stay_zone)

[1] "integer"

typeof(Final_data$sign_exemptions)

[1] "character"

typeof(Final_data$restriction_and_payment_display)

[1] "character"

typeof(Final_data$time_restrictions_start)

[1] "double"

typeof(Final_data$time_restrictions_finish)

[1] "double"

typeof(Final_data$day_of_week)

[1] "integer"

typeof(Final_data$minimum_stay)

[1] "integer"

typeof(Final_data$maximum_stay)

[1] "integer"

typeof(Final_data$cost_per_hour)

[1] "integer"

Tidy & Manipulate Data I

Our data set was already in tidy format. So, it doesn’t need any changes or alterations. The ‘Final_data’ is in tidy format because of the following reasons:

1)Each variable have its own column.

2)Each observation have its own row.

3)Each value have its own cell.

Tidy & Manipulate Data II

STEPS PERFORMED

In this section, a new variable called “Total_number_of_hours_restricted” has been created using ‘mutate()’ function. This gives the total number of restricted hours for vehicles in a day. This was obtained by subtracting the restriction start_time from the end_time of the parking space. As mentioned in the previous step, the hms() function gives the exact time format of the total restriction hours in a day. By using the ‘head()’ function it is clearly visible that the new variable(“Total_number_of_hours_restricted”) is added into the data set.

Final_data<- mutate(Final_data,Total_number_of_hours_restricted=Final_data$end_time-Final_data$start_time)
head(Final_data)

Scan I

STEPS PERFORMED

1. In this section, ‘sum(is.na())’ function has been used to find the count of NA’s in the data frame(‘Final_data’).

2. Then, ‘colSums(is.na())’ has been used to find the missing values in each column of the data frame seperately. It was observed that few variables contain missing values.

3. All the variables of the data frame were checked whether ‘not a number’ or not by using ‘sum(is.nan())’.It has returned ‘0’ which means not a number (FALSE).

4. ‘sign_exemptions’ was removed using ‘Final_data[-3]’ beacause the entire column has no values. So, imputing that particular column won’t be that efficient.

5. ‘Impute()’ function has been used handle The variables having missing values. This has been achieved by replacing the missing values by their mean and mode values.

6. Finally, again by observing the result of ‘sum(is.na())’, it has been proved that all the missing values were handled successfully.

sum(is.na(Final_data))

[1] 15346

# Checking NA's
colSums(is.na(Final_data))

                   pay_stay_zone  restriction_and_payment_display                  sign_exemptions 
                               0                                0                            11300 
             days_of_restriction          time_restrictions_start         time_restrictions_finish 
                            4031                                0                                0 
                     day_of_week                       start_time                         end_time 
                               0                                0                                0 
                    minimum_stay                     maximum_stay                    cost_per_hour 
                              15                                0                                0 
Total_number_of_hours_restricted 
                               0 

sum(is.nan(Final_data$pay_stay_zone))

[1] 0

sum(is.nan(Final_data$days_of_restriction))

[1] 0

sum(is.nan(Final_data$time_restrictions_start))

[1] 0

sum(is.nan(Final_data$time_restrictions_finish))

[1] 0

sum(is.nan(Final_data$day_of_week))

[1] 0

sum(is.nan(Final_data$start_time))

[1] 0

sum(is.nan(Final_data$end_time))

[1] 0

sum(is.nan(Final_data$minimum_stay))

[1] 0

sum(is.nan(Final_data$maximum_stay))

[1] 0

sum(is.nan(Final_data$cost_per_hour))

[1] 0

sum(is.nan(Final_data$Total_number_of_hours_restricted))

[1] 0

sum(is.nan(Final_data$restriction_and_payment_display))

[1] 0

Final_data<-Final_data[-3]
# Replacing NA's
Final_data$days_of_restriction <- impute(Final_data$days_of_restriction,fun = mode)
Final_data$minimum_stay <- impute(Final_data$minimum_stay,fun = mean)
colSums(is.na(Final_data))

                   pay_stay_zone  restriction_and_payment_display              days_of_restriction 
                               0                                0                                0 
         time_restrictions_start         time_restrictions_finish                      day_of_week 
                               0                                0                                0 
                      start_time                         end_time                     minimum_stay 
                               0                                0                                0 
                    maximum_stay                    cost_per_hour Total_number_of_hours_restricted 
                               0                                0                                0 

Scan II

STEPS PERFORMED

1. The main aim of this section is to scrutinize the numeric data for outliers. For this purpose, all the numeric data were cosidered.

2. By using ‘boxplot()’ function boxplots were created for the all the numeric variables. As our data set contains 5 numeric variables, 5 box plots were created.

3. It has been noted from the below box plot that, ‘maximum_stay’ do have outliers as it has data points falling outside the bound.

4. Later, the outliers of this variable were handled using ‘capping’ method. This method helped in replacing the outliers with the nearest neighbours that are not outliers. A user defined function() was used to cap the outliers.

5. Finally, it has been justified that outliers were handled from the boxplot(After handling outliers).’par(mfrow())’function has been used to create a side by side box plot for easy visualization.

# Boxplot for showing outliers
Final_data$pay_stay_zone<-as.numeric(Final_data$pay_stay_zone)
boxplot(Final_data$pay_stay_zone, main="boxplot of pay_stay_zone", col = "burlywood")

Final_data$day_of_week<-as.numeric(Final_data$day_of_week)
boxplot(Final_data$day_of_week, main="boxplot of day_of_week", col = "burlywood")

Final_data$cost_per_hour<-as.numeric(Final_data$cost_per_hour)
boxplot(Final_data$cost_per_hour, main="boxplot of cost_per_hour", col = "burlywood")

Final_data$minimum_stay<-as.numeric(Final_data$minimum_stay )
boxplot(Final_data$minimum_stay, main="boxplot of minimum_stay", col = "burlywood")

Final_data$maximum_stay<-as.numeric(Final_data$maximum_stay)
boxplot(Final_data$maximum_stay, main="boxplot of maximum stay", col = "burlywood")

# Handing outliers
cap <- function(x){
    quantiles <- quantile( x, c(.05, 0.25, 0.75 ) )
    x[ x < quantiles[2] - 1.5*IQR(x) ] <- quantiles[1]
    x[ x > quantiles[3] + 1.5*IQR(x) ] <- quantiles[4]
    x
}
par(mfrow=c(1,2))
boxplot(Final_data$maximum_stay,main="Before handling outliers", col = "burlywood")
max_capped <- Final_data$maximum_stay %>% cap()
boxplot(max_capped, main="After handling outliers", col = "burlywood")

Transform

STEPS PERFORMED

In this section, two histograms has been created using ‘hist()’ function. Before transformation, the histogram tend to be right skewed. So, ‘log10()’ function has been used to gain the normally distributed histogram. On using this, a normal distributed histogram has been created which was titled as ‘hist after transformation’.

par(mfrow=c(1,2))
hist(Final_data$maximum_stay,main = "Hist Before Transformation", col = "burlywood")
Max_stay_log <- log10(Final_data$maximum_stay)
hist(Max_stay_log,main = "Hist After Transformation", col = "burlywood")