MATH2349 Semester 2, 2018

Assignment 3

Required packages

library(readr)
library(dplyr)
library(tidyr)
library(Hmisc)
library(stringr)
library(lubridate)
library(outliers)
library(MVN)
library(ggplot2)
library(knitr)
library(gdata)
library(mlr)
library(forecast)

Executive Summary

Data preprocessing plays a crucial role in statistical analysis as it can significantly influence the statistical conclusion based on data. It can broadly be described as the cleaning or preparing all forms of untidy data(incomplete, noisy and inconsistent data) for statistical analysis. We have performed all the 5 major tasks of preprocessing namely- get, understand, tidy& manipulate,scan and transform.

First we imported the two datasets using readr function, these two datasets were sorted and subset was created before merging by ‘inner_join’ function. This merged dataset is understood in terms of its class and structure and, wherever needed,the class of variables were converted into date, factor etc.

After understanding the data, the data is checked for its tidyness using Hadley Wickham and Grolemund (2016) tidy data rules and the PlayerID which is alpha-numeric is separated into two variables Name and ID. The data is also manipulated using ‘dyplyr’ package wherein the career in terms of years of a player is found out using mutate() function. Later, for scanning the missing values and outliers the data is again subset with two numerical and one character variables. Both the numerical variables were scanned for the missing values using ‘is.na’ function and the missing values were imputed with the mean of the respective variables. Both the numerical variables are also scanned for outliers by creating using Tukey’s fencing method and number of outliers were confirmed by distance method(Z score) and the outliers detected were removed by capping function.

In the end, the values of one of the numerical variable were transformed by mathematical operations to decrease the skewness and convert it to a normal distribution.This was attained by raising the data to higher power(power transformation) and by taking the root(square root transformation).

Data

The data chosen for this assignment is about National Football League (NFL), a professional American football league.The NFL is one of the four major professional sports leagues in North America, and the highest professional level of American football in the world. The data is sourced from Kaggle.com, an open data source. The selected datasets can be found at the URL: https://www.kaggle.com/kendallgillies/nflstatistics/home.

There are two sets of data- main dataset and allied dataset. The first main group of statistics is ‘basic.csv’ where the basic statistics are provided for each player. The data has the following variables- Age Birth Place Birthday College Attended Current Satus Current Team Experience(number of seasons played) Height(inches) High School Attended High School Location Name of the player Number PlayerId Position Weight(lbs) Years played

The second main group of statistics gathered for each player are their career statistics. Each player has a main role but they have statistics in other areas as well. we have taken the career staistics of the players on the offensive line, stored in ‘offensive.csv’ and has following variables: PlayerId Name Position Year Team GamesPlayed GamesStarted

The Working directory is loaded and data is imported with readr “read_csv” function. The csv files can also be imported into R using base R “read.csv” function, however, read.csv function automatically converts strings/character(class/data type) into factor which is taken care of by changing “stringsAsFactors” clause.

setwd("C:/Users/Neeraj/Desktop/data preprocessing/assignment 3")
basic <- read_csv("basic.csv")

Parsed with column specification:
cols(
  Age = col_integer(),
  `Birth Place` = col_character(),
  Birthday = col_character(),
  College = col_character(),
  `Current Status` = col_character(),
  `Current Team` = col_character(),
  Experience = col_character(),
  `Height (inches)` = col_integer(),
  `High School` = col_character(),
  `High School Location` = col_character(),
  Name = col_character(),
  Number = col_integer(),
  `Player Id` = col_character(),
  Position = col_character(),
  `Weight (lbs)` = col_integer(),
  `Years Played` = col_character()
)

offensive <- read_csv("offensive.csv")

Parsed with column specification:
cols(
  `Player Id` = col_character(),
  Name = col_character(),
  Position = col_character(),
  Year = col_integer(),
  Team = col_character(),
  `Games Played` = col_integer(),
  `Games Started` = col_character()
)

The first six rows of the main dataset is displayed using head() function.

head(basic)

The first six rows of the allied dataset is displayed using head() function.

head(offensive)

In the below chunk, the main data is sorted for clarity and easy computation. The original data is subset and seven useful variables are taken from a total of 16 variables. these 7 variables are: PlayerID,Birthday, college, current status,Experience, Height, Weight.

basic_new <- basic[ ,c(13,3,4,5,7,8,15)]
basic_new

The columns of the allied dataset are also sorted and renamed to enhance the coherence of the dataset. Here, the 7 variables from the original dataset are summarised in to minimum and maximum year with the total number of games played and grouped on their playerID to better summarise the data. The first few observations are also displayed.

offensive_new <- offensive %>% group_by(`Player Id`) %>% summarise(start_year = min(Year, na.rm = FALSE),
          end_year = max(Year, na.rm = FALSE),
          total_games = sum(`Games Played`, na.rm = FALSE))
head(offensive_new)

Both the subsets of main dataset(“basic_new”) and the allied dataset(“offensive_new”) is merged together as “NFLdata”" using the inner_join() function on the key variable(“Player Id”). This merging condensed/shrinked the data into 2885 observations of 10 variables, the first six observations of which are displayed here using the head() function.

NFLdata <- basic_new %>% inner_join(offensive_new, by = "Player Id")
head(NFLdata)

Understand

The class() function is used to check the class/type of an individual variable/object.

class(NFLdata$College)

[1] "character"

class(NFLdata$`Height (inches)`)

[1] "integer"

The class of Birthday variable is shown as “character”. This is because when date and time data are imported into R they will often default to a character string (or factors if you are using stringsAsFactors = FALSE option). Hence, we need to convert strings to proper date format.

class(NFLdata$Birthday)

[1] "character"

There are different ways to convert this variable into date format.First one is by using “as.Date()” function under Base R and the other way is by using “lubridate” package. However, under the Base R function the default date format is YYYY-MM-DD and for the different format the format argument must be specified explicitly Whereas the “lubridate” package can automatically recognise the common separators and we only need to specify the order of the date elments. Here, the class of Birthday variable using “lubridate” package and checked again for confirmation.

NFLdata$Birthday <- mdy(NFLdata$Birthday)
class(NFLdata$Birthday)

[1] "Date"

The ‘Current Status’ variable is also converted to factor from character variable and checked.

NFLdata$`Current Status`<- as.factor(NFLdata$`Current Status`)
class(NFLdata$`Current Status`)

[1] "factor"

The ‘Experience’ variable is sorted into similar types, factored, levelled and labelled and ordered as per the player experience.

NFLdata$Experience <- factor( c(NFLdata$Experience), 
levels = c("Rookie",    "0 Season", "1 Season", "1st season",   "2 Seasons",    "2nd season",   "3 Seasons",    "3rd season",   "4 Seasons",    "4th season",   "5 Seasons",    "5th season",   "6 Seasons",    "6th season",   "7 Seasons",    "7th season",   "8 Seasons",    "8th season",   "9 Seasons",    "9th season",   "10 Seasons",   "10th season",  "11 Seasons",   "11th season",  "12 Seasons",   "13 Seasons",   "14 Seasons",   "14th season",  "17 Seasons",   "18 Seasons",   "18th season"), 
labels = c("Rookie",    "0 Season", "1 Season", "1 Season", "2 Seasons",    "2 Seasons",    "3 Seasons",    "3 Seasons",    "4 Seasons",    "4 Seasons",    "5 Seasons",    "5 Seasons",    "6 Seasons",    "6 Seasons",    "7 Seasons",    "7 Seasons",    "8 Seasons",    "8 Seasons",    "9 Seasons",    "9 Seasons",    "10 Seasons",   "10 Seasons",   "11 Seasons",   "11 Seasons",   "12 Seasons",   "13 Seasons",   "14 Seasons",   "14 Seasons",   "17 Seasons",   "18 Seasons",   "18 Seasons"), ordered = TRUE)

we can also check the class and other characteristics of data using str()function which compactly displays the structure of the whole dataset.

str(NFLdata)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   2885 obs. of  10 variables:
 $ Player Id      : chr  "jimraiff/2523700" "rosspetty/2523136" "billschuler/2525128" "georgesergienko/2525335" ...
 $ Birthday       : Date, format: "1930-12-21" ...
 $ College        : chr  "Dayton" "Illinois" "Yale" "American International" ...
 $ Current Status : Factor w/ 4 levels "Active","Physically unable to perform",..: 3 3 3 3 3 3 1 1 3 1 ...
 $ Experience     : Ord.factor w/ 18 levels "Rookie"<"0 Season"<..: 3 3 4 6 10 3 10 5 6 4 ...
 $ Height (inches): int  70 73 72 73 77 78 76 75 76 77 ...
 $ Weight (lbs)   : int  235 180 215 248 325 275 305 309 265 316 ...
 $ start_year     : num  1954 1920 1947 1943 1997 ...
 $ end_year       : num  1954 1920 1948 1946 2004 ...
 $ total_games    : int  3 10 23 30 54 6 112 17 35 14 ...

Tidy & Manipulate Data I

As per Hadley Wickham and Grolemund (2016), the three tidy data rules are: (a)Each variable must have its own column.(b) Each observation must have its own row.(c)Each value must have its own cell.On the basis of these tidy rules, the dataset in general can be considered as tidy data. However, in the original dataset, it is found that both the first name and last name of players are given in one column separated by a coma. As we are not sure about which one is first and last name of the players, the name is not separated in order to avoid confusion. In the below chunk, the PlayerId is separated into Names and ID variables using the ‘tidyr’ package separate() function.

NFLdata1 <- NFLdata %>% separate(`Player Id`, into = c("Name", "ID"), sep = "/")
head(NFLdata1)

Tidy & Manipulate Data II

To calculate the time frame of a player’s career, the mutate() function is used to create a new column,‘career_year’, by using the data of start_year and end_year. It seems that few players have entered and exited in their playing career in the same year however taking only the difference of their start year and end year will give zero year which is not correct logically. In order to solve this practical conundrum, one is added implicitly so that both the start year and end year are included to correctly represent the career_years.

mutate(NFLdata1,
       career_years = end_year - start_year + 1)

Scan I

In the below chunk, the numerical variables of the dataset are further subset into ‘NFL_subset’ with three variables namely- ID, Height(inches) and Weight(lbs) to scan the missing values/incostistencies and outliers and for data transformation.

NFL_subset<- NFLdata1[ ,c(2,7,8)]
NFL_subset

The colSums() function is applied to compute the total missing values in each column of the dataset.There are 56 and 18 missing values(NA’s) in Height and weight respectively.

colSums(is.na(NFL_subset))

             ID Height (inches)    Weight (lbs) 
              0              56              18 

The which() function is used to identify the location of NA’s in the Height variable.

which(is.na(NFL_subset$`Height (inches)`))

 [1]   13   86   93  109  144  225  254  273  319  351  368  395  416
[14]  498  551  606  633  654  676  700  740  741  786  873  956 1042
[27] 1123 1129 1321 1507 1555 1658 1693 1724 1804 1874 1916 1996 2043
[40] 2061 2194 2342 2358 2423 2453 2469 2541 2557 2610 2745 2756 2778
[53] 2797 2813 2815 2882

There are various ways to handle the missing values in the dataset like excluding the missing values or replace the missing values with some constant or with mean,median or mode. Although the number of NA’s is very small(almost 2%) and can be excluded. However, the exclusion/deletion of NA’s sometimes leads to biased subset of data. Hence, the missing values are imputed, here, with the mean of the Height variable using base R functions. Alternatively, Hmisc package can also be used for imputation of mean, median or mode.

NFL_subset$`Height (inches)`[is.na(NFL_subset$`Height (inches)`)] <- mean(NFL_subset$`Height (inches)`, na.rm = TRUE)

The Height variable is checked again for NA’s to confirm the imputation of mean into the missing values.

which(is.na(NFL_subset$`Height (inches)`))

integer(0)

After the height variable, the same procedure is followed to impute the NA’s in weight variable as well and the number of NA’s are checked and confirmed after the imputation of the mean of the Weight variable.

NFL_subset$`Weight (lbs)`[is.na(NFL_subset$`Weight (lbs)`)] <- mean(NFL_subset$`Weight (lbs)`, na.rm = TRUE)
which(is.na(NFL_subset$`Weight (lbs)`))

integer(0)

Scan II

There are two main methods for detecting the univariate outliers- Tukey’s method of outlier detection and distance based method. In the below chunk, the box plot is created using the Tukey’s method and outliers are seen lying outside the limits called ‘fences’.

NFL_subset$`Height (inches)` %>%  boxplot(main="Box Plot of Height", ylab = "Inches")

As box plot is a visual representation of outliers and in order to proceed with the handling of the outliers, their number should be confirmed. The distance based methods(like Z-score method etc.) help in confirming the exact number of outliers. Hence, in the chunk below, the Z-Score is calculated for Height variable using score() function from “outlier” package. The summary() function displays the minimum(-3.4370) & maximum(2.4930) Z-score of the Height variable.

zscores_Height <- NFL_subset$`Height (inches)` %>%  scores(type = "z")
zscores_Height %>% summary()

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-3.4370 -0.6697  0.1210  0.0000  0.5164  2.4930 

An observation is regarded as an outlier if the absolute value of its z-score is greater than 3. Therefore, in the following chunk, the total number of outliers according to the Z-score is calculated using length() function which comes out to be 14.

length (which( abs(zscores_Height) >3 ))

[1] 14

There are various approaches to handle the outliers like- excluding or deleting, imputing, capping/winsorising or transforming and binning. However,before deciding the approach to handle them, the outliers should be analysed as to whether they are a result of data entry/processing error and if so, they can be deleted or imputed. Due to lack of certainty for the reasons of outliers, the capping(a.k.a winsorising) approach is used. For capping, the user-defined function mentioned below(as cap)is applied to the height variable and assigned to Height_capped.

cap <- function(x){
    quantiles <- quantile( x, c(.05, 0.25, 0.75, .95 ) )
    x[ x < quantiles[2] - 1.5*IQR(x) ] <- quantiles[1]
    x[ x > quantiles[3] + 1.5*IQR(x) ] <- quantiles[4]
    x
}
Height_capped <- NFL_subset$`Height (inches)` %>% cap()

A boxplot of Height_capped is created again to confirm there are no remaining outliers.

Height_capped %>%  boxplot(main="Box Plot of Height", ylab = "Inches")

After Height, the Weight is also checked for the outliers and is handled in the similar fashion.(i.e - creation of boxplot, checking exact number of outliers via Z-score method and then handling the outliers via capping.) In the end, the boxplot is created again to check for the remaining outliers.

NFL_subset$`Weight (lbs)` %>%  boxplot(main="Box Plot of Weight", ylab = "Lbs")

zscores_Weight <- NFL_subset$`Weight (lbs)` %>%  scores(type = "z")
zscores_Weight %>% summary()

    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
-5.52961 -0.81285 -0.05903  0.00000  0.91017  2.52550 

length (which( abs(zscores_Weight) >3 ))

[1] 10

Weight_capped <- NFL_subset$`Weight (lbs)` %>% cap()

Weight_capped %>%  boxplot(main="Box Plot of Weight", ylab = "Lbs")

Transform

It is necessary to perform some data transformations on the tidy data before it can be used for modeling. Data transformation can be used in a variety of situations like: (1) To change the scale of a variable or standardise the values of a variable for better understanding. (2)To transform complex non-linear relationships into linear relationships (3) To reduce skewness and/or heterogeneity of variance and make it a normal distribution.

Here, the distribution of Height variable is visualised with the help of a histogram. The resulting histogram appears to be slightly left skewed.

hist(Height_capped)

As the histogram of height distribution is slightly skewed to the left, the data is transformed by raising the power to 5 and further by taking the fourth root. After performing the mathematical operations, a histogram is again plotted to check the transformed distribution. The transformation not always results in a perfect symmetrical shape. The distribution got better than before,hence, we can proceed with the resulting transformation.

height_transformed<- (Height_capped^5)^1/4
hist(height_transformed)

NAs produced by integer overflow