MATH2349 Semester 1, 2019
Assignment 3
Abhileen Khurana(s3774173), Shivani(s3763179)
Required packages
library(readr)
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionlibrary(lubridate)##
## Attaching package: 'lubridate'## The following object is masked from 'package:base':
##
## datelibrary(tidyr)
library(Hmisc)## Loading required package: lattice## Loading required package: survival## Loading required package: Formula## Loading required package: ggplot2##
## Attaching package: 'Hmisc'## The following objects are masked from 'package:dplyr':
##
## src, summarize## The following objects are masked from 'package:base':
##
## format.pval, unitslibrary(outliers)
library(ggplot2)
library(infotheo)
library(forecast)
library(knitr)
require("knitr")
opts_knit$set(root.dir = "~/Downloads")Executive Summary
- For this assignment we dealt with dataset about FIFA world cup. We merged 3 different datasets WorldcupMatches, WorldcupPlayers, Worldcups. Which had information about matches, the players, coaches, where and when was it played etc. The new dataset was saved under the name world_cup1. The new table (world_cup1) had 39256 observation of 36 variables. We chose the variables that needed data type conversions like date, year etc. one of the variables was also converted into factor type and levels were labeled. We also renamed columns as they were two variables with same column name. Our dataset was already in tidy form and required no changes. Using mutate function we created three new variables that helped us in understanding the dataset more clearly. As already mentioned above our dataset had 36 variables including few which weren’t necessary, we subsetted our dataset by only selecting the variables that will help us in giving accurate analysis. Larger the dataset higher is the possibility of having missing values. Therefore, we scanned our dataset to find how many missing values are present in each column. Only match attendance was the one which had them. Further those were imputed with mean values of the variable. Boxplots were made to check the outliers for a few variables. The outliers were then handled using cap function. In the last step i.e. transformation we tried several ways to reduce the skewness of our variable Match_Attendance and also to make distribution closer to normal distribution. Squareroot transformation suited our variable the best.
Data
- FIFA world cup is a global competition in which all football playing nations participate. It is held after every four years and is one of the most important and of highest honour in the sport of football. The datasets was taken from https://www.kaggle.com. It has 3 datasets worldcup matches, worldcup players and worldcups which gives us all details of FIFA.
- Worldcup Matches dataset has 20 variables which describes us where the match was played by which team in respect to the date and year in which it was being played. The following are the variables in Worlcup Matches dataset :-
- Year: The year in which the match was played
- Datetime: The date on which the match was played along with a 24hour format time
- Stage: the stage at which match was played
- Stadium: Stadium name where the match was held
- City: The city name, where the match was played
- Home.Team.Name: Home team country name
- Home.Team.Goals: Total goals scored by the home team by the end of the match
- Away.Team.Goals: Total goals scored by the away team by the end of the match
- Away.Team.Name: Away team country name
- Win.onditions: special win condition (if any)
- Attendance: Total crowd present at the stadium
- Half.time.Home.Goals: Goals scored by the home team until half time
- Half.Time.Away.Goals: Goals scored by the away team until half time
- Referee: Name of the first referee
- Assistant.1: Name of the first assistant referee (linesman)
- Assistant.2: Name of the second assistant referee (linesman)
- RoundID: Unique ID of the round
- MatchID: Unique ID of the match
- Home.Team.Initials: Home team country’s three letter initials
- Away.Team.Initials: Away team country’s three letter initials
- Worldcup Players dataset has 9 variables that provide us information about all the players participating in the matches with the country they are representing and whom they are coached by and many other variables. The following are the variables in the Worlcup Players dataset:-
- RoundID: Unique ID of the round
- RoundID: Unique ID of the round
- MatchID: Unique ID of the match
- Team Initials: Player’s team initials
- Coach name: Name and country of the team coach
- Line-up: S=Line-up, N=Substitute
- Shirt number: Shirt number of the player (if available)
- Player Name: Name of the player
- Position: C=Captain, GK=Goalkeeper
- Event: G=Goal, OG=Own Goal, Y=Yellow card, R=Red Card, SY=Red Card by Second Yellow, P=Penalty, MP=Missed Penalty, I=Substitution In, 0=Substitute Out
- WorldCups dataset has 10 variables which helps us with information about ranking and position of countries in each worlcup along with how many total goals are scored and other information. The following are the variables in the worlcup dataset :-
- Year: Year of the worldcup
- Country: Country of the worldcup
- Winner: Team who won the worldcup
- Runners-up: Team who was the second place
- Third: Team who was the third place
- Fourth: Team who was the fourth place
- GoalsScored: Toatl goals scored in the world cup
- QualifiedTeams: Total participating teams
- MatchesPlayed: Total matches played in the cup
- Attendance: Total attendance of the worldcup
All the 3 datasets were related and had common variables. Therefore worldcupMatches and Worldcups tables shared common variable, using Inner Join both tables were joined using common variable ‘Year’ and saved under a new table ‘world_cup’. Further, World_cup and worldcupPlayers also had common variables ‘roundID’ and ‘matchID’ which were used to inner join the tables. The tables formed was then saved under ‘world_cup1’. Head() function was used to check if all the variables are present in the new table ‘world_cup1’.
WorldCupMatches <- read.csv("~/Downloads/fifa-world-cup/WorldCupMatches.csv")
head(WorldCupMatches, n=2)WorldCupPlayers <- read.csv("~/Downloads/fifa-world-cup/WorldCupPlayers.csv")
head(WorldCupPlayers, n=2)WorldCups <- read.csv("~/Downloads/fifa-world-cup/WorldCups.csv")
head(WorldCups, n=2)world_cup <- inner_join(WorldCupMatches, WorldCups, by = "Year")
world_cup1 <- inner_join(world_cup, WorldCupPlayers, by = c("RoundID","MatchID"))
head(world_cup1, n=2)Understand
- Structure of the dataset ‘world_cup1’ was checked using str() function. In addition, class for all the attributes were checked. They were all in the data frame format.
- The initial datasets had a few variables with same name. Attendance variable was present in both worldcups and worldcupMatches dataset. Therefore, After combining the dataset using inner join R assigned names to them itself to differentiate them ‘Attendance.x’ and ‘Attendance.y’. We further renamed them ourselves according to the description of the respective variables using name() function. Attendance.x in worldcupMatches was renamed to ‘Match_Attendance’. Similarly, ‘Attendance.y’ in worldcups was renamed to ‘Total_Attendance’.
- Data conversions were made in the final table. The DateTime variable was converted from character form to date type using dmy_hm() function. Data type of both Attendance variables was changed. Match_Attendance was in numeric form while Total_Attendance was in factor form were converted into integer form using as.integer() function. Lastly, year variables was in numeric form and converted into factor using as.factor() function.
- The line up variable had two levels “L” and “S” which were changed into respective labels “Line up” and “substitute”.
#understand
str(world_cup1)## 'data.frame': 39256 obs. of 36 variables:
## $ Year : int 1930 1930 1930 1930 1930 1930 1930 1930 1930 1930 ...
## $ Datetime : Factor w/ 603 levels "","01 Jul 1950 - 15:00 ",..: 206 206 206 206 206 206 206 206 206 206 ...
## $ Stage : Factor w/ 24 levels "","Final","First round",..: 4 4 4 4 4 4 4 4 4 4 ...
## $ Stadium : Factor w/ 182 levels "","Arena Amazonia",..: 126 126 126 126 126 126 126 126 126 126 ...
## $ City : Factor w/ 152 levels "","Alicante ",..: 85 85 85 85 85 85 85 85 85 85 ...
## $ Home.Team.Name : Factor w/ 79 levels "","Algeria","Angola",..: 25 25 25 25 25 25 25 25 25 25 ...
## $ Home.Team.Goals : int 4 4 4 4 4 4 4 4 4 4 ...
## $ Away.Team.Goals : int 1 1 1 1 1 1 1 1 1 1 ...
## $ Away.Team.Name : Factor w/ 84 levels "","Algeria","Angola",..: 47 47 47 47 47 47 47 47 47 47 ...
## $ Win.conditions : Factor w/ 44 levels ""," "," win on penalties (2 - 3) ",..: 2 2 2 2 2 2 2 2 2 2 ...
## $ Attendance.x : int 4444 4444 4444 4444 4444 4444 4444 4444 4444 4444 ...
## $ Half.time.Home.Goals: int 3 3 3 3 3 3 3 3 3 3 ...
## $ Half.time.Away.Goals: int 0 0 0 0 0 0 0 0 0 0 ...
## $ Referee : Factor w/ 367 levels "","ABD EL FATAH Essam (EGY)",..: 193 193 193 193 193 193 193 193 193 193 ...
## $ Assistant.1 : Factor w/ 388 levels "","ABDUL HAMID Halim (MAS)",..: 88 88 88 88 88 88 88 88 88 88 ...
## $ Assistant.2 : Factor w/ 409 levels "","ABDUL HAMID Halim (MAS)",..: 293 293 293 293 293 293 293 293 293 293 ...
## $ RoundID : int 201 201 201 201 201 201 201 201 201 201 ...
## $ MatchID : int 1096 1096 1096 1096 1096 1096 1096 1096 1096 1096 ...
## $ Home.Team.Initials : Factor w/ 78 levels "","ALG","ANG",..: 26 26 26 26 26 26 26 26 26 26 ...
## $ Away.Team.Initials : Factor w/ 83 levels "","ALG","ANG",..: 48 48 48 48 48 48 48 48 48 48 ...
## $ Country : Factor w/ 15 levels "Argentina","Brazil",..: 14 14 14 14 14 14 14 14 14 14 ...
## $ Winner : Factor w/ 9 levels "Argentina","Brazil",..: 9 9 9 9 9 9 9 9 9 9 ...
## $ Runners.Up : Factor w/ 10 levels "Argentina","Brazil",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ Third : Factor w/ 14 levels "Austria","Brazil",..: 14 14 14 14 14 14 14 14 14 14 ...
## $ Fourth : Factor w/ 16 levels "Austria","Belgium",..: 16 16 16 16 16 16 16 16 16 16 ...
## $ GoalsScored : int 70 70 70 70 70 70 70 70 70 70 ...
## $ QualifiedTeams : int 13 13 13 13 13 13 13 13 13 13 ...
## $ MatchesPlayed : int 18 18 18 18 18 18 18 18 18 18 ...
## $ Attendance.y : Factor w/ 20 levels "1.045.246","1.545.791",..: 17 17 17 17 17 17 17 17 17 17 ...
## $ Team.Initials : Factor w/ 82 levels "ALG","ANG","ARG",..: 26 47 26 47 26 47 26 47 26 47 ...
## $ Coach.Name : Factor w/ 335 levels "ACOSTA Nelson (URU)",..: 50 169 50 169 50 169 50 169 50 169 ...
## $ Line.up : Factor w/ 2 levels "N","S": 2 2 2 2 2 2 2 2 2 2 ...
## $ Shirt.Number : int 0 0 0 0 0 0 0 0 0 0 ...
## $ Player.Name : Factor w/ 7663 levels "?URI?I?","A BAUTISTA",..: 317 5512 4706 3856 1988 5981 482 2988 2035 1666 ...
## $ Position : Factor w/ 4 levels "","C","GK","GKC": 3 3 1 1 1 2 1 1 1 1 ...
## $ Event : Factor w/ 1894 levels "","G1'","G1' G42'",..: 1 1 308 527 1 1 325 1 1 1 ...attributes(world_cup1)$class## [1] "data.frame"#Change Columns Name
names(world_cup1)[names(world_cup1) == "Attendance.x"] <- "Match_Attendance"
names(world_cup1)[names(world_cup1) == "Attendance.y"] <- "Total_Attendance"
#Data Type Coversion
world_cup1$Datetime <- dmy_hm(world_cup1$Datetime)
world_cup1$Match_Attendance <- as.integer(world_cup1$Match_Attendance)
world_cup1$Total_Attendance <- as.integer(world_cup1$Total_Attendance)
world_cup1$Year <- as.factor(world_cup1$Year)
#Labelled Factor Variable
world_cup1$Line.up <- world_cup1$Line.up %>% factor(levels = c("S","N"), labels = c("Line Up","Substitute"))Tidy & Manipulate Data I
Dataset world_cup1 was joined with 3 datasets as mentioned above. All the 3 datasets worldcupMatches, wordcupPlayers, Worldcups were in tidy format.
Tidy & Manipulate Data II
- Number of goals scored over the world cup were recorded for each team. We were given total number of goals as well as The number of goals scored during half-time for each game. Using mutate() function we created new columns.
- Total_Goals column was created which recorded the sum of “Home Team Goals” and “Away Team Goals”.
- Total_HalfTime_Goals column was created which recorded sum of “Half-Time Home Goals” and “Half-Time Away Goals”.
- Average_Goal column was created which recorded “GoalsScored” over “MatchesPlayed”.
- Further the table was subsetted using select() function where few columns were selected to be removed like “win conditions”, all the columns from “referee to Assistant 2”, “Home Team initials to Away team Initials” and “Event”.
#Creating new columns
world_cup1 <- mutate(world_cup1, Total_Goals = Home.Team.Goals + Away.Team.Goals)
world_cup1 <- mutate(world_cup1, Total_HalfTime_Goals = Half.time.Home.Goals + Half.time.Away.Goals)
world_cup1 <- mutate(world_cup1, Average_Goal = `GoalsScored`/ `MatchesPlayed`)
#Selecting the the required columns
world_cup1 <- world_cup1 %>% select(-(Win.conditions), -(Referee:Assistant.2),-(Home.Team.Initials: Away.Team.Initials), -(Event) )
head(world_cup1, n=2)str(world_cup1)## 'data.frame': 39256 obs. of 32 variables:
## $ Year : Factor w/ 20 levels "1930","1934",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ Datetime : POSIXct, format: "1930-07-13 15:00:00" "1930-07-13 15:00:00" ...
## $ Stage : Factor w/ 24 levels "","Final","First round",..: 4 4 4 4 4 4 4 4 4 4 ...
## $ Stadium : Factor w/ 182 levels "","Arena Amazonia",..: 126 126 126 126 126 126 126 126 126 126 ...
## $ City : Factor w/ 152 levels "","Alicante ",..: 85 85 85 85 85 85 85 85 85 85 ...
## $ Home.Team.Name : Factor w/ 79 levels "","Algeria","Angola",..: 25 25 25 25 25 25 25 25 25 25 ...
## $ Home.Team.Goals : int 4 4 4 4 4 4 4 4 4 4 ...
## $ Away.Team.Goals : int 1 1 1 1 1 1 1 1 1 1 ...
## $ Away.Team.Name : Factor w/ 84 levels "","Algeria","Angola",..: 47 47 47 47 47 47 47 47 47 47 ...
## $ Match_Attendance : int 4444 4444 4444 4444 4444 4444 4444 4444 4444 4444 ...
## $ Half.time.Home.Goals: int 3 3 3 3 3 3 3 3 3 3 ...
## $ Half.time.Away.Goals: int 0 0 0 0 0 0 0 0 0 0 ...
## $ RoundID : int 201 201 201 201 201 201 201 201 201 201 ...
## $ MatchID : int 1096 1096 1096 1096 1096 1096 1096 1096 1096 1096 ...
## $ Country : Factor w/ 15 levels "Argentina","Brazil",..: 14 14 14 14 14 14 14 14 14 14 ...
## $ Winner : Factor w/ 9 levels "Argentina","Brazil",..: 9 9 9 9 9 9 9 9 9 9 ...
## $ Runners.Up : Factor w/ 10 levels "Argentina","Brazil",..: 1 1 1 1 1 1 1 1 1 1 ...
## $ Third : Factor w/ 14 levels "Austria","Brazil",..: 14 14 14 14 14 14 14 14 14 14 ...
## $ Fourth : Factor w/ 16 levels "Austria","Belgium",..: 16 16 16 16 16 16 16 16 16 16 ...
## $ GoalsScored : int 70 70 70 70 70 70 70 70 70 70 ...
## $ QualifiedTeams : int 13 13 13 13 13 13 13 13 13 13 ...
## $ MatchesPlayed : int 18 18 18 18 18 18 18 18 18 18 ...
## $ Total_Attendance : int 17 17 17 17 17 17 17 17 17 17 ...
## $ Team.Initials : Factor w/ 82 levels "ALG","ANG","ARG",..: 26 47 26 47 26 47 26 47 26 47 ...
## $ Coach.Name : Factor w/ 335 levels "ACOSTA Nelson (URU)",..: 50 169 50 169 50 169 50 169 50 169 ...
## $ Line.up : Factor w/ 2 levels "Line Up","Substitute": 1 1 1 1 1 1 1 1 1 1 ...
## $ Shirt.Number : int 0 0 0 0 0 0 0 0 0 0 ...
## $ Player.Name : Factor w/ 7663 levels "?URI?I?","A BAUTISTA",..: 317 5512 4706 3856 1988 5981 482 2988 2035 1666 ...
## $ Position : Factor w/ 4 levels "","C","GK","GKC": 3 3 1 1 1 2 1 1 1 1 ...
## $ Total_Goals : int 5 5 5 5 5 5 5 5 5 5 ...
## $ Total_HalfTime_Goals: int 3 3 3 3 3 3 3 3 3 3 ...
## $ Average_Goal : num 3.89 3.89 3.89 3.89 3.89 ...Scan I
- Missing values were checked in all columns using colsums(is.na()) function for the subsetted dataset world_cup1. After checking we found 184 ‘NA’ values in “Match_Attendance”.
- The NA values in Match_Attendance was imputed with mean of the variable because the number of NA were limited and could be imputed easily.
- Similarly, ‘Infinite’ and ‘NaN’ values were also checked in the world_cup1 tables using is.specialinfinite() function and is.specialNaN() function respectively. There were no infinite and NaN values present in the table.
#Scanning for missing Values
colSums(is.na(world_cup1))## Year Datetime Stage
## 0 0 0
## Stadium City Home.Team.Name
## 0 0 0
## Home.Team.Goals Away.Team.Goals Away.Team.Name
## 0 0 0
## Match_Attendance Half.time.Home.Goals Half.time.Away.Goals
## 184 0 0
## RoundID MatchID Country
## 0 0 0
## Winner Runners.Up Third
## 0 0 0
## Fourth GoalsScored QualifiedTeams
## 0 0 0
## MatchesPlayed Total_Attendance Team.Initials
## 0 0 0
## Coach.Name Line.up Shirt.Number
## 0 0 0
## Player.Name Position Total_Goals
## 0 0 0
## Total_HalfTime_Goals Average_Goal
## 0 0#Imputing values
world_cup1$Match_Attendance <- as.integer(impute(world_cup1$Match_Attendance, fun = mean))
is.specialInfinite <- function(x){
sum(is.infinite(x))}
sapply(world_cup1,is.specialInfinite)## Year Datetime Stage
## 0 0 0
## Stadium City Home.Team.Name
## 0 0 0
## Home.Team.Goals Away.Team.Goals Away.Team.Name
## 0 0 0
## Match_Attendance Half.time.Home.Goals Half.time.Away.Goals
## 0 0 0
## RoundID MatchID Country
## 0 0 0
## Winner Runners.Up Third
## 0 0 0
## Fourth GoalsScored QualifiedTeams
## 0 0 0
## MatchesPlayed Total_Attendance Team.Initials
## 0 0 0
## Coach.Name Line.up Shirt.Number
## 0 0 0
## Player.Name Position Total_Goals
## 0 0 0
## Total_HalfTime_Goals Average_Goal
## 0 0is.specialNan <- function(x){
sum(is.nan(x))}
sapply(world_cup1,is.specialNan)## Year Datetime Stage
## 0 0 0
## Stadium City Home.Team.Name
## 0 0 0
## Home.Team.Goals Away.Team.Goals Away.Team.Name
## 0 0 0
## Match_Attendance Half.time.Home.Goals Half.time.Away.Goals
## 0 0 0
## RoundID MatchID Country
## 0 0 0
## Winner Runners.Up Third
## 0 0 0
## Fourth GoalsScored QualifiedTeams
## 0 0 0
## MatchesPlayed Total_Attendance Team.Initials
## 0 0 0
## Coach.Name Line.up Shirt.Number
## 0 0 0
## Player.Name Position Total_Goals
## 0 0 0
## Total_HalfTime_Goals Average_Goal
## 0 0Scan II
- Outliers were scanned for various variables using boxplot. We only checked for limited variables and results were that no outliers were found in ‘GoalsScored’ and ‘Total_Attendance’. While outliers were found in variables ‘Match_Attendance’, ‘Total_HalfTime_Goals’, ‘Total_Goals’, ‘Average_Goal’.
- Goals recorded in the matches are not errors but accurate values of how many goals are scored by players during matches played in worldcups. Therefore they can be extreme as well. So the outliers should not be removed as it will not give an accurate analysis of the dataset.
- Outliers for variable “Match_Attendance” were approached and dealt using capping function. As we can see from the boxplot of Match_Attendance that outliers were close to 95th percentaile. Therefore they were replaced by values near third quartile and then checked again if they were handled or not.
par(mfrow = c(2,2))
#scanning for outliers
world_cup1$GoalsScored %>% boxplot(main="Box Plot of Goals Scored", ylab="Goals", col = "grey")
world_cup1$Total_Attendance %>% boxplot(main="Box Plot of Total Attendance", ylab="Attendance", col = "grey")
world_cup1$Match_Attendance %>% boxplot(main="Box Plot of Match Attendance", ylab="Attendance", col = "grey")
world_cup1$Total_HalfTime_Goals %>% boxplot(main="Box Plot of Total Half Time Goals", ylab="Goals", col = "grey")
world_cup1$Total_Goals %>% boxplot(main="Box Plot of Total Goals", ylab="Goals", col = "grey")
world_cup1$Average_Goal %>% boxplot(main="Box Plot of Average Goals per Match", ylab="Goals", col = "grey")
#capping Outliers
cap <- function(x){
quantiles <- quantile( x, c(.05, 0.25, 0.75, .95 ) )
x[ x < quantiles[2] - 1.5*IQR(x) ] <- quantiles[1]
x[ x > quantiles[3] + 1.5*IQR(x) ] <- quantiles[4]
x}
world_cup1$Match_Attendance <- world_cup1$Match_Attendance %>% cap()
#checking again for outliers
world_cup1$Match_Attendance %>% boxplot(main="Box Plot of Match Attendance", ylab="Attendance", col = "grey")
Transform
- To perform transformation on we selected ‘Match_Attendance’ variable. Hist() function was used to check if the variable was normally distributed or was left/right skewed.
- Various data transformations were used to check which one of them most suited the data by making it less skewed.
- Both log transformations were making the histogram inclined more towards left skewed also Box-cox transformation did the same.
- Reciprocal transformation did the opposite of the above and made the histogram right skewed.
- Only square root transformation was the one which suited the most to the histogram and reduced the right skewness making it the only one closer to being normally distributed.
#transformation
hist(world_cup1$Match_Attendance)
#The Log transformation
log_Match_Attendance <- log10(world_cup1$Match_Attendance)
hist(log_Match_Attendance)
ln_Match_Attendance <- log(world_cup1$Match_Attendance)
hist(ln_Match_Attendance)
#square transformation
sqrt_Match_Attendance <- sqrt(world_cup1$Match_Attendance)
hist(sqrt_Match_Attendance)
#Box-Cox transformation
boxcox_Match_Attendance<- BoxCox(world_cup1$Match_Attendance,lambda = "auto")
hist(boxcox_Match_Attendance)
#reciprocal transformation
reci_Match_Attendance <- 1/world_cup1$Match_Attendance
hist(reci_Match_Attendance)