MATH2349 Semester 2, 2018

Assignment 3

Required packages

library(readr)
library(tidyr)
library(dplyr)

Attaching package: <U+393C><U+3E31>dplyr<U+393C><U+3E32>

The following objects are masked from <U+393C><U+3E31>package:stats<U+393C><U+3E32>:

    filter, lag

The following objects are masked from <U+393C><U+3E31>package:base<U+393C><U+3E32>:

    intersect, setdiff, setequal, union

library(Hmisc)

Loading required package: lattice
Loading required package: survival
Loading required package: Formula
Loading required package: ggplot2

Attaching package: <U+393C><U+3E31>Hmisc<U+393C><U+3E32>

The following objects are masked from <U+393C><U+3E31>package:dplyr<U+393C><U+3E32>:

    src, summarize

The following objects are masked from <U+393C><U+3E31>package:base<U+393C><U+3E32>:

    format.pval, units

library(knitr)
library(outliers)

Executive Summary

During this course we have learnt that ‘Data Preprocessing’ is a process required to prepare all forms of untidy data for statistical analysis. Once I found my data for this assignment, I began preproessing my data. The steps I took throughout this assignment were: 1. Got my two data sets (First data set on Birth Rates from 1960-2017 in various countries, and the second set of data on Death Rates from 1960-2017 in various countries) 2. Understand both data sets and what’s consisted within it (where I decided to only look at data from both tables for only three countries, which were India, Australia and United States) 3. Tidy and manipulate the data (where I tidied the data to a format where I was able to merge both data sets into one) 4. Scan the data for any errors or inconsistencies (where I removed all na’s within the data sets and replaced them with the mean of the columns where the data was missing) 5. Transformed the data to get another view of what it shows (where I transformed the death rate data as this data set had the most outliers)

Data

First set of data is based on birth rates within various countries from 1960 to 2017

Birth Rate Data Source: https://data.worldbank.org/indicator/SP.DYN.CBRT.IN Data fields within thie table are: 1. CountryName (name of country being represented in the data) 2. CountryCode (consisting three letter codes representing the country) 3. IndicatorName (indicating the data value being analysed in this data, which is the Birth rate, crude (per 1,000 people)) 4. IndicatorCode (the code that represents the birth rate indicator) 5. The remainder columns are the years from 1960 to 2017

Second set of data is based on death rates within various countries from 1960 to 2017

Death Rate data source: https://data.worldbank.org/indicator/SP.DYN.CDRT.IN Data fields within thie table are: 1. CountryName (name of country being represented in the data) 2. CountryCode (consisting three letter codes representing the country) 3. IndicatorName (indicating the data value being analysed in this data, which is the Death rate, crude (per 1,000 people)) 4. IndicatorCode (the code that represents the death rate indicator) 5. The remainder columns are the years from 1960 to 2017

Births <- read_csv("Birth Rate.csv") #loading the Birth Rate data from the working directory

Parsed with column specification:
cols(
  .default = col_double(),
  CountryName = col_character(),
  CountryCode = col_character(),
  IndicatorName = col_character(),
  IndicatorCode = col_character(),
  `2017` = col_character()
)
See spec(...) for full column specifications.

Deaths <- read_csv("Death Rate.csv") #loading the Death Rate data from the working directory

Parsed with column specification:
cols(
  .default = col_double(),
  CountryName = col_character(),
  CountryCode = col_character(),
  IndicatorName = col_character(),
  IndicatorCode = col_character(),
  `2017` = col_character()
)
See spec(...) for full column specifications.

head(Births)

head(Deaths)

Births <- Births %>%  filter( CountryName %in% c("Australia","India", "United States")) #filtering the birth rate data to only consist Australia, India and USA
Births <- Births %>% gather(Year, Birth_Rate, '1960':'2017') #creating a new column 'Year', and gathering the year columns under it
Births <- Births %>% select(CountryName, CountryCode, Year, Birth_Rate) #selecting the required columns I wish to use in this assignment
head(Births)

Deaths <- Deaths %>%  filter( CountryName %in% c("Australia","India", "United States")) #filtering the death rate data to only consist Australia, India and USA
Deaths <- Deaths %>% gather(Year, Death_Rate, '1960':'2017') #creating a new column 'Year', and gathering the year columns under it
Deaths <- Deaths %>% select(CountryName, CountryCode, Year, Death_Rate) #selecting the required columns I wish to use in this assignment
head(Deaths)

Data <- Births %>% left_join(select(Deaths,CountryName, Year, Death_Rate), by= c ("CountryName", "Year")) #joining the death rate data with the birth rate data
head(Data)

Understand

str(Data) #checking the structure and attributes of the merged data set

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   174 obs. of  5 variables:
 $ CountryName: chr  "Australia" "India" "United States" "Australia" ...
 $ CountryCode: chr  "AUS" "IND" "USA" "AUS" ...
 $ Year       : chr  "1960" "1960" "1960" "1961" ...
 $ Birth_Rate : chr  "22.4" "42.1" "23.7" "22.9" ...
 $ Death_Rate : chr  "8.6" "22.422" "9.5" "8.5" ...

Data$Year <- as.numeric(Data$Year) #converting Year from character to numeric value
class(Data$Year) #checking class of Year is numeric now

[1] "numeric"

Data$Birth_Rate <- as.numeric(Data$Birth_Rate) #converting Birth_Rate from character to numeric value
class(Data$Birth_Rate) #checking class of Birth_Rate is numeric now

[1] "numeric"

Data$Death_Rate <- as.numeric(Data$Death_Rate) #converting Death_Rate from character to numeric value
class(Data$Death_Rate) #checking class of Death_Rate is numeric now

[1] "numeric"

Data$CountryCode <- Data$CountryCode %>% factor(levels= c("AUS", "IND", "USA"), labels =c("AUS", "IND", "USA"), ordered = TRUE) #converting Countrycode from charatcer to factor while ordering the labels in the column
class(Data$CountryCode)

[1] "ordered" "factor" 

str(Data) #checking the structure and attributes of the updated final data set

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   174 obs. of  5 variables:
 $ CountryName: chr  "Australia" "India" "United States" "Australia" ...
 $ CountryCode: Ord.factor w/ 3 levels "AUS"<"IND"<"USA": 1 2 3 1 2 3 1 2 3 1 ...
 $ Year       : num  1960 1960 1960 1961 1961 ...
 $ Birth_Rate : num  22.4 42.1 23.7 22.9 41.9 ...
 $ Death_Rate : num  8.6 22.4 9.5 8.5 21.9 ...

Tidy & Manipulate Data I

#Prior to merging the Births and Deaths data, I had already structured the data in tidy format which requires each variable in its own column, each observation in its own row and finally each value in its own cell. This can be seen below.
Data

Tidy & Manipulate Data II

#Looking at the Data table above we can see the birth rate as well as the death rate, as a whole we can see that births seem higher than deaths, which shows a increase in population year on year. To be able to visibaly see this I have created a new column using mutate for growth as below
Data <- mutate(Data, Growth = Data$Birth_Rate - Data$Death_Rate)
head(Data)

Scan I

colSums(is.na(Data)) #this is showing the total number of na's per column of the data table

CountryName CountryCode        Year  Birth_Rate  Death_Rate      Growth 
          0           0           0           3           3           3 

#location of na's
head(which(is.na(Data)))

[1] 694 695 696 868 869 870

#checking for inconsistencies in all columns
sum(is.nan(Data$CountryName)) 

[1] 0

sum(is.nan(Data$CountryCode))

[1] 0

sum(is.nan(Data$Year))

[1] 0

sum(is.nan(Data$Death_Rate))

[1] 0

sum(is.nan(Data$Birth_Rate))

[1] 0

sum(is.nan(Data$Growth))

[1] 0

#count for infinite value in all columns
sum(is.infinite(Data$CountryName)) 

[1] 0

sum(is.infinite(Data$CountryCode))

[1] 0

sum(is.infinite(Data$Year))

[1] 0

sum(is.infinite(Data$Death_Rate))

[1] 0

sum(is.infinite(Data$Birth_Rate))

[1] 0

sum(is.infinite(Data$Growth))

[1] 0

#replacing all na's with column mean
Data$Birth_Rate[is.na(Data$Birth_Rate)] <- round(mean(Data$Birth_Rate, na.rm = TRUE))
Data$Death_Rate[is.na(Data$Death_Rate)] <- round(mean(Data$Death_Rate, na.rm = TRUE))
Data$Growth[is.na(Data$Growth)] <- round(mean(Data$Growth, na.rm = TRUE))
colSums(is.na(Data)) #after replacing na's with mean of column, just checking to ensure no na's left in data

CountryName CountryCode        Year  Birth_Rate  Death_Rate      Growth 
          0           0           0           0           0           0 

Scan II

Data$Birth_Rate %>%  boxplot(main="BoxPlot - Birth Rate", ylab="Birth_Rate", col = "grey")  #boxplot to show where the outliers sit within the Birth_Rate data

Data$Death_Rate %>%  boxplot(main="BoxPlot - Death Rate", ylab="Death_Rate", col = "grey")  #boxplot to show where the outliers sit within the Death_Rate data

Data$Growth %>%  boxplot(main="BoxPlot - Growth", ylab="Growth", col = "grey")  #boxplot to show where the outliers sit within the Growth data

summary(Data$Birth_Rate) #summary of Birth_Rate column to ensure no errors in data

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  12.30   14.53   16.95   21.09   24.99   42.10 

summary(Data$Death_Rate) #summary of Death_Rate column to ensure no errors in data

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   6.40    7.50    8.60    9.52    9.50   22.42 

summary(Data$Growth) #summary of Growth column to ensure no errors in data

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   3.96    6.75    8.40   11.59   16.71   22.82 

z.scores.b <- Data$Birth_Rate %>%  scores(type = "z")
z.score.b %>% summary() #z score summary result for Birth_Rate column

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-1.0025 -0.7489 -0.4724  0.0000  0.4445  2.3947 

z.scores.d <- Data$Death_Rate %>%  scores(type = "z")
z.scores.d %>% summary() #z score summary result for Death_Rate column

     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
-0.929066 -0.601483 -0.273900  0.000000 -0.005877  3.842331 

z.scores.g <- Data$Growth %>%  scores(type = "z")
z.scores.g %>% summary() #z score summary result for Growth column

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
-1.2640 -0.8019 -0.5286  0.0000  0.8471  1.8597 

Data$Birth_Rate[ which( abs(z.scores.b) >3 )] #locating outliers in column Birth_Rate

numeric(0)

Data$Death_Rate[ which( abs(z.scores.d) >3 )] #locating outliers in column Death_Rate

[1] 22.422 21.876 21.335 20.798 20.265 19.737

Data$Growth[ which( abs(z.scores.g) >3 )] #locating outliers for Growth column

numeric(0)

summary(Data$Death_Rate)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  6.400   7.500   8.600   9.121   9.500  19.215 

Transform

hist(Data$Death_Rate, main = "Death Rate Histogram") 

#histogram to show distribution of death rate without any manipulation. We can see the histogram is skewed towwrds the left

Transformed_Death_Rate <- sqrt(Data$Death_Rate)
hist(Transformed_Death_Rate, main = "Transformed Death Rate Histogram") 

#used sqare root of death rate  to get more even distridution on the histogram. Still not symmetrical due to the number of outliers within the data