MATH2349 Semester 1, 2020

Assignment 2

Required packages

The packages used in this project.

library(dplyr)
library(readr)
library(ggplot2)
library(stringr)
library(tidyr)
library(lubridate)
library(magrittr)
library(outliers)

Executive Summary

Data preprocessing is an imperative step in the data preparation stage of the Data Science process. For learning purposes, the decision was made to merge and preprocess two datasets. To keep with the times, a Covid-19 dataset and a SARS dataset was taken from Kaggle in order to investigate the idea of which pandemic is more fatal. The two selected datasets required much preprocessing. While both datasets required some common preprocessing such as the renaming of columns and selected row values, one dataset was relatively untidy. In the Covid dataset, one of the columns had both date and time combined inconsistently, while also having different date formats. This column needed to be split up and formatted appropriately in preparation for merging the two selected datasets for this project. Before merging the two datasets, the Covid dataset had to be further manipulated in order to be able to be properly merged with the SARS dataset. Once the datasets have been merged, one final data cleaning needed to be done to the newly merged dataset. With the merged dataset, a new variable was mutated in order to compare the deadliness of the two pandemics. The dataset also needed to be scanned for missing values, special values and outliers. However, at that point the dataset had been appropriately cleaned so no missing values or special values were found. The outliers were found but was justifiably left alone as they were important values. While data normalisation was attempted on the variables of the dataset, the results of the transformation did not yield any interesting results. Overall, the main goal of the project was successfully reached – with results supporting the notion that SARS is the more fatal pandemic compared to Covid-19.

Data description & Data importing

The Covid dataset was collected for research purposes by SRK from the John Hopkins University and posted on Kaggle. The dataset can be found on Kaggle(https://www.kaggle.com/sudalairajkumar/novel-corona-virus-2019-dataset?select=covid_19_data.csv). This repository contains many different csv files. But for this project, only the Covid-19 dataset will be the point of focus from this repository. The aforementioned Covid-19 dataset in the “covid_19_data.csv” file contains the following variables:

• Sno: Serial number
• ObservationDate: Date of the observation
• Province/State: Province or state of the observation
• Country/Region: Country of observation
• Last Update: Date and time at which the row is updated
• Confirmed: Cumulative number of confirmed cases up to that date
• Deaths: Cumulative number of deaths up to that date
• Recovered: Cumulative number of recovered cases up to that date

The SARS dataset was collected for research purposes by Devakumar kp from the World Health Organization website and posted on Kaggle. The dataset can be found on Kaggle(https://www.kaggle.com/imdevskp/sars-outbreak-2003-complete-dataset). This repository contains two different csv files. But for this project, only the cleaned summary dataset will be the point of focus from this repository. The aforementioned Sars dataset in the “summary_data_clean.csv” file contains the following variables:

• Country/Region: Country of observation
• Cumulative male cases: Total male cases
• Cumulative female cases: Total female cases
• Cumulative total cases: Total cases
• No. of deaths: Number of deaths
• Case fatalities ratio (%): Case fatalities in percentage
• Date onset first probable case: The date of the first probable Sars case
• Date onset last probable case: The date of the last probable Sars case
• Median age: The median age
• Age range: The age range of the data
• Percentage of Imported cases: The percentage of cases due to immigration
• Number of HCW affected: The number of health care worker’s affected
• Percentage of HCW affected: The percentage of health care worker’s affected

———————————————————————————————————–

The goal of this project is to clean the Covid-19 dataset and merge it with a cleaned SARS dataset. Then, mutate a Covid fatality ratio variable so it can be compared to the Sars fatality ratio in order to find out which pandemic is deadlier based on countries.

———————————————————————————————————–

To achieve this, the datasets will firstly be imported and subsetted. The datasets need to be subsetted as both datasets contain columns that are not relevant to the goal of this project. The datasets were subsetted as follows:

• Covid: “SNo” and “Province/State” columns were removed as this project focuses more on Countries. The SNo column was removed as the serial number of the cases is not necessary for the goal of this project.

• Sars: “Cumulative male cases”, “Cumulative female cases”, “Age range”, “Percentage of Imported cases”, “Number of HCW affected” and “Percentage of HCW affected” columns were removed as these columns are not necessary for the goal of this project.

# Import the data
covidRaw <- read_csv("covid_19_data.csv")

Parsed with column specification:
cols(
  SNo = col_double(),
  ObservationDate = col_character(),
  `Province/State` = col_character(),
  `Country/Region` = col_character(),
  `Last Update` = col_character(),
  Confirmed = col_double(),
  Deaths = col_double(),
  Recovered = col_double()
)

sarsRaw <- read_csv("summary_data_clean.csv")

Parsed with column specification:
cols(
  `Country/Region` = col_character(),
  `Cumulative male cases` = col_double(),
  `Cumulative female cases` = col_double(),
  `Cumulative total cases` = col_double(),
  `No. of deaths` = col_double(),
  `Case fatalities ratio (%)` = col_double(),
  `Date onset first probable case` = col_date(format = ""),
  `Date onset last probable case` = col_date(format = ""),
  `Median age` = col_double(),
  `Age range` = col_character(),
  `Number of Imported cases` = col_double(),
  `Percentage of Imported cases` = col_double(),
  `Number of HCW affected` = col_double(),
  `Percentage of HCW affected` = col_double()
)

# Subset data
covid <- covidRaw[, c(2,4:7)]
sars <- sarsRaw[, c(1,4:8)]
# Print the data
head(covid)

head(sars)

Data cleaning I

After importing the datasets, the data needs to be briefly cleaned before tidying begins. The first thing that was done was to rename the columns for both datasets for consistency purposes. Then, selected rows of both datasets are also renamed for consistency and correctness purposes (Eg. in the Sars dataset, Vietnam was written as Viet Nam which is incorrect). Then, the structure of the datasets were inspected, and various data types were identified for both datasets. While the data type conversions will occur later in the process (Data cleaning II), the dates for both datasets were formatted for consistency.

# Inspecting and renaming columns for consistency purposes
colnames(covid)

[1] "ObservationDate" "Country/Region"  "Last Update"     "Confirmed"       "Deaths"         

covid %<>% rename(`CovidConfirmedCases` = "Confirmed", `CovidDeaths` = "Deaths",
                 `Covid First Observations` = "ObservationDate", `Covid Last Update` = "Last Update")
colnames(sars)

[1] "Country/Region"                 "Cumulative total cases"         "No. of deaths"                 
[4] "Case fatalities ratio (%)"      "Date onset first probable case" "Date onset last probable case" 

sars %<>% rename(`Sars Total Cases` = "Cumulative total cases",
                 `Sars Deaths` = "No. of deaths",
                 `Sars Fatality Ratio (%)` = "Case fatalities ratio (%)",
                 `Sars First Probable Case Date` = "Date onset first probable case",
                 `Sars Last Probable Case Date` = "Date onset last probable case")
# Inspecting and renaming rows for consistency purposes
unique(covid$`Country/Region`)

  [1] "Mainland China"                   "Hong Kong"                       
  [3] "Macau"                            "Taiwan"                          
  [5] "US"                               "Japan"                           
  [7] "Thailand"                         "South Korea"                     
  [9] "Singapore"                        "Philippines"                     
 [11] "Malaysia"                         "Vietnam"                         
 [13] "Australia"                        "Mexico"                          
 [15] "Brazil"                           "Colombia"                        
 [17] "France"                           "Nepal"                           
 [19] "Canada"                           "Cambodia"                        
 [21] "Sri Lanka"                        "Ivory Coast"                     
 [23] "Germany"                          "Finland"                         
 [25] "United Arab Emirates"             "India"                           
 [27] "Italy"                            "UK"                              
 [29] "Russia"                           "Sweden"                          
 [31] "Spain"                            "Belgium"                         
 [33] "Others"                           "Egypt"                           
 [35] "Iran"                             "Israel"                          
 [37] "Lebanon"                          "Iraq"                            
 [39] "Oman"                             "Afghanistan"                     
 [41] "Bahrain"                          "Kuwait"                          
 [43] "Austria"                          "Algeria"                         
 [45] "Croatia"                          "Switzerland"                     
 [47] "Pakistan"                         "Georgia"                         
 [49] "Greece"                           "North Macedonia"                 
 [51] "Norway"                           "Romania"                         
 [53] "Denmark"                          "Estonia"                         
 [55] "Netherlands"                      "San Marino"                      
 [57] "Azerbaijan"                       "Belarus"                         
 [59] "Iceland"                          "Lithuania"                       
 [61] "New Zealand"                      "Nigeria"                         
 [63] "North Ireland"                    "Ireland"                         
 [65] "Luxembourg"                       "Monaco"                          
 [67] "Qatar"                            "Ecuador"                         
 [69] "Czech Republic"                   "Armenia"                         
 [71] "Dominican Republic"               "Indonesia"                       
 [73] "Portugal"                         "Andorra"                         
 [75] "Latvia"                           "Morocco"                         
 [77] "Saudi Arabia"                     "Senegal"                         
 [79] "Argentina"                        "Chile"                           
 [81] "Jordan"                           "Ukraine"                         
 [83] "Saint Barthelemy"                 "Hungary"                         
 [85] "Faroe Islands"                    "Gibraltar"                       
 [87] "Liechtenstein"                    "Poland"                          
 [89] "Tunisia"                          "Palestine"                       
 [91] "Bosnia and Herzegovina"           "Slovenia"                        
 [93] "South Africa"                     "Bhutan"                          
 [95] "Cameroon"                         "Costa Rica"                      
 [97] "Peru"                             "Serbia"                          
 [99] "Slovakia"                         "Togo"                            
[101] "Vatican City"                     "French Guiana"                   
[103] "Malta"                            "Martinique"                      
[105] "Republic of Ireland"              "Bulgaria"                        
[107] "Maldives"                         "Bangladesh"                      
[109] "Moldova"                          "Paraguay"                        
[111] "Albania"                          "Cyprus"                          
[113] "St. Martin"                       "Brunei"                          
[115] "occupied Palestinian territory"   "('St. Martin',)"                 
[117] "Burkina Faso"                     "Channel Islands"                 
[119] "Holy See"                         "Mongolia"                        
[121] "Panama"                           "Bolivia"                         
[123] "Honduras"                         "Congo (Kinshasa)"                
[125] "Jamaica"                          "Reunion"                         
[127] "Turkey"                           "Cuba"                            
[129] "Guyana"                           "Kazakhstan"                      
[131] "Cayman Islands"                   "Guadeloupe"                      
[133] "Ethiopia"                         "Sudan"                           
[135] "Guinea"                           "Antigua and Barbuda"             
[137] "Aruba"                            "Kenya"                           
[139] "Uruguay"                          "Ghana"                           
[141] "Jersey"                           "Namibia"                         
[143] "Seychelles"                       "Trinidad and Tobago"             
[145] "Venezuela"                        "Curacao"                         
[147] "Eswatini"                         "Gabon"                           
[149] "Guatemala"                        "Guernsey"                        
[151] "Mauritania"                       "Rwanda"                          
[153] "Saint Lucia"                      "Saint Vincent and the Grenadines"
[155] "Suriname"                         "Kosovo"                          
[157] "Central African Republic"         "Congo (Brazzaville)"             
[159] "Equatorial Guinea"                "Uzbekistan"                      
[161] "Guam"                             "Puerto Rico"                     
[163] "Benin"                            "Greenland"                       
[165] "Liberia"                          "Mayotte"                         
[167] "Republic of the Congo"            "Somalia"                         
[169] "Tanzania"                         "The Bahamas"                     
[171] "Barbados"                         "Montenegro"                      
[173] "The Gambia"                       "Kyrgyzstan"                      
[175] "Mauritius"                        "Zambia"                          
[177] "Djibouti"                         "Gambia, The"                     
[179] "Bahamas, The"                     "Chad"                            
[181] "El Salvador"                      "Fiji"                            
[183] "Nicaragua"                        "Madagascar"                      
[185] "Haiti"                            "Angola"                          
[187] "Cabo Verde"                       "Niger"                           
[189] "Papua New Guinea"                 "Zimbabwe"                        
[191] "Cape Verde"                       "East Timor"                      
[193] "Eritrea"                          "Uganda"                          
[195] "Bahamas"                          "Dominica"                        
[197] "Gambia"                           "Grenada"                         
[199] "Mozambique"                       "Syria"                           
[201] "Timor-Leste"                      "Belize"                          
[203] "Laos"                             "Libya"                           
[205] "Diamond Princess"                 "Guinea-Bissau"                   
[207] "Mali"                             "Saint Kitts and Nevis"           
[209] "West Bank and Gaza"               "Burma"                           
[211] "MS Zaandam"                       "Botswana"                        
[213] "Burundi"                          "Sierra Leone"                    
[215] "Malawi"                           "South Sudan"                     
[217] "Western Sahara"                   "Sao Tome and Principe"           
[219] "Yemen"                            "Comoros"                         
[221] "Tajikistan"                       "Lesotho"                         

covid$`Country/Region`[covid$`Country/Region` == "UK"] <- "United Kingdom"
covid$`Country/Region`[covid$`Country/Region` == "US"] <- "United States"
unique(sars$`Country/Region`)

 [1] "Australia"            "Canada"               "China"                "Hong Kong SAR, China"
 [5] "Macao SAR, China"     "Taiwan"               "France"               "Germany"             
 [9] "India"                "Indonesia"            "Italy"                "Kuwait"              
[13] "Malaysia"             "Mongolia"             "New Zealand"          "Philippines"         
[17] "Republic of Ireland"  "Republic of Korea"    "Romania"              "Russian Federation"  
[21] "Singapore"            "South Africa"         "Spain"                "Sweden"              
[25] "Switzerland"          "Thailand"             "United Kingdom"       "United States"       
[29] "Viet Nam"            

sars[3, 1] <- "Mainland China"
sars[4, 1] <- "Hong Kong"
sars[5, 1] <- "Macau"
sars[18, 1] <- "South Korea"
sars[20, 1] <- "Russia"
sars[29, 1] <- "Vietnam"# Renaming mispelled `Viet Nam` to `Vietnam`
# Inspecting the structure of the datasets and making the date format consistent for the Covid dataset
str(covid)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   27618 obs. of  5 variables:
 $ Covid First Observations: chr  "01/22/2020" "01/22/2020" "01/22/2020" "01/22/2020" ...
 $ Country/Region          : chr  "Mainland China" "Mainland China" "Mainland China" "Mainland China" ...
 $ Covid Last Update       : chr  "1/22/2020 17:00" "1/22/2020 17:00" "1/22/2020 17:00" "1/22/2020 17:00" ...
 $ CovidConfirmedCases     : num  1 14 6 1 0 26 2 1 4 1 ...
 $ CovidDeaths             : num  0 0 0 0 0 0 0 0 0 0 ...

covid$`Covid First Observations` <- as.Date(covid$`Covid First Observations`, "%m/%d/%Y")
covid$`Covid First Observations` <- format(as.Date(covid$`Covid First Observations`, format = "%Y-%m-%d"), "%d/%m/%Y")
head(covid)

# Inspecting the structure of the datasets and making the date format consistent for the Sars dataset
str(sars)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   29 obs. of  6 variables:
 $ Country/Region               : chr  "Australia" "Canada" "Mainland China" "Hong Kong" ...
 $ Sars Total Cases             : num  6 251 5327 1755 1 ...
 $ Sars Deaths                  : num  0 43 349 299 0 37 1 0 0 0 ...
 $ Sars Fatality Ratio (%)      : num  0 17 7 17 0 11 14 0 0 0 ...
 $ Sars First Probable Case Date: Date, format: "2003-02-26" "2003-02-23" "2002-11-16" ...
 $ Sars Last Probable Case Date : Date, format: "2003-04-01" "2003-06-12" "2003-06-03" ...

sars$`Sars First Probable Case Date` <- strftime(sars$`Sars First Probable Case Date`, "%d/%m/%Y")
sars$`Sars Last Probable Case Date` <- strftime(sars$`Sars Last Probable Case Date`, "%d/%m/%Y")
head(sars)

Tidy & Manipulate Data I

Once the data has been briefly cleaned, it is time to tidy up the dataset. The Covid dataset is untidy as it contains multiple coulmns that are not necessary for the projects goal; which has been removed in the first section. The Covid dataset also possesses a column that contains both date and time (and some of them have letters instead of whitespaces as seperators). This is not tidy according to Hadley Wickham’s tidy principles which states that each value must have its own cell. To remedy this problem, the column was split into two using a tidyr function with a regex seperator. The date column was then restructured using the parse_date_time() function as the column had multiple different date formats. The Covid dataset was now much tidier than before once the unnecessary columns were removed and the untidy column was separated and formatted appropriately.

# Splitting the covidLastUpdate column
splitCovid <- separate(covid, `Covid Last Update`, c("Covid Last Update", "Last Update Time"), sep = "T|[:space:]")
# Tidying up the new column
unique(splitCovid$`Covid Last Update`)

  [1] "1/22/2020"  "1/23/20"    "1/24/20"    "1/25/20"    "1/26/20"    "1/27/20"    "1/28/20"   
  [8] "1/29/20"    "1/30/20"    "1/31/2020"  "2/1/2020"   "2020-02-02" "2020-02-01" "2020-01-31"
 [15] "2020-02-03" "2020-02-04" "2020-02-05" "2020-02-06" "2020-02-07" "2020-02-08" "2020-02-09"
 [22] "2020-02-10" "2020-02-11" "2020-02-12" "2020-02-13" "2020-02-14" "2020-02-15" "2020-02-16"
 [29] "2020-02-17" "2020-02-18" "2020-02-19" "2020-02-20" "2020-02-21" "2020-02-22" "2020-02-23"
 [36] "2020-02-24" "2020-02-25" "2020-02-26" "2020-02-27" "2020-02-28" "2020-02-29" "2020-03-01"
 [43] "2020-03-02" "2020-03-03" "2020-03-04" "2020-03-05" "2020-03-06" "2020-03-07" "2020-03-08"
 [50] "2020-03-09" "2020-03-10" "2020-03-11" "2020-03-12" "2020-03-13" "2020-03-14" "2020-03-15"
 [57] "2020-03-16" "2020-03-17" "2020-03-18" "2020-03-19" "2020-03-20" "2020-03-21" "3/8/20"    
 [64] "2020-03-23" "2020-03-24" "2020-03-25" "2020-03-26" "2020-03-27" "2020-03-31" "2020-04-01"
 [71] "4/2/20"     "2020-04-03" "4/4/20"     "2020-04-05" "4/6/20"     "2020-04-07" "2020-04-08"
 [78] "2020-04-09" "2020-04-10" "2020-04-11" "2020-04-12" "2020-04-13" "2020-04-14" "2020-04-15"
 [85] "2020-04-16" "2020-04-17" "2020-04-18" "2020-04-19" "2020-04-20" "2020-04-21" "2020-04-22"
 [92] "2020-04-24" "2020-04-25" "2020-04-26" "2020-04-27" "2020-04-28" "2020-04-29" "2020-04-30"
 [99] "2020-05-01" "2020-05-02" "2020-05-03" "2020-05-04" "2020-05-05" "2020-05-06" "2020-05-07"
[106] "2020-05-08" "2020-05-09" "2020-05-10" "2020-05-11" "2020-05-12" "2020-05-13" "2020-05-14"
[113] "2020-05-15" "2020-05-16" "2020-05-17" "2020-05-18" "2020-05-19" "2020-05-20" "2020-05-21"
[120] "2020-05-22"

splitCovid$`Covid Last Update` <- parse_date_time(x = splitCovid$`Covid Last Update`, orders = c("m d y", "y m d"))
head(splitCovid)

str(splitCovid)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   27618 obs. of  6 variables:
 $ Covid First Observations: chr  "22/01/2020" "22/01/2020" "22/01/2020" "22/01/2020" ...
 $ Country/Region          : chr  "Mainland China" "Mainland China" "Mainland China" "Mainland China" ...
 $ Covid Last Update       : POSIXct, format: "2020-01-22" "2020-01-22" "2020-01-22" ...
 $ Last Update Time        : chr  "17:00" "17:00" "17:00" "17:00" ...
 $ CovidConfirmedCases     : num  1 14 6 1 0 26 2 1 4 1 ...
 $ CovidDeaths             : num  0 0 0 0 0 0 0 0 0 0 ...

Merging the Data

Once the Covid dataset has been made tidy, the Sars dataset and Covid dataset can be merged together. The first step is to remove more unnecessary columns. The first was to drop the time column in the splitCovid dataset, that was attained from the section above as it serves no purpose for the goal of this project. The next step was to subset the splitCovid to a dataset that contained the countries and the concatenation of the total covid cases and total covid deaths by country. This dataset was called covidCases.

After that step, it was time to focus on the dates. The splitCovid was subset to two datasets, one for the latest updated Covid date by country and one by the earliest Covid date by country. Once these two datasets were complete, they were then merged together along with the covidCases set in order to attain the final covid dataset called covidSet. A right_join was used to combine covidFirstDate and covidCases as we want to prioritize the covidCases dataset. A left_join was used to merge the covidLastDate in as we want to prioritize the prior combined covidFirstDate and covidCases dataset.

Once the covidSet was completed, it was then finally merged with the Sars dataset using a left_join as we only want the countries that are in the Sars dataset (as the covid dataset has more countries than the Sars dataset)

# Dropping the time column
splitCovid <- select(splitCovid, -c(`Last Update Time`))
# Summarise all the total covid cases and total covid deaths by country
covidCases <- splitCovid %>% 
  group_by(`Country/Region`) %>% 
  summarise(sum(CovidConfirmedCases), sum(CovidDeaths))
# Get the latest updated Covid date by country
covidLastDate <- splitCovid %>% 
  group_by(`Country/Region`) %>% 
  filter(`Covid Last Update` == max(`Covid Last Update`)) %>% 
  select(-c(`Covid First Observations`, `CovidConfirmedCases`, `CovidDeaths`))
# Get the earliest Covid date by country
covidFirstDate <- splitCovid %>% 
  group_by(`Country/Region`) %>% 
  filter(`Covid First Observations` == min(`Covid First Observations`)) %>% 
  select(-c(`Covid Last Update`, `CovidConfirmedCases`, `CovidDeaths`))
# Combine both dates with the covidCases while removing all duplicate rows
covidSet <- covidFirstDate %>% 
  group_by(`Country/Region`, `Covid First Observations`) %>% 
  right_join(covidCases, by="Country/Region") %>% 
  left_join(covidLastDate, by="Country/Region") %>% 
  distinct()
# Combine the covidSet with the Sars dataset
combinedSet <- left_join(sars, covidSet)

Joining, by = "Country/Region"

str(combinedSet)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   29 obs. of  10 variables:
 $ Country/Region               : chr  "Australia" "Canada" "Mainland China" "Hong Kong" ...
 $ Sars Total Cases             : num  6 251 5327 1755 1 ...
 $ Sars Deaths                  : num  0 43 349 299 0 37 1 0 0 0 ...
 $ Sars Fatality Ratio (%)      : num  0 17 7 17 0 11 14 0 0 0 ...
 $ Sars First Probable Case Date: chr  "26/02/2003" "23/02/2003" "16/11/2002" "15/02/2003" ...
 $ Sars Last Probable Case Date : chr  "01/04/2003" "12/06/2003" "03/06/2003" "31/05/2003" ...
 $ Covid First Observations     : chr  "01/02/2020" "01/02/2020" "01/02/2020" "01/02/2020" ...
 $ sum(CovidConfirmedCases)     : num  370129 2464766 8386881 61059 3165 ...
 $ sum(CovidDeaths)             : num  4055 146966 361112 346 0 ...
 $ Covid Last Update            : POSIXct, format: "2020-05-22" "2020-05-22" "2020-05-22" ...

head(combinedSet)

Data cleaning II

Once the datasets have been successfully merged, it is time to clean up the new dataset. This step entails the renaming and reordering of columns for consistency in the structure of the dataset. Appropriate data type conversions will also be performed, converting the “Country/Region” column to a factor data type, and also converting the date columns appropriately using lubridate functions.

# Factor data type conversion
combinedSet$`Country/Region` <- factor(combinedSet$`Country/Region`, levels = c("Australia", "Canada", "France", "Germany", "Hong Kong", "India", "Indonesia", "Italy", "Kuwait", "Macau", "Mainland China", "Malaysia", "Mongolia", "New Zealand", "Philippines", "Republic of Ireland", "Romania", "Russia", "Singapore", "South Africa", "South Korea", "Spain", "Sweden", "Switzerland", "Taiwan", "Thailand", "United Kingdom", "United States", "Vietnam"), labels = c("Australia", "Canada", "France", "Germany", "Hong Kong", "India", "Indonesia", "Italy", "Kuwait", "Macau", "Mainland China", "Malaysia", "Mongolia", "New Zealand", "Philippines", "Republic of Ireland", "Romania", "Russia", "Singapore", "South Africa", "South Korea", "Spain", "Sweden", "Switzerland", "Taiwan", "Thailand", "United Kingdom", "United States", "Vietnam"), ordered = FALSE)
# Name and structure cleaning
names(combinedSet)[7] <- "Covid First Probable Case Date"
names(combinedSet)[8] <- "Covid Total Cases"
names(combinedSet)[9] <- "Covid Deaths"
names(combinedSet)[10] <- "Covid Last Updated Case Date"
combinedSet$`Covid Last Updated Case Date` <- strftime(combinedSet$`Covid Last Updated Case Date`, "%d/%m/%Y")
col_order <- c("Country/Region", "Sars First Probable Case Date", "Sars Last Probable Case Date", "Sars Total Cases", "Sars Deaths", "Sars Fatality Ratio (%)", "Covid First Probable Case Date", "Covid Last Updated Case Date", "Covid Total Cases", "Covid Deaths")
cleanSet <- combinedSet[, col_order]
# Date data type conversion
cleanSet$`Sars First Probable Case Date` <- dmy(cleanSet$`Sars First Probable Case Date`)
cleanSet$`Sars Last Probable Case Date` <- dmy(cleanSet$`Sars Last Probable Case Date`)
cleanSet$`Covid First Probable Case Date` <- dmy(cleanSet$`Covid First Probable Case Date`)
cleanSet$`Covid Last Updated Case Date` <- dmy(cleanSet$`Covid Last Updated Case Date`)
str(cleanSet)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   29 obs. of  10 variables:
 $ Country/Region                : Factor w/ 29 levels "Australia","Canada",..: 1 2 11 5 10 25 3 4 6 7 ...
 $ Sars First Probable Case Date : Date, format: "2003-02-26" "2003-02-23" "2002-11-16" ...
 $ Sars Last Probable Case Date  : Date, format: "2003-04-01" "2003-06-12" "2003-06-03" ...
 $ Sars Total Cases              : num  6 251 5327 1755 1 ...
 $ Sars Deaths                   : num  0 43 349 299 0 37 1 0 0 0 ...
 $ Sars Fatality Ratio (%)       : num  0 17 7 17 0 11 14 0 0 0 ...
 $ Covid First Probable Case Date: Date, format: "2020-02-01" "2020-02-01" "2020-02-01" ...
 $ Covid Last Updated Case Date  : Date, format: "2020-05-22" "2020-05-22" "2020-05-22" ...
 $ Covid Total Cases             : num  370129 2464766 8386881 61059 3165 ...
 $ Covid Deaths                  : num  4055 146966 361112 346 0 ...

head(cleanSet)

Tidy & Manipulate Data II

Once we have a cleaned set that has both Sars and Covid cases; referring to the original goals of the project, one of which was to find the deadlier pandemic. This requires a mutation of a Covid Fatality Ratio variable so it can be compared to the Sars Fatality Ratio (%) variable (which is already in the dataset). To achieve this, a mutation where the Covid Deaths are divided by Covid Total Cases and multiplied by 100 in order to attain the ratio value in percentage - exactly like the Sars Fatality Ratio (%) variable.

# Mutating the Covid Fatality Ratio (%) variable by using the following formula and rounding the answer
cleanSet %<>% mutate(`Covid Fatality Ratio (%)` = round((`Covid Deaths`/`Covid Total Cases`*100)))
head(cleanSet)

Scan I

Once the dataset is clean and tidy, we can then move on to scanning for missing or special values. To scan for missing values, using the print(colSums(is.na())) function is very helpful. For scanning special values, a function was made that checks every numerical column if they possess infinite or NaN values called valueChecker. This was then applied to the cleaned dataset. The results showed that the dataset contains no missing values or special values.

# Scanning for missing values
print(colSums(is.na(cleanSet)))

                Country/Region  Sars First Probable Case Date   Sars Last Probable Case Date 
                             0                              0                              0 
              Sars Total Cases                    Sars Deaths        Sars Fatality Ratio (%) 
                             0                              0                              0 
Covid First Probable Case Date   Covid Last Updated Case Date              Covid Total Cases 
                             0                              0                              0 
                  Covid Deaths       Covid Fatality Ratio (%) 
                             0                              0 

# Scanning for special values
specialValueChecker <- function(x) {
  if(is.numeric(x)) (is.infinite(x) | is.nan(x))
}
sapply(cleanSet, specialValueChecker)

$`Country/Region`
NULL

$`Sars First Probable Case Date`
NULL

$`Sars Last Probable Case Date`
NULL

$`Sars Total Cases`
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[18] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

$`Sars Deaths`
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[18] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

$`Sars Fatality Ratio (%)`
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[18] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

$`Covid First Probable Case Date`
NULL

$`Covid Last Updated Case Date`
NULL

$`Covid Total Cases`
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[18] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

$`Covid Deaths`
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[18] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

$`Covid Fatality Ratio (%)`
 [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
[18] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

Scan II

Once the dataset has been scanned for missing and special values, the next step would be to scan the numeric data for outliers. To achieve this, Tukey’s method of outlier detection was used. This was performed by plotting boxplots for all numeric variables in the dataset, while printing the value of the outliers found in the boxplots. From the boxplots, it can be seen that there are a number of suggested outliers for each variable. While these anomalies may be problematic for statistical analysis - however, none will be performed for this project. Additionally, these anomalies represent important values that are needed in reaching the goal of this project.

# Boxplots to identify outliers
sarsCase_outliers <- cleanSet$`Sars Total Cases` %>% boxplot(main = "Box Plot of Sars Cases", ylab = "Total Cases")

covidCase_outliers <- cleanSet$`Covid Total Cases` %>% boxplot(main = "Box Plot of Covid Cases", ylab = "Total Cases")

sarsDeaths_outliers <- cleanSet$`Sars Deaths` %>% boxplot(main = "Box Plot of Sars Deaths", ylab = "Total Cases")

covidDeaths_outliers <- cleanSet$`Covid Deaths` %>% boxplot(main = "Box Plot of Covid Deaths", ylab = "Total Cases")

sarsFatality_outliers <- cleanSet$`Sars Fatality Ratio (%)` %>% boxplot(main = "Box plot of Sars Fatality Ratio", ylab = "Total Cases")

covidFatality_outliers <- cleanSet$`Covid Fatality Ratio (%)` %>% boxplot(main = "Box plot of Covid Fatality Ratio", ylab = "Total Cases")

# Outlier values
sarsCase_outliers$out

[1]  251 5327 1755  346  238   63

covidCase_outliers$out

[1] 48882141

sarsDeaths_outliers$out

[1]  43 349 299  37  33

covidDeaths_outliers$out

[1] 1067943 1404119 1161985 1083065 2671622

sarsFatality_outliers$out

[1]  40 100

covidFatality_outliers$out

[1] 14

Transform

From the section above, we can see that the Box plot of the Covid Fatality Ratio variable is the most suitable variable compared to the rest for an attempt at a data transformation. This will be performed by firstly plotting a histogram of the variable to get a visualization on what the distribution looks like. Then, we attempt to normalise the variable by using the z score standardisation. The resulting histogram does not reflect that of a normal distribution, instead it produced a cleaner U-shaped distribution.

# Histogram of the Covid Fatality Ratio variable
cleanSet$`Covid Fatality Ratio (%)` %>% hist(main = "Histogram of the Covid Fatality Ratio", xlab = "Fatality Ratio")

# Attempt to normalize the Covid Fatality Ratio variable using z score standardisation
zScoreStd_Test <- scale(cleanSet$`Covid Fatality Ratio (%)`, center = TRUE, scale = TRUE)
hist(zScoreStd_Test)

Final results

This final table represents the results of this project’s goal. From this table, while there are far more Covid cases and Covid deaths than SARS, the latter is still the more fatal pandemic. It seems that Covid-19 has a higher infection rate compared to SARS, however - the ratio of patients dying once infected is higher for SARS. This project could be further extended by adding the datasets of other pandemics in order to create an informational pandemic dataset.

head(cleanSet, n = 29)