MATH2349 (Data Preprocessing) Semester 1, 2018

Assignment 1

Setup

library(stringr)
library(readr)
library(knitr) 
library(dplyr)

setwd("C:\\Users\\Parikshit\\Documents\\Master Of Analytics\\Semester 3\\Data Preprocessing")

Data Description

The data used for this assignment has been downloaded from https://catalog.data.gov/dataset/drug-poisoning-mortality-by-state-united-states

The dataset describes drug poisoning death rates in the US at the state level and by selected demographic characteristics.

The crude death rate and the age adjusted death rates are the death rates per 100,000 U.S. Standard Population for 2000.

Both the crude death rate and age adjusted death rate have a lower and an upper confidence limit which capture the uncertainty in the estimates of the death rates.

State crude rate in Range is the interval range of crude death rate of that respective state

The rates are calculated based on multiple cause-of-death mortality data from National Vital Statistics [Prevention, C & Statistics, N 2018, “Drug Poisoning Mortality by State: United States - Data.gov”, Catalog.data.gov, viewed 24 March, 2018, https://catalog.data.gov/dataset/drug-poisoning-mortality-by-state-united-states.].

Read/Import Data

#Reading in the drug dataset at the same time trimming leading and trailing
#whitespaces
drug<-read.csv("Drug_Poisoning_Mortality_by_State__United_States.csv",
               strip.white = TRUE)

drug[drug$State=="California",] #printing out observations from the state of California

drug_tail_10<-tail(drug,10) #Subsetting out the last 10 observations of the drug dataset and storing it as a dataframe named drug_tail_10

• The drug dataset has been read using the read.csv() function from the base R package
• I have subsetted and listed out the observations from the state of California
• The last 10 observations of the drug dataframe has been subsetted (using the tail() function)and stored in a dataframe named drug_tail_10

Inspect and Understand

dim(drug)[1] #This gives the number of rows in the drug dataframe: 2862
dim(drug)[2] #This gives the number of columns in the drug dataframe: 18

str(drug) #Chekcing the structure of the drug dataframe

##Rearranging the columns of the dataframe
drug<-drug%>%select(1:5,16,6:15,17,18)


##Renaming columns of the drug dataframe
names(drug)[4]<-"Age_Group"
names(drug)[5]<-"Race_Origin"
names(drug)[6]<-"ST_CR_Range"
names(drug)[9]<-"CR"
names(drug)[10]<-"SE_CR"
names(drug)[11]<-"LCL_CR"
names(drug)[12]<-"UCL_CR"
names(drug)[13]<-"AACR"
names(drug)[14]<-"SE_AACR"
names(drug)[15]<-"LCL_AACR"
names(drug)[16]<-"UCL_AACR"
names(drug)[17]<-"US_CR"
names(drug)[18]<-"US_AACR"


##Setting appropriate data types to the variables
drug$State<-as.character(drug$State)

drug$Year<-as.numeric(drug$Year)

drug$Sex<-factor(drug$Sex,levels=c("Male","Female","Both Sexes"))

drug$Age_Group<-factor(drug$Age_Group,levels=c("0-14","15-24","25-34",
                                                   "35-44","45-54","55-64",
                                                   "65-74","75+","All Ages"))

drug$Race_Origin<-factor(drug$Race_Origin,levels=
                                         c("Hispanic","Non-Hispanic Black",
                                          "Non-Hispanic White","All Races-All Origins"))

drug$ST_CR_Range<-as.character(drug$ST_CR_Range)

##Converting column number 7 to column 18 to a numeric data type
for(i in 7:18){
  drug[,i]<-as.numeric(drug[,i])
} 


##Printing the levels of the factor variables and range of numeric variables

sapply(drug,function(x){
  if(is.factor(x)){
    return(levels(x))
  }else if(is.numeric(x)){
    return(range(x,na.rm=TRUE))
  }
})

#####Complete Cases and Missing Values in the Drug dataframe

miss_val<-data.frame(sapply(drug,function(x){missing_no<-sum(is.na(x))}))

names(miss_val)[1]<-"# Missing Values"

kable(miss_val,format="html",caption="Number of missing values in each attribute")

NA_no<-complete.cases(drug)

sum(NA_no) #Number of rows that have no missing values in the entire row sequence: 1134

The drug data set has been inspected and preprocessed on the following lines:

• Dimensions of the data has been checked
• Structure of the data frame has been checked
• Leading and trailing blanks (if any) have been removed from every column
• The columns of the drug dataframe has been rearranged
• Suitable renaming of the attributes
• Setting appropriate data types to the variables
  • Levels of the factor variables have been arranged
• Printing out the levels of all the factor variables and ranges(min and max value) of the numeric variables
• Inspecting missing values and complete cases

With regards to the naming conventions of the columns of the drug dataframe:

• Age.Group has been renamed to Age_Group
• Race.and.Hispanic.Origin has been renamed to Race_Origin
• State.Crude.Rate.in.Range has been renamed to ST_CR_Range
• Crude.Death.Rate has been renamed to CR
• Age-adjusted has been renamed to AACR
• Standard.Error.for.Crude.Rate has been renamed to SE_CR
• Lower.Confidence.Limit.for.Crude.Rate has been renamed to LCL_CR
• Upper.Confidence.Limit.for.Crude.Rate has been renamed to UCL_CR
• US.Crude.Rate has been renamed to US_CR
• US.Age-adjusted.Rate has been renamed to US_AACR

With regards to rearrangment in columns, State.Crude.Rate.in.Range has been brought after the Race.and.Hispanic.Origin column.

Data types of the attributes of the drug dataset has been set as follows:

• State and ST_CR_Range has been set to a Character data type
• Year has been set to a Numeric data type
• Sex, Age_Group and Race_Origin has been type casted to a Factor data type
• All of the other columns has been type casted to a Numeric data type

Subsetting I

Now, I subset the first 10 rows of the drug dataframe and create a matrix out of it.

Since a matrix data structure in R can hold only atomic elements (all elements being of the same data type), all of the 10 observations across the attributes of the drug dataset are coerced to a character data type which is evident when we execute the code print(drug_subset_matrix[1,])

drug_subset<-drug[1:10,] #Subsetting the first 10 rows of the drug dataframe

drug_subset_matrix<-as.matrix(drug_subset) #creating a matrix out of the subsetted data

print(drug_subset_matrix[1,])#Printing out the first row of the matrix

Subsetting II

Now, I subset the original drug dataframe by retaining only the first and the last column.

I save it as an R object. The reason for doing this is to free up some memory space and to load the data frame (in future, if needed) into memory in an as-is state without loss of information.

drug_subset_2<-data.frame(drug[,c(1,dim(drug)[2])])
save(drug_subset_2,file="drug_subsetted.Rda")

REFERENCES

Prevention, C & Statistics, N 2018, “Drug Poisoning Mortality by State: United States - Data.gov”, Catalog.data.gov, viewed 24 March, 2018, https://catalog.data.gov/dataset/drug-poisoning-mortality-by-state-united-states.