MATH2349 Semester 2, 2019

Assignment 3

Required packages

library(dplyr)
library(readr)
library(tidyr) 
library(Hmisc) 
library(outliers) 
library(forecast)

Executive Summary

The report is to exhibit various skills that we have achieved in preprocessing the data throughout the semester. Open data sets have been taken for cleaning and transforming them into a tidy format which can be used for analysis. The follwing measures are taken for obtaining the desired data format.

• The csv file data sets are imported in R Studio using readr functions
• Due to the large size of the dat set, they are subsetted and them merged using join() and merge() functions of dplyr package
• The structure of the merged final dataset is obtained for further analysis of the attributes and understand the data types. Data type conversion has been applied to two variables which has been properly factorised, ordered and labelled appropriately.
• The created dataframe is verified whether it is in tidy format or not.
• A new column has been added to the final data set by using mutate().
• The dataframe is scanned for missing values and the NA values are replaced with mean of the value.
• Numeric values are taken for additional evaluation for chekcing the presence of outliers.The outliers found are removem using Tukey’s methor or z score method.
• Data transformation using square and log transformation method has been applied to one of the columns for cteating suitable data for statistical anylsyis purposes.

Data

The data set is a program conducted by “National Health and Nutrition Examination Survey” to assess the diet and medicationss in adults and children in United States.

The dataset the used are downloaded from the below link: https://www.kaggle.com/cdc/national-health-and-nutrition-examination-survey#medications.csv

Three data sets have been selected and merged together to form new data set. demographic.csv diet.csv medication.csv

These data sets are imported using readr function:

DEMOGRAPHICS_INTIAL <- read.csv("C:/Users/Niki/Desktop/DataPreprocessing/Assignment3/national-health-and-nutrition-examination-survey/demographic.csv",stringsAsFactors = FALSE)
head(DEMOGRAPHICS_INTIAL)

DIET_INITIAL <- read.csv("C:/Users/Niki/Desktop/DataPreprocessing/Assignment3/national-health-and-nutrition-examination-survey/diet.csv",stringsAsFactors = FALSE)
head(DIET_INITIAL)

MEDICATIONS_INITIAL <-read.csv("C:/Users/Niki/Desktop/DataPreprocessing/Assignment3/national-health-and-nutrition-examination-survey/medications.csv",stringsAsFactors = FALSE)
head(MEDICATIONS_INITIAL)

Due to the large size of the data set, data is subsetted and the attributes of the subsetted data is explained below:

Attributes of demographic.csv

SEQN - Respondent sequence number SDDSRVYR - Data release cycle RIDSTATR - Interview/Examination status RIAGENDR - Gender RIDAGEYR - Age in years at screening

Attributes of diet.csv

SEQN - Respondent sequence number WTDRD1 - Dietary day one sample weight WTDR2D - Dietary two-day sample weight

Attributes of medication.csv

SEQN - Respondent sequence number RXDUSE - prescription medicine taken in past month RXDDRUG - Generic drug name RXDDRGID - Generic drug code RXDRSD1 - Drug description

DEMOGRAPHICS <-DEMOGRAPHICS_INTIAL[1:100,1:5]
head(DEMOGRAPHICS)

DIET <- DIET_INITIAL[1:100,1:3]
head(DIET)

MEDICATIONS<- MEDICATIONS_INITIAL[1:100,1:5]
head(MEDICATIONS)

The subsetted data is merged using the common attribute SEQN

JOINED_DATA <- DEMOGRAPHICS %>% left_join(DIET,BY="SEQN")

Joining, by = "SEQN"

head(JOINED_DATA)

FINALDATA <-merge(JOINED_DATA,MEDICATIONS, BY="SEQN",ALL=T)
head(FINALDATA)

NA

Understand

The steps for understanding the dtaa are as follows:

• The merged dataset contains different datatypes as its attributes which is obtained using str().
• The intervie status variable RIDSTATR and Gender specific variable RIAGENDR are converted to factor variables with properly ordered and labelled appropriately.

str(FINALDATA)

'data.frame':   100 obs. of  11 variables:
 $ SEQN    : int  73557 73557 73558 73558 73558 73558 73559 73559 73559 73559 ...
 $ SDDSRVYR: int  8 8 8 8 8 8 8 8 8 8 ...
 $ RIDSTATR: int  2 2 2 2 2 2 2 2 2 2 ...
 $ RIAGENDR: int  1 1 1 1 1 1 1 1 1 1 ...
 $ RIDAGEYR: int  69 69 54 54 54 54 72 72 72 72 ...
 $ WTDRD1  : num  16888 16888 17932 17932 17932 ...
 $ WTDR2D  : num  12931 12931 12684 12684 12684 ...
 $ RXDUSE  : int  1 1 1 1 1 1 1 1 1 1 ...
 $ RXDDRUG : chr  "GABAPENTIN" "INSULIN" "GABAPENTIN" "INSULIN GLARGINE" ...
 $ RXDDRGID: chr  "d03182" "d00262" "d03182" "d04538" ...
 $ RXDRSD1 : chr  "Muscle spasm" "Type 2 diabetes mellitus" "Restless legs syndrome" "Type 2 diabetes mellitus" ...

FINALDATA$RIDSTATR = FINALDATA$RIDSTATR %>% factor(levels = c("1","2"),
                                                   labels = c("Interviewed","Interviewed and Examined"),ordered = TRUE)

FINALDATA$RIAGENDR = FINALDATA$RIAGENDR %>% factor(levels = c("1","2"),
                                                   labels = c("Male","Female"),ordered = TRUE)

Tidy & Manipulate Data I

• To obtain analysis on the final dataset is done through str() where the type,range of each variable is shown in the output.
• It is clear from the output that the data is in tidy format and no additional steps are required.

str(FINALDATA)

'data.frame':   100 obs. of  11 variables:
 $ SEQN    : int  73557 73557 73558 73558 73558 73558 73559 73559 73559 73559 ...
 $ SDDSRVYR: int  8 8 8 8 8 8 8 8 8 8 ...
 $ RIDSTATR: Ord.factor w/ 2 levels "Interviewed"<..: 2 2 2 2 2 2 2 2 2 2 ...
 $ RIAGENDR: Ord.factor w/ 2 levels "Male"<"Female": 1 1 1 1 1 1 1 1 1 1 ...
 $ RIDAGEYR: int  69 69 54 54 54 54 72 72 72 72 ...
 $ WTDRD1  : num  16888 16888 17932 17932 17932 ...
 $ WTDR2D  : num  12931 12931 12684 12684 12684 ...
 $ RXDUSE  : int  1 1 1 1 1 1 1 1 1 1 ...
 $ RXDDRUG : chr  "GABAPENTIN" "INSULIN" "GABAPENTIN" "INSULIN GLARGINE" ...
 $ RXDDRGID: chr  "d03182" "d00262" "d03182" "d04538" ...
 $ RXDRSD1 : chr  "Muscle spasm" "Type 2 diabetes mellitus" "Restless legs syndrome" "Type 2 diabetes mellitus" ...

Tidy & Manipulate Data II

• A new column is created “DIFF_IN_FOODINTAKE” using mutate() where difference between dietary intake of day1 (WTDRD1) and dietary intake of day2 (WTDR2D) is taken.

FINALDATA = mutate(FINALDATA, DIFF_IN_FOODINTAKE=(WTDRD1-WTDR2D))
head(FINALDATA)

Scan I

• Missing values present in the data frames have been scanned in the below section
• Missing values are substitued with appropriate values. The missing values in sequence id is replaced with unspecified and data release cycle with the mean of the value

colSums(is.na(FINALDATA))

              SEQN           SDDSRVYR           RIDSTATR           RIAGENDR           RIDAGEYR             WTDRD1 
                 0                  0                  0                  0                  0                  2 
            WTDR2D             RXDUSE            RXDDRUG           RXDDRGID            RXDRSD1 DIFF_IN_FOODINTAKE 
                 5                  0                  0                  0                  0                  5 

FINALDATA$SEQN[is.na(FINALDATA$SEQN)] <-"Unspecified"
FINALDATA$SDDSRVYR <- impute(FINALDATA$SDDSRVYR, fun=mean)
FINALDATA= na.omit(FINALDATA)
head(FINALDATA,10)

Scan II

• The ouliers are plotted using boxplot and z score method.
• The outiers are treated using Tukey’s method and z score method
• Plotted boxplots after the above method treatment for confirming the removal of the outliers.

FINALDATA$WTDRD1 %>% boxplot(main="Boxplot of Dietary - Day 1 Sample weight", ylab="Sample Weight-Day1",col = "grey")

IQRSLOTS <- IQR(FINALDATA$WTDRD1, na.rm = TRUE)
q3SLOTS = quantile(FINALDATA$WTDRD1, .75, na.rm = TRUE)
FINALDATA<- filter(FINALDATA, FINALDATA$WTDRD1 <= 1.5 * IQRSLOTS + q3SLOTS) 
FINALDATA$WTDRD1 %>%  boxplot(main="Boxplot of Dietary - Day 1 Sample weight (No Outliers)", ylab="Sample Weight-Day1", col = "grey")

FINALDATA$WTDR2D %>% boxplot(main="Boxplot of Dietary - Day 2 Sample weight", ylab="Sample Weight-Day2",col = "grey")

Z.SCORE <- FINALDATA$WTDR2D  %>%  scores(type = "z") 
FINALDATA<- FINALDATA[-which( abs(Z.SCORE) >3 ),] 
FINALDATA

IQRDIETOFDAY2 <- IQR(FINALDATA$WTDR2D, na.rm = TRUE) 
Q3WTDR2D=quantile(FINALDATA$WTDR2D, .75, na.rm = TRUE) 
FINALDATA<- filter(FINALDATA, FINALDATA$WTDR2D <= 1.5 * IQRDIETOFDAY2 + Q3WTDR2D)

FINALDATA$WTDR2D  %>%  boxplot(main="Boxplot of Dietary - Day 2 Sample weight (No Outliers)", ylab="Sample Weight-Day2", col = "grey",outline = FALSE)

NA
NA

Transform

• Histogram have been plotted for one of the numeric attribute
• The plot is viewd to be slightly right skewed
• Transformation have been achieved by converting the values into higher power and taking log transformation.

hist(FINALDATA$WTDR2D)

SQRFINALDATA<- FINALDATA$WTDR2D^2 
hist(SQRFINALDATA,main = "Distribution of Diet Intake in Day 2 samples",xlab = "Day 2 intake of diet(square transformation applied)")

SQELOGMERGE<-log(SQRFINALDATA) 
hist(SQELOGMERGE,main = "Distribution of Diet Intake in Day 2 samples after log transformation is applied",xlab="Day 2 intake of diet(log transformation applied on top of square transformation)")