title: '

The Exercises of Week 1
' author: '
Reza Mohammadi
' date: '
r Sys.Date()
' output: htmldocument: numbersections: true figcaption: true toc: true figwidth: 7 figheight: 5 theme: cosmo highlight: tango codefolding: show ---

{r setup, include = FALSE} knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE, comment = " ", error = FALSE, fig.align = 'center')

Full Name: [Lisette de Vries | 1447]{style="color: purple;"}

Online Exercises on DataCamp: Please enroll in the free online course "Data Analytics 2024" on DataCamp. There are three online assignments to prepare you for the computer labs. Please note that the online-assignments at the DataCamp are not mandatory.

Assignment: Respond to the questions in this R-markdown file. This exercise is worth 100 points, with an additional Bonus section offering 30 extra points. Please upload both your R-markdown (.Rmd)and HTML files to Canvas. Ensure your R-markdown is formatted correctly.

How to use R-Markdown in RStudio? {-}

You're viewing an R Markdown document. Markdown offers a straightforward syntax to create documents that can be formatted into HTML, PDF, and MS Word. Dive deeper into R Markdown by visiting R Markdown's official page.

For a hands-on demonstration of using R Markdown in RStudio, watch this video tutorial.

Retrieve the 'Exercises_week1.Rmd' file from Canvas > Assignments > Exercises - Week 1 and download it to your computer. Once downloaded, open this Rmd file in RStudio. To generate its HTML version, navigate to the Workspace Tab and select Knit > Knit to HTML.

Loading/Attaching R packages {-}

At times, the function or dataset you need isn't natively available in R. When this happens, you'll need to install and then load the specific package containing the required function or dataset. Remember, you only need to install a package once. In subsequent uses of R, you'll just need to load it. As an example, let's consider the need for the following R packages:

  • ggplot2: This package offers robust data visualization capabilities. Besides, the diamonds dataset is available in this package.
  • plyr: Needed for the mutate() function.
  • liver: Contains the adult dataset.
  • Hmisc: Useful for handling missing values.
  • forcats: Needed for the fct_collapse() function.
  • naniar: For visualizing missing values.

To use these packages in R, you'd load them as follows: {r message = FALSE, warning = FALSE} library(ggplot2) # For the diamonds dataset and the ggplot function library(plyr) # For the 'mutate' function library(liver) # For the adult dataset library(Hmisc) # For missing values library(forcats) # For the "fct_collapse" function library(naniar) # For visualizing missing values

IMPORTANT: If you haven't installed these packages yet, you'll need to do so first.

To install a package, head over to RStudio and click on the "Tools" tab. From there, select "Install Packages...". In the ensuing dialog, enter the name of the package you wish to install into the Packages field and hit the install button. Make sure to check the "Install dependencies" option.

Your task is to ensure these packages are installed on your computer.

Diamonds Dataset (50 points)

In this section, we'll utilize the diamonds dataset to demonstrate handling unusual values and missing values. The diamonds dataset is available in the ggplot2 package in R.

While datasets can typically be imported into R from a personal computer or an online source using the read.csv() function, for our purpose, the diamonds dataset is available in the ggplot2 package. So, we can upload this dataset into our RStudio environment by running the following command: {r} data(diamonds) # This loads the "diamonds" dataset into your RStudio environment

To get a grasp of the dataset's structure in R, several functions can be employed:

  • str() offers a concise overview of the dataset's structure.
  • View() presents the data in a spreadsheet format. Do remember that R distinguishes between lowercase and uppercase letters.
  • head() displays the initial segment of the data (typically the first six rows).
  • summary() provides an aggregated view of each variable.

For our immediate needs, we'll utilize the str() function to outline the diamonds dataset: {r} str(diamonds)

From the output, we can discern that the dataset comprises r nrow(diamonds) entries spanning across r ncol(diamonds) distinct variables:

  • carat: Represents the diamond's weight (ranging from 0.2 to 5.01).
  • cut: Indicates the quality of the cut (categories: Fair, Good, Very Good, Premium, Ideal).
  • color: Grades the diamond color, from D (lowest grade)to J (highest grade).
  • clarity: Measures the diamond's clarity (ranked from I1 (lowest), SI2, SI1, VS2, VS1, VVS2, VVS1, to IF (highest)).
  • depth: Specifies the total depth percentage, calculated as z / mean(x, y)= 2 * z / (x + y).
  • table: The width of the diamond's top in relation to its broadest point (ranges between 43 and 95).
  • price: Quotes the price in US dollars (from \$326 to \$18,823).
  • x: Denotes the length in mm (0 to 10.74).
  • y: Specifies the width in mm (0 to 58.9).
  • z: Represents the depth in mm (0 to 31.8).

To view the initial segment of the data: {r} head(diamonds)

Further insights related to this dataset can be found here.

a. Which variables within the dataset are considered nominal? Which are deemed ordinal? And which are classified as numerical? There are no nominal variabbles. Cut, color and clarity are considered ordinal values. Carat, depth, table, price, x, y and z are considered numerical b. Use the summary() function to furnish a comprehensive summary of the diamonds dataset. {r} summary(diamonds) c. Determine and report the average, lowest, and highest price of the diamonds. ```{r}

Calculate the average price

average_price <- mean(diamonds$price)

Calculate the lowest price

lowest_price <- min(diamonds$price)

Calculate the highest price

highest_price <- max(diamonds$price)

Print the results

cat("Average Price:", averageprice, "\n") cat("Lowest Price:", lowestprice, "\n") cat("Highest Price:", highest_price, "\n") ```

Outliers represent data points that deviate significantly from the overall pattern of the dataset. They can sometimes result from data entry mistakes, but in other instances, they can offer valuable insights. When sifting through a large volume of data, outliers can often become obscured in visual representations like histograms. As an illustration, consider the distribution of the y variable within the diamonds dataset. The presence of outliers might only be hinted at by the unusually expanded y-axis scale:

{r fig.align = 'center'} ggplot(data = diamonds) + geom_histogram(aes(x = y), binwidth = 0.5)

If you are curious about how to create these types of visualizations in R, you can find more information here; This is the topic of next week. Given that the majority of observations cluster around common intervals, the less frequent ones might be less evident (though there's a noticeable minor bin at 0). To highlight these atypical values, we can zoom-in to a narrower range on the y-axis:

{r fig.align = 'center'} ggplot(data = diamonds) + geom_histogram(mapping = aes(x = y), binwidth = 0.5) + coord_cartesian(ylim = c(0, 30))

We also report the scatter plot for variable y vs price: {r fig.align = 'center'} ggplot(data = diamonds) + geom_point( mapping = aes(x = y, y = price), colour = 'blue')

The y variable measures one of the three dimensions of these diamonds, in mm. We know that diamonds can’t have a width of 0mm, so these values must be incorrect. We might also suspect that measurements of 32mm and 59mm are implausible: those diamonds are over an inch long, but don’t cost hundreds of thousands of dollars!

It’s good practice to repeat your analysis with and without the outliers. If they have minimal effect on the results, and you can’t figure out why they’re there, it’s reasonable to replace them with missing values, and move on. However, if they have a substantial effect on your results, you shouldn’t drop them without justification. You’ll need to figure out what caused them (e.g. a data entry error)and disclose that you removed them in your write-up.

I recommend replacing the unusual values with missing values. The easiest way to do this is to use mutate() to replace the variable with a modified copy. You can use the ifelse() function to replace unusual values with NA:

{r} diamonds_2 = mutate(diamonds, y = ifelse(y == 0 | y > 30, NA, y))

ifelse()has three arguments; The first argument test should be a logical vector. The result will contain the value of the second argument, NA, when test is TRUE, and the value of the third argument, y, when it is FALSE.

{r fig.align = 'center'} ggplot(data = diamonds_2, mapping = aes(x = y, y = price)) + geom_point(colour = 'blue')

To see the summary of the variable y with missing values: {r} summary(diamonds_2$y)

We see that in total r sum(is.na(diamonds_2$y)) unusual values are replaced by NA values.

Dealing with missing values

Here we impute the missing values with random values (which are proportional to categories' records)by using the function impute() from R package Hmisc as follows: {r} diamonds_2$y = impute(diamonds_2$y, 'random')

Note that we can replace the missing values by e.g. the mean of the data by replacing random with mean in in the impute() function. If you want to know more about the impute() function, just type ?impute in the Console of your RStudio.

To see the summary of the variable y: {r} summary(diamonds_2$y) You see that the outliers are imputed.

Unusual Values in the variables x and z

Here, in the diamonds dataset, we want to check if there is any unusual or outliers in the variables x and z.

Your task here is to check if there is any unusual or outliers in the variables x and z. If so, handle the unusual/outliers values. Follow the same steps as we did for the variable y in the previous sub-section. There seem to be some unusual or outliers in the variables

Adult Dataset (50 points)

We want to find out which type of people can earn more than 50K per year. Thus, the prediction task is to determine whether a person makes over $50K a year. For this classification task, in week 7, we will apply the Decision Tree as well as the Random Forest algorithm.

Data Understanding

You could find more information about this dataset here: https://rdrr.io/cran/liver/man/adult.html.

The adult dataset is from the US Census Bureauwith the primary task to predict whether a given adult makes more than $50K a year based on attributes such as education, hours of work per week, etc. The target feature is income with two levels "<=50K" and ">50K", and the remaining 14 variables are predictors.

The adult dataset, as a data frame, contains 48598 records (rows)with 15 variables/features (columns):

  • age: age in years (numerical).
  • workclass: a factor with 6 levels (categorical-nominal).
  • demogweight: the demographics to describe a person (categorical-nominal).
  • education: a factor with 16 levels (categorical-nominal).
  • education.num: number of years of education (numerical-discrete).
  • marital.status: a factor with 5 levels (categorical-nominal).
  • occupation: a factor with 15 levels (categorical-nominal).
  • relationship: a factor with 6 levels (categorical-nominal).
  • race: a factor with 5 levels (categorical-nominal).
  • gender: a factor with levels "Female","Male" (categorical-binary).
  • capital.gain: capital gains (numerical-discrete).
  • capital.loss: capital losses (numerical-discrete).
  • hours.per.week: number of hours of work per week (numerical-discrete).
  • native.country: a factor with 42 levels (categorical-nominal).
  • income: yearly income as a factor with levels "<=50K" and ">50K" (categorical-binary).

You can find more information related to this dataset at:

https://www.rdocumentation.org/packages/liver/versions/1.3/topics/adult

We import the dataset and report the structure of the dataset:

```{r} data(adult)

str(adult) ```

It shows the dataset contains r nrow(adult) records and r ncol(adult) variables/features. The dataset has r ncol(adult) - 1 predictors along with a target variable income as a binary variable with two levels "<=50K" and ">50K".

a. Which of the variables in the dataset are nominal? And which ones are numerical? Nominal: workclass, demogweight, education, marital.status, occupation, relationship, race, native.country Numerical: age, education.num, gender, income

b. Report the summary of the adult dataset by using function summary(). {r} summary(adult) c. Report the proportion of the people who earn more than 50K (>50K). {r} proportion_over_50K <- mean(adult$income == ">50K") print(proportion_over_50K)

Data Cleaning

In the dataset, some of the categorical variables (native.county and workclass) have a lot of categories. Here, we reduce the number of factors.

Using table() for checking the frequency of the native country column. {r} table(adult$native.country)

We group these countries together into continents by using function fct_collapse(): ```{r} Europe = c("England", "France", "Germany", "Greece", "Holand-Netherlands", "Hungary", "Ireland", "Italy", "Poland", "Portugal", "Scotland", "Yugoslavia")

Asia = c("China", "Hong", "India", "Iran", "Cambodia", "Japan", "Laos", "Philippines", "Vietnam", "Taiwan", "Thailand")

N.America = c("Canada", "United-States", "Puerto-Rico")

S.America = c("Columbia", "Cuba", "Dominican-Republic", "Ecuador", "El-Salvador", "Guatemala", "Haiti", "Honduras", "Mexico", "Nicaragua", "Outlying-US(Guam-USVI-etc)", "Peru", "Jamaica", "Trinadad&Tobago")

Other = c("South")

adult$native.country = fctcollapse(adult$native.country, "Europe" = Europe ) adult$native.country = fctcollapse(adult$native.country, "Asia" = Asia ) adult$native.country = fctcollapse(adult$native.country, "N.America" = N.America) adult$native.country = fctcollapse(adult$native.country, "S.America" = S.America) adult$native.country = fct_collapse(adult$native.country, "Other" = Other ) ```

Using table() for checking the frequency of the native country column. {r} table(adult$native.country)

For the variable workclass, we report the frequency of workclass as {r} table(adult$workclass)

We reduce two categories "Never-worked" and "Without-pay" to "Unemployed" as follows: {r} adult$workclass = fct_collapse(adult$workclass, "Unemployed" = c("Never-worked", "Without-pay"))

Missing Values

We can see from the output of the summary() function that variable workclass has a category of ? with r sum(adult$workclass=="?") records. The variable native.country also has a category of ? with r sum(adult$native.country=="?") records. Since these ? are the missing values in the dataset, we convert them to NA values as follows: {r message = FALSE } adult[ adult == "?" ] = NA

To remove the category "?" {r message = FALSE } adult$workclass = factor(adult$workclass , levels = levels(adult$workclass )[-1]) adult$native.country = factor(adult$native.country, levels = levels(adult$native.country)[-1]) adult$occupation = factor(adult$occupation , levels = levels(adult$occupation )[-1])

To visualize the missing values {r fig.align = 'center'} gg_miss_var(adult, show_pct = TRUE)

The above plot shows that the variables workclass, occupation, and native.country have missing values. It shows that the variables workclass, occupation, and native.country have around less than 0.06 percent missing values and the variable native.country has around 0.02 percent missing values.

Here we impute the missing values with random values (which is proportional to categories' records)by using the function impute() from R package Hmisc as follows: {r} # impute missing values with random value adult$workclass = impute(adult$workclass , 'random') adult$native.country = impute(adult$native.country, 'random') adult$occupation = impute(adult$occupation , 'random')

The below plot shows that all the missing values are imputed. {r fig.align = 'center'} gg_miss_var(adult, show_pct = TRUE)

The impute() function imputes missing values using a user-defined statistical method (e.g. mean, median, max). Its default is median. For more advanced approaches, users could apply the aregImpute() function (from the R package Hmisc)which provides mean imputation using additive regression, bootstrapping, and predictive mean matching.

We also can remove NA values from the adult dataset by using na.omit(). Note, we normally should not remove the missing values. It depends on the situation whether it's a good idea or not. You shouldn't always just drop NA values.

Outliers for variable capital.loss

Here we want to check if there are any outliers in the variable capital.loss or not. For this, first, we report the summary of the variable capital.loss as follow: {r} summary(adult$capital.loss)

We also report the boxplot and histogram of the variable capital.loss as follow: {r fig.align = 'center'} ggplot(data = adult, aes(y = capital.loss)) + geom_boxplot()

{r fig.align = 'center'} ggplot(data = adult, aes(x = capital.loss)) + geom_histogram(bins = 30, color = "blue", fill = "lightblue")

What would be your interpretation of the above outputs for the variable capital.loss? Are there any outliers in the variable capital.loss? If so, what would be a good strategy to deal with the outliers?

Outliers for variable capital.gain

Here we want to check if there is any outliers in the variable capital.gain.

a. Similar to the previous sub-section report the summary, boxplot, and histogram for the variable capital.gain. {r fig.align = 'center'} ggplot(data = adult) + geom_histogram(mapping = aes(x = capital.gain), binwidth = 0.5) + coord_cartesian(ylim = c(0, 30))

{r fig.align = 'center'} ggplot(data = adult) + geom_point( mapping = aes(x = y, y = capital.gain), colour = 'blue')

{r fig.align = 'center'} ggplot(data = adult, mapping = aes(x = y, y = capital.gain)) + geom_point(colour = 'blue')

b. What would be your interpretation of the above outputs for the variable capital.gain? Are there any outliers in the variable capital.gain? If so, what would be a good strategy to deal with the outliers?

Bonus: Main project in Data Science (30 points)

If you do not have your own dataset, do not worry, you can find a suitable public datasets in the follow links:

  • Kaggle: https://www.kaggle.com
  • UCL Machine Learning Repository: https://archive.ics.uci.edu/ml/index.php

In the Kaggle website you there are a lot of project examples with read-world datasets for Data Science.

Research/Business Understaning Stage (15 points)

For data-driven business-making, it is vital to understand the problem to be solved. This may seem obvious, but research/business projects seldom come pre-packaged as clear and unambiguous data mining problems.

The Research/Business Understanding stage represents a key part of the craft where the analysts' creativity plays a large role. Data Scientist has some things to say, as we will describe, but often the key to a great success is a creative problem formulation by some analyst regarding how to cast the business problem as one or more data science problems. High-level knowledge of the fundamentals helps creative business analysts see novel formulations. The Business Understanding stage can be done by following steps:

  1. First, clearly enunciate the project objectives and requirements in terms of the business or research unit as a whole.
  2. Then, translate these goals and restrictions into the formulation of a problem that can be solved using data science.
  3. Finally, prepare a preliminary strategy for achieving these objectives.

For this part, your task is to write a research/business problem that you want to address in your main project. Write the problem done by following the above three steps for the Business Understanding stage and discuss it with your teammate.

Selected dataset is from Kaggle and it is the retail sales dataset. 1. Objective of the analysis is to determine how customer age and gender influences the purchasing behaviour of the customer. For this it is necessary to have insights in the gender and age of the customer, as well as granular data providing insights in product groups. 2. Does the age and gender of a customer influence the purchase behaviour and what is the impact on monetary value of this 3. preliminary strategy is to first analyse an create customer groups based on both gender and age groups, afterwards categorizing the product categories over the groups. Then, I would like to analyze what the variances are between the customer groups to gain insights from thid.

Data Preparation Stage (15 points)

Raw data from data repositories are seldom ready for the algorithms straight out of the box. Instead, it needs to be cleaned or "prepared for analysis".

  • Check for the type of variables in the dataset. Nominal values: CustomerID, Gender, Product category Ordinal values:Date Numerical values: transaction ID, age, Quantity, Price per Unit, Total Amount
  • Check for missing values and determine what to do about them. No missing vallues identified
  • Identifying outliers and determining what to do about them.

For the data understanding part:

  • Report some important information about the dataset. For example, how and where the dataset is collected.
  • Report a summary of the dataset.

{r} summary(retail_sales_dataset)