title: '
r Sys.Date()
{r setup, include = FALSE} knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE, comment = " ", error = FALSE, fig.align = 'center')
Full Name: [Lisette de Vries | 1447]{style="color: purple;"}
Online Exercises on DataCamp: Please enroll in the free online course "Data Analytics 2024" on DataCamp. There are three online assignments to prepare you for the computer labs. Please note that the online-assignments at the DataCamp are not mandatory.
Assignment: Respond to the questions in this R-markdown file. This exercise is worth 100 points, with an additional Bonus section offering 30 extra points. Please upload both your R-markdown (.Rmd)and HTML files to Canvas. Ensure your R-markdown is formatted correctly.
You're viewing an R Markdown document. Markdown offers a straightforward syntax to create documents that can be formatted into HTML, PDF, and MS Word. Dive deeper into R Markdown by visiting R Markdown's official page.
For a hands-on demonstration of using R Markdown in RStudio, watch this video tutorial.
Retrieve the 'Exercises_week1.Rmd' file from Canvas >
Assignments > Exercises - Week 1 and download it to your
computer. Once downloaded, open this Rmd file in RStudio. To generate
its HTML version, navigate to the Workspace Tab and select
Knit > Knit to HTML.
At times, the function or dataset you need isn't natively available in R. When this happens, you'll need to install and then load the specific package containing the required function or dataset. Remember, you only need to install a package once. In subsequent uses of R, you'll just need to load it. As an example, let's consider the need for the following R packages:
mutate() function.fct_collapse() function.To use these packages in R, you'd load them as follows:
{r message = FALSE, warning = FALSE} library(ggplot2) # For the diamonds dataset and the ggplot function library(plyr) # For the 'mutate' function library(liver) # For the adult dataset library(Hmisc) # For missing values library(forcats) # For the "fct_collapse" function library(naniar) # For visualizing missing values
IMPORTANT: If you haven't installed these packages yet, you'll need to do so first.
To install a package, head over to RStudio and click on the "Tools" tab. From there, select "Install Packages...". In the ensuing dialog, enter the name of the package you wish to install into the Packages field and hit the install button. Make sure to check the "Install dependencies" option.
Your task is to ensure these packages are installed on your computer.
In this section, we'll utilize the diamonds dataset to demonstrate handling unusual values and missing values. The diamonds dataset is available in the ggplot2 package in R.
While datasets can typically be imported into R from
a personal computer or an online source using the
read.csv() function, for our purpose, the diamonds
dataset is available in the ggplot2 package. So, we can
upload this dataset into our RStudio environment by running the
following command:
{r} data(diamonds) # This loads the "diamonds" dataset into your RStudio environment
To get a grasp of the dataset's structure in R, several functions can be employed:
str() offers a concise overview of the dataset's
structure.View() presents the data in a spreadsheet format. Do
remember that R distinguishes between lowercase and
uppercase letters.head() displays the initial segment of the data
(typically the first six rows).summary() provides an aggregated view of each
variable.For our immediate needs, we'll utilize the str()
function to outline the diamonds dataset:
{r} str(diamonds)
From the output, we can discern that the dataset comprises
r nrow(diamonds) entries spanning across
r ncol(diamonds) distinct variables:
carat: Represents the diamond's weight (ranging from
0.2 to 5.01).cut: Indicates the quality of the cut (categories:
Fair, Good, Very Good, Premium, Ideal).color: Grades the diamond color, from D (lowest
grade)to J (highest grade).clarity: Measures the diamond's clarity (ranked from I1
(lowest), SI2, SI1, VS2, VS1, VVS2, VVS1, to IF (highest)).depth: Specifies the total depth percentage, calculated
as z / mean(x, y)= 2 *
z / (x + y).table: The width of the diamond's top in relation to
its broadest point (ranges between 43 and 95).price: Quotes the price in US dollars (from \$326 to
\$18,823).x: Denotes the length in mm (0 to 10.74).y: Specifies the width in mm (0 to 58.9).z: Represents the depth in mm (0 to 31.8).To view the initial segment of the data:
{r} head(diamonds)
Further insights related to this dataset can be found here.
a. Which variables within the dataset are considered
nominal? Which are deemed ordinal? And which are
classified as numerical? There are no nominal
variabbles. Cut, color and clarity are considered ordinal values. Carat,
depth, table, price, x, y and z are considered numerical b. Use
the summary() function to furnish a comprehensive summary
of the diamonds dataset.
{r} summary(diamonds) c. Determine and report the
average, lowest, and highest price of the diamonds. ```{r}
average_price <- mean(diamonds$price)
lowest_price <- min(diamonds$price)
highest_price <- max(diamonds$price)
cat("Average Price:", averageprice, "\n") cat("Lowest Price:", lowestprice, "\n") cat("Highest Price:", highest_price, "\n") ```
Outliers represent data points that deviate significantly from the
overall pattern of the dataset. They can sometimes result from data
entry mistakes, but in other instances, they can offer valuable
insights. When sifting through a large volume of data, outliers can
often become obscured in visual representations like histograms. As an
illustration, consider the distribution of the y variable
within the diamonds dataset. The presence of outliers might
only be hinted at by the unusually expanded y-axis scale:
{r fig.align = 'center'} ggplot(data = diamonds) + geom_histogram(aes(x = y), binwidth = 0.5)
If you are curious about how to create these types of visualizations in R, you can find more information here; This is the topic of next week. Given that the majority of observations cluster around common intervals, the less frequent ones might be less evident (though there's a noticeable minor bin at 0). To highlight these atypical values, we can zoom-in to a narrower range on the y-axis:
{r fig.align = 'center'} ggplot(data = diamonds) + geom_histogram(mapping = aes(x = y), binwidth = 0.5) + coord_cartesian(ylim = c(0, 30))
We also report the scatter plot for variable y vs
price:
{r fig.align = 'center'} ggplot(data = diamonds) + geom_point( mapping = aes(x = y, y = price), colour = 'blue')
The y variable measures one of the three dimensions of
these diamonds, in mm. We know that diamonds can’t have a width of 0mm,
so these values must be incorrect. We might also suspect that
measurements of 32mm and 59mm are implausible: those diamonds are over
an inch long, but don’t cost hundreds of thousands of dollars!
It’s good practice to repeat your analysis with and without the outliers. If they have minimal effect on the results, and you can’t figure out why they’re there, it’s reasonable to replace them with missing values, and move on. However, if they have a substantial effect on your results, you shouldn’t drop them without justification. You’ll need to figure out what caused them (e.g. a data entry error)and disclose that you removed them in your write-up.
I recommend replacing the unusual values with missing values. The
easiest way to do this is to use mutate() to replace the
variable with a modified copy. You can use the ifelse()
function to replace unusual values with NA:
{r} diamonds_2 = mutate(diamonds, y = ifelse(y == 0 | y > 30, NA, y))
ifelse()has
three arguments; The first argument test should be a logical vector. The
result will contain the value of the second argument, NA,
when test is TRUE, and the value of the third argument,
y, when it is FALSE.
{r fig.align = 'center'} ggplot(data = diamonds_2, mapping = aes(x = y, y = price)) + geom_point(colour = 'blue')
To see the summary of the variable y with missing
values: {r} summary(diamonds_2$y)
We see that in total r sum(is.na(diamonds_2$y)) unusual
values are replaced by NA values.
Here we impute the missing values with random values (which
are proportional to categories' records)by using the function
impute() from R package Hmisc as follows:
{r} diamonds_2$y = impute(diamonds_2$y, 'random')
Note that we can replace the missing values by e.g. the mean
of the data by replacing random with mean in
in the impute() function. If you want to know more about
the impute() function, just type ?impute in
the Console of your RStudio.
To see the summary of the variable y:
{r} summary(diamonds_2$y) You see that the outliers are
imputed.
x and
zHere, in the diamonds dataset, we want to check if there is
any unusual or outliers in the variables x and
z.
Your task here is to check if there is any unusual or
outliers in the variables x and z. If so,
handle the unusual/outliers values. Follow the same steps as we did for
the variable y in the previous sub-section. There
seem to be some unusual or outliers in the variables
We want to find out which type of people can earn more than 50K per year. Thus, the prediction task is to determine whether a person makes over $50K a year. For this classification task, in week 7, we will apply the Decision Tree as well as the Random Forest algorithm.
You could find more information about this dataset here: https://rdrr.io/cran/liver/man/adult.html.
The adult dataset is from the US Census Bureauwith the primary task to predict whether a given adult makes more than $50K a year based on attributes such as education, hours of work per week, etc. The target feature is income with two levels "<=50K" and ">50K", and the remaining 14 variables are predictors.
The adult dataset, as a data frame, contains 48598 records (rows)with 15 variables/features (columns):
age: age in years (numerical).workclass: a factor with 6 levels
(categorical-nominal).demogweight: the demographics to describe a person
(categorical-nominal).education: a factor with 16 levels
(categorical-nominal).education.num: number of years of education
(numerical-discrete).marital.status: a factor with 5 levels
(categorical-nominal).occupation: a factor with 15 levels
(categorical-nominal).relationship: a factor with 6 levels
(categorical-nominal).race: a factor with 5 levels
(categorical-nominal).gender: a factor with levels "Female","Male"
(categorical-binary).capital.gain: capital gains (numerical-discrete).capital.loss: capital losses (numerical-discrete).hours.per.week: number of hours of work per week
(numerical-discrete).native.country: a factor with 42 levels
(categorical-nominal).income: yearly income as a factor with levels
"<=50K" and ">50K" (categorical-binary).You can find more information related to this dataset at:
https://www.rdocumentation.org/packages/liver/versions/1.3/topics/adult
We import the dataset and report the structure of the dataset:
```{r} data(adult)
str(adult) ```
It shows the dataset contains r nrow(adult) records and
r ncol(adult) variables/features. The dataset has
r ncol(adult) - 1 predictors along with a target variable
income as a binary variable with two levels "<=50K" and
">50K".
a. Which of the variables in the dataset are nominal? And which ones are numerical? Nominal: workclass, demogweight, education, marital.status, occupation, relationship, race, native.country Numerical: age, education.num, gender, income
b. Report the summary of the adult dataset by using
function summary().
{r} summary(adult) c. Report the proportion of the
people who earn more than 50K (>50K).
{r} proportion_over_50K <- mean(adult$income == ">50K") print(proportion_over_50K)
In the dataset, some of the categorical variables
(native.county and workclass) have a lot of
categories. Here, we reduce the number of factors.
Using table() for checking the frequency of the native
country column. {r} table(adult$native.country)
We group these countries together into continents by using function
fct_collapse(): ```{r} Europe = c("England", "France",
"Germany", "Greece", "Holand-Netherlands", "Hungary", "Ireland",
"Italy", "Poland", "Portugal", "Scotland", "Yugoslavia")
Asia = c("China", "Hong", "India", "Iran", "Cambodia", "Japan", "Laos", "Philippines", "Vietnam", "Taiwan", "Thailand")
N.America = c("Canada", "United-States", "Puerto-Rico")
S.America = c("Columbia", "Cuba", "Dominican-Republic", "Ecuador", "El-Salvador", "Guatemala", "Haiti", "Honduras", "Mexico", "Nicaragua", "Outlying-US(Guam-USVI-etc)", "Peru", "Jamaica", "Trinadad&Tobago")
Other = c("South")
adult$native.country = fctcollapse(adult$native.country, "Europe" = Europe ) adult$native.country = fctcollapse(adult$native.country, "Asia" = Asia ) adult$native.country = fctcollapse(adult$native.country, "N.America" = N.America) adult$native.country = fctcollapse(adult$native.country, "S.America" = S.America) adult$native.country = fct_collapse(adult$native.country, "Other" = Other ) ```
Using table() for checking the frequency of the native
country column. {r} table(adult$native.country)
For the variable workclass, we report the frequency of
workclass as {r} table(adult$workclass)
We reduce two categories "Never-worked" and "Without-pay" to
"Unemployed" as follows:
{r} adult$workclass = fct_collapse(adult$workclass, "Unemployed" = c("Never-worked", "Without-pay"))
We can see from the output of the summary() function
that variable workclass has a category of
? with r sum(adult$workclass=="?")
records. The variable native.country also has a category of
? with r sum(adult$native.country=="?")
records. Since these ? are the missing values in the
dataset, we convert them to NA values as follows:
{r message = FALSE } adult[ adult == "?" ] = NA
To remove the category "?"
{r message = FALSE } adult$workclass = factor(adult$workclass , levels = levels(adult$workclass )[-1]) adult$native.country = factor(adult$native.country, levels = levels(adult$native.country)[-1]) adult$occupation = factor(adult$occupation , levels = levels(adult$occupation )[-1])
To visualize the missing values
{r fig.align = 'center'} gg_miss_var(adult, show_pct = TRUE)
The above plot shows that the variables workclass,
occupation, and native.country have missing
values. It shows that the variables workclass,
occupation, and native.country have around
less than 0.06 percent missing values and the variable
native.country has around 0.02 percent missing values.
Here we impute the missing values with random values (which
is proportional to categories' records)by using the function
impute() from R package Hmisc as follows:
{r} # impute missing values with random value adult$workclass = impute(adult$workclass , 'random') adult$native.country = impute(adult$native.country, 'random') adult$occupation = impute(adult$occupation , 'random')
The below plot shows that all the missing values are imputed.
{r fig.align = 'center'} gg_miss_var(adult, show_pct = TRUE)
The impute() function imputes missing values using a
user-defined statistical method (e.g. mean, median, max). Its default is
median. For more advanced approaches, users could apply the
aregImpute() function (from the R package
Hmisc)which provides mean imputation using additive
regression, bootstrapping, and predictive mean matching.
We also can remove NA values from the adult dataset by using
na.omit(). Note, we normally should not remove the missing
values. It depends on the situation whether it's a good idea or not. You
shouldn't always just drop NA values.
capital.lossHere we want to check if there are any outliers in the variable
capital.loss or not. For this, first, we report the summary
of the variable capital.loss as follow:
{r} summary(adult$capital.loss)
We also report the boxplot and histogram of the variable
capital.loss as follow:
{r fig.align = 'center'} ggplot(data = adult, aes(y = capital.loss)) + geom_boxplot()
{r fig.align = 'center'} ggplot(data = adult, aes(x = capital.loss)) + geom_histogram(bins = 30, color = "blue", fill = "lightblue")
What would be your interpretation of the above outputs for
the variable capital.loss? Are there any outliers in the
variable capital.loss? If so, what would be a good strategy
to deal with the outliers?
capital.gainHere we want to check if there is any outliers in the variable
capital.gain.
a. Similar to the previous sub-section report the
summary, boxplot, and histogram
for the variable capital.gain.
{r fig.align = 'center'} ggplot(data = adult) + geom_histogram(mapping = aes(x = capital.gain), binwidth = 0.5) + coord_cartesian(ylim = c(0, 30))
{r fig.align = 'center'} ggplot(data = adult) + geom_point( mapping = aes(x = y, y = capital.gain), colour = 'blue')
{r fig.align = 'center'} ggplot(data = adult, mapping = aes(x = y, y = capital.gain)) + geom_point(colour = 'blue')
b. What would be your interpretation of the above outputs for
the variable capital.gain? Are there any outliers in the
variable capital.gain? If so, what would be a good strategy
to deal with the outliers?
If you do not have your own dataset, do not worry, you can find a suitable public datasets in the follow links:
In the Kaggle website you there are a lot of project examples with read-world datasets for Data Science.
For data-driven business-making, it is vital to understand the problem to be solved. This may seem obvious, but research/business projects seldom come pre-packaged as clear and unambiguous data mining problems.
The Research/Business Understanding stage represents a key part of the craft where the analysts' creativity plays a large role. Data Scientist has some things to say, as we will describe, but often the key to a great success is a creative problem formulation by some analyst regarding how to cast the business problem as one or more data science problems. High-level knowledge of the fundamentals helps creative business analysts see novel formulations. The Business Understanding stage can be done by following steps:
For this part, your task is to write a research/business problem that you want to address in your main project. Write the problem done by following the above three steps for the Business Understanding stage and discuss it with your teammate.
Selected dataset is from Kaggle and it is the retail sales dataset. 1. Objective of the analysis is to determine how customer age and gender influences the purchasing behaviour of the customer. For this it is necessary to have insights in the gender and age of the customer, as well as granular data providing insights in product groups. 2. Does the age and gender of a customer influence the purchase behaviour and what is the impact on monetary value of this 3. preliminary strategy is to first analyse an create customer groups based on both gender and age groups, afterwards categorizing the product categories over the groups. Then, I would like to analyze what the variances are between the customer groups to gain insights from thid.
Raw data from data repositories are seldom ready for the algorithms straight out of the box. Instead, it needs to be cleaned or "prepared for analysis".
For the data understanding part:
{r} summary(retail_sales_dataset)