As we move on from week to week and task to task, the code that you have already completed, will stay on the template but will not run, this is possible by adding eval=FALSE to the corresponding code chunk. Note that the libraries need to be linked to this program as well.
# Install and load necessary libraries
#install.packages("ggplot2") # Install ggplot2 for plotting, if you have already installed the packages, comment this out by enterring a # in front of this command
#install.packages("scales") # Install scales for formatting
#install.packages("moments") # Install moments for skewness and kurtosis
library(ggplot2) # Load ggplot2 library
library(scales) # Load scales libraryThis needs to be addressed here.
# Check the current working directory
getwd()## [1] "C:/Users/sarah.serrano/Downloads/RStudio"# in the next line, change the directory to the place where you saved the
# data file, if you prefer you can save your data.csv file in the directory
# that command 7 indicated.
# for example your next line should like something similar to this: setwd("C:/Users/tsapara/Documents")
# Set the working directory to where the data file is located
# This ensures the program can access the file correctly
setwd("C:/Users/sarah.serrano/Downloads/RStudio")
### Choose an already existing directory in your computer.# Read the CSV file
# The header parameter ensures column names are correctly read
# sep defines the delimiter (comma in this case)
# stringsAsFactors prevents automatic conversion of strings to factors
df <- read.csv("my_data.csv.csv", header = TRUE, sep = ",", stringsAsFactors = TRUE)
##########################################################
# Define variables A and B based on your student ID
# A represents the first 3 digits, B represents the last 3 digits
A <- 34
B <- 99
Randomizer <- A + B # Randomizer ensures a consistent seed value for reproducibility
# Generate a random sample of 500 rows from the dataset
set.seed(Randomizer) # Set the seed for reproducibility
sample_size <- 500
df <- df[sample(nrow(df), sample_size, replace = TRUE), ] # Sample the dataset
write.csv(df, file = "my_data.csv", row.names = FALSE) # this command may take some time to run once it is done, it will create the desired data file locally in your directoryAs practice, you may want now to knit your file in an html. To do this, you should click on the knit button on the top panel, and wait for the rendering file. The HTML will open once it is done for you to review.
It is recommended to practice with RMD and download and review the following cheatsheets: https://rmarkdown.rstudio.com/lesson-15.HTML
In addition, you may want to alter some of the editor components and re-knit your file to gain some knowledge and understanding of RMD. For a complete tutorial, visit: https://rmarkdown.rstudio.com/lesson-2.html
df <- read.csv("my_data.csv", header = TRUE, sep = ",", stringsAsFactors = TRUE)Step 0. Now that you read the file, you want to learn few information about your data
The following commands will not be explained here, do your research, review your csv file and answer the questions related with this part of your code.
# Basic exploratory commands
nrow(df) # Number of rows in the dataset## [1] 500length(df) # Number of columns (or variables) in the dataset## [1] 15str(df) # Structure of the dataset (data types and a preview)## 'data.frame': 500 obs. of 15 variables:
## $ Gender : Factor w/ 3 levels "Female","Male",..: 1 2 2 1 2 2 1 1 3 1 ...
## $ Age : int 25 30 29 25 32 29 27 26 NA 27 ...
## $ Height : int 165 180 175 165 175 175 158 160 165 168 ...
## $ Weight : int 58 75 70 58 70 70 NA 55 56 65 ...
## $ Education : Factor w/ 5 levels "","Bachelor's",..: 2 4 5 2 5 5 2 1 5 2 ...
## $ Income : int 45000 60000 60000 45000 60000 60000 42000 48000 55000 45000 ...
## $ MaritalStatus: Factor w/ 3 levels "","Married","Single": 3 2 1 3 2 2 3 3 3 3 ...
## $ Employment : Factor w/ 3 levels "","Employed",..: 1 2 2 1 2 2 3 2 3 2 ...
## $ Score : num 6.2 7.5 7.8 6.2 7.8 7.8 NA 6.1 7.8 6.2 ...
## $ Rating : Factor w/ 3 levels "A","B","C": 2 3 3 2 3 3 1 2 2 2 ...
## $ Category : Factor w/ 3 levels "Art","Music",..: 2 2 3 2 3 3 1 1 2 1 ...
## $ Color : Factor w/ 4 levels "","Blue","Green",..: 3 2 4 3 4 4 3 2 2 2 ...
## $ Hobby : Factor w/ 4 levels "Photography",..: 4 2 4 4 4 4 3 3 4 2 ...
## $ Happiness : num 7 8 9 7 9 9 6.5 7 8.5 7 ...
## $ Location : Factor w/ 4 levels "","City","Rural",..: 3 2 4 3 4 4 2 3 4 2 ...summary(df) # Summary statistics for each column## Gender Age Height Weight Education
## Female:201 Min. :25.00 Min. :155.0 Min. :54.00 : 9
## Male :195 1st Qu.:27.00 1st Qu.:165.0 1st Qu.:58.00 Bachelor's :191
## Other :104 Median :28.00 Median :175.0 Median :70.00 High School: 96
## Mean :28.42 Mean :172.4 Mean :69.58 Master's : 99
## 3rd Qu.:30.00 3rd Qu.:182.0 3rd Qu.:80.00 PhD :103
## Max. :34.00 Max. :190.0 Max. :90.00 NAs : 2
## NAs :42 NAs :28 NAs :41
## Income MaritalStatus Employment Score Rating
## Min. :32000 : 19 : 18 Min. :5.500 A:147
## 1st Qu.:45000 Married:192 Employed :365 1st Qu.:6.100 B:219
## Median :50000 Single :289 Unemployed:117 Median :6.200 C:134
## Mean :51961 Mean :6.779
## 3rd Qu.:60000 3rd Qu.:7.800
## Max. :70000 Max. :8.900
## NAs :85 NAs :58
## Category Color Hobby Happiness Location
## Art :174 : 2 Photography: 96 Min. :6.000 : 3
## Music :146 Blue :223 Reading :131 1st Qu.:7.000 City :206
## Sports:180 Green:134 Swimming :115 Median :7.000 Rural :181
## Red :141 Traveling :158 Mean :7.512 Suburb:110
## 3rd Qu.:8.500
## Max. :9.000
## NAs :18Please answer the following questions, by typing information after the question.
Question 1
What type of variables does your file include?
Answer 1:
Question 2
Specific data types?
Answer 2:
Question 3
Are they read properly?
Answer 3:
Question 4
Are there any issues ?
Answer 4:
Question 5
Does your file includes both NAs and blanks?
Answer 5:
Question 6
How many NAs do you have and
Answer 6:
Question 7
How many blanks?
Answer 7:
Step 1: Handling both blanks and NAs is not simple so first we want to eliminate some of those, let’s eliminate the blanks and change them to NAs
#
# Step 1: # Handling both blanks and NAs is not simple so first we want to eliminate
# some of those, let's eliminate the blanks and change them to NAs
#
# Replace blanks with NAs across the dataset
# This ensures that blank values are consistently treated as missing data
df[df == ""] <- NA
# Convert specific columns to factors
# This step ensures categorical variables are treated correctly after replacing blanks
factor_columns <- c("Gender", "Education", "Rating", "MaritalStatus", "Category",
"Employment", "Color", "Hobby", "Location")
df[factor_columns] <- lapply(df[factor_columns], function(col) as.factor(as.character(col)))Step 2: Count NAs in the entire dataset
#
# Step 2: Count NAs in the entire dataset
# Count the total number of NAs in the dataset
total_nas <- sum(is.na(df))
total_nas # Print the total number of missing values## [1] 325Please answer the following questions, by typing information after the question.
Question 8
Explain what the printed number is, what is the information that relays and how can you use it in your analysis?
Answer 8:
Step 3: Count rows with NAs.
#
# Step 3: Count rows with NAs
#
# Count rows with at least one NA
rows_with_nas <- sum(rowSums(is.na(df)) > 0)
Percent_row_NA <- percent(rows_with_nas / nrow(df)) # Percentage of rows with NAs
rows_with_nas## [1] 272Percent_row_NA## [1] "54%"Question 9
How large is the proportion of the rows with NAs, we can drop up to 5%?
Answer 9:
Question 10
Do you think that would be wise to drop the above percent?
Answer 10:
Question 11
How this will affect your dataset?
Answer 11:
Step 4: Count columns with NAs
#
# Step 4: Count columns with NAs
# Count columns with at least one NA
cols_with_nas <- sum(colSums(is.na(df)) > 0)
Percent_col_NA <- percent(cols_with_nas / length(df)) # Percentage of columns with NAs
cols_with_nas## [1] 11Percent_col_NA## [1] "73%"Question 12
How large is the proportion of the cols with NAs, we never want to drop entire columnes as this would mean that we will loose variables and associations but do you think that would be wise to drop the above percent?
Answer 12:
Question 13
How this will affect your dataset?
Answer 13:
Step 5: Replace NAs with appropriate values (mean for numeric and integer,mode for factor, “NA” for character)
In later weeks we will learn how to replace the NAs properly based on the descriptive statistics and you will discuss this code.For now, you can assume that by setting the mean of the variable for numeric and mode for categorical it is correct - this is not always the case of course but the code will become much more complicated in that case.
#
# Step 5: Replace NAs with appropriate values (mean for numeric and integer,
# mode for factor, "NA" for character)
# In later weeks we will learn how to replace the NAs properly based on the
# descriptive statistics and you will discuss this code.
# for now, you can assume that by setting the mean of the variable for numeric
# and mode for categorical it is correct - this is not always the case of course
# but the code will become much more complicated in that case.
# Replace NAs with appropriate values
# Numeric: Replace with the mean if sufficient data is available
# Categorical: Replace with the mode (most common value)
# Character: Replace with the string "NA"
df <- lapply(df, function(col) {
if (is.numeric(col) || is.integer(col)) { # Numeric or integer columns
if (sum(!is.na(col)) > 10) {
col[is.na(col)] <- mean(col, na.rm = TRUE) # Replace with mean
} else {
col[is.na(col)] <- approx(seq_along(col), col, n = length(col))[["y"]][is.na(col)] # Interpolation
}
} else if (is.factor(col)) { # Factor columns
mode_val <- names(sort(-table(col)))[1] # Mode (most common value)
col[is.na(col)] <- mode_val
} else if (is.character(col)) { # Character columns
col[is.na(col)] <- "NA" # Replace with "NA"
}
return(col) # Return the modified column
})
df <- as.data.frame(df) # Convert the list back to a dataframe
#
# following the above method to impute, has now changed some of the statistics
# Check the updated dataset and ensure no remaining NAs
summary(df)## Gender Age Height Weight Education
## Female:201 Min. :25.00 Min. :155.0 Min. :54.00 Bachelor's :202
## Male :195 1st Qu.:27.00 1st Qu.:165.0 1st Qu.:65.00 High School: 96
## Other :104 Median :28.42 Median :173.7 Median :69.58 Master's : 99
## Mean :28.42 Mean :172.4 Mean :69.58 PhD :103
## 3rd Qu.:30.00 3rd Qu.:180.0 3rd Qu.:80.00
## Max. :34.00 Max. :190.0 Max. :90.00
## Income MaritalStatus Employment Score Rating
## Min. :32000 Married:192 Employed :383 Min. :5.500 A:147
## 1st Qu.:45000 Single :308 Unemployed:117 1st Qu.:6.100 B:219
## Median :51961 Median :6.700 C:134
## Mean :51961 Mean :6.779
## 3rd Qu.:60000 3rd Qu.:7.500
## Max. :70000 Max. :8.900
## Category Color Hobby Happiness Location
## Art :174 Blue :225 Photography: 96 Min. :6.000 City :209
## Music :146 Green:134 Reading :131 1st Qu.:7.000 Rural :181
## Sports:180 Red :141 Swimming :115 Median :7.000 Suburb:110
## Traveling :158 Mean :7.512
## 3rd Qu.:8.500
## Max. :9.000Essay Question
Run summary(df) and compare with the previous statistics. Do you observe any undesired changes? Explain in detail. Are there any more NA’s in your file?What is the information that is printed by the summary? How can this be interpreted? what are your observations? Verify the effects of imputation, and explain in detail. Compare the updated summary with the earlier statistics and note changes. Explain everything that you obsevrve.
Answer