Descriptive

Already complete Code

As we move on from week to week and task to task, the code that you have already completed, will stay on the template but will not run, this is possible by adding eval=FALSE to the corresponding code chunk. Note that the libraries need to be linked to this program as well.

# Install and load necessary libraries
#install.packages("ggplot2") # Install ggplot2 for plotting, if you have already installed the packages, comment this out by enterring a # in front of this command
#install.packages("scales")  # Install scales for formatting
#install.packages("moments") # Install moments for skewness and kurtosis
library(ggplot2)            # Load ggplot2 library
library(scales)             # Load scales library
library(moments)

Setting up your directory in your computer.

This needs to be addressed here.

# Check the current working directory
getwd()

## [1] "C:/Users/sarah.serrano/Downloads/RStudio/Descriptive Data Assignment"

# in the next line, change the directory to the place where you saved the
# data file, if you prefer you can save your data.csv file in the directory
# that command 7 indicated.
# for example your next line should like something similar to this: setwd("C:/Users/tsapara/Documents")

# Set the working directory to where the data file is located
# This ensures the program can access the file correctly

setwd("C:/Users/sarah.serrano/Downloads/RStudio/Descriptive Data Assignment")

### Choose an already existing directory in your computer.

Setting up your personalized data

# Read the CSV file
# The header parameter ensures column names are correctly read
# sep defines the delimiter (comma in this case)
# stringsAsFactors prevents automatic conversion of strings to factors
df <- read.csv("data.csv", header = TRUE, sep = ",", stringsAsFactors = TRUE)

##########################################################
# Define variables A and B based on your student ID
# A represents the first 3 digits, B represents the last 3 digits
A <- 34
B <- 99
Randomizer <- A + B # Randomizer ensures a consistent seed value for reproducibility


# Generate a random sample of 500 rows from the dataset
set.seed(Randomizer) # Set the seed for reproducibility
sample_size <- 500
df <- df[sample(nrow(df), sample_size, replace = TRUE), ] # Sample the dataset

write.csv(df, file = "my_data.csv", row.names = FALSE) # this command may take some time to run once it is done, it will create the desired data file locally in your directory

Knit your file

As practice, you may want now to knit your file in an html. To do this, you should click on the knit button on the top panel, and wait for the rendering file. The HTML will open once it is done for you to review.

It is recommended to practice with RMD and download and review the following cheatsheets: https://rmarkdown.rstudio.com/lesson-15.HTML

In addition, you may want to alter some of the editor components and re-knit your file to gain some knowledge and understanding of RMD. For a complete tutorial, visit: https://rmarkdown.rstudio.com/lesson-2.html

df <- read.csv("my_data.csv", header = TRUE, sep = ",", stringsAsFactors = TRUE)

Cleaning up your data

Step 0. Now that you read the file, you want to learn few information about your data

The following commands will not be explained here, do your research, review your csv file and answer the questions related with this part of your code.

# Basic exploratory commands
nrow(df)       # Number of rows in the dataset

## [1] 500

length(df)     # Number of columns (or variables) in the dataset

## [1] 15

str(df)        # Structure of the dataset (data types and a preview)

## 'data.frame':    500 obs. of  15 variables:
##  $ Gender       : Factor w/ 4 levels "","Female","Male",..: 2 2 2 2 3 2 2 4 2 2 ...
##  $ Age          : int  27 27 27 27 30 27 26 28 29 NA ...
##  $ Height       : int  158 168 168 168 182 158 160 185 155 160 ...
##  $ Weight       : int  NA 65 65 65 80 NA 55 85 54 55 ...
##  $ Education    : Factor w/ 5 levels "","Bachelor's",..: 2 2 2 2 4 2 2 3 4 3 ...
##  $ Income       : Factor w/ 15 levels "","32000","35000",..: 6 7 7 7 13 6 8 1 11 5 ...
##  $ MaritalStatus: Factor w/ 3 levels "","Married","Single": 3 3 3 3 2 3 3 3 2 2 ...
##  $ Employment   : Factor w/ 4 levels "","Employed",..: 4 2 2 2 2 4 2 4 2 2 ...
##  $ Score        : Factor w/ 15 levels "","5.5","5.7",..: NA 6 6 6 1 1 5 3 10 5 ...
##  $ Rating       : Factor w/ 5 levels "","5.7","A","B",..: 3 4 4 4 5 3 4 3 4 3 ...
##  $ Category     : Factor w/ 5 levels "","A","Art","Music",..: 3 3 3 3 4 3 3 5 4 3 ...
##  $ Color        : Factor w/ 5 levels "","Blue","Green",..: 3 2 2 2 4 3 2 3 2 2 ...
##  $ Hobby        : Factor w/ 6 levels "","Green","Photography",..: 5 4 4 4 6 5 5 3 5 5 ...
##  $ Happiness    : Factor w/ 10 levels "","6","6.5","7",..: 3 4 4 4 8 3 4 2 7 4 ...
##  $ Location     : Factor w/ 5 levels "","6","City",..: 3 3 3 3 3 3 4 4 3 3 ...

summary(df)    # Summary statistics for each column

##     Gender         Age            Height          Weight            Education  
##        :  2   Min.   :25.00   Min.   :155.0   Min.   :54.00              : 18  
##  Female:201   1st Qu.:27.00   1st Qu.:165.0   1st Qu.:62.00   Bachelor's :181  
##  Male  :188   Median :28.00   Median :175.0   Median :70.00   High School:112  
##  Other :109   Mean   :28.45   Mean   :172.6   Mean   :70.32   Master's   :100  
##               3rd Qu.:30.00   3rd Qu.:182.0   3rd Qu.:80.00   PhD        : 85  
##               Max.   :34.00   Max.   :190.0   Max.   :90.00   NAs        :  4  
##               NAs    :34      NAs    :26      NAs    :56                       
##      Income    MaritalStatus      Employment      Score     Rating   
##         : 85          : 22             : 14   6.2    : 85      :  3  
##  45000  : 84   Married:189   Employed  :355   7.8    : 70   5.7:  1  
##  60000  : 80   Single :289   Single    :  1          : 64   A  :175  
##  48000  : 63                 Unemployed:130   5.7    : 58   B  :194  
##  65000  : 44                                  6.1    : 54   C  :127  
##  42000  : 28                                  (Other):162            
##  (Other):116                                  NAs    :  7            
##    Category      Color             Hobby       Happiness     Location  
##        :  2         :  3              :  3   7      :152         :  3  
##  A     :  1   Blue  :202   Green      :  1   6      : 82   6     :  1  
##  Art   :182   Green :155   Photography: 94   9      : 72   City  :214  
##  Music :131   Red   :139   Reading    :143   8      : 63   Rural :183  
##  Sports:184   Sports:  1   Swimming   :108   8.5    : 54   Suburb: 99  
##                            Traveling  :151   (Other): 72               
##                                              NAs    :  5

Your Turn

Please answer the following questions, by typing information after the question.

Question 1

What type of variables does your file include?

Answer 1:

Question 2

Specific data types?

Answer 2:

Question 3

Are they read properly?

Answer 3:

Question 4

Are there any issues ?

Answer 4:

Question 5

Does your file includes both NAs and blanks?

Answer 5:

Question 6

How many NAs do you have and

Answer 6:

Question 7

How many blanks?

Answer 7:

Cleanup Continued

Step 1: Handling both blanks and NAs is not simple so first we want to eliminate some of those, let’s eliminate the blanks and change them to NAs

#
# Step 1:  # Handling both blanks and NAs is not simple so first we want to eliminate
# some of those, let's eliminate the blanks and change them to NAs
#


# Replace blanks with NAs across the dataset
# This ensures that blank values are consistently treated as missing data
df[df == ""] <- NA

# Convert specific columns to factors
# This step ensures categorical variables are treated correctly after replacing blanks
factor_columns <- c("Gender", "Education", "Rating", "MaritalStatus", "Category", 
                    "Employment", "Color", "Hobby", "Location")
df[factor_columns] <- lapply(df[factor_columns], function(col) as.factor(as.character(col)))
df$Income <- as.numeric(as.character(df$Income))

## Warning: NAs introduced by coercion

df$Score <- as.numeric(as.character(df$Score))

## Warning: NAs introduced by coercion

df$Happiness <- as.numeric(as.character(df$Happiness))

## Warning: NAs introduced by coercion

Cleanup Continued

Step 2: Count NAs in the entire dataset

#
# Step 2: Count NAs in the entire dataset


# Count the total number of NAs in the dataset
total_nas <- sum(is.na(df))
total_nas # Print the total number of missing values

## [1] 377

Your Turn

Please answer the following questions, by typing information after the question.

Question 8

Explain what the printed number is, what is the information that relays and how can you use it in your analysis?

Answer 8:

Clean Up Continued

Step 3: Count rows with NAs.

#
# Step 3: Count rows with NAs
#

# Count rows with at least one NA
rows_with_nas <- sum(rowSums(is.na(df)) > 0)
Percent_row_NA <- percent(rows_with_nas / nrow(df)) # Percentage of rows with NAs
rows_with_nas

## [1] 278

Percent_row_NA

## [1] "56%"

Your Turn

Question 9

How large is the proportion of the rows with NAs, we can drop up to 5%?

Answer 9:

Question 10

Do you think that would be wise to drop the above percent?

Answer 10:

Question 11

How this will affect your dataset?

Answer 11:

CleanUp Continued

Step 4: Count columns with NAs

#  
# Step 4: Count columns with NAs

# Count columns with at least one NA
cols_with_nas <- sum(colSums(is.na(df)) > 0)
Percent_col_NA <- percent(cols_with_nas / length(df)) # Percentage of columns with NAs
cols_with_nas

## [1] 15

Percent_col_NA

## [1] "100%"

Your Turn

Question 12

How large is the proportion of the cols with NAs, we never want to drop entire columnes as this would mean that we will loose variables and associations but do you think that would be wise to drop the above percent?

Answer 12:

Question 13

How this will affect your dataset?

Answer 13:

Imputation

Step 5: Replace NAs with appropriate values (mean for numeric and integer,mode for factor, “NA” for character)

In later weeks we will learn how to replace the NAs properly based on the descriptive statistics and you will discuss this code.For now, you can assume that by setting the mean of the variable for numeric and mode for categorical it is correct - this is not always the case of course but the code will become much more complicated in that case.

# 
# Step 5: Replace NAs with appropriate values (mean for numeric and integer,
# mode for factor, "NA" for character)
# In later weeks we will learn how to replace the NAs properly based on the
# descriptive statistics and you will discuss this code.
# for now, you can assume that by setting the mean of the variable for numeric
# and mode for categorical it is correct - this is not always the case of course
# but the code will become much more complicated in that case.


# Replace NAs with appropriate values
# Numeric: Replace with the mean if sufficient data is available
# Categorical: Replace with the mode (most common value)
# Character: Replace with the string "NA"
df <- lapply(df, function(col) {
  if (is.numeric(col) || is.integer(col)) { # Numeric or integer columns
    if (sum(!is.na(col)) > 10) {
      col[is.na(col)] <- mean(col, na.rm = TRUE) # Replace with mean
    } else {
      col[is.na(col)] <- approx(seq_along(col), col, n = length(col))[["y"]][is.na(col)] # Interpolation
    }
  } else if (is.factor(col)) { # Factor columns
    mode_val <- names(sort(-table(col)))[1] # Mode (most common value)
    col[is.na(col)] <- mode_val
  } else if (is.character(col)) { # Character columns
    col[is.na(col)] <- "NA" # Replace with "NA"
  }
  return(col) # Return the modified column
})

df <- as.data.frame(df) # Convert the list back to a dataframe


#
# following the above method to impute, has now changed some of the statistics


# Check the updated dataset and ensure no remaining NAs
summary(df)

##     Gender         Age            Height          Weight            Education  
##  Female:203   Min.   :25.00   Min.   :155.0   Min.   :54.00   Bachelor's :203  
##  Male  :188   1st Qu.:27.00   1st Qu.:165.0   1st Qu.:65.00   High School:112  
##  Other :109   Median :28.00   Median :172.6   Median :70.00   Master's   :100  
##               Mean   :28.45   Mean   :172.6   Mean   :70.32   PhD        : 85  
##               3rd Qu.:30.00   3rd Qu.:182.0   3rd Qu.:80.00                    
##               Max.   :34.00   Max.   :190.0   Max.   :90.00                    
##      Income      MaritalStatus      Employment      Score      Rating   
##  Min.   :32000   Married:189   Employed  :369   Min.   :5.50   5.7:  1  
##  1st Qu.:45000   Single :311   Single    :  1   1st Qu.:6.10   A  :175  
##  Median :51341                 Unemployed:130   Median :6.68   B  :197  
##  Mean   :51341                                  Mean   :6.68   C  :127  
##  3rd Qu.:60000                                  3rd Qu.:7.50            
##  Max.   :70000                                  Max.   :8.90            
##    Category      Color             Hobby       Happiness       Location  
##  A     :  1   Blue  :205   Green      :  1   Min.   :6.000   6     :  1  
##  Art   :182   Green :155   Photography: 94   1st Qu.:7.000   City  :217  
##  Music :131   Red   :139   Reading    :143   Median :7.000   Rural :183  
##  Sports:186   Sports:  1   Swimming   :108   Mean   :7.458   Suburb: 99  
##                            Traveling  :154   3rd Qu.:8.500               
##                                              Max.   :9.000

Your Turn

Essay Question

Run summary(df) and compare with the previous statistics. Do you observe any undesired changes? Explain in detail. Are there any more NA’s in your file?What is the information that is printed by the summary? How can this be interpreted? what are your observations? Verify the effects of imputation, and explain in detail. Compare the updated summary with the earlier statistics and note changes. Explain everything that you obsevrve.

Answer

Descriptive Statistics

Step 6: Create descriptive statistics for all variables

We run all the descriptive statistics for all the numeric variables

################################################################### 
# 
# Step 6: Create descriptive statistics for all variables
# We run all the descriptive statistics for all the numeric variables
#
###################################################################
# Initialize a function to compute descriptive statistics
compute_stats <- function(column, name) {
  if (is.numeric(column) || is.integer(column)) {
    data.frame(
      Variable = name,
      Mean = round(mean(column, na.rm = TRUE), 2),
      Median = round(median(column, na.rm = TRUE), 2),
      St.Deviation = round(sd(column, na.rm = TRUE), 2),
      Range = round(diff(range(column, na.rm = TRUE)), 2),
      IQR = round(IQR(column, na.rm = TRUE), 2),
      Skewness = round(skewness(column, na.rm = TRUE), 2),
      Kurtosis = round(kurtosis(column, na.rm = TRUE), 2),
      stringsAsFactors = FALSE
    )
  } else {
    NULL
  }
}

# Apply the function to each numeric or integer column in the dataset
descriptive_stats <- do.call(
  rbind,
  lapply(names(df), function(col) compute_stats(df[[col]], col))
)

# Print the descriptive statistics dataframe
descriptive_stats

##    Variable     Mean   Median St.Deviation   Range     IQR Skewness Kurtosis
## 1       Age    28.45    28.00         1.96     9.0     3.0     0.68     3.43
## 2    Height   172.56   172.56         9.34    35.0    17.0    -0.03     1.85
## 3    Weight    70.32    70.00        10.13    36.0    15.0     0.07     2.02
## 4    Income 51340.58 51340.58      8431.14 38000.0 15000.0     0.13     2.68
## 5     Score     6.68     6.68         0.86     3.4     1.4     0.66     2.59
## 6 Happiness     7.46     7.00         0.98     3.0     1.5     0.11     1.89

Descriptive Statistics Continued

Step 7: Print Descriptive Statistics

Now you have all the descriptive statistics for all numeric variables Create a professional table in your paper. The library(KableExtra), can help you create the table here. If you have no programming experience you can cut and paste in Excel and beautify the table in Excel.

#############################################################
# 
# Step 7: Print Descriptive Statistics
# Now you have all the descriptive statistics for all numeric variables
# Create a professional table in your paper.
# the library(KableExtra), can help you create the table here.
# if you have no programming experience you can cut and paste in Excel
# and beautify the table in Excel
#############################################################
  
  print("Descriptive Statistics:")

## [1] "Descriptive Statistics:"

  print(descriptive_stats)

##    Variable     Mean   Median St.Deviation   Range     IQR Skewness Kurtosis
## 1       Age    28.45    28.00         1.96     9.0     3.0     0.68     3.43
## 2    Height   172.56   172.56         9.34    35.0    17.0    -0.03     1.85
## 3    Weight    70.32    70.00        10.13    36.0    15.0     0.07     2.02
## 4    Income 51340.58 51340.58      8431.14 38000.0 15000.0     0.13     2.68
## 5     Score     6.68     6.68         0.86     3.4     1.4     0.66     2.59
## 6 Happiness     7.46     7.00         0.98     3.0     1.5     0.11     1.89

Your Turn

Essay Question

Review and compare with the previous statistics. Do you observe any undesired changes? Explain in detail. How can this be interpreted? what are your observations? Verify the descriptive statistics, and explain in detail. Explain everything that you obsevrve. Complete your research compare your variables and complete your paper

Answer

Visual Representations

Step 8: Create graphs using ggplot2

For this part there are parts that you will need to change to create your graphs. The example is set to work with Income. Make the necessary changes to create the rest of the graphs. You may also want to change the colors, the dimensions etc…

  #######################################################################
  # 
  # Step 8: Create graphs using ggplot2
  # For this part there are parts that you will need to change to create 
  # your graphs.
  # The example is set to work with Income
  # Make the necessary changes to create the rest of the graphs
  # You may also want to change the colors, the dimensions etc...
  #############################################################
  
  #############################################################
  #
  # STEP 8a: Create a bargraph or a histogram
  # Explain what graph was that and why?
  # Set col to the desired column name
  #############################################################
  #
  ##
  # In this code we start you of with an example of Happiness, later in the code
  # you should replace this with your desired variable.
  #
  
    col = "Location"  # This is an example, try to do the same with a different variable

Bargraph - if the variable of your choice is categorical

  # Bar graphs for the three selected categorical variables

categorical_vars <- c("Education", "Employment", "Location")

for (col in categorical_vars) {
  
  p <- ggplot(df, aes(x = .data[[col]], fill = .data[[col]])) +
    geom_bar() +
    labs(
      title = paste("Bar Graph for", col),
      x = col,
      y = "Count"
    ) +
    theme_minimal() +
    theme(legend.position = "right")
  
  print(p)
}

Histogram - If the varaibles of your choice is Numerical

You can also copy the chunk and create more graphs by resetting the col variable appropriately

# Histograms for the three selected numerical variables

numerical_vars <- c("Income", "Score", "Happiness")

for (col in numerical_vars) {

  p <- ggplot(df, aes(x = .data[[col]])) +
    geom_histogram(bins = 30, fill = "steelblue", color = "black") +
    labs(
      title = paste("Histogram for", col),
      x = col,
      y = "Frequency"
    ) +
    theme_minimal()

  print(p)
}

Your Turn

Essay Question

Now that you can observe graphically your data, explain the importance of graphical representations and how this helps to communicate data with other parties. Explain what graph was that and why?

Answer

Your Turn

STEP 8b: Create a boxplot and a Histogram for numeric variables note the the Bin width cannot be set up in the same way to work with Age or Happiness that has a small range and Income that the range is in thousands. Change this appropriately

Please note that this part of the code will not run for the demo code. You will need to change the value of eval=FALSE to eval=TRUE, after you introduce your code, to run it and add it to your knitted file.

 #############################################################
      #
      # STEP 8b: Create a boxplot  and Histogram for numeric variables
      # note the the Bin width cannot be set up in the same way to work with 
      # Age or Happiness that has a small range and Income that the range is in thousands
      # Change this appropriately
      #############################################################
      #
      # Choose a numeric variable (i.e., Age) set the col variable to the name of the column then you rerun the code that is commented out here.

#col = ____ Add the variable of your choice  

# Uncomment the code and you will create a Bar graph or a Histogram of a different variable here.
# Do not forget to change the value of eval=TRUE to run and knit this chunk
    
#  if (is.factor(df[[col]])) { # if the col is categorical, then the code will
      # create two graphs the Bar graph 
      # Highlight and run until the line that start with `# Boxplot for numeric variables
      #
      # If the col is numeric, then it will create the histogram
      # Bar graph for factors
#      ggplot(df, aes(x = .data[[col]], fill = .data[[col]])) +
#        geom_bar() +
#        labs(title = paste("Bar Graph for", col), x = col, y = "Count") +
#        theme_minimal() +
#        theme(legend.position = "right")
#    } else if (is.numeric(df[[col]]) || is.integer(df[[col]])) {
      
#      ggplot(df, aes(x = .data[[col]])) +
#        geom_histogram(binwidth = 0.3) +
#        labs(title = paste("Histogram for", col), x = col, y = "Count") +
#        theme_minimal()
    }

Your Turn

Essay Question

Now explain this graph. Focus on the information extracted, anomalies, outliers, relationships.

Answer

Your Turn

***Step 8c: NOTE that you should run this part with the latest value of col. Do not forget to change the eval=TRUE to knit it.

Boxplot for numeric variables

 # Boxplots for the three selected numerical variables

numerical_vars <- c("Income", "Score", "Happiness")

for (col in numerical_vars) {

  p <- ggplot(df, aes(x = "", y = .data[[col]])) +
    geom_boxplot(
      fill = "skyblue",
      color = "darkblue",
      width = 0.3,
      outlier.color = "red",
      outlier.size = 2
    ) +
    labs(
      title = paste("Box Plot for", col),
      x = NULL,
      y = col
    ) +
    theme_minimal() +
    theme(
      axis.text.x = element_blank(),
      axis.ticks.x = element_blank(),
      plot.title = element_text(hjust = 0.5, size = 16, face = "bold"),
      axis.title.y = element_text(size = 14),
      axis.text.y = element_text(size = 12)
    )

  print(p)
}

Your Turn

Essay Question

Explain the findings of your Boxplot. Are there any outliers? What is the IQR? Focus on the information extracted, anomalies, outliers, relationships.

Answer

Tabular Representations

Step 9: Tables

Creating tables to understand how the different categorical variables interconnect. Tabular information can be provided in both tables and parallel barplots. The following is an example on two variables, choose two others to get more valuable insights.

# Step 9: Contingency table for Employment and Location

Employment_Location <- table(df$Employment, df$Location)

# Display contingency table
Employment_Location

##             
##                6 City Rural Suburb
##   Employed     0  195    87     87
##   Single       1    0     0      0
##   Unemployed   0   22    96     12

# Display contingency table with totals
addmargins(Employment_Location)

##             
##                6 City Rural Suburb Sum
##   Employed     0  195    87     87 369
##   Single       1    0     0      0   1
##   Unemployed   0   22    96     12 130
##   Sum          1  217   183     99 500

# Create clustered barplot
barplot(
  Employment_Location,
  beside = TRUE,
  main = "Employment Status by Location",
  xlab = "Location",
  ylab = "Count",
  legend.text = rownames(Employment_Location),
  args.legend = list(x = "topright", cex = 0.7)
)

# Print contingency table with totals
print(addmargins(Employment_Location))

##             
##                6 City Rural Suburb Sum
##   Employed     0  195    87     87 369
##   Single       1    0     0      0   1
##   Unemployed   0   22    96     12 130
##   Sum          1  217   183     99 500

Making Pretty Tables

library(knitr)
library(kableExtra)

# Create the Employment by Location contingency table
Employment_Location <- table(df$Employment, df$Location)

# Add row and column totals
Employment_Location_margins <- addmargins(Employment_Location)

# Create a professional formatted table
kable(
  Employment_Location_margins,
  caption = "Employment Status by Location",
  align = "c"
) %>%
  kable_styling(
    full_width = FALSE,
    bootstrap_options = c("striped", "hover", "condensed")
  ) %>%
  row_spec(
    0,
    bold = TRUE,
    background = "#D3D3D3"
  )

Employment Status by Location
	6	City	Rural	Suburb	Sum
Employed	0	195	87	87	369
Single	1	0	0	0	1
Unemployed	0	22	96	12	130
Sum	1	217	183	99	500

Your Turn

Essay Question

Explain the table in details. Focus on the information extracted, anomalies, outliers, relationships.

Answer

Descriptive_Data

I.Tsapara

2025-04-28

Already complete Code

Setting up your directory in your computer.

Setting up your personalized data

Knit your file

Cleaning up your data

Your Turn

Cleanup Continued

Cleanup Continued

Your Turn

Clean Up Continued

Your Turn

CleanUp Continued

Your Turn

Imputation

Your Turn

Descriptive Statistics

Descriptive Statistics Continued

Your Turn

Visual Representations

Bargraph - if the variable of your choice is categorical

Histogram - If the varaibles of your choice is Numerical

Your Turn

Your Turn

Your Turn

Your Turn

Your Turn

Tabular Representations

Making Pretty Tables

Your Turn