Exploratory Data Analysis Part One
Miao Yu
2025-02-12
Process of EDA
The process of EDA usually involves the cycle of
asking questions out of specific or common interest;
answer them with data transformation, visualization and modeling, which typically would generate new questions and thus close the loop.
Here we will focus on data transformation and visualization, and leave the modeling part to future classes.
Change the y variable in bar charts with
stat_summary()
So far our bar plots are all plotting for counts. In many cases we
hope to plot for other descriptive measures. In the
diamonds data set, for example, we may hope to the mean
price for diamonds with various combination of color/clarity levels. The
following code does the job for us:
ggplot(data = diamonds) +
stat_summary(mapping = aes(x = clarity, y = price, fill = color), fun = "mean", geom = "bar", position = "dodge") +
labs(title = "Diamond Price by clarity and Color",
x = "Clarity Level",
y = "Mean Price (US dollar)") +
theme(plot.title = element_text(hjust = 0.5, size = rel(1.5), margin = margin(15,15,15,15)),
axis.title = element_text(size = rel(1.2)),
axis.title.x = element_text(margin = margin(10,5,5,5)),
axis.title.y = element_text(margin = margin(5,10,5,5)),
axis.text = element_text(size = rel(1.2)))
Basically, stat_summary or any other
<STAT_FUNCTION> is an alternative to to
<GEOM_FUNCTIONS>. The different is that,
-<GEOM_FUNCTIONS> creates a graph of a particular
graph type (point, line, bar plot etc.) and we need to specify the
mapping in it. -<STAT_FUNCTION> creates a graph that
plots a particular statistic (count, proportion, density, or any
specified variables and their functions), and we need to specify the
graph type in the function by using the geom argument.
An Example of EDA (exploratory data analysis)
To resolve the effect of carat, one way (of course not
the best way here) to handle this is to create a new ordinal variable
named carat_group. Then we label all samples into one group
in <= 1 carat, 1-2 carat,
2-3 carat, and > 3 carat based on the
diamond carat.
diamonds <- mutate(diamonds, carat_group = cut(diamonds$carat, c(0, 1,2,3, Inf), c('<= 1 carat', '1-2 carat', '2-3 carat', '> 3 carat')))
ggplot(data = diamonds) +
stat_summary(mapping = aes(x = carat_group, y = price, fill = color), fun = "mean", geom = "bar", position = "dodge") +
labs(title = "Diamond Price by Carat and Color",
x = "Carat",
y = "Mean Price (US dollar)") +
theme(plot.title = element_text(hjust = 0.5, size = rel(1.5), margin = margin(15,15,15,15)),
axis.title = element_text(size = rel(1.2)),
axis.title.x = element_text(margin = margin(10,5,5,5)),
axis.title.y = element_text(margin = margin(5,10,5,5)),
axis.text = element_text(size = rel(1.2)))
Graph Analysis
This graph makes much more sense now since for diamonds of less than 3 carats, the mean price decreases as the color level downgrades as an overall trend. However, the group “> 3 carat” still looks weird since the worst color still has the highest mean price.
In this case, we need to speculate why this may happen and investigate whether our speculation is correct or not by graphs. Here a natural speculation is that “> 3 carat” diamonds are very rare so the sample size is too small to reflect the trend. We may verify our speculation with the following graph:
ggplot(data = diamonds) +
geom_bar(mapping = aes(x = carat_group, fill = color) , position = "dodge") +
labs(title = "Diamond Data by Carat and Color",
x = "Carat",
y = "Counts") +
theme(plot.title = element_text(hjust = 0.5, size = rel(1.5), margin = margin(15,15,15,15)),
axis.title = element_text(size = rel(1.2)),
axis.title.x = element_text(margin = margin(10,5,5,5)),
axis.title.y = element_text(margin = margin(5,10,5,5)),
axis.text = element_text(size = rel(1.2))) 
The graph above clearly shows that the sample size in the “> 3 carats” group is simply too small, so we are not able to draw reliable conclusions for that group.
Exploratory Data Analysis (Cont’d)
It is also helpful to check the mean carat for each
color level:
ggplot(data = diamonds) +
stat_summary(mapping = aes(x = color, y = carat, fill = color), fun = "mean", geom = "bar") +
labs(title = "Mean Diamond Weight by Color",
x = "Color Level",
y = "Mean Weight (carat)") +
theme(plot.title = element_text(hjust = 0.5, size = rel(1.5), margin = margin(15,15,15,15)),
axis.title = element_text(size = rel(1.2)),
axis.title.x = element_text(margin = margin(10,5,5,5)),
axis.title.y = element_text(margin = margin(5,10,5,5)),
axis.text = element_text(size = rel(1.2))) 
As we see, in this data set diamonds with lower color quality are heavier on average, and that’s why we have to consider this factor when analyzing the relationship between price and color level.
Exploratory Data Analysis (Cont’d)
It is also helpful to study diamonds of less than 2 carat to exclude
big diamonds that are rare. In this case, we need to first filter our
data using the filter function in dplyr
package.
ggplot(data = filter(diamonds, carat < 2)) +
stat_summary(mapping = aes(x = color, y = carat, fill = color), fun = "mean", geom = "bar") +
labs(title = "Mean Weight vs Color for '< 2 carat' Diamonds",
x = "Color Level",
y = "Mean Weight (carat)") +
theme(plot.title = element_text(hjust = 0.5, size = rel(1.5), margin = margin(15,15,15,15)),
axis.title = element_text(size = rel(1.2)),
axis.title.x = element_text(margin = margin(10,5,5,5)),
axis.title.y = element_text(margin = margin(5,10,5,5)),
axis.text = element_text(size = rel(1.2))) 
The trend changes little compared with the previous graph. Therefore
we must take into account the effect of carat when
analyzing the effect of color on price.
This also shows that diamond color indeed impacts the price because although diamonds with color “D” and “F” share similar mean price, but on average “F” is heavier than “D”.
For categorical (ordinal) variables, create a bar plot or make a frequency table
For example, if we hope to know the distribution of
Attrition_Flag, which is a categorical variable. We can do
the following:
table(bank_data$Attrition_Flag)##
## Attrited Customer Existing Customer
## 1627 8500prop.table(table(bank_data$Attrition_Flag))##
## Attrited Customer Existing Customer
## 0.1606596 0.8393404The table() function creates a frequency table for a
categorical variable. Further put that table into
prop.table function results in the proportion of each
category. Here we see that there are only two values in the variable -
existing customer and attriting customer. Also, there are many more
existing customers than attriting customers. This information is very
important in modeling - the data is unbalanced in
quantity between the two key groups that we hope to study.
We may also create a bar plot to show the same information
ggplot(bank_data) + geom_bar(aes(Attrition_Flag))
For numeric variables, create a histogram and a summary
To understand the distribution of a numeric variable, the most
typical action is to create a histogram. But usually it is good to
complement it with a summary as well. Let’s take the variable
Credit_Limit as an example:
summary(bank_data$Credit_Limit)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 1438 2555 4549 8632 11068 34516ggplot(bank_data) + geom_histogram(aes(Credit_Limit))
Analysis: This histogram provides some useful information for us. We see that this is a bimodal distribution - one around $2000, and one around the maximum value of $34516. The origin of the second mode may need some investigation if necessary. Now we just keep this information in mind.
Treat discrete variables as categorical variable
For discrete numeric variables, you are recommended to treat it as categorical variable if the number of possible values is small, or treat it as a continuous variable if the number of possible values is larage. But in many cases you can do both.
For example, the variable Dependent_count is a discrete
random variable with values to be from 0 to 5:
table(bank_data$Dependent_count)##
## 0 1 2 3 4 5
## 904 1838 2655 2732 1574 424unique(bank_data$Dependent_count)## [1] 3 5 4 2 0 1The function unique lists all unique values of a
variable.
Therefore it’s reasonable to treat Dependent_count as a
categorical variable. Use as.factor() function to convert
it into a categorical variable when creating a bar plot.
ggplot(bank_data) +
geom_bar(aes(x = as.factor(Dependent_count))) +
labs(x = "Number of Dependents")
Check for unusual values (potential outliers)
Outliers are observations that are unusual; data points that don’t seem to fit the overall data pattern. Sometimes outliers are data entry errors; other times outliers suggest important new science. It is very important to be aware of them before our analysis.
Using the diamonds data set as an example. If we create
a histogram regarding the y variable, the width, we see
something strange.
ggplot(diamonds) + geom_histogram(aes(y))
We see that most data are within 1-10 mm but the plot extends itself
to 60 mm. This is an indicator that there is some unusually large values
in y to trigger this. There are so many observations in the
common bins that the rare bins are so short that you can’t see them.
To make it easy to see the unusual values, we need to zoom to small
values of the y-axis with coord_cartesian():
ggplot(diamonds) +
geom_histogram(mapping = aes(x = y), binwidth = 0.5) +
coord_cartesian(ylim = c(0, 50))
Now we see that there is a few suspicious data points with
y being 0 or 30, 60 mm. They do not look correct (how can a
diamond be 6 cm long?) and we will inspect them more closely.
An important side note here is that we cannot use ylim
here to zoom because that would remove all bins that
are higher than 50!
ggplot(diamonds) +
geom_histogram(mapping = aes(x = y), binwidth = 0.5) +
ylim(0, 50)