Exploratory Data Analysis Part Two
Miao Yu
2024-02-12
Example with the bank customer data
Let’s check whether Gender and
Dependent_count has a significant impact on
Attrition_Flag. We can do by graph or by table:
ggplot(bank_data) +
geom_bar(aes(x = Gender, fill = Attrition_Flag), position = "fill") +
labs(y = "proportion")
bank_data %>%
group_by(Gender) %>%
summarize(Attrition_rate = prop.table(table(Attrition_Flag))[1])## # A tibble: 2 × 2
## Gender Attrition_rate
## <chr> <dbl>
## 1 F 0.174
## 2 M 0.146So we see that the attrition rate is significantly higher in female cutomers than male ones. In other words, it seems that female customers are more likely to cancel their credit cards as suggested by the data set.
But please keep in mind that this difference is not necessarily statistical significant (depending on some other factors such as sample size). Later we will learn how to run statistical tests to draw a more quantitative conclusion.
Two numeric variables
For numeric variables, we simply create a scatter plot and/or a line graph. A correlation analysis is usually performed, too.
Let’s take Customer_Age and Credit_Limit as
an example:
ggplot(bank_data, aes(Customer_Age, Credit_Limit)) + geom_point() + geom_smooth()
The points are on vertical lines because Customer_Age is
a discrete variable and only takes integer values. It seems that the
correlation between the two variables are quite weak. But we do see a
reasonable trend in the smoothed version of line graph that, the
mid-aged group has higher credit limits than young professional and
retired people, which agrees with common sense.
Pearson’s Correlation Coefficient
It is also common to check Pearson’s correlation coefficient betweeb two numeric variables. But please be noted that it only measures linear correlation and cannot capture nonlinear ones.
Use cor(<data1>, <data2>) to compute the
coefficient in R. A value around zero indicates little correlation. A
value around \(1\) or \(-1\) indicates strong linear correlation,
as shown in the figure below.
One numeric variable and one categorical variable
For one numeric variable, and one categorical variable, we commonly
use boxplots to visualize their relationships. However, boxplots are
sometimes not informative enough. Here you are recommended to compare
the density curve between two groups. Let’s take
Attrition_Flag and Customer_Age as an
example.
ggplot(bank_data) +
geom_density(aes(x = Customer_Age, fill = Attrition_Flag), alpha = 0.5)
We see the two distributions nearly overlapping with each other, so
at least we can say that the effect of Customer_Age on
Attrition_Flag is not quite significant.
Another example
In comparison, we can create the same plot between
Attrition_Flag and Total_Trans_Ct. A evidently
more significant difference is observed.
ggplot(bank_data) +
geom_density(aes(x = Total_Trans_Ct, fill = Attrition_Flag), alpha = 0.5)
We see that customers with more transaction counts (over 50 or so) are much less likely to cancel their credit card service.
Analyze composite effect of 2 or more variables.
An interesting while challenging part of data exploration is that, sometimes there are complicated composite effect between variables. For example, if we only look at the effect of credit limit on churning, it’s not that significant:
ggplot(bank_data) +
geom_density(aes(x = Credit_Limit, fill = Attrition_Flag), alpha = 0.5)
ggplot(bank_data) +
geom_boxplot(aes(x = Credit_Limit, y = Attrition_Flag)) 
Example (Cont’d)
However, if we put Customer_Age also in the picture, we
see some composite effect:
ggplot(bank_data) +
geom_point(aes(x = Customer_Age, y = Credit_Limit, color = Attrition_Flag), alpha = 0.5) +
geom_smooth(aes(x = Customer_Age, y = Credit_Limit, color = Attrition_Flag))
So it seems that for young customers (less than 55), churning customers tended to have an lower credit limit on average. But the trend becomes opposite for aged customers, who were more likely to churn when they had a higher credit limit. Can you try to explain why?
In this example, we try to analyze the effect of two numeric variables on a categorical variable. So we use a grouped scatter/smooth plot.
What if we try to analyze the effect of two categorical variables?
Example (Cont’d)
We can also use facet_wrap to help use visualize the
effect:
# Relevel the income category, will learn later, not required
bank_data$Income_Category <- factor(bank_data$Income_Category,
levels = unique(bank_data$Income_Category)[c(2,4,1,3,5,6)])
ggplot(bank_data) +
geom_bar(aes(y = Income_Category, fill = Attrition_Flag), position = "fill") +
facet_wrap(~ Gender, nrow = 2)
Question: What can we learn from the result?