IntrotoBA_asm1
Bui Ha Linh
2025-04-07
1 Import dataset
Reproducible report
library(readxl)
df <- read_excel("german_credit_data_formatted-2.xlsx")
df <- as.data.frame(df)2 Clean dataset
The dim() command helps count the number of rows and columns in a dataset. The names() command returns the column names.
dim(df)## [1] 1000 10#summary(df)
str(df)## 'data.frame': 1000 obs. of 10 variables:
## $ ...1 : num 0 1 2 3 4 5 6 7 8 9 ...
## $ Age : num 67 22 49 45 53 35 53 35 61 28 ...
## $ Sex : chr "male" "female" "male" "male" ...
## $ Job : num 2 2 1 2 2 1 2 3 1 3 ...
## $ Housing : chr "own" "own" "own" "free" ...
## $ Saving accounts : chr NA "little" "little" "little" ...
## $ Checking account: chr "little" "moderate" NA "little" ...
## $ Credit amount : num 1169 5951 2096 7882 4870 ...
## $ Duration : num 6 48 12 42 24 36 24 36 12 30 ...
## $ Purpose : chr "radio/TV" "radio/TV" "education" "furniture/equipment" ...names(df)## [1] "...1" "Age" "Sex" "Job"
## [5] "Housing" "Saving accounts" "Checking account" "Credit amount"
## [9] "Duration" "Purpose"sapply(df,class)## ...1 Age Sex Job
## "numeric" "numeric" "character" "numeric"
## Housing Saving accounts Checking account Credit amount
## "character" "character" "character" "numeric"
## Duration Purpose
## "numeric" "character"df <- df[ , -1 ]
head(df)## Age Sex Job Housing Saving accounts Checking account Credit amount
## 1 67 male 2 own <NA> little 1169
## 2 22 female 2 own little moderate 5951
## 3 49 male 1 own little <NA> 2096
## 4 45 male 2 free little little 7882
## 5 53 male 2 free little little 4870
## 6 35 male 1 free <NA> <NA> 9055
## Duration Purpose
## 1 6 radio/TV
## 2 48 radio/TV
## 3 12 education
## 4 42 furniture/equipment
## 5 24 car
## 6 36 education2.1 Error Identification and Correction:
Identify inconsistencies, duplicate records, and inaccurate data entries.
Provide a clear explanation of how these errors were detected and rectified.
Since this data does not have customer ID codes, we check for duplicated records by comparing observations with each other using the duplicated() command. The result shows that these 1000 records are not duplicated.
table(duplicated(df))##
## FALSE
## 1000To check for inaccurate data entries, we use the table() command for character columns and the range() command for numeric columns.
table(df$Age, useNA = "always")##
## 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
## 2 14 14 27 48 44 41 50 51 43 37 40 38 34 33 32
## 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
## 40 39 29 24 21 25 17 22 17 17 15 18 17 12 14 12
## 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66
## 8 9 7 10 8 3 9 5 3 6 7 2 8 5 5 5
## 67 68 70 74 75 <NA>
## 3 3 1 4 2 0range(df$Age) # kết quả này phù hợp với nhóm khách hàng credit## [1] 19 75table(df$Sex, useNA = "always")##
## female male <NA>
## 310 690 0table(df$Job, useNA = "always")##
## 0 1 2 3 <NA>
## 22 200 630 148 0table(df$`Saving accounts`, useNA = "always")##
## little moderate quite rich rich <NA>
## 603 103 63 48 183# table(df$`Credit amount`)
summary(df$`Credit amount`)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 250 1366 2320 3271 3972 18424summary(df$Duration)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 4.0 12.0 18.0 20.9 24.0 72.0range(df$Duration)## [1] 4 722.2 Handling Missing Values:
Identify missing data points and determine the best strategy to address them.
In the ‘Saving accounts’ column, there are 183 observations with no NA information.
table(is.na(df$`Saving accounts`))##
## FALSE TRUE
## 817 183sapply(X = df,
FUN = function(x){ length(which(is.na(x))) } )## Age Sex Job Housing
## 0 0 0 0
## Saving accounts Checking account Credit amount Duration
## 183 394 0 0
## Purpose
## 0Discuss the methods used (e.g., mean/median imputation, forward-fill, or removal of affected records).
In the ‘Saving accounts’ column, there are 183 NA values. To fill in the missing values, we use a probabilistic method based on the proportion of the remaining values.
saving_account <- na.omit(df$`Saving accounts`)
attributes(saving_account)$na.action <- NULL
sample(x = saving_account,
size = 183,
replace = TRUE) -> saving_account_sim
prop.table(table(df$`Saving accounts`))##
## little moderate quite rich rich
## 0.73806610 0.12607099 0.07711138 0.05875153prop.table(table(saving_account_sim))## saving_account_sim
## little moderate quite rich rich
## 0.72677596 0.10928962 0.10928962 0.05464481df$`Saving accounts`[which(is.na(df$`Saving accounts`))] <- saving_account_simChecking account
checking_account <- na.omit(df$`Checking account`)
attributes(checking_account)$na.action <- NULL
sample(x = checking_account,
size = 394,
replace = TRUE) -> checking_account_sim
prop.table(table(df$`Checking account`))##
## little moderate rich
## 0.4521452 0.4438944 0.1039604prop.table(table(checking_account_sim))## checking_account_sim
## little moderate rich
## 0.4543147 0.4187817 0.1269036df$`Checking account`[which(is.na(df$`Checking account`))] <- checking_account_simJustify why your chosen approach is appropriate for this dataset.
2.3 Outlier Detection and Management:
Identify potential outliers that may distort the analysis.
Describe the method used to detect outliers (e.g., Z-score, IQR, visualization techniques).
sapply(df, class)## Age Sex Job Housing
## "numeric" "character" "numeric" "character"
## Saving accounts Checking account Credit amount Duration
## "character" "character" "numeric" "numeric"
## Purpose
## "character"An outlier is an extreme value, meaning a value that is unusually high or low compared to the majority of the values in a variable. In the case of this dataset, although the ‘Age’ and ‘Duration’ columns are numeric, they only represent information about the credit user. Therefore, we only need to consider outliers for the `Credit column.
# library(car)
# car:::qqPlot(df$Age)
# car:::qqPlot(df$Duration)
# car:::qqPlot(df$`Credit amount`,
# ylab="Credit Amount",
# main = "QQ plot for credit amount")
credit_outlier <- boxplot(df$`Credit amount`)
sort(credit_outlier$out)## [1] 7966 7980 8065 8072 8086 8133 8229 8318 8335 8358 8386 8471
## [13] 8487 8588 8613 8648 8858 8947 8978 9034 9055 9157 9271 9277
## [25] 9283 9398 9436 9566 9572 9629 9857 9960 10127 10144 10222 10297
## [37] 10366 10477 10623 10722 10875 10961 10974 11054 11328 11560 11590 11760
## [49] 11816 11938 11998 12169 12204 12389 12579 12612 12680 12749 12976 13756
## [61] 14027 14179 14318 14421 14555 14782 14896 15653 15672 15857 15945 18424# df[c(96, 916), ]
# summary(df$`Credit amount`)
# quantile(df$`Credit amount`,
# probs = c(0.75, 0.8, 0.85, 0.9, 0.95, 1))
#
# quantile(df$`Credit amount`)
#
# IQR(df$`Credit amount`)
1365.50 - 1.5*IQR(df$`Credit amount`)## [1] -2544.6253972.25 + 1.5*IQR(df$`Credit amount`)## [1] 7882.375# hist(df$`Credit amount`,
# breaks = 30)https://www.geo.fu-berlin.de/en/v/soga-r/Basics-of-statistics/Descriptive-Statistics/Measures-of-Position/Outliers-and-Boxplots/index.html
Decide whether to remove, transform, or retain the outliers and justify your choice.
We observe that, according to the IQR standard, there are 72 individuals with a credit amount considered as outliers. However, these values are still within a normal range (meaning that having a credit debt of up to 18,000 is not unusual). Therefore, I have decided to keep all of these values to include them in the subsequent analysis.
2.4 Presentation of Pre-processing Steps:
Bước 1: là đánh giá tổng thể dữ liệu sử dụng lệnh
table
lapply(df[ , -7], table)## $Age
##
## 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
## 2 14 14 27 48 44 41 50 51 43 37 40 38 34 33 32 40 39 29 24 21 25 17 22 17 17
## 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 70 74
## 15 18 17 12 14 12 8 9 7 10 8 3 9 5 3 6 7 2 8 5 5 5 3 3 1 4
## 75
## 2
##
## $Sex
##
## female male
## 310 690
##
## $Job
##
## 0 1 2 3
## 22 200 630 148
##
## $Housing
##
## free own rent
## 108 713 179
##
## $`Saving accounts`
##
## little moderate quite rich rich
## 736 123 83 58
##
## $`Checking account`
##
## little moderate rich
## 453 434 113
##
## $Duration
##
## 4 5 6 7 8 9 10 11 12 13 14 15 16 18 20 21 22 24 26 27
## 6 1 75 5 7 49 28 9 179 4 4 64 2 113 8 30 2 184 1 13
## 28 30 33 36 39 40 42 45 47 48 54 60 72
## 3 40 3 83 5 1 11 5 1 48 2 13 1
##
## $Purpose
##
## business car domestic appliances education
## 97 337 12 59
## furniture/equipment radio/TV repairs vacation/others
## 181 280 22 12summary(df$`Credit amount`)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 250 1366 2320 3271 3972 18424Step 2: is to clean the data, mainly filling in the NA values in the
Saving account and Checking account columns by
creating a vector containing random values based on the frequency of the
values in the columns to ensure the data is close to the real
values.
3 Data viz
3.1 Khảo sát ’Saving
accountsvàChecking account`
table(saving_account = df$`Saving accounts`,
checking_account = df$`Checking account`) -> saving_vs_checkingchisq.test(saving_vs_checking)##
## Pearson's Chi-squared test
##
## data: saving_vs_checking
## X-squared = 23.233, df = 6, p-value = 0.000722https://www.geo.fu-berlin.de/en/v/soga-r/Basics-of-statistics/Hypothesis-Tests/Chi-Square-Tests/Chi-Square-Independence-Test/index.html
3.2 Khảo sát
Job và Credit amount
# Job (numeric: 0 - unskilled and non-resident, 1 - unskilled and resident, 2 - skilled, 3 - highly skilled)
library(car)
df$Job_chr <- car:::recode(df$Job,
" '0' = 'unskilled and non-resident';
'1' = 'unskilled and resident';
'2' = 'skilled';
'3' = 'highly skilled';
else = NA ")library(ggplot2)
library(extrafont)
ggplot(data = df,
mapping = aes(x = Job,
y = `Credit amount`,
fill = Job_chr)) +
geom_boxplot() +
scale_fill_manual(values = c("green",
"coral",
"#eec750",
"lightblue")) +
theme_bw() +
theme(axis.text.x = element_text(angle = 90,
hjust = 1)) +
theme(text = element_text(family = "Times New Roman"))
## Khảo sát Credit amount, Job và
Purpose
addmargins(table(purpose = df$Purpose,
job = df$Job_chr))## job
## purpose highly skilled skilled unskilled and non-resident
## business 15 60 2
## car 69 190 12
## domestic appliances 0 10 1
## education 8 35 1
## furniture/equipment 21 126 1
## radio/TV 26 195 2
## repairs 0 13 2
## vacation/others 9 1 1
## Sum 148 630 22
## job
## purpose unskilled and resident Sum
## business 20 97
## car 66 337
## domestic appliances 1 12
## education 15 59
## furniture/equipment 33 181
## radio/TV 57 280
## repairs 7 22
## vacation/others 1 12
## Sum 200 10003.3 Khảo sát
Credit amount, Job và
Purpose
boxplot
library(ggplot2)
ggplot(data = df,
mapping = aes(x = Purpose,
y = `Credit amount`,
fill = Purpose)) +
geom_boxplot() +
facet_wrap(~Job_chr) +
theme_bw() +
theme(axis.text.x = element_text(angle = 90,
hjust = 1))
mean col
library(ggplot2)
library(dplyr)
df |> dplyr:::group_by(Job_chr, Purpose) |>
dplyr:::summarise(mean_credit = mean(`Credit amount`),
person = n()) -> df_sum
df_sum <- as.data.frame(df_sum)
ggplot(data = df_sum,
mapping = aes(x = Purpose,
y = mean_credit,
fill = Purpose)) +
geom_col() +
facet_wrap(~Job_chr) +
theme_bw() +
theme(axis.text.x = element_text(angle = 90,
hjust = 1))
3.4 Khảo sát
Age, ’Purpose’và Credit amount
ggplot(data=df,
mapping=aes(x=Age,
y=`Credit amount`,
color=Age)) +
geom_point() +
# geom_smooth(method = "lm") +
facet_wrap(~Purpose,
scales = "free")
cor.test(df$Age,
df$`Credit amount`)##
## Pearson's product-moment correlation
##
## data: df$Age and df$`Credit amount`
## t = 1.0341, df = 998, p-value = 0.3013
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
## -0.02933617 0.09451780
## sample estimates:
## cor
## 0.03271642cor.test(df$`Credit amount`,
df$Job)##
## Pearson's product-moment correlation
##
## data: df$`Credit amount` and df$Job
## t = 9.4069, df = 998, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
## 0.2274157 0.3413395
## sample estimates:
## cor
## 0.2853853library(dplyr)
df |> dplyr:::group_by(`Saving accounts`) |>
dplyr:::summarise(mean_credit = mean(`Credit amount`),
person = n()) -> df_sum
library(ggplot2)
ggplot(data = df_sum,
mapping = aes(x = `Saving accounts`,
y = person,
fill = `Saving accounts`)) +
geom_col() +
scale_fill_manual(values = c("green",
"coral",
"#eec750",
"lightblue"))
3.5 Khảo sát ‘Age’, ‘Credit amount’và ’Duration’
df_cor <- df[ , c("Age", "Credit amount",
"Duration")]
cor_matrix <- cor(df_cor)
ggplot(data=df,
mapping=aes(x=Duration,
y=`Credit amount`)) +
geom_point() +
geom_smooth() +
theme_bw()
4 So sánh
Credit ammount giữa male và female
df_male <- df |> subset(Sex == "male")
mean(df_male$`Credit amount`)## [1] 3448.041df_female <- df |> subset(Sex == "female")
mean(df_female$`Credit amount`)## [1] 2877.774### check normality
library(car)
car:::qqPlot(df_male$`Credit amount`)
## [1] 69 558car:::qqPlot(df_female$`Credit amount`)
## [1] 288 112### check variance
var.test(x = df_male$`Credit amount`,
y = df_female$`Credit amount`)##
## F test to compare two variances
##
## data: df_male$`Credit amount` and df_female$`Credit amount`
## F = 1.2415, num df = 689, denom df = 309, p-value = 0.02863
## alternative hypothesis: true ratio of variances is not equal to 1
## 95 percent confidence interval:
## 1.023039 1.496141
## sample estimates:
## ratio of variances
## 1.241496# p-value = 0.0286 < 0.05 cho thấy phương sai của hai vector này không bằng nhau
t.test(x = df_male$`Credit amount`,
y = df_female$`Credit amount`)##
## Welch Two Sample t-test
##
## data: df_male$`Credit amount` and df_female$`Credit amount`
## t = 3.0904, df = 658.03, p-value = 0.002084
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 207.9260 932.6068
## sample estimates:
## mean of x mean of y
## 3448.041 2877.774wilcox.test(x = df_male$`Credit amount`,
y = df_female$`Credit amount`)##
## Wilcoxon rank sum test with continuity correction
##
## data: df_male$`Credit amount` and df_female$`Credit amount`
## W = 122417, p-value = 0.0002507
## alternative hypothesis: true location shift is not equal to 0p-value = 0.002 < 0.05 shows that the average credit ammount of male is different from female at the significance level of 0.05, or specifically, male has significantly higher credit ammount than female.
Because the credit ammount data of male and female are not normally distributed, in this case we test the difference between these two groups using wilcoxon test, we have p-value = 0.00025 < 0.05 for the same conclusion as t-test.
5 References
https://www.kaggle.com/datasets/uciml/german-credit
https://www.geo.fu-berlin.de/en/v/soga-r/Basics-of-statistics/Discrete-Random-Variables/The-Mean-and-Standard-Deviation/index.html
https://www.data-to-viz.com/#barplot