Introduction & RPubs link information

• This report is published to RPubs here.
• This section is a quick introduction to provide a brief background to the problem, to define important terms, and lead to a strong rationale.
• In this assignment, I will be exploring the impact of a person’s demographic attributes on the person’s insurance details in the US like insurance premium charges etc.
• In practice, the premium is decided based on a score/value that is calculated based on important attributes involved in the decision process
• I will start the experiment by importing an open-source dataset from kaggle corresponding to insurance details in the US
• I’ll proceed to explore and describe the data and briefly explore uni-variate statistics and plots to understand the properties of the attributes in focus
• I’ll move on to calculating summary statistics of the focus attributes at the relevant levels and identify important summary characteristics
• I’ll formulate directed hypotheses about a research question in focus (defined soon, in the upcoming problem statement) and define the null, alternate hypothesis for the question in focus

Introduction Cont.

• Based on the results, i will either reject or fail-to-reject the null hypothesis
• Based on the above result, i will briefly discuss and present my findings and provide the conclusion on the research question in focus.
• I will add relevant images to make the presentation more appealing

Data Details:

• The data is imported from open-source kaggle link
• The 7 columns that are present in the available dataset - age, sex, bmi, children, smoker, region, charges
• We will be focusing on the impact of ‘sex’ on ‘charges’ AKA premium charges
• sex is a binary variable with 2 levels - ‘male’ and ‘female’
• Our focus will be on checking is there is any difference in the charges across males and females
• We will formulate our null hypothesis, H0 : There is no difference between male and female charges
• We will explore the average charges across both and either reject/fail to reject this hypothesis thus making a finding

Problem Statement & Data

• Problem statement: Check the difference between the insurance premium charges between male and females in the US

• I will explore the data in R, generate summary statistics, visually explore data, formulate hypothesis, reject/fail-to-reject the null hypothesis and analyze the statistical results to solve the problem statement selected

• Data: I collected the data from kaggle at https://www.kaggle.com/teertha/ushealthinsurancedataset and it is open source

• This includes various demographic attributes of a person and the corresponding insurance charges

• Demographics included are 7 in total and are : age, sex, bmi, children, smoker, region, charges

• I will focus on the impact of sex on premium charges, and will focus on 2 variables ‘sex’ and ‘charges’

• The dataset includes the charges for 1,338 people

• Sex is binary and contains ‘male’ & ’female

• This is considered a factor variable and will be converted to the same during data exploration and preparation

Data Cont.

• charges is a continuous variable and ranges from $1,121.8 to $62,770
• The data doesn’t contain any missing or undefined values, and will not require any imputation analysis
• Further details on the data generation are not provided at the source and i have assumed it to be user generated
• The above analysis can be extended to other attributes based on the research question chosen but will not be in scope of current analysis Other variables are as below:
• age is a factor variable with range 18 to 64
• Body Mass Index (bmi) ranging 15.96 to 53.13
• children is a factor variable from 0 to 5 (means count of children)
• smoker (flag) corresponds to a smoking status with ‘yes’ and ‘no’ (This can be an interesting research question to explore next)
• region is a factor variable having 4 locations ‘northeast’, ‘northwest’, ‘southeast’, ‘southwest’

Data Import, Descriptive Statistics and Visualisation

#Import csv data and visualize top 5 rows
insurance <- read.csv("D:/Material/Applied Analytics/Assignments/Assignment 2/insurance.csv")
insurance %>% head()

#convert sex and other relevant factor variables to factor() type
insurance$sex <- as.factor(insurance$sex)
insurance$smoker <- as.factor(insurance$smoker)
insurance$region <- as.factor(insurance$region)
insurance[c("sex", "smoker", "region")] %>% str() #converted to factor

## 'data.frame':    1338 obs. of  3 variables:
##  $ sex   : Factor w/ 2 levels "female","male": 1 2 2 2 2 1 1 1 2 1 ...
##  $ smoker: Factor w/ 2 levels "no","yes": 2 1 1 1 1 1 1 1 1 1 ...
##  $ region: Factor w/ 4 levels "northeast","northwest",..: 4 3 3 2 2 3 3 2 1 2 ...

Descriptive Statistics and Visualization - Cont.

#Check for missing values and quickly check statistical attributes for each column
colSums(is.na(insurance)) #No missing values

##      age      sex      bmi children   smoker   region  charges 
##        0        0        0        0        0        0        0

summary(insurance)

##       age            sex           bmi           children     smoker    
##  Min.   :18.00   female:662   Min.   :15.96   Min.   :0.000   no :1064  
##  1st Qu.:27.00   male  :676   1st Qu.:26.30   1st Qu.:0.000   yes: 274  
##  Median :39.00                Median :30.40   Median :1.000             
##  Mean   :39.21                Mean   :30.66   Mean   :1.095             
##  3rd Qu.:51.00                3rd Qu.:34.69   3rd Qu.:2.000             
##  Max.   :64.00                Max.   :53.13   Max.   :5.000             
##        region       charges     
##  northeast:324   Min.   : 1122  
##  northwest:325   1st Qu.: 4740  
##  southeast:364   Median : 9382  
##  southwest:325   Mean   :13270  
##                  3rd Qu.:16640  
##                  Max.   :63770

Descriptive Statistics and Visualization - Cont.

#Split dataset into male and female for descriptive statistics. Also, keep relevant variables only
insurance <- insurance[, c("sex", "charges")]
male_i <- insurance %>% filter(sex=='male');
female_i <- insurance %>% filter(sex=='female')

#Visualize the data for male and female sexes using histograms
par(mfrow = c(2,2)); 
male_i$charges %>% hist(main="Distribution of charges for male", xlab = "male charges", col = "blue")
female_i$charges %>% hist(main="Distribution of charges for female", xlab = "female charges", col = "red")

Descriptive Statistics and Visualization - Cont.

#Visualize the data for male and female sexes using boxplots. The interpretation of the graphs is done in the following text block
par(mfrow = c(1,2))
male_i$charges %>% boxplot(main="boxplot of charges for male", ylab = "male charges", col = "blue")
female_i$charges %>% boxplot(main="boxplot of charges for female", ylab = "female charges", col = "red")

Descriptive Statistics and Visualization - Findings:

1. Both male and female charges are skewed to the right with considerable tails
   
2. male charges seem to have more values around 40000 value range while compared to females as seen in the histograms
   
3. Both males and females seem to have multiple outliers in the tails
4. As seen in the boxplot, females do have more outliers than male as the IQR range is smaller in females
5. There aren’t any inconsistent values for any of the distributions (both are bound to a range of (0, ~60000) which is understood to be normal for a currency column)

Decsriptive Statistics and Visualization - Cont. 2

#Find summary statistics all-up
insurance %>% summarise(Min = min(charges,na.rm = TRUE), Q1 = quantile(charges,probs = .25,na.rm = TRUE),
                                           Median = median(charges, na.rm = TRUE), Q3 = quantile(charges,probs = .75,na.rm = TRUE),
                                           Max = max(charges,na.rm = TRUE), Mean = mean(charges, na.rm = TRUE),
                                           SD = sd(charges, na.rm = TRUE), n = n(), Missing = sum(is.na(charges))) -> table1
table1 <- cbind(sex="All", table1) #Add label for sex as 'All' (To be rbinded with sex-level stats next)
#Find summary statistics by sex
insurance %>% group_by(sex) %>% summarise(Min = min(charges,na.rm = TRUE),  Q1 = quantile(charges,probs = .25,na.rm = TRUE),
                                           Median = median(charges, na.rm = TRUE), Q3 = quantile(charges,probs = .75,na.rm = TRUE),
                                           Max = max(charges,na.rm = TRUE), Mean = mean(charges, na.rm = TRUE),
                                           SD = sd(charges, na.rm = TRUE), n = n(), Missing = sum(is.na(charges))) -> table2
table3 <- rbind(table1, table2)
knitr::kable(table3)

Descriptive Statistics and Visualization - Findings 2:

1. Males have a higher mean of 13956.75 while females have slightly lower mean of 12569.58 (This shows that males are likely to have higher charges on average, but we will need to do further analysis to understand if this really is the case)
   
2. Males have a larger inter-quartile range (4619.134, 18989.59) thus leading to considerably lower outliers as compared to females which have a smaller range (4885.159, 14454.69) for IQR.
   
3. Medians are closer to each other (male, female) as compared to the variation in mean due to the presence of heavy outliers which skews the mean up.
   
4. The all-up statistics seem to be in between male and female with mean = 13270.42 and median = 9382.033 (Total observations = 1338)
5. While the number of observations for male and female are not equal, they are well in similar range with 676 and 662 observations respectively. Given the closeness of the range, we can treat these as similar clusters of observations.
   
6. Also, since sample size is quite large i.e. >30, the analysis can be carried out without concerning with the normality of the underlying data. If this was <30, then normality would have to be checked. I will still plot the normality tests, i.e. qqplots subsequently and briefly.
   
7. We will next move to hypothesis testing to do further analysis and get more concrete results

Hypothesis Testing

• To explore the equality of charges across males and females, we are going to use a 2-sample t-test to explore the difference, if it exists
• We are going to formulate the below 2 hypothesis to provide direction to our analysis
  • Null Hypothesis: The average charges for males is the same as the average charges for female \[H_0: \mu_1 = \mu_2 \]
  • Alternate Hypothesis: The average charges for males is NOT the same as the average charges for female \[H_A: \mu_1 \ne \mu_2\]

Hypothesis Testing - Cont.

• Assumptions to analyze before proceeding with a 2-sample t-test:

1. If sample size <30, then normality is expected from the samples. If its larger, then normality is not mandatory since large samples generally lead to normality of sampling distribution. T-tests are generally robust against minor departures from normality
   • As seen in the subsequent qqplots, the male and female samples are not exactly normal as seen in the tails which digress form the diagonal line). Fortunately, since we have large sample size, this is not a deal breaker
2. Male, Female population is expected to be independent of each other. Given that male and female data is mutually exclusive and separated earlier, this is verified as independent. And of course, they are independent in common sense as well
3. 2-sample test also expects equal variance among the 2 samples. (This is also checked below using lavene test). If unequal variance, then relevant parameters have to be used while running a t-test (var.equal = FALSE)
   • As seen in the subsequent levene test, p < 0.05 and thus we cannot assume equal variance (i.e. we will have to use var.equal = FALSE in t test)
4. If the above assumptions are satisfied, then we can proceed with the t-test appropriately (next section)

Hypothesis Testing - Cont.

#Check for equal variances
leveneTest(charges ~ sex ,data = insurance) #unequal variance identified due to p < 0.05

#Check for normality
par(mfrow = (c(2,2)))
male_i$charges %>% qqPlot(dist="norm", main = "male charges vs theory_quantiles")

## [1] 658 623

female_i$charges %>% qqPlot(dist="norm", main = "female charges vs theory_quantiles")

## [1] 266 286

Hypothesis Testing Cont.

• Hypothesis test is executed below

t.test(data = insurance, charges ~ sex, var.equal = FALSE, alternative = "two.sided")

## 
##  Welch Two Sample t-test
## 
## data:  charges by sex
## t = -2.1009, df = 1313.4, p-value = 0.03584
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -2682.48932   -91.85535
## sample estimates:
## mean in group female   mean in group male 
##             12569.58             13956.75

• As seen in the t-test, p value = 0.036 which is, <0.05. Hence the null hypothesis is rejected and we can conclude that the average charges for males and female are not the same.
• We need to identify which in fact of the 2 has higher average. We can run the experiment for a 1-sided test. But as we have seen from the results of the t-test that this information is already provided.
• Mean of group male = ~13.9k and mean of group female = ~12.6k, hence we can identify that males have higher premium charges as compared to females. Thus we have provided a conclusion to the experiment in focus.

Discussion

• We started the analysis by selecting a research question to answer, which was to check if insurance charges are, on an average, the same across both male and female
  
• We identified an appropriate dataset from kaggle to solve the problem. We leveraged the sex and charges columns from the data for our analysis
• We explored the characteristics of the general data and specifically charges column, overall and across the two sexes to obtain initial findings. We found the below details
  • charges range from $1,122 to $63,770 and the data didnt have any missing values
  • sex is a character column, which was converted to a factor variable
  • We used histograms to explore the normality of both male and female charges, which turned out to be non-normal and skewed to the right with multiple outliers
  • We validated the same using boxplots where females had more outliers than male, which was attributed to the IQR of males being larger than females, thus generating more outliers for a otherwise similar spread of distribution
  • We explored the summary statistics of both male and female charges and saw that the male mean was higher than female (due to outliers) and the medians were closer to each other in comparison
  • both male and female observations were similar in number (676, 662 respectively), which also helped us to validate that the normality of the samples was not critical due to the large sample size

Discussion - Cont.

• We proceeded with hypothesis testing next
• We generated null hypothesis, H0: average charges across both male and females are same
• We then analyzed the requirements/assumptions around t tests which required the following - 1. data independence (which was present), 2. similar variance (which wasn’t there) and sample normality (which wasn’t normal)
• We used qqplots and levene test and data exploration to find the above details
• We took relevant decisions to tailor subsequent steps in line with the standard approach (like using var.equal = FALSE in t-test)
• t-test was implemented with relevant modifications as per the sample characterestics
  • it was found that the p-value, p was <0.05, due to which, we rejected the null hypothesis and accepted the alternate hypothesis
  • Alternate hypothesis concluded that the average charges across males and female are NOT the same
  • Based on further analysis of the results, we deduced that males in fact have HIGHER average charges as compared to females
• The above analysis is done for sex, and can be extended to other demographic attributes like age, region, smoker type etc, which will follow a similar approach
• Usually, either 2-sample tests can be implemented (sample 1 avg = sample 2 avg) or a 1-sample test can be implemented at an overall level (sample avg == research value)

References

Below is a list of any references i have used in the presentation.

• https://astral-theory-157510.appspot.com/secured/MATH1324_Module_07.html#
  
• https://www.kaggle.com/teertha/ushealthinsurancedataset
  
• https://stats.stackexchange.com/questions/18803/data-sets-and-problems-for-learning-hypothesis-testing