Problem Statement
Gender bias in earnings is an ongoing issue. In 2019, the American Association of University Women (AAUW) stated that women who work full-time earn about 80 percent of what their male counterparts make. This report identified the following problems which we attempt to investigate in our study:
Data and Methodology
For this study, we would use the Women in Workforce data which is a historical data about womens’ earnings and employment status, specific occupation and earnings from 2013-2016, compiled from the Bureau of Labor Statistics and the Census Bureau. We intend to analyze the data using the following methodologies: Trend analysis, descriptive analysis, data visualization, and applicable statistical tests.
Proposed Analysis
We would investigate the gender pay gap over the years and the changes in the total female salary as a percentage of male salary by age group. We would conduct an exploratory analysis across the different fields/occupation groups to identify the male-dominated vs female-dominated occupation types. Some statistical tests would be employed to determine the differences in pay gap across fields and by age group. Finally, we hope to identify some factors contributing to the differences in wage gap using regression analysis.
Our Contribution
Our results would provide more insights on the gender pay gap across fields, occupations, and age groups. Our results would show the trend in gender pay gap over the years and some of the factors contributing to gender pay gap. We hope that our study would contribute to the discussion on gender pay gap.
We used the following packages:
library(readxl) #used to import Excel files into R
library(tidyverse) #used for data manipulation
library(dplyr) # used for data manipulation
library(DT) # used for displaying R data objects (matrices or data frames) as tables on HTML pages
library(knitr) #used to display an aligned table on the screen
library(kableExtra)#used to build with straightforward formatting optionsFor this study, we use the Women in Workforce data which is a historical data about womens’ earnings and employment status, specific occupation and earnings from 2013-2016, compiled from the Bureau of Labor Statistics and the Census Bureau. The data was provided in March 2019 as part of the #TidyTuesday project to celebrate the Women’s History month.
The entire data is spread into 3 files: jobs_gender.csv, earnings_female.csv, employed_gender.csv and are described in the next tab.
The three datasets are first imported from csv files into dataframes named jobs_gender, earnings_female,and employed_gender
This dataset contains information on the total number of male and female workers and the total estimated median earnings for all employees at various occupation levels, from 2013-2016. The dataset has 12 variables, with a total of 2,088 data points, across 8 major job categories, 23 minor job categories and 522 occupation types. There are some missing values recorded as “NA” and are taken care of during the data cleaning process.
More information on this dataset can be found here
#Import Dataset 1
jobs_gender <- read.csv("~/BANA/Coursework/BANA 7025 Data wrangling/Mid term assignment/women_in_the_workplace_data/jobs_gender.csv", header = TRUE)#Check the names of the data columns
colnames(jobs_gender)## [1] "year" "occupation"
## [3] "major_category" "minor_category"
## [5] "total_workers" "workers_male"
## [7] "workers_female" "percent_female"
## [9] "total_earnings" "total_earnings_male"
## [11] "total_earnings_female" "wage_percent_of_male"#Check the dimension of the dataset
dim(jobs_gender)## [1] 2088 12#Count the number of distinct values/observations
library(dplyr)
jobs_gender %>%
summarise_each(n_distinct, "occupation","minor_category", "major_category")## occupation minor_category major_category
## 1 522 23 8#Check the number of missing values per column
colSums(is.na(jobs_gender))## year occupation major_category
## 0 0 0
## minor_category total_workers workers_male
## 0 0 0
## workers_female percent_female total_earnings
## 0 0 0
## total_earnings_male total_earnings_female wage_percent_of_male
## 4 65 846This dataset contains the historic information of female salary as a percent of male salary for various age groups, from year 1979 to 2011. The dataset has 3 variables with 264 observations. This dataset has no missing values.
The dataset can be found here
#Imports Dataset 2
earnings_female <- read.csv("~/BANA/Coursework/BANA 7025 Data wrangling/Mid term assignment/women_in_the_workplace_data/earnings_female.csv", header = TRUE)#Check the names of the data columns
colnames(earnings_female)## [1] "Year" "group" "percent"#Check the dimension of the dataset
dim(earnings_female)## [1] 264 3#Count the number of missing values per column
colSums(is.na(earnings_female))## Year group percent
## 0 0 0This dataset shows the percentage of part-time and full-time employees for each year at the gender level. The dataset has 7 variables with 49 observations each, from year 1968 to 2016. This dataset has no missing values and can be accessed here
#Import Dataset 3
employed_gender <- read.csv("~/BANA/Coursework/BANA 7025 Data wrangling/Mid term assignment/women_in_the_workplace_data/employed_gender.csv", header = TRUE)#Check the names of the data columns
colnames(employed_gender)## [1] "year" "total_full_time" "total_part_time"
## [4] "full_time_female" "part_time_female" "full_time_male"
## [7] "part_time_male"#Check the dimension of the dataset
dim(employed_gender)## [1] 49 7#Count the number of missing values per column
colSums(is.na(employed_gender))## year total_full_time total_part_time full_time_female
## 0 0 0 0
## part_time_female full_time_male part_time_male
## 0 0 0Our data cleaning focuses on Dataset 1 - jobs_gender because this is the only dataset with some missing values.
First, we notice that there are 65 cases with missing values for the total earnings for female workers. Of this 65 cases, there are 19 cases where the total number of female workers in the field or occupation types is zero and the total earnings for female is missing “NA”. We impute these missing values as zero because no female workers implies no total female earnings.
#Impute missing values as zero
jobs_gender$total_earnings_female <- ifelse(jobs_gender$workers_female == 0, 0, jobs_gender$total_earnings_female)For the remaining 46 observations with missing values for the total female earnings,we observe that the number of female workers is reported, indicating that these values are non-zero and missing. Hence, we remove the entire row from the dataset. We believe that this removal would not affect our analysis and goals for this study because, it’s just about 2 percent of the entire dataset.
#Remove rows with missing values for female earnings
jobs_gender_vector <- complete.cases(jobs_gender[,'total_earnings_female'])
jobs_gender <- jobs_gender[jobs_gender_vector,]Next, we notice that there are 4 observations where the total earnings for male workers is missing. In 3 of these cases, the number of male workers is zero and the total earnings for male is missing “NA”. A closer look at these three observations revealed that in 2013, 2014, and 2016, there were no male workers who worked as “Nurse Midwives”. Therefore, we imputed the missing total male earnings as zero because no male workers implies no male earnings.
#Impute missing values as zero
jobs_gender$total_earnings_male <- ifelse(jobs_gender$workers_male == 0, 0, jobs_gender$total_earnings_male)In the fourth case which was in 2015, a total of 53 male workers was reported for Nurse midwives. This is very different from what was reported in 2013, 2014, and 2016. This could mean that the occupation type (Nurse Midwives) had male workers in 2015 or an error occurred during the data reporting process. We decide to remove the entire row associated with this observation not because of the anomaly, but because the male total earnings is missing and reported as “NA”.
#Remove row with missing values for male earnings
jobs_gender_m_vector <- complete.cases(jobs_gender[,'total_earnings_male'])
jobs_gender <- jobs_gender[jobs_gender_m_vector,]Finally, we notice that the column “wage_percent_of_male” showing the percentage of the total female earnings to total male earnings has a total of 846 missing values. Given that this is the percentage of the total female earnings to the total male earnings, we calculate and impute this variable using the total_earnings_male and total_earning_female columns.
#Calculate and impute results for wage_percent_of_male
jobs_gender$wage_percent_of_male<- ifelse(is.na(jobs_gender$wage_percent_of_male),jobs_gender$total_earnings_female/jobs_gender$total_earnings_male*100,jobs_gender$wage_percent_of_male)Dataset 1 - jobs_gender
Dataset 2 - earnings_female
Dataset 3- employed_gender
Dataset 1 - jobs_gender
| Variable | Description |
|---|---|
| year | Year |
| occupation | Specific job/career |
| major_category | Broad category of occupation |
| minor_category | Fine category of occupation |
| total_workers | Total estimated full-time workers > 16 years old |
| workers_male | Estimated MALE full-time workers > 16 years old |
| workers_female | Estimated FEMALE full-time workers > 16 years old |
| percent_female | The percent of females for specific occupation |
| total_earnings | Total estimated median earnings for full-time workers > 16 years old |
| total_earnings_male | Estimated MALE median earnings for full-time workers > 16 years old |
| total_earnings_female | Estimated FEMALE median earnings for full-time workers > 16 years old |
| wage_percent_of_male | Female wages as percent of male wages - NA for occupations with small sample size |
Dataset 2 - earnings_female
| Variable | Description |
|---|---|
| Year | Year |
| group | Age group |
| percent | Female salary percent of male salary |
Dataset 3 - employed_gender
| Variable | Description |
|---|---|
| year | Year |
| total_full_time | Percent of total employed people usually working full time |
| total_part_time | Percent of total employed people usually working part time |
| full_time_female | Percent of employed women usually working full time |
| part_time_female | Percent of employed women usually working part time |
| full_time_male | Percent of employed men usually working full time |
| part_time_male | Percent of employed men usually working part time |
To investigate and address the problem statement, we would explore the individual datasets and combine them where appropriate. Our proposed approach is as follows:
Descriptive Analysis
We would provide some descriptive statistics and plots such as bar charts, line charts, and ribbon charts showing the representation of male and female workers across the occupation types, minor and major job groups. We would show the major job groups that are male-dominated vs female-dominated; summary statistics of earnings by gender; and the job types with the widest, narrowest, or no gender pay gap.
Trend Analysis with Interactive Plots
We would examine the gender wage gap over time through a trend analysis. This would show the reported and the change in median earnings for male and female workers from 2013 to 2016. Using the combined dataset, we would explore the percentage change in the number of full-time and part-time workers across both genders, from 1968 to 2016. We would employ bar charts, ribbon charts, and interactive plots to show how the trend has evolved over the years.
Analysis of Variance (ANOVA) Test
We would investigate if there’s an increase in pay gap as women get older. To achieve this, we would create a line chart showing female salary as a percentage of male salary across different age groups, from 1979 to 2011. We would also consider an Analysis of Variance Test (ANOVA) to test the hypothesis that the gender pay gap gets wider as women get older. Is there a difference in the mean earnings of women as a percentage of male earnings across different age groups?
Further Data Slicing and Exploration
The data would be sliced to investigate the claims that women make more than men in the five occupations (wholesale and retail buyers except farm products; police and sheriffs patrol officers; bookkeeping, accounting, and auditing clerks; general office clerks; and data entry keepers) and less in pink-collar jobs. The results would be displayed using tables and some summary statistics. We would further explore our dataset to see if some job types changed from been male-dominated to female-dominated and vice versa. If yes, did the earnings change as a result?
Machine Learning Technique: Linear Regression Analysis
Finally, we would employ the linear regression analysis technique to identify some of the factors that contribute to the differences in wage gap.
Need: More Expertise and Information