In the wake of the Great Recession of 2009, there has been a good deal of focus on employment statistics, one of the most important metrics policymakers use to gauge the overall strength of the economy. In the United States, the government measures unemployment using the Current Population Survey (CPS), which collects demographic and employment information from a wide range of Americans each month. In this exercise, we will employ the topics reviewed in the lectures as well as a few new techniques using the September 2013 version of this rich, nationally representative dataset (available online).
The observations in the dataset represent people surveyed in the September 2013 CPS who actually completed a survey. While the full dataset has 385 variables, in this exercise we will use a more compact version of the dataset, CPSData.csv, which has the following variables:
PeopleInHousehold: The number of people in the interviewee’s household.
Region: The census region where the interviewee lives.
State: The state where the interviewee lives.
MetroAreaCode: A code that identifies the metropolitan area in which the interviewee lives (missing if the interviewee does not live in a metropolitan area). The mapping from codes to names of metropolitan areas is provided in the file MetroAreaCodes.csv.
Age: The age, in years, of the interviewee. 80 represents people aged 80-84, and 85 represents people aged 85 and higher.
Married: The marriage status of the interviewee.
Sex: The sex of the interviewee.
Education: The maximum level of education obtained by the interviewee.
Race: The race of the interviewee.
Hispanic: Whether the interviewee is of Hispanic ethnicity.
CountryOfBirthCode: A code identifying the country of birth of the interviewee. The mapping from codes to names of countries is provided in the file CountryCodes.csv.
Citizenship: The United States citizenship status of the interviewee.
EmploymentStatus: The status of employment of the interviewee.
Industry: The industry of employment of the interviewee (only available if they are employed).
Load the dataset from CPSData.csv into a data frame called CPS, and view the dataset with the summary() and str() commands.
str(CPSData)Classes ‘tbl_df’, ‘tbl’ and 'data.frame': 131302 obs. of 14 variables:
$ PeopleInHousehold : int 1 3 3 3 3 3 3 2 2 2 ...
$ Region : chr "South" "South" "South" "South" ...
$ State : chr "Alabama" "Alabama" "Alabama" "Alabama" ...
$ MetroAreaCode : int 26620 13820 13820 13820 26620 26620 26620 33660 33660 26620 ...
$ Age : int 85 21 37 18 52 24 26 71 43 52 ...
$ Married : chr "Widowed" "Never Married" "Never Married" "Never Married" ...
$ Sex : chr "Female" "Male" "Female" "Male" ...
$ Education : chr "Associate degree" "High school" "High school" "No high school diploma" ...
$ Race : chr "White" "Black" "Black" "Black" ...
$ Hispanic : int 0 0 0 0 0 0 0 0 0 0 ...
$ CountryOfBirthCode: int 57 57 57 57 57 57 57 57 57 57 ...
$ Citizenship : chr "Citizen, Native" "Citizen, Native" "Citizen, Native" "Citizen, Native" ...
$ EmploymentStatus : chr "Retired" "Unemployed" "Disabled" "Not in Labor Force" ...
$ Industry : chr NA "Professional and business services" NA NA ...
- attr(*, "spec")=List of 2
..$ cols :List of 14
.. ..$ PeopleInHousehold : list()
.. .. ..- attr(*, "class")= chr "collector_integer" "collector"
.. ..$ Region : list()
.. .. ..- attr(*, "class")= chr "collector_character" "collector"
.. ..$ State : list()
.. .. ..- attr(*, "class")= chr "collector_character" "collector"
.. ..$ MetroAreaCode : list()
.. .. ..- attr(*, "class")= chr "collector_integer" "collector"
.. ..$ Age : list()
.. .. ..- attr(*, "class")= chr "collector_integer" "collector"
.. ..$ Married : list()
.. .. ..- attr(*, "class")= chr "collector_character" "collector"
.. ..$ Sex : list()
.. .. ..- attr(*, "class")= chr "collector_character" "collector"
.. ..$ Education : list()
.. .. ..- attr(*, "class")= chr "collector_character" "collector"
.. ..$ Race : list()
.. .. ..- attr(*, "class")= chr "collector_character" "collector"
.. ..$ Hispanic : list()
.. .. ..- attr(*, "class")= chr "collector_integer" "collector"
.. ..$ CountryOfBirthCode: list()
.. .. ..- attr(*, "class")= chr "collector_integer" "collector"
.. ..$ Citizenship : list()
.. .. ..- attr(*, "class")= chr "collector_character" "collector"
.. ..$ EmploymentStatus : list()
.. .. ..- attr(*, "class")= chr "collector_character" "collector"
.. ..$ Industry : list()
.. .. ..- attr(*, "class")= chr "collector_character" "collector"
..$ default: list()
.. ..- attr(*, "class")= chr "collector_guess" "collector"
..- attr(*, "class")= chr "col_spec"131302[1] 131302Among the interviewees with a value reported for the Industry variable, what is the most common industry of employment? Please enter the name exactly how you see it.
sort(table(CPSData$Industry))
Armed forces Mining
29 550
Agriculture, forestry, fishing, and hunting Information
1307 1328
Public administration Other services
3186 3224
Transportation and utilities Financial
3260 4347
Construction Leisure and hospitality
4387 6364
Manufacturing Professional and business services
6791 7519
Trade Educational and health services
8933 15017 "Educational and health services"[1] "Educational and health services"Recall from the homework assignment “The Analytical Detective” that you can call the sort() function on the output of the table() function to obtain a sorted breakdown of a variable. For instance, sort(table(CPS$Region)) sorts the regions by the number of interviewees from that region.
Which state has the fewest interviewees?
sort(table(CPSData$State))
New Mexico Montana Mississippi Alabama
1102 1214 1230 1376
West Virginia Arkansas Louisiana Idaho
1409 1421 1450 1518
Oklahoma Arizona Alaska Wyoming
1523 1528 1590 1624
North Dakota South Carolina Tennessee District of Columbia
1645 1658 1784 1791
Kentucky Utah Nevada Vermont
1841 1842 1856 1890
Kansas Oregon Nebraska Massachusetts
1935 1943 1949 1987
South Dakota Indiana Hawaii Missouri
2000 2004 2099 2145
Rhode Island Delaware Maine Washington
2209 2214 2263 2366
Iowa New Jersey North Carolina New Hampshire
2528 2567 2619 2662
Wisconsin Georgia Connecticut Colorado
2686 2807 2836 2925
Virginia Michigan Minnesota Maryland
2953 3063 3139 3200
Ohio Illinois Pennsylvania Florida
3678 3912 3930 5149
New York Texas California
5595 7077 11570 Which state has the largest number of interviewees?
"California"[1] "California"What proportion of interviewees are citizens of the United States?
table(CPSData$Citizenship)
Citizen, Native Citizen, Naturalized Non-Citizen
116639 7073 7590 123712/131302[1] 0.9421943The CPS differentiates between race (with possible values American Indian, Asian, Black, Pacific Islander, White, or Multiracial) and ethnicity. A number of interviewees are of Hispanic ethnicity, as captured by the Hispanic variable. For which races are there at least 250 interviewees in the CPS dataset of Hispanic ethnicity? (Select all that apply.)
table(CPSData$Race, CPSData$Hispanic)
0 1
American Indian 1129 304
Asian 6407 113
Black 13292 621
Multiracial 2449 448
Pacific Islander 541 77
White 89190 16731Which variables have at least one interviewee with a missing (NA) value? (Select all that apply.)
summary(CPSData) PeopleInHousehold Region State MetroAreaCode Age
Min. : 1.000 Length:131302 Length:131302 Min. :10420 Min. : 0.00
1st Qu.: 2.000 Class :character Class :character 1st Qu.:21780 1st Qu.:19.00
Median : 3.000 Mode :character Mode :character Median :34740 Median :39.00
Mean : 3.284 Mean :35075 Mean :38.83
3rd Qu.: 4.000 3rd Qu.:41860 3rd Qu.:57.00
Max. :15.000 Max. :79600 Max. :85.00
NA's :34238
Married Sex Education Race Hispanic
Length:131302 Length:131302 Length:131302 Length:131302 Min. :0.0000
Class :character Class :character Class :character Class :character 1st Qu.:0.0000
Mode :character Mode :character Mode :character Mode :character Median :0.0000
Mean :0.1393
3rd Qu.:0.0000
Max. :1.0000
CountryOfBirthCode Citizenship EmploymentStatus Industry
Min. : 57.00 Length:131302 Length:131302 Length:131302
1st Qu.: 57.00 Class :character Class :character Class :character
Median : 57.00 Mode :character Mode :character Mode :character
Mean : 82.68
3rd Qu.: 57.00
Max. :555.00
Often when evaluating a new dataset, we try to identify if there is a pattern in the missing values in the dataset. We will try to determine if there is a pattern in the missing values of the Married variable. The function
is.na(CPS$Married) returns a vector of TRUE/FALSE values for whether the Married variable is missing. We can see the breakdown of whether Married is missing based on the reported value of the Region variable with the function
table(CPS$Region, is.na(CPS$Married))Which is the most accurate:
table(CPSData$Region, is.na(CPSData$Married))
FALSE TRUE
Midwest 24609 6075
Northeast 21432 4507
South 33535 7967
West 26388 6789table(CPSData$Sex, is.na(CPSData$Married))
FALSE TRUE
Female 55264 12217
Male 50700 13121As mentioned in the variable descriptions, MetroAreaCode is missing if an interviewee does not live in a metropolitan area. Using the same technique as in the previous question, answer the following questions about people who live in non-metropolitan areas.
How many states had all interviewees living in a non-metropolitan area (aka they have a missing MetroAreaCode value)? For this question, treat the District of Columbia as a state (even though it is not technically a state).
table(CPSData$State, is.na(CPSData$MetroAreaCode))
FALSE TRUE
Alabama 1020 356
Alaska 0 1590
Arizona 1327 201
Arkansas 724 697
California 11333 237
Colorado 2545 380
Connecticut 2593 243
Delaware 1696 518
District of Columbia 1791 0
Florida 4947 202
Georgia 2250 557
Hawaii 1576 523
Idaho 761 757
Illinois 3473 439
Indiana 1420 584
Iowa 1297 1231
Kansas 1234 701
Kentucky 908 933
Louisiana 1216 234
Maine 909 1354
Maryland 2978 222
Massachusetts 1858 129
Michigan 2517 546
Minnesota 2150 989
Mississippi 376 854
Missouri 1440 705
Montana 199 1015
Nebraska 816 1133
Nevada 1609 247
New Hampshire 1148 1514
New Jersey 2567 0
New Mexico 832 270
New York 5144 451
North Carolina 1642 977
North Dakota 432 1213
Ohio 2754 924
Oklahoma 1024 499
Oregon 1519 424
Pennsylvania 3245 685
Rhode Island 2209 0
South Carolina 1139 519
South Dakota 595 1405
Tennessee 1149 635
Texas 6060 1017
Utah 1455 387
Vermont 657 1233
Virginia 2367 586
Washington 1937 429
West Virginia 344 1065
Wisconsin 1882 804
Wyoming 0 1624How many states had all interviewees living in a metropolitan area? Again, treat the District of Columbia as a state.
table(CPSData$State, is.na(CPSData$MetroAreaCode))
FALSE TRUE
Alabama 1020 356
Alaska 0 1590
Arizona 1327 201
Arkansas 724 697
California 11333 237
Colorado 2545 380
Connecticut 2593 243
Delaware 1696 518
District of Columbia 1791 0
Florida 4947 202
Georgia 2250 557
Hawaii 1576 523
Idaho 761 757
Illinois 3473 439
Indiana 1420 584
Iowa 1297 1231
Kansas 1234 701
Kentucky 908 933
Louisiana 1216 234
Maine 909 1354
Maryland 2978 222
Massachusetts 1858 129
Michigan 2517 546
Minnesota 2150 989
Mississippi 376 854
Missouri 1440 705
Montana 199 1015
Nebraska 816 1133
Nevada 1609 247
New Hampshire 1148 1514
New Jersey 2567 0
New Mexico 832 270
New York 5144 451
North Carolina 1642 977
North Dakota 432 1213
Ohio 2754 924
Oklahoma 1024 499
Oregon 1519 424
Pennsylvania 3245 685
Rhode Island 2209 0
South Carolina 1139 519
South Dakota 595 1405
Tennessee 1149 635
Texas 6060 1017
Utah 1455 387
Vermont 657 1233
Virginia 2367 586
Washington 1937 429
West Virginia 344 1065
Wisconsin 1882 804
Wyoming 0 1624Which region of the United States has the largest proportion of interviewees living in a non-metropolitan area?
table(CPSData$Region, is.na(CPSData$MetroAreaCode))
FALSE TRUE
Midwest 20010 10674
Northeast 20330 5609
South 31631 9871
West 25093 8084While we were able to use the table() command to compute the proportion of interviewees from each region not living in a metropolitan area, it was somewhat tedious (it involved manually computing the proportion for each region) and isn’t something you would want to do if there were a larger number of options. It turns out there is a less tedious way to compute the proportion of values that are TRUE. The mean() function, which takes the average of the values passed to it, will treat TRUE as 1 and FALSE as 0, meaning it returns the proportion of values that are true. For instance, mean(c(TRUE, FALSE, TRUE, TRUE)) returns 0.75. Knowing this, use tapply() with the mean function to answer the following questions:
Which state has a proportion of interviewees living in a non-metropolitan area closest to 30%?
sort(tapply(is.na(CPSData$MetroAreaCode), CPSData$State, mean))District of Columbia New Jersey Rhode Island California
0.00000000 0.00000000 0.00000000 0.02048401
Florida Massachusetts Maryland New York
0.03923092 0.06492199 0.06937500 0.08060769
Connecticut Illinois Colorado Arizona
0.08568406 0.11221881 0.12991453 0.13154450
Nevada Texas Louisiana Pennsylvania
0.13308190 0.14370496 0.16137931 0.17430025
Michigan Washington Georgia Virginia
0.17825661 0.18131868 0.19843249 0.19844226
Utah Oregon Delaware New Mexico
0.21009772 0.21821925 0.23396567 0.24500907
Hawaii Ohio Alabama Indiana
0.24916627 0.25122349 0.25872093 0.29141717
Wisconsin South Carolina Minnesota Oklahoma
0.29932986 0.31302774 0.31506849 0.32764281
Missouri Tennessee Kansas North Carolina
0.32867133 0.35594170 0.36227390 0.37304315
Iowa Arkansas Idaho Kentucky
0.48694620 0.49049965 0.49868248 0.50678979
New Hampshire Nebraska Maine Vermont
0.56874530 0.58132376 0.59832081 0.65238095
Mississippi South Dakota North Dakota West Virginia
0.69430894 0.70250000 0.73738602 0.75585522
Montana Alaska Wyoming
0.83607908 1.00000000 1.00000000 Which state has the largest proportion of non-metropolitan interviewees, ignoring states where all interviewees were non-metropolitan?
Codes like MetroAreaCode and CountryOfBirthCode are a compact way to encode factor variables with text as their possible values, and they are therefore quite common in survey datasets. In fact, all but one of the variables in this dataset were actually stored by a numeric code in the original CPS datafile.
When analyzing a variable stored by a numeric code, we will often want to convert it into the values the codes represent. To do this, we will use a dictionary, which maps the the code to the actual value of the variable. We have provided dictionaries MetroAreaCodes.csv and CountryCodes.csv, which respectively map MetroAreaCode and CountryOfBirthCode into their true values. Read these two dictionaries into data frames MetroAreaMap and CountryMap.
How many observations (codes for metropolitan areas) are there in MetroAreaMap?
How many observations (codes for countries) are there in CountryMap?
To merge in the metropolitan areas, we want to connect the field MetroAreaCode from the CPS data frame with the field Code in MetroAreaMap. The following command merges the two data frames on these columns, overwriting the CPS data frame with the result:
CPS = merge(CPS, MetroAreaMap, by.x="MetroAreaCode", by.y="Code", all.x=TRUE)The first two arguments determine the data frames to be merged (they are called “x” and “y”, respectively, in the subsequent parameters to the merge function). by.x=“MetroAreaCode” means we’re matching on the MetroAreaCode variable from the “x” data frame (CPS), while by.y=“Code” means we’re matching on the Code variable from the “y” data frame (MetroAreaMap). Finally, all.x=TRUE means we want to keep all rows from the “x” data frame (CPS), even if some of the rows’ MetroAreaCode doesn’t match any codes in MetroAreaMap (for those familiar with database terminology, this parameter makes the operation a left outer join instead of an inner join).
Review the new version of the CPS data frame with the summary() and str() functions. What is the name of the variable that was added to the data frame by the merge() operation?
How many interviewees have a missing value for the new metropolitan area variable? Note that all of these interviewees would have been removed from the merged data frame if we did not include the all.x=TRUE parameter.
Which of the following metropolitan areas has the largest number of interviewees?
Which metropolitan area has the highest proportion of interviewees of Hispanic ethnicity? Hint: Use tapply() with mean, as in the previous subproblem. Calling sort() on the output of tapply() could also be helpful here.
Remembering that CPS$Race == “Asian” returns a TRUE/FALSE vector of whether an interviewee is Asian, determine the number of metropolitan areas in the United States from which at least 20% of interviewees are Asian.
Normally, we would look at the sorted proportion of interviewees from each metropolitan area who have not received a high school diploma with the command:
sort(tapply(CPS$Education == "No high school diploma", CPS$MetroArea, mean))However, none of the interviewees aged 14 and younger have an education value reported, so the mean value is reported as NA for each metropolitan area. To get mean (and related functions, like sum) to ignore missing values, you can pass the parameter na.rm=TRUE. Passing na.rm=TRUE to the tapply function, determine which metropolitan area has the smallest proportion of interviewees who have received no high school diploma.
Just as we did with the metropolitan area information, merge in the country of birth information from the CountryMap data frame, replacing the CPS data frame with the result. If you accidentally overwrite CPS with the wrong values, remember that you can restore it by re-loading the data frame from CPSData.csv and then merging in the metropolitan area information using the command provided in the previous subproblem.
What is the name of the variable added to the CPS data frame by this merge operation?
How many interviewees have a missing value for the new country of birth variable?
Among all interviewees born outside of North America, which country was the most common place of birth?
What proportion of the interviewees from the “New York-Northern New Jersey-Long Island, NY-NJ-PA” metropolitan area have a country of birth that is not the United States? For this computation, don’t include people from this metropolitan area who have a missing country of birth.
Which metropolitan area has the largest number (note – not proportion) of interviewees with a country of birth in India? Hint – remember to include na.rm=TRUE if you are using tapply() to answer this question.
In Brazil?
In Somalia?