Homework # 5

library(tidyverse)
library(openintro)
library(tibble)
library(readr)
library(Lahman)
library(babynames)
library(nasaweather)
library(fueleconomy)
library(nycflights13)

10.5 Practice Problem 2

This exercise highlights key differences between base R data.frame behavior and tibbles when extracting columns.

1. df$x

In a base R data.frame, the $ operator allows partial matching. Even though there is no column named x, R matches x to xyz and returns: 1 “a” In contrast, a tibble does not allow partial matching. Attempting to access a non-existent column like tbl$x would not match xyz, making the behavior stricter and safer.

2. df[, "xyz"]

In a base R data.frame, selecting a single column with [, ] returns a vector: 1 “a”

In a tibble, this same operation returns a one-column tibble, not a vector. Tibbles preserve structure rather than simplifying output.

3. df[, c("abc", "xyz")]

When selecting multiple columns, a base R data.frame returns a data.frame, and a tibble returns a tibble. This behavior is consistent across both structures.

The default behavior of R data.frame can lead to confusion and bugs:

• Partial matching (e.g., df$x returning xyz) can produce unexpected results.
• Selecting one column returns a vector, but selecting multiple returns a data frame.
• The inconsistency in output types makes code harder to debug and predict.

Tibbles improve usability by enforcing stricter rules and returning consistent data types, which makes analysis workflows more reliable.

df <- data.frame(abc = 1, xyz = "a")

df$x

## [1] "a"

df[, "xyz"]

## [1] "a"

df[, c("abc", "xyz")]

##   abc xyz
## 1   1   a

11.2.2 Question 1

To read a file where fields are separated by the pipe character (|), I would use the read_delim() function from the readr package and specify the delimiter using delim = “|”. This function is designed for reading files with non-standard separators and provides flexibility compared to functions like read_csv().

data <- read_delim("file.txt", delim = "|")

11.3.5 Question 7

The correct format strings are:

• d1: “%B %d, %Y”
• d2: “%Y-%b-%d”
• d3: “%d-%b-%Y”
• d4: “%B %d (%Y)”
• d5: “%m/%d/%y”
• t1: “%H%M”
• t2: “%I:%M:%OS %p”

These strings work because: - %B = full month name (e.g., January, August) - %b = abbreviated month name (e.g., Mar, Jun) - %d = day of the month - %Y = four-digit year - %y = two-digit year - %m = two-digit month number - %H = hour in 24-hour time - %M = minutes - %OS = seconds with fractional seconds - %I = hour in 12-hour time - %p = AM/PM marker

This formatting allows R to correctly interpret each date and time value.

d1 <- "January 1, 2010"
d2 <- "2015-Mar-07"
d3 <- "06-Jun-2017"
d4 <- c("August 19 (2015)", "July 1 (2015)")
d5 <- "12/30/14" # Dec 30, 2014
t1 <- "1705"
t2 <- "11:15:10.12 PM"

12.2.1 Exercises

1

The sample tables organize variables and observations in different ways:

• table1 is a tidy dataset. Each variable (country, year, cases, population) has its own column, and each observation (a country-year combination) has its own row.

• table2 is not tidy. The cases and population values are stored in a single column called count, and another column (type) indicates whether the value represents cases or population. This means one variable is spread across rows instead of having separate columns.

• table3 is not tidy because cases and population are combined into a single column as a ratio (e.g., “745/19987071”), instead of being separated into two variables.

• table4a contains only case counts, with each year as a separate column. This spreads a variable (year) across multiple columns.

• table4b contains population data, also with each year as a separate column. Like table4a, it is not tidy because a variable is stored in the column names.

2

Table2

table2_rates <- table2 %>%
  pivot_wider(names_from = type, values_from = count) %>%
  mutate(rate = (cases / population) * 10000)

table2_rates

## # A tibble: 6 × 5
##   country      year  cases population  rate
##   <chr>       <dbl>  <dbl>      <dbl> <dbl>
## 1 Afghanistan  1999    745   19987071 0.373
## 2 Afghanistan  2000   2666   20595360 1.29 
## 3 Brazil       1999  37737  172006362 2.19 
## 4 Brazil       2000  80488  174504898 4.61 
## 5 China        1999 212258 1272915272 1.67 
## 6 China        2000 213766 1280428583 1.67

To compute the rate for table2, I first reshaped the data using pivot_wider() so that cases and population became separate columns. Then I divided cases by population and multiplied by 10,000 to create a new variable called rate.

Table4a and Table4b

table4a_long <- table4a %>%
  pivot_longer(cols = c(`1999`, `2000`),
               names_to = "year",
               values_to = "cases")

table4b_long <- table4b %>%
  pivot_longer(cols = c(`1999`, `2000`),
               names_to = "year",
               values_to = "population")

table4_rates <- table4a_long %>%
  left_join(table4b_long, by = c("country", "year")) %>%
  mutate(rate = (cases / population) * 10000)

table4_rates

## # A tibble: 6 × 5
##   country     year   cases population  rate
##   <chr>       <chr>  <dbl>      <dbl> <dbl>
## 1 Afghanistan 1999     745   19987071 0.373
## 2 Afghanistan 2000    2666   20595360 1.29 
## 3 Brazil      1999   37737  172006362 2.19 
## 4 Brazil      2000   80488  174504898 4.61 
## 5 China       1999  212258 1272915272 1.67 
## 6 China       2000  213766 1280428583 1.67

To compute the rate for table4a and table4b, I first converted both tables from wide format to long format using pivot_longer(). This moved the year values into a single year column. I then joined the two tables by country and year so that each row contained both the number of cases and the population for the same country and year. Finally, I calculated the rate by dividing cases by population and multiplying by 10,000.

3

The easiest representation to work with is table1 because it is already tidy. Each variable has its own column, each observation has its own row, and calculations can be done directly without reshaping the data. The hardest representation to work with is table3 because cases and population are combined into a single column. Before any analysis can be done, the combined values would need to be split into separate variables. This adds extra steps and makes the data less convenient to use. table2, table4a, and table4b are somewhat harder to use than table1 because they require reshaping before analysis, but they are still easier to work with than table3.

4

To recreate the plot showing change in cases over time using table2, I first needed to reshape the data. In table2, the values for cases and population are stored together in a single column called count, while another column called type identifies whether the value is for cases or population. Because cases is not already its own variable, I had to use pivot_wider() to convert the data into a tidy format with separate cases and population columns. After that, I used ggplot() to plot the number of cases over time by country.

table2_tidy <- table2 %>%
  pivot_wider(names_from = type, values_from = count)

ggplot(table2_tidy, aes(x = year, y = cases, color = country, group = country)) +
  geom_line() +
  geom_point()

12.3.3 Question 1

pivot_longer() and pivot_wider() are not perfectly symmetrical because information can be lost or changed during the transformation process, particularly when column names are created or modified.

In the example, when pivot_wider() is applied, the values from the year column become column names (e.g., "2015" and "2016"). These column names are stored as character strings, even though the original year variable was numeric.

When the data is then transformed back using pivot_longer(), the year values are reconstructed from the column names, but they remain character type, not numeric. As a result, the dataset is not identical to the original because the variable type has changed.

Additionally, when reshaping data, there is a risk of losing information if the data structure is not uniquely identified (e.g., if there are duplicate combinations of keys). This also contributes to the lack of perfect symmetry.

stocks <- tibble(
  year   = c(2015, 2015, 2016, 2016),
  half  = c(   1,    2,     1,    2),
  return = c(1.88, 0.59, 0.92, 0.17)
)
stocks %>% 
  pivot_wider(names_from = year, values_from = return) %>% 
  pivot_longer(`2015`:`2016`, names_to = "year", values_to = "return")

## # A tibble: 4 × 3
##    half year  return
##   <dbl> <chr>  <dbl>
## 1     1 2015    1.88
## 2     1 2016    0.92
## 3     2 2015    0.59
## 4     2 2016    0.17

The names_ptypes argument in pivot_longer() is used to enforce the data type of the new column created from column names. However, it requires that the values can already be interpreted as that type. If they cannot be safely converted, it will produce an error.

In contrast, names_transform can be used to explicitly convert the column to a new type (for example, numeric) after it is created. This is often more flexible when working with column names like years that are initially stored as character values.

stocks %>%
  pivot_wider(names_from = year, values_from = return) %>%
  pivot_longer(`2015`:`2016`,
               names_to = "year",
               values_to = "return",
               names_transform = list(year = as.numeric))

12.5.1 Exercises

1

The fill argument in pivot_wider() and complete() serves a similar purpose but is used in slightly different contexts.

In pivot_wider(), the values_fill argument is used to replace missing values that arise when reshaping data from long format to wide format. If a combination of variables does not exist in the original data, pivot_wider() will create an NA value, and values_fill allows you to specify what value should be used instead (such as 0).

In contrast, complete() is used to generate missing combinations of variables in a dataset. The fill argument in complete() is then used to assign values to those newly created rows. This means that complete() actively adds missing rows to the dataset, whereas pivot_wider() only fills in missing cells created during reshaping.

In summary: - pivot_wider() uses values_fill to replace missing values in an existing structure after transformation. - complete() uses fill to assign values to newly created rows that represent missing combinations of variables.

2

The direction argument in the fill() function controls how missing values (NAs) are filled within a column.

• "down" fills missing values using the last observed non-missing value moving downward.
• "up" fills missing values using the next observed non-missing value moving upward.
• "downup" first fills downward, then upward.
• "updown" first fills upward, then downward.

This argument allows you to control the direction in which values are carried forward or backward, which is especially useful when working with grouped or sequential data.

12.6.1 Question 1

In this case study, using values_drop_na = TRUE is reasonable for simplifying the dataset and making it easier to check whether the values have been correctly transformed. By removing rows with missing values, it becomes easier to focus only on the observed data and verify accuracy.

However, whether this is appropriate depends on how missing values are represented in the dataset. In this case, there are implicit missing values, meaning that some combinations of variables are absent from the data rather than explicitly recorded as NA. When values_drop_na = TRUE is used, these implicit missing values are effectively ignored, which may lead to loss of potentially meaningful information.

An important distinction is between missing values (NA) and zeros:

• An NA represents a value that is unknown, not recorded, or not applicable. It indicates missing data.
• A value of zero represents a known quantity, meaning that the measured value truly is zero.

This distinction matters because removing NA values does not imply that the value is zero. Treating missing values as zero could lead to incorrect conclusions, especially in analyses where absence of data is not the same as absence of events.

13.2.1 Practice Problem

1

To draw the route each plane flies from its origin to its destination, I would need variables that describe both the departure and arrival locations. Specifically, I would need:

• Origin airport code (origin)
• Destination airport code (dest)
• Latitude and longitude for each airport

To obtain this information, I would need to combine:

• The flights table (for origin and destination airport codes)
• The airports table (for latitude and longitude)

I would join the flights data to the airports table twice: - Once to get the coordinates of the origin airport - Once to get the coordinates of the destination airport

This would allow me to map the flight paths between airports.

2

The relationship between weather and airports is based on the airport location. Specifically, each weather observation corresponds to a particular airport, identified by the origin code.

This is a many-to-one relationship: - Many rows in the weather table correspond to one airport in the airports table

In a relational diagram, this relationship should be shown as a link between weather.origin and airports.faa, indicating that multiple weather records are associated with each airport.

3

The weather table currently only contains information for origin (NYC) airports. If it instead contained weather data for all airports in the United States, it would define an additional relationship with the flights table.

In this case, there would be two relationships between weather and flights: - One linking weather to the origin airport - One linking weather to the destination airport

Both relationships would likely also involve time variables (such as date and hour), since weather conditions change over time. This would create a more complex relationship where each flight could be associated with weather conditions at both departure and arrival locations.

4

To represent “special” days when fewer people fly, I would create a new data frame with a variable indicating whether a day is special.

For example, the table might include: - year - month - day - special_day (e.g., TRUE/FALSE or a category such as holiday or event)

The primary key for this table would be the combination of year, month, and day, since each date should be unique.

This table would connect to the flights table through these same variables (year, month, and day). This creates a one-to-many relationship, where each day in the new table corresponds to many flights in the flights table.

This structure would allow analysis of how flight patterns differ on special days compared to regular days.

13.3.1 Question 2

The keys in each dataset are:

• Lahman::Batting: playerID, yearID, and stint
  The documentation describes playerID as the player identifier, yearID as the year, and stint as the player’s stint within a season, which together identify each batting record.

• babynames::babynames: year, sex, and name
  The dataset contains one row for each baby name by year and sex, with n and prop describing that combination, so the identifying variables are year, sex, and name.

• nasaweather::atmos: lat, long, year, and month
  The documentation says the dataset records measurements by location (lat, long) and time (year, month), so those four variables together identify each observation.

• fueleconomy::vehicles: id
  The dataset documentation explicitly states that id is the unique EPA identifier, so this is the key.

• ggplot2::diamonds: no obvious natural key
  The documentation lists 10 descriptive variables (such as carat, cut, color, clarity, price, and dimensions), but it does not include any variable described as a unique identifier. In practice, this means diamonds does not have a simple natural key, and if a unique key were needed, an artificial row ID would likely need to be added.

names(Batting)

##  [1] "playerID" "yearID"   "stint"    "teamID"   "lgID"     "G"       
##  [7] "AB"       "R"        "H"        "X2B"      "X3B"      "HR"      
## [13] "RBI"      "SB"       "CS"       "BB"       "SO"       "IBB"     
## [19] "HBP"      "SH"       "SF"       "GIDP"

13.4.6 Question 1

To compute the average delay by destination and visualize it spatially, I first summarized the average arrival delay for each destination airport. Then I joined this summary to the airports dataset to obtain latitude and longitude for mapping.

# Step 1: Compute average delay by destination
avg_delay <- flights %>%
  group_by(dest) %>%
  summarise(avg_delay = mean(arr_delay, na.rm = TRUE))

# Step 2: Join with airports data
delay_map <- airports %>%
  semi_join(flights, c("faa" = "dest")) %>%
  left_join(avg_delay, by = c("faa" = "dest"))

# Step 3: Plot the map
ggplot(delay_map, aes(x = lon, y = lat)) +
  borders("state") +
  geom_point(aes(color = avg_delay, size = avg_delay), alpha = 0.7) +
  coord_quickmap() +
  labs(
    title = "Average Arrival Delay by Destination Airport",
    x = "Longitude",
    y = "Latitude",
    color = "Avg Delay",
    size = "Avg Delay"
  )

## Warning: `borders()` was deprecated in ggplot2 4.0.0.
## ℹ Please use `annotation_borders()` instead.
## This warning is displayed once per session.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.

## Warning: Removed 1 row containing missing values or values outside the scale range
## (`geom_point()`).

13.5.1 Exercises

1

A flight has a missing tailnum when the aircraft registration number was not recorded for that flight. This means the flight cannot be matched to a specific plane in the planes table.

Tail numbers in flights that do not have a matching record in planes are likely missing from the FAA aircraft registry data used in the planes dataset. A major reason for this is that many unmatched or missing tail numbers are associated with regional carriers. In the nycflights13 data, one variable that explains most of the issue is carrier, because certain airlines account for most of the unmatched plane records.

To investigate this, I would compare missing and unmatched tail numbers across carriers.

# Flights with missing tail numbers
flights %>%
  filter(is.na(tailnum)) %>%
  count(carrier, sort = TRUE)

## # A tibble: 7 × 2
##   carrier     n
##   <chr>   <int>
## 1 9E       1044
## 2 UA        686
## 3 US        663
## 4 AA         84
## 5 WN         30
## 6 F9          3
## 7 MQ          2

# Flights with tail numbers that do not match planes
flights %>%
  filter(!is.na(tailnum)) %>%
  anti_join(planes, by = "tailnum") %>%
  count(carrier, sort = TRUE)

## # A tibble: 9 × 2
##   carrier     n
##   <chr>   <int>
## 1 MQ      25395
## 2 AA      22474
## 3 UA       1007
## 4 B6        830
## 5 FL        187
## 6 DL        110
## 7 F9         47
## 8 US         36
## 9 WN          8

2

To show only flights operated by planes that flew at least 100 times, I first counted the number of flights for each tailnum, filtered to keep only those with at least 100 flights, and then used semi_join() to retain only matching rows in the flights dataset.

planes_100 <- flights %>%
  count(tailnum) %>%
  filter(!is.na(tailnum), n >= 100)

flights_100 <- flights %>%
  semi_join(planes_100, by = "tailnum")

flights_100

## # A tibble: 228,390 × 19
##     year month   day dep_time sched_dep_time dep_delay arr_time sched_arr_time
##    <int> <int> <int>    <int>          <int>     <dbl>    <int>          <int>
##  1  2013     1     1      517            515         2      830            819
##  2  2013     1     1      533            529         4      850            830
##  3  2013     1     1      544            545        -1     1004           1022
##  4  2013     1     1      554            558        -4      740            728
##  5  2013     1     1      555            600        -5      913            854
##  6  2013     1     1      557            600        -3      709            723
##  7  2013     1     1      557            600        -3      838            846
##  8  2013     1     1      558            600        -2      849            851
##  9  2013     1     1      558            600        -2      853            856
## 10  2013     1     1      558            600        -2      923            937
## # ℹ 228,380 more rows
## # ℹ 11 more variables: arr_delay <dbl>, carrier <chr>, flight <int>,
## #   tailnum <chr>, origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>,
## #   hour <dbl>, minute <dbl>, time_hour <dttm>

``