Data manipulation with tidyverse (examples from r4ds)

library(tidyverse)
library(nycflights13)

For the examples we use a set of all flights from NYC in 2013. The dataset consists of several tables:

• Airlines: Names of the airlines
• Airports: Information about the various airports
• Flighs : Information about the flights such as origin, destination, delays etc.
• Planes : Plane information, make, manufacture year etc.
• Weather : Weather informaton for the origin airports

Select,Filter,Sort & Mutate

Filter rows with filter()

Find all flights that departed 22 december 2013 from JFK.

filter(nycflights13::flights,month == 12 & day == 22 & origin == "JFK")

Select rows with select()

Select specific rows

select(nycflights13::flights,dest,origin)

All rows between two variables

select(nycflights13::flights,year:day)

There are a number of helper functions you can use within select():

• starts_with(“abc”): matches names that begin with “abc”.
• ends_with(“xyz”): matches names that end with “xyz”.
• contains(“ijk”): matches names that contain “ijk”.
• matches(“(.)\”): selects variables that match a regular expression. This one matches any variables that contain repeated characters. You’ll learn more about regular expressions in strings.
• num_range(“x”, 1:3) matches x1, x2 and x3.

Add new variables using mutate(), using the pipe operator %>% to pass on data

flights_subset <-  select(nycflights13::flights,dest,tailnum,contains("delay"),distance,air_time) %>%
      mutate(
           air_speed=distance/air_time*60, #Mph
           cumsum_delay=cumsum(arr_delay),
           lagged_delay2=lag(x = cumsum_delay,2)) 
flights_subset

Summarizing data with summarise()

summarise(flights_subset,mean_speed=mean(air_speed,na.rm=T))

Combine with group_by for more usefulness.

Here we get the top10 fastest tailnums pr. destination.

group_by(flights_subset,dest,tailnum) %>% 
    summarise(mean_speed_dest = mean(air_speed,na.rm=T),n=n()) %>%  #Calculate speed for pr. tailnum pr dest
          arrange(dest,desc(mean_speed_dest)) %>%                   #Sort by destination and tailnum
            mutate(rown=row_number()) %>%                           #Get the (intra group) row number (rank)
              filter(rown <= 10) %>%                                #Keep only the 10 fastest tailnums pr dest
                arrange(dest,desc(mean_speed_dest))                 #Arrange by destination and fastest tailnums

Another example: For each destination, compute the total minutes of delay. For each, flight, compute the proportion of the total delay for its destination.

group_by(flights_subset,dest) %>%
  mutate(dest_delay=sum(arr_delay,na.rm=T),prop_delay=arr_delay/dest_delay) %>%
    arrange(dest,desc(prop_delay)) %>%
      select(dest,tailnum,arr_delay,dest_delay,prop_delay)

Tidy data with spread, gather, separate and unite

Gathering

We can use the gather function to gather data where the columns contain values of a variable rather than variables:

table4a

Here the two columns 1999 and 2000 are not variables but values of a variable indicating time of observation. We can use gather to put these variables in to a column called year instead:

table4a %>% 
  gather(`1999`,`2000`, key="Year",value="Cases") %>% 
    arrange(country) 

Gather makes long data wider and shorter.

Spreading

Spread is used when an observation is located over several rows:

table2

Here each observation (a country in a year) is spread across two rows and indicated by the type variable. We can use spread to fix this:

table2 %>% spread(key=type,value=count)

Separate

We use separate to divide variables that are put in to one column when they belong in several columns:

table3

We use separate() to put the components of the rate into two new columns: Cases & Population

table3 %>% separate(rate,sep="/",into = c("Cases","Population"))