Advanced Data Manipulation

Introduction to dplyr

When working with data you must:

• Figure out what you want to do.

• Precisely describe what you want in the form of a computer program.

• Execute the code.

The dplyr package makes each of these steps as fast and easy as possible by:

• Elucidating the most common data manipulation operations, so that your options are helpfully constrained when thinking about how to tackle a problem.

• Providing simple functions that correspond to the most common data manipulation verbs, so that you can easily translate your thoughts into code.

• Using efficient data storage backends, so that you spend as little time waiting for the computer as possible.

To explore the basic data manipulation verbs of dplyr, we'll start with the built in nycflights13 data frame.

All 336,776 flights that departed from New York City in 2013.

The data comes from the US Bureau of Transporation Statistics, and is documented in ?nycflights13

library(nycflights13)
dim(flights)

[1] 336776     16

head(flights)

dplyr can work with data frames as is, but if you're dealing with large data, it's worthwhile to convert them to a tbl_df

This is a wrapper around a data frame that won't accidentally print a lot of data to the screen.

library(dplyr)
flights <- tbl_df(flights)

dplyr aims to provide a function for each basic verb of data manipulating:

• filter() (and slice())
• arrange()
• select() (and rename())
• distinct()
• mutate() (and transmute())
• summarise()
• sample_n() and sample_frac()
  

If you’ve used plyr before, many of these will be familar.

filter() allows you to select a subset of the rows of a data frame.

The first argument is the name of the data frame.

The second and subsequent are filtering expressions evaluated in the context of that data frame.

For example, we can select all flights on January 1st with:

filter(flights, month == 1, day == 1)

Source: local data frame [842 x 16]

   year month day dep_time dep_delay arr_time arr_delay carrier tailnum
1  2013     1   1      517         2      830        11      UA  N14228
2  2013     1   1      533         4      850        20      UA  N24211
3  2013     1   1      542         2      923        33      AA  N619AA
4  2013     1   1      544        -1     1004       -18      B6  N804JB
5  2013     1   1      554        -6      812       -25      DL  N668DN
6  2013     1   1      554        -4      740        12      UA  N39463
7  2013     1   1      555        -5      913        19      B6  N516JB
8  2013     1   1      557        -3      709       -14      EV  N829AS
9  2013     1   1      557        -3      838        -8      B6  N593JB
10 2013     1   1      558        -2      753         8      AA  N3ALAA
..  ...   ... ...      ...       ...      ...       ...     ...     ...
Variables not shown: flight (int), origin (chr), dest (chr), air_time
  (dbl), distance (dbl), hour (dbl), minute (dbl)

filter(flights, month == 1, day == 1)

What would the equivalent be in base R?

filter(flights, month == 1, day == 1)

What would the equivalent be in base R?

flights[flights$month == 1 & flights$day == 1, ]

slice(flights, 1:10)

Source: local data frame [10 x 16]

   year month day dep_time dep_delay arr_time arr_delay carrier tailnum
1  2013     1   1      517         2      830        11      UA  N14228
2  2013     1   1      533         4      850        20      UA  N24211
3  2013     1   1      542         2      923        33      AA  N619AA
4  2013     1   1      544        -1     1004       -18      B6  N804JB
5  2013     1   1      554        -6      812       -25      DL  N668DN
6  2013     1   1      554        -4      740        12      UA  N39463
7  2013     1   1      555        -5      913        19      B6  N516JB
8  2013     1   1      557        -3      709       -14      EV  N829AS
9  2013     1   1      557        -3      838        -8      B6  N593JB
10 2013     1   1      558        -2      753         8      AA  N3ALAA
Variables not shown: flight (int), origin (chr), dest (chr), air_time
  (dbl), distance (dbl), hour (dbl), minute (dbl)

slice(flights, 1:10)

What would the equivalent be in base R?

slice(flights, 1:10)

What would the equivalent be in base R?

flights[1:10,]

1. Select all of the cases in iris where Species is “virginica” & Petal.Width is >2.
   

Use both a dplyr and a base R solution.

arrange() works like filter() except that instead of selecting rows, it reorders them.

It takes a data frame and a set of column names (or more complicated expressions) to order by.

If you provide more than one column name, each additional column will be used to break ties in the values of preceding columns

arrange(flights, year, month, day)

Use desc() to order a column in descending order:

arrange(flights, desc(arr_delay))

If you wanted to do that with base R you would have typed:

flights[order(flights$year, flights$month, flights$day), ]

flights[order(desc(flights$arr_delay)), ]

Often you work with large datasets with many columns where only a few are actually of interest to you.

select() allows you to rapidly zoom in on a useful subset using operations that usually only work on numeric variable positions

# Select columns by name
select(flights, year, month, day)

# Select all columns between year and day (inclusive)
select(flights, year:day)

# Select all columns except those from year to day (inclusive)
select(flights, -(year:day))

There are a number of helper functions you can use within select()

• starts_with()
• ends_with()
• matches()
• contains()
  

These let you quickly match larger blocks of variable that meet some criterion. See ?select for more details.

rename(flights, tail_num = tailnum)

A common use of select() is to find out which values a set of variables takes.

This is particularly useful in conjunction with the distinct() verb which only returns the unique values in a table.

distinct(select(flights, tailnum))

distinct(select(flights, origin, dest))

This is very similar to base::unique() but should be much faster.

As well as selecting from the set of existing columns, it’s often useful to add new columns that are functions of existing columns.

This is the job of mutate()

mutate(flights,
  gain = arr_delay - dep_delay,
  speed = distance / air_time * 60)

You can even refer to columns that you just created.

mutate(flights,
  gain = arr_delay - dep_delay,
  gain_per_hour = gain / (air_time / 60)
)

The last verb is summarise(), which collapses a data frame to a single row. It's not very useful yet:

summarise(flights,
  delay = mean(dep_delay, na.rm = TRUE))

Source: local data frame [1 x 1]

     delay
1 12.63907

You can use either sample_n() to sample a fixed number or sample_frac() to sample a fixed fraction.

sample_n(flights, 10)

sample_frac(flights, 0.01)

You may have noticed that all of these functions are very similar:

• The first argument is a data frame.
• The subsequent arguments describe what to do with it, and you can refer to columns in the data frame directly without using $.
• The result is a new data frame
  

Together these properties make it easy to chain together multiple simple steps to achieve a complex result.

In dplyr, you use the group_by() function to describe how to break a dataset down into groups of rows.

You can then use the resulting object in exactly the same functions as above.

They’ll automatically work “by group” when the input is a grouped.

The verbs are affected by grouping as follows:

• grouped select() retains grouping variables
• grouped arrange() orders first by grouping variables
• sample_n() and sample_frac() samples rows from each group
• slice() extracts rows from each group.
• summarise() is easy to understand and very useful, and is described in more detail below.

You use summarise() with aggregate functions, which take a vector of values, and return a single number.

There are many useful functions in base R like:

• min() and max()
• mean() and median()
• sum()
• sd()
• IQR()

dplyr provides a handful of other aggregate functions:

• n(): # of observations in the current group
• n_distinct(x): # of unique values in x
• first(x): like x[1]
• last(x): like x[length(x)]
• nth(x, n): like x[n]

Use group_by() and summarise() to find the number of planes and the number of flights that go to each possible destination:

destinations <- group_by(flights, dest)

summarise(destinations,
  planes = n_distinct(tailnum),
  flights = n()
)

1. Use summarise() and group_by to find the mean Sepal.Length of each species in iris

Performing many operations, step-by-step does not produce elegant code

a1 <- group_by(flights, year, month, day) 

a2 <- select(a1, arr_delay, dep_delay) 

a3 <- summarise(a2, 
  arr = mean(arr_delay, na.rm = TRUE),
  dep = mean(dep_delay, na.rm = TRUE)) 

a4 <- filter(a3, arr > 30 | dep > 30) 

Or if you don’t want to save the intermediate results, you need to wrap the function calls inside each other:

filter(
  summarise(
    select(
      group_by(flights, year, month, day),
      arr_delay, dep_delay
    ),
    arr = mean(arr_delay, na.rm = TRUE),
    dep = mean(dep_delay, na.rm = TRUE)
  ),
  arr > 30 | dep > 30
)

To get around this problem, dplyr provides the %>% operator.

x %>% f(y) turns into f(x, y)

You can rewrite multiple operations so you can read from left-to-right, top-to-bottom.

flights %>%
  group_by(year, month, day) %>%
  select(arr_delay, dep_delay) %>%
  summarise(
    arr = mean(arr_delay, na.rm = TRUE),
    dep = mean(dep_delay, na.rm = TRUE)
  ) %>%
  filter(arr > 30 | dep > 30)

1. Use chaining to find the mean and standard deviation of Petal.Width of each species in iris

Often our data look like this:

data <- data.frame(species = c(rep(1, 1000), rep(2, 1000)),
                   treatment = rep(c("Exp","Cont"),1000),
                   weight = rnorm(2000, 10, 1),
                   length = rnorm(2000, 50, 5))

But instead of:

colnames(data)

[1] "species"   "treatment" "weight"    "length"   

we want species, treatment, variable, value

This manipulation is called melting and can be done with the package reshape2

library(reshape2)
melted <- melt(data, 
               id.vars=c("species", "treatment"))

Chaining dplyr commands:

melted %>% 
  group_by(species,treatment) %>%
    summarise(mean=mean(value), sd=sd(value))

Source: local data frame [4 x 4]
Groups: species

  species treatment     mean       sd
1       1      Cont 30.05909 20.31349
2       1       Exp 30.00171 20.24931
3       2      Cont 30.16614 20.42312
4       2       Exp 30.01305 20.38472