Grouping and Chaining with dplyr

---

In the last lesson, you learned about the five main data manipulation ‘verbs’ in dplyr: select(), filter(), arrange(), mutate(), and summarize(). The last of these, summarize(), is most powerful when applied to grouped data.

The main idea behind grouping data is that you want to break up your dataset into groups of rows based on the values of one or more variables. The group_by() function is reponsible for doing this.

We’ll continue where we left off with RStudio’s CRAN download log from July 8, 2014, which contains information on roughly 225,000 R package downloads (http://cran-logs.rstudio.com/).

As with the last lesson, the dplyr package was automatically installed (if necessary) and loaded at the beginning of this lesson. Normally, this is something you would have to do on your own. Just to build the habit, type library(dplyr) now to load the package again.

library(dplyr)

I’ve made the dataset available to you in a data frame called mydf. Put it in a ‘data frame tbl’ using the tbl_df() function and store the result in a object called cran. If you’re not sure what I’m talking about, you should start with the previous lesson. Otherwise, practice makes perfect!

cran <- tbl_df(mydf)

To avoid confusion and keep things running smoothly, let’s remove the original dataframe from your workspace with rm(“mydf”).

rm("mydf")

Print cran to the console.

cran

Our first goal is to group the data by package name. Bring up the help file for group_by().

?group_by

Group cran by the package variable and store the result in a new object called by_package.

by_package <- group_by(cran, package)

Let’s take a look at by_package. Print it to the console.

by_package

At the top of the output above, you’ll see ‘Groups: package’, which tells us that this tbl has been grouped by the package variable. Everything else looks the same, but now any operation we apply to the grouped data will take place on a per package basis.

Recall that when we applied mean(size) to the original tbl_df via summarize(), it returned a single number – the mean of all values in the size column. We may care about what that number is, but wouldn’t it be so much more interesting to look at the mean download size for each unique package?

That’s exactly what you’ll get if you use summarize() to apply mean(size) to the grouped data in by_package. Give it a shot.

summarize(by_package, mean(size))

Instead of returning a single value, summarize() now returns the mean size for EACH package in our dataset.

Let’s take it a step further. I just opened an R script for you that contains a partially constructed call to summarize(). Follow the instructions in the script comments.

When you are ready to move on, save the script and type submit(), or type reset() to reset the script to its original state.

Script: summarize1.R

Compute four values, in the following order, from the grouped data:

1. count = n()
2. unique = n_distinct(ip_id)
3. countries = n_distinct(country)
4. avg_bytes = mean(size)

A few thing to be careful of:

1. Separate arguments by commas
2. Make sure you have a closing parenthesis
3. Check your spelling!
4. Store the result in pack_sum (for ‘package summary’)

You should also take a look at ?n and ?n_distinct, so that you really understand what is going on.

pack_sum <- summarize(by_package,
                      count = n(),
                      unique = n_distinct(ip_id),
                      countries = n_distinct(country),
                      avg_bytes = mean(size))

submit()

Print the resulting tbl, pack_sum, to the console to examine its contents.

pack_sum

• The ‘count’ column, created with n(), contains the total number of rows (i.e. downloads) for each package.
• The ‘unique’ column, created with n_distinct(ip_id), gives the total number of unique downloads for each package, as measured by the number of distinct ip_id’s.
• The ‘countries’ column, created with n_distinct(country), provides the number of countries in which each package was downloaded.
• The ‘avg_bytes’ column, created with mean(size), contains the mean download size (in bytes) for each package.

It’s important that you understand how each column of pack_sum was created and what it means. Now that we’ve summarized the data by individual packages, let’s play around with it some more to see what we can learn.

Naturally, we’d like to know which packages were most popular on the day these data were collected (July 8, 2014). Let’s start by isolating the top 1% of packages, based on the total number of downloads as measured by the ‘count’ column.

We need to know the value of ‘count’ that splits the data into the top 1% and bottom 99% of packages based on total downloads. In statistics, this is called the 0.99, or 99%, sample quantile. Use quantile(pack_sum$count, probs = 0.99) to determine this number.

quantile(pack_sum$count, probs = 0.99)

   99% 
679.56 

Now we can isolate only those packages which had more than 679 total downloads. Use filter() to select all rows from pack_sum for which ‘count’ is strictly greater (>) than 679. Store the result in a new object called top_counts.

top_counts <- filter(pack_sum, count > 679)

Let’s take a look at top_counts. Print it to the console.

top_counts

There are only 61 packages in our top 1%, so we’d like to see all of them. Since dplyr only shows us the first 10 rows, we can use the View() function to see more.

View all 61 rows with View(top_counts). Note that the ‘V’ in View() is capitalized.

View(top_counts)

arrange() the rows of top_counts based on the ‘count’ column and assign the result to a new object called top_counts_sorted. We want the packages with the highest number of downloads at the top, which means we want ‘count’ to be in descending order. If you need help, check out ?arrange and/or ?desc.

top_counts_sorted <- arrange(top_counts, desc(count))

Now use View() again to see all 61 rows of top_counts_sorted.

View(top_counts_sorted)

If we use total number of downloads as our metric for popularity, then the above output shows us the most popular packages downloaded from the RStudio CRAN mirror on July 8, 2014. Not surprisingly, ggplot2 leads the pack with 4602 downloads, followed by Rcpp, plyr, rJava, ….

…And if you keep on going, you’ll see swirl at number 43, with 820 total downloads. Sweet!

Perhaps we’re more interested in the number of unique downloads on this particular day. In other words, if a package is downloaded ten times in one day from the same computer, we may wish to count that as only one download. That’s what the ‘unique’ column will tell us.

Like we did with ‘count’, let’s find the 0.99, or 99%, quantile for the ‘unique’ variable with quantile(pack_sum$unique, probs = 0.99).

quantile(pack_sum$unique, probs = 0.99)

99% 
465 

Apply filter() to pack_sum to select all rows corresponding to values of ‘unique’ that are strictly greater than 465. Assign the result to a object called top_unique.

top_unique <- filter(pack_sum, unique > 465)

Let’s View() our top contenders!

View(top_unique)

Now arrange() top_unique by the ‘unique’ column, in descending order, to see which packages were downloaded from the greatest number of unique IP addresses. Assign the result to top_unique_sorted.

top_unique_sorted <- arrange(top_unique, desc(unique))

View() the sorted data.

View(top_unique_sorted)

Now Rcpp is in the lead, followed by stringr, digest, plyr, and ggplot2. swirl moved up a few spaces to number 40, with 698 unique downloads. Nice!

Our final metric of popularity is the number of distinct countries from which each package was downloaded. We’ll approach this one a little differently to introduce you to a method called ‘chaining’ (or ‘piping’).

Chaining allows you to string together multiple function calls in a way that is compact and readable, while still accomplishing the desired result. To make it more concrete, let’s compute our last popularity metric from scratch, starting with our original data.

I’ve opened up a script that contains code similar to what you’ve seen so far. Don’t change anything. Just study it for a minute, make sure you understand everything that’s there, then submit() when you are ready to move on.

Script: summarize2.R

Don’t change any of the code below. Just type submit() when you think you understand it.

We’ve already done this part, but we’re repeating it here for clarity.

by_package <- group_by(cran, package)
pack_sum <- summarize(by_package,
                      count = n(),
                      unique = n_distinct(ip_id),
                      countries = n_distinct(country),
                      avg_bytes = mean(size))

Here’s the new bit, but using the same approach we’ve been using this whole time.

top_countries <- filter(pack_sum, countries > 60)
result1 <- arrange(top_countries, desc(countries), avg_bytes)
# Print the results to the console.
print(result1)

submit()

It’s worth noting that we sorted primarily by country, but used avg_bytes (in ascending order) as a tie breaker. This means that if two packages were downloaded from the same number of countries, the package with a smaller average download size received a higher ranking.

We’d like to accomplish the same result as the last script, but avoid saving our intermediate results. This requires embedding function calls within one another.

That’s exactly what we’ve done in this script. The result is equivalent, but the code is much less readable and some of the arguments are far away from the function to which they belong. Again, just try to understand what is going on here, then submit() when you are ready to see a better solution.

Script: summarize3.R

Don’t change any of the code below. Just type submit() when you think you understand it. If you find it confusing, you’re absolutely right!

result2 <-
  arrange(
    filter(
      summarize(
        group_by(cran,
                 package
        ),
        count = n(),
        unique = n_distinct(ip_id),
        countries = n_distinct(country),
        avg_bytes = mean(size)
      ),
      countries > 60
    ),
    desc(countries),
    avg_bytes
  )
print(result2)

submit()

In this script, we’ve used a special chaining operator, %>%, which was originally introduced in the magrittr R package and has now become a key component of dplyr. You can pull up the related documentation with ?chain. The benefit of %>% is that it allows us to chain the function calls in a linear fashion. The code to the right of %>% operates on the result from the code to the left of %>%.

Once again, just try to understand the code, then type submit() to continue.

Script: summarize4.R

Read the code below, but don’t change anything. As you read it, you can pronounce the %>% operator as the word ‘then’.

Type submit() when you think you understand everything here.

result3 <-
  cran %>%
  group_by(package) %>%
  summarize(count = n(),
            unique = n_distinct(ip_id),
            countries = n_distinct(country),
            avg_bytes = mean(size)
  ) %>%
  filter(countries > 60) %>%
  arrange(desc(countries), avg_bytes)
# Print result to console
print(result3)

submit()

So, the results of the last three scripts are all identical. But, the third script provides a convenient and concise alternative to the more traditional method that we’ve taken previously, which involves saving results as we go along.

Once again, let’s View() the full data, which has been stored in result3.

View(result3)

It looks like Rcpp is on top with downloads from 84 different countries, followed by digest, stringr, plyr, and ggplot2. swirl jumped up the rankings again, this time to 27th.

To help drive the point home, let’s work through a few more examples of chaining.

Let’s build a chain of dplyr commands one step at a time, starting with the script I just opened for you.

Script: chain1.R

select() the following columns from cran. Keep in mind that when you’re using the chaining operator, you don’t need to specify the name of the data tbl in your call to select().

1. ip_id
2. country
3. package
4. size

The call to print() at the end of the chain is optional, but necessary if you want your results printed to the console. Note that since there are no additional arguments to print(), you can leave off the parentheses after the function name. This is a convenient feature of the %>% operator.

cran %>%
  select(ip_id, country, package, size) %>%
    print

submit()

Let’s add to the chain.

Script: chain2.R

Use mutate() to add a column called size_mb that contains the size of each download in megabytes (i.e. size / 2^20).

If you want your results printed to the console, add print to the end of your chain.

cran %>%
  select(ip_id, country, package, size) %>%
  mutate(size_mb = size / 2^20) %>%
  print

submit()

A little bit more now.

Script: chain3.R

Use filter() to select all rows for which size_mb is less than or equal to (<=) 0.5.

If you want your results printed to the console, add print to the end of your chain.

cran %>%
  select(ip_id, country, package, size) %>%
  mutate(size_mb = size / 2^20) %>%
  filter(size_mb <= 0.5) %>%
  print

submit()

And finish it off.

Script: chain4.R

arrange() the result by size_mb, in descending order.

If you want your results printed to the console, add print to the end of your chain.

cran %>%
  select(ip_id, country, package, size) %>%
  mutate(size_mb = size / 2^20) %>%
  filter(size_mb <= 0.5) %>%
  arrange(desc(size_mb)) %>%
  print

submit()

In this lesson, you learned about grouping and chaining using dplyr. You combined some of the things you learned in the previous lesson with these more advanced ideas to produce concise, readable, and highly effective code. Welcome to the wonderful world of dplyr!

---

END

---