This course introduces a tidyverse environment, which is a collection of R packages for transforming and visualizing data. This is not the only set of tools in R, but it’s a powerful and popular approach for exploring data. To dive into a dataset and discover interesting things using Tidyverse environment. The two popular packages dplyr and ggplot2 are used to manipulate and visualize data. In detail, the objective of this course introduce:

  • For data manipulation: filtering, sorting, and summarizing a real dataset
  • For data visualization: line plots, bar plots, histograms; which turn processed data into information

DataCamp.

1 Data wrangling

1.1 The gapminder dataset

Gapminder tracks economic and social indicators like life expectancy and GDP per capita of countries over time. To take a brief look at the data, the function glimpse(data) could be used.

Exercise 1.1. Loading the gapminder and dplyr packages

# Load the gapminder package for dataset
# Load the dplyr package for function glimpse
library(gapminder) 
library(dplyr)
library(tidyverse)
library(tibble)
library(magrittr)
dplyr::glimpse(gapminder)
## Rows: 1,704
## Columns: 6
## $ country   <fct> "Afghanistan", "Afghanistan", "Afghanistan", "Afghanistan", ~
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, ~
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 1997, ~
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 40.8~
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372, 12~
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.1134, ~

Exercise 1.2. Understanding a data frame

After loading the gapminder dataset, we can examine for a better understanding it. To do so, type gapminder in your R terminal, and see how many observations are there in the dataset?

  1. 1704
  2. 6
  3. 1694
  4. 1952

The correct answer is: a.

1.2 The tibble data frame

Tibble is dataframe, which tweaks (ie. make small adjustments) some older behaviors. What do “older behavior” mean in this context? After importing a dataframe, people from older behaviors (like me, hihi) have the habit of assessing such following information from the data:

  • types of variables
  • number of rows and columns
  • brief look of a subset data

Exercise 1.3. Assess the iris data from older behaviors by means of following functions: str(), head(), nrow() and ncol().

data("iris")
str(iris)
head(iris)
nrow(iris)
ncol(iris)

Tibble is a modern dataframe, which takes into account older behaviors. In other words, tibble is lazy dataframe, but convenient. In R environment, the function as_tibble() from the package tibble could be used to convert traditional dataframe to tibble.

tibble::as_tibble(iris)
## # A tibble: 150 x 5
##    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
##           <dbl>       <dbl>        <dbl>       <dbl> <fct>  
##  1          5.1         3.5          1.4         0.2 setosa 
##  2          4.9         3            1.4         0.2 setosa 
##  3          4.7         3.2          1.3         0.2 setosa 
##  4          4.6         3.1          1.5         0.2 setosa 
##  5          5           3.6          1.4         0.2 setosa 
##  6          5.4         3.9          1.7         0.4 setosa 
##  7          4.6         3.4          1.4         0.3 setosa 
##  8          5           3.4          1.5         0.2 setosa 
##  9          4.4         2.9          1.4         0.2 setosa 
## 10          4.9         3.1          1.5         0.1 setosa 
## # ... with 140 more rows

Beside of taking into account older behaviors, tibble possesses some characteristics:

  • never convert string as factor (suitable for time series, names data)
  • never create row names (flexible for data wrangling)
  • give a better printing appearance

When printed, each column will be assigned in 4 types:

  1. logical vectors: denote <lgl>, which contain TRUE or FALSE.
  2. integer vectors: denote <int>, which contain integers.
  3. double vectors: denote <dbl>, which contain real numbers.
  4. character vectors: denote <chr>, which contain strings made with "__".

All vectors can also contain the missing value, denote NA. Collectively integer and double vectors are known as numeric vectors.

1.3 The filter() verb

The objective of the filter() function is to extract a subset of observations in a dataset based on a defined condition.

Exercise 1.4. Filtering observations for one year

Extracting particular observations based on a condition, eg. extracting data for Vietnam.

# Filter the gapminder dataset for Vietnam
gapminder %>%
  filter(country=="Vietnam")
## # A tibble: 12 x 6
##    country continent  year lifeExp      pop gdpPercap
##    <fct>   <fct>     <int>   <dbl>    <int>     <dbl>
##  1 Vietnam Asia       1952    40.4 26246839      605.
##  2 Vietnam Asia       1957    42.9 28998543      676.
##  3 Vietnam Asia       1962    45.4 33796140      772.
##  4 Vietnam Asia       1967    47.8 39463910      637.
##  5 Vietnam Asia       1972    50.3 44655014      700.
##  6 Vietnam Asia       1977    55.8 50533506      714.
##  7 Vietnam Asia       1982    58.8 56142181      707.
##  8 Vietnam Asia       1987    62.8 62826491      821.
##  9 Vietnam Asia       1992    67.7 69940728      989.
## 10 Vietnam Asia       1997    70.7 76048996     1386.
## 11 Vietnam Asia       2002    73.0 80908147     1764.
## 12 Vietnam Asia       2007    74.2 85262356     2442.

Exercise 1.5. Filtering for one country and one year

You can also use the filter() verb to set two conditions. Note that we can do this in one line of code by starting with gapminder %>%, then having the filter().

However, we should Keep one verb on each line, this could help the code being readable. At each code line, we put the pipe %>% at the end of the first line (like gapminder %>%), then putting the verb at the beginning of the second line.

# Filter for Vietnam in 2002
gapminder %>% 
  filter(country=="Vietnam", year=="2002")
## # A tibble: 1 x 6
##   country continent  year lifeExp      pop gdpPercap
##   <fct>   <fct>     <int>   <dbl>    <int>     <dbl>
## 1 Vietnam Asia       2002    73.0 80908147     1764.

1.4 The arrange() verb

The objective of the arrange() function is to sort observations in a dataset in ascending or descending order. The statement for:

  • Ascending order: data %>% arrange(variable), and for
  • Descending order: data %>% arrange(desc(variable))

We can flexiable combine filter() and arrange() verb to extract a subset of observations, and then sort the subset data extracted.

# statement
# data %>% filter(variable_1 == condition) %>% ## extracting
#          arrange(variable_2)                 ## descending order

Exercise 1.6. Arranging observations by life expectancy

You use arrange() to sort observations in ascending or descending order of a particular variable. In this case, you’ll sort the dataset based on the lifeExp variable.

# Sort in ascending order of lifeExp
gapminder %>% 
  arrange(lifeExp)
## # A tibble: 1,704 x 6
##    country      continent  year lifeExp     pop gdpPercap
##    <fct>        <fct>     <int>   <dbl>   <int>     <dbl>
##  1 Rwanda       Africa     1992    23.6 7290203      737.
##  2 Afghanistan  Asia       1952    28.8 8425333      779.
##  3 Gambia       Africa     1952    30    284320      485.
##  4 Angola       Africa     1952    30.0 4232095     3521.
##  5 Sierra Leone Africa     1952    30.3 2143249      880.
##  6 Afghanistan  Asia       1957    30.3 9240934      821.
##  7 Cambodia     Asia       1977    31.2 6978607      525.
##  8 Mozambique   Africa     1952    31.3 6446316      469.
##  9 Sierra Leone Africa     1957    31.6 2295678     1004.
## 10 Burkina Faso Africa     1952    32.0 4469979      543.
## # ... with 1,694 more rows
# Sort in descending order of lifeExp
gapminder %>% 
  arrange(desc(lifeExp))
## # A tibble: 1,704 x 6
##    country          continent  year lifeExp       pop gdpPercap
##    <fct>            <fct>     <int>   <dbl>     <int>     <dbl>
##  1 Japan            Asia       2007    82.6 127467972    31656.
##  2 Hong Kong, China Asia       2007    82.2   6980412    39725.
##  3 Japan            Asia       2002    82   127065841    28605.
##  4 Iceland          Europe     2007    81.8    301931    36181.
##  5 Switzerland      Europe     2007    81.7   7554661    37506.
##  6 Hong Kong, China Asia       2002    81.5   6762476    30209.
##  7 Australia        Oceania    2007    81.2  20434176    34435.
##  8 Spain            Europe     2007    80.9  40448191    28821.
##  9 Sweden           Europe     2007    80.9   9031088    33860.
## 10 Israel           Asia       2007    80.7   6426679    25523.
## # ... with 1,694 more rows

Exercise 1.7. Filtering and arranging

You’ll often need to use the pipe operator (%>%) to combine multiple dplyr verbs in a row. In this case, you’ll combine a filter() with an arrange() to find the highest population countries in a particular year.

# Filter for the year 1957, then arrange in descending order of population
gapminder %>% 
  filter(year==1957) %>% 
  arrange(desc(pop))
## # A tibble: 142 x 6
##    country        continent  year lifeExp       pop gdpPercap
##    <fct>          <fct>     <int>   <dbl>     <int>     <dbl>
##  1 China          Asia       1957    50.5 637408000      576.
##  2 India          Asia       1957    40.2 409000000      590.
##  3 United States  Americas   1957    69.5 171984000    14847.
##  4 Japan          Asia       1957    65.5  91563009     4318.
##  5 Indonesia      Asia       1957    39.9  90124000      859.
##  6 Germany        Europe     1957    69.1  71019069    10188.
##  7 Brazil         Americas   1957    53.3  65551171     2487.
##  8 United Kingdom Europe     1957    70.4  51430000    11283.
##  9 Bangladesh     Asia       1957    39.3  51365468      662.
## 10 Italy          Europe     1957    67.8  49182000     6249.
## # ... with 132 more rows

1.5 The mutate() verb

The verb mutate, from the English dictionary, means to change from one thing or type of thing into another. In tidyverse environment, the objective of mutate is to: (1) change a variable and/or (2) add a new variable.

  • Statement for changing a variable is as data %>% mutate(var_1 = function(var_1)); Let’s say: replace var_1 by a new variable, which is a function of var_1. As consequence, the var_1 in the dataset is going to be dropped.

  • Statement for adding a new variable is as data %>% mutate(var_2 = function(var_1)). Let’s say: create var_2, which is a function of var_1, and the var_1 is still in the dataset.

Exercise 1.8. Using mutate to change or create a column

Suppose we want life expectancy to be measured in months instead of years: you’d have to multiply the existing value by 12. You can use the mutate() verb to change this column, or to create a new column that’s calculated this way.

# Ex1. Use mutate to change lifeExp to be in months
gapminder %>% 
  mutate(lifeExp = lifeExp*12)
## # A tibble: 1,704 x 6
##    country     continent  year lifeExp      pop gdpPercap
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>
##  1 Afghanistan Asia       1952    346.  8425333      779.
##  2 Afghanistan Asia       1957    364.  9240934      821.
##  3 Afghanistan Asia       1962    384. 10267083      853.
##  4 Afghanistan Asia       1967    408. 11537966      836.
##  5 Afghanistan Asia       1972    433. 13079460      740.
##  6 Afghanistan Asia       1977    461. 14880372      786.
##  7 Afghanistan Asia       1982    478. 12881816      978.
##  8 Afghanistan Asia       1987    490. 13867957      852.
##  9 Afghanistan Asia       1992    500. 16317921      649.
## 10 Afghanistan Asia       1997    501. 22227415      635.
## # ... with 1,694 more rows
# Ex2. Use mutate to create a new column called lifeExpMonths
gapminder %>% 
  mutate(lifeExpMonths = lifeExp*12)
## # A tibble: 1,704 x 7
##    country     continent  year lifeExp      pop gdpPercap lifeExpMonths
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>         <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.          346.
##  2 Afghanistan Asia       1957    30.3  9240934      821.          364.
##  3 Afghanistan Asia       1962    32.0 10267083      853.          384.
##  4 Afghanistan Asia       1967    34.0 11537966      836.          408.
##  5 Afghanistan Asia       1972    36.1 13079460      740.          433.
##  6 Afghanistan Asia       1977    38.4 14880372      786.          461.
##  7 Afghanistan Asia       1982    39.9 12881816      978.          478.
##  8 Afghanistan Asia       1987    40.8 13867957      852.          490.
##  9 Afghanistan Asia       1992    41.7 16317921      649.          500.
## 10 Afghanistan Asia       1997    41.8 22227415      635.          501.
## # ... with 1,694 more rows

As you can see, the value of lifeExp from example 1 is same as that of lifeExpMonths from example 2. Additionally, var lifeExp was dropped from example 1, but not example 2.

Exercise 1.9. Combining filter(), mutate(), and arrange()

In this exercise, you’ll combine all three of the verbs you’ve learned in this chapter, to find the countries with the highest life expectancy, in months, in the year 2007.

# Filter, mutate, and arrange the gapminder dataset
gapminder %>% filter(year == 2007) %>% 
   mutate(lifeExpMonths = 12*lifeExp) %>%
   arrange(desc(lifeExpMonths))
## # A tibble: 142 x 7
##    country          continent  year lifeExp       pop gdpPercap lifeExpMonths
##    <fct>            <fct>     <int>   <dbl>     <int>     <dbl>         <dbl>
##  1 Japan            Asia       2007    82.6 127467972    31656.          991.
##  2 Hong Kong, China Asia       2007    82.2   6980412    39725.          986.
##  3 Iceland          Europe     2007    81.8    301931    36181.          981.
##  4 Switzerland      Europe     2007    81.7   7554661    37506.          980.
##  5 Australia        Oceania    2007    81.2  20434176    34435.          975.
##  6 Spain            Europe     2007    80.9  40448191    28821.          971.
##  7 Sweden           Europe     2007    80.9   9031088    33860.          971.
##  8 Israel           Asia       2007    80.7   6426679    25523.          969.
##  9 France           Europe     2007    80.7  61083916    30470.          968.
## 10 Canada           Americas   2007    80.7  33390141    36319.          968.
## # ... with 132 more rows

1.6 The pivot_longer() and pivot_wider()

1.6.1 From wide to long format using pivot_longer()

Let’s take the example_eagle_nests for an illustration. Figure below shows our starting point (i.e. the wide format) and expected results (ie. the long format).

# https://dcl-wrangle.stanford.edu/pivot_basic.html
library(dcldata)
library(tidyr)
# data(package = "dcldata")
example_eagle_nests
## # A tibble: 3 x 3
##   region                     `2007` `2009`
##   <chr>                       <dbl>  <dbl>
## 1 Pacific                      1039   2587
## 2 Southwest                      51    176
## 3 Rocky Mountains and Plains    200    338

The command lines below illustrates how to use pivot_longer() for the purpose of converting from wide format to long format. The concept of the command lines can be explained in 4 steps.

example_eagle_nests %>% 
  pivot_longer(
    cols = c(`2007`, `2009`), 
    names_to = "year", 
    values_to = "num_nests"
  )
## # A tibble: 6 x 3
##   region                     year  num_nests
##   <chr>                      <chr>     <dbl>
## 1 Pacific                    2007       1039
## 2 Pacific                    2009       2587
## 3 Southwest                  2007         51
## 4 Southwest                  2009        176
## 5 Rocky Mountains and Plains 2007        200
## 6 Rocky Mountains and Plains 2009        338

Step 1: Identify the variables from dataset. From example_eagle_nests data, there are three variables in this dataset:

  • region: The US region where the nests are located.
  • year: The year the nests were found.
  • num_nests: The number of nests found.

Step 2: Identify the columns to pivot. Identify which columns are keeping the data from being tidy. In our example, those columns are 2007 and 2009.

  • The cols argument controls which columns will be stored in the new column of the pivot.
  • Besides of the cols argument, the columns can be specified by other arguments, for instance: contains(), starts_with().
  • When the column names start with numbers as in our case, put them in backticks (eg. 2007)

Step 3: Name new columns of pivot_longer(), which consists of all the columns identified in Step 2.

  • The names_to argument controls the name of the column that will store the old column names. In our case, we want to name this new column “year”. The argument is called names_to because you’re specifying which column to move the column names to.

Note that the argument to names_to has to be in quotes, while the arguments cols do not. It’s easy to get confused about which pivot arguments need to be quoted. Here’s the general rule: if you’re identifying an existing column (e.g., 2007), do not quote. If you’re talking about a column that does not currently exist (e.g., year), quote it.

Step 4: Name the column that will store the column values. That column will store the values from 2007 and 2009.

  • values_to controls the name of column for the values. In example_eagle_nests, the column values represent the number of nests, so we’ll name the new column “num_nests”.

1.6.2 From long to wide format using pivot_wider()

Let’s take the example_eagle_nests for an illustration. Figure below shows our starting point (i.e. the wide format) and expected results (ie. the long format).

As you could have some idea from the previous section, here is the data in form of wide format.

example_eagle_nests_tidy
## # A tibble: 6 x 3
##   region                     year  num_nests
##   <chr>                      <chr>     <dbl>
## 1 Pacific                    2007       1039
## 2 Pacific                    2009       2587
## 3 Southwest                  2007         51
## 4 Southwest                  2009        176
## 5 Rocky Mountains and Plains 2007        200
## 6 Rocky Mountains and Plains 2009        338

pivot_wider() turns the values from one column and turns them into column names, also converting from wide to long format. These command lines could be interpreted by 2 steps.

example_eagle_nests_tidy %>% 
  pivot_wider(
    names_from = year, 
    values_from = num_nests
  )
## # A tibble: 3 x 3
##   region                     `2007` `2009`
##   <chr>                       <dbl>  <dbl>
## 1 Pacific                      1039   2587
## 2 Southwest                      51    176
## 3 Rocky Mountains and Plains    200    338

Step 1: Identify the column whose values will supply the column names. In our example, we want the unique values from year to become column names.

  • names_from argument controls which column is pivoted into column names

Step 2: Identify the column whose values will supply the column values.

  • values_from argument identify num_nests as the column supplying the values of 2007 and 2009.

1.6.3 Unique column issue

# load iris data
library(tibble)
library(dplyr)
iris_tib <- as.tibble(iris)
## Warning: `as.tibble()` was deprecated in tibble 2.0.0.
## Please use `as_tibble()` instead.
## The signature and semantics have changed, see `?as_tibble`.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was generated.
iris_tib %>% mutate("ID"=c(1:nrow(iris)))
## # A tibble: 150 x 6
##    Sepal.Length Sepal.Width Petal.Length Petal.Width Species    ID
##           <dbl>       <dbl>        <dbl>       <dbl> <fct>   <int>
##  1          5.1         3.5          1.4         0.2 setosa      1
##  2          4.9         3            1.4         0.2 setosa      2
##  3          4.7         3.2          1.3         0.2 setosa      3
##  4          4.6         3.1          1.5         0.2 setosa      4
##  5          5           3.6          1.4         0.2 setosa      5
##  6          5.4         3.9          1.7         0.4 setosa      6
##  7          4.6         3.4          1.4         0.3 setosa      7
##  8          5           3.4          1.5         0.2 setosa      8
##  9          4.4         2.9          1.4         0.2 setosa      9
## 10          4.9         3.1          1.5         0.1 setosa     10
## # ... with 140 more rows
# using pivot_longer
# - data: data frame to pivot
# - cols: columns to become one new column in longer format.
#         for selecting multiple columns: cols = c("var1", "var2")
# - names_to: "new column name"
# - values_to: "new value column name"
iris_tib_long <- pivot_longer(iris_tib, 
                              cols = "Sepal.Length":"Petal.Width",
               names_to = "iris_part",
               values_to = "measure_cm")
iris_tib_long
## # A tibble: 600 x 3
##    Species iris_part    measure_cm
##    <fct>   <chr>             <dbl>
##  1 setosa  Sepal.Length        5.1
##  2 setosa  Sepal.Width         3.5
##  3 setosa  Petal.Length        1.4
##  4 setosa  Petal.Width         0.2
##  5 setosa  Sepal.Length        4.9
##  6 setosa  Sepal.Width         3  
##  7 setosa  Petal.Length        1.4
##  8 setosa  Petal.Width         0.2
##  9 setosa  Sepal.Length        4.7
## 10 setosa  Sepal.Width         3.2
## # ... with 590 more rows
# using pivot_wider
# - data: data frame to pivot
# - cols: identifier columns not to pivot.
#         for selecting multiple columns: cols = c("var1", "var2")
# - names_from: "new column name"
# - values_from: "new value column name"
# iris_tib_wide <- pivot_wider(iris_tib_long,
#                             names_from = iris_part,
#                       values_from = measure_cm)

2 Data visualization

This chapter will show you how to create informative graphs, for instance: scatterplots, histogram, and boxplot using the two packages dplyr and ggplot2.

2.1 Visualizing with ggplot2

Assignment, denoted <-, this is a less and then a minus sign. This tibble has 142 rows, 6 columns and comes from the year 2007. In this example, we will:

  • create a gapminder_2007 dataset, which is a subset of the gapminder data from the year 1952, then
  • create a scatter plot where GDP per capita (gdpPercap) on the X-axis and life expectancy (lifeExp) on the Y-axis.
# extract the subset data (year == 2007)
gapminder_2007 <- gapminder %>% 
  filter (year == 2007)
# create scatter plot
library(ggplot2)
ggplot2::ggplot(gapminder_2007, 
                aes(x=gdpPercap, y=lifeExp)) + 
  geom_point() +
  theme_bw()

This plot communicates some interesting information: the higher income countries tend to have higher life expectancy.

Exercise 2.1. Comparing population and life expectancy

In this exercise, you’ll use ggplot2 to create a scatter plot to compare each country’s population with its life expectancy in the year 1952. Run the command below, then explain this informative graph.

# extract the subset data
gapminder_1952 <- gapminder %>%
  filter(year == 1952)
# create a scatter plot with pop on the x-axis and lifeExp on the y-axis
ggplot(gapminder_1952, aes(x=pop, y=lifeExp)) + 
  geom_point() +
  theme_bw()

2.2 Transforming log scales

One problem with the plot above is that a lot of countries get crammed into the leftmost part of the x-axis. The reason for that is because the distribution of GDP per capita (gdpPercap) spans several orders of magnitude. In this, some countries in the tens of thousands dollars and others in the hundreds. When one of your axes has that kind of distribution, it’s useful to work with a logarithmic scale. In logarithmic scale, each fixed distance represents a multiplication of the value. Below is what the scatter plot where x is represented on a log scale.

ggplot2::ggplot(gapminder_2007, 
                aes(x=gdpPercap, y=lifeExp)) + 
  geom_point() +
  scale_x_log10() +
  theme_bw()

You can see the axis goes from \(300\) to \(30.000\) and the next step of an equal distance would be \(100.000\). On this scale, the relationship between GDP per capita and life expectancy looks more linear. As consequence, you can more easily distinguish the countries at the lower end of the spectrum.

To create this plot, we add a command line scale_x_log10() just after the geom_point(). This command line specifies that we want the x-axis on a log10 scale. It’s not necessary in this case, but if you want to put the y-axis on a log10 scale as well, you can add another command line as scale_y_log10().

Exercise 2.2. Putting the x-axis on a log scale

You previously created a scatter plot with population on the x-axis and life expectancy on the y-axis. Since population is spread over several orders of magnitude, with some countries having a much higher population than others, it’s a good idea to put the x-axis on a log scale.

gapminder_1952 <- gapminder %>%
  filter(year == 1952)

# Change this plot to put the x-axis on a log scale
ggplot(gapminder_1952, aes(x = pop, y = lifeExp)) +
  geom_point() + 
  scale_x_log10()

Exercise 2.3. Putting both the x- and y- axes on a log scale

Suppose you want to create a scatter plot with population on the x-axis and GDP per capita on the y-axis. Both population and GDP per-capita are better represented with log scales, since they vary over many orders of magnitude.

gapminder_1952 <- gapminder %>%
  dplyr::filter(year == 1952)

# Scatter plot comparing pop and gdpPercap, with both axes on a log scale
ggplot2::ggplot(gapminder_1952, aes(x=pop,y=gdpPercap)) + 
  geom_point() + 
  scale_x_log10() + 
  scale_y_log10()

2.3 Additional aesthetics with the arguments color and size

Besides gdpPercap and lifeExp, we also have continent and population variables that are informative. We could examine relationships among all these variables in the same plot. Due to the fact that we already use the x-axis to represent GDP per capita and the y-axis to represent life expectancy, we will add three more aesthetics: color, size and facet. From technical point of view, the argument color allows to add information from a qualitative variable, whereas the argument size allows to add information from a quantitative variable. Additional aesthetics is a useful way to communicate the scatter plot with more than 2 quantitative variables.

ggplot2::ggplot(gapminder_2007, 
                aes(x=gdpPercap, y=lifeExp,
                    color = continent,
                    size = pop)) + 
  geom_point() +
  scale_x_log10() +
  theme_bw()

Exercise 2.4. Adding color to a scatter plot

In this lesson you learned how to use the color aesthetic, which can be used to show which continent each point in a scatter plot represents.

gapminder_1952 <- gapminder %>% 
  dplyr::filter(year == 1952)

# Scatter plot comparing pop and lifeExp, with color representing continent
ggplot2::ggplot(gapminder_1952, 
                aes(x=pop, y=lifeExp, color=continent)) + 
  geom_point() + 
  scale_x_log10()

Exercise 2.5. Adding size and color to a plot

In the last exercise, you created a scatter plot communicating information about each country’s population, life expectancy, and continent. Now you’ll use the size of the points to communicate even more.

gapminder_1952 <- gapminder %>%
  dplyr::filter(year == 1952)

# Add the size aesthetic to represent a country's gdpPercap
ggplot2::ggplot(gapminder_1952, 
                aes(x = pop, y = lifeExp, 
                    color = continent, 
                    size = gdpPercap)) + 
  geom_point() + 
  scale_x_log10()

2.4 Faceting

Facet means one side of something many-sided. Faceting in ggplot2 divides your plot into subplots to get smaller graph for each level of the categorical variable (ie. one side, the so-call faceting). Faceting is an alternative way to the argument color for the purpose of adding a qualitative information to scatter plot.

To facet a plot, we add an option facet_wrap(~variable). Wrap means to cover or surround something with paper, cloth, or other material. Tilde sign (~) means by. The command line facet_wrap(~continent) tells ggplot() to divide data into subplots based on the continent variable

ggplot2::ggplot(gapminder_2007, 
                aes(x=gdpPercap, y=lifeExp)) + 
  geom_point() +
  scale_x_log10() +
  facet_wrap(~continent) + 
  theme_bw()

Exercise 2.6. Creating a subgraph for each continent

You’ve learned to use faceting to divide a graph into subplots based on one of its variables, such as the continent.

gapminder_1952 <- gapminder %>%
  filter(year == 1952)

# Scatter plot comparing pop and lifeExp, faceted by continent
ggplot2::ggplot(gapminder_1952, aes(x=pop, y=lifeExp)) + 
  geom_point() + 
  scale_x_log10() + 
  facet_wrap(~continent) + 
  theme_bw()

Exercise 2.7 . Faceting by year

All of the graphs in this chapter have been visualizing statistics within one year. Now that you’re able to use faceting, however, you can create a graph showing all the country-level data from 1952 to 2007, to understand how global statistics have changed over time.

# Scatter plot comparing gdpPercap and lifeExp, with color representing continent
# and size representing population, faceted by year
ggplot(gapminder, 
       aes(x=gdpPercap, y=lifeExp, color=continent, size=pop)) +
  geom_point() +
  scale_x_log10() + 
  facet_wrap(~year) + 
  theme_bw()

3 Grouping and summarizing

3.1 The summarize() verb, not summarise()

Suppose we want to know the average life expectancy across all countries and years in the dataset. From older fashion, I use the function mean() as follow:

mean(gapminder$lifeExp)
## [1] 59.47444

With the modern fashion, the function summarize() could be used as command line below. First, take gapminder data, pipe it into summarize, and specify that we want to create a summary column called meanLifeExp.

gapminder %>% 
  summarize(meanLifeExp = mean(lifeExp))
## # A tibble: 1 x 1
##   meanLifeExp
##         <dbl>
## 1        59.5

If we just compare the two coding fashion in calculating the mean, it does not make sense to learn and apply the verb summarize(). It makes more sense to ask questions about: (1) the average in a particular condition and (2) creating multiple summarize at once. Interestingly, all the statistic functions, for instance: mean(), sum(), median(), min(), max(), var(), or sd() could be used in combination with the fuction summarize().

For example, we want to find the average of life expectancy for the year 2007 in the one part, and also the media population in that year. To do that, we add a comma after the mean of the life expectancy, then add comment medianPop=median(pop). As you can see in the results, a new column totalPop is added into the output table.

gapminder %>% 
  filter(year == 2007) %>% 
  summarize(meanLifeExp = mean(lifeExp),
            medianPop = median(pop),
            varLifeExp = var(lifeExp),
            sdPop=sd(pop))
## # A tibble: 1 x 4
##   meanLifeExp medianPop varLifeExp      sdPop
##         <dbl>     <dbl>      <dbl>      <dbl>
## 1        67.0  10517531       146. 147621398.

Exercise 3.1. Summarizing the median life expectancy

# Summarize to find the median life expectancy
gapminder %>% 
  summarize(medianLifeExp = median(lifeExp))

Exercise 3.2. Summarizing the median life expectancy in 1957

# Filter for 1957 then summarize the median life expectancy
gapminder %>% 
  filter(year==1957) %>% 
  summarize(medianLifeExp = median(lifeExp))

Exercise 3.3. Summarizing multiple variables in 1957

# Filter for 1957, then summarize the median life expectancy
#   and the maximum GDP per capita
gapminder %>% 
  filter(year == 1957) %>% 
  summarize(medianLifeExp = median(lifeExp), 
            maxGdpPercap=max(gdpPercap))

3.2 The group_by() verb

What if we were not interested in the year 2007, but for each year in the dataset? We could re-run this code and change the year for each time, but that’s very tedious. Instead, we can used the group_by verb to summarize within group, not the entire dataset. Let’s do this by replacing the command filter(year == 2007) by group_by(year):

# define pop as numeric
# as.numeric is identical to as.double
gapminder$pop <- as.numeric(gapminder$pop)
# summarize meanLifeExp and totalPop by year
gapminder %>% 
  group_by(year) %>%
  summarize(meanLifeExp = mean(lifeExp),
            totalPop=sum(pop))
## # A tibble: 12 x 3
##     year meanLifeExp   totalPop
##    <int>       <dbl>      <dbl>
##  1  1952        49.1 2406957150
##  2  1957        51.5 2664404580
##  3  1962        53.6 2899782974
##  4  1967        55.7 3217478384
##  5  1972        57.6 3576977158
##  6  1977        59.6 3930045807
##  7  1982        61.5 4289436840
##  8  1987        63.2 4691477418
##  9  1992        64.2 5110710260
## 10  1997        65.0 5515204472
## 11  2002        65.7 5886977579
## 12  2007        67.0 6251013179

As you can see, we now get one row for each year. Rather than that, we have the average life expectancy and total population in 2007 within each continent. A little bit difficult, in case where we want to assess the information for both all the years and for all continents, we can add all the two variables in group_by().

gapminder %>% 
  group_by(year, continent) %>%
  summarize(meanLifeExp = mean(lifeExp),
            totalPop=sum(pop))
## `summarise()` has grouped output by 'year'. You can override using the `.groups` argument.
## # A tibble: 60 x 4
## # Groups:   year [12]
##     year continent meanLifeExp   totalPop
##    <int> <fct>           <dbl>      <dbl>
##  1  1952 Africa           39.1  237640501
##  2  1952 Americas         53.3  345152446
##  3  1952 Asia             46.3 1395357351
##  4  1952 Europe           64.4  418120846
##  5  1952 Oceania          69.3   10686006
##  6  1957 Africa           41.3  264837738
##  7  1957 Americas         56.0  386953916
##  8  1957 Asia             49.3 1562780599
##  9  1957 Europe           66.7  437890351
## 10  1957 Oceania          70.3   11941976
## # ... with 50 more rows

Exercise 3.4. Running the code below and tell me what happens.

Remark that continent is put before year, what if this order will be changed?

gapminder$pop <- as.numeric(gapminder$pop)
gapminder %>% 
   group_by(continent, year) %>%
   summarize(meanLifeExp = mean(lifeExp),
             totalPop=sum(pop))
## `summarise()` has grouped output by 'continent'. You can override using the `.groups` argument.

Exercise 3.5. Summarizing by year

In a previous exercise, you found the median life expectancy and the maximum GDP per capita in the year 1957. Now, you’ll perform those two summaries within each year in the dataset, using the group_by verb.

# Find median life expectancy and 
#      maximum GDP per capita in each year
gapminder %>% 
  group_by(year) %>% 
  summarize(medianLifeExp = median(lifeExp), 
            maxGdpPercap = max(gdpPercap))

Exercise 3.6. Summarizing by continent

You can group by any variable in your dataset to create a summary. Rather than comparing across time, you might be interested in comparing among continents. You’ll want to do that within one year of the dataset: let’s use 1957.

# Find median life expectancy and 
#      maximum GDP per capita in each continent in 1957
gapminder %>% 
  filter(year == 1957) %>% 
  group_by(continent) %>% 
  summarize(medianLifeExp = median(lifeExp), 
            maxGdpPercap = max(gdpPercap))

Exercise 3.7. Summarizing by continent and year

Instead of grouping just by year, or just by continent, you’ll now group by both continent and year to summarize within each.

# Find median life expectancy and 
#      maximum GDP per capita in each continent, 
#      per year combination
gapminder %>% 
  group_by(continent, year) %>% 
  summarize(medianLifeExp = median(lifeExp), 
            maxGdpPercap = max(gdpPercap))
## `summarise()` has grouped output by 'continent'. You can override using the `.groups` argument.

3.3 Visualizing summarized data

After summarizing the data, we can graphically visualize this using ggplot2 package.

# extract the by_year data
by_year <- gapminder %>% 
  group_by(year) %>% 
  summarize(totalPop = sum(pop), 
            meanLifeExp=mean(lifeExp))
# scatter plot
ggplot2::ggplot(by_year, aes(x=year, y=totalPop)) + 
  geom_point() + 
  theme_bw()

The resulting graph of population by year shows the change in the total population, which is going up over time. It starts under 3 times 10 to the 9th power (ie. 3 billions) and go up to more than 6 billions. This graph is a little misleading because it doesn’t include zero. In fact, we do not have a sense of how much the population grew relative to where it was when it started.

In order to add/start with zero point for the y-axis, we use a new graphical option as expand_limits(y=0) to the end of the script.

# summarize by year
by_year <- gapminder %>% 
  group_by(year) %>% 
  summarize(totalPop = sum(pop), 
            meanLifeExp=mean(lifeExp))
# plot 
ggplot2::ggplot(by_year, aes(x=year, y=totalPop)) + 
  geom_point() + 
  expand_limits(y=0) +
  theme_bw()

By separated by continent and visualizing the data overtime, we can say that Asia is populated continent. And Africa has grown to surpass

# summarize by year and continent
by_year_continent <- gapminder %>% 
  group_by(year, continent) %>%
  summarize(totalPop = sum(pop), 
            meanLifeExp=mean(lifeExp))
## `summarise()` has grouped output by 'year'. You can override using the `.groups` argument.
# scatter plot with adding aesthetic as continent
ggplot(by_year_continent, 
       aes(x=year, y=totalPop, color = continent)) + 
  geom_point(alpha=0.7) + 
  expand_limits(y=0) + 
  theme_bw()

You could often combine the verb supported by dplyr and ggplot2 as a part of data exploration phase. So it’s important to get into the habit of visualizing summarized or processed data.

Exercise 3.8. Visualizing median life expectancy over time

In the last chapter, you summarized the gapminder data to calculate the median life expectancy within each year. This code is provided for you, and is saved (with <-) as the by_year dataset. Now you can use the ggplot2 package to turn this into a visualization of changing life expectancy over time.

by_year <- gapminder %>%
  group_by(year) %>%
  summarize(medianLifeExp = median(lifeExp),
            maxGdpPercap = max(gdpPercap))
# Create a scatter plot showing the change 
#        in medianLifeExp over time
ggplot2::ggplot(by_year, aes(x=year, y=medianLifeExp)) + 
  geom_point() + 
  expand_limits(y=0)

Exercise 3.9 Visualizing median GDP per capita per continent over time

In the last exercise you were able to see how the median life expectancy of countries changed over time. Now you’ll examine the median GDP per capita instead, and see how the trend differs among continents.

# Summarize medianGdpPercap within each continent within each year: 
by_year_continent <- gapminder %>% 
  group_by(year,continent) %>% 
  summarize(medianGdpPercap = median(gdpPercap))
## `summarise()` has grouped output by 'year'. You can override using the `.groups` argument.
# Plot the change in medianGdpPercap in each continent over time
ggplot2::ggplot(by_year_continent, 
                aes(x=year, y=medianGdpPercap, color=continent)) + 
  geom_point() + 
  expand_limits(y=0)

Exercise 3.10. Comparing median life expectancy and median GDP per continent in 2007

In these exercises you’ve generally created plots that show change over time. But as another way of exploring your data visually, you can also use ggplot2 to plot summarized data to compare continents within a single year.

# Summarize the median GDP and median life expectancy 
#   per continent in 2007
by_continent_2007 <- gapminder %>% 
  filter(year==2007) %>% 
  group_by(continent) %>%
  summarize(medianLifeExp = median(lifeExp), 
            medianGdpPercap = median(gdpPercap))
# Use a scatter plot to compare the median GDP 
#    and median life expectancy
ggplot2::ggplot(by_continent_2007, 
                aes(x=medianGdpPercap, y=medianLifeExp, color=continent)) +
  geom_point() + 
  expand_limits(y=0)

4 Type of visualization

4.1 Line plots

A line plot is useful for visualizing trends over time. In this section, we’ll examine how the median GDP per capita has changed over time using the argument geom_line().

# Summarize the median gdpPercap by year, then save it as by_year
by_year <- gapminder %>% 
  group_by(year) %>% 
  summarize(medianGdpPercap = median(gdpPercap))

# Create a line plot showing the change in medianGdpPercap over time
ggplot2::ggplot(by_year, 
                aes(x=year, y=medianGdpPercap)) + 
  geom_line() + 
  expand_limits(y=0) +
  theme_bw()

Exercise 4.1. Visualizing median GDP per capita by continent over time

In the last exercise you used a line plot to visualize the increase in median GDP per capita over time. Now you’ll examine the change within each continent.

# Summarize the median gdpPercap by year & continent, save as by_year_continent
by_year_continent <- gapminder %>% 
  group_by(year,continent) %>% 
  summarize(medianGdpPercap = median(gdpPercap))
## `summarise()` has grouped output by 'year'. You can override using the `.groups` argument.
# Create a line plot showing the change 
#  in medianGdpPercap by continent over time
ggplot2::ggplot(by_year_continent, 
                aes(x=year, y=medianGdpPercap, color=continent)) + 
  geom_line() + 
  expand_limits(y=0) + 
  theme_bw()

4.2 Bar plots

Bar plots are useful for comparing values across categories. The by_continent data show the average lifeExp within each continent in the year 2007. Instead of just printing the table, we want to represent graphically. To do so, barplot could be used. In such graph, one bar represents a continent, whose height represent the mean of life expectancy.

To create a bar plot, we add a command geom_col where col short for column. Remarkly in aesthetic, x has to be a qualitative variable and y has to be a quantitative variable. Unlike scatter plot or line plot, bar plot always starts from 0.

# extract data
by_continent <- gapminder %>% 
  filter(year==2007) %>%
  group_by(continent) %>% 
  summarize(meanLifeExp = mean(lifeExp))
# barplot
ggplot2::ggplot(by_continent, 
                aes(x=continent, y=meanLifeExp)) +
  geom_col() +
  theme_bw()

Exercise 4.2. Visualizing median GDP per capita by continent

A bar plot is useful for visualizing summary statistics, such as the median GDP in each continent.

# summarize the median gdpPercap by continent in 1952
by_continent <- gapminder %>% 
  filter(year==1952) %>%
  group_by(continent) %>% 
  summarize(medianGdpPercap = median(gdpPercap))
# create a bar plot showing medianGdp by continent
ggplot2::ggplot(by_continent, 
                aes(x=continent, y=medianGdpPercap)) + 
  geom_col() +
  theme_bw()

Exercise 4.3. Visualizing GDP per capita by country in Oceania

You’ve created a plot where each bar represents one continent, showing the median GDP per capita for each. But the x-axis of the bar plot doesn’t have to be the continent: you can instead create a bar plot where each bar represents a country. In this exercise, you’ll create a bar plot comparing the GDP per capita between the two countries in the Oceania continent (Australia and New Zealand).

# Filter for observations in the Oceania continent in 1952
oceania_1952 <- gapminder %>% 
  filter(continent=="Oceania", year==1952)

# Create a bar plot of gdpPercap by country
ggplot2::ggplot(oceania_1952, aes(x=country, y=gdpPercap)) + 
  geom_col() +
  theme_bw()

4.3 Histograms

Each kind of graph offers a different way to investigate the data. Histogram allows to investigate one dimension (ie. one variable) of the data at a time. Histogram shows information about the distribution of that variable along with its shape. Below is an illustration of life expectancy across countries in the year 2007. Each bar represents a bin of life expectancies and the height represents frequency (how many countries) fall into that bin.

A histogram is created with argument geom_histogram(). Remarkly, histogram has only one aesthetic (ie. the x-axis) that is the variable we are assessing. As you can see in the graph below, most of countries have expectancies between 70 and 80. Another countries have expectancies between 40 and 65.

# extract data
gapminder_2007 <- gapminder %>% 
  filter(year==2007)
# plot histogram
ggplot2::ggplot(gapminder_2007, aes(x=lifeExp)) + 
  geom_histogram(binwidth = 5) +
  theme_bw()

The width of each bin in the histogram is chosen automatically, and it has a large effect on how the histogram communicates the distribution. You may need customize that width. You can do so with the binwidth option. Setting binwidth=5 means that each of the bars in the histogram represents a width of 5 years. In some cases, you need to put x-axis of a histogram on a log scale for a better understandable. To do so, you could use argument scale_x_log10() as mentioned before.

Exercise 4.4. Visualizing population

A histogram is useful for examining the distribution of a numeric variable. In this exercise, you’ll create a histogram showing the distribution of country populations (by millions) in the year 1952. Code for generating this dataset, gapminder_1952, is provided.

gapminder_1952 <- gapminder %>%
  filter(year == 1952) %>%
  mutate(pop_by_mil = pop / 1000000)
# Create a histogram of population (pop_by_mil)
ggplot2::ggplot(gapminder_1952, aes(x=pop_by_mil)) + 
  geom_histogram(bins=50) +
  theme_bw()

Exercise 4.5. Visualizing population with x-axis on a log scale

In the last exercise you created a histogram of populations across countries. You might have noticed that there were several countries with a much higher population than others, which causes the distribution to be very skewed, with most of the distribution crammed into a small part of the graph. (Consider that it’s hard to tell the median or the minimum population from that histogram). To make the histogram more informative, you can try putting the x-axis on a log scale.

gapminder_1952 <- gapminder %>%
  filter(year == 1952)
# Create a histogram of population (pop), with x on a log scale
ggplot2::ggplot(gapminder_1952, aes(x=pop)) + 
  scale_x_log10() + 
  geom_histogram()
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

4.4 Boxplots

Histogram shows distribution of a (quantitative variable) across all continents. What if we want to compare among the continents in terms of lifeExp ? To do so, boxplot is a good solution. Boxplot is created by using argument geom_boxplot() with two information for aes: x-axis is a qualitative variable and y-axis is a quantitative variable.

ggplot2::ggplot(gapminder_2007, aes(x=continent, y=lifeExp)) + 
  geom_boxplot() +
  theme_bw()

Exercise 4.6. Comparing GDP per capita across continents

A boxplot is useful for comparing a distribution of values across several groups. In this exercise, you’ll examine the distribution of GDP per capita by continent. Since GDP per capita varies across several orders of magnitude, you’ll need to put the y-axis on a log scale.

gapminder_1952 <- gapminder %>%
  filter(year == 1952)
# Create a boxplot comparing gdpPercap among continents
ggplot2::ggplot(gapminder_1952, aes(x=continent, y=gdpPercap)) + 
  geom_boxplot() + 
  scale_y_log10() +
  theme_bw()

Exercise 4.7. Adding a title to your graph

There are many other options for customizing a ggplot2 graph, which you can learn about in other DataCamp courses. You can also learn about them from online resources, which is an important skill to develop. As the final exercise in this course, you’ll practice looking up ggplot2 instructions by completing a task we haven’t shown you how to do.

# Add a title to this graph: "Comparing GDP per capita across continents"
ggplot(gapminder_1952, aes(x = continent, y = gdpPercap)) +
  geom_boxplot() +
  scale_y_log10() + 
  theme_bw() +
  ggtitle("Comparing GDP per capita across continents")

4.5 Conclusions

Next steps, enroll some courses:

  1. Data visualization with ggplot2
  2. Data manipulation with dplyr
  3. Import and cleaning data
  4. Interactive Data Visualization with rbokeh
  5. Exploratory Data Analysis in R: Case study