Data Science: Visualization

  • Course Instructor: Rafael Irizarry

Abstract

This is the second in a series of courses in a Professional Certificate in Data Science program. The courses in the Professional Certificate program are designed to prepare you to do data analysis in R, from simple computations to machine learning. If you need a refresher of some basic R, check out Data Science: R Basics, the first course in this series.

The textbook for the Data Science course series is freely available online.

Learning Objectives

  • Data visualization principles to better communicate data-driven findings
  • How to use ggplot2 to create custom plots
  • The weaknesses of several widely used plots and why you should avoid them

Course Overview

Section 1: Introduction to Data Visualization and Distributions
You will get started with data visualization and distributions in R.

Section 2: Introduction to ggplot2
You will learn how to use ggplot2 to create plots.

Section 3: Summarizing with dplyr
You will learn how to summarize data using dplyr.

Section 4: Gapminder
You will see examples of ggplot2 and dplyr in action with the Gapminder dataset.

Section 5: Data Visualization Principles
You will learn general principles to guide you in developing effective data visualizations.

Section 1 Overview

Section 1 introduces you to Data Visualization and Distributions.

After completing Section 1, you will:

  • understand the importance of data visualization for communicating data-driven findings.
  • be able to use distributions to summarize data.
  • be able to use the average and the standard deviation to understand the normal distribution.
  • be able to assess how well a normal distribution fits the data using a quantile-quantile plot.
  • be able to interpret data from a boxplot.

The textbook for this section is available here

Assessment 1 (Data Types)

  1. Variable names
    Let’s start by reviewing how to extract the variable names from a dataset using the names function. What are the two variable names used in the heights dataset?
## [1] "sex"    "height"
  1. Variable type
    We saw that sex is the first variable. We know what values are represented by this variable and can confirm this by looking at the first few entires:

What data type is the sex variable?

A. Continuous

B. Categorical

C. Ordinal

D. None of the above

  1. Numerical values
    Use the unique and length functions to determine how many unique heights were reported.
## [1] 139
  1. Tables
    Use the table function to compute the frequencies of each unique height value. Because we are using the resulting frequency table in a later exercise we want you to save the results into an object and call it tab.
  1. Indicator variables
    In the previous exercise we computed the variable tab which reports the number of times each unique value appears. For values reported only once tab will be 1. Use logicals and the function sum to count the number of times this happens.
## [1] 63
  1. Data types - heights
    Since there are a finite number of reported heights and technically the height can be considered ordinal, which of the following is true:

A. It is more effective to consider heights to be numerical given the number of unique values we observe and the fact that if we keep collecting data even more will be observed.

B. It is actually preferable to consider heights ordinal since on a computer there are only a finite number of possibilities.

C. This is actually a categorical variable: tall, medium or short.

D. This is a numerical variable because numbers are used to represent it.

Assessment 2 (Distribution)

  1. Distributions - 1
    In the murders dataset, the region is a categorical variable and the following is its distribution:
    distribution

To the closet 5%, what proportion of the states are in the North Central region?

A. 75%

B. 50%

C. 20%

D. 5%

  1. Distributions - 2
    Which of the following is true:

A. The graph above is a histogram.

B. The graph above shows only four numbers with a bar plot.

C. Categories are not numbers, so it does not make sense to graph the distribution.

D. The colors, not the height of the bars, describe the distribution.

  1. The plot below shows the eCDF for male heights:
    eCDF

Based on the plot, what percentage of males are shorter than 75 inches?

A. 100%

B. 95%

C. 80%

D. 72 inches

  1. To the closest inch, what height m has the property that 1/2 of the male students are taller than m and 1/2 are shorter?

A. 61 inches

B. 64 inches

C. 69 inches

D. 74 inches

  1. Here is an eCDF of the murder rates across states:
    eCDF

Knowing that there are 51 states (counting DC) and based on this plot, how many states have murder rates larger than 10 per 100,000 people?

A. 1

B. 5

C. 10

D. 50

  1. Based on the eCDF above, which of the following statements are true:

A. About half the states have murder rates above 7 per 100,000 and the other half below.

B. Most states have murder rates below 2 per 100,000.

C. All the states have murder rates above 2 per 100,000.

D. With the exception of 4 states, the murder rates are below 5 per 100,000.

  1. Below is a histogram of male heights in our heights dataset:
    histogram-male-heights

Based on this plot, how many males are between 62.5 and 65.5?

A. 5

B. 24

C. 44

D. 100

  1. About what percentage are shorter than 60 inches?

A. 1%

B. 10%

C. 25%

D. 50%

  1. Based on the density plot below, about what proportion of US states have populations larger than 10 million?
    density-plot-populations

A. 0.02

B. 0.15

C. 0.50

D. 0.55

  1. Below are three density plots. Is it possible that they are from the same dataset?
    3-density-plot

Which of the following statements is true:

A. It is impossible that they are from the same dataset.

B. They are from the same dataset, but the plots are different due to code errors.

C. They are the same dataset, but the first and second plot undersmooth and the third oversmooths.

D. They are the same dataset, but the first is not in the log scale, the second undersmooths and the third oversmooths.

Assessment 3 (Normal Distribution)

  1. Proportions
    Histograms and density plots provide excellent summaries of a distribution. But can we summarize even further? We often see the average and standard deviation used as summary statistics: a two number summary! To understand what these summaries are and why they are so widely used, we need to understand the normal distribution.

The normal distribution, also known as the bell curve and as the Gaussian distribution, is one of the most famous mathematical concepts in history. A reason for this is that approximately normal distributions occur in many situations. Examples include gambling winnings, heights, weights, blood pressure, standardized test scores, and experimental measurement errors. Often data visualization is needed to confirm that our data follows a normal distribution.

Here we focus on how the normal distribution helps us summarize data and can be useful in practice.

One way the normal distribution is useful is that it can be used to approximate the distribution of a list of numbers without having access to the entire list. We will demonstrate this with the heights dataset.

Load the height data set and create a vector x with just the male heights:

What proportion of the data is between 69 and 72 inches (taller than 69 but shorter or equal to 72)?

## [1] 0.3337438
  1. Averages and Standard Deviations
    Suppose all you know about the height data from the previous exercise is the average and the standard deviation and that its distribution is approximated by the normal distribution. We can compute the average and standard deviation like this:

Suppose you only have avg and stdev below, but no access to x, can you approximate the proportion of the data that is between 69 and 72 inches?

Use the normal approximation to estimate the proportion the proportion of the data that is between 69 and 72 inches.
Note that you can’t use x in your code, only avg and stdev. Also note that R has a function that may prove very helpful here - check out the pnorm function (and remember that you can get help by using ?pnorm)

## [1] 0.3061779
  1. Approximations
    Notice that the approximation calculated in the second question is very close to the exact calculation in the first question. The normal distribution was a useful approximation for this case.

However, the approximation is not always useful. An example is for the more extreme values, often called the “tails” of the distribution. Let’s look at an example. We can compute the proportion of heights between 79 and 81.

## [1] 0.004926108

Use normal approximation to estimate the proportion of heights between 79 and 81 inches and save it in an object called approx.
Report how many times bigger the actual proportion is compared to the approximation.

## [1] 1.614261
  1. Seven footers and the NBA
    Someone asks you what percent of seven footers are in the National Basketball Association (NBA). Can you provide an estimate? Let’s try using the normal approximation to answer this question.

First, we will estimate the proportion of adult men that are 7 feet tall or taller.

Assume that the distribution of adult men in the world as normally distributed with an average of 69 inches and a standard deviation of 3 inches.
Using this approximation, estimate the proportion of adult men that are 7 feet tall or taller, referred to as seven footers. Print out your estimate; don’t store it in an object.

## [1] 2.866516e-07
  1. Estimating the number seven footers
    Now we have an approximation for the proportion, call it p, of men that are 7 feet tall or taller.

We know that there are about 1 billion men between the ages of 18 and 40 in the world, the age range for the NBA.

Can we use the normal distribution to estimate how many of these 1 billion men are at least seven feet tall? Use your answer to the previous exercise to estimate the proportion of men that are seven feet tall or taller in the world and store that value as p.
Then round the number of 18-40 year old men who are seven feet tall or taller to the nearest integer. (Do not store this value in an object.)

## [1] 48
  1. How many seven footers are in the NBA?
    There are about 10 National Basketball Association (NBA) players that are 7 feet tall or higher.

Use your answer to exercise 4 to estimate the proportion of men that are seven feet tall or taller in the world and store that value as p. 

Use your answer to the previous exercise (exercise 5) to round the number of 18-40 year old men who are seven feet tall or taller to the nearest integer and store that value as N.

Then calculate the proportion of the world’s 18 to 40 year old seven footers that are in the NBA. (Do not store this value in an object.)

## [1] 0.03484321
  1. Lebron James’ height
    In the previous exerceise we estimated the proportion of seven footers in the NBA using this simple code:
## [1] 0.03484321

Repeat the calculations performed in the previous question for Lebron James’ height: 6 feet 8 inches. There are about 150 players, instead of 10, that are at least that tall in the NBA.

Report the estimated proportion of people at least Lebron’s height that are in the NBA.

## [1] 0.001220842
  1. Interpretation In answering the previous questions, we found that it is not at all rare for a seven footer to become an NBA player.

What would be a fair critique of our calculations?

A. Practice and talent are what make a great basketball player, not height.

B. The normal approximation is not appropriate for heights.

C. As seen in exercise 3, the normal approximation tends to underestimate the extreme values. It's possible that there are more seven footers than we predicted.

D. As seen in exercise 3, the normal approximation tends to overestimate the extreme values. It’s possible that there are less seven footers than we predicted.

Assessment 4 (Quantiles, percentiles, and boxplots)

  1. Vector lengths
    When analyzing data it’s often important to know the number of measurements you have for each category.
    Define a variable male that contains the male heights.
    Define a variable female that contains the female heights.
    Report the length of each variable.
## [1] 812
## [1] 238
  1. Percentiles
    Suppose we can’t make a plot and want to compare the distributions side by side. We can’t just list all the numbers. Instead, we will look at the percentiles. Create a five row table showing female_percentiles and male_percentiles with the 10th, 30th, 50th, ., 90th percentiles for each sex. Then create a data frame with these two as columns.
  1. Interpretating Boxplots - 1
    Study the boxplots summarizing the distributions of populations sizes by country.

Which continent has the country with the largest population size?
Interpretating Boxplots-1

A. Africa

B. Americas

C. Asia

D. Europe

E. Oceania

  1. Interpretating Boxplots - 2
    Study the boxplots summarizing the distributions of populations sizes by country.

Which continent has median country with the largest population?
Interpretating Boxplots-1

A. Africa

B. Americas

C. Asia

D. Europe

E. Oceania

  1. Interpretating Boxplots - 3
    Again, look at the boxplots summarizing the distributions of populations sizes by country. To the nearest million, what is the median population size for Africa?
    Interpretating Boxplots-1

A. 100 million

B. 25 million

C. 10 million

D. 5 million

E. 1 million

  1. Low quantiles
    Examine the following boxplots and report approximately what proportion of countries in Europe have populations below 14 million:
    Interpretating Boxplots-1

A. 0.75

B. 0.50

C. 0.25

D. 0.01

  1. Interquantile Range (IQR)
    Based on the boxplot, if we use a log transformation, which continent shown below has the largest interquartile range?
    Interpretating Boxplots-1

A. Africa

B. Americas

C. Asia

D. Europe

E. Oceania

Assessment 5 (Robust Summaries With Outliers)

  1. Exploring the Galton Dataset - Average and Median
    For this chapter, we will use height data collected by Francis Galton for his genetics studies. Here we just use height of the children in the dataset:

Compute the average and median of these data. Note: do not assign them to a variable.

## [1] 68.08847
## [1] 68.2
  1. Exploring the Galton Dataset - SD and MAD
    Now for the same data compute the standard deviation and the median absolute deviation (MAD).
## [1] 2.517941
## [1] 2.9652
  1. Error impact on average
    In the previous exercises we saw that the mean and median are very similar and so are the standard deviation and MAD. This is expected since the data is approximated by a normal distribution which has this propoerty.

Now suppose that suppose Galton made a mistake when entering the first value, forgetting to use the decimal point. You can imitate this error by typing:

The data now has an outlier that the normal approximation does not account for. Let’s see how this affects the average.

Report how many inches the average grow after this mistake. Specifically, report the difference between the average of the data with the mistake x_with_error and the data without the mistake x.

## [1] 0.5983836
  1. Error impact on SD
    In the previous exercise we saw how a simple mistake can result in the average of our data increasing more than half a foot, which is a large difference in practical terms. Now let’s explore the effect this outlier has on the standard deviation.

Report how many inches the SD grows after this mistake. Specifically, report the difference between the SD of the data with the mistake x_with_error and the data without the mistake x.

## [1] 15.6746
  1. Error impact on median
    In the previous exercises we saw how one mistake can have a substantial effect on the average and the standard deviation.

Now we are going to see how the median and MAD are much more resistant to outliers. For this reason we say that they are robust summaries.

Report how many inches the median grows after the mistake. Specifically, report the difference between the median of the data with the mistake x_with_error and the data without the mistake x.

## [1] 0
  1. Error impact on MAD
    We saw that the median barely changes. Now let’s see how the MAD is affected.

Report how many inches the MAD grows after the mistake. Specifically, report the difference between the MAD of the data with the mistake x_with_error and the data without the mistake x.

## [1] 0
  1. Usefulness of EDA
    How could you use exploratory data analysis to detect that an error was made?

A. Since it is only one value out of many, we will not be able to detect this.

B. We would see an obvious shift in the distribution.

C. A boxplot, histogram, or qq-plot would reveal a clear outlier.

D. A scatter plot would show high levels of measurement error.

  1. Using EDA to explore changes We have seen how the average can be affected by outliers. But how large can this effect get? This of course depends on the size of the outlier and the size of the dataset.

To see how outliers can affect the average of a dataset, let’s write a simple function that takes the size of the outlier as input and returns the average.

Write a function called error_avg that takes a value k and returns the average of the vector x after the first entry changed to k. Show the results for k=10000 and k=-10000.

## [1] 78.79784
## [1] 57.24612

Section 2 Overview

n Section 2, you will learn how to create data visualizations in R using ggplot2.

After completing Section 2, you will:

  • be able to use ggplot2 to create data visualizations in R.
  • be able to explain what the data component of a graph is.
  • be able to identify the geometry component of a graph and know when to use which type of geometry.
  • be able to explain what the aesthetic mapping component of a graph is.
  • be able to understand the scale component of a graph and select an appropriate scale component to use.

The textbook for this section is available here

Assessment 6 (ggplot2)

Start by loading the dplyr and ggplot2 library as well as the murders and heights data.

  1. ggplot2 basics
    With ggplot2 plots can be saved as objects. For example we can associate a dataset with a plot object like this

Because data is the first argument we don’t need to spell it out

or, if we load dplyr, we can also use the pipe:

Remember the pipe sends the object on the left of %>% to be the first argument for the function the right of %>%.
What is class of the object p?

## [1] "gg"     "ggplot"
  1. printing
    Remember that to print an object you can use the command print or simply type the object. For example
## [1] 2
## [1] 2

Print the object p defined in exercise one and describe what you see.

A. Nothing happens.

B. A blank slate plot.

C. A scatter plot.

D. A histogram.

  1. Pipes
    Using the pipe %>%, create an object p but this time associated with the heights dataset instead of the murders dataset.
## [1] "gg"     "ggplot"
  1. Layers
    Now we are going to add a layers and the corresponding aesthetic mappings. For the murders data we plotted total murders versus population sizes. Explore the murders data frame to remind yourself what are the names for these two variables and select the correct answer. Hint: Look at ?murders.

A. state and abb.

B. total_murers and population_size.

C. total and population.

D. murders and size.

  1. geom_point 1
    To create the scatter plot we add a layer with geom_point. The aesthetic mappings require us to define the x-axis and y-axis variables respectively. So the code looks like this:

murders %>% ggplot(aes(x = , y = )) + geom_point()

except we have to define the two variables x and y. Fill this out with the correct variable names.

  1. geom_point 1
    Note that if we don’t use argument names, we can obtain the same plot by making sure we enter the variable names in the right order like this:

Remake the plot but now with total in the x-axis and population in the y-axis.

  1. geom_point text
    If instead of points we want to add text, we can use the geom_text() or geom_label() geometries. The following code

murders %>% ggplot(aes(population, total)) + geom_label()

will give us the error message: Error: geom_label requires the following missing aesthetics: label

Why is this?

A. We need to map a character to each point through the label argument in aes.

B. We need to let geom_label know what character to use in the plot.

C. The geom_label geometry does not require x-axis and y-axis values.

D. geom_label is not a ggplot2 command.

  1. Rewrite the code from the previous exercise to add the state abbreviation as the label through aes.

  1. geom_point colors
    Change the color of the labels through blue. How will we do this?

A. Adding a column called blue to murders

B. Because each label needs a different color we map the colors through aes

C. Use the color argument in ggplot

D. Because we want all colors to be blue, we do not need to map colors, just use the color argument in geom_label

  1. Rewrite the code above to make the labels blue.

  1. geom_labels by region
    Now suppose we want to use color to represent the different regions. In this case which of the following is most appropriate:

A. Adding a column called color to murders with the color we want to use

B. Mapping the colors through the color argument of aes because each label needs a different color

C. Using the color argument in ggplot

D. Using the color argument in geom_label because we want all colors to be blue so we do not need to map colors

  1. geom_label colors
    Rewrite the code above to make the label color correspond to the state’s region.

  1. Log-scale
    Now we are going to change the x-axis to a log scale to account for the fact the distribution of population is skewed. Let’s start by define an object p holding the plot we have made up to now

To change the y-axis to a log scale we learned about the scale_x_log10() function. Add this layer to the object p to change the scale and render the plot.

  1. Titles
    Now edit the code above to add the title “Gun murder data” to the plot. Hint: use the ggtitle function.

  1. Histograms
    We are going to shift our focus from the murders dataset to explore the heights dataset.

We use the geom_histogram function to make a histogram of the heights in the heights data frame. When reading the documentation for this function we see that it requires just one mapping, the values to be used for the histogram.

What is the variable containing the heights in inches in the heights data frame?

A. sex

B. heights

C. height

D. heights$height

  1. A second example
    We are now going to make a histogram of the heights so we will load the heights dataset. The following code has been pre-run for you to load the heights dataset:

Create a ggplot object called p using the pipe to assign the heights data to a ggplot object. Assign height to the x values through the aes function.

  1. Histograms 2
    Now we are ready to add a layer to actually make the histogram.

Add a layer to the object p (created in the previous exercise) using the geom_histogram function to make the histogram.

18. Histogram binwidth
Note that when we run the code from the previous exercise we get the following warning:

stat_bin() using bins = 30. Pick better value with binwidth.

Use the binwidth argument to change the histogram made in the previous exercise to use bins of size 1 inch.

  1. Smooth density plot Now instead of a histogram we are going to make a smooth density plot. In this case, we will not make an object p. Instead we will render the plot using a single line of code. In the previous exercise, we could have created a histogram using one line of code like this:

Now instead of geom_histogram we will use geom_density to create a smooth density plot.
Add the appropriate layer to create a smooth density plot of heights.

  1. Two smooth density plots
    Now we are going to make density plots for males and females separately. We can do this using the group argument within the aes mapping. Because each point will be assigned to a different density depending on a variable from the dataset, we need to map within aes.
    Create separte smooth density plots for males and females by defining group by sex.

  1. Two smooth density plots 2
    In the previous exercise we made the two density plots, one for each sex, using:

We can also assign groups through the color or fill argument. For example, if you type color = sex ggplot knows you want a different color for each sex. So two densities must be drawn. You can therefore skip the group = sex mapping. Using color has the added benefit that it uses color to distinguish the groups. Change the density plots from the previous exercise to add color.

  1. Two smooth density plots 3
    We can also assign groups using the fill argument. When using the geom_density geometry, color creates a colored line for the smooth density plot while fill colors in the area under the curve.

We can see what this looks like by running the following code:

However, here the second density is drawn over the other. We can change this by using something called alpha blending. Set the alpha parameter to 0.2 in the geom_density function to make this change.

Section 3 Overview

Section 3 introduces you to summarizing with dplyr.

After completing Section 3, you will:

  • understand the importance of summarizing data in exploratory data analysis.
  • be able to use the “summarize” verb in dplyr to facilitate summarizing data.
  • be able to use the “group_by” verb in dplyr to facilitate summarizing data.
  • be able to access values using the dot placeholder.
  • be able to use “arrange” to examine data after sorting.

The textbook for this section is available here

Assessment 7 (Summarizing with dplyr)

Practice Exercise. National Center for Health Statistics
To practice our dplyr skills we will be working with data from the survey collected by the United States National Center for Health Statistics (NCHS). This center has conducted a series of health and nutrition surveys since the 1960’s.

Starting in 1999, about 5,000 individuals of all ages have been interviewed every year and then they complete the health examination component of the survey. Part of this dataset is made available via the NHANES package which can be loaded this way:

The NHANES data has many missing values. Remember that the main summarization function in R will return NA if any of the entries of the input vector is an NA. Here is an example:

## [1] NA
## [1] NA

To ignore the NAs, we can use the na.rm argument:

## [1] 2.301754
## [1] 1.22338
  1. Blood pressure 1
    Let’s explore the NHANES data. We will be exploring blood pressure in this dataset.

First let’s select a group to set the standard. We will use 20-29 year old females. Note that the category is coded with 20-29, with a space in front of the 20! The AgeDecade is a categorical variable with these ages.

To know if someone is female, you can look at the Gender variable.

Filter the NHANES dataset so that only 20-29 year old females are included and assign this new data frame to the object tab.
Use the pipe to apply the function filter, with the appropriate logicals, to NHANES.
Remember that this age group is coded with 20-29, which includes a space. You can use head to explore the NHANES table to construct the correct call to filter.

  1. Blood pressure 2
    Now we will compute the average and standard deviation for the subgroup we defined in the previous exercise (20-29 year old females), which we will use reference for what is typical.

You will determine the average and standard deviation of systolic blood pressure, which are stored in the BPSysAve variable in the NHANES dataset.

Complete the line of code to save the average and standard deviation of systolic blood pressure as average and standard_deviation to a variable called ref.
Use the summarize function after filtering for 20-29 year old females and connect the results using the pipe %>%. When doing this remember there are NAs in the data!

  1. Summarizing averages
    Now we will repeat the exercise and generate only the average blood pressure for 20-29 year old females. For this exercise, you should review how to use the place holder . in dplyr.

Modify the line of sample code to assign the average to a numeric variable called ref_avg.

  1. Min and max
    Let’s continue practicing by calculating two other data summaries: the minimum and the maximum.

Again we will do it for the BPSysAve variable and the group of 20-29 year old females.

Report the min and max values for the same group as in the previous exercises.
Use filter and summarize connected by the pipe %>% again. The functions min and max can be used to get the values you want.
Within summarize, save the min and max of systolic blood pressure as min and max.

  1. group_by
    Now let’s practice using the group_by function.

What we are about to do is a very common operation in data science: you will split a data table into groups and then compute summary statistics for each group.

We will compute the average and standard deviation of systolic blood pressure for females for each age group separately. Remember that the age groups are contained in AgeDecade.

Use the functions filter, group_by, summarize, and the pipe %>% to compute the average and standard deviation of systolic blood pressure for females for each age group separately.

Within summarize, save the average and standard deviation of systolic blood pressure (BPSysAve) as average and standard_deviation.

  1. group_by example 2
    Now let’s practice using group_by some more. We are going to repeat the previous exercise of calculating the average and standard deviation of systolic blood pressure, but for males instead of females.

This time we will not provide much sample code. You are on your own!

Calculate the average and standard deviation of systolic blood pressure for males for each age group separately using the same methods as in the previous exercise.

  1. group_by example 3
    We can actually combine both of these summaries into a single line of code. This is because group_by permits us to group by more than one variable.

We can use group_by(AgeDecade, Gender) to group by both age decades and gender.

Create a single summary table for the average and standard deviation of systolic blood pressure using group_by(AgeDecade, Gender).
Note that we no longer have to filter!
Your code within summarize should remain the same as in the previous exercises.

  1. Arrange
    Now we are going to explore differences in systolic blood pressure across races, as reported in the Race1 variable.

We will learn to use the arrange function to order the outcome acording to one variable.

Note that this function can be used to order any table by a given outcome. Here is an example that arranges by systolic blood pressure.

If we want it in descending order we can use the desc function like this:

In this example, we will compare systolic blood pressure across values of the Race1 variable for males between the ages of 40-49.

Compute the average and standard deviation for each value of Race1 for males in the age decade 40-49.
Order the resulting table from lowest to highest average systolic blood pressure.
Use the functions filter, group_by, summarize, arrange, and the pipe %>% to do this in one line of code.
Within summarize, save the average and standard deviation of systolic blood pressure as average and standard_deviation.

Section 4 Overview

n Section 4, you will look at a case study involving data from the Gapminder Foundation about trends in world health and economics.

After completing Section 4, you will:

  • understand how Hans Rosling and the Gapminder Foundation use effective data visualization to convey data-based trends.
  • be able to apply the ggplot2 techniques from the previous section to answer questions using data.
  • understand how fixed scales across plots can ease comparisons.
  • be able to modify graphs to improve data visualization.

The textbook for this section is available here

Assessment 8 (Exploring the Gapminder Dataset)

  1. Life expectancy vs fertility - part 1
    The Gapminder Foundation (www.gapminder.org) is a non-profit organization based in Sweden that promotes global development through the use of statistics that can help reduce misconceptions about global development.
  • Using ggplot and the points layer, create a scatter plot of life expectancy versus fertility for the African continent in 2012.
  • Remember that you can use the R console to explore the gapminder dataset to figure out the names of the columns in the dataframe.
  • In this exercise we provide parts of code to get you going. You need to fill out what is missing. But note that going forward, in the next exercises, you will be required to write most of the code.

  1. Life expectancy vs fertility - part 2 - coloring your plot
    Note that there is quite a bit of variability in life expectancy and fertility with some African countries having very high life expectancies. There also appear to be three clusters in the plot.

Remake the plot from the previous exercises but this time use color to dinstinguish the different regions of Africa to see if this explains the clusters. Remember that you can explore the gapminder data to see how the regions of Africa are labeled in the dataframe!

  1. Life expectancy vs fertility - part 3 - selecting country and region
    While many of the countries in the high life expectancy/low fertility cluster are from Northern Africa, three countries are not.
  • Create a table showing the country and region for the African countries (use select) that in 2012 had fertility rates of 3 or less and life expectancies of at least 70.
  • Assign your result to a data frame called df.
  1. Life expectancy and the Vietnam War - part 1
    The Vietnam War lasted from 1955 to 1975. Do the data support war having a negative effect on life expectancy? We will create a time series plot that covers the period from 1960 to 2010 of life expectancy for Vietnam and the United States, using color to distinguish the two countries. In this start we start the analysis by generating a table.
  • Use filter to create a table with data for the years from 1960 to 2010 in Vietnam and the United States.
  • Save the table in an object called tab.
  1. Life expectancy and the Vietnam War - part 2
    Now that you have created the data table in Exercise 4, it is time to plot the data for the two countries.
  • Use geom_line to plot life expectancy vs year for Vietnam and the United States. The data table is stored in tab.
  • Use color to distinguish the two countries.

  1. Life expectancy in Cambodia
    Cambodia was also involved in this conflict and, after the war, Pol Pot and his communist Khmer Rouge took control and ruled Cambodia from 1975 to 1979. He is considered one of the most brutal dictators in history. Do the data support this claim?

Use a single line of code to create a time series plot from 1960 to 2010 of life expectancy vs year for Cambodia.

  1. Dollars per day - part 1
    Now we are going to calculate and plot dollars per day for African countries in 2010 using GDP data.

In the first part of this analysis, we will create the dollars per day variable.

  • Use mutate to create a dollars_per_day variable, which is defined as gdp/population/365.
  • Create the dollars_per_day variable for African countries for the year 2010.
  • Remove any NA values.
  • Save the mutated dataset as daydollars.
  1. Dollars per day - part 2
    Now we are going to calculate and plot dollars per day for African countries in 2010 using GDP data.

In the second part of this analysis, we will plot the smooth density plot using a log (base 2) x axis.

  • The dataset including the dollars_per_day variable is preloaded as daydollars.
  • Create a smooth density plot of dollars per day from daydollars.
  • Use a log (base 2) scale for the x axis.

  1. Dollars per day - part 3 - multiple density plots
    Now we are going to combine the plotting tools we have used in the past two exercises to create density plots for multiple years.
  • Create the dollars_per_day variable as in Exercise 7, but for African countries in the years 1970 and 2010 this time.
  • Make sure you remove any NA values.
  • Create a smooth density plot of dollars per day for 1970 and 2010 using a log (base 2) scale for the x axis.
  • Use facet_grid to show a different density plot for 1970 and 2010.

  1. Dollars per day - part 4 - stacked histograms
    Now we are going to edit the code from Exercise 9 to show stacked histograms of each region in Africa.

Much of the code will be the same as in Exercise 9:

  • Create the dollars_per_day variable as in Exercise 7, but for African countries in the years 1970 and 2010 this time.
  • Make sure you remove any NA values.
  • Create a smooth density plot of dollars per day for 1970 and 2010 using a log (base 2) scale for the x axis.
  • Use facet_grid to show a different density plot for 1970 and 2010.
  • Make sure the densities are smooth by using bw = 0.5.
  • Use the fill and position arguments where appropriate to create the stacked histograms of each region.

  1. Infant mortality scatter plot - part 1
    We are going to continue looking at patterns in the gapminder dataset by plotting infant mortality rates versus dollars per day for African countries.
  • Generate dollars_per_day using mutate and filter for the year 2010 for African countries.
  • Remember to remove NA values.
  • Store the mutated dataset in gapminder_Africa_2010.
  • Make a scatter plot of infant_mortaility versus dollars_per_day for countries in the African continent.
  • Use color to denote the different regions of Africa.

  1. Infant mortality scatter plot - part 2 - logarithmic axis
    Now we are going to transform the x axis of the plot from the previous exercise.
  • The mutated dataset is preloaded as gapminder_Africa_2010.
  • As in the previous exercise, make a scatter plot of infant_mortaility versus dollars_per_day for countries in the African continent.
  • As in the previous exercise, use color to denote the different regions of Africa.
  • Transform the x axis to be in the log (base 2) scale.

  1. Infant mortality scatter plot - part 3 - adding labels
    Note that there is a large variation in infant mortality and dollars per day among African countries.

As an example, one country has infant mortality rates of less than 20 per 1000 and dollars per day of 16, while another country has infant mortality rates over 10% and dollars per day of about 1.

In this exercise, we will remake the plot from Exercise 12 with country names instead of points so we can identify which countries are which.

  • The mutated dataset is preloaded as gapminder_Africa_2010.
  • As in the previous exercise, make a scatter plot of infant_mortaility versus dollars_per_day for countries in the African continent.
  • As in the previous exercise, use color to denote the different regions of Africa.
  • As in the previous exercise, transform the x axis to be in the log (base 2) scale.
  • Add a layer to display country names instead of points.

  1. Infant mortality scatter plot - part 4 - comparison of scatter plots
    Now we are going to look at changes in the infant mortality and dollars per day patterns African countries between 1970 and 2010.
  • Generate dollars_per_day using mutate and filter for the years 1970 and 2010 for African countries.
  • Remember to remove NA values.
  • As in the previous exercise, make a scatter plot of infant_mortaility versus dollars_per_day for countries in the African continent.
  • As in the previous exercise, use color to denote the different regions of Africa.
  • As in the previous exercise, transform the x axis to be in the log (base 2) scale.
  • As in the previous exercise, add a layer to display country names instead of points.
  • Use facet_grid to show different plots for 1970 and 2010.

Section 5 Overview

Section 5 covers some general principles that can serve as guides for effective data visualization.

After completing Section 5, you will:

  • understand basic principles of effective data visualization.
  • understand the importance of keeping your goal in mind when deciding on a visualization approach.
  • understand principles for encoding data, including position, aligned lengths, angles, area, brightness, and color hue.
  • know when to include the number zero in visualizations.
  • be able to use techniques to ease comparisons, such as using common axes, putting visual cues to be compared adjacent to one another, and using color effectively.

The textbook for this section is available here

Assessment 9 (Data Visualization Principles, Part 1)

1: Customizing plots - Pie charts
Pie charts are appropriate:

A. When we want to display percentages.

B. When ggplot2 is not available.

C. When I am in a bakery.

D. Never. Barplots and tables are always better.

  1. Customizing plots - What’s wrong?
    What is the problem with this plot?
    distribution

A. The values are wrong. The final vote was 306 to 232.

B. The axis does not start at 0. Judging by the length, it appears Trump received 3 times as many votes when in fact it was about 30% more.

C. The colors should be the same.

D. Percentages should be shown as a pie chart.

3: Customizing plots - What’s wrong 2?.
Take a look at the following two plots. They show the same information: rates of measles by state in the United States for 1928.
distribution

A. Both plots provide the same information, so they are equally good.

B. The plot on the left is better because it orders the states alphabetically.

C. The plot on the right is better because it orders the states by disease rate so we can quickly see the states with highest and lowest rates.

D. Both plots should be pie charts instead.

Assessment 10 (Data Visualization Principles, Part 2)

1: Customizing plots - watch and learn
To make the plot on the right in the exercise from the last set of assessments, we had to reorder the levels of the states’ variables.

  • Redefine the state object so that the levels are re-ordered by rate.
  • Print the new object state and its levels so you can see that the vector is now re-ordered by the levels.
##  [1] Alabama              Alaska               Arizona             
##  [4] Arkansas             California           Colorado            
##  [7] Connecticut          Delaware             District Of Columbia
## [10] Florida              Georgia              Hawaii              
## [13] Idaho                Illinois             Indiana             
## [16] Iowa                 Kansas               Kentucky            
## [19] Louisiana            Maine                Maryland            
## [22] Massachusetts        Michigan             Minnesota           
## [25] Mississippi          Missouri             Montana             
## [28] Nebraska             Nevada               New Hampshire       
## [31] New Jersey           New Mexico           New York            
## [34] North Carolina       North Dakota         Ohio                
## [37] Oklahoma             Oregon               Pennsylvania        
## [40] Rhode Island         South Carolina       South Dakota        
## [43] Tennessee            Texas                Utah                
## [46] Vermont              Virginia             Washington          
## [49] West Virginia        Wisconsin            Wyoming             
## attr(,"scores")
##              Alabama               Alaska              Arizona 
##           4.16107582           5.46389893           6.32695891 
##             Arkansas           California             Colorado 
##           6.87899954           2.79313560           7.96331905 
##          Connecticut             Delaware District Of Columbia 
##           0.36986840           1.13098183           0.35873614 
##              Florida              Georgia               Hawaii 
##           2.89358806           0.09987991           2.50173748 
##                Idaho             Illinois              Indiana 
##           6.03115170           1.20115480           1.34027323 
##                 Iowa               Kansas             Kentucky 
##           2.94948911           0.66386422           4.74576011 
##            Louisiana                Maine             Maryland 
##           0.46088071           2.57520433           0.49922233 
##        Massachusetts             Michigan            Minnesota 
##           0.74762338           1.33466700           0.37722410 
##          Mississippi             Missouri              Montana 
##           3.11366532           0.75696354           5.00433320 
##             Nebraska               Nevada        New Hampshire 
##           3.64389801           6.43683882           0.47181511 
##           New Jersey           New Mexico             New York 
##           0.88414264           6.15969926           0.66849058 
##       North Carolina         North Dakota                 Ohio 
##           1.92529764          14.48024642           1.16382241 
##             Oklahoma               Oregon         Pennsylvania 
##           3.27496900           8.75036439           0.67687303 
##         Rhode Island       South Carolina         South Dakota 
##           0.68207448           2.10412531           0.90289534 
##            Tennessee                Texas                 Utah 
##           5.47344506          12.49773953           4.03005836 
##              Vermont             Virginia           Washington 
##           1.00970314           5.28270939          17.65180349 
##        West Virginia            Wisconsin              Wyoming 
##           8.59456463           4.96246019           6.97303449 
## 51 Levels: Georgia District Of Columbia Connecticut ... Washington
##  [1] "Georgia"              "District Of Columbia" "Connecticut"         
##  [4] "Minnesota"            "Louisiana"            "New Hampshire"       
##  [7] "Maryland"             "Kansas"               "New York"            
## [10] "Pennsylvania"         "Rhode Island"         "Massachusetts"       
## [13] "Missouri"             "New Jersey"           "South Dakota"        
## [16] "Vermont"              "Delaware"             "Ohio"                
## [19] "Illinois"             "Michigan"             "Indiana"             
## [22] "North Carolina"       "South Carolina"       "Hawaii"              
## [25] "Maine"                "California"           "Florida"             
## [28] "Iowa"                 "Mississippi"          "Oklahoma"            
## [31] "Nebraska"             "Utah"                 "Alabama"             
## [34] "Kentucky"             "Wisconsin"            "Montana"             
## [37] "Virginia"             "Alaska"               "Tennessee"           
## [40] "Idaho"                "New Mexico"           "Arizona"             
## [43] "Nevada"               "Arkansas"             "Wyoming"             
## [46] "Colorado"             "West Virginia"        "Oregon"              
## [49] "Texas"                "North Dakota"         "Washington"

2: Customizing plots - redefining
Now we are going to customize this plot a little more by creating a rate variable and reordering by that variable instead.

  • Add a single line of code to the definition of the dat table that uses mutate to reorder the states by the rate variable.
  • The sample code provided will then create a bar plot using the newly defined dat.

3: Showing the data and customizing plots
Say we are interested in comparing gun homicide rates across regions of the US. We see this plot:

and decide to move to a state in the western region. What is the main problem with this interpretaion?

A. The categories are ordered alphabetically.

B. The graph does not show standard errors.

C. It does not show all the data. We do not see the variability within a region and it's possible that the safest states are not in the West.

D. The Northeast has the lowest average.

4: Making a box plot
To further investigate whether moving to the western region is a wise decision, let’s make a box plot of murder rates by region, showing all points.

  • Make a box plot of the murder rates by region.
  • Order the regions by their median murder rate.
  • Show all of the points on the box plot.

Assessment 11 (Data Visualization Principles, Part 3)

  1. Tile plot - measles and smallpox
    The sample code given creates a tile plot showing the rate of measles cases per population. We are going to modify the tile plot to look at smallpox cases instead.

  • Modify the tile plot to show the rate of smallpox cases instead of measles cases.
  • Exclude years in which cases were reported in fewer than 10 weeks from the plot.

  1. Time series plot - measles and smallpox
    The sample code given creates a time series plot showing the rate of measles cases per population by state. We are going to again modify this plot to look at smallpox cases instead.
## 'data.frame':    3724 obs. of  7 variables:
##  $ disease        : Factor w/ 7 levels "Hepatitis A",..: 2 2 2 2 2 2 2 2 2 2 ...
##  $ state          : Factor w/ 51 levels "Mississippi",..: 9 9 9 9 9 9 9 9 9 9 ...
##   ..- attr(*, "scores")= num [1:51(1d)] 9.27 NA 24.15 9.37 19.16 ...
##   .. ..- attr(*, "dimnames")=List of 1
##   .. .. ..$ : chr  "Alabama" "Alaska" "Arizona" "Arkansas" ...
##  $ year           : num  1928 1929 1930 1931 1932 ...
##  $ weeks_reporting: int  52 49 52 49 41 51 52 49 40 49 ...
##  $ count          : num  8843 2959 4156 8934 270 ...
##  $ population     : num  2589923 2619131 2646248 2670818 2693027 ...
##  $ rate           : num  34.1 11.3 15.7 33.5 1 ...

  • Modify the sample code for the time series plot to plot data for smallpox instead of for measles.
  • Once again, restrict the plot to years in which cases were reported in at least 10 weeks.
## 'data.frame':    1014 obs. of  7 variables:
##  $ disease        : Factor w/ 7 levels "Hepatitis A",..: 7 7 7 7 7 7 7 7 7 7 ...
##  $ state          : Factor w/ 51 levels "Rhode Island",..: 17 17 17 17 17 17 17 17 17 17 ...
##   ..- attr(*, "scores")= num [1:51(1d)] 0.382 NA 2.011 0.805 0.924 ...
##   .. ..- attr(*, "dimnames")=List of 1
##   .. .. ..$ : chr  "Alabama" "Alaska" "Arizona" "Arkansas" ...
##  $ year           : num  1928 1929 1930 1931 1932 ...
##  $ weeks_reporting: int  51 52 52 52 52 52 52 52 51 52 ...
##  $ count          : num  341 378 192 295 467 82 23 42 12 54 ...
##  $ population     : num  2589923 2619131 2646248 2670818 2693027 ...
##  $ rate           : num  1.317 1.443 0.726 1.105 1.734 ...

  1. Time series plot - all diseases in California
    Now we are going to look at the rates of all diseases in one state. Again, you will be modifying the sample code to produce the desired plot.
  • For the state of California, make a time series plot showing rates for all diseases.
  • Include only years with 10 or more weeks reporting.
  • Use a different color for each disease.

  1. Time series plot - all diseases in the United States
    Now we are going to make a time series plot for the rates of all diseases in the United States. For this exercise, we have provided less sample code - you can take a look at the previous exercise to get you started.
  • Compute the US rate by using summarize to sum over states.
  • The US rate for each disease will be the total number of cases divided by the total population.
  • Remember to convert to cases per 10,000.
  • You will need to filter for !is.na(population) to get all the data.
  • Plot each disease in a different color.