Introduction


In this class, we continue to learn how to visualize data with the ggplot2 package in R. We will learn the following topics

  1. How to create a histogram and/or a density plot
  2. How to customize histograms and density plots
  3. Use aesthetic grouping to explore the effect of an additional categorical variable
  4. Save your figure to disk
  5. Control the appearance of data-related components


library(tidyverse)

Lending Club data set


In this class, we will use the data set loans_full_schema to show examples. The data set is from the package openintro.

library(openintro) # Install the package if it's not available 


If you don’t have the package installed, use install.packages("openintro") to install it first. After loading the package, we can take a look at the data.

glimpse(loans_full_schema)


Question: How many samples are there? How many variables are there?

Histogram


To summarize a single numeric variable, the most commonly used chart is a histogram. As below is an example


A good histogram shows the distribution shape of the data set.

Selected variables


There are too many variables in the data set. To make things simpler, we will only handle 8 of them for now by executing the following code in R.

loans <- select(loans_full_schema, loan_amount, interest_rate, term, 
         grade, state, annual_income, homeownership, debt_to_income)


This operation of selecting some variables from the original data set belongs to data transformation, which will be our topic in the next chapter.


Now we create a new data set named loans that only stores the selected variables (loan_amount, interest_rate etc.) from the original set.

Glimpse of the data


Question: What is the data set about?

glimpse(loans)
## Rows: 10,000
## Columns: 8
## $ loan_amount    <int> 28000, 5000, 2000, 21600, 23000, 5000, 24000, 20000, 20…
## $ interest_rate  <dbl> 14.07, 12.61, 17.09, 6.72, 14.07, 6.72, 13.59, 11.99, 1…
## $ term           <dbl> 60, 36, 36, 36, 36, 36, 60, 60, 36, 36, 60, 60, 36, 60,…
## $ grade          <fct> C, C, D, A, C, A, C, B, C, A, C, B, C, B, D, D, D, F, E…
## $ state          <fct> NJ, HI, WI, PA, CA, KY, MI, AZ, NV, IL, IL, FL, SC, CO,…
## $ annual_income  <dbl> 90000, 40000, 40000, 30000, 35000, 34000, 35000, 110000…
## $ homeownership  <fct> MORTGAGE, RENT, RENT, RENT, RENT, OWN, MORTGAGE, MORTGA…
## $ debt_to_income <dbl> 18.01, 5.04, 21.15, 10.16, 57.96, 6.46, 23.66, 16.19, 3…

Meaning of variables


variable description
loan_amount Amount of the loan received, in US dollars
interest_rate Interest rate on the loan, in an annual percentage
term The length of the loan, which is always set as a whole number of months
grade Loan grade, which takes a values A through G and represents the quality of the loan and its likelihood of being repaid
state US state where the borrower resides
annual_income Borrower’s annual income, including any second income, in US dollars
homeownership Indicates whether the person owns, owns but has a mortgage, or rents
debt_to_income Debt-to-income ratio, in percentage

Variable types


<fct> means factor, which is a data type in R used to store and process categorical data.

variable type
loan_amount numerical, continuous
interest_rate numerical, continuous
term numerical, discrete
grade categorical, ordinal
state categorical, not ordinal
annual_income numerical, continuous
homeownership categorical, not ordinal
debt_to_income numerical, continuous

table and unique function


Oftentimes, we hope to quickly check all values for a categorical or discrete variable. There are two functions to fulfill the job:

unique(loans$term)
## [1] 60 36

The unique function returns all unique values for a vector. As we can see here, for all the loans in the data set, the term length is either 36 or 60 months.

An even more powerful function is the table function, which creates a frequency table for any given categorical or discrete variable.

table(loans$term)
## 
##   36   60 
## 6970 3030

As we see, the function not only lists all values of the variable, but also lists the counts (frequency) for each value.

Lab Exercise


Answer the following questions by using unique or table function:

  1. How many distinct values are there for homeownership variable? Which value is the most common one?

  2. How many distinct interest rates are there? Which value is the most common one?

  3. Apply table function to the annual_income variable. Do you think the result is helpful or not?

Create a histogram


Now let’s learn how to summarize a numeric variable by creating a histogram. Let’s pick the annual_income since its meaning is easy to understand.

ggplot(data = loans) +
  geom_histogram(mapping = aes(x = interest_rate))

By default, there are 30 bins and their ranges are automatically determined. In many cases, this won’t give us a satisfactory graph. We need to customize it.

Changing the bin number - bins=20


In the previous graph, around 10% there are two “missing” bars which look unnatural. We can resolve this by reducing the number of bins (groups).

ggplot(data = loans) +
  geom_histogram(mapping = aes(x = interest_rate), bins = 20) 

Changing the bin number - bins=10


We can make the bin number even smaller to be 10. Among the three histograms, Which one do you think is the best? Why?

ggplot(data = loans) +
  geom_histogram(mapping = aes(x = interest_rate), bins = 10) 

Change the bin width


In some situations, we hope to specify the bin width instead of bin number. We may do the following:

ggplot(data = loans) +
  geom_histogram(mapping = aes(x = interest_rate), binwidth = 5) 

Adjust the position of bins


We can specify the center or boundary of one bin to make adjustments to the position of bins.

ggplot(data = loans) +
  geom_histogram(mapping = aes(x = interest_rate), binwidth = 5, boundary = 10) 

Adjust the position of bins


We can specify the center or boundary of one bin to make adjustments to the position of bins.

ggplot(data = loans) +
  geom_histogram(mapping = aes(x = interest_rate), binwidth = 1, center = 10) 

Adjust the plotting range of x


It may look a little unnatural if we don’t plot x starting from zero. We may fix it by using the function xlim.

ggplot(data = loans) +
  geom_histogram(mapping = aes(x = interest_rate), binwidth = 1, center = 10) +
  xlim(0, 40)

Questions on data


  • Why is the lowest interest rate 5%-ish and there was no lower interest rate? Can you explain?

  • Why are there some peak interest rates around 7%, 10%, 14%? Can you explain?

Lab Exercise


  • Create a histogram of loan_amount. Customize your plot to give a graph that looks most reasonable to you.

  • Create a histogram of annual_income. What is the issue with your graph?

Density plot


For continuous random variables, it is common practice to have a density plot which plots a smoothed version of the histogram.


The density here is same as “probability density” in statistics. The density plot can be understood as a probability density function fit to the histogram. As below is an example.

ggplot(loans, aes(x = loan_amount)) +
  geom_density()


Here geom_density() function creates a density plot:

  • The total area under the density plot is one.

  • By default, the density curve is fit to the histogram with the default bin number (30 bins, see the graph below).

  • To adjust the “smoothness” of the plot, change the variable adjust.

Plot histogram and density together


Usually, it can be a good idea to plot histogram and density in one plot:

ggplot(loans, aes(x = loan_amount)) +
  geom_histogram(aes(y = after_stat(density)),  
                 boundary = 0, colour = "black", fill = "white") +
  geom_density(linewidth = 1.2)


Note that to make the histogram also plotted with y-axis being density, we need to add y = after_stat(density) in the aes function.

Adjust density smoothness


Larger adjust value gives more smooth density curves. For example, we can set it to be 30/8 to fit the curve to a histogram with 8 bins (which is smoother).

ggplot(loans, aes(x = loan_amount)) +
  geom_histogram(aes(y = after_stat(density)), binwidth = 5000, 
                 boundary = 0, colour = "black", fill = "white") +
  geom_density(adjust = 30/8, linewidth = 1.2) #30/8 = 30bins/8bins

Create a relative frequency histogram (optional)


The after_stat function is used in the example above because the y-axis is mapped to data that are not a variable in the original data frame. For histograms, the default y-axis is mapped to counts, which is computed by another function stat_bin(). The function after_stat(count) or after_stat(density) indicates that count or density are only available after the original data are transformed.

With this understanding, we can create a relative frequency histogram (although rarely used) where the y-axis is the proportion of samples in each bin rather than the counts:

ggplot(loans) + 
  geom_histogram(mapping = aes(x = loan_amount, y = after_stat(count/sum(count))), 
                 binwidth = 5000, boundary = 0, colour = "black", fill = "white")

Lab Exercise


Create a histogram of variable debt_to_income in loans with the following requirements:

  1. The plotting range of x is between 0 and 100
  2. The binwidth is 2
  3. Create a density plot on top of the histogram


Question: Can you explain the distribution of debt_to_income?

Aesthetic Grouping


In many cases, it is very useful to add another level of aesthetic mapping to a figure. For example, in the mpg data set, if we plot hwy vs displ, we would see a plot like this:


Example: How would we explain the red dots which seem to deviate the overall trend (larger displ leading to lower hwy)?

The same plot with color groups in vehicle class


How to explain those data points which has good fuel economy with large engine displacement? The following plot answers the question:

ggplot(data = mpg) + 
  geom_point(mapping = aes(x = displ, y = hwy, color = class))


We see that most of these cars are 2seaters, e.g., sports cars. Those cars are lighter with a powerful engine, thus having better mpg in highway than other non-sports cars (SUV, pickups).


Here, we put a color argument inside the aes function, this creates a new aesthetic group (in color) by the categorical variable class.

Template for aesthetic grouping


Similarly, we can map other aesthetic components (shape, color, size, linetype, transparency etc.) to any categorical variable. The template to do this is:

<GEOM_FUNCTION>(mapping = aes(x = ..., y = ..., 
                color/shape/size/linetype/alpha = <VARIABLE_NAME>))


Note that the argument must be inside the aes() function.

Example - group by color and size (for ordinal variable)


ggplot(data = mpg) + 
     geom_point(mapping = aes(x = displ, y = hwy, color = cyl, size = cyl))

Question: What can we learn from this graph?

Lab Exercise


For loans data, create a scatter plot of interest_rate vs debt_to_income with mapping color to grade. What can you learn from the graph?

Aesthetic grouping for histogram


Sometimes we hope to compare histograms between groups. One way to do this is to use fill in the aes function. For example, if we want to investigate the effect of homeownership to loan_amount, we can do the following:

ggplot(loans, aes(loan_amount, fill = homeownership)) + 
  geom_histogram(binwidth = 5000, alpha = 0.5) 


The argument alpha is between 0 and 1 that controls the transparency of each histogram. The smaller alpha is, the more transparent. It is very useful when we plot multiple charts that overlap with each other.

Aesthetic grouping for density curve

When the absolute counts between groups are quite different, comparing histograms is not a good idea. Instead, we would like to compare density curves between groups to see their differences.

ggplot(loans, aes(x = loan_amount, fill = homeownership)) + 
  geom_density(adjust = 2, alpha = 0.5)   # Transparency is necessary

So this graph shows the relatively insignificant differences between groups - people with mortgage tend to borrow more money than those renting houses (maybe they care less about having more debts since they are indebted anyway).

Ridge plots for density curves


When there are too many categories, a density ridge plot can be useful.

library(ggridges)   # The package "ggridges" must be installed 
ggplot(loans, aes(x = loan_amount, y = grade, 
                  fill = grade, color = grade)) + 
  geom_density_ridges(alpha = 0.5)


This enhanced graph becomes available after installing the package ggridges. You should be able to understand the code now without my explanation after studying many examples!


Question: What can we learn from this graph?

Save your figure


Use ggsave to save a figure as a file in the current working folder.

ggplot(mpg) + geom_point(aes(cty, hwy)) 
ggsave("my-plot.pdf")


Check the current working folder to find the file you just saved on your disk. You may use the following command to know your current working folder.

getwd()


You may save your figure as pdf, png, jpeg and other compatible formats.

R Markdown Basics

R Markdown is a format for writing reproducible, dynamic reports with R. Use it to embed R code and results into slideshows, pdfs, html documents, Word files and more.

For beginners, you are recommended to check the cheatsheet at: https://rstudio.github.io/cheatsheets/html/rmarkdown.html

Knit to pdf


You need to knit your “.rmd” file to an output format. To knit to pdf, you need to install “TinyTex” package by executing the following codes (line by line):

install.packages('tinytex')
tinytex::install_tinytex()

Another way to create a pdf is to knit to html file first, then print to pdf with your browser.

Publish your report online (and make it public)


There is another convenient way to share your report with other people. After knitting your report to “html” webpage, click the “publish” button on the top right.

After creating a Rpub account, you can publish your report on the server of Rpub with a permanent link. You can share the link with other people to give them access.

Lab Homework #1


  1. Finish all Lab Exercises

  2. Create a scatter plot of loan_amount vs interest_rate with a color grouping using term variable (please use factor(term) to convert it into a categorical variable). Save your plot to your local folder.

  3. Submit your homework using R Markdown in pdf format.

Lab Homework #2


In the following, we will use an example to learn how to use R Markdown to write a report. You should follow the same manner when you finish future homework in R. Let’s write a data analysis report to answer the following questions:


Regarding mpg data set:

  1. What is the most popular fuel type in this data set?
  2. Regarding the fuel type variable, the value “d” represents diesel, “p” represents premium (petrol) and “r” represents regular (petrol). Do you think there is an effect of fuel type on how many miles a vehicle can run on average per gallon of fuel?
  3. Do you think there is a difference in fuel economy for vehicles made in 1999 and 2008? (When plotting with “year” variable, use as.factor(year) to convert it to categorical variables. This will be explained in future classes.)
  4. What happens if you make a scatter plot of class vs drv? Do you think this plot is useful or not?

Submit a pdf created from an R markdown file. Submit both your pdf and the .rmd file.