Lab 1: Intro to R

---

Loading packages and data

I am using here the special template provided wiht oilabs. It has a non-hierarchical table of content including just first and second order.

Instead of loading dplyr and ggplot2 separatly it would be better to load tidyverse. The tidyverse is a collection of R packages that share common structure and are designed to work together. See in detail the unified philsosophy by the main developer Hadley Wickham The tidy tools manifesto.

Currently tidyverse loads the following packages:

• ggplot2: Data Visualisations (Using the Grammar of Graphics)
• tibble: Simple Data Frames
• tidyr: Tidy Data with ‘spread()’ and ‘gather()’
• readr: Read Tabular Data
• purrr: Functional Programming Tools
• dplyr: A Grammar of Data Manipulation

But with the same unified principles also come the following packages:

• forcats: Tools for Working with Categorical Variables (Factors)
• haven: Import and Export ‘SPSS’, ‘Stata’ and ‘SAS’ Files
• stringr: Simple, Consistent Wrappers for Common String Operations

and maybe also (I think so, but this is not confirmed): * broom: Convert Statistical Analysis Objects into Tidy Data Frames * reshape Flexibly Reshape Data: A Reboot of the Reshape Package

For the special advantages of each packages see the adequate vignettes of these pacckages.

Note:

There is some conflict of dplyr with the stat package: filer() and lag()

Those packages above are a set of general working tools for all data science tasks and should always have priority for the older and not with a unified structure programmed basic R packages.

library(tidyverse)

## Loading tidyverse: ggplot2
## Loading tidyverse: tibble
## Loading tidyverse: tidyr
## Loading tidyverse: readr
## Loading tidyverse: purrr
## Loading tidyverse: dplyr

## Conflicts with tidy packages ----------------------------------------------

## filter(): dplyr, stats
## lag():    dplyr, stats

Dr. Arbuthnot’s Baptism Records

Note: there is also a longer lab text, which includes two extra exercise about finding duplicates.

To get you started load the arbuthnotdata set.

data(arbuthnot)

The Arbuthnot data set refers to Dr. John Arbuthnot, an 18th century physician, writer, and mathematician. He was interested in the ratio of newborn boys to newborn girls, so he gathered the baptism records for children born in London for every year from 1629 to 1710. We can view the data by typing its name into the console.

However printing the whole dataset in the console is not that useful. Instead use the integrated data viewer of RStudio or use also some other commands like head, tail, str. As the data in this case is a tibble you can also use glimpse, which similiar to str.

glimpse(arbuthnot)

## Observations: 82
## Variables: 3
## $ year  <int> 1629, 1630, 1631, 1632, 1633, 1634, 1635, 1636, 1637, 16...
## $ boys  <int> 5218, 4858, 4422, 4994, 5158, 5035, 5106, 4917, 4703, 53...
## $ girls <int> 4683, 4457, 4102, 4590, 4839, 4820, 4928, 4605, 4457, 49...

We can see that there are 82 observations and 3 variables in this dataset. The variable names are year, boys, and girls.

Some Exploration

We can access the data in a single column of a data frame separately.

arbuthnot$boys

##  [1] 5218 4858 4422 4994 5158 5035 5106 4917 4703 5359 5366 5518 5470 5460
## [15] 4793 4107 4047 3768 3796 3363 3079 2890 3231 3220 3196 3441 3655 3668
## [29] 3396 3157 3209 3724 4748 5216 5411 6041 5114 4678 5616 6073 6506 6278
## [43] 6449 6443 6073 6113 6058 6552 6423 6568 6247 6548 6822 6909 7577 7575
## [57] 7484 7575 7737 7487 7604 7909 7662 7602 7676 6985 7263 7632 8062 8426
## [71] 7911 7578 8102 8031 7765 6113 8366 7952 8379 8239 7840 7640

This command will only show the number of boys baptized each year. The dollar sign basically says “go to the data frame that comes before me, and find the variable that comes after me”.

Exercise 1:

What command would you use to extract just the counts of girls baptized?

arbuthnot$girls

##  [1] 4683 4457 4102 4590 4839 4820 4928 4605 4457 4952 4784 5332 5200 4910
## [15] 4617 3997 3919 3395 3536 3181 2746 2722 2840 2908 2959 3179 3349 3382
## [29] 3289 3013 2781 3247 4107 4803 4881 5681 4858 4319 5322 5560 5829 5719
## [43] 6061 6120 5822 5738 5717 5847 6203 6033 6041 6299 6533 6744 7158 7127
## [57] 7246 7119 7214 7101 7167 7302 7392 7316 7483 6647 6713 7229 7767 7626
## [71] 7452 7061 7514 7656 7683 5738 7779 7417 7687 7623 7380 7288

Notice that the way R has printed these data is different. When we looked at the complete data frame, we saw 82 rows, one on each line of the display. These data are no longer structured in a table with other variables, so they are displayed one right after another. Objects that print out in this way are called vectors; they represent a set of numbers. R has added numbers in [brackets] along the left side of the printout to indicate locations within the vector. For example, 5218 follows [1], indicating that 5218 is the first entry in the vector. And if [43] starts a line, then that would mean the first number on that line would represent the 43rd entry in the vector.

Data visualisation

R has some powerful functions for making graphics. We can create a simple plot of the number of girls baptized per year with qplot.

qplot(x = year, y = girls, data = arbuthnot)

The qplot() function (meaning “quick plot”) considers the type of data you have provided it and makes the decision to visualize it with a scatterplot. The plot should appear under the Plots tab of the lower right panel of RStudio. Notice that the command above again looks like a function, this time with three arguments separated by commas. The first two arguments in the qplot() function specify the variables for the x-axis and the y-axis and the third provides the name of the data set where they can be found. If we wanted to connect the data points with lines, we could add a fourth argument to specify the geometry that we’d like.

qplot(x = year, y = girls, data = arbuthnot, geom = "line")

You might wonder how you are supposed to know that it was possible to add that fourth argument. Thankfully, R documents all of its functions extensively. To read what a function does and learn the arguments that are available to you, just type in a question mark followed by the name of the function that you’re interested in. Try the following.

?qplot

Exercise 2:

Is there an apparent trend in the number of girls baptized over the years? How would you describe it? (To ensure that your lab report is comprehensive, be sure to include the code needed to make the plot as well as your written interpretation.)

qplot(x = year, y = girls, data = arbuthnot, geom = "line")

After a twenty year decline (1640 - approx. 1660) the number of baptized girls generally is increasing.

R as a big calulator

Now, suppose we want to plot the total number of baptisms. To compute this, we could use the fact that R is really just a big calculator. We can type in mathematical expressions like

5218 + 4683

## [1] 9901

to see the total number of baptisms in 1629. We could repeat this once for each year, but there is a faster way. If we add the vector for baptisms for boys to that of girls, R will compute all sums simultaneously.

arbuthnot$boys + arbuthnot$girls

##  [1]  9901  9315  8524  9584  9997  9855 10034  9522  9160 10311 10150
## [12] 10850 10670 10370  9410  8104  7966  7163  7332  6544  5825  5612
## [23]  6071  6128  6155  6620  7004  7050  6685  6170  5990  6971  8855
## [34] 10019 10292 11722  9972  8997 10938 11633 12335 11997 12510 12563
## [45] 11895 11851 11775 12399 12626 12601 12288 12847 13355 13653 14735
## [56] 14702 14730 14694 14951 14588 14771 15211 15054 14918 15159 13632
## [67] 13976 14861 15829 16052 15363 14639 15616 15687 15448 11851 16145
## [78] 15369 16066 15862 15220 14928

What you will see are 82 numbers (in that packed display, because we aren’t looking at a data frame here), each one representing the sum we’re after. Take a look at a few of them and verify that they are right.

Adding a new variable to the data frame

We’ll be using this new vector to generate some plots, so we’ll want to save it as a permanent column in our data frame.

arbuthnot <- arbuthnot %>%
  mutate(total = boys + girls)

The %>% operator is called the piping operator. It takes the output of the previous expression and pipes it into the first argument of the function in the following one. To continue our analogy with mathematical functions, x %>% f(y) is equivalent to f(x, y).

A note on piping: Note that we can read these three lines of code as the following: “Take the arbuthnot dataset and pipe it into the mutate function. Mutate the arbuthnot data set by creating a new variable called total that is the sum of the variables called boys and girls. Then assign the resulting dataset to the object called arbuthnot, i.e. overwrite the old arbuthnot dataset with the new one containing the new variable.” This is equivalent to going through each row and adding up the boys and girls counts for that year and recording that value in a new column called total.

Where is the new variable? When you make changes to variables in your dataset, click on the name of the dataset again to update it in the data viewer. You’ll see that there is now a new column called total that has been tacked on to the data frame. The special symbol <- performs an assignment, taking the output of one line of code and saving it into an object in your workspace. In this case, you already have an object called arbuthnot, so this command updates that data set with the new mutated column.

We can make a plot of the total number of baptisms per year with the command. But here is always used the qplot command.

qplot(x = year, y = total, data = arbuthnot, geom = "line")

> qplot is a shortcut designed to be familiar if you’re used to base plot(). It’s a convenient wrapper for creating a number of different types of plots using a consistent calling scheme. It’s great for allowing you to produce plots quickly, but I [Hadley Wickham] highly recommend learning ggplot() as it makes it easier to create complex graphics.

What follow is a (long?) version of the qplot graph with ggplot command:

p <- ggplot(data = arbuthnot, mapping = aes(x = year, y = total))
p + geom_line()

I am now providing a short version of the plot graph with the pipecommand, dropping the names of the arguments for default values but adding labels and title.

arbuthnot %>% ggplot(aes(year, total)) +
        geom_line() +
        xlab("Year") + ylab("Baptized children (boys and girls)") +
        ggtitle("Baptized children, born in London. Recorded by Dr. John Arbuthnot.")

Similarly to how we computed the total number of births, we can compute different kinds of ratios (boys to girls, boys to total, girls to total).

Additionalley I will plot the change of the boy to girl ratio over the year.

arbuthnot <- arbuthnot %>%
  mutate(boy_to_girl_ratio = boys / girls)
arbuthnot <- arbuthnot %>%
  mutate(boy_ratio = boys / total)
arbuthnot <- arbuthnot %>%
  mutate(girl_ratio = girls / total)
arbuthnot %>% 
        ggplot(aes(year, boy_to_girl_ratio)) +
        geom_line() +
        xlab("Year") + ylab("Baptized ratio (boys to girls)") +
        ggtitle("Baptized children, born in London. Recorded by Dr. John Arbuthnot.")

Exercise 3:

In addition to simple mathematical operators like subtraction and division, you can ask R to make comparisons like greater than, >, less than, <, and equality, ==. For example, we can ask if boys outnumber girls in each year with the expression

arbuthnot <- arbuthnot %>%
  mutate(more_boys = boys > girls)

This command add a new variable to the arbuthnot dataframe containing the values of either TRUE if that year had more boys than girls, or FALSE if that year did not (the answer may surprise you). This variable contains a different kind of data than we have encountered so far. All other columns in the arbuthnot data frame have values that are numerical (the year, the number of boys and girls). Here, we’ve asked R to create logical data, data where the values are either TRUE or FALSE. In general, data analysis will involve many different kinds of data types, and one reason for using R is that it is able to represent and compute with many of them.

More Practice

Until now you recreated some of the displays and preliminary analysis of Arbuthnot’s baptism data. Your assignment involves repeating these steps, but for present day birth records in the United States. Load the present day data with the following command:

data(present)

Interlude: Download and import of a real data set

WONDER interactive website

Data up to 2002 appear in Mathews TJ, and Hamilton BE. 2005. Trend analysis of the sex ratio at birth in the United States. National Vital Statistics Reports 53(20):1-17. Data for 2003 - 2013 have been collected from annual National Vital Statistics Reports published by the US Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics.

The source of the data set present is on the one hand a report by the Centers for Disease Control. But for the newer data beginning with 2003-2013 the data is from the CDC website. The most recent file (until 2015) are downloadable from the CDC portal. See also the new WONDER website where data can manipulated interactively in the browser.

See for more info the WONDER about file:

CDC Wide-ranging ONline Data for Epidemiologic Research (CDC WONDER) is a web application that makes many health-related data sets available to the worldwide public health community. Users include state and local health departments, academic researchers, healthcare providers, CDC surveillance programs, and the general public. The data found on CDC WONDER aid users in public health research, decision making, priority setting, program evaluation, and resource allocation.

For using the WONDER website one has to read the detailed description in the Dataset Documentation.

Download and import from CDC

I downloaded one data file in zip format April 1, 2010, July 1, 2010 - July 1, 2015. It is a flat text files with a size of 307 MB. The description about the columns can be found in Documentation for bridged-race postcensal Vintage 2015 population estimates for April 1, 2010 - July 1, 2015.

In the PDF file a special file layout is explained:

Location	size	Item and code outline	Format
1-4	4	Series vintage (2015)	Numeric
5-6	2	State FIPS code	Numeric
7-9	3	County FIPS code	Numeric
10-11	2	Age (0, 1, 2,…, 85 years and over)	Numeric
12	1	Bridged-race-sex	Numeric
13	1	Hispanic origin	Numeric
14-21	8	April 1, 2010 base population estimate	Numeric
22-29	8	July 1, 2010 postcensal resident population estimate	Numeric
30-37	8	July 1, 2011 postcensal resident population estimate	Numeric
38-45	8	July 1, 2012 postcensal resident population estimate	Numeric
46-53	8	July 1, 2013 postcensal resident population estimate	Numeric
54-61	8	July 1, 2014 postcensal resident population estimate	Numeric
62-69	8	July 1, 2015 postcensal resident population estimate	Numeric

I am going to use the fixed with format of the function read_fwf() from the readr package. Strange: It is much faster to use the “Import Dataset” Utility from RStudio than to laod the file programmatically.

birth_dataset <- read_fwf("data/pcen_v2015_y1015_txt.txt", fwf_widths(c(4, 2, 3, 2, 1, 1, 8, 8, 8, 8, 8, 8, 8), c("year", "state", "county", "age", "race.sex", "hisp", "pop.est1", "pop.est2", "pop.est3", "pop.est4", "pop.est5", "pop.est6", "pop.est7")))

Exercise 4:

What years are included in this data set? What are the dimensions of the data frame? What are the variable (column) names?

glimpse(present)

## Observations: 74
## Variables: 3
## $ year  <dbl> 1940, 1941, 1942, 1943, 1944, 1945, 1946, 1947, 1948, 19...
## $ boys  <dbl> 1211684, 1289734, 1444365, 1508959, 1435301, 1404587, 16...
## $ girls <dbl> 1148715, 1223693, 1364631, 1427901, 1359499, 1330869, 15...

head(present)

## # A tibble: 6 × 3
##    year    boys   girls
##   <dbl>   <dbl>   <dbl>
## 1  1940 1211684 1148715
## 2  1941 1289734 1223693
## 3  1942 1444365 1364631
## 4  1943 1508959 1427901
## 5  1944 1435301 1359499
## 6  1945 1404587 1330869

tail(present)

## # A tibble: 6 × 3
##    year    boys   girls
##   <dbl>   <dbl>   <dbl>
## 1  2008 2173625 2074069
## 2  2009 2113739 2016926
## 3  2010 2046561 1952825
## 4  2011 2024068 1929522
## 5  2012 2021800 1931041
## 6  2013 2013108 1919073

dim(present)

## [1] 74  3

names(present)

## [1] "year"  "boys"  "girls"

• What years are included in this data set? 1940 - 2013
• What are the dimensions of the data frame? 3 variable (columns)
• What are the variable (column) names? year, boys and girls.

Exercise 5:

How do these counts compare to Arbuthnot’s? Are they of a similar magnitude?

present <- present %>%
  mutate(more_boys = boys > girls)

The preset data set has a much higher magnitude.

Exercise 6:

Make a plot that displays the proportion of boys born over time. What do you see? Does Arbuthnot’s observation about boys being born in greater proportion than girls hold up in the U.S.? Include the plot in your response. Hint: You should be able to reuse your code from Ex 3 above, just replace the dataframe name.

present <- present %>%
  mutate(boy_to_girl_ratio = boys / girls)
present %>% 
        ggplot(aes(year, boy_to_girl_ratio)) +
        geom_line() +
        xlab("Year") + ylab("Birth ratio (boys to girls)") +
        ggtitle("Children born in US, recorded by CDC.")

Exercise 7:

In what year did we see the most total number of births in the U.S.? Hint: First calculate the totals and save it as a new variable. Then, sort your dataset in descending order based on the total column. You can do this interactively in the data viewer by clicking on the arrows next to the variable names. To include the sorted result in your report you will need to use two new functions: arrange (for sorting). We can arrange the data in a descending order with another function: desc (for descending order). Sample code provided below.

present <- present %>%
        mutate(total = boys + girls)
present <- present %>%
        arrange(desc(total))
head(present, 10)

## # A tibble: 10 × 6
##     year    boys   girls more_boys boy_to_girl_ratio   total
##    <dbl>   <dbl>   <dbl>     <lgl>             <dbl>   <dbl>
## 1   2007 2208071 2108162      TRUE          1.047392 4316233
## 2   1961 2186274 2082052      TRUE          1.050057 4268326
## 3   2006 2184237 2081318      TRUE          1.049449 4265555
## 4   1960 2179708 2078142      TRUE          1.048873 4257850
## 5   1957 2179960 2074824      TRUE          1.050672 4254784
## 6   2008 2173625 2074069      TRUE          1.048000 4247694
## 7   1959 2173638 2071158      TRUE          1.049480 4244796
## 8   1958 2152546 2051266      TRUE          1.049374 4203812
## 9   1962 2132466 2034896      TRUE          1.047948 4167362
## 10  1956 2133588 2029502      TRUE          1.051286 4163090

ggplot(present, aes(year, total)) +
        geom_line()

Further resources

That was a short introduction to R and RStudio, but we will provide you with more functions and a more complete sense of the language as the course progresses.

In this course we will be using R packages called dplyr for data wrangling and ggplot2 for data visualization. If you are googling for R code, make sure to also include these package names in your search query. For example, instead of googling “scatterplot in R”, google “scatterplot in R with ggplot2”.

There are some other resources for learning R and working in RStudio. This list of “cheatsheets” may come in handy:

• RMarkdown cheatsheet
• Data wrangling cheatsheet
• Data visualization cheatsheet

Chester Ismay has put together a resource for new users of R, RStudio, and R Markdown. It includes examples showing working with R Markdown files in RStudio recorded as GIFs.