In This Demonstration….

The Data Science Workflow:

Offered in this document is a demonstration of the following elments in the data science workflow: importing data; understanding data through processes for transformation, visualization, and modeling; and communication through processes of explanation and documentation of information from these data.

Structure of the Demonstration

– Elements

In this demonstration I use mtcars, a dataset available through the datasets package that accompanies the base R installation.

  • First, one way to refine this dataset is demonstrated (There are many ways, as with all of the processes I demonstrate with mtcars).

  • Then, some variables in mtcars are transformed in various ways using functions available in the R package called dplyr.

  • Next, vizualizations of some variables in mtcars are shown using functions available in the R package ggplot2.

  • Last, several simple statistical hypotheses are modeled with some variables in the mtcars dataset.

– Explanation and documentation?

The example of these processes in the data sciences is provided in the file that created this web page, R_Example_30April2019.Rmd, which is an RMarkdown file. RMarkdown is an authoring language that enables easy creation of dynamic documents, presentations, and reports from R. It combines the core syntax of markdown (an easy-to-write plain text format commonly applied in data science) with embedded “chunks” of R code that are run so their output can be included in a final document.

– Same analysis is in RScript file

The RScript file, ‘R_Example_30April2019.R’, implements all of the R processes explained on this HTML page.

– R code is embedded in this demonstration

Any line that starts with a “#” is treated as a comment in R code and is not executed by R. Annotation of RScripts with comments aids in reproducibility of analyses. In most cases, the R code used to creat the analyses shown on thsi HTML page is reproduced. Typically, however, RMarkdown reports hide the R code useds to create the page.

– Note: Demo, not instruction

This demonstration is not meant to instruct you in the use of R or RStudio. It merely is an introductory tour of many R and RStudio capabilities.

1. Importing the dataset, mtcars

1.a. Structure of mtcars

To demonstrate processes in R, I import mtcars, a dataset with a small number of observations and variables. The data in mtcars were extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models). Said in another way, the data set, mtcars, has 32 observations (cars) with measures for 11 variables (aspects of design and performance). The mtcars dataset is “built-in” to RStudio, so we do not need to used methods available in R for importing data from external sources.

Here is a listing of the 32-observation-by-11-variable mtcars dataset to which you can refer throughout this demonstration.

1.b. Loading mtcars in RStudio

By entering the name of the dataset in the R console quadrant or in the source quadrant will print the dataset in the console:

mtcars

1.c. Printing mtcars to the RStudio console

Printing a dataset like mtcars is suitable for viewing in the console. It contains few observations and variables. However, what about datasets that contain hundreds … thousands … hundreds of thousands of observations?

Another approach is to print in the console the first six lines of the dataset using the R function, head.

head(mtcars)

Or, to print the last six lines.

tail(mtcars)

2. Transforming mtcars using dplyr

2.a. Loading dplyr

We can load an R “package” (a special purpose library of code in R that accomplishes a bundle of computing tasks) called dplyr to provide some order to the mtcars data.

library(dplyr)

2.b. Exploring mtcars with dplyr

We create a more orderly table of the mtcars dataset by executing what is called a tibble function, tbl_df, to transform mtcars into a new table of data, mtcars_df. (tibble….tbl_df….get it?)

Now, a print of mtcars_df is more orderly set of columns (variables) and only the first 10 rows (observations…or cars):

mtcars_df <- tbl_df(mtcars)
mtcars_df

Now that mtcars has been transformed, the result of the transformation, mtcars_df, is described in R as a dataframe rather than a dataset.

Also, note that the size of the table of data also is printed (32 rows by 11 columns….or, in our case, 32 cars by 11 aspects of performance).

We can use the dplyr function, glimpse, to view variable names, the type of variables (in this case “dbl” meaning numeric data), and a sample of some of the data points in mtcars_df:

glimpse(mtcars_df)
  Observations: 32
  Variables: 11
  $ mpg  <dbl> 21.0, 21.0, 22.8, 21.4, 18.7, 18.1, 14.3, 24.4, 22.8, 19.2,…
  $ cyl  <dbl> 6, 6, 4, 6, 8, 6, 8, 4, 4, 6, 6, 8, 8, 8, 8, 8, 8, 4, 4, 4,…
  $ disp <dbl> 160.0, 160.0, 108.0, 258.0, 360.0, 225.0, 360.0, 146.7, 140…
  $ hp   <dbl> 110, 110, 93, 110, 175, 105, 245, 62, 95, 123, 123, 180, 18…
  $ drat <dbl> 3.90, 3.90, 3.85, 3.08, 3.15, 2.76, 3.21, 3.69, 3.92, 3.92,…
  $ wt   <dbl> 2.620, 2.875, 2.320, 3.215, 3.440, 3.460, 3.570, 3.190, 3.1…
  $ qsec <dbl> 16.46, 17.02, 18.61, 19.44, 17.02, 20.22, 15.84, 20.00, 22.…
  $ vs   <dbl> 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1,…
  $ am   <dbl> 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,…
  $ gear <dbl> 4, 4, 4, 3, 3, 3, 3, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 4, 4, 4,…
  $ carb <dbl> 4, 4, 1, 1, 2, 1, 4, 2, 2, 4, 4, 3, 3, 3, 4, 4, 4, 1, 2, 1,…

The dataset, mtcars, and the dataframe created from it, mtcars_df, do not contain any missing data. It is quite common in practical research settings to have missing data for some cases (in mtcars_df, cars) within variables (in mtcars_df, aspects of performance). In WF ED 540 you will learn how to handle missing data in data modeling.

2.c. Summarising mtcars_df variables using dplyr functions

Rarely do researchers analyze the raw data that they collect. The R package, dplyr, makes reshaping data for analysis a relatively easy task.

It is common to filter a subset of rows in a data table according to criteria (dplyr’s filter function). Or, perhaps to select only some variables (dplyr’s select function).

Often, researchers wish to create new variables from existing variables (dplyr’s mutate function). And, statistical summaries of variables are desired (dplyr’s summarise function; note the spelling!).

The important and efficient use of dplyr is to use a small set of verbs to reshape data. Among the dplyr verbs you will learn to use to reshape data tables are:

  • 2.c.1. filter

  • 2.c.2. select

  • 2.c.3. mutate

  • 2.c.4 summarise

Provided are some examples (2..c.1. through 2.c.4.) of use of these verbs with the dataset, mtcars_df. Compare the transformations made with the raw data shown in the information sheet about mtcars that you received at the beginning of the first class meeting.

2.c.1. Filter mtcars

Here is a filter of only the cars that have a straight engine:

filter(mtcars_df, vs == 1)

Notice that 14 of the 32 cars meet this filtering criterion (see “Source: local data frame [14 x 11]”).

The filter verb has two parameters: the dataset to be filtered (mtcars_df); the filtering criterion (vs == 1, where “==” means “the same as”).

2.c.2. Select mtcars variables

Suppose I wish to select only mtcars_df variables that contain measures of weight (in pounds/1,000) and number of forward gears:

select(mtcars_df, wt, gear)

Notice that the selection retains all 32 cars, but only two variables (see “Source: local data frame [32 x 2]”)

2.c.3 Mutate mtcars variables

Now, suppose I wish to mutate mtcars_df by calculating a new variable from disp, the engine displacment in cubic inches: disp_l, the displacement in liters:

displace <- mutate(mtcars, displ_l = disp / 61.0237)
select (displace, disp, displ_l)

I created a new variable, displ_l, by converting cubic inches to liters (there are 61.0237 cubic inches in a liter). I created a new data table, displace, from which I, then, selected only disp and displ_l, the displacement in cubic inches and liters, respectively.

2.c.4 Summarize mtcars variables

Last, let’s calculate some summary statistics for a few variables in the new data I created, displace. Calculated are the means for disp and displ_l:

summarise(displace, count=n(), mean_cu_in = mean(disp), mean_ltr=mean(displ_l))

I added the function, count, to return the number of cars in mtcars_df. The means are the sum of all disp or displ_l divided by the number of cars.

3. Visualizing mtcars_df variables using ggplot2

As indicated in a white paper by Fusion Charts,

We visualize information to meet a very basic need – to tell a story. It’s one of the most primitive forms of communication known to man, having its origins in cave drawings dated as early as 30,000 B.C., even before written communication, which emerged in 3,000 B.C. Vision is the single most important faculty we use to communicate information.

A R package, ggplot2, provides a grammar for commands that create visualizations of data.

The concept behind ggplot2 divides plot into three different fundamental parts: Plot = data + Aesthetics + Geometry. The principal components of every plot can be defined as follow:

3.a. Simple bivariate plot

I wish to plot the relationship between the weight of a car and its mileage. Make sure you have ggplot2 installed. We load the ggplot2 library. Then, I use the ggplot2 code to creates a scatterplot of car miles per gallon by car weight.

In this R code, a plot is created using a dataframe (mtcars_df) + aesthetics (the x and y axes) + a geometry (geom_points, which plots points):

# Make sure you have installed the ggplot2 package.
library(ggplot2)
  ggplot(mtcars_df, aes(x=wt, y=mpg)) + 
   geom_point()

Plotted on the graph are the weight (wt) and lines per gallon (mpg) for every car. The joint occurence of wt and mpg is what is called a Cartesian coordinate and is show by a point for each car.

3.b. Change size of points on plot

But, hmmm, the points are small, aren’t they? So, let’s change the size of the points by modifying geom_point() to geom_point(size=5).

  ggplot(mtcars_df, aes(x=wt, y=mpg)) + 
   geom_point(size=5)

3.c. Change the shape of points on plot

Let’s change the plot so that the shape of points is now a diamond (shape=23), the size is larger (size=8_), the color is red (color=“red”), and the diamonds are not filled (fill=NA:

   ggplot(mtcars_df, aes(x=wt, y=mpg)) + 
   geom_point(size=8, shape=23, color="red", fill=NA)

3.d. Draw a straight line through points on plot

How about adding a line though black, round points (minus some of the other previously used plot charactertistics) that minimizes the distance between the line and the swarm of points?

  ggplot(mtcars_df, aes(x=wt, y=mpg)) +
  geom_point(size=4) +
  geom_smooth(method=lm, se=FALSE)

3.e. Change the line to follow contour of the relationship

  ggplot(mtcars_df, aes(x=wt, y=mpg)) +
  geom_point(size=4) +
  geom_smooth(se=FALSE)

3.f. Change the color of the line

Change the color of the smooth line? Add color=“red” to the geom_smooth characteristic.

  ggplot(mtcars_df, aes(x=wt, y=mpg)) +
  geom_point(size=4) +
  geom_smooth(se=FALSE, color="red")

3.g. Label axes and add a title

Pretty. But, what about a title and labels for axes of the graph?

  ggplot(mtcars_df, aes(x=wt, y=mpg)) +
  geom_point(size=4) +
  geom_smooth(se=FALSE, color="red") +
  labs(title="Relationship Between Miles Per Gallon and Weight",
       x="Weight", y="Miles Per Gallon")

3.h. Add a third variable to the plot

Now, let’s plot wt and mpg by the number of cylinders (cyl), make points based on different numbers of cylinders different shapes and colors, and add a legend to the plot?

mtcars_df$cyl <- as.factor(mtcars_df$cyl) 
# Previous command changes the cyl variable from numeric 
# to a factor (a categorical variable).
 ggplot(mtcars_df, aes(x=wt, y=mpg, shape=cyl, color=cyl)) +
   geom_point(size=4) +
  labs(title="Relationship Between Miles Per Gallon and Weight",
       x="Weight", y="Miles Per Gallon")

3.i. Move the legend to the top of the plot

 ggplot(mtcars_df, aes(x=wt, y=mpg, shape=cyl, color=cyl)) +
   geom_point(size=4) +
  labs(title="Relationship Between Miles Per Gallon and Weight",
       x="Weight", y="Miles Per Gallon") +
   theme(legend.position="top")

3.j. Change the look and feel of the plot using themes

Change the “look and feel” of the plot I added theme_economist, a based on plot design for The Economist magazine, after installing and loading the package, “ggthemes.”

# Make sure you have installed the ggthemes package.
library(ggthemes)
mtcars_df$cyl <- as.factor(mtcars_df$cyl) 
 ggplot(mtcars_df, aes(x=wt, y=mpg, shape=cyl, color=cyl)) +
   geom_point(size=4) +
  labs(title="Relationship Between Miles Per Gallon and Weight",
       x="Weight", y="Miles Per Gallon") +
   theme(legend.position="top") + 
   theme_economist()

Some academic journals do not accept color graphics. This plot uses a grey scale color palette entirely:

 ggplot(mtcars_df, aes(x=wt, y=mpg, shape=cyl, color=cyl)) +
   geom_point(size=4) +
  labs(title="Relationship Between Miles Per Gallon and Weight",
       x="Weight", y="Miles Per Gallon") +
   theme(legend.position="top",
         axis.title=element_text(size=17),
         axis.text=element_text(size=13),
         plot.title=element_text(size=19)) + 
   scale_color_grey()

4. Statistical Modeling with Some Variables in mtcars_df

4.a. Simple t-test

Suppose I am interested in testing the null hypothesis that fuel efficiency, miles/gallon, is equal for cars with automatic and manual transmissions. There are observable differences in our 32-car sample. But, mtcars is just a sample from a population of cars. Can I infer that mileage is the same or different in the population of cars based on this sample of data.

To test this hypothesis, I structure a t-test of the differences between mean mileage by transmission type:

mpg.at <- mtcars_df[mtcars$am == 0,]$mpg
mpg.mt <- mtcars_df[mtcars$am == 1,]$mpg
t.test(mpg.at, mpg.mt) 
  
    Welch Two Sample t-test
  
  data:  mpg.at and mpg.mt
  t = -3.7671, df = 18.332, p-value = 0.001374
  alternative hypothesis: true difference in means is not equal to 0
  95 percent confidence interval:
   -11.280194  -3.209684
  sample estimates:
  mean of x mean of y 
   17.14737  24.39231

The mean mileage for cars with manual transmissions is approximatly 24, while for automatics is approximatly 17 – a 7 miles/gallon difference in the sample. According to my analysis, I am 95% confident that the actual mileage difference between automatic and manual transmissions is between 3 and 11 miles per gallon.

If I set the probability of Type 1 error (error made rejecting the null hypothesis when it should not be rejected) to 0.05, I can reject the null hypothesis of “no difference” because the probability is 0.0013 of observing a t-value of 3.76, with 18 degrees of freedom, if the null hypothesis is true.

Hey, now, that’s a mouthful. But, you will not only learn to talk that gibberish in WF ED 540, but you also will learn and practice to understand it!

Now, this whole thing in a graphic:

boxplot(mpg~am, data=mtcars_df, main ='Fig. 1. Fuel Efficiency',
        ylab='Miles per gallon',names=c("Automatic","Manual"),notch=FALSE, col=(c("gold","skyblue")))

4.b. Simple Regression

OK, now, I test this hypothesis in another way – through linear regression.

Linear regression fits a line a “best fit” through a plot of points formed by the intersection of coordinates formed by values of two variables. The equation of a regression line is y = a + bx, where y is a depndent variable (mpg in our case), x is an independent variable (type of transmission), a is an intercpt term, and b is the slope of the “best fit” rgression line.

So, in R:

fit_1var <- lm(mtcars$mpg ~ mtcars$am)
summary(fit_1var)$coefficients
               Estimate Std. Error   t value     Pr(>|t|)
  (Intercept) 17.147368   1.124603 15.247492 1.133983e-15
  mtcars$am    7.244939   1.764422  4.106127 2.850207e-04
par(mfrow = c(1,2))
plot(mtcars$am, mtcars$mpg, main = 'MPG & Transmission')
abline(fit_1var)

The slope of the regression line is approximately 7, which is change in mpg for a change of one unit in am (the difference between automatic transmissions, coded “0”, and manual transmissions, coded “1”). This is the same result as for the t-test.

5. Explanation and Documentation

This report is an example of explanation of findings of data analysis. Moreover, this report also is an example of documentation through use of RMarkdown.

You can learn and practice explanation and documentation of analyses using RMarkdown to create HTML reports, slide presentations, and PDF files. You will host your HTML reports at RPubs, a free RMarkdown document hosting service.

This document is hosted at http://rpubs.com/davidpassmore/491603.