The Goal

This course is designed to help you develop your coding skills and your analysis skills in order to prepare you for DataFest. At the start of DataFest, you will be given a data set and provided with the general research goals your client would like you to explore - and that’s it! How you approach the problem, what model you choose…everything is completely up to you.

Throughout this course, we are going to practice some of the skills that will help you tackle these sorts of open ended problems. For today, we are going to start with exploring how to visualize data in R. We will also see some best practices for visualizations (like adding appropriate labels and titles to graphs) that your team should use for DataFest and indeed any formal report or presentation!

As you work through this activity, please use RMarkdown or Quarto so it’s easy to submit your work. We strongly recommend storing your work in a folder you can find later, so you can refer to it during DataFest!

The Data

For our first activity, the data can be downloaded from dropbox. We’ll practice with different ways of loading data as we go through the semester, but for today, all you need to do is create a chunk in RMarkdown, paste in the code below, and press play.

checkpoints_eoc <-read.csv("https://www.dropbox.com/scl/fi/3xda6hny48ojknoifupul/checkpoints_eoc.csv?rlkey=q6rksum9wc91pkte2qdg31s33&st=fr0rynut&dl=1")

Note: If any of those terms were unfamiliar to you, just let us know!

Client Request

Our data today is provided by a client who is interested in exploring how students engage with online statistics textbooks and how successfully students are performing using the different books. We are provided a data set of 10570 rows and 13 columns.

Note: Our client actually gave us multiple data files, and we will bring in the rest soon! For today, we are just going to focus on this one. DataFest data is either (1) huge, (2) messy, or (3) contained in multiples files - or sometimes a mixture of all 3. The CourseKata data will allow us to practice with merging multiple files a little later in the course.

Because the textbooks are online, students have online tests at the end of each chapter. Each row in our dataset represents the performance of a certain student in the year 2023 on a certain chapter of the book. Our 13 columns are:

EDA

Today, we are going to focus on EDA, which means using graphs, tables, and other visualization tools to see what is going on in the data. We have had people win DataFest with no models at all if they do EDA really well and present it well!!

In the real competition, we often need to explore multiple variables as we get to know the data set. This is a good time to split up the variables you might want to visualize among your team - it’s more efficient to split up the work!!

table( checkpoints_eoc$book )

length(  unique(  checkpoints_eoc$class_id) )

Making Plots

So far, we have used tables to explore a few of our variables. Now we are going to consider using graphs instead. To make visualizations, we will be working with a group of functions in the ggplot2 package in R. If you have already installed ggplot2, skip to the next section.

library(rlang)

suppressMessages(library(ggplot2))

EDA: Categorical Variables

We have seen that we can use a table to see the distribution of a categorical variable like book, but we have also seen that tables can sometimes be hard to read. Instead of a table, we can use bar plots.

To create a bar plot of release, for instance, paste the following code in a chunk and hit play.

ggplot(checkpoints_eoc, aes(x=release)) +
  geom_bar(fill='blue', col = 'black')

The creation of plots in ggplot2 requires building the plot in layers. First, we build the background, the grid on which we will be building our graph. This is the job of the ggplot() part of the code. Once we have built the background, we are ready to plot our data. The command we will use for this depends upon the data type we are working with. In this case, we want a bar plot. The command we use to build a bar plot is geom_bar(). In this same layer, we specify that we want the bars to be filled ('fill') in blue and we want them outlined ('col') in black.

Notice that to add each layer to the graph, in the code we use a plus sign. We add the background AND THEN the bars to make the final graph. Let’s break that down in more detail:

• ggplot(checkpoints_eoc, aes(x=release)): This part of the code creates the background of the plot. The two arguments are the data set we are using (checkpoints_eoc) and the variable(s) that will be used to define the axis/axes(aes). In this case, we defined only that the x-axis would contain information on release (aes(x=release)).
• geom_bar(fill='blue', col = 'black'): Once the axes are set, we are adding on (+) the actual data. In this case, we want a bar plot, so we add bars (geom_bar). We also specify those bars should colored blue and outlined in black.

ggplot(checkpoints_eoc, aes(x=release)) +
  geom_bar(fill='blue', col = 'black') +  
  labs(title="Figure 1:", x = "Release", y = "Number of Rows")

ggplot(checkpoints_eoc, aes(x=release)) +
  geom_bar(fill='blue', col = 'black') +  
  labs(title="Figure 1:", x = "Release", y = "Number of Rows",
  caption = "A bar plot of release number.")

ggplot(checkpoints_eoc, aes(x=eoc)) +
  geom_histogram(fill='blue', col = 'black', bins = 20) + 
  labs (title="Figure 1:", x = "EOC score", y = "Frequency", caption = "A histogram of EOC score.") +
  theme_light()

ggplot (checkpoints_eoc, aes(x=eoc)) +
  geom_histogram (fill='blue', col = 'black', bins = 20) +
  labs (title="Figure 1:", x = "EOC score", y = "Frequency", caption = "A histogram of EOC score.") +
  theme_bw()

ggplot (checkpoints_eoc, aes(x=eoc)) +
  geom_histogram (fill='blue', col = 'black', bins = 20) +
  labs (title="Figure 1:", x = "EOC score", y = "Frequency", caption = "A histogram of EOC score.") +
  theme_minimal()

Plotting Two Variables

Our client has asked us to explore student success, so one thing we may want to do is see if student EOC scores tend to be different depending on the book being used, or the version of the book. This means that we want to create a graph to examine the relationship between two variables.

Let’s say we want to explore whether EOC scores differ from release to release. We know that we can use a box plot to visualize EOC score and a bar plot to visualize book…but how can we visualize both??

We can actually create something called a side-by-side boxplot.

ggplot(checkpoints_eoc, aes(release, eoc)) + 
  geom_boxplot(fill = "gold", col = "blue") 

You’ll notice that this is the same as a standard boxplot except that we added a categorical variable on the x axis.

In this plot, you will notice that the x axis is really hard to read!! This is because the book titles are really long. This means we need to change the book titles to something shorter. This kind of data cleaning is also a big part of DataFest!

There are a lot of ways we can do this, but let’s start with a simple one.

checkpoints_eoc$book <- ifelse(checkpoints_eoc$book == "College / Statistics and Data Science (ABC)", "College", checkpoints_eoc$book)

This command says that if the book is College / Statistics and Data Science (ABC), we want to rename the book College. This is shorter and easier for us to work with! If the book is named something else, do not change the name.

Now that we have a graph, we want to use it to try to relate back to the client’s request. They wanted us to explore student success. We now have a plot that shows how EOC scores differ across different textbooks.

Question 13 is exactly the kind of thing you will do at the DataFest competition - you will be explaining your graphs and results to a panel of judges. Being clear here is key!

Okay, so we can look at EOC scores across different books. What if we want to look at how it differs by book and version at the same time? We also have the ability to use faceted plots to do this.

ggplot(checkpoints_eoc, aes(release, eoc)) + 
  geom_boxplot(fill = "gold", col = "blue") +
  facet_wrap( ~ book , ncol = 3)

You’ll notice that we know have three sets of side by side box plots grouped together by the textbook. This is what faceting by a categorical variable means!

If you feel the plot is too compressed, you can change ncol = 3 to ncol = 2. Try it and see what you think!

Stacking Graphs

We have seen how to make a few different kinds of graphs so far. Great. However, we have a lot of variables to explore, so we have the potential for a lot of plots. That starts to take up a lot of space in a report. One nice way to present multiple graphs is to stack them.

Suppose I want to make a histogram of EOC score. The code I need for that is:

ggplot(checkpoints_eoc, aes(x=eoc)) +  
  geom_histogram(fill='blue', col = 'black', bins = 20) +
  labs(title="Figure 1:", x = "EOC score", y = "Frequency")

If I put that in a chunk and press play, one histogram will appear on my screen. Let’s suppose I want to show a box plot along with this histogram. I can create both graphs and have them both print out separately in my knit document. However, I can also tell the computer to print more than one graph at once to save space. Try the following:

# Create the histogram 
g1 <-  ggplot(checkpoints_eoc, aes(x=eoc)) +
  geom_histogram(fill='blue', col = 'black', bins = 20) +
  labs (title="Figure 1:", x = "EOC score", y = "Frequency")
  
# Create the boxplot 
g2 <-  ggplot(checkpoints_eoc, aes(x=eoc)) + geom_boxplot(fill='gold') +  labs (title="Figure 2:", x = "EOC score", y = "")

# Show both plots 
gridExtra::grid.arrange(g1,g2, ncol = 2)

If you get a warning message about not having gridExtra, this means you will need to install the package using the same steps we did above to install ggplot2. We have to install packages all the time with R, depending on the version of R you have and your computer system.

What you will notice is that we have stored each of the two graphs under a name. Our histogram is stored under g1 and our box plot is stored under g2. Then, we use a special code called gridExtra::grid.arrange to help us arrange the graphs in a grid. In our case, we have to graphs, and we want them side by side. This means we want the graphs in a 1 (row) by 2 (column) grid.

To create the 1 by 2 grid, we feed the computer our two graphs, and then tell it we want the figures in 2 columns (next to each other) by specifying ncol = 2. In other words, the number of columns we want is 2!

We have made a good start with exploring the data! We will keep working with this data set in future activities.

Submitting your assignment

For today, everyone is going to submit their own assignment. Knit your Markdown (or Quarto if you prefer) file and submit either a PDF or html. Welcome to STA 175!!!