The first step of any analysis is reading in the data. We’ve already seen that the dataset is a spreadsheet saved as a comma separated value file. These files can be read with the built-in read.csv() function. You can get more info on this function by typing
?read.csv
at the R console.
The main argument to this function is the name of the file, and for most .csv files this is enough to read in the data correctly.
After reading in the data and saving it in a variable, it’s a good idea to look at the first and last few rows to make sure things loaded properly. The functions head() and tail() can be used to display the first few and last few rows of a varaible.
The next thing that should always be done with a new dataset, is to get an idea of how big it is. The function dim() gives the number of rows and columns on a data.frame
Show/Hide Code
dim(myiris)
[1] 150 5
It’s always a good idea to do some sanity checking, to make sure your data conforms to expected ranges and values. It’s common to have missing data, and or wild values, especially from spreadsheets containing entries that were manually edited. For small datasets, this can be done by hand/eye right in excel, but the task gets harder the more observations are involved.
As long as you know what to look for, the computer can be used to makes the checks.
One idea is to look at the range of the values. The minimum and maximums of each column.
Columns can be accessed by the $ data.frame operator, or by selecting specific columns with [i,j] notation. You can get the names of the columns with the colnames function.
The functions min() and max() will find the minimum and maximum values in a numeric vector.
For categorical variablers like Species, it doesn’t make sense to look for min/max values. Instead, you can look at the # of occurances for each level using the table() function.
Show/Hide Code
table(myiris$Species)
setosa versicolor virginica
50 50 50
These sanity checks are so common and necessary, that R has a built-in way to running them all at once in one shot. The summary() function computes a bunch of simple stats on each column of a data.frame.
Show/Hide Code
summary(myiris)
Sepal.Length Sepal.Width Petal.Length Petal.Width
Min. :4.300 Min. :2.000 Min. :-92932.42 Min. :0.100
1st Qu.:5.100 1st Qu.:2.800 1st Qu.: 1.52 1st Qu.:0.300
Median :5.800 Median :3.000 Median : 4.30 Median :1.300
Mean :5.843 Mean :3.057 Mean : -615.82 Mean :1.199
3rd Qu.:6.400 3rd Qu.:3.300 3rd Qu.: 5.10 3rd Qu.:1.800
Max. :7.900 Max. :4.400 Max. : 6.90 Max. :2.500
Species
Length:150
Class :character
Mode :character
Fix the data
From the summary, it should be immediately clear that there’s a problem with one of the fields. The Petal.Length column has at least 1 extreme value, that is probably a mistake. We need to decide what to do with this.
In practice, if you encounter a wierd value it by be possible to figure out how/why it came about and maybe even fix or remeasure it. As an example, if you have a column in a clinical dataset for pulse oximeter readings with measurments in %’s in the expected range of 0-100, what if you see a measurements of 0.98?
In some cases, you can’t fix the data, and have to instead treat it as a missing value. R has a special value for this called NA, which stands for Not Available. All the built in operations in R understand this special value and will return results accordingly.
Let’s assign an NA to the wierd value, then run our summaries again. To do this, first we have to find the index of the row with the bad entry, then we have to update the data.frame with the NA value. For this dataset, we’ll make the executive decision to call any measurement less than 0 a bad value.
Show/Hide Code
ix = myiris$Petal.Length <0myiris$Petal.Length[ix] =NAsummary(myiris)
Sepal.Length Sepal.Width Petal.Length Petal.Width
Min. :4.300 Min. :2.000 Min. :1.00 Min. :0.100
1st Qu.:5.100 1st Qu.:2.800 1st Qu.:1.60 1st Qu.:0.300
Median :5.800 Median :3.000 Median :4.30 Median :1.300
Mean :5.843 Mean :3.057 Mean :3.75 Mean :1.199
3rd Qu.:6.400 3rd Qu.:3.300 3rd Qu.:5.10 3rd Qu.:1.800
Max. :7.900 Max. :4.400 Max. :6.90 Max. :2.500
NA's :1
Species
Length:150
Class :character
Mode :character
Let’s also try our previous min/max functions on this newly updated column to see what happens
Show/Hide Code
min(myiris$Petal.Length)
[1] NA
Show/Hide Code
max(myiris$Petal.Length)
[1] NA
What happens if you try to add NA to a number?
You should always be careful computing a mathematical operations on a vector that contains NA’s. Most times, especially with summary stats like min,max,mean,etc., the best option is to just ignore the NA’s and use only the legitimate values. All of the built-in R functions have the capablity of ignoring NA values, and you can enable this option by adding a , na.rm = T parameter argument to the function call. This na.rm stands for NA remove, and will ignore that observation completely.
Show/Hide Code
min(myiris$Petal.Length, na.rm=T)
[1] 1
Show/Hide Code
max(myiris$Petal.Length, na.rm=T)
[1] 6.9
Save the data
Once we’ve made modifications to the original dataset, it’s a good idea to save it in a new file in case we need to access it again or share it with collaborators. For this example, editing a single line of a spreadsheet is quick and doesn’t really need to be saved in a separate file, but most real world datasets involves data cleanup that takes time and effort, sometimes more than is spent on the actual analysis. You need to keep track of all the changes you make to your data and an .R script is a great way to record all the steps that were taken.
saving the file as a comma separate value spreadsheet can be done with the built in write.csv() function
Show/Hide Code
#this one will add a column for the rownames by defaultwrite.csv(file ="myiris_modified.csv", x = myiris)#this one will ignore rownameswrite.csv(file ="myiris_modified2.csv", x = myiris, row.names=F)
View the data
Summaries like the built-in summary() function are good at getting a quick look at your data, but they don’t always tell you the whole picture. A plot can sometimes give a better understanding to what’s going on than a bunch of numbers can.
The most common ways to plot numerical data and look for associations is with histograms, boxplots and scatterplots and these are all available in R using built-in functions: hist(), boxplot(), plot().
Let’s see what these look like on the first 4 columns of the dataset.
Show/Hide Code
hist(myiris[,1])
Show/Hide Code
boxplot(myiris[,2])
Show/Hide Code
plot(x=myiris[,3], y=myiris[,4])
We’re going to go into the details of these functions in the 2nd week of class, but this is a good place to pause show off some of the benefits of using AI to help with our coding.
A boxplot of the Sepal.Width’s that separates each group by Species can be made with the following command.
Show/Hide Code
boxplot(myiris[,2] ~ myiris[,5])
If you have no or little experience with R, it’s very unlikely that you’ll be familiar with this command, or recognize the formula notation with the ~ that’s being used in the function call.
Luckily, chatbots like chatGPT have improved so much in the last few years, that you can use them to help with almost all your coding problems. For this example, I want you to ask a chatbot of your choice to make you a boxplot of the 2nd column of your dataset split by the 5th column.
I have a .csv file named "myiris.csv". I want you to load this in,
and make a boxplot of the 2nd column, split by the 5th column.
Give me the base R code that does this.
This is the response I got back using this exact prompt into chatGPT.
Show/Hide Code
# Load the csv filedf <-read.csv("myiris.csv")# Make a boxplot of the 2nd column, grouped by the 5th columnboxplot(df[[2]] ~ df[[5]],xlab =names(df)[5],ylab =names(df)[2],main =paste("Boxplot of", names(df)[2], "by", names(df)[5]))
If you understand the basics of function calling in R, it shouldn’t be too much a stretch to figure out what’s going on here. The boxplot() has some additional parameters that we haven’t gone over yet, named xlab and ylab that stand for the labels of the X and Y axes of the plot respectively. What’s going on with the last parameter main is a bit trickier to understand. main is the title of the plot and it will show up in bold letters at the top of the figure. The paste() function is a way to combine parts of character strings to form new ones. I know this because I have many years of experience with R, but how can you figure this out?
Ask the chatbot.
what does
main = paste("Boxplot of", names(df)[2], "by", names(df)[5]))
mean in your last response?
You should get a very clear line by line explanation that tells you exactly what’s going on. And if things are still not clear, ask again!
I don't understand, can you explain it in simpler terms?
Asking a chatgpt is not a cheat or a hack. It’s a way to help you learn the material, get ideas about the capabilities of R, and to work with the tricky syntax required for programming (in any language).
Every time you want to make a plot, process a file, or run a statistical test, it’s ok to start by asking a chatbot for help.
The key here is to understand that you’re not relying on the chatbot to do your work. You using it to help you figure out how to do the work youself.
Source Code
---title: "Interactive Data Exploration"format: html: toc: true toc-location: left code-fold: show code-tools: true code-summary: "Show/Hide Code" include-after-body: - text: | <script> document.addEventListener("DOMContentLoaded", function() { // Find all Quarto output blocks and make them behave like details tags const outputs = document.querySelectorAll('.cell-output'); outputs.forEach(output => { const details = document.createElement('details'); const summary = document.createElement('summary'); summary.textContent = 'Click to show output'; summary.style.cursor = 'pointer'; summary.style.fontWeight = 'bold'; summary.style.color = '#007bc2'; // Move output content into the details tag output.parentNode.insertBefore(details, output); details.appendChild(summary); details.appendChild(output); }); }); </script>execute: warning: false message: false---## Read in the dataThe first step of any analysis is reading in the data. We'vealready seen that the dataset is a spreadsheet savedas a *comma separated value* file. These files can be read with the built-in`read.csv()` function. You can get more info on this function bytyping `?read.csv`at the R console.The main argument to this function is the name of the file, and for most .csvfiles this is enough to read in the data correctly.After reading in the data and saving it in a variable, it's a good idea to look at the first and last few rows to make surethings loaded properly. The functions `head()` and `tail()` can be used to displaythe first few and last few rows of a varaible.```{r}myiris <-read.csv("myiris.csv")head(myiris)tail(myiris)```## Check the dataThe next thing that should always be done with a new dataset, is to get an idea ofhow big it is. The function `dim()` gives the number of rows and columns on a data.frame```{r}dim(myiris)```It's always a good idea to do some sanity checking, to make sure your dataconforms to expected ranges and values. It's common to have missing data,and or wild values, especially from spreadsheets containing entries thatwere manually edited. For small datasets, this can be done by hand/eye rightin excel, but the task gets harder the more observations are involved. As long as you know what to look for, the computer can be used to makes the checks.One idea is to look at the range of the values. The minimum and maximums of each column.Columns can be accessed by the `$` data.frame operator, or by selecting specific columnswith `[i,j]` notation. You can get the names of the columns with the `colnames` function.The functions `min()` and `max()` will find the minimum and maximum values in a numericvector.```{r}colnames(myiris)min(myiris$Sepal.Length)max(myiris$Sepal.Length)min(myiris$Sepal.Width)max(myiris$Sepal.Width)min(myiris$Petal.Length)max(myiris$Petal.Length)min(myiris$Petal.Width)max(myiris$Petal.Width)```For categorical variablers like *Species*, it doesn't make senseto look for min/max values. Instead, you can look at the # ofoccurances for each level using the `table()` function. ```{r}table(myiris$Species)```These *sanity checks* are so common and necessary, that R has a built-inway to running them all at once in one shot. The `summary()` function computesa bunch of simple stats on each column of a data.frame.```{r}summary(myiris)```## Fix the dataFrom the summary, it should be immediately clear that there's a problem with one of the fields.The *Petal.Length* column has at least 1 extreme value, that is probably a mistake. Weneed to decide what to do with this.In practice, if you encounter a wierd value it by be possible to figure out how/why it came about andmaybe even fix or remeasure it. As an example, if you have a column in a clinical dataset forpulse oximeter readings with measurments in %'s in the expected range of 0-100, what if yousee a measurements of 0.98?In some cases, you can't fix the data, and have to instead treat it as a missing value. R hasa special value for this called `NA`, which stands for *Not Available*. All the built inoperations in R understand this special value and will return results accordingly.Let's assign an `NA` to the wierd value, then run our summaries again. To do this, first we have tofind the index of the row with the bad entry, then we have to update the data.frame with the`NA` value. For this dataset, we'll make the executive decision to call any measurement less than 0 abad value.```{r}ix = myiris$Petal.Length <0myiris$Petal.Length[ix] =NAsummary(myiris)```Let's also try our previous min/max functions on this newly updated column to see what happens```{r}min(myiris$Petal.Length)max(myiris$Petal.Length)```What happens if you try to add `NA` to a number?You should always be careful computing a mathematical operations on a vector that contains `NA`'s. Most times, especially with summary stats like min,max,mean,etc., the best option is to just ignore the `NA`'s and use only the legitimate values. Allof the built-in R functions have the capablity of ignoring `NA` values, and you canenable this option by adding a `, na.rm = T` parameter argument to the function call.This `na.rm` stands for *NA remove*, and will ignore that observation completely.```{r}min(myiris$Petal.Length, na.rm=T)max(myiris$Petal.Length, na.rm=T)```## Save the dataOnce we've made modifications to the original dataset, it's a goodidea to save it in a new file in case we need to access it again orshare it with collaborators. For this example, editing a single line of a spreadsheet is quick and doesn't really need to besaved in a separate file, but most real world datasets involves datacleanup that takes time and effort, sometimes more than is spent on the actual analysis. You need to keep track of all the changes you maketo your data and an .R script is a great way to record all the steps that weretaken.saving the file as a comma separate value spreadsheet can be done with the built in`write.csv()` function```{r}#this one will add a column for the rownames by defaultwrite.csv(file ="myiris_modified.csv", x = myiris)#this one will ignore rownameswrite.csv(file ="myiris_modified2.csv", x = myiris, row.names=F)```## View the dataSummaries like the built-in `summary()` function are good at gettinga quick look at your data, but they don't always tell you the whole picture.A plot can sometimes give a better understanding to what's going on than a bunch of numbers can.The most common ways to plot numerical data and look for associations is with histograms, boxplots and scatterplots and these are all availablein R using built-in functions: `hist()`, `boxplot()`, `plot()`.Let's see what these look like on the first 4 columns of the dataset.```{r}hist(myiris[,1])boxplot(myiris[,2])plot(x=myiris[,3], y=myiris[,4])```We're going to go into the details of these functions in the 2nd week of class, but thisis a good place to pause show off some of the benefits of using AI to help withour coding.A boxplot of the *Sepal.Width*'s that separates each group by Species can be made with the followingcommand.```{r}boxplot(myiris[,2] ~ myiris[,5])```If you have no or little experience with R, it's very unlikely that you'll be familiarwith this command, or recognize the `formula` notation with the *~* that's being used in the function call.Luckily, chatbots like chatGPT have improved so much in the last few years, that youcan use them to help with almost all your coding problems. For this example, I want you toask a chatbot of your choice to make you a boxplot of the 2nd column of your datasetsplit by the 5th column. ```{}I have a .csv file named "myiris.csv". I want you to load this in, and make a boxplot of the 2nd column, split by the 5th column. Give me the base R code that does this.```This is the response I got back using this exact prompt into chatGPT. ```{r}# Load the csv filedf <-read.csv("myiris.csv")# Make a boxplot of the 2nd column, grouped by the 5th columnboxplot(df[[2]] ~ df[[5]],xlab =names(df)[5],ylab =names(df)[2],main =paste("Boxplot of", names(df)[2], "by", names(df)[5]))```If you understand the basics of function calling in R, it shouldn't betoo much a stretch to figure out what's going on here. The `boxplot()`has some additional parameters that we haven't gone over yet, named*xlab* and *ylab* that stand for thelabels of the X and Y axes of the plot respectively. What's going on with the last parameter *main* is a bit trickier to understand. *main* is the title of the plot and it willshow up in bold letters at the top of the figure. The `paste()` functionis a way to combine parts of character strings to form new ones. I know this becauseI have many years of experience with R, but how can you figure this out?Ask the chatbot.```{}what does main = paste("Boxplot of", names(df)[2], "by", names(df)[5]))mean in your last response?```You should get a very clear line by line explanation that tells you exactlywhat's going on. And if things are still not clear, ask again!```{}I don't understand, can you explain it in simpler terms?```Asking a chatgpt is not a cheat or a hack. It's a way to help you learn the material,get ideas about the capabilities of R, and to work with the tricky syntax required forprogramming (in any language).Every time you want to make a plot, process a file, or run a statistical test, it's ok to startby asking a chatbot for help.The key here is to understand that you're not relying on the chatbot to do your work.You using it to help you figure out how to do the work youself.
