MAS 261 - Lecture 4

Data Visualizations, Boxplots, Outliers

Author

Penelope Pooler Eisenbies

Published

September 1, 2024

Housekeeping

Today’s plan
- Review Question about Measures of Variability
- A few minutes for R Questions :magic_wand:
- More about Boxplots
  - Review of summary values and where they are shown
- Outliers
  - IQR - Inter-quartile Range
  - UL - Upper limit and LL Lower Limit to detect outliers
- In-class Exercises

R and RStudio

In this course we will use R and RStudio to understand statistical concepts.
You will access R and RStudio through Posit Cloud.
- Sign up for a Free Posit Cloud Account
I will post R/RStudio files on Posit Cloud that you can access in provided links.
I will also provide demo videos that show how to access files and complete exercises.
NOTE: The free Posit Cloud account is limited to 25 hours per month.
- I will demo how to download completed work so that you can use this allotment efficiently.
- For those who want to go further with R/RStudio:
  - After Test 1, I will provide videos on how to download the software (R/RStudio/Quarto) and lecture files to your computer.

Lecture 4 In-class Exercises - Q1

Session ID: MAS261f24

In Lecture 3 we discussed the measures of variability, including TSS, variance, standard deviation, CV, and range.

Recall that all of these measures, except range, are closely related and can be calculated from each other.

An online electronics store has a selection of 31 $(n=31)$ different headphones. The sample mean $(\overline{X})$ price is $97.

Recall that $SD = \sqrt{Var}$ and $CV = \frac{SD}{\overline{X}}$

Also recall that the R console (lower left pane) can be used like a calculator

If the variance in these prices is 1256, what is the coefficient of variation, CV? Round answer to two decimal places.

Boxplot Review

Annotated with Five Number Summary

Boxplot Review and More Information

Technical term is a Box and Whiskers Plot. This version shows where top whisker ends and circles above that.

Some new terms

Terminology

IQR - Interquartile Range $IQR = Q3 - Q1$
UL - Upper Limit $UL = Q3 + 1.5\times IQR$
LL - Lower Limit $LL = Q1 - 1.5\times IQR$

Notes

Values above Upper Limit are High outliers.
Values below Lower Limit are LOW outliers.
Not all datasets have high and/or low outliers.
Vermont and Maine data have no low outliers.
Lower Limit can be negative even if all data are positive.

Steps for Determining Outliers

It is useful, but not required, to examine a boxplot (or histogram) to examine the data distribution.
- We will introduce histograms at the end of this lecture.
- Data visualizations can indicate if there are high or low outliers present.
Find Q1 (25th Percentile) and Q3 (75th Percentile)
- In R these values are found using the summary command.
Calculate IQR, the Interquartile Range, $IQR = Q3 - Q1$
Calculate the LL, Lower Limit and UL, Upper Limit for determining outliers.
- $LL = Q1 - 1.5\times IQR$
- $UL = Q3 + 1.5\times IQR$
Examine values in sorted data to determine which values are
- HIGH Outliers, values above the UL
- Low Outliers, values below the LL

Boxplots - Lifetime Gross for All 3 Markets

Lecture 4 In-class Exercises - Q2

Session ID: MAS261f24

Based on the boxplots below, which market, domestic or foreign, has a higher median for the lifetime gross data?

A. Domestic

B Foreign

C. The median values for these two markets are approximately equal.

Lecture 4 In-class Exercises - Q3

Session ID: MAS261f24

Which statement(s) are true about all three movie gross markets?

A. There are no outliers in these data.

B. There are only low outliers in these data.

C. There are only high outliers in these data.

D. There are low and high outliers in these data.

Calculations to Determine Outliers - Domestic Data

NOTE: All saved calculations are enclosed in parentheses so they will ALSO be displayed.

Use summary command to find Q1 (25th Percentile or Quantile) and Q3 (75th Percentile or Quantile).

Code

```{r summary of domestic_gross_mil, echo=T}
summary(movie_gross$domestic_gross_mil)
```

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
   0.12  226.70  305.79  326.43  387.83  936.66       2

Calculate $IQR = Q3 - Q1$ for domestic data and save calculation

Code

```{r calculate and save iqr for dom, echo=T}
(iqr_dom = 387.83 - 226.7)
```

[1] 161.13

Calculate and save calculations for LL (Lower Limit) and UL (Upper Limit) for domestic data.

Code

```{r calculate and save ll and ul for dom, echo=T}
(ll_dom = 226.7 - 1.5*iqr_dom)
(ul_dom = 387.83 + 1.5*iqr_dom)
```

[1] -14.995
[1] 629.525

Final Step to determine outliers in Domestic Data

Examine domestic data to determine if there are

HIGH Outliers - values above the upper limit (UL = ul_dom)
LOW Outliers - values below the lower limit (LL = ll_dom)

Code

```{r compare ul to domestic data, echo=T}
ul_dom

movie_gross$domestic_gross_mil |> 
  sort(decreasing = T) |> head(15) |> cbind()
```

[1] 629.525
        [,1]
 [1,] 936.66
 [2,] 858.37
 [3,] 814.87
 [4,] 785.22
 [5,] 718.73
 [6,] 700.43
 [7,] 684.08
 [8,] 678.82
 [9,] 674.29
[10,] 653.41
[11,] 651.25
[12,] 636.24
[13,] 623.36
[14,] 620.18
[15,] 608.58

Code

```{r compare ll to domestic data, echo=T}
ll_dom

movie_gross$domestic_gross_mil |> 
  sort(decreasing = T) |> tail(15) |> cbind()
```

[1] -14.995
        [,1]
 [1,] 165.25
 [2,] 161.32
 [3,] 160.89
 [4,] 159.56
 [5,] 146.13
 [6,] 144.33
 [7,] 137.72
 [8,] 130.17
 [9,] 124.99
[10,]   5.97
[11,]   3.70
[12,]   2.72
[13,]   1.54
[14,]   0.34
[15,]   0.12

Lecture 4 In-class Exercises - Q4-Q6

Session ID: MAS261f24

Q4: How many LOW outliers are in the domestic gross data?
- Hint: This confirms what we saw in the boxplot data visualization.

Q5: How many HIGH outliers are in the domestic gross data?

Q6: What is the LOWEST value in the domestic gross data that is also a HIGH outlier?

Finding Outliers in the Foreign Gross Data

Instructions

Use the previous example and the summary command to find Q1 and Q3 for the foreign gross data.
Find the IQR, LL, and UL for the foreign gross data.
Examine the sorted foreign gross data to determine how many high outliers are present.

Lecture 4 In-class Exercises - Q7-Q10

Q7: What is the Inter-quartile range (IQR) of the foreign gross data?

Q8: What is the upper limit (UL) for the foreign gross data?

Q9: How many HIGH outliers are in the foreign gross data?

Q10: What is the LOWEST value in the foreign gross data that is also a HIGH outlier?

Another way to look at data - Histograms

Boxplots and the side-by-side boxplots are great for comparing the central tendancy and variability or two or more groups of data.
Another tool for examining the entire distribution of values is a histogram.

Looking Ahead

Next lecture we’re going to examine both categorical and quantitative data.
For categorical data, we’ll look at
- Frequency Tables and Terminology
- Bar Charts
- Pie Charts
For quantitative data we’ll talk more about histograms
- How are they created by the software
- How can we modify what the software is doing to better understand the data.
- What does the shape of the histogram tell us. For example, are the data
  - left-skewed?
  - right-skewed?
  - normally distributed?

Key Points from Today

More about Boxplots
Identifying outliers visually from a boxplot
Identifying outliers numerically
- Defining and calculating the Inter-quartile Range (IQR)
- Defining upper and lower limits for determining outliers
- Using lower and upper limits to identify outliers
Introduction to histograms

To submit an Engagement Question or Comment about material from Lecture 4: Submit it by midnight today (day of lecture).

--- title: "MAS 261 - Lecture 4" subtitle: "Data Visualizations, Boxplots, Outliers" author: "Penelope Pooler Eisenbies" date: last-modified toc: true toc-depth: 3 toc-location: left toc-title: "Table of Contents" toc-expand: 1 format: html: code-line-numbers: true code-fold: true code-tools: true execute: echo: fenced --- ## Housekeeping ```{r setup, echo=FALSE, include=F, message=F, warning=F} # this line specifies options for default options for all R Chunks knitr::opts_chunk$set(echo=F) # suppress scientific notation options(scipen=100) # install helper package that loads and installs other packages, if needed if (!require("pacman")) install.packages("pacman", repos = "http://lib.stat.cmu.edu/R/CRAN/") # install and load required packages pacman::p_load(pacman,tidyverse, magrittr, olsrr, shadowtext, mapproj, knitr, kableExtra, countrycode, usdata, maps, RColorBrewer, gridExtra, ggthemes, gt, mosaicData) # verify packages # p_loaded() ``` - Today's plan - Review Question about Measures of Variability - A few minutes for R Questions :magic_wand: - More about Boxplots - Review of summary values and where they are shown - Outliers - IQR - Inter-quartile Range - UL - Upper limit and LL Lower Limit to detect outliers - In-class Exercises ## R and RStudio - In this course we will use R and RStudio to understand statistical concepts. - You will access R and RStudio through **Posit Cloud**. - Sign up for a [Free Posit Cloud Account](https://posit.cloud/plans/free) - I will post R/RStudio files on Posit Cloud that you can access in provided links. - I will also provide demo videos that show how to access files and complete exercises. - NOTE: The free Posit Cloud account is limited to 25 hours per month. - I will demo how to download completed work so that you can use this allotment efficiently. - For those who want to go further with R/RStudio: - After Test 1, I will provide videos on how to download the software (R/RStudio/Quarto) and lecture files to your computer. ## Lecture 4 In-class Exercises - Q1 ***Session ID: MAS261f24*** In Lecture 3 we discussed the measures of variability, including TSS, variance, standard deviation, CV, and range. Recall that all of these measures, except range, are closely related and can be calculated from each other. An online electronics store has a selection of 31 $(n=31)$ different headphones. The sample mean $(\overline{X})$ price is $97. Recall that $SD = \sqrt{Var}$ and $CV = \frac{SD}{\overline{X}}$ Also recall that the R console (lower left pane) can be used like a calculator **If the variance in these prices is 1256, what is the coefficient of variation, CV? Round answer to two decimal places.** ## Boxplot Review **Annotated with Five Number Summary** ![](img/ME_VT_annotated.png){fig-align="center"} ## Boxplot Review and More Information Technical term is a Box and Whiskers Plot. This version shows where top whisker ends and circles above that. ![](img/ME_VT_annotated_outliers.png){fig-align="center"} ## Some new terms ::: columns ::: {.column width="35%"} ![](img/VT_annotated_outliers_defined.png) ::: ::: {.column width="65%"} ::: fragment **Terminology** ::: - IQR - Interquartile Range $IQR = Q3 - Q1$ - UL - Upper Limit $UL = Q3 + 1.5\times IQR$ - LL - Lower Limit $LL = Q1 - 1.5\times IQR$ ::: fragment **Notes** ::: - Values above Upper Limit are High outliers. - Values below Lower Limit are LOW outliers. - Not all datasets have high and/or low outliers. - Vermont and Maine data have no low outliers. - Lower Limit can be negative even if all data are positive. ::: ::: ## Steps for Determining Outliers 0. It is useful, but not required, to examine a boxplot (or histogram) to examine the data distribution. - We will introduce histograms at the end of this lecture. - Data visualizations can indicate if there are high or low outliers present. 1. Find Q1 (25th Percentile) and Q3 (75th Percentile) - In R these values are found using the `summary` command. 2. Calculate IQR, the Interquartile Range, $IQR = Q3 - Q1$ 3. Calculate the LL, Lower Limit and UL, Upper Limit for determining outliers. - $LL = Q1 - 1.5\times IQR$ - $UL = Q3 + 1.5\times IQR$ 4. Examine values in sorted data to determine which values are - HIGH Outliers, values above the UL - Low Outliers, values below the LL ## {} ### Boxplots - Lifetime Gross for All 3 Markets ```{r alltime best movies data mgmt, eval=F} movies <- read_csv("data/bom_alltime_worldwide_2024_raw.csv", show_col_types=F, skip=11) |> slice(1:200) |> mutate(rank = as.integer(Rank), year = as.integer(Year), title=Title, worldwide_gross = gsub("$","", `Worldwide Lifetime Gross`, fixed = T), worldwide_gross = gsub(",","", worldwide_gross, fixed = T) |> as.numeric(), domestic_gross = gsub("$","", `Domestic Lifetime Gross`, fixed = T), domestic_gross = gsub(",","", domestic_gross, fixed = T) |> as.numeric(), foreign_gross = gsub("$","", `Foreign Lifetime Gross`, fixed = T), foreign_gross = gsub(",","", foreign_gross, fixed = T) |> as.numeric(), worldwide_gross_mil = round(worldwide_gross/1000000,2) , domestic_gross_mil = round(domestic_gross/1000000,2), foreign_gross_mil = round(foreign_gross/1000000,2)) |> glimpse() |> select(rank, title, worldwide_gross_mil, domestic_gross_mil, foreign_gross_mil, year) movies |> write_csv("data/All_Time_Movies_20240901.csv") ``` ```{r import clean movie data} movie_gross <- read_csv("data/All_Time_Movies_20240901.csv", show_col_types = F) ``` ```{r create boxplots of foreign and domestic gross data, warning=F, eval=F} movie_long <- movie_gross |> pivot_longer(cols=worldwide_gross_mil:foreign_gross_mil, names_to = "market", values_to = "gross_mil") |> mutate(market = gsub("_gross_mil", "", market, fixed = T), market = str_to_title(market)) movie_gross_box <- movie_long |> ggplot() + geom_boxplot(aes(x=market,y=gross_mil, fill=market))+ theme_classic() + theme(legend.position = "none") + labs(y="Lifetime Gross ($mil)", x="", title="Top 200 Lifetime Gross Movie Totals") + theme(axis.text=element_text(size=10), axis.title=element_text(size=15), plot.title=element_text(size=15)) ggsave("img/movie_gross_box_2024.png", plot = movie_gross_box, width = 7, height = 4, dpi = 300) ``` ![](img/movie_gross_box_2024.png){fig-align="center"} ## Lecture 4 In-class Exercises - Q2 ***Session ID: MAS261f24*** **Based on the boxplots below, which market, domestic or foreign, has a higher median for the lifetime gross data?** A. Domestic B Foreign C. The median values for these two markets are approximately equal. ![](img/movie_gross_box_2024.png){fig-align="center"} ## Lecture 4 In-class Exercises - Q3 ***Session ID: MAS261f24*** **Which statement(s) are true about all three movie gross markets?** A. There are no outliers in these data. B. There are only low outliers in these data. C. There are only high outliers in these data. D. There are low and high outliers in these data. ## {} ### Calculations to Determine Outliers - Domestic Data **NOTE:** All saved calculations are enclosed in parentheses so they will ALSO be displayed. ::: {.fragment} 1. Use `summary` command to find Q1 (25th Percentile or Quantile) and Q3 (75th Percentile or Quantile). ::: ::: {.fragment} ```{r summary of domestic_gross_mil, echo=T} summary(movie_gross$domestic_gross_mil) ``` ::: ::: {.fragment} 2. Calculate $IQR = Q3 - Q1$ for domestic data and save calculation ::: ::: {.fragment} ```{r calculate and save iqr for dom, echo=T} (iqr_dom = 387.83 - 226.7) ``` ::: ::: {.fragment} 3. Calculate and save calculations for LL (Lower Limit) and UL (Upper Limit) for domestic data. ::: ::: {.fragment} ```{r calculate and save ll and ul for dom, echo=T} (ll_dom = 226.7 - 1.5*iqr_dom) (ul_dom = 387.83 + 1.5*iqr_dom) ``` ::: ## {} ### Final Step to determine outliers in Domestic Data Examine domestic data to determine if there are ::: {.fragment} - HIGH Outliers - values above the upper limit (UL = `ul_dom`) - LOW Outliers - values below the lower limit (LL = `ll_dom`) ::: :::: {.columns} ::: {.column width="48%"} ::: {.fragment} ```{r compare ul to domestic data, echo=T} ul_dom movie_gross$domestic_gross_mil |> sort(decreasing = T) |> head(15) |> cbind() ``` ::: ::: ::: {.column width="4%"} ::: ::: {.column width="48%"} ::: {.fragment} ```{r compare ll to domestic data, echo=T} ll_dom movie_gross$domestic_gross_mil |> sort(decreasing = T) |> tail(15) |> cbind() ``` ::: ::: ::: ## {} ### Lecture 4 In-class Exercises - Q4-Q6 ***Session ID: MAS261f24*** - **Q4: How many LOW outliers are in the domestic gross data?** - Hint: This confirms what we saw in the boxplot data visualization. - **Q5: How many HIGH outliers are in the domestic gross data?** - **Q6: What is the LOWEST value in the domestic gross data that is also a HIGH outlier?** ## Finding Outliers in the Foreign Gross Data ![](img/movie_gross_box_2024.png){fig-align="center"} ::: fragment **Instructions** ::: 1. Use the previous example and the `summary` command to find Q1 and Q3 for the foreign gross data. 2. Find the IQR, LL, and UL for the foreign gross data. 3. Examine the sorted foreign gross data to determine how many high outliers are present. ## {} ### Lecture 4 In-class Exercises - Q7-Q10 - **Q7: What is the Inter-quartile range (IQR) of the foreign gross data?** - **Q8: What is the upper limit (UL) for the foreign gross data?** - **Q9: How many HIGH outliers are in the foreign gross data?** - **Q10: What is the LOWEST value in the foreign gross data that is also a HIGH outlier?** ## Another way to look at data - Histograms - Boxplots and the side-by-side boxplots are great for comparing the central tendancy and variability or two or more groups of data. - Another tool for examining the entire distribution of values is a histogram. ::: fragment ```{r histogram of domestic gross, warning=F} # Define vertical lines and their labels vertical_lines <- data.frame( xintercepts = c(301.43, 318.49), labels = c("Median", "Mean"), Colors = c("blue", "red") ) (movie_gross |> filter(!is.na(domestic_gross_mil)) |> ggplot() + geom_histogram(aes(x=domestic_gross_mil), color="darkblue", fill="lightblue", bins = 25) + geom_vline(data=vertical_lines, aes(xintercept = xintercepts, color=Colors, label=labels), linetype="dashed", linewidth=1) + scale_color_manual(values = vertical_lines$Colors, labels=vertical_lines$labels) + theme_classic() + labs(x="Domestic Lifetime Gross ($mil)", y="Frequency", title="Distribution of Domestic Lifetime Gross", color="", linetype="") + theme(legend.position = "bottom")) ``` ::: ## Looking Ahead - Next lecture we're going to examine both categorical and quantitative data. - For categorical data, we'll look at - Frequency Tables and Terminology - Bar Charts - Pie Charts - For quantitative data we'll talk more about histograms - How are they created by the software - How can we modify what the software is doing to better understand the data. - **What does the shape of the histogram tell us. For example, are the data** - left-skewed? - right-skewed? - normally distributed? ## {background-image="img/tired_panda_faded.png"} ### Key Points from Today - More about Boxplots - Identifying outliers visually from a boxplot - Identifying outliers numerically - Defining and calculating the Inter-quartile Range (IQR) - Defining upper and lower limits for determining outliers - Using lower and upper limits to identify outliers - Introduction to histograms ::: fragment **To submit an Engagement Question or Comment about material from Lecture 4:** Submit it by midnight today (day of lecture). :::