Model Critique

For this lab, you’ll be working with a group of other classmates, and each group will be assigned a lab from a previous week. Your goal is to critique the models (or analyses) present in the lab.

First, review the materials from the Lesson on Ethics and Epistemology (week 5?). This includes lecture slides, the lecture video, or the reading. You can use these as reference materials for this lab. You may even consider the reading for the week associated with the lab, or even supplementary research on the topic at hand (e.g., news outlets, historical articles, etc.).

For the lab your group has been assigned, consider issues with models, statistical improvements, interpretations, analyses, visualizations, etc. Use this notebook as a sandbox for trying out different code, and investigating the data from a different perspective. Take notes on all the issues you see, and possible solutions (even if you would need to request more data or resources to accomplish those solutions).

Share your model critique in this notebook as your data dive submission for the week.

As a start, think about the context of the lab and consider the following:

Analytical issues, such as model assumptions
Statistical improvements; what do we know now that we didn’t know then?
Are there better visualizations which could have been used?
Overcoming biases (existing or potential)
Possible risks or societal implications
Crucial issues which might not be measurable

Treat this exercise as if the analyses in your assigned lab (i.e., the one you are critiquing) were to be published, made available to the public in a press release, or used at some large company (e.g., for mpg data, imagine if Toyota used the conclusions to drive strategic decisions).

If you were unable to attend class, select a notes_*.Rmd file from a previous week (not including weeks 1 or 3), and complete the analysis above. Share your critique below.

Example

For example, in Week 11, we used the year built, square footage, elevation, and the number of bedrooms to determine the price of an apartment. A few questions you might ask are:

Is this a “good” selection of variables? What could we be missing, or are there potential biases inherent in the groups of apartments here?
Nowhere in the lab do we investigate the assumptions of a linear model. Is the relationship between the response (i.e., \(\log(\text{price})\)) and each of these variables linear? Are the error terms evenly distributed?
Is it possible that our conclusions are more appropriate for some group(s) of the data and not others?
What if assumptions are not met? What could happen to this model if it were deployed on a platform like Zillow?
Consider different evaluation metrics between models. What is a practical use for these values?

Week 7 Lab Model Critique:

Model Assumptions:
- One of the most important assumption we make with this kind of time related data is the independence between observations, we should make sure that there are no dependence between observations. Hence, we can go for time series analysis to find out things such as seasonality, trends and periodicity.
- Another assumption is that the revenue data is normally distributed, It’s very important to check whether the data is normally distributed or not when comparing the averages of two different populations.
Statistical Improvements:
- Advanced statistical techniques such as time series analysis could be used to account for any time related dependencies in the data.
- Bayesian methods could be used to incorporate prior knowledge or beliefs into the analysis, especially if there are subjective interpretations involved.
Ethical Considerations:
- We aren’t sure of how the data was collected and if the data provided is right or wrong. In this case it is very important to verify the source of data and how reliable the source is.
- We aren’t also sure of what procedures were used to create this dataset. There is no transparency in data engineering methods used in creating the dataset.
- It is very important to consider going through the official documentation of the data and to know the answers for all the above questions before analyzing it.
Biases:
- There might be biases in the data collection process, such as selection bias if certain types of advertisements are overrepresented in the dataset.
- We also need to look for representation bias to check all the different categories are being fairly represented.
Crucial Issues:
- It’s important to consider the limitations of the data. For example, the dataset might not capture all relevant factors influencing advertisement performance, such as market competition or consumer behavior.
- Additionally, the dataset does not include information on the content or creative elements of the advertisements, which could significantly impact their effectiveness.

Ways to Improve Model:

One way to improve the model and gain insights from the data using a simple visualization is by creating a comparative box plot of the daily revenue for each variation of advertisement. This visualization can help identify any differences in the distribution of revenue between the different advertisement variations.

library(tidyverse)

## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.4
## ✔ forcats   1.0.0     ✔ stringr   1.5.1
## ✔ ggplot2   3.4.4     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.0
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

library(ggplot2)

url_ <- "https://raw.githubusercontent.com/leontoddjohnson/datasets/main/data/marketing/marketing.csv"
data <- read_delim(url_, delim = ",")

## Rows: 40 Columns: 8
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## dbl (8): spend, clicks, impressions, display, transactions, revenue, ctr, co...
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

data$display <- factor(data$display)

# Create the box plot
ggplot(data, aes(x = display, y = revenue)) +
  geom_boxplot() +
  labs(x = "Advertisement Variation", y = "Daily Revenue", title = "Box Plot of Daily Revenue for Advertisement Variations") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

By looking at the above box plot we can see that when display = 1 there seems to be outliers, we need understand the context of these outliers in order to work on a model as an outlier as big as this will skew the mean of this population and this should be considered while creating a model.