Model Critique

For this lab, you’ll be working with a group of other classmates, and each group will be assigned a lab from a previous week. Your goal is to critique the models (or analyses) present in the lab.

First, review the materials from the Lesson on Ethics and Epistemology (week 5?). This includes lecture slides, the lecture video, or the reading. You can use these as reference materials for this lab. You may even consider the reading for the week associated with the lab, or even supplementary research on the topic at hand (e.g., news outlets, historical articles, etc.).

For the lab your group has been assigned, consider issues with models, statistical improvements, interpretations, analyses, visualizations, etc. Use this notebook as a sandbox for trying out different code, and investigating the data from a different perspective. Take notes on all the issues you see, and possible solutions (even if you would need to request more data or resources to accomplish those solutions).

Share your model critique in this notebook as your data dive submission for the week.

As a start, think about the context of the lab and consider the following:

Treat this exercise as if the analyses in your assigned lab (i.e., the one you are critiquing) were to be published, made available to the public in a press release, or used at some large company (e.g., for mpg data, imagine if Toyota used the conclusions to drive strategic decisions).

If you were unable to attend class, select a notes_*.Rmd file from a previous week (not including weeks 1 or 3), and complete the analysis above. Share your critique below.

Example

For example, in Week 11, we used the year built, square footage, elevation, and the number of bedrooms to determine the price of an apartment. A few questions you might ask are:

  • Is this a “good” selection of variables? What could we be missing, or are there potential biases inherent in the groups of apartments here?
  • Nowhere in the lab do we investigate the assumptions of a linear model. Is the relationship between the response (i.e., \(\log(\text{price})\)) and each of these variables linear? Are the error terms evenly distributed?
  • Is it possible that our conclusions are more appropriate for some group(s) of the data and not others?
  • What if assumptions are not met? What could happen to this model if it were deployed on a platform like Zillow?
  • Consider different evaluation metrics between models. What is a practical use for these values?

Week 7 Lab Model Critique

Ethical Considerations:

  • Where was the data sourced from? Do users know their data is being collected and analyzed?
    • How do we know that we have enough data? What measures will be made if we need to collect more data?
  • What is being advertised? What are the means of this advertisement? What platforms are the ads being distributed on?
    • Is what we are marketing 18+?
    • Is it “good” for society or could it be used for harm?
    • Do we need to be careful about who our audience is? Is this a reason why we have a lack of data?
  • We do not have documentation for this data set to clearly state what features are being captured, what column names mean, etc.
    • This also means we are making a lot of assumptions, such as our data being independent.
  • There are a lot of outside factors that could be influencing this data we do not know about.
    • What time of the year is it? Are there any certain holidays or seasons that could be affecting marketing?
  • We do not know what their goals are. The lack of data and focus on revenue seems to make us think that they are focused on the short-term, but this is an assumption that we are making. Ultimately, we do not know what their goals are for using this data.
  • Are there “better” metrics that we could have focused on than just display and revenue? We found that CTR was more correlated to the display than the revenue.

Logistical/Statistical Considerations:

  • What processes were used for choosing significance and power levels? Who got to choose these and why?

  • Is data normal or is it skewed?

  • Data is very minimal, especially for digital marketing data, causing us to rely greatly on sampling.

    • Aside from just a lack of rows, the lack of columns also prevents us from making further conclusions and observations about the data (geographical data, demographics, device used to click on ad, platforms, timestamps, etc.)

  • Having more data would have helped the bootstrapping be more realistic. It is difficult to tell if a data point is actually significant or not.

  • Further hypothesis testing should have been done. Does the use of a display effect more than just revenue? Could it also impact clicks, conversion rate, impressions, etc.

Ways to Improve:

  • The Null Hypothesis of “Average daily revenue remains equal for both variations of advertisement.” can be improved. There is nothing time-related in the data set, yet the word “daily” is used. This is assuming each row of data is a day, but we do not know this for certain. To resolve this, we could either improve the documentation to tell us whether or not each row of data is supposed to represent a day, or we could change our wording.

    • We also believe that more hypothesis testing needing to be done to examine how the use of a display did or did not impact a certain row of data. Perhaps we also could have also used:

\[ H_0: \text{There is no difference in click-through-rate between both variations of advertisement.} \\ H_0: \text{The average of impressions for advertisements with and without a display are the same.} \]

  • On the “Advertisement Effect on Revenue” box plot, there is an outlier that we do not know anything about. With so little data, this could be an important point. Perhaps it would be useful to plot additional variables to learn more about this point.

  • A lot of our data is count data, we may possibly need to do some sort of transformation in order to analyze our data much further. Could we possibly have a poisson distribution? Note: Revenue being 0 could cause us a problem if we use a log transformation. We may possibly need to add 1 or small value in order to do log transformation.

log_marketing <- marketing |>
  select(spend, clicks, impressions, transactions, revenue, ctr, con_rate) |>
  log()

log_marketing$display <-marketing$display
log_marketing |>
  ggplot() +
  geom_boxplot(mapping =
                 aes(x = revenue,
                     y = factor(display, levels = c(0,1),
                                labels = c("Normal", "With Display")))) +
  labs(title = "Advertisement Effect on Revenue",
       x = "Revenue (in dollars)",
       y = "Advertisement Variation") +
  theme_minimal()