Goal 1: Business Scenario

Customer or Audience

The primary audience is ShopFast’s marketing team, specifically:

  • Digital Ad Managers (who allocate budgets)

  • Data Analysts (who interpret A/B test results)

  • UX Designers (who optimize ad creatives)

Problem Statement

To maximize return on ad spend, ShopFast must determine whether replacing its standard text-based ads (Version A) with display ads (Version B) will increase revenue per impression without disproportionately increasing costs. This decision must be made within 4 weeks to align with the Q3 marketing budget cycle.

Scope

Variables from the lab data to address the problem:

  • revenue (to measure sales impact)

  • clicks and impressions (to calculate CTR)

  • spend (to evaluate cost efficiency)

Key Analyses:

  1. Two-sample t-test for revenue difference between ad versions.

  2. Chi-squared test for CTR independence.

  3. Paired t-test for daily profit (revenue - spend).

Assumptions:

  • Daily ad performances are independent (no seasonality).

  • Click-through rates are binomially distributed.

  • Ad placement and audience demographics are consistent.

Objective

Determine with 95% confidence whether Version B (display ads) yields statistically higher revenue per impression or CTR compared to Version A, while maintaining a profit margin ≥15%.

Success Criteria:

  • Reject the null hypothesis (no difference) for revenue or CTR with p < 0.05.

  • Cohen’s d ≥ 0.5 (medium effect size).

  • Net profit margin ≥15% in Version B.

Goal 2: Model Critique

Issues with Current Lab Analyses

  • Ignoring Temporal Trends

    • The lab assumes daily data points are independent, but ad performance may vary by weekday (e.g., weekends could have higher engagement).

    • Solution: Add a weekday variable and use ANCOVA to adjust for time effects.

#marketing$weekday <- weekdays(as.Date(marketing$date))  
#anova(lm(revenue ~ display + weekday, data = marketing))

  • Multiple Testing Without Adjustment

    • The lab runs separate tests for revenue, CTR, and profit without correcting for family-wise error rates.

    • Solution: Apply Bonferroni correction.

#p_values <- c(0.01, 0.04, 0.03)  # Example p-values from tests  
#p.adjust(p_values, method = "bonferroni")

  • Profitability Not Integrated

    • Revenue and CTR are tested in isolation, but the business needs a combined ROI metric.

    • Solution: Use a profitability ratio ((revenue - spend) / spend).

#marketing$roi <- (marketing$revenue - marketing$spend) / marketing$spend  
#t.test(roi ~ display, data = marketing)

Proposed Improvements

  • Bayesian A/B Testing

    • Provides probabilistic interpretations (e.g., “Version B has an 80% chance of being better”).
#bayes_test <- bayesTest(marketing$revenue[marketing$display == 1],  
                       #marketing$revenue[marketing$display == 0],  
                       #priors = c("mu" = 100, "lambda" = 1),  
                       #distribution = "bernoulli")  
#summary(bayes_test)

  • Segmented Analysis

    • Test effects across user segments (e.g., new vs. returning visitors).
#t.test(revenue ~ segment + display, data = marketing)

  • Time-Series Visualization

    • Plot daily trends to identify anomalies or autocorrelation.
#ggplot(marketing, aes(x = date, y = revenue, color = factor(display))) +  
  #geom_line() +  
  #labs(title = "Daily Revenue by Ad Version")

Goal 3: Ethical and Epistemological Concerns

Ethical Concerns

  1. Privacy Risks

    • Display ads might use intrusive tracking (e.g., cookies). Mitigation: Anonymise user data.

  2. Dark Patterns

    • If Version B has deceptive design (e.g., fake urgency), it could harm trust. Mitigation: UX ethics review.

  3. Algorithmic Bias

    • Display ads might perform differently for demographic groups. Mitigation: Disaggregate analysis by age/gender.

Epistemological Concerns

  1. Causality Ambiguity

    • Higher CTR in Version B could stem from novelty bias (users react to newness, not ad quality).

  2. Generalizability

    • Results from a 2-week test may not apply to holiday seasons.

  3. Metric Myopia

    • Optimizing for short-term revenue might ignore long-term brand damage (e.g., ad fatigue).

Affected Stakeholders

  • Users: Could experience annoyance or privacy breaches.

  • Shareholders: Might prioritize short-term gains over sustainable growth.

  • Competitors: Aggressive ad strategies could trigger an “arms race.”