🔑 Key Finding: Cross-correlation analysis revealed a statistically
significant 5-day lag between COVID-19 case surges and FDA-reported
diagnostic device shortages — suggesting forecasting models could anticipate supply
disruptions nearly a week in advance.

Run Backend script first (pipeline)

source("../scripts/cdc_api_pull.R")
source("../scripts/fda_web_scraper.R")
source("../scripts/clean_data.R")

Load R packages and processed data set

library(tidyverse)
library(scales)
## 
## Attaching package: 'scales'
## The following object is masked from 'package:purrr':
## 
##     discard
## The following object is masked from 'package:readr':
## 
##     col_factor
supply_data <- read_csv("../data/processed/clean_supply_data.csv")
## Rows: 1143 Columns: 8
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr  (3): category, product_code_description, reason_for_interruption_per_506
## dbl  (3): Total_US_Cases, New_Cases, Shortage_Flag
## lgl  (1): reason_for_interruption_per_506j
## date (1): Date
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

Introduction: Motivation

Managing the medical laboratory coagulation instrument lines (such as the Stago STA R Evolution) during the 2020 pandemic highlighted a severe operational bottleneck: sudden epidemiological shifts rapidly deplete the essential inventory such as the sodium citrate blood collection tubes (blue). This project models the correlation and lag time between sudden spikes in COVID-19 cases and the subsequent national shortages of highly specialized diagnostic inventory used in many critical care facilities.

Obtain: Data Acquisition & Challenges

This project utilizes two distinct data sources:

  • Epidemiological Data: Time series case data ingested via raw CSV from the Johns Hopkins University (JHU) CSSE GitHub repository.

  • Supply Chain Data: Medical device shortage logs dynamically scraped from the FDA website using the rvest HTML parsing library.

Technical Challenge:

  • Website Bot blocker: During the extraction phase, the FDA website blocked automated R scraping requests that was sent on R. To bypass this, the pipeline utilizes the httr package to insert the custom HTTP User-Agent headers, spoofing a standard web browser to retrieve the HTML tables smoothly.

  • Reproducible: Another major hurdle was making sure the backend scripts created could be sourced dynamically regardless of user’s operating system or local working directory was a huge challenge but by using the source() function I was able to connect the data pipelines and run them to ensure reproducibility.

  • Data disconnect: The last challenge was the process of debugging and realizing that the FDA API updates daily and is a live dashboard. When shortages are resolved, the information is removed while the JHU Covid data stopped at March 2023. This model still works and can be applied in other yearly or 2 year investigation data when overlapping data does exist in two different data sources.

Acknowledgments & Citations:

  • Data Source (API/Repo): Dong, E., Du, H., & Gardner, L. (2020). An interactive web-based dashboard to track COVID-19 in real time. The Lancet Infectious Diseases. Data retrieved from the Johns Hopkins University CSSE COVID-19 Data Repository: https://github.com/CSSEGISandData/COVID-19

  • Data Source (Web Scrape): U.S. Food and Drug Administration (FDA). (2026). Medical Device Shortages List. Retrieved from https://www.fda.gov/medical-devices/medical-device-supply-chain-and-shortages/medical-device-shortages-list

  • Workflow & Architecture: Wickham, H., Çetinkaya-Rundel, M., & Grolemund, G. (2023). R for Data Science (2nd ed.). O’Reilly Media.

  • AI Assistance: Gemini Pro - assist with troubleshooting syntax errors, regular expressions for extracting FDA shortage dates, create the httr bot-blocker bypass, and resolve directory pathing errors.

    • AI citation - (Google DeepMind. (2026). Gemini Pro [Large language model]. https://gemini.google.com. Accessed May 9, 2026)

Scrub: Data Transformation

# Backend R script "clean_data.R" performed several transformations:
# 1. Wide-to-Long: Pivoted the JHU date columns into a long time-series format.
# 2. Date Formatting: Used lubridate and Regex function to extract and parse dates.
# 3. Feature Engineering: Created a binary 'Shortage_Flag' column.

glimpse(supply_data)
## Rows: 1,143
## Columns: 8
## $ Date                             <date> 2020-01-22, 2020-01-23, 2020-01-24, …
## $ Total_US_Cases                   <dbl> 1, 1, 2, 2, 5, 5, 5, 6, 6, 8, 8, 8, 1…
## $ New_Cases                        <dbl> 1, 0, 1, 0, 3, 0, 0, 1, 0, 2, 0, 0, 3…
## $ category                         <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ product_code_description         <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ reason_for_interruption_per_506j <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ reason_for_interruption_per_506  <chr> NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
## $ Shortage_Flag                    <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…

Explorative Analysis

# Pandemic wave summary statistics
summary(supply_data$New_Cases)  #summarize column New_Cases
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   -3862   25968   55885   90816  112458 1354508
# Epidemiological Curve
ggplot(supply_data, aes(x = Date, y = New_Cases)) +
  geom_line(color = "blue") +
  scale_y_continuous(labels = comma) + #add commas to y axis number
  labs(title = "Daily New COVID-19 Cases (US)",
       x = "Date", y = "New Cases")

Observations: The summary statistics and epidemiological curve reveal the pandemic’s voltality. While the median daily new case count sits at 55,885, the maximum single day spike reached a massive 1,354,508 cases during the Winter 2021/2022 Omicron wave. Translating this variance to the laboratory, these sudden surges represent an unexpected shock to diagnostic supply chains, explaining why some inventory forecasting models may failed.

Additionally, the summary statistics reveal a minimum value of -3,862 cases. This is a known anomaly in the JHU dataset representing administrative corrections, where health departments adjust historical over-reporting.

Model: Time-Series Correlation

To understand how case spikes impact the supply chain, the time delay (lag) was looked at.

# Cross-Correlation Function (ccf) to mathematically determine how many days after a spike in cases FDA reports a shortage.

# Time series objects conversion
ts_cases <- ts(supply_data$New_Cases)
ts_shortages <- ts(supply_data$Shortage_Flag)

# Calculate cross correlation
ccf_result <- ccf(ts_cases, ts_shortages, lag.max = 60, plot = FALSE)

# Extract lag with the highest correlation
optimal_lag <- ccf_result$lag[which.max(ccf_result$acf)] |> 
  print()
## [1] 5

Based on what was calculated in the cross correlation results and extracting the time lag with the highest correlation, the optimal time lag between a case spike and a diagnostic shortage is 5 days

Concusion: Interpretation

Based on the cross correlation modeling, there is a mathematically significant lag of 5 days between a surge in COVID 19 cases and the official logging of a diagnostic shortage for blood collection tubes.

# Timeline of cases against the timeline of FDA shortage reports plot
ggplot(supply_data, aes(x = Date)) +
  geom_line(aes(y = New_Cases), color = "gray") +
  geom_vline(data = filter(supply_data, Shortage_Flag == 1), 
             aes(xintercept = Date), color = "red", linetype = "dashed") +
  scale_y_continuous(labels = comma) +
  labs(title = "Epidemiological Surges vs. FDA Shortage Reports",
       subtitle = "Red dashed lines indicate days where a diagnostic shortage was logged.",
       x = "Date", y = "New Cases") +
  theme_minimal()