Hypothesis - Data Sets will keep getting larger

In other words: Disks are always full. It is futile to try to get more disk space. Data expands to fill any void (Murphy’s Law).

Magnitudes of Storage Capacity for well known systems

• 1000^1 kB kilobyte <- Space Shuttle Computer Memory Storage
• 1000^2 MB megabyte <- IBM PC-XT Hard Disk 1984
• 1000^3 GB gigabyte <- Smart Phone Memory
• 1000^4 TB terabyte <- Personal Computer SSD Storage
• 1000^5 PB petabyte <- Server Storage
• 1000^6 EB exabyte <- Cloud Storage
• 1000^7 ZB zettabyte <- Global Information Storage Capacity
• 1000^8 YB yottabyte . . .

Large Complex Datasets

• We will review two packages to access research data remotely
  • Connect to NHANES datasets online
  • Read (as well as write) research data (CSV, XLS, RData) from (to) GitHub

Best practices taught in the class

• Re-use of code (functions, packages)
• Avoid creating unneccassary copies of data
• Collabrate with ease by simply sharing the code with others
• Staging of data processing ahead of analysis
  • Repeatability of results
  • Down stream performance improvements

Libraries used for this presentation:

• nhanesA (and survey to calculate weighted means)
• fst

Epidemiologic dataset can get very large and cumbersome to manipulate

Access NHANES dataset using

library(nhanesA)

DEMO_I includes Demographic Variables and Sample Weights - Year 2015-2016

nhanesDemo <- nhanes("DEMO_I")

Demonstrate the differences between population distributions between unweighted interview sample and the weighted interview sample using survey::svymean

Same Code for a different NHANES data requires the change of one letter

Access NHANES dataset using

library(nhanesA)

DEMO_H includes Demographic Variables and Sample Weights - Year 2013-2014

nhanesDemo <- nhanes("DEMO_H")

Demonstrate the differences between population distributions between unweighted interview sample and the weighted interview sample using survey::svymean

Flattening the curve before it flattens us: hospital critical care capacity limits and mortality from novel coronavirus (SARS-CoV2) cases in US counties. Branas et al. (2020)

The preprint article is based on simulation of COVID cases in the US with respect social mobility. The model predicts the ICU bed availability contingent upon various parameters, at county level granularity

Challenges:

• Demonstrate simulation results via two ggplot line-graphs side-by-side
• Present a user interface with parameters to visualize various what-if scenarios
• Receive 12 large CSV files from GitHub, representing simulation results every week
• Upload data to cloud for access from RShiny platform
• Read all data files for each user session
• Consider future growth of model (increase in the number and size of files to read)

Simulation Data

• Time to read CSV files and plot graphs much improved from first iteration 300 seconds
• Used pivot_long, pivot_wide and pipes for data staging efficiencies
• However online users are impatient, need instant gratification
• Splash screen and hour-glass to inform user about the load time
• Researched fast data serialization models
• Among several alternatives library(fst) provided highest performance and highest compression combinaton
• Seperated data staging from online application
• Created dedicated data processing program to run once a week (see Benchmark 1)
• Saved resulting dataframe in form of a RData file
• Read RData projections into the Shiny App dataframe, ready for graphing takes a lot less time (see Benchmark 2)

## [1] "Benchmark 1 (data staging process):"

## Time difference of 35.24 secs

## [1] "Benchmark 2 (read data frame for user interface):"

##    user  system elapsed 
##    1.17    0.38    1.66

Data Files

References