dbverse:
composable database libraries for larger-than-memory scientific analytics

Motivation

Current challenges with scientific data analysis

dbverse overview

{dbverse} adopts familiar syntax

Example with {dbMatrix}

rownames(dgc)[1:5]

[1] "Gna12"  "Ccnd2"  "Btbd17" "Sox9"   "Sez6"  

colnames(dgc)[1:5]

[1] "AAAGGGATGTAGCAAG-1" "AAATGGCATGTCTTGT-1" "AAATGGTCAATGTGCC-1"
[4] "AAATTAACGGGTAGCT-1" "AACAACTGGTAGTTGC-1"

dim(dgc)

[1] 634 624

Loaded dbMatrix

rownames(dbMatrix)[1:5]

[1] "Gna12"  "Ccnd2"  "Btbd17" "Sox9"   "Sez6"  

colnames(dbMatrix)[1:5]

[1] "AAAGGGATGTAGCAAG-1" "AAATGGCATGTCTTGT-1" "AAATGGTCAATGTGCC-1"
[4] "AAATTAACGGGTAGCT-1" "AACAACTGGTAGTTGC-1"

dim(dgc)

[1] 634 624

How does it work? dbMatrix example

dbMatrix adopts familiar {Matrix} syntax…

scaled <- dbMatrix[,"cell_1"] * 10

scaled <- dplyr::tbl(con, "dbMatrix") |>
  dplyr::select(cell_id = "cell_1") |>
  dplyr::mutate(scaled = expression * 10)

SELECT cell_id, expression * 10 AS scaled
FROM my_cells.db WEHRE cell_id = 'cell_1';

Illustrative {dbMatrix} benchmark

{dbMatrix} performs larger-than-memory sparse matrix operations and outperforms HDF5Matrix

norm_mat <- t(t(dbMatrix) / libsizes) * scalefactor
lib_norm_mat <- log(norm_mat + offset) / log(base)
log_norm_mat <- t(norm_mat) - colMeans(lib_norm_mat)
# ...additional matrix operations

Illustrative {dbSpatial} benchmark

{dbSptaial} outperforms existing in-memory methods for spatial intersections

Task: find the intersection between cell polygons in tissue region of interests (ROIs)

Illustrative {dbSequence} benchmark

{dbSequence} outperforms competing methods

Task: filter reads in a genomic *.bam file (28GB, 285e6 reads)

• samtools v1.20 (Li et al. 2009)
• q01: chromosome region
• q02: q01 + flag
• q03: q02 + CIGAR string (samtools + awk)
  

Conclusions

Advantages of using DuckDB for scientific data analysis

Limitations and future directions

Acknowledgements

Questions?