The purpose of this document is to examine the File IO speed and file size of commonly used file IO functions. It includes base R (CSV, RDA, RDS), readr, data.table, and feather. The script was written and run on a iMac (3.2 Ghz 4 core 8 hyperthreaded i7 processor) with a Crucial MX 200 SSD and 16 GB of RAM.The codebase was borrowed directly from rmflight’s gist and edited for further comparison and testing. Many thanks to arun_sriniv for helping me get openMP working.
Zipping files when saving will save space, but not as much space as RDS/RDA+zipping does and comes at the cost of time. If you have disk space saving in the new feather format is the fastest. Writing to CSV with data.table is extremely fast now, but does come at the expense of file size. base::write.csv() produced files ~20% larger than data.table and readr::write_csv().
Feather was the fastest (on average) at reading the feather formatted data back into R, but it was also the most inconsistent. Second was the RDS format. As with writing, adding compression to cut file size down significally increased the time required to import. data.table was faster than readr, but neither were as fast as RDA/RDS files.
If you can spare some disk space and the speed of feather is useful to you, it seems like the way to go for now. If disk space is a premium and you don’t mind waiting a little longer then a compressed RDS/RDA file makes the most sense. If you want smaller size and portability, then compressing a csv makes a lot of sense, but is one of the slowest to read in. Base functions for reading/writing CSV files are clearly outpaced by these other methods.
## Session info -------------------------------------------------------------## setting value
## version R version 3.4.0 (2017-04-21)
## system x86_64, darwin15.6.0
## ui X11
## language (EN)
## collate en_US.UTF-8
## tz America/Los_Angeles
## date 2017-05-12## Packages -----------------------------------------------------------------## package * version date source
## assertthat 0.2.0 2017-04-11 CRAN (R 3.4.0)
## backports 1.0.5 2017-01-18 CRAN (R 3.4.0)
## base * 3.4.0 2017-04-21 local
## colorspace 1.3-2 2016-12-14 CRAN (R 3.4.0)
## compiler 3.4.0 2017-04-21 local
## crosstalk 1.0.0 2016-12-21 cran (@1.0.0)
## data.table * 1.10.5 2017-05-12 local
## datasets * 3.4.0 2017-04-21 local
## DBI 0.6-1 2017-04-01 cran (@0.6-1)
## devtools * 1.13.0 2017-05-08 CRAN (R 3.4.0)
## digest 0.6.12 2017-01-27 CRAN (R 3.4.0)
## dplyr 0.5.0 2016-06-24 cran (@0.5.0)
## evaluate 0.10 2016-10-11 CRAN (R 3.4.0)
## feather * 0.3.1 2017-05-12 Github (wesm/feather@8872294)
## ggplot2 * 2.2.1 2016-12-30 CRAN (R 3.4.0)
## graphics * 3.4.0 2017-04-21 local
## grDevices * 3.4.0 2017-04-21 local
## grid 3.4.0 2017-04-21 local
## gtable 0.2.0 2016-02-26 CRAN (R 3.4.0)
## hms 0.3 2016-11-22 cran (@0.3)
## htmltools 0.3.6 2017-04-28 CRAN (R 3.4.0)
## htmlwidgets 0.8 2016-11-09 cran (@0.8)
## httpuv 1.3.3 2015-08-04 cran (@1.3.3)
## httr 1.2.1 2016-07-03 CRAN (R 3.4.0)
## iotools * 0.1-12 2015-07-31 CRAN (R 3.4.0)
## jsonlite 1.4 2017-04-08 CRAN (R 3.4.0)
## knitr 1.15.1 2016-11-22 CRAN (R 3.4.0)
## labeling 0.3 2014-08-23 CRAN (R 3.4.0)
## lazyeval 0.2.0 2016-06-12 CRAN (R 3.4.0)
## magrittr 1.5 2014-11-22 CRAN (R 3.4.0)
## memoise 1.1.0 2017-04-21 CRAN (R 3.4.0)
## methods * 3.4.0 2017-04-21 local
## microbenchmark * 1.4-2.1 2015-11-25 CRAN (R 3.4.0)
## mime 0.5 2016-07-07 CRAN (R 3.4.0)
## munsell 0.4.3 2016-02-13 CRAN (R 3.4.0)
## plotly * 4.6.0.9000 2017-05-12 Github (ropensci/plotly@12cbb72)
## plyr 1.8.4 2016-06-08 CRAN (R 3.4.0)
## purrr 0.2.2 2016-06-18 cran (@0.2.2)
## R.methodsS3 * 1.7.1 2016-02-16 CRAN (R 3.4.0)
## R.oo * 1.21.0 2016-11-01 CRAN (R 3.4.0)
## R.utils * 2.5.0 2016-11-07 CRAN (R 3.4.0)
## R6 2.2.1 2017-05-10 cran (@2.2.1)
## Rcpp 0.12.10 2017-03-19 CRAN (R 3.4.0)
## readr * 1.1.0 2017-03-22 cran (@1.1.0)
## rmarkdown 1.5 2017-04-26 CRAN (R 3.4.0)
## rprojroot 1.2 2017-01-16 CRAN (R 3.4.0)
## scales 0.4.1 2016-11-09 CRAN (R 3.4.0)
## shiny 1.0.3 2017-04-26 cran (@1.0.3)
## stats * 3.4.0 2017-04-21 local
## stringi 1.1.5 2017-04-07 CRAN (R 3.4.0)
## stringr 1.2.0 2017-02-18 CRAN (R 3.4.0)
## tibble 1.3.0 2017-04-01 CRAN (R 3.4.0)
## tidyr 0.6.2 2017-05-04 cran (@0.6.2)
## tools 3.4.0 2017-04-21 local
## utils * 3.4.0 2017-04-21 local
## viridisLite 0.2.0 2017-03-24 cran (@0.2.0)
## withr 1.0.2 2016-06-20 CRAN (R 3.4.0)
## xtable 1.8-2 2016-02-05 CRAN (R 3.4.0)
## yaml 2.1.14 2016-11-12 CRAN (R 3.4.0)