Data Management Tips

The 3 R's of Data Management:

• Redundancy
  • Backup your data safely!
  • Use GitHub to track changes to files.
• Reliability
  • Organize your files for efficient access
  • Never change your raw data directly
• Reproducibility
  • Metadata helps others learn about your data
  • Communicate edits to raw data via scripts
  • Provide public access (when applicable)

For important data / files, you need to have a backup of your data (at least one, but preferrably more)!

Example:

• Files in (reliably) organized system on personal computer
• File system synced automatically to cloud (BoxSync)
• File system synced periodically to physical device (external hard-drive)
• Scripts and data version control system (GitHub)

Separate file system into major professional events. Further breakdown by courses, research projects, extracurricular programs, personal files.

phd

• courses
  • stat_500
  • stat_510
• research
  • bmp
  • conformal
• datafewsion
• personal

Iowa State University provides free unlimited cloud storage via Box.com

A helpful tool to enable automatic backup of your data to Box.com, is to use BoxSync.

• Store folders / files you want to backup in the BoxSync folder
• As you add / edit files in these folders, changes are automatically backed up to Box.com

Follow the link below for more information on installing BoxSync on Windows or Mac computers:

https://support.box.com/hc/en-us/articles/360043697194-Installing-Box-Sync

Within your backed up personal file system, you will likely have data related to your research.

After you have a organized file system, you can worry about your research projects:

• Redundancy
  • Use GitHub to track changes to files.
• Reliability
  • Never change your raw data directly
• Reproducibility
  • Metadata helps others learn about your data
  • Communicate edits to raw data via scripts
  • Provide public access (when applicable)

GitHub is a web platform for tracking changes to your computer codes and scripts, that you should be using in combination with RStudio, whenever you work with data!

If you don't have an account: https://www.github.com

Once you have an account, add the free education pack through ISU: https://education.github.com/pack

• Free GitHub Pro while a student
  • Extra Storage
  • Create a free personal webpage (Example: https://labuzzetta.github.io)
  • Private repositories

Using GitHub and RStudio, you can create a data project which is:

• Redundant: Backed up and tracked via GitHub

• Reliable: Your raw data is never altered directly

• Reproducible: Other scientists can easily understand and access your work

Project Subdirectories:

• data-raw: Put the .csv and other data files from your experiments here NEVER TOUCH AGAIN

  • It is so important not to edit your raw data directly, even if the formatting is terrible.
  • We can write R scripts to transform the raw data and clean it up.
  • The cleaned / transformed data is saved in a new location.

• data: Cleaned data tables, with nicely formated column names, data types, outliers removed

• R: Metadata files which describe the cleaned data tables. Might include data units or collection methods

• vignettes: Scripts that generate your analyses / graphics

• presentation: Scripts to generate a presenetation with easy access to graphs / tables

Dr. Lisa Schulte-Moore's STRIPS (Science-based Trials of Rowcrops Integrated with Prairie Strips) Project is one group that has transitioned to using the R Data Package Format.

• Over 350,000 collected datapoints
• Over 20,000 quadrat estimations
• Nearly 2,500 nest checks

And that's just 2015-2016!

As an example, we will look into the challenges of storing and analyzing (fake) data similar to those collected by the STRIPS project and examine how the R Data Package format would be useful.

When we take a look back our raw data, there may be many inconsistencies that we'd like to change:

• Naming Standards
  • Filenames
  • Column names
• Data Types
  • Dates
  • Booleans vs. Yes/No
• Database Design
  • Should I store everything in one table?
  • Will my database design make it easy to analyze my data?

In the column names, look for:

• Inconsistency
• Spaces / Weird Characters

In the column names, look for:

• full words
• lower case with underscores

There are many ways to name files / columns with consistency:

• lowercase_underscore
• camelCaseNoSpaces
• CapitalizeEveryWord
• PLEASE_DONT_DO_THIS

Suggestion: lowercase_underscore

Important People in the “R” World suggest: http://r-pkgs.had.co.nz/style.html

There are so many ways to store dates:

• 2000-07-04 14:00:00
• 2000-07-04 2:00 PM
• 2000/07/04 02:00:00 PM
• 04/07/2000

Suggestion: When possible, YYYY-mm-dd or YYYY/mm/dd or YYYYmmdd, and hh:mm:ss or hhmmss

• Stores in chronological order
• Bypasses AM/PM Errors
• Reduces Month/Day confusion
• May be combined into a single column when analyzing raw data

Main objective: Choose one preferred format

Suggestion: If using R, look into the lubridate package

How/where should measurement units be stored?

• In raw data, note the unit as full words in the column name after a dash
• But we will want to remove these units from the polished data
• Put the units in documentation / metadata files
• Consider changing booleans to “Yes”/“No”
• Use appropriate data types (integers vs decimals) and check your abbreviations

What's wrong with the table below?

Again, when making an R Data Package, look at: https://github.com/jarad/RDataPackageTemplate

• Store raw data in /data-raw
• Clean raw data with R scripts
• Import clean data to R package
• Easily manipulate and analyze clean data

How to think about database design:

• A database holds info about objects, events, observations
• A Relational Database uses ID's to link entries together

A Simple Example:

Single Table Format:

Instead we could use a relational database format:

Fruits

Stores

Consider the following scenario:

• An organization wants to have a bird watching competition
• Competitors will register with:
  • Name, Birthday, Email
• Competitors will record where, when and which birds they see
  • List of locations (Name, Address, GPS Coordinates)
  • List of observations (Who, When, Where, Which Species, Observation details)
  • List of all species observed (Common Name, Species, Genus)

How could this data be managed?

Avoid using repetitive groups of columns

See dataManagement::sightings_really_bad for the full table.

Relational database design helps to:

• Reduce the amount of repeat data
• Helps standardize data tables
• Can reduce errors and NA entries
• Makes it easier to understand the metadata / documentation files

Look further at the following tables and documentation:

• dataManagement::observer
• dataManagement::location
• dataManagement::sighting
• dataManagement::species

To view documentation for a dataset:

• Use ? before the dataset name
• Example: ?dataManagement::sighting

https://github.com/labuzzetta/apsimo

The 3 R's of Data Management:

• Redundancy
  • Backup your data safely!
  • Use GitHub to track changes to files.
• Reliability
  • Organize your files for efficient access
  • Never change your raw data directly
• Reproducibility
  • Metadata helps others learn about your data
  • Communicate edits to raw data via scripts
  • Provide public access (when applicable)

• Organize your personal file system

• Backup your data and install BoxSync

• Git your GitHub account and Github Education Pack

• Store research data in R Data Packages

• Never touch your raw data again!

• Support reproducible research by sharing your repositories on GitHub

• Introduction to RStudio

• Introduction to GitHub

• Making an R Data Package

• Introduction to Tidyverse

• Using GitHub within RStudio

You can find this presentation and the associated data on my GitHub page:

https://github.com/labuzzetta/dataManagement

Instructions for installing the package from GitHub are listed on the webpage.

A link to an online version of the presentation alone is:

https://rpubs.com/clabuzze/datamanagement