Introduction

This guide will help you manage your data effectively in lab experiments, ensuring all necessary libraries are installed and loaded, and your working directories are correctly set. The following analysis is done in RStudio, but you can easily adapt these practiices to your language of choice.

Setting Up the Environment

Clearing the workspace at the beginning of an analysis ensures that no residual data or variables from previous sessions interfere with your current work. This step is crucial for reproducibility, as it guarantees that your script’s output is consistent and solely dependent on the input data and the script itself, not on any leftover variables.

Clear workspace and remove all existing variables

rm(list=ls(all=T))

Load Libraries

Loading all required libraries at the start consolidates all dependencies in one place. This makes the script self-contained and easier to run on different machines or by different researchers. It also helps in identifying any missing packages early in the workflow.

# List of required packages
list.of.packages = c("data.table", "dplyr", "R.utils", "reshape", "reshape2", 
                     "stringr", "chron", "zoo", "RSQLite", "dplyr", "ggplot2", "rstudioapi", 
                     "data.table", "stringr", "chron", "zoo", "sendmailR", "cowplot", "RCurl", 
                     "tidyr", "lubridate", "collapse", "readr", "scales", "cowplot", "knitr", 
                     "googlesheets4", "DBI", "RPostgres", "odbc", "RJDBC", "DBI", "dplyr", "tidyr")


# Install and load new packages if running from RStudio
run_from_rstudio = TRUE

if (run_from_rstudio) {
  new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
  if(length(new.packages)) {install.packages(new.packages)}}

# Load installed packages
invisible(lapply(list.of.packages, require, character.only = TRUE))

Loading and Organizing Experimental Data

Systematically loading and organizing data, such as through structured folder paths and standardized naming conventions, facilitates easy access and manipulation. This structured approach aids in tracking different versions of data sets and ensures that the analysis is performed on the correct data set.

Start by naming your experiment

experiment = "Example qPCR Experiment"

Define File Locations

Before we start processing the data, it’s essential to know where our files are. This involves setting up a specific folder for raw data and identifying the files that will be used in the analysis. I try to do as little modification to files as possible outside of R. For example, I always copy and paste the raw file from a qPCR machine directly into the folder and edit downstream. This generally means less work in the long run, because you can just run the same chunk of code over and over again on different experiemnts.

When working from Rstudio, the parent working directory can be easily set to the location where the analysis script is saved. All folders will also be saved in this location (i.e. 2023-12-30 Example Experiment Analysis Folder below).

I always name my experiemnts with the following: YYYY-MM-DD Experimental Details, which is easiest for sorting and searching, and save the raw data in a “Data” Folder within the experiment folder.

Set the current working directory

Setting a specific working directory for each experiment ensures that all files related to that experiment are stored in a designated place. This organization is vital for managing multiple experiments simultaneously, as it prevents data from different experiments from getting mixed up and makes it easier to locate and reference data for each specific experiment, and use the same naming consistencies from experiment to experiment without loading the wrong data.

current_working_dir <- dirname(rstudioapi::getActiveDocumentContext()$path)
setwd(current_working_dir)

Confirm you saved your script in the right experiment folder

cat(current_working_dir)

## /Users/emmachory/Library/CloudStorage/Box-Box/ChoryLab/Experimental_Data_Repository/Emma_Robot_Experiments/2023-12-30 Example Experiment Analysis

Define variables for your raw data files (useful if you want to loop through many similar files)

# Setting the directory for where you've stored your raw data (a sub folder within your experimental folder)
file_location = "Data/"

# Listing all files in the current working directory, including experimental summaries, repositories, etc. 
experiment_files = list.files()

# Listing data files specifically in the 'Raw Data' folder
data_files = list.files(file_location)

Create Directories for Exported Data and Figures

Below, the file_location variable points to a dedicated folder for raw data, ensuring all initial, unprocessed data files are kept separate and unchanged. Automatically listing files using list.files() reduces manual errors and makes the script adaptable to various data sets.

To maintain an organized workflow, we create separate directories for storing processed data and figures. This keeps our project structured and makes it easier to locate specific outputs.

# Directory for exported, processed data
export_location = "Output Data/"

# Directory for storing figures
figure_location = "Figures/"

# Automatically create these directories if they don't exist. I tend to *not* create the folders in advance, in case I type something wrong, which makes it easier to repurpose code.
dir.create(export_location)
dir.create(figure_location)

Loading Reference Data

Loading something like a standard curve (that is used regularly between experiments) directly from a Google Sheets document ensures the use of the most up-to-date and centrally maintained version of the data. The googlesheets4 package makes this incredibly easy to do, and you can authenticate your access to google directly in Rstudio.

Import data from google drive

sheet_file = "https://docs.google.com/spreadsheets/d/1pX0yOutSFAcyyrPT6WlEK8SSc4dz_JsBpt08IRLWCXU/edit?usp=sharing"
standard_curves = googlesheets4::read_sheet(sheet_file,
                                            sheet = 'Sheet1')

View the reference data you’ve imported

kable(standard_curves)

Name	Intercept	Slope	Rsquared	Fwd	Rev	Fwd_seq	Rev_seq	Cutoff
phage qPCR-1	41.80512	-3.279375	0.9987223	Ep282	Ep283	TGTACTCTGTTTCGCGCTTG	GGCTACAGAGGCTTTGAGGA	10000
phage qPCR-2	42.77788	-3.387000	0.9997027	Ep284	Ep285	TGTTCTCGATGAGTGCGGTA	TTCTGTCCAGACGACGACAA	10000
phage qPCR-3	42.64402	-3.278244	0.9983784	Ep286	Ep287	GCTACAGCACCAGATTCAGC	AGGAAGCCCGAAAGACTTCA	10000
phage qPCR-4	41.40271	-3.208220	0.9597559	Ep288	Ep289	TGCGTGATGGACAGACTCTT	AGCTAAACAGGAGGCCGATT	10000

# Keep important columns
standard_curves = standard_curves %>% select(Name, Intercept, Slope, Cutoff) 

kable(standard_curves)

Name	Intercept	Slope	Cutoff
phage qPCR-1	41.80512	-3.279375	10000
phage qPCR-2	42.77788	-3.387000	10000
phage qPCR-3	42.64402	-3.278244	10000
phage qPCR-4	41.40271	-3.208220	10000

Create and Load an Experimental Manifest

For each experiment, I create an experimental manifest file contains crucial metadata for each well, including strain, media, inducers, concentrations, etc. It’s essential to load this manifest to link experimental data with these metadata.

Loading the manifest file

manifest <- fread(paste0("", experiment_files[grep(" Manifest.csv", experiment_files)]))

Your manifest file can contain as much data as you’d like, but I’d recommend keeping your headings and details as simple as possible to make it possible to use from experiment to experiment. For example, instead of making once column called “IPTG” and then the concentrations in the column, I make an “Inducer1” column, and a “Conc1” column. That way you can mix and match the various inducers etc, without changing the column titles (see below).

I use a manifest for EVERY experiment, no matter how small. It serves as a second laboratory notebook record. This data frame also makes it very easy to merge your experimental details with the data that comes from any plate reader, robot experiment, etc. I also use a standard naming convention (i.e. I always end the file name in Manifest.csv, which makes it easy to find the file without needing to change the name in the script).

print(head(manifest))

##    Well Experiment Strain Media              Inducer Conc Plasmid   Phage
## 1:   A1      T7 AP  S2060   DRM Trimethoprim (ng/mL)  100   T7 AP T7 RNAP
## 2:   B1      T7 AP  S2060   DRM Trimethoprim (ng/mL)  100   T7 AP T7 RNAP
## 3:   C1      T7 AP  S2060   DRM Trimethoprim (ng/mL)  100   T7 AP T7 RNAP
## 4:   D1      T7 AP  S2060   DRM    Aztreonam (ng/mL)  100   T7 AP T7 RNAP
## 5:   E1      T7 AP  S2060   DRM    Aztreonam (ng/mL)  100   T7 AP T7 RNAP
## 6:   F1      T7 AP  S2060   DRM    Aztreonam (ng/mL)  100   T7 AP T7 RNAP

Load and process your data

As an example below, qPCR data is loaded and processed to calculate the phage titers. This involves merging with standard curves, adjusting for dilutions, and converting Ct values to numeric, etc.

Import your data

data <- fread(paste0(file_location, data_files[grep("qPCR results.csv", data_files)]))
data$Ct <- as.numeric(data$Ct)

print(head(data))

##    Plate_num         Name Time Well     Ct Dilution
## 1:         1 phage qPCR-2    8   A1 29.262        1
## 2:         1 phage qPCR-2    8   A2 29.288        1
## 3:         1 phage qPCR-2    8   A3 29.974        1
## 4:         1 phage qPCR-2    8   A4 29.114        1
## 5:         1 phage qPCR-2    8   A5 29.707        1
## 6:         1 phage qPCR-2    8   A6 28.993        1

Merging with standard curves and calculating PFU/mL

If your columns are named the same, it’s easy to append all the details from a reference file to your data file with the merge function. In this case, the column in common is “Name” which is what qPCR primer pair is being used:

data <- merge(data, standard_curves)
data <- data %>% mutate(PFUmL = 10^((Ct - Intercept)/Slope)) %>% data.frame()

print(head(data))

##           Name Plate_num Time Well     Ct Dilution Intercept  Slope Cutoff
## 1 phage qPCR-2         1    8   A1 29.262        1  42.77788 -3.387  10000
## 2 phage qPCR-2         1    8   A2 29.288        1  42.77788 -3.387  10000
## 3 phage qPCR-2         1    8   A3 29.974        1  42.77788 -3.387  10000
## 4 phage qPCR-2         1    8   A4 29.114        1  42.77788 -3.387  10000
## 5 phage qPCR-2         1    8   A5 29.707        1  42.77788 -3.387  10000
## 6 phage qPCR-2         1    8   A6 28.993        1  42.77788 -3.387  10000
##       PFUmL
## 1  9783.972
## 2  9612.554
## 3  6029.755
## 4 10819.612
## 5  7229.870
## 6 11747.258

Merge Data with manifest file

Next, since I added a “Well” ID to my manifest file, I can simply merge the qPCR results with my experimental details:

data <- merge(data, manifest)

print(head(data))

##   Well         Name Plate_num Time     Ct Dilution Intercept  Slope Cutoff
## 1   A1 phage qPCR-2         1    8 29.262        1  42.77788 -3.387  10000
## 2  A11 phage qPCR-2         1    8 29.702        1  42.77788 -3.387  10000
## 3   A3 phage qPCR-2         1    8 29.974        1  42.77788 -3.387  10000
## 4   A5 phage qPCR-2         1    8 29.707        1  42.77788 -3.387  10000
## 5   A7 phage qPCR-2         1    8 29.534        1  42.77788 -3.387  10000
## 6   A9 phage qPCR-2         1    8 29.654        1  42.77788 -3.387  10000
##      PFUmL Experiment Strain Media               Inducer Conc Plasmid   Phage
## 1 9783.972      T7 AP  S2060   DRM  Trimethoprim (ng/mL)  100   T7 AP T7 RNAP
## 2 7254.487      T7 AP  S2133   DRM D-cycloserine (ng/mL)    1   T7 AP T7 RNAP
## 3 6029.755      T7 AP  S2069   DRM  Trimethoprim (ng/mL)   10   T7 AP T7 RNAP
## 4 7229.870      T7 AP  S2085   DRM  Trimethoprim (ng/mL)    1   T7 AP T7 RNAP
## 5 8132.202      T7 AP  S2101   DRM D-cycloserine (ng/mL)  100   T7 AP T7 RNAP
## 6 7495.120      T7 AP  S2117   DRM D-cycloserine (ng/mL)   10   T7 AP T7 RNAP

Exporting Processed Data

Exporting processed data at each stage of the analysis is a critical practice, especially in the context of scientific research and publishing. When preparing manuscripts for publication, many journals now require the submission of raw data alongside figures and results. By exporting data throughout the analysis process, you ensure that all relevant datasets are readily available and accurately reflect the analysis performed.

This approach saves significant time and effort, as it eliminates the need to retrospectively compile and export data when preparing for publication, which can be particularly challenging for experiments conducted years earlier. Moreover, naming these exported files in a way that correlates them with their corresponding figures or experimental conditions facilitates easier tracking and referencing during the manuscript preparation and peer review process.

Exporting Processed Data with Descriptive Naming

Below is the code to export processed qPCR data. The file is named to indicate the specific experiment and analysis date, which helps in linking the data file to the relevant figure or result section in a publication.

# File name includes experiment details and date for easy identification
export_file_name <- paste0(export_location, Sys.Date(), " ", experiment, " Analyzed Data.csv")

# Exporting processed qPCR data
write.csv(x = data, file = export_file_name)

Summarizing Data for Publication

To summarize the qPCR data, averages and standard deviations are calculated for each set of experimental conditions. Additionally, we emphasize the significance of exporting these summarized data for publishing purposes.

data_average <- data %>% 
  group_by(Name, Media, Inducer, Conc, Plasmid, Phage, Time) %>%
  filter(!is.na(PFUmL)) %>% 
  summarise(sd = sd(PFUmL, na.rm = TRUE), PFUmL = mean(PFUmL, na.rm = TRUE), n = n())

## `summarise()` has grouped output by 'Name', 'Media', 'Inducer', 'Conc',
## 'Plasmid', 'Phage'. You can override using the `.groups` argument.

Summarizing data, such as calculating averages and standard deviations, is a crucial step in the analysis process. It provides a concise overview of the experimental results, making them more accessible and interpretable. Most (if not soon to be all) scientific journals require the submission both the unprocessed and summarized data. By exporting this summarized data systematically during the analysis, you ensure that the required datasets are readily available when you are ready to publish your findings. Many journals now also require plotting the raw data points on top of the averages.

Exporting Summarized Data

Below is the code to export the summarized qPCR data. The file name is structured to include experiment details and the analysis date, facilitating easy identification and linkage to publication materials.

# File name includes experiment details and date for easy identification
export_summary_file_name <- paste0(export_location, Sys.Date(), " ", experiment, " Summarized Data.csv")

# Exporting summarized qPCR data
write.csv(x = data_average, file = export_summary_file_name)

Visualizing Data and Exporting Figures

Visualizing the data throughout the analysis is essential for quality control checking, and identifying underlying trends or analysis strategies that you may not have immediately realized are possible. I export plots across every step of analysis. Creating figures that represent your data graphically is a powerful way to communicate your findings. When preparing your research for publication, it’s essential to export these figures as PDF files so they can be edited later in adobe illustrator, but it’s equally important to ensure that you can reproduce them accurately.

Furthermore, adopting a consistent naming convention for these exported figures, such as including experiment details and analysis date, simplifies the process of linking figures to their corresponding sections in your publication.

Generating plots with ggplot

# Create a ggplot object for visualizing qPCR data
qPCR_grid_plot <- ggplot(data, 
                        aes(x = factor(Conc),
                            y = log10(PFUmL),
                            color = log10(PFUmL),
                            group = Inducer)) + 
  geom_point() +  # Add data points to the plot
  geom_line(aes(group = interaction(Inducer, Conc)), alpha = 0.5) +  # Add lines connecting data points
  facet_wrap(Inducer ~ ., scales = "free") +  # Create facets for different Inducers
  ylab("Phage Titre (Average PFU/mL)") +  # Label the y-axis
  ylab(paste("Concentration", unique(data$Inducer))) +  # Add a custom y-axis label
  ggtitle(paste(experiment, "(Average PFU/mL), n =", max(data_average$n))) +  # Add a custom title that also shows the number of replicates
  theme_classic() +  # Use a classic theme for the plot
  theme(legend.position = "bottom") +  # Set legend position to bottom
  guides(fill = guide_colourbar(barwidth = 10, barheight = 1))  # Customize color bar legend


print(qPCR_grid_plot)

Exporting Plots as PDFs

The exported PDF file name includes the experiment details, such as the experiment name and the analysis date. This naming convention makes it straightforward to identify and associate the figure with the specific experiment, facilitating seamless integration into research papers or presentations. The experiment is definted at the beginning of the code, to make things easier.

Exporting with ggsave()

The ggsave() function is used to save the qPCR grid plot as a PDF file. This function allows you to specify the file name, dimensions (width and height), units (inches), and scale of the plot. In this example, the width is set to 8 inches, and the height is set to 5 inches, ensuring that the resulting PDF is of appropriate size for inclusion in documents. PDFs make it very easy to make changes to the font size, color, etc, later in illustrator.

out_file_name <- paste0(figure_location, Sys.Date(), " ", experiment, " qPCR Plot.pdf")
ggsave(out_file_name, qPCR_grid_plot, width = 8, height = 5, units = "in", scale = 1)

Conclusion

In this guide, we’ve explored best practices in experimental data management, from setting up the environment to systematically organizing data, and generating figures. These practices not only contribute to the reproducibility and transparency of your research but also align with the requirements of many scientific journals when it comes time to publish your findings.

By following these guidelines and adopting a systematic approach to data management, you can save time and effort in the long run. Exporting raw data, summarized data, and figures with descriptive naming conventions ensures that your research remains accessible and reproducible, even years after the initial analysis.

As you continue your work in the lab and embark on new experiments, remember that good data management is a fundamental aspect of robust scientific research. Whether it’s for your own reference, collaboration with colleagues, or sharing your discoveries with the scientific community, these practices will serve you well throughout your research journey.

Godspeed!

Author: Emma Chory Date: 30 December, 2023

Chory Lab’s Best Practicies in Experimental Data Management

Emma Chory

30 December, 2023