• What is reproducible research?
• Why do we care?
• Why reproducibility questions arise?
• The cost of reproducibility
• Reproducibility and statistics
• Current status of reproducibility
• What can we do?

• Science is the systematic enterprise of gathering knowledge about the universe and organizing and condensing that knowledge into testable laws and theories

• The success and credibility of science are anchored in the willingness of scientists to expose their ideas and results to independent testing and replication by other scientists.

Reproducible research is the ultimate standard for strengthening scientific evidence by independent:

• Investigators
• Data
• Analytical methods
• Laboratories
• Instruments

• Galileo’s notes directly integrated his data (drawings of Jupiter and its moons), key metadata (timing of each observation, weather, and telescope properties), and text (methods, analysis, and conclusions)

• High-throughput biology generates volumes of data

• Data-generating technologies are increasingly used to make clinical recommendations and treatment decisions

• A problem may be overlooked .. Published .. Get in clinical trials

• A total of 162 co-authors
• 40 papers
• Two-thirds are partially or completely retracted

• Data/metadata used to develop test should be made publicly available

• The computer code and fully specified computational procedures used for development of the candidate omics-based test should be made sustainably available

• "… the computer code … will encompass all of the steps of computational analysis, including all data preprocessing steps … All aspects of the analysis need to be transparently reported"

• Humans are good at recognizing patterns

• Our intuition about patterns in high dimensional data quickly drops with the increased dimensionality of the data
• We rely on computation to uncover patterns
• P-values, the ‘gold standard’ of statistical validity, are not as reliable as many researchers assume.

1. Remember that the more hypotheses which are tested and the less selection which goes into choosing hypotheses - the more likely it is that you are looking at noise

2. Bigger samples are better

3. Small effects are to be distrusted

4. Multiple sources and types of evidence are desirable

5. Evaluate literatures not individual papers

6. Trust empirical papers which test other people's theories more than empirical papers which test the author's theory

7. As an editor or referee, don't reject papers that fail to reject the null

• Large-scale collaborative research

• Adoption of replication culture

• Registration (of studies, protocols, analysis codes, datasets, raw data, and results)

• Sharing (of data, protocols, materials, software)

• More appropriate statistical methods

• Standardization of definitions and analyses

• More stringent thresholds for claiming "breakthroughs"

• …

1. P-values can indicate how incompatible the data are with a specified statistical model

2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone

3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold

4. Proper inference requires full reporting and transparency

5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result

6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis

• Empirical reproducibility

• Computational reproducibility

• Statistical reproducibility

• Empirical reproducibility

• Computational reproducibility

• Statistical reproducibility

The most important is the mindset, when starting, that the end product will be reproducible.

The most important is the mindset, when starting, that the end product will be reproducible.

• Experimental design
• Data generation
• Data analysis
• Results interpretation
• Dissemination of results

Problems

• With point-and-click, there’s no way to record/save the steps that generated the (copy/pasted) results

• Data files are kept separately from the analysis code, and from reports

• After modifications of one of the files, it becomes unclear which version corresponds exactly to the reported results

• Every time something changes, you have to regenerate the figures/results/reports by hand – very time consuming

• The report is automated via code

• Data is attached to the well-documented code

• History of any changes should be preserved

• The final report should be self-sufficient and reproducible with a single command

• Fully automated report
• Including code and data

• Unix is a family of operating systems and environments that exploits the power of linguistic abstractions to perform tasks

• Unix users spend a lot of time at the command line

• In Unix, a command is worth a thousand mouse clicks

• Make is a tool which controls the workflow of generating target/result files from the dependencies/source files

• Automates/documents a workflow

• Intelligently handles the dependencies among data files, code

• Accounts for the updates in data, code

• R – free/open source programming language

• Runs on Windows, Mac, and Linux

• Extensible with a very large collection of actively developing packages

• Excellent graphics & report-creating capabilities

• A report containing a stream of text and code chunks

• Each code chunk loads data, computes results, shows figures

• Each text chunk explains how the code chunks work

• The resulting report is human- and machine readable

• HTML - HyperText Markup Language, used to create web pages. Developed in 1993

• LaTeX – a typesetting system for production of technical/scientific documentation, PDF output. Developed in 1994

• Sweave – a tool that allows embedding of the R code in LaTeX documents, PDF output. Developed in 2002

• reStructuredText - a markup syntax that plays well with Python and Sphinx documentation. Developed in 2002

• Markdown – a lightweight markup language for plain text formatting syntax. Easily converted to HTML. Developed in 2004

• knitR - a package for dynamic report generation written in R Markdown

• Supports RMarkdown, LaTeX, MathJax. PDF, HTML, DOCX output

• Developed in 2012 by Yihui Xie

• Code chunks separated by ```. Inline code allowed

• Graphics, code generated and external images

• Tables, code generated and in RMarkdown format

• Caching long code chunks

• Code chunks and results output fully customizable

• Mix markdown with code

Git and GitHub – version control system

• Each project stored in its own repository

• Keep history of changes – track what you did

• Ability to go back if something breaks

• Branch out, go creative, then merge or revert the changes

• Collaborate through merging changes from multiple people

• Git is a command line tool
• GitHub.com is a web-based storage for your project repositories

• Subversion, https://subversion.apache.org/
• Mercurial, https://www.mercurial-scm.org/

• Git is a command line tool
• GitHub.com is a web-based storage for your project repositories

• Git add – add a file to version control system
• Git commit – make a snapshot of current changes
• Git push/pull – send/get changes to/from GitHub

IPython/Jupyter Notebooks

• Combine text, equations, code, and graphics
• Markdown support
• Multiple languages support (>40)

Research environments

• Galaxy, https://usegalaxy.org/

• GenePattern, http://www.broadinstitute.org/cancer/software/genepattern/

• Taverna, https://taverna.incubator.apache.org/introduction/

Docker - an envelope (or container) for the whole project environment ~ lightweight virtual machine

• OS-independent, portable application images
• Preserves all application dependencies
• Easy to distribute

• Are the tables and figures reproducible from the code and data?

• Does the code actually do what you think it does?

• In addition to what was done, is it clear why it was done? (e.g., how parameter settings were chosen?)

• Is your code scalable to accommodate more data/methods?

1. The results cannot be reproduced

2. The results cannot seem to be reproduced

3. Reproducibility requires extreme effort

4. Reproducibility requires considerable effort

5. Easy reproducibility, but require some proprietary source packages (MATLAB, SAS, etc.)

6. The results can be easily reproduced by an independent researcher with at most 15 min of user effort, requiring only standard, freely available tools (C compiler, R, Python, etc.)

• Begin with the final product in mind

• Use literate programming (self-documenting code)

• Keep history of changes via code versioning and sharing

• Get basic statistics right

• Set stringent cutoffs, correct p-values for multiple testing

• Be critical, consider batch effects, visualize, do sanity checks, use random controls, cross-validation

• Follow reporting guidelines

• BIOS 692 “Reproducible Research Tools”

• https://mdozmorov.github.io/BIOS692/

• References https://mdozmorov.github.io/BIOS692/pages/references.html