• The motivations for reproducibility
• How reproducible is Regional Science?
• Preliminary results
• Measures to improve reproducibility

• Many powerful arguments put forward in medicine, physical sciences and health
• But how do these relate to the discipline of Regional Science?
• Do we even need reprodicibility?
• We'll consider 6 reasons

• In April 2013, a mistake in an Excel spreadsheet was found.
• This mistake meant that an influential paper (Reinhart and Rogof, 2010) contained faulty results

"The intellectual edifice of austerity economics rests largely on two academic papers that were seized on by policy makers, without ever having been properly vetted" (Krugman)

• Reinhart and Rogof deserve credit for publishing methods and admitting mistake.

• Millions of the world's poorest people lack access to data
• Decisions usually made behind closed doors using unknown methods
• The 'black box' economic evaluation of new roads is a good example
• Open source software and data remove barriers to entry
• Bridges digitial divide and prevents academic elitism

• Reproducibility taken seriously in the physical sciences
• Recent debate in Psychology sparked by a Special Issue (Pashler et al., 2012) asking if there is a 'crisis of confidence' in the field
• Reproducibility especially critical in areas with human consequences e.g. epidemiology:

The replication of important findings by multiple independent investigators is fundamental to the accumulation of scientific evidence (Peng et al., 2006)

• Regional Scientists pride their 'scientific' credentials
• Method based on premises:

Reproducibility is a good thing

Reproducibility -> falsifiability -> scientific credebility

• The hypotheses:

H0: Most Regional Science research papers contain reproducible results.

H1: Most Regional Science research lacks reproducibility

• Different ways to test reproducibility
• Most ambitious: re-do experiment and analysis (Open Science Collaboration, 2012)
• Use of established criteria for reproducibility (Peng et al., 2006)
• Select sample of papers representative of Regional Science and test Peng's criteria

• Peng et al. (2006) provided simple and clear criteria in epidemiology
• Minimum standards for data, methods, documentation and distribution
• These standards were used as the basis to assess the reproducibility of papers on a scale of 1 to 4:

Inadequate description of data and methods

Basic description of datasets and outline of methodology

Detailed description of data and methodology

Provision of sample data and code/procedure enabling reproduction of results

• Related issue of accessibility (available/unavailable) to public scrutiny also recorded

• Only empirical (based on analysis of data) papers published in leading regional science journals selected:
• Google Scholar search term: "regional science" data between 2000 and 2010 in the following journals:

Papers in Regional Science

Regional Science and Urban Economics

Journal of Regional Science

Regional Studies

• Most cited articles analysed (most influential)
• Plans to expand sampling (ideas?)

A weighted total was applied to the data scores for each paper:

Total <- Rating + Availability + 2 * Software

• Yu and Wei (2008) describe software and specific packages used
• Cited for methods, not just findings (benefit of reproducibility)
• Still far from fully reproducible: no code or data
• Full version available online

• Steps needed to improve reproducibility are simple and relatively easy to implement
• Reproducibility is being advocated on 3 levels:

Individual authors

Academia

Civil society and funding councils

3 levels recommended for Regional Scientists:

• Minimum standard: clear method used including software and links to data source
• Sample procedure and data: sample datasets and 'code snippets' provided
• Completely reproducible: e.g. everything needed to reproduce results available in a documented zip file

Lovelace and Ballas (2013)

• Koole and Lakens (2012) recommend parallel publication of 'replications' to incentivise reproducibility and testing findings
• PLOSONE and some other journals insist on publication of code and data source
• Move towards interactive graphics encourages reproducibility

"PLOS journals require authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception." (plosone.org)

• Rapid uptake of open source software
• Open data movements making it easier to share datasets
• Funding councils increasingly demand data outputs
• Want methods to be of use to others

"Data are the main assets of economic and social research. We recognise publicly-funded research data as valuable, long-term resources that, where practical, must be made available for secondary scientific research." (ESRC.ac.uk)

• Reproducibility is important for Regional Science
• Regional Scientists tend to describe data and methods well (by social science standards) but much room for improvement
• The steps needed to make your work more reproducible are easy to implement and may make your work more citeable and useful to others
• This will help Regional Science grow as a trustworthy, robust and scientific discipline