How Registered Reports Turn Bad Sciences Good?

Sau-Chin Chen, Tzu-Chi University & Erasmus University Rotterdam
2016/10/16

Take Home Messages

hypothetico-deductive model
Hypothetico-deductive model of the scientific method. by Chambers, Feredoes, Muthukumaraswamy, and Etchells(2014)

empirical cycle
By Empirical_Cycle.png: TesseUndDaan derivative work: Beao (Empirical_Cycle.png) [CC BY 3.0 (http://creativecommons.org/licenses/by/3.0)], via Wikimedia Commons

References

  1. Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. J. (2014). Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Environmental Science, 1, 4-17. doi
  2. Wagenmakers, E.-J., Wetzels, R., Borsboom, D., Maas, H. L. J. van der, & Kievit, R. A. (2012). An Agenda for Purely Confirmatory Research. Perspectives on Psychological Science, 7, 632-638. doi
  3. de Groot, A. D. (2014). The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychologica, 148C, 188-194. doi

class Sciences("Bad"):

hypothetico-deductive model
Bad factors hidden in Hypothetico-Deductive Model (Remake by COS, 2014)

def Statistical Power("Low"):

Estimated powers lower than true powers
Bakker, Hartgerink, Wicherts, and Maas (2016)

def p-hacking("significance"):


by xkcd(CC BY-NC 2.5)

def Effect Size("Publication bias", "Statistical Power"):

plot of chunk EPUB Nuijten et al. (2015) Figure 1 Remake

  • d = Expected effect size
  • D = Actual effect size because of publication bias
  • D - d = Relative bias on the effect size.
  • \[ Relative\ bias = \frac{1 - pub}{1 + pub \frac{\beta}{1 - \beta}} \]
  • Meta analysis on D will have a overesitmated effect size.

def Effect Size("Publication bias", "Statistical Power"):

  • Sample size is smaller, publication bias is bigger. sample size X publication bias
    Nuijten et al. (2015) Figure 2

def HARKing:

def Replication( ):

  • Direct Replication: Repeat the original experimental procedure as exact as possible.

  • Conceptual Replication: Repeat the test of a hypotheis or experimental results in use of different methods.

  • During 1900 to 2011, there are 1.6%(5,051 out of 321,411) of psychological papers mentioned replication(Makel, Plucker & Hegarty, 2012).

def Replication( ):

More conceptual replications than direct replications(Makel, Plucker & Hegarty, 2012) Replication Types

Why psycholgists do not like Replication?

empirical Cycle: From Exploratory to Confirmatory

empirical cycle
by Adrianus Dingeman de Groot (1956, 2014)

  • Exploratory(Generating Hypothesis): Observation -> Induction -> Deduction -> [1 or more testable hypothesis]
  • Confirmatory(Testing Hypothesis): [testable hypothesis] -> Testing -> Evaluation
  • New hypotheis could bring new findings.
  • Psychologisits have prefered Exploratory than Confirmatory.
  • Hypothetico-Deductive Model is for testing hypothesis rather than generating hypothesis.

Hypothetico-Deductive Model is for testing hypothesis rather than generating hypothesis.


Wagenmakers, Wetzels, Borsboom, Maas, and Kievit (2012) Figure 1

Why Sciences being Bad?

class Testing("Generating"):  
    def Statistical Power:  
        "no matter how low it is"
    def p value:
        "smaller is better"
    def Conclusion:
        if ture:
            "Take the words form my proposal"
        else:
            "Take the words from SPSS sheet"
    def Publication:
        "Paper is everything!"
    def Replication:
        "Inedible!"

Good Sciences need not only Exploratory Research but also Confirmatory Research

Solution: Registered Research(RR)

It has to be Confirmatory Research

Am I conducting a Confirmatory Research?

Organize your proposal in AsPredicted format.

RR workflow

RR workflow
Chambers, Feredoes, Muthukumaraswamy, and Etchells (2014) Figure 2

Why Sciences being Good?

class Testing("Protocol", "IPA"):  
    def Statistical Power("Protocol"):  
        pilots("data1", "data2",...)
    def significance("Protocol"):
        methods("between_group", "within_group", ...)
    def Conclusion("Protocol"):
        if match:
            Rule.1
        else:
            Rule.2
    def Data_Collection("Protocol"):
        if "IPA" is true:
            "Do as protocol"
    def Replication:
        if "IPA" is true:
            Open("materials", "analysis scripts", ...)

Registered Replication Research(RRR)

  • Aims to Capstone Studies
    • High citations
    • Lack of direct replications
    • Meta analysis shows publication bias
  • Multiple branches: APS RRR
    • Leading Lab manages the protocol
    • Collobrative Labs start to collect data after IPA
    • Leading Lab is going to finish Stage 2

Original Registered Research

Obstacles

  • No new findings
  • More writings
  • Fears to open

Suggestions

  • Change from the beginning: Restate PSY101, Stat101, …
  • Thinking your research project in the empirical cycle.
  • Apply RR workflow to the grant application, the production of dissertation, etc.


To be Continued…