Effect size measurement allows researchers to quantify their effects and assess the extent to which they are meaningful1. However, effect size is often undervalued in behavioral research—including relationship science—with some going as far as to say it is unimportant for theory testing2. Given the applicability of relationship science’s findings to potential intervention development, we find this phenomenon concerning. Recently, some relationships researchers have called for a “credibility revolution” within the discipline3. We assert that proper reporting and interpreting of effect size should be a part of this movement.
In this study, we will manually code published articles from Journal of Social and Personal Relationships (JSPR) and Personal Relationships (PR) to examine the extent to which relationships research has reported and interpreted effect size in recent years (2020-2024). Furthermore, we will examine historical trends in effect size reporting since the early 2000s. Although previous work has coded relationship science publications for other concepts related to the Open Science movement such as preregistration3, this will be the first study to quantify the extent to which relationships researchers utilize effect size estimates.
This study aims to answer the following research questions:
To what extent do relationship scientists report measures of effect size for the primary effect(s) of interest within their JSPR and PR publications? Furthermore, which standardized measures of effect size are typically reported (e.g., Cohen’s d, r, standardized beta)?
To what extent do researchers interpret effect size as a part of their published work in JSPR and PR?
Given that effect size has been a topic of interest in the behavioral sciences for decades4,5, have trends in effect size reporting within JSPR and PR changes over the last 25 years (2000-2024)?
To begin to answer our research questions, we will manually code the articles within randomly selected JSPR and PR issues from the following years: 2024, 2023, 2022, 2021, 2020, 2015, 2010, 2005, and 2000. For each year, we will randomly select two issues of each journal using code written in R (see Analysis). Due to potential variations in topic and scope, we will not code special issues as a part of this project (these will be removed prior to random selection). After selecting journal issues for coding, we will screen each article to identify theoretical papers, review papers, meta-analyses, and other articles which do not include firsthand, empirical data. These articles will not be coded further. Additionally, we will eliminate articles which contain solely qualitative work, as these articles are not relevant to our current research aims.
For each quantitative and mixed-methods article, we will identify the primary statistical effect(s) of interest from the abstract. This means that if the authors focused their abstract on the results of a non-hypothesized, exploratory analysis, we would still treat this as the primary effect of interest for the article. In the event that a paper contains multiple studies, we will assess each study which contains statistical tests of the identified primary effect(s) of interest. After identifying the primary statistical effect(s) of interest, we will code whether each article reported at least one standardized measure of effect size for this effect/these effects and note which measure(s) are reported. In the event that a paper contains Bayesian or other non-null hypothesis significance testing analyses, we will note whether effect size is reported in some capacity (e.g., Bayes factor). Lastly, we will code whether each article contained an interpretation of their effect size(s) as a part of their results and/or discussion section, meaning that the authors addressed the meaningfulness and/or relative importance of their effects beyond their initial reporting. This will be a binary variable.
Although not central to our research questions, we will also code (1) whether each article contained preregistered work and (2) whether each article contained dyadic analyses. We may report these codes and their relations to effect size reporting and interpretation as a part of our descriptive analyses.
Articles will be coded in a Google Sheet (see here for format). Each article will be coded by the first author, with a subset of 20% double-coded by the second author or an undergraduate research assistant. We will assess reliability using percent agreement with a threshold of 80%. If agreement falls below this threshold, all coders will meet and develop a plan to address discrepancies and difficulties in coding.
To randomly select the issues coded, we will use the following code (note that this code has not been run with this particular random seed prior to the posting of this preregistration):
## loading the necessary package
library(dplyr)
## setting our random seed - my cat's birthday
set.seed(815)
## loading our prepared datasets listing all issues from each year of interest
PR <- read.csv("/Users/suddu013/Desktop/Research/EFFSIZE/Coding/PR.csv")
JSPR <- read.csv("/Users/suddu013/Desktop/Research/EFFSIZE/Coding/JSPR.csv")
head(PR)## Journal Year Issue Month SpecialIssue
## 1 PR 2024 1 March N
## 2 PR 2024 2 June N
## 3 PR 2024 3 September N
## 4 PR 2024 4 December N
## 5 PR 2023 1 March Y
## 6 PR 2023 2 June Nhead(JSPR)## Journal Year Issue Month SpecialIssue
## 1 JSPR 2024 1 January N
## 2 JSPR 2024 2 February Y
## 3 JSPR 2024 3 March Y
## 4 JSPR 2024 4 April N
## 5 JSPR 2024 5 May Y
## 6 JSPR 2024 6 June N## setting the years we want issues from
years <- c(2024, 2023, 2022, 2021, 2020, 2015, 2010, 2005, 2000)
## writing a function to randomly select our issues
pick.issues <- function(data, years) {
data %>%
filter(Year %in% years & SpecialIssue == "N") %>%
group_by(Year) %>%
slice_sample(n = 2, replace = FALSE) %>%
ungroup()
}
## viewing our final PR issues to code
PR.sample <- pick.issues(PR, years)
PR.sample## # A tibble: 18 × 5
## Journal Year Issue Month SpecialIssue
## <chr> <int> <int> <chr> <chr>
## 1 PR 2000 2 June N
## 2 PR 2000 1 March N
## 3 PR 2005 4 December N
## 4 PR 2005 2 June N
## 5 PR 2010 4 December N
## 6 PR 2010 2 June N
## 7 PR 2015 3 September N
## 8 PR 2015 2 June N
## 9 PR 2020 4 December N
## 10 PR 2020 2 June N
## 11 PR 2021 4 December N
## 12 PR 2021 2 June N
## 13 PR 2022 4 December N
## 14 PR 2022 3 September N
## 15 PR 2023 4 December N
## 16 PR 2023 3 September N
## 17 PR 2024 2 June N
## 18 PR 2024 3 September N## viewing our final JSPR issues to code
JSPR.sample <- pick.issues(JSPR, years)
JSPR.sample## # A tibble: 18 × 5
## Journal Year Issue Month SpecialIssue
## <chr> <int> <chr> <chr> <chr>
## 1 JSPR 2000 1 February N
## 2 JSPR 2000 3 June N
## 3 JSPR 2005 5 October N
## 4 JSPR 2005 1 February N
## 5 JSPR 2010 1 February N
## 6 JSPR 2010 3 May N
## 7 JSPR 2015 2 March N
## 8 JSPR 2015 3 May N
## 9 JSPR 2020 3 March N
## 10 JSPR 2020 4 April N
## 11 JSPR 2021 8 August N
## 12 JSPR 2021 3 March N
## 13 JSPR 2022 12 December N
## 14 JSPR 2022 2 February N
## 15 JSPR 2023 10 October N
## 16 JSPR 2023 8 August N
## 17 JSPR 2024 12 December N
## 18 JSPR 2024 9 September NOur research questions will be analyzed as follows:
1. To what extent do relationship scientists report measures of effect size for the primary effect(s) of interest within their JSPR and PR publications? Furthermore, which standardized measures of effect size are typically reported?
We will compute the overall percentage of articles in the last five years (2020-2024) that reported standardized effect size coefficients. Furthermore, we will generate a descriptive table showcasing the different measures of effect size seen across both journals, along with individual tables for each journal.
2. To what extent do researchers interpret effect size as a part of their published work in JSPR and PR?
We will compute the overall percentage of articles in the last five years (2020-2024) that interpreted the size(s) of their effect(s).
3. Given that effect size has been a topic of interest in the behavioral sciences for decades, have trends in effect size reporting within JSPR and PR changes over the last 25 years?
To examine trends in effect size reporting since 2000, we will generate plots of the percentage of studies that reported and interpreted effect size over time. Furthermore, we will generate a table displaying these rates by year.