articleID <- "2-2-2015_PS" # insert the article ID code here e.g., "10-3-2015_PS"
reportType <- 'final' # specify whether this is the 'pilot' report or 'final' report
pilotNames <- "Bria Long" # insert the pilot's name here e.g., "Tom Hardwicke". If there are multiple cpilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
copilotNames <- "Tom Hardwicke" # # insert the co-pilot's name here e.g., "Michael Frank". If there are multiple co-pilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
pilotTTC <- 350 # insert the pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
copilotTTC <- 240 # insert the co-pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
pilotStartDate <- as.Date("06/13/18", format = "%m/%d/%y") # insert the pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
copilotStartDate <- as.Date("03/14/18", format = "%m/%d/%y") # insert the co-pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
completionDate <- as.Date("10/21/19", format = "%m/%d/%y") # copilot insert the date of final report completion (after any necessary rounds of author assistance) in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")In this task, participants continuously indicated which of two foods they preferredusing their mouse; these foods varied in both healthiness and tastiness. The authors then conducted trajectory analysis using linear regressions for every subject to estimate the degree to which the trajectories where influenced by the food attributes at every time point; the coefficients were then used to assess the earliest time points at which the mouse trajectories were influenced by these attributes for each subject, and statistical tests were performed over these time points.
We note that the author’s code was written in Matlab 2014; we attempted to reproduce it in Matlab 2015/Matlab 2018. The following command was run in Matlab to assess the dependencies used: ## [fList,pList] = matlab.codetools.requiredFilesAndProducts(‘code.m’); The pilot/co-pilot found that the Mapping Toolbox, Matlab, and Statisitcs and Machine Learning toolbox were required.
The only other file that was required and not provided was a plotting tool (shadedErrorBar.m) which the pilot already had installed from the Matlab file exchange.
To address whether individual differences in dietary self control were associated with differences in the relative speed at which the healthfulness and tastiness attributes were computed, we began by comparing how taste and health information were reflected in the mouse trajectories for individuals with high versus low self-control, as defined using a median split of the SCSR statistic. As shown in Figure 5a, we found substantial differences across the groups. In particular, in the high-self-control group, the paths for tastiness and healthfulness were quite similar and became significantly greater than zero at approximately the same time (healthfulness: t = 67; tastiness: t = 60). In contrast, for the low-self-control group, the latency at which tastiness became significant was similar to that of the high-self-control group (t = 56), but the latency for healthfulness occurred much later (t = 85). As before, we tested for the significance of these differences by estimating at the individual level the earliest time at which healthfulness became significantly greater than zero and then comparing the distributions. The normalized time at which tastiness and healthfulness became significant did not differ significantly for high-self-control subjects (mean healthfulness: t = 69.50; mean tastiness: t = 68.54; mean difference = −2.9), t(9) = −0.46, p = .33 (one-tailed), but they were significantly different in the low-self-control group (mean healthfulness: t = 81.22; mean tastiness: t = 61.35; mean difference = −16), t(8) = −2.5, p = .02 (one-tailed).
# load packages
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
library(ReproReports) # custom report functions# Prepare report object. This will be updated automatically by the reproCheck function each time values are compared
reportObject <- data.frame(dummyRow = TRUE, reportedValue = NA, obtainedValue = NA, valueType = NA, percentageError = NA, comparisonOutcome = NA, eyeballCheck = NA)Commented out exclusions that were flagged as optional in the author’s code, made modified file author_code_optional_values_commented.m; reran code. Copy and pasted outcome sentences from matlab console, but did not directly compare values when saw that they also did not match.
PAPER: in the high-self-control group, the paths for tastiness and healthfulness were quite similar and became significantly greater than zero at approximately the same time (healthfulness: t = 67; tastiness: t = 60).
MATLAB OUTPUT (Code provided by authors): in the high self-control group, the paths for taste and health were quite similar and became significantly greater than zero at approximately the same time (health: t = 72; taste: t = 63)
MATLAB OUTPUT (Optional exclusions commented): in the high self-control group, the paths for taste and health were quite similar and became significantly greater than zero at approximately the same time (health: t = 72; taste: t = 66)
MATLAB OUTPUT (Optional exclusions commented): in the high-self-control group, the paths for tastiness and healthfulness were quite similar and became significantly greater than zero at approximately the same time (healthfulness: t = 67; tastiness: t = 60).
## Original provided code
## High self-control
# heathfulness
# reportObject <- reproCheck(reportedValue = '67', obtainedValue = 72, valueType = 'mean')
#tastiness
# reportObject <- reproCheck(reportedValue = '60', obtainedValue = 63, valueType = 'mean')
## Corrected code
reportObject <- reproCheck(reportedValue = '67', obtainedValue = 67, valueType = 'mean')## [1] "MATCH for mean. The reported value (67) and the obtained value (67) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."#tastiness
reportObject <- reproCheck(reportedValue = '60', obtainedValue = 60, valueType = 'mean')## [1] "MATCH for mean. The reported value (60) and the obtained value (60) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."PAPER: for the low-self-control group, the latency at which tastiness became significant was similar to that of the high-self-control group (t = 56), but the latency for healthfulness occurred much later (t = 85).
MATLAB OUTPUT (Code provided by authors): for the low self-control group, the latency at which taste became significant was similar to that of the high self-control group (t = 57), but that for health occurred much later (t = 85).
MATLAB OUTPUT (Optional exclusions commented): for the low self-control group, the latency at which taste became significant was similar to that of the high self-control group (t = 58), but that for health occurred much later (t = 88).
MATLAB OUTPUT (Corrected code): In contrast, for the low-self-control group, the latency at which tastiness became significant was similar to that of the high-self-control group (t = 56), but the latency for healthfulness occurred much later (t = 85).’
## Raw output (optional exclusions NOT commented)
## Low self-control
# heathfulness
# reportObject <- reproCheck(reportedValue = '56', obtainedValue = 57, valueType = 'mean')
#tastiness
# reportObject <- reproCheck(reportedValue = '85', obtainedValue = 85, valueType = 'mean')
## Corrected code
reportObject <- reproCheck(reportedValue = '56', obtainedValue = 56, valueType = 'mean')## [1] "MATCH for mean. The reported value (56) and the obtained value (56) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."#tastiness
reportObject <- reproCheck(reportedValue = '85', obtainedValue = 85, valueType = 'mean')## [1] "MATCH for mean. The reported value (85) and the obtained value (85) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."PAPER: The normalized time at which tastiness and healthfulness became significant did not differ significantly for high-self-control subjects (mean healthfulness: t = 69.50; mean tastiness: t = 68.54; mean difference = −2.9), t(9) = −0.46, p = .33 (one-tailed).
MATLAB OUTPUT (Code provided by authors): The normalized time at which taste and health became significant did not differ for high self-control subjects (mean health: t = 75.00; mean taste: t = 72.27; mean difference = -4.30; t(9)=-0.74, p=0.24, one-tailed)
MATLAB OUTPUT (Optional exclusions commented): The normalized time at which taste and health became significant did not differ for high self-control subjects (mean health: t = 74.30; mean taste: t = 69.00; mean difference = -6.80; t(9)=-1.67, p=0.06, one-tailed)
MATLAB OUTPUT (Corrected code): The normalized time at which tastiness and healthfulness became significant did not differ significantly for high-self-control subjects (mean healthfulness: t = 69.50; mean tastiness: t = 68.55; mean difference = -2.9, t(9) = -0.46, p = 0.33 (one-tailed)
## High self-control statistics
# # heathfulness
# reportObject <- reproCheck(reportedValue = '69.50', obtainedValue = 75.00, valueType = 'mean')
#
# # tastiness
# reportObject <- reproCheck(reportedValue = '68.54', obtainedValue = 72.27, valueType = 'mean')
#
# # t-test values
# reportObject <- reproCheck(reportedValue = '9', obtainedValue = 9, valueType = 'df')
# reportObject <- reproCheck(reportedValue = '-.46', obtainedValue = -.74, valueType = 't')
# reportObject <- reproCheck(reportedValue = '.33', obtainedValue = .24, valueType = 'p')
## Corrected code
# heathfulness
reportObject <- reproCheck(reportedValue = '69.50', obtainedValue = 69.50, valueType = 'mean')## [1] "MATCH for mean. The reported value (69.5) and the obtained value (69.5) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."# tastiness
reportObject <- reproCheck(reportedValue = '68.54', obtainedValue = 68.55, valueType = 'mean')## [1] "MINOR_ERROR for mean. The reported value (68.54) and the obtained value (68.55) differed by 0.01%. Note that the obtained value was rounded to 2 decimal places to match the reported value."# t-test values
reportObject <- reproCheck(reportedValue = '9', obtainedValue = 9, valueType = 'df')## [1] "MATCH for df. The reported value (9) and the obtained value (9) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-.46', obtainedValue = -.46, valueType = 't')## [1] "MATCH for t. The reported value (-0.46) and the obtained value (-0.46) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.33', obtainedValue = .33, valueType = 'p')## [1] "MATCH for p. The reported value (0.33) and the obtained value (0.33) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."PAPER: …but they were significantly different in the low-self-control group (mean healthfulness: t = 81.22; mean tastiness: t = 61.35; mean difference = −16), t(8) = −2.5, p = .02 (one-tailed).
MATLAB OUTPUT (Code provided by authors): but they were significantly different in the low self-control group (mean health: t = 82.38; mean taste: t = 63.65; mean difference = -12.50, t(7)=-2.06, p=0.04, one-tailed).
MATLAB OUPUT (Optional exclusions commented): but they were significantly different in the low self-control group (mean health: t = 84.50; mean taste: t = 67.35; mean difference = -15.70, t(9)=-2.84, p=0.01, one-tailed).
MATLAB OUTPUT (Corrected code): but they were significantly different in the low-self-control group (mean healthfulness: t = 81.22; mean tastiness: t = 61.35; mean difference = -16), t(8) = -2.5, p = 0.02 (one-tailed)
## High self-control statistics
# heathfulness
# reportObject <- reproCheck(reportedValue = '81.22', obtainedValue = 75.00, valueType = 'mean')
#
# # tastiness
# reportObject <- reproCheck(reportedValue = '61.35', obtainedValue = 72.27, valueType = 'mean')
#
# # t-test values
# reportObject <- reproCheck(reportedValue = '8', obtainedValue = 7, valueType = 'df')
# reportObject <- reproCheck(reportedValue = '-2.5', obtainedValue = -2.06, valueType = 't')
# reportObject <- reproCheck(reportedValue = '.02', obtainedValue = .04, valueType = 'p')
## Corrected code
reportObject <- reproCheck(reportedValue = '81.22', obtainedValue = 81.22, valueType = 'mean')## [1] "MATCH for mean. The reported value (81.22) and the obtained value (81.22) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."# tastiness
reportObject <- reproCheck(reportedValue = '61.35', obtainedValue = 61.35, valueType = 'mean')## [1] "MATCH for mean. The reported value (61.35) and the obtained value (61.35) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."# t-test values
reportObject <- reproCheck(reportedValue = '8', obtainedValue =8, valueType = 'df')## [1] "MATCH for df. The reported value (8) and the obtained value (8) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '-2.5', obtainedValue = -2.5, valueType = 't')## [1] "MATCH for t. The reported value (-2.5) and the obtained value (-2.5) differed by 0%. Note that the obtained value was rounded to 1 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.02', obtainedValue = .02, valueType = 'p')## [1] "MATCH for p. The reported value (0.02) and the obtained value (0.02) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."The authors provided extensive code to reproduce the analyses and figures in the manuscript. The code provided by the authors also outputted both the primary sentences and figures in the paper. However, these did not match the numbers provided in the targetOutcomes.md taken from the main manuscript. We noticed that there were some “optional” exclusions in the main analysis file. As such, we ran the code both when excluding these optional trials and when including them; both versions are reported below; neither matched the values in the main manuscript.
Most of the values did not match, with one exception, with a variety of both minor and major numerical errors. For example, we noticed that there were certian optional exclusion criterion that were included in the main analyses (marked with optional in code.m). However, commenting out these optional exclusion criterion did not allow us to reproduce the figures or values in the main paper; these values are also reported above in the report but not directly compared to the reported values in the paper for brevity.
Update: We contacted the authors to ask if they knew what the problem was. It turns out that they had already realised the original code they uploaded did not completely reproduce the reported findings, and had additionally discovered “a mistake in the data posted to OSF”. They had posted corrected data and code to the OSF (https://osf.io/mx92g/). Our team was not aware of this and it is not mentioned in the paper. The authors say that they had considered posting a corrigendum to the paper but decided not to pursue that because Psychological Science only publishes one if the error “significantly affect an article’s findings or conclusions or a reader’s understanding.” They “did not feel this met that threshold”.
A readme file accompanying the updated OSF materials cotnains further details about the original errors:
- The “Example analysis code” component uploaded to OSF 10/17/2014 (https://osf.io/7gvh9/) was not sufficiently documented and hard to use in replicating the published results. The code presented here is a commented version of the code used to produce the reported analyses, and therefore identically replications the results from the paper.
- The data file with the mouse traces (sullivanEtAl2014ForPub_mouse.csv) contained several errors, leading to another source of discrepancy from the paper’s reported statistics. The mouse tracking data uploaded in this component (newData.csv) is the original raw data.
- This code reports that the percent of trials excluded from analysis is 8.1%, but the paper reports that it was 13.3%. This updated 8.1% figure is correct, and the reported number in the paper is a mistake."
We are now able to reproduce the values and have attributed the original reproducibility problems to the error in the data file with the mouse traces, as specified by the author.
Author_Assistance = TRUE # was author assistance provided? (if so, enter TRUE)
Insufficient_Information_Errors <- 0 # how many discrete insufficient information issues did you encounter?
# Assess the causal locus (discrete reproducibility issues) of any reproducibility errors. Note that there doesn't necessarily have to be a one-to-one correspondance between discrete reproducibility issues and reproducibility errors. For example, it could be that the original article neglects to mention that a Greenhouse-Geisser correct was applied to ANOVA outcomes. This might result in multiple reproducibility errors, but there is a single causal locus (discrete reproducibility issue).
locus_typo <- 0 # how many discrete issues did you encounter that related to typographical errors?
locus_specification <- 0 # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis <- 0 # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data <- 4 # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified <- 0 # how many discrete issues were there for which you could not identify the cause
# How many of the above issues were resolved through author assistance?
locus_typo_resolved <- 0 # how many discrete issues did you encounter that related to typographical errors?
locus_specification_resolved <- 0 # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis_resolved <- 0 # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data_resolved <- 4 # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified_resolved <- 0 # how many discrete issues were there for which you could not identify the cause
Affects_Conclusion <- FALSE # Do any reproducibility issues encounter appear to affect the conclusions made in the original article? TRUE, FALSE, or NA. This is a subjective judgement, but you should taking into account multiple factors, such as the presence/absence of decision errors, the number of target outcomes that could not be reproduced, the type of outcomes that could or could not be reproduced, the difference in magnitude of effect sizes, and the predictions of the specific hypothesis under scrutiny.reportObject <- reportObject %>%
filter(dummyRow == FALSE) %>% # remove the dummy row
select(-dummyRow) %>% # remove dummy row designation
mutate(articleID = articleID) %>% # add variables to report
select(articleID, everything()) # make articleID first column
# decide on final outcome
if(any(!(reportObject$comparisonOutcome %in% c("MATCH", "MINOR_ERROR"))) | Insufficient_Information_Errors > 0){
finalOutcome <- "Failure without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Failure despite author assistance"
}
}else{
finalOutcome <- "Success without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Success with author assistance"
}
}
# collate report extra details
reportExtras <- data.frame(articleID, pilotNames, copilotNames, pilotTTC, copilotTTC, pilotStartDate, copilotStartDate, completionDate, Author_Assistance, finalOutcome, Insufficient_Information_Errors, locus_typo, locus_specification, locus_analysis, locus_data, locus_unidentified, locus_typo_resolved, locus_specification_resolved, locus_analysis_resolved, locus_data_resolved, locus_unidentified_resolved)
# save report objects
if(reportType == "pilot"){
write_csv(reportObject, "pilotReportDetailed.csv")
write_csv(reportExtras, "pilotReportExtras.csv")
}
if(reportType == "final"){
write_csv(reportObject, "finalReportDetailed.csv")
write_csv(reportExtras, "finalReportExtras.csv")
}devtools::session_info()## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.0.0 (2020-04-24)
## os macOS Catalina 10.15.4
## system x86_64, darwin17.0
## ui X11
## language (EN)
## collate en_US.UTF-8
## ctype en_US.UTF-8
## tz Europe/London
## date 2020-05-12
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date lib
## assertthat 0.2.1 2019-03-21 [1]
## backports 1.1.6 2020-04-05 [1]
## broom 0.5.6 2020-04-20 [1]
## callr 3.4.3 2020-03-28 [1]
## cellranger 1.1.0 2016-07-27 [1]
## cli 2.0.2 2020-02-28 [1]
## colorspace 1.4-1 2019-03-18 [1]
## crayon 1.3.4 2017-09-16 [1]
## DBI 1.1.0 2019-12-15 [1]
## dbplyr 1.4.3 2020-04-19 [1]
## desc 1.2.0 2018-05-01 [1]
## devtools 2.3.0 2020-04-10 [1]
## digest 0.6.25 2020-02-23 [1]
## dplyr * 0.8.5 2020-03-07 [1]
## ellipsis 0.3.0 2019-09-20 [1]
## evaluate 0.14 2019-05-28 [1]
## fansi 0.4.1 2020-01-08 [1]
## forcats * 0.5.0 2020-03-01 [1]
## fs 1.4.1 2020-04-04 [1]
## generics 0.0.2 2018-11-29 [1]
## ggplot2 * 3.3.0 2020-03-05 [1]
## glue 1.4.0 2020-04-03 [1]
## gtable 0.3.0 2019-03-25 [1]
## haven * 2.2.0 2019-11-08 [1]
## hms 0.5.3 2020-01-08 [1]
## htmltools 0.4.0 2019-10-04 [1]
## httr 1.4.1 2019-08-05 [1]
## jsonlite 1.6.1 2020-02-02 [1]
## knitr * 1.28 2020-02-06 [1]
## lattice 0.20-41 2020-04-02 [1]
## lifecycle 0.2.0 2020-03-06 [1]
## lubridate 1.7.8 2020-04-06 [1]
## magrittr 1.5 2014-11-22 [1]
## memoise 1.1.0 2017-04-21 [1]
## modelr 0.1.7 2020-04-30 [1]
## munsell 0.5.0 2018-06-12 [1]
## nlme 3.1-147 2020-04-13 [1]
## pillar 1.4.4 2020-05-05 [1]
## pkgbuild 1.0.7 2020-04-25 [1]
## pkgconfig 2.0.3 2019-09-22 [1]
## pkgload 1.0.2 2018-10-29 [1]
## prettyunits 1.1.1 2020-01-24 [1]
## processx 3.4.2 2020-02-09 [1]
## ps 1.3.2 2020-02-13 [1]
## purrr * 0.3.4 2020-04-17 [1]
## R6 2.4.1 2019-11-12 [1]
## Rcpp 1.0.4.6 2020-04-09 [1]
## readr * 1.3.1 2018-12-21 [1]
## readxl * 1.3.1 2019-03-13 [1]
## remotes 2.1.1 2020-02-15 [1]
## reprex 0.3.0 2019-05-16 [1]
## ReproReports * 0.1 2020-05-06 [1]
## rlang 0.4.6 2020-05-02 [1]
## rmarkdown 2.1 2020-01-20 [1]
## rprojroot 1.3-2 2018-01-03 [1]
## rstudioapi 0.11 2020-02-07 [1]
## rvest 0.3.5 2019-11-08 [1]
## scales 1.1.0 2019-11-18 [1]
## sessioninfo 1.1.1 2018-11-05 [1]
## stringi 1.4.6 2020-02-17 [1]
## stringr * 1.4.0 2019-02-10 [1]
## testthat 2.3.2 2020-03-02 [1]
## tibble * 3.0.1 2020-04-20 [1]
## tidyr * 1.0.2 2020-01-24 [1]
## tidyselect 1.0.0 2020-01-27 [1]
## tidyverse * 1.3.0 2019-11-21 [1]
## usethis 1.6.1 2020-04-29 [1]
## vctrs 0.2.4 2020-03-10 [1]
## withr 2.2.0 2020-04-20 [1]
## xfun 0.13 2020-04-13 [1]
## xml2 1.3.2 2020-04-23 [1]
## yaml 2.2.1 2020-02-01 [1]
## source
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## Github (METRICS-CARPS/CARPSreports@3277f85)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
##
## [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library