CARPS Reproducibility Report
Report Details
articleID <- "8-12-2014_PS" # insert the article ID code here e.g., "10-3-2015_PS"
reportType <- 'final' # specify whether this is the 'pilot' report or 'final' report
pilotNames <- 'Bria Long' # insert the pilot's name here e.g., "Tom Hardwicke". If there are multiple cpilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
copilotNames <- 'Tom Hardwicke' # # insert the co-pilot's name here e.g., "Michael Frank". If there are multiple co-pilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
pilotTTC <- 180 # insert the pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
copilotTTC <- 180 # insert the co-pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
pilotStartDate <- as.Date("10/19/18", format = "%m/%d/%y") # insert the pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
copilotStartDate <- as.Date("06/29/19", format = "%m/%d/%y") # insert the co-pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
completionDate <- as.Date("08/24/19", format = "%m/%d/%y") # copilot insert the date of final report completion (after any necessary rounds of author assistance) in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")Methods summary:
The authors calculated the d’1 and d’2 for a grammatical judgement experiment in which participants gave both a response and made a confidence judgement about this response. These d’1 and d’2 values are compared in subjects who showed a d’1 performance below zero in the first 75% of trials vs. subjects who did not.
Target outcomes:
Figure 3 illustrates the mean d′1 and d′2 for the analysis trials (final 25%); results are plotted separately for participants who performed above chance in the selection trials (first 75%) and those who performed at or below chance in the selection trials. Analyses were conducted listwise to ensure that the means for each index were based on the same participants. Among participants who exhibited first-order accuracy in the selection trials, d′2 was significantly greater than chance, t(164) = 4.65, p < .001, d = 0.36, and showed the typical relationship with d′1, specifically that the mean of d′2 is approximately half that of d′1 (Krueger, Klapoetke, & Mattler, 2011).
Crucially,among participants who did not exhibit first-order accuracy, d′2 remained significantly greater than chance, t(32) = 2.30, p = .028, d = 0.40, and was not significantly different from the d′2 of participants who did exhibit first order accuracy, t(196) = 0.17, p = .868, d = 0.03. Thus, the analysis revealed reliable metacognitive performance among participants who did not exhibit first-order decision accuracy.
Step 1: Load packages and prepare report object
# load packages
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
library(ReproReports) # custom report functions
library(lsr)# Prepare report object. This will be updated automatically by the reproCheck function each time values are compared
reportObject <- data.frame(dummyRow = TRUE, reportedValue = NA, obtainedValue = NA, valueType = NA, percentageError = NA, comparisonOutcome = NA, eyeballCheck = NA)Step 2: Load data
spss_file = "originalAnalysis/Type1and2.sav"
spss_data = read_spss(spss_file)
matlab_output_data = read_excel("data/ResultFile.xls", col_names = c("pnum","t1dp_1st3Q","t2dp_1st3Q","t1dp_lastQ","t2dp_lastQ"))
d_raw = read_excel("data/RawData.xlsx")Step 3. Pre-process the raw data to recalculate the d-prime measurements; tidy data.
trial_counts <- d_raw %>%
count(Pnum) %>% ## count number of trials per participation, some participants have 64 vs. 60...
mutate(end3Q = (n/4)*3) ## Estimate 3/4 of the way through as in author's matlab script.
## Now filter by these values ad estimate d-primes
first_3quarters_data <- d_raw %>%
group_by(Pnum) %>%
left_join(trial_counts) %>%
filter(PresentationOrder < end3Q+1) %>%
summarize(d1_HR_3q = sum(Judgement==1 & Grammaticality==1) / sum(Grammaticality==1),
d1_FA_3q = sum(Judgement==1 & Grammaticality==0) / sum(Grammaticality==0),
d2_HR_3q = sum(BinaryConf==1 & Accuracy==1) / sum(Accuracy==1),
d2_FA_3q = sum(BinaryConf==1 & Accuracy==0) / sum(Accuracy==0)) %>%
mutate(t1dp_1st3Q = qnorm(d1_HR_3q) - qnorm(d1_FA_3q), t2dp_1st3Q=qnorm(d2_HR_3q) - qnorm(d2_FA_3q))
## replace infs with nans, as authors do.
first_3quarters_data$t1dp_1st3Q[first_3quarters_data$t1dp_1st3Q==Inf | first_3quarters_data$t1dp_1st3Q==-Inf]=NaN
first_3quarters_data$t2dp_1st3Q[first_3quarters_data$t2dp_1st3Q==Inf | first_3quarters_data$t2dp_1st3Q==-Inf]=NaN
## repeat same process for last quarter data.
last_quarter_data <- d_raw %>%
group_by(Pnum) %>%
left_join(trial_counts) %>%
filter(PresentationOrder > end3Q) %>%
summarize(d1_HR_lq = sum(Judgement==1 & Grammaticality==1) / sum(Grammaticality==1),
d1_FA_lq = sum(Judgement==1 & Grammaticality==0) / sum(Grammaticality==0),
d2_HR_lq = sum(BinaryConf==1 & Accuracy==1) / sum(Accuracy==1),
d2_FA_lq = sum(BinaryConf==1 & Accuracy==0) / sum(Accuracy==0)) %>%
mutate(t1dp_lastQ = qnorm(d1_HR_lq) - qnorm(d1_FA_lq), t2dp_lastQ=qnorm(d2_HR_lq) - qnorm(d2_FA_lq))
## replace infs with nans, as authors do.
last_quarter_data$t1dp_lastQ[last_quarter_data$t1dp_lastQ==Inf | last_quarter_data$t1dp_lastQ==-Inf]=NaN
##
last_quarter_data$t2dp_lastQ[last_quarter_data$t2dp_lastQ==Inf | last_quarter_data$t2dp_lastQ==-Inf]=NaN
summary_data <- last_quarter_data %>%
left_join(first_3quarters_data) %>%
rename(pnum = 'Pnum')Look at correlations between the dprime values calculated in both R and in Matlab.
Values are a bit off but pretty good – might be due to a rounding errors?
compare_data <- summary_data %>%
left_join(spss_data)
plot(summary_data$t1dp_1st3Q, spss_data$t1dp_1st3Q)
# plot(summary_data$t2dp_1st3Q, spss_data$t2dp_1st3Q)
# # #
# plot(summary_data$t1dp_lastQ, spss_data$t1dp_lastQ)
# plot(summary_data$t2dp_lastQ, spss_data$t2dp_lastQ)Step 4: Run analysis
Subject-level exclusions in the following analyses taken from paper (pg.2202) “Only participants for whom both d′1 and d′2 could be computed for the analysis subset (i.e., who had nonzero counts in every cell) were included; above chance: n = 165, at chance: n = 33.” However, we cannot reproduce the number of subjects included in any of the analyses.
## Those who had low accuracy in the first 3 quarters
# without_first_order_acc <- summary_data %>%
# filter(!is.na(t1dp_1st3Q) & !is.na(t2dp_1st3Q) & !is.na(t1dp_lastQ) & !is.na(t2dp_lastQ)) %>% # exclude subjects who have a NA in any of the 4 values
# filter(t1dp_1st3Q<=0) %>% ## d'1 is less than or equal to zero in first 3Q
# summarize(count_participants = length(unique(pnum)), mean_d1_first3q = mean(t1dp_1st3Q), mean_d2_first3q = mean(t2dp_1st3Q), mean_d1_last_q=mean(t1dp_lastQ), mean_d2_last_q=mean(t2dp_lastQ))
#
# kable(without_first_order_acc)# ## Those who had above chance accuracy in the first 3 quarters
# with_first_order_acc <- summary_data %>%
# filter(!is.na(t1dp_1st3Q) & !is.na(t2dp_1st3Q) & !is.na(t1dp_lastQ) & !is.na(t2dp_lastQ)) %>% # exclude subjects who have a NA in any of the 4 values
# filter(t1dp_1st3Q>0) %>% ## d'1 is greater tha zero in first 3Q
# summarize(count_participants = length(unique(pnum)), mean_d1_first3q = mean(t1dp_1st3Q), mean_d2_first3q = mean(t2dp_1st3Q), mean_d1_last_q=mean(t1dp_lastQ), mean_d2_last_q=mean(t2dp_lastQ))
#
# kable(with_first_order_acc)Try getting calculations from the spss datafile, which already has the exclusions.
Number of participants is still off in both cases.
# d <- spss_data
# ##
# without_first_order_acc <- d %>%
# filter(Filter1st3QzeroOrLess == "Selected" ) %>%
# filter(!is.na(t1dp_1st3Q) & !is.na(t2dp_1st3Q) & !is.na(t1dp_lastQ) & !is.na(t2dp_lastQ)) %>%
# summarize(count_participants = length(unique(pnum)), mean_d1_first3q = mean(t1dp_1st3Q), mean_d2_last_q=mean(t2dp_lastQ))
#
# kable(without_first_order_acc)# with_first_order_acc <- d %>%
# filter(Filter1st3QgreaterThanZero == "Selected" ) %>%
# filter(!is.na(t1dp_1st3Q) & !is.na(t2dp_1st3Q) & !is.na(t1dp_lastQ) & !is.na(t2dp_lastQ)) %>%
# summarize(count_participants = length(unique(pnum)), mean_d1_first3q = mean(t1dp_1st3Q), mean_d2_last_q=mean(t2dp_lastQ))
#
# kable(with_first_order_acc)Try getting the descriptives/results from the matlab output file.
Number of participants is still off in both cases, but the same as the spss discrepancies.
# d <- matlab_output_data
# ## Create basic descriptives of two sets of participants
# ## Those who had low accuracy in the first third
# without_first_order_acc <- d %>%
# filter(!is.na(t1dp_1st3Q) & !is.na(t2dp_1st3Q) & !is.na(t1dp_lastQ) & !is.na(t2dp_lastQ)) %>% # exclude subjects who have a NA in any of the 4 values
# filter(t1dp_1st3Q<=0) %>% ## d'1 is less than or equal to zero in first 3Q
# summarize(count_participants = length(unique(pnum)), mean_d1_first3q = mean(t1dp_1st3Q), mean_d2_first3q = mean(t2dp_1st3Q), mean_d1_last_q=mean(t1dp_lastQ), mean_d2_last_q=mean(t2dp_lastQ))
#
# kable(without_first_order_acc)# with_first_order_acc <- d %>%
# filter(!is.na(t1dp_1st3Q) & !is.na(t2dp_1st3Q) & !is.na(t1dp_lastQ) & !is.na(t2dp_lastQ)) %>%
# filter(t1dp_1st3Q>0) %>%
# summarize(count_participants = length(unique(pnum)), mean_d1_first3q = mean(t1dp_1st3Q), mean_d2_first3q = mean(t2dp_1st3Q), mean_d1_last_q=mean(t1dp_lastQ), mean_d2_last_q=mean(t2dp_lastQ))
#
# kable(with_first_order_acc)Inferential statistics
“Among participants who exhibited first-order accuracy in the selection trials, d′2 was significantly greater than chance, t(164) = 4.65, p < .001, d = 0.36”
out <- t.test(with_first_order_acc$t2dp_lastQ)
out$d <- cohensD(with_first_order_acc$t2dp_lastQ)
reportObject <- reproCheck(reportedValue = '4.65', obtainedValue = out$statistic, valueType = 't')## [1] "MINOR_ERROR for t. The reported value (4.65) and the obtained value (4.64) differed by 0.22%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.001', obtainedValue = out$p.value, valueType = 'p', eyeballCheck=TRUE)## [1] "MATCH for p. Eyeball comparison only."reportObject <- reproCheck(reportedValue = '164', obtainedValue = out$parameter, valueType = 'df')## [1] "MATCH for df. The reported value (164) and the obtained value (164) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '0.36', obtainedValue = out$d, valueType = 'd')## [1] "MATCH for d. The reported value (0.36) and the obtained value (0.36) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."“Crucially,among participants who did not exhibit first-order accuracy, d′2 remained significantly greater than chance, t(32) = 2.30, p = .028, d = 0.40”
out <- t.test(without_first_order_acc$t2dp_lastQ)
out$d <- cohensD(without_first_order_acc$t2dp_lastQ)
reportObject <- reproCheck(reportedValue = '2.30', obtainedValue = out$statistic, valueType = 't')## [1] "MATCH for t. The reported value (2.3) and the obtained value (2.3) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.028', obtainedValue = out$p.value, valueType = 'p')## [1] "MATCH for p. The reported value (0.028) and the obtained value (0.028) differed by 0%. Note that the obtained value was rounded to 3 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '32', obtainedValue = out$parameter, valueType = 'df')## [1] "MATCH for df. The reported value (32) and the obtained value (32) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '0.40', obtainedValue = out$d, valueType = 'd')## [1] "MATCH for d. The reported value (0.4) and the obtained value (0.4) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."“…and was not significantly different from the d′2 of participants who did exhibit first order accuracy, t(196) = 0.17, p = .868, d = 0.03.”
out <- t.test(without_first_order_acc$t2dp_lastQ, with_first_order_acc$t2dp_lastQ, var.equal = T)
out$d <- cohensD(without_first_order_acc$t2dp_lastQ, with_first_order_acc$t2dp_lastQ)
reportObject <- reproCheck(reportedValue = '.17', obtainedValue = out$statistic, valueType = 't')## [1] "MAJOR_ERROR for t. The reported value (0.17) and the obtained value (0.12) differed by 29.41%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '.868', obtainedValue = out$p.value, valueType = 'p')## [1] "MINOR_ERROR for p. The reported value (0.868) and the obtained value (0.908) differed by 4.61%. Note that the obtained value was rounded to 3 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '196', obtainedValue = out$parameter, valueType = 'df')## [1] "MATCH for df. The reported value (196) and the obtained value (196) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = '0.03', obtainedValue = out$d, valueType = 'd')## [1] "MAJOR_ERROR for d. The reported value (0.03) and the obtained value (0.02) differed by 33.33%. Note that the obtained value was rounded to 2 decimal places to match the reported value."We have two “major errors” here. We contacted the authors and they said they also could not reproduce these values - in their own reanalysis they obtain the same values as we do here. The reason for the differences is unknown. The authors suggest it might be due to change in version of the SPSS software that was used to run the original analysis. Although they reach the 10% threshold of a reproducibiltiy issues according to our operational definition, these differences are clearly not consequential to the authors’ original conclusions.
Step 5: Conclusion
We initially ran into a problem reproducing the exclusions reported in the paper. We used both the SPSS and Matlab files provided by the authors and attempted to apply the exclusions in several different ways. As all were unsuccessful, we contacted the original authors to ask for assistance. They created a video of themselves performing the analysis in SPSS and reproducing the exclusion numbers reported in the paper. Watching this video helped us to identify that we had misunderstood the sentence that reported the exlcusions, specifically: “Only participants for whom both d′1 and d′2 could be computed for the analysis subset (i.e., who had nonzero counts in every cell) were included; above chance: n = 165, at chance: n = 33.” We had excluded participants with missing values in any of the cells, whereas the original analysis excluded participants with missing values in cells relevant to the analysis subset. After being sent the video, we were able to reproduce this part of the analysis successfully and proceeded with the remainder of the analysis.
We were unable to reproduce two values - a t-value (0.17 reported vs. 0.12 in our analysis) and a d-value (0.03 reported vs.0.02 in our analysis). We contacted the authors again and they said they also could not reproduce these values - in their own reanalysis they obtain the same values as we do here. The reason for the differences is unknown. The authors suggest it might be due to change in version of the SPSS software that was used to run the original analysis. Although they reach the 10% threshold of a reproducibiltiy issues according to our operational definition, these differences are clearly not consequential to the authors’ original conclusions.
Author_Assistance = TRUE # was author assistance provided? (if so, enter TRUE)
Insufficient_Information_Errors <- 0 # how many discrete insufficient information issues did you encounter?
# Assess the causal locus (discrete reproducibility issues) of any reproducibility errors. Note that there doesn't necessarily have to be a one-to-one correspondance between discrete reproducibility issues and reproducibility errors. For example, it could be that the original article neglects to mention that a Greenhouse-Geisser correct was applied to ANOVA outcomes. This might result in multiple reproducibility errors, but there is a single causal locus (discrete reproducibility issue).
locus_typo <- 0 # how many discrete issues did you encounter that related to typographical errors?
locus_specification <- 1 # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis <- 0 # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data <- 0 # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified <- 1 # how many discrete issues were there for which you could not identify the cause
# How many of the above issues were resolved through author assistance?
locus_typo_resolved <- 0 # how many discrete issues did you encounter that related to typographical errors?
locus_specification_resolved <- 1 # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis_resolved <- 0 # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data_resolved <- 0 # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified_resolved <- 0 # how many discrete issues were there for which you could not identify the cause
Affects_Conclusion <- FALSE # Do any reproducibility issues encounter appear to affect the conclusions made in the original article? TRUE, FALSE, or NA. This is a subjective judgement, but you should taking into account multiple factors, such as the presence/absence of decision errors, the number of target outcomes that could not be reproduced, the type of outcomes that could or could not be reproduced, the difference in magnitude of effect sizes, and the predictions of the specific hypothesis under scrutiny.reportObject <- reportObject %>%
filter(dummyRow == FALSE) %>% # remove the dummy row
select(-dummyRow) %>% # remove dummy row designation
mutate(articleID = articleID) %>% # add variables to report
select(articleID, everything()) # make articleID first column
# decide on final outcome
if(any(!(reportObject$comparisonOutcome %in% c("MATCH", "MINOR_ERROR"))) | Insufficient_Information_Errors > 0){
finalOutcome <- "Failure without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Failure despite author assistance"
}
}else{
finalOutcome <- "Success without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Success with author assistance"
}
}
# collate report extra details
reportExtras <- data.frame(articleID, pilotNames, copilotNames, pilotTTC, copilotTTC, pilotStartDate, copilotStartDate, completionDate, Author_Assistance, finalOutcome, Insufficient_Information_Errors, locus_typo, locus_specification, locus_analysis, locus_data, locus_unidentified, locus_typo_resolved, locus_specification_resolved, locus_analysis_resolved, locus_data_resolved, locus_unidentified_resolved)
# save report objects
if(reportType == "pilot"){
write_csv(reportObject, "pilotReportDetailed.csv")
write_csv(reportExtras, "pilotReportExtras.csv")
}
if(reportType == "final"){
write_csv(reportObject, "finalReportDetailed.csv")
write_csv(reportExtras, "finalReportExtras.csv")
}Session information
devtools::session_info()## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.0.0 (2020-04-24)
## os macOS Catalina 10.15.4
## system x86_64, darwin17.0
## ui X11
## language (EN)
## collate en_US.UTF-8
## ctype en_US.UTF-8
## tz Europe/London
## date 2020-05-19
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date lib
## assertthat 0.2.1 2019-03-21 [1]
## backports 1.1.6 2020-04-05 [1]
## broom 0.5.6 2020-04-20 [1]
## callr 3.4.3 2020-03-28 [1]
## cellranger 1.1.0 2016-07-27 [1]
## cli 2.0.2 2020-02-28 [1]
## colorspace 1.4-1 2019-03-18 [1]
## crayon 1.3.4 2017-09-16 [1]
## DBI 1.1.0 2019-12-15 [1]
## dbplyr 1.4.3 2020-04-19 [1]
## desc 1.2.0 2018-05-01 [1]
## devtools 2.3.0 2020-04-10 [1]
## digest 0.6.25 2020-02-23 [1]
## dplyr * 0.8.5 2020-03-07 [1]
## ellipsis 0.3.0 2019-09-20 [1]
## evaluate 0.14 2019-05-28 [1]
## fansi 0.4.1 2020-01-08 [1]
## forcats * 0.5.0 2020-03-01 [1]
## fs 1.4.1 2020-04-04 [1]
## generics 0.0.2 2018-11-29 [1]
## ggplot2 * 3.3.0 2020-03-05 [1]
## glue 1.4.0 2020-04-03 [1]
## gtable 0.3.0 2019-03-25 [1]
## haven * 2.2.0 2019-11-08 [1]
## hms 0.5.3 2020-01-08 [1]
## htmltools 0.4.0 2019-10-04 [1]
## httr 1.4.1 2019-08-05 [1]
## jsonlite 1.6.1 2020-02-02 [1]
## knitr * 1.28 2020-02-06 [1]
## lattice 0.20-41 2020-04-02 [1]
## lifecycle 0.2.0 2020-03-06 [1]
## lsr * 0.5 2015-03-02 [1]
## lubridate 1.7.8 2020-04-06 [1]
## magrittr 1.5 2014-11-22 [1]
## memoise 1.1.0 2017-04-21 [1]
## modelr 0.1.7 2020-04-30 [1]
## munsell 0.5.0 2018-06-12 [1]
## nlme 3.1-147 2020-04-13 [1]
## pillar 1.4.4 2020-05-05 [1]
## pkgbuild 1.0.7 2020-04-25 [1]
## pkgconfig 2.0.3 2019-09-22 [1]
## pkgload 1.0.2 2018-10-29 [1]
## prettyunits 1.1.1 2020-01-24 [1]
## processx 3.4.2 2020-02-09 [1]
## ps 1.3.2 2020-02-13 [1]
## purrr * 0.3.4 2020-04-17 [1]
## R6 2.4.1 2019-11-12 [1]
## Rcpp 1.0.4.6 2020-04-09 [1]
## readr * 1.3.1 2018-12-21 [1]
## readxl * 1.3.1 2019-03-13 [1]
## remotes 2.1.1 2020-02-15 [1]
## reprex 0.3.0 2019-05-16 [1]
## ReproReports * 0.1 2020-05-06 [1]
## rlang 0.4.6 2020-05-02 [1]
## rmarkdown 2.1 2020-01-20 [1]
## rprojroot 1.3-2 2018-01-03 [1]
## rstudioapi 0.11 2020-02-07 [1]
## rvest 0.3.5 2019-11-08 [1]
## scales 1.1.0 2019-11-18 [1]
## sessioninfo 1.1.1 2018-11-05 [1]
## stringi 1.4.6 2020-02-17 [1]
## stringr * 1.4.0 2019-02-10 [1]
## testthat 2.3.2 2020-03-02 [1]
## tibble * 3.0.1 2020-04-20 [1]
## tidyr * 1.0.2 2020-01-24 [1]
## tidyselect 1.0.0 2020-01-27 [1]
## tidyverse * 1.3.0 2019-11-21 [1]
## usethis 1.6.1 2020-04-29 [1]
## vctrs 0.2.4 2020-03-10 [1]
## withr 2.2.0 2020-04-20 [1]
## xfun 0.13 2020-04-13 [1]
## xml2 1.3.2 2020-04-23 [1]
## yaml 2.2.1 2020-02-01 [1]
## source
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## Github (METRICS-CARPS/CARPSreports@3277f85)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
##
## [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library