CARPS Reproducibility Report
articleID <- "3-4-2015_PS" # insert the article ID code here e.g., "10-3-2015_PS"
reportType <- 'final'
pilotNames <- "Erik Santoro, Tysen Dauer, Jaclyn Schwartz" # insert the pilot's name here e.g., "Tom Hardwicke". If there are multiple pilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
copilotNames <- "Erica Yoon, Tom Hardwicke" # insert the co-pilot's name here e.g., "Michael Frank". If there are multiple co-pilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
pilotTTC <- 720 # insert the pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
copilotTTC <- 240 # insert the co- pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
pilotStartDate <- as.Date("10/31/17", format = "%m/%d/%y") # insert the pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
copilotStartDate <- as.Date("06/13/18", format = "%m/%d/%y") # insert the co-pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
completionDate <- as.Date("09/07/18", format = "%m/%d/%y") # copilot insert the date of final report completion (after any necessary rounds of author assistance) in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")Methods summary:
This paper tested the effects of proprioceptive information – aka our sense of body movement – on pain; the outstanding theories on pain up to that point had covered nocioception, or internal-stimuli. To examine the effects of perception on pain, participants wore a virtual reality headset while rotating their heads until they felt pain; the distance between the center and where they felt pain as measured by degrees, the “pain-free range of motion”, was the main dependent variable. Participants rotated their head to the left and to the right for 3 conditions: perceived movement understated true movement (e.g. gain = 0.8, or virtual rotation was 80% of actual rotation), was the same as true movement (e.g. gain = 1, or 100% of actual rotation), or overstated true movement (e.g. gain = 1.2, or virutal rotation was 120% of actual rotation). The order in which participants experienced the conditions was counterbalanced. To minimize the detection of virtual reality manipulation, participants were exposed to a different visual scene for each of the 6 trials (3 conditions * 2 directions of rotation). The participants and experimenters were blinded. Finally, there were two “manipulation checks”: the first piloted 9 healthy participants to find the ranges within which participants would not be able to determine virtual reality manipulation, and the second assessed the quality check of the machine.
Target outcomes:
The repeated measures ANOVA revealed a large overall effect of visual-proprioceptive feedback (condition) on pain-free range of motion F(2, 94) = 18.9, p < .001, η·p2 = 0.29. All pairwise comparisons were significant (ps < .01). As shown in Figure 3, when vision understated true rotation, pain-free range of motion was increased, and this was a medium-sized effect, p = .006, d = 0.67; when vision overstated true rotation, pain-free range of motion was decreased, and this was a large effect, p = .001, d = 0.80. Specifically, during visual feedback that understated true rotation, pain-free range of motion was increased by 6% (95% confidence interval, or CI = [2%, 11%]); during visual feedback that overstated true rotation, pain-free range of motion decreased by 7% (95% CI = [3%, 11%]). Therefore, our results show an overall effect of the manipulation of 13%.
Step 1: Load packages
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
library(ReproReports) # custom report functions
library(ez) # for ezANOVA
library(afex) # for repeated ANOVAs
library(lsr) #for cohen's d# Prepare report object. This will be updated automatically by the reproCheck function each time values are compared.
reportObject <- data.frame(dummyRow = TRUE, reportedValue = NA, obtainedValue = NA, valueType = NA, percentageError = NA, comparisonOutcome = NA, eyeballCheck = NA)Step 2: Load data
We initially encountered reproducibility issues and contacted the authors for assistance. According to the author (email correspondence), different normalization procedures were used for the descriptive statistics, ANOVA, and t-tests (this was not reported in the paper). For ANOVA and descriptive statistics, the authors “transformed data for each participant to a proportion of the average range of motion demonstrated in the neutral condition”. For t-tests, the authors normalised each participants data “to an average of the three conditions”. Below we prepare separate data sets using each of the normalization procedures.
d <- read_sav("data/BogusVisualFeedbackData.sav")
d_anova <- d %>%
select(Participant, DirectionofRotation, Condition1_Gain0.8, Condition2_Gain1, Condition3_Gain1.2)
d_ttest <- d %>%
select(Participant, DirectionofRotation, Point.8, One1, One1.2)Step 3: Tidy data
Make tidy data for both ANOVA and ttest.
#Create tidy data set
d.tidy_anova <- d_anova %>%
gather(condition,rangeofmotion,starts_with("condition")) %>% #the value various condition columns contains are the range of motion
#Need to convert following columns to proper type
mutate(Participant = as.factor(Participant),
condition = as.factor(condition),
DirectionofRotation = as.factor(DirectionofRotation))
d.tidy_ttest <- d_ttest %>%
rename(Condition1_Gain0.8 = Point.8,
Condition2_Gain1.0 = One1,
Condition3_Gain1.2 = One1.2
) %>%
mutate(Participant = as.factor(Participant),
DirectionofRotation = as.factor(DirectionofRotation))Step 4: Run analysis
Pre-processing
For reference, make a tidy table that groups by participant and averages across direction of rotation (e.g. left or right).
d.comparison <- d_anova %>%
group_by(Participant) %>%
summarise(mean_c1 = mean(Condition1_Gain0.8),
mean_c2 = mean(Condition2_Gain1),
mean_c3 = mean(Condition3_Gain1.2)
)Descriptive statistics
Figure 3
Here are means and CIs of pain-free range of motion per condition.
mean0.8 <- mean(d.comparison$mean_c1)
ci0.8 <- t.test(d.comparison$mean_c1)$conf.int
mean0.8## [1] 1.065417ci0.8## [1] 1.021867 1.108967
## attr(,"conf.level")
## [1] 0.95mean0.8 <- mean(d_anova$Condition1_Gain0.8)
mean1 <- mean(d_anova$Condition2_Gain1)
mean1.2 <- mean(d_anova$Condition3_Gain1.2)
mean1.2 <- mean(mean(d.comparison$mean_c3))
ci1.2 <- t.test(d.comparison$mean_c3)$conf.int
mean1.2## [1] 0.93125ci1.2## [1] 0.8903752 0.9721248
## attr(,"conf.level")
## [1] 0.95All seem to match.
reportObject <- reproCheck(reportedValue = "figure", obtainedValue = mean0.8, valueType = 'mean', eyeballCheck = TRUE)## [1] "MATCH for mean. Eyeball comparison only."reportObject <- reproCheck(reportedValue = "figure", obtainedValue = ci0.8[1], valueType = 'ci', eyeballCheck = TRUE)## [1] "MATCH for ci. Eyeball comparison only."reportObject <- reproCheck(reportedValue = "figure", obtainedValue = ci0.8[2], valueType = 'ci', eyeballCheck = TRUE)## [1] "MATCH for ci. Eyeball comparison only."reportObject <- reproCheck(reportedValue = "figure", obtainedValue = mean1, valueType = 'mean', eyeballCheck = TRUE)## [1] "MATCH for mean. Eyeball comparison only."reportObject <- reproCheck(reportedValue = "figure", obtainedValue = mean1.2, valueType = 'mean', eyeballCheck = TRUE)## [1] "MATCH for mean. Eyeball comparison only."reportObject <- reproCheck(reportedValue = "figure", obtainedValue = ci1.2[1], valueType = 'ci', eyeballCheck = TRUE)## [1] "MATCH for ci. Eyeball comparison only."reportObject <- reproCheck(reportedValue = "figure", obtainedValue = ci1.2[2], valueType = 'ci', eyeballCheck = TRUE)## [1] "MATCH for ci. Eyeball comparison only."Inferential statistics
First, we look at the ANOVA. Note that, according to the author (in an email correspondence), each participant x direction of rotation was considered to be a unique case. This means the unit of analysis is different from the descriptive statistics and the t-tests below (where the unit of analysis is an individual participant). This was not specified in the paper.
The repeated measures ANOVA revealed a large overall effect of visual-proprioceptive feedback (condition) on pain-free range of motion F(2, 94) = 18.9, p < .001, ηp 2 = 0.29. (from Harvie et al. p.388)
#Repeated Measures ANOVA
aov.out <- aov_ez(data = d.tidy_anova %>%
mutate(participant_direction = paste(Participant, DirectionofRotation, sep = "_")),
id = "participant_direction",
dv = "rangeofmotion",
within = c("condition"),
anova_table = list(correction = 'none', es = 'pes'))reportObject <- reproCheck(reportedValue = "18.9", obtainedValue = aov.out$anova_table$F, valueType = 'F')## [1] "MATCH for F. The reported value (18.9) and the obtained value (18.9) differed by 0%. Note that the obtained value was rounded to 1 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = "2", obtainedValue = aov.out$anova_table$`num Df`, valueType = 'df')## [1] "MATCH for df. The reported value (2) and the obtained value (2) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = "94", obtainedValue = aov.out$anova_table$`den Df`, valueType = 'df')## [1] "MATCH for df. The reported value (94) and the obtained value (94) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = "<.001", obtainedValue = aov.out$anova_table$`Pr(>F)`, valueType = 'p', eyeballCheck = TRUE)## [1] "MATCH for p. Eyeball comparison only."reportObject <- reproCheck(reportedValue = "0.29", obtainedValue = aov.out$anova_table$pes, valueType = 'pes')## [1] "MATCH for pes. The reported value (0.29) and the obtained value (0.29) differed by 0%. Note that the obtained value was rounded to 2 decimal places to match the reported value."All the values match up when we run the analysis as specified in the author’s correspondance.
Now, we look at all pairwise comparisons. The statistical test was not specified in the paper. However, the authors confirmed in email correspondence that they used t-tests. Note that the authors also informed us that the values used in this analysis are different from those used for the descriptive statistics and ANOVA, because they are normalised to the average rating across three conditions for each participant.
All pairwise comparisons were significant (ps < .01). (from Harvie et al. p.388)
ttest0.8v1 <- t.test(d.tidy_ttest$Condition1_Gain0.8, d.tidy_ttest$Condition2_Gain1.0, paired = TRUE)
ttest0.8v1.2 <- t.test(d.tidy_ttest$Condition1_Gain0.8, d.tidy_ttest$Condition3_Gain1.2, paired = TRUE)
ttest1.2v1 <- t.test(d.tidy_ttest$Condition3_Gain1.2, d.tidy_ttest$Condition2_Gain1.0, paired = TRUE)reportObject <- reproCheck(reportedValue = "<.01", obtainedValue = ttest0.8v1$p.value, valueType = 'p', eyeballCheck = TRUE)## [1] "MATCH for p. Eyeball comparison only."reportObject <- reproCheck(reportedValue = "<.01", obtainedValue = ttest0.8v1.2$p.value, valueType = 'p', eyeballCheck = TRUE)## [1] "MATCH for p. Eyeball comparison only."reportObject <- reproCheck(reportedValue = "<.01", obtainedValue = ttest1.2v1$p.value, valueType = 'p', eyeballCheck = TRUE)## [1] "MATCH for p. Eyeball comparison only."Now, we compare effect sizes and related p values:
As shown in Figure 3, when vision understated true rotation, pain-free range of motion was increased, and this was a medium-sized effect, p = .006, d = 0.67; when vision overstated true rotation, pain-free range of motion was decreased, and this was a large effect, p = .001, d = 0.80. (from Harvie et al. p.388-9)
ttest_cohen1vs2 <- pairedSamplesTTest(formula = ~Condition1_Gain0.8 + Condition2_Gain1.0, data=as.data.frame(d.tidy_ttest))
ttest_cohen2vs3 <- pairedSamplesTTest(formula = ~Condition2_Gain1.0 + Condition3_Gain1.2, data=as.data.frame(d.tidy_ttest))We cannot reproduce the effect sizes:
reportObject <- reproCheck(reportedValue = ".67", obtainedValue = ttest_cohen1vs2$effect.size, valueType = 'd')## [1] "MAJOR_ERROR for d. The reported value (0.67) and the obtained value (0.42) differed by 37.31%. Note that the obtained value was rounded to 2 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = ".006", obtainedValue = ttest_cohen1vs2$p.value, valueType = 'p')## [1] "MATCH for p. The reported value (0.006) and the obtained value (0.006) differed by 0%. Note that the obtained value was rounded to 3 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = ".8", obtainedValue = ttest_cohen2vs3$effect.size, valueType = 'd')## [1] "MAJOR_ERROR for d. The reported value (0.8) and the obtained value (0.6) differed by 25%. Note that the obtained value was rounded to 1 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = ".001", obtainedValue = ttest_cohen2vs3$p.value, valueType = 'p')## [1] "MAJOR_ERROR for p. The reported value (0.001) and the obtained value (0) differed by 100%. Note that the obtained value was rounded to 3 decimal places to match the reported value."Now, we compare the percentage change and confidence intervals.
Specifically, during visual feedback that understated true rotation, pain-free range of motion was increased by 6% (95% confidence interval, or CI = [2%, 11%]); during visual feedback that overstated true rotation, pain-free range of motion decreased by 7% (95% CI = [3%, 11%]). Therefore, our results show an overall effect of the manipulation of 13%." (from Harvie et al. p.389)
#Vision understated true rotation
confint_1vs2 <- (ttest_cohen1vs2$conf.int)*100
mean_1vs2 <- (ttest_cohen1vs2$mean[1] - ttest_cohen1vs2$mean[2])*100
#Vision overstated true rotation
confint_2vs3 <- (ttest_cohen2vs3$conf.int)*100
mean_2vs3 <- (ttest_cohen2vs3$mean[1] - ttest_cohen2vs3$mean[2])*100reportObject <- reproCheck(reportedValue = "6", obtainedValue = mean_1vs2, valueType = 'mean')## [1] "MATCH for mean. The reported value (6) and the obtained value (6) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = "2", obtainedValue = confint_1vs2[1], valueType = 'ci')## [1] "MATCH for ci. The reported value (2) and the obtained value (2) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = "11", obtainedValue = confint_1vs2[2], valueType = 'ci')## [1] "MINOR_ERROR for ci. The reported value (11) and the obtained value (10) differed by 9.09%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = "7", obtainedValue = mean_2vs3, valueType = 'mean')## [1] "MAJOR_ERROR for mean. The reported value (7) and the obtained value (8) differed by 14.29%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = "3", obtainedValue = confint_2vs3[1], valueType = 'ci')## [1] "MAJOR_ERROR for ci. The reported value (3) and the obtained value (4) differed by 33.33%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = "11", obtainedValue = confint_2vs3[2], valueType = 'ci')## [1] "MATCH for ci. The reported value (11) and the obtained value (11) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."reportObject <- reproCheck(reportedValue = "13", obtainedValue = mean_1vs2+mean_2vs3, valueType = 'mean')## [1] "MATCH for mean. The reported value (13) and the obtained value (13) differed by 0%. Note that the obtained value was rounded to 0 decimal places to match the reported value."The percentage change for the understatement (e.g. 0.8 vs 1) had a minor numerical error, and the confidence interval for the overstatement had two major numerical errors.
Step 5: Conclusion
In this reproducibility check we were able to reproduce the decriptive statistics (Figure 3) but ran into difficulties reproducing some of the inferential statistics. We contacted the authors and received asssitance that resolved some issues. Specifically, information was provided about the identity of statistical tests, units of analysis, and normalization procedures, that were not stated in the paper. Unfortunately, we then lost contact with the authors and some reproducibility issues remained that we could not resolve.
We could not reproduce two effect sizes (d). The authors reported that they tried to reproduce these values themselves and could “closely reproduce the Cohen’s d values”. However, they did not share the exact values they obtained in their re-analysis. We could also not reproduce one mean and one bound of a confidence interval. The causal locus of these issues is unclear.
The reproducibility issues do not appear to undermine the original conclusions. The obtained effect sizes are smaller than those reported, but not substantially so. The mean and ci discrepancies are of low magnitude.
Author_Assistance = TRUE # was author assistance provided? (if so, enter TRUE)
Insufficient_Information_Errors <- 0 # how many discrete insufficient information issues did you encounter?
# Assess the causal locus (discrete reproducibility issues) of any reproducibility errors. Note that there doesn't necessarily have to be a one-to-one correspondance between discrete reproducibility issues and reproducibility errors. For example, it could be that the original article neglects to mention that a Greenhouse-Geisser correct was applied to ANOVA outcomes. This might result in multiple reproducibility errors, but there is a single causal locus (discrete reproducibility issue).
locus_typo <- 0 # how many discrete issues did you encounter that related to typographical errors?
locus_specification <- 3 # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis <- 0 # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data <- 0 # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified <- 2 # how many discrete issues were there for which you could not identify the cause
# How many of the above issues were resolved through author assistance?
locus_typo_resolved <- 0 # how many discrete issues did you encounter that related to typographical errors?
locus_specification_resolved <- 3 # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis_resolved <- 0 # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data_resolved <- 0 # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified_resolved <- 0 # how many discrete issues were there for which you could not identify the cause
Affects_Conclusion <- FALSE # Do any reproducibility issues encounter appear to affect the conclusions made in the original article? This is a subjective judgement, but you should taking into account multiple factors, such as the presence/absence of decision errors, the number of target outcomes that could not be reproduced, the type of outcomes that could or could not be reproduced, the difference in magnitude of effect sizes, and the predictions of the specific hypothesis under scrutiny.reportObject <- reportObject %>%
filter(dummyRow == FALSE) %>% # remove the dummy row
select(-dummyRow) %>% # remove dummy row designation
mutate(articleID = articleID) %>% # add the articleID
select(articleID, everything()) # make articleID first column
# decide on final outcome
if(any(!(reportObject$comparisonOutcome %in% c("MATCH", "MINOR_ERROR"))) | Insufficient_Information_Errors > 0){
finalOutcome <- "Failure without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Failure despite author assistance"
}
}else{
finalOutcome <- "Success without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Success with author assistance"
}
}
# collate report extra details
reportExtras <- data.frame(articleID, pilotNames, copilotNames, pilotTTC, copilotTTC, pilotStartDate, copilotStartDate, completionDate, Author_Assistance, finalOutcome, Insufficient_Information_Errors, locus_typo, locus_specification, locus_analysis, locus_data, locus_unidentified, locus_typo_resolved, locus_specification_resolved, locus_analysis_resolved, locus_data_resolved, locus_unidentified_resolved)
# save report objects
if(reportType == "pilot"){
write_csv(reportObject, "pilotReportDetailed.csv")
write_csv(reportExtras, "pilotReportExtras.csv")
}
if(reportType == "final"){
write_csv(reportObject, "finalReportDetailed.csv")
write_csv(reportExtras, "finalReportExtras.csv")
}Session information
devtools::session_info()## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.0.0 (2020-04-24)
## os macOS Catalina 10.15.4
## system x86_64, darwin17.0
## ui X11
## language (EN)
## collate en_US.UTF-8
## ctype en_US.UTF-8
## tz Europe/London
## date 2020-05-13
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date lib
## abind 1.4-5 2016-07-21 [1]
## afex * 0.27-2 2020-03-28 [1]
## assertthat 0.2.1 2019-03-21 [1]
## backports 1.1.6 2020-04-05 [1]
## boot 1.3-24 2019-12-20 [1]
## broom 0.5.6 2020-04-20 [1]
## callr 3.4.3 2020-03-28 [1]
## car 3.0-7 2020-03-11 [1]
## carData 3.0-3 2019-11-16 [1]
## cellranger 1.1.0 2016-07-27 [1]
## cli 2.0.2 2020-02-28 [1]
## colorspace 1.4-1 2019-03-18 [1]
## crayon 1.3.4 2017-09-16 [1]
## curl 4.3 2019-12-02 [1]
## data.table 1.12.8 2019-12-09 [1]
## DBI 1.1.0 2019-12-15 [1]
## dbplyr 1.4.3 2020-04-19 [1]
## desc 1.2.0 2018-05-01 [1]
## devtools 2.3.0 2020-04-10 [1]
## digest 0.6.25 2020-02-23 [1]
## dplyr * 0.8.5 2020-03-07 [1]
## ellipsis 0.3.0 2019-09-20 [1]
## emmeans 1.4.6 2020-04-19 [1]
## estimability 1.3 2018-02-11 [1]
## evaluate 0.14 2019-05-28 [1]
## ez * 4.4-0 2016-11-02 [1]
## fansi 0.4.1 2020-01-08 [1]
## forcats * 0.5.0 2020-03-01 [1]
## foreign 0.8-78 2020-04-13 [1]
## fs 1.4.1 2020-04-04 [1]
## generics 0.0.2 2018-11-29 [1]
## ggplot2 * 3.3.0 2020-03-05 [1]
## glue 1.4.0 2020-04-03 [1]
## gtable 0.3.0 2019-03-25 [1]
## haven * 2.2.0 2019-11-08 [1]
## hms 0.5.3 2020-01-08 [1]
## htmltools 0.4.0 2019-10-04 [1]
## httr 1.4.1 2019-08-05 [1]
## jsonlite 1.6.1 2020-02-02 [1]
## knitr * 1.28 2020-02-06 [1]
## lattice 0.20-41 2020-04-02 [1]
## lifecycle 0.2.0 2020-03-06 [1]
## lme4 * 1.1-23 2020-04-07 [1]
## lmerTest 3.1-2 2020-04-08 [1]
## lsr * 0.5 2015-03-02 [1]
## lubridate 1.7.8 2020-04-06 [1]
## magrittr 1.5 2014-11-22 [1]
## MASS 7.3-51.5 2019-12-20 [1]
## Matrix * 1.2-18 2019-11-27 [1]
## memoise 1.1.0 2017-04-21 [1]
## mgcv 1.8-31 2019-11-09 [1]
## minqa 1.2.4 2014-10-09 [1]
## modelr 0.1.7 2020-04-30 [1]
## munsell 0.5.0 2018-06-12 [1]
## mvtnorm 1.1-0 2020-02-24 [1]
## nlme 3.1-147 2020-04-13 [1]
## nloptr 1.2.2.1 2020-03-11 [1]
## numDeriv 2016.8-1.1 2019-06-06 [1]
## openxlsx 4.1.4 2019-12-06 [1]
## pillar 1.4.4 2020-05-05 [1]
## pkgbuild 1.0.7 2020-04-25 [1]
## pkgconfig 2.0.3 2019-09-22 [1]
## pkgload 1.0.2 2018-10-29 [1]
## plyr 1.8.6 2020-03-03 [1]
## prettyunits 1.1.1 2020-01-24 [1]
## processx 3.4.2 2020-02-09 [1]
## ps 1.3.2 2020-02-13 [1]
## purrr * 0.3.4 2020-04-17 [1]
## R6 2.4.1 2019-11-12 [1]
## Rcpp 1.0.4.6 2020-04-09 [1]
## readr * 1.3.1 2018-12-21 [1]
## readxl * 1.3.1 2019-03-13 [1]
## remotes 2.1.1 2020-02-15 [1]
## reprex 0.3.0 2019-05-16 [1]
## ReproReports * 0.1 2020-05-06 [1]
## reshape2 1.4.4 2020-04-09 [1]
## rio 0.5.16 2018-11-26 [1]
## rlang 0.4.6 2020-05-02 [1]
## rmarkdown 2.1 2020-01-20 [1]
## rprojroot 1.3-2 2018-01-03 [1]
## rstudioapi 0.11 2020-02-07 [1]
## rvest 0.3.5 2019-11-08 [1]
## scales 1.1.0 2019-11-18 [1]
## sessioninfo 1.1.1 2018-11-05 [1]
## statmod 1.4.34 2020-02-17 [1]
## stringi 1.4.6 2020-02-17 [1]
## stringr * 1.4.0 2019-02-10 [1]
## testthat 2.3.2 2020-03-02 [1]
## tibble * 3.0.1 2020-04-20 [1]
## tidyr * 1.0.2 2020-01-24 [1]
## tidyselect 1.0.0 2020-01-27 [1]
## tidyverse * 1.3.0 2019-11-21 [1]
## usethis 1.6.1 2020-04-29 [1]
## vctrs 0.2.4 2020-03-10 [1]
## withr 2.2.0 2020-04-20 [1]
## xfun 0.13 2020-04-13 [1]
## xml2 1.3.2 2020-04-23 [1]
## xtable 1.8-4 2019-04-21 [1]
## yaml 2.2.1 2020-02-01 [1]
## zip 2.0.4 2019-09-01 [1]
## source
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## Github (METRICS-CARPS/CARPSreports@3277f85)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
## CRAN (R 4.0.0)
##
## [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library