CARPS Reproducibility Report
[PILOT/COPILOT - TEXT IN SQUARE BRACKETS IS HERE FOR GUIDANCE. COPILOT PLEASE DELETE BEFORE KNITTING THE FINAL REPORT]
Report Details
articleID <- 2-11-2015 # insert the article ID code here e.g., "10-3-2015_PS"
reportType <- 'Pilot' # specify whether this is the 'pilot' report or 'final' report
pilotNames <- "Pérez, Greses" # insert the pilot's name here e.g., "Tom Hardwicke". If there are multiple cpilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
copilotNames <- "Cachia, Julie Youko Anne" # # insert the co-pilot's name here e.g., "Michael Frank". If there are multiple co-pilots enter both names in a character string e.g., "Tom Hardwicke, Bob Dylan"
pilotTTC <- 1200 # insert the pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
copilotTTC <- 180 # insert the co-pilot's estimated time to complete (in minutes, fine to approximate) e.g., 120
pilotStartDate <- as.Date("11/02/18",format= "%m/%d/%y") # insert the pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
copilotStartDate <- as.Date("11/03/18",format= "%m/%d/%y") # insert the co-pilot's start date in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")
completionDate <- as.Date("11/08/18",format= "%m/%d/%y") # copilot insert the date of final report completion (after any necessary rounds of author assistance) in US format e.g., as.Date("01/25/18", format = "%m/%d/%y")Methods summary:
The authors assamble an experiment where 5-years-olds learned a sets of pictures with pairs of objects associated to a cartoon character to investigate the effects of consolidation on children’s memory by introducing a 1 min delay between learning and testing in the no-delay experiment. hree learning phases, 1 and 3 presented identical contingencies and 2 different contingencies. Darby & Sloutsky (2015) also evaluated the results for interference and savings following a 2 (phase: 1 vs. 3) × 2 (pair type: overlapping vs. unique) analysis of variance (ANOVA).
Target outcomes:
For this article you should focus on the findings reported in the results section for Experiment 1.
Specifically, you should attempt to reproduce all descriptive and inferential analyses reported in the text below and associated tables/figures:
No-delay condition. Results in the no-delay condition pointed to substantial retroactive interference, which was confirmed by a 2 (phase: 1 vs. 3) × 2 (pair type: overlapping vs. unique) repeated measures analysis of variance (ANOVA), which revealed a significant interaction, F(1, 24) = 13.04, p = .001, ηp 2 = .35. Planned comparisons indicated that accuracy significantly decreased for overlapping pairs, t(24) = 6.82, p < .001, Cohen’s d = 1.39, but not for unique pairs, p = .46. To determine the extent of retroactive interference effects on children’s memory, we also measured savings effects. Results pointed to savings for unique but not for overlapping pairs: A 2 (phase: 1 vs. 3) × 2 (pair type: overlapping vs. unique) repeated measures ANOVA revealed a significant interaction, F(1, 22) = 12.02, p = .002, ηp 2 = .33, with savings transpiring for unique pairs, t(24) = 5.82, p < .001, d = 1.19, but not for overlapping pairs, p = .56. Substantial retroactive interference coupled with no savings for overlapping pairs points to catastrophic- like levels of retroactive interference, which replicates our previous findings (Darby & Sloutsky, 2015).
Also attempt to reproduce Figure2 Experiment 1 No Delay only.
Note Make sure to use the original article for additional context and information about any necessary pre-processing steps. Also check for additional supplementary materials that may provide supporting documentation for analysis procedures.
Step 1: Load packages and prepare report object
# load packages
library(tidyverse) # for data munging
library(knitr) # for kable table formating
library(haven) # import and export 'SPSS', 'Stata' and 'SAS' Files
library(readxl) # import excel files
library(CARPSreports) # custom report functions
library('dplyr') # for data manipulation
library('tidyr') # for reshaping data
library('ggplot2') # plotting data
library('scales') # for scale_y_continuous(label = percent)
library('ggthemes') # for scale_fill_few('medium')
knitr::opts_chunk$set(comment = NA)
options(ztable.type = 'html')
library(ez)# Prepare report object. This will be updated automatically by the reproCheck function each time values are compared
reportObject <- data.frame(dummyRow = TRUE, reportedValue = NA, obtainedValue = NA, valueType = NA, percentageError = NA, comparisonOutcome = NA, eyeballCheck = NA)Step 2: Load data
Data <- read_excel("~/Documents/GitHub/CARPS_EXT_2-11-2015/data/All Data for Repository-Final_gp.xlsx")
str(Data)Classes 'tbl_df', 'tbl' and 'data.frame': 2370 obs. of 6 variables:
$ experiment: chr "Experiment 1 No Delay" "Experiment 1 No Delay" "Experiment 1 No Delay" "Experiment 1 No Delay" ...
$ subject : num 3049 3049 3049 3049 3049 ...
$ phase : num 1 1 1 1 1 1 1 1 1 1 ...
$ block : num 1 1 2 2 3 3 4 4 5 5 ...
$ pair type : chr "overlapping" "unique" "overlapping" "unique" ...
$ accuracy : num 1 0.5 1 0.75 1 1 1 1 1 1 ...head(Data)# A tibble: 6 x 6
experiment subject phase block `pair type` accuracy
<chr> <dbl> <dbl> <dbl> <chr> <dbl>
1 Experiment 1 No Delay 3049 1 1 overlapping 1
2 Experiment 1 No Delay 3049 1 1 unique 0.5
3 Experiment 1 No Delay 3049 1 2 overlapping 1
4 Experiment 1 No Delay 3049 1 2 unique 0.75
5 Experiment 1 No Delay 3049 1 3 overlapping 1
6 Experiment 1 No Delay 3049 1 3 unique 1 #Take a look at the data
summary(Data) experiment subject phase block
Length:2370 Min. : 3007 Min. :1 Min. :1
Class :character 1st Qu.: 3034 1st Qu.:1 1st Qu.:2
Mode :character Median : 3059 Median :2 Median :3
Mean : 22472 Mean :2 Mean :3
3rd Qu.: 3136 3rd Qu.:3 3rd Qu.:4
Max. :900010 Max. :3 Max. :5
pair type accuracy
Length:2370 Min. :0.0000
Class :character 1st Qu.:0.7500
Mode :character Median :1.0000
Mean :0.8402
3rd Qu.:1.0000
Max. :1.0000 Step 3: Tidy data
#The data is already tidyStep 4: Run analysis
Pre-processing
# get the data clean for analysis
d <- Data %>%
rename(pair_type = `pair type`)
head(d)# A tibble: 6 x 6
experiment subject phase block pair_type accuracy
<chr> <dbl> <dbl> <dbl> <chr> <dbl>
1 Experiment 1 No Delay 3049 1 1 overlapping 1
2 Experiment 1 No Delay 3049 1 1 unique 0.5
3 Experiment 1 No Delay 3049 1 2 overlapping 1
4 Experiment 1 No Delay 3049 1 2 unique 0.75
5 Experiment 1 No Delay 3049 1 3 overlapping 1
6 Experiment 1 No Delay 3049 1 3 unique 1 Descriptive statistics
#Replication of Table 1 on page 1941 of the article
# summarize by the right grouping factors
d_summary <- d %>%
filter(experiment == "Experiment 1 No Delay") %>%
group_by(experiment, phase, pair_type, block) %>%
summarise(mean = mean(accuracy),
sd = sd(accuracy))
# output step - format into a wide table
d_summary %>%
select(-sd) %>%
spread(block, mean)# A tibble: 6 x 8
# Groups: experiment, phase, pair_type [6]
experiment phase pair_type `1` `2` `3` `4` `5`
<chr> <dbl> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Experiment 1 No Delay 1 overlapping 0.6 0.91 0.91 0.92 0.93
2 Experiment 1 No Delay 1 unique 0.56 0.88 0.88 0.93 0.92
3 Experiment 1 No Delay 2 overlapping 0.48 0.78 0.91 0.88 0.91
4 Experiment 1 No Delay 2 unique 0.7 0.93 0.93 0.97 0.97
5 Experiment 1 No Delay 3 overlapping 0.64 0.85 0.89 0.93 0.95
6 Experiment 1 No Delay 3 unique 0.88 0.93 0.96 0.92 0.94d_summary %>%
select(-mean) %>%
spread(block, sd)# A tibble: 6 x 8
# Groups: experiment, phase, pair_type [6]
experiment phase pair_type `1` `2` `3` `4` `5`
<chr> <dbl> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Experiment 1 No Delay 1 overlapping 0.25 0.142 0.142 0.173 0.153
2 Experiment 1 No Delay 1 unique 0.181 0.193 0.163 0.135 0.157
3 Experiment 1 No Delay 2 overlapping 0.216 0.263 0.122 0.163 0.159
4 Experiment 1 No Delay 2 unique 0.217 0.135 0.135 0.110 0.0829
5 Experiment 1 No Delay 3 overlapping 0.240 0.204 0.192 0.135 0.125
6 Experiment 1 No Delay 3 unique 0.218 0.153 0.0935 0.139 0.109 # Initial calculations for savings
d_savings = d %>%
filter(experiment == "Experiment 1 No Delay") %>%
filter((phase %in% c(1, 3)) & block == 1) %>%
spread(phase, accuracy) %>%
mutate(savings=`3`-`1`)
head(d_savings)# A tibble: 6 x 7
experiment subject block pair_type `1` `3` savings
<chr> <dbl> <dbl> <chr> <dbl> <dbl> <dbl>
1 Experiment 1 No Delay 3049 1 overlapping 1 0.75 -0.25
2 Experiment 1 No Delay 3049 1 unique 0.5 0.75 0.25
3 Experiment 1 No Delay 3052 1 overlapping 0.75 0.5 -0.25
4 Experiment 1 No Delay 3052 1 unique 0.5 1 0.5
5 Experiment 1 No Delay 3053 1 overlapping 0.75 0.5 -0.25
6 Experiment 1 No Delay 3053 1 unique 0.75 1 0.25# Spread savings by pair type
d_savings_split <- d_savings%>%
select(-block)%>%
select(-`1`)%>%
select(-`3`)%>%
spread(pair_type, savings)
head(d_savings_split)# A tibble: 6 x 4
experiment subject overlapping unique
<chr> <dbl> <dbl> <dbl>
1 Experiment 1 No Delay 3049 -0.25 0.25
2 Experiment 1 No Delay 3052 -0.25 0.5
3 Experiment 1 No Delay 3053 -0.25 0.25
4 Experiment 1 No Delay 3056 0.25 0
5 Experiment 1 No Delay 3066 0 0.5
6 Experiment 1 No Delay 3068 -0.75 -0.25# Initial calculations for retroactive interference
d_pre_ri = d %>%
filter(experiment == "Experiment 1 No Delay") %>%
filter((phase == 1 & block == 5) | (phase == 3 & block ==1 ))%>%
select(-block)%>%
spread(phase, accuracy)%>%
mutate(pre_ri =`1` - `3`) #This is Phase 1 Block 5 and Phase 3 Block 1 separately for overlapping and unique pairs as indicated on p.1939 from the article
head(d_pre_ri)# A tibble: 6 x 6
experiment subject pair_type `1` `3` pre_ri
<chr> <dbl> <chr> <dbl> <dbl> <dbl>
1 Experiment 1 No Delay 3049 overlapping 1 0.75 0.25
2 Experiment 1 No Delay 3049 unique 1 0.75 0.25
3 Experiment 1 No Delay 3052 overlapping 1 0.5 0.5
4 Experiment 1 No Delay 3052 unique 0.75 1 -0.25
5 Experiment 1 No Delay 3053 overlapping 1 0.5 0.5
6 Experiment 1 No Delay 3053 unique 1 1 0 # Calculation of retroactive interference (named as "rio" in this code)
# I attempted to create figure 2 on page 1941 as requested for this project, but I was unable to replicate it and given the fact that I have spend a considerable amount of time in this project, I decided to move on to the next problem set. If someone gets to do it in the near future, I would love to have access to the complete code (greses@stanford.edu)
d_ri <- d_pre_ri %>%
select(-`1`,-`3`)%>%
spread(pair_type, pre_ri)%>%
mutate(rio = overlapping-unique)%>% #This is determining the difference between results for each pair type as indicated on p.1939 from the article
mutate(mean_rio=mean(rio))
head(d_ri) # A tibble: 6 x 6
experiment subject overlapping unique rio mean_rio
<chr> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Experiment 1 No Delay 3049 0.25 0.25 0 0.25
2 Experiment 1 No Delay 3052 0.5 -0.25 0.75 0.25
3 Experiment 1 No Delay 3053 0.5 0 0.5 0.25
4 Experiment 1 No Delay 3056 0 0 0 0.25
5 Experiment 1 No Delay 3066 0.25 0 0.25 0.25
6 Experiment 1 No Delay 3068 0.25 0.5 -0.25 0.25ggplot(d_ri, aes(x = "Retroactive Interference", y = mean_rio)) +
geom_count() # I did not know how to turn these values into rows, so I could create teh graph as shown on page p. 1939 of the study. 
# Calculations to take the data apart to graph figure 1 and 2 of the paper. I was unable to bring together the savings data and retroactive inference, so I could not graph figure 1.
total <- merge(d_savings_split, d_ri, by="subject")%>%
select(-`unique.y`,-`overlapping.y`,-`experiment.y`)%>%
mutate(mean_overlapping=mean(`overlapping.x`))%>%
mutate(mean_unique=mean(`unique.x`))
head(total) subject experiment.x overlapping.x unique.x rio mean_rio
1 3049 Experiment 1 No Delay -0.25 0.25 0.00 0.25
2 3052 Experiment 1 No Delay -0.25 0.50 0.75 0.25
3 3053 Experiment 1 No Delay -0.25 0.25 0.50 0.25
4 3056 Experiment 1 No Delay 0.25 0.00 0.00 0.25
5 3066 Experiment 1 No Delay 0.00 0.50 0.25 0.25
6 3068 Experiment 1 No Delay -0.75 -0.25 -0.25 0.25
mean_overlapping mean_unique
1 0.04 0.32
2 0.04 0.32
3 0.04 0.32
4 0.04 0.32
5 0.04 0.32
6 0.04 0.32# As described in the text above, I was unable to replicate the graphs 1 and 2
ggplot(total, aes(x = "rio", y = mean_rio), aes(x2="unique", y2=mean_unique)) +
geom_col()
Inferential statistics
substantial retroactive interference, confirmed by a 2 (phase: 1 vs. 3) × 2 (pair type: overlapping vs. unique) repeated measures analysis of variance (ANOVA), which revealed a significant interaction, F(1, 24) = 13.04, p = .001, ηp 2 = .35.
d_anova <-d %>%
filter(experiment == "Experiment 1 No Delay") %>%
filter((phase == 1 & block == 5) | (phase == 3 & block ==1 ))
View(d_anova)
model_anova <-aov(accuracy ~ phase + pair_type, data=d_anova)
summary.aov(model_anova) #I have followed the steps indicated in the study, but the description of the analysis in the paper is very ambiguous. I consulted with the TA's and they said that this paper is a good example of "what not to do when describing the analysis" Df Sum Sq Mean Sq F value Pr(>F)
phase 1 0.681 0.6806 16.219 0.000112 ***
pair_type 1 0.331 0.3306 7.879 0.006047 **
Residuals 97 4.071 0.0420
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1##Planned comparisons indicated that accuracy significantly decreased for overlapping pairs, t(24) = 6.82, p < .001, Cohen’s d = 1.39
t <- t.test(d_ri$overlapping)
head(t)$statistic
t
6.819598
$parameter
df
24
$p.value
[1] 4.711565e-07
$conf.int
[1] 0.2022337 0.3777663
attr(,"conf.level")
[1] 0.95
$estimate
mean of x
0.29
$null.value
mean
0 # accuracy did not decrease for unique pairs, p = .46.
t.ri.u <- t.test(d_ri$unique)
head(t.ri.u)$statistic
t
0.7492686
$parameter
df
24
$p.value
[1] 0.4609768
$conf.int
[1] -0.07018203 0.15018203
attr(,"conf.level")
[1] 0.95
$estimate
mean of x
0.04
$null.value
mean
0 #To determine the extent of retroactive interference effects on children’s memory, we also measured savings effects. Results pointed to savings for unique but not for overlapping pairs: A 2 (phase: 1 vs. 3) × 2 (pair type: overlapping vs. unique) repeated measures ANOVA revealed a significant interaction, F(1, 22) = 12.02, p = .002, ηp 2 = .33
# I have followed the steps indicated in the study, but the description of the analysis in the paper is very ambiguous. I consulted with the TA's and they said that this paper is a good example of "what not to do when describing the analysis"
das <- d%>%
filter(experiment=="Experiment 1 No Delay" & block==1)%>%
filter((phase==1) |(phase==3))
View(das)
model_das <-aov(accuracy~ phase + pair_type, data = das)
summary.aov(model_das) Df Sum Sq Mean Sq F value Pr(>F)
phase 1 0.81 0.8100 14.825 0.000212 ***
pair_type 1 0.25 0.2500 4.575 0.034943 *
Residuals 97 5.30 0.0546
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1# savings transpiring for unique pairs, t(24) = 5.82, p < .001, d = 1.19, but not for overlapping pairs, p = .56.
t.savings <- t.test(d_savings_split$unique)
head(t.savings)$statistic
t
5.818182
$parameter
df
24
$p.value
[1] 5.336859e-06
$conf.int
[1] 0.2064856 0.4335144
attr(,"conf.level")
[1] 0.95
$estimate
mean of x
0.32
$null.value
mean
0 t.overlapping <- t.test(d_savings_split$overlapping) # savings did not transpire for overlapping pairs, p= .56
head(t.overlapping)$statistic
t
0.5951836
$parameter
df
24
$p.value
[1] 0.5572916
$conf.int
[1] -0.09870668 0.17870668
attr(,"conf.level")
[1] 0.95
$estimate
mean of x
0.04
$null.value
mean
0 Step 5: Conclusion
The findings of this replication suggest a substantial retroactive interference confirmed by the analysis of variance with savings only for unique pairs, but not for overlapping pairs. The t-test analysis suggest that accuracy decrease fro overlapping pairs and increase for unique pairs.
The reproducibility check was not a failure, but parts of the study were not replicated. The description of the analysis of variance on page 1940 are ambiguous and it is not clear what the authors did in the analysis. Therefore, when running the anova, I was not clear on which ones are the variables under consideration in the model and the nature of the model.
Author_Assistance = FALSE # was author assistance provided? (if so, enter TRUE)
Insufficient_Information_Errors <- 1 # how many discrete insufficient information issues did you encounter?
# Assess the causal locus (discrete reproducibility issues) of any reproducibility errors. Note that there doesn't necessarily have to be a one-to-one correspondance between discrete reproducibility issues and reproducibility errors. For example, it could be that the original article neglects to mention that a Greenhouse-Geisser correct was applied to ANOVA outcomes. This might result in multiple reproducibility errors, but there is a single causal locus (discrete reproducibility issue).
locus_typo <- 3 # how many discrete issues did you encounter that related to typographical errors?
locus_specification <- 2 # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis <- NA # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data <- NA # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified <- 1 # how many discrete issues were there for which you could not identify the cause
# How many of the above issues were resolved through author assistance?
locus_typo_resolved <- NA # how many discrete issues did you encounter that related to typographical errors?
locus_specification_resolved <- NA # how many discrete issues did you encounter that related to incomplete, incorrect, or unclear specification of the original analyses?
locus_analysis_resolved <- NA # how many discrete issues did you encounter that related to errors in the authors' original analyses?
locus_data_resolved <- NA # how many discrete issues did you encounter that related to errors in the data files shared by the authors?
locus_unidentified_resolved <- 2 # how many discrete issues were there for which you could not identify the cause
Affects_Conclusion <- FALSE # Do any reproducibility issues encounter appear to affect the conclusions made in the original article? TRUE, FALSE, or NA. This is a subjective judgement, but you should taking into account multiple factors, such as the presence/absence of decision errors, the number of target outcomes that could not be reproduced, the type of outcomes that could or could not be reproduced, the difference in magnitude of effect sizes, and the predictions of the specific hypothesis under scrutiny.reportObject <- reportObject %>%
filter(dummyRow == FALSE) %>% # remove the dummy row DO NOT CHANGE ANYTHING BELOW HERE
select(-dummyRow) %>% # remove dummy row designation
mutate(articleID = articleID) %>% # add variables to report
select(articleID, everything()) # make articleID first column
# decide on final outcome
if(any(reportObject$comparisonOutcome %in% c("MAJOR_ERROR", "DECISION_ERROR")) | Insufficient_Information_Errors > 0){
finalOutcome <- "Failure without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Failure despite author assistance"
}
}else{
finalOutcome <- "Success without author assistance"
if(Author_Assistance == T){
finalOutcome <- "Success with author assistance"
}
}
# collate report extra details
reportExtras <- data.frame(articleID, pilotNames, copilotNames, pilotTTC, copilotTTC, pilotStartDate, copilotStartDate, completionDate, Author_Assistance, finalOutcome, Insufficient_Information_Errors, locus_typo, locus_specification, locus_analysis, locus_data, locus_unidentified, locus_typo_resolved, locus_specification_resolved, locus_analysis_resolved, locus_data_resolved, locus_unidentified_resolved)
# save report objects
if(reportType == "pilot"){
write_csv(reportObject, "pilotReportDetailed.csv")
write_csv(reportExtras, "pilotReportExtras.csv")
}
if(reportType == "final"){
write_csv(reportObject, "finalReportDetailed.csv")
write_csv(reportExtras, "finalReportExtras.csv")
}Session information
[This function will output information about the package versions used in this report:]
devtools::session_info()─ Session info ──────────────────────────────────────────────────────────
setting value
version R version 3.5.1 (2018-07-02)
os macOS High Sierra 10.13.6
system x86_64, darwin15.6.0
ui X11
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz America/Los_Angeles
date 2018-11-08
─ Packages ──────────────────────────────────────────────────────────────
package * version date lib
abind 1.4-5 2016-07-21 [1]
assertthat 0.2.0 2017-04-11 [1]
backports 1.1.2 2017-12-13 [1]
base64enc 0.1-3 2015-07-28 [1]
bindr 0.1.1 2018-03-13 [1]
bindrcpp * 0.2.2 2018-03-29 [1]
broom 0.5.0 2018-07-17 [1]
callr 3.0.0 2018-08-24 [1]
car 3.0-2 2018-08-23 [1]
carData 3.0-2 2018-09-30 [1]
CARPSreports * 0.1 2018-11-03 [1]
cellranger 1.1.0 2016-07-27 [1]
cli 1.0.1 2018-09-25 [1]
colorspace 1.3-2 2016-12-14 [1]
crayon 1.3.4 2017-09-16 [1]
curl 3.2 2018-03-28 [1]
data.table 1.11.8 2018-09-30 [1]
desc 1.2.0 2018-05-01 [1]
devtools 2.0.1 2018-10-26 [1]
digest 0.6.18 2018-10-10 [1]
dplyr * 0.7.7 2018-10-16 [1]
evaluate 0.12 2018-10-09 [1]
ez * 4.4-0 2016-11-02 [1]
fansi 0.4.0 2018-10-05 [1]
forcats * 0.3.0 2018-02-19 [1]
foreign 0.8-70 2017-11-28 [1]
fs 1.2.6 2018-08-23 [1]
ggplot2 * 3.1.0 2018-10-25 [1]
ggthemes * 4.0.1 2018-08-24 [1]
glue 1.3.0 2018-07-17 [1]
gtable 0.2.0 2016-02-26 [1]
haven * 1.1.2 2018-06-27 [1]
hms 0.4.2 2018-03-10 [1]
htmltools 0.3.6 2017-04-28 [1]
httr 1.3.1 2017-08-20 [1]
jsonlite 1.5 2017-06-01 [1]
knitr * 1.20 2018-02-20 [1]
labeling 0.3 2014-08-23 [1]
lattice 0.20-35 2017-03-25 [1]
lazyeval 0.2.1 2017-10-29 [1]
lme4 1.1-18-1 2018-08-17 [1]
lubridate 1.7.4 2018-04-11 [1]
magrittr 1.5 2014-11-22 [1]
MASS 7.3-50 2018-04-30 [1]
Matrix 1.2-14 2018-04-13 [1]
memoise 1.1.0 2017-04-21 [1]
mgcv 1.8-24 2018-06-23 [1]
minqa 1.2.4 2014-10-09 [1]
modelr 0.1.2 2018-05-11 [1]
munsell 0.5.0 2018-06-12 [1]
nlme 3.1-137 2018-04-07 [1]
nloptr 1.2.1 2018-10-03 [1]
openxlsx 4.1.0 2018-05-26 [1]
pillar 1.3.0 2018-07-14 [1]
pkgbuild 1.0.2 2018-10-16 [1]
pkgconfig 2.0.2 2018-08-16 [1]
pkgload 1.0.2 2018-10-29 [1]
plyr 1.8.4 2016-06-08 [1]
prettyunits 1.0.2 2015-07-13 [1]
processx 3.2.0 2018-08-16 [1]
ps 1.2.0 2018-10-16 [1]
purrr * 0.2.5 2018-05-29 [1]
R6 2.3.0 2018-10-04 [1]
Rcpp 0.12.19 2018-10-01 [1]
readr * 1.1.1 2017-05-16 [1]
readxl * 1.1.0 2018-04-20 [1]
remotes 2.0.2 2018-10-30 [1]
reshape2 1.4.3 2017-12-11 [1]
rio 0.5.10 2018-03-29 [1]
rlang 0.3.0.1 2018-10-25 [1]
rmarkdown 1.10 2018-06-11 [1]
rprojroot 1.3-2 2018-01-03 [1]
rstudioapi 0.8 2018-10-02 [1]
rvest 0.3.2 2016-06-17 [1]
scales * 1.0.0 2018-08-09 [1]
sessioninfo 1.1.0 2018-09-25 [1]
stringi 1.2.4 2018-07-20 [1]
stringr * 1.3.1 2018-05-10 [1]
testthat 2.0.0 2017-12-13 [1]
tibble * 1.4.2 2018-01-22 [1]
tidyr * 0.8.2 2018-10-28 [1]
tidyselect 0.2.5 2018-10-11 [1]
tidyverse * 1.2.1 2017-11-14 [1]
usethis 1.4.0 2018-08-14 [1]
utf8 1.1.4 2018-05-24 [1]
withr 2.1.2 2018-03-15 [1]
xml2 1.2.0 2018-01-24 [1]
yaml 2.2.0 2018-07-25 [1]
zip 1.0.0 2017-04-25 [1]
source
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
Github (METRICS-CARPS/CARPSreports@89db4a9)
CRAN (R 3.5.0)
CRAN (R 3.5.1)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.1)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.1)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.1)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.1)
CRAN (R 3.5.1)
CRAN (R 3.5.0)
CRAN (R 3.5.1)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.1)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
CRAN (R 3.5.0)
[1] /Library/Frameworks/R.framework/Versions/3.5/Resources/library