A generative model for vocabulary estimation
Mike Frank
July 17, 2015
We’re going to use Stan. Set some settings:
library(rstan)
library(rstanmulticore)
rstan_options(auto_write = TRUE)
options(mc.cores = parallel::detectCores())First, connect to the Wordbank database and pull out the English WS and WG data.
wordbank <- connect_to_wordbank()
admins <- get_administration_data() %>%
filter(language == "English")
items <- get_item_data() %>%
filter(language == "English", type == "word") %>%
select(item_id, form, category, lexical_category, item) %>%
mutate(num_item_id = as.numeric(substr(item_id, 6, nchar(item_id))))
get_form_data <- function(input_language, input_form) {
form_items <- filter(items, form == input_form)
get_instrument_data(input_language, input_form,
form_items$item_id, iteminfo=form_items) %>%
mutate(produces = value == "produces",
understands = value == "understands" | value == "produces") %>%
select(-value) %>%
gather(measure, value, produces, understands) %>%
left_join(select(admins, data_id, age)) %>%
filter(!is.na(age)) %>%
mutate(form = input_form)
}
ws <- get_form_data("English", "WS") %>%
filter(measure == "produces")Now aggregate by item.
items <- ws %>%
group_by(item, age) %>%
summarise(produces = mean(value, na.rm=TRUE))
write_csv(items, "ws.csv")Now arrange.
ordered <- items %>%
group_by() %>%
arrange(age, desc(produces)) %>%
group_by(age) %>%
mutate(index = 1:n()) Plot this with a glm sinusoid like in the Mayor & Plunkett (2011) paper.
qplot(index, produces, facets= ~age,
geom = "line",
data = ordered) +
ylim(c(0,1)) +
geom_smooth(method = "glm",
family = "binomial",
formula = y ~ x) 
Try a polynomial fit.
qplot(index, produces, facets= ~age,
geom = "line",
data = ordered) +
ylim(c(0,1)) +
geom_smooth(method = "lm",
family = "binomial",
color = "red",
formula = y ~ poly(x, 3)) 
Doesn’t work that well for the younger ages, though looks fine later.
Note age interactions in the stats.
mp.glm <- glm(produces ~ index * age, family = "binomial", data = ordered)
summary(mp.glm)##
## Call:
## glm(formula = produces ~ index * age, family = "binomial", data = ordered)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -0.51045 -0.09580 -0.01266 0.07168 1.16561
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -4.695e+00 2.839e-01 -16.538 < 2e-16 ***
## index -7.769e-03 8.480e-04 -9.161 < 2e-16 ***
## age 2.640e-01 1.251e-02 21.100 < 2e-16 ***
## index:age 9.173e-05 3.489e-05 2.629 0.00855 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 4140.24 on 10199 degrees of freedom
## Residual deviance: 175.65 on 10196 degrees of freedom
## AIC: 5984.7
##
## Number of Fisher Scoring iterations: 5Further thoughts on Mayor & Plunkett model
The Mayor & Plunkett (2011) model has two corrections. The first uses the parametric form of the logistic to fill in low-frequency words that are not on the CDI, the second uses the difference between diary study counts and CDI counts to find a multiplier for higher-frequency words that are omitted.
I think there’s a bit of a conceptual issue here, as these two corrections should essentially be the same thing - there are some words that are not on the CDI, and more of these are the low frequency/hard words. So really, it all is a correction for missing words.
Also - the first correction, which depends on the parametric form of the logistic, is much much smaller than the second. Take a look at this.
Equation 3:
\[ p(w_i) = 1 - \frac{1}{1 + e^{\frac{-(i-a)}{b}}} \]
a <- 600 # from figure 6c for a 30mo
b <- 180 # from figure 6c
ranks <- 0:3000
age <- 20
ys <- (1 - (1 / (1 + exp((-(ranks - a ))/b))))
qplot(ranks, ys,
geom = "line") +
geom_vline(xintercept = 680, lty = 2) +
geom_polygon(aes(x = c(ranks[ranks > 680],
rev(ranks[ranks > 680])),
y = c(ys[ranks > 680],
rep(0, length(ranks[ranks > 680])))),
fill = "blue", alpha = .5) +
ylim(c(0,1))
So adding the gray area gives us
\[ V_{corr_1} = b \log (1 + e^(a / b)) \]
but then adding the second correction is just a multiplier on this:
\[ V_{corr_2} = \alpha * V_{corr_1} \]
Note that (strikingly), M&P2011 never give their value of \(\alpha\) in the text. I estimate it below so that I can make an estimate of what correction 2 actually looks like…
a <- 600
b <- 180
ranks <- 1:3000
age <- 20
ys <- (1 - (1 / (1 + exp((-(ranks - a ))/b))))
area.uncorrected <- sum(ys[1:680])
area.c1 <- b * log(1 + exp(a / b))
area.c2 <- 1142 # from lookup table A1
alpha = area.c2 / area.c1
ys2 <- (1 - (1 / (1 + exp((-(ranks - (a*alpha) ))/(alpha * b)))))
area.corrected <- sum(ys2)
qplot(ranks, ys,
geom = "line") +
geom_vline(xintercept = 680, lty = 2) +
geom_polygon(aes(x = c(ranks[ranks > 680],
rev(ranks[ranks > 680])),
y = c(ys[ranks > 680],
rep(0, length(ranks[ranks > 680])))),
fill = "blue", alpha = .5) +
geom_line(aes(y = ys2), lty = 3) +
geom_polygon(aes(x = c(ranks, rev(ranks)),
y = c(ys2, rev(ys))),
fill = "red",
alpha = .5) +
ylim(c(0,1)) 
So you can see that the second correction dwarfs the first correction in size, and is really based on a few small diary studies.
In sum, I’m worried about this model for a few reasons:
- The distributional form (logit) is clearly not correct, so using this distributional form for extrapolation may have bad consequences.
- The first and second corrections aren’t conceptually distinct: they both concern missing words. Both have to do with the sampling of words on the CDI from the broader vocabulary.
- The second correction, which does most of the work, is - for reasonable reasons of data etc.- assumed to be a strict multiplier, which makes it do a ton of work at the higher end of vocabulary.
McMurray Model
This is the normal distribution model from the McMurray (2007) “Defusing the Vocabulary Explosion” paper. Assume kids are \(k\) and words \(w\). Kids’ abilities are a function of age (\(a\)). We model production probability \(p\).
\[ k_i \sim \mathcal{N}(\mu_1, \sigma_1) \\ w_j \sim \mathcal{N}(\mu_2, \sigma_2) \\ k_i(a) = k_i * a \\ p(k_i, w_j, a) = k_i(a) > w_j \\ \]
Try this with arbitrary parameters and make the same plot as above.
n.words <- 500
n.kids <- 1000
difficulty <- rnorm(n = n.words, m = 100, sd = 20)
ability <- rnorm(n = n.kids, m = 4, sd = 1)
age <- 16:30
sims <- expand.grid(difficulty = difficulty, ability = ability, age = age) %>%
group_by(ability, age) %>%
mutate(item = 1:n()) %>%
group_by() %>%
mutate(produces = difficulty < ability * age) %>%
group_by(item, age) %>%
summarise(produces = mean(produces)) %>%
group_by() %>%
arrange(age, desc(produces)) %>%
group_by(age) %>%
mutate(index = 1:n())
qplot(index, produces, facets = ~age,
geom = "line",
data = sims)
Cool - so this looks almost identical in distributional form! Just for kicks, let’s plot these on top of one another. For now I’m just tweaking parameters, though all four (\(\mu_1\), \(\mu_2\), \(\sigma_1\), and \(\sigma_2\)) matter to the fit.
n.words <- 680 # same as on the CDI
n.kids <- 500 # reasonable number, so the curves are smoothish
difficulty <- rnorm(n = n.words, m = 100, sd = 20)
ability <- rnorm(n = n.kids, m = 4, sd = 1)
age <- 16:30
sims <- expand.grid(difficulty = difficulty, ability = ability, age = age) %>%
group_by(ability, age) %>%
mutate(item = 1:n()) %>%
group_by() %>%
mutate(produces = difficulty < ability * age) %>%
group_by(item, age) %>%
summarise(produces = mean(produces)) %>%
group_by() %>%
arrange(age, desc(produces)) %>%
group_by(age) %>%
mutate(index = 1:n())
sims$dataset <- "simulations"
ordered$dataset <- "empirical WS"
d <- bind_rows(select(sims, index, produces, age, dataset),
select(ordered, index, produces, age, dataset))
qplot(index, produces, facets = ~age,
geom = "line",
col = dataset,
data = d)
Extending the McMurray Model
So if the McMurray model is more or less the right model of vocabulary growth, how do we use knowledge about its parameters to correct CDI data? Two points here.
First, we have to define a process by which words are excluded and then estimate the parameters of the full model based on the CDI model.
Second, to do this we’re still going to need some facts about total vocabulary to compare to the CDI vocabulary estimates, so we’ll still need a study like Robinson & Mervis (1999).
So let’s consider the McMurray model as the base of our generative model, having the steps:
- Generate a vocabulary with normally distributed difficulties,
- Generate a set of kids with normally distributed learning rates, and
- Generate a set of words to be on an instrument.
Let’s consider a model of selecting words for an instrument where the probability of selecting a word gradually decreases with its difficulty rank. The easiest words (e.g. momma, no, ball) will all be on the list, but only a smaller number of hard words will make it.
\[ p_{inclusion}(i) \sim \operatorname{Bern} (\frac{1}{i^{\alpha}}) \]
So each word is included as a function of a Bernoulli draw (coin flip) with probability proportional to its rank.
xs <- 1:100
ys <- 1 / xs^(1/4)
qplot(xs, ys)
So let’s see what this looks like. We’ll do the same generative process as before but this time with many more words, which we’ll exclude at random.
n.words <- 4000
n.kids <- 100
alpha = .25
difficulty <- rnorm(n = n.words, m = 100, sd = 20)
ability <- rnorm(n = n.kids, m = 4, sd = 1)
age <- 16:30
sims <- expand.grid(difficulty = difficulty, ability = ability, age = age) %>%
group_by(ability, age) %>%
mutate(item = 1:n()) %>%
group_by() %>%
mutate(produces = difficulty < ability * age) %>%
group_by(item, age) %>%
summarise(produces = mean(produces)) %>%
group_by() %>%
arrange(age, desc(produces)) %>%
group_by(age) %>%
mutate(index = 1:n())Now choose the words to exclude, based on actual ground-truth difficulty.
words.to.include <- expand.grid(difficulty = difficulty) %>%
mutate(item = 1:n()) %>%
arrange(difficulty) %>%
mutate(diff.idx = 1:n()) %>%
rowwise %>%
mutate(inc = rbinom(1, size = 1, prob = 1/diff.idx^alpha) == 1) %>%
filter(inc) %>%
rowwise %>%
mutate(sim.idx = sims$index[sims$item == item][1])First visualize this with the words, with red being more included words.
qplot(index, produces, facets = ~age,
geom = "line",
data = sims) +
geom_vline(data = words.to.include,
aes(xintercept = sim.idx),
col = "red", alpha = .03)
Next show the curves with only those words included.
sims.subset <- sims %>%
filter(item %in% words.to.include$item) %>%
mutate(index = 1:n())
sims$dataset <- "full simulation"
sims.subset$dataset <- "item subset"
d <- bind_rows(sims, sims.subset)
qplot(index, produces, facets = ~age,
geom = "line", col = dataset,
data = d) 
Now plot the multiplier in this simulation, which we can get based on the area under the curve. (As M&P2011 note, the area under the curve is the expected vocabulary estimate for that measure).
ms <- d %>%
group_by(age, dataset) %>%
summarise(produces = sum(produces))
qplot(age, produces, col = dataset,
geom = "line", data = ms) +
geom_hline(yintercept = 680, lty = 2) +
geom_hline(yintercept = 3000, lty = 2)
It’s pretty clear that we’re actually getting something similar to Bernard & Mervis (1999) here, and our correction isn’t looking that dissimilar to M&P2011. Of course, this is all with six parameters I made up: \(\mu_1\), \(\mu_2\), \(\sigma_1\), \(\sigma_2\), \(\alpha\) (the parameter on the words that got dropped), and \(N_{total}\) (the total number of words we considered). But we (sort of) fit the first four to the empriical data. And the combo of \(\alpha\) and \(N_{total}\) wasn’t totally invented either.
But really the next step is to use the empirical data and B&M1999 (plus other data) to fit this model, since with the addition of some hyper-priors it’s a well-defined generative model.
Inferring Model Parameters
Let’s start slow, first defining the McMurray model.
mcmurray <-'
data {
int<lower=1> W; // number of words
int<lower=1> S; // number of subjects
int<lower=1> N; // number of datapoints (W*S)
vector[S] age; // subject ages
int<lower=0,upper=1> produces[N]; // actual data
}
parameters {
real<lower=0> mu_w; // word mean
real<lower=0> mu_s; // subject mean
real<lower=0> sigma_w; // word SD
real<lower=0> sigma_s; // subject SD
real<lower=0> difficulty[W]; // word mean
real<lower=0> ability[S]; // subject mean
}
model {
mu_w ~ normal(0, 10);
mu_s ~ normal(0, 10);
sigma_w ~ normal(0, 5);
sigma_s ~ normal(0, 5);
difficulty ~ normal(mu_w, sigma_w);
ability ~ normal(mu_s, sigma_s);
for (s in 1:S)
for (w in 1:W)
produces[W*(s-1) + w] ~ bernoulli_logit((ability[s] * age[s]) - difficulty[w]);
}
'Fake data simulations.
Let’s start by doing simulations with fake data. These are six kids and five words, with the kids gradually increasing in age. Their abilities should be lower and lower, and words 1, 3, and 5 should be estimated as hard.
dat <- list(age = c(12, 14, 16, 18, 20, 22),
produces = rep(c(0, 1, 0, 1, 0), 6),
W = 5,
S = 6,
N = 30)Run!
samps <- pstan(model_code = mcmurray, model_name = "mcmurray_simple",
data = dat, iter = 2000, warmup=200, thin=20, chains = 4,
pars = c('mu_w','mu_s','sigma_w', 'sigma_s',
'difficulty', 'ability'),
verbose = F, refresh=2000/4) Now examine output.
samps## Inference for Stan model: mcmurray_simple.
## 4 chains, each with iter=2000; warmup=200; thin=20;
## post-warmup draws per chain=90, total post-warmup draws=360.
##
## mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
## mu_w 13.29 0.39 5.25 3.28 9.71 13.07 17.01 24.50 183 1.01
## mu_s 0.67 0.04 0.31 0.25 0.43 0.58 0.90 1.34 55 1.04
## sigma_w 8.30 0.23 2.87 3.60 6.25 7.98 9.89 15.01 157 1.02
## sigma_s 0.17 0.01 0.21 0.02 0.06 0.11 0.20 0.73 291 1.00
## difficulty[1] 21.53 0.44 8.12 9.38 15.70 20.68 26.06 41.14 339 1.00
## difficulty[2] 4.27 0.36 3.50 0.10 1.46 3.38 6.55 12.66 95 1.02
## difficulty[3] 21.18 0.47 8.05 10.08 14.90 19.80 25.74 39.36 290 1.00
## difficulty[4] 4.59 0.52 3.67 0.18 1.99 3.64 6.27 13.32 51 1.04
## difficulty[5] 21.29 0.46 7.98 8.50 15.88 20.32 26.19 38.70 296 1.00
## ability[1] 0.73 0.03 0.37 0.24 0.45 0.64 1.00 1.67 151 1.02
## ability[2] 0.68 0.04 0.32 0.23 0.43 0.60 0.93 1.33 56 1.04
## ability[3] 0.67 0.04 0.32 0.23 0.42 0.59 0.86 1.38 61 1.04
## ability[4] 0.66 0.04 0.31 0.20 0.41 0.60 0.89 1.33 71 1.03
## ability[5] 0.65 0.03 0.30 0.23 0.42 0.56 0.85 1.33 95 1.03
## ability[6] 0.60 0.04 0.28 0.21 0.39 0.53 0.79 1.17 44 1.05
## lp__ 2.60 0.72 5.45 -8.57 -0.85 2.41 6.19 12.45 57 1.05
##
## Samples were drawn using NUTS(diag_e) at Tue Aug 4 21:08:31 2015.
## For each parameter, n_eff is a crude measure of effective sample size,
## and Rhat is the potential scale reduction factor on split chains (at
## convergence, Rhat=1).Looks reasonable to me.
Simulations with a subset of the data.
I started out these simulations looking at many words, and a small number of kids, and the results looked funky. Not sure if that’s a convergence issue, or an issue with not having the appropriate diversity of kids. So let’s try a smaller number of words.
n.words <- 6
n.subs <- 500
model.data <- ws %>%
filter(data_id <= n.subs,
item %in% c("mommy","dog","up","table","truck","alligator")) %>%
select(data_id, num_item_id, item, value, age) %>%
rename(produces = value) %>%
arrange(data_id) # critical to get the words nested inside subjects
# for simulations, fill in false for missing data
model.data$produces[is.na(model.data$produces)] <- FALSE
ages <- model.data %>%
group_by(data_id) %>%
summarise(age = mean(age)) %>%
select(age)
dat <- list(age = ages$age,
produces = model.data$produces,
W = n.words,
S = n.subs,
N = n.words * n.subs)
model.data %>%
group_by(num_item_id) %>%
summarise(produces = mean(produces))## Source: local data frame [6 x 2]
##
## num_item_id produces
## 1 13 0.264
## 2 26 0.878
## 3 69 0.670
## 4 290 0.448
## 5 366 0.948
## 6 635 0.564Run the model.
samps <- pstan(model_code = mcmurray, model_name = "mcmurray_simple",
data = dat, iter = 2000, warmup=200, thin=20, chains = 4,
pars = c('mu_w','mu_s','sigma_w', 'sigma_s',
'difficulty', 'ability'),
verbose = F, refresh=2000/4) Diagnostics. Order of words is “alligator”, “dog”, “truck”, “table”, “mommy”, “up.”
traceplot(samps, pars = c("mu_w", "mu_s", "sigma_w", "sigma_s"))
samplerpar = get_sampler_params(samps)[[1]]
summary(samplerpar)## accept_stat__ stepsize__ treedepth__ n_leapfrog__
## Min. :0.0006583 Min. :0.007812 Min. :2.00 Min. : 3.00
## 1st Qu.:0.9101136 1st Qu.:0.119155 1st Qu.:5.00 1st Qu.: 31.00
## Median :0.9566540 Median :0.119155 Median :5.00 Median : 31.00
## Mean :0.8885530 Mean :0.123745 Mean :4.93 Mean : 31.04
## 3rd Qu.:0.9815654 3rd Qu.:0.119155 3rd Qu.:5.00 3rd Qu.: 31.00
## Max. :1.0000000 Max. :0.514698 Max. :7.00 Max. :127.00
## n_divergent__
## Min. :0.00
## 1st Qu.:0.00
## Median :0.00
## Mean :0.01
## 3rd Qu.:0.00
## Max. :1.00pairs(samps, pars=c('mu_w', 'mu_s'))
Simulations with full dataset
We need to use only complete cases. But even here, these simulations take a while.
n.words <- 680
complete <- ws %>%
group_by(data_id) %>%
summarize(complete = sum(!is.na(value)) == n.words) %>%
filter(complete)
n.subs <- 10
model.data <- ws %>%
select(data_id, num_item_id, item, value, age) %>%
filter(data_id %in% complete$data_id) %>%
mutate(data_id = as.numeric(as.factor(data_id))) %>% # hack to renumber
filter(data_id <= n.subs) %>%
rename(produces = value) %>%
arrange(data_id) # critical to get the words nested inside subjects
ages <- model.data %>%
group_by(data_id) %>%
summarise(age = mean(age)) %>%
select(age)
dat <- list(age = ages$age,
produces = model.data$produces,
W = n.words,
S = n.subs,
N = n.words * n.subs)Run the model. Unfortunately this takes around 45 minutes, even for 100 subjects. So do it for 10 (~90s).
samps <- pstan(model_code = mcmurray, model_name = "mcmurray_simple",
data = dat, iter = 2000, warmup=200, thin=10, chains = 4,
pars = c('mu_w','mu_s','sigma_w', 'sigma_s',
'difficulty', 'ability'),
verbose = F, refresh=2000/4) Re-merge coefficients with word data.
coefs <- data.frame(summary(samps)$summary)
coefs$name <- rownames(coefs)
word_ids <- ws %>%
group_by(num_item_id) %>%
summarise(item = unique(item))
words <- coefs %>%
filter(str_detect(name, "difficulty")) %>%
separate(name, c("name", "num_item_id"), "\\[") %>%
mutate(num_item_id = as.numeric(str_replace(num_item_id, "]", ""))) %>%
left_join(word_ids) %>%
select(mean, se_mean, sd, item) %>%
arrange(mean) %>%
mutate(item = factor(item,
levels = item,
labels = item))And plot the first 100 words or so… even within this, you can see that there is some good signal here and we are learning good stuff.
qplot(item, mean, data=words[1:100,]) +
theme(axis.text.x = element_text(angle = 90, hjust = 1))
Attempt #2: no by-subject coefficients
The problem here is computational complexity. Let’s rewrite the model so that we’re not actually learning anything about each subject, just learning a mean and variance that gets multiplied by age.
mcmurray_nosubs <-'
data {
int<lower=1> W; // number of words
int<lower=1> S; // number of subjects
int<lower=1> N; // number of datapoints (W*S)
vector[S] age; // subject ages
int<lower=0,upper=1> produces[N]; // actual data
}
parameters {
real<lower=0> mu_w; // word mean
real<lower=0> mu_s; // subject mean
real<lower=0> sigma_w; // word SD
real<lower=0> difficulty[W]; // word mean
}
model {
mu_w ~ normal(0, 10);
mu_s ~ normal(0, 10);
sigma_w ~ normal(0, 5);
difficulty ~ normal(mu_w, sigma_w);
for (s in 1:S)
for (w in 1:W)
produces[W*(s-1) + w] ~ bernoulli_logit((mu_s * age[s]) - difficulty[w]);
}
'Again, simulations with fake data. to begin These are six kids and five words, with the kids gradually increasing in age. Their abilities should be lower and lower, and words 1, 3, and 5 should be estimated as hard.
dat <- list(age = c(12, 14, 16, 18, 20, 22),
produces = rep(c(0, 1, 0, 1, 0), 6),
W = 5,
S = 6,
N = 30)Run!
samps <- pstan(model_code = mcmurray_nosubs, model_name = "mcmurray_nosubs",
data = dat, iter = 2000, warmup=200, thin=20, chains = 4,
pars = c('mu_w','mu_s','sigma_w', 'difficulty'),
verbose = F, refresh=2000/4) Now examine output.
samps## Inference for Stan model: mcmurray_nosubs.
## 4 chains, each with iter=2000; warmup=200; thin=20;
## post-warmup draws per chain=90, total post-warmup draws=360.
##
## mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
## mu_w 11.26 0.30 4.91 2.58 7.84 10.87 14.45 22.69 272 1.02
## mu_s 0.57 0.01 0.25 0.21 0.39 0.53 0.71 1.21 360 1.01
## sigma_w 8.00 0.14 2.74 3.75 6.12 7.60 9.42 14.65 360 1.00
## difficulty[1] 19.19 0.41 7.86 7.41 13.80 17.80 22.79 40.05 360 1.00
## difficulty[2] 3.93 0.16 2.93 0.22 1.75 3.32 5.56 10.77 343 1.01
## difficulty[3] 18.86 0.44 8.37 7.53 13.00 17.67 22.12 40.04 360 1.01
## difficulty[4] 3.97 0.16 3.03 0.06 1.63 3.49 5.57 11.19 339 1.01
## difficulty[5] 19.18 0.42 7.94 8.44 13.26 17.63 23.79 37.46 360 1.00
## lp__ -2.83 0.14 2.74 -8.75 -4.38 -2.59 -0.91 1.67 360 1.00
##
## Samples were drawn using NUTS(diag_e) at Wed Aug 5 09:44:53 2015.
## For each parameter, n_eff is a crude measure of effective sample size,
## and Rhat is the potential scale reduction factor on split chains (at
## convergence, Rhat=1).Looks fine. On to a subset of the dataset.
n.words <- 680
complete <- ws %>%
group_by(data_id) %>%
summarize(complete = sum(!is.na(value)) == n.words) %>%
filter(complete)
n.total.subs <- length(unique(ws$data_id))
n.subs <- 100
model.data <- ws %>%
select(data_id, num_item_id, item, value, age) %>%
filter(data_id %in% complete$data_id) %>%
mutate(data_id = as.numeric(as.factor(data_id))) %>% # hack to renumber
filter(data_id %in% sample(n.total.subs)[1:n.subs]) %>%
rename(produces = value) %>%
arrange(data_id) # critical to get the words nested inside subjects
ages <- model.data %>%
group_by(data_id) %>%
summarise(age = mean(age)) %>%
select(age)
dat <- list(age = ages$age,
produces = model.data$produces,
W = n.words,
S = n.subs,
N = n.words * n.subs)
samps <- pstan(model_code = mcmurray_nosubs, model_name = "mcmurray_nosubs",
data = dat, iter = 1000, warmup=100, thin=10, chains = 4,
pars = c('mu_w','mu_s','sigma_w','difficulty'),
verbose = F, refresh=2000/4)
samps## Inference for Stan model: mcmurray_nosubs.
## 4 chains, each with iter=1000; warmup=100; thin=10;
## post-warmup draws per chain=90, total post-warmup draws=360.
##
## mean se_mean sd 2.5% 25% 50%
## mu_w 7.41 0.23 0.51 6.30 7.11 7.36
## mu_s 0.30 0.01 0.02 0.25 0.29 0.30
## sigma_w 1.33 0.03 0.09 1.16 1.25 1.33
## difficulty[1] 5.60 0.20 0.49 4.62 5.30 5.56
## difficulty[2] 5.66 0.21 0.51 4.58 5.33 5.68
## difficulty[3] 7.41 0.22 0.55 6.31 7.04 7.38
## difficulty[4] 4.94 0.22 0.68 3.91 4.69 4.96
## difficulty[5] 4.77 0.25 1.11 0.23 4.56 4.94
## difficulty[6] 3.72 0.64 1.99 0.01 4.04 4.60
## difficulty[7] 5.22 0.19 0.49 4.27 4.88 5.22
## difficulty[8] 5.35 0.20 0.50 4.32 5.01 5.35
## difficulty[9] 4.11 0.22 0.52 3.18 3.70 4.10
## difficulty[10] 5.68 0.18 0.48 4.59 5.40 5.71
## difficulty[11] 5.08 0.20 0.49 4.11 4.78 5.09
## difficulty[12] 4.22 1.72 2.47 0.01 2.94 5.38
## difficulty[13] 7.21 0.20 0.52 6.15 6.87 7.21
## difficulty[14] 6.88 0.38 1.37 0.06 6.72 7.06
## difficulty[15] 7.47 0.21 0.54 6.38 7.10 7.52
## difficulty[16] 5.92 0.19 0.48 4.91 5.61 5.98
## difficulty[17] 6.59 0.23 0.53 5.49 6.25 6.53
## difficulty[18] 5.30 0.20 0.48 4.31 4.97 5.26
## difficulty[19] 5.12 0.72 2.05 0.01 5.42 5.83
## difficulty[20] 6.44 0.20 0.52 5.36 6.11 6.46
## difficulty[21] 6.71 0.23 0.52 5.72 6.34 6.69
## difficulty[22] 5.39 0.21 0.49 4.46 5.08 5.36
## difficulty[23] 6.37 0.21 0.50 5.32 6.03 6.33
## difficulty[24] 5.79 0.20 0.51 4.74 5.48 5.77
## difficulty[25] 8.10 0.23 0.55 7.03 7.76 8.11
## difficulty[26] 4.23 0.18 0.53 3.21 3.86 4.24
## difficulty[27] 8.17 0.24 0.55 7.04 7.80 8.11
## difficulty[28] 4.92 0.17 0.48 3.96 4.58 4.96
## difficulty[29] 6.84 0.20 0.53 5.73 6.50 6.87
## difficulty[30] 5.14 0.67 1.86 0.02 5.31 5.71
## difficulty[31] 6.59 0.22 0.51 5.58 6.23 6.56
## difficulty[32] 7.28 0.22 0.54 6.09 6.95 7.31
## difficulty[33] 8.36 0.24 0.57 7.22 7.98 8.36
## difficulty[34] 9.06 0.23 0.59 7.82 8.69 9.03
## difficulty[35] 6.23 0.33 1.07 5.11 6.01 6.36
## difficulty[36] 5.47 0.19 0.49 4.30 5.18 5.46
## difficulty[37] 8.26 0.23 0.58 6.96 7.86 8.32
## difficulty[38] 6.79 0.20 0.54 5.58 6.47 6.81
## difficulty[39] 6.56 0.20 0.49 5.50 6.28 6.56
## difficulty[40] 8.85 0.22 0.57 7.70 8.46 8.87
## difficulty[41] 6.67 0.19 0.49 5.63 6.34 6.67
## difficulty[42] 7.19 0.22 0.54 6.08 6.85 7.16
## difficulty[43] 8.04 0.24 0.57 6.79 7.66 7.97
## difficulty[44] 6.10 0.17 0.49 5.09 5.76 6.15
## difficulty[45] 8.57 0.24 0.57 7.39 8.16 8.54
## difficulty[46] 6.24 0.20 0.50 5.22 5.91 6.23
## difficulty[47] 8.40 0.24 0.58 7.21 7.99 8.35
## difficulty[48] 6.82 0.20 0.51 5.75 6.47 6.81
## difficulty[49] 7.52 0.22 0.55 6.46 7.17 7.55
## difficulty[50] 7.12 0.19 0.51 6.01 6.83 7.12
## difficulty[51] 7.19 0.21 0.52 6.06 6.84 7.24
## difficulty[52] 7.57 0.20 0.52 6.51 7.20 7.54
## difficulty[53] 6.90 0.21 0.53 5.71 6.58 6.92
## difficulty[54] 8.72 0.24 0.59 7.45 8.32 8.76
## difficulty[55] 7.35 0.23 0.54 6.25 6.98 7.36
## difficulty[56] 6.01 0.19 0.49 5.03 5.70 6.03
## difficulty[57] 6.61 0.21 0.51 5.50 6.27 6.60
## difficulty[58] 6.00 0.21 0.68 4.94 5.69 6.08
## difficulty[59] 6.25 0.19 0.50 5.27 5.94 6.24
## difficulty[60] 4.75 0.25 1.04 0.29 4.61 4.95
## difficulty[61] 7.27 0.20 0.52 6.07 6.95 7.29
## difficulty[62] 7.84 0.22 0.52 6.77 7.51 7.86
## difficulty[63] 7.86 0.23 0.56 6.71 7.49 7.81
## difficulty[64] 9.04 0.24 0.61 7.72 8.66 9.06
## difficulty[65] 7.65 0.22 0.53 6.65 7.32 7.58
## difficulty[66] 7.80 0.23 0.55 6.70 7.48 7.75
## difficulty[67] 6.61 0.20 0.52 5.46 6.26 6.63
## difficulty[68] 8.61 0.22 0.59 7.34 8.22 8.62
## difficulty[69] 5.54 0.18 0.48 4.54 5.23 5.56
## difficulty[70] 3.46 0.32 1.07 0.14 3.27 3.67
## difficulty[71] 5.59 0.19 0.49 4.56 5.30 5.61
## difficulty[72] 7.63 0.20 0.53 6.45 7.30 7.64
## difficulty[73] 6.58 0.20 0.49 5.51 6.27 6.55
## difficulty[74] 5.05 0.19 0.59 4.08 4.79 5.06
## difficulty[75] 5.80 0.19 0.49 4.84 5.49 5.80
## difficulty[76] 8.41 0.22 0.59 7.21 7.96 8.42
## difficulty[77] 7.05 0.20 0.52 5.90 6.70 7.07
## difficulty[78] 6.84 0.20 0.53 5.75 6.50 6.82
## difficulty[79] 7.71 0.22 0.56 6.45 7.36 7.77
## difficulty[80] 8.51 0.24 0.57 7.29 8.13 8.51
## difficulty[81] 7.10 0.21 0.53 5.99 6.75 7.12
## difficulty[82] 7.56 0.20 0.54 6.33 7.24 7.60
## difficulty[83] 8.18 0.21 0.54 7.07 7.86 8.20
## difficulty[84] 7.59 0.21 0.54 6.46 7.26 7.62
## difficulty[85] 7.13 0.21 0.53 5.94 6.81 7.15
## difficulty[86] 7.33 0.21 0.53 6.25 6.97 7.28
## difficulty[87] 6.40 0.19 0.50 5.40 6.09 6.38
## difficulty[88] 5.45 0.16 0.93 3.81 5.23 5.50
## difficulty[89] 7.05 0.36 1.19 5.69 6.85 7.18
## difficulty[90] 5.09 0.19 0.50 4.00 4.72 5.12
## difficulty[91] 7.40 0.22 0.55 6.29 7.08 7.40
## difficulty[92] 6.59 0.21 0.52 5.46 6.24 6.58
## difficulty[93] 7.40 0.23 0.53 6.36 7.05 7.36
## difficulty[94] 6.78 0.19 0.49 5.69 6.47 6.81
## difficulty[95] 5.82 1.01 2.45 0.01 6.33 6.66
## difficulty[96] 7.09 0.21 0.53 5.99 6.75 7.11
## difficulty[97] 6.84 0.25 0.54 5.79 6.47 6.77
## difficulty[98] 7.13 0.21 0.52 6.03 6.83 7.11
## difficulty[99] 5.09 0.62 1.75 0.01 5.22 5.58
## difficulty[100] 6.58 0.19 0.49 5.54 6.25 6.60
## difficulty[101] 7.47 0.20 0.55 6.25 7.14 7.50
## difficulty[102] 7.66 0.23 0.56 6.50 7.35 7.62
## difficulty[103] 8.06 0.21 0.54 6.69 7.73 8.12
## difficulty[104] 5.56 0.19 0.48 4.54 5.26 5.54
## difficulty[105] 7.06 0.20 0.51 6.01 6.78 7.05
## difficulty[106] 5.77 0.22 0.52 4.69 5.46 5.73
## difficulty[107] 7.60 0.22 0.55 6.44 7.28 7.57
## difficulty[108] 6.14 0.18 0.48 5.12 5.84 6.15
## difficulty[109] 6.58 0.20 0.53 5.41 6.30 6.55
## difficulty[110] 6.68 0.21 0.52 5.66 6.30 6.75
## difficulty[111] 7.04 0.22 0.53 5.93 6.70 7.05
## difficulty[112] 6.76 0.19 0.50 5.72 6.42 6.80
## difficulty[113] 6.72 0.20 0.51 5.65 6.40 6.73
## difficulty[114] 8.06 0.23 0.55 6.92 7.71 8.06
## difficulty[115] 7.96 0.21 0.56 6.72 7.57 7.97
## difficulty[116] 7.45 0.21 0.53 6.30 7.11 7.43
## difficulty[117] 6.84 0.21 0.53 5.68 6.47 6.88
## difficulty[118] 6.53 0.21 0.51 5.49 6.21 6.51
## difficulty[119] 8.05 0.22 0.55 6.91 7.70 8.04
## difficulty[120] 8.31 0.21 0.54 7.12 8.00 8.29
## difficulty[121] 4.36 1.71 2.46 0.04 3.27 5.49
## difficulty[122] 8.36 0.22 0.55 7.20 7.98 8.35
## difficulty[123] 7.51 0.23 0.54 6.40 7.16 7.47
## difficulty[124] 8.44 0.23 0.59 7.24 8.05 8.46
## difficulty[125] 3.90 1.60 2.29 0.00 2.55 4.97
## difficulty[126] 7.66 0.23 0.55 6.50 7.28 7.60
## difficulty[127] 7.49 0.23 0.55 6.33 7.11 7.44
## difficulty[128] 8.23 0.22 0.56 7.11 7.85 8.24
## difficulty[129] 6.63 0.19 0.51 5.53 6.25 6.66
## difficulty[130] 7.23 0.23 0.52 6.17 6.94 7.16
## difficulty[131] 7.54 0.22 0.53 6.44 7.25 7.53
## difficulty[132] 7.42 0.21 0.54 6.29 7.07 7.45
## difficulty[133] 7.72 0.26 0.57 6.62 7.33 7.67
## difficulty[134] 6.23 0.21 0.50 5.21 5.92 6.19
## difficulty[135] 7.08 0.20 0.51 5.99 6.75 7.06
## difficulty[136] 7.55 0.21 0.54 6.46 7.18 7.57
## difficulty[137] 7.27 0.20 0.53 6.06 6.91 7.37
## difficulty[138] 7.06 0.22 0.54 5.93 6.75 7.08
## difficulty[139] 7.56 0.21 0.54 6.43 7.22 7.58
## difficulty[140] 8.61 0.24 0.59 7.33 8.20 8.64
## difficulty[141] 7.32 0.24 0.56 6.22 6.95 7.28
## difficulty[142] 7.67 0.21 0.54 6.57 7.31 7.66
## difficulty[143] 8.66 0.23 0.57 7.50 8.28 8.68
## difficulty[144] 7.34 0.21 0.51 6.38 7.01 7.33
## difficulty[145] 8.24 0.21 0.55 7.08 7.88 8.28
## difficulty[146] 7.90 0.24 0.58 6.71 7.54 7.86
## difficulty[147] 7.52 0.22 0.55 6.50 7.14 7.51
## difficulty[148] 6.98 0.20 0.51 5.85 6.66 6.96
## difficulty[149] 7.20 0.22 0.53 6.05 6.88 7.18
## difficulty[150] 7.00 0.21 0.52 5.88 6.67 7.00
## difficulty[151] 9.13 0.23 0.60 7.81 8.77 9.14
## difficulty[152] 9.17 0.22 0.61 7.88 8.83 9.21
## difficulty[153] 8.28 0.24 0.57 7.06 7.91 8.28
## difficulty[154] 6.00 0.20 0.50 5.02 5.70 5.99
## difficulty[155] 7.20 0.22 0.54 6.12 6.85 7.17
## difficulty[156] 8.44 0.24 0.58 7.35 8.02 8.43
## difficulty[157] 7.75 0.22 0.55 6.59 7.40 7.71
## difficulty[158] 7.46 0.21 0.53 6.45 7.06 7.54
## difficulty[159] 7.12 0.21 0.51 6.05 6.79 7.18
## difficulty[160] 6.63 0.21 0.77 5.55 6.35 6.65
## difficulty[161] 6.60 0.20 0.49 5.55 6.30 6.58
## difficulty[162] 5.93 0.20 0.51 4.83 5.61 5.95
## difficulty[163] 7.85 0.20 0.52 6.72 7.53 7.87
## difficulty[164] 7.97 0.23 0.53 6.80 7.63 7.93
## difficulty[165] 5.10 0.21 0.50 4.12 4.72 5.09
## difficulty[166] 7.04 0.22 0.52 5.97 6.67 7.01
## difficulty[167] 7.94 0.22 0.54 6.79 7.56 7.99
## difficulty[168] 8.55 0.23 0.56 7.33 8.20 8.54
## difficulty[169] 7.86 0.22 0.55 6.65 7.55 7.88
## difficulty[170] 7.02 0.19 0.51 5.88 6.71 7.05
## difficulty[171] 6.79 0.22 0.52 5.76 6.43 6.73
## difficulty[172] 9.24 0.21 0.56 8.10 8.87 9.22
## difficulty[173] 6.61 0.20 0.48 5.58 6.29 6.58
## difficulty[174] 3.65 1.34 2.04 0.01 3.56 4.58
## difficulty[175] 7.50 0.22 0.55 6.43 7.13 7.48
## difficulty[176] 8.15 0.24 0.58 6.93 7.76 8.11
## difficulty[177] 8.63 0.22 0.58 7.45 8.24 8.64
## difficulty[178] 9.38 0.23 0.59 8.13 9.01 9.37
## difficulty[179] 6.20 0.19 0.51 5.04 5.90 6.19
## difficulty[180] 7.36 0.21 0.56 6.28 6.97 7.38
## difficulty[181] 9.24 0.22 0.58 8.01 8.87 9.21
## difficulty[182] 8.22 0.23 0.55 7.13 7.86 8.20
## difficulty[183] 7.45 0.23 0.54 6.37 7.10 7.43
## difficulty[184] 8.97 0.23 0.60 7.68 8.59 8.94
## difficulty[185] 6.56 0.19 0.50 5.52 6.27 6.56
## difficulty[186] 6.47 0.20 0.51 5.45 6.17 6.44
## difficulty[187] 6.52 0.21 0.52 5.50 6.17 6.48
## difficulty[188] 7.24 0.21 0.52 6.06 6.91 7.25
## difficulty[189] 7.06 0.21 0.52 5.99 6.71 7.08
## difficulty[190] 5.40 0.17 0.48 4.29 5.09 5.50
## difficulty[191] 4.48 0.66 1.73 0.01 4.63 5.12
## difficulty[192] 7.41 0.20 0.52 6.34 7.06 7.39
## difficulty[193] 6.62 0.20 0.49 5.57 6.31 6.65
## difficulty[194] 6.24 0.21 0.50 5.17 5.93 6.21
## difficulty[195] 5.95 0.19 0.50 4.89 5.62 5.97
## difficulty[196] 6.36 0.22 0.52 5.32 6.03 6.34
## difficulty[197] 6.33 0.25 0.82 5.21 6.09 6.36
## difficulty[198] 6.95 0.22 0.53 5.91 6.62 6.94
## difficulty[199] 6.93 0.21 0.52 5.79 6.63 6.90
## difficulty[200] 7.71 0.22 0.54 6.52 7.41 7.73
## difficulty[201] 6.00 0.18 0.50 4.88 5.69 6.05
## difficulty[202] 5.35 0.18 0.50 4.31 5.00 5.37
## difficulty[203] 4.79 1.01 2.13 0.02 5.20 5.60
## difficulty[204] 7.49 0.21 0.52 6.33 7.13 7.51
## difficulty[205] 7.97 0.22 0.55 6.88 7.62 7.97
## difficulty[206] 6.25 0.20 0.51 5.23 5.90 6.27
## difficulty[207] 6.81 0.20 0.51 5.66 6.49 6.81
## difficulty[208] 6.65 0.19 0.51 5.63 6.34 6.68
## difficulty[209] 6.57 0.21 0.52 5.48 6.22 6.55
## difficulty[210] 8.93 0.21 0.56 7.83 8.56 8.90
## difficulty[211] 7.43 0.21 0.55 6.24 7.05 7.47
## difficulty[212] 6.12 0.19 0.51 5.02 5.81 6.14
## difficulty[213] 6.15 0.21 0.62 5.12 5.86 6.18
## difficulty[214] 6.59 0.22 0.53 5.56 6.26 6.57
## difficulty[215] 6.80 0.21 0.53 5.70 6.47 6.84
## difficulty[216] 7.13 0.23 0.53 6.08 6.76 7.14
## difficulty[217] 6.33 0.34 1.31 0.23 6.21 6.53
## difficulty[218] 7.72 0.23 0.55 6.62 7.40 7.68
## difficulty[219] 7.69 0.21 0.52 6.60 7.35 7.68
## difficulty[220] 7.89 0.21 0.55 6.72 7.54 7.92
## difficulty[221] 7.01 0.20 0.52 5.89 6.68 7.06
## difficulty[222] 7.43 0.21 0.53 6.28 7.09 7.43
## difficulty[223] 4.87 1.06 2.11 0.03 5.13 5.65
## difficulty[224] 7.97 0.23 0.57 6.80 7.59 8.00
## difficulty[225] 6.30 0.20 0.49 5.32 5.98 6.28
## difficulty[226] 7.55 0.21 0.55 6.40 7.23 7.59
## difficulty[227] 7.77 0.24 0.57 6.50 7.41 7.79
## difficulty[228] 5.13 2.10 3.00 0.00 3.92 6.68
## difficulty[229] 7.66 0.22 0.52 6.57 7.30 7.65
## difficulty[230] 7.94 0.23 0.55 6.84 7.57 7.96
## difficulty[231] 6.37 0.20 0.52 5.29 6.06 6.41
## difficulty[232] 7.51 0.21 0.53 6.30 7.17 7.50
## difficulty[233] 8.35 0.22 0.54 7.18 8.01 8.33
## difficulty[234] 5.93 0.20 0.50 4.86 5.65 5.90
## difficulty[235] 7.22 0.22 0.54 6.11 6.87 7.21
## difficulty[236] 7.11 0.22 0.53 6.01 6.76 7.10
## difficulty[237] 8.38 0.24 0.57 7.14 7.99 8.40
## difficulty[238] 8.38 0.24 0.58 7.23 7.97 8.38
## difficulty[239] 7.45 0.21 0.52 6.36 7.13 7.41
## difficulty[240] 7.05 0.20 0.52 5.92 6.72 7.08
## difficulty[241] 7.70 0.21 0.53 6.72 7.34 7.66
## difficulty[242] 7.39 0.22 0.54 6.32 7.02 7.36
## difficulty[243] 6.90 0.21 0.52 5.88 6.57 6.89
## difficulty[244] 8.02 0.23 0.54 6.86 7.67 7.99
## difficulty[245] 7.13 0.20 0.53 5.96 6.83 7.11
## difficulty[246] 7.26 0.21 0.53 6.12 6.93 7.24
## difficulty[247] 8.36 0.23 0.57 7.12 8.05 8.36
## difficulty[248] 7.64 0.24 0.56 6.50 7.29 7.62
## difficulty[249] 6.56 0.21 0.52 5.47 6.23 6.57
## difficulty[250] 4.55 1.86 2.66 0.00 3.32 5.89
## difficulty[251] 7.85 0.22 0.54 6.72 7.49 7.85
## difficulty[252] 6.23 0.28 1.11 1.45 6.01 6.36
## difficulty[253] 7.28 0.21 0.53 6.22 6.92 7.24
## difficulty[254] 6.17 0.27 1.06 4.75 5.94 6.24
## difficulty[255] 6.64 0.20 0.51 5.62 6.28 6.65
## difficulty[256] 6.59 0.30 1.19 0.84 6.44 6.74
## difficulty[257] 9.51 0.22 0.59 8.32 9.13 9.49
## difficulty[258] 7.22 0.21 0.53 6.19 6.86 7.19
## difficulty[259] 9.52 0.23 0.59 8.29 9.12 9.53
## difficulty[260] 7.25 0.21 0.51 6.12 6.92 7.27
## difficulty[261] 9.05 0.24 0.58 7.79 8.68 9.02
## difficulty[262] 7.00 0.21 0.53 5.92 6.68 6.99
## difficulty[263] 6.36 0.19 0.50 5.26 6.10 6.35
## difficulty[264] 6.07 0.20 0.49 5.02 5.77 6.05
## difficulty[265] 7.46 0.22 0.54 6.27 7.11 7.47
## difficulty[266] 9.09 0.23 0.59 7.77 8.71 9.10
## difficulty[267] 6.31 0.19 0.49 5.23 6.01 6.35
## difficulty[268] 7.62 0.20 0.52 6.51 7.27 7.63
## difficulty[269] 7.27 0.22 0.51 6.26 6.94 7.25
## difficulty[270] 7.71 0.24 0.56 6.60 7.34 7.68
## difficulty[271] 5.93 0.19 0.51 4.91 5.60 5.99
## difficulty[272] 7.81 0.23 0.55 6.63 7.48 7.79
## difficulty[273] 8.43 0.24 0.59 7.19 8.02 8.44
## difficulty[274] 8.18 0.22 0.56 7.05 7.83 8.19
## difficulty[275] 7.50 0.21 0.53 6.37 7.19 7.49
## difficulty[276] 7.23 0.21 0.52 6.14 6.91 7.22
## difficulty[277] 7.96 0.23 0.56 6.74 7.62 7.94
## difficulty[278] 8.04 0.23 0.55 6.92 7.70 8.04
## difficulty[279] 8.92 0.22 0.57 7.74 8.58 8.94
## difficulty[280] 8.69 0.23 0.58 7.42 8.32 8.71
## difficulty[281] 6.56 0.19 0.51 5.57 6.20 6.58
## difficulty[282] 7.65 0.22 0.56 6.45 7.30 7.66
## difficulty[283] 7.73 0.22 0.56 6.50 7.36 7.72
## difficulty[284] 7.13 0.20 0.53 5.98 6.81 7.16
## difficulty[285] 7.16 0.19 0.53 5.88 6.83 7.21
## difficulty[286] 7.52 0.22 0.56 6.40 7.12 7.55
## difficulty[287] 9.16 0.23 0.57 8.04 8.79 9.12
## difficulty[288] 7.60 0.21 0.56 6.37 7.24 7.65
## difficulty[289] 7.76 0.21 0.52 6.72 7.44 7.79
## difficulty[290] 7.16 0.20 0.52 6.04 6.83 7.16
## difficulty[291] 6.81 0.21 0.53 5.74 6.42 6.76
## difficulty[292] 7.53 0.22 0.54 6.42 7.16 7.52
## difficulty[293] 7.85 0.22 0.54 6.70 7.51 7.84
## difficulty[294] 8.06 0.23 0.56 6.82 7.72 8.04
## difficulty[295] 7.37 0.22 0.52 6.34 7.00 7.34
## difficulty[296] 7.89 0.20 0.55 6.77 7.54 7.93
## difficulty[297] 5.98 0.20 0.52 4.83 5.71 5.97
## difficulty[298] 8.93 0.23 0.59 7.66 8.58 8.94
## difficulty[299] 7.27 0.22 0.53 6.12 6.92 7.31
## difficulty[300] 8.25 0.21 0.55 7.07 7.87 8.28
## difficulty[301] 7.87 0.22 0.53 6.65 7.53 7.86
## difficulty[302] 8.15 0.24 0.55 7.03 7.81 8.13
## difficulty[303] 6.13 0.18 0.51 5.05 5.82 6.11
## difficulty[304] 6.95 0.21 0.52 5.77 6.65 6.96
## difficulty[305] 6.56 0.20 0.51 5.47 6.21 6.59
## difficulty[306] 6.76 0.21 0.52 5.63 6.44 6.75
## difficulty[307] 8.51 0.21 0.56 7.38 8.13 8.55
## difficulty[308] 8.19 0.22 0.54 7.08 7.80 8.25
## difficulty[309] 7.66 0.21 0.53 6.55 7.32 7.68
## difficulty[310] 8.08 0.22 0.53 6.97 7.76 8.05
## difficulty[311] 7.06 0.20 0.52 5.93 6.72 7.06
## difficulty[312] 6.72 0.21 0.52 5.57 6.39 6.71
## difficulty[313] 7.52 0.21 0.54 6.39 7.18 7.50
## difficulty[314] 8.18 0.23 0.58 6.89 7.81 8.25
## difficulty[315] 8.82 0.23 0.58 7.68 8.44 8.75
## difficulty[316] 6.56 0.22 0.52 5.59 6.22 6.52
## difficulty[317] 7.20 0.20 0.53 6.04 6.85 7.28
## difficulty[318] 9.03 0.22 0.57 7.76 8.72 9.03
## difficulty[319] 7.62 0.22 0.54 6.47 7.28 7.62
## difficulty[320] 6.90 0.21 0.51 5.84 6.57 6.93
## difficulty[321] 6.78 0.20 0.51 5.68 6.44 6.85
## difficulty[322] 5.96 0.18 0.50 4.88 5.66 6.00
## difficulty[323] 6.13 0.20 0.50 5.18 5.81 6.09
## difficulty[324] 7.76 0.23 0.55 6.57 7.41 7.73
## difficulty[325] 8.14 0.24 0.56 7.03 7.75 8.12
## difficulty[326] 9.43 0.22 0.58 8.31 9.05 9.42
## difficulty[327] 8.00 0.23 0.56 6.84 7.62 8.00
## difficulty[328] 9.04 0.25 0.60 7.88 8.62 9.05
## difficulty[329] 10.14 0.21 0.60 8.87 9.79 10.16
## difficulty[330] 9.27 0.22 0.58 7.99 8.91 9.30
## difficulty[331] 8.27 0.24 0.56 7.20 7.87 8.23
## difficulty[332] 8.44 0.21 0.55 7.20 8.15 8.45
## difficulty[333] 6.13 0.19 0.50 5.01 5.82 6.12
## difficulty[334] 6.79 0.20 0.52 5.71 6.46 6.74
## difficulty[335] 7.86 0.21 0.53 6.75 7.54 7.87
## difficulty[336] 5.87 0.18 0.50 4.75 5.57 5.91
## difficulty[337] 6.79 0.19 0.52 5.72 6.47 6.81
## difficulty[338] 7.55 0.22 0.53 6.50 7.18 7.58
## difficulty[339] 9.21 0.22 0.59 7.90 8.81 9.26
## difficulty[340] 7.71 0.22 0.54 6.50 7.38 7.68
## difficulty[341] 6.87 0.21 0.53 5.65 6.56 6.88
## difficulty[342] 7.03 0.22 0.52 5.98 6.72 6.97
## difficulty[343] 9.32 0.23 0.59 8.06 8.94 9.35
## difficulty[344] 6.82 0.21 0.51 5.84 6.47 6.77
## difficulty[345] 8.37 0.25 0.59 7.24 7.99 8.30
## difficulty[346] 7.23 0.21 0.52 6.08 6.94 7.19
## difficulty[347] 7.53 0.22 0.54 6.29 7.21 7.57
## difficulty[348] 4.66 0.18 0.60 3.53 4.35 4.65
## difficulty[349] 9.21 0.25 0.60 7.99 8.85 9.16
## difficulty[350] 7.21 0.23 0.54 6.06 6.86 7.18
## difficulty[351] 7.14 0.20 0.52 5.94 6.80 7.16
## difficulty[352] 7.44 0.20 0.55 6.23 7.08 7.46
## difficulty[353] 9.24 0.23 0.57 8.09 8.89 9.24
## difficulty[354] 7.96 0.22 0.55 6.84 7.59 7.96
## difficulty[355] 8.78 0.22 0.57 7.54 8.43 8.80
## difficulty[356] 2.85 0.23 0.98 0.20 2.55 3.01
## difficulty[357] 7.44 0.22 0.53 6.45 7.10 7.41
## difficulty[358] 8.21 0.22 0.55 7.15 7.89 8.20
## difficulty[359] 7.85 0.21 0.54 6.77 7.52 7.86
## difficulty[360] 7.11 0.22 0.54 6.03 6.78 7.10
## difficulty[361] 5.23 0.21 0.52 4.21 4.91 5.17
## difficulty[362] 5.61 0.19 0.50 4.59 5.26 5.61
## difficulty[363] 8.10 0.23 0.56 6.97 7.74 8.07
## difficulty[364] 8.74 0.24 0.58 7.60 8.35 8.69
## difficulty[365] 7.45 0.22 0.54 6.34 7.11 7.45
## difficulty[366] 1.22 0.70 1.49 0.03 0.10 0.26
## difficulty[367] 9.54 0.23 0.61 8.14 9.19 9.58
## difficulty[368] 6.33 0.21 0.51 5.31 5.96 6.34
## difficulty[369] 7.91 0.24 0.54 6.86 7.56 7.87
## difficulty[370] 9.25 0.22 0.59 7.98 8.89 9.27
## difficulty[371] 5.02 2.05 2.93 0.00 3.87 6.44
## difficulty[372] 8.44 0.23 0.57 7.28 8.11 8.45
## difficulty[373] 7.96 0.21 0.54 6.82 7.61 7.98
## difficulty[374] 8.29 0.22 0.54 7.18 7.97 8.29
## difficulty[375] 7.63 0.20 0.51 6.52 7.28 7.65
## difficulty[376] 5.18 0.18 0.48 4.18 4.88 5.22
## difficulty[377] 7.23 0.21 0.51 6.11 6.94 7.25
## difficulty[378] 2.99 1.18 1.88 0.03 0.15 3.85
## difficulty[379] 6.74 0.21 0.52 5.69 6.43 6.68
## difficulty[380] 7.13 0.21 0.53 6.02 6.78 7.14
## difficulty[381] 7.34 0.22 0.55 6.12 7.02 7.38
## difficulty[382] 7.36 0.22 0.52 6.30 7.04 7.35
## difficulty[383] 6.67 0.19 0.48 5.74 6.35 6.71
## difficulty[384] 5.22 0.14 0.80 3.92 4.99 5.25
## difficulty[385] 4.43 0.46 1.40 0.04 4.43 4.79
## difficulty[386] 7.07 0.22 0.54 5.93 6.72 7.07
## difficulty[387] 6.46 0.21 0.52 5.32 6.15 6.48
## difficulty[388] 4.88 0.50 1.45 0.05 4.86 5.20
## difficulty[389] 4.00 0.17 0.64 2.68 3.67 4.12
## difficulty[390] 7.29 0.20 0.53 6.17 6.95 7.27
## difficulty[391] 5.31 0.43 1.27 0.14 5.24 5.55
## difficulty[392] 4.24 1.73 2.48 0.01 3.13 5.41
## difficulty[393] 5.37 0.20 0.48 4.41 5.06 5.31
## difficulty[394] 7.58 0.23 0.56 6.30 7.24 7.58
## difficulty[395] 7.21 0.22 0.53 6.05 6.85 7.22
## difficulty[396] 7.92 0.23 0.56 6.72 7.56 7.92
## difficulty[397] 3.73 1.58 2.68 0.00 0.03 5.39
## difficulty[398] 7.43 0.21 0.52 6.32 7.12 7.45
## difficulty[399] 7.69 0.23 0.53 6.57 7.33 7.68
## difficulty[400] 5.24 0.18 0.53 4.21 4.94 5.28
## difficulty[401] 6.46 0.22 0.51 5.38 6.11 6.41
## difficulty[402] 7.04 0.20 0.51 5.93 6.70 7.08
## difficulty[403] 7.36 0.21 0.53 6.23 6.99 7.42
## difficulty[404] 7.81 0.22 0.53 6.69 7.51 7.80
## difficulty[405] 8.09 0.22 0.56 6.92 7.73 8.10
## difficulty[406] 7.76 0.21 0.54 6.62 7.42 7.75
## difficulty[407] 8.12 0.21 0.54 7.00 7.78 8.12
## difficulty[408] 7.60 0.21 0.52 6.44 7.25 7.65
## difficulty[409] 7.58 0.24 0.56 6.44 7.21 7.56
## difficulty[410] 8.04 0.23 0.53 7.04 7.71 7.99
## difficulty[411] 6.74 0.19 0.52 5.61 6.40 6.81
## difficulty[412] 5.41 2.21 3.16 0.00 4.01 7.03
## difficulty[413] 7.54 0.22 0.53 6.45 7.18 7.52
## difficulty[414] 7.23 0.20 0.53 6.09 6.91 7.25
## difficulty[415] 7.17 0.20 0.52 5.98 6.85 7.16
## difficulty[416] 8.12 0.23 0.57 6.96 7.76 8.14
## difficulty[417] 6.91 0.22 0.54 5.74 6.55 6.90
## difficulty[418] 7.87 0.22 0.55 6.79 7.49 7.89
## difficulty[419] 6.81 0.20 0.52 5.67 6.49 6.86
## difficulty[420] 7.51 0.22 0.54 6.36 7.18 7.49
## difficulty[421] 5.64 0.54 1.59 0.01 5.69 5.95
## difficulty[422] 7.29 0.22 0.55 6.12 6.92 7.29
## difficulty[423] 7.63 0.22 0.56 6.48 7.26 7.63
## difficulty[424] 7.73 0.23 0.56 6.62 7.39 7.65
## difficulty[425] 8.70 0.24 0.58 7.48 8.36 8.69
## difficulty[426] 5.55 0.35 1.26 0.09 5.47 5.78
## difficulty[427] 7.15 0.22 0.54 5.98 6.79 7.16
## difficulty[428] 7.87 0.21 0.55 6.67 7.56 7.87
## difficulty[429] 7.53 0.21 0.56 6.32 7.17 7.55
## difficulty[430] 8.28 0.24 0.59 7.00 7.89 8.30
## difficulty[431] 8.27 0.23 0.58 7.07 7.89 8.27
## difficulty[432] 7.33 0.21 0.54 6.12 7.03 7.34
## difficulty[433] 6.97 0.20 0.51 5.94 6.66 6.97
## difficulty[434] 7.58 0.23 0.55 6.45 7.21 7.55
## difficulty[435] 5.57 0.18 0.50 4.50 5.21 5.61
## difficulty[436] 9.42 0.24 0.60 8.20 9.03 9.40
## difficulty[437] 7.57 0.22 0.55 6.38 7.20 7.54
## difficulty[438] 7.65 0.20 0.54 6.40 7.30 7.67
## difficulty[439] 6.81 0.21 0.52 5.74 6.49 6.81
## difficulty[440] 7.39 0.21 0.54 6.20 7.02 7.44
## difficulty[441] 7.37 0.21 0.53 6.33 7.00 7.38
## difficulty[442] 7.45 0.22 0.54 6.31 7.12 7.42
## difficulty[443] 6.24 0.43 1.35 0.16 6.12 6.43
## difficulty[444] 7.84 0.22 0.53 6.72 7.50 7.81
## difficulty[445] 7.04 0.22 0.55 5.91 6.67 6.96
## difficulty[446] 7.01 0.21 0.54 5.85 6.68 7.03
## difficulty[447] 6.25 0.27 1.03 5.19 6.04 6.35
## difficulty[448] 7.52 0.23 0.54 6.38 7.18 7.55
## difficulty[449] 7.81 0.23 0.55 6.71 7.45 7.75
## difficulty[450] 7.57 0.22 0.56 6.42 7.21 7.59
## difficulty[451] 8.05 0.22 0.54 6.93 7.71 8.07
## difficulty[452] 6.89 0.23 0.88 5.77 6.61 6.96
## difficulty[453] 6.48 0.41 1.48 0.09 6.41 6.79
## difficulty[454] 7.60 0.24 0.56 6.39 7.27 7.59
## difficulty[455] 6.20 0.32 1.18 0.24 6.05 6.36
## difficulty[456] 8.10 0.23 0.56 6.91 7.73 8.05
## difficulty[457] 8.19 0.22 0.56 6.97 7.86 8.19
## difficulty[458] 6.43 0.20 0.50 5.39 6.11 6.41
## difficulty[459] 8.22 0.24 0.56 6.92 7.91 8.19
## difficulty[460] 9.35 0.24 0.60 8.07 8.96 9.32
## difficulty[461] 7.94 0.22 0.54 6.77 7.62 7.94
## difficulty[462] 7.24 0.21 0.53 6.14 6.92 7.19
## difficulty[463] 7.72 0.22 0.55 6.60 7.37 7.71
## difficulty[464] 6.83 0.24 0.51 5.80 6.49 6.78
## difficulty[465] 6.88 0.23 0.55 5.72 6.51 6.89
## difficulty[466] 9.01 0.23 0.58 7.83 8.66 8.99
## difficulty[467] 7.34 0.22 0.54 6.19 7.03 7.29
## difficulty[468] 7.84 0.21 0.54 6.74 7.47 7.88
## difficulty[469] 6.58 0.24 0.53 5.56 6.22 6.58
## difficulty[470] 7.68 0.23 0.54 6.55 7.34 7.64
## difficulty[471] 7.90 0.21 0.54 6.64 7.59 7.92
## difficulty[472] 8.10 0.23 0.57 6.93 7.75 8.03
## difficulty[473] 6.94 0.20 0.51 5.77 6.62 6.97
## difficulty[474] 6.39 0.21 0.52 5.39 6.05 6.35
## difficulty[475] 9.27 0.23 0.58 7.97 8.93 9.30
## difficulty[476] 6.51 0.20 0.50 5.51 6.15 6.49
## difficulty[477] 7.10 0.21 0.53 5.98 6.74 7.11
## difficulty[478] 7.76 0.22 0.53 6.60 7.41 7.71
## difficulty[479] 7.66 0.22 0.54 6.60 7.32 7.63
## difficulty[480] 7.51 0.21 0.54 6.30 7.16 7.58
## difficulty[481] 7.57 0.22 0.55 6.38 7.21 7.55
## difficulty[482] 7.55 0.23 0.54 6.45 7.20 7.49
## difficulty[483] 6.39 0.19 0.49 5.33 6.10 6.36
## difficulty[484] 7.46 0.20 0.54 6.35 7.16 7.47
## difficulty[485] 7.28 0.21 0.52 6.12 6.97 7.25
## difficulty[486] 6.88 0.21 0.50 5.83 6.54 6.88
## difficulty[487] 7.76 0.21 0.56 6.61 7.38 7.84
## difficulty[488] 7.45 0.23 0.53 6.38 7.11 7.42
## difficulty[489] 8.13 0.23 0.54 7.02 7.78 8.08
## difficulty[490] 8.73 0.24 0.58 7.54 8.35 8.72
## difficulty[491] 9.06 0.24 0.59 7.83 8.71 9.05
## difficulty[492] 7.12 0.21 0.50 6.12 6.79 7.09
## difficulty[493] 6.85 0.20 0.51 5.76 6.54 6.88
## difficulty[494] 7.39 0.21 0.54 6.19 7.05 7.40
## difficulty[495] 7.80 0.23 0.57 6.60 7.46 7.76
## difficulty[496] 7.85 0.22 0.54 6.69 7.55 7.83
## difficulty[497] 6.73 0.20 0.53 5.65 6.38 6.78
## difficulty[498] 7.09 0.23 0.53 6.03 6.74 7.12
## difficulty[499] 7.38 0.24 0.56 6.23 6.97 7.34
## difficulty[500] 7.61 0.20 0.54 6.42 7.25 7.65
## difficulty[501] 10.33 0.22 0.63 9.11 9.88 10.36
## difficulty[502] 7.02 0.22 0.53 6.04 6.67 6.99
## difficulty[503] 7.93 0.23 0.55 6.80 7.58 7.95
## difficulty[504] 5.70 0.17 0.49 4.70 5.36 5.70
## difficulty[505] 6.85 0.20 0.52 5.82 6.50 6.86
## difficulty[506] 7.61 0.22 0.56 6.46 7.26 7.58
## difficulty[507] 6.97 0.20 0.53 5.88 6.60 6.96
## difficulty[508] 8.04 0.22 0.54 6.85 7.69 8.04
## difficulty[509] 6.54 0.22 0.51 5.55 6.23 6.48
## difficulty[510] 7.79 0.21 0.53 6.71 7.46 7.76
## difficulty[511] 6.69 0.22 0.50 5.73 6.36 6.62
## difficulty[512] 7.07 0.22 0.55 5.87 6.70 7.06
## difficulty[513] 8.21 0.23 0.56 7.06 7.87 8.18
## difficulty[514] 7.58 0.21 0.54 6.44 7.25 7.58
## difficulty[515] 7.09 0.22 0.52 6.01 6.78 7.02
## difficulty[516] 6.01 0.21 0.51 4.97 5.68 5.99
## difficulty[517] 7.80 0.21 0.53 6.64 7.48 7.80
## difficulty[518] 7.91 0.23 0.56 6.78 7.53 7.91
## difficulty[519] 6.62 0.21 0.51 5.61 6.28 6.64
## difficulty[520] 7.44 0.25 0.56 6.39 7.04 7.39
## difficulty[521] 7.76 0.22 0.56 6.65 7.41 7.74
## difficulty[522] 7.61 0.23 0.55 6.50 7.22 7.61
## difficulty[523] 9.27 0.22 0.60 8.01 8.89 9.31
## difficulty[524] 8.30 0.21 0.55 7.12 7.94 8.34
## difficulty[525] 8.27 0.21 0.55 7.08 7.91 8.31
## difficulty[526] 7.90 0.22 0.55 6.72 7.55 7.86
## difficulty[527] 6.94 0.22 0.52 5.86 6.60 6.91
## difficulty[528] 7.60 0.22 0.53 6.50 7.26 7.61
## difficulty[529] 7.27 0.22 0.53 6.12 6.91 7.24
## difficulty[530] 8.15 0.24 0.56 6.96 7.78 8.13
## difficulty[531] 7.53 0.21 0.54 6.33 7.18 7.57
## difficulty[532] 7.56 0.23 0.53 6.40 7.21 7.53
## difficulty[533] 4.89 0.17 0.48 3.82 4.64 4.89
## difficulty[534] 7.21 0.22 0.54 6.11 6.88 7.22
## difficulty[535] 6.99 0.21 0.52 5.83 6.68 6.97
## difficulty[536] 9.02 0.21 0.57 7.74 8.62 9.09
## difficulty[537] 7.68 0.21 0.54 6.61 7.36 7.67
## difficulty[538] 8.73 0.22 0.55 7.55 8.40 8.72
## difficulty[539] 7.67 0.21 0.52 6.61 7.33 7.64
## difficulty[540] 8.36 0.22 0.55 7.20 7.99 8.38
## difficulty[541] 9.08 0.23 0.60 7.83 8.71 8.99
## difficulty[542] 8.16 0.21 0.55 6.99 7.83 8.15
## difficulty[543] 7.18 0.21 0.52 6.08 6.84 7.20
## difficulty[544] 8.70 0.22 0.58 7.35 8.33 8.71
## difficulty[545] 8.64 0.24 0.56 7.40 8.29 8.62
## difficulty[546] 7.32 0.23 0.54 6.19 6.96 7.28
## difficulty[547] 9.36 0.21 0.58 8.07 9.01 9.40
## difficulty[548] 6.64 0.21 0.51 5.60 6.32 6.64
## difficulty[549] 7.84 0.20 0.54 6.67 7.53 7.85
## difficulty[550] 7.41 0.21 0.53 6.26 7.06 7.42
## difficulty[551] 7.62 0.22 0.55 6.52 7.27 7.64
## difficulty[552] 7.76 0.22 0.54 6.70 7.39 7.76
## difficulty[553] 7.81 0.25 0.57 6.66 7.44 7.75
## difficulty[554] 7.59 0.23 0.58 6.31 7.26 7.62
## difficulty[555] 8.58 0.23 0.59 7.39 8.19 8.60
## difficulty[556] 7.37 0.20 0.52 6.27 7.06 7.40
## difficulty[557] 7.59 0.22 0.55 6.40 7.27 7.61
## difficulty[558] 7.01 0.20 0.53 5.85 6.64 6.98
## difficulty[559] 7.75 0.21 0.52 6.71 7.42 7.77
## difficulty[560] 8.98 0.25 0.60 7.84 8.56 8.95
## difficulty[561] 7.72 0.21 0.53 6.60 7.40 7.73
## difficulty[562] 6.53 0.22 0.53 5.45 6.20 6.53
## difficulty[563] 8.04 0.23 0.55 6.82 7.68 8.00
## difficulty[564] 8.22 0.21 0.55 6.99 7.90 8.22
## difficulty[565] 7.16 0.21 0.50 6.12 6.87 7.10
## difficulty[566] 6.54 0.20 0.51 5.50 6.19 6.54
## difficulty[567] 8.68 0.22 0.56 7.30 8.31 8.69
## difficulty[568] 9.45 0.22 0.59 8.16 9.08 9.45
## difficulty[569] 8.09 0.22 0.56 6.98 7.75 8.09
## difficulty[570] 8.11 0.21 0.53 6.93 7.78 8.12
## difficulty[571] 7.97 0.21 0.56 6.81 7.62 7.97
## difficulty[572] 7.06 0.21 0.53 5.95 6.73 7.07
## difficulty[573] 7.53 0.21 0.52 6.38 7.19 7.55
## difficulty[574] 9.14 0.24 0.59 7.89 8.70 9.15
## difficulty[575] 8.79 0.23 0.58 7.58 8.43 8.79
## difficulty[576] 8.45 0.22 0.57 7.18 8.08 8.46
## difficulty[577] 8.88 0.26 0.61 7.70 8.49 8.86
## difficulty[578] 9.43 0.23 0.63 8.25 9.03 9.39
## difficulty[579] 8.34 0.23 0.56 7.12 8.00 8.32
## difficulty[580] 8.83 0.22 0.57 7.74 8.43 8.79
## difficulty[581] 9.54 0.24 0.61 8.37 9.15 9.53
## difficulty[582] 8.82 0.24 0.59 7.46 8.45 8.82
## difficulty[583] 8.82 0.25 0.59 7.67 8.43 8.79
## difficulty[584] 7.33 0.20 0.53 6.17 7.00 7.38
## difficulty[585] 7.70 0.23 0.55 6.46 7.36 7.70
## difficulty[586] 6.27 0.20 0.50 5.29 5.96 6.26
## difficulty[587] 5.87 0.21 0.50 4.89 5.51 5.83
## difficulty[588] 7.17 0.21 0.52 5.99 6.83 7.18
## difficulty[589] 8.83 0.23 0.60 7.52 8.48 8.84
## difficulty[590] 9.25 0.22 0.58 8.03 8.88 9.27
## difficulty[591] 8.39 0.22 0.55 7.17 8.06 8.38
## difficulty[592] 6.89 0.19 0.50 5.83 6.59 6.90
## difficulty[593] 9.69 0.24 0.61 8.49 9.29 9.73
## difficulty[594] 9.00 0.23 0.57 7.76 8.64 9.01
## difficulty[595] 8.77 0.23 0.57 7.60 8.40 8.76
## difficulty[596] 9.22 0.25 0.61 7.95 8.80 9.24
## difficulty[597] 7.36 0.20 0.53 6.25 7.00 7.42
## difficulty[598] 8.77 0.22 0.57 7.58 8.44 8.76
## difficulty[599] 9.51 0.24 0.61 8.27 9.14 9.47
## difficulty[600] 8.97 0.21 0.59 7.69 8.60 9.00
## difficulty[601] 7.45 0.23 0.54 6.32 7.07 7.46
## difficulty[602] 8.49 0.22 0.56 7.30 8.15 8.50
## difficulty[603] 10.36 0.19 0.64 9.04 9.95 10.43
## difficulty[604] 9.23 0.24 0.60 8.03 8.82 9.20
## difficulty[605] 7.15 0.22 0.54 6.04 6.78 7.18
## difficulty[606] 9.19 0.22 0.59 7.83 8.84 9.25
## difficulty[607] 7.54 0.21 0.56 6.34 7.16 7.60
## difficulty[608] 9.42 0.25 0.61 8.18 9.04 9.40
## difficulty[609] 8.40 0.25 0.57 7.24 8.02 8.39
## difficulty[610] 8.26 0.24 0.57 7.06 7.90 8.27
## difficulty[611] 10.00 0.22 0.63 8.73 9.59 10.04
## difficulty[612] 9.94 0.24 0.62 8.53 9.53 10.03
## difficulty[613] 9.06 0.23 0.58 7.80 8.71 9.04
## difficulty[614] 8.27 0.23 0.59 7.06 7.89 8.30
## difficulty[615] 8.02 0.22 0.56 6.76 7.68 7.99
## difficulty[616] 7.68 0.26 0.56 6.53 7.31 7.64
## difficulty[617] 8.68 0.23 0.59 7.42 8.32 8.70
## difficulty[618] 9.66 0.23 0.61 8.33 9.24 9.73
## difficulty[619] 8.85 0.23 0.58 7.63 8.51 8.83
## difficulty[620] 5.97 0.19 0.49 4.97 5.59 6.01
## difficulty[621] 8.51 0.22 0.57 7.23 8.15 8.52
## difficulty[622] 7.28 0.22 0.54 6.21 6.91 7.31
## difficulty[623] 6.93 0.21 0.51 5.88 6.59 6.94
## difficulty[624] 9.61 0.24 0.59 8.41 9.24 9.58
## difficulty[625] 9.07 0.24 0.60 7.93 8.64 9.06
## difficulty[626] 9.33 0.23 0.58 8.05 8.98 9.32
## difficulty[627] 6.36 0.19 0.50 5.29 6.06 6.35
## difficulty[628] 6.48 0.20 0.52 5.34 6.14 6.54
## difficulty[629] 8.56 0.23 0.56 7.49 8.18 8.60
## difficulty[630] 6.74 0.19 0.49 5.59 6.47 6.75
## difficulty[631] 8.15 0.25 0.58 6.87 7.80 8.12
## difficulty[632] 7.58 0.22 0.56 6.44 7.21 7.61
## difficulty[633] 8.01 0.21 0.54 6.84 7.66 8.02
## difficulty[634] 7.74 0.24 0.55 6.67 7.35 7.76
## difficulty[635] 5.91 0.19 0.50 4.93 5.51 5.96
## difficulty[636] 8.38 0.22 0.57 7.21 8.04 8.37
## difficulty[637] 7.68 0.20 0.55 6.54 7.36 7.75
## difficulty[638] 7.64 0.21 0.55 6.39 7.32 7.65
## difficulty[639] 8.93 0.23 0.61 7.67 8.50 8.97
## difficulty[640] 10.47 0.25 0.65 9.27 10.06 10.47
## difficulty[641] 8.25 0.22 0.56 7.03 7.91 8.25
## difficulty[642] 8.88 0.23 0.57 7.60 8.52 8.89
## difficulty[643] 10.00 0.24 0.60 8.82 9.59 9.95
## difficulty[644] 9.33 0.22 0.62 7.95 8.99 9.36
## difficulty[645] 5.62 0.25 0.89 4.59 5.42 5.67
## difficulty[646] 8.91 0.23 0.59 7.59 8.55 8.92
## difficulty[647] 9.10 0.24 0.59 7.80 8.72 9.09
## difficulty[648] 8.44 0.24 0.57 7.34 8.09 8.45
## difficulty[649] 8.53 0.21 0.56 7.37 8.15 8.56
## difficulty[650] 8.80 0.25 0.59 7.64 8.38 8.74
## difficulty[651] 7.76 0.20 0.52 6.66 7.42 7.76
## difficulty[652] 8.16 0.21 0.55 6.94 7.84 8.17
## difficulty[653] 7.59 0.22 0.56 6.31 7.25 7.61
## difficulty[654] 8.69 0.23 0.57 7.50 8.32 8.70
## difficulty[655] 8.73 0.22 0.57 7.51 8.40 8.68
## difficulty[656] 9.09 0.22 0.58 7.95 8.71 9.09
## difficulty[657] 8.46 0.23 0.58 7.20 8.12 8.46
## difficulty[658] 9.91 0.23 0.65 8.57 9.47 9.98
## difficulty[659] 8.51 0.21 0.57 7.33 8.15 8.51
## difficulty[660] 7.31 0.20 0.51 6.16 6.99 7.33
## difficulty[661] 9.70 0.22 0.59 8.53 9.34 9.75
## difficulty[662] 7.66 0.23 0.54 6.50 7.32 7.64
## difficulty[663] 8.10 0.21 0.55 6.95 7.75 8.15
## difficulty[664] 8.86 0.23 0.58 7.59 8.46 8.87
## difficulty[665] 8.69 0.23 0.57 7.48 8.33 8.70
## difficulty[666] 8.05 0.23 0.57 6.76 7.66 8.01
## difficulty[667] 7.90 0.21 0.54 6.62 7.58 7.89
## difficulty[668] 8.23 0.21 0.56 7.09 7.88 8.26
## difficulty[669] 8.31 0.23 0.56 7.08 7.92 8.31
## difficulty[670] 7.18 0.20 0.53 6.13 6.85 7.21
## difficulty[671] 9.35 0.23 0.60 8.11 8.98 9.34
## difficulty[672] 9.80 0.21 0.60 8.54 9.40 9.88
## difficulty[673] 9.15 0.25 0.60 7.83 8.77 9.15
## difficulty[674] 10.36 0.20 0.67 8.87 9.90 10.43
## difficulty[675] 8.42 0.23 0.54 7.35 8.10 8.40
## difficulty[676] 8.51 0.22 0.55 7.38 8.15 8.53
## difficulty[677] 9.35 0.23 0.58 8.11 9.00 9.38
## difficulty[678] 9.93 0.21 0.62 8.51 9.49 10.02
## difficulty[679] 9.14 0.23 0.57 7.97 8.78 9.15
## difficulty[680] 9.58 0.24 0.59 8.29 9.21 9.56
## lp__ -30505.97 324.95 726.44 -32156.05 -31010.75 -30254.29
## 75% 97.5% n_eff Rhat
## mu_w 7.92 8.15 5 2.18
## mu_s 0.32 0.33 5 2.20
## sigma_w 1.39 1.48 7 1.80
## difficulty[1] 5.98 6.49 6 1.62
## difficulty[2] 6.02 6.57 6 1.68
## difficulty[3] 7.88 8.36 6 1.77
## difficulty[4] 5.39 5.87 10 1.28
## difficulty[5] 5.35 6.00 20 1.16
## difficulty[6] 4.98 5.59 10 1.74
## difficulty[7] 5.57 6.09 7 1.45
## difficulty[8] 5.71 6.22 6 1.54
## difficulty[9] 4.48 5.09 6 1.52
## difficulty[10] 6.02 6.50 7 1.54
## difficulty[11] 5.48 5.95 6 1.53
## difficulty[12] 5.93 6.49 2 9.73
## difficulty[13] 7.62 8.15 7 1.67
## difficulty[14] 7.51 8.03 13 1.22
## difficulty[15] 7.86 8.47 6 1.75
## difficulty[16] 6.24 6.76 7 1.59
## difficulty[17] 6.99 7.57 5 1.77
## difficulty[18] 5.65 6.12 6 1.61
## difficulty[19] 6.19 6.75 8 2.44
## difficulty[20] 6.81 7.32 7 1.75
## difficulty[21] 7.10 7.61 5 1.71
## difficulty[22] 5.76 6.25 6 1.60
## difficulty[23] 6.74 7.29 6 1.69
## difficulty[24] 6.16 6.74 6 1.59
## difficulty[25] 8.53 9.04 6 1.76
## difficulty[26] 4.51 5.25 8 1.51
## difficulty[27] 8.63 9.04 5 1.70
## difficulty[28] 5.25 5.87 8 1.54
## difficulty[29] 7.24 7.71 7 1.65
## difficulty[30] 6.08 6.57 8 3.01
## difficulty[31] 7.03 7.48 6 1.66
## difficulty[32] 7.68 8.21 6 1.60
## difficulty[33] 8.80 9.43 6 1.70
## difficulty[34] 9.51 10.07 7 1.67
## difficulty[35] 6.75 7.18 11 1.27
## difficulty[36] 5.83 6.32 7 1.54
## difficulty[37] 8.72 9.20 7 1.76
## difficulty[38] 7.21 7.65 7 1.63
## difficulty[39] 6.93 7.38 6 1.64
## difficulty[40] 9.24 9.80 7 1.66
## difficulty[41] 7.02 7.58 7 1.60
## difficulty[42] 7.61 8.12 6 1.81
## difficulty[43] 8.53 8.98 6 1.80
## difficulty[44] 6.47 6.87 8 1.63
## difficulty[45] 9.04 9.51 6 1.73
## difficulty[46] 6.63 7.15 6 1.70
## difficulty[47] 8.87 9.45 6 1.72
## difficulty[48] 7.23 7.71 7 1.74
## difficulty[49] 7.93 8.38 6 1.64
## difficulty[50] 7.53 7.91 7 1.58
## difficulty[51] 7.56 8.11 6 1.63
## difficulty[52] 7.96 8.53 7 1.61
## difficulty[53] 7.30 7.77 7 1.69
## difficulty[54] 9.11 9.77 6 1.72
## difficulty[55] 7.78 8.30 6 1.79
## difficulty[56] 6.35 6.88 7 1.57
## difficulty[57] 6.98 7.50 6 1.64
## difficulty[58] 6.43 6.90 11 1.30
## difficulty[59] 6.61 7.16 7 1.63
## difficulty[60] 5.23 5.86 18 1.19
## difficulty[61] 7.65 8.08 7 1.59
## difficulty[62] 8.23 8.72 6 1.78
## difficulty[63] 8.29 8.83 6 1.86
## difficulty[64] 9.49 10.09 6 1.65
## difficulty[65] 8.09 8.63 6 1.78
## difficulty[66] 8.21 8.74 6 1.84
## difficulty[67] 7.00 7.54 7 1.65
## difficulty[68] 9.08 9.59 7 1.63
## difficulty[69] 5.91 6.39 7 1.59
## difficulty[70] 4.08 4.84 11 1.34
## difficulty[71] 5.94 6.46 7 1.64
## difficulty[72] 8.01 8.50 7 1.61
## difficulty[73] 6.99 7.44 6 1.69
## difficulty[74] 5.43 5.87 10 1.36
## difficulty[75] 6.21 6.68 7 1.60
## difficulty[76] 8.83 9.41 7 1.72
## difficulty[77] 7.46 7.92 7 1.65
## difficulty[78] 7.26 7.74 7 1.68
## difficulty[79] 8.14 8.66 7 1.59
## difficulty[80] 8.93 9.49 6 1.70
## difficulty[81] 7.49 7.97 6 1.73
## difficulty[82] 7.97 8.43 7 1.58
## difficulty[83] 8.58 9.07 7 1.64
## difficulty[84] 7.97 8.55 7 1.70
## difficulty[85] 7.54 8.04 6 1.66
## difficulty[86] 7.75 8.23 6 1.81
## difficulty[87] 6.77 7.34 7 1.63
## difficulty[88] 5.98 6.53 33 1.11
## difficulty[89] 7.59 8.13 11 1.25
## difficulty[90] 5.44 5.97 7 1.45
## difficulty[91] 7.78 8.34 6 1.66
## difficulty[92] 6.98 7.49 6 1.73
## difficulty[93] 7.83 8.35 5 1.75
## difficulty[94] 7.12 7.60 7 1.74
## difficulty[95] 7.13 7.74 6 2.60
## difficulty[96] 7.50 8.04 7 1.71
## difficulty[97] 7.27 7.82 5 1.70
## difficulty[98] 7.52 8.04 6 1.68
## difficulty[99] 5.99 6.44 8 2.37
## difficulty[100] 6.92 7.46 7 1.68
## difficulty[101] 7.88 8.32 7 1.58
## difficulty[102] 8.09 8.68 6 1.74
## difficulty[103] 8.46 8.93 6 1.63
## difficulty[104] 5.90 6.41 6 1.63
## difficulty[105] 7.43 7.92 7 1.55
## difficulty[106] 6.18 6.66 6 1.72
## difficulty[107] 8.02 8.56 6 1.75
## difficulty[108] 6.52 6.95 7 1.59
## difficulty[109] 6.95 7.51 7 1.64
## difficulty[110] 7.08 7.51 6 1.66
## difficulty[111] 7.46 7.91 6 1.69
## difficulty[112] 7.15 7.57 7 1.54
## difficulty[113] 7.08 7.63 6 1.65
## difficulty[114] 8.46 8.99 6 1.78
## difficulty[115] 8.38 8.91 7 1.62
## difficulty[116] 7.87 8.33 6 1.76
## difficulty[117] 7.24 7.69 7 1.68
## difficulty[118] 6.92 7.41 6 1.66
## difficulty[119] 8.49 9.00 6 1.71
## difficulty[120] 8.69 9.25 7 1.56
## difficulty[121] 6.09 6.65 2 9.61
## difficulty[122] 8.79 9.27 6 1.69
## difficulty[123] 7.95 8.50 6 1.70
## difficulty[124] 8.93 9.43 7 1.65
## difficulty[125] 5.48 5.96 2 8.96
## difficulty[126] 8.13 8.58 6 1.78
## difficulty[127] 7.91 8.46 6 1.62
## difficulty[128] 8.63 9.17 6 1.72
## difficulty[129] 7.02 7.45 7 1.62
## difficulty[130] 7.60 8.22 5 1.72
## difficulty[131] 7.96 8.47 6 1.64
## difficulty[132] 7.79 8.35 7 1.75
## difficulty[133] 8.16 8.76 5 1.83
## difficulty[134] 6.63 7.09 6 1.62
## difficulty[135] 7.45 7.96 6 1.76
## difficulty[136] 7.99 8.45 7 1.78
## difficulty[137] 7.65 8.17 7 1.62
## difficulty[138] 7.46 8.03 6 1.67
## difficulty[139] 7.96 8.48 7 1.73
## difficulty[140] 9.09 9.63 6 1.75
## difficulty[141] 7.81 8.27 5 1.77
## difficulty[142] 8.11 8.63 6 1.68
## difficulty[143] 9.13 9.61 6 1.71
## difficulty[144] 7.77 8.23 6 1.70
## difficulty[145] 8.67 9.13 7 1.62
## difficulty[146] 8.37 8.86 6 1.83
## difficulty[147] 7.95 8.45 6 1.73
## difficulty[148] 7.38 7.82 7 1.68
## difficulty[149] 7.58 8.16 6 1.70
## difficulty[150] 7.39 8.02 6 1.68
## difficulty[151] 9.64 10.07 7 1.57
## difficulty[152] 9.58 10.35 7 1.48
## difficulty[153] 8.70 9.32 6 1.63
## difficulty[154] 6.35 6.94 6 1.57
## difficulty[155] 7.61 8.14 6 1.75
## difficulty[156] 8.92 9.39 6 1.62
## difficulty[157] 8.14 8.73 6 1.70
## difficulty[158] 7.91 8.27 6 1.68
## difficulty[159] 7.49 7.99 6 1.66
## difficulty[160] 7.01 7.65 13 1.26
## difficulty[161] 6.98 7.43 6 1.71
## difficulty[162] 6.32 6.77 6 1.68
## difficulty[163] 8.20 8.79 7 1.59
## difficulty[164] 8.39 8.91 6 1.71
## difficulty[165] 5.51 5.99 6 1.60
## difficulty[166] 7.45 8.04 6 1.79
## difficulty[167] 8.33 8.86 6 1.66
## difficulty[168] 9.00 9.46 6 1.71
## difficulty[169] 8.25 8.82 6 1.74
## difficulty[170] 7.42 7.93 8 1.64
## difficulty[171] 7.20 7.74 5 1.75
## difficulty[172] 9.68 10.21 7 1.63
## difficulty[173] 6.94 7.52 6 1.59
## difficulty[174] 4.97 5.58 2 2.92
## difficulty[175] 7.89 8.53 6 1.70
## difficulty[176] 8.60 9.22 6 1.75
## difficulty[177] 9.08 9.61 7 1.67
## difficulty[178] 9.80 10.38 7 1.57
## difficulty[179] 6.56 7.10 7 1.57
## difficulty[180] 7.78 8.28 7 1.66
## difficulty[181] 9.68 10.27 7 1.63
## difficulty[182] 8.67 9.13 6 1.70
## difficulty[183] 7.88 8.32 6 1.72
## difficulty[184] 9.41 9.98 7 1.66
## difficulty[185] 6.92 7.42 7 1.62
## difficulty[186] 6.87 7.42 6 1.61
## difficulty[187] 6.94 7.41 6 1.73
## difficulty[188] 7.65 8.10 6 1.69
## difficulty[189] 7.46 7.93 6 1.71
## difficulty[190] 5.74 6.20 8 1.53
## difficulty[191] 5.38 5.83 7 4.97
## difficulty[192] 7.81 8.31 7 1.66
## difficulty[193] 7.00 7.44 6 1.63
## difficulty[194] 6.61 7.19 6 1.67
## difficulty[195] 6.30 6.88 7 1.66
## difficulty[196] 6.71 7.27 6 1.61
## difficulty[197] 6.78 7.27 11 1.27
## difficulty[198] 7.34 7.90 6 1.71
## difficulty[199] 7.30 7.77 6 1.65
## difficulty[200] 8.11 8.54 6 1.62
## difficulty[201] 6.36 6.85 8 1.60
## difficulty[202] 5.72 6.23 7 1.59
## difficulty[203] 5.96 6.45 4 2.21
## difficulty[204] 7.91 8.39 6 1.72
## difficulty[205] 8.40 8.96 6 1.65
## difficulty[206] 6.63 7.15 7 1.68
## difficulty[207] 7.20 7.61 6 1.74
## difficulty[208] 7.01 7.54 7 1.55
## difficulty[209] 6.99 7.49 6 1.71
## difficulty[210] 9.37 9.87 7 1.64
## difficulty[211] 7.87 8.33 7 1.72
## difficulty[212] 6.47 7.00 7 1.60
## difficulty[213] 6.53 7.03 9 1.32
## difficulty[214] 6.96 7.57 6 1.65
## difficulty[215] 7.17 7.72 7 1.70
## difficulty[216] 7.56 8.07 5 1.81
## difficulty[217] 6.91 7.39 15 1.28
## difficulty[218] 8.09 8.74 6 1.73
## difficulty[219] 8.05 8.57 6 1.72
## difficulty[220] 8.31 8.74 6 1.66
## difficulty[221] 7.40 7.86 7 1.61
## difficulty[222] 7.84 8.33 6 1.87
## difficulty[223] 6.06 6.57 4 2.58
## difficulty[224] 8.39 8.94 6 1.71
## difficulty[225] 6.66 7.20 6 1.69
## difficulty[226] 7.96 8.48 7 1.64
## difficulty[227] 8.20 8.76 6 1.71
## difficulty[228] 7.10 7.60 2 11.43
## difficulty[229] 8.05 8.57 6 1.80
## difficulty[230] 8.36 8.91 5 1.71
## difficulty[231] 6.77 7.24 7 1.66
## difficulty[232] 7.93 8.40 6 1.71
## difficulty[233] 8.78 9.29 6 1.72
## difficulty[234] 6.31 6.75 6 1.61
## difficulty[235] 7.63 8.15 6 1.75
## difficulty[236] 7.50 8.06 6 1.69
## difficulty[237] 8.82 9.33 6 1.73
## difficulty[238] 8.85 9.30 6 1.70
## difficulty[239] 7.86 8.38 6 1.77
## difficulty[240] 7.41 7.97 7 1.68
## difficulty[241] 8.12 8.61 6 1.72
## difficulty[242] 7.82 8.37 6 1.78
## difficulty[243] 7.34 7.76 6 1.75
## difficulty[244] 8.46 8.89 6 1.77
## difficulty[245] 7.53 8.03 7 1.71
## difficulty[246] 7.69 8.14 6 1.71
## difficulty[247] 8.76 9.41 6 1.65
## difficulty[248] 8.09 8.62 5 1.76
## difficulty[249] 6.96 7.43 6 1.67
## difficulty[250] 6.31 6.83 2 9.94
## difficulty[251] 8.31 8.75 6 1.69
## difficulty[252] 6.73 7.25 15 1.25
## difficulty[253] 7.71 8.18 6 1.76
## difficulty[254] 6.71 7.17 16 1.24
## difficulty[255] 7.00 7.57 6 1.59
## difficulty[256] 7.14 7.69 16 1.25
## difficulty[257] 9.96 10.58 7 1.53
## difficulty[258] 7.64 8.14 6 1.67
## difficulty[259] 9.92 10.66 6 1.55
## difficulty[260] 7.65 8.13 6 1.73
## difficulty[261] 9.49 10.10 6 1.58
## difficulty[262] 7.40 7.90 6 1.74
## difficulty[263] 6.72 7.25 7 1.55
## difficulty[264] 6.45 6.99 6 1.66
## difficulty[265] 7.87 8.38 6 1.69
## difficulty[266] 9.56 9.99 7 1.65
## difficulty[267] 6.67 7.16 7 1.56
## difficulty[268] 8.03 8.56 7 1.71
## difficulty[269] 7.63 8.27 5 1.71
## difficulty[270] 8.14 8.71 5 1.78
## difficulty[271] 6.32 6.78 7 1.61
## difficulty[272] 8.25 8.75 6 1.81
## difficulty[273] 8.87 9.40 6 1.72
## difficulty[274] 8.60 9.13 6 1.72
## difficulty[275] 7.89 8.41 6 1.63
## difficulty[276] 7.60 8.19 6 1.66
## difficulty[277] 8.37 8.93 6 1.71
## difficulty[278] 8.49 9.00 6 1.76
## difficulty[279] 9.34 9.92 7 1.52
## difficulty[280] 9.15 9.64 7 1.61
## difficulty[281] 6.99 7.41 7 1.70
## difficulty[282] 8.08 8.64 6 1.62
## difficulty[283] 8.17 8.62 6 1.70
## difficulty[284] 7.53 8.03 7 1.55
## difficulty[285] 7.55 8.01 8 1.61
## difficulty[286] 7.96 8.41 6 1.78
## difficulty[287] 9.57 10.21 6 1.70
## difficulty[288] 8.02 8.54 7 1.67
## difficulty[289] 8.12 8.66 6 1.73
## difficulty[290] 7.55 8.07 7 1.64
## difficulty[291] 7.22 7.71 6 1.73
## difficulty[292] 7.92 8.45 6 1.68
## difficulty[293] 8.29 8.77 6 1.64
## difficulty[294] 8.42 9.10 6 1.78
## difficulty[295] 7.78 8.28 6 1.67
## difficulty[296] 8.31 8.82 7 1.66
## difficulty[297] 6.31 6.95 7 1.62
## difficulty[298] 9.36 9.91 7 1.57
## difficulty[299] 7.63 8.17 6 1.65
## difficulty[300] 8.67 9.17 7 1.66
## difficulty[301] 8.32 8.71 6 1.72
## difficulty[302] 8.57 9.20 5 1.73
## difficulty[303] 6.53 6.95 8 1.56
## difficulty[304] 7.35 7.85 6 1.65
## difficulty[305] 6.90 7.47 6 1.65
## difficulty[306] 7.13 7.65 6 1.68
## difficulty[307] 8.92 9.52 7 1.66
## difficulty[308] 8.61 9.02 6 1.72
## difficulty[309] 8.09 8.53 6 1.71
## difficulty[310] 8.47 9.02 6 1.65
## difficulty[311] 7.46 7.97 7 1.69
## difficulty[312] 7.11 7.67 6 1.69
## difficulty[313] 7.94 8.44 7 1.65
## difficulty[314] 8.63 9.13 7 1.73
## difficulty[315] 9.29 9.81 6 1.69
## difficulty[316] 6.98 7.54 6 1.75
## difficulty[317] 7.61 8.08 7 1.68
## difficulty[318] 9.46 9.97 6 1.65
## difficulty[319] 8.05 8.54 6 1.62
## difficulty[320] 7.27 7.79 6 1.65
## difficulty[321] 7.16 7.61 7 1.70
## difficulty[322] 6.36 6.77 8 1.70
## difficulty[323] 6.55 7.01 6 1.67
## difficulty[324] 8.20 8.66 6 1.74
## difficulty[325] 8.57 9.13 6 1.70
## difficulty[326] 9.88 10.47 7 1.58
## difficulty[327] 8.44 8.94 6 1.73
## difficulty[328] 9.48 10.08 6 1.68
## difficulty[329] 10.54 11.17 8 1.38
## difficulty[330] 9.67 10.31 7 1.58
## difficulty[331] 8.73 9.26 5 1.79
## difficulty[332] 8.87 9.34 7 1.61
## difficulty[333] 6.51 7.02 7 1.52
## difficulty[334] 7.19 7.73 6 1.59
## difficulty[335] 8.25 8.81 6 1.75
## difficulty[336] 6.26 6.64 8 1.60
## difficulty[337] 7.16 7.67 7 1.67
## difficulty[338] 7.96 8.46 6 1.83
## difficulty[339] 9.66 10.21 7 1.59
## difficulty[340] 8.13 8.61 6 1.75
## difficulty[341] 7.27 7.77 6 1.71
## difficulty[342] 7.42 8.01 5 1.72
## difficulty[343] 9.77 10.30 6 1.62
## difficulty[344] 7.25 7.74 6 1.66
## difficulty[345] 8.84 9.45 5 1.83
## difficulty[346] 7.64 8.09 6 1.77
## difficulty[347] 7.93 8.42 6 1.70
## difficulty[348] 5.03 5.61 12 1.24
## difficulty[349] 9.68 10.27 6 1.66
## difficulty[350] 7.62 8.18 6 1.73
## difficulty[351] 7.53 8.04 7 1.63
## difficulty[352] 7.86 8.38 8 1.64
## difficulty[353] 9.69 10.26 6 1.61
## difficulty[354] 8.40 8.90 6 1.67
## difficulty[355] 9.19 9.82 7 1.60
## difficulty[356] 3.46 4.24 18 1.25
## difficulty[357] 7.83 8.38 6 1.72
## difficulty[358] 8.64 9.19 6 1.66
## difficulty[359] 8.27 8.76 7 1.68
## difficulty[360] 7.50 8.03 6 1.79
## difficulty[361] 5.66 6.20 6 1.66
## difficulty[362] 5.99 6.46 7 1.71
## difficulty[363] 8.49 9.09 6 1.69
## difficulty[364] 9.20 9.70 6 1.67
## difficulty[365] 7.90 8.38 6 1.83
## difficulty[366] 2.99 4.03 5 2.20
## difficulty[367] 9.97 10.59 7 1.52
## difficulty[368] 6.69 7.21 6 1.66
## difficulty[369] 8.33 8.85 5 1.79
## difficulty[370] 9.68 10.32 7 1.66
## difficulty[371] 7.02 7.50 2 11.79
## difficulty[372] 8.81 9.43 6 1.67
## difficulty[373] 8.37 8.86 6 1.64
## difficulty[374] 8.71 9.18 6 1.73
## difficulty[375] 8.04 8.49 7 1.68
## difficulty[376] 5.54 5.99 7 1.55
## difficulty[377] 7.64 8.04 6 1.74
## difficulty[378] 4.40 5.07 3 3.02
## difficulty[379] 7.14 7.69 6 1.75
## difficulty[380] 7.54 8.07 6 1.76
## difficulty[381] 7.73 8.25 6 1.70
## difficulty[382] 7.77 8.20 6 1.75
## difficulty[383] 7.02 7.51 6 1.66
## difficulty[384] 5.68 6.21 32 1.11
## difficulty[385] 5.14 5.61 9 1.77
## difficulty[386] 7.50 7.98 6 1.73
## difficulty[387] 6.85 7.38 6 1.62
## difficulty[388] 5.62 6.10 8 1.74
## difficulty[389] 4.46 4.87 14 1.27
## difficulty[390] 7.72 8.17 7 1.69
## difficulty[391] 5.90 6.48 9 1.42
## difficulty[392] 5.94 6.44 2 9.84
## difficulty[393] 5.73 6.21 6 1.66
## difficulty[394] 8.04 8.45 6 1.72
## difficulty[395] 7.60 8.14 6 1.72
## difficulty[396] 8.34 8.95 6 1.65
## difficulty[397] 5.80 6.24 3 3.15
## difficulty[398] 7.79 8.34 6 1.61
## difficulty[399] 8.08 8.60 6 1.70
## difficulty[400] 5.63 6.07 9 1.39
## difficulty[401] 6.84 7.44 5 1.77
## difficulty[402] 7.44 7.86 6 1.79
## difficulty[403] 7.75 8.33 6 1.67
## difficulty[404] 8.20 8.71 6 1.74
## difficulty[405] 8.49 9.01 6 1.71
## difficulty[406] 8.22 8.64 6 1.68
## difficulty[407] 8.56 8.97 7 1.66
## difficulty[408] 7.98 8.50 6 1.67
## difficulty[409] 8.02 8.55 5 1.78
## difficulty[410] 8.48 8.97 5 1.80
## difficulty[411] 7.09 7.69 7 1.69
## difficulty[412] 7.52 7.97 2 11.05
## difficulty[413] 7.92 8.48 6 1.75
## difficulty[414] 7.61 8.21 7 1.66
## difficulty[415] 7.58 8.02 7 1.76
## difficulty[416] 8.52 9.15 6 1.69
## difficulty[417] 7.29 7.85 6 1.71
## difficulty[418] 8.28 8.84 6 1.88
## difficulty[419] 7.17 7.66 6 1.68
## difficulty[420] 7.88 8.43 6 1.66
## difficulty[421] 6.38 6.98 9 1.59
## difficulty[422] 7.68 8.22 6 1.62
## difficulty[423] 8.04 8.67 6 1.81
## difficulty[424] 8.14 8.81 6 1.72
## difficulty[425] 9.17 9.71 6 1.69
## difficulty[426] 6.14 6.58 13 1.32
## difficulty[427] 7.58 8.07 6 1.78
## difficulty[428] 8.30 8.81 7 1.75
## difficulty[429] 7.98 8.50 7 1.59
## difficulty[430] 8.77 9.24 6 1.67
## difficulty[431] 8.69 9.25 6 1.72
## difficulty[432] 7.73 8.22 7 1.70
## difficulty[433] 7.36 7.78 7 1.62
## difficulty[434] 8.04 8.51 6 1.75
## difficulty[435] 5.96 6.41 8 1.69
## difficulty[436] 9.88 10.44 6 1.63
## difficulty[437] 8.02 8.47 6 1.68
## difficulty[438] 8.09 8.51 7 1.59
## difficulty[439] 7.23 7.73 6 1.78
## difficulty[440] 7.78 8.31 7 1.77
## difficulty[441] 7.77 8.28 6 1.71
## difficulty[442] 7.85 8.39 6 1.71
## difficulty[443] 6.90 7.35 10 1.33
## difficulty[444] 8.23 8.78 6 1.71
## difficulty[445] 7.51 7.98 6 1.74
## difficulty[446] 7.44 7.89 6 1.68
## difficulty[447] 6.72 7.28 14 1.20
## difficulty[448] 7.93 8.45 6 1.70
## difficulty[449] 8.29 8.72 6 1.81
## difficulty[450] 7.98 8.55 6 1.66
## difficulty[451] 8.46 8.96 6 1.71
## difficulty[452] 7.36 7.84 14 1.23
## difficulty[453] 7.15 7.60 13 1.34
## difficulty[454] 8.04 8.58 6 1.71
## difficulty[455] 6.76 7.26 14 1.21
## difficulty[456] 8.53 9.07 6 1.69
## difficulty[457] 8.61 9.13 6 1.65
## difficulty[458] 6.82 7.36 6 1.65
## difficulty[459] 8.69 9.24 6 1.71
## difficulty[460] 9.78 10.38 6 1.67
## difficulty[461] 8.32 8.90 6 1.69
## difficulty[462] 7.65 8.15 6 1.71
## difficulty[463] 8.15 8.65 6 1.75
## difficulty[464] 7.24 7.73 5 1.75
## difficulty[465] 7.29 7.83 6 1.62
## difficulty[466] 9.49 10.02 6 1.69
## difficulty[467] 7.77 8.32 6 1.76
## difficulty[468] 8.24 8.78 7 1.71
## difficulty[469] 7.00 7.50 5 1.68
## difficulty[470] 8.10 8.61 5 1.71
## difficulty[471] 8.28 8.80 7 1.66
## difficulty[472] 8.57 9.12 6 1.74
## difficulty[473] 7.32 7.81 6 1.73
## difficulty[474] 6.75 7.34 6 1.79
## difficulty[475] 9.69 10.25 6 1.64
## difficulty[476] 6.92 7.35 6 1.67
## difficulty[477] 7.50 8.00 7 1.69
## difficulty[478] 8.19 8.73 6 1.69
## difficulty[479] 8.05 8.68 6 1.71
## difficulty[480] 7.88 8.42 7 1.65
## difficulty[481] 7.99 8.48 6 1.64
## difficulty[482] 7.99 8.46 6 1.76
## difficulty[483] 6.71 7.30 6 1.60
## difficulty[484] 7.86 8.41 7 1.73
## difficulty[485] 7.68 8.18 6 1.70
## difficulty[486] 7.27 7.78 6 1.76
## difficulty[487] 8.16 8.74 7 1.59
## difficulty[488] 7.88 8.38 5 1.68
## difficulty[489] 8.54 9.09 6 1.82
## difficulty[490] 9.18 9.77 6 1.71
## difficulty[491] 9.50 10.04 6 1.63
## difficulty[492] 7.53 8.04 6 1.78
## difficulty[493] 7.22 7.67 7 1.77
## difficulty[494] 7.79 8.35 7 1.59
## difficulty[495] 8.24 8.77 6 1.73
## difficulty[496] 8.25 8.82 6 1.69
## difficulty[497] 7.14 7.60 7 1.67
## difficulty[498] 7.50 8.02 6 1.68
## difficulty[499] 7.86 8.29 5 1.83
## difficulty[500] 8.04 8.42 7 1.62
## difficulty[501] 10.77 11.47 8 1.39
## difficulty[502] 7.41 7.98 6 1.74
## difficulty[503] 8.31 8.89 6 1.71
## difficulty[504] 6.04 6.56 8 1.60
## difficulty[505] 7.23 7.76 7 1.75
## difficulty[506] 8.08 8.62 6 1.71
## difficulty[507] 7.34 7.85 7 1.72
## difficulty[508] 8.41 9.04 6 1.69
## difficulty[509] 6.96 7.36 5 1.80
## difficulty[510] 8.19 8.70 6 1.68
## difficulty[511] 7.09 7.62 5 1.73
## difficulty[512] 7.54 8.00 6 1.70
## difficulty[513] 8.64 9.19 6 1.74
## difficulty[514] 8.00 8.54 6 1.69
## difficulty[515] 7.52 8.07 6 1.75
## difficulty[516] 6.40 6.92 6 1.57
## difficulty[517] 8.21 8.72 6 1.70
## difficulty[518] 8.32 8.95 6 1.75
## difficulty[519] 6.98 7.52 6 1.62
## difficulty[520] 7.91 8.43 5 1.84
## difficulty[521] 8.18 8.76 7 1.70
## difficulty[522] 8.08 8.48 6 1.74
## difficulty[523] 9.67 10.33 7 1.54
## difficulty[524] 8.71 9.20 7 1.67
## difficulty[525] 8.69 9.14 7 1.64
## difficulty[526] 8.34 8.87 6 1.68
## difficulty[527] 7.37 7.79 5 1.71
## difficulty[528] 8.02 8.53 6 1.70
## difficulty[529] 7.68 8.17 6 1.70
## difficulty[530] 8.60 9.11 6 1.72
## difficulty[531] 7.93 8.44 7 1.65
## difficulty[532] 8.00 8.51 6 1.74
## difficulty[533] 5.21 5.72 8 1.53
## difficulty[534] 7.59 8.18 6 1.67
## difficulty[535] 7.40 7.93 6 1.71
## difficulty[536] 9.43 9.94 7 1.57
## difficulty[537] 8.09 8.62 7 1.71
## difficulty[538] 9.16 9.74 6 1.64
## difficulty[539] 8.04 8.68 6 1.67
## difficulty[540] 8.80 9.27 6 1.75
## difficulty[541] 9.56 10.13 7 1.59
## difficulty[542] 8.54 9.13 7 1.61
## difficulty[543] 7.61 8.01 6 1.61
## difficulty[544] 9.15 9.68 7 1.73
## difficulty[545] 9.08 9.67 6 1.68
## difficulty[546] 7.77 8.28 5 1.80
## difficulty[547] 9.80 10.29 8 1.58
## difficulty[548] 7.01 7.53 6 1.65
## difficulty[549] 8.25 8.73 8 1.63
## difficulty[550] 7.81 8.27 6 1.61
## difficulty[551] 8.06 8.61 6 1.66
## difficulty[552] 8.19 8.72 6 1.66
## difficulty[553] 8.27 8.87 5 1.78
## difficulty[554] 8.02 8.51 6 1.61
## difficulty[555] 9.07 9.54 7 1.69
## difficulty[556] 7.76 8.28 7 1.63
## difficulty[557] 8.00 8.49 6 1.71
## difficulty[558] 7.43 7.89 7 1.72
## difficulty[559] 8.13 8.65 6 1.70
## difficulty[560] 9.40 10.18 6 1.70
## difficulty[561] 8.09 8.62 6 1.73
## difficulty[562] 6.94 7.44 6 1.79
## difficulty[563] 8.47 8.97 6 1.76
## difficulty[564] 8.63 9.17 7 1.64
## difficulty[565] 7.53 8.02 6 1.74
## difficulty[566] 6.87 7.42 6 1.67
## difficulty[567] 9.09 9.65 6 1.63
## difficulty[568] 9.83 10.52 7 1.50
## difficulty[569] 8.47 9.07 6 1.71
## difficulty[570] 8.51 9.00 6 1.67
## difficulty[571] 8.42 8.94 7 1.67
## difficulty[572] 7.48 7.92 6 1.71
## difficulty[573] 7.89 8.50 6 1.64
## difficulty[574] 9.59 10.23 6 1.73
## difficulty[575] 9.22 9.77 6 1.68
## difficulty[576] 8.90 9.40 7 1.64
## difficulty[577] 9.41 9.93 6 1.86
## difficulty[578] 9.90 10.48 7 1.63
## difficulty[579] 8.75 9.33 6 1.66
## difficulty[580] 9.28 9.85 7 1.64
## difficulty[581] 9.96 10.68 7 1.56
## difficulty[582] 9.28 9.87 6 1.68
## difficulty[583] 9.27 9.87 6 1.68
## difficulty[584] 7.72 8.19 7 1.58
## difficulty[585] 8.11 8.63 6 1.73
## difficulty[586] 6.68 7.15 6 1.70
## difficulty[587] 6.24 6.75 5 1.74
## difficulty[588] 7.61 8.01 6 1.73
## difficulty[589] 9.24 9.83 7 1.55
## difficulty[590] 9.67 10.35 7 1.53
## difficulty[591] 8.81 9.34 6 1.61
## difficulty[592] 7.26 7.76 7 1.72
## difficulty[593] 10.15 10.71 7 1.52
## difficulty[594] 9.42 9.98 6 1.76
## difficulty[595] 9.22 9.77 6 1.70
## difficulty[596] 9.66 10.27 6 1.68
## difficulty[597] 7.76 8.25 7 1.66
## difficulty[598] 9.18 9.79 7 1.67
## difficulty[599] 9.94 10.68 6 1.56
## difficulty[600] 9.39 9.98 8 1.55
## difficulty[601] 7.82 8.40 6 1.83
## difficulty[602] 8.90 9.47 6 1.59
## difficulty[603] 10.84 11.39 11 1.34
## difficulty[604] 9.70 10.36 6 1.66
## difficulty[605] 7.54 8.08 6 1.63
## difficulty[606] 9.64 10.06 7 1.56
## difficulty[607] 7.96 8.45 7 1.64
## difficulty[608] 9.87 10.50 6 1.59
## difficulty[609] 8.86 9.40 5 1.72
## difficulty[610] 8.71 9.26 6 1.71
## difficulty[611] 10.47 11.08 9 1.48
## difficulty[612] 10.46 10.87 7 1.58
## difficulty[613] 9.51 10.08 6 1.60
## difficulty[614] 8.76 9.24 7 1.71
## difficulty[615] 8.44 9.00 6 1.73
## difficulty[616] 8.15 8.62 5 1.81
## difficulty[617] 9.11 9.71 7 1.56
## difficulty[618] 10.10 10.68 7 1.56
## difficulty[619] 9.30 9.88 6 1.62
## difficulty[620] 6.34 6.81 7 1.68
## difficulty[621] 8.92 9.49 7 1.64
## difficulty[622] 7.70 8.18 6 1.75
## difficulty[623] 7.30 7.86 6 1.65
## difficulty[624] 10.04 10.71 6 1.47
## difficulty[625] 9.53 10.11 6 1.61
## difficulty[626] 9.74 10.31 7 1.54
## difficulty[627] 6.72 7.25 7 1.60
## difficulty[628] 6.84 7.38 7 1.60
## difficulty[629] 8.98 9.53 6 1.75
## difficulty[630] 7.12 7.63 7 1.57
## difficulty[631] 8.60 9.14 5 1.78
## difficulty[632] 8.00 8.54 7 1.65
## difficulty[633] 8.43 8.89 6 1.76
## difficulty[634] 8.13 8.73 5 1.68
## difficulty[635] 6.29 6.75 7 1.70
## difficulty[636] 8.78 9.40 7 1.71
## difficulty[637] 8.11 8.60 7 1.74
## difficulty[638] 8.08 8.53 7 1.70
## difficulty[639] 9.35 9.96 7 1.63
## difficulty[640] 10.90 11.70 7 1.40
## difficulty[641] 8.66 9.24 6 1.69
## difficulty[642] 9.29 9.87 6 1.67
## difficulty[643] 10.45 11.14 6 1.51
## difficulty[644] 9.76 10.43 8 1.63
## difficulty[645] 6.05 6.61 13 1.22
## difficulty[646] 9.37 9.87 7 1.65
## difficulty[647] 9.53 10.09 6 1.61
## difficulty[648] 8.86 9.40 6 1.69
## difficulty[649] 8.97 9.48 7 1.73
## difficulty[650] 9.22 9.86 5 1.76
## difficulty[651] 8.14 8.67 7 1.72
## difficulty[652] 8.55 9.17 7 1.63
## difficulty[653] 8.02 8.46 7 1.72
## difficulty[654] 9.14 9.75 6 1.60
## difficulty[655] 9.11 9.79 7 1.69
## difficulty[656] 9.53 10.12 7 1.63
## difficulty[657] 8.93 9.39 6 1.73
## difficulty[658] 10.41 10.98 8 1.53
## difficulty[659] 8.95 9.49 7 1.68
## difficulty[660] 7.70 8.18 6 1.64
## difficulty[661] 10.11 10.70 7 1.60
## difficulty[662] 8.08 8.56 5 1.73
## difficulty[663] 8.51 8.97 7 1.64
## difficulty[664] 9.32 9.81 6 1.61
## difficulty[665] 9.10 9.64 6 1.69
## difficulty[666] 8.51 9.05 6 1.71
## difficulty[667] 8.31 8.81 6 1.72
## difficulty[668] 8.65 9.15 7 1.70
## difficulty[669] 8.78 9.25 6 1.73
## difficulty[670] 7.59 8.11 7 1.62
## difficulty[671] 9.79 10.48 7 1.58
## difficulty[672] 10.22 10.83 9 1.49
## difficulty[673] 9.59 10.30 6 1.60
## difficulty[674] 10.86 11.43 11 1.42
## difficulty[675] 8.85 9.39 6 1.70
## difficulty[676] 8.89 9.48 6 1.69
## difficulty[677] 9.77 10.34 7 1.57
## difficulty[678] 10.38 11.00 9 1.44
## difficulty[679] 9.56 10.11 6 1.62
## difficulty[680] 10.04 10.63 6 1.69
## lp__ -29787.17 -29680.00 5 2.17
##
## Samples were drawn using NUTS(diag_e) at Wed Aug 5 10:00:09 2015.
## For each parameter, n_eff is a crude measure of effective sample size,
## and Rhat is the potential scale reduction factor on split chains (at
## convergence, Rhat=1).Now re-merge.
coefs <- data.frame(summary(samps)$summary)
coefs$name <- rownames(coefs)
word_ids <- ws %>%
group_by(num_item_id) %>%
summarise(item = unique(item))
words <- coefs %>%
filter(str_detect(name, "difficulty")) %>%
separate(name, c("name", "num_item_id"), "\\[") %>%
mutate(num_item_id = as.numeric(str_replace(num_item_id, "]", ""))) %>%
left_join(word_ids) %>%
select(mean, se_mean, sd, item) %>%
arrange(mean) %>%
mutate(item = factor(item,
levels = item,
labels = item))and plot again:
qplot(item, mean, data=words[1:50,]) +
theme(axis.text.x = element_text(angle = 90, hjust = 1))
So clearly this model is getting us something. The only worry is that it’s still so slow.
Let’s try rewriting it so it is more vectorized? Hard to do under current Stan infrastructure.
Attempt #3: Binomials
But we can do an age aggregation and use binomials. (Since we’re not fitting subjects).
mcmurray_binom <-'
data {
int<lower=1> W; // number of words
int<lower=1> A; // number of ages
int<lower=1> N; // number of datapoints (W*A)
vector[A] age; // subject ages
int produces[N]; // count data
int attempts[N]; // count data
}
parameters {
real<lower=0> mu_w; // word mean
real<lower=0> mu_s; // subject mean
real<lower=0> sigma_w; // word SD
real<lower=0> difficulty[W]; // word mean
}
model {
mu_w ~ normal(0, 10);
mu_s ~ normal(0, 10);
sigma_w ~ normal(0, 5);
difficulty ~ normal(mu_w, sigma_w);
for (a in 1:A)
for (w in 1:W)
produces[W*(a-1) + w] ~ binomial_logit(attempts[W*(a-1) + w], (mu_s * age[a]) - difficulty[w]);
}
'Now run this with the full dataset. Interestingly, thinning is not recommended by many authors: see [http://doingbayesiandataanalysis.blogspot.ca/2011/11/thinning-to-reduce-autocorrelation.html]
n.words <- 680
complete <- ws %>%
group_by(data_id) %>%
summarize(complete = sum(!is.na(value)) == n.words) %>%
filter(complete)
model.data <- ws %>%
select(data_id, num_item_id, item, value, age) %>%
filter(data_id %in% complete$data_id) %>%
group_by(age, num_item_id) %>%
summarise(n = sum(value),
N = length(value))
ages <- unique(model.data$age)
dat <- list(age = ages,
produces = model.data$n,
attempts = model.data$N,
W = n.words,
A = length(ages),
N = n.words * length(ages))
ptm <- proc.time()
samps <- stan(model_code = mcmurray_binom,
cores = 4,
data = dat, iter = 300, warmup=100, thin=10, chains = 24,
pars = c("mu_w", "mu_s", "sigma_w", "difficulty")) ## COMPILING THE C++ CODE FOR MODEL '5ce82d8c94af70eb89782a919fbc669f' NOW.proc.time() - ptm## user system elapsed
## 32.402 2.807 204.931Diagnostics.
traceplot(samps, pars = c('mu_w','mu_s','sigma_w'))
Success!
coefs <- data.frame(summary(samps)$summary)
coefs$name <- rownames(coefs)
word_ids <- ws %>%
group_by(num_item_id) %>%
summarise(item = unique(item))
words <- coefs %>%
filter(str_detect(name, "difficulty")) %>%
separate(name, c("name", "num_item_id"), "\\[") %>%
mutate(num_item_id = as.numeric(str_replace(num_item_id, "]", ""))) %>%
left_join(word_ids) %>%
select(mean, se_mean, sd, item) %>%
arrange(mean) %>%
mutate(item = factor(item,
levels = item,
labels = item))and plot again:
qplot(mean, as.numeric(item),
geom = "point", data=words) +
geom_text(aes(x = mean + .2, label = item),
data = words[seq(1,680, 10),],
size = 2)
Check the histogram:
qplot(mean, geom = "blank", data = words) +
geom_histogram(aes(y = ..density..)) +
geom_density(col = "red")
What are the words with means below 1?
filter(words, mean < 1)## mean se_mean sd item
## 1 0.07866137 0.02970395 0.04227042 daddy
## 2 0.08430639 0.03116399 0.04425437 mommy
## 3 0.10447006 0.01989400 0.02834735 ball
## 4 0.10484346 0.02664622 0.03800157 bye
## 5 0.12863741 0.02971150 0.04219324 uh_oh
## 6 0.13190844 0.03823441 0.05421208 no
## 7 0.13629563 0.05586900 0.07915484 book
## 8 0.13650146 0.03106074 0.04407932 dog
## 9 0.15771244 0.05563103 0.07887128 baby
## 10 0.16074499 0.05847588 0.08304233 hi
## 11 0.17849331 0.07511224 0.10746221 car
## 12 0.18017045 0.02596099 0.03767650 shoe
## 13 0.18602319 0.07130439 0.10204358 juice
## 14 0.18850008 0.07338718 0.10457646 eye
## 15 0.20954523 0.02196423 0.03199990 moo
## 16 0.21262604 0.04603872 0.06738063 bird
## 17 0.21585838 0.07554811 0.10759685 woof_woof
## 18 0.22250346 0.02982616 0.04342450 banana
## 19 0.24575384 0.02708712 0.04170806 duck
## 20 0.25068177 0.10549353 0.15214774 balloon
## 21 0.27378803 0.03022874 0.04759371 meow
## 22 0.29671951 0.11898750 0.17312928 baa_baa
## 23 0.31312457 0.09542164 0.14351618 nose
## 24 0.35050920 0.13096858 0.19123260 apple
## 25 0.35677276 0.13592624 0.19878051 milk
## 26 0.37641563 0.18093183 0.26568744 ouch
## 27 0.38399311 0.06002686 0.09711413 hot
## 28 0.40662639 0.14753957 0.22004622 cheese
## 29 0.40687269 0.15173097 0.22780143 nightnight
## 30 0.46467848 0.20734438 0.30986546 quack_quack
## 31 0.52822860 0.23399173 0.35907858 hat
## 32 0.57959851 0.25838279 0.39580834 kitty
## 33 0.58218936 0.24439748 0.41509839 grandma
## 34 0.63385357 0.26639390 0.46131911 grrr
## 35 0.65212596 0.23823529 0.43062827 cookie
## 36 0.67479705 0.27047432 0.46438231 ear
## 37 0.75866744 0.39814320 0.60867500 water.beverage
## 38 0.79949471 0.53812639 0.78697826 cup
## 39 0.95883248 0.78482962 1.11847386 bubbles
## 40 0.96265875 0.47192226 0.71488456 peekaboo
## 41 0.97274645 0.76826542 1.09251239 yes
## 42 0.97636252 0.52863052 0.77498290 owie