Particularly regarding the recently collected pilot data, consisting of 33 mTurk subjects.
Marginal distributions of ratings
First thing, let’s take a look at the overall distribution of responses to see if there are any red flags.
msubj <- subjdat[!str_detect(uid,"test") & task_complete,uid]
scendat <- scendat[uid %in% msubj]
clickdat <- clickdat[!is.na(guilty)]
dat <- merge(scendat,clickdat,by=c("uid","scenario"))
dat[,period:=cut_number(question,4)]
qplot(dat$rating) + xlab("case strength")
qplot(dat$rate_punishment) + xlab("punishability")
For case strength we’re getting a lot of clumping on the lower end of the scale, but overall people are using the entire scale. Interesting how U-shaped it seems to be, maybe because it’s clear what a 0 pt case is and what a 100 pt case is but it’s not so obvious what a 50 pt case looks like. For punishment, on the other hand, we see a lot of people maxing out the scale.
Of course what we really want to see is that individuals are using the whole scale, not just the population. That’s tougher to visualize with so many participants but we can give it a go.
ggplot(dat,aes(x=rating,color=uid)) + stat_ecdf(geom="step", pad=FALSE) + theme(legend.position = "none") + xlab("case strength") + ylab("proportion")
ggplot(dat,aes(x=rate_punishment,color=uid)) + stat_ecdf(geom="step", pad=FALSE) + theme(legend.position = "none") + xlab("punishability") + ylab("proportion")
Here we’re looking at the empirical cummulative distributions for each subject – so, for each value on the x-axis, what proportion of their ratings were below that value. It appears that use of the case strength scale is fairly consistent across subjects, but punishment seems more… idiosyncratic, with more subjects leaning more heavily on one side of the scale or another.
We can also see this by just plotting the standard deviations of each participant’s ratings:
ggplot(dat[,.(rating=sd(rating),rate_punishment=sd(rate_punishment)),by=uid],aes(y=rating,x=rate_punishment)) + geom_point() + coord_fixed() + geom_abline() + ylab("case strength (sd)") + xlab("punishability (sd)")
For any given subject, case strength ratings tend to be a more variable than punishability ratings.
Evidence manipulation check
Moving on, we can repeat our previous sanity check of looking at punishment ratings as a function of the total number of inclupatory versus exculpatory pieces of evidence in a case:
evidat <- dat[,.(n_exculp=str_detect(c(physical,document,witness,character),"ex") %>% sum(),
n_inculp=str_detect(c(physical,document,witness,character),"in") %>% sum(),
rating, period),by=.(uid,scenario)]
ggplot(evidat,aes(x=n_inculp-n_exculp,y=rating)) + geom_smooth(method="lm", se=F) + geom_point(position=position_jitter(width=0.1),alpha=0.25) + ylim(c(-1,101)) + ylab("case strength")
Lookin’ good. However, Pate brought up that we should check whether specific cases might be behaving badly due to confusing wording, etc, so we can look at this relationship on a case-by-case basis.
qplot(evidat[,cor(n_inculp-n_exculp,rating),by=scenario]$V1) + xlab("correlation") + ylab("number of scenarios")
Here we’re looking at the (distribution of) the correlation between the case strength rating and the exculpatory/inculpatory balance. Every case seems to have at least a modest relationship in the right direction. Though there appears to be some heterogeneity, it’s hard to tell how much of that heterogeneity is “real” versus how much is just the consequence of noise.
We can take a crack at that latter question by fitting a simple mixed-effects model, which will do it’s best to sensibly distribute variability across scenarios and response noise:
fit <- lmer(rating ~ 1 + n_exculp + n_inculp + (1|uid) + (0+n_exculp|uid) + (0+n_inculp|uid) +
(1|scenario) + (0+n_exculp|scenario) + (0+n_inculp|scenario),data=evidat, REML=F)
print(fit, digits=2)
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: rating ~ 1 + n_exculp + n_inculp + (1 | uid) + (0 + n_exculp |
uid) + (0 + n_inculp | uid) + (1 | scenario) + (0 + n_exculp | scenario) + (0 + n_inculp | scenario)
Data: evidat
AIC BIC logLik deviance df.resid
9769.310 9818.914 -4874.655 9749.310 1044
Random effects:
Groups Name Std.Dev.
uid (Intercept) 7.54398
uid.1 n_exculp 0.00034
uid.2 n_inculp 0.00084
scenario (Intercept) 6.51870
scenario.1 n_exculp 0.00129
scenario.2 n_inculp 1.83279
Residual 23.61145
Number of obs: 1054, groups: uid, 33; scenario, 32
Fixed Effects:
(Intercept) n_exculp n_inculp
33 -10 18
convergence code 0; 1 optimizer warnings; 0 lme4 warnings
The main takeaway here is that we see negligible variability across subjects and scenarios in the marginal effect of a piece of evidence. The one exception is inculpatory evidence across scenarios, but even the standard deviation is only about 10% of the size of the fixed effect for inculpatory evidence. The best interpretation of this is less that there is actually little to no variability across subjects or scenarios in the evidence effects, but rather that the model doesn’t see any convincing evidence for such variability. We do see more substantial variability across both subjects and scenarios in the baselines, which I believe is consistent with the previous findings.
Time on task and consequences thereof
It’s worth checking how much time participants are actually spending on the task to see if the projection we give in the mTurk listing is accurate.
timedat <- dat[,.(task_time=difftime(max(as.POSIXct(stop)),min(as.POSIXct(start)), units="mins")),by=uid]
timedat <- merge(timedat,subjdat[uid %in% dat$uid,.(uid, qualtrics_time=difftime(as.POSIXct(EndDate),as.POSIXct(start),units="mins"))])
timedat[,qualtrics_time:=qualtrics_time-task_time]
ggplot(melt(timedat),aes(x=reorder(uid,-value),y=value,fill=ordered(variable,levels=c("qualtrics_time","task_time")))) + geom_bar(stat = "identity") + scale_fill_discrete(NULL) + scale_x_discrete(labels=NULL) + ylab("time (mins)")
That’s quite a spread, with some participants going crazy fast. The mTurk listing currently says that the task takes around 30 minutes, which is quite a bit longer than most people actually take. Still, as we’ve had no difficulty recruiting subjects, I’d rather remain conservative with our advertised time than risk subjects getting impatient.
On a related note, given that our task is a bit on the long side by mTurk standards, I was worried that subjects might be getting less contemplitive in their responses towards the end. Accordingly I redid some of the above analyses across four different time bins.
ggplot(dat,aes(x=rating,fill=period)) + geom_histogram(bins=20) + xlab("case strength")
ggplot(evidat,aes(x=n_inculp-n_exculp,y=rating, color=period)) + geom_smooth(method="lm", se=F) + geom_point(position=position_jitter(width=0.1),alpha=0.3) + ylim(c(-1,101)) + ylab("case strength")
If there are any changes over the course of the task, they’re certainly subtle.
Somewhat surprisingly, we don’t even see much of a change in response times over the course of the task:
rtdat <- dat[,.(rt=c(0,difftime(as.POSIXct(stop), min(as.POSIXct(start)), units="secs")) %>% diff(),question),by=uid]
ggplot(rtdat,aes(y=rt,x=ordered(question+1))) + geom_boxplot() + coord_cartesian(ylim=c(125,0)) + ylab("response time") + xlab("question number") + geom_smooth(aes(y=rt,x=question+1),method="loess")
While participants do speed up over the course of the task, it’s again pretty subtle. Note that a) I am chopping off a number of much longer outliers which render the graph unreadable, and b) the questions which appear to be sticking up compared to their neighbors follow catch trials, and the time on the comprehension question isn’t being subtracted out.
Conclusion
Data quality continues to look excellent. Going in I was very concerned about the length of the task, but I see no indication that this is actually a problem. There are only two issues we may want to address.
First, the bunching around zero of the case strength ratings. While this is hardly catastrophic, data near the lower bound is intrinsically less informative. I suggest that instead of fully randomizing the evidence as we are doing now, instead each scenario should have at least one piece of evidence that is either inculpatory or ambiguous. I imagine that this would also be more ecologically valid.
Second, the inflated 100s for punishment. This is even less of a problem because frankly, I have no particular plans for that data anyway. However, we might want to revisit the wording again. I changed it from “life in prison” to “extremely severe punishment” so that people wouldn’t be trying to figure out how to weigh different kinds of punishment against each other – e.g. fines vs prison time, but as a consequence subjects might be internally normalizing the value of the slider to the severity of the crime. I’m open to suggestions on this matter.
LS0tCnRpdGxlOiAiUHJlc2VudGluZzogQW4gRXhjZXNzaXZlIFF1YW50aXR5IG9mIFBsb3RzIgpvdXRwdXQ6CiAgaHRtbF9ub3RlYm9vazogZGVmYXVsdAogIGh0bWxfZG9jdW1lbnQ6CiAgICBkZl9wcmludDogcGFnZWQKICBwZGZfZG9jdW1lbnQ6IGRlZmF1bHQKLS0tCgpgYGB7ciBzZXR1cCwgaW5jbHVkZT1GQUxTRX0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KG1lc3NhZ2UgPSBGQUxTRSwgd2FybmluZyA9IEZBTFNFLCB0aWR5ID0gVFJVRSwgZmlnLmhlaWdodD0zLCBmaWcud2lkdGg9NSkKbGlicmFyeShzdHJpbmdyKQpsaWJyYXJ5KGdyaWRFeHRyYSkKbGlicmFyeShsbWU0KQpvcHRpb25zKHNjaXBlbiA9IDk5OSkKc291cmNlKCIvaG9tZS9zZXRoL2NvZGUvY2FzZXJldmVhbC9hbmFseXNpcy9wcm9jZXNzX2RhdGEuUiIpCmBgYAoKUGFydGljdWxhcmx5IHJlZ2FyZGluZyB0aGUgcmVjZW50bHkgY29sbGVjdGVkIHBpbG90IGRhdGEsIGNvbnNpc3Rpbmcgb2YgMzMgbVR1cmsgc3ViamVjdHMuCgojIyBNYXJnaW5hbCBkaXN0cmlidXRpb25zIG9mIHJhdGluZ3MKCkZpcnN0IHRoaW5nLCBsZXQncyB0YWtlIGEgbG9vayBhdCB0aGUgb3ZlcmFsbCBkaXN0cmlidXRpb24gb2YgcmVzcG9uc2VzIHRvIHNlZSBpZiB0aGVyZSBhcmUgYW55IHJlZCBmbGFncy4KCmBgYHtyfQptc3ViaiA8LSBzdWJqZGF0WyFzdHJfZGV0ZWN0KHVpZCwidGVzdCIpICYgdGFza19jb21wbGV0ZSx1aWRdCnNjZW5kYXQgPC0gc2NlbmRhdFt1aWQgJWluJSBtc3Vial0KY2xpY2tkYXQgPC0gY2xpY2tkYXRbIWlzLm5hKGd1aWx0eSldCmRhdCA8LSBtZXJnZShzY2VuZGF0LGNsaWNrZGF0LGJ5PWMoInVpZCIsInNjZW5hcmlvIikpCmRhdFsscGVyaW9kOj1jdXRfbnVtYmVyKHF1ZXN0aW9uLDQpXQoKcXBsb3QoZGF0JHJhdGluZykgKyB4bGFiKCJjYXNlIHN0cmVuZ3RoIikKcXBsb3QoZGF0JHJhdGVfcHVuaXNobWVudCkgKyB4bGFiKCJwdW5pc2hhYmlsaXR5IikKYGBgCgpGb3IgY2FzZSBzdHJlbmd0aCB3ZSdyZSBnZXR0aW5nIGEgbG90IG9mIGNsdW1waW5nIG9uIHRoZSBsb3dlciBlbmQgb2YgdGhlIHNjYWxlLCBidXQgb3ZlcmFsbCBwZW9wbGUgYXJlIHVzaW5nIHRoZSBlbnRpcmUgc2NhbGUuCkludGVyZXN0aW5nIGhvdyBVLXNoYXBlZCBpdCBzZWVtcyB0byBiZSwgbWF5YmUgYmVjYXVzZSBpdCdzIGNsZWFyIHdoYXQgYSAwIHB0IGNhc2UgaXMgYW5kIHdoYXQgYSAxMDAgcHQgY2FzZSBpcyBidXQgaXQncyBub3Qgc28gb2J2aW91cyB3aGF0IGEgNTAgcHQgY2FzZSBsb29rcyBsaWtlLgpGb3IgcHVuaXNobWVudCwgb24gdGhlIG90aGVyIGhhbmQsIHdlIHNlZSBhIGxvdCBvZiBwZW9wbGUgbWF4aW5nIG91dCB0aGUgc2NhbGUuCgpPZiBjb3Vyc2Ugd2hhdCB3ZSByZWFsbHkgd2FudCB0byBzZWUgaXMgdGhhdCBpbmRpdmlkdWFscyBhcmUgdXNpbmcgdGhlIHdob2xlIHNjYWxlLCBub3QganVzdCB0aGUgcG9wdWxhdGlvbi4KVGhhdCdzIHRvdWdoZXIgdG8gdmlzdWFsaXplIHdpdGggc28gbWFueSBwYXJ0aWNpcGFudHMgYnV0IHdlIGNhbiBnaXZlIGl0IGEgZ28uCmBgYHtyfQpnZ3Bsb3QoZGF0LGFlcyh4PXJhdGluZyxjb2xvcj11aWQpKSArIHN0YXRfZWNkZihnZW9tPSJzdGVwIiwgcGFkPUZBTFNFKSArIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICJub25lIikgKyB4bGFiKCJjYXNlIHN0cmVuZ3RoIikgKyB5bGFiKCJwcm9wb3J0aW9uIikKZ2dwbG90KGRhdCxhZXMoeD1yYXRlX3B1bmlzaG1lbnQsY29sb3I9dWlkKSkgKyBzdGF0X2VjZGYoZ2VvbT0ic3RlcCIsIHBhZD1GQUxTRSkgKyB0aGVtZShsZWdlbmQucG9zaXRpb24gPSAibm9uZSIpICsgeGxhYigicHVuaXNoYWJpbGl0eSIpICsgeWxhYigicHJvcG9ydGlvbiIpCmBgYAoKSGVyZSB3ZSdyZSBsb29raW5nIGF0IHRoZSBlbXBpcmljYWwgY3VtbXVsYXRpdmUgZGlzdHJpYnV0aW9ucyBmb3IgZWFjaCBzdWJqZWN0IC0tIHNvLCBmb3IgZWFjaCB2YWx1ZSBvbiB0aGUgeC1heGlzLCB3aGF0IHByb3BvcnRpb24gb2YgdGhlaXIgcmF0aW5ncyB3ZXJlIGJlbG93IHRoYXQgdmFsdWUuCkl0IGFwcGVhcnMgdGhhdCB1c2Ugb2YgdGhlIGNhc2Ugc3RyZW5ndGggc2NhbGUgaXMgZmFpcmx5IGNvbnNpc3RlbnQgYWNyb3NzIHN1YmplY3RzLCBidXQgcHVuaXNobWVudCBzZWVtcyBtb3JlLi4uIGlkaW9zeW5jcmF0aWMsIHdpdGggbW9yZSBzdWJqZWN0cyBsZWFuaW5nIG1vcmUgaGVhdmlseSBvbiBvbmUgc2lkZSBvZiB0aGUgc2NhbGUgb3IgYW5vdGhlci4KCldlIGNhbiBhbHNvIHNlZSB0aGlzIGJ5IGp1c3QgcGxvdHRpbmcgdGhlIHN0YW5kYXJkIGRldmlhdGlvbnMgb2YgZWFjaCBwYXJ0aWNpcGFudCdzIHJhdGluZ3M6CgpgYGB7cn0KZ2dwbG90KGRhdFssLihyYXRpbmc9c2QocmF0aW5nKSxyYXRlX3B1bmlzaG1lbnQ9c2QocmF0ZV9wdW5pc2htZW50KSksYnk9dWlkXSxhZXMoeT1yYXRpbmcseD1yYXRlX3B1bmlzaG1lbnQpKSArIGdlb21fcG9pbnQoKSArIGNvb3JkX2ZpeGVkKCkgKyBnZW9tX2FibGluZSgpICsgeWxhYigiY2FzZSBzdHJlbmd0aCAoc2QpIikgKyB4bGFiKCJwdW5pc2hhYmlsaXR5IChzZCkiKQpgYGAKCkZvciBhbnkgZ2l2ZW4gc3ViamVjdCwgY2FzZSBzdHJlbmd0aCByYXRpbmdzIHRlbmQgdG8gYmUgYSBtb3JlIHZhcmlhYmxlIHRoYW4gcHVuaXNoYWJpbGl0eSByYXRpbmdzLgoKCiMjIEV2aWRlbmNlIG1hbmlwdWxhdGlvbiBjaGVjawoKTW92aW5nIG9uLCB3ZSBjYW4gcmVwZWF0IG91ciBwcmV2aW91cyBzYW5pdHkgY2hlY2sgb2YgbG9va2luZyBhdCBwdW5pc2htZW50IHJhdGluZ3MgYXMgYSBmdW5jdGlvbiBvZiB0aGUgdG90YWwgbnVtYmVyIG9mIGluY2x1cGF0b3J5IHZlcnN1cyBleGN1bHBhdG9yeSBwaWVjZXMgb2YgZXZpZGVuY2UgaW4gYSBjYXNlOgpgYGB7cn0KZXZpZGF0IDwtIGRhdFssLihuX2V4Y3VscD1zdHJfZGV0ZWN0KGMocGh5c2ljYWwsZG9jdW1lbnQsd2l0bmVzcyxjaGFyYWN0ZXIpLCJleCIpICU+JSBzdW0oKSwKICAgICAgIG5faW5jdWxwPXN0cl9kZXRlY3QoYyhwaHlzaWNhbCxkb2N1bWVudCx3aXRuZXNzLGNoYXJhY3RlciksImluIikgJT4lIHN1bSgpLAogICAgICAgcmF0aW5nLCBwZXJpb2QpLGJ5PS4odWlkLHNjZW5hcmlvKV0KCmdncGxvdChldmlkYXQsYWVzKHg9bl9pbmN1bHAtbl9leGN1bHAseT1yYXRpbmcpKSArIGdlb21fc21vb3RoKG1ldGhvZD0ibG0iLCBzZT1GKSArIGdlb21fcG9pbnQocG9zaXRpb249cG9zaXRpb25faml0dGVyKHdpZHRoPTAuMSksYWxwaGE9MC4yNSkgKyB5bGltKGMoLTEsMTAxKSkgKyB5bGFiKCJjYXNlIHN0cmVuZ3RoIikKYGBgCgpMb29raW4nIGdvb2QuCkhvd2V2ZXIsIFBhdGUgYnJvdWdodCB1cCB0aGF0IHdlIHNob3VsZCBjaGVjayB3aGV0aGVyIHNwZWNpZmljIGNhc2VzIG1pZ2h0IGJlIGJlaGF2aW5nIGJhZGx5IGR1ZSB0byBjb25mdXNpbmcgd29yZGluZywgZXRjLCBzbyB3ZSBjYW4gbG9vayBhdCB0aGlzIHJlbGF0aW9uc2hpcCBvbiBhIGNhc2UtYnktY2FzZSBiYXNpcy4KCmBgYHtyfQpxcGxvdChldmlkYXRbLGNvcihuX2luY3VscC1uX2V4Y3VscCxyYXRpbmcpLGJ5PXNjZW5hcmlvXSRWMSkgKyB4bGFiKCJjb3JyZWxhdGlvbiIpICsgeWxhYigibnVtYmVyIG9mIHNjZW5hcmlvcyIpCmBgYAoKSGVyZSB3ZSdyZSBsb29raW5nIGF0IHRoZSAoZGlzdHJpYnV0aW9uIG9mKSB0aGUgY29ycmVsYXRpb24gYmV0d2VlbiB0aGUgY2FzZSBzdHJlbmd0aCByYXRpbmcgYW5kIHRoZSBleGN1bHBhdG9yeS9pbmN1bHBhdG9yeSBiYWxhbmNlLgpFdmVyeSBjYXNlIHNlZW1zIHRvIGhhdmUgYXQgbGVhc3QgYSBtb2Rlc3QgcmVsYXRpb25zaGlwIGluIHRoZSByaWdodCBkaXJlY3Rpb24uClRob3VnaCB0aGVyZSBhcHBlYXJzIHRvIGJlIHNvbWUgaGV0ZXJvZ2VuZWl0eSwgaXQncyBoYXJkIHRvIHRlbGwgaG93IG11Y2ggb2YgdGhhdCBoZXRlcm9nZW5laXR5IGlzICJyZWFsIiB2ZXJzdXMgaG93IG11Y2ggaXMganVzdCB0aGUgY29uc2VxdWVuY2Ugb2Ygbm9pc2UuCgpXZSBjYW4gdGFrZSBhIGNyYWNrIGF0IHRoYXQgbGF0dGVyIHF1ZXN0aW9uIGJ5IGZpdHRpbmcgYSBzaW1wbGUgbWl4ZWQtZWZmZWN0cyBtb2RlbCwgd2hpY2ggd2lsbCBkbyBpdCdzIGJlc3QgdG8gc2Vuc2libHkgZGlzdHJpYnV0ZSB2YXJpYWJpbGl0eSBhY3Jvc3Mgc2NlbmFyaW9zIGFuZCByZXNwb25zZSBub2lzZToKYGBge3J9CmZpdCA8LSBsbWVyKHJhdGluZyB+IDEgKyBuX2V4Y3VscCArIG5faW5jdWxwICsgKDF8dWlkKSArICgwK25fZXhjdWxwfHVpZCkgKyAoMCtuX2luY3VscHx1aWQpICsgCiAgICAgICAoMXxzY2VuYXJpbykgKyAoMCtuX2V4Y3VscHxzY2VuYXJpbykgKyAoMCtuX2luY3VscHxzY2VuYXJpbyksZGF0YT1ldmlkYXQsIFJFTUw9RikKcHJpbnQoZml0LCBkaWdpdHM9MikKYGBgCgpUaGUgbWFpbiB0YWtlYXdheSBoZXJlIGlzIHRoYXQgd2Ugc2VlIG5lZ2xpZ2libGUgdmFyaWFiaWxpdHkgYWNyb3NzIHN1YmplY3RzIGFuZCBzY2VuYXJpb3MgaW4gdGhlIG1hcmdpbmFsIGVmZmVjdCBvZiBhIHBpZWNlIG9mIGV2aWRlbmNlLgpUaGUgb25lIGV4Y2VwdGlvbiBpcyBpbmN1bHBhdG9yeSBldmlkZW5jZSBhY3Jvc3Mgc2NlbmFyaW9zLCBidXQgZXZlbiB0aGUgc3RhbmRhcmQgZGV2aWF0aW9uIGlzIG9ubHkgYWJvdXQgMTAlIG9mIHRoZSBzaXplIG9mIHRoZSBmaXhlZCBlZmZlY3QgZm9yIGluY3VscGF0b3J5IGV2aWRlbmNlLgpUaGUgYmVzdCBpbnRlcnByZXRhdGlvbiBvZiB0aGlzIGlzIGxlc3MgdGhhdCB0aGVyZSBpcyBfYWN0dWFsbHlfIGxpdHRsZSB0byBubyB2YXJpYWJpbGl0eSBhY3Jvc3Mgc3ViamVjdHMgb3Igc2NlbmFyaW9zIGluIHRoZSBldmlkZW5jZSBlZmZlY3RzLCBidXQgcmF0aGVyIHRoYXQgdGhlIG1vZGVsIGRvZXNuJ3Qgc2VlIGFueSBjb252aW5jaW5nIGV2aWRlbmNlIGZvciBzdWNoIHZhcmlhYmlsaXR5LgpXZSBkbyBzZWUgbW9yZSBzdWJzdGFudGlhbCB2YXJpYWJpbGl0eSBhY3Jvc3MgYm90aCBzdWJqZWN0cyBhbmQgc2NlbmFyaW9zIGluIHRoZSBiYXNlbGluZXMsIHdoaWNoIEkgYmVsaWV2ZSBpcyBjb25zaXN0ZW50IHdpdGggdGhlIHByZXZpb3VzIGZpbmRpbmdzLgoKIyMgVGltZSBvbiB0YXNrIGFuZCBjb25zZXF1ZW5jZXMgdGhlcmVvZgoKSXQncyB3b3J0aCBjaGVja2luZyBob3cgbXVjaCB0aW1lIHBhcnRpY2lwYW50cyBhcmUgYWN0dWFsbHkgc3BlbmRpbmcgb24gdGhlIHRhc2sgdG8gc2VlIGlmIHRoZSBwcm9qZWN0aW9uIHdlIGdpdmUgaW4gdGhlIG1UdXJrIGxpc3RpbmcgaXMgYWNjdXJhdGUuCgpgYGB7ciwgZmlnLndpZHRoPTZ9CnRpbWVkYXQgPC0gZGF0WywuKHRhc2tfdGltZT1kaWZmdGltZShtYXgoYXMuUE9TSVhjdChzdG9wKSksbWluKGFzLlBPU0lYY3Qoc3RhcnQpKSwgdW5pdHM9Im1pbnMiKSksYnk9dWlkXQp0aW1lZGF0IDwtIG1lcmdlKHRpbWVkYXQsc3ViamRhdFt1aWQgJWluJSBkYXQkdWlkLC4odWlkLCBxdWFsdHJpY3NfdGltZT1kaWZmdGltZShhcy5QT1NJWGN0KEVuZERhdGUpLGFzLlBPU0lYY3Qoc3RhcnQpLHVuaXRzPSJtaW5zIikpXSkKdGltZWRhdFsscXVhbHRyaWNzX3RpbWU6PXF1YWx0cmljc190aW1lLXRhc2tfdGltZV0KZ2dwbG90KG1lbHQodGltZWRhdCksYWVzKHg9cmVvcmRlcih1aWQsLXZhbHVlKSx5PXZhbHVlLGZpbGw9b3JkZXJlZCh2YXJpYWJsZSxsZXZlbHM9YygicXVhbHRyaWNzX3RpbWUiLCJ0YXNrX3RpbWUiKSkpKSArIGdlb21fYmFyKHN0YXQgPSAiaWRlbnRpdHkiKSArIHNjYWxlX2ZpbGxfZGlzY3JldGUoTlVMTCkgKyBzY2FsZV94X2Rpc2NyZXRlKGxhYmVscz1OVUxMKSArIHlsYWIoInRpbWUgKG1pbnMpIikKCmBgYApUaGF0J3MgcXVpdGUgYSBzcHJlYWQsIHdpdGggc29tZSBwYXJ0aWNpcGFudHMgZ29pbmcgY3JhenkgZmFzdC4KVGhlIG1UdXJrIGxpc3RpbmcgY3VycmVudGx5IHNheXMgdGhhdCB0aGUgdGFzayB0YWtlcyBhcm91bmQgMzAgbWludXRlcywgd2hpY2ggaXMgcXVpdGUgYSBiaXQgbG9uZ2VyIHRoYW4gbW9zdCBwZW9wbGUgYWN0dWFsbHkgdGFrZS4KU3RpbGwsIGFzIHdlJ3ZlIGhhZCBubyBkaWZmaWN1bHR5IHJlY3J1aXRpbmcgc3ViamVjdHMsIEknZCByYXRoZXIgcmVtYWluIGNvbnNlcnZhdGl2ZSB3aXRoIG91ciBhZHZlcnRpc2VkIHRpbWUgdGhhbiByaXNrIHN1YmplY3RzIGdldHRpbmcgaW1wYXRpZW50LgoKT24gYSByZWxhdGVkIG5vdGUsIGdpdmVuIHRoYXQgb3VyIHRhc2sgaXMgYSBiaXQgb24gdGhlIGxvbmcgc2lkZSBieSBtVHVyayBzdGFuZGFyZHMsIEkgd2FzIHdvcnJpZWQgdGhhdCBzdWJqZWN0cyBtaWdodCBiZSBnZXR0aW5nIGxlc3MgY29udGVtcGxpdGl2ZSBpbiB0aGVpciByZXNwb25zZXMgdG93YXJkcyB0aGUgZW5kLgpBY2NvcmRpbmdseSBJIHJlZGlkIHNvbWUgb2YgdGhlIGFib3ZlIGFuYWx5c2VzIGFjcm9zcyBmb3VyIGRpZmZlcmVudCB0aW1lIGJpbnMuCgpgYGB7cn0KZ2dwbG90KGRhdCxhZXMoeD1yYXRpbmcsZmlsbD1wZXJpb2QpKSArIGdlb21faGlzdG9ncmFtKGJpbnM9MjApICsgeGxhYigiY2FzZSBzdHJlbmd0aCIpCmdncGxvdChldmlkYXQsYWVzKHg9bl9pbmN1bHAtbl9leGN1bHAseT1yYXRpbmcsIGNvbG9yPXBlcmlvZCkpICsgZ2VvbV9zbW9vdGgobWV0aG9kPSJsbSIsIHNlPUYpICsgZ2VvbV9wb2ludChwb3NpdGlvbj1wb3NpdGlvbl9qaXR0ZXIod2lkdGg9MC4xKSxhbHBoYT0wLjMpICsgeWxpbShjKC0xLDEwMSkpICsgeWxhYigiY2FzZSBzdHJlbmd0aCIpCmBgYAoKSWYgdGhlcmUgYXJlIGFueSBjaGFuZ2VzIG92ZXIgdGhlIGNvdXJzZSBvZiB0aGUgdGFzaywgdGhleSdyZSBjZXJ0YWlubHkgc3VidGxlLgoKU29tZXdoYXQgc3VycHJpc2luZ2x5LCB3ZSBkb24ndCBldmVuIHNlZSBtdWNoIG9mIGEgY2hhbmdlIGluIHJlc3BvbnNlIHRpbWVzIG92ZXIgdGhlIGNvdXJzZSBvZiB0aGUgdGFzazoKYGBge3IsIGZpZy53aWR0aD03fQpydGRhdCA8LSBkYXRbLC4ocnQ9YygwLGRpZmZ0aW1lKGFzLlBPU0lYY3Qoc3RvcCksIG1pbihhcy5QT1NJWGN0KHN0YXJ0KSksIHVuaXRzPSJzZWNzIikpICU+JSBkaWZmKCkscXVlc3Rpb24pLGJ5PXVpZF0KZ2dwbG90KHJ0ZGF0LGFlcyh5PXJ0LHg9b3JkZXJlZChxdWVzdGlvbisxKSkpICsgZ2VvbV9ib3hwbG90KCkgKyBjb29yZF9jYXJ0ZXNpYW4oeWxpbT1jKDEyNSwwKSkgKyB5bGFiKCJyZXNwb25zZSB0aW1lIikgKyB4bGFiKCJxdWVzdGlvbiBudW1iZXIiKSArIGdlb21fc21vb3RoKGFlcyh5PXJ0LHg9cXVlc3Rpb24rMSksbWV0aG9kPSJsb2VzcyIpCmBgYApXaGlsZSBwYXJ0aWNpcGFudHMgZG8gc3BlZWQgdXAgb3ZlciB0aGUgY291cnNlIG9mIHRoZSB0YXNrLCBpdCdzIGFnYWluIHByZXR0eSBzdWJ0bGUuCk5vdGUgdGhhdCBhKSBJIGFtIGNob3BwaW5nIG9mZiBhIG51bWJlciBvZiBtdWNoIGxvbmdlciBvdXRsaWVycyB3aGljaCByZW5kZXIgdGhlIGdyYXBoIHVucmVhZGFibGUsIGFuZCBiKSB0aGUgcXVlc3Rpb25zIHdoaWNoIGFwcGVhciB0byBiZSBzdGlja2luZyB1cCBjb21wYXJlZCB0byB0aGVpciBuZWlnaGJvcnMgZm9sbG93IGNhdGNoIHRyaWFscywgYW5kIHRoZSB0aW1lIG9uIHRoZSBjb21wcmVoZW5zaW9uIHF1ZXN0aW9uIGlzbid0IGJlaW5nIHN1YnRyYWN0ZWQgb3V0LgoKIyMgQ29uY2x1c2lvbgoKRGF0YSBxdWFsaXR5IGNvbnRpbnVlcyB0byBsb29rIGV4Y2VsbGVudC4KR29pbmcgaW4gSSB3YXMgdmVyeSBjb25jZXJuZWQgYWJvdXQgdGhlIGxlbmd0aCBvZiB0aGUgdGFzaywgYnV0IEkgc2VlIG5vIGluZGljYXRpb24gdGhhdCB0aGlzIGlzIGFjdHVhbGx5IGEgcHJvYmxlbS4KVGhlcmUgYXJlIG9ubHkgdHdvIGlzc3VlcyB3ZSBtYXkgd2FudCB0byBhZGRyZXNzLgoKRmlyc3QsIHRoZSBidW5jaGluZyBhcm91bmQgemVybyBvZiB0aGUgY2FzZSBzdHJlbmd0aCByYXRpbmdzLgpXaGlsZSB0aGlzIGlzIGhhcmRseSBjYXRhc3Ryb3BoaWMsIGRhdGEgbmVhciB0aGUgbG93ZXIgYm91bmQgaXMgaW50cmluc2ljYWxseSBsZXNzIGluZm9ybWF0aXZlLgpJIHN1Z2dlc3QgdGhhdCBpbnN0ZWFkIG9mIGZ1bGx5IHJhbmRvbWl6aW5nIHRoZSBldmlkZW5jZSBhcyB3ZSBhcmUgZG9pbmcgbm93LCBpbnN0ZWFkIGVhY2ggc2NlbmFyaW8gc2hvdWxkIGhhdmUgYXQgbGVhc3Qgb25lIHBpZWNlIG9mIGV2aWRlbmNlIHRoYXQgaXMgZWl0aGVyIGluY3VscGF0b3J5IG9yIGFtYmlndW91cy4KSSBpbWFnaW5lIHRoYXQgdGhpcyB3b3VsZCBhbHNvIGJlIG1vcmUgZWNvbG9naWNhbGx5IHZhbGlkLgoKU2Vjb25kLCB0aGUgaW5mbGF0ZWQgMTAwcyBmb3IgcHVuaXNobWVudC4KVGhpcyBpcyBldmVuIGxlc3Mgb2YgYSBwcm9ibGVtIGJlY2F1c2UgZnJhbmtseSwgSSBoYXZlIG5vIHBhcnRpY3VsYXIgcGxhbnMgZm9yIHRoYXQgZGF0YSBhbnl3YXkuCkhvd2V2ZXIsIHdlIG1pZ2h0IHdhbnQgdG8gcmV2aXNpdCB0aGUgd29yZGluZyBhZ2Fpbi4KSSBjaGFuZ2VkIGl0IGZyb20gImxpZmUgaW4gcHJpc29uIiB0byAiZXh0cmVtZWx5IHNldmVyZSBwdW5pc2htZW50IiBzbyB0aGF0IHBlb3BsZSB3b3VsZG4ndCBiZSB0cnlpbmcgdG8gZmlndXJlIG91dCBob3cgdG8gd2VpZ2ggZGlmZmVyZW50IGtpbmRzIG9mIHB1bmlzaG1lbnQgYWdhaW5zdCBlYWNoIG90aGVyIC0tIGUuZy4gZmluZXMgdnMgcHJpc29uIHRpbWUsIGJ1dCBhcyBhIGNvbnNlcXVlbmNlIHN1YmplY3RzIG1pZ2h0IGJlIGludGVybmFsbHkgbm9ybWFsaXppbmcgdGhlIHZhbHVlIG9mIHRoZSBzbGlkZXIgdG8gdGhlIHNldmVyaXR5IG9mIHRoZSBjcmltZS4KSSdtIG9wZW4gdG8gc3VnZ2VzdGlvbnMgb24gdGhpcyBtYXR0ZXIu