“Block what you can, and randomize what you cannot”
The ideas and examples from this article primarily come from Chapter 3 and 4 of Gerber and Green’s Field Experiment Design, Interpretation, & Analysis book.
Blocking
Blocking is a special type of randomization technique where participants are randomized from their “subgroups”, called blocks, and then completely randomized within the block.
An easy example to grasp the concept is imagine an experiment is imagining an experiment with say we had 20 participants with 10 men and 10 women, and we needed a treatment and control with 10 people each.
Using complete random assignment, there are 20 C 10 combinations. It is likely that the groups will have an unequal number of men and women. It is even possible with bad enough luck, it is possible to draw a group with all 10 women in the same group. Random is random, right? Block randomization ensures that an equal number of men are women will be assigned to the treatment and control condition.
How is blocking different than just controlling for a variable?
Actually, the two ideas are pretty similar. Blocking is in essence, “manually” controlling for a variable. This is because is because blocking eliminates the correlation between the assigned treatment and the variables used to form the blocks. In other words, this is essentially like in regression adjustment without using the additional control(s). Therefore, blocking is particularly advantageous in small samples. When there are over 100 participants in each condition, the advantages of blocking are reduced.
Blocking is most useful when there are less than 100 participants in each condition
Gerber & Green lay out two main advantages of blocking: * Blocking helps reduce sampling variability * Blocking ensures that certain subgroups are available for analysis
What are the best variables to block on?
Blocking produces the greatest gains in precision when the variables used to form blocks strongly predict outcomes. This is because by blocking (“controlling”) on removing that predict the outcomes w cause reduces variability.
Is it ever bad to block?
Basically no, in practice blocking is almost never going to hurt your experiment unless you analyze your data wrong. Here’s an example from the DeclareDesign blog where they construct a perverse example intentionally to showcase when blocking can reduce precision.
How do I block in DeclareDesign?
It is easy to block using DeclareDesign, in the using the declare_assignment function. Using the “men and women” example from above:
library(tidyverse)
library(DeclareDesign)
library(DT)
library(knitr)
library(ggrepel)
library(ggpubr)
You’ll notice that this argument doesn’t show-up when using ?declare_assignment. This is because declare_assignment uses the conduct_ra() function as it’s handler, so in order to see all the blocking arguments we’ll need to use ?conduct_ra to see the correct documentation.
Other important arguments in regard to blocking for two-arm designs are:
block_m The number of subjects assigned to the treatment within each blockblock_prob The probability that participants in each block get assigned to the treatment (for when probability of assingment differs in each block unlike the above example).
If had more than two arms and wanted to differ the probability of assignments in each block, we could use the block_m_each or block_prob_each arguments instead.
How do I block on multiple variables in DeclareDesign?
Neil Fultz offers an easy solution using the base R interaction function. Here I wish to block on gender and eye_color. The interaction function basically “pastes” the two columns together.
block_gender_and_eyes <-
declare_population(N = 100,
gender = draw_categorical(N = N, prob = c(.5, .5), category_labels = c("Male", "Female")),
eye_color = sample(c("blue", "green", "brown"),
N,
replace=TRUE,
prob = c(.2, .2, .6)),
blocks = interaction(gender, eye_color))() %>% # uses interaction to combiune gender and eye_color
declare_assignment(blocks = blocks,
prob = .5)()
block_gender_and_eyes %>%
datatable()
From this example we can see that our groups are split almost perfectly on the basis on gender and eye color.
block_gender_and_eyes %>%
group_by(Z) %>%
count(blocks) %>%
pivot_wider(id_cols = blocks, names_from = Z, values_from = n) %>%
rename("Treatment" = `1`,
"Control" = `0`) %>%
kable()
| Female.blue |
3 |
2 |
| Male.blue |
5 |
5 |
| Female.brown |
16 |
16 |
| Male.brown |
15 |
15 |
| Female.green |
5 |
6 |
| Male.green |
6 |
6 |
How do you evaluate blocked designs? (blocked estimators)
Lastly, the most important question, how do we evaluate blocked designs? This is most important when blocks on have different probability of being assigned to the treatment. The DeclareDesign team has a really cool blog post about this where they try out different methods. Here I recreate a simplified version of their post focusing instead of even and uneven probability of assignment of assignment.
# Model ------------------------------------------------------------------------
U <- declare_population(block = add_level(N = 3,
tau = c(4, 2, 0)),
indiv = add_level(N = 100, e = rnorm(N))) # each of the 3 blocks has 100 people
Y <- declare_potential_outcomes(Y_Z_0 = e,
Y_Z_1 = e + tau)
# Inquiry ----------------------------------------------------------------------
Q <- declare_estimand(ATE = mean(Y_Z_1 - Y_Z_0))
# Data Strategy ----------------------------------------------------------------
Z <- declare_assignment(blocks = block, block_prob = c(.5, .7, .9)) # probability of assignment
R <- declare_reveal(Y, Z)
Warning: `quo_expr()` is deprecated as of rlang 0.2.0.
Please use `quo_squash()` instead.
This warning is displayed once per session.
Warning: Assigning non-quosure objects to quosure lists is deprecated as of rlang 0.3.0.
Please coerce to a bare list beforehand with `as.list()`
This warning is displayed once per session.
# Answer Strategy --------------------------------------------------------------
A0 <- declare_estimator(Y ~ Z, estimand = Q,
model = lm_robust, label = "A0: (No Controls)")
A1 <- declare_estimator(Y ~ Z + block, estimand = Q,
model = lm_robust, label = "A1: (Fixed Effects)")
A2 <- declare_estimator(Y ~ Z, blocks = block, estimand = Q,
model = difference_in_means, label = "A2: (Blocked DIM)")
# Design -----------------------------------------------------------------------
design <- U + Y + Z + Q + R + A0 + A1 + A2
design2 <- replace_step(design = design,
step = Z, declare_assignment(blocks = block, block_prob = c(.5, .5, .5))) # new design, with same chance of assingment
simulations <- simulate_design(design, design2, sims = 500)
Warning: `lang_modify()` is deprecated as of rlang 0.2.0.
Please use `call_modify()` instead.
This warning is displayed once per session.
theme_set(theme_pubr())
simulations %>%
mutate(design_label = case_when(design_label == "design" ~ "Unequal assignment probability blocks",
design_label == "design2" ~ "Equal assignment probability blocks")) %>%
group_by(design_label, estimator_label) %>%
summarize(SE_bias = mean(std.error - sd(estimate)),
ATE_bias = mean(estimate - estimand)) %>%
ggplot(aes(x = ATE_bias, y = SE_bias, color = design_label, label = estimator_label)) +
geom_point() +
geom_hline(yintercept = 0, size = .1, linetype = "dashed") +
geom_vline(xintercept = 0, size = .1, linetype = "dashed") +
geom_text_repel(show.legend = FALSE,
box.padding = .4,
point.padding = .65,
segment.alpha = .5) +
scale_x_continuous(limits = c(-.6, .6)) +
scale_y_continuous(limits = c(-.1, .1)) +
theme(legend.title = element_blank()) +
labs(x = "Average Treatment Effect (ATE) Bias",
y = "Standard Error Bias",
title = "Bias in Even and Uneven Probability of Assignment")
The key takeaway here is the old fashioned weighted-average difference-in-means in the best for both even and uneven block sizes. When the blocks have an equal assignment probability there is a slight overestimation of the standard error, but it is very small. When there is unequal assignment probability, although the fixed effect approach is overestimating the ATE, this kind of method has the advantage that it would allow additional covariates.
It is also worth noting that there appear to be better methods of evaluating the ATEs as shown in the blog post, but I do not delve into them here.
LS0tCnRpdGxlOiAiU2ltdWxhdGluZyBSZXNlYXJjaCBEZXNpZ25zIGZvciBTb2NpYWwgU2NpZW5jZSBwdCAzIgpzdWJ0aXRsZTogIkJsb2NrcyBpbiBFeHBlcmltZW50YWwgRGVzaWduIgphdXRob3I6ICJKb2huLUhlbnJ5IFBlenp1dG8iCm91dHB1dDogCiAgcm1hcmtkb3duOjpodG1sX25vdGVib29rOgogICAgc21hcnQ6IGZhbHNlCiAgICB0aGVtZTogc3BhY2VsYWIKICAgIG51bWJlcl9zZWN0aW9uczogbm8KICAgIHRvYzogdHJ1ZQogICAgdG9jX2Zsb2F0OgogICAgICBjb2xsYXBzZWQ6IGZhbHNlCiAgICB0b2NfZGVwdGg6IDQKLS0tCgpfIkJsb2NrIHdoYXQgeW91IGNhbiwgYW5kIHJhbmRvbWl6ZSB3aGF0IHlvdSBjYW5ub3QiXyAKClRoZSBpZGVhcyBhbmQgZXhhbXBsZXMgZnJvbSB0aGlzIGFydGljbGUgcHJpbWFyaWx5IGNvbWUgZnJvbSBDaGFwdGVyIDMgYW5kIDQgb2YgR2VyYmVyIGFuZCBHcmVlbidzIEZpZWxkIEV4cGVyaW1lbnQgRGVzaWduLCBJbnRlcnByZXRhdGlvbiwgJiBBbmFseXNpcyBib29rLgoKIyMgQmxvY2tpbmcKCgpCbG9ja2luZyBpcyBhIHNwZWNpYWwgdHlwZSBvZiByYW5kb21pemF0aW9uIHRlY2huaXF1ZSB3aGVyZSBwYXJ0aWNpcGFudHMgYXJlIHJhbmRvbWl6ZWQgZnJvbSB0aGVpciAic3ViZ3JvdXBzIiwgY2FsbGVkIF9ibG9ja3NfLCBhbmQgdGhlbiBjb21wbGV0ZWx5IHJhbmRvbWl6ZWQgd2l0aGluIHRoZSBibG9jay4KCkFuIGVhc3kgZXhhbXBsZSB0byBncmFzcCB0aGUgY29uY2VwdCBpcyBpbWFnaW5lIGFuIGV4cGVyaW1lbnQgaXMgaW1hZ2luaW5nIGFuIGV4cGVyaW1lbnQgd2l0aCBzYXkgd2UgaGFkIDIwIHBhcnRpY2lwYW50cyB3aXRoIDEwIG1lbiBhbmQgMTAgd29tZW4sIGFuZCB3ZSBuZWVkZWQgYSB0cmVhdG1lbnQgYW5kIGNvbnRyb2wgd2l0aCAxMCBwZW9wbGUgZWFjaC4gCgpVc2luZyBjb21wbGV0ZSByYW5kb20gYXNzaWdubWVudCwgdGhlcmUgYXJlIDIwIEMgMTAgY29tYmluYXRpb25zLiBJdCBpcyBsaWtlbHkgdGhhdCB0aGUgZ3JvdXBzIHdpbGwgaGF2ZSBhbiB1bmVxdWFsIG51bWJlciBvZiBtZW4gYW5kIHdvbWVuLiBJdCBpcyBldmVuIHBvc3NpYmxlIHdpdGggYmFkIGVub3VnaCBsdWNrLCBpdCBpcyBwb3NzaWJsZSB0byBkcmF3IGEgZ3JvdXAgd2l0aCBhbGwgMTAgd29tZW4gaW4gdGhlIHNhbWUgZ3JvdXAuIFJhbmRvbSBpcyByYW5kb20sIHJpZ2h0PyBCbG9jayByYW5kb21pemF0aW9uIGVuc3VyZXMgdGhhdCBhbiBlcXVhbCBudW1iZXIgb2YgbWVuIGFyZSB3b21lbiB3aWxsIGJlIGFzc2lnbmVkIHRvIHRoZSB0cmVhdG1lbnQgYW5kIGNvbnRyb2wgY29uZGl0aW9uLgoKCiMjIEhvdyBpcyBibG9ja2luZyBkaWZmZXJlbnQgdGhhbiBqdXN0IGNvbnRyb2xsaW5nIGZvciBhIHZhcmlhYmxlPwoKQWN0dWFsbHksIHRoZSB0d28gaWRlYXMgYXJlIHByZXR0eSBzaW1pbGFyLiBCbG9ja2luZyBpcyBpbiBlc3NlbmNlLCAibWFudWFsbHkiIGNvbnRyb2xsaW5nIGZvciBhIHZhcmlhYmxlLiBUaGlzIGlzIGJlY2F1c2UgaXMgYmVjYXVzZSBibG9ja2luZyBlbGltaW5hdGVzIHRoZSBjb3JyZWxhdGlvbiBiZXR3ZWVuIHRoZSBhc3NpZ25lZCB0cmVhdG1lbnQgYW5kIHRoZSB2YXJpYWJsZXMgdXNlZCB0byBmb3JtIHRoZSBibG9ja3MuIEluIG90aGVyIHdvcmRzLCB0aGlzIGlzIGVzc2VudGlhbGx5IGxpa2UgaW4gcmVncmVzc2lvbiBhZGp1c3RtZW50IHdpdGhvdXQgdXNpbmcgdGhlIGFkZGl0aW9uYWwgY29udHJvbChzKS4gVGhlcmVmb3JlLCBibG9ja2luZyBpcyBwYXJ0aWN1bGFybHkgYWR2YW50YWdlb3VzIGluIHNtYWxsIHNhbXBsZXMuIFdoZW4gdGhlcmUgYXJlIG92ZXIgMTAwIHBhcnRpY2lwYW50cyBpbiBlYWNoIGNvbmRpdGlvbiwgdGhlIGFkdmFudGFnZXMgb2YgYmxvY2tpbmcgYXJlIHJlZHVjZWQuIAoKX19fQmxvY2tpbmcgaXMgbW9zdCB1c2VmdWwgd2hlbiB0aGVyZSBhcmUgbGVzcyB0aGFuIDEwMCBwYXJ0aWNpcGFudHMgaW4gZWFjaCBjb25kaXRpb25fX18KCgpHZXJiZXIgJiBHcmVlbiBsYXkgb3V0IHR3byBtYWluIGFkdmFudGFnZXMgb2YgYmxvY2tpbmc6CiogQmxvY2tpbmcgaGVscHMgcmVkdWNlIHNhbXBsaW5nIHZhcmlhYmlsaXR5CiogQmxvY2tpbmcgZW5zdXJlcyB0aGF0IGNlcnRhaW4gc3ViZ3JvdXBzIGFyZSBhdmFpbGFibGUgZm9yIGFuYWx5c2lzCgoKIyMgV2hhdCBhcmUgdGhlIGJlc3QgdmFyaWFibGVzIHRvIGJsb2NrIG9uPwoKX19fQmxvY2tpbmcgcHJvZHVjZXMgdGhlIGdyZWF0ZXN0IGdhaW5zIGluIHByZWNpc2lvbiB3aGVuIHRoZSB2YXJpYWJsZXMgdXNlZCB0byBmb3JtIGJsb2NrcyBzdHJvbmdseSBwcmVkaWN0IG91dGNvbWVzX19fLiBUaGlzIGlzIGJlY2F1c2UgYnkgYmxvY2tpbmcgKCJjb250cm9sbGluZyIpIG9uIHJlbW92aW5nIHRoYXQgcHJlZGljdCB0aGUgb3V0Y29tZXMgdyBjYXVzZSByZWR1Y2VzIHZhcmlhYmlsaXR5LgoKIyMgSXMgaXQgZXZlciBiYWQgdG8gYmxvY2s/CgpCYXNpY2FsbHkgbm8sIGluIHByYWN0aWNlIGJsb2NraW5nIGlzIGFsbW9zdCBuZXZlciBnb2luZyB0byBodXJ0IHlvdXIgZXhwZXJpbWVudCBfdW5sZXNzIHlvdSBhbmFseXplIHlvdXIgZGF0YSB3cm9uZ18uIFtIZXJlJ3MgYW4gZXhhbXBsZSBmcm9tIHRoZSBEZWNsYXJlRGVzaWduIGJsb2ddKGh0dHBzOi8vZGVjbGFyZWRlc2lnbi5vcmcvYmxvZy8yMDE4LTA5LTI0LWJhZF9ibG9ja3NfYmFkLmh0bWwpIHdoZXJlIHRoZXkgY29uc3RydWN0IGEgcGVydmVyc2UgZXhhbXBsZSBpbnRlbnRpb25hbGx5IHRvIHNob3djYXNlIHdoZW4gYmxvY2tpbmcgY2FuIHJlZHVjZSBwcmVjaXNpb24uCgoKIyMgSG93IGRvIEkgYmxvY2sgaW4gRGVjbGFyZURlc2lnbj8KSXQgaXMgZWFzeSB0byBibG9jayB1c2luZyBEZWNsYXJlRGVzaWduLCBpbiB0aGUgdXNpbmcgdGhlIGBkZWNsYXJlX2Fzc2lnbm1lbnRgIGZ1bmN0aW9uLiBVc2luZyB0aGUgIm1lbiBhbmQgd29tZW4iIGV4YW1wbGUgZnJvbSBhYm92ZToKYGBge3IsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9CmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KERlY2xhcmVEZXNpZ24pCmxpYnJhcnkoRFQpCmxpYnJhcnkoa25pdHIpCmxpYnJhcnkoZ2dyZXBlbCkKbGlicmFyeShnZ3B1YnIpCmBgYAoKCllvdSdsbCBub3RpY2UgdGhhdCB0aGlzIGFyZ3VtZW50IGRvZXNuJ3Qgc2hvdy11cCB3aGVuIHVzaW5nIGA/ZGVjbGFyZV9hc3NpZ25tZW50YC4gVGhpcyBpcyBiZWNhdXNlIGBkZWNsYXJlX2Fzc2lnbm1lbnRgIHVzZXMgdGhlIGBjb25kdWN0X3JhKClgIGZ1bmN0aW9uIGFzIGl0J3MgaGFuZGxlciwgc28gaW4gb3JkZXIgdG8gc2VlIGFsbCB0aGUgYmxvY2tpbmcgYXJndW1lbnRzIHdlJ2xsIG5lZWQgdG8gdXNlIGA/Y29uZHVjdF9yYWAgdG8gc2VlIHRoZSBjb3JyZWN0IGRvY3VtZW50YXRpb24uCgpPdGhlciBpbXBvcnRhbnQgYXJndW1lbnRzIGluIHJlZ2FyZCB0byBibG9ja2luZyBmb3IgdHdvLWFybSBkZXNpZ25zIGFyZToKCiogYGJsb2NrX21gIFRoZSBudW1iZXIgb2Ygc3ViamVjdHMgYXNzaWduZWQgdG8gdGhlIHRyZWF0bWVudCB3aXRoaW4gZWFjaCBibG9jawoqIGBibG9ja19wcm9iYCBUaGUgcHJvYmFiaWxpdHkgdGhhdCBwYXJ0aWNpcGFudHMgaW4gZWFjaCBibG9jayBnZXQgYXNzaWduZWQgdG8gdGhlIHRyZWF0bWVudCAoZm9yIHdoZW4gcHJvYmFiaWxpdHkgb2YgYXNzaW5nbWVudCBkaWZmZXJzIGluIGVhY2ggYmxvY2sgdW5saWtlIHRoZSBhYm92ZSBleGFtcGxlKS4KCklmIGhhZCBtb3JlIHRoYW4gdHdvIGFybXMgYW5kIHdhbnRlZCB0byBkaWZmZXIgdGhlIHByb2JhYmlsaXR5IG9mIGFzc2lnbm1lbnRzIGluIGVhY2ggYmxvY2ssIHdlIGNvdWxkIHVzZSB0aGUgYGJsb2NrX21fZWFjaGAgb3IgYGJsb2NrX3Byb2JfZWFjaGAgYXJndW1lbnRzIGluc3RlYWQuCgojIyBIb3cgZG8gSSBibG9jayBvbiBtdWx0aXBsZSB2YXJpYWJsZXMgaW4gRGVjbGFyZURlc2lnbj8KCltOZWlsIEZ1bHR6IG9mZmVycyBhbiBlYXN5IHNvbHV0aW9uXShodHRwOi8vZGlzY3Vzcy5kZWNsYXJlZGVzaWduLm9yZy90L2RlY2xhcmVkZXNpZ24tZHBseXItYmxvY2tpbmctb24tbXVsdGlwbGUtdmFycy1pbi1hbi1lbGVnYW50LXdheS85MCkgdXNpbmcgdGhlIGJhc2UgUiBgaW50ZXJhY3Rpb25gIGZ1bmN0aW9uLiBIZXJlIEkgd2lzaCB0byBibG9jayBvbiBgZ2VuZGVyYCBhbmQgYGV5ZV9jb2xvcmAuIFRoZSBgaW50ZXJhY3Rpb25gIGZ1bmN0aW9uIGJhc2ljYWxseSAicGFzdGVzIiB0aGUgdHdvIGNvbHVtbnMgdG9nZXRoZXIuCgpgYGB7ciwgd2FybmluZz1GQUxTRX0KYmxvY2tfZ2VuZGVyX2FuZF9leWVzIDwtIAogIGRlY2xhcmVfcG9wdWxhdGlvbihOID0gMTAwLAogICAgICAgICAgICAgICAgICAgZ2VuZGVyID0gZHJhd19jYXRlZ29yaWNhbChOID0gTiwgcHJvYiA9IGMoLjUsIC41KSwgY2F0ZWdvcnlfbGFiZWxzID0gYygiTWFsZSIsICJGZW1hbGUiKSksCiAgICAgICAgICAgICAgICAgICBleWVfY29sb3IgPSBzYW1wbGUoYygiYmx1ZSIsICJncmVlbiIsICJicm93biIpLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBOLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICByZXBsYWNlPVRSVUUsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJvYiA9IGMoLjIsIC4yLCAuNikpLAogICAgICAgICAgICAgICAgICAgYmxvY2tzID0gaW50ZXJhY3Rpb24oZ2VuZGVyLCBleWVfY29sb3IpKSgpICU+JSAjIHVzZXMgaW50ZXJhY3Rpb24gdG8gY29tYml1bmUgZ2VuZGVyIGFuZCBleWVfY29sb3IKICBkZWNsYXJlX2Fzc2lnbm1lbnQoYmxvY2tzID0gYmxvY2tzLCAKICAgICAgICAgICAgICAgICAgICAgcHJvYiA9IC41KSgpCgoKYmxvY2tfZ2VuZGVyX2FuZF9leWVzICU+JSAKICBkYXRhdGFibGUoKQpgYGAKCkZyb20gdGhpcyBleGFtcGxlIHdlIGNhbiBzZWUgdGhhdCBvdXIgZ3JvdXBzIGFyZSBzcGxpdCBhbG1vc3QgcGVyZmVjdGx5IG9uIHRoZSBiYXNpcyBvbiBnZW5kZXIgYW5kIGV5ZSBjb2xvci4KYGBge3J9CmJsb2NrX2dlbmRlcl9hbmRfZXllcyAlPiUgCiAgZ3JvdXBfYnkoWikgJT4lIAogIGNvdW50KGJsb2NrcykgJT4lIAogIHBpdm90X3dpZGVyKGlkX2NvbHMgPSBibG9ja3MsIG5hbWVzX2Zyb20gPSBaLCB2YWx1ZXNfZnJvbSA9IG4pICU+JSAKICByZW5hbWUoIlRyZWF0bWVudCIgPSBgMWAsCiAgICAgICAgICJDb250cm9sIiA9IGAwYCkgJT4lIAogIGthYmxlKCkKYGBgCgoKIyMgSG93IGRvIHlvdSBldmFsdWF0ZSBibG9ja2VkIGRlc2lnbnM/IChibG9ja2VkIGVzdGltYXRvcnMpCgpMYXN0bHksIHRoZSBtb3N0IGltcG9ydGFudCBxdWVzdGlvbiwgaG93IGRvIHdlIGV2YWx1YXRlIGJsb2NrZWQgZGVzaWducz8gVGhpcyBpcyBtb3N0IGltcG9ydGFudCB3aGVuIGJsb2NrcyBvbiBoYXZlIGRpZmZlcmVudCBwcm9iYWJpbGl0eSBvZiBiZWluZyBhc3NpZ25lZCB0byB0aGUgdHJlYXRtZW50LiBUaGUgRGVjbGFyZURlc2lnbiB0ZWFtIGhhcyBhIFtyZWFsbHkgY29vbCBibG9nIHBvc3QgYWJvdXQgdGhpc10oaHR0cHM6Ly9kZWNsYXJlZGVzaWduLm9yZy9ibG9nL2JpYXNlZC1maXhlZC1lZmZlY3RzLmh0bWwpIHdoZXJlIHRoZXkgdHJ5IG91dCBkaWZmZXJlbnQgbWV0aG9kcy4gSGVyZSBJIHJlY3JlYXRlIGEgc2ltcGxpZmllZCB2ZXJzaW9uIG9mIHRoZWlyIHBvc3QgZm9jdXNpbmcgaW5zdGVhZCBvZiBldmVuIGFuZCB1bmV2ZW4gcHJvYmFiaWxpdHkgb2YgYXNzaWdubWVudCBvZiBhc3NpZ25tZW50LgoKCgpgYGB7cn0KIyBNb2RlbCAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KVSA8LSBkZWNsYXJlX3BvcHVsYXRpb24oYmxvY2sgPSBhZGRfbGV2ZWwoTiA9IDMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRhdSA9IGMoNCwgMiwgMCkpLAogICAgICAgICAgICAgICAgICAgICAgICBpbmRpdiA9IGFkZF9sZXZlbChOID0gMTAwLCBlID0gcm5vcm0oTikpKSAjIGVhY2ggb2YgdGhlIDMgYmxvY2tzIGhhcyAxMDAgcGVvcGxlCgpZIDwtIGRlY2xhcmVfcG90ZW50aWFsX291dGNvbWVzKFlfWl8wID0gZSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBZX1pfMSA9IGUgKyB0YXUpCgojIElucXVpcnkgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpRIDwtIGRlY2xhcmVfZXN0aW1hbmQoQVRFID0gbWVhbihZX1pfMSAtIFlfWl8wKSkKCiMgRGF0YSBTdHJhdGVneSAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tClogPC0gZGVjbGFyZV9hc3NpZ25tZW50KGJsb2NrcyA9IGJsb2NrLCBibG9ja19wcm9iID0gYyguNSwgLjcsIC45KSkgIyBwcm9iYWJpbGl0eSBvZiBhc3NpZ25tZW50ClIgPC0gZGVjbGFyZV9yZXZlYWwoWSwgWikKCiMgQW5zd2VyIFN0cmF0ZWd5IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCkEwIDwtIGRlY2xhcmVfZXN0aW1hdG9yKFkgfiBaLCBlc3RpbWFuZCA9IFEsICAKICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWwgPSAgbG1fcm9idXN0LCBsYWJlbCA9ICJBMDogKE5vIENvbnRyb2xzKSIpCgpBMSA8LSBkZWNsYXJlX2VzdGltYXRvcihZIH4gWiArIGJsb2NrLCBlc3RpbWFuZCA9IFEsICAKICAgICAgICAgICAgICAgICAgICAgICAgbW9kZWwgPSAgbG1fcm9idXN0LCBsYWJlbCA9ICJBMTogKEZpeGVkIEVmZmVjdHMpIikKCkEyIDwtIGRlY2xhcmVfZXN0aW1hdG9yKFkgfiBaLCBibG9ja3MgPSBibG9jaywgZXN0aW1hbmQgPSBRLCAgCiAgICAgICAgICAgICAgICAgICAgICAgIG1vZGVsID0gIGRpZmZlcmVuY2VfaW5fbWVhbnMsIGxhYmVsID0gIkEyOiAoQmxvY2tlZCBESU0pIikKCiMgRGVzaWduIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCmRlc2lnbiA8LSBVICsgWSArIFogKyBRICsgUiArIEEwICsgQTEgKyBBMgoKZGVzaWduMiA8LSByZXBsYWNlX3N0ZXAoZGVzaWduID0gZGVzaWduLCAKICAgICAgICAgICAgICAgICAgICAgICAgc3RlcCA9IFosIGRlY2xhcmVfYXNzaWdubWVudChibG9ja3MgPSBibG9jaywgYmxvY2tfcHJvYiA9IGMoLjUsIC41LCAuNSkpKSAjIG5ldyBkZXNpZ24sIHdpdGggc2FtZSBjaGFuY2Ugb2YgYXNzaW5nbWVudAoKc2ltdWxhdGlvbnMgPC0gc2ltdWxhdGVfZGVzaWduKGRlc2lnbiwgZGVzaWduMiwgc2ltcyA9IDUwMCkKYGBgCgpgYGB7cn0KdGhlbWVfc2V0KHRoZW1lX3B1YnIoKSkKCnNpbXVsYXRpb25zICU+JQogIG11dGF0ZShkZXNpZ25fbGFiZWwgPSBjYXNlX3doZW4oZGVzaWduX2xhYmVsID09ICJkZXNpZ24iIH4gIlVuZXF1YWwgYXNzaWdubWVudCBwcm9iYWJpbGl0eSBibG9ja3MiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGVzaWduX2xhYmVsID09ICJkZXNpZ24yIiB+ICJFcXVhbCBhc3NpZ25tZW50IHByb2JhYmlsaXR5IGJsb2NrcyIpKSAlPiUKICBncm91cF9ieShkZXNpZ25fbGFiZWwsIGVzdGltYXRvcl9sYWJlbCkgJT4lCiAgc3VtbWFyaXplKFNFX2JpYXMgPSBtZWFuKHN0ZC5lcnJvciAtIHNkKGVzdGltYXRlKSksCiAgICAgICAgICAgIEFURV9iaWFzID0gbWVhbihlc3RpbWF0ZSAtIGVzdGltYW5kKSkgJT4lCiAgZ2dwbG90KGFlcyh4ID0gQVRFX2JpYXMsIHkgPSBTRV9iaWFzLCBjb2xvciA9IGRlc2lnbl9sYWJlbCwgbGFiZWwgPSBlc3RpbWF0b3JfbGFiZWwpKSArCiAgZ2VvbV9wb2ludCgpICsKICBnZW9tX2hsaW5lKHlpbnRlcmNlcHQgPSAwLCBzaXplID0gLjEsIGxpbmV0eXBlID0gImRhc2hlZCIpICsKICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQgPSAwLCBzaXplID0gLjEsIGxpbmV0eXBlID0gImRhc2hlZCIpICsKICBnZW9tX3RleHRfcmVwZWwoc2hvdy5sZWdlbmQgPSBGQUxTRSwKICAgICAgICAgICAgICAgICAgYm94LnBhZGRpbmcgPSAuNCwKICAgICAgICAgICAgICAgICAgcG9pbnQucGFkZGluZyA9IC42NSwKICAgICAgICAgICAgICAgICAgc2VnbWVudC5hbHBoYSA9IC41KSArCiAgc2NhbGVfeF9jb250aW51b3VzKGxpbWl0cyA9IGMoLS42LCAuNikpICsKICBzY2FsZV95X2NvbnRpbnVvdXMobGltaXRzID0gYygtLjEsIC4xKSkgKwogIHRoZW1lKGxlZ2VuZC50aXRsZSA9IGVsZW1lbnRfYmxhbmsoKSkgKwogIGxhYnMoeCA9ICJBdmVyYWdlIFRyZWF0bWVudCBFZmZlY3QgKEFURSkgQmlhcyIsCiAgICAgICB5ID0gIlN0YW5kYXJkIEVycm9yIEJpYXMiLAogICAgICAgdGl0bGUgPSAiQmlhcyBpbiBFdmVuIGFuZCBVbmV2ZW4gUHJvYmFiaWxpdHkgb2YgQXNzaWdubWVudCIpCmBgYAoKVGhlIGtleSB0YWtlYXdheSBoZXJlIGlzIHRoZSBvbGQgZmFzaGlvbmVkIHdlaWdodGVkLWF2ZXJhZ2UgZGlmZmVyZW5jZS1pbi1tZWFucyBpbiB0aGUgYmVzdCBmb3IgYm90aCBldmVuIGFuZCB1bmV2ZW4gYmxvY2sgc2l6ZXMuIFdoZW4gdGhlIGJsb2NrcyBoYXZlIGFuIGVxdWFsIGFzc2lnbm1lbnQgcHJvYmFiaWxpdHkgdGhlcmUgaXMgYSBzbGlnaHQgb3ZlcmVzdGltYXRpb24gb2YgdGhlIHN0YW5kYXJkIGVycm9yLCBidXQgaXQgaXMgdmVyeSBzbWFsbC4gV2hlbiB0aGVyZSBpcyB1bmVxdWFsIGFzc2lnbm1lbnQgcHJvYmFiaWxpdHksIGFsdGhvdWdoIHRoZSBmaXhlZCBlZmZlY3QgYXBwcm9hY2ggaXMgIG92ZXJlc3RpbWF0aW5nIHRoZSBBVEUsIHRoaXMga2luZCBvZiBtZXRob2QgaGFzIHRoZSBhZHZhbnRhZ2UgdGhhdCBpdCB3b3VsZCBhbGxvdyBhZGRpdGlvbmFsIGNvdmFyaWF0ZXMuCgpJdCBpcyBhbHNvIHdvcnRoIG5vdGluZyB0aGF0IHRoZXJlIGFwcGVhciB0byBiZSBiZXR0ZXIgbWV0aG9kcyBvZiBldmFsdWF0aW5nIHRoZSBBVEVzIGFzIHNob3duIGluIFt0aGUgYmxvZyBwb3N0XShodHRwczovL2RlY2xhcmVkZXNpZ24ub3JnL2xpYnJhcnkvYXJ0aWNsZXMvYmxvY2tfY2x1c3Rlcl90d29fYXJtLmh0bWwpLCBidXQgSSBkbyBub3QgZGVsdmUgaW50byB0aGVtIGhlcmUuIA==