In a randomized encouragement design (RED), participants are randomized to receive an invitation or special encouragement to receive a treatment (e.g., some new service). Not everyone who is encouraged will take up the service, but as long as those randomly selected to receive the encouragement (the treatment group) use the service at a higher rate than the control group, the impact of the service can be estimated.

REDs typically result in two-sided non-compliance with respect to random assignment:

  1. Everyone randomized to the encouragement arm will receive a special invitation to try the intervention, but only a subset of people in this group will take up this offer.

  2. People randomized to the control arm will NOT receive a special invitation to try the intervention, but some will learn about it through other channels and try it out on their own.

Encouragement designs account for this non-compliance by estimating the local average treatment effect (LATE). LATE is the effect of the intervention on ‘compliers’—those who tried the intervention because they were randomly encouraged to do so but would not have tried if not encouraged.

Create Toy Example

First we’ll create a dataframe with 134 participants and randomize them to the encouragement arm (trt==1) or the control arm (trt==0), 1:1 allocation to arms.

# setup
  library(dplyr)

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union
  library(arm)
Loading required package: MASS

Attaching package: ‘MASS’

The following object is masked from ‘package:dplyr’:

    select

Loading required package: Matrix
Loading required package: lme4

arm (Version 1.9-3, built: 2016-11-21)

Working directory is /Users/ericgreen/Box Sync/SENSITIVE Folder epg4/Repositories/bitbucket/nivi-red/reports
  library(AER)
Loading required package: car

Attaching package: ‘car’

The following object is masked from ‘package:arm’:

    logit

The following object is masked from ‘package:dplyr’:

    recode

Loading required package: lmtest
Loading required package: zoo

Attaching package: ‘zoo’

The following objects are masked from ‘package:base’:

    as.Date, as.Date.numeric

Loading required package: sandwich
Loading required package: survival
  library(ivpack)
  library(stargazer)

Please cite as: 

 Hlavac, Marek (2015). stargazer: Well-Formatted Regression and Summary Statistics Tables.
 R package version 5.2. http://CRAN.R-project.org/package=stargazer 
  n <- 134
  
# https://rpubs.com/wsundstrom/t_ivreg
# function to calculate corrected SEs for OLS regression 
  cse = function(reg) {
      rob = sqrt(diag(vcovHC(reg, type = "HC1")))
      return(rob)
  }
# corrected SEs for IV regressions... slight difference from S&W method
  ivse = function(reg) {
      rob = robust.se(reg)[,2]
      return(rob)
  }
  
# create dataframe
  dat <- data.frame(partID=seq(1, n, 1),
                    trt=c(rep(0, n/2), 
                          rep(1, n/2)))

Next we’ll set the proportion of intervention use for both arms. In other words, what percentage of people in each arm actually try the intervention, regardless of their encouragement status?

# set proportion use
  useT <- .8  # treatment group (encouraged)
  useC <- .2  # control group (not encouraged)
  
# create use variable
  set.seed(493)
  dat$use <- c(rbinom(n/2, 1, useC),
               rbinom(n/2, 1, useT))
# check
  use <- dat %>% 
    group_by(trt) %>%
    summarize(prop.use = mean(use))
  use

We can also add covariates. Let’s add age and make age correlated with intervention use.

# http://stackoverflow.com/questions/42147053/simulate-continuous-variable-that-is-correlated-to-existing-binary-variable
  x1    <- dat$use               # fixed given data
  rho   <- 0.1                   # desired correlation = cos(angle)
  theta <- acos(rho)             # corresponding angle
  x2    <- rnorm(n, 2, 0.5)      # new random data
  X     <- cbind(x1, x2)         # matrix
  Xctr  <- scale(X, center=TRUE, 
                 scale=FALSE)    # centered columns (mean 0)
  
  Id   <- diag(n)                           # identity matrix
  Q    <- qr.Q(qr(Xctr[ , 1, drop=FALSE]))  # QR-decomposition, just matrix Q
  P    <- tcrossprod(Q)          # = Q Q'   # projection onto space defined by x1
  x2o  <- (Id-P) %*% Xctr[ , 2]                 # x2ctr made orthogonal to x1ctr
  Xc2  <- cbind(Xctr[ , 1], x2o)                # bind to matrix
  Y    <- Xc2 %*% diag(1/sqrt(colSums(Xc2^2)))  # scale columns to length 1
  x <- Y[ , 2] + (1 / tan(theta)) * Y[ , 1]     # final new vector
  dat$age <- (1 + x) * 25 
  cor(dat$use, dat$age)
[1] 0.1
  dat$age <- round(dat$age, 0)

Now let’s simulate a binary outcome that is equal to 1 for 5% of the control group and 35% of the treatment group.

  outT <- .35
  outC <- .05
  dat$y <- c(rbinom(n/2, 1, outC),
             rbinom(n/2, 1, outT))
  dat %>% 
    group_by(trt) %>%
    summarize(prop.out = mean(y))
  head(dat)

Estimate Impact

1. Divide ITT estimate by difference in usage rates

One approach is to scale the intent-to-treat estimate of the impact of encouragement on the outcome by the difference in usage rates. First, we get the ITT estimate.

  itt <- lm(y ~ trt + rescale(age), data=dat)
  stargazer(itt,  
            se=list(cse(itt)),
            title="ITT estimate", 
            df=FALSE, digits=5,
            type="text")

ITT estimate
===============================================
                        Dependent variable:    
                    ---------------------------
                                 y             
-----------------------------------------------
trt                         0.24029***         
                             (0.05928)         
                                               
rescale(age)                 -0.01103          
                             (0.06097)         
                                               
Constant                      0.02911          
                             (0.02128)         
                                               
-----------------------------------------------
Observations                    134            
R2                            0.11251          
Adjusted R2                   0.09896          
Residual Std. Error           0.33952          
F Statistic                 8.30400***         
===============================================
Note:               *p<0.1; **p<0.05; ***p<0.01

Now divide 0.24 by (0.79 - 0.24)=0.55.

  LATE1 <- as.numeric(itt$coefficients[2]/(use[2,2]-use[1,2]))
  LATE1
[1] 0.4351147
  ctrlP1 <- itt$coefficients[1]
  new1 <- LATE1+ctrlP1

This means that the intervention shifted the prevalence of the outcome among compliers by 43.5 percentage points, from 2.9 to 46.4. BUT IS THIS THE RIGHT INTERPRETATION???

2. Two-Stage Regression

Another approach is to regress use on treatment, obtain the predicted values for use, and then regress the outcome on these predicted values. This estimate is nearly identical, but it comes with an estimate of the standard error.

  dat$predUse <- predict(lm(use ~ trt + rescale(age), data=dat))
  twoStage <- lm(y ~ predUse + rescale(age), data=dat)
  stargazer(twoStage,  
            se=list(cse(twoStage)),
            title="Two Stage Regression", 
            type="text", 
            df=FALSE, 
            digits=5,
            ci=TRUE)

Two Stage Regression
===============================================
                        Dependent variable:    
                    ---------------------------
                                 y             
-----------------------------------------------
predUse                     0.43805***         
                        (0.22624, 0.64986)     
                                               
rescale(age)                 -0.02312          
                        (-0.14360, 0.09737)    
                                               
Constant                     -0.07631*         
                        (-0.15330, 0.00068)    
                                               
-----------------------------------------------
Observations                    134            
R2                            0.11251          
Adjusted R2                   0.09896          
Residual Std. Error           0.33952          
F Statistic                 8.30400***         
===============================================
Note:               *p<0.1; **p<0.05; ***p<0.01

3. IV Regression

A third option is to run an IV regression. I’m following this guide for getting correct standard errors, but I’m not sure I’m implementing correctly.

  ivR = ivreg(y ~ use + rescale(age) | rescale(age) + trt , data = dat)
  stargazer(twoStage, ivR, 
            se=list(cse(twoStage), ivse(ivR)),
            title="Two Stage vs IV Regression", 
            type="text", 
            df=FALSE, 
            digits=5,
            ci=TRUE) 
[1] "Robust Standard Errors"

Two Stage vs IV Regression
===========================================================
                              Dependent variable:          
                    ---------------------------------------
                                       y                   
                            OLS            instrumental    
                                             variable      
                            (1)                 (2)        
-----------------------------------------------------------
predUse                 0.43805***                         
                    (0.22624, 0.64986)                     
                                                           
use                                         0.43805***     
                                        (0.20296, 0.67315) 
                                                           
rescale(age)             -0.02312            -0.02312      
                    (-0.14360, 0.09737) (-0.16765, 0.12141)
                                                           
Constant                 -0.07631*           -0.07631      
                    (-0.15330, 0.00068) (-0.18097, 0.02835)
                                                           
-----------------------------------------------------------
Observations                134                 134        
R2                        0.11251            -0.13434      
Adjusted R2               0.09896            -0.15166      
Residual Std. Error       0.33952             0.38384      
F Statistic             8.30400***                         
===========================================================
Note:                           *p<0.1; **p<0.05; ***p<0.01

Indrani obtained the same results in Stata:

ivreg2 y (use=trt) age, r

# IV (2SLS) estimation
# --------------------
# 
# Estimates efficient for homoskedasticity only
# Statistics robust to heteroskedasticity
# 
#                                                       Number of obs =      134
#                                                       F(  2,   131) =     6.54
#                                                       Prob > F      =   0.0020
# Total (centered) SS     =  17.01492537                Centered R2   =  -0.1343
# Total (uncentered) SS   =           20                Uncentered R2 =   0.0350
# Residual SS             =   19.3007356                Root MSE      =    .3795
# 
# ------------------------------------------------------------------------------
#              |               Robust
#            y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
# -------------+----------------------------------------------------------------
#          use |   .4380521   .1199491     3.65   0.000     .2029562     .673148
#          age |  -.0050714    .016178    -0.31   0.754    -.0367798     .026637
#        _cons |   .0506636   .3978013     0.13   0.899    -.7290127    .8303399
# ------------------------------------------------------------------------------
# Underidentification test (Kleibergen-Paap rk LM statistic):             38.900
#                                                    Chi-sq(1) P-val =    0.0000
# ------------------------------------------------------------------------------
# Weak identification test (Kleibergen-Paap rk Wald F statistic):         55.284
# Stock-Yogo weak ID test critical values: 10% maximal IV size             16.38
#                                          15% maximal IV size              8.96
#                                          20% maximal IV size              6.66
#                                          25% maximal IV size              5.53
# Source: Stock-Yogo (2005).  Reproduced by permission.
# NB: Critical values are for Cragg-Donald F statistic and i.i.d. errors.
# ------------------------------------------------------------------------------
# Hansen J statistic (overidentification test of all instruments):         0.000
#                                                  (equation exactly identified)
# ------------------------------------------------------------------------------
# Instrumented:         use
# Included instruments: age
# Excluded instruments: trt
# ------------------------------------------------------------------------------
LS0tCnRpdGxlOiAiUmFuZG9taXplZCBFbmNvdXJhZ2VtZW50IEFuYWx5c2lzIgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCgpJbiBhIHJhbmRvbWl6ZWQgZW5jb3VyYWdlbWVudCBkZXNpZ24gKFJFRCksIHBhcnRpY2lwYW50cyBhcmUgcmFuZG9taXplZCB0byByZWNlaXZlIGFuIGludml0YXRpb24gb3Igc3BlY2lhbCBlbmNvdXJhZ2VtZW50IHRvIHJlY2VpdmUgYSB0cmVhdG1lbnQgKGUuZy4sIHNvbWUgbmV3IHNlcnZpY2UpLiBOb3QgZXZlcnlvbmUgd2hvIGlzIGVuY291cmFnZWQgd2lsbCB0YWtlIHVwIHRoZSBzZXJ2aWNlLCBidXQgYXMgbG9uZyBhcyB0aG9zZSByYW5kb21seSBzZWxlY3RlZCB0byByZWNlaXZlIHRoZSBlbmNvdXJhZ2VtZW50ICh0aGUgdHJlYXRtZW50IGdyb3VwKSB1c2UgdGhlIHNlcnZpY2UgYXQgYSBoaWdoZXIgcmF0ZSB0aGFuIHRoZSBjb250cm9sIGdyb3VwLCB0aGUgaW1wYWN0IG9mIHRoZSBzZXJ2aWNlIGNhbiBiZSBlc3RpbWF0ZWQuCgpSRURzIHR5cGljYWxseSByZXN1bHQgaW4gdHdvLXNpZGVkIG5vbi1jb21wbGlhbmNlIHdpdGggcmVzcGVjdCB0byByYW5kb20gYXNzaWdubWVudDoKCjEuIEV2ZXJ5b25lIHJhbmRvbWl6ZWQgdG8gdGhlIGVuY291cmFnZW1lbnQgYXJtIHdpbGwgcmVjZWl2ZSBhIHNwZWNpYWwgaW52aXRhdGlvbiB0byB0cnkgdGhlIGludGVydmVudGlvbiwgYnV0IG9ubHkgYSBzdWJzZXQgb2YgcGVvcGxlIGluIHRoaXMgZ3JvdXAgd2lsbCB0YWtlIHVwIHRoaXMgb2ZmZXIuCgoyLiBQZW9wbGUgcmFuZG9taXplZCB0byB0aGUgY29udHJvbCBhcm0gd2lsbCBOT1QgcmVjZWl2ZSBhIHNwZWNpYWwgaW52aXRhdGlvbiB0byB0cnkgdGhlIGludGVydmVudGlvbiwgYnV0IHNvbWUgd2lsbCBsZWFybiBhYm91dCBpdCB0aHJvdWdoIG90aGVyIGNoYW5uZWxzIGFuZCB0cnkgaXQgb3V0IG9uIHRoZWlyIG93bi4KCkVuY291cmFnZW1lbnQgZGVzaWducyBhY2NvdW50IGZvciB0aGlzIG5vbi1jb21wbGlhbmNlIGJ5IGVzdGltYXRpbmcgdGhlIGxvY2FsIGF2ZXJhZ2UgdHJlYXRtZW50IGVmZmVjdCAoTEFURSkuIExBVEUgaXMgdGhlIGVmZmVjdCBvZiB0aGUgaW50ZXJ2ZW50aW9uIG9uICdjb21wbGllcnMn4oCUdGhvc2Ugd2hvIHRyaWVkIHRoZSBpbnRlcnZlbnRpb24gYmVjYXVzZSB0aGV5IHdlcmUgcmFuZG9tbHkgZW5jb3VyYWdlZCB0byBkbyBzbyBidXQgd291bGQgbm90IGhhdmUgdHJpZWQgaWYgbm90IGVuY291cmFnZWQuIAoKIyMgQ3JlYXRlIFRveSBFeGFtcGxlCmBgYHtyIGluY2x1ZGU9RkFMU0V9CiAgTiA8LSAxMzQKYGBgCgpGaXJzdCB3ZSdsbCBjcmVhdGUgYSBkYXRhZnJhbWUgd2l0aCBgciBOYCBwYXJ0aWNpcGFudHMgYW5kIHJhbmRvbWl6ZSB0aGVtIHRvIHRoZSBlbmNvdXJhZ2VtZW50IGFybSAoYHRydD09MWApIG9yIHRoZSBjb250cm9sIGFybSAoYHRydD09MGApLCAxOjEgYWxsb2NhdGlvbiB0byBhcm1zLgoKYGBge3J9CiMgc2V0dXAKICBsaWJyYXJ5KGRwbHlyKQogIGxpYnJhcnkoYXJtKQogIGxpYnJhcnkoQUVSKQogIGxpYnJhcnkoaXZwYWNrKQogIGxpYnJhcnkoc3RhcmdhemVyKQogIG4gPC0gMTM0CiAgCiMgaHR0cHM6Ly9ycHVicy5jb20vd3N1bmRzdHJvbS90X2l2cmVnCiMgZnVuY3Rpb24gdG8gY2FsY3VsYXRlIGNvcnJlY3RlZCBTRXMgZm9yIE9MUyByZWdyZXNzaW9uIAogIGNzZSA9IGZ1bmN0aW9uKHJlZykgewogICAgICByb2IgPSBzcXJ0KGRpYWcodmNvdkhDKHJlZywgdHlwZSA9ICJIQzEiKSkpCiAgICAgIHJldHVybihyb2IpCiAgfQojIGNvcnJlY3RlZCBTRXMgZm9yIElWIHJlZ3Jlc3Npb25zLi4uIHNsaWdodCBkaWZmZXJlbmNlIGZyb20gUyZXIG1ldGhvZAogIGl2c2UgPSBmdW5jdGlvbihyZWcpIHsKICAgICAgcm9iID0gcm9idXN0LnNlKHJlZylbLDJdCiAgICAgIHJldHVybihyb2IpCiAgfQogIAojIGNyZWF0ZSBkYXRhZnJhbWUKICBkYXQgPC0gZGF0YS5mcmFtZShwYXJ0SUQ9c2VxKDEsIG4sIDEpLAogICAgICAgICAgICAgICAgICAgIHRydD1jKHJlcCgwLCBuLzIpLCAKICAgICAgICAgICAgICAgICAgICAgICAgICByZXAoMSwgbi8yKSkpCmBgYAoKTmV4dCB3ZSdsbCBzZXQgdGhlIHByb3BvcnRpb24gb2YgaW50ZXJ2ZW50aW9uIHVzZSBmb3IgYm90aCBhcm1zLiBJbiBvdGhlciB3b3Jkcywgd2hhdCBwZXJjZW50YWdlIG9mIHBlb3BsZSBpbiBlYWNoIGFybSBhY3R1YWxseSB0cnkgdGhlIGludGVydmVudGlvbiwgcmVnYXJkbGVzcyBvZiB0aGVpciBlbmNvdXJhZ2VtZW50IHN0YXR1cz8KCmBgYHtyfQojIHNldCBwcm9wb3J0aW9uIHVzZQogIHVzZVQgPC0gLjggICMgdHJlYXRtZW50IGdyb3VwIChlbmNvdXJhZ2VkKQogIHVzZUMgPC0gLjIgICMgY29udHJvbCBncm91cCAobm90IGVuY291cmFnZWQpCiAgCiMgY3JlYXRlIHVzZSB2YXJpYWJsZQogIHNldC5zZWVkKDQ5MykKICBkYXQkdXNlIDwtIGMocmJpbm9tKG4vMiwgMSwgdXNlQyksCiAgICAgICAgICAgICAgIHJiaW5vbShuLzIsIDEsIHVzZVQpKQoKIyBjaGVjawogIHVzZSA8LSBkYXQgJT4lIAogICAgZ3JvdXBfYnkodHJ0KSAlPiUKICAgIHN1bW1hcml6ZShwcm9wLnVzZSA9IG1lYW4odXNlKSkKICB1c2UKYGBgCgpXZSBjYW4gYWxzbyBhZGQgY292YXJpYXRlcy4gTGV0J3MgYWRkIGFnZSBhbmQgbWFrZSBhZ2UgY29ycmVsYXRlZCB3aXRoIGludGVydmVudGlvbiB1c2UuCgpgYGB7ciwgZWNobz1UUlVFfQojIGh0dHA6Ly9zdGFja292ZXJmbG93LmNvbS9xdWVzdGlvbnMvNDIxNDcwNTMvc2ltdWxhdGUtY29udGludW91cy12YXJpYWJsZS10aGF0LWlzLWNvcnJlbGF0ZWQtdG8tZXhpc3RpbmctYmluYXJ5LXZhcmlhYmxlCgogIHgxICAgIDwtIGRhdCR1c2UgICAgICAgICAgICAgICAjIGZpeGVkIGdpdmVuIGRhdGEKICByaG8gICA8LSAwLjEgICAgICAgICAgICAgICAgICAgIyBkZXNpcmVkIGNvcnJlbGF0aW9uID0gY29zKGFuZ2xlKQogIHRoZXRhIDwtIGFjb3MocmhvKSAgICAgICAgICAgICAjIGNvcnJlc3BvbmRpbmcgYW5nbGUKICB4MiAgICA8LSBybm9ybShuLCAyLCAwLjUpICAgICAgIyBuZXcgcmFuZG9tIGRhdGEKICBYICAgICA8LSBjYmluZCh4MSwgeDIpICAgICAgICAgIyBtYXRyaXgKICBYY3RyICA8LSBzY2FsZShYLCBjZW50ZXI9VFJVRSwgCiAgICAgICAgICAgICAgICAgc2NhbGU9RkFMU0UpICAgICMgY2VudGVyZWQgY29sdW1ucyAobWVhbiAwKQogIAogIElkICAgPC0gZGlhZyhuKSAgICAgICAgICAgICAgICAgICAgICAgICAgICMgaWRlbnRpdHkgbWF0cml4CiAgUSAgICA8LSBxci5RKHFyKFhjdHJbICwgMSwgZHJvcD1GQUxTRV0pKSAgIyBRUi1kZWNvbXBvc2l0aW9uLCBqdXN0IG1hdHJpeCBRCiAgUCAgICA8LSB0Y3Jvc3Nwcm9kKFEpICAgICAgICAgICMgPSBRIFEnICAgIyBwcm9qZWN0aW9uIG9udG8gc3BhY2UgZGVmaW5lZCBieSB4MQogIHgybyAgPC0gKElkLVApICUqJSBYY3RyWyAsIDJdICAgICAgICAgICAgICAgICAjIHgyY3RyIG1hZGUgb3J0aG9nb25hbCB0byB4MWN0cgogIFhjMiAgPC0gY2JpbmQoWGN0clsgLCAxXSwgeDJvKSAgICAgICAgICAgICAgICAjIGJpbmQgdG8gbWF0cml4CiAgWSAgICA8LSBYYzIgJSolIGRpYWcoMS9zcXJ0KGNvbFN1bXMoWGMyXjIpKSkgICMgc2NhbGUgY29sdW1ucyB0byBsZW5ndGggMQoKICB4IDwtIFlbICwgMl0gKyAoMSAvIHRhbih0aGV0YSkpICogWVsgLCAxXSAgICAgIyBmaW5hbCBuZXcgdmVjdG9yCgogIGRhdCRhZ2UgPC0gKDEgKyB4KSAqIDI1IAogIGNvcihkYXQkdXNlLCBkYXQkYWdlKQogIGRhdCRhZ2UgPC0gcm91bmQoZGF0JGFnZSwgMCkKYGBgCgpOb3cgbGV0J3Mgc2ltdWxhdGUgYSBiaW5hcnkgb3V0Y29tZSB0aGF0IGlzIGVxdWFsIHRvIDEgZm9yIDUlIG9mIHRoZSBjb250cm9sIGdyb3VwIGFuZCAzNSUgb2YgdGhlIHRyZWF0bWVudCBncm91cC4KCmBgYHtyfQogIG91dFQgPC0gLjM1CiAgb3V0QyA8LSAuMDUKICBkYXQkeSA8LSBjKHJiaW5vbShuLzIsIDEsIG91dEMpLAogICAgICAgICAgICAgcmJpbm9tKG4vMiwgMSwgb3V0VCkpCiAgZGF0ICU+JSAKICAgIGdyb3VwX2J5KHRydCkgJT4lCiAgICBzdW1tYXJpemUocHJvcC5vdXQgPSBtZWFuKHkpKQogIGhlYWQoZGF0KQpgYGAKCiMjIEVzdGltYXRlIEltcGFjdAoKIyMjIDEuIERpdmlkZSBJVFQgZXN0aW1hdGUgYnkgZGlmZmVyZW5jZSBpbiB1c2FnZSByYXRlcwoKT25lIGFwcHJvYWNoIGlzIHRvIHNjYWxlIHRoZSBpbnRlbnQtdG8tdHJlYXQgZXN0aW1hdGUgb2YgdGhlIGltcGFjdCBvZiBlbmNvdXJhZ2VtZW50IG9uIHRoZSBvdXRjb21lIGJ5IHRoZSBkaWZmZXJlbmNlIGluIHVzYWdlIHJhdGVzLiBGaXJzdCwgd2UgZ2V0IHRoZSBJVFQgZXN0aW1hdGUuCgpgYGB7ciwgcmVzdWx0cz0ibWFya2Rvd24ifQogIGl0dCA8LSBsbSh5IH4gdHJ0ICsgcmVzY2FsZShhZ2UpLCBkYXRhPWRhdCkKICBzdGFyZ2F6ZXIoaXR0LCAgCiAgICAgICAgICAgIHNlPWxpc3QoY3NlKGl0dCkpLAogICAgICAgICAgICB0aXRsZT0iSVRUIGVzdGltYXRlIiwgCiAgICAgICAgICAgIGRmPUZBTFNFLCBkaWdpdHM9NSwKICAgICAgICAgICAgdHlwZT0idGV4dCIpCmBgYAoKTm93IGRpdmlkZSBgciByb3VuZChpdHQkY29lZmZpY2llbnRzWzJdLCAyKWAgYnkgYChgciByb3VuZChhcy5udW1lcmljKHVzZVsyLDJdKSwgMilgIC0gYHIgcm91bmQoYXMubnVtZXJpYyh1c2VbMSwyXSksIDIpYCk9YHIgcm91bmQoYXMubnVtZXJpYyh1c2VbMiwyXS11c2VbMSwyXSksIDIpYGAuCgpgYGB7cn0KICBMQVRFMSA8LSBhcy5udW1lcmljKGl0dCRjb2VmZmljaWVudHNbMl0vKHVzZVsyLDJdLXVzZVsxLDJdKSkKICBMQVRFMQogIGN0cmxQMSA8LSBpdHQkY29lZmZpY2llbnRzWzFdCiAgbmV3MSA8LSBMQVRFMStjdHJsUDEKYGBgCgpUaGlzIG1lYW5zIHRoYXQgdGhlIGludGVydmVudGlvbiBzaGlmdGVkIHRoZSBwcmV2YWxlbmNlIG9mIHRoZSBvdXRjb21lIGFtb25nIGNvbXBsaWVycyBieSBgciByb3VuZChMQVRFMSoxMDAsIDEpYCBwZXJjZW50YWdlIHBvaW50cywgZnJvbSBgciByb3VuZChjdHJsUDEqMTAwLCAxKWAgdG8gYHIgcm91bmQobmV3MSoxMDAsIDEpYC4gQlVUIElTIFRISVMgVEhFIFJJR0hUIElOVEVSUFJFVEFUSU9OPz8/CgojIyMgMi4gVHdvLVN0YWdlIFJlZ3Jlc3Npb24KCkFub3RoZXIgYXBwcm9hY2ggaXMgdG8gcmVncmVzcyB1c2Ugb24gdHJlYXRtZW50LCBvYnRhaW4gdGhlIHByZWRpY3RlZCB2YWx1ZXMgZm9yIHVzZSwgYW5kIHRoZW4gcmVncmVzcyB0aGUgb3V0Y29tZSBvbiB0aGVzZSBwcmVkaWN0ZWQgdmFsdWVzLiBUaGlzIGVzdGltYXRlIGlzIG5lYXJseSBpZGVudGljYWwsIGJ1dCBpdCBjb21lcyB3aXRoIGFuIGVzdGltYXRlIG9mIHRoZSBzdGFuZGFyZCBlcnJvci4KCmBgYHtyLCByZXN1bHRzPSJtYXJrZG93biJ9CiAgZGF0JHByZWRVc2UgPC0gcHJlZGljdChsbSh1c2UgfiB0cnQgKyByZXNjYWxlKGFnZSksIGRhdGE9ZGF0KSkKICB0d29TdGFnZSA8LSBsbSh5IH4gcHJlZFVzZSArIHJlc2NhbGUoYWdlKSwgZGF0YT1kYXQpCiAgc3RhcmdhemVyKHR3b1N0YWdlLCAgCiAgICAgICAgICAgIHNlPWxpc3QoY3NlKHR3b1N0YWdlKSksCiAgICAgICAgICAgIHRpdGxlPSJUd28gU3RhZ2UgUmVncmVzc2lvbiIsIAogICAgICAgICAgICB0eXBlPSJ0ZXh0IiwgCiAgICAgICAgICAgIGRmPUZBTFNFLCAKICAgICAgICAgICAgZGlnaXRzPTUsCiAgICAgICAgICAgIGNpPVRSVUUpCmBgYAoKIyMjIDMuIElWIFJlZ3Jlc3Npb24KCkEgdGhpcmQgb3B0aW9uIGlzIHRvIHJ1biBhbiBJViByZWdyZXNzaW9uLiBJJ20gZm9sbG93aW5nIFt0aGlzIGd1aWRlXShodHRwczovL3JwdWJzLmNvbS93c3VuZHN0cm9tL3RfaXZyZWcpIGZvciBnZXR0aW5nIGNvcnJlY3Qgc3RhbmRhcmQgZXJyb3JzLCBidXQgSSdtIG5vdCBzdXJlIEknbSBpbXBsZW1lbnRpbmcgY29ycmVjdGx5LgoKYGBge3J9CiAgaXZSID0gaXZyZWcoeSB+IHVzZSArIHJlc2NhbGUoYWdlKSB8IHJlc2NhbGUoYWdlKSArIHRydCAsIGRhdGEgPSBkYXQpCiAgc3RhcmdhemVyKHR3b1N0YWdlLCBpdlIsIAogICAgICAgICAgICBzZT1saXN0KGNzZSh0d29TdGFnZSksIGl2c2UoaXZSKSksCiAgICAgICAgICAgIHRpdGxlPSJUd28gU3RhZ2UgdnMgSVYgUmVncmVzc2lvbiIsIAogICAgICAgICAgICB0eXBlPSJ0ZXh0IiwgCiAgICAgICAgICAgIGRmPUZBTFNFLCAKICAgICAgICAgICAgZGlnaXRzPTUsCiAgICAgICAgICAgIGNpPVRSVUUpIApgYGAKCkluZHJhbmkgb2J0YWluZWQgdGhlIHNhbWUgcmVzdWx0cyBpbiBTdGF0YToKCmBpdnJlZzIgeSAodXNlPXRydCkgYWdlLCByYAoKYGBge3J9CiMgSVYgKDJTTFMpIGVzdGltYXRpb24KIyAtLS0tLS0tLS0tLS0tLS0tLS0tLQojIAojIEVzdGltYXRlcyBlZmZpY2llbnQgZm9yIGhvbW9za2VkYXN0aWNpdHkgb25seQojIFN0YXRpc3RpY3Mgcm9idXN0IHRvIGhldGVyb3NrZWRhc3RpY2l0eQojIAojICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIE51bWJlciBvZiBvYnMgPSAgICAgIDEzNAojICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEYoICAyLCAgIDEzMSkgPSAgICAgNi41NAojICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFByb2IgPiBGICAgICAgPSAgIDAuMDAyMAojIFRvdGFsIChjZW50ZXJlZCkgU1MgICAgID0gIDE3LjAxNDkyNTM3ICAgICAgICAgICAgICAgIENlbnRlcmVkIFIyICAgPSAgLTAuMTM0MwojIFRvdGFsICh1bmNlbnRlcmVkKSBTUyAgID0gICAgICAgICAgIDIwICAgICAgICAgICAgICAgIFVuY2VudGVyZWQgUjIgPSAgIDAuMDM1MAojIFJlc2lkdWFsIFNTICAgICAgICAgICAgID0gICAxOS4zMDA3MzU2ICAgICAgICAgICAgICAgIFJvb3QgTVNFICAgICAgPSAgICAuMzc5NQojIAojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQojICAgICAgICAgICAgICB8ICAgICAgICAgICAgICAgUm9idXN0CiMgICAgICAgICAgICB5IHwgICAgICBDb2VmLiAgIFN0ZC4gRXJyLiAgICAgIHogICAgUD58enwgICAgIFs5NSUgQ29uZi4gSW50ZXJ2YWxdCiMgLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgICAgICAgICAgdXNlIHwgICAuNDM4MDUyMSAgIC4xMTk5NDkxICAgICAzLjY1ICAgMC4wMDAgICAgIC4yMDI5NTYyICAgICAuNjczMTQ4CiMgICAgICAgICAgYWdlIHwgIC0uMDA1MDcxNCAgICAuMDE2MTc4ICAgIC0wLjMxICAgMC43NTQgICAgLS4wMzY3Nzk4ICAgICAuMDI2NjM3CiMgICAgICAgIF9jb25zIHwgICAuMDUwNjYzNiAgIC4zOTc4MDEzICAgICAwLjEzICAgMC44OTkgICAgLS43MjkwMTI3ICAgIC44MzAzMzk5CiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgVW5kZXJpZGVudGlmaWNhdGlvbiB0ZXN0IChLbGVpYmVyZ2VuLVBhYXAgcmsgTE0gc3RhdGlzdGljKTogICAgICAgICAgICAgMzguOTAwCiMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgQ2hpLXNxKDEpIFAtdmFsID0gICAgMC4wMDAwCiMgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiMgV2VhayBpZGVudGlmaWNhdGlvbiB0ZXN0IChLbGVpYmVyZ2VuLVBhYXAgcmsgV2FsZCBGIHN0YXRpc3RpYyk6ICAgICAgICAgNTUuMjg0CiMgU3RvY2stWW9nbyB3ZWFrIElEIHRlc3QgY3JpdGljYWwgdmFsdWVzOiAxMCUgbWF4aW1hbCBJViBzaXplICAgICAgICAgICAgIDE2LjM4CiMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAxNSUgbWF4aW1hbCBJViBzaXplICAgICAgICAgICAgICA4Ljk2CiMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAyMCUgbWF4aW1hbCBJViBzaXplICAgICAgICAgICAgICA2LjY2CiMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAyNSUgbWF4aW1hbCBJViBzaXplICAgICAgICAgICAgICA1LjUzCiMgU291cmNlOiBTdG9jay1Zb2dvICgyMDA1KS4gIFJlcHJvZHVjZWQgYnkgcGVybWlzc2lvbi4KIyBOQjogQ3JpdGljYWwgdmFsdWVzIGFyZSBmb3IgQ3JhZ2ctRG9uYWxkIEYgc3RhdGlzdGljIGFuZCBpLmkuZC4gZXJyb3JzLgojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQojIEhhbnNlbiBKIHN0YXRpc3RpYyAob3ZlcmlkZW50aWZpY2F0aW9uIHRlc3Qgb2YgYWxsIGluc3RydW1lbnRzKTogICAgICAgICAwLjAwMAojICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAoZXF1YXRpb24gZXhhY3RseSBpZGVudGlmaWVkKQojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQojIEluc3RydW1lbnRlZDogICAgICAgICB1c2UKIyBJbmNsdWRlZCBpbnN0cnVtZW50czogYWdlCiMgRXhjbHVkZWQgaW5zdHJ1bWVudHM6IHRydAojIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpgYGAKCgo=