Tutorial 1: Develop a prediction model
New to R? Before you start, install R and RStudio from posit.co and work through the PREDICT Practical 1 (Getting started with R) and the Introduction to R: the chapters “Start”, “Set-up” (R, RStudio) and “The basics”, and the exercises at the bottom of the pages Creating data, Using data, Packages and External data.
Overview
In this tutorial you will go from raw data to a final prediction model for a time-to-event outcome. It has four parts:
Load data and make Table 1: describe the study population.
Describe the survival data: define the outcome, make a Kaplan-Meier curve, choose a time horizon and calculate the median follow-up time.
Multiple imputation: deal with missing data.
Fit the models: check non-linear effects, select predictors and present the final model.
Tutorial 2 continues with the final model and validates it
(shrinkage, optimism-corrected C-index, calibration and decision
curves). We use the pbc data (primary biliary cirrhosis)
from the survival package as dummy data.
Download the tutorial
Do not just read this page: run the code yourself.
You only learn R by doing it. Download the R Markdown file of this
tutorial (Tutorial_1_Develop_Prediction_Model.Rmd) by
clicking the Code button at the top right of this page
and choosing Download Rmd, and run every chunk in
RStudio, one by one. Read the output, change the code, break it and fix
it, and try the exercises before you open the answers. Reading the
results on this page is not a substitute.
To make sure everything works:
Create a folder for the course, for example
Predictive_models/, and put the downloaded.Rmdfile in it.Open the
.Rmdfile in RStudio. When you knit or run chunks from an Rmd file, R uses the folder that contains the file as the working directory. Check this by running:getwd()The output should be the path of the folder from step 1. If it is not, set it with
setwd("path/to/your/folder")or in RStudio via Session > Set Working Directory > To Source File Location.Do not use file paths in the code. All files are read from and saved to the working directory, so keep all files in this one folder.
Running the tutorial saves its output (for example the Kaplan-Meier figure
KM_plot_10_years.pngand Table 1 as a Word file) in this same folder.
The folder will then look like this:
Predictive_models/
├── Tutorial_1_Develop_Prediction_Model.Rmd
├── Tutorial_2_Validate_Prediction_Model.Rmd
├── Descriptives.docx (created by Tutorial 1)
└── KM_plot_10_years.png (created by Tutorial 1)Setup
Clear the environment and load libraries
Start with a clean environment so that results do not depend on
objects left over from earlier work. A package is a collection
of functions that someone wrote and shared.
library(package) loads a package, so that you can use its
functions directly. We only load rms, because we use its
functions (such as cph, Predict and
fastbw) without the package:: prefix. All
other packages are called with the :: notation (see below),
so they do not have to be loaded, but they do have to be installed once.
We use:
survival: the data and survival analysisdplyr: data manipulationarsenalandflextable: Table 1 and saving it to Wordsurvminerandprodlim: Kaplan-Meier curves and follow-up timevisdatandmice: visualizing and imputing missing datarmsandHmisc: fitting and presenting prediction models (cph,fit.mult.impute)ggplot2andggpubr: saving and combining figures
# clear the environment: remove all objects that R currently remembers
rm(list=ls())
# load the rms package: after this, all its functions can be used directly
library(rms)
# The other packages must be installed once on your computer (not run here):
# install.packages(c("survival", "dplyr", "arsenal", "flextable", "survminer",
# "prodlim", "visdat", "mice", "Hmisc", "ggplot2", "ggpubr"))The :: notation
In this tutorial you will often see code like
dplyr::select(...). The notation
package::function() means “use the function
function from the package package”.
You do not have to write it this way once you have loaded a package
with library(): after library(dplyr),
select(...) works as well. We still write it out because it
is nice to see where each function comes from, and it avoids mistakes
when two packages have a function with the same name (R then uses the
one from the package that was loaded last). For a package that you do
not load with library(), such as openxlsx or
foreign in the templates below, the ::
notation lets you use one of its functions without loading the whole
package.
Set the seed
Some steps in this tutorial use random numbers, most importantly
multiple imputation. Without a seed, you would get different imputed
values, and therefore slightly different results, every time you run the
code. set.seed() fixes the starting point of R’s random
number generator, so everyone who runs the code obtains the same
numbers:
# the same seed gives the same random numbers
set.seed(1)
rnorm(3)## [1] -0.6264538 0.1836433 -0.8356286set.seed(1)
rnorm(3)## [1] -0.6264538 0.1836433 -0.8356286We set the random seed once, here at the top. In
addition, mice() has its own seed argument,
which we use when we impute, so that the imputations do not depend on
what was run before. If you change the order of the code or add extra
random steps, the numbers after that point can change.
# set the seed once for the whole tutorial
set.seed(1)Work in one folder
Everything in this tutorial is saved in one folder: the folder that contains this Rmd file. When you knit the document (or run the chunks in RStudio), the working directory is this folder, so tables, figures and R data files are saved right next to the Rmd and you do not need to set any file paths.
Your turn: check where your files will be saved.
# all files are saved in the working directory, which is the folder of this Rmd
getwd()1 Load data and make Table 1
Every prediction model project starts by looking at the data. In this part you load the data, decide which variables are candidate predictors, make sure every variable has the correct type, and create a “Table 1” that describes your study population. You save this table to Word so you can use it in a manuscript.
1.1 Load the data
In your own project you will read your own data into R, most often
from an Excel or SPSS file. The next two blocks show how to do that.
They are not run in this tutorial
(eval=FALSE), because we do not have your files; they are
here as templates for your own projects. After them, we load the dummy
data that we actually use.
1.1.1 Template: load an Excel file (not run)
openxlsx::read.xlsx() reads an .xlsx file
into a data frame (older .xls files must first be saved as
.xlsx). Replace the path with the location of your file. If
your file is in the same folder as this Rmd, the file name alone is
enough. If the data are not on the first sheet, add the argument
sheet="Sheet name" (or the sheet number). Install the
package once with install.packages("openxlsx") if you do
not have it yet.
# load Excel data (this block is not run)
data <- openxlsx::read.xlsx("C:/.../file_data_name.xlsx")1.1.2 Template: load an SPSS file (not run)
foreign::read.spss() reads an SPSS .sav
file. With to.data.frame=TRUE the result is a data frame
instead of a list. SPSS stores dates as seconds since a starting date,
so origin="01-01-1970" tells R how to convert them to
dates. Variables with value labels in SPSS become factors in R, so check
afterwards that every variable has the type you expect (see the next
section).
# load SPSS file (this block is not run)
data <- foreign::read.spss(file="C:/.../file_data_name.sav", to.data.frame=TRUE, origin="01-01-1970")1.1.3 Load the dummy data that we use
Here we use the dummy data from the survival package.
The data dictionary is available at https://cran.r-project.org/web/packages/survival/survival.pdf.
# load the pbc data set from the survival package into R's memory
data(pbc, package="survival")
# copy it to an object with the general name `data` (<- assigns a value to a name)
data <- pbcWe also add a center variable. Note: this
variable is artificial. It simply alternates between 1 and 2,
so it carries no real between-center differences. That is fine for a
demonstration; we use it in Tutorial 2 to illustrate internal-external
cross-validation.
# add center variable for illustration (artificial, fine for a demo)
# rep(1:2, length.out=n) repeats 1, 2, 1, 2, ... until we have n values, with
# n = nrow(data) the number of patients; `data$center <-` stores them as a new column
data$center <- rep(1:2, length.out=nrow(data))Picking a column: $ versus
[[. Here and below we use data$center
to pick a column, which is short to type. R has two ways to take one
column out of a data frame, and they behave differently:
[keeps the container,[[takes one element out of it:data["center"]is still a data frame, whereasdata[["center"]]is the column itself.data$centeralso gives the column itself.$does partial matching:data$censilently gives you thecentercolumn. In the console that is convenient, but in a script a typo returns the wrong column instead of an error.$takes the name literally, so it cannot use a column name that is stored in a variable.[[evaluates the variable first, which is what makes loops over variable names possible (we use this in Part 3).
# the column itself
head(data$center)## [1] 1 2 1 2 1 2head(data[["center"]])## [1] 1 2 1 2 1 2class(data["center"]) # a data frame with one column## [1] "data.frame"class(data[["center"]]) # the column itself## [1] "integer"# partial matching with $
head(data$cen)## [1] 1 2 1 2 1 2# column name stored in a variable
my_var <- "center"
data$my_var # NULL: looks for a column called my_var## NULLhead(data[[my_var]]) # works## [1] 1 2 1 2 1 2In scripts, [["name"]] is therefore safer, and it is the
only option when the column name is stored in a variable.
1.2 Select variables and set variable types
Select the outcome variables (time and
status) and the variables that you want to describe and use
as candidate predictors, based on literature and clinical knowledge.
Binary and categorical variables must be factor variables,
otherwise R treats them as numbers and reports a mean and SD for
them.
Two variables deserve attention: trt (treatment, coded
1/2) and edema (coded 0, 0.5, 1) are categories, not
quantities, so they are also converted to factors. We set every factor
separately with factor(): levels are the
values as they are stored in the data and labels are the
names that we want to see in tables, figures and model output. The
labels must be in the same order as the levels. Check the codebook of
your data to see what each value means (in pbc,
edema 0 = no edema, 0.5 = edema untreated or successfully
treated, 1 = edema despite diuretic therapy, and sex is
stored as “m” and “f”).
Note: check how binary variables are coded.
trt is coded 1/2 here. This often happens when you load
data from SPSS, where categories are stored as 1/2 instead of 0/1.
Always check this (for example with table(data$trt) or
str(data)), because a variable coded 1/2 is easily mistaken
for a quantity, and the reference category is not obvious. It is
probably best to recode binary variables to 0/1 (for example 0 =
no/control, 1 = yes/treatment), or to give them readable labels as we do
below. Never rely on as.factor() alone, because it keeps
the stored values (1 and 2) as labels.
# select the outcome and candidate variables based on literature
# (dplyr::select() keeps only the columns that we name)
data <- dplyr::select(data, time, status,
trt, age, sex, ascites, hepato, spiders, edema,
bili, chol, albumin, copper, alk.phos, ast, trig,
platelet, protime, stage, center)
# binary and categorical variables should be factor variables:
# factor(x, levels=..., labels=...) takes the values that are stored in the data
# (levels) and gives each of them a readable label (in the same order)
data$trt <- factor(data$trt, levels=c(1, 2),
labels=c("D-penicillamine", "Placebo"))
data$sex <- factor(data$sex, levels=c("m", "f"),
labels=c("Male", "Female"))
data$ascites <- factor(data$ascites, levels=c(0, 1),
labels=c("No", "Yes"))
data$hepato <- factor(data$hepato, levels=c(0, 1),
labels=c("No", "Yes"))
data$spiders <- factor(data$spiders, levels=c(0, 1),
labels=c("No", "Yes"))
data$edema <- factor(data$edema, levels=c(0, 0.5, 1),
labels=c("No edema", "Untreated or successfully treated",
"Despite diuretics"))
data$stage <- factor(data$stage, levels=1:4,
labels=c("1", "2", "3", "4"))
data$center <- factor(data$center, levels=c(1, 2),
labels=c("Center 1", "Center 2"))Your turn: run str(data) and check that
every variable has the type you expect.
Check the labels. factor() silently
turns every value that is not listed in levels into a
missing value (NA), for example when a level is misspelled
or the data are coded differently than you think. After setting the
labels, always count the values per level and include the missing values
with useNA="ifany", and compare the counts with those
before recoding:
# table() counts how often each value occurs; useNA="ifany" also counts the
# missing values (NA), so that wrongly labelled values show up
table(data$trt, useNA="ifany")##
## D-penicillamine Placebo <NA>
## 158 154 106table(data$sex, useNA="ifany")##
## Male Female
## 44 374table(data$edema, useNA="ifany")##
## No edema Untreated or successfully treated
## 354 44
## Despite diuretics
## 20The first level is the reference category in models:
the coefficients of a factor compare each other level with the first
one. That is why the model output later shows names such as
ascites=Yes (compared with “No”) and
sex=Female (compared with “Male”). Use
relevel(x, ref="...") to choose another reference
category.
1.3 Make Table 1
arsenal::tableby() summarizes each variable, here by
center. We show the mean (SD), median (Q1, Q3), range and
number of missing values for continuous variables, and counts
(percentages) and missing values for categorical variables. The outcome
variables time and status are left out of the
table. We switch off the statistical tests (test=FALSE)
because a Table 1 in a prediction model paper describes the population
and does not compare groups.
# center ~ . means: one column per center, describing all other variables (.)
# data=...: all variables except time and status (-time, -status leaves them out)
table_one <- arsenal::tableby(center ~ .,
data=dplyr::select(data, -time, -status),
total.pos="before", # put the total column first
digits=1, # number of decimals
test=FALSE, # no p-values
total=TRUE, # add a column for all patients
numeric.test="kwt",
cat.test="chisq",
cat.simplify=TRUE, # one row for binary variables
# statistics shown for continuous variables ...
numeric.stats=c("meansd", "medianq1q3",
"range", "Nmiss2"),
# ... and for categorical variables
cat.stats=c("countpct", "Nmiss2"),
# readable names for the statistics
stats.labels=list(
meansd="Mean (SD)",
medianq1q3="Median (Q1, Q3)",
range="Min - Max",
Nmiss2="Missing"))Here is the table as it will look in the report:
summary(table_one, text="html")| Total (N=418) | Center 1 (N=209) | Center 2 (N=209) | |
|---|---|---|---|
| trt | |||
| D-penicillamine | 158 (50.6%) | 71 (45.5%) | 87 (55.8%) |
| Placebo | 154 (49.4%) | 85 (54.5%) | 69 (44.2%) |
| Missing | 106 | 53 | 53 |
| age | |||
| Mean (SD) | 50.7 (10.4) | 50.5 (10.0) | 51.0 (10.9) |
| Median (Q1, Q3) | 51.0 (42.8, 58.2) | 49.6 (43.0, 58.0) | 52.0 (42.8, 58.4) |
| Min - Max | 26.3 - 78.4 | 29.6 - 78.4 | 26.3 - 76.7 |
| Missing | 0 | 0 | 0 |
| sex | |||
| Male | 44 (10.5%) | 20 (9.6%) | 24 (11.5%) |
| Female | 374 (89.5%) | 189 (90.4%) | 185 (88.5%) |
| Missing | 0 | 0 | 0 |
| ascites | |||
| No | 288 (92.3%) | 141 (90.4%) | 147 (94.2%) |
| Yes | 24 (7.7%) | 15 (9.6%) | 9 (5.8%) |
| Missing | 106 | 53 | 53 |
| hepato | |||
| No | 152 (48.7%) | 76 (48.7%) | 76 (48.7%) |
| Yes | 160 (51.3%) | 80 (51.3%) | 80 (51.3%) |
| Missing | 106 | 53 | 53 |
| spiders | |||
| No | 222 (71.2%) | 111 (71.2%) | 111 (71.2%) |
| Yes | 90 (28.8%) | 45 (28.8%) | 45 (28.8%) |
| Missing | 106 | 53 | 53 |
| edema | |||
| No edema | 354 (84.7%) | 172 (82.3%) | 182 (87.1%) |
| Untreated or successfully treated | 44 (10.5%) | 25 (12.0%) | 19 (9.1%) |
| Despite diuretics | 20 (4.8%) | 12 (5.7%) | 8 (3.8%) |
| Missing | 0 | 0 | 0 |
| bili | |||
| Mean (SD) | 3.2 (4.4) | 3.2 (4.5) | 3.2 (4.3) |
| Median (Q1, Q3) | 1.4 (0.8, 3.4) | 1.3 (0.8, 3.2) | 1.5 (0.8, 3.5) |
| Min - Max | 0.3 - 28.0 | 0.3 - 24.5 | 0.3 - 28.0 |
| Missing | 0 | 0 | 0 |
| chol | |||
| Mean (SD) | 369.5 (231.9) | 368.9 (221.7) | 370.2 (243.2) |
| Median (Q1, Q3) | 309.5 (249.5, 400.0) | 316.0 (254.5, 400.0) | 302.0 (246.0, 396.0) |
| Min - Max | 120.0 - 1775.0 | 120.0 - 1712.0 | 127.0 - 1775.0 |
| Missing | 134 | 62 | 72 |
| albumin | |||
| Mean (SD) | 3.5 (0.4) | 3.5 (0.4) | 3.5 (0.4) |
| Median (Q1, Q3) | 3.5 (3.2, 3.8) | 3.6 (3.2, 3.8) | 3.5 (3.3, 3.8) |
| Min - Max | 2.0 - 4.6 | 2.0 - 4.5 | 2.2 - 4.6 |
| Missing | 0 | 0 | 0 |
| copper | |||
| Mean (SD) | 97.6 (85.6) | 99.0 (81.9) | 96.3 (89.5) |
| Median (Q1, Q3) | 73.0 (41.2, 123.0) | 75.0 (44.0, 129.0) | 67.0 (39.0, 121.0) |
| Min - Max | 4.0 - 588.0 | 4.0 - 558.0 | 11.0 - 588.0 |
| Missing | 108 | 53 | 55 |
| alk.phos | |||
| Mean (SD) | 1982.7 (2140.4) | 2022.6 (2118.2) | 1942.8 (2168.5) |
| Median (Q1, Q3) | 1259.0 (871.5, 1980.0) | 1300.0 (857.8, 2040.5) | 1224.0 (877.5, 1929.2) |
| Min - Max | 289.0 - 13862.4 | 377.0 - 12258.8 | 289.0 - 13862.4 |
| Missing | 106 | 53 | 53 |
| ast | |||
| Mean (SD) | 122.6 (56.7) | 123.3 (56.3) | 121.9 (57.3) |
| Median (Q1, Q3) | 114.7 (80.6, 151.9) | 110.5 (82.2, 153.4) | 116.6 (80.6, 148.0) |
| Min - Max | 26.4 - 457.2 | 41.9 - 338.0 | 26.4 - 457.2 |
| Missing | 106 | 53 | 53 |
| trig | |||
| Mean (SD) | 124.7 (65.1) | 125.4 (68.6) | 123.9 (61.5) |
| Median (Q1, Q3) | 108.0 (84.2, 151.0) | 112.0 (85.0, 149.0) | 106.0 (84.0, 153.0) |
| Min - Max | 33.0 - 598.0 | 44.0 - 598.0 | 33.0 - 382.0 |
| Missing | 136 | 64 | 72 |
| platelet | |||
| Mean (SD) | 257.0 (98.3) | 260.2 (98.6) | 253.8 (98.2) |
| Median (Q1, Q3) | 251.0 (188.5, 318.0) | 256.0 (195.0, 319.0) | 244.0 (181.0, 317.2) |
| Min - Max | 62.0 - 721.0 | 62.0 - 563.0 | 70.0 - 721.0 |
| Missing | 11 | 6 | 5 |
| protime | |||
| Mean (SD) | 10.7 (1.0) | 10.8 (1.1) | 10.7 (0.9) |
| Median (Q1, Q3) | 10.6 (10.0, 11.1) | 10.6 (10.1, 11.1) | 10.6 (10.0, 11.0) |
| Min - Max | 9.0 - 18.0 | 9.3 - 18.0 | 9.0 - 14.1 |
| Missing | 2 | 1 | 1 |
| stage | |||
| 1 | 21 (5.1%) | 9 (4.4%) | 12 (5.8%) |
| 2 | 92 (22.3%) | 47 (22.9%) | 45 (21.7%) |
| 3 | 155 (37.6%) | 67 (32.7%) | 88 (42.5%) |
| 4 | 144 (35.0%) | 82 (40.0%) | 62 (30.0%) |
| Missing | 6 | 4 | 2 |
1.4 Save Table 1 to Word
# convert the table to a data frame, make a flextable of it and save it as a
# Word file in the working directory. The pipe |> passes the result on to the
# next function
flextable::flextable(data.frame(summary(table_one, text=TRUE))) |>
flextable::save_as_docx(path="Descriptives.docx")1.5 Exercises: Part 1
Try each exercise yourself first. Then open the field below it to see the answer, the code and its results. The answers use new object names, so they do not change anything in the tutorial.
Exercise 1. Which variables have the most missing values? Think about what this means for the modelling in Part 3.
Show answer, code and results
Count the missing values per variable, and the percentage of patients with a complete record:
# number of missing values per variable, most missing first
sort(colSums(is.na(data)), decreasing=TRUE)## trig chol copper trt ascites hepato spiders alk.phos
## 136 134 108 106 106 106 106 106
## ast platelet stage protime time status age sex
## 106 11 6 2 0 0 0 0
## edema bili albumin center
## 0 0 0 0# percentage of patients without any missing value
round(100*mean(complete.cases(data)), 1)## [1] 66Several variables are missing for many patients. If you deleted every
patient with at least one missing value (complete case analysis), you
would throw away a large part of the data. In pbc, the
patients who did not take part in the randomized trial (missing
trt) also have many other missing values, so the
missingness is related to other variables. Both are reasons to impute
(Part 3) instead of using complete cases.
Exercise 2. Change center ~ . to
trt ~ . or sex ~ . and look at the new table.
Which would be most useful in a paper?
Show answer, code and results
table_trt <- arsenal::tableby(trt ~ .,
data=dplyr::select(data, -time, -status),
test=FALSE, digits=1)
summary(table_trt, text="html")| D-penicillamine (N=158) | Placebo (N=154) | Total (N=312) | |
|---|---|---|---|
| age | |||
| Mean (SD) | 51.4 (11.0) | 48.6 (10.0) | 50.0 (10.6) |
| Range | 26.3 - 78.4 | 30.6 - 74.5 | 26.3 - 78.4 |
| sex | |||
| Male | 21 (13.3%) | 15 (9.7%) | 36 (11.5%) |
| Female | 137 (86.7%) | 139 (90.3%) | 276 (88.5%) |
| ascites | |||
| No | 144 (91.1%) | 144 (93.5%) | 288 (92.3%) |
| Yes | 14 (8.9%) | 10 (6.5%) | 24 (7.7%) |
| hepato | |||
| No | 85 (53.8%) | 67 (43.5%) | 152 (48.7%) |
| Yes | 73 (46.2%) | 87 (56.5%) | 160 (51.3%) |
| spiders | |||
| No | 113 (71.5%) | 109 (70.8%) | 222 (71.2%) |
| Yes | 45 (28.5%) | 45 (29.2%) | 90 (28.8%) |
| edema | |||
| No edema | 132 (83.5%) | 131 (85.1%) | 263 (84.3%) |
| Untreated or successfully treated | 16 (10.1%) | 13 (8.4%) | 29 (9.3%) |
| Despite diuretics | 10 (6.3%) | 10 (6.5%) | 20 (6.4%) |
| bili | |||
| Mean (SD) | 2.9 (3.6) | 3.6 (5.3) | 3.3 (4.5) |
| Range | 0.3 - 20.0 | 0.3 - 28.0 | 0.3 - 28.0 |
| chol | |||
| N-Miss | 18 | 10 | 28 |
| Mean (SD) | 365.0 (209.5) | 373.9 (252.5) | 369.5 (231.9) |
| Range | 127.0 - 1712.0 | 120.0 - 1775.0 | 120.0 - 1775.0 |
| albumin | |||
| Mean (SD) | 3.5 (0.4) | 3.5 (0.4) | 3.5 (0.4) |
| Range | 2.1 - 4.6 | 2.0 - 4.4 | 2.0 - 4.6 |
| copper | |||
| N-Miss | 1 | 1 | 2 |
| Mean (SD) | 97.6 (90.6) | 97.7 (80.5) | 97.6 (85.6) |
| Range | 9.0 - 588.0 | 4.0 - 558.0 | 4.0 - 588.0 |
| alk.phos | |||
| Mean (SD) | 2021.3 (2183.4) | 1943.0 (2101.7) | 1982.7 (2140.4) |
| Range | 369.0 - 11552.0 | 289.0 - 13862.4 | 289.0 - 13862.4 |
| ast | |||
| Mean (SD) | 120.2 (54.5) | 125.0 (58.9) | 122.6 (56.7) |
| Range | 26.4 - 338.0 | 28.4 - 457.2 | 26.4 - 457.2 |
| trig | |||
| N-Miss | 19 | 11 | 30 |
| Mean (SD) | 124.1 (71.5) | 125.3 (58.5) | 124.7 (65.1) |
| Range | 33.0 - 598.0 | 44.0 - 432.0 | 33.0 - 598.0 |
| platelet | |||
| N-Miss | 2 | 2 | 4 |
| Mean (SD) | 258.8 (100.3) | 265.2 (90.7) | 261.9 (95.6) |
| Range | 62.0 - 563.0 | 71.0 - 487.0 | 62.0 - 563.0 |
| protime | |||
| Mean (SD) | 10.7 (0.9) | 10.8 (1.1) | 10.7 (1.0) |
| Range | 9.0 - 14.1 | 9.2 - 17.1 | 9.0 - 17.1 |
| stage | |||
| 1 | 12 (7.6%) | 4 (2.6%) | 16 (5.1%) |
| 2 | 35 (22.2%) | 32 (20.8%) | 67 (21.5%) |
| 3 | 56 (35.4%) | 64 (41.6%) | 120 (38.5%) |
| 4 | 55 (34.8%) | 54 (35.1%) | 109 (34.9%) |
| center | |||
| Center 1 | 71 (44.9%) | 85 (55.2%) | 156 (50.0%) |
| Center 2 | 87 (55.1%) | 69 (44.8%) | 156 (50.0%) |
Which split is most useful depends on the purpose. In a prediction
model paper, Table 1 usually describes the whole development cohort,
sometimes split by center or by the outcome. A table by treatment helps
to check whether the treatment groups are comparable. Note that patients
with missing trt do not appear in either column.
Exercise 3. Remove "Nmiss2" from
cat.stats and see what changes.
Show answer, code and results
table_no_miss <- arsenal::tableby(center ~ .,
data=dplyr::select(data, -time, -status),
total.pos="before",
digits=1,
test=FALSE,
total=TRUE,
cat.simplify=TRUE,
numeric.stats=c("meansd", "medianq1q3",
"range", "Nmiss2"),
cat.stats=c("countpct"),
stats.labels=list(
meansd="Mean (SD)",
medianq1q3="Median (Q1, Q3)",
range="Min - Max",
Nmiss2="Missing"))
summary(table_no_miss, text="html")| Total (N=418) | Center 1 (N=209) | Center 2 (N=209) | |
|---|---|---|---|
| trt | 154 (49.4%) | 85 (54.5%) | 69 (44.2%) |
| age | |||
| Mean (SD) | 50.7 (10.4) | 50.5 (10.0) | 51.0 (10.9) |
| Median (Q1, Q3) | 51.0 (42.8, 58.2) | 49.6 (43.0, 58.0) | 52.0 (42.8, 58.4) |
| Min - Max | 26.3 - 78.4 | 29.6 - 78.4 | 26.3 - 76.7 |
| Missing | 0 | 0 | 0 |
| sex | 374 (89.5%) | 189 (90.4%) | 185 (88.5%) |
| ascites | 24 (7.7%) | 15 (9.6%) | 9 (5.8%) |
| hepato | 160 (51.3%) | 80 (51.3%) | 80 (51.3%) |
| spiders | 90 (28.8%) | 45 (28.8%) | 45 (28.8%) |
| edema | |||
| No edema | 354 (84.7%) | 172 (82.3%) | 182 (87.1%) |
| Untreated or successfully treated | 44 (10.5%) | 25 (12.0%) | 19 (9.1%) |
| Despite diuretics | 20 (4.8%) | 12 (5.7%) | 8 (3.8%) |
| bili | |||
| Mean (SD) | 3.2 (4.4) | 3.2 (4.5) | 3.2 (4.3) |
| Median (Q1, Q3) | 1.4 (0.8, 3.4) | 1.3 (0.8, 3.2) | 1.5 (0.8, 3.5) |
| Min - Max | 0.3 - 28.0 | 0.3 - 24.5 | 0.3 - 28.0 |
| Missing | 0 | 0 | 0 |
| chol | |||
| Mean (SD) | 369.5 (231.9) | 368.9 (221.7) | 370.2 (243.2) |
| Median (Q1, Q3) | 309.5 (249.5, 400.0) | 316.0 (254.5, 400.0) | 302.0 (246.0, 396.0) |
| Min - Max | 120.0 - 1775.0 | 120.0 - 1712.0 | 127.0 - 1775.0 |
| Missing | 134 | 62 | 72 |
| albumin | |||
| Mean (SD) | 3.5 (0.4) | 3.5 (0.4) | 3.5 (0.4) |
| Median (Q1, Q3) | 3.5 (3.2, 3.8) | 3.6 (3.2, 3.8) | 3.5 (3.3, 3.8) |
| Min - Max | 2.0 - 4.6 | 2.0 - 4.5 | 2.2 - 4.6 |
| Missing | 0 | 0 | 0 |
| copper | |||
| Mean (SD) | 97.6 (85.6) | 99.0 (81.9) | 96.3 (89.5) |
| Median (Q1, Q3) | 73.0 (41.2, 123.0) | 75.0 (44.0, 129.0) | 67.0 (39.0, 121.0) |
| Min - Max | 4.0 - 588.0 | 4.0 - 558.0 | 11.0 - 588.0 |
| Missing | 108 | 53 | 55 |
| alk.phos | |||
| Mean (SD) | 1982.7 (2140.4) | 2022.6 (2118.2) | 1942.8 (2168.5) |
| Median (Q1, Q3) | 1259.0 (871.5, 1980.0) | 1300.0 (857.8, 2040.5) | 1224.0 (877.5, 1929.2) |
| Min - Max | 289.0 - 13862.4 | 377.0 - 12258.8 | 289.0 - 13862.4 |
| Missing | 106 | 53 | 53 |
| ast | |||
| Mean (SD) | 122.6 (56.7) | 123.3 (56.3) | 121.9 (57.3) |
| Median (Q1, Q3) | 114.7 (80.6, 151.9) | 110.5 (82.2, 153.4) | 116.6 (80.6, 148.0) |
| Min - Max | 26.4 - 457.2 | 41.9 - 338.0 | 26.4 - 457.2 |
| Missing | 106 | 53 | 53 |
| trig | |||
| Mean (SD) | 124.7 (65.1) | 125.4 (68.6) | 123.9 (61.5) |
| Median (Q1, Q3) | 108.0 (84.2, 151.0) | 112.0 (85.0, 149.0) | 106.0 (84.0, 153.0) |
| Min - Max | 33.0 - 598.0 | 44.0 - 598.0 | 33.0 - 382.0 |
| Missing | 136 | 64 | 72 |
| platelet | |||
| Mean (SD) | 257.0 (98.3) | 260.2 (98.6) | 253.8 (98.2) |
| Median (Q1, Q3) | 251.0 (188.5, 318.0) | 256.0 (195.0, 319.0) | 244.0 (181.0, 317.2) |
| Min - Max | 62.0 - 721.0 | 62.0 - 563.0 | 70.0 - 721.0 |
| Missing | 11 | 6 | 5 |
| protime | |||
| Mean (SD) | 10.7 (1.0) | 10.8 (1.1) | 10.7 (0.9) |
| Median (Q1, Q3) | 10.6 (10.0, 11.1) | 10.6 (10.1, 11.1) | 10.6 (10.0, 11.0) |
| Min - Max | 9.0 - 18.0 | 9.3 - 18.0 | 9.0 - 14.1 |
| Missing | 2 | 1 | 1 |
| stage | |||
| 1 | 21 (5.1%) | 9 (4.4%) | 12 (5.8%) |
| 2 | 92 (22.3%) | 47 (22.9%) | 45 (21.7%) |
| 3 | 155 (37.6%) | 67 (32.7%) | 88 (42.5%) |
| 4 | 144 (35.0%) | 82 (40.0%) | 62 (30.0%) |
The “Missing” rows of the categorical variables disappear, so you can no longer see how many values are missing for them. Always report missing data in Table 1.
Exercise 4. Open Descriptives.docx in
your folder. Note that you still need to adjust things before you can
put this table in a manuscript. What would you change?
Show answer, code and results
The Word table is created from the same table as above, with generic column names:
names(data.frame(summary(table_one, text=TRUE)))## [1] "Var.1" "Total..N.418." "Center.1..N.209." "Center.2..N.209."Things to check or improve in Word: the column headers (replace the generic names by clear ones), the number of decimals, readable variable labels and units (for example “alk.phos” is not a label for a paper), and add a table caption and a footnote that explains the abbreviations.
2 Describe the survival data
Our outcome is time to death, which is time-to-event data: some patients are followed until they die, others are censored (still alive at last contact). In this part you define the outcome, draw a Kaplan-Meier curve, restrict the outcome to a clinically relevant time horizon, and calculate the median follow-up time correctly.
2.1 Define the outcome
Time-to-event data are the most common type of outcome data in
clinical prediction research, so it is worth learning how to work with
them. (If your own outcome is binary without follow-up time, you can
skip the parts about follow-up.) We create a Surv object
that combines follow-up time and event status. We convert time from days
to years, and dichotomize the event: in pbc the status is
coded 0 = censored, 1 = transplant, 2 = dead, and we model death.
Important: with status==2 we treat a
liver transplant as a censored observation. We do
not treat transplant as a competing risk. This is a
simplification: censoring at transplant assumes that patients who
receive a transplant would otherwise have the same risk of death as
those who remain in follow-up, which is unlikely to be true. If
transplant is an important competing event in your own project, use a
competing risk approach (for example cause-specific hazards or
Fine-Gray).
# generate the Surv object for the outcome: it combines follow-up time and event
# time: transform the time from days to years (divide by 365)
# event: status==2 is TRUE for patients who died and FALSE for the others;
# as.numeric() turns TRUE/FALSE into 1/0
# (transplant is treated as censored, NOT as a competing risk)
S <- survival::Surv(time=data$time/365, event=as.numeric(data$status==2))2.2 Kaplan-Meier curve
The Kaplan-Meier method estimates survival over time while correctly
handling censoring. We make one curve per treatment group, here for all
follow-up (unrestricted, S).
# make Kaplan-Meier curve (all follow-up)
# the formula S ~ trt means: one survival curve for each treatment group
KM_unrestricted <- survival::survfit(S ~ trt, data=data)Note: some patients are silently dropped.
trt is missing for some patients (those not in the
randomized trial), and survfit() removes them without a
warning. Check how many:
# number of patients with missing treatment, these are not in the curves
sum(is.na(data$trt))## [1] 106Only the remaining patients are used in the curves. In the prediction models in Part 4 we do keep everyone, because missing values are imputed.
First look at the curve with the basic plot()
function:
# basic Kaplan-Meier curve: one color per treatment group, with axis labels
plot(KM_unrestricted, col=c("blue", "red"), xlab="Time (years)", ylab="Survival")
For a nicer figure with confidence intervals and a risk table we use
ggsurvplot() from survminer.
# Kaplan-Meier curve with risk table (all follow-up)
KM_plot_unrestricted <- survminer::ggsurvplot(KM_unrestricted,
data=data,
risk.table=TRUE, # table below the curve
risk.table.y.text=FALSE, # no group names in the table
legend="top",
legend.title="Treatment",
legend.labs=levels(data$trt), # names of the groups
xlab="Time (years)",
conf.int=TRUE) # confidence intervals
# typing the name of an object prints it
KM_plot_unrestricted
Two details in the code make the figure tidier.
legend.labs=levels(data$trt) replaces the default labels
(trt=D-penicillamine, trt=Placebo) of the two
treatment arms in the legend by the factor labels we set earlier; to use
other names, give your own vector, in the same order as the levels, for
example legend.labs=c("Penicillamine", "Control"). And
risk.table.y.text=FALSE removes the long group names from
the y-axis of the risk table (the colors in the table match the legend),
so that the y-axes of the curve and the table line up.
Look at the right end of the curves: the confidence intervals become wider, and the risk table shows why: fewer and fewer patients are still followed after about 10 years, so the estimates there are less certain. A prediction model should not rely much on this part of the follow-up. Moreover, a prediction horizon of 10 years may be more clinically useful than one of 12 years. That is why we truncate the outcome at 10 years in the next section; for the purpose of this exercise we use 10 years as the horizon.
2.3 Limit the outcome to the time horizon
A prediction model predicts risk at a specific time point. We choose 10 years. Everyone followed longer than 10 years is censored at 10 years, because events after the horizon are not part of the outcome we predict.
# limit the outcome to your time horizon
horizon <- 10
S_10 <- S # start from a copy of the full outcome
# S_10[, 1] is the time column and S_10[, 2] is the event column;
# S_10[, 1] > horizon is TRUE for everyone followed longer than 10 years
S_10[S_10[, 1]>horizon, 2] <- 0 # for them: no event (censored)
S_10[S_10[, 1]>horizon, 1] <- horizon # and the time becomes 10 yearsNow fit the Kaplan-Meier curve for the restricted outcome
S_10 and make the same figure as before:
# the same Kaplan-Meier fit, now for the outcome restricted to 10 years
KM_10 <- survival::survfit(S_10 ~ trt, data=data)
# Kaplan-Meier curve with risk table, restricted to 10 years
KM_plot <- survminer::ggsurvplot(KM_10,
data=data,
risk.table=TRUE,
risk.table.y.text=FALSE,
legend="top",
legend.title="Treatment",
legend.labs=levels(data$trt),
xlab="Time (years)",
conf.int=TRUE)
KM_plot
# combine the curve and the risk table and save
ggplot2::ggsave(filename="KM_plot_10_years.png",
plot=ggpubr::ggarrange(KM_plot$plot, KM_plot$table,
nrow=2, heights=c(3, 1), align="v"),
width=7, height=7, dpi=300)Now display the Kaplan-Meier estimates at the horizon, for the unrestricted and the restricted outcome:
# display the Kaplan-Meier estimates (survival probability) at the horizon,
# time=horizon asks for the estimate at 10 years only
summary(KM_unrestricted, time=horizon)## Call: survfit(formula = S ~ trt, data = data)
##
## 106 observations deleted due to missingness
## trt=D-penicillamine
## time n.risk n.event survival std.err lower 95% CI
## 10.0000 16.0000 63.0000 0.4247 0.0603 0.3216
## upper 95% CI
## 0.5611
##
## trt=Placebo
## time n.risk n.event survival std.err lower 95% CI
## 10.0000 16.0000 57.0000 0.4575 0.0612 0.3520
## upper 95% CI
## 0.5946summary(KM_10, time=horizon)## Call: survfit(formula = S_10 ~ trt, data = data)
##
## 106 observations deleted due to missingness
## trt=D-penicillamine
## time n.risk n.event survival std.err lower 95% CI
## 10.0000 16.0000 63.0000 0.4247 0.0603 0.3216
## upper 95% CI
## 0.5611
##
## trt=Placebo
## time n.risk n.event survival std.err lower 95% CI
## 10.0000 16.0000 57.0000 0.4575 0.0612 0.3520
## upper 95% CI
## 0.59462.4 Median follow-up time
The median of the observed times is not the median follow-up
time, because patients who died have a short observed time and drag the
median down. The correct estimate uses the reverse Kaplan-Meier
method: we flip the roles of event and censoring, so the “event” is
being censored and we estimate how long patients are followed in a
typical situation. In survfit() this means that the event
indicator becomes 1 - status.
First plot the reverse Kaplan-Meier curve. It shows the probability that a patient is still being followed at each time point; the time at which this curve crosses 0.5 is the median follow-up time.
# reverse Kaplan-Meier: the "event" is being censored (not dying)
# S[, "time"] is the time column of S; 1 - S[, "status"] flips 1 and 0, so
# that being censored counts as the event
reverse_KM <- survival::survfit(survival::Surv(S[, "time"], 1 - S[, "status"]) ~ 1,
data=data, conf.type="plain")
survminer::ggsurvplot(reverse_KM,
data=data,
conf.int=TRUE,
legend="none",
xlab="Time (years)",
ylab="Probability of still being followed")
Then calculate the median follow-up time from this curve:
# median follow-up time with the reverse Kaplan-Meier method (in years)
# probs=0.5 asks for the 50th percentile, the median
stats::quantile(reverse_KM, probs=0.5)## $quantile
## 50
## 6.479452
##
## $lower
## 50
## 6.093151
##
## $upper
## 50
## 7.021918The result gives the median follow-up time (with
probs=0.5) with its confidence interval. We use
conf.type="plain" in survfit() to get the same
confidence interval as prodlim below; the default of
survfit() is conf.type="log", which gives a
slightly different lower limit.
The prodlim package does the same in one line: with
reverse=TRUE it flips event and censoring for you, so you
do not have to define 1 - status yourself. Install it once
with install.packages("prodlim") if you do not have it yet.
The row for 0.5 gives the same median follow-up time and
confidence interval. Do not confuse this interval with the line “Median
with interquartile range (IQR)” below the table: that is the median
together with the 25th and 75th percentile of the follow-up times (the
rows for 0.75 and 0.25), which describes how
much the follow-up varies between patients. It is not a confidence
interval of the median.
# the same in one line with prodlim
stats::quantile(prodlim::prodlim(prodlim::Hist(time/365, status==2) ~ 1,
data=data, reverse=TRUE))## Quantiles of the potential follow up time distribution based on the Kaplan-Meier method
## applied to the censored times reversing the roles of event status and censored.
##
## Table of quantiles and corresponding confidence limits:
## q quantile lower upper
## <num> <num> <num> <num>
## 1: 0.00 NA NA NA
## 2: 0.25 9.1 8.4 9.8
## 3: 0.50 6.5 6.1 7.0
## 4: 0.75 4.4 3.9 4.8
## 5: 1.00 1.5 1.5 2.0
##
##
## Median with interquartile range (IQR):
## Median (IQR)
## <char>
## 1: 6.48 (4.36;9.14)2.5 Restricted mean survival time (RMST)
The restricted mean survival time (RMST) is the area under the Kaplan-Meier curve up to a chosen time point. It is the average number of years a patient is alive during that period, for example during the first 10 years. For a quick guide, see Han & Jung (2022).
A hazard ratio is a relative measure that assumes the effect is constant over time (proportional hazards) and is hard to translate into an absolute effect. The RMST is in the unit of time (years gained or lost), so it is easy to interpret, it does not need the proportional hazards assumption, and it can still show a difference when the survival curves cross. The difference in RMST between two groups (the area between their Kaplan-Meier curves) also gives a better picture of the absolute treatment benefit than the risk difference at a single time point, which can lead to treating the wrong patients. For that reason it is recommended for assessing absolute treatment effects in time-to-event data (Maas et al., 2025).
You can get it from the summary() of the Kaplan-Meier
fit by setting rmean to the time point. The result contains
a table with the RMST (rmean) and its standard
error (se(rmean)) per group. We use the outcome restricted
to the 10-year horizon:
# restricted mean survival time up to the horizon (10 years)
# $table takes the table with the RMST out of the summary
summary(KM_10, rmean=horizon)$table## records n.max n.start events rmean se(rmean) median
## trt=D-penicillamine 158 158 158 63 7.148479 0.2827057 8.991781
## trt=Placebo 154 154 154 57 7.285271 0.2954189 9.391781
## 0.95LCL 0.95UCL
## trt=D-penicillamine 7.076712 NA
## trt=Placebo 8.465753 NA# extract the RMST (and its standard error) for one treatment arm:
# [row name, column names] picks one row and two columns of the table
summary(KM_10, rmean=horizon)$table["trt=Placebo", c("rmean", "se(rmean)")]## rmean se(rmean)
## 7.2852709 0.2954189For example, an rmean of 7.1 means that patients in that
group live on average 7.1 of the first 10 years. The RMST needs a time
point to restrict at, otherwise R restricts at the largest observed
time. This is another reason to limit the outcome to the horizon.
2.6 Exercises: Part 2
Try each exercise yourself first. Then open the field below it to see the answer, the code and its results.
Exercise 1. What are the Kaplan-Meier estimates at 5 years? How do they compare with the estimates at the 10-year horizon?
Show answer, code and results
A Kaplan-Meier estimate at 5 years does not depend on follow-up after
5 years, so you do not need to define a new outcome. Use
summary() of the fit with S_10 and set
time=5:
# Kaplan-Meier estimates at 5 years
summary(KM_10, time=5)## Call: survfit(formula = S_10 ~ trt, data = data)
##
## 106 observations deleted due to missingness
## trt=D-penicillamine
## time n.risk n.event survival std.err lower 95% CI
## 5.0000 82.0000 43.0000 0.7077 0.0379 0.6371
## upper 95% CI
## 0.7861
##
## trt=Placebo
## time n.risk n.event survival std.err lower 95% CI
## 5.0000 77.0000 42.0000 0.7146 0.0376 0.6445
## upper 95% CI
## 0.7923Compare these estimates with the 10-year estimates above. A Kaplan-Meier curve can only go down over time, so the survival estimate at 5 years is at least as high as the estimate at 10 years.
Exercise 2. Calculate the naive median,
median(data$time)/365, and compare it with the reverse
Kaplan-Meier estimate. Which one would you report, and why?
Show answer, code and results
# naive median of the observed times (in years)
median(data$time)/365## [1] 4.739726# median follow-up with the reverse Kaplan-Meier method (in years)
stats::quantile(reverse_KM, probs=0.5)## $quantile
## 50
## 6.479452
##
## $lower
## 50
## 6.093151
##
## $upper
## 50
## 7.021918Report the reverse Kaplan-Meier estimate. The naive median mixes patients who were followed until the end of the study with patients whose follow-up stopped early because they died, which makes the typical follow-up look shorter than it is.
Exercise 3. Calculate the restricted mean survival time (RMST) up to 5 years instead of 10 years for each treatment group, and extract the RMST of the placebo arm only. How many of the first 5 years do patients on average live?
Show answer, code and results
# RMST up to 5 years for both treatment arms
summary(KM_10, rmean=5)$table## records n.max n.start events rmean se(rmean) median
## trt=D-penicillamine 158 158 158 63 4.30216 0.1060121 8.991781
## trt=Placebo 154 154 154 57 4.18246 0.1190897 9.391781
## 0.95LCL 0.95UCL
## trt=D-penicillamine 7.076712 NA
## trt=Placebo 8.465753 NA# extract the RMST (and its standard error) of the placebo arm only
summary(KM_10, rmean=5)$table["trt=Placebo", c("rmean", "se(rmean)")]## rmean se(rmean)
## 4.1824596 0.1190897The fit KM_10 covers 10 years, but rmean=5
restricts the area under the curve to the first 5 years, so you do not
need a new fit. The RMST up to 5 years is at most 5, and smaller than
the RMST up to 10 years.
Exercise 4. Why do we truncate the outcome at the horizon instead of just using all follow-up?
Show answer, code and results
# how many patients are followed for longer than the 10-year horizon?
table(S[, 1] > horizon)##
## FALSE TRUE
## 383 35A prediction model gives the risk at a specific time point, here 10 years, and the baseline hazard is estimated at that time point. Later follow-up is based on few patients at risk and would not be part of the risk that we predict. Truncating at the horizon keeps the outcome consistent with the prediction.
3 Multiple imputation
Most clinical data sets have missing values. Deleting every patient with a missing value (complete case analysis) wastes data and can bias the model. Multiple imputation fills in missing values several times, using the observed data, so that the uncertainty due to missingness is reflected in the results. In this part you visualize the missing data, build an imputation model that suits survival data, and check the imputed values.
3.1 Visualize the missing data
Before imputing, look at how much is missing and where.
# visualize missingness: every column is a variable, every row a patient,
# and the black cells are missing values
visdat::vis_miss(data)
3.2 Build the imputation model for survival data
The imputation model must contain the outcome, otherwise the imputed
predictors are unrelated to the outcome and the model’s effects are
biased towards zero. For time-to-event data, it is not
a good idea to put time and status in the
imputation model directly. The recommended approach (White &
Royston, 2009) is to include:
- the event indicator, and
- the Nelson-Aalen estimate of the cumulative hazard at each patient’s observed time.
We calculate both on the same outcome that we model later: death
within the 10-year horizon (S_10), with transplant treated
as censored. The raw time and status variables
are then left out of the imputation model.
# outcome information for the imputation model, based on the analysis outcome S_10
data_mi <- data # work on a copy, so that `data` stays as it is
data_mi$time <- NULL # NULL removes a column
data_mi$status <- NULL
# 1. the event indicator (0/1) of the outcome within 10 years
data_mi$event_10 <- S_10[, "status"]
# 2. the Nelson-Aalen estimate of the cumulative hazard at each patient's time
data_mi$cum_hazard <- mice::nelsonaalen(data.frame(time=S_10[, "time"],
status=S_10[, "status"]),
time, status)3.3 Impute
We create m = 5 imputed data sets. seed=1
makes the result reproducible.
Note: five imputations is on the small side. A rule of thumb is to use at least as many imputations as the percentage of incomplete cases (White et al., 2011). Let’s check that percentage here:
# percentage of patients with at least one missing value
round(100*mean(!complete.cases(data_mi)), 1)## [1] 34With this percentage, m should probably be larger than 5
(maybe 40 or more). For this tutorial we keep m = 5 so that
all following steps run quickly, but in your own project you should use
a larger m based on the fraction of incomplete cases.
# multiple imputation (m should maybe be larger, see the text above)
m <- 5 # number of imputed data sets
# mice() imputes the missing values: m=number of data sets, seed=1 makes the
# result reproducible, print=FALSE hides the progress messages
imputed_data <- mice::mice(data_mi, m=m, seed=1, print=FALSE)
# problems that mice ran into (NULL means none)
imputed_data$loggedEvents## NULLloggedEvents lists problems that mice ran
into; NULL means none.
3.4 Fraction of missing information (FMI)
The fraction of missing information (FMI) is the part of the
uncertainty (variance) of an estimate that is due to missing data. It
can be much lower or higher than the percentage of missing values,
because it depends on how well the other variables predict the missing
ones. We calculate it per variable: fit an intercept-only model on every
imputed data set (its estimate is the mean, or the log odds for a binary
variable), pool the results with Rubin’s rules using
pool(), and take the FMI from the pooled result. We skip
categorical variables with more than 2 levels, because a single
intercept does not describe them.
To do this for every variable we need two basic tools: a function and a for loop.
A function is a piece of code that you give a name, so that you can run it again and again with different input. For example, a function that squares a number:
# define a function: x is the input, the last line is the output
square <- function(x){
x^2
}
square(3)## [1] 9square(10)## [1] 100A for loop repeats code for every element of a vector. For example, print the square of each number in a vector:
# the loop runs once for every value of `number`
for (number in c(2, 3, 4)){
print(square(number))
}## [1] 4
## [1] 9
## [1] 16Now we write a function that calculates the FMI for one variable, and use a for loop to run it for every variable with missing values.
# ---- 1. Define the function -------------------------------------------------
# A function is defined with function(argument){ ... }. Here the argument `var`
# is the name of one variable (for example "chol"), and the function returns a
# small table with the percentage missing and the FMI of that variable.
calculate_fmi <- function(var){
# data_mi[[var]] is the column with the name stored in `var`.
# is.na() gives TRUE for missing values, mean() of TRUE/FALSE gives the
# proportion of TRUE, and 100* turns it into a percentage (rounded to 1 decimal)
pct_missing <- round(100*mean(is.na(data_mi[[var]])), 1)
# We fit a model with only an intercept ("var ~ 1") on each of the m imputed
# data sets. with(imputed_data, ...) runs the model on every imputed data set.
# paste(var, "~ 1") builds the formula as text, for example "chol ~ 1", and
# as.formula() turns the text into a formula.
if (is.factor(data_mi[[var]]) && nlevels(data_mi[[var]]) == 2){
# binary variable: logistic regression (the intercept is the log odds)
fit <- with(imputed_data, glm(as.formula(paste(var, "~ 1")), family=binomial))
} else if (is.numeric(data_mi[[var]])){
# continuous variable: linear regression (the intercept is the mean)
fit <- with(imputed_data, lm(as.formula(paste(var, "~ 1"))))
} else {
# other variables (categorical with more than 2 levels): no FMI, so we
# return a row with NA and stop here (return() ends the function)
return(data.frame(variable=var, pct_missing=pct_missing, fmi=NA))
}
# pool() combines the m models into one estimate with Rubin's rules. The
# pooled result contains the fraction of missing information (`fmi`), a
# proportion between 0 and 1; we show it as a percentage.
pooled <- mice::pool(fit)
# The last line of a function is its result: a one-row table for this variable
data.frame(variable=var, pct_missing=pct_missing,
fmi=round(100*pooled$pooled$fmi, 1))
}
# ---- 2. Use the function in a for loop ----------------------------------------
# colSums(is.na(data_mi)) counts the missing values per column; we keep the names
# of the columns that have at least one missing value.
vars_missing <- names(data_mi)[colSums(is.na(data_mi)) > 0]
# Start with an empty table and add one row per variable. In every round of the
# loop `var` is the next name in `vars_missing`; rbind() puts the new row under
# the rows that we already have.
fmi_results <- data.frame()
for (var in vars_missing){
fmi_results <- rbind(fmi_results, calculate_fmi(var))
}
# print the table
fmi_resultsA high FMI means that the missing data add a lot of uncertainty and
that more imputations are needed to estimate it stably. A common rule of
thumb is that m should be at least 100 times the largest
FMI (as a proportion, so an FMI of 0.3 gives m = 30) (White
et al., 2011).
3.5 Check the imputations
Compare the distribution of observed values (blue) with the imputed values (red) for each imputation. They do not need to be identical (missingness may depend on other variables), but strange differences deserve a closer look.
# visualize imputations
mice::densityplot(imputed_data)
3.6 Exercises: Part 3
Try each exercise yourself first. Then open the field below it to see the answer, the code and its results.
Exercise 1. Which variables have the most missing values, and what percentage of patients have at least one missing value?
Show answer, code and results
# percentage missing per variable, most missing first
round(100*sort(colMeans(is.na(data)), decreasing=TRUE), 1)## trig chol copper trt ascites hepato spiders alk.phos
## 32.5 32.1 25.8 25.4 25.4 25.4 25.4 25.4
## ast platelet stage protime time status age sex
## 25.4 2.6 1.4 0.5 0.0 0.0 0.0 0.0
## edema bili albumin center
## 0.0 0.0 0.0 0.0# percentage of patients with at least one missing value
round(100*mean(!complete.cases(data_mi)), 1)## [1] 34The first table gives the percentage of missing values per variable, the second number the percentage of patients with at least one missing value. The second number is the one used for the rule of thumb for the number of imputations.
Exercise 2. Look at the density plots. Do the
imputed values (red) look plausible compared with the observed values
(blue) for chol, trig and
copper?
Show answer, code and results
mice::densityplot(imputed_data, ~ chol)
mice::densityplot(imputed_data, ~ trig)
mice::densityplot(imputed_data, ~ copper)
Compare the shape of the red lines (imputed) with the blue line
(observed). These variables are right-skewed, and the imputed values do
not need to match the observed values exactly: if missingness depends on
other variables, such as being in the trial or not, the imputed values
may legitimately differ. By default mice uses predictive
mean matching for continuous variables, which only imputes values that
occur in the observed data, so impossible values (for example negative
cholesterol) cannot appear. Be suspicious if the imputed distribution is
completely different or has a strange shape.
Exercise 3. Increase m to the
rule-of-thumb value and re-run. How long does it take, and do the
density plots change?
Show answer, code and results
This can take a minute or two. The new imputations are stored in
imputed_data_big, so the tutorial keeps using
m = 5.
# rule of thumb: at least as many imputations as the percentage of incomplete cases
m_rule <- ceiling(100*mean(!complete.cases(data_mi)))
m_rule## [1] 34# impute again with more imputations and time it
time_big <- system.time(
imputed_data_big <- mice::mice(data_mi, m=m_rule, seed=1, print=FALSE)
)
time_big["elapsed"]## elapsed
## 18.85# density plot with more imputations
mice::densityplot(imputed_data_big, ~ chol)
The time increases roughly in proportion to the number of imputations. The density plots look similar (more red lines), because the imputation model is the same. The benefit of more imputations is more stable estimates and standard errors of the final model, not a different imputation distribution.
Exercise 4. Suppose the outcome is missing for some
patients (for example, we lost track of their vital status). Should you
delete these patients before imputation? Show how to impute with the
outcome in the imputation model and delete the patients with a missing
outcome afterwards. To mimic this, set event_10 to missing
for every 20th patient.
Show answer, code and results
Do not delete them before imputation: these patients still have observed predictor values that help to impute the missing predictors of other patients. Instead (1) first create a variable that flags who has a missing outcome, (2) impute with the outcome in the imputation model, and (3) delete the flagged rows from every imputed data set afterwards. This is called multiple imputation, then deletion (von Hippel, 2007).
# mimic missing outcomes: set event_10 to missing for every 20th patient
data_mi_out <- data_mi
data_mi_out$event_10[seq(1, nrow(data_mi_out), by=20)] <- NA
# 1. flag the patients with a missing outcome BEFORE imputation
data_mi_out$outcome_missing <- is.na(data_mi_out$event_10)
# 2. impute, with the outcome in the imputation model
# (the flag itself is not used to impute other variables)
pred_matrix <- mice::make.predictorMatrix(data_mi_out)
pred_matrix[, "outcome_missing"] <- 0
imputed_out <- mice::mice(data_mi_out, m=5, predictorMatrix=pred_matrix,
seed=1, print=FALSE)
# 3. after imputation: delete the rows with a missing outcome in each imputed data set
imputed_sets <- mice::complete(imputed_out, "all")
for (i in seq_along(imputed_sets)){
imputed_sets[[i]] <- imputed_sets[[i]][!imputed_sets[[i]]$outcome_missing, ]
}
sapply(imputed_sets, nrow)## 1 2 3 4 5
## 397 397 397 397 397Each imputed data set now has the patients with an observed outcome only, and the predictors of these patients were imputed using all patients. The imputed outcomes themselves are not analysed. This approach is for a missing outcome only; always keep the outcome in the imputation model.
4 Fit the models
In this part you fit a Cox proportional hazards model on the imputed
data, check whether continuous predictors need a non-linear effect
(using rcs()), select predictors with backward selection,
and present the final model (effects, variable importance and a
nomogram). cph (from rms) fits a Cox model
with extra tools for prediction models, and fit.mult.impute
fits the model on every imputed data set and pools the results with
Rubin’s rules.
4.1 Set data distributions
datadist stores the ranges of the predictors, which
rms needs for summaries, plots and nomograms.
# set data distributions: datadist() stores the ranges of all predictors, and
# options() tells rms to use it (it looks for the object called "dd")
dd <- rms::datadist(data)
options(datadist="dd")4.2 Non-linear effects
with rcs()
Continuous predictors such as age, bilirubin (bili) and
albumin are often assumed to have a linear effect: every extra
year or unit changes the log hazard by the same amount. That is rarely
exactly true. A flexible way to model a non-linear effect is a
restricted cubic spline, rcs(x, k) from
rms:
kis the number of knots, the points where the curve is allowed to bend. With 3 knots, they are placed at the 10th, 50th and 90th percentile of the variable (with more knots, at other percentiles). Use 3 to 5 knots: more knots make the curve more flexible, but need more parameters and can overfit.rcs(x, 3)uses 2 coefficients: one forxitself (the linear part) and one forx'(the non-linear part).rcs(x, 4)uses 3, and so on. A linear effect uses 1.- The curve is restricted to be linear before the first and after the last knot, which keeps it stable in the tails.
We first model all continuous predictors flexibly with
rcs(..., 3) and test whether the non-linear part is
significant. Fitting flexible effects for all continuous variables is a
good way to see if linearity is a reasonable assumption.
# fit full model with all candidate variables on each imputed data set
# model all continuous variables flexible to see if that is needed
# fit.mult.impute() arguments:
# - the formula: outcome ~ predictor 1 + predictor 2 + ...
# - cph: the function that fits the Cox model on each imputed data set
# - fitargs: extra arguments for cph (keep x, y and the survival estimates)
# - xtrans: the imputed data sets
# - data: the original data, with the missing values
# - fit.reps=TRUE: keep the model of every imputed data set
full_non_linear_model <- Hmisc::fit.mult.impute(S_10 ~ rcs(age, 3) +
rcs(bili, 3) +
rcs(albumin, 3) + sex +
ascites + hepato + stage,
cph,
fitargs=list(x=TRUE, y=TRUE, surv=TRUE),
xtrans=imputed_data,
data=data,
fit.reps=TRUE)##
## Wald Statistic Information
##
## Variance Inflation Factors Due to Imputation:
##
## age age' bili bili' albumin albumin'
## 1.01 1.01 1.01 1.00 1.00 1.01
## sex=Female ascites=Yes hepato=Yes stage=2 stage=3 stage=4
## 1.00 1.15 1.09 1.09 1.08 1.09
##
## Fraction of Missing Information:
##
## age age' bili bili' albumin albumin'
## 0.01 0.01 0.01 0.00 0.00 0.01
## sex=Female ascites=Yes hepato=Yes stage=2 stage=3 stage=4
## 0.00 0.13 0.08 0.08 0.07 0.08
##
## d.f. for t-distribution for Tests of Single Coefficients:
##
## age age' bili bili' albumin albumin'
## 70992.09 29693.89 106704.51 199948.55 205552.84 32084.48
## sex=Female ascites=Yes hepato=Yes stage=2 stage=3 stage=4
## 358897.26 242.90 630.61 651.50 799.20 602.05
##
## The following fit components were averaged over the 5 model fits:
##
## linear.predictors means stats centersummary(full_non_linear_model)## Effects Response : S_10
##
## Factor Low High Diff. Effect S.E. Lower 0.95
## age 42.8320 58.241 15.4090 0.556350 0.15466 0.2532300
## Hazard Ratio 42.8320 58.241 15.4090 1.744300 NA 1.2882000
## bili 0.8000 3.400 2.6000 1.495800 0.20069 1.1025000
## Hazard Ratio 0.8000 3.400 2.6000 4.462900 NA 3.0116000
## albumin 3.2425 3.770 0.5275 -0.360650 0.12539 -0.6064000
## Hazard Ratio 3.2425 3.770 0.5275 0.697220 NA 0.5453100
## sex - Male:Female 2.0000 1.000 NA 0.081707 0.25309 -0.4143400
## Hazard Ratio 2.0000 1.000 NA 1.085100 NA 0.6607700
## ascites - Yes:No 1.0000 2.000 NA 0.438920 0.26828 -0.0868970
## Hazard Ratio 1.0000 2.000 NA 1.551000 NA 0.9167700
## hepato - No:Yes 2.0000 1.000 NA -0.087502 0.22035 -0.5193800
## Hazard Ratio 2.0000 1.000 NA 0.916220 NA 0.5948900
## stage - 1:3 3.0000 1.000 NA -0.730420 0.73833 -2.1775000
## Hazard Ratio 3.0000 1.000 NA 0.481710 NA 0.1133200
## stage - 2:3 3.0000 2.000 NA -0.482070 0.27063 -1.0125000
## Hazard Ratio 3.0000 2.000 NA 0.617500 NA 0.3633100
## stage - 4:3 3.0000 4.000 NA 0.415320 0.21663 -0.0092709
## Hazard Ratio 3.0000 4.000 NA 1.514900 NA 0.9907700
## Upper 0.95
## 0.859470
## 2.361900
## 1.889100
## 6.613600
## -0.114890
## 0.891460
## 0.577760
## 1.782000
## 0.964730
## 2.624100
## 0.344380
## 1.411100
## 0.716690
## 2.047600
## 0.048357
## 1.049500
## 0.839920
## 2.316200The anova table displays chi-square tests for
non-linearity. Look at the row “Nonlinear” under each continuous
predictor: a small p-value means that the effect is not a straight
line.
# display chi-square tests for non-linearities
anova(full_non_linear_model)## Wald Statistics Response: S_10
##
## Factor Chi-Square d.f. P
## age 13.11 2 0.0014
## Nonlinear 3.51 1 0.0609
## bili 95.92 2 <.0001
## Nonlinear 35.10 1 <.0001
## albumin 11.80 2 0.0027
## Nonlinear 0.12 1 0.7263
## sex 0.10 1 0.7468
## ascites 2.68 1 0.1018
## hepato 0.16 1 0.6913
## stage 11.06 3 0.0114
## TOTAL NONLINEAR 36.64 3 <.0001
## TOTAL 215.30 12 <.0001Here bilirubin has a clearly non-linear effect, age is borderline, and albumin is close to linear. Plot the estimated effects to see what this looks like:
# display predictor effects (log hazard) for the flexible model
plot(Predict(full_non_linear_model))
The curve for bilirubin rises steeply at low values and flattens out
at high values, which a straight line would miss. Based on these results
we keep rcs() for age and bilirubin and a linear effect for
albumin in the full model.
4.3 Fit the full model
Now we fit the full model with rcs() for age and
bilirubin, a linear effect for albumin, and the other candidate
predictors. This model is the starting point for variable selection.
# fit full model with all candidate variables on each imputed data set
# we store the formula in an object, so that we can reuse it later (Tutorial 2)
# rcs(age, 3): spline for age; albumin without rcs(): a straight line
form_full <- S_10 ~ rcs(age, 3) + rcs(bili, 3) + albumin + sex +
ascites + hepato + stage
full_model <- Hmisc::fit.mult.impute(form_full,
cph,
fitargs=list(x=TRUE, y=TRUE, surv=TRUE),
xtrans=imputed_data,
data=data,
fit.reps=TRUE)##
## Wald Statistic Information
##
## Variance Inflation Factors Due to Imputation:
##
## age age' bili bili' albumin sex=Female
## 1.01 1.01 1.01 1.00 1.00 1.00
## ascites=Yes hepato=Yes stage=2 stage=3 stage=4
## 1.14 1.10 1.08 1.08 1.09
##
## Fraction of Missing Information:
##
## age age' bili bili' albumin sex=Female
## 0.01 0.01 0.01 0.00 0.00 0.00
## ascites=Yes hepato=Yes stage=2 stage=3 stage=4
## 0.13 0.09 0.08 0.07 0.08
##
## d.f. for t-distribution for Tests of Single Coefficients:
##
## age age' bili bili' albumin sex=Female
## 66673.51 25006.17 103533.09 203798.25 31013743.69 337198.38
## ascites=Yes hepato=Yes stage=2 stage=3 stage=4
## 251.90 492.12 655.56 784.71 576.02
##
## The following fit components were averaged over the 5 model fits:
##
## linear.predictors means stats center# display predictor effects (hazard ratios, comparing the 25th and 75th percentile)
summary(full_model)## Effects Response : S_10
##
## Factor Low High Diff. Effect S.E. Lower 0.95
## age 42.8320 58.241 15.4090 0.557540 0.15483 0.254070
## Hazard Ratio 42.8320 58.241 15.4090 1.746400 NA 1.289300
## bili 0.8000 3.400 2.6000 1.502000 0.20048 1.109100
## Hazard Ratio 0.8000 3.400 2.6000 4.490800 NA 3.031600
## albumin 3.2425 3.770 0.5275 -0.380950 0.11312 -0.602660
## Hazard Ratio 3.2425 3.770 0.5275 0.683210 NA 0.547360
## sex - Male:Female 2.0000 1.000 NA 0.084776 0.25332 -0.411730
## Hazard Ratio 2.0000 1.000 NA 1.088500 NA 0.662500
## ascites - Yes:No 1.0000 2.000 NA 0.453960 0.26293 -0.061373
## Hazard Ratio 1.0000 2.000 NA 1.574500 NA 0.940470
## hepato - No:Yes 2.0000 1.000 NA -0.081244 0.22032 -0.513070
## Hazard Ratio 2.0000 1.000 NA 0.921970 NA 0.598660
## stage - 1:3 3.0000 1.000 NA -0.731010 0.73871 -2.178800
## Hazard Ratio 3.0000 1.000 NA 0.481420 NA 0.113170
## stage - 2:3 3.0000 2.000 NA -0.483820 0.27056 -1.014100
## Hazard Ratio 3.0000 2.000 NA 0.616420 NA 0.362720
## stage - 4:3 3.0000 4.000 NA 0.412780 0.21622 -0.010994
## Hazard Ratio 3.0000 4.000 NA 1.511000 NA 0.989070
## Upper 0.95
## 0.86101
## 2.36550
## 1.89500
## 6.65230
## -0.15924
## 0.85279
## 0.58128
## 1.78830
## 0.96929
## 2.63610
## 0.35058
## 1.41990
## 0.71683
## 2.04790
## 0.04647
## 1.04760
## 0.83656
## 2.30840# display chi-square tests for variable importance: a larger chi-square
# (relative to its degrees of freedom) means a more important predictor
anova(full_model)## Wald Statistics Response: S_10
##
## Factor Chi-Square d.f. P
## age 13.18 2 0.0014
## Nonlinear 3.70 1 0.0546
## bili 95.81 2 <.0001
## Nonlinear 35.72 1 <.0001
## albumin 11.34 1 0.0008
## sex 0.11 1 0.7379
## ascites 2.98 1 0.0842
## hepato 0.14 1 0.7123
## stage 11.03 3 0.0116
## TOTAL NONLINEAR 36.34 2 <.0001
## TOTAL 214.11 11 <.0001We check the proportional hazards assumption, which says that the effect of a predictor (the hazard ratio) is constant over time. A small p-value for a predictor suggests a violation.
# test proportional hazards assumption: a small p-value suggests that the
# effect of that predictor changes over time
survival::cox.zph(full_model)## chisq df p
## rcs(age, 3) 0.5573 66674 1
## rcs(bili, 3) 0.4378 25006 1
## albumin 0.6281 103533 1
## sex 0.5998 203798 1
## ascites 2.2708 31013744 1
## hepato 0.0879 337198 1
## stage 4.3057 252 1
## GLOBAL 6.8567 31752713 1# display predictor effects: Predict() calculates the effect of each predictor
# on the log hazard (others held at their median), plot() draws it
plot(Predict(full_model))
4.4 Backward variable selection
We use backward selection with fastbw(): starting from
the full model, variables with a p-value above 0.05 are removed one by
one. type="individual" tests each variable separately. Note
that rcs(age, 3) is one variable with two terms
(age and age'), so it is listed once, as
age.
For this illustrative exercise fastbw() is fine. There
are better ways to select variables, for example penalized regression
(LASSO or elastic net) or relying on clinical knowledge instead of
p-values, as backward selection tends to overfit. This is also why we
estimate a shrinkage factor in Tutorial 2.
# perform backward selection with fastbw():
# type="individual": test each variable separately
# rule="p": remove variables based on their p-value
# sls=0.05: remove a variable if its p-value is above 0.05
fastbw(full_model, type="individual", rule="p", sls=0.05)##
## Deleted Chi-Sq d.f. P Residual d.f. P AIC
## sex 0.11 1 0.7379 0.11 1 0.7379 -1.89
## hepato 0.15 1 0.6988 0.26 2 0.8773 -3.74
## ascites 3.05 1 0.0807 3.31 3 0.3460 -2.69
##
## Approximate Estimates after Deleting Factors
##
## Coef S.E. Wald Z P
## age 0.06770 0.02400 2.8207 4.792e-03
## age' -0.03661 0.02257 -1.6224 1.047e-01
## bili 0.79304 0.10919 7.2627 3.796e-13
## bili' -2.30232 0.37783 -6.0936 1.104e-09
## albumin -0.83765 0.20184 -4.1501 3.323e-05
## stage=2 0.29603 0.74623 0.3967 6.916e-01
## stage=3 0.73805 0.72262 1.0214 3.071e-01
## stage=4 1.23083 0.72734 1.6922 9.060e-02
##
## Factors in Final Model
##
## [1] age bili albumin stage# this gives final model: age, bili, albumin, stageWe then fit the backward selected model.
# fit backward selected model: the same as before, with only the selected variables
form_bw <- S_10 ~ rcs(age, 3) + rcs(bili, 3) + albumin + stage
bw_model <- Hmisc::fit.mult.impute(form_bw,
cph,
fitargs=list(x=TRUE, y=TRUE, surv=TRUE),
xtrans=imputed_data,
data=data,
fit.reps=TRUE)##
## Wald Statistic Information
##
## Variance Inflation Factors Due to Imputation:
##
## age age' bili bili' albumin stage=2 stage=3 stage=4
## 1.00 1.00 1.00 1.00 1.01 1.09 1.06 1.07
##
## Fraction of Missing Information:
##
## age age' bili bili' albumin stage=2 stage=3 stage=4
## 0.00 0.00 0.00 0.00 0.01 0.08 0.06 0.06
##
## d.f. for t-distribution for Tests of Single Coefficients:
##
## age age' bili bili' albumin stage=2
## 20946309.50 131064158.78 381202.54 364582.19 97205.48 592.59
## stage=3 stage=4
## 1199.64 1069.91
##
## The following fit components were averaged over the 5 model fits:
##
## linear.predictors means stats center# display predictor effects
summary(bw_model)## Effects Response : S_10
##
## Factor Low High Diff. Effect S.E. Lower 0.95 Upper 0.95
## age 42.8320 58.241 15.4090 0.59769 0.14878 0.306080 0.88930
## Hazard Ratio 42.8320 58.241 15.4090 1.81790 NA 1.358100 2.43340
## bili 0.8000 3.400 2.6000 1.51860 0.19498 1.136400 1.90070
## Hazard Ratio 0.8000 3.400 2.6000 4.56560 NA 3.115500 6.69060
## albumin 3.2425 3.770 0.5275 -0.44221 0.10613 -0.650230 -0.23419
## Hazard Ratio 3.2425 3.770 0.5275 0.64261 NA 0.521920 0.79121
## stage - 1:3 3.0000 1.000 NA -0.71732 0.72234 -2.133100 0.69845
## Hazard Ratio 3.0000 1.000 NA 0.48806 NA 0.118470 2.01060
## stage - 2:3 3.0000 2.000 NA -0.45077 0.27036 -0.980680 0.07913
## Hazard Ratio 3.0000 2.000 NA 0.63713 NA 0.375060 1.08230
## stage - 4:3 3.0000 4.000 NA 0.48047 0.19963 0.089196 0.87175
## Hazard Ratio 3.0000 4.000 NA 1.61680 NA 1.093300 2.391104.5 Present the final model
The effect of age per decade. Because age has a non-linear effect (a spline), the hazard ratio for 10 years of age depends on where you start. We therefore compare ages within the observed range, here 50 versus 60 years. (A comparison of age 0 with age 10 would be outside the range of the data and meaningless.)
# present effect for age per decade (from 50 to 60 years)
# age=c(50, 60) sets the two values that are compared
summary(bw_model, age=c(50, 60))## Effects Response : S_10
##
## Factor Low High Diff. Effect S.E. Lower 0.95 Upper 0.95
## age 50.0000 60.00 10.0000 0.24457 0.094077 0.060179 0.42895
## Hazard Ratio 50.0000 60.00 10.0000 1.27710 NA 1.062000 1.53560
## bili 0.8000 3.40 2.6000 1.51860 0.194980 1.136400 1.90070
## Hazard Ratio 0.8000 3.40 2.6000 4.56560 NA 3.115500 6.69060
## albumin 3.2425 3.77 0.5275 -0.44221 0.106130 -0.650230 -0.23419
## Hazard Ratio 3.2425 3.77 0.5275 0.64261 NA 0.521920 0.79121
## stage - 1:3 3.0000 1.00 NA -0.71732 0.722340 -2.133100 0.69845
## Hazard Ratio 3.0000 1.00 NA 0.48806 NA 0.118470 2.01060
## stage - 2:3 3.0000 2.00 NA -0.45077 0.270360 -0.980680 0.07913
## Hazard Ratio 3.0000 2.00 NA 0.63713 NA 0.375060 1.08230
## stage - 4:3 3.0000 4.00 NA 0.48047 0.199630 0.089196 0.87175
## Hazard Ratio 3.0000 4.00 NA 1.61680 NA 1.093300 2.39110Variable importance with the chi-square test statistic (minus its degrees of freedom is a fair way to compare):
# display variable importance using chi-square test statistic
anova(bw_model)## Wald Statistics Response: S_10
##
## Factor Chi-Square d.f. P
## age 16.27 2 0.0003
## Nonlinear 2.65 1 0.1033
## bili 116.25 2 <.0001
## Nonlinear 37.53 1 <.0001
## albumin 17.36 1 <.0001
## stage 14.66 3 0.0021
## TOTAL NONLINEAR 37.70 2 <.0001
## TOTAL 206.90 8 <.0001# visualize predictor effects
plot(Predict(bw_model))
A nomogram lets you calculate the risk of an individual patient by hand.
# nomogram: nomogram() calculates the points of every predictor value,
# plot() draws it
plot(rms::nomogram(bw_model))
4.6 Exercises: Part 4
Try each exercise yourself first. Then open the field below it to see the answer, the code and its results. The answers use new object names, so the saved models do not change.
Exercise 1. In the nomogram, what is the total number of points for a 75-year-old patient with a bilirubin of 2, an albumin of 3.6 and stage 4? Can you tell which variable contributes the most?
Show answer, code and results
Printing the nomogram gives the points for every value of every predictor (for age, in steps of 5 years):
nom <- rms::nomogram(bw_model)
print(nom)## Points per unit of linear predictor: 25.99955
## Linear predictor units per point : 0.03846221
##
##
## age Points
## 25 0
## 30 9
## 35 18
## 40 26
## 45 34
## 50 41
## 55 45
## 60 47
## 65 49
## 70 50
## 75 51
## 80 52
##
##
## bili Points
## 0 0
## 2 38
## 4 61
## 6 71
## 8 75
## 10 77
## 12 80
## 14 82
## 16 85
## 18 87
## 20 90
## 22 92
## 24 95
## 26 97
## 28 100
##
##
## albumin Points
## 1.8 65
## 2.0 61
## 2.2 57
## 2.4 52
## 2.6 48
## 2.8 44
## 3.0 39
## 3.2 35
## 3.4 31
## 3.6 26
## 3.8 22
## 4.0 17
## 4.2 13
## 4.4 9
## 4.6 4
## 4.8 0
##
##
## stage Points
## 1 0
## 2 7
## 3 19
## 4 31Look up the points for age 75, bilirubin 2, albumin 3.6 and stage 4, and add them up. The total is read on the “Total Points” axis, which converts the points to a risk. The variable with the widest range of points (the longest axis in the nomogram) contributes the most to the prediction.
Exercise 2. Change the age comparison to
c(40, 50) and c(60, 70). Why do the hazard
ratios differ?
Show answer, code and results
summary(bw_model, age=c(40, 50))## Effects Response : S_10
##
## Factor Low High Diff. Effect S.E. Lower 0.95 Upper 0.95
## age 40.0000 50.00 10.0000 0.55853 0.17340 0.218660 0.89839
## Hazard Ratio 40.0000 50.00 10.0000 1.74810 NA 1.244400 2.45570
## bili 0.8000 3.40 2.6000 1.51860 0.19498 1.136400 1.90070
## Hazard Ratio 0.8000 3.40 2.6000 4.56560 NA 3.115500 6.69060
## albumin 3.2425 3.77 0.5275 -0.44221 0.10613 -0.650230 -0.23419
## Hazard Ratio 3.2425 3.77 0.5275 0.64261 NA 0.521920 0.79121
## stage - 1:3 3.0000 1.00 NA -0.71732 0.72234 -2.133100 0.69845
## Hazard Ratio 3.0000 1.00 NA 0.48806 NA 0.118470 2.01060
## stage - 2:3 3.0000 2.00 NA -0.45077 0.27036 -0.980680 0.07913
## Hazard Ratio 3.0000 2.00 NA 0.63713 NA 0.375060 1.08230
## stage - 4:3 3.0000 4.00 NA 0.48047 0.19963 0.089196 0.87175
## Hazard Ratio 3.0000 4.00 NA 1.61680 NA 1.093300 2.39110summary(bw_model, age=c(60, 70))## Effects Response : S_10
##
## Factor Low High Diff. Effect S.E. Lower 0.95 Upper 0.95
## age 60.0000 70.00 10.0000 0.10094 0.15461 -0.202090 0.40397
## Hazard Ratio 60.0000 70.00 10.0000 1.10620 NA 0.817020 1.49780
## bili 0.8000 3.40 2.6000 1.51860 0.19498 1.136400 1.90070
## Hazard Ratio 0.8000 3.40 2.6000 4.56560 NA 3.115500 6.69060
## albumin 3.2425 3.77 0.5275 -0.44221 0.10613 -0.650230 -0.23419
## Hazard Ratio 3.2425 3.77 0.5275 0.64261 NA 0.521920 0.79121
## stage - 1:3 3.0000 1.00 NA -0.71732 0.72234 -2.133100 0.69845
## Hazard Ratio 3.0000 1.00 NA 0.48806 NA 0.118470 2.01060
## stage - 2:3 3.0000 2.00 NA -0.45077 0.27036 -0.980680 0.07913
## Hazard Ratio 3.0000 2.00 NA 0.63713 NA 0.375060 1.08230
## stage - 4:3 3.0000 4.00 NA 0.48047 0.19963 0.089196 0.87175
## Hazard Ratio 3.0000 4.00 NA 1.61680 NA 1.093300 2.39110# plot the effect of age on the log hazard
plot(Predict(bw_model, age))
The plot shows why: age has a non-linear effect (modelled with
rcs()), so the effect of age is not a straight line. The
change in log hazard for a 10-year difference therefore depends on where
you start. With a linear effect, the hazard ratio per decade would be
the same everywhere.
Exercise 3. Change sls=0.05 to
sls=0.157 (similar to AIC). Does the selected model
change?
Show answer, code and results
fastbw(full_model, type="individual", rule="p", sls=0.157)##
## Deleted Chi-Sq d.f. P Residual d.f. P AIC
## sex 0.11 1 0.7379 0.11 1 0.7379 -1.89
## hepato 0.15 1 0.6988 0.26 2 0.8773 -3.74
##
## Approximate Estimates after Deleting Factors
##
## Coef S.E. Wald Z P
## age 0.07049 0.02406 2.9304 3.386e-03
## age' -0.04335 0.02289 -1.8933 5.831e-02
## bili 0.79745 0.10922 7.3011 2.854e-13
## bili' -2.34284 0.37854 -6.1892 6.048e-10
## albumin -0.72787 0.21140 -3.4431 5.751e-04
## ascites=Yes 0.45878 0.26270 1.7464 8.074e-02
## stage=2 0.25941 0.74652 0.3475 7.282e-01
## stage=3 0.75467 0.72268 1.0443 2.964e-01
## stage=4 1.18395 0.72783 1.6267 1.038e-01
##
## Factors in Final Model
##
## [1] age bili albumin ascites stageCompare the variables that are kept with the result for
sls=0.05. With sls=0.157 the model usually
keeps a variable or two that were removed with sls=0.05
(here ascites, which has a p-value between 0.05 and
0.157).
Why 0.157? In backward selection a variable is
removed when its p-value is larger than sls. The threshold
determines how strict the selection is:
- The AIC (Akaike information criterion) is
-2 x log-likelihood + 2 x number of parameters. Removing a variable with 1 degree of freedom lowers the AIC when the chi-square of that variable is smaller than 2. A chi-square of 2 with 1 degree of freedom corresponds to a p-value of 0.157. So selecting withsls=0.157is (for variables with 1 degree of freedom) the same as removing variables as long as that improves the AIC. For variables with more degrees of freedom the match is only approximate. - With
sls=0.05a variable is only kept when its chi-square is larger than 3.84 (for 1 degree of freedom). That is a much higher bar than 2, so fewer variables are kept and the model is smaller.
A less strict threshold (0.157) keeps more variables: you are less likely to miss a variable that matters, but the model has more parameters and a larger risk of overfitting (more optimism and more shrinkage needed, see Tutorial 2). A stricter threshold (0.05) gives a smaller, simpler model, but you might remove variables that carry a bit of information. There is no single correct choice: report which threshold you used, and validate the whole procedure, including the selection, as we do in Tutorial 2. Also remember that selecting variables on p-values at all is a debatable strategy (see the note above on penalized regression).
Exercise 4. What does the cox.zph
output tell you? Which predictor (if any) might violate proportional
hazards?
Show answer, code and results
survival::cox.zph(full_model)## chisq df p
## rcs(age, 3) 0.5573 66674 1
## rcs(bili, 3) 0.4378 25006 1
## albumin 0.6281 103533 1
## sex 0.5998 203798 1
## ascites 2.2708 31013744 1
## hepato 0.0879 337198 1
## stage 4.3057 252 1
## GLOBAL 6.8567 31752713 1Each row tests whether the effect of that predictor changes over
time. A small p-value (for example below 0.05) suggests that the
proportional hazards assumption is violated for that predictor, and the
GLOBAL row tests all predictors together. If a predictor
violates the assumption, you could add an interaction with time,
stratify on it, or use a different model. With few events these tests
have little power, so also look at the plots.
Exercise 5. Refit the full model with 4 knots for
bilirubin, rcs(bili, 4), instead of 3. Does the estimated
effect of bilirubin change? How many parameters does it add?
Show answer, code and results
We fit the model in a new object, full_model_4knots, so
full_model stays as it is.
form_4knots <- S_10 ~ rcs(age, 3) + rcs(bili, 4) + albumin + sex +
ascites + hepato + stage
full_model_4knots <- Hmisc::fit.mult.impute(form_4knots,
cph,
fitargs=list(x=TRUE, y=TRUE, surv=TRUE),
xtrans=imputed_data,
data=data,
fit.reps=TRUE)##
## Wald Statistic Information
##
## Variance Inflation Factors Due to Imputation:
##
## age age' bili bili' bili'' albumin
## 1.01 1.01 1.01 1.01 1.01 1.00
## sex=Female ascites=Yes hepato=Yes stage=2 stage=3 stage=4
## 1.00 1.14 1.09 1.08 1.07 1.08
##
## Fraction of Missing Information:
##
## age age' bili bili' bili'' albumin
## 0.01 0.01 0.01 0.01 0.01 0.00
## sex=Female ascites=Yes hepato=Yes stage=2 stage=3 stage=4
## 0.00 0.12 0.08 0.07 0.07 0.08
##
## d.f. for t-distribution for Tests of Single Coefficients:
##
## age age' bili bili' bili'' albumin
## 44243.18 20152.96 82587.68 62450.86 61604.99 26275689.48
## sex=Female ascites=Yes hepato=Yes stage=2 stage=3 stage=4
## 497354.41 267.67 571.30 741.29 881.78 669.50
##
## The following fit components were averaged over the 5 model fits:
##
## linear.predictors means stats center# effect of bilirubin with 3 and 4 knots
plot(Predict(full_model, bili))
plot(Predict(full_model_4knots, bili))
# number of coefficients
length(coef(full_model))## [1] 11length(coef(full_model_4knots))## [1] 12With 4 knots the curve can bend in more places, but with this number of events the shape is very similar to the curve with 3 knots, while the model needs one more parameter. More knots give more flexibility, but also a higher risk of overfitting, so 3 knots is a sensible default in a data set of this size.
Exercise 6. Add edema and
copper to form_full. How does variable
importance change?
Show answer, code and results
We fit the larger model in a new object,
full_model_extra, so full_model stays as it
is.
form_extra <- S_10 ~ rcs(age, 3) + rcs(bili, 3) + albumin + sex +
ascites + hepato + stage + edema + copper
full_model_extra <- Hmisc::fit.mult.impute(form_extra,
cph,
fitargs=list(x=TRUE, y=TRUE, surv=TRUE),
xtrans=imputed_data,
data=data,
fit.reps=TRUE)##
## Wald Statistic Information
##
## Variance Inflation Factors Due to Imputation:
##
## age age'
## 1.00 1.01
## bili bili'
## 1.03 1.03
## albumin sex=Female
## 1.01 1.06
## ascites=Yes hepato=Yes
## 1.22 1.05
## stage=2 stage=3
## 1.07 1.07
## stage=4 edema=Untreated or successfully treated
## 1.07 1.02
## edema=Despite diuretics copper
## 1.01 1.79
##
## Fraction of Missing Information:
##
## age age'
## 0.00 0.01
## bili bili'
## 0.03 0.02
## albumin sex=Female
## 0.01 0.05
## ascites=Yes hepato=Yes
## 0.18 0.05
## stage=2 stage=3
## 0.07 0.07
## stage=4 edema=Untreated or successfully treated
## 0.06 0.02
## edema=Despite diuretics copper
## 0.01 0.44
##
## d.f. for t-distribution for Tests of Single Coefficients:
##
## age age'
## 257842.87 108635.55
## bili bili'
## 5498.37 6417.54
## albumin sex=Female
## 104515.53 1369.17
## ascites=Yes hepato=Yes
## 123.74 1760.96
## stage=2 stage=3
## 842.00 825.73
## stage=4 edema=Untreated or successfully treated
## 951.11 9363.50
## edema=Despite diuretics copper
## 36821.90 20.51
##
## The following fit components were averaged over the 5 model fits:
##
## linear.predictors means stats centeranova(full_model_extra)## Wald Statistics Response: S_10
##
## Factor Chi-Square d.f. P
## age 13.18 2 0.0014
## Nonlinear 3.67 1 0.0553
## bili 66.07 2 <.0001
## Nonlinear 31.75 1 <.0001
## albumin 6.12 1 0.0133
## sex 0.00 1 0.9914
## ascites 0.82 1 0.3658
## hepato 0.11 1 0.7360
## stage 8.98 3 0.0296
## edema 13.24 2 0.0013
## copper 2.15 1 0.1429
## TOTAL NONLINEAR 32.71 2 <.0001
## TOTAL 236.75 14 <.0001Compare the chi-square values (minus their degrees of freedom) with
the table for full_model. Copper is a known predictor in
primary biliary cirrhosis, and it is missing for about a quarter of the
patients, so its effect is estimated on imputed values. Note that
copper is right-skewed, so a transformation (for example a
log transformation) could be considered.
LS0tCnRpdGxlOiAiVHV0b3JpYWwgMTogRGV2ZWxvcCBhIHByZWRpY3Rpb24gbW9kZWwiCm91dHB1dDoKICBodG1sX2RvY3VtZW50OgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKICAgIG51bWJlcl9zZWN0aW9uczogdHJ1ZQogICAgZGZfcHJpbnQ6IHBhZ2VkCiAgICBjb2RlX2Rvd25sb2FkOiB0cnVlCi0tLQoKYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvPVRSVUUsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0UpCmBgYAoKPHN0eWxlPgpkZXRhaWxzIHsgYm9yZGVyOiAxcHggc29saWQgI2M4YzhjODsgYm9yZGVyLXJhZGl1czogNHB4OyBwYWRkaW5nOiAwLjRlbSAxZW07IG1hcmdpbjogMC41ZW0gMCAxLjVlbSAwOyBiYWNrZ3JvdW5kOiAjZjhmOGY4OyB9CnN1bW1hcnkgeyBjdXJzb3I6IHBvaW50ZXI7IGZvbnQtd2VpZ2h0OiBib2xkOyB9Cjwvc3R5bGU+Cgo8ZGl2IHN0eWxlPSJib3JkZXItbGVmdDogNHB4IHNvbGlkICM2MTliZmY7IGJhY2tncm91bmQ6ICNmMGY1ZmY7IHBhZGRpbmc6IDAuNmVtIDFlbTsgbWFyZ2luOiAxZW0gMDsiPgoKKipOZXcgdG8gUj8qKiBCZWZvcmUgeW91IHN0YXJ0LCBpbnN0YWxsIFIgYW5kIFJTdHVkaW8gZnJvbSBbcG9zaXQuY29dKGh0dHBzOi8vcG9zaXQuY28vZG93bmxvYWQvcnN0dWRpby1kZXNrdG9wLykgYW5kIHdvcmsgdGhyb3VnaCB0aGUgW1BSRURJQ1QgUHJhY3RpY2FsIDFdKGh0dHBzOi8vcnB1YnMuY29tL0NITU1hYXMvUFJFRElDVF9wcmFjdGljYWxfMSkgKEdldHRpbmcgc3RhcnRlZCB3aXRoIFIpIGFuZCB0aGUgW0ludHJvZHVjdGlvbiB0byBSXShodHRwczovL3JqamFuc2UuZ2l0aHViLmlvL3J0Lyk6IHRoZSBjaGFwdGVycyAiU3RhcnQiLCAiU2V0LXVwIiAoW1JdKGh0dHBzOi8vcmpqYW5zZS5naXRodWIuaW8vcnQvci5odG1sKSwgW1JTdHVkaW9dKGh0dHBzOi8vcmpqYW5zZS5naXRodWIuaW8vcnQvcnN0dWRpby5odG1sKSkgYW5kICJUaGUgYmFzaWNzIiwgYW5kIHRoZSBleGVyY2lzZXMgYXQgdGhlIGJvdHRvbSBvZiB0aGUgcGFnZXMgW0NyZWF0aW5nIGRhdGFdKGh0dHBzOi8vcmpqYW5zZS5naXRodWIuaW8vcnQvY3JlYXRpbmcuaHRtbCNleGVyY2lzZXMpLCBbVXNpbmcgZGF0YV0oaHR0cHM6Ly9yamphbnNlLmdpdGh1Yi5pby9ydC91c2luZy5odG1sI2V4ZXJjaXNlcyksIFtQYWNrYWdlc10oaHR0cHM6Ly9yamphbnNlLmdpdGh1Yi5pby9ydC9wYWNrYWdlcy5odG1sI2V4ZXJjaXNlcykgYW5kIFtFeHRlcm5hbCBkYXRhXShodHRwczovL3JqamFuc2UuZ2l0aHViLmlvL3J0L2RhdGEuaHRtbCNleGVyY2lzZXMpLgo8L2Rpdj4KCiMgT3ZlcnZpZXcgey19CgpJbiB0aGlzIHR1dG9yaWFsIHlvdSB3aWxsIGdvIGZyb20gcmF3IGRhdGEgdG8gYSBmaW5hbCBwcmVkaWN0aW9uIG1vZGVsIGZvciBhIHRpbWUtdG8tZXZlbnQgb3V0Y29tZS4gSXQgaGFzIGZvdXIgcGFydHM6CgoxLiAqKkxvYWQgZGF0YSBhbmQgbWFrZSBUYWJsZSAxOioqIGRlc2NyaWJlIHRoZSBzdHVkeSBwb3B1bGF0aW9uLgoKMi4gKipEZXNjcmliZSB0aGUgc3Vydml2YWwgZGF0YToqKiBkZWZpbmUgdGhlIG91dGNvbWUsIG1ha2UgYSBLYXBsYW4tTWVpZXIgY3VydmUsIGNob29zZSBhIHRpbWUgaG9yaXpvbiBhbmQgY2FsY3VsYXRlIHRoZSBtZWRpYW4gZm9sbG93LXVwIHRpbWUuCgozLiAqKk11bHRpcGxlIGltcHV0YXRpb246KiogZGVhbCB3aXRoIG1pc3NpbmcgZGF0YS4KCjQuICoqRml0IHRoZSBtb2RlbHM6KiogY2hlY2sgbm9uLWxpbmVhciBlZmZlY3RzLCBzZWxlY3QgcHJlZGljdG9ycyBhbmQgcHJlc2VudCB0aGUgZmluYWwgbW9kZWwuCgpUdXRvcmlhbCAyIGNvbnRpbnVlcyB3aXRoIHRoZSBmaW5hbCBtb2RlbCBhbmQgdmFsaWRhdGVzIGl0IChzaHJpbmthZ2UsIG9wdGltaXNtLWNvcnJlY3RlZCBDLWluZGV4LCBjYWxpYnJhdGlvbiBhbmQgZGVjaXNpb24gY3VydmVzKS4gV2UgdXNlIHRoZSBgcGJjYCBkYXRhIChwcmltYXJ5IGJpbGlhcnkgY2lycmhvc2lzKSBmcm9tIHRoZSBgc3Vydml2YWxgIHBhY2thZ2UgYXMgZHVtbXkgZGF0YS4KCiMgRG93bmxvYWQgdGhlIHR1dG9yaWFsIHstfQoKKipEbyBub3QganVzdCByZWFkIHRoaXMgcGFnZTogcnVuIHRoZSBjb2RlIHlvdXJzZWxmLioqIFlvdSBvbmx5IGxlYXJuIFIgYnkgZG9pbmcgaXQuIERvd25sb2FkIHRoZSBSIE1hcmtkb3duIGZpbGUgb2YgdGhpcyB0dXRvcmlhbCAoYFR1dG9yaWFsXzFfRGV2ZWxvcF9QcmVkaWN0aW9uX01vZGVsLlJtZGApIGJ5IGNsaWNraW5nIHRoZSAqKkNvZGUqKiBidXR0b24gYXQgdGhlIHRvcCByaWdodCBvZiB0aGlzIHBhZ2UgYW5kIGNob29zaW5nICoqRG93bmxvYWQgUm1kKiosIGFuZCBydW4gZXZlcnkgY2h1bmsgaW4gUlN0dWRpbywgb25lIGJ5IG9uZS4gUmVhZCB0aGUgb3V0cHV0LCBjaGFuZ2UgdGhlIGNvZGUsIGJyZWFrIGl0IGFuZCBmaXggaXQsIGFuZCB0cnkgdGhlIGV4ZXJjaXNlcyBiZWZvcmUgeW91IG9wZW4gdGhlIGFuc3dlcnMuIFJlYWRpbmcgdGhlIHJlc3VsdHMgb24gdGhpcyBwYWdlIGlzIG5vdCBhIHN1YnN0aXR1dGUuCgpUbyBtYWtlIHN1cmUgZXZlcnl0aGluZyB3b3JrczoKCjEuICoqQ3JlYXRlIGEgZm9sZGVyKiogZm9yIHRoZSBjb3Vyc2UsIGZvciBleGFtcGxlIGBQcmVkaWN0aXZlX21vZGVscy9gLCBhbmQgcHV0IHRoZSBkb3dubG9hZGVkIGAuUm1kYCBmaWxlIGluIGl0LgoKMi4gKipPcGVuIHRoZSBgLlJtZGAgZmlsZSBpbiBSU3R1ZGlvLioqIFdoZW4geW91IGtuaXQgb3IgcnVuIGNodW5rcyBmcm9tIGFuIFJtZCBmaWxlLCBSIHVzZXMgdGhlIGZvbGRlciB0aGF0IGNvbnRhaW5zIHRoZSBmaWxlIGFzIHRoZSB3b3JraW5nIGRpcmVjdG9yeS4gQ2hlY2sgdGhpcyBieSBydW5uaW5nOgoKICAgYGBgcgogICBnZXR3ZCgpCiAgIGBgYAoKICAgVGhlIG91dHB1dCBzaG91bGQgYmUgdGhlIHBhdGggb2YgdGhlIGZvbGRlciBmcm9tIHN0ZXAgMS4gSWYgaXQgaXMgbm90LCBzZXQgaXQgd2l0aCBgc2V0d2QoInBhdGgvdG8veW91ci9mb2xkZXIiKWAgb3IgaW4gUlN0dWRpbyB2aWEgKlNlc3Npb24gPiBTZXQgV29ya2luZyBEaXJlY3RvcnkgPiBUbyBTb3VyY2UgRmlsZSBMb2NhdGlvbiouCgozLiAqKkRvIG5vdCB1c2UgZmlsZSBwYXRocyoqIGluIHRoZSBjb2RlLiBBbGwgZmlsZXMgYXJlIHJlYWQgZnJvbSBhbmQgc2F2ZWQgdG8gdGhlIHdvcmtpbmcgZGlyZWN0b3J5LCBzbyBrZWVwIGFsbCBmaWxlcyBpbiB0aGlzIG9uZSBmb2xkZXIuCgo0LiBSdW5uaW5nIHRoZSB0dXRvcmlhbCBzYXZlcyBpdHMgb3V0cHV0IChmb3IgZXhhbXBsZSB0aGUgS2FwbGFuLU1laWVyIGZpZ3VyZSBgS01fcGxvdF8xMF95ZWFycy5wbmdgIGFuZCBUYWJsZSAxIGFzIGEgV29yZCBmaWxlKSBpbiB0aGlzIHNhbWUgZm9sZGVyLgoKVGhlIGZvbGRlciB3aWxsIHRoZW4gbG9vayBsaWtlIHRoaXM6CgpgYGAKUHJlZGljdGl2ZV9tb2RlbHMvCuKUnOKUgOKUgCBUdXRvcmlhbF8xX0RldmVsb3BfUHJlZGljdGlvbl9Nb2RlbC5SbWQK4pSc4pSA4pSAIFR1dG9yaWFsXzJfVmFsaWRhdGVfUHJlZGljdGlvbl9Nb2RlbC5SbWQK4pSc4pSA4pSAIERlc2NyaXB0aXZlcy5kb2N4ICAgICAgICAoY3JlYXRlZCBieSBUdXRvcmlhbCAxKQrilJTilIDilIAgS01fcGxvdF8xMF95ZWFycy5wbmcgICAgIChjcmVhdGVkIGJ5IFR1dG9yaWFsIDEpCmBgYAoKIyBTZXR1cCB7LX0KCiMjIENsZWFyIHRoZSBlbnZpcm9ubWVudCBhbmQgbG9hZCBsaWJyYXJpZXMgey19CgpTdGFydCB3aXRoIGEgY2xlYW4gZW52aXJvbm1lbnQgc28gdGhhdCByZXN1bHRzIGRvIG5vdCBkZXBlbmQgb24gb2JqZWN0cyBsZWZ0IG92ZXIgZnJvbSBlYXJsaWVyIHdvcmsuIEEgKnBhY2thZ2UqIGlzIGEgY29sbGVjdGlvbiBvZiBmdW5jdGlvbnMgdGhhdCBzb21lb25lIHdyb3RlIGFuZCBzaGFyZWQuIGBsaWJyYXJ5KHBhY2thZ2UpYCBsb2FkcyBhIHBhY2thZ2UsIHNvIHRoYXQgeW91IGNhbiB1c2UgaXRzIGZ1bmN0aW9ucyBkaXJlY3RseS4gV2Ugb25seSBsb2FkIGBybXNgLCBiZWNhdXNlIHdlIHVzZSBpdHMgZnVuY3Rpb25zIChzdWNoIGFzIGBjcGhgLCBgUHJlZGljdGAgYW5kIGBmYXN0YndgKSB3aXRob3V0IHRoZSBgcGFja2FnZTo6YCBwcmVmaXguIEFsbCBvdGhlciBwYWNrYWdlcyBhcmUgY2FsbGVkIHdpdGggdGhlIGA6OmAgbm90YXRpb24gKHNlZSBiZWxvdyksIHNvIHRoZXkgZG8gbm90IGhhdmUgdG8gYmUgbG9hZGVkLCBidXQgdGhleSBkbyBoYXZlIHRvIGJlIGluc3RhbGxlZCBvbmNlLiBXZSB1c2U6CgoqIGBzdXJ2aXZhbGA6IHRoZSBkYXRhIGFuZCBzdXJ2aXZhbCBhbmFseXNpcwoqIGBkcGx5cmA6IGRhdGEgbWFuaXB1bGF0aW9uCiogYGFyc2VuYWxgIGFuZCBgZmxleHRhYmxlYDogVGFibGUgMSBhbmQgc2F2aW5nIGl0IHRvIFdvcmQKKiBgc3Vydm1pbmVyYCBhbmQgYHByb2RsaW1gOiBLYXBsYW4tTWVpZXIgY3VydmVzIGFuZCBmb2xsb3ctdXAgdGltZQoqIGB2aXNkYXRgIGFuZCBgbWljZWA6IHZpc3VhbGl6aW5nIGFuZCBpbXB1dGluZyBtaXNzaW5nIGRhdGEKKiBgcm1zYCBhbmQgYEhtaXNjYDogZml0dGluZyBhbmQgcHJlc2VudGluZyBwcmVkaWN0aW9uIG1vZGVscyAoYGNwaGAsIGBmaXQubXVsdC5pbXB1dGVgKQoqIGBnZ3Bsb3QyYCBhbmQgYGdncHVicmA6IHNhdmluZyBhbmQgY29tYmluaW5nIGZpZ3VyZXMKCmBgYHtyIGxpYnJhcmllc30KIyBjbGVhciB0aGUgZW52aXJvbm1lbnQ6IHJlbW92ZSBhbGwgb2JqZWN0cyB0aGF0IFIgY3VycmVudGx5IHJlbWVtYmVycwpybShsaXN0PWxzKCkpCgojIGxvYWQgdGhlIHJtcyBwYWNrYWdlOiBhZnRlciB0aGlzLCBhbGwgaXRzIGZ1bmN0aW9ucyBjYW4gYmUgdXNlZCBkaXJlY3RseQpsaWJyYXJ5KHJtcykKCiMgVGhlIG90aGVyIHBhY2thZ2VzIG11c3QgYmUgaW5zdGFsbGVkIG9uY2Ugb24geW91ciBjb21wdXRlciAobm90IHJ1biBoZXJlKToKIyBpbnN0YWxsLnBhY2thZ2VzKGMoInN1cnZpdmFsIiwgImRwbHlyIiwgImFyc2VuYWwiLCAiZmxleHRhYmxlIiwgInN1cnZtaW5lciIsCiMgICAgICAgICAgICAgICAgICAgICJwcm9kbGltIiwgInZpc2RhdCIsICJtaWNlIiwgIkhtaXNjIiwgImdncGxvdDIiLCAiZ2dwdWJyIikpCmBgYAoKIyMgVGhlIGA6OmAgbm90YXRpb24gey19CgpJbiB0aGlzIHR1dG9yaWFsIHlvdSB3aWxsIG9mdGVuIHNlZSBjb2RlIGxpa2UgYGRwbHlyOjpzZWxlY3QoLi4uKWAuIFRoZSBub3RhdGlvbiBgcGFja2FnZTo6ZnVuY3Rpb24oKWAgbWVhbnMgInVzZSB0aGUgZnVuY3Rpb24gYGZ1bmN0aW9uYCBmcm9tIHRoZSBwYWNrYWdlIGBwYWNrYWdlYCIuCgpZb3UgZG8gbm90IGhhdmUgdG8gd3JpdGUgaXQgdGhpcyB3YXkgb25jZSB5b3UgaGF2ZSBsb2FkZWQgYSBwYWNrYWdlIHdpdGggYGxpYnJhcnkoKWA6IGFmdGVyIGBsaWJyYXJ5KGRwbHlyKWAsIGBzZWxlY3QoLi4uKWAgd29ya3MgYXMgd2VsbC4gV2Ugc3RpbGwgd3JpdGUgaXQgb3V0IGJlY2F1c2UgaXQgaXMgbmljZSB0byBzZWUgd2hlcmUgZWFjaCBmdW5jdGlvbiBjb21lcyBmcm9tLCBhbmQgaXQgYXZvaWRzIG1pc3Rha2VzIHdoZW4gdHdvIHBhY2thZ2VzIGhhdmUgYSBmdW5jdGlvbiB3aXRoIHRoZSBzYW1lIG5hbWUgKFIgdGhlbiB1c2VzIHRoZSBvbmUgZnJvbSB0aGUgcGFja2FnZSB0aGF0IHdhcyBsb2FkZWQgbGFzdCkuIEZvciBhIHBhY2thZ2UgdGhhdCB5b3UgZG8gbm90IGxvYWQgd2l0aCBgbGlicmFyeSgpYCwgc3VjaCBhcyBgb3Blbnhsc3hgIG9yIGBmb3JlaWduYCBpbiB0aGUgdGVtcGxhdGVzIGJlbG93LCB0aGUgYDo6YCBub3RhdGlvbiBsZXRzIHlvdSB1c2Ugb25lIG9mIGl0cyBmdW5jdGlvbnMgd2l0aG91dCBsb2FkaW5nIHRoZSB3aG9sZSBwYWNrYWdlLgoKCiMjIFNldCB0aGUgc2VlZCB7LX0KClNvbWUgc3RlcHMgaW4gdGhpcyB0dXRvcmlhbCB1c2UgcmFuZG9tIG51bWJlcnMsIG1vc3QgaW1wb3J0YW50bHkgbXVsdGlwbGUgaW1wdXRhdGlvbi4gV2l0aG91dCBhIHNlZWQsIHlvdSB3b3VsZCBnZXQgZGlmZmVyZW50IGltcHV0ZWQgdmFsdWVzLCBhbmQgdGhlcmVmb3JlIHNsaWdodGx5IGRpZmZlcmVudCByZXN1bHRzLCBldmVyeSB0aW1lIHlvdSBydW4gdGhlIGNvZGUuIGBzZXQuc2VlZCgpYCBmaXhlcyB0aGUgc3RhcnRpbmcgcG9pbnQgb2YgUidzIHJhbmRvbSBudW1iZXIgZ2VuZXJhdG9yLCBzbyBldmVyeW9uZSB3aG8gcnVucyB0aGUgY29kZSBvYnRhaW5zIHRoZSBzYW1lIG51bWJlcnM6CgpgYGB7ciBzZWVkLWV4YW1wbGV9CiMgdGhlIHNhbWUgc2VlZCBnaXZlcyB0aGUgc2FtZSByYW5kb20gbnVtYmVycwpzZXQuc2VlZCgxKQpybm9ybSgzKQpzZXQuc2VlZCgxKQpybm9ybSgzKQpgYGAKCldlIHNldCB0aGUgcmFuZG9tIHNlZWQgKipvbmNlLCBoZXJlIGF0IHRoZSB0b3AqKi4gSW4gYWRkaXRpb24sIGBtaWNlKClgIGhhcyBpdHMgb3duIGBzZWVkYCBhcmd1bWVudCwgd2hpY2ggd2UgdXNlIHdoZW4gd2UgaW1wdXRlLCBzbyB0aGF0IHRoZSBpbXB1dGF0aW9ucyBkbyBub3QgZGVwZW5kIG9uIHdoYXQgd2FzIHJ1biBiZWZvcmUuIElmIHlvdSBjaGFuZ2UgdGhlIG9yZGVyIG9mIHRoZSBjb2RlIG9yIGFkZCBleHRyYSByYW5kb20gc3RlcHMsIHRoZSBudW1iZXJzIGFmdGVyIHRoYXQgcG9pbnQgY2FuIGNoYW5nZS4KCmBgYHtyIHNlZWR9CiMgc2V0IHRoZSBzZWVkIG9uY2UgZm9yIHRoZSB3aG9sZSB0dXRvcmlhbApzZXQuc2VlZCgxKQpgYGAKCiMjIFdvcmsgaW4gb25lIGZvbGRlciB7LX0KCkV2ZXJ5dGhpbmcgaW4gdGhpcyB0dXRvcmlhbCBpcyBzYXZlZCBpbiBvbmUgZm9sZGVyOiB0aGUgZm9sZGVyIHRoYXQgY29udGFpbnMgdGhpcyBSbWQgZmlsZS4gV2hlbiB5b3Uga25pdCB0aGUgZG9jdW1lbnQgKG9yIHJ1biB0aGUgY2h1bmtzIGluIFJTdHVkaW8pLCB0aGUgd29ya2luZyBkaXJlY3RvcnkgaXMgdGhpcyBmb2xkZXIsIHNvIHRhYmxlcywgZmlndXJlcyBhbmQgUiBkYXRhIGZpbGVzIGFyZSBzYXZlZCByaWdodCBuZXh0IHRvIHRoZSBSbWQgYW5kIHlvdSBkbyBub3QgbmVlZCB0byBzZXQgYW55IGZpbGUgcGF0aHMuCgoqKllvdXIgdHVybjoqKiBjaGVjayB3aGVyZSB5b3VyIGZpbGVzIHdpbGwgYmUgc2F2ZWQuCgpgYGB7ciBwYXRoLCBldmFsPUZBTFNFfQojIGFsbCBmaWxlcyBhcmUgc2F2ZWQgaW4gdGhlIHdvcmtpbmcgZGlyZWN0b3J5LCB3aGljaCBpcyB0aGUgZm9sZGVyIG9mIHRoaXMgUm1kCmdldHdkKCkKYGBgCgojIExvYWQgZGF0YSBhbmQgbWFrZSBUYWJsZSAxCgpFdmVyeSBwcmVkaWN0aW9uIG1vZGVsIHByb2plY3Qgc3RhcnRzIGJ5IGxvb2tpbmcgYXQgdGhlIGRhdGEuIEluIHRoaXMgcGFydCB5b3UgbG9hZCB0aGUgZGF0YSwgZGVjaWRlIHdoaWNoIHZhcmlhYmxlcyBhcmUgY2FuZGlkYXRlIHByZWRpY3RvcnMsIG1ha2Ugc3VyZSBldmVyeSB2YXJpYWJsZSBoYXMgdGhlIGNvcnJlY3QgdHlwZSwgYW5kIGNyZWF0ZSBhICJUYWJsZSAxIiB0aGF0IGRlc2NyaWJlcyB5b3VyIHN0dWR5IHBvcHVsYXRpb24uIFlvdSBzYXZlIHRoaXMgdGFibGUgdG8gV29yZCBzbyB5b3UgY2FuIHVzZSBpdCBpbiBhIG1hbnVzY3JpcHQuCgojIyBMb2FkIHRoZSBkYXRhCgpJbiB5b3VyIG93biBwcm9qZWN0IHlvdSB3aWxsIHJlYWQgeW91ciBvd24gZGF0YSBpbnRvIFIsIG1vc3Qgb2Z0ZW4gZnJvbSBhbiBFeGNlbCBvciBTUFNTIGZpbGUuIFRoZSBuZXh0IHR3byBibG9ja3Mgc2hvdyBob3cgdG8gZG8gdGhhdC4gKipUaGV5IGFyZSBub3QgcnVuIGluIHRoaXMgdHV0b3JpYWwqKiAoYGV2YWw9RkFMU0VgKSwgYmVjYXVzZSB3ZSBkbyBub3QgaGF2ZSB5b3VyIGZpbGVzOyB0aGV5IGFyZSBoZXJlIGFzIHRlbXBsYXRlcyBmb3IgeW91ciBvd24gcHJvamVjdHMuIEFmdGVyIHRoZW0sIHdlIGxvYWQgdGhlIGR1bW15IGRhdGEgdGhhdCB3ZSBhY3R1YWxseSB1c2UuCgojIyMgVGVtcGxhdGU6IGxvYWQgYW4gRXhjZWwgZmlsZSAobm90IHJ1bikKCmBvcGVueGxzeDo6cmVhZC54bHN4KClgIHJlYWRzIGFuIGAueGxzeGAgZmlsZSBpbnRvIGEgZGF0YSBmcmFtZSAob2xkZXIgYC54bHNgIGZpbGVzIG11c3QgZmlyc3QgYmUgc2F2ZWQgYXMgYC54bHN4YCkuIFJlcGxhY2UgdGhlIHBhdGggd2l0aCB0aGUgbG9jYXRpb24gb2YgeW91ciBmaWxlLiBJZiB5b3VyIGZpbGUgaXMgaW4gdGhlIHNhbWUgZm9sZGVyIGFzIHRoaXMgUm1kLCB0aGUgZmlsZSBuYW1lIGFsb25lIGlzIGVub3VnaC4gSWYgdGhlIGRhdGEgYXJlIG5vdCBvbiB0aGUgZmlyc3Qgc2hlZXQsIGFkZCB0aGUgYXJndW1lbnQgYHNoZWV0PSJTaGVldCBuYW1lImAgKG9yIHRoZSBzaGVldCBudW1iZXIpLiBJbnN0YWxsIHRoZSBwYWNrYWdlIG9uY2Ugd2l0aCBgaW5zdGFsbC5wYWNrYWdlcygib3Blbnhsc3giKWAgaWYgeW91IGRvIG5vdCBoYXZlIGl0IHlldC4KCmBgYHtyIHAxLWxvYWQtZXhjZWwsIGV2YWw9RkFMU0V9CiMgbG9hZCBFeGNlbCBkYXRhICh0aGlzIGJsb2NrIGlzIG5vdCBydW4pCmRhdGEgPC0gb3Blbnhsc3g6OnJlYWQueGxzeCgiQzovLi4uL2ZpbGVfZGF0YV9uYW1lLnhsc3giKQpgYGAKCiMjIyBUZW1wbGF0ZTogbG9hZCBhbiBTUFNTIGZpbGUgKG5vdCBydW4pCgpgZm9yZWlnbjo6cmVhZC5zcHNzKClgIHJlYWRzIGFuIFNQU1MgYC5zYXZgIGZpbGUuIFdpdGggYHRvLmRhdGEuZnJhbWU9VFJVRWAgdGhlIHJlc3VsdCBpcyBhIGRhdGEgZnJhbWUgaW5zdGVhZCBvZiBhIGxpc3QuIFNQU1Mgc3RvcmVzIGRhdGVzIGFzIHNlY29uZHMgc2luY2UgYSBzdGFydGluZyBkYXRlLCBzbyBgb3JpZ2luPSIwMS0wMS0xOTcwImAgdGVsbHMgUiBob3cgdG8gY29udmVydCB0aGVtIHRvIGRhdGVzLiBWYXJpYWJsZXMgd2l0aCB2YWx1ZSBsYWJlbHMgaW4gU1BTUyBiZWNvbWUgZmFjdG9ycyBpbiBSLCBzbyBjaGVjayBhZnRlcndhcmRzIHRoYXQgZXZlcnkgdmFyaWFibGUgaGFzIHRoZSB0eXBlIHlvdSBleHBlY3QgKHNlZSB0aGUgbmV4dCBzZWN0aW9uKS4KCmBgYHtyIHAxLWxvYWQtc3BzcywgZXZhbD1GQUxTRX0KIyBsb2FkIFNQU1MgZmlsZSAodGhpcyBibG9jayBpcyBub3QgcnVuKQpkYXRhIDwtIGZvcmVpZ246OnJlYWQuc3BzcyhmaWxlPSJDOi8uLi4vZmlsZV9kYXRhX25hbWUuc2F2IiwgdG8uZGF0YS5mcmFtZT1UUlVFLCBvcmlnaW49IjAxLTAxLTE5NzAiKQpgYGAKCiMjIyBMb2FkIHRoZSBkdW1teSBkYXRhIHRoYXQgd2UgdXNlCgpIZXJlIHdlIHVzZSB0aGUgZHVtbXkgZGF0YSBmcm9tIHRoZSBgc3Vydml2YWxgIHBhY2thZ2UuIFRoZSBkYXRhIGRpY3Rpb25hcnkgaXMgYXZhaWxhYmxlIGF0IDxodHRwczovL2NyYW4uci1wcm9qZWN0Lm9yZy93ZWIvcGFja2FnZXMvc3Vydml2YWwvc3Vydml2YWwucGRmPi4KCmBgYHtyIHAxLWxvYWQtZGF0YX0KIyBsb2FkIHRoZSBwYmMgZGF0YSBzZXQgZnJvbSB0aGUgc3Vydml2YWwgcGFja2FnZSBpbnRvIFIncyBtZW1vcnkKZGF0YShwYmMsIHBhY2thZ2U9InN1cnZpdmFsIikKCiMgY29weSBpdCB0byBhbiBvYmplY3Qgd2l0aCB0aGUgZ2VuZXJhbCBuYW1lIGBkYXRhYCAoPC0gYXNzaWducyBhIHZhbHVlIHRvIGEgbmFtZSkKZGF0YSA8LSBwYmMKYGBgCgpXZSBhbHNvIGFkZCBhIGBjZW50ZXJgIHZhcmlhYmxlLiAqKk5vdGU6IHRoaXMgdmFyaWFibGUgaXMgYXJ0aWZpY2lhbC4qKiBJdCBzaW1wbHkgYWx0ZXJuYXRlcyBiZXR3ZWVuIDEgYW5kIDIsIHNvIGl0IGNhcnJpZXMgbm8gcmVhbCBiZXR3ZWVuLWNlbnRlciBkaWZmZXJlbmNlcy4gVGhhdCBpcyBmaW5lIGZvciBhIGRlbW9uc3RyYXRpb247IHdlIHVzZSBpdCBpbiBUdXRvcmlhbCAyIHRvIGlsbHVzdHJhdGUgaW50ZXJuYWwtZXh0ZXJuYWwgY3Jvc3MtdmFsaWRhdGlvbi4KCmBgYHtyIHAxLWNlbnRlcn0KIyBhZGQgY2VudGVyIHZhcmlhYmxlIGZvciBpbGx1c3RyYXRpb24gKGFydGlmaWNpYWwsIGZpbmUgZm9yIGEgZGVtbykKIyByZXAoMToyLCBsZW5ndGgub3V0PW4pIHJlcGVhdHMgMSwgMiwgMSwgMiwgLi4uIHVudGlsIHdlIGhhdmUgbiB2YWx1ZXMsIHdpdGgKIyBuID0gbnJvdyhkYXRhKSB0aGUgbnVtYmVyIG9mIHBhdGllbnRzOyBgZGF0YSRjZW50ZXIgPC1gIHN0b3JlcyB0aGVtIGFzIGEgbmV3IGNvbHVtbgpkYXRhJGNlbnRlciA8LSByZXAoMToyLCBsZW5ndGgub3V0PW5yb3coZGF0YSkpCmBgYAoKKipQaWNraW5nIGEgY29sdW1uOiBgJGAgdmVyc3VzIGBbW2AuKiogSGVyZSBhbmQgYmVsb3cgd2UgdXNlIGBkYXRhJGNlbnRlcmAgdG8gcGljayBhIGNvbHVtbiwgd2hpY2ggaXMgc2hvcnQgdG8gdHlwZS4gUiBoYXMgdHdvIHdheXMgdG8gdGFrZSBvbmUgY29sdW1uIG91dCBvZiBhIGRhdGEgZnJhbWUsIGFuZCB0aGV5IGJlaGF2ZSBkaWZmZXJlbnRseToKCiogYFtgIGtlZXBzIHRoZSBjb250YWluZXIsIGBbW2AgdGFrZXMgb25lIGVsZW1lbnQgb3V0IG9mIGl0OiBgZGF0YVsiY2VudGVyIl1gIGlzIHN0aWxsIGEgZGF0YSBmcmFtZSwgd2hlcmVhcyBgZGF0YVtbImNlbnRlciJdXWAgaXMgdGhlIGNvbHVtbiBpdHNlbGYuIGBkYXRhJGNlbnRlcmAgYWxzbyBnaXZlcyB0aGUgY29sdW1uIGl0c2VsZi4KKiBgJGAgZG9lcyAqcGFydGlhbCBtYXRjaGluZyo6IGBkYXRhJGNlbmAgc2lsZW50bHkgZ2l2ZXMgeW91IHRoZSBgY2VudGVyYCBjb2x1bW4uIEluIHRoZSBjb25zb2xlIHRoYXQgaXMgY29udmVuaWVudCwgYnV0IGluIGEgc2NyaXB0IGEgdHlwbyByZXR1cm5zIHRoZSB3cm9uZyBjb2x1bW4gaW5zdGVhZCBvZiBhbiBlcnJvci4KKiBgJGAgdGFrZXMgdGhlIG5hbWUgbGl0ZXJhbGx5LCBzbyBpdCBjYW5ub3QgdXNlIGEgY29sdW1uIG5hbWUgdGhhdCBpcyBzdG9yZWQgaW4gYSB2YXJpYWJsZS4gYFtbYCBldmFsdWF0ZXMgdGhlIHZhcmlhYmxlIGZpcnN0LCB3aGljaCBpcyB3aGF0IG1ha2VzIGxvb3BzIG92ZXIgdmFyaWFibGUgbmFtZXMgcG9zc2libGUgKHdlIHVzZSB0aGlzIGluIFBhcnQgMykuCgpgYGB7ciBwMS1kb2xsYXItZXhhbXBsZX0KIyB0aGUgY29sdW1uIGl0c2VsZgpoZWFkKGRhdGEkY2VudGVyKQpoZWFkKGRhdGFbWyJjZW50ZXIiXV0pCmNsYXNzKGRhdGFbImNlbnRlciJdKSAgICAgICAgICAjIGEgZGF0YSBmcmFtZSB3aXRoIG9uZSBjb2x1bW4KY2xhc3MoZGF0YVtbImNlbnRlciJdXSkgICAgICAgICMgdGhlIGNvbHVtbiBpdHNlbGYKCiMgcGFydGlhbCBtYXRjaGluZyB3aXRoICQKaGVhZChkYXRhJGNlbikKCiMgY29sdW1uIG5hbWUgc3RvcmVkIGluIGEgdmFyaWFibGUKbXlfdmFyIDwtICJjZW50ZXIiCmRhdGEkbXlfdmFyICAgICAgICAgICAgICAgICAgICAjIE5VTEw6IGxvb2tzIGZvciBhIGNvbHVtbiBjYWxsZWQgbXlfdmFyCmhlYWQoZGF0YVtbbXlfdmFyXV0pICAgICAgICAgICAjIHdvcmtzCmBgYAoKSW4gc2NyaXB0cywgYFtbIm5hbWUiXV1gIGlzIHRoZXJlZm9yZSBzYWZlciwgYW5kIGl0IGlzIHRoZSBvbmx5IG9wdGlvbiB3aGVuIHRoZSBjb2x1bW4gbmFtZSBpcyBzdG9yZWQgaW4gYSB2YXJpYWJsZS4KCiMjIFNlbGVjdCB2YXJpYWJsZXMgYW5kIHNldCB2YXJpYWJsZSB0eXBlcwoKU2VsZWN0IHRoZSBvdXRjb21lIHZhcmlhYmxlcyAoYHRpbWVgIGFuZCBgc3RhdHVzYCkgYW5kIHRoZSB2YXJpYWJsZXMgdGhhdCB5b3Ugd2FudCB0byBkZXNjcmliZSBhbmQgdXNlIGFzIGNhbmRpZGF0ZSBwcmVkaWN0b3JzLCBiYXNlZCBvbiBsaXRlcmF0dXJlIGFuZCBjbGluaWNhbCBrbm93bGVkZ2UuIEJpbmFyeSBhbmQgY2F0ZWdvcmljYWwgdmFyaWFibGVzIG11c3QgYmUgYGZhY3RvcmAgdmFyaWFibGVzLCBvdGhlcndpc2UgUiB0cmVhdHMgdGhlbSBhcyBudW1iZXJzIGFuZCByZXBvcnRzIGEgbWVhbiBhbmQgU0QgZm9yIHRoZW0uCgpUd28gdmFyaWFibGVzIGRlc2VydmUgYXR0ZW50aW9uOiBgdHJ0YCAodHJlYXRtZW50LCBjb2RlZCAxLzIpIGFuZCBgZWRlbWFgIChjb2RlZCAwLCAwLjUsIDEpIGFyZSBjYXRlZ29yaWVzLCBub3QgcXVhbnRpdGllcywgc28gdGhleSBhcmUgYWxzbyBjb252ZXJ0ZWQgdG8gZmFjdG9ycy4gV2Ugc2V0IGV2ZXJ5IGZhY3RvciBzZXBhcmF0ZWx5IHdpdGggYGZhY3RvcigpYDogYGxldmVsc2AgYXJlIHRoZSB2YWx1ZXMgYXMgdGhleSBhcmUgc3RvcmVkIGluIHRoZSBkYXRhIGFuZCBgbGFiZWxzYCBhcmUgdGhlIG5hbWVzIHRoYXQgd2Ugd2FudCB0byBzZWUgaW4gdGFibGVzLCBmaWd1cmVzIGFuZCBtb2RlbCBvdXRwdXQuIFRoZSBsYWJlbHMgbXVzdCBiZSBpbiB0aGUgc2FtZSBvcmRlciBhcyB0aGUgbGV2ZWxzLiBDaGVjayB0aGUgY29kZWJvb2sgb2YgeW91ciBkYXRhIHRvIHNlZSB3aGF0IGVhY2ggdmFsdWUgbWVhbnMgKGluIGBwYmNgLCBgZWRlbWFgIDAgPSBubyBlZGVtYSwgMC41ID0gZWRlbWEgdW50cmVhdGVkIG9yIHN1Y2Nlc3NmdWxseSB0cmVhdGVkLCAxID0gZWRlbWEgZGVzcGl0ZSBkaXVyZXRpYyB0aGVyYXB5LCBhbmQgYHNleGAgaXMgc3RvcmVkIGFzICJtIiBhbmQgImYiKS4KCioqTm90ZTogY2hlY2sgaG93IGJpbmFyeSB2YXJpYWJsZXMgYXJlIGNvZGVkLioqIGB0cnRgIGlzIGNvZGVkIDEvMiBoZXJlLiBUaGlzIG9mdGVuIGhhcHBlbnMgd2hlbiB5b3UgbG9hZCBkYXRhIGZyb20gU1BTUywgd2hlcmUgY2F0ZWdvcmllcyBhcmUgc3RvcmVkIGFzIDEvMiBpbnN0ZWFkIG9mIDAvMS4gQWx3YXlzIGNoZWNrIHRoaXMgKGZvciBleGFtcGxlIHdpdGggYHRhYmxlKGRhdGEkdHJ0KWAgb3IgYHN0cihkYXRhKWApLCBiZWNhdXNlIGEgdmFyaWFibGUgY29kZWQgMS8yIGlzIGVhc2lseSBtaXN0YWtlbiBmb3IgYSBxdWFudGl0eSwgYW5kIHRoZSByZWZlcmVuY2UgY2F0ZWdvcnkgaXMgbm90IG9idmlvdXMuIEl0IGlzIHByb2JhYmx5IGJlc3QgdG8gcmVjb2RlIGJpbmFyeSB2YXJpYWJsZXMgdG8gMC8xIChmb3IgZXhhbXBsZSAwID0gbm8vY29udHJvbCwgMSA9IHllcy90cmVhdG1lbnQpLCBvciB0byBnaXZlIHRoZW0gcmVhZGFibGUgbGFiZWxzIGFzIHdlIGRvIGJlbG93LiBOZXZlciByZWx5IG9uIGBhcy5mYWN0b3IoKWAgYWxvbmUsIGJlY2F1c2UgaXQga2VlcHMgdGhlIHN0b3JlZCB2YWx1ZXMgKDEgYW5kIDIpIGFzIGxhYmVscy4KCmBgYHtyIHAxLXNlbGVjdH0KIyBzZWxlY3QgdGhlIG91dGNvbWUgYW5kIGNhbmRpZGF0ZSB2YXJpYWJsZXMgYmFzZWQgb24gbGl0ZXJhdHVyZQojIChkcGx5cjo6c2VsZWN0KCkga2VlcHMgb25seSB0aGUgY29sdW1ucyB0aGF0IHdlIG5hbWUpCmRhdGEgPC0gZHBseXI6OnNlbGVjdChkYXRhLCB0aW1lLCBzdGF0dXMsCiAgICAgICAgICAgICAgIHRydCwgYWdlLCBzZXgsIGFzY2l0ZXMsIGhlcGF0bywgc3BpZGVycywgZWRlbWEsCiAgICAgICAgICAgICAgIGJpbGksIGNob2wsIGFsYnVtaW4sIGNvcHBlciwgYWxrLnBob3MsIGFzdCwgdHJpZywKICAgICAgICAgICAgICAgcGxhdGVsZXQsIHByb3RpbWUsIHN0YWdlLCBjZW50ZXIpCgojIGJpbmFyeSBhbmQgY2F0ZWdvcmljYWwgdmFyaWFibGVzIHNob3VsZCBiZSBmYWN0b3IgdmFyaWFibGVzOgojIGZhY3Rvcih4LCBsZXZlbHM9Li4uLCBsYWJlbHM9Li4uKSB0YWtlcyB0aGUgdmFsdWVzIHRoYXQgYXJlIHN0b3JlZCBpbiB0aGUgZGF0YQojIChsZXZlbHMpIGFuZCBnaXZlcyBlYWNoIG9mIHRoZW0gYSByZWFkYWJsZSBsYWJlbCAoaW4gdGhlIHNhbWUgb3JkZXIpCmRhdGEkdHJ0ICAgICA8LSBmYWN0b3IoZGF0YSR0cnQsIGxldmVscz1jKDEsIDIpLAogICAgICAgICAgICAgICAgICAgICAgIGxhYmVscz1jKCJELXBlbmljaWxsYW1pbmUiLCAiUGxhY2VibyIpKQpkYXRhJHNleCAgICAgPC0gZmFjdG9yKGRhdGEkc2V4LCBsZXZlbHM9YygibSIsICJmIiksCiAgICAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoIk1hbGUiLCAiRmVtYWxlIikpCmRhdGEkYXNjaXRlcyA8LSBmYWN0b3IoZGF0YSRhc2NpdGVzLCBsZXZlbHM9YygwLCAxKSwKICAgICAgICAgICAgICAgICAgICAgICBsYWJlbHM9YygiTm8iLCAiWWVzIikpCmRhdGEkaGVwYXRvICA8LSBmYWN0b3IoZGF0YSRoZXBhdG8sIGxldmVscz1jKDAsIDEpLAogICAgICAgICAgICAgICAgICAgICAgIGxhYmVscz1jKCJObyIsICJZZXMiKSkKZGF0YSRzcGlkZXJzIDwtIGZhY3RvcihkYXRhJHNwaWRlcnMsIGxldmVscz1jKDAsIDEpLAogICAgICAgICAgICAgICAgICAgICAgIGxhYmVscz1jKCJObyIsICJZZXMiKSkKZGF0YSRlZGVtYSAgIDwtIGZhY3RvcihkYXRhJGVkZW1hLCBsZXZlbHM9YygwLCAwLjUsIDEpLAogICAgICAgICAgICAgICAgICAgICAgIGxhYmVscz1jKCJObyBlZGVtYSIsICJVbnRyZWF0ZWQgb3Igc3VjY2Vzc2Z1bGx5IHRyZWF0ZWQiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJEZXNwaXRlIGRpdXJldGljcyIpKQpkYXRhJHN0YWdlICAgPC0gZmFjdG9yKGRhdGEkc3RhZ2UsIGxldmVscz0xOjQsCiAgICAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoIjEiLCAiMiIsICIzIiwgIjQiKSkKZGF0YSRjZW50ZXIgIDwtIGZhY3RvcihkYXRhJGNlbnRlciwgbGV2ZWxzPWMoMSwgMiksCiAgICAgICAgICAgICAgICAgICAgICAgbGFiZWxzPWMoIkNlbnRlciAxIiwgIkNlbnRlciAyIikpCmBgYAoKKipZb3VyIHR1cm46KiogcnVuIGBzdHIoZGF0YSlgIGFuZCBjaGVjayB0aGF0IGV2ZXJ5IHZhcmlhYmxlIGhhcyB0aGUgdHlwZSB5b3UgZXhwZWN0LgoKKipDaGVjayB0aGUgbGFiZWxzLioqIGBmYWN0b3IoKWAgc2lsZW50bHkgdHVybnMgZXZlcnkgdmFsdWUgdGhhdCBpcyBub3QgbGlzdGVkIGluIGBsZXZlbHNgIGludG8gYSBtaXNzaW5nIHZhbHVlIChgTkFgKSwgZm9yIGV4YW1wbGUgd2hlbiBhIGxldmVsIGlzIG1pc3NwZWxsZWQgb3IgdGhlIGRhdGEgYXJlIGNvZGVkIGRpZmZlcmVudGx5IHRoYW4geW91IHRoaW5rLiBBZnRlciBzZXR0aW5nIHRoZSBsYWJlbHMsIGFsd2F5cyBjb3VudCB0aGUgdmFsdWVzIHBlciBsZXZlbCBhbmQgaW5jbHVkZSB0aGUgbWlzc2luZyB2YWx1ZXMgd2l0aCBgdXNlTkE9ImlmYW55ImAsIGFuZCBjb21wYXJlIHRoZSBjb3VudHMgd2l0aCB0aG9zZSBiZWZvcmUgcmVjb2Rpbmc6CgpgYGB7ciBwMS1jaGVjay1mYWN0b3JzfQojIHRhYmxlKCkgY291bnRzIGhvdyBvZnRlbiBlYWNoIHZhbHVlIG9jY3VyczsgdXNlTkE9ImlmYW55IiBhbHNvIGNvdW50cyB0aGUKIyBtaXNzaW5nIHZhbHVlcyAoTkEpLCBzbyB0aGF0IHdyb25nbHkgbGFiZWxsZWQgdmFsdWVzIHNob3cgdXAKdGFibGUoZGF0YSR0cnQsIHVzZU5BPSJpZmFueSIpCnRhYmxlKGRhdGEkc2V4LCB1c2VOQT0iaWZhbnkiKQp0YWJsZShkYXRhJGVkZW1hLCB1c2VOQT0iaWZhbnkiKQpgYGAKClRoZSAqKmZpcnN0IGxldmVsIGlzIHRoZSByZWZlcmVuY2UgY2F0ZWdvcnkqKiBpbiBtb2RlbHM6IHRoZSBjb2VmZmljaWVudHMgb2YgYSBmYWN0b3IgY29tcGFyZSBlYWNoIG90aGVyIGxldmVsIHdpdGggdGhlIGZpcnN0IG9uZS4gVGhhdCBpcyB3aHkgdGhlIG1vZGVsIG91dHB1dCBsYXRlciBzaG93cyBuYW1lcyBzdWNoIGFzIGBhc2NpdGVzPVllc2AgKGNvbXBhcmVkIHdpdGggIk5vIikgYW5kIGBzZXg9RmVtYWxlYCAoY29tcGFyZWQgd2l0aCAiTWFsZSIpLiBVc2UgYHJlbGV2ZWwoeCwgcmVmPSIuLi4iKWAgdG8gY2hvb3NlIGFub3RoZXIgcmVmZXJlbmNlIGNhdGVnb3J5LgoKIyMgTWFrZSBUYWJsZSAxCgpgYXJzZW5hbDo6dGFibGVieSgpYCBzdW1tYXJpemVzIGVhY2ggdmFyaWFibGUsIGhlcmUgYnkgYGNlbnRlcmAuIFdlIHNob3cgdGhlIG1lYW4gKFNEKSwgbWVkaWFuIChRMSwgUTMpLCByYW5nZSBhbmQgbnVtYmVyIG9mIG1pc3NpbmcgdmFsdWVzIGZvciBjb250aW51b3VzIHZhcmlhYmxlcywgYW5kIGNvdW50cyAocGVyY2VudGFnZXMpIGFuZCBtaXNzaW5nIHZhbHVlcyBmb3IgY2F0ZWdvcmljYWwgdmFyaWFibGVzLiBUaGUgb3V0Y29tZSB2YXJpYWJsZXMgYHRpbWVgIGFuZCBgc3RhdHVzYCBhcmUgbGVmdCBvdXQgb2YgdGhlIHRhYmxlLiBXZSBzd2l0Y2ggb2ZmIHRoZSBzdGF0aXN0aWNhbCB0ZXN0cyAoYHRlc3Q9RkFMU0VgKSBiZWNhdXNlIGEgVGFibGUgMSBpbiBhIHByZWRpY3Rpb24gbW9kZWwgcGFwZXIgZGVzY3JpYmVzIHRoZSBwb3B1bGF0aW9uIGFuZCBkb2VzIG5vdCBjb21wYXJlIGdyb3Vwcy4KCmBgYHtyIHAxLXRhYmxlLW9uZX0KIyBjZW50ZXIgfiAuIG1lYW5zOiBvbmUgY29sdW1uIHBlciBjZW50ZXIsIGRlc2NyaWJpbmcgYWxsIG90aGVyIHZhcmlhYmxlcyAoLikKIyBkYXRhPS4uLjogYWxsIHZhcmlhYmxlcyBleGNlcHQgdGltZSBhbmQgc3RhdHVzICgtdGltZSwgLXN0YXR1cyBsZWF2ZXMgdGhlbSBvdXQpCnRhYmxlX29uZSA8LSBhcnNlbmFsOjp0YWJsZWJ5KGNlbnRlciB+IC4sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGE9ZHBseXI6OnNlbGVjdChkYXRhLCAtdGltZSwgLXN0YXR1cyksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRvdGFsLnBvcz0iYmVmb3JlIiwgICAjIHB1dCB0aGUgdG90YWwgY29sdW1uIGZpcnN0CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRpZ2l0cz0xLCAgICAgICAgICAgICAjIG51bWJlciBvZiBkZWNpbWFscwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0ZXN0PUZBTFNFLCAgICAgICAgICAgIyBubyBwLXZhbHVlcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0b3RhbD1UUlVFLCAgICAgICAgICAgIyBhZGQgYSBjb2x1bW4gZm9yIGFsbCBwYXRpZW50cwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBudW1lcmljLnRlc3Q9Imt3dCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNhdC50ZXN0PSJjaGlzcSIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNhdC5zaW1wbGlmeT1UUlVFLCAgICAjIG9uZSByb3cgZm9yIGJpbmFyeSB2YXJpYWJsZXMKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBzdGF0aXN0aWNzIHNob3duIGZvciBjb250aW51b3VzIHZhcmlhYmxlcyAuLi4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbnVtZXJpYy5zdGF0cz1jKCJtZWFuc2QiLCAibWVkaWFucTFxMyIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAicmFuZ2UiLCAiTm1pc3MyIiksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgLi4uIGFuZCBmb3IgY2F0ZWdvcmljYWwgdmFyaWFibGVzCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNhdC5zdGF0cz1jKCJjb3VudHBjdCIsICJObWlzczIiKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIyByZWFkYWJsZSBuYW1lcyBmb3IgdGhlIHN0YXRpc3RpY3MKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3RhdHMubGFiZWxzPWxpc3QoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWVhbnNkPSJNZWFuIChTRCkiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1lZGlhbnExcTM9Ik1lZGlhbiAoUTEsIFEzKSIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcmFuZ2U9Ik1pbiAtIE1heCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgTm1pc3MyPSJNaXNzaW5nIikpCmBgYAoKSGVyZSBpcyB0aGUgdGFibGUgYXMgaXQgd2lsbCBsb29rIGluIHRoZSByZXBvcnQ6CgpgYGB7ciBwMS10YWJsZS1vbmUtaHRtbCwgcmVzdWx0cz0iYXNpcyJ9CnN1bW1hcnkodGFibGVfb25lLCB0ZXh0PSJodG1sIikKYGBgCgojIyBTYXZlIFRhYmxlIDEgdG8gV29yZAoKYGBge3IgcDEtdGFibGUtb25lLXdvcmR9CiMgY29udmVydCB0aGUgdGFibGUgdG8gYSBkYXRhIGZyYW1lLCBtYWtlIGEgZmxleHRhYmxlIG9mIGl0IGFuZCBzYXZlIGl0IGFzIGEKIyBXb3JkIGZpbGUgaW4gdGhlIHdvcmtpbmcgZGlyZWN0b3J5LiBUaGUgcGlwZSB8PiBwYXNzZXMgdGhlIHJlc3VsdCBvbiB0byB0aGUKIyBuZXh0IGZ1bmN0aW9uCmZsZXh0YWJsZTo6ZmxleHRhYmxlKGRhdGEuZnJhbWUoc3VtbWFyeSh0YWJsZV9vbmUsIHRleHQ9VFJVRSkpKSB8PgogIGZsZXh0YWJsZTo6c2F2ZV9hc19kb2N4KHBhdGg9IkRlc2NyaXB0aXZlcy5kb2N4IikKYGBgCgojIyBFeGVyY2lzZXM6IFBhcnQgMQoKVHJ5IGVhY2ggZXhlcmNpc2UgeW91cnNlbGYgZmlyc3QuIFRoZW4gb3BlbiB0aGUgZmllbGQgYmVsb3cgaXQgdG8gc2VlIHRoZSBhbnN3ZXIsIHRoZSBjb2RlIGFuZCBpdHMgcmVzdWx0cy4gVGhlIGFuc3dlcnMgdXNlIG5ldyBvYmplY3QgbmFtZXMsIHNvIHRoZXkgZG8gbm90IGNoYW5nZSBhbnl0aGluZyBpbiB0aGUgdHV0b3JpYWwuCgoqKkV4ZXJjaXNlIDEuKiogV2hpY2ggdmFyaWFibGVzIGhhdmUgdGhlIG1vc3QgbWlzc2luZyB2YWx1ZXM/IFRoaW5rIGFib3V0IHdoYXQgdGhpcyBtZWFucyBmb3IgdGhlIG1vZGVsbGluZyBpbiBQYXJ0IDMuCgo8ZGV0YWlscz4KPHN1bW1hcnk+U2hvdyBhbnN3ZXIsIGNvZGUgYW5kIHJlc3VsdHM8L3N1bW1hcnk+CgpDb3VudCB0aGUgbWlzc2luZyB2YWx1ZXMgcGVyIHZhcmlhYmxlLCBhbmQgdGhlIHBlcmNlbnRhZ2Ugb2YgcGF0aWVudHMgd2l0aCBhIGNvbXBsZXRlIHJlY29yZDoKCmBgYHtyIGV4LXAxLTEsIGVycm9yPVRSVUV9CiMgbnVtYmVyIG9mIG1pc3NpbmcgdmFsdWVzIHBlciB2YXJpYWJsZSwgbW9zdCBtaXNzaW5nIGZpcnN0CnNvcnQoY29sU3Vtcyhpcy5uYShkYXRhKSksIGRlY3JlYXNpbmc9VFJVRSkKCiMgcGVyY2VudGFnZSBvZiBwYXRpZW50cyB3aXRob3V0IGFueSBtaXNzaW5nIHZhbHVlCnJvdW5kKDEwMCptZWFuKGNvbXBsZXRlLmNhc2VzKGRhdGEpKSwgMSkKYGBgCgpTZXZlcmFsIHZhcmlhYmxlcyBhcmUgbWlzc2luZyBmb3IgbWFueSBwYXRpZW50cy4gSWYgeW91IGRlbGV0ZWQgZXZlcnkgcGF0aWVudCB3aXRoIGF0IGxlYXN0IG9uZSBtaXNzaW5nIHZhbHVlIChjb21wbGV0ZSBjYXNlIGFuYWx5c2lzKSwgeW91IHdvdWxkIHRocm93IGF3YXkgYSBsYXJnZSBwYXJ0IG9mIHRoZSBkYXRhLiBJbiBgcGJjYCwgdGhlIHBhdGllbnRzIHdobyBkaWQgbm90IHRha2UgcGFydCBpbiB0aGUgcmFuZG9taXplZCB0cmlhbCAobWlzc2luZyBgdHJ0YCkgYWxzbyBoYXZlIG1hbnkgb3RoZXIgbWlzc2luZyB2YWx1ZXMsIHNvIHRoZSBtaXNzaW5nbmVzcyBpcyByZWxhdGVkIHRvIG90aGVyIHZhcmlhYmxlcy4gQm90aCBhcmUgcmVhc29ucyB0byBpbXB1dGUgKFBhcnQgMykgaW5zdGVhZCBvZiB1c2luZyBjb21wbGV0ZSBjYXNlcy4KCjwvZGV0YWlscz4KCioqRXhlcmNpc2UgMi4qKiBDaGFuZ2UgYGNlbnRlciB+IC5gIHRvIGB0cnQgfiAuYCBvciBgc2V4IH4gLmAgYW5kIGxvb2sgYXQgdGhlIG5ldyB0YWJsZS4gV2hpY2ggd291bGQgYmUgbW9zdCB1c2VmdWwgaW4gYSBwYXBlcj8KCjxkZXRhaWxzPgo8c3VtbWFyeT5TaG93IGFuc3dlciwgY29kZSBhbmQgcmVzdWx0czwvc3VtbWFyeT4KCmBgYHtyIGV4LXAxLTIsIHJlc3VsdHM9ImFzaXMiLCBlcnJvcj1UUlVFfQp0YWJsZV90cnQgPC0gYXJzZW5hbDo6dGFibGVieSh0cnQgfiAuLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkYXRhPWRwbHlyOjpzZWxlY3QoZGF0YSwgLXRpbWUsIC1zdGF0dXMpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0ZXN0PUZBTFNFLCBkaWdpdHM9MSkKc3VtbWFyeSh0YWJsZV90cnQsIHRleHQ9Imh0bWwiKQpgYGAKCldoaWNoIHNwbGl0IGlzIG1vc3QgdXNlZnVsIGRlcGVuZHMgb24gdGhlIHB1cnBvc2UuIEluIGEgcHJlZGljdGlvbiBtb2RlbCBwYXBlciwgVGFibGUgMSB1c3VhbGx5IGRlc2NyaWJlcyB0aGUgd2hvbGUgZGV2ZWxvcG1lbnQgY29ob3J0LCBzb21ldGltZXMgc3BsaXQgYnkgY2VudGVyIG9yIGJ5IHRoZSBvdXRjb21lLiBBIHRhYmxlIGJ5IHRyZWF0bWVudCBoZWxwcyB0byBjaGVjayB3aGV0aGVyIHRoZSB0cmVhdG1lbnQgZ3JvdXBzIGFyZSBjb21wYXJhYmxlLiBOb3RlIHRoYXQgcGF0aWVudHMgd2l0aCBtaXNzaW5nIGB0cnRgIGRvIG5vdCBhcHBlYXIgaW4gZWl0aGVyIGNvbHVtbi4KCjwvZGV0YWlscz4KCioqRXhlcmNpc2UgMy4qKiBSZW1vdmUgYCJObWlzczIiYCBmcm9tIGBjYXQuc3RhdHNgIGFuZCBzZWUgd2hhdCBjaGFuZ2VzLgoKPGRldGFpbHM+CjxzdW1tYXJ5PlNob3cgYW5zd2VyLCBjb2RlIGFuZCByZXN1bHRzPC9zdW1tYXJ5PgoKYGBge3IgZXgtcDEtMywgcmVzdWx0cz0iYXNpcyIsIGVycm9yPVRSVUV9CnRhYmxlX25vX21pc3MgPC0gYXJzZW5hbDo6dGFibGVieShjZW50ZXIgfiAuLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGF0YT1kcGx5cjo6c2VsZWN0KGRhdGEsIC10aW1lLCAtc3RhdHVzKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRvdGFsLnBvcz0iYmVmb3JlIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRpZ2l0cz0xLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGVzdD1GQUxTRSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRvdGFsPVRSVUUsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjYXQuc2ltcGxpZnk9VFJVRSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG51bWVyaWMuc3RhdHM9YygibWVhbnNkIiwgIm1lZGlhbnExcTMiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJyYW5nZSIsICJObWlzczIiKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNhdC5zdGF0cz1jKCJjb3VudHBjdCIpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc3RhdHMubGFiZWxzPWxpc3QoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1lYW5zZD0iTWVhbiAoU0QpIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWVkaWFucTFxMz0iTWVkaWFuIChRMSwgUTMpIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcmFuZ2U9Ik1pbiAtIE1heCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIE5taXNzMj0iTWlzc2luZyIpKQpzdW1tYXJ5KHRhYmxlX25vX21pc3MsIHRleHQ9Imh0bWwiKQpgYGAKClRoZSAiTWlzc2luZyIgcm93cyBvZiB0aGUgY2F0ZWdvcmljYWwgdmFyaWFibGVzIGRpc2FwcGVhciwgc28geW91IGNhbiBubyBsb25nZXIgc2VlIGhvdyBtYW55IHZhbHVlcyBhcmUgbWlzc2luZyBmb3IgdGhlbS4gQWx3YXlzIHJlcG9ydCBtaXNzaW5nIGRhdGEgaW4gVGFibGUgMS4KCjwvZGV0YWlscz4KCioqRXhlcmNpc2UgNC4qKiBPcGVuIGBEZXNjcmlwdGl2ZXMuZG9jeGAgaW4geW91ciBmb2xkZXIuIE5vdGUgdGhhdCB5b3Ugc3RpbGwgbmVlZCB0byBhZGp1c3QgdGhpbmdzIGJlZm9yZSB5b3UgY2FuIHB1dCB0aGlzIHRhYmxlIGluIGEgbWFudXNjcmlwdC4gV2hhdCB3b3VsZCB5b3UgY2hhbmdlPwoKPGRldGFpbHM+CjxzdW1tYXJ5PlNob3cgYW5zd2VyLCBjb2RlIGFuZCByZXN1bHRzPC9zdW1tYXJ5PgoKVGhlIFdvcmQgdGFibGUgaXMgY3JlYXRlZCBmcm9tIHRoZSBzYW1lIHRhYmxlIGFzIGFib3ZlLCB3aXRoIGdlbmVyaWMgY29sdW1uIG5hbWVzOgoKYGBge3IgZXgtcDEtNCwgZXJyb3I9VFJVRX0KbmFtZXMoZGF0YS5mcmFtZShzdW1tYXJ5KHRhYmxlX29uZSwgdGV4dD1UUlVFKSkpCmBgYAoKVGhpbmdzIHRvIGNoZWNrIG9yIGltcHJvdmUgaW4gV29yZDogdGhlIGNvbHVtbiBoZWFkZXJzIChyZXBsYWNlIHRoZSBnZW5lcmljIG5hbWVzIGJ5IGNsZWFyIG9uZXMpLCB0aGUgbnVtYmVyIG9mIGRlY2ltYWxzLCByZWFkYWJsZSB2YXJpYWJsZSBsYWJlbHMgYW5kIHVuaXRzIChmb3IgZXhhbXBsZSAiYWxrLnBob3MiIGlzIG5vdCBhIGxhYmVsIGZvciBhIHBhcGVyKSwgYW5kIGFkZCBhIHRhYmxlIGNhcHRpb24gYW5kIGEgZm9vdG5vdGUgdGhhdCBleHBsYWlucyB0aGUgYWJicmV2aWF0aW9ucy4KCjwvZGV0YWlscz4KCiMgRGVzY3JpYmUgdGhlIHN1cnZpdmFsIGRhdGEKCk91ciBvdXRjb21lIGlzIHRpbWUgdG8gZGVhdGgsIHdoaWNoIGlzICp0aW1lLXRvLWV2ZW50KiBkYXRhOiBzb21lIHBhdGllbnRzIGFyZSBmb2xsb3dlZCB1bnRpbCB0aGV5IGRpZSwgb3RoZXJzIGFyZSBjZW5zb3JlZCAoc3RpbGwgYWxpdmUgYXQgbGFzdCBjb250YWN0KS4gSW4gdGhpcyBwYXJ0IHlvdSBkZWZpbmUgdGhlIG91dGNvbWUsIGRyYXcgYSBLYXBsYW4tTWVpZXIgY3VydmUsIHJlc3RyaWN0IHRoZSBvdXRjb21lIHRvIGEgY2xpbmljYWxseSByZWxldmFudCB0aW1lIGhvcml6b24sIGFuZCBjYWxjdWxhdGUgdGhlIG1lZGlhbiBmb2xsb3ctdXAgdGltZSBjb3JyZWN0bHkuCgojIyBEZWZpbmUgdGhlIG91dGNvbWUKClRpbWUtdG8tZXZlbnQgZGF0YSBhcmUgdGhlIG1vc3QgY29tbW9uIHR5cGUgb2Ygb3V0Y29tZSBkYXRhIGluIGNsaW5pY2FsIHByZWRpY3Rpb24gcmVzZWFyY2gsIHNvIGl0IGlzIHdvcnRoIGxlYXJuaW5nIGhvdyB0byB3b3JrIHdpdGggdGhlbS4gKElmIHlvdXIgb3duIG91dGNvbWUgaXMgYmluYXJ5IHdpdGhvdXQgZm9sbG93LXVwIHRpbWUsIHlvdSBjYW4gc2tpcCB0aGUgcGFydHMgYWJvdXQgZm9sbG93LXVwLikgV2UgY3JlYXRlIGEgYFN1cnZgIG9iamVjdCB0aGF0IGNvbWJpbmVzIGZvbGxvdy11cCB0aW1lIGFuZCBldmVudCBzdGF0dXMuIFdlIGNvbnZlcnQgdGltZSBmcm9tIGRheXMgdG8geWVhcnMsIGFuZCBkaWNob3RvbWl6ZSB0aGUgZXZlbnQ6IGluIGBwYmNgIHRoZSBzdGF0dXMgaXMgY29kZWQgMCA9IGNlbnNvcmVkLCAxID0gdHJhbnNwbGFudCwgMiA9IGRlYWQsIGFuZCB3ZSBtb2RlbCBkZWF0aC4KCioqSW1wb3J0YW50OioqIHdpdGggYHN0YXR1cz09MmAgd2UgdHJlYXQgYSBsaXZlciB0cmFuc3BsYW50IGFzIGEgKmNlbnNvcmVkKiBvYnNlcnZhdGlvbi4gV2UgZG8gKipub3QqKiB0cmVhdCB0cmFuc3BsYW50IGFzIGEgY29tcGV0aW5nIHJpc2suIFRoaXMgaXMgYSBzaW1wbGlmaWNhdGlvbjogY2Vuc29yaW5nIGF0IHRyYW5zcGxhbnQgYXNzdW1lcyB0aGF0IHBhdGllbnRzIHdobyByZWNlaXZlIGEgdHJhbnNwbGFudCB3b3VsZCBvdGhlcndpc2UgaGF2ZSB0aGUgc2FtZSByaXNrIG9mIGRlYXRoIGFzIHRob3NlIHdobyByZW1haW4gaW4gZm9sbG93LXVwLCB3aGljaCBpcyB1bmxpa2VseSB0byBiZSB0cnVlLiBJZiB0cmFuc3BsYW50IGlzIGFuIGltcG9ydGFudCBjb21wZXRpbmcgZXZlbnQgaW4geW91ciBvd24gcHJvamVjdCwgdXNlIGEgY29tcGV0aW5nIHJpc2sgYXBwcm9hY2ggKGZvciBleGFtcGxlIGNhdXNlLXNwZWNpZmljIGhhemFyZHMgb3IgRmluZS1HcmF5KS4KCmBgYHtyIHAyLXN1cnZ9CiMgZ2VuZXJhdGUgdGhlIFN1cnYgb2JqZWN0IGZvciB0aGUgb3V0Y29tZTogaXQgY29tYmluZXMgZm9sbG93LXVwIHRpbWUgYW5kIGV2ZW50CiMgdGltZTogdHJhbnNmb3JtIHRoZSB0aW1lIGZyb20gZGF5cyB0byB5ZWFycyAoZGl2aWRlIGJ5IDM2NSkKIyBldmVudDogc3RhdHVzPT0yIGlzIFRSVUUgZm9yIHBhdGllbnRzIHdobyBkaWVkIGFuZCBGQUxTRSBmb3IgdGhlIG90aGVyczsKIyAgICAgICAgYXMubnVtZXJpYygpIHR1cm5zIFRSVUUvRkFMU0UgaW50byAxLzAKIyAodHJhbnNwbGFudCBpcyB0cmVhdGVkIGFzIGNlbnNvcmVkLCBOT1QgYXMgYSBjb21wZXRpbmcgcmlzaykKUyA8LSBzdXJ2aXZhbDo6U3Vydih0aW1lPWRhdGEkdGltZS8zNjUsIGV2ZW50PWFzLm51bWVyaWMoZGF0YSRzdGF0dXM9PTIpKQpgYGAKCiMjIEthcGxhbi1NZWllciBjdXJ2ZQoKVGhlIEthcGxhbi1NZWllciBtZXRob2QgZXN0aW1hdGVzIHN1cnZpdmFsIG92ZXIgdGltZSB3aGlsZSBjb3JyZWN0bHkgaGFuZGxpbmcgY2Vuc29yaW5nLiBXZSBtYWtlIG9uZSBjdXJ2ZSBwZXIgdHJlYXRtZW50IGdyb3VwLCBoZXJlIGZvciBhbGwgZm9sbG93LXVwICh1bnJlc3RyaWN0ZWQsIGBTYCkuCgpgYGB7ciBwMi1rbX0KIyBtYWtlIEthcGxhbi1NZWllciBjdXJ2ZSAoYWxsIGZvbGxvdy11cCkKIyB0aGUgZm9ybXVsYSBTIH4gdHJ0IG1lYW5zOiBvbmUgc3Vydml2YWwgY3VydmUgZm9yIGVhY2ggdHJlYXRtZW50IGdyb3VwCktNX3VucmVzdHJpY3RlZCA8LSBzdXJ2aXZhbDo6c3VydmZpdChTIH4gdHJ0LCBkYXRhPWRhdGEpCmBgYAoKKipOb3RlOiBzb21lIHBhdGllbnRzIGFyZSBzaWxlbnRseSBkcm9wcGVkLioqIGB0cnRgIGlzIG1pc3NpbmcgZm9yIHNvbWUgcGF0aWVudHMgKHRob3NlIG5vdCBpbiB0aGUgcmFuZG9taXplZCB0cmlhbCksIGFuZCBgc3VydmZpdCgpYCByZW1vdmVzIHRoZW0gd2l0aG91dCBhIHdhcm5pbmcuIENoZWNrIGhvdyBtYW55OgoKYGBge3IgcDIta20tbWlzc2luZ30KIyBudW1iZXIgb2YgcGF0aWVudHMgd2l0aCBtaXNzaW5nIHRyZWF0bWVudCwgdGhlc2UgYXJlIG5vdCBpbiB0aGUgY3VydmVzCnN1bShpcy5uYShkYXRhJHRydCkpCmBgYAoKT25seSB0aGUgcmVtYWluaW5nIHBhdGllbnRzIGFyZSB1c2VkIGluIHRoZSBjdXJ2ZXMuIEluIHRoZSBwcmVkaWN0aW9uIG1vZGVscyBpbiBQYXJ0IDQgd2UgZG8ga2VlcCBldmVyeW9uZSwgYmVjYXVzZSBtaXNzaW5nIHZhbHVlcyBhcmUgaW1wdXRlZC4KCkZpcnN0IGxvb2sgYXQgdGhlIGN1cnZlIHdpdGggdGhlIGJhc2ljIGBwbG90KClgIGZ1bmN0aW9uOgoKYGBge3IgcDIta20tYmFzZX0KIyBiYXNpYyBLYXBsYW4tTWVpZXIgY3VydmU6IG9uZSBjb2xvciBwZXIgdHJlYXRtZW50IGdyb3VwLCB3aXRoIGF4aXMgbGFiZWxzCnBsb3QoS01fdW5yZXN0cmljdGVkLCBjb2w9YygiYmx1ZSIsICJyZWQiKSwgeGxhYj0iVGltZSAoeWVhcnMpIiwgeWxhYj0iU3Vydml2YWwiKQpgYGAKCkZvciBhIG5pY2VyIGZpZ3VyZSB3aXRoIGNvbmZpZGVuY2UgaW50ZXJ2YWxzIGFuZCBhIHJpc2sgdGFibGUgd2UgdXNlIGBnZ3N1cnZwbG90KClgIGZyb20gYHN1cnZtaW5lcmAuIAoKYGBge3IgcDIta20tcGxvdC11bnJlc3RyaWN0ZWR9CiMgS2FwbGFuLU1laWVyIGN1cnZlIHdpdGggcmlzayB0YWJsZSAoYWxsIGZvbGxvdy11cCkKS01fcGxvdF91bnJlc3RyaWN0ZWQgPC0gc3Vydm1pbmVyOjpnZ3N1cnZwbG90KEtNX3VucmVzdHJpY3RlZCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGE9ZGF0YSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJpc2sudGFibGU9VFJVRSwgICAgICAgICAgICMgdGFibGUgYmVsb3cgdGhlIGN1cnZlCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICByaXNrLnRhYmxlLnkudGV4dD1GQUxTRSwgICAjIG5vIGdyb3VwIG5hbWVzIGluIHRoZSB0YWJsZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGVnZW5kPSJ0b3AiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGVnZW5kLnRpdGxlPSJUcmVhdG1lbnQiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGVnZW5kLmxhYnM9bGV2ZWxzKGRhdGEkdHJ0KSwgICMgbmFtZXMgb2YgdGhlIGdyb3VwcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeGxhYj0iVGltZSAoeWVhcnMpIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbmYuaW50PVRSVUUpICAgICAgICAgICAgICMgY29uZmlkZW5jZSBpbnRlcnZhbHMKIyB0eXBpbmcgdGhlIG5hbWUgb2YgYW4gb2JqZWN0IHByaW50cyBpdApLTV9wbG90X3VucmVzdHJpY3RlZApgYGAKClR3byBkZXRhaWxzIGluIHRoZSBjb2RlIG1ha2UgdGhlIGZpZ3VyZSB0aWRpZXIuIGBsZWdlbmQubGFicz1sZXZlbHMoZGF0YSR0cnQpYCByZXBsYWNlcyB0aGUgZGVmYXVsdCBsYWJlbHMgKGB0cnQ9RC1wZW5pY2lsbGFtaW5lYCwgYHRydD1QbGFjZWJvYCkgb2YgdGhlIHR3byB0cmVhdG1lbnQgYXJtcyBpbiB0aGUgbGVnZW5kIGJ5IHRoZSBmYWN0b3IgbGFiZWxzIHdlIHNldCBlYXJsaWVyOyB0byB1c2Ugb3RoZXIgbmFtZXMsIGdpdmUgeW91ciBvd24gdmVjdG9yLCBpbiB0aGUgc2FtZSBvcmRlciBhcyB0aGUgbGV2ZWxzLCBmb3IgZXhhbXBsZSBgbGVnZW5kLmxhYnM9YygiUGVuaWNpbGxhbWluZSIsICJDb250cm9sIilgLiBBbmQgYHJpc2sudGFibGUueS50ZXh0PUZBTFNFYCByZW1vdmVzIHRoZSBsb25nIGdyb3VwIG5hbWVzIGZyb20gdGhlIHktYXhpcyBvZiB0aGUgcmlzayB0YWJsZSAodGhlIGNvbG9ycyBpbiB0aGUgdGFibGUgbWF0Y2ggdGhlIGxlZ2VuZCksIHNvIHRoYXQgdGhlIHktYXhlcyBvZiB0aGUgY3VydmUgYW5kIHRoZSB0YWJsZSBsaW5lIHVwLgoKTG9vayBhdCB0aGUgcmlnaHQgZW5kIG9mIHRoZSBjdXJ2ZXM6IHRoZSBjb25maWRlbmNlIGludGVydmFscyBiZWNvbWUgKip3aWRlcioqLCBhbmQgdGhlIHJpc2sgdGFibGUgc2hvd3Mgd2h5OiBmZXdlciBhbmQgZmV3ZXIgcGF0aWVudHMgYXJlIHN0aWxsIGZvbGxvd2VkIGFmdGVyIGFib3V0IDEwIHllYXJzLCBzbyB0aGUgZXN0aW1hdGVzIHRoZXJlIGFyZSBsZXNzIGNlcnRhaW4uIEEgcHJlZGljdGlvbiBtb2RlbCBzaG91bGQgbm90IHJlbHkgbXVjaCBvbiB0aGlzIHBhcnQgb2YgdGhlIGZvbGxvdy11cC4gTW9yZW92ZXIsIGEgcHJlZGljdGlvbiBob3Jpem9uIG9mIDEwIHllYXJzIG1heSBiZSBtb3JlIGNsaW5pY2FsbHkgdXNlZnVsIHRoYW4gb25lIG9mIDEyIHllYXJzLiBUaGF0IGlzIHdoeSB3ZSAqKnRydW5jYXRlIHRoZSBvdXRjb21lIGF0IDEwIHllYXJzKiogaW4gdGhlIG5leHQgc2VjdGlvbjsgZm9yIHRoZSBwdXJwb3NlIG9mIHRoaXMgZXhlcmNpc2Ugd2UgdXNlIDEwIHllYXJzIGFzIHRoZSBob3Jpem9uLgoKIyMgTGltaXQgdGhlIG91dGNvbWUgdG8gdGhlIHRpbWUgaG9yaXpvbgoKQSBwcmVkaWN0aW9uIG1vZGVsIHByZWRpY3RzIHJpc2sgYXQgYSBzcGVjaWZpYyB0aW1lIHBvaW50LiBXZSBjaG9vc2UgMTAgeWVhcnMuIEV2ZXJ5b25lIGZvbGxvd2VkIGxvbmdlciB0aGFuIDEwIHllYXJzIGlzIGNlbnNvcmVkIGF0IDEwIHllYXJzLCBiZWNhdXNlIGV2ZW50cyBhZnRlciB0aGUgaG9yaXpvbiBhcmUgbm90IHBhcnQgb2YgdGhlIG91dGNvbWUgd2UgcHJlZGljdC4KCmBgYHtyIHAyLWhvcml6b259CiMgbGltaXQgdGhlIG91dGNvbWUgdG8geW91ciB0aW1lIGhvcml6b24KaG9yaXpvbiA8LSAxMApTXzEwIDwtIFMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgc3RhcnQgZnJvbSBhIGNvcHkgb2YgdGhlIGZ1bGwgb3V0Y29tZQojIFNfMTBbLCAxXSBpcyB0aGUgdGltZSBjb2x1bW4gYW5kIFNfMTBbLCAyXSBpcyB0aGUgZXZlbnQgY29sdW1uOwojIFNfMTBbLCAxXSA+IGhvcml6b24gaXMgVFJVRSBmb3IgZXZlcnlvbmUgZm9sbG93ZWQgbG9uZ2VyIHRoYW4gMTAgeWVhcnMKU18xMFtTXzEwWywgMV0+aG9yaXpvbiwgMl0gPC0gMCAgICAgICAjIGZvciB0aGVtOiBubyBldmVudCAoY2Vuc29yZWQpClNfMTBbU18xMFssIDFdPmhvcml6b24sIDFdIDwtIGhvcml6b24gIyBhbmQgdGhlIHRpbWUgYmVjb21lcyAxMCB5ZWFycwpgYGAKCk5vdyBmaXQgdGhlIEthcGxhbi1NZWllciBjdXJ2ZSBmb3IgdGhlIHJlc3RyaWN0ZWQgb3V0Y29tZSBgU18xMGAgYW5kIG1ha2UgdGhlIHNhbWUgZmlndXJlIGFzIGJlZm9yZToKCmBgYHtyIHAyLWttLXBsb3R9CiMgdGhlIHNhbWUgS2FwbGFuLU1laWVyIGZpdCwgbm93IGZvciB0aGUgb3V0Y29tZSByZXN0cmljdGVkIHRvIDEwIHllYXJzCktNXzEwIDwtIHN1cnZpdmFsOjpzdXJ2Zml0KFNfMTAgfiB0cnQsIGRhdGE9ZGF0YSkKCiMgS2FwbGFuLU1laWVyIGN1cnZlIHdpdGggcmlzayB0YWJsZSwgcmVzdHJpY3RlZCB0byAxMCB5ZWFycwpLTV9wbG90IDwtIHN1cnZtaW5lcjo6Z2dzdXJ2cGxvdChLTV8xMCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGF0YT1kYXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICByaXNrLnRhYmxlPVRSVUUsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJpc2sudGFibGUueS50ZXh0PUZBTFNFLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsZWdlbmQ9InRvcCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxlZ2VuZC50aXRsZT0iVHJlYXRtZW50IiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGVnZW5kLmxhYnM9bGV2ZWxzKGRhdGEkdHJ0KSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeGxhYj0iVGltZSAoeWVhcnMpIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY29uZi5pbnQ9VFJVRSkKS01fcGxvdAoKIyBjb21iaW5lIHRoZSBjdXJ2ZSBhbmQgdGhlIHJpc2sgdGFibGUgYW5kIHNhdmUKZ2dwbG90Mjo6Z2dzYXZlKGZpbGVuYW1lPSJLTV9wbG90XzEwX3llYXJzLnBuZyIsCiAgICAgICBwbG90PWdncHVicjo6Z2dhcnJhbmdlKEtNX3Bsb3QkcGxvdCwgS01fcGxvdCR0YWJsZSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbnJvdz0yLCBoZWlnaHRzPWMoMywgMSksIGFsaWduPSJ2IiksCiAgICAgICB3aWR0aD03LCBoZWlnaHQ9NywgZHBpPTMwMCkKYGBgCgpOb3cgZGlzcGxheSB0aGUgS2FwbGFuLU1laWVyIGVzdGltYXRlcyBhdCB0aGUgaG9yaXpvbiwgZm9yIHRoZSB1bnJlc3RyaWN0ZWQgYW5kIHRoZSByZXN0cmljdGVkIG91dGNvbWU6CgpgYGB7ciBwMi1rbS1zdW1tYXJ5fQojIGRpc3BsYXkgdGhlIEthcGxhbi1NZWllciBlc3RpbWF0ZXMgKHN1cnZpdmFsIHByb2JhYmlsaXR5KSBhdCB0aGUgaG9yaXpvbiwKIyB0aW1lPWhvcml6b24gYXNrcyBmb3IgdGhlIGVzdGltYXRlIGF0IDEwIHllYXJzIG9ubHkKc3VtbWFyeShLTV91bnJlc3RyaWN0ZWQsIHRpbWU9aG9yaXpvbikKc3VtbWFyeShLTV8xMCwgdGltZT1ob3Jpem9uKQpgYGAKCiMjIE1lZGlhbiBmb2xsb3ctdXAgdGltZQoKVGhlIG1lZGlhbiBvZiB0aGUgb2JzZXJ2ZWQgdGltZXMgaXMgKm5vdCogdGhlIG1lZGlhbiBmb2xsb3ctdXAgdGltZSwgYmVjYXVzZSBwYXRpZW50cyB3aG8gZGllZCBoYXZlIGEgc2hvcnQgb2JzZXJ2ZWQgdGltZSBhbmQgZHJhZyB0aGUgbWVkaWFuIGRvd24uIFRoZSBjb3JyZWN0IGVzdGltYXRlIHVzZXMgdGhlICpyZXZlcnNlIEthcGxhbi1NZWllciogbWV0aG9kOiB3ZSBmbGlwIHRoZSByb2xlcyBvZiBldmVudCBhbmQgY2Vuc29yaW5nLCBzbyB0aGUgImV2ZW50IiBpcyBiZWluZyBjZW5zb3JlZCBhbmQgd2UgZXN0aW1hdGUgaG93IGxvbmcgcGF0aWVudHMgYXJlIGZvbGxvd2VkIGluIGEgdHlwaWNhbCBzaXR1YXRpb24uIEluIGBzdXJ2Zml0KClgIHRoaXMgbWVhbnMgdGhhdCB0aGUgZXZlbnQgaW5kaWNhdG9yIGJlY29tZXMgYDEgLSBzdGF0dXNgLgoKRmlyc3QgcGxvdCB0aGUgcmV2ZXJzZSBLYXBsYW4tTWVpZXIgY3VydmUuIEl0IHNob3dzIHRoZSBwcm9iYWJpbGl0eSB0aGF0IGEgcGF0aWVudCBpcyBzdGlsbCBiZWluZyBmb2xsb3dlZCBhdCBlYWNoIHRpbWUgcG9pbnQ7IHRoZSB0aW1lIGF0IHdoaWNoIHRoaXMgY3VydmUgY3Jvc3NlcyAwLjUgaXMgdGhlIG1lZGlhbiBmb2xsb3ctdXAgdGltZS4KCmBgYHtyIHAyLWZvbGxvdy11cC1wbG90fQojIHJldmVyc2UgS2FwbGFuLU1laWVyOiB0aGUgImV2ZW50IiBpcyBiZWluZyBjZW5zb3JlZCAobm90IGR5aW5nKQojIFNbLCAidGltZSJdIGlzIHRoZSB0aW1lIGNvbHVtbiBvZiBTOyAxIC0gU1ssICJzdGF0dXMiXSBmbGlwcyAxIGFuZCAwLCBzbwojIHRoYXQgYmVpbmcgY2Vuc29yZWQgY291bnRzIGFzIHRoZSBldmVudApyZXZlcnNlX0tNIDwtIHN1cnZpdmFsOjpzdXJ2Zml0KHN1cnZpdmFsOjpTdXJ2KFNbLCAidGltZSJdLCAxIC0gU1ssICJzdGF0dXMiXSkgfiAxLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGE9ZGF0YSwgY29uZi50eXBlPSJwbGFpbiIpCnN1cnZtaW5lcjo6Z2dzdXJ2cGxvdChyZXZlcnNlX0tNLAogICAgICAgICAgICAgICAgICAgICAgZGF0YT1kYXRhLAogICAgICAgICAgICAgICAgICAgICAgY29uZi5pbnQ9VFJVRSwKICAgICAgICAgICAgICAgICAgICAgIGxlZ2VuZD0ibm9uZSIsCiAgICAgICAgICAgICAgICAgICAgICB4bGFiPSJUaW1lICh5ZWFycykiLAogICAgICAgICAgICAgICAgICAgICAgeWxhYj0iUHJvYmFiaWxpdHkgb2Ygc3RpbGwgYmVpbmcgZm9sbG93ZWQiKQpgYGAKClRoZW4gY2FsY3VsYXRlIHRoZSBtZWRpYW4gZm9sbG93LXVwIHRpbWUgZnJvbSB0aGlzIGN1cnZlOgoKYGBge3IgcDItZm9sbG93LXVwfQojIG1lZGlhbiBmb2xsb3ctdXAgdGltZSB3aXRoIHRoZSByZXZlcnNlIEthcGxhbi1NZWllciBtZXRob2QgKGluIHllYXJzKQojIHByb2JzPTAuNSBhc2tzIGZvciB0aGUgNTB0aCBwZXJjZW50aWxlLCB0aGUgbWVkaWFuCnN0YXRzOjpxdWFudGlsZShyZXZlcnNlX0tNLCBwcm9icz0wLjUpCmBgYAoKVGhlIHJlc3VsdCBnaXZlcyB0aGUgbWVkaWFuIGZvbGxvdy11cCB0aW1lICh3aXRoIGBwcm9icz0wLjVgKSB3aXRoIGl0cyBjb25maWRlbmNlIGludGVydmFsLiBXZSB1c2UgYGNvbmYudHlwZT0icGxhaW4iYCBpbiBgc3VydmZpdCgpYCB0byBnZXQgdGhlIHNhbWUgY29uZmlkZW5jZSBpbnRlcnZhbCBhcyBgcHJvZGxpbWAgYmVsb3c7IHRoZSBkZWZhdWx0IG9mIGBzdXJ2Zml0KClgIGlzIGBjb25mLnR5cGU9ImxvZyJgLCB3aGljaCBnaXZlcyBhIHNsaWdodGx5IGRpZmZlcmVudCBsb3dlciBsaW1pdC4KClRoZSBgcHJvZGxpbWAgcGFja2FnZSBkb2VzIHRoZSBzYW1lIGluIG9uZSBsaW5lOiB3aXRoIGByZXZlcnNlPVRSVUVgIGl0IGZsaXBzIGV2ZW50IGFuZCBjZW5zb3JpbmcgZm9yIHlvdSwgc28geW91IGRvIG5vdCBoYXZlIHRvIGRlZmluZSBgMSAtIHN0YXR1c2AgeW91cnNlbGYuIEluc3RhbGwgaXQgb25jZSB3aXRoIGBpbnN0YWxsLnBhY2thZ2VzKCJwcm9kbGltIilgIGlmIHlvdSBkbyBub3QgaGF2ZSBpdCB5ZXQuIFRoZSByb3cgZm9yIGAwLjVgIGdpdmVzIHRoZSBzYW1lIG1lZGlhbiBmb2xsb3ctdXAgdGltZSBhbmQgY29uZmlkZW5jZSBpbnRlcnZhbC4gRG8gbm90IGNvbmZ1c2UgdGhpcyBpbnRlcnZhbCB3aXRoIHRoZSBsaW5lICJNZWRpYW4gd2l0aCBpbnRlcnF1YXJ0aWxlIHJhbmdlIChJUVIpIiBiZWxvdyB0aGUgdGFibGU6IHRoYXQgaXMgdGhlIG1lZGlhbiB0b2dldGhlciB3aXRoIHRoZSAyNXRoIGFuZCA3NXRoIHBlcmNlbnRpbGUgb2YgdGhlIGZvbGxvdy11cCB0aW1lcyAodGhlIHJvd3MgZm9yIGAwLjc1YCBhbmQgYDAuMjVgKSwgd2hpY2ggZGVzY3JpYmVzIGhvdyBtdWNoIHRoZSBmb2xsb3ctdXAgdmFyaWVzIGJldHdlZW4gcGF0aWVudHMuIEl0IGlzIG5vdCBhIGNvbmZpZGVuY2UgaW50ZXJ2YWwgb2YgdGhlIG1lZGlhbi4KCmBgYHtyIHAyLWZvbGxvdy11cC1wcm9kbGltfQojIHRoZSBzYW1lIGluIG9uZSBsaW5lIHdpdGggcHJvZGxpbQpzdGF0czo6cXVhbnRpbGUocHJvZGxpbTo6cHJvZGxpbShwcm9kbGltOjpIaXN0KHRpbWUvMzY1LCBzdGF0dXM9PTIpIH4gMSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZGF0YT1kYXRhLCByZXZlcnNlPVRSVUUpKQpgYGAKCiMjIFJlc3RyaWN0ZWQgbWVhbiBzdXJ2aXZhbCB0aW1lIChSTVNUKQoKVGhlICpyZXN0cmljdGVkIG1lYW4gc3Vydml2YWwgdGltZSogKFJNU1QpIGlzIHRoZSBhcmVhIHVuZGVyIHRoZSBLYXBsYW4tTWVpZXIgY3VydmUgdXAgdG8gYSBjaG9zZW4gdGltZSBwb2ludC4gSXQgaXMgdGhlIGF2ZXJhZ2UgbnVtYmVyIG9mIHllYXJzIGEgcGF0aWVudCBpcyBhbGl2ZSBkdXJpbmcgdGhhdCBwZXJpb2QsIGZvciBleGFtcGxlIGR1cmluZyB0aGUgZmlyc3QgMTAgeWVhcnMuIEZvciBhIHF1aWNrIGd1aWRlLCBzZWUgW0hhbiAmIEp1bmcgKDIwMjIpXShodHRwczovL3BtYy5uY2JpLm5sbS5uaWguZ292L2FydGljbGVzL1BNQzkwODE2ODYvKS4KCkEgaGF6YXJkIHJhdGlvIGlzIGEgcmVsYXRpdmUgbWVhc3VyZSB0aGF0IGFzc3VtZXMgdGhlIGVmZmVjdCBpcyBjb25zdGFudCBvdmVyIHRpbWUgKHByb3BvcnRpb25hbCBoYXphcmRzKSBhbmQgaXMgaGFyZCB0byB0cmFuc2xhdGUgaW50byBhbiBhYnNvbHV0ZSBlZmZlY3QuIFRoZSBSTVNUIGlzIGluIHRoZSB1bml0IG9mIHRpbWUgKHllYXJzIGdhaW5lZCBvciBsb3N0KSwgc28gaXQgaXMgZWFzeSB0byBpbnRlcnByZXQsIGl0IGRvZXMgbm90IG5lZWQgdGhlIHByb3BvcnRpb25hbCBoYXphcmRzIGFzc3VtcHRpb24sIGFuZCBpdCBjYW4gc3RpbGwgc2hvdyBhIGRpZmZlcmVuY2Ugd2hlbiB0aGUgc3Vydml2YWwgY3VydmVzIGNyb3NzLiBUaGUgZGlmZmVyZW5jZSBpbiBSTVNUIGJldHdlZW4gdHdvIGdyb3VwcyAodGhlIGFyZWEgYmV0d2VlbiB0aGVpciBLYXBsYW4tTWVpZXIgY3VydmVzKSBhbHNvIGdpdmVzIGEgYmV0dGVyIHBpY3R1cmUgb2YgdGhlIGFic29sdXRlIHRyZWF0bWVudCBiZW5lZml0IHRoYW4gdGhlIHJpc2sgZGlmZmVyZW5jZSBhdCBhIHNpbmdsZSB0aW1lIHBvaW50LCB3aGljaCBjYW4gbGVhZCB0byB0cmVhdGluZyB0aGUgd3JvbmcgcGF0aWVudHMuIEZvciB0aGF0IHJlYXNvbiBpdCBpcyByZWNvbW1lbmRlZCBmb3IgYXNzZXNzaW5nIGFic29sdXRlIHRyZWF0bWVudCBlZmZlY3RzIGluIHRpbWUtdG8tZXZlbnQgZGF0YSAoW01hYXMgZXQgYWwuLCAyMDI1XShodHRwczovL3B1Ym1lZC5uY2JpLm5sbS5uaWguZ292LzQxMzE3Mzc3LykpLgoKWW91IGNhbiBnZXQgaXQgZnJvbSB0aGUgYHN1bW1hcnkoKWAgb2YgdGhlIEthcGxhbi1NZWllciBmaXQgYnkgc2V0dGluZyBgcm1lYW5gIHRvIHRoZSB0aW1lIHBvaW50LiBUaGUgcmVzdWx0IGNvbnRhaW5zIGEgYHRhYmxlYCB3aXRoIHRoZSBSTVNUIChgcm1lYW5gKSBhbmQgaXRzIHN0YW5kYXJkIGVycm9yIChgc2Uocm1lYW4pYCkgcGVyIGdyb3VwLiBXZSB1c2UgdGhlIG91dGNvbWUgcmVzdHJpY3RlZCB0byB0aGUgMTAteWVhciBob3Jpem9uOgoKYGBge3IgcDItcm1zdH0KIyByZXN0cmljdGVkIG1lYW4gc3Vydml2YWwgdGltZSB1cCB0byB0aGUgaG9yaXpvbiAoMTAgeWVhcnMpCiMgJHRhYmxlIHRha2VzIHRoZSB0YWJsZSB3aXRoIHRoZSBSTVNUIG91dCBvZiB0aGUgc3VtbWFyeQpzdW1tYXJ5KEtNXzEwLCBybWVhbj1ob3Jpem9uKSR0YWJsZQoKIyBleHRyYWN0IHRoZSBSTVNUIChhbmQgaXRzIHN0YW5kYXJkIGVycm9yKSBmb3Igb25lIHRyZWF0bWVudCBhcm06CiMgW3JvdyBuYW1lLCBjb2x1bW4gbmFtZXNdIHBpY2tzIG9uZSByb3cgYW5kIHR3byBjb2x1bW5zIG9mIHRoZSB0YWJsZQpzdW1tYXJ5KEtNXzEwLCBybWVhbj1ob3Jpem9uKSR0YWJsZVsidHJ0PVBsYWNlYm8iLCBjKCJybWVhbiIsICJzZShybWVhbikiKV0KYGBgCgpGb3IgZXhhbXBsZSwgYW4gYHJtZWFuYCBvZiA3LjEgbWVhbnMgdGhhdCBwYXRpZW50cyBpbiB0aGF0IGdyb3VwIGxpdmUgb24gYXZlcmFnZSA3LjEgb2YgdGhlIGZpcnN0IDEwIHllYXJzLiBUaGUgUk1TVCBuZWVkcyBhIHRpbWUgcG9pbnQgdG8gcmVzdHJpY3QgYXQsIG90aGVyd2lzZSBSIHJlc3RyaWN0cyBhdCB0aGUgbGFyZ2VzdCBvYnNlcnZlZCB0aW1lLiBUaGlzIGlzIGFub3RoZXIgcmVhc29uIHRvIGxpbWl0IHRoZSBvdXRjb21lIHRvIHRoZSBob3Jpem9uLgoKIyMgRXhlcmNpc2VzOiBQYXJ0IDIKClRyeSBlYWNoIGV4ZXJjaXNlIHlvdXJzZWxmIGZpcnN0LiBUaGVuIG9wZW4gdGhlIGZpZWxkIGJlbG93IGl0IHRvIHNlZSB0aGUgYW5zd2VyLCB0aGUgY29kZSBhbmQgaXRzIHJlc3VsdHMuCgoqKkV4ZXJjaXNlIDEuKiogV2hhdCBhcmUgdGhlIEthcGxhbi1NZWllciBlc3RpbWF0ZXMgYXQgNSB5ZWFycz8gSG93IGRvIHRoZXkgY29tcGFyZSB3aXRoIHRoZSBlc3RpbWF0ZXMgYXQgdGhlIDEwLXllYXIgaG9yaXpvbj8KCjxkZXRhaWxzPgo8c3VtbWFyeT5TaG93IGFuc3dlciwgY29kZSBhbmQgcmVzdWx0czwvc3VtbWFyeT4KCkEgS2FwbGFuLU1laWVyIGVzdGltYXRlIGF0IDUgeWVhcnMgZG9lcyBub3QgZGVwZW5kIG9uIGZvbGxvdy11cCBhZnRlciA1IHllYXJzLCBzbyB5b3UgZG8gbm90IG5lZWQgdG8gZGVmaW5lIGEgbmV3IG91dGNvbWUuIFVzZSBgc3VtbWFyeSgpYCBvZiB0aGUgZml0IHdpdGggYFNfMTBgIGFuZCBzZXQgYHRpbWU9NWA6CgpgYGB7ciBleC1wMi0xLCBlcnJvcj1UUlVFfQojIEthcGxhbi1NZWllciBlc3RpbWF0ZXMgYXQgNSB5ZWFycwpzdW1tYXJ5KEtNXzEwLCB0aW1lPTUpCmBgYAoKQ29tcGFyZSB0aGVzZSBlc3RpbWF0ZXMgd2l0aCB0aGUgMTAteWVhciBlc3RpbWF0ZXMgYWJvdmUuIEEgS2FwbGFuLU1laWVyIGN1cnZlIGNhbiBvbmx5IGdvIGRvd24gb3ZlciB0aW1lLCBzbyB0aGUgc3Vydml2YWwgZXN0aW1hdGUgYXQgNSB5ZWFycyBpcyBhdCBsZWFzdCBhcyBoaWdoIGFzIHRoZSBlc3RpbWF0ZSBhdCAxMCB5ZWFycy4KCjwvZGV0YWlscz4KCioqRXhlcmNpc2UgMi4qKiBDYWxjdWxhdGUgdGhlIG5haXZlIG1lZGlhbiwgYG1lZGlhbihkYXRhJHRpbWUpLzM2NWAsIGFuZCBjb21wYXJlIGl0IHdpdGggdGhlIHJldmVyc2UgS2FwbGFuLU1laWVyIGVzdGltYXRlLiBXaGljaCBvbmUgd291bGQgeW91IHJlcG9ydCwgYW5kIHdoeT8KCjxkZXRhaWxzPgo8c3VtbWFyeT5TaG93IGFuc3dlciwgY29kZSBhbmQgcmVzdWx0czwvc3VtbWFyeT4KCmBgYHtyIGV4LXAyLTIsIGVycm9yPVRSVUV9CiMgbmFpdmUgbWVkaWFuIG9mIHRoZSBvYnNlcnZlZCB0aW1lcyAoaW4geWVhcnMpCm1lZGlhbihkYXRhJHRpbWUpLzM2NQoKIyBtZWRpYW4gZm9sbG93LXVwIHdpdGggdGhlIHJldmVyc2UgS2FwbGFuLU1laWVyIG1ldGhvZCAoaW4geWVhcnMpCnN0YXRzOjpxdWFudGlsZShyZXZlcnNlX0tNLCBwcm9icz0wLjUpCmBgYAoKUmVwb3J0IHRoZSByZXZlcnNlIEthcGxhbi1NZWllciBlc3RpbWF0ZS4gVGhlIG5haXZlIG1lZGlhbiBtaXhlcyBwYXRpZW50cyB3aG8gd2VyZSBmb2xsb3dlZCB1bnRpbCB0aGUgZW5kIG9mIHRoZSBzdHVkeSB3aXRoIHBhdGllbnRzIHdob3NlIGZvbGxvdy11cCBzdG9wcGVkIGVhcmx5IGJlY2F1c2UgdGhleSBkaWVkLCB3aGljaCBtYWtlcyB0aGUgdHlwaWNhbCBmb2xsb3ctdXAgbG9vayBzaG9ydGVyIHRoYW4gaXQgaXMuCgo8L2RldGFpbHM+CgoqKkV4ZXJjaXNlIDMuKiogQ2FsY3VsYXRlIHRoZSByZXN0cmljdGVkIG1lYW4gc3Vydml2YWwgdGltZSAoUk1TVCkgdXAgdG8gNSB5ZWFycyBpbnN0ZWFkIG9mIDEwIHllYXJzIGZvciBlYWNoIHRyZWF0bWVudCBncm91cCwgYW5kIGV4dHJhY3QgdGhlIFJNU1Qgb2YgdGhlIHBsYWNlYm8gYXJtIG9ubHkuIEhvdyBtYW55IG9mIHRoZSBmaXJzdCA1IHllYXJzIGRvIHBhdGllbnRzIG9uIGF2ZXJhZ2UgbGl2ZT8KCjxkZXRhaWxzPgo8c3VtbWFyeT5TaG93IGFuc3dlciwgY29kZSBhbmQgcmVzdWx0czwvc3VtbWFyeT4KCmBgYHtyIGV4LXAyLTMsIGVycm9yPVRSVUV9CiMgUk1TVCB1cCB0byA1IHllYXJzIGZvciBib3RoIHRyZWF0bWVudCBhcm1zCnN1bW1hcnkoS01fMTAsIHJtZWFuPTUpJHRhYmxlCgojIGV4dHJhY3QgdGhlIFJNU1QgKGFuZCBpdHMgc3RhbmRhcmQgZXJyb3IpIG9mIHRoZSBwbGFjZWJvIGFybSBvbmx5CnN1bW1hcnkoS01fMTAsIHJtZWFuPTUpJHRhYmxlWyJ0cnQ9UGxhY2VibyIsIGMoInJtZWFuIiwgInNlKHJtZWFuKSIpXQpgYGAKClRoZSBmaXQgYEtNXzEwYCBjb3ZlcnMgMTAgeWVhcnMsIGJ1dCBgcm1lYW49NWAgcmVzdHJpY3RzIHRoZSBhcmVhIHVuZGVyIHRoZSBjdXJ2ZSB0byB0aGUgZmlyc3QgNSB5ZWFycywgc28geW91IGRvIG5vdCBuZWVkIGEgbmV3IGZpdC4gVGhlIFJNU1QgdXAgdG8gNSB5ZWFycyBpcyBhdCBtb3N0IDUsIGFuZCBzbWFsbGVyIHRoYW4gdGhlIFJNU1QgdXAgdG8gMTAgeWVhcnMuCgo8L2RldGFpbHM+CgoqKkV4ZXJjaXNlIDQuKiogV2h5IGRvIHdlIHRydW5jYXRlIHRoZSBvdXRjb21lIGF0IHRoZSBob3Jpem9uIGluc3RlYWQgb2YganVzdCB1c2luZyBhbGwgZm9sbG93LXVwPwoKPGRldGFpbHM+CjxzdW1tYXJ5PlNob3cgYW5zd2VyLCBjb2RlIGFuZCByZXN1bHRzPC9zdW1tYXJ5PgoKYGBge3IgZXgtcDItNCwgZXJyb3I9VFJVRX0KIyBob3cgbWFueSBwYXRpZW50cyBhcmUgZm9sbG93ZWQgZm9yIGxvbmdlciB0aGFuIHRoZSAxMC15ZWFyIGhvcml6b24/CnRhYmxlKFNbLCAxXSA+IGhvcml6b24pCmBgYAoKQSBwcmVkaWN0aW9uIG1vZGVsIGdpdmVzIHRoZSByaXNrIGF0IGEgc3BlY2lmaWMgdGltZSBwb2ludCwgaGVyZSAxMCB5ZWFycywgYW5kIHRoZSBiYXNlbGluZSBoYXphcmQgaXMgZXN0aW1hdGVkIGF0IHRoYXQgdGltZSBwb2ludC4gTGF0ZXIgZm9sbG93LXVwIGlzIGJhc2VkIG9uIGZldyBwYXRpZW50cyBhdCByaXNrIGFuZCB3b3VsZCBub3QgYmUgcGFydCBvZiB0aGUgcmlzayB0aGF0IHdlIHByZWRpY3QuIFRydW5jYXRpbmcgYXQgdGhlIGhvcml6b24ga2VlcHMgdGhlIG91dGNvbWUgY29uc2lzdGVudCB3aXRoIHRoZSBwcmVkaWN0aW9uLgoKPC9kZXRhaWxzPgoKIyBNdWx0aXBsZSBpbXB1dGF0aW9uCgpNb3N0IGNsaW5pY2FsIGRhdGEgc2V0cyBoYXZlIG1pc3NpbmcgdmFsdWVzLiBEZWxldGluZyBldmVyeSBwYXRpZW50IHdpdGggYSBtaXNzaW5nIHZhbHVlIChjb21wbGV0ZSBjYXNlIGFuYWx5c2lzKSB3YXN0ZXMgZGF0YSBhbmQgY2FuIGJpYXMgdGhlIG1vZGVsLiAqTXVsdGlwbGUgaW1wdXRhdGlvbiogZmlsbHMgaW4gbWlzc2luZyB2YWx1ZXMgc2V2ZXJhbCB0aW1lcywgdXNpbmcgdGhlIG9ic2VydmVkIGRhdGEsIHNvIHRoYXQgdGhlIHVuY2VydGFpbnR5IGR1ZSB0byBtaXNzaW5nbmVzcyBpcyByZWZsZWN0ZWQgaW4gdGhlIHJlc3VsdHMuIEluIHRoaXMgcGFydCB5b3UgdmlzdWFsaXplIHRoZSBtaXNzaW5nIGRhdGEsIGJ1aWxkIGFuIGltcHV0YXRpb24gbW9kZWwgdGhhdCBzdWl0cyBzdXJ2aXZhbCBkYXRhLCBhbmQgY2hlY2sgdGhlIGltcHV0ZWQgdmFsdWVzLgoKIyMgVmlzdWFsaXplIHRoZSBtaXNzaW5nIGRhdGEKCkJlZm9yZSBpbXB1dGluZywgbG9vayBhdCBob3cgbXVjaCBpcyBtaXNzaW5nIGFuZCB3aGVyZS4KCmBgYHtyIHAzLW1pc3Npbmd9CiMgdmlzdWFsaXplIG1pc3NpbmduZXNzOiBldmVyeSBjb2x1bW4gaXMgYSB2YXJpYWJsZSwgZXZlcnkgcm93IGEgcGF0aWVudCwKIyBhbmQgdGhlIGJsYWNrIGNlbGxzIGFyZSBtaXNzaW5nIHZhbHVlcwp2aXNkYXQ6OnZpc19taXNzKGRhdGEpCmBgYAoKIyMgQnVpbGQgdGhlIGltcHV0YXRpb24gbW9kZWwgZm9yIHN1cnZpdmFsIGRhdGEKClRoZSBpbXB1dGF0aW9uIG1vZGVsIG11c3QgY29udGFpbiB0aGUgb3V0Y29tZSwgb3RoZXJ3aXNlIHRoZSBpbXB1dGVkIHByZWRpY3RvcnMgYXJlIHVucmVsYXRlZCB0byB0aGUgb3V0Y29tZSBhbmQgdGhlIG1vZGVsJ3MgZWZmZWN0cyBhcmUgYmlhc2VkIHRvd2FyZHMgemVyby4gRm9yIHRpbWUtdG8tZXZlbnQgZGF0YSwgaXQgaXMgKipub3QqKiBhIGdvb2QgaWRlYSB0byBwdXQgYHRpbWVgIGFuZCBgc3RhdHVzYCBpbiB0aGUgaW1wdXRhdGlvbiBtb2RlbCBkaXJlY3RseS4gVGhlIHJlY29tbWVuZGVkIGFwcHJvYWNoIChXaGl0ZSAmIFJveXN0b24sIDIwMDkpIGlzIHRvIGluY2x1ZGU6CgoxLiB0aGUgKipldmVudCBpbmRpY2F0b3IqKiwgYW5kCjIuIHRoZSAqKk5lbHNvbi1BYWxlbiBlc3RpbWF0ZSBvZiB0aGUgY3VtdWxhdGl2ZSBoYXphcmQqKiBhdCBlYWNoIHBhdGllbnQncyBvYnNlcnZlZCB0aW1lLgoKV2UgY2FsY3VsYXRlIGJvdGggb24gdGhlIHNhbWUgb3V0Y29tZSB0aGF0IHdlIG1vZGVsIGxhdGVyOiBkZWF0aCB3aXRoaW4gdGhlIDEwLXllYXIgaG9yaXpvbiAoYFNfMTBgKSwgd2l0aCB0cmFuc3BsYW50IHRyZWF0ZWQgYXMgY2Vuc29yZWQuIFRoZSByYXcgYHRpbWVgIGFuZCBgc3RhdHVzYCB2YXJpYWJsZXMgYXJlIHRoZW4gbGVmdCBvdXQgb2YgdGhlIGltcHV0YXRpb24gbW9kZWwuCgpgYGB7ciBwMy1pbXB1dGF0aW9uLWRhdGF9CiMgb3V0Y29tZSBpbmZvcm1hdGlvbiBmb3IgdGhlIGltcHV0YXRpb24gbW9kZWwsIGJhc2VkIG9uIHRoZSBhbmFseXNpcyBvdXRjb21lIFNfMTAKZGF0YV9taSA8LSBkYXRhICAgICAgICAgICAgICAjIHdvcmsgb24gYSBjb3B5LCBzbyB0aGF0IGBkYXRhYCBzdGF5cyBhcyBpdCBpcwpkYXRhX21pJHRpbWUgPC0gTlVMTCAgICAgICAgICMgTlVMTCByZW1vdmVzIGEgY29sdW1uCmRhdGFfbWkkc3RhdHVzIDwtIE5VTEwKIyAxLiB0aGUgZXZlbnQgaW5kaWNhdG9yICgwLzEpIG9mIHRoZSBvdXRjb21lIHdpdGhpbiAxMCB5ZWFycwpkYXRhX21pJGV2ZW50XzEwIDwtIFNfMTBbLCAic3RhdHVzIl0KIyAyLiB0aGUgTmVsc29uLUFhbGVuIGVzdGltYXRlIG9mIHRoZSBjdW11bGF0aXZlIGhhemFyZCBhdCBlYWNoIHBhdGllbnQncyB0aW1lCmRhdGFfbWkkY3VtX2hhemFyZCA8LSBtaWNlOjpuZWxzb25hYWxlbihkYXRhLmZyYW1lKHRpbWU9U18xMFssICJ0aW1lIl0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHN0YXR1cz1TXzEwWywgInN0YXR1cyJdKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRpbWUsIHN0YXR1cykKYGBgCgojIyBJbXB1dGUKCldlIGNyZWF0ZSBgbSA9IDVgIGltcHV0ZWQgZGF0YSBzZXRzLiBgc2VlZD0xYCBtYWtlcyB0aGUgcmVzdWx0IHJlcHJvZHVjaWJsZS4KCioqTm90ZToqKiBmaXZlIGltcHV0YXRpb25zIGlzIG9uIHRoZSBzbWFsbCBzaWRlLiBBIHJ1bGUgb2YgdGh1bWIgaXMgdG8gdXNlIGF0IGxlYXN0IGFzIG1hbnkgaW1wdXRhdGlvbnMgYXMgdGhlIHBlcmNlbnRhZ2Ugb2YgaW5jb21wbGV0ZSBjYXNlcyAoV2hpdGUgZXQgYWwuLCAyMDExKS4gTGV0J3MgY2hlY2sgdGhhdCBwZXJjZW50YWdlIGhlcmU6CgpgYGB7ciBwMy1mcmFjdGlvbn0KIyBwZXJjZW50YWdlIG9mIHBhdGllbnRzIHdpdGggYXQgbGVhc3Qgb25lIG1pc3NpbmcgdmFsdWUKcm91bmQoMTAwKm1lYW4oIWNvbXBsZXRlLmNhc2VzKGRhdGFfbWkpKSwgMSkKYGBgCgpXaXRoIHRoaXMgcGVyY2VudGFnZSwgYG1gIHNob3VsZCBwcm9iYWJseSBiZSBsYXJnZXIgdGhhbiA1IChtYXliZSA0MCBvciBtb3JlKS4gRm9yIHRoaXMgdHV0b3JpYWwgd2Uga2VlcCBgbSA9IDVgIHNvIHRoYXQgYWxsIGZvbGxvd2luZyBzdGVwcyBydW4gcXVpY2tseSwgYnV0IGluIHlvdXIgb3duIHByb2plY3QgeW91IHNob3VsZCB1c2UgYSBsYXJnZXIgYG1gIGJhc2VkIG9uIHRoZSBmcmFjdGlvbiBvZiBpbmNvbXBsZXRlIGNhc2VzLgoKYGBge3IgcDMtbWljZX0KIyBtdWx0aXBsZSBpbXB1dGF0aW9uIChtIHNob3VsZCBtYXliZSBiZSBsYXJnZXIsIHNlZSB0aGUgdGV4dCBhYm92ZSkKbSA8LSA1ICAgICAgICAgICAgICAgICAgICAgICAjIG51bWJlciBvZiBpbXB1dGVkIGRhdGEgc2V0cwojIG1pY2UoKSBpbXB1dGVzIHRoZSBtaXNzaW5nIHZhbHVlczogbT1udW1iZXIgb2YgZGF0YSBzZXRzLCBzZWVkPTEgbWFrZXMgdGhlCiMgcmVzdWx0IHJlcHJvZHVjaWJsZSwgcHJpbnQ9RkFMU0UgaGlkZXMgdGhlIHByb2dyZXNzIG1lc3NhZ2VzCmltcHV0ZWRfZGF0YSA8LSBtaWNlOjptaWNlKGRhdGFfbWksIG09bSwgc2VlZD0xLCBwcmludD1GQUxTRSkKIyBwcm9ibGVtcyB0aGF0IG1pY2UgcmFuIGludG8gKE5VTEwgbWVhbnMgbm9uZSkKaW1wdXRlZF9kYXRhJGxvZ2dlZEV2ZW50cwpgYGAKCmBsb2dnZWRFdmVudHNgIGxpc3RzIHByb2JsZW1zIHRoYXQgYG1pY2VgIHJhbiBpbnRvOyBgTlVMTGAgbWVhbnMgbm9uZS4KCiMjIEZyYWN0aW9uIG9mIG1pc3NpbmcgaW5mb3JtYXRpb24gKEZNSSkKClRoZSAqZnJhY3Rpb24gb2YgbWlzc2luZyBpbmZvcm1hdGlvbiogKEZNSSkgaXMgdGhlIHBhcnQgb2YgdGhlIHVuY2VydGFpbnR5ICh2YXJpYW5jZSkgb2YgYW4gZXN0aW1hdGUgdGhhdCBpcyBkdWUgdG8gbWlzc2luZyBkYXRhLiBJdCBjYW4gYmUgbXVjaCBsb3dlciBvciBoaWdoZXIgdGhhbiB0aGUgcGVyY2VudGFnZSBvZiBtaXNzaW5nIHZhbHVlcywgYmVjYXVzZSBpdCBkZXBlbmRzIG9uIGhvdyB3ZWxsIHRoZSBvdGhlciB2YXJpYWJsZXMgcHJlZGljdCB0aGUgbWlzc2luZyBvbmVzLiBXZSBjYWxjdWxhdGUgaXQgcGVyIHZhcmlhYmxlOiBmaXQgYW4gaW50ZXJjZXB0LW9ubHkgbW9kZWwgb24gZXZlcnkgaW1wdXRlZCBkYXRhIHNldCAoaXRzIGVzdGltYXRlIGlzIHRoZSBtZWFuLCBvciB0aGUgbG9nIG9kZHMgZm9yIGEgYmluYXJ5IHZhcmlhYmxlKSwgcG9vbCB0aGUgcmVzdWx0cyB3aXRoIFJ1YmluJ3MgcnVsZXMgdXNpbmcgYHBvb2woKWAsIGFuZCB0YWtlIHRoZSBGTUkgZnJvbSB0aGUgcG9vbGVkIHJlc3VsdC4gV2Ugc2tpcCBjYXRlZ29yaWNhbCB2YXJpYWJsZXMgd2l0aCBtb3JlIHRoYW4gMiBsZXZlbHMsIGJlY2F1c2UgYSBzaW5nbGUgaW50ZXJjZXB0IGRvZXMgbm90IGRlc2NyaWJlIHRoZW0uCgpUbyBkbyB0aGlzIGZvciBldmVyeSB2YXJpYWJsZSB3ZSBuZWVkIHR3byBiYXNpYyB0b29sczogYSAqZnVuY3Rpb24qIGFuZCBhICpmb3IgbG9vcCouCgoqKkEgZnVuY3Rpb24qKiBpcyBhIHBpZWNlIG9mIGNvZGUgdGhhdCB5b3UgZ2l2ZSBhIG5hbWUsIHNvIHRoYXQgeW91IGNhbiBydW4gaXQgYWdhaW4gYW5kIGFnYWluIHdpdGggZGlmZmVyZW50IGlucHV0LiBGb3IgZXhhbXBsZSwgYSBmdW5jdGlvbiB0aGF0IHNxdWFyZXMgYSBudW1iZXI6CgpgYGB7ciBwMy1mdW5jdGlvbi1leGFtcGxlfQojIGRlZmluZSBhIGZ1bmN0aW9uOiB4IGlzIHRoZSBpbnB1dCwgdGhlIGxhc3QgbGluZSBpcyB0aGUgb3V0cHV0CnNxdWFyZSA8LSBmdW5jdGlvbih4KXsKICB4XjIKfQpzcXVhcmUoMykKc3F1YXJlKDEwKQpgYGAKCioqQSBmb3IgbG9vcCoqIHJlcGVhdHMgY29kZSBmb3IgZXZlcnkgZWxlbWVudCBvZiBhIHZlY3Rvci4gRm9yIGV4YW1wbGUsIHByaW50IHRoZSBzcXVhcmUgb2YgZWFjaCBudW1iZXIgaW4gYSB2ZWN0b3I6CgpgYGB7ciBwMy1sb29wLWV4YW1wbGV9CiMgdGhlIGxvb3AgcnVucyBvbmNlIGZvciBldmVyeSB2YWx1ZSBvZiBgbnVtYmVyYApmb3IgKG51bWJlciBpbiBjKDIsIDMsIDQpKXsKICBwcmludChzcXVhcmUobnVtYmVyKSkKfQpgYGAKCk5vdyB3ZSB3cml0ZSBhIGZ1bmN0aW9uIHRoYXQgY2FsY3VsYXRlcyB0aGUgRk1JIGZvciBvbmUgdmFyaWFibGUsIGFuZCB1c2UgYSBmb3IgbG9vcCB0byBydW4gaXQgZm9yIGV2ZXJ5IHZhcmlhYmxlIHdpdGggbWlzc2luZyB2YWx1ZXMuCgpgYGB7ciBwMy1mbWl9CiMgLS0tLSAxLiBEZWZpbmUgdGhlIGZ1bmN0aW9uIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KIyBBIGZ1bmN0aW9uIGlzIGRlZmluZWQgd2l0aCBmdW5jdGlvbihhcmd1bWVudCl7IC4uLiB9LiBIZXJlIHRoZSBhcmd1bWVudCBgdmFyYAojIGlzIHRoZSBuYW1lIG9mIG9uZSB2YXJpYWJsZSAoZm9yIGV4YW1wbGUgImNob2wiKSwgYW5kIHRoZSBmdW5jdGlvbiByZXR1cm5zIGEKIyBzbWFsbCB0YWJsZSB3aXRoIHRoZSBwZXJjZW50YWdlIG1pc3NpbmcgYW5kIHRoZSBGTUkgb2YgdGhhdCB2YXJpYWJsZS4KY2FsY3VsYXRlX2ZtaSA8LSBmdW5jdGlvbih2YXIpewogICMgZGF0YV9taVtbdmFyXV0gaXMgdGhlIGNvbHVtbiB3aXRoIHRoZSBuYW1lIHN0b3JlZCBpbiBgdmFyYC4KICAjIGlzLm5hKCkgZ2l2ZXMgVFJVRSBmb3IgbWlzc2luZyB2YWx1ZXMsIG1lYW4oKSBvZiBUUlVFL0ZBTFNFIGdpdmVzIHRoZQogICMgcHJvcG9ydGlvbiBvZiBUUlVFLCBhbmQgMTAwKiB0dXJucyBpdCBpbnRvIGEgcGVyY2VudGFnZSAocm91bmRlZCB0byAxIGRlY2ltYWwpCiAgcGN0X21pc3NpbmcgPC0gcm91bmQoMTAwKm1lYW4oaXMubmEoZGF0YV9taVtbdmFyXV0pKSwgMSkKCiAgIyBXZSBmaXQgYSBtb2RlbCB3aXRoIG9ubHkgYW4gaW50ZXJjZXB0ICgidmFyIH4gMSIpIG9uIGVhY2ggb2YgdGhlIG0gaW1wdXRlZAogICMgZGF0YSBzZXRzLiB3aXRoKGltcHV0ZWRfZGF0YSwgLi4uKSBydW5zIHRoZSBtb2RlbCBvbiBldmVyeSBpbXB1dGVkIGRhdGEgc2V0LgogICMgcGFzdGUodmFyLCAifiAxIikgYnVpbGRzIHRoZSBmb3JtdWxhIGFzIHRleHQsIGZvciBleGFtcGxlICJjaG9sIH4gMSIsIGFuZAogICMgYXMuZm9ybXVsYSgpIHR1cm5zIHRoZSB0ZXh0IGludG8gYSBmb3JtdWxhLgogIGlmIChpcy5mYWN0b3IoZGF0YV9taVtbdmFyXV0pICYmIG5sZXZlbHMoZGF0YV9taVtbdmFyXV0pID09IDIpewogICAgIyBiaW5hcnkgdmFyaWFibGU6IGxvZ2lzdGljIHJlZ3Jlc3Npb24gKHRoZSBpbnRlcmNlcHQgaXMgdGhlIGxvZyBvZGRzKQogICAgZml0IDwtIHdpdGgoaW1wdXRlZF9kYXRhLCBnbG0oYXMuZm9ybXVsYShwYXN0ZSh2YXIsICJ+IDEiKSksIGZhbWlseT1iaW5vbWlhbCkpCiAgfSBlbHNlIGlmIChpcy5udW1lcmljKGRhdGFfbWlbW3Zhcl1dKSl7CiAgICAjIGNvbnRpbnVvdXMgdmFyaWFibGU6IGxpbmVhciByZWdyZXNzaW9uICh0aGUgaW50ZXJjZXB0IGlzIHRoZSBtZWFuKQogICAgZml0IDwtIHdpdGgoaW1wdXRlZF9kYXRhLCBsbShhcy5mb3JtdWxhKHBhc3RlKHZhciwgIn4gMSIpKSkpCiAgfSBlbHNlIHsKICAgICMgb3RoZXIgdmFyaWFibGVzIChjYXRlZ29yaWNhbCB3aXRoIG1vcmUgdGhhbiAyIGxldmVscyk6IG5vIEZNSSwgc28gd2UKICAgICMgcmV0dXJuIGEgcm93IHdpdGggTkEgYW5kIHN0b3AgaGVyZSAocmV0dXJuKCkgZW5kcyB0aGUgZnVuY3Rpb24pCiAgICByZXR1cm4oZGF0YS5mcmFtZSh2YXJpYWJsZT12YXIsIHBjdF9taXNzaW5nPXBjdF9taXNzaW5nLCBmbWk9TkEpKQogIH0KCiAgIyBwb29sKCkgY29tYmluZXMgdGhlIG0gbW9kZWxzIGludG8gb25lIGVzdGltYXRlIHdpdGggUnViaW4ncyBydWxlcy4gVGhlCiAgIyBwb29sZWQgcmVzdWx0IGNvbnRhaW5zIHRoZSBmcmFjdGlvbiBvZiBtaXNzaW5nIGluZm9ybWF0aW9uIChgZm1pYCksIGEKICAjIHByb3BvcnRpb24gYmV0d2VlbiAwIGFuZCAxOyB3ZSBzaG93IGl0IGFzIGEgcGVyY2VudGFnZS4KICBwb29sZWQgPC0gbWljZTo6cG9vbChmaXQpCgogICMgVGhlIGxhc3QgbGluZSBvZiBhIGZ1bmN0aW9uIGlzIGl0cyByZXN1bHQ6IGEgb25lLXJvdyB0YWJsZSBmb3IgdGhpcyB2YXJpYWJsZQogIGRhdGEuZnJhbWUodmFyaWFibGU9dmFyLCBwY3RfbWlzc2luZz1wY3RfbWlzc2luZywKICAgICAgICAgICAgIGZtaT1yb3VuZCgxMDAqcG9vbGVkJHBvb2xlZCRmbWksIDEpKQp9CgojIC0tLS0gMi4gVXNlIHRoZSBmdW5jdGlvbiBpbiBhIGZvciBsb29wIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KIyBjb2xTdW1zKGlzLm5hKGRhdGFfbWkpKSBjb3VudHMgdGhlIG1pc3NpbmcgdmFsdWVzIHBlciBjb2x1bW47IHdlIGtlZXAgdGhlIG5hbWVzCiMgb2YgdGhlIGNvbHVtbnMgdGhhdCBoYXZlIGF0IGxlYXN0IG9uZSBtaXNzaW5nIHZhbHVlLgp2YXJzX21pc3NpbmcgPC0gbmFtZXMoZGF0YV9taSlbY29sU3Vtcyhpcy5uYShkYXRhX21pKSkgPiAwXQoKIyBTdGFydCB3aXRoIGFuIGVtcHR5IHRhYmxlIGFuZCBhZGQgb25lIHJvdyBwZXIgdmFyaWFibGUuIEluIGV2ZXJ5IHJvdW5kIG9mIHRoZQojIGxvb3AgYHZhcmAgaXMgdGhlIG5leHQgbmFtZSBpbiBgdmFyc19taXNzaW5nYDsgcmJpbmQoKSBwdXRzIHRoZSBuZXcgcm93IHVuZGVyCiMgdGhlIHJvd3MgdGhhdCB3ZSBhbHJlYWR5IGhhdmUuCmZtaV9yZXN1bHRzIDwtIGRhdGEuZnJhbWUoKQpmb3IgKHZhciBpbiB2YXJzX21pc3NpbmcpewogIGZtaV9yZXN1bHRzIDwtIHJiaW5kKGZtaV9yZXN1bHRzLCBjYWxjdWxhdGVfZm1pKHZhcikpCn0KCiMgcHJpbnQgdGhlIHRhYmxlCmZtaV9yZXN1bHRzCmBgYAoKQSBoaWdoIEZNSSBtZWFucyB0aGF0IHRoZSBtaXNzaW5nIGRhdGEgYWRkIGEgbG90IG9mIHVuY2VydGFpbnR5IGFuZCB0aGF0IG1vcmUgaW1wdXRhdGlvbnMgYXJlIG5lZWRlZCB0byBlc3RpbWF0ZSBpdCBzdGFibHkuIEEgY29tbW9uIHJ1bGUgb2YgdGh1bWIgaXMgdGhhdCBgbWAgc2hvdWxkIGJlIGF0IGxlYXN0IDEwMCB0aW1lcyB0aGUgbGFyZ2VzdCBGTUkgKGFzIGEgcHJvcG9ydGlvbiwgc28gYW4gRk1JIG9mIDAuMyBnaXZlcyBgbSA9IDMwYCkgKFdoaXRlIGV0IGFsLiwgMjAxMSkuCgojIyBDaGVjayB0aGUgaW1wdXRhdGlvbnMKCkNvbXBhcmUgdGhlIGRpc3RyaWJ1dGlvbiBvZiBvYnNlcnZlZCB2YWx1ZXMgKGJsdWUpIHdpdGggdGhlIGltcHV0ZWQgdmFsdWVzIChyZWQpIGZvciBlYWNoIGltcHV0YXRpb24uIFRoZXkgZG8gbm90IG5lZWQgdG8gYmUgaWRlbnRpY2FsIChtaXNzaW5nbmVzcyBtYXkgZGVwZW5kIG9uIG90aGVyIHZhcmlhYmxlcyksIGJ1dCBzdHJhbmdlIGRpZmZlcmVuY2VzIGRlc2VydmUgYSBjbG9zZXIgbG9vay4KCmBgYHtyIHAzLWRlbnNpdHksIGZpZy5oZWlnaHQ9MTAsIGZpZy53aWR0aD0xMH0KIyB2aXN1YWxpemUgaW1wdXRhdGlvbnMKbWljZTo6ZGVuc2l0eXBsb3QoaW1wdXRlZF9kYXRhKQpgYGAKCiMjIEV4ZXJjaXNlczogUGFydCAzCgpUcnkgZWFjaCBleGVyY2lzZSB5b3Vyc2VsZiBmaXJzdC4gVGhlbiBvcGVuIHRoZSBmaWVsZCBiZWxvdyBpdCB0byBzZWUgdGhlIGFuc3dlciwgdGhlIGNvZGUgYW5kIGl0cyByZXN1bHRzLgoKKipFeGVyY2lzZSAxLioqIFdoaWNoIHZhcmlhYmxlcyBoYXZlIHRoZSBtb3N0IG1pc3NpbmcgdmFsdWVzLCBhbmQgd2hhdCBwZXJjZW50YWdlIG9mIHBhdGllbnRzIGhhdmUgYXQgbGVhc3Qgb25lIG1pc3NpbmcgdmFsdWU/Cgo8ZGV0YWlscz4KPHN1bW1hcnk+U2hvdyBhbnN3ZXIsIGNvZGUgYW5kIHJlc3VsdHM8L3N1bW1hcnk+CgpgYGB7ciBleC1wMy0xLCBlcnJvcj1UUlVFfQojIHBlcmNlbnRhZ2UgbWlzc2luZyBwZXIgdmFyaWFibGUsIG1vc3QgbWlzc2luZyBmaXJzdApyb3VuZCgxMDAqc29ydChjb2xNZWFucyhpcy5uYShkYXRhKSksIGRlY3JlYXNpbmc9VFJVRSksIDEpCgojIHBlcmNlbnRhZ2Ugb2YgcGF0aWVudHMgd2l0aCBhdCBsZWFzdCBvbmUgbWlzc2luZyB2YWx1ZQpyb3VuZCgxMDAqbWVhbighY29tcGxldGUuY2FzZXMoZGF0YV9taSkpLCAxKQpgYGAKClRoZSBmaXJzdCB0YWJsZSBnaXZlcyB0aGUgcGVyY2VudGFnZSBvZiBtaXNzaW5nIHZhbHVlcyBwZXIgdmFyaWFibGUsIHRoZSBzZWNvbmQgbnVtYmVyIHRoZSBwZXJjZW50YWdlIG9mIHBhdGllbnRzIHdpdGggYXQgbGVhc3Qgb25lIG1pc3NpbmcgdmFsdWUuIFRoZSBzZWNvbmQgbnVtYmVyIGlzIHRoZSBvbmUgdXNlZCBmb3IgdGhlIHJ1bGUgb2YgdGh1bWIgZm9yIHRoZSBudW1iZXIgb2YgaW1wdXRhdGlvbnMuCgo8L2RldGFpbHM+CgoqKkV4ZXJjaXNlIDIuKiogTG9vayBhdCB0aGUgZGVuc2l0eSBwbG90cy4gRG8gdGhlIGltcHV0ZWQgdmFsdWVzIChyZWQpIGxvb2sgcGxhdXNpYmxlIGNvbXBhcmVkIHdpdGggdGhlIG9ic2VydmVkIHZhbHVlcyAoYmx1ZSkgZm9yIGBjaG9sYCwgYHRyaWdgIGFuZCBgY29wcGVyYD8KCjxkZXRhaWxzPgo8c3VtbWFyeT5TaG93IGFuc3dlciwgY29kZSBhbmQgcmVzdWx0czwvc3VtbWFyeT4KCmBgYHtyIGV4LXAzLTIsIGVycm9yPVRSVUV9Cm1pY2U6OmRlbnNpdHlwbG90KGltcHV0ZWRfZGF0YSwgfiBjaG9sKQptaWNlOjpkZW5zaXR5cGxvdChpbXB1dGVkX2RhdGEsIH4gdHJpZykKbWljZTo6ZGVuc2l0eXBsb3QoaW1wdXRlZF9kYXRhLCB+IGNvcHBlcikKYGBgCgpDb21wYXJlIHRoZSBzaGFwZSBvZiB0aGUgcmVkIGxpbmVzIChpbXB1dGVkKSB3aXRoIHRoZSBibHVlIGxpbmUgKG9ic2VydmVkKS4gVGhlc2UgdmFyaWFibGVzIGFyZSByaWdodC1za2V3ZWQsIGFuZCB0aGUgaW1wdXRlZCB2YWx1ZXMgZG8gbm90IG5lZWQgdG8gbWF0Y2ggdGhlIG9ic2VydmVkIHZhbHVlcyBleGFjdGx5OiBpZiBtaXNzaW5nbmVzcyBkZXBlbmRzIG9uIG90aGVyIHZhcmlhYmxlcywgc3VjaCBhcyBiZWluZyBpbiB0aGUgdHJpYWwgb3Igbm90LCB0aGUgaW1wdXRlZCB2YWx1ZXMgbWF5IGxlZ2l0aW1hdGVseSBkaWZmZXIuIEJ5IGRlZmF1bHQgYG1pY2VgIHVzZXMgcHJlZGljdGl2ZSBtZWFuIG1hdGNoaW5nIGZvciBjb250aW51b3VzIHZhcmlhYmxlcywgd2hpY2ggb25seSBpbXB1dGVzIHZhbHVlcyB0aGF0IG9jY3VyIGluIHRoZSBvYnNlcnZlZCBkYXRhLCBzbyBpbXBvc3NpYmxlIHZhbHVlcyAoZm9yIGV4YW1wbGUgbmVnYXRpdmUgY2hvbGVzdGVyb2wpIGNhbm5vdCBhcHBlYXIuIEJlIHN1c3BpY2lvdXMgaWYgdGhlIGltcHV0ZWQgZGlzdHJpYnV0aW9uIGlzIGNvbXBsZXRlbHkgZGlmZmVyZW50IG9yIGhhcyBhIHN0cmFuZ2Ugc2hhcGUuCgo8L2RldGFpbHM+CgoqKkV4ZXJjaXNlIDMuKiogSW5jcmVhc2UgYG1gIHRvIHRoZSBydWxlLW9mLXRodW1iIHZhbHVlIGFuZCByZS1ydW4uIEhvdyBsb25nIGRvZXMgaXQgdGFrZSwgYW5kIGRvIHRoZSBkZW5zaXR5IHBsb3RzIGNoYW5nZT8KCjxkZXRhaWxzPgo8c3VtbWFyeT5TaG93IGFuc3dlciwgY29kZSBhbmQgcmVzdWx0czwvc3VtbWFyeT4KClRoaXMgY2FuIHRha2UgYSBtaW51dGUgb3IgdHdvLiBUaGUgbmV3IGltcHV0YXRpb25zIGFyZSBzdG9yZWQgaW4gYGltcHV0ZWRfZGF0YV9iaWdgLCBzbyB0aGUgdHV0b3JpYWwga2VlcHMgdXNpbmcgYG0gPSA1YC4KCmBgYHtyIGV4LXAzLTMsIGVycm9yPVRSVUV9CiMgcnVsZSBvZiB0aHVtYjogYXQgbGVhc3QgYXMgbWFueSBpbXB1dGF0aW9ucyBhcyB0aGUgcGVyY2VudGFnZSBvZiBpbmNvbXBsZXRlIGNhc2VzCm1fcnVsZSA8LSBjZWlsaW5nKDEwMCptZWFuKCFjb21wbGV0ZS5jYXNlcyhkYXRhX21pKSkpCm1fcnVsZQoKIyBpbXB1dGUgYWdhaW4gd2l0aCBtb3JlIGltcHV0YXRpb25zIGFuZCB0aW1lIGl0CnRpbWVfYmlnIDwtIHN5c3RlbS50aW1lKAogIGltcHV0ZWRfZGF0YV9iaWcgPC0gbWljZTo6bWljZShkYXRhX21pLCBtPW1fcnVsZSwgc2VlZD0xLCBwcmludD1GQUxTRSkKKQp0aW1lX2JpZ1siZWxhcHNlZCJdCgojIGRlbnNpdHkgcGxvdCB3aXRoIG1vcmUgaW1wdXRhdGlvbnMKbWljZTo6ZGVuc2l0eXBsb3QoaW1wdXRlZF9kYXRhX2JpZywgfiBjaG9sKQpgYGAKClRoZSB0aW1lIGluY3JlYXNlcyByb3VnaGx5IGluIHByb3BvcnRpb24gdG8gdGhlIG51bWJlciBvZiBpbXB1dGF0aW9ucy4gVGhlIGRlbnNpdHkgcGxvdHMgbG9vayBzaW1pbGFyIChtb3JlIHJlZCBsaW5lcyksIGJlY2F1c2UgdGhlIGltcHV0YXRpb24gbW9kZWwgaXMgdGhlIHNhbWUuIFRoZSBiZW5lZml0IG9mIG1vcmUgaW1wdXRhdGlvbnMgaXMgbW9yZSBzdGFibGUgZXN0aW1hdGVzIGFuZCBzdGFuZGFyZCBlcnJvcnMgb2YgdGhlIGZpbmFsIG1vZGVsLCBub3QgYSBkaWZmZXJlbnQgaW1wdXRhdGlvbiBkaXN0cmlidXRpb24uCgo8L2RldGFpbHM+CgoqKkV4ZXJjaXNlIDQuKiogU3VwcG9zZSB0aGUgb3V0Y29tZSBpcyBtaXNzaW5nIGZvciBzb21lIHBhdGllbnRzIChmb3IgZXhhbXBsZSwgd2UgbG9zdCB0cmFjayBvZiB0aGVpciB2aXRhbCBzdGF0dXMpLiBTaG91bGQgeW91IGRlbGV0ZSB0aGVzZSBwYXRpZW50cyBiZWZvcmUgaW1wdXRhdGlvbj8gU2hvdyBob3cgdG8gaW1wdXRlIHdpdGggdGhlIG91dGNvbWUgaW4gdGhlIGltcHV0YXRpb24gbW9kZWwgYW5kIGRlbGV0ZSB0aGUgcGF0aWVudHMgd2l0aCBhIG1pc3Npbmcgb3V0Y29tZSBhZnRlcndhcmRzLiBUbyBtaW1pYyB0aGlzLCBzZXQgYGV2ZW50XzEwYCB0byBtaXNzaW5nIGZvciBldmVyeSAyMHRoIHBhdGllbnQuCgo8ZGV0YWlscz4KPHN1bW1hcnk+U2hvdyBhbnN3ZXIsIGNvZGUgYW5kIHJlc3VsdHM8L3N1bW1hcnk+CgpEbyBub3QgZGVsZXRlIHRoZW0gYmVmb3JlIGltcHV0YXRpb246IHRoZXNlIHBhdGllbnRzIHN0aWxsIGhhdmUgb2JzZXJ2ZWQgcHJlZGljdG9yIHZhbHVlcyB0aGF0IGhlbHAgdG8gaW1wdXRlIHRoZSBtaXNzaW5nIHByZWRpY3RvcnMgb2Ygb3RoZXIgcGF0aWVudHMuIEluc3RlYWQgKDEpIGZpcnN0IGNyZWF0ZSBhIHZhcmlhYmxlIHRoYXQgZmxhZ3Mgd2hvIGhhcyBhIG1pc3Npbmcgb3V0Y29tZSwgKDIpIGltcHV0ZSB3aXRoIHRoZSBvdXRjb21lIGluIHRoZSBpbXB1dGF0aW9uIG1vZGVsLCBhbmQgKDMpIGRlbGV0ZSB0aGUgZmxhZ2dlZCByb3dzIGZyb20gZXZlcnkgaW1wdXRlZCBkYXRhIHNldCBhZnRlcndhcmRzLiBUaGlzIGlzIGNhbGxlZCAqbXVsdGlwbGUgaW1wdXRhdGlvbiwgdGhlbiBkZWxldGlvbiogKHZvbiBIaXBwZWwsIDIwMDcpLgoKYGBge3IgZXgtcDMtNCwgZXJyb3I9VFJVRX0KIyBtaW1pYyBtaXNzaW5nIG91dGNvbWVzOiBzZXQgZXZlbnRfMTAgdG8gbWlzc2luZyBmb3IgZXZlcnkgMjB0aCBwYXRpZW50CmRhdGFfbWlfb3V0IDwtIGRhdGFfbWkKZGF0YV9taV9vdXQkZXZlbnRfMTBbc2VxKDEsIG5yb3coZGF0YV9taV9vdXQpLCBieT0yMCldIDwtIE5BCgojIDEuIGZsYWcgdGhlIHBhdGllbnRzIHdpdGggYSBtaXNzaW5nIG91dGNvbWUgQkVGT1JFIGltcHV0YXRpb24KZGF0YV9taV9vdXQkb3V0Y29tZV9taXNzaW5nIDwtIGlzLm5hKGRhdGFfbWlfb3V0JGV2ZW50XzEwKQoKIyAyLiBpbXB1dGUsIHdpdGggdGhlIG91dGNvbWUgaW4gdGhlIGltcHV0YXRpb24gbW9kZWwKIyAgICAodGhlIGZsYWcgaXRzZWxmIGlzIG5vdCB1c2VkIHRvIGltcHV0ZSBvdGhlciB2YXJpYWJsZXMpCnByZWRfbWF0cml4IDwtIG1pY2U6Om1ha2UucHJlZGljdG9yTWF0cml4KGRhdGFfbWlfb3V0KQpwcmVkX21hdHJpeFssICJvdXRjb21lX21pc3NpbmciXSA8LSAwCmltcHV0ZWRfb3V0IDwtIG1pY2U6Om1pY2UoZGF0YV9taV9vdXQsIG09NSwgcHJlZGljdG9yTWF0cml4PXByZWRfbWF0cml4LAogICAgICAgICAgICAgICAgICAgICAgICAgIHNlZWQ9MSwgcHJpbnQ9RkFMU0UpCgojIDMuIGFmdGVyIGltcHV0YXRpb246IGRlbGV0ZSB0aGUgcm93cyB3aXRoIGEgbWlzc2luZyBvdXRjb21lIGluIGVhY2ggaW1wdXRlZCBkYXRhIHNldAppbXB1dGVkX3NldHMgPC0gbWljZTo6Y29tcGxldGUoaW1wdXRlZF9vdXQsICJhbGwiKQpmb3IgKGkgaW4gc2VxX2Fsb25nKGltcHV0ZWRfc2V0cykpewogIGltcHV0ZWRfc2V0c1tbaV1dIDwtIGltcHV0ZWRfc2V0c1tbaV1dWyFpbXB1dGVkX3NldHNbW2ldXSRvdXRjb21lX21pc3NpbmcsIF0KfQpzYXBwbHkoaW1wdXRlZF9zZXRzLCBucm93KQpgYGAKCkVhY2ggaW1wdXRlZCBkYXRhIHNldCBub3cgaGFzIHRoZSBwYXRpZW50cyB3aXRoIGFuIG9ic2VydmVkIG91dGNvbWUgb25seSwgYW5kIHRoZSBwcmVkaWN0b3JzIG9mIHRoZXNlIHBhdGllbnRzIHdlcmUgaW1wdXRlZCB1c2luZyBhbGwgcGF0aWVudHMuIFRoZSBpbXB1dGVkIG91dGNvbWVzIHRoZW1zZWx2ZXMgYXJlIG5vdCBhbmFseXNlZC4gVGhpcyBhcHByb2FjaCBpcyBmb3IgYSBtaXNzaW5nIG91dGNvbWUgb25seTsgYWx3YXlzIGtlZXAgdGhlIG91dGNvbWUgaW4gdGhlIGltcHV0YXRpb24gbW9kZWwuCgo8L2RldGFpbHM+CgojIEZpdCB0aGUgbW9kZWxzCgpJbiB0aGlzIHBhcnQgeW91IGZpdCBhIENveCBwcm9wb3J0aW9uYWwgaGF6YXJkcyBtb2RlbCBvbiB0aGUgaW1wdXRlZCBkYXRhLCBjaGVjayB3aGV0aGVyIGNvbnRpbnVvdXMgcHJlZGljdG9ycyBuZWVkIGEgbm9uLWxpbmVhciBlZmZlY3QgKHVzaW5nIGByY3MoKWApLCBzZWxlY3QgcHJlZGljdG9ycyB3aXRoIGJhY2t3YXJkIHNlbGVjdGlvbiwgYW5kIHByZXNlbnQgdGhlIGZpbmFsIG1vZGVsIChlZmZlY3RzLCB2YXJpYWJsZSBpbXBvcnRhbmNlIGFuZCBhIG5vbW9ncmFtKS4gYGNwaGAgKGZyb20gYHJtc2ApIGZpdHMgYSBDb3ggbW9kZWwgd2l0aCBleHRyYSB0b29scyBmb3IgcHJlZGljdGlvbiBtb2RlbHMsIGFuZCBgZml0Lm11bHQuaW1wdXRlYCBmaXRzIHRoZSBtb2RlbCBvbiBldmVyeSBpbXB1dGVkIGRhdGEgc2V0IGFuZCBwb29scyB0aGUgcmVzdWx0cyB3aXRoIFJ1YmluJ3MgcnVsZXMuCgojIyBTZXQgZGF0YSBkaXN0cmlidXRpb25zCgpgZGF0YWRpc3RgIHN0b3JlcyB0aGUgcmFuZ2VzIG9mIHRoZSBwcmVkaWN0b3JzLCB3aGljaCBgcm1zYCBuZWVkcyBmb3Igc3VtbWFyaWVzLCBwbG90cyBhbmQgbm9tb2dyYW1zLgoKYGBge3IgcDQtZGF0YWRpc3R9CiMgc2V0IGRhdGEgZGlzdHJpYnV0aW9uczogZGF0YWRpc3QoKSBzdG9yZXMgdGhlIHJhbmdlcyBvZiBhbGwgcHJlZGljdG9ycywgYW5kCiMgb3B0aW9ucygpIHRlbGxzIHJtcyB0byB1c2UgaXQgKGl0IGxvb2tzIGZvciB0aGUgb2JqZWN0IGNhbGxlZCAiZGQiKQpkZCA8LSBybXM6OmRhdGFkaXN0KGRhdGEpCm9wdGlvbnMoZGF0YWRpc3Q9ImRkIikKYGBgCgojIyBOb24tbGluZWFyIGVmZmVjdHMgd2l0aCBgcmNzKClgCgpDb250aW51b3VzIHByZWRpY3RvcnMgc3VjaCBhcyBhZ2UsIGJpbGlydWJpbiAoYGJpbGlgKSBhbmQgYWxidW1pbiBhcmUgb2Z0ZW4gYXNzdW1lZCB0byBoYXZlIGEgKmxpbmVhciogZWZmZWN0OiBldmVyeSBleHRyYSB5ZWFyIG9yIHVuaXQgY2hhbmdlcyB0aGUgbG9nIGhhemFyZCBieSB0aGUgc2FtZSBhbW91bnQuIFRoYXQgaXMgcmFyZWx5IGV4YWN0bHkgdHJ1ZS4gQSBmbGV4aWJsZSB3YXkgdG8gbW9kZWwgYSBub24tbGluZWFyIGVmZmVjdCBpcyBhICpyZXN0cmljdGVkIGN1YmljIHNwbGluZSosIGByY3MoeCwgaylgIGZyb20gYHJtc2A6CgoqIGBrYCBpcyB0aGUgbnVtYmVyIG9mICprbm90cyosIHRoZSBwb2ludHMgd2hlcmUgdGhlIGN1cnZlIGlzIGFsbG93ZWQgdG8gYmVuZC4gV2l0aCAzIGtub3RzLCB0aGV5IGFyZSBwbGFjZWQgYXQgdGhlIDEwdGgsIDUwdGggYW5kIDkwdGggcGVyY2VudGlsZSBvZiB0aGUgdmFyaWFibGUgKHdpdGggbW9yZSBrbm90cywgYXQgb3RoZXIgcGVyY2VudGlsZXMpLiBVc2UgMyB0byA1IGtub3RzOiBtb3JlIGtub3RzIG1ha2UgdGhlIGN1cnZlIG1vcmUgZmxleGlibGUsIGJ1dCBuZWVkIG1vcmUgcGFyYW1ldGVycyBhbmQgY2FuIG92ZXJmaXQuCiogYHJjcyh4LCAzKWAgdXNlcyAyIGNvZWZmaWNpZW50czogb25lIGZvciBgeGAgaXRzZWxmICh0aGUgbGluZWFyIHBhcnQpIGFuZCBvbmUgZm9yIGB4J2AgKHRoZSBub24tbGluZWFyIHBhcnQpLiBgcmNzKHgsIDQpYCB1c2VzIDMsIGFuZCBzbyBvbi4gQSBsaW5lYXIgZWZmZWN0IHVzZXMgMS4KKiBUaGUgY3VydmUgaXMgcmVzdHJpY3RlZCB0byBiZSBsaW5lYXIgYmVmb3JlIHRoZSBmaXJzdCBhbmQgYWZ0ZXIgdGhlIGxhc3Qga25vdCwgd2hpY2gga2VlcHMgaXQgc3RhYmxlIGluIHRoZSB0YWlscy4KCldlIGZpcnN0IG1vZGVsIGFsbCBjb250aW51b3VzIHByZWRpY3RvcnMgZmxleGlibHkgd2l0aCBgcmNzKC4uLiwgMylgIGFuZCB0ZXN0IHdoZXRoZXIgdGhlIG5vbi1saW5lYXIgcGFydCBpcyBzaWduaWZpY2FudC4gRml0dGluZyBmbGV4aWJsZSBlZmZlY3RzIGZvciBhbGwgY29udGludW91cyB2YXJpYWJsZXMgaXMgYSBnb29kIHdheSB0byBzZWUgaWYgbGluZWFyaXR5IGlzIGEgcmVhc29uYWJsZSBhc3N1bXB0aW9uLgoKYGBge3IgcDQtbm9ubGluZWFyfQojIGZpdCBmdWxsIG1vZGVsIHdpdGggYWxsIGNhbmRpZGF0ZSB2YXJpYWJsZXMgb24gZWFjaCBpbXB1dGVkIGRhdGEgc2V0CiMgbW9kZWwgYWxsIGNvbnRpbnVvdXMgdmFyaWFibGVzIGZsZXhpYmxlIHRvIHNlZSBpZiB0aGF0IGlzIG5lZWRlZAojIGZpdC5tdWx0LmltcHV0ZSgpIGFyZ3VtZW50czoKIyAgIC0gdGhlIGZvcm11bGE6IG91dGNvbWUgfiBwcmVkaWN0b3IgMSArIHByZWRpY3RvciAyICsgLi4uCiMgICAtIGNwaDogdGhlIGZ1bmN0aW9uIHRoYXQgZml0cyB0aGUgQ294IG1vZGVsIG9uIGVhY2ggaW1wdXRlZCBkYXRhIHNldAojICAgLSBmaXRhcmdzOiBleHRyYSBhcmd1bWVudHMgZm9yIGNwaCAoa2VlcCB4LCB5IGFuZCB0aGUgc3Vydml2YWwgZXN0aW1hdGVzKQojICAgLSB4dHJhbnM6IHRoZSBpbXB1dGVkIGRhdGEgc2V0cwojICAgLSBkYXRhOiB0aGUgb3JpZ2luYWwgZGF0YSwgd2l0aCB0aGUgbWlzc2luZyB2YWx1ZXMKIyAgIC0gZml0LnJlcHM9VFJVRToga2VlcCB0aGUgbW9kZWwgb2YgZXZlcnkgaW1wdXRlZCBkYXRhIHNldApmdWxsX25vbl9saW5lYXJfbW9kZWwgPC0gSG1pc2M6OmZpdC5tdWx0LmltcHV0ZShTXzEwIH4gcmNzKGFnZSwgMykgKwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJjcyhiaWxpLCAzKSArCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcmNzKGFsYnVtaW4sIDMpICsgc2V4ICsKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhc2NpdGVzICsgaGVwYXRvICsgc3RhZ2UsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNwaCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZml0YXJncz1saXN0KHg9VFJVRSwgeT1UUlVFLCBzdXJ2PVRSVUUpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4dHJhbnM9aW1wdXRlZF9kYXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkYXRhPWRhdGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZpdC5yZXBzPVRSVUUpCnN1bW1hcnkoZnVsbF9ub25fbGluZWFyX21vZGVsKQpgYGAKClRoZSBgYW5vdmFgIHRhYmxlIGRpc3BsYXlzIGNoaS1zcXVhcmUgdGVzdHMgZm9yIG5vbi1saW5lYXJpdHkuIExvb2sgYXQgdGhlIHJvdyAiTm9ubGluZWFyIiB1bmRlciBlYWNoIGNvbnRpbnVvdXMgcHJlZGljdG9yOiBhIHNtYWxsIHAtdmFsdWUgbWVhbnMgdGhhdCB0aGUgZWZmZWN0IGlzIG5vdCBhIHN0cmFpZ2h0IGxpbmUuCgpgYGB7ciBwNC1ub25saW5lYXItYW5vdmF9CiMgZGlzcGxheSBjaGktc3F1YXJlIHRlc3RzIGZvciBub24tbGluZWFyaXRpZXMKYW5vdmEoZnVsbF9ub25fbGluZWFyX21vZGVsKQpgYGAKCkhlcmUgYmlsaXJ1YmluIGhhcyBhIGNsZWFybHkgbm9uLWxpbmVhciBlZmZlY3QsIGFnZSBpcyBib3JkZXJsaW5lLCBhbmQgYWxidW1pbiBpcyBjbG9zZSB0byBsaW5lYXIuIFBsb3QgdGhlIGVzdGltYXRlZCBlZmZlY3RzIHRvIHNlZSB3aGF0IHRoaXMgbG9va3MgbGlrZToKCmBgYHtyIHA0LW5vbmxpbmVhci1wbG90fQojIGRpc3BsYXkgcHJlZGljdG9yIGVmZmVjdHMgKGxvZyBoYXphcmQpIGZvciB0aGUgZmxleGlibGUgbW9kZWwKcGxvdChQcmVkaWN0KGZ1bGxfbm9uX2xpbmVhcl9tb2RlbCkpCmBgYAoKVGhlIGN1cnZlIGZvciBiaWxpcnViaW4gcmlzZXMgc3RlZXBseSBhdCBsb3cgdmFsdWVzIGFuZCBmbGF0dGVucyBvdXQgYXQgaGlnaCB2YWx1ZXMsIHdoaWNoIGEgc3RyYWlnaHQgbGluZSB3b3VsZCBtaXNzLiBCYXNlZCBvbiB0aGVzZSByZXN1bHRzIHdlIGtlZXAgYHJjcygpYCBmb3IgYWdlIGFuZCBiaWxpcnViaW4gYW5kIGEgbGluZWFyIGVmZmVjdCBmb3IgYWxidW1pbiBpbiB0aGUgZnVsbCBtb2RlbC4KCiMjIEZpdCB0aGUgZnVsbCBtb2RlbAoKTm93IHdlIGZpdCB0aGUgZnVsbCBtb2RlbCB3aXRoIGByY3MoKWAgZm9yIGFnZSBhbmQgYmlsaXJ1YmluLCBhIGxpbmVhciBlZmZlY3QgZm9yIGFsYnVtaW4sIGFuZCB0aGUgb3RoZXIgY2FuZGlkYXRlIHByZWRpY3RvcnMuIFRoaXMgbW9kZWwgaXMgdGhlIHN0YXJ0aW5nIHBvaW50IGZvciB2YXJpYWJsZSBzZWxlY3Rpb24uCgpgYGB7ciBwNC1mdWxsfQojIGZpdCBmdWxsIG1vZGVsIHdpdGggYWxsIGNhbmRpZGF0ZSB2YXJpYWJsZXMgb24gZWFjaCBpbXB1dGVkIGRhdGEgc2V0CiMgd2Ugc3RvcmUgdGhlIGZvcm11bGEgaW4gYW4gb2JqZWN0LCBzbyB0aGF0IHdlIGNhbiByZXVzZSBpdCBsYXRlciAoVHV0b3JpYWwgMikKIyByY3MoYWdlLCAzKTogc3BsaW5lIGZvciBhZ2U7IGFsYnVtaW4gd2l0aG91dCByY3MoKTogYSBzdHJhaWdodCBsaW5lCmZvcm1fZnVsbCA8LSBTXzEwIH4gcmNzKGFnZSwgMykgKyByY3MoYmlsaSwgMykgKyBhbGJ1bWluICsgc2V4ICsKICBhc2NpdGVzICsgaGVwYXRvICsgc3RhZ2UKZnVsbF9tb2RlbCA8LSBIbWlzYzo6Zml0Lm11bHQuaW1wdXRlKGZvcm1fZnVsbCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY3BoLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmaXRhcmdzPWxpc3QoeD1UUlVFLCB5PVRSVUUsIHN1cnY9VFJVRSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHh0cmFucz1pbXB1dGVkX2RhdGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGE9ZGF0YSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZml0LnJlcHM9VFJVRSkKIyBkaXNwbGF5IHByZWRpY3RvciBlZmZlY3RzIChoYXphcmQgcmF0aW9zLCBjb21wYXJpbmcgdGhlIDI1dGggYW5kIDc1dGggcGVyY2VudGlsZSkKc3VtbWFyeShmdWxsX21vZGVsKQpgYGAKCmBgYHtyIHA0LWZ1bGwtYW5vdmF9CiMgZGlzcGxheSBjaGktc3F1YXJlIHRlc3RzIGZvciB2YXJpYWJsZSBpbXBvcnRhbmNlOiBhIGxhcmdlciBjaGktc3F1YXJlCiMgKHJlbGF0aXZlIHRvIGl0cyBkZWdyZWVzIG9mIGZyZWVkb20pIG1lYW5zIGEgbW9yZSBpbXBvcnRhbnQgcHJlZGljdG9yCmFub3ZhKGZ1bGxfbW9kZWwpCmBgYAoKV2UgY2hlY2sgdGhlIHByb3BvcnRpb25hbCBoYXphcmRzIGFzc3VtcHRpb24sIHdoaWNoIHNheXMgdGhhdCB0aGUgZWZmZWN0IG9mIGEgcHJlZGljdG9yICh0aGUgaGF6YXJkIHJhdGlvKSBpcyBjb25zdGFudCBvdmVyIHRpbWUuIEEgc21hbGwgcC12YWx1ZSBmb3IgYSBwcmVkaWN0b3Igc3VnZ2VzdHMgYSB2aW9sYXRpb24uCgpgYGB7ciBwNC16cGh9CiMgdGVzdCBwcm9wb3J0aW9uYWwgaGF6YXJkcyBhc3N1bXB0aW9uOiBhIHNtYWxsIHAtdmFsdWUgc3VnZ2VzdHMgdGhhdCB0aGUKIyBlZmZlY3Qgb2YgdGhhdCBwcmVkaWN0b3IgY2hhbmdlcyBvdmVyIHRpbWUKc3Vydml2YWw6OmNveC56cGgoZnVsbF9tb2RlbCkKYGBgCgpgYGB7ciBwNC1mdWxsLXBsb3R9CiMgZGlzcGxheSBwcmVkaWN0b3IgZWZmZWN0czogUHJlZGljdCgpIGNhbGN1bGF0ZXMgdGhlIGVmZmVjdCBvZiBlYWNoIHByZWRpY3RvcgojIG9uIHRoZSBsb2cgaGF6YXJkIChvdGhlcnMgaGVsZCBhdCB0aGVpciBtZWRpYW4pLCBwbG90KCkgZHJhd3MgaXQKcGxvdChQcmVkaWN0KGZ1bGxfbW9kZWwpKQpgYGAKCiMjIEJhY2t3YXJkIHZhcmlhYmxlIHNlbGVjdGlvbgoKV2UgdXNlIGJhY2t3YXJkIHNlbGVjdGlvbiB3aXRoIGBmYXN0YncoKWA6IHN0YXJ0aW5nIGZyb20gdGhlIGZ1bGwgbW9kZWwsIHZhcmlhYmxlcyB3aXRoIGEgcC12YWx1ZSBhYm92ZSAwLjA1IGFyZSByZW1vdmVkIG9uZSBieSBvbmUuIGB0eXBlPSJpbmRpdmlkdWFsImAgdGVzdHMgZWFjaCB2YXJpYWJsZSBzZXBhcmF0ZWx5LiBOb3RlIHRoYXQgYHJjcyhhZ2UsIDMpYCBpcyBvbmUgdmFyaWFibGUgd2l0aCB0d28gdGVybXMgKGBhZ2VgIGFuZCBgYWdlJ2ApLCBzbyBpdCBpcyBsaXN0ZWQgb25jZSwgYXMgYGFnZWAuCgpGb3IgdGhpcyBpbGx1c3RyYXRpdmUgZXhlcmNpc2UgYGZhc3RidygpYCBpcyBmaW5lLiBUaGVyZSBhcmUgYmV0dGVyIHdheXMgdG8gc2VsZWN0IHZhcmlhYmxlcywgZm9yIGV4YW1wbGUgcGVuYWxpemVkIHJlZ3Jlc3Npb24gKExBU1NPIG9yIGVsYXN0aWMgbmV0KSBvciByZWx5aW5nIG9uIGNsaW5pY2FsIGtub3dsZWRnZSBpbnN0ZWFkIG9mIHAtdmFsdWVzLCBhcyBiYWNrd2FyZCBzZWxlY3Rpb24gdGVuZHMgdG8gb3ZlcmZpdC4gVGhpcyBpcyBhbHNvIHdoeSB3ZSBlc3RpbWF0ZSBhIHNocmlua2FnZSBmYWN0b3IgaW4gVHV0b3JpYWwgMi4KCmBgYHtyIHA0LWJ3fQojIHBlcmZvcm0gYmFja3dhcmQgc2VsZWN0aW9uIHdpdGggZmFzdGJ3KCk6CiMgICB0eXBlPSJpbmRpdmlkdWFsIjogdGVzdCBlYWNoIHZhcmlhYmxlIHNlcGFyYXRlbHkKIyAgIHJ1bGU9InAiOiByZW1vdmUgdmFyaWFibGVzIGJhc2VkIG9uIHRoZWlyIHAtdmFsdWUKIyAgIHNscz0wLjA1OiByZW1vdmUgYSB2YXJpYWJsZSBpZiBpdHMgcC12YWx1ZSBpcyBhYm92ZSAwLjA1CmZhc3RidyhmdWxsX21vZGVsLCB0eXBlPSJpbmRpdmlkdWFsIiwgcnVsZT0icCIsIHNscz0wLjA1KQojIHRoaXMgZ2l2ZXMgZmluYWwgbW9kZWw6IGFnZSwgYmlsaSwgYWxidW1pbiwgc3RhZ2UKYGBgCgpXZSB0aGVuIGZpdCB0aGUgYmFja3dhcmQgc2VsZWN0ZWQgbW9kZWwuCgpgYGB7ciBwNC1idy1maXR9CiMgZml0IGJhY2t3YXJkIHNlbGVjdGVkIG1vZGVsOiB0aGUgc2FtZSBhcyBiZWZvcmUsIHdpdGggb25seSB0aGUgc2VsZWN0ZWQgdmFyaWFibGVzCmZvcm1fYncgPC0gU18xMCB+IHJjcyhhZ2UsIDMpICsgcmNzKGJpbGksIDMpICsgYWxidW1pbiArIHN0YWdlCmJ3X21vZGVsIDwtIEhtaXNjOjpmaXQubXVsdC5pbXB1dGUoZm9ybV9idywKICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNwaCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgIGZpdGFyZ3M9bGlzdCh4PVRSVUUsIHk9VFJVRSwgc3Vydj1UUlVFKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHh0cmFucz1pbXB1dGVkX2RhdGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBkYXRhPWRhdGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBmaXQucmVwcz1UUlVFKQojIGRpc3BsYXkgcHJlZGljdG9yIGVmZmVjdHMKc3VtbWFyeShid19tb2RlbCkKYGBgCgojIyBQcmVzZW50IHRoZSBmaW5hbCBtb2RlbAoKKipUaGUgZWZmZWN0IG9mIGFnZSBwZXIgZGVjYWRlLioqIEJlY2F1c2UgYWdlIGhhcyBhIG5vbi1saW5lYXIgZWZmZWN0IChhIHNwbGluZSksIHRoZSBoYXphcmQgcmF0aW8gZm9yIDEwIHllYXJzIG9mIGFnZSBkZXBlbmRzIG9uIHdoZXJlIHlvdSBzdGFydC4gV2UgdGhlcmVmb3JlIGNvbXBhcmUgYWdlcyB3aXRoaW4gdGhlIG9ic2VydmVkIHJhbmdlLCBoZXJlIDUwIHZlcnN1cyA2MCB5ZWFycy4gKEEgY29tcGFyaXNvbiBvZiBhZ2UgMCB3aXRoIGFnZSAxMCB3b3VsZCBiZSBvdXRzaWRlIHRoZSByYW5nZSBvZiB0aGUgZGF0YSBhbmQgbWVhbmluZ2xlc3MuKQoKYGBge3IgcDQtYWdlLWRlY2FkZX0KIyBwcmVzZW50IGVmZmVjdCBmb3IgYWdlIHBlciBkZWNhZGUgKGZyb20gNTAgdG8gNjAgeWVhcnMpCiMgYWdlPWMoNTAsIDYwKSBzZXRzIHRoZSB0d28gdmFsdWVzIHRoYXQgYXJlIGNvbXBhcmVkCnN1bW1hcnkoYndfbW9kZWwsIGFnZT1jKDUwLCA2MCkpCmBgYAoKVmFyaWFibGUgaW1wb3J0YW5jZSB3aXRoIHRoZSBjaGktc3F1YXJlIHRlc3Qgc3RhdGlzdGljIChtaW51cyBpdHMgZGVncmVlcyBvZiBmcmVlZG9tIGlzIGEgZmFpciB3YXkgdG8gY29tcGFyZSk6CgpgYGB7ciBwNC1idy1hbm92YX0KIyBkaXNwbGF5IHZhcmlhYmxlIGltcG9ydGFuY2UgdXNpbmcgY2hpLXNxdWFyZSB0ZXN0IHN0YXRpc3RpYwphbm92YShid19tb2RlbCkKYGBgCgpgYGB7ciBwNC1idy1wbG90fQojIHZpc3VhbGl6ZSBwcmVkaWN0b3IgZWZmZWN0cwpwbG90KFByZWRpY3QoYndfbW9kZWwpKQpgYGAKCkEgbm9tb2dyYW0gbGV0cyB5b3UgY2FsY3VsYXRlIHRoZSByaXNrIG9mIGFuIGluZGl2aWR1YWwgcGF0aWVudCBieSBoYW5kLgoKYGBge3IgcDQtbm9tb2dyYW0sIGZpZy5oZWlnaHQ9NywgZmlnLndpZHRoPTl9CiMgbm9tb2dyYW06IG5vbW9ncmFtKCkgY2FsY3VsYXRlcyB0aGUgcG9pbnRzIG9mIGV2ZXJ5IHByZWRpY3RvciB2YWx1ZSwKIyBwbG90KCkgZHJhd3MgaXQKcGxvdChybXM6Om5vbW9ncmFtKGJ3X21vZGVsKSkKYGBgCgojIyBFeGVyY2lzZXM6IFBhcnQgNAoKVHJ5IGVhY2ggZXhlcmNpc2UgeW91cnNlbGYgZmlyc3QuIFRoZW4gb3BlbiB0aGUgZmllbGQgYmVsb3cgaXQgdG8gc2VlIHRoZSBhbnN3ZXIsIHRoZSBjb2RlIGFuZCBpdHMgcmVzdWx0cy4gVGhlIGFuc3dlcnMgdXNlIG5ldyBvYmplY3QgbmFtZXMsIHNvIHRoZSBzYXZlZCBtb2RlbHMgZG8gbm90IGNoYW5nZS4KCioqRXhlcmNpc2UgMS4qKiBJbiB0aGUgbm9tb2dyYW0sIHdoYXQgaXMgdGhlIHRvdGFsIG51bWJlciBvZiBwb2ludHMgZm9yIGEgNzUteWVhci1vbGQgcGF0aWVudCB3aXRoIGEgYmlsaXJ1YmluIG9mIDIsIGFuIGFsYnVtaW4gb2YgMy42IGFuZCBzdGFnZSA0PyBDYW4geW91IHRlbGwgd2hpY2ggdmFyaWFibGUgY29udHJpYnV0ZXMgdGhlIG1vc3Q/Cgo8ZGV0YWlscz4KPHN1bW1hcnk+U2hvdyBhbnN3ZXIsIGNvZGUgYW5kIHJlc3VsdHM8L3N1bW1hcnk+CgpQcmludGluZyB0aGUgbm9tb2dyYW0gZ2l2ZXMgdGhlIHBvaW50cyBmb3IgZXZlcnkgdmFsdWUgb2YgZXZlcnkgcHJlZGljdG9yIChmb3IgYWdlLCBpbiBzdGVwcyBvZiA1IHllYXJzKToKCmBgYHtyIGV4LXA0LTEsIGVycm9yPVRSVUV9Cm5vbSA8LSBybXM6Om5vbW9ncmFtKGJ3X21vZGVsKQpwcmludChub20pCmBgYAoKTG9vayB1cCB0aGUgcG9pbnRzIGZvciBhZ2UgNzUsIGJpbGlydWJpbiAyLCBhbGJ1bWluIDMuNiBhbmQgc3RhZ2UgNCwgYW5kIGFkZCB0aGVtIHVwLiBUaGUgdG90YWwgaXMgcmVhZCBvbiB0aGUgIlRvdGFsIFBvaW50cyIgYXhpcywgd2hpY2ggY29udmVydHMgdGhlIHBvaW50cyB0byBhIHJpc2suIFRoZSB2YXJpYWJsZSB3aXRoIHRoZSB3aWRlc3QgcmFuZ2Ugb2YgcG9pbnRzICh0aGUgbG9uZ2VzdCBheGlzIGluIHRoZSBub21vZ3JhbSkgY29udHJpYnV0ZXMgdGhlIG1vc3QgdG8gdGhlIHByZWRpY3Rpb24uCgo8L2RldGFpbHM+CgoqKkV4ZXJjaXNlIDIuKiogQ2hhbmdlIHRoZSBhZ2UgY29tcGFyaXNvbiB0byBgYyg0MCwgNTApYCBhbmQgYGMoNjAsIDcwKWAuIFdoeSBkbyB0aGUgaGF6YXJkIHJhdGlvcyBkaWZmZXI/Cgo8ZGV0YWlscz4KPHN1bW1hcnk+U2hvdyBhbnN3ZXIsIGNvZGUgYW5kIHJlc3VsdHM8L3N1bW1hcnk+CgpgYGB7ciBleC1wNC0yLCBlcnJvcj1UUlVFfQpzdW1tYXJ5KGJ3X21vZGVsLCBhZ2U9Yyg0MCwgNTApKQpzdW1tYXJ5KGJ3X21vZGVsLCBhZ2U9Yyg2MCwgNzApKQoKIyBwbG90IHRoZSBlZmZlY3Qgb2YgYWdlIG9uIHRoZSBsb2cgaGF6YXJkCnBsb3QoUHJlZGljdChid19tb2RlbCwgYWdlKSkKYGBgCgpUaGUgcGxvdCBzaG93cyB3aHk6IGFnZSBoYXMgYSBub24tbGluZWFyIGVmZmVjdCAobW9kZWxsZWQgd2l0aCBgcmNzKClgKSwgc28gdGhlIGVmZmVjdCBvZiBhZ2UgaXMgbm90IGEgc3RyYWlnaHQgbGluZS4gVGhlIGNoYW5nZSBpbiBsb2cgaGF6YXJkIGZvciBhIDEwLXllYXIgZGlmZmVyZW5jZSB0aGVyZWZvcmUgZGVwZW5kcyBvbiB3aGVyZSB5b3Ugc3RhcnQuIFdpdGggYSBsaW5lYXIgZWZmZWN0LCB0aGUgaGF6YXJkIHJhdGlvIHBlciBkZWNhZGUgd291bGQgYmUgdGhlIHNhbWUgZXZlcnl3aGVyZS4KCjwvZGV0YWlscz4KCioqRXhlcmNpc2UgMy4qKiBDaGFuZ2UgYHNscz0wLjA1YCB0byBgc2xzPTAuMTU3YCAoc2ltaWxhciB0byBBSUMpLiBEb2VzIHRoZSBzZWxlY3RlZCBtb2RlbCBjaGFuZ2U/Cgo8ZGV0YWlscz4KPHN1bW1hcnk+U2hvdyBhbnN3ZXIsIGNvZGUgYW5kIHJlc3VsdHM8L3N1bW1hcnk+CgpgYGB7ciBleC1wNC0zLCBlcnJvcj1UUlVFfQpmYXN0YncoZnVsbF9tb2RlbCwgdHlwZT0iaW5kaXZpZHVhbCIsIHJ1bGU9InAiLCBzbHM9MC4xNTcpCmBgYAoKQ29tcGFyZSB0aGUgdmFyaWFibGVzIHRoYXQgYXJlIGtlcHQgd2l0aCB0aGUgcmVzdWx0IGZvciBgc2xzPTAuMDVgLiBXaXRoIGBzbHM9MC4xNTdgIHRoZSBtb2RlbCB1c3VhbGx5IGtlZXBzIGEgdmFyaWFibGUgb3IgdHdvIHRoYXQgd2VyZSByZW1vdmVkIHdpdGggYHNscz0wLjA1YCAoaGVyZSBgYXNjaXRlc2AsIHdoaWNoIGhhcyBhIHAtdmFsdWUgYmV0d2VlbiAwLjA1IGFuZCAwLjE1NykuCgoqKldoeSAwLjE1Nz8qKiBJbiBiYWNrd2FyZCBzZWxlY3Rpb24gYSB2YXJpYWJsZSBpcyByZW1vdmVkIHdoZW4gaXRzIHAtdmFsdWUgaXMgbGFyZ2VyIHRoYW4gYHNsc2AuIFRoZSB0aHJlc2hvbGQgZGV0ZXJtaW5lcyBob3cgc3RyaWN0IHRoZSBzZWxlY3Rpb24gaXM6CgoqIFRoZSAqQUlDKiAoQWthaWtlIGluZm9ybWF0aW9uIGNyaXRlcmlvbikgaXMgYC0yIHggbG9nLWxpa2VsaWhvb2QgKyAyIHggbnVtYmVyIG9mIHBhcmFtZXRlcnNgLiBSZW1vdmluZyBhIHZhcmlhYmxlIHdpdGggMSBkZWdyZWUgb2YgZnJlZWRvbSBsb3dlcnMgdGhlIEFJQyB3aGVuIHRoZSBjaGktc3F1YXJlIG9mIHRoYXQgdmFyaWFibGUgaXMgc21hbGxlciB0aGFuIDIuIEEgY2hpLXNxdWFyZSBvZiAyIHdpdGggMSBkZWdyZWUgb2YgZnJlZWRvbSBjb3JyZXNwb25kcyB0byBhIHAtdmFsdWUgb2YgMC4xNTcuIFNvIHNlbGVjdGluZyB3aXRoIGBzbHM9MC4xNTdgIGlzIChmb3IgdmFyaWFibGVzIHdpdGggMSBkZWdyZWUgb2YgZnJlZWRvbSkgdGhlIHNhbWUgYXMgcmVtb3ZpbmcgdmFyaWFibGVzIGFzIGxvbmcgYXMgdGhhdCBpbXByb3ZlcyB0aGUgQUlDLiBGb3IgdmFyaWFibGVzIHdpdGggbW9yZSBkZWdyZWVzIG9mIGZyZWVkb20gdGhlIG1hdGNoIGlzIG9ubHkgYXBwcm94aW1hdGUuCiogV2l0aCBgc2xzPTAuMDVgIGEgdmFyaWFibGUgaXMgb25seSBrZXB0IHdoZW4gaXRzIGNoaS1zcXVhcmUgaXMgbGFyZ2VyIHRoYW4gMy44NCAoZm9yIDEgZGVncmVlIG9mIGZyZWVkb20pLiBUaGF0IGlzIGEgbXVjaCBoaWdoZXIgYmFyIHRoYW4gMiwgc28gZmV3ZXIgdmFyaWFibGVzIGFyZSBrZXB0IGFuZCB0aGUgbW9kZWwgaXMgc21hbGxlci4KCkEgbGVzcyBzdHJpY3QgdGhyZXNob2xkICgwLjE1Nykga2VlcHMgbW9yZSB2YXJpYWJsZXM6IHlvdSBhcmUgbGVzcyBsaWtlbHkgdG8gbWlzcyBhIHZhcmlhYmxlIHRoYXQgbWF0dGVycywgYnV0IHRoZSBtb2RlbCBoYXMgbW9yZSBwYXJhbWV0ZXJzIGFuZCBhIGxhcmdlciByaXNrIG9mIG92ZXJmaXR0aW5nIChtb3JlIG9wdGltaXNtIGFuZCBtb3JlIHNocmlua2FnZSBuZWVkZWQsIHNlZSBUdXRvcmlhbCAyKS4gQSBzdHJpY3RlciB0aHJlc2hvbGQgKDAuMDUpIGdpdmVzIGEgc21hbGxlciwgc2ltcGxlciBtb2RlbCwgYnV0IHlvdSBtaWdodCByZW1vdmUgdmFyaWFibGVzIHRoYXQgY2FycnkgYSBiaXQgb2YgaW5mb3JtYXRpb24uIFRoZXJlIGlzIG5vIHNpbmdsZSBjb3JyZWN0IGNob2ljZTogcmVwb3J0IHdoaWNoIHRocmVzaG9sZCB5b3UgdXNlZCwgYW5kIHZhbGlkYXRlIHRoZSB3aG9sZSBwcm9jZWR1cmUsIGluY2x1ZGluZyB0aGUgc2VsZWN0aW9uLCBhcyB3ZSBkbyBpbiBUdXRvcmlhbCAyLiBBbHNvIHJlbWVtYmVyIHRoYXQgc2VsZWN0aW5nIHZhcmlhYmxlcyBvbiBwLXZhbHVlcyBhdCBhbGwgaXMgYSBkZWJhdGFibGUgc3RyYXRlZ3kgKHNlZSB0aGUgbm90ZSBhYm92ZSBvbiBwZW5hbGl6ZWQgcmVncmVzc2lvbikuCgo8L2RldGFpbHM+CgoqKkV4ZXJjaXNlIDQuKiogV2hhdCBkb2VzIHRoZSBgY294LnpwaGAgb3V0cHV0IHRlbGwgeW91PyBXaGljaCBwcmVkaWN0b3IgKGlmIGFueSkgbWlnaHQgdmlvbGF0ZSBwcm9wb3J0aW9uYWwgaGF6YXJkcz8KCjxkZXRhaWxzPgo8c3VtbWFyeT5TaG93IGFuc3dlciwgY29kZSBhbmQgcmVzdWx0czwvc3VtbWFyeT4KCmBgYHtyIGV4LXA0LTQsIGVycm9yPVRSVUV9CnN1cnZpdmFsOjpjb3guenBoKGZ1bGxfbW9kZWwpCmBgYAoKRWFjaCByb3cgdGVzdHMgd2hldGhlciB0aGUgZWZmZWN0IG9mIHRoYXQgcHJlZGljdG9yIGNoYW5nZXMgb3ZlciB0aW1lLiBBIHNtYWxsIHAtdmFsdWUgKGZvciBleGFtcGxlIGJlbG93IDAuMDUpIHN1Z2dlc3RzIHRoYXQgdGhlIHByb3BvcnRpb25hbCBoYXphcmRzIGFzc3VtcHRpb24gaXMgdmlvbGF0ZWQgZm9yIHRoYXQgcHJlZGljdG9yLCBhbmQgdGhlIGBHTE9CQUxgIHJvdyB0ZXN0cyBhbGwgcHJlZGljdG9ycyB0b2dldGhlci4gSWYgYSBwcmVkaWN0b3IgdmlvbGF0ZXMgdGhlIGFzc3VtcHRpb24sIHlvdSBjb3VsZCBhZGQgYW4gaW50ZXJhY3Rpb24gd2l0aCB0aW1lLCBzdHJhdGlmeSBvbiBpdCwgb3IgdXNlIGEgZGlmZmVyZW50IG1vZGVsLiBXaXRoIGZldyBldmVudHMgdGhlc2UgdGVzdHMgaGF2ZSBsaXR0bGUgcG93ZXIsIHNvIGFsc28gbG9vayBhdCB0aGUgcGxvdHMuCgo8L2RldGFpbHM+CgoqKkV4ZXJjaXNlIDUuKiogUmVmaXQgdGhlIGZ1bGwgbW9kZWwgd2l0aCA0IGtub3RzIGZvciBiaWxpcnViaW4sIGByY3MoYmlsaSwgNClgLCBpbnN0ZWFkIG9mIDMuIERvZXMgdGhlIGVzdGltYXRlZCBlZmZlY3Qgb2YgYmlsaXJ1YmluIGNoYW5nZT8gSG93IG1hbnkgcGFyYW1ldGVycyBkb2VzIGl0IGFkZD8KCjxkZXRhaWxzPgo8c3VtbWFyeT5TaG93IGFuc3dlciwgY29kZSBhbmQgcmVzdWx0czwvc3VtbWFyeT4KCldlIGZpdCB0aGUgbW9kZWwgaW4gYSBuZXcgb2JqZWN0LCBgZnVsbF9tb2RlbF80a25vdHNgLCBzbyBgZnVsbF9tb2RlbGAgc3RheXMgYXMgaXQgaXMuCgpgYGB7ciBleC1wNC01LCBlcnJvcj1UUlVFfQpmb3JtXzRrbm90cyA8LSBTXzEwIH4gcmNzKGFnZSwgMykgKyByY3MoYmlsaSwgNCkgKyBhbGJ1bWluICsgc2V4ICsKICBhc2NpdGVzICsgaGVwYXRvICsgc3RhZ2UKZnVsbF9tb2RlbF80a25vdHMgPC0gSG1pc2M6OmZpdC5tdWx0LmltcHV0ZShmb3JtXzRrbm90cywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY3BoLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmaXRhcmdzPWxpc3QoeD1UUlVFLCB5PVRSVUUsIHN1cnY9VFJVRSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHh0cmFucz1pbXB1dGVkX2RhdGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGE9ZGF0YSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZml0LnJlcHM9VFJVRSkKCiMgZWZmZWN0IG9mIGJpbGlydWJpbiB3aXRoIDMgYW5kIDQga25vdHMKcGxvdChQcmVkaWN0KGZ1bGxfbW9kZWwsIGJpbGkpKQpwbG90KFByZWRpY3QoZnVsbF9tb2RlbF80a25vdHMsIGJpbGkpKQoKIyBudW1iZXIgb2YgY29lZmZpY2llbnRzCmxlbmd0aChjb2VmKGZ1bGxfbW9kZWwpKQpsZW5ndGgoY29lZihmdWxsX21vZGVsXzRrbm90cykpCmBgYAoKV2l0aCA0IGtub3RzIHRoZSBjdXJ2ZSBjYW4gYmVuZCBpbiBtb3JlIHBsYWNlcywgYnV0IHdpdGggdGhpcyBudW1iZXIgb2YgZXZlbnRzIHRoZSBzaGFwZSBpcyB2ZXJ5IHNpbWlsYXIgdG8gdGhlIGN1cnZlIHdpdGggMyBrbm90cywgd2hpbGUgdGhlIG1vZGVsIG5lZWRzIG9uZSBtb3JlIHBhcmFtZXRlci4gTW9yZSBrbm90cyBnaXZlIG1vcmUgZmxleGliaWxpdHksIGJ1dCBhbHNvIGEgaGlnaGVyIHJpc2sgb2Ygb3ZlcmZpdHRpbmcsIHNvIDMga25vdHMgaXMgYSBzZW5zaWJsZSBkZWZhdWx0IGluIGEgZGF0YSBzZXQgb2YgdGhpcyBzaXplLgoKPC9kZXRhaWxzPgoKKipFeGVyY2lzZSA2LioqIEFkZCBgZWRlbWFgIGFuZCBgY29wcGVyYCB0byBgZm9ybV9mdWxsYC4gSG93IGRvZXMgdmFyaWFibGUgaW1wb3J0YW5jZSBjaGFuZ2U/Cgo8ZGV0YWlscz4KPHN1bW1hcnk+U2hvdyBhbnN3ZXIsIGNvZGUgYW5kIHJlc3VsdHM8L3N1bW1hcnk+CgpXZSBmaXQgdGhlIGxhcmdlciBtb2RlbCBpbiBhIG5ldyBvYmplY3QsIGBmdWxsX21vZGVsX2V4dHJhYCwgc28gYGZ1bGxfbW9kZWxgIHN0YXlzIGFzIGl0IGlzLgoKYGBge3IgZXgtcDQtNiwgZXJyb3I9VFJVRX0KZm9ybV9leHRyYSA8LSBTXzEwIH4gcmNzKGFnZSwgMykgKyByY3MoYmlsaSwgMykgKyBhbGJ1bWluICsgc2V4ICsKICBhc2NpdGVzICsgaGVwYXRvICsgc3RhZ2UgKyBlZGVtYSArIGNvcHBlcgpmdWxsX21vZGVsX2V4dHJhIDwtIEhtaXNjOjpmaXQubXVsdC5pbXB1dGUoZm9ybV9leHRyYSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY3BoLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmaXRhcmdzPWxpc3QoeD1UUlVFLCB5PVRSVUUsIHN1cnY9VFJVRSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHh0cmFucz1pbXB1dGVkX2RhdGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGE9ZGF0YSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZml0LnJlcHM9VFJVRSkKYW5vdmEoZnVsbF9tb2RlbF9leHRyYSkKYGBgCgpDb21wYXJlIHRoZSBjaGktc3F1YXJlIHZhbHVlcyAobWludXMgdGhlaXIgZGVncmVlcyBvZiBmcmVlZG9tKSB3aXRoIHRoZSB0YWJsZSBmb3IgYGZ1bGxfbW9kZWxgLiBDb3BwZXIgaXMgYSBrbm93biBwcmVkaWN0b3IgaW4gcHJpbWFyeSBiaWxpYXJ5IGNpcnJob3NpcywgYW5kIGl0IGlzIG1pc3NpbmcgZm9yIGFib3V0IGEgcXVhcnRlciBvZiB0aGUgcGF0aWVudHMsIHNvIGl0cyBlZmZlY3QgaXMgZXN0aW1hdGVkIG9uIGltcHV0ZWQgdmFsdWVzLiBOb3RlIHRoYXQgYGNvcHBlcmAgaXMgcmlnaHQtc2tld2VkLCBzbyBhIHRyYW5zZm9ybWF0aW9uIChmb3IgZXhhbXBsZSBhIGxvZyB0cmFuc2Zvcm1hdGlvbikgY291bGQgYmUgY29uc2lkZXJlZC4KCjwvZGV0YWlscz4K