Tidy Week 2: Data Cleansing
Sofia Animato, Nel Mazur
Introduction
Analysis of data is a process of inspecting, cleaning, transforming, and modeling data with the goal of highlighting useful information, suggesting conclusions and supporting decision making.
Many times in the beginning we spend hours on handling problems with missing values, logical inconsistencies or outliers in our datasets. In this tutorial we will go through the most popular techniques in data cleansing.
We will be working with the messy dataset iris.
Originally published at UCI Machine Learning Repository: Iris Data Set,
this small dataset from 1936 is often used for testing out machine
learning algorithms and visualizations. Each row of the table represents
an iris flower, including its species and dimensions of its botanical
parts, sepal and petal, in centimeters.
Take a look at this dataset here:
mydata## Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1 6.4 3.2 4.500 1.5 versicolor
## 2 6.3 3.3 6.000 2.5 virginica
## 3 6.2 NA 5.400 2.3 virginica
## 4 5.0 3.4 1.600 0.4 setosa
## 5 5.7 2.6 3.500 1.0 versicolor
## 6 5.3 NA NA 0.2 setosa
## 7 6.4 2.7 5.300 NA virginica
## 8 5.9 3.0 5.100 1.8 virginica
## 9 5.8 2.7 4.100 1.0 versicolor
## 10 4.8 3.1 1.600 0.2 setosa
## 11 5.0 3.5 1.600 0.6 setosa
## 12 6.0 2.7 5.100 1.6 versicolor
## 13 6.0 3.0 4.800 NA virginica
## 14 6.8 2.8 4.800 1.4 versicolor
## 15 NA 3.9 1.700 0.4 setosa
## 16 5.0 -3.0 3.500 1.0 versicolor
## 17 5.5 NA 4.000 1.3 versicolor
## 18 4.7 3.2 1.300 0.2 setosa
## 19 NA 4.0 NA 0.2 setosa
## 20 5.6 NA 4.200 1.3 versicolor
## 21 4.9 3.6 NA 0.1 setosa
## 22 5.4 NA 4.500 1.5 versicolor
## 23 6.2 2.8 NA 1.8 virginica
## 24 6.7 3.3 5.700 2.5 virginica
## 25 NA 3.0 5.900 2.1 virginica
## 26 4.6 3.2 1.400 0.2 setosa
## 27 4.9 3.1 1.500 0.1 setosa
## 28 73.0 29.0 63.000 NA virginica
## 29 6.5 3.2 5.100 2.0 virginica
## 30 NA 2.8 0.820 1.3 versicolor
## 31 4.4 3.2 NA 0.2 setosa
## 32 5.9 3.2 4.800 NA versicolor
## 33 5.7 2.8 4.500 1.3 versicolor
## 34 6.2 2.9 NA 1.3 versicolor
## 35 6.6 2.9 23.000 1.3 versicolor
## 36 4.8 3.0 1.400 0.1 setosa
## 37 6.5 3.0 5.500 1.8 virginica
## 38 6.2 2.2 4.500 1.5 versicolor
## 39 6.7 2.5 5.800 1.8 virginica
## 40 5.0 3.0 1.600 0.2 setosa
## 41 5.0 NA 1.200 0.2 setosa
## 42 5.8 2.7 3.900 1.2 versicolor
## 43 0.0 NA 1.300 0.4 setosa
## 44 5.8 2.7 5.100 1.9 virginica
## 45 5.5 4.2 1.400 0.2 setosa
## 46 7.7 2.8 6.700 2.0 virginica
## 47 5.7 NA NA 0.4 setosa
## 48 7.0 3.2 4.700 1.4 versicolor
## 49 6.5 3.0 5.800 2.2 virginica
## 50 6.0 3.4 4.500 1.6 versicolor
## 51 5.5 2.6 4.400 1.2 versicolor
## 52 4.9 3.1 NA 0.2 setosa
## 53 5.2 2.7 3.900 1.4 versicolor
## 54 4.8 3.4 1.600 0.2 setosa
## 55 6.3 3.3 4.700 1.6 versicolor
## 56 7.7 3.8 6.700 2.2 virginica
## 57 5.1 3.8 1.500 0.3 setosa
## 58 NA 2.9 4.500 1.5 versicolor
## 59 6.4 2.8 5.600 NA virginica
## 60 6.4 2.8 5.600 2.1 virginica
## 61 5.0 2.3 3.300 NA versicolor
## 62 7.4 2.8 6.100 1.9 virginica
## 63 4.3 3.0 1.100 0.1 setosa
## 64 5.0 3.3 1.400 0.2 setosa
## 65 7.2 3.0 5.800 1.6 virginica
## 66 6.3 2.5 4.900 1.5 versicolor
## 67 5.1 2.5 NA 1.1 versicolor
## 68 NA 3.2 5.700 2.3 virginica
## 69 5.1 3.5 NA NA setosa
## 70 5.0 3.5 1.300 0.3 setosa
## 71 6.1 3.0 4.600 1.4 versicolor
## 72 6.9 3.1 5.100 2.3 virginica
## 73 5.1 3.5 1.400 0.3 setosa
## 74 6.5 NA 4.600 1.5 versicolor
## 75 5.6 2.8 4.900 2.0 virginica
## 76 4.9 2.5 4.500 NA virginica
## 77 5.5 3.5 1.300 0.2 setosa
## 78 7.6 3.0 6.600 2.1 virginica
## 79 5.1 3.8 0.000 0.2 setosa
## 80 7.9 3.8 6.400 2.0 virginica
## 81 6.1 2.6 5.600 1.4 virginica
## 82 5.4 3.4 1.700 0.2 setosa
## 83 6.1 2.9 4.700 1.4 versicolor
## 84 5.4 3.7 1.500 0.2 setosa
## 85 6.7 3.0 5.200 2.3 virginica
## 86 5.1 3.8 1.900 Inf setosa
## 87 6.4 2.9 4.300 1.3 versicolor
## 88 5.7 2.9 4.200 1.3 versicolor
## 89 4.4 2.9 1.400 0.2 setosa
## 90 6.3 2.5 5.000 1.9 virginica
## 91 7.2 3.2 6.000 1.8 virginica
## 92 4.9 NA 3.300 1.0 versicolor
## 93 5.2 3.4 1.400 0.2 setosa
## 94 5.8 2.7 5.100 1.9 virginica
## 95 6.0 2.2 5.000 1.5 virginica
## 96 6.9 3.1 NA 1.5 versicolor
## 97 5.5 2.3 4.000 1.3 versicolor
## 98 6.7 NA 5.000 1.7 versicolor
## 99 5.7 3.0 4.200 1.2 versicolor
## 100 6.3 2.8 5.100 1.5 virginica
## 101 5.4 3.4 1.500 0.4 setosa
## 102 7.2 3.6 NA 2.5 virginica
## 103 6.3 2.7 4.900 NA virginica
## 104 5.6 3.0 4.100 1.3 versicolor
## 105 5.1 3.7 NA 0.4 setosa
## 106 5.5 NA 0.925 1.0 versicolor
## 107 6.5 3.0 5.200 2.0 virginica
## 108 4.8 3.0 1.400 NA setosa
## 109 6.1 2.8 NA 1.3 versicolor
## 110 4.6 3.4 1.400 0.3 setosa
## 111 6.3 3.4 NA 2.4 virginica
## 112 5.0 3.4 1.500 0.2 setosa
## 113 5.1 3.4 1.500 0.2 setosa
## 114 NA 3.3 5.700 2.1 virginica
## 115 6.7 3.1 4.700 1.5 versicolor
## 116 7.7 2.6 6.900 2.3 virginica
## 117 6.3 NA 4.400 1.3 versicolor
## 118 4.6 3.1 1.500 0.2 setosa
## 119 NA 3.0 5.500 2.1 virginica
## 120 NA 2.8 4.700 1.2 versicolor
## 121 5.9 3.0 NA 1.5 versicolor
## 122 4.5 2.3 1.300 0.3 setosa
## 123 6.4 3.2 5.300 2.3 virginica
## 124 5.2 4.1 1.500 0.1 setosa
## 125 49.0 30.0 14.000 2.0 setosa
## 126 5.6 2.9 3.600 1.3 versicolor
## 127 6.8 3.2 5.900 2.3 virginica
## 128 5.8 NA 5.100 2.4 virginica
## 129 4.6 3.6 NA 0.2 setosa
## 130 5.7 0.0 1.700 0.3 setosa
## 131 5.6 2.5 3.900 1.1 versicolor
## 132 6.7 3.1 4.400 1.4 versicolor
## 133 4.8 NA 1.900 0.2 setosa
## 134 5.1 3.3 1.700 0.5 setosa
## 135 4.4 3.0 1.300 NA setosa
## 136 7.7 3.0 NA 2.3 virginica
## 137 4.7 3.2 1.600 0.2 setosa
## 138 NA 3.0 4.900 1.8 virginica
## 139 6.9 3.1 5.400 2.1 virginica
## 140 6.0 2.2 4.000 1.0 versicolor
## 141 5.0 NA 1.400 0.2 setosa
## 142 5.5 NA 3.800 1.1 versicolor
## 143 6.6 3.0 4.400 1.4 versicolor
## 144 6.3 2.9 5.600 1.8 virginica
## 145 5.7 2.5 5.000 2.0 virginica
## 146 6.7 3.1 5.600 2.4 virginica
## 147 5.6 3.0 4.500 1.5 versicolor
## 148 5.2 3.5 1.500 0.2 setosa
## 149 6.4 3.1 NA 1.8 virginica
## 150 5.8 2.6 4.000 NA versicolorDealing with NA
In all of our data analyses so far we implicitly assumed that we don’t have any missing values in our data. In practice, that is often not the case. While some statistical and machine learning methods work with missing data, many commonly used methods can’t, so it is important to learn how to deal with missing values.
The severity of NA
If we judge by the imputation or data removing methods that are most commonly used in practice, we might conclude that missing data is a relatively simple problem that is secondary to the inference, predictive modelling, etc. that are the primary goal of our analysis. Unfortunately, that is not the case. Dealing with missing values is very challenging, often in itself a modelling problem.
Three classes of NA’s
The choice of an appropriate method is inseparable from our understanding of or assumptions about the process that generated the missing values (the missingness mechanism). Based on the characteristics of this process we typically characterize the missing data problem as one of these three cases:
MCAR (Missing Completely At Random): Whether or not a value is missing is independent of both the observed values and the missing (unobserved) values. For example, if we had temperature measuring devices at different locations and they occassionally and random intervals stopped working. Or, in surveys, where respondents don’t respond with a certain probability, independent of the characteristics that we are surveying.
MAR (Missing At Random): Whether or not a value is missing is independent of the missing (unobserved) values but depends on the observed values. That is, there is a pattern to how the values are missing, but we could fully explain that pattern given only the observed data. For example, if our temperature measuring devices stopped working more often in certain locations than in others. Or, in surveys, if women are less likely to report their weight than men.
MNAR (Missing Not At Random): Whether or not a value is missing also depends on the missing (unobserved) values in a way that can’t be explained by the observed values. That is, there is a pattern to how the values are missing, but we wouldn’t be able to fully explain it without observing the values that are missing. For example, if our temperature measuring device had a tendency to stop working when the temperature is very low. Or, in surveys, if a person was less likely to report their salary if their salary was high.
The NA’s mechanism
Every variable in our data might have a different missingness mechanism. So, how do we determine whether it is MCAR, MAR, or MNAR?
Showing with a reasonable degree of certainty that the mechanism is not MCAR is equivalent to showing that the missingness (whether or not a value is missing) can be predicted from observed values. That is, it is a prediction problem and it is sufficient to show one way that missingness can be predicted. On the other hand, it is infeasible to show that the mechanism is MCAR, because that would require us to show that there is no way of predicting missingness from observed values. We can, however, rule out certain kinds of dependency (for example, linear dependency).
For MNAR, the situation is even worse. In general, it is impossible to determine from the data the relationship between missingness and the value that is missing, because we don’t know what is missing. That is, unless we are able to somehow measure the values that are missing, we won’t be able to determine whether or not the missingness regime is MNAR. Getting our hands on the missing values, however, is in most cases impossible or infeasible.
To summarize, we’ll often be able to show that our missingness regime is not MCAR and never that it is MCAR. Subsequently, we’ll often know that the missingness regime is at least MAR, but we’ll rarely be able to determine whether it is MAR or MNAR, unless we can get our hands on the missing data. Therefore, it becomes very important to utilize not only data but also domain-specific background knowledge, when applicable. In particular, known relationships between the variables in our data and what caused the values to be missing.
Causes for NA’s
Understanding the cause for missing data can often help us identify the missingness mechanism and avoid introducing a bias. In general, we can split the causes into two classes: intentional or unintentional.
Intentionally or systematically missing data are missing by design. For example:
- patients that did not experience pain were not asked to rate their pain,
- a patient that has not been discharged from the hospital doesn’t have a ‘days spent in hospital care’ data point (although we can infer a lower bound from the day of arrival) and
- a particular prediction algorithm’s performance was measured on datasets with fewer than 100 variables, due to its time complexity and
- some measurements were not made because it was too costly to make all of them.
Unintentionally missing data were not planned. For example:
- data missing due to measurement error,
- a subject skipping a survey question,
- a patient prematurely dropping out of a study
and other reasons not planned by and outside the control of the data collector.
There is no general rule and further conclusions can only be made on a case-by-case basis, using domain specific knowledge. Measurement error can range from completly random to completely not-at-random, such a temperature sensor breaking down at high temperatures. Subjects typically do not drop out of studies at random, but that is also a possiblity. When not all measurements are made to reduce cost, they are often omitted completely at random, but sometimes a different design is used.
Detecting NA
A missing value, represented by NA in R, is a placeholder for a datum of which the type is known but its value isn’t. Therefore, it is impossible to perform statistical analysis on data where one or more values in the data are missing. One may choose to either omit elements from a dataset that contain missing values or to impute a value, but missingness is something to be dealt with prior to any analysis.
Can you see that many values in our dataset have status NA = Not Available? Count (or plot), how many (%) of all 150 rows is complete.
sum(complete.cases(mydata))## [1] 96nrow(mydata[complete.cases(mydata), ])/nrow(mydata)*100## [1] 64Does the data contain other special values? If it does, replace them with NA.
is.special <- function(x){
if (is.numeric(x)) !is.finite(x) else is.na(x)
}
sapply(mydata, is.special)## Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## [1,] FALSE FALSE FALSE FALSE FALSE
## [2,] FALSE FALSE FALSE FALSE FALSE
## [3,] FALSE TRUE FALSE FALSE FALSE
## [4,] FALSE FALSE FALSE FALSE FALSE
## [5,] FALSE FALSE FALSE FALSE FALSE
## [6,] FALSE TRUE TRUE FALSE FALSE
## [7,] FALSE FALSE FALSE TRUE FALSE
## [8,] FALSE FALSE FALSE FALSE FALSE
## [9,] FALSE FALSE FALSE FALSE FALSE
## [10,] FALSE FALSE FALSE FALSE FALSE
## [11,] FALSE FALSE FALSE FALSE FALSE
## [12,] FALSE FALSE FALSE FALSE FALSE
## [13,] FALSE FALSE FALSE TRUE FALSE
## [14,] FALSE FALSE FALSE FALSE FALSE
## [15,] TRUE FALSE FALSE FALSE FALSE
## [16,] FALSE FALSE FALSE FALSE FALSE
## [17,] FALSE TRUE FALSE FALSE FALSE
## [18,] FALSE FALSE FALSE FALSE FALSE
## [19,] TRUE FALSE TRUE FALSE FALSE
## [20,] FALSE TRUE FALSE FALSE FALSE
## [21,] FALSE FALSE TRUE FALSE FALSE
## [22,] FALSE TRUE FALSE FALSE FALSE
## [23,] FALSE FALSE TRUE FALSE FALSE
## [24,] FALSE FALSE FALSE FALSE FALSE
## [25,] TRUE FALSE FALSE FALSE FALSE
## [26,] FALSE FALSE FALSE FALSE FALSE
## [27,] FALSE FALSE FALSE FALSE FALSE
## [28,] FALSE FALSE FALSE TRUE FALSE
## [29,] FALSE FALSE FALSE FALSE FALSE
## [30,] TRUE FALSE FALSE FALSE FALSE
## [31,] FALSE FALSE TRUE FALSE FALSE
## [32,] FALSE FALSE FALSE TRUE FALSE
## [33,] FALSE FALSE FALSE FALSE FALSE
## [34,] FALSE FALSE TRUE FALSE FALSE
## [35,] FALSE FALSE FALSE FALSE FALSE
## [36,] FALSE FALSE FALSE FALSE FALSE
## [37,] FALSE FALSE FALSE FALSE FALSE
## [38,] FALSE FALSE FALSE FALSE FALSE
## [39,] FALSE FALSE FALSE FALSE FALSE
## [40,] FALSE FALSE FALSE FALSE FALSE
## [41,] FALSE TRUE FALSE FALSE FALSE
## [42,] FALSE FALSE FALSE FALSE FALSE
## [43,] FALSE TRUE FALSE FALSE FALSE
## [44,] FALSE FALSE FALSE FALSE FALSE
## [45,] FALSE FALSE FALSE FALSE FALSE
## [46,] FALSE FALSE FALSE FALSE FALSE
## [47,] FALSE TRUE TRUE FALSE FALSE
## [48,] FALSE FALSE FALSE FALSE FALSE
## [49,] FALSE FALSE FALSE FALSE FALSE
## [50,] FALSE FALSE FALSE FALSE FALSE
## [51,] FALSE FALSE FALSE FALSE FALSE
## [52,] FALSE FALSE TRUE FALSE FALSE
## [53,] FALSE FALSE FALSE FALSE FALSE
## [54,] FALSE FALSE FALSE FALSE FALSE
## [55,] FALSE FALSE FALSE FALSE FALSE
## [56,] FALSE FALSE FALSE FALSE FALSE
## [57,] FALSE FALSE FALSE FALSE FALSE
## [58,] TRUE FALSE FALSE FALSE FALSE
## [59,] FALSE FALSE FALSE TRUE FALSE
## [60,] FALSE FALSE FALSE FALSE FALSE
## [61,] FALSE FALSE FALSE TRUE FALSE
## [62,] FALSE FALSE FALSE FALSE FALSE
## [63,] FALSE FALSE FALSE FALSE FALSE
## [64,] FALSE FALSE FALSE FALSE FALSE
## [65,] FALSE FALSE FALSE FALSE FALSE
## [66,] FALSE FALSE FALSE FALSE FALSE
## [67,] FALSE FALSE TRUE FALSE FALSE
## [68,] TRUE FALSE FALSE FALSE FALSE
## [69,] FALSE FALSE TRUE TRUE FALSE
## [70,] FALSE FALSE FALSE FALSE FALSE
## [71,] FALSE FALSE FALSE FALSE FALSE
## [72,] FALSE FALSE FALSE FALSE FALSE
## [73,] FALSE FALSE FALSE FALSE FALSE
## [74,] FALSE TRUE FALSE FALSE FALSE
## [75,] FALSE FALSE FALSE FALSE FALSE
## [76,] FALSE FALSE FALSE TRUE FALSE
## [77,] FALSE FALSE FALSE FALSE FALSE
## [78,] FALSE FALSE FALSE FALSE FALSE
## [79,] FALSE FALSE FALSE FALSE FALSE
## [80,] FALSE FALSE FALSE FALSE FALSE
## [81,] FALSE FALSE FALSE FALSE FALSE
## [82,] FALSE FALSE FALSE FALSE FALSE
## [83,] FALSE FALSE FALSE FALSE FALSE
## [84,] FALSE FALSE FALSE FALSE FALSE
## [85,] FALSE FALSE FALSE FALSE FALSE
## [86,] FALSE FALSE FALSE TRUE FALSE
## [87,] FALSE FALSE FALSE FALSE FALSE
## [88,] FALSE FALSE FALSE FALSE FALSE
## [89,] FALSE FALSE FALSE FALSE FALSE
## [90,] FALSE FALSE FALSE FALSE FALSE
## [91,] FALSE FALSE FALSE FALSE FALSE
## [92,] FALSE TRUE FALSE FALSE FALSE
## [93,] FALSE FALSE FALSE FALSE FALSE
## [94,] FALSE FALSE FALSE FALSE FALSE
## [95,] FALSE FALSE FALSE FALSE FALSE
## [96,] FALSE FALSE TRUE FALSE FALSE
## [97,] FALSE FALSE FALSE FALSE FALSE
## [98,] FALSE TRUE FALSE FALSE FALSE
## [99,] FALSE FALSE FALSE FALSE FALSE
## [100,] FALSE FALSE FALSE FALSE FALSE
## [101,] FALSE FALSE FALSE FALSE FALSE
## [102,] FALSE FALSE TRUE FALSE FALSE
## [103,] FALSE FALSE FALSE TRUE FALSE
## [104,] FALSE FALSE FALSE FALSE FALSE
## [105,] FALSE FALSE TRUE FALSE FALSE
## [106,] FALSE TRUE FALSE FALSE FALSE
## [107,] FALSE FALSE FALSE FALSE FALSE
## [108,] FALSE FALSE FALSE TRUE FALSE
## [109,] FALSE FALSE TRUE FALSE FALSE
## [110,] FALSE FALSE FALSE FALSE FALSE
## [111,] FALSE FALSE TRUE FALSE FALSE
## [112,] FALSE FALSE FALSE FALSE FALSE
## [113,] FALSE FALSE FALSE FALSE FALSE
## [114,] TRUE FALSE FALSE FALSE FALSE
## [115,] FALSE FALSE FALSE FALSE FALSE
## [116,] FALSE FALSE FALSE FALSE FALSE
## [117,] FALSE TRUE FALSE FALSE FALSE
## [118,] FALSE FALSE FALSE FALSE FALSE
## [119,] TRUE FALSE FALSE FALSE FALSE
## [120,] TRUE FALSE FALSE FALSE FALSE
## [121,] FALSE FALSE TRUE FALSE FALSE
## [122,] FALSE FALSE FALSE FALSE FALSE
## [123,] FALSE FALSE FALSE FALSE FALSE
## [124,] FALSE FALSE FALSE FALSE FALSE
## [125,] FALSE FALSE FALSE FALSE FALSE
## [126,] FALSE FALSE FALSE FALSE FALSE
## [127,] FALSE FALSE FALSE FALSE FALSE
## [128,] FALSE TRUE FALSE FALSE FALSE
## [129,] FALSE FALSE TRUE FALSE FALSE
## [130,] FALSE FALSE FALSE FALSE FALSE
## [131,] FALSE FALSE FALSE FALSE FALSE
## [132,] FALSE FALSE FALSE FALSE FALSE
## [133,] FALSE TRUE FALSE FALSE FALSE
## [134,] FALSE FALSE FALSE FALSE FALSE
## [135,] FALSE FALSE FALSE TRUE FALSE
## [136,] FALSE FALSE TRUE FALSE FALSE
## [137,] FALSE FALSE FALSE FALSE FALSE
## [138,] TRUE FALSE FALSE FALSE FALSE
## [139,] FALSE FALSE FALSE FALSE FALSE
## [140,] FALSE FALSE FALSE FALSE FALSE
## [141,] FALSE TRUE FALSE FALSE FALSE
## [142,] FALSE TRUE FALSE FALSE FALSE
## [143,] FALSE FALSE FALSE FALSE FALSE
## [144,] FALSE FALSE FALSE FALSE FALSE
## [145,] FALSE FALSE FALSE FALSE FALSE
## [146,] FALSE FALSE FALSE FALSE FALSE
## [147,] FALSE FALSE FALSE FALSE FALSE
## [148,] FALSE FALSE FALSE FALSE FALSE
## [149,] FALSE FALSE TRUE FALSE FALSE
## [150,] FALSE FALSE FALSE TRUE FALSEfor (n in colnames(mydata)){
is.na(mydata[[n]]) <- is.special(mydata[[n]])
}
summary(mydata)## Sepal.Length Sepal.Width Petal.Length Petal.Width
## Min. : 0.000 Min. :-3.000 Min. : 0.00 Min. :0.100
## 1st Qu.: 5.100 1st Qu.: 2.800 1st Qu.: 1.60 1st Qu.:0.300
## Median : 5.750 Median : 3.000 Median : 4.50 Median :1.300
## Mean : 6.559 Mean : 3.391 Mean : 4.45 Mean :1.207
## 3rd Qu.: 6.400 3rd Qu.: 3.300 3rd Qu.: 5.10 3rd Qu.:1.800
## Max. :73.000 Max. :30.000 Max. :63.00 Max. :2.500
## NA's :10 NA's :17 NA's :19 NA's :13
## Species
## Length:150
## Class :character
## Mode :character
##
##
##
## NaniaR
OMG! So hard :-) It’s better to use visualizations, tests… or automatically detect NA’s with some function…
There are a variety of different plots to explore missing data
available in the naniar package. If you would like to know more about
the philosophy of the naniar package, you should read the
vignette Getting
Started with naniar.
A key point to remember with the visualisation tools in
naniar is that there is a way to get the data from the plot
out from the visualisation.
Exploring NA’s
One of the first plots that I recommend you start with when you are
first exploring your missing data, is the vis_miss() plot,
which is re-exported from visdat.
vis_miss(airquality)
This plot provides a specific visualiation of the amount of missing data, showing in black the location of missing values, and also providing information on the overall percentage of missing values overall (in the legend), and in each variable.
Patterns
An upset plot from the UpSetR package can be used to
visualise the patterns of missingness, or rather the combinations of
missingness across cases. To see combinations of missingness and
intersections of missingness amongst variables, use the
gg_miss_upset function:
gg_miss_upset(airquality)
We can explore this with more complex data, such as riskfactors:
gg_miss_upset(riskfactors)
NA’s Mechanisms
There are a few different ways to explore different missing data
mechanisms and relationships. One way incorporates the method of
shifting missing values so that they can be visualised on the same axes
as the regular values, and then colours the missing and not missing
points. This is implemented with geom_miss_point().
# using regular geom_point()
ggplot(airquality,
aes(x = Ozone,
y = Solar.R)) +
geom_miss_point()
NA’s in vars
This plot shows the number of missing values in each variable in a
dataset. It is powered by the miss_var_summary()
function.
gg_miss_var(airquality)
gg_miss_var(airquality) + labs(y = "Look at all the missing ones")
NA’s in cases
This plot shows the number of missing values in each case. It is
powered by the miss_case_summary() function.
gg_miss_case(airquality) + labs(x = "Number of Cases")
NA’s across factors
This plot shows the number of missings in each column, broken down by
a categorical variable from the dataset. It is powered by a
dplyr::group_by statement followed by
miss_var_summary().
gg_miss_fct(x = riskfactors, fct = marital)
Summaries for NA’s
naniar provides numerical summaries of missing data, that follow a consistent rule that uses a syntax begining with miss_.
miss_var_summary(airquality)## # A tibble: 6 × 3
## variable n_miss pct_miss
## <chr> <int> <dbl>
## 1 Ozone 37 24.2
## 2 Solar.R 7 4.58
## 3 Wind 0 0
## 4 Temp 0 0
## 5 Month 0 0
## 6 Day 0 0You could also group_by() to work out the number of missings in each variable across the levels within it.
airquality %>%
group_by(Month) %>%
miss_var_summary()## # A tibble: 25 × 4
## # Groups: Month [5]
## Month variable n_miss pct_miss
## <int> <chr> <int> <dbl>
## 1 5 Ozone 5 16.1
## 2 5 Solar.R 4 12.9
## 3 5 Wind 0 0
## 4 5 Temp 0 0
## 5 5 Day 0 0
## 6 6 Ozone 21 70
## 7 6 Solar.R 0 0
## 8 6 Wind 0 0
## 9 6 Temp 0 0
## 10 6 Day 0 0
## # ℹ 15 more rowsIdentify outliers
Thanks to rstatix library we can easily detect outliers using boxplot methods. Boxplots are a popular and an easy method for identifying outliers.
# Convert the variable dose from a numeric to a factor variable
tooth <- as.data.frame(ToothGrowth)
tooth$dose <- as.factor(tooth$dose)
# Change box plot line colors by groups
p<-ggplot(tooth, aes(x=dose, y=len, color=dose)) +
geom_boxplot()
p
There are two categories of outlier: (1) outliers and (2) extreme points.
Values above Q3 + 1.5xIQR or below Q1 - 1.5xIQR are considered as outliers. Values above Q3 + 3xIQR or below Q1 - 3xIQR are considered as extreme points (or extreme outliers).
Q1 and Q3 are the first and third quartile, respectively. IQR is the interquartile range (IQR = Q3 - Q1).
Generally speaking, data points that are labelled outliers in boxplots are not considered as troublesome as those considered extreme points and might even be ignored. Note that, any NA and NaN are automatically removed before the quantiles are computed.
Example
tooth %>%
group_by(dose) %>%
identify_outliers(len)## # A tibble: 1 × 5
## dose len supp is.outlier is.extreme
## <fct> <dbl> <fct> <lgl> <lgl>
## 1 0.5 21.5 OJ TRUE FALSESee? one outlier! (that was visible on the boxplot)
Checking consistency
Consistent data are technically correct data that are fit for statistical analysis. They are data in which missing values, special values, (obvious) errors and outliers are either removed, corrected or imputed. The data are consistent with constraints based on real-world knowledge about the subject that the data describe.
We have the following background knowledge:
Species should be one of the following values: setosa, versicolor or virginica.
All measured numerical properties of an iris should be positive.
The petal length of an iris is at least 2 times its petal width.
The sepal length of an iris cannot exceed 30 cm.
The sepals of an iris are longer than its petals.
Define these rules in a separate object ‘RULE’ and read them into R using editset (package ‘editrules’). Print the resulting constraint object.
RULE <- editset(c("Sepal.Length <= 30","Species %in% c('setosa','versicolor','virginica')"
, "Sepal.Length > 0", "Sepal.Width > 0", "Petal.Length > 0", "Petal.Width > 0",
"Petal.Length >= 2 * Petal.Width", "Sepal.Length>Petal.Length"))
RULE##
## Data model:
## dat1 : Species %in% c('setosa', 'versicolor', 'virginica')
##
## Edit set:
## num1 : Sepal.Length <= 30
## num2 : 0 < Sepal.Length
## num3 : 0 < Sepal.Width
## num4 : 0 < Petal.Length
## num5 : 0 < Petal.Width
## num6 : 2*Petal.Width <= Petal.Length
## num7 : Petal.Length < Sepal.LengthNow we are ready to determine how often each rule is broken (violatedEdits). Also we can summarize and plot the result.
summary(violatedEdits(RULE, mydata))## Edit violations, 150 observations, 0 completely missing (0%):
##
## editname freq rel
## num6 3 2%
## num1 2 1.3%
## num3 2 1.3%
## num7 2 1.3%
## num2 1 0.7%
## num4 1 0.7%
##
## Edit violations per record:
##
## errors freq rel
## 0 90 60%
## 1 17 11.3%
## 2 13 8.7%
## 3 25 16.7%
## 4 4 2.7%
## 5 1 0.7%What percentage of the data has no errors?
violated <- violatedEdits(RULE, mydata)
summary(violated)## Edit violations, 150 observations, 0 completely missing (0%):
##
## editname freq rel
## num6 3 2%
## num1 2 1.3%
## num3 2 1.3%
## num7 2 1.3%
## num2 1 0.7%
## num4 1 0.7%
##
## Edit violations per record:
##
## errors freq rel
## 0 90 60%
## 1 17 11.3%
## 2 13 8.7%
## 3 25 16.7%
## 4 4 2.7%
## 5 1 0.7%plot(violated)
Exercise 1.
Find out which observations have too long sepals using the result of violatedEdits.
violated_obs <- mydata[violated, ]Exercise 2.
Find outliers in sepal length using boxplot and boxplot.stats. Retrieve the corresponding observations and look at the other values. Any ideas what might have happened? Set the outliers to NA (or a value that you find more appropiate)
boxplot(mydata$Sepal.Length)
outliers <- boxplot.stats(mydata$Sepal.Length)$out
outliers_idx <- which(mydata$Sepal.Length %in% outliers)
mydata[outliers_idx,]## Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 28 73 29 63.0 NA virginica
## 43 0 NA 1.3 0.4 setosa
## 125 49 30 14.0 2.0 setosa# they all seem to be too big... may they were measured in mm i.o cm?
mydata[outliers_idx,1:4] <- mydata[outliers_idx,1:4]/10
summary(mydata)## Sepal.Length Sepal.Width Petal.Length Petal.Width
## Min. :0.000 Min. :-3.000 Min. : 0.000 Min. :0.040
## 1st Qu.:5.100 1st Qu.: 2.800 1st Qu.: 1.600 1st Qu.:0.300
## Median :5.700 Median : 3.000 Median : 4.400 Median :1.300
## Mean :5.775 Mean : 2.992 Mean : 3.912 Mean :1.192
## 3rd Qu.:6.400 3rd Qu.: 3.300 3rd Qu.: 5.100 3rd Qu.:1.800
## Max. :7.900 Max. : 4.200 Max. :23.000 Max. :2.500
## NA's :10 NA's :17 NA's :19 NA's :13
## Species
## Length:150
## Class :character
## Mode :character
##
##
##
## Note that simple boxplot shows an extra outlier!
boxplot(Sepal.Length ~ Species, data=mydata)
Corrections
Replace non positive values from Sepal.Width with NA using correctWithRules from the library ‘deducorrect’.
cr <- correctionRules(expression(
if (!is.na(Sepal.Width) && Sepal.Width <=0 ) Sepal.Width = NA
))
correctWithRules(cr, mydata)## $corrected
## Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1 6.4 3.2 4.500 1.50 versicolor
## 2 6.3 3.3 6.000 2.50 virginica
## 3 6.2 NA 5.400 2.30 virginica
## 4 5.0 3.4 1.600 0.40 setosa
## 5 5.7 2.6 3.500 1.00 versicolor
## 6 5.3 NA NA 0.20 setosa
## 7 6.4 2.7 5.300 NA virginica
## 8 5.9 3.0 5.100 1.80 virginica
## 9 5.8 2.7 4.100 1.00 versicolor
## 10 4.8 3.1 1.600 0.20 setosa
## 11 5.0 3.5 1.600 0.60 setosa
## 12 6.0 2.7 5.100 1.60 versicolor
## 13 6.0 3.0 4.800 NA virginica
## 14 6.8 2.8 4.800 1.40 versicolor
## 15 NA 3.9 1.700 0.40 setosa
## 16 5.0 NA 3.500 1.00 versicolor
## 17 5.5 NA 4.000 1.30 versicolor
## 18 4.7 3.2 1.300 0.20 setosa
## 19 NA 4.0 NA 0.20 setosa
## 20 5.6 NA 4.200 1.30 versicolor
## 21 4.9 3.6 NA 0.10 setosa
## 22 5.4 NA 4.500 1.50 versicolor
## 23 6.2 2.8 NA 1.80 virginica
## 24 6.7 3.3 5.700 2.50 virginica
## 25 NA 3.0 5.900 2.10 virginica
## 26 4.6 3.2 1.400 0.20 setosa
## 27 4.9 3.1 1.500 0.10 setosa
## 28 7.3 2.9 6.300 NA virginica
## 29 6.5 3.2 5.100 2.00 virginica
## 30 NA 2.8 0.820 1.30 versicolor
## 31 4.4 3.2 NA 0.20 setosa
## 32 5.9 3.2 4.800 NA versicolor
## 33 5.7 2.8 4.500 1.30 versicolor
## 34 6.2 2.9 NA 1.30 versicolor
## 35 6.6 2.9 23.000 1.30 versicolor
## 36 4.8 3.0 1.400 0.10 setosa
## 37 6.5 3.0 5.500 1.80 virginica
## 38 6.2 2.2 4.500 1.50 versicolor
## 39 6.7 2.5 5.800 1.80 virginica
## 40 5.0 3.0 1.600 0.20 setosa
## 41 5.0 NA 1.200 0.20 setosa
## 42 5.8 2.7 3.900 1.20 versicolor
## 43 0.0 NA 0.130 0.04 setosa
## 44 5.8 2.7 5.100 1.90 virginica
## 45 5.5 4.2 1.400 0.20 setosa
## 46 7.7 2.8 6.700 2.00 virginica
## 47 5.7 NA NA 0.40 setosa
## 48 7.0 3.2 4.700 1.40 versicolor
## 49 6.5 3.0 5.800 2.20 virginica
## 50 6.0 3.4 4.500 1.60 versicolor
## 51 5.5 2.6 4.400 1.20 versicolor
## 52 4.9 3.1 NA 0.20 setosa
## 53 5.2 2.7 3.900 1.40 versicolor
## 54 4.8 3.4 1.600 0.20 setosa
## 55 6.3 3.3 4.700 1.60 versicolor
## 56 7.7 3.8 6.700 2.20 virginica
## 57 5.1 3.8 1.500 0.30 setosa
## 58 NA 2.9 4.500 1.50 versicolor
## 59 6.4 2.8 5.600 NA virginica
## 60 6.4 2.8 5.600 2.10 virginica
## 61 5.0 2.3 3.300 NA versicolor
## 62 7.4 2.8 6.100 1.90 virginica
## 63 4.3 3.0 1.100 0.10 setosa
## 64 5.0 3.3 1.400 0.20 setosa
## 65 7.2 3.0 5.800 1.60 virginica
## 66 6.3 2.5 4.900 1.50 versicolor
## 67 5.1 2.5 NA 1.10 versicolor
## 68 NA 3.2 5.700 2.30 virginica
## 69 5.1 3.5 NA NA setosa
## 70 5.0 3.5 1.300 0.30 setosa
## 71 6.1 3.0 4.600 1.40 versicolor
## 72 6.9 3.1 5.100 2.30 virginica
## 73 5.1 3.5 1.400 0.30 setosa
## 74 6.5 NA 4.600 1.50 versicolor
## 75 5.6 2.8 4.900 2.00 virginica
## 76 4.9 2.5 4.500 NA virginica
## 77 5.5 3.5 1.300 0.20 setosa
## 78 7.6 3.0 6.600 2.10 virginica
## 79 5.1 3.8 0.000 0.20 setosa
## 80 7.9 3.8 6.400 2.00 virginica
## 81 6.1 2.6 5.600 1.40 virginica
## 82 5.4 3.4 1.700 0.20 setosa
## 83 6.1 2.9 4.700 1.40 versicolor
## 84 5.4 3.7 1.500 0.20 setosa
## 85 6.7 3.0 5.200 2.30 virginica
## 86 5.1 3.8 1.900 NA setosa
## 87 6.4 2.9 4.300 1.30 versicolor
## 88 5.7 2.9 4.200 1.30 versicolor
## 89 4.4 2.9 1.400 0.20 setosa
## 90 6.3 2.5 5.000 1.90 virginica
## 91 7.2 3.2 6.000 1.80 virginica
## 92 4.9 NA 3.300 1.00 versicolor
## 93 5.2 3.4 1.400 0.20 setosa
## 94 5.8 2.7 5.100 1.90 virginica
## 95 6.0 2.2 5.000 1.50 virginica
## 96 6.9 3.1 NA 1.50 versicolor
## 97 5.5 2.3 4.000 1.30 versicolor
## 98 6.7 NA 5.000 1.70 versicolor
## 99 5.7 3.0 4.200 1.20 versicolor
## 100 6.3 2.8 5.100 1.50 virginica
## 101 5.4 3.4 1.500 0.40 setosa
## 102 7.2 3.6 NA 2.50 virginica
## 103 6.3 2.7 4.900 NA virginica
## 104 5.6 3.0 4.100 1.30 versicolor
## 105 5.1 3.7 NA 0.40 setosa
## 106 5.5 NA 0.925 1.00 versicolor
## 107 6.5 3.0 5.200 2.00 virginica
## 108 4.8 3.0 1.400 NA setosa
## 109 6.1 2.8 NA 1.30 versicolor
## 110 4.6 3.4 1.400 0.30 setosa
## 111 6.3 3.4 NA 2.40 virginica
## 112 5.0 3.4 1.500 0.20 setosa
## 113 5.1 3.4 1.500 0.20 setosa
## 114 NA 3.3 5.700 2.10 virginica
## 115 6.7 3.1 4.700 1.50 versicolor
## 116 7.7 2.6 6.900 2.30 virginica
## 117 6.3 NA 4.400 1.30 versicolor
## 118 4.6 3.1 1.500 0.20 setosa
## 119 NA 3.0 5.500 2.10 virginica
## 120 NA 2.8 4.700 1.20 versicolor
## 121 5.9 3.0 NA 1.50 versicolor
## 122 4.5 2.3 1.300 0.30 setosa
## 123 6.4 3.2 5.300 2.30 virginica
## 124 5.2 4.1 1.500 0.10 setosa
## 125 4.9 3.0 1.400 0.20 setosa
## 126 5.6 2.9 3.600 1.30 versicolor
## 127 6.8 3.2 5.900 2.30 virginica
## 128 5.8 NA 5.100 2.40 virginica
## 129 4.6 3.6 NA 0.20 setosa
## 130 5.7 NA 1.700 0.30 setosa
## 131 5.6 2.5 3.900 1.10 versicolor
## 132 6.7 3.1 4.400 1.40 versicolor
## 133 4.8 NA 1.900 0.20 setosa
## 134 5.1 3.3 1.700 0.50 setosa
## 135 4.4 3.0 1.300 NA setosa
## 136 7.7 3.0 NA 2.30 virginica
## 137 4.7 3.2 1.600 0.20 setosa
## 138 NA 3.0 4.900 1.80 virginica
## 139 6.9 3.1 5.400 2.10 virginica
## 140 6.0 2.2 4.000 1.00 versicolor
## 141 5.0 NA 1.400 0.20 setosa
## 142 5.5 NA 3.800 1.10 versicolor
## 143 6.6 3.0 4.400 1.40 versicolor
## 144 6.3 2.9 5.600 1.80 virginica
## 145 5.7 2.5 5.000 2.00 virginica
## 146 6.7 3.1 5.600 2.40 virginica
## 147 5.6 3.0 4.500 1.50 versicolor
## 148 5.2 3.5 1.500 0.20 setosa
## 149 6.4 3.1 NA 1.80 virginica
## 150 5.8 2.6 4.000 NA versicolor
##
## $corrections
## row variable old new
## 1 16 Sepal.Width -3 NA
## 2 130 Sepal.Width 0 NA
## how
## 1 if (!is.na(Sepal.Width) && Sepal.Width <= 0) Sepal.Width = NA
## 2 if (!is.na(Sepal.Width) && Sepal.Width <= 0) Sepal.Width = NAReplace all erroneous values with NA using (the result of) localizeErrors:
mydata[localizeErrors(RULE, mydata)$adapt] <- NA
# anything violated?
any(violatedEdits(RULE,mydata), na.rm=TRUE)## [1] FALSEWell done! No errors!
dlookr package
After you have acquired the data, you should do the following:
- Diagnose data quality.
- If there is a problem with data quality,
- The data must be corrected or re-acquired.
- Explore data to understand the data and find scenarios for performing the analysis.
- Derive new variables or perform variable transformations.
The dlookr package makes these steps fast and easy:
- Performs an data diagnosis or automatically generates a data diagnosis report.
- Discover data in a variety of ways, and automatically generate EDA(exploratory data analysis) report.
- Impute missing values and outliers, resolve skewed data, and categorize continuous variables into categorical variables. And generates an automated report to support it.
Here we will introduce data transformation methods
provided by the dlookr package. You will learn how to transform of
tbl_df data that inherits from data.frame and
data.frame with functions provided by dlookr.
dlookr increases synergy with dplyr. Particularly in
data transformation and data wrangle, it increases the efficiency of the
tidyverse package group.
To illustrate the basic use of data transformation in the dlookr
package, I use a Carseats dataset. Carseats in
the ISLR package is simulation dataset that sells
children’s car seats at 400 stores. This data is a data.frame created
for the purpose of predicting sales volume.
str(Carseats)## 'data.frame': 400 obs. of 11 variables:
## $ Sales : num 9.5 11.22 10.06 7.4 4.15 ...
## $ CompPrice : num 138 111 113 117 141 124 115 136 132 132 ...
## $ Income : num 73 48 35 100 64 113 105 81 110 113 ...
## $ Advertising: num 11 16 10 4 3 13 0 15 0 0 ...
## $ Population : num 276 260 269 466 340 501 45 425 108 131 ...
## $ Price : num 120 83 80 97 128 72 108 120 124 124 ...
## $ ShelveLoc : Factor w/ 3 levels "Bad","Good","Medium": 1 2 3 3 1 1 3 2 3 3 ...
## $ Age : num 42 65 59 55 38 78 71 67 76 76 ...
## $ Education : num 17 10 12 14 13 16 15 10 10 17 ...
## $ Urban : Factor w/ 2 levels "No","Yes": 2 2 2 2 2 1 2 2 1 1 ...
## $ US : Factor w/ 2 levels "No","Yes": 2 2 2 2 1 2 1 2 1 2 ...The contents of individual variables are as follows. (Refer to ISLR::Carseats Man page)
- Sales
- Unit sales (in thousands) at each location
- CompPrice
- Price charged by competitor at each location
- Income
- Community income level (in thousands of dollars)
- Advertising
- Local advertising budget for company at each location (in thousands of dollars)
- Population
- Population size in region (in thousands)
- Price
- Price company charges for car seats at each site
- ShelveLoc
- A factor with levels Bad, Good and Medium indicating the quality of the shelving location for the car seats at each site
- Age
- Average age of the local population
- Education
- Education level at each location
- Urban
- A factor with levels No and Yes to indicate whether the store is in an urban or rural location
- US
- A factor with levels No and Yes to indicate whether the store is in the US or not
When data analysis is performed, data containing missing values is
often encountered. However, Carseats is complete data
without missing. Therefore, the missing values are generated as follows.
And I created a data.frame object named carseats.
carseats <- ISLR::Carseats
suppressWarnings(RNGversion("3.5.0"))
set.seed(123)
carseats[sample(seq(NROW(carseats)), 20), "Income"] <- NA
suppressWarnings(RNGversion("3.5.0"))
set.seed(456)
carseats[sample(seq(NROW(carseats)), 10), "Urban"] <- NAFunctions
dlookr imputes missing values and outliers and resolves skewed data. It also provides the ability to bin continuous variables as categorical variables.
Here is a list of the data conversion functions and functions provided by dlookr:
find_na()finds a variable that contains the missing values variable, andimputate_na()imputes the missing values.find_outliers()finds a variable that contains the outliers, andimputate_outlier()imputes the outlier.summary.imputation()andplot.imputation()provide information and visualization of the imputed variables.find_skewness()finds the variables of the skewed data, andtransform()performs the resolving of the skewed data.transform()also performs standardization of numeric variables.summary.transform()andplot.transform()provide information and visualization of transformed variables.binning()andbinning_by()convert binational data into categorical data.print.bins()andsummary.bins()show and summarize the binning results.plot.bins()andplot.optimal_bins()provide visualization of the binning result.transformation_report()performs the data transform and reports the result.
Imputations of NA’s
imputate_na() imputes the missing value contained in the
variable. The predictor with missing values support both numeric and
categorical variables, and supports the following
method.
Imputation is the process of estimating or deriving values for fields where data is missing. There is a vast body of literature on imputation methods and it goes beyond the scope of this tutorial to discuss all of them.
There is no one single best imputation method that works in all cases. The imputation model of choice depends on what auxiliary information is available and whether there are (multivariate) edit restrictions on the data to be imputed. The availability of R software for imputation under edit restrictions is, to our best knowledge, limited. However, a viable strategy for imputing numerical data is to first impute missing values without restrictions, and then minimally adjust the imputed values so that the restrictions are obeyed. Separately, these methods are available in R.
- predictor is numerical variable
- “mean” : arithmetic mean
- “median” : median
- “mode” : mode
- “knn” : K-nearest neighbors
- target variable must be specified
- “rpart” : Recursive Partitioning and Regression Trees
- target variable must be specified
- target variable must be specified
- “mice” : Multivariate Imputation by Chained Equations
- target variable must be specified
- random seed must be set
- target variable must be specified
- predictor is categorical variable
- “mode” : mode
- “rpart” : Recursive Partitioning and Regression Trees
- target variable must be specified
- target variable must be specified
- “mice” : Multivariate Imputation by Chained Equations
- target variable must be specified
- random seed must be set
- target variable must be specified
Example
In the following example, imputate_na() imputes the
missing value of Income, a numeric variable of carseats,
using the “rpart” method. summary() summarizes missing
value imputation information, and plot() visualizes missing
information.
income <- imputate_na(carseats, Income, US, method = "rpart")
# summary of imputation
summary(income)## * Impute missing values based on Recursive Partitioning and Regression Trees
## - method : rpart
##
## * Information of Imputation (before vs after)
## Original Imputation
## described_variables "value" "value"
## n "380" "400"
## na "20" " 0"
## mean "68.86053" "69.05073"
## sd "28.09161" "27.57382"
## se_mean "1.441069" "1.378691"
## IQR "48.25" "46.00"
## skewness "0.04490600" "0.02935732"
## kurtosis "-1.089201" "-1.035086"
## p00 "21" "21"
## p01 "21.79" "21.99"
## p05 "26" "26"
## p10 "30.0" "30.9"
## p20 "39" "40"
## p25 "42.75" "44.00"
## p30 "48.00000" "51.58333"
## p40 "62" "63"
## p50 "69" "69"
## p60 "78.0" "77.4"
## p70 "86.3" "84.3"
## p75 "91" "90"
## p80 "96.2" "96.0"
## p90 "108.1" "106.1"
## p95 "115.05" "115.00"
## p99 "119.21" "119.01"
## p100 "120" "120" # viz of imputation
plot(income)
Example
The following imputes the categorical variable urban by
the “mice” method.
library(mice)## Warning: il pacchetto 'mice' è stato creato con R versione 4.3.2##
## Caricamento pacchetto: 'mice'## Il seguente oggetto è mascherato da 'package:stats':
##
## filter## I seguenti oggetti sono mascherati da 'package:base':
##
## cbind, rbindurban <- imputate_na(carseats, Urban, US, method = "mice")##
## iter imp variable
## 1 1 Income Urban
## 1 2 Income Urban
## 1 3 Income Urban
## 1 4 Income Urban
## 1 5 Income Urban
## 2 1 Income Urban
## 2 2 Income Urban
## 2 3 Income Urban
## 2 4 Income Urban
## 2 5 Income Urban
## 3 1 Income Urban
## 3 2 Income Urban
## 3 3 Income Urban
## 3 4 Income Urban
## 3 5 Income Urban
## 4 1 Income Urban
## 4 2 Income Urban
## 4 3 Income Urban
## 4 4 Income Urban
## 4 5 Income Urban
## 5 1 Income Urban
## 5 2 Income Urban
## 5 3 Income Urban
## 5 4 Income Urban
## 5 5 Income Urban# result of imputation
urban## [1] Yes Yes Yes Yes Yes No Yes Yes No No No Yes Yes Yes Yes No Yes Yes
## [19] No Yes Yes No Yes Yes Yes No No Yes Yes Yes Yes Yes Yes Yes Yes No
## [37] No Yes Yes No No Yes Yes Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes
## [55] No Yes Yes Yes Yes Yes Yes No Yes Yes No No Yes Yes Yes Yes Yes No
## [73] Yes No No No Yes No Yes Yes Yes Yes Yes No No No Yes No Yes No
## [91] No Yes Yes Yes Yes Yes No Yes No No No Yes No Yes Yes Yes No Yes
## [109] Yes No Yes Yes Yes Yes Yes Yes No Yes Yes Yes Yes Yes Yes No Yes No
## [127] Yes Yes Yes No Yes Yes Yes Yes Yes No No Yes Yes No Yes Yes Yes Yes
## [145] No Yes Yes No No Yes No No No No No Yes Yes No No No No No
## [163] Yes No No Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes No Yes No Yes
## [181] Yes Yes Yes Yes No Yes No Yes Yes No No Yes No Yes Yes Yes Yes Yes
## [199] Yes Yes No Yes No Yes Yes Yes Yes No Yes No No Yes Yes Yes Yes Yes
## [217] Yes No Yes Yes Yes Yes Yes Yes No Yes Yes Yes No No No No Yes No
## [235] No Yes Yes Yes Yes Yes Yes Yes No Yes Yes No Yes Yes Yes Yes Yes Yes
## [253] Yes No Yes Yes Yes Yes No No Yes Yes Yes Yes Yes Yes No No Yes Yes
## [271] Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes No Yes No No Yes No Yes
## [289] No Yes No Yes Yes Yes Yes No Yes Yes Yes No Yes Yes Yes Yes Yes Yes
## [307] Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No No Yes Yes Yes Yes
## [325] Yes Yes Yes Yes Yes Yes No Yes Yes Yes Yes Yes Yes Yes No Yes Yes No
## [343] No Yes No Yes No No Yes No No No Yes No Yes Yes Yes Yes Yes Yes
## [361] No No Yes Yes Yes No No Yes No Yes Yes Yes No Yes Yes Yes Yes No
## [379] Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes Yes Yes Yes Yes No Yes Yes
## [397] No Yes Yes Yes
## attr(,"var_type")
## [1] categorical
## attr(,"method")
## [1] mice
## attr(,"na_pos")
## [1] 33 36 84 94 113 132 151 292 313 339
## attr(,"seed")
## [1] 24283
## attr(,"type")
## [1] missing values
## attr(,"message")
## [1] complete imputation
## attr(,"success")
## [1] TRUE
## Levels: No Yes# summary of imputation
summary(urban)## * Impute missing values based on Multivariate Imputation by Chained Equations
## - method : mice
## - random seed : 24283
##
## * Information of Imputation (before vs after)
## original imputation original_percent imputation_percent
## No 115 119 28.75 29.75
## Yes 275 281 68.75 70.25
## <NA> 10 0 2.50 0.00# viz of imputation
plot(urban)
In the following exercise try to impute the missing value of the
Income variable, and then calculates the arithmetic mean
for each level of US. In this case, dplyr
should be used, and it is easily interpreted logically using pipes!
Exercise 3.
carseats %>%
mutate(Income = ifelse(is.na(Income), mean(Income, na.rm = TRUE), Income)) %>%
group_by(US) %>%
summarise(Avg_Income = mean(Income, na.rm = TRUE))## # A tibble: 2 × 2
## US Avg_Income
## <fct> <dbl>
## 1 No 66.1
## 2 Yes 70.4Imputation of outliers
imputate_outlier() imputes the outliers value. The
predictor with outliers supports only numeric variables and supports the
following methods.
- predictor is numerical variable
- “mean” : arithmetic mean
- “median” : median
- “mode” : mode
- “capping” : Impute the upper outliers with 95 percentile, and Impute the bottom outliers with 5 percentile.
Exercise 4.
Use the function imputate_outlier() to impute the
outliers of the numeric variable Price as the “capping”
method. Hint: summary() summarizes outliers imputation
information, and plot() visualizes imputation
information.
carseats %>%
mutate(Price_imp = imputate_outlier(carseats, Price, method = "capping")) %>%
summary()## Impute outliers with capping
##
## * Information of Imputation (before vs after)
## Original Imputation
## described_variables "value" "value"
## n "400" "400"
## na "0" "0"
## mean "115.7950" "115.8928"
## sd "23.67666" "22.61092"
## se_mean "1.183833" "1.130546"
## IQR "31" "31"
## skewness "-0.1252862" "-0.0461621"
## kurtosis " 0.4518850" "-0.3030578"
## p00 "24" "54"
## p01 "54.99" "67.96"
## p05 "77" "77"
## p10 "87" "87"
## p20 "96.8" "96.8"
## p25 "100" "100"
## p30 "104" "104"
## p40 "110" "110"
## p50 "117" "117"
## p60 "122" "122"
## p70 "128.3" "128.3"
## p75 "131" "131"
## p80 "134" "134"
## p90 "146" "146"
## p95 "155.0500" "155.0025"
## p99 "166.05" "164.02"
## p100 "191" "173"## Sales CompPrice Income Advertising
## Min. : 0.000 Min. : 77 Min. : 21.00 Min. : 0.000
## 1st Qu.: 5.390 1st Qu.:115 1st Qu.: 42.75 1st Qu.: 0.000
## Median : 7.490 Median :125 Median : 69.00 Median : 5.000
## Mean : 7.496 Mean :125 Mean : 68.86 Mean : 6.635
## 3rd Qu.: 9.320 3rd Qu.:135 3rd Qu.: 91.00 3rd Qu.:12.000
## Max. :16.270 Max. :175 Max. :120.00 Max. :29.000
## NA's :20
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
## Population Price ShelveLoc Age Education
## Min. : 10.0 Min. : 24.0 Bad : 96 Min. :25.00 Min. :10.0
## 1st Qu.:139.0 1st Qu.:100.0 Good : 85 1st Qu.:39.75 1st Qu.:12.0
## Median :272.0 Median :117.0 Medium:219 Median :54.50 Median :14.0
## Mean :264.8 Mean :115.8 Mean :53.32 Mean :13.9
## 3rd Qu.:398.5 3rd Qu.:131.0 3rd Qu.:66.00 3rd Qu.:16.0
## Max. :509.0 Max. :191.0 Max. :80.00 Max. :18.0
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
##
## Urban US Price_imp.Original Price_imp.Imputation
## No :115 No :142 value value
## Yes :275 Yes:258 400 400
## NA's: 10 0 0
## 115.7950 115.8928
## 23.67666 22.61092
## 1.183833 1.130546
## 31 31
## -0.1252862 -0.0461621
## 0.4518850 -0.3030578
## 24 54
## 54.99 67.96
## 77 77
## 87 87
## 96.8 96.8
## 100 100
## 104 104
## 110 110
## 117 117
## 122 122
## 128.3 128.3
## 131 131
## 134 134
## 146 146
## 155.0500 155.0025
## 166.05 164.02
## 191 173The following example imputes the outliers of the Price
variable, and then calculates the arithmetic mean for each level of
US. In this case, dplyr is used, and it is
easily interpreted logically using pipes.
Example
# The mean before and after the imputation of the Price variable
carseats %>%
mutate(Price_imp = imputate_outlier(carseats, Price, method = "capping")) %>%
group_by(US) %>%
summarise(orig = mean(Price, na.rm = TRUE),
imputation = mean(Price_imp, na.rm = TRUE))## # A tibble: 2 × 3
## US orig imputation
## <fct> <dbl> <dbl>
## 1 No 114. 114.
## 2 Yes 117. 117.Transformations
Finally, we sometimes encounter the situation where we have problems with skewed distributions or we just want to transform, recode or perform discretization. Let’s review some of the most popular transformation methods.
First, standardization (also known as normalization):
Z-score approach - standardization procedure, using the formula: \(z=\frac{x-\mu}{\sigma}\) where \(\mu\) = mean and \(\sigma\) = standard deviation. Z-scores are also known as standardized scores; they are scores (or data values) that have been given a common standard. This standard is a mean of zero and a standard deviation of 1.
minmax approach - An alternative approach to Z-score normalization (or standardization) is the so-called MinMax scaling (often also simply called “normalization” - a common cause for ambiguities). In this approach, the data is scaled to a fixed range - usually 0 to 1. The cost of having this bounded range - in contrast to standardization - is that we will end up with smaller standard deviations, which can suppress the effect of outliers. If you would like to perform MinMax scaling - simply substract minimum value and divide it by range:\((x-min)/(max-min)\)
In order to solve problems with very skewed distributions we can also use several types of simple transformations:
+ "log" : log transformation. log(x)
+ "log+1" : log transformation. log(x + 1). Used for values that contain 0.
+ "sqrt" : square root transformation.
+ "1/x" : 1 / x transformation
+ "x^2" : x square transformation
+ "x^3" : x^3 square transformationStandardization
Use the methods “zscore” and “minmax” to perform standardization.
carseats %>%
mutate(Income_minmax = transform(carseats$Income, method = "minmax"),
Sales_minmax = transform(carseats$Sales, method = "minmax")) %>%
select(Income_minmax, Sales_minmax) %>%
boxplot()
Binning
Sometimes we just would like to perform so called ‘binning’ procedure to be able to analyze our categorical data, to compare several categorical variables, to construct statistical models etc. Thanks to the ‘binning’ function we can transform quantitative variables into categorical using several methods:
binning() transforms a numeric variable into a
categorical variable by binning it. The following types of binning are
supported.
- “quantile” : categorize using quantile to include the same frequencies
- “equal” : categorize to have equal length segments
- “pretty” : categorized into moderately good segments
- “kmeans” : categorization using K-means clustering
- “bclust” : categorization using bagged clustering technique
Here are some examples of how to bin Income using
binning().:
Example:
# Binning the carat variable. default type argument is "quantile"
bin <- binning(carseats$Income)
# Print bins class object
bin## binned type: quantile
## number of bins: 10
## x
## [21,30] (30,39] (39,48] (48,62] (62,69]
## 40 37 38 40 42
## (69,78] (78,86.56667] (86.56667,96.6] (96.6,108.6333] (108.6333,120]
## 33 36 38 38 38
## <NA>
## 20# Summarize bins class object
summary(bin)## levels freq rate
## 1 [21,30] 40 0.1000
## 2 (30,39] 37 0.0925
## 3 (39,48] 38 0.0950
## 4 (48,62] 40 0.1000
## 5 (62,69] 42 0.1050
## 6 (69,78] 33 0.0825
## 7 (78,86.56667] 36 0.0900
## 8 (86.56667,96.6] 38 0.0950
## 9 (96.6,108.6333] 38 0.0950
## 10 (108.6333,120] 38 0.0950
## 11 <NA> 20 0.0500# Plot bins class object
plot(bin)
Using pipes & dplyr:
carseats %>%
mutate(Income_bin = binning(carseats$Income) %>%
extract()) %>%
group_by(ShelveLoc, Income_bin) %>%
summarise(freq = n()) %>%
arrange(desc(freq)) %>%
head(10)## `summarise()` has grouped output by 'ShelveLoc'. You can override using the
## `.groups` argument.## # A tibble: 10 × 3
## # Groups: ShelveLoc [1]
## ShelveLoc Income_bin freq
## <fct> <ord> <int>
## 1 Medium [21,30] 25
## 2 Medium (62,69] 24
## 3 Medium (48,62] 23
## 4 Medium (39,48] 21
## 5 Medium (30,39] 20
## 6 Medium (86.56667,96.6] 20
## 7 Medium (108.6333,120] 20
## 8 Medium (69,78] 18
## 9 Medium (96.6,108.6333] 18
## 10 Medium (78,86.56667] 17Example: Recode the original distribution of incomes using fixed length of intervals and assign them labels.
income_fixed<- binning(carseats$Income, nbins = 4,
labels = c("low", "average", "high", "very high"))
summary(income_fixed)## levels freq rate
## 1 low 95 0.2375
## 2 average 102 0.2550
## 3 high 89 0.2225
## 4 very high 94 0.2350
## 5 <NA> 20 0.0500plot(income_fixed)
In case of statistical modeling (i.e. credit scoring purposes) - we need to be aware of the fact, that the optimal discretization of the original distribution must be achieved. The ‘binning_by’ function comes with some help here.
Example: Perform discretization of the variable ‘Advertising’ using optimal binning.
bin <- binning_by(carseats, "US", "Advertising")## Warning in binning_by(carseats, "US", "Advertising"): The factor y has been changed to a numeric vector consisting of 0 and 1.
## 'Yes' changed to 1 (positive) and 'No' changed to 0 (negative).summary(bin)## ── Binning Table ──────────────────────── Several Metrics ──
## Bin CntRec CntPos CntNeg RatePos RateNeg Odds WoE IV JSD
## 1 [-1,0] 144 19 125 0.07364 0.88028 0.1520 -2.48101 2.00128 0.20093
## 2 (0,6] 69 54 15 0.20930 0.10563 3.6000 0.68380 0.07089 0.00869
## 3 (6,29] 187 185 2 0.71705 0.01408 92.5000 3.93008 2.76272 0.21861
## 4 Total 400 258 142 1.00000 1.00000 1.8169 NA 4.83489 0.42823
## AUC
## 1 0.03241
## 2 0.01883
## 3 0.00903
## 4 0.06028
##
## ── General Metrics ─────────────────────────────────────────
## • Gini index : -0.87944
## • IV (Jeffrey) : 4.83489
## • JS (Jensen-Shannon) Divergence : 0.42823
## • Kolmogorov-Smirnov Statistics : 0.80664
## • HHI (Herfindahl-Hirschman Index) : 0.37791
## • HHI (normalized) : 0.06687
## • Cramer's V : 0.81863
##
## ── Significance Tests ──────────────────── Chisquare Test ──
## Bin A Bin B statistics p_value
## 1 [-1,0] (0,6] 87.67064 7.731349e-21
## 2 (0,6] (6,29] 34.73349 3.780706e-09plot(bin)
Please take a look once again at the final report of our binning procedure. Try to interpret information criterion (Jeffrey’s or K-S) and plots.
We can finally print some summary reports after the cleansing procedures are all done - in the PDF format (TEX installation is required first) or HTML format. Please use this chunk in your own RStudio in order to produce some reports.
carseats %>%
transformation_report(target = US, output_format = "html",
output_file = "transformation_carseats.html")Exercise 5.
Use quantile approach to perform binning of the variable ‘Income’ (4 equal categories).
carseats %>%
mutate(Income_Category = ntile(Income, 4))## Sales CompPrice Income Advertising Population Price ShelveLoc Age Education
## 1 9.50 138 73 11 276 120 Bad 42 17
## 2 11.22 111 48 16 260 83 Good 65 10
## 3 10.06 113 35 10 269 80 Medium 59 12
## 4 7.40 117 100 4 466 97 Medium 55 14
## 5 4.15 141 64 3 340 128 Bad 38 13
## 6 10.81 124 113 13 501 72 Bad 78 16
## 7 6.63 115 105 0 45 108 Medium 71 15
## 8 11.85 136 81 15 425 120 Good 67 10
## 9 6.54 132 110 0 108 124 Medium 76 10
## 10 4.69 132 113 0 131 124 Medium 76 17
## 11 9.01 121 78 9 150 100 Bad 26 10
## 12 11.96 117 94 4 503 94 Good 50 13
## 13 3.98 122 35 2 393 136 Medium 62 18
## 14 10.96 115 28 11 29 86 Good 53 18
## 15 11.17 107 117 11 148 118 Good 52 18
## 16 8.71 149 95 5 400 144 Medium 76 18
## 17 7.58 118 NA 0 284 110 Good 63 13
## 18 12.29 147 NA 13 251 131 Good 52 10
## 19 13.91 110 110 0 408 68 Good 46 17
## 20 8.73 129 76 16 58 121 Medium 69 12
## 21 6.41 125 90 2 367 131 Medium 35 18
## 22 12.13 134 29 12 239 109 Good 62 18
## 23 5.08 128 46 6 497 138 Medium 42 13
## 24 5.87 121 31 0 292 109 Medium 79 10
## 25 10.14 145 119 16 294 113 Bad 42 12
## 26 14.90 139 32 0 176 82 Good 54 11
## 27 8.33 107 115 11 496 131 Good 50 11
## 28 5.27 98 118 0 19 107 Medium 64 17
## 29 2.99 103 74 0 359 97 Bad 55 11
## 30 7.81 104 99 15 226 102 Bad 58 17
## 31 13.55 125 94 0 447 89 Good 30 12
## 32 8.25 136 58 16 241 131 Medium 44 18
## 33 6.20 107 32 12 236 137 Good 64 10
## 34 8.77 114 38 13 317 128 Good 50 16
## 35 2.67 115 54 0 406 128 Medium 42 17
## 36 11.07 131 84 11 29 96 Medium 44 17
## 37 8.89 122 76 0 270 100 Good 60 18
## 38 4.95 121 41 5 412 110 Medium 54 10
## 39 6.59 109 73 0 454 102 Medium 65 15
## 40 3.24 130 NA 0 144 138 Bad 38 10
## 41 2.07 119 98 0 18 126 Bad 73 17
## 42 7.96 157 53 0 403 124 Bad 58 16
## 43 10.43 77 69 0 25 24 Medium 50 18
## 44 4.12 123 42 11 16 134 Medium 59 13
## 45 4.16 85 79 6 325 95 Medium 69 13
## 46 4.56 141 63 0 168 135 Bad 44 12
## 47 12.44 127 90 14 16 70 Medium 48 15
## 48 4.38 126 98 0 173 108 Bad 55 16
## 49 3.91 116 52 0 349 98 Bad 69 18
## 50 10.61 157 93 0 51 149 Good 32 17
## 51 1.42 99 32 18 341 108 Bad 80 16
## 52 4.42 121 90 0 150 108 Bad 75 16
## 53 7.91 153 40 3 112 129 Bad 39 18
## 54 6.92 109 64 13 39 119 Medium 61 17
## 55 4.90 134 103 13 25 144 Medium 76 17
## 56 6.85 143 81 5 60 154 Medium 61 18
## 57 11.91 133 82 0 54 84 Medium 50 17
## 58 0.91 93 91 0 22 117 Bad 75 11
## 59 5.42 103 93 15 188 103 Bad 74 16
## 60 5.21 118 71 4 148 114 Medium 80 13
## 61 8.32 122 102 19 469 123 Bad 29 13
## 62 7.32 105 32 0 358 107 Medium 26 13
## 63 1.82 139 45 0 146 133 Bad 77 17
## 64 8.47 119 88 10 170 101 Medium 61 13
## 65 7.80 100 67 12 184 104 Medium 32 16
## 66 4.90 122 26 0 197 128 Medium 55 13
## 67 8.85 127 92 0 508 91 Medium 56 18
## 68 9.01 126 61 14 152 115 Medium 47 16
## 69 13.39 149 69 20 366 134 Good 60 13
## 70 7.99 127 59 0 339 99 Medium 65 12
## 71 9.46 89 81 15 237 99 Good 74 12
## 72 6.50 148 51 16 148 150 Medium 58 17
## 73 5.52 115 45 0 432 116 Medium 25 15
## 74 12.61 118 90 10 54 104 Good 31 11
## 75 6.20 150 68 5 125 136 Medium 64 13
## 76 8.55 88 111 23 480 92 Bad 36 16
## 77 10.64 102 87 10 346 70 Medium 64 15
## 78 7.70 118 71 12 44 89 Medium 67 18
## 79 4.43 134 48 1 139 145 Medium 65 12
## 80 9.14 134 67 0 286 90 Bad 41 13
## 81 8.01 113 100 16 353 79 Bad 68 11
## 82 7.52 116 72 0 237 128 Good 70 13
## 83 11.62 151 83 4 325 139 Good 28 17
## 84 4.42 109 36 7 468 94 Bad 56 11
## 85 2.23 111 25 0 52 121 Bad 43 18
## 86 8.47 125 103 0 304 112 Medium 49 13
## 87 8.70 150 84 9 432 134 Medium 64 15
## 88 11.70 131 67 7 272 126 Good 54 16
## 89 6.56 117 42 7 144 111 Medium 62 10
## 90 7.95 128 66 3 493 119 Medium 45 16
## 91 5.33 115 22 0 491 103 Medium 64 11
## 92 4.81 97 46 11 267 107 Medium 80 15
## 93 4.53 114 113 0 97 125 Medium 29 12
## 94 8.86 145 30 0 67 104 Medium 55 17
## 95 8.39 115 NA 5 134 84 Bad 55 11
## 96 5.58 134 25 10 237 148 Medium 59 13
## 97 9.48 147 42 10 407 132 Good 73 16
## 98 7.45 161 82 5 287 129 Bad 33 16
## 99 12.49 122 77 24 382 127 Good 36 16
## 100 4.88 121 47 3 220 107 Bad 56 16
## 101 4.11 113 69 11 94 106 Medium 76 12
## 102 6.20 128 93 0 89 118 Medium 34 18
## 103 5.30 113 22 0 57 97 Medium 65 16
## 104 5.07 123 91 0 334 96 Bad 78 17
## 105 4.62 121 96 0 472 138 Medium 51 12
## 106 5.55 104 100 8 398 97 Medium 61 11
## 107 0.16 102 33 0 217 139 Medium 70 18
## 108 8.55 134 107 0 104 108 Medium 60 12
## 109 3.47 107 79 2 488 103 Bad 65 16
## 110 8.98 115 65 0 217 90 Medium 60 17
## 111 9.00 128 62 7 125 116 Medium 43 14
## 112 6.62 132 118 12 272 151 Medium 43 14
## 113 6.67 116 99 5 298 125 Good 62 12
## 114 6.01 131 29 11 335 127 Bad 33 12
## 115 9.31 122 87 9 17 106 Medium 65 13
## 116 8.54 139 NA 0 95 129 Medium 42 13
## 117 5.08 135 75 0 202 128 Medium 80 10
## 118 8.80 145 53 0 507 119 Medium 41 12
## 119 7.57 112 88 2 243 99 Medium 62 11
## 120 7.37 130 94 8 137 128 Medium 64 12
## 121 6.87 128 105 11 249 131 Medium 63 13
## 122 11.67 125 89 10 380 87 Bad 28 10
## 123 6.88 119 100 5 45 108 Medium 75 10
## 124 8.19 127 103 0 125 155 Good 29 15
## 125 8.87 131 113 0 181 120 Good 63 14
## 126 9.34 89 NA 0 181 49 Medium 43 15
## 127 11.27 153 68 2 60 133 Good 59 16
## 128 6.52 125 48 3 192 116 Medium 51 14
## 129 4.96 133 100 3 350 126 Bad 55 13
## 130 4.47 143 120 7 279 147 Bad 40 10
## 131 8.41 94 84 13 497 77 Medium 51 12
## 132 6.50 108 69 3 208 94 Medium 77 16
## 133 9.54 125 87 9 232 136 Good 72 10
## 134 7.62 132 98 2 265 97 Bad 62 12
## 135 3.67 132 31 0 327 131 Medium 76 16
## 136 6.44 96 94 14 384 120 Medium 36 18
## 137 5.17 131 75 0 10 120 Bad 31 18
## 138 6.52 128 42 0 436 118 Medium 80 11
## 139 10.27 125 103 12 371 109 Medium 44 10
## 140 12.30 146 62 10 310 94 Medium 30 13
## 141 6.03 133 60 10 277 129 Medium 45 18
## 142 6.53 140 42 0 331 131 Bad 28 15
## 143 7.44 124 84 0 300 104 Medium 77 15
## 144 0.53 122 88 7 36 159 Bad 28 17
## 145 9.09 132 68 0 264 123 Good 34 11
## 146 8.77 144 63 11 27 117 Medium 47 17
## 147 3.90 114 83 0 412 131 Bad 39 14
## 148 10.51 140 54 9 402 119 Good 41 16
## 149 7.56 110 119 0 384 97 Medium 72 14
## 150 11.48 121 120 13 140 87 Medium 56 11
## 151 10.49 122 84 8 176 114 Good 57 10
## 152 10.77 111 58 17 407 103 Good 75 17
## 153 7.64 128 78 0 341 128 Good 45 13
## 154 5.93 150 36 7 488 150 Medium 25 17
## 155 6.89 129 69 10 289 110 Medium 50 16
## 156 7.71 98 72 0 59 69 Medium 65 16
## 157 7.49 146 34 0 220 157 Good 51 16
## 158 10.21 121 58 8 249 90 Medium 48 13
## 159 12.53 142 90 1 189 112 Good 39 10
## 160 9.32 119 60 0 372 70 Bad 30 18
## 161 4.67 111 28 0 486 111 Medium 29 12
## 162 2.93 143 21 5 81 160 Medium 67 12
## 163 3.63 122 NA 0 424 149 Medium 51 13
## 164 5.68 130 64 0 40 106 Bad 39 17
## 165 8.22 148 64 0 58 141 Medium 27 13
## 166 0.37 147 58 7 100 191 Bad 27 15
## 167 6.71 119 67 17 151 137 Medium 55 11
## 168 6.71 106 73 0 216 93 Medium 60 13
## 169 7.30 129 89 0 425 117 Medium 45 10
## 170 11.48 104 41 15 492 77 Good 73 18
## 171 8.01 128 39 12 356 118 Medium 71 10
## 172 12.49 93 106 12 416 55 Medium 75 15
## 173 9.03 104 102 13 123 110 Good 35 16
## 174 6.38 135 91 5 207 128 Medium 66 18
## 175 0.00 139 24 0 358 185 Medium 79 15
## 176 7.54 115 89 0 38 122 Medium 25 12
## 177 5.61 138 NA 9 480 154 Medium 47 11
## 178 10.48 138 72 0 148 94 Medium 27 17
## 179 10.66 104 NA 14 89 81 Medium 25 14
## 180 7.78 144 25 3 70 116 Medium 77 18
## 181 4.94 137 112 15 434 149 Bad 66 13
## 182 7.43 121 83 0 79 91 Medium 68 11
## 183 4.74 137 60 4 230 140 Bad 25 13
## 184 5.32 118 74 6 426 102 Medium 80 18
## 185 9.95 132 33 7 35 97 Medium 60 11
## 186 10.07 130 100 11 449 107 Medium 64 10
## 187 8.68 120 51 0 93 86 Medium 46 17
## 188 6.03 117 32 0 142 96 Bad 62 17
## 189 8.07 116 37 0 426 90 Medium 76 15
## 190 12.11 118 117 18 509 104 Medium 26 15
## 191 8.79 130 37 13 297 101 Medium 37 13
## 192 6.67 156 42 13 170 173 Good 74 14
## 193 7.56 108 26 0 408 93 Medium 56 14
## 194 13.28 139 70 7 71 96 Good 61 10
## 195 7.23 112 98 18 481 128 Medium 45 11
## 196 4.19 117 93 4 420 112 Bad 66 11
## 197 4.10 130 28 6 410 133 Bad 72 16
## 198 2.52 124 61 0 333 138 Medium 76 16
## 199 3.62 112 80 5 500 128 Medium 69 10
## 200 6.42 122 88 5 335 126 Medium 64 14
## 201 5.56 144 92 0 349 146 Medium 62 12
## 202 5.94 138 83 0 139 134 Medium 54 18
## 203 4.10 121 78 4 413 130 Bad 46 10
## 204 2.05 131 82 0 132 157 Bad 25 14
## 205 8.74 155 80 0 237 124 Medium 37 14
## 206 5.68 113 22 1 317 132 Medium 28 12
## 207 4.97 162 67 0 27 160 Medium 77 17
## 208 8.19 111 105 0 466 97 Bad 61 10
## 209 7.78 86 NA 0 497 64 Bad 33 12
## 210 3.02 98 21 11 326 90 Bad 76 11
## 211 4.36 125 41 2 357 123 Bad 47 14
## 212 9.39 117 118 14 445 120 Medium 32 15
## 213 12.04 145 69 19 501 105 Medium 45 11
## 214 8.23 149 84 5 220 139 Medium 33 10
## 215 4.83 115 115 3 48 107 Medium 73 18
## 216 2.34 116 83 15 170 144 Bad 71 11
## 217 5.73 141 NA 0 243 144 Medium 34 17
## 218 4.34 106 44 0 481 111 Medium 70 14
## 219 9.70 138 61 12 156 120 Medium 25 14
## 220 10.62 116 79 19 359 116 Good 58 17
## 221 10.59 131 120 15 262 124 Medium 30 10
## 222 6.43 124 NA 0 125 107 Medium 80 11
## 223 7.49 136 119 6 178 145 Medium 35 13
## 224 3.45 110 45 9 276 125 Medium 62 14
## 225 4.10 134 82 0 464 141 Medium 48 13
## 226 6.68 107 25 0 412 82 Bad 36 14
## 227 7.80 119 33 0 245 122 Good 56 14
## 228 8.69 113 64 10 68 101 Medium 57 16
## 229 5.40 149 73 13 381 163 Bad 26 11
## 230 11.19 98 104 0 404 72 Medium 27 18
## 231 5.16 115 60 0 119 114 Bad 38 14
## 232 8.09 132 69 0 123 122 Medium 27 11
## 233 13.14 137 80 10 24 105 Good 61 15
## 234 8.65 123 76 18 218 120 Medium 29 14
## 235 9.43 115 62 11 289 129 Good 56 16
## 236 5.53 126 32 8 95 132 Medium 50 17
## 237 9.32 141 34 16 361 108 Medium 69 10
## 238 9.62 151 28 8 499 135 Medium 48 10
## 239 7.36 121 24 0 200 133 Good 73 13
## 240 3.89 123 105 0 149 118 Bad 62 16
## 241 10.31 159 80 0 362 121 Medium 26 18
## 242 12.01 136 63 0 160 94 Medium 38 12
## 243 4.68 124 46 0 199 135 Medium 52 14
## 244 7.82 124 25 13 87 110 Medium 57 10
## 245 8.78 130 30 0 391 100 Medium 26 18
## 246 10.00 114 43 0 199 88 Good 57 10
## 247 6.90 120 56 20 266 90 Bad 78 18
## 248 5.04 123 114 0 298 151 Bad 34 16
## 249 5.36 111 52 0 12 101 Medium 61 11
## 250 5.05 125 67 0 86 117 Bad 65 11
## 251 9.16 137 105 10 435 156 Good 72 14
## 252 3.72 139 111 5 310 132 Bad 62 13
## 253 8.31 133 97 0 70 117 Medium 32 16
## 254 5.64 124 24 5 288 122 Medium 57 12
## 255 9.58 108 104 23 353 129 Good 37 17
## 256 7.71 123 81 8 198 81 Bad 80 15
## 257 4.20 147 40 0 277 144 Medium 73 10
## 258 8.67 125 62 14 477 112 Medium 80 13
## 259 3.47 108 38 0 251 81 Bad 72 14
## 260 5.12 123 36 10 467 100 Bad 74 11
## 261 7.67 129 117 8 400 101 Bad 36 10
## 262 5.71 121 42 4 188 118 Medium 54 15
## 263 6.37 120 NA 15 86 132 Medium 48 18
## 264 7.77 116 26 6 434 115 Medium 25 17
## 265 6.95 128 29 5 324 159 Good 31 15
## 266 5.31 130 35 10 402 129 Bad 39 17
## 267 9.10 128 93 12 343 112 Good 73 17
## 268 5.83 134 82 7 473 112 Bad 51 12
## 269 6.53 123 57 0 66 105 Medium 39 11
## 270 5.01 159 69 0 438 166 Medium 46 17
## 271 11.99 119 26 0 284 89 Good 26 10
## 272 4.55 111 56 0 504 110 Medium 62 16
## 273 12.98 113 33 0 14 63 Good 38 12
## 274 10.04 116 106 8 244 86 Medium 58 12
## 275 7.22 135 93 2 67 119 Medium 34 11
## 276 6.67 107 119 11 210 132 Medium 53 11
## 277 6.93 135 69 14 296 130 Medium 73 15
## 278 7.80 136 48 12 326 125 Medium 36 16
## 279 7.22 114 113 2 129 151 Good 40 15
## 280 3.42 141 57 13 376 158 Medium 64 18
## 281 2.86 121 86 10 496 145 Bad 51 10
## 282 11.19 122 69 7 303 105 Good 45 16
## 283 7.74 150 96 0 80 154 Good 61 11
## 284 5.36 135 110 0 112 117 Medium 80 16
## 285 6.97 106 46 11 414 96 Bad 79 17
## 286 7.60 146 26 11 261 131 Medium 39 10
## 287 7.53 117 118 11 429 113 Medium 67 18
## 288 6.88 95 44 4 208 72 Bad 44 17
## 289 6.98 116 40 0 74 97 Medium 76 15
## 290 8.75 143 77 25 448 156 Medium 43 17
## 291 9.49 107 111 14 400 103 Medium 41 11
## 292 6.64 118 70 0 106 89 Bad 39 17
## 293 11.82 113 66 16 322 74 Good 76 15
## 294 11.28 123 84 0 74 89 Good 59 10
## 295 12.66 148 76 3 126 99 Good 60 11
## 296 4.21 118 35 14 502 137 Medium 79 10
## 297 8.21 127 44 13 160 123 Good 63 18
## 298 3.07 118 83 13 276 104 Bad 75 10
## 299 10.98 148 63 0 312 130 Good 63 15
## 300 9.40 135 40 17 497 96 Medium 54 17
## 301 8.57 116 78 1 158 99 Medium 45 11
## 302 7.41 99 93 0 198 87 Medium 57 16
## 303 5.28 108 77 13 388 110 Bad 74 14
## 304 10.01 133 52 16 290 99 Medium 43 11
## 305 11.93 123 98 12 408 134 Good 29 10
## 306 8.03 115 29 26 394 132 Medium 33 13
## 307 4.78 131 32 1 85 133 Medium 48 12
## 308 5.90 138 92 0 13 120 Bad 61 12
## 309 9.24 126 80 19 436 126 Medium 52 10
## 310 11.18 131 111 13 33 80 Bad 68 18
## 311 9.53 175 65 29 419 166 Medium 53 12
## 312 6.15 146 68 12 328 132 Bad 51 14
## 313 6.80 137 117 5 337 135 Bad 38 10
## 314 9.33 103 81 3 491 54 Medium 66 13
## 315 7.72 133 NA 10 333 129 Good 71 14
## 316 6.39 131 21 8 220 171 Good 29 14
## 317 15.63 122 36 5 369 72 Good 35 10
## 318 6.41 142 30 0 472 136 Good 80 15
## 319 10.08 116 72 10 456 130 Good 41 14
## 320 6.97 127 45 19 459 129 Medium 57 11
## 321 5.86 136 70 12 171 152 Medium 44 18
## 322 7.52 123 39 5 499 98 Medium 34 15
## 323 9.16 140 50 10 300 139 Good 60 15
## 324 10.36 107 105 18 428 103 Medium 34 12
## 325 2.66 136 65 4 133 150 Bad 53 13
## 326 11.70 144 69 11 131 104 Medium 47 11
## 327 4.69 133 30 0 152 122 Medium 53 17
## 328 6.23 112 38 17 316 104 Medium 80 16
## 329 3.15 117 66 1 65 111 Bad 55 11
## 330 11.27 100 54 9 433 89 Good 45 12
## 331 4.99 122 59 0 501 112 Bad 32 14
## 332 10.10 135 63 15 213 134 Medium 32 10
## 333 5.74 106 33 20 354 104 Medium 61 12
## 334 5.87 136 60 7 303 147 Medium 41 10
## 335 7.63 93 117 9 489 83 Bad 42 13
## 336 6.18 120 70 15 464 110 Medium 72 15
## 337 5.17 138 35 6 60 143 Bad 28 18
## 338 8.61 130 38 0 283 102 Medium 80 15
## 339 5.97 112 24 0 164 101 Medium 45 11
## 340 11.54 134 44 4 219 126 Good 44 15
## 341 7.50 140 29 0 105 91 Bad 43 16
## 342 7.38 98 120 0 268 93 Medium 72 10
## 343 7.81 137 102 13 422 118 Medium 71 10
## 344 5.99 117 42 10 371 121 Bad 26 14
## 345 8.43 138 80 0 108 126 Good 70 13
## 346 4.81 121 68 0 279 149 Good 79 12
## 347 8.97 132 NA 0 144 125 Medium 33 13
## 348 6.88 96 39 0 161 112 Good 27 14
## 349 12.57 132 102 20 459 107 Good 49 11
## 350 9.32 134 27 18 467 96 Medium 49 14
## 351 8.64 111 NA 17 266 91 Medium 63 17
## 352 10.44 124 115 16 458 105 Medium 62 16
## 353 13.44 133 103 14 288 122 Good 61 17
## 354 9.45 107 67 12 430 92 Medium 35 12
## 355 5.30 133 31 1 80 145 Medium 42 18
## 356 7.02 130 100 0 306 146 Good 42 11
## 357 3.58 142 109 0 111 164 Good 72 12
## 358 13.36 103 73 3 276 72 Medium 34 15
## 359 4.17 123 96 10 71 118 Bad 69 11
## 360 3.13 130 62 11 396 130 Bad 66 14
## 361 8.77 118 86 7 265 114 Good 52 15
## 362 8.68 131 25 10 183 104 Medium 56 15
## 363 5.25 131 55 0 26 110 Bad 79 12
## 364 10.26 111 NA 1 377 108 Good 25 12
## 365 10.50 122 21 16 488 131 Good 30 14
## 366 6.53 154 30 0 122 162 Medium 57 17
## 367 5.98 124 56 11 447 134 Medium 53 12
## 368 14.37 95 106 0 256 53 Good 52 17
## 369 10.71 109 22 10 348 79 Good 74 14
## 370 10.26 135 100 22 463 122 Medium 36 14
## 371 7.68 126 41 22 403 119 Bad 42 12
## 372 9.08 152 81 0 191 126 Medium 54 16
## 373 7.80 121 NA 0 508 98 Medium 65 11
## 374 5.58 137 NA 0 402 116 Medium 78 17
## 375 9.44 131 47 7 90 118 Medium 47 12
## 376 7.90 132 46 4 206 124 Medium 73 11
## 377 16.27 141 60 19 319 92 Good 44 11
## 378 6.81 132 61 0 263 125 Medium 41 12
## 379 6.11 133 88 3 105 119 Medium 79 12
## 380 5.81 125 111 0 404 107 Bad 54 15
## 381 9.64 106 64 10 17 89 Medium 68 17
## 382 3.90 124 65 21 496 151 Bad 77 13
## 383 4.95 121 28 19 315 121 Medium 66 14
## 384 9.35 98 117 0 76 68 Medium 63 10
## 385 12.85 123 37 15 348 112 Good 28 12
## 386 5.87 131 73 13 455 132 Medium 62 17
## 387 5.32 152 116 0 170 160 Medium 39 16
## 388 8.67 142 73 14 238 115 Medium 73 14
## 389 8.14 135 89 11 245 78 Bad 79 16
## 390 8.44 128 42 8 328 107 Medium 35 12
## 391 5.47 108 75 9 61 111 Medium 67 12
## 392 6.10 153 63 0 49 124 Bad 56 16
## 393 4.53 129 42 13 315 130 Bad 34 13
## 394 5.57 109 51 10 26 120 Medium 30 17
## 395 5.35 130 58 19 366 139 Bad 33 16
## 396 12.57 138 108 17 203 128 Good 33 14
## 397 6.14 139 NA 3 37 120 Medium 55 11
## 398 7.41 162 26 12 368 159 Medium 40 18
## 399 5.94 100 79 7 284 95 Bad 50 12
## 400 9.71 134 37 0 27 120 Good 49 16
## Urban US Income_Category
## 1 Yes Yes 3
## 2 Yes Yes 2
## 3 Yes Yes 1
## 4 Yes Yes 4
## 5 Yes No 2
## 6 No Yes 4
## 7 Yes No 4
## 8 Yes Yes 3
## 9 No No 4
## 10 No Yes 4
## 11 No Yes 3
## 12 Yes Yes 4
## 13 Yes No 1
## 14 Yes Yes 1
## 15 Yes Yes 4
## 16 No No 4
## 17 Yes No NA
## 18 Yes Yes NA
## 19 No Yes 4
## 20 Yes Yes 3
## 21 Yes Yes 3
## 22 No Yes 1
## 23 Yes No 2
## 24 Yes No 1
## 25 Yes Yes 4
## 26 No No 1
## 27 No Yes 4
## 28 Yes No 4
## 29 Yes Yes 3
## 30 Yes Yes 4
## 31 Yes No 4
## 32 Yes Yes 2
## 33 <NA> Yes 1
## 34 Yes Yes 1
## 35 Yes Yes 2
## 36 <NA> Yes 3
## 37 No No 3
## 38 Yes Yes 1
## 39 Yes No 3
## 40 No No NA
## 41 No No 4
## 42 Yes No 2
## 43 Yes No 2
## 44 Yes Yes 1
## 45 Yes Yes 3
## 46 Yes Yes 2
## 47 No Yes 3
## 48 Yes No 4
## 49 Yes No 2
## 50 Yes No 4
## 51 Yes Yes 1
## 52 Yes No 3
## 53 Yes Yes 1
## 54 Yes Yes 2
## 55 No Yes 4
## 56 Yes Yes 3
## 57 Yes No 3
## 58 Yes No 3
## 59 Yes Yes 4
## 60 Yes No 3
## 61 Yes Yes 4
## 62 No No 1
## 63 Yes Yes 2
## 64 Yes Yes 3
## 65 No Yes 2
## 66 No No 1
## 67 Yes No 4
## 68 Yes Yes 2
## 69 Yes Yes 2
## 70 Yes No 2
## 71 Yes Yes 3
## 72 No Yes 2
## 73 Yes No 2
## 74 No Yes 3
## 75 No Yes 2
## 76 No Yes 4
## 77 Yes Yes 3
## 78 No Yes 3
## 79 Yes Yes 2
## 80 Yes No 2
## 81 Yes Yes 4
## 82 Yes No 3
## 83 Yes Yes 3
## 84 <NA> Yes 1
## 85 No No 1
## 86 No No 4
## 87 Yes No 3
## 88 No Yes 2
## 89 Yes Yes 1
## 90 No No 2
## 91 No No 1
## 92 Yes Yes 2
## 93 Yes No 4
## 94 <NA> No 1
## 95 Yes Yes NA
## 96 Yes Yes 1
## 97 No Yes 1
## 98 Yes Yes 3
## 99 No Yes 3
## 100 No Yes 2
## 101 No Yes 2
## 102 Yes No 4
## 103 No No 1
## 104 Yes Yes 3
## 105 Yes No 4
## 106 Yes Yes 4
## 107 No No 1
## 108 Yes No 4
## 109 Yes No 3
## 110 No No 2
## 111 Yes Yes 2
## 112 Yes Yes 4
## 113 <NA> Yes 4
## 114 Yes Yes 1
## 115 Yes Yes 3
## 116 Yes No NA
## 117 No No 3
## 118 Yes No 2
## 119 Yes Yes 3
## 120 Yes Yes 4
## 121 Yes Yes 4
## 122 Yes Yes 3
## 123 Yes Yes 4
## 124 No Yes 4
## 125 Yes No 4
## 126 No No NA
## 127 Yes Yes 2
## 128 Yes Yes 2
## 129 Yes Yes 4
## 130 No Yes 4
## 131 Yes Yes 3
## 132 <NA> No 2
## 133 Yes Yes 3
## 134 Yes Yes 4
## 135 Yes No 1
## 136 No Yes 4
## 137 No No 3
## 138 Yes No 1
## 139 Yes Yes 4
## 140 No Yes 2
## 141 Yes Yes 2
## 142 Yes No 1
## 143 Yes No 3
## 144 Yes Yes 3
## 145 No No 2
## 146 Yes Yes 2
## 147 Yes No 3
## 148 No Yes 2
## 149 No Yes 4
## 150 Yes Yes 4
## 151 <NA> Yes 3
## 152 No Yes 2
## 153 No No 3
## 154 No Yes 1
## 155 No Yes 3
## 156 Yes No 3
## 157 Yes No 1
## 158 No Yes 2
## 159 No Yes 3
## 160 No No 2
## 161 No No 1
## 162 No Yes 1
## 163 Yes No NA
## 164 No No 2
## 165 No Yes 2
## 166 Yes Yes 2
## 167 Yes Yes 2
## 168 Yes No 3
## 169 Yes No 3
## 170 Yes Yes 1
## 171 Yes Yes 1
## 172 Yes Yes 4
## 173 Yes Yes 4
## 174 Yes Yes 4
## 175 No No 1
## 176 Yes No 3
## 177 No Yes NA
## 178 Yes Yes 3
## 179 No Yes NA
## 180 Yes Yes 1
## 181 Yes Yes 4
## 182 Yes No 3
## 183 Yes No 2
## 184 Yes Yes 3
## 185 No Yes 1
## 186 Yes Yes 4
## 187 No No 2
## 188 Yes No 1
## 189 Yes No 1
## 190 No Yes 4
## 191 No Yes 1
## 192 Yes Yes 1
## 193 No No 1
## 194 Yes Yes 3
## 195 Yes Yes 4
## 196 Yes Yes 4
## 197 Yes Yes 1
## 198 Yes No 2
## 199 Yes Yes 3
## 200 Yes Yes 3
## 201 No No 4
## 202 Yes No 3
## 203 No Yes 3
## 204 Yes No 3
## 205 Yes No 3
## 206 Yes No 1
## 207 Yes Yes 2
## 208 No No 4
## 209 Yes No NA
## 210 No Yes 1
## 211 No Yes 1
## 212 Yes Yes 4
## 213 Yes Yes 3
## 214 Yes Yes 3
## 215 Yes Yes 4
## 216 Yes Yes 3
## 217 Yes No NA
## 218 No No 2
## 219 Yes Yes 2
## 220 Yes Yes 3
## 221 Yes Yes 4
## 222 Yes No NA
## 223 Yes Yes 4
## 224 Yes Yes 2
## 225 No No 3
## 226 Yes No 1
## 227 Yes No 1
## 228 Yes Yes 2
## 229 No Yes 3
## 230 No No 4
## 231 No No 2
## 232 No No 3
## 233 Yes Yes 3
## 234 No Yes 3
## 235 No Yes 2
## 236 Yes Yes 1
## 237 Yes Yes 1
## 238 Yes Yes 1
## 239 Yes No 1
## 240 Yes Yes 4
## 241 Yes No 3
## 242 Yes No 2
## 243 No No 2
## 244 Yes Yes 1
## 245 Yes No 1
## 246 No Yes 2
## 247 Yes Yes 2
## 248 Yes No 4
## 249 Yes Yes 2
## 250 Yes No 2
## 251 Yes Yes 4
## 252 Yes Yes 4
## 253 Yes No 4
## 254 No Yes 1
## 255 Yes Yes 4
## 256 Yes Yes 3
## 257 Yes No 1
## 258 Yes Yes 2
## 259 No No 1
## 260 No Yes 1
## 261 Yes Yes 4
## 262 Yes Yes 1
## 263 Yes Yes NA
## 264 Yes Yes 1
## 265 Yes Yes 1
## 266 Yes Yes 1
## 267 No Yes 4
## 268 No Yes 3
## 269 Yes No 2
## 270 Yes No 3
## 271 Yes No 1
## 272 Yes No 2
## 273 Yes No 1
## 274 Yes Yes 4
## 275 Yes Yes 4
## 276 Yes Yes 4
## 277 Yes Yes 3
## 278 Yes Yes 2
## 279 No Yes 4
## 280 Yes Yes 2
## 281 Yes Yes 3
## 282 No Yes 3
## 283 Yes No 4
## 284 No No 4
## 285 No No 2
## 286 Yes Yes 1
## 287 No Yes 4
## 288 Yes Yes 2
## 289 No No 1
## 290 Yes Yes 3
## 291 No Yes 4
## 292 <NA> No 3
## 293 Yes Yes 2
## 294 Yes No 3
## 295 Yes Yes 3
## 296 No Yes 1
## 297 Yes Yes 2
## 298 Yes Yes 3
## 299 Yes No 2
## 300 No Yes 1
## 301 Yes Yes 3
## 302 Yes Yes 4
## 303 Yes Yes 3
## 304 Yes Yes 2
## 305 Yes Yes 4
## 306 Yes Yes 1
## 307 Yes Yes 1
## 308 Yes No 4
## 309 Yes Yes 3
## 310 Yes Yes 4
## 311 Yes Yes 2
## 312 Yes Yes 2
## 313 <NA> Yes 4
## 314 Yes No 3
## 315 Yes Yes NA
## 316 Yes Yes 1
## 317 Yes Yes 1
## 318 No No 1
## 319 No Yes 3
## 320 No Yes 2
## 321 Yes Yes 3
## 322 Yes No 1
## 323 Yes Yes 2
## 324 Yes Yes 4
## 325 Yes Yes 2
## 326 Yes Yes 3
## 327 Yes No 1
## 328 Yes Yes 1
## 329 Yes Yes 2
## 330 Yes Yes 2
## 331 No No 2
## 332 Yes Yes 2
## 333 Yes Yes 1
## 334 Yes Yes 2
## 335 Yes Yes 4
## 336 Yes Yes 3
## 337 Yes No 1
## 338 Yes No 1
## 339 <NA> No 1
## 340 Yes Yes 2
## 341 Yes No 1
## 342 No No 4
## 343 No Yes 4
## 344 Yes Yes 1
## 345 No Yes 3
## 346 Yes No 2
## 347 No No NA
## 348 No No 1
## 349 Yes Yes 4
## 350 No Yes 1
## 351 No Yes NA
## 352 No Yes 4
## 353 Yes Yes 4
## 354 No Yes 2
## 355 Yes Yes 1
## 356 Yes No 4
## 357 Yes No 4
## 358 Yes Yes 3
## 359 Yes Yes 4
## 360 Yes Yes 2
## 361 No Yes 3
## 362 No Yes 1
## 363 Yes Yes 2
## 364 Yes No NA
## 365 Yes Yes 1
## 366 No No 1
## 367 No Yes 2
## 368 Yes No 4
## 369 No Yes 1
## 370 Yes Yes 4
## 371 Yes Yes 1
## 372 Yes No 3
## 373 No No NA
## 374 Yes No NA
## 375 Yes Yes 2
## 376 Yes No 2
## 377 Yes Yes 2
## 378 No No 2
## 379 Yes Yes 3
## 380 Yes No 4
## 381 Yes Yes 2
## 382 Yes Yes 2
## 383 Yes Yes 1
## 384 Yes No 4
## 385 Yes Yes 1
## 386 Yes Yes 3
## 387 Yes No 4
## 388 No Yes 3
## 389 Yes Yes 3
## 390 Yes Yes 1
## 391 Yes Yes 3
## 392 Yes No 2
## 393 Yes Yes 1
## 394 No Yes 2
## 395 Yes Yes 2
## 396 Yes Yes 4
## 397 No Yes NA
## 398 Yes Yes 1
## 399 Yes Yes 3
## 400 Yes Yes 1Automated report
dlookr provides two automated data transformation reports:
- Web page-based dynamic reports can perform in-depth analysis through visualization and statistical tables.
- Static reports generated as pdf files or html files can be archived as output of data analysis.
Dynamic report
transformation_web_report() creates dynamic report for
object inherited from data.frame(tbl_df, tbl,
etc) or data.frame.
The contents of the report are as follows.:
- Overview
- Data Structures
- Data Types
- Job Informations
- Imputation
- Missing Values
- Outliers
- Resolving Skewness
- Binning
- Optimal Binning
The following script creates a data transformation report for the
tbl_df class object, heartfailure.
heartfailure %>% transformation_web_report(target = “death_event”, subtitle = “heartfailure”,output_dir = “./”, output_file = “transformation.html”, theme = “blue”)
- The dynamic contents of the report is shown in the following figure.:
Static report
transformation_paged_report() create static report for
object inherited from data.frame(tbl_df, tbl,
etc) or data.frame.
The contents of the report are as follows.:
- Overview
- Data Structures
- Job Informations
- Imputation
- Missing Values
- Outliers
- Resolving Skewness
- Binning
- Optimal Binning
The following script creates a data transformation report for the
data.frame class object, heartfailure.
heartfailure %>% transformation_paged_report(target = “death_event”, subtitle = “heartfailure”, output_dir = “./”, output_file = “transformation.pdf”, theme = “blue”)
- The cover of the report is shown in the following figure.:
Diagnostic report
diagnose_paged_report() creates static report for object inherited from data.frame(tbl_df, tbl, etc) or data.frame.
The following script creates a quality diagnosis report for the tbl_df class object, flights.
flights %>% diagnose_paged_report(subtitle = “flights”, output_dir = “./”, output_file = “Diagn.pdf”, theme = “blue”)
The cover of the report is shown in the following figure.:
EDA Report
dlookr provides two automated EDA reports:
Web page-based dynamic reports can perform in-depth analysis through visualization and statistical tables.
Static reports generated as pdf files or html files can be archived as output of data analysis.
eda_web_report() creates dynamic report for object inherited from data.frame(tbl_df, tbl, etc) or data.frame.
The following script creates a EDA report for the data.frame class object, heartfailure:
heartfailure %>% eda_web_report(target = “death_event”, subtitle = “heartfailure”, output_dir = “./”, output_file = “EDA.html”, theme = “blue”)
The dynamic contents of the report is shown in the following figure.:
Transformation Report
dlookr provides two automated data transformation reports:
- Web page-based dynamic reports can perform in-depth analysis through visualization and statistical tables.
- Static reports generated as pdf files or html files can be archived as output of data analysis.
transformation_web_report() creates dynamic report for object inherited from data.frame(tbl_df, tbl, etc) or data.frame.
The following script creates a data transformation report for the tbl_df class object, heartfailure:
heartfailure %>% transformation_web_report(target = “death_event”, subtitle = “heartfailure”, output_dir = “./”, output_file = “transformation.html”, theme = “blue”)
The dynamic contents of the report is shown in the following figure.:
Summary
If we are to take away one thing from this TIDY WEEK it should be that dealing with missing data is not an easy problem. Data will rarely be missing in a nice (completely random) way, so if we want to resort to more simple removal or imputation techniques, we must put reasonable effort into determining whether or not the dependencies between observed data and missingness are a cause for concern. If strong dependencies exist and/or there is a lot of data missing, the missing data problem becomes a prediction problem (or a series of prediction problems). We should also be aware of the possibility that missingness depends on the missing value in a way that can’t be explained by observed variables. This can also cause bias in our analyses and we will not be able to detect it unless we get our hands on some of the missing values.
Note that we focused only on standard cross-sectional data. There are many other types of data, such as time-series data, spatial data, images, sound, graphs, etc. The basic principles remain unchanged. We must be aware of the missingness mechanism and introducing bias. We can deal with missing values either by removing observations/variables or by imputing them. However, different, sometimes additional models and techniques will be appropriate. For example, temporal and spatial data lend themselves to interpolation of missing values.
Further reading
A gentle introduction from a practitioners perspective: Blankers, M., Koeter, M. W., & Schippers, G. M. (2010). Missing data approaches in eHealth research: simulation study and a tutorial for nonmathematically inclined researchers. Journal of medical Internet research, 12(5), e54.
A great book on basic and some advance techniques: Allison, P. D. (2001). Missing data (Vol. 136). Sage publications.
Another great book with many examples and case-studies: Van Buuren, S. (2018). Flexible imputation of missing data. Chapman and Hall/CRC.
Understanding multiple imputation with chained equations (in R): Buuren, S. V., & Groothuis-Oudshoorn, K. (2010). mice: Multivariate imputation by chained equations in R. Journal of statistical software, 1-68.
When doing missing data value handling and prediction separately, we should be aware that certain types of prediction models might work better with certain methods for handling missing values. A paper that illustrate this: Yadav, M. L., & Roychoudhury, B. (2018). Handling missing values: A study of popular imputation packages in R. Knowledge-Based Systems, 160, 104-118.
Learning outcomes
Data science students should work towards obtaining the knowledge and the skills that enable them to:
- Reproduce the techniques demonstrated in this chapter using their language/tool of choice.
- Analyze the severity of the missing data problem.
- Recognize when a technique is appropriate and what are its limitations.
LS0tDQp0aXRsZTogIlRpZHkgV2VlayAyOiBEYXRhIENsZWFuc2luZyINCmF1dGhvcjogIlNvZmlhIEFuaW1hdG8sIE5lbCBNYXp1ciINCm91dHB1dDoNCiAgaHRtbF9kb2N1bWVudDogDQogICAgdGhlbWU6IGNlcnVsZWFuDQogICAgaGlnaGxpZ2h0OiB0ZXh0bWF0ZQ0KICAgIGZvbnRzaXplOiA4cHQNCiAgICB0b2M6IHllcw0KICAgIGNvZGVfZG93bmxvYWQ6IHllcw0KICAgIHRvY19mbG9hdDoNCiAgICAgIGNvbGxhcHNlZDogbm8NCiAgICBkZl9wcmludDogZGVmYXVsdA0KICAgIHRvY19kZXB0aDogNQ0KZWRpdG9yX29wdGlvbnM6IA0KICBtYXJrZG93bjogDQogICAgd3JhcDogNzINCi0tLQ0KDQpgYGB7ciBzZXR1cCwgaW5jbHVkZT1GQUxTRX0NCmxpYnJhcnkoZGxvb2tyKQ0KbGlicmFyeShlZGl0cnVsZXMpDQpsaWJyYXJ5KGRlZHVjb3JyZWN0KQ0KbGlicmFyeShJU0xSKSANCmxpYnJhcnkocmFuZG9tRm9yZXN0KQ0KbGlicmFyeShkcGx5cikNCmxpYnJhcnkoVklNKQ0KbGlicmFyeShnZ3Bsb3QyKQ0KbGlicmFyeShyc3RhdGl4KQ0KbGlicmFyeShuYW5pYXIpDQpteWRhdGEgPC0gcmVhZC5jc3YoImh0dHBzOi8vcmF3LmdpdGh1Yi5jb20vZWR3aW5kai9kYXRhY2xlYW5pbmcvbWFzdGVyL2RhdGEvZGlydHlfaXJpcy5jc3YiLCBoZWFkZXI9VFJVRSwgc2VwPSIsIikNCg0KY2Fyc2VhdHMgPC0gSVNMUjo6Q2Fyc2VhdHMgIA0Kc3VwcHJlc3NXYXJuaW5ncyhSTkd2ZXJzaW9uKCIzLjUuMCIpKSANCnNldC5zZWVkKDEyMykgDQpjYXJzZWF0c1tzYW1wbGUoc2VxKE5ST1coY2Fyc2VhdHMpKSwgMjApLCAiSW5jb21lIl0gPC0gTkEgDQpzdXBwcmVzc1dhcm5pbmdzKFJOR3ZlcnNpb24oIjMuNS4wIikpIA0Kc2V0LnNlZWQoNDU2KSANCmNhcnNlYXRzW3NhbXBsZShzZXEoTlJPVyhjYXJzZWF0cykpLCAxMCksICJVcmJhbiJdIDwtIE5BDQpgYGANCg0KIyMgSW50cm9kdWN0aW9uDQoNCkFuYWx5c2lzIG9mIGRhdGEgaXMgYSBwcm9jZXNzIG9mIGluc3BlY3RpbmcsIGNsZWFuaW5nLCB0cmFuc2Zvcm1pbmcsIGFuZA0KbW9kZWxpbmcgZGF0YSB3aXRoIHRoZSBnb2FsIG9mIGhpZ2hsaWdodGluZyB1c2VmdWwgaW5mb3JtYXRpb24sDQpzdWdnZXN0aW5nIGNvbmNsdXNpb25zIGFuZCBzdXBwb3J0aW5nIGRlY2lzaW9uIG1ha2luZy4NCg0KYGBge3IgZmlnMSwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiNjAlIiwgZmlnLmFsaWduID0gJ2NlbnRlcid9DQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tmbGlzaWtvd3NraS9kcy9tYXN0ZXIvZHMucG5nIikNCmBgYA0KDQpNYW55IHRpbWVzIGluIHRoZSBiZWdpbm5pbmcgd2Ugc3BlbmQgaG91cnMgb24gaGFuZGxpbmcgcHJvYmxlbXMgd2l0aA0KbWlzc2luZyB2YWx1ZXMsIGxvZ2ljYWwgaW5jb25zaXN0ZW5jaWVzIG9yIG91dGxpZXJzIGluIG91ciBkYXRhc2V0cy4gSW4NCnRoaXMgdHV0b3JpYWwgd2Ugd2lsbCBnbyB0aHJvdWdoIHRoZSBtb3N0IHBvcHVsYXIgdGVjaG5pcXVlcyBpbiBkYXRhDQpjbGVhbnNpbmcuDQoNCldlIHdpbGwgYmUgd29ya2luZyB3aXRoIHRoZSBtZXNzeSBkYXRhc2V0IGBpcmlzYC4gT3JpZ2luYWxseSBwdWJsaXNoZWQNCmF0IFVDSSBNYWNoaW5lIExlYXJuaW5nIFJlcG9zaXRvcnk6IElyaXMgRGF0YSBTZXQsIHRoaXMgc21hbGwgZGF0YXNldA0KZnJvbSAxOTM2IGlzIG9mdGVuIHVzZWQgZm9yIHRlc3Rpbmcgb3V0IG1hY2hpbmUgbGVhcm5pbmcgYWxnb3JpdGhtcyBhbmQNCnZpc3VhbGl6YXRpb25zLiBFYWNoIHJvdyBvZiB0aGUgdGFibGUgcmVwcmVzZW50cyBhbiBpcmlzIGZsb3dlciwNCmluY2x1ZGluZyBpdHMgc3BlY2llcyBhbmQgZGltZW5zaW9ucyBvZiBpdHMgYm90YW5pY2FsIHBhcnRzLCBzZXBhbCBhbmQNCnBldGFsLCBpbiBjZW50aW1ldGVycy4NCg0KVGFrZSBhIGxvb2sgYXQgdGhpcyBkYXRhc2V0IGhlcmU6DQoNCmBgYHtyIGVjaG8gPSBUUlVFfQ0KbXlkYXRhDQpgYGANCg0KIyMgRGVhbGluZyB3aXRoIE5BDQoNCkluIGFsbCBvZiBvdXIgZGF0YSBhbmFseXNlcyBzbyBmYXIgd2UgaW1wbGljaXRseSBhc3N1bWVkIHRoYXQgd2UgZG9uJ3QNCmhhdmUgYW55IG1pc3NpbmcgdmFsdWVzIGluIG91ciBkYXRhLiBJbiBwcmFjdGljZSwgdGhhdCBpcyBvZnRlbiBub3QgdGhlDQpjYXNlLiBXaGlsZSBzb21lIHN0YXRpc3RpY2FsIGFuZCBtYWNoaW5lIGxlYXJuaW5nIG1ldGhvZHMgd29yayB3aXRoDQptaXNzaW5nIGRhdGEsIG1hbnkgY29tbW9ubHkgdXNlZCBtZXRob2RzIGNhbid0LCBzbyBpdCBpcyBpbXBvcnRhbnQgdG8NCmxlYXJuIGhvdyB0byBkZWFsIHdpdGggbWlzc2luZyB2YWx1ZXMuDQoNCiMjIyBUaGUgc2V2ZXJpdHkgb2YgTkENCg0KSWYgd2UganVkZ2UgYnkgdGhlIGltcHV0YXRpb24gb3IgZGF0YSByZW1vdmluZyBtZXRob2RzIHRoYXQgYXJlIG1vc3QNCmNvbW1vbmx5IHVzZWQgaW4gcHJhY3RpY2UsIHdlIG1pZ2h0IGNvbmNsdWRlIHRoYXQgbWlzc2luZyBkYXRhIGlzIGENCnJlbGF0aXZlbHkgc2ltcGxlIHByb2JsZW0gdGhhdCBpcyBzZWNvbmRhcnkgdG8gdGhlIGluZmVyZW5jZSwgcHJlZGljdGl2ZQ0KbW9kZWxsaW5nLCBldGMuIHRoYXQgYXJlIHRoZSBwcmltYXJ5IGdvYWwgb2Ygb3VyIGFuYWx5c2lzLg0KVW5mb3J0dW5hdGVseSwgdGhhdCBpcyBub3QgdGhlIGNhc2UuIERlYWxpbmcgd2l0aCBtaXNzaW5nIHZhbHVlcyBpcyB2ZXJ5DQpjaGFsbGVuZ2luZywgb2Z0ZW4gaW4gaXRzZWxmIGEgbW9kZWxsaW5nIHByb2JsZW0uDQoNCiMjIyBUaHJlZSBjbGFzc2VzIG9mIE5BJ3MNCg0KVGhlIGNob2ljZSBvZiBhbiBhcHByb3ByaWF0ZSBtZXRob2QgaXMgaW5zZXBhcmFibGUgZnJvbSBvdXINCnVuZGVyc3RhbmRpbmcgb2Ygb3IgYXNzdW1wdGlvbnMgYWJvdXQgdGhlIHByb2Nlc3MgdGhhdCBnZW5lcmF0ZWQgdGhlDQptaXNzaW5nIHZhbHVlcyAodGhlICptaXNzaW5nbmVzcyBtZWNoYW5pc20qKS4gQmFzZWQgb24gdGhlDQpjaGFyYWN0ZXJpc3RpY3Mgb2YgdGhpcyBwcm9jZXNzIHdlIHR5cGljYWxseSBjaGFyYWN0ZXJpemUgdGhlIG1pc3NpbmcNCmRhdGEgcHJvYmxlbSBhcyBvbmUgb2YgdGhlc2UgdGhyZWUgY2FzZXM6DQoNCmEuICAqKk1DQVIqKiAoTWlzc2luZyBDb21wbGV0ZWx5IEF0IFJhbmRvbSk6IFdoZXRoZXIgb3Igbm90IGEgdmFsdWUgaXMNCiAgICBtaXNzaW5nIGlzIGluZGVwZW5kZW50IG9mIGJvdGggdGhlIG9ic2VydmVkIHZhbHVlcyBhbmQgdGhlIG1pc3NpbmcNCiAgICAodW5vYnNlcnZlZCkgdmFsdWVzLiBGb3IgZXhhbXBsZSwgaWYgd2UgaGFkIHRlbXBlcmF0dXJlIG1lYXN1cmluZw0KICAgIGRldmljZXMgYXQgZGlmZmVyZW50IGxvY2F0aW9ucyBhbmQgdGhleSBvY2Nhc3Npb25hbGx5IGFuZCByYW5kb20NCiAgICBpbnRlcnZhbHMgc3RvcHBlZCB3b3JraW5nLiBPciwgaW4gc3VydmV5cywgd2hlcmUgcmVzcG9uZGVudHMgZG9uJ3QNCiAgICByZXNwb25kIHdpdGggYSBjZXJ0YWluIHByb2JhYmlsaXR5LCBpbmRlcGVuZGVudCBvZiB0aGUNCiAgICBjaGFyYWN0ZXJpc3RpY3MgdGhhdCB3ZSBhcmUgc3VydmV5aW5nLg0KDQpiLiAgKipNQVIqKiAoTWlzc2luZyBBdCBSYW5kb20pOiBXaGV0aGVyIG9yIG5vdCBhIHZhbHVlIGlzIG1pc3NpbmcgaXMNCiAgICBpbmRlcGVuZGVudCBvZiB0aGUgbWlzc2luZyAodW5vYnNlcnZlZCkgdmFsdWVzIGJ1dCBkZXBlbmRzIG9uIHRoZQ0KICAgIG9ic2VydmVkIHZhbHVlcy4gVGhhdCBpcywgdGhlcmUgaXMgYSBwYXR0ZXJuIHRvIGhvdyB0aGUgdmFsdWVzIGFyZQ0KICAgIG1pc3NpbmcsIGJ1dCB3ZSBjb3VsZCBmdWxseSBleHBsYWluIHRoYXQgcGF0dGVybiBnaXZlbiBvbmx5IHRoZQ0KICAgIG9ic2VydmVkIGRhdGEuIEZvciBleGFtcGxlLCBpZiBvdXIgdGVtcGVyYXR1cmUgbWVhc3VyaW5nIGRldmljZXMNCiAgICBzdG9wcGVkIHdvcmtpbmcgbW9yZSBvZnRlbiBpbiBjZXJ0YWluIGxvY2F0aW9ucyB0aGFuIGluIG90aGVycy4gT3IsDQogICAgaW4gc3VydmV5cywgaWYgd29tZW4gYXJlIGxlc3MgbGlrZWx5IHRvIHJlcG9ydCB0aGVpciB3ZWlnaHQgdGhhbg0KICAgIG1lbi4NCg0KYy4gICoqTU5BUioqIChNaXNzaW5nIE5vdCBBdCBSYW5kb20pOiBXaGV0aGVyIG9yIG5vdCBhIHZhbHVlIGlzIG1pc3NpbmcNCiAgICBhbHNvIGRlcGVuZHMgb24gdGhlIG1pc3NpbmcgKHVub2JzZXJ2ZWQpIHZhbHVlcyBpbiBhIHdheSB0aGF0IGNhbid0DQogICAgYmUgZXhwbGFpbmVkIGJ5IHRoZSBvYnNlcnZlZCB2YWx1ZXMuIFRoYXQgaXMsIHRoZXJlIGlzIGEgcGF0dGVybiB0bw0KICAgIGhvdyB0aGUgdmFsdWVzIGFyZSBtaXNzaW5nLCBidXQgd2Ugd291bGRuJ3QgYmUgYWJsZSB0byBmdWxseSBleHBsYWluDQogICAgaXQgd2l0aG91dCBvYnNlcnZpbmcgdGhlIHZhbHVlcyB0aGF0IGFyZSBtaXNzaW5nLiBGb3IgZXhhbXBsZSwgaWYNCiAgICBvdXIgdGVtcGVyYXR1cmUgbWVhc3VyaW5nIGRldmljZSBoYWQgYSB0ZW5kZW5jeSB0byBzdG9wIHdvcmtpbmcgd2hlbg0KICAgIHRoZSB0ZW1wZXJhdHVyZSBpcyB2ZXJ5IGxvdy4gT3IsIGluIHN1cnZleXMsIGlmIGEgcGVyc29uIHdhcyBsZXNzDQogICAgbGlrZWx5IHRvIHJlcG9ydCB0aGVpciBzYWxhcnkgaWYgdGhlaXIgc2FsYXJ5IHdhcyBoaWdoLg0KDQojIyMgVGhlIE5BJ3MgbWVjaGFuaXNtDQoNCkV2ZXJ5IHZhcmlhYmxlIGluIG91ciBkYXRhIG1pZ2h0IGhhdmUgYSBkaWZmZXJlbnQgbWlzc2luZ25lc3MgbWVjaGFuaXNtLg0KU28sIGhvdyBkbyB3ZSBkZXRlcm1pbmUgd2hldGhlciBpdCBpcyBNQ0FSLCBNQVIsIG9yIE1OQVI/DQoNClNob3dpbmcgd2l0aCBhIHJlYXNvbmFibGUgZGVncmVlIG9mIGNlcnRhaW50eSB0aGF0IHRoZSBtZWNoYW5pc20gaXMgbm90DQpNQ0FSIGlzIGVxdWl2YWxlbnQgdG8gc2hvd2luZyB0aGF0IHRoZSBtaXNzaW5nbmVzcyAod2hldGhlciBvciBub3QgYQ0KdmFsdWUgaXMgbWlzc2luZykgY2FuIGJlIHByZWRpY3RlZCBmcm9tIG9ic2VydmVkIHZhbHVlcy4gVGhhdCBpcywgaXQgaXMNCmEgcHJlZGljdGlvbiBwcm9ibGVtIGFuZCBpdCBpcyBzdWZmaWNpZW50IHRvIHNob3cgKm9uZSB3YXkqIHRoYXQNCm1pc3NpbmduZXNzIGNhbiBiZSBwcmVkaWN0ZWQuIE9uIHRoZSBvdGhlciBoYW5kLCBpdCBpcyBpbmZlYXNpYmxlIHRvDQpzaG93IHRoYXQgdGhlIG1lY2hhbmlzbSBpcyBNQ0FSLCBiZWNhdXNlIHRoYXQgd291bGQgcmVxdWlyZSB1cyB0byBzaG93DQp0aGF0IHRoZXJlIGlzICpubyB3YXkqIG9mIHByZWRpY3RpbmcgbWlzc2luZ25lc3MgZnJvbSBvYnNlcnZlZCB2YWx1ZXMuDQpXZSBjYW4sIGhvd2V2ZXIsIHJ1bGUgb3V0IGNlcnRhaW4ga2luZHMgb2YgZGVwZW5kZW5jeSAoZm9yIGV4YW1wbGUsDQpsaW5lYXIgZGVwZW5kZW5jeSkuDQoNCkZvciBNTkFSLCB0aGUgc2l0dWF0aW9uIGlzIGV2ZW4gd29yc2UuIEluIGdlbmVyYWwsIGl0IGlzIGltcG9zc2libGUgdG8NCmRldGVybWluZSBmcm9tIHRoZSBkYXRhIHRoZSByZWxhdGlvbnNoaXAgYmV0d2VlbiBtaXNzaW5nbmVzcyBhbmQgdGhlDQp2YWx1ZSB0aGF0IGlzIG1pc3NpbmcsIGJlY2F1c2Ugd2UgZG9uJ3Qga25vdyB3aGF0IGlzIG1pc3NpbmcuIFRoYXQgaXMsDQp1bmxlc3Mgd2UgYXJlIGFibGUgdG8gc29tZWhvdyBtZWFzdXJlIHRoZSB2YWx1ZXMgdGhhdCBhcmUgbWlzc2luZywgd2UNCndvbid0IGJlIGFibGUgdG8gZGV0ZXJtaW5lIHdoZXRoZXIgb3Igbm90IHRoZSBtaXNzaW5nbmVzcyByZWdpbWUgaXMNCk1OQVIuIEdldHRpbmcgb3VyIGhhbmRzIG9uIHRoZSBtaXNzaW5nIHZhbHVlcywgaG93ZXZlciwgaXMgaW4gbW9zdCBjYXNlcw0KaW1wb3NzaWJsZSBvciBpbmZlYXNpYmxlLg0KDQpUbyBzdW1tYXJpemUsIHdlJ2xsIG9mdGVuIGJlIGFibGUgdG8gc2hvdyB0aGF0IG91ciBtaXNzaW5nbmVzcyByZWdpbWUgaXMNCm5vdCBNQ0FSIGFuZCBuZXZlciB0aGF0IGl0IGlzIE1DQVIuIFN1YnNlcXVlbnRseSwgd2UnbGwgb2Z0ZW4ga25vdyB0aGF0DQp0aGUgbWlzc2luZ25lc3MgcmVnaW1lIGlzIGF0IGxlYXN0IE1BUiwgYnV0IHdlJ2xsIHJhcmVseSBiZSBhYmxlIHRvDQpkZXRlcm1pbmUgd2hldGhlciBpdCBpcyBNQVIgb3IgTU5BUiwgdW5sZXNzIHdlIGNhbiBnZXQgb3VyIGhhbmRzIG9uIHRoZQ0KbWlzc2luZyBkYXRhLiBUaGVyZWZvcmUsIGl0IGJlY29tZXMgdmVyeSBpbXBvcnRhbnQgdG8gdXRpbGl6ZSBub3Qgb25seQ0KZGF0YSBidXQgYWxzbyBkb21haW4tc3BlY2lmaWMgYmFja2dyb3VuZCBrbm93bGVkZ2UsIHdoZW4gYXBwbGljYWJsZS4gSW4NCnBhcnRpY3VsYXIsIGtub3duIHJlbGF0aW9uc2hpcHMgYmV0d2VlbiB0aGUgdmFyaWFibGVzIGluIG91ciBkYXRhIGFuZA0Kd2hhdCBjYXVzZWQgdGhlIHZhbHVlcyB0byBiZSBtaXNzaW5nLg0KDQojIyMgQ2F1c2VzIGZvciBOQSdzDQoNClVuZGVyc3RhbmRpbmcgdGhlIGNhdXNlIGZvciBtaXNzaW5nIGRhdGEgY2FuIG9mdGVuIGhlbHAgdXMgaWRlbnRpZnkgdGhlDQptaXNzaW5nbmVzcyBtZWNoYW5pc20gYW5kIGF2b2lkIGludHJvZHVjaW5nIGEgYmlhcy4gSW4gZ2VuZXJhbCwgd2UgY2FuDQpzcGxpdCB0aGUgY2F1c2VzIGludG8gdHdvIGNsYXNzZXM6ICppbnRlbnRpb25hbCogb3IgKnVuaW50ZW50aW9uYWwqLg0KDQpJbnRlbnRpb25hbGx5IG9yIHN5c3RlbWF0aWNhbGx5IG1pc3NpbmcgZGF0YSBhcmUgbWlzc2luZyBieSBkZXNpZ24uIEZvcg0KZXhhbXBsZToNCg0KLSAgIHBhdGllbnRzIHRoYXQgZGlkIG5vdCBleHBlcmllbmNlIHBhaW4gd2VyZSBub3QgYXNrZWQgdG8gcmF0ZSB0aGVpcg0KICAgIHBhaW4sDQotICAgYSBwYXRpZW50IHRoYXQgaGFzIG5vdCBiZWVuIGRpc2NoYXJnZWQgZnJvbSB0aGUgaG9zcGl0YWwgZG9lc24ndA0KICAgIGhhdmUgYSAnZGF5cyBzcGVudCBpbiBob3NwaXRhbCBjYXJlJyBkYXRhIHBvaW50IChhbHRob3VnaCB3ZSBjYW4NCiAgICBpbmZlciBhIGxvd2VyIGJvdW5kIGZyb20gdGhlIGRheSBvZiBhcnJpdmFsKSBhbmQNCi0gICBhIHBhcnRpY3VsYXIgcHJlZGljdGlvbiBhbGdvcml0aG0ncyBwZXJmb3JtYW5jZSB3YXMgbWVhc3VyZWQgb24NCiAgICBkYXRhc2V0cyB3aXRoIGZld2VyIHRoYW4gMTAwIHZhcmlhYmxlcywgZHVlIHRvIGl0cyB0aW1lIGNvbXBsZXhpdHkNCiAgICBhbmQNCi0gICBzb21lIG1lYXN1cmVtZW50cyB3ZXJlIG5vdCBtYWRlIGJlY2F1c2UgaXQgd2FzIHRvbyBjb3N0bHkgdG8gbWFrZQ0KICAgIGFsbCBvZiB0aGVtLg0KDQpVbmludGVudGlvbmFsbHkgbWlzc2luZyBkYXRhIHdlcmUgbm90IHBsYW5uZWQuIEZvciBleGFtcGxlOg0KDQotICAgZGF0YSBtaXNzaW5nIGR1ZSB0byBtZWFzdXJlbWVudCBlcnJvciwNCi0gICBhIHN1YmplY3Qgc2tpcHBpbmcgYSBzdXJ2ZXkgcXVlc3Rpb24sDQotICAgYSBwYXRpZW50IHByZW1hdHVyZWx5IGRyb3BwaW5nIG91dCBvZiBhIHN0dWR5DQoNCmFuZCBvdGhlciByZWFzb25zIG5vdCBwbGFubmVkIGJ5IGFuZCBvdXRzaWRlIHRoZSBjb250cm9sIG9mIHRoZSBkYXRhDQpjb2xsZWN0b3IuDQoNClRoZXJlIGlzIG5vIGdlbmVyYWwgcnVsZSBhbmQgZnVydGhlciBjb25jbHVzaW9ucyBjYW4gb25seSBiZSBtYWRlIG9uIGENCmNhc2UtYnktY2FzZSBiYXNpcywgdXNpbmcgZG9tYWluIHNwZWNpZmljIGtub3dsZWRnZS4gTWVhc3VyZW1lbnQgZXJyb3INCmNhbiByYW5nZSBmcm9tIGNvbXBsZXRseSByYW5kb20gdG8gY29tcGxldGVseSBub3QtYXQtcmFuZG9tLCBzdWNoIGENCnRlbXBlcmF0dXJlIHNlbnNvciBicmVha2luZyBkb3duIGF0IGhpZ2ggdGVtcGVyYXR1cmVzLiBTdWJqZWN0cw0KdHlwaWNhbGx5IGRvIG5vdCBkcm9wIG91dCBvZiBzdHVkaWVzIGF0IHJhbmRvbSwgYnV0IHRoYXQgaXMgYWxzbyBhDQpwb3NzaWJsaXR5LiBXaGVuIG5vdCBhbGwgbWVhc3VyZW1lbnRzIGFyZSBtYWRlIHRvIHJlZHVjZSBjb3N0LCB0aGV5IGFyZQ0Kb2Z0ZW4gb21pdHRlZCBjb21wbGV0ZWx5IGF0IHJhbmRvbSwgYnV0IHNvbWV0aW1lcyBhIGRpZmZlcmVudCBkZXNpZ24gaXMNCnVzZWQuDQoNCiMjIyBEZXRlY3RpbmcgTkENCg0KQSBtaXNzaW5nIHZhbHVlLCByZXByZXNlbnRlZCBieSBOQSBpbiBSLCBpcyBhIHBsYWNlaG9sZGVyIGZvciBhIGRhdHVtIG9mDQp3aGljaCB0aGUgdHlwZSBpcyBrbm93biBidXQgaXRzIHZhbHVlIGlzbid0LiBUaGVyZWZvcmUsIGl0IGlzIGltcG9zc2libGUNCnRvIHBlcmZvcm0gc3RhdGlzdGljYWwgYW5hbHlzaXMgb24gZGF0YSB3aGVyZSBvbmUgb3IgbW9yZSB2YWx1ZXMgaW4gdGhlDQpkYXRhIGFyZSBtaXNzaW5nLiBPbmUgbWF5IGNob29zZSB0byBlaXRoZXIgb21pdCBlbGVtZW50cyBmcm9tIGEgZGF0YXNldA0KdGhhdCBjb250YWluIG1pc3NpbmcgdmFsdWVzIG9yIHRvIGltcHV0ZSBhIHZhbHVlLCBidXQgbWlzc2luZ25lc3MgaXMNCnNvbWV0aGluZyB0byBiZSBkZWFsdCB3aXRoIHByaW9yIHRvIGFueSBhbmFseXNpcy4NCg0KYGBge3IgZmlnMiwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiNjAlIiwgZmlnLmFsaWduID0gJ2NlbnRlcid9DQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tmbGlzaWtvd3NraS9kcy9tYXN0ZXIvZGF0YV9jb3dib3kucG5nIikNCmBgYA0KDQpDYW4geW91IHNlZSB0aGF0IG1hbnkgdmFsdWVzIGluIG91ciBkYXRhc2V0IGhhdmUgc3RhdHVzIE5BID0gTm90DQpBdmFpbGFibGU/IENvdW50IChvciBwbG90KSwgaG93IG1hbnkgKCUpIG9mIGFsbCAxNTAgcm93cyBpcyBjb21wbGV0ZS4NCg0KYGBge3IgY291bnQtZGVtbywgZWNobz1UUlVFfQ0Kc3VtKGNvbXBsZXRlLmNhc2VzKG15ZGF0YSkpDQpucm93KG15ZGF0YVtjb21wbGV0ZS5jYXNlcyhteWRhdGEpLCBdKS9ucm93KG15ZGF0YSkqMTAwDQpgYGANCg0KRG9lcyB0aGUgZGF0YSBjb250YWluIG90aGVyIHNwZWNpYWwgdmFsdWVzPyBJZiBpdCBkb2VzLCByZXBsYWNlIHRoZW0NCndpdGggTkEuDQoNCmBgYHtyIGNvdW50LWRlbW8yLCBlY2hvPVRSVUV9DQppcy5zcGVjaWFsIDwtIGZ1bmN0aW9uKHgpew0KICBpZiAoaXMubnVtZXJpYyh4KSkgIWlzLmZpbml0ZSh4KSBlbHNlIGlzLm5hKHgpDQp9DQoNCnNhcHBseShteWRhdGEsIGlzLnNwZWNpYWwpDQpmb3IgKG4gaW4gY29sbmFtZXMobXlkYXRhKSl7DQogIGlzLm5hKG15ZGF0YVtbbl1dKSA8LSBpcy5zcGVjaWFsKG15ZGF0YVtbbl1dKQ0KfQ0Kc3VtbWFyeShteWRhdGEpDQpgYGANCg0KIyMjIE5hbmlhUg0KDQpPTUchIFNvIGhhcmQgOi0pIEl0J3MgYmV0dGVyIHRvIHVzZSB2aXN1YWxpemF0aW9ucywgdGVzdHMuLi4gb3INCmF1dG9tYXRpY2FsbHkgZGV0ZWN0IE5BJ3Mgd2l0aCBzb21lIGZ1bmN0aW9uLi4uDQoNCmBgYHtyIG5hbmlhciwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiNjAlIiwgZmlnLmFsaWduID0gJ2NlbnRlcid9DQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tmbGlzaWtvd3NraS9kcy9tYXN0ZXIvbmFuaWFyLmpwZyIpDQpgYGANCg0KVGhlcmUgYXJlIGEgdmFyaWV0eSBvZiBkaWZmZXJlbnQgcGxvdHMgdG8gZXhwbG9yZSBtaXNzaW5nIGRhdGEgYXZhaWxhYmxlDQppbiB0aGUgbmFuaWFyIHBhY2thZ2UuIElmIHlvdSB3b3VsZCBsaWtlIHRvIGtub3cgbW9yZSBhYm91dCB0aGUNCnBoaWxvc29waHkgb2YgdGhlIGBuYW5pYXJgIHBhY2thZ2UsIHlvdSBzaG91bGQgcmVhZCB0aGUgdmlnbmV0dGUNCltHZXR0aW5nIFN0YXJ0ZWQgd2l0aA0KbmFuaWFyXShodHRwOi8vbmFuaWFyLm5qdGllcm5leS5jb20vYXJ0aWNsZXMvZ2V0dGluZy1zdGFydGVkLXctbmFuaWFyLmh0bWwpLg0KDQpBIGtleSBwb2ludCB0byByZW1lbWJlciB3aXRoIHRoZSB2aXN1YWxpc2F0aW9uIHRvb2xzIGluIGBuYW5pYXJgIGlzIHRoYXQNCnRoZXJlIGlzIGEgd2F5IHRvIGdldCB0aGUgZGF0YSBmcm9tIHRoZSBwbG90IG91dCBmcm9tIHRoZSB2aXN1YWxpc2F0aW9uLg0KDQojIyMjIEV4cGxvcmluZyBOQSdzDQoNCk9uZSBvZiB0aGUgZmlyc3QgcGxvdHMgdGhhdCBJIHJlY29tbWVuZCB5b3Ugc3RhcnQgd2l0aCB3aGVuIHlvdSBhcmUNCmZpcnN0IGV4cGxvcmluZyB5b3VyIG1pc3NpbmcgZGF0YSwgaXMgdGhlIGB2aXNfbWlzcygpYCBwbG90LCB3aGljaCBpcw0KcmUtZXhwb3J0ZWQgZnJvbSBbYHZpc2RhdGBdKGh0dHBzOi8vZ2l0aHViLmNvbS9yb3BlbnNjaS92aXNkYXQpLg0KDQpgYGB7ciB2aXMtbWlzc30NCnZpc19taXNzKGFpcnF1YWxpdHkpDQpgYGANCg0KVGhpcyBwbG90IHByb3ZpZGVzIGEgc3BlY2lmaWMgdmlzdWFsaWF0aW9uIG9mIHRoZSBhbW91bnQgb2YgbWlzc2luZw0KZGF0YSwgc2hvd2luZyBpbiBibGFjayB0aGUgbG9jYXRpb24gb2YgbWlzc2luZyB2YWx1ZXMsIGFuZCBhbHNvDQpwcm92aWRpbmcgaW5mb3JtYXRpb24gb24gdGhlIG92ZXJhbGwgcGVyY2VudGFnZSBvZiBtaXNzaW5nIHZhbHVlcw0Kb3ZlcmFsbCAoaW4gdGhlIGxlZ2VuZCksIGFuZCBpbiBlYWNoIHZhcmlhYmxlLg0KDQojIyMjIFBhdHRlcm5zDQoNCkFuIHVwc2V0IHBsb3QgZnJvbSB0aGUgYFVwU2V0UmAgcGFja2FnZSBjYW4gYmUgdXNlZCB0byB2aXN1YWxpc2UgdGhlDQpwYXR0ZXJucyBvZiBtaXNzaW5nbmVzcywgb3IgcmF0aGVyIHRoZSBjb21iaW5hdGlvbnMgb2YgbWlzc2luZ25lc3MNCmFjcm9zcyBjYXNlcy4gVG8gc2VlIGNvbWJpbmF0aW9ucyBvZiBtaXNzaW5nbmVzcyBhbmQgaW50ZXJzZWN0aW9ucyBvZg0KbWlzc2luZ25lc3MgYW1vbmdzdCB2YXJpYWJsZXMsIHVzZSB0aGUgYGdnX21pc3NfdXBzZXRgIGZ1bmN0aW9uOg0KDQpgYGB7ciBnZy1taXNzLXVwc2V0fQ0KZ2dfbWlzc191cHNldChhaXJxdWFsaXR5KQ0KYGBgDQoNCldlIGNhbiBleHBsb3JlIHRoaXMgd2l0aCBtb3JlIGNvbXBsZXggZGF0YSwgc3VjaCBhcyByaXNrZmFjdG9yczoNCg0KYGBge3J9DQpnZ19taXNzX3Vwc2V0KHJpc2tmYWN0b3JzKQ0KYGBgDQoNCiMjIyMgTkEncyBNZWNoYW5pc21zDQoNClRoZXJlIGFyZSBhIGZldyBkaWZmZXJlbnQgd2F5cyB0byBleHBsb3JlIGRpZmZlcmVudCBtaXNzaW5nIGRhdGENCm1lY2hhbmlzbXMgYW5kIHJlbGF0aW9uc2hpcHMuIE9uZSB3YXkgaW5jb3Jwb3JhdGVzIHRoZSBtZXRob2Qgb2YNCnNoaWZ0aW5nIG1pc3NpbmcgdmFsdWVzIHNvIHRoYXQgdGhleSBjYW4gYmUgdmlzdWFsaXNlZCBvbiB0aGUgc2FtZSBheGVzDQphcyB0aGUgcmVndWxhciB2YWx1ZXMsIGFuZCB0aGVuIGNvbG91cnMgdGhlIG1pc3NpbmcgYW5kIG5vdCBtaXNzaW5nDQpwb2ludHMuIFRoaXMgaXMgaW1wbGVtZW50ZWQgd2l0aCBgZ2VvbV9taXNzX3BvaW50KClgLg0KDQpgYGB7ciBnZ3Bsb3QtZ2VvbS1taXNzLXBvaW50LCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KIyB1c2luZyByZWd1bGFyIGdlb21fcG9pbnQoKQ0KZ2dwbG90KGFpcnF1YWxpdHksDQogICAgICAgYWVzKHggPSBPem9uZSwNCiAgICAgICAgICAgeSA9IFNvbGFyLlIpKSArDQpnZW9tX21pc3NfcG9pbnQoKQ0KYGBgDQoNCiMjIyMgTkEncyBpbiB2YXJzDQoNClRoaXMgcGxvdCBzaG93cyB0aGUgbnVtYmVyIG9mIG1pc3NpbmcgdmFsdWVzIGluIGVhY2ggdmFyaWFibGUgaW4gYQ0KZGF0YXNldC4gSXQgaXMgcG93ZXJlZCBieSB0aGUgYG1pc3NfdmFyX3N1bW1hcnkoKWAgZnVuY3Rpb24uDQoNCmBgYHtyIGdnLW1pc3MtdmFyfQ0KZ2dfbWlzc192YXIoYWlycXVhbGl0eSkNCmdnX21pc3NfdmFyKGFpcnF1YWxpdHkpICsgbGFicyh5ID0gIkxvb2sgYXQgYWxsIHRoZSBtaXNzaW5nIG9uZXMiKQ0KYGBgDQoNCiMjIyMgTkEncyBpbiBjYXNlcw0KDQpUaGlzIHBsb3Qgc2hvd3MgdGhlIG51bWJlciBvZiBtaXNzaW5nIHZhbHVlcyBpbiBlYWNoIGNhc2UuIEl0IGlzIHBvd2VyZWQNCmJ5IHRoZSBgbWlzc19jYXNlX3N1bW1hcnkoKWAgZnVuY3Rpb24uDQoNCmBgYHtyIGdnLW1pc3MtY2FzZX0NCmdnX21pc3NfY2FzZShhaXJxdWFsaXR5KSArIGxhYnMoeCA9ICJOdW1iZXIgb2YgQ2FzZXMiKQ0KYGBgDQoNCiMjIyMgTkEncyBhY3Jvc3MgZmFjdG9ycw0KDQpUaGlzIHBsb3Qgc2hvd3MgdGhlIG51bWJlciBvZiBtaXNzaW5ncyBpbiBlYWNoIGNvbHVtbiwgYnJva2VuIGRvd24gYnkgYQ0KY2F0ZWdvcmljYWwgdmFyaWFibGUgZnJvbSB0aGUgZGF0YXNldC4gSXQgaXMgcG93ZXJlZCBieSBhDQpgZHBseXI6Omdyb3VwX2J5YCBzdGF0ZW1lbnQgZm9sbG93ZWQgYnkgYG1pc3NfdmFyX3N1bW1hcnkoKWAuDQoNCmBgYHtyIGdnLW1pc3MtZmN0LCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQ0KZ2dfbWlzc19mY3QoeCA9IHJpc2tmYWN0b3JzLCBmY3QgPSBtYXJpdGFsKQ0KYGBgDQoNCiMjIyMgU3VtbWFyaWVzIGZvciBOQSdzDQoNCm5hbmlhciBwcm92aWRlcyBudW1lcmljYWwgc3VtbWFyaWVzIG9mIG1pc3NpbmcgZGF0YSwgdGhhdCBmb2xsb3cgYQ0KY29uc2lzdGVudCBydWxlIHRoYXQgdXNlcyBhIHN5bnRheCBiZWdpbmluZyB3aXRoIG1pc3NcXy4NCg0KYGBge3J9DQptaXNzX3Zhcl9zdW1tYXJ5KGFpcnF1YWxpdHkpDQpgYGANCg0KWW91IGNvdWxkIGFsc28gZ3JvdXBfYnkoKSB0byB3b3JrIG91dCB0aGUgbnVtYmVyIG9mIG1pc3NpbmdzIGluIGVhY2gNCnZhcmlhYmxlIGFjcm9zcyB0aGUgbGV2ZWxzIHdpdGhpbiBpdC4NCg0KYGBge3J9DQphaXJxdWFsaXR5ICU+JQ0KICBncm91cF9ieShNb250aCkgJT4lDQogIG1pc3NfdmFyX3N1bW1hcnkoKQ0KYGBgDQoNCiMjIElkZW50aWZ5IG91dGxpZXJzDQoNClRoYW5rcyB0byAqcnN0YXRpeCogbGlicmFyeSB3ZSBjYW4gZWFzaWx5IGRldGVjdCBvdXRsaWVycyB1c2luZyBib3hwbG90DQptZXRob2RzLiBCb3hwbG90cyBhcmUgYSBwb3B1bGFyIGFuZCBhbiBlYXN5IG1ldGhvZCBmb3IgaWRlbnRpZnlpbmcNCm91dGxpZXJzLg0KDQpgYGB7cn0NCiMgQ29udmVydCB0aGUgdmFyaWFibGUgZG9zZSBmcm9tIGEgbnVtZXJpYyB0byBhIGZhY3RvciB2YXJpYWJsZQ0KdG9vdGggPC0gYXMuZGF0YS5mcmFtZShUb290aEdyb3d0aCkNCnRvb3RoJGRvc2UgPC0gYXMuZmFjdG9yKHRvb3RoJGRvc2UpDQojIENoYW5nZSBib3ggcGxvdCBsaW5lIGNvbG9ycyBieSBncm91cHMNCnA8LWdncGxvdCh0b290aCwgYWVzKHg9ZG9zZSwgeT1sZW4sIGNvbG9yPWRvc2UpKSArDQogIGdlb21fYm94cGxvdCgpDQpwDQpgYGANCg0KVGhlcmUgYXJlIHR3byBjYXRlZ29yaWVzIG9mIG91dGxpZXI6ICgxKSBvdXRsaWVycyBhbmQgKDIpIGV4dHJlbWUNCnBvaW50cy4NCg0KVmFsdWVzIGFib3ZlIFEzICsgMS41eElRUiBvciBiZWxvdyBRMSAtIDEuNXhJUVIgYXJlIGNvbnNpZGVyZWQgYXMNCm91dGxpZXJzLiBWYWx1ZXMgYWJvdmUgUTMgKyAzeElRUiBvciBiZWxvdyBRMSAtIDN4SVFSIGFyZSBjb25zaWRlcmVkIGFzDQpleHRyZW1lIHBvaW50cyAob3IgZXh0cmVtZSBvdXRsaWVycykuDQoNClExIGFuZCBRMyBhcmUgdGhlIGZpcnN0IGFuZCB0aGlyZCBxdWFydGlsZSwgcmVzcGVjdGl2ZWx5LiBJUVIgaXMgdGhlDQppbnRlcnF1YXJ0aWxlIHJhbmdlIChJUVIgPSBRMyAtIFExKS4NCg0KR2VuZXJhbGx5IHNwZWFraW5nLCBkYXRhIHBvaW50cyB0aGF0IGFyZSBsYWJlbGxlZCBvdXRsaWVycyBpbiBib3hwbG90cw0KYXJlIG5vdCBjb25zaWRlcmVkIGFzIHRyb3VibGVzb21lIGFzIHRob3NlIGNvbnNpZGVyZWQgZXh0cmVtZSBwb2ludHMgYW5kDQptaWdodCBldmVuIGJlIGlnbm9yZWQuIE5vdGUgdGhhdCwgYW55IE5BIGFuZCBOYU4gYXJlIGF1dG9tYXRpY2FsbHkNCnJlbW92ZWQgYmVmb3JlIHRoZSBxdWFudGlsZXMgYXJlIGNvbXB1dGVkLg0KDQoqKkV4YW1wbGUqKg0KDQpgYGB7cn0NCnRvb3RoICU+JQ0KICBncm91cF9ieShkb3NlKSAlPiUNCiAgaWRlbnRpZnlfb3V0bGllcnMobGVuKQ0KYGBgDQoNClNlZT8gb25lIG91dGxpZXIhICh0aGF0IHdhcyB2aXNpYmxlIG9uIHRoZSBib3hwbG90KQ0KDQojIyBDaGVja2luZyBjb25zaXN0ZW5jeQ0KDQpDb25zaXN0ZW50IGRhdGEgYXJlIHRlY2huaWNhbGx5IGNvcnJlY3QgZGF0YSB0aGF0IGFyZSBmaXQgZm9yDQpzdGF0aXN0aWNhbCBhbmFseXNpcy4gVGhleSBhcmUgZGF0YSBpbiB3aGljaCBtaXNzaW5nIHZhbHVlcywgc3BlY2lhbA0KdmFsdWVzLCAob2J2aW91cykgZXJyb3JzIGFuZCBvdXRsaWVycyBhcmUgZWl0aGVyIHJlbW92ZWQsIGNvcnJlY3RlZCBvcg0KaW1wdXRlZC4gVGhlIGRhdGEgYXJlIGNvbnNpc3RlbnQgd2l0aCBjb25zdHJhaW50cyBiYXNlZCBvbiByZWFsLXdvcmxkDQprbm93bGVkZ2UgYWJvdXQgdGhlIHN1YmplY3QgdGhhdCB0aGUgZGF0YSBkZXNjcmliZS4NCg0KYGBge3IgZmlnMywgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiNjAlIiwgZmlnLmFsaWduID0gJ2NlbnRlcid9DQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tmbGlzaWtvd3NraS9kcy9tYXN0ZXIvaXJpcy5wbmciKQ0KYGBgDQoNCldlIGhhdmUgdGhlIGZvbGxvd2luZyBiYWNrZ3JvdW5kIGtub3dsZWRnZToNCg0KLSAgIFNwZWNpZXMgc2hvdWxkIGJlIG9uZSBvZiB0aGUgZm9sbG93aW5nIHZhbHVlczogc2V0b3NhLCB2ZXJzaWNvbG9yIG9yDQogICAgdmlyZ2luaWNhLg0KDQotICAgQWxsIG1lYXN1cmVkIG51bWVyaWNhbCBwcm9wZXJ0aWVzIG9mIGFuIGlyaXMgc2hvdWxkIGJlIHBvc2l0aXZlLg0KDQotICAgVGhlIHBldGFsIGxlbmd0aCBvZiBhbiBpcmlzIGlzIGF0IGxlYXN0IDIgdGltZXMgaXRzIHBldGFsIHdpZHRoLg0KDQotICAgVGhlIHNlcGFsIGxlbmd0aCBvZiBhbiBpcmlzIGNhbm5vdCBleGNlZWQgMzAgY20uDQoNCi0gICBUaGUgc2VwYWxzIG9mIGFuIGlyaXMgYXJlIGxvbmdlciB0aGFuIGl0cyBwZXRhbHMuDQoNCkRlZmluZSB0aGVzZSBydWxlcyBpbiBhIHNlcGFyYXRlIG9iamVjdCAnUlVMRScgYW5kIHJlYWQgdGhlbSBpbnRvIFINCnVzaW5nIGVkaXRzZXQgKHBhY2thZ2UgJ2VkaXRydWxlcycpLiBQcmludCB0aGUgcmVzdWx0aW5nIGNvbnN0cmFpbnQNCm9iamVjdC4NCg0KYGBge3IgZWNobyA9IFRSVUV9DQpSVUxFIDwtIGVkaXRzZXQoYygiU2VwYWwuTGVuZ3RoIDw9IDMwIiwiU3BlY2llcyAlaW4lIGMoJ3NldG9zYScsJ3ZlcnNpY29sb3InLCd2aXJnaW5pY2EnKSINCiAgICAgICAgICAgICAgICwgIlNlcGFsLkxlbmd0aCA+IDAiLCAiU2VwYWwuV2lkdGggPiAwIiwgIlBldGFsLkxlbmd0aCA+IDAiLCAiUGV0YWwuV2lkdGggPiAwIiwNCiJQZXRhbC5MZW5ndGggPj0gMiAqIFBldGFsLldpZHRoIiwgIlNlcGFsLkxlbmd0aD5QZXRhbC5MZW5ndGgiKSkNClJVTEUNCmBgYA0KDQpOb3cgd2UgYXJlIHJlYWR5IHRvIGRldGVybWluZSBob3cgb2Z0ZW4gZWFjaCBydWxlIGlzIGJyb2tlbg0KKHZpb2xhdGVkRWRpdHMpLiBBbHNvIHdlIGNhbiBzdW1tYXJpemUgYW5kIHBsb3QgdGhlIHJlc3VsdC4NCg0KYGBge3IgZWNobyA9IFRSVUV9DQpzdW1tYXJ5KHZpb2xhdGVkRWRpdHMoUlVMRSwgbXlkYXRhKSkNCmBgYA0KDQpXaGF0IHBlcmNlbnRhZ2Ugb2YgdGhlIGRhdGEgaGFzIG5vIGVycm9ycz8NCg0KYGBge3Igbm8tZXJyb3JzLCBlY2hvPVRSVUV9DQp2aW9sYXRlZCA8LSB2aW9sYXRlZEVkaXRzKFJVTEUsIG15ZGF0YSkNCnN1bW1hcnkodmlvbGF0ZWQpDQpwbG90KHZpb2xhdGVkKQ0KYGBgDQoNCiMjIyBFeGVyY2lzZSAxLg0KDQpGaW5kIG91dCB3aGljaCBvYnNlcnZhdGlvbnMgaGF2ZSB0b28gbG9uZyBzZXBhbHMgdXNpbmcgdGhlIHJlc3VsdCBvZg0KdmlvbGF0ZWRFZGl0cy4NCg0KYGBge3J9DQp2aW9sYXRlZF9vYnMgPC0gbXlkYXRhW3Zpb2xhdGVkLCBdDQpgYGANCg0KIyMjIEV4ZXJjaXNlIDIuDQoNCkZpbmQgb3V0bGllcnMgaW4gc2VwYWwgbGVuZ3RoIHVzaW5nIGJveHBsb3QgYW5kIGJveHBsb3Quc3RhdHMuIFJldHJpZXZlDQp0aGUgY29ycmVzcG9uZGluZyBvYnNlcnZhdGlvbnMgYW5kIGxvb2sgYXQgdGhlIG90aGVyIHZhbHVlcy4gQW55IGlkZWFzDQp3aGF0IG1pZ2h0IGhhdmUgaGFwcGVuZWQ/IFNldCB0aGUgb3V0bGllcnMgdG8gTkEgKG9yIGEgdmFsdWUgdGhhdCB5b3UNCmZpbmQgbW9yZSBhcHByb3BpYXRlKQ0KDQpgYGB7ciBvdXRsaWVyMSwgZWNobz1UUlVFfQ0KYm94cGxvdChteWRhdGEkU2VwYWwuTGVuZ3RoKQ0Kb3V0bGllcnMgPC0gYm94cGxvdC5zdGF0cyhteWRhdGEkU2VwYWwuTGVuZ3RoKSRvdXQNCm91dGxpZXJzX2lkeCA8LSB3aGljaChteWRhdGEkU2VwYWwuTGVuZ3RoICVpbiUgb3V0bGllcnMpDQpteWRhdGFbb3V0bGllcnNfaWR4LF0NCiMgdGhleSBhbGwgc2VlbSB0byBiZSB0b28gYmlnLi4uIG1heSB0aGV5IHdlcmUgbWVhc3VyZWQgaW4gbW0gaS5vIGNtPw0KbXlkYXRhW291dGxpZXJzX2lkeCwxOjRdIDwtIG15ZGF0YVtvdXRsaWVyc19pZHgsMTo0XS8xMA0Kc3VtbWFyeShteWRhdGEpDQpgYGANCg0KTm90ZSB0aGF0IHNpbXBsZSBib3hwbG90IHNob3dzIGFuIGV4dHJhIG91dGxpZXIhDQoNCmBgYHtyIGJveHBsb3QsIGVjaG89VFJVRX0NCmJveHBsb3QoU2VwYWwuTGVuZ3RoIH4gU3BlY2llcywgZGF0YT1teWRhdGEpDQpgYGANCg0KIyMgQ29ycmVjdGlvbnMNCg0KUmVwbGFjZSBub24gcG9zaXRpdmUgdmFsdWVzIGZyb20gU2VwYWwuV2lkdGggd2l0aCBOQSB1c2luZw0KY29ycmVjdFdpdGhSdWxlcyBmcm9tIHRoZSBsaWJyYXJ5ICdkZWR1Y29ycmVjdCcuDQoNCmBgYHtyIGNvcnJlY3Qtc2ltcGxlLCBlY2hvID0gVFJVRX0NCg0KY3IgPC0gY29ycmVjdGlvblJ1bGVzKGV4cHJlc3Npb24oDQogIGlmICghaXMubmEoU2VwYWwuV2lkdGgpICYmIFNlcGFsLldpZHRoIDw9MCApIFNlcGFsLldpZHRoID0gTkENCiAgKSkNCmNvcnJlY3RXaXRoUnVsZXMoY3IsIG15ZGF0YSkNCmBgYA0KDQpSZXBsYWNlIGFsbCBlcnJvbmVvdXMgdmFsdWVzIHdpdGggTkEgdXNpbmcgKHRoZSByZXN1bHQgb2YpDQpsb2NhbGl6ZUVycm9yczoNCg0KYGBge3IgY29ycmVjdC1zaW1wbGUyLCBlY2hvID0gVFJVRX0NCm15ZGF0YVtsb2NhbGl6ZUVycm9ycyhSVUxFLCBteWRhdGEpJGFkYXB0XSA8LSBOQQ0KIyBhbnl0aGluZyB2aW9sYXRlZD8NCmFueSh2aW9sYXRlZEVkaXRzKFJVTEUsbXlkYXRhKSwgbmEucm09VFJVRSkNCmBgYA0KDQpXZWxsIGRvbmUhIE5vIGVycm9ycyENCg0KIyMgZGxvb2tyIHBhY2thZ2UNCg0KYGBge3IgZmlnNCwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiODAlIiwgZmlnLmFsaWduID0gJ2NlbnRlcid9DQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tmbGlzaWtvd3NraS9kcy9tYXN0ZXIvdGlkeWRhdGEucG5nIikNCmBgYA0KDQpBZnRlciB5b3UgaGF2ZSBhY3F1aXJlZCB0aGUgZGF0YSwgeW91IHNob3VsZCBkbyB0aGUgZm9sbG93aW5nOg0KDQotICAgRGlhZ25vc2UgZGF0YSBxdWFsaXR5Lg0KICAgIC0gICBJZiB0aGVyZSBpcyBhIHByb2JsZW0gd2l0aCBkYXRhIHF1YWxpdHksDQogICAgLSAgIFRoZSBkYXRhIG11c3QgYmUgY29ycmVjdGVkIG9yIHJlLWFjcXVpcmVkLg0KLSAgIEV4cGxvcmUgZGF0YSB0byB1bmRlcnN0YW5kIHRoZSBkYXRhIGFuZCBmaW5kIHNjZW5hcmlvcyBmb3INCiAgICBwZXJmb3JtaW5nIHRoZSBhbmFseXNpcy4NCi0gICAqKkRlcml2ZSBuZXcgdmFyaWFibGVzIG9yIHBlcmZvcm0gdmFyaWFibGUgdHJhbnNmb3JtYXRpb25zLioqDQoNClRoZSBkbG9va3IgcGFja2FnZSBtYWtlcyB0aGVzZSBzdGVwcyBmYXN0IGFuZCBlYXN5Og0KDQotICAgUGVyZm9ybXMgYW4gZGF0YSBkaWFnbm9zaXMgb3IgYXV0b21hdGljYWxseSBnZW5lcmF0ZXMgYSBkYXRhDQogICAgZGlhZ25vc2lzIHJlcG9ydC4NCi0gICBEaXNjb3ZlciBkYXRhIGluIGEgdmFyaWV0eSBvZiB3YXlzLCBhbmQgYXV0b21hdGljYWxseSBnZW5lcmF0ZQ0KICAgIEVEQShleHBsb3JhdG9yeSBkYXRhIGFuYWx5c2lzKSByZXBvcnQuDQotICAgKipJbXB1dGUgbWlzc2luZyB2YWx1ZXMgYW5kIG91dGxpZXJzLCByZXNvbHZlIHNrZXdlZCBkYXRhLCBhbmQNCiAgICBjYXRlZ29yaXplIGNvbnRpbnVvdXMgdmFyaWFibGVzIGludG8gY2F0ZWdvcmljYWwgdmFyaWFibGVzLiBBbmQNCiAgICBnZW5lcmF0ZXMgYW4gYXV0b21hdGVkIHJlcG9ydCB0byBzdXBwb3J0IGl0LioqDQoNCkhlcmUgd2Ugd2lsbCBpbnRyb2R1Y2UgKipkYXRhIHRyYW5zZm9ybWF0aW9uKiogbWV0aG9kcyBwcm92aWRlZCBieSB0aGUNCmRsb29rciBwYWNrYWdlLiBZb3Ugd2lsbCBsZWFybiBob3cgdG8gdHJhbnNmb3JtIG9mIGB0YmxfZGZgIGRhdGEgdGhhdA0KaW5oZXJpdHMgZnJvbSBkYXRhLmZyYW1lIGFuZCBgZGF0YS5mcmFtZWAgd2l0aCBmdW5jdGlvbnMgcHJvdmlkZWQgYnkNCmRsb29rci4NCg0KZGxvb2tyIGluY3JlYXNlcyBzeW5lcmd5IHdpdGggYGRwbHlyYC4gUGFydGljdWxhcmx5IGluIGRhdGENCnRyYW5zZm9ybWF0aW9uIGFuZCBkYXRhIHdyYW5nbGUsIGl0IGluY3JlYXNlcyB0aGUgZWZmaWNpZW5jeSBvZiB0aGUNCmB0aWR5dmVyc2VgIHBhY2thZ2UgZ3JvdXAuDQoNClRvIGlsbHVzdHJhdGUgdGhlIGJhc2ljIHVzZSBvZiBkYXRhIHRyYW5zZm9ybWF0aW9uIGluIHRoZSBkbG9va3INCnBhY2thZ2UsIEkgdXNlIGEgYENhcnNlYXRzYCBkYXRhc2V0LiBgQ2Fyc2VhdHNgIGluIHRoZSBgSVNMUmAgcGFja2FnZSBpcw0Kc2ltdWxhdGlvbiBkYXRhc2V0IHRoYXQgc2VsbHMgY2hpbGRyZW4ncyBjYXIgc2VhdHMgYXQgNDAwIHN0b3Jlcy4gVGhpcw0KZGF0YSBpcyBhIGRhdGEuZnJhbWUgY3JlYXRlZCBmb3IgdGhlIHB1cnBvc2Ugb2YgcHJlZGljdGluZyBzYWxlcyB2b2x1bWUuDQoNCmBgYHtyIGltcG9ydF9kYXRhfQ0Kc3RyKENhcnNlYXRzKQ0KYGBgDQoNClRoZSBjb250ZW50cyBvZiBpbmRpdmlkdWFsIHZhcmlhYmxlcyBhcmUgYXMgZm9sbG93cy4gKFJlZmVyIHRvDQpJU0xSOjpDYXJzZWF0cyBNYW4gcGFnZSkNCg0KLSAgIFNhbGVzDQogICAgLSAgIFVuaXQgc2FsZXMgKGluIHRob3VzYW5kcykgYXQgZWFjaCBsb2NhdGlvbg0KLSAgIENvbXBQcmljZQ0KICAgIC0gICBQcmljZSBjaGFyZ2VkIGJ5IGNvbXBldGl0b3IgYXQgZWFjaCBsb2NhdGlvbg0KLSAgIEluY29tZQ0KICAgIC0gICBDb21tdW5pdHkgaW5jb21lIGxldmVsIChpbiB0aG91c2FuZHMgb2YgZG9sbGFycykNCi0gICBBZHZlcnRpc2luZw0KICAgIC0gICBMb2NhbCBhZHZlcnRpc2luZyBidWRnZXQgZm9yIGNvbXBhbnkgYXQgZWFjaCBsb2NhdGlvbiAoaW4NCiAgICAgICAgdGhvdXNhbmRzIG9mIGRvbGxhcnMpDQotICAgUG9wdWxhdGlvbg0KICAgIC0gICBQb3B1bGF0aW9uIHNpemUgaW4gcmVnaW9uIChpbiB0aG91c2FuZHMpDQotICAgUHJpY2UNCiAgICAtICAgUHJpY2UgY29tcGFueSBjaGFyZ2VzIGZvciBjYXIgc2VhdHMgYXQgZWFjaCBzaXRlDQotICAgU2hlbHZlTG9jDQogICAgLSAgIEEgZmFjdG9yIHdpdGggbGV2ZWxzIEJhZCwgR29vZCBhbmQgTWVkaXVtIGluZGljYXRpbmcgdGhlIHF1YWxpdHkNCiAgICAgICAgb2YgdGhlIHNoZWx2aW5nIGxvY2F0aW9uIGZvciB0aGUgY2FyIHNlYXRzIGF0IGVhY2ggc2l0ZQ0KLSAgIEFnZQ0KICAgIC0gICBBdmVyYWdlIGFnZSBvZiB0aGUgbG9jYWwgcG9wdWxhdGlvbg0KLSAgIEVkdWNhdGlvbg0KICAgIC0gICBFZHVjYXRpb24gbGV2ZWwgYXQgZWFjaCBsb2NhdGlvbg0KLSAgIFVyYmFuDQogICAgLSAgIEEgZmFjdG9yIHdpdGggbGV2ZWxzIE5vIGFuZCBZZXMgdG8gaW5kaWNhdGUgd2hldGhlciB0aGUgc3RvcmUgaXMNCiAgICAgICAgaW4gYW4gdXJiYW4gb3IgcnVyYWwgbG9jYXRpb24NCi0gICBVUw0KICAgIC0gICBBIGZhY3RvciB3aXRoIGxldmVscyBObyBhbmQgWWVzIHRvIGluZGljYXRlIHdoZXRoZXIgdGhlIHN0b3JlIGlzDQogICAgICAgIGluIHRoZSBVUyBvciBub3QNCg0KV2hlbiBkYXRhIGFuYWx5c2lzIGlzIHBlcmZvcm1lZCwgZGF0YSBjb250YWluaW5nIG1pc3NpbmcgdmFsdWVzIGlzIG9mdGVuDQplbmNvdW50ZXJlZC4gSG93ZXZlciwgYENhcnNlYXRzYCBpcyBjb21wbGV0ZSBkYXRhIHdpdGhvdXQgbWlzc2luZy4NClRoZXJlZm9yZSwgdGhlIG1pc3NpbmcgdmFsdWVzIGFyZSBnZW5lcmF0ZWQgYXMgZm9sbG93cy4gQW5kIEkgY3JlYXRlZCBhDQpkYXRhLmZyYW1lIG9iamVjdCBuYW1lZCBjYXJzZWF0cy4NCg0KYGBge3IgbWlzc2luZ30NCmNhcnNlYXRzIDwtIElTTFI6OkNhcnNlYXRzDQpzdXBwcmVzc1dhcm5pbmdzKFJOR3ZlcnNpb24oIjMuNS4wIikpDQpzZXQuc2VlZCgxMjMpDQpjYXJzZWF0c1tzYW1wbGUoc2VxKE5ST1coY2Fyc2VhdHMpKSwgMjApLCAiSW5jb21lIl0gPC0gTkENCnN1cHByZXNzV2FybmluZ3MoUk5HdmVyc2lvbigiMy41LjAiKSkNCnNldC5zZWVkKDQ1NikNCmNhcnNlYXRzW3NhbXBsZShzZXEoTlJPVyhjYXJzZWF0cykpLCAxMCksICJVcmJhbiJdIDwtIE5BDQpgYGANCg0KIyMjIEZ1bmN0aW9ucw0KDQpkbG9va3IgaW1wdXRlcyBtaXNzaW5nIHZhbHVlcyBhbmQgb3V0bGllcnMgYW5kIHJlc29sdmVzIHNrZXdlZCBkYXRhLiBJdA0KYWxzbyBwcm92aWRlcyB0aGUgYWJpbGl0eSB0byBiaW4gY29udGludW91cyB2YXJpYWJsZXMgYXMgY2F0ZWdvcmljYWwNCnZhcmlhYmxlcy4NCg0KSGVyZSBpcyBhIGxpc3Qgb2YgdGhlIGRhdGEgY29udmVyc2lvbiBmdW5jdGlvbnMgYW5kIGZ1bmN0aW9ucyBwcm92aWRlZA0KYnkgZGxvb2tyOg0KDQotICAgYGZpbmRfbmEoKWAgZmluZHMgYSB2YXJpYWJsZSB0aGF0IGNvbnRhaW5zIHRoZSBtaXNzaW5nIHZhbHVlcw0KICAgIHZhcmlhYmxlLCBhbmQgYGltcHV0YXRlX25hKClgIGltcHV0ZXMgdGhlIG1pc3NpbmcgdmFsdWVzLg0KLSAgIGBmaW5kX291dGxpZXJzKClgIGZpbmRzIGEgdmFyaWFibGUgdGhhdCBjb250YWlucyB0aGUgb3V0bGllcnMsIGFuZA0KICAgIGBpbXB1dGF0ZV9vdXRsaWVyKClgIGltcHV0ZXMgdGhlIG91dGxpZXIuDQotICAgYHN1bW1hcnkuaW1wdXRhdGlvbigpYCBhbmQgYHBsb3QuaW1wdXRhdGlvbigpYCBwcm92aWRlIGluZm9ybWF0aW9uDQogICAgYW5kIHZpc3VhbGl6YXRpb24gb2YgdGhlIGltcHV0ZWQgdmFyaWFibGVzLg0KLSAgIGBmaW5kX3NrZXduZXNzKClgIGZpbmRzIHRoZSB2YXJpYWJsZXMgb2YgdGhlIHNrZXdlZCBkYXRhLCBhbmQNCiAgICBgdHJhbnNmb3JtKClgIHBlcmZvcm1zIHRoZSByZXNvbHZpbmcgb2YgdGhlIHNrZXdlZCBkYXRhLg0KLSAgIGB0cmFuc2Zvcm0oKWAgYWxzbyBwZXJmb3JtcyBzdGFuZGFyZGl6YXRpb24gb2YgbnVtZXJpYyB2YXJpYWJsZXMuDQotICAgYHN1bW1hcnkudHJhbnNmb3JtKClgIGFuZCBgcGxvdC50cmFuc2Zvcm0oKWAgcHJvdmlkZSBpbmZvcm1hdGlvbiBhbmQNCiAgICB2aXN1YWxpemF0aW9uIG9mIHRyYW5zZm9ybWVkIHZhcmlhYmxlcy4NCi0gICBgYmlubmluZygpYCBhbmQgYGJpbm5pbmdfYnkoKWAgY29udmVydCBiaW5hdGlvbmFsIGRhdGEgaW50bw0KICAgIGNhdGVnb3JpY2FsIGRhdGEuDQotICAgYHByaW50LmJpbnMoKWAgYW5kIGBzdW1tYXJ5LmJpbnMoKWAgc2hvdyBhbmQgc3VtbWFyaXplIHRoZSBiaW5uaW5nDQogICAgcmVzdWx0cy4NCi0gICBgcGxvdC5iaW5zKClgIGFuZCBgcGxvdC5vcHRpbWFsX2JpbnMoKWAgcHJvdmlkZSB2aXN1YWxpemF0aW9uIG9mIHRoZQ0KICAgIGJpbm5pbmcgcmVzdWx0Lg0KLSAgIGB0cmFuc2Zvcm1hdGlvbl9yZXBvcnQoKWAgcGVyZm9ybXMgdGhlIGRhdGEgdHJhbnNmb3JtIGFuZCByZXBvcnRzDQogICAgdGhlIHJlc3VsdC4NCg0KIyMjIEltcHV0YXRpb25zIG9mIE5BJ3MNCg0KYGltcHV0YXRlX25hKClgIGltcHV0ZXMgdGhlIG1pc3NpbmcgdmFsdWUgY29udGFpbmVkIGluIHRoZSB2YXJpYWJsZS4gVGhlDQpwcmVkaWN0b3Igd2l0aCBtaXNzaW5nIHZhbHVlcyBzdXBwb3J0IGJvdGggbnVtZXJpYyBhbmQgY2F0ZWdvcmljYWwNCnZhcmlhYmxlcywgYW5kIHN1cHBvcnRzIHRoZSBmb2xsb3dpbmcgYG1ldGhvZGAuDQoNCkltcHV0YXRpb24gaXMgdGhlIHByb2Nlc3Mgb2YgZXN0aW1hdGluZyBvciBkZXJpdmluZyB2YWx1ZXMgZm9yIGZpZWxkcw0Kd2hlcmUgZGF0YSBpcyBtaXNzaW5nLiBUaGVyZSBpcyBhIHZhc3QgYm9keSBvZiBsaXRlcmF0dXJlIG9uIGltcHV0YXRpb24NCm1ldGhvZHMgYW5kIGl0IGdvZXMgYmV5b25kIHRoZSBzY29wZSBvZiB0aGlzIHR1dG9yaWFsIHRvIGRpc2N1c3MgYWxsIG9mDQp0aGVtLg0KDQpUaGVyZSBpcyBubyBvbmUgc2luZ2xlIGJlc3QgaW1wdXRhdGlvbiBtZXRob2QgdGhhdCB3b3JrcyBpbiBhbGwgY2FzZXMuDQpUaGUgaW1wdXRhdGlvbiBtb2RlbCBvZiBjaG9pY2UgZGVwZW5kcyBvbiB3aGF0IGF1eGlsaWFyeSBpbmZvcm1hdGlvbiBpcw0KYXZhaWxhYmxlIGFuZCB3aGV0aGVyIHRoZXJlIGFyZSAobXVsdGl2YXJpYXRlKSBlZGl0IHJlc3RyaWN0aW9ucyBvbiB0aGUNCmRhdGEgdG8gYmUgaW1wdXRlZC4gVGhlIGF2YWlsYWJpbGl0eSBvZiBSIHNvZnR3YXJlIGZvciBpbXB1dGF0aW9uIHVuZGVyDQplZGl0IHJlc3RyaWN0aW9ucyBpcywgdG8gb3VyIGJlc3Qga25vd2xlZGdlLCBsaW1pdGVkLiBIb3dldmVyLCBhIHZpYWJsZQ0Kc3RyYXRlZ3kgZm9yIGltcHV0aW5nIG51bWVyaWNhbCBkYXRhIGlzIHRvIGZpcnN0IGltcHV0ZSBtaXNzaW5nIHZhbHVlcw0Kd2l0aG91dCByZXN0cmljdGlvbnMsIGFuZCB0aGVuIG1pbmltYWxseSBhZGp1c3QgdGhlIGltcHV0ZWQgdmFsdWVzIHNvDQp0aGF0IHRoZSByZXN0cmljdGlvbnMgYXJlIG9iZXllZC4gU2VwYXJhdGVseSwgdGhlc2UgbWV0aG9kcyBhcmUNCmF2YWlsYWJsZSBpbiBSLg0KDQotICAgcHJlZGljdG9yIGlzIG51bWVyaWNhbCB2YXJpYWJsZQ0KICAgIC0gICAibWVhbiIgOiBhcml0aG1ldGljIG1lYW4NCiAgICAtICAgIm1lZGlhbiIgOiBtZWRpYW4NCiAgICAtICAgIm1vZGUiIDogbW9kZQ0KICAgIC0gICAia25uIiA6IEstbmVhcmVzdCBuZWlnaGJvcnMNCiAgICAgICAgLSAgIHRhcmdldCB2YXJpYWJsZSBtdXN0IGJlIHNwZWNpZmllZA0KICAgIC0gICAicnBhcnQiIDogUmVjdXJzaXZlIFBhcnRpdGlvbmluZyBhbmQgUmVncmVzc2lvbiBUcmVlcw0KICAgICAgICAtICAgdGFyZ2V0IHZhcmlhYmxlIG11c3QgYmUgc3BlY2lmaWVkXA0KICAgIC0gICAibWljZSIgOiBNdWx0aXZhcmlhdGUgSW1wdXRhdGlvbiBieSBDaGFpbmVkIEVxdWF0aW9ucw0KICAgICAgICAtICAgdGFyZ2V0IHZhcmlhYmxlIG11c3QgYmUgc3BlY2lmaWVkXA0KICAgICAgICAtICAgcmFuZG9tIHNlZWQgbXVzdCBiZSBzZXQNCi0gICBwcmVkaWN0b3IgaXMgY2F0ZWdvcmljYWwgdmFyaWFibGUNCiAgICAtICAgIm1vZGUiIDogbW9kZQ0KICAgIC0gICAicnBhcnQiIDogUmVjdXJzaXZlIFBhcnRpdGlvbmluZyBhbmQgUmVncmVzc2lvbiBUcmVlcw0KICAgICAgICAtICAgdGFyZ2V0IHZhcmlhYmxlIG11c3QgYmUgc3BlY2lmaWVkXA0KICAgIC0gICAibWljZSIgOiBNdWx0aXZhcmlhdGUgSW1wdXRhdGlvbiBieSBDaGFpbmVkIEVxdWF0aW9ucw0KICAgICAgICAtICAgdGFyZ2V0IHZhcmlhYmxlIG11c3QgYmUgc3BlY2lmaWVkXA0KICAgICAgICAtICAgcmFuZG9tIHNlZWQgbXVzdCBiZSBzZXQNCg0KKipFeGFtcGxlKioNCg0KSW4gdGhlIGZvbGxvd2luZyBleGFtcGxlLCBgaW1wdXRhdGVfbmEoKWAgaW1wdXRlcyB0aGUgbWlzc2luZyB2YWx1ZSBvZg0KYEluY29tZWAsIGEgbnVtZXJpYyB2YXJpYWJsZSBvZiBjYXJzZWF0cywgdXNpbmcgdGhlICJycGFydCIgbWV0aG9kLg0KYHN1bW1hcnkoKWAgc3VtbWFyaXplcyBtaXNzaW5nIHZhbHVlIGltcHV0YXRpb24gaW5mb3JtYXRpb24sIGFuZA0KYHBsb3QoKWAgdmlzdWFsaXplcyBtaXNzaW5nIGluZm9ybWF0aW9uLg0KDQpgYGB7ciBpbXB1dGF0ZV9uYSwgZmlnLmFsaWduPSdjZW50ZXInLCBmaWcud2lkdGggPSA3LCBmaWcuaGVpZ2h0ID0gNX0NCmluY29tZSA8LSBpbXB1dGF0ZV9uYShjYXJzZWF0cywgSW5jb21lLCBVUywgbWV0aG9kID0gInJwYXJ0IikNCiAgIyBzdW1tYXJ5IG9mIGltcHV0YXRpb24NCiAgc3VtbWFyeShpbmNvbWUpDQogICMgdml6IG9mIGltcHV0YXRpb24NCiAgcGxvdChpbmNvbWUpDQpgYGANCg0KKipFeGFtcGxlKioNCg0KVGhlIGZvbGxvd2luZyBpbXB1dGVzIHRoZSBjYXRlZ29yaWNhbCB2YXJpYWJsZSBgdXJiYW5gIGJ5IHRoZSAibWljZSINCm1ldGhvZC4NCg0KYGBge3IgaW1wdXRhdGVfbmEyLCBmaWcuYWxpZ249J2NlbnRlcicsIGZpZy53aWR0aCA9IDcsIGZpZy5oZWlnaHQgPSA1fQ0KbGlicmFyeShtaWNlKQ0KdXJiYW4gPC0gaW1wdXRhdGVfbmEoY2Fyc2VhdHMsIFVyYmFuLCBVUywgbWV0aG9kID0gIm1pY2UiKQ0KIyByZXN1bHQgb2YgaW1wdXRhdGlvbg0KdXJiYW4NCiMgc3VtbWFyeSBvZiBpbXB1dGF0aW9uDQpzdW1tYXJ5KHVyYmFuKQ0KIyB2aXogb2YgaW1wdXRhdGlvbg0KcGxvdCh1cmJhbikNCmBgYA0KDQpJbiB0aGUgZm9sbG93aW5nIGV4ZXJjaXNlIHRyeSB0byBpbXB1dGUgdGhlIG1pc3NpbmcgdmFsdWUgb2YgdGhlDQpgSW5jb21lYCB2YXJpYWJsZSwgYW5kIHRoZW4gY2FsY3VsYXRlcyB0aGUgYXJpdGhtZXRpYyBtZWFuIGZvciBlYWNoDQpsZXZlbCBvZiBgVVNgLiBJbiB0aGlzIGNhc2UsIGBkcGx5cmAgc2hvdWxkIGJlIHVzZWQsIGFuZCBpdCBpcyBlYXNpbHkNCmludGVycHJldGVkIGxvZ2ljYWxseSB1c2luZyBwaXBlcyENCg0KIyMjIEV4ZXJjaXNlIDMuDQoNCmBgYHtyIGV4M30NCmNhcnNlYXRzICU+JQ0KICBtdXRhdGUoSW5jb21lID0gaWZlbHNlKGlzLm5hKEluY29tZSksIG1lYW4oSW5jb21lLCBuYS5ybSA9IFRSVUUpLCBJbmNvbWUpKSAlPiUNCiAgZ3JvdXBfYnkoVVMpICU+JQ0KICBzdW1tYXJpc2UoQXZnX0luY29tZSA9IG1lYW4oSW5jb21lLCBuYS5ybSA9IFRSVUUpKQ0KYGBgDQoNCiMjIyBJbXB1dGF0aW9uIG9mIG91dGxpZXJzDQoNCmBpbXB1dGF0ZV9vdXRsaWVyKClgIGltcHV0ZXMgdGhlIG91dGxpZXJzIHZhbHVlLiBUaGUgcHJlZGljdG9yIHdpdGgNCm91dGxpZXJzIHN1cHBvcnRzIG9ubHkgbnVtZXJpYyB2YXJpYWJsZXMgYW5kIHN1cHBvcnRzIHRoZSBmb2xsb3dpbmcNCm1ldGhvZHMuDQoNCi0gICBwcmVkaWN0b3IgaXMgbnVtZXJpY2FsIHZhcmlhYmxlDQogICAgLSAgICJtZWFuIiA6IGFyaXRobWV0aWMgbWVhbg0KICAgIC0gICAibWVkaWFuIiA6IG1lZGlhbg0KICAgIC0gICAibW9kZSIgOiBtb2RlDQogICAgLSAgICJjYXBwaW5nIiA6IEltcHV0ZSB0aGUgdXBwZXIgb3V0bGllcnMgd2l0aCA5NSBwZXJjZW50aWxlLCBhbmQNCiAgICAgICAgSW1wdXRlIHRoZSBib3R0b20gb3V0bGllcnMgd2l0aCA1IHBlcmNlbnRpbGUuDQoNCiMjIyBFeGVyY2lzZSA0Lg0KDQpVc2UgdGhlIGZ1bmN0aW9uIGBpbXB1dGF0ZV9vdXRsaWVyKClgIHRvIGltcHV0ZSB0aGUgb3V0bGllcnMgb2YgdGhlDQpudW1lcmljIHZhcmlhYmxlIGBQcmljZWAgYXMgdGhlICJjYXBwaW5nIiBtZXRob2QuIEhpbnQ6IGBzdW1tYXJ5KClgDQpzdW1tYXJpemVzIG91dGxpZXJzIGltcHV0YXRpb24gaW5mb3JtYXRpb24sIGFuZCBgcGxvdCgpYCB2aXN1YWxpemVzDQppbXB1dGF0aW9uIGluZm9ybWF0aW9uLg0KDQpgYGB7ciBleDR9DQpjYXJzZWF0cyAlPiUNCiAgbXV0YXRlKFByaWNlX2ltcCA9IGltcHV0YXRlX291dGxpZXIoY2Fyc2VhdHMsIFByaWNlLCBtZXRob2QgPSAiY2FwcGluZyIpKSAlPiUNCiAgc3VtbWFyeSgpDQpgYGANCg0KVGhlIGZvbGxvd2luZyBleGFtcGxlIGltcHV0ZXMgdGhlIG91dGxpZXJzIG9mIHRoZSBgUHJpY2VgIHZhcmlhYmxlLCBhbmQNCnRoZW4gY2FsY3VsYXRlcyB0aGUgYXJpdGhtZXRpYyBtZWFuIGZvciBlYWNoIGxldmVsIG9mIGBVU2AuIEluIHRoaXMNCmNhc2UsIGBkcGx5cmAgaXMgdXNlZCwgYW5kIGl0IGlzIGVhc2lseSBpbnRlcnByZXRlZCBsb2dpY2FsbHkgdXNpbmcNCnBpcGVzLg0KDQoqKkV4YW1wbGUqKg0KDQpgYGB7ciBpbXB1dGF0ZV9vdXRsaWVyMn0NCiMgVGhlIG1lYW4gYmVmb3JlIGFuZCBhZnRlciB0aGUgaW1wdXRhdGlvbiBvZiB0aGUgUHJpY2UgdmFyaWFibGUNCmNhcnNlYXRzICU+JQ0KICBtdXRhdGUoUHJpY2VfaW1wID0gaW1wdXRhdGVfb3V0bGllcihjYXJzZWF0cywgUHJpY2UsIG1ldGhvZCA9ICJjYXBwaW5nIikpICU+JQ0KICBncm91cF9ieShVUykgJT4lDQogIHN1bW1hcmlzZShvcmlnID0gbWVhbihQcmljZSwgbmEucm0gPSBUUlVFKSwNCiAgICBpbXB1dGF0aW9uID0gbWVhbihQcmljZV9pbXAsIG5hLnJtID0gVFJVRSkpDQpgYGANCg0KIyMgVHJhbnNmb3JtYXRpb25zDQoNCkZpbmFsbHksIHdlIHNvbWV0aW1lcyBlbmNvdW50ZXIgdGhlIHNpdHVhdGlvbiB3aGVyZSB3ZSBoYXZlIHByb2JsZW1zDQp3aXRoIHNrZXdlZCBkaXN0cmlidXRpb25zIG9yIHdlIGp1c3Qgd2FudCB0byBbdHJhbnNmb3JtXXsudW5kZXJsaW5lfSwNCltyZWNvZGVdey51bmRlcmxpbmV9IG9yIHBlcmZvcm0gW2Rpc2NyZXRpemF0aW9uXXsudW5kZXJsaW5lfS4gTGV0J3MNCnJldmlldyBzb21lIG9mIHRoZSBtb3N0IHBvcHVsYXIgdHJhbnNmb3JtYXRpb24gbWV0aG9kcy4NCg0KRmlyc3QsIHN0YW5kYXJkaXphdGlvbiAoYWxzbyBrbm93biBhcyBub3JtYWxpemF0aW9uKToNCg0KLSAgICoqWi1zY29yZSoqIGFwcHJvYWNoIC0gc3RhbmRhcmRpemF0aW9uIHByb2NlZHVyZSwgdXNpbmcgdGhlIGZvcm11bGE6DQogICAgJHo9XGZyYWN7eC1cbXV9e1xzaWdtYX0kIHdoZXJlICRcbXUkID0gbWVhbiBhbmQgJFxzaWdtYSQgPSBzdGFuZGFyZA0KICAgIGRldmlhdGlvbi4gWi1zY29yZXMgYXJlIGFsc28ga25vd24gYXMgc3RhbmRhcmRpemVkIHNjb3JlczsgdGhleSBhcmUNCiAgICBzY29yZXMgKG9yIGRhdGEgdmFsdWVzKSB0aGF0IGhhdmUgYmVlbiBnaXZlbiBhIGNvbW1vbiAqc3RhbmRhcmQqLg0KICAgIFRoaXMgc3RhbmRhcmQgaXMgYSBtZWFuIG9mIHplcm8gYW5kIGEgc3RhbmRhcmQgZGV2aWF0aW9uIG9mIDEuDQoNCi0gICAqKm1pbm1heCoqIGFwcHJvYWNoIC0gQW4gYWx0ZXJuYXRpdmUgYXBwcm9hY2ggdG8gWi1zY29yZQ0KICAgIG5vcm1hbGl6YXRpb24gKG9yIHN0YW5kYXJkaXphdGlvbikgaXMgdGhlIHNvLWNhbGxlZCBNaW5NYXggc2NhbGluZw0KICAgIChvZnRlbiBhbHNvIHNpbXBseSBjYWxsZWQgIm5vcm1hbGl6YXRpb24iIC0gYSBjb21tb24gY2F1c2UgZm9yDQogICAgYW1iaWd1aXRpZXMpLiBJbiB0aGlzIGFwcHJvYWNoLCB0aGUgZGF0YSBpcyBzY2FsZWQgdG8gYSBmaXhlZA0KICAgIHJhbmdlIC0gdXN1YWxseSAwIHRvIDEuIFRoZSBjb3N0IG9mIGhhdmluZyB0aGlzIGJvdW5kZWQgcmFuZ2UgLSBpbg0KICAgIGNvbnRyYXN0IHRvIHN0YW5kYXJkaXphdGlvbiAtIGlzIHRoYXQgd2Ugd2lsbCBlbmQgdXAgd2l0aCBzbWFsbGVyDQogICAgc3RhbmRhcmQgZGV2aWF0aW9ucywgd2hpY2ggY2FuIHN1cHByZXNzIHRoZSBlZmZlY3Qgb2Ygb3V0bGllcnMuIElmDQogICAgeW91IHdvdWxkIGxpa2UgdG8gcGVyZm9ybSBNaW5NYXggc2NhbGluZyAtIHNpbXBseSBzdWJzdHJhY3QgbWluaW11bQ0KICAgIHZhbHVlIGFuZCBkaXZpZGUgaXQgYnkgcmFuZ2U6JCh4LW1pbikvKG1heC1taW4pJA0KDQpJbiBvcmRlciB0byBzb2x2ZSBwcm9ibGVtcyB3aXRoIHZlcnkgc2tld2VkIGRpc3RyaWJ1dGlvbnMgd2UgY2FuIGFsc28NCnVzZSBzZXZlcmFsIHR5cGVzIG9mIHNpbXBsZSB0cmFuc2Zvcm1hdGlvbnM6DQoNCmBgYCAgICAgICAgIA0KKyAibG9nIiA6IGxvZyB0cmFuc2Zvcm1hdGlvbi4gbG9nKHgpDQorICJsb2crMSIgOiBsb2cgdHJhbnNmb3JtYXRpb24uIGxvZyh4ICsgMSkuIFVzZWQgZm9yIHZhbHVlcyB0aGF0IGNvbnRhaW4gMC4NCisgInNxcnQiIDogc3F1YXJlIHJvb3QgdHJhbnNmb3JtYXRpb24uDQorICIxL3giIDogMSAvIHggdHJhbnNmb3JtYXRpb24NCisgInheMiIgOiB4IHNxdWFyZSB0cmFuc2Zvcm1hdGlvbg0KKyAieF4zIiA6IHheMyBzcXVhcmUgdHJhbnNmb3JtYXRpb24NCmBgYA0KDQojIyMgU3RhbmRhcmRpemF0aW9uDQoNClVzZSB0aGUgbWV0aG9kcyAienNjb3JlIiBhbmQgIm1pbm1heCIgdG8gcGVyZm9ybSBzdGFuZGFyZGl6YXRpb24uDQoNCmBgYHtyIHN0YW5kYXJkaXphdGlvbiwgZmlnLmFsaWduPSdjZW50ZXInLCBmaWcud2lkdGggPSA3LCBmaWcuaGVpZ2h0ID0gNX0NCmNhcnNlYXRzICU+JSANCiAgbXV0YXRlKEluY29tZV9taW5tYXggPSB0cmFuc2Zvcm0oY2Fyc2VhdHMkSW5jb21lLCBtZXRob2QgPSAibWlubWF4IiksDQogICAgU2FsZXNfbWlubWF4ID0gdHJhbnNmb3JtKGNhcnNlYXRzJFNhbGVzLCBtZXRob2QgPSAibWlubWF4IikpICU+JSANCiAgc2VsZWN0KEluY29tZV9taW5tYXgsIFNhbGVzX21pbm1heCkgJT4lIA0KICBib3hwbG90KCkNCmBgYA0KDQojIyMgQmlubmluZw0KDQpTb21ldGltZXMgd2UganVzdCB3b3VsZCBsaWtlIHRvIHBlcmZvcm0gc28gY2FsbGVkICdiaW5uaW5nJyBwcm9jZWR1cmUgdG8NCmJlIGFibGUgdG8gYW5hbHl6ZSBvdXIgY2F0ZWdvcmljYWwgZGF0YSwgdG8gY29tcGFyZSBzZXZlcmFsIGNhdGVnb3JpY2FsDQp2YXJpYWJsZXMsIHRvIGNvbnN0cnVjdCBzdGF0aXN0aWNhbCBtb2RlbHMgZXRjLiBUaGFua3MgdG8gdGhlICdiaW5uaW5nJw0KZnVuY3Rpb24gd2UgY2FuIHRyYW5zZm9ybSBxdWFudGl0YXRpdmUgdmFyaWFibGVzIGludG8gY2F0ZWdvcmljYWwgdXNpbmcNCnNldmVyYWwgbWV0aG9kczoNCg0KYGJpbm5pbmcoKWAgdHJhbnNmb3JtcyBhIG51bWVyaWMgdmFyaWFibGUgaW50byBhIGNhdGVnb3JpY2FsIHZhcmlhYmxlIGJ5DQpiaW5uaW5nIGl0LiBUaGUgZm9sbG93aW5nIHR5cGVzIG9mIGJpbm5pbmcgYXJlIHN1cHBvcnRlZC4NCg0KLSAgICJxdWFudGlsZSIgOiBjYXRlZ29yaXplIHVzaW5nIHF1YW50aWxlIHRvIGluY2x1ZGUgdGhlIHNhbWUNCiAgICBmcmVxdWVuY2llcw0KLSAgICJlcXVhbCIgOiBjYXRlZ29yaXplIHRvIGhhdmUgZXF1YWwgbGVuZ3RoIHNlZ21lbnRzDQotICAgInByZXR0eSIgOiBjYXRlZ29yaXplZCBpbnRvIG1vZGVyYXRlbHkgZ29vZCBzZWdtZW50cw0KLSAgICJrbWVhbnMiIDogY2F0ZWdvcml6YXRpb24gdXNpbmcgSy1tZWFucyBjbHVzdGVyaW5nDQotICAgImJjbHVzdCIgOiBjYXRlZ29yaXphdGlvbiB1c2luZyBiYWdnZWQgY2x1c3RlcmluZyB0ZWNobmlxdWUNCg0KSGVyZSBhcmUgc29tZSBleGFtcGxlcyBvZiBob3cgdG8gYmluIGBJbmNvbWVgIHVzaW5nIGBiaW5uaW5nKClgLjoNCg0KKipFeGFtcGxlOioqDQoNCmBgYHtyIGJpbjF9DQojIEJpbm5pbmcgdGhlIGNhcmF0IHZhcmlhYmxlLiBkZWZhdWx0IHR5cGUgYXJndW1lbnQgaXMgInF1YW50aWxlIg0KYmluIDwtIGJpbm5pbmcoY2Fyc2VhdHMkSW5jb21lKQ0KIyBQcmludCBiaW5zIGNsYXNzIG9iamVjdA0KYmluDQojIFN1bW1hcml6ZSBiaW5zIGNsYXNzIG9iamVjdA0Kc3VtbWFyeShiaW4pDQojIFBsb3QgYmlucyBjbGFzcyBvYmplY3QNCnBsb3QoYmluKQ0KYGBgDQoNClVzaW5nIHBpcGVzICYgZHBseXI6DQoNCmBgYHtyIGJpbm5pbmcsIGVjaG8gPSBUUlVFfQ0KIGNhcnNlYXRzICU+JQ0KIG11dGF0ZShJbmNvbWVfYmluID0gYmlubmluZyhjYXJzZWF0cyRJbmNvbWUpICU+JSANCiAgICAgICAgICAgICAgICAgICAgIGV4dHJhY3QoKSkgJT4lDQogZ3JvdXBfYnkoU2hlbHZlTG9jLCBJbmNvbWVfYmluKSAlPiUNCiBzdW1tYXJpc2UoZnJlcSA9IG4oKSkgJT4lDQogYXJyYW5nZShkZXNjKGZyZXEpKSAlPiUNCiBoZWFkKDEwKQ0KYGBgDQoNCioqRXhhbXBsZToqKiBSZWNvZGUgdGhlIG9yaWdpbmFsIGRpc3RyaWJ1dGlvbiBvZiBpbmNvbWVzIHVzaW5nIGZpeGVkDQpsZW5ndGggb2YgaW50ZXJ2YWxzIGFuZCBhc3NpZ24gdGhlbSBsYWJlbHMuDQoNCmBgYHtyIGV4YW1wbGU3LCBlY2hvID0gVFJVRX0NCmluY29tZV9maXhlZDwtIGJpbm5pbmcoY2Fyc2VhdHMkSW5jb21lLCBuYmlucyA9IDQsDQogICAgICAgICAgICAgICAgICAgbGFiZWxzID0gYygibG93IiwgImF2ZXJhZ2UiLCAiaGlnaCIsICJ2ZXJ5IGhpZ2giKSkNCnN1bW1hcnkoaW5jb21lX2ZpeGVkKQ0KcGxvdChpbmNvbWVfZml4ZWQpDQpgYGANCg0KSW4gY2FzZSBvZiBzdGF0aXN0aWNhbCBtb2RlbGluZyAoaS5lLiBjcmVkaXQgc2NvcmluZyBwdXJwb3NlcykgLSB3ZSBuZWVkDQp0byBiZSBhd2FyZSBvZiB0aGUgZmFjdCwgdGhhdCB0aGUgWyoqKm9wdGltYWwqKipdey51bmRlcmxpbmV9DQpkaXNjcmV0aXphdGlvbiBvZiB0aGUgb3JpZ2luYWwgZGlzdHJpYnV0aW9uIG11c3QgYmUgYWNoaWV2ZWQuIFRoZQ0KJypiaW5uaW5nX2J5KicgZnVuY3Rpb24gY29tZXMgd2l0aCBzb21lIGhlbHAgaGVyZS4NCg0KKipFeGFtcGxlOioqIFBlcmZvcm0gZGlzY3JldGl6YXRpb24gb2YgdGhlIHZhcmlhYmxlICdBZHZlcnRpc2luZycgdXNpbmcNCm9wdGltYWwgYmlubmluZy4NCg0KYGBge3IgZXhhbXBsZTgsIGVjaG8gPSBUUlVFfQ0KYmluIDwtIGJpbm5pbmdfYnkoY2Fyc2VhdHMsICJVUyIsICJBZHZlcnRpc2luZyIpDQpzdW1tYXJ5KGJpbikNCnBsb3QoYmluKQ0KYGBgDQoNClBsZWFzZSB0YWtlIGEgbG9vayBvbmNlIGFnYWluIGF0IHRoZSBmaW5hbCByZXBvcnQgb2Ygb3VyIGJpbm5pbmcNCnByb2NlZHVyZS4gVHJ5IHRvIGludGVycHJldCBpbmZvcm1hdGlvbiBjcml0ZXJpb24gKEplZmZyZXkncyBvciBLLVMpIGFuZA0KcGxvdHMuDQoNCldlIGNhbiBmaW5hbGx5IHByaW50IHNvbWUgc3VtbWFyeSByZXBvcnRzIGFmdGVyIHRoZSBjbGVhbnNpbmcgcHJvY2VkdXJlcw0KYXJlIGFsbCBkb25lIC0gaW4gdGhlIFBERiBmb3JtYXQgKFRFWCBpbnN0YWxsYXRpb24gaXMgcmVxdWlyZWQgZmlyc3QpIG9yDQpIVE1MIGZvcm1hdC4gUGxlYXNlIHVzZSB0aGlzIGNodW5rIGluIHlvdXIgb3duIFJTdHVkaW8gaW4gb3JkZXIgdG8NCnByb2R1Y2Ugc29tZSByZXBvcnRzLg0KDQpgYGAgICAgICAgICANCmNhcnNlYXRzICU+JQ0KICB0cmFuc2Zvcm1hdGlvbl9yZXBvcnQodGFyZ2V0ID0gVVMsIG91dHB1dF9mb3JtYXQgPSAiaHRtbCIsIA0KICAgICAgICAgICAgICAgICAgICAgICAgb3V0cHV0X2ZpbGUgPSAidHJhbnNmb3JtYXRpb25fY2Fyc2VhdHMuaHRtbCIpDQpgYGANCg0KIyMjIEV4ZXJjaXNlIDUuDQoNClVzZSBxdWFudGlsZSBhcHByb2FjaCB0byBwZXJmb3JtIGJpbm5pbmcgb2YgdGhlIHZhcmlhYmxlICdJbmNvbWUnICg0DQplcXVhbCBjYXRlZ29yaWVzKS4NCg0KYGBge3IgZXg1fQ0KY2Fyc2VhdHMgJT4lDQogICBtdXRhdGUoSW5jb21lX0NhdGVnb3J5ID0gbnRpbGUoSW5jb21lLCA0KSkNCmBgYA0KDQojIyBBdXRvbWF0ZWQgcmVwb3J0DQoNCmRsb29rciBwcm92aWRlcyB0d28gYXV0b21hdGVkIGRhdGEgdHJhbnNmb3JtYXRpb24gcmVwb3J0czoNCg0KLSAgIFdlYiBwYWdlLWJhc2VkIGR5bmFtaWMgcmVwb3J0cyBjYW4gcGVyZm9ybSBpbi1kZXB0aCBhbmFseXNpcyB0aHJvdWdoDQogICAgdmlzdWFsaXphdGlvbiBhbmQgc3RhdGlzdGljYWwgdGFibGVzLg0KLSAgIFN0YXRpYyByZXBvcnRzIGdlbmVyYXRlZCBhcyBwZGYgZmlsZXMgb3IgaHRtbCBmaWxlcyBjYW4gYmUgYXJjaGl2ZWQNCiAgICBhcyBvdXRwdXQgb2YgZGF0YSBhbmFseXNpcy4NCg0KIyMjIER5bmFtaWMgcmVwb3J0DQoNCmB0cmFuc2Zvcm1hdGlvbl93ZWJfcmVwb3J0KClgIGNyZWF0ZXMgZHluYW1pYyByZXBvcnQgZm9yIG9iamVjdA0KaW5oZXJpdGVkIGZyb20gZGF0YS5mcmFtZShgdGJsX2RmYCwgYHRibGAsIGV0Yykgb3IgZGF0YS5mcmFtZS4NCg0KVGhlIGNvbnRlbnRzIG9mIHRoZSByZXBvcnQgYXJlIGFzIGZvbGxvd3MuOg0KDQotICAgT3ZlcnZpZXcNCiAgICAtICAgRGF0YSBTdHJ1Y3R1cmVzDQogICAgLSAgIERhdGEgVHlwZXMNCiAgICAtICAgSm9iIEluZm9ybWF0aW9ucw0KLSAgIEltcHV0YXRpb24NCiAgICAtICAgTWlzc2luZyBWYWx1ZXMNCiAgICAtICAgT3V0bGllcnMNCi0gICBSZXNvbHZpbmcgU2tld25lc3MNCi0gICBCaW5uaW5nDQotICAgT3B0aW1hbCBCaW5uaW5nDQoNClRoZSBmb2xsb3dpbmcgc2NyaXB0IGNyZWF0ZXMgYSBkYXRhIHRyYW5zZm9ybWF0aW9uIHJlcG9ydCBmb3IgdGhlDQpgdGJsX2RmYCBjbGFzcyBvYmplY3QsIGBoZWFydGZhaWx1cmVgLg0KDQpoZWFydGZhaWx1cmUgJVw+JSB0cmFuc2Zvcm1hdGlvbl93ZWJfcmVwb3J0KHRhcmdldCA9ICJkZWF0aF9ldmVudCIsDQpzdWJ0aXRsZSA9ICJoZWFydGZhaWx1cmUiLG91dHB1dF9kaXIgPSAiLi8iLCBvdXRwdXRfZmlsZSA9DQoidHJhbnNmb3JtYXRpb24uaHRtbCIsIHRoZW1lID0gImJsdWUiKQ0KDQotICAgVGhlIGR5bmFtaWMgY29udGVudHMgb2YgdGhlIHJlcG9ydCBpcyBzaG93biBpbiB0aGUgZm9sbG93aW5nDQogICAgZmlndXJlLjoNCg0KYGBge3IgcmVwb3J0LCBlY2hvID0gRkFMU0UsIG91dC53aWR0aCA9ICI2MCUiLCBmaWcuYWxpZ24gPSAnY2VudGVyJ30NCmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKCJodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20va2ZsaXNpa293c2tpL2RzL21hc3Rlci9yZXAxLmpwZyIpDQpgYGANCg0KIyMjIFN0YXRpYyByZXBvcnQNCg0KYHRyYW5zZm9ybWF0aW9uX3BhZ2VkX3JlcG9ydCgpYCBjcmVhdGUgc3RhdGljIHJlcG9ydCBmb3Igb2JqZWN0DQppbmhlcml0ZWQgZnJvbSBkYXRhLmZyYW1lKGB0YmxfZGZgLCBgdGJsYCwgZXRjKSBvciBkYXRhLmZyYW1lLg0KDQpUaGUgY29udGVudHMgb2YgdGhlIHJlcG9ydCBhcmUgYXMgZm9sbG93cy46DQoNCi0gICBPdmVydmlldw0KICAgIC0gICBEYXRhIFN0cnVjdHVyZXMNCiAgICAtICAgSm9iIEluZm9ybWF0aW9ucw0KLSAgIEltcHV0YXRpb24NCiAgICAtICAgTWlzc2luZyBWYWx1ZXMNCiAgICAtICAgT3V0bGllcnMNCi0gICBSZXNvbHZpbmcgU2tld25lc3MNCi0gICBCaW5uaW5nDQotICAgT3B0aW1hbCBCaW5uaW5nDQoNClRoZSBmb2xsb3dpbmcgc2NyaXB0IGNyZWF0ZXMgYSBkYXRhIHRyYW5zZm9ybWF0aW9uIHJlcG9ydCBmb3IgdGhlDQpgZGF0YS5mcmFtZWAgY2xhc3Mgb2JqZWN0LCBgaGVhcnRmYWlsdXJlYC4NCg0KaGVhcnRmYWlsdXJlICVcPiUgdHJhbnNmb3JtYXRpb25fcGFnZWRfcmVwb3J0KHRhcmdldCA9ICJkZWF0aF9ldmVudCIsDQpzdWJ0aXRsZSA9ICJoZWFydGZhaWx1cmUiLCBvdXRwdXRfZGlyID0gIi4vIiwgb3V0cHV0X2ZpbGUgPQ0KInRyYW5zZm9ybWF0aW9uLnBkZiIsIHRoZW1lID0gImJsdWUiKQ0KDQotICAgVGhlIGNvdmVyIG9mIHRoZSByZXBvcnQgaXMgc2hvd24gaW4gdGhlIGZvbGxvd2luZyBmaWd1cmUuOg0KDQpgYGB7ciByZXBvcnQyLCBlY2hvID0gRkFMU0UsIG91dC53aWR0aCA9ICI2MCUiLCBmaWcuYWxpZ24gPSAnY2VudGVyJ30NCmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKCJodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20va2ZsaXNpa293c2tpL2RzL21hc3Rlci9yZXAyLmpwZyIpDQpgYGANCg0KIyMjIERpYWdub3N0aWMgcmVwb3J0DQoNCipkaWFnbm9zZV9wYWdlZF9yZXBvcnQoKSogY3JlYXRlcyBzdGF0aWMgcmVwb3J0IGZvciBvYmplY3QgaW5oZXJpdGVkDQpmcm9tIGRhdGEuZnJhbWUodGJsX2RmLCB0YmwsIGV0Yykgb3IgZGF0YS5mcmFtZS4NCg0KVGhlIGZvbGxvd2luZyBzY3JpcHQgY3JlYXRlcyBhIHF1YWxpdHkgZGlhZ25vc2lzIHJlcG9ydCBmb3IgdGhlIHRibF9kZg0KY2xhc3Mgb2JqZWN0LCBmbGlnaHRzLg0KDQpmbGlnaHRzICVcPiUgZGlhZ25vc2VfcGFnZWRfcmVwb3J0KHN1YnRpdGxlID0gImZsaWdodHMiLCBvdXRwdXRfZGlyID0NCiIuLyIsIG91dHB1dF9maWxlID0gIkRpYWduLnBkZiIsIHRoZW1lID0gImJsdWUiKQ0KDQpUaGUgY292ZXIgb2YgdGhlIHJlcG9ydCBpcyBzaG93biBpbiB0aGUgZm9sbG93aW5nIGZpZ3VyZS46DQoNCmBgYHtyIHJlcG9ydDMsIGVjaG8gPSBGQUxTRSwgb3V0LndpZHRoID0gIjYwJSIsIGZpZy5hbGlnbiA9ICdjZW50ZXInfQ0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoImh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9rZmxpc2lrb3dza2kvZHMvbWFzdGVyL3JlcDMuanBnIikNCmBgYA0KDQojIyMgRURBIFJlcG9ydA0KDQpkbG9va3IgcHJvdmlkZXMgdHdvIGF1dG9tYXRlZCBFREEgcmVwb3J0czoNCg0KLSAgIFdlYiBwYWdlLWJhc2VkIGR5bmFtaWMgcmVwb3J0cyBjYW4gcGVyZm9ybSBpbi1kZXB0aCBhbmFseXNpcyB0aHJvdWdoDQogICAgdmlzdWFsaXphdGlvbiBhbmQgc3RhdGlzdGljYWwgdGFibGVzLg0KDQotICAgU3RhdGljIHJlcG9ydHMgZ2VuZXJhdGVkIGFzIHBkZiBmaWxlcyBvciBodG1sIGZpbGVzIGNhbiBiZSBhcmNoaXZlZA0KICAgIGFzIG91dHB1dCBvZiBkYXRhIGFuYWx5c2lzLg0KDQplZGFfd2ViX3JlcG9ydCgpIGNyZWF0ZXMgZHluYW1pYyByZXBvcnQgZm9yIG9iamVjdCBpbmhlcml0ZWQgZnJvbQ0KZGF0YS5mcmFtZSh0YmxfZGYsIHRibCwgZXRjKSBvciBkYXRhLmZyYW1lLg0KDQpUaGUgZm9sbG93aW5nIHNjcmlwdCBjcmVhdGVzIGEgRURBIHJlcG9ydCBmb3IgdGhlIGRhdGEuZnJhbWUgY2xhc3MNCm9iamVjdCwgaGVhcnRmYWlsdXJlOg0KDQpoZWFydGZhaWx1cmUgJVw+JSBlZGFfd2ViX3JlcG9ydCh0YXJnZXQgPSAiZGVhdGhfZXZlbnQiLCBzdWJ0aXRsZSA9DQoiaGVhcnRmYWlsdXJlIiwgb3V0cHV0X2RpciA9ICIuLyIsIG91dHB1dF9maWxlID0gIkVEQS5odG1sIiwgdGhlbWUgPQ0KImJsdWUiKQ0KDQpUaGUgZHluYW1pYyBjb250ZW50cyBvZiB0aGUgcmVwb3J0IGlzIHNob3duIGluIHRoZSBmb2xsb3dpbmcgZmlndXJlLjoNCg0KYGBge3IgcmVwb3J0NCwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiNjAlIiwgZmlnLmFsaWduID0gJ2NlbnRlcid9DQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tmbGlzaWtvd3NraS9kcy9tYXN0ZXIvcmVwNC5qcGciKQ0KYGBgDQoNCiMjIyBUcmFuc2Zvcm1hdGlvbiBSZXBvcnQNCg0KZGxvb2tyIHByb3ZpZGVzIHR3byBhdXRvbWF0ZWQgZGF0YSB0cmFuc2Zvcm1hdGlvbiByZXBvcnRzOg0KDQotICAgV2ViIHBhZ2UtYmFzZWQgZHluYW1pYyByZXBvcnRzIGNhbiBwZXJmb3JtIGluLWRlcHRoIGFuYWx5c2lzIHRocm91Z2gNCiAgICB2aXN1YWxpemF0aW9uIGFuZCBzdGF0aXN0aWNhbCB0YWJsZXMuDQotICAgU3RhdGljIHJlcG9ydHMgZ2VuZXJhdGVkIGFzIHBkZiBmaWxlcyBvciBodG1sIGZpbGVzIGNhbiBiZSBhcmNoaXZlZA0KICAgIGFzIG91dHB1dCBvZiBkYXRhIGFuYWx5c2lzLg0KDQp0cmFuc2Zvcm1hdGlvbl93ZWJfcmVwb3J0KCkgY3JlYXRlcyBkeW5hbWljIHJlcG9ydCBmb3Igb2JqZWN0IGluaGVyaXRlZA0KZnJvbSBkYXRhLmZyYW1lKHRibF9kZiwgdGJsLCBldGMpIG9yIGRhdGEuZnJhbWUuDQoNClRoZSBmb2xsb3dpbmcgc2NyaXB0IGNyZWF0ZXMgYSBkYXRhIHRyYW5zZm9ybWF0aW9uIHJlcG9ydCBmb3IgdGhlIHRibF9kZg0KY2xhc3Mgb2JqZWN0LCBoZWFydGZhaWx1cmU6DQoNCmhlYXJ0ZmFpbHVyZSAlXD4lIHRyYW5zZm9ybWF0aW9uX3dlYl9yZXBvcnQodGFyZ2V0ID0gImRlYXRoX2V2ZW50IiwNCnN1YnRpdGxlID0gImhlYXJ0ZmFpbHVyZSIsIG91dHB1dF9kaXIgPSAiLi8iLCBvdXRwdXRfZmlsZSA9DQoidHJhbnNmb3JtYXRpb24uaHRtbCIsIHRoZW1lID0gImJsdWUiKQ0KDQpUaGUgZHluYW1pYyBjb250ZW50cyBvZiB0aGUgcmVwb3J0IGlzIHNob3duIGluIHRoZSBmb2xsb3dpbmcgZmlndXJlLjoNCg0KYGBge3IgcmVwb3J0NSwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiNjAlIiwgZmlnLmFsaWduID0gJ2NlbnRlcid9DQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tmbGlzaWtvd3NraS9kcy9tYXN0ZXIvcmVwNS5qcGciKQ0KYGBgDQoNCiMjIFN1bW1hcnkNCg0KSWYgd2UgYXJlIHRvIHRha2UgYXdheSBvbmUgdGhpbmcgZnJvbSB0aGlzIFRJRFkgV0VFSyBpdCBzaG91bGQgYmUgdGhhdA0KZGVhbGluZyB3aXRoIG1pc3NpbmcgZGF0YSBpcyBub3QgYW4gZWFzeSBwcm9ibGVtLiBEYXRhIHdpbGwgcmFyZWx5IGJlDQptaXNzaW5nIGluIGEgKm5pY2UqIChjb21wbGV0ZWx5IHJhbmRvbSkgd2F5LCBzbyBpZiB3ZSB3YW50IHRvIHJlc29ydCB0bw0KbW9yZSBzaW1wbGUgcmVtb3ZhbCBvciBpbXB1dGF0aW9uIHRlY2huaXF1ZXMsIHdlIG11c3QgcHV0IHJlYXNvbmFibGUNCmVmZm9ydCBpbnRvIGRldGVybWluaW5nIHdoZXRoZXIgb3Igbm90IHRoZSBkZXBlbmRlbmNpZXMgYmV0d2VlbiBvYnNlcnZlZA0KZGF0YSBhbmQgbWlzc2luZ25lc3MgYXJlIGEgY2F1c2UgZm9yIGNvbmNlcm4uIElmIHN0cm9uZyBkZXBlbmRlbmNpZXMNCmV4aXN0IGFuZC9vciB0aGVyZSBpcyBhIGxvdCBvZiBkYXRhIG1pc3NpbmcsIHRoZSBtaXNzaW5nIGRhdGEgcHJvYmxlbQ0KYmVjb21lcyBhIHByZWRpY3Rpb24gcHJvYmxlbSAob3IgYSBzZXJpZXMgb2YgcHJlZGljdGlvbiBwcm9ibGVtcykuIFdlDQpzaG91bGQgYWxzbyBiZSBhd2FyZSBvZiB0aGUgcG9zc2liaWxpdHkgdGhhdCBtaXNzaW5nbmVzcyBkZXBlbmRzIG9uIHRoZQ0KbWlzc2luZyB2YWx1ZSBpbiBhIHdheSB0aGF0IGNhbid0IGJlIGV4cGxhaW5lZCBieSBvYnNlcnZlZCB2YXJpYWJsZXMuDQpUaGlzIGNhbiBhbHNvIGNhdXNlIGJpYXMgaW4gb3VyIGFuYWx5c2VzIGFuZCB3ZSB3aWxsIG5vdCBiZSBhYmxlIHRvDQpkZXRlY3QgaXQgdW5sZXNzIHdlIGdldCBvdXIgaGFuZHMgb24gc29tZSBvZiB0aGUgbWlzc2luZyB2YWx1ZXMuDQoNCk5vdGUgdGhhdCB3ZSBmb2N1c2VkIG9ubHkgb24gc3RhbmRhcmQgY3Jvc3Mtc2VjdGlvbmFsIGRhdGEuIFRoZXJlIGFyZQ0KbWFueSBvdGhlciB0eXBlcyBvZiBkYXRhLCBzdWNoIGFzIHRpbWUtc2VyaWVzIGRhdGEsIHNwYXRpYWwgZGF0YSwNCmltYWdlcywgc291bmQsIGdyYXBocywgZXRjLiBUaGUgYmFzaWMgcHJpbmNpcGxlcyByZW1haW4gdW5jaGFuZ2VkLiBXZQ0KbXVzdCBiZSBhd2FyZSBvZiB0aGUgbWlzc2luZ25lc3MgbWVjaGFuaXNtIGFuZCBpbnRyb2R1Y2luZyBiaWFzLiBXZSBjYW4NCmRlYWwgd2l0aCBtaXNzaW5nIHZhbHVlcyBlaXRoZXIgYnkgcmVtb3Zpbmcgb2JzZXJ2YXRpb25zL3ZhcmlhYmxlcyBvciBieQ0KaW1wdXRpbmcgdGhlbS4gSG93ZXZlciwgZGlmZmVyZW50LCBzb21ldGltZXMgYWRkaXRpb25hbCBtb2RlbHMgYW5kDQp0ZWNobmlxdWVzIHdpbGwgYmUgYXBwcm9wcmlhdGUuIEZvciBleGFtcGxlLCB0ZW1wb3JhbCBhbmQgc3BhdGlhbCBkYXRhDQpsZW5kIHRoZW1zZWx2ZXMgdG8gaW50ZXJwb2xhdGlvbiBvZiBtaXNzaW5nIHZhbHVlcy4NCg0KIyMgRnVydGhlciByZWFkaW5nDQoNCi0gICBBIGdlbnRsZSBpbnRyb2R1Y3Rpb24gZnJvbSBhIHByYWN0aXRpb25lcnMgcGVyc3BlY3RpdmU6IEJsYW5rZXJzLA0KICAgIE0uLCBLb2V0ZXIsIE0uIFcuLCAmIFNjaGlwcGVycywgRy4gTS4gKDIwMTApLiBNaXNzaW5nIGRhdGENCiAgICBhcHByb2FjaGVzIGluIGVIZWFsdGggcmVzZWFyY2g6IHNpbXVsYXRpb24gc3R1ZHkgYW5kIGEgdHV0b3JpYWwgZm9yDQogICAgbm9ubWF0aGVtYXRpY2FsbHkgaW5jbGluZWQgcmVzZWFyY2hlcnMuIEpvdXJuYWwgb2YgbWVkaWNhbCBJbnRlcm5ldA0KICAgIHJlc2VhcmNoLCAxMig1KSwgZTU0Lg0KDQotICAgQSBncmVhdCBib29rIG9uIGJhc2ljIGFuZCBzb21lIGFkdmFuY2UgdGVjaG5pcXVlczogKkFsbGlzb24sIFAuIEQuDQogICAgKDIwMDEpLiBNaXNzaW5nIGRhdGEgKFZvbC4gMTM2KS4gU2FnZSBwdWJsaWNhdGlvbnMuKg0KDQotICAgQW5vdGhlciBncmVhdCBib29rIHdpdGggbWFueSBleGFtcGxlcyBhbmQgY2FzZS1zdHVkaWVzOiAqVmFuDQogICAgQnV1cmVuLCBTLiAoMjAxOCkuIEZsZXhpYmxlIGltcHV0YXRpb24gb2YgbWlzc2luZyBkYXRhLiBDaGFwbWFuIGFuZA0KICAgIEhhbGwvQ1JDLioNCg0KLSAgIFVuZGVyc3RhbmRpbmcgbXVsdGlwbGUgaW1wdXRhdGlvbiB3aXRoIGNoYWluZWQgZXF1YXRpb25zIChpbiBSKToNCiAgICBCdXVyZW4sIFMuIFYuLCAmIEdyb290aHVpcy1PdWRzaG9vcm4sIEsuICgyMDEwKS4gbWljZTogTXVsdGl2YXJpYXRlDQogICAgaW1wdXRhdGlvbiBieSBjaGFpbmVkIGVxdWF0aW9ucyBpbiBSLiBKb3VybmFsIG9mIHN0YXRpc3RpY2FsDQogICAgc29mdHdhcmUsIDEtNjguDQoNCi0gICBXaGVuIGRvaW5nIG1pc3NpbmcgZGF0YSB2YWx1ZSBoYW5kbGluZyBhbmQgcHJlZGljdGlvbiBzZXBhcmF0ZWx5LCB3ZQ0KICAgIHNob3VsZCBiZSBhd2FyZSB0aGF0IGNlcnRhaW4gdHlwZXMgb2YgcHJlZGljdGlvbiBtb2RlbHMgbWlnaHQgd29yaw0KICAgIGJldHRlciB3aXRoIGNlcnRhaW4gbWV0aG9kcyBmb3IgaGFuZGxpbmcgbWlzc2luZyB2YWx1ZXMuIEEgcGFwZXINCiAgICB0aGF0IGlsbHVzdHJhdGUgdGhpczogKllhZGF2LCBNLiBMLiwgJiBSb3ljaG91ZGh1cnksIEIuICgyMDE4KS4NCiAgICBIYW5kbGluZyBtaXNzaW5nIHZhbHVlczogQSBzdHVkeSBvZiBwb3B1bGFyIGltcHV0YXRpb24gcGFja2FnZXMNCiAgICBpbiBSLiBLbm93bGVkZ2UtQmFzZWQgU3lzdGVtcywgMTYwLCAxMDQtMTE4LioNCg0KIyMgTGVhcm5pbmcgb3V0Y29tZXMNCg0KYGBge3IgZmlnNSwgZWNobyA9IEZBTFNFLCBvdXQud2lkdGggPSAiNjAlIiwgZmlnLmFsaWduID0gJ2NlbnRlcid9DQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tmbGlzaWtvd3NraS9kcy9tYXN0ZXIvbW9uc3Rlcl9zdXBwb3J0LnBuZyIpDQpgYGANCg0KRGF0YSBzY2llbmNlIHN0dWRlbnRzIHNob3VsZCB3b3JrIHRvd2FyZHMgb2J0YWluaW5nIHRoZSBrbm93bGVkZ2UgYW5kDQp0aGUgc2tpbGxzIHRoYXQgZW5hYmxlIHRoZW0gdG86DQoNCi0gICBSZXByb2R1Y2UgdGhlIHRlY2huaXF1ZXMgZGVtb25zdHJhdGVkIGluIHRoaXMgY2hhcHRlciB1c2luZyB0aGVpcg0KICAgIGxhbmd1YWdlL3Rvb2wgb2YgY2hvaWNlLg0KLSAgIEFuYWx5emUgdGhlIHNldmVyaXR5IG9mIHRoZSBtaXNzaW5nIGRhdGEgcHJvYmxlbS4NCi0gICBSZWNvZ25pemUgd2hlbiBhIHRlY2huaXF1ZSBpcyBhcHByb3ByaWF0ZSBhbmQgd2hhdCBhcmUgaXRzDQogICAgbGltaXRhdGlvbnMuDQo=