R Notebook
library(dplyr)
library(car)
library(ggplot2)
library(pgirmess)
library(clinfun)source("../data/load_experiment_results.r")Warning: The working directory was changed to C:/Users/Benoit/Insync/UHasselt/Workspaces/RStudio/processdiscoverybenchmark inside a notebook chunk. The working directory will be reset when the chunk is finished running.Data
As we can see below, there are 3472 experiments (= 4 Miners x 2 levels of Frequent Paths x 7 levels of reoccuring tasks x 62 logs). For each experiment, we measured the recall, precision and f1 score as the average based on 10 fold CV.
glimpse(results)Observations: 3,472
Variables: 8
$ infrequentpaths <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALSE...
$ reoccuringtasks <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
$ tree <int> 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 6...
$ miner <fctr> Alpha +, Heuristics, ILP, Inductive, Alpha +, Heuristics, IL...
$ coverability <dbl> 9.009003, 9.009003, 9.009003, 9.009003, 9.009003, 9.009003, 9...
$ avg_recall <dbl> 1.0000000, 0.0000000, 1.0000000, 0.5502521, 0.9000000, 0.6848...
$ avg_precision <dbl> 0.4992754, 0.0000000, 1.0000000, 1.0000000, 0.4455056, 0.4458...
$ avg_f1 <dbl> 0.6660194, 0.0000000, 1.0000000, 0.7073211, 0.5959900, 0.5392...The three plots below show the marginal distribution of recall, precision, f1.
results %>%
ggplot(aes(x=avg_recall)) +
geom_histogram(binwidth=0.01) +
ggtitle("Histogram of Recall Values")
The Recall values appear to be U-shaped distributed, with high peaks at values 0 and 1
results %>%
ggplot(aes(x=avg_precision)) +
geom_histogram(binwidth=0.01) +
ggtitle("Histogram of Precision Values")
The Precision Values have three peaks (modi), at values 0, 0.5 and 1.
results %>%
ggplot(aes(x=avg_f1)) +
geom_histogram(binwidth=0.01) +
ggtitle("Histogram of F1 Values")
F1 values are the avg between recall and precision. Consequently, we see the three modi pattern of precision reappearing.
It is clear that the three metrics are not normally distributed.
Now lets do some statistical analysis and see where we get. We start with analysing recall and apply one of the simplest analysis, a one-way ANOVA and select miner as the independent variable. The other variables are considered part of the error term.
Recall
Let’s start with recall. First we will verify if there are significant differences between the models generated by four different miners in terms of recall value.
Differences between models generated by different miners
First we check the homogeneity assumption. The Levene’s test shows that this assumption is not met and we should use Welch’s F-ratio
leveneTest(results$avg_recall, results$miner, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 3 352.35 < 2.2e-16 ***
3468
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1To check the normality assumption, we create an anova model and evaluate the residuals.
model1 <- aov(avg_recall ~ miner, data=results)
plot(model1, which=2)
The plot clearly illustrates that the normality assumption is not met either. So, it is best to abandon the ANOVA analysis and apply the non-parametric variant Kruskal-Wallis test. The idea behind the KW test is to rank all the observations and then test whether the average rank differs between the groups.
Therefore, we start by ranking the discovered models based on their recall value and compare the average ranking per miner.
print("Average Rank per Miner")[1] "Average Rank per Miner"results$rank_recall <- rank(-results$avg_recall)
by(results$rank_recall, results$miner, mean)results$miner: Alpha +
[1] 1974.647
-------------------------------------------------------------------
results$miner: Heuristics
[1] 1810.96
-------------------------------------------------------------------
results$miner: ILP
[1] 951.0363
-------------------------------------------------------------------
results$miner: Inductive
[1] 2209.357Based on the average (recall) ranking, these results show that ILP creates the best models, followed by Heuristics, Alpha+ and Inductive (in that order). To get an idea of the distribution of the ranks, we create some violin plots. These plots show the the density curve, mirrored around a central vertical axis, together with three horizontal lines which represent the 25%, 50% and 75% percentile. A violin plot is somehow related to a boxplot, but more suitable when the distribution of the data is multimodal.
results %>%
ggplot(aes(x= miner ,y =rank_recall)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on recall)")
These plots reveal for example that 75% of the models discovered by ILP have a higher recall value (i.e. are ranked higher) than 75% of the models discovered by Inductive miner en 50% of the models discovered by Heuristics and Alpha+.
Note, that these distributions do not tell us anything about the absolute differences in recall values. Below, one can find violin plots for the distribution of the actual recall values, which clearly show a different picture. Note that our analysis are based on the ranking of discovered models rather than the actual quaility of the models and that we will show these plots based on actual values merely for the purpose of illustration. All statistical claims will be about the ranking of the models.
results %>%
ggplot(aes(x= miner ,y =avg_recall)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of recall values")
Based on the above analysis, it appears that the four miners generate models which rank on average differently (based on recall values). We will use a Kruskal-Wallis test to see if these differences in average ranking are statistically significant.
kruskal.test(avg_recall ~ miner, results)
Kruskal-Wallis rank sum test
data: avg_recall by miner
Kruskal-Wallis chi-squared = 852.25, df = 3, p-value < 2.2e-16These results show that there are significant ranking differences between the models created by the four miners in terms of recall. The order suggested by the plot is ILP > Heuristics > Alpha+ > Inductive. We can use a post-hoc test to see which of these differences are significant.
kruskalmc(avg_recall ~ miner, results)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
Alpha +-Heuristics 163.6878 126.9474 TRUE
Alpha +-ILP 1023.6112 126.9474 TRUE
Alpha +-Inductive 234.7091 126.9474 TRUE
Heuristics-ILP 859.9234 126.9474 TRUE
Heuristics-Inductive 398.3969 126.9474 TRUE
ILP-Inductive 1258.3203 126.9474 TRUEIt appears that all pairwise comparisons are statisically significant different from each other. Thus, based on recall values, models generated by ILP rank on average better than models generated by the Heuristics Miner, which rank on average better than models generated by the Alpha+ Miner, which subsequently rank better on average than models generated by the Inductive Miner.
Impact of Infrequent Behavior on performance of the Mining algoritms
Next, we want to test whether the presence/absence of infrequent behavior in the log has an impact on the average ranking based on recall values. For this we will split the data between experiments with infrequent behavior and experiments without infrequent behavior and verify the ranking for these two subsets. We will start the analysis for experiments where infrequent behavior was absent.
results_freq <- results %>%
filter(infrequentpaths==F)
results_infreq <- results %>%
filter(infrequentpaths==T)Absence of Infrequent Behavior
First we will test to see if we can apply a standard ANOVA analysis
leveneTest(results_freq$avg_recall, results_freq$miner, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 3 387.29 < 2.2e-16 ***
1732
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_recall ~ miner, data=results_freq)
plot(model1, which=2)
Both the Levene’s test as the QQ-plot reveal that both the assumption of Homogeneity and Normality are violated. Therefore we will focus on the comparison of average rankings.
print("Average Rank per Miner")[1] "Average Rank per Miner"results_freq$rank_recall <- rank(-results_freq$avg_recall)
by(results_freq$rank_recall, results_freq$miner, mean)results_freq$miner: Alpha +
[1] 1008.922
-------------------------------------------------------------------
results_freq$miner: Heuristics
[1] 938.4712
-------------------------------------------------------------------
results_freq$miner: ILP
[1] 481.4194
-------------------------------------------------------------------
results_freq$miner: Inductive
[1] 1045.188results_freq %>%
ggplot(aes(x= miner ,y =rank_recall)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on recall)")
These analysis show a similar ranking between the four miners. However, it appears that in case of no infrequent behavior in the log, ILP outperforms the other algorithms even more. 75% of the models generated by ILP are better than 75% of the models generated by Alpha+ en Heuristics and all the models generated by ILP are better than 75% of the models generated by Inductive Miner.
We will use a Kruskal-Wallis test to see if these differences in average ranking are statistically significant.
kruskal.test(avg_recall ~ miner, results_freq)
Kruskal-Wallis rank sum test
data: avg_recall by miner
Kruskal-Wallis chi-squared = 407.62, df = 3, p-value < 2.2e-16These results show that there are significant ranking differences between the models created by the four miners in terms of recall. The order suggested by the plot is ILP > Heuristics > Alpha+ > Inductive. We can use a post-hoc test to see which of these differences are significant.
kruskalmc(avg_recall ~ miner, results_freq)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
Alpha +-Heuristics 70.45046 89.77831 FALSE
Alpha +-ILP 527.50230 89.77831 TRUE
Alpha +-Inductive 36.26613 89.77831 FALSE
Heuristics-ILP 457.05184 89.77831 TRUE
Heuristics-Inductive 106.71659 89.77831 TRUE
ILP-Inductive 563.76843 89.77831 TRUEIt appears that all pairwise comparisons are statisically significant different from each other, except for the differences between the models generated by Alpha+ on the one hand and the models generated by Inductive and Heuristics.
Presence of Infrequent Behavior
First we will test to see if we can apply a standard ANOVA analysis
leveneTest(results_infreq$avg_recall, results_infreq$miner, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 3 138.85 < 2.2e-16 ***
1732
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_recall ~ miner, data=results_infreq)
plot(model1, which=2)
Both the Levene’s test as the QQ-plot reveal that both the assumption of Homogeneity and Normality are violated. Therefore we will focus on the comparison of average rankings.
print("Average Rank per Miner")[1] "Average Rank per Miner"results_infreq$rank_recall <- rank(-results_infreq$avg_recall)
by(results_infreq$rank_recall, results_infreq$miner, mean)results_infreq$miner: Alpha +
[1] 964.6763
-------------------------------------------------------------------
results_infreq$miner: Heuristics
[1] 867.9965
-------------------------------------------------------------------
results_infreq$miner: ILP
[1] 465.9389
-------------------------------------------------------------------
results_infreq$miner: Inductive
[1] 1175.388results_infreq %>%
ggplot(aes(x= miner ,y =rank_recall)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on recall)")
These analysis show again a similar ranking between the four miners. However, it appears that the differences between ILP on the one hand and Alpha+ and Heuristics on the other hand are less extreme than in the case of logs without infrequent behavior. Yet, it is still clear that ILP outperforms the other algorithms in terms of recall.
We will use a Kruskal-Wallis test to see if these differences in average ranking are statistically significant.
kruskal.test(avg_recall ~ miner, results_infreq)
Kruskal-Wallis rank sum test
data: avg_recall by miner
Kruskal-Wallis chi-squared = 484.1, df = 3, p-value < 2.2e-16These results show that there are significant ranking differences between the models created by the four miners in terms of recall. The order suggested by the plot is ILP > Heuristics > Alpha+ > Inductive. We can use a post-hoc test to see which of these differences are significant.
kruskalmc(avg_recall ~ miner, results_infreq)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
Alpha +-Heuristics 96.67972 89.77831 TRUE
Alpha +-ILP 498.73733 89.77831 TRUE
Alpha +-Inductive 210.71198 89.77831 TRUE
Heuristics-ILP 402.05760 89.77831 TRUE
Heuristics-Inductive 307.39171 89.77831 TRUE
ILP-Inductive 709.44931 89.77831 TRUEIt appears that all pairwise comparisons are statisically significant different from each other.
Impact of amount of reoccuring tasks on performance of the Mining Algorithm
Next, we will investigate how the performance each of the mining algorithm is influenced by the level of reoccuring tasks in the log files.
results_alpha <- results %>%
filter(miner=="Alpha +")%>%
mutate(reoccuringtasks = as.factor(reoccuringtasks))
results_heuristics <- results %>%
filter(miner=="Heuristics")%>%
mutate(reoccuringtasks = as.factor(reoccuringtasks))
results_ilp <- results %>%
filter(miner=="ILP")%>%
mutate(reoccuringtasks = as.factor(reoccuringtasks))
results_inductive <- results %>%
filter(miner=="Inductive")%>%
mutate(reoccuringtasks = as.factor(reoccuringtasks))Alpha+
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_alpha$avg_recall, results_alpha$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 5.7921 6.298e-06 ***
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_recall ~ reoccuringtasks, data= results_alpha)
plot(model1, which=2)
Both the assumptions of normality and homogeneity appear to be violated, thus we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_alpha$rank_recall <- rank(-results_alpha$avg_recall)
by(results_alpha$rank_recall, results_alpha$reoccuringtasks, mean)results_alpha$reoccuringtasks: 0
[1] 354.6653
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.05
[1] 386.4556
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.1
[1] 414.3145
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.15
[1] 436.9718
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.2
[1] 449.3226
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.25
[1] 464.9355
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.3
[1] 534.8347results_alpha %>%
ggplot(aes(x= reoccuringtasks ,y =rank_recall)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on recall)")
These data seem to suggest that as the level of reoccuring tasks increases, the models generated by Alpha miner deteriorate. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test. The latter test whether there is a significant trend within the data (across increasing levels of reoccuringtasks)
kruskal.test(avg_recall~ reoccuringtasks, results_alpha)
Kruskal-Wallis rank sum test
data: avg_recall by reoccuringtasks
Kruskal-Wallis chi-squared = 43.546, df = 6, p-value = 9.094e-08jonckheere.test(results_alpha$avg_recall, as.numeric(results_alpha$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 135510, p-value = 0.002
alternative hypothesis: two.sidedThese analyses show that there is statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. Let’s verify this further by comparing different levels pairwise.
kruskalmc(avg_recall ~reoccuringtasks, data=results_alpha)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
0-0.05 31.79032 96.73456 FALSE
0-0.1 59.64919 96.73456 FALSE
0-0.15 82.30645 96.73456 FALSE
0-0.2 94.65726 96.73456 FALSE
0-0.25 110.27016 96.73456 TRUE
0-0.3 180.16935 96.73456 TRUE
0.05-0.1 27.85887 96.73456 FALSE
0.05-0.15 50.51613 96.73456 FALSE
0.05-0.2 62.86694 96.73456 FALSE
0.05-0.25 78.47984 96.73456 FALSE
0.05-0.3 148.37903 96.73456 TRUE
0.1-0.15 22.65726 96.73456 FALSE
0.1-0.2 35.00806 96.73456 FALSE
0.1-0.25 50.62097 96.73456 FALSE
0.1-0.3 120.52016 96.73456 TRUE
0.15-0.2 12.35081 96.73456 FALSE
0.15-0.25 27.96371 96.73456 FALSE
0.15-0.3 97.86290 96.73456 TRUE
0.2-0.25 15.61290 96.73456 FALSE
0.2-0.3 85.51210 96.73456 FALSE
0.25-0.3 69.89919 96.73456 FALSEWhile the previous analysis show that there is a statisically significant trend, the pairwise comparisons do not provide a clear picture how this trend looks like, with many comparisons statistically insignificant. Note however, that this does not prove that there is no difference, but could also be caused by a lack of power. The Multiple comparison test incorporates some kind of Bonferroni correction to correct for the fact that we are performing many tests simultaneously, which often errs on the conservative side.
Heuristics
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_heuristics$avg_recall, results_heuristics$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 3.2421 0.003715 **
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_recall ~ reoccuringtasks, data= results_heuristics)
plot(model1, which=2)
Both the assumptions of normality and homogeneity appear to be violated, thus we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_heuristics$rank_recall <- rank(-results_heuristics$avg_recall)
by(results_heuristics$rank_recall, results_heuristics$reoccuringtasks, mean)results_heuristics$reoccuringtasks: 0
[1] 363.1331
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.05
[1] 431.5726
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.1
[1] 436.5242
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.15
[1] 475.2702
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.2
[1] 439.9153
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.25
[1] 438.6169
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.3
[1] 456.4677results_heuristics %>%
ggplot(aes(x= reoccuringtasks ,y =rank_recall)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on recall)")
These data seem to suggest that as we move from no reoccuring tasks to some reoccuring tasks, the models generated by Heuristics miner deteriorate some. However, as the level of reocurring tasks increase, there appears to be no clear trend in the further deterioration of the relative quality of the models. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test.
kruskal.test(avg_recall~ reoccuringtasks, results_heuristics)
Kruskal-Wallis rank sum test
data: avg_recall by reoccuringtasks
Kruskal-Wallis chi-squared = 15.509, df = 6, p-value = 0.01665jonckheere.test(results_heuristics$avg_recall, as.numeric(results_heuristics$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 150830, p-value = 0.008
alternative hypothesis: two.sidedThese analyses show that there is indeed a statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. Let’s verify this further by comparing different levels pairwise.
kruskalmc(avg_recall ~reoccuringtasks, data=results_heuristics)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
0-0.05 68.439516 96.73456 FALSE
0-0.1 73.391129 96.73456 FALSE
0-0.15 112.137097 96.73456 TRUE
0-0.2 76.782258 96.73456 FALSE
0-0.25 75.483871 96.73456 FALSE
0-0.3 93.334677 96.73456 FALSE
0.05-0.1 4.951613 96.73456 FALSE
0.05-0.15 43.697581 96.73456 FALSE
0.05-0.2 8.342742 96.73456 FALSE
0.05-0.25 7.044355 96.73456 FALSE
0.05-0.3 24.895161 96.73456 FALSE
0.1-0.15 38.745968 96.73456 FALSE
0.1-0.2 3.391129 96.73456 FALSE
0.1-0.25 2.092742 96.73456 FALSE
0.1-0.3 19.943548 96.73456 FALSE
0.15-0.2 35.354839 96.73456 FALSE
0.15-0.25 36.653226 96.73456 FALSE
0.15-0.3 18.802419 96.73456 FALSE
0.2-0.25 1.298387 96.73456 FALSE
0.2-0.3 16.552419 96.73456 FALSE
0.25-0.3 17.850806 96.73456 FALSEThese results seem to confirm our impression that Heuristics Miner is sensitive for the presence of reoccuring tasks, but does not seem to produce models of decreasing relative quality as the level of reoccuring tasks increases.
ILP
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_ilp$avg_recall, results_ilp$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 0.8767 0.5114
861 model1 <- aov(avg_recall ~ reoccuringtasks, data= results_ilp)
plot(model1, which=2)
Both the assumptions of normality and homogeneity appear to be violated, thus we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_ilp$rank_recall <- rank(-results_ilp$avg_recall)
by(results_ilp$rank_recall, results_ilp$reoccuringtasks, mean)results_ilp$reoccuringtasks: 0
[1] 457.2177
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.05
[1] 444.5282
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.1
[1] 423.9798
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.15
[1] 435.1048
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.2
[1] 424.1008
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.25
[1] 434.8306
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.3
[1] 421.7379results_ilp %>%
ggplot(aes(x= reoccuringtasks ,y =rank_recall)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on recall)")
These data seem to suggest that ILP is not sensitive to the level of reoccuring tasks. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test. The latter test whether there is a significant trend within the data (across increasing levels of reoccuringtasks)
kruskal.test(avg_recall~ reoccuringtasks, results_ilp)
Kruskal-Wallis rank sum test
data: avg_recall by reoccuringtasks
Kruskal-Wallis chi-squared = 3.7951, df = 6, p-value = 0.7044jonckheere.test(results_ilp$avg_recall, as.numeric(results_ilp$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 165880, p-value = 0.106
alternative hypothesis: two.sidedThese analyses confirm our impression and show that there is no statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. The quality of models generated by ILP miner is insensitive to the level of reoccuring tasks, when measured in terms of recall values.
Inductive
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_inductive$avg_recall, results_inductive$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 23.723 < 2.2e-16 ***
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_recall ~ reoccuringtasks, data= results_inductive)
plot(model1, which=2)
Both the assumptions of normality and homogeneity appear to be violated, thus we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_inductive$rank_recall <- rank(-results_inductive$avg_recall)
by(results_inductive$rank_recall, results_inductive$reoccuringtasks, mean)results_inductive$reoccuringtasks: 0
[1] 245.2298
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.05
[1] 299.379
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.1
[1] 408.1935
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.15
[1] 497.879
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.2
[1] 518.125
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.25
[1] 496.5806
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.3
[1] 576.1129results_inductive %>%
ggplot(aes(x= reoccuringtasks ,y =rank_recall)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on recall)")
These data seem to suggest that as the level of reoccuring tasks increases, the models generated by Inductive miner deteriorate, although the effect seems to level of as we reach higher levels of reoccuring tasks. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test. The latter test whether there is a significant trend within the data (across increasing levels of reoccuringtasks)
kruskal.test(avg_recall~ reoccuringtasks, results_inductive)
Kruskal-Wallis rank sum test
data: avg_recall by reoccuringtasks
Kruskal-Wallis chi-squared = 178.13, df = 6, p-value < 2.2e-16jonckheere.test(results_inductive$avg_recall, as.numeric(results_inductive$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 107610, p-value = 0.002
alternative hypothesis: two.sidedThese analyses show that there is statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. Let’s verify this further by comparing different levels pairwise.
kruskalmc(avg_recall ~reoccuringtasks, data=results_inductive)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
0-0.05 54.149194 96.73456 FALSE
0-0.1 162.963710 96.73456 TRUE
0-0.15 252.649194 96.73456 TRUE
0-0.2 272.895161 96.73456 TRUE
0-0.25 251.350806 96.73456 TRUE
0-0.3 330.883065 96.73456 TRUE
0.05-0.1 108.814516 96.73456 TRUE
0.05-0.15 198.500000 96.73456 TRUE
0.05-0.2 218.745968 96.73456 TRUE
0.05-0.25 197.201613 96.73456 TRUE
0.05-0.3 276.733871 96.73456 TRUE
0.1-0.15 89.685484 96.73456 FALSE
0.1-0.2 109.931452 96.73456 TRUE
0.1-0.25 88.387097 96.73456 FALSE
0.1-0.3 167.919355 96.73456 TRUE
0.15-0.2 20.245968 96.73456 FALSE
0.15-0.25 1.298387 96.73456 FALSE
0.15-0.3 78.233871 96.73456 FALSE
0.2-0.25 21.544355 96.73456 FALSE
0.2-0.3 57.987903 96.73456 FALSE
0.25-0.3 79.532258 96.73456 FALSEThese results seem to confirm our impression. Inductive Miner is sensitive to levels for reoccuring tasks, resulting in models of lower quality (measured in terms of recall values). However, from a level of around 15% reoccuring tasks in the log, this effect seems to have reached a plateau and stays stable.
Precision
Let’s continue with precision. First we will verify if there are significant differences between the models generated by four different miners in terms of recall value.
Differences between models generated by different miners
First we check the homogeneity the normality assumption.
leveneTest(results$avg_precision, results$miner, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 3 103 < 2.2e-16 ***
3468
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_precision ~ miner, data=results)
plot(model1, which=2)
The analysis illustrate that both assumptions are violated, so we proceed by focusing on the relative differences and the ranks of the generated models.
Therefore, we start by ranking the discovered models based on their precision value and compare the average ranking per miner.
print("Average Rank per Miner")[1] "Average Rank per Miner"results$rank_precision <- rank(-results$avg_precision)
by(results$rank_precision, results$miner, mean)results$miner: Alpha +
[1] 2184.514
-------------------------------------------------------------------
results$miner: Heuristics
[1] 2422.416
-------------------------------------------------------------------
results$miner: ILP
[1] 1002.904
-------------------------------------------------------------------
results$miner: Inductive
[1] 1336.165Based on the average (precision) ranking, these results show again ILP creates the best models. However, the order between the other three miners is now slightly different. Based on the precision metric, Inductive miner produces relatively better models than Alpha+ which seems to outperform Heuristics miner. To get an idea of the distribution of the ranks, we create some violin plots.
results %>%
ggplot(aes(x= miner ,y =rank_precision)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on precision)")
These plots reveal a clear distinction between the models generated by ILP and Inductive Miner on the one hand and Alpha+ and Heuristics Miner on the other hand.
Note, that these distributions focus on the relative differences between the quality of the models, not the absolute differences. For purpose of illustration, one can find a violin plot below which shows the distribution of the actual precision values (instead of the ranks).
results %>%
ggplot(aes(x= miner ,y =avg_precision)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of precision values")
We will now continue focusing on the relative differences instead. Based on the above analysis, it appears that the four miners generate models which rank on average differently (based on recall values). We will use a Kruskal-Wallis test to see if these differences in average ranking are statistically significant.
kruskal.test(avg_precision ~ miner, results)
Kruskal-Wallis rank sum test
data: avg_precision by miner
Kruskal-Wallis chi-squared = 1191.9, df = 3, p-value < 2.2e-16These results show that there precision. The order suggested by the plot is ILP > Inductive > Alpha+ > Heuristics. We can use a post-hoc test to see which of these differences are significant.
kruskalmc(avg_precision ~ miner, results)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
Alpha +-Heuristics 237.9021 126.9474 TRUE
Alpha +-ILP 1181.6106 126.9474 TRUE
Alpha +-Inductive 848.3491 126.9474 TRUE
Heuristics-ILP 1419.5127 126.9474 TRUE
Heuristics-Inductive 1086.2512 126.9474 TRUE
ILP-Inductive 333.2615 126.9474 TRUEIt appears that all pairwise comparisons are statisically significant different from each other. Thus, based on precision values, models generated by ILP rank on average better than models generated by the ILP Miner, which rank on average better than models generated by the Alpha+ Miner, which subsequently rank better on average than models generated by the Heuristics Miner.
Impact of Infrequent Behavior on performance of the Mining algoritms
Next, we want to test whether the presence/absence of infrequent behavior in the log has an impact on the average ranking based on precision values. For this we will split the data between experiments with infrequent behavior and experiments without infrequent behavior and verify the ranking for these two subsets. We will start the analysis for experiments where infrequent behavior was absent.
Absence of Infrequent Behavior
First we will test to see if we can apply a standard ANOVA analysis
leveneTest(results_freq$avg_precision, results_freq$miner, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 3 57.78 < 2.2e-16 ***
1732
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_precision ~ miner, data=results_freq)
plot(model1, which=2)
Both the Levene’s test as the QQ-plot reveal that both the assumption of Homogeneity and Normality are violated. Therefore we will focus on the comparison of average rankings.
print("Average Rank per Miner")[1] "Average Rank per Miner"results_freq$rank_precision <- rank(-results_freq$avg_precision)
by(results_freq$rank_precision, results_freq$miner, mean)results_freq$miner: Alpha +
[1] 1091.543
-------------------------------------------------------------------
results_freq$miner: Heuristics
[1] 1246.615
-------------------------------------------------------------------
results_freq$miner: ILP
[1] 508.985
-------------------------------------------------------------------
results_freq$miner: Inductive
[1] 626.8571results_freq %>%
ggplot(aes(x= miner ,y =rank_precision)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on precision)")
These analysis show a similar ranking between the four miners.
We will use a Kruskal-Wallis test to see if these differences in average ranking are statistically significant.
kruskal.test(avg_precision ~ miner, results_freq)
Kruskal-Wallis rank sum test
data: avg_precision by miner
Kruskal-Wallis chi-squared = 661.04, df = 3, p-value < 2.2e-16These results show that there are significant ranking differences between the models created by the four miners in terms of precision. The order suggested by the plot is ILP > Inductive > Alpha+ > Heuristics. We can use a post-hoc test to see which of these differences are significant.
kruskalmc(avg_precision ~ miner, results_freq)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
Alpha +-Heuristics 155.0726 89.77831 TRUE
Alpha +-ILP 582.5576 89.77831 TRUE
Alpha +-Inductive 464.6855 89.77831 TRUE
Heuristics-ILP 737.6302 89.77831 TRUE
Heuristics-Inductive 619.7581 89.77831 TRUE
ILP-Inductive 117.8721 89.77831 TRUEIt appears that all pairwise comparisons are statisically significant different from each other.
Presence of Infrequent Behavior
First we will test to see if we can apply a standard ANOVA analysis
leveneTest(results_infreq$avg_precision, results_infreq$miner, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 3 56.602 < 2.2e-16 ***
1732
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_precision ~ miner, data=results_infreq)
plot(model1, which=2)
Both the Levene’s test as the QQ-plot reveal that both the assumption of Homogeneity and Normality are violated. Therefore we will focus on the comparison of average rankings.
print("Average Rank per Miner")[1] "Average Rank per Miner"results_infreq$rank_precision <- rank(-results_infreq$avg_precision)
by(results_infreq$rank_precision, results_infreq$miner, mean)results_infreq$miner: Alpha +
[1] 1091.851
-------------------------------------------------------------------
results_infreq$miner: Heuristics
[1] 1177.096
-------------------------------------------------------------------
results_infreq$miner: ILP
[1] 495.6532
-------------------------------------------------------------------
results_infreq$miner: Inductive
[1] 709.3998results_infreq %>%
ggplot(aes(x= miner ,y =rank_precision)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on precision)")
These analysis show again a similar ranking between the four miners.
We will use a Kruskal-Wallis test to see if these differences in average ranking are statistically significant.
kruskal.test(avg_precision ~ miner, results_infreq)
Kruskal-Wallis rank sum test
data: avg_precision by miner
Kruskal-Wallis chi-squared = 539.13, df = 3, p-value < 2.2e-16These results show that there are significant ranking differences between the models created by the four miners in terms of precision. The order suggested by the plot is ILP > Inductive > Alpha+ > Heuristics. We can use a post-hoc test to see which of these differences are significant.
kruskalmc(avg_precision ~ miner, results_infreq)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
Alpha +-Heuristics 85.24424 89.77831 FALSE
Alpha +-ILP 596.19816 89.77831 TRUE
Alpha +-Inductive 382.45161 89.77831 TRUE
Heuristics-ILP 681.44240 89.77831 TRUE
Heuristics-Inductive 467.69585 89.77831 TRUE
ILP-Inductive 213.74654 89.77831 TRUEIt appears that all pairwise comparisons are statisically significant different from each other, except for the difference between Alpha+ and Heuristics.
Impact of amount of reoccuring tasks on performance of the Mining Algorithm
Next, we will investigate how the performance each of the mining algorithm is influenced by the level of reoccuring tasks in the log files.
Alpha+
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_alpha$avg_precision, results_alpha$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 0.3516 0.9091
861 model1 <- aov(avg_recall ~ reoccuringtasks, data= results_alpha)
plot(model1, which=2)
While the assumption of homogneity seems to hold, the assumptions of normality appears to be violated. To remain consistent, we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_alpha$rank_precision <- rank(-results_alpha$avg_precision)
by(results_alpha$rank_precision, results_alpha$reoccuringtasks, mean)results_alpha$reoccuringtasks: 0
[1] 319.0242
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.05
[1] 346.6169
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.1
[1] 394.3508
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.15
[1] 435.4879
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.2
[1] 483.2177
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.25
[1] 506.5887
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.3
[1] 556.2137results_alpha %>%
ggplot(aes(x= reoccuringtasks ,y =rank_precision)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on precision)")
These data seem to suggest that as the level of reoccuring tasks increases, the models generated by Alpha miner deteriorate. However, it seems that this effect really starts to show from a level of 15% reoccuring tasks. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test.
kruskal.test(avg_precision~ reoccuringtasks, results_alpha)
Kruskal-Wallis rank sum test
data: avg_precision by reoccuringtasks
Kruskal-Wallis chi-squared = 90.762, df = 6, p-value < 2.2e-16jonckheere.test(results_alpha$avg_precision, as.numeric(results_alpha$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 121400, p-value = 0.002
alternative hypothesis: two.sidedThese analyses show that there is statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. Let’s verify this further by comparing different levels pairwise.
kruskalmc(avg_precision ~reoccuringtasks, data=results_alpha)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
0-0.05 27.59274 96.73456 FALSE
0-0.1 75.32661 96.73456 FALSE
0-0.15 116.46371 96.73456 TRUE
0-0.2 164.19355 96.73456 TRUE
0-0.25 187.56452 96.73456 TRUE
0-0.3 237.18952 96.73456 TRUE
0.05-0.1 47.73387 96.73456 FALSE
0.05-0.15 88.87097 96.73456 FALSE
0.05-0.2 136.60081 96.73456 TRUE
0.05-0.25 159.97177 96.73456 TRUE
0.05-0.3 209.59677 96.73456 TRUE
0.1-0.15 41.13710 96.73456 FALSE
0.1-0.2 88.86694 96.73456 FALSE
0.1-0.25 112.23790 96.73456 TRUE
0.1-0.3 161.86290 96.73456 TRUE
0.15-0.2 47.72984 96.73456 FALSE
0.15-0.25 71.10081 96.73456 FALSE
0.15-0.3 120.72581 96.73456 TRUE
0.2-0.25 23.37097 96.73456 FALSE
0.2-0.3 72.99597 96.73456 FALSE
0.25-0.3 49.62500 96.73456 FALSEThese results illustrate that the quality of the models decrease significantly whenever the level of reoccuring tasks increase by 15%.
Heuristics
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_heuristics$avg_precision, results_heuristics$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 1.8471 0.08723 .
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_precision ~ reoccuringtasks, data= results_heuristics)
plot(model1, which=2)
Again, the Levene’s test is not conclusive about the violation of the homogeneity assumption, but the QQ-plot shows that the reisudals are not normally distribution. Therefore, we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_heuristics$rank_precision <- rank(-results_heuristics$avg_precision)
by(results_heuristics$rank_precision, results_heuristics$reoccuringtasks, mean)results_heuristics$reoccuringtasks: 0
[1] 398.6694
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.05
[1] 416.375
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.1
[1] 449.6492
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.15
[1] 473.2984
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.2
[1] 434.3911
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.25
[1] 443.6331
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.3
[1] 425.4839results_heuristics %>%
ggplot(aes(x= reoccuringtasks ,y =rank_precision)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on precision)")
These data seem to suggest that the precision score of models discovered by the Heuristics Miner is relatively stable against increasing levels of reoccuring tasks. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test.
kruskal.test(avg_precision~ reoccuringtasks, results_heuristics)
Kruskal-Wallis rank sum test
data: avg_precision by reoccuringtasks
Kruskal-Wallis chi-squared = 6.9349, df = 6, p-value = 0.3269jonckheere.test(results_heuristics$avg_precision, as.numeric(results_heuristics$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 157040, p-value = 0.262
alternative hypothesis: two.sidedThese analyses show that there is indeed no statistically significant trend in the relative quality of the generated models as the amount of reoccuring tasks increases.
ILP
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_ilp$avg_precision, results_ilp$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 29.985 < 2.2e-16 ***
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_precision ~ reoccuringtasks, data= results_ilp)
plot(model1, which=2)
Both the assumptions of normality and homogeneity appear to be violated, thus we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_ilp$rank_precision <- rank(-results_ilp$avg_precision)
by(results_ilp$rank_precision, results_ilp$reoccuringtasks, mean)results_ilp$reoccuringtasks: 0
[1] 154.9677
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.05
[1] 296.2298
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.1
[1] 413.1613
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.15
[1] 441.6976
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.2
[1] 521.246
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.25
[1] 599.3185
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.3
[1] 614.879results_ilp %>%
ggplot(aes(x= reoccuringtasks ,y =rank_precision)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on Precision)")
These data seem to suggest that the precision score of the models generated by ILP is very sensitive to the level of reoccuring tasks. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test.
kruskal.test(avg_precision~ reoccuringtasks, results_ilp)
Kruskal-Wallis rank sum test
data: avg_precision by reoccuringtasks
Kruskal-Wallis chi-squared = 330.72, df = 6, p-value < 2.2e-16jonckheere.test(results_ilp$avg_precision, as.numeric(results_ilp$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 83914, p-value = 0.002
alternative hypothesis: two.sidedThese analyses show that there is a statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. Let’s verify this further by comparing different levels pairwise.
kruskalmc(avg_precision ~reoccuringtasks, data=results_ilp)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
0-0.05 141.26210 96.73456 TRUE
0-0.1 258.19355 96.73456 TRUE
0-0.15 286.72984 96.73456 TRUE
0-0.2 366.27823 96.73456 TRUE
0-0.25 444.35081 96.73456 TRUE
0-0.3 459.91129 96.73456 TRUE
0.05-0.1 116.93145 96.73456 TRUE
0.05-0.15 145.46774 96.73456 TRUE
0.05-0.2 225.01613 96.73456 TRUE
0.05-0.25 303.08871 96.73456 TRUE
0.05-0.3 318.64919 96.73456 TRUE
0.1-0.15 28.53629 96.73456 FALSE
0.1-0.2 108.08468 96.73456 TRUE
0.1-0.25 186.15726 96.73456 TRUE
0.1-0.3 201.71774 96.73456 TRUE
0.15-0.2 79.54839 96.73456 FALSE
0.15-0.25 157.62097 96.73456 TRUE
0.15-0.3 173.18145 96.73456 TRUE
0.2-0.25 78.07258 96.73456 FALSE
0.2-0.3 93.63306 96.73456 FALSE
0.25-0.3 15.56048 96.73456 FALSEThese results illustrate there is a clear significant trend, which somehow levels of around 25%.
Inductive
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_inductive$avg_precision, results_inductive$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 28.328 < 2.2e-16 ***
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_precision ~ reoccuringtasks, data= results_inductive)
plot(model1, which=2)
Both the assumptions of normality and homogeneity appear to be violated, thus we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_inductive$rank_precision <- rank(-results_inductive$avg_precision)
by(results_inductive$rank_precision, results_inductive$reoccuringtasks, mean)results_inductive$reoccuringtasks: 0
[1] 159.5323
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.05
[1] 302.2823
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.1
[1] 406.5565
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.15
[1] 511.8911
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.2
[1] 514.3024
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.25
[1] 557.5403
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.3
[1] 589.3952results_inductive %>%
ggplot(aes(x= reoccuringtasks ,y =rank_precision)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on precision)")
These data seem to suggest that as the level of reoccuring tasks increases, the models generated by Inductive miner deteriorate. It is clear from the violin plot that the presence of reoccuring tasks vs the absence of reoccuring tasks makes a big difference for the precision values of models generated by Inductive Miner. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test.
kruskal.test(avg_precision~ reoccuringtasks, results_inductive)
Kruskal-Wallis rank sum test
data: avg_precision by reoccuringtasks
Kruskal-Wallis chi-squared = 293.56, df = 6, p-value < 2.2e-16jonckheere.test(results_inductive$avg_precision, as.numeric(results_inductive$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 91378, p-value = 0.002
alternative hypothesis: two.sidedThese analyses show that there is statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. Let’s verify this further by comparing different levels pairwise.
kruskalmc(avg_precision ~reoccuringtasks, data=results_inductive)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
0-0.05 142.75000 96.73456 TRUE
0-0.1 247.02419 96.73456 TRUE
0-0.15 352.35887 96.73456 TRUE
0-0.2 354.77016 96.73456 TRUE
0-0.25 398.00806 96.73456 TRUE
0-0.3 429.86290 96.73456 TRUE
0.05-0.1 104.27419 96.73456 TRUE
0.05-0.15 209.60887 96.73456 TRUE
0.05-0.2 212.02016 96.73456 TRUE
0.05-0.25 255.25806 96.73456 TRUE
0.05-0.3 287.11290 96.73456 TRUE
0.1-0.15 105.33468 96.73456 TRUE
0.1-0.2 107.74597 96.73456 TRUE
0.1-0.25 150.98387 96.73456 TRUE
0.1-0.3 182.83871 96.73456 TRUE
0.15-0.2 2.41129 96.73456 FALSE
0.15-0.25 45.64919 96.73456 FALSE
0.15-0.3 77.50403 96.73456 FALSE
0.2-0.25 43.23790 96.73456 FALSE
0.2-0.3 75.09274 96.73456 FALSE
0.25-0.3 31.85484 96.73456 FALSEThese results seem to confirm our impression. Inductive Miner is sensitive to levels for reoccuring tasks, resulting in models of lower quality (measured in terms of precision values). However, from a level of around 15% reoccuring tasks in the log, this effect seems to have reached a plateau and stays stable.
F1
Let’s continue with the F1-score. First we will verify if there are significant differences between the models generated by four different miners in terms of recall value.
Differences between models generated by different miners
First we check the homogeneity the normality assumption.
leveneTest(results$avg_f1, results$miner, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 3 175.25 < 2.2e-16 ***
3468
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_f1 ~ miner, data=results)
plot(model1, which=2)
The analysis illustrate that both assumptions are violated, so we proceed by focusing on the relative differences and the ranks of the generated models.
Therefore, we start by ranking the discovered models based on their precision value and compare the average ranking per miner.
print("Average Rank per Miner")[1] "Average Rank per Miner"results$rank_f1 <- rank(-results$avg_f1)
by(results$rank_f1, results$miner, mean)results$miner: Alpha +
[1] 2321.445
-------------------------------------------------------------------
results$miner: Heuristics
[1] 2307.161
-------------------------------------------------------------------
results$miner: ILP
[1] 678.5081
-------------------------------------------------------------------
results$miner: Inductive
[1] 1638.887Based on the average (precision) ranking, these results show again ILP creates the best models. The order between the other three miners is now such that Inductive miner produces relatively better models than Heuristics which seems to outperform Alpha+ miner. To get an idea of the distribution of the ranks, we create some violin plots.
results %>%
ggplot(aes(x= miner ,y =rank_f1)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on f1)")
These plots reveal a mixed picture between the results on recall and precision, which was to be expected. ILP outperforms the other miners, followed by Inductive and then by Alpha+ and Heuristics which seem more or less equally well performing.
Note, that these distributions focus on the relative differences between the quality of the models, not the absolute differences. For purpose of illustration, one can find a violin plot below which shows the distribution of the actual f1 values (instead of the ranks).
results %>%
ggplot(aes(x= miner ,y =avg_f1)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of f1 values")
We will now continue to focus on the relative differences instead. Based on the above analysis, it appears that the four miners generate models which rank on average differently (based on recall values). We will use a Kruskal-Wallis test to see if these differences in average ranking are statistically significant.
kruskal.test(avg_f1 ~ miner, results)
Kruskal-Wallis rank sum test
data: avg_f1 by miner
Kruskal-Wallis chi-squared = 1555.8, df = 3, p-value < 2.2e-16These results show that there is a significant difference between the four miners. The order suggested by the results is ILP > Inductive > Heuristics > Alpha+ . We can use a post-hoc test to see which of these differences are significant.
kruskalmc(avg_f1 ~ miner, results)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
Alpha +-Heuristics 14.28399 126.9474 FALSE
Alpha +-ILP 1642.93664 126.9474 TRUE
Alpha +-Inductive 682.55818 126.9474 TRUE
Heuristics-ILP 1628.65265 126.9474 TRUE
Heuristics-Inductive 668.27419 126.9474 TRUE
ILP-Inductive 960.37846 126.9474 TRUEIt appears that except for the difference between Alpha+ and Heuristics, all pairwise comparisons are statisically significant. Thus, based on f1 values, models generated by ILP rank on average better than models generated by the ILP Miner, which rank on average better than models generated by the Alpha+ or Heuristics Miner.
Impact of Infrequent Behavior on performance of the Mining algoritms
Next, we want to test whether the presence/absence of infrequent behavior in the log has an impact on the average ranking based on f1 values. For this we will split the data between experiments with infrequent behavior and experiments without infrequent behavior and verify the ranking for these two subsets. We will start the analysis for experiments where infrequent behavior was absent.
Absence of Infrequent Behavior
First we will test to see if we can apply a standard ANOVA analysis
leveneTest(results_freq$avg_f1, results_freq$miner, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 3 86.517 < 2.2e-16 ***
1732
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_f1 ~ miner, data=results_freq)
plot(model1, which=2)
Both the Levene’s test as the QQ-plot reveal that both the assumption of Homogeneity and Normality are violated. Therefore we will focus on the comparison of average rankings.
print("Average Rank per Miner")[1] "Average Rank per Miner"results_freq$rank_f1 <- rank(-results_freq$avg_f1)
by(results_freq$rank_f1, results_freq$miner, mean)results_freq$miner: Alpha +
[1] 1166.771
-------------------------------------------------------------------
results_freq$miner: Heuristics
[1] 1177.698
-------------------------------------------------------------------
results_freq$miner: ILP
[1] 341.1406
-------------------------------------------------------------------
results_freq$miner: Inductive
[1] 788.3906results_freq %>%
ggplot(aes(x= miner ,y =rank_f1)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on f1)")
These analysis show a similar ranking between the four miners.
We will use a Kruskal-Wallis test to see if these differences in average ranking are statistically significant.
kruskal.test(avg_f1 ~ miner, results_freq)
Kruskal-Wallis rank sum test
data: avg_f1 by miner
Kruskal-Wallis chi-squared = 812.34, df = 3, p-value < 2.2e-16These results show that there are significant ranking differences between the models created by the four miners in terms of precision. The order suggested by the results is ILP > Inductive > Alpha+ > Heuristics. We can use a post-hoc test to see which of these differences are significant.
kruskalmc(avg_f1 ~ miner, results_freq)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
Alpha +-Heuristics 10.92742 89.77831 FALSE
Alpha +-ILP 825.63018 89.77831 TRUE
Alpha +-Inductive 378.38018 89.77831 TRUE
Heuristics-ILP 836.55760 89.77831 TRUE
Heuristics-Inductive 389.30760 89.77831 TRUE
ILP-Inductive 447.25000 89.77831 TRUEIt appears that all pairwise comparisons are statisically significant different from each other, except again between Alpha+ and Heuristics.
Presence of Infrequent Behavior
First we will test to see if we can apply a standard ANOVA analysis
leveneTest(results_infreq$avg_f1, results_infreq$miner, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 3 95.61 < 2.2e-16 ***
1732
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_f1 ~ miner, data=results_infreq)
plot(model1, which=2)
Both the Levene’s test as the QQ-plot reveal that both the assumption of Homogeneity and Normality are violated. Therefore we will focus on the comparison of average rankings.
print("Average Rank per Miner")[1] "Average Rank per Miner"results_infreq$rank_f1 <- rank(-results_infreq$avg_f1)
by(results_infreq$rank_f1, results_infreq$miner, mean)results_infreq$miner: Alpha +
[1] 1150.52
-------------------------------------------------------------------
results_infreq$miner: Heuristics
[1] 1135.862
-------------------------------------------------------------------
results_infreq$miner: ILP
[1] 334.2385
-------------------------------------------------------------------
results_infreq$miner: Inductive
[1] 853.3802results_infreq %>%
ggplot(aes(x= miner ,y =rank_f1)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on F1)")
These analysis show again a similar ranking between the four miners.
We will use a Kruskal-Wallis test to see if these differences in average ranking are statistically significant.
kruskal.test(avg_f1 ~ miner, results_infreq)
Kruskal-Wallis rank sum test
data: avg_f1 by miner
Kruskal-Wallis chi-squared = 756.01, df = 3, p-value < 2.2e-16These results show that there are significant ranking differences between the models created by the four miners in terms of F1 The order suggested by the plot is ILP > Inductive > Heuristics > Alpha+. We can use a post-hoc test to see which of these differences are significant.
kruskalmc(avg_f1 ~ miner, results_infreq)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
Alpha +-Heuristics 14.65783 89.77831 FALSE
Alpha +-ILP 816.28111 89.77831 TRUE
Alpha +-Inductive 297.13940 89.77831 TRUE
Heuristics-ILP 801.62327 89.77831 TRUE
Heuristics-Inductive 282.48157 89.77831 TRUE
ILP-Inductive 519.14171 89.77831 TRUEIt appears that all pairwise comparisons are statisically significant different from each other, except for the difference between Alpha+ and Heuristics.
Impact of amount of reoccuring tasks on performance of the Mining Algorithm
Next, we will investigate how the performance of each of the mining algorithm is influenced by the level of reoccuring tasks in the log files.
Alpha+
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_alpha$avg_f1, results_alpha$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 3.4901 0.002044 **
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_f1 ~ reoccuringtasks, data= results_alpha)
plot(model1, which=2)
Both the assumption of homogneity and normality are violated. Therefore, we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_alpha$rank_f1 <- rank(-results_alpha$avg_f1)
by(results_alpha$rank_f1, results_alpha$reoccuringtasks, mean)results_alpha$reoccuringtasks: 0
[1] 319.2621
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.05
[1] 358.5645
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.1
[1] 399.8911
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.15
[1] 452.9113
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.2
[1] 469.1492
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.25
[1] 493.7944
-------------------------------------------------------------------
results_alpha$reoccuringtasks: 0.3
[1] 547.9274results_alpha %>%
ggplot(aes(x= reoccuringtasks ,y =rank_f1)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on F1)")
These data seem to suggest that as the level of reoccuring tasks increases, the models generated by Alpha miner deteriorate. However, it seems that this effect really starts to show from a level of 15% reoccuring tasks. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test.
kruskal.test(avg_f1~ reoccuringtasks, results_alpha)
Kruskal-Wallis rank sum test
data: avg_f1 by reoccuringtasks
Kruskal-Wallis chi-squared = 76.605, df = 6, p-value = 1.793e-14jonckheere.test(results_alpha$avg_f1, as.numeric(results_alpha$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 124790, p-value = 0.002
alternative hypothesis: two.sidedThese analyses show that there is statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. Let’s verify this further by comparing different levels pairwise.
kruskalmc(avg_f1 ~reoccuringtasks, data=results_alpha)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
0-0.05 39.30242 96.73456 FALSE
0-0.1 80.62903 96.73456 FALSE
0-0.15 133.64919 96.73456 TRUE
0-0.2 149.88710 96.73456 TRUE
0-0.25 174.53226 96.73456 TRUE
0-0.3 228.66532 96.73456 TRUE
0.05-0.1 41.32661 96.73456 FALSE
0.05-0.15 94.34677 96.73456 FALSE
0.05-0.2 110.58468 96.73456 TRUE
0.05-0.25 135.22984 96.73456 TRUE
0.05-0.3 189.36290 96.73456 TRUE
0.1-0.15 53.02016 96.73456 FALSE
0.1-0.2 69.25806 96.73456 FALSE
0.1-0.25 93.90323 96.73456 FALSE
0.1-0.3 148.03629 96.73456 TRUE
0.15-0.2 16.23790 96.73456 FALSE
0.15-0.25 40.88306 96.73456 FALSE
0.15-0.3 95.01613 96.73456 FALSE
0.2-0.25 24.64516 96.73456 FALSE
0.2-0.3 78.77823 96.73456 FALSE
0.25-0.3 54.13306 96.73456 FALSEThese results illustrate that the quality of the models decrease significantly whenever the level of reoccuring tasks increase by 15%.
Heuristics
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_heuristics$avg_f1, results_heuristics$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 2.5386 0.01924 *
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_f1 ~ reoccuringtasks, data= results_heuristics)
plot(model1, which=2)
Both assumptions of Normality and Homogeneity are violated. We therefore continue with the KW-test and compare average ranks between the different levels of reoccuring tasks.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_heuristics$rank_f1 <- rank(-results_heuristics$avg_f1)
by(results_heuristics$rank_f1, results_heuristics$reoccuringtasks, mean)results_heuristics$reoccuringtasks: 0
[1] 385.2903
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.05
[1] 415.8306
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.1
[1] 443.1411
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.15
[1] 484.4516
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.2
[1] 434.3911
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.25
[1] 436.2379
-------------------------------------------------------------------
results_heuristics$reoccuringtasks: 0.3
[1] 442.1573results_heuristics %>%
ggplot(aes(x= reoccuringtasks ,y =rank_f1)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on f1)")
These data seem to suggest that while the F1 score might be negatively influenced by the presence of reoccuring tasks, the F1 score of models discovered by the Heuristics Miner is relatively stable against increasing levels of reoccuring tasks. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test.
kruskal.test(avg_f1~ reoccuringtasks, results_heuristics)
Kruskal-Wallis rank sum test
data: avg_f1 by reoccuringtasks
Kruskal-Wallis chi-squared = 10.666, df = 6, p-value = 0.09925jonckheere.test(results_heuristics$avg_f1, as.numeric(results_heuristics$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 154160, p-value = 0.08
alternative hypothesis: two.sidedThese analyses show that the data does not support the claim of a statistically significant trend in the relative quality of the generated models as the amount of reoccuring tasks increases.
ILP
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_ilp$avg_f1, results_ilp$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 26.868 < 2.2e-16 ***
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_f1 ~ reoccuringtasks, data= results_ilp)
plot(model1, which=2)
Both the assumptions of normality and homogeneity appear to be violated, thus we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_ilp$rank_f1 <- rank(-results_ilp$avg_f1)
by(results_ilp$rank_f1, results_ilp$reoccuringtasks, mean)results_ilp$reoccuringtasks: 0
[1] 160.1935
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.05
[1] 294.1089
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.1
[1] 411.7702
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.15
[1] 444.0685
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.2
[1] 519.3508
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.25
[1] 598.1492
-------------------------------------------------------------------
results_ilp$reoccuringtasks: 0.3
[1] 613.8589results_ilp %>%
ggplot(aes(x= reoccuringtasks ,y =rank_f1)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on f1)")
These data seem to suggest that the precision score of the models generated by ILP is very sensitive to the level of reoccuring tasks. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test.
kruskal.test(avg_f1~ reoccuringtasks, results_ilp)
Kruskal-Wallis rank sum test
data: avg_f1 by reoccuringtasks
Kruskal-Wallis chi-squared = 320.91, df = 6, p-value < 2.2e-16jonckheere.test(results_ilp$avg_f1, as.numeric(results_ilp$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 84772, p-value = 0.002
alternative hypothesis: two.sidedThese analyses show that there is a statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. Let’s verify this further by comparing different levels pairwise.
kruskalmc(avg_f1 ~reoccuringtasks, data=results_ilp)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
0-0.05 133.91532 96.73456 TRUE
0-0.1 251.57661 96.73456 TRUE
0-0.15 283.87500 96.73456 TRUE
0-0.2 359.15726 96.73456 TRUE
0-0.25 437.95565 96.73456 TRUE
0-0.3 453.66532 96.73456 TRUE
0.05-0.1 117.66129 96.73456 TRUE
0.05-0.15 149.95968 96.73456 TRUE
0.05-0.2 225.24194 96.73456 TRUE
0.05-0.25 304.04032 96.73456 TRUE
0.05-0.3 319.75000 96.73456 TRUE
0.1-0.15 32.29839 96.73456 FALSE
0.1-0.2 107.58065 96.73456 TRUE
0.1-0.25 186.37903 96.73456 TRUE
0.1-0.3 202.08871 96.73456 TRUE
0.15-0.2 75.28226 96.73456 FALSE
0.15-0.25 154.08065 96.73456 TRUE
0.15-0.3 169.79032 96.73456 TRUE
0.2-0.25 78.79839 96.73456 FALSE
0.2-0.3 94.50806 96.73456 FALSE
0.25-0.3 15.70968 96.73456 FALSEThese results illustrate there is a clear significant trend, which somehow levels of around 25%.
Inductive
First, we will check the assumptions for a standard ANOVA analysis
leveneTest(results_inductive$avg_f1, results_inductive$reoccuringtasks, center=median)Levene's Test for Homogeneity of Variance (center = median)
Df F value Pr(>F)
group 6 23.671 < 2.2e-16 ***
861
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1model1 <- aov(avg_f1 ~ reoccuringtasks, data= results_inductive)
plot(model1, which=2)
Both the assumptions of normality and homogeneity appear to be violated, thus we fall back to comparing the average rankings of the models instead.
print("Average Rank per level Reoccuring Tasks")[1] "Average Rank per level Reoccuring Tasks"results_inductive$rank_f1 <- rank(-results_inductive$avg_f1)
by(results_inductive$rank_f1, results_inductive$reoccuringtasks, mean)results_inductive$reoccuringtasks: 0
[1] 208.9435
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.05
[1] 287.1774
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.1
[1] 408.2137
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.15
[1] 511.9355
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.2
[1] 516.0323
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.25
[1] 521.1774
-------------------------------------------------------------------
results_inductive$reoccuringtasks: 0.3
[1] 588.0202results_inductive %>%
ggplot(aes(x= reoccuringtasks ,y =rank_f1)) + geom_violin(scale="width", draw_quantiles = c(0.25, 0.5, 0.75)) +
ggtitle("Distribution of ranks (based on f1s)")
These data seem to suggest that as the level of reoccuring tasks increases, the models generated by Inductive miner deteriorate. To test this impression statistically, we will rely on the KW-test and the Jonckheere Test.
kruskal.test(avg_f1~ reoccuringtasks, results_inductive)
Kruskal-Wallis rank sum test
data: avg_f1 by reoccuringtasks
Kruskal-Wallis chi-squared = 231.79, df = 6, p-value < 2.2e-16jonckheere.test(results_inductive$avg_f1, as.numeric(results_inductive$reoccuringtasks), nperm=1000)
Jonckheere-Terpstra test
data:
JT = 99196, p-value = 0.002
alternative hypothesis: two.sidedThese analyses show that there is statistically significant negative trend in the relative quality of the generated models as the amount of reoccuring tasks increases. Let’s verify this further by comparing different levels pairwise.
kruskalmc(avg_f1 ~reoccuringtasks, data=results_inductive)Multiple comparison test after Kruskal-Wallis
p.value: 0.05
Comparisons
obs.dif critical.dif difference
0-0.05 78.233871 96.73456 FALSE
0-0.1 199.270161 96.73456 TRUE
0-0.15 302.991935 96.73456 TRUE
0-0.2 307.088710 96.73456 TRUE
0-0.25 312.233871 96.73456 TRUE
0-0.3 379.076613 96.73456 TRUE
0.05-0.1 121.036290 96.73456 TRUE
0.05-0.15 224.758065 96.73456 TRUE
0.05-0.2 228.854839 96.73456 TRUE
0.05-0.25 234.000000 96.73456 TRUE
0.05-0.3 300.842742 96.73456 TRUE
0.1-0.15 103.721774 96.73456 TRUE
0.1-0.2 107.818548 96.73456 TRUE
0.1-0.25 112.963710 96.73456 TRUE
0.1-0.3 179.806452 96.73456 TRUE
0.15-0.2 4.096774 96.73456 FALSE
0.15-0.25 9.241935 96.73456 FALSE
0.15-0.3 76.084677 96.73456 FALSE
0.2-0.25 5.145161 96.73456 FALSE
0.2-0.3 71.987903 96.73456 FALSE
0.25-0.3 66.842742 96.73456 FALSEThese results seem to confirm our impression. Inductive Miner is sensitive to levels for reoccuring tasks, resulting in models of lower quality (measured in terms of precision values). However, from a level of around 15% reoccuring tasks in the log, this effect seems to have reached a plateau and stays stable.
LS0tDQp0aXRsZTogIlIgTm90ZWJvb2siDQpvdXRwdXQ6DQogIGh0bWxfbm90ZWJvb2s6IA0KICAgIHRoZW1lOiByZWFkYWJsZQ0KICAgIHRvYzogeWVzDQogIHBkZl9kb2N1bWVudDogZGVmYXVsdA0KLS0tDQoNCg0KYGBge3IgbG9hZF9saWJyYXJpZXMsIG1lc3NhZ2U9RkFMU0V9DQpsaWJyYXJ5KGRwbHlyKQ0KbGlicmFyeShjYXIpDQpsaWJyYXJ5KGdncGxvdDIpDQpsaWJyYXJ5KHBnaXJtZXNzKQ0KbGlicmFyeShjbGluZnVuKQ0KYGBgDQoNCg0KDQpgYGB7ciBsb2FkX2RhdGEsIHdhcm5pbmc9Rn0NCnNvdXJjZSgiLi4vZGF0YS9sb2FkX2V4cGVyaW1lbnRfcmVzdWx0cy5yIikNCmBgYA0KDQoNCiMgRGF0YQ0KDQpBcyB3ZSBjYW4gc2VlIGJlbG93LCB0aGVyZSBhcmUgMzQ3MiBleHBlcmltZW50cyAoPSA0IE1pbmVycyB4IDIgbGV2ZWxzIG9mIEZyZXF1ZW50IFBhdGhzIHggNyBsZXZlbHMgb2YgcmVvY2N1cmluZyB0YXNrcyB4IDYyIGxvZ3MpLiBGb3IgZWFjaCBleHBlcmltZW50LCB3ZSBtZWFzdXJlZCB0aGUgcmVjYWxsLCBwcmVjaXNpb24gYW5kIGYxIHNjb3JlIGFzIHRoZSBhdmVyYWdlIGJhc2VkIG9uIDEwIGZvbGQgQ1YuDQoNCmBgYHtyfQ0KZ2xpbXBzZShyZXN1bHRzKQ0KYGBgDQoNClRoZSB0aHJlZSBwbG90cyBiZWxvdyBzaG93IHRoZSBtYXJnaW5hbCBkaXN0cmlidXRpb24gb2YgcmVjYWxsLCBwcmVjaXNpb24sIGYxLg0KDQpgYGB7cn0NCnJlc3VsdHMgJT4lDQogIGdncGxvdChhZXMoeD1hdmdfcmVjYWxsKSkgKyANCiAgZ2VvbV9oaXN0b2dyYW0oYmlud2lkdGg9MC4wMSkgKyANCiAgZ2d0aXRsZSgiSGlzdG9ncmFtIG9mIFJlY2FsbCBWYWx1ZXMiKQ0KYGBgDQoNClRoZSBSZWNhbGwgdmFsdWVzIGFwcGVhciB0byBiZSBVLXNoYXBlZCBkaXN0cmlidXRlZCwgd2l0aCBoaWdoIHBlYWtzIGF0IHZhbHVlcyAwIGFuZCAxDQoNCmBgYHtyfQ0KcmVzdWx0cyAlPiUNCiAgZ2dwbG90KGFlcyh4PWF2Z19wcmVjaXNpb24pKSArIA0KICBnZW9tX2hpc3RvZ3JhbShiaW53aWR0aD0wLjAxKSArIA0KICBnZ3RpdGxlKCJIaXN0b2dyYW0gb2YgUHJlY2lzaW9uIFZhbHVlcyIpDQpgYGANCg0KVGhlIFByZWNpc2lvbiBWYWx1ZXMgaGF2ZSB0aHJlZSBwZWFrcyAobW9kaSksIGF0IHZhbHVlcyAwLCAwLjUgYW5kIDEuDQpgYGB7cn0NCnJlc3VsdHMgJT4lDQogIGdncGxvdChhZXMoeD1hdmdfZjEpKSArIA0KICBnZW9tX2hpc3RvZ3JhbShiaW53aWR0aD0wLjAxKSArIA0KICBnZ3RpdGxlKCJIaXN0b2dyYW0gb2YgRjEgVmFsdWVzIikNCmBgYA0KDQpGMSB2YWx1ZXMgYXJlIHRoZSBhdmcgYmV0d2VlbiByZWNhbGwgYW5kIHByZWNpc2lvbi4gQ29uc2VxdWVudGx5LCB3ZSBzZWUgdGhlIHRocmVlIG1vZGkgcGF0dGVybiBvZiBwcmVjaXNpb24gcmVhcHBlYXJpbmcuDQoNCkl0IGlzIGNsZWFyIHRoYXQgdGhlIHRocmVlIG1ldHJpY3MgYXJlIG5vdCBub3JtYWxseSBkaXN0cmlidXRlZC4NCg0KTm93IGxldHMgZG8gc29tZSBzdGF0aXN0aWNhbCBhbmFseXNpcyBhbmQgc2VlIHdoZXJlIHdlIGdldC4gV2Ugc3RhcnQgd2l0aCBhbmFseXNpbmcgcmVjYWxsIGFuZCBhcHBseSBvbmUgb2YgdGhlIHNpbXBsZXN0IGFuYWx5c2lzLCBhIG9uZS13YXkgQU5PVkEgYW5kIHNlbGVjdCBfbWluZXJfIGFzIHRoZSBpbmRlcGVuZGVudCB2YXJpYWJsZS4gVGhlIG90aGVyIHZhcmlhYmxlcyBhcmUgY29uc2lkZXJlZCBwYXJ0IG9mIHRoZSBlcnJvciB0ZXJtLg0KDQojIFJlY2FsbA0KTGV0J3Mgc3RhcnQgd2l0aCByZWNhbGwuIEZpcnN0IHdlIHdpbGwgdmVyaWZ5IGlmIHRoZXJlIGFyZSBzaWduaWZpY2FudCBkaWZmZXJlbmNlcyBiZXR3ZWVuIHRoZSBtb2RlbHMgZ2VuZXJhdGVkIGJ5IGZvdXIgZGlmZmVyZW50IG1pbmVycyBpbiB0ZXJtcyBvZiByZWNhbGwgdmFsdWUuDQoNCiMjIERpZmZlcmVuY2VzIGJldHdlZW4gbW9kZWxzIGdlbmVyYXRlZCBieSBkaWZmZXJlbnQgbWluZXJzDQpGaXJzdCB3ZSBjaGVjayB0aGUgaG9tb2dlbmVpdHkgYXNzdW1wdGlvbi4gVGhlIExldmVuZSdzIHRlc3Qgc2hvd3MgdGhhdCB0aGlzIGFzc3VtcHRpb24gaXMgbm90IG1ldCBhbmQgd2Ugc2hvdWxkIHVzZSBXZWxjaCdzIEYtcmF0aW8NCg0KYGBge3J9DQpsZXZlbmVUZXN0KHJlc3VsdHMkYXZnX3JlY2FsbCwgcmVzdWx0cyRtaW5lciwgY2VudGVyPW1lZGlhbikNCmBgYA0KDQpUbyBjaGVjayB0aGUgbm9ybWFsaXR5IGFzc3VtcHRpb24sIHdlIGNyZWF0ZSBhbiBhbm92YSBtb2RlbCBhbmQgZXZhbHVhdGUgdGhlIHJlc2lkdWFscy4gDQoNCmBgYHtyfQ0KbW9kZWwxIDwtIGFvdihhdmdfcmVjYWxsIH4gbWluZXIsIGRhdGE9cmVzdWx0cykNCnBsb3QobW9kZWwxLCB3aGljaD0yKQ0KYGBgDQoNClRoZSBwbG90IGNsZWFybHkgaWxsdXN0cmF0ZXMgdGhhdCB0aGUgbm9ybWFsaXR5IGFzc3VtcHRpb24gaXMgbm90IG1ldCBlaXRoZXIuIFNvLCBpdCBpcyBiZXN0IHRvIGFiYW5kb24gdGhlIEFOT1ZBIGFuYWx5c2lzIGFuZCBhcHBseSB0aGUgbm9uLXBhcmFtZXRyaWMgdmFyaWFudCBLcnVza2FsLVdhbGxpcyB0ZXN0LiBUaGUgaWRlYSBiZWhpbmQgdGhlIEtXIHRlc3QgaXMgdG8gcmFuayBhbGwgdGhlIG9ic2VydmF0aW9ucyBhbmQgdGhlbiB0ZXN0IHdoZXRoZXIgdGhlIGF2ZXJhZ2UgcmFuayBkaWZmZXJzIGJldHdlZW4gdGhlIGdyb3Vwcy4gDQoNClRoZXJlZm9yZSwgd2Ugc3RhcnQgYnkgcmFua2luZyB0aGUgZGlzY292ZXJlZCBtb2RlbHMgYmFzZWQgb24gdGhlaXIgcmVjYWxsIHZhbHVlIGFuZCBjb21wYXJlIHRoZSBhdmVyYWdlIHJhbmtpbmcgcGVyIG1pbmVyLg0KDQpgYGB7cn0NCnByaW50KCJBdmVyYWdlIFJhbmsgcGVyIE1pbmVyIikNCnJlc3VsdHMkcmFua19yZWNhbGwgPC0gcmFuaygtcmVzdWx0cyRhdmdfcmVjYWxsKQ0KYnkocmVzdWx0cyRyYW5rX3JlY2FsbCwgcmVzdWx0cyRtaW5lciwgbWVhbikNCg0KYGBgDQoNCkJhc2VkIG9uIHRoZSBhdmVyYWdlIChyZWNhbGwpIHJhbmtpbmcsIHRoZXNlIHJlc3VsdHMgc2hvdyB0aGF0IF9JTFBfIGNyZWF0ZXMgdGhlIGJlc3QgbW9kZWxzLCBmb2xsb3dlZCBieSBfSGV1cmlzdGljc18sIF9BbHBoYStfIGFuZCBfSW5kdWN0aXZlXyAoaW4gdGhhdCBvcmRlcikuIFRvIGdldCBhbiBpZGVhIG9mIHRoZSBkaXN0cmlidXRpb24gb2YgdGhlIHJhbmtzLCB3ZSBjcmVhdGUgc29tZSB2aW9saW4gcGxvdHMuIFRoZXNlIHBsb3RzIHNob3cgdGhlIHRoZSBkZW5zaXR5IGN1cnZlLCBtaXJyb3JlZCBhcm91bmQgYSBjZW50cmFsIHZlcnRpY2FsIGF4aXMsIHRvZ2V0aGVyIHdpdGggdGhyZWUgaG9yaXpvbnRhbCBsaW5lcyB3aGljaCByZXByZXNlbnQgdGhlIDI1JSwgNTAlIGFuZCA3NSUgcGVyY2VudGlsZS4gQSB2aW9saW4gcGxvdCBpcyBzb21laG93IHJlbGF0ZWQgdG8gYSBib3hwbG90LCBidXQgbW9yZSBzdWl0YWJsZSB3aGVuIHRoZSBkaXN0cmlidXRpb24gb2YgdGhlIGRhdGEgaXMgbXVsdGltb2RhbC4NCg0KYGBge3J9DQpyZXN1bHRzICU+JQ0KICBnZ3Bsb3QoYWVzKHg9IG1pbmVyICx5ID1yYW5rX3JlY2FsbCkpICsgZ2VvbV92aW9saW4oc2NhbGU9IndpZHRoIiwgZHJhd19xdWFudGlsZXMgPSBjKDAuMjUsIDAuNSwgMC43NSkpICsNCiAgZ2d0aXRsZSgiRGlzdHJpYnV0aW9uIG9mIHJhbmtzIChiYXNlZCBvbiByZWNhbGwpIikNCmBgYA0KDQpUaGVzZSBwbG90cyByZXZlYWwgZm9yIGV4YW1wbGUgdGhhdCA3NSUgb2YgdGhlIG1vZGVscyBkaXNjb3ZlcmVkIGJ5IElMUCBoYXZlIGEgaGlnaGVyIHJlY2FsbCB2YWx1ZSAoaS5lLiBhcmUgcmFua2VkIGhpZ2hlcikgdGhhbiA3NSUgb2YgdGhlIG1vZGVscyBkaXNjb3ZlcmVkIGJ5IEluZHVjdGl2ZSBtaW5lciBlbiA1MCUgb2YgdGhlIG1vZGVscyBkaXNjb3ZlcmVkIGJ5IEhldXJpc3RpY3MgYW5kIEFscGhhKy4NCg0KTm90ZSwgdGhhdCB0aGVzZSBkaXN0cmlidXRpb25zIGRvIG5vdCB0ZWxsIHVzIGFueXRoaW5nIGFib3V0IHRoZSBhYnNvbHV0ZSBkaWZmZXJlbmNlcyBpbiByZWNhbGwgdmFsdWVzLiBCZWxvdywgb25lIGNhbiBmaW5kIHZpb2xpbiBwbG90cyBmb3IgdGhlIGRpc3RyaWJ1dGlvbiBvZiB0aGUgYWN0dWFsIHJlY2FsbCB2YWx1ZXMsIHdoaWNoIGNsZWFybHkgc2hvdyBhIGRpZmZlcmVudCBwaWN0dXJlLiBOb3RlIHRoYXQgb3VyIGFuYWx5c2lzIGFyZSBiYXNlZCBvbiB0aGUgcmFua2luZyBvZiBkaXNjb3ZlcmVkIG1vZGVscyByYXRoZXIgdGhhbiB0aGUgYWN0dWFsIHF1YWlsaXR5IG9mIHRoZSBtb2RlbHMgYW5kIHRoYXQgd2Ugd2lsbCBzaG93IHRoZXNlIHBsb3RzIGJhc2VkIG9uIGFjdHVhbCB2YWx1ZXMgbWVyZWx5IGZvciB0aGUgcHVycG9zZSBvZiBpbGx1c3RyYXRpb24uIEFsbCBzdGF0aXN0aWNhbCBjbGFpbXMgd2lsbCBiZSBhYm91dCB0aGUgcmFua2luZyBvZiB0aGUgbW9kZWxzLg0KDQpgYGB7cn0NCnJlc3VsdHMgJT4lDQogIGdncGxvdChhZXMoeD0gbWluZXIgLHkgPWF2Z19yZWNhbGwpKSArIGdlb21fdmlvbGluKHNjYWxlPSJ3aWR0aCIsIGRyYXdfcXVhbnRpbGVzID0gYygwLjI1LCAwLjUsIDAuNzUpKSArDQogIGdndGl0bGUoIkRpc3RyaWJ1dGlvbiBvZiByZWNhbGwgdmFsdWVzIikNCmBgYA0KDQpCYXNlZCBvbiB0aGUgYWJvdmUgYW5hbHlzaXMsIGl0IGFwcGVhcnMgdGhhdCB0aGUgZm91ciBtaW5lcnMgZ2VuZXJhdGUgbW9kZWxzIHdoaWNoIHJhbmsgb24gYXZlcmFnZSBkaWZmZXJlbnRseSAoYmFzZWQgb24gcmVjYWxsIHZhbHVlcykuIFdlIHdpbGwgdXNlIGEgS3J1c2thbC1XYWxsaXMgdGVzdCB0byBzZWUgaWYgdGhlc2UgZGlmZmVyZW5jZXMgaW4gYXZlcmFnZSByYW5raW5nIGFyZSBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50Lg0KDQpgYGB7cn0NCmtydXNrYWwudGVzdChhdmdfcmVjYWxsIH4gbWluZXIsIHJlc3VsdHMpDQoNCmBgYA0KDQoNCg0KVGhlc2UgcmVzdWx0cyBzaG93IHRoYXQgdGhlcmUgYXJlIHNpZ25pZmljYW50IHJhbmtpbmcgZGlmZmVyZW5jZXMgYmV0d2VlbiB0aGUgbW9kZWxzIGNyZWF0ZWQgYnkgdGhlIGZvdXIgbWluZXJzIGluIHRlcm1zIG9mIHJlY2FsbC4gVGhlIG9yZGVyIHN1Z2dlc3RlZCBieSB0aGUgcGxvdCBpcyBfSUxQXyA+IF9IZXVyaXN0aWNzXyA+IF9BbHBoYStfID4gX0luZHVjdGl2ZV8uIFdlIGNhbiB1c2UgYSBwb3N0LWhvYyB0ZXN0IHRvIHNlZSB3aGljaCBvZiB0aGVzZSBkaWZmZXJlbmNlcyBhcmUgc2lnbmlmaWNhbnQuDQoNCmBgYHtyfQ0Ka3J1c2thbG1jKGF2Z19yZWNhbGwgfiBtaW5lciwgcmVzdWx0cykNCmBgYA0KDQpJdCBhcHBlYXJzIHRoYXQgYWxsIHBhaXJ3aXNlIGNvbXBhcmlzb25zIGFyZSBzdGF0aXNpY2FsbHkgc2lnbmlmaWNhbnQgZGlmZmVyZW50IGZyb20gZWFjaCBvdGhlci4gVGh1cywgYmFzZWQgb24gcmVjYWxsIHZhbHVlcywgbW9kZWxzIGdlbmVyYXRlZCBieSBJTFAgcmFuayBvbiBhdmVyYWdlIGJldHRlciB0aGFuIG1vZGVscyBnZW5lcmF0ZWQgYnkgdGhlIEhldXJpc3RpY3MgTWluZXIsIHdoaWNoIHJhbmsgb24gYXZlcmFnZSBiZXR0ZXIgdGhhbiBtb2RlbHMgZ2VuZXJhdGVkIGJ5IHRoZSBBbHBoYSsgTWluZXIsIHdoaWNoIHN1YnNlcXVlbnRseSByYW5rIGJldHRlciBvbiBhdmVyYWdlIHRoYW4gbW9kZWxzIGdlbmVyYXRlZCBieSB0aGUgSW5kdWN0aXZlIE1pbmVyLg0KDQojIyBJbXBhY3Qgb2YgSW5mcmVxdWVudCBCZWhhdmlvciBvbiBwZXJmb3JtYW5jZSBvZiB0aGUgTWluaW5nIGFsZ29yaXRtcw0KDQpOZXh0LCB3ZSB3YW50IHRvIHRlc3Qgd2hldGhlciB0aGUgcHJlc2VuY2UvYWJzZW5jZSBvZiBpbmZyZXF1ZW50IGJlaGF2aW9yIGluIHRoZSBsb2cgaGFzIGFuIGltcGFjdCBvbiB0aGUgYXZlcmFnZSByYW5raW5nIGJhc2VkIG9uIHJlY2FsbCB2YWx1ZXMuIEZvciB0aGlzIHdlIHdpbGwgc3BsaXQgdGhlIGRhdGEgYmV0d2VlbiBleHBlcmltZW50cyB3aXRoIGluZnJlcXVlbnQgYmVoYXZpb3IgYW5kIGV4cGVyaW1lbnRzIHdpdGhvdXQgaW5mcmVxdWVudCBiZWhhdmlvciBhbmQgdmVyaWZ5IHRoZSByYW5raW5nIGZvciB0aGVzZSB0d28gc3Vic2V0cy4gV2Ugd2lsbCBzdGFydCB0aGUgYW5hbHlzaXMgZm9yIGV4cGVyaW1lbnRzIHdoZXJlIGluZnJlcXVlbnQgYmVoYXZpb3Igd2FzIGFic2VudC4NCg0KYGBge3J9DQpyZXN1bHRzX2ZyZXEgPC0gcmVzdWx0cyAlPiUNCiAgZmlsdGVyKGluZnJlcXVlbnRwYXRocz09RikNCg0KcmVzdWx0c19pbmZyZXEgPC0gcmVzdWx0cyAlPiUNCiAgZmlsdGVyKGluZnJlcXVlbnRwYXRocz09VCkNCmBgYA0KDQojIyMgQWJzZW5jZSBvZiBJbmZyZXF1ZW50IEJlaGF2aW9yDQoNCkZpcnN0IHdlIHdpbGwgdGVzdCB0byBzZWUgaWYgd2UgY2FuIGFwcGx5IGEgc3RhbmRhcmQgQU5PVkEgYW5hbHlzaXMNCmBgYHtyfQ0KbGV2ZW5lVGVzdChyZXN1bHRzX2ZyZXEkYXZnX3JlY2FsbCwgcmVzdWx0c19mcmVxJG1pbmVyLCBjZW50ZXI9bWVkaWFuKQ0KbW9kZWwxIDwtIGFvdihhdmdfcmVjYWxsIH4gbWluZXIsIGRhdGE9cmVzdWx0c19mcmVxKQ0KcGxvdChtb2RlbDEsIHdoaWNoPTIpDQpgYGANCg0KQm90aCB0aGUgTGV2ZW5lJ3MgdGVzdCBhcyB0aGUgUVEtcGxvdCByZXZlYWwgdGhhdCBib3RoIHRoZSBhc3N1bXB0aW9uIG9mIEhvbW9nZW5laXR5IGFuZCBOb3JtYWxpdHkgYXJlIHZpb2xhdGVkLiBUaGVyZWZvcmUgd2Ugd2lsbCBmb2N1cyBvbiB0aGUgY29tcGFyaXNvbiBvZiBhdmVyYWdlIHJhbmtpbmdzLg0KDQpgYGB7cn0NCnByaW50KCJBdmVyYWdlIFJhbmsgcGVyIE1pbmVyIikNCnJlc3VsdHNfZnJlcSRyYW5rX3JlY2FsbCA8LSByYW5rKC1yZXN1bHRzX2ZyZXEkYXZnX3JlY2FsbCkNCmJ5KHJlc3VsdHNfZnJlcSRyYW5rX3JlY2FsbCwgcmVzdWx0c19mcmVxJG1pbmVyLCBtZWFuKQ0KDQpgYGANCg0KYGBge3J9DQpyZXN1bHRzX2ZyZXEgJT4lDQogIGdncGxvdChhZXMoeD0gbWluZXIgLHkgPXJhbmtfcmVjYWxsKSkgKyBnZW9tX3Zpb2xpbihzY2FsZT0id2lkdGgiLCBkcmF3X3F1YW50aWxlcyA9IGMoMC4yNSwgMC41LCAwLjc1KSkgKw0KICBnZ3RpdGxlKCJEaXN0cmlidXRpb24gb2YgcmFua3MgKGJhc2VkIG9uIHJlY2FsbCkiKQ0KYGBgDQoNClRoZXNlIGFuYWx5c2lzIHNob3cgYSBzaW1pbGFyIHJhbmtpbmcgYmV0d2VlbiB0aGUgZm91ciBtaW5lcnMuIEhvd2V2ZXIsIGl0IGFwcGVhcnMgdGhhdCBpbiBjYXNlIG9mIG5vIGluZnJlcXVlbnQgYmVoYXZpb3IgaW4gdGhlIGxvZywgSUxQIG91dHBlcmZvcm1zIHRoZSBvdGhlciBhbGdvcml0aG1zIGV2ZW4gbW9yZS4gNzUlIG9mIHRoZSBtb2RlbHMgZ2VuZXJhdGVkIGJ5IElMUCBhcmUgYmV0dGVyIHRoYW4gNzUlIG9mIHRoZSBtb2RlbHMgZ2VuZXJhdGVkIGJ5IEFscGhhKyBlbiBIZXVyaXN0aWNzIGFuZCBhbGwgdGhlIG1vZGVscyBnZW5lcmF0ZWQgYnkgSUxQIGFyZSBiZXR0ZXIgdGhhbiA3NSUgb2YgdGhlIG1vZGVscyBnZW5lcmF0ZWQgYnkgSW5kdWN0aXZlIE1pbmVyLiANCg0KV2Ugd2lsbCB1c2UgYSBLcnVza2FsLVdhbGxpcyB0ZXN0IHRvIHNlZSBpZiB0aGVzZSBkaWZmZXJlbmNlcyBpbiBhdmVyYWdlIHJhbmtpbmcgYXJlIHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnQuDQoNCmBgYHtyfQ0Ka3J1c2thbC50ZXN0KGF2Z19yZWNhbGwgfiBtaW5lciwgcmVzdWx0c19mcmVxKQ0KDQpgYGANCg0KDQoNClRoZXNlIHJlc3VsdHMgc2hvdyB0aGF0IHRoZXJlIGFyZSBzaWduaWZpY2FudCByYW5raW5nIGRpZmZlcmVuY2VzIGJldHdlZW4gdGhlIG1vZGVscyBjcmVhdGVkIGJ5IHRoZSBmb3VyIG1pbmVycyBpbiB0ZXJtcyBvZiByZWNhbGwuIFRoZSBvcmRlciBzdWdnZXN0ZWQgYnkgdGhlIHBsb3QgaXMgX0lMUF8gPiBfSGV1cmlzdGljc18gPiBfQWxwaGErXyA+IF9JbmR1Y3RpdmVfLiBXZSBjYW4gdXNlIGEgcG9zdC1ob2MgdGVzdCB0byBzZWUgd2hpY2ggb2YgdGhlc2UgZGlmZmVyZW5jZXMgYXJlIHNpZ25pZmljYW50Lg0KDQpgYGB7cn0NCmtydXNrYWxtYyhhdmdfcmVjYWxsIH4gbWluZXIsIHJlc3VsdHNfZnJlcSkNCmBgYA0KDQpJdCBhcHBlYXJzIHRoYXQgYWxsIHBhaXJ3aXNlIGNvbXBhcmlzb25zIGFyZSBzdGF0aXNpY2FsbHkgc2lnbmlmaWNhbnQgZGlmZmVyZW50IGZyb20gZWFjaCBvdGhlciwgZXhjZXB0IGZvciB0aGUgZGlmZmVyZW5jZXMgYmV0d2VlbiB0aGUgbW9kZWxzIGdlbmVyYXRlZCBieSBBbHBoYSsgb24gdGhlIG9uZSBoYW5kIGFuZCB0aGUgbW9kZWxzIGdlbmVyYXRlZCBieSBJbmR1Y3RpdmUgYW5kIEhldXJpc3RpY3MuIA0KDQoNCiMjIyBQcmVzZW5jZSBvZiBJbmZyZXF1ZW50IEJlaGF2aW9yDQoNCkZpcnN0IHdlIHdpbGwgdGVzdCB0byBzZWUgaWYgd2UgY2FuIGFwcGx5IGEgc3RhbmRhcmQgQU5PVkEgYW5hbHlzaXMNCmBgYHtyfQ0KbGV2ZW5lVGVzdChyZXN1bHRzX2luZnJlcSRhdmdfcmVjYWxsLCByZXN1bHRzX2luZnJlcSRtaW5lciwgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX3JlY2FsbCB+IG1pbmVyLCBkYXRhPXJlc3VsdHNfaW5mcmVxKQ0KcGxvdChtb2RlbDEsIHdoaWNoPTIpDQpgYGANCg0KQm90aCB0aGUgTGV2ZW5lJ3MgdGVzdCBhcyB0aGUgUVEtcGxvdCByZXZlYWwgdGhhdCBib3RoIHRoZSBhc3N1bXB0aW9uIG9mIEhvbW9nZW5laXR5IGFuZCBOb3JtYWxpdHkgYXJlIHZpb2xhdGVkLiBUaGVyZWZvcmUgd2Ugd2lsbCBmb2N1cyBvbiB0aGUgY29tcGFyaXNvbiBvZiBhdmVyYWdlIHJhbmtpbmdzLg0KDQpgYGB7cn0NCnByaW50KCJBdmVyYWdlIFJhbmsgcGVyIE1pbmVyIikNCnJlc3VsdHNfaW5mcmVxJHJhbmtfcmVjYWxsIDwtIHJhbmsoLXJlc3VsdHNfaW5mcmVxJGF2Z19yZWNhbGwpDQpieShyZXN1bHRzX2luZnJlcSRyYW5rX3JlY2FsbCwgcmVzdWx0c19pbmZyZXEkbWluZXIsIG1lYW4pDQoNCmBgYA0KDQpgYGB7cn0NCnJlc3VsdHNfaW5mcmVxICU+JQ0KICBnZ3Bsb3QoYWVzKHg9IG1pbmVyICx5ID1yYW5rX3JlY2FsbCkpICsgZ2VvbV92aW9saW4oc2NhbGU9IndpZHRoIiwgZHJhd19xdWFudGlsZXMgPSBjKDAuMjUsIDAuNSwgMC43NSkpICsNCiAgZ2d0aXRsZSgiRGlzdHJpYnV0aW9uIG9mIHJhbmtzIChiYXNlZCBvbiByZWNhbGwpIikNCmBgYA0KDQpUaGVzZSBhbmFseXNpcyBzaG93IGFnYWluIGEgc2ltaWxhciByYW5raW5nIGJldHdlZW4gdGhlIGZvdXIgbWluZXJzLiBIb3dldmVyLCBpdCBhcHBlYXJzIHRoYXQgdGhlIGRpZmZlcmVuY2VzIGJldHdlZW4gSUxQIG9uIHRoZSBvbmUgaGFuZCBhbmQgQWxwaGErIGFuZCBIZXVyaXN0aWNzIG9uIHRoZSBvdGhlciBoYW5kIGFyZSBsZXNzIGV4dHJlbWUgdGhhbiBpbiB0aGUgY2FzZSBvZiBsb2dzIHdpdGhvdXQgaW5mcmVxdWVudCBiZWhhdmlvci4gWWV0LCBpdCBpcyBzdGlsbCBjbGVhciB0aGF0IElMUCBvdXRwZXJmb3JtcyB0aGUgb3RoZXIgYWxnb3JpdGhtcyBpbiB0ZXJtcyBvZiByZWNhbGwuDQoNCldlIHdpbGwgdXNlIGEgS3J1c2thbC1XYWxsaXMgdGVzdCB0byBzZWUgaWYgdGhlc2UgZGlmZmVyZW5jZXMgaW4gYXZlcmFnZSByYW5raW5nIGFyZSBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50Lg0KDQpgYGB7cn0NCmtydXNrYWwudGVzdChhdmdfcmVjYWxsIH4gbWluZXIsIHJlc3VsdHNfaW5mcmVxKQ0KDQpgYGANCg0KDQoNClRoZXNlIHJlc3VsdHMgc2hvdyB0aGF0IHRoZXJlIGFyZSBzaWduaWZpY2FudCByYW5raW5nIGRpZmZlcmVuY2VzIGJldHdlZW4gdGhlIG1vZGVscyBjcmVhdGVkIGJ5IHRoZSBmb3VyIG1pbmVycyBpbiB0ZXJtcyBvZiByZWNhbGwuIFRoZSBvcmRlciBzdWdnZXN0ZWQgYnkgdGhlIHBsb3QgaXMgX0lMUF8gPiBfSGV1cmlzdGljc18gPiBfQWxwaGErXyA+IF9JbmR1Y3RpdmVfLiBXZSBjYW4gdXNlIGEgcG9zdC1ob2MgdGVzdCB0byBzZWUgd2hpY2ggb2YgdGhlc2UgZGlmZmVyZW5jZXMgYXJlIHNpZ25pZmljYW50Lg0KDQpgYGB7cn0NCmtydXNrYWxtYyhhdmdfcmVjYWxsIH4gbWluZXIsIHJlc3VsdHNfaW5mcmVxKQ0KYGBgDQoNCkl0IGFwcGVhcnMgdGhhdCBhbGwgcGFpcndpc2UgY29tcGFyaXNvbnMgYXJlIHN0YXRpc2ljYWxseSBzaWduaWZpY2FudCBkaWZmZXJlbnQgZnJvbSBlYWNoIG90aGVyLg0KDQojIyBJbXBhY3Qgb2YgYW1vdW50IG9mIHJlb2NjdXJpbmcgdGFza3Mgb24gcGVyZm9ybWFuY2Ugb2YgdGhlIE1pbmluZyBBbGdvcml0aG0NCg0KTmV4dCwgd2Ugd2lsbCBpbnZlc3RpZ2F0ZSBob3cgdGhlIHBlcmZvcm1hbmNlIGVhY2ggb2YgdGhlIG1pbmluZyBhbGdvcml0aG0gaXMgaW5mbHVlbmNlZCBieSB0aGUgbGV2ZWwgb2YgcmVvY2N1cmluZyB0YXNrcyBpbiB0aGUgbG9nIGZpbGVzLg0KDQpgYGB7cn0NCnJlc3VsdHNfYWxwaGEgPC0gcmVzdWx0cyAlPiUNCiAgZmlsdGVyKG1pbmVyPT0iQWxwaGEgKyIpJT4lDQogIG11dGF0ZShyZW9jY3VyaW5ndGFza3MgPSBhcy5mYWN0b3IocmVvY2N1cmluZ3Rhc2tzKSkNCnJlc3VsdHNfaGV1cmlzdGljcyA8LSByZXN1bHRzICU+JQ0KICBmaWx0ZXIobWluZXI9PSJIZXVyaXN0aWNzIiklPiUNCiAgbXV0YXRlKHJlb2NjdXJpbmd0YXNrcyA9IGFzLmZhY3RvcihyZW9jY3VyaW5ndGFza3MpKQ0KcmVzdWx0c19pbHAgPC0gcmVzdWx0cyAlPiUNCiAgZmlsdGVyKG1pbmVyPT0iSUxQIiklPiUNCiAgbXV0YXRlKHJlb2NjdXJpbmd0YXNrcyA9IGFzLmZhY3RvcihyZW9jY3VyaW5ndGFza3MpKQ0KcmVzdWx0c19pbmR1Y3RpdmUgPC0gcmVzdWx0cyAlPiUNCiAgZmlsdGVyKG1pbmVyPT0iSW5kdWN0aXZlIiklPiUNCiAgbXV0YXRlKHJlb2NjdXJpbmd0YXNrcyA9IGFzLmZhY3RvcihyZW9jY3VyaW5ndGFza3MpKQ0KYGBgDQoNCiMjIyBBbHBoYSsNCg0KRmlyc3QsIHdlIHdpbGwgY2hlY2sgdGhlIGFzc3VtcHRpb25zIGZvciBhIHN0YW5kYXJkIEFOT1ZBIGFuYWx5c2lzIA0KDQpgYGB7cn0NCmxldmVuZVRlc3QocmVzdWx0c19hbHBoYSRhdmdfcmVjYWxsLCByZXN1bHRzX2FscGhhJHJlb2NjdXJpbmd0YXNrcywgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX3JlY2FsbCB+IHJlb2NjdXJpbmd0YXNrcywgZGF0YT0gcmVzdWx0c19hbHBoYSkNCnBsb3QobW9kZWwxLCB3aGljaD0yKQ0KYGBgDQoNCkJvdGggdGhlIGFzc3VtcHRpb25zIG9mIG5vcm1hbGl0eSBhbmQgaG9tb2dlbmVpdHkgYXBwZWFyIHRvIGJlIHZpb2xhdGVkLCB0aHVzIHdlIGZhbGwgYmFjayB0byBjb21wYXJpbmcgdGhlIGF2ZXJhZ2UgcmFua2luZ3Mgb2YgdGhlIG1vZGVscyBpbnN0ZWFkLg0KDQoNCmBgYHtyfQ0KcHJpbnQoIkF2ZXJhZ2UgUmFuayBwZXIgbGV2ZWwgUmVvY2N1cmluZyBUYXNrcyIpDQpyZXN1bHRzX2FscGhhJHJhbmtfcmVjYWxsIDwtIHJhbmsoLXJlc3VsdHNfYWxwaGEkYXZnX3JlY2FsbCkNCmJ5KHJlc3VsdHNfYWxwaGEkcmFua19yZWNhbGwsIHJlc3VsdHNfYWxwaGEkcmVvY2N1cmluZ3Rhc2tzLCBtZWFuKQ0KDQpgYGANCg0KYGBge3J9DQpyZXN1bHRzX2FscGhhICU+JQ0KICBnZ3Bsb3QoYWVzKHg9IHJlb2NjdXJpbmd0YXNrcyAseSA9cmFua19yZWNhbGwpKSArIGdlb21fdmlvbGluKHNjYWxlPSJ3aWR0aCIsIGRyYXdfcXVhbnRpbGVzID0gYygwLjI1LCAwLjUsIDAuNzUpKSArDQogIGdndGl0bGUoIkRpc3RyaWJ1dGlvbiBvZiByYW5rcyAoYmFzZWQgb24gcmVjYWxsKSIpDQpgYGANCg0KVGhlc2UgZGF0YSBzZWVtIHRvIHN1Z2dlc3QgdGhhdCBhcyB0aGUgbGV2ZWwgb2YgcmVvY2N1cmluZyB0YXNrcyBpbmNyZWFzZXMsIHRoZSBtb2RlbHMgZ2VuZXJhdGVkIGJ5IEFscGhhIG1pbmVyIGRldGVyaW9yYXRlLiBUbyB0ZXN0IHRoaXMgaW1wcmVzc2lvbiBzdGF0aXN0aWNhbGx5LCB3ZSB3aWxsIHJlbHkgb24gdGhlIEtXLXRlc3QgYW5kIHRoZSBKb25ja2hlZXJlIFRlc3QuIFRoZSBsYXR0ZXIgdGVzdCB3aGV0aGVyIHRoZXJlIGlzIGEgc2lnbmlmaWNhbnQgdHJlbmQgd2l0aGluIHRoZSBkYXRhIChhY3Jvc3MgaW5jcmVhc2luZyBsZXZlbHMgb2YgcmVvY2N1cmluZ3Rhc2tzKQ0KDQpgYGB7cn0NCmtydXNrYWwudGVzdChhdmdfcmVjYWxsfiByZW9jY3VyaW5ndGFza3MsIHJlc3VsdHNfYWxwaGEpDQpqb25ja2hlZXJlLnRlc3QocmVzdWx0c19hbHBoYSRhdmdfcmVjYWxsLCBhcy5udW1lcmljKHJlc3VsdHNfYWxwaGEkcmVvY2N1cmluZ3Rhc2tzKSwgbnBlcm09MTAwMCkNCg0KYGBgDQoNClRoZXNlIGFuYWx5c2VzIHNob3cgdGhhdCB0aGVyZSBpcyBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50IG5lZ2F0aXZlIHRyZW5kIGluIHRoZSByZWxhdGl2ZSBxdWFsaXR5IG9mIHRoZSBnZW5lcmF0ZWQgbW9kZWxzIGFzIHRoZSBhbW91bnQgb2YgcmVvY2N1cmluZyB0YXNrcyBpbmNyZWFzZXMuIExldCdzIHZlcmlmeSB0aGlzIGZ1cnRoZXIgYnkgY29tcGFyaW5nIGRpZmZlcmVudCBsZXZlbHMgcGFpcndpc2UuDQoNCmBgYHtyfQ0Ka3J1c2thbG1jKGF2Z19yZWNhbGwgfnJlb2NjdXJpbmd0YXNrcywgZGF0YT1yZXN1bHRzX2FscGhhKQ0KYGBgDQoNCldoaWxlIHRoZSBwcmV2aW91cyBhbmFseXNpcyBzaG93IHRoYXQgdGhlcmUgaXMgYSBzdGF0aXNpY2FsbHkgc2lnbmlmaWNhbnQgdHJlbmQsIHRoZSBwYWlyd2lzZSBjb21wYXJpc29ucyBkbyBub3QgcHJvdmlkZSBhIGNsZWFyIHBpY3R1cmUgaG93IHRoaXMgdHJlbmQgbG9va3MgbGlrZSwgd2l0aCBtYW55IGNvbXBhcmlzb25zIHN0YXRpc3RpY2FsbHkgaW5zaWduaWZpY2FudC4gTm90ZSBob3dldmVyLCB0aGF0IHRoaXMgZG9lcyBub3QgcHJvdmUgdGhhdCB0aGVyZSBpcyBubyBkaWZmZXJlbmNlLCBidXQgY291bGQgYWxzbyBiZSBjYXVzZWQgYnkgYSBsYWNrIG9mIHBvd2VyLiBUaGUgTXVsdGlwbGUgY29tcGFyaXNvbiB0ZXN0IGluY29ycG9yYXRlcyBzb21lIGtpbmQgb2YgQm9uZmVycm9uaSBjb3JyZWN0aW9uIHRvIGNvcnJlY3QgZm9yIHRoZSBmYWN0IHRoYXQgd2UgYXJlIHBlcmZvcm1pbmcgbWFueSB0ZXN0cyBzaW11bHRhbmVvdXNseSwgd2hpY2ggb2Z0ZW4gZXJycyBvbiB0aGUgY29uc2VydmF0aXZlIHNpZGUuDQoNCiMjIyBIZXVyaXN0aWNzDQoNCkZpcnN0LCB3ZSB3aWxsIGNoZWNrIHRoZSBhc3N1bXB0aW9ucyBmb3IgYSBzdGFuZGFyZCBBTk9WQSBhbmFseXNpcyANCg0KYGBge3J9DQpsZXZlbmVUZXN0KHJlc3VsdHNfaGV1cmlzdGljcyRhdmdfcmVjYWxsLCByZXN1bHRzX2hldXJpc3RpY3MkcmVvY2N1cmluZ3Rhc2tzLCBjZW50ZXI9bWVkaWFuKQ0KbW9kZWwxIDwtIGFvdihhdmdfcmVjYWxsIH4gcmVvY2N1cmluZ3Rhc2tzLCBkYXRhPSByZXN1bHRzX2hldXJpc3RpY3MpDQpwbG90KG1vZGVsMSwgd2hpY2g9MikNCmBgYA0KDQpCb3RoIHRoZSBhc3N1bXB0aW9ucyBvZiBub3JtYWxpdHkgYW5kIGhvbW9nZW5laXR5IGFwcGVhciB0byBiZSB2aW9sYXRlZCwgdGh1cyB3ZSBmYWxsIGJhY2sgdG8gY29tcGFyaW5nIHRoZSBhdmVyYWdlIHJhbmtpbmdzIG9mIHRoZSBtb2RlbHMgaW5zdGVhZC4NCg0KDQpgYGB7cn0NCnByaW50KCJBdmVyYWdlIFJhbmsgcGVyIGxldmVsIFJlb2NjdXJpbmcgVGFza3MiKQ0KcmVzdWx0c19oZXVyaXN0aWNzJHJhbmtfcmVjYWxsIDwtIHJhbmsoLXJlc3VsdHNfaGV1cmlzdGljcyRhdmdfcmVjYWxsKQ0KYnkocmVzdWx0c19oZXVyaXN0aWNzJHJhbmtfcmVjYWxsLCByZXN1bHRzX2hldXJpc3RpY3MkcmVvY2N1cmluZ3Rhc2tzLCBtZWFuKQ0KDQpgYGANCg0KYGBge3J9DQpyZXN1bHRzX2hldXJpc3RpY3MgJT4lDQogIGdncGxvdChhZXMoeD0gcmVvY2N1cmluZ3Rhc2tzICx5ID1yYW5rX3JlY2FsbCkpICsgZ2VvbV92aW9saW4oc2NhbGU9IndpZHRoIiwgZHJhd19xdWFudGlsZXMgPSBjKDAuMjUsIDAuNSwgMC43NSkpICsNCiAgZ2d0aXRsZSgiRGlzdHJpYnV0aW9uIG9mIHJhbmtzIChiYXNlZCBvbiByZWNhbGwpIikNCmBgYA0KDQpUaGVzZSBkYXRhIHNlZW0gdG8gc3VnZ2VzdCB0aGF0IGFzIHdlIG1vdmUgZnJvbSBubyByZW9jY3VyaW5nIHRhc2tzIHRvIHNvbWUgcmVvY2N1cmluZyB0YXNrcywgdGhlIG1vZGVscyBnZW5lcmF0ZWQgYnkgSGV1cmlzdGljcyBtaW5lciBkZXRlcmlvcmF0ZSBzb21lLiBIb3dldmVyLCBhcyB0aGUgbGV2ZWwgb2YgcmVvY3VycmluZyB0YXNrcyBpbmNyZWFzZSwgdGhlcmUgYXBwZWFycyB0byBiZSBubyBjbGVhciB0cmVuZCBpbiB0aGUgZnVydGhlciBkZXRlcmlvcmF0aW9uIG9mIHRoZSByZWxhdGl2ZSBxdWFsaXR5IG9mIHRoZSBtb2RlbHMuIFRvIHRlc3QgdGhpcyBpbXByZXNzaW9uIHN0YXRpc3RpY2FsbHksIHdlIHdpbGwgcmVseSBvbiB0aGUgS1ctdGVzdCBhbmQgdGhlIEpvbmNraGVlcmUgVGVzdC4gDQoNCmBgYHtyfQ0Ka3J1c2thbC50ZXN0KGF2Z19yZWNhbGx+IHJlb2NjdXJpbmd0YXNrcywgcmVzdWx0c19oZXVyaXN0aWNzKQ0Kam9uY2toZWVyZS50ZXN0KHJlc3VsdHNfaGV1cmlzdGljcyRhdmdfcmVjYWxsLCBhcy5udW1lcmljKHJlc3VsdHNfaGV1cmlzdGljcyRyZW9jY3VyaW5ndGFza3MpLCBucGVybT0xMDAwKQ0KDQpgYGANCg0KVGhlc2UgYW5hbHlzZXMgc2hvdyB0aGF0IHRoZXJlIGlzIGluZGVlZCBhIHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnQgbmVnYXRpdmUgdHJlbmQgaW4gdGhlIHJlbGF0aXZlIHF1YWxpdHkgb2YgdGhlIGdlbmVyYXRlZCBtb2RlbHMgYXMgdGhlIGFtb3VudCBvZiByZW9jY3VyaW5nIHRhc2tzIGluY3JlYXNlcy4gTGV0J3MgdmVyaWZ5IHRoaXMgZnVydGhlciBieSBjb21wYXJpbmcgZGlmZmVyZW50IGxldmVscyBwYWlyd2lzZS4NCg0KYGBge3J9DQprcnVza2FsbWMoYXZnX3JlY2FsbCB+cmVvY2N1cmluZ3Rhc2tzLCBkYXRhPXJlc3VsdHNfaGV1cmlzdGljcykNCmBgYA0KDQpUaGVzZSByZXN1bHRzIHNlZW0gdG8gY29uZmlybSBvdXIgaW1wcmVzc2lvbiB0aGF0IEhldXJpc3RpY3MgTWluZXIgaXMgc2Vuc2l0aXZlIGZvciB0aGUgcHJlc2VuY2Ugb2YgcmVvY2N1cmluZyB0YXNrcywgYnV0IGRvZXMgbm90IHNlZW0gdG8gcHJvZHVjZSBtb2RlbHMgb2YgZGVjcmVhc2luZyByZWxhdGl2ZSBxdWFsaXR5IGFzIHRoZSBsZXZlbCBvZiByZW9jY3VyaW5nIHRhc2tzIGluY3JlYXNlcy4NCg0KIyMjIElMUA0KDQpGaXJzdCwgd2Ugd2lsbCBjaGVjayB0aGUgYXNzdW1wdGlvbnMgZm9yIGEgc3RhbmRhcmQgQU5PVkEgYW5hbHlzaXMgDQoNCmBgYHtyfQ0KbGV2ZW5lVGVzdChyZXN1bHRzX2lscCRhdmdfcmVjYWxsLCByZXN1bHRzX2lscCRyZW9jY3VyaW5ndGFza3MsIGNlbnRlcj1tZWRpYW4pDQptb2RlbDEgPC0gYW92KGF2Z19yZWNhbGwgfiByZW9jY3VyaW5ndGFza3MsIGRhdGE9IHJlc3VsdHNfaWxwKQ0KcGxvdChtb2RlbDEsIHdoaWNoPTIpDQpgYGANCg0KQm90aCB0aGUgYXNzdW1wdGlvbnMgb2Ygbm9ybWFsaXR5IGFuZCBob21vZ2VuZWl0eSBhcHBlYXIgdG8gYmUgdmlvbGF0ZWQsIHRodXMgd2UgZmFsbCBiYWNrIHRvIGNvbXBhcmluZyB0aGUgYXZlcmFnZSByYW5raW5ncyBvZiB0aGUgbW9kZWxzIGluc3RlYWQuDQoNCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBsZXZlbCBSZW9jY3VyaW5nIFRhc2tzIikNCnJlc3VsdHNfaWxwJHJhbmtfcmVjYWxsIDwtIHJhbmsoLXJlc3VsdHNfaWxwJGF2Z19yZWNhbGwpDQpieShyZXN1bHRzX2lscCRyYW5rX3JlY2FsbCwgcmVzdWx0c19pbHAkcmVvY2N1cmluZ3Rhc2tzLCBtZWFuKQ0KDQpgYGANCg0KYGBge3J9DQpyZXN1bHRzX2lscCAlPiUNCiAgZ2dwbG90KGFlcyh4PSByZW9jY3VyaW5ndGFza3MgLHkgPXJhbmtfcmVjYWxsKSkgKyBnZW9tX3Zpb2xpbihzY2FsZT0id2lkdGgiLCBkcmF3X3F1YW50aWxlcyA9IGMoMC4yNSwgMC41LCAwLjc1KSkgKw0KICBnZ3RpdGxlKCJEaXN0cmlidXRpb24gb2YgcmFua3MgKGJhc2VkIG9uIHJlY2FsbCkiKQ0KYGBgDQoNClRoZXNlIGRhdGEgc2VlbSB0byBzdWdnZXN0IHRoYXQgSUxQIGlzIG5vdCBzZW5zaXRpdmUgdG8gdGhlIGxldmVsIG9mIHJlb2NjdXJpbmcgdGFza3MuIFRvIHRlc3QgdGhpcyBpbXByZXNzaW9uIHN0YXRpc3RpY2FsbHksIHdlIHdpbGwgcmVseSBvbiB0aGUgS1ctdGVzdCBhbmQgdGhlIEpvbmNraGVlcmUgVGVzdC4gVGhlIGxhdHRlciB0ZXN0IHdoZXRoZXIgdGhlcmUgaXMgYSBzaWduaWZpY2FudCB0cmVuZCB3aXRoaW4gdGhlIGRhdGEgKGFjcm9zcyBpbmNyZWFzaW5nIGxldmVscyBvZiByZW9jY3VyaW5ndGFza3MpDQoNCmBgYHtyfQ0Ka3J1c2thbC50ZXN0KGF2Z19yZWNhbGx+IHJlb2NjdXJpbmd0YXNrcywgcmVzdWx0c19pbHApDQpqb25ja2hlZXJlLnRlc3QocmVzdWx0c19pbHAkYXZnX3JlY2FsbCwgYXMubnVtZXJpYyhyZXN1bHRzX2lscCRyZW9jY3VyaW5ndGFza3MpLCBucGVybT0xMDAwKQ0KDQpgYGANCg0KVGhlc2UgYW5hbHlzZXMgY29uZmlybSBvdXIgaW1wcmVzc2lvbiBhbmQgc2hvdyB0aGF0IHRoZXJlIGlzIG5vIHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnQgbmVnYXRpdmUgdHJlbmQgaW4gdGhlIHJlbGF0aXZlIHF1YWxpdHkgb2YgdGhlIGdlbmVyYXRlZCBtb2RlbHMgYXMgdGhlIGFtb3VudCBvZiByZW9jY3VyaW5nIHRhc2tzIGluY3JlYXNlcy4gVGhlIHF1YWxpdHkgb2YgbW9kZWxzIGdlbmVyYXRlZCBieSBJTFAgbWluZXIgaXMgaW5zZW5zaXRpdmUgdG8gdGhlIGxldmVsIG9mIHJlb2NjdXJpbmcgdGFza3MsIHdoZW4gbWVhc3VyZWQgaW4gdGVybXMgb2YgcmVjYWxsIHZhbHVlcy4NCg0KIyMjIEluZHVjdGl2ZQ0KDQpGaXJzdCwgd2Ugd2lsbCBjaGVjayB0aGUgYXNzdW1wdGlvbnMgZm9yIGEgc3RhbmRhcmQgQU5PVkEgYW5hbHlzaXMgDQoNCmBgYHtyfQ0KbGV2ZW5lVGVzdChyZXN1bHRzX2luZHVjdGl2ZSRhdmdfcmVjYWxsLCByZXN1bHRzX2luZHVjdGl2ZSRyZW9jY3VyaW5ndGFza3MsIGNlbnRlcj1tZWRpYW4pDQptb2RlbDEgPC0gYW92KGF2Z19yZWNhbGwgfiByZW9jY3VyaW5ndGFza3MsIGRhdGE9IHJlc3VsdHNfaW5kdWN0aXZlKQ0KcGxvdChtb2RlbDEsIHdoaWNoPTIpDQpgYGANCg0KQm90aCB0aGUgYXNzdW1wdGlvbnMgb2Ygbm9ybWFsaXR5IGFuZCBob21vZ2VuZWl0eSBhcHBlYXIgdG8gYmUgdmlvbGF0ZWQsIHRodXMgd2UgZmFsbCBiYWNrIHRvIGNvbXBhcmluZyB0aGUgYXZlcmFnZSByYW5raW5ncyBvZiB0aGUgbW9kZWxzIGluc3RlYWQuDQoNCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBsZXZlbCBSZW9jY3VyaW5nIFRhc2tzIikNCnJlc3VsdHNfaW5kdWN0aXZlJHJhbmtfcmVjYWxsIDwtIHJhbmsoLXJlc3VsdHNfaW5kdWN0aXZlJGF2Z19yZWNhbGwpDQpieShyZXN1bHRzX2luZHVjdGl2ZSRyYW5rX3JlY2FsbCwgcmVzdWx0c19pbmR1Y3RpdmUkcmVvY2N1cmluZ3Rhc2tzLCBtZWFuKQ0KDQpgYGANCg0KYGBge3J9DQpyZXN1bHRzX2luZHVjdGl2ZSAlPiUNCiAgZ2dwbG90KGFlcyh4PSByZW9jY3VyaW5ndGFza3MgLHkgPXJhbmtfcmVjYWxsKSkgKyBnZW9tX3Zpb2xpbihzY2FsZT0id2lkdGgiLCBkcmF3X3F1YW50aWxlcyA9IGMoMC4yNSwgMC41LCAwLjc1KSkgKw0KICBnZ3RpdGxlKCJEaXN0cmlidXRpb24gb2YgcmFua3MgKGJhc2VkIG9uIHJlY2FsbCkiKQ0KYGBgDQoNClRoZXNlIGRhdGEgc2VlbSB0byBzdWdnZXN0IHRoYXQgYXMgdGhlIGxldmVsIG9mIHJlb2NjdXJpbmcgdGFza3MgaW5jcmVhc2VzLCB0aGUgbW9kZWxzIGdlbmVyYXRlZCBieSBJbmR1Y3RpdmUgbWluZXIgZGV0ZXJpb3JhdGUsIGFsdGhvdWdoIHRoZSBlZmZlY3Qgc2VlbXMgdG8gbGV2ZWwgb2YgYXMgd2UgcmVhY2ggaGlnaGVyIGxldmVscyBvZiByZW9jY3VyaW5nIHRhc2tzLiBUbyB0ZXN0IHRoaXMgaW1wcmVzc2lvbiBzdGF0aXN0aWNhbGx5LCB3ZSB3aWxsIHJlbHkgb24gdGhlIEtXLXRlc3QgYW5kIHRoZSBKb25ja2hlZXJlIFRlc3QuIFRoZSBsYXR0ZXIgdGVzdCB3aGV0aGVyIHRoZXJlIGlzIGEgc2lnbmlmaWNhbnQgdHJlbmQgd2l0aGluIHRoZSBkYXRhIChhY3Jvc3MgaW5jcmVhc2luZyBsZXZlbHMgb2YgcmVvY2N1cmluZ3Rhc2tzKQ0KDQpgYGB7cn0NCmtydXNrYWwudGVzdChhdmdfcmVjYWxsfiByZW9jY3VyaW5ndGFza3MsIHJlc3VsdHNfaW5kdWN0aXZlKQ0Kam9uY2toZWVyZS50ZXN0KHJlc3VsdHNfaW5kdWN0aXZlJGF2Z19yZWNhbGwsIGFzLm51bWVyaWMocmVzdWx0c19pbmR1Y3RpdmUkcmVvY2N1cmluZ3Rhc2tzKSwgbnBlcm09MTAwMCkNCg0KYGBgDQoNClRoZXNlIGFuYWx5c2VzIHNob3cgdGhhdCB0aGVyZSBpcyBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50IG5lZ2F0aXZlIHRyZW5kIGluIHRoZSByZWxhdGl2ZSBxdWFsaXR5IG9mIHRoZSBnZW5lcmF0ZWQgbW9kZWxzIGFzIHRoZSBhbW91bnQgb2YgcmVvY2N1cmluZyB0YXNrcyBpbmNyZWFzZXMuIExldCdzIHZlcmlmeSB0aGlzIGZ1cnRoZXIgYnkgY29tcGFyaW5nIGRpZmZlcmVudCBsZXZlbHMgcGFpcndpc2UuDQoNCmBgYHtyfQ0Ka3J1c2thbG1jKGF2Z19yZWNhbGwgfnJlb2NjdXJpbmd0YXNrcywgZGF0YT1yZXN1bHRzX2luZHVjdGl2ZSkNCmBgYA0KDQpUaGVzZSByZXN1bHRzIHNlZW0gdG8gY29uZmlybSBvdXIgaW1wcmVzc2lvbi4gSW5kdWN0aXZlIE1pbmVyIGlzIHNlbnNpdGl2ZSB0byBsZXZlbHMgZm9yIHJlb2NjdXJpbmcgdGFza3MsIHJlc3VsdGluZyBpbiBtb2RlbHMgb2YgbG93ZXIgcXVhbGl0eSAobWVhc3VyZWQgaW4gdGVybXMgb2YgcmVjYWxsIHZhbHVlcykuIEhvd2V2ZXIsIGZyb20gYSBsZXZlbCBvZiBhcm91bmQgMTUlIHJlb2NjdXJpbmcgdGFza3MgaW4gdGhlIGxvZywgdGhpcyBlZmZlY3Qgc2VlbXMgdG8gaGF2ZSByZWFjaGVkIGEgcGxhdGVhdSBhbmQgc3RheXMgc3RhYmxlLg0KDQojIFByZWNpc2lvbg0KDQpMZXQncyBjb250aW51ZSB3aXRoIHByZWNpc2lvbi4gRmlyc3Qgd2Ugd2lsbCB2ZXJpZnkgaWYgdGhlcmUgYXJlIHNpZ25pZmljYW50IGRpZmZlcmVuY2VzIGJldHdlZW4gdGhlIG1vZGVscyBnZW5lcmF0ZWQgYnkgZm91ciBkaWZmZXJlbnQgbWluZXJzIGluIHRlcm1zIG9mIHJlY2FsbCB2YWx1ZS4NCg0KIyMgRGlmZmVyZW5jZXMgYmV0d2VlbiBtb2RlbHMgZ2VuZXJhdGVkIGJ5IGRpZmZlcmVudCBtaW5lcnMNCkZpcnN0IHdlIGNoZWNrIHRoZSBob21vZ2VuZWl0eSB0aGUgbm9ybWFsaXR5IGFzc3VtcHRpb24uDQoNCmBgYHtyfQ0KbGV2ZW5lVGVzdChyZXN1bHRzJGF2Z19wcmVjaXNpb24sIHJlc3VsdHMkbWluZXIsIGNlbnRlcj1tZWRpYW4pDQpgYGANCg0KDQpgYGB7cn0NCm1vZGVsMSA8LSBhb3YoYXZnX3ByZWNpc2lvbiB+IG1pbmVyLCBkYXRhPXJlc3VsdHMpDQpwbG90KG1vZGVsMSwgd2hpY2g9MikNCmBgYA0KDQpUaGUgYW5hbHlzaXMgaWxsdXN0cmF0ZSB0aGF0IGJvdGggYXNzdW1wdGlvbnMgYXJlIHZpb2xhdGVkLCBzbyB3ZSBwcm9jZWVkIGJ5IGZvY3VzaW5nIG9uIHRoZSByZWxhdGl2ZSBkaWZmZXJlbmNlcyBhbmQgdGhlIHJhbmtzIG9mIHRoZSBnZW5lcmF0ZWQgbW9kZWxzLg0KDQpUaGVyZWZvcmUsIHdlIHN0YXJ0IGJ5IHJhbmtpbmcgdGhlIGRpc2NvdmVyZWQgbW9kZWxzIGJhc2VkIG9uIHRoZWlyIHByZWNpc2lvbiB2YWx1ZSBhbmQgY29tcGFyZSB0aGUgYXZlcmFnZSByYW5raW5nIHBlciBtaW5lci4NCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBNaW5lciIpDQpyZXN1bHRzJHJhbmtfcHJlY2lzaW9uIDwtIHJhbmsoLXJlc3VsdHMkYXZnX3ByZWNpc2lvbikNCmJ5KHJlc3VsdHMkcmFua19wcmVjaXNpb24sIHJlc3VsdHMkbWluZXIsIG1lYW4pDQoNCmBgYA0KDQpCYXNlZCBvbiB0aGUgYXZlcmFnZSAocHJlY2lzaW9uKSByYW5raW5nLCB0aGVzZSByZXN1bHRzIHNob3cgYWdhaW4gX0lMUF8gY3JlYXRlcyB0aGUgYmVzdCBtb2RlbHMuIEhvd2V2ZXIsIHRoZSBvcmRlciBiZXR3ZWVuIHRoZSBvdGhlciB0aHJlZSBtaW5lcnMgaXMgbm93IHNsaWdodGx5IGRpZmZlcmVudC4gQmFzZWQgb24gdGhlIHByZWNpc2lvbiBtZXRyaWMsIF9JbmR1Y3RpdmVfIG1pbmVyIHByb2R1Y2VzIHJlbGF0aXZlbHkgYmV0dGVyIG1vZGVscyB0aGFuIF9BbHBoYStfIHdoaWNoIHNlZW1zIHRvIG91dHBlcmZvcm0gX0hldXJpc3RpY3NfIG1pbmVyLiBUbyBnZXQgYW4gaWRlYSBvZiB0aGUgZGlzdHJpYnV0aW9uIG9mIHRoZSByYW5rcywgd2UgY3JlYXRlIHNvbWUgdmlvbGluIHBsb3RzLg0KDQpgYGB7cn0NCnJlc3VsdHMgJT4lDQogIGdncGxvdChhZXMoeD0gbWluZXIgLHkgPXJhbmtfcHJlY2lzaW9uKSkgKyBnZW9tX3Zpb2xpbihzY2FsZT0id2lkdGgiLCBkcmF3X3F1YW50aWxlcyA9IGMoMC4yNSwgMC41LCAwLjc1KSkgKw0KICBnZ3RpdGxlKCJEaXN0cmlidXRpb24gb2YgcmFua3MgKGJhc2VkIG9uIHByZWNpc2lvbikiKQ0KYGBgDQoNClRoZXNlIHBsb3RzIHJldmVhbCBhIGNsZWFyIGRpc3RpbmN0aW9uIGJldHdlZW4gdGhlIG1vZGVscyBnZW5lcmF0ZWQgYnkgSUxQIGFuZCBJbmR1Y3RpdmUgTWluZXIgb24gdGhlIG9uZSBoYW5kIGFuZCBBbHBoYSsgYW5kIEhldXJpc3RpY3MgTWluZXIgb24gdGhlIG90aGVyIGhhbmQuIA0KDQpOb3RlLCB0aGF0IHRoZXNlIGRpc3RyaWJ1dGlvbnMgZm9jdXMgb24gdGhlIHJlbGF0aXZlIGRpZmZlcmVuY2VzIGJldHdlZW4gdGhlIHF1YWxpdHkgb2YgdGhlIG1vZGVscywgbm90IHRoZSBhYnNvbHV0ZSBkaWZmZXJlbmNlcy4gRm9yIHB1cnBvc2Ugb2YgaWxsdXN0cmF0aW9uLCBvbmUgY2FuIGZpbmQgYSB2aW9saW4gcGxvdCBiZWxvdyB3aGljaCBzaG93cyB0aGUgZGlzdHJpYnV0aW9uIG9mIHRoZSBhY3R1YWwgcHJlY2lzaW9uIHZhbHVlcyAoaW5zdGVhZCBvZiB0aGUgcmFua3MpLg0KDQpgYGB7cn0NCnJlc3VsdHMgJT4lDQogIGdncGxvdChhZXMoeD0gbWluZXIgLHkgPWF2Z19wcmVjaXNpb24pKSArIGdlb21fdmlvbGluKHNjYWxlPSJ3aWR0aCIsIGRyYXdfcXVhbnRpbGVzID0gYygwLjI1LCAwLjUsIDAuNzUpKSArDQogIGdndGl0bGUoIkRpc3RyaWJ1dGlvbiBvZiBwcmVjaXNpb24gdmFsdWVzIikNCmBgYA0KDQpXZSB3aWxsIG5vdyBjb250aW51ZSBmb2N1c2luZyBvbiB0aGUgcmVsYXRpdmUgZGlmZmVyZW5jZXMgaW5zdGVhZC4gQmFzZWQgb24gdGhlIGFib3ZlIGFuYWx5c2lzLCBpdCBhcHBlYXJzIHRoYXQgdGhlIGZvdXIgbWluZXJzIGdlbmVyYXRlIG1vZGVscyB3aGljaCByYW5rIG9uIGF2ZXJhZ2UgZGlmZmVyZW50bHkgKGJhc2VkIG9uIHJlY2FsbCB2YWx1ZXMpLiBXZSB3aWxsIHVzZSBhIEtydXNrYWwtV2FsbGlzIHRlc3QgdG8gc2VlIGlmIHRoZXNlIGRpZmZlcmVuY2VzIGluIGF2ZXJhZ2UgcmFua2luZyBhcmUgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudC4NCg0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX3ByZWNpc2lvbiB+IG1pbmVyLCByZXN1bHRzKQ0KYGBgDQoNCg0KDQpUaGVzZSByZXN1bHRzIHNob3cgdGhhdCB0aGVyZSBwcmVjaXNpb24uIFRoZSBvcmRlciBzdWdnZXN0ZWQgYnkgdGhlIHBsb3QgaXMgX0lMUF8gPiBfSW5kdWN0aXZlXyA+IF9BbHBoYStfID4gX0hldXJpc3RpY3NfLiBXZSBjYW4gdXNlIGEgcG9zdC1ob2MgdGVzdCB0byBzZWUgd2hpY2ggb2YgdGhlc2UgZGlmZmVyZW5jZXMgYXJlIHNpZ25pZmljYW50Lg0KDQpgYGB7cn0NCmtydXNrYWxtYyhhdmdfcHJlY2lzaW9uIH4gbWluZXIsIHJlc3VsdHMpDQpgYGANCg0KSXQgYXBwZWFycyB0aGF0IGFsbCBwYWlyd2lzZSBjb21wYXJpc29ucyBhcmUgc3RhdGlzaWNhbGx5IHNpZ25pZmljYW50IGRpZmZlcmVudCBmcm9tIGVhY2ggb3RoZXIuIFRodXMsIGJhc2VkIG9uIHByZWNpc2lvbiB2YWx1ZXMsIG1vZGVscyBnZW5lcmF0ZWQgYnkgSUxQIHJhbmsgb24gYXZlcmFnZSBiZXR0ZXIgdGhhbiBtb2RlbHMgZ2VuZXJhdGVkIGJ5IHRoZSBJTFAgTWluZXIsIHdoaWNoIHJhbmsgb24gYXZlcmFnZSBiZXR0ZXIgdGhhbiBtb2RlbHMgZ2VuZXJhdGVkIGJ5IHRoZSBBbHBoYSsgTWluZXIsIHdoaWNoIHN1YnNlcXVlbnRseSByYW5rIGJldHRlciBvbiBhdmVyYWdlIHRoYW4gbW9kZWxzIGdlbmVyYXRlZCBieSB0aGUgSGV1cmlzdGljcyBNaW5lci4NCg0KIyMgSW1wYWN0IG9mIEluZnJlcXVlbnQgQmVoYXZpb3Igb24gcGVyZm9ybWFuY2Ugb2YgdGhlIE1pbmluZyBhbGdvcml0bXMNCg0KTmV4dCwgd2Ugd2FudCB0byB0ZXN0IHdoZXRoZXIgdGhlIHByZXNlbmNlL2Fic2VuY2Ugb2YgaW5mcmVxdWVudCBiZWhhdmlvciBpbiB0aGUgbG9nIGhhcyBhbiBpbXBhY3Qgb24gdGhlIGF2ZXJhZ2UgcmFua2luZyBiYXNlZCBvbiBwcmVjaXNpb24gdmFsdWVzLiBGb3IgdGhpcyB3ZSB3aWxsIHNwbGl0IHRoZSBkYXRhIGJldHdlZW4gZXhwZXJpbWVudHMgd2l0aCBpbmZyZXF1ZW50IGJlaGF2aW9yIGFuZCBleHBlcmltZW50cyB3aXRob3V0IGluZnJlcXVlbnQgYmVoYXZpb3IgYW5kIHZlcmlmeSB0aGUgcmFua2luZyBmb3IgdGhlc2UgdHdvIHN1YnNldHMuIFdlIHdpbGwgc3RhcnQgdGhlIGFuYWx5c2lzIGZvciBleHBlcmltZW50cyB3aGVyZSBpbmZyZXF1ZW50IGJlaGF2aW9yIHdhcyBhYnNlbnQuDQoNCiMjIyBBYnNlbmNlIG9mIEluZnJlcXVlbnQgQmVoYXZpb3INCg0KRmlyc3Qgd2Ugd2lsbCB0ZXN0IHRvIHNlZSBpZiB3ZSBjYW4gYXBwbHkgYSBzdGFuZGFyZCBBTk9WQSBhbmFseXNpcw0KDQpgYGB7cn0NCmxldmVuZVRlc3QocmVzdWx0c19mcmVxJGF2Z19wcmVjaXNpb24sIHJlc3VsdHNfZnJlcSRtaW5lciwgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX3ByZWNpc2lvbiB+IG1pbmVyLCBkYXRhPXJlc3VsdHNfZnJlcSkNCnBsb3QobW9kZWwxLCB3aGljaD0yKQ0KYGBgDQoNCkJvdGggdGhlIExldmVuZSdzIHRlc3QgYXMgdGhlIFFRLXBsb3QgcmV2ZWFsIHRoYXQgYm90aCB0aGUgYXNzdW1wdGlvbiBvZiBIb21vZ2VuZWl0eSBhbmQgTm9ybWFsaXR5IGFyZSB2aW9sYXRlZC4gVGhlcmVmb3JlIHdlIHdpbGwgZm9jdXMgb24gdGhlIGNvbXBhcmlzb24gb2YgYXZlcmFnZSByYW5raW5ncy4NCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBNaW5lciIpDQpyZXN1bHRzX2ZyZXEkcmFua19wcmVjaXNpb24gPC0gcmFuaygtcmVzdWx0c19mcmVxJGF2Z19wcmVjaXNpb24pDQpieShyZXN1bHRzX2ZyZXEkcmFua19wcmVjaXNpb24sIHJlc3VsdHNfZnJlcSRtaW5lciwgbWVhbikNCg0KYGBgDQoNCmBgYHtyfQ0KcmVzdWx0c19mcmVxICU+JQ0KICBnZ3Bsb3QoYWVzKHg9IG1pbmVyICx5ID1yYW5rX3ByZWNpc2lvbikpICsgZ2VvbV92aW9saW4oc2NhbGU9IndpZHRoIiwgZHJhd19xdWFudGlsZXMgPSBjKDAuMjUsIDAuNSwgMC43NSkpICsNCiAgZ2d0aXRsZSgiRGlzdHJpYnV0aW9uIG9mIHJhbmtzIChiYXNlZCBvbiBwcmVjaXNpb24pIikNCmBgYA0KDQpUaGVzZSBhbmFseXNpcyBzaG93IGEgc2ltaWxhciByYW5raW5nIGJldHdlZW4gdGhlIGZvdXIgbWluZXJzLg0KDQpXZSB3aWxsIHVzZSBhIEtydXNrYWwtV2FsbGlzIHRlc3QgdG8gc2VlIGlmIHRoZXNlIGRpZmZlcmVuY2VzIGluIGF2ZXJhZ2UgcmFua2luZyBhcmUgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudC4NCg0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX3ByZWNpc2lvbiB+IG1pbmVyLCByZXN1bHRzX2ZyZXEpDQoNCmBgYA0KDQoNCg0KVGhlc2UgcmVzdWx0cyBzaG93IHRoYXQgdGhlcmUgYXJlIHNpZ25pZmljYW50IHJhbmtpbmcgZGlmZmVyZW5jZXMgYmV0d2VlbiB0aGUgbW9kZWxzIGNyZWF0ZWQgYnkgdGhlIGZvdXIgbWluZXJzIGluIHRlcm1zIG9mIHByZWNpc2lvbi4gVGhlIG9yZGVyIHN1Z2dlc3RlZCBieSB0aGUgcGxvdCBpcyBfSUxQXyA+IF9JbmR1Y3RpdmVfID4gX0FscGhhK18gPiBfSGV1cmlzdGljc18uIFdlIGNhbiB1c2UgYSBwb3N0LWhvYyB0ZXN0IHRvIHNlZSB3aGljaCBvZiB0aGVzZSBkaWZmZXJlbmNlcyBhcmUgc2lnbmlmaWNhbnQuDQoNCmBgYHtyfQ0Ka3J1c2thbG1jKGF2Z19wcmVjaXNpb24gfiBtaW5lciwgcmVzdWx0c19mcmVxKQ0KYGBgDQoNCkl0IGFwcGVhcnMgdGhhdCBhbGwgcGFpcndpc2UgY29tcGFyaXNvbnMgYXJlIHN0YXRpc2ljYWxseSBzaWduaWZpY2FudCBkaWZmZXJlbnQgZnJvbSBlYWNoIG90aGVyLiANCg0KDQojIyMgUHJlc2VuY2Ugb2YgSW5mcmVxdWVudCBCZWhhdmlvcg0KDQpGaXJzdCB3ZSB3aWxsIHRlc3QgdG8gc2VlIGlmIHdlIGNhbiBhcHBseSBhIHN0YW5kYXJkIEFOT1ZBIGFuYWx5c2lzDQpgYGB7cn0NCmxldmVuZVRlc3QocmVzdWx0c19pbmZyZXEkYXZnX3ByZWNpc2lvbiwgcmVzdWx0c19pbmZyZXEkbWluZXIsIGNlbnRlcj1tZWRpYW4pDQptb2RlbDEgPC0gYW92KGF2Z19wcmVjaXNpb24gfiBtaW5lciwgZGF0YT1yZXN1bHRzX2luZnJlcSkNCnBsb3QobW9kZWwxLCB3aGljaD0yKQ0KYGBgDQoNCkJvdGggdGhlIExldmVuZSdzIHRlc3QgYXMgdGhlIFFRLXBsb3QgcmV2ZWFsIHRoYXQgYm90aCB0aGUgYXNzdW1wdGlvbiBvZiBIb21vZ2VuZWl0eSBhbmQgTm9ybWFsaXR5IGFyZSB2aW9sYXRlZC4gVGhlcmVmb3JlIHdlIHdpbGwgZm9jdXMgb24gdGhlIGNvbXBhcmlzb24gb2YgYXZlcmFnZSByYW5raW5ncy4NCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBNaW5lciIpDQpyZXN1bHRzX2luZnJlcSRyYW5rX3ByZWNpc2lvbiA8LSByYW5rKC1yZXN1bHRzX2luZnJlcSRhdmdfcHJlY2lzaW9uKQ0KYnkocmVzdWx0c19pbmZyZXEkcmFua19wcmVjaXNpb24sIHJlc3VsdHNfaW5mcmVxJG1pbmVyLCBtZWFuKQ0KDQpgYGANCg0KYGBge3J9DQpyZXN1bHRzX2luZnJlcSAlPiUNCiAgZ2dwbG90KGFlcyh4PSBtaW5lciAseSA9cmFua19wcmVjaXNpb24pKSArIGdlb21fdmlvbGluKHNjYWxlPSJ3aWR0aCIsIGRyYXdfcXVhbnRpbGVzID0gYygwLjI1LCAwLjUsIDAuNzUpKSArDQogIGdndGl0bGUoIkRpc3RyaWJ1dGlvbiBvZiByYW5rcyAoYmFzZWQgb24gcHJlY2lzaW9uKSIpDQpgYGANCg0KVGhlc2UgYW5hbHlzaXMgc2hvdyBhZ2FpbiBhIHNpbWlsYXIgcmFua2luZyBiZXR3ZWVuIHRoZSBmb3VyIG1pbmVycy4gDQoNCldlIHdpbGwgdXNlIGEgS3J1c2thbC1XYWxsaXMgdGVzdCB0byBzZWUgaWYgdGhlc2UgZGlmZmVyZW5jZXMgaW4gYXZlcmFnZSByYW5raW5nIGFyZSBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50Lg0KDQpgYGB7cn0NCmtydXNrYWwudGVzdChhdmdfcHJlY2lzaW9uIH4gbWluZXIsIHJlc3VsdHNfaW5mcmVxKQ0KDQpgYGANCg0KVGhlc2UgcmVzdWx0cyBzaG93IHRoYXQgdGhlcmUgYXJlIHNpZ25pZmljYW50IHJhbmtpbmcgZGlmZmVyZW5jZXMgYmV0d2VlbiB0aGUgbW9kZWxzIGNyZWF0ZWQgYnkgdGhlIGZvdXIgbWluZXJzIGluIHRlcm1zIG9mIHByZWNpc2lvbi4gVGhlIG9yZGVyIHN1Z2dlc3RlZCBieSB0aGUgcGxvdCBpcyBfSUxQXyA+IF9JbmR1Y3RpdmVfID4gX0FscGhhK18gPiBfSGV1cmlzdGljc18uIFdlIGNhbiB1c2UgYSBwb3N0LWhvYyB0ZXN0IHRvIHNlZSB3aGljaCBvZiB0aGVzZSBkaWZmZXJlbmNlcyBhcmUgc2lnbmlmaWNhbnQuDQoNCmBgYHtyfQ0Ka3J1c2thbG1jKGF2Z19wcmVjaXNpb24gfiBtaW5lciwgcmVzdWx0c19pbmZyZXEpDQpgYGANCg0KSXQgYXBwZWFycyB0aGF0IGFsbCBwYWlyd2lzZSBjb21wYXJpc29ucyBhcmUgc3RhdGlzaWNhbGx5IHNpZ25pZmljYW50IGRpZmZlcmVudCBmcm9tIGVhY2ggb3RoZXIsIGV4Y2VwdCBmb3IgdGhlIGRpZmZlcmVuY2UgYmV0d2VlbiBBbHBoYSsgYW5kIEhldXJpc3RpY3MuDQoNCiMjIEltcGFjdCBvZiBhbW91bnQgb2YgcmVvY2N1cmluZyB0YXNrcyBvbiBwZXJmb3JtYW5jZSBvZiB0aGUgTWluaW5nIEFsZ29yaXRobQ0KDQpOZXh0LCB3ZSB3aWxsIGludmVzdGlnYXRlIGhvdyB0aGUgcGVyZm9ybWFuY2UgZWFjaCBvZiB0aGUgbWluaW5nIGFsZ29yaXRobSBpcyBpbmZsdWVuY2VkIGJ5IHRoZSBsZXZlbCBvZiByZW9jY3VyaW5nIHRhc2tzIGluIHRoZSBsb2cgZmlsZXMuDQoNCiMjIyBBbHBoYSsNCg0KRmlyc3QsIHdlIHdpbGwgY2hlY2sgdGhlIGFzc3VtcHRpb25zIGZvciBhIHN0YW5kYXJkIEFOT1ZBIGFuYWx5c2lzIA0KDQpgYGB7cn0NCmxldmVuZVRlc3QocmVzdWx0c19hbHBoYSRhdmdfcHJlY2lzaW9uLCByZXN1bHRzX2FscGhhJHJlb2NjdXJpbmd0YXNrcywgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX3JlY2FsbCB+IHJlb2NjdXJpbmd0YXNrcywgZGF0YT0gcmVzdWx0c19hbHBoYSkNCnBsb3QobW9kZWwxLCB3aGljaD0yKQ0KYGBgDQoNCldoaWxlIHRoZSBhc3N1bXB0aW9uIG9mIGhvbW9nbmVpdHkgc2VlbXMgdG8gaG9sZCwgdGhlIGFzc3VtcHRpb25zIG9mIG5vcm1hbGl0eSBhcHBlYXJzIHRvIGJlIHZpb2xhdGVkLiBUbyByZW1haW4gY29uc2lzdGVudCwgd2UgZmFsbCBiYWNrIHRvIGNvbXBhcmluZyB0aGUgYXZlcmFnZSByYW5raW5ncyBvZiB0aGUgbW9kZWxzIGluc3RlYWQuDQoNCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBsZXZlbCBSZW9jY3VyaW5nIFRhc2tzIikNCnJlc3VsdHNfYWxwaGEkcmFua19wcmVjaXNpb24gPC0gcmFuaygtcmVzdWx0c19hbHBoYSRhdmdfcHJlY2lzaW9uKQ0KYnkocmVzdWx0c19hbHBoYSRyYW5rX3ByZWNpc2lvbiwgcmVzdWx0c19hbHBoYSRyZW9jY3VyaW5ndGFza3MsIG1lYW4pDQoNCmBgYA0KDQpgYGB7cn0NCnJlc3VsdHNfYWxwaGEgJT4lDQogIGdncGxvdChhZXMoeD0gcmVvY2N1cmluZ3Rhc2tzICx5ID1yYW5rX3ByZWNpc2lvbikpICsgZ2VvbV92aW9saW4oc2NhbGU9IndpZHRoIiwgZHJhd19xdWFudGlsZXMgPSBjKDAuMjUsIDAuNSwgMC43NSkpICsNCiAgZ2d0aXRsZSgiRGlzdHJpYnV0aW9uIG9mIHJhbmtzIChiYXNlZCBvbiBwcmVjaXNpb24pIikNCmBgYA0KDQpUaGVzZSBkYXRhIHNlZW0gdG8gc3VnZ2VzdCB0aGF0IGFzIHRoZSBsZXZlbCBvZiByZW9jY3VyaW5nIHRhc2tzIGluY3JlYXNlcywgdGhlIG1vZGVscyBnZW5lcmF0ZWQgYnkgQWxwaGEgbWluZXIgZGV0ZXJpb3JhdGUuIEhvd2V2ZXIsIGl0IHNlZW1zIHRoYXQgdGhpcyBlZmZlY3QgcmVhbGx5IHN0YXJ0cyB0byBzaG93IGZyb20gYSBsZXZlbCBvZiAxNSUgcmVvY2N1cmluZyB0YXNrcy4gVG8gdGVzdCB0aGlzIGltcHJlc3Npb24gc3RhdGlzdGljYWxseSwgd2Ugd2lsbCByZWx5IG9uIHRoZSBLVy10ZXN0IGFuZCB0aGUgSm9uY2toZWVyZSBUZXN0LiANCg0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX3ByZWNpc2lvbn4gcmVvY2N1cmluZ3Rhc2tzLCByZXN1bHRzX2FscGhhKQ0Kam9uY2toZWVyZS50ZXN0KHJlc3VsdHNfYWxwaGEkYXZnX3ByZWNpc2lvbiwgYXMubnVtZXJpYyhyZXN1bHRzX2FscGhhJHJlb2NjdXJpbmd0YXNrcyksIG5wZXJtPTEwMDApDQoNCmBgYA0KDQpUaGVzZSBhbmFseXNlcyBzaG93IHRoYXQgdGhlcmUgaXMgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCBuZWdhdGl2ZSB0cmVuZCBpbiB0aGUgcmVsYXRpdmUgcXVhbGl0eSBvZiB0aGUgZ2VuZXJhdGVkIG1vZGVscyBhcyB0aGUgYW1vdW50IG9mIHJlb2NjdXJpbmcgdGFza3MgaW5jcmVhc2VzLiBMZXQncyB2ZXJpZnkgdGhpcyBmdXJ0aGVyIGJ5IGNvbXBhcmluZyBkaWZmZXJlbnQgbGV2ZWxzIHBhaXJ3aXNlLg0KDQpgYGB7cn0NCmtydXNrYWxtYyhhdmdfcHJlY2lzaW9uIH5yZW9jY3VyaW5ndGFza3MsIGRhdGE9cmVzdWx0c19hbHBoYSkNCmBgYA0KDQpUaGVzZSByZXN1bHRzIGlsbHVzdHJhdGUgdGhhdCB0aGUgcXVhbGl0eSBvZiB0aGUgbW9kZWxzIGRlY3JlYXNlIHNpZ25pZmljYW50bHkgd2hlbmV2ZXIgdGhlIGxldmVsIG9mIHJlb2NjdXJpbmcgdGFza3MgaW5jcmVhc2UgYnkgMTUlLg0KDQojIyMgSGV1cmlzdGljcw0KDQpGaXJzdCwgd2Ugd2lsbCBjaGVjayB0aGUgYXNzdW1wdGlvbnMgZm9yIGEgc3RhbmRhcmQgQU5PVkEgYW5hbHlzaXMgDQoNCmBgYHtyfQ0KbGV2ZW5lVGVzdChyZXN1bHRzX2hldXJpc3RpY3MkYXZnX3ByZWNpc2lvbiwgcmVzdWx0c19oZXVyaXN0aWNzJHJlb2NjdXJpbmd0YXNrcywgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX3ByZWNpc2lvbiB+IHJlb2NjdXJpbmd0YXNrcywgZGF0YT0gcmVzdWx0c19oZXVyaXN0aWNzKQ0KcGxvdChtb2RlbDEsIHdoaWNoPTIpDQpgYGANCg0KQWdhaW4sIHRoZSBMZXZlbmUncyB0ZXN0IGlzIG5vdCBjb25jbHVzaXZlIGFib3V0IHRoZSB2aW9sYXRpb24gb2YgdGhlIGhvbW9nZW5laXR5IGFzc3VtcHRpb24sIGJ1dCB0aGUgUVEtcGxvdCBzaG93cyB0aGF0IHRoZSByZWlzdWRhbHMgYXJlIG5vdCBub3JtYWxseSBkaXN0cmlidXRpb24uIFRoZXJlZm9yZSwgd2UgZmFsbCBiYWNrIHRvIGNvbXBhcmluZyB0aGUgYXZlcmFnZSByYW5raW5ncyBvZiB0aGUgbW9kZWxzIGluc3RlYWQuDQoNCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBsZXZlbCBSZW9jY3VyaW5nIFRhc2tzIikNCnJlc3VsdHNfaGV1cmlzdGljcyRyYW5rX3ByZWNpc2lvbiA8LSByYW5rKC1yZXN1bHRzX2hldXJpc3RpY3MkYXZnX3ByZWNpc2lvbikNCmJ5KHJlc3VsdHNfaGV1cmlzdGljcyRyYW5rX3ByZWNpc2lvbiwgcmVzdWx0c19oZXVyaXN0aWNzJHJlb2NjdXJpbmd0YXNrcywgbWVhbikNCg0KYGBgDQoNCmBgYHtyfQ0KcmVzdWx0c19oZXVyaXN0aWNzICU+JQ0KICBnZ3Bsb3QoYWVzKHg9IHJlb2NjdXJpbmd0YXNrcyAseSA9cmFua19wcmVjaXNpb24pKSArIGdlb21fdmlvbGluKHNjYWxlPSJ3aWR0aCIsIGRyYXdfcXVhbnRpbGVzID0gYygwLjI1LCAwLjUsIDAuNzUpKSArDQogIGdndGl0bGUoIkRpc3RyaWJ1dGlvbiBvZiByYW5rcyAoYmFzZWQgb24gcHJlY2lzaW9uKSIpDQpgYGANCg0KVGhlc2UgZGF0YSBzZWVtIHRvIHN1Z2dlc3QgdGhhdCB0aGUgcHJlY2lzaW9uIHNjb3JlIG9mIG1vZGVscyBkaXNjb3ZlcmVkIGJ5IHRoZSBIZXVyaXN0aWNzIE1pbmVyIGlzIHJlbGF0aXZlbHkgc3RhYmxlIGFnYWluc3QgaW5jcmVhc2luZyBsZXZlbHMgb2YgcmVvY2N1cmluZyB0YXNrcy4gVG8gdGVzdCB0aGlzIGltcHJlc3Npb24gc3RhdGlzdGljYWxseSwgd2Ugd2lsbCByZWx5IG9uIHRoZSBLVy10ZXN0IGFuZCB0aGUgSm9uY2toZWVyZSBUZXN0LiANCg0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX3ByZWNpc2lvbn4gcmVvY2N1cmluZ3Rhc2tzLCByZXN1bHRzX2hldXJpc3RpY3MpDQpqb25ja2hlZXJlLnRlc3QocmVzdWx0c19oZXVyaXN0aWNzJGF2Z19wcmVjaXNpb24sIGFzLm51bWVyaWMocmVzdWx0c19oZXVyaXN0aWNzJHJlb2NjdXJpbmd0YXNrcyksIG5wZXJtPTEwMDApDQoNCmBgYA0KDQpUaGVzZSBhbmFseXNlcyBzaG93IHRoYXQgdGhlcmUgaXMgaW5kZWVkIG5vIHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnQgIHRyZW5kIGluIHRoZSByZWxhdGl2ZSBxdWFsaXR5IG9mIHRoZSBnZW5lcmF0ZWQgbW9kZWxzIGFzIHRoZSBhbW91bnQgb2YgcmVvY2N1cmluZyB0YXNrcyBpbmNyZWFzZXMuDQoNCiMjIyBJTFANCg0KRmlyc3QsIHdlIHdpbGwgY2hlY2sgdGhlIGFzc3VtcHRpb25zIGZvciBhIHN0YW5kYXJkIEFOT1ZBIGFuYWx5c2lzIA0KDQpgYGB7cn0NCmxldmVuZVRlc3QocmVzdWx0c19pbHAkYXZnX3ByZWNpc2lvbiwgcmVzdWx0c19pbHAkcmVvY2N1cmluZ3Rhc2tzLCBjZW50ZXI9bWVkaWFuKQ0KbW9kZWwxIDwtIGFvdihhdmdfcHJlY2lzaW9uIH4gcmVvY2N1cmluZ3Rhc2tzLCBkYXRhPSByZXN1bHRzX2lscCkNCnBsb3QobW9kZWwxLCB3aGljaD0yKQ0KYGBgDQoNCkJvdGggdGhlIGFzc3VtcHRpb25zIG9mIG5vcm1hbGl0eSBhbmQgaG9tb2dlbmVpdHkgYXBwZWFyIHRvIGJlIHZpb2xhdGVkLCB0aHVzIHdlIGZhbGwgYmFjayB0byBjb21wYXJpbmcgdGhlIGF2ZXJhZ2UgcmFua2luZ3Mgb2YgdGhlIG1vZGVscyBpbnN0ZWFkLg0KDQoNCmBgYHtyfQ0KcHJpbnQoIkF2ZXJhZ2UgUmFuayBwZXIgbGV2ZWwgUmVvY2N1cmluZyBUYXNrcyIpDQpyZXN1bHRzX2lscCRyYW5rX3ByZWNpc2lvbiA8LSByYW5rKC1yZXN1bHRzX2lscCRhdmdfcHJlY2lzaW9uKQ0KYnkocmVzdWx0c19pbHAkcmFua19wcmVjaXNpb24sIHJlc3VsdHNfaWxwJHJlb2NjdXJpbmd0YXNrcywgbWVhbikNCg0KYGBgDQoNCmBgYHtyfQ0KcmVzdWx0c19pbHAgJT4lDQogIGdncGxvdChhZXMoeD0gcmVvY2N1cmluZ3Rhc2tzICx5ID1yYW5rX3ByZWNpc2lvbikpICsgZ2VvbV92aW9saW4oc2NhbGU9IndpZHRoIiwgZHJhd19xdWFudGlsZXMgPSBjKDAuMjUsIDAuNSwgMC43NSkpICsNCiAgZ2d0aXRsZSgiRGlzdHJpYnV0aW9uIG9mIHJhbmtzIChiYXNlZCBvbiBQcmVjaXNpb24pIikNCmBgYA0KDQpUaGVzZSBkYXRhIHNlZW0gdG8gc3VnZ2VzdCB0aGF0IHRoZSBwcmVjaXNpb24gc2NvcmUgb2YgdGhlIG1vZGVscyBnZW5lcmF0ZWQgYnkgSUxQIGlzIHZlcnkgc2Vuc2l0aXZlIHRvIHRoZSBsZXZlbCBvZiByZW9jY3VyaW5nIHRhc2tzLiBUbyB0ZXN0IHRoaXMgaW1wcmVzc2lvbiBzdGF0aXN0aWNhbGx5LCB3ZSB3aWxsIHJlbHkgb24gdGhlIEtXLXRlc3QgYW5kIHRoZSBKb25ja2hlZXJlIFRlc3QuIA0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX3ByZWNpc2lvbn4gcmVvY2N1cmluZ3Rhc2tzLCByZXN1bHRzX2lscCkNCmpvbmNraGVlcmUudGVzdChyZXN1bHRzX2lscCRhdmdfcHJlY2lzaW9uLCBhcy5udW1lcmljKHJlc3VsdHNfaWxwJHJlb2NjdXJpbmd0YXNrcyksIG5wZXJtPTEwMDApDQoNCmBgYA0KDQpUaGVzZSBhbmFseXNlcyBzaG93IHRoYXQgdGhlcmUgaXMgYSBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50IG5lZ2F0aXZlIHRyZW5kIGluIHRoZSByZWxhdGl2ZSBxdWFsaXR5IG9mIHRoZSBnZW5lcmF0ZWQgbW9kZWxzIGFzIHRoZSBhbW91bnQgb2YgcmVvY2N1cmluZyB0YXNrcyBpbmNyZWFzZXMuIExldCdzIHZlcmlmeSB0aGlzIGZ1cnRoZXIgYnkgY29tcGFyaW5nIGRpZmZlcmVudCBsZXZlbHMgcGFpcndpc2UuDQoNCmBgYHtyfQ0Ka3J1c2thbG1jKGF2Z19wcmVjaXNpb24gfnJlb2NjdXJpbmd0YXNrcywgZGF0YT1yZXN1bHRzX2lscCkNCmBgYA0KDQpUaGVzZSByZXN1bHRzIGlsbHVzdHJhdGUgdGhlcmUgaXMgYSBjbGVhciBzaWduaWZpY2FudCB0cmVuZCwgd2hpY2ggc29tZWhvdyBsZXZlbHMgb2YgYXJvdW5kIDI1JS4NCg0KIyMjIEluZHVjdGl2ZQ0KDQpGaXJzdCwgd2Ugd2lsbCBjaGVjayB0aGUgYXNzdW1wdGlvbnMgZm9yIGEgc3RhbmRhcmQgQU5PVkEgYW5hbHlzaXMgDQoNCmBgYHtyfQ0KbGV2ZW5lVGVzdChyZXN1bHRzX2luZHVjdGl2ZSRhdmdfcHJlY2lzaW9uLCByZXN1bHRzX2luZHVjdGl2ZSRyZW9jY3VyaW5ndGFza3MsIGNlbnRlcj1tZWRpYW4pDQptb2RlbDEgPC0gYW92KGF2Z19wcmVjaXNpb24gfiByZW9jY3VyaW5ndGFza3MsIGRhdGE9IHJlc3VsdHNfaW5kdWN0aXZlKQ0KcGxvdChtb2RlbDEsIHdoaWNoPTIpDQpgYGANCg0KQm90aCB0aGUgYXNzdW1wdGlvbnMgb2Ygbm9ybWFsaXR5IGFuZCBob21vZ2VuZWl0eSBhcHBlYXIgdG8gYmUgdmlvbGF0ZWQsIHRodXMgd2UgZmFsbCBiYWNrIHRvIGNvbXBhcmluZyB0aGUgYXZlcmFnZSByYW5raW5ncyBvZiB0aGUgbW9kZWxzIGluc3RlYWQuDQoNCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBsZXZlbCBSZW9jY3VyaW5nIFRhc2tzIikNCnJlc3VsdHNfaW5kdWN0aXZlJHJhbmtfcHJlY2lzaW9uIDwtIHJhbmsoLXJlc3VsdHNfaW5kdWN0aXZlJGF2Z19wcmVjaXNpb24pDQpieShyZXN1bHRzX2luZHVjdGl2ZSRyYW5rX3ByZWNpc2lvbiwgcmVzdWx0c19pbmR1Y3RpdmUkcmVvY2N1cmluZ3Rhc2tzLCBtZWFuKQ0KDQpgYGANCg0KYGBge3J9DQpyZXN1bHRzX2luZHVjdGl2ZSAlPiUNCiAgZ2dwbG90KGFlcyh4PSByZW9jY3VyaW5ndGFza3MgLHkgPXJhbmtfcHJlY2lzaW9uKSkgKyBnZW9tX3Zpb2xpbihzY2FsZT0id2lkdGgiLCBkcmF3X3F1YW50aWxlcyA9IGMoMC4yNSwgMC41LCAwLjc1KSkgKw0KICBnZ3RpdGxlKCJEaXN0cmlidXRpb24gb2YgcmFua3MgKGJhc2VkIG9uIHByZWNpc2lvbikiKQ0KYGBgDQoNClRoZXNlIGRhdGEgc2VlbSB0byBzdWdnZXN0IHRoYXQgYXMgdGhlIGxldmVsIG9mIHJlb2NjdXJpbmcgdGFza3MgaW5jcmVhc2VzLCB0aGUgbW9kZWxzIGdlbmVyYXRlZCBieSBJbmR1Y3RpdmUgbWluZXIgZGV0ZXJpb3JhdGUuIEl0IGlzIGNsZWFyIGZyb20gdGhlIHZpb2xpbiBwbG90IHRoYXQgdGhlIHByZXNlbmNlIG9mIHJlb2NjdXJpbmcgdGFza3MgdnMgdGhlIGFic2VuY2Ugb2YgcmVvY2N1cmluZyB0YXNrcyBtYWtlcyBhIGJpZyBkaWZmZXJlbmNlIGZvciB0aGUgcHJlY2lzaW9uIHZhbHVlcyBvZiBtb2RlbHMgZ2VuZXJhdGVkIGJ5IEluZHVjdGl2ZSBNaW5lci4gVG8gdGVzdCB0aGlzIGltcHJlc3Npb24gc3RhdGlzdGljYWxseSwgd2Ugd2lsbCByZWx5IG9uIHRoZSBLVy10ZXN0IGFuZCB0aGUgSm9uY2toZWVyZSBUZXN0LiANCg0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX3ByZWNpc2lvbn4gcmVvY2N1cmluZ3Rhc2tzLCByZXN1bHRzX2luZHVjdGl2ZSkNCmpvbmNraGVlcmUudGVzdChyZXN1bHRzX2luZHVjdGl2ZSRhdmdfcHJlY2lzaW9uLCBhcy5udW1lcmljKHJlc3VsdHNfaW5kdWN0aXZlJHJlb2NjdXJpbmd0YXNrcyksIG5wZXJtPTEwMDApDQoNCmBgYA0KDQpUaGVzZSBhbmFseXNlcyBzaG93IHRoYXQgdGhlcmUgaXMgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCBuZWdhdGl2ZSB0cmVuZCBpbiB0aGUgcmVsYXRpdmUgcXVhbGl0eSBvZiB0aGUgZ2VuZXJhdGVkIG1vZGVscyBhcyB0aGUgYW1vdW50IG9mIHJlb2NjdXJpbmcgdGFza3MgaW5jcmVhc2VzLiBMZXQncyB2ZXJpZnkgdGhpcyBmdXJ0aGVyIGJ5IGNvbXBhcmluZyBkaWZmZXJlbnQgbGV2ZWxzIHBhaXJ3aXNlLg0KDQpgYGB7cn0NCmtydXNrYWxtYyhhdmdfcHJlY2lzaW9uIH5yZW9jY3VyaW5ndGFza3MsIGRhdGE9cmVzdWx0c19pbmR1Y3RpdmUpDQpgYGANCg0KVGhlc2UgcmVzdWx0cyBzZWVtIHRvIGNvbmZpcm0gb3VyIGltcHJlc3Npb24uIEluZHVjdGl2ZSBNaW5lciBpcyBzZW5zaXRpdmUgdG8gbGV2ZWxzIGZvciByZW9jY3VyaW5nIHRhc2tzLCByZXN1bHRpbmcgaW4gbW9kZWxzIG9mIGxvd2VyIHF1YWxpdHkgKG1lYXN1cmVkIGluIHRlcm1zIG9mIHByZWNpc2lvbiB2YWx1ZXMpLiBIb3dldmVyLCBmcm9tIGEgbGV2ZWwgb2YgYXJvdW5kIDE1JSByZW9jY3VyaW5nIHRhc2tzIGluIHRoZSBsb2csIHRoaXMgZWZmZWN0IHNlZW1zIHRvIGhhdmUgcmVhY2hlZCBhIHBsYXRlYXUgYW5kIHN0YXlzIHN0YWJsZS4NCg0KDQojIEYxDQoNCkxldCdzIGNvbnRpbnVlIHdpdGggdGhlIEYxLXNjb3JlLiBGaXJzdCB3ZSB3aWxsIHZlcmlmeSBpZiB0aGVyZSBhcmUgc2lnbmlmaWNhbnQgZGlmZmVyZW5jZXMgYmV0d2VlbiB0aGUgbW9kZWxzIGdlbmVyYXRlZCBieSBmb3VyIGRpZmZlcmVudCBtaW5lcnMgaW4gdGVybXMgb2YgcmVjYWxsIHZhbHVlLg0KDQojIyBEaWZmZXJlbmNlcyBiZXR3ZWVuIG1vZGVscyBnZW5lcmF0ZWQgYnkgZGlmZmVyZW50IG1pbmVycw0KRmlyc3Qgd2UgY2hlY2sgdGhlIGhvbW9nZW5laXR5IHRoZSBub3JtYWxpdHkgYXNzdW1wdGlvbi4NCg0KYGBge3J9DQpsZXZlbmVUZXN0KHJlc3VsdHMkYXZnX2YxLCByZXN1bHRzJG1pbmVyLCBjZW50ZXI9bWVkaWFuKQ0KYGBgDQoNCg0KYGBge3J9DQptb2RlbDEgPC0gYW92KGF2Z19mMSB+IG1pbmVyLCBkYXRhPXJlc3VsdHMpDQpwbG90KG1vZGVsMSwgd2hpY2g9MikNCmBgYA0KDQpUaGUgYW5hbHlzaXMgaWxsdXN0cmF0ZSB0aGF0IGJvdGggYXNzdW1wdGlvbnMgYXJlIHZpb2xhdGVkLCBzbyB3ZSBwcm9jZWVkIGJ5IGZvY3VzaW5nIG9uIHRoZSByZWxhdGl2ZSBkaWZmZXJlbmNlcyBhbmQgdGhlIHJhbmtzIG9mIHRoZSBnZW5lcmF0ZWQgbW9kZWxzLg0KDQpUaGVyZWZvcmUsIHdlIHN0YXJ0IGJ5IHJhbmtpbmcgdGhlIGRpc2NvdmVyZWQgbW9kZWxzIGJhc2VkIG9uIHRoZWlyIHByZWNpc2lvbiB2YWx1ZSBhbmQgY29tcGFyZSB0aGUgYXZlcmFnZSByYW5raW5nIHBlciBtaW5lci4NCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBNaW5lciIpDQpyZXN1bHRzJHJhbmtfZjEgPC0gcmFuaygtcmVzdWx0cyRhdmdfZjEpDQpieShyZXN1bHRzJHJhbmtfZjEsIHJlc3VsdHMkbWluZXIsIG1lYW4pDQoNCmBgYA0KDQpCYXNlZCBvbiB0aGUgYXZlcmFnZSAocHJlY2lzaW9uKSByYW5raW5nLCB0aGVzZSByZXN1bHRzIHNob3cgYWdhaW4gX0lMUF8gY3JlYXRlcyB0aGUgYmVzdCBtb2RlbHMuIFRoZSBvcmRlciBiZXR3ZWVuIHRoZSBvdGhlciB0aHJlZSBtaW5lcnMgaXMgbm93IHN1Y2ggdGhhdCBfSW5kdWN0aXZlXyBtaW5lciBwcm9kdWNlcyByZWxhdGl2ZWx5IGJldHRlciBtb2RlbHMgdGhhbiBfSGV1cmlzdGljc18gd2hpY2ggc2VlbXMgdG8gb3V0cGVyZm9ybSBfQWxwaGErXyBtaW5lci4gVG8gZ2V0IGFuIGlkZWEgb2YgdGhlIGRpc3RyaWJ1dGlvbiBvZiB0aGUgcmFua3MsIHdlIGNyZWF0ZSBzb21lIHZpb2xpbiBwbG90cy4NCg0KYGBge3J9DQpyZXN1bHRzICU+JQ0KICBnZ3Bsb3QoYWVzKHg9IG1pbmVyICx5ID1yYW5rX2YxKSkgKyBnZW9tX3Zpb2xpbihzY2FsZT0id2lkdGgiLCBkcmF3X3F1YW50aWxlcyA9IGMoMC4yNSwgMC41LCAwLjc1KSkgKw0KICBnZ3RpdGxlKCJEaXN0cmlidXRpb24gb2YgcmFua3MgKGJhc2VkIG9uIGYxKSIpDQpgYGANCg0KVGhlc2UgcGxvdHMgcmV2ZWFsIGEgbWl4ZWQgcGljdHVyZSBiZXR3ZWVuIHRoZSByZXN1bHRzIG9uIHJlY2FsbCBhbmQgcHJlY2lzaW9uLCB3aGljaCB3YXMgdG8gYmUgZXhwZWN0ZWQuIElMUCBvdXRwZXJmb3JtcyB0aGUgb3RoZXIgbWluZXJzLCBmb2xsb3dlZCBieSBJbmR1Y3RpdmUgYW5kIHRoZW4gYnkgQWxwaGErIGFuZCBIZXVyaXN0aWNzIHdoaWNoIHNlZW0gbW9yZSBvciBsZXNzIGVxdWFsbHkgd2VsbCBwZXJmb3JtaW5nLg0KDQpOb3RlLCB0aGF0IHRoZXNlIGRpc3RyaWJ1dGlvbnMgZm9jdXMgb24gdGhlIHJlbGF0aXZlIGRpZmZlcmVuY2VzIGJldHdlZW4gdGhlIHF1YWxpdHkgb2YgdGhlIG1vZGVscywgbm90IHRoZSBhYnNvbHV0ZSBkaWZmZXJlbmNlcy4gRm9yIHB1cnBvc2Ugb2YgaWxsdXN0cmF0aW9uLCBvbmUgY2FuIGZpbmQgYSB2aW9saW4gcGxvdCBiZWxvdyB3aGljaCBzaG93cyB0aGUgZGlzdHJpYnV0aW9uIG9mIHRoZSBhY3R1YWwgZjEgdmFsdWVzIChpbnN0ZWFkIG9mIHRoZSByYW5rcykuDQoNCmBgYHtyfQ0KcmVzdWx0cyAlPiUNCiAgZ2dwbG90KGFlcyh4PSBtaW5lciAseSA9YXZnX2YxKSkgKyBnZW9tX3Zpb2xpbihzY2FsZT0id2lkdGgiLCBkcmF3X3F1YW50aWxlcyA9IGMoMC4yNSwgMC41LCAwLjc1KSkgKw0KICBnZ3RpdGxlKCJEaXN0cmlidXRpb24gb2YgZjEgdmFsdWVzIikNCmBgYA0KDQpXZSB3aWxsIG5vdyBjb250aW51ZSB0byBmb2N1cyBvbiB0aGUgcmVsYXRpdmUgZGlmZmVyZW5jZXMgaW5zdGVhZC4gQmFzZWQgb24gdGhlIGFib3ZlIGFuYWx5c2lzLCBpdCBhcHBlYXJzIHRoYXQgdGhlIGZvdXIgbWluZXJzIGdlbmVyYXRlIG1vZGVscyB3aGljaCByYW5rIG9uIGF2ZXJhZ2UgZGlmZmVyZW50bHkgKGJhc2VkIG9uIHJlY2FsbCB2YWx1ZXMpLiBXZSB3aWxsIHVzZSBhIEtydXNrYWwtV2FsbGlzIHRlc3QgdG8gc2VlIGlmIHRoZXNlIGRpZmZlcmVuY2VzIGluIGF2ZXJhZ2UgcmFua2luZyBhcmUgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudC4NCg0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX2YxIH4gbWluZXIsIHJlc3VsdHMpDQpgYGANCg0KDQoNClRoZXNlIHJlc3VsdHMgc2hvdyB0aGF0IHRoZXJlIGlzIGEgc2lnbmlmaWNhbnQgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSBmb3VyIG1pbmVycy4gVGhlIG9yZGVyIHN1Z2dlc3RlZCBieSB0aGUgcmVzdWx0cyBpcyBfSUxQXyA+IF9JbmR1Y3RpdmVfID4gX0hldXJpc3RpY3NfID4gX0FscGhhK18gLiBXZSBjYW4gdXNlIGEgcG9zdC1ob2MgdGVzdCB0byBzZWUgd2hpY2ggb2YgdGhlc2UgZGlmZmVyZW5jZXMgYXJlIHNpZ25pZmljYW50Lg0KDQpgYGB7cn0NCmtydXNrYWxtYyhhdmdfZjEgfiBtaW5lciwgcmVzdWx0cykNCmBgYA0KDQpJdCBhcHBlYXJzIHRoYXQgZXhjZXB0IGZvciB0aGUgZGlmZmVyZW5jZSBiZXR3ZWVuIEFscGhhKyBhbmQgSGV1cmlzdGljcywgYWxsIHBhaXJ3aXNlIGNvbXBhcmlzb25zIGFyZSBzdGF0aXNpY2FsbHkgc2lnbmlmaWNhbnQuIFRodXMsIGJhc2VkIG9uIGYxIHZhbHVlcywgbW9kZWxzIGdlbmVyYXRlZCBieSBJTFAgcmFuayBvbiBhdmVyYWdlIGJldHRlciB0aGFuIG1vZGVscyBnZW5lcmF0ZWQgYnkgdGhlIElMUCBNaW5lciwgd2hpY2ggcmFuayBvbiBhdmVyYWdlIGJldHRlciB0aGFuIG1vZGVscyBnZW5lcmF0ZWQgYnkgdGhlIEFscGhhKyBvciBIZXVyaXN0aWNzIE1pbmVyLg0KDQojIyBJbXBhY3Qgb2YgSW5mcmVxdWVudCBCZWhhdmlvciBvbiBwZXJmb3JtYW5jZSBvZiB0aGUgTWluaW5nIGFsZ29yaXRtcw0KDQpOZXh0LCB3ZSB3YW50IHRvIHRlc3Qgd2hldGhlciB0aGUgcHJlc2VuY2UvYWJzZW5jZSBvZiBpbmZyZXF1ZW50IGJlaGF2aW9yIGluIHRoZSBsb2cgaGFzIGFuIGltcGFjdCBvbiB0aGUgYXZlcmFnZSByYW5raW5nIGJhc2VkIG9uIGYxIHZhbHVlcy4gRm9yIHRoaXMgd2Ugd2lsbCBzcGxpdCB0aGUgZGF0YSBiZXR3ZWVuIGV4cGVyaW1lbnRzIHdpdGggaW5mcmVxdWVudCBiZWhhdmlvciBhbmQgZXhwZXJpbWVudHMgd2l0aG91dCBpbmZyZXF1ZW50IGJlaGF2aW9yIGFuZCB2ZXJpZnkgdGhlIHJhbmtpbmcgZm9yIHRoZXNlIHR3byBzdWJzZXRzLiBXZSB3aWxsIHN0YXJ0IHRoZSBhbmFseXNpcyBmb3IgZXhwZXJpbWVudHMgd2hlcmUgaW5mcmVxdWVudCBiZWhhdmlvciB3YXMgYWJzZW50Lg0KDQojIyMgQWJzZW5jZSBvZiBJbmZyZXF1ZW50IEJlaGF2aW9yDQoNCkZpcnN0IHdlIHdpbGwgdGVzdCB0byBzZWUgaWYgd2UgY2FuIGFwcGx5IGEgc3RhbmRhcmQgQU5PVkEgYW5hbHlzaXMNCg0KYGBge3J9DQpsZXZlbmVUZXN0KHJlc3VsdHNfZnJlcSRhdmdfZjEsIHJlc3VsdHNfZnJlcSRtaW5lciwgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX2YxIH4gbWluZXIsIGRhdGE9cmVzdWx0c19mcmVxKQ0KcGxvdChtb2RlbDEsIHdoaWNoPTIpDQpgYGANCg0KQm90aCB0aGUgTGV2ZW5lJ3MgdGVzdCBhcyB0aGUgUVEtcGxvdCByZXZlYWwgdGhhdCBib3RoIHRoZSBhc3N1bXB0aW9uIG9mIEhvbW9nZW5laXR5IGFuZCBOb3JtYWxpdHkgYXJlIHZpb2xhdGVkLiBUaGVyZWZvcmUgd2Ugd2lsbCBmb2N1cyBvbiB0aGUgY29tcGFyaXNvbiBvZiBhdmVyYWdlIHJhbmtpbmdzLg0KDQpgYGB7cn0NCnByaW50KCJBdmVyYWdlIFJhbmsgcGVyIE1pbmVyIikNCnJlc3VsdHNfZnJlcSRyYW5rX2YxIDwtIHJhbmsoLXJlc3VsdHNfZnJlcSRhdmdfZjEpDQpieShyZXN1bHRzX2ZyZXEkcmFua19mMSwgcmVzdWx0c19mcmVxJG1pbmVyLCBtZWFuKQ0KDQpgYGANCg0KYGBge3J9DQpyZXN1bHRzX2ZyZXEgJT4lDQogIGdncGxvdChhZXMoeD0gbWluZXIgLHkgPXJhbmtfZjEpKSArIGdlb21fdmlvbGluKHNjYWxlPSJ3aWR0aCIsIGRyYXdfcXVhbnRpbGVzID0gYygwLjI1LCAwLjUsIDAuNzUpKSArDQogIGdndGl0bGUoIkRpc3RyaWJ1dGlvbiBvZiByYW5rcyAoYmFzZWQgb24gZjEpIikNCmBgYA0KDQpUaGVzZSBhbmFseXNpcyBzaG93IGEgc2ltaWxhciByYW5raW5nIGJldHdlZW4gdGhlIGZvdXIgbWluZXJzLg0KDQpXZSB3aWxsIHVzZSBhIEtydXNrYWwtV2FsbGlzIHRlc3QgdG8gc2VlIGlmIHRoZXNlIGRpZmZlcmVuY2VzIGluIGF2ZXJhZ2UgcmFua2luZyBhcmUgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudC4NCg0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX2YxIH4gbWluZXIsIHJlc3VsdHNfZnJlcSkNCg0KYGBgDQoNCg0KDQpUaGVzZSByZXN1bHRzIHNob3cgdGhhdCB0aGVyZSBhcmUgc2lnbmlmaWNhbnQgcmFua2luZyBkaWZmZXJlbmNlcyBiZXR3ZWVuIHRoZSBtb2RlbHMgY3JlYXRlZCBieSB0aGUgZm91ciBtaW5lcnMgaW4gdGVybXMgb2YgcHJlY2lzaW9uLiBUaGUgb3JkZXIgc3VnZ2VzdGVkIGJ5IHRoZSByZXN1bHRzIGlzIF9JTFBfID4gX0luZHVjdGl2ZV8gPiBfQWxwaGErXyA+IF9IZXVyaXN0aWNzXy4gV2UgY2FuIHVzZSBhIHBvc3QtaG9jIHRlc3QgdG8gc2VlIHdoaWNoIG9mIHRoZXNlIGRpZmZlcmVuY2VzIGFyZSBzaWduaWZpY2FudC4NCg0KYGBge3J9DQprcnVza2FsbWMoYXZnX2YxIH4gbWluZXIsIHJlc3VsdHNfZnJlcSkNCmBgYA0KDQpJdCBhcHBlYXJzIHRoYXQgYWxsIHBhaXJ3aXNlIGNvbXBhcmlzb25zIGFyZSBzdGF0aXNpY2FsbHkgc2lnbmlmaWNhbnQgZGlmZmVyZW50IGZyb20gZWFjaCBvdGhlciwgZXhjZXB0IGFnYWluIGJldHdlZW4gQWxwaGErIGFuZCBIZXVyaXN0aWNzLiANCg0KDQojIyMgUHJlc2VuY2Ugb2YgSW5mcmVxdWVudCBCZWhhdmlvcg0KDQpGaXJzdCB3ZSB3aWxsIHRlc3QgdG8gc2VlIGlmIHdlIGNhbiBhcHBseSBhIHN0YW5kYXJkIEFOT1ZBIGFuYWx5c2lzDQpgYGB7cn0NCmxldmVuZVRlc3QocmVzdWx0c19pbmZyZXEkYXZnX2YxLCByZXN1bHRzX2luZnJlcSRtaW5lciwgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX2YxIH4gbWluZXIsIGRhdGE9cmVzdWx0c19pbmZyZXEpDQpwbG90KG1vZGVsMSwgd2hpY2g9MikNCmBgYA0KDQpCb3RoIHRoZSBMZXZlbmUncyB0ZXN0IGFzIHRoZSBRUS1wbG90IHJldmVhbCB0aGF0IGJvdGggdGhlIGFzc3VtcHRpb24gb2YgSG9tb2dlbmVpdHkgYW5kIE5vcm1hbGl0eSBhcmUgdmlvbGF0ZWQuIFRoZXJlZm9yZSB3ZSB3aWxsIGZvY3VzIG9uIHRoZSBjb21wYXJpc29uIG9mIGF2ZXJhZ2UgcmFua2luZ3MuDQoNCmBgYHtyfQ0KcHJpbnQoIkF2ZXJhZ2UgUmFuayBwZXIgTWluZXIiKQ0KcmVzdWx0c19pbmZyZXEkcmFua19mMSA8LSByYW5rKC1yZXN1bHRzX2luZnJlcSRhdmdfZjEpDQpieShyZXN1bHRzX2luZnJlcSRyYW5rX2YxLCByZXN1bHRzX2luZnJlcSRtaW5lciwgbWVhbikNCg0KYGBgDQoNCmBgYHtyfQ0KcmVzdWx0c19pbmZyZXEgJT4lDQogIGdncGxvdChhZXMoeD0gbWluZXIgLHkgPXJhbmtfZjEpKSArIGdlb21fdmlvbGluKHNjYWxlPSJ3aWR0aCIsIGRyYXdfcXVhbnRpbGVzID0gYygwLjI1LCAwLjUsIDAuNzUpKSArDQogIGdndGl0bGUoIkRpc3RyaWJ1dGlvbiBvZiByYW5rcyAoYmFzZWQgb24gRjEpIikNCmBgYA0KDQpUaGVzZSBhbmFseXNpcyBzaG93IGFnYWluIGEgc2ltaWxhciByYW5raW5nIGJldHdlZW4gdGhlIGZvdXIgbWluZXJzLiANCg0KV2Ugd2lsbCB1c2UgYSBLcnVza2FsLVdhbGxpcyB0ZXN0IHRvIHNlZSBpZiB0aGVzZSBkaWZmZXJlbmNlcyBpbiBhdmVyYWdlIHJhbmtpbmcgYXJlIHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnQuDQoNCmBgYHtyfQ0Ka3J1c2thbC50ZXN0KGF2Z19mMSB+IG1pbmVyLCByZXN1bHRzX2luZnJlcSkNCg0KYGBgDQoNClRoZXNlIHJlc3VsdHMgc2hvdyB0aGF0IHRoZXJlIGFyZSBzaWduaWZpY2FudCByYW5raW5nIGRpZmZlcmVuY2VzIGJldHdlZW4gdGhlIG1vZGVscyBjcmVhdGVkIGJ5IHRoZSBmb3VyIG1pbmVycyBpbiB0ZXJtcyBvZiBGMSBUaGUgb3JkZXIgc3VnZ2VzdGVkIGJ5IHRoZSBwbG90IGlzIF9JTFBfID4gX0luZHVjdGl2ZV8gPiBfSGV1cmlzdGljc18gPiBfQWxwaGErXy4gV2UgY2FuIHVzZSBhIHBvc3QtaG9jIHRlc3QgdG8gc2VlIHdoaWNoIG9mIHRoZXNlIGRpZmZlcmVuY2VzIGFyZSBzaWduaWZpY2FudC4NCg0KYGBge3J9DQprcnVza2FsbWMoYXZnX2YxIH4gbWluZXIsIHJlc3VsdHNfaW5mcmVxKQ0KYGBgDQoNCkl0IGFwcGVhcnMgdGhhdCBhbGwgcGFpcndpc2UgY29tcGFyaXNvbnMgYXJlIHN0YXRpc2ljYWxseSBzaWduaWZpY2FudCBkaWZmZXJlbnQgZnJvbSBlYWNoIG90aGVyLCBleGNlcHQgZm9yIHRoZSBkaWZmZXJlbmNlIGJldHdlZW4gQWxwaGErIGFuZCBIZXVyaXN0aWNzLg0KDQojIyBJbXBhY3Qgb2YgYW1vdW50IG9mIHJlb2NjdXJpbmcgdGFza3Mgb24gcGVyZm9ybWFuY2Ugb2YgdGhlIE1pbmluZyBBbGdvcml0aG0NCg0KTmV4dCwgd2Ugd2lsbCBpbnZlc3RpZ2F0ZSBob3cgdGhlIHBlcmZvcm1hbmNlIG9mIGVhY2ggb2YgdGhlIG1pbmluZyBhbGdvcml0aG0gaXMgaW5mbHVlbmNlZCBieSB0aGUgbGV2ZWwgb2YgcmVvY2N1cmluZyB0YXNrcyBpbiB0aGUgbG9nIGZpbGVzLg0KDQojIyMgQWxwaGErDQoNCkZpcnN0LCB3ZSB3aWxsIGNoZWNrIHRoZSBhc3N1bXB0aW9ucyBmb3IgYSBzdGFuZGFyZCBBTk9WQSBhbmFseXNpcyANCg0KYGBge3J9DQpsZXZlbmVUZXN0KHJlc3VsdHNfYWxwaGEkYXZnX2YxLCByZXN1bHRzX2FscGhhJHJlb2NjdXJpbmd0YXNrcywgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX2YxIH4gcmVvY2N1cmluZ3Rhc2tzLCBkYXRhPSByZXN1bHRzX2FscGhhKQ0KcGxvdChtb2RlbDEsIHdoaWNoPTIpDQpgYGANCg0KQm90aCB0aGUgYXNzdW1wdGlvbiBvZiBob21vZ25laXR5IGFuZCBub3JtYWxpdHkgYXJlIHZpb2xhdGVkLiBUaGVyZWZvcmUsIHdlIGZhbGwgYmFjayB0byBjb21wYXJpbmcgdGhlIGF2ZXJhZ2UgcmFua2luZ3Mgb2YgdGhlIG1vZGVscyBpbnN0ZWFkLg0KDQoNCmBgYHtyfQ0KcHJpbnQoIkF2ZXJhZ2UgUmFuayBwZXIgbGV2ZWwgUmVvY2N1cmluZyBUYXNrcyIpDQpyZXN1bHRzX2FscGhhJHJhbmtfZjEgPC0gcmFuaygtcmVzdWx0c19hbHBoYSRhdmdfZjEpDQpieShyZXN1bHRzX2FscGhhJHJhbmtfZjEsIHJlc3VsdHNfYWxwaGEkcmVvY2N1cmluZ3Rhc2tzLCBtZWFuKQ0KDQpgYGANCg0KYGBge3J9DQpyZXN1bHRzX2FscGhhICU+JQ0KICBnZ3Bsb3QoYWVzKHg9IHJlb2NjdXJpbmd0YXNrcyAseSA9cmFua19mMSkpICsgZ2VvbV92aW9saW4oc2NhbGU9IndpZHRoIiwgZHJhd19xdWFudGlsZXMgPSBjKDAuMjUsIDAuNSwgMC43NSkpICsNCiAgZ2d0aXRsZSgiRGlzdHJpYnV0aW9uIG9mIHJhbmtzIChiYXNlZCBvbiBGMSkiKQ0KYGBgDQoNClRoZXNlIGRhdGEgc2VlbSB0byBzdWdnZXN0IHRoYXQgYXMgdGhlIGxldmVsIG9mIHJlb2NjdXJpbmcgdGFza3MgaW5jcmVhc2VzLCB0aGUgbW9kZWxzIGdlbmVyYXRlZCBieSBBbHBoYSBtaW5lciBkZXRlcmlvcmF0ZS4gSG93ZXZlciwgaXQgc2VlbXMgdGhhdCB0aGlzIGVmZmVjdCByZWFsbHkgc3RhcnRzIHRvIHNob3cgZnJvbSBhIGxldmVsIG9mIDE1JSByZW9jY3VyaW5nIHRhc2tzLiBUbyB0ZXN0IHRoaXMgaW1wcmVzc2lvbiBzdGF0aXN0aWNhbGx5LCB3ZSB3aWxsIHJlbHkgb24gdGhlIEtXLXRlc3QgYW5kIHRoZSBKb25ja2hlZXJlIFRlc3QuIA0KDQpgYGB7cn0NCmtydXNrYWwudGVzdChhdmdfZjF+IHJlb2NjdXJpbmd0YXNrcywgcmVzdWx0c19hbHBoYSkNCmpvbmNraGVlcmUudGVzdChyZXN1bHRzX2FscGhhJGF2Z19mMSwgYXMubnVtZXJpYyhyZXN1bHRzX2FscGhhJHJlb2NjdXJpbmd0YXNrcyksIG5wZXJtPTEwMDApDQoNCmBgYA0KDQpUaGVzZSBhbmFseXNlcyBzaG93IHRoYXQgdGhlcmUgaXMgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCBuZWdhdGl2ZSB0cmVuZCBpbiB0aGUgcmVsYXRpdmUgcXVhbGl0eSBvZiB0aGUgZ2VuZXJhdGVkIG1vZGVscyBhcyB0aGUgYW1vdW50IG9mIHJlb2NjdXJpbmcgdGFza3MgaW5jcmVhc2VzLiBMZXQncyB2ZXJpZnkgdGhpcyBmdXJ0aGVyIGJ5IGNvbXBhcmluZyBkaWZmZXJlbnQgbGV2ZWxzIHBhaXJ3aXNlLg0KDQpgYGB7cn0NCmtydXNrYWxtYyhhdmdfZjEgfnJlb2NjdXJpbmd0YXNrcywgZGF0YT1yZXN1bHRzX2FscGhhKQ0KYGBgDQoNClRoZXNlIHJlc3VsdHMgaWxsdXN0cmF0ZSB0aGF0IHRoZSBxdWFsaXR5IG9mIHRoZSBtb2RlbHMgZGVjcmVhc2Ugc2lnbmlmaWNhbnRseSB3aGVuZXZlciB0aGUgbGV2ZWwgb2YgcmVvY2N1cmluZyB0YXNrcyBpbmNyZWFzZSBieSAxNSUuDQoNCiMjIyBIZXVyaXN0aWNzDQoNCkZpcnN0LCB3ZSB3aWxsIGNoZWNrIHRoZSBhc3N1bXB0aW9ucyBmb3IgYSBzdGFuZGFyZCBBTk9WQSBhbmFseXNpcyANCg0KYGBge3J9DQpsZXZlbmVUZXN0KHJlc3VsdHNfaGV1cmlzdGljcyRhdmdfZjEsIHJlc3VsdHNfaGV1cmlzdGljcyRyZW9jY3VyaW5ndGFza3MsIGNlbnRlcj1tZWRpYW4pDQptb2RlbDEgPC0gYW92KGF2Z19mMSB+IHJlb2NjdXJpbmd0YXNrcywgZGF0YT0gcmVzdWx0c19oZXVyaXN0aWNzKQ0KcGxvdChtb2RlbDEsIHdoaWNoPTIpDQpgYGANCg0KQm90aCBhc3N1bXB0aW9ucyBvZiBOb3JtYWxpdHkgYW5kIEhvbW9nZW5laXR5IGFyZSB2aW9sYXRlZC4gV2UgdGhlcmVmb3JlIGNvbnRpbnVlIHdpdGggdGhlIEtXLXRlc3QgYW5kIGNvbXBhcmUgYXZlcmFnZSByYW5rcyBiZXR3ZWVuIHRoZSBkaWZmZXJlbnQgbGV2ZWxzIG9mIHJlb2NjdXJpbmcgdGFza3MuDQoNCg0KYGBge3J9DQpwcmludCgiQXZlcmFnZSBSYW5rIHBlciBsZXZlbCBSZW9jY3VyaW5nIFRhc2tzIikNCnJlc3VsdHNfaGV1cmlzdGljcyRyYW5rX2YxIDwtIHJhbmsoLXJlc3VsdHNfaGV1cmlzdGljcyRhdmdfZjEpDQpieShyZXN1bHRzX2hldXJpc3RpY3MkcmFua19mMSwgcmVzdWx0c19oZXVyaXN0aWNzJHJlb2NjdXJpbmd0YXNrcywgbWVhbikNCg0KYGBgDQoNCmBgYHtyfQ0KcmVzdWx0c19oZXVyaXN0aWNzICU+JQ0KICBnZ3Bsb3QoYWVzKHg9IHJlb2NjdXJpbmd0YXNrcyAseSA9cmFua19mMSkpICsgZ2VvbV92aW9saW4oc2NhbGU9IndpZHRoIiwgZHJhd19xdWFudGlsZXMgPSBjKDAuMjUsIDAuNSwgMC43NSkpICsNCiAgZ2d0aXRsZSgiRGlzdHJpYnV0aW9uIG9mIHJhbmtzIChiYXNlZCBvbiBmMSkiKQ0KYGBgDQoNClRoZXNlIGRhdGEgc2VlbSB0byBzdWdnZXN0IHRoYXQgd2hpbGUgdGhlIEYxIHNjb3JlIG1pZ2h0IGJlIG5lZ2F0aXZlbHkgaW5mbHVlbmNlZCBieSB0aGUgcHJlc2VuY2Ugb2YgcmVvY2N1cmluZyB0YXNrcywgdGhlIEYxIHNjb3JlIG9mIG1vZGVscyBkaXNjb3ZlcmVkIGJ5IHRoZSBIZXVyaXN0aWNzIE1pbmVyIGlzIHJlbGF0aXZlbHkgc3RhYmxlIGFnYWluc3QgaW5jcmVhc2luZyBsZXZlbHMgb2YgcmVvY2N1cmluZyB0YXNrcy4gVG8gdGVzdCB0aGlzIGltcHJlc3Npb24gc3RhdGlzdGljYWxseSwgd2Ugd2lsbCByZWx5IG9uIHRoZSBLVy10ZXN0IGFuZCB0aGUgSm9uY2toZWVyZSBUZXN0LiANCg0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX2YxfiByZW9jY3VyaW5ndGFza3MsIHJlc3VsdHNfaGV1cmlzdGljcykNCmpvbmNraGVlcmUudGVzdChyZXN1bHRzX2hldXJpc3RpY3MkYXZnX2YxLCBhcy5udW1lcmljKHJlc3VsdHNfaGV1cmlzdGljcyRyZW9jY3VyaW5ndGFza3MpLCBucGVybT0xMDAwKQ0KDQpgYGANCg0KVGhlc2UgYW5hbHlzZXMgc2hvdyB0aGF0IHRoZSBkYXRhIGRvZXMgbm90IHN1cHBvcnQgdGhlIGNsYWltIG9mIGEgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCAgdHJlbmQgaW4gdGhlIHJlbGF0aXZlIHF1YWxpdHkgb2YgdGhlIGdlbmVyYXRlZCBtb2RlbHMgYXMgdGhlIGFtb3VudCBvZiByZW9jY3VyaW5nIHRhc2tzIGluY3JlYXNlcy4NCg0KIyMjIElMUA0KDQpGaXJzdCwgd2Ugd2lsbCBjaGVjayB0aGUgYXNzdW1wdGlvbnMgZm9yIGEgc3RhbmRhcmQgQU5PVkEgYW5hbHlzaXMgDQoNCmBgYHtyfQ0KbGV2ZW5lVGVzdChyZXN1bHRzX2lscCRhdmdfZjEsIHJlc3VsdHNfaWxwJHJlb2NjdXJpbmd0YXNrcywgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX2YxIH4gcmVvY2N1cmluZ3Rhc2tzLCBkYXRhPSByZXN1bHRzX2lscCkNCnBsb3QobW9kZWwxLCB3aGljaD0yKQ0KYGBgDQoNCkJvdGggdGhlIGFzc3VtcHRpb25zIG9mIG5vcm1hbGl0eSBhbmQgaG9tb2dlbmVpdHkgYXBwZWFyIHRvIGJlIHZpb2xhdGVkLCB0aHVzIHdlIGZhbGwgYmFjayB0byBjb21wYXJpbmcgdGhlIGF2ZXJhZ2UgcmFua2luZ3Mgb2YgdGhlIG1vZGVscyBpbnN0ZWFkLg0KDQoNCmBgYHtyfQ0KcHJpbnQoIkF2ZXJhZ2UgUmFuayBwZXIgbGV2ZWwgUmVvY2N1cmluZyBUYXNrcyIpDQpyZXN1bHRzX2lscCRyYW5rX2YxIDwtIHJhbmsoLXJlc3VsdHNfaWxwJGF2Z19mMSkNCmJ5KHJlc3VsdHNfaWxwJHJhbmtfZjEsIHJlc3VsdHNfaWxwJHJlb2NjdXJpbmd0YXNrcywgbWVhbikNCg0KYGBgDQoNCmBgYHtyfQ0KcmVzdWx0c19pbHAgJT4lDQogIGdncGxvdChhZXMoeD0gcmVvY2N1cmluZ3Rhc2tzICx5ID1yYW5rX2YxKSkgKyBnZW9tX3Zpb2xpbihzY2FsZT0id2lkdGgiLCBkcmF3X3F1YW50aWxlcyA9IGMoMC4yNSwgMC41LCAwLjc1KSkgKw0KICBnZ3RpdGxlKCJEaXN0cmlidXRpb24gb2YgcmFua3MgKGJhc2VkIG9uIGYxKSIpDQpgYGANCg0KVGhlc2UgZGF0YSBzZWVtIHRvIHN1Z2dlc3QgdGhhdCB0aGUgcHJlY2lzaW9uIHNjb3JlIG9mIHRoZSBtb2RlbHMgZ2VuZXJhdGVkIGJ5IElMUCBpcyB2ZXJ5IHNlbnNpdGl2ZSB0byB0aGUgbGV2ZWwgb2YgcmVvY2N1cmluZyB0YXNrcy4gVG8gdGVzdCB0aGlzIGltcHJlc3Npb24gc3RhdGlzdGljYWxseSwgd2Ugd2lsbCByZWx5IG9uIHRoZSBLVy10ZXN0IGFuZCB0aGUgSm9uY2toZWVyZSBUZXN0LiANCg0KYGBge3J9DQprcnVza2FsLnRlc3QoYXZnX2YxfiByZW9jY3VyaW5ndGFza3MsIHJlc3VsdHNfaWxwKQ0Kam9uY2toZWVyZS50ZXN0KHJlc3VsdHNfaWxwJGF2Z19mMSwgYXMubnVtZXJpYyhyZXN1bHRzX2lscCRyZW9jY3VyaW5ndGFza3MpLCBucGVybT0xMDAwKQ0KDQpgYGANCg0KVGhlc2UgYW5hbHlzZXMgc2hvdyB0aGF0IHRoZXJlIGlzIGEgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCBuZWdhdGl2ZSB0cmVuZCBpbiB0aGUgcmVsYXRpdmUgcXVhbGl0eSBvZiB0aGUgZ2VuZXJhdGVkIG1vZGVscyBhcyB0aGUgYW1vdW50IG9mIHJlb2NjdXJpbmcgdGFza3MgaW5jcmVhc2VzLiBMZXQncyB2ZXJpZnkgdGhpcyBmdXJ0aGVyIGJ5IGNvbXBhcmluZyBkaWZmZXJlbnQgbGV2ZWxzIHBhaXJ3aXNlLg0KDQpgYGB7cn0NCmtydXNrYWxtYyhhdmdfZjEgfnJlb2NjdXJpbmd0YXNrcywgZGF0YT1yZXN1bHRzX2lscCkNCmBgYA0KDQpUaGVzZSByZXN1bHRzIGlsbHVzdHJhdGUgdGhlcmUgaXMgYSBjbGVhciBzaWduaWZpY2FudCB0cmVuZCwgd2hpY2ggc29tZWhvdyBsZXZlbHMgb2YgYXJvdW5kIDI1JS4NCg0KIyMjIEluZHVjdGl2ZQ0KDQpGaXJzdCwgd2Ugd2lsbCBjaGVjayB0aGUgYXNzdW1wdGlvbnMgZm9yIGEgc3RhbmRhcmQgQU5PVkEgYW5hbHlzaXMgDQoNCmBgYHtyfQ0KbGV2ZW5lVGVzdChyZXN1bHRzX2luZHVjdGl2ZSRhdmdfZjEsIHJlc3VsdHNfaW5kdWN0aXZlJHJlb2NjdXJpbmd0YXNrcywgY2VudGVyPW1lZGlhbikNCm1vZGVsMSA8LSBhb3YoYXZnX2YxIH4gcmVvY2N1cmluZ3Rhc2tzLCBkYXRhPSByZXN1bHRzX2luZHVjdGl2ZSkNCnBsb3QobW9kZWwxLCB3aGljaD0yKQ0KYGBgDQoNCkJvdGggdGhlIGFzc3VtcHRpb25zIG9mIG5vcm1hbGl0eSBhbmQgaG9tb2dlbmVpdHkgYXBwZWFyIHRvIGJlIHZpb2xhdGVkLCB0aHVzIHdlIGZhbGwgYmFjayB0byBjb21wYXJpbmcgdGhlIGF2ZXJhZ2UgcmFua2luZ3Mgb2YgdGhlIG1vZGVscyBpbnN0ZWFkLg0KDQoNCmBgYHtyfQ0KcHJpbnQoIkF2ZXJhZ2UgUmFuayBwZXIgbGV2ZWwgUmVvY2N1cmluZyBUYXNrcyIpDQpyZXN1bHRzX2luZHVjdGl2ZSRyYW5rX2YxIDwtIHJhbmsoLXJlc3VsdHNfaW5kdWN0aXZlJGF2Z19mMSkNCmJ5KHJlc3VsdHNfaW5kdWN0aXZlJHJhbmtfZjEsIHJlc3VsdHNfaW5kdWN0aXZlJHJlb2NjdXJpbmd0YXNrcywgbWVhbikNCg0KYGBgDQoNCmBgYHtyfQ0KcmVzdWx0c19pbmR1Y3RpdmUgJT4lDQogIGdncGxvdChhZXMoeD0gcmVvY2N1cmluZ3Rhc2tzICx5ID1yYW5rX2YxKSkgKyBnZW9tX3Zpb2xpbihzY2FsZT0id2lkdGgiLCBkcmF3X3F1YW50aWxlcyA9IGMoMC4yNSwgMC41LCAwLjc1KSkgKw0KICBnZ3RpdGxlKCJEaXN0cmlidXRpb24gb2YgcmFua3MgKGJhc2VkIG9uIGYxcykiKQ0KYGBgDQoNClRoZXNlIGRhdGEgc2VlbSB0byBzdWdnZXN0IHRoYXQgYXMgdGhlIGxldmVsIG9mIHJlb2NjdXJpbmcgdGFza3MgaW5jcmVhc2VzLCB0aGUgbW9kZWxzIGdlbmVyYXRlZCBieSBJbmR1Y3RpdmUgbWluZXIgZGV0ZXJpb3JhdGUuIFRvIHRlc3QgdGhpcyBpbXByZXNzaW9uIHN0YXRpc3RpY2FsbHksIHdlIHdpbGwgcmVseSBvbiB0aGUgS1ctdGVzdCBhbmQgdGhlIEpvbmNraGVlcmUgVGVzdC4gDQoNCmBgYHtyfQ0Ka3J1c2thbC50ZXN0KGF2Z19mMX4gcmVvY2N1cmluZ3Rhc2tzLCByZXN1bHRzX2luZHVjdGl2ZSkNCmpvbmNraGVlcmUudGVzdChyZXN1bHRzX2luZHVjdGl2ZSRhdmdfZjEsIGFzLm51bWVyaWMocmVzdWx0c19pbmR1Y3RpdmUkcmVvY2N1cmluZ3Rhc2tzKSwgbnBlcm09MTAwMCkNCg0KYGBgDQoNClRoZXNlIGFuYWx5c2VzIHNob3cgdGhhdCB0aGVyZSBpcyBzdGF0aXN0aWNhbGx5IHNpZ25pZmljYW50IG5lZ2F0aXZlIHRyZW5kIGluIHRoZSByZWxhdGl2ZSBxdWFsaXR5IG9mIHRoZSBnZW5lcmF0ZWQgbW9kZWxzIGFzIHRoZSBhbW91bnQgb2YgcmVvY2N1cmluZyB0YXNrcyBpbmNyZWFzZXMuIExldCdzIHZlcmlmeSB0aGlzIGZ1cnRoZXIgYnkgY29tcGFyaW5nIGRpZmZlcmVudCBsZXZlbHMgcGFpcndpc2UuDQoNCmBgYHtyfQ0Ka3J1c2thbG1jKGF2Z19mMSB+cmVvY2N1cmluZ3Rhc2tzLCBkYXRhPXJlc3VsdHNfaW5kdWN0aXZlKQ0KYGBgDQoNClRoZXNlIHJlc3VsdHMgc2VlbSB0byBjb25maXJtIG91ciBpbXByZXNzaW9uLiBJbmR1Y3RpdmUgTWluZXIgaXMgc2Vuc2l0aXZlIHRvIGxldmVscyBmb3IgcmVvY2N1cmluZyB0YXNrcywgcmVzdWx0aW5nIGluIG1vZGVscyBvZiBsb3dlciBxdWFsaXR5IChtZWFzdXJlZCBpbiB0ZXJtcyBvZiBwcmVjaXNpb24gdmFsdWVzKS4gSG93ZXZlciwgZnJvbSBhIGxldmVsIG9mIGFyb3VuZCAxNSUgcmVvY2N1cmluZyB0YXNrcyBpbiB0aGUgbG9nLCB0aGlzIGVmZmVjdCBzZWVtcyB0byBoYXZlIHJlYWNoZWQgYSBwbGF0ZWF1IGFuZCBzdGF5cyBzdGFibGUu