Study 2 Analysis Standardised test scores
Assumes libraries tidyverse, descriptr, gridExtra
Read and clean data
df_raw = read_excel("S2_Pre_Post_full.xlsx")Rescaling needed: * P1Q1 max 6, Q2 max 4, Q3 max 3, Q4 max 2, total max 15 * P2Q1 max 4, Q2 max 4, Q3 max 3, Q4 max 2, total max 13 * TRANSFER1 and 2 max 10 each.
Instead of the absolute scores we need the percentage in terms of the maximum score. We can use a scale from 0 to 10 with integer values.
df <- df_raw
df$ROLE <- factor(df$ROLE, levels = c(1,9), labels = c("tutor_first", "tutee_first"))
df$Pair <- factor(df$Pair)
# Pretest Scatterplot
df$P1Q1 <-as.integer(round(df$P1Q1/6.0 * 10, digits = 0))
df$P1Q2 <-as.integer(round(df$P1Q2/4.0 * 10, digits = 0))
df$P1Q3 <-as.integer(round(df$P1Q3/3.0 * 10, digits = 0))
df$P1Q4 <-as.integer(round(df$P1Q4/2.0 * 10, digits = 0))
# Pretest BWD
df$P2Q1 <-as.integer(round(df$P2Q1/4.0 * 10, digits = 0))
df$P2Q2 <-as.integer(round(df$P2Q2/4.0 * 10, digits = 0))
df$P2Q3 <-as.integer(round(df$P2Q3/3.0 * 10, digits = 0))
df$P2Q4 <-as.integer(round(df$P2Q4/2.0 * 10, digits = 0))
# Post-test Scatterplot
df$POSTP1Q1 <-as.integer(round(df$POSTP1Q1/6.0 * 10, digits = 0))
df$POSTP1Q2 <-as.integer(round(df$POSTP1Q2/4.0 * 10, digits = 0))
df$POSTP1Q3 <-as.integer(round(df$POSTP1Q3/3.0 * 10, digits = 0))
df$POSTP1Q4 <-as.integer(round(df$POSTP1Q4/2.0 * 10, digits = 0))
# Post-test BWD
df$POSTP2Q1 <-as.integer(round(df$POSTP2Q1/4.0 * 10, digits = 0))
df$POSTP2Q2 <-as.integer(round(df$POSTP2Q2/4.0 * 10, digits = 0))
df$POSTP2Q3 <-as.integer(round(df$POSTP2Q3/3.0 * 10, digits = 0))
df$POSTP2Q4 <-as.integer(round(df$POSTP2Q4/2.0 * 10, digits = 0))
# Transfer items to integers
df$TRANSFER1 <- as.integer(df$TRANSFER1)
df$TRANSFER2 <- as.integer(df$TRANSFER2)Now we need to re-compute the marginal scores. Let’s first drop the old columns:
df <- select(df, -c("PRE-SCORE", "POST-SCORE"))And now compute the new marginal scores:
# average scores:
There were 23 warnings (use warnings() to see them)df <- mutate (df, PREAVG = PRESCORE/8)I think we are ready now for the analysis.
Univarite analysis
Pretest
The principe maximal score in the test is 80.
ds_summary_stats(df,PRESCORE)────────────────────────────────────────────── Variable: PRESCORE ──────────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 394.23
Missing 0.00 Std Deviation 19.86
Mean 38.89 Range 75.00
Median 37.50 Interquartile Range 27.25
Mode 13.00 Uncorrected SS 87317.00
Trimmed Mean 38.50 Corrected SS 17740.46
Skewness 0.35 Coeff Variation 51.05
Kurtosis -0.78 Std Error Mean 2.93
Quantiles
Quantile Value
Max 80.00
99% 79.10
95% 72.00
90% 68.00
Q3 51.75
Median 37.50
Q1 24.50
10% 14.50
5% 12.25
1% 6.80
Min 5.00
Extreme Values
Low High
Obs Value Obs Value
38 5 37 80
25 9 35 78
32 12 8 73
21 13 4 69
26 13 16 69 ggplot(df, aes(PRESCORE)) +
geom_histogram(bins = 10)
ggplot(df, aes(x = 1, y = PRESCORE)) +
geom_boxplot() +
scale_x_continuous(breaks = NULL) +
theme(axis.title.x = element_blank())
To interpret changes due to ROLE later, is there a differnce in the pre-test between students that subseqently were in the tutor_first or tutee_first role?
ggplot(df, aes(x = ROLE, y = PRESCORE)) +
geom_boxplot() +
xlab("Tutor role")
While the tutor_first is slightly better, this is likely random. A t-test agrees, with p greater than 0.05.
t.test(df$PRESCORE ~df$ROLE)
Welch Two Sample t-test
data: df$PRESCORE by df$ROLE
t = 0.52296, df = 43.403, p-value = 0.6037
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-8.814075 14.987988
sample estimates:
mean in group tutor_first mean in group tutee_first
40.43478 37.34783 The non-parametric Wilcoxon test further confirms that there is no significant difference between the two groups:
wilcox.test(df$PRESCORE ~df$ROLE)cannot compute exact p-value with ties
Wilcoxon rank sum test with continuity correction
data: df$PRESCORE by df$ROLE
W = 288, p-value = 0.6133
alternative hypothesis: true location shift is not equal to 0Post test
ds_summary_stats(df,POSTSCORE)─────────────────────────────────────────────── Variable: POSTSCORE ───────────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 384.17
Missing 0.00 Std Deviation 19.60
Mean 52.30 Range 68.00
Median 58.50 Interquartile Range 31.00
Mode 26.00 Uncorrected SS 143132.00
Trimmed Mean 52.86 Corrected SS 17287.74
Skewness -0.55 Coeff Variation 37.47
Kurtosis -0.91 Std Error Mean 2.89
Quantiles
Quantile Value
Max 80.00
99% 79.10
95% 77.25
90% 74.50
Q3 68.00
Median 58.50
Q1 37.00
10% 23.00
5% 19.00
1% 13.80
Min 12.00
Extreme Values
Low High
Obs Value Obs Value
19 12 35 80
6 16 37 78
13 19 39 78
32 19 9 75
38 21 24 75 # ggplot(df, aes(POSTSCORE)) + geom_bar()
ggplot(df, aes(POSTSCORE)) +
geom_histogram(bins = 10)
Check:we’d like to know if the cases with the low values are the same from pre to post test. That would indicate non-engagment.
ggplot(df, aes(x = 1, y = POSTSCORE)) +
geom_boxplot() +
scale_x_continuous(breaks = NULL) +
theme(axis.title.x = element_blank())
By role:
ggplot(df, aes(x = ROLE, y = POSTSCORE)) +
geom_boxplot() +
xlab("Tutor role")
The difference between the two conditions is marginal, by inspection, also keeping in mind that the pre-test scores where sliglyt elavated for the tutor_first condition. A test reveals no significant difference.
t.test(df$POSTSCORE ~df$ROLE)
Welch Two Sample t-test
data: df$POSTSCORE by df$ROLE
t = 0.53732, df = 43.954, p-value = 0.5938
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-8.611533 14.872402
sample estimates:
mean in group tutor_first mean in group tutee_first
53.86957 50.73913 In the further analysis we treat the two groups as comparable.
Treatment effects
Summary scores
The intervention was clearly effective:
t.test(df$POSTSCORE, df$PRESCORE, paired=T)
Paired t-test
data: df$POSTSCORE and df$PRESCORE
t = 5.7233, df = 45, p-value = 8.025e-07
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
8.692841 18.133246
sample estimates:
mean of the differences
13.41304 Looking at the individual gain scores, we see that hile most of the gain scores are positive, with negative values being few and small values only, the loss score of 33 points for student in position 8 (S2-AM-9) is rather extreme.
df$gain <- df$POSTSCORE - df$PRESCORE
df$gain [1] 16 10 12 0 10 -1 20 -33 21 13 43 23 -3 29 7 -10 27 -11 -7 10 12 1 41 8 17 33 16
[28] 26 37 26 17 7 32 20 2 -5 -2 16 28 40 2 -2 21 20 32 -4This can be explained by this student not actually engaging with the post-test….
ds_summary_stats(df, gain)──────────────────────────────────────────────── Variable: gain ────────────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 252.65
Missing 0.00 Std Deviation 15.89
Mean 13.41 Range 76.00
Median 14.50 Interquartile Range 24.00
Mode 10.00 Uncorrected SS 19645.00
Trimmed Mean 13.74 Corrected SS 11369.15
Skewness -0.32 Coeff Variation 118.50
Kurtosis 0.27 Std Error Mean 2.34
Quantiles
Quantile Value
Max 43.00
99% 42.10
95% 39.25
90% 32.50
Q3 25.25
Median 14.50
Q1 1.25
10% -4.50
5% -9.25
1% -23.10
Min -33.00
Extreme Values
Low High
Obs Value Obs Value
8 -33 11 43
18 -11 23 41
16 -10 40 40
19 -7 29 37
36 -5 26 33 Averaged scores
Strong learning gains:
t.test(df$POSTAVG, df$PREAVG, paired=T)
Paired t-test
data: df$POSTAVG and df$PREAVG
t = 5.7233, df = 45, p-value = 8.025e-07
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
1.086605 2.266656
sample estimates:
mean of the differences
1.67663 df$gain_avg <- df$POSTAVG - df$PREAVG
df$gain_avg [1] 2.000 1.250 1.500 0.000 1.250 -0.125 2.500 -4.125 2.625 1.625 5.375 2.875 -0.375 3.625 0.875
[16] -1.250 3.375 -1.375 -0.875 1.250 1.500 0.125 5.125 1.000 2.125 4.125 2.000 3.250 4.625 3.250
[31] 2.125 0.875 4.000 2.500 0.250 -0.625 -0.250 2.000 3.500 5.000 0.250 -0.250 2.625 2.500 4.000
[46] -0.500ds_summary_stats(df,gain_avg)────────────────────────────────────────────── Variable: gain_avg ──────────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 3.95
Missing 0.00 Std Deviation 1.99
Mean 1.68 Range 9.50
Median 1.81 Interquartile Range 3.00
Mode 1.25 Uncorrected SS 306.95
Trimmed Mean 1.72 Corrected SS 177.64
Skewness -0.32 Coeff Variation 118.50
Kurtosis 0.27 Std Error Mean 0.29
Quantiles
Quantile Value
Max 5.38
99% 5.26
95% 4.91
90% 4.06
Q3 3.16
Median 1.81
Q1 0.16
10% -0.56
5% -1.16
1% -2.89
Min -4.12
Extreme Values
Low High
Obs Value Obs Value
8 -4.125 11 5.375
18 -1.375 23 5.125
16 -1.25 40 5
19 -0.875 29 4.625
36 -0.625 26 4.125 Analysis of no learning cases
Given that the students had no very little initial knowledge, losses in the post test need to be explained on a case by case basis. We do this in three sections:
Low pre-test scores and no gain
No gain or loss likely means non-engagement. This pattern holds for these cases:
High pre-test score and loss
S2-AM-9:
Others?
Average scores analysis
Average Pretest
The principe maximal score in the test is 80.
ds_summary_stats(df,PREAVG)───────────────────────────────────────────── Variable: PREAVG ────────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 6.16
Missing 0.00 Std Deviation 2.48
Mean 4.86 Range 9.38
Median 4.69 Interquartile Range 3.41
Mode 1.62 Uncorrected SS 1364.33
Trimmed Mean 4.81 Corrected SS 277.19
Skewness 0.35 Coeff Variation 51.05
Kurtosis -0.78 Std Error Mean 0.37
Quantiles
Quantile Value
Max 10.00
99% 9.89
95% 9.00
90% 8.50
Q3 6.47
Median 4.69
Q1 3.06
10% 1.81
5% 1.53
1% 0.85
Min 0.62
Extreme Values
Low High
Obs Value Obs Value
38 0.625 37 10
25 1.125 35 9.75
32 1.5 8 9.125
21 1.625 4 8.625
26 1.625 16 8.625 There were 26 warnings (use warnings() to see them)ggplot(df, aes(PREAVG)) +
geom_histogram(bins = 10)
ggplot(df, aes(x = 1, y = PREAVG)) +
geom_boxplot() +
scale_x_continuous(breaks = NULL) +
theme(axis.title.x = element_blank())
To interpret changes due to ROLE later, is there a differnce in the pre-test between students that subseqently were in the tutor_first or tutee_first role?
ggplot(df, aes(x = ROLE, y = PREAVG)) +
geom_boxplot() +
xlab("Tutor role")
While the tutor_first is slightly better, this is likely random. A t-test agrees, with p greater than 0.05.
t.test(df$PREAVG ~df$ROLE)
Welch Two Sample t-test
data: df$PREAVG by df$ROLE
t = 0.52296, df = 43.403, p-value = 0.6037
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.101759 1.873498
sample estimates:
mean in group tutor_first mean in group tutee_first
5.054348 4.668478 The non-parametric Wilcoxon test further confirms that there is no significant difference between the two groups:
wilcox.test(df$PREAVG ~df$ROLE)cannot compute exact p-value with ties
Wilcoxon rank sum test with continuity correction
data: df$PREAVG by df$ROLE
W = 288, p-value = 0.6133
alternative hypothesis: true location shift is not equal to 0Average Posttest
The principe maximal score in the test is 80.
ds_summary_stats(df,POSTAVG)─────────────────────────────── Variable: POSTAVG ──────────────────────────────
Univariate Analysis
N 46.00 Variance 6.00
Missing 0.00 Std Deviation 2.45
Mean 6.54 Range 8.50
Median 7.31 Interquartile Range 3.88
Mode 3.25 Uncorrected SS 2236.44
Trimmed Mean 6.61 Corrected SS 270.12
Skewness -0.55 Coeff Variation 37.47
Kurtosis -0.91 Std Error Mean 0.36
Quantiles
Quantile Value
Max 10.00
99% 9.89
95% 9.66
90% 9.31
Q3 8.50
Median 7.31
Q1 4.62
10% 2.88
5% 2.38
1% 1.73
Min 1.50
Extreme Values
Low High
Obs Value Obs Value
19 1.5 35 10
6 2 37 9.75
13 2.375 39 9.75
32 2.375 9 9.375
38 2.625 24 9.375 There were 23 warnings (use warnings() to see them)ggplot(df, aes(POSTAVG)) +
geom_histogram(bins = 10)
ggplot(df, aes(x = 1, y = POSTAVG)) +
geom_boxplot() +
scale_x_continuous(breaks = NULL) +
theme(axis.title.x = element_blank())
is there a differnce between students that subseqently were in the tutor_first or tutee_first role?
ggplot(df, aes(x = ROLE, y = POSTAVG)) +
geom_boxplot() +
xlab("Tutor role")
While the tutor_first is slightly better, this is likely random. A t-test agrees, with p greater than 0.05.
t.test(df$POSTAVG ~df$ROLE)
Welch Two Sample t-test
data: df$POSTAVG by df$ROLE
t = 0.53732, df = 43.954, p-value = 0.5938
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.076442 1.859050
sample estimates:
mean in group tutor_first mean in group tutee_first
6.733696 6.342391 The non-parametric Wilcoxon test further confirms that there is no significant difference between the two groups:
wilcox.test(df$POSTAVG ~df$ROLE)cannot compute exact p-value with ties
Wilcoxon rank sum test with continuity correction
data: df$POSTAVG by df$ROLE
W = 289, p-value = 0.5978
alternative hypothesis: true location shift is not equal to 0Item level analysis Pre and Post.
Scatterplot Q1
df$P1Q1 [1] 2 3 8 10 2 3 3 3 7 3 3 8 3 5 10 10 8 7 5 3 3 7 0 10 3 3 10 8 3 2 5 2 5 5 8 0 10
[38] 3 8 3 2 7 7 3 2 5df$POSTP1Q1 [1] 3 3 10 10 0 2 3 0 10 5 3 8 0 8 10 10 10 8 5 3 3 10 7 10 3 10 8 3 10 10 7 5 10 7 10 3 8
[38] 3 8 10 0 10 10 8 7 2ds_summary_stats(df,P1Q1, POSTP1Q1 )────────────────────────────────────────────────── Variable: P1Q1 ─────────────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 8.62
Missing 0.00 Std Deviation 2.94
Mean 5.00 Range 10.00
Median 4.00 Interquartile Range 4.75
Mode 3.00 Uncorrected SS 1538.00
Trimmed Mean 5.00 Corrected SS 388.00
Skewness 0.39 Coeff Variation 58.73
Kurtosis -1.06 Std Error Mean 0.43
Quantiles
Quantile Value
Max 10.00
99% 10.00
95% 10.00
90% 10.00
Q3 7.75
Median 4.00
Q1 3.00
10% 2.00
5% 2.00
1% 0.00
Min 0.00
Extreme Values
Low High
Obs Value Obs Value
23 0 4 10
36 0 15 10
1 2 16 10
5 2 24 10
30 2 27 10
──────────────────────────────────────────────── Variable: POSTP1Q1 ───────────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 12.24
Missing 0.00 Std Deviation 3.50
Mean 6.37 Range 10.00
Median 7.50 Interquartile Range 7.00
Mode 10.00 Uncorrected SS 2417.00
Trimmed Mean 6.50 Corrected SS 550.72
Skewness -0.43 Coeff Variation 54.92
Kurtosis -1.27 Std Error Mean 0.52
Quantiles
Quantile Value
Max 10.00
99% 10.00
95% 10.00
90% 10.00
Q3 10.00
Median 7.50
Q1 3.00
10% 2.00
5% 0.00
1% 0.00
Min 0.00
Extreme Values
Low High
Obs Value Obs Value
5 0 3 10
8 0 4 10
13 0 9 10
41 0 15 10
6 2 16 10 boxplot(df$P1Q1, data = df)
boxplot(df$POSTP1Q1, data = df)
ggplot(df, aes(P1Q1)) + geom_bar()
ggplot(df, aes(POSTP1Q1)) + geom_bar()
Scatterplot Q2
df$P1Q2
df$POSTP1Q2ds_summary_stats(df,P1Q2, POSTP1Q2)ggplot(df, aes(P1Q2)) + geom_bar()
ggplot(df, aes(POSTP1Q2)) + geom_bar()The prestest has a very odd distribution: Check!
Scatterplot Q3
df$P1Q3
df$POSTP1Q3ds_summary_stats(df,P1Q3, POSTP1Q3 )boxplot(df$P1Q3, data = df)
boxplot(df$POSTP1Q3, data = df)ggplot(df, aes(P1Q3)) + geom_bar()
ggplot(df, aes(POSTP1Q3)) + geom_bar()Scatterplot Q4
df$P1Q4
df$POSTP1Q4ds_summary_stats(df,P1Q4, POSTP1Q4 )boxplot(df$P1Q4, data = df)
boxplot(df$POSTP1Q4, data = df)ggplot(df, aes(P1Q4)) + geom_bar()
ggplot(df, aes(POSTP1Q4)) + geom_bar()BWD Q1
df$P2Q1
df$POSTP2Q1ds_summary_stats(df,P2Q1, POSTP2Q1 )boxplot(df$P2Q1, data = df)
boxplot(df$POSTP2Q1, data = df)ggplot(df, aes(P2Q1)) + geom_bar()
ggplot(df, aes(POSTP2Q1)) + geom_bar()BWD Q2
df$P2Q2
df$POSTP2Q2ds_summary_stats(df,P2Q2, POSTP2Q2 )boxplot(df$P2Q2, data = df)
boxplot(df$POSTP2Q2, data = df)ggplot(df, aes(P2Q2)) + geom_bar()
ggplot(df, aes(POSTP2Q2)) + geom_bar()BWD Q3
df$P2Q3
df$POSTP2Q3ds_summary_stats(df,P2Q3, POSTP2Q3 )boxplot(df$P2Q3, data = df)
boxplot(df$POSTP2Q3, data = df)ggplot(df, aes(P2Q3)) + geom_bar()
ggplot(df, aes(POSTP2Q3)) + geom_bar()BWD Q4
df$P2Q4
df$POSTP2Q4ds_summary_stats(df,P2Q4, POSTP2Q4 )boxplot(df$P2Q4, data = df)
boxplot(df$POSTP2Q4, data = df)ggplot(df, aes(P2Q4)) + geom_bar()
ggplot(df, aes(POSTP2Q4)) + geom_bar()Scatterplot Transfer
df$TRANSFER1 [1] 5 10 10 8 0 7 9 0 7 5 7 7 3 5 9 6 5 3 0 6 0 0 5 9 4 0 5 6 8 9 9 0 8 7 10 5 7
[38] 2 10 8 5 2 8 5 0 0ds_summary_stats(df,TRANSFER1 )─────────────────────────────────────────────── Variable: TRANSFER1 ───────────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 11.11
Missing 0.00 Std Deviation 3.33
Mean 5.30 Range 10.00
Median 5.50 Interquartile Range 5.00
Mode 0.00 Uncorrected SS 1794.00
Trimmed Mean 5.33 Corrected SS 499.74
Skewness -0.41 Coeff Variation 62.83
Kurtosis -1.03 Std Error Mean 0.49
Quantiles
Quantile Value
Max 10.00
99% 10.00
95% 10.00
90% 9.00
Q3 8.00
Median 5.50
Q1 3.00
10% 0.00
5% 0.00
1% 0.00
Min 0.00
Extreme Values
Low High
Obs Value Obs Value
5 0 2 10
8 0 3 10
19 0 35 10
21 0 39 10
22 0 7 9 boxplot(df$TRANSFER1, data = df)
ggplot(df, aes(TRANSFER1)) + geom_histogram(bins = 6)
Does ROLE make a difference?
ggplot(df, aes(x = ROLE, y = TRANSFER1)) +
geom_boxplot() +
xlab("Tutor role")
Visually yes and also statistically significant:
t.test(df$TRANSFER1 ~df$ROLE)
Welch Two Sample t-test
data: df$TRANSFER1 by df$ROLE
t = 2.112, df = 43.782, p-value = 0.04042
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.09127597 3.90872403
sample estimates:
mean in group tutor_first mean in group tutee_first
6.304348 4.304348 BWD Transfer
df$TRANSFER2 [1] 0 9 0 9 0 6 10 0 8 9 10 10 0 0 10 8 8 3 0 9 0 0 9 10 4 3 8 9 10 10 9 0 9 8 10 8 8
[38] 0 10 9 0 2 9 6 0 0ds_summary_stats(df,TRANSFER2 )─────────────────────────────────────────────── Variable: TRANSFER2 ───────────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 17.74
Missing 0.00 Std Deviation 4.21
Mean 5.65 Range 10.00
Median 8.00 Interquartile Range 9.00
Mode 0.00 Uncorrected SS 2268.00
Trimmed Mean 5.71 Corrected SS 798.43
Skewness -0.44 Coeff Variation 74.52
Kurtosis -1.66 Std Error Mean 0.62
Quantiles
Quantile Value
Max 10.00
99% 10.00
95% 10.00
90% 10.00
Q3 9.00
Median 8.00
Q1 0.00
10% 0.00
5% 0.00
1% 0.00
Min 0.00
Extreme Values
Low High
Obs Value Obs Value
1 0 7 10
3 0 11 10
5 0 12 10
8 0 15 10
13 0 24 10 boxplot(df$TRANSFER2, data = df)
ggplot(df, aes(TRANSFER2)) + geom_histogram(bins = 6)
This might indicate that in the post test we have two groups of students: Those who pretty much got it and those who pretty much not.
Does ROLE make a difference?
ggplot(df, aes(x = ROLE, y = TRANSFER2)) +
geom_boxplot() +
xlab("Tutor role")
Visually yes but missing statistical significance because of the large variation in the scores:
t.test(df$TRANSFER2 ~df$ROLE)
Welch Two Sample t-test
data: df$TRANSFER2 by df$ROLE
t = 1.5647, df = 43.993, p-value = 0.1248
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.5509621 4.3770490
sample estimates:
mean in group tutor_first mean in group tutee_first
6.608696 4.695652 For the qualitative analysis it would be worth finding out what can be said about the two ‘extreme’ groups. This item would also be the one as a basis for extrem group comparisons, splitting on the median.
A combined transfer score
df$TRANSFER <- df$TRANSFER1 + df$TRANSFER2ds_summary_stats(df,TRANSFER )───────────────────────────────────────── Variable: TRANSFER ────────────────────────────────────────
Univariate Analysis
N 46.00 Variance 50.84
Missing 0.00 Std Deviation 7.13
Mean 10.96 Range 20.00
Median 13.50 Interquartile Range 12.75
Mode 0.00 Uncorrected SS 7810.00
Trimmed Mean 11.05 Corrected SS 2287.91
Skewness -0.44 Coeff Variation 65.08
Kurtosis -1.39 Std Error Mean 1.05
Quantiles
Quantile Value
Max 20.00
99% 20.00
95% 19.00
90% 19.00
Q3 17.00
Median 13.50
Q1 4.25
10% 0.00
5% 0.00
1% 0.00
Min 0.00
Extreme Values
Low High
Obs Value Obs Value
5 0 35 20
8 0 39 20
19 0 2 19
21 0 7 19
22 0 15 19 boxplot(df$TRANSFER, data = df)
ggplot(df, aes(TRANSFER)) + geom_histogram(bins = 8)
ggplot(df, aes(x = ROLE, y = TRANSFER)) +
geom_boxplot() +
xlab("Tutor role")
t.test(df$TRANSFER ~df$ROLE)
Welch Two Sample t-test
data: df$TRANSFER by df$ROLE
t = 1.9154, df = 43.668, p-value = 0.062
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.2050992 8.0311861
sample estimates:
mean in group tutor_first mean in group tutee_first
12.91304 9.00000 Low and highscoring students based on various criteria:
Dalal, this is for cases where you may want to make extrem group comparisons later. But you picked a subset anyway. So just for completenes:
So, a median split could be on a score 8.0 of TRANSFER2, the BWD item. It would yield the following 20 students for low score (Less than 8):
low_scorers <- filter(df, TRANSFER2 < 8.0)
low_scorers <- as_vector(low_scorers$Stdcode)
low_scorers [1] "S2_AM_1" "S2_AM_3" "S2_AM_5" "S2_AM_6" "S2_AM_9" "S2_AM_14" "S2_AM_01" "S2_AM_05"
[9] "S2_AM_06" "S2_AM_08" "S2_AM_010" "S2_AM_013" "S2_AM_014" "S2_PM_6" "S2_PM_02" "S2_PM_05"
[17] "S2_PM_06" "S2_PM_08" "S2_PM_09" "S2_PM_010"And these 26 for scores GEQ 8.0:
high_scorers <- filter(df, TRANSFER2 >= 8.0)
high_scorers <- as_vector(high_scorers$Stdcode)
high_scorers [1] "S2_AM_2" "S2_AM_4" "S2_AM_7" "S2_AM_10" "S2_AM_11" "S2_AM_12" "S2_AM_13" "S2_AM_02"
[9] "S2_AM_03" "S2_AM_04" "S2_AM_07" "S2_AM_011" "S2_AM_012" "S2_PM_1" "S2_PM_2" "S2_PM_3"
[17] "S2_PM_4" "S2_PM_5" "S2_PM_7" "S2_PM_8" "S2_PM_9" "S2_PM_10" "S2_PM_01" "S2_PM_03"
[25] "S2_PM_04" "S2_PM_07" Non-engaged students (ignore this; only for completeness)
Filtering out six likely cases of non-engagement:
df <- filter(df, Stdcode != "S2_AM_9", Stdcode != "S2_AM_6", Stdcode != "S2_AM_06",
Stdcode != "S2_AM_14", Stdcode != "S2_PM_010", Stdcode != "S2_PM_05")Treatment effect general :
t.test(df$POST.SCORE, df$PRE.SCORE, paired=T)Strong as ever.
Significant differences regarding role before intervention?
t.test(df$PRE.SCORE ~df$ROLE)No.
After intervention?
t.test(df$POST.SCORE ~df$ROLE)ALso not.
How about Transfer item Scatterplat?
t.test(df$TRANSFER1 ~df$ROLE)Not quite but close.
And BWD Transfer?
t.test(df$TRANSFER2 ~df$ROLE)More clearly not.
So, removing the non-engaged students doesn’t help with the significance testing. Suggestion is to not do analysis with these removed because it raises issues with selection criteria. Keep discussion to qualitative to have explanation for no learning/loss cases.
Reset df in case the selection of non engaged was peformed:
df = read.csv("S2_Pre_Post_full.csv", header = T)
df$ROLE <- factor(df$ROLE, levels = c(1,9), labels = c("tutor_first", "tutee_first"))Correlations
Let’s concentrate on the posttest items because we assume more or less zero knowledge in pre-test.
postdf <- df %>% select(starts_with("POST"))Note: We use the Stdcode and the factors here was well. Perhaps this can be done more elegantly?
Look at the correlations of the post-test items. rcorr() is from the package Hmisc.
rcorr(as.matrix(postdf)) POSTP1Q1 POSTP1Q2 POSTP1Q3 POSTP1Q4 POSTP2Q1 POSTP2Q2 POSTP2Q3 POSTP2Q4 POSTSCORE
POSTP1Q1 1.00 0.57 0.22 0.27 0.56 0.37 0.29 0.11 0.63
POSTP1Q2 0.57 1.00 0.32 0.67 0.60 0.69 0.30 0.36 0.87
POSTP1Q3 0.22 0.32 1.00 0.28 0.24 0.33 0.39 -0.06 0.43
POSTP1Q4 0.27 0.67 0.28 1.00 0.50 0.45 0.15 0.20 0.68
POSTP2Q1 0.56 0.60 0.24 0.50 1.00 0.61 0.19 0.26 0.76
POSTP2Q2 0.37 0.69 0.33 0.45 0.61 1.00 0.40 0.43 0.83
POSTP2Q3 0.29 0.30 0.39 0.15 0.19 0.40 1.00 0.47 0.55
POSTP2Q4 0.11 0.36 -0.06 0.20 0.26 0.43 0.47 1.00 0.55
POSTSCORE 0.63 0.87 0.43 0.68 0.76 0.83 0.55 0.55 1.00
n= 46
P
POSTP1Q1 POSTP1Q2 POSTP1Q3 POSTP1Q4 POSTP2Q1 POSTP2Q2 POSTP2Q3 POSTP2Q4 POSTSCORE
POSTP1Q1 0.0000 0.1410 0.0689 0.0000 0.0103 0.0537 0.4467 0.0000
POSTP1Q2 0.0000 0.0320 0.0000 0.0000 0.0000 0.0458 0.0148 0.0000
POSTP1Q3 0.1410 0.0320 0.0559 0.1115 0.0243 0.0069 0.6966 0.0031
POSTP1Q4 0.0689 0.0000 0.0559 0.0004 0.0016 0.3325 0.1737 0.0000
POSTP2Q1 0.0000 0.0000 0.1115 0.0004 0.0000 0.2183 0.0844 0.0000
POSTP2Q2 0.0103 0.0000 0.0243 0.0016 0.0000 0.0057 0.0026 0.0000
POSTP2Q3 0.0537 0.0458 0.0069 0.3325 0.2183 0.0057 0.0011 0.0000
POSTP2Q4 0.4467 0.0148 0.6966 0.1737 0.0844 0.0026 0.0011 0.0000
POSTSCORE 0.0000 0.0000 0.0031 0.0000 0.0000 0.0000 0.0000 0.0000 Any P LEQ0.05 can be considered significant.
Cluster analysis on post-test data
We can think of a students’ scores in the post-test as a kind of profile, and ask if there are clusters of students with similar profiles. This is what a cluster analysis lets us find out.
We may have to think about what the post-test values mean and if a standardisation is required. We might need to standardise if the maximal scores are different between items.
No standardisation
Using Euclidian distance, we compute the distance between the students:
dist.eucl <- dist(postdf, method = "euclidean")The first 10 students’ distances are:
round(as.matrix(dist.eucl)[1:10, 1:10], 1)The smaller the value, the more similar the students’ score profile.
Lets’ find clusters and visualise them.
posthc <- hclust(d = dist.eucl, method = "ward.D2")# cex: label size
fviz_dend(posthc, cex = 0.5)
Dalal, the students are the row numbers in the excel table minus 1 for variable names.
and can you see a patern at the level where we have tree clusters?
You read the dendrogram from bottom to top, see here
With standardisation
Need to have a look at the differences in the clustering results once I understand the implications of standardisation more. If the items have differnt maximal scores, the standardisation is necessary in any case. But we already rescaled items on 1-10 values, so I think at this stage that scalilng in the CA sense is not needed.
postdf_std <- scale(postdf)
head(postdf_std, nrow=6) POSTP1Q1 POSTP1Q2 POSTP1Q3 POSTP1Q4 POSTP2Q1 POSTP2Q2 POSTP2Q3 POSTP2Q4
[1,] -0.9631983 0.6488973 -0.08071886 0.6175402 -0.05361556 -1.195598 0.2464416 -0.5387811
[2,] -0.9631983 0.6488973 -0.08071886 0.6175402 -0.05361556 1.058399 1.2770158 0.7004155
[3,] 1.0377685 0.6488973 -0.08071886 0.6175402 1.31655988 1.058399 0.2464416 0.7004155
[4,] 1.0377685 0.6488973 -0.08071886 0.6175402 -0.05361556 1.058399 0.2464416 0.7004155
[5,] -1.8207556 -1.7778731 -2.20247167 0.6175402 -1.42379100 -1.195598 -1.1276572 0.7004155
[6,] -1.2490508 -1.7778731 -0.08071886 -1.8526207 -1.42379100 -1.195598 0.2464416 -1.7779778
POSTSCORE
[1,] -0.2706258
[2,] 0.6477273
[3,] 1.1069038
[4,] 0.8518057
[5,] -1.3420376
[6,] -1.8522338dist.eucl <- dist(postdf_std, method = "euclidean")
posthc <- hclust(d = dist.eucl, method = "ward.D2")
fviz_dend(posthc, cex = 0.5)
Analysis based on the coding categories
df_coded = read_excel("coding-counts.xlsx")
df_coded$Study <- factor(df_coded$Study)
df_codes <- select(df_coded, -c("Stdcode", "Study"))Correlations
There are lots of zeros, so this by and large may not be a good idea.
# rcorr(as.matrix(df_codes))Cluster analysis on codes
We can think of a students’ scores in the post-test as a kind of profile, and ask if there are clusters of students with similar profiles. This is what a cluster analysis lets us find out.
We may have to think about what the post-test values mean and if a standardisation is required. We might need to standardise if the maximal scores are different between items.
No standardisation
Using Euclidian distance, we compute the distance between the students:
dist.eucl <- dist(df_codes, method = "euclidean")The first 10 students’ distances are:
round(as.matrix(dist.eucl)[1:10, 1:10], 1) 1 2 3 4 5 6 7 8 9 10
1 0.0 13.0 13.6 24.9 13.4 22.7 11.1 19.4 12.4 13.6
2 13.0 0.0 13.5 20.6 13.4 20.3 15.0 15.8 15.6 9.4
3 13.6 13.5 0.0 26.5 17.3 26.5 16.9 23.5 19.8 12.9
4 24.9 20.6 26.5 0.0 19.1 8.4 21.0 8.3 17.7 17.7
5 13.4 13.4 17.3 19.1 0.0 17.3 8.6 15.4 13.3 13.2
6 22.7 20.3 26.5 8.4 17.3 0.0 17.5 7.9 15.9 17.6
7 11.1 15.0 16.9 21.0 8.6 17.5 0.0 15.9 11.0 13.7
8 19.4 15.8 23.5 8.3 15.4 7.9 15.9 0.0 12.6 14.0
9 12.4 15.6 19.8 17.7 13.3 15.9 11.0 12.6 0.0 14.5
10 13.6 9.4 12.9 17.7 13.2 17.6 13.7 14.0 14.5 0.0The smaller the value, the more similar the students’ score profile.
Lets’ find clusters and visualise them.
posthc <- hclust(d = dist.eucl, method = "ward.D2")# cex: label size
fviz_dend(posthc, cex = 0.5)
With standardisation
postdf_std <- scale(df_codes)
dist.eucl <- dist(postdf_std, method = "euclidean")
posthc <- hclust(d = dist.eucl, method = "ward.D2")
fviz_dend(posthc, cex = 0.5)
LS0tCnRpdGxlOiAiU3R1ZHkgMiBBbmFseXNpcyBTdGFuZGFyZGlzZWQgdGVzdCBzY29yZXMiCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KCgpBc3N1bWVzIGxpYnJhcmllcyB0aWR5dmVyc2UsIGRlc2NyaXB0ciwgZ3JpZEV4dHJhCgoKYGBge3IgaW5jbHVkZT1GQUxTRX0KbGlicmFyeSgidGlkeXZlcnNlIikKbGlicmFyeSgiZGVzY3JpcHRyIikKbGlicmFyeSgiZ3JpZEV4dHJhIikKbGlicmFyeSgicmVhZHhsIikKYGBgCgojIyMgUmVhZCBhbmQgY2xlYW4gZGF0YSAKCgpgYGB7cn0KZGZfcmF3ID0gcmVhZF9leGNlbCgiUzJfUHJlX1Bvc3RfZnVsbC54bHN4IikKYGBgCgpSZXNjYWxpbmcgbmVlZGVkOiAKKiBQMVExIG1heCA2LCBRMiBtYXggNCwgUTMgbWF4IDMsIFE0IG1heCAyLCB0b3RhbCBtYXggMTUKKiBQMlExIG1heCA0LCBRMiBtYXggNCwgUTMgbWF4IDMsIFE0IG1heCAyLCB0b3RhbCBtYXggMTMKKiBUUkFOU0ZFUjEgYW5kIDIgbWF4IDEwIGVhY2guIAoKSW5zdGVhZCBvZiB0aGUgYWJzb2x1dGUgc2NvcmVzIHdlIG5lZWQgdGhlIHBlcmNlbnRhZ2UgaW4gdGVybXMgb2YgdGhlIG1heGltdW0gc2NvcmUuIApXZSBjYW4gdXNlIGEgc2NhbGUgZnJvbSAwIHRvIDEwIHdpdGggaW50ZWdlciB2YWx1ZXMuIAoKYGBge3J9CmRmIDwtIGRmX3JhdwpkZiRST0xFIDwtIGZhY3RvcihkZiRST0xFLCBsZXZlbHMgPSBjKDEsOSksIGxhYmVscyA9IGMoInR1dG9yX2ZpcnN0IiwgInR1dGVlX2ZpcnN0IikpCmRmJFBhaXIgPC0gZmFjdG9yKGRmJFBhaXIpCiMgUHJldGVzdCBTY2F0dGVycGxvdApkZiRQMVExIDwtYXMuaW50ZWdlcihyb3VuZChkZiRQMVExLzYuMCAqIDEwLCBkaWdpdHMgPSAwKSkKZGYkUDFRMiA8LWFzLmludGVnZXIocm91bmQoZGYkUDFRMi80LjAgKiAxMCwgZGlnaXRzID0gMCkpCmRmJFAxUTMgPC1hcy5pbnRlZ2VyKHJvdW5kKGRmJFAxUTMvMy4wICogMTAsIGRpZ2l0cyA9IDApKQpkZiRQMVE0IDwtYXMuaW50ZWdlcihyb3VuZChkZiRQMVE0LzIuMCAqIDEwLCBkaWdpdHMgPSAwKSkKIyBQcmV0ZXN0IEJXRApkZiRQMlExIDwtYXMuaW50ZWdlcihyb3VuZChkZiRQMlExLzQuMCAqIDEwLCBkaWdpdHMgPSAwKSkKZGYkUDJRMiA8LWFzLmludGVnZXIocm91bmQoZGYkUDJRMi80LjAgKiAxMCwgZGlnaXRzID0gMCkpCmRmJFAyUTMgPC1hcy5pbnRlZ2VyKHJvdW5kKGRmJFAyUTMvMy4wICogMTAsIGRpZ2l0cyA9IDApKQpkZiRQMlE0IDwtYXMuaW50ZWdlcihyb3VuZChkZiRQMlE0LzIuMCAqIDEwLCBkaWdpdHMgPSAwKSkKIyBQb3N0LXRlc3QgU2NhdHRlcnBsb3QKZGYkUE9TVFAxUTEgPC1hcy5pbnRlZ2VyKHJvdW5kKGRmJFBPU1RQMVExLzYuMCAqIDEwLCBkaWdpdHMgPSAwKSkKZGYkUE9TVFAxUTIgPC1hcy5pbnRlZ2VyKHJvdW5kKGRmJFBPU1RQMVEyLzQuMCAqIDEwLCBkaWdpdHMgPSAwKSkKZGYkUE9TVFAxUTMgPC1hcy5pbnRlZ2VyKHJvdW5kKGRmJFBPU1RQMVEzLzMuMCAqIDEwLCBkaWdpdHMgPSAwKSkKZGYkUE9TVFAxUTQgPC1hcy5pbnRlZ2VyKHJvdW5kKGRmJFBPU1RQMVE0LzIuMCAqIDEwLCBkaWdpdHMgPSAwKSkKIyBQb3N0LXRlc3QgQldECmRmJFBPU1RQMlExIDwtYXMuaW50ZWdlcihyb3VuZChkZiRQT1NUUDJRMS80LjAgKiAxMCwgZGlnaXRzID0gMCkpCmRmJFBPU1RQMlEyIDwtYXMuaW50ZWdlcihyb3VuZChkZiRQT1NUUDJRMi80LjAgKiAxMCwgZGlnaXRzID0gMCkpCmRmJFBPU1RQMlEzIDwtYXMuaW50ZWdlcihyb3VuZChkZiRQT1NUUDJRMy8zLjAgKiAxMCwgZGlnaXRzID0gMCkpCmRmJFBPU1RQMlE0IDwtYXMuaW50ZWdlcihyb3VuZChkZiRQT1NUUDJRNC8yLjAgKiAxMCwgZGlnaXRzID0gMCkpCiMgVHJhbnNmZXIgaXRlbXMgdG8gaW50ZWdlcnMKZGYkVFJBTlNGRVIxIDwtIGFzLmludGVnZXIoZGYkVFJBTlNGRVIxKQpkZiRUUkFOU0ZFUjIgPC0gYXMuaW50ZWdlcihkZiRUUkFOU0ZFUjIpCmBgYAoKTm93IHdlIG5lZWQgdG8gcmUtY29tcHV0ZSB0aGUgbWFyZ2luYWwgc2NvcmVzLiBMZXQncyBmaXJzdCBkcm9wIHRoZSBvbGQgY29sdW1uczogCgpgYGB7cn0KZGYgPC0gc2VsZWN0KGRmLCAtYygiUFJFLVNDT1JFIiwgIlBPU1QtU0NPUkUiKSkKYGBgCgpBbmQgbm93IGNvbXB1dGUgdGhlIG5ldyBtYXJnaW5hbCBzY29yZXM6IAoKYGBge3J9CmRmIDwtIG11dGF0ZShkZiwgUFJFU0NPUkUgPSBQMVExICsgUDFRMiArIFAxUTMgKyBQMVE0ICsgUDJRMSArIFAyUTIgKyBQMlEzICsgUDJRNCkKZGYgPC0gbXV0YXRlKGRmLCBQT1NUU0NPUkUgPSBQT1NUUDFRMSArIFBPU1RQMVEyICsgUE9TVFAxUTMgKyBQT1NUUDFRNCArIFBPU1RQMlExICsgUE9TVFAyUTIgKyBQT1NUUDJRMyArIFBPU1RQMlE0KQojIGF2ZXJhZ2UgIHNjb3JlczoKZGYgPC0gbXV0YXRlIChkZiwgUFJFQVZHID0gUFJFU0NPUkUvOCkKZGYgPC0gbXV0YXRlIChkZiwgUE9TVEFWRyA9IFBPU1RTQ09SRS84KQpgYGAKCkkgdGhpbmsgd2UgYXJlIHJlYWR5IG5vdyBmb3IgdGhlIGFuYWx5c2lzLiAKCiMgVW5pdmFyaXRlIGFuYWx5c2lzCgojIyBQcmV0ZXN0IAoKVGhlIHByaW5jaXBlICBtYXhpbWFsIHNjb3JlIGluIHRoZSB0ZXN0IGlzIDgwLiAKCmBgYHtyfQpkc19zdW1tYXJ5X3N0YXRzKGRmLFBSRVNDT1JFKQpgYGAKCiAKYGBge3J9CmdncGxvdChkZiwgYWVzKFBSRVNDT1JFKSkgKyAKICBnZW9tX2hpc3RvZ3JhbShiaW5zID0gMTApCmBgYAoKCmBgYHtyfQpnZ3Bsb3QoZGYsIGFlcyh4ID0gMSwgeSA9IFBSRVNDT1JFKSkgKwogIGdlb21fYm94cGxvdCgpICsgCiAgc2NhbGVfeF9jb250aW51b3VzKGJyZWFrcyA9IE5VTEwpICsgCiAgdGhlbWUoYXhpcy50aXRsZS54ID0gZWxlbWVudF9ibGFuaygpKQpgYGAKVG8gaW50ZXJwcmV0IGNoYW5nZXMgZHVlIHRvIFJPTEUgbGF0ZXIsIGlzIHRoZXJlIGEgZGlmZmVybmNlIGluIHRoZSBwcmUtdGVzdCBiZXR3ZWVuIHN0dWRlbnRzIHRoYXQgc3Vic2VxZW50bHkgd2VyZSBpbiB0aGUgdHV0b3JfZmlyc3Qgb3IgdHV0ZWVfZmlyc3Qgcm9sZT8gCgpgYGB7cn0KZ2dwbG90KGRmLCBhZXMoeCA9IFJPTEUsIHkgPSBQUkVTQ09SRSkpICsKICBnZW9tX2JveHBsb3QoKSArIAogIHhsYWIoIlR1dG9yIHJvbGUiKQpgYGAKV2hpbGUgdGhlIHR1dG9yX2ZpcnN0IGlzIHNsaWdodGx5IGJldHRlciwgdGhpcyBpcyBsaWtlbHkgcmFuZG9tLiBBIHQtdGVzdCBhZ3JlZXMsIHdpdGggcCBncmVhdGVyIHRoYW4gMC4wNS4gCgoKYGBge3J9CnQudGVzdChkZiRQUkVTQ09SRSB+ZGYkUk9MRSkKYGBgClRoZSBub24tcGFyYW1ldHJpYyBXaWxjb3hvbiB0ZXN0IGZ1cnRoZXIgY29uZmlybXMgdGhhdCB0aGVyZSBpcyBubyBzaWduaWZpY2FudCBkaWZmZXJlbmNlIGJldHdlZW4gdGhlIHR3byBncm91cHM6IAoKYGBge3J9CndpbGNveC50ZXN0KGRmJFBSRVNDT1JFIH5kZiRST0xFKQpgYGAKCgojIyAgUG9zdCB0ZXN0CgpgYGB7cn0KZHNfc3VtbWFyeV9zdGF0cyhkZixQT1NUU0NPUkUpCmBgYAoKYGBge3J9CiMgZ2dwbG90KGRmLCBhZXMoUE9TVFNDT1JFKSkgKyBnZW9tX2JhcigpCmdncGxvdChkZiwgYWVzKFBPU1RTQ09SRSkpICsgCiAgZ2VvbV9oaXN0b2dyYW0oYmlucyA9IDEwKQpgYGAKCkNoZWNrOndlJ2QgbGlrZSB0byBrbm93IGlmIHRoZSBjYXNlcyB3aXRoIHRoZSBsb3cgdmFsdWVzIGFyZSB0aGUgc2FtZSBmcm9tIHByZSB0byBwb3N0IHRlc3QuIFRoYXQgd291bGQgaW5kaWNhdGUgbm9uLWVuZ2FnbWVudC4gCgpgYGB7cn0KZ2dwbG90KGRmLCBhZXMoeCA9IDEsIHkgPSBQT1NUU0NPUkUpKSArCiAgZ2VvbV9ib3hwbG90KCkgKyAKICBzY2FsZV94X2NvbnRpbnVvdXMoYnJlYWtzID0gTlVMTCkgKyAKICB0aGVtZShheGlzLnRpdGxlLnggPSBlbGVtZW50X2JsYW5rKCkpCmBgYAoKQnkgcm9sZTogCgpgYGB7cn0KZ2dwbG90KGRmLCBhZXMoeCA9IFJPTEUsIHkgPSBQT1NUU0NPUkUpKSArCiAgZ2VvbV9ib3hwbG90KCkgKyAKICB4bGFiKCJUdXRvciByb2xlIikKYGBgClRoZSBkaWZmZXJlbmNlIGJldHdlZW4gdGhlIHR3byBjb25kaXRpb25zIGlzIG1hcmdpbmFsLCBieSBpbnNwZWN0aW9uLCBhbHNvIGtlZXBpbmcgaW4gbWluZCB0aGF0IHRoZSBwcmUtdGVzdCBzY29yZXMgd2hlcmUgc2xpZ2x5dCBlbGF2YXRlZCBmb3IgdGhlIHR1dG9yX2ZpcnN0IGNvbmRpdGlvbi4gQSB0ZXN0IHJldmVhbHMgbm8gc2lnbmlmaWNhbnQgZGlmZmVyZW5jZS4gCgpgYGB7cn0KdC50ZXN0KGRmJFBPU1RTQ09SRSB+ZGYkUk9MRSkKYGBgCkluIHRoZSBmdXJ0aGVyIGFuYWx5c2lzIHdlIHRyZWF0IHRoZSB0d28gZ3JvdXBzIGFzIGNvbXBhcmFibGUuIAoKIyBUcmVhdG1lbnQgZWZmZWN0cwoKIyMgU3VtbWFyeSBzY29yZXMKClRoZSBpbnRlcnZlbnRpb24gd2FzIGNsZWFybHkgZWZmZWN0aXZlOiAKCmBgYHtyfQp0LnRlc3QoZGYkUE9TVFNDT1JFLCBkZiRQUkVTQ09SRSwgcGFpcmVkPVQpCmBgYAoKTG9va2luZyBhdCB0aGUgaW5kaXZpZHVhbCBnYWluIHNjb3Jlcywgd2Ugc2VlIHRoYXQgaGlsZSBtb3N0IG9mIHRoZSBnYWluIHNjb3JlcyBhcmUgcG9zaXRpdmUsIHdpdGggbmVnYXRpdmUgdmFsdWVzIGJlaW5nIGZldyBhbmQgc21hbGwgdmFsdWVzIG9ubHksIHRoZSBsb3NzIHNjb3JlIG9mIDMzIHBvaW50cyBmb3Igc3R1ZGVudCBpbiBwb3NpdGlvbiA4IChTMi1BTS05KSBpcyByYXRoZXIgZXh0cmVtZS4gCmBgYHtyfQpkZiRnYWluIDwtIGRmJFBPU1RTQ09SRSAtIGRmJFBSRVNDT1JFCmRmJGdhaW4KYGBgCgpUaGlzIGNhbiBiZSBleHBsYWluZWQgYnkgdGhpcyBzdHVkZW50IG5vdCBhY3R1YWxseSBlbmdhZ2luZyB3aXRoIHRoZSBwb3N0LXRlc3QuLi4uCgpgYGB7cn0KZHNfc3VtbWFyeV9zdGF0cyhkZiwgZ2FpbikKYGBgCgojIyBBdmVyYWdlZCBzY29yZXMKClN0cm9uZyBsZWFybmluZyAgZ2FpbnM6IApgYGB7cn0KdC50ZXN0KGRmJFBPU1RBVkcsIGRmJFBSRUFWRywgcGFpcmVkPVQpCmBgYAoKYGBge3J9CmRmJGdhaW5fYXZnIDwtIGRmJFBPU1RBVkcgLSBkZiRQUkVBVkcKZGYkZ2Fpbl9hdmcKYGBgCgpgYGB7cn0KZHNfc3VtbWFyeV9zdGF0cyhkZixnYWluX2F2ZykKYGBgCgojIyBBbmFseXNpcyBvZiBubyBsZWFybmluZyBjYXNlcyAKCkdpdmVuIHRoYXQgdGhlIHN0dWRlbnRzIGhhZCBubyB2ZXJ5IGxpdHRsZSBpbml0aWFsIGtub3dsZWRnZSwgbG9zc2VzIGluIHRoZSBwb3N0IHRlc3QgbmVlZCB0byBiZSBleHBsYWluZWQgb24gYSBjYXNlIGJ5IGNhc2UgYmFzaXMuIFdlIGRvIHRoaXMgaW4gdGhyZWUgc2VjdGlvbnM6IAoKIyMjIExvdyBwcmUtdGVzdCBzY29yZXMgYW5kIG5vIGdhaW4KCk5vIGdhaW4gb3IgbG9zcyBsaWtlbHkgbWVhbnMgbm9uLWVuZ2FnZW1lbnQuIFRoaXMgcGF0dGVybiBob2xkcyBmb3IgdGhlc2UgY2FzZXM6IAoKIyMjIEhpZ2ggcHJlLXRlc3Qgc2NvcmUgYW5kIGxvc3MKClMyLUFNLTk6IAoKT3RoZXJzPyAKCgojIEF2ZXJhZ2Ugc2NvcmVzICBhbmFseXNpcyAKCgojIyBBdmVyYWdlIFByZXRlc3QgCgpUaGUgcHJpbmNpcGUgIG1heGltYWwgc2NvcmUgaW4gdGhlIHRlc3QgaXMgODAuIAoKYGBge3J9CmRzX3N1bW1hcnlfc3RhdHMoZGYsUFJFQVZHKQpgYGAKCiAKYGBge3J9CmdncGxvdChkZiwgYWVzKFBSRUFWRykpICsgCiAgZ2VvbV9oaXN0b2dyYW0oYmlucyA9IDEwKQpgYGAKCgpgYGB7cn0KZ2dwbG90KGRmLCBhZXMoeCA9IDEsIHkgPSBQUkVBVkcpKSArCiAgZ2VvbV9ib3hwbG90KCkgKyAKICBzY2FsZV94X2NvbnRpbnVvdXMoYnJlYWtzID0gTlVMTCkgKyAKICB0aGVtZShheGlzLnRpdGxlLnggPSBlbGVtZW50X2JsYW5rKCkpCmBgYApUbyBpbnRlcnByZXQgY2hhbmdlcyBkdWUgdG8gUk9MRSBsYXRlciwgaXMgdGhlcmUgYSBkaWZmZXJuY2UgaW4gdGhlIHByZS10ZXN0IGJldHdlZW4gc3R1ZGVudHMgdGhhdCBzdWJzZXFlbnRseSB3ZXJlIGluIHRoZSB0dXRvcl9maXJzdCBvciB0dXRlZV9maXJzdCByb2xlPyAKCmBgYHtyfQpnZ3Bsb3QoZGYsIGFlcyh4ID0gUk9MRSwgeSA9IFBSRUFWRykpICsKICBnZW9tX2JveHBsb3QoKSArIAogIHhsYWIoIlR1dG9yIHJvbGUiKQpgYGAKV2hpbGUgdGhlIHR1dG9yX2ZpcnN0IGlzIHNsaWdodGx5IGJldHRlciwgdGhpcyBpcyBsaWtlbHkgcmFuZG9tLiBBIHQtdGVzdCBhZ3JlZXMsIHdpdGggcCBncmVhdGVyIHRoYW4gMC4wNS4gCgoKYGBge3J9CnQudGVzdChkZiRQUkVBVkcgfmRmJFJPTEUpCmBgYApUaGUgbm9uLXBhcmFtZXRyaWMgV2lsY294b24gdGVzdCBmdXJ0aGVyIGNvbmZpcm1zIHRoYXQgdGhlcmUgaXMgbm8gc2lnbmlmaWNhbnQgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSB0d28gZ3JvdXBzOiAKCmBgYHtyfQp3aWxjb3gudGVzdChkZiRQUkVBVkcgfmRmJFJPTEUpCmBgYAoKIyMgQXZlcmFnZSBQb3N0dGVzdAoKVGhlIHByaW5jaXBlICBtYXhpbWFsIHNjb3JlIGluIHRoZSB0ZXN0IGlzIDgwLiAKCmBgYHtyfQpkc19zdW1tYXJ5X3N0YXRzKGRmLFBPU1RBVkcpCmBgYAoKIApgYGB7cn0KZ2dwbG90KGRmLCBhZXMoUE9TVEFWRykpICsgCiAgZ2VvbV9oaXN0b2dyYW0oYmlucyA9IDEwKQpgYGAKCgpgYGB7cn0KZ2dwbG90KGRmLCBhZXMoeCA9IDEsIHkgPSBQT1NUQVZHKSkgKwogIGdlb21fYm94cGxvdCgpICsgCiAgc2NhbGVfeF9jb250aW51b3VzKGJyZWFrcyA9IE5VTEwpICsgCiAgdGhlbWUoYXhpcy50aXRsZS54ID0gZWxlbWVudF9ibGFuaygpKQpgYGAKIGlzIHRoZXJlIGEgZGlmZmVybmNlICBiZXR3ZWVuIHN0dWRlbnRzIHRoYXQgc3Vic2VxZW50bHkgd2VyZSBpbiB0aGUgdHV0b3JfZmlyc3Qgb3IgdHV0ZWVfZmlyc3Qgcm9sZT8gCgpgYGB7cn0KZ2dwbG90KGRmLCBhZXMoeCA9IFJPTEUsIHkgPSBQT1NUQVZHKSkgKwogIGdlb21fYm94cGxvdCgpICsgCiAgeGxhYigiVHV0b3Igcm9sZSIpCmBgYApXaGlsZSB0aGUgdHV0b3JfZmlyc3QgaXMgc2xpZ2h0bHkgYmV0dGVyLCB0aGlzIGlzIGxpa2VseSByYW5kb20uIEEgdC10ZXN0IGFncmVlcywgd2l0aCBwIGdyZWF0ZXIgdGhhbiAwLjA1LiAKCgpgYGB7cn0KdC50ZXN0KGRmJFBPU1RBVkcgfmRmJFJPTEUpCmBgYApUaGUgbm9uLXBhcmFtZXRyaWMgV2lsY294b24gdGVzdCBmdXJ0aGVyIGNvbmZpcm1zIHRoYXQgdGhlcmUgaXMgbm8gc2lnbmlmaWNhbnQgZGlmZmVyZW5jZSBiZXR3ZWVuIHRoZSB0d28gZ3JvdXBzOiAKCmBgYHtyfQp3aWxjb3gudGVzdChkZiRQT1NUQVZHIH5kZiRST0xFKQpgYGAKCgojIEl0ZW0gbGV2ZWwgYW5hbHlzaXMgUHJlIGFuZCBQb3N0LiAKCiMjIFNjYXR0ZXJwbG90IFExCgpgYGB7cn0KZGYkUDFRMQpkZiRQT1NUUDFRMQpgYGAKCgpgYGB7cn0KZHNfc3VtbWFyeV9zdGF0cyhkZixQMVExLCBQT1NUUDFRMSApCmBgYAoKCgpgYGB7cn0KYm94cGxvdChkZiRQMVExLCAgZGF0YSA9IGRmKQpib3hwbG90KGRmJFBPU1RQMVExLCAgZGF0YSA9IGRmKQpgYGAKCmBgYHtyfQpnZ3Bsb3QoZGYsIGFlcyhQMVExKSkgKyBnZW9tX2JhcigpCmdncGxvdChkZiwgYWVzKFBPU1RQMVExKSkgKyBnZW9tX2JhcigpCmBgYAoKIyMgU2NhdHRlcnBsb3QgIFEyCgpgYGB7cn0KZGYkUDFRMgpkZiRQT1NUUDFRMgpgYGAKCgpgYGB7cn0KZHNfc3VtbWFyeV9zdGF0cyhkZixQMVEyLCBQT1NUUDFRMikKYGBgCgpgYGB7cn0KZ2dwbG90KGRmLCBhZXMoUDFRMikpICsgZ2VvbV9iYXIoKQpnZ3Bsb3QoZGYsIGFlcyhQT1NUUDFRMikpICsgZ2VvbV9iYXIoKQpgYGAKPiBUaGUgcHJlc3Rlc3QgaGFzIGEgdmVyeSBvZGQgZGlzdHJpYnV0aW9uOiBDaGVjayEgCgojIyBTY2F0dGVycGxvdCBRMwoKYGBge3J9CmRmJFAxUTMKZGYkUE9TVFAxUTMKYGBgCgoKYGBge3J9CmRzX3N1bW1hcnlfc3RhdHMoZGYsUDFRMywgUE9TVFAxUTMgKQpgYGAKCgoKYGBge3J9CmJveHBsb3QoZGYkUDFRMywgIGRhdGEgPSBkZikKYm94cGxvdChkZiRQT1NUUDFRMywgIGRhdGEgPSBkZikKYGBgCgpgYGB7cn0KZ2dwbG90KGRmLCBhZXMoUDFRMykpICsgZ2VvbV9iYXIoKQpnZ3Bsb3QoZGYsIGFlcyhQT1NUUDFRMykpICsgZ2VvbV9iYXIoKQpgYGAKCiMjIFNjYXR0ZXJwbG90IFE0CgpgYGB7cn0KZGYkUDFRNApkZiRQT1NUUDFRNApgYGAKCgpgYGB7cn0KZHNfc3VtbWFyeV9zdGF0cyhkZixQMVE0LCBQT1NUUDFRNCApCmBgYAoKCgpgYGB7cn0KYm94cGxvdChkZiRQMVE0LCAgZGF0YSA9IGRmKQpib3hwbG90KGRmJFBPU1RQMVE0LCAgZGF0YSA9IGRmKQpgYGAKCmBgYHtyfQpnZ3Bsb3QoZGYsIGFlcyhQMVE0KSkgKyBnZW9tX2JhcigpCmdncGxvdChkZiwgYWVzKFBPU1RQMVE0KSkgKyBnZW9tX2JhcigpCmBgYAoKIyMgQldEIFExCgpgYGB7cn0KZGYkUDJRMQpkZiRQT1NUUDJRMQpgYGAKCgpgYGB7cn0KZHNfc3VtbWFyeV9zdGF0cyhkZixQMlExLCBQT1NUUDJRMSApCmBgYAoKCgpgYGB7cn0KYm94cGxvdChkZiRQMlExLCAgZGF0YSA9IGRmKQpib3hwbG90KGRmJFBPU1RQMlExLCAgZGF0YSA9IGRmKQpgYGAKCmBgYHtyfQpnZ3Bsb3QoZGYsIGFlcyhQMlExKSkgKyBnZW9tX2JhcigpCmdncGxvdChkZiwgYWVzKFBPU1RQMlExKSkgKyBnZW9tX2JhcigpCmBgYAoKIyMgQldEIFEyCgpgYGB7cn0KZGYkUDJRMgpkZiRQT1NUUDJRMgpgYGAKCgpgYGB7cn0KZHNfc3VtbWFyeV9zdGF0cyhkZixQMlEyLCBQT1NUUDJRMiApCmBgYAoKCgpgYGB7cn0KYm94cGxvdChkZiRQMlEyLCAgZGF0YSA9IGRmKQpib3hwbG90KGRmJFBPU1RQMlEyLCAgZGF0YSA9IGRmKQpgYGAKCmBgYHtyfQpnZ3Bsb3QoZGYsIGFlcyhQMlEyKSkgKyBnZW9tX2JhcigpCmdncGxvdChkZiwgYWVzKFBPU1RQMlEyKSkgKyBnZW9tX2JhcigpCmBgYAojIyBCV0QgUTMKCmBgYHtyfQpkZiRQMlEzCmRmJFBPU1RQMlEzCmBgYAoKCmBgYHtyfQpkc19zdW1tYXJ5X3N0YXRzKGRmLFAyUTMsIFBPU1RQMlEzICkKYGBgCgoKCmBgYHtyfQpib3hwbG90KGRmJFAyUTMsICBkYXRhID0gZGYpCmJveHBsb3QoZGYkUE9TVFAyUTMsICBkYXRhID0gZGYpCmBgYAoKYGBge3J9CmdncGxvdChkZiwgYWVzKFAyUTMpKSArIGdlb21fYmFyKCkKZ2dwbG90KGRmLCBhZXMoUE9TVFAyUTMpKSArIGdlb21fYmFyKCkKYGBgCgojIyBCV0QgUTQKCmBgYHtyfQpkZiRQMlE0CmRmJFBPU1RQMlE0CmBgYAoKCmBgYHtyfQpkc19zdW1tYXJ5X3N0YXRzKGRmLFAyUTQsIFBPU1RQMlE0ICkKYGBgCgoKCmBgYHtyfQpib3hwbG90KGRmJFAyUTQsICBkYXRhID0gZGYpCmJveHBsb3QoZGYkUE9TVFAyUTQsICBkYXRhID0gZGYpCmBgYAoKYGBge3J9CmdncGxvdChkZiwgYWVzKFAyUTQpKSArIGdlb21fYmFyKCkKZ2dwbG90KGRmLCBhZXMoUE9TVFAyUTQpKSArIGdlb21fYmFyKCkKYGBgCgoKCiMjIFNjYXR0ZXJwbG90IFRyYW5zZmVyCgpgYGB7cn0KZGYkVFJBTlNGRVIxCmBgYAoKCmBgYHtyfQpkc19zdW1tYXJ5X3N0YXRzKGRmLFRSQU5TRkVSMSApCmBgYAoKCgpgYGB7cn0KYm94cGxvdChkZiRUUkFOU0ZFUjEsICBkYXRhID0gZGYpCmBgYAoKYGBge3J9CmdncGxvdChkZiwgYWVzKFRSQU5TRkVSMSkpICsgZ2VvbV9oaXN0b2dyYW0oYmlucyA9IDYpCmBgYAoKIyMjIERvZXMgUk9MRSBtYWtlIGEgZGlmZmVyZW5jZT8gIAoKYGBge3J9CmdncGxvdChkZiwgYWVzKHggPSBST0xFLCB5ID0gVFJBTlNGRVIxKSkgKwogIGdlb21fYm94cGxvdCgpICsgCiAgeGxhYigiVHV0b3Igcm9sZSIpCmBgYApWaXN1YWxseSB5ZXMgYW5kIGFsc28gc3RhdGlzdGljYWxseSBzaWduaWZpY2FudDogCmBgYHtyfQp0LnRlc3QoZGYkVFJBTlNGRVIxIH5kZiRST0xFKQpgYGAKIAoKIyMgQldEIFRyYW5zZmVyCgpgYGB7cn0KZGYkVFJBTlNGRVIyCmBgYAoKCmBgYHtyfQpkc19zdW1tYXJ5X3N0YXRzKGRmLFRSQU5TRkVSMiApCmBgYAoKCgpgYGB7cn0KYm94cGxvdChkZiRUUkFOU0ZFUjIsICBkYXRhID0gZGYpCmBgYAoKYGBge3J9CmdncGxvdChkZiwgYWVzKFRSQU5TRkVSMikpICsgZ2VvbV9oaXN0b2dyYW0oYmlucyA9IDYpCmBgYApUaGlzIG1pZ2h0IGluZGljYXRlIHRoYXQgaW4gdGhlIHBvc3QgdGVzdCB3ZSBoYXZlIHR3byBncm91cHMgb2Ygc3R1ZGVudHM6IFRob3NlIHdobyBwcmV0dHkgbXVjaCBnb3QgaXQgYW5kIHRob3NlIHdobyBwcmV0dHkgbXVjaCBub3QuIAoKIyMjIERvZXMgUk9MRSBtYWtlIGEgZGlmZmVyZW5jZT8gIAoKYGBge3J9CmdncGxvdChkZiwgYWVzKHggPSBST0xFLCB5ID0gVFJBTlNGRVIyKSkgKwogIGdlb21fYm94cGxvdCgpICsgCiAgeGxhYigiVHV0b3Igcm9sZSIpCmBgYApWaXN1YWxseSB5ZXMgYnV0ICBtaXNzaW5nIHN0YXRpc3RpY2FsIHNpZ25pZmljYW5jZSBiZWNhdXNlIG9mIHRoZSBsYXJnZSB2YXJpYXRpb24gaW4gdGhlIHNjb3JlczogCmBgYHtyfQp0LnRlc3QoZGYkVFJBTlNGRVIyIH5kZiRST0xFKQpgYGAKRm9yIHRoZSBxdWFsaXRhdGl2ZSBhbmFseXNpcyBpdCB3b3VsZCBiZSB3b3J0aCBmaW5kaW5nIG91dCB3aGF0IGNhbiBiZSBzYWlkIGFib3V0IHRoZSB0d28gJ2V4dHJlbWUnIGdyb3Vwcy4gVGhpcyBpdGVtIHdvdWxkIGFsc28gYmUgdGhlIG9uZSBhcyBhIGJhc2lzIGZvciBleHRyZW0gZ3JvdXAgY29tcGFyaXNvbnMsIHNwbGl0dGluZyBvbiB0aGUgbWVkaWFuLiAKCiMjIyBBIGNvbWJpbmVkIHRyYW5zZmVyIHNjb3JlCgpgYGB7cn0KZGYkVFJBTlNGRVIgPC0gZGYkVFJBTlNGRVIxICsgZGYkVFJBTlNGRVIyCmBgYAoKYGBge3J9CmRzX3N1bW1hcnlfc3RhdHMoZGYsVFJBTlNGRVIgKQpgYGAKCgoKCmBgYHtyfQpib3hwbG90KGRmJFRSQU5TRkVSLCAgZGF0YSA9IGRmKQpgYGAKCmBgYHtyfQpnZ3Bsb3QoZGYsIGFlcyhUUkFOU0ZFUikpICsgZ2VvbV9oaXN0b2dyYW0oYmlucyA9IDgpCmBgYAoKCmBgYHtyfQpnZ3Bsb3QoZGYsIGFlcyh4ID0gUk9MRSwgeSA9IFRSQU5TRkVSKSkgKwogIGdlb21fYm94cGxvdCgpICsgCiAgeGxhYigiVHV0b3Igcm9sZSIpCmBgYAoKYGBge3J9CnQudGVzdChkZiRUUkFOU0ZFUiB+ZGYkUk9MRSkKYGBgCgojIyMgTG93IGFuZCBoaWdoc2NvcmluZyBzdHVkZW50cyBiYXNlZCBvbiB2YXJpb3VzIGNyaXRlcmlhOiAKCkRhbGFsLCB0aGlzIGlzIGZvciBjYXNlcyB3aGVyZSB5b3UgbWF5IHdhbnQgdG8gbWFrZSBleHRyZW0gZ3JvdXAgY29tcGFyaXNvbnMgbGF0ZXIuIEJ1dCB5b3UgcGlja2VkIGEgc3Vic2V0IGFueXdheS4gU28ganVzdCBmb3IgY29tcGxldGVuZXM6CgpTbywgYSBtZWRpYW4gc3BsaXQgY291bGQgYmUgb24gYSBzY29yZSAgOC4wICBvZiBUUkFOU0ZFUjIsIHRoZSBCV0QgaXRlbS4gSXQgd291bGQgeWllbGQgdGhlIGZvbGxvd2luZyAyMCBzdHVkZW50cyBmb3IgbG93IHNjb3JlIChMZXNzIHRoYW4gOCk6IAoKYGBge3J9Cmxvd19zY29yZXJzIDwtIGZpbHRlcihkZiwgVFJBTlNGRVIyIDwgOC4wKQpsb3dfc2NvcmVycyA8LSBhc192ZWN0b3IobG93X3Njb3JlcnMkU3RkY29kZSkKbG93X3Njb3JlcnMKYGBgCkFuZCB0aGVzZSAyNiBmb3Igc2NvcmVzIEdFUSA4LjA6IApgYGB7cn0KaGlnaF9zY29yZXJzIDwtIGZpbHRlcihkZiwgVFJBTlNGRVIyID49IDguMCkKaGlnaF9zY29yZXJzIDwtIGFzX3ZlY3RvcihoaWdoX3Njb3JlcnMkU3RkY29kZSkKaGlnaF9zY29yZXJzCmBgYAoKCiMjIE5vbi1lbmdhZ2VkIHN0dWRlbnRzIChpZ25vcmUgdGhpczsgb25seSBmb3IgY29tcGxldGVuZXNzKQoKRmlsdGVyaW5nIG91dCBzaXggbGlrZWx5IGNhc2VzIG9mIG5vbi1lbmdhZ2VtZW50OiAKCmBgYHtyfQpkZiA8LSBmaWx0ZXIoZGYsIFN0ZGNvZGUgIT0gIlMyX0FNXzkiLCBTdGRjb2RlICE9ICJTMl9BTV82IiwgU3RkY29kZSAhPSAiUzJfQU1fMDYiLCAKICAgICAgICAgICAgIFN0ZGNvZGUgIT0gIlMyX0FNXzE0IiwgU3RkY29kZSAhPSAiUzJfUE1fMDEwIiwgU3RkY29kZSAhPSAiUzJfUE1fMDUiKQpgYGAKClRyZWF0bWVudCBlZmZlY3QgZ2VuZXJhbCA6IAoKYGBge3J9CnQudGVzdChkZiRQT1NULlNDT1JFLCBkZiRQUkUuU0NPUkUsIHBhaXJlZD1UKQpgYGAKU3Ryb25nIGFzIGV2ZXIuIAoKU2lnbmlmaWNhbnQgZGlmZmVyZW5jZXMgcmVnYXJkaW5nIHJvbGUgYmVmb3JlIGludGVydmVudGlvbj8gCgpgYGB7cn0KdC50ZXN0KGRmJFBSRS5TQ09SRSB+ZGYkUk9MRSkKYGBgCk5vLiAKCkFmdGVyIGludGVydmVudGlvbj8gCgpgYGB7cn0KdC50ZXN0KGRmJFBPU1QuU0NPUkUgfmRmJFJPTEUpCmBgYAoKQUxzbyBub3QuIAoKSG93IGFib3V0IFRyYW5zZmVyIGl0ZW0gU2NhdHRlcnBsYXQ/CgoKYGBge3J9CnQudGVzdChkZiRUUkFOU0ZFUjEgfmRmJFJPTEUpCmBgYApOb3QgcXVpdGUgYnV0IGNsb3NlLiAKCkFuZCBCV0QgVHJhbnNmZXI/IAoKYGBge3J9CnQudGVzdChkZiRUUkFOU0ZFUjIgfmRmJFJPTEUpCmBgYAoKTW9yZSBjbGVhcmx5IG5vdC4gCgpTbywgcmVtb3ZpbmcgdGhlIG5vbi1lbmdhZ2VkIHN0dWRlbnRzIGRvZXNuJ3QgaGVscCB3aXRoIHRoZSBzaWduaWZpY2FuY2UgdGVzdGluZy4gU3VnZ2VzdGlvbiBpcyB0byBub3QgZG8gYW5hbHlzaXMgd2l0aCB0aGVzZSByZW1vdmVkIGJlY2F1c2UgaXQgcmFpc2VzIGlzc3VlcyB3aXRoIHNlbGVjdGlvbiBjcml0ZXJpYS4gS2VlcCBkaXNjdXNzaW9uIHRvIHF1YWxpdGF0aXZlIHRvIGhhdmUgZXhwbGFuYXRpb24gZm9yIG5vIGxlYXJuaW5nL2xvc3MgY2FzZXMuIAoKUmVzZXQgZGYgaW4gY2FzZSB0aGUgc2VsZWN0aW9uIG9mIG5vbiBlbmdhZ2VkIHdhcyBwZWZvcm1lZDogCgpgYGB7cn0KZGYgPSByZWFkLmNzdigiUzJfUHJlX1Bvc3RfZnVsbC5jc3YiLCBoZWFkZXIgPSBUKQpkZiRST0xFIDwtIGZhY3RvcihkZiRST0xFLCBsZXZlbHMgPSBjKDEsOSksIGxhYmVscyA9IGMoInR1dG9yX2ZpcnN0IiwgInR1dGVlX2ZpcnN0IikpCmBgYAoKCgoKCiMgQ29ycmVsYXRpb25zCgpMZXQncyBjb25jZW50cmF0ZSBvbiB0aGUgcG9zdHRlc3QgaXRlbXMgYmVjYXVzZSB3ZSBhc3N1bWUgbW9yZSBvciBsZXNzIHplcm8ga25vd2xlZGdlIGluIHByZS10ZXN0LiAKCmBgYHtyfQpwb3N0ZGYgPC0gZGYgJT4lIHNlbGVjdChzdGFydHNfd2l0aCgiUE9TVCIpKQpgYGAKCk5vdGU6IFdlIHVzZSB0aGUgU3RkY29kZSBhbmQgdGhlIGZhY3RvcnMgaGVyZSB3YXMgd2VsbC4gUGVyaGFwcyB0aGlzIGNhbiBiZSBkb25lIG1vcmUgZWxlZ2FudGx5PyAKCmBgYHtyIGluY2x1ZGU9RkFMU0V9CmxpYnJhcnkoIkhtaXNjIikKYGBgCgoKTG9vayBhdCB0aGUgY29ycmVsYXRpb25zIG9mIHRoZSBwb3N0LXRlc3QgaXRlbXMuIGByY29ycigpYCBpcyBmcm9tIHRoZSBwYWNrYWdlIGBIbWlzY2AuIAoKCmBgYHtyfQpyY29ycihhcy5tYXRyaXgocG9zdGRmKSkKYGBgCgpBbnkgUCBMRVEwLjA1IGNhbiBiZSBjb25zaWRlcmVkIHNpZ25pZmljYW50LiAKCgojIENsdXN0ZXIgYW5hbHlzaXMgb24gcG9zdC10ZXN0IGRhdGEKCmBgYHtyIGluY2x1ZGU9RkFMU0V9CmxpYnJhcnkoImNsdXN0ZXIiKQpsaWJyYXJ5KCJmYWN0b2V4dHJhIikKYGBgCgpXZSBjYW4gdGhpbmsgb2YgYSBzdHVkZW50cycgc2NvcmVzIGluIHRoZSBwb3N0LXRlc3QgYXMgYSBraW5kIG9mIHByb2ZpbGUsIGFuZCBhc2sgaWYgdGhlcmUgYXJlIGNsdXN0ZXJzIG9mIHN0dWRlbnRzIHdpdGggc2ltaWxhciBwcm9maWxlcy4gVGhpcyBpcyB3aGF0IGEgY2x1c3RlciBhbmFseXNpcyBsZXRzIHVzIGZpbmQgb3V0LiAKCldlIG1heSBoYXZlIHRvIHRoaW5rIGFib3V0IHdoYXQgdGhlIHBvc3QtdGVzdCB2YWx1ZXMgbWVhbiBhbmQgaWYgYSBzdGFuZGFyZGlzYXRpb24gaXMgcmVxdWlyZWQuIFdlIG1pZ2h0IG5lZWQgdG8gc3RhbmRhcmRpc2UgaWYgdGhlIG1heGltYWwgc2NvcmVzIGFyZSBkaWZmZXJlbnQgYmV0d2VlbiBpdGVtcy4KCiMjIE5vIHN0YW5kYXJkaXNhdGlvbgoKVXNpbmcgRXVjbGlkaWFuIGRpc3RhbmNlLCB3ZSBjb21wdXRlIHRoZSBkaXN0YW5jZSBiZXR3ZWVuIHRoZSBzdHVkZW50czogCgpgYGB7cn0KZGlzdC5ldWNsIDwtIGRpc3QocG9zdGRmLCBtZXRob2QgPSAiZXVjbGlkZWFuIikKYGBgCgpUaGUgZmlyc3QgMTAgc3R1ZGVudHMnIGRpc3RhbmNlcyBhcmU6IAoKYGBge3J9CnJvdW5kKGFzLm1hdHJpeChkaXN0LmV1Y2wpWzE6MTAsIDE6MTBdLCAxKQpgYGAKVGhlIHNtYWxsZXIgdGhlIHZhbHVlLCB0aGUgbW9yZSBzaW1pbGFyIHRoZSBzdHVkZW50cycgc2NvcmUgcHJvZmlsZS4gCgpMZXRzJyBmaW5kIGNsdXN0ZXJzIGFuZCB2aXN1YWxpc2UgdGhlbS4gCgpgYGB7cn0KcG9zdGhjIDwtIGhjbHVzdChkID0gZGlzdC5ldWNsLCBtZXRob2QgPSAid2FyZC5EMiIpCmBgYAoKYGBge3J9CiMgY2V4OiBsYWJlbCBzaXplIApmdml6X2RlbmQocG9zdGhjLCBjZXggPSAwLjUpCmBgYAo+IERhbGFsLCB0aGUgc3R1ZGVudHMgYXJlIHRoZSByb3cgbnVtYmVycyBpbiB0aGUgZXhjZWwgdGFibGUgbWludXMgMSBmb3IgdmFyaWFibGUgbmFtZXMuIAoKPiBhbmQgY2FuIHlvdSBzZWUgYSBwYXRlcm4gIGF0IHRoZSBsZXZlbCB3aGVyZSB3ZSBoYXZlIHRyZWUgY2x1c3RlcnM/IAoKPiBZb3UgcmVhZCB0aGUgZGVuZHJvZ3JhbSBmcm9tIGJvdHRvbSB0byB0b3AsIHNlZSBbaGVyZV0oaHR0cHM6Ly93d3cuZGlzcGxheXIuY29tL3doYXQtaXMtZGVuZHJvZ3JhbS8jOn46dGV4dD1Ib3clMjB0byUyMHJlYWQlMjBhJTIwZGVuZHJvZ3JhbSxvYmplY3RzJTIwYXJlJTIwQSUyMGFuZCUyMEIuKQoKIyMgV2l0aCBzdGFuZGFyZGlzYXRpb24KCk5lZWQgdG8gaGF2ZSBhIGxvb2sgYXQgdGhlIGRpZmZlcmVuY2VzIGluIHRoZSBjbHVzdGVyaW5nIHJlc3VsdHMgb25jZSBJIHVuZGVyc3RhbmQgdGhlIGltcGxpY2F0aW9ucyBvZiBzdGFuZGFyZGlzYXRpb24gbW9yZS4gSWYgdGhlIGl0ZW1zIGhhdmUgZGlmZmVybnQgbWF4aW1hbCBzY29yZXMsIHRoZSBzdGFuZGFyZGlzYXRpb24gaXMgbmVjZXNzYXJ5IGluIGFueSBjYXNlLiBCdXQgd2UgYWxyZWFkeSByZXNjYWxlZCBpdGVtcyBvbiAxLTEwIHZhbHVlcywgc28gSSB0aGluayBhdCB0aGlzIHN0YWdlIHRoYXQgc2NhbGlsbmcgIGluIHRoZSBDQSBzZW5zZSBpcyBub3QgbmVlZGVkLiAKCmBgYHtyfQpwb3N0ZGZfc3RkIDwtIHNjYWxlKHBvc3RkZikKaGVhZChwb3N0ZGZfc3RkLCBucm93PTYpCmBgYAoKYGBge3J9CmRpc3QuZXVjbCA8LSBkaXN0KHBvc3RkZl9zdGQsIG1ldGhvZCA9ICJldWNsaWRlYW4iKQpwb3N0aGMgPC0gaGNsdXN0KGQgPSBkaXN0LmV1Y2wsIG1ldGhvZCA9ICJ3YXJkLkQyIikKZnZpel9kZW5kKHBvc3RoYywgY2V4ID0gMC41KQpgYGAKCgojIEFuYWx5c2lzIGJhc2VkIG9uIHRoZSBjb2RpbmcgY2F0ZWdvcmllcyAKCmBgYHtyfQpkZl9jb2RlZCA9IHJlYWRfZXhjZWwoImNvZGluZy1jb3VudHMueGxzeCIpCmRmX2NvZGVkJFN0dWR5IDwtIGZhY3RvcihkZl9jb2RlZCRTdHVkeSkKZGZfY29kZXMgPC0gc2VsZWN0KGRmX2NvZGVkLCAtYygiU3RkY29kZSIsICJTdHVkeSIpKQpgYGAKIyMgIENvcnJlbGF0aW9ucwpUaGVyZSBhcmUgbG90cyBvZiB6ZXJvcywgc28gdGhpcyBieSBhbmQgbGFyZ2UgbWF5IG5vdCBiZSBhIGdvb2QgaWRlYS4gCgpgYGB7cn0KIyByY29ycihhcy5tYXRyaXgoZGZfY29kZXMpKQpgYGAKCgojIENsdXN0ZXIgYW5hbHlzaXMgb24gY29kZXMKCldlIGNhbiB0aGluayBvZiBhIHN0dWRlbnRzJyBzY29yZXMgaW4gdGhlIHBvc3QtdGVzdCBhcyBhIGtpbmQgb2YgcHJvZmlsZSwgYW5kIGFzayBpZiB0aGVyZSBhcmUgY2x1c3RlcnMgb2Ygc3R1ZGVudHMgd2l0aCBzaW1pbGFyIHByb2ZpbGVzLiBUaGlzIGlzIHdoYXQgYSBjbHVzdGVyIGFuYWx5c2lzIGxldHMgdXMgZmluZCBvdXQuIAoKV2UgbWF5IGhhdmUgdG8gdGhpbmsgYWJvdXQgd2hhdCB0aGUgcG9zdC10ZXN0IHZhbHVlcyBtZWFuIGFuZCBpZiBhIHN0YW5kYXJkaXNhdGlvbiBpcyByZXF1aXJlZC4gV2UgbWlnaHQgbmVlZCB0byBzdGFuZGFyZGlzZSBpZiB0aGUgbWF4aW1hbCBzY29yZXMgYXJlIGRpZmZlcmVudCBiZXR3ZWVuIGl0ZW1zLgoKIyMgTm8gc3RhbmRhcmRpc2F0aW9uCgpVc2luZyBFdWNsaWRpYW4gZGlzdGFuY2UsIHdlIGNvbXB1dGUgdGhlIGRpc3RhbmNlIGJldHdlZW4gdGhlIHN0dWRlbnRzOiAKCmBgYHtyfQpkaXN0LmV1Y2wgPC0gZGlzdChkZl9jb2RlcywgbWV0aG9kID0gImV1Y2xpZGVhbiIpCmBgYAoKVGhlIGZpcnN0IDEwIHN0dWRlbnRzJyBkaXN0YW5jZXMgYXJlOiAKCmBgYHtyfQpyb3VuZChhcy5tYXRyaXgoZGlzdC5ldWNsKVsxOjEwLCAxOjEwXSwgMSkKYGBgClRoZSBzbWFsbGVyIHRoZSB2YWx1ZSwgdGhlIG1vcmUgc2ltaWxhciB0aGUgc3R1ZGVudHMnIHNjb3JlIHByb2ZpbGUuIAoKTGV0cycgZmluZCBjbHVzdGVycyBhbmQgdmlzdWFsaXNlIHRoZW0uIAoKYGBge3J9CnBvc3RoYyA8LSBoY2x1c3QoZCA9IGRpc3QuZXVjbCwgbWV0aG9kID0gIndhcmQuRDIiKQpgYGAKCmBgYHtyfQojIGNleDogbGFiZWwgc2l6ZSAKZnZpel9kZW5kKHBvc3RoYywgY2V4ID0gMC41KQpgYGAKIyMjIFdpdGggc3RhbmRhcmRpc2F0aW9uIAoKYGBge3J9CnBvc3RkZl9zdGQgPC0gc2NhbGUoZGZfY29kZXMpCmRpc3QuZXVjbCA8LSBkaXN0KHBvc3RkZl9zdGQsIG1ldGhvZCA9ICJldWNsaWRlYW4iKQpwb3N0aGMgPC0gaGNsdXN0KGQgPSBkaXN0LmV1Y2wsIG1ldGhvZCA9ICJ3YXJkLkQyIikKZnZpel9kZW5kKHBvc3RoYywgY2V4ID0gMC41KQpgYGA=