library(tidyverse); library(gt); library(gtExtras); library(epitools)
# Reading in the nfl drives data
drives <- read.csv("https://raw.githubusercontent.com/Shammalamala/STA4504/refs/heads/main/data/ch2/nfl%20drives.csv")
Looking at the way drives can begin and end:
unique(drives$drive_start)
## [1] "Fumble" "Interception" "Kickoff" "Punt"
unique(drives$drive_end)
## [1] "Field Goal" "Punt" "Touchdown" "Turnover"
Motivating Question: Is a team more or less likely
to score a touchdown depending on how the drive began?
- Explanatory Variable: Drive Start - Punt(1), Kickoff(2),
Interception(3), Fumble(4)
- Response Variable: Drive End: Touchdown(1) or No Touchdown(2)
Now we can store our results in an \(I\times 2\) table. Not as simple as a \(2 \times 2\) table, but simpler than an
\(I \times J\) table!
drives2 <-
drives |>
mutate(drive_start = factor(drive_start,
levels = c("Punt", "Kickoff", "Interception", "Fumble")), # Reordering the groups
# And let's convert drive end to a binary variable again
drive_end = if_else(drive_end == "Touchdown",
true = "TD",
false = "noTD") |>
factor(levels = c("TD", "noTD")))
# Creating our resulting 4x2 table
drive_freq <-
xtabs(formula = ~ drive_start + drive_end,
data = drives2)
# Make the table look nice
drive_freq |>
data.frame() |>
pivot_wider(names_from = "drive_end",
values_from = "Freq") |>
rename(Touchdown = TD,
`No Touchdown` = noTD,
`Drive Start` = drive_start) |>
gt() |>
cols_align(align = "center") |>
gt_add_divider(columns = everything(),
color = "black",
sides = "all")
| Drive Start |
Touchdown |
No Touchdown |
| Punt |
537 |
1862 |
| Kickoff |
663 |
2193 |
| Interception |
114 |
253 |
| Fumble |
125 |
318 |
Let’s convert from counts to conditional proportions using
prop.table():
drive_freq |>
prop.table(margin = "drive_start") |>
signif(digits = 3) |>
data.frame() |>
pivot_wider(names_from = "drive_end",
values_from = "Freq") |>
rename(Touchdown = TD,
`No Touchdown` = noTD,
`Drive Start` = drive_start) |>
gt() |>
cols_align(align = "center") |>
gt_add_divider(columns = everything(),
color = "black",
sides = "all")
| Drive Start |
Touchdown |
No Touchdown |
| Punt |
0.224 |
0.776 |
| Kickoff |
0.232 |
0.768 |
| Interception |
0.311 |
0.689 |
| Fumble |
0.282 |
0.718 |
Always a good idea to include a graph of the data!
drive_freq |>
# Calculate the conditional proportions
prop.table(margin = "drive_start") |>
# Convert the table to a data frame to be used by ggplot()
data.frame() |>
# Rename Freq to prop
rename(prop = Freq) |>
ggplot(mapping = aes(x = drive_start,
fill = drive_end,
y = prop)) +
geom_col(width = 0.5) +
labs(y = "Proportion",
x = "How Drive Began",
fill = "Drive \nResult") +
theme_test() +
scale_y_continuous(expand = c(0, 0, 0, 0.05)) +
scale_fill_manual(labels = c("Touchdown", "No Touchdown"),
values = c("steelblue", "darkorange"))

There are some differences, but it isn’t a very noticeable
difference.
Summarizing the associations
When \(Y\) is binary, we can still
calculate risk ratios and odds ratios in a fairly straight forward
manner. The main difference is we need to decide on a baseline or
reference group to use for the explanatory variable (Say group 1). From
there, we’ll calculate the relative risk and odds ratio for each other
group in comparison to the reference group:
- Odds of a TD if the drive starts from a kickoff compared to a
punt
- Odds of a TD if the drive starts from an interception compared to a
punt
- Odds of a TD if the drive starts from a fumble compared to a
punt
While the list above mentions the odds, we can use the probability
(or risk) as well!
Since the data was collected randomly with regards to \(X\) and \(Y\), we can calculate both the relative
risk and odds ratio (instead of just the odds ratio)!
Risk Ratios
We’ll use the same function from the epitools package to
calculate the risk ratio that we did when we were analyzing two binary
variables.
The riskratio() function wants a table with the first
column being failures (noTD) and the second being successes (TD) and the
first row should be the reference group. The order of the remaining
columns is not that important.
td_rr <-
riskratio(x = drive_freq,
rev = "c",
method = "wald")
# p.values
td_rr$p.value |>
data.frame() |>
signif(digits = 3)
## midp.exact fisher.exact chi.square
## Punt NA NA NA
## Kickoff 0.476000 0.489000 0.475000
## Interception 0.000381 0.000444 0.000262
## Fumble 0.008720 0.008530 0.007630
# Estimates and confidence intervals
td_rr$measure |>
data.frame() |>
signif(digits = 3)
## estimate lower upper
## Punt 1.00 NA NA
## Kickoff 1.04 0.938 1.15
## Interception 1.39 1.170 1.64
## Fumble 1.26 1.070 1.49
Since the confidence interval for Kickoff has 1 inside, we
don’t have evidence that the probability a drive ends in a touchdown is
different for kickoffs compared to a punt at a 5% significance
level.
But we can be 95% confident that the probability a drive ends in a
touchdown is 1.170 to 1.64 times higher if the drive starts from an
interception and 1.07 to 1.46 higher for a fumble than if the drive
started with a punt.
Visualizing the confidence intervals for relative risk
We can visualize the association using relative risk with a
Cleveland plot, which has
- the values on the x-axis
- the groups on the y-axis
- the estimates represented by a point
- the confidence intervals represented by a line
td_rr$measure |>
data.frame() |>
# Changing the groups from row names to having their own column:
rownames_to_column(var = "drive_start") |>
# Let's remove the row for the reference group
filter(!is.na(lower)) |>
# use ggplot() to create our graph
ggplot(mapping = aes(x = estimate,
y = reorder(drive_start, -estimate),
color = drive_start)) +
# Adding the estimate
geom_point(size = 3) +
# Adding the line for the confidence interval with geom_segment()
geom_segment(mapping = aes(x = lower,
xend = upper,
y = drive_start,
yend = drive_start),
linewidth = 1) +
labs(x = NULL, # Removing the x-axis label
y = "Drive Beginning",
title = "Relative Risk of NFL Drives Ending in a TD \nfor how They Start Compared to a Punt") +
theme_bw() +
theme(legend.position = 'none',
plot.title = element_text(hjust = 0.5)) +
# and let's add a vertical line at 1 to indicate no difference compared to a punt
geom_vline(xintercept = 1,
linewidth = 1,
linetype = "dashed")

The graph is nice because it let’s us quickly see where significant
differences occur, and the direction of the difference:
If the entire interval is on the left of the dashed line, then
that means the probability of a success for the group is lower than the
reference group.
If the entire interval is on the right of the dashed line, then
that indicates the probability of a success for that group is higher
than the reference group.
If the interval overlaps with the dashed line, then we don’t have
any evidence that there is a difference in the probability of success
for the group compared to the reference group.
Odds Ratios
We’ll use the oddsratio() function from the
epitools package. oddsratio() wants the same
table as riskratio() does. In fact, you can just copy and
paste the code for riskratio() and just change the function
name!
td_or <-
oddsratio(x = drive_freq,
rev = "c",
method = "wald")
# p.values
td_or$p.value |>
data.frame() |>
signif(digits = 3)
## midp.exact fisher.exact chi.square
## Punt NA NA NA
## Kickoff 0.476000 0.489000 0.475000
## Interception 0.000381 0.000444 0.000262
## Fumble 0.008720 0.008530 0.007630
# Estimates and confidence intervals
td_or$measure |>
data.frame() |>
signif(digits = 3)
## estimate lower upper
## Punt 1.00 NA NA
## Kickoff 1.05 0.921 1.19
## Interception 1.56 1.230 1.99
## Fumble 1.36 1.080 1.71
Since the confidence interval for Kickoff has 1 inside, we
don’t have evidence that the odds a drive ends in a
touchdown is different for kickoffs compared to a punt at a 5%
significance level.
But we can be 95% confident that the **odds* a drive ends in a
touchdown is 1.230 to 1.99 times higher if the drive starts from an
interception and 1.08 to 1.71 times higher for a fumble than if the
drive started with a punt.
we can also create the same type of graph, but now representing the
oddsratio instead:
td_or$measure |>
data.frame() |>
# Changing the groups from row names to having their own column:
rownames_to_column(var = "drive_start") |>
# Let's remove the row for the reference group
filter(!is.na(lower)) |>
# use ggplot() to create our graph
ggplot(mapping = aes(x = estimate,
y = reorder(drive_start, -estimate),
color = drive_start)) +
# Adding the estimate
geom_point(size = 3) +
# Adding the line for the confidence interval with geom_segment()
geom_segment(mapping = aes(x = lower,
xend = upper,
y = drive_start,
yend = drive_start),
linewidth = 1) +
labs(x = NULL, # Removing the x-axis label
y = "Drive Beginning",
title = "Odds ratio of NFL Drives Ending in a TD \nfor how They Start Compared to a Punt") +
theme_bw() +
theme(legend.position = 'none',
plot.title = element_text(hjust = 0.5)) +
# and let's add a vertical line at 1 to indicate no difference compared to a punt
geom_vline(xintercept = 1,
linewidth = 1,
linetype = "dashed") +
# You can optionally add the end points to the graph using 2 instances of geom_text()
geom_text(mapping = aes(x = lower,
label = round(lower, digits = 2)),
vjust = -0.5) +
geom_text(mapping = aes(x = upper,
label = round(upper, digits = 2)),
vjust = -0.5)

LS0tDQp0aXRsZTogIlR3byBDYXRlZ29yaWNhbCBWYXJpYWJsZXMgLSBCaW5hcnkgUmVzcG9uc2UgYW5kIE5vbmJpbmFyeSBFeHBsYW5hdG9yeSINCmF1dGhvcjogIkNoYXB0ZXIgMiINCmRhdGU6ICJTVEEgNDUwNCINCm91dHB1dDoNCiAgaHRtbF9kb2N1bWVudDoNCiAgICBmaWdfd2lkdGg6IDEwDQogICAgZmlnX2hlaWdodDogNg0KICAgIGZpZ19jYXB0aW9uOiB0cnVlDQogICAgdG9jOiB0cnVlDQogICAgdG9jX2Zsb2F0OiB0cnVlDQogICAgbnVtYmVyX3NlY3Rpb25zOiBmYWxzZQ0KICAgIGNvZGVfZm9sZGluZzogaGlkZQ0KICAgIGNvZGVfZG93bmxvYWQ6IHRydWUNCiAgICBzbW9vdGhfc2Nyb2xsOiB0cnVlDQogICAgdGhlbWU6IGx1bWVuDQotLS0NCg0KYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9DQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUsDQogICAgICAgICAgICAgICAgICAgICAgZmlnLmFsaWduID0gImNlbnRlciIpDQpgYGANCg0KDQpgYGB7ciBwYWNrYWdlc19kYXRhLCBtZXNzYWdlID0gRiwgd2FybmluZyA9IEZ9DQpsaWJyYXJ5KHRpZHl2ZXJzZSk7IGxpYnJhcnkoZ3QpOyBsaWJyYXJ5KGd0RXh0cmFzKTsgbGlicmFyeShlcGl0b29scykNCg0KIyBSZWFkaW5nIGluIHRoZSBuZmwgZHJpdmVzIGRhdGENCmRyaXZlcyA8LSByZWFkLmNzdigiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL1NoYW1tYWxhbWFsYS9TVEE0NTA0L3JlZnMvaGVhZHMvbWFpbi9kYXRhL2NoMi9uZmwlMjBkcml2ZXMuY3N2IikgDQpgYGANCg0KDQpMb29raW5nIGF0IHRoZSB3YXkgZHJpdmVzIGNhbiBiZWdpbiBhbmQgZW5kOg0KDQoNCmBgYHtyIGRyaXZlX3N0YXJ0X2VuZH0NCnVuaXF1ZShkcml2ZXMkZHJpdmVfc3RhcnQpDQp1bmlxdWUoZHJpdmVzJGRyaXZlX2VuZCkNCg0KYGBgDQoNCioqTW90aXZhdGluZyBRdWVzdGlvbjoqKiBJcyBhIHRlYW0gbW9yZSBvciBsZXNzIGxpa2VseSB0byBzY29yZSBhIHRvdWNoZG93biBkZXBlbmRpbmcgb24gaG93IHRoZSBkcml2ZSBiZWdhbj8NCg0KLSBFeHBsYW5hdG9yeSBWYXJpYWJsZTogRHJpdmUgU3RhcnQgLSBQdW50KDEpLCBLaWNrb2ZmKDIpLCBJbnRlcmNlcHRpb24oMyksIEZ1bWJsZSg0KQ0KLSBSZXNwb25zZSBWYXJpYWJsZTogRHJpdmUgRW5kOiBUb3VjaGRvd24oMSkgb3IgTm8gVG91Y2hkb3duKDIpDQoNCg0KTm93IHdlIGNhbiBzdG9yZSBvdXIgcmVzdWx0cyBpbiBhbiAkSVx0aW1lcyAyJCB0YWJsZS4gTm90IGFzIHNpbXBsZSBhcyBhICQyIFx0aW1lcyAyJCB0YWJsZSwgYnV0IHNpbXBsZXIgdGhhbiBhbiAkSSBcdGltZXMgSiQgdGFibGUhDQoNCmBgYHtyIGRyaXZlczJ9DQpkcml2ZXMyIDwtIA0KICBkcml2ZXMgfD4gDQogIG11dGF0ZShkcml2ZV9zdGFydCA9IGZhY3Rvcihkcml2ZV9zdGFydCwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxldmVscyA9IGMoIlB1bnQiLCAiS2lja29mZiIsICJJbnRlcmNlcHRpb24iLCAiRnVtYmxlIikpLCAjIFJlb3JkZXJpbmcgdGhlIGdyb3Vwcw0KICAgICAgICAgDQogICAgICAgICAjIEFuZCBsZXQncyBjb252ZXJ0IGRyaXZlIGVuZCB0byBhIGJpbmFyeSB2YXJpYWJsZSBhZ2Fpbg0KICAgICAgICAgZHJpdmVfZW5kID0gaWZfZWxzZShkcml2ZV9lbmQgPT0gIlRvdWNoZG93biIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRydWUgPSAiVEQiLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmYWxzZSA9ICJub1REIikgfD4gDQogICAgICAgICAgICAgICAgICAgICBmYWN0b3IobGV2ZWxzID0gYygiVEQiLCAibm9URCIpKSkNCg0KIyBDcmVhdGluZyBvdXIgcmVzdWx0aW5nIDR4MiB0YWJsZQ0KZHJpdmVfZnJlcSA8LSANCiAgeHRhYnMoZm9ybXVsYSA9IH4gZHJpdmVfc3RhcnQgKyBkcml2ZV9lbmQsDQogICAgICAgIGRhdGEgPSBkcml2ZXMyKQ0KDQojIE1ha2UgdGhlIHRhYmxlIGxvb2sgbmljZQ0KZHJpdmVfZnJlcSB8PiANCiAgZGF0YS5mcmFtZSgpIHw+IA0KICBwaXZvdF93aWRlcihuYW1lc19mcm9tID0gImRyaXZlX2VuZCIsDQogICAgICAgICAgICAgIHZhbHVlc19mcm9tID0gIkZyZXEiKSB8PiANCiAgcmVuYW1lKFRvdWNoZG93biA9IFRELA0KICAgICAgICAgYE5vIFRvdWNoZG93bmAgPSBub1RELA0KICAgICAgICAgYERyaXZlIFN0YXJ0YCA9IGRyaXZlX3N0YXJ0KSB8PiANCiAgZ3QoKSB8PiANCiAgY29sc19hbGlnbihhbGlnbiA9ICJjZW50ZXIiKSB8PiANCiAgZ3RfYWRkX2RpdmlkZXIoY29sdW1ucyA9IGV2ZXJ5dGhpbmcoKSwNCiAgICAgICAgICAgICAgICAgY29sb3IgPSAiYmxhY2siLA0KICAgICAgICAgICAgICAgICBzaWRlcyA9ICJhbGwiKQ0KYGBgDQoNCg0KDQpMZXQncyBjb252ZXJ0IGZyb20gY291bnRzIHRvIGNvbmRpdGlvbmFsIHByb3BvcnRpb25zIHVzaW5nIGBwcm9wLnRhYmxlKClgOg0KDQpgYGB7cn0NCmRyaXZlX2ZyZXEgfD4gDQogIHByb3AudGFibGUobWFyZ2luID0gImRyaXZlX3N0YXJ0IikgfD4gDQogIHNpZ25pZihkaWdpdHMgPSAzKSB8PiANCiAgZGF0YS5mcmFtZSgpIHw+IA0KICBwaXZvdF93aWRlcihuYW1lc19mcm9tID0gImRyaXZlX2VuZCIsDQogICAgICAgICAgICAgIHZhbHVlc19mcm9tID0gIkZyZXEiKSB8PiANCiAgcmVuYW1lKFRvdWNoZG93biA9IFRELA0KICAgICAgICAgYE5vIFRvdWNoZG93bmAgPSBub1RELA0KICAgICAgICAgYERyaXZlIFN0YXJ0YCA9IGRyaXZlX3N0YXJ0KSB8PiANCiAgZ3QoKSB8PiANCiAgY29sc19hbGlnbihhbGlnbiA9ICJjZW50ZXIiKSB8PiANCiAgZ3RfYWRkX2RpdmlkZXIoY29sdW1ucyA9IGV2ZXJ5dGhpbmcoKSwNCiAgICAgICAgICAgICAgICAgY29sb3IgPSAiYmxhY2siLA0KICAgICAgICAgICAgICAgICBzaWRlcyA9ICJhbGwiKQ0KYGBgDQoNCg0KDQoNCkFsd2F5cyBhIGdvb2QgaWRlYSB0byBpbmNsdWRlIGEgZ3JhcGggb2YgdGhlIGRhdGEhDQoNCmBgYHtyIGJhcl9jaGFydH0NCmRyaXZlX2ZyZXEgfD4gDQogICMgQ2FsY3VsYXRlIHRoZSBjb25kaXRpb25hbCBwcm9wb3J0aW9ucw0KICBwcm9wLnRhYmxlKG1hcmdpbiA9ICJkcml2ZV9zdGFydCIpIHw+IA0KICAjIENvbnZlcnQgdGhlIHRhYmxlIHRvIGEgZGF0YSBmcmFtZSB0byBiZSB1c2VkIGJ5IGdncGxvdCgpDQogIGRhdGEuZnJhbWUoKSB8PiANCiAgIyBSZW5hbWUgRnJlcSB0byBwcm9wDQogIHJlbmFtZShwcm9wID0gRnJlcSkgfD4gDQogIA0KICBnZ3Bsb3QobWFwcGluZyA9IGFlcyh4ID0gZHJpdmVfc3RhcnQsDQogICAgICAgICAgICAgICAgICAgICAgIGZpbGwgPSBkcml2ZV9lbmQsDQogICAgICAgICAgICAgICAgICAgICAgIHkgPSBwcm9wKSkgKyANCiAgDQogIGdlb21fY29sKHdpZHRoID0gMC41KSArIA0KICANCiAgbGFicyh5ID0gIlByb3BvcnRpb24iLA0KICAgICAgIHggPSAiSG93IERyaXZlIEJlZ2FuIiwNCiAgICAgICBmaWxsID0gIkRyaXZlIFxuUmVzdWx0IikgKyANCiAgDQogIHRoZW1lX3Rlc3QoKSArIA0KICANCiAgc2NhbGVfeV9jb250aW51b3VzKGV4cGFuZCA9IGMoMCwgMCwgMCwgMC4wNSkpICsgDQogIA0KICBzY2FsZV9maWxsX21hbnVhbChsYWJlbHMgPSBjKCJUb3VjaGRvd24iLCAiTm8gVG91Y2hkb3duIiksDQogICAgICAgICAgICAgICAgICAgIHZhbHVlcyA9IGMoInN0ZWVsYmx1ZSIsICJkYXJrb3JhbmdlIikpDQoNCg0KYGBgDQoNClRoZXJlIGFyZSBzb21lIGRpZmZlcmVuY2VzLCBidXQgaXQgaXNuJ3QgYSB2ZXJ5IG5vdGljZWFibGUgZGlmZmVyZW5jZS4gDQoNCg0KIyMgU3VtbWFyaXppbmcgdGhlIGFzc29jaWF0aW9ucw0KDQpXaGVuICRZJCBpcyBiaW5hcnksIHdlIGNhbiBzdGlsbCBjYWxjdWxhdGUgcmlzayByYXRpb3MgYW5kIG9kZHMgcmF0aW9zIGluIGEgZmFpcmx5IHN0cmFpZ2h0IGZvcndhcmQgbWFubmVyLiBUaGUgbWFpbiBkaWZmZXJlbmNlIGlzIHdlIG5lZWQgdG8gZGVjaWRlIG9uIGEgYmFzZWxpbmUgb3IgcmVmZXJlbmNlIGdyb3VwIHRvIHVzZSBmb3IgdGhlIGV4cGxhbmF0b3J5IHZhcmlhYmxlIChTYXkgZ3JvdXAgMSkuIEZyb20gdGhlcmUsIHdlJ2xsIGNhbGN1bGF0ZSB0aGUgcmVsYXRpdmUgcmlzayBhbmQgb2RkcyByYXRpbyBmb3IgZWFjaCBvdGhlciBncm91cCBpbiBjb21wYXJpc29uIHRvIHRoZSByZWZlcmVuY2UgZ3JvdXA6DQoNCjEpIE9kZHMgb2YgYSBURCBpZiB0aGUgZHJpdmUgc3RhcnRzIGZyb20gYSBraWNrb2ZmIGNvbXBhcmVkIHRvIGEgcHVudA0KMikgT2RkcyBvZiBhIFREIGlmIHRoZSBkcml2ZSBzdGFydHMgZnJvbSBhbiBpbnRlcmNlcHRpb24gY29tcGFyZWQgdG8gYSBwdW50DQozKSBPZGRzIG9mIGEgVEQgaWYgdGhlIGRyaXZlIHN0YXJ0cyBmcm9tIGEgZnVtYmxlIGNvbXBhcmVkIHRvIGEgcHVudA0KDQpXaGlsZSB0aGUgbGlzdCBhYm92ZSBtZW50aW9ucyB0aGUgb2Rkcywgd2UgY2FuIHVzZSB0aGUgcHJvYmFiaWxpdHkgKG9yIHJpc2spIGFzIHdlbGwhDQoNCg0KU2luY2UgdGhlIGRhdGEgd2FzIGNvbGxlY3RlZCByYW5kb21seSB3aXRoIHJlZ2FyZHMgdG8gJFgkIGFuZCAkWSQsIHdlIGNhbiBjYWxjdWxhdGUgYm90aCB0aGUgcmVsYXRpdmUgcmlzayBhbmQgb2RkcyByYXRpbyAoaW5zdGVhZCBvZiBqdXN0IHRoZSBvZGRzIHJhdGlvKSENCg0KDQoNCiMjIFJpc2sgUmF0aW9zDQoNCldlJ2xsIHVzZSB0aGUgc2FtZSBmdW5jdGlvbiBmcm9tIHRoZSBgZXBpdG9vbHNgIHBhY2thZ2UgdG8gY2FsY3VsYXRlIHRoZSByaXNrIHJhdGlvIHRoYXQgd2UgZGlkIHdoZW4gd2Ugd2VyZSBhbmFseXppbmcgdHdvIGJpbmFyeSB2YXJpYWJsZXMuIA0KDQpUaGUgYHJpc2tyYXRpbygpYCBmdW5jdGlvbiB3YW50cyBhIHRhYmxlIHdpdGggdGhlIGZpcnN0IGNvbHVtbiBiZWluZyBmYWlsdXJlcyAobm9URCkgYW5kIHRoZSBzZWNvbmQgYmVpbmcgc3VjY2Vzc2VzIChURCkgYW5kIHRoZSBmaXJzdCByb3cgc2hvdWxkIGJlIHRoZSByZWZlcmVuY2UgZ3JvdXAuIFRoZSBvcmRlciBvZiB0aGUgcmVtYWluaW5nIGNvbHVtbnMgaXMgbm90IHRoYXQgaW1wb3J0YW50Lg0KDQoNCmBgYHtyIHJpc2tyYXRpb30NCnRkX3JyIDwtIA0KICByaXNrcmF0aW8oeCA9IGRyaXZlX2ZyZXEsDQogICAgICAgICAgICByZXYgPSAiYyIsDQogICAgICAgICAgICBtZXRob2QgPSAid2FsZCIpDQoNCiMgcC52YWx1ZXMNCnRkX3JyJHAudmFsdWUgfD4gDQogIGRhdGEuZnJhbWUoKSB8PiANCiAgc2lnbmlmKGRpZ2l0cyA9IDMpDQoNCiMgRXN0aW1hdGVzIGFuZCBjb25maWRlbmNlIGludGVydmFscw0KdGRfcnIkbWVhc3VyZSB8PiANCiAgZGF0YS5mcmFtZSgpIHw+IA0KICBzaWduaWYoZGlnaXRzID0gMykNCg0KYGBgDQoNClNpbmNlIHRoZSBjb25maWRlbmNlIGludGVydmFsIGZvciAqS2lja29mZiogaGFzIDEgaW5zaWRlLCB3ZSBkb24ndCBoYXZlIGV2aWRlbmNlIHRoYXQgdGhlIHByb2JhYmlsaXR5IGEgZHJpdmUgZW5kcyBpbiBhIHRvdWNoZG93biBpcyBkaWZmZXJlbnQgZm9yIGtpY2tvZmZzIGNvbXBhcmVkIHRvIGEgcHVudCBhdCBhIDUlIHNpZ25pZmljYW5jZSBsZXZlbC4NCg0KQnV0IHdlIGNhbiBiZSA5NSUgY29uZmlkZW50IHRoYXQgdGhlIHByb2JhYmlsaXR5IGEgZHJpdmUgZW5kcyBpbiBhIHRvdWNoZG93biBpcyAxLjE3MCB0byAxLjY0IHRpbWVzIGhpZ2hlciBpZiB0aGUgZHJpdmUgc3RhcnRzIGZyb20gYW4gaW50ZXJjZXB0aW9uIGFuZCAxLjA3IHRvIDEuNDYgaGlnaGVyIGZvciBhIGZ1bWJsZSB0aGFuIGlmIHRoZSBkcml2ZSBzdGFydGVkIHdpdGggYSBwdW50Lg0KDQoNCiMjIyMgVmlzdWFsaXppbmcgdGhlIGNvbmZpZGVuY2UgaW50ZXJ2YWxzIGZvciByZWxhdGl2ZSByaXNrDQoNCldlIGNhbiB2aXN1YWxpemUgdGhlIGFzc29jaWF0aW9uIHVzaW5nIHJlbGF0aXZlIHJpc2sgd2l0aCBhICoqQ2xldmVsYW5kIHBsb3QqKiwgd2hpY2ggaGFzIA0KDQotIHRoZSB2YWx1ZXMgb24gdGhlIHgtYXhpcw0KLSB0aGUgZ3JvdXBzIG9uIHRoZSB5LWF4aXMNCi0gdGhlIGVzdGltYXRlcyByZXByZXNlbnRlZCBieSBhIHBvaW50DQotIHRoZSBjb25maWRlbmNlIGludGVydmFscyByZXByZXNlbnRlZCBieSBhIGxpbmUNCg0KDQoNCmBgYHtyIGNsZXZlbGFuZH0NCnRkX3JyJG1lYXN1cmUgfD4gDQogIGRhdGEuZnJhbWUoKSB8Pg0KICANCiAgIyBDaGFuZ2luZyB0aGUgZ3JvdXBzIGZyb20gcm93IG5hbWVzIHRvIGhhdmluZyB0aGVpciBvd24gY29sdW1uOg0KICByb3duYW1lc190b19jb2x1bW4odmFyID0gImRyaXZlX3N0YXJ0IikgfD4gDQogIA0KICAjIExldCdzIHJlbW92ZSB0aGUgcm93IGZvciB0aGUgcmVmZXJlbmNlIGdyb3VwDQogIGZpbHRlcighaXMubmEobG93ZXIpKSB8PiANCiAgDQogICMgdXNlIGdncGxvdCgpIHRvIGNyZWF0ZSBvdXIgZ3JhcGgNCiAgZ2dwbG90KG1hcHBpbmcgPSBhZXMoeCA9IGVzdGltYXRlLA0KICAgICAgICAgICAgICAgICAgICAgICB5ID0gcmVvcmRlcihkcml2ZV9zdGFydCwgLWVzdGltYXRlKSwNCiAgICAgICAgICAgICAgICAgICAgICAgY29sb3IgPSBkcml2ZV9zdGFydCkpICsgDQogIA0KICAjIEFkZGluZyB0aGUgZXN0aW1hdGUNCiAgZ2VvbV9wb2ludChzaXplID0gMykgKw0KICANCiAgIyBBZGRpbmcgdGhlIGxpbmUgZm9yIHRoZSBjb25maWRlbmNlIGludGVydmFsIHdpdGggZ2VvbV9zZWdtZW50KCkNCiAgZ2VvbV9zZWdtZW50KG1hcHBpbmcgPSBhZXMoeCA9IGxvd2VyLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICB4ZW5kID0gdXBwZXIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHkgPSBkcml2ZV9zdGFydCwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeWVuZCA9IGRyaXZlX3N0YXJ0KSwNCiAgICAgICAgICAgICAgIGxpbmV3aWR0aCA9IDEpICsgDQogIA0KICBsYWJzKHggPSBOVUxMLCAgIyBSZW1vdmluZyB0aGUgeC1heGlzIGxhYmVsDQogICAgICAgeSA9ICJEcml2ZSBCZWdpbm5pbmciLA0KICAgICAgIHRpdGxlID0gIlJlbGF0aXZlIFJpc2sgb2YgTkZMIERyaXZlcyBFbmRpbmcgaW4gYSBURCBcbmZvciBob3cgVGhleSBTdGFydCBDb21wYXJlZCB0byBhIFB1bnQiKSArDQogIA0KICB0aGVtZV9idygpICsNCiAgDQogIHRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICdub25lJywNCiAgICAgICAgcGxvdC50aXRsZSA9IGVsZW1lbnRfdGV4dChoanVzdCA9IDAuNSkpICArDQogIA0KICAjIGFuZCBsZXQncyBhZGQgYSB2ZXJ0aWNhbCBsaW5lIGF0IDEgdG8gaW5kaWNhdGUgbm8gZGlmZmVyZW5jZSBjb21wYXJlZCB0byBhIHB1bnQNCiAgZ2VvbV92bGluZSh4aW50ZXJjZXB0ID0gMSwNCiAgICAgICAgICAgICBsaW5ld2lkdGggPSAxLA0KICAgICAgICAgICAgIGxpbmV0eXBlID0gImRhc2hlZCIpIA0KICANCmBgYA0KDQpUaGUgZ3JhcGggaXMgbmljZSBiZWNhdXNlIGl0IGxldCdzIHVzIHF1aWNrbHkgc2VlIHdoZXJlIHNpZ25pZmljYW50IGRpZmZlcmVuY2VzIG9jY3VyLCBhbmQgdGhlIGRpcmVjdGlvbiBvZiB0aGUgZGlmZmVyZW5jZTogDQoNCi0gSWYgdGhlIGVudGlyZSBpbnRlcnZhbCBpcyBvbiB0aGUgbGVmdCBvZiB0aGUgZGFzaGVkIGxpbmUsIHRoZW4gdGhhdCBtZWFucyB0aGUgcHJvYmFiaWxpdHkgb2YgYSBzdWNjZXNzIGZvciB0aGUgZ3JvdXAgaXMgbG93ZXIgdGhhbiB0aGUgcmVmZXJlbmNlIGdyb3VwLg0KDQotIElmIHRoZSBlbnRpcmUgaW50ZXJ2YWwgaXMgb24gdGhlIHJpZ2h0IG9mIHRoZSBkYXNoZWQgbGluZSwgdGhlbiB0aGF0IGluZGljYXRlcyB0aGUgcHJvYmFiaWxpdHkgb2YgYSBzdWNjZXNzIGZvciB0aGF0IGdyb3VwIGlzIGhpZ2hlciB0aGFuIHRoZSByZWZlcmVuY2UgZ3JvdXAuDQoNCi0gSWYgdGhlIGludGVydmFsIG92ZXJsYXBzIHdpdGggdGhlIGRhc2hlZCBsaW5lLCB0aGVuIHdlIGRvbid0IGhhdmUgYW55IGV2aWRlbmNlIHRoYXQgdGhlcmUgaXMgYSBkaWZmZXJlbmNlIGluIHRoZSBwcm9iYWJpbGl0eSBvZiBzdWNjZXNzIGZvciB0aGUgZ3JvdXAgY29tcGFyZWQgdG8gdGhlIHJlZmVyZW5jZSBncm91cC4NCg0KDQoNCg0KDQoNCiMjIE9kZHMgUmF0aW9zDQoNCldlJ2xsIHVzZSB0aGUgYG9kZHNyYXRpbygpYCBmdW5jdGlvbiBmcm9tIHRoZSBgZXBpdG9vbHNgIHBhY2thZ2UuIGBvZGRzcmF0aW8oKWAgd2FudHMgdGhlIHNhbWUgdGFibGUgYXMgYHJpc2tyYXRpbygpYCBkb2VzLiBJbiBmYWN0LCB5b3UgY2FuIGp1c3QgY29weSBhbmQgcGFzdGUgdGhlIGNvZGUgZm9yIGByaXNrcmF0aW8oKWAgYW5kIGp1c3QgY2hhbmdlIHRoZSBmdW5jdGlvbiBuYW1lIQ0KDQoNCmBgYHtyIG9kZHNyYXRpb30NCnRkX29yIDwtIA0KICBvZGRzcmF0aW8oeCA9IGRyaXZlX2ZyZXEsDQogICAgICAgICAgICByZXYgPSAiYyIsDQogICAgICAgICAgICBtZXRob2QgPSAid2FsZCIpDQoNCiMgcC52YWx1ZXMNCnRkX29yJHAudmFsdWUgfD4gDQogIGRhdGEuZnJhbWUoKSB8PiANCiAgc2lnbmlmKGRpZ2l0cyA9IDMpDQoNCiMgRXN0aW1hdGVzIGFuZCBjb25maWRlbmNlIGludGVydmFscw0KdGRfb3IkbWVhc3VyZSB8PiANCiAgZGF0YS5mcmFtZSgpIHw+IA0KICBzaWduaWYoZGlnaXRzID0gMykNCg0KYGBgDQoNClNpbmNlIHRoZSBjb25maWRlbmNlIGludGVydmFsIGZvciAqS2lja29mZiogaGFzIDEgaW5zaWRlLCB3ZSBkb24ndCBoYXZlIGV2aWRlbmNlIHRoYXQgdGhlICoqb2RkcyoqIGEgZHJpdmUgZW5kcyBpbiBhIHRvdWNoZG93biBpcyBkaWZmZXJlbnQgZm9yIGtpY2tvZmZzIGNvbXBhcmVkIHRvIGEgcHVudCBhdCBhIDUlIHNpZ25pZmljYW5jZSBsZXZlbC4NCg0KQnV0IHdlIGNhbiBiZSA5NSUgY29uZmlkZW50IHRoYXQgdGhlICoqb2RkcyogYSBkcml2ZSBlbmRzIGluIGEgdG91Y2hkb3duIGlzIDEuMjMwIHRvIDEuOTkgdGltZXMgaGlnaGVyIGlmIHRoZSBkcml2ZSBzdGFydHMgZnJvbSBhbiBpbnRlcmNlcHRpb24gYW5kIDEuMDggdG8gMS43MSB0aW1lcyBoaWdoZXIgZm9yIGEgZnVtYmxlIHRoYW4gaWYgdGhlIGRyaXZlIHN0YXJ0ZWQgd2l0aCBhIHB1bnQuDQoNCg0KDQp3ZSBjYW4gYWxzbyBjcmVhdGUgdGhlIHNhbWUgdHlwZSBvZiBncmFwaCwgYnV0IG5vdyByZXByZXNlbnRpbmcgdGhlIG9kZHNyYXRpbyBpbnN0ZWFkOg0KDQoNCmBgYHtyIE9SX2dyYXBofQ0KdGRfb3IkbWVhc3VyZSB8PiANCiAgZGF0YS5mcmFtZSgpIHw+DQogIA0KICAjIENoYW5naW5nIHRoZSBncm91cHMgZnJvbSByb3cgbmFtZXMgdG8gaGF2aW5nIHRoZWlyIG93biBjb2x1bW46DQogIHJvd25hbWVzX3RvX2NvbHVtbih2YXIgPSAiZHJpdmVfc3RhcnQiKSB8PiANCiAgDQogICMgTGV0J3MgcmVtb3ZlIHRoZSByb3cgZm9yIHRoZSByZWZlcmVuY2UgZ3JvdXANCiAgZmlsdGVyKCFpcy5uYShsb3dlcikpIHw+IA0KICANCiAgIyB1c2UgZ2dwbG90KCkgdG8gY3JlYXRlIG91ciBncmFwaA0KICBnZ3Bsb3QobWFwcGluZyA9IGFlcyh4ID0gZXN0aW1hdGUsDQogICAgICAgICAgICAgICAgICAgICAgIHkgPSByZW9yZGVyKGRyaXZlX3N0YXJ0LCAtZXN0aW1hdGUpLA0KICAgICAgICAgICAgICAgICAgICAgICBjb2xvciA9IGRyaXZlX3N0YXJ0KSkgKyANCiAgDQogICMgQWRkaW5nIHRoZSBlc3RpbWF0ZQ0KICBnZW9tX3BvaW50KHNpemUgPSAzKSArDQogIA0KICAjIEFkZGluZyB0aGUgbGluZSBmb3IgdGhlIGNvbmZpZGVuY2UgaW50ZXJ2YWwgd2l0aCBnZW9tX3NlZ21lbnQoKQ0KICBnZW9tX3NlZ21lbnQobWFwcGluZyA9IGFlcyh4ID0gbG93ZXIsDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHhlbmQgPSB1cHBlciwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeSA9IGRyaXZlX3N0YXJ0LA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICB5ZW5kID0gZHJpdmVfc3RhcnQpLA0KICAgICAgICAgICAgICAgbGluZXdpZHRoID0gMSkgKyANCiAgDQogIGxhYnMoeCA9IE5VTEwsICAjIFJlbW92aW5nIHRoZSB4LWF4aXMgbGFiZWwNCiAgICAgICB5ID0gIkRyaXZlIEJlZ2lubmluZyIsDQogICAgICAgdGl0bGUgPSAiT2RkcyByYXRpbyBvZiBORkwgRHJpdmVzIEVuZGluZyBpbiBhIFREIFxuZm9yIGhvdyBUaGV5IFN0YXJ0IENvbXBhcmVkIHRvIGEgUHVudCIpICsNCiAgDQogIHRoZW1lX2J3KCkgKw0KICANCiAgdGhlbWUobGVnZW5kLnBvc2l0aW9uID0gJ25vbmUnLA0KICAgICAgICBwbG90LnRpdGxlID0gZWxlbWVudF90ZXh0KGhqdXN0ID0gMC41KSkgICsNCiAgDQogICMgYW5kIGxldCdzIGFkZCBhIHZlcnRpY2FsIGxpbmUgYXQgMSB0byBpbmRpY2F0ZSBubyBkaWZmZXJlbmNlIGNvbXBhcmVkIHRvIGEgcHVudA0KICBnZW9tX3ZsaW5lKHhpbnRlcmNlcHQgPSAxLA0KICAgICAgICAgICAgIGxpbmV3aWR0aCA9IDEsDQogICAgICAgICAgICAgbGluZXR5cGUgPSAiZGFzaGVkIikgICsgDQogIA0KICAjIFlvdSBjYW4gb3B0aW9uYWxseSBhZGQgdGhlIGVuZCBwb2ludHMgdG8gdGhlIGdyYXBoIHVzaW5nIDIgaW5zdGFuY2VzIG9mIGdlb21fdGV4dCgpDQogIGdlb21fdGV4dChtYXBwaW5nID0gYWVzKHggPSBsb3dlciwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgbGFiZWwgPSByb3VuZChsb3dlciwgZGlnaXRzID0gMikpLA0KICAgICAgICAgICAgdmp1c3QgPSAtMC41KSArIA0KICANCiAgZ2VvbV90ZXh0KG1hcHBpbmcgPSBhZXMoeCA9IHVwcGVyLA0KICAgICAgICAgICAgICAgICAgICAgICAgICBsYWJlbCA9IHJvdW5kKHVwcGVyLCBkaWdpdHMgPSAyKSksDQogICAgICAgICAgICB2anVzdCA9IC0wLjUpDQogIA0KYGBgDQoNCg0KDQoNCg==