O objeto principal da análise são as buscas e a navegação depois da busca. Criamos esses dados a partir dos dados originais da wikimedia em /data/search_data.csv.

Aqui, exploramos esses dados.

library(tidyverse)
── Attaching packages ──────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
✔ ggplot2 3.1.1       ✔ purrr   0.3.2  
✔ tibble  2.1.1       ✔ dplyr   0.8.0.1
✔ tidyr   0.8.3       ✔ stringr 1.4.0  
✔ readr   1.3.1       ✔ forcats 0.4.0  
── Conflicts ─────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
library(here)
here() starts at /Users/marcoswilliamscavalcantidearruda/Workspace/Mestrado/fpcc2/l1p2-marcoswca
library(lubridate)

Attaching package: ‘lubridate’

The following object is masked from ‘package:here’:

    here

The following object is masked from ‘package:base’:

    date
theme_set(theme_bw())
options(warn=-1) # ignore warns
buscas = read_csv(here::here("data/search_data.csv"))
Parsed with column specification:
cols(
  session_id = col_character(),
  search_index = col_double(),
  session_length = col_double(),
  session_start_timestamp = col_double(),
  session_start_date = col_datetime(format = ""),
  group = col_character(),
  results = col_double(),
  num_clicks = col_double(),
  first_click = col_double()
)

Explorando os dados

Aqui observamos a distribuição da variável número de clicks, podemos observar que sua maior concentração está entre 0 e 2.Também observamos outliers que foram removidos.

buscas %>% 
    filter(num_clicks <= 7) %>%
    ggplot(aes(x = num_clicks)) + 
    geom_density(fill='black') +
    labs(title = "density num_clicks")

A distribuição possui uma longa calda a direita, para facilitar a interpretação os outliers foram removidos.

buscas %>% 
  filter(session_length <= 5000) %>%
  ggplot(aes(x=session_length)) +
  geom_density(fill='blue') +
  labs(title = "density session_length")

Observando a relação entre session_length x num_clicks não aperenta ter correlação de qualquer natureza. Analisando os coeficientes de correlação percebemos que nenhum possui valor significativo, ou seja, realmente não existe correlação entre as variáveis.

buscas %>% 
  filter(session_length <= 100000 & num_clicks <= 25) %>%
  ggplot(aes(x = session_length, y = num_clicks)) +
  geom_point() +
  labs(title = "session_length x num_clicks")

buscas %>% 
    filter(session_length <= 100000 & num_clicks <= 25) %>%
    summarise(
      pearson = cor(session_length, num_clicks, method = "pearson"), 
      spearman = cor(session_length, num_clicks, method = "spearman"), 
      kendall = cor(session_length, num_clicks, method = "kendall")
    )

As distribuições possuem uma longa calda a direita, para facilitar a interpretação os outliers foram removidos.

buscas %>% 
  filter(results <= 25) %>%
  ggplot(aes(x=results)) +
  geom_density(alpha = .4, fill='green') +
  labs(title = "density results")

buscas %>% 
  filter(search_index <= 30) %>%
  ggplot(aes(x=search_index)) +
  geom_density(fill='blue') +
  labs(title = "density search_index")

Observando a relação entre results e search_index, novamente, não é possível observar correlação de qualquer natureza e estimando os coeficientes de correlação não encontramos valores significativos.

buscas %>% 
  filter(results <= 25 & search_index <= 30) %>%
  ggplot(aes(x=results, y=search_index)) +
  geom_point() +
  labs(title = "results x search_index")
buscas %>% 
    filter(results <= 25 & search_index <= 30) %>%
    summarise(
      pearson = cor(results, search_index, method = "pearson"), 
      spearman = cor(results, search_index, method = "spearman"), 
      kendall = cor(results, search_index, method = "kendall")
    )

1. What is our daily overall clickthrough rate? How does it vary between the groups?

O gráfico exibe a relação entre data e número de clicks, é possível perceber que os dias 1,2 e 3 de março possuem mais clicks. O dia 5 de março é o dia com menos clicks.

buscas$session_start_date <- buscas$session_start_date %>% as.Date("%d/%m/%Y")
unknown timezone '%d/%m/%Y'
buscas %>%
  group_by(session_start_date) %>%
  mutate(clicks_per_day = sum(num_clicks)) %>%
  ggplot(aes(x=session_start_date, y=clicks_per_day)) +
  geom_point() +
  geom_line() +
  labs(title = "clicks_per_day")

Considerando a mesma relação, mas dessa vez considerando os grupos, observamos que o grupo A possui um maior número de clicks do que o grupo B em todo o período observado. O grupo B possui maior regularidade na quantidade de clicks do que o grupo A, isso pode ser obersavado comparando a variância dos dois grupos.

buscas %>%
  group_by(group, session_start_date) %>%
  mutate(clicks_per_day = sum(num_clicks)) %>%
  ggplot(mapping = aes(x=session_start_date, y=clicks_per_day, color = group)) +
  geom_point() +
  geom_line() +
  facet_grid(group ~ .) +
  labs(title = "clicks_per_day")

buscas %>% 
    group_by(group) %>%
    summarise(variancia = var(session_length, results))

2. Which results do people tend to try first? How does it change day-to-day?

Analisando a distribuição observamos que uma maior concentração no valor 1 e depois vai decrescendo, ou seja, as pessoas clicam mais nos primeiros links de retorno de uma pesquisa.

buscas %>% 
  filter(!is.na(first_click)) %>% 
  group_by(first_click) %>% 
  filter(n() >= 20) %>%
  ggplot(aes(x=first_click)) +
  geom_density(fill = "blue", color= "green", alpha = .4)

Analisando a distribuição dos primeiros clicks, mas dessa vez agrupando por dia, observamos que todos os dias tem uma distribuição muito parecida, com concentração em torno de 1. Então, ao calcular o percentual de clicks diferentes de 1 percebemos que os grupos se diferenciam um pouco, 2016-03-05 foi o dia em que mais se clicou em links que não fossem o primeiro, com quase 2% a mais do que o dia 2016-03-01 em que menos se clicou.

buscas %>% 
  filter(!is.na(first_click)) %>% 
  group_by(first_click, session_start_date) %>% 
  summarise(num_clicks = n()) %>%
  filter(num_clicks >= 20) %>%
  ggplot(aes(x=first_click, y = num_clicks)) +
  facet_wrap(~session_start_date) +
  geom_area(fill = "green", color = "red")

percent_first_click <- buscas %>% 
                        filter(!is.na(first_click)) %>% 
                        group_by(session_start_date, first_click) %>%
                        summarise(total = n())
percent_first_click <- percent_first_click %>%
  group_by(session_start_date) %>%
  mutate(percent = total / sum(total))
percent_first_click <- percent_first_click %>%
  filter(first_click != 1) %>%
  select(session_start_date, percent) %>%
  group_by(session_start_date) %>%
  mutate(percent_dif_1 = sum(percent) * 100) %>%
  select(-percent)
percent_first_click[!duplicated(percent_first_click[c("session_start_date","percent_dif_1")]),]

3. What is our daily overall zero results rate? How does it vary between the groups?

O gráfico exibe a distribuição de pesquisas que retornaram 0 resultados ao longo do periodo de coleta. O dia 5 é o dia com menor quantidade de resultados 0, mas como visto na questão 1 também é o dia com menos quantidade de clicks.

buscas %>% 
    filter(results == 0) %>% 
    group_by(session_start_date) %>% 
    mutate(num_results_0 = n()) %>% 
    ggplot(aes(x = session_start_date, y = num_results_0)) +
    geom_line() +
    labs(
      title = "Distribution of results 0",
      x = "Date",
      y = "Number of results 0"
    )

O gráfico exibe a mesma relação do anterior, porém dessa vez considerando os grupos. O grupo B apresenta menos resultados 0 do que o grupo A e também uma maior regularidade, ou seja, a mesma relação observada para o número de clicks.

buscas %>% 
    filter(results == 0) %>% 
    group_by(group, session_start_date) %>% 
    mutate(num_results_0 = n()) %>% 
    ggplot(aes(x = session_start_date, y = num_results_0, color = group)) +
    geom_line() +
    facet_grid(group ~ .) +
    labs(
      title = "Distribution of results 0",
      x = "Date",
      y = "Number of results 0"
    )

4. Let session length be approximately the time between the first event and the last event in a session. Choose a variable from the dataset and describe its relationship to session length. Visualize the relationship.

Aqui observamos a relação entre o tamanho da sessão e o valor de resultados retornados em pesquisaas dentro dessa mesma sessão. Não é possível perceber uma correção entre as variáveis, de qualquer natureza. Ao estimar as correlações nenhuma retorna um valor significante, ou seja, não existe correlação linear ou não linear entre as variáveis. Em outras palavras o tamanho da sessão e o valor de resultados retornados em pesquisaas dentro dessa mesma sessão são variáveis independentes entre si.

buscas %>%
    filter(session_length <= 100000) %>%
    ggplot(aes(x = session_length, y = results)) +
    geom_point()

buscas %>% 
    filter(session_length <= 100000) %>%
    summarise(pearson = cor(session_length, results, method = "pearson"), 
        spearman = cor(session_length, results, method = "spearman"), 
        kendall = cor(session_length, results, method = "kendall"))
LS0tCnRpdGxlOiAiRURBIGJ1c2NhcyIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKTyBvYmpldG8gcHJpbmNpcGFsIGRhIGFuw6FsaXNlIHPDo28gYXMgYnVzY2FzIGUgYSBuYXZlZ2HDp8OjbyBkZXBvaXMgZGEgYnVzY2EuIENyaWFtb3MgZXNzZXMgZGFkb3MgYSBwYXJ0aXIgZG9zIGRhZG9zIG9yaWdpbmFpcyBkYSB3aWtpbWVkaWEgZW0gYC9kYXRhL3NlYXJjaF9kYXRhLmNzdmAuIAoKQXF1aSwgZXhwbG9yYW1vcyBlc3NlcyBkYWRvcy4gCgpgYGB7ciBzZXR1cH0KbGlicmFyeSh0aWR5dmVyc2UpCmxpYnJhcnkoaGVyZSkKbGlicmFyeShsdWJyaWRhdGUpCnRoZW1lX3NldCh0aGVtZV9idygpKQpvcHRpb25zKHdhcm49LTEpICMgaWdub3JlIHdhcm5zCmBgYAoKYGBge3IgRVRMfQpidXNjYXMgPSByZWFkX2NzdihoZXJlOjpoZXJlKCJkYXRhL3NlYXJjaF9kYXRhLmNzdiIpKQpgYGAKCiMgRXhwbG9yYW5kbyBvcyBkYWRvcwoKQXF1aSBvYnNlcnZhbW9zIGEgZGlzdHJpYnVpw6fDo28gZGEgdmFyacOhdmVsIG7Dum1lcm8gZGUgY2xpY2tzLCBwb2RlbW9zIG9ic2VydmFyIHF1ZSBzdWEgbWFpb3IgY29uY2VudHJhw6fDo28gZXN0w6EgZW50cmUgMCBlIDIuVGFtYsOpbSBvYnNlcnZhbW9zIG91dGxpZXJzIHF1ZSBmb3JhbSByZW1vdmlkb3MuCgpgYGB7cn0KYnVzY2FzICU+JSAKICAgIGZpbHRlcihudW1fY2xpY2tzIDw9IDcpICU+JQogICAgZ2dwbG90KGFlcyh4ID0gbnVtX2NsaWNrcykpICsgCiAgICBnZW9tX2RlbnNpdHkoZmlsbD0nYmxhY2snKSArCiAgICBsYWJzKHRpdGxlID0gImRlbnNpdHkgbnVtX2NsaWNrcyIpCmBgYAoKQSBkaXN0cmlidWnDp8OjbyBwb3NzdWkgdW1hIGxvbmdhIGNhbGRhIGEgZGlyZWl0YSwgcGFyYSBmYWNpbGl0YXIgYSBpbnRlcnByZXRhw6fDo28gb3Mgb3V0bGllcnMgZm9yYW0gcmVtb3ZpZG9zLiAKCmBgYHtyfQpidXNjYXMgJT4lIAogIGZpbHRlcihzZXNzaW9uX2xlbmd0aCA8PSA1MDAwKSAlPiUKICBnZ3Bsb3QoYWVzKHg9c2Vzc2lvbl9sZW5ndGgpKSArCiAgZ2VvbV9kZW5zaXR5KGZpbGw9J2JsdWUnKSArCiAgbGFicyh0aXRsZSA9ICJkZW5zaXR5IHNlc3Npb25fbGVuZ3RoIikKYGBgCgpPYnNlcnZhbmRvIGEgcmVsYcOnw6NvIGVudHJlIHNlc3Npb25fbGVuZ3RoIHggbnVtX2NsaWNrcyBuw6NvIGFwZXJlbnRhIHRlciBjb3JyZWxhw6fDo28gZGUgcXVhbHF1ZXIgbmF0dXJlemEuIEFuYWxpc2FuZG8gb3MgY29lZmljaWVudGVzIGRlIGNvcnJlbGHDp8OjbyBwZXJjZWJlbW9zIHF1ZSBuZW5odW0gcG9zc3VpIHZhbG9yIHNpZ25pZmljYXRpdm8sIG91IHNlamEsIHJlYWxtZW50ZSBuw6NvIGV4aXN0ZSBjb3JyZWxhw6fDo28gZW50cmUgYXMgdmFyacOhdmVpcy4KCmBgYHtyfQpidXNjYXMgJT4lIAogIGZpbHRlcihzZXNzaW9uX2xlbmd0aCA8PSAxMDAwMDAgJiBudW1fY2xpY2tzIDw9IDI1KSAlPiUKICBnZ3Bsb3QoYWVzKHggPSBzZXNzaW9uX2xlbmd0aCwgeSA9IG51bV9jbGlja3MpKSArCiAgZ2VvbV9wb2ludCgpICsKICBsYWJzKHRpdGxlID0gInNlc3Npb25fbGVuZ3RoIHggbnVtX2NsaWNrcyIpCmJ1c2NhcyAlPiUgCiAgICBmaWx0ZXIoc2Vzc2lvbl9sZW5ndGggPD0gMTAwMDAwICYgbnVtX2NsaWNrcyA8PSAyNSkgJT4lCiAgICBzdW1tYXJpc2UoCiAgICAgIHBlYXJzb24gPSBjb3Ioc2Vzc2lvbl9sZW5ndGgsIG51bV9jbGlja3MsIG1ldGhvZCA9ICJwZWFyc29uIiksIAogICAgICBzcGVhcm1hbiA9IGNvcihzZXNzaW9uX2xlbmd0aCwgbnVtX2NsaWNrcywgbWV0aG9kID0gInNwZWFybWFuIiksIAogICAgICBrZW5kYWxsID0gY29yKHNlc3Npb25fbGVuZ3RoLCBudW1fY2xpY2tzLCBtZXRob2QgPSAia2VuZGFsbCIpCiAgICApCmBgYAoKQXMgZGlzdHJpYnVpw6fDtWVzIHBvc3N1ZW0gdW1hIGxvbmdhIGNhbGRhIGEgZGlyZWl0YSwgcGFyYSBmYWNpbGl0YXIgYSBpbnRlcnByZXRhw6fDo28gb3Mgb3V0bGllcnMgZm9yYW0gcmVtb3ZpZG9zLgogCmBgYHtyfQpidXNjYXMgJT4lIAogIGZpbHRlcihyZXN1bHRzIDw9IDI1KSAlPiUKICBnZ3Bsb3QoYWVzKHg9cmVzdWx0cykpICsKICBnZW9tX2RlbnNpdHkoYWxwaGEgPSAuNCwgZmlsbD0nZ3JlZW4nKSArCiAgbGFicyh0aXRsZSA9ICJkZW5zaXR5IHJlc3VsdHMiKQpidXNjYXMgJT4lIAogIGZpbHRlcihzZWFyY2hfaW5kZXggPD0gMzApICU+JQogIGdncGxvdChhZXMoeD1zZWFyY2hfaW5kZXgpKSArCiAgZ2VvbV9kZW5zaXR5KGZpbGw9J2JsdWUnKSArCiAgbGFicyh0aXRsZSA9ICJkZW5zaXR5IHNlYXJjaF9pbmRleCIpCmBgYAoKT2JzZXJ2YW5kbyBhIHJlbGHDp8OjbyBlbnRyZSByZXN1bHRzIGUgc2VhcmNoX2luZGV4LCBub3ZhbWVudGUsIG7Do28gw6kgcG9zc8OtdmVsIG9ic2VydmFyIGNvcnJlbGHDp8OjbyBkZSBxdWFscXVlciBuYXR1cmV6YSBlIGVzdGltYW5kbyBvcyBjb2VmaWNpZW50ZXMgZGUgY29ycmVsYcOnw6NvIG7Do28gZW5jb250cmFtb3MgdmFsb3JlcyBzaWduaWZpY2F0aXZvcy4KCmBgYHtyfQpidXNjYXMgJT4lIAogIGZpbHRlcihyZXN1bHRzIDw9IDI1ICYgc2VhcmNoX2luZGV4IDw9IDMwKSAlPiUKICBnZ3Bsb3QoYWVzKHg9cmVzdWx0cywgeT1zZWFyY2hfaW5kZXgpKSArCiAgZ2VvbV9wb2ludCgpICsKICBsYWJzKHRpdGxlID0gInJlc3VsdHMgeCBzZWFyY2hfaW5kZXgiKQpidXNjYXMgJT4lIAogICAgZmlsdGVyKHJlc3VsdHMgPD0gMjUgJiBzZWFyY2hfaW5kZXggPD0gMzApICU+JQogICAgc3VtbWFyaXNlKAogICAgICBwZWFyc29uID0gY29yKHJlc3VsdHMsIHNlYXJjaF9pbmRleCwgbWV0aG9kID0gInBlYXJzb24iKSwgCiAgICAgIHNwZWFybWFuID0gY29yKHJlc3VsdHMsIHNlYXJjaF9pbmRleCwgbWV0aG9kID0gInNwZWFybWFuIiksIAogICAgICBrZW5kYWxsID0gY29yKHJlc3VsdHMsIHNlYXJjaF9pbmRleCwgbWV0aG9kID0gImtlbmRhbGwiKQogICAgKQpgYGAKCgojIDEuIFdoYXQgaXMgb3VyIGRhaWx5IG92ZXJhbGwgY2xpY2t0aHJvdWdoIHJhdGU/IEhvdyBkb2VzIGl0IHZhcnkgYmV0d2VlbiB0aGUgZ3JvdXBzPwoKTyBncsOhZmljbyBleGliZSBhIHJlbGHDp8OjbyBlbnRyZSBkYXRhIGUgbsO6bWVybyBkZSBjbGlja3MsIMOpIHBvc3PDrXZlbCBwZXJjZWJlciBxdWUgb3MgZGlhcyAxLDIgZSAzIGRlIG1hcsOnbyBwb3NzdWVtIG1haXMgY2xpY2tzLiBPIGRpYSA1IGRlIG1hcsOnbyDDqSBvIGRpYSBjb20gbWVub3MgY2xpY2tzLgoKYGBge3J9CmJ1c2NhcyRzZXNzaW9uX3N0YXJ0X2RhdGUgPC0gYnVzY2FzJHNlc3Npb25fc3RhcnRfZGF0ZSAlPiUgYXMuRGF0ZSgiJWQvJW0vJVkiKQpidXNjYXMgJT4lCiAgZ3JvdXBfYnkoc2Vzc2lvbl9zdGFydF9kYXRlKSAlPiUKICBtdXRhdGUoY2xpY2tzX3Blcl9kYXkgPSBzdW0obnVtX2NsaWNrcykpICU+JQogIGdncGxvdChhZXMoeD1zZXNzaW9uX3N0YXJ0X2RhdGUsIHk9Y2xpY2tzX3Blcl9kYXkpKSArCiAgZ2VvbV9wb2ludCgpICsKICBnZW9tX2xpbmUoKSArCiAgbGFicyh0aXRsZSA9ICJjbGlja3NfcGVyX2RheSIpCmBgYAoKQ29uc2lkZXJhbmRvIGEgbWVzbWEgcmVsYcOnw6NvLCBtYXMgZGVzc2EgdmV6IGNvbnNpZGVyYW5kbyBvcyBncnVwb3MsIG9ic2VydmFtb3MgcXVlIG8gZ3J1cG8gQSBwb3NzdWkgdW0gbWFpb3IgbsO6bWVybyBkZSBjbGlja3MgZG8gcXVlIG8gZ3J1cG8gQiBlbSB0b2RvIG8gcGVyw61vZG8gb2JzZXJ2YWRvLiBPIGdydXBvIEIgcG9zc3VpIG1haW9yIHJlZ3VsYXJpZGFkZSBuYSBxdWFudGlkYWRlIGRlIGNsaWNrcyBkbyBxdWUgbyBncnVwbyBBLCBpc3NvIHBvZGUgc2VyIG9iZXJzYXZhZG8gY29tcGFyYW5kbyBhIHZhcmnDom5jaWEgZG9zIGRvaXMgZ3J1cG9zLgoKYGBge3J9CmJ1c2NhcyAlPiUKICBncm91cF9ieShncm91cCwgc2Vzc2lvbl9zdGFydF9kYXRlKSAlPiUKICBtdXRhdGUoY2xpY2tzX3Blcl9kYXkgPSBzdW0obnVtX2NsaWNrcykpICU+JQogIGdncGxvdChtYXBwaW5nID0gYWVzKHg9c2Vzc2lvbl9zdGFydF9kYXRlLCB5PWNsaWNrc19wZXJfZGF5LCBjb2xvciA9IGdyb3VwKSkgKwogIGdlb21fcG9pbnQoKSArCiAgZ2VvbV9saW5lKCkgKwogIGZhY2V0X2dyaWQoZ3JvdXAgfiAuKSArCiAgbGFicyh0aXRsZSA9ICJjbGlja3NfcGVyX2RheSIpCmJ1c2NhcyAlPiUgCiAgICBncm91cF9ieShncm91cCkgJT4lCiAgICBzdW1tYXJpc2UodmFyaWFuY2lhID0gdmFyKHNlc3Npb25fbGVuZ3RoLCByZXN1bHRzKSkKYGBgCgojIDIuIFdoaWNoIHJlc3VsdHMgZG8gcGVvcGxlIHRlbmQgdG8gdHJ5IGZpcnN0PyBIb3cgZG9lcyBpdCBjaGFuZ2UgZGF5LXRvLWRheT8KCkFuYWxpc2FuZG8gYSBkaXN0cmlidWnDp8OjbyBvYnNlcnZhbW9zIHF1ZSB1bWEgbWFpb3IgY29uY2VudHJhw6fDo28gbm8gdmFsb3IgMSBlIGRlcG9pcyB2YWkgZGVjcmVzY2VuZG8sIG91IHNlamEsIGFzIHBlc3NvYXMgY2xpY2FtIG1haXMgbm9zIHByaW1laXJvcyBsaW5rcyBkZSByZXRvcm5vIGRlIHVtYSBwZXNxdWlzYS4KCmBgYHtyfQpidXNjYXMgJT4lIAogIGZpbHRlcighaXMubmEoZmlyc3RfY2xpY2spKSAlPiUgCiAgZ3JvdXBfYnkoZmlyc3RfY2xpY2spICU+JSAKICBmaWx0ZXIobigpID49IDIwKSAlPiUKICBnZ3Bsb3QoYWVzKHg9Zmlyc3RfY2xpY2spKSArCiAgZ2VvbV9kZW5zaXR5KGZpbGwgPSAiYmx1ZSIsIGNvbG9yPSAiZ3JlZW4iLCBhbHBoYSA9IC40KQpgYGAKCkFuYWxpc2FuZG8gYSBkaXN0cmlidWnDp8OjbyBkb3MgcHJpbWVpcm9zIGNsaWNrcywgbWFzIGRlc3NhIHZleiBhZ3J1cGFuZG8gcG9yIGRpYSwgb2JzZXJ2YW1vcyBxdWUgdG9kb3Mgb3MgZGlhcyB0ZW0gdW1hIGRpc3RyaWJ1acOnw6NvIG11aXRvIHBhcmVjaWRhLCBjb20gY29uY2VudHJhw6fDo28gZW0gdG9ybm8gZGUgMS4gRW50w6NvLCBhbyBjYWxjdWxhciBvIHBlcmNlbnR1YWwgZGUgY2xpY2tzIGRpZmVyZW50ZXMgZGUgMSBwZXJjZWJlbW9zIHF1ZSBvcyBncnVwb3Mgc2UgZGlmZXJlbmNpYW0gdW0gcG91Y28sIDIwMTYtMDMtMDUgZm9pIG8gZGlhIGVtIHF1ZSBtYWlzIHNlIGNsaWNvdSBlbSBsaW5rcyBxdWUgbsOjbyBmb3NzZW0gbyBwcmltZWlybywgY29tIHF1YXNlIDIlIGEgbWFpcyBkbyBxdWUgbyBkaWEgMjAxNi0wMy0wMSBlbSBxdWUgbWVub3Mgc2UgY2xpY291LiAgIAoKYGBge3J9CmJ1c2NhcyAlPiUgCiAgZmlsdGVyKCFpcy5uYShmaXJzdF9jbGljaykpICU+JSAKICBncm91cF9ieShmaXJzdF9jbGljaywgc2Vzc2lvbl9zdGFydF9kYXRlKSAlPiUgCiAgc3VtbWFyaXNlKG51bV9jbGlja3MgPSBuKCkpICU+JQogIGZpbHRlcihudW1fY2xpY2tzID49IDIwKSAlPiUKICBnZ3Bsb3QoYWVzKHg9Zmlyc3RfY2xpY2ssIHkgPSBudW1fY2xpY2tzKSkgKwogIGZhY2V0X3dyYXAofnNlc3Npb25fc3RhcnRfZGF0ZSkgKwogIGdlb21fYXJlYShmaWxsID0gImdyZWVuIiwgY29sb3IgPSAicmVkIikKcGVyY2VudF9maXJzdF9jbGljayA8LSBidXNjYXMgJT4lIAogICAgICAgICAgICAgICAgICAgICAgICBmaWx0ZXIoIWlzLm5hKGZpcnN0X2NsaWNrKSkgJT4lIAogICAgICAgICAgICAgICAgICAgICAgICBncm91cF9ieShzZXNzaW9uX3N0YXJ0X2RhdGUsIGZpcnN0X2NsaWNrKSAlPiUKICAgICAgICAgICAgICAgICAgICAgICAgc3VtbWFyaXNlKHRvdGFsID0gbigpKQpwZXJjZW50X2ZpcnN0X2NsaWNrIDwtIHBlcmNlbnRfZmlyc3RfY2xpY2sgJT4lCiAgZ3JvdXBfYnkoc2Vzc2lvbl9zdGFydF9kYXRlKSAlPiUKICBtdXRhdGUocGVyY2VudCA9IHRvdGFsIC8gc3VtKHRvdGFsKSkKcGVyY2VudF9maXJzdF9jbGljayA8LSBwZXJjZW50X2ZpcnN0X2NsaWNrICU+JQogIGZpbHRlcihmaXJzdF9jbGljayAhPSAxKSAlPiUKICBzZWxlY3Qoc2Vzc2lvbl9zdGFydF9kYXRlLCBwZXJjZW50KSAlPiUKICBncm91cF9ieShzZXNzaW9uX3N0YXJ0X2RhdGUpICU+JQogIG11dGF0ZShwZXJjZW50X2RpZl8xID0gc3VtKHBlcmNlbnQpICogMTAwKSAlPiUKICBzZWxlY3QoLXBlcmNlbnQpCnBlcmNlbnRfZmlyc3RfY2xpY2tbIWR1cGxpY2F0ZWQocGVyY2VudF9maXJzdF9jbGlja1tjKCJzZXNzaW9uX3N0YXJ0X2RhdGUiLCJwZXJjZW50X2RpZl8xIildKSxdCmBgYAoKIyAzLiBXaGF0IGlzIG91ciBkYWlseSBvdmVyYWxsIHplcm8gcmVzdWx0cyByYXRlPyBIb3cgZG9lcyBpdCB2YXJ5IGJldHdlZW4gdGhlIGdyb3Vwcz8KCk8gZ3LDoWZpY28gZXhpYmUgYSBkaXN0cmlidWnDp8OjbyBkZSBwZXNxdWlzYXMgcXVlIHJldG9ybmFyYW0gMCByZXN1bHRhZG9zIGFvIGxvbmdvIGRvIHBlcmlvZG8gZGUgY29sZXRhLiBPIGRpYSA1IMOpIG8gZGlhIGNvbSBtZW5vciBxdWFudGlkYWRlIGRlIHJlc3VsdGFkb3MgMCwgbWFzIGNvbW8gdmlzdG8gbmEgcXVlc3TDo28gMSB0YW1iw6ltIMOpIG8gZGlhIGNvbSBtZW5vcyBxdWFudGlkYWRlIGRlIGNsaWNrcy4KCmBgYHtyfQpidXNjYXMgJT4lIAogICAgZmlsdGVyKHJlc3VsdHMgPT0gMCkgJT4lIAogICAgZ3JvdXBfYnkoc2Vzc2lvbl9zdGFydF9kYXRlKSAlPiUgCiAgICBtdXRhdGUobnVtX3Jlc3VsdHNfMCA9IG4oKSkgJT4lIAogICAgZ2dwbG90KGFlcyh4ID0gc2Vzc2lvbl9zdGFydF9kYXRlLCB5ID0gbnVtX3Jlc3VsdHNfMCkpICsKICAgIGdlb21fbGluZSgpICsKICAgIGxhYnMoCiAgICAgIHRpdGxlID0gIkRpc3RyaWJ1dGlvbiBvZiByZXN1bHRzIDAiLAogICAgICB4ID0gIkRhdGUiLAogICAgICB5ID0gIk51bWJlciBvZiByZXN1bHRzIDAiCiAgICApCmBgYAoKTyBncsOhZmljbyBleGliZSBhIG1lc21hIHJlbGHDp8OjbyBkbyBhbnRlcmlvciwgcG9yw6ltIGRlc3NhIHZleiBjb25zaWRlcmFuZG8gb3MgZ3J1cG9zLiBPIGdydXBvIEIgYXByZXNlbnRhIG1lbm9zIHJlc3VsdGFkb3MgMCBkbyBxdWUgbyBncnVwbyBBIGUgdGFtYsOpbSB1bWEgbWFpb3IgcmVndWxhcmlkYWRlLCBvdSBzZWphLCBhIG1lc21hIHJlbGHDp8OjbyBvYnNlcnZhZGEgcGFyYSBvIG7Dum1lcm8gZGUgY2xpY2tzLgoKYGBge3J9CmJ1c2NhcyAlPiUgCiAgICBmaWx0ZXIocmVzdWx0cyA9PSAwKSAlPiUgCiAgICBncm91cF9ieShncm91cCwgc2Vzc2lvbl9zdGFydF9kYXRlKSAlPiUgCiAgICBtdXRhdGUobnVtX3Jlc3VsdHNfMCA9IG4oKSkgJT4lIAogICAgZ2dwbG90KGFlcyh4ID0gc2Vzc2lvbl9zdGFydF9kYXRlLCB5ID0gbnVtX3Jlc3VsdHNfMCwgY29sb3IgPSBncm91cCkpICsKICAgIGdlb21fbGluZSgpICsKICAgIGZhY2V0X2dyaWQoZ3JvdXAgfiAuKSArCiAgICBsYWJzKAogICAgICB0aXRsZSA9ICJEaXN0cmlidXRpb24gb2YgcmVzdWx0cyAwIiwKICAgICAgeCA9ICJEYXRlIiwKICAgICAgeSA9ICJOdW1iZXIgb2YgcmVzdWx0cyAwIgogICAgKQpgYGAKCgojIDQuIExldCBzZXNzaW9uIGxlbmd0aCBiZSBhcHByb3hpbWF0ZWx5IHRoZSB0aW1lIGJldHdlZW4gdGhlIGZpcnN0IGV2ZW50IGFuZCB0aGUgbGFzdCBldmVudCBpbiBhIHNlc3Npb24uIENob29zZSBhIHZhcmlhYmxlIGZyb20gdGhlIGRhdGFzZXQgYW5kIGRlc2NyaWJlIGl0cyByZWxhdGlvbnNoaXAgdG8gc2Vzc2lvbiBsZW5ndGguIFZpc3VhbGl6ZSB0aGUgcmVsYXRpb25zaGlwLgoKQXF1aSBvYnNlcnZhbW9zIGEgcmVsYcOnw6NvIGVudHJlIG8gdGFtYW5obyBkYSBzZXNzw6NvIGUgbyB2YWxvciBkZSByZXN1bHRhZG9zIHJldG9ybmFkb3MgZW0gcGVzcXVpc2FhcyBkZW50cm8gZGVzc2EgbWVzbWEgc2Vzc8Ojby4gTsOjbyDDqSBwb3Nzw612ZWwgcGVyY2ViZXIgdW1hIGNvcnJlw6fDo28gZW50cmUgYXMgdmFyacOhdmVpcywgZGUgcXVhbHF1ZXIgbmF0dXJlemEuIEFvIGVzdGltYXIgYXMgY29ycmVsYcOnw7VlcyBuZW5odW1hIHJldG9ybmEgdW0gdmFsb3Igc2lnbmlmaWNhbnRlLCBvdSBzZWphLCBuw6NvIGV4aXN0ZSBjb3JyZWxhw6fDo28gbGluZWFyIG91IG7Do28gbGluZWFyIGVudHJlIGFzIHZhcmnDoXZlaXMuIEVtIG91dHJhcyBwYWxhdnJhcyBvIHRhbWFuaG8gZGEgc2Vzc8OjbyBlIG8gdmFsb3IgZGUgcmVzdWx0YWRvcyByZXRvcm5hZG9zIGVtIHBlc3F1aXNhYXMgZGVudHJvIGRlc3NhIG1lc21hIHNlc3PDo28gc8OjbyB2YXJpw6F2ZWlzIGluZGVwZW5kZW50ZXMgZW50cmUgc2kuCgpgYGB7cn0KYnVzY2FzICU+JQogICAgZmlsdGVyKHNlc3Npb25fbGVuZ3RoIDw9IDEwMDAwMCkgJT4lCiAgICBnZ3Bsb3QoYWVzKHggPSBzZXNzaW9uX2xlbmd0aCwgeSA9IHJlc3VsdHMpKSArCiAgICBnZW9tX3BvaW50KCkKYnVzY2FzICU+JSAKICAgIGZpbHRlcihzZXNzaW9uX2xlbmd0aCA8PSAxMDAwMDApICU+JQogICAgc3VtbWFyaXNlKHBlYXJzb24gPSBjb3Ioc2Vzc2lvbl9sZW5ndGgsIHJlc3VsdHMsIG1ldGhvZCA9ICJwZWFyc29uIiksIAogICAgICAgIHNwZWFybWFuID0gY29yKHNlc3Npb25fbGVuZ3RoLCByZXN1bHRzLCBtZXRob2QgPSAic3BlYXJtYW4iKSwgCiAgICAgICAga2VuZGFsbCA9IGNvcihzZXNzaW9uX2xlbmd0aCwgcmVzdWx0cywgbWV0aG9kID0gImtlbmRhbGwiKSkKYGBg