O objeto principal da análise são as buscas e a navegação depois da busca. Criamos esses dados a partir dos dados originais da wikimedia em /data/search_data.csv.

Aqui, exploramos esses dados.

library(tidyverse)
library(here)
library(lubridate)
theme_set(theme_bw())
options(warn=-1) # ignore warns
buscas = read_csv(here::here("data/search_data.csv"))
Parsed with column specification:
cols(
  session_id = col_character(),
  search_index = col_double(),
  session_length = col_double(),
  session_start_timestamp = col_double(),
  session_start_date = col_datetime(format = ""),
  group = col_character(),
  results = col_double(),
  num_clicks = col_double(),
  first_click = col_double()
)

Explorando os dados

Aqui observamos a distribuição da variável número de clicks, podemos observar que sua maior concentração está entre 0 e 2.Também observamos outliers que foram removidos.

Os gráficos são referentes as distribuições das variáveis session_start_date e session_start_timestamp, respectivamente. Em ambos os casos é perceptível padrões nos horários de início das sessões.

buscas %>% ggplot(aes(x= session_start_date)) +
    geom_histogram() + 
    coord_flip()


buscas %>% 
  ggplot(aes(x=session_start_timestamp)) +
  geom_density(alpha = .4, fill='purple') +
  labs(title = "density session_start_timestamp")

O gráfico exibe a distribuição da variável session_length, a distibuição se concentra em session_length menores que 1000. A distribuição possui uma longa calda a direita, para facilitar a interpretação os outliers foram removidos.

Aqui observamos a relação entre session_length x num_clicks que não aperenta ter correlação de qualquer natureza. Estimando os coeficientes de correlação percebemos que nenhum possui valor significativo, ou seja, não existe correlação entre as variáveis.

buscas %>% 
  filter(session_length <= 100000) %>%
  filter(num_clicks <= 25) %>%
  ggplot(aes(x = session_length, y = num_clicks)) +
  geom_point() +
  labs(title = "session_length x num_clicks")


buscas %>% 
    filter(session_length <= 100000) %>%
    filter(num_clicks <= 25) %>%
    summarise(pearson = cor(session_length, num_clicks, method = "pearson"), 
        spearman = cor(session_length, num_clicks, method = "spearman"), 
        kendall = cor(session_length, num_clicks, method = "kendall"))

Os gráficos mostram a distribuição das variáveis results e search_index. A distribuição da variável results se concentra em torno de 20 e 0, já para a variável search_index a distribuição se concentra para valores menores que 5. Ambas as distribuições possuem uma longa calda a direita, para facilitar a interpretação os outliers foram removidos.

buscas %>% 
  filter(results <= 25) %>%
  ggplot(aes(x=results)) +
  geom_density(alpha = .4, fill='orange') +
  labs(title = "density results")



buscas %>% 
  filter(search_index <= 30) %>%
  ggplot(aes(x=search_index)) +
  geom_density(alpha = .4, fill='red') +
  labs(title = "density search_index")

Aqui observamos a relação entre results e search_index, novamente, não é possível observar correlação de qualquer natureza. Estimando os coeficientes de correlação não encontramos valores significativos.

buscas %>% 
  filter(results <= 25 & search_index <= 30) %>%
  ggplot(aes(x=results, y=search_index)) +
  geom_point() +
  labs(title = "results x search_index")


buscas %>% 
    filter(results <= 25 & search_index <= 30) %>%
    summarise(
      pearson = cor(results, search_index, method = "pearson"), 
      spearman = cor(results, search_index, method = "spearman"), 
      kendall = cor(results, search_index, method = "kendall")
    )

1. What is our daily overall clickthrough rate? How does it vary between the groups?

O gráfico exibe a relação entre data e número de clicks, é possível perceber que os dias 1,2 e 3 de março possuem mais clicks. O dia 5 de março é o dia com menos clicks.

buscas$session_start_date <- buscas$session_start_date %>% as.Date("%d/%m/%Y")
unknown timezone '%d/%m/%Y'
buscas %>%
  group_by(session_start_date) %>%
  mutate(clicks_per_day = sum(num_clicks)) %>%
  ggplot(aes(x=session_start_date, y=clicks_per_day)) +
  geom_point() +
  geom_line() +
  labs(title = "clicks_per_day")

Considerando a mesma relação, mas dessa vez considerando os grupos, observamos que o grupo A possui um maior número de clicks do que o grupo B em todo o período observado. O grupo B possui maior regularidade na quantidade de clicks do que o grupo A, isso pode ser obersavado comparando a variância dos dois grupos.

buscas %>%
  group_by(group, session_start_date) %>%
  mutate(clicks_per_day = sum(num_clicks)) %>%
  ggplot(mapping = aes(x=session_start_date, y=clicks_per_day, color = group)) +
  geom_point() +
  geom_line() +
  facet_grid(group ~ .) +
  labs(title = "clicks_per_day")


buscas %>% 
    group_by(group) %>%
    summarise(variancia = var(session_length, results))

2. Which results do people tend to try first? How does it change day-to-day?

Analisando a distribuição dos primeiros clicks, observamos que essa tem maior concentração no valor 1 e depois vai decrescendo proporcionalmente, ou seja, as pessoas clicam mais nos primeiros links de retorno de uma pesquisa.

buscas %>% 
  filter(!is.na(first_click)) %>% 
  group_by(first_click) %>% 
  filter(n() >= 20) %>%
  ggplot(aes(x=first_click)) +
  geom_density(fill = "green", color= "red", alpha = .4)

Analisando a distribuição dos primeiros clicks, mas dessa vez agrupando por dia, observamos que todos os dias tem uma distribuição muito parecida, com concentração em torno de 1. Então, ao calcular o percentual de clicks diferentes de 1 percebemos que os grupos se diferenciam um pouco, 2016-03-05 foi o dia em que mais se clicou em links que não fossem o primeiro, com quase 2% a mais do que o dia 2016-03-01 em que menos se clicou.

buscas %>% 
  filter(!is.na(first_click)) %>% 
  group_by(first_click, session_start_date) %>% 
  summarise(num_clicks = n()) %>%
  filter(num_clicks >= 20) %>%
  ggplot(aes(x=first_click, y = num_clicks)) +
  facet_wrap(~session_start_date) +
  geom_area(fill = "green", color = "red")



percent_first_click <- buscas %>% 
                        filter(!is.na(first_click)) %>% 
                        group_by(session_start_date, first_click) %>%
                        summarise(total = n())

percent_first_click <- percent_first_click %>%
  group_by(session_start_date) %>%
  mutate(percent = total / sum(total))

percent_first_click <- percent_first_click %>%
  filter(first_click != 1) %>%
  select(session_start_date, percent) %>%
  group_by(session_start_date) %>%
  mutate(percent_dif_1 = sum(percent) * 100) %>%
  select(-percent)

percent_first_click[!duplicated(percent_first_click[c("session_start_date","percent_dif_1")]),]

3. What is our daily overall zero results rate? How does it vary between the groups?

O gráfico exibe a distribuição de pesquisas que retornaram 0 resultados ao longo do periodo de coleta. O dia 5 é o dia com menor quantidade de resultados 0, mas como visto na questão 1 também é o dia com menos quantidade de clicks.

buscas %>% 
    filter(results == 0) %>% 
    group_by(session_start_date) %>% 
    mutate(num_results_0 = n()) %>% 
    ggplot(aes(x = session_start_date, y = num_results_0)) +
    geom_line() +
    labs(
      title = "Distribution of results 0",
      x = "Date",
      y = "Number of results 0"
    )

O gráfico exibe a mesma relação do anterior, porém dessa vez considerando os grupos. O grupo B apresenta menos resultados 0 do que o grupo A e também uma maior regularidade, ou seja, a mesma relação observada para o número de clicks.

buscas %>% 
    filter(results == 0) %>% 
    group_by(group, session_start_date) %>% 
    mutate(num_results_0 = n()) %>% 
    ggplot(aes(x = session_start_date, y = num_results_0, color = group)) +
    geom_line() +
    facet_grid(group ~ .) +
    labs(
      title = "Distribution of results 0",
      x = "Date",
      y = "Number of results 0"
    )

4. Let session length be approximately the time between the first event and the last event in a session. Choose a variable from the dataset and describe its relationship to session length. Visualize the relationship.

Aqui observamos a relação entre o tamanho da sessão e o valor de resultados retornados em pesquisaas dentro dessa mesma sessão. Não é possível perceber uma correção entre as variáveis, de qualquer natureza. Ao estimar as correlações nenhuma retorna um valor significante, ou seja, não existe correlação linear ou não linear entre as variáveis. Em outras palavras o tamanho da sessão e e o valor de resultados retornados em pesquisaas dentro dessa mesma sessão são variáveis independentes entre si.

buscas %>%
    filter(session_length <= 100000) %>%
    ggplot(aes(x = session_length, y = results)) +
    geom_point()


buscas %>% 
    filter(session_length <= 100000) %>%
    summarise(pearson = cor(session_length, results, method = "pearson"), 
        spearman = cor(session_length, results, method = "spearman"), 
        kendall = cor(session_length, results, method = "kendall"))
NA
LS0tDQp0aXRsZTogIkVEQSBidXNjYXMiDQpvdXRwdXQ6IA0KICBodG1sX25vdGVib29rOg0KICAgIHRoZW1lOiBsdW1lbg0KICAgIGZpZ193aWR0aDogNw0KICAgIHRvYzogdHJ1ZQ0KICAgIHRvY19mbG9hdDogdHJ1ZQ0KICAgIGNvZGVfZm9sZGluZzogaGlkZQ0KZWRpdG9yX29wdGlvbnM6IA0KICBjaHVua19vdXRwdXRfdHlwZTogaW5saW5lDQotLS0NCg0KTyBvYmpldG8gcHJpbmNpcGFsIGRhIGFuw6FsaXNlIHPDo28gYXMgYnVzY2FzIGUgYSBuYXZlZ2HDp8OjbyBkZXBvaXMgZGEgYnVzY2EuIENyaWFtb3MgZXNzZXMgZGFkb3MgYSBwYXJ0aXIgZG9zIGRhZG9zIG9yaWdpbmFpcyBkYSB3aWtpbWVkaWEgZW0gYC9kYXRhL3NlYXJjaF9kYXRhLmNzdmAuIA0KDQpBcXVpLCBleHBsb3JhbW9zIGVzc2VzIGRhZG9zLiANCg0KYGBge3Igc2V0dXB9DQpsaWJyYXJ5KHRpZHl2ZXJzZSkNCmxpYnJhcnkoaGVyZSkNCmxpYnJhcnkobHVicmlkYXRlKQ0KdGhlbWVfc2V0KHRoZW1lX2J3KCkpDQpvcHRpb25zKHdhcm49LTEpICMgaWdub3JlIHdhcm5zDQpgYGANCg0KYGBge3IgRVRMfQ0KYnVzY2FzID0gcmVhZF9jc3YoaGVyZTo6aGVyZSgiZGF0YS9zZWFyY2hfZGF0YS5jc3YiKSkNCmBgYA0KDQojIEV4cGxvcmFuZG8gb3MgZGFkb3MNCg0KQXF1aSBvYnNlcnZhbW9zIGEgZGlzdHJpYnVpw6fDo28gZGEgdmFyacOhdmVsIG7Dum1lcm8gZGUgY2xpY2tzLCBwb2RlbW9zIG9ic2VydmFyIHF1ZSBzdWEgbWFpb3IgY29uY2VudHJhw6fDo28gZXN0w6EgZW50cmUgMCBlIDIuVGFtYsOpbSBvYnNlcnZhbW9zIG91dGxpZXJzIHF1ZSBmb3JhbSByZW1vdmlkb3MuDQoNCmBgYHtyfQ0KYnVzY2FzICU+JSANCiAgICBmaWx0ZXIobnVtX2NsaWNrcyA8PSA3KSAlPiUNCiAgICBnZ3Bsb3QoYWVzKHggPSBudW1fY2xpY2tzKSkgKyANCiAgICBnZW9tX2RlbnNpdHkoZmlsbD0nYmxhY2snKSArDQogICAgbGFicyh0aXRsZSA9ICJkZW5zaXR5IG51bV9jbGlja3MiKQ0KYGBgDQoNCk9zIGdyw6FmaWNvcyBzw6NvIHJlZmVyZW50ZXMgYXMgZGlzdHJpYnVpw6fDtWVzIGRhcyB2YXJpw6F2ZWlzIHNlc3Npb25fc3RhcnRfZGF0ZSBlIHNlc3Npb25fc3RhcnRfdGltZXN0YW1wLCByZXNwZWN0aXZhbWVudGUuIEVtIGFtYm9zIG9zIGNhc29zIMOpIHBlcmNlcHTDrXZlbCBwYWRyw7VlcyBub3MgaG9yw6FyaW9zIGRlIGluw61jaW8gZGFzIHNlc3PDtWVzLg0KDQpgYGB7cn0NCmJ1c2NhcyAlPiUgZ2dwbG90KGFlcyh4PSBzZXNzaW9uX3N0YXJ0X2RhdGUpKSArDQogICAgZ2VvbV9oaXN0b2dyYW0oKSArIA0KICAgIGNvb3JkX2ZsaXAoKSArDQogICAgbGFicyh0aXRsZSA9ICJkaXN0cmlidXRpb24gc2Vzc2lvbl9zdGFydF9kYXRlIikNCg0KYnVzY2FzICU+JSANCiAgZ2dwbG90KGFlcyh4PXNlc3Npb25fc3RhcnRfdGltZXN0YW1wKSkgKw0KICBnZW9tX2RlbnNpdHkoYWxwaGEgPSAuNCwgZmlsbD0ncHVycGxlJykgKw0KICBsYWJzKHRpdGxlID0gImRlbnNpdHkgc2Vzc2lvbl9zdGFydF90aW1lc3RhbXAiKQ0KDQpgYGANCg0KTyBncsOhZmljbyBleGliZSBhIGRpc3RyaWJ1acOnw6NvIGRhIHZhcmnDoXZlbCBzZXNzaW9uX2xlbmd0aCwgYSBkaXN0aWJ1acOnw6NvIHNlIGNvbmNlbnRyYSBlbSBzZXNzaW9uX2xlbmd0aCBtZW5vcmVzIHF1ZSAxMDAwLiBBIGRpc3RyaWJ1acOnw6NvIHBvc3N1aSB1bWEgbG9uZ2EgY2FsZGEgYSBkaXJlaXRhLCBwYXJhIGZhY2lsaXRhciBhIGludGVycHJldGHDp8OjbyBvcyBvdXRsaWVycyBmb3JhbSByZW1vdmlkb3MuIA0KDQpgYGB7cn0NCmJ1c2NhcyAlPiUgDQogIGZpbHRlcihzZXNzaW9uX2xlbmd0aCA8PSA1MDAwKSAlPiUNCiAgZ2dwbG90KGFlcyh4PXNlc3Npb25fbGVuZ3RoKSkgKw0KICBnZW9tX2RlbnNpdHkoYWxwaGEgPSAuNCwgZmlsbD0ncmVkJykgKw0KICBsYWJzKHRpdGxlID0gImRlbnNpdHkgc2Vzc2lvbl9sZW5ndGgiKQ0KYGBgDQoNCkFxdWkgb2JzZXJ2YW1vcyBhIHJlbGHDp8OjbyBlbnRyZSBzZXNzaW9uX2xlbmd0aCB4IG51bV9jbGlja3MgcXVlIG7Do28gYXBlcmVudGEgdGVyIGNvcnJlbGHDp8OjbyBkZSBxdWFscXVlciBuYXR1cmV6YS4gRXN0aW1hbmRvIG9zIGNvZWZpY2llbnRlcyBkZSBjb3JyZWxhw6fDo28gcGVyY2ViZW1vcyBxdWUgbmVuaHVtIHBvc3N1aSB2YWxvciBzaWduaWZpY2F0aXZvLCBvdSBzZWphLCBuw6NvIGV4aXN0ZSBjb3JyZWxhw6fDo28gZW50cmUgYXMgdmFyacOhdmVpcy4NCg0KYGBge3J9DQpidXNjYXMgJT4lIA0KICBmaWx0ZXIoc2Vzc2lvbl9sZW5ndGggPD0gMTAwMDAwICYgbnVtX2NsaWNrcyA8PSAyNSkgJT4lDQogIGdncGxvdChhZXMoeCA9IHNlc3Npb25fbGVuZ3RoLCB5ID0gbnVtX2NsaWNrcykpICsNCiAgZ2VvbV9wb2ludCgpICsNCiAgbGFicyh0aXRsZSA9ICJzZXNzaW9uX2xlbmd0aCB4IG51bV9jbGlja3MiKQ0KDQpidXNjYXMgJT4lIA0KICAgIGZpbHRlcihzZXNzaW9uX2xlbmd0aCA8PSAxMDAwMDAgJiBudW1fY2xpY2tzIDw9IDI1KSAlPiUNCiAgICBzdW1tYXJpc2UoDQogICAgICBwZWFyc29uID0gY29yKHNlc3Npb25fbGVuZ3RoLCBudW1fY2xpY2tzLCBtZXRob2QgPSAicGVhcnNvbiIpLCANCiAgICAgIHNwZWFybWFuID0gY29yKHNlc3Npb25fbGVuZ3RoLCBudW1fY2xpY2tzLCBtZXRob2QgPSAic3BlYXJtYW4iKSwgDQogICAgICBrZW5kYWxsID0gY29yKHNlc3Npb25fbGVuZ3RoLCBudW1fY2xpY2tzLCBtZXRob2QgPSAia2VuZGFsbCIpDQogICAgKQ0KYGBgDQoNCk9zIGdyw6FmaWNvcyBtb3N0cmFtIGEgZGlzdHJpYnVpw6fDo28gZGFzIHZhcmnDoXZlaXMgcmVzdWx0cyBlIHNlYXJjaF9pbmRleC4gQSBkaXN0cmlidWnDp8OjbyBkYSB2YXJpw6F2ZWwgcmVzdWx0cyBzZSBjb25jZW50cmEgZW0gdG9ybm8gZGUgMjAgZSAwLCBqw6EgcGFyYSBhIHZhcmnDoXZlbCBzZWFyY2hfaW5kZXgNCiBhIGRpc3RyaWJ1acOnw6NvIHNlIGNvbmNlbnRyYSBwYXJhIHZhbG9yZXMgbWVub3JlcyBxdWUgNS4gQW1iYXMgYXMgZGlzdHJpYnVpw6fDtWVzIHBvc3N1ZW0gdW1hIGxvbmdhIGNhbGRhIGEgZGlyZWl0YSwgcGFyYSBmYWNpbGl0YXIgYSBpbnRlcnByZXRhw6fDo28gb3Mgb3V0bGllcnMgZm9yYW0gcmVtb3ZpZG9zLg0KIA0KYGBge3J9DQpidXNjYXMgJT4lIA0KICBmaWx0ZXIocmVzdWx0cyA8PSAyNSkgJT4lDQogIGdncGxvdChhZXMoeD1yZXN1bHRzKSkgKw0KICBnZW9tX2RlbnNpdHkoYWxwaGEgPSAuNCwgZmlsbD0nb3JhbmdlJykgKw0KICBsYWJzKHRpdGxlID0gImRlbnNpdHkgcmVzdWx0cyIpDQoNCg0KYnVzY2FzICU+JSANCiAgZmlsdGVyKHNlYXJjaF9pbmRleCA8PSAzMCkgJT4lDQogIGdncGxvdChhZXMoeD1zZWFyY2hfaW5kZXgpKSArDQogIGdlb21fZGVuc2l0eShhbHBoYSA9IC40LCBmaWxsPSdyZWQnKSArDQogIGxhYnModGl0bGUgPSAiZGVuc2l0eSBzZWFyY2hfaW5kZXgiKQ0KDQpgYGANCg0KQXF1aSBvYnNlcnZhbW9zIGEgcmVsYcOnw6NvIGVudHJlIHJlc3VsdHMgZSBzZWFyY2hfaW5kZXgsIG5vdmFtZW50ZSwgbsOjbyDDqSBwb3Nzw612ZWwgb2JzZXJ2YXIgY29ycmVsYcOnw6NvIGRlIHF1YWxxdWVyIG5hdHVyZXphLiBFc3RpbWFuZG8gb3MgY29lZmljaWVudGVzIGRlIGNvcnJlbGHDp8OjbyBuw6NvIGVuY29udHJhbW9zIHZhbG9yZXMgc2lnbmlmaWNhdGl2b3MuDQoNCmBgYHtyfQ0KYnVzY2FzICU+JSANCiAgZmlsdGVyKHJlc3VsdHMgPD0gMjUgJiBzZWFyY2hfaW5kZXggPD0gMzApICU+JQ0KICBnZ3Bsb3QoYWVzKHg9cmVzdWx0cywgeT1zZWFyY2hfaW5kZXgpKSArDQogIGdlb21fcG9pbnQoKSArDQogIGxhYnModGl0bGUgPSAicmVzdWx0cyB4IHNlYXJjaF9pbmRleCIpDQoNCmJ1c2NhcyAlPiUgDQogICAgZmlsdGVyKHJlc3VsdHMgPD0gMjUgJiBzZWFyY2hfaW5kZXggPD0gMzApICU+JQ0KICAgIHN1bW1hcmlzZSgNCiAgICAgIHBlYXJzb24gPSBjb3IocmVzdWx0cywgc2VhcmNoX2luZGV4LCBtZXRob2QgPSAicGVhcnNvbiIpLCANCiAgICAgIHNwZWFybWFuID0gY29yKHJlc3VsdHMsIHNlYXJjaF9pbmRleCwgbWV0aG9kID0gInNwZWFybWFuIiksIA0KICAgICAga2VuZGFsbCA9IGNvcihyZXN1bHRzLCBzZWFyY2hfaW5kZXgsIG1ldGhvZCA9ICJrZW5kYWxsIikNCiAgICApDQpgYGANCg0KIyAxLiBXaGF0IGlzIG91ciBkYWlseSBvdmVyYWxsIGNsaWNrdGhyb3VnaCByYXRlPyBIb3cgZG9lcyBpdCB2YXJ5IGJldHdlZW4gdGhlIGdyb3Vwcz8NCg0KTyBncsOhZmljbyBleGliZSBhIHJlbGHDp8OjbyBlbnRyZSBkYXRhIGUgbsO6bWVybyBkZSBjbGlja3MsIMOpIHBvc3PDrXZlbCBwZXJjZWJlciBxdWUgb3MgZGlhcyAxLDIgZSAzIGRlIG1hcsOnbyBwb3NzdWVtIG1haXMgY2xpY2tzLiBPIGRpYSA1IGRlIG1hcsOnbyDDqSBvIGRpYSBjb20gbWVub3MgY2xpY2tzLg0KDQpgYGB7cn0NCmJ1c2NhcyRzZXNzaW9uX3N0YXJ0X2RhdGUgPC0gYnVzY2FzJHNlc3Npb25fc3RhcnRfZGF0ZSAlPiUgYXMuRGF0ZSgiJWQvJW0vJVkiKQ0KDQpidXNjYXMgJT4lDQogIGdyb3VwX2J5KHNlc3Npb25fc3RhcnRfZGF0ZSkgJT4lDQogIG11dGF0ZShjbGlja3NfcGVyX2RheSA9IHN1bShudW1fY2xpY2tzKSkgJT4lDQogIGdncGxvdChhZXMoeD1zZXNzaW9uX3N0YXJ0X2RhdGUsIHk9Y2xpY2tzX3Blcl9kYXkpKSArDQogIGdlb21fcG9pbnQoKSArDQogIGdlb21fbGluZSgpICsNCiAgbGFicyh0aXRsZSA9ICJjbGlja3NfcGVyX2RheSIpDQpgYGANCg0KQ29uc2lkZXJhbmRvIGEgbWVzbWEgcmVsYcOnw6NvLCBtYXMgZGVzc2EgdmV6IGNvbnNpZGVyYW5kbyBvcyBncnVwb3MsIG9ic2VydmFtb3MgcXVlIG8gZ3J1cG8gQSBwb3NzdWkgdW0gbWFpb3IgbsO6bWVybyBkZSBjbGlja3MgZG8gcXVlIG8gZ3J1cG8gQiBlbSB0b2RvIG8gcGVyw61vZG8gb2JzZXJ2YWRvLiBPIGdydXBvIEIgcG9zc3VpIG1haW9yIHJlZ3VsYXJpZGFkZSBuYSBxdWFudGlkYWRlIGRlIGNsaWNrcyBkbyBxdWUgbyBncnVwbyBBLCBpc3NvIHBvZGUgc2VyIG9iZXJzYXZhZG8gY29tcGFyYW5kbyBhIHZhcmnDom5jaWEgZG9zIGRvaXMgZ3J1cG9zLg0KDQpgYGB7cn0NCmJ1c2NhcyAlPiUNCiAgZ3JvdXBfYnkoZ3JvdXAsIHNlc3Npb25fc3RhcnRfZGF0ZSkgJT4lDQogIG11dGF0ZShjbGlja3NfcGVyX2RheSA9IHN1bShudW1fY2xpY2tzKSkgJT4lDQogIGdncGxvdChtYXBwaW5nID0gYWVzKHg9c2Vzc2lvbl9zdGFydF9kYXRlLCB5PWNsaWNrc19wZXJfZGF5LCBjb2xvciA9IGdyb3VwKSkgKw0KICBnZW9tX3BvaW50KCkgKw0KICBnZW9tX2xpbmUoKSArDQogIGZhY2V0X2dyaWQoZ3JvdXAgfiAuKSArDQogIGxhYnModGl0bGUgPSAiY2xpY2tzX3Blcl9kYXkiKQ0KDQpidXNjYXMgJT4lIA0KICAgIGdyb3VwX2J5KGdyb3VwKSAlPiUNCiAgICBzdW1tYXJpc2UodmFyaWFuY2lhID0gdmFyKHNlc3Npb25fbGVuZ3RoLCByZXN1bHRzKSkNCmBgYA0KDQojIDIuIFdoaWNoIHJlc3VsdHMgZG8gcGVvcGxlIHRlbmQgdG8gdHJ5IGZpcnN0PyBIb3cgZG9lcyBpdCBjaGFuZ2UgZGF5LXRvLWRheT8NCg0KQW5hbGlzYW5kbyBhIGRpc3RyaWJ1acOnw6NvIGRvcyBwcmltZWlyb3MgY2xpY2tzLCBvYnNlcnZhbW9zIHF1ZSBlc3NhIHRlbSBtYWlvciBjb25jZW50cmHDp8OjbyBubyB2YWxvciAxIGUgZGVwb2lzIHZhaSBkZWNyZXNjZW5kbyBwcm9wb3JjaW9uYWxtZW50ZSwgb3Ugc2VqYSwgYXMgcGVzc29hcyBjbGljYW0gbWFpcyBub3MgcHJpbWVpcm9zIGxpbmtzIGRlIHJldG9ybm8gZGUgdW1hIHBlc3F1aXNhLg0KDQpgYGB7cn0NCmJ1c2NhcyAlPiUgDQogIGZpbHRlcighaXMubmEoZmlyc3RfY2xpY2spKSAlPiUgDQogIGdyb3VwX2J5KGZpcnN0X2NsaWNrKSAlPiUgDQogIGZpbHRlcihuKCkgPj0gMjApICU+JQ0KICBnZ3Bsb3QoYWVzKHg9Zmlyc3RfY2xpY2spKSArDQogIGdlb21fZGVuc2l0eShmaWxsID0gImdyZWVuIiwgY29sb3I9ICJyZWQiLCBhbHBoYSA9IC40KQ0KYGBgDQoNCkFuYWxpc2FuZG8gYSBkaXN0cmlidWnDp8OjbyBkb3MgcHJpbWVpcm9zIGNsaWNrcywgbWFzIGRlc3NhIHZleiBhZ3J1cGFuZG8gcG9yIGRpYSwgb2JzZXJ2YW1vcyBxdWUgdG9kb3Mgb3MgZGlhcyB0ZW0gdW1hIGRpc3RyaWJ1acOnw6NvIG11aXRvIHBhcmVjaWRhLCBjb20gY29uY2VudHJhw6fDo28gZW0gdG9ybm8gZGUgMS4gRW50w6NvLCBhbyBjYWxjdWxhciBvIHBlcmNlbnR1YWwgZGUgY2xpY2tzIGRpZmVyZW50ZXMgZGUgMSBwZXJjZWJlbW9zIHF1ZSBvcyBncnVwb3Mgc2UgZGlmZXJlbmNpYW0gdW0gcG91Y28sIDIwMTYtMDMtMDUgZm9pIG8gZGlhIGVtIHF1ZSBtYWlzIHNlIGNsaWNvdSBlbSBsaW5rcyBxdWUgbsOjbyBmb3NzZW0gbyBwcmltZWlybywgY29tIHF1YXNlIDIlIGEgbWFpcyBkbyBxdWUgbyBkaWEgMjAxNi0wMy0wMSBlbSBxdWUgbWVub3Mgc2UgY2xpY291LiAgIA0KDQpgYGB7cn0NCmJ1c2NhcyAlPiUgDQogIGZpbHRlcighaXMubmEoZmlyc3RfY2xpY2spKSAlPiUgDQogIGdyb3VwX2J5KGZpcnN0X2NsaWNrLCBzZXNzaW9uX3N0YXJ0X2RhdGUpICU+JSANCiAgc3VtbWFyaXNlKG51bV9jbGlja3MgPSBuKCkpICU+JQ0KICBmaWx0ZXIobnVtX2NsaWNrcyA+PSAyMCkgJT4lDQogIGdncGxvdChhZXMoeD1maXJzdF9jbGljaywgeSA9IG51bV9jbGlja3MpKSArDQogIGZhY2V0X3dyYXAofnNlc3Npb25fc3RhcnRfZGF0ZSkgKw0KICBnZW9tX2FyZWEoZmlsbCA9ICJncmVlbiIsIGNvbG9yID0gInJlZCIpDQoNCg0KcGVyY2VudF9maXJzdF9jbGljayA8LSBidXNjYXMgJT4lIA0KICAgICAgICAgICAgICAgICAgICAgICAgZmlsdGVyKCFpcy5uYShmaXJzdF9jbGljaykpICU+JSANCiAgICAgICAgICAgICAgICAgICAgICAgIGdyb3VwX2J5KHNlc3Npb25fc3RhcnRfZGF0ZSwgZmlyc3RfY2xpY2spICU+JQ0KICAgICAgICAgICAgICAgICAgICAgICAgc3VtbWFyaXNlKHRvdGFsID0gbigpKQ0KDQpwZXJjZW50X2ZpcnN0X2NsaWNrIDwtIHBlcmNlbnRfZmlyc3RfY2xpY2sgJT4lDQogIGdyb3VwX2J5KHNlc3Npb25fc3RhcnRfZGF0ZSkgJT4lDQogIG11dGF0ZShwZXJjZW50ID0gdG90YWwgLyBzdW0odG90YWwpKQ0KDQpwZXJjZW50X2ZpcnN0X2NsaWNrIDwtIHBlcmNlbnRfZmlyc3RfY2xpY2sgJT4lDQogIGZpbHRlcihmaXJzdF9jbGljayAhPSAxKSAlPiUNCiAgc2VsZWN0KHNlc3Npb25fc3RhcnRfZGF0ZSwgcGVyY2VudCkgJT4lDQogIGdyb3VwX2J5KHNlc3Npb25fc3RhcnRfZGF0ZSkgJT4lDQogIG11dGF0ZShwZXJjZW50X2RpZl8xID0gc3VtKHBlcmNlbnQpICogMTAwKSAlPiUNCiAgc2VsZWN0KC1wZXJjZW50KQ0KDQpwZXJjZW50X2ZpcnN0X2NsaWNrWyFkdXBsaWNhdGVkKHBlcmNlbnRfZmlyc3RfY2xpY2tbYygic2Vzc2lvbl9zdGFydF9kYXRlIiwicGVyY2VudF9kaWZfMSIpXSksXQ0KYGBgDQoNCiMgMy4gV2hhdCBpcyBvdXIgZGFpbHkgb3ZlcmFsbCB6ZXJvIHJlc3VsdHMgcmF0ZT8gSG93IGRvZXMgaXQgdmFyeSBiZXR3ZWVuIHRoZSBncm91cHM/DQoNCk8gZ3LDoWZpY28gZXhpYmUgYSBkaXN0cmlidWnDp8OjbyBkZSBwZXNxdWlzYXMgcXVlIHJldG9ybmFyYW0gMCByZXN1bHRhZG9zIGFvIGxvbmdvIGRvIHBlcmlvZG8gZGUgY29sZXRhLiBPIGRpYSA1IMOpIG8gZGlhIGNvbSBtZW5vciBxdWFudGlkYWRlIGRlIHJlc3VsdGFkb3MgMCwgbWFzIGNvbW8gdmlzdG8gbmEgcXVlc3TDo28gMSB0YW1iw6ltIMOpIG8gZGlhIGNvbSBtZW5vcyBxdWFudGlkYWRlIGRlIGNsaWNrcy4NCg0KYGBge3J9DQpidXNjYXMgJT4lIA0KICAgIGZpbHRlcihyZXN1bHRzID09IDApICU+JSANCiAgICBncm91cF9ieShzZXNzaW9uX3N0YXJ0X2RhdGUpICU+JSANCiAgICBtdXRhdGUobnVtX3Jlc3VsdHNfMCA9IG4oKSkgJT4lIA0KICAgIGdncGxvdChhZXMoeCA9IHNlc3Npb25fc3RhcnRfZGF0ZSwgeSA9IG51bV9yZXN1bHRzXzApKSArDQogICAgZ2VvbV9saW5lKCkgKw0KICAgIGxhYnMoDQogICAgICB0aXRsZSA9ICJEaXN0cmlidXRpb24gb2YgcmVzdWx0cyAwIiwNCiAgICAgIHggPSAiRGF0ZSIsDQogICAgICB5ID0gIk51bWJlciBvZiByZXN1bHRzIDAiDQogICAgKQ0KYGBgDQoNCk8gZ3LDoWZpY28gZXhpYmUgYSBtZXNtYSByZWxhw6fDo28gZG8gYW50ZXJpb3IsIHBvcsOpbSBkZXNzYSB2ZXogY29uc2lkZXJhbmRvIG9zIGdydXBvcy4gTyBncnVwbyBCIGFwcmVzZW50YSBtZW5vcyByZXN1bHRhZG9zIDAgZG8gcXVlIG8gZ3J1cG8gQSBlIHRhbWLDqW0gdW1hIG1haW9yIHJlZ3VsYXJpZGFkZSwgb3Ugc2VqYSwgYSBtZXNtYSByZWxhw6fDo28gb2JzZXJ2YWRhIHBhcmEgbyBuw7ptZXJvIGRlIGNsaWNrcy4NCg0KYGBge3J9DQpidXNjYXMgJT4lIA0KICAgIGZpbHRlcihyZXN1bHRzID09IDApICU+JSANCiAgICBncm91cF9ieShncm91cCwgc2Vzc2lvbl9zdGFydF9kYXRlKSAlPiUgDQogICAgbXV0YXRlKG51bV9yZXN1bHRzXzAgPSBuKCkpICU+JSANCiAgICBnZ3Bsb3QoYWVzKHggPSBzZXNzaW9uX3N0YXJ0X2RhdGUsIHkgPSBudW1fcmVzdWx0c18wLCBjb2xvciA9IGdyb3VwKSkgKw0KICAgIGdlb21fbGluZSgpICsNCiAgICBmYWNldF9ncmlkKGdyb3VwIH4gLikgKw0KICAgIGxhYnMoDQogICAgICB0aXRsZSA9ICJEaXN0cmlidXRpb24gb2YgcmVzdWx0cyAwIiwNCiAgICAgIHggPSAiRGF0ZSIsDQogICAgICB5ID0gIk51bWJlciBvZiByZXN1bHRzIDAiDQogICAgKQ0KYGBgDQoNCg0KIyA0LiBMZXQgc2Vzc2lvbiBsZW5ndGggYmUgYXBwcm94aW1hdGVseSB0aGUgdGltZSBiZXR3ZWVuIHRoZSBmaXJzdCBldmVudCBhbmQgdGhlIGxhc3QgZXZlbnQgaW4gYSBzZXNzaW9uLiBDaG9vc2UgYSB2YXJpYWJsZSBmcm9tIHRoZSBkYXRhc2V0IGFuZCBkZXNjcmliZSBpdHMgcmVsYXRpb25zaGlwIHRvIHNlc3Npb24gbGVuZ3RoLiBWaXN1YWxpemUgdGhlIHJlbGF0aW9uc2hpcC4NCg0KQXF1aSBvYnNlcnZhbW9zIGEgcmVsYcOnw6NvIGVudHJlIG8gdGFtYW5obyBkYSBzZXNzw6NvIGUgbyB2YWxvciBkZSByZXN1bHRhZG9zIHJldG9ybmFkb3MgZW0gcGVzcXVpc2FhcyBkZW50cm8gZGVzc2EgbWVzbWEgc2Vzc8Ojby4gTsOjbyDDqSBwb3Nzw612ZWwgcGVyY2ViZXIgdW1hIGNvcnJlw6fDo28gZW50cmUgYXMgdmFyacOhdmVpcywgZGUgcXVhbHF1ZXIgbmF0dXJlemEuIEFvIGVzdGltYXIgYXMgY29ycmVsYcOnw7VlcyBuZW5odW1hIHJldG9ybmEgdW0gdmFsb3Igc2lnbmlmaWNhbnRlLCBvdSBzZWphLCBuw6NvIGV4aXN0ZSBjb3JyZWxhw6fDo28gbGluZWFyIG91IG7Do28gbGluZWFyIGVudHJlIGFzIHZhcmnDoXZlaXMuIEVtIG91dHJhcyBwYWxhdnJhcyBvIHRhbWFuaG8gZGEgc2Vzc8OjbyBlIGUgbyB2YWxvciBkZSByZXN1bHRhZG9zIHJldG9ybmFkb3MgZW0gcGVzcXVpc2FhcyBkZW50cm8gZGVzc2EgbWVzbWEgc2Vzc8OjbyBzw6NvIHZhcmnDoXZlaXMgaW5kZXBlbmRlbnRlcyBlbnRyZSBzaS4NCg0KYGBge3J9DQpidXNjYXMgJT4lDQogICAgZmlsdGVyKHNlc3Npb25fbGVuZ3RoIDw9IDEwMDAwMCkgJT4lDQogICAgZ2dwbG90KGFlcyh4ID0gc2Vzc2lvbl9sZW5ndGgsIHkgPSByZXN1bHRzKSkgKw0KICAgIGdlb21fcG9pbnQoKQ0KDQpidXNjYXMgJT4lIA0KICAgIGZpbHRlcihzZXNzaW9uX2xlbmd0aCA8PSAxMDAwMDApICU+JQ0KICAgIHN1bW1hcmlzZShwZWFyc29uID0gY29yKHNlc3Npb25fbGVuZ3RoLCByZXN1bHRzLCBtZXRob2QgPSAicGVhcnNvbiIpLCANCiAgICAgICAgc3BlYXJtYW4gPSBjb3Ioc2Vzc2lvbl9sZW5ndGgsIHJlc3VsdHMsIG1ldGhvZCA9ICJzcGVhcm1hbiIpLCANCiAgICAgICAga2VuZGFsbCA9IGNvcihzZXNzaW9uX2xlbmd0aCwgcmVzdWx0cywgbWV0aG9kID0gImtlbmRhbGwiKSkNCg0KYGBgDQo=