Séries Temporais: cross-validation com pacote fable
Adriano Marcos Rodrigues Figueiredo, *e-mail: adriano.figueiredo@ufms.br*
26 julho 2021
Abstract
The example is a simplified exercise using cross-validation in time series with the fable package.Licença
This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
License: CC BY-SA 4.0
Citação
Sugestão de citação: FIGUEIREDO, Adriano Marcos Rodrigues. Séries Temporais: cross-validation com pacote fable. Campo Grande-MS,Brasil: RStudio/Rpubs, 2021. Disponível em http://rpubs.com/amrofi/cross_validation_fable.
1 Introdução
Usarei o pacote do fpp3 para exemplo com o dataset de produção trimestral de cerveja Australiana. Instruções interessantes estão em <https://robjhyndman.com/hyndsight/tscv-fable/>, de Hyndman (2021).
2 Dados e pacote
O dataset é trimestral do primeiro trimestre de 1956 (1956Q1) ao segundo trimestre de 2010 (2010Q2). O pacote fpp3 automaticamente carrega os pacotes principais para este exemplo, como o tsibble, feasts e fable. Primeiro vou gerar o aus_production para olhar a série original. A série original tem 218 observações e 7 colunas sendo a primeira para identificação do trimestre.
library(fpp3)
beer <- aus_production
beerQuarter <qtr> | Beer <dbl> | Tobacco <dbl> | Bricks <dbl> | Cement <dbl> | Electricity <dbl> | Gas <dbl> |
|---|---|---|---|---|---|---|
| 1956 Q1 | 284 | 5225 | 189 | 465 | 3923 | 5 |
| 1956 Q2 | 213 | 5178 | 204 | 532 | 4436 | 6 |
| 1956 Q3 | 227 | 5297 | 208 | 561 | 4806 | 7 |
| 1956 Q4 | 308 | 5681 | 197 | 570 | 4418 | 6 |
| 1957 Q1 | 262 | 5577 | 187 | 529 | 4339 | 5 |
| 1957 Q2 | 228 | 5651 | 214 | 604 | 4811 | 7 |
| 1957 Q3 | 236 | 5317 | 227 | 603 | 5259 | 7 |
| 1957 Q4 | 320 | 6152 | 222 | 582 | 4735 | 6 |
| 1958 Q1 | 272 | 5758 | 199 | 554 | 4608 | 5 |
| 1958 Q2 | 233 | 5641 | 229 | 620 | 5196 | 7 |
names(beer)[1] "Quarter" "Beer" "Tobacco" "Bricks" "Cement"
[6] "Electricity" "Gas" Segundo, criamos um objeto de dados contendo muitos conjuntos de treinamento começando com 3 anos (12 observações ou trimestres) e adicionando um trimestre de cada vez até que todos os dados sejam incluídos. Ou seja, o conjunto .id=2017 irá até 2010Q2 .
Observe que o objeto beer agora terá 207 conjuntos de treinamento de tamanho crescente, totalizando 23805 observações para 3 colunas (selecionamos apenas a variável Beer), outra é a coluna indicadora do trimestre (Quarter) e é criada a coluna .id para indicar a subamostra de conjunto de treinamento.
library(fpp3)
beer <- aus_production %>%
select(Beer) %>%
stretch_tsibble(.init = 12, .step = 1)
beerBeer <dbl> | Quarter <qtr> | .id <int> | ||
|---|---|---|---|---|
| 284 | 1956 Q1 | 1 | ||
| 213 | 1956 Q2 | 1 | ||
| 227 | 1956 Q3 | 1 | ||
| 308 | 1956 Q4 | 1 | ||
| 262 | 1957 Q1 | 1 | ||
| 228 | 1957 Q2 | 1 | ||
| 236 | 1957 Q3 | 1 | ||
| 320 | 1957 Q4 | 1 | ||
| 272 | 1958 Q1 | 1 | ||
| 233 | 1958 Q2 | 1 |
names(beer)[1] "Beer" "Quarter" ".id" 3 Estimação
Ajustamos, seguindo Hyndman (2021), um modelo ETS a cada conjunto de treinamento e produzimos um ano de previsões (4 previsões) para cada modelo. Como desejo calcular o RMSE para cada horizonte de previsão, adicionarei o horizonte h ao objeto fc resultante. O tempo de estimação dependerá da sua capacidade de processamento de máquina.
fc <- beer %>%
model(ETS(Beer)) %>%
forecast(h = "1 year") %>%
group_by(.id) %>%
mutate(h = row_number()) %>%
ungroup()
fc.id <int> | .model <chr> | Quarter <qtr> | Beer <dist> | .mean <dbl> | h <int> |
|---|---|---|---|---|---|
| 1 | ETS(Beer) | 1959 Q1 | <dist> | 272.8851 | 1 |
| 1 | ETS(Beer) | 1959 Q2 | <dist> | 224.8880 | 2 |
| 1 | ETS(Beer) | 1959 Q3 | <dist> | 232.9877 | 3 |
| 1 | ETS(Beer) | 1959 Q4 | <dist> | 313.6845 | 4 |
| 2 | ETS(Beer) | 1959 Q2 | <dist> | 226.2297 | 1 |
| 2 | ETS(Beer) | 1959 Q3 | <dist> | 235.9480 | 2 |
| 2 | ETS(Beer) | 1959 Q4 | <dist> | 314.8028 | 3 |
| 2 | ETS(Beer) | 1960 Q1 | <dist> | 269.2523 | 4 |
| 3 | ETS(Beer) | 1959 Q3 | <dist> | 233.8160 | 1 |
| 3 | ETS(Beer) | 1959 Q4 | <dist> | 313.4209 | 2 |
O exame do objeto fc permitirá verificar que este tem 828 linhas, ou seja, quatro previsões para cada um dos 207 conjuntos de treinamento, iniciando cada previsão ao término de cada conjunto. A coluna .model identifica o modelo ETS solicitado; a coluna Beer mostra a distribuição associada a cada estimação; a estimativa média está na coluna .mean; e a coluna h identifica cada horizonte de previsão de cada linha.
Por fim, comparamos as previsões com os valores reais e a média nas subamostras. As previsões de 1 e 2 trimestres à frente têm aproximadamente a mesma precisão aqui, mas então o erro aumenta para os horizontes 3 e 4.
fc %>%
accuracy(aus_production, by = "h") %>%
select(h, RMSE)h <int> | RMSE <dbl> | |||
|---|---|---|---|---|
| 1 | 17.05099 | |||
| 2 | 16.68515 | |||
| 3 | 18.14166 | |||
| 4 | 19.16382 |
Podemos também olhar a acurácia por subamostra e plotar.
fc %>%
accuracy(aus_production, by = ".id") %>%
select(.id, RMSE) -> tabela
tabela.id <int> | RMSE <dbl> | |||
|---|---|---|---|---|
| 1 | 10.431115 | |||
| 2 | 10.945103 | |||
| 3 | 11.526015 | |||
| 4 | 13.400453 | |||
| 5 | 13.968624 | |||
| 6 | 15.936552 | |||
| 7 | 16.056605 | |||
| 8 | 10.153741 | |||
| 9 | 18.970878 | |||
| 10 | 11.029045 |
library(ggplot2)
ggplot(tabela) + aes(x = .id, y = RMSE) + geom_line(size = 1.15, colour = "#112446") +
labs(x = "Subconjunto de treinamento", y = "Valores de RMSE", title = "Valores de RMSE para cada subconjunto de treinamento",
subtitle = "Produção trimestral de cerveja Australiana", caption = "Fonte: Adaptado de Hyndman (2021).") +
ggthemes::theme_economist()
Conforme <https://math.unm.edu/~lil/Stat581/3b-accuracy.pdf>, uma boa utilização do cross-validation é aplicar em diferentes modelos (métodos) e usar o resultado da acurácia do cross-validation para escolher o melhor modelo para a série temporal utilizada.
Referências
Referências de pacotes utilizados - lista todos os pacotes que foram abertos em algum procedimento (mesmo que dentro de rotinas de outros pacotes). Ou seja, chamamos apenas o fpp3, mas este usa muitos outros!
Hyndman, Rob J. (2021). Time series cross-validation using fable. https://robjhyndman.com/hyndsight/tscv-fable/. Accessed on July, 26th, 2021.
Allaire, JJ, Yihui Xie, Jonathan McPherson, Javier Luraschi, Kevin Ushey, Aron Atkins, Hadley Wickham, Joe Cheng, Winston Chang, and Richard Iannone. 2021. Rmarkdown: Dynamic Documents for r. https://CRAN.R-project.org/package=rmarkdown.
Barnier, Julien. 2021. Rmdformats: HTML Output Formats and Templates for Rmarkdown Documents. https://github.com/juba/rmdformats.
Grolemund, Garrett, and Hadley Wickham. 2011. “Dates and Times Made Easy with lubridate.” Journal of Statistical Software 40 (3): 1–25. https://www.jstatsoft.org/v40/i03/.
Hyndman, Rob. 2021. Fpp3: Data for "Forecasting: Principles and Practice" (3rd Edition). https://CRAN.R-project.org/package=fpp3.
Müller, Kirill, and Hadley Wickham. 2021. Tibble: Simple Data Frames. https://CRAN.R-project.org/package=tibble.
O’Hara-Wild, Mitchell, Rob Hyndman, and Earo Wang. 2021a. Fable: Forecasting Models for Tidy Time Series. https://CRAN.R-project.org/package=fable.
———. 2021b. Fabletools: Core Tools for Packages in the Fable Framework. https://CRAN.R-project.org/package=fabletools.
———. 2021c. Feasts: Feature Extraction and Statistics for Time Series. https://CRAN.R-project.org/package=feasts.
———. 2021d. Tsibbledata: Diverse Datasets for Tsibble. https://CRAN.R-project.org/package=tsibbledata.
R Core Team. 2021. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Spinu, Vitalie, Garrett Grolemund, and Hadley Wickham. 2021. Lubridate: Make Dealing with Dates a Little Easier. https://CRAN.R-project.org/package=lubridate.
Wang, Earo, Dianne Cook, and Rob J Hyndman. 2020. “A New Tidy Data Structure to Support Exploration and Modeling of Temporal Data.” Journal of Computational and Graphical Statistics 29 (3): 466–78. https://doi.org/10.1080/10618600.2019.1695624.
Wang, Earo, Di Cook, Rob Hyndman, and Mitchell O’Hara-Wild. 2021. Tsibble: Tidy Temporal Data Frames and Tools. https://tsibble.tidyverts.org.
Wickham, Hadley. 2016. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. https://ggplot2.tidyverse.org.
———. 2021. Tidyr: Tidy Messy Data. https://CRAN.R-project.org/package=tidyr.
Wickham, Hadley, Winston Chang, Lionel Henry, Thomas Lin Pedersen, Kohske Takahashi, Claus Wilke, Kara Woo, Hiroaki Yutani, and Dewey Dunnington. 2021. Ggplot2: Create Elegant Data Visualisations Using the Grammar of Graphics. https://CRAN.R-project.org/package=ggplot2.
Wickham, Hadley, Romain François, Lionel Henry, and Kirill Müller. 2021. Dplyr: A Grammar of Data Manipulation. https://CRAN.R-project.org/package=dplyr.
Xie, Yihui. 2014. “Knitr: A Comprehensive Tool for Reproducible Research in R.” In Implementing Reproducible Computational Research, edited by Victoria Stodden, Friedrich Leisch, and Roger D. Peng. Chapman; Hall/CRC. http://www.crcpress.com/product/isbn/9781466561595.
———. 2015. Dynamic Documents with R and Knitr. 2nd ed. Boca Raton, Florida: Chapman; Hall/CRC. https://yihui.org/knitr/.
———. 2016. Bookdown: Authoring Books and Technical Documents with R Markdown. Boca Raton, Florida: Chapman; Hall/CRC. https://bookdown.org/yihui/bookdown.
———. 2021a. Bookdown: Authoring Books and Technical Documents with r Markdown. https://CRAN.R-project.org/package=bookdown.
———. 2021b. Knitr: A General-Purpose Package for Dynamic Report Generation in r. https://yihui.org/knitr/.
Xie, Yihui, J. J. Allaire, and Garrett Grolemund. 2018. R Markdown: The Definitive Guide. Boca Raton, Florida: Chapman; Hall/CRC. https://bookdown.org/yihui/rmarkdown.
Xie, Yihui, Christophe Dervieux, and Emily Riederer. 2020. R Markdown Cookbook. Boca Raton, Florida: Chapman; Hall/CRC. https://bookdown.org/yihui/rmarkdown-cookbook.
LS0tDQp0aXRsZTogIlPDqXJpZXMgVGVtcG9yYWlzOiBjcm9zcy12YWxpZGF0aW9uIGNvbSBwYWNvdGUgZmFibGUiDQphdXRob3I6ICJBZHJpYW5vIE1hcmNvcyBSb2RyaWd1ZXMgRmlndWVpcmVkbywgKmUtbWFpbDogYWRyaWFuby5maWd1ZWlyZWRvQHVmbXMuYnIqIg0KYWJzdHJhY3Q6IA0KICBUaGUgZXhhbXBsZSBpcyBhIHNpbXBsaWZpZWQgZXhlcmNpc2UgdXNpbmcgY3Jvc3MtdmFsaWRhdGlvbiBpbiB0aW1lIHNlcmllcyB3aXRoIHRoZSBmYWJsZSBwYWNrYWdlLiANCmRhdGU6ICJgciBmb3JtYXQoU3lzLkRhdGUoKSwgJyVkICVCICVZJylgIg0KYmlibGlvZ3JhcGh5OiBwYWNrYWdlcy5iaWINCm5vY2l0ZTogJ0AqJw0Kb3V0cHV0Og0KICBodG1sX2RvY3VtZW50Og0KICAgIGNvZGVfZG93bmxvYWQ6IHllcw0KICAgIHRoZW1lOiBkZWZhdWx0DQogICAgbnVtYmVyX3NlY3Rpb25zOiB5ZXMNCiAgICB0b2M6IHllcw0KICAgIHRvY19mbG9hdDogbm8NCiAgICBkZl9wcmludDogcGFnZWQNCiAgICBmaWdfY2FwdGlvbjogeWVzDQogIHBkZl9kb2N1bWVudDoNCiAgICB0b2M6IHllcw0KICB3b3JkX2RvY3VtZW50Og0KICAgIHRvYzogeWVzDQotLS0NCg0KYGBge3Iga25pdHJfaW5pdCwgZWNobz1GQUxTRSwgY2FjaGU9RkFMU0V9DQpsaWJyYXJ5KGtuaXRyKQ0KbGlicmFyeShybWFya2Rvd24pDQpsaWJyYXJ5KHJtZGZvcm1hdHMpDQoNCiMjIEdsb2JhbCBvcHRpb25zDQpvcHRpb25zKG1heC5wcmludD0iMTAwIikNCm9wdHNfY2h1bmskc2V0KGVjaG89VFJVRSwNCgkgICAgICAgICAgICAgY2FjaGU9VFJVRSwNCiAgICAgICAgICAgICAgIHByb21wdD1GQUxTRSwNCiAgICAgICAgICAgICAgIHRpZHk9VFJVRSwNCiAgICAgICAgICAgICAgIGNvbW1lbnQ9TkEsDQogICAgICAgICAgICAgICBtZXNzYWdlPUZBTFNFLA0KICAgICAgICAgICAgICAgd2FybmluZz1GQUxTRSwNCiAgICAgICAgICAgICAgIG91dC53aWR0aD03NTAsIA0KICAgICAgICAgICAgICAgZmlnLmhlaWdodD04LCANCiAgICAgICAgICAgICAgIGZpZy53aWR0aD04KQ0Kb3B0c19rbml0JHNldCh3aWR0aD0xMDApDQpgYGANCg0KIyBMaWNlbsOnYSB7I0xpY2Vuw6dhIC51bm51bWJlcmVkfQ0KDQpUaGlzIHdvcmsgaXMgbGljZW5zZWQgdW5kZXIgdGhlIENyZWF0aXZlIENvbW1vbnMgQXR0cmlidXRpb24tU2hhcmVBbGlrZSA0LjAgSW50ZXJuYXRpb25hbCBMaWNlbnNlLiBUbyB2aWV3IGEgY29weSBvZiB0aGlzIGxpY2Vuc2UsIHZpc2l0IDxodHRwOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS1zYS80LjAvPiBvciBzZW5kIGEgbGV0dGVyIHRvIENyZWF0aXZlIENvbW1vbnMsIFBPIEJveCAxODY2LCBNb3VudGFpbiBWaWV3LCBDQSA5NDA0MiwgVVNBLg0KDQohW0xpY2Vuc2U6IENDIEJZLVNBIDQuMF0oaHR0cHM6Ly9taXJyb3JzLmNyZWF0aXZlY29tbW9ucy5vcmcvcHJlc3NraXQvYnV0dG9ucy84OHgzMS9wbmcvYnktc2EucG5nKXt3aWR0aD0iMjUlIn0NCg0KIyBDaXRhw6fDo28geyNDaXRhw6fDo28gLnVubnVtYmVyZWR9DQoNClN1Z2VzdMOjbyBkZSBjaXRhw6fDo286IEZJR1VFSVJFRE8sIEFkcmlhbm8gTWFyY29zIFJvZHJpZ3Vlcy4gU8OpcmllcyBUZW1wb3JhaXM6IGNyb3NzLXZhbGlkYXRpb24gY29tIHBhY290ZSBgZmFibGVgLiBDYW1wbyBHcmFuZGUtTVMsQnJhc2lsOiBSU3R1ZGlvL1JwdWJzLCAyMDIxLiBEaXNwb27DrXZlbCBlbSA8aHR0cDovL3JwdWJzLmNvbS9hbXJvZmkvY3Jvc3NfdmFsaWRhdGlvbl9mYWJsZT4uDQoNCiMgSW50cm9kdcOnw6NvDQoNClVzYXJlaSBvIHBhY290ZSBkbyBgZnBwM2AgcGFyYSBleGVtcGxvIGNvbSBvIGRhdGFzZXQgZGUgcHJvZHXDp8OjbyB0cmltZXN0cmFsIGRlIGNlcnZlamEgQXVzdHJhbGlhbmEuIEluc3RydcOnw7VlcyBpbnRlcmVzc2FudGVzIGVzdMOjbyBlbSBcPDxodHRwczovL3JvYmpoeW5kbWFuLmNvbS9oeW5kc2lnaHQvdHNjdi1mYWJsZS8+XD4sIGRlIEh5bmRtYW4gKDIwMjEpLg0KDQojIERhZG9zIGUgcGFjb3RlDQoNCk8gZGF0YXNldCDDqSB0cmltZXN0cmFsIGRvIHByaW1laXJvIHRyaW1lc3RyZSBkZSAxOTU2ICgxOTU2UTEpIGFvIHNlZ3VuZG8gdHJpbWVzdHJlIGRlIDIwMTAgKDIwMTBRMikuIE8gcGFjb3RlIGBmcHAzYCBhdXRvbWF0aWNhbWVudGUgY2FycmVnYSBvcyBwYWNvdGVzIHByaW5jaXBhaXMgcGFyYSBlc3RlIGV4ZW1wbG8sIGNvbW8gbyBgdHNpYmJsZWAsIGBmZWFzdHNgIGUgYGZhYmxlYC4gUHJpbWVpcm8gdm91IGdlcmFyIG8gYXVzX3Byb2R1Y3Rpb24gcGFyYSBvbGhhciBhIHPDqXJpZSBvcmlnaW5hbC4gQSBzw6lyaWUgb3JpZ2luYWwgdGVtIDIxOCBvYnNlcnZhw6fDtWVzIGUgNyBjb2x1bmFzIHNlbmRvIGEgcHJpbWVpcmEgcGFyYSBpZGVudGlmaWNhw6fDo28gZG8gdHJpbWVzdHJlLg0KDQpgYGB7ciBkYWRvc30NCmxpYnJhcnkoZnBwMykNCmJlZXIgPC0gYXVzX3Byb2R1Y3Rpb24gDQpiZWVyDQpuYW1lcyhiZWVyKQ0KYGBgDQoNClNlZ3VuZG8sIGNyaWFtb3MgdW0gb2JqZXRvIGRlIGRhZG9zIGNvbnRlbmRvIG11aXRvcyBjb25qdW50b3MgZGUgdHJlaW5hbWVudG8gY29tZcOnYW5kbyBjb20gMyBhbm9zICgxMiBvYnNlcnZhw6fDtWVzIG91IHRyaW1lc3RyZXMpIGUgYWRpY2lvbmFuZG8gdW0gdHJpbWVzdHJlIGRlIGNhZGEgdmV6IGF0w6kgcXVlIHRvZG9zIG9zIGRhZG9zIHNlamFtIGluY2x1w61kb3MuIE91IHNlamEsIG8gY29uanVudG8gYC5pZD0yMDE3YCBpcsOhIGF0w6kgYDIwMTBRMmAgLg0KDQpPYnNlcnZlIHF1ZSBvIG9iamV0byBiZWVyIGFnb3JhIHRlcsOhIDIwNyBjb25qdW50b3MgZGUgdHJlaW5hbWVudG8gZGUgdGFtYW5obyBjcmVzY2VudGUsIHRvdGFsaXphbmRvIDIzODA1IG9ic2VydmHDp8O1ZXMgcGFyYSAzIGNvbHVuYXMgKHNlbGVjaW9uYW1vcyBhcGVuYXMgYSB2YXJpw6F2ZWwgYEJlZXJgKSwgb3V0cmEgw6kgYSBjb2x1bmEgaW5kaWNhZG9yYSBkbyB0cmltZXN0cmUgKGBRdWFydGVyYCkgZSDDqSBjcmlhZGEgYSBjb2x1bmEgYC5pZGAgcGFyYSBpbmRpY2FyIGEgc3ViYW1vc3RyYSBkZSBjb25qdW50byBkZSB0cmVpbmFtZW50by4NCg0KYGBge3IgZGFkb3MyfQ0KbGlicmFyeShmcHAzKQ0KYmVlciA8LSBhdXNfcHJvZHVjdGlvbiAlPiUNCiAgc2VsZWN0KEJlZXIpICU+JQ0KICBzdHJldGNoX3RzaWJibGUoLmluaXQgPSAxMiwgLnN0ZXA9MSkNCmJlZXINCm5hbWVzKGJlZXIpDQpgYGANCg0KIyBFc3RpbWHDp8Ojbw0KDQpBanVzdGFtb3MsIHNlZ3VpbmRvIEh5bmRtYW4gKDIwMjEpLCB1bSBtb2RlbG8gYEVUU2AgYSBjYWRhIGNvbmp1bnRvIGRlIHRyZWluYW1lbnRvIGUgcHJvZHV6aW1vcyB1bSBhbm8gZGUgcHJldmlzw7VlcyAoNCBwcmV2aXPDtWVzKSBwYXJhIGNhZGEgbW9kZWxvLiBDb21vIGRlc2VqbyBjYWxjdWxhciBvIGBSTVNFYCBwYXJhIGNhZGEgaG9yaXpvbnRlIGRlIHByZXZpc8OjbywgYWRpY2lvbmFyZWkgbyBob3Jpem9udGUgYGhgIGFvIG9iamV0byBgZmNgIHJlc3VsdGFudGUuIE8gdGVtcG8gZGUgZXN0aW1hw6fDo28gZGVwZW5kZXLDoSBkYSBzdWEgY2FwYWNpZGFkZSBkZSBwcm9jZXNzYW1lbnRvIGRlIG3DoXF1aW5hLg0KDQpgYGB7ciBlc3RpbWFjYW99DQpmYyA8LSBiZWVyICU+JQ0KICBtb2RlbChFVFMoQmVlcikpICU+JQ0KICBmb3JlY2FzdChoID0gIjEgeWVhciIpICU+JQ0KICBncm91cF9ieSguaWQpICU+JQ0KICBtdXRhdGUoaCA9IHJvd19udW1iZXIoKSkgJT4lDQogIHVuZ3JvdXAoKQ0KZmMNCmBgYA0KDQpPIGV4YW1lIGRvIG9iamV0byBmYyBwZXJtaXRpcsOhIHZlcmlmaWNhciBxdWUgZXN0ZSB0ZW0gODI4IGxpbmhhcywgb3Ugc2VqYSwgcXVhdHJvIHByZXZpc8O1ZXMgcGFyYSBjYWRhIHVtIGRvcyAyMDcgY29uanVudG9zIGRlIHRyZWluYW1lbnRvLCBpbmljaWFuZG8gY2FkYSBwcmV2aXPDo28gYW8gdMOpcm1pbm8gZGUgY2FkYSBjb25qdW50by4gQSBjb2x1bmEgYC5tb2RlbGAgaWRlbnRpZmljYSBvIG1vZGVsbyBFVFMgc29saWNpdGFkbzsgYSBjb2x1bmEgYEJlZXJgIG1vc3RyYSBhIGRpc3RyaWJ1acOnw6NvIGFzc29jaWFkYSBhIGNhZGEgZXN0aW1hw6fDo287IGEgZXN0aW1hdGl2YSBtw6lkaWEgZXN0w6EgbmEgY29sdW5hIGAubWVhbmA7IGUgYSBjb2x1bmEgYGhgIGlkZW50aWZpY2EgY2FkYSBob3Jpem9udGUgZGUgcHJldmlzw6NvIGRlIGNhZGEgbGluaGEuDQoNClBvciBmaW0sIGNvbXBhcmFtb3MgYXMgcHJldmlzw7VlcyBjb20gb3MgdmFsb3JlcyByZWFpcyBlIGEgbcOpZGlhIG5hcyBzdWJhbW9zdHJhcy4gQXMgcHJldmlzw7VlcyBkZSAxIGUgMiB0cmltZXN0cmVzIMOgIGZyZW50ZSB0w6ptIGFwcm94aW1hZGFtZW50ZSBhIG1lc21hIHByZWNpc8OjbyBhcXVpLCBtYXMgZW50w6NvIG8gZXJybyBhdW1lbnRhIHBhcmEgb3MgaG9yaXpvbnRlcyAzIGUgNC4NCg0KYGBge3IgYWN1cmFjaWFfYnloLHdhcm5pbmc9RkFMU0UsY29tbWVudD1GQUxTRX0NCmZjICU+JQ0KICBhY2N1cmFjeShhdXNfcHJvZHVjdGlvbiwgYnk9ImgiKSAlPiUNCiAgc2VsZWN0KGgsIFJNU0UpDQpgYGANCg0KUG9kZW1vcyB0YW1iw6ltIG9saGFyIGEgYWN1csOhY2lhIHBvciBzdWJhbW9zdHJhIGUgcGxvdGFyLg0KDQpgYGB7ciBhY3VyYWNpYV9ieWlkLHdhcm5pbmc9RkFMU0UsY29tbWVudD1GQUxTRX0NCmZjICU+JQ0KICBhY2N1cmFjeShhdXNfcHJvZHVjdGlvbiwgYnk9Ii5pZCIpICU+JQ0KICBzZWxlY3QoLmlkLCBSTVNFKS0+dGFiZWxhDQp0YWJlbGENCmxpYnJhcnkoZ2dwbG90MikNCg0KZ2dwbG90KHRhYmVsYSkgKw0KIGFlcyh4ID0gLmlkLCB5ID0gUk1TRSkgKw0KIGdlb21fbGluZShzaXplID0gMS4xNSwgY29sb3VyID0gIiMxMTI0NDYiKSArDQogbGFicyh4ID0gIlN1YmNvbmp1bnRvIGRlIHRyZWluYW1lbnRvIiwgDQogeSA9ICJWYWxvcmVzIGRlIFJNU0UiLCB0aXRsZSA9ICJWYWxvcmVzIGRlIFJNU0UgcGFyYSBjYWRhIHN1YmNvbmp1bnRvIGRlIHRyZWluYW1lbnRvIiwgc3VidGl0bGUgPSAiUHJvZHXDp8OjbyB0cmltZXN0cmFsIGRlIGNlcnZlamEgQXVzdHJhbGlhbmEiLCANCiBjYXB0aW9uID0gIkZvbnRlOiBBZGFwdGFkbyBkZSBIeW5kbWFuICgyMDIxKS4iKSArDQogZ2d0aGVtZXM6OnRoZW1lX2Vjb25vbWlzdCgpDQoNCmBgYA0KDQpDb25mb3JtZSBcPDxodHRwczovL21hdGgudW5tLmVkdS9+bGlsL1N0YXQ1ODEvM2ItYWNjdXJhY3kucGRmPlw+LCB1bWEgYm9hIHV0aWxpemHDp8OjbyBkbyBjcm9zcy12YWxpZGF0aW9uIMOpIGFwbGljYXIgZW0gZGlmZXJlbnRlcyBtb2RlbG9zIChtw6l0b2RvcykgZSB1c2FyIG8gcmVzdWx0YWRvIGRhIGFjdXLDoWNpYSBkbyBjcm9zcy12YWxpZGF0aW9uIHBhcmEgZXNjb2xoZXIgbyBtZWxob3IgbW9kZWxvIHBhcmEgYSBzw6lyaWUgdGVtcG9yYWwgdXRpbGl6YWRhLg0KDQojIFJlZmVyw6puY2lhcyB7I1JlZmVyw6puY2lhcyAudW5udW1iZXJlZH0NCg0KKipSZWZlcsOqbmNpYXMgZGUgcGFjb3RlcyB1dGlsaXphZG9zIC0qKiBsaXN0YSB0b2RvcyBvcyBwYWNvdGVzIHF1ZSBmb3JhbSBhYmVydG9zIGVtIGFsZ3VtIHByb2NlZGltZW50byAobWVzbW8gcXVlIGRlbnRybyBkZSByb3RpbmFzIGRlIG91dHJvcyBwYWNvdGVzKS4gT3Ugc2VqYSwgY2hhbWFtb3MgYXBlbmFzIG8gYGZwcDNgLCBtYXMgZXN0ZSB1c2EgbXVpdG9zIG91dHJvcyENCg0KSHluZG1hbiwgUm9iIEouICgyMDIxKS4gVGltZSBzZXJpZXMgY3Jvc3MtdmFsaWRhdGlvbiB1c2luZyBmYWJsZS4gPGh0dHBzOi8vcm9iamh5bmRtYW4uY29tL2h5bmRzaWdodC90c2N2LWZhYmxlLz4uIEFjY2Vzc2VkIG9uIEp1bHksIDI2dGgsIDIwMjEuDQoNCmBgYHtyLGluY2x1ZGU9RkFMU0V9DQprbml0cjo6d3JpdGVfYmliKGMoLnBhY2thZ2VzKCksICJib29rZG93biIpLCAicGFja2FnZXMuYmliIiwgd2lkdGggPSA2MCkNCmBgYA0K