Introduction
According to Nielsen, Marketing mix modeling (MMM) is the use of statistical analysis to estimate the past impact and predict the future impact of various marketing tactics on sales. These marketing tactics typically consist of controlling certain variables, often summarized as the four Ps of marketing: product, price, promotion and place(McCarthy 1978). Some examples of variables related to the product are newness of the product, differentiation from competitors, etc. Price refers to the product’s list price and incentives such as discounts and deals. Promotion refers to advertising and displays. Place refers to the delivery of the product measured by variables such as distribution, availability and shelf-space (Tellis 2006).
This article attempts to provide a mathematically rigorous introduction to MMM, and is targeted towards readers with some quantitative background, but no previous experience in marketing science or econometrics. We mainly consider the effect of price and promotion in this article as these factors are readily quantifiable. We begin by formulating the regression problem. We then incorporate certain stylized facts about the impact of price and promotion on sales into the model. We then show how this analysis can e performed in R using a well-known dataset.
Problem Formulation
A common methodology employed to study the effect of a response variable (sales, in this case) on certain explanatory variables (the four Ps) is multivariate linear regression. Let \(y_t\) denote the sales at time \(t\). Typically, sales volume in units sold is used and not revenue from sales. This is because the revenue from sales is dependent on the price, which is an explanatory variable. Let \(x_{it}\), \(i = 1, \ldots, K\) denote the \(K\) explanatory variables. These could include the price, measures of ad exposure, shelf placement, store location etc. To estimate the effect of each of the explanatory variables on the sales volume, we fit a regression model given by \[
\tag{1}
y_t = \beta_0 + \sum_{i = 1}^{K}\beta_ix_{it} + \varepsilon_t
\] where \(\beta_i\), \(i=0,\ldots,K\) denote the regression coefficients. Assuming the Gauss-Markov assumptions hold, the ordinary least squares (OLS) estimators for \(\beta_i\), denoted by \(\hat{\beta}_i\) are the best linear unbiased estimators. In practice, however, this is rarely the case since the process \(y_t\) is seldom stationary or independent. In a later section we see how we can adjust for this using a practical example.
There are several key stylized facts about the effect of advertising on sales
- Exposure to advertising today has effects on consumers’ purchasing decisions in the future (Carryover Effect)
- The effect of ad exposure on purchasing decisions decays with time (Carryover Effect)
- The effect of ad exposure on sales is subject to the law of diminishing returns (Shape Effect)
- Changing the level of ad exposure brings about a relative change in sales volumes (Shape Effect)
There are several other effects that can be taken into consideration like those of competition, content and advertising media(Tellis 2006). However, to avoid complicating the model, we focus on these effects for now. Let us look at these one by one and express them mathematically
Carryover Effects
Consider the model in \((1)\). Let us simplify the model, wherein the sales at time \(t\) can by explained by a constant base value and the additional effect of ad exposure at time \(t\), measured by a variable \(x_{t}\). This is denoted by \[
y_t = \beta_0 + \beta_1 x_t + \varepsilon_t
\] We can incorporate Effects 1 and 2 into the above model by incorporating the lagged effects of all previous ad exposures. This model is given by \[
y_t = \beta_0 + \beta_1 s_t + \varepsilon_t \\
s_t = x_t + \lambda s_{t-1}
\] where \(s_0\) is typically set to \(0\).
Shape Effects
To clearly express Effect 3, we rewrite the above equation as follows \[
y_t = \beta_0 + \beta_1 x_t + \lambda \beta_1 s_{t-1} + \varepsilon_t
\] The above expression implies that the rate of change of sales volumes with respect to ad exposure in the same time period is constant, i.e, \(\partial y_t/ \partial x_t = \beta_1\). However, Effect 3 implies that this rate tapers off as we increase ad exposure and therefore the following must hold true \[
\lim_{x_t \to \infty} \frac{\partial y_t}{\partial x_t} = 0
\] Several functions satisfy this property(Joy 2006), e.g, the log function and the logistic function. We pick the logistic function with parameter \(\alpha\) for its flexibility. The model now becomes \[
\tag{2}
y_t = \beta_0 + \beta_1 s_t + \varepsilon_t \\
s_t = \frac{1}{1+ e^{-\alpha x_t}} + \lambda s_{t-1}
\] Now, \((2)\) implies that the increase in sales from increase in ad exposure is independent of \(y_t\). In other words, it is absolute. This means that increasing ad exposure by one unit cause the sales to increase by a certain amount irrespective of the magnitude of \(y_t\). For example, assume \(\beta_0 = 0\), \(\beta_1 = 1\), \(\alpha = 1\), \(\lambda = 0\). Let us increase \(x_t\) to \(x'_t\). Assuming the increment is small enough, We can approximate the new sales value by \[
y'_t \approx y_t + \frac{\partial y_t}{\partial x_t}(x'_t - x_t)
\] If \((x'_t - x_t) = 0.1\), and \(x_t=1\), we have \(y'_t - y_t \approx 0.2\). Therefore, sales increase by 0.2 units regardless of whether \(y_t\) is 100 or 1000. In practice, we usually see that sales increase in a relative manner, i.e by a percentage rather than in absolute terms. To take this into account we employ a logarithmic transformation on the dependent variable and obtain \[
\log y_t = \beta_0 + \beta_1 s_t + \varepsilon_t \\
s_t = \frac{1}{1+ e^{-\alpha x_t}} + \lambda s_{t-1}
\] It is trivial to check that \({\partial y_t}/{\partial x_t}\) indeed depends on \(y_t\) after this transformation.
Impact of Price
In order to bring price into the model, we rely on the price elasticity of demand, a measure of responsiveness of the quantity of the good demanded to a change in its price. We can argue that sales volumes directly reflect the demand for the product, assuming that production can keep up with demand linearly. The price elasticity \(E\) is expressed as \[
E = \frac{\partial q_t/q_t}{\partial p_t /p_t}
\] where \(p_t\) is the price of the product at time \(t\) and \(q_t\) is the quantity of the product demanded. Assuming sales volumes are directly proportional to quantity demanded, we can write \(y_t = k q_t\), where \(k\) is a constant of proportionality. Using this in the above expression we have \[
\frac{\partial \log y_t}{\partial p_t} = kE \log p_t
\] We can readily merge \(kE\) into one term and include the above expression into the regression model. Finally, we have \[
\log y_t = \beta_0 + \beta_1 s_t + \beta_2 \log p_t +\varepsilon_t \\
s_t = \frac{1}{1+ e^{-\alpha x_t}} + \lambda s_{t-1}
\] We see that the model has 5 parameters that need to be estimated, and it is nonlinear in \(\lambda\) and \(\alpha\). In what follows we fix \(\lambda\) and \(\alpha\) to regain linearity in the parameter space. To fix \(\lambda\) we use the empirical observation that the half-life of an ad copy is approximately 2.5 weeks. This yields \(\lambda=0.25\). We will simply assume \(\alpha=2\), though in reality, this is calibrated more carefully.
An Example
We use the Sliced Cheese Dataset from the bayesm
package to illustrate how to fit the model.
library(bayesm)
data("cheese")
summary(cheese)
RETAILER VOLUME DISP PRICE
BALTI/WASH - SAFEWAY : 68 Min. : 231 Min. :0.00000 Min. :1.320
BALTI/WASH - SUPER FRESH : 68 1st Qu.: 1990 1st Qu.:0.00000 1st Qu.:2.457
BIRMINGHAM/MONTGOM - KROGER : 68 Median : 3408 Median :0.04737 Median :2.703
BOSTON - STAR MARKET : 68 Mean : 4771 Mean :0.11428 Mean :2.869
BUFFALO/ROCHESTER - TOPS MARKETS: 68 3rd Qu.: 5520 3rd Qu.:0.16600 3rd Qu.:3.203
BUFFALO/ROCHESTER - WEGMANS : 68 Max. :148109 Max. :1.00000 Max. :4.642
(Other) :5147
The data consists of weekly observations of the volume of sales of sliced cheese over different retailers in the United States. Here, VOLUME
corresponds to \(y_t\), DISP
to \(x_t\) and PRICE
to \(p_t\). We are not yet looking at hierarchical models, so we focus on one retailer, say Dominick’s in Chicago. We then apply the required transformations to obtain the regression dataset. these are shown below.
library(dplyr)
adstockTransform <- function(x){
stats::filter( 1/(1+exp(-2*x)), 0.25, method = "recursive")
}
mmm.data <- cheese %>% filter(RETAILER == 'CHICAGO - DOMINICK') %>%
select(-RETAILER) %>%
mutate(log.volume = log(VOLUME), log.price = log(PRICE), adstock= adstockTransform(DISP)) %>%
select(-VOLUME, -DISP, -PRICE)
After checking the ACF and PACF of log.volume
to ensure stationarity and independence, we fit a linear model
fit <- lm(log.volume ~ adstock + log.price, data=mmm.data)
summary(fit)
Call:
lm(formula = log.volume ~ adstock + log.price, data = mmm.data)
Residuals:
Min 1Q Median 3Q Max
-0.53274 -0.13505 -0.02216 0.08934 0.58237
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 11.6611 0.3029 38.50 <2e-16 ***
adstock 0.6719 0.2615 2.57 0.0128 *
log.price -2.9680 0.1442 -20.58 <2e-16 ***
---
Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1
Residual standard error: 0.2468 on 58 degrees of freedom
Multiple R-squared: 0.9016, Adjusted R-squared: 0.8982
F-statistic: 265.6 on 2 and 58 DF, p-value: < 2.2e-16
From the above, we see that the regression coefficients are statistically significant with \(R^2 = 0.8982\). as expected, we can infer that ad exposure has a net positive effect on sales volumes. Furthermore, we find that the price elasticity is estimated to be -2.96, indicating that increasing the price has a negative impact on sales.
In Part II, we will see how we can estimate the same model without fixing \(\lambda\) and \(\alpha\) using a Bayesian approach. In Part III, we will construct a hierarchical model to guage the impact of another P, Place on the sales volume.
References
McCarthy, J. E. 1978. Basic Marketing: A Managerial Approach. Homewood, IL: RD Irwin.
Tellis, G. J. 2006. “Modeling Marketing Mix.” In The Handbook of Marketing Research, edited by M Grover R. & Vriens, 506–22. Thousand Oaks, CA: SAGE Publications Ltd.
LS0tDQp0aXRsZTogJ1VuZGVyc3RhbmRpbmcgTWFya2V0aW5nIE1peCBNb2RlbGxpbmc6IFBhcnQgSScNCm91dHB1dDoNCiAgaHRtbF9ub3RlYm9vazogZGVmYXVsdA0KICBodG1sX2RvY3VtZW50OiBkZWZhdWx0DQogIHBkZl9kb2N1bWVudDogZGVmYXVsdA0KYmlibGlvZ3JhcGh5OiBCaWJsaW9ncmFwaHkuYmliDQotLS0NCg0KDQojI0ludHJvZHVjdGlvbg0KQWNjb3JkaW5nIHRvIFtOaWVsc2VuXShodHRwOi8vd3d3Lm5pZWxzZW4uY29tL3VzL2VuL2luc2lnaHRzL3JlcG9ydHMvMjAxNC9tYXJrZXRpbmctbWl4LW1vZGVsaW5nLXdoYXQtbWFya2V0ZXJzLW5lZWQtdG8ta25vdy5odG1sKSwgTWFya2V0aW5nIG1peCBtb2RlbGluZyAoTU1NKSBpcyB0aGUgdXNlIG9mIHN0YXRpc3RpY2FsIGFuYWx5c2lzIHRvIGVzdGltYXRlIHRoZSBwYXN0IGltcGFjdCBhbmQgcHJlZGljdCB0aGUgZnV0dXJlIGltcGFjdCBvZiB2YXJpb3VzIG1hcmtldGluZyB0YWN0aWNzIG9uIHNhbGVzLiBUaGVzZSBtYXJrZXRpbmcgdGFjdGljcyB0eXBpY2FsbHkgY29uc2lzdCBvZiBjb250cm9sbGluZyBjZXJ0YWluIHZhcmlhYmxlcywgb2Z0ZW4gc3VtbWFyaXplZCBhcyB0aGUgZm91ciBQcyBvZiBtYXJrZXRpbmc6IHByb2R1Y3QsIHByaWNlLCBwcm9tb3Rpb24gYW5kIHBsYWNlW0BtY2NhcnRoeTE5NzhiYXNpY10uIFNvbWUgZXhhbXBsZXMgb2YgdmFyaWFibGVzIHJlbGF0ZWQgdG8gdGhlIHByb2R1Y3QgYXJlIG5ld25lc3Mgb2YgdGhlIHByb2R1Y3QsIGRpZmZlcmVudGlhdGlvbiBmcm9tIGNvbXBldGl0b3JzLCBldGMuIFByaWNlIHJlZmVycyB0byB0aGUgcHJvZHVjdCdzIGxpc3QgcHJpY2UgYW5kIGluY2VudGl2ZXMgc3VjaCBhcyBkaXNjb3VudHMgYW5kIGRlYWxzLiBQcm9tb3Rpb24gcmVmZXJzIHRvIGFkdmVydGlzaW5nIGFuZCBkaXNwbGF5cy4gUGxhY2UgcmVmZXJzIHRvIHRoZSBkZWxpdmVyeSBvZiB0aGUgcHJvZHVjdCBtZWFzdXJlZCBieSB2YXJpYWJsZXMgc3VjaCBhcyBkaXN0cmlidXRpb24sIGF2YWlsYWJpbGl0eSBhbmQgc2hlbGYtc3BhY2UgW0B0ZWxsaXMyMDA2XS4NCg0KVGhpcyBhcnRpY2xlIGF0dGVtcHRzIHRvIHByb3ZpZGUgYSBtYXRoZW1hdGljYWxseSByaWdvcm91cyBpbnRyb2R1Y3Rpb24gdG8gTU1NLCBhbmQgaXMgdGFyZ2V0ZWQgdG93YXJkcyByZWFkZXJzIHdpdGggc29tZSBxdWFudGl0YXRpdmUgYmFja2dyb3VuZCwgYnV0IG5vIHByZXZpb3VzIGV4cGVyaWVuY2UgaW4gbWFya2V0aW5nIHNjaWVuY2Ugb3IgZWNvbm9tZXRyaWNzLiBXZSBtYWlubHkgY29uc2lkZXIgdGhlIGVmZmVjdCBvZiBwcmljZSBhbmQgcHJvbW90aW9uIGluIHRoaXMgYXJ0aWNsZSBhcyB0aGVzZSBmYWN0b3JzIGFyZSByZWFkaWx5IHF1YW50aWZpYWJsZS4gV2UgYmVnaW4gYnkgZm9ybXVsYXRpbmcgdGhlIHJlZ3Jlc3Npb24gcHJvYmxlbS4gV2UgdGhlbiBpbmNvcnBvcmF0ZSBjZXJ0YWluIFtzdHlsaXplZCBmYWN0c10oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvU3R5bGl6ZWRfZmFjdCkgYWJvdXQgdGhlIGltcGFjdCBvZiBwcmljZSBhbmQgcHJvbW90aW9uIG9uIHNhbGVzIGludG8gdGhlIG1vZGVsLiBXZSB0aGVuIHNob3cgaG93IHRoaXMgYW5hbHlzaXMgY2FuIGUgcGVyZm9ybWVkIGluIFIgdXNpbmcgYSB3ZWxsLWtub3duIGRhdGFzZXQuIA0KDQojI1Byb2JsZW0gRm9ybXVsYXRpb24NCkEgY29tbW9uIG1ldGhvZG9sb2d5IGVtcGxveWVkIHRvIHN0dWR5IHRoZSBlZmZlY3Qgb2YgYSByZXNwb25zZSB2YXJpYWJsZSAoc2FsZXMsIGluIHRoaXMgY2FzZSkgb24gY2VydGFpbiBleHBsYW5hdG9yeSB2YXJpYWJsZXMgKHRoZSBmb3VyIFBzKSBpcyBbbXVsdGl2YXJpYXRlIGxpbmVhciByZWdyZXNzaW9uXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9MaW5lYXJfcmVncmVzc2lvbikuIExldCAkeV90JCBkZW5vdGUgdGhlIHNhbGVzIGF0IHRpbWUgJHQkLiBUeXBpY2FsbHksIHNhbGVzIHZvbHVtZSBpbiB1bml0cyBzb2xkIGlzIHVzZWQgYW5kIG5vdCByZXZlbnVlIGZyb20gc2FsZXMuIFRoaXMgaXMgYmVjYXVzZSB0aGUgcmV2ZW51ZSBmcm9tIHNhbGVzIGlzIGRlcGVuZGVudCBvbiB0aGUgcHJpY2UsIHdoaWNoIGlzIGFuIGV4cGxhbmF0b3J5IHZhcmlhYmxlLiBMZXQgJHhfe2l0fSQsICRpID0gMSwgXGxkb3RzLCBLJCBkZW5vdGUgdGhlICRLJCBleHBsYW5hdG9yeSB2YXJpYWJsZXMuIFRoZXNlIGNvdWxkIGluY2x1ZGUgdGhlIHByaWNlLCBtZWFzdXJlcyBvZiBhZCBleHBvc3VyZSwgc2hlbGYgcGxhY2VtZW50LCBzdG9yZSBsb2NhdGlvbiBldGMuIFRvIGVzdGltYXRlIHRoZSBlZmZlY3Qgb2YgZWFjaCBvZiB0aGUgZXhwbGFuYXRvcnkgdmFyaWFibGVzIG9uIHRoZSBzYWxlcyB2b2x1bWUsIHdlIGZpdCBhIHJlZ3Jlc3Npb24gbW9kZWwgZ2l2ZW4gYnkgDQokJA0KXHRhZ3sxfQ0KeV90ID0gXGJldGFfMCArIFxzdW1fe2kgPSAxfV57S31cYmV0YV9peF97aXR9ICsgXHZhcmVwc2lsb25fdA0KJCQNCndoZXJlICRcYmV0YV9pJCwgJGk9MCxcbGRvdHMsSyQgZGVub3RlIHRoZSByZWdyZXNzaW9uIGNvZWZmaWNpZW50cy4gQXNzdW1pbmcgdGhlIEdhdXNzLU1hcmtvdiBhc3N1bXB0aW9ucyBob2xkLCB0aGUgb3JkaW5hcnkgbGVhc3Qgc3F1YXJlcyAoT0xTKSBlc3RpbWF0b3JzIGZvciAkXGJldGFfaSQsIGRlbm90ZWQgYnkgJFxoYXR7XGJldGF9X2kkIGFyZSB0aGUgYmVzdCBsaW5lYXIgdW5iaWFzZWQgZXN0aW1hdG9ycy4gSW4gcHJhY3RpY2UsIGhvd2V2ZXIsIHRoaXMgaXMgcmFyZWx5IHRoZSBjYXNlIHNpbmNlIHRoZSBwcm9jZXNzICR5X3QkIGlzIHNlbGRvbSBzdGF0aW9uYXJ5IG9yIGluZGVwZW5kZW50LiBJbiBhIGxhdGVyIHNlY3Rpb24gd2Ugc2VlIGhvdyB3ZSBjYW4gYWRqdXN0IGZvciB0aGlzIHVzaW5nIGEgcHJhY3RpY2FsIGV4YW1wbGUuIA0KDQpUaGVyZSBhcmUgc2V2ZXJhbCBrZXkgc3R5bGl6ZWQgZmFjdHMgYWJvdXQgdGhlIGVmZmVjdCBvZiBhZHZlcnRpc2luZyBvbiBzYWxlcw0KDQogIDEuIEV4cG9zdXJlIHRvIGFkdmVydGlzaW5nIHRvZGF5IGhhcyBlZmZlY3RzIG9uIGNvbnN1bWVycycgcHVyY2hhc2luZyBkZWNpc2lvbnMgaW4gdGhlIGZ1dHVyZSAoQ2FycnlvdmVyIEVmZmVjdCkNCiAgMi4gVGhlIGVmZmVjdCBvZiBhZCBleHBvc3VyZSBvbiBwdXJjaGFzaW5nIGRlY2lzaW9ucyBkZWNheXMgd2l0aCB0aW1lIChDYXJyeW92ZXIgRWZmZWN0KQ0KICAzLiBUaGUgZWZmZWN0IG9mIGFkIGV4cG9zdXJlIG9uIHNhbGVzIGlzIHN1YmplY3QgdG8gdGhlIGxhdyBvZiBkaW1pbmlzaGluZyByZXR1cm5zIChTaGFwZSBFZmZlY3QpDQogIDQuIENoYW5naW5nIHRoZSBsZXZlbCBvZiBhZCBleHBvc3VyZSBicmluZ3MgYWJvdXQgYSByZWxhdGl2ZSBjaGFuZ2UgaW4gc2FsZXMgdm9sdW1lcyAoU2hhcGUgRWZmZWN0KQ0KDQpUaGVyZSBhcmUgc2V2ZXJhbCBvdGhlciBlZmZlY3RzIHRoYXQgY2FuIGJlIHRha2VuIGludG8gY29uc2lkZXJhdGlvbiBsaWtlIHRob3NlIG9mIGNvbXBldGl0aW9uLCBjb250ZW50IGFuZCBhZHZlcnRpc2luZyBtZWRpYVtAdGVsbGlzMjAwNl0uIEhvd2V2ZXIsIHRvIGF2b2lkIGNvbXBsaWNhdGluZyB0aGUgbW9kZWwsIHdlIGZvY3VzIG9uIHRoZXNlIGVmZmVjdHMgZm9yIG5vdy4gTGV0IHVzIGxvb2sgYXQgdGhlc2Ugb25lIGJ5IG9uZSBhbmQgZXhwcmVzcyB0aGVtIG1hdGhlbWF0aWNhbGx5IA0KDQojIyMjIENhcnJ5b3ZlciBFZmZlY3RzDQpDb25zaWRlciB0aGUgbW9kZWwgaW4gJCgxKSQuIExldCB1cyBzaW1wbGlmeSB0aGUgbW9kZWwsIHdoZXJlaW4gdGhlIHNhbGVzIGF0IHRpbWUgJHQkIGNhbiBieSBleHBsYWluZWQgYnkgYSBjb25zdGFudCBiYXNlIHZhbHVlIGFuZCB0aGUgYWRkaXRpb25hbCBlZmZlY3Qgb2YgYWQgZXhwb3N1cmUgYXQgdGltZSAkdCQsIG1lYXN1cmVkIGJ5IGEgdmFyaWFibGUgJHhfe3R9JC4gVGhpcyBpcyBkZW5vdGVkIGJ5DQokJA0KeV90ID0gXGJldGFfMCArIFxiZXRhXzEgeF90ICsgXHZhcmVwc2lsb25fdA0KJCQNCldlIGNhbiBpbmNvcnBvcmF0ZSBFZmZlY3RzIDEgYW5kIDIgaW50byB0aGUgYWJvdmUgbW9kZWwgYnkgaW5jb3Jwb3JhdGluZyB0aGUgbGFnZ2VkIGVmZmVjdHMgb2YgYWxsIHByZXZpb3VzIGFkIGV4cG9zdXJlcy4gVGhpcyBtb2RlbCBpcyBnaXZlbiBieSANCiQkDQp5X3QgPSBcYmV0YV8wICsgXGJldGFfMSBzX3QgKyBcdmFyZXBzaWxvbl90IFxcDQpzX3QgPSB4X3QgKyBcbGFtYmRhIHNfe3QtMX0NCiQkDQp3aGVyZSAkc18wJCBpcyB0eXBpY2FsbHkgc2V0IHRvICQwJC4gDQoNCiMjIyMgU2hhcGUgRWZmZWN0cw0KVG8gY2xlYXJseSBleHByZXNzIEVmZmVjdCAzLCB3ZSByZXdyaXRlIHRoZSBhYm92ZSBlcXVhdGlvbiBhcyBmb2xsb3dzDQokJA0KeV90ID0gXGJldGFfMCArIFxiZXRhXzEgeF90ICArIFxsYW1iZGEgXGJldGFfMSBzX3t0LTF9ICsgXHZhcmVwc2lsb25fdA0KJCQNClRoZSBhYm92ZSBleHByZXNzaW9uIGltcGxpZXMgdGhhdCB0aGUgcmF0ZSBvZiBjaGFuZ2Ugb2Ygc2FsZXMgdm9sdW1lcyB3aXRoIHJlc3BlY3QgdG8gYWQgZXhwb3N1cmUgKmluIHRoZSBzYW1lIHRpbWUgcGVyaW9kKiBpcyBjb25zdGFudCwgaS5lLCAkXHBhcnRpYWwgeV90LyBccGFydGlhbCB4X3QgPSBcYmV0YV8xJC4gSG93ZXZlciwgRWZmZWN0IDMgaW1wbGllcyB0aGF0IHRoaXMgcmF0ZSB0YXBlcnMgb2ZmIGFzIHdlIGluY3JlYXNlIGFkIGV4cG9zdXJlIGFuZCB0aGVyZWZvcmUgdGhlIGZvbGxvd2luZyBtdXN0IGhvbGQgdHJ1ZQ0KJCQNCiBcbGltX3t4X3QgXHRvIFxpbmZ0eX0gXGZyYWN7XHBhcnRpYWwgeV90fXtccGFydGlhbCB4X3R9ID0gMA0KJCQNClNldmVyYWwgZnVuY3Rpb25zIHNhdGlzZnkgdGhpcyBwcm9wZXJ0eVtAam9zZXBoMjAwNnVuZGVyc3RhbmRpbmddLCBlLmcsIHRoZSBsb2cgZnVuY3Rpb24gYW5kIHRoZSBsb2dpc3RpYyBmdW5jdGlvbi4gV2UgcGljayB0aGUgbG9naXN0aWMgZnVuY3Rpb24gd2l0aCBwYXJhbWV0ZXIgJFxhbHBoYSQgZm9yIGl0cyBmbGV4aWJpbGl0eS4gVGhlIG1vZGVsIG5vdyBiZWNvbWVzDQokJA0KXHRhZ3syfQ0KeV90ID0gXGJldGFfMCArIFxiZXRhXzEgc190ICsgXHZhcmVwc2lsb25fdCBcXA0Kc190ID0gXGZyYWN7MX17MSsgZV57LVxhbHBoYSB4X3R9fSArIFxsYW1iZGEgc197dC0xfQ0KJCQNCk5vdywgJCgyKSQgaW1wbGllcyB0aGF0IHRoZSBpbmNyZWFzZSBpbiBzYWxlcyBmcm9tIGluY3JlYXNlIGluIGFkIGV4cG9zdXJlIGlzIGluZGVwZW5kZW50IG9mICR5X3QkLiBJbiBvdGhlciB3b3JkcywgaXQgaXMgKmFic29sdXRlKi4gVGhpcyBtZWFucyB0aGF0IGluY3JlYXNpbmcgYWQgZXhwb3N1cmUgYnkgb25lIHVuaXQgY2F1c2UgdGhlIHNhbGVzIHRvIGluY3JlYXNlIGJ5IGEgY2VydGFpbiBhbW91bnQgaXJyZXNwZWN0aXZlIG9mIHRoZSBtYWduaXR1ZGUgb2YgJHlfdCQuIEZvciBleGFtcGxlLCBhc3N1bWUgJFxiZXRhXzAgPSAwJCwgJFxiZXRhXzEgPSAxJCwgJFxhbHBoYSA9IDEkLCAkXGxhbWJkYSA9IDAkLiBMZXQgdXMgaW5jcmVhc2UgJHhfdCQgdG8gJHgnX3QkLiBBc3N1bWluZyB0aGUgaW5jcmVtZW50IGlzIHNtYWxsIGVub3VnaCwgIFdlIGNhbiBhcHByb3hpbWF0ZSB0aGUgbmV3IHNhbGVzIHZhbHVlIGJ5IA0KJCQNCnknX3QgXGFwcHJveCB5X3QgKyBcZnJhY3tccGFydGlhbCB5X3R9e1xwYXJ0aWFsIHhfdH0oeCdfdCAtIHhfdCkNCiQkDQpJZiAkKHgnX3QgLSB4X3QpID0gMC4xJCwgYW5kICR4X3Q9MSQsIHdlIGhhdmUgJHknX3QgLSB5X3QgXGFwcHJveCAwLjIkLiBUaGVyZWZvcmUsIHNhbGVzIGluY3JlYXNlIGJ5IDAuMiB1bml0cyByZWdhcmRsZXNzIG9mIHdoZXRoZXIgJHlfdCQgaXMgMTAwIG9yIDEwMDAuIEluIHByYWN0aWNlLCB3ZSB1c3VhbGx5IHNlZSB0aGF0IHNhbGVzIGluY3JlYXNlIGluIGEgKnJlbGF0aXZlKiBtYW5uZXIsIGkuZSBieSBhIHBlcmNlbnRhZ2UgcmF0aGVyIHRoYW4gaW4gYWJzb2x1dGUgdGVybXMuIFRvIHRha2UgdGhpcyBpbnRvIGFjY291bnQgd2UgZW1wbG95IGEgbG9nYXJpdGhtaWMgdHJhbnNmb3JtYXRpb24gb24gdGhlIGRlcGVuZGVudCB2YXJpYWJsZSBhbmQgb2J0YWluDQokJA0KXGxvZyB5X3QgPSBcYmV0YV8wICsgXGJldGFfMSBzX3QgKyBcdmFyZXBzaWxvbl90IFxcDQpzX3QgPSBcZnJhY3sxfXsxKyBlXnstXGFscGhhIHhfdH19ICsgXGxhbWJkYSBzX3t0LTF9DQokJA0KSXQgaXMgdHJpdmlhbCB0byBjaGVjayB0aGF0ICR7XHBhcnRpYWwgeV90fS97XHBhcnRpYWwgeF90fSQgaW5kZWVkIGRlcGVuZHMgb24gJHlfdCQgYWZ0ZXIgdGhpcyB0cmFuc2Zvcm1hdGlvbi4gDQoNCiMjIyMgSW1wYWN0IG9mIFByaWNlDQpJbiBvcmRlciB0byBicmluZyBwcmljZSBpbnRvIHRoZSBtb2RlbCwgd2UgcmVseSBvbiB0aGUgW3ByaWNlIGVsYXN0aWNpdHkgb2YgZGVtYW5kXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9QcmljZV9lbGFzdGljaXR5X29mX2RlbWFuZCksIGEgbWVhc3VyZSBvZiByZXNwb25zaXZlbmVzcyBvZiB0aGUgcXVhbnRpdHkgb2YgdGhlIGdvb2QgZGVtYW5kZWQgdG8gYSBjaGFuZ2UgaW4gaXRzIHByaWNlLiBXZSBjYW4gYXJndWUgdGhhdCBzYWxlcyB2b2x1bWVzIGRpcmVjdGx5IHJlZmxlY3QgdGhlIGRlbWFuZCBmb3IgdGhlIHByb2R1Y3QsIGFzc3VtaW5nIHRoYXQgcHJvZHVjdGlvbiBjYW4ga2VlcCB1cCB3aXRoIGRlbWFuZCBsaW5lYXJseS4gVGhlIHByaWNlIGVsYXN0aWNpdHkgJEUkIGlzIGV4cHJlc3NlZCBhcyANCiQkDQpFID0gXGZyYWN7XHBhcnRpYWwgcV90L3FfdH17XHBhcnRpYWwgcF90IC9wX3R9DQokJA0Kd2hlcmUgJHBfdCQgaXMgdGhlIHByaWNlIG9mIHRoZSBwcm9kdWN0IGF0IHRpbWUgJHQkIGFuZCAkcV90JCBpcyB0aGUgcXVhbnRpdHkgb2YgdGhlIHByb2R1Y3QgZGVtYW5kZWQuIEFzc3VtaW5nIHNhbGVzIHZvbHVtZXMgYXJlIGRpcmVjdGx5IHByb3BvcnRpb25hbCB0byBxdWFudGl0eSBkZW1hbmRlZCwgd2UgY2FuIHdyaXRlICR5X3QgPSBrIHFfdCQsIHdoZXJlICRrJCBpcyBhIGNvbnN0YW50IG9mIHByb3BvcnRpb25hbGl0eS4gVXNpbmcgdGhpcyBpbiB0aGUgYWJvdmUgZXhwcmVzc2lvbiB3ZSBoYXZlDQokJA0KXGZyYWN7XHBhcnRpYWwgXGxvZyB5X3R9e1xwYXJ0aWFsIHBfdH0gPSBrRSBcbG9nIHBfdA0KJCQNCldlIGNhbiByZWFkaWx5IG1lcmdlICRrRSQgaW50byBvbmUgdGVybSBhbmQgaW5jbHVkZSB0aGUgYWJvdmUgZXhwcmVzc2lvbiBpbnRvIHRoZSByZWdyZXNzaW9uIG1vZGVsLiBGaW5hbGx5LCB3ZSBoYXZlDQokJA0KXGxvZyB5X3QgPSBcYmV0YV8wICsgXGJldGFfMSBzX3QgKyBcYmV0YV8yIFxsb2cgcF90ICtcdmFyZXBzaWxvbl90IFxcDQpzX3QgPSBcZnJhY3sxfXsxKyBlXnstXGFscGhhIHhfdH19ICsgXGxhbWJkYSBzX3t0LTF9DQokJA0KV2Ugc2VlIHRoYXQgdGhlIG1vZGVsIGhhcyA1IHBhcmFtZXRlcnMgdGhhdCBuZWVkIHRvIGJlIGVzdGltYXRlZCwgYW5kIGl0IGlzIG5vbmxpbmVhciBpbiAkXGxhbWJkYSQgYW5kICRcYWxwaGEkLiBJbiB3aGF0IGZvbGxvd3Mgd2UgZml4ICRcbGFtYmRhJCBhbmQgJFxhbHBoYSQgdG8gcmVnYWluIGxpbmVhcml0eSBpbiB0aGUgcGFyYW1ldGVyIHNwYWNlLiBUbyBmaXggJFxsYW1iZGEkIHdlIHVzZSB0aGUgZW1waXJpY2FsIG9ic2VydmF0aW9uIHRoYXQgdGhlIGhhbGYtbGlmZSBvZiBhbiBhZCBjb3B5IGlzIGFwcHJveGltYXRlbHkgMi41IHdlZWtzLiBUaGlzIHlpZWxkcyAkXGxhbWJkYT0wLjI1JC4gV2Ugd2lsbCBzaW1wbHkgYXNzdW1lICRcYWxwaGE9MiQsIHRob3VnaCBpbiByZWFsaXR5LCB0aGlzIGlzIGNhbGlicmF0ZWQgbW9yZSBjYXJlZnVsbHkuIA0KDQojIyBBbiBFeGFtcGxlDQpXZSB1c2UgdGhlIFtTbGljZWQgQ2hlZXNlIERhdGFzZXRdKGh0dHBzOi8vd3d3LnJkb2N1bWVudGF0aW9uLm9yZy9wYWNrYWdlcy9iYXllc20vdmVyc2lvbnMvMy4wLTIvdG9waWNzL2NoZWVzZSkgZnJvbSB0aGUgYGJheWVzbWAgcGFja2FnZSB0byBpbGx1c3RyYXRlIGhvdyB0byBmaXQgdGhlIG1vZGVsLiANCmBgYHtyLCBlY2hvPVRSVUV9DQpsaWJyYXJ5KGJheWVzbSkNCmRhdGEoImNoZWVzZSIpDQpzdW1tYXJ5KGNoZWVzZSkNCmBgYA0KVGhlIGRhdGEgY29uc2lzdHMgb2Ygd2Vla2x5IG9ic2VydmF0aW9ucyBvZiB0aGUgdm9sdW1lIG9mIHNhbGVzIG9mIHNsaWNlZCBjaGVlc2Ugb3ZlciBkaWZmZXJlbnQgcmV0YWlsZXJzIGluIHRoZSBVbml0ZWQgU3RhdGVzLiBIZXJlLCBgVk9MVU1FYCBjb3JyZXNwb25kcyB0byAkeV90JCwgYERJU1BgIHRvICR4X3QkIGFuZCBgUFJJQ0VgIHRvICRwX3QkLiBXZSBhcmUgbm90IHlldCBsb29raW5nIGF0IGhpZXJhcmNoaWNhbCBtb2RlbHMsIHNvIHdlIGZvY3VzIG9uIG9uZSByZXRhaWxlciwgc2F5IERvbWluaWNrJ3MgaW4gQ2hpY2Fnby4gV2UgdGhlbiBhcHBseSB0aGUgcmVxdWlyZWQgdHJhbnNmb3JtYXRpb25zIHRvIG9idGFpbiB0aGUgcmVncmVzc2lvbiBkYXRhc2V0LiB0aGVzZSBhcmUgc2hvd24gYmVsb3cuDQoNCmBgYHtyLCBldmFsPUZBTFNFLCBpbmNsdWRlPVRSVUV9DQpsaWJyYXJ5KGRwbHlyKQ0KDQphZHN0b2NrVHJhbnNmb3JtIDwtIGZ1bmN0aW9uKHgpew0KICBzdGF0czo6ZmlsdGVyKCAxLygxK2V4cCgtMip4KSksIDAuMjUsIG1ldGhvZCA9ICJyZWN1cnNpdmUiKQ0KfQ0KDQptbW0uZGF0YSA8LSBjaGVlc2UgJT4lIGZpbHRlcihSRVRBSUxFUiA9PSAnQ0hJQ0FHTyAtIERPTUlOSUNLJykgJT4lDQogIHNlbGVjdCgtUkVUQUlMRVIpICU+JSANCiAgbXV0YXRlKGxvZy52b2x1bWUgPSBsb2coVk9MVU1FKSwgbG9nLnByaWNlID0gbG9nKFBSSUNFKSwgYWRzdG9jaz0gYWRzdG9ja1RyYW5zZm9ybShESVNQKSkgJT4lDQogIHNlbGVjdCgtVk9MVU1FLCAtRElTUCwgLVBSSUNFKQ0KDQpgYGANCkFmdGVyIGNoZWNraW5nIHRoZSBBQ0YgYW5kIFBBQ0Ygb2YgYGxvZy52b2x1bWVgIHRvIGVuc3VyZSBzdGF0aW9uYXJpdHkgYW5kIGluZGVwZW5kZW5jZSwgd2UgZml0IGEgbGluZWFyIG1vZGVsDQpgYGB7cn0NCmZpdCA8LSBsbShsb2cudm9sdW1lIH4gYWRzdG9jayArIGxvZy5wcmljZSwgZGF0YT1tbW0uZGF0YSkNCg0Kc3VtbWFyeShmaXQpDQpgYGANCg0KRnJvbSB0aGUgYWJvdmUsIHdlIHNlZSB0aGF0IHRoZSByZWdyZXNzaW9uIGNvZWZmaWNpZW50cyBhcmUgc3RhdGlzdGljYWxseSBzaWduaWZpY2FudCB3aXRoICRSXjIgPSAwLjg5ODIkLiBhcyBleHBlY3RlZCwgd2UgY2FuIGluZmVyIHRoYXQgYWQgZXhwb3N1cmUgaGFzIGEgbmV0IHBvc2l0aXZlIGVmZmVjdCBvbiBzYWxlcyB2b2x1bWVzLiBGdXJ0aGVybW9yZSwgd2UgZmluZCB0aGF0IHRoZSBwcmljZSBlbGFzdGljaXR5IGlzIGVzdGltYXRlZCB0byBiZSAtMi45NiwgaW5kaWNhdGluZyB0aGF0IGluY3JlYXNpbmcgdGhlIHByaWNlIGhhcyBhIG5lZ2F0aXZlIGltcGFjdCBvbiBzYWxlcy4gDQoNCkluIFBhcnQgSUksIHdlIHdpbGwgc2VlIGhvdyB3ZSBjYW4gZXN0aW1hdGUgdGhlIHNhbWUgbW9kZWwgd2l0aG91dCBmaXhpbmcgJFxsYW1iZGEkIGFuZCAkXGFscGhhJCB1c2luZyBhIEJheWVzaWFuIGFwcHJvYWNoLiBJbiBQYXJ0IElJSSwgd2Ugd2lsbCBjb25zdHJ1Y3QgYSBoaWVyYXJjaGljYWwgbW9kZWwgdG8gZ3VhZ2UgdGhlIGltcGFjdCBvZiBhbm90aGVyIFAsIFBsYWNlIG9uIHRoZSBzYWxlcyB2b2x1bWUuIA0KDQojIyBSZWZlcmVuY2Vz