This week, we are going to use data from Gavin and Hofmann (2002), a study on organizational climate and attitudes published in Leadership Quarterly. Here, we have individuals soldiers nested within companies. This is the same dataset that Garson uses in Chapter 6, so you can recreate his analysis.

Load Some Packages to Help with the Analysis and Data Management:

library(tidyverse)
library(psych)
library(lme4)

Load and Explore the Data

glimpse(lq2002)

Rows: 2,042
Columns: 28
$ ROWID    <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,…
$ COMPID   <dbl> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, …
$ SUB      <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,…
$ LEAD01   <dbl> 2, 4, 4, 1, 1, 2, 3, 3, 3, 4, 2, 2, 4, 4, 1, 2, 4, 4, 4, 4, 4, 3, …
$ LEAD02   <dbl> 2, 4, 4, 1, 1, 3, 2, 3, 4, 3, 4, 3, 4, 4, 1, 1, 5, 4, 5, 4, 2, 4, …
$ LEAD03   <dbl> 2, 2, 2, 1, 1, 2, 2, 2, 3, 3, 1, 1, 4, 4, 1, 2, 3, 4, 1, 4, 2, 4, …
$ LEAD04   <dbl> 3, 4, 4, 1, 1, 4, 1, 4, 2, 4, 3, 1, 4, 4, 1, 4, 5, 4, 1, 5, 1, 4, …
$ LEAD05   <dbl> 2, 4, 2, 1, 3, 2, 2, 2, 3, 3, 1, 2, 4, 4, 1, 3, 3, 4, 1, 2, 2, 3, …
$ LEAD06   <dbl> 2, 4, 4, 1, 2, 4, 2, 2, 3, 4, 3, 4, 4, 4, 1, 3, 5, 4, 4, 2, 2, 4, …
$ LEAD07   <dbl> 5, 4, 4, 4, 1, 3, 2, 2, 3, 3, 3, 1, 4, 3, 1, 5, 3, 4, 1, 3, 2, 3, …
$ LEAD08   <dbl> 5, 4, 4, 2, 1, 4, 2, 4, 1, 4, 4, 3, 4, 4, 1, 2, 5, 4, 1, 3, 2, 4, …
$ LEAD09   <dbl> 5, 4, 4, 3, 1, 2, 3, 2, 3, 3, 2, 2, 4, 4, 1, 4, 1, 4, 1, 4, 2, 2, …
$ LEAD10   <dbl> 5, 4, 4, 3, 1, 4, 4, 2, 2, 4, 3, 2, 4, 4, 1, 2, 5, 4, 4, 4, 1, 4, …
$ LEAD11   <dbl> 4, 3, 2, 2, 1, 3, 2, 4, 3, 3, 2, 1, 4, 4, 1, 2, 4, 4, 1, 4, 2, 3, …
$ TSIG01   <dbl> 1, 3, 5, 1, 4, 4, 2, 2, 4, 4, 3, 3, 4, 4, 1, 1, 4, 4, 1, 5, 1, 4, …
$ TSIG02   <dbl> 5, 4, 5, 3, 4, 4, 4, 2, 3, 4, 4, 3, 4, 4, 1, 5, 4, 4, 5, 5, 4, 4, …
$ TSIG03   <dbl> 5, 3, 5, 2, 4, 4, 4, 4, 4, 4, 3, 3, 4, 4, 1, 4, 5, 4, 5, 5, 4, 4, …
$ HOSTIL01 <dbl> 0, 3, 1, 2, 2, 0, 2, 0, 0, 1, 1, 2, 0, 0, 4, 3, 1, 0, 4, 3, 4, 1, …
$ HOSTIL02 <dbl> 0, 2, 0, 1, 0, 0, 0, 0, 4, 0, 0, 2, 0, 0, 4, 0, 0, 0, 0, 1, 4, 1, …
$ HOSTIL03 <dbl> 0, 0, 0, 3, 0, 0, 1, 0, 4, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 2, 2, 0, …
$ HOSTIL04 <dbl> 0, 0, 0, 3, 0, 0, 1, 0, 4, 0, 0, 1, 0, 0, 4, 0, 0, 0, 4, 3, 2, 0, …
$ HOSTIL05 <dbl> 0, 1, 1, 0, 0, 0, 0, 0, 4, 0, 0, 2, 0, 0, 2, 0, 0, 0, 4, 2, 3, 0, …
$ LEAD     <dbl> 3.363636, 3.727273, 3.454545, 1.818182, 1.272727, 3.000000, 2.2727…
$ TSIG     <dbl> 3.666667, 3.333333, 5.000000, 2.000000, 4.000000, 4.000000, 3.3333…
$ HOSTILE  <dbl> 0.0, 1.2, 0.4, 1.8, 0.4, 0.0, 0.8, 0.0, 3.2, 0.2, 0.2, 1.4, 0.0, 0…
$ GLEAD    <dbl> 2.882576, 2.882576, 2.882576, 2.882576, 2.882576, 2.882576, 2.8825…
$ GTSIG    <dbl> 3.541667, 3.541667, 3.541667, 3.541667, 3.541667, 3.541667, 3.5416…
$ GHOSTILE <dbl> 1.0416667, 1.0416667, 1.0416667, 1.0416667, 1.0416667, 1.0416667, …

Remember our old friend describe from the psych package a few weeks ago? This is a great way to get a detailed summary of all the variables in a dataset.

Calculing Scale Scores and Cronbach’s Alpha

This week, we are going to get a crash course in how to calculate reliability using Cronbach’s alpha. The alpha function in the psych package is a quick and easy way to calculate alpha:

alpha(hostile_items)


Reliability analysis   
Call: alpha(x = hostile_items)

 

 lower alpha upper     95% confidence boundaries
0.86 0.87 0.88 

 Reliability if an item is dropped:

 Item statistics 

Non missing response frequency for each item
            0    1    2    3    4 miss
HOSTIL01 0.32 0.25 0.17 0.13 0.13    0
HOSTIL02 0.67 0.17 0.07 0.04 0.05    0
HOSTIL03 0.53 0.17 0.09 0.08 0.12    0
HOSTIL04 0.60 0.15 0.08 0.07 0.10    0
HOSTIL05 0.66 0.18 0.08 0.04 0.03    0

Now, we can create a new variable, hostile, that is constructed by taking the mean of our 5 hostile items.

my.keys.list <- list(hostile = c("HOSTIL01","HOSTIL02","HOSTIL03", "HOSTIL04","HOSTIL05"))
                     
my.scales <- scoreItems(my.keys.list, lq2002, impute = "none")

Scale reliability estimates, etc. for Likert-type scales

print(my.scales, short = FALSE)

Call: scoreItems(keys = my.keys.list, items = lq2002, impute = "none")

(Standardized) Alpha:
      hostile
alpha    0.87

Standard errors of unstandardized Alpha:
      hostile
ASE     0.011

Standardized Alpha of observed scales:
     hostile
[1,]    0.87

Average item correlation:
          hostile
average.r    0.58

Median item correlation:
hostile 
   0.56 

 Guttman 6* reliability: 
         hostile
Lambda.6    0.87

Signal/Noise based upon av.r : 
             hostile
Signal/Noise     6.9

Scale intercorrelations corrected for attenuation 
 raw correlations below the diagonal, alpha on the diagonal 
 corrected correlations above the diagonal:

Note that these are the correlations of the complete scales based on the correlation matrix,
 not the observed scales based on the raw items.
        hostile
hostile    0.87

Item by scale correlations:
 corrected for item overlap and scale reliability
         hostile
HOSTIL01    0.70
HOSTIL02    0.73
HOSTIL03    0.81
HOSTIL04    0.84
HOSTIL05    0.74

Non missing response frequency for each item
            0    1    2    3    4 miss
HOSTIL01 0.32 0.25 0.17 0.13 0.13    0
HOSTIL02 0.67 0.17 0.07 0.04 0.05    0
HOSTIL03 0.53 0.17 0.09 0.08 0.12    0
HOSTIL04 0.60 0.15 0.08 0.07 0.10    0
HOSTIL05 0.66 0.18 0.08 0.04 0.03    0

Save Scored Scales as New Variables

my.scores <- as_tibble(my.scales$scores)

Attach New Variable to Old Data Set (LQ2002)

glimpse(lq2002.1)

Rows: 2,042
Columns: 29
$ ROWID    <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,…
$ COMPID   <dbl> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, …
$ SUB      <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,…
$ LEAD01   <dbl> 2, 4, 4, 1, 1, 2, 3, 3, 3, 4, 2, 2, 4, 4, 1, 2, 4, 4, 4, 4, 4, 3, …
$ LEAD02   <dbl> 2, 4, 4, 1, 1, 3, 2, 3, 4, 3, 4, 3, 4, 4, 1, 1, 5, 4, 5, 4, 2, 4, …
$ LEAD03   <dbl> 2, 2, 2, 1, 1, 2, 2, 2, 3, 3, 1, 1, 4, 4, 1, 2, 3, 4, 1, 4, 2, 4, …
$ LEAD04   <dbl> 3, 4, 4, 1, 1, 4, 1, 4, 2, 4, 3, 1, 4, 4, 1, 4, 5, 4, 1, 5, 1, 4, …
$ LEAD05   <dbl> 2, 4, 2, 1, 3, 2, 2, 2, 3, 3, 1, 2, 4, 4, 1, 3, 3, 4, 1, 2, 2, 3, …
$ LEAD06   <dbl> 2, 4, 4, 1, 2, 4, 2, 2, 3, 4, 3, 4, 4, 4, 1, 3, 5, 4, 4, 2, 2, 4, …
$ LEAD07   <dbl> 5, 4, 4, 4, 1, 3, 2, 2, 3, 3, 3, 1, 4, 3, 1, 5, 3, 4, 1, 3, 2, 3, …
$ LEAD08   <dbl> 5, 4, 4, 2, 1, 4, 2, 4, 1, 4, 4, 3, 4, 4, 1, 2, 5, 4, 1, 3, 2, 4, …
$ LEAD09   <dbl> 5, 4, 4, 3, 1, 2, 3, 2, 3, 3, 2, 2, 4, 4, 1, 4, 1, 4, 1, 4, 2, 2, …
$ LEAD10   <dbl> 5, 4, 4, 3, 1, 4, 4, 2, 2, 4, 3, 2, 4, 4, 1, 2, 5, 4, 4, 4, 1, 4, …
$ LEAD11   <dbl> 4, 3, 2, 2, 1, 3, 2, 4, 3, 3, 2, 1, 4, 4, 1, 2, 4, 4, 1, 4, 2, 3, …
$ TSIG01   <dbl> 1, 3, 5, 1, 4, 4, 2, 2, 4, 4, 3, 3, 4, 4, 1, 1, 4, 4, 1, 5, 1, 4, …
$ TSIG02   <dbl> 5, 4, 5, 3, 4, 4, 4, 2, 3, 4, 4, 3, 4, 4, 1, 5, 4, 4, 5, 5, 4, 4, …
$ TSIG03   <dbl> 5, 3, 5, 2, 4, 4, 4, 4, 4, 4, 3, 3, 4, 4, 1, 4, 5, 4, 5, 5, 4, 4, …
$ HOSTIL01 <dbl> 0, 3, 1, 2, 2, 0, 2, 0, 0, 1, 1, 2, 0, 0, 4, 3, 1, 0, 4, 3, 4, 1, …
$ HOSTIL02 <dbl> 0, 2, 0, 1, 0, 0, 0, 0, 4, 0, 0, 2, 0, 0, 4, 0, 0, 0, 0, 1, 4, 1, …
$ HOSTIL03 <dbl> 0, 0, 0, 3, 0, 0, 1, 0, 4, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 2, 2, 0, …
$ HOSTIL04 <dbl> 0, 0, 0, 3, 0, 0, 1, 0, 4, 0, 0, 1, 0, 0, 4, 0, 0, 0, 4, 3, 2, 0, …
$ HOSTIL05 <dbl> 0, 1, 1, 0, 0, 0, 0, 0, 4, 0, 0, 2, 0, 0, 2, 0, 0, 0, 4, 2, 3, 0, …
$ LEAD     <dbl> 3.363636, 3.727273, 3.454545, 1.818182, 1.272727, 3.000000, 2.2727…
$ TSIG     <dbl> 3.666667, 3.333333, 5.000000, 2.000000, 4.000000, 4.000000, 3.3333…
$ HOSTILE  <dbl> 0.0, 1.2, 0.4, 1.8, 0.4, 0.0, 0.8, 0.0, 3.2, 0.2, 0.2, 1.4, 0.0, 0…
$ GLEAD    <dbl> 2.882576, 2.882576, 2.882576, 2.882576, 2.882576, 2.882576, 2.8825…
$ GTSIG    <dbl> 3.541667, 3.541667, 3.541667, 3.541667, 3.541667, 3.541667, 3.5416…
$ GHOSTILE <dbl> 1.0416667, 1.0416667, 1.0416667, 1.0416667, 1.0416667, 1.0416667, …
$ hostile  <dbl> 0.0, 1.2, 0.4, 1.8, 0.4, 0.0, 0.8, 0.0, 3.2, 0.2, 0.2, 1.4, 0.0, 0…

Visualizing the Key Variables

Let’s take a look at some of the key variables for our analysis. We have three variables, all which are scale scores constructed just like we did for the example above.

hist(lq2002$HOSTILE, 
     main = "Distribution of Hostility Scores",
     sub = "N = 2,042 U.S. Army active duty soliders. Data from Gavin and Hofmann (2002).",
     xlab = "Scaled Hostility Score")

hist(lq2002$TSIG, 
     main = "Distribution of Task Significance Scores",
     sub = "N = 2,042 U.S. Army active duty soliders. Data from Gavin and Hofmann (2002).",
     xlab = "Scaled Task Significance Score")

hist(lq2002$LEAD, 
     main = "Distribution of Leadership Climate Scores",
     sub = "N = 2,042 U.S. Army active duty soliders. Data from Gavin and Hofmann (2002).",
     xlab = "Scaled Leadership Climate Score")

Running the Multilevel Models

Step One: The Null Model

models <- list()
model.0 <- lmer(HOSTILE ~ (1|COMPID), REML = FALSE, data = lq2002.1)
summary(model.0)
Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: HOSTILE ~ (1 | COMPID)
   Data: lq2002.1

     AIC      BIC   logLik deviance df.resid 
  5888.9   5905.8  -2941.5   5882.9     2039 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-1.3826 -0.7294 -0.3230  0.5086  3.1628 

Random effects:
 Groups   Name        Variance Std.Dev.
 COMPID   (Intercept) 0.05765  0.2401  
 Residual             1.01668  1.0083  
Number of obs: 2042, groups:  COMPID, 49

Fixed effects:
            Estimate Std. Error t value
(Intercept)  0.90632    0.04308   21.04

Post-Estimation Task: Calculate ICC

Here’s a quick and easy way to get our friend the ICC!


null.ICC <- 0.05765/(0.05765 + 1.01668)
null.ICC
[1] 0.05366135

Using lmerTest to Evaluate Random Effects:

lmerTest::rand(model.0)
ANOVA-like table for random-effects: Single term deletions

Model:
HOSTILE ~ (1 | COMPID)
             npar  logLik    AIC  LRT Df Pr(>Chisq)    
<none>          3 -2941.5 5888.9                       
(1 | COMPID)    2 -2970.2 5944.3 57.4  1  3.556e-14 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Step Two: Adding a Level One Predictor, TSIG

model.1 <- lmer(HOSTILE ~ TSIG + (1|COMPID), REML = FALSE, data = lq2002.1)
summary(model.1)
Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: HOSTILE ~ TSIG + (1 | COMPID)
   Data: lq2002.1

     AIC      BIC   logLik deviance df.resid 
  5668.4   5690.9  -2830.2   5660.4     2038 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-1.9601 -0.7135 -0.2688  0.4734  3.4848 

Random effects:
 Groups   Name        Variance Std.Dev.
 COMPID   (Intercept) 0.02817  0.1678  
 Residual             0.91938  0.9588  
Number of obs: 2042, groups:  COMPID, 49

Fixed effects:
            Estimate Std. Error t value
(Intercept)  1.96574    0.07574   25.95
TSIG        -0.33180    0.02145  -15.47

Correlation of Fixed Effects:
     (Intr)
TSIG -0.894

Step Three: Adding an Additional Level One Predictor, LEAD

model.2 <- lmer(HOSTILE ~ TSIG + LEAD + (1|COMPID), REML = FALSE, data = lq2002.1)
summary(model.2)
Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: HOSTILE ~ TSIG + LEAD + (1 | COMPID)
   Data: lq2002.1

     AIC      BIC   logLik deviance df.resid 
  5540.7   5568.9  -2765.4   5530.7     2037 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.3574 -0.6918 -0.2278  0.5180  3.4823 

Random effects:
 Groups   Name        Variance Std.Dev.
 COMPID   (Intercept) 0.02028  0.1424  
 Residual             0.86544  0.9303  
Number of obs: 2042, groups:  COMPID, 49

Fixed effects:
            Estimate Std. Error t value
(Intercept)  2.56714    0.08847  29.017
TSIG        -0.18610    0.02434  -7.646
LEAD        -0.35010    0.03021 -11.588

Correlation of Fixed Effects:
     (Intr) TSIG  
TSIG -0.328       
LEAD -0.577 -0.523

Step Four: Testing the Interaction of LEAD and TSIG

model.3 <- lmer(HOSTILE ~ TSIG + LEAD + TSIG:LEAD + (1|COMPID), REML = FALSE, data = lq2002.1)
summary(model.3)
Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: HOSTILE ~ TSIG + LEAD + TSIG:LEAD + (1 | COMPID)
   Data: lq2002.1

     AIC      BIC   logLik deviance df.resid 
  5539.6   5573.3  -2763.8   5527.6     2036 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.5016 -0.6809 -0.2376  0.5107  3.4871 

Random effects:
 Groups   Name        Variance Std.Dev.
 COMPID   (Intercept) 0.02012  0.1419  
 Residual             0.86416  0.9296  
Number of obs: 2042, groups:  COMPID, 49

Fixed effects:
            Estimate Std. Error t value
(Intercept)  2.89374    0.20342  14.225
TSIG        -0.29435    0.06543  -4.499
LEAD        -0.47236    0.07498  -6.300
TSIG:LEAD    0.03858    0.02166   1.781

Correlation of Fixed Effects:
          (Intr) TSIG   LEAD  
TSIG      -0.889              
LEAD      -0.925  0.771       
TSIG:LEAD  0.901 -0.928 -0.915

Step Five: Adding a Level Two Predictor, GTSIG

model.4 <- lmer(HOSTILE ~ TSIG + LEAD + GTSIG + (1|COMPID), REML = FALSE, data = lq2002.1)
summary(model.4)
Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: HOSTILE ~ TSIG + LEAD + GTSIG + (1 | COMPID)
   Data: lq2002.1

     AIC      BIC   logLik deviance df.resid 
  5533.7   5567.4  -2760.8   5521.7     2036 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.3166 -0.6960 -0.2211  0.5170  3.4998 

Random effects:
 Groups   Name        Variance Std.Dev.
 COMPID   (Intercept) 0.009905 0.09952 
 Residual             0.867050 0.93116 
Number of obs: 2042, groups:  COMPID, 49

Fixed effects:
            Estimate Std. Error t value
(Intercept)  3.40718    0.25785  13.214
TSIG        -0.17234    0.02479  -6.953
LEAD        -0.35177    0.02996 -11.741
GTSIG       -0.27729    0.08258  -3.358

Correlation of Fixed Effects:
      (Intr) TSIG   LEAD  
TSIG   0.105              
LEAD  -0.205 -0.511       
GTSIG -0.942 -0.227  0.010

Step Six: Adding a 2nd Level Two Predictor, GLEAD

model.5 <- lmer(HOSTILE ~ TSIG + LEAD + GTSIG + GLEAD + (1|COMPID), REML = FALSE, data = lq2002.1)
summary(model.5)
Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: HOSTILE ~ TSIG + LEAD + GTSIG + GLEAD + (1 | COMPID)
   Data: lq2002.1

     AIC      BIC   logLik deviance df.resid 
  5535.2   5574.6  -2760.6   5521.2     2035 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.3187 -0.6941 -0.2225  0.5113  3.4930 

Random effects:
 Groups   Name        Variance Std.Dev.
 COMPID   (Intercept) 0.009737 0.09868 
 Residual             0.866974 0.93111 
Number of obs: 2042, groups:  COMPID, 49

Fixed effects:
            Estimate Std. Error t value
(Intercept)  3.52131    0.31006  11.357
TSIG        -0.17486    0.02508  -6.971
LEAD        -0.34583    0.03132 -11.042
GTSIG       -0.24989    0.09273  -2.695
GLEAD       -0.06981    0.10715  -0.651

Correlation of Fixed Effects:
      (Intr) TSIG   LEAD   GTSIG 
TSIG   0.000                     
LEAD   0.000 -0.528              
GTSIG -0.437 -0.271  0.143       
GLEAD -0.559  0.154 -0.292 -0.460

Should We Include That Interaction? Comparing model.2 with model.3:

anova(model.2, model.3)
Data: lq2002.1
Models:
model.2: HOSTILE ~ TSIG + LEAD + (1 | COMPID)
model.3: HOSTILE ~ TSIG + LEAD + TSIG:LEAD + (1 | COMPID)
        npar    AIC    BIC  logLik deviance  Chisq Df Pr(>Chisq)  
model.2    5 5540.7 5568.9 -2765.4   5530.7                       
model.3    6 5539.6 5573.3 -2763.8   5527.6 3.1706  1    0.07498 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Using the modelsummary and broom.mixed Packages to Organize Your Results:

Model 1 Model 2 Model 3 Model 4 Model 5 Model 6
(Intercept) 0.906 1.966 2.567 2.894 3.407 3.521
(0.043) (0.076) (0.088) (0.203) (0.258) (0.310)
sd__(Intercept) 0.240 0.168 0.142 0.142 0.100 0.099
sd__Observation 1.008 0.959 0.930 0.930 0.931 0.931
TSIG -0.332 -0.186 -0.294 -0.172 -0.175
(0.021) (0.024) (0.065) (0.025) (0.025)
LEAD -0.350 -0.472 -0.352 -0.346
(0.030) (0.075) (0.030) (0.031)
TSIG × LEAD 0.039
(0.022)
GTSIG -0.277 -0.250
(0.083) (0.093)
GLEAD -0.070
(0.107)
AIC 5888.9 5668.4 5540.7 5539.6 5533.7 5535.2
BIC 5905.8 5690.9 5568.9 5573.3 5567.4 5574.6
Log.Lik. -2941.475 -2830.193 -2765.375 -2763.789 -2760.833 -2760.621

HTML Version That You Can Open in Word:

modelsummary(models, output = 'msum.html', title = 'MLM Estimates')
[WARNING] This document format requires a nonempty <title> element.
  Please specify either 'title' or 'pagetitle' in the metadata.
  Falling back to 'msum'
LS0tCnRpdGxlOiAnTXVsdGlsZXZlbCBNb2RlbGluZywgV2VlayA0OiBDb25kaXRpb25hbCBSYW5kb20gSW50ZXJjZXB0IE1vZGVscycKYXV0aG9yOiAiRHIuIEJyb2RhIgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCgpUaGlzIHdlZWssIHdlIGFyZSBnb2luZyB0byB1c2UgZGF0YSBmcm9tIEdhdmluIGFuZCBIb2ZtYW5uICgyMDAyKSwgYSBzdHVkeSBvbiBvcmdhbml6YXRpb25hbCBjbGltYXRlIGFuZCBhdHRpdHVkZXMgcHVibGlzaGVkIGluIExlYWRlcnNoaXAgUXVhcnRlcmx5LiBIZXJlLCB3ZSBoYXZlIGluZGl2aWR1YWxzIHNvbGRpZXJzIG5lc3RlZCB3aXRoaW4gY29tcGFuaWVzLiBUaGlzIGlzIHRoZSBzYW1lIGRhdGFzZXQgdGhhdCBHYXJzb24gdXNlcyBpbiBDaGFwdGVyIDYsIHNvIHlvdSBjYW4gcmVjcmVhdGUgaGlzIGFuYWx5c2lzLgoKIyBMb2FkIFNvbWUgUGFja2FnZXMgdG8gSGVscCB3aXRoIHRoZSBBbmFseXNpcyBhbmQgRGF0YSBNYW5hZ2VtZW50OgpgYGB7ciwgcHJpbnQ9RkFMU0V9CmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KHBzeWNoKQpsaWJyYXJ5KGxtZTQpCmBgYAoKIyBMb2FkIGFuZCBFeHBsb3JlIHRoZSBEYXRhCmBgYHtyfQoKbHEyMDAyIDwtIHJlYWRfY3N2KCJscTIwMDIuY3N2IikKZ2xpbXBzZShscTIwMDIpCgpgYGAKClJlbWVtYmVyIG91ciBvbGQgZnJpZW5kIGBkZXNjcmliZWAgZnJvbSB0aGUgYHBzeWNoYCBwYWNrYWdlIGEgZmV3IHdlZWtzIGFnbz8gVGhpcyBpcyBhIGdyZWF0IHdheSB0byBnZXQgYSBkZXRhaWxlZCBzdW1tYXJ5IG9mIGFsbCB0aGUgdmFyaWFibGVzIGluIGEgZGF0YXNldC4KCmBgYHtyfQpwc3ljaDo6ZGVzY3JpYmUobHEyMDAyLCAKICAgICAgICAgICAgICAgIGZhc3QgPSBUUlVFKQpgYGAKCiMgQ2FsY3VsaW5nIFNjYWxlIFNjb3JlcyBhbmQgQ3JvbmJhY2gncyBBbHBoYQoKVGhpcyB3ZWVrLCB3ZSBhcmUgZ29pbmcgdG8gZ2V0IGEgY3Jhc2ggY291cnNlIGluIGhvdyB0byBjYWxjdWxhdGUgcmVsaWFiaWxpdHkgdXNpbmcgQ3JvbmJhY2gncyBhbHBoYS4gVGhlIGBhbHBoYWAgZnVuY3Rpb24gaW4gdGhlIGBwc3ljaGAgcGFja2FnZSAgaXMgYSBxdWljayBhbmQgZWFzeSB3YXkgdG8gY2FsY3VsYXRlIGFscGhhOgoKYGBge3J9Cmhvc3RpbGVfaXRlbXMgPC0gbHEyMDAyICU+JSAKICBzZWxlY3QoLiwgSE9TVElMMDEsCiAgICAgICAgIEhPU1RJTDAyLAogICAgICAgICBIT1NUSUwwMywKICAgICAgICAgSE9TVElMMDQsCiAgICAgICAgIEhPU1RJTDA1KQoKYWxwaGEoaG9zdGlsZV9pdGVtcykKYGBgCgpOb3csIHdlIGNhbiBjcmVhdGUgYSBuZXcgdmFyaWFibGUsIGBob3N0aWxlYCwgdGhhdCBpcyBjb25zdHJ1Y3RlZCBieSB0YWtpbmcgdGhlIG1lYW4gb2Ygb3VyIDUgYGhvc3RpbGVgIGl0ZW1zLiAKCmBgYHtyfQpteS5rZXlzLmxpc3QgPC0gbGlzdChob3N0aWxlID0gYygiSE9TVElMMDEiLCJIT1NUSUwwMiIsIkhPU1RJTDAzIiwgIkhPU1RJTDA0IiwiSE9TVElMMDUiKSkKICAgICAgICAgICAgICAgICAgICAgCm15LnNjYWxlcyA8LSBzY29yZUl0ZW1zKG15LmtleXMubGlzdCwgbHEyMDAyLCBpbXB1dGUgPSAibm9uZSIpCmBgYAoKIyBTY2FsZSByZWxpYWJpbGl0eSBlc3RpbWF0ZXMsIGV0Yy4gZm9yIExpa2VydC10eXBlIHNjYWxlcwpgYGB7cn0KcHJpbnQobXkuc2NhbGVzLCBzaG9ydCA9IEZBTFNFKQpgYGAKCiMgU2F2ZSBTY29yZWQgU2NhbGVzIGFzIE5ldyBWYXJpYWJsZXMKYGBge3J9Cm15LnNjb3JlcyA8LSBhc190aWJibGUobXkuc2NhbGVzJHNjb3JlcykKYGBgCgojIEF0dGFjaCBOZXcgVmFyaWFibGUgdG8gT2xkIERhdGEgU2V0IChMUTIwMDIpCmBgYHtyfQpscTIwMDIuMSA8LSBiaW5kX2NvbHMobHEyMDAyLCBteS5zY29yZXMpCgpnbGltcHNlKGxxMjAwMi4xKQpgYGAKCiMgVmlzdWFsaXppbmcgdGhlIEtleSBWYXJpYWJsZXMKTGV0J3MgdGFrZSBhIGxvb2sgYXQgc29tZSBvZiB0aGUga2V5IHZhcmlhYmxlcyBmb3Igb3VyIGFuYWx5c2lzLiBXZSBoYXZlIHRocmVlIHZhcmlhYmxlcywgYWxsIHdoaWNoIGFyZSBzY2FsZSBzY29yZXMgY29uc3RydWN0ZWQganVzdCBsaWtlIHdlIGRpZCBmb3IgdGhlIGV4YW1wbGUgYWJvdmUuCgpgYGB7cn0KaGlzdChscTIwMDIkSE9TVElMRSwgCiAgICAgbWFpbiA9ICJEaXN0cmlidXRpb24gb2YgSG9zdGlsaXR5IFNjb3JlcyIsCiAgICAgc3ViID0gIk4gPSAyLDA0MiBVLlMuIEFybXkgYWN0aXZlIGR1dHkgc29saWRlcnMuIERhdGEgZnJvbSBHYXZpbiBhbmQgSG9mbWFubiAoMjAwMikuIiwKICAgICB4bGFiID0gIlNjYWxlZCBIb3N0aWxpdHkgU2NvcmUiKQpgYGAKCmBgYHtyfQpoaXN0KGxxMjAwMiRUU0lHLCAKICAgICBtYWluID0gIkRpc3RyaWJ1dGlvbiBvZiBUYXNrIFNpZ25pZmljYW5jZSBTY29yZXMiLAogICAgIHN1YiA9ICJOID0gMiwwNDIgVS5TLiBBcm15IGFjdGl2ZSBkdXR5IHNvbGlkZXJzLiBEYXRhIGZyb20gR2F2aW4gYW5kIEhvZm1hbm4gKDIwMDIpLiIsCiAgICAgeGxhYiA9ICJTY2FsZWQgVGFzayBTaWduaWZpY2FuY2UgU2NvcmUiKQpgYGAKCmBgYHtyfQpoaXN0KGxxMjAwMiRMRUFELCAKICAgICBtYWluID0gIkRpc3RyaWJ1dGlvbiBvZiBMZWFkZXJzaGlwIENsaW1hdGUgU2NvcmVzIiwKICAgICBzdWIgPSAiTiA9IDIsMDQyIFUuUy4gQXJteSBhY3RpdmUgZHV0eSBzb2xpZGVycy4gRGF0YSBmcm9tIEdhdmluIGFuZCBIb2ZtYW5uICgyMDAyKS4iLAogICAgIHhsYWIgPSAiU2NhbGVkIExlYWRlcnNoaXAgQ2xpbWF0ZSBTY29yZSIpCmBgYAoKIyBSdW5uaW5nIHRoZSBNdWx0aWxldmVsIE1vZGVscwoKIyMgU3RlcCBPbmU6IFRoZSBOdWxsIE1vZGVsCgpgYGB7cn0KbW9kZWxzIDwtIGxpc3QoKQptb2RlbC4wIDwtIGxtZXIoSE9TVElMRSB+ICgxfENPTVBJRCksIFJFTUwgPSBGQUxTRSwgZGF0YSA9IGxxMjAwMi4xKQpzdW1tYXJ5KG1vZGVsLjApCmBgYAoKIyMjIFBvc3QtRXN0aW1hdGlvbiBUYXNrOiBDYWxjdWxhdGUgSUNDCgpIZXJlJ3MgYSBxdWljayBhbmQgZWFzeSB3YXkgdG8gZ2V0IG91ciBmcmllbmQgdGhlIElDQyEKYGBge3J9CgpudWxsLklDQyA8LSAwLjA1NzY1LygwLjA1NzY1ICsgMS4wMTY2OCkKbnVsbC5JQ0MKCmBgYAoKIyMjIFVzaW5nIGBsbWVyVGVzdGAgdG8gRXZhbHVhdGUgUmFuZG9tIEVmZmVjdHM6CmBgYHtyfQpsbWVyVGVzdDo6cmFuZChtb2RlbC4wKQpgYGAKCiMjIFN0ZXAgVHdvOiBBZGRpbmcgYSBMZXZlbCBPbmUgUHJlZGljdG9yLCBgVFNJR2AKCmBgYHtyfQptb2RlbC4xIDwtIGxtZXIoSE9TVElMRSB+IFRTSUcgKyAoMXxDT01QSUQpLCBSRU1MID0gRkFMU0UsIGRhdGEgPSBscTIwMDIuMSkKc3VtbWFyeShtb2RlbC4xKQpgYGAKCiMjIFN0ZXAgVGhyZWU6IEFkZGluZyBhbiBBZGRpdGlvbmFsIExldmVsIE9uZSBQcmVkaWN0b3IsIGBMRUFEYApgYGB7cn0KbW9kZWwuMiA8LSBsbWVyKEhPU1RJTEUgfiBUU0lHICsgTEVBRCArICgxfENPTVBJRCksIFJFTUwgPSBGQUxTRSwgZGF0YSA9IGxxMjAwMi4xKQpzdW1tYXJ5KG1vZGVsLjIpCmBgYAoKIyMgU3RlcCBGb3VyOiBUZXN0aW5nIHRoZSBJbnRlcmFjdGlvbiBvZiBgTEVBRGAgYW5kIGBUU0lHYApgYGB7cn0KbW9kZWwuMyA8LSBsbWVyKEhPU1RJTEUgfiBUU0lHICsgTEVBRCArIFRTSUc6TEVBRCArICgxfENPTVBJRCksIFJFTUwgPSBGQUxTRSwgZGF0YSA9IGxxMjAwMi4xKQpzdW1tYXJ5KG1vZGVsLjMpCmBgYAoKCiMjIFN0ZXAgRml2ZTogQWRkaW5nIGEgTGV2ZWwgVHdvIFByZWRpY3RvciwgYEdUU0lHYApgYGB7cn0KbW9kZWwuNCA8LSBsbWVyKEhPU1RJTEUgfiBUU0lHICsgTEVBRCArIEdUU0lHICsgKDF8Q09NUElEKSwgUkVNTCA9IEZBTFNFLCBkYXRhID0gbHEyMDAyLjEpCnN1bW1hcnkobW9kZWwuNCkKYGBgCgojIyBTdGVwIFNpeDogQWRkaW5nIGEgMm5kIExldmVsIFR3byBQcmVkaWN0b3IsIGBHTEVBRGAKYGBge3J9Cm1vZGVsLjUgPC0gbG1lcihIT1NUSUxFIH4gVFNJRyArIExFQUQgKyBHVFNJRyArIEdMRUFEICsgKDF8Q09NUElEKSwgUkVNTCA9IEZBTFNFLCBkYXRhID0gbHEyMDAyLjEpCnN1bW1hcnkobW9kZWwuNSkKYGBgCgojIFNob3VsZCBXZSBJbmNsdWRlIFRoYXQgSW50ZXJhY3Rpb24/IENvbXBhcmluZyBgbW9kZWwuMmAgd2l0aCBgbW9kZWwuM2A6CmBgYHtyfQphbm92YShtb2RlbC4yLCBtb2RlbC4zKQpgYGAKCiMgVXNpbmcgdGhlIGBtb2RlbHN1bW1hcnlgIGFuZCBgYnJvb20ubWl4ZWRgIFBhY2thZ2VzIHRvIE9yZ2FuaXplIFlvdXIgUmVzdWx0czoKYGBge3J9CmxpYnJhcnkobW9kZWxzdW1tYXJ5KQpsaWJyYXJ5KGJyb29tLm1peGVkKQoKbW9kZWxzIDwtIGxpc3QobW9kZWwuMCwgbW9kZWwuMSwgbW9kZWwuMiwgbW9kZWwuMywgbW9kZWwuNCwgbW9kZWwuNSkKbW9kZWxzdW1tYXJ5KG1vZGVscykKYGBgCgojIEhUTUwgVmVyc2lvbiBUaGF0IFlvdSBDYW4gT3BlbiBpbiBXb3JkOgpgYGB7cn0KbW9kZWxzdW1tYXJ5KG1vZGVscywgb3V0cHV0ID0gJ21zdW0uaHRtbCcsIHRpdGxlID0gJ01MTSBFc3RpbWF0ZXMnKQoKYGBgCgo=