If we want to be able to make comparisons across groups, we must show our latent variable(s) is functioning the same way across groups. This is referred to as measurement invariance. If the model is not functioning the same way across the groups, then it makes no sense to say, for example, that one group has a higher mean with respect to the measured construct than another group because the constructs are functioning differently. Similarly, if we want to show that it’s fair to compare scores on a construct at time 1 with scores on that construct at time 2, we need demonstrate measurement invariance.

We implicitly assume measurement invariance, when we make comparsions on some construct across groups. We are assuming that the construct functions the same way when we decide to do regressions, t-test, mixed effects model, etc.

We’ll use the Holzinger data set in lavaan, to show how to assess measurement invariance. Here’s the description of the data set:

The classic Holzinger and Swineford (1939) dataset consists of mental ability test scores of seventh- and eighth-grade children from two different schools (Pasteur and Grant-White). In the original dataset (available in the MBESS package), there are scores for 26 tests. However, a smaller subset with 9 variables is more widely used in the literature (for example in Joreskog’s 1969 paper, which also uses the 145 subjects from the Grant-White school only).

library("lavaan")
This is lavaan 0.5-20
lavaan is BETA software! Please report any bugs.
data("HolzingerSwineford1939")
head(HolzingerSwineford1939)
  id sex ageyr agemo  school grade       x1   x2    x3       x4   x5        x6       x7   x8       x9
1  1   1    13     1 Pasteur     7 3.333333 7.75 0.375 2.333333 5.75 1.2857143 3.391304 5.75 6.361111
2  2   2    13     7 Pasteur     7 5.333333 5.25 2.125 1.666667 3.00 1.2857143 3.782609 6.25 7.916667
3  3   2    13     1 Pasteur     7 4.500000 5.25 1.875 1.000000 1.75 0.4285714 3.260870 3.90 4.416667
4  4   1    13     2 Pasteur     7 5.333333 7.75 3.000 2.666667 4.50 2.4285714 3.000000 5.30 4.861111
5  5   2    12     2 Pasteur     7 4.833333 4.75 0.875 2.666667 4.00 2.5714286 3.695652 6.30 5.916667
6  6   2    14     1 Pasteur     7 5.333333 5.00 2.250 1.000000 3.00 0.8571429 4.347826 6.65 7.500000

What we are testing is whether our model is invariant across these two schools.

Subtests x1, x2, and x3 are manifestations of the visual factor; x4, x5, and x6 are manifestations of the textual factor; and x7, x8, and x9 are manifestations of the speed factor. This is what our model looks like overall (ignoring the schools).

HS.model <- ' visual  =~ x1 + x2 + x3
              textual =~ x4 + x5 + x6
              speed   =~ x7 + x8 + x9 '
fit <- cfa(HS.model, data=HolzingerSwineford1939)
summary(fit, fit.measures=TRUE)
lavaan (0.5-20) converged normally after  35 iterations

  Number of observations                           301

  Estimator                                         ML
  Minimum Function Test Statistic               85.306
  Degrees of freedom                                24
  P-value (Chi-square)                           0.000

Model test baseline model:

  Minimum Function Test Statistic              918.852
  Degrees of freedom                                36
  P-value                                        0.000

User model versus baseline model:

  Comparative Fit Index (CFI)                    0.931
  Tucker-Lewis Index (TLI)                       0.896

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)              -3737.745
  Loglikelihood unrestricted model (H1)      -3695.092

  Number of free parameters                         21
  Akaike (AIC)                                7517.490
  Bayesian (BIC)                              7595.339
  Sample-size adjusted Bayesian (BIC)         7528.739

Root Mean Square Error of Approximation:

  RMSEA                                          0.092
  90 Percent Confidence Interval          0.071  0.114
  P-value RMSEA <= 0.05                          0.001

Standardized Root Mean Square Residual:

  SRMR                                           0.065

Parameter Estimates:

  Information                                 Expected
  Standard Errors                             Standard

Latent Variables:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual =~                                           
    x1                1.000                           
    x2                0.554    0.100    5.554    0.000
    x3                0.729    0.109    6.685    0.000
  textual =~                                          
    x4                1.000                           
    x5                1.113    0.065   17.014    0.000
    x6                0.926    0.055   16.703    0.000
  speed =~                                            
    x7                1.000                           
    x8                1.180    0.165    7.152    0.000
    x9                1.082    0.151    7.155    0.000

Covariances:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual ~~                                           
    textual           0.408    0.074    5.552    0.000
    speed             0.262    0.056    4.660    0.000
  textual ~~                                          
    speed             0.173    0.049    3.518    0.000

Variances:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                0.549    0.114    4.833    0.000
    x2                1.134    0.102   11.146    0.000
    x3                0.844    0.091    9.317    0.000
    x4                0.371    0.048    7.779    0.000
    x5                0.446    0.058    7.642    0.000
    x6                0.356    0.043    8.277    0.000
    x7                0.799    0.081    9.823    0.000
    x8                0.488    0.074    6.573    0.000
    x9                0.566    0.071    8.003    0.000
    visual            0.809    0.145    5.564    0.000
    textual           0.979    0.112    8.737    0.000
    speed             0.384    0.086    4.451    0.000

Our model has adequate fit and the fit statistics are just ok. They are not awful but if this was your data set, you would hope for a better model fit or you might consider modifications (e.g. removing/swapping in indicators, allowing residuals to correlate, etc).

Configural model

The first step in demonstrating measurement invariance is to show that the model fits for each of our groups. This can be done two ways. 1) You could fit the data to each subset of the data or 2) you can use a multiple group model. We’ll use this second apprach and to do this in lavaan, we simply add the group = "school" argument and refit our model. This argument will allow all of our parameters to differ across the groups.

Note, it is inappropriate to use standardize coefficients here. The reason is that we have no reason to expect the variances are the same across the groups and

configural <- cfa(HS.model, data=HolzingerSwineford1939, group = "school")
summary(configural, fit.measures=TRUE)
lavaan (0.5-20) converged normally after  57 iterations

  Number of observations per group         
  Pasteur                                          156
  Grant-White                                      145

  Estimator                                         ML
  Minimum Function Test Statistic              115.851
  Degrees of freedom                                48
  P-value (Chi-square)                           0.000

Chi-square for each group:

  Pasteur                                       64.309
  Grant-White                                   51.542

Model test baseline model:

  Minimum Function Test Statistic              957.769
  Degrees of freedom                                72
  P-value                                        0.000

User model versus baseline model:

  Comparative Fit Index (CFI)                    0.923
  Tucker-Lewis Index (TLI)                       0.885

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)              -3682.198
  Loglikelihood unrestricted model (H1)      -3624.272

  Number of free parameters                         60
  Akaike (AIC)                                7484.395
  Bayesian (BIC)                              7706.822
  Sample-size adjusted Bayesian (BIC)         7516.536

Root Mean Square Error of Approximation:

  RMSEA                                          0.097
  90 Percent Confidence Interval          0.075  0.120
  P-value RMSEA <= 0.05                          0.001

Standardized Root Mean Square Residual:

  SRMR                                           0.068

Parameter Estimates:

  Information                                 Expected
  Standard Errors                             Standard


Group 1 [Pasteur]:

Latent Variables:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual =~                                           
    x1                1.000                           
    x2                0.394    0.122    3.220    0.001
    x3                0.570    0.140    4.076    0.000
  textual =~                                          
    x4                1.000                           
    x5                1.183    0.102   11.613    0.000
    x6                0.875    0.077   11.421    0.000
  speed =~                                            
    x7                1.000                           
    x8                1.125    0.277    4.057    0.000
    x9                0.922    0.225    4.104    0.000

Covariances:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual ~~                                           
    textual           0.479    0.106    4.531    0.000
    speed             0.185    0.077    2.397    0.017
  textual ~~                                          
    speed             0.182    0.069    2.628    0.009

Intercepts:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                4.941    0.095   52.249    0.000
    x2                5.984    0.098   60.949    0.000
    x3                2.487    0.093   26.778    0.000
    x4                2.823    0.092   30.689    0.000
    x5                3.995    0.105   38.183    0.000
    x6                1.922    0.079   24.321    0.000
    x7                4.432    0.087   51.181    0.000
    x8                5.563    0.078   71.214    0.000
    x9                5.418    0.079   68.440    0.000
    visual            0.000                           
    textual           0.000                           
    speed             0.000                           

Variances:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                0.298    0.232    1.286    0.198
    x2                1.334    0.158    8.464    0.000
    x3                0.989    0.136    7.271    0.000
    x4                0.425    0.069    6.138    0.000
    x5                0.456    0.086    5.292    0.000
    x6                0.290    0.050    5.780    0.000
    x7                0.820    0.125    6.580    0.000
    x8                0.510    0.116    4.406    0.000
    x9                0.680    0.104    6.516    0.000
    visual            1.097    0.276    3.967    0.000
    textual           0.894    0.150    5.963    0.000
    speed             0.350    0.126    2.778    0.005


Group 2 [Grant-White]:

Latent Variables:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual =~                                           
    x1                1.000                           
    x2                0.736    0.155    4.760    0.000
    x3                0.925    0.166    5.583    0.000
  textual =~                                          
    x4                1.000                           
    x5                0.990    0.087   11.418    0.000
    x6                0.963    0.085   11.377    0.000
  speed =~                                            
    x7                1.000                           
    x8                1.226    0.187    6.569    0.000
    x9                1.058    0.165    6.429    0.000

Covariances:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual ~~                                           
    textual           0.408    0.098    4.153    0.000
    speed             0.276    0.076    3.639    0.000
  textual ~~                                          
    speed             0.222    0.073    3.022    0.003

Intercepts:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                4.930    0.095   51.696    0.000
    x2                6.200    0.092   67.416    0.000
    x3                1.996    0.086   23.195    0.000
    x4                3.317    0.093   35.625    0.000
    x5                4.712    0.096   48.986    0.000
    x6                2.469    0.094   26.277    0.000
    x7                3.921    0.086   45.819    0.000
    x8                5.488    0.087   63.174    0.000
    x9                5.327    0.085   62.571    0.000
    visual            0.000                           
    textual           0.000                           
    speed             0.000                           

Variances:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                0.715    0.126    5.676    0.000
    x2                0.899    0.123    7.339    0.000
    x3                0.557    0.103    5.409    0.000
    x4                0.315    0.065    4.870    0.000
    x5                0.419    0.072    5.812    0.000
    x6                0.406    0.069    5.880    0.000
    x7                0.600    0.091    6.584    0.000
    x8                0.401    0.094    4.249    0.000
    x9                0.535    0.089    6.010    0.000
    visual            0.604    0.160    3.762    0.000
    textual           0.942    0.152    6.177    0.000
    speed             0.461    0.118    3.910    0.000

The model again fits just ok. At least it doesn’t fit worse when we fit a multiple group model. The configural model is a prerequisite to investigating measurement invariance.

Weak Invariance

To assess for weak (metric) invariance, we need to constrain the factor loadings to be equal across the groups. This shows that the factor(s) has the same meaning across the groups. Are they attributing the same meaning to the latent constructs? If we don’t have weak invariance, they this implies the the meaning of the manifest variables are different across the groups. For example, on the Minnesota Student Survey, there are many items that ask about bullying and a specific item that asks about frequency of being bullied because of your race/ethnic group. Racism functions differently for white students than non-white students and because of this, this item ends up not being invariant because white students are much more likely to say never whereas non-white students would be more likely to say it occurs more frequently.

At this stage, and in future steps, if you have only partial invariance of the factor loadings, constrain these invariant loadings to be equal and allow the other, ones to be freed. At this point, we are yet not estimating the factor means. This happens in the next step.

You fit this model in lavaan using the group.equal argument.

weak.invariance <- cfa(HS.model, data=HolzingerSwineford1939, group = "school", group.equal = "loadings")
summary(weak.invariance, fit.measures = TRUE)
lavaan (0.5-20) converged normally after  42 iterations

  Number of observations per group         
  Pasteur                                          156
  Grant-White                                      145

  Estimator                                         ML
  Minimum Function Test Statistic              124.044
  Degrees of freedom                                54
  P-value (Chi-square)                           0.000

Chi-square for each group:

  Pasteur                                       68.825
  Grant-White                                   55.219

Model test baseline model:

  Minimum Function Test Statistic              957.769
  Degrees of freedom                                72
  P-value                                        0.000

User model versus baseline model:

  Comparative Fit Index (CFI)                    0.921
  Tucker-Lewis Index (TLI)                       0.895

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)              -3686.294
  Loglikelihood unrestricted model (H1)      -3624.272

  Number of free parameters                         54
  Akaike (AIC)                                7480.587
  Bayesian (BIC)                              7680.771
  Sample-size adjusted Bayesian (BIC)         7509.514

Root Mean Square Error of Approximation:

  RMSEA                                          0.093
  90 Percent Confidence Interval          0.071  0.114
  P-value RMSEA <= 0.05                          0.001

Standardized Root Mean Square Residual:

  SRMR                                           0.072

Parameter Estimates:

  Information                                 Expected
  Standard Errors                             Standard


Group 1 [Pasteur]:

Latent Variables:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual =~                                           
    x1                1.000                           
    x2      (.p2.)    0.599    0.100    5.979    0.000
    x3      (.p3.)    0.784    0.108    7.267    0.000
  textual =~                                          
    x4                1.000                           
    x5      (.p5.)    1.083    0.067   16.049    0.000
    x6      (.p6.)    0.912    0.058   15.785    0.000
  speed =~                                            
    x7                1.000                           
    x8      (.p8.)    1.201    0.155    7.738    0.000
    x9      (.p9.)    1.038    0.136    7.629    0.000

Covariances:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual ~~                                           
    textual           0.416    0.097    4.271    0.000
    speed             0.169    0.064    2.643    0.008
  textual ~~                                          
    speed             0.176    0.061    2.882    0.004

Intercepts:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                4.941    0.093   52.991    0.000
    x2                5.984    0.100   60.096    0.000
    x3                2.487    0.094   26.465    0.000
    x4                2.823    0.093   30.371    0.000
    x5                3.995    0.101   39.714    0.000
    x6                1.922    0.081   23.711    0.000
    x7                4.432    0.086   51.540    0.000
    x8                5.563    0.078   71.087    0.000
    x9                5.418    0.079   68.153    0.000
    visual            0.000                           
    textual           0.000                           
    speed             0.000                           

Variances:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                0.551    0.137    4.010    0.000
    x2                1.258    0.155    8.117    0.000
    x3                0.882    0.128    6.884    0.000
    x4                0.434    0.070    6.238    0.000
    x5                0.508    0.082    6.229    0.000
    x6                0.266    0.050    5.294    0.000
    x7                0.849    0.114    7.468    0.000
    x8                0.515    0.095    5.409    0.000
    x9                0.658    0.096    6.865    0.000
    visual            0.805    0.171    4.714    0.000
    textual           0.913    0.137    6.651    0.000
    speed             0.305    0.078    3.920    0.000


Group 2 [Grant-White]:

Latent Variables:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual =~                                           
    x1                1.000                           
    x2      (.p2.)    0.599    0.100    5.979    0.000
    x3      (.p3.)    0.784    0.108    7.267    0.000
  textual =~                                          
    x4                1.000                           
    x5      (.p5.)    1.083    0.067   16.049    0.000
    x6      (.p6.)    0.912    0.058   15.785    0.000
  speed =~                                            
    x7                1.000                           
    x8      (.p8.)    1.201    0.155    7.738    0.000
    x9      (.p9.)    1.038    0.136    7.629    0.000

Covariances:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual ~~                                           
    textual           0.437    0.099    4.423    0.000
    speed             0.314    0.079    3.958    0.000
  textual ~~                                          
    speed             0.226    0.072    3.144    0.002

Intercepts:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                4.930    0.097   50.763    0.000
    x2                6.200    0.091   68.379    0.000
    x3                1.996    0.085   23.455    0.000
    x4                3.317    0.092   35.950    0.000
    x5                4.712    0.100   47.173    0.000
    x6                2.469    0.091   27.248    0.000
    x7                3.921    0.086   45.555    0.000
    x8                5.488    0.087   63.257    0.000
    x9                5.327    0.085   62.786    0.000
    visual            0.000                           
    textual           0.000                           
    speed             0.000                           

Variances:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                0.645    0.127    5.084    0.000
    x2                0.933    0.121    7.732    0.000
    x3                0.605    0.096    6.282    0.000
    x4                0.329    0.062    5.279    0.000
    x5                0.384    0.073    5.270    0.000
    x6                0.437    0.067    6.576    0.000
    x7                0.599    0.090    6.651    0.000
    x8                0.406    0.089    4.541    0.000
    x9                0.532    0.086    6.202    0.000
    visual            0.722    0.161    4.490    0.000
    textual           0.906    0.136    6.646    0.000
    speed             0.475    0.109    4.347    0.000

All of the measurement invariance models are nested within one another. The weak invariance model is nested within the configural model. Therefore, we can do a chi-square difference test.

anova(weak.invariance, configural)
Chi Square Difference Test

                Df    AIC    BIC  Chisq Chisq diff Df diff Pr(>Chisq)
configural      48 7484.4 7706.8 115.85                              
weak.invariance 54 7480.6 7680.8 124.04     8.1922       6     0.2244

The null hypothesis is that the configural model is not an improvement in fit over the weak invariance model. We fail to reject the null hypothesis. This is evidence that we have weak invariance. We can also investigate other fit statistics.

fit.stats <- rbind(fitmeasures(configural, fit.measures = c("chisq", "df", "rmsea", "tli", "cfi", "aic")),
fitmeasures(weak.invariance, fit.measures = c("chisq", "df", "rmsea", "tli", "cfi", "aic")))
rownames(fit.stats) <- c("configural", "weak invariance")
fit.stats
                   chisq df      rmsea       tli       cfi      aic
configural      115.8513 48 0.09691486 0.8850976 0.9233984 7484.395
weak invariance 124.0435 54 0.09283654 0.8945646 0.9209235 7480.587

The fit statistics haven’t really changed, but recall that the configural model is the model were all the parameters are all freely estimated across the groups, so the weak invariance model is the more parsimonious model estimating 6 fewer parameters. This is evidence that we have weak invariance. Based on this, we can safely conclude that we have weak invariance.

Strong Invariance

Strong (scalar) invariance adds the additional constraint that intercepts are equal across groups. Now we have both loadings and intercepts held constrant across groups.This implies that the meaning of the construct (the factor loadings), and the levels of the underlying manifest variables (intercepts) are equal in both groups. If we have strong invariance, we are able to compare differences in the mean across the latent construct(s). In this stage we estimate the factor means.

strong.invariance <- cfa(HS.model, data=HolzingerSwineford1939, group = "school", group.equal = c( "loadings", "intercepts"))
summary(strong.invariance, fit.measures = TRUE)
lavaan (0.5-20) converged normally after  60 iterations

  Number of observations per group         
  Pasteur                                          156
  Grant-White                                      145

  Estimator                                         ML
  Minimum Function Test Statistic              164.103
  Degrees of freedom                                60
  P-value (Chi-square)                           0.000

Chi-square for each group:

  Pasteur                                       90.210
  Grant-White                                   73.892

Model test baseline model:

  Minimum Function Test Statistic              957.769
  Degrees of freedom                                72
  P-value                                        0.000

User model versus baseline model:

  Comparative Fit Index (CFI)                    0.882
  Tucker-Lewis Index (TLI)                       0.859

Loglikelihood and Information Criteria:

  Loglikelihood user model (H0)              -3706.323
  Loglikelihood unrestricted model (H1)      -3624.272

  Number of free parameters                         48
  Akaike (AIC)                                7508.647
  Bayesian (BIC)                              7686.588
  Sample-size adjusted Bayesian (BIC)         7534.359

Root Mean Square Error of Approximation:

  RMSEA                                          0.107
  90 Percent Confidence Interval          0.088  0.127
  P-value RMSEA <= 0.05                          0.000

Standardized Root Mean Square Residual:

  SRMR                                           0.082

Parameter Estimates:

  Information                                 Expected
  Standard Errors                             Standard


Group 1 [Pasteur]:

Latent Variables:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual =~                                           
    x1                1.000                           
    x2      (.p2.)    0.576    0.101    5.713    0.000
    x3      (.p3.)    0.798    0.112    7.146    0.000
  textual =~                                          
    x4                1.000                           
    x5      (.p5.)    1.120    0.066   16.965    0.000
    x6      (.p6.)    0.932    0.056   16.608    0.000
  speed =~                                            
    x7                1.000                           
    x8      (.p8.)    1.130    0.145    7.786    0.000
    x9      (.p9.)    1.009    0.132    7.667    0.000

Covariances:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual ~~                                           
    textual           0.410    0.095    4.293    0.000
    speed             0.178    0.066    2.687    0.007
  textual ~~                                          
    speed             0.180    0.062    2.900    0.004

Intercepts:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1      (.25.)    5.001    0.090   55.760    0.000
    x2      (.26.)    6.151    0.077   79.905    0.000
    x3      (.27.)    2.271    0.083   27.387    0.000
    x4      (.28.)    2.778    0.087   31.953    0.000
    x5      (.29.)    4.035    0.096   41.858    0.000
    x6      (.30.)    1.926    0.079   24.426    0.000
    x7      (.31.)    4.242    0.073   57.975    0.000
    x8      (.32.)    5.630    0.072   78.531    0.000
    x9      (.33.)    5.465    0.069   79.016    0.000
    visual            0.000                           
    textual           0.000                           
    speed             0.000                           

Variances:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                0.555    0.139    3.983    0.000
    x2                1.296    0.158    8.186    0.000
    x3                0.944    0.136    6.929    0.000
    x4                0.445    0.069    6.430    0.000
    x5                0.502    0.082    6.136    0.000
    x6                0.263    0.050    5.264    0.000
    x7                0.888    0.120    7.416    0.000
    x8                0.541    0.095    5.706    0.000
    x9                0.654    0.096    6.805    0.000
    visual            0.796    0.172    4.641    0.000
    textual           0.879    0.131    6.694    0.000
    speed             0.322    0.082    3.914    0.000


Group 2 [Grant-White]:

Latent Variables:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual =~                                           
    x1                1.000                           
    x2      (.p2.)    0.576    0.101    5.713    0.000
    x3      (.p3.)    0.798    0.112    7.146    0.000
  textual =~                                          
    x4                1.000                           
    x5      (.p5.)    1.120    0.066   16.965    0.000
    x6      (.p6.)    0.932    0.056   16.608    0.000
  speed =~                                            
    x7                1.000                           
    x8      (.p8.)    1.130    0.145    7.786    0.000
    x9      (.p9.)    1.009    0.132    7.667    0.000

Covariances:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual ~~                                           
    textual           0.427    0.097    4.417    0.000
    speed             0.329    0.082    4.006    0.000
  textual ~~                                          
    speed             0.236    0.073    3.224    0.001

Intercepts:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1      (.25.)    5.001    0.090   55.760    0.000
    x2      (.26.)    6.151    0.077   79.905    0.000
    x3      (.27.)    2.271    0.083   27.387    0.000
    x4      (.28.)    2.778    0.087   31.953    0.000
    x5      (.29.)    4.035    0.096   41.858    0.000
    x6      (.30.)    1.926    0.079   24.426    0.000
    x7      (.31.)    4.242    0.073   57.975    0.000
    x8      (.32.)    5.630    0.072   78.531    0.000
    x9      (.33.)    5.465    0.069   79.016    0.000
    visual           -0.148    0.122   -1.211    0.226
    textual           0.576    0.117    4.918    0.000
    speed            -0.177    0.090   -1.968    0.049

Variances:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                0.654    0.128    5.094    0.000
    x2                0.964    0.123    7.812    0.000
    x3                0.641    0.101    6.316    0.000
    x4                0.343    0.062    5.534    0.000
    x5                0.376    0.073    5.133    0.000
    x6                0.437    0.067    6.559    0.000
    x7                0.625    0.095    6.574    0.000
    x8                0.434    0.088    4.914    0.000
    x9                0.522    0.086    6.102    0.000
    visual            0.708    0.160    4.417    0.000
    textual           0.870    0.131    6.659    0.000
    speed             0.505    0.115    4.379    0.000

You’ll note that the latent means for visual, textual, and speed are now estimated. However, they shouldn’t be compared across group until we test if we have strong invariance.

The strong invariance model is nested within the weak invariance model. Therefore, we can again perform a chi-square difference test.

anova(strong.invariance, weak.invariance)
Chi Square Difference Test

                  Df    AIC    BIC  Chisq Chisq diff Df diff Pr(>Chisq)    
weak.invariance   54 7480.6 7680.8 124.04                                  
strong.invariance 60 7508.6 7686.6 164.10     40.059       6  4.435e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

We see that we reject the null hypothesis and that the weak invariance model fits better than the strong invariance model. This is also evident by the lower AIC and BIC. What we should now know is examine whether there is partial invariance. We can do this by using the lavTestScore() function in the lavaan function. This function allows us to see the effect of releasing equality constraints across the groups. The modindices() will only show modification indices for newly added parameters associated with new paths. These parameters aren’t newly estimated in the model, we’re just freeing them across groups.

lavTestScore(strong.invariance)
$test

total score test:

   test     X2 df p.value
1 score 46.956 15       0

$uni

univariate score tests:

     lhs op   rhs     X2 df p.value
1   .p2. == .p38.  0.306  1   0.580
2   .p3. == .p39.  1.636  1   0.201
3   .p5. == .p41.  2.744  1   0.098
4   .p6. == .p42.  2.627  1   0.105
5   .p8. == .p44.  0.027  1   0.871
6   .p9. == .p45.  0.004  1   0.952
7  .p25. == .p61.  5.847  1   0.016
8  .p26. == .p62.  6.863  1   0.009
9  .p27. == .p63. 19.193  1   0.000
10 .p28. == .p64.  2.139  1   0.144
11 .p29. == .p65.  1.563  1   0.211
12 .p30. == .p66.  0.032  1   0.857
13 .p31. == .p67. 15.021  1   0.000
14 .p32. == .p68.  4.710  1   0.030
15 .p33. == .p69.  1.498  1   0.221

What are all these .ps?

parTable(strong.invariance)
   id     lhs op     rhs user group free ustart exo label plabel start    est    se
1   1  visual =~      x1    1     1    0      1   0         .p1. 1.000  1.000 0.000
2   2  visual =~      x2    1     1    1     NA   0  .p2.   .p2. 0.769  0.576 0.101
3   3  visual =~      x3    1     1    2     NA   0  .p3.   .p3. 1.186  0.798 0.112
4   4 textual =~      x4    1     1    0      1   0         .p4. 1.000  1.000 0.000
5   5 textual =~      x5    1     1    3     NA   0  .p5.   .p5. 1.237  1.120 0.066
6   6 textual =~      x6    1     1    4     NA   0  .p6.   .p6. 0.865  0.932 0.056
7   7   speed =~      x7    1     1    0      1   0         .p7. 1.000  1.000 0.000
8   8   speed =~      x8    1     1    5     NA   0  .p8.   .p8. 1.227  1.130 0.145
9   9   speed =~      x9    1     1    6     NA   0  .p9.   .p9. 0.827  1.009 0.132
10 10      x1 ~~      x1    0     1    7     NA   0        .p10. 0.698  0.555 0.139
11 11      x2 ~~      x2    0     1    8     NA   0        .p11. 0.752  1.296 0.158
12 12      x3 ~~      x3    0     1    9     NA   0        .p12. 0.673  0.944 0.136
13 13      x4 ~~      x4    0     1   10     NA   0        .p13. 0.660  0.445 0.069
14 14      x5 ~~      x5    0     1   11     NA   0        .p14. 0.854  0.502 0.082
15 15      x6 ~~      x6    0     1   12     NA   0        .p15. 0.487  0.263 0.050
16 16      x7 ~~      x7    0     1   13     NA   0        .p16. 0.585  0.888 0.120
17 17      x8 ~~      x8    0     1   14     NA   0        .p17. 0.476  0.541 0.095
18 18      x9 ~~      x9    0     1   15     NA   0        .p18. 0.489  0.654 0.096
19 19  visual ~~  visual    0     1   16     NA   0        .p19. 0.050  0.796 0.172
20 20 textual ~~ textual    0     1   17     NA   0        .p20. 0.050  0.879 0.131
21 21   speed ~~   speed    0     1   18     NA   0        .p21. 0.050  0.322 0.082
22 22  visual ~~ textual    0     1   19     NA   0        .p22. 0.000  0.410 0.095
23 23  visual ~~   speed    0     1   20     NA   0        .p23. 0.000  0.178 0.066
24 24 textual ~~   speed    0     1   21     NA   0        .p24. 0.000  0.180 0.062
25 25      x1 ~1            0     1   22     NA   0 .p25.  .p25. 4.941  5.001 0.090
26 26      x2 ~1            0     1   23     NA   0 .p26.  .p26. 5.984  6.151 0.077
27 27      x3 ~1            0     1   24     NA   0 .p27.  .p27. 2.487  2.271 0.083
28 28      x4 ~1            0     1   25     NA   0 .p28.  .p28. 2.823  2.778 0.087
29 29      x5 ~1            0     1   26     NA   0 .p29.  .p29. 3.995  4.035 0.096
30 30      x6 ~1            0     1   27     NA   0 .p30.  .p30. 1.922  1.926 0.079
31 31      x7 ~1            0     1   28     NA   0 .p31.  .p31. 4.432  4.242 0.073
32 32      x8 ~1            0     1   29     NA   0 .p32.  .p32. 5.563  5.630 0.072
33 33      x9 ~1            0     1   30     NA   0 .p33.  .p33. 5.418  5.465 0.069
34 34  visual ~1            0     1    0      0   0        .p34. 0.000  0.000 0.000
35 35 textual ~1            0     1    0      0   0        .p35. 0.000  0.000 0.000
36 36   speed ~1            0     1    0      0   0        .p36. 0.000  0.000 0.000
37 37  visual =~      x1    1     2    0      1   0        .p37. 1.000  1.000 0.000
38 38  visual =~      x2    1     2   31     NA   0  .p2.  .p38. 0.896  0.576 0.101
39 39  visual =~      x3    1     2   32     NA   0  .p3.  .p39. 1.155  0.798 0.112
40 40 textual =~      x4    1     2    0      1   0        .p40. 1.000  1.000 0.000
41 41 textual =~      x5    1     2   33     NA   0  .p5.  .p41. 0.991  1.120 0.066
42 42 textual =~      x6    1     2   34     NA   0  .p6.  .p42. 0.962  0.932 0.056
43 43   speed =~      x7    1     2    0      1   0        .p43. 1.000  1.000 0.000
44 44   speed =~      x8    1     2   35     NA   0  .p8.  .p44. 1.282  1.130 0.145
45 45   speed =~      x9    1     2   36     NA   0  .p9.  .p45. 0.895  1.009 0.132
46 46      x1 ~~      x1    0     2   37     NA   0        .p46. 0.659  0.654 0.128
47 47      x2 ~~      x2    0     2   38     NA   0        .p47. 0.613  0.964 0.123
48 48      x3 ~~      x3    0     2   39     NA   0        .p48. 0.537  0.641 0.101
49 49      x4 ~~      x4    0     2   40     NA   0        .p49. 0.629  0.343 0.062
50 50      x5 ~~      x5    0     2   41     NA   0        .p50. 0.671  0.376 0.073
51 51      x6 ~~      x6    0     2   42     NA   0        .p51. 0.640  0.437 0.067
52 52      x7 ~~      x7    0     2   43     NA   0        .p52. 0.531  0.625 0.095
53 53      x8 ~~      x8    0     2   44     NA   0        .p53. 0.547  0.434 0.088
54 54      x9 ~~      x9    0     2   45     NA   0        .p54. 0.526  0.522 0.086
55 55  visual ~~  visual    0     2   46     NA   0        .p55. 0.050  0.708 0.160
56 56 textual ~~ textual    0     2   47     NA   0        .p56. 0.050  0.870 0.131
57 57   speed ~~   speed    0     2   48     NA   0        .p57. 0.050  0.505 0.115
58 58  visual ~~ textual    0     2   49     NA   0        .p58. 0.000  0.427 0.097
59 59  visual ~~   speed    0     2   50     NA   0        .p59. 0.000  0.329 0.082
60 60 textual ~~   speed    0     2   51     NA   0        .p60. 0.000  0.236 0.073
61 61      x1 ~1            0     2   52     NA   0 .p25.  .p61. 4.930  5.001 0.090
62 62      x2 ~1            0     2   53     NA   0 .p26.  .p62. 6.200  6.151 0.077
63 63      x3 ~1            0     2   54     NA   0 .p27.  .p63. 1.996  2.271 0.083
64 64      x4 ~1            0     2   55     NA   0 .p28.  .p64. 3.317  2.778 0.087
65 65      x5 ~1            0     2   56     NA   0 .p29.  .p65. 4.712  4.035 0.096
66 66      x6 ~1            0     2   57     NA   0 .p30.  .p66. 2.469  1.926 0.079
67 67      x7 ~1            0     2   58     NA   0 .p31.  .p67. 3.921  4.242 0.073
68 68      x8 ~1            0     2   59     NA   0 .p32.  .p68. 5.488  5.630 0.072
69 69      x9 ~1            0     2   60     NA   0 .p33.  .p69. 5.327  5.465 0.069
70 70  visual ~1            0     2   61     NA   0        .p70. 0.000 -0.148 0.122
71 71 textual ~1            0     2   62     NA   0        .p71. 0.000  0.576 0.117
72 72   speed ~1            0     2   63     NA   0        .p72. 0.000 -0.177 0.090
73 73    .p2. ==   .p38.    2     0    0     NA   0              0.000  0.000 0.000
74 74    .p3. ==   .p39.    2     0    0     NA   0              0.000  0.000 0.000
75 75    .p5. ==   .p41.    2     0    0     NA   0              0.000  0.000 0.000
76 76    .p6. ==   .p42.    2     0    0     NA   0              0.000  0.000 0.000
77 77    .p8. ==   .p44.    2     0    0     NA   0              0.000  0.000 0.000
78 78    .p9. ==   .p45.    2     0    0     NA   0              0.000  0.000 0.000
79 79   .p25. ==   .p61.    2     0    0     NA   0              0.000  0.000 0.000
80 80   .p26. ==   .p62.    2     0    0     NA   0              0.000  0.000 0.000
81 81   .p27. ==   .p63.    2     0    0     NA   0              0.000  0.000 0.000
82 82   .p28. ==   .p64.    2     0    0     NA   0              0.000  0.000 0.000
83 83   .p29. ==   .p65.    2     0    0     NA   0              0.000  0.000 0.000
84 84   .p30. ==   .p66.    2     0    0     NA   0              0.000  0.000 0.000
85 85   .p31. ==   .p67.    2     0    0     NA   0              0.000  0.000 0.000
86 86   .p32. ==   .p68.    2     0    0     NA   0              0.000  0.000 0.000
87 87   .p33. ==   .p69.    2     0    0     NA   0              0.000  0.000 0.000

The first output is a multivariate score test (i.e. Lagrange multipler test) and this is a test of whether freeing all equality constraints represents an improvement in fit over the base model. We reject the null hypothesis. Therefore, we should look at the univariate score tests (i.e., the chi-square difference tests) to see which equality constraints should be relaxed and these parameters should be freed. We see that we should free the x3 (this is the .p27 == .p63) intercept has the largest change in chi-square difference (X2). Let’s fit this model first.

strong.invariance.x3 <- cfa(HS.model, data=HolzingerSwineford1939, group = "school", group.equal = c( "loadings", "intercepts"), group.partial = c("x3 ~ 1"))
lavTestScore(strong.invariance.x3)
$test

total score test:

   test     X2 df p.value
1 score 27.528 14   0.016

$uni

univariate score tests:

     lhs op   rhs     X2 df p.value
1   .p2. == .p38.  0.734  1   0.392
2   .p3. == .p39.  0.485  1   0.486
3   .p5. == .p41.  2.760  1   0.097
4   .p6. == .p42.  2.630  1   0.105
5   .p8. == .p44.  0.026  1   0.872
6   .p9. == .p45.  0.002  1   0.960
7  .p25. == .p61.  2.833  1   0.092
8  .p26. == .p62.  2.833  1   0.092
9  .p28. == .p64.  2.136  1   0.144
10 .p29. == .p65.  1.560  1   0.212
11 .p30. == .p66.  0.032  1   0.857
12 .p31. == .p67. 15.023  1   0.000
13 .p32. == .p68.  4.727  1   0.030
14 .p33. == .p69.  1.492  1   0.222

Again, we reject the multivariate score test and see that we can free x7 (.p31. == .p67.). We could potentially free x8 (.p32. == .p68.) but as before, let’s free x7 first.

strong.invariance.x3x7 <- cfa(HS.model, data=HolzingerSwineford1939, group = "school", group.equal = c( "loadings", "intercepts"), group.partial = c("x3 ~ 1", "x7 ~ 1"))
lavTestScore(strong.invariance.x3x7)
$test

total score test:

   test     X2 df p.value
1 score 12.583 13   0.481

$uni

univariate score tests:

     lhs op   rhs    X2 df p.value
1   .p2. == .p38. 0.734  1   0.391
2   .p3. == .p39. 0.492  1   0.483
3   .p5. == .p41. 2.769  1   0.096
4   .p6. == .p42. 2.631  1   0.105
5   .p8. == .p44. 0.013  1   0.910
6   .p9. == .p45. 0.062  1   0.803
7  .p25. == .p61. 2.832  1   0.092
8  .p26. == .p62. 2.832  1   0.092
9  .p28. == .p64. 2.135  1   0.144
10 .p29. == .p65. 1.563  1   0.211
11 .p30. == .p66. 0.032  1   0.858
12 .p32. == .p68. 0.053  1   0.818
13 .p33. == .p69. 0.053  1   0.818

Nothing else should be freed (i.e., we fail to reject multivariate score test). Let’s return back to using lavaan and add these additional constriants.

strong.invariance.x3x7 <- cfa(HS.model, data=HolzingerSwineford1939, group = "school", group.equal = c( "loadings", "intercepts"), group.partial = c("x3 ~ 1", "x7 ~ 1"))
anova(strong.invariance.x3x7, weak.invariance)
Chi Square Difference Test

                       Df    AIC    BIC  Chisq Chisq diff Df diff Pr(>Chisq)
weak.invariance        54 7480.6 7680.8 124.04                              
strong.invariance.x3x7 58 7478.0 7663.3 129.42     5.3789       4     0.2506

Now, we see that have partial strong (scalar) invariance.

fit.stats2 <- rbind(fitmeasures(strong.invariance, fit.measures = c("chisq", "df", "rmsea", "tli", "cfi", "aic")),
fitmeasures(strong.invariance.x3x7, fit.measures = c("chisq", "df", "rmsea", "tli", "cfi", "aic")))
rownames(fit.stats2) <- c("strong", "strong with x3 x7")
fit.stats <- rbind(fit.stats, fit.stats2)
round(fit.stats, 4)
                     chisq df  rmsea    tli    cfi      aic
configural        115.8513 48 0.0969 0.8851 0.9234 7484.395
weak invariance   124.0435 54 0.0928 0.8946 0.9209 7480.587
strong            164.1028 60 0.1074 0.8590 0.8825 7508.646
strong with x3 x7 129.4225 58 0.0905 0.8999 0.9194 7477.966

Because we have strong invariance, we can now make mean comparisons for the latent variables

summary(strong.invariance.x3x7)
lavaan (0.5-20) converged normally after  61 iterations

  Number of observations per group         
  Pasteur                                          156
  Grant-White                                      145

  Estimator                                         ML
  Minimum Function Test Statistic              129.422
  Degrees of freedom                                58
  P-value (Chi-square)                           0.000

Chi-square for each group:

  Pasteur                                       71.170
  Grant-White                                   58.253

Parameter Estimates:

  Information                                 Expected
  Standard Errors                             Standard


Group 1 [Pasteur]:

Latent Variables:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual =~                                           
    x1                1.000                           
    x2      (.p2.)    0.606    0.101    5.988    0.000
    x3      (.p3.)    0.791    0.109    7.259    0.000
  textual =~                                          
    x4                1.000                           
    x5      (.p5.)    1.120    0.066   16.960    0.000
    x6      (.p6.)    0.932    0.056   16.606    0.000
  speed =~                                            
    x7                1.000                           
    x8      (.p8.)    1.200    0.155    7.741    0.000
    x9      (.p9.)    1.041    0.136    7.635    0.000

Covariances:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual ~~                                           
    textual           0.404    0.095    4.247    0.000
    speed             0.168    0.064    2.647    0.008
  textual ~~                                          
    speed             0.172    0.060    2.882    0.004

Intercepts:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1      (.25.)    4.914    0.092   53.538    0.000
    x2      (.26.)    6.087    0.079   76.999    0.000
    x3                2.487    0.094   26.474    0.000
    x4      (.28.)    2.778    0.087   31.953    0.000
    x5      (.29.)    4.035    0.096   41.861    0.000
    x6      (.30.)    1.926    0.079   24.425    0.000
    x7                4.432    0.086   51.533    0.000
    x8      (.32.)    5.569    0.074   75.328    0.000
    x9      (.33.)    5.409    0.070   77.182    0.000
    visual            0.000                           
    textual           0.000                           
    speed             0.000                           

Variances:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                0.560    0.137    4.086    0.000
    x2                1.267    0.156    8.105    0.000
    x3                0.879    0.128    6.850    0.000
    x4                0.446    0.069    6.432    0.000
    x5                0.502    0.082    6.132    0.000
    x6                0.263    0.050    5.258    0.000
    x7                0.850    0.114    7.471    0.000
    x8                0.516    0.095    5.429    0.000
    x9                0.656    0.096    6.852    0.000
    visual            0.796    0.170    4.691    0.000
    textual           0.879    0.131    6.693    0.000
    speed             0.304    0.078    3.918    0.000


Group 2 [Grant-White]:

Latent Variables:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual =~                                           
    x1                1.000                           
    x2      (.p2.)    0.606    0.101    5.988    0.000
    x3      (.p3.)    0.791    0.109    7.259    0.000
  textual =~                                          
    x4                1.000                           
    x5      (.p5.)    1.120    0.066   16.960    0.000
    x6      (.p6.)    0.932    0.056   16.606    0.000
  speed =~                                            
    x7                1.000                           
    x8      (.p8.)    1.200    0.155    7.741    0.000
    x9      (.p9.)    1.041    0.136    7.635    0.000

Covariances:
                   Estimate  Std.Err  Z-value  P(>|z|)
  visual ~~                                           
    textual           0.426    0.097    4.412    0.000
    speed             0.312    0.079    3.955    0.000
  textual ~~                                          
    speed             0.223    0.071    3.163    0.002

Intercepts:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1      (.25.)    4.914    0.092   53.538    0.000
    x2      (.26.)    6.087    0.079   76.999    0.000
    x3                1.955    0.108   18.170    0.000
    x4      (.28.)    2.778    0.087   31.953    0.000
    x5      (.29.)    4.035    0.096   41.861    0.000
    x6      (.30.)    1.926    0.079   24.425    0.000
    x7                3.992    0.094   42.478    0.000
    x8      (.32.)    5.569    0.074   75.328    0.000
    x9      (.33.)    5.409    0.070   77.182    0.000
    visual            0.051    0.129    0.393    0.695
    textual           0.576    0.117    4.918    0.000
    speed            -0.071    0.089   -0.800    0.424

Variances:
                   Estimate  Std.Err  Z-value  P(>|z|)
    x1                0.651    0.127    5.138    0.000
    x2                0.939    0.122    7.721    0.000
    x3                0.603    0.096    6.248    0.000
    x4                0.343    0.062    5.532    0.000
    x5                0.377    0.073    5.136    0.000
    x6                0.437    0.067    6.556    0.000
    x7                0.599    0.090    6.655    0.000
    x8                0.407    0.089    4.570    0.000
    x9                0.531    0.086    6.186    0.000
    visual            0.715    0.160    4.473    0.000
    textual           0.870    0.131    6.659    0.000
    speed             0.475    0.109    4.344    0.000

We have strong invariance for textual and partial strong invariance for both visual and speed factors. We see that Grant-White, school 2, students do better on the textual latent construct than Pasteur. They are expected to be 0.576 higher than students in Pasteur on textual. If we allow for partial strong invariance, we see that there is no difference between Grant-White and Pasteur students on visual or speed.

Strict Invariance

For strict invariance, we add the additional constraint of equal residual variances for the manifest variables across the groups. Because we have only partial invariance, we will allow those parameters to be free that should be (e.g., the intercepts for x3 and x7) and restrict all the other parameters (i.e., factor loadings and intercepts) as well as the residual variances. If we want to make comparsions based on the raw scores (e.g. based on the sum of the scores on the variables), we require strict invariance. This is because the observed variance is the sum of true score variance and residual/error variance. If the residual variances are the same, they have the same amount of true scure variance. As with before, we estimate the latent means.

strict.invariance.x3x7 <- cfa(HS.model, data=HolzingerSwineford1939, group = "school", group.equal = c( "loadings", "intercepts", "residuals"), group.partial = c("x3 ~ 1", "x7 ~ 1"))
anova(strong.invariance.x3x7, strict.invariance.x3x7)
Chi Square Difference Test

                       Df    AIC    BIC  Chisq Chisq diff Df diff Pr(>Chisq)  
strong.invariance.x3x7 58 7478.0 7663.3 129.42                                
strict.invariance.x3x7 67 7477.8 7629.8 147.26     17.838       9     0.0371 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The chi-square difference test is significant. BUT, we have performed a lot of statistical tests. Therefore, a p-value of .037, isn’t that unlike. Let’s look at the other fit measures.

fit.stats <- rbind(fit.stats, fitmeasures(strict.invariance.x3x7, fit.measures = c("chisq", "df", "rmsea", "tli", "cfi", "aic")))
rownames(fit.stats)[5] <- "strict invariance"
round(fit.stats, 4)
                     chisq df  rmsea    tli    cfi      aic
configural        115.8513 48 0.0969 0.8851 0.9234 7484.395
weak invariance   124.0435 54 0.0928 0.8946 0.9209 7480.587
strong            164.1028 60 0.1074 0.8590 0.8825 7508.646
strong with x3 x7 129.4225 58 0.0905 0.8999 0.9194 7477.966
strict invariance 147.2605 67 0.0892 0.9026 0.9094 7477.804

Strict invariance is a marginal improvement in fit over strong invariance where the intercepts for x3 and x7 are free. If we had complete strict invariance, we can say manifest variables are equal reliable across the groups. This means we could just use sum scores for mean comparsions or to perform regressions across groups. We dont’ have full strict invariance, therefore the only scale we could compare in this nature is the textual factor.

When we talk about structural models, we’ll return to this issue and discuss structural invariance, specifically factor variances, factor covariances, and regression coefficents in SEM.

LS0tCnRpdGxlOiAiTWVhc3VyZW1lbnQgSW52YXJpYW5jZSIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKYGBge3Igc2V0dXAsIGluY2x1ZGU9RkFMU0V9CmtuaXRyOjpvcHRzX2NodW5rJHNldChlY2hvID0gVFJVRSwgZmlnLmtlZXAgPSBUKQpgYGAKCklmIHdlIHdhbnQgdG8gYmUgYWJsZSB0byBtYWtlIGNvbXBhcmlzb25zIGFjcm9zcyBncm91cHMsIHdlIG11c3Qgc2hvdyBvdXIgbGF0ZW50IHZhcmlhYmxlKHMpIGlzIGZ1bmN0aW9uaW5nIHRoZSBzYW1lIHdheSBhY3Jvc3MgZ3JvdXBzLiBUaGlzIGlzIHJlZmVycmVkIHRvIGFzIG1lYXN1cmVtZW50IGludmFyaWFuY2UuIElmIHRoZSBtb2RlbCBpcyBub3QgZnVuY3Rpb25pbmcgdGhlIHNhbWUgd2F5IGFjcm9zcyB0aGUgZ3JvdXBzLCB0aGVuIGl0IG1ha2VzIG5vIHNlbnNlIHRvIHNheSwgZm9yIGV4YW1wbGUsIHRoYXQgb25lIGdyb3VwIGhhcyBhIGhpZ2hlciBtZWFuIHdpdGggcmVzcGVjdCB0byB0aGUgbWVhc3VyZWQgY29uc3RydWN0IHRoYW4gYW5vdGhlciBncm91cCBiZWNhdXNlIHRoZSBjb25zdHJ1Y3RzIGFyZSBmdW5jdGlvbmluZyBkaWZmZXJlbnRseS4gU2ltaWxhcmx5LCBpZiB3ZSB3YW50IHRvIHNob3cgdGhhdCBpdCdzIGZhaXIgdG8gY29tcGFyZSAgc2NvcmVzIG9uIGEgY29uc3RydWN0IGF0IHRpbWUgMSB3aXRoIHNjb3JlcyBvbiB0aGF0IGNvbnN0cnVjdCBhdCB0aW1lIDIsIHdlIG5lZWQgZGVtb25zdHJhdGUgbWVhc3VyZW1lbnQgaW52YXJpYW5jZS4gIAoKV2UgaW1wbGljaXRseSBhc3N1bWUgbWVhc3VyZW1lbnQgaW52YXJpYW5jZSwgd2hlbiB3ZSBtYWtlIGNvbXBhcnNpb25zIG9uIHNvbWUgY29uc3RydWN0IGFjcm9zcyBncm91cHMuIFdlIGFyZSBhc3N1bWluZyB0aGF0IHRoZSBjb25zdHJ1Y3QgZnVuY3Rpb25zIHRoZSBzYW1lIHdheSB3aGVuIHdlIGRlY2lkZSB0byBkbyByZWdyZXNzaW9ucywgdC10ZXN0LCBtaXhlZCBlZmZlY3RzIG1vZGVsLCBldGMuIAoKV2UnbGwgdXNlIHRoZSBIb2x6aW5nZXIgZGF0YSBzZXQgaW4gYGxhdmFhbmAsIHRvIHNob3cgaG93IHRvIGFzc2VzcyBtZWFzdXJlbWVudCBpbnZhcmlhbmNlLiBIZXJlJ3MgdGhlIGRlc2NyaXB0aW9uIG9mIHRoZSBkYXRhIHNldDoKClRoZSBjbGFzc2ljIEhvbHppbmdlciBhbmQgU3dpbmVmb3JkICgxOTM5KSBkYXRhc2V0IGNvbnNpc3RzIG9mIG1lbnRhbCBhYmlsaXR5IHRlc3Qgc2NvcmVzIG9mIHNldmVudGgtIGFuZCBlaWdodGgtZ3JhZGUgY2hpbGRyZW4gZnJvbSB0d28gZGlmZmVyZW50IHNjaG9vbHMgKFBhc3RldXIgYW5kIEdyYW50LVdoaXRlKS4gSW4gdGhlIG9yaWdpbmFsIGRhdGFzZXQgKGF2YWlsYWJsZSBpbiB0aGUgTUJFU1MgcGFja2FnZSksIHRoZXJlIGFyZSBzY29yZXMgZm9yIDI2IHRlc3RzLiBIb3dldmVyLCBhIHNtYWxsZXIgc3Vic2V0IHdpdGggOSB2YXJpYWJsZXMgaXMgbW9yZSB3aWRlbHkgdXNlZCBpbiB0aGUgbGl0ZXJhdHVyZSAoZm9yIGV4YW1wbGUgaW4gSm9yZXNrb2cncyAxOTY5IHBhcGVyLCB3aGljaCBhbHNvIHVzZXMgdGhlIDE0NSBzdWJqZWN0cyBmcm9tIHRoZSBHcmFudC1XaGl0ZSBzY2hvb2wgb25seSkuCgoKYGBge3J9CmxpYnJhcnkoImxhdmFhbiIpCmRhdGEoIkhvbHppbmdlclN3aW5lZm9yZDE5MzkiKQpoZWFkKEhvbHppbmdlclN3aW5lZm9yZDE5MzkpCmBgYAoKV2hhdCB3ZSBhcmUgdGVzdGluZyBpcyB3aGV0aGVyIG91ciBtb2RlbCBpcyBpbnZhcmlhbnQgYWNyb3NzIHRoZXNlIHR3byBzY2hvb2xzLgoKU3VidGVzdHMgeDEsIHgyLCBhbmQgeDMgYXJlIG1hbmlmZXN0YXRpb25zIG9mIHRoZSB2aXN1YWwgZmFjdG9yOyB4NCwgeDUsIGFuZCB4NiBhcmUgbWFuaWZlc3RhdGlvbnMgb2YgdGhlIHRleHR1YWwgZmFjdG9yOyBhbmQgeDcsIHg4LCBhbmQgeDkgYXJlIG1hbmlmZXN0YXRpb25zIG9mIHRoZSBzcGVlZCBmYWN0b3IuIFRoaXMgaXMgd2hhdCBvdXIgbW9kZWwgbG9va3MgbGlrZSBvdmVyYWxsIChpZ25vcmluZyB0aGUgc2Nob29scykuCgpgYGB7cn0KSFMubW9kZWwgPC0gJyB2aXN1YWwgID1+IHgxICsgeDIgKyB4MwogICAgICAgICAgICAgIHRleHR1YWwgPX4geDQgKyB4NSArIHg2CiAgICAgICAgICAgICAgc3BlZWQgICA9fiB4NyArIHg4ICsgeDkgJwoKZml0IDwtIGNmYShIUy5tb2RlbCwgZGF0YT1Ib2x6aW5nZXJTd2luZWZvcmQxOTM5KQpzdW1tYXJ5KGZpdCwgZml0Lm1lYXN1cmVzPVRSVUUpCmBgYAoKT3VyIG1vZGVsIGhhcyBhZGVxdWF0ZSBmaXQgYW5kIHRoZSBmaXQgc3RhdGlzdGljcyBhcmUganVzdCBvay4gVGhleSBhcmUgbm90IGF3ZnVsIGJ1dCBpZiB0aGlzIHdhcyB5b3VyIGRhdGEgc2V0LCB5b3Ugd291bGQgaG9wZSBmb3IgYSBiZXR0ZXIgbW9kZWwgZml0IG9yIHlvdSBtaWdodCBjb25zaWRlciBtb2RpZmljYXRpb25zIChlLmcuIHJlbW92aW5nL3N3YXBwaW5nIGluIGluZGljYXRvcnMsIGFsbG93aW5nIHJlc2lkdWFscyB0byBjb3JyZWxhdGUsIGV0YykuCgojIyBDb25maWd1cmFsIG1vZGVsIAoKVGhlIGZpcnN0IHN0ZXAgaW4gZGVtb25zdHJhdGluZyBtZWFzdXJlbWVudCBpbnZhcmlhbmNlIGlzIHRvIHNob3cgdGhhdCB0aGUgbW9kZWwgZml0cyBmb3IgZWFjaCBvZiBvdXIgZ3JvdXBzLiBUaGlzIGNhbiBiZSBkb25lIHR3byB3YXlzLiAxKSBZb3UgY291bGQgZml0IHRoZSBkYXRhIHRvIGVhY2ggc3Vic2V0IG9mIHRoZSBkYXRhIG9yIDIpIHlvdSBjYW4gdXNlIGEgbXVsdGlwbGUgZ3JvdXAgbW9kZWwuIFdlJ2xsIHVzZSB0aGlzIHNlY29uZCBhcHByYWNoIGFuZCB0byBkbyB0aGlzIGluIGxhdmFhbiwgd2Ugc2ltcGx5IGFkZCB0aGUgYGdyb3VwID0gInNjaG9vbCJgIGFyZ3VtZW50IGFuZCByZWZpdCBvdXIgbW9kZWwuIFRoaXMgYXJndW1lbnQgd2lsbCBhbGxvdyBhbGwgb2Ygb3VyIHBhcmFtZXRlcnMgdG8gZGlmZmVyIGFjcm9zcyB0aGUgZ3JvdXBzLgoKTm90ZSwgaXQgaXMgaW5hcHByb3ByaWF0ZSB0byB1c2Ugc3RhbmRhcmRpemUgY29lZmZpY2llbnRzIGhlcmUuIFRoZSByZWFzb24gaXMgdGhhdCB3ZSBoYXZlIG5vIHJlYXNvbiB0byBleHBlY3QgdGhlIHZhcmlhbmNlcyBhcmUgdGhlIHNhbWUgYWNyb3NzIHRoZSBncm91cHMgYW5kIAoKYGBge3J9CmNvbmZpZ3VyYWwgPC0gY2ZhKEhTLm1vZGVsLCBkYXRhPUhvbHppbmdlclN3aW5lZm9yZDE5MzksIGdyb3VwID0gInNjaG9vbCIpCnN1bW1hcnkoY29uZmlndXJhbCwgZml0Lm1lYXN1cmVzPVRSVUUpCmBgYAoKVGhlIG1vZGVsIGFnYWluIGZpdHMganVzdCBvay4gQXQgbGVhc3QgaXQgZG9lc24ndCBmaXQgd29yc2Ugd2hlbiB3ZSBmaXQgYSBtdWx0aXBsZSBncm91cCBtb2RlbC4gVGhlIGNvbmZpZ3VyYWwgbW9kZWwgaXMgYSBwcmVyZXF1aXNpdGUgdG8gaW52ZXN0aWdhdGluZyBtZWFzdXJlbWVudCBpbnZhcmlhbmNlLiAKCiMjIFdlYWsgSW52YXJpYW5jZQoKVG8gYXNzZXNzIGZvciB3ZWFrIChtZXRyaWMpIGludmFyaWFuY2UsIHdlIG5lZWQgdG8gY29uc3RyYWluIHRoZSBmYWN0b3IgbG9hZGluZ3MgdG8gYmUgZXF1YWwgYWNyb3NzIHRoZSBncm91cHMuIFRoaXMgc2hvd3MgdGhhdCB0aGUgZmFjdG9yKHMpIGhhcyB0aGUgc2FtZSBtZWFuaW5nIGFjcm9zcyB0aGUgZ3JvdXBzLiBBcmUgdGhleSBhdHRyaWJ1dGluZyB0aGUgc2FtZSBtZWFuaW5nIHRvIHRoZSBsYXRlbnQgY29uc3RydWN0cz8gSWYgd2UgZG9uJ3QgaGF2ZSB3ZWFrIGludmFyaWFuY2UsIHRoZXkgdGhpcyBpbXBsaWVzIHRoZSB0aGUgbWVhbmluZyBvZiB0aGUgbWFuaWZlc3QgdmFyaWFibGVzIGFyZSBkaWZmZXJlbnQgYWNyb3NzIHRoZSBncm91cHMuIEZvciBleGFtcGxlLCBvbiB0aGUgTWlubmVzb3RhIFN0dWRlbnQgU3VydmV5LCB0aGVyZSBhcmUgbWFueSBpdGVtcyB0aGF0IGFzayBhYm91dCBidWxseWluZyBhbmQgYSBzcGVjaWZpYyBpdGVtIHRoYXQgYXNrcyBhYm91dCBmcmVxdWVuY3kgb2YgYmVpbmcgYnVsbGllZCBiZWNhdXNlIG9mIHlvdXIgcmFjZS9ldGhuaWMgZ3JvdXAuIFJhY2lzbSBmdW5jdGlvbnMgZGlmZmVyZW50bHkgZm9yIHdoaXRlIHN0dWRlbnRzIHRoYW4gbm9uLXdoaXRlIHN0dWRlbnRzIGFuZCBiZWNhdXNlIG9mIHRoaXMsIHRoaXMgaXRlbSBlbmRzIHVwIG5vdCBiZWluZyBpbnZhcmlhbnQgYmVjYXVzZSB3aGl0ZSBzdHVkZW50cyBhcmUgbXVjaCBtb3JlIGxpa2VseSB0byBzYXkgbmV2ZXIgd2hlcmVhcyBub24td2hpdGUgc3R1ZGVudHMgd291bGQgYmUgbW9yZSBsaWtlbHkgdG8gc2F5IGl0IG9jY3VycyBtb3JlIGZyZXF1ZW50bHkuIAoKQXQgdGhpcyBzdGFnZSwgYW5kIGluIGZ1dHVyZSBzdGVwcywgaWYgeW91IGhhdmUgb25seSBwYXJ0aWFsIGludmFyaWFuY2Ugb2YgdGhlIGZhY3RvciBsb2FkaW5ncywgY29uc3RyYWluIHRoZXNlIGludmFyaWFudCBsb2FkaW5ncyB0byBiZSBlcXVhbCBhbmQgYWxsb3cgdGhlIG90aGVyLCBvbmVzIHRvIGJlIGZyZWVkLiBBdCB0aGlzIHBvaW50LCB3ZSBhcmUgeWV0IG5vdCBlc3RpbWF0aW5nIHRoZSBmYWN0b3IgbWVhbnMuIFRoaXMgaGFwcGVucyBpbiB0aGUgbmV4dCBzdGVwLiAgCgpZb3UgZml0IHRoaXMgbW9kZWwgaW4gYGxhdmFhbmAgdXNpbmcgdGhlIGBncm91cC5lcXVhbGAgYXJndW1lbnQuCgpgYGB7cn0Kd2Vhay5pbnZhcmlhbmNlIDwtIGNmYShIUy5tb2RlbCwgZGF0YT1Ib2x6aW5nZXJTd2luZWZvcmQxOTM5LCBncm91cCA9ICJzY2hvb2wiLCBncm91cC5lcXVhbCA9ICJsb2FkaW5ncyIpCnN1bW1hcnkod2Vhay5pbnZhcmlhbmNlLCBmaXQubWVhc3VyZXMgPSBUUlVFKQpgYGAKCkFsbCBvZiB0aGUgbWVhc3VyZW1lbnQgaW52YXJpYW5jZSBtb2RlbHMgYXJlIG5lc3RlZCB3aXRoaW4gb25lIGFub3RoZXIuIFRoZSB3ZWFrIGludmFyaWFuY2UgbW9kZWwgaXMgbmVzdGVkIHdpdGhpbiB0aGUgY29uZmlndXJhbCBtb2RlbC4gVGhlcmVmb3JlLCB3ZSBjYW4gZG8gYSBjaGktc3F1YXJlIGRpZmZlcmVuY2UgdGVzdC4KCmBgYHtyfQphbm92YSh3ZWFrLmludmFyaWFuY2UsIGNvbmZpZ3VyYWwpCmBgYAoKVGhlIG51bGwgaHlwb3RoZXNpcyBpcyB0aGF0IHRoZSBjb25maWd1cmFsIG1vZGVsIGlzIG5vdCBhbiBpbXByb3ZlbWVudCBpbiBmaXQgb3ZlciB0aGUgd2VhayBpbnZhcmlhbmNlIG1vZGVsLiBXZSBmYWlsIHRvIHJlamVjdCB0aGUgbnVsbCBoeXBvdGhlc2lzLiBUaGlzIGlzIGV2aWRlbmNlIHRoYXQgd2UgaGF2ZSB3ZWFrIGludmFyaWFuY2UuIFdlIGNhbiBhbHNvIGludmVzdGlnYXRlIG90aGVyIGZpdCBzdGF0aXN0aWNzLgoKYGBge3J9CmZpdC5zdGF0cyA8LSByYmluZChmaXRtZWFzdXJlcyhjb25maWd1cmFsLCBmaXQubWVhc3VyZXMgPSBjKCJjaGlzcSIsICJkZiIsICJybXNlYSIsICJ0bGkiLCAiY2ZpIiwgImFpYyIpKSwKZml0bWVhc3VyZXMod2Vhay5pbnZhcmlhbmNlLCBmaXQubWVhc3VyZXMgPSBjKCJjaGlzcSIsICJkZiIsICJybXNlYSIsICJ0bGkiLCAiY2ZpIiwgImFpYyIpKSkKcm93bmFtZXMoZml0LnN0YXRzKSA8LSBjKCJjb25maWd1cmFsIiwgIndlYWsgaW52YXJpYW5jZSIpCmZpdC5zdGF0cwpgYGAKVGhlIGZpdCBzdGF0aXN0aWNzIGhhdmVuJ3QgcmVhbGx5IGNoYW5nZWQsIGJ1dCByZWNhbGwgdGhhdCB0aGUgY29uZmlndXJhbCBtb2RlbCBpcyB0aGUgbW9kZWwgd2VyZSBhbGwgdGhlIHBhcmFtZXRlcnMgYXJlIGFsbCBmcmVlbHkgZXN0aW1hdGVkIGFjcm9zcyB0aGUgZ3JvdXBzLCBzbyB0aGUgd2VhayBpbnZhcmlhbmNlIG1vZGVsIGlzIHRoZSBtb3JlIHBhcnNpbW9uaW91cyBtb2RlbCBlc3RpbWF0aW5nIDYgZmV3ZXIgcGFyYW1ldGVycy4gVGhpcyBpcyBldmlkZW5jZSB0aGF0IHdlIGhhdmUgd2VhayBpbnZhcmlhbmNlLiBCYXNlZCBvbiB0aGlzLCB3ZSBjYW4gc2FmZWx5IGNvbmNsdWRlIHRoYXQgd2UgaGF2ZSB3ZWFrIGludmFyaWFuY2UuIAoKIyMgU3Ryb25nIEludmFyaWFuY2UKClN0cm9uZyAoc2NhbGFyKSBpbnZhcmlhbmNlIGFkZHMgdGhlIGFkZGl0aW9uYWwgY29uc3RyYWludCB0aGF0IGludGVyY2VwdHMgYXJlIGVxdWFsIGFjcm9zcyBncm91cHMuIE5vdyB3ZSBoYXZlIGJvdGggbG9hZGluZ3MgYW5kIGludGVyY2VwdHMgaGVsZCBjb25zdHJhbnQgYWNyb3NzIGdyb3Vwcy5UaGlzIGltcGxpZXMgdGhhdCB0aGUgbWVhbmluZyBvZiB0aGUgY29uc3RydWN0ICh0aGUgZmFjdG9yIGxvYWRpbmdzKSwgYW5kIHRoZSBsZXZlbHMgb2YgdGhlIHVuZGVybHlpbmcgbWFuaWZlc3QgdmFyaWFibGVzIChpbnRlcmNlcHRzKSBhcmUgZXF1YWwgaW4gYm90aCBncm91cHMuICBJZiB3ZSBoYXZlIHN0cm9uZyBpbnZhcmlhbmNlLCB3ZSBhcmUgYWJsZSB0byBjb21wYXJlIGRpZmZlcmVuY2VzIGluIHRoZSBtZWFuIGFjcm9zcyB0aGUgbGF0ZW50IGNvbnN0cnVjdChzKS4gSW4gdGhpcyBzdGFnZSB3ZSBlc3RpbWF0ZSB0aGUgZmFjdG9yIG1lYW5zLgoKYGBge3J9CnN0cm9uZy5pbnZhcmlhbmNlIDwtIGNmYShIUy5tb2RlbCwgZGF0YT1Ib2x6aW5nZXJTd2luZWZvcmQxOTM5LCBncm91cCA9ICJzY2hvb2wiLCBncm91cC5lcXVhbCA9IGMoICJsb2FkaW5ncyIsICJpbnRlcmNlcHRzIikpCnN1bW1hcnkoc3Ryb25nLmludmFyaWFuY2UsIGZpdC5tZWFzdXJlcyA9IFRSVUUpCmBgYAoKWW91J2xsIG5vdGUgdGhhdCB0aGUgbGF0ZW50IG1lYW5zIGZvciB2aXN1YWwsIHRleHR1YWwsIGFuZCBzcGVlZCBhcmUgbm93IGVzdGltYXRlZC4gSG93ZXZlciwgdGhleSBzaG91bGRuJ3QgYmUgY29tcGFyZWQgYWNyb3NzIGdyb3VwIHVudGlsIHdlIHRlc3QgaWYgd2UgaGF2ZSBzdHJvbmcgaW52YXJpYW5jZS4gCgpUaGUgc3Ryb25nIGludmFyaWFuY2UgbW9kZWwgaXMgbmVzdGVkIHdpdGhpbiB0aGUgd2VhayBpbnZhcmlhbmNlIG1vZGVsLiBUaGVyZWZvcmUsIHdlIGNhbiBhZ2FpbiBwZXJmb3JtIGEgY2hpLXNxdWFyZSBkaWZmZXJlbmNlIHRlc3QuCgpgYGB7cn0KYW5vdmEoc3Ryb25nLmludmFyaWFuY2UsIHdlYWsuaW52YXJpYW5jZSkKYGBgCgpXZSBzZWUgdGhhdCB3ZSByZWplY3QgdGhlIG51bGwgaHlwb3RoZXNpcyBhbmQgdGhhdCB0aGUgd2VhayBpbnZhcmlhbmNlIG1vZGVsIGZpdHMgYmV0dGVyIHRoYW4gdGhlIHN0cm9uZyBpbnZhcmlhbmNlIG1vZGVsLiBUaGlzIGlzIGFsc28gZXZpZGVudCBieSB0aGUgbG93ZXIgQUlDIGFuZCBCSUMuIFdoYXQgd2Ugc2hvdWxkIG5vdyBrbm93IGlzIGV4YW1pbmUgd2hldGhlciB0aGVyZSBpcyBwYXJ0aWFsIGludmFyaWFuY2UuIFdlIGNhbiBkbyB0aGlzIGJ5IHVzaW5nIHRoZSBgbGF2VGVzdFNjb3JlKClgIGZ1bmN0aW9uIGluIHRoZSBsYXZhYW4gZnVuY3Rpb24uIFRoaXMgZnVuY3Rpb24gYWxsb3dzIHVzIHRvIHNlZSB0aGUgZWZmZWN0IG9mIHJlbGVhc2luZyBlcXVhbGl0eSBjb25zdHJhaW50cyBhY3Jvc3MgdGhlIGdyb3Vwcy4gVGhlIGBtb2RpbmRpY2VzKClgIHdpbGwgb25seSBzaG93IG1vZGlmaWNhdGlvbiBpbmRpY2VzIGZvciBuZXdseSBhZGRlZCBwYXJhbWV0ZXJzIGFzc29jaWF0ZWQgd2l0aCBuZXcgcGF0aHMuIFRoZXNlIHBhcmFtZXRlcnMgYXJlbid0IG5ld2x5IGVzdGltYXRlZCBpbiB0aGUgbW9kZWwsIHdlJ3JlIGp1c3QgZnJlZWluZyB0aGVtIGFjcm9zcyBncm91cHMuICAKCmBgYHtyfQpsYXZUZXN0U2NvcmUoc3Ryb25nLmludmFyaWFuY2UpCmBgYAoKV2hhdCBhcmUgYWxsIHRoZXNlIGAucGBzPwoKYGBge3J9CnBhclRhYmxlKHN0cm9uZy5pbnZhcmlhbmNlKQpgYGAKCgpUaGUgZmlyc3Qgb3V0cHV0IGlzIGEgbXVsdGl2YXJpYXRlIHNjb3JlIHRlc3QgKGkuZS4gTGFncmFuZ2UgbXVsdGlwbGVyIHRlc3QpIGFuZCB0aGlzIGlzIGEgdGVzdCBvZiB3aGV0aGVyIGZyZWVpbmcgYWxsIGVxdWFsaXR5IGNvbnN0cmFpbnRzIHJlcHJlc2VudHMgYW4gaW1wcm92ZW1lbnQgaW4gZml0IG92ZXIgdGhlIGJhc2UgbW9kZWwuIFdlIHJlamVjdCB0aGUgbnVsbCBoeXBvdGhlc2lzLiBUaGVyZWZvcmUsIHdlIHNob3VsZCBsb29rIGF0IHRoZSB1bml2YXJpYXRlIHNjb3JlIHRlc3RzIChpLmUuLCB0aGUgY2hpLXNxdWFyZSBkaWZmZXJlbmNlIHRlc3RzKSB0byBzZWUgd2hpY2ggZXF1YWxpdHkgY29uc3RyYWludHMgc2hvdWxkIGJlIHJlbGF4ZWQgYW5kIHRoZXNlIHBhcmFtZXRlcnMgc2hvdWxkIGJlIGZyZWVkLiBXZSBzZWUgdGhhdCB3ZSBzaG91bGQgZnJlZSB0aGUgeDMgKHRoaXMgaXMgdGhlIGAucDI3ID09IC5wNjNgKSBpbnRlcmNlcHQgaGFzIHRoZSBsYXJnZXN0IGNoYW5nZSBpbiBjaGktc3F1YXJlIGRpZmZlcmVuY2UgKGBYMmApLiBMZXQncyBmaXQgdGhpcyBtb2RlbCBmaXJzdC4KCmBgYHtyfQpzdHJvbmcuaW52YXJpYW5jZS54MyA8LSBjZmEoSFMubW9kZWwsIGRhdGE9SG9semluZ2VyU3dpbmVmb3JkMTkzOSwgZ3JvdXAgPSAic2Nob29sIiwgZ3JvdXAuZXF1YWwgPSBjKCAibG9hZGluZ3MiLCAiaW50ZXJjZXB0cyIpLCBncm91cC5wYXJ0aWFsID0gYygieDMgfiAxIikpCmxhdlRlc3RTY29yZShzdHJvbmcuaW52YXJpYW5jZS54MykKYGBgCgpBZ2Fpbiwgd2UgcmVqZWN0IHRoZSBtdWx0aXZhcmlhdGUgc2NvcmUgdGVzdCBhbmQgc2VlIHRoYXQgd2UgY2FuIGZyZWUgeDcgKGAucDMxLiA9PSAucDY3LmApLiBXZSBjb3VsZCBwb3RlbnRpYWxseSBmcmVlIHg4IChgLnAzMi4gPT0gLnA2OC5gKSBidXQgYXMgYmVmb3JlLCBsZXQncyBmcmVlIHg3IGZpcnN0LgoKYGBge3J9CnN0cm9uZy5pbnZhcmlhbmNlLngzeDcgPC0gY2ZhKEhTLm1vZGVsLCBkYXRhPUhvbHppbmdlclN3aW5lZm9yZDE5MzksIGdyb3VwID0gInNjaG9vbCIsIGdyb3VwLmVxdWFsID0gYyggImxvYWRpbmdzIiwgImludGVyY2VwdHMiKSwgZ3JvdXAucGFydGlhbCA9IGMoIngzIH4gMSIsICJ4NyB+IDEiKSkKbGF2VGVzdFNjb3JlKHN0cm9uZy5pbnZhcmlhbmNlLngzeDcpCmBgYAoKTm90aGluZyBlbHNlIHNob3VsZCBiZSBmcmVlZCAoaS5lLiwgd2UgZmFpbCB0byByZWplY3QgbXVsdGl2YXJpYXRlIHNjb3JlIHRlc3QpLiBMZXQncyByZXR1cm4gYmFjayB0byB1c2luZyBgbGF2YWFuYCBhbmQgYWRkIHRoZXNlIGFkZGl0aW9uYWwgY29uc3RyaWFudHMuCgpgYGB7cn0Kc3Ryb25nLmludmFyaWFuY2UueDN4NyA8LSBjZmEoSFMubW9kZWwsIGRhdGE9SG9semluZ2VyU3dpbmVmb3JkMTkzOSwgZ3JvdXAgPSAic2Nob29sIiwgZ3JvdXAuZXF1YWwgPSBjKCAibG9hZGluZ3MiLCAiaW50ZXJjZXB0cyIpLCBncm91cC5wYXJ0aWFsID0gYygieDMgfiAxIiwgIng3IH4gMSIpKQphbm92YShzdHJvbmcuaW52YXJpYW5jZS54M3g3LCB3ZWFrLmludmFyaWFuY2UpCmBgYAoKTm93LCB3ZSBzZWUgdGhhdCBoYXZlIHBhcnRpYWwgc3Ryb25nIChzY2FsYXIpIGludmFyaWFuY2UuCgpgYGB7cn0KZml0LnN0YXRzMiA8LSByYmluZChmaXRtZWFzdXJlcyhzdHJvbmcuaW52YXJpYW5jZSwgZml0Lm1lYXN1cmVzID0gYygiY2hpc3EiLCAiZGYiLCAicm1zZWEiLCAidGxpIiwgImNmaSIsICJhaWMiKSksCmZpdG1lYXN1cmVzKHN0cm9uZy5pbnZhcmlhbmNlLngzeDcsIGZpdC5tZWFzdXJlcyA9IGMoImNoaXNxIiwgImRmIiwgInJtc2VhIiwgInRsaSIsICJjZmkiLCAiYWljIikpKQpyb3duYW1lcyhmaXQuc3RhdHMyKSA8LSBjKCJzdHJvbmciLCAic3Ryb25nIHdpdGggeDMgeDciKQpmaXQuc3RhdHMgPC0gcmJpbmQoZml0LnN0YXRzLCBmaXQuc3RhdHMyKQpyb3VuZChmaXQuc3RhdHMsIDQpCmBgYAoKQmVjYXVzZSB3ZSBoYXZlIHN0cm9uZyBpbnZhcmlhbmNlLCB3ZSBjYW4gbm93IG1ha2UgbWVhbiBjb21wYXJpc29ucyBmb3IgdGhlIGxhdGVudCB2YXJpYWJsZXMKYGBge3J9CnN1bW1hcnkoc3Ryb25nLmludmFyaWFuY2UueDN4NykKYGBgCgpXZSBoYXZlIHN0cm9uZyBpbnZhcmlhbmNlIGZvciB0ZXh0dWFsIGFuZCBwYXJ0aWFsIHN0cm9uZyBpbnZhcmlhbmNlIGZvciBib3RoIHZpc3VhbCBhbmQgc3BlZWQgZmFjdG9ycy4gV2Ugc2VlIHRoYXQgR3JhbnQtV2hpdGUsIHNjaG9vbCAyLCBzdHVkZW50cyBkbyBiZXR0ZXIgb24gdGhlIHRleHR1YWwgbGF0ZW50IGNvbnN0cnVjdCB0aGFuIFBhc3RldXIuIFRoZXkgYXJlIGV4cGVjdGVkIHRvIGJlIDAuNTc2IGhpZ2hlciB0aGFuIHN0dWRlbnRzIGluIFBhc3RldXIgb24gdGV4dHVhbC4gSWYgd2UgYWxsb3cgZm9yIHBhcnRpYWwgc3Ryb25nIGludmFyaWFuY2UsIHdlIHNlZSB0aGF0IHRoZXJlIGlzIG5vIGRpZmZlcmVuY2UgYmV0d2VlbiBHcmFudC1XaGl0ZSBhbmQgUGFzdGV1ciBzdHVkZW50cyBvbiB2aXN1YWwgb3Igc3BlZWQuIAoKIyMgU3RyaWN0IEludmFyaWFuY2UKCkZvciBzdHJpY3QgaW52YXJpYW5jZSwgd2UgYWRkIHRoZSBhZGRpdGlvbmFsIGNvbnN0cmFpbnQgb2YgZXF1YWwgcmVzaWR1YWwgdmFyaWFuY2VzIGZvciB0aGUgbWFuaWZlc3QgdmFyaWFibGVzIGFjcm9zcyB0aGUgZ3JvdXBzLiBCZWNhdXNlIHdlIGhhdmUgb25seSBwYXJ0aWFsIGludmFyaWFuY2UsIHdlIHdpbGwgYWxsb3cgdGhvc2UgcGFyYW1ldGVycyB0byBiZSBmcmVlIHRoYXQgc2hvdWxkIGJlIChlLmcuLCB0aGUgaW50ZXJjZXB0cyBmb3IgeDMgYW5kIHg3KSBhbmQgcmVzdHJpY3QgYWxsIHRoZSBvdGhlciBwYXJhbWV0ZXJzIChpLmUuLCBmYWN0b3IgbG9hZGluZ3MgYW5kIGludGVyY2VwdHMpIGFzIHdlbGwgYXMgdGhlIHJlc2lkdWFsIHZhcmlhbmNlcy4gSWYgd2Ugd2FudCB0byBtYWtlIGNvbXBhcnNpb25zIGJhc2VkIG9uIHRoZSByYXcgc2NvcmVzIChlLmcuIGJhc2VkIG9uIHRoZSBzdW0gb2YgdGhlIHNjb3JlcyBvbiB0aGUgdmFyaWFibGVzKSwgd2UgcmVxdWlyZSBzdHJpY3QgaW52YXJpYW5jZS4gVGhpcyBpcyBiZWNhdXNlIHRoZSBvYnNlcnZlZCB2YXJpYW5jZSBpcyB0aGUgc3VtIG9mIHRydWUgc2NvcmUgdmFyaWFuY2UgYW5kIHJlc2lkdWFsL2Vycm9yIHZhcmlhbmNlLiBJZiB0aGUgcmVzaWR1YWwgdmFyaWFuY2VzIGFyZSB0aGUgc2FtZSwgdGhleSBoYXZlIHRoZSBzYW1lIGFtb3VudCBvZiB0cnVlIHNjdXJlIHZhcmlhbmNlLiBBcyB3aXRoIGJlZm9yZSwgd2UgZXN0aW1hdGUgdGhlIGxhdGVudCBtZWFucy4KCmBgYHtyfQpzdHJpY3QuaW52YXJpYW5jZS54M3g3IDwtIGNmYShIUy5tb2RlbCwgZGF0YT1Ib2x6aW5nZXJTd2luZWZvcmQxOTM5LCBncm91cCA9ICJzY2hvb2wiLCBncm91cC5lcXVhbCA9IGMoICJsb2FkaW5ncyIsICJpbnRlcmNlcHRzIiwgInJlc2lkdWFscyIpLCBncm91cC5wYXJ0aWFsID0gYygieDMgfiAxIiwgIng3IH4gMSIpKQphbm92YShzdHJvbmcuaW52YXJpYW5jZS54M3g3LCBzdHJpY3QuaW52YXJpYW5jZS54M3g3KQpgYGAKClRoZSBjaGktc3F1YXJlIGRpZmZlcmVuY2UgdGVzdCBpcyBzaWduaWZpY2FudC4gQlVULCB3ZSBoYXZlIHBlcmZvcm1lZCBhIGxvdCBvZiBzdGF0aXN0aWNhbCB0ZXN0cy4gVGhlcmVmb3JlLCBhIHAtdmFsdWUgb2YgLjAzNywgaXNuJ3QgdGhhdCB1bmxpa2UuIExldCdzIGxvb2sgYXQgdGhlIG90aGVyIGZpdCBtZWFzdXJlcy4KCmBgYHtyfQpmaXQuc3RhdHMgPC0gcmJpbmQoZml0LnN0YXRzLCBmaXRtZWFzdXJlcyhzdHJpY3QuaW52YXJpYW5jZS54M3g3LCBmaXQubWVhc3VyZXMgPSBjKCJjaGlzcSIsICJkZiIsICJybXNlYSIsICJ0bGkiLCAiY2ZpIiwgImFpYyIpKSkKcm93bmFtZXMoZml0LnN0YXRzKVs1XSA8LSAic3RyaWN0IGludmFyaWFuY2UiCnJvdW5kKGZpdC5zdGF0cywgNCkKYGBgCgpTdHJpY3QgaW52YXJpYW5jZSBpcyBhIG1hcmdpbmFsIGltcHJvdmVtZW50IGluIGZpdCBvdmVyIHN0cm9uZyBpbnZhcmlhbmNlIHdoZXJlIHRoZSBpbnRlcmNlcHRzIGZvciB4MyBhbmQgeDcgYXJlIGZyZWUuIElmIHdlIGhhZCBjb21wbGV0ZSBzdHJpY3QgaW52YXJpYW5jZSwgd2UgY2FuIHNheSBtYW5pZmVzdCB2YXJpYWJsZXMgYXJlIGVxdWFsIHJlbGlhYmxlIGFjcm9zcyB0aGUgZ3JvdXBzLiBUaGlzIG1lYW5zIHdlIGNvdWxkIGp1c3QgdXNlIHN1bSBzY29yZXMgZm9yIG1lYW4gY29tcGFyc2lvbnMgb3IgdG8gcGVyZm9ybSByZWdyZXNzaW9ucyBhY3Jvc3MgZ3JvdXBzLiBXZSBkb250JyBoYXZlIGZ1bGwgc3RyaWN0IGludmFyaWFuY2UsIHRoZXJlZm9yZSB0aGUgb25seSBzY2FsZSB3ZSBjb3VsZCBjb21wYXJlIGluIHRoaXMgbmF0dXJlIGlzIHRoZSB0ZXh0dWFsIGZhY3Rvci4KCldoZW4gd2UgdGFsayBhYm91dCBzdHJ1Y3R1cmFsIG1vZGVscywgd2UnbGwgcmV0dXJuIHRvIHRoaXMgaXNzdWUgYW5kIGRpc2N1c3Mgc3RydWN0dXJhbCBpbnZhcmlhbmNlLCBzcGVjaWZpY2FsbHkgZmFjdG9yIHZhcmlhbmNlcywgZmFjdG9yIGNvdmFyaWFuY2VzLCBhbmQgcmVncmVzc2lvbiBjb2VmZmljZW50cyBpbiBTRU0uCgo=