Conjoint Analysis is a survey based technique to identify how customers value various attributes that make up an individual product. Products and services are bundles of features that customers consider jointly and while making a purchase decision they must make trade offs. Conjoint analysis helps companies and business owners determine importance of different features and their money value. This also helps them identify optimal price for their products and services.

Basic Setup In this example, we will discover how to design a survey to get the information necessary, analyze and quantify importance of various features in a product and how to interpret the results. But first , let’s begin by setting up R environment and loading required libraries.

tidyverse
Error: object 'tidyverse' not found

Experiment Design: We want to know what car manufacturer/car type respondents find most important when choosing to buy a car.


###Experiment Design for asking Conjoint Questions

#- identify number of questions required. Define level and factors
NumberOfQuestions = nrow(oa.design(nlevels=c(7,2,2)))
creating full factorial with 28 runs ...
# create dummy data
bettervehicle = expand.grid(
                
car_manufacturer = c('chevrolet','dodge','honda','lincoln','mercury','pontiac','toyota'),
 car_type= c('compact','suv','twoseater','midsize','minivan','pickup','subcompact'))
#combination to enquire
selectedComb = caFactorialDesign(
  data  = bettervehicle,
type = 'fractional',
cards = NumberOfQuestions)

# print select combinations

kable(selectedComb) %>% 
  kable_styling(bootstrap_options = c('striped','hover','condensed','responsive'))
car_manufacturer car_type
4 lincoln compact
5 mercury compact
6 pontiac compact
7 toyota compact
8 chevrolet suv
9 dodge suv
12 mercury suv
13 pontiac suv
15 chevrolet twoseater
17 honda twoseater
18 lincoln twoseater
21 toyota twoseater
23 dodge midsize
24 honda midsize
27 pontiac midsize
28 toyota midsize
29 chevrolet minivan
31 honda minivan
32 lincoln minivan
34 pontiac minivan
37 dodge pickup
38 honda pickup
39 lincoln pickup
40 mercury pickup
43 chevrolet subcompact
44 dodge subcompact
47 mercury subcompact
49 toyota subcompact

Predicting Responses for other combinations Using orthogonal factorial design, we identified 28 combinations to include in customer survey out of possible 49 combinations. It saves time, money and effort. Survey takers were asked if they liked the combination or not and their responses were noted for all 28 combinations. We will use these 28 responses to train our logistic regression model and use to predict rest of combinations.

# logistic regression
logisticmodel=glm(Response ~ factor(car_manufacturer) + factor(car_type),
            family=binomial(link='logit'), data=selectedComb)
logisticmodel

Call:  glm(formula = Response ~ factor(car_manufacturer) + factor(car_type), 
    family = binomial(link = "logit"), data = selectedComb)

Coefficients:
                    (Intercept)    factor(car_manufacturer)dodge  
           37.17375406612627131            -18.66970866954126862  
  factor(car_manufacturer)honda  factor(car_manufacturer)lincoln  
          -24.18860483890588853            -56.42542531667980654  
factor(car_manufacturer)mercury  factor(car_manufacturer)pontiac  
          -37.17375406612629263            -37.17375406612627131  
 factor(car_manufacturer)toyota              factor(car_type)suv  
            0.00000000000000367              0.00000000000000819  
      factor(car_type)twoseater          factor(car_type)midsize  
          -37.17375406612627131             19.79288025159280195  
        factor(car_type)minivan           factor(car_type)pickup  
          -56.83058479194890822             39.00537632897358975  
     factor(car_type)subcompact  
          -37.17375406612627842  

Degrees of Freedom: 27 Total (i.e. Null);  15 Residual
Null Deviance:      38.7 
Residual Deviance: 11.1     AIC: 37.1
# predict
bettervehicle$response = ifelse(predict(logisticmodel,bettervehicle,type="response") > 0.5,1,0)
bettervehicle$response 
 [1] 1 1 1 0 0 0 1 1 1 1 0 0 1 1 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0
[45] 0 0 0 0 0

Conjoint Analysis

Now, that we have all the data we need, lets run a conjoint analysis and summarize importance of various features for customers.


Call:
lm(formula = frml)

Residuals:
    Min      1Q  Median      3Q     Max 
-0,6364 -0,2245  0,0612  0,2041  0,6327 

Coefficients: (1 not defined because of singularities)
                                        Estimate           Std. Error t value
(Intercept)                  0,50811688311688263  0,02847261231624853   17,85
factor(x$car_manufacturer)1 -3,44318181818182811  0,51981640713137645   -6,62
factor(x$car_manufacturer)2  0,57351576994434250  0,10435622599741000    5,50
factor(x$car_manufacturer)3  0,57351576994434050  0,10435622599740983    5,50
factor(x$car_manufacturer)4  0,57351576994434172  0,10435622599740989    5,50
factor(x$car_manufacturer)5  0,28780148423005619  0,10435622599741004    2,76
factor(x$car_manufacturer)6  0,28780148423005719  0,10435622599741004    2,76
factor(x$car_manufacturer)7  0,43065862708719999  0,10435622599741008    4,13
factor(x$car_type)1          3,57142857142857872  0,51323656942362439    6,96
factor(x$car_type)2         -0,42857142857143121  0,11199740137158185   -3,83
factor(x$car_type)3         -0,28571428571428786  0,11199740137158190   -2,55
factor(x$car_type)4         -0,85714285714285821  0,11199740137158173   -7,65
factor(x$car_type)5         -0,00000000000000116  0,11199740137158175    0,00
factor(x$car_type)6         -1,00000000000000111  0,11199740137158172   -8,93
factor(x$car_type)7                           NA                   NA      NA
                                        Pr(>|t|)    
(Intercept)                 < 0,0000000000000002 ***
factor(x$car_manufacturer)1    0,000000002089818 ***
factor(x$car_manufacturer)2    0,000000325636423 ***
factor(x$car_manufacturer)3    0,000000325636423 ***
factor(x$car_manufacturer)4    0,000000325636423 ***
factor(x$car_manufacturer)5              0,00698 ** 
factor(x$car_manufacturer)6              0,00698 ** 
factor(x$car_manufacturer)7    0,000078922487768 ***
factor(x$car_type)1            0,000000000438388 ***
factor(x$car_type)2                      0,00023 ***
factor(x$car_type)3                      0,01234 *  
factor(x$car_type)4            0,000000000016081 ***
factor(x$car_type)5                      1,00000    
factor(x$car_type)6            0,000000000000032 ***
factor(x$car_type)7                           NA    
---
Signif. codes:  0 ‘***’ 0,001 ‘**’ 0,01 ‘*’ 0,05 ‘.’ 0,1 ‘ ’ 1

Residual standard error: 0,296 on 95 degrees of freedom
Multiple R-squared:  0,694, Adjusted R-squared:  0,652 
F-statistic: 16,6 on 13 and 95 DF,  p-value: <0,0000000000000002

[1] "Part worths (utilities) of levels (model parameters for whole sample):"
[1] "Average importance of factors (attributes):"
[1] 30 70
[1] Sum of average importance:  100
[1] "Chart of average factors importance"

Feature Importance
The Importance of features plot shows that buyers of vehicles care more about the type of car it is than they do the manufacturer. Which in way makes sense as it provides them with a way of understanding how the car might best suit their needs.

# Feature Importance
Importance2 = data.frame(Feature = c('car_manufacturer','car_type'),
                        Importance=caImportance(y=Survey_b,
   x=bettervehicle[,1:2]))
Importance2
ggplot(Importance2,aes(reorder(Feature,-Importance),Importance))+
  geom_bar(stat='identity',width=.8,fill='orange3')+
  ggtitle('Importance of different features')+
  xlab('')+theme_classic()+coord_flip()





Feature Utilities

Now, for vehicle options, we understand that car type is the key factor, although the manufacturer is still seen to be of some importance. But overall which type of car design do consumers find most paramount in their reason of wanting to buy it? We can get our answer by comparing the levels of each feature one at a time

#summarize utilities


util = data.frame(Utilities = t
        (data.frame(caPartUtilities(y=Survey,
          x=bettervehicle[,1:2],z=levels))))


util$levels = row.names(util)

util$levels
 [1] "intercept"  "chevrolet"  "dodge"      "honda"      "lincoln"    "mercury"   
 [7] "pontiac"    "toyota"     "compact"    "suv"        "twoseater"  "midsize"   
[13] "minivan"    "pickup"     "subcompact"
util %>% 
  arrange(desc(Utilities))
#  Type
ggplot(data = util[which(util$levels %in% c('compact','suv','twoseater','midsize','minivan',      'pickup','subcompact')),], aes(x = levels, y = Utilities,fill=Utilities)) +
  geom_bar(stat='identity') +
  ggtitle("Utilities of Car Classes") + xlab("")+coord_flip()+
  geom_hline(yintercept = 0,lty=2,col='black',linewidth=.7)+scale_fill_gradient(high='cyan',low='black')+theme(plot.title =  element_text(size=14),vjust=1)+theme_classic()+theme(legend.position = 'bottom')



ggplot(data = util[which(util$levels %in% c('compact','suv','midsize','pickup')),], aes(x = levels, y = Utilities,fill=Utilities)) +
  geom_bar(stat='identity') +
  ggtitle("Utilities of Car Classes") + xlab("")+coord_flip()+
  geom_hline(yintercept = 0,lty=2,col='black',linewidth=.7)+scale_fill_gradient(high='cyan',low='black')+theme(plot.title =  element_text(size=14),vjust=1)+theme_classic()+theme(legend.position = 'bottom')

In conclusion:


Findings

After comparing the levels of each feature in our utility summary it’s clear from the results of the survey that the majority of respondents most prefer to buy cars based off of their type or general design rather than the car’s manufacturer. In addition to this, we also discovered that the 51% of respondents who happened to buy a vehicle ended getting one because it was a pick up truck while another 51% of respondents did so because of it being midsize. Meanwhile, 22.4% purchased the vehicle because of it being an SUV and less than 10% for being compact. Whereas, respondents found two seaters, subcompact and minivan car types to be of no importance in choosing a car to buy.

Insights
What this tells us is that we should focus the majority of our resources more heavily toward the continued production and advertising alongside the addition of more technological enhancements in pick up trucks and midsize cars while still allocating a certain percent of resources to the production,enhancement and advertisement of SUV types.

LS0tDQp0aXRsZTogIkNvbmpvaW50IEFuYWx5c2lzOiBWZWhpY2xlIENob2ljZSINCm91dHB1dDogaHRtbF9ub3RlYm9vaw0KYXV0aG9yOiBoYXJyaXNvbg0KLS0tDQoNCjxicj4NCjxicj4NCjxicj4NCjxiPkNvbmpvaW50IEFuYWx5c2lzPC9iPiBpcyBhIHN1cnZleSBiYXNlZCB0ZWNobmlxdWUgdG8gaWRlbnRpZnkgaG93IGN1c3RvbWVycyB2YWx1ZSB2YXJpb3VzIGF0dHJpYnV0ZXMgdGhhdCBtYWtlIHVwIGFuIGluZGl2aWR1YWwgcHJvZHVjdC4gUHJvZHVjdHMgYW5kIHNlcnZpY2VzIGFyZSBidW5kbGVzIG9mIGZlYXR1cmVzIHRoYXQgY3VzdG9tZXJzIGNvbnNpZGVyIGpvaW50bHkgYW5kIHdoaWxlIG1ha2luZyBhIHB1cmNoYXNlIGRlY2lzaW9uIHRoZXkgbXVzdCBtYWtlIHRyYWRlIG9mZnMuIENvbmpvaW50IGFuYWx5c2lzIGhlbHBzIGNvbXBhbmllcyBhbmQgYnVzaW5lc3Mgb3duZXJzIGRldGVybWluZSBpbXBvcnRhbmNlIG9mIGRpZmZlcmVudCBmZWF0dXJlcyBhbmQgdGhlaXIgbW9uZXkgdmFsdWUuIFRoaXMgYWxzbyBoZWxwcyB0aGVtIGlkZW50aWZ5IG9wdGltYWwgcHJpY2UgZm9yIHRoZWlyIHByb2R1Y3RzIGFuZCBzZXJ2aWNlcy4NCg0KQmFzaWMgU2V0dXANCkluIHRoaXMgZXhhbXBsZSwgd2Ugd2lsbCBkaXNjb3ZlciBob3cgdG8gZGVzaWduIGEgc3VydmV5IHRvIGdldCB0aGUgaW5mb3JtYXRpb24gbmVjZXNzYXJ5LCBhbmFseXplIGFuZCBxdWFudGlmeSBpbXBvcnRhbmNlIG9mIHZhcmlvdXMgZmVhdHVyZXMgaW4gYSBwcm9kdWN0IGFuZCBob3cgdG8gaW50ZXJwcmV0IHRoZSByZXN1bHRzLiBCdXQgZmlyc3QgLCBsZXTigJlzIGJlZ2luIGJ5IHNldHRpbmcgdXAgUiBlbnZpcm9ubWVudCBhbmQgbG9hZGluZyByZXF1aXJlZCBsaWJyYXJpZXMuDQoNCmBgYHtyIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0V9DQpsaWJyYXJ5KHRpZHl2ZXJzZSkNCm9wdGlvbnMoc2NpcGVuID0gOTk5KQ0Kb3B0aW9ucyhkaWdpdHM9MykNCmxpYnJhcnkoY29uam9pbnQpDQppbnN0YWxsLnBhY2thZ2VzKCdwYWNtYW4nKQ0KbGlicmFyeShwYWNtYW4pDQpwYWNtYW46OnBfbG9hZChjb25qb2ludCwgRG9FLmJhc2UsIGtuaXRyLCBkcGx5ciwga2FibGVFeHRyYSwgZ2dwbG90MikNCmBgYA0KDQo8Yj5FeHBlcmltZW50IERlc2lnbjo8L2I+DQpXZSB3YW50IHRvIGtub3cgd2hhdCBjYXIgbWFudWZhY3R1cmVyL2NhciB0eXBlIHJlc3BvbmRlbnRzIGZpbmQgbW9zdCBpbXBvcnRhbnQgd2hlbiBjaG9vc2luZyB0byBidXkgYSBjYXIuDQoNCmBgYHtyfQ0KDQojIyNFeHBlcmltZW50IERlc2lnbiBmb3IgYXNraW5nIENvbmpvaW50IFF1ZXN0aW9ucw0KDQojLSBpZGVudGlmeSBudW1iZXIgb2YgcXVlc3Rpb25zIHJlcXVpcmVkLiBEZWZpbmUgbGV2ZWwgYW5kIGZhY3RvcnMNCk51bWJlck9mUXVlc3Rpb25zID0gbnJvdyhvYS5kZXNpZ24obmxldmVscz1jKDcsMiwyKSkpDQoNCiMgY3JlYXRlIGR1bW15IGRhdGENCmJldHRlcnZlaGljbGUgPSBleHBhbmQuZ3JpZCgNCiAgICAgICAgICAgICAgICANCmNhcl9tYW51ZmFjdHVyZXIgPSBjKCdjaGV2cm9sZXQnLCdkb2RnZScsJ2hvbmRhJywnbGluY29sbicsJ21lcmN1cnknLCdwb250aWFjJywndG95b3RhJyksDQogY2FyX3R5cGU9IGMoJ2NvbXBhY3QnLCdzdXYnLCd0d29zZWF0ZXInLCdtaWRzaXplJywnbWluaXZhbicsJ3BpY2t1cCcsJ3N1YmNvbXBhY3QnKSkNCiNjb21iaW5hdGlvbiB0byBlbnF1aXJlDQpzZWxlY3RlZENvbWIgPSBjYUZhY3RvcmlhbERlc2lnbigNCiAgZGF0YSAgPSBiZXR0ZXJ2ZWhpY2xlLA0KdHlwZSA9ICdmcmFjdGlvbmFsJywNCmNhcmRzID0gTnVtYmVyT2ZRdWVzdGlvbnMpDQoNCiMgcHJpbnQgc2VsZWN0IGNvbWJpbmF0aW9ucw0KDQprYWJsZShzZWxlY3RlZENvbWIpICU+JSANCiAga2FibGVfc3R5bGluZyhib290c3RyYXBfb3B0aW9ucyA9IGMoJ3N0cmlwZWQnLCdob3ZlcicsJ2NvbmRlbnNlZCcsJ3Jlc3BvbnNpdmUnKSkNCmBgYA0KDQoNCg0KDQoNCmBgYHtyfQ0KYGBgDQoNClByZWRpY3RpbmcgUmVzcG9uc2VzIGZvciBvdGhlciBjb21iaW5hdGlvbnMNClVzaW5nIG9ydGhvZ29uYWwgZmFjdG9yaWFsIGRlc2lnbiwgd2UgaWRlbnRpZmllZCAyOCBjb21iaW5hdGlvbnMgdG8gaW5jbHVkZSBpbiBjdXN0b21lciBzdXJ2ZXkgb3V0IG9mIHBvc3NpYmxlIDQ5IGNvbWJpbmF0aW9ucy4gSXQgc2F2ZXMgdGltZSwgbW9uZXkgYW5kIGVmZm9ydC4gU3VydmV5IHRha2VycyB3ZXJlIGFza2VkIGlmIHRoZXkgbGlrZWQgdGhlIGNvbWJpbmF0aW9uIG9yIG5vdCBhbmQgdGhlaXIgcmVzcG9uc2VzIHdlcmUgbm90ZWQgZm9yIGFsbCAyOCBjb21iaW5hdGlvbnMuIFdlIHdpbGwgdXNlIHRoZXNlIDI4IHJlc3BvbnNlcyB0byB0cmFpbiBvdXIgbG9naXN0aWMgcmVncmVzc2lvbiBtb2RlbCBhbmQgdXNlIHRvIHByZWRpY3QgcmVzdCBvZiBjb21iaW5hdGlvbnMuDQoNCmBgYHtyfQ0KIyBhZGQgcmVzcG9uc2UgY29sdW1uDQpzZWxlY3RlZENvbWIkUmVzcG9uc2UgPSANCiAgYygwLDAsMSwxLDEsMSwxLDAsMCwwLDAsMSwxLDEsMSwxLDAsMCwwLDAsMSwxLDEsMSwxLDAsMCwwKQ0KICAgICAgICAgICAgICAgICAgICAgICAgICANCnNlbGVjdGVkQ29tYiRSZXNwb25zZQ0KYGBgDQoNCg0KYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0NCiMgbG9naXN0aWMgcmVncmVzc2lvbg0KbG9naXN0aWNtb2RlbD1nbG0oUmVzcG9uc2UgfiBmYWN0b3IoY2FyX21hbnVmYWN0dXJlcikgKyBmYWN0b3IoY2FyX3R5cGUpLA0KICAgICAgICAgICAgZmFtaWx5PWJpbm9taWFsKGxpbms9J2xvZ2l0JyksIGRhdGE9c2VsZWN0ZWRDb21iKQ0KbG9naXN0aWNtb2RlbA0KYGBgDQoNCg0KYGBge3J9DQojIHByZWRpY3QNCmJldHRlcnZlaGljbGUkcmVzcG9uc2UgPSBpZmVsc2UocHJlZGljdChsb2dpc3RpY21vZGVsLGJldHRlcnZlaGljbGUsdHlwZT0icmVzcG9uc2UiKSA+IDAuNSwxLDApDQpiZXR0ZXJ2ZWhpY2xlJHJlc3BvbnNlIA0KYGBgDQoNCg0KYGBge3J9DQpgYGANCg0KPGI+Q29uam9pbnQgQW5hbHlzaXMgPC9iPg0KDQpOb3csIHRoYXQgd2UgaGF2ZSBhbGwgdGhlIGRhdGEgd2UgbmVlZCwgbGV0cyBydW4gYSBjb25qb2ludCBhbmFseXNpcyBhbmQgc3VtbWFyaXplIGltcG9ydGFuY2Ugb2YgdmFyaW91cyBmZWF0dXJlcyBmb3IgY3VzdG9tZXJzLg0KYGBge3IsIGVycm9yPUZBTFNFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFLGVjaG89RkFMU0V9DQoNCiMgU3VydmV5IFJlc3BvbnNlDQpTdXJ2ZXlfYiA9IGRhdGEuZnJhbWUobWF0cml4KGJldHRlcnZlaGljbGUkcmVzcG9uc2UsDQogICAgICAgICAgICAgICAgICAgICAgICAgICBuY29sPTEwOSxucm93PTEpKQ0KDQojIHNhdmUgbGV2ZWxzIG9mIHRoZSByZXNwb25zZQ0KbGV2ZWxzID0gYygnY2hldnJvbGV0JywnZG9kZ2UnLCdob25kYScsJ2xpbmNvbG4nLCdtZXJjdXJ5JywncG9udGlhYycsJ3RveW90YScsJ2NvbXBhY3QnLCdzdXYnLCd0d29zZWF0ZXInLCdtaWRzaXplJywnbWluaXZhbicsJ3BpY2t1cCcsJ3N1YmNvbXBhY3QnKQ0KDQojQ29uam9pbnQNCkNvbmpvaW50KFN1cnZleV9iLGJldHRlcnZlaGljbGVbLDE6Ml0sIHo9bGV2ZWxzKQ0KYGBgDQoNCjxiPkZlYXR1cmUgSW1wb3J0YW5jZTwvYj4NCjxicj4NCjxpPg0KVGhlIEltcG9ydGFuY2Ugb2YgZmVhdHVyZXMgcGxvdCBzaG93cyB0aGF0IGJ1eWVycyBvZiB2ZWhpY2xlcyBjYXJlIG1vcmUgYWJvdXQgdGhlIHR5cGUgb2YgY2FyIGl0IGlzIHRoYW4gdGhleSBkbyB0aGUgbWFudWZhY3R1cmVyLiBXaGljaCBpbiB3YXkgbWFrZXMgc2Vuc2UgYXMgaXQgcHJvdmlkZXMgdGhlbSB3aXRoIGEgd2F5IG9mIHVuZGVyc3RhbmRpbmcgaG93IHRoZSBjYXIgbWlnaHQgYmVzdCBzdWl0IHRoZWlyIG5lZWRzLg0KPC9pPg0KDQpgYGB7cn0NCiMgRmVhdHVyZSBJbXBvcnRhbmNlDQpJbXBvcnRhbmNlMiA9IGRhdGEuZnJhbWUoRmVhdHVyZSA9IGMoJ2Nhcl9tYW51ZmFjdHVyZXInLCdjYXJfdHlwZScpLA0KICAgICAgICAgICAgICAgICAgICAgICAgSW1wb3J0YW5jZT1jYUltcG9ydGFuY2UoeT1TdXJ2ZXlfYiwNCiAgIHg9YmV0dGVydmVoaWNsZVssMToyXSkpDQpJbXBvcnRhbmNlMg0KYGBgDQoNCg0KYGBge3J9DQpnZ3Bsb3QoSW1wb3J0YW5jZTIsYWVzKHJlb3JkZXIoRmVhdHVyZSwtSW1wb3J0YW5jZSksSW1wb3J0YW5jZSkpKw0KICBnZW9tX2JhcihzdGF0PSdpZGVudGl0eScsd2lkdGg9LjgsZmlsbD0nb3JhbmdlMycpKw0KICBnZ3RpdGxlKCdJbXBvcnRhbmNlIG9mIGRpZmZlcmVudCBmZWF0dXJlcycpKw0KICB4bGFiKCcnKSt0aGVtZV9jbGFzc2ljKCkrY29vcmRfZmxpcCgpDQpgYGANCg0KPGJyPg0KPGJyPg0KPGJyPg0KPGJyPg0KYGBge3J9DQpgYGANCg0KPGI+RmVhdHVyZSBVdGlsaXRpZXM8L2I+DQo8YnI+PGJyPg0KPGk+DQpOb3csIGZvciB2ZWhpY2xlIG9wdGlvbnMsIHdlIHVuZGVyc3RhbmQgdGhhdCBjYXIgdHlwZSBpcyB0aGUga2V5IGZhY3RvciwgYWx0aG91Z2ggIHRoZSBtYW51ZmFjdHVyZXIgaXMgc3RpbGwgc2VlbiB0byBiZSBvZiBzb21lIGltcG9ydGFuY2UuIEJ1dCBvdmVyYWxsIHdoaWNoIHR5cGUgb2YgY2FyIGRlc2lnbiBkbyBjb25zdW1lcnMgZmluZCBtb3N0IHBhcmFtb3VudCBpbiB0aGVpciByZWFzb24gb2Ygd2FudGluZyB0byBidXkgaXQ/IFdlIGNhbiBnZXQgb3VyIGFuc3dlciBieSBjb21wYXJpbmcgdGhlIGxldmVscyBvZiBlYWNoIGZlYXR1cmUgb25lIGF0IGEgdGltZQ0KPC9pPg0KDQoNCg0KYGBge3J9DQojc3VtbWFyaXplIHV0aWxpdGllcw0KDQoNCnV0aWwgPSBkYXRhLmZyYW1lKFV0aWxpdGllcyA9IHQNCiAgICAgICAgKGRhdGEuZnJhbWUoY2FQYXJ0VXRpbGl0aWVzKHk9U3VydmV5LA0KICAgICAgICAgIHg9YmV0dGVydmVoaWNsZVssMToyXSx6PWxldmVscykpKSkNCg0KDQp1dGlsJGxldmVscyA9IHJvdy5uYW1lcyh1dGlsKQ0KDQp1dGlsJGxldmVscw0KDQp1dGlsICU+JSANCiAgYXJyYW5nZShkZXNjKFV0aWxpdGllcykpDQpgYGANCg0KDQoNCg0KYGBge3IgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0NCiMgIFR5cGUNCmdncGxvdChkYXRhID0gdXRpbFt3aGljaCh1dGlsJGxldmVscyAlaW4lIGMoJ2NvbXBhY3QnLCdzdXYnLCd0d29zZWF0ZXInLCdtaWRzaXplJywnbWluaXZhbicsICAgICAgJ3BpY2t1cCcsJ3N1YmNvbXBhY3QnKSksXSwgYWVzKHggPSBsZXZlbHMsIHkgPSBVdGlsaXRpZXMsZmlsbD1VdGlsaXRpZXMpKSArDQogIGdlb21fYmFyKHN0YXQ9J2lkZW50aXR5JykgKw0KICBnZ3RpdGxlKCJVdGlsaXRpZXMgb2YgQ2FyIENsYXNzZXMiKSArIHhsYWIoIiIpK2Nvb3JkX2ZsaXAoKSsNCiAgZ2VvbV9obGluZSh5aW50ZXJjZXB0ID0gMCxsdHk9Mixjb2w9J2JsYWNrJyxsaW5ld2lkdGg9LjcpK3NjYWxlX2ZpbGxfZ3JhZGllbnQoaGlnaD0nY3lhbicsbG93PSdibGFjaycpK3RoZW1lKHBsb3QudGl0bGUgPSAgZWxlbWVudF90ZXh0KHNpemU9MTQpLHZqdXN0PTEpK3RoZW1lX2NsYXNzaWMoKSt0aGVtZShsZWdlbmQucG9zaXRpb24gPSAnYm90dG9tJykNCg0KDQpnZ3Bsb3QoZGF0YSA9IHV0aWxbd2hpY2godXRpbCRsZXZlbHMgJWluJSBjKCdjb21wYWN0Jywnc3V2JywnbWlkc2l6ZScsJ3BpY2t1cCcpKSxdLCBhZXMoeCA9IGxldmVscywgeSA9IFV0aWxpdGllcyxmaWxsPVV0aWxpdGllcykpICsNCiAgZ2VvbV9iYXIoc3RhdD0naWRlbnRpdHknKSArDQogIGdndGl0bGUoIlV0aWxpdGllcyBvZiBDYXIgQ2xhc3NlcyIpICsgeGxhYigiIikrY29vcmRfZmxpcCgpKw0KICBnZW9tX2hsaW5lKHlpbnRlcmNlcHQgPSAwLGx0eT0yLGNvbD0nYmxhY2snLGxpbmV3aWR0aD0uNykrc2NhbGVfZmlsbF9ncmFkaWVudChoaWdoPSdjeWFuJyxsb3c9J2JsYWNrJykrdGhlbWUocGxvdC50aXRsZSA9ICBlbGVtZW50X3RleHQoc2l6ZT0xNCksdmp1c3Q9MSkrdGhlbWVfY2xhc3NpYygpK3RoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICdib3R0b20nKQ0KYGBgDQo8aDM+DQo8Yj5JbiBjb25jbHVzaW9uOjwvYj4NCjwvaDM+DQo8YnI+DQo8Yj5GaW5kaW5nczwvYj4NCjxicj4NCg0KPGk+DQpBZnRlciBjb21wYXJpbmcgdGhlIGxldmVscyBvZiBlYWNoIGZlYXR1cmUgaW4gb3VyIHV0aWxpdHkgc3VtbWFyeSBpdCdzIGNsZWFyIGZyb20gdGhlIHJlc3VsdHMgb2YgdGhlIHN1cnZleSB0aGF0IHRoZSBtYWpvcml0eSBvZiByZXNwb25kZW50cyBtb3N0IHByZWZlciB0byBidXkgY2FycyBiYXNlZCBvZmYgb2YgdGhlaXIgdHlwZSBvciBnZW5lcmFsIGRlc2lnbiByYXRoZXIgdGhhbiB0aGUgY2FyJ3MgbWFudWZhY3R1cmVyLiBJbiBhZGRpdGlvbiB0byB0aGlzLCB3ZSBhbHNvIGRpc2NvdmVyZWQgdGhhdCB0aGUgNTElIG9mIHJlc3BvbmRlbnRzIHdobyBoYXBwZW5lZCB0byBidXkgYSB2ZWhpY2xlIGVuZGVkIGdldHRpbmcgb25lIGJlY2F1c2UgaXQgd2FzIGEgcGljayB1cCB0cnVjayB3aGlsZSBhbm90aGVyIDUxJSBvZiByZXNwb25kZW50cyBkaWQgc28gYmVjYXVzZSBvZiBpdCBiZWluZyBtaWRzaXplLiBNZWFud2hpbGUsIDIyLjQlIHB1cmNoYXNlZCB0aGUgdmVoaWNsZSBiZWNhdXNlIG9mIGl0IGJlaW5nIGFuIFNVViBhbmQgbGVzcyB0aGFuIDEwJSBmb3IgYmVpbmcgY29tcGFjdC4gV2hlcmVhcywgcmVzcG9uZGVudHMgZm91bmQgdHdvIHNlYXRlcnMsIHN1YmNvbXBhY3QgYW5kIG1pbml2YW4gY2FyIHR5cGVzIHRvIGJlIG9mIG5vIGltcG9ydGFuY2UgaW4gY2hvb3NpbmcgYSBjYXIgdG8gYnV5LiANCjwvaT4NCjxicj48YnI+DQo8Yj5JbnNpZ2h0czwvYj4NCjxicj4NCjxpPg0KV2hhdCB0aGlzIHRlbGxzIHVzIGlzIHRoYXQgd2Ugc2hvdWxkIGZvY3VzIHRoZSBtYWpvcml0eSBvZiBvdXIgcmVzb3VyY2VzIG1vcmUgaGVhdmlseSB0b3dhcmQgdGhlIGNvbnRpbnVlZCBwcm9kdWN0aW9uIGFuZCBhZHZlcnRpc2luZyBhbG9uZ3NpZGUgdGhlIGFkZGl0aW9uIG9mIG1vcmUgdGVjaG5vbG9naWNhbCBlbmhhbmNlbWVudHMgaW4gcGljayB1cCB0cnVja3MgYW5kIG1pZHNpemUgY2FycyB3aGlsZSBzdGlsbCBhbGxvY2F0aW5nIGEgY2VydGFpbiBwZXJjZW50IG9mIHJlc291cmNlcyB0byB0aGUgcHJvZHVjdGlvbixlbmhhbmNlbWVudCBhbmQgYWR2ZXJ0aXNlbWVudCBvZiBTVVYgdHlwZXMuICAgDQo8L2k+DQoNCg0KDQpgYGB7cn0NCg0KYGBgDQoNCg0KDQoNCg0KYGBge3J9DQpgYGANCg0K