Max-diff is used to measure relative preferences, with obvious applications in market research. Respondents are shown a series of lists and asked which alternative they like best and which worst. The technique can be used to elicit preferences for relative preferences between brands, particular attributes of one brand, or for different bundles of attributes. Example: I just bought a new TV (after relentless pressure from my younger family members!). I wandered around for hours, staring at all the different options and hoping for some inspiration. It was really difficult: do I go for the 55 inch with this and that, but is the brand reliable? Unbeknown to myself I was in fact using max-diff by ranking the different bundles of alternatives.
Max-diff is a form of discrete choice experiment, a hugely studied area in statistics. For example, a town-planner might want to know where to locate a new bus-stop. The intended users will be making choices: walk to work/take the bus/drive. The location of the stop has to be the solution which passengers will find optimal: less cost (in walking from home to the stop) and all the other factors, such as relative safety. Statistical techniques such as multinomial logit can solve these problems, but the parameter estimates can be a little biased.
One way forward is to use a form of 'tricked' logit. The 'trick' is to transform the data before using logistic regression. Alternative methods include a multi-level approach and a hierarchical Bayesian approach. There is some commercial software available ('Sawtooth') but I have not used it. From what I have read, Sawtooth uses the same technique as described below.
Under the 'incomplete block design approach' respondents are asked to complete sets of questions. The first question contains one subset of alternatives, the next question contains a different subset of alternatives. The questions obviously need to be randomised to reduce bias. Fortunately R contains the package 'AlgDesign' which helps with algorithmic experimental design (and thanks to Bob Wheeler for that). Below is an example of a block from this package, with 10 alternatives and 11 blocks (it could be any reasonable combination). In this example, the respondent is shown 5 alternatives in each block (count'em!). Again, the number can be whatever is required.
block
alternative 1 2 3 4 5 6 7 8 9 10 11
1 0 0 0 1 0 1 1 0 1 1 0
2 0 1 1 1 1 0 0 0 0 1 0
3 0 0 1 0 1 1 1 1 0 0 0
4 1 1 0 0 0 0 1 1 0 1 0
5 1 0 1 1 0 0 0 1 1 0 0
6 1 1 0 1 0 1 1 0 0 0 1
7 0 1 0 1 1 0 0 1 1 0 1
8 1 0 1 0 1 0 1 0 1 0 1
9 1 0 0 0 1 1 0 1 0 1 1
10 0 1 1 0 0 1 0 0 1 1 1
This example comes from Tim Bock with some modifications (by me). First, here is a matrix of responses. The first set of responses had three possible responses (C,D,E). The responses are written in the same way as a dummy variable. The respondent chose C as the best (the 1 in the very first row). In lines 3 to 6, we see the data for Set 2. The respondent is being asked to identify the worst. In line 6 you can see the -1 for Alternative E. Recording the response with a negative sign is the 'trick'. That's all there is to it!
ID Block Set Choice A B C D E F
1 1 1 1 1 0 0 1 0 0 0
2 1 1 1 0 0 0 0 1 0 0
3 1 1 1 0 0 0 0 0 1 0
4 1 1 2 0 0 0 -1 0 0 0
5 1 1 2 0 0 0 0 -1 0 0
6 1 1 2 1 0 0 0 0 -1 0
7 1 2 3 1 1 0 0 0 0 0
8 1 2 3 0 0 0 0 1 0 0
9 1 2 3 0 0 0 0 0 1 0
10 1 2 4 0 -1 0 0 0 0 0
11 1 2 4 0 0 0 0 -1 0 0
12 1 2 4 1 0 0 0 0 -1 0
13 1 3 5 1 0 1 0 0 0 0
14 1 3 5 0 0 0 0 1 0 0
15 1 3 5 0 0 0 0 0 0 1
16 1 3 6 0 0 -1 0 0 0 0
17 1 3 6 0 0 0 0 -1 0 0
18 1 3 6 1 0 0 0 0 0 -1
19 1 4 7 0 1 0 0 0 0 0
20 1 4 7 1 0 0 1 0 0 0
21 1 4 7 0 0 0 0 0 0 1
22 1 4 8 0 -1 0 0 0 0 0
23 1 4 8 0 0 0 -1 0 0 0
24 1 4 8 1 0 0 0 0 0 -1
25 1 5 9 0 1 0 0 0 0 0
26 1 5 9 0 0 1 0 0 0 0
27 1 5 9 1 0 0 1 0 0 0
28 1 5 10 0 -1 0 0 0 0 0
29 1 5 10 1 0 -1 0 0 0 0
30 1 5 10 0 0 0 -1 0 0 0
31 1 6 11 1 0 1 0 0 0 0
32 1 6 11 0 0 0 0 0 1 0
33 1 6 11 0 0 0 0 0 0 1
34 1 6 12 0 0 -1 0 0 0 0
35 1 6 12 0 0 0 0 0 -1 0
36 1 6 12 1 0 0 0 0 0 -1
Using multinomial logistic regression, we can find the rank ordering of the choices. What we are doing is adding up the number of occurrences of 1 or 0 under the column 'Choice' and then atttributing it to each alternative.
Note that alternative A is not one of the choices because it is the 'reference level'. The parameter estimates are with reference to alternative A. We could use any of the possible choices as the reference level.
Call:
mlogit(formula = Choice ~ B + C + D + E + F | 0, data = trickedData, alt.levels = paste(1:nAltsPerSet), shape = "long", method = "nr", print.level = 0)
Coefficients:
B C D E F
-16.6 17.7 -32.9 -50.0 -66.9
Giving alternative A a value of 0, we can see that 17.715 > 0 > -16.572 > -32.902 > -50.011 > - 66.934. it's apparent that alternative C is preferred to all other choices. In rank order of preferences: C > A > B > D > E > F.
We can work out the probability that a respondent would have chosen an alternative when presented with all the alternatives in one go, using inverse logit. For example, probability of choosing C when offered all six is (hardly a surprise) 1. But what about B? We can do this by dividing the exponent of interest through by the sum of all the exponents.
This method violates some assumptions especially the iid assumption and as a result the parameter estimates may be biased. As a result it is probably safest to stick to the rank ordering results.