HDR ASSET Workshop #9

Choice Modelling: a summary

Choice models attract academic and applied interest for their ability to generate economically grounded outputs such as demand forecasts, willingness-to-pay estimates, and price elasticities that inform policy and strategic decisions.

(most) Choice models are based on random utility theory and use a form of logistic regression to make estimates about the relative utility decision makers derive from a set of alternatives.

Statistical software that can estimate a logit model can be used to run choice models of this type (e.g., base R, core Python, SPSS, Stata, Matlab, SAS) as well as commercial software packages designed specifically for choice models (NLogit, ALogit, Sawtooth, Biogeme).

Stated preference (SP) experiments

SP experiments (also known as “choice experiments”) are a specific type of data generation process that choice modellers use to record the choices of decision makers.

The task for a respondent in a SP experiment is to select the alternative before them which best maximises their utility. The alternatives are described by attributes, and those attributes have levels.

Typically analysts will observe respondents making several choices. Having multiple observations per respondent allows for more precise estimation of model parameters, as well as allows the analyst to make rich behavioural insights.

Wine choice example

In this example, there are 8 alternatives in the set shown. Each alternative has attributes (price, grape type, etc), and these attributes vary across alternatives ($9.99 vs $13.99, Cabernet vs Shiraz, etc). After making a choice, the respondent will be shown another choice set (another “shelf of wine”) and asked to choose again. The screen is also formatted aesthetically to resemble a real decision making context.

Which bottle would you choose?

Attributes and Levels

The analyst has to carefully select the attributes and levels they will use to describe the range of hypothetical alternatves to display to respondents.

The attributes should reflect the real-world. The attributes should be important, plausible, and understandable to respondents. The levels of the attributes should have realistic ranges, and be sufficiently separated such that decision makers can reasonably trade them off.

Next, the analyst has to combine the attributes and levels together in such a way that is statistically robust and be suitable for respondents.

How do we actually create a SP experiment?

Designing SP experiments

The design of SP experiments typically relies on experimental design plans, which are either generated using specialised software or sourced from established design catalogues.

The design of SP experiments constitutes a specialised field in its own right, encompassing diverse methods of algorithmic optimisation, combinatorial logic, and simulation techniques. These approaches are used to systematically combine attributes and levels into presentable “choice sets” that balance statistical rigour with respondent realism.

Designing SP experiments can be really hard.

Design plans

A design plan is a table of rows and colums, where each row represents an alternative within a choice set, and each column represents an attribute. The cells in the table show which level of each attribute is used for each alternative.

Design plans are structured to reduce overlap between attributes and to cover the range of possibilities efficiently. If researchers already know how people are likely to respond (based on prior studies) they can use Bayesian methods to create more efficient designs tailored to those expectations.

Once a design plan is located/generated, the choice sets need to be constructed and made presentable to respondents.

Example design plan

Creating formatted choice sets

The next step is to transpose the information from the design plan to formatted choice sets.

Survey design software (e.g., Qualtrics) can be used for this purpose using a “tick one” rating scale question with each of the attributes and levels for each alternative described above the selection button.

This method requires manual construction of each alternative within each choice set (a tedious, but doable, process!)

Creating formatted choice sets

becomes choice set 1…

Creating formatted choice sets

becomes choice set 2…

Creating formatted choice sets

becomes choice set 3…

Creating formatted choice sets

becomes choice set 4…

Creating formatted choice sets

becomes choice set 5…

Creating formatted choice sets

becomes choice set 6…

Creating formatted choice sets

becomes choice set 7…

Creating formatted choice sets

becomes choice set 8…

Specialised SP design platforms

Rather than manually transposing the design plan to choice sets in generic survey software, a better approach is to use platforms more specifically developed for the creation of SP experiments to remove the need to manually construct choice sets or reshape data for modelling.

SurveyEngine is the most recommended platform to do for academic research purposes. It provides high degree of statistical rigor and control the analyst to design their SP experiment with full transparency using the most advanced techniques available.

SurveyEngine was formed in Sydney in 2001 by Ben White whilst working as a programmer for Jordan Louviere. The platform integrates Ngene for SP design and Apollo for SP data modelling.

Apollo and Ngene

Apollo is an R package for the estimation of a wide range of choice models, developed and maintained by Stephane Hess & David Palma.

Ngene is specialised design software developed and maintained by John Rose, Michiel Bliemer, Andrew Collins and David Hensher.

SurveyEngine pricing

HDR Workshop access to SurveyEngine

Students in this workshop can access a standard account for FREE for a period of 3 months. Access the sign up page from the QR code or link, select the Standard option and use the promo code below. Once complete login to your SurveyEngine account on your laptop.

SurveyEngine home page

To create a new survey, click “Create Survey”.

Create new survey

To create a new SP choice experiment click “Experiment”.

Create an experiment

SurveyEngine experimental design

Fill in the fields in the SurveyEngine experimental design generator.

Attributes should have a descriptive name.

Attributes should have at least two levels.

Once ready to generate the experimental design plan, click “Design” then click “Generate Orthogonal”.

Generate orthogonal design

Design tips

Stated preference experiments can quickly become too cognitively burdensome for respondents to engage with if they have too many attributes, too many levels or too many alternatives.

Complex designs with too many elements are difficult to find statistically tractable ways to combine together into choice sets.

Simple designs are the best (and worst). Designs with about 4 to 6 attributes, 2 to 5 levels per attribute, and about 3 to 4 alternatives per choice set are close to the “sweet spot” when it comes to designing experiments. However, if a decision context has much more to consider than this in reality, such designs might not accurately lead to respondent behaviour that matches behaviour in that context.

Add choice sets to survey

Collecting data

Once you have completed the design of your experiment and the survey pages surrounding it, you are ready to launch the survey to a panel of respondents.

SurveyEngine offers a panel provider service, or alternatively you can distribute the published URL to an alternative panel provider (e.g., Prolific), Dynata, PureProfile, etc.

Publishing survey

Modelling the data

Once the data has been collected from a panel of respondents, SurveyEngine can generate an MNL model of the SP data within the browser platform.

Models can be generated on the fly as data is being collected. This enables you to track how results are tracking. It is also a good way to check they your experiment (and survey more generally) is behaving as expected.

SurveyEngine will also generate a .r file which can be used to run models offline using Apollo in R.

Estimate an MNL (in SurveyEngine)

Downloading data and R script

Using R

If you have not already, install R and R Studio on your computer.

Download and install R from: https://cran.rstudio.com/

Download and install R Studio from: https://posit.co/download/rstudio-desktop/

Install apollo using:

install.packages("apollo")

Download data and .r file

SurveyEngine packages the data and R script for your first MNL model into a .zip file. Unzip this file into a directory on your computer and open the .r file in R Studio.

Opening the .r file from where you have saved it on your computer should automatically set the correct working directory. If you need help setting your working directory in R, reach out for help.

Once opened in R, run the whole .r file. This will read in the data from SurveyEngine, run the MNL model in Apollo and print the output to the console and files within your working directory.

Estimate MNL in R using Apollo

Interpreting MNL Coefficients

The estimated coefficients from a Multinomial Logit (MNL) model indicate how changes in attribute levels influence decision-makers’ preferences within the context of the study.

• Coefficients reflect changes in utility relative to a preference level (baseline) for each attribute.
• A positive coefficient suggests that increasing the attribute level raises utility (i.e., makes the option more preferred), while a negative coefficient implies a decrease in utility.
• If a cost attribute is included in the model, willingness-to-pay (WTP) estimates can be derived by dividing the coefficient of a non-cost attribute by the cost coefficient to express how much respondents are willing to pay for a one-unit change in the attribute level.

Learning more about choice models

Choice modelling is an advanced methodology. There is only so much we can cover in a 2hr workshop!

Today we put the theory learnt in the previous workshop (Fundamentals of Choice Modelling) into quick practice, covering how to design SP experiments in SurveyEngine and run a first MNL model in Apollo.

ICMC2026

UQ is hosting the 2026 International Choice Modelling Conference on the Gold Coast (20 - 22 July 2026). If you have a study which uses choice modelling, consider submitting it for presentation at ICMC2026! The call for abstracts will come out soon…

Stephane Hess (the author of the Apollo package) will be running day long Apollo workshops here on campus at UQ either before or after the conference (TBA). These workshops will provide a deep dive into running and interpreting models models of both RP and SP data, as well other advanced usage of the Apollo package.

More information will be available at https://www.icmconference.org.uk/ soon. We will also distribute the information to all workshop attendees.

ICMC2026