Basics of Experimental Designs with Applications
Clinton L. Neill, Ph.D.
2023-04-27
Introduction
This document was created as part of a guest lecture for FST 5984: SS Data Analytics for Food and Ag on Experimental Design and Applications. Note that there is an associated PowerPoint that has information not included here. This document is to serve as reference for several applications covered as part of the guest lecture.
This document and associated R code is focused on exploring different aspects of experimental design and only serves as an introduction of the process. The science of experimental design is very broad and has branched into several sub specialties across applied disciplines. Consider this your jumping off point into the fun world of experiments.
Basics about how the document is formatted and presented
To give you some basics about the document, the following is an example of R code and output.
summary(cars)## speed dist
## Min. : 4.0 Min. : 2.00
## 1st Qu.:12.0 1st Qu.: 26.00
## Median :15.0 Median : 36.00
## Mean :15.4 Mean : 42.98
## 3rd Qu.:19.0 3rd Qu.: 56.00
## Max. :25.0 Max. :120.00Note that the output is denoted by the ## on each line.
Also the text style is also different from regular text. I have used R
Markdown to create this document to easily introduce code and output to
you throughout this tutorial.
Including Plots
Sometimes I will also embed plots, for example:
Note that the echo = FALSE parameter was added to the
code chunk to prevent printing of the R code that generated the
plot.
Note that I sometimes suppress output for easier viewing of the document. If you have the correct data and code, you should be able to run the code and see the output yourself.
Defining your Question
The first step in any experiment is to define your question. What phenomenon are you interested in studying? Why is it important that you study this question? Who/What will it impact? I will work through a few examples of such questions throughout this document, but I encourage you to think of your own question/research and code with that in mind. By defining your question before you even think about the experimental design will help facilitate more robust research.
As an aside, economics is the study of trade offs, choices, and scarcity. Most of the choices we make everyday have an economic impact on ourselves, our local economies, and beyond. As an economist, I keep these types of questions in mind and will use this framework in our examples. Enjoy your dive into my areas of economics.
Some quick definitions/notations: treatments are the general things you want to test and levels are the variations within treatments. For example, if I wanted to test the effect of amount of sugar on perceptions of sweetness in tea, my treatment would be the amount of sugar placed in the tea. Within the sugar treatment I could have 3 levels - none, 1 tablespoon, and 2 tablespoons. So we have treatments - the thing we are varying; and levels - how does it vary.
Example #1: Randomized Control Trial: CRD & RCBD
Randomized Control Trials (RCTs) are common forms of experiments that include a control group that does not receive any form of treatment. This general set up of an experiment allows for direct comparison to a baseline group. Any treatments are then applied to other groups with clear distinctions between the various groups. The control group (sometimes called the placebo group in medical studies) has the zero level of the treatment applied to it. This can sometimes be done by just simply observing the group or providing them with a benign (placebo) type treatment. How data is collected must remain constant across groups and information about what group participants are in (if doing work with humans) must remain confidential to prevent spillover effects.
Organizational Interventions to burnout among veterinarians
A current study that I am working on with Behavioral Scientists at Cornell University is a RCT testing different types of interventions to reduce the rate of burnout among clinical veterinarians. This project proposes to investigate how organizational interventions can alleviate burnout among U.S. based Veterinarians. We will investigate two major interventions for lowering burnout and improving job satisfaction in veterinary practice, with a particular focus on:
leadership quality, team communication and teamwork for organizational-directed interventions
individual job crafting for a combined organizational and individual intervention
As such this RCT has 2 different treatments plus a control:
| Condition: | Organizational | Organizational + Individual | Control |
|---|---|---|---|
| Organizational: | Present | Present | None |
| Individual: | None | Present | None |
What is unique about this RCT is that the unit of measurement is the veterinarian. In other words, we want to see the extent of the effect, if any, each type of intervention has on the individual. We are measuring a number of outcomes, but the main one of interest is a psychometric on burnout. However, implementing an organizational level intervention means that all veterinarians that work at a clinic would need to be under the same treatment since we cannot separate veterinarians and randomly assign those in one clinic to one treatment and others to another treatment. This would allow for too much confounding and be impossible to actually implement.
Let’s generate some data to begin our example. Assume we have 240 veterinarians to enroll in our study and 27 clinics:
library(ggplot2)## Warning: package 'ggplot2' was built under R version 4.2.3library(ggpubr)## Warning: package 'ggpubr' was built under R version 4.2.3vet_id <- 1:240
clinic_id <- sample(1:27, size=240, replace=TRUE) #This randomly assigns vets to a clinic.
rct_data <- data.frame(vet_id, clinic_id)
#Graph to vizualize distribution of clinics
ggplot(rct_data, aes(clinic_id)) +
geom_bar(fill = "#0073C2FF") +
theme_pubclean()
Completely Randomized Design
For sake of learning, let’s assume that we can abstract from reality
for a moment. The simplest experimental design would be a Completely
Randomized Design (CRD) in which every veterinarian is randomly assigned
to one of the two treatment groups or the control. Note that in sampling
we focus on a very narrow population of potential clinics that are all
similar in size, type, and general function. In order to implement the
CRD, we would place all of the veterinarians in a dataframe and then use
the package randomizr to assign the participants to each of
the three groups.
#install.packages("randomizr")
library(randomizr)
crd_design <- complete_ra(N=length(rct_data$vet_id), conditions = c("Org", "OrgInd", "Control"))
table(crd_design) #generates the treatment assignment sequence## crd_design
## Org OrgInd Control
## 80 80 80rct_data <- cbind(rct_data, crd_design) #combine to original dataAs you can see from the data, veterinarians from different clinics are now assigned to different groups. If we want to check how many veterinarians are assigned to each treatment or control we simply do the following:
ggplot(rct_data, aes(crd_design)) +
geom_bar(fill = "#0073C2FF") +
theme_pubclean()
This shows us that there is indeed an equal number of units (veterinarians) in each treatment group. If we were to have a sample size that is not divisible by 3 (say 250 veterinarians), then we can still use the same code as before and it will randomly assign the odd number and you will have unequal number of observations in each treatment. This can easily be handled in the analysis phase. This package also allows you to use probability weighting or setting the number of observations in each treatment. I would not do this unless you can easily justify doing so based on previous literature.
Randomized Complete Block Design
As mentioned earlier, a CRD is not really realistic in this case as
we are doing organizational interventions. Instead we want to use a
Randomized Complete Block Design (RCBD). This design is similar to a
CRD, but now allows us to assign clinics to treatments instead of
veterinarians. From a coding perspective, this is done via a different
function in randomizr package:
rcbd_design <- cluster_ra(clusters=rct_data$clinic_id, conditions = c("Org", "OrgInd", "Control"))
table(rcbd_design, rct_data$clinic_id) #generates the treatment assignment sequence##
## rcbd_design 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
## Org 7 0 0 0 0 0 8 0 9 0 0 0 0 0 9 0 0 0 0 5 0 12
## OrgInd 0 11 5 0 9 8 0 7 0 7 4 13 0 0 0 0 0 0 0 0 0 0
## Control 0 0 0 11 0 0 0 0 0 0 0 0 9 14 0 6 5 13 8 0 11 0
##
## rcbd_design 23 24 25 26 27
## Org 7 0 0 11 10
## OrgInd 0 12 0 0 0
## Control 0 0 9 0 0rct_data <- cbind(rct_data, rcbd_design) #combine to original dataWe can see that all veterinarians from one clinic are now assigned to one of the treatments. To ensure equal clinic numbers in each treatment we can analyze this by the following:
library(dplyr)## Warning: package 'dplyr' was built under R version 4.2.3##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unioncounts <- group_by(rct_data, clinic_id, rcbd_design)
table(counts$rcbd_design)##
## Org OrgInd Control
## 78 76 86ggplot(counts, aes(counts$rcbd_design)) +
geom_bar(fill = "#0073C2FF") +
theme_pubclean()
Again, these functions can deal with number of clinics not divisible by the number of treatments. To determine how many veterinarians we have in each treatment, we can again visualize this using similar code:
ggplot(rct_data, aes(rct_data$rcbd_design)) +
geom_bar(fill = "#0073C2FF") +
theme_pubclean()## Warning: Use of `rct_data$rcbd_design` is discouraged.
## ℹ Use `rcbd_design` instead.
In this case, we end up with an equal number of clinics in each treatment as needed, but we have more individual veterinarians in the control due to clinics having differing numbers of staff. Again, we can handle this in the analysis stage quite easily, but we will not get into that here.
Example 2: Choice Experiments and Incomplete Block Designs
Choice experiments are a common method to elicit consumer prefences for attributes on specific products.
State Ag marketing label WTP and competition Effects
A lot of my work has been on marketing labels and consumer
preferences for various types of informative labels. My early work was
on the competitng effects of state agricultural marketing labels across
state borders. The intended effects of such marketing programs is to
increase in state demand for locally made/grown food products. However,
neighboring states also participate in such programs and is sold across
state borders with the neighboring state marketing label, which is known
as a “beggar-thy-neighbor” scenario. There are also regional and
national brands that compete in these marketing schemes. So, we designed
an experiment to test how big the “beggar-thy-neighbor” effects were and
anchored the effect to something homogeneous and produced in every state
- milk. The number of treatments in this case was quite small with only
two attributes, but the number of levels was quite large. We planned to
sample consumers from 8 states, which would require 8 state agricultural
marketing labels plus a regional and a national label - so 10 labels in
total. When you include a price attribute with 2 levels, a factorial
design with a reasonable level of design efficiency would be far too
large, even with blocking. So, we went with a partially balanced
incomplete block design. Incomplete block designs allow for smaller
designs while also minimizing correlation between treatments/attributes.
Also, these designs balance how often the number of times the
combination of attributes appear throughout the design. The package we
use for incomplete block designs is crossdes:
#install.packages("crossdes")
library(crossdes)## Warning: package 'crossdes' was built under R version 4.2.3## Loading required package: AlgDesign## Loading required package: gtools## Warning: package 'gtools' was built under R version 4.2.3Without solving some combonatorics math, BIBDs are difficult to
create appropriately. See the following document (specifically page 13-1
- which is the second page) https://www.stat.purdue.edu/~bacraig/notes1/topic13.pdf
for the specific equations. The function we use for BIBDs is the
find.BIB('treatments', 'blocks', 'elements). Without using
the combonatorics math tricks, we can attempt to create a BIBD through
trial and error as follows:
find.BIB(5, 4, 3) ## [,1] [,2] [,3]
## [1,] 2 3 4
## [2,] 1 3 4
## [3,] 1 2 5
## [4,] 3 4 5isGYD(find.BIB(5, 4, 3), tables = TRUE, type=TRUE) ##
## [1] The design is neither balanced w.r.t. rows nor w.r.t. columns.
##
## $`Number of occurrences of treatments in d`
## 1 2 3 4 5
## 3 2 3 2 2
##
## $`Row incidence matrix of d`
## 1 2 3 4
## 1 1 1 0 1
## 2 0 0 1 1
## 3 1 1 0 1
## 4 0 1 1 0
## 5 1 0 1 0
##
## $`Column incidence matrix of d`
## 1 2 3
## 1 3 0 0
## 2 1 1 0
## 3 0 2 1
## 4 0 1 1
## 5 0 0 2
##
## $`Concurrence w.r.t. rows`
## 1 2 3 4 5
## 1 3 1 3 1 1
## 2 1 2 1 1 1
## 3 3 1 3 1 1
## 4 1 1 1 2 1
## 5 1 1 1 1 2
##
## $`Concurrence w.r.t. columns`
## 1 2 3 4 5
## 1 9 3 0 0 0
## 2 3 2 2 1 0
## 3 0 2 5 3 2
## 4 0 1 3 2 2
## 5 0 0 2 2 4We attempted to create the following design: five treatments in four
blocks of three elements. The isGYD() function tests to see
if the design is Balanced. Obviously this is not a balanced design. If
you think of each row as a block there is clearly not a balance between
the treatments. If we had used the combonatorics math, we would have
found that we need 10 blocks to create a balanced design:
find.BIB(5, 10, 3) ## [,1] [,2] [,3]
## [1,] 2 3 5
## [2,] 1 2 4
## [3,] 1 4 5
## [4,] 1 3 5
## [5,] 2 3 4
## [6,] 1 3 4
## [7,] 2 4 5
## [8,] 1 2 5
## [9,] 1 2 3
## [10,] 3 4 5isGYD(find.BIB(5, 10, 3), tables = TRUE, type=TRUE) ##
## [1] The design is a balanced incomplete block design w.r.t. rows.
##
## $`Number of occurrences of treatments in d`
## 1 2 3 4 5
## 6 6 6 6 6
##
## $`Row incidence matrix of d`
## 1 2 3 4 5 6 7 8 9 10
## 1 1 1 0 0 1 0 1 1 0 1
## 2 0 1 0 1 1 1 0 0 1 1
## 3 1 0 1 1 0 1 1 0 0 1
## 4 1 0 1 0 1 1 0 1 1 0
## 5 0 1 1 1 0 0 1 1 1 0
##
## $`Column incidence matrix of d`
## 1 2 3
## 1 6 0 0
## 2 3 3 0
## 3 1 4 1
## 4 0 3 3
## 5 0 0 6
##
## $`Concurrence w.r.t. rows`
## 1 2 3 4 5
## 1 6 3 3 3 3
## 2 3 6 3 3 3
## 3 3 3 6 3 3
## 4 3 3 3 6 3
## 5 3 3 3 3 6
##
## $`Concurrence w.r.t. columns`
## 1 2 3 4 5
## 1 36 18 6 0 0
## 2 18 18 15 9 0
## 3 6 15 18 15 6
## 4 0 9 15 18 18
## 5 0 0 6 18 36This design would be considered a BIBD.
For the purpose of the State Marketing label example, we would have 20 potential options when we combine the label and price alternatives. In addition, we wanted each choice question to have 4 options plus an additional “no buy” option. We found that 20 blocks or questions would maximize efficiency without having too many questions. So, following our previous code we generate the following design:
find.BIB(20, 20, 4) ## [,1] [,2] [,3] [,4]
## [1,] 3 5 14 18
## [2,] 4 5 9 19
## [3,] 1 5 8 13
## [4,] 9 13 18 20
## [5,] 6 8 11 14
## [6,] 2 4 15 20
## [7,] 11 13 15 19
## [8,] 6 15 16 18
## [9,] 4 7 14 17
## [10,] 3 11 16 20
## [11,] 2 6 9 10
## [12,] 1 10 14 15
## [13,] 6 7 12 13
## [14,] 1 9 11 17
## [15,] 12 17 18 19
## [16,] 2 3 8 17
## [17,] 2 5 7 16
## [18,] 8 10 12 20
## [19,] 3 7 10 19
## [20,] 1 4 12 16isGYD(find.BIB(20, 20, 4), tables = TRUE, type=TRUE) ##
## [1] The design is neither balanced w.r.t. rows nor w.r.t. columns.
##
## $`Number of occurrences of treatments in d`
## 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
## 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
##
## $`Row incidence matrix of d`
## 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
## 1 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0
## 2 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0
## 3 0 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0
## 4 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0
## 5 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0 0
## 6 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0
## 7 0 0 0 0 1 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0
## 8 0 0 0 1 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0
## 9 0 0 1 0 0 0 0 0 0 0 1 1 0 0 1 0 0 0 0 0
## 10 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 1
## 11 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0
## 12 0 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1
## 13 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 1 0 0 0 0
## 14 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 1
## 15 0 0 0 1 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0
## 16 1 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 0
## 17 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1
## 18 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 0 0 1 0 0
## 19 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 0
## 20 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0
##
## $`Column incidence matrix of d`
## 1 2 3 4
## 1 4 0 0 0
## 2 3 1 0 0
## 3 3 1 0 0
## 4 3 0 1 0
## 5 1 3 0 0
## 6 3 0 1 0
## 7 0 3 1 0
## 8 1 2 1 0
## 9 0 3 0 1
## 10 1 2 1 0
## 11 0 1 3 0
## 12 0 2 0 2
## 13 1 0 3 0
## 14 0 1 2 1
## 15 0 0 2 2
## 16 0 1 2 1
## 17 0 0 2 2
## 18 0 0 0 4
## 19 0 0 1 3
## 20 0 0 0 4
##
## $`Concurrence w.r.t. rows`
## 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
## 1 4 1 1 1 0 0 1 0 1 1 0 1 0 0 1 1 0 1 1 1
## 2 1 4 0 1 1 0 1 0 0 0 1 1 0 1 1 0 1 1 1 1
## 3 1 0 4 0 1 1 1 1 1 0 1 1 0 1 1 0 1 0 1 0
## 4 1 1 0 4 1 0 1 1 1 1 1 1 1 0 1 1 0 0 0 0
## 5 0 1 1 1 4 1 0 0 1 0 1 1 1 0 0 1 1 1 0 1
## 6 0 0 1 0 1 4 1 1 0 1 1 1 0 0 1 1 1 1 0 1
## 7 1 1 1 1 0 1 4 1 1 0 0 0 0 1 1 1 1 0 0 1
## 8 0 0 1 1 0 1 1 4 1 0 1 1 1 1 1 0 0 1 1 0
## 9 1 0 1 1 1 0 1 1 4 1 0 0 1 0 0 0 1 1 1 1
## 10 1 0 0 1 0 1 0 0 1 4 1 1 1 1 1 0 1 1 0 1
## 11 0 1 1 1 1 1 0 1 0 1 4 0 0 1 1 1 0 0 1 1
## 12 1 1 1 1 1 1 0 1 0 1 0 4 1 1 0 0 1 0 1 0
## 13 0 0 0 1 1 0 0 1 1 1 0 1 4 1 1 1 0 1 1 1
## 14 0 1 1 0 0 0 1 1 0 1 1 1 1 4 0 1 1 1 0 1
## 15 1 1 1 1 0 1 1 1 0 1 1 0 1 0 4 0 0 1 1 0
## 16 1 0 0 1 1 1 1 0 0 0 1 0 1 1 0 4 1 1 1 1
## 17 0 1 1 0 1 1 1 0 1 1 0 1 0 1 0 1 4 1 1 0
## 18 1 1 0 0 1 1 0 1 1 1 0 0 1 1 1 1 1 4 0 0
## 19 1 1 1 0 0 0 0 1 1 0 1 1 1 0 1 1 1 0 4 1
## 20 1 1 0 0 1 1 1 0 1 1 1 0 1 1 0 1 0 0 1 4
##
## $`Concurrence w.r.t. columns`
## 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
## 1 16 12 12 12 4 12 0 4 0 4 0 0 4 0 0 0 0 0 0 0
## 2 12 10 10 9 6 9 3 5 3 5 1 2 3 1 0 1 0 0 0 0
## 3 12 10 10 9 6 9 3 5 3 5 1 2 3 1 0 1 0 0 0 0
## 4 12 9 9 10 3 10 1 4 0 4 3 0 6 2 2 2 2 0 1 0
## 5 4 6 6 3 10 3 9 7 9 7 3 6 1 3 0 3 0 0 0 0
## 6 12 9 9 10 3 10 1 4 0 4 3 0 6 2 2 2 2 0 1 0
## 7 0 3 3 1 9 1 10 7 9 7 6 6 3 5 2 5 2 0 1 0
## 8 4 5 5 4 7 4 7 6 6 6 5 4 4 4 2 4 2 0 1 0
## 9 0 3 3 0 9 0 9 6 10 6 3 8 0 4 2 4 2 4 3 4
## 10 4 5 5 4 7 4 7 6 6 6 5 4 4 4 2 4 2 0 1 0
## 11 0 1 1 3 3 3 6 5 3 5 10 2 9 7 6 7 6 0 3 0
## 12 0 2 2 0 6 0 6 4 8 4 2 8 0 4 4 4 4 8 6 8
## 13 4 3 3 6 1 6 3 4 0 4 9 0 10 6 6 6 6 0 3 0
## 14 0 1 1 2 3 2 5 4 4 4 7 4 6 6 6 6 6 4 5 4
## 15 0 0 0 2 0 2 2 2 2 2 6 4 6 6 8 6 8 8 8 8
## 16 0 1 1 2 3 2 5 4 4 4 7 4 6 6 6 6 6 4 5 4
## 17 0 0 0 2 0 2 2 2 2 2 6 4 6 6 8 6 8 8 8 8
## 18 0 0 0 0 0 0 0 0 4 0 0 8 0 4 8 4 8 16 12 16
## 19 0 0 0 1 0 1 1 1 3 1 3 6 3 5 8 5 8 12 10 12
## 20 0 0 0 0 0 0 0 0 4 0 0 8 0 4 8 4 8 16 12 16What we find is that this design is not fully balanced. Thus, we have a partially balanced incomplete block design. It is partially balanced as all 20 treatments appear equally throughout the design, but the co-occurrence of treatments is not equal. While this was not optimal, we sampled a large enough group of consumers and used an analysis that accounts for the wider variance in treatment estimates. An example of the choice experiment questions is seen here:
Overall, this study proved a success and allowed for a better understanding of the competitive effects of state marketing labels. If you would like to read more about it, feel free to read the published version here.
Best-Worst Experiments
An interesting type of economic experiment is a Best-Worst study that forces survey participants to make trade offs between alternatives/treatments. This type of experiment helps in creating rankings and does not allow people to avoid making a choice (choice experiments typically allow for a ‘no buy’ option). This requires a completely balanced incomplete block design - where treatments and co-occurrence of treatments appear the same number of times. A recent example of this was part of a research study on Chinese consumer preferences for U.S. grown rice, we asked participants in the U.S., Mainland China, and Hong Kong to complete a best-worst experiment on rice attributes to determine which ones were most important. We had 7 attributes and wanted blocks of 4 treatments. So a BIBD would be:
find.BIB(7, 7, 4) ## [,1] [,2] [,3] [,4]
## [1,] 3 4 6 7
## [2,] 1 2 3 4
## [3,] 2 4 5 6
## [4,] 1 2 6 7
## [5,] 1 3 5 6
## [6,] 1 4 5 7
## [7,] 2 3 5 7isGYD(find.BIB(7, 7, 4), tables = TRUE, type=TRUE) ##
## [1] The design is a balanced incomplete block design w.r.t. rows.
##
## $`Number of occurrences of treatments in d`
## 1 2 3 4 5 6 7
## 4 4 4 4 4 4 4
##
## $`Row incidence matrix of d`
## 1 2 3 4 5 6 7
## 1 1 1 0 0 0 1 1
## 2 1 1 1 0 1 0 0
## 3 0 1 0 1 1 0 1
## 4 1 0 0 1 1 1 0
## 5 0 0 1 0 1 1 1
## 6 1 0 1 1 0 0 1
## 7 0 1 1 1 0 1 0
##
## $`Column incidence matrix of d`
## 1 2 3 4
## 1 4 0 0 0
## 2 2 2 0 0
## 3 1 2 1 0
## 4 0 2 2 0
## 5 0 1 2 1
## 6 0 0 2 2
## 7 0 0 0 4
##
## $`Concurrence w.r.t. rows`
## 1 2 3 4 5 6 7
## 1 4 2 2 2 2 2 2
## 2 2 4 2 2 2 2 2
## 3 2 2 4 2 2 2 2
## 4 2 2 2 4 2 2 2
## 5 2 2 2 2 4 2 2
## 6 2 2 2 2 2 4 2
## 7 2 2 2 2 2 2 4
##
## $`Concurrence w.r.t. columns`
## 1 2 3 4 5 6 7
## 1 16 8 4 0 0 0 0
## 2 8 8 6 4 2 0 0
## 3 4 6 6 6 4 2 0
## 4 0 4 6 8 6 4 0
## 5 0 2 4 6 6 6 4
## 6 0 0 2 4 6 8 8
## 7 0 0 0 0 4 8 16An example of the best-worst question would look like the following:
The presentation of the order of the treatments/attributes is randomized for each person to prevent any order effects. The data allows us to create a ranking of attributes for each individual and average across the different groups for comparison.
Example 3: Basket Based Designs with Fractional/Full Factorials
The most common way to design choice experiments is to use a full or
fractional factorial. This type of design allows for design
orthogonality (no correlation) between treatments and offers a degree of
randomization between the choice options. There are a number of packages
you can use to render the designs such as AlgDesign,
conjoint, and support.CEs. The package
AlgDesigns allows for the most flexibility and calculates
design efficiency for you. However, designs that require higher numbers
of levels for the treatments (more than 2 or 3), a lot of manual
manipulation is required. The conjoint and
support.CEs packages are much easier to use and can handle
treatments with greater numbers of levels. It is important to note that
choice experiments also require a design that handles orthogonality for
multiple options within and across repeated choice sets questions.
Recent advancements in choice experiments have focused on designs that allow the participants to choose multiple alternatives rather than a single, discrete option. In addition, some designs also allow for participants to choose quantities of each good. In essence, designs are now allowing for participants to make multiple, continuous choices. Such designs open the world of choice experiments to examining a wider variety of questions such as examining substitutes and complements among alternatives, satiation of goods, and examine degree of variety seeking.
To showcase the full and fractional factorial designs we will walk
through a couple basket based examples using the conjoint
package.
#install.packages("conjoint")
library(conjoint)## Warning: package 'conjoint' was built under R version 4.2.3Student vegetable choice
A few years ago, Dr. Lahne and I started to think about basket based designs as we wanted to better extract how people made food choices with and without sensory (taste) components. Basket designs had just been proposed but these options limited the experiment to just choosing multiple options and did not allow for quantities to be chosen. So, we created a choice experiment design that would allow for such an option. Our design, the Basket and Expenditure Based Choice Experiment, allowed participants to choose any continuous amount of product and choose as many products as they wish. We also include a repeated choice component where we vary the prices of all the goods in a statistically efficient way. This creates a complex analysis on the back end, but is a much more realistic scenario for someone who is grocery shopping. You can read the entire paper here.
In our initial experiment we had 6 vegetables each varied at 3 price per pound levels. Each of the vegetable options were available at all times, we just varied the prices. We performed the experiment with students in a large undergraduate class and some did a sensory experiment along side the choice experiment while others only did the choice experiment. Students could choose to purchase none, some, or all of the vegetables and indicated how much of their normal weekly food budget they intended to spend on each vegetable item. An example of the question can be seen here:
They then had a follow up to make sure they were spending an amount reasonable with their budget:
To create the basket based question, we need a factorial design that has 6 options always available but varies the prices per pound in such a way that they are not correlated throughout the repeated questions.
Before we dig into that, let’s quickly talk about the math behind a full factorial. Assuming we only have one choice option, the one we design or a ‘no buy’ option, we can determine the number of options a full factorial would contain. Assume L=“Levels” and T=“Treatments”, then the formula for the full factorial is L^(# of T with same # of L). If we have 2 treatments each with 3 levels: 3^2 = 9 total choice questions. However, if we reduced the number of levels of one treatment to 2, then the math is a little different: 3^(1)*2(1) = 6 total choices.
To do this in our case with 6 treatments all at 3 level (3^6=729) we
use the following code to generate a full factorial. We use
expand.grid to allow us to create an easy
data.frame for our attributes and levels
experiment<-expand.grid( pepper_price = c("$1.80", "$3.40", "$5.20"),
turnip_price = c("$1.00", "$2.00", "$3.00"),
carrot_price = c("$0.60", "$1.00", "$1.60"),
tomato_price = c("$0.60", "$1.20", "$1.80"),
kale_price = c("$1.40", "$3.00", "$4.40"),
lettuce_price = c("$0.80", "$1.60", "$2.20"))Now to create the design we use the
caFactorialDesign(data, type='', seed=' ') to create the
design. The type is to specify a full or fractional
factorial, while seed tells you the random set to start the
algorithm. Let’s start with a full factorial.
design=caFactorialDesign(data=experiment,type="full", seed=55751)
#print(design)
print(cor(caEncodedDesign(design))) ## pepper_price turnip_price carrot_price tomato_price kale_price
## pepper_price 1 0 0 0 0
## turnip_price 0 1 0 0 0
## carrot_price 0 0 1 0 0
## tomato_price 0 0 0 1 0
## kale_price 0 0 0 0 1
## lettuce_price 0 0 0 0 0
## lettuce_price
## pepper_price 0
## turnip_price 0
## carrot_price 0
## tomato_price 0
## kale_price 0
## lettuce_price 1As we can see, all combinations are presented. The
cor(caEncodedDesign(design)) just tells us whether their
correlation between treatments. If the correlation is high, you likely
need a different design as you might be confounding/aliasing treatment
effects. In this case, the full factorial is orthogonal and since there
are no cross-vegetable correlations. Yet, 729 questions is unreasonable
for one person to examine. As such, we can use a design called a
fractional factorial which reduces the experiment design and attempts to
minimize the amount of cross attribute correlation. We can do this by
the following:
design.fract=caFactorialDesign(data=experiment,type="ca", seed=55751)
print(design.fract)## pepper_price turnip_price carrot_price tomato_price kale_price
## 26 $3.40 $3.00 $1.60 $0.60 $1.40
## 31 $1.80 $2.00 $0.60 $1.20 $1.40
## 105 $5.20 $2.00 $1.60 $0.60 $3.00
## 154 $1.80 $1.00 $1.60 $1.80 $3.00
## 173 $3.40 $1.00 $1.00 $0.60 $4.40
## 225 $5.20 $3.00 $0.60 $1.80 $4.40
## 246 $5.20 $1.00 $0.60 $0.60 $1.40
## 367 $1.80 $3.00 $1.00 $1.20 $3.00
## 482 $3.40 $2.00 $1.60 $1.80 $4.40
## 555 $5.20 $2.00 $1.00 $1.80 $1.40
## 596 $3.40 $1.00 $0.60 $1.20 $3.00
## 655 $1.80 $3.00 $0.60 $0.60 $4.40
## 696 $5.20 $1.00 $1.60 $1.20 $4.40
## lettuce_price
## 26 $0.80
## 31 $0.80
## 105 $0.80
## 154 $0.80
## 173 $0.80
## 225 $0.80
## 246 $1.60
## 367 $1.60
## 482 $1.60
## 555 $2.20
## 596 $2.20
## 655 $2.20
## 696 $2.20print(cor(caEncodedDesign(design.fract)))## pepper_price turnip_price carrot_price tomato_price kale_price
## pepper_price 1.00000000 -0.21551724 0.1058626 0.00862069 -0.00862069
## turnip_price -0.21551724 1.00000000 -0.1058626 -0.00862069 0.00862069
## carrot_price 0.10586263 -0.10586263 1.0000000 0.10586263 0.10586263
## tomato_price 0.00862069 -0.00862069 0.1058626 1.00000000 0.12068966
## kale_price -0.00862069 0.00862069 0.1058626 0.12068966 1.00000000
## lettuce_price 0.12407292 -0.12407292 -0.2031498 0.09098681 0.12407292
## lettuce_price
## pepper_price 0.12407292
## turnip_price -0.12407292
## carrot_price -0.20314980
## tomato_price 0.09098681
## kale_price 0.12407292
## lettuce_price 1.00000000As you can see the design only returns specific rows of the full factorial. One thing to note, you need to ensure that all of the treatment levels are present across the fractional design. If all treatment levels are not present, you are missing variation and we prefer at least one direct observation of that level within the choice experiment.
Also, you can see the cross-correlations are very small which means this design does not have higher order effects. By changing type to ‘orthogonal’, we obtain a design that has zero cross attribute correlation. This is typically what you want, but not necessary if cross correlations are small.Once you have the statistical design created and duplicates removed, you can design your experiment in Qualtrics or other program. This is where the “Art” of the process comes in. There are plenty of samples of visually appealing surveys on the internet. If you would like to see one of mine, just ask!
Cider choice among alcoholic beverages
Our last example is one from a more recent basket based study about how hard (alcoholic) cider choices are made in comparison to wine and beer. In this study we are more interested in types of each alcoholic beverage and how they influence preferences and choices. As such, we have to vary more attributes than price in this situation. In this study we have 4 options always available: red wine, white wine, beer, and cider. The attributes we vary are varieties/descriptors of each option, price, and in cider we vary whether it is packaged in bottles and cans. For red wine this is Cabernet Sauvignon, Merlot, and Pinot Noir; white wine is Pinot Gris, Sauvignon Blanc, and Chardonnay; Beer is Pilsner, IPA, and Stout; Cider is Fruity, Tart, and Funky.
Once again a full factorial would be far too large in this case -
13,122 options. In this case, we were interested in a orthogonal design.
Unfortunately, with this large of a design the conjoint
package takes awhile to find the one that meets our needs given the
search algorithm. Instead we will use the support.CEs
package as it has alternative algorithms that are much faster in
generating the design.
#install.packages("support.CEs")
library(support.CEs)## Warning: package 'support.CEs' was built under R version 4.2.3## Registered S3 method overwritten by 'DoE.base':
## method from
## factorize.factor conf.designThe code for generating the fractional factorial with orthogonality is straight forward:
d.BEBCE.cider <- Lma.design(
attribute.names = list(
p_wine_red = c("$10", "$17", "$24"),
type_wine_red = c("CabSav", "Merlot", "PinotNoir"), #"Zinfandel", "Malbec"),
p_wine_white = c("$10", "$15", "$20"),
type_wine_white = c("Chard", "PinotGris", "SauvBlanc"), #"Riesling", "Chenin Blanc"),
p_beer = c("$5", "$8", "$11"),
type_beer = c("Pilsner", "IPA", "Stout"), #"Sour", "Gose", "Wheat Beer/Hefeweizen"),
p_cider = c("$10", "$15", "$20"),
type_cider = c("Fruity", "Funky", "Tart"), #"Type4")
pack_cider = c("750ml Bottle", "6-pack of Cans")
),
nalternatives = 1,
nblocks = 1,
row.renames = FALSE,
seed = 55751)## The columns of the array have been used in order of appearance.
## For designs with relatively few columns,
## the properties can sometimes be substantially improved
## using option columns with min3 or even min34.d.BEBCE.cider##
## Choice sets:
## alternative 1 in each choice set
## BLOCK QES ALT p_wine_red type_wine_red p_wine_white type_wine_white p_beer
## 19 1 1 1 $17 Merlot $20 SauvBlanc $5
## 15 1 2 1 $24 Merlot $10 PinotGris $8
## 18 1 3 1 $10 CabSav $15 PinotGris $11
## 31 1 4 1 $10 CabSav $10 Chard $8
## 21 1 5 1 $10 CabSav $15 PinotGris $5
## 32 1 6 1 $10 PinotNoir $20 PinotGris $5
## 6 1 7 1 $17 Merlot $20 SauvBlanc $8
## 25 1 8 1 $24 CabSav $20 PinotGris $8
## 13 1 9 1 $24 PinotNoir $10 Chard $8
## 24 1 10 1 $17 CabSav $10 SauvBlanc $8
## 3 1 11 1 $17 Merlot $15 PinotGris $8
## 35 1 12 1 $17 CabSav $20 Chard $5
## 5 1 13 1 $10 CabSav $10 Chard $5
## 29 1 14 1 $17 CabSav $10 SauvBlanc $11
## 8 1 15 1 $10 Merlot $10 SauvBlanc $5
## 20 1 16 1 $24 Merlot $15 Chard $11
## 4 1 17 1 $17 PinotNoir $15 Chard $5
## 23 1 18 1 $17 PinotNoir $10 PinotGris $11
## 34 1 19 1 $24 CabSav $15 SauvBlanc $8
## 36 1 20 1 $17 PinotNoir $10 PinotGris $5
## 26 1 21 1 $10 Merlot $20 Chard $11
## 30 1 22 1 $17 CabSav $20 Chard $11
## 1 1 23 1 $10 PinotNoir $15 SauvBlanc $8
## 33 1 24 1 $17 PinotNoir $15 Chard $8
## 7 1 25 1 $10 Merlot $20 Chard $8
## 11 1 26 1 $24 Merlot $10 PinotGris $5
## 22 1 27 1 $24 PinotNoir $20 SauvBlanc $5
## 12 1 28 1 $24 PinotNoir $10 Chard $11
## 9 1 29 1 $24 CabSav $20 PinotGris $11
## 17 1 30 1 $24 PinotNoir $20 SauvBlanc $11
## 2 1 31 1 $17 Merlot $15 PinotGris $11
## 27 1 32 1 $10 Merlot $10 SauvBlanc $11
## 16 1 33 1 $24 Merlot $15 Chard $5
## 14 1 34 1 $10 PinotNoir $15 SauvBlanc $11
## 10 1 35 1 $24 CabSav $15 SauvBlanc $5
## 28 1 36 1 $10 PinotNoir $20 PinotGris $8
## type_beer p_cider type_cider pack_cider
## 19 Pilsner $10 Fruity 750ml Bottle
## 15 Pilsner $20 Fruity 6-pack of Cans
## 18 Stout $20 Tart 750ml Bottle
## 31 IPA $20 Tart 750ml Bottle
## 21 Pilsner $15 Funky 750ml Bottle
## 32 IPA $20 Fruity 6-pack of Cans
## 6 IPA $20 Tart 750ml Bottle
## 25 Stout $15 Fruity 750ml Bottle
## 13 IPA $15 Funky 750ml Bottle
## 24 Stout $10 Funky 6-pack of Cans
## 3 IPA $15 Funky 750ml Bottle
## 35 Stout $15 Tart 6-pack of Cans
## 5 Pilsner $10 Fruity 750ml Bottle
## 29 IPA $15 Fruity 6-pack of Cans
## 8 Stout $15 Tart 750ml Bottle
## 20 Pilsner $15 Tart 6-pack of Cans
## 4 IPA $10 Tart 750ml Bottle
## 23 Pilsner $15 Tart 6-pack of Cans
## 34 Pilsner $10 Tart 6-pack of Cans
## 36 Stout $20 Funky 6-pack of Cans
## 26 IPA $15 Fruity 6-pack of Cans
## 30 Pilsner $20 Funky 6-pack of Cans
## 1 Stout $15 Fruity 6-pack of Cans
## 33 Pilsner $20 Fruity 750ml Bottle
## 7 Stout $10 Funky 6-pack of Cans
## 11 IPA $10 Tart 6-pack of Cans
## 22 Pilsner $15 Funky 750ml Bottle
## 12 Stout $10 Fruity 750ml Bottle
## 9 IPA $10 Funky 750ml Bottle
## 17 Stout $20 Tart 750ml Bottle
## 2 Stout $10 Fruity 750ml Bottle
## 27 Pilsner $20 Funky 750ml Bottle
## 16 Stout $20 Funky 6-pack of Cans
## 14 IPA $10 Funky 6-pack of Cans
## 10 IPA $20 Fruity 6-pack of Cans
## 28 Pilsner $10 Tart 6-pack of Cans
##
## Candidate design:
## A B C D E F G H J
## 1 1 3 2 3 2 3 2 1 2
## 2 2 2 2 2 3 3 1 1 1
## 3 2 2 2 2 2 2 2 2 1
## 4 2 3 2 1 1 2 1 3 1
## 5 1 1 1 1 1 1 1 1 1
## 6 2 2 3 3 2 2 3 3 1
## 7 1 2 3 1 2 3 1 2 2
## 8 1 2 1 3 1 3 2 3 1
## 9 3 1 3 2 3 2 1 2 1
## 10 3 1 2 3 1 2 3 1 2
## 11 3 2 1 2 1 2 1 3 2
## 12 3 3 1 1 3 3 1 1 1
## 13 3 3 1 1 2 2 2 2 1
## 14 1 3 2 3 3 2 1 2 2
## 15 3 2 1 2 2 1 3 1 2
## 16 3 2 2 1 1 3 3 2 2
## 17 3 3 3 3 3 3 3 3 1
## 18 1 1 2 2 3 3 3 3 1
## 19 2 2 3 3 1 1 1 1 1
## 20 3 2 2 1 3 1 2 3 2
## 21 1 1 2 2 1 1 2 2 1
## 22 3 3 3 3 1 1 2 2 1
## 23 2 3 1 2 3 1 2 3 2
## 24 2 1 1 3 2 3 1 2 2
## 25 3 1 3 2 2 3 2 1 1
## 26 1 2 3 1 3 2 2 1 2
## 27 1 2 1 3 3 1 3 2 1
## 28 1 3 3 2 2 1 1 3 2
## 29 2 1 1 3 3 2 2 1 2
## 30 2 1 3 1 3 1 3 2 2
## 31 1 1 1 1 2 2 3 3 1
## 32 1 3 3 2 1 2 3 1 2
## 33 2 3 2 1 2 1 3 1 1
## 34 3 1 2 3 2 1 1 3 2
## 35 2 1 3 1 1 3 2 3 2
## 36 2 3 1 2 1 3 3 2 2
## class=design, type= oa
##
## Design information:
## number of blocks = 1
## number of questions per block = 36
## number of alternatives per choice set = 1
## number of attributes per alternative = 9As you can see, this design generates 36 choice questions. This is
still too many for any one person to answer. Instead of doing what we
did before and have a non-orthogonal, we can create blocks of questions
that are orthogonal within and across blocks. To do this, we just change
the nblocks = ' ' in the previous code. Three blocks would
create a reasonable number of questions (12) for people to answer:
d.BEBCE.cider.blocks <- Lma.design(
attribute.names = list(
p_wine_red = c("$10", "$17", "$24"),
type_wine_red = c("CabSav", "Merlot", "PinotNoir"), #"Zinfandel", "Malbec"),
p_wine_white = c("$10", "$15", "$20"),
type_wine_white = c("Chard", "PinotGris", "SauvBlanc"), #"Riesling", "Chenin Blanc"),
p_beer = c("$5", "$8", "$11"),
type_beer = c("Pilsner", "IPA", "Stout"), #"Sour", "Gose", "Wheat Beer/Hefeweizen"),
p_cider = c("$10", "$15", "$20"),
type_cider = c("Fruity", "Funky", "Tart"), #"Type4")
pack_cider = c("750ml Bottle", "6-pack of Cans")
),
nalternatives = 1,
nblocks = 3,
row.renames = FALSE,
seed = 987)## The columns of the array have been used in order of appearance.
## For designs with relatively few columns,
## the properties can sometimes be substantially improved
## using option columns with min3 or even min34.d.BEBCE.cider.blocks##
## Choice sets:
## alternative 1 in each choice set
## BLOCK QES ALT p_wine_red type_wine_red p_wine_white type_wine_white p_beer
## 35 1 1 1 $24 Merlot $10 PinotGris $5
## 30 1 2 1 $24 PinotNoir $20 SauvBlanc $5
## 15 1 3 1 $17 Merlot $20 SauvBlanc $8
## 8 1 4 1 $24 CabSav $20 PinotGris $8
## 28 1 5 1 $17 PinotNoir $15 Chard $8
## 3 1 6 1 $17 CabSav $10 SauvBlanc $8
## 16 1 7 1 $17 PinotNoir $10 PinotGris $11
## 20 1 8 1 $10 CabSav $15 PinotGris $11
## 10 1 9 1 $10 CabSav $10 Chard $5
## 24 1 10 1 $24 Merlot $15 Chard $5
## 21 1 11 1 $10 PinotNoir $15 SauvBlanc $11
## 25 1 12 1 $10 Merlot $20 Chard $11
## 26 2 1 1 $24 CabSav $15 SauvBlanc $5
## 19 2 2 1 $10 PinotNoir $20 PinotGris $8
## 17 2 3 1 $24 Merlot $15 Chard $11
## 11 2 4 1 $10 Merlot $10 SauvBlanc $11
## 29 2 5 1 $10 CabSav $10 Chard $8
## 1 2 6 1 $10 PinotNoir $15 SauvBlanc $8
## 27 2 7 1 $17 Merlot $20 SauvBlanc $5
## 22 2 8 1 $24 PinotNoir $10 Chard $11
## 4 2 9 1 $17 PinotNoir $10 PinotGris $5
## 5 2 10 1 $24 CabSav $20 PinotGris $11
## 18 2 11 1 $17 Merlot $15 PinotGris $8
## 2 2 12 1 $17 CabSav $20 Chard $5
## 13 3 1 1 $24 PinotNoir $20 SauvBlanc $11
## 23 3 2 1 $17 CabSav $20 Chard $11
## 36 3 3 1 $10 PinotNoir $20 PinotGris $5
## 6 3 4 1 $10 Merlot $10 SauvBlanc $5
## 14 3 5 1 $17 CabSav $10 SauvBlanc $11
## 33 3 6 1 $10 CabSav $15 PinotGris $5
## 12 3 7 1 $17 PinotNoir $15 Chard $5
## 7 3 8 1 $24 PinotNoir $10 Chard $8
## 31 3 9 1 $17 Merlot $15 PinotGris $11
## 34 3 10 1 $10 Merlot $20 Chard $8
## 32 3 11 1 $24 Merlot $10 PinotGris $8
## 9 3 12 1 $24 CabSav $15 SauvBlanc $8
## type_beer p_cider type_cider pack_cider
## 35 IPA $10 Tart 6-pack of Cans
## 30 Pilsner $15 Funky 750ml Bottle
## 15 IPA $20 Tart 750ml Bottle
## 8 Stout $15 Fruity 750ml Bottle
## 28 Pilsner $20 Fruity 750ml Bottle
## 3 Stout $10 Funky 6-pack of Cans
## 16 Pilsner $15 Tart 6-pack of Cans
## 20 Stout $20 Tart 750ml Bottle
## 10 Pilsner $10 Fruity 750ml Bottle
## 24 Stout $20 Funky 6-pack of Cans
## 21 IPA $10 Funky 6-pack of Cans
## 25 IPA $15 Fruity 6-pack of Cans
## 26 IPA $20 Fruity 6-pack of Cans
## 19 Pilsner $10 Tart 6-pack of Cans
## 17 Pilsner $15 Tart 6-pack of Cans
## 11 Pilsner $20 Funky 750ml Bottle
## 29 IPA $20 Tart 750ml Bottle
## 1 Stout $15 Fruity 6-pack of Cans
## 27 Pilsner $10 Fruity 750ml Bottle
## 22 Stout $10 Fruity 750ml Bottle
## 4 Stout $20 Funky 6-pack of Cans
## 5 IPA $10 Funky 750ml Bottle
## 18 IPA $15 Funky 750ml Bottle
## 2 Stout $15 Tart 6-pack of Cans
## 13 Stout $20 Tart 750ml Bottle
## 23 Pilsner $20 Funky 6-pack of Cans
## 36 IPA $20 Fruity 6-pack of Cans
## 6 Stout $15 Tart 750ml Bottle
## 14 IPA $15 Fruity 6-pack of Cans
## 33 Pilsner $15 Funky 750ml Bottle
## 12 IPA $10 Tart 750ml Bottle
## 7 IPA $15 Funky 750ml Bottle
## 31 Stout $10 Fruity 750ml Bottle
## 34 Stout $10 Funky 6-pack of Cans
## 32 Pilsner $20 Fruity 6-pack of Cans
## 9 Pilsner $10 Tart 6-pack of Cans
##
## Candidate design:
## A B C D E F G H J K
## 1 1 3 2 3 2 3 2 1 2 2
## 2 2 1 3 1 1 3 2 3 2 2
## 3 2 1 1 3 2 3 1 2 2 1
## 4 2 3 1 2 1 3 3 2 2 2
## 5 3 1 3 2 3 2 1 2 1 2
## 6 1 2 1 3 1 3 2 3 1 3
## 7 3 3 1 1 2 2 2 2 1 3
## 8 3 1 3 2 2 3 2 1 1 1
## 9 3 1 2 3 2 1 1 3 2 3
## 10 1 1 1 1 1 1 1 1 1 1
## 11 1 2 1 3 3 1 3 2 1 2
## 12 2 3 2 1 1 2 1 3 1 3
## 13 3 3 3 3 3 3 3 3 1 3
## 14 2 1 1 3 3 2 2 1 2 3
## 15 2 2 3 3 2 2 3 3 1 1
## 16 2 3 1 2 3 1 2 3 2 1
## 17 3 2 2 1 3 1 2 3 2 2
## 18 2 2 2 2 2 2 2 2 1 2
## 19 1 3 3 2 2 1 1 3 2 2
## 20 1 1 2 2 3 3 3 3 1 1
## 21 1 3 2 3 3 2 1 2 2 1
## 22 3 3 1 1 3 3 1 1 1 2
## 23 2 1 3 1 3 1 3 2 2 3
## 24 3 2 2 1 1 3 3 2 2 1
## 25 1 2 3 1 3 2 2 1 2 1
## 26 3 1 2 3 1 2 3 1 2 2
## 27 2 2 3 3 1 1 1 1 1 2
## 28 2 3 2 1 2 1 3 1 1 1
## 29 1 1 1 1 2 2 3 3 1 2
## 30 3 3 3 3 1 1 2 2 1 1
## 31 2 2 2 2 3 3 1 1 1 3
## 32 3 2 1 2 2 1 3 1 2 3
## 33 1 1 2 2 1 1 2 2 1 3
## 34 1 2 3 1 2 3 1 2 2 3
## 35 3 2 1 2 1 2 1 3 2 1
## 36 1 3 3 2 1 2 3 1 2 3
## class=design, type= oa
##
## Design information:
## number of blocks = 3
## number of questions per block = 12
## number of alternatives per choice set = 1
## number of attributes per alternative = 9From here we can then program this in Qualtrics and we would get something like the following:
Conclusions
We have covered a number of examples of experimental designs applied to a number of research questions. I hope this overview of experimental design provides some insight to the art and science of the process. There is much I didn’t cover in this short lecture, but I encourage your to think deeply whenever you have to use experimental design in your research and know that there are a number of resources available to you to ensure the best possible outcome.