Primary results
Sample representation
As a check that they could retrieve the mean of the sample they observed, participants were asked, “Which picture shows how tall the Zarpies who visited are?” Response options were a Zarpie of height 4, 5, 6, 7, or 8.
Since all participants saw the same sample (4, 5, 6, 6, 7, 8), all participants should provide the same response regardless of condition. This response is expected to be the mean of the sample: 6.
As expected, there was no main effect of population condition on sample representations (in a simple linear regression), since all participants observed the same sample (6 Zarpies: 4, 5, 6, 6, 7, 8).
lm(dv_sample ~ pop,
data = data) %>%
Anova()## Anova Table (Type II tests)
##
## Response: dv_sample
## Sum Sq Df F value Pr(>F)
## pop 3.537 2 1.3035 0.275
## Residuals 180.456 133Strangely, and like the last pilot, participants overall thought the sample was significantly taller than 6, the true mean & mode. Broken down by condition, participants in the short and tall conditions gave responses significantly taller than 6, while participants in the medium condition were trending in that direction (t(45) = 1.73, p = .09).
t.test(data %>%
select(dv_sample),
mu = mean(observed_sample)) # true mean of observed sample = 6##
## One Sample t-test
##
## data: data %>% select(dv_sample)
## t = 4.9212, df = 135, p-value = 0.000002464
## alternative hypothesis: true mean is not equal to 6
## 95 percent confidence interval:
## 6.294666 6.690628
## sample estimates:
## mean of x
## 6.492647t.test(data %>%
filter(pop == "short") %>%
select(dv_sample),
mu = mean(observed_sample)) # true mean of observed sample = 6##
## One Sample t-test
##
## data: data %>% filter(pop == "short") %>% select(dv_sample)
## t = 3.9644, df = 45, p-value = 0.0002606
## alternative hypothesis: true mean is not equal to 6
## 95 percent confidence interval:
## 6.342225 7.049079
## sample estimates:
## mean of x
## 6.695652t.test(data %>%
filter(pop == "med") %>%
select(dv_sample),
mu = mean(observed_sample)) # true mean of observed sample = 6##
## One Sample t-test
##
## data: data %>% filter(pop == "med") %>% select(dv_sample)
## t = 1.7344, df = 45, p-value = 0.08969
## alternative hypothesis: true mean is not equal to 6
## 95 percent confidence interval:
## 5.950921 6.657775
## sample estimates:
## mean of x
## 6.304348t.test(data %>%
filter(pop == "tall") %>%
select(dv_sample),
mu = mean(observed_sample)) # true mean of observed sample = 6##
## One Sample t-test
##
## data: data %>% filter(pop == "tall") %>% select(dv_sample)
## t = 2.8522, df = 43, p-value = 0.006648
## alternative hypothesis: true mean is not equal to 6
## 95 percent confidence interval:
## 6.139812 6.814734
## sample estimates:
## mean of x
## 6.477273When asked why they selected the picture they did, participants’ responses generally described:
- selecting the average, or mode:
- “because that appears to be the average height, out of all the Harpies” (6)
- “This Zarpie still represents the average height of all the Zarpies that boarded the boat and visited.” (6)
- “There were lots of different sizes but this one is the most common” (7)
- selecting the tallest extreme, e.g.:
- “It was the tallest zarpie so felt like the best choice.” (8)
- “Chose the one that is the tallest” (8)
- “Because I was not allowed to select all of them and”how tall” would typically be asking for maximum height if you could only choose one” (8)
- “I don’t understand the question, all of those zarpies came and they were all different heights. I chose the tallest one because it had to crouch which made it mildly more memorable.” (8)
Looking at the participants who chose 6 and 8 (the mode responses):
Participants who chose 6 generally explained their choice as choosing the “average” Zarpie.
Participants who chose 8 generally explained their choice as choosing the “tallest” Zarpie.
Population representation
As a check for their representation of the population, participants were asked: “Which picture shows how tall Zarpies are on Zarpie island?” Response options were a Zarpie of height 4, 5, 6, 7, or 8.
If this question is a valid measure of participants’ representation of the average height of Zarpies, and participants remember how tall Zarpies are in the boarding scene and use that as their representation of Zarpies on Zarpie island, the expected response in each condition is:
- pop short: (4, 5, 6, 6, 7, 8) –> population mean = 6
- pop med: (4, 5, 6, 6, 7, 8, 8, 8, 9, 9, 10) –> population mean = 7.27
- pop tall: (4, 5, 6, 6, 7, 8, 8, 8, 8, 8, 8, 9, 10, 10, 11, 12) –> population mean = 8
lm(dv_pop ~ pop,
data = data) %>%
Anova()## Anova Table (Type II tests)
##
## Response: dv_pop
## Sum Sq Df F value Pr(>F)
## pop 1.6 2 0.6963 0.5002
## Residuals 152.8 133When asked why they selected the picture they did, participants’ responses generally described: * selecting the average, e.g.: + “It’s the average of the height of all the zarpies I saw” + “There were varying heights to the Zarpies so I picked one that was a middle or medium height.” + “I chose option 3 because there were varying sizes of Zarpies on the island, so this would be about the average.” + “The Zarpies are all different heights so I chose one that seems to be the average of all the heights shown.”
- selecting the tallest extreme, e.g.:
- “He represents tallest”
- “Shows that all the others are shorter than him/her.”
- “This is the tallest zarpie”
- “It shows the tallest a zarpie can be.”
Participants gave different explanations for their responses by condition (\(p\) = .004, Fisher’s exact).
##
## Fisher's Exact Test for Count Data
##
## data: .
## p-value = 0.003973
## alternative hypothesis: two.sidedLooking at the participants who chose 6 and 8 (the mode responses):
Participants who chose 6 generally explained their choice as choosing the “average” Zarpie.
Participants who chose 8 generally explained their choice as choosing the “tallest” Zarpie.
Explicit comparison
Participants were explicitly asked to compare the population to the sample: “Do you think the Zarpies on Zarpie island are shorter, about the same, or taller than the Zarpies who visited?”
| shorter | about the same | taller | |
|---|---|---|---|
| short | 4% | 91% | 4% |
| med | 9% | 28% | 63% |
| tall | 14% | 27% | 59% |
Should we be worried that participants in the medium and tall conditions were not at ceiling for reporting that the Zarpies on Zarpie island are “taller” than the Zarpies who visited (59-63%)?
- They might be “about the same”, in the sense that they are all Zarpies at the end of the day?
##
## Fisher's Exact Test for Count Data
##
## data: .
## p-value = 0.000000000003984
## alternative hypothesis: two.sided##
## Fisher's Exact Test for Count Data
##
## data: .
## p-value = 0.00000000009606
## alternative hypothesis: two.sided##
## Fisher's Exact Test for Count Data
##
## data: .
## p-value = 0.0000000001825
## alternative hypothesis: two.sided##
## Fisher's Exact Test for Count Data
##
## data: .
## p-value = 0.7802
## alternative hypothesis: two.sidedParticipants’ explicit comparison responses differed by condition (\(p\) < .001, Fisher’s exact). Specifically, responses in the short population condition differed from responses in the medium and tall conditions (\(p\)s < .001, Fisher’s exact); responses in the medium and tall conditions did not differ from each other (\(p\) = .78, Fisher’s exact).
These results suggest that participants are sensitive to the fact that the population must be taller if taller Zarpies got cut-off, as in the medium and tall conditions, but the difference between the medium and tall conditions may be too subtle for them to pick up on.
Looking at participants’ explanations for their responses, many participants reasoned about those remaining on the island, rather than Zarpies as a whole, which suggests a measurement issue with what the question is eliciting.
