Overview

Project goals

The goals of this project are to establish:

  1. if children and adults generalize from a sample to a social group from a sample that is, unbeknownst to them, structurally skewed, resulting in inaccurate beliefs about the group

  2. if children and adults can adjust their generalization from a sample to a social group to account for the fact that the sample was skewed by a structural process

This study focuses on question (2).

Previously on…

Previously in pilot 3, we found a paradigm and measures that successfully elicit the average representation of Zarpies’ height.

  • When given full access to some population information, adults were sensitive to the increasing heights of the population in each successive condition in their population representation, suggesting that measure captures sensitivity to the population information. However, their reported averages still fell short of the true average of the population, suggesting adults knew the population was taller and taller in each condition, but not by how much.

Study goals

The primary goal of this pilot was to see if adults, given the same sample but varying boat heights, make different population inferences.

Changes from last pilot:

  • go back to varying boat height and boarding outcomes (some Zarpies get turned away from short boat, or all Zarpies board tall boat)

  • obscure population information (cannot see heights of Zarpies boarding) to force participants to infer the population height

  • drop the multiple-select “identify the things mentioned in the task” check question, instead exclude those who failed memory check or comprehension check

Results

  • Promisingly, adults were sensitive to the structural information, making taller inferences about the population in the skewed boat height 6 condition, compared to the not skewed boat height 10 condition (see population inference).

    • Adults were also more likely to explicitly report that the population is taller than the sample, and less likely to report that they are shorter than the sample, in the skewed boat height 6 condition compared to the not skewed boat height 10 condition (see explicit comparison).

    • Adults in the skewed boat height 6 condition made inferences about the population that were significantly taller than their own sample responses, as well as significantly taller than the true sample mean (see sample vs population).

  • However, the effect is relatively weak and fragile.

    • Adults in the skewed boat height 6 condition did not make significantly taller inferences about the population than their own sample responses, when including adults who failed the memory check or comprehension check (see sample vs population).

    • The main effect of condition on population inferences was tempered by a significant order effect and marginal order by condition interaction. In particular, the condition effect is only present when participants were asked about the population after being asked about the sample, rather than first being asked about the population (see order effects).

      • Speculative explanation: asking adults about the sample first helps them reason about how the population might look different from the sample, but asking adults about the population right off the bat makes them just generalize directly off the sample and/or makes them forget about the structural skew.

      • Note: there were no order effects on the explicit comparison question, which was always asked after both population inference and sample representation questions.

    • Pairwise forced-choice measures were a complete wash, i.e., everything at chance across conditions, likely because they were our coarsest measure (see pairwise forced-choice).

Methods

Participants

Data was collected from 100 adults recruited via Prolific on Mon 6/2/2025. Participants were required to be in the United States, fluent in English, and have not participated in the earlier pilot of this study.

Participants were paid $2.75 for an estimated 10-12 minute task. In fact, the study generally took about 14 minutes for participants.

The final sample included 88 adults (n = 44-44 in each of the 2 conditions).

condition (boat height) participants
not skewed (10) 44
skewed (6) 44

Exclusion criteria

12 participants (12.0% of all participants) were excluded for meeting at least 1 of the following exclusion criteria:

  • failing the sound check (n = 3 participants)

  • failing to select the correct task description (i.e., did not select “Watching videos about fictional people from an island”) (n = 2 participants)

  • failing the memory check (n = 5 participants)

  • failing the comprehension check (n = 6 participants)

Including participants who failed the memory check or comprehension check (but who passed the other exclusion criteria) results in largely the same effects as described below, other than the participants in the skewed boat height (6) condition, no longer making population inferences that are significantly taller than their sample inferences (see sample vs population).

Memory check

Participants overwhelmingly passed the memory check for the Quaffa boarding sequence, i.e., “no”, not all the Quaffas made it onto the boat.

Participants who made incorrect responses were excluded.

Comprehension check

Participants overwhelmingly passed the comprehension check for the Zarpie boarding sequence. Note the correct answer to this question depends on condition:

  • In the boat height 6 condition, the correct answer is “no”, not all of the Zarpies made it onto the boat.

  • In the boat height 10 condition, the correct answer is “yes”, all of the Zarpies made it onto the boat.

Participants who made incorrect responses or non-responses were excluded. Note that there are some non-responses (NAs), because I forgot to require a response on this question in Qualtrics.

Demographics

age
mean sd n
39.70 13.94 88
  • The sample was largely young and middle-aged.
gender n prop
Female 45 51.1%
Male 42 47.7%
Prefer not to specify 1 1.1%
  • The sample was diverse in terms of gender identities in the US.
race n prop
White, Caucasian, or European American 45 51.1%
Black or African American 19 21.6%
Hispanic or Latino/a 8 9.1%
White, Caucasian, or European American,Hispanic or Latino/a 3 3.4%
Hispanic or Latino/a,Black or African American 2 2.3%
Middle Eastern or North African 2 2.3%
White, Caucasian, or European American,Black or African American 2 2.3%
East Asian 1 1.1%
East Asian,Native Hawaiian or other Pacific Islander 1 1.1%
Hispanic or Latino/a,South or Southeast Asian 1 1.1%
Native American, American Indian, or Alaska Native 1 1.1%
Prefer not to specify 1 1.1%
South or Southeast Asian 1 1.1%
White, Caucasian, or European American,Native American, American Indian, or Alaska Native 1 1.1%
  • The sample was also racially diverse.
education n prop
High school/GED 9 10.2%
Some college 25 28.4%
Bachelor's (B.A., B.S.) 30 34.1%
Master's (M.A., M.S.) 20 22.7%
Doctoral (Ph.D., J.D., M.D.) 3 3.4%
Prefer not to specify 1 1.1%
  • The sample was mostly college-educated.

Procedure

This study was administered as a Qualtrics survey, and approved by the NYU IRB (IRB-FY2024-9169).

After providing their consent, participants completed a captcha and sound check, and were asked to watch videos sound on. Participants then watched the following videos in order:

  1. In the prior setting and familiarization phase, participants saw a photorealistic picture of 5 human adults and then another picture of a different 5 adults appear on screen against a grid. These adults were all 10 gridline units tall.

  2. In the boat training phase, participants were randomly assigned to see a boat that was either 6 or 10 units tall.

    • The boat height was specified to be accidental (“When the boat builders were building the boat, they started building the boat from the bottom, but ran out of the special wood they needed for the boat! So the boat ended up being this tall. It might be hard for anyone who is taller than the boat to get on the boat.”), to avoid any justificatory reasoning about the height of the boat being informative about the height of Zarpies or vice versa.

    • To communicate how the boat functions to exclude those shorter than the boat, participants then watched a parade of 20 fictional animals (Quaffas, taken from Foster-Hanson et al., 2019) attempt to board the boat, one at a time, from shortest to tallest.

    • The height of animals were scaled to the height of the boat, such that 10 animals were always shorter than the boat (these animals boarded successfully) and 10 animals were always taller than the boat (all but one were unable to board; the third quaffa successfully boards by bending its head).

    • Participants were asked a memory check: “Did all of the animals board the boat?” (yes/no), and received an affirmation (if they said “no”) or correction (if they said “yes”).

  3. In the boat boarding phase, participants learned that Zarpies live on Zarpie island, and saw an island with many Zarpies overhead. Participants learned that all the grownup Zarpies’ names were put into a hat, and some of their names “were drawn out of a hat to try and visit us”. Participants saw then saw a parade of Zarpies attempt to board the boat to visit us, one at a time, behind a curtain that obscured their heights. Participants were told that they were all grown-up Zarpies.

    • In the boat height 6 condition, 20 Zarpies attempt to board, 6 of whom successfully make it on (6 out of 16 successful = 30% successful). Of the 6 who make it on, 2 had to stoop to board.
Boarding for boat height 6 condition.
Boarding for boat height 6 condition.
* In the boat height 10 condition, 6 Zarpies attempt to board, all of whom successfully make it on without stooping. 
Boarding for boat height 10 condition.
Boarding for boat height 10 condition.
  1. After the boat boarding phase, participants were asked a comprehension check: “Did all of the Zarpies board the boat?” (yes/no), and received either an affirmation (if they said “no” in boat height 6, or “yes” in boat height 10) or correction (if they said “yes” in boat height 6, or “no” in boat height 10).

  2. In the sample observation phase, all participants saw the Zarpies who successfully boarded the boat get off the boat to visit us. The Zarpies got off one at a time, and each waved/descrunched if relevant. The height of this observed sample (4, 5, 6, 6, 7, 8) was held constant across conditions.

    • To emphasize the height of the Zarpies relative to the boat, participants watched Zarpies deboard the boat, wave, reboard the boat (with any Zarpies taller than the boat stooping down again to board again), and deboard again (with any Zarpies taller than the boat straightening up again).
Observed sample in boat height 10 condition.
Observed sample in boat height 10 condition.

Participants were asked the following DVs in the following order:

  1. Participants were asked the average height of the Zarpies who visited (Sample representation) and the average height of Zarpies on Zarpie island (Population inference), in counterbalanced order.

  2. Participants were then asked an explicit comparison question asking them to compare the heights of Zarpies on Zarpie island to that of Zarpies who visited: shorter, about the same, or taller (see explicit comparison).

  3. Participants were then shown pairs of Zarpies (6v7, 6v8, 7v8) and told one Zarpie is from Zarpie island and one is a Zarpie who visited, and asked to guess which one is the Zarpie on Zarpie island. Pairs were presented in randomized order. (see pairwise forced-choice)

Finally, participants were asked for any problems or confusion they had, what they thought the task was about (see [Participant feedback]), and demographic information.

Primary results

Sample representation

As a check that they could retrieve the mean of the sample they observed, participants were asked, “Which picture shows the average height of the Zarpies who visited?”, and had to choose between a Zarpie of height 4, 5, 6, 7, or 8.

Sample representation question for boat height 6 condition, with the response options in red boxes.
Sample representation question for boat height 6 condition, with the response options in red boxes.

Since all participants saw the same sample, participants should provide the same response, regardless of condition. The sample was (4, 5, 6, 6, 7, 8), so the the response is expected to be the mean of the sample, 6 (indicated with a red line below).

As expected, there was no main effect of condition (boat height) on sample representations (t = 0.23, p = 0.817), since all participants observed the same sample.

Participants on average reported taller heights than the true mean of the sample, 6 (t(87) = 2.1, p = 0.038).

Population inference

To assess how tall participants thought Zarpies in general are, participants were asked: “Which picture shows the average height of Zarpies on Zarpie island?” Response options were a Zarpie of height 4, 5, 6, 7, or 8.

Population question for boat height 6 condition. If participants adjust their inferences about the population based on biases in the sampling process:

  • Participants in the unrestricted (boat height 10) condition should infer that the population is like the sample, since everyone was able to board, so they should respond with something similar to sample mean, i.e., 6.
  • Participants in the restricted (boat height 6) condition should infer the population is taller than the sample, since Zarpies taller than the boat were unable to board, so they should report significantly taller height than those in the unrestricted condition.

Participants thought the population was taller in the skewed boat height 6 condition than the not skewed boat height 10 condition (partial \(\eta^2\) = 0.06, t = -2.43, p = 0.017).

Power analysis: \(f^2 = t^2/df = (-2.435)^2/86 = 0.0689\), which is a small effect size. Using GPower on a linear multiple regression, fixed model, R2 deviation from 0, looking for 95% power, GPower suggests n = 191 total sample size.

In the skewed boat height 6 condition, participants’ population inferences were significantly taller than the sample mean of 6 (t(43) = 2.64, p = 0.011), and taller than their sample representations (see sample vs population).

In the not skewed boat height 10 condition, participants’ population inferences were not different from the sample mean of 6 (t(43) = -0.27, p = 0.785) nor from their sample representations (see sample vs population).

Explicit comparison

Participants were explicitly asked to compare the population to the sample: “Do you think the Zarpies on Zarpie island are shorter, about the same, or taller than the Zarpies who visited?”

boatheight_label shorter about the same taller
not skewed (10) 9% 86% 5%
skewed (6) 5% 57% 39%

Participants’ explicit comparisons were significantly different between the two conditions (p < .001, Fisher’s exact).

## 
## Call:
## glm(formula = dv_comp_taller ~ boatheight, family = binomial, 
##     data = .)
## 
## Coefficients:
##              Estimate Std. Error z value Pr(>|z|)   
## (Intercept)   -0.4626     0.3096  -1.494  0.13512   
## boatheight10  -2.5819     0.7871  -3.280  0.00104 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 91.816  on 87  degrees of freedom
## Residual deviance: 74.976  on 86  degrees of freedom
## AIC: 78.976
## 
## Number of Fisher Scoring iterations: 5
## 
## Call:
## glm(formula = dv_comp_same ~ boatheight, family = binomial, data = .)
## 
## Coefficients:
##              Estimate Std. Error z value Pr(>|z|)   
## (Intercept)    0.2744     0.3044   0.902  0.36722   
## boatheight10   1.5714     0.5344   2.940  0.00328 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 105.033  on 87  degrees of freedom
## Residual deviance:  95.227  on 86  degrees of freedom
## AIC: 99.227
## 
## Number of Fisher Scoring iterations: 4
## 
## Call:
## glm(formula = dv_comp_shorter ~ boatheight, family = binomial, 
##     data = .)
## 
## Coefficients:
##              Estimate Std. Error z value  Pr(>|z|)    
## (Intercept)   -3.0445     0.7237  -4.207 0.0000259 ***
## boatheight10   0.7419     0.8937   0.830     0.406    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 43.808  on 87  degrees of freedom
## Residual deviance: 43.080  on 86  degrees of freedom
## AIC: 47.08
## 
## Number of Fisher Scoring iterations: 5

When comparing the height of Zarpies on Zarpie island to those who visited, participants in the skewed boat height 6 condition were significantly more likely to say that they are “taller” (z = -3.33, p < .001), and significantly less likely to say that they are “about the same” (z = 3.18, p = .0015), compared to those in the not skewed boat height 10 condition. Responses that they are “shorter” were rare and did not differ across conditions (p > .60).

Pairwise forced-choice

Participants saw pairs of Zarpies of different heights, and were told one is a Zarpie on Zarpie island and the other one a Zarpie who visited. Participants asked to guess which was the Zarpie on Zarpie island.

The pairs tested were 6v8, 7v8, and 6v7, in randomized order.

6v8

6v8 forced choice question for boat height 6.
6v8 forced choice question for boat height 6.

Participants’ choices did not differ between conditions (z = -0.72, p = .47).

## 
## Call:
## glm(formula = dv_fc_6v8 ~ boatheight, family = binomial, data = data)
## 
## Coefficients:
##              Estimate Std. Error z value Pr(>|z|)
## (Intercept)    0.1823     0.3028   0.602    0.547
## boatheight10  -0.1823     0.4273  -0.427    0.670
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 121.81  on 87  degrees of freedom
## Residual deviance: 121.63  on 86  degrees of freedom
## AIC: 125.63
## 
## Number of Fisher Scoring iterations: 3

Participants’ choices were not different from chance in either condition (ps > .57).

## 
##  One Sample t-test
## 
## data:  data %>% filter(boatheight == "6") %>% pull(dv_fc_6v8) %>% as.character() %>% as.numeric()
## t = 0.59861, df = 43, p-value = 0.5526
## alternative hypothesis: true mean is not equal to 7
## 95 percent confidence interval:
##  6.784640 7.397178
## sample estimates:
## mean of x 
##  7.090909
## 
##  One Sample t-test
## 
## data:  data %>% filter(boatheight == "10") %>% pull(dv_fc_6v8) %>% as.character() %>% as.numeric()
## t = 0, df = 43, p-value = 1
## alternative hypothesis: true mean is not equal to 7
## 95 percent confidence interval:
##  6.692457 7.307543
## sample estimates:
## mean of x 
##         7

6v7

6v7 forced choice question for boat height 6.
6v7 forced choice question for boat height 6.

Participants’ choices did not differ between conditions (z = -0.50, p = .62).

## 
## Call:
## glm(formula = dv_fc_6v7 ~ boatheight, family = binomial, data = data)
## 
## Coefficients:
##              Estimate Std. Error z value Pr(>|z|)
## (Intercept)    0.4626     0.3096   1.494    0.135
## boatheight10  -0.3717     0.4324  -0.860    0.390
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 120.35  on 87  degrees of freedom
## Residual deviance: 119.61  on 86  degrees of freedom
## AIC: 123.61
## 
## Number of Fisher Scoring iterations: 4

Participants’ choices were not different from chance in either condition (ps > .15).

## 
##  One Sample t-test
## 
## data:  data %>% filter(boatheight == "6") %>% pull(dv_fc_6v7) %>% as.character() %>% as.numeric()
## t = 1.5304, df = 43, p-value = 0.1332
## alternative hypothesis: true mean is not equal to 6.5
## 95 percent confidence interval:
##  6.463889 6.763384
## sample estimates:
## mean of x 
##  6.613636
## 
##  One Sample t-test
## 
## data:  data %>% filter(boatheight == "10") %>% pull(dv_fc_6v7) %>% as.character() %>% as.numeric()
## t = 0.29837, df = 43, p-value = 0.7669
## alternative hypothesis: true mean is not equal to 6.5
## 95 percent confidence interval:
##  6.369115 6.676340
## sample estimates:
## mean of x 
##  6.522727

7v8

7v8 forced choice question for boat height 6.
7v8 forced choice question for boat height 6.

Participants’ choices did not differ between conditions (z = 0.30, p = .77).

## 
## Call:
## glm(formula = dv_fc_7v8 ~ boatheight, family = binomial, data = data)
## 
## Coefficients:
##              Estimate Std. Error z value Pr(>|z|)
## (Intercept)   -0.3677     0.3066  -1.199     0.23
## boatheight10   0.2768     0.4302   0.643     0.52
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 120.86  on 87  degrees of freedom
## Residual deviance: 120.44  on 86  degrees of freedom
## AIC: 124.44
## 
## Number of Fisher Scoring iterations: 4

Participants’ choices were not different from chance in either condition (ps > .25).

## 
##  One Sample t-test
## 
## data:  data %>% filter(boatheight == "6") %>% pull(dv_fc_7v8) %>% as.character() %>% as.numeric()
## t = -1.2125, df = 43, p-value = 0.232
## alternative hypothesis: true mean is not equal to 7.5
## 95 percent confidence interval:
##  7.257883 7.560299
## sample estimates:
## mean of x 
##  7.409091
## 
##  One Sample t-test
## 
## data:  data %>% filter(boatheight == "10") %>% pull(dv_fc_7v8) %>% as.character() %>% as.numeric()
## t = -0.29837, df = 43, p-value = 0.7669
## alternative hypothesis: true mean is not equal to 7.5
## 95 percent confidence interval:
##  7.323660 7.630885
## sample estimates:
## mean of x 
##  7.477273

Secondary results

Sample vs population

As an implicit comparison, we can compare each participant’s sample representation and population inference to each other.

As expected, participants in the unrestricted (boat height 10) condition did not give different responses to sample and population questions (t(43) = 1.63, p = 0.11). This is expected since in the unrestricted boat height 10 condition, the sample and the population are identical.

## 
##  Paired t-test
## 
## data:  data %>% filter(boatheight == "6") %>% pull(dv_sample) and data %>% filter(boatheight == "6") %>% pull(dv_pop)
## t = -2.3023, df = 43, p-value = 0.02622
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
##  -0.55425882 -0.03665027
## sample estimates:
## mean difference 
##      -0.2954545

Participants in the restricted (boat height 6) condition gave significantly taller responses to the population question than to the sample question (\(t\)(43) = -2.30, \(p\) = .026).

However, this effect disappears when including those participants who failed the memory check or comprehension check (\(t\)(47) = -1.61, \(p\) = .11).

## 
## Call:
## lm(formula = response ~ boatheight * dv, data = data_tidy)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.38636 -0.11364 -0.09091  0.02273  1.90909 
## 
## Coefficients:
##                       Estimate Std. Error t value            Pr(>|t|)    
## (Intercept)            6.09091    0.09718  62.674 <0.0000000000000002 ***
## boatheight10           0.02273    0.13744   0.165              0.8689    
## dvdv_pop               0.29545    0.13744   2.150              0.0330 *  
## boatheight10:dvdv_pop -0.43182    0.19437  -2.222              0.0276 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.6446 on 172 degrees of freedom
## Multiple R-squared:  0.05264,    Adjusted R-squared:  0.03612 
## F-statistic: 3.186 on 3 and 172 DF,  p-value: 0.02524

Were participants in the restricted boat height 6 condition significantly more likely than participants in the unrestricted boat height 10 condition to give taller responses to the population than sample questions? There is a marginal interaction between boat height condition (6 vs 10) and dv (sample vs population) (t = -2.22, p = .028).

This interaction becomes marginal when including participants who failed memory check or comprehension check (t = 1.78, p = .076).

Order effects

Participants saw the two DVs in counterbalanced order:

  • pop_sample = population DV first, then sample DV
  • sample_pop = sample DV first, then population DV

There was an effect of DV order on population responses, but not on sample responses.

Sample representation order effects

There is no order effect on sample representation.

## Anova Table (Type II tests)
## 
## Response: dv_sample
##                        Sum Sq Df F value Pr(>F)
## boatheight             0.0120  1  0.0559 0.8137
## cb_dvorder             0.0171  1  0.0798 0.7783
## boatheight:cb_dvorder  0.0172  1  0.0800 0.7780
## Residuals             18.0339 84

Population inference order effects

In a linear model with main effects of condition, DV order, and their interaction on population inferences, there was no effect of condition, a main effect of DV order, and a marginal interaction between condition and DV order.

Participants gave higher population estimates after they gave sample estimates, then when they were asked before the sample, as indicated by a main effect of DV order on population estimates (t = 2.14, p = .036).

There is also a marginal interaction of order and condition (t = -1.64, p = .10), suggesting the effect may only emerge when participants are asked about the sample then about the population, rather than just about the population right off the bat.

## 
## Call:
## lm(formula = dv_pop ~ boatheight * cb_dvorder, data = data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.6364 -0.1364  0.0000  0.1235  1.8636 
## 
## Coefficients:
##                                   Estimate Std. Error t value
## (Intercept)                         6.1364     0.1655  37.070
## boatheight10                       -0.1364     0.2369  -0.576
## cb_dvordersample_pop                0.5000     0.2341   2.136
## boatheight10:cb_dvordersample_pop  -0.5435     0.3312  -1.641
##                                              Pr(>|t|)    
## (Intercept)                       <0.0000000000000002 ***
## boatheight10                                   0.5664    
## cb_dvordersample_pop                           0.0356 *  
## boatheight10:cb_dvordersample_pop              0.1046    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.7764 on 84 degrees of freedom
## Multiple R-squared:  0.113,  Adjusted R-squared:  0.08134 
## F-statistic: 3.568 on 3 and 84 DF,  p-value: 0.01746

Indeed, the effect of boat height condition on population inferences is only found when population inferences are made after the sample question (t = -2.93, p = .0054), and not when it is made before (t = -0.58, p = .58).

## 
## Call:
## lm(formula = dv_pop ~ boatheight, data = data %>% filter(cb_dvorder == 
##     "sample_pop"))
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -2.63636 -0.63636  0.04348  0.36364  1.36364 
## 
## Coefficients:
##              Estimate Std. Error t value             Pr(>|t|)    
## (Intercept)    6.6364     0.1659  39.994 < 0.0000000000000002 ***
## boatheight10  -0.6798     0.2321  -2.929              0.00542 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.7783 on 43 degrees of freedom
## Multiple R-squared:  0.1663, Adjusted R-squared:  0.1469 
## F-statistic: 8.579 on 1 and 43 DF,  p-value: 0.00542
## 
## Call:
## lm(formula = dv_pop ~ boatheight, data = data %>% filter(cb_dvorder == 
##     "pop_sample"))
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.1364 -0.1364  0.0000  0.0000  1.8636 
## 
## Coefficients:
##              Estimate Std. Error t value            Pr(>|t|)    
## (Intercept)    6.1364     0.1651  37.164 <0.0000000000000002 ***
## boatheight10  -0.1364     0.2363  -0.577               0.567    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.7745 on 41 degrees of freedom
## Multiple R-squared:  0.008059,   Adjusted R-squared:  -0.01613 
## F-statistic: 0.3331 on 1 and 41 DF,  p-value: 0.567

Explicit comparison order effects

There is no effect of the order of sample and population questions (\(\chi^2\) = 0.69, p = .71) or interaction with condition (\(\chi^2\) = 3.61, p = .16) on the explicit comparison question, which was asked after both of them.

## # weights:  15 (8 variable)
## initial  value 96.677881 
## iter  10 value 56.085756
## iter  20 value 55.682399
## iter  30 value 55.681605
## final  value 55.676355 
## converged
## Call:
## multinom(formula = dv_comp ~ boatheight * cb_dvorder, data = data)
## 
## Coefficients:
##                (Intercept) boatheight10 cb_dvordersample_pop
## about the same    1.871939     1.072024             8.424506
## taller            1.252935    -1.253610             8.861231
##                boatheight10:cb_dvordersample_pop
## about the same                         -9.522510
## taller                                 -9.958811
## 
## Std. Errors:
##                (Intercept) boatheight10 cb_dvordersample_pop
## about the same   0.7595991     1.276386             49.69587
## taller           0.8018227     1.625629             49.69670
##                boatheight10:cb_dvordersample_pop
## about the same                          49.71034
## taller                                  49.73022
## 
## Residual Deviance: 111.3527 
## AIC: 127.3527
## Analysis of Deviance Table (Type II tests)
## 
## Response: dv_comp
##                       LR Chisq Df Pr(>Chisq)    
## boatheight             17.1776  2  0.0001862 ***
## cb_dvorder              0.6894  2  0.7084233    
## boatheight:cb_dvorder   3.6101  2  0.1644649    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Session info

## R version 4.4.2 (2024-10-31)
## Platform: aarch64-apple-darwin20
## Running under: macOS Sequoia 15.5
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## time zone: America/New_York
## tzcode source: internal
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] effectsize_1.0.0 emmeans_1.10.4   nnet_7.3-19      lmerTest_3.1-3  
##  [5] lme4_1.1-35.5    Matrix_1.7-1     car_3.1-3        carData_3.0-5   
##  [9] ggtext_0.1.2     lubridate_1.9.3  forcats_1.0.0    stringr_1.5.1   
## [13] dplyr_1.1.4      purrr_1.0.2      readr_2.1.5      tidyr_1.3.1     
## [17] tibble_3.2.1     ggplot2_3.5.1    tidyverse_2.0.0  gt_0.11.1       
## [21] scales_1.3.0     janitor_2.2.0    here_1.0.1      
## 
## loaded via a namespace (and not attached):
##  [1] gridExtra_2.3       sandwich_3.1-1      rlang_1.1.4        
##  [4] magrittr_2.0.3      multcomp_1.4-26     snakecase_0.11.1   
##  [7] compiler_4.4.2      systemfonts_1.1.0   vctrs_0.6.5        
## [10] pkgconfig_2.0.3     crayon_1.5.3        fastmap_1.2.0      
## [13] backports_1.5.0     labeling_0.4.3      rmarkdown_2.29     
## [16] markdown_1.13       tzdb_0.4.0          nloptr_2.1.1       
## [19] ragg_1.3.2          bit_4.5.0.1         xfun_0.49          
## [22] cachem_1.1.0        jsonlite_1.8.9      parallel_4.4.2     
## [25] cluster_2.1.6       R6_2.5.1            bslib_0.8.0        
## [28] stringi_1.8.4       boot_1.3-31         rpart_4.1.23       
## [31] jquerylib_0.1.4     numDeriv_2016.8-1.1 estimability_1.5.1 
## [34] Rcpp_1.0.13         knitr_1.49          zoo_1.8-12         
## [37] base64enc_0.1-3     parameters_0.24.0   splines_4.4.2      
## [40] timechange_0.3.0    tidyselect_1.2.1    rstudioapi_0.17.1  
## [43] abind_1.4-8         yaml_2.3.10         codetools_0.2-20   
## [46] lattice_0.22-6      withr_3.0.2         bayestestR_0.15.0  
## [49] coda_0.19-4.1       evaluate_1.0.1      foreign_0.8-87     
## [52] survival_3.7-0      xml2_1.3.6          pillar_1.10.0      
## [55] checkmate_2.3.2     insight_1.0.0       generics_0.1.3     
## [58] vroom_1.6.5         rprojroot_2.0.4     hms_1.1.3          
## [61] commonmark_1.9.2    munsell_0.5.1       minqa_1.2.8        
## [64] glue_1.8.0          Hmisc_5.1-3         tools_4.4.2        
## [67] data.table_1.15.4   mvtnorm_1.3-1       grid_4.4.2         
## [70] datawizard_0.13.0   colorspace_2.1-1    nlme_3.1-166       
## [73] htmlTable_2.4.3     Formula_1.2-5       cli_3.6.3          
## [76] textshaping_0.4.0   ggthemes_5.1.0      viridisLite_0.4.2  
## [79] gtable_0.3.5        sass_0.4.9          digest_0.6.37      
## [82] TH.data_1.1-2       htmlwidgets_1.6.4   farver_2.1.2       
## [85] htmltools_0.5.8.1   lifecycle_1.0.4     gridtext_0.1.5     
## [88] bit64_4.5.2         MASS_7.3-61