Primary results
Induction task
Analyses of the induction task were logistic regressions unless otherwise specified, predicting prevalence (.01-.99) with participant and test feature as random intercepts. Test feature (“can snap with their toes”, etc.) is technically nested within test feature type (physical, diet, personality), but since each test feature is unique to each test feature type, a model with the nesting term is analytically equivalent to the previous model, so the nesting term was omitted for simplicity of specification.
By test feature
We can look at how prevalence judgments vary by condition and individual test feature.
By test feature type
We can look at how prevalence judgments vary by condition and test feature type (i.e., physical, diet, or personality).
If the chosen clusters capture some systematicity in how people generalize, the physical condition should make the highest prevalence estimates for physical test features, the diet condition for the diet test features, and the personality condition for personality test features. This appears to be true for the physical and personality conditions, but not for the diet condition.
# condition * test feature type
glmm_condition_testfeaturetype <-
glmmTMB(prevalence ~ condition * test_feature_type + (1|participant) + (1|test_feature),
data = data_tidy,
family = beta_family(link = "logit"))
glmm_condition_testfeaturetype %>%
Anova()
## Analysis of Deviance Table (Type II Wald chisquare tests)
##
## Response: prevalence
## Chisq Df Pr(>Chisq)
## condition 11.6643 3 0.008626 **
## test_feature_type 6.9583 2 0.030833 *
## condition:test_feature_type 79.6473 6 0.000000000000004225 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
glmm_condition_testfeaturetype_pairs <- glmm_condition_testfeaturetype %>%
emmeans(~ condition * test_feature_type) %>%
contrast(method = "pairwise") %>%
summary()
glmm_condition_testfeaturetype_pairs
## contrast estimate SE df
## physical physical - diet physical 0.551661 0.147 Inf
## physical physical - personality physical 0.587304 0.147 Inf
## physical physical - heterogeneous physical 0.551367 0.146 Inf
## physical physical - physical diet 0.069805 0.210 Inf
## physical physical - diet diet 0.139959 0.250 Inf
## physical physical - personality diet 0.479164 0.249 Inf
## physical physical - heterogeneous diet 0.518802 0.249 Inf
## physical physical - physical personality -0.245210 0.211 Inf
## physical physical - diet personality -0.008632 0.250 Inf
## physical physical - personality personality -0.215079 0.249 Inf
## physical physical - heterogeneous personality 0.097944 0.249 Inf
## diet physical - personality physical 0.035643 0.148 Inf
## diet physical - heterogeneous physical -0.000294 0.147 Inf
## diet physical - physical diet -0.481856 0.250 Inf
## diet physical - diet diet -0.411702 0.211 Inf
## diet physical - personality diet -0.072496 0.250 Inf
## diet physical - heterogeneous diet -0.032859 0.250 Inf
## diet physical - physical personality -0.796871 0.250 Inf
## diet physical - diet personality -0.560293 0.211 Inf
## diet physical - personality personality -0.766740 0.250 Inf
## diet physical - heterogeneous personality -0.453717 0.250 Inf
## personality physical - heterogeneous physical -0.035937 0.146 Inf
## personality physical - physical diet -0.517499 0.249 Inf
## personality physical - diet diet -0.447345 0.250 Inf
## personality physical - personality diet -0.108139 0.210 Inf
## personality physical - heterogeneous diet -0.068502 0.249 Inf
## personality physical - physical personality -0.832514 0.249 Inf
## personality physical - diet personality -0.595936 0.250 Inf
## personality physical - personality personality -0.802383 0.211 Inf
## personality physical - heterogeneous personality -0.489360 0.249 Inf
## heterogeneous physical - physical diet -0.481562 0.249 Inf
## heterogeneous physical - diet diet -0.411409 0.250 Inf
## heterogeneous physical - personality diet -0.072203 0.249 Inf
## heterogeneous physical - heterogeneous diet -0.032565 0.210 Inf
## heterogeneous physical - physical personality -0.796577 0.249 Inf
## heterogeneous physical - diet personality -0.559999 0.250 Inf
## heterogeneous physical - personality personality -0.766446 0.249 Inf
## heterogeneous physical - heterogeneous personality -0.453423 0.210 Inf
## physical diet - diet diet 0.070153 0.147 Inf
## physical diet - personality diet 0.409359 0.147 Inf
## physical diet - heterogeneous diet 0.448997 0.146 Inf
## physical diet - physical personality -0.315015 0.211 Inf
## physical diet - diet personality -0.078437 0.250 Inf
## physical diet - personality personality -0.284884 0.249 Inf
## physical diet - heterogeneous personality 0.028139 0.249 Inf
## diet diet - personality diet 0.339206 0.148 Inf
## diet diet - heterogeneous diet 0.378844 0.147 Inf
## diet diet - physical personality -0.385169 0.250 Inf
## diet diet - diet personality -0.148591 0.211 Inf
## diet diet - personality personality -0.355038 0.250 Inf
## diet diet - heterogeneous personality -0.042015 0.250 Inf
## personality diet - heterogeneous diet 0.039638 0.147 Inf
## personality diet - physical personality -0.724375 0.250 Inf
## personality diet - diet personality -0.487797 0.250 Inf
## personality diet - personality personality -0.694244 0.211 Inf
## personality diet - heterogeneous personality -0.381221 0.250 Inf
## heterogeneous diet - physical personality -0.764012 0.249 Inf
## heterogeneous diet - diet personality -0.527434 0.250 Inf
## heterogeneous diet - personality personality -0.733881 0.250 Inf
## heterogeneous diet - heterogeneous personality -0.420859 0.211 Inf
## physical personality - diet personality 0.236578 0.148 Inf
## physical personality - personality personality 0.030131 0.147 Inf
## physical personality - heterogeneous personality 0.343154 0.147 Inf
## diet personality - personality personality -0.206447 0.149 Inf
## diet personality - heterogeneous personality 0.106576 0.148 Inf
## personality personality - heterogeneous personality 0.313023 0.147 Inf
## z.ratio p.value
## 3.751 0.0096
## 4.004 0.0036
## 3.779 0.0087
## 0.332 1.0000
## 0.561 1.0000
## 1.921 0.7460
## 2.083 0.6348
## -1.165 0.9915
## -0.035 1.0000
## -0.862 0.9994
## 0.393 1.0000
## 0.241 1.0000
## -0.002 1.0000
## -1.931 0.7397
## -1.955 0.7241
## -0.290 1.0000
## -0.132 1.0000
## -3.191 0.0631
## -2.658 0.2475
## -3.066 0.0905
## -1.817 0.8088
## -0.245 1.0000
## -2.076 0.6401
## -1.790 0.8241
## -0.514 1.0000
## -0.275 1.0000
## -3.337 0.0403
## -2.382 0.4182
## -3.810 0.0077
## -1.962 0.7193
## -1.935 0.7370
## -1.649 0.8909
## -0.290 1.0000
## -0.155 1.0000
## -3.199 0.0617
## -2.242 0.5184
## -3.073 0.0887
## -2.155 0.5827
## 0.476 1.0000
## 2.787 0.1854
## 3.070 0.0896
## -1.496 0.9421
## -0.314 1.0000
## -1.142 0.9928
## 0.113 1.0000
## 2.291 0.4832
## 2.569 0.2974
## -1.542 0.9290
## -0.705 0.9999
## -1.419 0.9601
## -0.168 1.0000
## 0.270 1.0000
## -2.902 0.1399
## -1.949 0.7280
## -3.295 0.0460
## -1.528 0.9332
## -3.065 0.0906
## -2.110 0.6153
## -2.940 0.1269
## -1.998 0.6950
## 1.600 0.9097
## 0.205 1.0000
## 2.340 0.4475
## -1.390 0.9657
## 0.721 0.9999
## 2.126 0.6040
##
## Results are given on the log odds ratio (not the response) scale.
## P value adjustment: tukey method for comparing a family of 12 estimates
Indeed, there is a significant interaction between condition and test feature type in an ANOVA conducted on a logistic regression with condition, test feature type, and their interaction as fixed effects, and with participant and test feature as random intercepts (\(\chi^2\)(6) = 79.65, p < .001). There is also a main effect of condition (\(\chi^2\)(3) = 11.66, p = 0.009) and test feature type (\(\chi^2\)(2) = 6.96, p = 0.031)
By test feature type match
Another way to look at the data is to code responses by whether the test feature htype matched the training condition. If they match (e.g., diet condition responding to a diet test question), we can code that as a match, or if they mismatch (e.g., diet condition responding to a personality test question), we can code that as a mismatch. We can leave the heterogeneous condition as its own category, since it’s a semi-match to everything.
If the chosen clusters capture some systematicity in how people generalize, matches should result in higher prevalence estimates than mismatches. Indeed, that’s what we find.
# condition
glmm_condition_test_match <-
glmmTMB(prevalence ~ condition_test_match + (1|participant) + (1|test_feature),
data = data_tidy,
family = beta_family(link = "logit"))
glmm_condition_test_match %>%
Anova()
## Analysis of Deviance Table (Type II Wald chisquare tests)
##
## Response: prevalence
## Chisq Df Pr(>Chisq)
## condition_test_match 78.898 2 < 0.00000000000000022 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
glmm_condition_test_match_pairs <- glmm_condition_test_match %>%
emmeans(~ condition_test_match) %>%
contrast(method = "pairwise") %>%
summary()
glmm_condition_test_match_pairs
## contrast estimate SE df z.ratio p.value
## match - heterogeneous 0.415 0.1150 Inf 3.593 0.0010
## match - mismatch 0.264 0.0306 Inf 8.632 <0.0001
## heterogeneous - mismatch -0.151 0.1140 Inf -1.324 0.3819
##
## Results are given on the log odds ratio (not the response) scale.
## P value adjustment: tukey method for comparing a family of 3 estimates
Indeed, there is a main effect of whether condition and test variables match (match, hetereogenous, or mismatch) on prevalence, in an ANOVA conducted on a logistic regression with match as a main effect, and with participant and test feature as random intercepts (\(\chi^2\)(2) = 78.9, p < .001).
Pairwise comparisons reveal that the matching condition results in higher prevalence estimates of test features than the heterogeneous condition (z = 3.59, p < .001) or the mismatching conditions (z = 8.63, p < .001)
By cosine similarity
Instead of grouping features into discrete types, we can also look at the distance (cosine similarity) between each individual test feature to the training features presented in each condition, in the multidimensional embedding space.
For each test feature, we can calculate the average distance to training features in each condition, and see if that metric of cosine similarity predicts measures of prevalence.
# average cosine similarity of the test feature, to the training features in that condition
glmm_cosine_similarity_avg <-
glmmTMB(prevalence ~ cosine_similarity_avg + (1|participant) + (1|test_feature),
data = data_tidy,
family = beta_family(link = "logit"))
glmm_cosine_similarity_avg %>%
summary()
Indeed, there is a significant effect of average cosine similarity of the test feature to the various training features in the condition (\(z\) = 7.39, p < .001), such that higher average cosine similarity predicts higher prevalence estimates, in a logistic model with random intercepts per participant and test feature.
We can also focus on maximum cosine similarity, i.e., the distance from the test feature to the closest training feature in a given condition, and see if that metric predicts prevalence judgments.
# max cosine similarity of the test feature, to the closest training features in that condition
glmm_cosine_similarity_max <-
glmmTMB(prevalence ~ cosine_similarity_max + (1|participant) + (1|test_feature),
data = data_tidy,
family = beta_family(link = "logit"))
glmm_cosine_similarity_max %>%
summary()
Indeed, maximum cosine similarity is also a significant predictor of prevalence estimates (\(z\) = 7.06, p < .001), such that higher maximum cosine similarity predicts higher prevalence estimates, in a logistic model with random intercepts per participant and test feature.
glmmTMB(prevalence ~ cosine_similarity_avg + cosine_similarity_max + (1|participant) + (1|test_feature),
data = data_tidy,
family = beta_family(link = "logit")) %>%
summary()
However, when including both average and max cosine similarity as predictors, only average cosine similarity remains a significant predictor of prevalence estimates, suggesting that people are integrating over all training features in the condition, rather than just attending to the most similar training feature.
Overall
We can look at prevalence estimates overall. If the heterogeneous condition leads to the highest overall coherence, we should see the highest prevalence estimates in that condition overall. However, that’s not what we find.
# condition * test feature type
glmm_condition <-
glmmTMB(prevalence ~ condition + (1|participant) + (1|test_feature_type),
data = data_tidy,
family = beta_family(link = "logit"))
glmm_condition %>%
Anova()
## Analysis of Deviance Table (Type II Wald chisquare tests)
##
## Response: prevalence
## Chisq Df Pr(>Chisq)
## condition 12.229 3 0.006638 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
glmm_condition_pairs <- glmm_condition %>%
emmeans("condition") %>%
contrast(method = "pairwise") %>%
summary()
glmm_condition_pairs
## contrast estimate SE df z.ratio p.value
## physical - diet 0.2842 0.132 Inf 2.147 0.1383
## physical - personality 0.3371 0.132 Inf 2.554 0.0520
## physical - heterogeneous 0.4377 0.131 Inf 3.332 0.0048
## diet - personality 0.0529 0.133 Inf 0.398 0.9787
## diet - heterogeneous 0.1534 0.132 Inf 1.160 0.6524
## personality - heterogeneous 0.1006 0.132 Inf 0.762 0.8714
##
## Results are given on the log odds ratio (not the response) scale.
## P value adjustment: tukey method for comparing a family of 4 estimates
There is an effect of condition on prevalence (\(\chi^2\)(3) = 12.23, p = 0.007).
Pairwise comparisons reveal significant differences between the physical condition and most other conditions (physical vs diet: z = 2.15, p = 0.138, physical vs personality: z = 2.55, p = 0.052, physical vs heterogeneous: z = 3.33, p = 0.005).
By test feature, vs model
We can get the model’s predictions and compare those to people’s ratings of prevalence. For now, we get the model’s “kind score” for each test feature, which is a measure of the expected value of the Gaussian function at that location in feature space.