This document was created from an R markdown file. The respository for the project can be found here: https://github.com/mllewis/keb_2019_reanalysis.

1 Card Sorting Task

1.1 Basic Clusterings

Dendrograms created based on a hierarchical cluster analysis of human judgements of similarity and language-based estimates of similarity.

1.1.1 Shape

1.1.2 Texture

1.1.3 Color

1.2 Entanglement Comparisons

Entanglement is a measure of how well the labels of two dendrograms are aligned. Entanglement values range from 0 (fully aligned labels) to 1 (fully mismatched labels). Entanglement is computed by numbering the labels (1 to the total number of labels) of each tree, and then computing the L-norm distance between these two vectors.

Below, we show pairwise comparisons of human judgement-based (blind and sigted participants) and language-based dendrograms in so-called tanglegrams, after using the untangle() method from the R package dendextend to minimize the amount of entanglement, i.e. to optimize the alignment of the labels from the two dendrograms without altering the underlying cluster structure. We also plot the minimum entanglement values found for each pairwise comparison (lower equals better alignment of the labels).

1.2.1 Shape

1.2.2 Texture

1.2.3 Color

1.2.4 Entanglement Values

1.3 Indices of Similarity between Clusterings

We also computed two indices of the similarity between the clusterings derived from human (blind and sighted participant data) and language-based similarity ratings for color, shape and texture: the Fowlkes-Mallows Index and the adjusted Rand index.

1.3.1 FM-Index (z-scored)

The Fowlkes-Mallows Index (FM-Index) is computed by comparing the two hierarchical clustering trees cut at a specific level k (i.e. split into k different clusters based on the hierarchical cluster). It varies from 0 to 1, with higher values indicating greater similarity. Intuitively, the FM-Index captures the degree to which two labels tend to fall in the same cluster in both tree 1 and tree 2. The Fowlkes-Mallows index is computed as the geometric mean of the ratio of the total number of labels sharing the same cluster in both trees to the number of labels sharing the same cluster in tree 1 and the ratio of the total number of labels sharing the same cluster in both trees to the number of labels sharing the same cluster in tree 2. The FM-Index of two given hierarchical clusterings is then compared to the expected value of the FM-Index under the hypothesis of no relation between the two clusterings.

The plots depict the z-scored FM-Index for each pairwise comparison of the hierarchical cluster trees derived from the human judgement and language similarity data, after cutting the trees into k = 5, 10, 15, and 20 clusters. The dashed line shows the critical value at \(\alpha = .05\) assuming a one-sided hypothesis test (\(z = 1.645\), i.e. the z-score with a tail area of .05).

1.3.2 Adjusted Rand Index

The Rand index is the ratio of the number of pairs of labels on which two clusterings agree (i.e. the number of pairs of labels in the same cluster in both trees and the number of pairs of labels in different clusters in both trees) to the total number of label pairs. The adjusted Rand index (here using Hubert and Arabie’s method) corrects the Rand index for the number of groupings one might expect by chance alone. An adjusted Rand index of 0 indicates two clusterings have a Rand index that matches the expected value for random groupings, with higher and lower values indicating higher- or lower-than-chance level similarity between the two clusterings.

The plots depict the adjusted Rand index for each pairwise comparison of the hierarchical cluster trees derived from the human judgement and language similarity data, after cutting the trees into k = 5, 10, 15, and 20 clusters.

2 Taxonomic Comparision

2.1 Heatmaps

2.1.1 Taxonomy

2.1.2 Language

2.2 Language - Taxonomy correlation

estimate statistic p.value method alternative
-0.2704078 17428394 0 Spearman’s rank correlation rho two.sided

2.3 Taxonomic and language similarity as predictors

3 Feature Choice Task (Texture)

3.1 Reproduction of KEB Figure 6b with language data

3.2 Language - Human correlations

group estimate statistic p.value method alternative
Blind 0.3184788 196264.5 0.0003926 Spearman’s rank correlation rho two.sided
Sighted 0.3226663 195058.6 0.0003253 Spearman’s rank correlation rho two.sided

4 Replication on Second Corpus

4.1 Card Sorting Task

4.2 Taxonomic Comparision

estimate statistic p.value method alternative
-0.2574201 17250219 1e-07 Spearman’s rank correlation rho two.sided

4.3 Feature Choice Task (Texture)

5 Bedny et al. (2019) Reanalysis

participant_group estimate statistic p.value method alternative
Blind 0.5304364 90588.2 0.0e+00 Spearman’s rank correlation rho two.sided
Sighted 0.4695225 102339.7 4.0e-07 Spearman’s rank correlation rho two.sided
Turk 0.4505641 105997.2 1.4e-06 Spearman’s rank correlation rho two.sided

References

Bedny M, Koster-Hale J, Elli G, Yazzolino L, Saxe R (2019) There’s more to “sparkle” than meets the eye: Knowledge of vision and light verbs among congenitally blind and sighted individuals. Cognition 189:105–115.

Benesty, M. (2018). fastrtext: ‘fastText’ Wrapper for Text Classification and Word Representation (R package version 0.2.5). https://CRAN.R-project.org/package=fastrtext/

Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2016). Enriching word vectors with subword information. https://arxiv.org/abs/1607.04606