Model RSMs

The following is quoted from Fernandino et al (2022, PNAS):

Taxonomic Models

WordNet is the most influential representational model based on taxonomic information, having been used in several neuroimaging studies to successfully model semantic content (2, 22, 54, 55). It is organized as a knowledge graph in which words are grouped into sets of synonyms (“synsets”), with each expressing a distinct concept. Synsets are richly interconnected according to taxonomic relations, resulting in a hierarchically structured network encompassing 81,426 noun concepts (in the English version). Concept similarity is computed based on the shortest path connecting the two concepts in this network.

In contrast to WordNet, which provides a comprehensive taxonomy of lexical concepts, the Categorical model was customized to encode the particular taxonomic structure of the concept set in each study, based on a set of a priori categories. Therefore, the Categorical model ignores concept categories that were not included in the study, such as “furniture” or “clothing.” To reduce the level of subjectivity involved in the assignment of items to categories and in the evaluation of intercategory similarities, we tested 18 a priori versions of the Categorical model, with different numbers of categories and different levels of hierarchical structure, for the concepts in Study 1 (SI Appendix, Fig. S1). Each version was tested via RSA against the fMRI data, and the best-performing version was selected for comparisons against other types of models. The selected model for Study 1 (model N in SI Appendix, Fig. S1) consisted of 19 hierarchically structured categories, as follows: abstract (mental abstract, social abstract, social event, other abstract), event (social event, concrete event), animate (animal, human, body part), inanimate (artifact [musical instrument, vehicle, other artifact], food, other inanimate), and place.

In Study 2, the Categorical model consisted of two higher-level categories—object and event—with each consisting of four subcategories (animal, plant/food, tool, and vehicle; sound event, social event, communication event, and negative event).

Experiential Models

The Exp48 model consists of 48 dimensions corresponding to distinct aspects of phenomenal experience, such as color, shape, manipulability, and pleasantness (SI Appendix, Table S2). This model is based on the experiential salience norms of Binder and colleagues (40), in which each dimension encodes the relative importance of an experiential attribute according to crowd-sourced ratings. Exp48 encompasses all perceptual, motor, spatial, temporal, causal, valuation, and valence dimensions present in those norms.

The SM8 model consists of a subset of the Exp48 dimensions, focusing exclusively on sensory-motor information. These dimensions represent the relevance of each sensory modality (vision, audition, touch, taste, and smell) and of action schemas performed with each motor effector (hand, foot, and mouth) to the concept. The concept “apple,” for instance, has high values for vision, touch, taste, mouth actions, and hand actions and low values for the other dimensions.

Distributional Models

Distributional information was modeled with two of the most prominent distributional models available. Namely, word2vec (56) uses a deep neural network trained to predict a word based on a context window of a few words preceding and following the target word. In contrast, the model GloVe (57) is based on the ratio of cooccurrence probabilities between pairs of words across the entire corpus. In a comparative evaluation of distributional semantic models (17), word2vec and GloVe emerged as the two top-performing models in predicting human behavior across a variety of semantic tasks.

Load data and setup

library(dplyr)
library(purrr)
library(tidyr)
library(ggplot2)

dsubj <- readRDS("data/lt_reg_stats.rds") |>
  bind_rows(readRDS("data/lt_reg_aITG_stats.rds")) |>
  bind_rows(readRDS("data/lt_reg_aFusG_stats.rds")) |>
  as_tibble() |>
  mutate(
    model_rsm = as.factor(code),
    test = factor(
      test,
      c("omnibus", "model_rsm", "confounds", "RC1_Fam", "RC2_Conc", "RC3_RT", "RC4_Orth")
    ),
    study_id = factor(study_id, c(1,2))
  ) |>
  select(-sid, -model_id, -code) |>
  group_by(study_id, region_id, hemisphere, abbreviation, model_rsm, test) |>
  summarize(n_subj = n(), r_avg = mean(r), se = sd(r) / sqrt(n_subj), p_med = median(p)) |>
  mutate(
    t_subjects_se = r_avg / se,
    p_subjects_se = pt(abs(t_subjects_se), df = n_subj - 1, lower.tail = FALSE) / 2
  ) |>
  group_by(study_id, region_id, hemisphere) |>
  mutate(p_subjects_se_fdr = p.adjust(p_subjects_se, method = "fdr")) |>
  ungroup()

Load data and setup

library(dplyr)
library(purrr)
library(tidyr)
library(ggplot2)

davg <- readRDS("data/lt_reg_COMBINED_avg_stats.rds") |>
  as_tibble() |>
  mutate(
    model_rsm = as.factor(code),
    test = factor(
      test,
      c("omnibus", "model_rsm", "confounds", "RC1_Fam", "RC2_Conc", "RC3_RT", "RC4_Orth")
    ),
    study_id = factor(study_id, c(1,2))
  ) |>
  select(-model_id, -code) |>
  group_by(study_id, region_id, hemisphere) |>
  mutate(p_fdr = p.adjust(p, method = "fdr")) |>
  ungroup()