This homework is due by Thursday, January 25th, 8:00pm. Upload a pdf file to Canvas called 2_wrangling.pdf

In this homework, you’ll combine data-wrangling and visualization to reverse-engineer a professional data visualization. The goal of this homework is to replicate Figure 1 as closely as possible. This is known as a “Copy the Master” assignment (Nolan and Perrett 2016). If you have any questions, feel free to come to section (Monday 3:30-4:20pm @ 160-124 or Friday 12:30-1:20pm McMurtry Art Building 350) or post on Ed Discussion.

Dataset description

You’ll be working with a dataset collected in the high throughput psychophysics experimental tradition (Rajalingham et al. 2018). One variant of this approach is to collect data from hundreds, even thousands of participants, in order to generate reliable estimates of perceptual behaviors. Typically, this is accomplished using online data collection. In this dataset there were originally 345 subjects who collectively performed 42,113 trials of a visual discrimination task.

In order to generate the main plot, we’ll analyze a subset of this human behavior. When this is done, we’ll compare human performance with the behavior of neurons and a computational model performing the same task. The neural data is from a neuroscience/psychophysics experiment conducted in non-human primates (Majaj et al. 2015). The computational model data, in our case, was used to validate its correspondence with visual area IT. If you’re interested in this model class, (Yamins and DiCarlo 2016) provides a nice intro. The basic idea is that we can use biologically plausible computational models to understand the relationship between the brain and behavior.

These data were collected to generate a stimulus set with an interesting pattern of perceptual properties: 1. Early visual regions (e.g. primate area V4) can’t do the task at all 2. Late visual regions (e.g. primate aread IT) can do the task on some items 3. Neurotypical humans can do the task on all items (outperforming IT and V4)

The beautiful plot you want to recreate.

The beautiful plot you want to recreate.

Panel A compares model predictions (x axis) with the neural predictions from V4 (y axis). Panel B shows the model predictions (x axis) compared with the neural predictions from IT (y axis), showing that the model is a good approximation for this visual region. Panel C shows the model predictions (x axis) with the behavior of neurotypical humans (y axis). Humans clearly outperform the model (and are clearly better than IT or V4). Panel D is a summary plot, with just the trends from the first three. Panel E is the relationship between reaction time (x axis) and the difference between human performance and neural area IT. This demonstrates that the degree to which human accuracy diverges from visual area IT (but not V4), the more time is required to perform these tasks. Together these results (alongside recent lesion studies) suggest that other brain regions, beyond these canonical sensory regions, are implicated in these kinds of behaviors.

What kind of behavior? Formally, they’re called concurrent visual discrimination (“oddity”) tasks that vary in difficulty. The task is to identify on each trial which one out of three images is the odd one. Here is a screenshot of an example trial. In this case, the lion is the odd one out.

Example trial. Elephant is the `typical object` while lion is the `oddity object.` In the control trials (`V0`), all objects are presented from the same viewpoint. In the experimental trials (`V3`), all objects are presented from different viewpoints. This trial here is one of the `V3` elephant trials.

Example trial. Elephant is the typical object while lion is the oddity object. In the control trials (V0), all objects are presented from the same viewpoint. In the experimental trials (V3), all objects are presented from different viewpoints. This trial here is one of the V3 elephant trials.

You’ll be working with two dataframes in this homework. First, we’ll provide you with a data frame that contains the human behavior you’ll be wrangling and summarizing. Second, a dataframe that contains the accuracy on those same items, but from several different sources: neural responses from two visual regions (V4 and IT) as well as a computational model.

Read in and inspect dataset

Great. Let’s start by loading the human behavioral data frame:

df.human_wide = read_csv("data/human_behavior_V03.csv")

Let’s take a look at the first 40 column names:

df.human_wide %>% 
  names() %>% 
  head(40)
##  [1] "index"              "0-array_type"       "0-choice"          
##  [4] "0-chosen_object"    "0-collection"       "0-database"        
##  [7] "0-iteration"        "0-oddity_category"  "0-oddity_index"    
## [10] "0-oddity_name"      "0-oddity_type"      "0-reaction_time"   
## [13] "0-trial_indices"    "0-trial_number"     "0-typical_category"
## [16] "0-typical_name"     "0-variation_level"  "0-worker_id"       
## [19] "1-array_type"       "1-choice"           "1-chosen_object"   
## [22] "1-collection"       "1-database"         "1-iteration"       
## [25] "1-oddity_category"  "1-oddity_index"     "1-oddity_name"     
## [28] "1-oddity_type"      "1-reaction_time"    "1-trial_indices"   
## [31] "1-trial_number"     "1-typical_category" "1-typical_name"    
## [34] "1-variation_level"  "1-worker_id"        "10-array_type"     
## [37] "10-choice"          "10-chosen_object"   "10-collection"     
## [40] "10-database"

Oh, wow. That’s really untidy data: each row here is a subject, and each column contains 1) a prefix indicating which trial these data correspond to, and 2) a variable name indicating what information is present. So, for example, if we’re interested in the variable variation_level, and there are \(n\) trials, this means there are \(n\) columns containing variation_level :(

Let’s take a look at the content of a single trial, for example, trial 13:

## # A tibble: 345 × 34
##    `13-array_type` `13-choice` `13-chosen_object` `13-collection` `13-database`
##    <chr>                 <dbl> <chr>              <chr>           <chr>        
##  1 3-way                     1 f16                lesion          oddity       
##  2 3-way                     3 _008               lesion          oddity       
##  3 3-way                     1 ELEPHANT_M         lesion          oddity       
##  4 3-way                     3 _010               lesion          oddity       
##  5 3-way                     3 junkers88          lesion          oddity       
##  6 3-way                     3 _008               lesion          oddity       
##  7 3-way                     2 GORILLA            lesion          oddity       
##  8 3-way                     2 _033               lesion          oddity       
##  9 3-way                     3 face0005           lesion          oddity       
## 10 3-way                     3 hedgehog           lesion          oddity       
## # ℹ 335 more rows
## # ℹ 29 more variables: `13-iteration` <chr>, `13-oddity_category` <chr>,
## #   `13-oddity_index` <dbl>, `13-oddity_name` <chr>, `13-oddity_type` <chr>,
## #   `13-reaction_time` <dbl>, `13-trial_indices` <chr>,
## #   `13-trial_number` <dbl>, `13-typical_category` <chr>,
## #   `13-typical_name` <chr>, `13-variation_level` <chr>, `13-worker_id` <dbl>,
## #   `113-array_type` <chr>, `113-choice` <lgl>, `113-chosen_object` <chr>, …
##    [1] "index"                "0-array_type"         "0-choice"            
##    [4] "0-chosen_object"      "0-collection"         "0-database"          
##    [7] "0-iteration"          "0-oddity_category"    "0-oddity_index"      
##   [10] "0-oddity_name"        "0-oddity_type"        "0-reaction_time"     
##   [13] "0-trial_indices"      "0-trial_number"       "0-typical_category"  
##   [16] "0-typical_name"       "0-variation_level"    "0-worker_id"         
##   [19] "1-array_type"         "1-choice"             "1-chosen_object"     
##   [22] "1-collection"         "1-database"           "1-iteration"         
##   [25] "1-oddity_category"    "1-oddity_index"       "1-oddity_name"       
##   [28] "1-oddity_type"        "1-reaction_time"      "1-trial_indices"     
##   [31] "1-trial_number"       "1-typical_category"   "1-typical_name"      
##   [34] "1-variation_level"    "1-worker_id"          "10-array_type"       
##   [37] "10-choice"            "10-chosen_object"     "10-collection"       
##   [40] "10-database"          "10-iteration"         "10-oddity_category"  
##   [43] "10-oddity_index"      "10-oddity_name"       "10-oddity_type"      
##   [46] "10-reaction_time"     "10-trial_indices"     "10-trial_number"     
##   [49] "10-typical_category"  "10-typical_name"      "10-variation_level"  
##   [52] "10-worker_id"         "11-array_type"        "11-choice"           
##   [55] "11-chosen_object"     "11-collection"        "11-database"         
##   [58] "11-iteration"         "11-oddity_category"   "11-oddity_index"     
##   [61] "11-oddity_name"       "11-oddity_type"       "11-reaction_time"    
##   [64] "11-trial_indices"     "11-trial_number"      "11-typical_category" 
##   [67] "11-typical_name"      "11-variation_level"   "11-worker_id"        
##   [70] "12-array_type"        "12-choice"            "12-chosen_object"    
##   [73] "12-collection"        "12-database"          "12-iteration"        
##   [76] "12-oddity_category"   "12-oddity_index"      "12-oddity_name"      
##   [79] "12-oddity_type"       "12-reaction_time"     "12-trial_indices"    
##   [82] "12-trial_number"      "12-typical_category"  "12-typical_name"     
##   [85] "12-variation_level"   "12-worker_id"         "13-array_type"       
##   [88] "13-choice"            "13-chosen_object"     "13-collection"       
##   [91] "13-database"          "13-iteration"         "13-oddity_category"  
##   [94] "13-oddity_index"      "13-oddity_name"       "13-oddity_type"      
##   [97] "13-reaction_time"     "13-trial_indices"     "13-trial_number"     
##  [100] "13-typical_category"  "13-typical_name"      "13-variation_level"  
##  [103] "13-worker_id"         "14-array_type"        "14-choice"           
##  [106] "14-chosen_object"     "14-collection"        "14-database"         
##  [109] "14-iteration"         "14-oddity_category"   "14-oddity_index"     
##  [112] "14-oddity_name"       "14-oddity_type"       "14-reaction_time"    
##  [115] "14-trial_indices"     "14-trial_number"      "14-typical_category" 
##  [118] "14-typical_name"      "14-variation_level"   "14-worker_id"        
##  [121] "15-array_type"        "15-choice"            "15-chosen_object"    
##  [124] "15-collection"        "15-database"          "15-iteration"        
##  [127] "15-oddity_category"   "15-oddity_index"      "15-oddity_name"      
##  [130] "15-oddity_type"       "15-reaction_time"     "15-trial_indices"    
##  [133] "15-trial_number"      "15-typical_category"  "15-typical_name"     
##  [136] "15-variation_level"   "15-worker_id"         "16-array_type"       
##  [139] "16-choice"            "16-chosen_object"     "16-collection"       
##  [142] "16-database"          "16-iteration"         "16-oddity_category"  
##  [145] "16-oddity_index"      "16-oddity_name"       "16-oddity_type"      
##  [148] "16-reaction_time"     "16-trial_indices"     "16-trial_number"     
##  [151] "16-typical_category"  "16-typical_name"      "16-variation_level"  
##  [154] "16-worker_id"         "17-array_type"        "17-choice"           
##  [157] "17-chosen_object"     "17-collection"        "17-database"         
##  [160] "17-iteration"         "17-oddity_category"   "17-oddity_index"     
##  [163] "17-oddity_name"       "17-oddity_type"       "17-reaction_time"    
##  [166] "17-trial_indices"     "17-trial_number"      "17-typical_category" 
##  [169] "17-typical_name"      "17-variation_level"   "17-worker_id"        
##  [172] "18-array_type"        "18-choice"            "18-chosen_object"    
##  [175] "18-collection"        "18-database"          "18-iteration"        
##  [178] "18-oddity_category"   "18-oddity_index"      "18-oddity_name"      
##  [181] "18-oddity_type"       "18-reaction_time"     "18-trial_indices"    
##  [184] "18-trial_number"      "18-typical_category"  "18-typical_name"     
##  [187] "18-variation_level"   "18-worker_id"         "19-array_type"       
##  [190] "19-choice"            "19-chosen_object"     "19-collection"       
##  [193] "19-database"          "19-iteration"         "19-oddity_category"  
##  [196] "19-oddity_index"      "19-oddity_name"       "19-oddity_type"      
##  [199] "19-reaction_time"     "19-trial_indices"     "19-trial_number"     
##  [202] "19-typical_category"  "19-typical_name"      "19-variation_level"  
##  [205] "19-worker_id"         "2-array_type"         "2-choice"            
##  [208] "2-chosen_object"      "2-collection"         "2-database"          
##  [211] "2-iteration"          "2-oddity_category"    "2-oddity_index"      
##  [214] "2-oddity_name"        "2-oddity_type"        "2-reaction_time"     
##  [217] "2-trial_indices"      "2-trial_number"       "2-typical_category"  
##  [220] "2-typical_name"       "2-variation_level"    "2-worker_id"         
##  [223] "20-array_type"        "20-choice"            "20-chosen_object"    
##  [226] "20-collection"        "20-database"          "20-iteration"        
##  [229] "20-oddity_category"   "20-oddity_index"      "20-oddity_name"      
##  [232] "20-oddity_type"       "20-reaction_time"     "20-trial_indices"    
##  [235] "20-trial_number"      "20-typical_category"  "20-typical_name"     
##  [238] "20-variation_level"   "20-worker_id"         "21-array_type"       
##  [241] "21-choice"            "21-chosen_object"     "21-collection"       
##  [244] "21-database"          "21-iteration"         "21-oddity_category"  
##  [247] "21-oddity_index"      "21-oddity_name"       "21-oddity_type"      
##  [250] "21-reaction_time"     "21-trial_indices"     "21-trial_number"     
##  [253] "21-typical_category"  "21-typical_name"      "21-variation_level"  
##  [256] "21-worker_id"         "22-array_type"        "22-choice"           
##  [259] "22-chosen_object"     "22-collection"        "22-database"         
##  [262] "22-iteration"         "22-oddity_category"   "22-oddity_index"     
##  [265] "22-oddity_name"       "22-oddity_type"       "22-reaction_time"    
##  [268] "22-trial_indices"     "22-trial_number"      "22-typical_category" 
##  [271] "22-typical_name"      "22-variation_level"   "22-worker_id"        
##  [274] "23-array_type"        "23-choice"            "23-chosen_object"    
##  [277] "23-collection"        "23-database"          "23-iteration"        
##  [280] "23-oddity_category"   "23-oddity_index"      "23-oddity_name"      
##  [283] "23-oddity_type"       "23-reaction_time"     "23-trial_indices"    
##  [286] "23-trial_number"      "23-typical_category"  "23-typical_name"     
##  [289] "23-variation_level"   "23-worker_id"         "24-array_type"       
##  [292] "24-choice"            "24-chosen_object"     "24-collection"       
##  [295] "24-database"          "24-iteration"         "24-oddity_category"  
##  [298] "24-oddity_index"      "24-oddity_name"       "24-oddity_type"      
##  [301] "24-reaction_time"     "24-trial_indices"     "24-trial_number"     
##  [304] "24-typical_category"  "24-typical_name"      "24-variation_level"  
##  [307] "24-worker_id"         "25-array_type"        "25-choice"           
##  [310] "25-chosen_object"     "25-collection"        "25-database"         
##  [313] "25-iteration"         "25-oddity_category"   "25-oddity_index"     
##  [316] "25-oddity_name"       "25-oddity_type"       "25-reaction_time"    
##  [319] "25-trial_indices"     "25-trial_number"      "25-typical_category" 
##  [322] "25-typical_name"      "25-variation_level"   "25-worker_id"        
##  [325] "26-array_type"        "26-choice"            "26-chosen_object"    
##  [328] "26-collection"        "26-database"          "26-iteration"        
##  [331] "26-oddity_category"   "26-oddity_index"      "26-oddity_name"      
##  [334] "26-oddity_type"       "26-reaction_time"     "26-trial_indices"    
##  [337] "26-trial_number"      "26-typical_category"  "26-typical_name"     
##  [340] "26-variation_level"   "26-worker_id"         "27-array_type"       
##  [343] "27-choice"            "27-chosen_object"     "27-collection"       
##  [346] "27-database"          "27-iteration"         "27-oddity_category"  
##  [349] "27-oddity_index"      "27-oddity_name"       "27-oddity_type"      
##  [352] "27-reaction_time"     "27-trial_indices"     "27-trial_number"     
##  [355] "27-typical_category"  "27-typical_name"      "27-variation_level"  
##  [358] "27-worker_id"         "28-array_type"        "28-choice"           
##  [361] "28-chosen_object"     "28-collection"        "28-database"         
##  [364] "28-iteration"         "28-oddity_category"   "28-oddity_index"     
##  [367] "28-oddity_name"       "28-oddity_type"       "28-reaction_time"    
##  [370] "28-trial_indices"     "28-trial_number"      "28-typical_category" 
##  [373] "28-typical_name"      "28-variation_level"   "28-worker_id"        
##  [376] "29-array_type"        "29-choice"            "29-chosen_object"    
##  [379] "29-collection"        "29-database"          "29-iteration"        
##  [382] "29-oddity_category"   "29-oddity_index"      "29-oddity_name"      
##  [385] "29-oddity_type"       "29-reaction_time"     "29-trial_indices"    
##  [388] "29-trial_number"      "29-typical_category"  "29-typical_name"     
##  [391] "29-variation_level"   "29-worker_id"         "3-array_type"        
##  [394] "3-choice"             "3-chosen_object"      "3-collection"        
##  [397] "3-database"           "3-iteration"          "3-oddity_category"   
##  [400] "3-oddity_index"       "3-oddity_name"        "3-oddity_type"       
##  [403] "3-reaction_time"      "3-trial_indices"      "3-trial_number"      
##  [406] "3-typical_category"   "3-typical_name"       "3-variation_level"   
##  [409] "3-worker_id"          "30-array_type"        "30-choice"           
##  [412] "30-chosen_object"     "30-collection"        "30-database"         
##  [415] "30-iteration"         "30-oddity_category"   "30-oddity_index"     
##  [418] "30-oddity_name"       "30-oddity_type"       "30-reaction_time"    
##  [421] "30-trial_indices"     "30-trial_number"      "30-typical_category" 
##  [424] "30-typical_name"      "30-variation_level"   "30-worker_id"        
##  [427] "31-array_type"        "31-choice"            "31-chosen_object"    
##  [430] "31-collection"        "31-database"          "31-iteration"        
##  [433] "31-oddity_category"   "31-oddity_index"      "31-oddity_name"      
##  [436] "31-oddity_type"       "31-reaction_time"     "31-trial_indices"    
##  [439] "31-trial_number"      "31-typical_category"  "31-typical_name"     
##  [442] "31-variation_level"   "31-worker_id"         "32-array_type"       
##  [445] "32-choice"            "32-chosen_object"     "32-collection"       
##  [448] "32-database"          "32-iteration"         "32-oddity_category"  
##  [451] "32-oddity_index"      "32-oddity_name"       "32-oddity_type"      
##  [454] "32-reaction_time"     "32-trial_indices"     "32-trial_number"     
##  [457] "32-typical_category"  "32-typical_name"      "32-variation_level"  
##  [460] "32-worker_id"         "33-array_type"        "33-choice"           
##  [463] "33-chosen_object"     "33-collection"        "33-database"         
##  [466] "33-iteration"         "33-oddity_category"   "33-oddity_index"     
##  [469] "33-oddity_name"       "33-oddity_type"       "33-reaction_time"    
##  [472] "33-trial_indices"     "33-trial_number"      "33-typical_category" 
##  [475] "33-typical_name"      "33-variation_level"   "33-worker_id"        
##  [478] "34-array_type"        "34-choice"            "34-chosen_object"    
##  [481] "34-collection"        "34-database"          "34-iteration"        
##  [484] "34-oddity_category"   "34-oddity_index"      "34-oddity_name"      
##  [487] "34-oddity_type"       "34-reaction_time"     "34-trial_indices"    
##  [490] "34-trial_number"      "34-typical_category"  "34-typical_name"     
##  [493] "34-variation_level"   "34-worker_id"         "35-array_type"       
##  [496] "35-choice"            "35-chosen_object"     "35-collection"       
##  [499] "35-database"          "35-iteration"         "35-oddity_category"  
##  [502] "35-oddity_index"      "35-oddity_name"       "35-oddity_type"      
##  [505] "35-reaction_time"     "35-trial_indices"     "35-trial_number"     
##  [508] "35-typical_category"  "35-typical_name"      "35-variation_level"  
##  [511] "35-worker_id"         "4-array_type"         "4-choice"            
##  [514] "4-chosen_object"      "4-collection"         "4-database"          
##  [517] "4-iteration"          "4-oddity_category"    "4-oddity_index"      
##  [520] "4-oddity_name"        "4-oddity_type"        "4-reaction_time"     
##  [523] "4-trial_indices"      "4-trial_number"       "4-typical_category"  
##  [526] "4-typical_name"       "4-variation_level"    "4-worker_id"         
##  [529] "5-array_type"         "5-choice"             "5-chosen_object"     
##  [532] "5-collection"         "5-database"           "5-iteration"         
##  [535] "5-oddity_category"    "5-oddity_index"       "5-oddity_name"       
##  [538] "5-oddity_type"        "5-reaction_time"      "5-trial_indices"     
##  [541] "5-trial_number"       "5-typical_category"   "5-typical_name"      
##  [544] "5-variation_level"    "5-worker_id"          "6-array_type"        
##  [547] "6-choice"             "6-chosen_object"      "6-collection"        
##  [550] "6-database"           "6-iteration"          "6-oddity_category"   
##  [553] "6-oddity_index"       "6-oddity_name"        "6-oddity_type"       
##  [556] "6-reaction_time"      "6-trial_indices"      "6-trial_number"      
##  [559] "6-typical_category"   "6-typical_name"       "6-variation_level"   
##  [562] "6-worker_id"          "7-array_type"         "7-choice"            
##  [565] "7-chosen_object"      "7-collection"         "7-database"          
##  [568] "7-iteration"          "7-oddity_category"    "7-oddity_index"      
##  [571] "7-oddity_name"        "7-oddity_type"        "7-reaction_time"     
##  [574] "7-trial_indices"      "7-trial_number"       "7-typical_category"  
##  [577] "7-typical_name"       "7-variation_level"    "7-worker_id"         
##  [580] "8-array_type"         "8-choice"             "8-chosen_object"     
##  [583] "8-collection"         "8-database"           "8-iteration"         
##  [586] "8-oddity_category"    "8-oddity_index"       "8-oddity_name"       
##  [589] "8-oddity_type"        "8-reaction_time"      "8-trial_indices"     
##  [592] "8-trial_number"       "8-typical_category"   "8-typical_name"      
##  [595] "8-variation_level"    "8-worker_id"          "9-array_type"        
##  [598] "9-choice"             "9-chosen_object"      "9-collection"        
##  [601] "9-database"           "9-iteration"          "9-oddity_category"   
##  [604] "9-oddity_index"       "9-oddity_name"        "9-oddity_type"       
##  [607] "9-reaction_time"      "9-trial_indices"      "9-trial_number"      
##  [610] "9-typical_category"   "9-typical_name"       "9-variation_level"   
##  [613] "9-worker_id"          "36-array_type"        "36-choice"           
##  [616] "36-chosen_object"     "36-collection"        "36-database"         
##  [619] "36-iteration"         "36-oddity_category"   "36-oddity_index"     
##  [622] "36-oddity_name"       "36-oddity_type"       "36-reaction_time"    
##  [625] "36-trial_indices"     "36-trial_number"      "36-typical_category" 
##  [628] "36-typical_name"      "36-variation_level"   "36-worker_id"        
##  [631] "37-array_type"        "37-choice"            "37-chosen_object"    
##  [634] "37-collection"        "37-database"          "37-iteration"        
##  [637] "37-oddity_category"   "37-oddity_index"      "37-oddity_name"      
##  [640] "37-oddity_type"       "37-reaction_time"     "37-trial_indices"    
##  [643] "37-trial_number"      "37-typical_category"  "37-typical_name"     
##  [646] "37-variation_level"   "37-worker_id"         "38-array_type"       
##  [649] "38-choice"            "38-chosen_object"     "38-collection"       
##  [652] "38-database"          "38-iteration"         "38-oddity_category"  
##  [655] "38-oddity_index"      "38-oddity_name"       "38-oddity_type"      
##  [658] "38-reaction_time"     "38-trial_indices"     "38-trial_number"     
##  [661] "38-typical_category"  "38-typical_name"      "38-variation_level"  
##  [664] "38-worker_id"         "39-array_type"        "39-choice"           
##  [667] "39-chosen_object"     "39-collection"        "39-database"         
##  [670] "39-iteration"         "39-oddity_category"   "39-oddity_index"     
##  [673] "39-oddity_name"       "39-oddity_type"       "39-reaction_time"    
##  [676] "39-trial_indices"     "39-trial_number"      "39-typical_category" 
##  [679] "39-typical_name"      "39-variation_level"   "39-worker_id"        
##  [682] "40-array_type"        "40-choice"            "40-chosen_object"    
##  [685] "40-collection"        "40-database"          "40-iteration"        
##  [688] "40-oddity_category"   "40-oddity_index"      "40-oddity_name"      
##  [691] "40-oddity_type"       "40-reaction_time"     "40-trial_indices"    
##  [694] "40-trial_number"      "40-typical_category"  "40-typical_name"     
##  [697] "40-variation_level"   "40-worker_id"         "41-array_type"       
##  [700] "41-choice"            "41-chosen_object"     "41-collection"       
##  [703] "41-database"          "41-iteration"         "41-oddity_category"  
##  [706] "41-oddity_index"      "41-oddity_name"       "41-oddity_type"      
##  [709] "41-reaction_time"     "41-trial_indices"     "41-trial_number"     
##  [712] "41-typical_category"  "41-typical_name"      "41-variation_level"  
##  [715] "41-worker_id"         "42-array_type"        "42-choice"           
##  [718] "42-chosen_object"     "42-collection"        "42-database"         
##  [721] "42-iteration"         "42-oddity_category"   "42-oddity_index"     
##  [724] "42-oddity_name"       "42-oddity_type"       "42-reaction_time"    
##  [727] "42-trial_indices"     "42-trial_number"      "42-typical_category" 
##  [730] "42-typical_name"      "42-variation_level"   "42-worker_id"        
##  [733] "43-array_type"        "43-choice"            "43-chosen_object"    
##  [736] "43-collection"        "43-database"          "43-iteration"        
##  [739] "43-oddity_category"   "43-oddity_index"      "43-oddity_name"      
##  [742] "43-oddity_type"       "43-reaction_time"     "43-trial_indices"    
##  [745] "43-trial_number"      "43-typical_category"  "43-typical_name"     
##  [748] "43-variation_level"   "43-worker_id"         "44-array_type"       
##  [751] "44-choice"            "44-chosen_object"     "44-collection"       
##  [754] "44-database"          "44-iteration"         "44-oddity_category"  
##  [757] "44-oddity_index"      "44-oddity_name"       "44-oddity_type"      
##  [760] "44-reaction_time"     "44-trial_indices"     "44-trial_number"     
##  [763] "44-typical_category"  "44-typical_name"      "44-variation_level"  
##  [766] "44-worker_id"         "45-array_type"        "45-choice"           
##  [769] "45-chosen_object"     "45-collection"        "45-database"         
##  [772] "45-iteration"         "45-oddity_category"   "45-oddity_index"     
##  [775] "45-oddity_name"       "45-oddity_type"       "45-reaction_time"    
##  [778] "45-trial_indices"     "45-trial_number"      "45-typical_category" 
##  [781] "45-typical_name"      "45-variation_level"   "45-worker_id"        
##  [784] "46-array_type"        "46-choice"            "46-chosen_object"    
##  [787] "46-collection"        "46-database"          "46-iteration"        
##  [790] "46-oddity_category"   "46-oddity_index"      "46-oddity_name"      
##  [793] "46-oddity_type"       "46-reaction_time"     "46-trial_indices"    
##  [796] "46-trial_number"      "46-typical_category"  "46-typical_name"     
##  [799] "46-variation_level"   "46-worker_id"         "47-array_type"       
##  [802] "47-choice"            "47-chosen_object"     "47-collection"       
##  [805] "47-database"          "47-iteration"         "47-oddity_category"  
##  [808] "47-oddity_index"      "47-oddity_name"       "47-oddity_type"      
##  [811] "47-reaction_time"     "47-trial_indices"     "47-trial_number"     
##  [814] "47-typical_category"  "47-typical_name"      "47-variation_level"  
##  [817] "47-worker_id"         "48-array_type"        "48-choice"           
##  [820] "48-chosen_object"     "48-collection"        "48-database"         
##  [823] "48-iteration"         "48-oddity_category"   "48-oddity_index"     
##  [826] "48-oddity_name"       "48-oddity_type"       "48-reaction_time"    
##  [829] "48-trial_indices"     "48-trial_number"      "48-typical_category" 
##  [832] "48-typical_name"      "48-variation_level"   "48-worker_id"        
##  [835] "49-array_type"        "49-choice"            "49-chosen_object"    
##  [838] "49-collection"        "49-database"          "49-iteration"        
##  [841] "49-oddity_category"   "49-oddity_index"      "49-oddity_name"      
##  [844] "49-oddity_type"       "49-reaction_time"     "49-trial_indices"    
##  [847] "49-trial_number"      "49-typical_category"  "49-typical_name"     
##  [850] "49-variation_level"   "49-worker_id"         "50-array_type"       
##  [853] "50-choice"            "50-chosen_object"     "50-collection"       
##  [856] "50-database"          "50-iteration"         "50-oddity_category"  
##  [859] "50-oddity_index"      "50-oddity_name"       "50-oddity_type"      
##  [862] "50-reaction_time"     "50-trial_indices"     "50-trial_number"     
##  [865] "50-typical_category"  "50-typical_name"      "50-variation_level"  
##  [868] "50-worker_id"         "51-array_type"        "51-choice"           
##  [871] "51-chosen_object"     "51-collection"        "51-database"         
##  [874] "51-iteration"         "51-oddity_category"   "51-oddity_index"     
##  [877] "51-oddity_name"       "51-oddity_type"       "51-reaction_time"    
##  [880] "51-trial_indices"     "51-trial_number"      "51-typical_category" 
##  [883] "51-typical_name"      "51-variation_level"   "51-worker_id"        
##  [886] "52-array_type"        "52-choice"            "52-chosen_object"    
##  [889] "52-collection"        "52-database"          "52-iteration"        
##  [892] "52-oddity_category"   "52-oddity_index"      "52-oddity_name"      
##  [895] "52-oddity_type"       "52-reaction_time"     "52-trial_indices"    
##  [898] "52-trial_number"      "52-typical_category"  "52-typical_name"     
##  [901] "52-variation_level"   "52-worker_id"         "53-array_type"       
##  [904] "53-choice"            "53-chosen_object"     "53-collection"       
##  [907] "53-database"          "53-iteration"         "53-oddity_category"  
##  [910] "53-oddity_index"      "53-oddity_name"       "53-oddity_type"      
##  [913] "53-reaction_time"     "53-trial_indices"     "53-trial_number"     
##  [916] "53-typical_category"  "53-typical_name"      "53-variation_level"  
##  [919] "53-worker_id"         "54-array_type"        "54-choice"           
##  [922] "54-chosen_object"     "54-collection"        "54-database"         
##  [925] "54-iteration"         "54-oddity_category"   "54-oddity_index"     
##  [928] "54-oddity_name"       "54-oddity_type"       "54-reaction_time"    
##  [931] "54-trial_indices"     "54-trial_number"      "54-typical_category" 
##  [934] "54-typical_name"      "54-variation_level"   "54-worker_id"        
##  [937] "55-array_type"        "55-choice"            "55-chosen_object"    
##  [940] "55-collection"        "55-database"          "55-iteration"        
##  [943] "55-oddity_category"   "55-oddity_index"      "55-oddity_name"      
##  [946] "55-oddity_type"       "55-reaction_time"     "55-trial_indices"    
##  [949] "55-trial_number"      "55-typical_category"  "55-typical_name"     
##  [952] "55-variation_level"   "55-worker_id"         "56-array_type"       
##  [955] "56-choice"            "56-chosen_object"     "56-collection"       
##  [958] "56-database"          "56-iteration"         "56-oddity_category"  
##  [961] "56-oddity_index"      "56-oddity_name"       "56-oddity_type"      
##  [964] "56-reaction_time"     "56-trial_indices"     "56-trial_number"     
##  [967] "56-typical_category"  "56-typical_name"      "56-variation_level"  
##  [970] "56-worker_id"         "57-array_type"        "57-choice"           
##  [973] "57-chosen_object"     "57-collection"        "57-database"         
##  [976] "57-iteration"         "57-oddity_category"   "57-oddity_index"     
##  [979] "57-oddity_name"       "57-oddity_type"       "57-reaction_time"    
##  [982] "57-trial_indices"     "57-trial_number"      "57-typical_category" 
##  [985] "57-typical_name"      "57-variation_level"   "57-worker_id"        
##  [988] "58-array_type"        "58-choice"            "58-chosen_object"    
##  [991] "58-collection"        "58-database"          "58-iteration"        
##  [994] "58-oddity_category"   "58-oddity_index"      "58-oddity_name"      
##  [997] "58-oddity_type"       "58-reaction_time"     "58-trial_indices"    
## [1000] "58-trial_number"      "58-typical_category"  "58-typical_name"     
## [1003] "58-variation_level"   "58-worker_id"         "59-array_type"       
## [1006] "59-choice"            "59-chosen_object"     "59-collection"       
## [1009] "59-database"          "59-iteration"         "59-oddity_category"  
## [1012] "59-oddity_index"      "59-oddity_name"       "59-oddity_type"      
## [1015] "59-reaction_time"     "59-trial_indices"     "59-trial_number"     
## [1018] "59-typical_category"  "59-typical_name"      "59-variation_level"  
## [1021] "59-worker_id"         "60-array_type"        "60-choice"           
## [1024] "60-chosen_object"     "60-collection"        "60-database"         
## [1027] "60-iteration"         "60-oddity_category"   "60-oddity_index"     
## [1030] "60-oddity_name"       "60-oddity_type"       "60-reaction_time"    
## [1033] "60-trial_indices"     "60-trial_number"      "60-typical_category" 
## [1036] "60-typical_name"      "60-variation_level"   "60-worker_id"        
## [1039] "61-array_type"        "61-choice"            "61-chosen_object"    
## [1042] "61-collection"        "61-database"          "61-iteration"        
## [1045] "61-oddity_category"   "61-oddity_index"      "61-oddity_name"      
## [1048] "61-oddity_type"       "61-reaction_time"     "61-trial_indices"    
## [1051] "61-trial_number"      "61-typical_category"  "61-typical_name"     
## [1054] "61-variation_level"   "61-worker_id"         "62-array_type"       
## [1057] "62-choice"            "62-chosen_object"     "62-collection"       
## [1060] "62-database"          "62-iteration"         "62-oddity_category"  
## [1063] "62-oddity_index"      "62-oddity_name"       "62-oddity_type"      
## [1066] "62-reaction_time"     "62-trial_indices"     "62-trial_number"     
## [1069] "62-typical_category"  "62-typical_name"      "62-variation_level"  
## [1072] "62-worker_id"         "63-array_type"        "63-choice"           
## [1075] "63-chosen_object"     "63-collection"        "63-database"         
## [1078] "63-iteration"         "63-oddity_category"   "63-oddity_index"     
## [1081] "63-oddity_name"       "63-oddity_type"       "63-reaction_time"    
## [1084] "63-trial_indices"     "63-trial_number"      "63-typical_category" 
## [1087] "63-typical_name"      "63-variation_level"   "63-worker_id"        
## [1090] "64-array_type"        "64-choice"            "64-chosen_object"    
## [1093] "64-collection"        "64-database"          "64-iteration"        
## [1096] "64-oddity_category"   "64-oddity_index"      "64-oddity_name"      
## [1099] "64-oddity_type"       "64-reaction_time"     "64-trial_indices"    
## [1102] "64-trial_number"      "64-typical_category"  "64-typical_name"     
## [1105] "64-variation_level"   "64-worker_id"         "65-array_type"       
## [1108] "65-choice"            "65-chosen_object"     "65-collection"       
## [1111] "65-database"          "65-iteration"         "65-oddity_category"  
## [1114] "65-oddity_index"      "65-oddity_name"       "65-oddity_type"      
## [1117] "65-reaction_time"     "65-trial_indices"     "65-trial_number"     
## [1120] "65-typical_category"  "65-typical_name"      "65-variation_level"  
## [1123] "65-worker_id"         "66-array_type"        "66-choice"           
## [1126] "66-chosen_object"     "66-collection"        "66-database"         
## [1129] "66-iteration"         "66-oddity_category"   "66-oddity_index"     
## [1132] "66-oddity_name"       "66-oddity_type"       "66-reaction_time"    
## [1135] "66-trial_indices"     "66-trial_number"      "66-typical_category" 
## [1138] "66-typical_name"      "66-variation_level"   "66-worker_id"        
## [1141] "67-array_type"        "67-choice"            "67-chosen_object"    
## [1144] "67-collection"        "67-database"          "67-iteration"        
## [1147] "67-oddity_category"   "67-oddity_index"      "67-oddity_name"      
## [1150] "67-oddity_type"       "67-reaction_time"     "67-trial_indices"    
## [1153] "67-trial_number"      "67-typical_category"  "67-typical_name"     
## [1156] "67-variation_level"   "67-worker_id"         "68-array_type"       
## [1159] "68-choice"            "68-chosen_object"     "68-collection"       
## [1162] "68-database"          "68-iteration"         "68-oddity_category"  
## [1165] "68-oddity_index"      "68-oddity_name"       "68-oddity_type"      
## [1168] "68-reaction_time"     "68-trial_indices"     "68-trial_number"     
## [1171] "68-typical_category"  "68-typical_name"      "68-variation_level"  
## [1174] "68-worker_id"         "69-array_type"        "69-choice"           
## [1177] "69-chosen_object"     "69-collection"        "69-database"         
## [1180] "69-iteration"         "69-oddity_category"   "69-oddity_index"     
## [1183] "69-oddity_name"       "69-oddity_type"       "69-reaction_time"    
## [1186] "69-trial_indices"     "69-trial_number"      "69-typical_category" 
## [1189] "69-typical_name"      "69-variation_level"   "69-worker_id"        
## [1192] "70-array_type"        "70-choice"            "70-chosen_object"    
## [1195] "70-collection"        "70-database"          "70-iteration"        
## [1198] "70-oddity_category"   "70-oddity_index"      "70-oddity_name"      
## [1201] "70-oddity_type"       "70-reaction_time"     "70-trial_indices"    
## [1204] "70-trial_number"      "70-typical_category"  "70-typical_name"     
## [1207] "70-variation_level"   "70-worker_id"         "71-array_type"       
## [1210] "71-choice"            "71-chosen_object"     "71-collection"       
## [1213] "71-database"          "71-iteration"         "71-oddity_category"  
## [1216] "71-oddity_index"      "71-oddity_name"       "71-oddity_type"      
## [1219] "71-reaction_time"     "71-trial_indices"     "71-trial_number"     
## [1222] "71-typical_category"  "71-typical_name"      "71-variation_level"  
## [1225] "71-worker_id"         "72-array_type"        "72-choice"           
## [1228] "72-chosen_object"     "72-collection"        "72-database"         
## [1231] "72-iteration"         "72-oddity_category"   "72-oddity_index"     
## [1234] "72-oddity_name"       "72-oddity_type"       "72-reaction_time"    
## [1237] "72-trial_indices"     "72-trial_number"      "72-typical_category" 
## [1240] "72-typical_name"      "72-variation_level"   "72-worker_id"        
## [1243] "73-array_type"        "73-choice"            "73-chosen_object"    
## [1246] "73-collection"        "73-database"          "73-iteration"        
## [1249] "73-oddity_category"   "73-oddity_index"      "73-oddity_name"      
## [1252] "73-oddity_type"       "73-reaction_time"     "73-trial_indices"    
## [1255] "73-trial_number"      "73-typical_category"  "73-typical_name"     
## [1258] "73-variation_level"   "73-worker_id"         "74-array_type"       
## [1261] "74-choice"            "74-chosen_object"     "74-collection"       
## [1264] "74-database"          "74-iteration"         "74-oddity_category"  
## [1267] "74-oddity_index"      "74-oddity_name"       "74-oddity_type"      
## [1270] "74-reaction_time"     "74-trial_indices"     "74-trial_number"     
## [1273] "74-typical_category"  "74-typical_name"      "74-variation_level"  
## [1276] "74-worker_id"         "75-array_type"        "75-choice"           
## [1279] "75-chosen_object"     "75-collection"        "75-database"         
## [1282] "75-iteration"         "75-oddity_category"   "75-oddity_index"     
## [1285] "75-oddity_name"       "75-oddity_type"       "75-reaction_time"    
## [1288] "75-trial_indices"     "75-trial_number"      "75-typical_category" 
## [1291] "75-typical_name"      "75-variation_level"   "75-worker_id"        
## [1294] "76-array_type"        "76-choice"            "76-chosen_object"    
## [1297] "76-collection"        "76-database"          "76-iteration"        
## [1300] "76-oddity_category"   "76-oddity_index"      "76-oddity_name"      
## [1303] "76-oddity_type"       "76-reaction_time"     "76-trial_indices"    
## [1306] "76-trial_number"      "76-typical_category"  "76-typical_name"     
## [1309] "76-variation_level"   "76-worker_id"         "77-array_type"       
## [1312] "77-choice"            "77-chosen_object"     "77-collection"       
## [1315] "77-database"          "77-iteration"         "77-oddity_category"  
## [1318] "77-oddity_index"      "77-oddity_name"       "77-oddity_type"      
## [1321] "77-reaction_time"     "77-trial_indices"     "77-trial_number"     
## [1324] "77-typical_category"  "77-typical_name"      "77-variation_level"  
## [1327] "77-worker_id"         "78-array_type"        "78-choice"           
## [1330] "78-chosen_object"     "78-collection"        "78-database"         
## [1333] "78-iteration"         "78-oddity_category"   "78-oddity_index"     
## [1336] "78-oddity_name"       "78-oddity_type"       "78-reaction_time"    
## [1339] "78-trial_indices"     "78-trial_number"      "78-typical_category" 
## [1342] "78-typical_name"      "78-variation_level"   "78-worker_id"        
## [1345] "79-array_type"        "79-choice"            "79-chosen_object"    
## [1348] "79-collection"        "79-database"          "79-iteration"        
## [1351] "79-oddity_category"   "79-oddity_index"      "79-oddity_name"      
## [1354] "79-oddity_type"       "79-reaction_time"     "79-trial_indices"    
## [1357] "79-trial_number"      "79-typical_category"  "79-typical_name"     
## [1360] "79-variation_level"   "79-worker_id"         "80-array_type"       
## [1363] "80-choice"            "80-chosen_object"     "80-collection"       
## [1366] "80-database"          "80-iteration"         "80-oddity_category"  
## [1369] "80-oddity_index"      "80-oddity_name"       "80-oddity_type"      
## [1372] "80-reaction_time"     "80-trial_indices"     "80-trial_number"     
## [1375] "80-typical_category"  "80-typical_name"      "80-variation_level"  
## [1378] "80-worker_id"         "81-array_type"        "81-choice"           
## [1381] "81-chosen_object"     "81-collection"        "81-database"         
## [1384] "81-iteration"         "81-oddity_category"   "81-oddity_index"     
## [1387] "81-oddity_name"       "81-oddity_type"       "81-reaction_time"    
## [1390] "81-trial_indices"     "81-trial_number"      "81-typical_category" 
## [1393] "81-typical_name"      "81-variation_level"   "81-worker_id"        
## [1396] "82-array_type"        "82-choice"            "82-chosen_object"    
## [1399] "82-collection"        "82-database"          "82-iteration"        
## [1402] "82-oddity_category"   "82-oddity_index"      "82-oddity_name"      
## [1405] "82-oddity_type"       "82-reaction_time"     "82-trial_indices"    
## [1408] "82-trial_number"      "82-typical_category"  "82-typical_name"     
## [1411] "82-variation_level"   "82-worker_id"         "83-array_type"       
## [1414] "83-choice"            "83-chosen_object"     "83-collection"       
## [1417] "83-database"          "83-iteration"         "83-oddity_category"  
## [1420] "83-oddity_index"      "83-oddity_name"       "83-oddity_type"      
## [1423] "83-reaction_time"     "83-trial_indices"     "83-trial_number"     
## [1426] "83-typical_category"  "83-typical_name"      "83-variation_level"  
## [1429] "83-worker_id"         "84-array_type"        "84-choice"           
## [1432] "84-chosen_object"     "84-collection"        "84-database"         
## [1435] "84-iteration"         "84-oddity_category"   "84-oddity_index"     
## [1438] "84-oddity_name"       "84-oddity_type"       "84-reaction_time"    
## [1441] "84-trial_indices"     "84-trial_number"      "84-typical_category" 
## [1444] "84-typical_name"      "84-variation_level"   "84-worker_id"        
## [1447] "85-array_type"        "85-choice"            "85-chosen_object"    
## [1450] "85-collection"        "85-database"          "85-iteration"        
## [1453] "85-oddity_category"   "85-oddity_index"      "85-oddity_name"      
## [1456] "85-oddity_type"       "85-reaction_time"     "85-trial_indices"    
## [1459] "85-trial_number"      "85-typical_category"  "85-typical_name"     
## [1462] "85-variation_level"   "85-worker_id"         "86-array_type"       
## [1465] "86-choice"            "86-chosen_object"     "86-collection"       
## [1468] "86-database"          "86-iteration"         "86-oddity_category"  
## [1471] "86-oddity_index"      "86-oddity_name"       "86-oddity_type"      
## [1474] "86-reaction_time"     "86-trial_indices"     "86-trial_number"     
## [1477] "86-typical_category"  "86-typical_name"      "86-variation_level"  
## [1480] "86-worker_id"         "87-array_type"        "87-choice"           
## [1483] "87-chosen_object"     "87-collection"        "87-database"         
## [1486] "87-iteration"         "87-oddity_category"   "87-oddity_index"     
## [1489] "87-oddity_name"       "87-oddity_type"       "87-reaction_time"    
## [1492] "87-trial_indices"     "87-trial_number"      "87-typical_category" 
## [1495] "87-typical_name"      "87-variation_level"   "87-worker_id"        
## [1498] "88-array_type"        "88-choice"            "88-chosen_object"    
## [1501] "88-collection"        "88-database"          "88-iteration"        
## [1504] "88-oddity_category"   "88-oddity_index"      "88-oddity_name"      
## [1507] "88-oddity_type"       "88-reaction_time"     "88-trial_indices"    
## [1510] "88-trial_number"      "88-typical_category"  "88-typical_name"     
## [1513] "88-variation_level"   "88-worker_id"         "89-array_type"       
## [1516] "89-choice"            "89-chosen_object"     "89-collection"       
## [1519] "89-database"          "89-iteration"         "89-oddity_category"  
## [1522] "89-oddity_index"      "89-oddity_name"       "89-oddity_type"      
## [1525] "89-reaction_time"     "89-trial_indices"     "89-trial_number"     
## [1528] "89-typical_category"  "89-typical_name"      "89-variation_level"  
## [1531] "89-worker_id"         "90-array_type"        "90-choice"           
## [1534] "90-chosen_object"     "90-collection"        "90-database"         
## [1537] "90-iteration"         "90-oddity_category"   "90-oddity_index"     
## [1540] "90-oddity_name"       "90-oddity_type"       "90-reaction_time"    
## [1543] "90-trial_indices"     "90-trial_number"      "90-typical_category" 
## [1546] "90-typical_name"      "90-variation_level"   "90-worker_id"        
## [1549] "91-array_type"        "91-choice"            "91-chosen_object"    
## [1552] "91-collection"        "91-database"          "91-iteration"        
## [1555] "91-oddity_category"   "91-oddity_index"      "91-oddity_name"      
## [1558] "91-oddity_type"       "91-reaction_time"     "91-trial_indices"    
## [1561] "91-trial_number"      "91-typical_category"  "91-typical_name"     
## [1564] "91-variation_level"   "91-worker_id"         "92-array_type"       
## [1567] "92-choice"            "92-chosen_object"     "92-collection"       
## [1570] "92-database"          "92-iteration"         "92-oddity_category"  
## [1573] "92-oddity_index"      "92-oddity_name"       "92-oddity_type"      
## [1576] "92-reaction_time"     "92-trial_indices"     "92-trial_number"     
## [1579] "92-typical_category"  "92-typical_name"      "92-variation_level"  
## [1582] "92-worker_id"         "93-array_type"        "93-choice"           
## [1585] "93-chosen_object"     "93-collection"        "93-database"         
## [1588] "93-iteration"         "93-oddity_category"   "93-oddity_index"     
## [1591] "93-oddity_name"       "93-oddity_type"       "93-reaction_time"    
## [1594] "93-trial_indices"     "93-trial_number"      "93-typical_category" 
## [1597] "93-typical_name"      "93-variation_level"   "93-worker_id"        
## [1600] "94-array_type"        "94-choice"            "94-chosen_object"    
## [1603] "94-collection"        "94-database"          "94-iteration"        
## [1606] "94-oddity_category"   "94-oddity_index"      "94-oddity_name"      
## [1609] "94-oddity_type"       "94-reaction_time"     "94-trial_indices"    
## [1612] "94-trial_number"      "94-typical_category"  "94-typical_name"     
## [1615] "94-variation_level"   "94-worker_id"         "95-array_type"       
## [1618] "95-choice"            "95-chosen_object"     "95-collection"       
## [1621] "95-database"          "95-iteration"         "95-oddity_category"  
## [1624] "95-oddity_index"      "95-oddity_name"       "95-oddity_type"      
## [1627] "95-reaction_time"     "95-trial_indices"     "95-trial_number"     
## [1630] "95-typical_category"  "95-typical_name"      "95-variation_level"  
## [1633] "95-worker_id"         "96-array_type"        "96-choice"           
## [1636] "96-chosen_object"     "96-collection"        "96-database"         
## [1639] "96-iteration"         "96-oddity_category"   "96-oddity_index"     
## [1642] "96-oddity_name"       "96-oddity_type"       "96-reaction_time"    
## [1645] "96-trial_indices"     "96-trial_number"      "96-typical_category" 
## [1648] "96-typical_name"      "96-variation_level"   "96-worker_id"        
## [1651] "100-array_type"       "100-choice"           "100-chosen_object"   
## [1654] "100-collection"       "100-database"         "100-iteration"       
## [1657] "100-oddity_category"  "100-oddity_index"     "100-oddity_name"     
## [1660] "100-oddity_type"      "100-reaction_time"    "100-trial_indices"   
## [1663] "100-trial_number"     "100-typical_category" "100-typical_name"    
## [1666] "100-variation_level"  "100-worker_id"        "101-array_type"      
## [1669] "101-choice"           "101-chosen_object"    "101-collection"      
## [1672] "101-database"         "101-iteration"        "101-oddity_category" 
## [1675] "101-oddity_index"     "101-oddity_name"      "101-oddity_type"     
## [1678] "101-reaction_time"    "101-trial_indices"    "101-trial_number"    
## [1681] "101-typical_category" "101-typical_name"     "101-variation_level" 
## [1684] "101-worker_id"        "102-array_type"       "102-choice"          
## [1687] "102-chosen_object"    "102-collection"       "102-database"        
## [1690] "102-iteration"        "102-oddity_category"  "102-oddity_index"    
## [1693] "102-oddity_name"      "102-oddity_type"      "102-reaction_time"   
## [1696] "102-trial_indices"    "102-trial_number"     "102-typical_category"
## [1699] "102-typical_name"     "102-variation_level"  "102-worker_id"       
## [1702] "103-array_type"       "103-choice"           "103-chosen_object"   
## [1705] "103-collection"       "103-database"         "103-iteration"       
## [1708] "103-oddity_category"  "103-oddity_index"     "103-oddity_name"     
## [1711] "103-oddity_type"      "103-reaction_time"    "103-trial_indices"   
## [1714] "103-trial_number"     "103-typical_category" "103-typical_name"    
## [1717] "103-variation_level"  "103-worker_id"        "104-array_type"      
## [1720] "104-choice"           "104-chosen_object"    "104-collection"      
## [1723] "104-database"         "104-iteration"        "104-oddity_category" 
## [1726] "104-oddity_index"     "104-oddity_name"      "104-oddity_type"     
## [1729] "104-reaction_time"    "104-trial_indices"    "104-trial_number"    
## [1732] "104-typical_category" "104-typical_name"     "104-variation_level" 
## [1735] "104-worker_id"        "105-array_type"       "105-choice"          
## [1738] "105-chosen_object"    "105-collection"       "105-database"        
## [1741] "105-iteration"        "105-oddity_category"  "105-oddity_index"    
## [1744] "105-oddity_name"      "105-oddity_type"      "105-reaction_time"   
## [1747] "105-trial_indices"    "105-trial_number"     "105-typical_category"
## [1750] "105-typical_name"     "105-variation_level"  "105-worker_id"       
## [1753] "106-array_type"       "106-choice"           "106-chosen_object"   
## [1756] "106-collection"       "106-database"         "106-iteration"       
## [1759] "106-oddity_category"  "106-oddity_index"     "106-oddity_name"     
## [1762] "106-oddity_type"      "106-reaction_time"    "106-trial_indices"   
## [1765] "106-trial_number"     "106-typical_category" "106-typical_name"    
## [1768] "106-variation_level"  "106-worker_id"        "107-array_type"      
## [1771] "107-choice"           "107-chosen_object"    "107-collection"      
## [1774] "107-database"         "107-iteration"        "107-oddity_category" 
## [1777] "107-oddity_index"     "107-oddity_name"      "107-oddity_type"     
## [1780] "107-reaction_time"    "107-trial_indices"    "107-trial_number"    
## [1783] "107-typical_category" "107-typical_name"     "107-variation_level" 
## [1786] "107-worker_id"        "108-array_type"       "108-choice"          
## [1789] "108-chosen_object"    "108-collection"       "108-database"        
## [1792] "108-iteration"        "108-oddity_category"  "108-oddity_index"    
## [1795] "108-oddity_name"      "108-oddity_type"      "108-reaction_time"   
## [1798] "108-trial_indices"    "108-trial_number"     "108-typical_category"
## [1801] "108-typical_name"     "108-variation_level"  "108-worker_id"       
## [1804] "109-array_type"       "109-choice"           "109-chosen_object"   
## [1807] "109-collection"       "109-database"         "109-iteration"       
## [1810] "109-oddity_category"  "109-oddity_index"     "109-oddity_name"     
## [1813] "109-oddity_type"      "109-reaction_time"    "109-trial_indices"   
## [1816] "109-trial_number"     "109-typical_category" "109-typical_name"    
## [1819] "109-variation_level"  "109-worker_id"        "110-array_type"      
## [1822] "110-choice"           "110-chosen_object"    "110-collection"      
## [1825] "110-database"         "110-iteration"        "110-oddity_category" 
## [1828] "110-oddity_index"     "110-oddity_name"      "110-oddity_type"     
## [1831] "110-reaction_time"    "110-trial_indices"    "110-trial_number"    
## [1834] "110-typical_category" "110-typical_name"     "110-variation_level" 
## [1837] "110-worker_id"        "111-array_type"       "111-choice"          
## [1840] "111-chosen_object"    "111-collection"       "111-database"        
## [1843] "111-iteration"        "111-oddity_category"  "111-oddity_index"    
## [1846] "111-oddity_name"      "111-oddity_type"      "111-reaction_time"   
## [1849] "111-trial_indices"    "111-trial_number"     "111-typical_category"
## [1852] "111-typical_name"     "111-variation_level"  "111-worker_id"       
## [1855] "112-array_type"       "112-choice"           "112-chosen_object"   
## [1858] "112-collection"       "112-database"         "112-iteration"       
## [1861] "112-oddity_category"  "112-oddity_index"     "112-oddity_name"     
## [1864] "112-oddity_type"      "112-reaction_time"    "112-trial_indices"   
## [1867] "112-trial_number"     "112-typical_category" "112-typical_name"    
## [1870] "112-variation_level"  "112-worker_id"        "113-array_type"      
## [1873] "113-choice"           "113-chosen_object"    "113-collection"      
## [1876] "113-database"         "113-iteration"        "113-oddity_category" 
## [1879] "113-oddity_index"     "113-oddity_name"      "113-oddity_type"     
## [1882] "113-reaction_time"    "113-trial_indices"    "113-trial_number"    
## [1885] "113-typical_category" "113-typical_name"     "113-variation_level" 
## [1888] "113-worker_id"        "97-array_type"        "97-choice"           
## [1891] "97-chosen_object"     "97-collection"        "97-database"         
## [1894] "97-iteration"         "97-oddity_category"   "97-oddity_index"     
## [1897] "97-oddity_name"       "97-oddity_type"       "97-reaction_time"    
## [1900] "97-trial_indices"     "97-trial_number"      "97-typical_category" 
## [1903] "97-typical_name"      "97-variation_level"   "97-worker_id"        
## [1906] "98-array_type"        "98-choice"            "98-chosen_object"    
## [1909] "98-collection"        "98-database"          "98-iteration"        
## [1912] "98-oddity_category"   "98-oddity_index"      "98-oddity_name"      
## [1915] "98-oddity_type"       "98-reaction_time"     "98-trial_indices"    
## [1918] "98-trial_number"      "98-typical_category"  "98-typical_name"     
## [1921] "98-variation_level"   "98-worker_id"         "99-array_type"       
## [1924] "99-choice"            "99-chosen_object"     "99-collection"       
## [1927] "99-database"          "99-iteration"         "99-oddity_category"  
## [1930] "99-oddity_index"      "99-oddity_name"       "99-oddity_type"      
## [1933] "99-reaction_time"     "99-trial_indices"     "99-trial_number"     
## [1936] "99-typical_category"  "99-typical_name"      "99-variation_level"  
## [1939] "99-worker_id"

Variables of interest we’ll be looking at:

Variables that you need to create:

Step 1: Data Wrangling

  1. Convert the data from wide to long format, such that each row corresponds to a single trial, preserving all the other useful information (like worker_id, typical_name, etc.). You might call this new data frame df.human_long. This may require two transformation steps. First, you’ll separate the current column headings into new columns: the trial and the variable name indicating what information was present. Second, you’ll structure the dataset so that each row corresponds to a single trial, with the variable names indicating what information was present being columns.
library(tidyr)

df.human_long = df.human_wide %>%
# pivoting
pivot_longer(cols = -index,
names_to = c("trial", "var_name"),
names_sep = "-",
values_to = "Var",
values_transform = list(Var = "as.character")) |>
pivot_wider(names_from = var_name,
values_from = Var)
  1. Select only those columns that you need (as indicated in the list of variables of interest mentioned above). Also remove the index column, rename the worker_id column to participant and put it as the left-most column of the data frame.

*index: an index for each row in the data frame * oddity_name: the object-level description for the oddity (e.g. face0001) * chosen_object: the name of the object that subjects selected as the oddity * typical_name: the object-level description for the typical objects (e.g. face0002) * variation_level: V0 are the easy control trials & V3 are the challenging experimental trials * reaction_time: amount of time before a subject responded (in milliseconds) * worker_id**: id for each participant

"worker_id" %in% names(df.human_long)
## [1] TRUE
library(dplyr)

df.human_long <- df.human_long %>%
  rename(participant = worker_id) %>%
  select(participant, oddity_name, chosen_object, typical_name, variation_level, reaction_time)
  1. Remove all the rows in the long data frame that contain missing values (NA) in the participant column (i.e. the column that used to be the worker_id column before you renamed it in step 2).

Tip: The drop_na() function will be helpful here.

library(dplyr)

df.human_long <- df.human_long %>%
  drop_na(participant)
  1. Clean up the data types by turning the <chr> columns that contain only numbers into numeric columns.
df.n <- df.human_long %>% 
  mutate(across(.cols = c(participant, reaction_time),
                .fns = ~ as.numeric(.)))

head(df.n)
## # A tibble: 6 × 6
##   participant oddity_name chosen_object typical_name variation_level
##         <dbl> <chr>       <chr>         <chr>        <chr>          
## 1           0 _011        _011          _033         V0             
## 2           0 face0001    face0001      face0004     V3             
## 3           0 _014        _014          _001         V3             
## 4           0 f16         f16           junkers88    V3             
## 5           0 face0005    face0005      face0002     V3             
## 6           0 f16         f16           rdbarren     V3             
## # ℹ 1 more variable: reaction_time <dbl>
  1. Generate a correct column by checking whether the object a participant chose (chosen_object) matches the oddity (oddity_name). That is, you want to check whether they correctly identified the odd one out.
df.n <- df.n %>% 
  mutate(correct = chosen_object == oddity_name)
  1. Print the first 10 rows of your data frame that you’ve created in 5).
head(df.n, 10)
## # A tibble: 10 × 7
##    participant oddity_name chosen_object typical_name variation_level
##          <dbl> <chr>       <chr>         <chr>        <chr>          
##  1           0 _011        _011          _033         V0             
##  2           0 face0001    face0001      face0004     V3             
##  3           0 _014        _014          _001         V3             
##  4           0 f16         f16           junkers88    V3             
##  5           0 face0005    face0005      face0002     V3             
##  6           0 f16         f16           rdbarren     V3             
##  7           0 LIONESS     LIONESS       TURTLE_L     V3             
##  8           0 _05_future  _05_future    f16          V3             
##  9           0 hedgehog    hedgehog      ELEPHANT_M   V3             
## 10           0 _004        _004          _010         V0             
## # ℹ 2 more variables: reaction_time <dbl>, correct <lgl>

Step 2: Analysis of human behavioral data

  1. Identify subjects who were more than 90% accurate on the control trials. Then, keep only those participants (who were more than 90% accurate on the control trials) in the data frame that you’ve created in 5).

Tip: Remember that the control trials are marked as “V0” in variation_level.

controltrials <- df.n %>%
  filter(variation_level == "V0") %>% group_by(participant) %>%
  filter(mean(correct == TRUE, na.rm = TRUE) > .90)
  1. Create a data frame that contains the average accuracy and reaction time for each item (as identified by typical_name). The estimates should be rounded to two decimal places.
df.ct <- controltrials %>%
  group_by(typical_name) %>%
  summarise(
    # Ensure that 'correct' is a logical vector. Adjust as needed.
    average_accuracy = round(mean(correct == TRUE, na.rm = TRUE), 2),
    average_reaction_time = round(mean(reaction_time, na.rm = TRUE), 2)
  )
  1. Print the top 10 rows of the data frame that you’ve created in 2).
head(df.ct, 10)
## # A tibble: 10 × 3
##    typical_name average_accuracy average_reaction_time
##    <chr>                   <dbl>                 <dbl>
##  1 ELEPHANT_M               1                    1788.
##  2 GORILLA                  1                    1552.
##  3 LIONESS                  0.98                 1935.
##  4 TURTLE_L                 0.98                 2003.
##  5 _001                     1                    1758.
##  6 _004                     0.99                 1901.
##  7 _008                     1                    1881.
##  8 _010                     1                    2142.
##  9 _011                     0.99                 1859.
## 10 _014                     0.99                 1813.

Step 3: Compare human and neural behavior

If all went well, you should have created the same data frame that is also saved in data/human_accuracy_and_reaction_time_per_item.csv. If you weren’t quite able to create this data frame, you can load it in the code chunk below and continue to work with it.

df.human_long = read_csv("data/human_accuracy_and_reaction_time_per_item.csv")

Now, let’s load in the data frame that contains the neural and computational performance on those same typical_name items.

df.neural_and_model = read_csv("data/neural_and_computational_predictions_per_item.csv")
  1. Combine the human with the neural and computational data by joining the df.human_long data frame with the df.neural_and_model data frame.

Tip: Use the typical_name column to join the data frames.

summary(df.neural_and_model)
##      model              it               v4         typical_name      
##  Min.   :0.2767   Min.   :0.2600   Min.   :0.2300   Length:32         
##  1st Qu.:0.4512   1st Qu.:0.4400   1st Qu.:0.3600   Class :character  
##  Median :0.7220   Median :0.5867   Median :0.3992   Mode  :character  
##  Mean   :0.6731   Mean   :0.5984   Mean   :0.4280                     
##  3rd Qu.:0.8731   3rd Qu.:0.7488   3rd Qu.:0.5065                     
##  Max.   :0.9525   Max.   :0.9400   Max.   :0.6520
joint_df <- df.human_long %>% 
  left_join(df.neural_and_model, 
            by = "typical_name")


tail(joint_df)
## # A tibble: 6 × 6
##   typical_name human_accuracy human_reaction_time model    it    v4
##   <chr>                 <dbl>               <dbl> <dbl> <dbl> <dbl>
## 1 junkers88              0.91               3082. 0.568 0.443 0.395
## 2 LIONESS                0.93               2278. 0.886 0.794 0.564
## 3 mig29                  0.84               3221. 0.592 0.452 0.38 
## 4 rdbarren               0.9                2657. 0.8   0.58  0.518
## 5 sopwith                0.91               2640. 0.858 0.74  0.394
## 6 TURTLE_L               0.94               2248. 0.942 0.87  0.652
  1. Create a new column in your data frame that contains the difference between human performance and (neural) IT performance on each item.
# YOUR CODE 

View(joint_df)


joint_df <- joint_df %>% ## creating a new variable
  mutate(dif.var = human_accuracy - it)
  1. Print out the top 10 rows of the data frame that you’ve created in 2).
head(joint_df, 10)
## # A tibble: 10 × 7
##    typical_name     human_accuracy human_reaction_time model    it    v4 dif.var
##    <chr>                     <dbl>               <dbl> <dbl> <dbl> <dbl>   <dbl>
##  1 _001                       0.89               2272. 0.912 0.74  0.506   0.15 
##  2 _004                       0.94               1971. 0.952 0.722 0.31    0.217
##  3 _008                       0.91               2517. 0.792 0.69  0.378   0.22 
##  4 _01_Airliner_2j…           0.89               2812. 0.545 0.378 0.485   0.512
##  5 _010                       0.94               2513. 0.335 0.5   0.36    0.44 
##  6 _011                       0.91               2380. 0.828 0.667 0.377   0.243
##  7 _014                       0.91               2351. 0.888 0.708 0.403   0.202
##  8 _031                       0.9                2590. 0.637 0.593 0.312   0.307
##  9 _033                       0.93               2391. 0.634 0.646 0.364   0.284
## 10 _05_future                 0.86               3121. 0.835 0.562 0.452   0.298
  1. Plot the correspondence between neural and human behaviors (i.e. recreate the figure panel shown at the very top of this document). Try to tweak the aspects of your figure to make it match the one shown above (but don’t sweat it if you can’t get everything to perfectly match).

Tip: For panels A to D the model accuracy (in the column model) is shown on the x-axis, and what is shown on the y-axis differs between the panels (A: v4, B: it, C: human_accuracy). For D, instead of showing a scatter plot, use geom_smooth(method = "lm") to show the regression lines. For E, use a scatter plot to show the relationship between the difference in accuracy between human and it performance on the y-axis (which you’ve already calculated above), and reaction time on the x-axis.

To combine the different plots into one figure panel, take a look at the patchwork package. You can find an example for how to use that package in the notes for the Visualization 2 lecture (visualization2.Rmd).

# YOUR CODE 

##plot1 

p1 <- ggplot(data = joint_df, 
            aes(x = `model`, y = v4)) +
       geom_point(shape = 21, 
                  colour = "black", 
                  fill = "grey", 
                  size = 3) +
       coord_cartesian(xlim = c(0.2, 1),
                       ylim = c(0.2, 1)) +
       geom_abline(linetype = "dashed", 
                   slope = 1,
                   intercept = 0) + 
       theme_classic() +
       labs(x = "Model Performance",
            y = "V4 Performance", 
            title = "Model vs. Neural Model V4 Comparison")


##plot2


p2 <- ggplot(data = joint_df, 
       aes(x = model, y = it)) + 
  geom_point(shape = 21, 
             colour = "black", 
             fill = "yellow", 
             size = 3) + 
  coord_cartesian(xlim = c(0.2, 1),
                  ylim = c(0.2, 1)) +
  geom_abline(linetype = "dashed", 
              slope = 1,
              intercept = 0)+ 
  theme_classic() +
  labs(x = "model performance",
       y = "it", 
       title = "model predicts neural area IT")


##Plot C
p3 <- ggplot(data = joint_df, 
       aes(x = model, y = human_accuracy)) + 
  geom_point(shape = 21, 
             colour = "black", 
             fill = "black", 
             size = 3) + 
  coord_cartesian(xlim = c(0.2, 1),
                  ylim = c(0.2, 1)) +
  geom_abline(linetype = "dashed", 
              slope = 1,
              intercept = 0)+ 
  theme_classic() +
  labs(x = "model performance",
       y = "it", 
       title = "humans outperform the model")

##plot E

p4 <- ggplot(data = joint_df, 
       aes(x = human_reaction_time, y = dif.var)) +
  geom_point(shape = 21, 
             colour = "black", 
             fill = "orange", 
             size = 3) +
  theme_classic() +
  labs(x = "Reaction Time",
       y = "Human Neural", 
       title = "Difference between Human and IT is Related to Reaction Time")

Grading rubric

References

Majaj, Najib J, Ha Hong, Ethan A Solomon, and James J DiCarlo. 2015. “Simple Learned Weighted Sums of Inferior Temporal Neuronal Firing Rates Accurately Predict Human Core Object Recognition Performance.” Journal of Neuroscience 35 (39): 13402–18.
Nolan, Deborah, and Jamis Perrett. 2016. “Teaching and Learning Data Visualization: Ideas and Assignments.” The American Statistician 70 (3): 260–69. https://doi.org/10.1080/00031305.2015.1123651.
Rajalingham, Rishi, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J DiCarlo. 2018. “Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks.” Journal of Neuroscience 38 (33): 7255–69.
Yamins, Daniel LK, and James J DiCarlo. 2016. “Using Goal-Driven Deep Learning Models to Understand Sensory Cortex.” Nature Neuroscience 19 (3): 356.