Probability assigned to correct answer

Using absolute length

To deal with length variability, we truncate at 20 words and downfill for utterances shorter than that.

Choosing a somewhat arbitrary set of word positions (1,5,10) to look at by expt differences.

Could use improved vis to add the actual tangram images…

Using fractional length

Rstudio / R is not behaving and so won’t show any alpha values less than .002 :(

Delta probs

Biggest changes

tangram text partial partial_length
E the bunny! the bunny! 2
I ice skater ice skater 2
I ice skater ice skater 2
E the bunny the bunny 2
E my b guys this is the bunny one my b guys this is the bunny 7
E it’s the bunny ears it’s the bunny 3
E It’s the bunny Two triangles as its ears atop a square head kinda facing down facing the right It’s the bunny 3
I looks like a figure skate one foot up at the right two arms out in front figure skater looks like a figure skate 5
E the one with bunny ears the one with bunny 4
E the one with bunny ears the one with bunny 4
I looks like an ice skater facing left looks like an ice skater 5
E The only one with rabbit ears Two triangles above the diamond head The only one with rabbit 5
E This is a sad bunny who is leaning on something with a straight back that is to their left This is a sad bunny 5
E this picture actually looks like a bunny rabbit this picture actually looks like a bunny 7
C Looks like a man falling backwards with his arms in the air Looks like a man falling 5
E sitting down facing right looks like a bunny arms not out sitting down facing right looks like a bunny 8
C Looks like someone with body and arms but no legs, and as though they are flying towards the left arms in the air yeah no legs Looks like someone with body and arms but no legs, and as though they are flying 16
I skater ice skater skater ice skater 3
C a person pretending to fly with no legs a person pretending to fly 5
H cat or swaddled baby again bottom part points out to right diamond head two sloping shoulders cat or swaddled 3
A protester walking left, triangle head down, wearing shoes square on its back protester walking 2
F man sitting, facing left knees bent off the ground, feet pointing left man sitting, 2
C A silly man who obviously thinks he can fly Squad head and outstretched arms, like a bird A silly man who obviously thinks he can fly 9
I select the figure skating select the figure skating 4
F looks like man sitting down looks sad pointing toes up knees up and bent looks like man sitting 4
H body like a swaddled baby body like a swaddled 4
H body like a swaddled baby and diamond head body like a swaddled 4
C looks like a flying ghost arms extended on either side of head facing up no legs looks like a flying 4
C this looks like a guy wearing a robe pretending like he’s flying both arms outstretched, head sort of leaning to the left parallelogram (i think ) shaped body this looks like a guy wearing a robe pretending like he’s flying 12
C This person is flying has the arms open wide Floating mid air no legs are shown, just flying This person is flying 4
F man facing left sitting on floor knees bent and head looking down man facing left sitting 4
E sitting down with a weird hat with his square head looking down to the right two pyramid shapes on his square head looks like he’s sitting on the floor looking to the right with his head leaning forward sitting down with a weird hat with his square head looking down to the right 15
A Pretty much sleepwalking to the left, leg raised up with a square on a back on his right Pretty much sleepwalking 3
C the person flying or ghost the person flying 3
C looks like a guy is flying diamond head looks like a guy is flying 6
F man sitting looking sad, facing left with his head down and his knees bent man sitting 2
C This one looks like a guy who’s flying, they’re facing left Arms outstretched *outstretched No legs This one looks like a guy who’s flying, 8
I person ice skating person ice skating 3
C figure flying like ghost triangle head figure flying 2
H bowling pin bowling pin 2
H bowling pin bowling pin 2
G Women in long dress facing right dancing the tango Women in long dress facing right dancing 7
E the one with a big bow or maybe bunny ears, sitting facing the right the one with a big bow 6
E person sitting with what looks like two bunny ears person sitting with what looks like two bunny 8
C Again this is the one that looks like it is a figure flying with both arms outstretched and leaning to the left Again this is the one that looks like it is a figure flying 13
C It looks like a guy flying to the left with outstretched wings It looks like a guy flying 6
E sitting down facing the left depressed with huge triangle ears facing right sitting down facing the left depressed with huge triangle ears 10
F A person slouched with their knees bent facing left A person slouched with their knees 6
C this looks like a person is flying with their arms out to the side this looks like a person is flying 7
F Man Facing left and sitting Looks like he is looking down or up Knees bent No arms Man Facing left and sitting 5
I it looks like an ice skater with arms in the air (left side) and one leg up or a ballerina it looks like an ice skater 6
C they’re flying with arms wide out they’re flying 2
A The chicken dance one that Hera described One food in the air, one on the ground Facing right The feet face left and one foot is up to the knee level The head faces to the right They have the arm on the left up in the air, with the elbow the same height as the top of head The chicken 2
I It’s the ice skater It’s the ice skater 4
A Ok this looks funny to me He is a small human, with a chicken face and 1 wing He is looking to the left direction He is looking down at his raised right foot Ok this looks funny to me He is a small human, with a chicken 14
C ok the one like the snowman flying arms above yes tilted to the left ok the one like the snowman flying 7
A Humanoid figure walking to the left One leg raised Square attached to shoulders on the back (right) Elongated nose beak Humanoid figure walking 3
F a guy leaning to the left, sitting looks like he is doing a exercise crunches no arms visible, only legs 1 leg he looks like he is doing crunches like the exercise for abs a guy leaning to the left, sitting 7
C a person flying a person flying 3
D praying man, arms pointing left, small triangle foot pointing left praying man, arms pointing 4
F sitting in a chair knees bent left head down to the left sitting in a chair knees bent left head down to the left 12
F The person looks like it’s sitting to the left with a square head facing to the left Their legs are slight up The person looks like it’s sitting 6
F deep in thought, sitting position just head no ear deep in thought, 3
F deep in thought, situp position no ears deep in thought, 3
E Girl sitting and facing to our right and has bunny ears Girl sitting and facing to our right and has bunny 10
B Looks like he went to a party with a tie and he did the splinting dance move Looks like he went to a party with a tie 10
F the man facing left, sitting down, no arms the man facing left, sitting 5
F Man facing left looking sad sat down with knees bent Man facing left looking sad sat down 7
E The picture is of a rabbit like body in the sense of two big ears, they are sitting down facing the right, head slanted slightly downwards so the main part of the body is almost a 90 degree triangle The picture is of a rabbit 6
E a hat with two triangles, square head looking down and they are sitting against a wall facing right a hat with two triangles, square head looking down and they are sitting against a wall 16
D Man in robe pointing left Man in robe 3
D man in robe point left man in robe 3
F Person sat down, legs bent in front of them to the left, head down Facing the left Person sat down, legs bent in front of them to the left, head down Facing the left 17
C body only with arms stretched wide tiny square head in the middle almost like they are flying body only with arms stretched wide tiny square head in the middle almost like they are flying 17
D Looks almost like a priest delivering a sermon; wearing a clock, with a foot pointing to the left, arm also pointing to the left *cloak, not a clock Looks almost like a priest 5
A the guy looks like holding a flag duckman the guy looks like holding a flag duckman 8
F The one that looks like he is sitting down The one that looks like he is sitting 8
H baby wrapped up like a grub baby wrapped up like a grub 6
E Flat back sitting to the right Two long ears pointed up Flat back sitting to the right Two long ears 9
E the person is sitting down, with a hat on the hat has 2 ears like a rabbit their body looks like a triangle the person is sitting down, with a hat on the hat has 2 ears like a rabbit 17
I okay this one is the ice skater one leg on the ground the other stuck out behind them two arms stretched out facing left okay this one is the ice skater 7
D It looks like a priest reading at church It looks like a priest 5
A The goofy looking one like a zombie ready to stomp something The goofy looking one like a zombie 7
C man leaning left with both arms in the air man leaning left with both arms in the air 9
F man sitting down with his knees up and feet on the ground facing the left man sitting down with his knees up and feet on the ground facing the left 15
D Looks like a priest with an offering plate Looks like a priest 4
C The figure is leaning sharply to the left, with their arms in the air There are no legs visible The figure is leaning sharply to the left, with their arms in the air 14
F man sat slumped slumped back to the wall facing left knees up toes up diamond head looking down man sat slumped slumped back to the wall 8
F Sitting on the floor with feet to the left, head tilted to the left Sitting on the floor with feet to the left, head tilted to the left 14
F sitting leaning against the wall, knees up, head down, pointing to the left sitting leaning against the wall, knees up, head down, pointing to the left 13
C both hands in the air to the sides no legs they are leaning left both hands in the air to the sides 8
E rabbit body facing left but head turned and looking right 2 different size triangles yeah well if the triangles are overlapped yes rabbit body facing left but head turned and looking 9
F sitting on the ground, head tilted up, looks like he’s thinking about life facing left sitting on the ground, head tilted up, looks like he’s thinking 11
F Man sat down, he has leant his back towards the right of the images, his feet are to the left and his head looks like he’s looking down Man sat down, he has leant his back towards the right 11
F man sitting down looking to his left can clearly see his leg and foot man sitting down looking to his left can clearly see his leg and foot 14
F sitting down, back is to the right, head tipped to the left knees up and feet on the ground to the left flat back sitting down, back is to the right, head tipped to the left knees up and feet 16
I A person with triangles for his body ice skating A person with triangles for his body ice skating 9
F sitting down with legs out, knees bent, looks like a jacket is worn they’re facing the left sitting down with legs out, knees bent, looks like a jacket is worn they’re facing the left 17
E it looks like a box with wings on top of a mountain it looks like a box with wings on top of a mountain 12
E fully sat figure with his head down and two triangles coming out of the head facing towards the right triangular body exactly the evo; rabbit fully sat figure with his head down and two triangles coming out of the head 15

I think final words are > average at diagnostic or content words?

Notes

Plots we want: Identifiability vs delta identifiability How identifiable after x fraction or x words (deal with shorter ones appropriately) Game condition / repNum / tangram

Model: P_correct ~ frac * condition * repNum + (all that | tangram) + more mixed effects Possibly log link function ? Also do this as delta per word?? Confusion matrix GIF is suggested as well as different programming langs Maybe bin either by quintile of length or of words (and possibly divide up by repNum) What to run on humans What words induce the biggest change in prob dist (see if those make sense via intuition?) Biggest KL divergence from one word to the next Biggest delta in correct probability For overall – sample across prob distribution of model (stratified sampling) then do on humans (as a human sample – model/human calibration) Could possibly use to adjust these calibration results to make model fit better in future Alvin also wants more model runs for more things just generally Which contexts - words have highest delta (or highest KL) (look at devbench /Alvin for KL divergence) Maybe more substring models? ? Is_retained_in_next_round ~ informativity (from positive/negative rolling window)