To deal with length variability, we truncate at 20 words and downfill for utterances shorter than that.
Choosing a somewhat arbitrary set of word positions (1,5,10) to look at by expt differences.
Could use improved vis to add the actual tangram images…
Rstudio / R is not behaving and so won’t show any alpha values less than .002 :(
tangram | text | partial | partial_length |
---|---|---|---|
E | the bunny! | the bunny! | 2 |
I | ice skater | ice skater | 2 |
I | ice skater | ice skater | 2 |
E | the bunny | the bunny | 2 |
E | my b guys this is the bunny one | my b guys this is the bunny | 7 |
E | it’s the bunny ears | it’s the bunny | 3 |
E | It’s the bunny Two triangles as its ears atop a square head kinda facing down facing the right | It’s the bunny | 3 |
I | looks like a figure skate one foot up at the right two arms out in front figure skater | looks like a figure skate | 5 |
E | the one with bunny ears | the one with bunny | 4 |
E | the one with bunny ears | the one with bunny | 4 |
I | looks like an ice skater facing left | looks like an ice skater | 5 |
E | The only one with rabbit ears Two triangles above the diamond head | The only one with rabbit | 5 |
E | This is a sad bunny who is leaning on something with a straight back that is to their left | This is a sad bunny | 5 |
E | this picture actually looks like a bunny rabbit | this picture actually looks like a bunny | 7 |
C | Looks like a man falling backwards with his arms in the air | Looks like a man falling | 5 |
E | sitting down facing right looks like a bunny arms not out | sitting down facing right looks like a bunny | 8 |
C | Looks like someone with body and arms but no legs, and as though they are flying towards the left arms in the air yeah no legs | Looks like someone with body and arms but no legs, and as though they are flying | 16 |
I | skater ice skater | skater ice skater | 3 |
C | a person pretending to fly with no legs | a person pretending to fly | 5 |
H | cat or swaddled baby again bottom part points out to right diamond head two sloping shoulders | cat or swaddled | 3 |
A | protester walking left, triangle head down, wearing shoes square on its back | protester walking | 2 |
F | man sitting, facing left knees bent off the ground, feet pointing left | man sitting, | 2 |
C | A silly man who obviously thinks he can fly Squad head and outstretched arms, like a bird | A silly man who obviously thinks he can fly | 9 |
I | select the figure skating | select the figure skating | 4 |
F | looks like man sitting down looks sad pointing toes up knees up and bent | looks like man sitting | 4 |
H | body like a swaddled baby | body like a swaddled | 4 |
H | body like a swaddled baby and diamond head | body like a swaddled | 4 |
C | looks like a flying ghost arms extended on either side of head facing up no legs | looks like a flying | 4 |
C | this looks like a guy wearing a robe pretending like he’s flying both arms outstretched, head sort of leaning to the left parallelogram (i think ) shaped body | this looks like a guy wearing a robe pretending like he’s flying | 12 |
C | This person is flying has the arms open wide Floating mid air no legs are shown, just flying | This person is flying | 4 |
F | man facing left sitting on floor knees bent and head looking down | man facing left sitting | 4 |
E | sitting down with a weird hat with his square head looking down to the right two pyramid shapes on his square head looks like he’s sitting on the floor looking to the right with his head leaning forward | sitting down with a weird hat with his square head looking down to the right | 15 |
A | Pretty much sleepwalking to the left, leg raised up with a square on a back on his right | Pretty much sleepwalking | 3 |
C | the person flying or ghost | the person flying | 3 |
C | looks like a guy is flying diamond head | looks like a guy is flying | 6 |
F | man sitting looking sad, facing left with his head down and his knees bent | man sitting | 2 |
C | This one looks like a guy who’s flying, they’re facing left Arms outstretched *outstretched No legs | This one looks like a guy who’s flying, | 8 |
I | person ice skating | person ice skating | 3 |
C | figure flying like ghost triangle head | figure flying | 2 |
H | bowling pin | bowling pin | 2 |
H | bowling pin | bowling pin | 2 |
G | Women in long dress facing right dancing the tango | Women in long dress facing right dancing | 7 |
E | the one with a big bow or maybe bunny ears, sitting facing the right | the one with a big bow | 6 |
E | person sitting with what looks like two bunny ears | person sitting with what looks like two bunny | 8 |
C | Again this is the one that looks like it is a figure flying with both arms outstretched and leaning to the left | Again this is the one that looks like it is a figure flying | 13 |
C | It looks like a guy flying to the left with outstretched wings | It looks like a guy flying | 6 |
E | sitting down facing the left depressed with huge triangle ears facing right | sitting down facing the left depressed with huge triangle ears | 10 |
F | A person slouched with their knees bent facing left | A person slouched with their knees | 6 |
C | this looks like a person is flying with their arms out to the side | this looks like a person is flying | 7 |
F | Man Facing left and sitting Looks like he is looking down or up Knees bent No arms | Man Facing left and sitting | 5 |
I | it looks like an ice skater with arms in the air (left side) and one leg up or a ballerina | it looks like an ice skater | 6 |
C | they’re flying with arms wide out | they’re flying | 2 |
A | The chicken dance one that Hera described One food in the air, one on the ground Facing right The feet face left and one foot is up to the knee level The head faces to the right They have the arm on the left up in the air, with the elbow the same height as the top of head | The chicken | 2 |
I | It’s the ice skater | It’s the ice skater | 4 |
A | Ok this looks funny to me He is a small human, with a chicken face and 1 wing He is looking to the left direction He is looking down at his raised right foot | Ok this looks funny to me He is a small human, with a chicken | 14 |
C | ok the one like the snowman flying arms above yes tilted to the left | ok the one like the snowman flying | 7 |
A | Humanoid figure walking to the left One leg raised Square attached to shoulders on the back (right) Elongated nose beak | Humanoid figure walking | 3 |
F | a guy leaning to the left, sitting looks like he is doing a exercise crunches no arms visible, only legs 1 leg he looks like he is doing crunches like the exercise for abs | a guy leaning to the left, sitting | 7 |
C | a person flying | a person flying | 3 |
D | praying man, arms pointing left, small triangle foot pointing left | praying man, arms pointing | 4 |
F | sitting in a chair knees bent left head down to the left | sitting in a chair knees bent left head down to the left | 12 |
F | The person looks like it’s sitting to the left with a square head facing to the left Their legs are slight up | The person looks like it’s sitting | 6 |
F | deep in thought, sitting position just head no ear | deep in thought, | 3 |
F | deep in thought, situp position no ears | deep in thought, | 3 |
E | Girl sitting and facing to our right and has bunny ears | Girl sitting and facing to our right and has bunny | 10 |
B | Looks like he went to a party with a tie and he did the splinting dance move | Looks like he went to a party with a tie | 10 |
F | the man facing left, sitting down, no arms | the man facing left, sitting | 5 |
F | Man facing left looking sad sat down with knees bent | Man facing left looking sad sat down | 7 |
E | The picture is of a rabbit like body in the sense of two big ears, they are sitting down facing the right, head slanted slightly downwards so the main part of the body is almost a 90 degree triangle | The picture is of a rabbit | 6 |
E | a hat with two triangles, square head looking down and they are sitting against a wall facing right | a hat with two triangles, square head looking down and they are sitting against a wall | 16 |
D | Man in robe pointing left | Man in robe | 3 |
D | man in robe point left | man in robe | 3 |
F | Person sat down, legs bent in front of them to the left, head down Facing the left | Person sat down, legs bent in front of them to the left, head down Facing the left | 17 |
C | body only with arms stretched wide tiny square head in the middle almost like they are flying | body only with arms stretched wide tiny square head in the middle almost like they are flying | 17 |
D | Looks almost like a priest delivering a sermon; wearing a clock, with a foot pointing to the left, arm also pointing to the left *cloak, not a clock | Looks almost like a priest | 5 |
A | the guy looks like holding a flag duckman | the guy looks like holding a flag duckman | 8 |
F | The one that looks like he is sitting down | The one that looks like he is sitting | 8 |
H | baby wrapped up like a grub | baby wrapped up like a grub | 6 |
E | Flat back sitting to the right Two long ears pointed up | Flat back sitting to the right Two long ears | 9 |
E | the person is sitting down, with a hat on the hat has 2 ears like a rabbit their body looks like a triangle | the person is sitting down, with a hat on the hat has 2 ears like a rabbit | 17 |
I | okay this one is the ice skater one leg on the ground the other stuck out behind them two arms stretched out facing left | okay this one is the ice skater | 7 |
D | It looks like a priest reading at church | It looks like a priest | 5 |
A | The goofy looking one like a zombie ready to stomp something | The goofy looking one like a zombie | 7 |
C | man leaning left with both arms in the air | man leaning left with both arms in the air | 9 |
F | man sitting down with his knees up and feet on the ground facing the left | man sitting down with his knees up and feet on the ground facing the left | 15 |
D | Looks like a priest with an offering plate | Looks like a priest | 4 |
C | The figure is leaning sharply to the left, with their arms in the air There are no legs visible | The figure is leaning sharply to the left, with their arms in the air | 14 |
F | man sat slumped slumped back to the wall facing left knees up toes up diamond head looking down | man sat slumped slumped back to the wall | 8 |
F | Sitting on the floor with feet to the left, head tilted to the left | Sitting on the floor with feet to the left, head tilted to the left | 14 |
F | sitting leaning against the wall, knees up, head down, pointing to the left | sitting leaning against the wall, knees up, head down, pointing to the left | 13 |
C | both hands in the air to the sides no legs they are leaning left | both hands in the air to the sides | 8 |
E | rabbit body facing left but head turned and looking right 2 different size triangles yeah well if the triangles are overlapped yes | rabbit body facing left but head turned and looking | 9 |
F | sitting on the ground, head tilted up, looks like he’s thinking about life facing left | sitting on the ground, head tilted up, looks like he’s thinking | 11 |
F | Man sat down, he has leant his back towards the right of the images, his feet are to the left and his head looks like he’s looking down | Man sat down, he has leant his back towards the right | 11 |
F | man sitting down looking to his left can clearly see his leg and foot | man sitting down looking to his left can clearly see his leg and foot | 14 |
F | sitting down, back is to the right, head tipped to the left knees up and feet on the ground to the left flat back | sitting down, back is to the right, head tipped to the left knees up and feet | 16 |
I | A person with triangles for his body ice skating | A person with triangles for his body ice skating | 9 |
F | sitting down with legs out, knees bent, looks like a jacket is worn they’re facing the left | sitting down with legs out, knees bent, looks like a jacket is worn they’re facing the left | 17 |
E | it looks like a box with wings on top of a mountain | it looks like a box with wings on top of a mountain | 12 |
E | fully sat figure with his head down and two triangles coming out of the head facing towards the right triangular body exactly the evo; rabbit | fully sat figure with his head down and two triangles coming out of the head | 15 |
I think final words are > average at diagnostic or content words?
Plots we want: Identifiability vs delta identifiability How identifiable after x fraction or x words (deal with shorter ones appropriately) Game condition / repNum / tangram
Model: P_correct ~ frac * condition * repNum + (all that | tangram) + more mixed effects Possibly log link function ? Also do this as delta per word?? Confusion matrix GIF is suggested as well as different programming langs Maybe bin either by quintile of length or of words (and possibly divide up by repNum) What to run on humans What words induce the biggest change in prob dist (see if those make sense via intuition?) Biggest KL divergence from one word to the next Biggest delta in correct probability For overall – sample across prob distribution of model (stratified sampling) then do on humans (as a human sample – model/human calibration) Could possibly use to adjust these calibration results to make model fit better in future Alvin also wants more model runs for more things just generally Which contexts - words have highest delta (or highest KL) (look at devbench /Alvin for KL divergence) Maybe more substring models? ? Is_retained_in_next_round ~ informativity (from positive/negative rolling window)