DisCon Pilot Summary
Successful communication requires coordination between speakers and listeners in a variety of ways. The accrual of shared knowledge between interlocutors (e.g. “common ground”) during discourse is of particular interest. To date, most theories characterize common ground in a categorical fashion - either you have it or you don’t. We explore the degree to which interlocutors display graded sensitivity to common ground, maintaining uncertainty over shared knowledge during a conversation.
Pilot Experiment
Task description
Each experiment contains 6 trials. Each trial contains 6 training slides (non-ambiguous utterance) and 1 test slide (ambiguous utterance), as shown below. For the first experiment, the ambiguous utterance consists of saying “Here’s another one.”
Left: training slide, Right: test slide
The items appearing in a trial belong to a set of 3 categories of items (for e.g. musical instruments, fruits, vehicles) that is unique across 6 trials. We run the experiment with distributions (6-0-0), (4-2-0), and (2-2-2), with each number corresponding to how many times a category has items appear in the trial.
Sample size
We remove responses that get less than 5/6 correct in training slides. If there are duplicated IPs, we only take the first response.
| n |
|---|
| 19 |
Finding
The dotted line represents performance expected by chance (1/3), and the error bar are 95% CIs.
In distribution (6-0-0), most participants choose the most frequent category.
There is a graded response in distribution (4-2-0) corresponding to the frequency of appearance of each category. This fits with our hypothesis that people use common ground to identify the reference of an ambiguous utterance.
However, this result could also be driven by a recency effect where participants simply choose according to the category of the last seen item.
We see that participants are choosing the item in the test slide in the same category as the last training item above chance, suggesting a possibly strong recency effect.
The reason we’re seeing this recency effect may be the semantics of “another” as it makes strong reference to the last category seen.
Recency Effect Experiment
In order to look at the recency effect more closely, we manipulate the experiment to look at distribution (5-1-0) where the last item preceding the test slide is never in the most frequent category. We vary the wording in the ambiguous utterance.
Task description
We use 2 new phrasings for the test slide, “Here’s the last one” and “Look at that.” Distribution (2-2-2), with all category appearances randomized, is used as the control.
Left: training slide, Right: test slide
Sample size
We remove responses that get less than 5/6 correct in training slides. If there are duplicated IPs, we only take the first response.
| wording | n |
|---|---|
| another one | 28 |
| last one | 28 |
| look at that | 25 |
Finding
Distribution (5, 1, 0) where the last item preceding the test slide is never in the most frequent category
We see that the recency effect is reduced when “Here’s the last one” and “Look at that” are used. We observe a higher increase in proportion of participants choosing the item from the least frequent category in the condition “Here’s the last one”, perhaps because it is possible to interpret “the last one” as the item from the category that has not appeared. “Look at that” looks closest to the graded response that corresponds to people using common ground to identify reference.
Proportion of participants choosing the item at test slide in the same category that appears on the last training slide.
Moving forward
We stay with “Look at that.” as the phrasing for the test slide because it produces an inference that is closest to a common ground inference. Furthermore, it’s also compatible with a potential speaker change condition.
We run 2 experiments with adults, with the same setup:
- (6-0-0) vs (4-2-0) vs (2-2-2)
- (3-0-0) vs (2-1-0) vs (1-1-1)
to test the effect of relative and absolute input.
We run the 3rd experiment with other distributions that are interesting, such as (1-0-0) and (3-2-1).
We start pilotting on kids, with the distributions (6-0-0) vs (4-2-0) vs (2-2-2).