condition | n |
|---|---|
No chatGPT | 106 |
chatGPT Only | 98 |
chatGPT Supplemented | 95 |
Generative AI and Ownership
Writeup, written for a fellowship proposal
Do people overestimate their contributions when working with generative AI?
Spouses overestimate their proportion of shared household chores (ross 1979 egocentric bias)
Ross & Sicoly, 1979
Study 1:
Each spouse in married couples estimated what % of household chores they did
If estimates are accurate, they should add up to 100%
Sum in each couple > 100%
Study 2:
Pairs of students worked on a problem
Students were either told they succeeded or failed
Students estimated what % of their group’s performance was due to their contributions
Individual’s estimates:
Success: 80%
Failure: 60%
Does the same bias occur when people use generative AI?
Research Qs:
Do people underestimate how much the generative AI tool contributed to a final product?
- 2 And is the bias exaggerated over time? i.e. when people recall work they did with generative AI, do they begin to “forget” how much generative AI contributed to the final product?
Do people feel more ownership of content created using generative AI than content created with another person?
Relevant References
Jago, A. S., & Carroll, G. R. (2023). Who made this? Algorithms and authorship credit. Personality and Social Psychology Bulletin, 01461672221149815.
Does artificial intelligence cause artificial confidence? Generative artificial intelligence as an emerging social referent.Reich, Taly; Teeny, Jacob D. - 4/28/2025
Do people underestimate how much a generative AI tool contributed to a final product?
Procedure:
Participants read an article and then either write a summary of it or have ChatGPT produce a summary of it.
Those who wrote their own summary then either edit their own summary or have ChatGPT edit it.
Participants report what % of the final product they wrote themselves and what % of the ideas were their own.
IV: proportion of work done by ChatGPT
| No ChatGPT. Write summary + edit summary | Supplemented w/ ChatGPT. Write summary + ChatGPT edits it. | Only ChatGPT. ChatGPT writes summary. No edits. |
DVs:
What percentage of the [edited] summary did you write yourself?
What percentage of the ideas in the [edited] summary are yours?
Results
Collected on 08/14/23 on Prolific. Final n = 299
Percent of content written and percent of ideas
[1] 62.27323Comparison of chatGPT’s actual contribution and participants’ perceived contribution
Call:
lm(formula = percent_written ~ ToolsADay_similarity * condition,
data = subset(study1_dat, ToolsADay_similarity < 100))
Residuals:
Min 1Q Median 3Q Max
-76.720 -9.772 2.605 9.040 74.756
Coefficients:
Estimate Std. Error t value
(Intercept) 64.2107 19.4405 3.303
ToolsADay_similarity 0.3632 0.2258 1.608
conditionchatGPT Supplemented -91.3759 21.1481 -4.321
ToolsADay_similarity:conditionchatGPT Supplemented 0.6228 0.2586 2.408
Pr(>|t|)
(Intercept) 0.00119 **
ToolsADay_similarity 0.10982
conditionchatGPT Supplemented 2.78e-05 ***
ToolsADay_similarity:conditionchatGPT Supplemented 0.01722 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 22.82 on 154 degrees of freedom
(4 observations deleted due to missingness)
Multiple R-squared: 0.6891, Adjusted R-squared: 0.683
F-statistic: 113.8 on 3 and 154 DF, p-value: < 2.2e-16exploring the prompts that people used in the chatGPT supplemented condition
[1] "I told it to summarize that large text in 6-8 sentences and it did it in 9 and I told it to fix that."
[2] "I copied & pasted the entire article & clicked to summarize it."
[3] "To summarize the article. I didn't see it actually to know if I did that or not. sorry "
[4] "I pretty much had to fight with it to get it edit the summary. I gave up & just gave it the whole article & told it to summarize it."
[5] "None"
[6] "Summarize this article in 5-6 sentences"
[7] "I didn't use any because they needed too much personal information, such as my age, full name, and cell phone number. "
[8] "I asked ChatGBT to summarize the following article in 6 sentences and then pasted the article."
[9] "I copied the passage and asked for a short summary. I reworded and condescend certain sentences to make it sound more natural. "
[10] "to summarize the article"
[11] "I just told it to summarize the article in six sentences."
[12] "I provided the text and then asked it simply \"Please summarize this in 7 sentences\""
[13] "I told Chat GPT to summarize the article in 6-8 sentences, and copied and pasted the entire article there. I do think the summary it gave was longer. "
[14] "Summarize the below into 5-6 sentances"
[15] "Summarize the following, then pasted in the original text"
[16] "I told the gpt to summarize the article in six sentences"
[17] "summerize article - pasted article in the message box and ChatGPT summerized it"
[18] "Shorten and rewrite this to be less formal"
[19] "Please write a summary \"insert summary\""
[20] "Tell me about death and findings "
[21] "Summarize the given article it did not work so I kept my original"
[22] "I copied the article into the chat box with Bard and asked for a summary of the article. That was the only way I knew how to get a summary of the article. I did not understand how Bard could edit my summary if he did not have the article. Bard's summary was similar to my summary but had more tedious details and was much more wordy."
[23] "I googled the story and then used a tool for summary. "
[24] "Summarize and copied article"
[25] "I told it to summarize the article. I then told it that it was too long and to shorten it. "
[26] "Tell me about finding the remains of King Richard the third "
[27] "Here is a short article about an event that occurred. Please read it carefully, and then thoroughly summarize it, using at least 6-8 sentences. \nHere is a short article about an event that occurred. Please read it carefully, and then thoroughly summarize it, using at least 6-8 sentences.\n"
[28] "I told the AI to summarize the article in 6 to 8 sentences."
[29] "\"summarize the following article in 5-6 sentences: [the article contents pasted here]\""
[30] "Summarize the following statement: X"
[31] "I pasted the text into the program and had it summarize it.\n"
[32] "Researchers from the University of Leicester made a significant discovery when they unearthed a skeleton, confirming it as the remains of English king Richard III. Their analysis revealed a number of injuries, predominantly concentrated on his skull. Notably, a substantial wound caused by a sharp object was identified, penetrating his skull. This finding has led to speculations that Richard III, along with his brother Edward, may have been murdered, possibly at the behest of their uncle. Richard's reign on the throne was short-lived, lasting merely two years before his demise at the Battle of Bosworth in 1485. Intriguingly, the positioning of the arms crossed within the coffin hints at the possibility of Richard being buried with bound wrists."
[33] "I asked chat GPT to summarize the article. I then pasted the article into the chat box."
[34] "I told it to summarize the following and copy pasted the text into the field. "
[35] "Summarize the article"
[36] "I asked it to summarize something for me, then provided the entirety of the article. "
[37] "Summarize for me."
[38] "Summarize this article"
[39] "I just copy and pasted the article text and had it write a summary for me."
[40] "summarize the following article: X"
[41] "#NAME?"
[42] "To summarize the article"
[43] "ChatGPT" Bin width defaults to 1/30 of the range of the data. Pick better value with
`binwidth`.
Notes:
43 out of 95 people in the chatGPT supplemented condition prompted chatGPT to “summarize the article” (or put some other other weird response, like said that they didn’t use generative AI) rather than edit their summary
Somebody said they used SIRI lol
it seems like some people (e.g. PID 162, 169, 270) asked chatGPT to do something but then didn’t change the summary they submitted? because the similarity scores are 100. I’m not sure how many in total were like this
it’s possible that some folks who asked chatGPT to ‘summarize the original article’ then used chatGPTs summary to edit their own (rather than simply copy-pasting chatGPT’s summary). So it’s hard to say whether using that prompt is a reason for exclusion or not?
ANALYSIS: - difference score between similarity score and estimate of contribution to edited summary -> - compare between no chatGPT & chatGPT supplemented conditions - could be that people are underestimating their contribution when working with chat gpt
- ppool
- target N = 200
- final N = 60
- replication of study 1, but with only two conditions, AI assisted or not
Results
Bin width defaults to 1/30 of the range of the data. Pick better value with
`binwidth`.
Bin width defaults to 1/30 of the range of the data. Pick better value with
`binwidth`.
Is the overestimation bias amplified when working with chatGPT vs a real person?
Procedure:
Participants randomly assigned to work with chatGPT or work with another participant.
Participants complete a creativity task (Ward 1994).
- “Imagine you Imagine you have gone to another planet in a far away galaxy that is very different from Earth. You have encountered some animals on this distant planet.
[Collaborate with each other to brainstorm]/[Collaborate with ChatGPT to brainstorm] what one of these animals could look like. Then draw a front view and a side view of the animal. Label the major parts of the animal, and write a short description (1 - 2 sentences) of the animal below it.
[Click this link to open a chat with ChatGPT and get started brainstorming.] Draw your animal on the paper provided. When your drawing & description are complete, click the arrow to continue.
- “Imagine you Imagine you have gone to another planet in a far away galaxy that is very different from Earth. You have encountered some animals on this distant planet.
Participants report what % of the final product they are responsible for and what % of the ideas were their own.
Participants rate themselves on creativity
IV: partner: chatGPT or person
| chatGPT. Collaborate with chatGPT on creativity task | person. Collaborate with partner on creativity task |
DVs:
What percentage of the work did you do in creating the animal drawing?
What percentage of the ideas in the animal drawing are yours?
In general, how would you rate yourself on your creativity? Much less creative than the average person – much more creative than the average person
Results
We collected 326 participants from the psychology participant pool. Participants were randomly assigned to work with chatGPT or to work with another participant. In the person to person condition, participants were instructed to communicate via Slack on their laptops. Accordingly, we maintained records of all communication that happened between participants and chatGPT, and between participants on teams in the person to person condition. Researchers kept participant logs to note down anything of concern that would compromise data. The following datapoints were excluded or fixed in Excel according to those notes:
PID 17 - 18: did not draw an alien. Seemed to have not seen the instructions
PID 184: qualtrics lost responses
277-280 assigned wrong condition on qualtrics1
162 - 163: did not respond to DVs
Final N:
condition | n |
|---|---|
twoPersonTeam | 169 |
generativeAI | 150 |
Total n = 319
T-tests
variable | statistic | df | p | effsize |
|---|---|---|---|---|
creativity | 0.78 | 314 | .439 | 0.09 |
percentage_ideas | -12.34 | 303 | < .001*** | -1.41 |
percentage_work | -1.22 | 312 | .223 | -0.14 |
Part 2
Participants from study 2 came back 2+ weeks later to rate their alien drawings again. They reports how much of the work they contributed, what percentage of the ideas were theirs, and how creative they are.
Results
We collected a total of N = 157 complete responses at time 2.
t-tests of the difference scores for ideas ownership, work ownership, and creativity scores | ||||
|---|---|---|---|---|
variable | statistic | df | p | effsize |
creativity_difference | -0.09 | 155 | .927 | -0.01 |
ideas_difference | 4.22 | 149 | < .001*** | 0.67 |
work_difference | 1.09 | 153 | .279 | 0.17 |
Means and SDs of difference scores of ownership of ideas | |||
|---|---|---|---|
condition | count | mean | sd |
generativeAI | 69 | 13.89 | 20.59 |
twoPersonTeam | 88 | 2.52 | 12.33 |
paired t-test on whether creativity differs between time 1 and time 2 (regardless of condition) | |||||||
|---|---|---|---|---|---|---|---|
.y. | group1 | group2 | n1 | η2 | statistic | df | p |
creativity | immediately | 2+ weeks | 157 | 157 | -4.19 | 156 | < .001*** |
- next study: three entity groups with 2 people and chatGPT
- you have to estimate how much you’re doing, how much another person is doing, how much chat gpt + other people are doing, and trying to mentalize about how other people use chatGPT to accurately estimate
- how do people accurately acknowledge authorship when people know others can be working with chatGPT
- double blind; tell peoiple they are working with people when they are acutally working with chatGPT
- look at time spent on drawing in chatgpt vs two person team
- test by regression controlling for time 1 instead of diference score tests
Transcript analysis
too difficult to automate - should produce a coding scheme with human coding (highest quality).
- code to trace where a final idea started at. count how many ideas generated by one party ended up
can we operationalize emergent properties to describe the qualitative differences between two-person teams and chatGPT teams?
Footnotes
Fixed in excel↩︎