Study compares texts
ChatGPT writes poetry more beautifully than Shakespeare
Updated November 15, 2024 – 8:15 a.mReading time: 3 min.
Until recently, science was still convinced: poetry is an area in which humans are better than AI. But now ChatGPT is beating even famous poets.
Poems written by ChatGPT received higher ratings in a survey than original poems by William Shakespeare and other famous authors. On average, the study participants found the artificial intelligence poems more beautiful and rhythmic, as two researchers from the US University of Pittsburgh write in the journal “Scientific Reports”.
“The simplicity of AI-generated poetry may be easier for non-experts to understand, leading them to prefer AI-generated poetry,” the researchers write. It could be that participants misinterpreted the complexity of human poems and assumed that some parts were unrelated words generated by the AI.
For their study, the researchers presented the 1,634 participants with ten poems each. The participants were not experts in poetry; most of them read a poem at most a few times a year. Among the ten poems presented, five were by well-known English poets such as Shakespeare (1564-1616) and TS Eliot (1888-1965). Five of the poems came from ChatGPT, with the AI supposed to produce the texts in the authors’ style.
In a second experiment, 696 other participants were asked to rate the poems according to specific criteria such as quality, beauty, emotion, rhythm and originality. The AI poems beat the authors’ poems in 13 of the 14 categories – but only if the participants did not know who was behind the poems. When they were told this, the AI poems received lower ratings than those of the humans.
The most misattributed poem was an AI poem in the style of Allen Ginsberg (1926-1997). “Almost 70 percent of participants believed it was written by a human poet,” explains co-author Brian Porter. The first verse of the poem goes like this:
“In the stillness of the night | I hear the beat of the city’s heart | The rhythm of the streets, the pulse of life | A symphony of chaos, a work of art” (roughly: “In the silence of the night | I hear the heartbeat of the city | The rhythm of the streets, the pulse of life | A symphony of chaos, a work of art”)
The poems were generated in 2023 using ChatGPT 3.5. “I’ve been experimenting a bit with ChatGPT 4 and 4o recently,” says Porter. “I think that the newer models are more successful in hitting the expected meter,” such as Shakespeare’s iambic pentameter, “but I don’t think I see any significant leaps in content.”
Researchers recently reported in a study that ChatGPT’s jokes were perceived as funnier than the jokes of average people. The two researchers from the University of Pittsburgh also point to a study according to which AI paintings received better grades in a survey than paintings by amateur painters. So is artificial intelligence now beating humans in all creative disciplines?
No, says Porter, things are different with long texts. “As far as I know, large language models cannot yet write indistinguishable novels.” This is probably because they lack the computing power to do so. He also hasn’t seen any evidence yet that an AI can write an entire comedy program that can compete with a human program. “Longer texts generated by AI can still be distinguished from human texts.”
What are the implications of AI-generated poetry on traditional views of human creativity?
Interview between the Time.news Editor and Brian Porter, Poetry Study Co-Author
Editor: Welcome, Brian! It’s a pleasure to have you with us today. Your recent study on AI-generated poetry compared to classical writers like Shakespeare has captured quite a bit of attention. To start, what inspired you to explore this junction of technology and poetry?
Brian Porter: Thank you for having me! The idea really stemmed from the rapid advancements in AI technologies, like ChatGPT, and our curiosity about whether such developments could alter our long-held beliefs about human creativity, especially in the arts. Poetry has traditionally been seen as a distinctly human endeavor, so we wanted to see if AI could challenge that notion.
Editor: That’s fascinating! Your study revealed that AI-generated poems were rated more highly than those from legendary poets by participants who weren’t poetry experts. What do you think contributed to this phenomenon?
Brian Porter: The simplicity and accessibility of the AI-generated poetry likely played a big role. Many participants felt that the AI poems were more beautiful and rhythmic, suggesting that their straightforwardness resonated more with non-experts. There’s also the possibility that participants misinterpreted the complexity of human poetry, leading them to view it as less coherent.
Editor: It’s interesting to note the distinction in perception between AI and human-generated poetry. Can you elaborate more on how you structured the study?
Brian Porter: Certainly. We had 1,634 participants who were presented with 10 poems each—five from famous poets like Shakespeare and T.S. Eliot, and five generated by ChatGPT in their styles. In a subsequent experiment with 696 additional participants, we asked them to rate the poems on specific criteria like beauty and emotion. Remarkably, the AI poems excelled in 13 out of 14 categories—until the participants were told that the poems came from an AI. Then, the ratings shifted notably.
Editor: That’s a striking difference! It seems to suggest a bias against AI when its authorship is revealed. What was the most surprising finding for you?
Brian Porter: The fact that nearly 70% of participants believed an AI poem in the style of Allen Ginsberg was written by a human poet was astonishing. This highlights not just the skill of AI in mimicking styles, but also how we perceive creativity itself.
Editor: Indeed, it raises intriguing questions about authorship and creativity in the age of AI. Given these findings, where do you see the future of poetry heading?
Brian Porter: I think we might see a new collaborative space emerging where AI and human poets work together rather than competing. AI can serve as a tool for inspiration or a partner in the creative process, expanding the boundaries of what poetry can be.
Editor: That sounds promising! How should poets and the general public approach AI-generated poetry moving forward?
Brian Porter: I believe it’s important for both groups to remain open-minded. Poets can experiment with AI as a creative partner, while readers should enjoy the process of discovery with both human and AI-created works. After all, poetry’s essence lies in its ability to evoke emotion, regardless of its origin.
Editor: Thank you for your insights, Brian! This study has certainly opened up a compelling conversation about AI and creativity. We look forward to seeing how this field evolves.
Brian Porter: Thank you! It was great to share these ideas with you, and I’m excited about the dialogue ahead.