Did a robot write that? Study suggests people prefer AI authors

By Stephen Beech

People prefer stories written by machine — especially when told they were written by a human, according to new research.

The highest ratings were given by participants in the study to stories generated by artificial intelligence (AI) that they were told had been written by a person.

They were unable to tell the difference between human- and AI-created stories, according to the findings.

The study, led by researchers at Villanova University, adds to growing evidence that many people's ability to spot AI-generated content is limited.

For the study, participants ages 18 to 81 were tested to see whether they could tell the difference between human-written and AI-generated short stories, how they rated the quality and appeal of each, and whether they could spot AI-generated content.

The researchers used three fictional short stories written by human authors published in a reputable literary journal or short story collection, alongside three corresponding short stories generated by ChatGPT.

In the first experiment, 1,682 participants were asked to read a story and were either correctly or falsely told that it was written by a human or AI.

They then rated the story's quality and how engaging they found it.

In the second and third experiments, participants received one human-written and one AI-generated story, but were not told the origin of the stories.

After reading them, they were asked to identify the author of each story.

Participants were also asked to rate their experience with using AI platforms such as Chat GPT, and their experience with fictional literature.

The results showed that AI-generated stories received higher ratings for both quality — how well-written the stories were — and absorption — how invested in the story the reader was — than human-written stories.

The findings, published in the journal Judgment and Decision Making, also showed that AI-generated stories were rated even more highly when participants believed they had been written by a human.

Participants' ability to identify AI-generated content was just 39.9% in experiment two and 51.9% in experiment three.

Study senior author Deena Weisberg said: "In our study, participants gave the highest ratings to stories that they had been told were written by humans but which were actually written by AI.

"I'd argue that this reveals a bias toward narratives written by real people.

"We assume creative writing requires uniquely human qualities, such as emotional understanding and lived experience, which leads people to underestimate AI's capabilities.

"I'd say that public assumptions about AI's capabilities are increasingly out of date."

The research team also found that participants who said they had greater AI literacy were better able to identify AI-written text, but participants who said they had greater expertise in literature were not.

In the second experiment, the findings showed that each one-point increase in participants' self-reported AI expertise score was associated with a 14% increase in the odds of correctly guessing the story's author.

In the third experiment, which used a previously validated measure called the Artificial Intelligence Literacy Scale (AILS), a one-point increase in participants' scores was associated with a 33% increase in the odds of correctly guessing the story's author.

Weisberg, from Villanova University's department of psychological and brain sciences, said: "One thing to note was that familiarity with AI systems appeared to help people recognize the patterns typical of AI-generated writing, such as em dashes and sentence structures such as 'it's not just X, it's Y.'

"That suggests that improving AI literacy would be one way to help people to navigate the new AI-enabled world that we're living in."

The study validates previous research that has shown that people often prefer AI-generated writing in various formats, including short stories, poetry and essays.

Speculating on why that might be, Weisberg said: "AI writing tends to be clearer, more direct and easier to process.

"By contrast, human-written stories, are often more subtle and complex.

"For example, the AI versions of our stories usually stated their themes explicitly, rather than allowing readers to infer meaning from the characters' words and actions.

"We hear a lot about the crisis in literacy, shrinking attention spans and the influence of platforms such as TikTok.

"But I don't think we can say that our findings in this study can be entirely explained by contemporary media consumption.

"It's more that these technologies amplify existing tendencies, rather than creating them."

She added: "Perhaps people generally prefer predictability, because difficult or subtle material requires more brain power."