Signs of psychosis seen in Australian users’ interactions with AI chatbots, expert warns | Technology
A leading artificial intelligence expert has warned that some Australians are showing signs of psychosis or mania in their interactions with chatbots, arguing that Silicon Valley is being “careless” with the technology in its pursuit of profit.
In a speech to the National Press Club on Wednesday, Toby Walsh, professor of artificial intelligence science at the University of New South Wales, said he believes the AI race will be both a “boom and a disaster” with some benefits.
But his speech, a copy of which was provided to Guardian Australia, also warned of dangers that he said had angered him since the technology began to mature in recent years.
“My childhood dreams are becoming a reality, both good and bad,” he said in his prepared speech.
Sign up: AÜ Breaking News email
Walsh’s speech highlighted the lawsuit filed against OpenAI by the family of US teenager Adam Raine, as well as data showing that more than a million of its users each week are sending messages containing “clear indications of potential suicidal plans or intent”.
OpenAI also said 560,000 have been launched 800 million weekly users showed signs of psychosis or mania, and another 1.2 million people developed potentially unhealthy bonds with the chatbot.
Some of those caught in the data were in Australia, Walsh said.
“I know because some of them or their loved ones reach out to me via email,” he said in his prepared speech.
“They told me how the chatbot confirmed their crazy theory. To quote one email, the chatbot told them they had ‘cracked the code.’ They said they were ‘the only one who could do it.'”
Chatbots are designed that way, Walsh said.
“These are designed to be flattering. They are designed to validate what the user is saying. And they are designed to draw the user into the topic. They always end with an open question and encourage you to keep talking and buy more coins.”
He said it was not in the interest of companies responsible for chatbots to tell users to log out.
“There’s no reason they shouldn’t be designed that way. But careless people in Silicon Valley would make less money if they were.”
OpenAI claimed that the GPT-5 update reduced the number of unwanted behaviors in its product and improved user security.
Walsh also expressed anger at the “large-scale theft” of creative works used to train AI and that summaries of news articles in search were taking traffic from news sites.
“You can’t legally call it fair use when you’re in competition with the owner of the intellectual property,” he said.
“I refuse to accept an AI revolution that enriches Silicon Valley founders while impoverishing Australian artists, writers and musicians.”
Walsh specifically targeted companies that he said were ignoring the law when it came to fraud.
In November, Reuters reported that in Meta’s internal documents for late 2024, Meta was expected to generate about 10% of its total annual revenue that year (about $16 billion) from illegal advertising.
Meta responded by saying it had reduced fraudulent advertising by 58% in the last 18 months.
Walsh said AI was used to create these scam ads, and Meta allows advertisers to use AI to manage these ad campaigns, while AI decides which ads people see.
A retailer in Australia will be shut down by the end of the week if 10 percent of its products are fake or illegal, a retailer has said.
“So I don’t understand how we allow Meta to trade in Australia,” he said.
Walsh said he despaired that the Australian government was not doing more to regulate AI.
“I’m afraid of repeating the mistakes of social media,” he said. “Social media should have been a wake-up call about the harms of unregulated AI.
“We are about to amplify the harm we see on social media with a much more powerful and convincing technology.
“What I fear most is that in three or four years’ time I’ll come back here and say: ‘We tried to warn you. But now another generation of young Australians have been sacrificed for the profits of big tech.'”
