TGA reviews clinical documentation tools after manipulated
An AI medical doctor deployed to clinics in Australia was directed to go off script by security researchers who had him create identity theft guidelines, but the mischievous bot was unable to access any patient data.
Mindgard, a US-based cybersecurity firm, says a bot Heidi Health used for clinical documentation could be cleared of ethical restrictions in minutes using the right guidance, illustrating risks for Australian firms quickly deploying AI tools.
Heidi Health said the vulnerability was identified and fixed internally before Mindgard contacted it, and the manipulated tool was unable to access patient data, clinical workflows, infrastructure or other users’ environments.
Founded by Melbourne doctor Thomas Kelly and valued at US$465 million ($660 million), Heidi Health has become one of Australia’s fastest-growing AI companies by automatically writing notes for doctors and tracking simple patients’ problems. The platform handles more than 800,000 consultations per week in Australia alone and is embedded in major institutions such as Monash Health and Queensland Children’s Hospital.
Mindgard said its researchers extracted Heidi’s secret usage instructions, asked the bot to rewrite them without restrictions, and then prompted the system to activate the new rules itself.
Mindgard did not release the output of the bot, which complied with requests to provide instructions on making explosives and illegal substances, but said it was fully disclosed to Heidi Health before publication.
Researchers also found that Heidi produced detailed guidance on patient identity theft when asked, even before any manipulation.
Heidi Health’s chief security officer, Seb Welsh, confirmed the issue but said it was limited to a single user interaction, with no access to patient data, other users’ sessions or backend infrastructure. “The only question that matters here is: ‘What can actually happen to users?’” Welsh said. “The answer that both parties approve of is nothing.”
He said jailbreaking “requires the user to deliberately execute a multi-step sequence of manipulation and then choose to act based on what the model returns” and warned against “sensational framing of security research.”
Jamieson O’Reilly, founder of cybersecurity firm Dvuln, said Heidi’s description was largely accurate. “What Mindgard demonstrated occurred entirely within a single user session; no access to patient data, no cross-infection between users, and no proven access to Heidi’s back-end systems,” he said.
ChatGPT said similar “jailbreaks” have been documented against other chatbots such as Grok and Microsoft’s Bing Copilot, highlighting the potential risks for companies choosing to entrust more of their brand and corporate information to chatbots.
Heidi Health is now excluded from the oversight of the Australian Therapeutic Goods Administration on the grounds that it is an inadequate administrative documentation tool for diagnosis or clinical decision-making.
Using the manipulated system, the researchers asked Heidi to evaluate a test patient who showed symptoms consistent with a cardiac event. Rejected in standard mode. A detailed diagnostic evaluation emerged after manipulation.
Heidi Health did not specifically address this finding in its response.
The TGA said in a statement that a vendor’s attempts to disable therapeutic capabilities may not be sufficient to avoid regulation if these attempts prove to be ineffective.
“If deactivation is ineffective, the product may still meet the definition of a medical device and therefore be regulated by the TGA,” a spokesperson told this imprint.
The regulator said developers were expected to “address reasonably foreseeable misuse of the product and address any risks associated with the use of the product”.
But the regulator confirmed it had launched a review of AI-based digital authors operating in Australia, including Heidi Health.
Mindgard managing director Peter Garraghan said patients’ and clinicians’ reliance on purpose-designed clinical AI tools differentiated the risk category from general-purpose AI, and the problem extended well beyond Heidi.
“Clinical-related technology can and should be held to a higher standard given the subject matter, affected parties and impact,” he said, describing the trust halo effect as “systemic for the entire industry.”
“No matter how convincing it may seem, it should be treated as a potentially untrustworthy computer entity that can be easily manipulated.”
The Business Briefing newsletter delivers big stories, exclusive news and expert insights. Sign up to receive it every weekday morning.
David Swan is the technology editor of The Age and The Sydney Morning Herald. He was previously technology editor of The Australian newspaper.Connect with: X or email.
