Sick and wrong: Ontario auditors find doctors' AI note takers routinely blow basic facts
60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say
Brandon Vigliarolo Brandon Vigliarolo Published thu 14 May 2026 // 21:50 UTCThe AI systems approved for Ontario healthcare providers routinely missed critical details, inserted incorrect information, and hallucinated content that neither patients nor clinicians mentioned, according to a provincial audit of 20 approved vendors’ systems.
The findings come from the Office of the Auditor General of Ontario, Canada, and are included in a larger report about the state of AI usage by public services in the province. They specifically address the AI Scribe program, the Ontario Ministry of Health initiated for physicians, nurse practitioners, and other healthcare professionals across the broader health sector.
As part of the procurement process, officials conducted evaluations using simulated doctor-patient recordings. Medical professionals then reviewed the original recordings alongside the AI-generated notes to evaluate their accuracy.
What they found was, frankly, shocking for anyone concerned about the accuracy of AI in critical situations.
Nine out of 20 AI systems reportedly “fabricated information and made suggestions to patients' treatment plans” that weren’t discussed in the recordings. According to the report, evaluators spotted potentially devastating incorrect information in the sample reports, such as no masses being found, or patients being anxious, even though these things were never discussed in the recordings.
Twelve of the 20 systems evaluated inserted incorrect drug information into patient notes, while 17 of the systems “missed key details about the patients’ mental health issues” that were discussed in the recordings. Six of the systems “missed the patients’ mental health issues fully or partially or were missing key details,” per the report.
OntarioMD, a group that offers support for physicians in adopting new technologies and was involved in the AI Scribe procurement process, has recommended that doctors manually review their AI notes for accuracy, but the report notes there’s no mandatory attestation feature in any of the AI Scribe-approved systems.
Bad evaluations don’t help, either
AI systems making mistakes isn’t exactly shocking. As we’ve reported previously, consumer-focused AI has a tendency to provide bad medical information to users, and some studies have found large language models failed to produce appropriate differential diagnoses in roughly 80 percent of tested cases. But the tools evaluated here are for doctors, not consumers, and such poor performance necessitates explanation. A good portion of the report blames how the systems were evaluated.
According to the report, the weight given to various categories of AI Scribe performances was wonky. While 30 percent of a platform’s evaluation score depended solely on whether they had a domestic presence in Ontario, the accuracy of medical notes contributed only 4 percent to the total score.
Bias controls accounted for only 2 percent of the total evaluation score; threat, risk, and privacy assessments counted for another 2 percent; and SOC 2 Type 2 compliance contributed an additional 4 percentage points.
In other words, criteria tied to accuracy, bias controls, and key security and privacy safeguards made up only a small portion of the total evaluation score for the AI Scribe systems.
“Inaccurate weightings could result in the selection of vendors whose AI tools may produce inaccurate or biased medical records or lack adequate protection to safeguard sensitive personal health information,” the report said of the scoring regime.
The Register reached out to the Ontario Health Ministry for its take on the report, and whether it was going to conform to its recommendations for the AI Scribe program, but we didn’t immediately hear back. A spokesperson for the Ministry told the CBC on Wednesday that more than 5,000 physicians in Ontario are participating in the AI Scribe program and there have been no known reports of patient harms associated with the technology. ®
ai and ml ai + ml ai software canada healthcarePossible Samsung strike puts even more pressure on memory pricing
As a senior policymaker ponders whether all South Koreans should enjoy an ‘AI dividend’
SecurityNobody believes the 'criminals and scumbags' who hacked Canvas really deleted stolen student data
Other than Instructure execs - maybe?
ZTE and Telkom Indonesia sign strategic MoU to accelerate digital solutions and infrastructure development
Strengthening Indonesia’s digital ecosystem through AI, cloud computing, and next-gen connectivity
AI + MLSick and wrong: Ontario auditors find doctors' AI note takers routinely blow basic facts
60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say
AWS to Quick admins: The access control didn't work, but you weren't using it anyway, so what's the problem?
If a setting fails in the forest and nobody hears it ...
OffbeatGrad-to-be turns graduation cap into Rust-powered light show
Eric Park tells us he doesn't plan to wear his modified cap to commencement, but his code's available for anyone with no such qualms and an upcoming ceremony
IBM Cloud evaporates as datacenter loses power
Google users fight for refunds as unauthorized API usage bills soar
Chrome silently installs a 4 GB local LLM on your computer
Anthropic’s bug-hunting Mythos was greatest marketing stunt ever, says cURL creator
HP stuffed a PC into a keyboard. We took it for a spin
From Prompt to Exploit: How LLMs Are Changing API Attacks
Modern applications are API-driven, interconnected, and often over-permissioned, making them an ideal target for AI-assisted attacks.
Identity Resilience: The New Mandate for Cyber Survival
Join Druva experts for a compelling deep dive into what it takes to build an identity-first recovery strategy in this new threat landscape.
Identity Resilience: The New Mandate for Cyber Survival
Join Druva experts for a compelling deep dive into what it takes to build an identity-first recovery strategy in this new threat landscape.
Unfriendly Followers: The Black Market For Your Identity
They’ll reveal how attackers use your profile as intel and show you how to make yourself harder to target
How Agents are Reshaping AI Security
AI adoption is accelerating and with it comes a new security challenge.
How Agents are Reshaping AI Security
AI adoption is accelerating and with it comes a new security challenge.
AI Found the Problem. Now What?
AI is transforming the software development lifecycle, helping teams identify and remediate vulnerabilities before they reach production.
AI
-
AI+ML
Cerebras risked it all on dinner plate-sized AI accelerators a decade ago. Today it’s worth $66 billion
Here's a look at the tech powering the first big IPO of 2026
-
Security
Nobody believes the 'criminals and scumbags' who hacked Canvas really deleted stolen student data
Other than Instructure execs - maybe?
-
AI + ML
Sick and wrong: Ontario auditors find doctors' AI note takers routinely blow basic facts
60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say
-
AI + ML
Anthropic tosses agents into the API billing pool
Limits Claude subscriptions to interactive use
-
Offbeat
Grad-to-be turns graduation cap into Rust-powered light show
Eric Park tells us he doesn't plan to wear his modified cap to commencement, but his code's available for anyone with no such qualms and an upcoming ceremony
Infosec
-
AI+ML
Cerebras risked it all on dinner plate-sized AI accelerators a decade ago. Today it’s worth $66 billion
Here's a look at the tech powering the first big IPO of 2026
-
Security
Nobody believes the 'criminals and scumbags' who hacked Canvas really deleted stolen student data
Other than Instructure execs - maybe?
-
AI + ML
Sick and wrong: Ontario auditors find doctors' AI note takers routinely blow basic facts
60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say
-
AI + ML
Anthropic tosses agents into the API billing pool
Limits Claude subscriptions to interactive use
-
Offbeat
Grad-to-be turns graduation cap into Rust-powered light show
Eric Park tells us he doesn't plan to wear his modified cap to commencement, but his code's available for anyone with no such qualms and an upcoming ceremony
FOSS
-
Cerebras risked it all on dinner plate-sized AI accelerators a decade ago. Today it’s worth $66 billion
Here's a look at the tech powering the first big IPO of 2026
-
Nobody believes the 'criminals and scumbags' who hacked Canvas really deleted stolen student data
Other than Instructure execs - maybe?
-
Sick and wrong: Ontario auditors find doctors' AI note takers routinely blow basic facts
60% of evaluated AI Scribe systems mixed up prescribed drugs in patient notes, auditors say
-
Anthropic tosses agents into the API billing pool
Limits Claude subscriptions to interactive use
-
Grad-to-be turns graduation cap into Rust-powered light show
Eric Park tells us he doesn't plan to wear his modified cap to commencement, but his code's available for anyone with no such qualms and an upcoming ceremony
-
KDE bags €1.3M as Europe realizes it might need an OS of its own
Germany's Sovereign Tech Fund backs the desktop project while public sector interest in homegrown alternatives grows