One conversation, followed all the way#
You sit down with your doctor. They tap a tablet and ask, "Mind if my assistant listens in? It writes the notes, so I can look at you instead of the screen." You nod. Most people do.
Twenty minutes later you leave. The note is already in your record, neat and complete. Your doctor saved time and you had their full attention. That part is real, and it is why this technology is spreading so fast.
Now the question almost nobody asks. Where did the conversation go after you left the room?
We picked the clinic because it is the clearest example we know. But look at the shape of it: some input goes in, a model reads it, a record comes out, and copies are kept along the way. That is the shape of nearly every AI system in a business — the invoice reader, the support-ticket router, the contract summariser. The words you will hear about the clinic are the same words you will hear in your next vendor meeting. They just matter more when the input is a person's health.
Why good doctors say yes#
This is not a story about careless people. The pressure is real.
About 48% of US physicians reported at least one symptom of burnout in 2023, and roughly one in five spent more than eight hours a week on the patient record outside working hours, according to the American Medical Association. One large medical group rolled AI scribes out to 7,260 physicians and estimated about 15,791 hours of note-writing saved across 2.5 million visits in just over a year (AMA summary).
The honest counterweight: a study of 1,800 clinicians at five academic centres found a more modest gain — about 16 minutes saved for every eight hours of patient care (STAT, April 2026). Helpful. Not magic.
Either way, the demand is legitimate. That is exactly why the defaults matter.
Where the audio actually goes#
Follow the conversation, one step at a time. Each step is a place where a copy can exist, and a place where someone may be able to read it.
A microphone on a phone or tablet records the visit. The first question is not technical at all: was the patient actually asked, by a person, in a way they could refuse?
The audio is turned into text. This may happen on the device, or the audio may be sent to a server first. Now there are two copies — the sound and the words.
A language model reads the transcript and drafts the note, sometimes with billing codes. It runs somewhere — the vendor's cloud, the hospital's own servers, or the device itself.
The doctor reviews the note and signs it into the electronic health record. That record is very often hosted in someone's data centre too.
The audio, transcript and draft may be kept for a while — for support, for quality checks, or to improve the model. This is the step almost nobody sees, and it is where most of the real answers live.
A recent perspective in npj Digital Medicine put it plainly: how raw audio, transcripts and draft outputs are governed is often unspecified. Health systems still have to decide how long each copy is kept and who can open it. The note is governed. The copies around it often are not.
The words you will hear, in plain English#
Every vendor conversation uses the same vocabulary. Here is what each word actually means, and the one question worth asking when you hear it.
PHI and PII
Protected health information and personally identifiable information — anything that points to a real person. A name, a date of birth, an address, a voice. Ask: which of these fields does the system actually need to do its job?
Ambient AI
A system that listens in the background instead of being switched on for each task. Ask: what tells a person, in the moment, that it is on — and how do they turn it off?
Consent
Permission, given freely, by someone who understood what they agreed to. In about a dozen US states, everyone in a conversation generally has to agree before it is recorded. Ask: who asks — a person, or a checkbox the software fills in?
BAA
A business associate agreement. The contract that lets a vendor handle health data on a provider's behalf under US law. It sets rules and assigns responsibility. Ask: what does it allow the vendor to do with the data, not just what does it forbid?
“Compliant”
A claim that a system follows a set of rules. It is not a certificate, and for most privacy laws there is no official certificate to hold. Ask: compliant with what, checked by whom, and when?
Subprocessor
Another company your vendor uses behind the scenes — a cloud host, a speech-to-text service, a model provider. Your data goes wherever they go. Ask: can I see the full list, and am I told when it changes?
Egress
Data leaving your walls — your building, your network, your cloud account. Ask: which fields leave, where to, and is that written down before the system runs, not after?
In-tenant and on-device
In-tenant means the model runs inside your own cloud account. On-device means it runs on the phone or tablet itself. Both reduce egress. Neither removes it. Ask: after processing, what still leaves, and where does it land?
Retention
How long a copy is kept before it is deleted. Ask: for each copy — audio, transcript, draft — what is the clock, and can we shorten it?
De-identified and pseudonymised
De-identified means details that point to a person have been removed. Pseudonymised means they have been swapped for a code, which someone may still be able to reverse. A voice is hard to de-identify. Ask: which method, and who holds the key?
“Model improvement”
Using your data to make the vendor's AI better — often after de-identification. It is common, and it is often allowed by default. Ask: is my data used for this, and can I say no without losing the service?
Human review
People at the vendor who may read or listen to samples to check quality. Ask: who, where, how many, and is each access logged?
Hallucination and omission
A hallucination is something the model wrote that was never said. An omission is something that was said and never made it in. The second is quieter and often more dangerous. Ask: how do you measure both, on data like ours?
Audit trail
A record of what went in, what came out, who saw it and what happened next — kept so someone can check later. Ask: could we reconstruct yesterday's visit from the record alone?
What “up to 90 days” looks like in practice#
These are not hidden terms. They are published, and they are legal.
One of the largest vendors in this market states in its own documentation that it keeps audio, transcripts and recognized text for up to 90 days, and that data is de-identified or pseudonymised for AI model improvement. Another's published privacy policy reserves de-identified data to refine its algorithms. Enterprise contracts may narrow these terms, which is exactly why the contract is worth reading.
And here is what it looks like when the first step goes wrong. In November 2025 a patient filed a proposed class action against a California health system, alleging that an ambient scribe recorded exam-room conversations without consent — possibly more than 100,000 patients — and that notes stated patients had been advised and had consented when they had not. According to the report, the clinic told the patient the recording could remain on the vendor's servers for about a month. These are allegations, not findings. But the detail worth remembering is the checkbox. Consent became a field the software filled in.
The line to remember
A signed agreement decides who may hold the data and who pays if it leaks. It does not decide that the data stays home, how long it lives, or whether anyone was asked.
Would a small model on the device fix this?#
This is the pitch you will hear next: run a small language model on the hospital's own tablet, and nothing ever leaves. Zero data leakage.
It is half right, and the half matters.
Local closes one door, not all of them. The finished note still lands in a health record that is usually hosted. The small model was trained on someone's notes. Updates, crash logs and usage data still travel. And a recording nobody agreed to is still a recording, even if it never touches the cloud. The California case above was about consent, not about where the server sat.
Small is not the same as weak. In one 2025 study, Llama models between 1 and 8 billion parameters, fine-tuned for medical coding, went from almost 0% to about 99% on matching ICD-10 code descriptions. On real discharge notes the exact-match rate was 69% — and 42% of the misses were cases where the notes simply did not contain the information needed to pick a code. For scale, an untuned GPT-4 matched only about 34% of ICD-10 codes in an earlier benchmark. A narrow, well-trained small model can do a narrow job very well. It still needs a person to check it.
Fine-tuned does not automatically mean fewer made-up answers. Research presented at EMNLP 2024 found that teaching a model facts it did not already know through fine-tuning can increase its tendency to hallucinate. Narrow helps. It does not guarantee.
Where this goes, done well#
The same listening is moving into more rooms. Video consultations, where the audio already travels over the internet. Remote monitoring and care at home, where there may be no one in the room to ask, “Wait, is this on?” For rural and housebound patients this could mean real access to care that was out of reach before.
Each step adds reach. Each step also removes a person who might have noticed something was wrong. That is why the safer defaults are worth deciding now, while the systems are still new:
- A person asks for consent. Software never ticks the box.
- Audio is deleted when the note is signed, not weeks later.
- Processing happens on the device or inside your own walls where it can.
- No training on anyone's data without a separate, clear yes.
- The retention clock is visible to the patient, and they can stop it.
- "Pause" is one word away, and it is honoured.
Small, local models make the middle three cheaper than they have ever been. None of the six needs to wait for a regulator.
Five questions for your next review#
These were written for a clinic. Swap “visit” for “invoice”, “ticket” or “contract”, and they work for almost any AI system in your business.
“Where is the input from yesterday right now, and who has read it?”
If nobody in the room can answer, that is the finding. Everything else on this list is a way of getting to the answer.
“How many copies exist, and what is the clock on each one?”
Audio, transcript, draft, final. Most teams know the answer for the final version only.
“Is our data used to improve the vendor's model, and did we choose that?”
It is often the default. A default is still a decision — just one somebody else made for you.
“Which subprocessors touch it, and would we know if the list changed?”
Your data follows the vendor's vendors. Ask for the list and the notice period in writing.
“When the output is wrong, who notices, and when?”
Name the person and the moment. If you cannot, the review step does not exist yet — however good the model is.
What this does not tell you#
It does not tell you which vendor to choose. The terms quoted here are published and legal, and several vendors offer tighter settings to customers who ask. Some offer them to everyone. The point is not that anyone is hiding something. The point is that the answers sit in documentation that few buyers read, and a signature on an agreement has quietly come to stand in for the reading.
It also does not tell you that AI scribes are a bad idea. They give doctors time back and patients a doctor who looks at them. That is worth protecting — which is exactly why the questions above are worth asking before the next rollout, not after the next lawsuit.
Related
