The hardest part of putting an AI agent near patient data is not the prompt. It is deciding where the boundary sits between your system and the model provider, because that decision fixes your architecture, your logging, your retention policy and your audit trail before you write a line of code.
Most guides on this subject read like a policy summary. They tell you HIPAA has a Privacy Rule and a Security Rule, that you need encryption in transit and at rest, and that you should sign a Business Associate Agreement. All true, and none of it tells you what to actually build.
This page is about the engineering. What counts as protected health information once it is serialised into a request body. Which of the three viable architectures you are choosing between and what each one costs you. Why your error tracker is the most likely place you will leak PHI. What an audit event for a model call has to contain to be worth anything six months later during an investigation. And why an agent that can write to a clinical record is a different class of system from one that can only read.
None of this is legal advice. It is the set of engineering consequences that follow from the rules, written for the person who has to ship the thing. Your compliance officer and your counsel decide what is permitted. You decide whether the code makes that possible.
What counts as PHI once it is inside a request body
PHI is health information that is individually identifiable and held by a covered entity or its business associate. In practice, engineers under-scope it in four predictable ways.
The identifiers are broader than you think. Name, address down to anything finer than the first three digits of a postcode, any date more specific than a year, phone, email, medical record number, account number, device serial number, IP address, full-face photograph, biometrics. Eighteen categories in the Safe Harbor list. A request that contains only "patient DOB 14 March 1981, presenting with chest pain" has no name in it and is still PHI, because the date plus the clinical detail plus your possession of the record makes it identifiable.
Free text carries identifiers you did not put there. This is the failure that catches teams who built a careful redaction layer over structured fields. A clinician's note reads "discussed with her husband Raj, who drove her in from the Adajan clinic". Nobody designed that field to contain a name. It contains two. If you are passing clinical notes to a model, you are passing unstructured identifiers whether you intended to or not.
The response is PHI too. A model summary of a patient's chart is derived health information about an identifiable person. It gets the same treatment as the input: same encryption, same retention rules, same audit trail. Teams routinely protect the request path and then write the response into an unencrypted cache, a websocket log, or a "drafts" table nobody classified.
Tool call arguments are requests. If your agent calls a tool with {"patient_id": "44192", "query": "recent HbA1c"}, that argument object was generated by the model and travels back through the provider. So does the tool result you return. In an agentic loop, every turn is another round trip with the same exposure as the first one, and there may be a dozen of them behind one user action.
The working definition to hold in your head: anything that crosses the network boundary to the provider, in either direction, at any turn of the loop.
The Business Associate Agreement question decides your architecture
If PHI reaches a vendor, that vendor is handling it on your behalf, and the relationship needs a Business Associate Agreement in place before it does. This is not a procurement formality you can run in parallel with the build. It determines which of three architectures you are allowed to have, and they are not small variations on each other.
Do not take anyone's blog post, including this one, as evidence of which vendors will sign one, under which plan, or on which endpoints. Terms change, they differ between a direct API account and the same model served through a cloud provider, and they sometimes differ between features within one account. Ask the vendor in writing, get the executed agreement, and read what it says about subprocessors, retention and abuse monitoring. Then design.
Pathway one: covered, with an agreement in place
PHI goes to the model as it is. You get full clinical context, the best output quality, and the simplest code. You inherit an obligation to configure the account the way the agreement assumes: any retention or human-review setting the vendor offers has to be set correctly, and you need to keep evidence that it is. Your architecture diagram now has a third party inside the trust boundary, which means it belongs in your risk analysis, your subprocessor list and your incident response plan.
Pathway two: de-identify at the boundary
PHI never leaves your infrastructure. A service in your VPC strips or tokenises identifiers, sends the de-identified text out, and rehydrates the response on the way back. You are no longer sending PHI, so the agreement question softens. In exchange you take on the hardest engineering in this document, and you lose some clinical signal, because dates and ages and geography sometimes matter to the answer.
Pathway three: keep the model inside your perimeter
An open-weights model running on infrastructure you control, or a managed endpoint inside your own cloud account with no data egress. No third party sees anything. You now own GPU capacity planning, model updates, evaluation and the quality gap against frontier models, which for hard clinical reasoning is real. This is the right answer more often than people expect for narrow tasks: classification, extraction, coding suggestions, template filling.
| Decision | Covered pathway | De-identify at boundary | Model in your perimeter |
|---|---|---|---|
| PHI leaves your network | Yes, under agreement | No, if de-identification holds | No |
| Hardest engineering | Config discipline and audit | De-identification and rehydration | Serving, updates, evaluation |
| Output quality on hard reasoning | Highest | Slightly degraded by lost context | Depends on the model you can run |
| Biggest failure mode | A logging sink nobody classified | An identifier in free text you missed | Quality drift with nobody watching |
| Blocked until | The agreement is executed | Your redaction passes review | Capacity exists |
One more thing that is easy to miss: the pathway is per feature, not per company. A discharge summary drafting tool and an appointment-reminder rewriter do not need the same boundary. Picking one global answer for the whole product usually means over-engineering the harmless features and under-serving the valuable one.
De-identification is a pipeline with a re-identification problem
Teams choose pathway two because it looks like the cautious option. It is the one with the most code in it.
Safe Harbor de-identification means removing all eighteen identifier categories and having no actual knowledge that the remainder could identify the person. The alternative, expert determination, means a qualified statistician certifies the residual re-identification risk is very small. Safe Harbor is mechanical and strict. Expert determination lets you keep useful things like a date shifted by a consistent offset, and it costs you an expert.
If you go the Safe Harbor route, build the pipeline in four stages and treat each as a separate testable component.
- Structured stripping. Known fields, deterministic. Easy, and the part everyone does.
- Free-text detection. Named entity recognition over notes. A clinical NER model, a rules layer for record numbers and phone formats, and a dictionary of your own staff and facility names. This is where the residual risk lives.
- Tokenisation with a local map. Replace each detected identifier with a stable placeholder,
PT_0001,PROV_0007, and keep the mapping in your own encrypted store, keyed to the request. The map never leaves. This is what lets the model produce coherent text about "the patient" and lets you put the real name back. - Rehydration on the return path. Substitute the real values back into the response before it reaches a human. Test what happens when the model invents a token you never issued, because it will.
Two practical warnings. First, rare attributes re-identify people even with every name gone: a diagnosis with a handful of cases in a region, a very unusual medication combination, a note that mentions an occupation and a district. Free text plus rarity beats field-level redaction. Second, measure your recall on real notes and publish the number internally. A detector that catches 95 percent of identifiers sounds good and means one identifier in twenty leaves your network. Decide out loud whether that is acceptable, rather than discovering the rate after an incident.
Your logs are the most likely place you leak
Almost every PHI exposure in an LLM feature comes from observability, not from the model call. The model call was designed. The logs happened.
Enumerate the sinks before you ship, because the defaults are against you. HTTP access logs that record query strings. Application logs where someone added logger.debug(payload) during a bad afternoon. APM traces that capture request and response bodies as span attributes. The error tracker, which by default attaches local variables to a stack frame, and the frame you crashed in was holding the prompt. LLM observability tools whose entire value proposition is storing prompts and completions, hosted somewhere you have not vetted. CI logs from an integration test that used a real note. The CDN. The load balancer. A queue's dead letter store, holding the full job payload, retained for a fortnight.
Two rules keep this tractable. Redact at the point of construction, not at the point of logging, so that a PHI-bearing object is never handed to a logger in the first place. And log identifiers instead of content: a request id, a patient reference that resolves only inside your own database, a prompt template id, a content hash. You want to be able to reconstruct what happened without the log itself being a medical record.
A model call log line that is safe to keep looks roughly like this.
{
"event": "llm.completion",
"request_id": "01JB8Z6QK4F2",
"actor_id": "user_2291",
"subject_ref": "pt_44192",
"purpose": "draft_visit_summary",
"prompt_template": "visit_summary.v7",
"context_doc_ids": ["note_88421", "lab_20114"],
"model_alias": "clinical-drafting-default",
"input_tokens": 3180,
"output_tokens": 512,
"boundary": "deidentified",
"deid_tokens_issued": 6,
"outcome": "accepted_with_edits",
"latency_ms": 4120
}No clinical content, no names. Enough to answer what the model was asked, on whose behalf, about which record, with which prompt, and what the human did with the answer.
Prompts and responses are records, so retention is a design decision
The moment you store a prompt that contains PHI, you have created a record, and everything that applies to records applies to it. A patient's right of access can extend to information you hold about them. A deletion request has to reach it. Your retention schedule has to cover it. Your backups contain it, which means "we deleted it" needs to be true of the backup rotation as well.
So decide, per feature, what you keep and for how long, and write it down before the table exists:
- Keep the full prompt and response when the output enters the clinical record and you may need to show what the model was given. Then it lives in your encrypted primary store, inside the same access control as the chart, with a defined expiry.
- Keep only the envelope when the output is a draft a human accepts or discards. Metadata, hashes, document ids, the decision. This is the right default for most features.
- Keep nothing beyond the session for ephemeral assistance like a search rephraser.
On the vendor side, ask specifically how long request and response data persists, whether a zero retention configuration exists for your account and endpoints, whether any human review or abuse-monitoring process can see payloads, and where subprocessors sit. Get the answers in the agreement, not from a marketing page, and re-check when you change plans or regions. If a feature depends on prompt caching or any server-side state, that state is storage and it needs the same treatment.
Minimum necessary is a retrieval constraint, not a policy paragraph
The minimum necessary principle says you use and disclose only what the purpose requires. In an LLM feature, the purpose is answering one question, and the amount is however much context your retrieval step decided to stuff into the prompt. Those two numbers are usually very far apart.
Two implementation rules follow, and both are about where the filter runs.
The agent acts as the user, never as the service. The easy build gives the retrieval layer a service account with access to every record, then asks the model to only look at what it needs. That is not access control, it is a request. Pass the caller's identity into retrieval and let the same authorisation rules that guard your API guard context assembly. If a receptionist cannot open a chart in the UI, the agent must not be able to put that chart in a prompt on her behalf.
Filter before assembly, not after. Scope the query, then build the context window from what comes back. Never assemble broadly and rely on instructions to ignore the surplus. A prompt injection in a scanned referral letter does not respect an instruction, and neither does a model having a bad day.
Multi-tenant scoping is the same problem one level up, and it is much cheaper to design in than to retrofit. In Denti360, our own dental practice management product, branch scoping went into the database schema from the first migration rather than being added later, because a multi-branch clinic's staff have different access at different branches. That shape is what a practice system's patient data actually looks like in production: appointments across branches and chairs, treatment plans, billing and receivables, all of it tenant-scoped and role-scoped at the query layer. Any context-assembly step in that kind of system inherits those filters or it is broken, and the same holds across clinical record systems generally.
An agent that can write to a clinical system is a different risk class
A read-only agent that summarises and drafts has a bounded failure mode: it says something wrong, a human reads it, the human catches it or does not. Give the same agent tools that mutate state and you have changed the category of system. A wrong tool call now cancels an appointment, amends a medication list, files a claim, or sends a message to a patient. There is no reviewer between the model and the consequence unless you built one.
Design for that gap explicitly.
- Split your tools by effect and be honest about the labels. Read tools, propose tools, commit tools. Most agents should only get the first two. A propose tool writes a pending object that a human confirms; the commit path is ordinary application code with ordinary authorisation, not a model decision.
- Validate arguments against a schema, then against reality. A well-formed
patient_idis not the same as a patient this user may touch, and a valid dosage field is not the same as a plausible dosage. Server-side checks, every call, as if the arguments came from an untrusted client, because they did. - Make every write idempotent and reversible. Idempotency keys on tool calls, because agentic loops retry. A reverse operation for anything a human might need undone, and an audit event for both directions.
- Cap the blast radius per session. Limits on how many write calls one session may make, which record types it may touch, and a hard stop on anything irreversible or externally visible. Cancelling one appointment is a mistake. Cancelling a hundred is an incident.
- Treat retrieved documents as untrusted input. Text that arrives from a fax, a patient portal message or an uploaded PDF can carry instructions. If that text sits in the same context as tool definitions, you have an injection path into your write tools. This is the single strongest argument for keeping write access out of the agent loop, and it is the part of agentic system design that most deserves a threat model of its own.
Evaluation, and what you do when the model is wrong
Assume it will be wrong. The compliance question is not whether the model errs, it is whether your system notices and whether you can reconstruct what happened.
Build an evaluation set from your own documents, scored by people who know what a correct output looks like on those documents. Public medical benchmarks measure exam-style recall, which tells you little about whether a summariser drops an anticoagulant from a medication list. Score two error types separately, because they behave differently: inventions, which a reviewer reading against the source can spot, and omissions, which look like clean output and are caught only by someone reading the whole source. Omission is the one that hurts in clinical text, and it is invisible in a thumbs-up metric.
Then instrument the human. Every accept, edit and reject is a labelled example arriving for free. A rising edit rate on one prompt template is your earliest signal that something moved, whether the model changed underneath a stable alias, your document mix drifted, or someone tuned a prompt on a Friday. Pin what you can, re-run your evaluation on every model or prompt change, and keep the results somewhere a regulator or a client's security reviewer can see.
Finally, write the incident path before you need it. If a prompt containing PHI landed in a third-party log store, you need to answer within days: which records, whose, over what window, who could have read it. That answer comes from the audit trail described above, or it comes from a forensic reconstruction you will hate. The rest of your healthcare product engineering already assumes breach analysis is possible. Your AI feature should not be the part of the system where it is not.
Putting an AI agent next to protected health information and not sure yet where the boundary should sit? A Scoping Sprint ($2,300, two weeks) ends with the architecture decision made for your case, the data-flow boundary drawn, a clickable prototype, and a fixed quote. Or just start a conversation.
