Naming the best model for clinical summarisation is a claim with a shelf life of about four months. The thing worth writing down is the evaluation method, because that is what tells you whether the model in front of you is safe on your documents.
This page used to be a head to head between Claude 3.5 Sonnet and GPT-4o. Both have been superseded. Anthropic's current line runs through Claude 5, with Opus 5, Sonnet 5 and Fable 5.1 alongside Haiku 4.5, and every other provider has moved in the same window. Whatever pair you crown next will be superseded too, so a ranking pinned to two version numbers decays on a schedule you do not control.
The shape of the problem does not decay. Clinical summarisation means taking a discharge summary, a run of consult notes, a lab panel, a referral letter, and producing something a clinician can read in thirty seconds without being misled. The published benchmarks for medical language models are mostly exam-style question answering, MedQA and USMLE style. Those measure recall of medical facts under multiple choice conditions. Summarisation is a different task with a different failure mode, and a high MedQA score tells you close to nothing about whether a model will quietly drop an anticoagulant from a medication list.
So the real work is building your own evaluation, on your own documents, scored by people who know what a good summary of those documents contains. It is a few weeks of unglamorous effort and it outlives every model release.
Why the model-versus-model comparison keeps expiring
Three separate clocks run against a published comparison, and they are not synchronised.
The first is release cadence. New frontier models arrive several times a year and each one resets the ordering, so any conclusion of the form "model A beats model B on clinical text" has to be re-earned. Nobody re-runs a blog post.
The second is silent revision. Providers update models behind a stable API name. Point releases, safety tuning, serving changes and default parameter shifts all land without you redeploying anything. The identifier in your config file is not a guarantee that the behaviour you validated in March is the behaviour you get in September.
The third is your own drift. The documents you feed the model change as your product grows. A summariser validated on cardiology discharge summaries meets its first psychiatric intake note and behaves differently, and no external benchmark warned you.
The response to all three is the same. Own an evaluation set, own a rubric, and re-run it. Then a model swap becomes a measurement instead of an argument.
Accuracy is the wrong headline metric for a summary
"Accuracy" is borrowed from classification, where an output is right or wrong. A summary is neither. It is a selection, and the selection is where the danger lives.
The distinction that matters clinically is between two kinds of error.
Errors of commission are inventions. The model states a diagnosis that is not in the source, attaches a dose to the wrong drug, or asserts a negative finding as a positive one. These are the failures everyone talks about, and they are comparatively easy to catch. A reviewer reading the summary against the source finds a sentence that has no support, and it stands out.
Errors of omission are silences. The model produces a clean, fluent, entirely truthful summary that leaves out the penicillin allergy, the fall two weeks ago, or the fact that the previous clinician was uncertain about the primary diagnosis. Nothing in the output looks wrong. A reviewer scanning the summary sees a coherent clinical story and has no prompt to go looking for the missing piece. Detection requires reading the whole source document, which is precisely the work the summary was supposed to save.
Omission is harder to detect and, in most clinical workflows, worse. It is also the failure mode that automated scoring is least able to see, because a summary missing one critical line still reads as an excellent summary by every surface measure.
A related failure sits next to omission: flattening. The source note says "possible pneumonia, consider repeat imaging." The summary says "pneumonia." Nothing was invented and nothing was dropped, but a hedge became a fact. Score this explicitly, because models are trained toward fluent declarative prose and hedges are exactly what fluent prose removes.
Build the evaluation set before you look at a single model
This is the part teams skip, and it is the part that decides whether the rest is worth anything.
Sample from real documents, across the real range. The instinct is to pick clean, well-structured examples, because they are pleasant to work with. Those are the ones every model handles. Your evaluation set needs the twelve-page transfer summary, the note that is half template boilerplate, the scanned letter that came through OCR with mangled numbers, the record where two clinicians disagree, and the one with a paragraph in another language. Stratify deliberately: short and long, single specialty and mixed, structured and free text, clean and messy. If the messy documents are ten percent of your traffic, they should be more than ten percent of your evaluation set, because that is where the differences between models actually appear.
Have clinicians define what a good summary contains. Not engineers, and not the model. For each document, a clinician who works with that document type writes down the items a correct summary must contain for your specific use case. A summary for a triage queue is not the same as a summary for a specialist handover, and the same source note has two different correct answers. This reference list is your ground truth, and building it is slow. Budget for it honestly.
Score against the rubric, not against a reference text. Automated similarity metrics, ROUGE, BLEU, embedding cosine distance and their relatives, measure whether the generated text resembles a reference string. That is almost useless here. Two summaries can share most of their wording while one of them omits the allergy. A summary phrased completely differently from the reference can be clinically perfect. Similarity scoring rewards phrasing and is blind to selection, which is the only thing you care about.
A model can help with the scoring pass, but only as a first filter that flags candidate omissions for a human to confirm. Treat a model grading a model as triage, never as the result. The clinical judgement calls have to be made by clinicians, and you need at least a subset double scored so you know how much your reviewers agree with each other. If two clinicians disagree about what belongs in the summary, no model was ever going to get it right, and the rubric needs fixing first.
The rubric: what to score, and how each dimension fails
Keep the rubric short enough that a clinician can score a document in a few minutes, and specific enough that two reviewers land in the same place. These are the dimensions worth carrying.
| Dimension | What you are scoring | How it fails quietly |
|---|---|---|
| Factual correctness | Every statement in the summary is supported by the source | Plausible detail invented to fill a gap, usually a dose, a date or a laterality |
| Completeness on management-changing items | Presence of the items the clinician listed as required for this document type | Allergies, anticoagulation, recent falls and pending results dropped from an otherwise clean summary |
| Preserved uncertainty | Hedges, differentials and provisional wording survive into the output | "Query sepsis" becomes "sepsis" and a differential becomes a diagnosis |
| Negation handling | Ruled-out and denied findings stay negative, or are correctly left out | "No chest pain" surfaces as chest pain in a symptom list |
| Attribution and recency | Who said it and when, especially across a multi-visit record | A resolved 2023 problem presented as an active one |
| Scannability | Structure a clinician can read under time pressure | Fluent paragraphs that bury the one line that mattered |
Score completeness and preserved uncertainty separately from factual correctness. If you collapse them into one number, the number will be dominated by commission errors, because those are the easy ones to see, and you will have built a metric that is blind to your worst failure mode.
One more scoring rule: weight by consequence, not by count. Missing a follow-up appointment date and missing an allergy are not the same error. Have the clinicians mark a subset of required items as critical, and report the critical miss rate as its own figure. A model with a good average and one critical miss is not a good model.
Cost per document, not cost per million tokens
Published prices are per million input and output tokens. That is the wrong unit for a budget conversation, because it hides everything that actually drives the bill.
Model the cost of processing one document of each type in your corpus, end to end, then multiply by your real document mix. The drivers below are what separate the modelled number from the invoice.
| Cost driver | Why the token price hides it | What to measure instead |
|---|---|---|
| Document length distribution | Averages are pulled around by a long tail of large records | Cost at the median and at the 95th percentile document, separately |
| Retries and failures | A rejected or malformed output is billed like a good one | Billed calls per accepted summary, not calls attempted |
| Prompt overhead | Instructions, rubric and few-shot examples are resent on every call | Fixed tokens per call before any patient data, and whether caching applies |
| Human correction time | Sits in a payroll line, not the API line | Minutes of clinician review per document, priced at clinician cost |
| Batch versus interactive | Same tokens, different rate and different infrastructure | Which share of volume can tolerate overnight turnaround |
The human correction row is the one that decides most comparisons. If a cheaper model needs an extra two minutes of clinician attention per document, it is not cheaper, and the gap is not close. Clinician time is the most expensive input in the workflow by a wide margin, and any saving on inference that adds review load is a loss dressed up as a saving. Measure review time as part of the evaluation, with the same reviewers scoring the same documents from each model, and put it in the cost model as a real line item.
Batch processing deserves a specific look. Overnight summarisation of yesterday's records is a different economic product from a summary generated while a clinician waits. If most of your volume is the former, you can often use a larger, slower, more careful model at a lower effective rate, and the entire cost argument changes shape. Our guide to healthcare app development costs covers where the surrounding integration and compliance line items usually land, which are typically larger than the inference bill.
Latency matters in one place and almost nowhere else
Latency is worth optimising when a human is sitting there waiting. A clinician who has finished a consultation and wants a draft note before the next patient walks in has a real time budget, and the difference between three seconds and twenty seconds decides whether the feature gets used or bypassed.
Everywhere else it mostly does not matter. A nightly job that summarises the day's discharges has hours of headroom, and so does a queue that prepares handover summaries before a shift change. Optimising latency in those paths trades quality for a property nobody experiences.
Measure the tail rather than the mean. Percentile latency at the 95th and 99th is what users feel, and it is disproportionately driven by your longest documents, which are also the ones where the summary matters most. If your interactive path can time out on a twelve-page record, decide in advance whether it degrades to a partial summary or refuses, and never let it silently return a summary of the first few pages. That is an omission error created by your own infrastructure. Getting this right inside a live clinical workflow rather than a notebook is the substance of production AI and agent engineering.
The data agreement usually decides the choice before quality does
In healthcare, model selection is frequently settled before anyone opens a benchmark, and teams that treat quality as the first filter waste weeks.
The questions that come first:
- Will the provider sign? A business associate agreement under HIPAA, or the equivalent processor terms under GDPR and your local regime. If the answer is no, or only on an enterprise tier you cannot reach, the model is not a candidate at any quality level.
- Where does the data physically go? Region of processing, region of storage, and whether inference can be pinned to a region. UK, EU, Australian and Canadian deployments frequently have hard requirements here, and the default endpoint is often not the compliant one.
- What is retained, and for how long? Zero retention is a contractual commitment, not a marketing line. Abuse monitoring buffers are a real exception worth reading carefully.
- Is your data used for training? Get the commitment in the contract, for the specific tier and endpoint you are calling.
- Subprocessors and audit trail. Who else touches the data, and can you produce a record of what was sent and returned for any given summary when someone asks in two years.
Run this filter first. It usually cuts the field to two or three viable options, and only then does the evaluation set earn its keep, by choosing among models you are actually allowed to use. A model that tops your rubric and cannot be used under your data terms is not a result, it is a distraction. We document the studio's own posture on our security practices page, and a healthcare client should expect the same from every vendor in the chain.
The evaluation is a standing job, and so is the reviewer
Because providers update models underneath you, a one-time evaluation is a snapshot with a decay curve. Treat the evaluation set as production infrastructure. Run it on a schedule, run it on every announced version change, run it before any model swap, and run it when you start ingesting a new document type. Keep the results in version control next to the prompt, so a regression has a date and a diff rather than a rumour.
Watch for drift in the rubric too. As clinicians use the output, their sense of what belongs in a good summary sharpens. That is a signal, not noise. Re-baseline when the rubric changes, and be explicit that scores before and after a rubric revision are not comparable.
Which leaves the last decision, and it is the one that changes what your product legally is. A summariser whose output a qualified clinician reads and signs off is a drafting aid. Remove that step, let the summary flow into a record or a decision without a human in the path, and you have built something that behaves like a clinical decision support tool, with the regulatory character that follows. That is a product and compliance decision, not an engineering optimisation, and it should never happen by accident because someone removed a review queue to cut latency.
Design the review step so it is fast enough that nobody wants to skip it: show the summary next to the source, highlight what the model claims to have taken from where, and make correcting an omission a two-second action. The rest of the surrounding product, the workflow, the audit trail, the integrations, is where most of the build effort in healthcare software development actually goes. We build healthcare software and run Denti360, a practice management product in daily clinical use, and the pattern holds: the model is a small part of the system, and the workflow around it is what makes it safe.
Trying to work out whether a language model is safe on your clinical documents? A Scoping Sprint ($2,300, two weeks) ends with an evaluation rubric and document sampling plan made for your case, a prototype, and a fixed quote. Or just start a conversation.
