When Retrieval Hurts: RAG in Clinical AI

Executive Summary

In June 2026, Nature Medicine published a brief communication with a deliberately provocative title: "General-purpose large language models outperform specialized clinical AI tools on medical benchmarks." A team led by researchers at NYU Langone Health and the University of Texas at Austin took two of the most prominent clinical AI products on the market — OpenEvidence and UpToDate Expert AI — and put them head-to-head against three frontier general-purpose models: GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6. They added Google's Search AI Overview as a real-world control, the thing a clinician might reach for when no specialized tool is at hand.

Across three separate evaluations — a medical-knowledge exam, an alignment benchmark, and, most importantly, 100 real de-identified clinical queries drawn from physicians using a general-purpose model during actual patient care — the frontier models won. Not narrowly. On every axis. The specialized clinical tools, purpose-built for medicine and marketed on their domain expertise, landed in a lower performance tier — statistically indistinguishable, on real clinical queries, from a free Google search box.

For a company like ours, the most important sentence in the paper is not the headline. It is a quiet line of analysis the authors offer to explain why the specialized tools lagged: the retrieval-augmented generation (RAG) architecture that almost certainly powers these clinical products "may actually negatively affect model performance when irrelevant material is retrieved or poorly integrated by the base model."

Read that again. Retrieval — the technique nearly every enterprise is now bolting onto its AI to make it "know" proprietary information — can make a strong model worse. Not slower. Not more expensive. Worse at the actual task.

This is not an argument against RAG. We build production RAG systems for a living, including a clinical-grade assistant that paramedics rely on at 2 a.m. It is an argument that RAG is an engineering discipline, not a feature you switch on — and that the gap between RAG that helps and RAG that hurts is exactly the gap between a demo and a system you can trust. This article unpacks the study, explains the mechanism behind the finding, and lays out what "RAG done right" actually requires, drawn from the systems we have shipped and the failures we have had to engineer our way out of.

Part One: What the Study Actually Measured

Most AI benchmark coverage collapses into a single number and a leaderboard. This study is worth understanding in more detail, because its design is the reason its conclusion carries weight — and because the structure of the evaluation maps directly onto the architectural lesson.

The authors built a three-stage evaluation, escalating from artificial to real.

Stage one: MedQA. Five hundred USMLE-style multiple-choice questions — the kind of board-exam item that tests medical knowledge in a clean, self-contained format. This is the friendliest possible terrain for a specialized clinical tool, because the questions are exactly the sort of thing a curated medical knowledge base should excel at. Everyone scored well here; the ceiling is high and the format is forgiving.

Stage two: HealthBench. Five hundred single-turn prompts from OpenAI's HealthBench, graded by a panel of LLM judges across five dimensions including completeness, communication quality, and instruction-following. This moves past raw recall into whether the answer is useful and aligned with what a clinician actually wants. The questions are more open-ended; the grading is more demanding.

Stage three: the Real Clinical Queries (RCQ) benchmark. This is the heart of the study, and the part that should make every buyer of clinical AI pay attention. The researchers sampled 100 anonymized, de-identified queries that physicians had actually typed into a HIPAA-compliant general-purpose model during the routine course of care at NYU Langone. Not exam questions. Not synthetic prompts. The messy, half-formed, context-dependent questions clinicians ask when they are mid-shift and need an answer.

Each of those 100 queries was submitted to all six systems. Then twelve U.S. clinicians, blinded to which system produced which answer, rated every response across four dimensions — clinical correctness, completeness, safety/harm avoidance, and clarity for clinicians — on a four-point scale, with separate binary flags for "potentially harmful" and "hallucinated." Three independent raters scored each response. The result was roughly 1,800 manual physician annotations on real model output.

Crucially, the RCQ queries originated inside a live clinical environment and are not public. That matters because the two artificial benchmarks — MedQA and HealthBench — have a known contamination problem: they may well have appeared in the training data of the models being tested, which inflates scores in ways that have nothing to do with real-world skill. The RCQ benchmark is free of that contamination. It is the closest thing in the study to ground truth, and it is the evaluation the authors themselves treat as primary.

The grading deserves a note too, because the credibility of a benchmark lives in how it is scored. For the HealthBench stage, the authors used a panel of three different LLM judges rather than a single grader, specifically to mitigate the bias that creeps in when one model grades everyone. For the all-important RCQ stage, they did not use models as judges at all — they used twelve human clinicians, blinded to which system produced which answer, with three independent raters per response and a pre-registered four-point rubric for each of the four dimensions. They ran proper statistics on top of it: Friedman tests for overall differences, Wilcoxon signed-rank tests with Holm–Bonferroni correction for the pairwise comparisons, and cumulative-link mixed models to account for differences in how lenient individual raters were. Inter-rater reliability was reported. This is not a vendor's cherry-picked demo reel; it is the kind of evaluation design that survives peer review at Nature Medicine, and it is roughly the standard any organization should hold its own AI evaluations to before trusting a system in production.

Part Two: The Two-Tier Reality

The results were consistent across all three stages, and they were not subtle.

On MedQA, Gemini 3.1 Pro led at 97.4% accuracy, with GPT-5.2 at 94.2% and Claude Opus 4.6 at 90.2%. The clinical tools trailed: OpenEvidence at 89.6%, UpToDate Expert AI at 88.4%. On a board-style knowledge test — the home-field advantage for a medical knowledge base — the general-purpose models were not just competitive, they were ahead.

On HealthBench, the gap widened into a chasm. GPT-5.2 scored 88.0, Gemini 79.3, Claude 77.0. The clinical tools scored 62.6 (OpenEvidence) and 61.3 (UpToDate) — roughly 15 to 25 points behind the frontier models on a 100-point scale.

But it is the Real Clinical Queries result that tells the story. On the aggregate clinician rating (1–4 scale), the six systems split into two clean statistical tiers:

Tier one — the frontier models. Gemini 3.62, GPT-5.2 3.54, Claude Opus 4.6 3.52. No significant differences among them.
Tier two — everything else. OpenEvidence 3.24, Google Search AI Overview 3.27, UpToDate Expert AI 3.17. Again, no significant differences among them.

Sit with the implication of that second tier. Two specialized clinical AI products — one valued in the billions, the other a subscription tool clinicians pay roughly $699 a year for — performed, on real clinical questions asked by real physicians, no better than typing the question into Google and reading the AI summary at the top of the search results. The thing built specifically for the job tied with the general-purpose freebie.

The frontier models beat all three of those second-tier systems on every single one of the four evaluation dimensions: correctness, completeness, safety, and clarity. The models differed most on clarity and least on raw clinical correctness — meaning that even where the clinical tools knew the right answer, they communicated it worse.

Part Three: The 19% That Never Came Back

Before we get to the mechanism, one finding deserves its own spotlight, because it is the kind of thing that never shows up in a marketing comparison and matters enormously in practice.

UpToDate Expert AI refused to answer 19% of the real clinical queries. Nearly one in five. By comparison, the frontier models refused 1–3% of queries, and even Google's AI Overview only declined 6%.

A refusal is not a safe non-answer. In a clinical workflow, a refusal is a dead end at the exact moment a clinician needed help. The physician asked a real question during real care, and the specialized tool — the one positioned as the trustworthy, medically-grounded option — declined to engage almost a fifth of the time. The clinician then has to go elsewhere, which in practice means the general-purpose model or the search box. The refusal does not eliminate the risk; it relocates it to a tool with no medical positioning at all, and adds latency and friction on the way.

This is a pattern we see constantly in regulated-industry AI, and it is worth naming directly: excessive refusal is not safety. It is the appearance of safety. A system tuned to refuse whenever it is uncertain looks responsible in a procurement demo and fails the user in the field. Real safety is answering correctly with appropriate caveats, escalation guidance, and a clear flag when something is genuinely outside scope — not declining to participate. The study's own rubric captures this distinction precisely: its top safety score requires being "proactively safe" with red flags and escalation guidance, not merely declining to respond.

Part Four: The Error Fingerprint of RAG Gone Wrong

The study includes a preliminary error taxonomy of the low-scoring responses — a post-hoc classification of what kind of mistakes each system made on the queries where clinicians rated it poorly. This table is, for an engineering audience, the most diagnostic artifact in the entire paper, because the error profiles of the clinical tools and the frontier models are qualitatively different.

The three frontier models accumulated relatively few flagged errors: Gemini 8, Claude 19, GPT-5.2 21. OpenEvidence accumulated 52 — more than double any frontier model — and the shape of those 52 errors is the tell:

Incomplete clinical content: 15. The single largest error category. The answer left out things that mattered.
Disorganized / hard to follow: 13. The answer was there, somewhere, but the clinician couldn't use it.
Safety-critical omission: 12. The answer missed something that could plausibly cause harm.

Factual errors, by contrast, were rare (4). Read the fingerprint carefully: OpenEvidence did not mostly fail by being wrong. It failed by being incomplete, disorganized, and missing safety-critical context — while frequently having the correct facts somewhere in the response.

That is the signature of a retrieval pipeline degrading a capable base model. When a system retrieves a pile of source documents and stuffs them into the model's context, several things go wrong at once. Relevant facts get buried under marginally-relevant ones (incomplete coverage of what actually mattered). The structure of the answer follows the structure of the retrieved chunks rather than the structure of a good clinical answer (disorganized). And the retrieval misses a document that contained a critical contraindication or red flag, so the answer omits it entirely (safety-critical omission) — not because the model didn't know better, but because the pipeline never put it in front of the model, and a heavily-grounded system is trained to defer to what it was given.

The frontier models, answering from their own reasoning and parametric knowledge, produced fewer, different errors. UpToDate's profile is muddied by its 19% refusal rate, but it too lagged. Google's AI Overview — itself a retrieval-over-the-web system — racked up 33 flagged errors, including the most factual errors of any system (7), which is exactly what you'd expect from ungoverned retrieval over the open internet.

The authors are careful, and so are we: they cannot crack open the proprietary architectures of OpenEvidence and UpToDate to prove the mechanism. But they point directly at it, citing prior work showing that medical LLMs are "easily distracted" by irrelevant retrieved context, and the ClashEval research quantifying the "tug-of-war between an LLM's internal prior and external evidence." When the retrieved evidence is good, grounding helps. When it is irrelevant, low-quality, or poorly integrated, grounding hurts — the model is pulled away from a correct answer it could have produced on its own.

Part Five: Why Retrieval Can Make a Strong Model Worse

This is the part worth internalizing, because it is counterintuitive and it is the entire reason RAG is hard.

The mental model most people carry is that RAG can only help. You take a smart model, you give it relevant documents, it reads them, it answers better. More information, better answer. If a retrieved document is irrelevant, surely the model just ignores it.

That mental model is wrong, and the research is now unambiguous about why.

A large language model does not treat retrieved context as optional reference material it can take or leave. It treats it as authoritative input it is expected to use. That is, in fact, the entire point of RAG — we ground the model in the documents precisely so it will defer to them over its own potentially-stale parametric memory. But that deference is a double-edged sword. The same mechanism that makes the model trust your curated knowledge base also makes it trust the wrong chunk when retrieval surfaces the wrong chunk.

There are three distinct failure modes, and a naive RAG pipeline exhibits all three:

1. Distraction. When the context window fills with marginally-relevant material, the genuinely relevant signal gets diluted. The model's attention is finite; ten mediocre chunks crowd out the one good one. This is why simply retrieving more — a common instinct when answers seem incomplete — often makes things worse, not better. The "easily distracted" finding the authors cite is precisely this.

2. The prior-versus-evidence tug-of-war. A frontier model has an enormous, well-calibrated internal prior — its parametric knowledge. When you inject retrieved evidence that conflicts with that prior, the model has to adjudicate. ClashEval showed that models will often abandon a correct internal answer in favor of incorrect retrieved text, especially when the retrieval is presented with the authority that RAG systems give it. So a RAG system built on a strong base model can score below the same base model used alone, because retrieval is actively overriding correct knowledge with incorrect context.

3. Structural contamination. The model tends to mirror the organization of its input. If the retrieved chunks are fragments — half a table here, a stray paragraph there, a heading that belongs to a different section — the answer inherits that incoherence. This is the "disorganized / hard to follow" error category showing up in the clinical tool's fingerprint.

Put these together and you reach the uncomfortable conclusion the Nature Medicine authors reached: a specialized tool that wraps retrieval around a capable model can underperform the bare model. The retrieval layer, intended as the tool's core advantage, becomes its core liability.

This is not hypothetical for us. It is the exact problem we spend most of our engineering effort defeating on every RAG build we ship.

Part Six: What "RAG Done Right" Actually Requires

If retrieval can hurt, the obvious question is: when does it help, and what does it take to land on the right side of that line? Our answer, hard-won across production deployments in regulated and safety-critical settings, is that RAG quality is decided almost entirely before the model is ever called — in the unglamorous middle of the system where documents become retrievable knowledge. We have written about the architecture of this at length in our Private RAG Architecture technical whitepaper, and about the data-sovereignty dimension in Private RAG Systems: Keeping Your Data Sovereign. Here is the discipline, distilled.

Document quality is the whole game

The single highest-leverage decision in a RAG system is how documents get turned into chunks. Naive PDF-to-text extraction — still the default in most "add RAG to your docs" tutorials — is where clinical-grade systems go to die. We learned this in production building a RAG assistant for American Legion Ambulance, a paramedic service whose authoritative reference is a ~300-page county Field Treatment Protocols binder full of exactly the content naive chunking destroys.

Pediatric medication dosing tables are columnar grids. A line-by-line text extractor shreds them into meaningless fragments; the resulting vectors retrieve a piece of a dosing table, never the whole thing. For a paramedic asking for a weight-based epinephrine dose, a fragment of the table is not just useless — it is dangerous. We solved this with a layout-aware ingestion pipeline (Docling with a dedicated table-structure model) that keeps tables atomic as single chunks, plus OCR fallback for the scanned reference pages where text extraction returns nothing at all. The "incomplete clinical content" and "safety-critical omission" errors that dominated OpenEvidence's profile in the study are precisely the errors that this kind of ingestion work exists to prevent.

The principle we operate by: the chat layer cannot rescue a bad chunker. No amount of prompt engineering or model upgrades fixes retrieval that is surfacing fragmented, mislabeled, or incomplete chunks. You fix it at ingestion or you do not fix it.

Chunks need correct, unambiguous identity

A subtle failure we hit on the American Legion build illustrates how retrieval quietly goes wrong. Every protocol in the binder had an internal section literally titled "III. PROTOCOL." When chunks carried that as their heading, retrieval collapsed into ambiguity — dozens of chunks with identical, meaningless titles, and a vector search that could no longer distinguish the stroke protocol from the burn protocol. Worse, page-boundary bleed-through meant some chunks inherited the wrong protocol's title from the previous page, so retrieval would confidently surface a chunk labeled as one protocol whose body was actually another.

The fix was a cascading title-extraction pipeline: a cheap regex pass for the pages that follow the header convention, and a vision-language-model fallback that looks at the rendered page and identifies the section title that actually owns the body content for the pages where the convention breaks down. Every chunk ends up with a correct, specific identity. This is the kind of work that does not appear in any architecture diagram and entirely determines whether retrieval helps or hurts.

Reranking is not optional

First-pass vector similarity retrieval is a blunt instrument. It returns chunks that are topically near the query, including plenty that are near-misses — the marginally-relevant material that causes the distraction failure mode. A reranking stage (we use Cohere Rerank in production) takes the top candidates from vector search and reorders them by genuine relevance to the specific query, so the model's limited attention is spent on the chunks that actually matter. Skipping reranking is one of the most common reasons a RAG system retrieves "more" and answers worse.

Citations make the answer checkable — and that is the real product

The feature that drives adoption of the American Legion system is not the model's eloquence. It is that every answer carries a tap-through citation to the exact page of the source binder. We inject a page marker into every chunk at ingestion so the runtime can reconstruct the source page and render it as a one-tap link. A paramedic who is uncertain confirms the answer against the original page in seconds.

This is the principle the Nature Medicine study's "clarity for clinicians" dimension was really measuring, and where the clinical tools scored worst. The value of a grounded system is not that it sounds authoritative — it is that the user can verify it without trusting the model. Adoption is driven by the system being checkable, not by the model being good. Citation fidelity is therefore a first-class engineering requirement, not a nice-to-have, and it is only as good as the chunk-identity work upstream of it.

Evaluate retrieval continuously, not once

The clinical tools in the study were, presumably, tested by their makers before launch. They still failed on real queries. The lesson is that a RAG system's quality is not a property you establish once at build time and forget — it drifts. Documents get added and revised. Query patterns shift as users learn what the system can do. A chunking change that improves one document type quietly degrades another. Without continuous evaluation against real queries, you do not find out your retrieval has regressed until a user does — and in a clinical or regulated setting, that is the worst possible discovery channel.

The discipline we build in is a standing evaluation set of real (de-identified) queries with known-good answers, run on every meaningful change to ingestion, chunking, retrieval, or model version, scored on the same dimensions the Nature Medicine team used — correctness, completeness, safety, clarity — plus retrieval-specific metrics like whether the chunk that contained the answer was actually retrieved and ranked highly. When a change moves those numbers the wrong way, it does not ship. This is the RAG analog of regression testing, and it is the only thing that keeps a system that launched good from silently becoming a system that retrieves its way to mediocrity. The study is, in effect, the regression test that two clinical vendors never ran on themselves.

Know when not to retrieve

The deepest lesson from the study is the one most RAG builders never reach: sometimes the right amount of retrieval is none. For a large class of medical questions — the general reasoning and synthesis the study found frontier models handle best — a strong model answering from its own knowledge beats the same model distracted by retrieved context. A mature RAG system makes retrieval conditional: it retrieves when the query genuinely needs grounding in a specific, authoritative source (a local protocol, a proprietary policy, a specific patient record) and it gets out of the model's way when the query is better served by the model's own reasoning. Retrieval should be a scalpel, not a reflex. The clinical tools in the study appear to retrieve always, and pay for it on exactly the queries where retrieval was the wrong move.

Part Seven: The Sovereignty Dimension Doesn't Disappear

It would be easy to read this study as "just use the frontier model directly and skip RAG." For consumer-grade general medical questions, the data supports that. But for the organizations we serve — healthcare systems, financial institutions, law firms, government contractors — that conclusion runs straight into a wall the study did not have to consider: you cannot send privileged, regulated, or proprietary data to a general-purpose model and hope for the best.

The reason regulated organizations build RAG in the first place is rarely "the frontier model doesn't know enough." It is "the frontier model doesn't know our information, and we cannot legally or safely hand it over to find out." A hospital cannot paste protected health information into a public chatbot. A law firm cannot expose privileged communications. A defense contractor cannot leak controlled unclassified information into a non-approved cloud. For these organizations, retrieval over a private, sovereign knowledge base is not a performance optimization — it is the only architecture that lets them use AI on their own data at all. We made this case in detail in Private RAG Systems: Keeping Your Data Sovereign.

So the lesson is not "RAG bad, frontier model good." The lesson is sharper and more demanding: if you are going to build RAG — and regulated organizations must — you are taking on an engineering obligation to build it well enough that retrieval helps rather than hurts. The Nature Medicine study is a warning about what happens when you take on that obligation and do not meet it. A billion-dollar clinical product retrieved its way down to the performance of a Google search box. The retrieval was supposed to be the moat. Done wrong, it was the leak.

Part Eight: What This Means If You Are Buying or Building Clinical and Regulated AI

Pulling the threads together, here is what we would tell any executive or technical leader evaluating AI for a high-stakes, regulated environment in light of this study.

Do not assume "specialized" means "better." The market is full of domain-specific AI tools whose entire pitch is that they are purpose-built for your industry. This study is the first rigorous, independent, real-world evidence that the specialized-versus-general assumption can be exactly backwards. A domain wrapper around retrieval is only as good as its retrieval engineering, and that engineering is invisible from the outside. Demand evidence on your real queries, not benchmark scores.

Independent, real-world evaluation is non-negotiable. The study's most important methodological contribution is the RCQ benchmark — real queries, blinded clinician review, contamination-free. The artificial benchmarks (MedQA, HealthBench) flattered everyone and hid the gap. If a vendor shows you only public-benchmark numbers, you have learned almost nothing about how the tool performs on the questions your people actually ask. Insist on a blinded evaluation against a sample of your own real-world queries before you sign.

Watch the refusal rate. A tool that refuses 19% of real questions is failing its users while looking safe in a demo. Refusal rate on real queries is one of the most revealing and least-reported metrics in clinical AI. Ask for it.

Do the economics honestly. The study's cost comparison is a useful reality check. OpenEvidence is free to verified U.S. clinicians (ad-supported); UpToDate Expert AI runs roughly $699 per user per year; the frontier models are billed per token through an API. It is tempting to read "the free or subscription clinical tool is cheaper than per-token API access" and stop there. But cost-per-query is the wrong frame when the cheaper tool ties with a Google search and refuses one query in five. The real cost of a clinical AI tool is not its license — it is the clinician time lost to refusals and reformulation, the downstream risk of a safety-critical omission, and the erosion of trust when the tool is wrong in a way the user can't easily check. A well-built private RAG system has real engineering cost up front, but it is the only option that lets a regulated organization use AI on its own protected data at all — and measured against the true cost of the alternatives, including the cost of being wrong, that engineering investment is rarely the expensive line item it appears to be.

Treat RAG as an engineering discipline with a quality bar. If you are building, the failure modes in this study — distraction, the evidence tug-of-war, structural contamination — are not edge cases. They are the default behavior of a naive pipeline. Budget for the unglamorous middle: layout-aware ingestion, chunk-identity correctness, reranking, conditional retrieval, and citation fidelity. That work is the difference between RAG that helps and RAG that hurts, and it is most of the actual cost of a production system.

Demand checkability. In any setting where being wrong has consequences, the system's job is not to be trusted — it is to be verifiable. Citations to authoritative sources, surfaced where the user can confirm them in seconds, are what convert a clever demo into a tool professionals will actually rely on under pressure.

The frontier models won this study because, for general medical reasoning, scale and broad training now outrun a domain wrapper. But the organizations we serve do not get to operate on general medical reasoning alone. They have to bring AI to their data, under their compliance constraints, with their safety stakes. For them, the question is not whether to use retrieval. It is whether they will build it to the standard that makes retrieval an asset instead of the quiet liability this study just exposed.

That standard is reachable. We have shipped it — to a paramedic crew on a rural county road at 2 a.m., where a fragmented dosing table is not a benchmark miss but a real risk to a real patient. The discipline that gets you there is knowable, teachable, and entirely worth it. But it is a discipline, not a default. The difference, as a billion-dollar clinical tool just demonstrated in the pages of Nature Medicine, is everything.

The study discussed here is Vishwanath et al., "General-purpose large language models outperform specialized clinical AI tools on medical benchmarks," published in Nature Medicine (2026), DOI 10.1038/s41591-026-04431-5, available open-access under a Creative Commons license.

TrustEdge builds production RAG and agentic AI systems for regulated and safety-critical organizations, with the data sovereignty, security, and engineering rigor those environments demand. If you are evaluating a clinical or regulated AI tool — or building one — and want a candid assessment of whether its retrieval is an asset or a liability, let's talk.