Demos
RAG quality is decided before a query ever runs. See why chunking, terminology normalization, and de-identification determine whether clinical RAG retrieval is reliable.
Retrieval-augmented generation lets a clinical LLM reason over guidelines, protocols, and patient records instead of relying only on what it memorized in training. The failure mode is hard to spot: a RAG system built on a poorly prepared corpus does not crash; it returns a fluent, plausible answer built on retrieved context that is fragmented, redundant, or matched on the wrong terminology, and the clinician reading it has no way to tell. The work that prevents that outcome- ingestion, de-identification, normalization, and chunking- happens before a single query runs.
The gap gets little attention because it sits below the retrieval and generation layers most RAG discussion covers. It is also, in practice, what decides whether a clinical RAG deployment is reliable enough to trust.
Why do clinical corpora break naive RAG pipelines?
A health system’s RAG corpus is not one kind of document. It is EHR exports, free-text notes in a dozen documentation styles, scanned records that need OCR, DICOM metadata, PDF pathology and radiology reports, and clinical guidelines that mix structured HTML with unstructured PDF. Each format carries different structural cues and different PHI exposure, and a pipeline that treats them all as generic text loses information a clinical retrieval system needs.
The information lost first is provenance. A chunk retrieved without knowing which document it came from, when that document was generated, and what type of document it is cannot be evaluated for currency or authority. Semantic similarity alone cannot tell a current guideline from a superseded one, or an institutional protocol from a general reference, because both can describe the same clinical concept in similar language. Metadata capture at ingestion is what makes that distinction possible at retrieval time. Skip it, and every downstream layer, chunking, embedding, indexing, inherits a corpus that looks uniform but is not.
Clinical RAG document-preparation pipeline: raw clinical corpus flows through ingest and metadata capture, de-identification, terminology normalization, and structure-aware chunking, before embedding and indexing feed retrieval and generation
Why does de-identification have to happen before embedding, not after?
Any RAG corpus built from clinical records carries PHI exposure risk the moment it is indexed into a vector store an LLM inference system can query. De-identifying the surface text is necessary but not sufficient if de-identification runs after embedding, because vector representations are not a one-way transformation. A February 2026 study on embedding inversion found that text embeddings can “expose sensitive attributes or reconstruct raw text,” a risk the authors describe as inherent to how embedding models represent language, not a defect specific to one vendor’s model (Tsai et al., 2026). De-identification belongs in the ingestion pipeline, ahead of the embedding step, not layered on afterward.
It also has to be accurate, and general-purpose LLM APIs applied zero-shot to this task are not consistently accurate enough for regulatory-grade use. An evaluation exercise conducted by John Snow Labs on commercial de-identification APIs on 48 expert-annotated clinical documents across six PHI entity types found Healthcare NLP reaching 96% F1, against 91% for Azure Health Data Services, 83% for AWS Comprehend Medical, and 79% for GPT-4o applied zero-shot. On NAME and DATE entities specifically, the gap widened further, and processing cost per million documents ran $2,418 for Healthcare NLP against $21,400 for GPT-4o (Kocaman et al., 2025). The gap holds against newer tooling, too: John Snow Labs’ August 2026 retest on the same benchmark put today’s frontier LLMs at 0.86–0.91 F1, still trailing Healthcare NLP’s 0.96, and Databricks’ native ai_mask() function at just 0.71 (John Snow Labs, 2026). A RAG pipeline that de-identifies with a general model, or a lakehouse platform’s built-in masking, absorbs both the accuracy gap and the cost gap before the first document is even chunked.
What makes retrieved content comparable across documents?
Clinical text describes the same concept in different surface forms depending on source, specialty, and era: myocardial infarction, MI, heart attack; metformin, glucophage; complete blood count, CBC. A retrieval system matching on embedding similarity alone will retrieve documents using the query’s exact phrasing and miss documents describing the identical concept in different words, because the similarity metric never learns the two forms are equivalent.
Normalizing clinical concepts to SNOMED CT, ICD-10, LOINC, and RxNorm before embedding closes that gap. It does not replace semantic embedding, it supplements it: normalized documents retrieve consistently regardless of which surface form the query happens to use, while embedding still captures conceptual relationships that terminology mapping alone does not encode. The scale of the problem this solves is visible in how badly general-purpose models handle it unassisted. A 2025 evaluation of LLMs mapping medical terms to standard ontology codes found GPT-4o reaching 93.75% precision on the task, while an open-source alternative, Llama 3.3 70B, managed only 19.19% on the same benchmark, both models weakest on multi-parameter clinical measurements (Mavridis et al., 2025). Embedding models trained on general text inherit the same weakness: a May 2025 evaluation found that a medical-domain embedding model sustained strong retrieval across both general and specialized benchmarks, including 74.25 on a medical QA retrieval task, where narrowly domain-adapted alternatives traded away general performance to get there (Selvadurai et al., 2025). Normalization before embedding is what lets a retrieval system treat “MI” and “myocardial infarction” as the same fact rather than two unrelated vectors.
Normalization doesn’t replace embedding — it supplements it. Mapped documents retrieve consistently no matter which surface form the query uses, while general-purpose ontology mapping drops as low as 19% precision on the same task.
Why is chunking the decision that most affects retrieval relevance?
Chunking sets both what can be retrieved and what context the LLM sees once it is. Chunks that are too large dilute relevance by mixing independent concepts; chunks that are too small separate a claim from the qualification that determines whether it applies to the patient in front of the clinician. Fact-lookup queries, a dosage, a diagnostic threshold, want small, isolated chunks. Queries about a clinical scenario or a guideline’s rationale want larger chunks that preserve the surrounding argument. A corpus serving both query types needs hierarchical chunking, not one fixed size applied everywhere.
The scale of the effect is not marginal. A 2026 evaluation of nine chunking methods across multiple retrieval datasets found accuracy at five retrieved chunks ranging from 25.8% to 99% depending on the method and dataset, with the authors’ central finding being that chunk structural coherence, not chunking sophistication or compute cost, is what predicts retrieval quality (Śmigielski et al., 2026). The gap is sharper on the long, multi-section documents that make up most clinical guidelines and full-length research articles. A March 2026 study from the Ottawa Hospital Research Institute compared naive fixed-size chunking against structure-aware chunking on full-text biomedical articles running 5,000 to 8,000 words, and found that content-only chunking retrieved from a single document section on every query it tested (section coverage of 1.0), while structure-aware chunking retrieved from up to 15.6 times more sections of the same document on the same queries (Mortezaagha and Rahgozar, 2026). A guideline’s recommendation and the contraindication qualifying it often live in different sections. A chunking strategy that only ever surfaces one section systematically misses that qualification, and the LLM has no way to know what it was not shown.
Chunking method, not compute cost, decides what the LLM ever sees. Naive fixed-size chunking retrieved from a single document section on every tested query, while structure-aware chunking reached up to 15.6 times more sections of the same documents.
What does a governed clinical RAG pipeline require?
Clinical guidelines get revised, protocols get updated, findings get superseded, and a RAG system that cannot say which version of a document it indexed, or when, will eventually retrieve accurate-at-the-time guidance that is no longer current. Version control has to operate at four layers: the source document, the processing pipeline, the embedding model, and the index itself, because a change at any layer can shift retrieval behavior. Ohio State University Wexner Medical Center runs an NLP pipeline processing more than 200 million Epic clinical notes across seven hospitals, covering de-identification, extraction and coding, and human-in-the-loop validation through Generative AI Lab, on both Azure Databricks and on-premises high-performance computing (Huerta, 2025). Running that consistently across seven facilities with different documentation cultures, and across two infrastructure environments, is what governed reproducibility looks like at production scale, not a one-time validation on a curated sample.
Ohio State’s pipeline processes 200 million-plus Epic notes across seven hospitals, running the identical workflow on both Azure Databricks and on-premises HPC — governed reproducibility at production scale, not a one-time validation on a curated sample.
John Snow Labs builds this document preparation layer on Healthcare NLP and the Terminology Server rather than assembling it from general-purpose components after the fact. De-identification runs on the same models validated at 96% F1 above, ahead of embedding, on-premises or in a private cloud so patient text never leaves the customer’s environment. Entity and assertion recognition anchor chunk boundaries to clinical concepts instead of fixed character counts, and terminology mapping to SNOMED CT, ICD-10, LOINC, and RxNorm runs at ingestion so retrieval is not dependent on which surface form of a term the query happens to use. Generative AI Lab captures the source, version, and document-type metadata that source hierarchy resolution depends on, and keeps the audit trail connecting every indexed chunk back to its origin. For a code walkthrough of the retrieval and generation layers built on top of this foundation, see John Snow Labs’ practical guide to RAG with Spark NLP; for the document preparation layer this article covers, see Healthcare NLP, Generative AI Lab, and de-identification.
For teams evaluating a clinical RAG deployment, the diligence question worth asking a vendor is not “what embedding model do you use.” It is what happens to a document before it reaches that embedding model: how it is de-identified, how its terminology is normalized, how it is chunked, and whether the pipeline can reconstruct exactly what was indexed a year from now. Request an architecture consultation scoped to your own document mix and EHR environment to get answers specific to your corpus rather than a benchmark average.
Frequently asked questions
What is RAG data engineering? RAG data engineering is the set of steps that prepare a document corpus before it is embedded and indexed for retrieval: ingestion and metadata capture, de-identification, terminology normalization, and chunking. Retrieval quality at query time is a direct function of how well these steps were done, more than which embedding model or LLM the system runs on top of them.
Why does de-identification need to happen before embedding rather than after? Text embeddings are not a one-way transformation. A 2026 study on embedding inversion found that vector representations can expose sensitive attributes or allow reconstruction of the original text, which means PHI embedded before de-identification remains a re-identification risk even if the surface text is later scrubbed (Tsai et al., 2026). De-identification has to run at ingestion, ahead of the embedding step.
Does chunk size matter more than which embedding model I use? Both matter, but chunking is the decision teams most often underinvest in relative to its effect. A 2026 evaluation of nine chunking methods found retrieval accuracy ranging from 25.8% to 99% depending on chunking approach and document type, with structural coherence, not method sophistication, predicting the outcome (Śmigielski et al., 2026). On long, multi-section clinical documents, naive fixed-size chunking retrieved from a single document section on every tested query, while structure-aware chunking retrieved from up to 15.6 times more sections of the same documents (Mortezaagha and Rahgozar, 2026).
Can general-purpose LLM embeddings handle clinical terminology well enough for RAG? Not consistently. A 2025 evaluation of LLMs mapping clinical terms to standard ontology codes found GPT-4o reaching 93.75% precision while an open-source general model reached only 19.19% on the same task (Mavridis et al., 2025), a gap wide enough that normalizing terminology before embedding, rather than relying on the embedding model to infer equivalence, materially changes retrieval consistency.
How should a clinical RAG system handle metadata for retrieval? Metadata, source document, date, document type, guideline version, needs to be captured at ingestion and preserved through normalization and indexing, because it is what lets the retrieval layer distinguish a current guideline from a superseded one or a primary source from a summary. Without it, semantic similarity alone cannot apply source-hierarchy rules, and a retrieval system can surface outdated guidance with the same confidence as current guidance.
What is the difference between chunking for fact lookup and chunking for clinical reasoning? Fact-lookup queries, a dosage or a diagnostic threshold, retrieve best from smaller chunks that isolate a single fact and reduce surrounding noise. Queries that require understanding a clinical scenario or a guideline’s rationale retrieve best from larger chunks that preserve the relationships between adjacent content. A corpus serving both query types needs hierarchical chunking rather than one fixed chunk size applied across the whole pipeline.
How is this different from a RAG implementation tutorial? A RAG tutorial typically walks through the retrieval and generation architecture, embeddings, vector store, prompt template, using general-purpose sample text. This article covers the layer before that architecture runs: how the clinical document corpus itself has to be de-identified, normalized, and chunked before it is fit to embed. Both matter, but a RAG system inherits every gap in the underlying corpus regardless of how well the retrieval and generation layers are built.
What production evidence exists for a governed clinical RAG document pipeline? Ohio State University Wexner Medical Center runs a pipeline processing more than 200 million Epic clinical notes across seven hospitals, covering de-identification, extraction, coding, and human-in-the-loop validation, consistently across both Azure Databricks and on-premises HPC environments (Huerta, 2025). Consistency across environments and facilities is the standard a production clinical RAG corpus has to meet, beyond accuracy on a single test set.
