Skip to main content

Ingest external FHIR documents without patient linkage

Upload FHIR files, run an ingestion job that assigns no patient and creates no OMOP person record, and group the ingested documents into a reusable dataset. This page ends where the dataset is created and handed to a document-level workflow: curation, information extraction, or de-identification. It does not cover patient matching or OMOP linkage.

Video walkthroughs​

Example 1 is the standard walkthrough, from upload through ingestion to the finished dataset.

Ingest External FHIR Documents Without Patient Linkage (example 1)

Example 2 starts from a loose set of standalone FHIR files with no surrounding context, and shows the same workflow applied to them.

Ingest External FHIR Documents Without Patient Linkage (example 2)

Both videos have no audio. They are visual walkthroughs of the screen.

Who this is for​

Anyone who receives FHIR files from outside the organization and has to do something useful with them before anyone decides whose records they are. That includes interoperability and integration engineers who set up the ingestion job, data engineers and platform administrators who own pipeline configuration, privacy and compliance teams running de-identification before wider release, clinical data curation and abstraction teams working document by document, and research or evaluation teams assessing a partner feed or a vendor sample.

Why it matters​

FHIR bundles arrive from partners, health information exchanges, and vendor evaluations. They cover people who are not in your OMOP database, or they carry no patient identifier you would trust: a Patient resource with a local MRN from another health system, a bundle stripped of identifiers before it was sent, a sample feed built from several sites at once. The clinical content is still worth reading. A DiagnosticReport carrying a pathology report, or a DocumentReference wrapping a discharge summary, has the staging or the medication history you want to look at.

Forcing those files into the patient layer to get them ingested is the expensive mistake. Each bundle that cannot be matched creates a new person record, and every one of those is a record you later have to unwind. That cost grows the longer it sits, because cohorts, care gap results, and exports all build on top of the person table. Unwinding a hundred speculative person records after downstream work depends on them is a much larger job than never creating them.

Document-first ingestion keeps the two problems apart. The documents become processable immediately, the patient layer stays clean, and the decision about whether these people belong in your OMOP database stays open until you have looked at the content.

What you gain​

A set of FHIR documents that are ingested, validated, countable, and traceable back to the job that loaded them, with no person records invented to hold them. From there you get a reusable dataset built on a document filter rather than a cohort, which any of the document-level workflows can consume.

What you do not get is patient-level consolidation. Two bundles describing the same person stay two independent sets of documents, and extraction results are per document, not per patient. That is the trade you are making on purpose, and it is the right one when the identifier is not reliable enough to merge on.

Patient Journey Intelligence structures these documents and surfaces the evidence behind every extracted fact. Any clinical conclusion drawn from them stays with the clinicians and reviewers reading the results.

Before you start​

  • Valid FHIR files, prepared and readable. Malformed files surface as validation errors at upload, and files that fail validation do not reach the dataset
  • Confirmation that the file format is supported by the ingestion configuration you plan to select. The same configuration also accepts HL7 and CDA files; for CDA, follow Ingest external CDA documents without patient linkage instead
  • An appropriate pipeline or template available in your deployment. Which pipelines and templates exist depends on how your instance is configured
  • For De-Identification, a profile compatible with the document format. Without one, the job has nothing to run against
  • Permission to create ingestion jobs and datasets. Missing permissions stop the workflow at Step 1 or Step 4

Step 1: Upload the external FHIR files​

  1. In the left navigation, go to Data Integration > Ingestion Jobs.
  2. Select Add Ingestion.
  3. Choose Local Upload.
  4. Upload the FHIR files.
  5. Wait for file validation to complete.
  6. Review validation warnings or errors before continuing.
  7. Continue without assigning the documents to an existing or new patient.

The same upload path accepts HL7 and CDA files, so a mixed batch is possible, but keep the batch to one format when you want the ingested document count to be easy to reconcile.

Step 2: Configure document-first ingestion​

  1. Enter a clear ingestion job name and description. Name it so that someone reviewing the audit trail in six months can tell which partner feed or evaluation batch it came from.
  2. Select an ingestion purpose or configuration that supports external documents.
  3. Select or create the appropriate pipeline configuration.
  4. Enable the processing steps required for the intended workflow.
  5. Leave patient- and OMOP-dependent steps disabled unless the documents will later be linked. This is the step that keeps the person table clean.
  6. Review detected document metadata and format information.
  7. Continue to Review & Start.
  8. Verify that patient assignment is not required.
  9. Select Start Ingestion.

Step 3: Validate the ingestion result​

  1. Monitor the job from the Ingestion Jobs page.
  2. Open the job details page after processing completes.
  3. Confirm that the expected number of documents was ingested.
  4. Review the document-ingestion step, its logs, and its validation results.
  5. Confirm that the documents are available even though no patient or OMOP record was assigned.
  6. Resolve failed or unsupported documents before creating the dataset. A file that failed here will not appear in the dataset, and the gap is easier to explain now than after extraction has run.

Step 4: Create a dataset from the ingested documents​

  1. In the left navigation, go to Data Curation Studio > Dataset Builder.
  2. Select Source Document Filter as the dataset mode.
  3. Select Unstructured, or the applicable document data type.
  4. Select the completed external-document ingestion job.
  5. Apply document filters if needed. Available filters include file name, document type, source, ingestion job, and document metadata.
  6. Review the document preview and the total document count.
  7. Confirm that the dataset does not require a cohort or patient selection.
  8. Enter a dataset name and description.
  9. Review the dataset configuration.
  10. Select Create Dataset.

Step 5: Use the dataset in a downstream workflow​

Once the dataset exists, it behaves like any other dataset in the document-level workflows. Pick the one that matches what you need from these files.

Data curation​

  1. Open Data Curation Studio > Curation Jobs.
  2. Start a new curation workflow.
  3. Select the external-document dataset.
  4. Use document-level extraction, since patient-level consolidation is not available for these documents.
  5. Review the extracted results and the linked source evidence.

Information extraction​

  1. Open Information Extraction.
  2. Create a new extraction job.
  3. Select the external-document dataset.
  4. Configure the required extraction pipeline.
  5. Review the extracted entities, attributes, and evidence.

De-identification​

  1. Open De-Identification.
  2. Start a new de-identification job.
  3. Select the external-document dataset.
  4. Select a profile compatible with the document format.
  5. Run the job and review the detected PHI.
  6. Review and export the de-identified results according to your governance requirements.

Expected result​

Your FHIR documents are ingested without patient assignment, and no new patient or OMOP person record was required to hold them. The documents remain available through their ingestion lineage, a reusable dataset sits directly on the ingestion output, and that dataset can be used for document-level curation, information extraction, or de-identification.

Confirm the ingestion is correct​

Check each of these before the dataset goes anywhere downstream.

  • The uploaded file count matches the ingested document count
  • Unsupported or failed files are clearly identified
  • No unintended patient or OMOP records were created
  • The dataset references the correct ingestion job
  • Dataset preview counts match the intended document scope
  • The selected downstream workflow supports the document format
  • The original ingestion and dataset lineage remain available for auditing

Limitations​

  • Processing is document-level only. Facts extracted from two bundles describing the same person are not merged, and no patient timeline is built from them
  • Patient- and OMOP-dependent processing steps should stay disabled for this workflow. Enabling them is only appropriate when you already know the documents will be linked
  • Files that fail validation or arrive in an unsupported format are excluded from the ingestion output, so the dataset silently under-counts unless you resolve them at Step 3
  • De-identification needs a profile that matches the document format. A profile built for another format will not run against these files
  • The available ingestion purposes, pipeline configurations, templates, and de-identification profiles depend on how your deployment is configured, so what you see in these menus may differ
  • Creating ingestion jobs and datasets is permission-controlled, and both permissions are required to finish the workflow
  • This workflow leaves later linkage open, but it does not perform it. Documents ingested this way stay document-level until you run a workflow that does patient matching