Skip to main content

Link document batches to patients with structured OMOP metadata

Prepare structured JSON metadata alongside a batch of clinical documents so PJI can link the uploaded files to patients and, when the source data holds it, to visits, providers, care sites, locations, and measurements during OMOP CDM ingestion. The page covers preparing the files, defining the linkage, running the ingestion job, and validating the result. It stops once the linked documents and their OMOP records check out in Patient Journey and Database Explorer.

Structured Metadata for OMOP CDM Ingest

This video has no audio. It is a visual walkthrough of the screen.

Who this is for​

Anyone who has to hand a batch of clinical documents to an ingestion job and be certain each file lands on the right patient:

  • Data engineers and integration engineers preparing batch uploads from a source system or archive
  • Clinical data managers responsible for the accuracy of patient-document linkage
  • Real-world evidence and informatics teams that need OMOP context alongside the document text
  • The platform administrator who configures the ingestion job and reviews the detected mappings

Why it matters​

A document that arrives without patient and visit context is a document the timeline cannot place. A pathology report lands in a batch of several hundred PDFs, ingestion has no metadata record for it, and there is nothing to attach it to: no patient, no visit, no care site. The file is processed and then sits outside every timeline it belongs on.

Deterministic linkage from a metadata file is different from inferring the patient from the document text. It does not depend on the text containing a legible identifier, and it does not fail silently. A source value that matches nothing in the related file is a validation error you can read before ingestion starts, rather than a gap someone finds months later.

Broken foreign-key and source-value references are the main cause of orphaned records. Orphaned records surface later as timeline gaps that look like missing care rather than missing links, and a reviewer reading that timeline has no way to tell the two apart. Getting the metadata right up front is cheaper than repairing it afterward, especially once cohorts and datasets have been built on top of it.

What you gain​

A batch of documents attached to the correct patients by explicit identifier match, with whatever visit, provider, care-site, location, personal-data, and measurement context the source system actually carried. The linkage is reproducible: the same metadata files produce the same OMOP relationships, and every relationship can be traced back to a source value you chose.

The linkage also holds up to inspection. Because the foreign keys are validated before ingestion runs and the resulting OMOP records are queryable in Database Explorer, you can show how a document reached a patient instead of asserting that it did. PJI structures and links these records; clinicians make the clinical decisions about the care they describe.

Before you start​

  • Supported clinical documents (PDFs, text files, images) collected in the expected batch-upload location, with their metadata files kept alongside them
  • Stable source identifiers that can be reused consistently across every metadata file. Identifiers that drift between files are the most common cause of documents that ingest successfully and stay unlinked
  • A unique source_document_id for every document metadata record. Duplicates here fail validation
  • A decision, per document, about whether it should be patient-linked. A document with source_patient_id left empty is treated as external
  • Access to Data Integration > Ingestion Jobs and to Metadata Configuration in your deployment

Step 1: Prepare the document batch​

  1. Collect the PDFs, text files, images, and other supported clinical documents for the ingestion batch.
  2. Keep the documents and their structured metadata files together in the expected batch-upload location.
  3. Assign stable source identifiers that can be reused consistently across the metadata files.
  4. Confirm that every document intended for patient linkage has a corresponding metadata record.

Step 2: Prepare the structured metadata files​

The structured metadata workflow uses eight JSON files. Two of them are required for patient linkage. Include the other six only when the source data actually contains that context.

FileRequired?What it carries
metadata.jsonRequired for patient linkageThe document-level hub for the batch: one record per document, each with a unique source_document_id and the source identifiers that point at the related files
person.jsonRequired for patient linkageOne record per patient, with person_source_value, gender, race, ethnicity, and year of birth
person_personal_data.jsonOptionalAdditional personal-data attributes for the patient records
visit.jsonOptionalVisit records keyed by visit_source_value, giving each document its encounter context
care_site.jsonOptionalCare-site records keyed by care_site_source_value
location.jsonOptionalLocation records keyed by location_source_value
provider.jsonOptionalProvider records keyed by provider_source_value
measurement.jsonOptionalMeasurement records, each referencing person_source_value and visit_source_value
  1. Prepare the eight JSON files listed above, either manually or through the ingestion wizard. Downloadable templates are available in Metadata Configuration.
  2. Include metadata.json and person.json for patient-linked ingestion.
  3. Give each document metadata record a unique source_document_id.
  4. Use source_patient_id only when the document should be linked to a patient.
  5. In person.json, provide person_source_value, gender, race, ethnicity, and year of birth for each patient record.

Step 3: Define document and patient linkage​

Use metadata.json as the document-level hub for the batch. Every relationship follows the same pattern: a source identifier in metadata.json matches a *_source_value field in the related file.

Field in metadata.jsonmatchesField in the related file
source_patient_id→person_source_value in person.json
source_visit_id→visit_source_value in visit.json
source_care_site→care_site_source_value in care_site.json
source_location→location_source_value in location.json
source_provider→provider_source_value in provider.json
  1. Link a document to a patient by matching metadata.source_patient_id to person.person_source_value.
  2. When visit context is available, match metadata.source_visit_id to visit.visit_source_value.
  3. Match optional context using the same source-value pattern shown in the table.
  4. Provide both document metadata and a matching person record when patient linkage is required.
  5. Leave source_patient_id empty only for an external document that should be processed without patient linkage.

Step 4: Validate foreign-key relationships​

  1. Verify that shared identifiers match across the JSON files.
  2. Confirm that the source identifiers in metadata.json match the corresponding *_source_value fields in the related files.
  3. Check that referenced person, visit, provider, care-site, and location records exist when those relationships are supplied.
  4. Resolve missing, duplicated, or inconsistent identifiers before starting ingestion.
  5. When measurement.json is included, validate its person_source_value and visit_source_value references and the required measurement fields.

Step 5: Configure and run the ingestion job​

  1. In the left navigation, go to Data Integration > Ingestion Jobs.
  2. Create a new ingestion job and select the source containing the document batch and JSON metadata files.
  3. Wait for dataset validation to complete, then review any missing-file, schema, or relationship errors.
  4. Continue to Metadata Configuration and review the detected mappings for document metadata, person, personal data, visit, care site, location, provider, and measurement records.
  5. Correct any source-to-target mappings that do not align with the intended OMOP fields.
  6. Review the job configuration and start ingestion.
  7. Monitor the job until it reaches its expected final status.

Step 6: Validate the OMOP and Patient Journey output​

  1. Open an ingested patient in Patient Journey.
  2. Confirm that the source documents are linked to the correct patient.
  3. Verify visit context when a visit identifier was provided.
  4. Review available provider, care-site, location, personal-data, and measurement context.
  5. Use Database Explorer when needed to confirm that the corresponding OMOP records and foreign-key relationships were created as expected.
  6. Confirm that the ingested patients and documents are available for downstream cohort, dataset, extraction, curation, and de-identification workflows.
  7. For external documents, confirm that NLP processing completes while the document remains unlinked and does not appear as a patient-linked document in Patient Timeline.

Limitations​

  • metadata.json and person.json are required when a document must be linked to a patient. Without both, there is nothing to link.
  • source_patient_id is optional at the document level. When it is absent, the document is treated as external and remains unlinked to a patient. That is a valid outcome when you intend it, and a silent data gap when you do not. See External FHIR documents without patient linkage for that path.
  • Visit linkage is optional, but the referenced visit must be consistent with the linked patient when supplied.
  • Foreign-key values must match across files. Inconsistent identifiers can produce validation failures or unlinked records.
  • Optional JSON files should be included only when the batch contains the corresponding data.
  • Structured metadata improves linkage and context but does not replace NLP when clinical facts must be extracted from narrative documents. A stage recorded only in a pathology report still needs document processing to become a queryable fact.