Ingestion jobs: from clinical source to OMOP CDM in one automated pipeline
An ingestion job in Patient Journey Intelligence connects to a clinical source, retrieves documents, and runs them through one automated pipeline: de-identification, NLP extraction, terminology normalization, OMOP CDM conversion, and deduplication. Every step is monitored while it runs and every decision is recorded with provenance back to the source document.
Connecting a new clinical source is usually an engineering project: weeks of custom ETL, a parser per document type, and manual reconciliation when two sources disagree about the same patient. Here you configure where the data lives and what to import, and the platform runs the pipeline. The output is a continuously updated, audit-ready OMOP dataset that refreshes as new data arrives at the source.
Local upload walkthrough: from files on your machine to patient-linked OMOP records
Two ways to bring data in
Every ingestion job starts with a choice: pull from a registered external source, or upload files directly. Both paths run the same automated pipeline and produce the same OMOP output, and they differ only in how the data arrives.
Pull from a registered source
Connect to an external system already configured in the Data Sources module: an EHR, SFTP endpoint, S3 bucket, AWS HealthLake, or HTTP plugin. Run imports on a schedule (daily, weekly, monthly) or trigger them on demand. For sources that support file browsing, you can navigate the directory structure and select exactly which files to import.
Upload files directly
Upload clinical documents from your local machine without a registered connector. Supports TXT, XML, HL7, PDF, and DICOM files. The right choice for pilot ingestions, one-off imports, or testing new document types before connecting a full source system.
Both methods use the same four-step configuration wizard. It takes minutes.
Setting up a new ingestion job
Choose your import method
Select 'Data Source' to pull from a registered external system, or 'Local Upload' to import files from your machine.
Select your source
For external sources, browse your configured connections, including HTTP plugins, SFTP endpoints, AWS HealthLake, and S3 buckets, in card or table view, and select the one you want to use.
Pick your files
Navigate the source's folder structure, filter by file type, search by name, and preview documents before selecting. Toggle de-identification on or off for this job.
Review and start
Confirm your import method, source, file list, and de-identification settings, then click Start Ingestion. The pipeline begins immediately.
The de-identification toggle at step 3 decides what the rest of the pipeline sees. When enabled, PHI removal runs as the first processing step, before NLP extraction and before any clinical fact is written anywhere. Enable it for any dataset built for research, quality improvement, or another secondary use. The de-identification pipeline supports HIPAA Safe Harbor and Expert Determination methods across text, PDF, DICOM, and images.
Choosing compute for the de-identification and NLP steps
De-identification and NLP extraction run on the platform's local compute by default. For large document volumes, each step can run on a Databricks cluster instead. The choice is made per job and per step, so de-identification can run on Databricks while NLP runs locally, or both can run on the cluster, according to volume and cluster availability.
When a step is configured for Databricks, the platform uses the Databricks credential configured in Administration. Documents are sent to the cluster, processed there, and the results are written back. No documents are permanently stored outside the platform, and the cluster holds only the active processing job.
Example: a weekly feed and a one-time backfill
A health system ingests about 3,000 documents a week from an S3 bucket and is also loading 11 years of archived discharge summaries from the same bucket. The weekly job keeps both steps on local compute. The backfill job runs de-identification and NLP on a Databricks cluster. Both jobs use the same source, the same de-identification profile, and the same pipeline configuration.
C-CDA ingestion
Patient Journey Intelligence ingests C-CDA (Continuity of Care Document) files and maps their structured content directly to OMOP CDM, without routing it through the NLP extraction step. Diagnoses, medications, allergies, procedures, and encounters already coded in the C-CDA XML are read from the structured sections and written to the matching OMOP domains. The NLP pipeline still runs over any free-text narrative sections in the same document.
A C-CDA from an ambulatory EHR transfer therefore carries its coded problem list and medication list into OMOP with the codes the sending system assigned, and NLP fills the gaps where structured coding is absent.
Example: a referral C-CDA
A C-CDA arrives with 14 coded problems, 9 coded medications, and a three-paragraph narrative summary from the referring physician. The 23 coded entries map to condition_occurrence and drug_exposure from the structured sections. NLP reads the narrative and contributes the symptom onset and the functional status that were never coded anywhere.
What happens after you click Start
Once a job launches, the platform runs seven sequential steps automatically. Click the job to open the Job Status panel and watch each one progress.
1. Document retrieval
Files are securely downloaded from the source system or read from the uploaded batch. Connection credentials are verified and file integrity is checked before processing begins.
2. De-identification
If enabled, PHI such as names, dates, identifiers, and geographic details is removed using HIPAA-compliant methods before any other processing touches the content. The de-identified text is what flows into all downstream steps.
3. Information extraction
Healthcare NLP reads every clinical document and extracts diagnoses, medications, procedures, lab results, imaging findings, and relationships, including assertion status (present, absent, historical, hypothetical) for each entity.
4. Terminology normalization
Every extracted and structured concept is resolved to a standard vocabulary code: SNOMED CT, RxNorm, LOINC, ICD-10-CM, CPT, or one of 40+ additional vocabularies. Local institutional codes are mapped automatically.
5. OMOP conversion
Normalized clinical facts are transformed into OMOP CDM v5.4: condition occurrences, drug exposures, procedure occurrences, measurements, observations, and note_nlp records, with standard concept IDs and full source provenance.
6. Merge and deduplication
New records are merged into the existing OMOP dataset. Duplicate entities across sources are detected and consolidated. Conflicts are resolved using the medical reasoning hierarchy: source priority, recency, confidence score, clinical logic.
7. Clinical measure recalculation
Any clinical quality metrics or registry measures that depend on the updated data are recalculated automatically, so dashboards and reports reflect the latest ingested content without a manual refresh.
For each step, the status panel shows a completion indicator, elapsed time, and processing statistics, for example "312 documents processed, 2,140 clinical facts extracted, 847 concepts resolved". A job is marked Completed only when all seven steps finish successfully.
The diagram below shows the pipeline architecture end to end. Input files arrive from source systems or local storage, pass through the content extraction stages, and flow through information extraction, concept resolution, OMOP modeling, and merge and deduplication before the curated output reaches downstream platform modules: Cohort Building, Clinical Measures, and Data Curation. Each processing stage writes intermediate results to Elastic Search, which holds pipeline state, provenance, and monitoring data.
End-to-end ingestion pipeline architecture. Source systems feed into local processing storage, where content extraction, information extraction, concept resolution, OMOP modeling, and merge/deduplication are applied in sequence. Curated outputs reach downstream platform modules including Cohort Building, Clinical Measures, and Data Curation.
Step 5, OMOP conversion, is where extracted clinical facts become a structured, queryable dataset. The pipeline reads the enriched NLP annotations produced in step 3, applies confidence and terminology filters, and maps each clinical entity to its OMOP CDM v5.4 domain. Diagnoses become condition occurrences, prescriptions become drug exposures, procedures become procedure occurrences, lab results become measurements, and free-text notes are preserved as note and note_nlp records. Each record carries its standard concept ID, source value, and provenance back to the originating document. The output is written to a temporary per-source schema before step 6 merges it into the Gold OMOP dataset, so a partial run never corrupts the canonical patient data.
Transforming enriched annotations into OMOP CDM. This pipeline reads enriched NLP annotations and source documents, applies confidence and terminology filters, maps clinical facts into OMOP CDM domains including condition occurrence, drug exposure, procedure occurrence, measurement, observation, note, and note_nlp, and prepares temporary schemas for merge into the governed OMOP dataset.
Monitoring the dashboard
The ingestion dashboard gives you one view of every job, scheduled and manual, running and historical. Each row shows:
| Field | What it tells you |
|---|---|
| Job ID | Unique tracking number for this import |
| Source Name | Where the data came from ("Main EHR", "Manual Upload") |
| Source Type | Connection method: S3, SFTP, HealthLake, HTTP Plugin, Local Upload |
| Created At / By | When the job started and who initiated it |
| Documents Processed | How many files were imported |
| Schedule Type | None (manual), Daily, Weekly, or Monthly |
| Pipeline Duration | Total elapsed time from start to finish |
| Status | In Progress, Completed, or Failed |
| Progress Bar | Visual indicator showing how far along the job is in its current step |
At a glance you can see what is running, what succeeded, and what needs attention, without opening individual jobs.
Scheduled ingestion: keeping your dataset current
A one-time import gives you a snapshot. A schedule gives you a dataset that stays current. For any registered data source, configure an automatic schedule, daily, weekly, or monthly, and the platform runs the full ingestion pipeline at that interval with no manual action.
Scheduled jobs appear on the dashboard with three additional fields:
- Frequency: How often they run (Daily, Weekly, Monthly)
- Next Execution: When the next automatic import will start
- Last Execution: When the most recent import completed
Combined with step 7, clinical measure recalculation, this keeps your OMOP dataset, quality metrics, and registry measures current as new clinical data arrives at the source.
When a job fails: logs and recovery
Connection timeouts, unsupported file formats, and NLP edge cases all cause individual steps to fail. The Logs tab on any job records what went wrong:
What the logs capture:
- Connection problems: authentication failures, network timeouts, credential expiry
- File format issues: unsupported types, corrupted files, encoding problems
- NLP extraction errors: documents that could not be parsed or exceeded processing limits
- Pipeline failures: which stage failed, the error message, and the affected record count
- Document-level warnings: individual files with issues that did not stop the overall job
Once you have resolved the underlying issue, by correcting a credential, removing a corrupted file, or adjusting a configuration, three controls are available on a job:
- Re-run Job: Restarts the same import with identical settings. Use this after fixing the underlying issue. The job runs with the same source, file selection, and de-identification configuration.
- Stop Job: Immediately cancels a running job. Use this if you notice a configuration mistake mid-run. No partial output is written to the OMOP dataset.
- Delete job: Available on any finished job, whether completed, failed, or stopped. Deleting first clears the documents and OMOP rows the job produced, then removes the job from the list. Only system administrators can use this action, and a confirmation window shows exactly what will be cleared before anything is removed. Imaging studies the job ingested are not removed automatically; you confirm that they stay before the job is deleted. If clearing does not finish completely, the job is kept so you can try again.
These controls let you recover from errors and iterate on an import configuration without building a new job from scratch.
Sending job data to Generative AI Lab
Every ingestion job has a GenAI Lab tab, which sends the job's data to Generative AI Lab as a new review project so a human reviewer can check it there.
You choose what to send:
- The ingested documents
- The PHI found during de-identification, which uploads the original, un-redacted document rather than the PHI spans alone
- The clinical entities NLP extracted
- The concept-resolved annotations
- Any combination of the above
Send the whole job or a single batch, and add to a project created from an earlier export instead of always starting a new one. The tab lists every export that has been started and shows how far its review has progressed.
Example: validating a new document type
A team ingests 200 operative notes from a new source and sends the extracted entities and concept-resolved annotations to Generative AI Lab. Two clinical reviewers confirm the procedure and laterality entities there, and the team uses what the review found to adjust the pipeline configuration before running the remaining 40,000 notes.
What the module enables
Manual and scheduled imports
Import data on demand or configure automatic schedules, daily, weekly, or monthly, for any registered source. Both run through the same full pipeline.
End-to-end processing
Every document passes through de-identification, NLP extraction, terminology normalization, OMOP conversion, and deduplication automatically, without custom engineering per source.
Compute chosen per job
Run de-identification and NLP extraction on local compute for routine jobs and on a Databricks cluster for large backfills, using the same source and the same configuration.
File preview before import
Browse the source directory, filter by file type, and preview individual documents before committing to an import, so you import exactly what you intend.
Real-time pipeline monitoring
Watch every step of every job as it runs, with per-step status, elapsed time, and processing statistics. No black-box processing.
Detailed logs for troubleshooting
Every connection failure, format error, and extraction issue is logged at the document level, giving you what you need to diagnose and fix problems quickly.
Best practices for production ingestion
Start with a pilot before scaling. Import 100 to 1,000 documents from a single source, review the extraction results in the Database Explorer, check that terminology mappings match your institution's coding patterns, and validate the OMOP output structure. A mapping issue corrected at this stage costs minutes. The same issue found after several million documents have been processed costs a full reprocessing run.
Enable de-identification by default for any job feeding a research or secondary-use dataset. Enabling it consistently is easier than auditing later which jobs ran with it on. The de-identified and identified OMOP datasets stay synchronized automatically, so the toggle creates no separate pipeline and no additional overhead.
For production sources, use scheduled ingestion. Manual jobs depend on someone remembering to run them. Scheduled jobs run whether or not anyone is watching, and the dashboard makes a missed run obvious.
Review logs actively on first-time imports. The most common issues surface in the first few hundred documents: local code mappings the terminology server does not recognize, document formats that need a parser configuration, and PHI patterns specific to your institution. Catching them there keeps the dataset clean from the start.
For local uploads: TXT, XML, HL7, PDF, and DICOM. For registered sources: SFTP endpoints, Amazon S3 buckets, AWS HealthLake, HTTP plugins, and FHIR endpoints. Structured EHR data arrives via HL7 v2 feeds and FHIR R4/STU3, and C-CDA documents are mapped to OMOP from their structured sections. Additional formats can be configured based on your Patient Journey Intelligence instance.
Before. De-identification is step 2 in the pipeline, immediately after document retrieval, and NLP extraction is step 3. PHI is therefore removed from the content before any clinical fact is extracted or written to storage. Terminology mapping, OMOP conversion, and deduplication all operate on de-identified text when de-identification is enabled.
Yes. The de-identification toggle is configured per job rather than per source. You can run a de-identified job for a research export pipeline and an identified job for your operational OMOP dataset from the same source, and both outputs are maintained as synchronized datasets.
Yes. The de-identification and NLP extraction steps each run on local compute or on a Databricks cluster, selected per job. The platform uses the Databricks credential configured in Administration, sends documents to the cluster, and writes the results back. No documents are permanently stored outside the platform.
Structured C-CDA sections are mapped directly to OMOP CDM without passing through NLP, so diagnoses, medications, allergies, procedures, and encounters keep the codes the sending system assigned. The NLP pipeline still runs over the free-text narrative sections of the same document, which is where findings that were never coded are recovered.
The job is marked Failed and the Logs tab shows which step failed and why. No partial output is written to the OMOP dataset, because the pipeline is atomic at the job level. Once you resolve the underlying issue, use Re-run Job to restart with the same settings. Completed steps are not re-executed; the job resumes from the failed step.
Scheduled jobs run daily, weekly, or monthly. For near-real-time ingestion from active EHR feeds, contact your John Snow Labs implementation team about event-driven ingestion configurations.
Use the Database Explorer to query the OMOP concept tables after a pilot ingestion. Check that local codes map to the expected standard concept IDs, and review any record where concept_id is 0, which marks an unmapped value. The Medical Terminology Server logs resolution statistics per job, which are also available in the Logs tab.
Related tutorials
Step-by-step walkthroughs that run ingestion jobs against this module, in the Patient Journey Intelligence cookbook:
- Ingest FHIR R4 into OMOP CDM, with or without NLP
- Ingest Epic EHR Patients and Clinical Notes over FHIR R4
- Ingest Clinical Documents from an S3 Source
- Ingest External FHIR Documents Without Patient Linkage
- Ingest External HL7 CDA Documents Without Patient Linkage
- Turn Clinical Documents into Curated, Reviewable Data