Ingest clinical documents from an S3 source

Bring clinical documents from an S3 bucket into Patient Journey Intelligence and confirm that what arrived matches what you selected. This page ends when the ingested documents are available for patient search, cohort filtering, and dataset creation. It does not build a cohort or a dataset.

Who this is for

Anyone who owns the first step of a document pipeline and will be asked later why a downstream count looks wrong: data engineers who configure sources and run ingestion, platform administrators who hold the credentials and the permissions, clinical data curators and abstractors waiting on documents to review, real-world evidence analysts who will build cohorts on top of this output, and registrars setting up a new registry project or a periodic data refresh.

Why it matters

Every other workflow in this library starts by reading ingestion output. Cohorts, datasets, information extraction, data curation, de-identification, and registry export all sit downstream of this step, and each of them inherits whatever ingestion produced, including its gaps. A document that never landed is not visible anywhere later. It is simply absent.

That absence is quiet. Suppose one folder of synoptic pathology reports sits under a prefix the source configuration does not cover, so those files fail dataset validation and the job finishes anyway. Nothing looks broken. Two stages later, a breast cancer cohort comes back smaller than the tumor registry says it should be, receptor status is missing for the patients who are in it, and the reason is three screens and several days away from where anyone is looking. The same pattern hits any fact that lives in a document rather than a code: ejection fraction in an echo report, functional status in an H&P, the reason a drug was stopped in a progress note, margin status in an operative note.

Read the validation output as a checkpoint, not a log. It is the last point where a shortfall is cheap to explain and cheap to fix, because the only thing built on top of it so far is nothing.

What you gain

A completed ingestion job, a set of clinical documents available inside Patient Journey Intelligence, and patients and documents ready for cohort, dataset, extraction, curation, and de-identification workflows. You also get a number you can defend: the count of documents processed, against the count you selected, with the failed, skipped, and unlinked ones named individually rather than rolled into a total.

The counts will not always match, and that is worth knowing early rather than late. Some documents fail on format or permissions. Others process cleanly but do not link to a patient, which is a review task, not an error. Ingestion structures and surfaces this material; the clinicians and reviewers reading it downstream make the clinical decisions.

Before you start

• The S3 bucket or path and its access details. Incorrect permissions are the most common reason a job cannot read files it can otherwise see listed
• The source type and the expected document structure for that bucket. A folder layout that does not match the source configuration silently narrows what gets picked up
• Permission to configure data sources and to run ingestion jobs. Without both, the workflow stops at Step 1 or Step 2
• An expected document count from the source side. Without it, you have nothing to validate the ingestion result against, and Step 3 becomes a formality

Step 1: Create or select an S3-based data source

1. In the left navigation, go to Data Integration > Source Registry.
2. Select an existing S3-based source, or create a new one.
3. Enter the bucket, path, and access details for the source.
4. Confirm the source settings, including source type and the expected document structure.
5. Preview the available documents where the source supports preview. A preview that returns nothing, or returns a folder you did not expect, is a configuration problem to solve now.

Step 2: Start an ingestion job

1. In the left navigation, go to Data Integration > Ingestion Jobs.
2. Select Add Ingestion.
3. Select the S3 data source configured in Step 1.
4. Select Browse Documents and choose the files to include in the ingestion job.
5. Wait for dataset validation to complete, and continue only after validation succeeds.
6. Review the remaining ingestion settings for the job, then continue to Review & Start.
7. Review the job summary and select Start Ingestion.

Step 3: Monitor the job until it completes

1. Track ingestion progress from the ingestion jobs table.
2. Open the ingestion job details page to inspect job status, logs, and ingested documents.
3. Wait for the job to reach its final status before drawing any conclusion from the counts. A job still running will report a document count that is simply incomplete.

Optional: monitor Databricks compute execution

This applies only to deployments that use Databricks compute. There, the ingestion job is configured and started in Patient Journey Intelligence while the processing workload runs on the associated Databricks compute cluster. If your deployment does not use Databricks, skip this and stay in the ingestion jobs table.

1. Start the ingestion job from Patient Journey Intelligence as described above.
2. In Databricks, open the job run or compute cluster associated with the ingestion workload.
3. Review the run status and driver logs when compute-level monitoring or troubleshooting is required.
4. Review the cluster libraries when environment-level validation is needed. Do not change or install dependencies unless that responsibility is part of your deployment's operating procedure.
5. After processing completes, return to Data Integration > Ingestion Jobs.
6. Confirm that the ingestion reached its expected final status and that the processed-document count matches the selected input.

This path adds compute-level visibility. It does not change the ingestion workflow or move the final validation, which still happens in Patient Journey Intelligence.

Step 4: Review failed, skipped, and unlinked documents

1. Compare the processed-document count against the count you selected in Step 2.
2. Open the job logs and list every document reported as failed or skipped, with its reason.
3. Identify documents that ingested but did not link cleanly to a patient, and queue them for review.
4. Fix the cause and re-run rather than continuing with a partial load. Permissions and folder structure are the usual culprits, and both are faster to correct here than to explain later.
5. Record the final counts somewhere the downstream owner can see them. The analyst building the cohort should not have to reconstruct what ingestion did.

Step 5: Confirm the documents are available downstream

1. Confirm the ingested documents appear on the ingestion job details page.
2. Confirm the documents are available for patient search and cohort filtering.
3. Confirm the ingestion job is selectable in Data Curation Studio > Dataset Builder for dataset creation.
4. Hand off the job name and the validated counts along with the data.

Limitations

• Incorrect S3 permissions or an unexpected folder structure can prevent ingestion. The job may still finish, so treat validation output as the check, not job status alone
• Some documents do not link cleanly to patients and need review. This is expected for external or partially identified source files, and it is a review queue rather than a failure
• A completed ingestion job does not create cohorts or datasets. Those are separate steps you run afterward, against this output
• Source preview availability depends on the source type and on how your deployment is configured
• Databricks compute monitoring applies only to deployments that run the processing workload on Databricks, and cluster library changes belong to whoever owns that deployment's operating procedure

Next step

For the full path from this ingestion output through cohort creation, dataset scoping, and evidence-linked curation, follow Turn clinical documents into curated, reviewable data.