Skip to main content

Turn clinical documents into curated, reviewable data

Ingest clinical documents from an S3 bucket and turn them into structured fields a reviewer can verify against the source. Ingestion, cohort selection, dataset scoping, and field extraction run as one pass, and every extracted value carries the evidence span it came from.

Ingest Clinical Documents from S3 and Run Data Curation

Who this is for​

Anyone who needs structured, reviewable fields out of clinical documents, whatever the downstream purpose: clinical data scientists and real-world evidence analysts building study variables, quality and population health analysts pulling measures that exist only in narrative text, research coordinators pre-screening for trials, pharmacovigilance teams reading discontinuation reasons out of progress notes, registrars working to a defined data dictionary, and the clinical data engineers who own the document pipeline feeding all of them.

Why it matters​

Registry reporting needs histology, grade, stage, receptor status, and treatment dates for every incident case. None of those fields exist in billing data. They exist in one sentence inside a synoptic pathology report, and an abstractor finds that sentence by opening the PDF.

The pattern repeats across specialties. Ejection fraction sits in an echo report, not a claim. Smoking and functional status sit in an H&P. The reason a medication was stopped sits in a progress note. Wherever the fact you need was written rather than coded, no SQL query against the structured warehouse will return it.

What you gain​

Ingestion, cohort selection, and field extraction run as one pass, and each extracted field carries the source span it came from. Your reviewers verify a highlighted value against the report on screen instead of reading the full document and typing the value. The output is a structured table with provenance attached, which is what an auditor asks for when they question a submitted value.

Two ways to scope the work follow from the same ingestion. A cohort is a set of patients; a dataset is a set of documents. Curation runs against either one, at patient level from a cohort or at document level from a dataset. Choose by the unit of analysis: every patient in a registry reporting year, or every synoptic pathology report in a date range.

Before you start​

  • An S3 source registered in Data Integration > Source Registry, with credentials the deployment can use
  • Dataset validation passing during ingestion. Failed validation here is the most common cause of an empty downstream cohort
  • The ingestion job finished. Cohort, dataset, and curation steps all read from ingestion output

Step 1: Ingest oncology documents from S3​

  1. In the left navigation, go to Data Integration > Ingestion Jobs.
  2. Click Add Ingestion.
  3. Select a data source from the available configured sources.
  4. Click Browse Documents and select the files that should be included in the ingestion job.
  5. Wait for dataset validation to complete. Continue only after validation succeeds.
  6. In the Ingestion Details section, review and configure the job name, scheduling options, and required ingestion settings.
  7. Continue to Pipeline Selection.
  8. Select an existing pipeline, or create a new configuration by using Modify Template or New Configuration.
  9. Continue to Metadata Configuration.
  10. Review the fetched metadata mappings for Document Metadata, Person, Person Personal Data, Provider, Care Site, Location, Visit, and Measurement.
  11. Adjust metadata mappings when the source values need to be corrected, normalized, or aligned with the target data model. Confirm that patient identifiers, visit context, document dates, provider details, and measurement fields are mapped consistently before continuing.
  12. Continue to Review & Start.
  13. Review the ingestion job summary and click Start Ingestion.
  14. Monitor ingestion progress from the ingestion jobs table.
  15. Open the ingestion job details page when needed to inspect job status, logs, and ingested documents.
  16. Confirm that the ingested documents are visible on the details page and ready for downstream cohort, dataset, and curation workflows.

Step 2: Use the Assistant to create a breast cancer cohort​

  1. In the left navigation, go to Agents and Tools > Assistant.
  2. Ask a cohort-building question, for example: "Find all patients with breast cancer."
  3. Review the Assistant response, including the interpreted criteria and returned patient population.
  4. If no additional clarification or follow-up question is needed, click Create Cohort to begin cohort creation.
  5. In the Advanced Filtering panel on the right side of the screen, apply any additional filters if needed.
  6. Click Next to continue to Patient Management.
  7. Review the patient list and exclude any patients that should not be included in the cohort.
  8. Click Next to continue to Cohort Details.
  9. Enter the cohort name, description, category, and any linked cohort information.
  10. Click Next to continue to Review Cohort.
  11. Review the cohort configuration and optionally define a sample size.
  12. Click Create Cohort to save the cohort.

Step 3: Build a dataset from the cohort, including pathology and oncology notes​

  1. In the left navigation, go to Data Curation Studio > Dataset Builder.
  2. Select Cohort & Clinical Filter as the dataset mode.
  3. Select Unstructured as the data type.
  4. Click Next to continue to Cohort Selection.
  5. Select the breast cancer cohort created in the previous step.
  6. Click Next to continue to Unstructured Criteria.
  7. In the Document Filters panel on the right side of the screen, apply the desired document filters.
  8. For example, filter by note type and select Pathology. Add oncology note filters as needed for the curation objective.
  9. Click Next to continue to Dataset Details.
  10. Enter the dataset name and description.
  11. Click Next to continue to Review Dataset.
  12. Review the dataset configuration, selected cohort, data type, and document filters.
  13. Click Create Dataset to create the dataset.

Step 4: Start Data Curation with a cancer registry or custom registry​

  1. In the left navigation, go to Data Curation Studio > Curation Jobs.
  2. Click Start Data Curation to begin creating a new automation.
  3. Select a registry from the table, for example Cancer Registry.
  4. Review the registry details in the Field Configuration area.
  5. Click Next to continue to Dataset Selection.
  6. Select the dataset created in the previous step.
  7. Click Next to continue to Configuration.
  8. Select the extraction method, for example Patient-level.
  9. Click Next to continue to Reference Results.
  10. If a prior data curation job or information extraction job exists for the selected dataset, select it as a reference result where appropriate.
  11. If no reference result is available or needed, continue to Automation Info.
  12. Enter the automation name and description.
  13. Click Next to continue to Review Automation.
  14. Review the automation details, including registry, dataset, extraction method, and reference-result selections.
  15. Click Create Automation to start the curation automation.

Step 5: Review extracted fields and evidence​

  1. Monitor the automation until its status is Completed.
  2. Open the automation details page.
  3. Review the curation results, extracted fields, confidence indicators, and linked evidence.
  4. Open patient-level or field-level details where needed to confirm that extracted values are supported by the source documents.
  5. Correct or flag results according to the review workflow available in the deployment.

Step 6: Export or save reviewed results​

  1. On the curation details page, click Export Curation.
  2. Export the reviewed results in CSV or JSON format, depending on the intended downstream use.
  3. To inspect patient-specific output, open a patient from the curation results.
  4. Review the patient-level extracted fields and evidence.
  5. Click Download to export the patient-level results in JSON format.
  6. Save or share exported files according to the deployment's governance, privacy, and customer-delivery process.

Recipe reference​

Each stage of this scenario is also a reusable building block.

These steps have their own pages: