S3 to cohort to dataset to Data Curation

This scenario demonstrates the full workflow from source document ingestion to cohort creation, dataset preparation, data curation, evidence review, and result export.

Best for: demonstrating the full journey from raw files to curated clinical data.

Step 1: Ingest oncology documents from S3

1. In the left navigation, go to Data Integration > Ingestion Jobs.
2. Click Add Ingestion.
3. Select a data source from the available configured sources.
4. Click Browse Documents and select the files that should be included in the ingestion job.
5. Wait for dataset validation to complete. Continue only after validation succeeds.
6. In the Ingestion Details section, review and configure the job name, scheduling options, and required ingestion settings.
7. Continue to Pipeline Selection.
8. Select an existing pipeline, or create a new configuration by using Modify Template or New Configuration.
9. Continue to Metadata Configuration.
10. Review the fetched metadata mappings for Document Metadata, Person, Person Personal Data, Provider, Care Site, Location, Visit, and Measurement.
11. Adjust metadata mappings when the source values need to be corrected, normalized, or aligned with the target data model. Confirm that patient identifiers, visit context, document dates, provider details, and measurement fields are mapped consistently before continuing.
12. Continue to Review & Start.
13. Review the ingestion job summary and click Start Ingestion.
14. Monitor ingestion progress from the ingestion jobs table.
15. Open the ingestion job details page when needed to inspect job status, logs, and ingested documents.
16. Confirm that the ingested documents are visible on the details page and ready for downstream cohort, dataset, and curation workflows.

Step 2: Use the Assistant to create a breast cancer cohort

1. In the left navigation, go to Agents and Tools > Assistant.
2. Ask a cohort-building question, for example: "Find all patients with breast cancer."
3. Review the Assistant response, including the interpreted criteria and returned patient population.
4. If no additional clarification or follow-up question is needed, click Create Cohort to begin cohort creation.
5. In the Advanced Filtering panel on the right side of the screen, apply any additional filters if needed.
6. Click Next to continue to Patient Management.
7. Review the patient list and exclude any patients that should not be included in the cohort.
8. Click Next to continue to Cohort Details.
9. Enter the cohort name, description, category, and any linked cohort information.
10. Click Next to continue to Review Cohort.
11. Review the cohort configuration and optionally define a sample size.
12. Click Create Cohort to save the cohort.

Step 3: Build a dataset from the cohort, including pathology and oncology notes

1. In the left navigation, go to Data Curation Studio > Dataset Builder.
2. Select Cohort & Clinical Filter as the dataset mode.
3. Select Unstructured as the data type.
4. Click Next to continue to Cohort Selection.
5. Select the breast cancer cohort created in the previous step.
6. Click Next to continue to Unstructured Criteria.
7. In the Document Filters panel on the right side of the screen, apply the desired document filters.
8. For example, filter by note type and select Pathology. Add oncology note filters as needed for the curation objective.
9. Click Next to continue to Dataset Details.
10. Enter the dataset name and description.
11. Click Next to continue to Review Dataset.
12. Review the dataset configuration, selected cohort, data type, and document filters.
13. Click Create Dataset to create the dataset.

Step 4: Start Data Curation with a cancer registry or custom registry

1. In the left navigation, go to Data Curation Studio > Curation Jobs.
2. Click Start Data Curation to begin creating a new automation.
3. Select a registry from the table, for example Cancer Registry.
4. Review the registry details in the Field Configuration area.
5. Click Next to continue to Dataset Selection.
6. Select the dataset created in the previous step.
7. Click Next to continue to Configuration.
8. Select the extraction method, for example Patient-level.
9. Click Next to continue to Reference Results.
10. If a prior data curation job or information extraction job exists for the selected dataset, select it as a reference result where appropriate.
11. If no reference result is available or needed, continue to Automation Info.
12. Enter the automation name and description.
13. Click Next to continue to Review Automation.
14. Review the automation details, including registry, dataset, extraction method, and reference-result selections.
15. Click Create Automation to start the curation automation.

Step 5: Review extracted fields and evidence

1. Monitor the automation until its status is Completed.
2. Open the automation details page.
3. Review the curation results, extracted fields, confidence indicators, and linked evidence.
4. Open patient-level or field-level details where needed to confirm that extracted values are supported by the source documents.
5. Correct or flag results according to the review workflow available in the deployment.

Step 6: Export or save reviewed results

1. On the curation details page, click Export Curation.
2. Export the reviewed results in CSV or JSON format, depending on the intended downstream use.
3. To inspect patient-specific output, open a patient from the curation results.
4. Review the patient-level extracted fields and evidence.
5. Click Download to export the patient-level results in JSON format.
6. Save or share exported files according to the deployment's governance, privacy, and customer-delivery process.

Recipe reference

Each stage of this scenario is also a standalone recipe you can reuse in other workflows.

Ingest clinical documents from S3 and prepare them for analysis

When to use on its own: At the start of a new customer demo, study setup, registry project, or data refresh.
Features involved: Data Sources, Ingestion Jobs, ingestion monitoring and logs, source document preview, Dataset Builder.
Edge cases / limitations: Incorrect S3 permissions or folder structure can prevent ingestion. Some documents may not link cleanly to patients. Ingestion completion does not automatically create cohorts or datasets.
Value: Turns raw source files into usable clinical assets inside Patient Journey Intelligence.

Create a patient cohort from the Assistant

When to use on its own: When you know the clinical population but don't want to manually build detailed filters.
Features involved: Assistant, Cohort Builder, patient search and filtering, saved cohorts.
Edge cases / limitations: A zero-patient result may mean the data is missing, not that the population doesn't exist. Cohorts should be reviewed before regulatory, trial, or operational use.
Value: Moves you from question to reusable patient population quickly.

Build a dataset from a cohort or ingestion results

When to use on its own: After ingestion and before Data Curation, Information Extraction, or De-Identification.
Features involved: Dataset Builder, Dataset Explorer, cohort filters, ingestion-result filters, dataset preview.
Edge cases / limitations: Data Curation and De-Identification require a dataset; a cohort alone is not enough. Too broad a dataset makes downstream jobs slow or noisy; too narrow risks missing evidence.
Value: Gives you a controlled, reusable work package instead of repeatedly selecting raw records.

Start Data Curation from a dataset

When to use on its own: When you need curated, field-level outputs for registries, studies, oncology abstraction, quality review, or analytics.
Features involved: Data Curation Studio, Dataset Explorer, Custom or Standard Registries, Curation Jobs, Curation Results, evidence review and version history.
Edge cases / limitations: Data Curation starts from a dataset, not directly from a raw source. A mismatched registry or template can produce sparse or inaccurate results.
Value: Converts unstructured clinical documents into structured, reviewable data.