Build a dataset from a cohort or ingestion results

A cohort names patients. A dataset names the documents about those patients that a specific job should read. This recipe builds that dataset, either from a saved cohort or straight from ingestion-result filters, and stops once the contents check out in Dataset Explorer and the dataset is ready for Data Curation, Information Extraction, or De-Identification.

Cohort or dataset

Data Curation and De-Identification require a dataset. A cohort alone is not enough, because a cohort says who to study and says nothing about which documents to read.

	Cohort	Dataset
What it holds	Patients matching clinical criteria	The patients, documents, images, and note types in scope for one job
How you build it	Assistant or Cohort Builder	Dataset Builder, from a saved cohort or from ingestion-result filters
What it feeds	Datasets, care-gap analysis, registry projects	Data Curation, Information Extraction, De-Identification
Reusable?	Yes, across many datasets	Yes, as a fixed work package for repeated or re-run jobs

Who this is for

Anyone who has to hand a scoped, repeatable work package to a curation, extraction, or privacy job:

• Data curators preparing input for abstraction or field-level review
• Clinical analysts and researchers defining the boundary of a study pull
• Registrars assembling the case material for a reporting year
• Privacy officers scoping the documents that go through De-Identification
• The data engineer who confirms ingestion finished and the documents are linked

Why it matters

Scope is the cheapest lever you have over a curation job. Send it every document in a chart and it spends compute on registration forms, insurance authorizations, and billing sheets, none of which contain the fact you asked for. Compute is the smaller cost. Those documents land in the reviewer's queue as candidate evidence, and someone has to open each one to decide it says nothing.

Filtering to pathology reports and oncology progress notes before a recurrence curation job runs is the difference between a result set a reviewer works through and a queue nobody finishes. The recurrence date lives in the pathology report and the oncologist's progress note. It does not live in the H&P from the patient's knee surgery three years earlier, and including that note only lowers the signal in the output.

The reverse failure is quieter. Cut the scope to pathology reports alone and a recurrence documented only in an oncology progress note is absent from the curated output, which reads as a patient without recurrence rather than a patient whose evidence was out of scope. Scope decisions are evidence decisions, so make them deliberately and write them down.

What you gain

A named, reusable work package instead of a fresh manual selection of raw records each time a job runs. The same dataset can feed a curation job, an extraction job, and a de-identification job, and re-running any of them a month later reads the same boundary you defined the first time.

The boundary is also inspectable. Because the dataset's contents are listed in Dataset Explorer, you can show which documents a curated field could have come from and which were never in scope. PJI scopes and structures those documents and links every curated value back to its source text; clinicians make the clinical decisions about the care described in them.

Before you start

• Ingestion is complete and the documents you expect are available in PJI. A dataset built while a job is still running captures whatever landed so far
• Documents are linked to patients where linkage was intended. Unlinked documents will not appear through a cohort-based selection, no matter how the cohort is defined
• A saved cohort from the Assistant or Cohort Builder, if you are starting from a patient population
• A decision about the unit of analysis: every patient in a registry reporting year, or every synoptic pathology report in a date range. This decision drives every other choice in the builder
• Permission to create datasets and to run the downstream curation, extraction, or de-identification job in your deployment

Step 1: Choose your starting point

1. Open Dataset Builder.
2. Choose whether to start from a saved cohort or to filter directly from ingestion results.
3. Start from a saved cohort when the unit of analysis is the patient, for example every patient in a registry reporting year. Curation then runs at patient level over a cohort-derived dataset, and each result belongs to a patient.
4. Filter directly from ingestion results when the unit of analysis is the document, for example every synoptic pathology report in a date range. Curation then runs at document level over a document-scoped dataset, and each result belongs to a document.
5. If both framings look plausible, pick the one that matches how the output will be read and reported. A registry case list is patient level. A report-quality review is document level.

Step 2: Set the dataset scope

1. Select the patients, documents, images, and note types the job needs.
2. When filtering from ingestion results, narrow by document type, note type, diagnosis, procedure, patient attributes, or source.
3. Include the note types where the target fact is actually written. For a recurrence field, that is usually the pathology report and the oncology progress note.
4. Exclude administrative material that cannot hold the fact, such as registration forms and billing sheets.
5. Add a date range when the study or reporting period has one, rather than filtering by date after the job has run.

Step 3: Preview and create the dataset

1. Preview the dataset contents.
2. Check the patient and document counts against what you expected. A count far below expectation usually points at unlinked documents or a filter that is stricter than intended.
3. Spot-check a few documents in the preview to confirm the note types are the ones you meant to include.
4. Adjust the filters and preview again until the scope is right.
5. Create the dataset and give it a name that states its scope and period, so the next person can tell what it covers without opening it.

Step 4: Confirm the contents in Dataset Explorer

1. Open Dataset Explorer and select the new dataset.
2. Confirm the patient, document, and image scope matches the definition you intended.
3. Verify that the note types present are the ones the downstream job depends on.
4. Note anything missing that the job will need, and rebuild the dataset rather than working around the gap later.

Step 5: Hand the dataset to the downstream job

1. Use the dataset as the input for Data Curation, Information Extraction, or De-Identification.
2. Keep the dataset for re-runs, so a repeated job reads the same boundary.
3. Build a separate dataset for a different unit of analysis or a different reporting period instead of editing one that jobs already depend on.

Limitations

• Data Curation and De-Identification require a dataset. A cohort alone is not enough.
• A dataset can be cohort-based or built directly from ingestion filters. The two produce different units of analysis downstream, so the choice is not cosmetic.
• A dataset that is too broad makes downstream jobs slow and their output noisy, which pushes the cost onto whoever reviews the results.
• A dataset that is too narrow risks leaving out documents that hold the evidence, and absent evidence reads like an absent finding.
• Documents that never linked to a patient during ingestion stay outside any cohort-based dataset. Fix the linkage upstream rather than widening the filters.
• What happens to a running or completed job when its dataset is later changed or deleted depends on your deployment configuration. Confirm the behavior before editing a dataset that other jobs point at.

Related pages

• From S3 ingestion to cohort, dataset, and curation runs this step in sequence with the ingestion and curation work on either side of it.