Skip to main content

Start data curation from a dataset

Run structured abstraction over a dataset so narrative clinical documents come back as a table with named columns: histology, grade, receptor status, treatment start date. This page covers selecting the registry or template, running the curation job, and reviewing each extracted field against the sentence it came from. It stops at a reviewed, versioned curation output ready for export.

Who this is for​

Anyone who has to produce field-level data from documents that were written as prose:

  • Data curators and clinical abstractors working through study, registry, or quality caseloads
  • Tumor registrars and other registry staff responsible for coded case data
  • Real-world evidence and outcomes researchers who need note-derived variables next to coded data
  • Clinical subject-matter experts who define what a field means and validate what comes back
  • The platform administrator who configures registries, templates, and reviewer access

Why it matters​

Take a breast cancer case. The histology and grade are in the pathology report, ER and PR status are in the same report, HER2 status arrives later in an addendum, and the date systemic therapy actually started is in a medical oncology progress note. None of it exists as a coded value anywhere in the record. Someone has to read all four documents and write four values into a form.

That is the work most abstraction backlogs are made of. It scales with staff hours, and it degrades in a specific way: the abstractor who is on case 60 of the day reads faster, and a value that was transcribed rather than verified is very hard to audit six months later when a reviewer asks where the grade came from.

Data Curation runs the extraction over a dataset and returns each field with a pointer to the text it was taken from. The abstractor's job changes from finding values to confirming them. PJI structures the data and surfaces the evidence; the clinical and coding judgment stays with the people doing the review.

What you gain​

A structured table of the fields you asked for, across every patient and document in the dataset, where each value carries the evidence span it was drawn from. That single property changes the shape of the work. A reviewer confirms a highlighted value against the sentence that produced it, which is a different and much faster task than reading a 14-page operative note to find one number.

It is also why the output holds up when a value is challenged. A curated grade of 3 arrives with a source document, a source sentence, a registry version, and a named reviewer who accepted it. Corrections you make during review are saved as a new version rather than overwriting the run, so the extracted value and the human decision both stay on the record.

Before you start​

  • A dataset built in Dataset Builder covering the patients, documents, and note types you want curated. Data Curation runs against a dataset, so a saved cohort on its own will not start a job
  • A registry or template that matches those documents: a standard registry, a custom registry, or a curation template. A registry aimed at documents the dataset does not contain is the most common cause of a run that returns almost nothing
  • Field definitions your subject-matter experts agree on, including what each field should hold when the note is silent
  • Reviewers assigned and permissioned for curation review. Results that no one is assigned to verify are extraction output, not curated data
  • Access to Data Curation and to the registry configuration your deployment uses

Step 1: Open the dataset and start a curation job​

  1. Open the dataset you want curated in Dataset Explorer and confirm its patient, document, and note-type scope.
  2. Go to Data Curation and start a new curation job against that dataset.
  3. Name the job so it can be identified later against the study, registry, or review cycle it belongs to.

Step 2: Select the registry or curation template​

  1. Choose the standard registry, custom registry, or curation template that defines the fields to extract.
  2. Review the field list before running anything, and confirm each field is one the documents in this dataset could plausibly support.
  3. Note the registry version you selected. It is what makes the run reproducible.
  4. Where your deployment supports it, bring in Information Extraction results as reviewer context. Confirm this handoff in your own environment before you build a workflow on it.

Step 3: Configure the scope and run the job​

  1. Set the curation scope: which patients, documents, and note types in the dataset the job should process.
  2. Start the job and monitor it to completion.
  3. Check the field coverage on the first results before committing reviewer time. Sparse output across the board usually points at a registry or template mismatch rather than at missing documentation.

Step 4: Review each field against its evidence​

  1. Open Curation Results and work through the extracted fields patient by patient.
  2. For each value, open the linked evidence and confirm the value against the source sentence.
  3. Use the confidence indicators to sequence the work, taking low-confidence and unresolved fields first.
  4. Correct values that do not match their evidence, and record why where the interface supports a note.
  5. Mark a field as not documented only when the documents genuinely do not state it. That result means the documentation is silent, not that the fact is absent.

Step 5: Save a reviewed version and export​

  1. Save the reviewed results as a version, so the extracted values and the reviewed values are both retrievable.
  2. Confirm the version history shows the registry version, the run, and the review.
  3. Export the curated output when downstream analysis needs it, and map free-text terms to standard concepts before joining the output to a cohort.

Rules for curated fields that OMOP cannot hold​

Most PJI workflows move and reshape data the OMOP Common Data Model already holds. Curation is often used for the opposite case: clinically decisive facts that live only in free text and have no native home in OMOP's coded domains of condition, drug, measurement, procedure, visit, and observation. A discontinuation reason, a device UDI, a recurrence event, a pedigree entry. Those fields need a custom registry rather than a built-in preset, and they carry more risk than a field with a standard concept behind it, so five rules apply to every one of them.

  • Version the custom registry. A run is reproducible and auditable only if you can say which version of the field definitions produced it.
  • Keep the evidence spans. Every curated value should link back to the source sentence, for review now and for provenance later.
  • Treat "not documented" as absence of documentation. It is not evidence that the fact is untrue, and analysis that reads it that way will be wrong.
  • Map free-text terms back to standard concepts before joining curated output to a cohort. Unmapped strings do not join cleanly and do not compare across sites.
  • Use human-in-the-loop review before any research, safety, or quality decision that rests on these fields.

The four recipes below apply these rules to specific field sets, each with an illustrative custom registry you can adapt:

Limitations​

  • Data Curation starts from a dataset, not directly from a raw source. Ingest and build the dataset first.
  • A custom registry or curation template that does not match the documents in the dataset produces sparse or inaccurate results. Check field coverage on early results rather than at the end of a review cycle.
  • Field lists in the advanced curation recipes are illustrative starting points, not shipped presets.
  • Some result-to-analysis flows require export or extra configuration before curated output can be queried elsewhere in PJI. Confirm what is enabled in your deployment.
  • Reuse of Information Extraction results inside a curation job depends on deployment configuration and should be confirmed before you rely on it.
  • Curated values are proposals until a qualified reviewer accepts them. Nothing here replaces clinical or coding judgment.