Skip to main content

Curation Jobs

A curation job identifies the patients who meet your selection criteria, extracts the clinical data your ontology defines, and keeps the resulting dataset current as new clinical data arrives. Jobs run on a schedule you set, so a dataset reflects what the record says today rather than what it said when someone last ran an extraction.

What a curation job does​

A clinical dataset is never finished. Patients receive new diagnoses, undergo procedures, start medications, and accumulate documentation. Curation Jobs automate the discovery and extraction work that keeps up with that.

You configure a job once. It then identifies patients matching your inclusion and exclusion criteria, extracts the clinical data elements you specified, transforms them according to your schema, and produces reports of what was found and processed. The dataset behind a quality report, a research study, or a registry submission stays current without anyone remembering to refresh it.

Setting up a curation job​

Creating a curation job takes four guided steps, from selecting the extraction schema to launching the automation.

Step 1: Ontology configuration​

Choose the ontology that defines what clinical information to extract and how to structure it. The ontology is the extraction schema: it specifies which fields to capture, what data types to expect, and what extraction logic to apply.

Browse the available ontologies and select the one that matches your dataset. The system displays each ontology's configuration, including its fields, categories, and extraction guidelines. A cancer registry needing detailed staging, a chronic disease cohort needing medication and lab data, and a custom research dataset each select or create the ontology that structures their data.

Curation job wizard step 1: selecting the ontology that defines the extraction schema

Step 2: Dataset selection​

Select the dataset holding the patients and documents to process. This defines the population that will be evaluated against the ontology's extraction criteria.

Choose from existing cohorts, patient lists, or document collections already defined in the system: a cohort of patients with a specific diagnosis, a group identified through an earlier query, or a document set filtered by type and date range.

The dataset selection is the job's scope. Only patients and documents inside it are processed during extraction, so a job targets the population you intend and spends no compute on the rest.

Curation job wizard step 2: selecting the dataset that defines the patient population

Step 3: Curation information​

Give the job metadata. Start with a descriptive name, which appears in your dashboard and reports, so something meaningful such as "Q4 2024 Breast Cancer Registry" or "Diabetes Cohort weekly update" pays off later. This step also specifies the target database where extracted results are stored, which determines where curated data becomes available for downstream analytics, reporting, and research.

A description is optional and worth writing. It records the job's purpose, scope, and any special considerations, which is what the next person on the team reads when a job has been running for six months.

Curation job wizard step 3: naming the job and selecting the target database

Step 4: Review and launch​

Before execution begins, review the configuration. The summary shows the selected ontology and its field count, the chosen dataset with patient and document counts, the target database, and the metadata you configured.

Verify that the ontology, the population, and the storage location are the ones you intended. This is the last point at which a correction costs nothing.

When everything is correct, click the start curation button. The job begins processing immediately, applying the ontology's extraction logic to the selected dataset and storing structured results in the target database.

Curation job wizard step 4: reviewing the configuration before launching the job

Monitoring job results​

After each execution, review the job logs and statistics. They report how many patients were identified, how many records were extracted, any errors or warnings encountered during processing, and summary metrics about the extracted data.

Reviewing these confirms that jobs run as expected, surfaces issues that need attention, and tracks registry growth over time.

Job status: Completed with errors​

A curation job reports its outcome per patient as well as for the run. When some patients are processed successfully and others encounter errors, the job reports a Completed with errors status.

The distinction is operational. A job that ran against 800 patients and hit extraction errors on 12 of them is a different situation from a job that failed to start. Completed with errors tells you that results are available for the 788 patients that succeeded, and that the 12 that failed need attention. Open the job, see which patients had errors, review the error detail for each one, and re-run extraction for that population alone, leaving the patients that already completed correctly untouched.

Reusing information extraction results from an ingestion job​

When a dataset is built from an ingestion job, either from the Datasets page or with Create Dataset on the job's results, the platform takes the NLP extraction results that job already produced and makes them available directly in Data Curation. A separate extraction run is not needed.

The run appears in Data Curation under Augment with Information Extraction → Use existing job. It covers every pipeline the ingestion step ran, including the OMOP NER pipeline. A dataset built from several ingestion jobs gets one combined run holding all of their results.

The run is prepared in the background as soon as the dataset is saved and shows as processing until it is ready. Deleting the run removes this copy alone and leaves the underlying ingestion results untouched.

Example: review starts the same afternoon​

A research team ingests 6,500 discharge summaries with the NLP extraction step enabled, which takes most of a day. When they create a dataset from that job, the extraction results are already present in Data Curation. Review starts that afternoon rather than after a second extraction run over the same 6,500 documents.

It means the job processed some patients successfully and hit errors on others. Results are available for the patients that succeeded. Open the job to see which patients had errors and the error detail for each, then re-run extraction for that subset alone. Patients that completed correctly are not re-processed.

No. When a dataset is created from an ingestion job that ran NLP extraction, those results are made available in Data Curation under Augment with Information Extraction, then Use existing job. The run covers every pipeline the ingestion step ran, including the OMOP NER pipeline, and a dataset built from several ingestion jobs produces one combined run.

Deleting the run removes only the copy held in Data Curation. The underlying ingestion job results are untouched and the run can be prepared again from the same dataset.

The ontology is the extraction schema. It specifies which fields to capture, what data type each field expects, which fields are required, and what extraction guidance applies. The wizard shows an ontology's fields, categories, and guidelines before you select it, and any custom ontology can be used.

Into the target database you select at step 3 of the wizard, which is where curated data becomes available for downstream analytics, reporting, and research. The job name given at the same step identifies the run in the dashboard and in reports.