Continuous NER Training with Active Learning in Generative AI Lab

On the first day of a NER project, the model assisting the annotators knows clinical language but nothing about the project. It has not seen how this organization defines a Diagnosis, where annotators draw the boundary of a Medication entity, or how the guidelines handle ambiguous cases.

Five hundred annotated notes later, all of that has been decided, documented, and validated by the team.

Unless the model is retrained during annotation, note 501 receives the same assistance as note 1.

In Generative AI Lab, Active Learning is automatic retraining. Once a set number of new human completions accumulates, the platform retrains the project’s NER model and, if configured, deploys the new model back into annotation. It does not choose which tasks annotators label next, which is what active learning means in the machine learning literature.

Every completed annotation is also training data

Annotators review documents, identify entities and correct predictions to produce a validated dataset for model training. Each completed task also records how the project applies its guidelines.

But every completed task also captures knowledge about how the project should work. Annotators establish where entities begin and end. They apply project-specific definitions. They resolve ambiguous mentions according to annotation guidelines. They correct predictions that do not match the requirements.

In a linear workflow, that information reaches the model only after annotation is finished:

Annotate → Complete Dataset → Train Model → Deploy Model

In that sequence, the model pre-annotating documents stays unchanged for the whole project while the validated dataset grows.

Retraining mid-project adapts the model to local guidelines

Consider a project extracting diagnoses, medications, tests, and treatments from clinical notes.

A pretrained clinical NER model recognizes common clinical terminology. Each project differs from its training data: Documentation styles vary between institutions and specialties. Annotation guidelines define entities differently. Some projects need broad clinical concepts; others require more granular distinctions.

Human review records those differences. The first 50 completions give the model an initial set of project-specific examples, and the next 100 add new terminology and edge cases. Each later batch adds more examples of every guideline decision.

An iterative workflow feeds validated completions back into model training while annotation continues:

Model predictions → Human annotation and correction → Validated completions → Model training → Updated model predictions

Annotators still create and validate the ground truth. The model trains on it before the project ends.

How Generative AI Lab turns annotations into an iterative training loop

Generative AI Lab integrates Active Learning directly into NER model training.

Project Owners and Managers can enable Active Learning from the Model Training tab and choose how many new completions should accumulate before training is triggered. Generative AI Lab currently supports thresholds of 50, 100, or 200 new completions. Once the selected threshold is reached, NER training starts automatically.

With a threshold of 100, a team annotates and validates 100 tasks, which triggers a training cycle, and keeps working until the next 100 completions trigger another. Each cycle trains a new model version on all validated completions so far.

Without Active Learning, someone on the team has to decide when enough new annotations exist, start training by hand, and repeat the process as the project grows.

Automated retraining helps most on long-running annotation projects, where training data is created incrementally rather than delivered as one finished dataset. Instead of depending on whether someone remembered to retrain the model, the retraining cadence becomes part of the workflow itself.

Choose when a newly trained model returns to annotation

When Active Learning is enabled, Generative AI Lab lets teams decide whether newly trained models should be deployed automatically after training. If automatic deployment is enabled, Generative AI Lab updates the Project Configuration for the labels included in the newly trained model so that the project uses it. Teams that prefer more control can leave automatic deployment disabled, in which case trained models remain available for on-demand selection from the project’s predefined labels, with their training date and time displayed.

Automatic deployment suits projects that want each new model in pre-annotation as soon as training finishes. Manual deployment suits projects that validate a model before annotators see its predictions.

Retraining can reduce correction work on later batches

A model retrained on the project’s validated completions can produce pre-annotations that better reflect the annotation guidelines than the initial pretrained model. When this happens, reviewers spend less time correcting entity labels and boundaries and more time on the remaining difficult cases.

Closer pre-annotations mean less to correct. Rather than adjusting boundaries and relabeling entities on nearly every document, annotators confirm predictions that are already largely right and spend their attention on the cases the model still gets wrong. That effort compounds across the project. The first batch is annotated with assistance from a generic clinical model. Later batches are annotated with assistance from a model shaped by the project’s own reviewed examples. With each training cycle the pre-annotations improve, and the time an annotator spends on a document near the end of the project is typically lower than the same document would have taken at the beginning.

Retraining teaches project-specific entity boundaries

Clinical language models provide a valuable starting point because they already recognize terminology and patterns common to healthcare documents. But a generic clinical model cannot know every decision encoded in a project’s annotation guidelines.

Suppose two teams encounter:

severe chronic systolic heart failure

One project may annotate the entire phrase as a single clinical problem. Another may follow guidelines that require different entity boundaries or labels. Neither convention can be inferred from clinical terminology alone. It comes from the annotation task itself, and human completions provide examples of those decisions.

As more reviewed annotations enter the training dataset, retraining gives the model an opportunity to adapt to the project’s actual labeled examples rather than relying exclusively on its original training. The objective is not to produce another version of a generic NER model. It is to make the model assisting annotators increasingly reflect the data and annotation decisions being produced inside the project.

Annotation and model training become one process

With Active Learning, annotation and model training stop being two phases separated by the end of a project. The human establishes ground truth, the validated completions become training data, and the retrained model returns to assist the next batch.

Active Learning keeps validated human annotation at the center of model development. As completions accumulate, they become training data for new NER model versions that can be returned to the same project. The result is a repeatable feedback loop in which annotation generates both the final dataset and the examples used to adapt the model to the project’s own guidelines.

Frequently asked questions

Q: What is Active Learning in Generative AI Lab?

Active Learning connects ongoing NER annotation with iterative model training. When a configured number of new completions is reached, Generative AI Lab automatically triggers NER training using the growing set of project completions.

Q: How often can Generative AI Lab retrain the model?

Project Owners or Managers can configure training to trigger after every 50, 100, or 200 new completions.

Q: Is a newly trained model automatically deployed?

It can be, but it does not have to be. When enabling Active Learning, teams choose whether the newly trained model should be deployed after training. If automatic deployment is disabled, the trained model can be selected and added to the project configuration later.

Q: Who can enable Active Learning?

Active Learning can be enabled by Project Owners or Managers from the Model Training tab.

Q: Does Active Learning remove the need for human annotation?

No. The workflow depends on human completions accumulating within the project. Those completions trigger subsequent NER training cycles, keeping validated human annotation at the center of the iterative training process.

Learn more

To see Active Learning in action, watch the Generative AI Lab demo or start a free trial.