was successfully added to your cart.

    BBox OCR Benchmark: #1 of 15 Models Including Every Frontier API

    Avatar photo
    Senior Data Scientist at John Snow Labs

    John Snow Labs’ JSL Vision OCR-1B ranked first in the published 15-model FUNSD grounded OCR comparison at 0.039 BBox CER, ahead of Gemini 2.5 Pro at 0.053. Grounded OCR returns recognized text with bounding-box coordinates, so your pipeline can locate words for redaction or field extraction.

    The 1B model supports deployment on a single T4 GPU. The measured 1.8-second result uses H100 via vLLM; the approximately 4-second T4 figure is a separate estimate. The cost and hardware sections distinguish those configurations.

    The comparison includes frontier APIs and traditional OCR engines on the FUNSD dataset. Read the results alongside the matching thresholds, parse rates, and serving configuration described below.

    Summary cards showing JSL Vision OCR at 0.039 BBox CER, 100% parse rate, a CER comparison with Tesseract, and H100 deployment.

    Published results on the evaluated FUNSD pages. The reported 100% parse rate describes parseable output; the 4.7× comparison refers to CER, not a percentage-point accuracy gain.

    What “grounded OCR” looks like

    The example shows the input document on the left and recognized text with bounding boxes on the right.

    Scanned form alongside recognized text regions and bounding boxes returned by JSL Vision OCR.

    Text recognition with bounding-box coordinates. Example names are fictional.

    Downstream code can use the returned coordinates to select text regions for review, field mapping, or masking. Validate recognition and spatial alignment against the requirements of your workflow.

    Why do healthcare workflows need bounding boxes?

    Grounded OCR associates recognized text with coordinates on the source image. In healthcare document processing, those coordinates support:

    • Visual de-identification: locate text regions before masking identified personal information.
    • Form-field extraction: associate text with its position on the form.
    • Audit review: link extracted text to the source region for inspection.
    • Region-level redaction: mask selected areas while retaining the rest of the page.

    The original evaluation reports 6 models that did not emit bounding boxes in the tested setup, including RolmOCR-8B, Nanonets-OCR2, Phi-4-Multimodal, DeepSeek-OCR, and Chandra OCR 2. This is a result for those prompts and configurations; it does not establish that every version or deployment of those models has the same limitation.

    Which models lead the published grounded OCR comparison?

    Grounded OCR podium: JSL Vision OCR 0.039 BBox CER, Gemini 2.5 Pro 0.053, and Gemini 2.5 Flash 0.092.

    Three lowest CERs in the published leaderboard. Lower BBox CER is better.

     

    The three lowest scores in the published CER leaderboard are JSL Vision OCR at 0.039 BBox CER, Gemini 2.5 Pro at 0.053, and Gemini 2.5 Flash at 0.092. JSL Vision OCR’s reported parse rate is 100%.

    Lower BBox CER is better. Parse rate measures whether the output could be parsed; it does not establish that every recognized word or coordinate is correct.

    How do the grounded OCR models compare?

    The table compares BBox CER, parse rates, estimated costs, and per-page latency.

    Model Parameters BBox CER ↓ Parse rate Estimated cost / 1,000 docs Seconds / page
    JSL Vision OCR 1B 0.039 100% ≈$0.56 (T4 estimate); ≈$1.22 (H100) 1.8 (H100 / vLLM)
    Gemini 2.5 Pro Not stated 0.053 100% ≈$6.75 (published API estimate) 50.0
    Gemini 2.5 Flash Not stated 0.092 100% — 12.5
    Claude Sonnet 4.5 Not stated 0.152 100% — 20.8
    Tesseract 5.5 Not applicable 0.180 100% — 1.0
    EasyOCR Not stated 0.237 100% — 2.8
    Qwen3-VL-8B 8B 0.254 91% — 12.8
    Claude Opus 4.5 Not stated 0.530 90% — 20.6
    Grok-4 Not stated 0.582 92% ≈$3.38 (published API estimate) 93.5
    GPT-5.2 Not stated 0.588 100% ≈$9.45 (published API estimate) 30.2
    Sonar Pro Not stated 0.676 100% — 13.5
    Llama-4-Maverick Not stated 0.723 81% — 8.0
    Ministral-14B 14B 0.736 93% — 14.0
    Pixtral-Large Not stated 0.783 12% — 20.0
    docTR (timing chart only) Not stated — — — 1.1

    — means a value is unavailable here, not zero. The CER leaderboard ranks 14 models. docTR appears in the timing chart only and has no CER ranking in this table. Parse rates come from the archived comparison data; latency values come from the timing chart.

    Costs are historical estimates. The $0.56 T4 estimate uses a different configuration from the H100/vLLM timing and approximately $1.22 H100 cost. The cost scatterplot uses H100 for self-hosted inference; some API price points differ from the prose estimates. For Pixtral-Large, the timing chart shows 20.0 seconds per page and the prose gives 69 seconds.

    Spatial errors in this evaluation

    GPT-5.2, Grok-4, and Claude Opus 4.5 each scored above 0.5 BBox CER in the published comparison. For workflows requiring text localization, evaluate the coordinate alignment as well as the extracted text.

    Gemini 2.5 Flash and Pro

    Gemini 2.5 Flash recorded 0.092 BBox CER, compared with 0.053 for Pro. Compare that error-rate difference with the cost and latency assumptions for your document mix.

    Pixtral-Large parse rate

    Pixtral-Large’s reported parse rate was 12%, leaving 88 of 100 pages without parseable bounding-box output. Your pipeline needs to detect and handle such failures.

    What does grounded OCR cost per document?

    The reported compute estimate for JSL Vision OCR on T4 is approximately $0.56 per 1,000 documents, using approximately 4 seconds per page at $0.50 per hour. This is a T4 estimate, separate from the measured H100 latency.

    Using the unrounded hourly rate, the calculation is $0.50 ÷ 3,600 × 4 × 1,000 ≈ $0.56. Include software licensing and operating costs when budgeting for a deployment.

    The original article quotes API estimates of approximately $6.75 per 1,000 documents for Gemini 2.5 Pro, $9.45 for GPT-5.2, and $3.38 for Grok-4. Treat these as historical figures; the chart and prose reflect different reported pricing assumptions.

    Grounded OCR cost-versus-CER scatterplot, with self-hosted H100 estimates and historical API price points.

    Estimated cost per 1,000 documents versus BBox CER. Self-hosted costs use H100; the approximately $0.56 T4 estimate uses a separate configuration. Some API price points differ from the historical estimates in the text.

    How long does grounded OCR take per page?

    For batch ingestion and interactive processing, evaluate per-page latency alongside extraction quality.

    Grounded OCR timing chart showing 15 models, with JSL Vision OCR at 1.8 seconds per page.

    Per-page latency for 15 models. JSL Vision OCR’s 1.8-second result uses H100/vLLM; approximately 4 seconds on T4 is a separate estimate. Pixtral-Large is shown at 20.0 seconds; the prose gives 69 seconds.

    JSL Vision OCR’s reported 1.8 seconds per page is the rounded H100/vLLM result; the more precise figure reported in the hardware section is 1.76 seconds. The approximately 4-second T4 figure belongs to the separate cost estimate.

    The published timing chart places JSL Vision OCR third, behind Tesseract and docTR. It reports 50.0 seconds per page for Gemini 2.5 Pro and 93.5 for Grok-4. The original prose’s rounded Grok-4 figure is 94 seconds.

    Pixtral-Large’s timing is inconsistent: the prose gives 69 seconds per page and the chart shows 20.0 seconds.

    How does JSL Vision OCR compare with traditional engines?

    Tesseract and EasyOCR return text regions and coordinates. This FUNSD comparison evaluates their spatially matched text against JSL Vision OCR using the same reported BBox CER metric.

    BBox CER comparison showing JSL Vision OCR at 0.039, Tesseract at 0.180, and EasyOCR at 0.237.

    Published CER comparison on FUNSD. The 4.7× and 6.1× labels are ratios of character error rates.

    The published comparison gives Tesseract 0.180 CER and EasyOCR 0.237, compared with JSL Vision OCR at 0.039. It reports their relative CERs as approximately 4.7× and 6.1× higher, respectively.

    JSL Vision OCR’s reported parse rate is 100%. These results describe the evaluated FUNSD pages; separate testing is needed for handwriting, other layouts, and additional languages.

     

    How can you deploy grounded OCR in healthcare?

    JSL Vision OCR supports document processing inside your environment. Review the complete deployment against your healthcare data-handling and operational requirements.

    • Data control: local inference supports control over document processing and data movement. HIPAA, GDPR, and SOC 2 requirements also depend on the deployment’s configuration and operating controls.
    • Deployment: Docker or Amazon SageMaker.
    • Hardware: a single T4 for the 1B model or A10G for the 8B configuration. Validate your model and serving configuration before deployment.
    • Repeatability: pin model versions and inference settings, then test repeatability in your own environment.
    • Inputs: PDF, PNG, and JPG, converted to images for inference.

    How did we evaluate grounded OCR accuracy?

    • Dataset: FUNSD, Form Understanding in Noisy Scanned Documents. 100 pages from the training split.
    • BBox CER metric: IoU-based spatial matching (threshold 0.3, score filter 0.15) followed by Levenshtein distance on matched text pairs. Unmatched regions counted as insertion/deletion errors.
    • Closed-source models: Evaluated via OpenRouter API.
    • Open-source VLMs: Evaluated via vLLM v0.17.1 on a single NVIDIA H100.
    • Traditional OCR: Native Python libraries (tesserocr, easyocr).
    • Parse rate: Percentage of pages where the model produced any parseable bounding-box output.

    The evaluation methodology is described above. Evaluation code and per-page JSON caches are archived internally.

    How do CPU and GPU OCR deployments compare?

    We compared H100 GPU and c5.2xlarge CPU inference on the same 5 FUNSD pages with the same prompts. The Transformers runs used bfloat16 on H100 and float32 on CPU. JSL Vision OCR ran separately on H100 via vLLM, so its result also reflects a different serving backend.

    H100 GPU versus c5.2xlarge CPU OCR latency comparison across six models.

    Hardware comparison on 5 FUNSD pages. The Transformers GPU configuration is labeled bfloat16 and the CPU configuration float32; JSL Vision OCR’s vLLM result uses a different serving backend.

    GPU processing was 2–7× faster per page in the Transformers comparisons, reaching 7.2× for 7–8B models. JSL Vision OCR took 1.76 seconds per page on H100 via vLLM, rounded to 1.8 seconds in the latency chart. The approximately 4-second T4 figure is a separate estimate.

    Estimated compute cost per 1,000 OCR documents on H100 GPU and c5.2xlarge CPU.

    Published compute estimates use $2.50 per H100 hour and $0.34 per CPU hour. The approximately $1.22 JSL Vision OCR figure is H100/vLLM; the approximately $0.56 T4 estimate is separate.

    At approximately $0.34 per CPU hour and $2.50 per H100 hour, CPU inference costs less for most models in this comparison despite taking longer per page. JSL Vision OCR’s H100/vLLM compute cost is approximately $1.22 per 1,000 documents. The separate T4 estimate is approximately $0.56 per 1,000 documents, based on approximately 4 seconds per page at $0.50 per hour.

    GPU VRAM and CPU RAM usage during OCR inference across six models.

    GPU VRAM and CPU RAM during inference, using bfloat16 on H100 and float32 on CPU.

    The memory comparison uses bfloat16 on GPU and float32 on CPU. When choosing a deployment, account for inference precision and serving software alongside hardware price and per-page latency. These compute estimates exclude software licensing and operating costs.

    Frequently asked questions

    What does grounded OCR return?

    Grounded OCR returns recognized text together with bounding-box coordinates. Coordinates let downstream code locate text regions for field mapping or redaction. Validate recognition and spatial alignment separately for your workflow.

    How was BBox CER calculated?

    The evaluation uses IoU-based spatial matching with threshold 0.3 and score filter 0.15, followed by Levenshtein distance on matched text pairs. Unmatched regions contribute insertion or deletion errors. Lower BBox CER is better.

    Does a 100% parse rate mean perfect OCR?

    Parse rate is the percentage of pages with any parseable bounding-box output. The model can still make text or location errors. JSL Vision OCR’s published figures are 100% parse rate and 0.039 BBox CER.

    Which hardware produced the 1.8-second result?

    The approximately 1.8-second result refers to H100 inference via vLLM, reported more precisely as 1.76 seconds per page. The approximately 4-second T4 figure is a separate estimate. They should not be used interchangeably in a cost calculation.

    How was the T4 cost estimate calculated?

    The estimate uses $0.50 per hour and approximately 4 seconds per page. At that processing time, $0.50 ÷ 3,600 × 4 × 1,000 is approximately $0.56 per 1,000 documents. This is a compute estimate, excluding licensing and operations.

    Where are the evaluation methods described?

    The methodology section defines the FUNSD dataset, spatial-matching thresholds, and scoring process. Evaluation code and per-page JSON caches are archived internally.

    What’s next

    This is the first article in a three-part OCR benchmark series:

    1. Grounded (BBox) OCR on FUNSD, covered here
    2. Plain-text OCR on FUNSD
    3. Schema-constrained JSON OCR on OmniOCR

    JSL Vision OCR recorded the lowest BBox CER in the published comparison. Assess text recognition, spatial alignment, and serving requirements on documents representative of your own workflow.

    Schedule a demo to evaluate JSL Vision OCR for your document processing pipeline.

    How useful was this post?

    Schedule a demo

    Book now
    Avatar photo
    Senior Data Scientist at John Snow Labs
    Our additional expert:
    Christian Kasim Loan is a computer scientist with over 10 years of coding experience who works for John Snow Labs as a Senior Data Scientist where he helps porting the latest and greatest Machine Learning Models to Spark and created the NLU library.

    Reliable and verified information compiled by our editorial and professional team. John Snow Labs' Editorial Policy.

    Plain-Text OCR Benchmark: The Gap Between On-Prem and API Is Gone

    For clinical search and RAG pipelines, OCR quality determines how reliably you can retrieve information from scanned documents. In John Snow Labs’...
    preloader