Sub 3 Millisecond Named Entity Recognition Ner Inference

Sub-3 millisecond named entity recognition (NER) inference Announcing our Series F . Learn more

Model performance

Sub-3 millisecond named entity recognition (NER) inference

Baseten Embeddings Inference (BEI) now supports NER inference at sub-3 ms client-side and 1 ms server-side, 7x faster than typical solutions.

Authors

Michael Feil

David Oy

Last updated April 6, 2026

Share

Baseten Embeddings Inference (BEI) started with a simple goal: to make encoder inference so fast that it vanishes into the background of your application. In service of this goal, we have expanded BEI beyond embeddings to include named entity recognition (NER): a token-classification workload that shows up everywhere in production, from document ingestion and support analytics to compliance pipelines, routing, and retrieval augmentation. With BEI’s serving path and runtime optimizations, we can run common BERT-class NER models with sub-3 millisecond client-side inference and 1 millisecond server-side inference under realistic production conditions. ✕ Using BEI, named entity recognition (NER) inference runs at 1 ms P50 and 3 ms P99 latency on the server side, and 3 ms P50 and 4 ms P99 latency end-to-end (including network latency, which can be variable). In comparison, an optimized PyTorch model implementation (transformers + the open-source Baseten Performance Client, which improves throughput up to 12x) achieves 16 ms P50 and 23 ms P99 latency (without the Performance Client we see up to 50 ms P99 latency, but use the more conservative estimate here). The BEI implementation is ~7.7x faster. This post covers: What NER models do (and why NER inference has different requirements than embeddings)

What we changed in BEI to make NER inference so fast

How to deploy an NER model with BEI

How to inspect token-level predictions

Optimizing NER inference: NER vs. embeddings Named entity recognition (NER) models turn free-form text into structured data by identifying character spans that represent entities such as people, organizations, locations, and dates. Most modern NER models are implemented as token classification: Tokenize the input text into word pieces (e.g., WordPiece/BPE).

Run an encoder model (often from the BERT family).

Produce a label distribution for every token (e.g., B-PER, I-ORG, O, etc.).

Optionally post-process token labels into entity spans aligned with the original string.

Unlike embedding models (which output one vector per input text), NER returns a prediction per token. That makes the inference compute relatively small compared to the overhead around inference (tokenization, HTTP parsing, JSON serialization, connection behavior, proxying, and framework overhead). Once the model is fast enough, the real work becomes removing everything else from the hot path. How we get sub-3 millisecond NER inference To achieve sub-3 ms NER inference, we focused on removing latency at every layer of the serving stack: the web framework and tokenizer, the proxy, connection handling, and the deployment topology. Low-overhead serving with Rust BEI uses a Rust-based web server and tokenizer stack designed for high-throughput encoder workloads. NER benefits even more than embeddings here, because token classification requests can be small enough that a Python web framework’s overhead dominates the end-to-end latency. By keeping tokenization and request handling in Rust, we minimize per-request overhead and keep CPU utilization predictable under concurrency. Nginx is configured to stay out of the way At sub-millisecond inference, “just the proxy” can become the bottleneck. We configure Nginx to reduce overhead on the serving path, minimizing unnecessary buffering and avoiding disk-backed behavior that can introduce latency variance. The goal is simple: keep the request path memory-backed and lightweight, so the model runtime is what you’re measuring — not incidental IO. HTTP/2 connection pooling in the Baseten Performance Client If you benchmark without connection reuse, you’re mostly measuring TLS handshakes, socket setup, and request lifecycle overhead. That’s not what production looks like, and it’s especially misleading when the model itself is extremely fast. Our Performance Client uses HTTP/2 connection pooling, so repeated requests reuse connections, and your latency reflects inference + minimal request handling rather than repeated setup costs. Geographical co-location If you’re calling your NER pipeline as part of a larger processing pipeline, e.g., for anonymization or query understanding, running it in the same physical datacenter can help reduce latency. Baseten has products like Chains that allow co-location of Baseten deployments. Calling /predict_tokens vs /hello_world in FastAPI A helpful mental model: A minimal FastAPI route like /hello_world is effectively a benchmark of framework overhead.

A Python-served NER endpoint often adds tokenization, model runtime, and serialization on top of that.

BEI’s /predict_tokens is designed to keep overhead low enough that token classification can run at the speed the hardware allows.

When you’re chasing sub-millisecond inference, these differences matter: at that point, even “small” overhead (routing layers, connection setup, JSON parsing/encoding) can dominate end-to-end latency. How to deploy an NER model with BEI Below is an example configuration for deploying a BERT-family NER model checkpoint via BEI on an L4 GPU. It sets BEI’s default route to /predict_tokens , which returns token-level predictions. 1 model_metadata: 2 example_model_input: 3 inputs: 4 - - Apple is looking at buying U.K. startup for $1 billion 5 - - John works at Google in Mountain View, California 6 raw_scores: true 7 truncate: true 8 truncation_direction: Right 9 model_name: BEI-Bert-ner-bert-base-ner-uncased-truss-example 10 python_version: py39 11 resources: 12 accelerator: L4 13 cpu: "1" 14 memory: 10Gi 15 use_gpu: true 16 trt_llm: 17 build: 18 base_model: encoder_bert 19 checkpoint_repository: 20 repo: baseten-admin/bert-base-ner-uncased 21 revision: main 22 source: HF 23 max_num_tokens: 16384 24 runtime: 25 webserver_default_route: /predict_tokens A few notes: base_model: encoder_bert selects the encoder runtime path suitable for token classification.

webserver_default_route: /predict_tokens makes the model behave like a dedicated token prediction endpoint.

raw_scores: true returns per-label probabilities per token (useful if you want to apply confidence...