Outperforming Frontier Models On Emergency Medicine Documentation

Outperforming frontier models on emergency medicine documentation Announcing our Series F . Learn more

AI engineering

Outperforming frontier models on emergency medicine documentation

We trained a specialist model that beats frontier Gemini models on clinical documentation and runs 6–8x faster.

Authors

Harry Partridge

Charles O'Neill

Mudith Jayasekara

Last updated March 2, 2026

Share

TL;DR A company that builds ambient scribes that convert emergency department (ED) conversations into structured clinical charts came to us with a specific challenge: build a model capable of handling the complexity of emergency medicine documentation while being fast enough for real-time use. The task involves two stages. First, convert (often messy) ED transcripts into structured JSON with 16 distinct sections. Second, merge that JSON with physical exam templates to produce the final chart. Each stage has dozens of interlocking rules that must be followed precisely. This is quite scary because getting things wrong means the model has introduced liability issues, as well as just generating a bad note. We built a model that achieves 84.8% accuracy on chart generation and 67.2% on summarization, compared to gemini-2.5-pro's 63.6% and 47.2%. It also runs 6-8x faster.

The problem: emergency documentation is complex Emergency physicians spend 2-3 hours per shift on documentation. In addition, emergency department notes aren't like other medical documentation. Information comes from multiple sources simultaneously: the patient, EMS, family members, and prior records. The clinician must document not just what happened but also what didn't happen, in particular by explicitly ruling out dangerous diagnoses. The system needed to handle: Complex routing logic where information is redistributed across sections based on priority

Conditional inclusion rules (e.g., include transport mode only if not self-transport)

Template modification that saves non-contradicted normal findings while adding new observations

High-risk diagnosis detection that matches chief complaints to potential emergencies and tracks rule-out criteria

General-purpose LLMs consistently failed at these tasks. They'd hallucinate provider names, place physical exam findings in the history section, or worse, would miss critical diagnoses that hadn't been ruled out. And frontier models failed at emergency documentation.

And it’s true, frontier models actually do fail a lot at emergency documentation. They need to maintain multiple concurrent constraints while preserving information semantics, which in some sense is actually quite out of distribution for them. When gemini-2.5-pro sees “regular rate, regular rhythm, no murmurs” and needs to incorporate tachycardia, it faces a constraint satisfaction problem: preserve non-contradicted facts while surgically removing contradicted ones. This requires understanding medical relationships (rate ≠ murmurs) plus precise text manipulation. Most models do one or the other; few do both. Building evaluators for emergency medicine We built comprehensive evaluation frameworks using Lumina for both stages of the pipeline. (Read more about our LLM-as-judge evaluation construction with Lumina here .) Our client provided 120 micro-checks they used for quality assurance, i.e., specific failure modes they'd identified over months of production use. We semantically partitioned these into our evaluator framework, ensuring complete coverage while organizing them into coherent evaluation dimensions, as well as supplementing them with additional errors and holistic checks for quality. For each stage – summarization and chart generation – we created five evaluators. Each evaluator uses binary pass/fail scoring on specific criteria. A chart passes only if it satisfies all requirements.

We also needed to validate the evaluators themselves. We ran a meta-evaluation process where we generated “perfect” outputs according to our evaluators, and had our customer's clinical team review these outputs. When they found issues our evaluators missed, we refined the evaluation prompts. When evaluators flagged clinically acceptable outputs, we traced misalignment back to task specifications. We also provided questions about ambiguities surfaced by Lumina for the customer to answer. This iteration continued until our evaluators aligned with expert clinical judgment. Simplifying the pipeline The original system used multiple prompts across different stages: separate prompts for each summarization section, another for PE template selection, and more for chart generation. This created compounding errors and latency. We consolidated this into two stages: Stage 1 : One prompt handles all 16 sections of summarization

Stage 2 : One prompt handles complete chart generation

For PE template selection, we noticed the LLM was essentially pattern-matching against rules. We replaced this with deterministic Python code, eliminating an unnecessary LLM call while improving accuracy to 100%. This simplification was only possible because we made the first stage robust enough to produce consistent, well-structured outputs that the second stage could reliably process. The training approach We used iterative SFT to train our model, which in this case is qwen3-32b , a dense model (we are also currently training an MoE model for the same task, qwen3-next-80b-a3b , which significantly speeds up inference at the cost of memory). For each training example, we: Generate initial output from base model

Run all evaluators to identify failures

Use gemini-2.5-pro to repair the output based on specific failure feedback

Repeat until all evaluators pass

Train on these perfect outputs

This process is more sample-efficient than standard SFT because we're training on outputs that score higher than what gemini-2.5-pro produces naturally. Each training example provides dense supervisory signal about what went wrong and how to fix it. We also enhanced our training data through prompt mutation , which is systematically varying the input format while preserving semantic content. This prevented overfitting to specific phrasings. For the chart generation stage, we added robustness by training on both perfect JSON outputs and unrefined outputs from the summarization stage. This ensures the model handles imperfect inputs gracefully in production. We include two examples of how iterative SFT (iSFT) allows us to bake in...