Building Production Ai For Regulated Industries With A Leading Digital Insurer

Building production AI for regulated industries with a leading digital insurer Announcing our Series F . Learn more

AI engineering

Building production AI for regulated industries with a leading digital insurer

From frontier OpenAI/Google models to open-source — delivering 8x the speed and outperforming GPT-5-level accuracy.

Authors

Charles O'Neill

Jonathon Liu

Last updated October 20, 2025

Share

TL;DR When this leading digital insurer approached Baseten, they had a clear challenge: build an AI system capable of handling customer insurance queries with perfect regulatory compliance, deep policy knowledge, and sub-3-second response times. The constraint is that their industry operates under strict Financial Conduct Authority (FCA) oversight, where a single hallucination or policy misinterpretation could result in regulatory sanctions. For instance, claiming to have “competitive” pricing without robust evidence of this can lead to multi-million dollar lawsuits. Insurance is a very spiky and chaotic task in this regard; slight perturbations of seemingly similar responses can really screw you. Six weeks later, we deployed a fine-tuned 32B parameter open-source model that outperforms all closed-source models (including GPT-5 with maximum reasoning mode) with maximum reasoning while responding in 2.88 seconds, which is 8.7x faster than Gemini-2.5-Pro and 4.2x faster than GPT-5.

Compliance Meets Conversational AI Insurance is uniquely challenging for language models. Consider a customer asking: "Am I covered if my phone gets stolen while I'm traveling in Thailand?" A correct answer requires: Retrieving the precise policy wording from complex legal documents

Understanding multi-layered eligibility criteria (coverage tier, destination, item limits)

Maintaining FCA-compliant language throughout the conversation

Never hallucinating coverage that doesn't exist

Handling edge cases with appropriate caveats

Traditional approaches fail here. General-purpose models hallucinate. RAG systems retrieve irrelevant chunks. Fine-tuned models lose reasoning capability. The customer needs accuracy, compliance, and speed simultaneously. Phase 1: Building the Retrieval System We began by constructing a state-of-the-art knowledge retrieval system. This wasn't a basic vector database—we implemented: Dynamic chunking strategies that preserve semantic coherence across policy sections

Hybrid retrieval combining dense embeddings with BM25 sparse retrieval

Multi-stage reranking using cross-encoders to surface the most relevant passages

Synthetic benchmark generation : 10,000 question-answer pairs derived from policy documents

Our initial benchmarking showed 94.3% retrieval accuracy on synthetic queries—a strong foundation, but only the beginning. ✕ Different RAG methods and our optimised performance ✕ Latency of different RAG methods Phase 2: The Evaluation Framework This is where our approach diverged from most people’s attempts to fine-tune/optimise a model. Typically, a customer might try something like generating a bunch of GPT-5 responses in data they already have, and end up disappointed when the fine-tuned model still has a significant gap to GPT-5 quality (we call this the distillation gap ). So we need a way to start above the quality of GPT-5 for optimising, regardless of if we use SFT or RL; our learning signal needs to be cleaner than “just slowly learn what GPT-5 would do”. However, to do this, we first need to actually decide on what quality means. Rather than manually defining success criteria, we use Baseten's internal evaluation harness constructor that discovers failure modes systematically. From a high level, the constructor functions as follows: Spectrum sampling : We ran 10,000 customer-style questions through eight models spanning the capability range, from Llama-3-8B (deliberately terrible) to GPT-5 with extended thinking and Gemini-2.5-Pro at maximum reasoning tokens.

Meta-error analysis : Using an ensemble of reasoning models, we analysed every response for failure modes: hallucinations, policy misinterpretations, compliance violations, inappropriate caveats, missing disclaimers. Just any general part of the output that was wrong, contradicted the instructions in the input, or wasn’t up to scratch, gets recorded.

Hierarchical clustering : We performed clustering analysis on the error corpus to identify semantic patterns—not just "what went wrong" but "what types of failures occur across the model spectrum."

Evaluation synthesis : From the clustered error space, we derived seven specialised evaluation prompts. Each prompt acts as an LLM-as-judge for a specific failure mode (e.g., "Does the response hallucinate coverage not present in retrieved policy text?", "Does the response use FCA-compliant language for pre-contractual information?"). We have pretty strong opinions on what makes a good LLM-as-judge prompt, and we have baked these into the prompt generation pipeline.

At this stage, we also do a bunch of analysis to ensure the consistency and transitivity of the evaluation prompts with the big reasoning models ie what’s the variance in pass rate when you feed it the same question+answer multiple times (a perfect evaluator should be zero)? Can it distinguish between really good models and really bad models? The Meta-Evaluation Loop Here's where it gets interesting. Evaluations themselves can be wrong ie misaligned with task requirements or missing critical checks. Our constructor also has a meta-evaluation process: Conflict detection : Identify when evaluation prompts contradict the generation prompt or task specification

Coverage analysis : Determine if the evaluation set comprehensively covers all failure modes

Expert alignment : If there are ambiguities in the task specification (ie provided generation prompt and provided customer context), we collect these as a list of questions which get sent back to the customer (in this case a leading digital insurer) to clarify.

We also provide some “gold” outputs (optimised with our process discussed in the next section) for this customer to examine; these gold outputs are perfect under the existing evaluators. When this digital insurer identified a compliance issue our evaluators missed, we refined the evaluation prompts. When evaluation prompts flagged responses that the digital insurer considered acceptable, we traced the misalignment back to the task specification. The result is a...