Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow

Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow | Databricks Blog Skip to main content

Summary

How Zepto builds evaluation-first AI agents on Databricks and MLflow, using traces, golden datasets, and LLM-as-judge evaluations as core infrastructure for LLM systems.

How a dual-loop architecture—development and production connected by a strict quality gate—provides a concrete pattern to engineer reliability, control cost, and manage risk for large-scale agents.

How this framework achieves 65% lower support costs, and sub-one-month payback, offering AI builders a reusable blueprint for high-impact, production-grade agent design.

Zepto's Push for Reliable, Real-Time Customer Support Zepto is one of India's fastest-growing quick-commerce platforms, with more than thousands of products, a presence in over 60 cities, and delivery windows measured in minutes. In a business where speed is the product, customer support has to move just as fast. To meet that expectation, Zepto runs customer support on a multi-agent AI system that processes over a hundred thousand tickets a day. Early on, the team could build and ship agents quickly. The harder question was how to keep those agents reliable as volume grew, categories expanded, and customer behavior kept changing. Zepto partnered with Databricks to answer that question, not by shipping more agents, but by making evaluation the primary way agents get built, tested, and operated . This blog walks through that journey: the system architecture, the evaluation framework on Databricks and MLflow, the production stories where it earned its keep, and the results and lessons that came out of it. Why "Just Ship the Agent" Breaks at Scale In a high-velocity business, "just ship the agent" works right up until it breaks at scale. At more than 100,000 AI-agent tickets a day, even a 1% error rate creates thousands of bad outcomes and real revenue leakage every single day. The pressure arrived in uneven waves. Weather events, Diwali, and the start of summer drove sharp spikes in ticket volume. Expansion from groceries into apparel, electronics, and beauty introduced new refund, exchange, and return journeys. Meanwhile, a more diverse, multilingual customer base brought a wider range of support requests—and new failure modes surfaced every few weeks. The deeper problem is the assurance gap . Agentic systems operate as multi-step workflows—classifying intent, retrieving knowledge, analyzing inputs, reasoning through decisions, calling transactional tools, and generating responses—so failures can emerge anywhere along the way, not only in the final answer. This assurance gap translated into concrete problems: Failures were invisible until customers complained Fixes were slow The final answer hid internal errors Lacked a principled way to balance cost, performance, and quality for agent Agent design did not capture all critical stakeholder perspectives Reliability was hard to assure in the face of rapid agent evolution

The objective became clear: engineer an evaluation framework on Databricks and MLflow so it functions as core AI infrastructure on which Agents are built and operated . Why Evaluation Framework and its Outcomes A strong evaluation framework directly affects five axes of production readiness: Reliability : system-level guarantees that agents behave correctly across steps, not just “sound right” Velocity : faster, safer iteration on prompts, policies, and models because changes are regression-tested automatically Cost vs Quality vs Performance control : ability to choose optimal models, Prompt strategies or hybrid routing strategies to hit sweet spot on constraints, backed by hard evaluation data Governance : auditable traces, versioned evaluation baselines, and well-defined thresholds for deployment and rollback. Moving decisions from gut-feel (“this version feels better”) to evidence (“this version beats the baseline on agreed metrics”) Stakeholder collaboration : capturing success and reliability criteria from stakeholders perspective, and making trade-offs explicit and measurable for everyone

With Databricks + MLflow as the evaluation backbone and an evaluation-first agent architecture, Zepto achieved Cost & efficiency 80% plus tickets fully managed by AI agents with human oversight 65% reduction in support cost or support tickets Payback period of less than one month

Quality & reputation 20% improvement in customer satisfaction (CSAT) 8% improvement in accuracy

Performance & operations 3x faster development cycles 4x faster time to resolution

Building the Framework: A Dual Loop for Confidence and Control At the core of this approach is the dual-loop model: a development loop and a production loop, connected by a quality gate. This section outlines how those loops work together. Development loop : where you design, iterate, and evaluate agent versions to build with confidence before shipping Production loop : where you monitor live behavior, and detect failures to operate agents with confidence Feedback loop : where production failures fed back to development to enrich next iteration Quality gate : controlling movement between loops, it decides which versions are allowed into production and which are pushed back to development for better iterations

Together, these two loops ensure that agents are built and operated with control. Any failure is automatically captured, fed back, and corrected. As a result, agents are built with confidence, run with control, and continuously improve to handle production failures better over time. The dual loop lies at the core of our framework . Phase 0: Enable Tracing: Transparent Agents by design Every agent invocation emits a rich execution trace that captures prompts, completions, retrieved documents, tool calls, latencies, and decision paths, so the whole workflow is observable at granular level rather than an input and final output. We enabled this with a hybrid approach using MLflow. A single line, mlflow..autolog() , turns on automatic tracing , and the @mlflow.trace decorator adds custom spans wherever we need more detail. Traces are emitted in real-time as OpenTelemetry spans with unique IDs so they stay composable, and MLflow's integration with Unity Catalog centralizes logging into Delta tables. Phase 1: Set Evaluation Dimensions: Pillars and Gates With tracing enabled, the next step is to capture, from each stakeholder’s perspective, “...