Announcing the Databricks Big Book of AgentOps

Announcing the Databricks Big Book of AgentOps | Databricks Blog Skip to main content

Summary

AgentOps is the operating discipline for building, evaluating, governing and improving AI agents in production.

The Big Book of AgentOps provides practical guidance on architecture, evaluation, observability, cost management and stakeholder alignment.

The book provides a blueprint that combines field-tested architectures, technical best practices, stakeholder alignment playbooks, and highlights common pitfalls to avoid.

What is AgentOps? AgentOps is the operating discipline for building, deploying and improving AI agents in production. It brings architecture, evaluation, observability, governance, security and cost management together into a process teams can repeat. An AI agent is more than a model producing a response. Agents can choose tools at runtime, retrieve enterprise data, call APIs and work through a multi-step task on their own. Every one of those capabilities is a place where something can go wrong, such as a bad tool call, an overbroad permission or a cost spike nobody planned for. AgentOps exists to keep that complexity from becoming a liability. Done well, it makes a system reliable enough that people trust it and simple enough that a team can actually run it. Why do AI agents need AgentOps? Generative AI has moved from experimentation into the enterprise faster than most technology waves. The next challenge is turning promising pilots into systems people can depend on. Most teams stall on the same operational questions: Is the agent producing the right result for this task? Can we trace what it did, including the tools and data it used? How do we control access to sensitive data and actions? What will a request cost once it triggers several model calls, retries or guardrail checks? Who decides the agent is ready to ship, and who’s watching once it’s live?

Answering those questions takes more than a stronger model. It takes an operating model for the agent itself. This is familiar territory. MLOps matured as teams moved machine learning models out of notebooks and into production. LLMOps followed, adding practices for prompt and model versioning, distributed serving, and cost control. AgentOps is the next layer and extends that discipline to systems that reason, use tools, and take action on their own. A production agent needs clear boundaries on what its tools can touch, a traceable record of multi-step execution, a way to measure quality, a plan for what happens when something fails, and enough alignment across engineering, product, security, compliance, and finance that nobody is surprised when it ships. Customer experience bears this out. FactSet’s text-to-code knowledge agent evolved from a single foundation model into a full agent system and delivered a 44% improvement in accuracy. Read the FactSet story . The Big Book of AgentOps codifies the practices that help enterprise teams make that transition: architecture patterns, a phased delivery pipeline, evaluation and feedback loops, governance, cost management and the stakeholder decisions that determine whether an agent actually reaches production. What’s inside The Big Book of AgentOps The book moves from concepts to implementation across six chapters. 1. Understanding AI agent architectures Agents are not the same as a prompt-and-response LLM call. Logging, evaluation gates, governance, rollback and monitoring all matter, but requirements shift across four agent architectures. We outline each architecture and their corresponding operational requirements for easy reference. Also, just as important are the anti-patterns that keep pilots from shipping: starting with a use case that’s too broad, reaching for multi-agent orchestration before the complexity is justified, having an unnecessary reasoning loop an leaving evaluation for later than it should be. We share a list of common ones we’ve seen so similar mistakes can be avoided. 2. AI agent deployment architecture patterns Deployment architectures range from simple to complex depending on a use case and an organizations need. We cover four deployment patterns spanning deploying from a single Databricks workspace to the most complex setup: a multi-account, multi-agent enterprise topology. Each pattern comes with guidance on how to select and evolve between patterns as your needs change. At each stage, Unity Catalog , Unity Gateway and MLflow remain at the core of support the architecture. 3. The AgentOps project lifecycle A seven-phase roadmap starting from how to form a team, select a use-case, through to setting up data infrastructure, evaluation loops, and governance best practices. We highlight important things to note for each phase. For example, cost is an important part of the lifecycle. A single user request can trigger several model calls once sub-agents, retries and guardrail checks are taken into account. That makes it critical to attribute usage, set limits and establish clear accountability for spend. 4. Applying DevOps principles to AI agents Teams need to iterate quickly to develop a high-quality agent. They also need to evolve and agent based on developments at the frontier of research and in response to the changing needs of their organization. To address this, we highlight how the principles of flow, feedback and continuous-learning, taken from The DevOps Handbook provide a useful foundation for operating AI agent systems. Applying these principles to agent systems means building a golden evaluation dataset from real traces, calibrating automated judges against subject-matter-expert feedback and using evaluation results to drive what gets built next. A worked example of a customer email agent shows human review, model-based judges and rules-based checks working together without turning every release into a manual audit. 5. How to operationalize AI agents A six-step planning sequence helps teams put effort where it actually changes the outcome: map the human workflow, translate it into a technical architecture, define observability needs by persona, design tracing into the system, map access controls to data and tools and identify what can be reused. A telecommunications customer-support agent puts the sequence into practice, down to the data schemas, the tools available to a billing sub-agent and the fine-grained controls that keep one customer from ever seeing another customer’s data. 6. Managing stakeholders for production AI agents Good...