SOP-Bench: A new benchmark for evaluating AI agents on real business procedures

SOP-Bench: A new benchmark for evaluating AI agents on real business procedures - Amazon Science

Close

Close

Social

bluesky

threads

twitter

instagram

youtube

facebook

linkedin

github

rss

Menu

Research

Research areas

Automated reasoning

Cloud and systems

Computer vision

Conversational AI

Economics

Information and knowledge management

Machine learning

Operations research and optimization

Quantum technologies

Robotics

Search and information retrieval

Security, privacy, and abuse prevention

Sustainability

Our scientific contributions

Publications

Research from our scientists and collaborators.

Conferences

Our experts present and discuss cutting-edge research at scientific meetings globally.

Research areas

Automated reasoning

Cloud and systems

Computer vision

Conversational AI

Economics

Information and knowledge management

Machine learning

Operations research and optimization

Quantum technologies

Robotics

Search and information retrieval

Security, privacy, and abuse prevention

Sustainability

Our scientific contributions

Publications

Research from our scientists and collaborators.

Conferences

Our experts present and discuss cutting-edge research at scientific meetings globally.

News & blog

The latest from Amazon researchers

Amazon Science Blog

Technical deep-dives and perspectives from our scientists.

News

Research milestones and recent achievements.

The latest from Amazon researchers

Amazon Science Blog

Technical deep-dives and perspectives from our scientists.

News

Research milestones and recent achievements.

Collaborations

Amazon Research Awards

Overview

Call for proposals

Latest news

Research stories

Recipients

Amazon Nova AI Challenge

Overview

Rules

FAQs

Teams

Research collaborations

Overview

Carnegie Mellon University

Columbia University

Hampton University

Howard University

IIT Bombay

Johns Hopkins University

Max Planck Society

MIT

Tennessee State University

University of California, Los Angeles

University of Illinois Urbana-Champaign

University of Southern California

University of Texas at Austin

Virginia Tech

University of Washington

Amazon Research Awards

Overview

Call for proposals

Latest news

Research stories

Recipients

Amazon Nova AI Challenge

Overview

Rules

FAQs

Teams

Research collaborations

Overview

Carnegie Mellon University

Columbia University

Hampton University

Howard University

IIT Bombay

Johns Hopkins University

Max Planck Society

MIT

Tennessee State University

University of California, Los Angeles

University of Illinois Urbana-Champaign

University of Southern California

University of Texas at Austin

Virginia Tech

University of Washington

Resources

Code and datasets

Amazon Nova

Try Amazon’s frontier foundation models.

Code and datasets

Amazon Nova

Try Amazon’s frontier foundation models.

Careers

Careers

Explore our open roles.

Amazon Scholars

Faculty research opportunities on industry-scale technical challenges.

Postdoctoral Science Program

Early-career research opportunities alongside experienced industry scientists.

Careers

Explore our open roles.

Amazon Scholars

Faculty research opportunities on industry-scale technical challenges.

Postdoctoral Science Program

Early-career research opportunities alongside experienced industry scientists.

Search

Submit Search

Conversational AI

SOP-Bench: A new benchmark for evaluating AI agents on real business procedures

Extendable framework enables testing agents on the full set of capabilities required to successfully complete a procedure, not isolated proxy tasks.

By Rohith Nama , Nandi Subhrangshu

August 21, 2026

9 min read

Share

Share

Copy link

Email

X

LinkedIn

Facebook

Line

Reddit

QZone

Sina Weibo

WeChat

WhatsApp

分享到微信

x

Key takeaways

SOP-Bench is an open benchmark that measures AI agent performance on authentic standard operating procedures (SOPs) across twelve business domains, featuring over 2,000 tasks paired with functioning tools and ground-truth answers for objective evaluation. Existing agent benchmarks fall short by testing isolated capabilities with clean, machine-formatted prompts, while real SOPs require coordinated multi-tool use, ambiguity interpretation, and domain expertise that previous datasets lack. Testing across eleven frontier models revealed that newer models don't always perform better, additional tools can reduce success rates, and no single model-agent combination excels across all procedures, requiring task-specific evaluation before production deployment. The benchmark framework allows teams to evaluate custom agents against existing procedures or extend it to new domains using a structured approach combining expert-authored SOPs, generated tool interfaces, and reproducible grading against ground truth.

Was this answer helpful?

A standard operating procedure, or SOP, is the written set of steps an organization follows to correctly complete an important piece of routine work the same way every time. Almost every industry runs on SOPs. A hospital uses one to register a new patient, a logistics team uses one to decide whether a shipment qualifies as hazardous, a bank uses one to verify a new business customer, and a trust and safety team uses one to decide whether to remove a piece of content. SOPs carry an organization's hard-won knowledge, its compliance rules, and its decision logic in a form that any trained person can adopt and follow. As a result, they keep operations consistent and safe across different employees, shifts, and sites. SOPs are hard for AI agents to execute because they look cleaner than they actually are. A real procedure asks the reader to interpret instructions that were never fully spelled out, to draw upon knowledge that everyone in the field already shares, and to make judgment calls as conditions change. Consider the following passage from a patient intake procedure:

An excerpt from a patient intake SOP.

Steps four and six tell the operator to verify the patient's insurance, without saying how the verification should be done or why it needs to be done twice. Someone who has worked an intake desk, however, knows that the first step confirms the patient’s coverage with the insurer, and the second step confirms that the patient’s information is correctly entered into the medical provider’s management system. An agent has none of that background, so it must guess what verification means here, remember what...