Introducing The Specialized Intelligence Index

Introducing The Specialized Intelligence Index

Join us for our inaugural conference, Forge 2026

Blog

Introducing The Specialized Intelligence Index Introducing The Specialized Intelligence Index

PUBLISHED 9/22/2026

Table of Contents The measure of real work

Developed by practitioners, for practitioners SII highlights at a glance

Healthcare

Legal

Finance

Cybersecurity

Customer Support

Productivity

Software Methodology overview Contribute a benchmark or a model

Table of Contents

Define AI performance in your industry

Join the index

The Specialized Intelligence Index (SII) is your one-stop destination to explore the performance of open, closed, and specialized models on domain-specific benchmarks. Each benchmark reflects real-world tasks designed by practitioners. Today, we are launching benchmarks in seven initial domains: healthcare, legal, cybersecurity, finance, customer support, productivity, and software. More are coming soon. The measure of real work

Public benchmarks provide common reference points for tracking progress and comparing models, but they are not a good measure of real work. These evals use bounded tasks, fixed datasets, and standardized scoring. Real work is messier. It involves incomplete information, changing scope, business constraints, complex judgment calls, multi-step workflows, and collaboration. This distinction matters for organizations seeking to determine if a model is good enough to automate human tasks. An acceptable result must satisfy the standards of real people responsible for real outcomes. Earlier this year, METR quantified the difference. In their work, 4 maintainers reviewed 296 AI-generated pull requests (PRs) from 3 SWE-bench Verified repositories. Maintainer acceptance scores averaged 24.2 percentage points below automated benchmark scores. In their own words: “many SWE-bench-passing PRs would not be merged into main.” To apply benchmarks to real work, we need to establish what a score measures, how closely the evaluation reflects the intended work, and whether better performance produces a useful operational result. Does the test measure the capability it claims? This is a question of construct validity: whether the evaluation supports the interpretation attached to its score. Bean and colleagues examined 445 LLM benchmarks and identified recurring gaps between the phenomena researchers intended to measure, their tasks, and their scoring methods. For example, a task intended to measure reasoning may also depend on memorized knowledge. This makes it difficult to determine whether a high score reflects reasoning, recall, or both. Does the test represent the work we care about? Representativeness concerns the coverage and composition of the task set. Wang and colleagues studied 43 agent benchmarks and found a concentration in computer and mathematical work, a category accounting for 7.6% of U.S. employment in their analysis. Management and legal work were underrepresented, as were interpersonal skills common across occupations. Real-work benchmarks must represent their intended domain. Evaluating models on real work then follows a logical progression, with each step requiring more evidence: Benchmark score → Capability claim → Business outcome The score records performance on a defined task set under a specified protocol. A capability claim requires evidence that the system can perform the relevant class of work reliably, including on unfamiliar cases. A business outcome requires evidence that this performance delivers the desired result at acceptable quality and cost. Three benchmarks across Software, Customer Support, and Finance. Visit the SII to explore the scores. Developed by practitioners, for practitioners

Real-work evals are defined by practitioners who help outline the work, the constraints, and the conditions for acceptance. Designing real-work evals generally involves these steps: 1. Define the job and its value. Specify the task, intended users, and level of human oversight. Set quality thresholds and time and cost limits. Establish a baseline for the current workflow, then test whether score improvements predict better outcomes in a pilot or controlled deployment. 2. Reflect the work. Sample routine tasks and difficult cases from the intended setting. Include realistic information, tools, permissions, and policy constraints. Add stress tests for consequential failures, but report them separately when their frequency differs from normal usage. A deliberately difficult test set should not be presented as an estimate of everyday performance. 3. Set acceptance criteria with practitioners. Translate professional standards into observable outcomes and explicit rubrics. Distinguish minor defects from failures that make an output unacceptable. Evaluate both the final result and any actions that matter, such as seeking approval before changing a protected resource. 4. Validate the grader. Compare automated judgments with expert review. Examine false acceptances, false rejections, and disagreements among reviewers. Refine the rubric or grading method where those differences reveal ambiguity. Continue sampling outputs for expert review as the system changes. 5. Test generalization and reliability. Keep development and held-out cases separate, check for contamination, and refresh the evaluation as the work changes. Repeat runs and report uncertainty, performance by task category, and critical failure rates. Record the model, prompts, agent harness, tools, resource limits, and grader version so comparisons remain interpretable. SII highlights at a glance

Healthcare

Doximity’s BedsideBench v0.2.0 evaluates frontier AI models across 500 physician-validated clinical cases spanning medical reasoning, calculations, drug safety, guideline adherence, hallucination, diagnostic safety, and treatment planning. Mercor’s APEX-1: General Practitioner (MD) Benchmark measures how well frontier AI models perform on real primary care physician tasks in diagnosis, workup, and safe escalation. HealthBench Professional evaluates whether frontier AI models can provide accurate, useful, and safe responses to challenging clinician-authored tasks. Legal

Harvey's Legal Agent Benchmark (LAB) measures how well frontier AI models perform on real legal work across 24 practice areas, requiring them to navigate files and produce work products graded against expert rubrics for factual accuracy, legal analysis, and format. Harvey’s LAB...