Dynamically Allocating Evaluation Effort For Model Ranking 2026 08 04

Dynamically Allocating Evaluation Effort for Model Ranking Skip to content Introducing North 2: AI without compromises Learn more about the updates

Products

Solutions

Resources

Blog

Research

Company

Sign in

Request a demo

Platform North

Enterprise-ready AI for business

Compass

Intelligent search and discovery

Models Command

Generative language models

Transcribe New

Speech recognition model

North Small Translate NEW

Machine translation model

Parse New

Document parsing model

Embed

Semantic representation model

Rerank

Retrieval optimization model

Models Overview

Product Products Overview

Total Cost of AI Ownership

Pricing

Featured North 2: Cohere's latest platform update — with 15+ new features

Deploy Model Vault

Dedicated model inference platform

Private Deployments

On-prem or isolated VPCs

Security

Protect your data at every stage

See deployment options

By Industry Financial Services

Public Sector

Technology

Telecommunications

Energy and Utilities

Healthcare

Manufacturing

Featured Model Vault provides fully-isolated, performant inference with Saas simplicity

Insights Customer Stories

For Developers Developers

Models Overview

Docs

Discord

LLM University

Connect Partners

Events

Webinars

Merch Store

Featured How CoreWeave used Cohere North to transform its customer support in 90 days

Blog

The latest news, launches, and insights

Read more

North 2: Enterprise AI without compromises

Introducing Embed 5—A new family of frontier embedding models

Cohere and Aleph Alpha sign agreement to become the first transatlantic sovereign AI solution

Research Cohere Labs

Cohere’s ML research lab

Explorations Future(s) of Work

How will AI change the way we work?

Aya Models

Multilingual AI at scale

All Papers

Initiatives Research Scholars

Finding the new generation of ML talent

Open Science Community

Championing global, open science

Catalyst Grants

Supporting impactful ML endeavors

Resources Blog

Hugging Face

Events

Featured The future of work debate has an evidence problem

About

Careers

Newsroom

Aug 04, 2026 Dynamically Allocating Evaluation Effort for Model Ranking While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach.

Read the paper

Authors

Vilém Zouhar, Julia Kreutzer, Alon Lavie, Tom Kocmi, Matt Post, Ondřej Bojar, Mrinmaya Sachan

Abstract

While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.

Evaluation

Related works

Research BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

Read

Research Findings of the WMT26 General Machine Translation Shared Task: Contrastive Dynamic Human Evaluation at Scale

Read

Research Contrastive ESA: Human Evaluation of Multiple Translations at Once

Read