Evaluating The Retrieval Robustness Of Large Language Models 2026 07 08

Evaluating the Retrieval Robustness of Large Language Models Skip to content AI for Empowerment: Your freedom. Your focus. See how AI gives you more time for what truly moves you. Explore now

Products

Solutions

Resources

Blog

Research

Company

Sign in

Request a demo

Platform North

Enterprise-ready AI for business

Compass

Intelligent search and discovery

Models Command

Generative language models

Transcribe New

Speech recognition model

North Small Translate NEW

Machine translation model

Parse New

Document parsing model

Embed

Semantic representation model

Rerank

Retrieval optimization model

Models Overview

Product Products Overview

Total Cost of AI Ownership

Pricing

Featured Command: High-performance generative AI models for real-world applications

Deploy Model Vault

Dedicated model inference platform

Private Deployments

On-prem or isolated VPCs

Security

Protect your data at every stage

See deployment options

By Industry Financial Services

Public Sector

Technology

Telecommunications

Energy and Utilities

Healthcare and Life Sciences

Manufacturing

Featured Model Vault provides fully-isolated, performant inference with Saas simplicity

Insights Customer Stories

For Developers Developers

Models Overview

Docs

Discord

LLM University

Connect Partners

Events

Webinars

Merch Store

Featured How CoreWeave used Cohere North to transform its customer support in 90 days

Blog

The latest news, launches, and insights

Read more

Cohere and Aleph Alpha sign agreement to become the first transatlantic sovereign AI solution

Who Gets to Define the Rules for AI?

The state of sovereign AI adoption in 2026

Research Cohere Labs

Cohere’s ML research lab

Explorations Future(s) of Work

How will AI change the way we work?

Aya Models

Multilingual AI at scale

All Papers

Initiatives Research Scholars

Finding the new generation of ML talent

Open Science Community

Championing global, open science

Catalyst Grants

Supporting impactful ML endeavors

Resources Blog

Hugging Face

Events

Featured The future of work debate has an evidence problem

About

Careers

Newsroom

Jul 08, 2026 Evaluating the Retrieval Robustness of Large Language Models Retrieval-augmented generation (RAG) generally enhances large language models' (LLMs) ability to solve knowledge-intensive tasks. But RAG could also lead to performance degradation due to imperfect retrieval and the model's limited ability to leverage retrieved content. In this work, we evaluate the robustness of LLMs in practical RAG setups (henceforth retrieval robustness).

Read the paper

Authors

Shuyang Cao, Karthik Radhakrishnan, David Rosenberg, Steven Lu, Pengxiang Cheng

Abstract

Retrieval-augmented generation (RAG) generally enhances large language models' (LLMs) ability to solve knowledge-intensive tasks. But RAG could also lead to performance degradation due to imperfect retrieval and the model's limited ability to leverage retrieved content. In this work, we evaluate the robustness of LLMs in practical RAG setups (henceforth retrieval robustness). We focus on three research questions: (1) whether RAG is always better than non-RAG; (2) whether more retrieved documents always lead to better performance; and (3) whether document order impacts results. To facilitate this study, we establish a benchmark of 1,891 samples spanning five datasets across three task categories, each with documents retrieved using both sparse and dense retrievers. We introduce three robustness metrics, each corresponding to one research question. Our experiments across 11 LLMs show that models achieve generally high retrieval robustness, but robustness varies substantially across tasks, suggesting that the decision to adopt RAG remains a case-by-case consideration. We further examine four additional prompting strategies that vary how models interact with retrieved documents. We find that Qwen and GPT models suffer notable robustness declines when reasoning is disabled, even on single-hop QA tasks, and that providing retrieved documents as tool responses improves Claude models but hurts Qwen and GPT models, highlighting potential issues of the GPT models regardless of their best overall robustness under vanilla prompting.

Language Models Robustness

Related works

Research LLM2Vec-Gen: Generative Embeddings from Large Language Models

Read

Research BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs

Read

Research SimMerge: Learning to Select Merge Operators from Similarity Signals

Read