NVIDIA/RULER

NVIDIA/RULER

Description: This repo contains the source code for RULER: What’s the Real Context Size of Your Long-Context Language Models?

Language: Python

License: Apache-2.0

Stars: 1568

Forks: 130

Open issues: 18

Created: 2024-04-09T23:13:20Z

Pushed: 2026-06-25T17:18:16Z

Default branch: main

Fork: no

Archived: no

README:

📏 RULER: What’s the Real Context Size of Your Long-Context Language Models?

This repository contains code for our paper RULER: What’s the Real Context Size of Your Long-Context Language Models. RULER generates synthetic examples to evaluate long-context language models with configurable sequence length and task complexity. We benchmark 17 open-source models across 4 task categories (in total 13 tasks) in RULER, evaluating long-context capabilities beyond simple in-context recall. Here are our main results.

|Models|Claimed Length|Effective Length|4K|8K|16K|32K|64K|128K|Avg.|wAvg. (inc)|wAvg. (dec)| |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Llama2 (7B)|4K||85.6| Jamba-1.5-large* (94B/398B)|256k|>128k|96.7|96.6|96.4|96.0|95.4|95.1|96.0|95.7 (1st)|96.3 (1st)| Gemini-1.5-pro,-Text%20and%20images)|1M|>128K|96.7|95.8|96.0|95.9|95.9|94.4|95.8|95.5 (2nd)|96.1 (2nd)| Qwen2.5-14B-Instruct-1M* (14B)|1M|>128K|97.5|97.1|94.6|94.9|94.9|92.2|95.7|TBD|TBD| Qwen3-235B-A22B* (235B)|128K|>128K|97.7|97.2|96.4|95.1|93.3|90.6|95.0|TBD|TBD| Qwen3-14B* (14B)|128K|>128K|98.0|97.8|96.4|96.1|94.0|85.1|94.6|TBD|TBD| Jamba-1.5-mini (12B/52B)|256K|>128K|95.6|95.6|94.8|94.6|92.8|90.0|93.9|93.1 (3rd)|94.8 (3rd) Qwen3-32B* (32B)|128K|>128K|98.4|96.0|96.2|94.4|91.8|85.6|93.7|TBD|TBD| EXAONE-4.0-32B* (32B)|128K|>128K|96.3|94.9|93.9|93.6|91.7|88.2|93.1|TBD|TBD| Qwen2.5-7B-Instruct-1M* (7B)|1M|>128K|96.8|95.3|93.0|91.1|90.4|84.4|91.8|TBD|TBD| Qwen3-30B-A3B* (30B)|128K|64K|96.5|97.0|95.3|92.4|89.1|79.2|91.6|TBD|TBD| GPT-4-1106-preview|128K|64K|96.6|96.3|95.2|93.2|87.0|81.2|91.6|89.0 (4th)|94.1 (4th)| Llama3.1 (70B)|128K|64K|96.5|95.8|95.4|94.8|88.4|66.6|89.6|85.5 (10th)|93.7 (5th)| Qwen3-8B* (8B)|128K|64K|96.3|96.0|91.8|91.2|82.1|77.4|89.1|TBD|TBD| Mistral-Large-2411 (123B)|128K|64K|96.4|96.3|95.3|94.0|85.9|48.1|86.0|79.5 (18th)|92.5 (6th)| Command-R-plus-0824 (104B)|128K|32K|96.0|95.1|94.0|92.4|85.4|64.6|87.9|83.4 (13th)|92.4 (7th)| Qwen2 (72B)|128K|32K|96.9|96.1|94.9|94.1|79.8|53.7|85.9|79.6 (17th)|92.3 (8th)| Command-R-plus (104B)|128K|32K|95.6|95.2|94.2|92.0|84.3|63.1|87.4|82.7 (14th)|92.1 (9th)| Command-R-0824 (32B)|128K|64K|94.7|93.7|93.1|90.8|86.6|74.7|88.9|86.0 (8th)|91.9 (10th)| GLM4 (9B)|1M|64K|94.7|92.8|92.1|89.9|86.7|83.1|89.9|88.0 (5th)|91.7 (11th)| Llama3.1 (8B)|128K|32K|95.5|93.8|91.6|87.4|84.7|77.0|88.3|85.4 (11th)|91.3 (12th)| ProLong (8B)|512K|32K|94.5|92.5|92.3|89.3|83.2|81.6|88.9|86.6 (7th)|91.2 (13th)| Command-R (35B)|128K|32K|93.8|93.3|92.4|89.5|84.9|76.0|88.3|85.5 (9th)|91.1 (14th)| MegaBeam-Mistral (7B)|512K|32K|93.8|92.5|92.0|89.2|83.7|83.7|89.1|87.3 (6th)|91.0 (15th)| Mistral-Large-2407 (123B)|128K|32K|96.2|96.1|95.1|93.0|78.8|23.7|80.5|70.6 (24th)|90.4 (16th)| GradientAI/Llama3 (70B)|1M|16K|95.1|94.4|90.8|85.4|80.9|72.1|86.5|82.6 (15th)|90.3 (17th)| Mixtral-8x22B (39B/141B)|64K|32K|95.6|94.9|93.4|90.9|84.7|31.7|81.9|73.5 (22nd)|90.3 (18th)| Yi (34B)|200K|32K|93.3|92.2|91.3|87.5|83.2|77.3|87.5|84.8 (12th)|90.1 (19th)| Qwen3-4B* (4B)|128K|64K|95.1|93.6|91.0|87.8|77.8|66.0|85.2|TBD|TBD| EXAONE-4.0-1.2B* (1.2B)|64K|32K|87.0|86.7|88.8|81.1|77.4|-|84.2|TBD|TBD| Phi3-mini (3.8B)|128K|32K|92.2|91.5|90.7|87.5|80.6|66.7|84.8|80.9 (16th)|88.7 (20th)| Phi3-medium (14B)|128K|32K|93.3|93.2|91.1|86.8|78.6|46.1|81.5|74.8 (21st)|88.3 (21st)| Mixtral-8x7B (12.9B/46.7B)|32K|32K|94.9|92.1|92.5|85.9|72.4|44.5|80.4|72.8 (23rd)|87.9 (22nd)| GradientAI/Llama3 (8B)|1M|16K|92.8|90.3|85.7|79.9|76.3|69.5|82.4|78.5 (19th)|86.3 (23rd)| FILM-7B* (7B)|32K|32K|92.8|88.2|88.1|86.9|70.1|27.1|75.5|66.4 (26th)|84.7 (24th)| InternLM2.5 (7B)|1M|4K|88.1|85.5|84.5|82.7|75.5|68.9|80.9| 77.8 (20th)|83.9 (25th)| Mistral (7B)|32K|16K|93.6|91.2|87.2|75.4|49.0|13.8|68.4|55.6...