google-deepmind/polaris-bench

google-deepmind/polaris-bench

Language: Python

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-09-03T23:45:45Z

Pushed: 2026-09-09T19:35:09Z

Default branch: main

Fork: no

Archived: no

README: The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

Polaris-Bench: Official Benchmark and Evaluation Suite

Xia Hu1, Zhenrui Yue1, Brian Potetz1, Howard Zhou1, Leonidas Guibas1,2, Chun-Ta Lu3, Zhicheng Wang1

1Google DeepMind 2Stanford University 3Google Research

---

Overview

Current Multimodal Large Language Models (MLLMs) achieve strong performance on visual reasoning benchmarks, but do these scores reflect genuine visual perception? In this work, we identify a pervasive vulnerability: the Cartesian Shortcut.

The Cartesian Shortcut

Standard visual reasoning benchmarks are predominantly structured around orthogonal, grid-based Cartesian layouts. We find that state-of-the-art models systematically exploit this structure: rather than performing true visual-spatial reasoning, they discretize 2D images into explicit textual coordinates (such as row and column indices) and offload spatial deduction onto pure text-based reasoning. This text-based shortcut inflates benchmark scores while masking critical deficiencies in genuine visual understanding.

Polaris-Bench

To dismantle the Cartesian Shortcut, we introduce Polaris-Bench, an evaluation benchmark that re-formulates 53 visual reasoning tasks across 5 cognitive categories into Polar coordinate space, paired directly with their Cartesian counterparts under identical logical constraints and rules. In Polar space, lines of constant coordinate curvature bend, distance metrics depend on radius, and coordinate discretization becomes non-trivial.

Under this controlled setting, frontier models that achieve 70-83% accuracy on Cartesian layouts experience a dramatic performance collapse to 31-39% on logically equivalent Polar tasks, while human performance remains robust (94.5% Cartesian vs. 88.8% Polar).

Leaderboard

All models evaluated under high reasoning mode. Sorted by Polar accuracy (P). Full per-category results on the project page.

| # | Model | Type | Cartesian (%) | Polar (%) | Drop (Δ) | |:-:|-------|:----:|:---:|:---:|:---:| | 👤 | Human | Baseline | 94.5 | 88.8 | -5.7 | | 1 | GPT-5.2 | Closed | 77.4 | 39.2 | -38.2 | | 2 | Gemini-3.1-Pro | Closed | 82.6 | 35.9 | -46.7 | | 3 | Qwen3.5-397B-A17B | Open | 72.9 | 35.0 | -37.9 | | 4 | Gemini-3-Flash | Closed | 71.0 | 33.8 | -37.2 | | 5 | Kimi-k2.5 | Open | 69.0 | 31.1 | -37.8 | | 6 | Gemma-4-31B | Open | 60.5 | 31.0 | -29.5 | | 7 | Claude-Sonnet-4.6 | Closed | 44.4 | 25.9 | -18.5 | | 8 | Gemini-2.5-Pro | Closed | 38.4 | 25.3 | -13.2 | | 9 | Gemini-3.1-Flash-Lite | Closed | 46.8 | 24.6 | -22.2 | | 10 | Gemma-4-26B | Open | 47.2 | 22.9 | -24.4 | | 11 | Grok-4-Fast-Reasoning | Closed | 31.0 | 22.3 | -8.7 | | 12 | Grok-4-0709 | Closed | 33.0 | 21.8 | -11.2 | | 13 | Gemini-2.5-Flash | Closed | 32.2 | 21.1 | -11.2 | | 14 | Mistral-Small-2503 | Open | 19.4 | 19.0 | -0.4 | | - | Random Baseline | Baseline | 15.8 | 15.8 | 0.0 |

Dataset on Hugging Face

The complete Polaris-Bench evaluation dataset (all 10,800 multimodal problem instances across 53 tasks and 4 coordinate systems) is hosted on the Hugging Face Hub:

[https://huggingface.co/datasets/google/polaris-bench](https://huggingface.co/datasets/google/polaris-bench)

You can load the full dataset directly using the Hugging Face datasets library:

from datasets import load_dataset

# Load the full 10,800 evaluation problem instances
dataset = load_dataset("google/polaris-bench", split="test")

# Inspect an example
sample = dataset[0]
print(f"Task: {sample['task']} | Coordinate System: {sample['question_type']}")
print(f"Question: {sample['question']}")
print(f"Answer: {sample['answer']}")
# sample["image"] is automatically decoded as a PIL Image object

> Offline sample tasks: If you want to explore the benchmark locally without downloading the full Hugging Face image dataset, a standalone set of 20 representative paired Cartesian-Polar tasks is included directly in this repository under [examples/sample_tasks/](examples/sample_tasks/).

Repository Structure

polaris-bench/
├── evaluation/ # Core evaluation module and benchmark scoring
│ ├── __init__.py # Package exports (PolarisDataLoader, Evaluator)
│ ├── data_loader.py # Dataset loader (supports Hugging Face, local JSON, and sample tasks)
│ └── evaluate.py # Benchmark evaluation script (accuracy by coordinate system and category)
├── examples/ # Quick start guide and local offline sample instances
│ ├── quick_start.py # Standalone demo script (loads data, paired tasks, and runs mock evaluation)
│ └── sample_tasks/ # 20 representative paired tasks for local offline inspection
│ ├── sample_tasks.json# Paired Cartesian vs. Polar questions and ground truth
│ ├── README.md # Documentation for sample tasks
│ └── images/ # PNG images organized by task
├── docs/ # Project webpage source (served via GitHub Pages)
│ ├── index.html # Interactive project page with leaderboard and task viewer
│ └── static/ # Paper figures, teasers, and website assets
├── tests/ # Unit test suite
│ └── test_benchmark.py # 9 automated tests for data loading, matching, and metrics
├── pyproject.toml # Python package build configuration
├── requirements.txt # Minimal dependencies (datasets, Pillow, etc.)
├── LICENSE # Apache 2.0 license
└── README.md # Project documentation and getting started guide

Installation

git clone https://github.com/google-deepmind/polaris-bench.git
cd polaris-bench
pip install -r requirements.txt

Quick Start

You can inspect the benchmark using either the HuggingFace Hub dataset or the local offline sample data:

from evaluation.data_loader import PolarisDataLoader

# 1. Load from local file or HuggingFace Hub ('google/polaris-bench')
loader = PolarisDataLoader("examples/sample_tasks/sample_tasks.json")

# 2. Filter by task and coordinate system
cart_examples = loader.get_dataset(task="sudoku", question_type="cartesian")
print(f"Loaded {len(cart_examples)} Cartesian Sudoku tasks.")

# 3. Retrieve paired instances (Cartesian vs. Polar)
pairs = loader.get_paired_dataset(task="sudoku")
cart_sample,...