Tencent-Hunyuan/HyperLogic

Tencent-Hunyuan/HyperLogic

Description: HyperLogic: a hard, forward-authored Chinese logical reasoning benchmark with execution-derived answers (Seed / Base / Hard)

Language: Python

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-10-27T11:05:36Z

Pushed: 2026-10-28T06:42:44Z

Default branch: main

Fork: no

Archived: no

README:

中文 ｜ English

HyperLogic: A Hard, Forward-Authored Chinese Logical Reasoning Benchmark with Execution-Derived Answers

📄 Paper | 🤗 Dataset | 🚀 Quick Start | 📮 Evaluate Your Model

---

🔥 News

• 2026.10.28: 💻 This repository and the Hugging Face dataset go public with about 30% of the items of every tier, their gold answers, the direct-answer prompt and scorer, and the complete Hard agent evaluation (both sandbox conditions, their exact payloads, and the logic modeling library).
• 2026.10.02: 🎉 The paper is released on arXiv as HyperLogic (v2 of arXiv:2605.19597) — HyperLogic-Base with 195 items and 922 sub-questions, HyperLogic-Hard with 100 items, and results for 7 frontier models. [[Paper]](https://arxiv.org/abs/2605.19597)

📋 Contents

• [Overview](#-overview)
• [Leaderboard](#-leaderboard)
• [Key Findings](#-key-findings)
• [What Is Released](#-what-is-released)
• [Quick Start](#-quick-start)
• [Data Format](#-data-format)
• [Repository Structure](#-repository-structure)
• [Evaluate Your Model](#-evaluate-your-model)
• [License](#-license)
• [Citation](#-citation)
• [Contact](#-contact)

📄 Overview

Existing logic benchmarks mostly measure whether a model can answer a reasoning question directly, and many of them no longer separate frontier models. Benchmarks that generate text from formal structures make answers easy to compute, but fix the formalization before the problem is written. HyperLogic is built forward: problems are written first, and their answers are derived afterwards by executing formal models of them.

The construction pipeline separates problem authoring from answer generation. A multi-agent workflow hardens Chinese seeds written by logic-trained undergraduates, without solving them. Two agents from different model families then independently translate each finished item into executable finite-domain models, and logic solvers derive the answers. Encodings and answers go through layered, agent-assisted adjudication under human-expert oversight; an item is released only when its two corrected encodings agree. Because the solvers do the search, items can be harder than the models that built them, while every answer stays executable and auditable.

*Overview of HyperLogic. (a) A four-role agent loop hardens seeds with tier-specific strategy libraries. (b) Two independent formalizations provide solver-derived answers for layered adjudication. (c) Both tiers are evaluated by direct answering; Hard also uses sandbox agents, with or without the logic modeling library.*

Stats: 228 Seed items · 195 Base items with 922 sub-questions · 100 Hard items · 7 frontier models · 3 rollouts per model and condition · every gold answer derived by solvers from two agreeing independent formalizations

Tiers:

• HyperLogic-Seed — 228 single-question seeds written by undergraduates with coursework in logic. Like existing benchmarks, Seed barely separates frontier models: all seven are within 5.9 points of each other.
• HyperLogic-Base — 195 items, each a shared background (about 3,300 Chinese characters) with 4.7 sub-questions on average. An item counts as correct only if every sub-question is. Frontier models can still answer these directly, and Base is designed to separate them.
• HyperLogic-Hard — 100 single-question items with larger, coupled search spaces (backgrounds of about 8,700 characters). Direct solving is no longer practical; Hard also evaluates agents that write code in a sandbox, with or without our logic modeling library.

🏆 Leaderboard

Accuracy (%) on the full benchmark, mean ± standard deviation over three rollouts, every model at its highest reasoning setting. Base Item counts an item as correct only if all its sub-questions are. Sandbox gives the agent a code sandbox; + Library also gives it the logic modeling library. Bold marks the best model in each column.

| Rank | Model | Seed | Base Item | Base Sub-Q | Hard Direct | Hard Sandbox | Hard + Library | |:---:|---|---:|---:|---:|---:|---:|---:| | 1 | GPT-5.6 Sol | 89.8 ± 3.7 | 77.6 ± 0.6 | 92.6 ± 0.2 | 13.7 ± 5.9 | 40.3 ± 4.2 | 45.7 ± 2.1 | | 2 | Claude Opus 5 | 86.5 ± 0.7 | 74.2† ± 2.7 | 88.9† ± 0.7 | 15.6† ± 1.8 | 55.7 ± 3.2 | 57.0 ± 2.6 | | 3 | Kimi K3 | 88.6 ± 0.0 | 57.9 ± 1.8 | 81.6 ± 1.8 | 4.0 ± 0.0 | 41.3 ± 1.5 | 43.0 ± 8.9 | | 4 | Grok 4.6 | 87.0 ± 1.8 | 52.5 ± 4.1 | 82.9 ± 0.7 | 12.0 ± 1.0 | 39.3 ± 1.5 | 38.3 ± 2.1 | | 5 | DeepSeek V4 Pro | 83.9 ± 1.7 | 51.8 ± 1.5 | 81.2 ± 1.2 | 9.3 ± 1.2 | 26.0 ± 5.6 | 20.0 ± 1.7 | | 6 | GLM-5.3 | 87.3 ± 0.9 | 45.6 ± 3.2 | 67.2 ± 4.9 | 3.0 ± 1.0 | 20.7 ± 0.6 | 31.3 ± 5.1 | | 7 | Doubao Seed 2.1 Pro | 85.8 ± 0.7 | 44.6 ± 2.2 | 73.2 ± 1.0 | 5.0 ± 1.0 | 26.0 ± 1.0 | 33.7 ± 0.6 |

†Excludes items refused by the model's API service. Grok 4.6 runs at xhigh, every other model at max. Models are ranked by Base item accuracy.

💡 Key Findings

• Base is hard and discriminative. On five of seven existing logic benchmarks, three frontier models differ by at most three points, and Seed behaves the same way (83.9–89.8%). On Base every model loses at least 12 points from Seed and the seven models spread over 33.0 points (44.6–77.6%).
• Longer reasoning does not mean higher accuracy. Median reasoning length on Base ranges from 15.2k to 52.7k tokens per item and does not track accuracy: the two most accurate models use less than half the tokens of GLM-5.3, which hits its 131k-token limit without a parseable answer in 99 of 585 runs.
• Error sources are shared across models, and large possibility spaces separate them. For most models errors rise with the size of the gold answer; the two leaders err *less* on the larger half.
• Hard is largely beyond direct answering, and a sandbox helps but does not solve it. No model exceeds 16% directly. A code sandbox lifts every model by 16.7–40.1 points, yet the best reaches only 55.7% (57.0% with the library).
• The logic modeling library does not consistently help. Adding it changes Hard accuracy by...