Tencent-Hunyuan/RepoProbe

Tencent-Hunyuan/RepoProbe

Description: Benchmark for repository-level code understanding: 500 open-ended questions from real GitHub Discussions across 50 repositories, graded by a checklist-based verification protocol (ASE 2026)

Language: Python

License: NOASSERTION

Stars: 6

Forks: 0

Open issues: 0

Created: 2026-08-05T12:42:23Z

Pushed: 2026-08-06T03:33:17Z

Default branch: main

Fork: no

Archived: no

README:

RepoProbe

Can an LLM explain how a codebase actually works — not just patch it?

Evaluation code and benchmark data for the paper:

> RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists > *41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026)* > Munich, Germany, October 12–16, 2026 > arXiv:2608.04783

RepoProbe is a discussion-based benchmark for repository-level code understanding. Unlike defect-centric benchmarks that supply strong localisation cues (stack traces, error logs, filenames) and thereby let models shortcut genuine comprehension, RepoProbe draws its questions from real GitHub Discussions — open-ended architectural inquiries about how a codebase actually works.

Answers are graded by a Checklist-Based Verification Protocol that decomposes each open-ended response into atomic, weighted, verifiable technical facts, replacing opaque scalar LLM-as-a-Judge ratings with objective item-level verification.

At a Glance

| | | |---|---| | Repositories | 50 | | Questions | 500 (3–30 per repository, median 8) | | Programming languages | 15 | | Question categories | Project Architecture / Business Logic / Implementation Details | | Grading | Weighted checklist items with tiered rubrics, scored by an LLM judge | | Agent scaffolding | Docker-isolated, identical for every model under test | | Models evaluated in the paper | 20 (13 closed-weight, 7 open-weight) |

Key Results

Each answer is worth 10 points: 9 for knowledge items (factual and technical correctness) and 1 for a clarity item (explanation quality). *Overall Performance* is the mean of the total over 10, *Perfect Solve Rate* the fraction of questions scoring a full 10/10. All values are percentages; the full table of 20 models is in the paper.

| Model | Overall | Knowledge | Clarity | Perfect Solve | |---|---|---|---|---| | GPT-5.2 | 62.7 | 60.3 | 84.7 | 26.0 | | Claude Opus 4.6 | 62.1 | 60.2 | 79.0 | 27.5 | | GPT-5.4 | 60.4 | 57.6 | 85.6 | 24.2 | | Claude Sonnet 4.6 | 60.2 | 58.3 | 78.2 | 27.2 | | GLM-5 *(best open-weight)* | 54.8 | 52.9 | 72.2 | 21.3 |

Two findings drive the benchmark's design:

• Fluency outruns correctness. Every model scores far higher on clarity than on knowledge

— GPT-5.2 reaches 84.7 clarity against 60.3 knowledge. Answers read as authoritative while the technical substance lags, which is precisely what scalar LLM-as-a-Judge scoring tends to reward.

• Full credit is rare. The best Perfect Solve Rate is 27.5%, so roughly three out of four

questions leave at least one checklist item unmet even for frontier models. Repository-level comprehension is far from saturated.

Repository Structure

.
├── README.md # This file
├── LICENSE # Apache-2.0, applies to source code
├── NOTICE # Copyright and licensing summary
├── CITATION.cff # Citation metadata
├── repos_info.json # Metadata and snapshot info for all 50 repositories
├── requirements.txt # Python dependencies
├── dataset/ # Benchmark Q&A data (one CSV per repository)
│ ├── LICENSE # CC BY 4.0, applies to benchmark data
│ ├── NOTICE # Scope of the data licence and provenance
│ ├── adk-python.csv
│ ├── ... # 50 CSV files
│ └── yasb.csv
├── docs/
│ └── AGENT_INTEGRATION.md # How to plug your own agent scaffolding in
├── candidate_repos/ # (Runtime, git-ignored) Cloned repositories
├── scripts_agent/ # Shell entrypoints
│ ├── fetch_repos.sh # Clone/update repos and generate repomix summaries
│ └── eval_models_multi_repo.sh # Run agent-based evaluation across all repos
├── evaluator.py # Main evaluation orchestrator (Docker-based agent runner)
├── scorer.py # Checklist-based scoring via LLM judge
├── cache.py # Caching layer for agent answers and scores
├── dataset_loader.py # CSV dataset loader
├── fetch_repos.py # Repository cloning and repomix generation
├── prompt_templates_en.py # Scoring prompt templates
└── agent_configs/ # Docker container and agent runtime configuration
├── Dockerfile.base # Base image (Python, Node.js)
├── Dockerfile # Per-repo image (copies repo + configs)
├── build_base_image.sh # Script to build the base Docker image
├── agent.json # Declares how to launch the agent under evaluation
├── example_agent.py # Reference agent scaffolding, replaceable
├── gateway.py # In-container OpenAI-compatible model gateway
├── gateway.sh # Gateway startup script
└── model_client.py # OpenAI-compatible API client

Dataset Format

Each CSV file in dataset/ corresponds to one repository. Columns:

| Column | Description | |---|---| | repo_name | Short name of the source repository (matches the CSV filename) | | question_id | Stable identifier, formatted as - | | discussion_id | GitHub Discussion node ID, for tracing back to the original thread | | taxonomy | Code-understanding category: Project Architecture / Business Logic / Implementation Details | | difficulty | Difficulty level | | question | The question, derived from a GitHub Discussion | | answer | Reference answer from the discussion maintainer | | checklist | Weighted checklist items with tiered scoring rubrics |

A checklist entry looks like this — each item carries a point value and tiered criteria, so partial credit is awarded explicitly rather than inferred by the judge:

(3 points) Bug Fix Confirmation and Commit Identification
- 3 points: Correctly confirms the fix is present in 8.0.2 and identifies the commit hash
- 1 point: Confirms the fix exists but does not provide the correct commit
- 0 points: Fails to confirm, or asserts the bug is unfixed

Prerequisites

• Python 3.11+
• Docker
• Node.js 20+ (for repomix)
• An OpenAI-compatible API endpoint and key

Agent Scaffolding

RepoProbe evaluates agents rather than raw model APIs: each question is answered by an agent that explores the repository before responding. The harness is scaffolding-agnostic — it defines a container contract (repository at /app/repo, question at /tmp/prompt/prompt.txt,...