zai-org/SurveyReview

zai-org/SurveyReview

Description: Official Code for SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators (KDD 2026)

Language: Python

License: Apache-2.0

Stars: 1

Forks: 0

Open issues: 0

Created: 2026-06-09T08:33:01Z

Pushed: 2026-08-07T09:55:44Z

Default branch: main

Fork: no

Archived: no

README:

SurveyReview

A Reviewer-Aligned Benchmark for Survey Evaluators

SurveyReview is a reviewer-aligned benchmark for evaluating survey papers. It converts real peer-review reports into multidimensional evaluation labels and rationales, allowing models to be tested against how human reviewers judge survey quality.

Version

• v1.0/: the original SurveyReview release for paper reproduction.
• v1.1/: the updated release using cleaned article texts.

The benchmark focuses on four survey-review dimensions: Readability, Criticalness, Comprehensiveness, and Structure. It provides standardized train/test splits, article metadata, prompt files, and an API-based evaluation pipeline.

Quick Start

Create an environment and install dependencies:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Configure the API client:

cp .env.example .env

Then edit .env:

API_KEY=your-api-key-here
BASE_URL=https://api.openai.com/v1
MODEL_NAME=gpt-5.2
JUDGE_MODEL=gpt-5.2
EVALUATE_REASONS=True

Run the default test-set evaluation:

python src/api_base_evaluate.py

Run on the train split:

EVAL_SPLIT=train python src/api_base_evaluate.py

Outputs are written to result//:

| File | Description | | --- | --- | | results.csv | MSE, MAE, accuracy, and sample counts for each dimension. | | predictions_.jsonl | Per-sample prediction records. | | run_config.json | Runtime configuration and split statistics. | | rqs_.json | Rationale quality results when EVALUATE_REASONS=True. |

Leaderboard(v1.0)

Lower MSE/MAE is better. Higher HAS/RQS is better.

Rank Model HAS Read. Crit. Comp. Stru. Average RQS

MSEMAE MSEMAE MSEMAE MSEMAE MSEMAE

1 SurveyReviewer 0.74 1.430.72 1.520.82 1.260.56 1.290.65 1.380.69 0.36

2 GPT-5.2 0.68 2.131.07 1.970.97 2.041.08 2.981.47 2.281.15 0.42

3 Claude-Opus-4.5 0.68 2.911.29 1.880.88 2.661.23 3.651.58 2.771.25 0.48

4 Qwen3-32B 0.61 3.051.45 3.241.51 3.221.54 3.351.53 3.211.51 0.36

5 GLM-4.7 0.60 3.431.50 2.581.21 3.661.57 4.831.95 3.621.56 0.37

6 gemini-3-pro 0.58 3.841.52 2.251.00 3.911.49 5.762.11 3.941.53 0.29

7 DeepSeek-v3.2 0.58 4.781.88 2.491.15 4.591.82 4.021.76 3.971.65 0.37

Notes

• v1.0/ contains the original SurveyReview release and v1.0-paper data for paper reproduction.
• v1.1/ contains the updated release using cleaned article texts.
• Article files are split into JSON shards to stay within GitHub file-size limits.
• Both releases keep the SurveyReview evaluation style: MSE/MAE plus optional RQS. If you only want to verify the pipeline, set EVALUATE_REASONS=False to skip the judge-model stage.

Citation

🌟 If you find our work helpful, please leave us a star and cite our paper.

@inproceedings{zhang2026surveyreview,
author = {Zhang, Yuheng and Wang, Yuanchun and Zhang, Fanjin and Zhao, Ruyu and Li, Juanzi and Tang, Jie and Zhang, Jing},
title = {{SurveyReview}: A Reviewer-Aligned Benchmark for Survey Evaluators},
booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
year = {2026},
pages = {10302--10313},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
location = {Jeju Island, Republic of Korea},
series = {KDD '26},
isbn = {979-8-4007-2259-2},
doi = {10.1145/3770855.3817505},
url = {https://doi.org/10.1145/3770855.3817505}
}