UpstageAI/ParseBench

UpstageAI/ParseBench

Description: ParseBench - A Document Parsing Benchmark for AI Agents

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 1

Created: 2026-09-17T05:59:39Z

Pushed: 2026-09-17T06:24:00Z

Default branch: main

Fork: yes

Parent repository: run-llama/ParseBench

Archived: no

README:

ParseBench

> Related: For schema-guided enterprise document extraction, see our companion benchmark ExtractBench.

ParseBench is a benchmark for evaluating how well document parsing tools convert PDFs into structured output that AI agents can reliably act on. It tests whether parsed output preserves the structure and meaning needed for autonomous decisions — not just whether it looks similar to a reference text.

The benchmark covers ~2,000 human-verified pages from real enterprise documents (insurance, finance, government), organized around five capability dimensions, each targeting a failure mode that breaks production agent workflows.

Leaderboard

_Top 10 by Overall score. For the full sortable, filterable leaderboard, see parsebench.ai; for raw data, see [leaderboard.csv](leaderboard.csv)._

| Rank | Provider | Category | Overall | Tables | Charts | Content Faith. | Sem. Format. | Visual Ground. | ¢ / Page | |---:|---|---|---:|---:|---:|---:|---:|---:|---:| | 1 | LlamaParse Agentic Plus | LlamaParse | 90.20 | 93.37 | 94.18 | 92.25 | 87.12 | 84.09 | 5.62¢ | | 2 | LlamaParse Agentic | LlamaParse | 87.01 | 88.88 | 88.68 | 91.78 | 81.44 | 84.25 | 1.25¢ | | 3 | Pulse Ultra 2 | Commercial - Startup APIs | 81.60 | 90.35 | 89.70 | 87.63 | 73.97 | 66.35 | 1.50¢ | | 4 | LlamaParse Cost Effective | LlamaParse | 80.61 | 84.19 | 77.91 | 89.87 | 67.29 | 83.77 | 0.38¢ | | 5 | Anthropic Fable 5.1 | VLM - Proprietary | 78.92 | 91.52 | 67.06 | 91.19 | 76.52 | 68.3 | 16.05¢ | | 6 | oi-parser | Commercial - Startup APIs | 78.30 | 92.62 | 78.28 | 86.17 | 66.88 | 67.53 | — | | 7 | rakedoc-nano | VLM - Open Weight | 77.23 | 86.44 | 64.89 | 88.84 | 71.68 | 74.28 | — | | 8 | florin-parser-nano | VLM - Open Weight | 76.69 | 86.10 | 65.19 | 87.37 | 70.64 | 74.14 | — | | 9 | KDL-Frontier-Parser-nano | VLM - Open Weight | 76.36 | 85.56 | 63.41 | 87.19 | 66.81 | 78.84 | — | | 10 | Extend (2.0) | Commercial - Startup APIs | 75.33 | 84.82 | 78.31 | 84.59 | 60.31 | 68.61 | 2.50¢ |

Inclusion criteria: 1. The model or API needs to be publicly accessible, either via open weights or a self-serve API that any user can sign up for. 2. The benchmark run needs to finish within a reasonable time (roughly single-digit hours). 3. We can adjust concurrency based on the provider's recommended settings, but providers should not require custom framework changes, so the evaluation stays fair across models.

Quick Start

Prerequisites: Create a .env file with the API key for the parsing tool you want to evaluate (see [Configuration](#configuration) for details).

# Install from PyPI (pick the extras for the providers you want to run)
pip install "parse-bench[runners]" # every provider SDK
pip install "parse-bench[llamaparse]" # or just one, e.g. llamaparse / openai / anthropic / google

# Or, from a checkout of this repo
uv sync --extra runners

# Optional: add the `fast` extra for a JIT-accelerated TEDS table metric (numba).
# Scores are identical to the default path — just faster on large tables.
pip install "parse-bench[runners,fast]"

# Quick test run (small dataset, 3 files per category — good for trying things out)
# (drop the `uv run` prefix if you installed from PyPI)
uv run parse-bench run llamaparse_agentic --test

# Full benchmark run (replace llamaparse_agentic with any pipeline name, see "Available Pipelines" below)
uv run parse-bench run llamaparse_agentic

# View interactive reports in your browser
uv run parse-bench serve llamaparse_agentic

Available Pipelines

A pipeline is a document parsing tool or configuration that you want to evaluate. There are 180+ pipelines available -- see [docs/pipelines.md](docs/pipelines.md) for the full list, or run uv run parse-bench pipelines. To add your own providers, pipelines, products or rule types from another package, see [docs/extending.md](docs/extending.md).

Paper baselines (21 pipelines)

| Pipeline name | Name in paper | |---------------|---------------| | llamaparse_agentic | LlamaParse Agentic | | llamaparse_cost_effective | LlamaParse Cost Effective | | openai_gpt5_mini_reasoning_medium_parse_with_layout_file | OpenAI GPT-5 Mini (Reasoning Medium) | | openai_gpt5_mini_reasoning_minimal_parse_with_layout_file | OpenAI GPT-5 Mini (Reasoning Minimal) | | openai_gpt_5_4_parse_with_layout_file | OpenAI GPT-5.4 | | anthropic_haiku_parse_with_layout_file | Anthropic Haiku 4.5 (Disable Thinking) | | anthropic_haiku_thinking_parse_with_layout_file | Anthropic Haiku 4.5 (Thinking) | | anthropic_opus_4_6_parse_with_layout_file | Anthropic Opus 4.6 | | google_gemini_3_flash_thinking_minimal_parse_with_layout_file | Google Gemini 3 Flash (Thinking Minimal) | | google_gemini_3_flash_thinking_high_parse_with_layout_file | Google Gemini 3 Flash (Thinking High) | | google_gemini_3_1_pro_parse_with_layout_file | Google Gemini 3.1 Pro | | azure_di_layout | Azure Document Intelligence | | aws_textract | AWS Textract | | google_docai_layout | Google Cloud Document AI | | reducto | Reducto | | reducto_agentic | Reducto (Agentic) | | extend_parse | Extend | | landingai_parse | LandingAI | | qwen3_5_4b_vllm_parse | Qwen 3 VL | | dots_ocr_1_5_parse | Dots OCR 1.5 | | docling_parse | Docling |

Dataset

Hosted on HuggingFace: `llamaindex/ParseBench`

The dataset is stratified into five capability dimensions, each with its own ground-truth format and evaluation metric:

| Dimension | File(s) | Metric | Pages | Docs | Rules | |-----------|---------|--------|------:|-----:|------:| | Tables | table.jsonl | GTRM (GriTS + TableRecordMatch) | 503 | 284 | --- | | Charts | chart.jsonl | ChartDataPointMatch | 568 | 99 | 4,864 | | Content Faithfulness | text_content.jsonl | Content Faithfulness Score | 506 | 506 | 141,322 | | Semantic Formatting | text_formatting.jsonl | Semantic Formatting Score | 476 | 476 | 5,997 | | Visual Grounding | layout.jsonl | Element Pass Rate | 500 | 321 | 16,325 | | Total (unique) | | | 2,078 |...