inclusionAI/PIBench

inclusionAI/PIBench

Description: PIBench is an open-source benchmark for evaluating AI coding agents on realistic, end-to-end payment integration tasks.

Language: Java

License: MIT

Stars: 13

Forks: 0

Open issues: 0

Created: 2026-07-06T08:13:44Z

Pushed: 2026-07-17T03:13:20Z

Default branch: main

Fork: no

Archived: no

README:

中文说明

---

🎯 Introduction

Alipay-PIBench is a repository-level benchmark proposed by Alipay for evaluating AI coding agents on realistic payment integration tasks. Built around Alipay Open Platform products and business-oriented repositories, it comprises nine product-specific projects and 18 task instances across Basic (Functional Payment Completion) and Advanced (Risk-Aware Payment Hardening) scenarios. The benchmark evaluates functional correctness, reliability, security, and business-state consistency.

• Payment Benchmark Construction. Each payment product is paired with a representative project, and each task combines the target product, business workflow, initial repository, and scenario-specific integration request.
• Rubric-Grounded Evaluation. Scenario-specific rubrics define deterministic checks and LLM-assisted criteria, producing rubric-aligned evidence for structural, executable, and payment-domain requirements.
• Paired Skill Study. Each agent is evaluated on the same task, project, instruction, and environment with and without `alipay-payment-integration`, enabling comparisons by product, scenario, agent, and skill condition.

[](assets/overview.pdf)

🏆 Main Results

Model Capability

[](assets/main-results.pdf)

Model capability explains much of the observed performance spread and varies across products and scenarios. Under the with-skill condition, overall mean Rubric Pass Rate (RPR) ranges from 68.58% to 91.37%. Basic and Advanced tasks provide complementary views of an agent's ability to construct a payment flow and preserve safety-related payment behavior.

For results and detailed analyses of the remaining research questions, please refer to the paper.

🚀 Quick Start

Requirements

Run the suite on a Linux host or cloud server with:

• Bash
• Python 3
• Git
• Docker Engine, with permission for the current user to access the Docker daemon
• Network access to the selected model provider, Docker image registry, and task-specific package repositories
• Enough disk space to build task images and store run artifacts

The default configuration enables Docker because each task instance defines its own application, database, and development environment.

Run the Benchmark

This release is self-contained. It includes the benchmark suite, deterministic and LLM-assisted evaluations, three agent adapters (Claude Code, OpenClaw, and Hermes), the PaySkills runtime source, Docker orchestration, concurrent execution, and a top-level run.sh. You do not need to install the PaySkills platform.

# Show command help.
./run.sh --help

# Create config/.env without overwriting an existing file.
./run.sh --init

# Configure the run and local secrets.
vim config/config.yaml
vim config/.env

# Validate package layout, configuration, credentials, tools, and Docker.
./run.sh --doctor

# List selected task instances without invoking an agent or judge.
./run.sh --dry-run

# Start the evaluation.
./run.sh

config/.env is loaded automatically and ignored by Git. Do not put real API keys or Alipay sandbox private keys directly in config/config.yaml.

To keep multiple local configurations, pass one explicitly or set PAYSKILLS_CONFIG:

./run.sh --config config/config.local.yaml --doctor
./run.sh --config config/config.local.yaml
PAYSKILLS_CONFIG=/absolute/path/to/config.yaml ./run.sh --doctor

⚙️ Configuration

The active configuration is config/config.yaml. config/config.example.yaml is the reference template.

Run

run:
parallelism: 4
output_dir: runs
timeout_sec: 3600
task_instances: []

• parallelism: maximum number of task instances executed concurrently
• output_dir: safe relative output directory; do not use package paths such as benchmark_suite/, config/, or src/
• timeout_sec: timeout applied separately to task/run.sh and evaluation/evaluate.sh
• task_instances: optional list of task instance IDs; an empty list selects all 18

Keep configuration types literal: positive integers for parallelism and timeout_sec, a YAML list for task_instances, mappings for env and runtime_inputs, and booleans for docker.enabled and docker.build.

Agent

agent:
type: claude-code
mode: no-skill
model: "your-agent-model"
base_url: "https://your-provider.example/api/anthropic"
api_key_env: AGENT_API_KEY

• type: claude-code, openclaw, or hermes
• mode: no-skill or with-skill
• model: model name understood by the selected adapter and provider
• base_url: provider endpoint; configure it together with api_key_env
• api_key_env: name of the environment variable containing the key, not the key itself

Set the referenced variable in config/.env:

AGENT_API_KEY="your-secret-key"

LLM Judge

judge:
base_url: "https://your-provider.example/v1"
api_key_env: JUDGE_API_KEY
model: "your-judge-model"

Fill all three judge settings together. Store the referenced key in config/.env:

JUDGE_API_KEY="your-secret-key"

The agent and judge may reference the same environment variable when they use the same provider credential.

Extra Environment Variables

Values under env are copied into every task instance process:

env:
CUSTOM_FLAG: value

Use runtime_inputs, not env, for structured files or secrets requested by specific task instances.

Alipay Sandbox Runtime Input

Some task instances require Alipay sandbox information. Obtain it from the Alipay Developer Sandbox, and keep it outside the benchmark suite and Git history.

For identity-sensitive tasks, app_id and seller_id are distinct: the app ID identifies the sandbox application, while the seller ID is the merchant PID receiving funds. Do not copy the app ID into the seller ID field.

To...