ByteDance-Seed/EdgeBench

ByteDance-Seed/EdgeBench

Description: EdgeBench: Unveiling scaling laws of learning from real-world environments

Language: Python

License: Apache-2.0

Stars: 68

Forks: 1

Open issues: 0

Created: 2026-06-23T14:19:39Z

Pushed: 2026-07-02T15:27:25Z

Default branch: main

Fork: no

Archived: no

README:

---

Overview

EdgeBench is a benchmark of 134 real-world tasks for evaluating how autonomous AI agents *learn from real-world environments*. Instead of measuring one-shot performance, EdgeBench places agents in executable task environments with realistic, multi-level feedback and lets them iterate for 12+ hours per task — tracking the full trajectory of improvement, not just the final score. We publicly release 51 tasks along with the full evaluation framework.

Analyzing ~38,000 hours of agent interaction on all 134 tasks, we find that performance follows a log-sigmoid scaling law as a function of interaction time ($R^2 = 0.998$). See the tech report for details.

Leaderboard

Full Benchmark (134 tasks)

| Model | @2h | @4h | @6h | @8h | @10h | @12h | |:------|:---:|:---:|:---:|:---:|:----:|:--------:| | Claude Opus 4.8 | 39.0 | 45.7 | 48.1 | 49.8 | 50.9 | 51.3 | | GPT-5.5 | 36.8 | 42.1 | 44.5 | 46.3 | 47.6 | 48.4 | | GPT-5.4 | 29.7 | 34.0 | 36.5 | 38.0 | 38.9 | 39.3 | | GLM-5.1 | 26.0 | 30.4 | 32.9 | 34.9 | 36.5 | 37.4 | | DS-V4-Pro | 23.3 | 27.1 | 29.0 | 29.9 | 30.9 | 31.0 |

Category Scores @12h (134 tasks)

| Model | Scientific & ML | Systems & SE | Optimization | Knowledge | Formal | Games | |:------|:-------:|:----:|:------------:|:---------:|:----:|:-----:| | Claude Opus 4.8 | 48.5 | 67.4 | 36.5 | 47.0 | 55.0 | 39.3 | | GPT-5.5 | 44.3 | 65.0 | 33.6 | 45.7 | 50.0 | 39.1 | | GPT-5.4 | 33.5 | 54.1 | 27.9 | 38.8 | 40.8 | 29.0 | | GLM-5.1 | 33.8 | 50.9 | 26.4 | 43.5 | 24.6 | 29.3 | | DS-V4-Pro | 30.0 | 43.0 | 21.5 | 37.0 | 14.1 | 16.9 |

Open-Source Subset (51 tasks)

| Model | @2h | @4h | @6h | @8h | @10h | @12h | |:------|:---:|:---:|:---:|:---:|:----:|:--------:| | Claude Opus 4.8 | 33.2 | 38.4 | 40.6 | 41.9 | 42.9 | 43.6 | | GPT-5.5 | 31.0 | 35.7 | 37.9 | 39.9 | 41.7 | 42.7 | | GPT-5.4 | 25.1 | 28.3 | 30.4 | 32.2 | 33.4 | 34.3 | | GLM-5.1 | 21.4 | 24.2 | 26.7 | 28.2 | 29.1 | 30.3 | | DS-V4-Pro | 17.1 | 21.0 | 22.8 | 23.5 | 24.6 | 25.1 |

Category Scores @12h (51 tasks)

| Model | Scientific & ML | Systems & SE | Optimization | Knowledge | Formal | Games | |:------|:-------:|:----:|:------------:|:---------:|:----:|:-----:| | Claude Opus 4.8 | 31.8 | 62.0 | 38.2 | 38.7 | 40.9 | 39.3 | | GPT-5.5 | 27.7 | 60.5 | 32.3 | 38.4 | 49.0 | 39.1 | | GPT-5.4 | 25.7 | 50.1 | 29.9 | 31.6 | 30.2 | 29.0 | | GLM-5.1 | 25.7 | 43.6 | 26.7 | 31.0 | 19.9 | 29.3 | | DS-V4-Pro | 23.8 | 37.6 | 24.1 | 33.2 | 12.7 | 16.9 |

Per-Task Scores by Time Budget (51 tasks)

Each model cell reports scores at @2h / @4h / @6h / @8h / @10h / @12h. Missing valid results are shown as —.

| Task | Category | Opus 4.8 | GPT-5.5 | GPT-5.4 | GLM-5.1 | DS-V4-Pro | |:-----|:-------|:---------|:--------|:--------|:--------|:----------| | bipedalwalker_locomotion_rl | Scientific & ML | 16.7/20.8/22.4/23.3/23.3/23.3 | 14.7/14.9/15.2/15.2/16.0/21.0 | 13.9/13.9/13.9/14.5/14.5/17.5 | 13.9/20.3/21.5/22.5/22.5/22.5 | 8.9/14.8/17.6/20.4/20.4/20.6 | | capecod_plume_reconstruction | Scientific & ML | 10.0/15.3/17.3/18.0/18.2/19.9 | 11.7/13.7/15.1/16.2/16.4/16.4 | 10.7/11.5/12.2/12.5/12.5/12.6 | 8.6/9.0/9.2/10.3/10.5/10.9 | 7.9/8.5/8.5/8.8/8.8/8.8 | | dabic_gravity_inversion | Scientific & ML | 9.5/15.2/15.7/17.4/17.5/17.5 | 15.9/16.2/16.7/17.0/17.2/17.3 | 14.6/14.6/15.5/15.5/15.0/15.0 | 9.2/13.7/16.0/16.5/16.5/17.1 | —/12.7/12.7/12.7/13.0/13.8 | | graph_node_classification | Scientific & ML | 59.4/62.7/65.0/65.6/66.5/66.6 | 54.7/55.1/55.1/55.3/55.9/56.0 | 54.9/56.2/56.5/56.9/57.5/57.6 | 49.4/52.3/52.3/52.3/52.3/52.3 | 46.0/48.2/49.2/51.3/51.7/51.8 | | ann_vector_search_qps | Systems & SE | 26.2/57.0/58.6/58.7/59.4/59.7 | 22.3/34.3/35.1/36.0/40.0/40.7 | 27.5/30.2/44.5/45.2/49.7/50.2 | 6.7/24.4/25.6/25.6/26.1/38.3 | 9.4/19.6/22.4/22.8/23.8/23.8 | | arc_compiler_runtime | Systems & SE | 49.3/52.0/52.0/52.0/52.0/52.0 | 55.5/56.5/60.9/70.3/71.0/72.4 | 45.1/46.5/49.8/49.8/50.0/50.0 | 47.7/48.0/48.4/48.7/48.7/48.7 | 40.3/41.7/44.2/44.2/44.2/44.2 | | exchange_core_throughput | Systems & SE | 40.7/57.0/58.5/58.9/59.7/59.7 | 15.4/37.2/39.9/44.3/51.3/53.2 | 14.3/40.8/41.0/45.2/46.4/47.3 | 29.2/43.7/46.5/48.6/50.3/52.6 | 32.9/33.8/45.0/47.7/48.4/48.6 | | ffmpeg_swscale_reimplementation | Systems & SE | 9.9/17.6/19.8/20.9/21.1/21.1 | 8.8/14.3/15.1/15.3/15.3/15.3 | 5.4/8.5/9.4/11.6/13.3/13.9 | 0.3/0.3/0.4/2.2/2.2/2.2 | 0.1/1.9/2.0/3.8/3.8/3.8 | | git_rewrite_in_zig | Systems & SE | 22.0/22.8/22.8/22.8/23.1/23.1 | 16.1/16.9/17.7/18.2/18.2/18.4 | 9.6/13.8/14.0/14.2/14.2/15.4 | 12.0/20.2/23.3/23.4/23.4/23.5 | 8.5/13.5/16.0/17.6/17.8/17.9 | | integer_compression_codec | Systems & SE | 69.4/69.7/74.8/74.9/75.2/75.3 | 61.1/67.6/73.9/73.9/74.3/74.4 | 38.6/40.9/41.2/42.2/42.2/42.3 | 23.5/27.3/28.5/28.7/28.9/28.9 | 15.9/16.0/16.2/16.2/16.2/16.2 | | juliet_vulnerability_analyzer | Systems & SE | 71.9/74.9/75.4/75.6/75.6/75.6 | 81.0/83.2/85.4/86.8/87.4/89.8 | 52.9/66.1/74.3/76.0/76.8/77.2 | 59.3/60.7/62.8/63.5/63.5/63.5 | 46.8/63.1/66.1/66.2/66.2/66.2 | | rust_multicrate_reconstruction | Systems & SE | —/—/—/—/—/— | 27.5/42.6/53.1/54.9/57.8/57.8 | 16.7/19.9/21.3/21.4/21.4/21.4 | 24.8/24.8/25.2/25.2/37.5/38.5 | 20.5/21.7/22.7/23.1/23.5/23.6 | | schemathesis_config_modernization | Systems & SE | 82.5/85.0/86.1/87.4/87.4/87.7 | 79.1/82.2/82.9/83.2/83.6/84.0 | 67.2/68.8/68.8/71.7/71.7/71.9 | 58.3/59.7/60.4/61.2/61.7/61.7 | 54.3/54.3/55.3/55.3/55.3/55.6 | | schemathesis_datagen_pipeline | Systems & SE | 68.0/70.2/70.2/70.2/70.2/70.2 | 54.6/54.6/56.7/56.7/56.7/56.7 | 56.6/56.6/56.6/56.6/56.6/56.6 | 62.1/64.2/64.2/67.0/67.0/67.0 | 47.9/50.1/52.3/52.3/52.3/52.3 | | schemathesis_reporting_observability | Systems & SE | 73.9/75.6/76.2/76.2/76.2/76.2 | 76.6/76.6/76.6/76.6/77.1/77.1 | 70.0/73.7/74.7/75.7/76.2/76.2 | 61.9/61.9/61.9/61.9/61.9/61.9 | 59.4/62.4/63.0/63.0/65.0/65.0 | | vliw_kernel_optimization | Systems & SE | 74.0/76.0/77.7/79.5/79.6/80.9 | 71.6/75.7/77.1/79.5/83.1/85.6 | 75.7/77.0/77.2/78.7/79.1/79.1 | 5.6/9.5/27.5/35.0/35.9/35.9 |...