NousResearch/hermes-toolperf-evals

NousResearch/hermes-toolperf-evals

Description: Core-toolset A/B eval harness + benchmark cases behind the 15-PR tool-efficiency tracker (hermes-agent #77056). Built from session-DB mining, NeMo Relay traces, and prod error taxonomy.

Language: Python

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-08-06T16:14:08Z

Pushed: 2026-08-06T17:16:06Z

Default branch: main

Fork: no

Archived: no

README:

hermes-toolperf-evals

The complete eval set behind the hermes-agent core-toolset performance batch — 15 PRs + 1 integration fix, all merged (tracker: hermes-agent#77056).

Built from production session-DB mining (~1.5M messages), a schema token audit, and NVIDIA NeMo Relay ATOF traces for ground-truth turn accounting.

Layout

| dir | what | |---|---| | abeval/ | The hard A/B eval: 9 error-inducing benchmark cases, baseline-tree vs fixes-tree (arms differ only by PYTHONPATH), 2 models × 3 reps, ATOF-scored. This is the harness behind the published before/after numbers. Parameterized — see its README. | | bench/ | The earlier 11-task baseline battery (8 generic + 3 hint-targeted) that established the waste profile. Preserved as recovered; original hardcoded paths noted in its README. | | analysis/ | The mined data: CANDIDATES.md (P1–P14 pain points with magnitudes from the 400k-message window) and PR_DESIGNS.md (per-PR designs with the DB frequency justifying each rule). | | results/ | Checked-in run data. 2026-08-06_rerun/ is the canonical reproducible dataset: pinned SHAs, meta.jsonl per cell, full ATOF traces, regeneration instructions, and an honest readout vs the Aug 2 published numbers. | | docs/ | METHODOLOGY.md — how the targets were derived (the answer to "how did you find these"); RESULTS.md — the Aug 2 published A/B tables + readout (raw traces lost; superseded as canonical by results/2026-08-06_rerun/). |

The 9 A/B benchmark cases

err_python_env, err_replay_patch, err_ambiguous_edit, err_case_search, err_hidden_search, err_big_output, err_multi_dir, err_inline_script, err_big_file_read — each derived from a mined production waste class, with a sandbox fixture and a deterministic success marker.

Headline numbers (weak model, where tool-ergonomics waste manifests)

−21% LLM turns · −29% tool calls · tool errors and retry-after-error → 0 · −33% result bytes into context · −23% wall clock. Strong model: parity (it already recovers in one turn — which is why weak-model runs are the signal).

Running the A/B eval against your own model (e.g. Nemotron Ultra)

cd abeval
./run_all.sh [reps] [models...]

The harness shells out to the hermes CLI; any provider model id works. Startup crashes (nonzero exit, empty output) retry on resume instead of being recorded as data points. Report the mean of ≥3 reps per cell; n=3 deltas under a few percent are noise.

The A/B harness also ships inside hermes-agent at scripts/toolperf_abeval/ (landed in PR #79162); this repo is the canonical home for the full eval set including the pieces that don't belong in the product tree.