inclusionAI/BiDex-100

inclusionAI/BiDex-100

Language: Python

License: MIT

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-10-12T05:59:07Z

Pushed: 2026-10-13T05:55:32Z

Default branch: master

Fork: no

Archived: no

README:

A comprehensive benchmark for bimanual dexterous manipulation across 100+ simulation tasks

Website · Datasets · 3D Assets

Overview

BiDex-100 evaluates vision-language-action policies on coordinated, bimanual, dexterous manipulation. It combines declarative task definitions, reproducible MuJoCo environments, matched in-domain and out-of-domain evaluation, 3D assets, and 8,080 success-validated demonstrations.

• 106 tasks across five capability axes: Memory, Long-Horizon,

Instruction-Following, Limited-Demonstrations, and Generalization.

• Two bimanual embodiments: dual Franka FR3 v2 arms and Astribot S1,

both equipped with dexterous hands.

• 8,080 demonstrations and 6.25M frames: six synchronized RGB-D streams at

25 FPS, released as task-level LeRobot v2.1 and v3.0 datasets.

• 58-D unified action space: 14 arm/end-effector dimensions plus 44

dexterous-hand joint dimensions.

• Matched ID/OOD protocol: deterministic seeds, scene and object shifts,

predicate-based progress scoring, and a strict score-100 success criterion.

Installation

git clone https://github.com/BiDex-100/BiDex-100.git
cd BiDex-100

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m pip install -e .

The Hugging Face dataset repository has three independently downloadable components:

assets/ compressed simulation asset packs
datasets/lerobotv21/ LeRobot v2.1 datasets
datasets/lerobotv30/ LeRobot v3.0 datasets

Download everything and prepare the simulation assets:

python -m pip install -U huggingface_hub
hf auth login # omit for public, unauthenticated downloads

bash scripts/download_release.sh all downloads/BiDex-100
bash scripts/setup_assets_links.sh downloads/BiDex-100

Each component can instead be downloaded separately into the same destination:

bash scripts/download_release.sh assets downloads/BiDex-100
bash scripts/download_release.sh lerobotv21 downloads/BiDex-100
bash scripts/download_release.sh lerobotv30 downloads/BiDex-100

scripts/download_assets.sh and scripts/download_datasets.sh remain as convenience wrappers. Run download_datasets.sh lerobotv21, download_datasets.sh lerobotv30, or download_datasets.sh all to fetch one or both dataset formats. The download destination preserves the repository layout shown above.

The asset setup script accepts either downloads/BiDex-100 or downloads/BiDex-100/assets. It verifies each archive against assets/SHA256SUMS, extracts it in place, creates the repository links, and recreates the relative aliases used by assembly tasks. It is idempotent, so an interrupted setup can be rerun safely. Alternatively set BIDEX_ASSETS=/path/to/BiDex-100/assets. Export HF_TOKEN instead of using hf auth login in non-interactive environments.

For headless GPU rendering, select EGL before starting Python:

export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl

Tasks

Task YAML files under bidex/tasks_data/suites//.yaml define the scene, robot, objects, initialization distribution, instruction, success predicates, subtasks, and OOD counterpart. The five suites contain 10, 10, 20, 26, and 40 tasks respectively (106 total).

This minimal example is complete and directly runnable. It disables cameras so that the first simulation check does not require a graphics context, resets one task, holds the reset robot pose, advances physics with env.step, and closes all MuJoCo resources:

from bidex import BiDexEnv, load_task

task = load_task("Memory", "uncover_red_cube")
env = BiDexEnv(task, control_mode="joint", cameras=[])

try:
obs = env.reset(seed=0)
action = {
"arm_joint_position": obs["arm_joint_position"].copy(),
"dex_joint_position": obs["dex_joint_position"].copy(),
}

for _ in range(64):
obs, reward, terminated, info = env.step(action)
if terminated:
break

print({"reward": reward, "terminated": terminated, "score": info.get("score")})
finally:
env.close()

Use list_suites() and list_tasks() to discover tasks. Pass cameras=["head", "left_wrist", "right_wrist"] when RGB observations are needed.

Training

BiDex-100 is model-agnostic: this repository provides the benchmark and data contract, while each VLA implementation owns its trainer. The supported 58-D representations are:

| Mode | Arm component | Hand component | | --- | --- | --- | | joint | actions.arm_joint_position (14-D) | actions.dex_joint_position (44-D) | | robot_ee | actions.abs_end_effector (14-D) | actions.dex_joint_position (44-D) | | camera_ee | actions.abs_end_effector_camera (14-D) | actions.dex_joint_position (44-D) |

Use bidex.tasks.benchmark.BENCHMARKS and MODES as the canonical suite and modality registry when adding a trainer integration.

Reported training setup

The currently reported model results use co-training: demonstrations from multiple benchmark tasks are merged into one training corpus and used to train one shared policy checkpoint. Evaluation then runs that same checkpoint across all applicable tasks; the reported numbers are not an ensemble of separately trained per-task policies.

Evaluation

BiDex-100 exposes one evaluation contract for different VLA policies. A model adapter receives the same three camera observations, proprioceptive state, and language instruction; the benchmark owns task discovery, seeded resets, ID/OOD randomization, rollout execution, scoring, videos, and result schemas. This keeps model comparisons independent of model-specific launch code.

ID/OOD protocol summary

The canonical protocol evaluates 25 episodes for each of two ID seeds (0,1) and two OOD seeds (2,3). Both domains use deterministic per-episode reset seeds and the same task instructions, success predicates, rollout horizon, and scoring rules.

| Setting | In-domain (ID) | Out-of-domain (OOD) | | --- | --- | --- | | Task definition | Base task YAML. | The task's committed __ood.yaml variant (all 106 tasks are covered). | | Robot initial state | Deterministically samples a recorded working pose from the task's evaluation H5 and restores robot joints only; objects retain the fresh base-YAML placement. | Uses the task's default robot reset pose; it does not sample an H5...