ByteDance-Seed/manip-as-in-sim-suite

ByteDance-Seed/manip-as-in-sim-suite

Description: Sim-to-real and CDM inference code for ManipAsInSim project.

Language: Python

License: Apache-2.0

Stars: 154

Forks: 6

Open issues: 9

Created: 2025-08-08T09:39:36Z

Pushed: 2025-12-09T08:31:31Z

Default branch: main

Fork: no

Archived: no

README:

Manipulation As in Simulation Suite

This repository contains the open-source implementation of the paper "Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots". The suite provides tools for bridging the sim-to-real gap in robotic manipulation through high-quality depth perception and automated demonstration generation.

📋 Table of Contents

• [🎯 Overview](#-overview)
• [🔍 CDM (Camera Depth Models)](#-cdm-camera-depth-models)
• [🤖 WBCMimic](#-wbcmimic)
• [✨ Key Features](#-key-features)
• [🚀 Quick Start](#-quick-start)
• [Prerequisites](#prerequisites)
• [Installation](#installation)
• [🔧 Environment Setup](#-environment-setup)
• [Environment Variables](#environment-variables)
• [Setup Options](#setup-options)
• [Verification](#verification)
• [Troubleshooting](#troubleshooting)
• [🚀 Usage](#-usage)
• [CDM Usage](#cdm-usage)
• [WBCMimic Usage](#wbcmimic-usage)
• [🔬 Research Contributions](#-research-contributions)
• [Camera Depth Models (CDMs)](#camera-depth-models-cdms)
• [WBCMimic Enhancements](#wbcmimic-enhancements)
• [🎯 Supported Tasks & Hardware](#-supported-tasks--hardware)
• [Robotic Tasks](#robotic-tasks)
• [Supported Cameras](#supported-cameras)
• [📚 Documentation](#-documentation)
• [📄 Citation](#-citation)
• [📝 License](#-license)
• [🔗 Links](#-links)
• [📧 Contact](#-contact)

🎯 Overview

The suite consists of two main components that enable seamless sim-to-real transfer for robotic manipulation:

🔍 CDM (Camera Depth Models)

A depth estimation package that produces clean, simulation-like depth maps from noisy real-world camera data. CDMs enable policies trained purely in simulation to transfer directly to real robots by providing perfect depth perception.

🤖 WBCMimic

An enhanced version of MimicGen that extends autonomous data generation to mobile manipulators with whole-body control. It enables efficient generation of high-quality manipulation demonstrations through automated pipelines with multi-GPU parallel simulation.

✨ Key Features

• Sim-to-Real Depth Transfer: Clean, metric depth estimation that matches simulation quality
• Multi-Camera Support: Pre-trained models for various depth sensors (RealSense, ZED, Kinect)
• Automated Data Generation: Scalable demonstration generation using enhanced MimicGen
• Whole-Body Control: Unified control for mobile manipulators for mimicgen
• Multi-GPU Parallelization: Distributed simulation for faster data collection
• VR Teleoperation: Intuitive demonstration recording using Meta Quest controllers

🚀 Quick Start

Prerequisites

• Python 3.10+
• CUDA-capable GPU (recommended)
• Isaac Lab 2.1
• CuRobo for motion planning

Installation

1. Clone the repository:

git clone https://github.com/your-org/manip-as-in-sim-suite.git
cd manip-as-in-sim-suite

2. Install CDM:

cd cdm
pip install -e .

3. Install WBCMimic:

cd ../wbcmimic
pip install -e source/isaaclab_mimic

4. Configure Environment Variables (Required for WBCMimic):

# Set paths to your Isaac Lab and CuRobo installations
export ISAACLAB_DIR="/path/to/IsaacLab"
export CUROBO_DIR="/path/to/curobo"

# Set paths to robot assets
export X7_URDF_PATH="/path/to/X7/urdf/X7_2.urdf"
export UR5_URDF_PATH="/path/to/UR5/ur5_isaac_simulation/robot.urdf"
export ROOM_USD_PATH="/path/to/Room_empty_table.usdc"
export ISAAC_SIM_FRANKA_PATH="/path/to/isaac-sim/src/franka"

🔧 Environment Setup

This section explains how to configure the environment variables needed for the manipulation-as-in-simulation suite. The codebase uses configurable environment variables and fallback paths to make the code portable and maintainable.

Environment Variables

Robot URDF Paths

##### X7_URDF_PATH

• Purpose: Path to the ARX-X7 robot URDF file
• Used in: wbc_controller_dual.py
• Default fallback: ../../../../../assets/X7/urdf/X7_2.urdf (relative to the controller file)
• Example:

export X7_URDF_PATH="/path/to/your/X7/urdf/X7_2.urdf"

##### UR5_URDF_PATH

• Purpose: Path to the UR5 robot URDF file
• Used in: wbc_controller.py, wbc_controller_cuda.py, test_robot_ik.ipynb
• Default fallback: ../../../../../assets/UR5/ur5_isaac_simulation/robot.urdf (relative to the controller file)
• Example:

export UR5_URDF_PATH="/path/to/your/UR5/ur5_isaac_simulation/robot.urdf"

Scene Assets

##### ROOM_USD_PATH

• Purpose: Path to the room USD scene file
• Used in: record_demos_quest.py
• Default fallback: ../../../assets/Room_empty_table.usdc (relative to the script)
• Example:

export ROOM_USD_PATH="/path/to/your/Room_empty_table.usdc"

Isaac Sim Integration

##### ISAAC_SIM_FRANKA_PATH

• Purpose: Path to Isaac Sim's Franka source directory
• Used in: vr_policy.py files
• Default fallback: Tries common installation paths:
• ~/.local/share/ov/pkg/isaac-sim-4.0.0/src/franka
• /opt/nvidia/isaac-sim/src/franka
• /usr/local/isaac-sim/src/franka
• Example:

export ISAAC_SIM_FRANKA_PATH="/your/isaac-sim/installation/src/franka"

Setup Options

Option 1: Set Environment Variables (Recommended)

Create a shell script or add to your .bashrc/.zshrc:

# Robot assets - adjust paths according to your setup
export X7_URDF_PATH="/path/to/IsaacLab/source/arxx7_assets/X7/urdf/X7_2.urdf"
export UR5_URDF_PATH="/path/to/IsaacLab/source/arxx7_assets/UR5/ur5_isaac_simulation/robot.urdf"

# Scene assets
export ROOM_USD_PATH="/path/to/your/scene/Room_empty_table.usdc"

# Isaac Sim (if not in standard location)
export ISAAC_SIM_FRANKA_PATH="/path/to/isaac-sim/src/franka"

# Source the environment
source ~/.bashrc # or ~/.zshrc

Option 2: Use Default Fallback Paths

If you don't set environment variables, the code will use relative paths. Ensure your asset directory structure follows this layout:

manip-as-in-sim-suite/
├── assets/
│ ├──…