OpenBMB/SimpleMemVLA

OpenBMB/SimpleMemVLA

Description: Native-video memory for vision-language-action models, using timestamped visual history and exact streaming inference for long-horizon robot manipulation.

Language: Python

License: MIT

Stars: 31

Forks: 5

Open issues: 1

Created: 2026-08-29T16:37:48Z

Pushed: 2026-09-09T17:25:47Z

Default branch: main

Fork: no

Archived: no

README:

🔥 SimpleMemVLA 🔥

A Simple but Effective Native-Video Memory for Vision-Language-Action Models

Run 1 — blue, green, red Run 2 — green, red, blue Run 3 — red, green, blue

🔗 Quick Links --------------

[🧠 Overview](#-overview) | [✨ Highlights](#-highlights) | [🏗️ Architecture](#%EF%B8%8F-architecture--streaming-inference) | [📊 Results](#-results) | [🤖 Benchmarks](#-supported-benchmarks) | [🛠️ Setup](#%EF%B8%8F-setup) | [💾 Data & Checkpoints](#-data--checkpoints) | [🚀 Training](#-training) | [🧪 Evaluation](#-evaluation) | [🦾 Real Robot](#-real-robot-deployment) | [🧩 Extending](#-extending-to-a-new-benchmark) | [📄 Citation](#-citation)

📰 News -------

• [2026-09] SimpleMemVLA deployed on a real dual-arm robot for the *cover blocks* memory

task — three autonomous rollouts, three block layouts, lids removed in red → green → blue ([🦾 Real-Robot Deployment](#-real-robot-deployment)).

• [2026-09] Preprint on arXiv: arXiv:2609.05533

(PDF).

• [2026-08] Initial release: paper, full training + closed-loop evaluation code for five

benchmarks, LeRobot-v3 datasets and one released checkpoint per suite.

📝 TODO -------

• [x] arXiv preprint release (arXiv:2609.05533)
• [x] Real-robot deployment — *cover blocks* on a dual-arm robot, policy running autonomously

([rollouts](#-real-robot-deployment))

🧠 Overview -----------

Long-horizon manipulation is partially observable: the information needed to choose the next action may appear only in observations from minutes earlier. Existing memory mechanisms — retrieval banks, learned compressors, recurrent states — must decide what to keep from the past before knowing what a future decision will require (*write-time commitment*): a relevant frame may be skipped by retrieval, visual detail lost in compression, earlier evidence overwritten by a recurrent update.

Those designs were motivated by the assumption that minute-scale history is too large to process directly — an assumption modern VLM backbones no longer make true. SimpleMemVLA is a VLA without a dedicated memory module: it keeps the sampled history intact, presents it in the timestamped video format the backbone was pretrained to process, and defers evidence selection to native self-attention at decision time. The hidden states of a generated sub-task then form the only channel from history to a standard flow-matching action head, and shared-prefix prefilling keeps decision latency close to a single-frame VLA.

✨ Highlights ------------

• 🥇 State of the art on four memory benchmarks, one model per suite — RMBench 94.0

(+11.0 over per-task specialists), RoboMME 88.3 (+43.7, above the ground-truth-perception oracle), MIKASA-Robo 74.0 (+29.6 over the best prior VLA), RoboMemArena 63.6 TSR (+17.4, above the benchmark's own GT reference) — with no cost on general-purpose control: 97.5 on LIBERO (ties the best reported average) and the strongest zero-shot transfer to LIBERO-Plus (78.4).

• 🧲 The gain is the memory interface, not the backbone — rebuilding retrieval, token

compression and recurrent state on the *same* backbone, data and trainer reaches only 31.5 / 22.6 / 20.6 on RoboMME vs 88.3 for native context.

• ⚡ Exact streaming inference — the shared history prefix is prefilled while the robot

executes the current chunk; decision latency drops 1.02 s → 0.68 s with byte-identical outputs.

• 🧰 One repo, five benchmarks — one model, one data pipeline, one trainer; everything

benchmark-specific is a small spec module under [simplememvla/benchmarks/](simplememvla/benchmarks/).

🏗️ Architecture & Streaming Inference --------------------------------------

Only standard VLA components; self-attention over plaintext-timestamped history serves as memory.

1. History as native video. At every decision the policy rebuilds a window over the last history_video_sec seconds, subsampled at history_video_fps from the native control-rate stream. The main camera's clip enters the Qwen3.5-4B backbone through its video channel — the processor prefixes every 2-frame temporal patch with a plaintext ` timestamp, exactly as in video pretraining. Wrist cameras contribute only their current frame through the image channel, so the input format itself separates past from present. The window is a *cap*, not a fixed length: young rollouts feed only their real frames, cropped/padded by one shared even-length rule ([simplememvla/data/messages.py](simplememvla/data/messages.py) variable_history_frames) used byte-identically by the training dataset and every rollout policy. 2. **A narrow text channel from history to action.** The backbone is supervised (token-level cross-entropy) to state the current **sub-task** as an ordinary assistant answer — no chain-of-thought, capped at 64 tokens. The hidden states and token embeddings of that span, plus one normalized proprio token, are the *only* conditioning a DiT flow-matching expert (~0.9B params) receives. The same span mask (simplememvla.data.collator.subtask_span_mask) selects the supervised tokens at training and the generated tokens at inference, so train/rollout conditioning can never drift. Deployment integrates the learned velocity field with 10 deterministic Euler steps, executes the first execute_horizon` actions of the chunk, and re-decides (receding horizon). 3. Exact streaming deployment. Consecutive decisions share nearly their entire video prefix, so it is prefilled during action execution (per-temporal-patch ViT feature cache + KV-prefix reuse) and only the newly arrived patch, wrist frames and text stay on the critical path. Outputs are exactly those of full recomputation — all reported evaluations run this path.

Every prompt/pipeline detail an eval policy needs (cameras, window, fps, robot tag, control frequency, sub-task column, variable-history flag) is persisted into the checkpoint's config.json, so closed-loop evaluation rebuilds a byte-identical pipeline from the...