nvidia/lidarverse

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

Henry Che1,2   Xuanchi Ren1   Sanja Fidler1   Shenlong Wang2   Laura Leal-Taixé1   Shengyu Huang1

1NVIDIA 2University of Illinois Urbana-Champaign

Watch the LidarVerse teaser

Quickstart | Fine-tuning | Model card

Abstract

LidarVerse is a real-time unified LiDAR world model for generation, point-cloud forecasting, and generalizable resimulation. It combines the flexibility of generative modeling with reconstruction-grounded simulation without per-scene optimization. An attention-based tokenizer maps different LiDAR scanning patterns into a shared latent space, while flexible conditioning on observed LiDAR, HDMaps, and accumulated geometry enables one model to perform all three tasks. Self-forcing distillation produces a four-step causal model; a rolling temporal K/V cache and a persistent spatial cache support long-horizon generation at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across the three evaluation protocols. The framework also supports cross-sensor generation with intensity, and its tokenizer generalizes to an unseen scanning pattern.

Stage 3 Quickstart

lidarverse-sfdmd-pai/ contains the EMA-only SF-DMD student. Use the current LidarVerse code and run from its repository root:

python scripts/download_checkpoints.py
CUDA_VISIBLE_DEVICES=0 STAGE=3 MODE=all NUM_SAMPLES=1 \
NUM_STEPS=4 SAMPLER=sde SDE_ETA=1 GUIDANCE=1 SEED=0 \
bash scripts/diffusion/getting_started.sh

The single-sample quickstart uses the second official NVIDIA PAI clip. Sampling uses four-frame blocks, KV9 including the sink, FP32 tokenizer encoding/decoding, and a persistent spatial cache with 0.10 m voxels. HDMap generation rasterizes raw lanes and actor edges with calibrated rays and rolling-shutter poses, then widens only bbox returns to 3x3 pixels before encoding. Set RAW_HDMAP_CONDITION_RASTERIZE=0 for the original cached condition; forecasting/resimulation do not use this HDMap override. Set PREPARE_DATA=0 to reuse prepared examples.

The package contains no regular-model, fake-score, optimizer, or training state. Fine-tuning from it starts a new run, not an exact full-state continuation. Its teacher came from the same causal-training lineage as the released Stage 2, but not from the exact Stage-2 snapshot packaged here.

LidarVerse Overview

> Current checkpoint scope: the quickstarts use the range-only NVIDIA PAI > tokenizer and PAI bidirectional, causal, and SF-DMD EMA > packages, with 128x1024 inputs at 10 Hz and 2.5--100 m support. They do not > include intensity or the paper's cross-sensor checkpoints. The broader > architecture and dataset descriptions below describe the research project, > not a claim that these particular packages support every sensor. Quickstart > throughput is not a paper benchmark.

Description:

LidarVerse is a real-time unified LiDAR world model for sensor simulation across tasks and sensors. LidarVerse bridges both generative and reconstruction-based simulation into a single, unified conditional generative framework. It is composed of a universal, fully attention-based tokenizer maps LiDAR range sequences with different scanning patterns into a shared latent space; a bidirectional diffusion teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text; and a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon roll-outs with real-time inference at 11 FPS on a single H100 GPU. LidarVerse was developed by NVIDIA. _This model is for research and development only._

License/Terms of Use:

NVIDIA Open Model License

Deployment Geography:

Global

Use Case:

This model is intended for research community to study, ablate, and build upon of.

Release Date:

GitHub: 09/11/2026 via https://github.com/nv-tlabs/lidarverse

Reference(s):

Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models

Model Architecture:

Architecture Type: Transformer Network Architecture: Diffusion Transformer (DiT) This model was developed based on Cosmos-Predict2-T2V-2B. Number of model parameters: 2B

Input:

Input Type(s): Video Input Format(s): Lidar Input Parameters: Three-Dimensional (3D) Other Properties Related to Input: Flexible resolution and length

Output:

Output Type(s): Video Output Format: Lidar Output Parameters: Three-Dimensional (3D) Other Properties Related to Output: Flexible resolution and length, streaming supported

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration:

Runtime Engine(s): Not Applicable (N/A) Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper
• NVIDIA Lovelace

Supported Operating System(s): Linux

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s):

LidarVerse v1.0 — 2B-parameter trained research model

Training, Testing, and Evaluation Datasets:

Training Dataset:

Data Modality:

• Video
• Text
• Image

Image Training Data Size: 1 Million to 1 Billion Images Text Training Data Size: Less than a Billion Tokens Video Training Data Size: 10,000 to 1 Million Hours **Data...