tencent/Hy-Nomas-Diffusers

中文 ｜ English

Hy-Nomas: Towards Free-Form Interactive World Generation

BF16 weights for Diffusers · HyNomasModularPipeline

🚀 Try the Demo →

Model Introduction

Hy-Nomas is a video world generation model that supports free-form, multi-turn interaction through text instructions and camera control. It generates a world one video segment at a time, allowing users to introduce objects, change styles and scenes, describe actions, or control the camera as generation continues.

The model combines a segment-autoregressive diffusion transformer (DiT), a Qwen text encoder, camera-conditioning modules, and a PixelUMM video decoder. A persistent key/value (KV) cache carries context between segments. This directory provides the distilled model in BF16 Diffusers format. The example below uses 4 denoising steps, with 704 × 1280 output at 24 FPS.

Key Features

• Free-form interaction: Guide successive segments with instructions about subjects, actions, appearance, style, and environment.
• Camera control: Supply camera trajectories alongside text to control viewpoint and movement.
• Continuous generation: Reuse earlier context through a KV cache when extending the same world.
• Distilled inference: Generate each segment with 4 DiT denoising steps in the example below and decode its latents with a fine-tuned PixelUMM decoder.
• Diffusers integration: Load the bundled transformer, text encoder, tokenizer, scheduler, and decoder with HyNomasModularPipeline.

Hy-Nomas-Bench

We also release [Hy-Nomas-Bench](https://huggingface.co/datasets/tencent/Hy-Nomas-Bench), a suite of four benchmarks with 280 test cases covering multi-turn free-form text interaction, camera navigation, interleaved text and camera control, and spatial memory. It evaluates interactive controllability and consistency during continuous world generation, providing structured prompts, interaction sequences, keyboard timelines, and reference camera trajectories. See the Hy-Nomas-Bench dataset repository for downloads, data formats, and inference instructions.

Model Architecture

Hy-Nomas performs diffusion in a compressed video latent space and autoregression across video segments. Text, camera motion, and cached history condition the generation of each new segment.


*Hy-Nomas framework. The inference configuration uses 54 dual-stream MMDiT blocks (N = 54). Purple, blue, and green indicate the text, video, and camera paths, respectively.*

The figure's video-encoder path illustrates how video is represented as DCAE-V latents. In the text-to-video inference flow below, each new segment starts from Gaussian noise in that latent space and uses the KV cache for generated history. The MLLM is the Qwen text encoder, and the default video decoder is the fine-tuned PixelUMM described below.

Text and Camera Conditioning

Text conditioning. The text encoder uses the language-model component of Qwen/Qwen2.5-VL-7B-Instruct. Its 3,584-dimensional hidden states pass through a two-layer token refiner and are projected into the DiT's 2,048-dimensional hidden space. Segment-specific prompts let users change the requested content as generation proceeds.

Online prompt enhancement. The original web demo uses Nomas-PE, a separate service that converts user instructions into structured generation prompts. This Diffusers pipeline accepts plain text or prepared structured prompts directly and does not run Nomas-PE.

Camera conditioning. The inference configuration uses extrinsics_attn. Normalized 4 × 4 camera extrinsic matrices are grouped over four video frames and flattened into 64-dimensional inputs, aligning camera features with the temporal compression of the video latents. Learned pose embeddings and projections implement the figure's Pose Encoder → Camera Proj path. At each double-stream block, video tokens query the projected camera features through cross-attention after the video MLP. With per_chunk_camera_attn=true and camera_chunk_size=1, camera attention is aligned to individual latent frames.

Segment-Autoregressive Diffusion Transformer

The generation backbone is a dense multimodal diffusion transformer (MMDiT). Video latents enter through Patch Embed, while refined text tokens enter a separate stream. The Timestep Embed supplies noise-level conditioning: each block uses it to modulate the two streams with learned shifts, scales, and residual gates.

Within a block, text and video interact through segment-wise causal joint attention, with 3D RoPE encoding video positions and the layer's KV cache supplying context from earlier segments. The two streams then pass through separate MLPs. Camera cross-attention updates the video branch using video queries and camera keys/values. Text features continue to the next block; after the last block, the output head uses only the video stream to predict latent-space flow velocity. The sampler uses these predictions to denoise the segment before video decoding.

The model architecture is illustrated in the following table:

| Property | Configuration | |:---|:---| | Model identifier | Hy-Nomas | | Transformer blocks | 54 double-stream blocks; 0 single-stream blocks | | Hidden size | 2,048 | | Attention heads | 16 heads; 128 dimensions per head | | Feed-forward expansion | 4×; intermediate size 8,192 | | Latent input / output channels | 32 / 32 | | Latent patch size, time × height × width | 1 × 1 × 1 | | Position encoding | 3D RoPE; temporal/height/width dimensions [16, 56, 56]; theta 256 | | Text conditioning | 3,584-dimensional Qwen features; two-layer token refiner | | Camera conditioning | Extrinsic embeddings with cross-attention in the double-stream blocks | | Denoising prediction | Flow velocity; Euler ODE sampling in the inference configuration |

At each turn, the DiT denoises the current segment while attending to all past KV cache and the current conditioning. Attention is causal across segments, so future segments are not required to generate the current one. Cached attention keys and values are reused when generation advances to the next segment....