stepfun-ai/Step-Video-T2V

stepfun-ai/Step-Video-T2V

Language: Python

License: MIT

Stars: 3185

Forks: 338

Open issues: 47

Created: 2025-02-08T08:46:51Z

Pushed: 2025-03-17T03:26:54Z

Default branch: main

Fork: no

Archived: no

README:

🔥🔥🔥 News!!

• Mar 17, 2025: 👋 We release the Step-Video-TI2V, an image-to-video model based on Step-Video-T2V.
• Feb 17, 2025: 👋 We release the inference code and model weights of Step-Video-T2V. Download
• Feb 17, 2025: 👋 We release the inference code and model weights of Step-Video-T2V-Turbo. Download
• Feb 17, 2025: 🎉 We have made our technical report available as open source. Read

Video Demos

Table of Contents

1. [Introduction](#1-introduction) 2. [Model Summary](#2-model-summary) 3. [Model Download](#3-model-download) 4. [Model Usage](#4-model-usage) 5. [Benchmark](#5-benchmark) 6. [Online Engine](#6-online-engine) 7. [Citation](#7-citation) 8. [Acknowledgement](#8-ackownledgement)

1. Introduction

We present Step-Video-T2V, a state-of-the-art (SoTA) text-to-video pre-trained model with 30 billion parameters and the capability to generate videos up to 204 frames. To enhance both training and inference efficiency, we propose a deep compression VAE for videos, achieving 16x16 spatial and 8x temporal compression ratios. Direct Preference Optimization (DPO) is applied in the final stage to further enhance the visual quality of the generated videos. Step-Video-T2V's performance is evaluated on a novel video generation benchmark, Step-Video-T2V-Eval, demonstrating its SoTA text-to-video quality compared to both open-source and commercial engines.

2. Model Summary

In Step-Video-T2V, videos are represented by a high-compression Video-VAE, achieving 16x16 spatial and 8x temporal compression ratios. User prompts are encoded using two bilingual pre-trained text encoders to handle both English and Chinese. A DiT with 3D full attention is trained using Flow Matching and is employed to denoise input noise into latent frames, with text embeddings and timesteps serving as conditioning factors. To further enhance the visual quality of the generated videos, a video-based DPO approach is applied, which effectively reduces artifacts and ensures smoother, more realistic video outputs.

2.1. Video-VAE

A deep compression Variational Autoencoder (VideoVAE) is designed for video generation tasks, achieving 16x16 spatial and 8x temporal compression ratios while maintaining exceptional video reconstruction quality. This compression not only accelerates training and inference but also aligns with the diffusion process's preference for condensed representations.

2.2. DiT w/ 3D Full Attention

Step-Video-T2V is built on the DiT architecture, which has 48 layers, each containing 48 attention heads, with each head’s dimension set to 128. AdaLN-Single is leveraged to incorporate the timestep condition, while QK-Norm in the self-attention mechanism is introduced to ensure training stability. Additionally, 3D RoPE is employed, playing a critical role in handling sequences of varying video lengths and resolutions.

2.3. Video-DPO

In Step-Video-T2V, we incorporate human feedback through Direct Preference Optimization (DPO) to further enhance the visual quality of the generated videos. DPO leverages human preference data to fine-tune the model, ensuring that the generated content aligns more closely with human expectations. The overall DPO pipeline is shown below, highlighting its critical role in improving both the consistency and quality of the video generation process.

3. Model Download

| Models | 🤗Huggingface | 🤖Modelscope | |:-------:|:-------:|:-------:| | Step-Video-T2V | download | download | Step-Video-T2V-Turbo (Inference Step Distillation) | download | download

4. Model Usage

📜 4.1 Requirements

The following table shows the requirements for running Step-Video-T2V model (batch size = 1, w/o cfg distillation) to generate videos:

| Model | height/width/frame | Peak GPU Memory | 50 steps w flash-attn | 50 steps w/o flash-attn | |:------------:|:------------:|:------------:|:------------:|:------------:| | Step-Video-T2V | 768px768px204f | 78.55 GB | 860 s | 1437 s | | Step-Video-T2V | 544px992px204f | 77.64 GB | 743 s | 1232 s | | Step-Video-T2V | 544px992px136f | 72.48 GB | 408 s | 605 s |

• An NVIDIA GPU with CUDA support is required.
• The model is tested on four GPUs.
• Recommended: We recommend to use GPUs with 80GB of memory for better generation quality.
• Tested operating system: Linux
• The self-attention in text-encoder (step_llm) only supports CUDA capabilities sm_80 sm_86 and sm_90

🔧 4.2 Dependencies and Installation

• Python >= 3.10.0 (Recommend to use Anaconda or Miniconda)
• PyTorch >= 2.3-cu121
• CUDA Toolkit
• FFmpeg

git clone https://github.com/stepfun-ai/Step-Video-T2V.git
conda create -n stepvideo python=3.10
conda activate stepvideo

cd Step-Video-T2V
pip install -e .
pip install flash-attn --no-build-isolation ## flash-attn is optional

🚀 4.3 Inference Scripts

Multi-GPU Parallel Deployment

• We employed a decoupling strategy for the text encoder, VAE decoding, and DiT to optimize GPU resource utilization by DiT. As a result, a dedicated GPU is needed to handle the API services for the text encoder's embeddings and VAE decoding.

python api/call_remote_server.py --model_dir where_you_download_dir & ## We assume you have more than 4 GPUs available. This command will return the URL for both the caption API and the VAE API. Please use the returned URL in the following command.

parallel=4 # or parallel=8
url='127.0.0.1'
model_dir=where_you_download_dir

tp_degree=2
ulysses_degree=2

# make sure tp_degree x ulysses_degree = parallel
torchrun --nproc_per_node $parallel run_parallel.py --model_dir $model_dir --vae_url $url --caption_url $url…