Tencent-Hunyuan/UniRL v0.2.0

UniRL v0.2.0

Repository: Tencent-Hunyuan/UniRL

Tag: v0.2.0

Published: 2026-10-08T08:43:40Z

Prerelease: no

Release notes:

Highlights

UniRL v0.2.0 is the first release since the initial v0.1.0 drop and folds in ~300 merged PRs. It focuses on four themes:

• Much broader model coverage: 12 new model packages — Z-Image, Qwen-Image-Edit-2511, Boogu-Image, LTX-2 / LTX-2.3 (text-to-audio+video), WAN 2.2 V2V, MiniMax-H3 (text-to-video+audio), SenseNova-U1.5, Janus-Pro, Qwen3.5, Qwen3-MoE (VeOmni EP), Qwen3-Omni Thinker, and Cosmos3 — plus new modes for BAGEL and HunyuanImage 3.0.
• New training modes: supervised fine-tuning (train_sft), multi-turn agentic RL (train_agentic), and disaggregated async RL for both AR (train_async_ar) and diffusion (train_async_diffusion).
• More algorithms: CPPO, AR DPPO, GSPO, PPO with GAE + value head, DiffusionOPD (on-policy distillation), and a reference-model KL for DiffusionNFT.
• Current rollout stacks: rebuilt SGLang Diffusion and vLLM-Omni engines, a new direct vLLM TP engine and FastVideo engine, TP/EP weight sync for full weights and LoRA, all on SGLang 0.5.19 / vLLM-Omni 0.28 / torch 2.13 + CUDA 13.

Models

• Image diffusion: Z-Image with trainside and SGLang Diffusion rollout (#41, #73); Qwen-Image-Edit-2511 with DiffusionNFT and FlowGRPO (#128, #171); Boogu-Image-0.1 (#219); FLUX.2-Klein image-edit conditioning and ti2i rollout (#79, #242); Qwen-Image SGLang / vLLM rollout engines (#34).
• Video diffusion: LTX-2 T2V and LTX-2.3 T2AV with a joint audio+video SDE policy (#91, #115, #233); WAN 2.2 V2V (#113); WAN 2.1 and WAN 2.2-A14B on SGLang Diffusion (#183, #184); MiniMax-H3 T2VA Flow-GRPO (#347, #403, #513); HunyuanVideo 1.5 FlashAttention (#427).
• Unified models: BAGEL-7B-MoT multi-mode RL (it2i / t2t / i2t / it2t / t2ti) and UniGRPO on a single trainside (#62, #100, #148); BAGEL on vLLM-Omni (#89); HunyuanImage 3.0 all-modality generation, it2i GRPO, pure-T2I FlowGRPO, and VeOmni expert parallelism (#61, #186, #342, #140); SenseNova-U1.5 pixel flow with OCR recipes (#389, #407).
• AR / omni: Janus-Pro i2t and t2i (#141); Qwen3.5 text + single-image (#174); Qwen3-Omni Thinker on vLLM-Omni with text, video, audio-in-video, standalone audio and standalone image training (#235, #260, #306, #309).
• World model: Cosmos3 omnimodal SFT adapters for video prediction and action BC (#200).

Training Modes & Trainers

• SFT domain covering CE and flow-matching over the shared TrainStack (#209), with agent-conversation, interleaved image-text trajectory, Qwen3-Omni audio, and WAN 2.1 full-transformer SFT (#234, #352, #345, #320).
• Agentic multi-turn rollout and training on Sample lineages (#214, #315, #363).
• Disaggregated async trainers: AsyncARTrainer and AsyncDiffusionTrainer, with staleness budgeting, dynamic prompt dispatch, and rollout refill before scoring (#68, #192, #310, #289, #213).
• Semi-independent reward on a dedicated GPU slab (#134); opt-in diffusion role residency (#302, #428).

Algorithms

• CPPO (#58), AR DPPO (#85), GSPO (#178), AR PPO with GAE and a value head (#254, #256, #259).
• DiffusionOPD on-policy distillation with multi-teacher and video-KL support (#232, #432); teacher-anchored capability negotiation in ARTrainer (#457).
• DiffusionNFT under the EMA shadow adapter and with a reference-model KL penalty (#21, #411); Flow-DPPO beta KL (#112); WindowScheduler decay / exp_decay (#489).
• ReFL / DRaFT-K differentiable-reward training, now hosted under experimental/ (#120, #210, #281).

Rollout Engines & Weight Sync

• SGLang Diffusion v2 and vLLM-Omni v2 engines replace v1; a canonical SGLang LLM/VLM engine replaces sglang_llm (#35, #36, #74).
• New direct vLLM TP rollout engine with vLLM-native IPC weight sync (#480); FastVideo rollout engine for WAN 2.1 / 2.2 (#149, #415).
• SGLang TP/EP full-weight and LoRA sync, plus checkpoint-engine IPC sync (#226, #227, #225).
• vLLM-Omni prompt-embed cache, sleep lifecycle, and MoE workspace offload on sleep (#449, #370).

Training Backend & Performance

• Configurable HSDP, fsdp_mode=no_shard, copy-engine all-gather, and VeOmni expert parallelism and Ulysses sequence parallelism (#410, #321, #467, #158, #71).
• Token-budget packed micro-batches and DP sequence-length balancing (#42, #45, #56); packed-varlen replay for Qwen3 and Qwen-VL (#75, #76).
• Batched-step diffusion replay extended to Qwen-Image, Z-Image, and FLUX.2-Klein (#156); BAGEL prompt-KV caching and batched rollouts (#145, #203, #253).
• Lower memory: CPU-resident eager AdamW step across offload, released grads at the optimizer step, and a lower expert-gather peak (#441, #460, #422).
• Remote rewards overlap generation via micro-batched diffusion rollout (#445).

Rewards

• New scorers: EditReward, HPSv3++, VideoAlign, CLAP / ImageBind audio rewards, EditScore, and rank-affine managed image scorers (#28, #142, #97, #168, #348, #303).
• math-verify now runs in a bounded child process (#462).

Checkpointing, Eval & Observability

• Checkpoint save/load for all trainers, LoRA checkpoints, and DCP sharded checkpoints with async save and an export reader (#33, #66, #83, #157).
• Periodic eval for PE / ReFL / unified trainers and eval sampling decoupled from the RL rollout (#202, #346).
• Unified logger, per-phase timing, Perfetto train-step profiler, GPU memory monitoring, and a rollout-replay K3 metric (#24, #37, #160, #179, #187).

Notable Fixes

• DRPO: deterministic inference no longer clones every sample in a group, which zeroed the DRPO signal, and rollout throughput is restored (#208, #535).
• AR loss aggregation is mask-aware and grouping-invariant, and text loss masks are respected (#448, #409).
• Advantage normalization stays finite for invalid rewards (#335); diffusion replay keeps transition means at SDE-native fp32 (#444).
• Distributed rank errors fail fast instead of hanging, and cold AdamW state survives checkpoint export (#258, #440).

Upgrade Notes

• Dependency stacks moved: SGLang 0.5.12.post1 → 0.5.19; vLLM / vLLM-Omni 0.20.0 → 0.28.0; torch 2.11.0 → 2.13.0, now +cu130 for both engines; transformers >=4.41 → >=5.12,=0.30 → >=0.38. The SGLang wheel needs glibc ≥ 2.34. See INSTALL.md.
• Recipes moved: examples are grouped into per-model folders (#84), examples/sft/validation is flattened into examples/sft (#349), and...