Rise Of Hybrid Llms

Attention was never enough: Tracing the rise of hybrid LLMs | AI21

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

Transformers have been the standard architecture for large language models, but the self-attention mechanism imposes quadratic complexity—making inference computationally expensive. As context length grows, so does the KV-cache, overwhelming GPU memory. And because attention operates over fixed context, models often lose track of global information, forcing workarounds like chunking and retrieval.

In December 2023, Albert Gu and Tri Dao introduced Mamba—a selective state-space model designed to improve throughput and efficiency without relying on attention. With input-dependent parameters and content-aware computation, Mamba offered a new framework for scalable sequence modeling.

This piece traces the architectural shifts shaping the rise of hybrid LLMs. Each entry marks a move away from the Transformer default—and a step toward what comes next.

Mamba (Gu & Dao, December 2023)

Paper Authors: Albert Gu, Tri Dao

The original Mamba paper introduced a selective state-space model (SSM) that achieved linear-time inference and 5× throughput gains over Transformers. By making core parameters input-dependent, the model enabled selective, content-aware processing of sequences—without relying on attention or MLP blocks. The architecture combined efficiency with generalization, showing strong results across language, audio, and genomics. This opened the door for a new class of models—simple, fast, and expressive—ready to power foundation models beyond the Transformer era.

Jamba (AI21 Labs, March 2024)

Paper Authors: Opher Lieber, Barak Lenz

Jamba introduced the first large-scale hybrid Transformer–Mamba–MoE language model. Its architecture interleaved attention and Mamba layers at a 1:7 ratio, with MoE layers added every two blocks—balancing long-context reasoning with efficient computation. The model supported context lengths up to 256K tokens while maintaining a compact memory footprint and high throughput. The team demonstrated that hybrid design choices—down to layer ordering and MoE sparsity—directly impact model quality and deployability. Jamba set a precedent for hybrid models built to scale, releasing both weights and architecture under Apache 2.0.

MambaVision (NVIDIA, July 2024)

Paper Authors: Ali Hatamizadeh, Jan Kautz

MambaVision reimagined Mamba for computer vision, combining it with Transformer blocks in a hierarchical layout. Early layers used convolutional blocks for efficient feature extraction; later stages added self-attention to recover long-range spatial dependencies. The result was a Pareto-optimal backbone achieving state-of-the-art accuracy and image throughput on ImageNet-1K. The team showed that Mamba’s sequential strengths could complement ViT-style architectures without incurring latency costs. MambaVision stood out for its architectural pragmatism—adapting recurrence for spatial reasoning at scale.

Codestral Mamba (Mistral, July 2024)

Blog Authors: Mistral AI team

Codestral Mamba marks Mistral’s first pure Mamba2-based code model, trained with guidance from Mamba’s original authors. Unlike Transformers, Mamba offers linear-time inference and the capacity to model sequences of theoretically infinite length—capabilities that suit large-context code workflows. The 7.3B parameter model supports up to 256K context tokens and is optimized for retrieval and code reasoning.

Jamba 1.5 (AI21 Labs, August 2024)

Paper Authors : Barak Lenz and Opher Lieber

AI21 Labs scaled its hybrid Mamba–Transformer architecture to 398B total parameters with 94B active—marking the first large-scale deployment of such a model. Jamba 1.5 interleaves Mamba and attention layers across 72 layers, using grouped-query attention, low-rank adaptation, and 16 MoE experts for efficient routing. It supports 256K-token context and achieves top scores on long-context benchmarks, including state-of-the-art performance on NVIDIA’s RULER . Designed for real-world enterprise applications, Jamba 1.5 balances compute efficiency with generalization, showing that hybrid architectures can scale without compromising inference speed or accuracy.

Mamba-Llama (Together AI, August 2024)

Paper Authors: Dmitriy Boylov, Ansh Radhakrishnan, Constantine Caramanis, Alexander M. Rush, Tri Dao

Together AI distilled LLaMA-3-8B-Instruct into a hybrid model by replacing 75% of its attention layers with Mamba blocks. The resulting architecture reduced inference latency by up to 5× while preserving chat performance across human evaluations and MMLU/ARC benchmarks. The team used exact weight mapping to retain instruction tuning and applied iterative distillation to recover quality after architectural modification. The model supports 128K context and is open-sourced under Apache 2.0. The main contribution of this work is showing that Transformer models can be “converted” into more efficient Mamba variants with relatively little compute.

Falcon Mamba (Technology Innovation Institute, July 2025)

Paper Authors: Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, Hakim Hacid

Falcon Mamba 7B is a pure Mamba-based language model trained on 5.8 trillion tokens with 7.27B parameters. It uses a state-space model architecture without any attention layers and supports arbitrary sequence lengths during inference. The model outperforms LLaMA3.1-8B, Mistral 7B, Falcon2-11B, and several hybrid models on benchmarks such as MMLU, GSM8K, ARC, and MuSR. It maintains constant memory and throughput across long sequences.

Nemotron-H (NVIDIA, April 2025)

Paper Authors : NVIDIA team

NVIDIA introduced Nemotron-H, a family of hybrid Mamba–Transformer models (8B, 47B, and 56B) optimized for inference-time scaling and speed. By replacing 92% of attention layers with Mamba2 blocks, Nemotron-H models deliver up to 3× faster throughput than similarly sized Transformers like LLaMA-3.1 and Qwen-2.5, while matching or exceeding accuracy on MMLU, GSM8K, HumanEval, and MATH. The 56B model was trained using an FP8-based recipe and compressed to 47B using MiniPuzzle, NVIDIA’s new distillation framework. All weights are open-sourced via Hugging Face and NeMo. Nemotron-H affirms that hybrid architectures can outperform traditional Transformers at scale—without compromising reasoning performance.

Bamba (IBM, April 2025)...