amazon/GDN-primed-HQwen3-8B-Reasoner

GDN-primed-HQwen3-8B-Reasoner

GDN-primed-HQwen3-8B-Reasoner is a Hybrid language model consisting of 50% Attention layers and 50% Gated DeltaNet (GDN) layers, primed from Qwen3-8B using the Hybrid Model Factory Priming pipeline. The model is trained for long-context reasoning and supports context lengths of 128K tokens.

GDN is a State-Space Model layer with constant memory and linear compute cost in the sequence length.

By combining Attention with GDN, our Hybrid model achieves up to 2× faster inference at long contexts while closely matching the base Transformer's quality.

Links

• 📄 Gated DeltaNet paper (ICLR 2025)
• 💻 GitHub: Hybrid Model Factory

Why Hybrid?

Each Primed Hybrid model is initialized from a base Transformer by converting a portion of its Attention layers into State-Space Model (SSM) layers that maintain a fixed-size recurrent state instead of a growing KV cache. At a 50% Hybrid ratio, roughly half the KV cache (which grows linearly with sequence length) is replaced with fixed-size SSM state. The practical benefits:

• Higher throughput at long contexts — less memory on KV cache means more memory for batching
• More concurrent sequences — ~2× as many concurrent sequences before hitting memory limits
• Growing advantage with context length — at long contexts, Attention dominates the forward pass while SSM layers remain negligible in cost. Since the Hybrid model makes roughly half as many Attention calls as the base Transformer, the throughput advantage grows with context length

Increasing hybridization ratio, replacing more Attention layers with SSM layers, further reduces memory and increases throughput, typically at the expense of performance.

Model Overview

• Type: Causal Language Model (Hybrid Attention + SSM)
• Base Model: Qwen3-8B
• Hybrid Layer Type: Gated DeltaNet (GDN)
• Hybrid Ratio: 50% (18 Attention + 18 GDN layers)
• Parameters: ~8B
• Context Length: 128K natively
• Precision: bfloat16
• License: Apache 2.0

Benchmark Results

We consider the following Transformer as a baseline:

1. Qwen3-8B (thinking, from HF): The original Qwen model evaluated in thinking mode, which is the intended mode for reasoning tasks. This serves as the base Transformer from which we start the Priming procedure.

Reasoning Benchmarks

Evaluations on math reasoning (AIME24/25), science (GPQA), coding (LiveCodeBenchv5, Scicode), tool-calling (BFCLv3/v4), and instruction-following (IFBench). Evaluations are done using the Nemo Evaluator SDK. We have provided the evaluation configuration examples/evaluation/nemo_reasoning_evals.yaml for reproducibility. Evaluations are done at 64K generation length.

| Model | AIME24 | AIME25 | GPQA | LiveCodeBench-v5 | BFCLv4 (minus web-search) | BFCLv3 | IFBench | SciCode | Average | |-------------------------------|------|-----------|--------|-----------|----------|------|----|-----|-----| | Qwen3-8B (thinking, from HF) | 78.67 | 71.0 | 57.77 | 57.94 | 68.30 | 66.46 | 31.60 | 10.63 | 55.29 | | GKA-primed-HQwen3-8B-Reasoner | 82.00 | 73.67 | 61.81 | 63.10 | 66.47 | 62.20 | 38.96 | 6.41 | 56.82 | | GDN-primed-HQwen3-8B-Reasoner | 82.00 | 73.33 | 61.49 | 62.94 | 63.27 | 57.44 | 37.80 | 2.50 | 55.10 |

*For BFCLv4, we remove the web-search subtask and weight each task by the number of entries (test examples) for that task.*

How close are the Hybrid models to the Transformer baseline on complex reasoning tasks? Our Primed Hybrid models are competitive with the Qwen3-8B (thinking, from HF) model despite [ [!TIP] > The --mamba-cache-dtype float32 and --mamba-ssm-cache-dtype float32 flags are important for accurate long-context generation. See the Inference guide for details on all recommended flags.

> [!TIP] > Similarly to NVIDIA-Nemotron-3-Nano-30B-A3B-BF16, for generic reasoning tasks (e.g. Math, Science) we recommend setting temperature=1.0 and top_p=1.0. For tool-calling we recommend temperature=0.6, top_p=0.95.

Thinking Versus Non-thinking Setting

Our reasoning model supports thinking on/off modes. Whenever thinking mode is on, the model will reason for multiple tokens in a segment delimited by ` and ` (which is extracted by the reasoning parser) before producing a response. This is necessary for difficult queries and increases response quality at the expense of higher latency. Thinking mode is enabled by default, however thinking mode can be turned off via the chat template.

If you want to query the model with thinking mode *off*, query the model as follows:

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "amazon/GDN-primed-HQwen3-8B-Reasoner",
"messages": [
{"role": "user", "content": "What is Linear Attention in the context of LLMs?"}
],
"chat_template_kwargs": {"enable_thinking": false}
}'

With Hugging Face Transformers

> [!WARNING] > Due to the long generations produced by reasoning models, the lower latency provided by vLLM is preferred over Hugging Face for evaluations and in production settings. We recommend Hugging Face generation primarily for quick debugging or testing.

from transformers import AutoModelForCausalLM, AutoTokenizer
import hmf.model.hybrid_zoo.models.model_register # Register Hybrid models

model = AutoModelForCausalLM.from_pretrained(
"amazon/GDN-primed-HQwen3-8B-Reasoner", trust_remote_code=True
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("amazon/GDN-primed-HQwen3-8B-Reasoner")

messages = [{"role": "user", "content": "What is linear attention in the context of LLMs?"}]
prompt = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True, enable_thinking=True
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=65536, temperature=1.0, top_p=1.0)
print(tokenizer.decode(outputs[0],…