Parcae: Doing more with fewer parameters using stable looped models

Parcae: Doing more with fewer parameters using stable looped models

⚡️ FlashAttention-4: up to 1.3× faster than cuDNN on NVIDIA Blackwell →

Introducing Together AI's new look →

🔎 ATLAS: runtime-learning accelerators delivering up to 4x faster LLM inference →

⚡ Together GPU Clusters: self-service NVIDIA GPUs, now generally available →

📦 Batch Inference API: Process billions of tokens at 50% lower cost for most models →

🪛 Fine-Tuning Platform Upgrades: Larger Models, Longer Contexts →

All blog posts

Research

Published 4/15/2026

Parcae: Doing more with fewer parameters using stable looped models

Authors

Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, Dan Fu

Table of contents

40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...

Links in this article

Paper Code Hugging Face

Summary

We present Parcae, one of the first stable architectures for looped language models , achieving the quality of a Transformer twice the size with clean, predictable training. Parcae creates a new medium to scale quality by increasing recurrence rather than purely scaling data, opening up an efficient frontier for training memory-constrained on-device models.

Getting the most out of your parameters Traditional scaling laws tell us that to achieve the best performance, we need to scale FLOPs, often with more parameters or data. But as models move to the edge and inference costs skyrocket, we wonder: Can we scale quality without inflating memory footprint? To that end, we’ve been exploring looped architectures, models that increase compute by passing activations through the same layers multiple times. While promising, these models have been unstable to train. We tackle this issue directly and introduce Parcae , a stable looped architecture that: Is better than prior looped models : Parcae achieves up to 6.3% lower validation perplexity than previous large-scale looped recipes. Punches above its weight : Our 770M Parcae matches the quality of a 1.3B parameter transformer trained on the same data, achieving the same performance with roughly half the parameters. Scales Predictably: We establish the first scaling laws for looping , finding that compute-optimal training requires increasing looping and data in tandem .

Looped models are cool, but hard to train in practice As models move to the edge and inference deployments take on larger portions of compute, there is an increasing interest in scaling model quality without increasing parameters. One mechanism we have been excited about is layer looping, where initial works have trained looped models that match the quality of larger fixed-depth architectures. To turn a vanilla Transformer into a looped model, we follow prior work and partition its layers into three functional blocks: a prelude ($\mathcal{P}$) , a recurrent ($\mathcal{R}$), and a coda ($\mathcal{C}$). The forward pass works in three stages: Embedding: The prelude transforms the input into a latent state $e$. Recurrence: The recurrent block iteratively updates a hidden state $h_t$ for $T$ loops. To maintain the input’s influence, $e$ is injected into each loop, typically via addition [1] ($h_{t+1} = \mathcal{R}(h_t + e)$) or concatenation with projection [2] ($h_{t+1} = \mathcal{R}(W[h_t; e])$). Output: The coda processes the final $h_T$ to generate the model’s output.

Unfortunately, looped models are a headache to train [2][3][4]. We personally found them to suffer from residual state explosion and loss spikes. What makes looped models even trickier is that the recurrent block is composed of several vanilla Transformer blocks, making it difficult to reason about the source of instability.

Understanding the instability of looping While instability is a fickle foe, we observed that a simple linear framework captured a significant source of instability. Specifically, we recast looping as a nonlinear time variant dynamical system over the residual, whose update rule is: $$h_{t+1} = \overline{A} h_t + \overline{B} e + \overline{\mathcal{R}}(h_t, e)$$ where $\overline{A}, \overline{B}$ perform injection and $\overline{\mathcal{R}}$ is the contribution of the Transformer blocks to the residual stream. For the subquadratic sequence mixing fanatics out there, observe that if we ignore the nonlinear term $\overline{\mathcal{R}}$, the resulting system is a discrete linear time-invariant (LTI) dynamical system over the residual state, across model depth. What's cool is that for discrete LTI systems, their stability and convergence are determined by the eigenvalues of $\overline{A}$. Specifically, stability is categorized using the spectral norm $\rho(\overline{A})$ (i.e., the absolute largest eigenvalue of $\overline{A}$), with stable systems (convergent) being $\rho(\overline{A})[2], a prior looped model, we tested against parameter- and data-matched RDMs, observing that Parcae reduces validation perplexity by up to 6.3%.

Params & Model Val. PPL (↓)

100M Scale

└ RDM 14.23

└ Parcae 13.59

350M Scale

└ RDM 10.76

└ Parcae 10.09

When retrofitting a very strong Transformer baseline into an RDM, without any hyperparameter tuning, we found Parcae to be robust over RDMs (which just flat-out diverged).

Params & Model Val. Loss (↓) Core (↑) Core-Extended (↑)

RDM Divergent Divergent Divergent

+ Parcae Constrained A 2.97 13.2 ± 0.2 9.1 ± 0.5

+ All Parcae Tricks 2.95 14.0 ± 0.2 9.7 ± 0.3

We also took Parcae and used it as a drop-in replacement for a standard fixed-depth Transformer. Using a nanochat-inspired setup, we train a series of language models on FineWeb-Edu, up to 1.3B parameters. We found that Parcae outperformed all parameter- and data-matched Transformers, with our 770M Parcae model almost achieving downstream quality equivalent to a Transformer twice its size!

Params & Model Val. PPL (↓) Core (↑) Core-Extended (↑)

140M Scale

└ Transformer 21.48 13.00 ± 0.15 8.80 ± 0.21

└ Parcae 19.06 14.04 ± 0.20 9.67 ± 0.28

370M Scale

└ Transformer 15.79 17.46 ± 0.03 11.71 ± 0.22

└ Parcae 14.49 20.00 ± 0.06 12.75 ± 0.31

770M Scale

└ Transformer 13.08 22.42 ± 0.20 14.20 ± 0.63

└ Parcae 12.49 25.07 ± 0.33 15.19 ± 0.43

1.3B Scale

└ Transformer 11.95 25.45 ± 0.08 15.90 ± 0.23

└ Parcae 11.42 28.44 ± 0.28 17.08 ± 0.09

To loop, or not to loop But is looping actually FLOP efficient? To study this, we explore a setting where, under a fixed parameter count and FLOP budget, we trade off mean recurrence in training…