Harel Lidar

Article summary · Oct 2026

Towards Looped Model Done Right

Mostly someone else’s words. I keep it as reading, not as my own writing.

The authors try to formalize and systematize the way we train, architect, and organize loop transformers. To do so they begin by defining the search space. They claim that the framework for looped models should be thought of as three separate choices that are often bundled together. These are:

  1. Where recurrence is placed
  2. How the input is reintroduced
  3. How recurrent states are organized And they explore each of these questions separately.

Notations

Let x∈VB×L\mathbf{x} \in \mathcal{V}^{B \times L} denote token indices and x0=Eθ(x)∈RB×L×d\mathbf{x}_0 = E_\theta(\mathbf{x}) \in \mathbb{R}^{B \times L \times d} their embeddings. A general tied iterative model can be written as

e=Pθ(x0).z0=ϕθ(e,ξ).z~t=Wθ(zt,e).zt+1=Rθ(z~t),t=0,…,T−1.h=Cθ(zT).\begin{aligned} \mathbf{e} &= P_\theta(\mathbf{x}_0). \\ \mathbf{z}_0 &= \phi_\theta(\mathbf{e}, \boldsymbol{\xi}). \\ \tilde{\mathbf{z}}_t &= W_\theta(\mathbf{z}_t, \mathbf{e}). \\ \mathbf{z}_{t+1} &= R_\theta(\tilde{\mathbf{z}}_t), \qquad t = 0, \dots, T-1. \\ \mathbf{h} &= C_\theta(\mathbf{z}_T). \end{aligned}

Here PθP_\theta is the prelude map [3], RθR_\theta is the tied iterative body, CθC_\theta is the coda map. All three are stacks of standard Transformer blocks; the prelude and coda are applied once, whereas the recurrent core is repeatedly applied with shared parameters. ϕθ\phi_\theta initializes the recurrent state, and WθW_\theta is the per-step input write. The fixed representation e\mathbf{e} is the prelude output, zt\mathbf{z}_t is the model’s recurrent hidden stream, and auxiliary randomness ξ\boldsymbol{\xi} is used only by variants with random state initialization.

Ouro vs Huginn

Under this notation, an Ouro-style topology sets Pθ,WθP_\theta, W_\theta, and CθC_\theta to identity maps, initializes z0=x0z_0=x_0, and applies the same stack for iterations. A Huginn-style topology keeps nontrivial prelude and coda maps and writes the fixed representation into the recurrent stream at each iteration

Results

Transfer to moe

Takeaways

We compare full-stack and middle-loop language models by disentangling three design choices that standard Ouro- and Huginn-style recipes vary simultaneously: the iteration envelope, input injection, and latent-state organization. Under matched parameter scale, logical depth, and training-token budget, we find that the prelude–loop–coda sandwich architecture improves performance on mathematical reasoning, context extraction, and other reasoning-intensive tasks. Input injection primarily benefits knowledge retrieval and context access, while contributing little to quantitative reasoning. Random state initialization yields mixed results, whereas the shared H/L hierarchy provides no consistent benefit.

Our MoE transfer study evaluates the two complete architectures in a modern sparse-model setting and shows that the advantages of the Huginn-style architecture persist. The Huginn MoE achieves more balanced expert utilization and approaches—and on some tasks surpasses—the performance of its feedforward MoE counterpart, despite being trained from scratch under the same data budget and matched memory and FLOP constraints.

More broadly, the architectural design space of looped models remains underexplored and calls for substantially more controlled ablation studies. Meaningful comparisons therefore require carefully designed, apples-to-apples evaluations that match parameter scale, training budget, inference cost, and evaluation protocol. Many important questions remain open, including how loop architectures interact with conditional sparsity and how architecture–system co-design can further advance their performance–efficiency Pareto frontier