Princeton’s Recurrent Looped Transformer Runs 96 Blocks Per Token

Princeton researcher Yifan Zhang published a technical report in September 2025 proposing the Recurrent Looped Transformer (RLT), an architecture that carries a decoder's full hidden state, including its layerwise sliding-window attention cache, across every token in both the prompt and response. Standard decoder-only LLMs never pass the last layer's output to the first layer of the next token. RLT closes that loop. The reference configuration runs 96 logical blocks per token and builds a state path that grows with sequence length. No empirical results have been reported yet.

What happened

Detail	Value
Researcher	Yifan Zhang, Princeton
Proposal	Recurrent Looped Transformer (RLT)
Reference config	48 encoder layers + 48 decoder layers (tied weights)
Logical blocks per token	96
State path after t tokens	48t decoder blocks
Measured results	None reported

In a standard decoder-only language model, the computation at the last layer of token t has no direct path to the first layer of token t+1. Tokens communicate only through attention over cached keys and values. Zhang’s RLT proposal closes that gap by carrying two things forward at every token boundary: the decoder’s final hidden state and its complete layerwise sliding-window attention (SWA) cache. There is no reset between the prompt and the response.

The architecture pairs a causal encoder with a recurrent decoder. The encoder processes tokens in parallel under a causal mask and produces representations that are projected into a key-value memory. That memory can either be shared across all decoder layers (G = 1) or kept layer-specific (G = L_D). The decoder then does the actual recurrence.

How the decoder loop works

At each token, a gated merge combines the encoder’s output for the current token with the previous decoder output. Each decoder block then runs three operations in sequence: causal SWA over decoder activations, cross-attention to the encoder memory, and a feed-forward network. The next-token probability distribution is read from the final decoder state. Before the first token, the model initializes with a learned start state and an empty cache.

The reference configuration ties the weights between the 48 encoder layers and 48 decoder layers. Because decoder blocks add cross-attention, per-block compute is not equal between encoder and decoder, so the “96 blocks” figure describes logical depth, not identical FLOPs. Zhang describes this as parameter reuse, not activation copying.

Why it matters

The structural appeal is that depth grows with the sequence for free. After processing t tokens, the state path from the initial state traverses 48t decoder blocks. Per-token compute stays fixed, but the effective reasoning depth expands as the sequence gets longer. That is a different trade-off than simply adding more layers at training time.

The report is careful to note the limits of this claim. Gating and gradient contraction can suppress the influence of earlier states, so structural depth is not a reasoning guarantee. No experiments have been run to show whether the long-range paths actually carry useful information.

Hardware and training implications

On the hardware side, the encoder and memory projections for known tokens can run with token-parallel kernels. The decoder transitions are sequential within a single sequence, but independent sequences can share one batched kernel. Zhang explicitly states that no parallel scan is assumed for the nonlinear decoder and no reduced-prefill speedup is claimed. Batching, kernel fusion, and checkpointing are listed as targets, not finished implementations.

For reinforcement learning training, the architecture requires rebuilding all encoder memory, recurrent outputs, and SWA caches from the sequence start under current parameters before scoring each action. Old rollout states are never reused. The sampler records each action’s log-probability under its actual sampling distribution, including temperature and truncation, so those recorded probabilities serve as ratio denominators during the RL update. A formal proposition in the paper states that moving the prompt-response split leaves the conditional distribution unchanged for a fixed token history.

Multi-turn serving

Serving across multiple turns requires saving an exact prefix snapshot that includes encoder cache and memory, the complete decoder state, position metadata, the window convention, and model version. Because the state is independent of the serving split, a fixed-weight snapshot can be reused across requests. Weight updates invalidate old snapshots, and editing a past prefix forces recomputation from an earlier checkpoint.

How it relates to prior work

Zhang situates RLT against several existing approaches. The encoder-derived memory follows patterns from YOCO and DeepSeek-V4.1-Flash, both of which project key-value memory from encoder states, though RLT does not skip decoder processing across the prompt. The temporal feedback mechanism builds on Feedback Transformer and Recurrent Transformer work, but RLT feeds the previous final decoder output into the next decoder input and runs recurrence over the prompt as well as the response. The depth-wise reuse connects to Universal Transformers and recurrent-depth latent reasoning research.

Our take

This is a thoughtful architecture proposal from a credible source, and the paper earns points for being unusually honest about what it has not done. Most research announcements bury the “no results yet” disclosure. Zhang leads with it. That matters for anyone trying to decide whether this is worth tracking.

The core idea, feeding the final decoder state back into the next token’s input, is not new in concept, but the specific formulation around SWA cache carryover and the RL replay contract is genuinely detailed. The multi-turn serving snapshot specification alone is more practical than most architecture papers bother with.

The gap between “structurally unbounded depth” and “actually useful long-range reasoning” is real and acknowledged. Until someone trains this at scale and measures whether those 48t-deep state paths actually help on hard benchmarks, RLT is a well-specified hypothesis. Watch for follow-up experiments, especially on tasks where state accumulation across long prompts matters, like multi-step tool use or long-document reasoning. If you are tracking the frontier of AI model architecture for business applications, our AI news coverage covers these developments as they land. For teams already exploring how to integrate emerging model capabilities into products, AI integration work often turns on exactly these architectural trade-offs around memory and state.

What to do about it

1. Bookmark the technical report if you work on LLM infrastructure or fine-tuning pipelines. The RL replay and multi-turn serving sections contain practical design notes regardless of whether RLT itself ships.
2. Do not adjust vendor or model selection based on this paper. There are no measured results to act on yet.
3. Watch for follow-up work from Zhang or Princeton on training runs, especially any benchmarks on long-context reasoning tasks where state accumulation would show the clearest signal.

The honest bottom line: RLT is a serious design document, not a product. Come back when the benchmarks exist.

Source: Marktechpost