Attention and model architecture
Transformers from Scratch: a tutorial by Peter Bloem
A blog post by Peter Bloem that explains transformers from the ground up. The post is shared as a learning resource on the transformer architecture.
A self-contained walkthrough of transformer internals, useful for engineers who want to understand attention from first principles.
Depth Hybrid Attention: depth-wise value mixing for hybrid architectures
The post introduces Depth Hybrid Attention (DHA), which adds depth-wise value mixing to hybrid attention models with little extra compute or cache. The author claims DWA and DDA variants beat full-attention and hybrid baselines while keeping a low KV-cache footprint.
It proposes a cheap way for layers to mix information across depth in hybrid attention models, with code provided.
- 11 days ago @ZyphraAI
Zyphra introduces Zyphra Hierarchical Attention for subquadratic attention
Zyphra announces ZHA, an attention method that searches a learned tree of compressed summaries. The author projects speedups of 35× over dense attention and 5× over MSA at 10M tokens.
It proposes a subquadratic attention design, a direction engineers track for long-context model efficiency.
- 11 days ago @industriaalist
Data-constrained scaling laws for underparameterized LLMs
The author claims LLMs are far underparameterized relative to the brain's synapse count and token exposure, and presents data-constrained scaling laws and learning algorithms for very large transformers. No link is included in the post.
It challenges Chinchilla-style compute-optimal assumptions and proposes scaling parameters versus data differently.
- 11 days ago @jcrwhittington
Latent Sparse Attention: subquadratic attention research release
Open research release of Latent Sparse Attention (LSA), which the author says matches full-attention accuracy at ultra-long context while speeding up attention up to 22.7x for sparse and 72.4x for full.
Shows where sparse attention time goes at 10M tokens and reports speedups for long-context attention.
- 13 days ago @divagr1925
QwenPU: a small LLM running inside Qwen 3.5-4B's weights
The author describes QwenPU, a stories260k LLM running on an NPU compiled into Qwen 3.5-4B's weights. A quoted post describes Qwenputer, a RISC-V CPU embedded in the same model's weights, using Gated DeltaNet layers as RAM.
Shows how computation can be compiled into LLM weights, relevant to Gated DeltaNet and model internals.
- 15 days ago @IntuitMachine
Sharing KV-caches between AI agents instead of text summaries
A thread describing research, attributed to MIT CSAIL and IBM, where agents pass raw KV-cache memory to each other instead of English text. The author claims 40% higher accuracy and 4x speed.
It proposes a cache-to-cache alternative to text handoffs in multi-agent pipelines, which matters for inference cost and latency.
- 16 days ago @_albertgu
Hierarchical byte-level networks (H-Nets) for end-to-end language modeling
The author announces a new version of H-Nets, hierarchical networks that learn end-to-end from raw bytes. The post claims 3-5x efficiency gains in scaling experiments and matching an open-source model trained on 11T tokens with about 10x less data and compute.
It proposes a tokenizer-free, byte-level architecture, a direction relevant to how future models handle raw data.
- 16 days ago @divagr1925
Qwenputer: RISC-V CPU embedded in Qwen3.5-4B weights
Author @divagr1925 describes Qwenputer, a Qwen3.5-4B model said to hold a RISC-V CPU in its weights, using Gated DeltaNet layers as RAM to run C and MicroPython during forward passes.
Shows Gated DeltaNet state used as general memory; the author's benchmark claims are unverified.
Qwen3.8-27B release with all layers converted to Gated DeltaNet
Qwen3.8-27B release with all 64 layers converted to Gated DeltaNet linear attention, the 16 former full-attention layers recovered via per-layer distillation and LoRA r32. English and Japanese.
Shows a concrete recipe for converting a pretrained full-attention transformer into linear attention via distillation.
- 22 days ago @thesupermannx
Meta paper proposes byte-level models without tokenizers
A post says Meta's paper 'Breaking the Token Ceiling' trains models on raw bytes (a 256-value vocabulary) and distills them, claiming 1B byte models match or beat Llama 3.2-1B and Gemma-3-1B.
Byte-level modeling could remove the tokenizer and shrink the vocabulary, but the claims come from a post with no link.
μLattice: Memory-scaling alternative to attention, per its author
Open-source project and paper by @Ex0byt describing μLattice, a memory design said to grow with input without quadratic cost. The author claims fewer operations than attention and perfect recall of thousands of associations.
Relevant to anyone weighing attention alternatives; the performance claims are unverified and come from the author.
LiFT: Loop Flow Transformers for Flow-Matching Image Generation
LiFT trains a flow-matching image generator with a shared transformer core so each loop progressively refines the initial velocity estimate. A 2-loop model can improve with up to 16 loops at inference, per the author.
Shows a way to trade inference loops for quality without retraining, relevant to compute-efficient diffusion transformers.
LiFT uses a shared DiT core for iterative inference
LiFT repeatedly applies a shared Diffusion Transformer core along a continuous depth path. On ImageNet 256×256, it beats a dense DiT baseline with about 60% fewer parameters, according to the post.
The approach offers a way to scale inference computation without retraining the model.
- 1 month ago @industriaalist
Dust Uses Activation Perturbations for Zeroth-Order Transformer Training
The post describes Dust, a zeroth-order method that uses parallel activation perturbations as a “virtual population” to pretrain transformers. It reports scaling experiments and comparisons with backpropagation and an evolution-strategies method.
It raises a practical alternative to backpropagation and reports how zeroth-order training behaves as model size and compute increase.
- 1 month ago @hyundongleee
Switching Linear Attention switches among multiple linear maps
The post describes Switching Linear Attention (SwiLA), a COLM 2026 paper that switches among multiple linear maps. It contrasts fixed-state linear attention with softmax attention's sequence-length-dependent KV cache.
It highlights an architecture trade-off between attention expressiveness and the memory cost of a growing KV cache.
Shared Memory in Looped Transformers Cuts Context Memory
The paper studies shared memory across recursions in looped Transformers. The post reports 76–79% lower context memory and improved quality over standard Transformers at 150M–1B parameters.
Engineers working on model architecture can assess a way to reduce the memory cost of looped Transformers.
A paper introduces Native Hybrid Attention
The post points to a paper introducing Native Hybrid Attention (NHA), which it describes as addressing the trade-off between Transformers’ speed on long contexts and linear attention’s ability to retain precise details.
Engineers evaluating attention architectures can examine the proposed approach to this speed-and-detail trade-off.
How attention and residual sinks rescale transformer components
The paper investigates attention sinks and residual sinks in large language models. It hypothesizes that these outliers, together with softmax attention and RMSNorm, rescale other components.
The analysis offers a functional account of emergent outliers and their role in transformer training.
BF16 FlashAttention-3 gradients can grow late in training
The paper reports that a 450M-parameter transformer trained with FlashAttention-3 in BF16 had a 1,000× gradient-norm increase after 25B tokens and ended with higher loss than FP32 attention, without NaNs. Recomputing two layers’ attention backward pass in FP32 removed almost all of the issue.
Engineers using fused BF16 attention can assess a reported late-training stability issue and a mitigation.
arxiv.org