Attention and model architecture
GOAT adds trainable priors to attention
The paper views standard attention as an Entropic Optimal Transport problem with an implicit uniform prior. It introduces GOAT, which uses a learnable continuous prior and remains compatible with optimized kernels such as FlashAttention.
Engineers can evaluate a way to add learnable attention priors while retaining compatibility with optimized kernels.
Looped Diffusion Transformer Reuses Blocks During Denoising
The paper describes a text-to-image model that reuses selected Transformer blocks within each denoising step. The post reports that a 260M model outperforms models up to 6.5× larger, and that extra loops help more than extra denoising steps at equal compute.
Block reuse offers a way to improve predictions without adding unique parameters at inference time.
LoopCD Improves Decoding in Looped Transformers
The paper presents LoopCD, which uses the difference between early and final recurrent-state predictions to refine decoding without extra training or recurrent iterations. It reports higher AIME 2024 accuracy on Ouro-2.6B-Thinking and comparable or better results using half the loops.
Engineers working on recurrent Transformers can assess a decoding method that may improve accuracy while reducing forward FLOPs.
LoopCD Uses Earlier Loops for Contrastive Decoding
The paper introduces LoopCD, a training-free contrastive decoding framework for Looped Transformers. It uses earlier loop representations, which provide aligned weaker predictions, to guide token selection.
Engineers working on recurrent Transformer inference can assess a decoding method that uses intermediate loop states without additional training.
When Transformers Need Chain of Thought for Serial Tasks
The paper studies whether chain-of-thought content does computational work or is largely decorative. The post says removing it lowers accuracy on serial tasks to a shallow-circuit ceiling.
It examines how chain-of-thought affects the computational limits of transformer reasoning.
Jagged Flash Attention with TLX on NVIDIA Blackwell
A PyTorch blog describes Jagged Flash Attention, the attention kernel behind Meta’s Generative Ads Model, built with TLX for NVIDIA Blackwell. The post reports better performance than FlashAttention-4 on GEM’s jagged shapes.
It offers engineers a concrete example of hardware-aware Triton extensions for optimizing attention kernels.
Looped-DiT Reuses Transformer Blocks During Denoising
Looped-DiT repeatedly runs shared Transformer blocks at each denoising step. The post reports that a 260M model outperformed a model 6.5 times larger on text-to-image benchmarks with 4.9 times lower inference compute.
Engineers can assess block reuse as an alternative to scaling model size or denoising steps.
- 1 month ago @che_shr_cat
Weight Tying and Gradient Flow in Attention
The post argues that factorizing an attention matrix as S = WWᵀ changes gradient flow and removes an information-exponent bottleneck that can stall unfactorized models.
The claim highlights how attention parameterization may affect optimization, not just model size.
DepthBench evaluates residual connections and computational depth
DepthBench examines whether methods including LayerNorm Scaling, mHC, and AttnRes help deeper Transformers use their layers effectively. The preview notes that it remains unclear whether these methods increase effective computational depth.
It offers a benchmark for comparing approaches to information flow and depth utilization in Transformers.
BF16 Rounding and Gradient Instability in FlashAttention
The authors report that BF16 rounding in FlashAttention breaks a softmax-gradient symmetry and can cause gradients to blow up late in training. They propose GProj as a fix.
This matters to engineers training models with fused attention kernels in BF16.
Attention to Mamba: Cross-Architecture Distillation
The paper studies how to use pretrained Transformers to train State Space Models such as Mamba. Its abstract notes SSMs’ reduced memory consumption and higher generation throughput compared with attention-based models.
It addresses how to transfer existing Transformer training knowledge to SSMs.
KITE expands Transformers while keeping KV computation invariant
KITE is a scaling paradigm that combines scaling with upcycling while keeping KV computation invariant. It expands decoder-side components that reuse a KV cache generated by a smaller prefiller.
Engineers can assess an approach aimed at scaling model quality without proportionally increasing prefill cost.
RESA uses residual estimation with sparse attention
The RESA paper proposes using residual estimation to recover context ignored by sparse attention. The post says it adds no compute overhead.
It describes an approach to retaining context when sparse attention selects only part of the KV cache.
OPAL Trains Linear-Attention Models on Their Own Outputs
On-Policy Attention Linearization trains distilled linear-attention models on their own long-context outputs under teacher supervision. The post reports full-attention retrieval recovery and 83–93% of math reasoning performance with 3B tokens.
It describes a training approach for reducing attention memory costs while retaining retrieval and reasoning performance.
PTXBench evaluates AI-generated GPU kernels
PTXBench tests models writing matrix multiplication and attention kernels in architecture-specific PTX on H100 and B200. It measures correctness, speed, and whether requested instructions actually execute.
It offers engineers a way to assess where AI-generated low-level GPU kernels fall short of expert libraries.
Kernel Design Agents Optimize Kimi Delta Attention
An NVIDIA blog describes using Kernel Design Agents to generate and optimize Kimi Delta Attention kernels. It reports up to 2.96× faster performance than FlashKDA on B300, with one-tenth the state error.
The post describes agent-driven kernel optimization and diagnostics across CUDA and domain-specific kernel languages.
Matryoshka Attribution Ranks Components by Behavioral Influence
Matryoshka Attribution learns a nested ranking of neural representations or fine-tuning weight changes by optimizing behavior preserved under soft interventions. The post reports that reverting 1% of weights in Llama 3.1 8B removes most refusal behavior while largely preserving capabilities.
It offers a way to localize behavioral circuits and identify influential fine-tuning weight changes.
Block-Sparse Attention with Log-Linear Block Selection
The paper introduces a hierarchical method that scores large context regions with LogSumExp, then recursively narrows selection to promising regions. It reduces block selection from O(N²) to O(N log N).
It targets the cost of selecting blocks in sparse attention for long-context models.
HySparse2 uses two-level KV sharing for sparse attention
HySparse2 is a hybrid sparse attention method with two-level KV sharing. It targets efficient prefill and compact KV-cache storage for long-horizon, multi-turn agents.
Engineers working on long-context models can assess an attention design aimed at reducing prefill and KV-cache demands.
- 1 month ago @Jo1uck
Memory Attention Reuses Keys in the Value Expression
A post describes preliminary ablation results suggesting that reusing keys in V=K+M is an effective architectural choice. The author says it still needs validation at larger scales.
The result may inform attention architecture experiments, but the post notes that larger-scale validation is still needed.
arxiv.org
