News
- Training and fine-tuningArticle
How to Scale Your Model: a free guide to LLM scaling on TPUs and GPUs
An online book that explains how TPUs and GPUs work, how LLMs run on real hardware, and how to parallelize training and inference at large scale. The linked page presents it as a demystification of LLM scaling.
Covers the hardware and parallelism math behind training and serving LLMs, useful for estimating training cost and memory.
- LLM inferencePost1 day ago @casper_hansen_
DeepSeek V4.1 Flash on one RTX 3090 with vLLM and ExLlamaV3
The author reports running DeepSeek V4.1 Flash on one 3090 with 64GB RAM at about 20 tokens/s, claiming no quality loss. The stack pairs a vLLM frontend for scheduling, tool calling and OpenAI/Anthropic APIs with ExLlamaV3 for the forward pass.
Shows a concrete consumer-GPU serving stack split between a scheduler frontend and a swappable forward pass.
- LLM inferencePost1 day ago @TeksEdge
Kinetic: Rust inference engine that processes prompts before Enter
Open-source Rust inference engine (Apache 2.0) that the post says starts processing a pasted prompt before Enter is pressed. Reported RTX 3060 laptop test: 0.13s wait vs 10.95s for llama.cpp.
Shows prefill and prompt-reuse ideas applied to local inference latency; the benchmarks are the author's reported numbers.
- CUDA and GPU programmingArticle
Modal's GPU Glossary: terms for GPU computing
A glossary of GPU-related terms published by Modal. The post shares it as a GPU Glossary reference.
A quick reference for GPU terminology that engineers can use when reading CUDA docs or profiler output.
- 3 days ago @SciTechera
Stanford memory system uses 16 tokens to recall conversation history
A post reports a Stanford memory system that recalls full conversation histories with 16 memory tokens. The author claims up to 4,212x fewer document tokens, 27x speed in a 512-document test, and wins on 5 of 8 benchmarks.
Offers an alternative to retrieving document tokens for long-context memory, but no primary source is linked.
- C and C++Article
Curated index of modern C++ articles on Modernes C++ blog
A table of contents page for the Modernes C++ blog, collecting its articles on modern C++ topics. The author of the post recommends saving it as a structured collection of C++ articles.
A structured index of modern C++ articles gives engineers a single place to find topic-by-topic reading.
- 4 days ago @MaximeRivest
Author argues fine-tuning may beat prompt optimization for cost and speed
A post arguing that fine-tuning model weights may be faster, cheaper and easier than prompt optimization such as GEPA. It suggests fine-tuning a small model with Tinker, starting from a paper and a dataset.
Offers a concrete cost and effort case for fine-tuning over prompt optimization.
- CUDA and GPU programmingArticle
Deep Learning Compilers: Architecture, Optimization, and Empirical Characterization
A paper on deep learning compilers, covering their architecture, optimization techniques, and empirical characterization. The linked hgpu.org page describes compilers as key to efficiency on heterogeneous hardware.
Useful for engineers who want a structured view of how DL compilers optimize models across heterogeneous hardware, including GPUs.
- LLM inferenceArticle
Making Deep Learning Go Brrrr From First Principles
Horace He's essay explains how to reason about deep learning performance from first principles, focusing on compute, memory bandwidth, and overhead. The item shares it as a recommended read.
It gives a framework for judging whether a model's bottleneck is compute, memory, or overhead, which matters for LLM inference costs.
- CUDA and GPU programmingArticle
WebGPU Fundamentals: introductory learning site
A post links to WebGPU Fundamentals, a site for learning the WebGPU API. The page preview describes it only as a place to learn WebGPU.
WebGPU is the browser's GPU API, useful for engineers exploring GPU programming beyond CUDA.
- LLM inferencePost6 days ago @TeksEdge
Running a 284B DeepSeek model in 64GB RAM on CPU via llama.cpp fork
A developer-modified llama.cpp fork, llama-modchip, reportedly runs DeepSeek-V4-Flash (284B total, ~13B active) in 64GB DDR5 RAM with no GPU, using low-bit quantization and custom AVX2/AVX-512 kernels.
Shows CPU-only inference of a large MoE model, with quantization and kernel gains; numbers are developer-reported.
- 6 days ago @aaronklaw
Fine-tuning a legal LLM with LoRA and the overfitting lesson from a hold-out set
The author describes learning to fine-tune Llama for a legal task with LoRA using Unsloth notebooks. The model memorized the training data and performed worse than the base model on unseen examples, so they stress validating on a hold-out set.
A concrete reminder that a strong training loss doesn't show generalization, so evaluate fine-tunes on held-out data.
- LLM inferenceArticle
NVIDIA: LLM agents can skip long context via frequent compaction
NVIDIA study finds agents can finish long-horizon tasks by periodically compacting state instead of keeping full history, reporting up to 8.8x lower estimated inference cost across six benchmarks.
Offers a concrete way to cut per-token inference cost in agent systems by shrinking the active context.
PTX ISA 9.5 adds new async copy, fence, and locality instructions
A post highlights new PTX 9.5 features, including narrower async copy waits, aliased-read release fences, locality domain queries, and timestamped trace markers. It links to the PTX ISA 9.5 documentation.
CUDA kernel authors targeting new GPUs need to know these new memory ordering and scheduling primitives.
An Even Easier Introduction to CUDA, by Mark Harris
A step-by-step NVIDIA Technical Blog post introducing CUDA C++ programming for GPUs, using a simple parallel example. The post is an updated version of an earlier introduction.
A short, hands-on starting point for engineers new to writing CUDA C++ kernels.
- AI agentsProject
Running isolated LM experiments with Codex CLI config to disable tool-calling
A post advising how to run isolated experiments with local models via Codex CLI, using config files that disable tool-calling so results stay shareable. It links to a config file in the lm_understand-logs repository.
Shows a reproducible harness setup for evaluating local models without tool access confounding results.
- LLM inferencePost8 days ago @oliviscusAI
tiny-llm: open-source course on building a vLLM-style inference engine
An open-source four-week course where you build a small vLLM-style inference engine around Qwen3, covering attention, KV cache, 4-bit weights, C++/Metal kernels and batching. It runs on Apple Silicon with MLX.
Hands-on, measured practice with KV cache, quantized weights and batching, runnable on a laptop without an NVIDIA GPU.
- CUDA and GPU programmingArticle
Formalizing PTX in Lean to Prove GPU Kernel Correctness
A write-up by the Koska team on formalizing NVIDIA's PTX instruction set in the Lean prover to formally verify GPU kernels. The author says the goal is confidence in agent-written kernels without reading them.
Shows a route to machine-checked correctness for low-level GPU kernels, relevant to verifying generated code.
- 9 days ago @BenKoska
Formalizing PTX in Lean to prove GPU kernels correct
The author says they formalized PTX, NVIDIA's GPU intermediate assembly, in Lean so GPU kernels can be formally proven correct. The post has no linked source yet.
Formal verification of PTX could let teams trust machine-written GPU kernels without manual code review.
- 9 days ago @VikParuchuri
Benchmark of sub-1B parameter OCR models on OmniParseBench
A post from @VikParuchuri says the author's team benchmarked recent small OCR model releases with under 1B parameters on OmniParseBench. The post links to no primary source or results page.
Gives engineers a reference point for choosing small OCR models, though the post itself lacks details.
- 9 days ago @vanstriendaniel
Claude Opus edits OCR model weights to stop repetition loops
The author reports asking Claude Opus to find a small OCR model and edit its weights to fix repetition loops. It reportedly changed 0.0085% of the weights with no training, in under 3 hours and about $10 of Hugging Face Jobs compute.
Shows a no-training route to fixing OCR failure modes, though it is an anecdotal, unverified report.
Depth Hybrid Attention: depth-wise value mixing for hybrid architectures
The post introduces Depth Hybrid Attention (DHA), which adds depth-wise value mixing to hybrid attention models with little extra compute or cache. The author claims DWA and DDA variants beat full-attention and hybrid baselines while keeping a low KV-cache footprint.
It proposes a cheap way for layers to mix information across depth in hybrid attention models, with code provided.
- C and C++Post10 days ago @Arikalabs
Using alarm(3) to stop a stuck C program on Linux
The post explains that calling alarm(3) makes the kernel send SIGALRM after about three seconds. If the signal's default action is unchanged, the process terminates, with no polling loop or watchdog thread needed.
A one-call timeout mechanism for runaway loops, useful for tools and tests, but it relies on signal handling and default behavior.
- 10 days ago @ZyphraAI
Zyphra introduces Zyphra Hierarchical Attention for subquadratic attention
Zyphra announces ZHA, an attention method that searches a learned tree of compressed summaries. The author projects speedups of 35× over dense attention and 5× over MSA at 10M tokens.
It proposes a subquadratic attention design, a direction engineers track for long-context model efficiency.
- 10 days ago @industriaalist
Data-constrained scaling laws for underparameterized LLMs
The author claims LLMs are far underparameterized relative to the brain's synapse count and token exposure, and presents data-constrained scaling laws and learning algorithms for very large transformers. No link is included in the post.
It challenges Chinchilla-style compute-optimal assumptions and proposes scaling parameters versus data differently.
- 10 days ago @Prince_Canuma
NVIDIA RE-USE speech enhancement model now available on MLX-Audio
NVIDIA's RE-USE, a small multilingual universal speech enhancement model, is now available on MLX-Audio. The post's author says it helps clean up noisy recordings that speech-to-text models often struggle with.
A lightweight speech enhancement model on MLX could improve ASR input quality for noisy audio on local hardware.
- 10 days ago @jcrwhittington
Latent Sparse Attention: subquadratic attention research release
Open research release of Latent Sparse Attention (LSA), which the author says matches full-attention accuracy at ultra-long context while speeding up attention up to 22.7x for sparse and 72.4x for full.
Shows where sparse attention time goes at 10M tokens and reports speedups for long-context attention.
Defaulted C++ constructors may hurt GCC compile performance
A post by Daniel Lemire asks whether C++ constructors should be defined with = default, claiming this can have notable negative performance effects with recent GCC versions. It points to his articles for details.
Choosing = default versus explicit constructor bodies may affect compile times or generated code, so it is worth verifying with your GCC version.
- LLM inferencePost10 days ago @TeksEdge
Pithos: C inference engine for Liquid AI LFM models
Pithos is a C inference engine for Liquid AI's LFM models. It keeps frequently used experts in VRAM, runs others on the CPU, and reads weights from disk when needed. Developer-reported: 17.5–21.9 tps vs 11.2–11.3 tps for llama.cpp CUDA on a 4GB GTX 1650.
Shows hybrid expert offloading on a small GPU, a concrete tradeoff for memory-limited local inference.
- 11 days ago @ChangJonathanC
Neuron Tuning: fine-tuning only context-firing neurons
The author introduces Neuron Tuning, which fine-tunes only a few neurons that fire in the target context. The author claims it changes the model 3-19x less on unrelated text than LoRA.
Offers a parameter-efficient alternative to LoRA that may limit unintended behavior changes outside the target context.
hgpu.org
