LLM inference
- LLM inferencePost1 day ago @casper_hansen_
DeepSeek V4.1 Flash on one RTX 3090 with vLLM and ExLlamaV3
The author reports running DeepSeek V4.1 Flash on one 3090 with 64GB RAM at about 20 tokens/s, claiming no quality loss. The stack pairs a vLLM frontend for scheduling, tool calling and OpenAI/Anthropic APIs with ExLlamaV3 for the forward pass.
Shows a concrete consumer-GPU serving stack split between a scheduler frontend and a swappable forward pass.
- LLM inferencePost1 day ago @TeksEdge
Kinetic: Rust inference engine that processes prompts before Enter
Open-source Rust inference engine (Apache 2.0) that the post says starts processing a pasted prompt before Enter is pressed. Reported RTX 3060 laptop test: 0.13s wait vs 10.95s for llama.cpp.
Shows prefill and prompt-reuse ideas applied to local inference latency; the benchmarks are the author's reported numbers.
- LLM inferenceArticle
Making Deep Learning Go Brrrr From First Principles
Horace He's essay explains how to reason about deep learning performance from first principles, focusing on compute, memory bandwidth, and overhead. The item shares it as a recommended read.
It gives a framework for judging whether a model's bottleneck is compute, memory, or overhead, which matters for LLM inference costs.
- LLM inferencePost6 days ago @TeksEdge
Running a 284B DeepSeek model in 64GB RAM on CPU via llama.cpp fork
A developer-modified llama.cpp fork, llama-modchip, reportedly runs DeepSeek-V4-Flash (284B total, ~13B active) in 64GB DDR5 RAM with no GPU, using low-bit quantization and custom AVX2/AVX-512 kernels.
Shows CPU-only inference of a large MoE model, with quantization and kernel gains; numbers are developer-reported.
- LLM inferenceArticle
NVIDIA: LLM agents can skip long context via frequent compaction
NVIDIA study finds agents can finish long-horizon tasks by periodically compacting state instead of keeping full history, reporting up to 8.8x lower estimated inference cost across six benchmarks.
Offers a concrete way to cut per-token inference cost in agent systems by shrinking the active context.
- LLM inferencePost8 days ago @oliviscusAI
tiny-llm: open-source course on building a vLLM-style inference engine
An open-source four-week course where you build a small vLLM-style inference engine around Qwen3, covering attention, KV cache, 4-bit weights, C++/Metal kernels and batching. It runs on Apple Silicon with MLX.
Hands-on, measured practice with KV cache, quantized weights and batching, runnable on a laptop without an NVIDIA GPU.
- LLM inferencePost10 days ago @TeksEdge
Pithos: C inference engine for Liquid AI LFM models
Pithos is a C inference engine for Liquid AI's LFM models. It keeps frequently used experts in VRAM, runs others on the CPU, and reads weights from disk when needed. Developer-reported: 17.5–21.9 tps vs 11.2–11.3 tps for llama.cpp CUDA on a 4GB GTX 1650.
Shows hybrid expert offloading on a small GPU, a concrete tradeoff for memory-limited local inference.
- LLM inferencePost11 days ago @TeksEdge
vLLM reads Qwen PLE table rows from safetensors to cut loaded memory
vLLM can now read only the needed rows of Qwen's 51B PLE lookup table directly from the original safetensors files. On one DGX Spark GB10, loaded memory fell from 99.09 to 72.09 GiB, with decode within ~3.5% at c=1.
Lets engineers serve large per-layer-embedding models on memory-limited hardware with little decode slowdown.
- LLM inferencePost14 days ago @TheZachMueller
Interconnect Bandwidth Limits Distributed LLM Inference and Training
A practitioner's reminder that communication overhead (RDMA, PCIe, InfiniBand) can leave multi-GPU or multi-machine setups far slower than one larger machine. The post has no linked source.
It helps engineers weigh splitting models across GPUs or nodes against keeping work on a single machine.
- LLM inferencePost14 days ago @superalesha
Author reports Qwen3.8-27B batched serving on a single RTX 3090
The author reports serving Qwen3.8-27B on one RTX 3090 capped at 250W to 64 concurrent chats, about 1035 tok/s total and 127 tok/s for one chat, with up to 150K context via a two-command Docker Compose setup.
Shows batched throughput figures for a 27B model on a consumer GPU, though the numbers are self-reported.
- LLM inferenceProject
DeepSeek open-sources DeepInfer, its in-house LLM inference framework
DeepSeek announced DeepInfer, its production inference framework, now open source, used to serve DeepSeek-V4.1-Flash. The repository describes it as a framework for high-performance, large-scale inference of DeepSeek models.
A production serving stack from a major lab offers concrete reference for throughput, latency and cost per token.
- LLM inferencePost23 days ago @ErikVoorhees
Diffusion text model generates whole output at once, unlike token-by-token LLMs
A post describes diffusion text models like Mercury, which produce the entire output simultaneously rather than sequentially token by token, as autoregressive LLMs do. The author says it works fast.
It offers a different decoding paradigm with a distinct latency and cost profile compared to standard autoregressive inference.
- LLM inferencePost24 days ago @lemire
Intel GPU D270 (Crescent Island) inference accelerator card
A post describing Intel's GPU D270 (Crescent Island), a PCIe 5.0 card built for inference with 32 Xe cores, XMX matrix engines and LPDDR5X memory of about 160 GB, expandable to 480 GB.
Memory capacity and matrix-engine design of inference cards shape how large models are served and at what cost per token.
- LLM inferenceArticle
Intel GPU D-Series: 480GB LPDDR5X inference card now sampling
Intel's GPU D-Series (formerly Crescent Island) offers up to 480GB LPDDR5X at 1.5TB/s and 1.3 PFLOPS FP8 in a 350W PCIe card, now sampling. Intel's own card is 160GB; partners reach 480GB.
Large on-card memory at 350W targets LLM inference capacity and cost per token for serving.
- LLM inferencePost1 month ago @thesupermannx
LittleBit: Samsung method compressing 13B LLMs under 1 GB
The author reports that Samsung open-sourced LittleBit, which uses latent factorization to compress 13B-parameter LLM weights to under 1 GB, with sub-1-bit levels and XOR in place of multiplication.
Relevant to engineers evaluating sub-1-bit quantization for local deployment; the speedup figures are the author's claims.
- LLM inferencePost1 month ago @MercityAI
Layer pruning of Qwen3-8B-Base with LaCo to cut model size
A post reports removing up to 10 layers from Qwen3-8B-Base using LaCo (Layer Collapse), with 6 layers deleted yielding a 16.7% smaller model. The author claims 78-94% of reasoning scores were kept at a 1.68x perplexity ratio.
Shows a concrete trade-off between layer pruning, model size, and quality, relevant to cheaper inference.
- LLM inferenceVideo
DeepSeek V4.1-Flash’s KV Cache and Attention Techniques
The post says DeepSeek V4.1-Flash uses 890 bytes of global KV cache per token and an 8× smaller persistent cache. It names Causal Encoder-Decoder, Compressed Sparse Attention, and SWA Bounded Replay as techniques.
The reported cache reductions and attention techniques may be relevant to engineers optimizing LLM inference costs.
- LLM inferenceProject
Strata runs Qwen3.8-Flash-Next on GPUs with 8GB+ VRAM
Strata is an inference engine for running the 125B MoE Qwen3.8-Flash-Next on NVIDIA GPUs with at least 8GB of VRAM. Its GitHub page describes one-click installation for Windows and Linux and localhost OpenAI/Anthropic APIs.
The repository may be useful to engineers evaluating local serving of a large MoE model on modest GPU memory.
- LLM inferenceProject
DeepEP-Ascend is a communication library for Huawei NPUs
DeepEP-Ascend is a high-performance communication library for machine learning training and inference on Huawei Ascend NPUs.
Engineers serving models on Ascend hardware can assess its communication support for inference workloads.
- LLM inferencePaper
Efficiently Scaling Transformer Inference
A paper by Reiner Pope and coauthors on scaling Transformer inference efficiently.
Relevant to engineers evaluating approaches to serve Transformer models at scale.
