CUDA and GPU programming
- CUDA and GPU programmingArticle
Inside NVIDIA’s Blackwell GPU
The article examines NVIDIA’s Blackwell GPU and places it in the company’s tradition of building large GPUs.
GPU hardware context can help engineers understand the architecture behind their workloads.
- 8 months ago @mohitwt_
A CuPy/CUDA FlashAttention Implementation from Scratch
The project implements tiled FlashAttention with online softmax in CuPy and custom CUDA kernels, avoiding materialization of the NxN score matrix. The author reports mathematical verification but no speedup over cuBLAS.
It describes a hands-on implementation of FlashAttention and notes the performance gap with cuBLAS.
- CUDA and GPU programmingArticle
A comparison of inference-focused AI hardware
Turing Post compares the 2026 inference chip landscape, including NVIDIA Vera Rubin, MatX’s programmable LLM accelerator, and Taalas’ model-as-hardware approach, with a focus on cost per token.
Useful for engineers evaluating specialized accelerators and their trade-offs against GPU infrastructure.
A Step-by-Step Introduction to CUDA C++
NVIDIA's blog post introduces CUDA programming with a simple, step-by-step parallel programming example.
Useful for engineers looking for an accessible starting point for writing GPU code with CUDA C++.
- CUDA and GPU programmingArticle
A Worklog on Optimizing CUDA Matrix Multiplication
Simon Boehm documents the iterative optimization of a CUDA matrix multiplication kernel, aiming to understand how to approach cuBLAS-like performance rather than build a cuBLAS replacement.
The worklog offers a concrete view of kernel optimization decisions for a common GPU operation.
- 8 months ago @remi_or_
Padding inputs to reuse variable-size CUDA graphs
The post recommends setting a maximum input size and padding inputs to the nearest interval, creating a family of graphs for different sequence lengths. It claims this avoids recompilation.
This is a concrete graph-capture strategy to consider for inference workloads with variable sequence lengths.
Efficient Matrix Transpose in CUDA C/C++
Mark Harris's NVIDIA blog post discusses performance gains achievable by using shared memory for matrix transpose in CUDA C/C++.
It explains how shared memory can improve the performance of a common GPU operation.
- 8 months ago @mohitwt_
CUDA Kernel Patterns: Reduction, Convolution, Histogram, and Softmax
The post lists parallel reduction, 1D convolution, histogram, and numerically stable softmax as core CUDA kernel patterns, mentioning techniques such as tiling, halo regions, data reuse, atomics, and multi-stage reduction.
These examples highlight common parallelization, memory-reuse, and contention challenges in CUDA kernels.
- CUDA and GPU programmingArticle
HPC++ Automatically Parallelizes C++ for CPUs and OpenCL GPUs
HPC++ is an LLVM-based framework that transforms sequential C++ programs into parallel implementations for multi-core CPUs and OpenCL-capable GPUs.
It may be relevant to engineers evaluating automatic parallelization across heterogeneous CPU–GPU systems.
- CUDA and GPU programmingProject
Booth compiler targets multiple GPU and CPU architectures
Booth is an open-source compiler for CUDA, Triton, and HIP that targets multiple GPU and CPU architectures.
Engineers can explore a compiler project aimed at running these programming models across different architectures.
- CUDA and GPU programmingProject
Booth is an open-source compiler for CUDA, Triton, and HIP
Booth targets multiple GPU and CPU architectures. The post describes BarraCUDA as compiling CUDA .cu files to AMD GFX11 machine code and ELF .hsaco binaries without an LLVM dependency.
It may help engineers explore compiling CUDA code for AMD GPUs and other architectures.
- CUDA and GPU programmingArticle
Inside VOLT: Designing an Open-Source GPU Compiler
An article about VOLT, an open-source GPU compiler. Its preview describes emerging open GPU architectures that define SIMT functionality.
Useful context for engineers exploring open-source GPU compiler design and architectures.
- CUDA and GPU programmingArticle
How GPUs Evolved from SIMD to SIMT
An ACM SIGGRAPH Blog post on how GPUs combined thread abstractions with SIMD hardware, becoming flexible processors for graphics, AI, and scientific computing.
Useful context for understanding the GPU execution model behind modern workloads.
- CUDA and GPU programmingArticle
CUDA for LLMs covers kernels, Flash Attention, and profiling
The Manning book introduces CUDA programming for NVIDIA GPUs, from first kernels to advanced LLM features such as Flash Attention. Its preview also mentions profiling with Nsight Compute.
Useful for engineers who want to learn GPU-level programming and profiling for LLM workloads.
CuTe Layouts Explain Matrix Shapes and Strides
The post recommends CUTLASS/CuTe’s Layout Algebra tutorials for understanding matrix shapes and strides. The linked NVIDIA documentation covers CuTe layouts.
Useful for engineers learning how CuTe represents tensor layouts in CUTLASS.
CUDA-L2 uses reinforcement learning to optimize HGEMM kernels
CUDA-L2 combines large language models and reinforcement learning to optimize half-precision General Matrix Multiply CUDA kernels, using execution speed as the reward. The paper evaluates kernels across 1,000 configurations and reports outperforming matmul baselines including cuBLAS.
The approach offers a way to automate tuning CUDA matrix multiplication kernels against execution-speed measurements.
- 11 months ago @marc_lelarge
A Numba-to-Triton Path for GPU Programming
The post describes a hands-on Jupyter notebook that introduces GPU programming with Numba, then progresses to Triton for writing kernels in a Python-like language.
It offers engineers a practical starting point for learning GPU kernel programming.
- 11 months ago @vivekgalatage
Video lectures for Programming Massively Parallel Processors
The post points to video lectures for the book “Programming Massively Parallel Processors” and highlights lecture 04 on GPU architecture.
The lectures offer material on GPU programming and architecture.
Red Hat’s Introduction to GPU Programming
The first article in a four-part series, it gives a basic overview of the GPU programming model. The post also points to a GPU architecture roadmap from Cornell.
A concise starting point for engineers learning the GPU programming model and architecture.
- CUDA and GPU programmingArticle
SASS Latency Tables and Instruction Reordering
The post argues that latency tables extracted from nvdisasm are not useful and that instruction reordering can improve performance by 3–4%, with a theoretical limit of 10%.
It raises practical questions about using SASS latency data and instruction scheduling when tuning GPU kernels.
hgpu.org
arxiv.org