Advanced CUDA Programming: High Performance Computing with GPUs (GPU Expert Engineering: Mastering Design, Programming, and Optimization)

Your Kernel Compiles. It Launches. It Returns the Right Answer. And It's Leaving Half Your GPU on the Floor. Here's the uncomfortable part: it'll never show up in your tests. A correct kernel and a fast kernel look identical from the outside. The difference is buried in instruction issue, occupancy, memory transactions, and how the hardware actually moves your data — and that's exactly where the official docs go quiet, scattered across release notes, tuning guides, and forum threads that never quite connect. This book connects them. It treats CUDA the way the people shipping the fastest kernels on the planet actually treat it: not as a way to launch parallel work, but as a performance-engineering discipline. At the source level, a kernel looks like a clean scalar function. At the machine level, performance is decided by orchestration — computation, data movement, synchronization, and numerical precision arranged like stages of an assembly line. This Second Edition teaches you to think at that level, on the hardware you're actually running: Hopper, Blackwell, and the rack-scale systems coming after them. Inside, you'll work through: Why a kernel that "runs fine" can still waste most of the chip — and the small set of numbers that tell you precisely how much, and where Reading the machine — turning SASS disassembly into compiler decisions you can act on instead of guess at Thread block clusters, the Tensor Memory Accelerator, and warp-group matrix instructions — handled not as exotic edge features, but as the model current hardware is built around Async copy pipelines — moving multidimensional tiles into shared memory without burning an instruction on every element FP8 and low-precision — where it buys real throughput, and where it quietly costs you accuracy FlashAttention-style fused kernels — streaming softmax that never materializes the full attention matrix Occupancy, bank conflicts, and divergence — what they actually cost you in cycles, not in rules of thumb Streams, events, and CUDA graphs — overlap that still holds up under real load Multi-GPU and rack-scale — what to do when the bottleneck leaves the SM and becomes the fabric Tile programming as a first-class path — tiles mapped automatically onto Tensor Cores and TMA Correctness and debugging — how to know a fast kernel is also a kernel you can trust Now the part most book descriptions won't tell you: This is not an introduction to CUDA. If you're looking for "hello world," your first kernel, or a gentle on-ramp to threads and blocks — this is the wrong book, and you'll be annoyed. Buy a beginner title first. But if you already write CUDA and you're tired of code that compiles clean and runs slow — if you want the deeper machinery of how warps are scheduled, why divergence and bank conflicts cost what they cost, how TMA and async copy shape throughput, how Tensor Cores get fed, and why the fastest kernels are designed around the architecture rather than merely compiled for it — then you're exactly who this was written for. The hardware is moving faster than the books that explain it. This one is current, and it goes straight at expert practice. Scroll up, click Buy Now, and start writing kernels that respect the machine underneath them.