Classify the bottleneck, reason from hardware constraints, profile before changing code, and learn the kernel patterns behind modern AI systems performance. Examples were originally worked against CUDA 12.2 and Turing sm_75, while the engineering principles generalize.
Every GPU kernel is first constrained by compute throughput, data-movement bandwidth, and the amount of parallel work available to hide latency. Classify the bottleneck before optimizing.
Open lesson →CUDA threads execute in 32-lane warps under the SIMT model. When lanes in one warp follow different control-flow paths, the hardware generally executes each path while masking inactive lanes…
Open lesson →Registers, shared memory/L1, L2, and DRAM have different capacities and latencies. Global-memory efficiency depends on whether a warp's addresses can be served by a small number of memory tr…
Open lesson →Occupancy is the ratio of active warps to the architectural maximum. It is limited by threads, registers per thread, shared memory per block, and architectural limits.
Open lesson →Shared memory is divided into banks. When lanes in a warp access different addresses in the same bank, the operation is split; broadcasts from the same address are a special case.
Open lesson →`__shfl_*_sync`, `__ballot_sync`, and related operations exchange register values or construct masks within a warp without shared memory or a block-wide barrier.
Open lesson →Efficient reductions are hierarchical: local thread accumulation, warp shuffle, one shared value per warp, then a final block reduction. Grid-wide reduction usually needs a second kernel or …
Open lesson →Tiling stages reusable blocks of data in shared memory or registers so that one DRAM load serves multiple computations.
Open lesson →High-performance GEMM uses hierarchical tiling: a CTA owns an output tile, warps and threads hold smaller accumulator tiles, and the K dimension advances through staged tiles.
Open lesson →Convolution can map to GEMM, but explicit im2col materializes a large matrix. Implicit GEMM generates coordinates inside the kernel while reusing tiles.
Open lesson →Stable Softmax subtracts each row's maximum, computes exponentials and their sum, then normalizes. Its core is two related reductions without large intermediates.
Open lesson →LayerNorm computes row mean and variance before an affine transform. Welford maintains `(count, mean, M2)` online and supports parallel combination.
Open lesson →Standard attention is `softmax(QK^T/sqrt(d))V`. Flash-style algorithms process Q/K/V tiles while maintaining running row maxima, normalizers, and output accumulators, avoiding a full NxN sco…
Open lesson →Fusion places producers and consumers in one kernel so intermediates remain in registers or shared memory, eliminating global write/read cycles and some launches.
Open lesson →Mixed precision uses FP16 inputs and Tensor Cores for suitable GEMM/Conv work while retaining FP32 accumulation, reductions, and sensitive operations.
Open lesson →A Triton program instance processes a logical data tile. The author describes tile shapes, addresses, masks, and reductions while the compiler maps work to warps and threads.
Open lesson →A kernel defines how the GPU computes. An operator also defines schema, mutation and aliasing, dispatch, fake metadata, backward behavior, and compiler integration.
Open lesson →The Roofline bound is `P <= min(P_compute_peak, BW_peak * AI)`. Low arithmetic intensity tends toward memory limits; high intensity tends toward compute limits; points far below both roofs o…
Open lesson →An active warp is resident but not necessarily ready. An eligible warp has its next instruction's dependencies satisfied. Schedulers need eligible warps to fill issue slots.
Open lesson →CUDA Graphs capture and instantiate a fixed GPU workflow, then submit it with one replay call, reducing CPU and driver overhead for many short kernels.
Open lesson →Operations in one stream are ordered; independent streams may overlap when dependencies and hardware permit. Across batches, use separate buffers, pinned host memory, and asynchronous copies…
Open lesson →With S SMs and R resident blocks per SM, approximate wave capacity is C=S*R. A grid requires `ceil(B/C)` waves, and an incomplete final wave leaves many SMs idle.
Open lesson →Registers are thread-private but consume a finite SM-wide register file. Excess live values and dynamically indexed local arrays can enter local memory; local means a private address space, …
Open lesson →`float2`, `float4`, or 128-bit load/store operations move multiple adjacent elements per thread. They mainly reduce memory-instruction count, address arithmetic, and loop iterations; they do…
Open lesson →Split a long K reduction into P partial GEMMs, then merge the partial output. Split-K is useful when M and N are small, K is large, and the ordinary GEMM launches too few output tiles to occ…
Open lesson →