CUDA Graphs: Remove Repeated Launch Overhead

Core Concept

CUDA Graphs capture and instantiate a fixed GPU workflow, then submit it with one replay call, reducing CPU and driver overhead for many short kernels.

Why It Affects Performance

Regular gaps between microsecond kernels can exceed computation time. Graphs do not change FLOPs, DRAM bytes, or internal stalls; they reduce submission gaps.

Example or Pseudocode

static_x = torch.empty(shape, device='cuda')
g = torch.cuda.CUDAGraph()
# Warm up first on a side stream.
with torch.cuda.graph(g): static_y = workload(static_x)
static_x.copy_(new_x); g.replay()

Graphs capture addresses, so update static buffers instead of rebinding variables.

Profiling and Performance Judgment

Compare eager and replay timelines in Nsight Systems. Measure both GPU Event time and CPU end-to-end time, explicitly deciding whether copies and capture amortization are included.

Five-Minute Exercise

Twenty 3-us kernels each have a 5-us eager gap; graph submission costs 10 us total. What are the steady-state times and speedup?