CUDA Streams: Hide Transfers Behind Computation
REFERENCE CONTEXT · CUDA 12.2; RTX 2070 SUPER; Turing sm_75
Core Concept
Operations in one stream are ordered; independent streams may overlap when dependencies and hardware permit. Across batches, use separate buffers, pinned host memory, and asynchronous copies to pipeline H2D, compute, and D2H.
Why It Affects Performance
non_blocking=True permits asynchronous behavior but does not guarantee overlap. A batch must be transferred before it is computed; the useful overlap is transfer of batch n+1 with computation of batch n.
Example or Pseudocode
with torch.cuda.stream(streams[slot]):
d_in[slot].copy_(h_in, non_blocking=True)
kernel(d_in[slot], d_out[slot])
h_out.copy_(d_out[slot], non_blocking=True)Profiling and Performance Judgment
In Nsight Systems, inspect HtoD, Kernel, and DtoH lanes. Verify pinned memory, separate streams and buffers, absence of implicit synchronization, and device asyncEngineCount.
Five-Minute Exercise
If H2D=4 ms, compute=7 ms, D2H=2 ms, what is the ideal time for 100 batches?
Show answer
Pipeline fill/drain costs 13 ms and the remaining 99 batches cost 99*7=693 ms, for about 706 ms total. Compute is the steady-state bottleneck.