Operator Fusion: Remove Intermediates Without Causing Spills

Core Concept

Fusion places producers and consumers in one kernel so intermediates remain in registers or shared memory, eliminating global write/read cycles and some launches.

Why It Affects Performance

Reduced traffic often directly accelerates bandwidth-bound chains. Excessive fusion, however, increases live values, registers, code size, and branches, potentially reducing occupancy or causing spills.

Example or Pseudocode

# Decomposed
t = a * x
y = torch.relu(t + b)
# Fused semantics
y = relu(a * x + b)

Profiling and Performance Judgment

Use PyTorch Profiler and Nsight Systems for kernel count and allocations; use Nsight Compute for DRAM bytes, registers/thread, local traffic, and duration.

Five-Minute Exercise

If fusion removes one FP32 intermediate write and reread, how much primary traffic is saved per element?