Why Attention Profiling Matters
Attention is the heart of Transformers, but its quadratic complexity makes it a prime target for optimization. While many tricks exist, the real challenge is understanding how each implementation behaves under the hood. In this tutorial, we'll profile four attention variants in PyTorch: a naive hand-written version, an in-place optimized version, and the built-in scaled_dot_product_attention (SDPA) with different backends. You'll learn to read profiler traces, spot inefficiencies, and make informed decisions for your models.
We'll use an NVIDIA A100 GPU and the Hugging Face trace-util tool. All code is available in the companion repository.
Step-by-Step: Naive Attention and In-Place Optimization
Let's start with a naive causal attention module built from primitives:
import torch
import torch.nn as nn
import math
class NaiveCausalAttention(nn.Module):
def __init__(self, head_dim):
super().__init__()
self.scale = 1.0 / math.sqrt(head_dim)
def forward(self, q, k, v, mask):
scores = torch.matmul(q, k.transpose(-2, -1))
scores = scores * self.scale
scores = scores.masked_fill(mask, float("-inf"))
attn = torch.softmax(scores, dim=-1)
out = torch.matmul(attn, v)
return out
Profiling this reveals a hidden Memcpy kernel caused by the out-of-place masked_fill. Switching to the in-place version (masked_fill_) eliminates that copy:
def forward(self, q, k, v, mask):
scores = torch.matmul(q, k.transpose(-2, -1))
scores = torch.mul(scores, self.scale)
scores.masked_fill_(mask, float("-inf")) # in-place
attn = torch.softmax(scores, dim=-1)
out = torch.matmul(attn, v)
return out
Key takeaway: In-place ops save memory and time, but only safe under torch.no_grad() (no backward pass).
Comparing SDPA Backends: Math, Efficient, Flash, cuDNN
PyTorch's F.scaled_dot_product_attention dispatches to multiple backends. Here's how they stack up:
| Backend | Kernels/forward | Key characteristic | Performance note |
|---|---|---|---|
| Math | 20 | Reference impl: FP32 on CUDA cores, rebuilds mask every call, uses _safe_softmax | ~3.7x slower than naive, but numerically safe |
| Efficient | 1 | Fused fmha_cutlassF kernel (xformers), stays in bf16 on Tensor Cores | Good balance, no FP32 upcast |
| Flash | 1 | Fused pytorch_flash (FlashAttention-2), low occupancy (~13%) but fastest | Best for large sequences, avoids HBM traffic |
| cuDNN | 1 | Per-problem generated kernel (cudnn_generated_fort...), no transposes, uses cuLaunchKernelEx | Cost moved to CPU (plan selection ~214µs) |
Profiling tip: Always guess what the trace should show before opening it. Mismatches are where the insights live. For example, Flash's low occupancy looks "wrong" but is intentional—it trades occupancy for on-chip data reuse.
Conclusion & Next Steps
Profiling is a superpower: form a hypothesis, open the trace, and chase the mismatch. We shaved a kernel with one underscore, discovered the math backend's 20-kernel overhead, and saw why Flash's low occupancy is actually a feature.
Your turn: Profile your own attention layer. Try pinning backends with torch.nn.attention.sdpa_kernel and compare traces. For a deeper dive into open-source collaboration at scale, check out how Meta deprecated its internal FFmpeg fork. Also, if you're into serverless architectures, see Serverless Genomics at Scale.
Happy profiling! 🤗