NVIDIA Dynamo Snapshot Slashing AI Inference Cold Start from Minutes to Seconds on Kubernetes

The Cold-Start Nightmare for AI Inference

In production, inference workloads face a brutal reality: demand spikes unpredictably, but scaling GPU replicas on Kubernetes takes minutes — not seconds. During that window, GPUs are allocated but idle, burning money and risking SLA violations.

For a single-GPU vLLM workload, cold-start latency breaks down like this:

• Model weight loading: 60-80% of total time
• CUDA graph compilation: 10-15%
• Kernel warm-up: 5-10%
• Distributed runtime registration: 5-10%

NVIDIA's new Dynamo Snapshot tackles this head-on with a checkpoint/restore approach that brings startup times close to "speed of light" (SOL) — the theoretical minimum for memory bandwidth. This post dives into the engineering behind it, based on the original NVIDIA Developer Blog.

How Dynamo Snapshot Works: CRIU + CUDA Checkpoint

The core insight: instead of re-initializing everything from scratch, snapshot the fully warmed-up worker and restore it on a new node in seconds.

The Two-Layer Checkpoint

These tools compose cleanly: cuda-checkpoint dumps GPU state into CPU memory, then CRIU serializes the entire process tree to disk. On restore (even on a different node), CRIU rehydrates the host state, and cuda-checkpoint reattaches GPU memory.

Kubernetes Integration via DaemonSet

Instead of relying on Kubernetes native checkpoint/restore (which delegates to CRIU but is cloud-vendor dependent), NVIDIA built a privileged DaemonSet called snapshot-agent. It:

• Runs on every node
• Handles checkpoint/restore for runc containers without modifying runc
• Writes artifacts to shared storage (NFS/SMB)
• Parallelizes across the cluster automatically

# Example: Deploying the snapshot-agent via Helm
helm repo add nvidia-dynamo https://nvidia.github.io/dynamo
helm install snapshot-agent nvidia-dynamo/snapshot-agent \
  --set storage.backend=nfs \
  --set storage.nfs.server=10.0.0.1 \
  --set storage.nfs.path=/exports/checkpoints

The Quiesce/Resume Pattern

The trickiest part: after restore, the worker must re-establish network connections and re-register with the control plane. Dynamo Snapshot uses a signal file pattern:

1. Worker initializes engine → writes ready_for_checkpoint.signal
2. Worker enters polling loop waiting for restore_complete.signal
3. snapshot-agent checkpoints the worker inside that polling loop
4. On restore, CRIU resumes at the exact same instruction → worker sees the new signal → proceeds with distributed runtime startup

# Pseudocode for the quiesce/resume hook
import os
import time

SIGNAL_DIR = "/tmp/dynamo"
READY_FILE = os.path.join(SIGNAL_DIR, "ready_for_checkpoint.signal")
RESTORE_FILE = os.path.join(SIGNAL_DIR, "restore_complete.signal")

def initialize_engine():
    # Load weights, compile graphs, warm up kernels
    print("[Engine] Initialization complete")
    # Signal that we're ready to be checkpointed
    with open(READY_FILE, "w") as f:
        f.write("ready")

def wait_for_restore():
    # Poll until restore is detected
    while not os.path.exists(RESTORE_FILE):
        time.sleep(0.1)
    print("[Worker] Restore detected, resuming runtime startup")
    # Now connect to control plane, register with discovery backend
    register_with_dynamo_control_plane()

if __name__ == "__main__":
    initialize_engine()
    wait_for_restore()

Three Key Optimizations That Make It Fast

1. KV Cache Unmap & Release

Before checkpointing, the worker deallocates the KV cache (since it hasn't served any requests yet). Using CUDA Virtual Memory Management (cuMemUnmap / cuMemRelease), the virtual address stays stable for CUDA graphs, but physical memory is freed. This reduces checkpoint size from ~190 GiB to ~6 GiB for a small model.

2. Parallel memfd Restore

Large model weights are stored in memfd (shared anonymous memory). Upstream CRIU restores them serially — one 2 GiB buffer at a time. NVIDIA modified CRIU to use a thread pool, restoring all memfd buffers in parallel. This alone gave a 5-8x speedup on large models.

3. Linux Native AIO for Anonymous Memory

Instead of a synchronous preadv loop (one read at a time), Dynamo Snapshot uses Linux native AIO with io_submit and io_getevents. Up to 128 reads in flight simultaneously, saturating fast NVMe or network storage. Combined with O_DIRECT to avoid polluting the page cache, this brings restore times close to SOL.

4. GPU Memory Service (GMS) — The Game Changer

Even with optimized CRIU, moving model weights from storage to CPU to GPU is a serial bottleneck. GMS decouples weight restoration from process restoration using CUDA VMM:

• CRIU checkpoint shrinks to ~5-7 GiB (just process state + metadata)
• Weight artifact (~74 GiB for gpt-oss-120b) is restored concurrently via GPUDirect Storage (GDS) or NVMe stripes
• End-to-end startup time drops from 5+ minutes to under 5 seconds — a 21x improvement

# Conceptual flow: GMS restores weights in parallel with CRIU
# Process 1: CRIU restore (5-7 GiB)
criu restore -d --images-dir /checkpoints/criu/ &

# Process 2: GMS weight restore via GPUDirect Storage
gms_restore --backend gds --source /checkpoints/weights/ --gpu-id 0 &

wait  # Both complete concurrently

Limitations & Caveats

• Single-GPU only for now: Multi-GPU and multi-node checkpoint/restore is on the roadmap but requires handling NCCL connections, RDMA state, and pod IP changes — significantly more complex.
• Not yet upstream: The CRIU optimizations (parallel memfd, AIO) are not merged into upstream CRIU; they'll ship as part of Dynamo Snapshot first.
• Storage dependency: Fast restore requires fast shared storage (NVMe, GDS). On slow NFS without O_DIRECT, gains are reduced.
• Workload awareness required: The quiesce/resume hooks need integration with the inference framework (vLLM, SGLang). Not all frameworks support sleep()/wake_up() natively.

What's Next?

• GMS with pluggable backends: GDS, UCX, peer-GPU RDMA/NVLink (pending CUDA driver patch)
• TensorRT-LLM support
• Multi-GPU and multi-node via hooks for PyTorch, NCCL, NIXL

If you're running production inference on Kubernetes, this is the most promising approach to solving the cold-start problem since GPUs went mainstream. The experimental release supports single-GPU vLLM and SGLang today — worth testing in your staging environment.

Together with: Microsoft Sovereign Cloud: Running Large AI Models Fully Disconnected — explore how disconnected AI governance complements fast inference startup for regulated industries.

Also see: Microsoft's 2026 Database Vision: Unified Data, AI Agents, and the New Fabric Hub — the data infrastructure that will feed these fast-scaling inference workloads.