Skip to content
senn-techsenn-tech
Back to blog
AI Hardware
Local AI and GPU hardware2026-11-15· By Franz Senn

NVIDIA Snapshot: restore in seconds, cold start in minutes

On November 11 we restarted the host of our standby lane by hand, and the container did not come back: its restart policy was set to no. Only the alert mail in the afternoon and a manual step brought the lane back, after which it was healthy within about 120 seconds (as our operations note of November 11, 2026 records it; this is the standby lane with a 27B model on one card, the production lane follows further down). That is exactly the gap NVIDIA Snapshot wants to close, open source since November 10, 2026.

What a GPU checkpoint is. A checkpoint freezes an already initialized process in full: CPU memory, GPU memory, open files, thread state. On restore the process wakes at the point where it was put to sleep. The engine skips weight loading, kernel compilation and CUDA graph capture. Saved is the whole cold start, and loading the files is only one part of it.

NVIDIA logo
NVIDIA open-sourced Snapshot on 10 November 2026; every Snapshot figure in this post is a vendor claim. (Quelle: Simple Icons, CC0)

What NVIDIA Snapshot delivers

Snapshot is a component of the DSX platform and sits as an Apache-2.0 repository in the ai-dynamo GitHub. Installed is one Helm release with two parts: an operator in the control plane, plus a privileged node agent as a DaemonSet on every GPU node. A pod is frozen through the PodSnapshot resource, the restore runs through the annotation nvidia.com/restore-from. Under the hood CRIU works on the process and cuda-checkpoint on the GPU part; the CUDA tool cuda-checkpoint itself stands, according to its repository, under the NVIDIA SDK EULA, separate from Apache.

The numbers from repository and blog (blog dated November 10, 2026), read again and recomputed on November 15, 2026, all of them vendor claims:

QuantityValueUnit
Tested configurations22 (vLLM, SGLang, TensorRT-LLM)count
Restore faster than cold start8.1 to 35.9factor
Restore time named in the blog3.6 to 42.7seconds
Checkpoint time15.3 to 327.2seconds
Heaviest model in the test913.09GB weights
GPU share of restore time on eight cards18.17 to 28.98seconds

The checkpoint range comes from the table in repository and blog: from the smallest model at 15.3 seconds up to GLM-5.3 on SGLang at 327.2 seconds, which is 5.5 minutes. Two details from the table deserve attention. In the smallest model of the test (Qwen3-0.6B), 3.66 of the 4.66 seconds of restore sit in agent setup and CRIU, leaving the GPU share a near footnote. And the restore asks for matching hardware: same GPU population, same driver. Anyone who freezes four cards in tensor-parallel operation needs four of the same to wake them again. The blog rounds the speedup to 8x to 36x, the table in the repository says 8.1 to 35.9. The blog gives the checkpoint time as “15 seconds to 5.5 minutes”; 327.2 seconds are 5.45 minutes. All measurements run, per performance.md, on B200 cards with NFS storage (VAST, NFSv3) and vLLM 0.31.0, driver 615.71.09. That is the same version number as ours, on different hardware; whether B200 and GeForce share the same driver package we did not check.

NVIDIA Snapshot: freezing and waking a GPU processEngine running,initializedweights loaded, kernelsreadyCheckpointCRIU saves the process,cuda-checkpoint the GPU partArtifact in theshared storeCPU state, GPU state, rootfsarchiveRestore viaannotationnvidia.com/restore-from onthe podProcess wakes upno weight reads, no kernelcompiles, no graph capture
Flow per NVIDIA repository and blog, vendor claim; not tried by us. (Quelle: NVIDIA blog on Snapshot, 10 Nov 2026)

Where the time really sits on our side

We run our production lane on a bare-metal host without virtualization, four RTX 5090, Docker instead of Kubernetes, vLLM 0.32.0, tensor parallelism 4, KV cache in nvfp4, 220,000 tokens of context. The weights of the running model occupy 122.5 GiB on disk (sum of file sizes via ls -l), spread over 53 safetensors files. The host has 91 GiB of RAM; free -g showed 1 GiB available on November 15, 2026. The weights therefore never fit fully into the page cache, and every cold start reads fresh from the NVMe.

We read six real restarts from October 18 to November 2 out of the container logs (docker logs --timestamps, evaluated again on November 15, 2026; all six values match the November 12 measurement), each measured from the first log line of the process to the message Application startup complete. No lab setup, these are the starts that came with image and config changes:

StartTotal time sOf that engine initialization s
2026-10-18376.1122.48
2026-10-28 (1)273.846.12
2026-10-28 (2)275.447.07
2026-10-28 (3)300.057.75
2026-10-28 (4)293.357.95
2026-11-02292.463.79

Unit seconds, source: docker logs --timestamps of the production container on our host with four RTX 5090, first measured November 12, 2026, re-read November 15, 2026. Three further starts on November 5 took 302, 297 and 347 seconds (two aborted attempts before them), inside the same range. They are not in the table and chart because we found them only afterwards and their engine initialization is not evaluated; we logged no fixed selection criterion. The median over all nine starts is 297 seconds, over the six in the table 293 (derived); the statement does not change.

Measured cold starts of our vLLM lane (seconds)Oct 18376 · slow startOct 28 (1)274Oct 28 (2)275Oct 28 (3)300Oct 28 (4)293Nov 22920400
Six real restarts, median 293 seconds (all nine known starts: 297). The spread from 274 to 376 we did not break down further. (Quelle: own measurement, container logs, November 15, 2026)

Interim verdict: our 274 to 376 seconds fall inside the cold-start range of NVIDIA's table, which runs from 87 to 1,111 seconds (B200, shared NFS). Closest are gpt-oss-120b on four cards at 156 to 269 seconds and GLM-5.3 on eight cards at 364 to 427 seconds; we did not compare the same model. Subtract the engine initialization and 228 to 254 seconds remain for start and weights. It works out to 0.48 to 0.54 GiB per second. Whether the disk or the deserialization is at fault, we did not measure separately. The initialization itself fluctuates by a factor of 2.7 on our side, because profiling, KV-cache setup and warm-up depend on the state of the machine. Whoever improves only the data throughput works at the wrong end.

What Snapshot blocks for us

The prerequisites from the documentation, as of November 12, 2026, against the measured state of today:

PrerequisiteState on our side, measured November 15, 2026Verdict
Kubernetes with Helmno kubectl, no kubelet, Docker 29.8.0install needed
NVIDIA GPU Operator 26.3 or newernot installedinstall needed
Linux kernel 5.13 or newer7.0.0-42-genericmet
Kernel driver 580 or newer615.71.09met
containerd or CRI-O as runtimeDocker 29.8.0, no CRI operationopen
MIG disabledno MIG-capable cardmet
Privileged agent with hostPID, hostIPC, hostNetworktechnically possible, reaches deep into the hostto weigh
Shared store: ReadWriteMany or S3local, 1.6 TB free on the NVMesingle-host path
User-space driver (libcuda) 615 for multi-GPU workloadstwo workload processes load libcuda.so.615.71.09 (/proc/<pid>/maps); we run no PageBroker or agentmet for the workload

On storage: the snapshot artifacts carry CPU state, GPU state and a rootfs archive. In NVIDIA's table they land between 1.1 and 1.8 times the weights for the vLLM runs on four to eight cards, and one SGLang run on four cards reaches a good ten times. One caveat applies: NVIDIA had the engines release their KV cache before the checkpoint. Our lane claims over 90 percent of GPU memory and has a fixed-size KV cache, and without a release beforehand the KV cache would end up inside the artifact. In NVIDIA's table the ratio for multi-card vLLM runs lies between 1.09 (DeepSeek-V4-Pro, 996.53 to 913.09 GB) and 1.83 (gpt-oss-120b on four cards, 119.69 to 65.25 GB). Applied to our 122.5 GiB of weights that gives 134 to 224 GiB per snapshot (derived, no measurement of ours), and only with the KV cache released. Without the release the artifact gets larger. The free space on the disk still covers it, also for several artifacts.

The real price is the stage. For a single Docker host that means: set up Kubernetes, introduce the GPU Operator, maintain two namespaces, all to set one annotation. Whether the GPU Operator manages a GeForce 5090 cleanly stands nowhere in the prerequisites; the limitations only say that Snapshot runs on GPUs the required CUDA driver supports. We will answer that once the operator is installed.

Striking for our configuration is the condition for multi-GPU workloads: per the documentation it builds on GPU CustomStorage and therefore asks for the user-space driver (libcuda) at version 615 or newer. If the workload does not meet that, the checkpoint hangs in cuCheckpointProcessLock and fails after the timeout with CUDA_ERROR_OPERATING_SYSTEM. A kernel driver below 580 makes the restore bounce off the driver-minimum check. Our lane talks over NCCL on four cards, and the workload loads libcuda 615.71.09. Whether NCCL multicast (NVLS) adds trouble at freeze time, the documentation does not say, and we did not test it on our stack.

The shortcut already sits in the image

Our production contains vLLM 0.32.0, and the command line of the engine offers two things that pursue the same purpose without Kubernetes (vllm --help inside the container, November 15, 2026). First vllm snapshot with the subcommands create, inspect and restore. The help says "Create a snapshot from an initialized TP1 engine" and "Restore a same-host TP1 snapshot". We have not tried the command, only read the help. Our lane runs with tensor parallelism 4, so it falls out. On a single-card lane this would be the fast road.

vllm preload starts one daemon per GPU that keeps the finished quantized weights in VRAM and hands them to other engines over CUDA-IPC. That sounds like our cold-start problem, yet it costs graphics memory. Our lane claims over 90 percent of GPU memory, so the buffer is missing (derived, untested). Then vllm download-kernels pulls precompiled kernels and lowers the compile time at start.

What we make of this

On a single inference machine the restore stays an expansion project. The steps until then are possible now:

  • Set restart policy and systemd unit so the lane comes up without a human. This is exactly where the standby lane failed on November 11.
  • Keep writing the cold-start log: the 102 seconds between the slowest and the fastest start (376.1 against 273.8) deserve an explanation before we buy memory for it.
  • Test a single-card lane through vllm snapshot as soon as one 5090 is free. TP1, same machine, without Kubernetes.
  • Take Snapshot seriously the moment a second reason arrives: several GPU nodes, autoscaling, or tenants with uneven traffic.

How our AI stack stands today stands in the post own AI stack with LiteLLM and vLLM, more operation topics in the collection AI in production.

Anyone who wants to check whether such an expansion pays off for their own company can send us the traffic curve and the hardware list. We measure the cold start on the machine you already have, before anyone buys: Corporate LLM consulting.

What remains

The technology is solidly documented, the numbers are traceable. For us, with one host, Docker and GeForce cards, the restore sits behind three things: a Kubernetes stage, a shared store, and the open question about the GPU Operator. Until then the lever is in our own hands, namely start policy, precompiled kernels and one card that stays free in case of doubt.

Open points: We tried neither Snapshot nor vllm snapshot; every Snapshot figure is a vendor number on B200 cards, and our size range is derived. Whether the GPU Operator runs on GeForce cards and whether NCCL multicast disturbs freezing, we do not know. The start times of our production lane are six runs from the container logs; what caused the outlier on October 18 is not explained. We did not restart the standby lane (120 seconds); that figure comes from our operations note of November 11, 2026.

Read next: H100 rental prices by provider · memory as the limit of local AI agents · Inkling 2 Small against our VRAM limit

Further Reading

Labels: vendor claim = from NVIDIA, retrieved and read by us on November 15, 2026, not re-measured by us; derived = calculated by us from it; own measurement = method stated in the text.

Questions?
Does NVIDIA Snapshot work with GeForce cards?+

The condition is the driver, ahead of the card. Our host runs 615.71.09, measured November 15, 2026 with nvidia-smi, and the documentation asks for kernel driver 580 and, for multi-GPU operation, libcuda 615. NVIDIA restricts the field to GPUs that the required CUDA driver supports. What stays open is whether the GPU Operator cooperates with an RTX 5090.

How large does a snapshot get for a 122 GiB model on four cards?+

Roughly 134 to 224 GiB, derived from the ratio of artifact to weights in NVIDIA's multi-card vLLM runs (1.09 to 1.83) times our 122.5 GiB. That assumes the KV cache is released before the checkpoint, which is how NVIDIA measured. Without a release the artifact would be larger. We have not measured a snapshot.

Is NVIDIA Snapshot production ready?+

No, the repository itself advises against production-critical workloads for now, and the API can change. For a single Docker host there is more: Kubernetes, the GPU Operator and a shared store would have to be built first. We would touch it on a test lane first and leave the production lane alone.