Skip to main content
A container that has scaled to zero has to come back before it can answer. For a small CPU endpoint that is a second or two. For an LLM it is minutes, because the blueprint has to be pulled, the weights read and the CUDA graphs captured before the first token. There are three ways to handle that, and they trade cost against latency in different places. Measured on an L4 with Qwen/Qwen2.5-1.5B-Instruct, same model and settings throughout:

​
What a call does while a container comes up

A call that arrives when nothing is ready waits for a container instead of failing. It waits for as long as a container of the endpoint is on its way: being placed, pulling its image, or running and still loading (its /health not answering yet, as opposed to answering 500), up to an hour. It’s answered the moment the container is ready, ahead of calls that came after it. If nothing is on its way, the wait ends after about 15 minutes with a 504. Hang up whenever you like. A caller that gives up while its call waits takes the call with it, so a client that retries after its own timeout doesn’t leave the earlier tries queued and counted against max_pending_tasks. That holds for request bodies up to 1 MiB, which covers chat requests; a larger body is noticed only once its call reaches the container.

​
Always on

The simplest answer, and usually the right one for production. keep_warm_seconds only says how long to linger after the last request, so any finite value eventually idles out and somebody pays for it. min_containers keeps a replica up, and brings it back if one dies:
app.py
You are billed for that replica whether or not anyone calls it. In the dashboard the same choice is the Availability field when you create an endpoint, and the list marks an always-on endpoint so it is clear which ones are costing you money between calls.

​
Checkpointing

checkpoint_enabled=True snapshots the container once it is warm, and later cold starts restore that snapshot instead of booting. The weights are already in GPU memory in the snapshot, which is the part of a vLLM start that takes minutes.
Nothing else is required. The first call after a deploy still pays a full cold start - that boot is what creates the snapshot. The container signals when it is ready (once every worker is serving), the snapshot is taken while it keeps running, and every cold start after that restores.

​
Sizing the container, which is the part that bites

A GPU checkpoint copies device memory into the container, into host allocations charged to its memory limit, at roughly 1.2x the live device footprint. So a container that will be checkpointed has to hold both its own resident memory and a copy of everything on the card:
vLLM carries about 3.8 GiB before a single weight loads. On a 22 GiB L4 at gpu_memory_utilization=0.6 that is 3.8 + 1.2 x 13.4 ≈ 21 GiB, so memory="21Gi". Get it wrong and the dump is killed part way through, and what you see is not an out-of-memory error:
which reads as though the GPU cannot be checkpointed at all. It can. The SDK checks this arithmetic at deploy time and tells you the number to use, so in practice you should see a warning rather than that log. Two consequences worth knowing:
  • The host needs the RAM, not just the GPU. A 16 GiB host (g4dn.xlarge, g6.xlarge) cannot hold a vLLM checkpoint at any setting that also boots. The GPU pools are sized for this; a self-hosted cluster needs hosts with meaningfully more RAM than the card has VRAM.
  • A failed dump takes the container with it, so the request that triggered it pays a second cold start. Heed the warning rather than deploying past it.

​
When a snapshot is used, and when it is not

A checkpoint belongs to a stub - the thing a deploy creates from your code plus its runtime config. When a container is scheduled, the scheduler looks up the newest available checkpoint for that stub id and attaches it; the worker restores whatever it is handed. So a snapshot is reused only for the same stub. Change anything that makes a new one - the model, an engine argument, memory, cpu, the blueprint - and there is no checkpoint for it yet, so the next boot is cold and takes a fresh snapshot. That is correct, because the old snapshot has the old settings baked into process memory, but it means iterating on settings means repeated cold starts. The snapshot also contains the weights, so its size scales with the footprint: about 3.2 GiB for a 1.5B model holding 13 GiB on the card.

​
Which to pick

Reach for always on when a person or a product waits on the endpoint. It is the only option with no first-call penalty, and for a small model it is often cheaper than the engineering time spent avoiding it. Reach for checkpointing when an endpoint is genuinely bursty - idle for hours, then busy - and a GPU sitting idle is not worth it. It turns minutes into tens of seconds, not into nothing. Leave scale to zero for development and for endpoints nobody is waiting on. They combine: an always-on endpoint with checkpoint_enabled=True pays its cold start once at deploy, and the snapshot is there if the replica is ever rescheduled onto another node.

Parameters

keep_warm_seconds, autoscaler, checkpoint_enabled and the rest