> ## Documentation Index > Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Cold starts > What a caller waits for when nothing is running, and the three ways to change it A container that has scaled to zero has to come back before it can answer. For a small CPU endpoint that is a second or two. For an LLM it is minutes, because the blueprint has to be pulled, the weights read and the CUDA graphs captured before the first token. There are three ways to handle that, and they trade cost against latency in different places. | | first call after idle | cost while idle | good for | | - | - | - | - | | **Scale to zero** (default) | full cold start | nothing | dev, bursty or rarely used endpoints | | **Checkpoint** | restore from a snapshot | nothing | an endpoint that is used in bursts and cannot justify a GPU sitting idle | | **Always on** | none | a container, continuously | anything a person or a product waits on | Measured on an L4 with `Qwen/Qwen2.5-1.5B-Instruct`, same model and settings throughout: | | | | - | - | | cold start | **\~470s** | | restore from a checkpoint | **\~20s** (CRIU itself 9-11s) | | always on, warm | **\~1.05s** | ## What a call does while a container comes up A call that arrives when nothing is ready waits for a container instead of failing. It waits for as long as a container of the endpoint is on its way: being placed, pulling its image, or running and still loading (its `/health` not answering yet, as opposed to answering `500`), up to an hour. It's answered the moment the container is ready, ahead of calls that came after it. If nothing is on its way, the wait ends after about 15 minutes with a `504`. Hang up whenever you like. A caller that gives up while its call waits takes the call with it, so a client that retries after its own timeout doesn't leave the earlier tries queued and counted against `max_pending_tasks`. That holds for request bodies up to 1 MiB, which covers chat requests; a larger body is noticed only once its call reaches the container. ## Always on The simplest answer, and usually the right one for production. `keep_warm_seconds` only says how long to linger *after the last request*, so any finite value eventually idles out and somebody pays for it. `min_containers` keeps a replica up, and brings it back if one dies: ```python app.py theme={null} from levers9 import QueueDepthAutoscaler from levers9.integrations import VLLM, VLLMArgs llm = VLLM( name="chat", gpu="L4", cpu=2, memory="21Gi", vllm_args=VLLMArgs(model="Qwen/Qwen2.5-1.5B-Instruct", max_model_len=8192), autoscaler=QueueDepthAutoscaler(min_containers=1, max_containers=4), ) ``` You are billed for that replica whether or not anyone calls it. In the dashboard the same choice is the **Availability** field when you create an endpoint, and the list marks an always-on endpoint so it is clear which ones are costing you money between calls. ## Checkpointing `checkpoint_enabled=True` snapshots the container once it is warm, and later cold starts restore that snapshot instead of booting. The weights are already in GPU memory in the snapshot, which is the part of a vLLM start that takes minutes. ```python theme={null} llm = VLLM( name="chat", gpu="L4", cpu=2, memory="21Gi", checkpoint_enabled=True, vllm_args=VLLMArgs(model="Qwen/Qwen2.5-1.5B-Instruct", gpu_memory_utilization=0.6), ) ``` Nothing else is required. The first call after a deploy still pays a full cold start - that boot is what *creates* the snapshot. The container signals when it is ready (once every worker is serving), the snapshot is taken while it keeps running, and every cold start after that restores. ### Sizing the container, which is the part that bites A GPU checkpoint copies device memory **into the container**, into host allocations charged to its memory limit, at roughly 1.2x the live device footprint. So a container that will be checkpointed has to hold both its own resident memory and a copy of everything on the card: ``` container_memory > process_rss + 1.2 x device_footprint + slack ``` vLLM carries about 3.8 GiB before a single weight loads. On a 22 GiB L4 at `gpu_memory_utilization=0.6` that is `3.8 + 1.2 x 13.4 ≈ 21 GiB`, so `memory="21Gi"`. Get it wrong and the dump is killed part way through, and what you see is **not** an out-of-memory error: ``` cuda_plugin: CHECKPOINT_DEVICES failed ... "OS call failed or operation not supported on this OS" Dumping FAILED. criu failed: type DUMP errno 3 ``` which reads as though the GPU cannot be checkpointed at all. It can. The SDK checks this arithmetic at deploy time and tells you the number to use, so in practice you should see a warning rather than that log. Two consequences worth knowing: * **The host needs the RAM, not just the GPU.** A 16 GiB host (`g4dn.xlarge`, `g6.xlarge`) cannot hold a vLLM checkpoint at any setting that also boots. The GPU pools are sized for this; a self-hosted cluster needs hosts with meaningfully more RAM than the card has VRAM. * **A failed dump takes the container with it**, so the request that triggered it pays a second cold start. Heed the warning rather than deploying past it. ### When a snapshot is used, and when it is not A checkpoint belongs to a **stub** - the thing a deploy creates from your code plus its runtime config. When a container is scheduled, the scheduler looks up the newest available checkpoint for that stub id and attaches it; the worker restores whatever it is handed. So a snapshot is reused only for the *same* stub. Change anything that makes a new one - the model, an engine argument, `memory`, `cpu`, the blueprint - and there is no checkpoint for it yet, so the next boot is cold and takes a fresh snapshot. That is correct, because the old snapshot has the old settings baked into process memory, but it means iterating on settings means repeated cold starts. The snapshot also contains the weights, so its size scales with the footprint: about 3.2 GiB for a 1.5B model holding 13 GiB on the card. ## Which to pick Reach for **always on** when a person or a product waits on the endpoint. It is the only option with no first-call penalty, and for a small model it is often cheaper than the engineering time spent avoiding it. Reach for **checkpointing** when an endpoint is genuinely bursty - idle for hours, then busy - and a GPU sitting idle is not worth it. It turns minutes into tens of seconds, not into nothing. Leave **scale to zero** for development and for endpoints nobody is waiting on. They combine: an always-on endpoint with `checkpoint_enabled=True` pays its cold start once at deploy, and the snapshot is there if the replica is ever rescheduled onto another node. > **[Parameters](https://levers9.mintlify.site/sdk/parameters.md)** > > `keep_warm_seconds`, `autoscaler`, `checkpoint_enabled` and the rest This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.