> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM serving

> Serve an open-weights model behind an OpenAI-compatible URL with vLLM

`VLLM` is an endpoint that runs vLLM for you: pick a model from the Hugging Face Hub, a GPU, and deploy. The URL
speaks the OpenAI API, so any client that takes a `base_url` works unchanged.

```python app.py theme={null}
from levers9 import Blueprint, QueueDepthAutoscaler
from levers9.integrations import VLLM, VLLMArgs

llm = VLLM(
    name="gemma",
    gpu="L40S", cpu=4, memory="24Gi",
    vllm_args=VLLMArgs(model="google/gemma-4-12B-it", max_model_len=8192),
    secrets=["HF_TOKEN"],                      # for gated repos
    keep_warm_seconds=300,                     # idles out after 5 min; see Cold starts
    autoscaler=QueueDepthAutoscaler(max_containers=2),
)
```

```bash theme={null}
levers9 deploy app.py:llm --name gemma
```

```python theme={null}
from openai import OpenAI
client = OpenAI(base_url="https://levers9.uselevers.com/gemma/v1", api_key=TOKEN)
print(client.chat.completions.create(model="google/gemma-4-12B-it", messages=[{"role": "user", "content": "hi"}]).choices[0].message.content)
```

An LLM that has idled out takes minutes to come back, because the weights have to be read and the CUDA
graphs captured again. [Cold starts](https://levers9.mintlify.site/sdk/cold-starts.md) covers what a caller waits for and the three ways to
change it - scaling to zero, checkpointing, or keeping a replica up.

`VLLMArgs` mirrors vLLM's own flags: `model`, `max_model_len`, `dtype`, `quantization`, `gpu_memory_utilization`,
`trust_remote_code`, `enable_lora` and `lora_modules` (to serve an adapter trained in the
[RL Arena](https://levers9.mintlify.site/lab/train-and-evaluate.md#weights)), `tensor_parallel_size` with `gpu_count` for multi-GPU.

## From the CLI

`levers9 deploy --llm` deploys a model without a Python file:

```bash theme={null}
levers9 deploy --llm --llm-model-id Qwen/Qwen3-8B --llm-engine vllm --llm-context-length 16384 --name qwen --gpu L40S
```

Engines: `vllm`, `sglang`, `tensorrt-llm`. The controlplane routes and meters these as OpenAI-compatible services.

## From the dashboard

**Endpoints** in the dashboard is the same thing with a catalog: featured open-weights models verified against
the Hub, a search box, a suggested GPU from the parameter count, and deploy, stop and start buttons. See
[Endpoints](https://levers9.mintlify.site/dashboard/endpoints.md).

> [!NOTE]
>
> Scaling: `LLMTokenPressureAutoscaler` scales on token pressure instead of queue depth, which suits chat traffic
> better than request counts.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.