LLM serving

VLLM is an endpoint that runs vLLM for you: pick a model from the Hugging Face Hub, a GPU, and deploy. The URL speaks the OpenAI API, so any client that takes a base_url works unchanged.

app.py

from levers9 import Blueprint, QueueDepthAutoscaler
from levers9.integrations import VLLM, VLLMArgs

llm = VLLM(
    name="gemma",
    gpu="L40S", cpu=4, memory="24Gi",
    vllm_args=VLLMArgs(model="google/gemma-4-12B-it", max_model_len=8192),
    secrets=["HF_TOKEN"],                      # for gated repos
    keep_warm_seconds=300,                     # idles out after 5 min; see Cold starts
    autoscaler=QueueDepthAutoscaler(max_containers=2),
)

levers9 deploy app.py:llm --name gemma

from openai import OpenAI
client = OpenAI(base_url="https://levers9.uselevers.com/gemma/v1", api_key=TOKEN)
print(client.chat.completions.create(model="google/gemma-4-12B-it", messages=[{"role": "user", "content": "hi"}]).choices[0].message.content)

An LLM that has idled out takes minutes to come back, because the weights have to be read and the CUDA graphs captured again. Cold starts covers what a caller waits for and the three ways to change it - scaling to zero, checkpointing, or keeping a replica up. VLLMArgs mirrors vLLM’s own flags: model, max_model_len, dtype, quantization, gpu_memory_utilization, trust_remote_code, enable_lora and lora_modules (to serve an adapter trained in the RL Arena), tensor_parallel_size with gpu_count for multi-GPU.

From the CLI

levers9 deploy --llm deploys a model without a Python file:

levers9 deploy --llm --llm-model-id Qwen/Qwen3-8B --llm-engine vllm --llm-context-length 16384 --name qwen --gpu L40S

Engines: vllm, sglang, tensorrt-llm. The controlplane routes and meters these as OpenAI-compatible services.

From the dashboard

Endpoints in the dashboard is the same thing with a catalog: featured open-weights models verified against the Hub, a search box, a suggested GPU from the parameter count, and deploy, stop and start buttons. See Endpoints.

Scaling: LLMTokenPressureAutoscaler scales on token pressure instead of queue depth, which suits chat traffic better than request counts.

Get started

Build with the SDK

Dashboard

Data Lab

RL Arena

Operate

From the CLI

From the dashboard

​From the CLI

​From the dashboard

From the CLI

From the dashboard