> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Endpoints

> Serve an open-weights model from a catalog, in one click, behind an OpenAI-compatible URL

**Endpoints** is managed inference. Pick a model, pick a GPU, deploy; the dashboard runs it with vLLM and gives
you a URL that speaks the OpenAI API.

## The catalog

* **Featured** models are open-weights checkpoints verified live against the Hugging Face Hub (Gemma, Qwen,
  Llama, Mistral and others), grouped by size, with a suggested GPU worked out from the parameter count.
* **Search** finds any transformers repo on the Hub; quantised-only formats (GGUF, MLX) are hidden because vLLM
  cannot serve them.
* Gated repos need a Hugging Face token: store it as a workspace secret (the Hugging Face template names it `HF_TOKEN`) and pick that secret when you deploy.

## Create endpoint

**Create endpoint** (or **Deploy** on a catalog card) opens the journey:

1. **Endpoint type**: dedicated, on your GPUs, billed by GPU second while it runs and scaled to zero when idle.
   (A shared, per-token endpoint is not something this cluster offers.)
2. **Weights**: **Open weights** picks a model from the catalog or a Hub search. **Custom weights** brings your
   own: a **Hugging Face** repo (with a revision and a token secret for private repos) or a **levers9 volume**
   folder, browsed in place so you see `config.json` before deploying. Say whether the weights are a **full
   model** or a **LoRA adapter on a base model**; for an adapter you pick the base model and vLLM serves the
   base with the adapter applied under the endpoint's name. That is how a version trained in the
   [RL Arena](https://levers9.mintlify.site/lab/train-and-evaluate.md#weights) becomes an endpoint.
3. **Endpoint name**, checked for availability as you type; it becomes the URL and the default served model
   name that clients pass as `model=`.
4. **Resources**: the GPU (suggested from the parameter count and the pools this cluster has), count, host CPU
   and memory, max context, keep-warm, served model name, token secret, trust remote code.

The **Fit preview** on the right estimates the weights in bf16 plus a third for the KV cache against the VRAM
of the chosen GPUs, says whether a pool with that GPU exists here, and shows the client snippet. **Deploy
endpoint** in the bottom bar registers it; the endpoint appears with its state, URL, and stop / start buttons.
The first start downloads the weights, so it takes minutes; later starts are faster since the weights are cached
on the volume.

Upload custom weights with `levers9 cp ./my-finetune levers9://models/my-finetune` (add `--no-multipart` from a
laptop outside the cluster), then pick the `models` volume and the `my-finetune` folder.

```python theme={null}
from openai import OpenAI
client = OpenAI(base_url="<endpoint url>/v1", api_key="<workspace token>")
client.chat.completions.create(model="<model id>", messages=[{"role": "user", "content": "hi"}])
```

> [!NOTE]
>
> A cluster with no GPU pool (a laptop k3d) accepts the deploy and leaves it pending; the page says so instead
> of pretending.

The same thing from Python or the CLI is on [LLM serving](https://levers9.mintlify.site/sdk/llm-serving.md). Endpoints trained in the RL Arena
(a LoRA adapter served with `enable_lora`) can be registered as policies there.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.