Skip to main content
Version: Latest (main)

Model runtime

The built-in model runtime serves the models the Router consults about each request: the decision model, which answers every System One question, and the embedding model that recall features use. You don't install it separately: it ships inside the Router images, and the Router starts and supervises it for you.

Serve a decision model and ask it directly
vllm-sr serve vllm-sr/vela3-fast --engine --platform cpu
curl -s localhost:8899/v1/systemone/models
note

The runtime is not where your chat models run. The models that answer your users stay behind your providers (vLLM, Ollama, a hosted API); the runtime serves the small models the Router asks about each request.

Frontend, optional decision engine, and on-demand model runtime

Three ways to use it​

You want toDo thisRead
Use the Router's built-in featuresNothing extra. The Router starts the runtime for you; --platform rocm or --platform cuda selects a GPU-capable imageConnect the Router
Share the models between Routers, or run them on another machineStart a runtime yourself and point the Router at it with endpointAttach a runtime you run
Call the models from your own codeRun vllm-sr serve ARTIFACT --engine and send HTTP requestsQuickstart

Managed workers run as vllm-srun processes inside the Router container. Starting vllm-sr serve ARTIFACT --engine keeps the same instance frontend and model management, with Chat routing disabled. Router mode can serve native System One requests at the same time as routed Chat requests. Component Architecture covers the request paths.

What it can serve​

TaskBuilt-in modelsGuide
Answer every routing judgment: the application signals, your own questions, the System One selector, guards and judgesVela 3.0 (vela3-fast by default), Decision 3.0, Decision 2.0 and 1.0, Vela 2.0Supported models
Recall for the response cache, memory and RAGVela 1.0 Embedding, Qwen3-Embedding-0.6BEmbeddings

Choose a model helps you pick a size and hardware.

How it starts and fails​

  • Only what you use loads. Built-in features resolve their model defaults automatically (Choose a model for a signal). The Router starts managed workers only for deployments that something resolves to, and for explicitly published native models. An unused deployment loads no weights.
  • Startup waits for required models. Attached model readiness follows the readiness rules.
  • A slow model never stalls a request. The signal deadline bounds how long a request waits; an unfinished signal gives no answer and resolves through its on_error or on_unknown policy.

What you can rely on​

  • Pinned and verified. Built-in models are pinned to an exact Hugging Face revision. Every file is checked against its recorded SHA-256 before it is loaded, and code shipped inside a model repository is never run.
  • Throughput first, at the published precision. Like vLLM, the runtime batches concurrent requests by default and runs each model in the precision it was published in. A batched answer can move by rounding, which the records measure per model; a faster setting whose records don't show unchanged accuracy is an explicit switch. See Engine arguments.
  • Bounded waits. Slow or unavailable models resolve through the signals' deadline and error policies. A model forward already running may continue after the caller times out and delay queued work. The Router restarts a crashed runtime.
  • Batched calls. Compatible model work from the same routing stage can travel together in one API call. That call may require several model forward passes, and later stages can make additional calls. Independent workers can answer in parallel when hardware capacity allows.
  • Pluggable. New model families, engines and hardware back ends are ordinary Python packages. See Add your own model family.

Hardware​

HardwareStatus
CPUValidated. Every Router image runs models on the CPU
AMD GPUs (MI300X, MI325X)Validated. vllm-sr serve --platform rocm
NVIDIA GPUsValidated. vllm-sr serve --platform cuda
Intel GPUs (xpu), Ascend NPUs (npu), Apple GPUs (mps)Available, not yet validated

Choose a model has the details, and Quantization the reduced-precision packages each GPU runs.

Coming from an older release?

The candle, ONNX Runtime and OpenVINO back ends are gone. Run vllm-sr config migrate to update your configuration; see Migrate from the native bindings.