Migrate from the native bindings

note

The legacy.yaml example comes from an earlier release and keeps its model fields, such as a module model_id and a binding contract and adapter. vllm-sr config migrate rewrites those as well; Migrate model resources shows each earlier field next to the deployment-and-binding form that replaces it.

Earlier releases ran models inside the router with three back ends: candle, ONNX Runtime (ort) and OpenVINO. Those in-process back ends are gone. Local routing models now run in the model runtime, and remote ones are System One APIs or embedding services. vllm-sr serve --platform auto detects the host or target; use --platform cpu to select the CPU image explicitly. The release note lists every breaking change of that release.

Review defaults as well as field migrations. Every routing signal now asks the decision model, vela3-fast unless you choose another. A historical configuration that relied on Vela 1.0 classifiers therefore changes answers even when it migrates cleanly. Review model and threshold choices and test representative requests before rollout. Migrate to v0.5 covers the configuration changes of the current release.

You need to migrate if your configuration contains any of these:

• provider: candle, provider: ort or provider: openvino;
• precision, custom_ops_profile or compilation_cache_dir on a deployment;
• variant, model_type, use_modernbert or use_mmbert_32k on a module;
• a binding head that names an ONNX or OpenVINO graph file;
• an older model name such as models/mom-domain-classifier, models/mmbert-embed-32k-2d-matryoshka or lettucedect;
• the NLI explainer (hallucination_explainer, nli_model, enable_nli_filtering, use_nli) or the response cache's polarity_guard setting;
• a hallucination detector with backend: endpoint;
• profile on a model runtime deployment (see Engine arguments).

The router refuses these settings at startup and tells you to run the migration command.

1. Run the migration

vllm-sr config migrate --config legacy.yaml --output legacy.migrated.yaml

The command writes the migrated file and prints every change it made, for example:

Changes to review
  global.model_catalog.deployments.vela-embedding
      provider ort is removed: the built-in model runtime serves every local model; rewritten as provider: model_runtime, on its device, with its artifact's Hub repository (2 places)
  global.model_catalog.deployments.vela-domain.precision
      the model runs in its published precision (dtype: auto); dtype: float16 runs it in half precision on a GPU

It prints a Warning for anything you still have to do yourself: a relative path to a model of your own that it could not rewrite safely (the migrated file keeps that value, and the router rejects it until you fix it), or stored vectors to re-embed because their embedding model changed (see Re-embed when the embedding model changes).

2. Review the changes

Before	After
provider: candle, ort or openvino	provider: model_runtime
device: cpu	device: cpu
device: cuda:N	device: cuda:N
device: rocm:N or migraphx:N	device: rocm:N
device: metal:0	device: mps
an OpenVINO device (CPU, GPU, NPU, ...)	device: cpu; Intel GPUs can use device: xpu:0 (not yet validated)
precision: fp16	removed: the model runs in its published precision; dtype: float16 runs it in half precision on a GPU (see Engine arguments)
precision: native or fp32	removed: the default dtype: auto already runs the published precision, FP32 for these models
custom_ops_profile, compilation_cache_dir	removed: the runtime picks its own kernels
a graph head such as onnx/model_fa.onnx	removed: the runtime picks the model's graphs
artifact: models/Vela-1.0-Encoder-307M-...	artifact: vllm-sr/Vela-1.0-Encoder-307M-..., the Hub repository
embeddings.semantic, with its embedding_config.backend: candle or openvino and its gemma_model_path or bert_model_path	removed: the runtime has no EmbeddingGemma or MiniLM family; embeddings.default names the embedding deployment, Vela Embedding unless you change it (see Embeddings)
embedding_model: bert or gemma on the response cache, memory or vector store	removed: the store embeds with embeddings.default; re-embed stored vectors
a vector size Vela Embedding does not serve (such as MiniLM's 384) on a store that embeds with it	kept: set 256 for memory, 768 for the response cache and the vector store, and re-create the collection or index at that size
global.router.model_selection.ml, with the trained knn, kmeans, svm and mlp selectors	removed: the systemone selector, which needs no trained model, replaces them
variant, model_type, use_modernbert, use_mmbert_32k on a module	removed: the runtime reads the architecture from the model
a label map inside an older model's directory, such as category_mapping_path: models/mom-domain-classifier/category_mapping.json	removed: the router reads the labels of the model it runs
grounding.nli_contradiction_penalty of the fusion algorithm	grounding.contradiction_penalty: grounding now reads the hallucination question
a hallucination detector with backend: endpoint, endpoint and model_id	removed: the hallucination signal asks a span-capable System One deployment

Model names that changed

The Router no longer runs task classifiers: a domain, PII, jailbreak, fact-check, feedback, modality or hallucination model of an earlier release, whether an older name such as mom-domain-classifier or a Vela 1.0 task model, is replaced by the decision model, which asks that signal's question. Its labels are the ones your rules already use. Embedding models are replaced as follows:

Old model	New model	What changes
mmbert-embed-32k-2d-matryoshka	Vela-1.0-Encoder-307M-Embedding	Re-embed stored vectors
EmbeddingGemma (mom-embedding-flash), MiniLM (mom-embedding-light)	Vela-1.0-Encoder-307M-Embedding, or an OpenAI-compatible embedding endpoint	Re-embed stored vectors
Qwen3-Embedding-0.6B (mom-embedding-pro)	unchanged

Features that were retired

• The NLI explainer (hallucination_explainer, enable_nli_filtering, include_explanation on a local detector, use_nli). Hallucination checks still mark the unsupported spans of an answer; they no longer add an NLI verdict per span.
• The response cache's polarity_guard setting. Its NLI tier is gone, so there is nothing left to choose: the lexical guard, which catches negations and antonyms, always runs. The migration removes the block.
• OpenVINO. Intel CPUs run models on cpu; Intel GPUs can use the xpu device.
• The ONNX Runtime MIGraphX and CK flash-attention paths. AMD GPUs run models through PyTorch for ROCm (device: rocm:N).

3. Validate and start

vllm-sr config validate --config legacy.migrated.yaml
vllm-sr serve --config legacy.migrated.yaml

For an existing local Docker stack, the CLI may report that it preserved the saved active configuration. Review and merge any Dashboard edits you need into legacy.migrated.yaml, then rerun with --replace-active-config to apply that file. If a Recipe package is active, change or deactivate it through the Recipe workflow first.

On the first start the runtime downloads the models it does not have yet. Run it with the router shows how to see when every deployment is ready.

A complete example

This configuration uses ONNX Runtime on an AMD GPU for embeddings, candle on an NVIDIA GPU for a domain classifier, and the response cache's NLI polarity guard:

legacy.yaml

version: v0.5
listeners: []
providers:
  defaults:
    model: answer-model
  models:
    - name: answer-model
      backend_refs:
        - name: answer
          endpoint: vllm:8000
          provider: vllm
recipes:
  - name: default
    model_bindings:
      embedding:
        deployment: vela-embedding
        contract: embedding.v1
        adapter: mmbert
        head: onnx/model_fa.onnx
global:
  model_catalog:
    deployments:
      vela-embedding:
        artifact: models/Vela-1.0-Encoder-307M-Embedding
        provider: ort
        device: rocm:0
        precision: native
        custom_ops_profile: ck_flash_attention
      vela-domain:
        artifact: models/Vela-1.0-Encoder-307M-Domain
        provider: candle
        device: cuda:0
        precision: fp16
  stores:
    response_cache:
      backend_type: memory
      polarity_guard:
        mode: lexical+nli
  plugins:
    - type: response_cache
      configuration:
        enabled: true
        scope: global

vllm-sr config migrate --config legacy.yaml writes:

legacy.migrated.yaml

version: v0.5
listeners: []
providers:
  defaults:
    model: answer-model
  models:
    - name: answer-model
      backend_refs:
        - name: answer
          endpoint: vllm:8000
          provider: vllm
recipes:
  - name: default
global:
  model_catalog:
    deployments:
      vela-domain:
        artifact: vllm-sr/Vela-1.0-Encoder-307M-Domain
        provider: model_runtime
        device: cuda:0
      vela-embedding:
        artifact: vllm-sr/Vela-1.0-Encoder-307M-Embedding
        provider: model_runtime
        device: rocm:0
  stores:
    response_cache:
      backend_type: memory
  plugins:
    - type: response_cache
      configuration:
        enabled: true
        scope: global

The embedding binding is dropped: the response cache embeds with embeddings.default, the vela-embedding deployment the file declares, now served by the model runtime on the AMD GPU. Running the command again on the migrated file changes nothing.

Engine arguments

The model runtime used to choose its batching and numerics with a profile: exact (the default), batching, shared_context or max_speed. It now batches concurrent requests by default, as vLLM does, and takes vLLM-style engine arguments instead. The router refuses a deployment that still sets profile, and vllm-sr config migrate rewrites it:

Before	After
profile: exact	removed. Concurrent requests now share forwards, so an answer can move by rounding; no engine argument restores one request per forward
profile: batching	removed: batching is the default
profile: shared_context	enable_shared_context: true; rows of other requests now also share forwards
profile: max_speed	enable_shared_context: true; its batching and faster kernels are the defaults
any other profile	removed, with a warning: set the engine arguments it stood for

An engine argument the deployment already sets is kept. The flags and request options changed with them; a removed flag fails with the one that replaces it:

Before	After
vllm-sr serve --runtime-profile P	engine flags such as --max-num-seqs and --dtype
vllm-srun serve --profile P	the same engine flags
vllm-srun serve --max-batch-tokens N	--max-num-batched-tokens N
"options": {"profile": "exact"} on a request	refused (invalid_request): a request runs with its deployment's engine arguments
profile in GET /v1/models and in a response's meta	engine_args, dtype and numerics (exact or approximate)

Clients of the public System One API see two of these changes. A request that sets options.profile returns 400 (invalid_request) from the Router, and the meta of a return_meta answer reports dtype and numerics instead of profile. Drop the option, and read numerics where you read profile. See System One API stability.

Re-embed when the embedding model changes

Changing the embedding model changes the vector space. The router keeps vectors of different models apart, so nothing breaks, but stored data does not carry over:

• Semantic cache: entries from the old model are no longer found; the cache fills again with new traffic.
• Memory: memories stored with the old model are not returned. Re-add the memories you need to keep.
• Vector stores and RAG: re-index your documents with the new model. Milvus, Qdrant and hybrid RAG backends embed the query with Vela Embedding (768 dimensions) where they used MiniLM (384), so re-embed their collections with Vela Embedding, or name the deployment that indexed a collection in its source's backend_config.embedding. Check similarity thresholds on your own traffic; the scores of a different model are not comparable.

Kubernetes and Docker

Router images no longer contain native libraries. They contain the CPU runtime, so managed models work out of the box. Remove any environment variables, init containers or volumes that existed only for candle, ONNX Runtime or OpenVINO. For GPUs, run a GPU runtime and attach to it; see Run it with the router.

If you deploy with the operator, remove embedding_models from your SemanticRouter resources before you upgrade. The operator refuses a resource that still sets it; its router embeds with embeddings.default, Vela Embedding unless the resource's configuration names another deployment. Re-embed the vectors that EmbeddingGemma stored (see Re-embed when the embedding model changes).