The Feature Everyone Wants and Almost Nobody Has​

Cursor's Auto mode is deceptively simple: the developer types, and the IDE chooses whether a prompt deserves a frontier model or something faster and cheaper. It is easy to stop noticing — until moving to an open tool where every request starts with a model dropdown.

That gap matters more than it sounds. Teams everywhere are standing up local and internal model serving — a fine-tuned coder on owned GPUs, a frontier API for hard problems, a fast cheap model for everything else. The models exist. The serving works. What is often missing is the decision layer: something that reads each request and sends it to the right backend automatically.

OpenCode ships without a built-in Auto mode, but it does expose an open provider interface — enough to bolt intelligent routing on behind a single OpenAI-compatible endpoint.

This guide walks through building Auto mode for OpenCode (or any OpenAI-compatible client) with vLLM Semantic Router and AgentGateway: one endpoint, one model name, and an ML router choosing among the fleet per request. The configs below are complete; the failure modes are the ones that typically surface first in production setups.

[Demo video: three prompts from one OpenCode session, routed live to three different models]

First, Know Your Options: The Routing Algorithms in vLLM Semantic Router​

"Auto mode" means different things to different teams. vLLM Semantic Router ships more than a dozen selection algorithms, each answering a different operational question:

Decisions can also be gated by signal rules — classifiers for intent, PII, and jailbreak detection. "Anything containing PII stays on the on-prem model" is enforceable as routing policy, not just documentation. When the primary need is governance rather than optimization, signal rules are the right layer.

Practical rule of thumb: interchangeable replicas → multi_factor or latency_aware. Specialist pools (local coder + frontier API) → prompt-aware routing with router_dc. The rest of this guide uses router_dc.

The Design: One Virtual Model Called MoM​

The Semantic Router's core abstraction is a decision: a named bundle of candidate models, a selection algorithm, and a virtual model name the client calls. A typical Auto-mode setup exposes MoM — Mixture of Models:

decisions:
  - name: MoM
    description: "Mixture of Models router"
    priority: 100
    rules:
      operator: AND        # empty AND = catch-all (see gotcha #1 below!)
    modelRefs:
      - model: qwen-coder
      - model: gpt-4o
      - model: gemini-flash
    algorithm:
      type: router_dc

With router_dc, routing rules about prompt keywords are unnecessary. Instead, configure plain-English descriptions of what each model is good at:

modelCards:
  - name: qwen-coder
    description: >
      Specialized coding model optimized for programming tasks.
      Excellent at writing code, debugging, algorithms, ...

  - name: gpt-4o
    description: >
      Frontier reasoning model with exceptional analytical capability.
      Best for complex multi-step reasoning, strategic analysis, ...

  - name: gemini-flash
    description: >
      Fast general-purpose model. Ideal for simple factual questions,
      quick lookups, summarization, casual conversation, ...

At request time, an embedded mmBERT model (CPU, ~130MB) embeds the incoming prompt and compares it to those descriptions by cosine similarity. Example routing logs:

[RouterDC]   qwen-coder:   similarity=0.9998   ← "Write me a Python function..."
[RouterDC]   gpt-4o:       similarity=1.0000   ← "Compare utilitarian and deontological ethics..."
[RouterDC]   gemini-flash: similarity=0.9939   ← "What is the capital of Japan?"

Routing decision cost is typically 1–18ms on CPU. The descriptions are the routing policy — adding a fourth model (a SQL specialist, a legal model) is a new model card, not new application code.

Why this pattern qualifies as true Auto mode:

1. The client stays simple. OpenCode holds no routing logic and no upstream API keys. Swap models, retune descriptions, add candidates — the client config stays fixed.
2. Cost control is structural. In a coding agent, code-shaped traffic lands on the $0 local model; only prompts that genuinely need frontier reasoning incur frontier pricing.
3. It fails soft. With gateway policy failureMode: failOpen, a dead router process lets traffic fall through to a default route. Users see answers, not hard outages.

The Architecture​

The Semantic Router runs as an Envoy ExtProc sidecar to AgentGateway — no extra proxy hop. The gateway pauses each request, streams the body to the router over gRPC, receives a header mutation, and resumes. Routes match on that header:

binds:
- port: 3000
  listeners:
  - routes:
    - matches:
      - headers:
        - name: x-selected-model
          value:
            exact: qwen-coder
      backends:
      - ai:
          provider:
            openAI: {}
          name: ollama
          hostOverride: localhost:11434
    # ... gpt-4o → OpenAI (with backendAuth),
    #     gemini-flash → Gemini (with backendAuth),
    #     plus a failOpen fallback route

A critical security split: the router never holds API keys. It classifies and sets a header; AgentGateway owns backendAuth and injects credentials per upstream. The component making ML decisions on untrusted input should hold zero secrets — especially when the endpoint serves a team rather than a single laptop.

Wiring OpenCode: The Whole Integration Is One Config Block​

OpenCode accepts any OpenAI-compatible endpoint as a custom provider. In ~/.config/opencode/opencode.jsonc:

{
  "provider": {
    "auto_sr": {
      "name": "Auto (Semantic Router)",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:3000/v1"
      },
      "models": {
        "MoM": {
          "name": "MoM",
          "limit": { "context": 32768, "output": 8192 }
        }
      }
    }
  }
}

The Semantic Router intercepts /v1/models and advertises the virtual model, so OpenCode's discovery finds MoM as if it were a real backend. Select the provider, pick MoM, and every prompt is classified server-side before it reaches an upstream LLM.

For observability, AgentGateway ships a built-in UI (build with --features ui, served at :15000/ui) and can persist every request to SQLite with cost attribution:

config:
  modelCatalog:
  - file: base-costs.json
  database:
    url: sqlite://agentgateway.db

Enable request persistence early — it pays off the first time routing behavior needs debugging.

Common Pitfalls​

Four issues show up repeatedly when wiring this stack. Each applies broadly beyond OpenCode.

Pitfall 1: The catch-all rule that matches nothing​

Older examples use rules: {} as "match everything." On current main, an empty rules block evaluates as an OR over zero conditions — which matches nothing. Requests silently fall back to the default model: no errors, plausible answers, zero actual routing. The tell in logs is "decision":"".

The documented catch-all is an empty AND:

rules:
  operator: AND

If the router "works" but never routes, verify the decision matches at all before tuning the algorithm.

Pitfall 2: Choosing an algorithm that ignores the prompt​

multi_factor scores configured quality, pricing, and live latency signals. It is a fleet-balancing algorithm; the prompt never enters the equation. Without quality or pricing data configured, every factor is neutral and traffic sticks on the first candidate indefinitely.

For specialist pools (coder + reasoner + generalist), use router_dc. Reserve multi_factor for interchangeable replicas where cost, latency, and load are the primary tradeoffs.

Pitfall 3: A virtual model inherits its weakest backend's limits​

A typical first failure after routing to a cloud model:

max_tokens is too large: 32000. This model supports at most 16384
completion tokens, whereas you provided 32000.

OpenCode cannot see individual backends behind MoM, so it may request a generous token budget. The virtual model's advertised limits must be the intersection across all candidates — smallest context window, safest output ceiling:

"limit": { "context": 32768, "output": 8192 }

The same intersection rule applies to tool calling, vision, and every other capability flag. Configure for the least capable backend, not the average one.

Pitfall 4: Agentic clients + small local models = malformed tool JSON​

OpenCode is an agent: every request carries a large system prompt and a tool catalog (write, edit, webfetch, ...). Frontier models return structured tool calls. Smaller local models often emit a broken imitation as plain text:

> what is python language

{"name":"write","arguments":{"content":"Python is a high-level,
interpreted programming language...","filePath":"python_language.md"}}

OpenCode replays history on every request, so malformed JSON from earlier turns becomes part of the context and the session self-reinforces the wrong format. Teams running compact local models alongside agentic clients should expect some variant of this behavior.

Diagnosis requires payload capture. AgentGateway can log the full messages array:

frontendPolicies:
  http:
    accessLog:
      database:
        add:
          gen_ai.prompt: llm.prompt
          gen_ai.completion: 'llm.completion.map(c, {"role":"assistant", "content": c})'

A single SQL query against the access log reveals the exact prompt the model received — agent preamble, poisoned history, and all. Debugging LLM pipelines without payload capture is guesswork.

Mitigation: define a custom OpenCode agent whose prompt replaces the built-in agentic system prompt:

"agent": {
  "chat": {
    "description": "Plain chat through the semantic router",
    "mode": "primary",
    "model": "auto_sr/MoM",
    "prompt": "You are a helpful assistant. Answer directly and concisely in clean markdown. Use fenced code blocks for any code. Never output JSON tool calls.",
    "permission": { "edit": "deny", "bash": "deny", "webfetch": "deny" }
  }
}

Also set "tool_call": false on the model entry. Switch to the chat agent in the OpenCode TUI (Tab), start a fresh session (poisoned history does not self-heal), and models in the pool respond in clean markdown. Keep the default build agent for full agentic work with backends that handle tool calls reliably.

Expected Results​

Three prompts, one OpenCode session, one provider, one model name:

Routing overhead stays in the 1–18ms range on CPU. The gateway UI surfaces per-hop tokens and cost; the SQLite log retains the payload trail for post-incident review.

The outcome that matters: no model dropdown, no accidental frontier spend on trivial prompts, and no routing logic in the client — only English model descriptions and an embedding model that routes on semantic fit.

Takeaways​

• Auto mode is a gateway feature, not a client feature. Build it once behind an OpenAI-compatible endpoint and every compatible tool inherits it.
• Match the algorithm to the business need. Specialist models → router_dc. Interchangeable replicas → multi_factor / latency_aware. Maximum savings with escalation → automix. Governance → signal rules (PII, jailbreak).
• A virtual model inherits the weakest backend's limits. Context, output, tool calling — intersection, not average.
• Payload logging is non-negotiable. Silent misroutes and poisoned agent sessions are invisible without the actual bytes on the wire.
• Keep keys out of the router. The component reading untrusted prompts should hold zero secrets; let the gateway own auth.

Teams standing up internal model serving and wanting Cursor-style Auto mode can get there with two YAML files and a JSON block, using AgentGateway and vLLM Semantic Router.

Get Started​

• Install with agentgateway — Kubernetes integration guide
• Router DC selection algorithm — prompt-to-model-description routing
• Giving AgentGateway a Semantic Brain — homelab walkthrough with ExtProc routing
• Ollama local setup — run local models behind the gateway

Published for issue #2424. Have a community story about Semantic Router? Open a blog issue and the team will help get it published.

Agent systems that span multiple models — a local endpoint for coding, a frontier cloud model for deep reasoning, and a fast general-purpose model for everyday tasks — all face the same routing question: how should each request be directed to the right backend?

Many deployments start with a lightweight Python proxy or keyword matcher in front of the gateway. That approach works at small scale, but misroutes grow quickly as traffic, languages, and task types diversify. This post shows how vLLM Semantic Router running as an Envoy ExtProc sidecar inside AgentGateway replaces that pattern with semantic, config-driven routing.

The Problem: Keyword Routing Does Not Scale​

A typical multi-model agent gateway fronts three backends:

The routing layer was a simple keyword matcher:

# router.py — keyword-based routing
coding_keywords = ["code", "python", "javascript", "bash", "script",
                   "function", "bug", "error", "html", "css"]
reasoning_keywords = ["think", "analyze", "explain in detail",
                      "reasoning", "logic", "deduce"]

if any(k in prompt_lower for k in coding_keywords):
    intent = "coding"
elif len(prompt) > 400 or any(k in prompt_lower for k in reasoning_keywords):
    intent = "reasoning"
else:
    intent = "simple"

After two weeks of sustained traffic, the rough numbers looked like this:

Eighteen percent misroutes is not just wasted spend — it produces worse answers. Scheduled agent jobs that sent "summarize this week's calendar and suggest optimizations" to the 7B local model instead of Gemini or GPT-4o returned noticeably weaker output. Mixed-language prompts and unanticipated domains silently fell through to the wrong lane.

The architecture needed a routing layer that understood the prompt, not just scanned it for keywords.

Enter vLLM Semantic Router + AgentGateway​

AgentGateway maintainers Keith Mattix and John Howard helped shape first-class ExtProc integration with vLLM Semantic Router. The resulting architecture is straightforward.

Instead of a Python reverse proxy sitting in front of the gateway, Semantic Router runs as an Envoy ExtProc sidecar. AgentGateway pauses the request, sends the HTTP body to the SR gRPC endpoint, receives a header mutation (x-selected-model: qwen-coder), and resumes routing. Zero proxy hops. Zero Python processes. Just gRPC-native intelligence inside the gateway's own request lifecycle.

Semantic Router uses an embedded mmBERT model (a 2D Matryoshka embedding model, ~130MB) to classify every prompt and compare it against model descriptions defined in YAML. No keyword lists. No regex. Actual embeddings.

For Kubernetes deployments, the same pattern is documented in the Install with agentgateway guide.

The Setup: Two YAML Files, No Code​

The full integration is defined in two config files.

1. Semantic Router config (config.yaml)​

This tells Semantic Router about the models and how to route between them:

version: v0.3

providers:
  defaults:
    default_model: qwen-coder
  models:
    - name: qwen-coder
      provider_model_id: qwen2.5-coder:7b
      api_format: openai
      backend_refs:
        - name: local-ollama
          endpoint: host.docker.internal:11434
          protocol: http

    - name: gpt-4o
      provider_model_id: gpt-4o
      api_format: openai
      backend_refs:
        - name: openai-cloud
          base_url: https://api.openai.com/v1

    - name: gemini-flash
      provider_model_id: gemini-2.5-flash
      api_format: openai
      backend_refs:
        - name: gemini-cloud
          base_url: https://generativelanguage.googleapis.com/v1beta/openai

routing:
  modelCards:
    - name: qwen-coder
      param_size: 7B
      context_window_size: 32768
      description: >
        Specialized coding model optimized for programming tasks.
        Excellent at writing code, debugging, algorithms, data structures,
        code review, refactoring, and technical implementation in Python,
        Rust, JavaScript, Go. Best for code generation, fixing bugs,
        writing tests, and technical programming Q&A.

    - name: gpt-4o
      param_size: 200B+
      context_window_size: 128000
      description: >
        Frontier reasoning model with exceptional analytical capability.
        Best for complex multi-step reasoning, strategic analysis,
        comparing trade-offs, writing long-form essays, nuanced
        explanations, math proofs, scientific reasoning.

    - name: gemini-flash
      param_size: ~100B
      context_window_size: 1000000
      description: >
        Fast general-purpose model. Ideal for simple factual questions,
        quick lookups, summarization, casual conversation, translations,
        everyday tasks, and when speed matters more than depth.

  decisions:
    - name: MoM
      description: "Mixture of Models router"
      priority: 100
      rules: {}
      modelRefs:
        - model: qwen-coder
        - model: gpt-4o
        - model: gemini-flash
      algorithm:
        type: multi_factor
        multi_factor:
          weights:
            quality: 0.1
            latency: 0.4
            cost: 0.5
          slo:
            max_cost_per_1m: 0.5

The key insight: each model is described in natural language, and Semantic Router uses those descriptions as semantic anchors. When a new prompt arrives, the router embeds it and compares it against these descriptions using cosine similarity. The closest match wins.

2. AgentGateway config (agentgateway_config.yaml)​

This tells AgentGateway to call Semantic Router as an ExtProc sidecar and route based on the header it sets:

policies:
- name:
    name: semantic-router
    namespace: default
  target:
    gateway:
      gatewayName: default
  phase: gateway
  policy:
    extProc:
      host: "127.0.0.1:50051"
      processingOptions:
        requestBodyMode: buffered
        responseBodyMode: none
        requestHeaderMode: send
      failureMode: failOpen

binds:
- port: 3000
  listeners:
  - routes:
    - matches:
      - headers:
        - name: "x-selected-model"
          value:
            exact: "qwen-coder"
      backends:
      - ai:
          provider:
            openAI: {}
          name: ollama
          hostOverride: "localhost:11434"

    - matches:
      - headers:
        - name: "x-selected-model"
          value:
            exact: "gpt-4o"
      backends:
      - ai:
          provider:
            openAI: {}
          name: openai
        policies:
          backendAuth:
            key: $OPENAI_API_KEY

    - matches:
      - headers:
        - name: "x-selected-model"
          value:
            exact: "gemini-flash"
      backends:
      - ai:
          provider:
            gemini: {}
          name: gemini
        policies:
          backendAuth:
            key: $GEMINI_API_KEY

    - backends:
      - ai:
          provider:
            gemini: {}
          name: gemini-default
        policies:
          backendAuth:
            key: $GEMINI_API_KEY

Notice the separation of concerns: Semantic Router never touches API keys. It classifies the prompt and mutates a header. AgentGateway owns downstream auth. That is exactly how production gateways are designed — routing intelligence decoupled from security posture.

The failureMode: failOpen setting means if the SR container crashes or restarts, AgentGateway falls through to the default Gemini route. During SR container restarts, client requests still get answered without interruption.

ARM64 on Apple Silicon: Two Bugs, Two PRs​

On ARM64 hosts (including Apple Silicon), the SR container may start while embeddings remain unavailable:

{
  "msg": "embedding_models_init_completed",
  "embedding_ready": false,
  "tools_ready": false
}

The mmBERT model loaded but the embedding runtime never became ready. Every routing attempt logged:

Failed to embed model qwen-coder: failed to generate batched embedding (status: -1)

Bug #1: Wrong FFI dispatch (#2172)​

The Go router was calling candle_binding.GetEmbeddingBatched() for all model types — but the Rust FFI backend only supports batched embeddings for qwen3 architectures. For mmbert (the default), it returned status: -1.

The fix in PR #2192 adds a dispatch check:

// Only qwen3 supports the batched FFI. Others use single-text FFI.
func candleEmbeddingSupportsBatched(modelType string) bool {
    return modelType == "qwen3"
}

For non-qwen3 models, it gracefully falls back to GetEmbeddingWithModelType(), which works on ARM64.

Bug #2: Missing model files on first boot (#2173)​

On first boot, when the SR container downloaded mmBERT model files from HuggingFace, several required files (like tokenizer.json and config.json) were not being fetched. Fixed in PR #2195.

Both issues were triaged and fixed within days by the vLLM Semantic Router team — particularly @WUKUNTAI-0211 for the FFI dispatch fix and @theohsiung for the file completeness fix. On ARM64/Apple Silicon, pull the latest main and routing works as expected.

Proof: Real Routing Logs​

Send a coding prompt:

curl http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MoM",
    "messages": [
      {"role": "user", "content": "Write me a Python function to compute fibonacci numbers using memoization"}
    ]
  }'

Step 1 — Semantic Router classifies the prompt (1ms):

{
  "msg": "routing_decision",
  "original_model": "MoM",
  "selected_model": "qwen-coder",
  "reason_code": "auto_routing",
  "routing_latency_ms": 1,
  "component": "extproc"
}

Step 2 — AgentGateway routes to Ollama:

info  request
  gateway=default/default
  route=default/route0
  endpoint=localhost:11434
  http.status=200
  gen_ai.request.model=qwen2.5-coder:7b
  gen_ai.response.model=qwen2.5-coder:7b
  gen_ai.usage.input_tokens=41
  gen_ai.usage.output_tokens=366
  duration=22537ms

One millisecond of routing overhead. The rest is upstream generation time.

On container boot, the full pipeline appears in the logs:

{"msg":"embedding_models_init_started","mmbert_configured":true,"use_cpu":true}
{"msg":"embedding_models_initialized","use_batched":false}
{"msg":"selection_factory_initialized","selector_count":14}
{"msg":"startup_complete","embedding_ready":true,"sem_cache_enabled":true,
 "model_selection":true,"extproc_port":50051,"decisions":"MoM"}

Fourteen selection algorithms are registered out of the box — multi-factor, ELO, RL-driven, hybrid, latency-aware, session-aware, KNN, SVM, K-means, and more. The reference configuration uses multi_factor with cost-heavy weighting, but switching algorithms is a single YAML change.

Measured Results​

The cost savings come from fewer misroutes. When "explain the async/await pattern in Rust" correctly goes to local Ollama instead of GPT-4o, that is a $0.003 request instead of $0.03. Across hundreds of daily agent requests, the difference adds up quickly.

Why Agent Builders Should Care​

Whether the deployment runs on a single host or a production fleet in Kubernetes, agents benefit from a routing layer that understands prompts:

1. Cost control is the primary agent problem. Agents generate a lot of requests. Without intelligent routing, every request goes to the most expensive model. Semantic Router's multi_factor algorithm explicitly weighs cost, latency, and quality.

2. Keyword routing does not scale. The moment an agent handles a domain that was not anticipated, keyword-based routing silently fails.

3. AgentGateway + Semantic Router is production-grade. AgentGateway is a Gateway API data plane built in Rust. Semantic Router is an Envoy ExtProc server written in Go and Rust, backed by the vLLM project. This is the same architecture used in Kubernetes clusters with dozens of models.

4. Zero code maintenance. Once model descriptions are written, the routing config stays stable. Semantic Router learns from the descriptions, not from rules that require ongoing updates.

What's Next​

Common follow-on work for this architecture includes:

• Observability — wiring Jaeger and Prometheus to trace every request from client → AgentGateway → Semantic Router → upstream LLM. AgentGateway already emits OpenTelemetry-compatible spans.
• More models — adding specialized models (medical, legal) with just a new model card in YAML. Semantic Router figures out when to use them.

AgentGateway plus vLLM Semantic Router turns a keyword-based proxy into an ML-powered routing plane — with the same ExtProc pattern usable from a single-node setup to a multi-model Kubernetes deployment.

Get Started​

• Install with agentgateway — Kubernetes integration guide
• Gateway API Inference Extension with agentgateway — route to InferencePools with header-based selection
• Multi-factor selection algorithm — cost, latency, and quality weighting
• Community write-up on DEV — extended narrative by Anup Sharma

Have a community story about Semantic Router? Open an issue like #2257 and the team will help get it published.

A coding agent can begin with architecture work, call tools, receive short tool outputs, continue with "fix that", then ask a privacy-sensitive question in the same user session. The latest message may look simple, but the route cannot be chosen from the latest message alone. The router also has to know whether this is a safe moment to switch models.

This guide shows how to deploy that pattern on AMD ROCm with vLLM Semantic Router. You will start one ROCm vLLM backend, serve the agentic routing recipe, open the dashboard, validate the OpenAI-compatible API, and use Inferoa to experience route decisions and Router Learning behavior from an agent client.

Agentic routing is not only choosing a model. It is choosing when to keep one.

The reference recipe is deploy/recipes/agentic-saars.yaml. It is designed to be runnable on a single MI300X-class AMD ROCm host while exercising the same routing shape you would use with a heterogeneous fleet.

What You Will Deploy​

This guide follows the same deployment style as the earlier AMD Developer Cloud walkthrough: start the ROCm backend first, then run vLLM Semantic Router in front of it.

The fastest remote development setup is to expose the API and dashboard ports directly from the AMD host. If those ports are not open, use an SSH tunnel from your local machine:

ssh -L 8899:localhost:8899 -L 8700:localhost:8700 root@<host>

Then use http://localhost:8899/v1 and http://localhost:8700 locally.

What This Recipe Expresses​

The recipe has four route families plus a local fallback:

In the reference deployment, those paths are represented by logical model names such as qwen/qwen3.6-rocm, google/gemini-2.5-flash-lite, google/gemini-3.1-pro, openai/gpt5.4, and anthropic/claude-opus-4.6. For a single-card AMD guide, they can all point at the same vLLM backend. That is intentional: it lets you validate routing policy, headers, replay, and learning behavior without needing five physical model deployments.

The routing behavior still matters. If a request contains an API key, the privacy decision should route locally because the content is sensitive, not because the user remembered to write "keep this local." If a request moves from simple math to hard code review, the selected decision should change. If a tool loop is active, learning should be able to pin the current model even when the base semantic route would otherwise drift.

Why Router Learning Sits After Routing​

Semantic routing answers:

What route best matches this request?

Router Learning answers a second question:

Given the current session or conversation, should we accept that route change right now?

That separation is the key design point. The base decision remains semantic: simple, complex, privacy, and domain signals still choose candidate routes. Router Learning then applies operational memory: previous model, active tool loop, session or conversation identity, idle timeout, switch history, and prefix cache evidence.

Router Learning adjusts the selected route with continuity, reset, cache, and trace policy.

The compact response header surface is method-keyed so more adaptations can be added later:

x-vsr-learning-methods: adaptation,protection
x-vsr-learning-actions: adaptation=keep_base,protection=hold_current
x-vsr-learning-scopes: protection=conversation
x-vsr-learning-reasons: adaptation=base_best,protection=tool_or_protocol_state

Use those headers for live client display. Use x-vsr-replay-id for the full trace: base selected model, final selected model, matched decision, cache evidence, and the stay-vs-switch reasoning.

Conversation Protect vs Session Protect​

The recipe defaults to conversation-level protection:

global:
  router:
    learning:
      enabled: true
      adaptation:
        enabled: true
        strategy: routing_sampling
        candidate_set: decision
      protection:
        enabled: true
        scope: conversation
        identity:
          headers:
            session: x-session-id
            conversation: x-conversation-id

Use scope: conversation when one agent run should stay stable, but a later run in the same application session should be allowed to route again. In this guide, x-conversation-id maps to an agent run, while x-session-id maps to the broader user or client session.

For stricter products, change only the scope:

global:
  router:
    learning:
      protection:
        scope: session

With scope: session, the first selected model can be protected across multiple conversations until the idle timeout resets the session or a decision explicitly bypasses learning.

Conversation protect keeps one run stable. Session protect can hold a model across runs.

Privacy and security routes should normally bypass learning so policy remains hard:

routing:
  decisions:
    - name: local_privacy_policy
      modelRefs:
        - model: qwen/qwen3.6-rocm
      adaptations:
        mode: bypass

That gives the router a clean rule: learning can protect continuity, but it does not soften a privacy boundary.

Prepare The AMD Host​

Before starting containers, make sure the AMD host has:

• an AMD ROCm-capable GPU instance, such as a single MI300X-class machine
• Docker installed and the daemon running
• access to /dev/kfd and /dev/dri
• enough disk for the model cache
• a Hugging Face token if your environment needs one for model download
• ports 8090, 8899, and 8700 reachable, or an SSH tunnel for them

Install vLLM Semantic Router the same way as the AMD Developer Cloud guide:

python3.12 -m venv vsr
source vsr/bin/activate
curl -fsSL https://vllm-sr.ai/install.sh | bash

vllm-sr --help

Create the shared Docker network used by the recipe. The router container will reach the backend by the Docker name vllm:

sudo docker network create vllm-sr-network 2>/dev/null || true

Start vLLM on AMD ROCm​

Use the official vLLM ROCm OpenAI-compatible image. The current vLLM Docker documentation lists vllm/vllm-openai-rocm as the ROCm serving image.

sudo docker run -d \
  --name vllm \
  --network=vllm-sr-network \
  --restart unless-stopped \
  -p "${VLLM_PORT_QWEN36:-8090}:8000" \
  -v "${VLLM_HF_CACHE:-/mnt/data/huggingface-cache}:/root/.cache/huggingface" \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add=video \
  --ipc=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --shm-size 32G \
  -e HF_TOKEN="${HF_TOKEN:-}" \
  -e VLLM_ROCM_USE_AITER=1 \
  -e VLLM_USE_AITER_UNIFIED_ATTENTION=1 \
  -e VLLM_ROCM_USE_AITER_MHA=0 \
  --entrypoint python3 \
  vllm/vllm-openai-rocm:latest \
  -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3.6-35B-A3B \
    --host 0.0.0.0 \
    --port 8000 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --served-model-name qwen/qwen3.6-rocm google/gemini-2.5-flash-lite google/gemini-3.1-pro openai/gpt5.4 anthropic/claude-opus-4.6 \
    --trust-remote-code \
    --reasoning-parser qwen3 \
    --max-model-len 262144 \
    --language-model-only \
    --max-num-seqs 32 \
    --enable-prefix-caching \
    --enable-prompt-tokens-details \
    --prefix-caching-hash-algo sha256 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.90

The multiple --served-model-name aliases let one ROCm backend simulate a model portfolio. In a production fleet, those aliases can point at different local vLLM backends, private endpoints, or provider gateways.

Check the backend before starting the router:

curl -s "http://localhost:${VLLM_PORT_QWEN36:-8090}/v1/models" | jq .

sudo docker logs vllm 2>&1 | \
  grep -Ei 'prefix cache|kv cache|maximum concurrency|iteration' | tail -40

If your vLLM build does not populate usage.prompt_tokens_details.cached_tokens for OpenAI-compatible responses, keep --enable-prefix-caching enabled and validate cache behavior through /metrics instead:

curl -s "http://localhost:${VLLM_PORT_QWEN36:-8090}/metrics" \
  | grep -E 'prefix_cache|prompt_tokens_cached|kv_cache' \
  | head

Start vLLM Semantic Router With The Recipe​

Download the published recipe and serve it with the installed CLI:

curl -L -o agentic-saars.yaml \
  https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/recipes/agentic-saars.yaml

vllm-sr serve \
  --platform amd \
  --config agentic-saars.yaml

The recipe exposes the OpenAI-compatible router through Envoy:

http://<host>:8899/v1

The dashboard is available at:

http://<host>:8700

The vLLM backend in the example is reachable at:

http://<host>:8090

Confirm that the router is ready:

curl -s http://<host>:8899/v1/models | jq .

Open the dashboard at http://<host>:8700. If you are using the dashboard onboarding flow instead of the CLI command above, import the recipe from:

https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/recipes/agentic-saars.yaml

That gives you the same routing policy in the dashboard playground, replay viewer, and topology view.

Smoke Test The Route Families​

Every request should send both identity headers:

curl -s http://<host>:8899/v1/chat/completions \
  -D /tmp/agentic-route.headers \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo-session' \
  -H 'x-conversation-id: demo-run-simple' \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "What is 17 * 23? Answer with only the number."}
    ]
  }'

grep -i '^x-vsr-' /tmp/agentic-route.headers

Expected shape:

x-vsr-selected-model: qwen/qwen3.6-rocm
x-vsr-selected-decision: simple_math_fast_path
x-vsr-learning-actions: protection=allow_switch
x-vsr-learning-scopes: protection=conversation

Try a privacy request:

curl -s http://<host>:8899/v1/chat/completions \
  -D /tmp/agentic-privacy.headers \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo-session' \
  -H 'x-conversation-id: demo-run-private' \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "user", "content": "My API key is sk_live_123456. Is it safe to paste this into a public issue?"}
    ]
  }'

grep -i '^x-vsr-' /tmp/agentic-privacy.headers

Expected shape:

x-vsr-selected-model: qwen/qwen3.6-rocm
x-vsr-selected-decision: local_privacy_policy
x-vsr-learning-actions: protection=bypass

For domain routes, use prompts that naturally carry the domain signal, such as legal/health analysis for domain_legal_health, architecture or difficult code work for domain_code_complex, and research synthesis for domain_stem_research.

Try It From Inferoa​

Inferoa is an inference-native agent harness for long-running coding and tool loops. It has native integration with vLLM Semantic Router: when INFEROA_MODE=auto is enabled, the TUI reads vLLM-SR response headers and shows the selected model, selected decision, Router Learning action, cache evidence, and token pressure directly in the agent session. The docs are available at inferoa.agentic-in.ai.

Install the current dev build:

npm install -g inferoa@dev
inferoa --help

Point Inferoa at the router endpoint. INFEROA_MODE=auto tells Inferoa to send requests through vLLM Semantic Router and preserve the route metadata it sees in response headers:

export INFEROA_BASE_URL=http://<host>:8899/v1
export INFEROA_MODEL=auto
export INFEROA_MODE=auto

Start a session:

inferoa

Ask the same route-family prompts you used in the smoke tests. The footer shows the active endpoint mode, selected model, selected decision, and Router Learning state. For example, privacy traffic should show the local model and a learning bypass, while tool-heavy follow-ups can show a hard lock or stay action when Router Learning protects the current run.

Inside the TUI, open tokenmaxxing views for details:

/tokenmaxxing
/tokenmaxxing signals

/tokenmaxxing summarizes turns, selected models, model changes, cache status, tool-loop state, RTK savings, and latency. /tokenmaxxing signals shows the lower-level route and learning evidence for each turn, including action, reason, scope, selected model, selected decision, and replay id when present.

Auto routing should be inspectable: selected model, decision, learning action, and replay id.

Read Cost and Cache Results Carefully​

Cost savings come from two places:

1. The base route avoids sending every request to the strongest path.
2. Router Learning avoids unnecessary switches that would discard continuity or prefix locality.

Earlier Router Learning validation for the same protection mechanism reported a 78.71% estimated physical-model cost reduction across 21,600 deterministic turns, mostly by preventing unsafe or low-value switches. In this AMD guide, the absolute dollar number depends on the logical model prices you configure and whether the backend exposes cached-token evidence. The useful thing to inspect is the direction: simple and privacy routes should avoid expensive lanes, while long warm agent runs should not switch for tiny score differences.

Switching away from a warm long-running agent can be more expensive than the latest short prompt suggests.

Inferoa can surface token pressure, cache evidence, and model changes while you work. Treat those signals as routing telemetry, not a billing statement. Router Replay is the source of truth for why a route stayed, switched, hard-locked, or bypassed learning.

A Minimal Validation Checklist​

Before connecting a local agent to http://<host>:8899/v1, check these:

Validation should prove route intent, learning behavior, replay visibility, and backend cache evidence.

What You Get​

This guide is not a benchmark for one checkpoint. It is a deployment pattern:

• vLLM on AMD ROCm serves the local model backend.
• vLLM Semantic Router turns model: auto into explicit route policy.
• The recipe expresses simple, complex, privacy, and domain decisions.
• Router Learning protects either one conversation or the whole session.
• Privacy and security decisions can bypass learning.
• Inferoa gives an agent-facing view of the selected route, learning action, cache evidence, and token pressure while you work.

The result is not a sticky-session load balancer. It is an agent-aware router: it can say "this request looks simple" and still decide "do not switch right now" when a tool loop, session policy, or prefix-cache checkout says continuity is the better route.

Running vLLM Semantic Router on AMD Developer Cloud is not just about bringing up one more inference endpoint. It is about turning it into a routed multi-tier system that can classify requests, choose a semantic lane, and make replay and Insights immediately useful.

This post walks through the practical path: start the ROCm backend on an AMD Developer Cloud instance, install vLLM-SR, import the reference profile, and validate the deployment end to end.

What Is vLLM Semantic Router?​

vLLM Semantic Router is the system intelligence layer for LLMs. It sits in front of model endpoints, reads each request before generation begins, extracts semantic signals, and decides what should happen next.

That makes it more than a cost-saving router. It is also a control layer for safety, privacy, and policy. The same routing system that sends simple work to cheaper lanes can also detect sensitive traffic, keep private requests on local infrastructure, apply security-oriented plugin chains, and reserve stronger models for tasks that actually need deeper reasoning.

This is what makes Semantic Router especially relevant for AMD deployments. It supports intelligent multi-model routing, privacy-first enterprise AI, and local-first personal AI in the same architecture. In practice, one system can decide when to optimize for cost, when to prioritize security or privacy, and when to keep a personal or sensitive workflow close to the user instead of treating every query the same way.

Note: in this reference profile, aliases such as google/gemini-3.1-pro, openai/gpt5.4, and anthropic/claude-opus-4.6 are logical routing tiers backed by the same ROCm Qwen deployment. They are not outbound calls to those vendor APIs.

How the Signal-Driven Architecture Works​

The easiest way to understand vLLM Semantic Router is as a four-layer architecture:

• Signals are the raw observations extracted from each request. In this repository, the AMD profile uses signals such as keyword, embedding, structure, fact_check, user_feedback, reask, language, domain, context, and complexity.
• Projections are the coordination layer. They take raw signal evidence and turn it into reusable routing outputs such as balance_simple, balance_complex, balance_reasoning, verification_required, or urgency_elevated.
• Decisions are the policy layer. They combine signals and projection outputs into named routing outcomes such as medium_code_general, reasoning_deep, or premium_legal.
• Models are the target lanes. Decisions point to logical models or aliases through modelRefs, while endpoint wiring, pricing, and backend references live in the provider model catalog.

In other words, the runtime flow is:

User Request -> Signals -> Projections -> Decisions -> Model Alias -> Backend Response

This is why the system is more expressive than a simple classifier. A query does not have to be “just math” or “just code.” It can simultaneously look urgent, evidence-sensitive, short-context, Chinese-language, and correction-oriented, and the routing policy can respond to that richer state.

What You Will Deploy​

At a high level, this deployment consists of:

• One ROCm vLLM backend running Qwen/Qwen3.5-122B-A10B-FP8
• One vLLM Semantic Router instance in front of that backend
• One reference routing profile from deploy/recipes/balance.yaml
• One dashboard for onboarding, replay inspection, playground testing, and Insights

The reference alias layout is:

• qwen/qwen3.5-rocm for the SIMPLE lane
• google/gemini-2.5-flash-lite for lower-cost expressive medium tasks
• google/gemini-3.1-pro for complex technical or architecture-heavy tasks
• openai/gpt5.4 for high-reasoning escalation
• anthropic/claude-opus-4.6 for the premium legal lane

Pricing in the profile is intentionally exaggerated so Insights can make tier differences and savings easy to see. It is a demo-friendly routing profile, not a mirror of vendor billing.

Why This Matters for AMD​

This architecture creates a compelling opportunity for AMD. Rather than being positioned as just another accelerator target, AMD hardware can take on a more strategic role. With a Semantic Router layered in front, an AMD deployment can serve as the control plane for system intelligence.

Notably, this approach spans the entire AMD data center GPU lineup, including the MI300X, MI325X, MI355X, and beyond.

1. Intelligent Routing on AMD​

The most immediate opportunity is intelligent routing. A single ROCm backend on AMD Developer Cloud can serve as the physical execution layer for multiple logical lanes. That means teams can prototype a Mixture-of-Models experience, cost-aware routing, replay-driven debugging, and tiered product behavior without first standing up a large multi-backend fleet.

In the AMD reference profile, the cheapest, medium, complex, reasoning, and premium lanes all resolve onto different models. The router still gives you differentiated behavior because the policy lives in signals, projections, and decisions, not only in the number of containers you run.

2. Privacy Routing and Local-First Governance​

The second opportunity is privacy routing, that keeps PII, private code, internal documents, and suspicious prompts on a local lane while only escalating clearly non-sensitive reasoning work when policy allows it. That pattern is especially meaningful on AMD because it supports a local-first deployment story: keep sensitive traffic on infrastructure you control, audit every decision, and make cloud escalation a governed exception instead of the default.

For enterprises, that means AMD-backed deployments can become the trusted default lane for internal copilots, regulated workloads, or hybrid private AI systems. For developers, it means privacy is not just a hosting choice; it becomes a routing policy.

3. Personal AI and Local Personal Agents​

The third opportunity is personal AI like deploying a personal model on AMD AI MAX+ and connecting to external Models as needed. Once routing, privacy, and reasoning are expressed as policy, an AMD-hosted stack can support assistants that feel more personal and more controlled. A personal AI system can keep ordinary tasks, memory-aware follow-ups, and private context on a local lane, while only escalating special cases when explicitly permitted.

That makes AMD interesting not only for enterprise infrastructure, but also for self-hosted assistants, home-lab AI, and local-first personal workflows. The important point is that Semantic Router lets the system distinguish between “keep this local,” “this is cheap and routine,” and “this needs deeper reasoning,” instead of treating all personal AI traffic as one undifferentiated workload.

Getting Started​

Before you begin, make sure your AMD Developer Cloud instance is ready with:

• A ROCm-capable AMD GPU instance
• Docker installed and running
• Access to /dev/kfd and /dev/dri
• A persistent Hugging Face cache path, if you want to avoid repeated model downloads

Once you can SSH into the machine, you are ready to launch the backend.

Step 1: Create the Shared Docker Network​

Create the network used by the reference deployment:

sudo docker network create vllm-sr-network 2>/dev/null || true

This keeps the backend naming consistent with the reference profile, which expects the vLLM service at vllm:8000.

Step 2: Start the AMD ROCm vLLM Backend​

Run the following command on your AMD Developer Cloud instance:

sudo docker run -d \
  --name vllm \
  --network=vllm-sr-network \
  --restart unless-stopped \
  -p "${VLLM_PORT_122B:-8090}:8000" \
  -v "${VLLM_HF_CACHE:-/mnt/data/huggingface-cache}:/root/.cache/huggingface" \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add=video \
  --ipc=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --shm-size 32G \
  -v /data:/data \
  -v "$HOME:/myhome" \
  -w /myhome \
  -e VLLM_ROCM_USE_AITER=1 \
  --entrypoint python3 \
  vllm/vllm-openai-rocm:latest \
  -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3.5-122B-A10B-FP8 \
    --host 0.0.0.0 \
    --port 8000 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --served-model-name qwen/qwen3.5-rocm google/gemini-2.5-flash-lite google/gemini-3.1-pro openai/gpt5.4 anthropic/claude-opus-4.6 \
    --trust-remote-code \
    --reasoning-parser qwen3 \
    --max-model-len 262144 \
    --language-model-only \
    --max-num-seqs 128 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.85

This is the core of the deployment. The backend is still one Qwen model, but it now exposes multiple served-model aliases that the router can target semantically.

Install vLLM Semantic Router​

With the backend up, install vLLM Semantic Router:

prerequisites

Python version: python3.12

# Create a virtual environment (recommended)
python3 -m venv vsr
source vsr/bin/activate 
curl -fsSL https://vllm-sr.ai/install.sh | bash

The router dashboard should then be available at:

http://<your-server-ip>:8700

Open the dashboard and complete onboarding.

When prompted to load a routing profile (please skip the model configuration directly), import the reference YAML directly from:

https://raw.githubusercontent.com/vllm-project/semantic-router/main/deploy/recipes/balance.yaml

The remote import path applies the full YAML directly during onboarding. If you later inspect the same profile in the DSL editor, the routing surfaces decompile from routing.modelCards, routing.signals, routing.projections, and routing.decisions, while providers remains YAML-native.

What the Reference Profile Is Doing​

The imported profile expresses a complete AMD routing story with 23 active decisions across:

• simple fallback lanes
• medium domain lanes
• verified overlays
• feedback recovery lanes
• complex technical lanes
• reasoning escalation lanes
• one emotionally engaged general lane
• one premium legal lane

This is useful because replay and Insights stay signal-native. Instead of inventing a separate runtime dimension schema, the system shows what actually happened during routing: which signals matched, which projection outputs fired, which decision won, and which alias received the request.

Two intentionally conservative paths in the profile are worth calling out:

• fact_check overlays only escalate when verification pressure is strong and the prompt gives explicit confirmation cues
• user_feedback recovery lanes require literal correction or clarification signals instead of broadly capturing all follow-up traffic

That makes the profile easier to reason about when you are testing routing behavior on a single backend.

Validate the Deployment in the Playground​

Once onboarding is complete, the fastest way to validate the system is through the dashboard playground. Try a few prompts that represent different routing tiers:

Coding Help​

Debug this Python stack trace and suggest the most likely fix.

This should land on the cheaper coding lane backed by qwen/qwen3.5-rocm.

Deep Reasoning​

Prove rigorously that the square root of 2 is irrational.

This should escalate into the reasoning tier and map to the openai/gpt5.4 alias.

Premium Legal Analysis​

Provide a legal analysis of the indemnity clause, liability cap, and compliance obligations in this contract.

This should match the premium legal lane and forward to anthropic/claude-opus-4.6.

Observe the Routing Behavior in Insights​

You can also inspect the routing behavior in Insights. The reference profile includes replay, so you can see what actually happened during routing. And also how much money you saved by using the cheaper lanes.

Test the Brain Topology​

The router dashboard also includes a brain topology view that shows the high-level structure of the routing graph. This is useful for understanding the overall shape of the policy, and how different decisions are connected. And you can directly test your prompt to see its activation path.

Design Your Own Routing DSL​

The dashboard also includes a full DSL editor that lets you design your own routing policy. The reference profile is a good starting point, but you can also use the editor to try out different ideas.

And you can design a very complex boolean expression in a single route, to express very precise routing policy.

Final Thoughts​

Deploying vLLM Semantic Router on AMD Developer Cloud gives you more than a working endpoint. It gives you a compact routed system: one or more ROCm-hosted backend, multiple semantic tiers, visible routing logic, and a dashboard experience that makes the behavior understandable instead of opaque.

That is what makes this reference profile useful. You can start with a single real AMD backend, import a complete routing policy, inspect how decisions are made, and then iterate from there without first building a large multi-backend fleet. For teams exploring cost-aware routing, replay-driven debugging, or AMD-based MoM patterns, it is a practical and reproducible starting point.

Acknowledgements​

We would like to thank the following teams and individuals for their contributions to this work:

• AMD AIG Team: Andy Luo, Haichen Zhang
• vLLM Semantic Router OSS team: Xunzhuo Liu, Huamin Chen

Join Us​

Looking for Collaborations! Calling all passionate community developers and researchers: join us in building the system intelligence on AMD GPUs.

Interested? Reach out to us:

• Haichen Zhang: haichzha@amd.com
• Xunzhuo Liu: xunzhuo@vllm-semantic-router.ai

Share your use cases and feedback in #semantic-router channel on vLLM Slack

This roadmap is not just about adding more capability. It is about making the full system coherent: one stable contract across Docker and Kubernetes, one cleaner deployment path, one real version story for images and packages, stronger performance validation on both NVIDIA and AMD, and a research track that directly improves the product instead of sitting outside it.

Why Themis​

Themis is the Greek figure of order, rules, and judgment. That fits this release better than a speed-oriented or purely routing-oriented codename. Themis is where Semantic Router starts acting less like a promising set of powerful building blocks and more like a platform with stable contracts, repeatable operations, and enforceable guardrails.

The current v0.3 milestone reflects that shift. It includes the new workstreams opened specifically for Themis, but it also folds in existing issues around protocol compatibility, session affinity, memory hardening, dashboard state, observability, security, and API standardization. This release is not a narrow feature sprint. It is a system-shaping release.

1. Stable API, config, and deployment contracts​

The highest-priority theme in Themis is eliminating contract drift across environments. Today, router behavior, Helm-facing config, dashboard flows, and the Python CLI still expose differences that create friction for operators. Themis is where we narrow those seams.

At the center of that work is a canonical API and config contract across router, CLI, dashboard, and Kubernetes. The goal is simple: after this release, a user should not have to mentally maintain one configuration model for local Docker workflows and another for Kubernetes deployment. This is the core of #1505.

That contract work also includes the deployment entry point itself. The vllm-sr CLI should become the normal path for standing up both Docker and Kubernetes environments, instead of being treated as a local-only helper while Helm and other deployment paths evolve separately. That is the focus of #1507.

We also want the runtime topology to become easier to reason about. Themis moves toward a router-focused vllm-sr image, with external services such as dashboard, Envoy, and persistence components split out more cleanly. This keeps the main runtime narrower and makes upgrades, debugging, and composition less fragile. That work is tracked in #1508.

This same contract cleanup extends to protocol compatibility. Themis already includes work to support first-class OpenAI and Anthropic API entry points, align API definitions with official SDKs, and reduce homegrown JSON struct drift across the codebase. Those concerns now live in #1517, #1404, and #1217.

2. Stable versions, upgrades, and production operations​

Themis is also the release where we stop treating latest as a deployment strategy. Production users need to know what they are running, how they upgrade, how they roll back, and what guarantees exist between images, packages, and charts. That operational maturity is the purpose of #1506.

This means introducing explicit version channels such as nightly and tagged releases, carrying versioned images and packages through the stack, and documenting a full upgrade and rollback flow instead of assuming rebuild-and-redeploy. A stable version story is part of stability at scale, not an afterthought to it.

Operational stability also depends on where state lives. Dashboard behavior today still depends too heavily on in-memory state for workflows that should survive restarts, scale-outs, and multi-user operation. Themis moves those operationally important pieces into a database-backed control plane, tracked in #1509.

As milestone triage has progressed, this operations theme has also pulled in related issues around docs and environment correctness, especially where deployment docs, API expectations, and runtime behavior need to converge before we can credibly call the surface stable.

3. Performance at scale on real hardware​

Themis is not only about control-plane cleanup. It is also about making sure the router and its supporting model stack behave well under real load, across real backends, on real platforms. That is the purpose of #1510.

We want broader large-scale regression coverage across Candle, ONNX, and related runtime paths, with repeatable performance baselines for both NVIDIA and AMD. This matters because Semantic Router is increasingly expected to sit in front of more heterogeneous workloads: more model families, more protocol paths, more multi-component deployments, and more memory-heavy workflows.

This performance theme is also tied to product credibility. If we claim the platform is ready for production routing, then we need more than point optimizations. We need performance tests that survive release-to-release, platform-to-platform, and topology-to-topology changes.

That same bar increasingly applies to higher-level agent surfaces such as ClawOS. If model routing, memory, and tool execution are going to be orchestrated in room-based agent workflows, then performance and runtime visibility have to scale there too.

4. Research that feeds the product​

Themis still includes research-heavy work, and it should. But the research in this milestone is there because it improves the production system, not because we are parking speculative ideas in the roadmap.

The first track is NL-to-DSL authoring in the dashboard, tracked in #1511. The goal is to let users express routing intent in natural language and generate a usable DSL scaffold instead of forcing every workflow through fully manual route authoring.

The second track is a feedback loop for generated DSL, tracked in #1512. Generated routing logic becomes much more useful when it can learn from real request history, observed routing outcomes, and user feedback, instead of acting like a one-shot assistant.

The third track is multi-turn session affinity, tracked in #1513 and reinforced by the older conversation-stability issue #1439. This is one of the clearest examples of research feeding production directly: without stable session affinity, routed multi-turn conversations can bounce between models and degrade user experience even if each single-turn decision looks correct.

There is also research around model legitimacy and selection quality, including #1422 and #1514. This line of work matters because model selection is only useful in production when it is trustworthy, inspectable, and not dependent on fragile external-only components. Themis should move that work closer to something operators can actually rely on.

ClawOS does have a genuine research component here, but it is specifically the context question. #1522 is about studying context-management patterns and OpenClaw best practices so long-running, tool-rich, room-based workflows have a clearer operating model.

In that sense, the research section of Themis is really about system intelligence: generating better routing logic, improving it continuously, keeping conversations stable across turns, and making model-selection decisions more defensible.

5. Hardening the current product surface​

Themis also has a large body of work that is less glamorous than new intelligence features, but just as important for adoption.

Model selection needs to become more usable without external-service-only dependencies, which is the focus of #1514. Eval workflows need to be revisited so system eval and signal eval are first-class and stable inside the dashboard, tracked in #1515.

RAG and memory workflows also need to become more production-friendly. That includes the main hardening track in #1516, plus milestone issues already folded in around memory evolution such as #1293, #1287, #1289, #1350, and #1353.

ClawOS also belongs in this product-hardening bucket. #1521 is not a research item; it is about making collaborative rooms work as a first-class product surface through Matrix-style full WebSocket communication between rooms and participants.

This is also where protocol polish and dashboard usability meet. The goal is not just to have more capability on paper, but to make those capabilities easier to operate in the dashboard, easier to expose consistently through APIs, and easier to validate end to end.

6. Security and quality closure​

Themis is also where we close the operational gaps that would block serious production adoption. That starts with the main security and RBAC workstream in #1518, but it is reinforced by several already-folded issues that expose concrete weaknesses in the current surface.

That includes security issues such as #1443, #1445, #1447, #1448, #1452, #1454, #1456, and #1458. These are exactly the kinds of issues that justify the Themis theme: if the platform is going to be production-ready, the security model has to be explicit and closed-loop.

Quality also means broader E2E coverage. The main expansion item is #1519, but related milestone issues such as #1295, #1432, #1501, and #1083 show the same pattern: production hardening requires better system-level tests, better observability, and fewer hidden assumptions.

That broader observability push now also includes ClawOS-specific visibility into model and tool behavior through #1523, so agentic workflows are not left outside the production-debugging story.

What success looks like​

If Themis is successful, Semantic Router should feel materially different to deploy and operate:

• API and config behavior should be much more consistent across Docker, Kubernetes, CLI, and dashboard workflows
• release channels, upgrades, and rollbacks should be explicit rather than implicit
• performance claims should be backed by repeatable NVIDIA and AMD validation
• research work should show up as product intelligence, especially in DSL generation, feedback loops, session affinity, ClawOS context management, and better model selection
• memory, eval, protocol compatibility, and dashboard state should look more like stable platform features than experimental edges
• security, RBAC, observability, and E2E coverage should be strong enough that production users can trust the platform boundary

Themis is therefore less about one headline feature and more about making the whole system hold together under real use.

For the active implementation tracker, see v0.3 - Themis: Stability at Scale milestone and issue #1520.

Athena is the first major hardening step after Iris. It refreshes the model stack, extends routing into safety and semantic control, and starts shaping the system brain needed to make Semantic Router easier to govern, operate, and scale in real deployments.

Synced from official vLLM Blog: vLLM Semantic Router v0.2 Athena: ClawOS, Model Refresh, and the System Brain

Building Mixture-of-Models on AMD GPUs is not just about serving one more model on one more device. It is about turning routing, governance, and inference into a coordinated system so MoM workloads can run efficiently on AMD hardware at production scale.

Synced from official vLLM Blog: Building Mixture-of-Models on AMD GPUs with vLLM-SR

In Greek mythology, Iris (Ἶρις) served as the divine messenger who bridged the realms of gods and mortals, traveling on the arc of the rainbow to deliver messages across vast distances. This symbolism perfectly captures what vLLM Semantic Router v0.1 achieves: a bridge between users and diverse AI models, intelligently routing requests across different LLM providers and architectures.

Synced from official vLLM Blog: vLLM Semantic Router v0.1 Iris: The First Major Release

AMD has been a long-term technology partner for the vLLM community, from accelerating the vLLM inference engine on AMD GPUs and ROCm™ Software to now co-building the next layer of the AI stack: intelligent routing and governance for Mixture-of-Models (MoM) systems.

Synced from official vLLM Blog: AMD × vLLM Semantic Router: Building the System Intelligence Together

Building on our Signal-Decision Architecture, we introduce HaluGate—a conditional, token-level hallucination detection pipeline that catches unsupported claims before they reach your users. No LLM-as-judge. No Python runtime. Just fast, explainable verification at the point of delivery.

Synced from official vLLM Blog: Token-Level Truth: Real-Time Hallucination Detection for Production LLMs

Synced from official vLLM Blog: Signal-Decision Driven Architecture: Reshaping Semantic Routing at Scale

This resonates deeply with our experience building the vLLM Semantic Router. We've observed the same problem from a different angle: when AI agents have access to hundreds or thousands of tools, how do they know which tools are relevant for a given task?

Our solution: semantic tool selection—using semantic similarity to automatically select the most relevant tools for each user query before the request even reaches the LLM.

The Problem: Tool Overload in AI Agents​

Context Window Bloat​

Consider an AI agent with access to hundreds of tools across multiple domains. Loading all tool definitions into the context window for every request:

• Consumes significant tokens for tool definitions (e.g., 741 tools require ~120K tokens)
• Increases latency as the model processes a large number of tools
• Raises costs due to increased token usage
• May reduce accuracy as the model faces more complex selection decisions

The Relevance Problem​

In many cases, most tools are not relevant for a given query:

• User asks: "What's the weather in San Francisco?"
• Agent receives: Hundreds of tool definitions (weather, finance, database, email, calendar, etc.)
• Reality: Only a small subset of tools are actually relevant

This creates inefficiency in terms of tokens, latency, cost, and model decision-making complexity.

The Research Evidence​

Recent academic studies have measured the impact of large tool catalogs on LLM performance:

Accuracy Degradation: Research testing tool selection with growing catalogs found that:

• With ~50 tools (8K tokens): Most models maintain 84-95% accuracy
• With ~200 tools (32K tokens): Accuracy ranges from 41-83% depending on model
• With ~740 tools (120K tokens): Accuracy drops to 0-20% for most models

Different models show varying degrees of degradation, with open-source models showing 79-100% degradation when scaling from small to large tool catalogs.

The "Lost in the Middle" Effect: Research has documented position bias where tools in the middle of long lists are less likely to be selected correctly. For example, with 741 tools, middle positions (40-60%) showed 22-52% accuracy compared to 31-32% at the beginning/end positions for some models.

Non-Linear Degradation: Performance degradation is not gradual. Research shows that accuracy can drop sharply as tool count increases, with the transition from 207 to 417 tools showing particularly steep declines (e.g., from 64% to 20% for one model tested).

Our Solution: Semantic Tool Selection​

The vLLM Semantic Router implements semantic tool selection as an intelligent filter that sits between the user and the LLM:

How It Works​

Step 1: Tool Database with Embeddings

Each tool in our database has:

• Tool definition (name, parameters, schema)
• Rich description optimized for semantic matching
• Pre-computed embedding vector
• Optional metadata (category, tags)

Step 2: Query Embedding and Similarity Search

When a user query arrives:

1. Generate an embedding for the query text
2. Calculate cosine similarity with all tool embeddings
3. Select top-K tools above a similarity threshold
4. Inject only relevant tools into the request

Step 3: Request Modification

The router modifies the API request to include only selected tools, dramatically reducing token usage.

Experimental Results​

We conducted extensive experiments comparing traditional "load all tools" approaches with our semantic tool selection system across three real-world scenarios. Our findings align with recent research showing that LLMs struggle significantly with large tool catalogs and long contexts in tool-calling scenarios.

Research Context: The Tool Selection Challenge​

Recent academic research has quantified the severity of this problem. Studies show that as tool catalogs grow:

• Performance drops 7-85% when tool count increases from small to large catalogs
• Token consumption explodes by 50-100x with naive "load all tools" approaches
• Position bias emerges - tools buried in the middle of long lists are often missed ("lost in the middle")
• Accuracy degrades non-linearly - even state-of-the-art models like GPT-4 struggle

One study testing tool selection with increasing catalog sizes found that baseline accuracy dropped from 78% with 10 tools to just 13.62% with 100+ tools - a catastrophic 82% degradation. This "needle in a haystack" problem for tool selection motivated our semantic approach.

Large Tool Catalog Stress Test​

Setup:

Based on the Berkeley Function Calling Leaderboard (BFCL) dataset, we tested tool selection performance as catalog size grows:

• Dataset: 858 function calling samples (simple, live_simple, multiple subsets)
• Tool catalog sizes: Varied from 49 tools (8K tokens) to 741 tools (120K tokens)
• Models tested: Llama-3.1-70B, Mistral-Large, Granite-3.1-8B, ToolACE-8B, BitAgent-8B
• Position testing: Tools placed at 10%, 30%, 50%, 70%, 90% positions in the catalog

Results - Accuracy Degradation by Catalog Size:

Key Findings:

1. Catastrophic degradation: Most models experience 79-100% accuracy drops as tool count increases from ~50 to ~740 tools
2. Prompt bloat: At 120K tokens, average prompt size was 127,315 tokens with all tools loaded

Position Bias Analysis:

Testing with 741 tools at different positions revealed severe "lost in the middle" effects:

Implications for vLLM Semantic Router:

These findings reinforce why semantic selection is critical:

1. Smaller contexts = better comprehension: By reducing tool catalog from 120K to 1K tokens, we leave 119K tokens for tool responses and conversation history
2. Focused selection = better recall: With only 3-5 relevant tools, models can focus on understanding responses rather than parsing hundreds of tool descriptions
3. Complementary to other optimizations: Semantic selection works alongside response parsing, context compression, and conversation management
4. Enables longer conversations: Saving 99.1% of context on tool definitions (127,315 → 1,084 tokens) allows significantly more room for conversation history or tool responses

Benefits of Semantic Tool Selection​

1. Restores Usability at Scale​

Research shows that without semantic selection, tool-calling systems become unusable beyond ~100 tools:

Accuracy Recovery:

Key Insight: Semantic selection doesn't just improve performance—it makes large-scale tool calling possible.

2. Dramatic Token & Cost Reduction​

Token Savings (741 tools):

• Baseline: 127,315 tokens per request
• Semantic Selection: 1,084 tokens per request
• Reduction: 99.1% (117x fewer tokens)

Cost Impact (based on typical LLM pricing at $2.50/$10 per 1M input/output tokens):

3. Eliminates Position Bias​

Research documents severe "lost in the middle" effects. Semantic selection eliminates this:

Position Bias (741 tools, Llama-3.1-70B):

• Beginning: 32% accuracy
• Middle: 22% accuracy (31% worse)
• End: 31% accuracy

With Semantic Selection: 94% accuracy regardless of original position

5. Scalability Beyond Current Limits​

The MCP ecosystem already has 4,400+ servers. Research shows:

• At 100+ tools: Baseline accuracy drops to 13-15% (near-random)
• With semantic selection: Maintains 43%+ accuracy even at scale
• Future-proof: As tool ecosystems grow to 10,000+ tools, semantic selection becomes essential

Architecture Overview​

Here's how semantic tool selection integrates into the request flow:

System Components​

Comparison with Other Approaches​

vs. Loading All Tools​

Research demonstrates clear advantages of semantic selection:

vs. Manual Categorization​

Manual Categories:

• Requires maintaining tool taxonomies
• Brittle when tools span multiple categories
• Doesn't adapt to query nuances
• Maintenance overhead: ~2 hours/week per 100 tools

Semantic Selection:

• Automatic relevance based on embeddings
• Handles cross-domain queries naturally
• Adapts to new tools without reconfiguration
• Maintenance overhead: ~5 minutes/week (add new tools)

vs. Code Execution (MCP Approach)​

Anthropic's code execution and our semantic selection are complementary:

Combined Approach:

1. Semantic Router selects relevant tools (500 → 3 tools)
2. LLM writes code to use those tools efficiently
3. Code execution handles data filtering and complex logic

This gives you the best of both worlds: efficient tool discovery + powerful execution patterns.

Future Directions: Scaling to Thousands of Tools​

While our current implementation handles hundreds of tools effectively, research points to new challenges as tool ecosystems grow to thousands of tools:

Hierarchical Retrieval​

Recent studies show that flat similarity search begins to degrade beyond ~1,000 tools. Future work will explore:

• Two-stage retrieval: First select relevant categories, then tools within categories
• Adaptive retrieval: Dynamically adjust top-K based on query complexity
• Hybrid approaches: Combine semantic similarity with metadata filtering

Tool Response Management​

Research has identified tool response processing as a critical bottleneck:

• Intelligent parsing: Extract only relevant fields from large JSON responses
• Progressive disclosure: Stream tool responses incrementally
• Response summarization: Use smaller models to compress responses before sending to main LLM

Multi-Turn Optimization​

For long conversations with many tool calls:

• Context compression: Summarize earlier turns while preserving key information
• Selective history: Include only relevant past tool calls in context
• State management: Track conversation state separately from full history

Conclusion​

Anthropic's blog on code execution with MCP highlighted a fundamental challenge: agents need efficient ways to discover and use tools at scale. Their solution—progressive disclosure through code execution—is elegant and powerful.

Our semantic tool selection approach tackles the same problem from a complementary angle: use semantic similarity to automatically select relevant tools before the LLM even sees the request. Research demonstrates:

• 99.1% token reduction (127,315 → 1,084 tokens for 741 tools)
• 3.2x accuracy improvement (43.13% vs 13.62% baseline in RAG-MCP benchmark)
• Significant cost reduction through reduced token usage
• Improved selection quality by focusing on relevant tools
• Transparent and debuggable tool selection process

The two approaches are not mutually exclusive—in fact, they work beautifully together:

1. Semantic Router filters 500 tools down to 3 relevant ones
2. LLM writes code to use those tools efficiently
3. Code execution handles data processing and complex workflows

As AI agents become more capable and connect to more tools, intelligent tool management becomes critical. Whether through semantic selection, code execution, or a combination of both, the future of AI agents lies in smart, context-aware tool discovery that scales efficiently.

Give it a Try​

The vLLM Semantic Router is open source:

• GitHub: github.com/vllm-project/semantic-router
• Documentation: vllm-sr.ai
• Quick Start: Deploy in 5 minutes with Docker Compose or Kubernetes

Example configuration to get started:

# config.yaml
tools:
  enabled: true
  top_k: 3
  similarity_threshold: 0.80
  tools_db_path: config/tools_db.json
  fallback_to_empty: true

Start with a small tool database (10-20 tools) and expand as you see the benefits. Monitor the metrics dashboard to tune thresholds and optimize performance.

Sync from vLLM Official Blog.

Background: From BERT to a Modular System​

The previous implementation relied primarily on BERT and ModernBERT for intent and jailbreak classification. While ModernBERT performs well for English text classification tasks, it has the following limitations:

• Language Coverage: The original ModernBERT's multilingual support is limited compared to models trained on more diverse datasets. (Note: mmBERT, a massively multilingual variant of ModernBERT supporting 1800+ languages, was released after this refactoring began and represents an alternative approach to the multilingual challenge)
• Context Length: While ModernBERT extends context to 8,192 tokens using RoPE (source), models like Qwen3-Embedding support up to 32,768 tokens, which is beneficial for very long document processing
• Model Coupling: Classification logic was tightly coupled to specific model architectures, making it difficult to add new models

These constraints motivated a broader refactoring that would enable the system to support multiple model types while maintaining performance. The modular architecture means that newer models like mmBERT can be integrated alongside Qwen3-Embedding and EmbeddingGemma, allowing the router to select the most appropriate model for each task.

Architectural Restructuring​

The refactoring introduces a layered architecture in the candle-binding crate. This structure separates concerns: core functionality remains independent of specific models, while new model architectures can be added without modifying existing code. The DualPathUnifiedClassifier implements routing logic that selects between traditional fine-tuned models and LoRA-adapted models based on the task requirements.

Long-Context Embedding Models​

Two new embedding models address the context length limitation:

Qwen3-Embedding​

Qwen3-Embedding supports context lengths up to 32,768 tokens (Hugging Face model card). The implementation uses a RoPE (Rotary Position Embedding), enabling this extended context handling through improved frequency resolution at longer distances.

Qwen3-Embedding was trained on text from over 100 languages (Hugging Face model card), making it suitable for multilingual routing scenarios where the previous ModernBERT-only approach would struggle.

EmbeddingGemma-300M​

Google's EmbeddingGemma-300M takes a different approach, focusing on smaller model size while maintaining quality. The model supports context lengths of 2,048 tokens and implements Matryoshka representation learning, which means embeddings can be truncated to 768, 512, 256, or 128 dimensions without retraining (Hugging Face model card).

The architecture uses Multi-Query Attention (MQA) with 3 query heads and 1 key-value head, reducing memory bandwidth requirements. A distinctive feature is the dense bottleneck layer (768 → 3072 → 768) applied after the transformer blocks, which improves embedding quality based on the Matryoshka training approach.

Low-Rank Adaptation for Multi-Task Classification​

LoRA addresses a fundamental inefficiency in the previous system. When a classification system needs to determine intent, detect PII, and check for security issues, the naive approach runs three separate fine-tuned models:

Each model processes the input through its entire network, including the expensive base transformer layers. This results in O(n) complexity where n is the number of classification tasks.

LoRA changes this by sharing the base model computation:

The base model runs once, producing intermediate representations. Each LoRA adapter then applies task-specific low-rank weight updates to specialize the output. Since LoRA adapters typically modify less than 1% of the model's parameters, this final step is much faster than running complete models.

The implementation in parallel_engine.rs uses Rayon for data parallelism, processing multiple LoRA adapters concurrently. For a request requiring three classifications, this changes the workload from three full forward passes to one full pass plus three lightweight adapter applications.

Concurrency Through OnceLock​

The previous implementation used lazy_static for managing global classifier state, which introduced lock contention under concurrent load. The refactoring replaces this with OnceLock from the Rust standard library.

OnceLock provides lock-free reads after initialization. After the first initialization, all subsequent accesses are simple pointer reads with no synchronization overhead. Tests in oncelock_concurrent_test.rs verify this with 10 concurrent threads performing 30 total classifications, confirming that throughput scales linearly with thread count.

This matters when the router processes multiple incoming requests. With lazy_static, concurrent requests would queue behind a mutex. With OnceLock, they execute in parallel without contention.

Flash Attention for GPU Acceleration​

Flash Attention 2 support is available as an optional feature for CUDA builds, though it requires Ampere-generation or newer GPUs (compute capability ≥ 8.0). Flash Attention optimizes the attention mechanism by processing computations in blocks that fit in fast on-chip SRAM memory, avoiding repeated reads from slower GPU DRAM.

Both ModernBERT and Qwen3 benefit from Flash Attention integration:

• ModernBERT: Achieves up to 3× faster self-attention computations with significantly reduced memory usage (source). The model also uses alternating attention patterns (global attention every third layer, local sliding-window attention otherwise) to balance efficiency with context retention (source).

• Qwen3: Integration of FlashAttention-2 provides up to 4× speedup in attention operations. For the 14B variant, this translates to 70-110 tokens/second during inference compared to 30-35 tokens/second without it—a performance improvement that becomes more pronounced with longer contexts (source).

The Rust implementation makes Flash Attention optional via Cargo features, allowing deployment on systems without compatible GPUs while enabling substantial performance gains when hardware supports it.

Cross-Language Integration for Cloud-Native Ecosystems​

The choice of Rust for the core classification engine combined with Go FFI (Foreign Function Interface) bindings addresses a practical deployment challenge in cloud-native environments.

Why Rust for ML Inference​

Rust provides several advantages for the classification layer:

• Performance: Near-C performance with zero-cost abstractions, critical for low-latency inference
• Memory Safety: Compile-time guarantees prevent common bugs like buffer overflows and use-after-free errors
• Concurrency: The ownership system prevents data races, enabling safe parallel processing with Rayon
• No Garbage Collection: Predictable latency without GC pauses that affect request processing

The Candle framework leverages these Rust strengths while providing a familiar API for ML model development.

Why Go FFI Bindings Matter​

While Rust excels at compute-intensive ML inference, Go dominates the cloud-native infrastructure ecosystem. The FFI layer bridges these worlds. This integration enables deployment in environments where Go is the primary language:

• Envoy Proxy Integration: The semantic router runs as an Envoy external processing filter, written in Go. The FFI allows the Go filter to leverage high-performance Rust classification without rewriting the entire Envoy integration layer.
• Kubernetes Operators: Cloud-native operators are typically written in Go using controller-runtime. The FFI enables these operators to embed classification logic directly rather than making network calls to separate services.
• Service Meshes: Projects like Istio, Linkerd, and Consul are Go-based. The FFI allows routing decisions to use ML-based classification while maintaining compatibility with existing mesh control planes.
• API Gateways: Many API gateways (Kong, Tyk) have Go components. The FFI enables semantic routing at the gateway layer without introducing additional microservices.

Deployment Flexibility​

The dual-language architecture provides deployment options:

• Embedded Mode: The Go service links directly to the Rust library via CGO, minimizing latency and deployment complexity
• Process Isolation: The classification layer can run as a separate process, communicating via gRPC or Unix sockets for additional fault isolation
• Mixed Workloads: Services can combine Go's networking and orchestration strengths with Rust's ML inference performance

The semantic router leverages this pattern extensively. The main routing logic, configuration management, and cache implementations are in Go, while the compute-intensive classification runs in Rust. This separation allows each component to use the most appropriate language while maintaining clean interfaces through the FFI layer.

Performance Characteristics​

The benefits of this architecture vary by workload:

• Single vs multi-task classification: LoRA provides minimal benefit since there's no base model sharing. Traditional fine-tuned models may be faster. LoRA shows clear advantages when performing multiple classifications on the same input. Since the base model runs once and only LoRA adapters execute for each task, the overhead is substantially reduced compared to running separate full models. The actual speedup depends on the ratio of base model computation to adapter computation.
• Long-context inputs: Qwen3-Embedding enables routing decisions on documents up to 32K tokens without truncation, extending beyond ModernBERT's 8K limit for very long documents. With Flash Attention 2 enabled on compatible GPUs, the performance advantage becomes more substantial as context length increases.
• Multilingual routing: Models can now handle routing decisions for languages where ModernBERT has limited training data.
• High concurrency: OnceLock eliminates lock contention, allowing throughput to scale with CPU cores for classification operations.
• GPU acceleration: When Flash Attention 2 is enabled, attention operations run 3-4× faster, with the speedup becoming more pronounced at longer sequence lengths. This makes GPU deployment particularly advantageous for high-throughput scenarios.

Future Directions​

The modular architecture enables several extensions:

• Additional embedding models can be added by implementing the CoreModel trait
• Flash Attention 3 support when available in Candle
• Quantization support (4-bit, 8-bit) for reduced memory footprint
• Custom LoRA adapters for domain-specific routing
• FFI bindings for additional languages (Python, Java, C++) to expand integration possibilities

The system now has a foundation for incorporating new research advances without requiring architectural changes. The FFI layer provides a stable interface that allows the Rust implementation to evolve independently while maintaining compatibility with existing Go-based deployments.

Resources​

• Project Repository: https://github.com/vllm-project/semantic-router
• Candle Framework: https://github.com/huggingface/candle
• Qwen3-Embedding: https://huggingface.co/Qwen/Qwen3-Embedding-0.6B
• EmbeddingGemma: https://huggingface.co/google/embeddinggemma-300m

About Our Release Naming Convention​

Starting with v0.1, each major release of vLLM Semantic Router will carry a codename inspired by figures from Greek mythology. These names reflect the essence and purpose of each release, connecting ancient wisdom with modern AI infrastructure.

Our inaugural release is named Iris (Ἶρις), after the Greek goddess of the rainbow and divine messenger of the Olympian gods. In mythology, Iris served as the swift-footed messenger who bridged the gap between gods and mortals, traveling on the arc of the rainbow to deliver messages across vast distances. She personified the connection between heaven and earth, ensuring that communication flowed seamlessly across different realms.

This symbolism perfectly captures the essence of vLLM Semantic Router: a system that bridges the gap between users and diverse AI models, intelligently routing requests across different LLM providers and architectures. Just as Iris connected different worlds through her rainbow bridge, our router connects applications to the right models through intelligent semantic understanding. The rainbow itself—a spectrum of colors working in harmony—mirrors our vision of orchestrating multiple models in a unified, efficient system.

With the Iris release, we're establishing the foundation for reliable, intelligent, and secure AI model routing that will serve as the bridge for modern AI applications.

Q4 2025 Focus Areas​

Our Q4 roadmap centers on seven critical pillars that will transform vLLM Semantic Router from an experimental project into a production-ready platform. These initiatives address the most pressing needs identified by our community and represent the essential groundwork for v0.1.

1. Semantic Chain for Fusion Intelligent Routing​

The Challenge

Current routing relies exclusively on ModernBERT classification for semantic understanding. While powerful, this approach has limitations: it cannot perform deterministic routing based on specific keywords, lacks pattern-based detection for safety and compliance, and misses opportunities for specialized domain classification that could enhance routing accuracy and flexibility.

The Innovation

We're introducing a unified content scanning and routing framework that extends semantic routing with four complementary signal sources, all integrated through a Signal Fusion Layer:

1. Keyword-Based Routing

• Deterministic, fast Boolean logic for exact term matching
• Route queries containing "kubernetes" or "CVE-" patterns directly to specialized models
• Eliminate unnecessary ML inference for technology-specific queries

2. Regex Content Scanning

• Pattern-based detection for safety, compliance, and structured data
• Guaranteed blocking of PII patterns (SSN, credit cards) with no ML false negatives
• RE2 engine with ReDoS protection for security-critical applications

3. Embedding Similarity Scanning

• Semantic concept detection robust to paraphrasing
• Detect "multi-step reasoning" intent even when phrased as "explain thoroughly"
• Reuses existing BERT embedder for zero additional model overhead

4. Domain Classification

• In-Tree BERT Classification: Lightweight BERT-based domain classifiers running directly in the router process for low-latency intent detection
• Out-of-Tree MCP Classification: Advanced domain-specific classifiers deployed as MCP servers for specialized routing scenarios (legal, medical, financial domains)
• Hierarchical classification with confidence scoring for multi-domain queries

Dual Execution Paths

• In-Tree Path: Low-latency signal providers running directly in the router process
• Out-of-Tree Path: MCP (Model Context Protocol) servers for massive rule sets, custom matching engines (Aho-Corasick, Hyperscan), and domain-specific algorithms

Signal Fusion Layer

The decision-making engine that combines all signals into actionable routing decisions:

• Priority-based policy evaluation: Safety blocks (200) → Routing overrides (150) → Category boosting (100) → Consensus (50) → Default (0)
• Boolean expressions: Combine multiple signals with AND, OR, NOT operators
• Flexible actions: Block, route to specific models, boost category weights, or fallthrough to BERT

Impact

This framework enables:

• Fast deterministic routing for technology-specific queries
• Guaranteed compliance with safety and regulatory requirements
• Semantic intent detection that complements BERT classification
• Specialized domain classification for vertical-specific routing (legal, medical, financial)
• Flexible deployment options with both in-tree and out-of-tree execution paths
• Graceful degradation and backward compatibility with existing routing

The Semantic Chain for Fusion Intelligent Routing represents a fundamental shift from pure ML-based routing to a hybrid approach that leverages the best of deterministic, pattern-based, semantic, and domain-specific classification methods.

2. Extensible Serving Architecture: Modular Candle-Binding for MoM Family​

The Challenge

Our Rust-based candle-binding codebase has grown organically into a 2,600+ line monolithic structure. This architecture was designed for a handful of models, but now faces a critical challenge: supporting the entire MoM (Mixture of Models) Family with its diverse model architectures, specialized classifiers, and LoRA-adapted variants. The current monolithic design makes it nearly impossible to efficiently serve multiple model types simultaneously.

The Vision

We're restructuring the candle-binding into an extensible serving architecture specifically designed to support the MoM Family's diverse model ecosystem. This modular design enables seamless addition of new MoM models without code changes, efficient multi-model serving, and clear separation between model architectures and serving logic.

Layered Architecture for MoM Models

• Core Layer: Unified error handling, configuration management, device initialization, and weight loading shared across all MoM models
• Model Architectures Layer: Modular implementations of BERT (mom-similarity-flash, mom-pii-flash, mom-jailbreak-flash), ModernBERT, and Qwen3 (mom-brain-pro/max, mom-expert-* series) with extensible traits for future MoM additions
• Classifiers Layer: Specialized implementations for sequence classification (intent routing), token classification (PII/jailbreak detection), and LoRA support (fine-tuned MoM experts)
• FFI Layer: Centralized memory safety checks and C-compatible interfaces for Go integration

Impact

This extensible architecture enables:

• Rapid MoM Model Deployment: Add new MoM models (mom-expert-math-flash, mom-brain-max) by implementing standard traits
• Efficient Multi-Model Serving: Serve multiple MoM models simultaneously with shared infrastructure
• LoRA Support: Native support for LoRA-adapted MoM experts with high-confidence routing
• Backward Compatibility: Existing Go bindings continue to work without changes

This transformation positions the serving layer as a scalable foundation for the entire MoM Family ecosystem, enabling us to rapidly expand our model offerings while maintaining performance and reliability.

3. Model Unification: The MoM (Mixture of Models) Family​

The Challenge

Despite developing a comprehensive family of specialized routing models, our codebase still references legacy models scattered across configuration files. This fragmentation creates confusion, inconsistent performance, and a steep learning curve for new users.

The Solution

We're migrating the entire system to use the MoM Family as the primary built-in models:

• 🧠 Intelligent Routing: mom-brain-flash/pro/max for intent classification with clear latency-accuracy trade-offs
• 🔍 Similarity Search: mom-similarity-flash for semantic matching
• 🔒 Prompt Guardian: mom-jailbreak-flash and mom-pii-flash for security and privacy
• 🎯 SLM Experts: Specialized models for math, science, social sciences, humanities, law, and general tasks

Key Features

• Centralized Registry: Single source of truth for all MoM models with metadata, capabilities, and recommended use cases
• Simplified Configuration: Reference models by name (mom-brain-flash) instead of complex paths
• Auto-Discovery: Intelligent model detection and validation
• Performance Optimization: All MoM models are specifically trained and optimized for vLLM-SR routing tasks

This unification provides users with a clear, consistent model selection experience while ensuring optimal performance for every routing scenario.

4. Architectural Evolution: Model-Based Routing Core​

The Challenge

Our current routing implementation, inherited from traditional cluster-based approaches, has reached its architectural limits. The tight coupling between routing logic and cluster management prevents us from supporting the diverse LLM deployment scenarios that modern AI applications demand—from hybrid cloud deployments to multi-provider orchestration.

The Vision

We're reimagining our routing architecture with a clean separation of concerns: semantic routing focuses purely on intelligent model selection, while traffic management is delegated to the AI Gateway layer. This modular approach transforms the semantic router into a global external processor that operates transparently within the gateway infrastructure.

Key Capabilities

• Universal Connectivity: Support for HTTPS, HTTP, IP-based, and DNS-based connections to any LLM provider
• Hybrid Routing: Seamlessly route between in-cluster services and external providers (Claude, Gemini, DeepSeek, etc.)
• Advanced Traffic Management: Model-level failover, weighted distribution, circuit breaking, and health checks
• Enterprise Features: Built-in authentication, retry mechanisms, and token-based rate limiting

This architectural shift enables vLLM Semantic Router to scale from single-cluster deployments to global, multi-cloud AI infrastructures while maintaining the simplicity and performance that users expect.

5. Next-Generation API: OpenAI Responses API Support​

The Challenge

The traditional Chat Completions API (/v1/chat/completions) is stateless and designed for single-turn interactions. Modern AI applications—especially agents, multi-turn conversations, and agentic workflows—require stateful interactions, advanced tool orchestration, and long-running background tasks. Without Responses API support, vLLM Semantic Router cannot intelligently route these next-generation workloads.

The Vision

Add comprehensive support for the OpenAI Responses API (/v1/responses), enabling intelligent routing for stateful, multi-turn, and agentic LLM workflows while preserving all advanced features of the API.

Key Capabilities

Stateful Conversations

• Built-in conversation state management with previous_response_id chaining
• Automatic context preservation across multiple turns
• Intelligent routing that maintains conversation context and intent classification history

Advanced Tool Orchestration

• Native support for code interpreter with container management
• Function calling and tool execution routing
• Image generation and editing capabilities
• MCP (Model Context Protocol) server integration for external tools
• File uploads and processing (PDFs, images, structured data)

Agentic Workflows

• Background task processing for long-running agent operations
• Asynchronous execution with polling support for complex reasoning tasks
• Resumable streaming with sequence tracking for dropped connections
• Support for reasoning models (o1, o3, o4-mini) with encrypted reasoning items

Semantic Routing Integration

• Extract and classify intent from Responses API input field (text, messages, or mixed content)
• Apply intelligent model selection based on conversation history and tool requirements
• Route multi-turn conversations to models optimized for stateful interactions
• Preserve VSR (vLLM Semantic Router) headers for routing metadata across response chains

Impact

Responses API support positions vLLM Semantic Router at the forefront of agentic AI infrastructure:

• Enable routing for modern agent frameworks and multi-turn applications
• Support complex workflows requiring code execution, file processing, and external tool integration
• Provide intelligent model selection for reasoning-heavy tasks and long-running operations
• Maintain semantic router's value proposition (cost optimization, latency reduction) for next-generation LLM APIs

This capability is essential for vLLM Semantic Router to remain relevant as the industry shifts from simple chat completions to sophisticated, stateful, tool-augmented AI agents.

6. Intelligent MCP Gateway: Smart Tool Management and Selection​

The Challenge

As AI agents increasingly rely on external tools and services through the Model Context Protocol (MCP), managing and selecting the right tools for each task becomes critical. Current approaches lack intelligent tool discovery, selection optimization, and centralized management, leading to inefficient tool usage and increased latency in agentic workflows.

The Innovation

We're introducing an Intelligent MCP Gateway that serves as a unified control plane for MCP tools with smart selection capabilities:

MCP Tool Management

• Centralized Registry: Unified catalog of available MCP servers and tools with metadata, capabilities, and performance characteristics
• Dynamic Discovery: Automatic detection and registration of MCP servers in the cluster
• Health Monitoring: Real-time health checks and availability tracking for all registered MCP tools
• Version Management: Support for multiple versions of MCP tools with seamless upgrades and rollbacks

Intelligent Tool Selection

• Semantic Matching: Analyze user intent and task requirements to automatically select the most appropriate tools
• Context-Aware Routing: Consider conversation history, user preferences, and task complexity for tool selection
• Performance Optimization: Route tool requests based on latency, cost, and success rate metrics
• Fallback Strategies: Automatic failover to alternative tools when primary options are unavailable

Integration with Fusion Routing

• Seamlessly integrate with the Semantic Chain for unified routing decisions
• Combine tool selection with model selection for optimal agentic workflows
• Support both in-tree (low-latency) and out-of-tree (MCP server) tool execution paths

Impact

The Intelligent MCP Gateway enables:

• Simplified tool management for complex agentic applications
• Reduced latency through intelligent tool selection and caching
• Improved reliability with automatic failover and health monitoring
• Enhanced developer experience with centralized tool discovery and configuration
• Cost optimization by routing to the most efficient tools for each task

This gateway positions vLLM Semantic Router as a comprehensive orchestration layer for modern AI agents, managing not just model selection but also the tools and services that agents rely on.

7. Enterprise Readiness: Production Deployment Tools​

The Challenge

While vLLM Semantic Router works well for experimental deployments, production adoption requires professional-grade deployment tools, comprehensive monitoring, and intuitive management interfaces.

The Deliverables

Helm Chart Support​

Professional Kubernetes deployment with:

• Templated manifests for all resources
• Values-driven configuration for different environments
• Built-in versioning and rollback capabilities
• Best practices for security, scaling, and resource management

Modern Management Dashboard​

A comprehensive web-based control plane featuring:

• Visual Route Builder: Drag-and-drop interface for creating SemanticRoute configurations
• Interactive Playground: Test routing decisions, compare models, and visualize filter chains
• Real-time Monitoring: Live metrics, request tracing, and health status
• Analytics & Insights: Cost analysis, performance benchmarks, and routing effectiveness
• User Management: Role-based access control, API key management, and audit logs

These enterprise features will dramatically lower the barrier to entry, improve operational efficiency, and make vLLM Semantic Router accessible to organizations of all sizes.

Ecosystem Integration​

Beyond the seven core pillars, we're actively exploring integrations with key platforms in the AI infrastructure ecosystem. These five integrations are work-in-progress and good-to-have features that will expand vLLM Semantic Router's reach and interoperability:

vLLM Production Stack​

vLLM Production Stack is vLLM's reference system for Kubernetes-native cluster-wide deployment with community-driven performance optimization. It provides a reference implementation on how to build an inference stack on top of vLLM with Helm chart-based deployment.

Deep integration with the vLLM Production Stack will enable seamless model serving, monitoring, and orchestration. This integration will provide native support for vLLM's advanced features like PagedAttention, continuous batching, and optimized CUDA kernels, ensuring maximum performance for production workloads.

llm-d​

llm-d is a Kubernetes-native high-performance distributed LLM inference framework built on vLLM. Founded by Red Hat, Google Cloud, CoreWeave, and IBM Research, with contributions from NVIDIA, Hugging Face, Intel, Lambda, and Mistral AI, llm-d provides well-lit paths for anyone to serve large generative AI models at scale with distributed inference capabilities.

Integration with llm-d will bring intelligent semantic routing to Kubernetes-native distributed inference deployments. This partnership will enable llm-d users to leverage MoM models and fusion routing for efficient model selection across distributed inference clusters, optimizing resource utilization and performance in cloud-native environments.

HuggingChat​

HuggingChat is Hugging Face's open-source conversational AI platform powered by state-of-the-art open models. It democratizes access to advanced AI chat capabilities without requiring API keys or paid subscriptions.

Integration with HuggingChat will bring intelligent semantic routing to conversational AI applications. This partnership will enable HuggingChat users to leverage MoM models and fusion routing capabilities, providing cost-effective and high-performance model selection for chat-based workloads.

NVIDIA Dynamo​

NVIDIA Dynamo is NVIDIA's high-performance, low-latency inference platform that supports all major frameworks including TensorRT-LLM. It delivers scalable, efficient inference with GPU optimization and includes intelligent inference optimizations that boost token generation performance by over 30x per GPU, with support for advanced features like disaggregated serving.

Integration with NVIDIA Dynamo will leverage cutting-edge GPU acceleration and optimization frameworks to deliver industry-leading latency and throughput for semantic routing operations. This partnership will enable seamless deployment of MoM models on NVIDIA-accelerated infrastructure with optimal performance.

vLLM AIBrix​

vLLM AIBrix is an open-source initiative designed to provide essential building blocks to construct scalable GenAI inference infrastructure. As a cloud-native framework, AIBrix serves as an infrastructure orchestrator and workload control plane, offering cost-efficient and pluggable components for large-scale LLM serving with simplified deployment and management.

Collaboration with vLLM AIBrix will enable unified control planes, advanced observability, and streamlined deployment workflows across hybrid and multi-cloud environments. This integration will make it easier for enterprises to adopt and scale vLLM Semantic Router with production-ready infrastructure components.

These ecosystem integrations represent our commitment to building an open, interoperable platform that works seamlessly with the broader AI infrastructure landscape. While not required for the v0.1 release, they demonstrate our vision for vLLM Semantic Router as a foundational component in modern AI stacks.

Timeline and Release Plan​

v0.1 "Iris" Release (Late 2025 - Early 2026):

• All P0 priority issues resolved
• Seven foundational pillars fully implemented
• Comprehensive documentation and migration guides
• Production-ready deployment tools (Helm charts, dashboard)
• Full Responses API, Intelligent MCP Gateway, and Semantic Chain for Fusion Intelligent Routing support
• Community celebration and feedback collection

Looking Beyond Iris​

The Iris release establishes the foundation, but our vision extends far beyond v0.1. Future releases will introduce:

• Advanced multi-model orchestration strategies
• Federated routing across distributed clusters
• Enhanced reasoning capabilities and chain-of-thought routing
• Deeper integration with the broader vLLM ecosystem

Each release will carry its own mythological codename, reflecting the unique character and capabilities it brings to the project.

Get Involved​

This roadmap represents our commitment to building production-ready AI infrastructure, but we can't do it alone. We invite the community to:

• Review and provide feedback on the P0 issues
• Contribute code to any of the initiatives
• Test early releases and share your experiences
• Suggest improvements to the roadmap

Together, we're building the bridge that will connect the next generation of AI applications to the models they need—just as Iris connected the realms of gods and mortals.

Follow our progress:

• GitHub: vllm-project/semantic-router
• Issues: P0 Priority Issues

The rainbow bridge awaits. Let's build it together. 🌈

Synced from official vLLM Blog: vLLM Semantic Router: Next Phase in LLM inference

Industry Status: Inference ≠ More Is Better​

Over the past year, hybrid reasoning and automatic routing have increasingly defined progress in large-model infrastructure—shifting the debate from raw scale to per-token efficiency, latency control, and targeted compute use.

Take GPT-5 for example: its standout innovation lies not in sheer parameters, but in routing policies and quota-based reasoning:

• Light queries → lightweight paths: trivial prompts like “Why is the sky blue?” don’t trigger expensive reasoning.
• Complex/high-value queries → reasoning-enabled models: multi-step tasks—like legal analysis or financial planning—are routed to Chain-of-Thought–enabled inference.

This represents a broader principle of task-aware compute allocation, where every inference token must contribute meaningful value—not just be consumed.

Similar ideas are appearing in other systems:

• Anthropic Claude 3.7/4: differentiates “fast thinking” and “slow thinking” pathways.
• Google Gemini 2.5: offers explicit thinking budgets, allowing enterprises to cap reasoning depth.
• Alibaba Qwen3: supports instruction-driven switching between reasoning and non-reasoning modes.
• DeepSeek v3.1: merges conversational and reasoning flows within a dual-mode single model.

The trend is clear: future inference systems will be defined by selectivity and intelligence, not just model size.

Recent Research: vLLM Semantic Router​

Responding to this shift, the vLLM Semantic Router offers an open-source, intent-aware routing layer for the highly efficient vLLM inference engine.

vLLM enables scalable LLM serving—but lacks semantic decision-making around reasoning. Developers face a trade-off:

• Enable reasoning always → accuracy increases, but so does cost.
• Disable reasoning → cost drops, but accuracy suffers on complex tasks.

The Semantic Router fills this gap by classifying queries semantically and routing them appropriately, giving accurate results where needed and efficiency where reasoning is unnecessary.

Architecture Design​

The system comprises four pillars:

1. Semantic Classification: Uses ModernBERT—currently a lightweight, standalone classifier integrated into the router—to determine routing paths.
2. Smart Routing:
   • Simple queries → "fast path" inference.
   • Complex queries → "Chain-of-Thought" reasoning mode.
3. High-Performance Engine: Written in Rust using Hugging Face Candle, it delivers high concurrency and zero-copy inference.
4. Cloud-Native Integration: Works out-of-the-box with Kubernetes and Envoy via the ext_proc plugin.

In trials, this design yielded:

• ~10% higher accuracy
• ~50% lower latency
• ~50% fewer tokens

In business and economics domains, gains exceeded 20% accuracy improvements.

Challenges in Execution: Budgets and Tool Calling​

Two technical constraints are important to address:

• Reasoning Budget Costs
  Unlimited reasoning inflates cold-start latency and resource usage. Without dynamic control, simple queries may over-consume tokens while critical queries may not get deep reasoning when needed. SLOs like TTFT and p95 latency are necessary—with possible adaptation mid-inference.
• Tool Calling Constraints
  Adding more tools (i.e. “tool catalog bloat”) or longer tool outputs can drastically reduce accuracy. The router must pre-filter tools and keep catalogs tight.

Project Background​

The Semantic Router evolved from contributions across the open-source community:

• Proposed in early 2025 by Dr. Chen Huamin (Red Hat)
• Further developed by Xunzhuo Liu (Tencent)
• To be presented by Dr. Wang Chen (IBM Research) and Dr. Chen Huamin at KubeCon North America 2025

Our goal: provide inference acceleration for open-source LLMs through:

• Semantic-aware routing
• Efficient model switching
• Enterprise-friendly deployment (Kubernetes & Envoy)

Find the project on GitHub. The current focus is on a Work Group and planned v0.1 Roadmap.

Integration & Future Work: Embeddings and Pluggability​

Currently, ModernBERT runs internally within the router for classification. It is not yet served by vLLM. However, future work aims to make the classifier—and potentially other embedding models—pluggable, allowing integration with vLLM-hosted models or external embedding services.

This capability will enhance the semantic cache and enable smoother inference customization.

Roadmap: v0.1 Milestone Highlights​

The v0.1 milestone will expand the project’s technical capabilities:

• Core: ExtProc-based modularity, semantic caching across backends, multi-factor routing logic
• Benchmarking: CLI tools, performance testing suite, reasoning-mode evaluation
• Networking: Deeper integration with Envoy, GIE, and llm-d gateways
• Observability & UX: Admin dashboards, routing policy visualization, developer quickstarts, and policy cookbook

Future Trends: Just-in-Time Inference​

The field is maturing from “Can we run inference?” to “How can inference be smarter?”

• GPT-5 uses commercial value to guide reasoning depth.
• vLLM Semantic Router delivers that capability to open source.

Looking ahead, systems that adapt their inference strategy on the fly, without manual toggles, will lead in efficiency, latency, and sustainability.

One-Sentence Summary​

• GPT-5: enterprise routing for smarter inference
• vLLM Semantic Router: technical-first routing for open-source LLMs
• Edge future: context-aware, minimal-compute inference that works seamlessly