Skip to main content
Version: Latest (main)

Connect an agent harness

Point your harness at a Router entrypoint such as vllm-sr/auto. The harness owns the agent loop, tools, and task state; the Router selects models or runs a bounded multi-model workflow under your policy. Inference backends execute the model calls.

Install the Router first, or ask an agent to install it.

Choose the inference connection​

Configure the harness's model provider. Setting names vary by harness version.

SettingWhat to use
Inference addressThe public inference listener or gateway. The local Quickstart uses http://localhost:8899; the management API and Dashboard have different addresses.
Client protocolOpenAI Chat Completions, OpenAI Responses, or Anthropic Messages, matching the requests your harness sends.
Public model IDvllm-sr/auto for the default policy, or a published entrypoint from GET /v1/models.
AuthenticationThe credentials required by your inference listener or gateway. Backend provider credentials are configured separately by the operator.
Model limits and capabilitiesContext, output, tools, vision, and reasoning settings that the selected recipe can support.

Clients may append /v1 themselves. Check that the final path matches:

Client APIFinal request pathRequirement
Chat CompletionsPOST /v1/chat/completionsSend the Chat Completions request shape.
ResponsesPOST /v1/responsesThe Router's global.services.response_api and its store must be available.
MessagesPOST /v1/messagesSend the Messages request shape and an appropriate anthropic-version header.

Client and backend protocols can differ, within the compatibility matrix. Free-form custom tools cannot reach a Messages backend; Codex CLI has additional limits listed there. Keep credentials in the harness's supported secret store or environment bindings.

Verify the public model​

Configure at least one backend, then list the local stack's public models:

curl -sS http://localhost:8899/v1/models

Use a virtual entrypoint to apply routing policy. Concrete provider model names bypass recipe routing.

Send a minimal request and include response headers:

curl -sS -i http://localhost:8899/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "vllm-sr/auto",
"messages": [{"role": "user", "content": "Hello!"}]
}'

For other deployments, use their inference address and authentication. Check the answer plus x-vsr-selected-recipe and x-vsr-selected-model; HTTP success alone does not verify the output or route. See routing receipts and CLI previews and probes.

Model names the harness picks itself​

A harness can request models other than the one you configured. Claude Code, for example, asks for its own Haiku model ID when it starts an Explore subagent. The Router answers a model it does not publish with "the requested model is not available", and that subagent fails. Point every model setting the harness has at an entrypoint (for Claude Code, ANTHROPIC_MODEL and the ANTHROPIC_DEFAULT_*_MODEL variables), or list the IDs it sends in the entrypoint's model_names:

entrypoints:
- model_names: [vllm-sr/auto, claude-haiku-5-5]
recipe: default

Claude Code and Codex CLI​

Claude Code on a Claude subscription runs Anthropic's models, and the vSR plugin adds guard, shield, redact and check to it without a gateway. Route Claude Code or Codex CLI through the Router when your own models or provider keys should answer: the Router then chooses the model for every call, the main conversation included, while vSR keeps judging tool output and shell commands inside Claude Code. Codex reaches the Router the same way, with its own vSR plugin.

The two harnesses need different Router-wide settings: the losses the Router accepts, the headers that identify a session and a subagent, and defaults for what the harness leaves out. Start with one Router for each; both can use the same backends and decision model. This was checked with Claude Code 2.1.322 with vSR 0.1.10, Claude Code 2.1.317 without it, and Codex CLI 0.166.0, against anthropic provider models served by vLLM 0.32.0.

Connect with vllm-sr connect​

export VLLM_SR_API_KEY=<the listener's API key> # leave unset when it has none
vllm-sr connect claude-code --config claude-code.yaml
vllm-sr connect codex --config codex.yaml
codex --profile vllm-sr

Here each harness has its own Router, served from claude-code.yaml and codex.yaml. The command reaches the gateway at the first listener of the configuration it is given (config.yaml by default); --base-url names another origin.

The command shows the changes and asks before writing them:

  • Claude Code: the env block of ~/.claude/settings.json (or $CLAUDE_CONFIG_DIR/settings.json; --scope project writes .claude/settings.local.json). The other settings stay as they are. The file holds the key when one is set, so only you can read it afterwards. In a git repository, --scope project adds the file and the command's undo record to .git/info/exclude when git would otherwise track them, and refuses to write a key into either one once git already tracks it. ANTHROPIC_API_KEY is set empty: otherwise Claude Code also sends a key your shell exports, as x-api-key beside the gateway's token.
  • Codex: the ~/.codex/vllm-sr.config.toml profile, which holds a vllm-sr provider, selects it and turns off hosted web search, which the Router does not serve. ~/.codex/config.toml is not touched.

Neither harness knows vllm-sr/auto, so the command gives them its context window: the smallest context_window_size of the models the entrypoint can reach in the configuration, which are the models its recipe's decisions name and providers.defaults.model (a catalog-backed model without one takes its card's limit). Image and System One models are left out. It says where the number came from and writes it as CLAUDE_CODE_MAX_CONTEXT_TOKENS for Claude Code and model_context_window for Codex. When a model states no window, the command names it and sets none: Claude Code then assumes a window of its own (1,000,000 tokens in 2.1.323) and says so at start, and Codex falls back to default model metadata. --context-window sets the window either way.

It then reports three checks, each on its own line:

  • Router configuration: what the configuration given with --config lacks for this harness, from the settings below, without changing it.
  • Basic reply: a plain request in the harness's protocol gets an answer, and the line names the model that gave it.
  • Streamed tool loop: a streamed tool call, then the answer once its result goes back, as the harness sends them. A stream that ends in an error fails it, and the check runs only after the basic reply passed.

On a failure the written changes stay, so fix the gateway and run the command again, or take them back with --undo. With the vSR plugin installed it warns while the plugin's router app is on, and when the listener also serves System One it prints the setting that points vSR at it. --dry-run prints the plan and writes nothing.

After writing, the command names any settings that win over what it wrote, and how to stop them. For Claude Code that is a .claude/settings.json or .claude/settings.local.json in the current directory that sets one of its env keys: Claude Code reads project settings only in the directory it starts in, and they win over the user's. Variables your shell exports do not win over the settings' env. For Codex it is a .codex/config.toml from the repository's top down to the current directory that sets model: once Codex trusts the project, it asks for that model instead of the profile's. OPENAI_BASE_URL does not change the vllm-sr provider. These were measured with Claude Code 2.1.323 and Codex CLI 0.166.0.

--status-line also sets Claude Code's status line to show which models answered this session, the last first, such as vllm-sr ⇢ qwen36 (4) · qwen38-27b (2). Claude Code asks for vllm-sr/auto, so its own model display names only the entrypoint. The line reads the session's transcript and its subagents' transcripts, where each reply keeps the model that wrote it, so it needs no Router access. The script sits beside the settings as vllm-sr-status-line.py, and --undo puts back the status line you had before. --undo puts back what each key, or the Codex profile, held before the first run, leaves alone what changed since, and keeps everything else in the files.

Try it without writing anything​

vllm-sr connect claude-code --config claude-code.yaml --launch
vllm-sr connect codex --config codex.yaml --launch -- exec "fix the failing test"

--launch runs the same checks and the same probe, then starts the harness through the gateway for that one session and exits with its status. Nothing is written: Claude Code gets the settings with --settings from a temporary file only you can read, which holds the key and is removed when Claude Code exits, and Codex gets them as -c overrides and reads the key from VLLM_SR_API_KEY, so it never appears on the command line. Both win over the user's and the project's settings, so the overrides above do not apply. Arguments after -- go to the harness, and a gateway that fails the probe means the harness is not started.

Router settings for Claude Code​

global:
router:
allow_lossy_translation: [tools.defer_loading]
learning:
enabled: true
protection:
enabled: true
scope: session
identity:
headers:
session: x-claude-code-session-id
agent: x-claude-code-agent-id

Claude Code's tool search marks tools for loading on demand. Without tools.defer_loading the Router refuses the first calls of every session, and Claude Code goes on without tool search. Toward an openai backend, also list tool_result.is_error, messages[].role and content.file, as protocol compatibility describes. An anthropic backend receives those as sent.

Claude Code on its own Claude login​

Claude Code signed in to a Claude subscription can use the gateway without an API key: give every backend auth: {credential: caller} (Security Hardening), and point Claude Code at the listener without a gateway token, so it sends its own login:

env -u ANTHROPIC_API_KEY -u ANTHROPIC_AUTH_TOKEN -u CLAUDE_CODE_OAUTH_TOKEN \
ANTHROPIC_BASE_URL=http://localhost:8899 \
ANTHROPIC_MODEL=vllm-sr/auto \
ANTHROPIC_DEFAULT_HAIKU_MODEL=vllm-sr/auto \
ANTHROPIC_DEFAULT_SONNET_MODEL=vllm-sr/auto \
ANTHROPIC_DEFAULT_OPUS_MODEL=vllm-sr/auto \
CLAUDE_CODE_MAX_CONTEXT_TOKENS=1000000 \
claude

CLAUDE_CODE_MAX_CONTEXT_TOKENS is the smallest context window among the models the entrypoint reaches; without it Claude Code does not know vllm-sr/auto and says so at start. vllm-sr connect claude-code --config <your config> writes the same settings for such a configuration: it leaves ANTHROPIC_AUTH_TOKEN empty, so neither a gateway key nor one your shell exports replaces the login, and it does not probe the gateway, which answers only on that login.

The Router then routes each call among Claude models on that login, and Insights shows the session's calls with the model each decision picked. A model's pricing is the API's list price, not the plan's, so leave it out or read the cost as an API-price estimate. The login stays its owner's: serve this to your own Claude Code, or to your organization's members, never to other people as a service (Security Hardening).

Router settings for Codex CLI​

recipes:
- name: default
decisions:
- name: agents
# rules and modelRefs as in your recipe
plugins:
- type: request_params
configuration:
default_max_tokens: 32000
global:
router:
allow_lossy_translation: [reasoning.signature]
learning:
enabled: true
protection:
enabled: true
scope: session
identity:
headers:
session: session-id
agent: thread-id
services:
response_api:
enabled: true
store_backend: memory

Codex sends no output limit, so a decision that sends it to an anthropic backend sets one. Thinking from an anthropic backend carries a signature that a Responses client cannot receive: without reasoning.signature the stream ends in a lossy_translation error where the thinking block closes, and Codex reconnects until it gives up. The Responses endpoint needs the Router's Responses service and store. See Codex CLI for the other fields Codex sends.

Codex on its own ChatGPT login​

Codex signed in with ChatGPT can use the gateway without an API key as well: give every backend provider: openai, the endpoint https://chatgpt.com/backend-api/codex and auth: {credential: caller} (Security Hardening), and add requires_openai_auth = true to the gateway's provider in the Codex profile, so Codex sends its own login with the request:

[model_providers.vllm-sr]
name = "vLLM Semantic Router"
base_url = "http://localhost:8899/v1"
wire_api = "responses"
requires_openai_auth = true

Keep web_search = "disabled" in the profile: the Router takes function tools only. vllm-sr connect codex --config <your config> writes this profile for such a configuration, with requires_openai_auth and without a key variable, and does not probe the gateway, which answers only on Codex's own login. Each connect command stops before writing anything for a gateway whose caller backends name the other provider: the Router takes an anthropic login only on Messages and an openai one only on Responses or Chat, so every call would be refused. The plan's usage limits apply to each call, and the login stays its owner's, as for Claude Code above.

With the vSR plugin​

One listener can serve both systems: the routed model calls for Claude Code and, at /v1/decisions, the System One questions vSR asks. Publish the decision model on the listener and point vSR at it, so the plugin and the gateway share one address, one API key and one decision model. A listener that names its apis serves only those, so list the model APIs the harnesses use next to systemone:

listeners:
- name: http
address: 0.0.0.0
port: 8899
apis:
messages: {} # Claude Code
responses: {} # Codex CLI
chat_completions: {}
systemone:
models: [vela3-lite] # the public names vSR may ask for
global:
model_catalog:
deployments:
primary:
provider: model_runtime
public_name: vela3-lite
artifact: vllm-sr/vela3-lite

vSR's apps are measured on vela3-lite. Models of the runtime's vela3 family, such as vela3-fast, answer no score question, which vSR's router app asks, and score risky commands far below the line guard holds at, so publish vela3-lite for vSR.

vllm-sr connect claude-code --config router.yaml
vllm-sr connect codex --config router.yaml --vsr-source /path/to/semantic-router

When the listener publishes a System One model, one command configures the gateway (System Two) and vSR (System One); --no-vsr leaves vSR out. The plugin installs from --vsr-source, this repository on GitHub by default, in the agent's own CLI (claude plugin or codex plugin), and only when it is missing. For Claude Code the command then sets the plugin's options through claude plugin configure: a remote runtime, the gateway's /v1/decisions, the first published System One model, and router: false so the gateway chooses the models. When the API key variable is set it also passes runtimeToken, which Claude Code keeps in its credential store, not in settings.json; the preview names the variable and never shows the key. For Codex it writes runtimeUrl and runtimeModel to $CODEX_HOME/vsr.json (default ~/.codex/vsr.json), keeping its other settings.

Both harnesses preview everything with --dry-run, the plugin commands included, which run only when writing. --undo sets back the plugin options the command changed (an option that was never set returns to its default) but leaves the plugin installed and the token in place. The gateway settings are written even when vSR cannot be: without the agent's CLI on PATH, when the plugin's options cannot be read, or when the plugin holds a token from before and no key is given (Claude Code cannot clear a stored token, and the plugin would send it to the new listener), the command says why and what to run. --launch writes nothing, so it leaves vSR alone.

The listener can also set which of vSR's apps run, for every plugin pointed at it, under systemone.vsr; the Dashboard shows each field as Default, On or Off under Config › Global Config › vSR agent plugins, where Default leaves the field out. Each field set there wins over the plugin's own option, so turning redact off here turns it off in every session that uses this listener, from its next prompt a minute or more later. A choice a person makes in Claude Code with /vsr stays theirs until /vsr follow:

systemone:
models: [vela3-lite]
vsr: {redact: true, guard: true, router: false}

Rules of your own for guard and redact​

vsr.rules binds rules of one recipe to guard and redact. A rule is an ordinary signal of that recipe: guard takes any keyword rule and a System One noul rule, or choice rule with labels; redact takes a rule that finds text, an OR keyword rule of literal or regex keywords without fuzzy_match, count or density, or a System One span rule. An entrypoint must reach the recipe, since the Router prepares the rules of a reachable recipe only.

systemone:
models: [vela3-lite]
vsr:
rules:
recipe: default
guard:
- {type: keyword, name: prod_database, enforce: required}
- {type: systemone, name: deletes_data}
redact:
- {type: keyword, name: employee_id, enforce: required}

In the Dashboard, Config › Global Config › vSR agent plugins › Rules of your own picks the recipe and lists its rules that fit each app, each Off, Advisory or Required. Once an app binds a rule, the rules left Off fold behind a "Show N more rules" button. Try it under each app sends a sample command or output through the Dashboard, which adds the listener's key on the server, and says which rules match or what they would hide, as the Router runs the saved rules.

enforce: required makes a rule fail closed for the plugin that applies it: when the rule cannot be checked, the command waits for the person and the output is withheld. advisory, the default, lets an unchecked rule step aside as the plugins' own checks do. An app binds at most 16 rules, each name once.

The listing (GET /v1/systemone/models) hands each rule's name, description and required to the plugins under vsr.rules, with a revision that changes with any rule's definition, binding or enforcement. A plugin that gets an answer for another revision, such as after a change in the Dashboard, reads the listing again and asks once more in the same call, so a change applies to a running session at once. A rule's pattern or question never leaves the Router. To evaluate them, a client sends the texts to the same listener, with the same key as any System One call:

curl -s http://localhost:8899/v1/systemone/rules \
-H 'Authorization: Bearer <listener key>' \
-d '{"app": "redact", "parts": [{"id": "out-1", "text": "Paid by E-104233"}]}'
{"revision": "3f9c0a1b2d4e", "parts": [{"id": "out-1", "coverage": "complete",
"matches": [], "spans": [{"start": 8, "end": 16, "rule": "employee_id"}], "unchecked": []}]}

Each part comes back with the guard rules it matches, or the redact spans in code points, overlapping spans of different rules merged into one under the rule that starts first, and the rules the Router could not check with a reason. coverage is complete, partial or failed. A request holds at most 16 parts and 256 KiB of text; a listener evaluates 8 such requests at once and answers 429 with Retry-After beyond that, 404 rules_not_published for an app it binds no rules to, and 401 without its key. Every call is logged with the rule names, counts and latency, never the text.

The vSR plugins for Claude Code and Codex read these rules from the listing and call this endpoint themselves. Guard sends the whole command beside its own questions and holds it when a rule matches; redact sends the tool output as the tool gave it and puts [vsr:<rule name>] where a rule found text. A required rule that cannot be checked holds the command or withholds the output in both; the rules last read from a listener are kept, so an outage does not drop one once a plugin has read the listing. Before that first read a plugin knows no rules, and its checks step aside when the listener does not answer. Turning guard or redact off for a session turns their rules off too, so a protection nobody may turn off belongs in a decision or in masking at the gateway.

A Router rule about what the agent reads, such as a prompt injection in a file or a web page, should read the tool results and leave the user turn out (input: {tool_results: true, request: false}). Claude Code's user turn carries its own instructions and reminders, and a rule that reads it also judges those.

Turn the plugin's router app off (/vsr router off) so the plugin and the Router do not both choose a subagent's model, and with it the plugin's routing of your own turns (routerTurns), which is for Claude Code talking to Anthropic directly. Its escalate app still asks Claude to re-check an answer you push back on, but no longer moves the next subagent up a model, since that step belongs to the router app. The other apps work as before. When the plugin's decision runtime does not answer, its tally says model offline and its checks step aside while Claude goes on, except guard's fixed rules, which still hold, and an app set to wait until done, which withholds the output or holds the command instead. A SKIPPED row says what one check left unchecked. Both are explained in the plugin README.

Settings by hand​

The command writes these. Claude Code reads them from its settings:

{
"env": {
"ANTHROPIC_BASE_URL": "http://localhost:8899",
"ANTHROPIC_AUTH_TOKEN": "<the listener's API key, or any value without one>",
"ANTHROPIC_API_KEY": "",
"ANTHROPIC_MODEL": "vllm-sr/auto",
"ANTHROPIC_DEFAULT_HAIKU_MODEL": "vllm-sr/auto",
"ANTHROPIC_DEFAULT_SONNET_MODEL": "vllm-sr/auto",
"ANTHROPIC_DEFAULT_OPUS_MODEL": "vllm-sr/auto",
"CLAUDE_CODE_MAX_OUTPUT_TOKENS": "32000"
}
}

Codex reads the profile from ~/.codex/vllm-sr.config.toml, which codex --profile vllm-sr layers over ~/.codex/config.toml, and the provider can live in the same file:

model = "vllm-sr/auto"
model_provider = "vllm-sr"
web_search = "disabled"

[model_providers.vllm-sr]
name = "vLLM Semantic Router"
base_url = "http://localhost:8899/v1"
wire_api = "responses"
env_key = "VLLM_SR_API_KEY"

Leave env_key out for a listener without keys: Codex refuses to start while the variable it names is unset, and vllm-sr connect writes it only when VLLM_SR_API_KEY is set.

Codex does not know vllm-sr/auto, so without a catalog entry for it, it warns that it uses fallback model metadata. vllm-sr connect writes one next to the profile, vllm-sr.models.json, named by model_catalog_json. The file replaces Codex's own catalog, so it holds the installed Codex's bundled entries (from codex debug models --bundled) and adds one for the model: the gpt-5.5 entry under the model's name, with tool search and the apply_patch custom tool turned off, since the Router does not serve the first and a Messages backend refuses the second, with Codex's fallback reasoning settings, and without gpt-5.5's upgrade prompt and Fast tier, which belong to that model. Codex then edits files through its shell. A model Codex already knows, such as --model gpt-5.5, keeps its own entry, and no catalog is written. A Codex that bundles no gpt-5.5 entry gets no catalog and keeps the warning. Run vllm-sr connect codex again after updating Codex, so the catalog follows its release. vllm-sr connect also sets model_context_window and the entry's window from the configuration, or from --context-window, so the budget follows your backends.

Codex 0.166 also says it runs without its shared background server: a profile, like -c overrides, starts a Codex process of its own. The session works as usual.

The Codex TUI sends a namespace tool on every turn. A Router that refuses namespace tools answers its first turn with unsupported_tool; on such a Router, run codex exec with [features] multi_agent = false instead.

Set budgets before running a task​

GET /v1/models discovers virtual names and routing metadata; it does not advertise their context windows or output limits. Configure these in the harness. To keep every candidate usable, take the smallest context window and output limit across the recipe's selectable models, including the configured default, and enable only their shared capabilities.

For heterogeneous pools, recipe-wide candidate_requirements can check output budgets and declared tool, reasoning, and structured-output support before scoring. This does not configure the harness's own context management. Follow Virtual Models: limits for agent clients for the example and exact eligibility behavior.

A listener's timeout (300 seconds by default) bounds a whole call, the streamed response included, and arriving output does not extend it. A call that reaches the limit is cut off: the stream is reset and the rest of the answer never arrives.

Long reasoning calls hit the listener timeout

For agent models, set the model's total_timeout to 0s and keep idle_timeout to bound a stalled stream; see model configuration. In live Terminal-Bench 2.1 runs of Claude Code 2.1.304 at xhigh effort on Qwen3.8-27B-FP8 (vLLM 0.32, one MI300X), 2.5% to 4% of calls took more than 300 seconds, and the longest about 17 minutes.

Preserve tool and conversation continuity​

Run a real tool loop: model requests a tool → harness executes it → next call carries the matching result. For streaming, check complete tool arguments, the terminal event, and final answer. Protocol compatibility alone does not verify the model and harness together.

When Router Learning protection is configured, it needs an explicit session identity. Send both headers on related calls:

x-session-id: harness-demo-session
x-conversation-id: harness-demo-conversation

These are example values:

  • Use real identities. Send distinct, stable identifiers for your sessions and conversations, from the harness or a trusted gateway.
  • With only the session header, scope: session holds the whole session, and the default scope: conversation holds each conversation, named by how it starts.
  • Without the session header the request stays routable, but no model is kept through protection.
  • Responses history, including previous_response_id, does not replace these headers.

See Session identification for identity priority and privacy boundaries.

When the task changes phase, the harness can ask for a fresh routing decision in the same session: it sends x-vsr-protection-release: 1 on the first request of the new phase, at a user turn rather than inside a tool loop. A refused release returns its reason in x-vsr-warnings. See Explicit release.

Send the release header on one request only

Don't set it through Claude Code's ANTHROPIC_CUSTOM_HEADERS, which sends it on every request.

Check the selected model on tool continuations and user follow-ups. The harness retains tool permissions, task state, and the outer loop; recipes do not configure delegated agents.

Claude Code sends x-claude-code-session-id instead of x-session-id, and Codex CLI sends session-id and thread-id. See Agents that delegate to subagents for the identity headers that hold their runs and their subagents.

Agents that send tool output as messages​

Some harnesses, such as mini-SWE-agent, don't use the protocol's tool calls. The model writes a command in its reply, and the harness sends the command's output back as the next user message. The Router can't tell that output from a new user turn, so it never sees a tool loop, and signals read the output as if the user had written it.

Under scope: conversation, a decision that changes on such an output releases the current model, and min_turns_before_switch does not hold it. Use scope: session with a new session ID for every agent run instead:

  • A changed decision alone then keeps the run on its model.
  • Idle timeout, a bypass decision, candidate exclusion or a qualified rescue can still release it.
  • To restart a task on another model, start a new run with a new session ID, or name the concrete model, which bypasses recipe routing.

Program the policy behind the entrypoint​

Connect models and publish a recipe behind the same entrypoint. Signals, decisions, and algorithms define which models are eligible and how they run.

The Agent Routing recipe is a starting point for local, specialist, and frontier model lanes. Review its model bindings, decision-model questions, and replay retention before using it; its checked-in configuration records request and response data. Use the recipe evaluation workflow to test representative tasks before adopting a policy change.

Decide reasoning in the decision as well. For a model with a reasoning family, such as a catalog model, the decision's modelRefs entry chooses whether the backend thinks and at what effort; the harness's own request does not:

Decision's modelRefs entryWhat the Router sends
use_reasoning left outReasoning off
use_reasoning: trueThe decision's reasoning_effort, or the family default when none is set

This holds whatever effort the harness asks for. Claude Code, for example, asks for adaptive thinking with its effort setting on every call. Set use_reasoning: true and the effort you want on the decisions behind an agent entrypoint. See Model Reasoning.

Keep agent calls on a warm prompt cache​

An agent resends its conversation on every call, so most of each prompt is a prefix the backend has already processed. Whether that prefix is still cached decides much of a call's latency and cost, and three things lose it.

  • A model change. A prompt cache belongs to one model, so a switch in the middle of a run computes the whole context again on the new model. Keep each run on one model as described above. The multi_factor selector also favors the model that served the session's previous turn.
  • A replica change. Each replica keeps its own prefix cache, and a model's lb_policy (round_robin or least_request, see model configuration) does not keep a session on one replica. When several replicas serve agent sessions, route the model to a pool whose scheduler places requests by prefix cache and load, for example llm-d (an experimental integration), instead of listing the replicas as backend_refs.
  • Eviction. When the sessions in flight need more KV cache than the GPU holds, a returning session finds its prefix evicted. A CPU offload tier on the backend, such as LMCache for vLLM, returns those prefixes instead of computing them again.

The harness can also start the prompt over itself. When it compacts its history, replacing earlier turns with a summary to stay within the context window, the backend computes most of the new prompt again on the same model and replica. The Router neither causes nor prevents this, so a drop in prefix reuse right after a compaction is not a routing or placement fault.

Leave cross-model KV transfer (global.integrations.kv_transfer) off for agent entrypoints. The receiving model must serve with prefix caching disabled, and the connector rejects scaled RoPE, so neither model can serve a context extended past its native length.