Connect an agent harness
Point your harness at a Router entrypoint such as vllm-sr/auto. The harness
owns the agent loop, tools, and task state; the Router selects models or runs a
bounded multi-model workflow under your policy. Inference backends execute the
model calls.
Install the Router first, or ask an agent to install it.
Choose the inference connection
Configure the harness's model provider. Setting names vary by harness version.
| Setting | What to use |
|---|---|
| Inference address | The public inference listener or gateway. The local Quickstart uses http://localhost:8899; the management API and Dashboard have different addresses. |
| Client protocol | OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages, matching the requests your harness sends. |
| Public model ID | vllm-sr/auto for the default policy, or a published entrypoint from GET /v1/models. |
| Authentication | The credentials required by your inference listener or gateway. Backend provider credentials are configured separately by the operator. |
| Model limits and capabilities | Context, output, tools, vision, and reasoning settings that the selected recipe can support. |
Clients may append /v1 themselves. Check that the final path matches:
| Client API | Final request path | Requirement |
|---|---|---|
| Chat Completions | POST /v1/chat/completions | Send the Chat Completions request shape. |
| Responses | POST /v1/responses | The Router's global.services.response_api and its store must be available. |
| Messages | POST /v1/messages | Send the Messages request shape and an appropriate anthropic-version header. |
Client and backend protocols can differ, within the compatibility matrix. Free-form custom tools cannot reach a Messages backend; Codex CLI has additional limits listed there. Keep credentials in the harness's supported secret store or environment bindings.
Verify the public model
Configure at least one backend, then list the local stack's public models:
curl -sS http://localhost:8899/v1/models
Use a virtual entrypoint to apply routing policy. Concrete provider model names bypass recipe routing.
Send a minimal request and include response headers:
curl -sS -i http://localhost:8899/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "vllm-sr/auto",
"messages": [{"role": "user", "content": "Hello!"}]
}'
For other deployments, use their inference address and authentication. Check
the answer plus x-vsr-selected-recipe and x-vsr-selected-model; HTTP success
alone does not verify the output or route. See routing
receipts and CLI previews and probes.
Model names the harness picks itself
A harness can request models other than the one you configured. Claude Code,
for example, asks for its own Haiku model ID when it starts an Explore
subagent. The Router answers a model it does not publish with "the requested
model is not available", and that subagent fails. Point every model setting
the harness has at an entrypoint (for Claude Code, ANTHROPIC_MODEL and the
ANTHROPIC_DEFAULT_*_MODEL variables), or list the IDs it sends in the
entrypoint's model_names:
entrypoints:
- model_names: [vllm-sr/auto, claude-haiku-5-5]
recipe: default
Claude Code and Codex CLI
Claude Code on a Claude subscription runs Anthropic's models, and the vSR plugin adds guard, shield, redact and check to it without a gateway. Route Claude Code or Codex CLI through the Router when your own models or provider keys should answer: the Router then chooses the model for every call, the main conversation included, while vSR keeps judging tool output and shell commands inside Claude Code. Codex reaches the Router the same way, with its own vSR plugin.
The two harnesses need different Router-wide settings: the losses the Router
accepts, the headers that identify a session and a subagent, and defaults for
what the harness leaves out. Start with one Router for each; both can use the
same backends and decision model. This was checked with Claude Code 2.1.322
with vSR 0.1.10, Claude Code 2.1.317 without it, and Codex CLI 0.166.0, against
anthropic provider models served by vLLM 0.32.0.
Connect with vllm-sr connect
export VLLM_SR_API_KEY=<the listener's API key> # leave unset when it has none
vllm-sr connect claude-code --config claude-code.yaml
vllm-sr connect codex --config codex.yaml
codex --profile vllm-sr
Here each harness has its own Router, served from claude-code.yaml and
codex.yaml. The command reaches the gateway at the first listener of the
configuration it is given (config.yaml by default); --base-url names
another origin.
The command shows the changes and asks before writing them:
- Claude Code: the
envblock of~/.claude/settings.json(or$CLAUDE_CONFIG_DIR/settings.json;--scope projectwrites.claude/settings.local.json). The other settings stay as they are. The file holds the key when one is set, so only you can read it afterwards. In a git repository,--scope projectadds the file and the command's undo record to.git/info/excludewhen git would otherwise track them, and refuses to write a key into either one once git already tracks it.ANTHROPIC_API_KEYis set empty: otherwise Claude Code also sends a key your shell exports, asx-api-keybeside the gateway's token. - Codex: the
~/.codex/vllm-sr.config.tomlprofile, which holds avllm-srprovider, selects it and turns off hosted web search, which the Router does not serve.~/.codex/config.tomlis not touched.
Neither harness knows vllm-sr/auto, so the command gives them its context
window: the smallest context_window_size of the models the entrypoint can
reach in the configuration, which are the models its recipe's decisions name
and providers.defaults.model (a catalog-backed model without one takes its
card's limit). Image and System One models are left out. It says where the
number came from and writes it as CLAUDE_CODE_MAX_CONTEXT_TOKENS for Claude
Code and model_context_window for Codex. When a model states no window, the
command names it and sets none: Claude Code then assumes a window of its own
(1,000,000 tokens in 2.1.323) and says so at start, and Codex falls back to
default model metadata. --context-window sets the window either way.
It then reports three checks, each on its own line:
Router configuration: what the configuration given with--configlacks for this harness, from the settings below, without changing it.Basic reply: a plain request in the harness's protocol gets an answer, and the line names the model that gave it.Streamed tool loop: a streamed tool call, then the answer once its result goes back, as the harness sends them. A stream that ends in an error fails it, and the check runs only after the basic reply passed.
On a failure the written changes stay, so fix the gateway and run the command
again, or take them back with --undo. With the vSR plugin installed it warns
while the plugin's router app is on, and when the listener also serves System
One it prints the setting that points vSR at it. --dry-run prints the plan
and writes nothing.
After writing, the command names any settings that win over what it wrote,
and how to stop them. For Claude Code that is a .claude/settings.json or
.claude/settings.local.json in the current directory that sets one of its
env keys: Claude Code reads project settings only in the directory it starts
in, and they win over the user's. Variables your shell exports do not win over
the settings' env. For Codex it is a .codex/config.toml from the
repository's top down to the current directory that sets model: once Codex
trusts the project, it asks for that model instead of the profile's.
OPENAI_BASE_URL does not change the vllm-sr provider. These were measured
with Claude Code 2.1.323 and Codex CLI 0.166.0.
--status-line also sets Claude Code's status line to show which models
answered this session, the last first, such as vllm-sr ⇢ qwen36 (4) · qwen38-27b (2). Claude Code asks for vllm-sr/auto, so its own model display
names only the entrypoint. The line reads the session's transcript and its
subagents' transcripts, where each reply keeps the model that wrote it, so it
needs no Router access. The script
sits beside the settings as vllm-sr-status-line.py, and --undo puts back
the status line you had before.
--undo puts back what each key, or the Codex profile, held before the first
run, leaves alone what changed since, and keeps everything else in the files.
Try it without writing anything
vllm-sr connect claude-code --config claude-code.yaml --launch
vllm-sr connect codex --config codex.yaml --launch -- exec "fix the failing test"
--launch runs the same checks and the same probe, then starts the harness
through the gateway for that one session and exits with its status. Nothing is
written: Claude Code gets the settings with --settings from a temporary file
only you can read, which holds the key and is removed when Claude Code exits,
and Codex gets them as -c overrides and reads the key from VLLM_SR_API_KEY,
so it never appears on the command line. Both win over the user's and the
project's settings, so the overrides above do not apply. Arguments after --
go to the harness, and a gateway that fails the probe means the harness is not
started.
Router settings for Claude Code
global:
router:
allow_lossy_translation: [tools.defer_loading]
learning:
enabled: true
protection:
enabled: true
scope: session
identity:
headers:
session: x-claude-code-session-id
agent: x-claude-code-agent-id
Claude Code's tool search marks tools for loading on demand. Without
tools.defer_loading the Router refuses the first calls of every session, and
Claude Code goes on without tool search. Toward an openai backend, also list
tool_result.is_error, messages[].role and content.file, as
protocol compatibility
describes. An anthropic backend receives those as sent.
Claude Code on its own Claude login
Claude Code signed in to a Claude subscription can use the gateway without an
API key: give every backend auth: {credential: caller}
(Security Hardening),
and point Claude Code at the listener without a gateway token, so it sends its
own login:
env -u ANTHROPIC_API_KEY -u ANTHROPIC_AUTH_TOKEN -u CLAUDE_CODE_OAUTH_TOKEN \
ANTHROPIC_BASE_URL=http://localhost:8899 \
ANTHROPIC_MODEL=vllm-sr/auto \
ANTHROPIC_DEFAULT_HAIKU_MODEL=vllm-sr/auto \
ANTHROPIC_DEFAULT_SONNET_MODEL=vllm-sr/auto \
ANTHROPIC_DEFAULT_OPUS_MODEL=vllm-sr/auto \
CLAUDE_CODE_MAX_CONTEXT_TOKENS=1000000 \
claude
CLAUDE_CODE_MAX_CONTEXT_TOKENS is the smallest context window among the
models the entrypoint reaches; without it Claude Code does not know
vllm-sr/auto and says so at start. vllm-sr connect claude-code --config <your config> writes the same settings for such a configuration: it leaves
ANTHROPIC_AUTH_TOKEN empty, so neither a gateway key nor one your shell
exports replaces the login, and it does not probe the gateway, which answers
only on that login.
The Router then routes each call among Claude models on that login, and
Insights shows the session's calls with the model each decision picked. A
model's pricing is the API's list price, not the plan's, so leave it out or
read the cost as an API-price estimate. The login stays its owner's: serve this
to your own Claude Code, or to your organization's members, never to other
people as a service
(Security Hardening).
Router settings for Codex CLI
recipes:
- name: default
decisions:
- name: agents
# rules and modelRefs as in your recipe
plugins:
- type: request_params
configuration:
default_max_tokens: 32000
global:
router:
allow_lossy_translation: [reasoning.signature]
learning:
enabled: true
protection:
enabled: true
scope: session
identity:
headers:
session: session-id
agent: thread-id
services:
response_api:
enabled: true
store_backend: memory
Codex sends no output limit, so a decision that sends it to an anthropic
backend sets one. Thinking from an anthropic backend carries a signature that
a Responses client cannot receive: without reasoning.signature the stream
ends in a lossy_translation error where the thinking block closes, and Codex
reconnects until it gives up. The Responses endpoint needs the Router's
Responses service and store. See Codex CLI
for the other fields Codex sends.
Codex on its own ChatGPT login
Codex signed in with ChatGPT can use the gateway without an API key as well:
give every backend provider: openai, the endpoint
https://chatgpt.com/backend-api/codex and auth: {credential: caller}
(Security Hardening),
and add requires_openai_auth = true to the gateway's provider in the Codex
profile, so Codex sends its own login with the request:
[model_providers.vllm-sr]
name = "vLLM Semantic Router"
base_url = "http://localhost:8899/v1"
wire_api = "responses"
requires_openai_auth = true
Keep web_search = "disabled" in the profile: the Router takes function tools
only. vllm-sr connect codex --config <your config> writes this profile for
such a configuration, with requires_openai_auth and without a key variable,
and does not probe the gateway, which answers only on Codex's own login. Each
connect command stops before writing anything for a gateway whose caller
backends name the other provider: the Router takes an anthropic login only on
Messages and an openai one only on Responses or Chat, so every call would be
refused. The plan's usage limits apply to each call, and the login stays its
owner's, as for Claude Code above.
With the vSR plugin
One listener can serve both systems: the routed model calls for Claude Code
and, at /v1/decisions, the System One questions vSR asks. Publish the decision
model on the listener and point vSR at it, so the plugin and the gateway share
one address, one API key and one decision model. A listener that names its
apis serves only those, so list the model APIs the harnesses use next to
systemone:
listeners:
- name: http
address: 0.0.0.0
port: 8899
apis:
messages: {} # Claude Code
responses: {} # Codex CLI
chat_completions: {}
systemone:
models: [vela3-lite] # the public names vSR may ask for
global:
model_catalog:
deployments:
primary:
provider: model_runtime
public_name: vela3-lite
artifact: vllm-sr/vela3-lite
vSR's apps are measured on vela3-lite. Models of the runtime's vela3 family,
such as vela3-fast, answer no score question, which vSR's router app asks,
and score risky commands far below the line guard holds at, so publish
vela3-lite for vSR.
vllm-sr connect claude-code --config router.yaml
vllm-sr connect codex --config router.yaml --vsr-source /path/to/semantic-router
When the listener publishes a System One model, one command configures the
gateway (System Two) and vSR (System One); --no-vsr leaves vSR out. The
plugin installs from --vsr-source, this repository on GitHub by default, in
the agent's own CLI (claude plugin or codex plugin), and only when it is
missing. For Claude Code the command then sets the plugin's options through
claude plugin configure: a remote runtime, the gateway's /v1/decisions, the
first published System One model, and router: false so the gateway chooses the
models. When the API key variable is set it also passes runtimeToken, which
Claude Code keeps in its credential store, not in settings.json; the preview
names the variable and never shows the key. For Codex it writes
runtimeUrl and runtimeModel to $CODEX_HOME/vsr.json (default
~/.codex/vsr.json), keeping its other settings.
Both harnesses preview everything with --dry-run, the plugin commands
included, which run only when writing. --undo sets back the plugin options the
command changed (an option that was never set returns to its default) but leaves
the plugin installed and the token in place. The gateway settings are written
even when vSR cannot be: without the agent's CLI on PATH, when the plugin's
options cannot be read, or when the plugin holds a token from before and no key
is given (Claude Code cannot clear a stored token, and the plugin would send it to
the new listener), the command says why and what to run. --launch writes
nothing, so it leaves vSR alone.
The listener can also set which of vSR's apps run, for every plugin pointed at
it, under systemone.vsr; the Dashboard shows each field as Default, On or Off
under Config › Global Config › vSR agent plugins, where Default leaves the field
out. Each field set there wins over the plugin's own option, so turning
redact off here turns it off in every session that uses this listener, from
its next prompt a minute or more later. A choice a person makes in Claude Code
with /vsr stays theirs until /vsr follow:
systemone:
models: [vela3-lite]
vsr: {redact: true, guard: true, router: false}
Rules of your own for guard and redact
vsr.rules binds rules of one recipe to guard and redact. A rule is an
ordinary signal of that recipe: guard takes any keyword
rule and a System
One noul rule, or choice rule with
labels; redact takes a rule that finds text, an OR keyword rule of literal
or regex keywords without fuzzy_match, count or density, or a System One
span rule. An entrypoint must reach the recipe, since the Router prepares the
rules of a reachable recipe only.
systemone:
models: [vela3-lite]
vsr:
rules:
recipe: default
guard:
- {type: keyword, name: prod_database, enforce: required}
- {type: systemone, name: deletes_data}
redact:
- {type: keyword, name: employee_id, enforce: required}
In the Dashboard, Config › Global Config › vSR agent plugins › Rules of your own picks the recipe and lists its rules that fit each app, each Off, Advisory or Required. Once an app binds a rule, the rules left Off fold behind a "Show N more rules" button. Try it under each app sends a sample command or output through the Dashboard, which adds the listener's key on the server, and says which rules match or what they would hide, as the Router runs the saved rules.
enforce: required makes a rule fail closed for the plugin that applies it:
when the rule cannot be checked, the command waits for the person and the
output is withheld. advisory, the default, lets an unchecked rule step aside
as the plugins' own checks do. An app binds at most 16 rules, each name once.
The listing (GET /v1/systemone/models) hands each rule's name, description
and required to the plugins under vsr.rules, with a revision that changes
with any rule's definition, binding or enforcement. A plugin that gets an
answer for another revision, such as after a change in the Dashboard, reads the
listing again and asks once more in the same call, so a change applies to a
running session at once. A rule's pattern or
question never leaves the Router. To evaluate them, a client sends the texts
to the same listener, with the same key as any System One call:
curl -s http://localhost:8899/v1/systemone/rules \
-H 'Authorization: Bearer <listener key>' \
-d '{"app": "redact", "parts": [{"id": "out-1", "text": "Paid by E-104233"}]}'
{"revision": "3f9c0a1b2d4e", "parts": [{"id": "out-1", "coverage": "complete",
"matches": [], "spans": [{"start": 8, "end": 16, "rule": "employee_id"}], "unchecked": []}]}
Each part comes back with the guard rules it matches, or the redact spans in
code points, overlapping spans of different rules merged into one under the
rule that starts first, and the rules the Router could not check with a reason.
coverage is complete, partial or failed. A request holds at most 16
parts and 256 KiB of text; a listener evaluates 8 such requests at once and
answers 429 with Retry-After beyond that, 404 rules_not_published for an
app it binds no rules to, and 401 without its key. Every call is logged with
the rule names, counts and latency, never the text.
The vSR plugins for Claude Code and Codex read these rules from the listing and
call this endpoint themselves. Guard sends the whole command beside its own
questions and holds it when a rule matches; redact sends the tool output as the
tool gave it and puts [vsr:<rule name>] where a rule found text. A required
rule that cannot be checked holds the command or withholds the output in both;
the rules last read from a listener are kept, so an outage does not drop one
once a plugin has read the listing. Before that first read a plugin knows no
rules, and its checks step aside when the listener does not answer.
Turning guard or redact off for a session turns their rules off too, so a
protection nobody may turn off belongs in a decision or in masking at the
gateway.
A Router rule about what the agent reads, such as a prompt injection in a file
or a web page, should read the tool results and leave the user turn out
(input: {tool_results: true, request: false}). Claude Code's user turn carries
its own instructions and reminders, and a rule that reads it also judges those.
Turn the plugin's router app off (/vsr router off) so the plugin and the
Router do not both choose a subagent's model, and with it the plugin's routing
of your own turns (routerTurns), which is for Claude Code talking to Anthropic
directly. Its escalate app still asks Claude
to re-check an answer you push back on, but no longer moves the next subagent
up a model, since that step belongs to the router app. The other apps work as
before. When the plugin's decision runtime does not answer, its tally says
model offline and its checks step aside while Claude goes on, except guard's
fixed rules, which still hold, and an app set to wait until done, which withholds
the output or holds the command instead. A SKIPPED row says what one check
left unchecked. Both are explained in the
plugin README.
Settings by hand
The command writes these. Claude Code reads them from its settings:
{
"env": {
"ANTHROPIC_BASE_URL": "http://localhost:8899",
"ANTHROPIC_AUTH_TOKEN": "<the listener's API key, or any value without one>",
"ANTHROPIC_API_KEY": "",
"ANTHROPIC_MODEL": "vllm-sr/auto",
"ANTHROPIC_DEFAULT_HAIKU_MODEL": "vllm-sr/auto",
"ANTHROPIC_DEFAULT_SONNET_MODEL": "vllm-sr/auto",
"ANTHROPIC_DEFAULT_OPUS_MODEL": "vllm-sr/auto",
"CLAUDE_CODE_MAX_OUTPUT_TOKENS": "32000"
}
}
Codex reads the profile from ~/.codex/vllm-sr.config.toml, which
codex --profile vllm-sr layers over ~/.codex/config.toml, and the provider
can live in the same file:
model = "vllm-sr/auto"
model_provider = "vllm-sr"
web_search = "disabled"
[model_providers.vllm-sr]
name = "vLLM Semantic Router"
base_url = "http://localhost:8899/v1"
wire_api = "responses"
env_key = "VLLM_SR_API_KEY"
Leave env_key out for a listener without keys: Codex refuses to start while
the variable it names is unset, and vllm-sr connect writes it only when
VLLM_SR_API_KEY is set.
Codex does not know vllm-sr/auto, so without a catalog entry for it, it warns
that it uses fallback model metadata. vllm-sr connect writes one next to the
profile, vllm-sr.models.json, named by model_catalog_json. The file
replaces Codex's own catalog, so it holds the installed Codex's bundled entries
(from codex debug models --bundled) and adds one for the model: the gpt-5.5
entry under the model's name, with tool search and the apply_patch custom
tool turned off, since the Router does not serve the first and a Messages
backend refuses the second, with Codex's fallback reasoning settings, and
without gpt-5.5's upgrade prompt and Fast tier, which belong to that model.
Codex then edits files through its shell. A model Codex already knows, such as
--model gpt-5.5, keeps its own entry, and no catalog is written. A Codex that
bundles no gpt-5.5 entry gets no catalog and keeps the warning. Run vllm-sr connect codex again after
updating Codex, so the catalog follows its release. vllm-sr connect also sets
model_context_window and the entry's window from the configuration, or from
--context-window, so the budget follows your backends.
Codex 0.166 also says it runs without its shared background server: a profile,
like -c overrides, starts a Codex process of its own. The session works as
usual.
The Codex TUI sends a namespace tool on every turn. A Router that refuses
namespace tools answers its first turn with unsupported_tool; on such a
Router, run codex exec with [features] multi_agent = false instead.
Set budgets before running a task
GET /v1/models discovers virtual names and routing metadata; it does not
advertise their context windows or output limits. Configure these in the
harness. To keep every candidate usable, take the smallest context window and
output limit across the recipe's selectable models, including the configured
default, and enable only their shared capabilities.
For heterogeneous pools, recipe-wide candidate_requirements can check output
budgets and declared tool, reasoning, and structured-output support before
scoring. This does not configure the harness's own context management. Follow
Virtual Models: limits for agent clients
for the example and exact eligibility behavior.
A listener's timeout (300 seconds by default) bounds a whole call, the
streamed response included, and arriving output does not extend it. A call
that reaches the limit is cut off: the stream is reset and the rest of the
answer never arrives.
For agent models, set the model's total_timeout to 0s and keep
idle_timeout to bound a stalled stream; see
model configuration.
In live Terminal-Bench 2.1 runs of Claude Code 2.1.304 at xhigh effort on
Qwen3.8-27B-FP8 (vLLM 0.32, one MI300X), 2.5% to 4% of calls took more than
300 seconds, and the longest about 17 minutes.
Preserve tool and conversation continuity
Run a real tool loop: model requests a tool → harness executes it → next call carries the matching result. For streaming, check complete tool arguments, the terminal event, and final answer. Protocol compatibility alone does not verify the model and harness together.
When Router Learning protection is configured, it needs an explicit session identity. Send both headers on related calls:
x-session-id: harness-demo-session
x-conversation-id: harness-demo-conversation
These are example values:
- Use real identities. Send distinct, stable identifiers for your sessions and conversations, from the harness or a trusted gateway.
- With only the session header,
scope: sessionholds the whole session, and the defaultscope: conversationholds each conversation, named by how it starts. - Without the session header the request stays routable, but no model is kept through protection.
- Responses history, including
previous_response_id, does not replace these headers.
See Session identification for identity priority and privacy boundaries.
When the task changes phase, the harness can ask for a fresh routing decision
in the same session: it sends x-vsr-protection-release: 1 on the first
request of the new phase, at a user turn rather than inside a tool loop. A
refused release returns its reason in x-vsr-warnings. See
Explicit release.
Don't set it through Claude Code's ANTHROPIC_CUSTOM_HEADERS, which sends it
on every request.
Check the selected model on tool continuations and user follow-ups. The harness retains tool permissions, task state, and the outer loop; recipes do not configure delegated agents.
Claude Code sends x-claude-code-session-id instead of x-session-id, and
Codex CLI sends session-id and thread-id. See
Agents that delegate to subagents
for the identity headers that hold their runs and their subagents.
Agents that send tool output as messages
Some harnesses, such as mini-SWE-agent, don't use the protocol's tool calls. The model writes a command in its reply, and the harness sends the command's output back as the next user message. The Router can't tell that output from a new user turn, so it never sees a tool loop, and signals read the output as if the user had written it.
Under scope: conversation, a decision that changes on such an output releases
the current model, and min_turns_before_switch does not hold it. Use
scope: session with a new session ID for every agent run instead:
- A changed decision alone then keeps the run on its model.
- Idle timeout, a bypass decision, candidate exclusion or a qualified rescue can still release it.
- To restart a task on another model, start a new run with a new session ID, or name the concrete model, which bypasses recipe routing.
Program the policy behind the entrypoint
Connect models and publish a recipe behind the same entrypoint. Signals, decisions, and algorithms define which models are eligible and how they run.
The Agent Routing recipe is a starting point for local, specialist, and frontier model lanes. Review its model bindings, decision-model questions, and replay retention before using it; its checked-in configuration records request and response data. Use the recipe evaluation workflow to test representative tasks before adopting a policy change.
Decide reasoning in the decision as well. For a model with a reasoning family,
such as a catalog model, the decision's modelRefs entry chooses whether the
backend thinks and at what effort; the harness's own request does not:
Decision's modelRefs entry | What the Router sends |
|---|---|
use_reasoning left out | Reasoning off |
use_reasoning: true | The decision's reasoning_effort, or the family default when none is set |
This holds whatever effort the harness asks for. Claude Code, for example,
asks for adaptive thinking with its effort setting on every call. Set
use_reasoning: true and the effort you want on the decisions behind an agent
entrypoint. See Model Reasoning.
Keep agent calls on a warm prompt cache
An agent resends its conversation on every call, so most of each prompt is a prefix the backend has already processed. Whether that prefix is still cached decides much of a call's latency and cost, and three things lose it.
- A model change. A prompt cache belongs to one model, so a switch in the middle of a run computes the whole context again on the new model. Keep each run on one model as described above. The multi_factor selector also favors the model that served the session's previous turn.
- A replica change. Each replica keeps its own prefix cache, and a model's
lb_policy(round_robinorleast_request, see model configuration) does not keep a session on one replica. When several replicas serve agent sessions, route the model to a pool whose scheduler places requests by prefix cache and load, for example llm-d (an experimental integration), instead of listing the replicas asbackend_refs. - Eviction. When the sessions in flight need more KV cache than the GPU holds, a returning session finds its prefix evicted. A CPU offload tier on the backend, such as LMCache for vLLM, returns those prefixes instead of computing them again.
The harness can also start the prompt over itself. When it compacts its history, replacing earlier turns with a summary to stay within the context window, the backend computes most of the new prompt again on the same model and replica. The Router neither causes nor prevents this, so a drop in prefix reuse right after a compaction is not a routing or placement fault.
Leave cross-model KV transfer (global.integrations.kv_transfer) off for agent
entrypoints. The receiving model must serve with prefix caching disabled, and
the connector
rejects scaled RoPE, so neither model can serve a context extended past its
native length.