Skip to main content
Version: Latest (main)

Use Cases

Route each model call by task, capability and policy. One stable entrypoint lets your harness use hosted APIs, data-center fleets, edge models or a mix, without changing its task loop or tools.

  • Inside an agent task: pick the right model for each call, keep a tool loop on one model, and verify selected answers.
  • Across environments: the same policy works for cloud, data-center, edge and hybrid pools.
  • Reusable patterns: capability, specialist, objective, recovery and policy-first routing.

Decisions inside an agent task​

Harness needRouter policyHarness responsibility
A coding or research call needs a different model capability.Match request signals to a specialist lane, and filter candidates by declared capabilities and context capacity.Choose the next task step and execute any returned tool calls.
A tool-using session continues across turns.With session protection configured, keep an eligible model through active tool loops or provider state that can't move.Supply the conversation and a stable session identity; manage tool results and task state.
A selected call benefits from verification.Run a configured, bounded escalation, panel or multi-model workflow.Decide how to use the returned answer in the larger task.
Sensitive work needs an approved execution path.Apply explicit eligibility and data-handling policy over local or remote backends.Enforce the same boundary in tools, task storage and other services.

Start with the agent harness guide and the Agent Routing recipe.

Deployment environments​

Most deployments mix environments: a data-center pool for general traffic, an edge model for private or offline work, and cloud providers for specialized skills. The virtual model name stays stable while each environment's endpoints change behind it.

EnvironmentTypical problemWhat Semantic Router decidesWhat stays outside the Router
CloudProvider APIs differ in capability, cost, latency and availability.Which approved provider or virtual objective handles the request.Provider capacity, billing, regional health and account policy.
Data centerA shared accelerator fleet serves models with different strengths.Which model pool fits the task, policy and objective.Replica placement, batching, autoscaling and device scheduling.
EdgeCapacity is limited, and data may need to stay local or work offline.Whether a local path is capable, and whether remote escalation is allowed.Device runtime, model packaging, power and network availability.
Enterprise hybridWorkloads cross tenants, regions, trust zones and providers.Which paths are eligible under identity, residency, capability and policy.Identity issuance, network controls, key management and data governance.

One policy over several model services. One provider may have the best vision model, another the longest context, and a small hosted model the cheapest routine chat.

The Router exposes one model API while decisions account for:

  • task intent, modality, context length and tool requirements;
  • provider allowlists and regional or contractual constraints;
  • cost, latency or quality objectives within the eligible set;
  • explicit escalation, or rejection when a path is ruled out.

Example: a document assistant sends ordinary text to an economical model, images to a vision-capable provider, and evidence-sensitive questions to a verification route. Clients keep requesting the same model name.

note

Provider accounts are not interchangeable. Manage credentials, quotas, data-use terms and regional availability explicitly.

Patterns that span environments​

PatternWhat it doesBuilt from
Capability routingDrops models that can't handle the request's modality, context, tools or language, before comparing cost or latency.Candidate requirements, input_modality, context
Specialist routingSends code, mathematics, research or domain work to a fitting pool.domain, systemone, keyword signals
Objective-based virtual modelsGives stable public names to balanced, fast, economical, accuracy-first or private behavior.Entrypoints, MoM V1
Bounded recovery and orchestrationEscalates on low confidence, compares answers or runs a workflow, only where the extra calls pay off.escalate, fusion, remom, workflows
Policy-first routingApplies authorization, locality, data handling and capability rules before any optimizer ranks candidates.authz signals, decisions

Choosing a starting point​

  1. Identify the model pools, and the system that runs each one.
  2. Define the hard eligibility boundaries before quality, latency or cost goals.
  3. Expose one public entrypoint for each behavior clients should choose.
  4. Start with a simple decision and static selection.
  5. Add signals or algorithms only when they change an observed routing outcome.
  6. Validate representative requests with probes before you change production policy.

Next​