Find Your Focus: Join the vLLM Semantic Router Workgroups

Open source grows when people can see where their work belongs, who they can build with, and how they can make a lasting contribution. vLLM Semantic Router now has seven direction-based Workgroups to make that path clear.

If you are new to the project, start here: vLLM Semantic Router sits between an AI application and a collection of models. It understands each request, chooses one model or a bounded way for several models to collaborate, executes that choice, and makes the result observable and measurable. The seven Workgroups own the durable technical directions needed to make that experience intelligent, fast, reliable, and easy to use.

How the Pieces Fit Together

A routed request crosses several responsibilities, but each responsibility has one clear technical home:

1. Developer Experience & Ecosystem gives users the CLI, Dashboard, APIs, recipes, and guidance needed to describe and operate the system.
2. Enterprise & Environment determines who may use each model, applies quotas and lifecycle policy, and keeps supported deployments operable.
3. Agentic & Context prepares a bounded context for long-running sessions, while Router Models & Inference Runtime turns the request into useful routing signals.
4. MoM & Routing uses those signals and a versioned recipe to choose a model or a bounded form of multi-model collaboration.
5. Data Plane & Networking executes that decision on the live request path, while Evaluation & Quality measures behavior and protects it from regression.

The Workgroups therefore form one system. They are separated by responsibility, not by isolated code ownership.

Choose the Direction that matches the problem you want to solve. Each section shows its motivation, ownership boundary, and accepted Epic directions.

MoM & Routing

Mission: Make a pool of models behave like one measurable, continuously improving Mixture-of-Models.

Why this matters

A user should be able to call one stable model name without manually deciding which backend to use for every request. A Mixture-of-Models (MoM) needs a model pool and a versioned recipe: the pool defines which models are available, while the recipe defines when and how they are used. Both must improve without making behavior unpredictable.

Scope

• Design objective-specific model pools and portable routing recipes.
• Support different forms of model collaboration: direct selection, fallback, cascade, parallel answers with a judge, synthesis, and bounded workflows.
• Define how models and recipes are evaluated, shadowed, promoted, retired, and rolled back.
• Connect offline evaluation, routing outcomes, replay data, and human feedback to reviewable recipe improvements.
• Improve cross-model efficiency, including safe reuse of computation when a request moves between compatible models.

Non-scope

This Workgroup does not train the lightweight models that produce routing signals, build the live network path, decide how conversation history is compressed, or operate a hosted service. Those responsibilities have their own Workgroups.

Epic directions

• Define portable MoM model pools, routing recipes, evaluation, and model cards
• Optimize routing recipes through an offline-to-online lifecycle
• Enable safe cross-model KV-cache reuse

Router Models & Inference Runtime

Mission: Build better Router Models and one extensible runtime for executing them across the ecosystem.

Why this matters

Before vLLM-SR can choose a route, it needs signals such as intent, complexity, safety, preference, or expected quality. Router Models are the specialized models that produce those signals and scores. Their quality should improve over time, and adding a new model should not require spreading engine-specific code throughout the Router.

Scope

• Post-train, distill, calibrate, and evaluate the Router Models already built into the project.
• Research routing-native Small Language Model families beyond BERT-only designs, including long-context, multimodal, retrieval, and agentic signals.
• Support privacy-aware fine-tuning and immutable candidate artifacts that can be evaluated before activation.
• Define typed inference inputs and outputs, model capabilities, artifact identity, activation, diagnostics, draining, and rollback.
• Qualify built-in and ecosystem execution through Candle, ONNX Runtime, vLLM, TEI, and other supported engines and hardware.

Non-scope

This Workgroup produces routing intelligence; it does not choose the end-user MoM pool, own generic gateway forwarding, manage users and quotas, or rebuild the tensor engines and GPU schedulers it integrates with.

Epic directions

• Build an extensible inference runtime for Router Models
• Improve current Router Models and develop routing-native model families
• Build a reproducible SLM self-improvement and Router Model fine-tuning pipeline

Data Plane & Networking

Mission: Execute every live routing decision through a fast, reliable, and portable request path.

Why this matters

A good decision is useless if the request path is slow, fragile, or behaves differently in each deployment. vLLM-SR needs one routing behavior whether it runs as a standalone OpenAI-compatible service or integrates with Envoy and an existing gateway environment.

Scope

• Process requests and responses, including streaming, dispatch, retries, fallback, errors, telemetry, and immediate responses.
• Provide a standalone mode for Docker and Kubernetes without making Envoy a mandatory dependency.
• Keep Envoy ExtProc and qualified gateway integrations first-class and behaviorally consistent with standalone mode.
• Define engine-neutral backend connectivity and cooperate with serving and load-balancing layers on inference-aware endpoint selection.
• Optimize latency, throughput, resource efficiency, streaming behavior, and failure recovery with reproducible measurements.

Non-scope

This Workgroup executes access and routing policy but does not define who may access a model, which quota applies, which hardware is officially supported, how a Router Model is trained, or which MoM recipe is best.

Epic directions

• Support standalone and gateway-integrated data plane modes
• Connect semantic routing to inference-aware backend selection
• Optimize data-plane performance, streaming, and failure recovery

Enterprise & Environment

Mission: Make vLLM Semantic Router production-grade across supported organizations, environments, and hardware.

Why this matters

Production users need clear answers to practical questions: Who can call each model? How much can they use? What changed? Is the system healthy? Can a model, recipe, or Router upgrade be rolled out and reversed safely? The answers must remain consistent across deployment environments.

Scope

• Registration, invitations, organizations, teams, serving identities, roles, and model access.
• API-key creation, rotation, revocation, request and token rate limits, usage accounting, tenant isolation, and audit.
• Stability, scalability, monitoring, diagnostics, multi-replica consistency, and failure recovery.
• Model and recipe onboarding, draining, gray release, promotion, rollback, and auditable vLLM-SR upgrades.
• A tested support matrix across Docker, Kubernetes, CPU, AMD, NVIDIA, precision, images, and maintained configurations.

Non-scope

This Workgroup does not promise a public hosted-service SLA, expose private infrastructure or credentials, define model quality, own evaluation standards, or implement networking protocols. It supplies reusable open-source production capabilities rather than publishing private product plans.

Epic directions

• Build multi-tenant inference access, quotas, and usage controls
• Build versioned configuration activation and rollback
• Establish production observability and supported-environment qualification

Agentic & Context

Mission: Keep long-running agent workloads context-efficient, state-aware, bounded, and safe.

Why this matters

Long conversations accumulate messages, retrieved memory, tool output, tokens, cost, and risk. Important instructions can be lost in that growth, and the best model may change as the task moves from planning to tool use to final response. vLLM-SR should manage those constraints without becoming a general-purpose agent framework.

Scope

• Context compression, message and tool-output pruning, retrieval and memory selection, and prompt restructuring.
• Explicit protection for system and developer instructions, authorization, safety constraints, tool contracts, and task-critical user intent.
• Session budgets, state boundaries, retention, recovery, and graceful degradation when context or memory capabilities are unavailable.
• Requirements and safety constraints for switching models or bounded workflows as a session evolves.
• Evaluation of instruction retention, task quality, token reduction, latency, cost, safety, and critical-information loss.

Non-scope

This Workgroup does not build an unrestricted agent orchestrator, own all MoM selection algorithms, transport KV caches between models, automate CLI installation, or allow silent lossy transformation and unbounded online training.

Epic directions

• Optimize context for long-session and agentic workloads
• Develop safe model and workflow switching for long-running agents
• Define a trusted gateway context envelope for agent memory, tools, and budgets

Developer Experience & Ecosystem

Mission: Make vLLM Semantic Router easy to discover, install, configure, extend, and operate.

Why this matters

Technical depth has little impact when a new user cannot reach a first request or understand what happened. The project also needs clear extension paths so contributors, model builders, infrastructure projects, and educators can build on it without reverse-engineering the repository.

Scope

• Installation, configuration, validation, deployment, tuning, diagnostics, and operations across the CLI, Dashboard, and APIs.
• Dashboard information architecture and developer or operator workflows built on maintained enterprise contracts.
• An agent-facing vLLM-SR skill for deployment, recipe generation, evaluation, tuning, and reviewed operations.
• Architecture documentation, tutorials, reference material, examples, model cards, evaluation results, blogs, video tutorials, and release guidance.
• Use case sharing, reference implementations, extension guidance, partner integrations, and contributor entry points.

Non-scope

This Workgroup does not control repository permissions or promotion, run marketing events or AMD-internal programs, or redefine algorithms, production policy, and quality standards owned by other Workgroups.

Epic directions

• Build agent-assisted vLLM-SR deployment, recipe, evaluation, and tuning workflows
• Reduce time to the first successful routed request
• Publish maintained reference recipes and end-to-end tutorials

Evaluation & Quality

Mission: Make every supported capability measurable and every change verifiable.

Why this matters

Claims about a Router Model, MoM recipe, runtime optimization, or deployment are difficult to trust when each uses a different dataset and reporting method. The project needs shared evaluation contracts and regression gates, while each technical Workgroup remains accountable for the quality of what it builds.

Scope

• Common benchmark structure, dataset provenance, metrics, comparison rules, reproducibility, and result publication.
• MoM, recipe, Router Model, context, serving, platform, and developer-workflow evaluation without replacing each domain's objectives.
• Model-card and evaluation-result requirements for published MoM identities.
• CI, E2E, compatibility, security, performance, and operational regression coverage across supported modes and environments.
• Shared definitions of done and the evaluation required for milestone readiness, upgrade, and rollback.

Non-scope

This Workgroup does not choose another direction's quality target, accept an issue, make the final release decision, replace Maintainer review, or own the model research itself. It defines shared measurement and gates.

Epic directions

• Build reproducible decision-level routing evaluation
• Establish cross-platform performance regression coverage
• Expand end-to-end quality coverage across Router, Dashboard, and deployments

How Workgroups Operate

A Workgroup is a technical home, not a permission level.

Each Workgroup owns one durable technical Direction, its charter, and the bounded Epics beneath it. It connects contributors, helps triage proposals, and recommends when work is ready to accept. The Open Source Team retains final acceptance, review, merge, role, and release authority.

Workgroups	Open Source Team
Direction, boundaries, Epics, and contributor focus	Project governance and repository permissions
Triage and acceptance recommendations	Final acceptance, merge, and release authority
Lead and Member roles	Maintainer, Committer, and Contributor roles

Direction → Workgroup → domain → bounded Epic → accepted issue

Lead. Every active Workgroup has at least one Lead and may have several. A Lead is a Committer or Maintainer, or a Contributor with a merged commit and an active Committer/Maintainer Sponsor. Leads maintain the charter and Epic map, coordinate triage, and ensure accepted initiatives have delivery owners.

Member. A Member has at least one merged repository commit and continues to build in the Direction. Anyone may collaborate before meeting the roster requirement, and people may join more than one Workgroup.

To join, open the linked charter and comment with your requested role, a merged contribution, your focus and availability, and your Sponsor when applying as a sponsored Lead. Confirmed Leads and Members appear on the public Workgroups map; these roles provide visibility and responsibility, not additional repository permissions.

Proposing a New Workgroup

Create a Workgroup only for a durable technical problem that spans releases, contains multiple bounded Epics, and cannot fit an existing charter. A single feature, algorithm, collaboration, integration, event, or milestone remains an Epic inside an existing Workgroup.

A proposal must define the problem, goal, scope, non-scope, interfaces, initial Epic map, why existing charters cannot own it, and at least one eligible Lead. After Maintainer acceptance, the project creates its charter and owner label, opens Lead and Member self-nomination, and adds it to the website.

Come Build With Us

Start with the Workgroups map, choose a charter, and introduce yourself. Join an Epic, shape a contributor-ready issue, or propose a focused piece of work.

Find your focus and passion. Build with the community. Grow together.