5 posts tagged with "community"

Every multi-agent framework asks one question before the first task arrives: which model does each agent use?

planner    = Agent(role="planner",    model="small-model")
summarizer = Agent(role="summarizer", model="small-model")
reviewer   = Agent(role="reviewer",   model="frontier-model")
writer     = Agent(role="writer",     model="small-model")

It looks like configuration. It is really a prediction, made once, about work nobody has seen yet.

A task never reaches a model as one request. It arrives as a stream of calls: list the changed files, summarize a module, search for a concurrency bug, write the final comment. Bind a model to an agent and every one of those calls gets the same answer, hard or trivial.

vLLM Semantic Router moves that choice onto the request. Every agent sends one model name, MoM, short for mixture of models. It is not a model but a virtual name the router resolves per call, from what the call contains, using signals and decisions written in YAML. Throughout this post, local means a small model on your own machine, free and slower, and frontier means a large hosted model, billed per token and fast.

To see what changes, we ran one four-agent crew four ways, all frontier, model per agent, model per call and all local, recording every call: which model served it, which decision chose it, what it cost, and how long the router took to decide.

Per-call routing sent 2 of 16 calls to the frontier model and cost $1.37 per 1,000 reviews. That is one fifth the cost of sending everything to the frontier model, and less than half the cost of a hand-tuned per-agent setup. It found 4 of 6 seeded bugs, matching the all-frontier arm and beating per-agent binding, which found 3. The router took 1.86 ms at p50 to make each choice.

An application should not need a new integration every time the best model changes. Hermes gives it one stable model name, while vLLM Semantic Router chooses from a qualified pool, delivers the answer, and measures whether that choice helped.

Themis (v0.3) made routing stateful and inspectable. Hermes turns that foundation into a model system that can improve as models and workloads change.

982 commits · 130 contributors · 107 first-time contributors since Themis. This is a milestone built by a growing community across models, runtime, evaluation, and the experience of putting it all to work.

Open source grows when people can see where their work belongs and who they can build with. vLLM Semantic Router now has seven Workgroups, each responsible for one durable technical direction.

If you are new to the project, start here: vLLM Semantic Router sits between an AI application and its model or agent backends. It understands the request, chooses how it should be handled, executes that route, and measures the result. The Workgroups divide that system into clear places to contribute.

One System, Seven Clear Owners​

A routed request crosses several responsibilities:

1. Developer Experience & Ecosystem provides the CLI, Dashboard, APIs, recipes, and learning path.
2. Enterprise & Environment secures management surfaces and owns lifecycle, capacity, and deployment policy.
3. Router Models & Inference Runtime produces the signals used to route.
4. MoM & Routing chooses models and multi-model strategies.
5. Agentic & Context manages bounded context, memory, and session continuity.
6. Data Plane & Networking executes the chosen path.
7. Evaluation & Quality measures results and catches regressions.

They form one system, separated by responsibility rather than isolated code ownership. Every Epic in the live views below has one owning Workgroup. Dependencies on other groups are recorded as shared interfaces in the linked charter.

Choose the direction that matches the problem you want to solve. GitHub labels on each linked Epic are the source of truth for acceptance and delivery status.

MoM & Routing​

Mission: Make a pool of models behave like one measurable and improving Mixture-of-Models.

The problem​

A user should be able to call one stable model name without choosing a backend for every request. A Mixture-of-Models (MoM) needs a qualified model pool and a versioned recipe that can improve without making behavior unpredictable.

What this Workgroup owns​

• Model pools, model roles, portable recipes, and their versioned lifecycle.
• Model selection and collaboration through fallback, cascade, judging, synthesis, and bounded workflows.
• Offline-to-online improvement of recipes and pool members against explicit quality, cost, latency, safety, domain, or modality objectives.
• Modality-aware pools, approved reasoning reuse, and safe reuse of compatible computation across models.

What it does not own​

This Workgroup does not train the lightweight models that produce routing signals, build the live network path, decide how conversation history is compressed, or operate a hosted service.

Owned Epics​

View all current MoM & Routing Epics

Router Models & Inference Runtime​

Mission: Build better Router Models and one extensible runtime for executing them across the ecosystem.

The problem​

Routing depends on signals such as intent, complexity, safety, preference, and expected quality. The models that produce those signals must improve over time, and new models should not spread engine-specific code throughout the Router.

What this Workgroup owns​

• Improve, calibrate, and release the Router Models built into the project.
• Develop routing-native model families beyond BERT-only designs.
• Build reproducible self-improvement, distillation, and fine-tuning pipelines.
• Provide one versioned execution contract across supported engines and hardware, with clear activation, diagnostics, and rollback.

What it does not own​

This Workgroup produces routing intelligence; it does not choose the end-user MoM pool, own generic gateway forwarding, secure management surfaces, or rebuild the tensor engines and GPU schedulers it integrates with.

Owned Epics​

View all current Router Models & Inference Runtime Epics

Data Plane & Networking​

Mission: Execute every live routing decision through a fast, reliable, and portable request path.

The problem​

A routing decision has little value if the request path is slow, fragile, or different in every deployment. Standalone serving and gateway integration need the same behavior and failure semantics.

What this Workgroup owns​

• Standalone OpenAI-compatible serving and Envoy or gateway integrations.
• Request, response, streaming, dispatch, retry, fallback, error, and telemetry behavior.
• Engine-neutral backend connectivity and inference-aware endpoint selection.
• Safe semantic caching, performance optimization, and failure recovery.

What it does not own​

This Workgroup executes request-path networking and deployment-provided access policy, but it does not define management identity and authorization, decide which hardware is officially supported, train Router Models, or choose the best MoM recipe.

Owned Epics​

View all current Data Plane & Networking Epics

Enterprise & Environment​

Mission: Make vLLM Semantic Router production-grade across supported environments and hardware.

The problem​

Production operators need clear answers to practical questions: Which management surfaces are protected, and by which provider-backed identity? What changed? Is the system healthy? Can a model, recipe, or Router upgrade be rolled out and reversed safely? Which deployment path is maintained, and which components does it own? The answers must remain consistent across deployment environments.

What this Workgroup owns​

• Management authentication, provider-backed identity integration, route-bound authorization, input and credential boundaries, and durable audit.
• Reliability, scalability, monitoring, diagnostics, and the existing Insights and operational surfaces.
• Model, recipe, configuration, and vLLM-SR activation, rollout, and rollback.
• Workload simulation and capacity planning that connect observed traffic, routing behavior, serving topology, and calibrated hardware profiles to reviewable deployment proposals.
• Stable deployment and lifecycle APIs, maintained reference stacks, and a tested support matrix across deployment environments and hardware.

What it does not own​

This Workgroup does not build organization, team, or project administration; virtual API keys; tenant quotas, token rate limits, budgets, billing, or usage settlement. It also does not plan routing analytics beyond the existing Insights and operational surfaces.

It does not promise a public hosted-service SLA, expose private infrastructure or credentials, define model quality, own evaluation standards, or implement networking protocols. It supplies reusable open-source production capabilities rather than publishing private product plans.

Owned Epics​

View all current Enterprise & Environment Epics

Agentic & Context​

Mission: Optimize bounded context, memory, session continuity, and safe model or workflow switching for long-running workloads.

The problem​

Long-running work accumulates messages, memory, tool output, cost, and risk. Important instructions can be lost, and model or workflow changes can break tool loops or provider state. The Router needs bounded continuity contracts without becoming a general-purpose agent framework.

What this Workgroup owns​

• Context compression, pruning, memory selection, prompt restructuring, and protection of critical instructions.
• Prompt-visible Router Memory with explicit persistence and lifecycle receipts.
• Session budgets, state boundaries, retention, tool-loop continuity, recovery, and graceful degradation.
• Safe model or workflow switching as a session evolves.
• Typed task, context-portability, capability, and collaboration receipts that external agent runtimes can consume.

What it does not own​

This Workgroup does not select, invoke, host, or compose agent endpoints inside the Router. It does not build an agent endpoint catalog, unrestricted agent orchestrator, tool platform, or workflow engine; own general MoM selection; transport KV caches; or allow silent lossy transformation and unbounded online training.

Owned Epics​

View all current Agentic & Context Epics

Developer Experience & Ecosystem​

Mission: Make vLLM Semantic Router easy to adopt, configure, extend, diagnose, and contribute to.

The problem​

Technical depth has little impact when a new user cannot reach a first request or understand what happened. The project also needs clear extension paths so contributors, model builders, infrastructure projects, and educators can build on it without reverse-engineering the repository.

What this Workgroup owns​

• One supported first-run path through the CLI, configuration, recipes, errors, and troubleshooting.
• Dashboard configuration and diagnostics built on canonical Router and deployment contracts.
• An agent-facing skill for deployment, recipe generation, evaluation, tuning, and reviewed operations.
• Documentation, localization, integration guides, Router Model development guides, technical content, and contributor entry points.
• Clear extension and contribution paths for models, runtimes, gateways, and deployment systems.

What it does not own​

This Workgroup does not control repository permissions or promotion, run marketing events or AMD-internal programs, or redefine algorithms, production policy, and quality standards owned by other Workgroups.

Owned Epics​

View all current Developer Experience & Ecosystem Epics

Evaluation & Quality​

Mission: Make every supported capability measurable and every change verifiable.

The problem​

Claims about a Router Model, MoM recipe, agent-selection policy, runtime optimization, or deployment are difficult to trust when each uses a different dataset and reporting method. The project needs shared evaluation contracts and regression gates. Each technical Workgroup remains accountable for what it builds; this Workgroup makes the results comparable.

What this Workgroup owns​

• Common benchmark, data provenance, metric, comparison, reproducibility, and publication contracts.
• First-class evaluation of each MoM against standalone models, with a common core and objective-specific extensions.
• Shared evaluation for routing, agents, context, serving, platform, and developer workflows.
• CI, E2E, compatibility, security, performance, and operational regression gates.

What it does not own​

This Workgroup does not choose another direction's quality target, accept an issue, make the final release decision, replace Maintainer review, or own the model research itself. It defines shared measurement and gates.

Owned Epics​

View all current Evaluation & Quality Epics

How to Join and Work Together​

A Workgroup is a technical home, not a permission level.

Each Workgroup owns one durable technical direction and its bounded Epics. It connects contributors, maintains the charter, and helps prepare work for acceptance. The Open Source Team retains final acceptance, merge, role, and release authority.

The normal Member path has four clear steps:

1. Apply — choose a Workgroup in the Workgroup join guide, open its charter, and leave a first-person application comment.
2. Claim — choose an AVAILABLE issue owned by that Workgroup, open the target issue, and comment /assign. The task counts as assigned only after your name appears in the issue's Assignees field.
3. Deliver — open a focused PR linked to the claimed issue and get a substantive contribution merged.
4. Join the roster — the Workgroup Operations run collects every eligible person into one reviewed roster PR. You become a canonical Member after that PR merges.

A contribution belongs to the Workgroup that owns its issue or PR. Work in a different Workgroup can qualify you there, but it does not complete the application you made elsewhere. A requested Lead follows the same contribution path and then receives a separate human role review.

If you have not applied​

Pick the Workgroup whose charter matches the work you want to do, then comment on that charter in your own words. This compact shape is enough:

I'd like to join this Workgroup.
Role: Member
Background: <relevant experience>
Interested in: <areas you want to help with>

You do not need to be on a roster before collaborating. After applying, open the current weekly Workgroup Issues or the Community Weekly Issues view and continue with any AVAILABLE issue under the same Workgroup.

If you have applied but do not know how to get started​

Choose any AVAILABLE issue under your applied-to Workgroup, open the target issue, comment /assign, and verify that GitHub lists you as an assignee. That assignee state is what makes the application TASK_ASSIGNED; the weekly listing or the /assign text by itself does not. Please claim one issue at a time so ownership stays clear and other contributors can find open work; finish or hand off the current issue before claiming another.

If none of the available issues fit, comment on the weekly Workgroup Issues entry with exactly:

Need task: wg/<slug>

The next operations run will bring that request into the task-replenishment discussion. It will not assign a task automatically.

Once a substantive PR owned by your applied-to Workgroup merges, your application becomes READY_FOR_ROSTER. The roster PR still needs review and merge before your Member card becomes canonical.

If you already joined​

Applicants, Members, and Leads use the same AVAILABLE list. Use /assign on the target issue, keep the issue updated when scope or progress changes, and keep one active implementation issue at a time. If nothing fits, use the same Need task: wg/<slug> comment on the weekly Workgroup Issues entry so the next replenishment discussion includes you.

When the weekly list changes​

The Workgroup Operations run posts one consolidated update on the weekly issue, the join guide when needed, and each Workgroup charter. A single comment may mention several current applicants, Members, and Leads so the charter does not fill with repeated one-person automation replies. The update links the shared Workgroup list and explains /assign; a mention or recommendation is not an assignment. Only the target issue's Assignees field establishes ownership.

What each role is responsible for​

Every active Workgroup has at least one Lead and may have several. A Lead is a Committer or Maintainer, or a Contributor with a merged commit and a Committer or Maintainer Sponsor. A Member has at least one substantive merged repository contribution and continues to build in the direction. People may join more than one Workgroup, and these roles provide visibility and responsibility rather than additional repository permissions.

Make Your Work Visible​

Contribution is more rewarding when people can see the work, the technical home behind it, and the responsibility it grows into. The Community Console turns public repository evidence into three shareable cards:

Workgroup role	Contribution impact	Project team
Your canonical Lead or Member identity after the roster PR merges.	Your public merged PRs, reviews, commits, discussions, and current rank.	Maintainer, Committer, or Emeritus responsibility after the project's separate promotion process.

Click a person's avatar or name on the Workgroups, Contributors, or Team page to create the live 3:4 card, copy it, or download the full-resolution PNG. The examples above are snapshots; the live cards update with the repository.

Proposing a New Workgroup​

Create a Workgroup only for a durable technical problem that spans releases, contains several Epics, and cannot fit an existing charter. A feature, algorithm, integration, event, or milestone belongs inside an existing group.

A proposal must define the problem, scope, non-scope, shared interfaces, initial Epic map, why existing groups cannot own it, and at least one eligible Lead. After Maintainer acceptance, it receives a charter, owner label, public roster, and Lead or Member self-nomination.

Come Build With Us​

Start with the Workgroups map, choose a charter, and introduce yourself. Join an Epic, shape a contributor-ready issue, or propose a focused piece of work.

Find your focus and passion. Build with the community. Grow together.

The Feature Everyone Wants and Almost Nobody Has​

Cursor's Auto mode is deceptively simple: the developer types, and the IDE chooses whether a prompt deserves a frontier model or something faster and cheaper. It is easy to stop noticing — until moving to an open tool where every request starts with a model dropdown.

Agent systems that span multiple models — a local endpoint for coding, a frontier cloud model for deep reasoning, and a fast general-purpose model for everyday tasks — all face the same routing question: how should each request be directed to the right backend?

Many deployments start with a lightweight Python proxy or keyword matcher in front of the gateway. That approach works at small scale, but misroutes grow quickly as traffic, languages, and task types diversify. This post shows how vLLM Semantic Router running as an Envoy ExtProc sidecar inside AgentGateway replaces that pattern with semantic, config-driven routing.