14 篇博文 含有标签「ecosystem」

Your LLM just called a tool, received accurate data, and still got the answer wrong. Welcome to the world of extrinsic hallucination—where models confidently ignore the ground truth sitting right in front of them.

Building on our Signal-Decision Architecture, we introduce HaluGate—a conditional, token-level hallucination detection pipeline that catches unsupported claims before they reach your users. No LLM-as-judge. No Python runtime. Just fast, explainable verification at the point of delivery.

The Problem: Hallucinations Block Production Deployment​

Hallucinations have become the single biggest barrier to deploying LLMs in production. Across industries—legal (fabricated case citations), healthcare (incorrect drug interactions), finance (invented financial data), customer service (non-existent policies)—the pattern is the same: AI generates plausible-sounding content that appears authoritative but crumbles under scrutiny.

The challenge isn't obvious nonsense. It's subtle fabrications embedded in otherwise accurate responses—errors that require domain expertise or external verification to catch. For enterprises, this uncertainty makes LLM deployment a liability rather than an asset.

The Scenario: When Tools Work But Models Don't​

Let's make this concrete. Consider a typical function-calling interaction:

User: "When was the Eiffel Tower built?"

Tool Call: get_landmark_info("Eiffel Tower")

Tool Response: {"name": "Eiffel Tower", "built": "1887-1889", "height": "330 meters", "location": "Paris, France"}

LLM Response: "The Eiffel Tower was built in 1950 and stands at 500 meters tall in Paris, France."

The tool returned correct data. The model's response contains facts. But two of those "facts" are fabricated—extrinsic hallucinations that directly contradict the provided context.

This failure mode is particularly insidious:

• Users trust it because they see the tool was called
• Traditional filters miss it because there's no toxic or harmful content
• Evaluation is expensive if you rely on another LLM to judge

What if we could detect these errors automatically, in real-time, with millisecond latency?

The Insight: Function Calling as Ground Truth​

Here's the key realization: modern function-calling APIs already provide grounding context. When users ask factual questions, models call tools—database lookups, API calls, document retrieval. These tool results are semantically equivalent to retrieved documents in RAG.

We don't need to build separate retrieval infrastructure. We don't need to call GPT-4 as a judge. We extract three components from the existing API flow:

The question becomes: Is the answer faithful to the context?

Why Not Just Use LLM-as-Judge?​

The obvious solution—call another LLM to verify—has fundamental problems in production:

LLM judges also suffer from:

• Position bias: Tendency to favor certain answer positions
• Verbosity bias: Longer answers rated higher regardless of accuracy
• Self-preference: Models favor outputs similar to their own style
• Inconsistency: Same input can yield different judgments

We needed something faster, cheaper, and more explainable.

HaluGate: A Two-Stage Detection Pipeline​

HaluGate implements a conditional two-stage pipeline that balances efficiency with precision:

Stage 1: HaluGate Sentinel (Prompt Classification)​

Not every query needs hallucination detection. Consider these prompts:

Running token-level detection on creative writing or code review is wasteful—and potentially produces false positives ("your poem contains unsupported claims!").

Why pre-classification matters: Token-level detection scales linearly with context length. For a 4K token RAG context, detection takes ~125ms; for 16K tokens, ~365ms. In production workloads where ~35% of queries are non-factual, pre-classification achieves a 72.2% efficiency gain—skipping expensive detection entirely for creative, coding, and opinion queries.

HaluGate Sentinel is a ModernBERT-based classifier that answers one question: Does this prompt warrant factual verification?

The model is trained on a carefully curated mix of:

Fact-Check Needed (Positive Class):

• Question Answering: SQuAD, TriviaQA, Natural Questions, HotpotQA
• Truthfulness: TruthfulQA (common misconceptions)
• Hallucination Benchmarks: HaluEval, FactCHD
• Information-Seeking Dialogue: FaithDial, CoQA
• RAG Datasets: neural-bridge/rag-dataset-12000

No Fact-Check Needed (Negative Class):

• Creative Writing: WritingPrompts, story generation
• Code: CodeSearchNet docstrings, programming tasks
• Opinion/Instruction: Dolly non-factual, Alpaca creative

This binary classification achieves 96.4% validation accuracy with ~12ms inference latency via native Rust/Candle integration.

Stage 2: Token-Level Detection + NLI Explanation​

For prompts classified as fact-seeking, we run a two-model detection pipeline.

Token-Level Hallucination Detection​

Unlike sentence-level classifiers that output a single "hallucinated/not hallucinated" label, token-level detection identifies exactly which tokens are unsupported by the context.

The model architecture:

Input: [CLS] context [SEP] question [SEP] answer [SEP]
                                          ↓
                              ModernBERT Encoder
                                          ↓
                    Token Classification Head (Binary per token)
                                          ↓
              Label: 0 = Supported, 1 = Hallucinated (for answer tokens only)

Key design decisions:

• Answer-only classification: We only classify tokens in the answer segment, not context or question
• Span merging: Consecutive hallucinated tokens are merged into spans for readability
• Confidence thresholding: Configurable threshold (default 0.8) to balance precision/recall

NLI Explanation Layer​

Knowing that something is hallucinated isn't enough—we need to know why. The NLI (Natural Language Inference) model classifies each detected span against the context:

Why the ensemble works: Token-level detection alone achieves only 59% F1 on the hallucinated class—nearly half of hallucinations are missed, and one-third of flags are false positives. We experimented with training a unified 5-class model (SUPPORTED/CONTRADICTION/FABRICATION/etc.) but it achieved only 21.7% F1—token-level classification simply cannot distinguish why something is wrong. The two-stage approach turns a mediocre detector into an actionable system: LettuceDetect provides recall (catching potential issues), while NLI provides precision (filtering false positives) and explainability (categorizing why each span is problematic).

Integration with Signal-Decision Architecture​

HaluGate doesn't operate in isolation—it's deeply integrated with our Signal-Decision Architecture as a new signal type and plugin.

fact_check as a Signal Type​

Just as we have keyword, embedding, and domain signals, fact_check is now a first-class signal type:

This allows decisions to be conditioned on whether the query is fact-seeking:

Note: Even frontier models show hallucination variance between releases. For example, GPT-5.2's system card demonstrates measurable hallucination delta compared to previous versions, highlighting the importance of continuous verification regardless of model sophistication.

decisions:
  - name: "factual-query-with-verification"
    priority: 100
    rules:
      operator: "AND"
      conditions:
        - type: "fact_check"
          name: "needs_fact_check"
        - type: "domain"
          name: "general"
    plugins:
      - type: "hallucination"
        configuration:
          enabled: true
          use_nli: true
          hallucination_action: "header"

Request-Response Context Propagation​

A key challenge: the classification happens at request time, but detection happens at response time. We need to propagate state across this boundary.

The RequestContext structure carries all necessary state:

RequestContext:
  # Classification results (set at request time)
  FactCheckNeeded: true
  FactCheckConfidence: 0.87

  # Tool context (extracted at request time)
  HasToolsForFactCheck: true
  ToolResultsContext: "Built 1887-1889, 330 meters..."
  UserContent: "When was the Eiffel Tower built?"

  # Detection results (set at response time)
  HallucinationDetected: true
  HallucinationSpans: ["1950", "500 meters"]
  HallucinationConfidence: 0.92

The hallucination Plugin​

The hallucination plugin is configured per-decision, allowing fine-grained control:

plugins:
  - type: "hallucination"
    configuration:
      enabled: true
      use_nli: true  # Enable NLI explanations

      # Action when hallucination detected
      hallucination_action: "header"  # "header" | "body" | "block" | "none"

      # Action when fact-check needed but no tool context
      unverified_factual_action: "header"

      # Include detailed info in response
      include_hallucination_details: true

Response Headers: Actionable Transparency​

Detection results are communicated via HTTP headers, enabling downstream systems to implement custom policies:

HTTP/1.1 200 OK
Content-Type: application/json
x-vsr-fact-check-needed: true
x-vsr-hallucination-detected: true
x-vsr-hallucination-spans: 1950; 500 meters
x-vsr-nli-contradictions: 2
x-vsr-max-severity: 4

For unverified factual responses (when tools aren't available):

HTTP/1.1 200 OK
x-vsr-fact-check-needed: true
x-vsr-unverified-factual-response: true
x-vsr-verification-context-missing: true

These headers enable:

• UI Disclaimers: Show warnings to users when confidence is low
• Human Review Queues: Route flagged responses for manual review
• Audit Logging: Track unverified claims for compliance
• Conditional Blocking: Block high-severity contradictions

The Complete Pipeline: Three Paths​

Model Architecture Deep Dive​

Let's look at the three models that power HaluGate:

HaluGate Sentinel: Binary Prompt Classification​

Architecture: ModernBERT-base + LoRA adapter + binary classification head

Training:

• Base Model: answerdotai/ModernBERT-base
• Fine-tuning: LoRA (rank=16, alpha=32, dropout=0.1)
• Training Data: 50,000 samples from 14 datasets
• Loss: CrossEntropy with class weights (handle imbalance)
• Optimization: AdamW, lr=2e-5, 3 epochs

Inference:

• Input: Raw prompt text
• Output: (class_id, confidence)
• Latency: ~12ms on CPU

The LoRA approach allows efficient fine-tuning while preserving the pretrained knowledge. Only 2.2% of parameters (3.4M out of 149M) are updated during training.

HaluGate Detector: Token-Level Binary Classification​

Architecture: ModernBERT-base + token classification head

Input Format:

[CLS] The Eiffel Tower was built in 1887-1889 and is 330 meters tall.
[SEP] When was the Eiffel Tower built?
[SEP] The Eiffel Tower was built in 1950 and is 500 meters tall. [SEP]
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    Answer tokens (classification targets)

Output: Binary label (0=Supported, 1=Hallucinated) for each answer token

Post-processing:

1. Filter predictions to answer segment only
2. Apply confidence threshold (default: 0.8)
3. Merge consecutive hallucinated tokens into spans
4. Return spans with confidence scores

HaluGate Explainer: Three-Way NLI Classification​

Architecture: ModernBERT-base fine-tuned on NLI

Input Format:

[CLS] The Eiffel Tower was built in 1887-1889. [SEP] built in 1950 [SEP]
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^       ^^^^^^^^^^^^^^^
                    Premise (context)                Hypothesis (span)

Output: Three-way classification with confidence:

• ENTAILMENT (0): Context supports the claim
• NEUTRAL (1): Cannot be determined from context
• CONTRADICTION (2): Context conflicts with claim

Severity Mapping:

Why Native Rust/Candle Matters​

All three models run natively via Candle (Hugging Face's Rust ML framework) with CGO bindings to Go:

Benefits of this approach:

This eliminates the need for a separate Python service, sidecars, or model servers—everything runs in-process.

Latency Breakdown​

Here's the measured latency for each component in the production pipeline:

The total overhead (76-162ms) is negligible compared to typical LLM generation times (5-30 seconds), making HaluGate practical for synchronous request processing.

Configuration Reference​

Complete configuration for hallucination mitigation:

# Model configuration
hallucination_mitigation:
  # Stage 1: Prompt classification
  fact_check_model:
    model_id: "models/halugate-sentinel"
    threshold: 0.6  # Confidence threshold for FACT_CHECK_NEEDED
    use_cpu: true

  # Stage 2a: Token-level detection
  hallucination_model:
    model_id: "models/halugate-detector"
    threshold: 0.8  # Token confidence threshold
    use_cpu: true

  # Stage 2b: NLI explanation
  nli_model:
    model_id: "models/halugate-explainer"
    threshold: 0.9  # NLI confidence threshold
    use_cpu: true

# Signal rules for fact-check classification
fact_check_rules:
  - name: needs_fact_check
    description: "Query contains factual claims that should be verified"
  - name: no_fact_check_needed
    description: "Query is creative, code-related, or opinion-based"

# Decision with hallucination plugin
decisions:
  - name: "verified-factual"
    priority: 100
    rules:
      operator: "AND"
      conditions:
        - type: "fact_check"
          name: "needs_fact_check"
    plugins:
      - type: "hallucination"
        configuration:
          enabled: true
          use_nli: true
          hallucination_action: "header"
          unverified_factual_action: "header"
          include_hallucination_details: true

Beyond Production: HaluGate as an Evaluation Framework​

While HaluGate is designed for real-time production use, the same pipeline can power offline model evaluation. Instead of intercepting live requests, we feed benchmark datasets through the detection pipeline to systematically measure hallucination rates across models.

Evaluation Workflow​

The evaluation framework treats HaluGate as a hallucination scorer:

1. Load Dataset: Use existing QA/RAG benchmarks (TriviaQA, Natural Questions, HotpotQA) or custom enterprise datasets with context-question pairs
2. Generate Responses: Run the model under test against each query with provided context
3. Detect Hallucinations: Pass (context, query, response) triples through HaluGate Detector
4. Classify Severity: Use HaluGate Explainer to categorize each flagged span
5. Aggregate Metrics: Compute hallucination rates, contradiction ratios, and per-category breakdowns

Limitations and Scope​

HaluGate specifically targets extrinsic hallucinations—where tool/RAG context provides grounding for verification. It has known limitations:

What HaluGate Cannot Detect​

Transparent Degradation​

For requests classified as fact-seeking but lacking tool context, we explicitly flag responses as "unverified factual" rather than silently passing them through:

x-vsr-fact-check-needed: true
x-vsr-unverified-factual-response: true
x-vsr-verification-context-missing: true

This transparency allows downstream systems to handle uncertainty appropriately.

Acknowledgments​

HaluGate builds on excellent work from the research community:

• Token-level detection architecture: Inspired by LettuceDetect from KRLabs—pioneering work in ModernBERT-based hallucination detection
• NLI models: Built on tasksource/ModernBERT-base-nli—high-quality NLI fine-tuning
• Training datasets: TruthfulQA, HaluEval, FaithDial, RAGTruth, and other publicly available benchmarks

We're grateful to these teams for advancing the field of hallucination detection.

Conclusion​

HaluGate brings principled hallucination detection to production LLM deployments:

• Conditional verification: Skip non-factual queries, verify factual ones
• Token-level precision: Know exactly which claims are unsupported
• Explainable results: NLI classification tells you why something is wrong
• Zero-latency integration: Native Rust inference, no Python sidecars
• Actionable transparency: Headers enable downstream policy enforcement

The next time your LLM calls a tool, receives accurate data, and still gets the answer wrong—HaluGate will catch it before your users do.

Resources:

• Signal-Decision Architecture Blog
• vLLM Semantic Router GitHub Repo
• vLLM Semantic Router Documentation

Join the discussion: Share your use cases and feedback in #semantic-router channel on vLLM Slack

The earlier versions of vLLM Semantic Router relied on classification-based routing, a straightforward approach where user queries are classified into one of 14 MMLU domain categories, and then routed to corresponding models. While this worked for basic scenarios, we quickly discovered its limitations when building production AI systems for enterprises.

Consider this real-world scenario: A user asks, "I need urgent help reviewing a security vulnerability in my authentication code." The classification-based router would identify this as a "computer science" query and route it to a general coding model. But it misses critical context:

• The urgency signal that requires immediate attention
• The security sensitivity that demands specialized expertise and jailbreak protection
• The code review intent that benefits from reasoning capabilities
• The authentication complexity that needs careful analysis

This single example reveals the fundamental constraint: classification-based routing captures only one dimension of user intent—the domain—while ignoring the rich, multi-dimensional signals embedded in natural language queries.

Today, we're introducing the Signal-Decision Architecture—a complete reimagining of semantic routing that scales from 14 fixed categories to unlimited intelligent routing decisions. This new architecture combines multi-dimensional signal extraction, flexible decision logic with AND/OR operators, and built-in plugin orchestration to deliver production-ready semantic intelligence.

The Problem: Why Classification-Based Routing Doesn't Scale​

The previous vLLM Semantic Router architecture followed a simple pipeline:

User Prompt → MMLU Domain Classification → Model Selection

This approach has several fundamental limitations that prevent it from scaling to enterprise requirements.

Single-Dimensional Analysis​

Classification-based routing only considers the domain or subject matter of the query. It cannot capture:

• Urgency signals: "urgent", "immediate", "critical"
• Security sensitivity: "vulnerability", "exploit", "breach"
• Intent types: code review, architecture design, troubleshooting
• Complexity levels: simple FAQ vs. complex reasoning tasks
• Compliance requirements: PII handling, regulatory constraints

Real Impact: A medical query about "urgent patient data breach" gets routed to a medical model but lacks PII protection and security filtering—potentially violating HIPAA compliance.

Fixed Category Constraint​

Limited to 14 predefined MMLU categories (math, physics, computer science, business, etc.), making it impossible to:

• Create custom categories for specific business domains
• Define fine-grained routing rules within a domain
• Scale beyond academic subject classification

Real Impact: An enterprise with 50+ specialized use cases (legal contracts, financial compliance, medical diagnostics, code security audits) cannot express their routing requirements within 14 categories.

Inflexible Logic​

Cannot combine multiple conditions or implement complex routing strategies:

• No support for AND/OR logic: "route to expert model only when query is both urgent AND security-related"
• No priority-based selection when multiple conditions match
• No conditional plugin application based on signal combinations

Real Impact: Cannot implement layered routing strategies like "high-priority security issues get reasoning + jailbreak protection, while general questions get cached responses."

Introducing Signal-Decision Architecture​

The Signal-Decision Architecture fundamentally reimagines semantic routing by separating signal extraction from routing decisions and introducing a flexible decision engine with built-in plugin orchestration.

Architecture Overview​

The new architecture introduces three key innovations:

1. Multi-Signal Extraction: Captures multiple dimensions of user intent simultaneously
2. Decision Engine: Combines signals using flexible AND/OR logic with priority-based selection
3. Plugin Chain: Provides built-in intelligence for caching, security, and optimization

Complete Request Flow​

Core Concepts​

Signals: Multi-Dimensional Prompt Analysis​

Instead of relying solely on domain classification, the Signal-Decision Architecture extracts three complementary types of signals from each user query. Each signal type leverages different AI/ML techniques and serves distinct purposes in the routing decision process.

Keyword Signals: Interpretable Pattern Matching​

Keyword signals use regex-based pattern matching to detect specific terms or phrases in user queries. This approach provides human-interpretable routing logic—you can easily understand why a query matched a particular rule by examining the keywords.

Technical Approach:

• Compiled regex patterns for efficient matching
• Support for AND/OR boolean operators
• Case-sensitive and case-insensitive modes
• No model inference required (zero ML overhead)

Key Advantage - Interpretability: Unlike black-box ML models, keyword signals provide complete transparency. When debugging routing decisions, you can trace exactly which keywords triggered which rules. This is critical for compliance auditing and troubleshooting production issues.

Use Cases:

• Detect urgency markers: "urgent", "immediate", "asap", "critical"
• Identify security keywords: "vulnerability", "exploit", "breach", "CVE"
• Flag compliance terms: "HIPAA", "GDPR", "PII", "confidential"
• Recognize intent patterns: "code review", "architecture design", "troubleshooting"

Embedding Signals: Scalable Semantic Understanding​

Embedding signals use neural embedding models to compute semantic similarity between user queries and candidate phrases. This approach provides scalable semantic matching that understands intent beyond exact keyword matches.

Technical Approach:

• Pre-computed embeddings for candidate phrases (offline)
• Runtime query embedding using lightweight models (e.g., sentence-transformers)
• Cosine similarity computation with configurable thresholds
• Multiple aggregation strategies: max (any match), mean (average similarity), any (threshold-based)

Key Advantage - Scalability: Embedding-based matching scales to thousands of candidate phrases efficiently. Adding new routing patterns doesn't require retraining models—simply add new candidate phrases and compute their embeddings. This enables rapid iteration and customization for specific business domains.

Use Cases:

• Intent understanding: "I need help" → "technical support request"
• Paraphrase matching: "How do I fix this bug?" ≈ "debugging assistance"
• Cross-lingual routing: Semantic similarity works across languages with multilingual embeddings
• Fuzzy matching: Handles typos, abbreviations, and informal language

Domain Signals: Dataset-Driven Classification​

Domain signals use MMLU-trained classification models to identify the academic or professional domain of user queries. This approach provides dataset-driven domain expertise with support for custom domain expansion.

Technical Approach:

• Fine-tuned classification models on MMLU dataset (14 base categories)
• Support for custom domain expansion via LoRA adapters
• Multi-label classification for queries spanning multiple domains
• Confidence scoring for domain predictions

Key Advantage - Extensibility via LoRA: While the base model covers 14 MMLU categories, enterprises can train lightweight LoRA adapters to add private domain categories without retraining the entire model. For example:

• Healthcare: Add "medical_imaging", "clinical_trials", "pharmaceutical_research"
• Finance: Add "risk_modeling", "algorithmic_trading", "regulatory_compliance"
• Legal: Add "contract_law", "intellectual_property", "litigation_support"

This enables organizations to extend domain classification to their specific verticals while maintaining the base model's general knowledge.

Use Cases:

• Route to domain-specific expert models (math queries → math-expert)
• Apply domain-appropriate policies (medical queries → PII protection)
• Select specialized knowledge bases (legal queries → legal document retrieval)
• Trigger domain-specific plugins (code queries → syntax validation)

Signal Comparison​

Why Three Signal Types?​

The three signal types are complementary, not redundant:

• Keyword signals provide fast, interpretable matching for known patterns
• Embedding signals handle semantic variations and scale to large phrase sets
• Domain signals leverage academic datasets and enable domain-specific expertise

By combining all three, the Signal-Decision Architecture captures multiple dimensions of user intent simultaneously, enabling far more sophisticated routing logic than any single signal type could achieve.

Decisions: Flexible Routing Logic​

Decisions are the core routing rules that combine multiple signals using AND/OR logic to determine model selection and plugin configuration.

Decision Structure​

Each decision consists of:

Signal Combination: AND/OR logic combining multiple signal conditions

• AND: All conditions must match (high precision)
• OR: Any condition matches (high recall)

Priority: Integer value for conflict resolution when multiple decisions match

• Higher priority wins
• Enables layered routing strategies

Model Reference: Specifies which model (and optional LoRA adapter) to use

• Supports base models with domain-specific LoRA adapters
• Configures reasoning mode and effort level

Plugin Chain: Ordered list of plugins to apply

• Semantic caching for cost optimization
• Jailbreak detection for security
• PII protection for compliance
• System prompt injection for behavior control
• Header mutation for metadata propagation

Decision Evaluation Flow​

When multiple decisions match, the system selects the one with the highest priority. If no decisions match, the system falls back to the default model.

Plugins: Built-in Intelligence​

The architecture includes five built-in plugins that can be configured per decision:

Plugins execute in the configured order, with each plugin able to modify the request, block execution, or add metadata for downstream processing.

Plugin Chain Execution Flow​

Scaling from 14 to Unlimited​

The Signal-Decision Architecture removes the fundamental constraint of fixed categories. Here's how it scales:

Traditional Approach (Limited)​

14 MMLU Categories → 14 Routing Rules → 14 Model Selections

Constraints:

• Cannot create custom categories
• Cannot combine multiple conditions
• Cannot apply different policies per rule
• Cannot scale beyond domain classification

Signal-Decision Approach (Unlimited)​

3 Signal Types × N Conditions × AND/OR Logic → Unlimited Decisions

Capabilities:

• Create unlimited custom routing rules
• Combine multiple signals with flexible logic
• Apply unique plugin chains per decision
• Scale to enterprise complexity

Scalability Example​

Consider an enterprise IT support system:

Traditional Routing: Limited to 14 domain-based routes

• "computer_science" → code-model
• "engineering" → engineering-model
• (12 more fixed categories)

Signal-Decision Routing: Hundreds of specialized routes

• Urgent + Security + Computer Science → security-expert + reasoning + jailbreak
• Code Review + High Complexity → architecture-model + reasoning
• FAQ + General → cached-model + semantic-cache
• Medical + PII Detected → medical-expert + PII-protection + disclaimer
• Legal + Confidential → law-expert + PII-hash + audit-headers
• (Hundreds more custom combinations)

Each decision can have unique model selection, reasoning configuration, and plugin chains—enabling fine-grained control at scale.

Kubernetes-Native Design​

The Signal-Decision Architecture is designed for cloud-native environments with two Custom Resource Definitions (CRDs):

Complete Example: Enterprise IT Support System​

Let's walk through a complete example that demonstrates how IntelligentPool and IntelligentRoute work together to build an enterprise IT support routing system.

IntelligentPool: Define Model Pool​

First, we define the available models and their LoRA adapters:

This pool defines:

• A base model "qwen3" with 4 specialized LoRA adapters
• A fallback "qwen3" model for non-specialized queries
• Reasoning family configuration for each model

IntelligentRoute: Define Routing Logic​

Next, we define the routing decisions with multi-signal extraction:

This configuration demonstrates:

Multi-Signal Extraction:

• 3 keyword signals (urgency, security, code-review)
• 2 embedding signals (technical-support, architecture-design)
• 1 domain signal (computer-science)

Layered Decision Logic:

• Priority 100: Urgent + Security + CS → security-expert + high reasoning + jailbreak + PII protection
• Priority 80: Code Review + CS → code-reviewer + medium reasoning + cache + custom prompt
• Priority 60: Architecture Design + CS → architecture-expert + high reasoning + cache
• Priority 40: General Support → base model + aggressive cache

Plugin Orchestration:

• Security-critical queries get jailbreak detection and PII protection
• Code reviews get semantic caching and custom system prompts
• Architecture queries get longer cache TTL (2h vs 1h)
• General queries get aggressive caching (0.90 threshold, 4h TTL)

Fallback Behavior:

• If no decision matches, route to defaultModel ("general-assistant")
• If multiple decisions match, select highest priority

Dynamic Configuration Flow​

The Kubernetes-native design enables:

• Zero-downtime configuration updates
• GitOps workflows for change management
• Multi-cluster deployment strategies
• Namespace-based isolation and RBAC

Real-World Applications​

Enterprise IT Support​

Challenge: Route support tickets based on urgency, technical domain, and security sensitivity.

Solution: Multi-layered decisions with priority-based selection

• Priority 100: Urgent + Security + CS → security-expert + reasoning + jailbreak
• Priority 80: Technical Support + Debugging → code-expert + semantic-cache
• Priority 60: General Questions → general-model + aggressive-cache

Results: Appropriate model selection, cost optimization through caching, security protection for sensitive issues.

Healthcare Platform​

Challenge: HIPAA compliance requiring PII protection and medical disclaimers.

Solution: Domain-based routing with mandatory compliance plugins

• Health Domain → medical-expert + PII-redaction + disclaimer-prompt + audit-headers

Results: Automatic PII protection, consistent disclaimers, audit trail for compliance.

Financial Services​

Challenge: Multi-layered security with PII protection, jailbreak detection, and cost optimization.

Solution: Comprehensive plugin chain for financial queries

• Economics Domain → finance-expert + jailbreak + PII-hash + disclaimer + cache + compliance-headers

Results: Enterprise-grade security, regulatory compliance, cost efficiency.

Educational Platform​

Challenge: Personalized learning experiences based on subject and learning intent.

Solution: Intent-based routing with customized teaching styles

• Math + Learning Intent → math-expert + reasoning + patient-tutor-prompt + cache
• Science + Tutorial → science-expert + engaging-educator-prompt

Results: Personalized teaching approaches, appropriate reasoning for complex topics, cost optimization.

Code Assistant​

Challenge: Different complexity levels require different model capabilities.

Solution: Complexity-aware routing with reasoning control

• Architecture Design → reasoning-model + high-effort + complexity-header
• Code Review → code-expert + medium-reasoning + cache
• Simple Questions → code-expert + cache-only

Results: Optimal model selection, cost-effective reasoning usage, fast responses for simple queries.

Future Roadmap​

The Signal-Decision Architecture provides a foundation for future enhancements across multiple dimensions:

Routing Core Performance Optimization​

Radix Tree for Keyword Matching: Replace regex-based keyword matching with radix tree data structures to achieve faster pattern matching for thousands of keyword patterns. This will enable enterprises to define 10,000+ keyword rules with consistent performance.

HNSW Index for Embedding Search: Implement Hierarchical Navigable Small World (HNSW) graphs for approximate nearest neighbor search in embedding space. This will significantly improve embedding signal performance while supporting millions of candidate phrases.

Parallel LoRA for Decode-Only Models: Enable parallel execution of multiple LoRA adapters during the decode phase, allowing a single base model to serve multiple specialized domains simultaneously. This will reduce model switching overhead and improve throughput for multi-tenant deployments.

Feature Enhancements​

Visual Configuration Console: Web-based UI for creating and managing decisions without YAML editing, with real-time validation and testing capabilities.

Custom Plugin Framework: SDK for developing custom plugins with community marketplace, enabling enterprises to build domain-specific intelligence layers.

Advanced Analytics: Real-time monitoring of decision performance, signal effectiveness, and cost optimization opportunities with ML-driven recommendations.

Model Evaluation via Multi-Turn Dialogue: Intelligent model selection through multi-turn conversation evaluation. The system automatically engages multiple candidate models in parallel conversations, using LLM-as-a-Judge to assess response quality across dimensions like coherence, relevance, safety, and domain expertise. This enables dynamic routing optimization based on actual model performance rather than static rules.

Intent-Aware Internal/External Model Selection: Smart routing between internal private models and external APIs (OpenAI, Anthropic, etc.) based on intent analysis. Sensitive data and proprietary information automatically route to internal models for privacy and compliance, while general queries leverage external APIs for broader knowledge. Cost, latency, and compliance requirements are balanced dynamically based on query characteristics.

Conclusion​

The Signal-Decision Architecture represents a fundamental shift in how we think about semantic routing. By moving from fixed classification to flexible signal-based decisions, we enable:

Unlimited Scalability: From 14 categories to unlimited custom routing rules

Multi-Dimensional Intelligence: Capture keyword, embedding, and domain signals simultaneously

Flexible Logic: Combine signals with AND/OR operators and priority-based selection

Built-in Security: Integrated plugins for jailbreak detection, PII protection, and compliance

Cloud-Native Design: Kubernetes CRDs with dynamic configuration and zero-downtime updates

Whether you're building an enterprise AI gateway, a multi-tenant SaaS platform, or an industry-specific AI assistant, the Signal-Decision Architecture provides the scalability, flexibility, and intelligence needed for production deployments.

Getting Started​

Ready to try Signal-Decision routing?

Join our community to share feedback and learn from other users building intelligent routing systems at scale.

Semantic routing systems face a scaling challenge. When each classification request requires running multiple fine-tuned models independently, the computational cost grows linearly with the number of models. This post examines how a recent refactoring of the vLLM Semantic Router's Rust-based classification layer addresses this problem through architectural modularity, Low-Rank Adaptation (LoRA), and concurrency optimization.

Background: From BERT to a Modular System​

The previous implementation relied primarily on BERT and ModernBERT for intent and jailbreak classification. While ModernBERT performs well for English text classification tasks, it has the following limitations:

• Language Coverage: The original ModernBERT's multilingual support is limited compared to models trained on more diverse datasets. (Note: mmBERT, a massively multilingual variant of ModernBERT supporting 1800+ languages, was released after this refactoring began and represents an alternative approach to the multilingual challenge)
• Context Length: While ModernBERT extends context to 8,192 tokens using RoPE (source), models like Qwen3-Embedding support up to 32,768 tokens, which is beneficial for very long document processing
• Model Coupling: Classification logic was tightly coupled to specific model architectures, making it difficult to add new models

These constraints motivated a broader refactoring that would enable the system to support multiple model types while maintaining performance. The modular architecture means that newer models like mmBERT can be integrated alongside Qwen3-Embedding and EmbeddingGemma, allowing the router to select the most appropriate model for each task.

Architectural Restructuring​

The refactoring introduces a layered architecture in the candle-binding crate. This structure separates concerns: core functionality remains independent of specific models, while new model architectures can be added without modifying existing code. The DualPathUnifiedClassifier implements routing logic that selects between traditional fine-tuned models and LoRA-adapted models based on the task requirements.

Long-Context Embedding Models​

Two new embedding models address the context length limitation:

Qwen3-Embedding​

Qwen3-Embedding supports context lengths up to 32,768 tokens (Hugging Face model card). The implementation uses a RoPE (Rotary Position Embedding), enabling this extended context handling through improved frequency resolution at longer distances.

Qwen3-Embedding was trained on text from over 100 languages (Hugging Face model card), making it suitable for multilingual routing scenarios where the previous ModernBERT-only approach would struggle.

EmbeddingGemma-300M​

Google's EmbeddingGemma-300M takes a different approach, focusing on smaller model size while maintaining quality. The model supports context lengths of 2,048 tokens and implements Matryoshka representation learning, which means embeddings can be truncated to 768, 512, 256, or 128 dimensions without retraining (Hugging Face model card).

The architecture uses Multi-Query Attention (MQA) with 3 query heads and 1 key-value head, reducing memory bandwidth requirements. A distinctive feature is the dense bottleneck layer (768 → 3072 → 768) applied after the transformer blocks, which improves embedding quality based on the Matryoshka training approach.

Low-Rank Adaptation for Multi-Task Classification​

LoRA addresses a fundamental inefficiency in the previous system. When a classification system needs to determine intent, detect PII, and check for security issues, the naive approach runs three separate fine-tuned models:

Each model processes the input through its entire network, including the expensive base transformer layers. This results in O(n) complexity where n is the number of classification tasks.

LoRA changes this by sharing the base model computation:

The base model runs once, producing intermediate representations. Each LoRA adapter then applies task-specific low-rank weight updates to specialize the output. Since LoRA adapters typically modify less than 1% of the model's parameters, this final step is much faster than running complete models.

The implementation in parallel_engine.rs uses Rayon for data parallelism, processing multiple LoRA adapters concurrently. For a request requiring three classifications, this changes the workload from three full forward passes to one full pass plus three lightweight adapter applications.

Concurrency Through OnceLock​

The previous implementation used lazy_static for managing global classifier state, which introduced lock contention under concurrent load. The refactoring replaces this with OnceLock from the Rust standard library.

OnceLock provides lock-free reads after initialization. After the first initialization, all subsequent accesses are simple pointer reads with no synchronization overhead. Tests in oncelock_concurrent_test.rs verify this with 10 concurrent threads performing 30 total classifications, confirming that throughput scales linearly with thread count.

This matters when the router processes multiple incoming requests. With lazy_static, concurrent requests would queue behind a mutex. With OnceLock, they execute in parallel without contention.

Flash Attention for GPU Acceleration​

Flash Attention 2 support is available as an optional feature for CUDA builds, though it requires Ampere-generation or newer GPUs (compute capability ≥ 8.0). Flash Attention optimizes the attention mechanism by processing computations in blocks that fit in fast on-chip SRAM memory, avoiding repeated reads from slower GPU DRAM.

Both ModernBERT and Qwen3 benefit from Flash Attention integration:

• ModernBERT: Achieves up to 3× faster self-attention computations with significantly reduced memory usage (source). The model also uses alternating attention patterns (global attention every third layer, local sliding-window attention otherwise) to balance efficiency with context retention (source).

• Qwen3: Integration of FlashAttention-2 provides up to 4× speedup in attention operations. For the 14B variant, this translates to 70-110 tokens/second during inference compared to 30-35 tokens/second without it—a performance improvement that becomes more pronounced with longer contexts (source).

The Rust implementation makes Flash Attention optional via Cargo features, allowing deployment on systems without compatible GPUs while enabling substantial performance gains when hardware supports it.

Cross-Language Integration for Cloud-Native Ecosystems​

The choice of Rust for the core classification engine combined with Go FFI (Foreign Function Interface) bindings addresses a practical deployment challenge in cloud-native environments.

Why Rust for ML Inference​

Rust provides several advantages for the classification layer:

• Performance: Near-C performance with zero-cost abstractions, critical for low-latency inference
• Memory Safety: Compile-time guarantees prevent common bugs like buffer overflows and use-after-free errors
• Concurrency: The ownership system prevents data races, enabling safe parallel processing with Rayon
• No Garbage Collection: Predictable latency without GC pauses that affect request processing

The Candle framework leverages these Rust strengths while providing a familiar API for ML model development.

Why Go FFI Bindings Matter​

While Rust excels at compute-intensive ML inference, Go dominates the cloud-native infrastructure ecosystem. The FFI layer bridges these worlds. This integration enables deployment in environments where Go is the primary language:

• Envoy Proxy Integration: The semantic router runs as an Envoy external processing filter, written in Go. The FFI allows the Go filter to leverage high-performance Rust classification without rewriting the entire Envoy integration layer.
• Kubernetes Operators: Cloud-native operators are typically written in Go using controller-runtime. The FFI enables these operators to embed classification logic directly rather than making network calls to separate services.
• Service Meshes: Projects like Istio, Linkerd, and Consul are Go-based. The FFI allows routing decisions to use ML-based classification while maintaining compatibility with existing mesh control planes.
• API Gateways: Many API gateways (Kong, Tyk) have Go components. The FFI enables semantic routing at the gateway layer without introducing additional microservices.

Deployment Flexibility​

The dual-language architecture provides deployment options:

• Embedded Mode: The Go service links directly to the Rust library via CGO, minimizing latency and deployment complexity
• Process Isolation: The classification layer can run as a separate process, communicating via gRPC or Unix sockets for additional fault isolation
• Mixed Workloads: Services can combine Go's networking and orchestration strengths with Rust's ML inference performance

The semantic router leverages this pattern extensively. The main routing logic, configuration management, and cache implementations are in Go, while the compute-intensive classification runs in Rust. This separation allows each component to use the most appropriate language while maintaining clean interfaces through the FFI layer.

Performance Characteristics​

The benefits of this architecture vary by workload:

• Single vs multi-task classification: LoRA provides minimal benefit since there's no base model sharing. Traditional fine-tuned models may be faster. LoRA shows clear advantages when performing multiple classifications on the same input. Since the base model runs once and only LoRA adapters execute for each task, the overhead is substantially reduced compared to running separate full models. The actual speedup depends on the ratio of base model computation to adapter computation.
• Long-context inputs: Qwen3-Embedding enables routing decisions on documents up to 32K tokens without truncation, extending beyond ModernBERT's 8K limit for very long documents. With Flash Attention 2 enabled on compatible GPUs, the performance advantage becomes more substantial as context length increases.
• Multilingual routing: Models can now handle routing decisions for languages where ModernBERT has limited training data.
• High concurrency: OnceLock eliminates lock contention, allowing throughput to scale with CPU cores for classification operations.
• GPU acceleration: When Flash Attention 2 is enabled, attention operations run 3-4× faster, with the speedup becoming more pronounced at longer sequence lengths. This makes GPU deployment particularly advantageous for high-throughput scenarios.

Future Directions​

The modular architecture enables several extensions:

• Additional embedding models can be added by implementing the CoreModel trait
• Flash Attention 3 support when available in Candle
• Quantization support (4-bit, 8-bit) for reduced memory footprint
• Custom LoRA adapters for domain-specific routing
• FFI bindings for additional languages (Python, Java, C++) to expand integration possibilities

The system now has a foundation for incorporating new research advances without requiring architectural changes. The FFI layer provides a stable interface that allows the Rust implementation to evolve independently while maintaining compatibility with existing Go-based deployments.

Resources​

• Project Repository
• Candle Framework
• Qwen3-Embedding
• EmbeddingGemma

Industry Status: Inference ≠ More Is Better​

Over the past year, hybrid reasoning and automatic routing have increasingly defined progress in large-model infrastructure—shifting the debate from raw scale to per-token efficiency, latency control, and targeted compute use.

Take GPT-5 for example: its standout innovation lies not in sheer parameters, but in routing policies and quota-based reasoning:

• Light queries → lightweight paths: trivial prompts like “Why is the sky blue?” don’t trigger expensive reasoning.
• Complex/high-value queries → reasoning-enabled models: multi-step tasks—like legal analysis or financial planning—are routed to Chain-of-Thought–enabled inference.

This represents a broader principle of task-aware compute allocation, where every inference token must contribute meaningful value—not just be consumed.

Similar ideas are appearing in other systems:

• Anthropic Claude 3.7/4: differentiates “fast thinking” and “slow thinking” pathways.
• Google Gemini 2.5: offers explicit thinking budgets, allowing enterprises to cap reasoning depth.
• Alibaba Qwen3: supports instruction-driven switching between reasoning and non-reasoning modes.
• DeepSeek v3.1: merges conversational and reasoning flows within a dual-mode single model.

The trend is clear: future inference systems will be defined by selectivity and intelligence, not just model size.

Recent Research: vLLM Semantic Router​

Responding to this shift, the vLLM Semantic Router offers an open-source, intent-aware routing layer for the highly efficient vLLM inference engine.

vLLM enables scalable LLM serving—but lacks semantic decision-making around reasoning. Developers face a trade-off:

• Enable reasoning always → accuracy increases, but so does cost.
• Disable reasoning → cost drops, but accuracy suffers on complex tasks.

The Semantic Router fills this gap by classifying queries semantically and routing them appropriately, giving accurate results where needed and efficiency where reasoning is unnecessary.

Architecture Design​

The system comprises four pillars:

1. Semantic Classification: Uses ModernBERT—currently a lightweight, standalone classifier integrated into the router—to determine routing paths.
2. Smart Routing:
   • Simple queries → "fast path" inference.
   • Complex queries → "Chain-of-Thought" reasoning mode.
3. High-Performance Engine: Written in Rust using Hugging Face Candle, it delivers high concurrency and zero-copy inference.
4. Cloud-Native Integration: Works out-of-the-box with Kubernetes and Envoy via the ext_proc plugin.

In trials, this design yielded:

• ~10% higher accuracy
• ~50% lower latency
• ~50% fewer tokens

In business and economics domains, gains exceeded 20% accuracy improvements.

Challenges in Execution: Budgets and Tool Calling​

Two technical constraints are important to address:

• Reasoning Budget Costs
  Unlimited reasoning inflates cold-start latency and resource usage. Without dynamic control, simple queries may over-consume tokens while critical queries may not get deep reasoning when needed. SLOs like TTFT and p95 latency are necessary—with possible adaptation mid-inference.
• Tool Calling Constraints
  Adding more tools (i.e. “tool catalog bloat”) or longer tool outputs can drastically reduce accuracy. The router must pre-filter tools and keep catalogs tight.

Project Background​

The Semantic Router evolved from contributions across the open-source community:

• Proposed in early 2025 by Dr. Chen Huamin (Red Hat)
• Further developed by Xunzhuo Liu (Tencent)
• To be presented by Dr. Wang Chen (IBM Research) and Dr. Chen Huamin at KubeCon North America 2025

Our goal: provide inference acceleration for open-source LLMs through:

• Semantic-aware routing
• Efficient model switching
• Enterprise-friendly deployment (Kubernetes & Envoy)

Find the project on GitHub. The current focus is on a Work Group and planned v0.1 Roadmap.

Integration & Future Work: Embeddings and Pluggability​

Currently, ModernBERT runs internally within the router for classification. It is not yet served by vLLM. However, future work aims to make the classifier—and potentially other embedding models—pluggable, allowing integration with vLLM-hosted models or external embedding services.

This capability will enhance the semantic cache and enable smoother inference customization.

Roadmap: v0.1 Milestone Highlights​

The v0.1 milestone will expand the project’s technical capabilities:

• Core: ExtProc-based modularity, semantic caching across backends, multi-factor routing logic
• Benchmarking: CLI tools, performance testing suite, reasoning-mode evaluation
• Networking: Deeper integration with Envoy, GIE, and llm-d gateways
• Observability & UX: Admin dashboards, routing policy visualization, developer quickstarts, and policy cookbook

Future Trends: Just-in-Time Inference​

The field is maturing from “Can we run inference?” to “How can inference be smarter?”

• GPT-5 uses commercial value to guide reasoning depth.
• vLLM Semantic Router delivers that capability to open source.

Looking ahead, systems that adapt their inference strategy on the fly, without manual toggles, will lead in efficiency, latency, and sustainability.

One-Sentence Summary​

• GPT-5: enterprise routing for smarter inference
• vLLM Semantic Router: technical-first routing for open-source LLMs
• Edge future: context-aware, minimal-compute inference that works seamlessly