OSPP2026: Multi-Architecture Runtime Support for vLLM Semantic Router
Two requests sent from the official Dashboard Playground to the native X60 Router.
Each response shows the selected model, decision and routing latency.
The test backend returns a fixed ok.
The standalone Router builds and runs on a SpacemiT X60 RISC-V server. On openKylin 2.0 SP1, the container passed eight routing and restart checks. With a two-CPU limit and concurrency 4–64, it handled about 270–280 requests/s in a short keyword/default-routing workload.
This work is part of OSPP project 2647b0097, 面向多架构环境的 vLLM Semantic Router 运行支持. It covers the Go Router and its container build. Model inference remains external. The port is proposed in PR #5259, with experimental platform status.
What runs on RISC-V
In standalone mode, the Router accepts HTTP requests and forwards them to the model API selected by its routing policy. It can also call an external classification service. The keyword and default policies tested here do not need a classifier.
The build changes
Building GLIDE from source
The first build blocker was GLIDE. Its Go client links to a Rust FFI library, and the v2.3.0 module used by the Router does not include the required RISC-V archive. Turning off Valkey-backed features in configuration still leaves that link dependency.
The build compiles GLIDE's ffi crate for riscv64gc-unknown-linux-gnu and links libglide_ffi.a into the Go binary. It checks the source archive's SHA256 and verifies the resulting archive's ELF architecture. A Cargo lock pins Rust dependencies because the v2.3.0 source archive does not include ffi/Cargo.lock.
We have requested upstream support in Valkey GLIDE #7388.
Separating the image build
tools/docker/riscv64/Dockerfile compiles GLIDE and exports the Go binary in a small intermediate image. The independent tools/docker/Dockerfile.attached accepts that binary image and packages the Router, configuration assets and startup script without bundled PyTorch or vllm-srun. It can also build the standard amd64/arm64 binary itself.
The port leaves the bundled Dockerfile.extproc unchanged. The proposal registers a separate vllm-sr-riscv64 artifact, keeping the existing CPU/CUDA/ROCm image definitions intact.
Build it on a RISC-V host
The SDK pins Go 1.25.14 and Rust 1.90.0 archives by URL and SHA256, and the Debian base by digest.
From the repository root on a native RISC-V64 Linux host with Python 3.10+ and Docker/buildx:
python3 tools/docker/riscv64/build.py --plan
python3 tools/docker/riscv64/build.py --tag local \
--output /tmp/vsr-riscv-build
This produces the SDK, native binary image and semantic-router-router:local runtime image. Use a new output directory for each run; it records the commands and build results. The Docker buildx docker driver is required to pass local images between stages. The host needs network access to the image, toolchain and dependency repositories.
The runner accepts --sdk-image to reuse an SDK and --no-cache for fresh-layer builds. This validation reused the SDK and cached layers.
Native container validation
The test host was a SpacemiT X60 with 8 cores and 16 GB RAM, running openKylin 2.0 SP1 and kernel 6.6.63-cloud. Compilation and container execution ran natively on the server. The Router ran as a non-root user with a read-only root filesystem.
Host architecture and operating system in the live SSH terminal.
The smoke test uses two routes: a code-related keyword selects code-model; other requests use general-model. A deterministic backend returns ok. For each route, the test checks a buffered response and an SSE response, restarts the container, and repeats the checks.
| Route | Buffered | SSE | Buffered after restart | SSE after restart |
|---|---|---|---|---|
Keyword → code-model | Pass | Pass | Pass | Pass |
Default → general-model | Pass | Pass | Pass | Pass |
All eight cases returned HTTP 200 with the expected model and content; SSE checks also verified stream termination.
The tested source is revision 2e899997. Its native runtime image ID is:
sha256:3ad42927fea7b3d11b041244f8eb54317994ffbde98a24e86172229d8f4371ea
The checks are in tools/test/smoke/riscv64-standalone.py; deployment commands are in tools/docker/riscv64/README.md.
Two routes in Playground
Both requests use vllm-sr/auto. The prompt write code for binary search
matches the keyword rule and selects code-model. Explain photosynthesis
falls through to general-model. In the Playground demonstration, an existing
RISC-V Dashboard image connects to this Router revision. Dashboard adaptation is
outside the scope of this work. The maintained provider-mocker backend serves
fixed ok responses over SSE.
The keyword request selects code-model; Playground displays the decision and response details.
Default request: the same endpoint selects general-model. Restart behavior
was checked separately in the smoke test above.
HTTP workload results
We compared the X60 deployment with an AMD Ryzen 9 9950X host, using the same Router revision and the same Go benchmark source compiled natively for each architecture. The performance backend returned a fixed OpenAI-compatible JSON response; no LLM generation or classifier inference was involved.
| Setting | RISC-V | x86 |
|---|---|---|
| CPU | SpacemiT X60, 8 cores, 16 GB RAM | AMD Ryzen 9 9950X, 16 cores / 32 threads |
| Host OS/kernel | openKylin 2.0 SP1 / 6.6.63-cloud | Linux / 6.8.0-147-generic |
| Router limits | 2 CPUs, 1 GiB, GOMAXPROCS=2 | Same |
| Router logging | SR_LOG_LEVEL=warn | Same |
| Client and backend | Separate native Go processes, each GOMAXPROCS=2; built with Go 1.25.14 | Same |
| Test | Concurrency 1, 4, 16, 64; 15 seconds per path; 3 repeats | Same |
Each host ran two loopback paths: directly to a fixed-response Go backend, and through the Router to that backend. Workers alternated two short prompts, checked the returned model and ok content, and waited for each response before sending the next request. The client used an HTTP/1.1 connection pool and prepared request bodies before timing. Responses were buffered (stream: false); SSE was tested separately above.
Both hosts passed all 24 phases with zero request errors. X60 completed 42,897 timed requests through the Router; x86 completed 1,556,699. The table gives the median of three phase results, with warmups excluded. The p50 and p95 columns are medians of phase percentiles.
Median throughput over three runs. The vertical axis uses a logarithmic scale; solid lines pass through the Router and dashed lines call the backend directly.
| Host | Concurrency | Direct backend requests/s | Via Router requests/s | Via Router p50 (ms) | Via Router p95 (ms) |
|---|---|---|---|---|---|
| RISC-V / X60 | 1 | 849.96 | 121.29 | 7.969 | 9.771 |
| RISC-V / X60 | 4 | 2,907.56 | 277.63 | 13.270 | 24.172 |
| RISC-V / X60 | 16 | 3,321.48 | 276.03 | 54.673 | 96.529 |
| RISC-V / X60 | 64 | 3,146.10 | 273.18 | 227.671 | 343.737 |
| x86 / Ryzen 9 9950X | 1 | 49,847.88 | 4,745.47 | 0.196 | 0.283 |
| x86 / Ryzen 9 9950X | 4 | 125,278.92 | 10,533.89 | 0.306 | 0.815 |
| x86 / Ryzen 9 9950X | 16 | 157,307.86 | 10,795.92 | 1.261 | 3.303 |
| x86 / Ryzen 9 9950X | 64 | 161,315.63 | 10,444.91 | 5.839 | 11.806 |
Where throughput levels off
At concurrency 4 and above, X60 sustained about 270–280 requests/s and x86 about 10,000–11,000. More concurrent requests increased latency without increasing throughput. Direct-backend throughput was about 10–12 times the routed throughput on X60 and 12–16 times on x86.
Average CPU cores used at concurrency 16, taking the median of three runs. Both Router containers had a two-CPU quota.
The X60 Router used 1.97 CPU cores, versus 0.43 for the client and 0.33 for the backend. On x86, the Router used 1.99 cores. The load generator and backend had room to spare on both hosts, while the Router approached its two-CPU allowance. The measured path includes HTTP handling, routing, forwarding and response processing.
Measurement details and variation
- Each path received 20 warmup requests, with another warmup attempt per worker before each phase. Path order alternated between repeats. This was a closed-loop test; actual connection reuse was not counted.
- Client and backend processes were outside the Router's CPU quota but shared its host. Other workloads remained running, without CPU pinning. One-minute load averages changed from 2.20 to 5.68 on X60 and from 0.15 to 5.92 on x86.
- Across both paths, X60 completed 504,194 timed requests and x86 22,913,211. Latency quantiles were computed per phase, not from pooled samples.
- CPU estimates used process user+system time sampled about every 0.5 seconds. Phase-boundary offsets reached 0.25 seconds on X60 and 0.47 seconds on x86. Accumulated CPU-quota throttled time was below 0.1 seconds in each roughly 15-second routed phase.
- x86 routed throughput at concurrency 4 ranged from 6,024 to 10,688 requests/s; at concurrency 64 it ranged from 10,339 to 10,509. The cause of this variation was not profiled.
Timing the internal routing stage
The Router reports its internal routing interval in x-vsr-routing-latency-ms. Every timed routed response contained a valid value.
| Host | Concurrency | Internal routing p50 (ms) | Internal routing p95 (ms) |
|---|---|---|---|
| RISC-V / X60 | 1 | 2.540 | 3.119 |
| RISC-V / X60 | 4 | 2.741 | 6.384 |
| RISC-V / X60 | 16 | 2.785 | 17.284 |
| RISC-V / X60 | 64 | 2.850 | 66.078 |
| x86 / Ryzen 9 9950X | 1 | 0.066 | 0.094 |
| x86 / Ryzen 9 9950X | 4 | 0.061 | 0.162 |
| x86 / Ryzen 9 9950X | 16 | 0.059 | 0.209 |
| x86 / Ryzen 9 9950X | 64 | 0.061 | 0.284 |
The header measures wall-clock time from request-body processing to the routing timestamp before forwarding, including scheduling waits. It excludes other gateway work, backend waiting and response processing. The end-to-end table includes those costs plus client-side work; its percentiles cannot be subtracted to obtain a Router-only latency.
At concurrency 4 and above, the 9950X deployment delivered about 38–39 times the X60's routed throughput. Both Routers were close to their two-CPU quota. The hosts differ in core design, clock speed, cache and kernel; the ratio describes these two systems running this workload, rather than RISC-V and x86 processors in general.
Reproducing the measurement
The benchmark tools are available in the author's fork. They contain the Go client and backend, tests and a measurement script. The script starts them with the Router, samples CPU use and cleans up afterward.
From the directory containing the accompanying benchmark files, check out the tested source revision:
git clone --no-checkout \
https://github.com/LeON-Nie-code/semantic-router.git semantic-router-tools
git -C semantic-router-tools checkout --detach \
cbeef26dae09690fb2f0b83c0114e4424df3706f
cd semantic-router-tools/bench/router_flow/standalone_http
git clone --no-checkout https://github.com/LeON-Nie-code/semantic-router.git \
semantic-router-benchmark
git -C semantic-router-benchmark checkout --detach \
2e89999771dc3bb2458e1bc86ef00c4e918dd591
VSR_SOURCE=$(realpath semantic-router-benchmark)
On RISC-V, build semantic-router-router:local from that checkout with the native build commands above. On x86, use the attached image's amd64 build path instead:
docker buildx build --platform linux/amd64 --target vllm-sr-attached --load \
-f "$VSR_SOURCE/tools/docker/Dockerfile.attached" \
-t semantic-router-router:local "$VSR_SOURCE"
Then build and run the benchmark on each host:
(cd go-router-probe && CGO_ENABLED=0 go build -trimpath -o ./go-router-probe .)
VSR_BENCH_RESULTS=$(mktemp -d)
sudo python3 ./run-go-probe.py \
--source "$VSR_SOURCE" \
--helper "$PWD/go-router-probe/go-router-probe" \
--image semantic-router-router:local \
--seconds 15 --repeats 3 --concurrencies 1,4,16,64 \
--output "$VSR_BENCH_RESULTS/run"
Use Go 1.25.14 and an image built from the pinned checkout. The script needs Docker access and permission to read container CPU counters, and uses host networking and temporary mounts. The results summary includes phase quantiles, repeated-run ranges and CPU estimates.
Support status
The proposed experimental support-matrix entry covers the standalone Router build and the routing, buffered/SSE response and restart checks described here. Envoy + ExtProc, local model inference, Dashboard, CLI stack orchestration, Operator and Kubernetes deployment are outside this PR's scope. Optional cache, knowledge-base, storage and plugin integrations have not been comprehensively tested.
The port is tracked in PR #5259 and #5310. Official RISC-V image publication remains to be arranged; the commands above build a local image.