Skip to main content
Version: Latest (main)

Deploy with Docker

Docker is the shortest path from a Semantic Router configuration to a running stack. It is a good fit for evaluation, development, CI, edge hosts, and single-host deployments that do not need Kubernetes scheduling or failover.

ComponentWho runs it
Router, Dashboard and the services the configuration needsThe CLI
An Envoy container in front of the RouterThe CLI, with --gateway extproc (see Gateway Modes)
The Router's decision and embedding modelsThe model runtime, which the Router starts
The model servers that answer usersYou
note

A healthy Router stack doesn't mean its provider endpoints are installed, running, or able to generate. Test them directly.

Start the stack​

Complete the Quickstart to install the CLI and create a configuration, or start from an existing canonical YAML file:

vllm-sr config validate --config config.yaml
vllm-sr serve --config config.yaml
CLIPulls Router and Dashboard images tagged
Stable releaseWith its own release version
Development buildWith the SHA of the main commit it was built from
Source checkout:latest

--image and the documented image environment overrides select a different build when needed.

With no --config, vllm-sr serve uses config.yaml in the current directory, or opens first-run setup in the Dashboard:

  • During setup the command keeps running. When you activate a config in the Dashboard, it starts the Router from it and waits until it is ready.
  • If you stop it first, the next vllm-sr serve starts the Router, and vllm-sr status says when setup is complete.
  • The Dashboard never gets the container runtime's socket.

The default local endpoints are:

EndpointDefaultPurpose
Dashboardhttp://localhost:8700Configure and inspect the stack.
Routed listenerhttp://localhost:8899Send OpenAI-compatible model requests.
Management APIhttp://localhost:8080Validate config and use evaluation, replay, or vector-store APIs.
Router metricshttp://localhost:9190/metricsPrometheus metrics, including the model runtime's vsr_model_runtime_*.

Ports can change with the active configuration or a stack port offset. Use vllm-sr status when you are unsure which endpoints are active.

The Dashboard binds to 127.0.0.1 by default, and first-run admin registration stays available from the local host. For a remote machine, forward the port with ssh -L 8700:127.0.0.1:8700 <host> and open the local Dashboard URL.

Publishing the Dashboard on all interfaces

Set VLLM_SR_DASHBOARD_HOST_BIND=0.0.0.0, and either provision both DASHBOARD_ADMIN_EMAIL and DASHBOARD_ADMIN_PASSWORD, or set DASHBOARD_ALLOW_OPEN_BOOTSTRAP=true when first-admin registration on that network is intended. The CLI rejects a wildcard bind with neither.

After starting containers, the CLI waits up to 1800 seconds for the Router to be ready, or for the Dashboard during first-run setup. Set --startup-timeout SECONDS when model loading or GPU compilation needs a different budget:

vllm-sr serve --config config.yaml --startup-timeout 7200
  • Measure first. Choose the budget from startup measurements for your models and hardware.
  • Startup only. It applies to the local Docker target, and doesn't change inference request deadlines.
  • On timeout, the CLI exits with an error and leaves the containers for vllm-sr status and vllm-sr logs router. Use vllm-sr stop to stop them.

Connect model backends​

Choose the connection that matches where the model server runs:

Model locationConfigure the backend with
On the Docker hosthost.docker.internal:<port>; the CLI adds the host-gateway mapping.
In a container on the same networkThe model container's DNS name and service port.
On another host or managed serviceIts reachable HTTPS base URL and environment-backed credentials.

For a small local model, follow Configure models with Ollama. For a GPU-backed vLLM server, choose a guide under Hardware. In every case, verify the model endpoint directly before debugging routing.

Operate a local stack​

vllm-sr status
vllm-sr logs router -f
vllm-sr dashboard
vllm-sr stop

With --gateway extproc, vllm-sr logs envoy shows the Envoy container's log too; in standalone mode there is no Envoy container, and the command says so.

Use --minimal to run without the Dashboard and the observability stack (Jaeger, Prometheus, Grafana). Use --readonly to keep the Dashboard available without allowing configuration changes. Pin images and review Security Hardening before exposing a listener beyond a trusted host.

Keep the stack running across crashes and reboots​

The containers that vllm-sr serve creates have no restart policy by default: a crashed container stays down, and the stack does not come back after a host reboot. On a server, start it with a policy:

vllm-sr serve --config config.yaml --restart-policy unless-stopped
PolicyBehavior
unless-stopped, alwaysRestart a container that exits, and start the stack again when the Docker daemon starts.
on-failureRestart only containers that fail.
  • vllm-sr stop still stops the stack.
  • The policy applies to the containers that serve creates, so pass it on every serve.
  • With Podman, also enable its restart service (systemctl enable --now podman-restart.service) so containers come back after a reboot.
Envoy debug logs

With --gateway extproc, Envoy logs at the safe info level by default. To troubleshoot, set VLLM_SR_ENVOY_LOG_LEVEL=debug before starting the stack, and unset it when finished: debug logs can expose forwarded request headers, including provider Authorization headers.

Expose the Dashboard through a reverse proxy​

Grafana Live validates the Origin header used to establish its WebSocket. When an HTTPS reverse proxy exposes the Dashboard on a public hostname, allow that exact origin before starting the stack:

export GF_LIVE_ALLOWED_ORIGINS='https://dashboard.example.com'
vllm-sr serve --config config.yaml
  • Separate multiple origins with commas, in scheme://host[:port] form, without paths, queries, fragments or trailing slashes.
  • Grafana supports wildcard patterns, but exact origins are safer for public deployments.
  • vllm-sr serve writes the value into the generated .vllm-sr/grafana/grafana.serve.ini; restart the stack after changing it.

When to move to Kubernetes​

Docker does not provide multi-node scheduling, rolling deployment control, or cluster-level recovery. Move to a Kubernetes path when you need multi-node Router replicas, declarative rollout, gateway integration, or platform-managed model discovery. The same canonical configuration can be deployed with the CLI and Helm or managed through the Semantic Router Operator.