Deploy with Docker
Docker is the shortest path from a Semantic Router configuration to a running stack. It is a good fit for evaluation, development, CI, edge hosts, and single-host deployments that do not need Kubernetes scheduling or failover.
| Component | Who runs it |
|---|---|
| Router, Dashboard and the services the configuration needs | The CLI |
| An Envoy container in front of the Router | The CLI, with --gateway extproc (see Gateway Modes) |
| The Router's decision and embedding models | The model runtime, which the Router starts |
| The model servers that answer users | You |
A healthy Router stack doesn't mean its provider endpoints are installed, running, or able to generate. Test them directly.
Start the stack
Complete the Quickstart to install the CLI and create a configuration, or start from an existing canonical YAML file:
vllm-sr config validate --config config.yaml
vllm-sr serve --config config.yaml
| CLI | Pulls Router and Dashboard images tagged |
|---|---|
| Stable release | With its own release version |
| Development build | With the SHA of the main commit it was built from |
| Source checkout | :latest |
--image and the documented image environment overrides select a different
build when needed.
With no --config, vllm-sr serve uses config.yaml in the current
directory, or opens first-run setup in the Dashboard:
- During setup the command keeps running. When you activate a config in the Dashboard, it starts the Router from it and waits until it is ready.
- If you stop it first, the next
vllm-sr servestarts the Router, andvllm-sr statussays when setup is complete. - The Dashboard never gets the container runtime's socket.
The default local endpoints are:
| Endpoint | Default | Purpose |
|---|---|---|
| Dashboard | http://localhost:8700 | Configure and inspect the stack. |
| Routed listener | http://localhost:8899 | Send OpenAI-compatible model requests. |
| Management API | http://localhost:8080 | Validate config and use evaluation, replay, or vector-store APIs. |
| Router metrics | http://localhost:9190/metrics | Prometheus metrics, including the model runtime's vsr_model_runtime_*. |
Ports can change with the active configuration or a stack port offset. Use
vllm-sr status when you are unsure which endpoints are active.
The Dashboard binds to 127.0.0.1 by default, and first-run admin
registration stays available from the local host. For a remote machine,
forward the port with ssh -L 8700:127.0.0.1:8700 <host> and open the local
Dashboard URL.
Set VLLM_SR_DASHBOARD_HOST_BIND=0.0.0.0, and either provision both
DASHBOARD_ADMIN_EMAIL and DASHBOARD_ADMIN_PASSWORD, or set
DASHBOARD_ALLOW_OPEN_BOOTSTRAP=true when first-admin registration on that
network is intended. The CLI rejects a wildcard bind with neither.
After starting containers, the CLI waits up to 1800 seconds for the Router to
be ready, or for the Dashboard during first-run setup. Set
--startup-timeout SECONDS when model loading or GPU compilation needs a
different budget:
vllm-sr serve --config config.yaml --startup-timeout 7200
- Measure first. Choose the budget from startup measurements for your models and hardware.
- Startup only. It applies to the local Docker target, and doesn't change inference request deadlines.
- On timeout, the CLI exits with an error and leaves the containers for
vllm-sr statusandvllm-sr logs router. Usevllm-sr stopto stop them.
Connect model backends
Choose the connection that matches where the model server runs:
| Model location | Configure the backend with |
|---|---|
| On the Docker host | host.docker.internal:<port>; the CLI adds the host-gateway mapping. |
| In a container on the same network | The model container's DNS name and service port. |
| On another host or managed service | Its reachable HTTPS base URL and environment-backed credentials. |
For a small local model, follow Configure models with Ollama. For a GPU-backed vLLM server, choose a guide under Hardware. In every case, verify the model endpoint directly before debugging routing.
Operate a local stack
vllm-sr status
vllm-sr logs router -f
vllm-sr dashboard
vllm-sr stop
With --gateway extproc, vllm-sr logs envoy shows the Envoy container's log
too; in standalone mode there is no Envoy container, and the command says so.
Use --minimal to run without the Dashboard and the observability stack
(Jaeger, Prometheus, Grafana). Use --readonly to keep the Dashboard available
without allowing configuration changes. Pin images and review
Security Hardening before exposing a listener beyond a
trusted host.
Keep the stack running across crashes and reboots
The containers that vllm-sr serve creates have no restart policy by default: a
crashed container stays down, and the stack does not come back after a host
reboot. On a server, start it with a policy:
vllm-sr serve --config config.yaml --restart-policy unless-stopped
| Policy | Behavior |
|---|---|
unless-stopped, always | Restart a container that exits, and start the stack again when the Docker daemon starts. |
on-failure | Restart only containers that fail. |
vllm-sr stopstill stops the stack.- The policy applies to the containers that
servecreates, so pass it on everyserve. - With Podman, also enable its restart service
(
systemctl enable --now podman-restart.service) so containers come back after a reboot.
With --gateway extproc, Envoy logs at the safe info level by default. To
troubleshoot, set VLLM_SR_ENVOY_LOG_LEVEL=debug before starting the stack,
and unset it when finished: debug logs can expose forwarded request headers,
including provider Authorization headers.
Expose the Dashboard through a reverse proxy
Grafana Live validates the Origin header used to establish its WebSocket.
When an HTTPS reverse proxy exposes the Dashboard on a public hostname, allow
that exact origin before starting the stack:
export GF_LIVE_ALLOWED_ORIGINS='https://dashboard.example.com'
vllm-sr serve --config config.yaml
- Separate multiple origins with commas, in
scheme://host[:port]form, without paths, queries, fragments or trailing slashes. - Grafana supports wildcard patterns, but exact origins are safer for public deployments.
vllm-sr servewrites the value into the generated.vllm-sr/grafana/grafana.serve.ini; restart the stack after changing it.
When to move to Kubernetes
Docker does not provide multi-node scheduling, rolling deployment control, or cluster-level recovery. Move to a Kubernetes path when you need multi-node Router replicas, declarative rollout, gateway integration, or platform-managed model discovery. The same canonical configuration can be deployed with the CLI and Helm or managed through the Semantic Router Operator.