One Conversation, Multiple Models: Runkite and vLLM Semantic Router
Runkite website · GitHub · Runnable example and recorded results
An incident investigation can begin with a small request: read the incident and tell me which service is affected. The next question asks why it is failing. Eventually, someone asks the agent to restart the service.
The conversation has moved from reading to reasoning to acting. Those requests can use different model routes, but changing the model should not change the agent's permission to act. And if the process running the agent restarts along the way, the next question still belongs to the same conversation.
That is the path I wanted to test by connecting Runkite with vLLM Semantic Router. Runkite's existing managed Vercel AI SDK runner would execute the agent. Semantic Router would select the model backend for its inference requests. The SDK would keep its native tool loop, while model selection would live in the router configuration.
I followed one conversation through three turns: read an incident, restart the runner and ask for a diagnosis, then request a service restart the agent was forbidden to perform. The incident and tool service are fictional. The model calls, streamed tool requests and durable results are real.
To understand the result, start with what happens between a user's question and an agent's answer.