KTMS LLM gateway added local nodes: own GPUs as a separate vendor

On October 30, 2026, an article was published about operating Keystone. The Model Switchboard. (KTMS) — an open-source LLM gateway (a layer between the application and models) in Go. Last time it was about routing and money. Now — about how to connect your own GPUs and what to do when something fails.

The difference from the previous material is in focus. Previously, we examined where the gateway sends requests and how it calculates cost. Now — operations: nodes, dependency failures, encryption key rotation, verification of the downloaded release, and zero-downtime updates. That is, the article is about what happens after the route is chosen.

In practice, the scheme is as follows. The gateway administrator issues a node key bound to its identifier. The key is shown once. Then registers a model with vendor (vendor — source of the model) local and parameters such as queue_timeout_ms. On the machine itself, installation comes down to one command: the script installs Ollama (local model runner), the model itself, and the node agent, and starts them as background services. In the example from the article — node anna-mac and model gemma4:26b-a4b-it-qat. Requests go to local machines first, cloud connects when nodes are busy. The logic is simple: many teams have a Mac Studio or a Linux machine with a card that sits idle, and a bill for cloud requests that a local model would have answered no worse.

What remained unknown. Details of handling failures of each dependency, the procedure for encryption key rotation, the method of verifying that exactly the required release was downloaded, and the mechanics of zero-downtime updates were announced by the author, but there are no specifics in the retelling. No data is provided in the note about how this is arranged internally.