Architecture

How Substrate works

A control plane, a router and a node agent. That is the whole system.

Three components

Each one is a single static binary. There is no database to operate — state lives in embedded Raft, replicated across the control plane nodes.

Holds cluster state, model registry and policy. Runs three or five replicas. Survives losing a minority without dropping traffic.

Stateless. Terminates the OpenAI-compatible API, applies per-tenant limits, and picks the node with the warmest matching cache.

Owns the GPUs on one machine. Loads engines, reports pressure, and handles graceful pre-emption on a signal from the scheduler.

Requests sharing a system prompt land on the node that already has those KV blocks resident.

Token counts and GPU-seconds attributed by API key, exportable as a metered usage feed.

Load the new weights beside the old ones, shift traffic by percentage, roll back on error rate.

Grammar-constrained decoding across every supported engine, with the same request shape.

Optional. When local capacity is exhausted, overflow to a provider you configure — off by default.

Every policy change and admin action, signed and appended, shipped wherever you send logs.

We will run a benchmark on your traces before you commit to anything.