Serving vs compute backends#

Note

“Backend” means two unrelated things in domyn-swarm. This page is about serving and compute backends, which decide where the model runs and where jobs execute. A data backend decides how job input and output are read and written — that is Choosing a data backend. Readers conflate these constantly.

Why two protocols instead of one backend type#

A single Backend abstraction would have to answer both “how do I serve a model here” and “how do I run a process here”, and those questions have different shapes and different failure modes. Splitting them buys three things:

Mixing. Serving and compute are chosen independently, so a model served in one place can be driven by work executing somewhere else.
Independent evolution. Adding a readiness strategy touches only the serving side; adding a way to launch processes touches only compute.
Honest health reporting. “The endpoint is ready” and “the job succeeded” are separate states with separate owners, which is why domyn-swarm status can report a healthy endpoint and a failed job at once.

Both are typing.Protocol definitions, so a backend conforms structurally — no inheritance required, and no base class to fight when a platform does something unusual.

ServingBackend#

Owns the model endpoint and its lifetime.

Method	Responsibility
`create_or_update(name, spec, extras)`	Create the endpoint or reconcile an existing one; returns a `ServingHandle`
`wait_ready(handle, timeout_s, extras)`	Block until the endpoint can serve
`ensure_ready(handle)`	Assert readiness for an already-created endpoint
`status(handle)`	Report a `ServingStatus`, carrying a `ServingPhase`
`delete(handle)`	Remove the endpoint

create_or_update rather than create is deliberate: re-running up against an existing swarm reconciles instead of failing or duplicating.

ComputeBackend#

Owns the processes that call the endpoint.

Method	Responsibility
`submit(...)`	Start a job; returns a `JobHandle`
`wait(handle, stream_logs=True)`	Block until completion; returns a `JobStatus`
`cancel(handle)`	Stop a running job
`probe(handle)`	Report a `JobProbe`, which is how `job status --refresh` works
`default_python(cfg)`	Interpreter to run the job with
`default_image(cfg)`	Container image, where the platform uses one
`default_resources(cfg)`	Platform resource request
`default_env(cfg)`	Environment the job needs

The four default_* methods keep platform knowledge out of the submission path: the caller does not need to know that Slurm wants a venv_path while Lepton wants a Docker image. DefaultComputeMixin supplies the common implementations, including a probe that most backends can use unchanged.

How the two platforms satisfy them#

Slurm. Serving is an array job of vLLM replicas plus an Nginx load-balancer job; readiness is an HTTP probe against /v1/health through the load balancer. Compute is srun into the allocation, with require_allocated_node guarding against work landing on the load-balancer node.

Lepton. Serving is a Lepton endpoint, which fronts its own replicas, so there is no load balancer to manage; readiness is deployment-state polling rather than an HTTP probe. Compute is a Lepton batch job.

Notice that the readiness strategies differ entirely while the protocol does not. That is the separation earning its keep.

Adding a platform#

The README describes new targets as “easy to add”, which is worth qualifying. What the protocols genuinely give you is a closed list of methods and no base class to satisfy — the interface is small and the compiler-checkable part is straightforward.

What they do not remove is the real work: readiness semantics, log retrieval, identifier mapping into ServingHandle and JobHandle, cancellation that actually stops things, and the config model plus its type discriminator. Look at backends/serving/ and backends/compute/ for either existing platform to gauge it honestly — the protocol is a day, the semantics are not.

For the step-by-step version — the method sets, the handle contract, the registration line, and which features are not backend-agnostic — see Implementing a backend.