Architecture#

Serving a model and running work against it are two different problems, and domyn-swarm keeps them separate. This boundary determines the structure of the system.

DomynLLMSwarm owns the lifecycle#

DomynLLMSwarm is a context manager. Entering it starts an endpoint and waits for readiness. Leaving it stops the endpoint unless delete_on_exit=False keeps the allocation alive.

with DomynLLMSwarm(cfg=cfg) as swarm:
    swarm.submit_job(
        job,
        run=JobRunSpec(input_path=Path("in.parquet"), output_path=Path("out.parquet")),
    )

It is a thin coordinator. The platform-specific work lives below it.

Deployment pairs a serving backend with a compute backend#

Deployment composes exactly two collaborators and nothing else:

  • ServingBackend owns the model endpoint.

  • ComputeBackend owns the processes that call that endpoint.

Its flow is up(name, ctx) → wait_ready(timeout_s) → run(...) → down(handle). It is also a context manager. A raised exception still deletes the endpoint instead of leaving an allocation active.

The handle passed between those calls is a ServingHandle, carrying the endpoint URL and whatever platform metadata the backend needs to find its resources again.

What each side is responsible for#

The serving side provides:

create_or_update

create the endpoint, or reconcile an existing one, returning a ServingHandle

wait_ready

block until the endpoint can serve requests

status

report a ServingStatus, which is what domyn-swarm status surfaces

delete

remove the endpoint

The compute side provides:

submit

start a job, returning a JobHandle

wait

block until the job finishes, returning a JobStatus

cancel

stop a running job

probe

report a JobProbe for a handle, which is how status refresh works

Compute backends also supply default_python, default_image, default_resources, and default_env. A job submission does not need to name values that the platform can infer. DefaultComputeMixin provides common implementations.

Both are Protocol definitions rather than base classes. A backend satisfies them through its methods. See Serving vs compute backends.

Jobs run beside the endpoint, not inside it#

A job never runs in the serving container. The compute backend starts a separate process. Slurm uses srun, and Lepton uses a batch job. The process receives ENDPOINT and MODEL in its environment and calls the endpoint over HTTP.

As a result, job submit-script can run arbitrary Python. The endpoint handles a SwarmJob and a submitted script in the same way. A new job type does not require a change to the serving backend.

See The SwarmJob lifecycle.

State makes swarms outlive processes#

Every swarm has a record in a local SQLite database. The record contains deployment metadata, resource handles, and the configuration that created the swarm. It also contains platform identifiers, node assignments, and the endpoint URL.

That record is why --name works. The original up process has ended by the time you run job submit --name my-swarm. The command restores the swarm from state. DomynLLMSwarm.from_state(name) and swarm list read the same record.

See Managing swarm state.

The load balancer reconciles rather than being configured#

Nginx needs the location of each replica. Slurm decides placement after submission, and replicas appear one at a time. The load-balancer configuration must converge as locations become available.

Each replica writes a replica-<id>.head file, holding its host:port, into the swarm’s shared serving directory. A supervisor process watches that directory and regenerates 00-upstreams.conf from the current files. It also writes Prometheus target files when monitoring is on. A file addition or removal represents a replica change.

The supervisor writes files but does not reload Nginx. Nginx runs in a separate Singularity instance with a private PID namespace. A process in another container cannot signal the Nginx master. The load-balancer script watches the generated file on the host. After a change, it runs nginx -t and then nginx -s reload through singularity exec instance://. It does not load an invalid configuration.

The same reconcile loop generates Prometheus targets from the replica head files. Prometheus can scrape a new replica without a manual configuration change. See Metrics and dashboards.

Health is reported, not inferred#

Each replica has a watchdog that probes it. The watchdog reports to one collector, which owns the health database for domyn-swarm status. See Watchdog and collector for the reason for this design.

Where to go next#