domyn-swarm#

A CLI and Python library for launching LLM serving endpoints and running high-throughput batch jobs against them. It supports Slurm and NVIDIA DGX Cloud Lepton.

domyn-swarm up -c config.yaml
domyn-swarm job submit --name my-swarm --input prompts.parquet --output answers.parquet
domyn-swarm down my-swarm

Why domyn-swarm#

Model serving and batch processing often use separate tools. domyn-swarm joins them in one workflow. A YAML file describes the swarm. up starts replicas behind a load balancer and waits for them to respond. job submit runs a typed job with batching, retries, and checkpointing. down removes the deployment.

The same commands work on an HPC cluster and in the cloud. Only the configuration’s backend section changes.

Get started

Install, launch your first swarm, and write your first custom job.

Getting started
Guides

Checkpointing, sharding, data backends, and the platform-specific details.

Guides
Concepts

How the pieces fit together, and why they are split the way they are.

Concepts
Reference

CLI, configuration and Python API, generated from the source.

Reference

What it does#

  • One CLI supports Slurm and Lepton. up, job submit, status, and down use the same interface on both platforms.

  • Health probes run before up returns. Jobs do not start while a model is still loading.

  • Jobs support bounded concurrency, retries with backoff, and Parquet checkpoints. An interrupted job can resume from completed work.

  • job submit-script submits a Python file when a job class does not fit the task.

  • A local SQLite database stores swarm state. Later commands can address the swarm by name.

  • Separate serving and compute protocols isolate platform-specific code.

Supported backends#

Serving and compute backends control where models and jobs run:

Backend

Notes

Slurm

Singularity containers and a job array for replicas, behind an Nginx load balancer

NVIDIA DGX Cloud Lepton

Endpoint plus batch job through the Lepton SDK (domyn-swarm[lepton])

Data backends control how jobs read, write, and iterate over data. See Choosing a data backend:

Backend

Install

pandas

always available and used by default

polars

domyn-swarm[polars]

ray

domyn-swarm[ray]

Where to go next#

New here? Installation then Quickstart is about fifteen minutes end to end.

Already running jobs? Checkpointing and resuming and Sharding and concurrency are the two guides that most change how a long run behaves.

Debugging something? Monitoring and troubleshooting starts from the symptom.