Guides

Task-oriented, and mostly independent of one another — read the one you need.

Platforms

• Running on Slurm — Singularity images, bind mounts, modules, node selection

• Running on Lepton — endpoint and job config, secrets, and how it differs from Slurm

Running work

• Submitting jobs — input formats, job classes, the flags that matter

• Checkpointing and resuming — surviving a failed run

• Sharding and concurrency — the two dials, and how they affect resume

• Choosing a data backend — pandas, polars or Ray

Operating

• Managing swarm state — the state database, db commands, inspecting swarms

• Monitoring and troubleshooting — reading status, and what to do when a replica is unhealthy

• Metrics and dashboards — Prometheus, GPU exporters and domyn-swarm monitor

• Swarm pools — incomplete; read before trying to use it

Extending

• Implementing a backend — adding a platform: the two protocols, the config model, and which features are not backend-agnostic

For why things are built this way rather than how to use them, see Concepts. For exhaustive flag and field lists, see Reference.