Running on Slurm#

On Slurm, a swarm is a job array of vLLM replicas fronted by an Nginx load balancer, with roles assigned by SLURM_NODEID.

What up submits#

domyn-swarm up -c config.yaml --replicas 3

  • -c/--config — path to your YAML config

  • -r/--replicas — override the replica count from the config

Three things happen:

  1. an array job starts one vLLM server per replica

  2. a load-balancer job starts Nginx, which waits until every replica is answering, then exposes a single endpoint

  3. a SQLite state record is created or updated, which is what lets later commands address the swarm by name

Building the Singularity images#

domyn-swarm uses Singularity as its container engine on Slurm. Build the two required images from the definition files in examples/singularity_images/, on a machine with sudo, or with --fakeroot if your site enables it:

# NGINX load balancer image
sudo singularity build nginx.sif examples/singularity_images/nginx.def

# vLLM runtime image
sudo singularity build vllm.sif examples/singularity_images/vllm.def

Then point the config at the results:

model: "deepseek-ai/DeepSeek-R1-0528"  # whatever model you want to deploy
image: /shared/images/vllm.sif         # vLLM container; optional if you run from a venv
backend:
  type: slurm
  endpoint:
    nginx_image: /shared/images/nginx.sif  # required for the load balancer

Practical notes:

  • Put the .sif files on a shared path readable by every compute node.

  • Singularity must be available on the execution nodes.

  • If your site disables --fakeroot, build with admin privileges elsewhere and copy the .sif onto the shared filesystem.

  • image is optional if you run vLLM from a virtual environment instead of a container; endpoint.nginx_image is not.

Bind mounts#

backend.mounts adds bind mounts to the vLLM containers:

backend:
  type: slurm
  mounts:
    - /scratch/datasets                # bound at the same path inside the container
    - /host/config:/etc/app/config:ro  # host:container, with an option

Each entry is either /path, bound at the same path inside the container, or /host/path:/container/path with an optional :ro or :rw suffix.

Entries are passed verbatim to Singularity’s --bind. domyn-swarm validates only the basic shape — an absolute source path, at most a source:dest:opts triple — and the container runtime does the actual binding and reports its own errors, for instance a missing host path or an invalid option. For the full bind syntax see Apptainer or SingularityCE.

Modules and sbatch preamble#

Two fields inject site-specific setup into the generated cluster script:

backend:
  type: slurm
  modules:
    - cuda/12.1
    - singularity
  preamble:
    - "#SBATCH --exclusive"
    - "export NCCL_DEBUG=WARN"

modules become module load lines; preamble lines are inserted near the top of the script, before the module loads, which makes them suitable for extra sbatch directives or shell setup.

Replicas larger than one node#

A replica that needs more GPUs than a single node has cannot be one process, so it becomes a Ray cluster: one head and enough workers to reach gpus_per_replica, with vLLM’s tensor parallelism spanning them.

You do not ask for this. It follows from the arithmetic:

model: "deepseek-ai/DeepSeek-R1-0528"
gpus_per_replica: 16     # more than one node holds
gpus_per_node: 4         # so each replica spans four nodes
replicas: 2

gpus_per_replica > gpus_per_node sets requires_ray, which in turn:

  • renders the sbatch script from llm_swarm_ray.sh.j2 instead of llm_swarm.sh.j2 — both share _swarm_common.sh.j2, so the two paths differ only where Ray genuinely requires it

  • enables watchdog.ray, so cluster liveness is checked alongside the HTTP probe

  • enables monitoring.ray_metrics when monitoring is on, adding Ray’s own metrics and a group of Ray panels to the dashboard

gpus_per_replica must be a multiple of gpus_per_node in this case. Anything else would leave a partly-used node inside a replica, and it is rejected with When gpus_per_replica > gpus_per_node, gpus_per_replica must be a multiple of gpus_per_node.

backend.template_path pins the template if you need your own, which also disables the automatic choice — so a custom template has to handle Ray itself if the deployment needs it.

Node selection and limits#

backend.partition, account and qos are required. Beyond those, exclude_nodes and node_list accept Slurm’s own syntax (node[001-004]), time_limit caps the allocation, and mail_user enables END and FAIL notifications. The load-balancer job is configured separately under backend.endpoint, including its own optional qos override.

Full field list: Configuration.

Shutting down#

domyn-swarm down my-swarm-name

Takes a swarm name and stops the load balancer and every replica job via scancel.

Metrics#

The load balancer can also run Prometheus and a GPU exporter as sidecars, giving you throughput and GPU dashboards over the swarm. Slurm-only, off by default — see Metrics and dashboards.