Submitting jobs#

domyn-swarm job submit runs a typed job — a DataFrame in, a DataFrame out — against a running swarm. This page covers the choices that matter; the CLI reference carries every flag and its exact signature.

domyn-swarm job submit \
  my_module:CustomCompletionJob \
  --name my-swarm-name \
  --job-kwargs '{"request_params":{"temperature":0.2}}' \
  --checkpoint-interval 16 \
  --input prompts.parquet \
  --output answers.parquet

Choosing the job class#

The positional argument is <module>:<ClassName>, naming a class that implements SwarmJob. It defaults to domyn_swarm.jobs:ChatCompletionJob.

--job-kwargs takes a JSON object passed to the job’s constructor as configuration overrides. OpenAI parameters reach the client through its request_params key:

--job-kwargs '{"request_params":{"temperature":0.2,"top_p":0.9}}'

Writing your own class: Your first custom job.

Input and output paths#

--input and --output take a Parquet file or a directory holding a Parquet dataset, and both must be on a filesystem shared with the compute nodes.

Input additionally supports:

  • numeric brace ranges — input_00{0978..1955}.parquet, and the {0978-1955} form, expanding to a file range

  • glob patterns with the pandas backend — data-*.parquet

--limit / -l caps how many rows are read, which is what you want while debugging against a small slice of a large dataset.

Attaching to a swarm#

Exactly one of:

  • --name — attach to an existing swarm, found via the state record

  • --config — start a fresh swarm from YAML for this job, then tear it down

Passing both is an error.

Row identity#

--id-column / --id-col names a column to use as a stable row id. It matters more than it looks:

  • without it, pandas uses the DataFrame index and polars generates a _row_id

  • resume is then stable only if the input ordering and scan are identical across runs

For robust resume across restarts, or across different scan graphs, provide a stable id column. See Checkpointing and resuming.

Checkpointing#

Progress is written out periodically so a failed run can resume instead of starting over. --checkpoint-interval sets the flush frequency in items, --checkpoint-tag gives a run a stable identity to resume against, and --no-resume and --no-checkpointing opt out in different ways.

Details, and the resume semantics that go with sharding: Checkpointing and resuming.

Concurrency and sharding#

--max-concurrency bounds in-flight requests to the endpoint. --num-shards splits the input into shards. --shard-mode picks id for stable hashing or index for legacy row-order sharding. --retries and --timeout govern individual request failures.

These interact with resume, so read Sharding and concurrency before changing --num-shards on a run you intend to resume.

Choosing a data backend#

--data-backend selects pandas, polars or ray, and --runner selects pandas or arrow for the non-Ray backends. --native-backend enables native execution and is required for Ray, with --ray-address pointing at the cluster. --native-batch-size, --backend-read-kwargs and --backend-write-kwargs tune the backend further.

Which to use, and why: Choosing a data backend.

Output layout#

With a directory output and the Polars runner, --shard-output writes one Parquet file per shard, based on --num-shards, using the checkpoint outputs as the source of truth.

Running detached#

--detach runs the job in a separate process and prints its PID, so the job survives your terminal. Track it afterwards with domyn-swarm job list, job status, job wait and job cancel — see Managing swarm state.

--mail-user enables email notification on completion where the compute backend supports it.

Sizing the job step on Slurm#

On Slurm, the job runs as an srun step. The step size depends on where you submit from:

  • Without endpoint.require_allocated_node, the step runs in the load-balancer allocation, with endpoint.mem and endpoint.cpus_per_task.

  • With endpoint.require_allocated_node: true, you must submit from your own sbatch or salloc allocation. The step gets all the memory of that allocation (--mem=0) and its --cpus-per-task value. A shell inside the load-balancer job, for example from srun --jobid=<lb_jobid> --pty bash, is refused.

--mem and --cpus-per-task override these defaults for one job:

domyn-swarm job submit \
  --name my-swarm-name \
  --input data.parquet --output out.parquet \
  --mem 64G --cpus-per-task 16

If your allocation does not set --cpus-per-task, srun gives the step one CPU. In that case, pass --cpus-per-task. Lepton rejects both options because backend.job.resource_shape sets the job size.

Submitting a plain script#

When the job shape does not fit, send a script to the head node instead:

domyn-swarm job submit-script \
  --name my-swarm-name \
  path/to/script.py -- --foo 1 --bar 2

  • the script file must exist

  • exactly one of --config or --name, as above

  • everything after -- is forwarded to your script, not consumed by domyn-swarm

  • --mem and --cpus-per-task size the step as for job submit

The script runs with ENDPOINT and MODEL already set in its environment, so it can build its own client against the swarm.