Metrics and dashboards#

domyn-swarm can run a Prometheus instance next to the load balancer, scrape every vLLM replica, and give you a dashboard over the result.

This is about quantities — tokens per second, queue depth, GPU utilisation. For is it broken, see Monitoring and troubleshooting, which reads health rather than metrics.

Off by default, and Slurm only: it is built out of sidecars on the load-balancer node, which the Lepton backend does not have.

Turning it on#

backend:
  type: slurm
  endpoint:
    monitoring:
      enabled: true
      prometheus_image: /path/to/prometheus.sif
      nginx_exporter_image: /path/to/nginx-prometheus-exporter.sif
      retention: 12h

enabled is a master switch: with it off, every other field here is ignored and the load balancer behaves exactly as it did before monitoring existed.

Two images are needed because two sidecars run — Prometheus itself, and nginx-prometheus-exporter, which turns Nginx’s status page into metrics. To run host binaries instead of containers:

monitoring:
  enabled: true
  mode: binary
  prometheus_binary: /path/to/prometheus
  nginx_exporter_binary: /path/to/nginx-prometheus-exporter

Both binaries default to being looked up on PATH by name, so if they are already installed cluster-wide, mode: binary alone is enough.

What gets scraped#

Job	Target	Source
`vllm`	every replica’s `/metrics`	discovered from a file the supervisor keeps current
`nginx`	`nginx-prometheus-exporter` on `exporter_port`	static
`gpu`	each node’s GPU exporter	only with `gpu_exporter.enabled`
`gpu_ownership`	`/gpu_ownership` on the endpoint	only with `gpu_exporter.enabled`
`ray`	each node’s Ray metrics port	only with `ray_metrics.enabled`

Replica targets are not static — replicas come and go, and their host and port are only known once Slurm has placed them. Each replica writes a replica-<id>.head file into the swarm’s serving directory, and the load-balancer supervisor turns those into both Nginx’s upstreams and Prometheus’s target file. Adding a replica therefore adds a scrape target with no reconfiguration.

Every series is labelled swarm with the swarm’s name, from Prometheus’s external_labels.

Reaching it#

Prometheus is served through the load balancer, at route_prefix on the same endpoint URL as the model:

http://<endpoint>/prometheus

The port field (default 9090) is where Prometheus listens on the load-balancer node, behind that proxy — it is not the port you connect to.

Warning

Nothing authenticates /prometheus. Anyone who can reach the swarm’s endpoint can read it, and can read the metrics of every replica. Treat it as internal to your cluster.

The database is node-local and ephemeral. It lives on the load-balancer node and disappears when that job ends, so retention (default 12h) only caps a single run’s history. If you need metrics to outlive a swarm, point an external Prometheus at the same /prometheus URL and let it federate.

Being a plain Prometheus over HTTP, it is not tied to any particular viewer — see Viewing the metrics.

Viewing the metrics#

Prometheus is a plain Prometheus, so anything that speaks to Prometheus works. There are two paths, and neither is privileged over the other.

Your own Grafana#

Add the swarm’s /prometheus URL as a Prometheus data source in an existing Grafana and build or import whatever dashboards you like. The dashboards domyn-swarm bundles are ordinary Grafana JSON, so you can import them directly:

Dashboard	Package path
vLLM and Nginx	`domyn_swarm/data/dashboards/vllm.json`
GPU, nvidia_smi vocabulary	`domyn_swarm/data/dashboards/gpu_nvidia_smi.json`
GPU, DCGM vocabulary	`domyn_swarm/data/dashboards/gpu_dcgm.json`
Ray panels, appended to another dashboard	`domyn_swarm/data/dashboards/ray_panels.json`

This is the right choice if you already run Grafana, want dashboards to outlive a swarm, or want to watch several swarms side by side. Remember the endpoint is only reachable from inside the cluster, and that the swarm’s Prometheus disappears with the job — so for anything long-lived, federate rather than pointing at it directly.

In the terminal#

domyn-swarm monitor my-swarm-name

This resolves the swarm’s Prometheus URL from its state record and hands it to grafatui, a terminal UI for Prometheus that renders Grafana dashboards, along with the bundled vLLM dashboard. It replaces the current process, so you get grafatui’s UI directly.

grafatui is a separate project and is not installed with domyn-swarm. See its installation guide — briefly, brew install fedexist/grafatui/grafatui, its install.sh, or cargo install grafatui.

Without it on PATH the command exits 127 and prints the Prometheus URL, so a missing install tells you exactly where to point Grafana instead.

Other exits worth knowing, because they are configuration problems rather than failures:

Exit	Meaning
`1`	monitoring is not enabled for this swarm — enable it and redeploy
`2`	`--gpu` was passed but no GPU exporter is configured, or `--dashboard` names a file that does not exist
`127`	`grafatui` is not on `PATH`

Note that “enable it and redeploy” is the whole story for exit 1: monitoring is wired into the load-balancer job at submission time, so it cannot be switched on under a swarm that is already up.

grafatui flags#

domyn-swarm monitor my-swarm --range 1h --step 15s
domyn-swarm monitor my-swarm -d ./my-dashboard.json
domyn-swarm monitor my-swarm --prometheus-url http://other-host:9090
domyn-swarm monitor my-swarm --var replicas=8

--range and --step pass straight through to grafatui. --prometheus-url overrides the resolved URL, which is how you point at a swarm’s Prometheus from outside the cluster or through a tunnel.

Dashboard variables#

The bundled dashboard is parameterised, and monitor fills two variables from the swarm config: vllm_job (the Prometheus job name, vllm) and replicas, which drives the Replicas down panel. Override either, or add variables your own dashboard needs, with repeatable --var KEY=VALUE.

Two variables you might expect are deliberately not filled in:

swarm — each swarm runs its own Prometheus scraping only itself, so filtering by swarm would be redundant.
model — vLLM labels metrics with the full resolved model path, not the model string from your config, so an auto-filled value would match nothing. Pass it explicitly with --var if your dashboard uses it.

GPU metrics#

Per-node GPU metrics come from a separate exporter, enabled underneath monitoring:

backend:
  endpoint:
    monitoring:
      enabled: true
      gpu_exporter:
        enabled: true
        kind: nvidia_smi
        image: /shared/images/gpu_exporter_nvidia_smi.sif

Then:

domyn-swarm monitor my-swarm-name --gpu

which loads the bundled dashboard matching the configured kind.

Choosing a kind#

nvidia_smi (the default): A small static binary that shells out to nvidia-smi. Portable, runs unprivileged anywhere nvidia-smi exists, and is the right default.
dcgm: NVIDIA’s Data Center GPU Manager, emitting the standard DCGM_FI_* series. Pinned to the 3.x line because 4.x aborts when run unprivileged. Driver-to-DCGM compatibility varies by site, so this is the one to try when you need the standard metric names and have checked your driver supports it.

Either way the exporter runs unprivileged inside the job’s GPU cgroup, and covers memory, utilisation, power, temperature, and clocks with throttle reasons. Profiling counters (DCGM_FI_PROF_*) need root and are not available.

Images and modes#

Build from the recipes shipped in images/:

sudo singularity build gpu_exporter_nvidia_smi.sif images/gpu_exporter_nvidia_smi.def
sudo singularity build gpu_exporter_dcgm.sif images/gpu_exporter_dcgm.def

The exporter follows monitoring.mode, and the two kinds do not support the same modes. The config rejects the impossible combinations at load time rather than failing on the node:

nvidia_smi with mode: container needs an explicit gpu_exporter.image — there is no default image to fall back to. Omitting it raises nvidia_smi container mode needs an explicit gpu_exporter.image.
dcgm only works with mode: container, because it is launched through singularity exec. Asking for mode: binary raises dcgm exporter is only supported in container mode.
dcgm with mode: container and no image falls back to a public NVIDIA image.

So mode: binary means kind: nvidia_smi, with binary defaulting to nvidia_gpu_exporter on PATH.

Which replica owns which GPU#

An exporter runs once per node, but a node can host several replicas. To attribute a GPU to a replica, each replica records the UUIDs of the GPUs it owns, and the supervisor renders them as a join metric:

dswarm_gpu_owner{uuid="GPU-...", UUID="GPU-...", replica="0"} 1

Both spellings of the label are emitted on purpose: the nvidia_smi exporter labels GPUs uuid and DCGM labels them UUID, so one series joins against either without the dashboard caring which exporter is running.

Prometheus scrapes this from /gpu_ownership on the endpoint, served by the same Nginx that fronts the model.

Ray metrics#

For multi-node replicas, Ray’s own ray_* metrics are scraped from every node. This needs no configuration: ray_metrics.enabled resolves itself to true when monitoring is on and the deployment requires Ray, and to false otherwise.

Set it explicitly to false to opt out, or true to force it on. An explicit value is always respected, and after validation the field is never left unset.

monitoring:
  enabled: true
  ray_metrics:
    enabled: false     # scrape vLLM but not Ray

When Ray metrics are active, domyn-swarm monitor appends a group of Ray panels to the bundled dashboard rather than using a separate one, so a Ray swarm’s dashboard is the vLLM dashboard plus cluster panels. Passing --dashboard suppresses this — your dashboard is used exactly as given.

ray_metrics.port (default 8090) is Ray’s --metrics-export-port. It is fixed rather than ephemeral so that the per-node files Prometheus discovers have stable contents.

What monitoring does not do#

It does not replace health checks. Prometheus tells you a replica is slow; the watchdog decides whether it is dead. See Watchdog and collector.
It does not persist. No metric outlives the load-balancer job unless something external is federating.
It does not alert. No Alertmanager is deployed and no rules are shipped.
It does not need domyn-swarm monitor. That command is a convenience for reading a swarm from a terminal; the metrics are plain Prometheus either way.

Full field reference#

Every field, with types and defaults, is generated from the models: MonitoringConfig, GpuExporterConfig and RayMetricsConfig in Configuration.