Skip to content

Swarm troubleshooting

Four things can go wrong, and they look different.

1. Cleat cannot reach the cluster

"Monitoring unreachable", or the cluster header showing a stale "updated N ago".

Cleat could not run a command on any candidate manager. Usually the SSH connection: a host that has moved, a key that is no longer authorised, a jump host that is down.

The Events tab shows which candidate was tried and why it failed. If a candidate's connection was deleted, startup logs it as a missing profile and skips it — the cluster runs on the remaining candidates, with fewer than you think.

2. The SSH user cannot run docker

Polls fail immediately with a permission error rather than timing out.

Add the user to the docker group, or set the cluster's Docker command to sudo -n docker. If you used sudo without -n, it is waiting for a password and the poll hangs instead of failing — always include it.

3. The cluster is unhealthy

Quorum lost, no leader, managers unreachable. These are Swarm problems rather than Cleat problems, and they are the ones that are never suppressed and never silenced.

If you see No leader without Quorum lost, the managers can still form a majority but have not settled on one — usually transient during a manager restart, and worth investigating if it persists.

4. Alerts are not arriving

Work down the chain, because each link is independently switchable:

  1. Is swarm.alerts.enabled on? Off means nothing is evaluated at all
  2. Is there a rule for it? Check Settings → Swarm → Alert rules
  3. Is it still in for dwell? A rule with 180 seconds of dwell says nothing for three minutes
  4. Is it suppressed? A rolling update or a drain excuses many rules, for a bounded time
  5. Is it silenced? Silences are per cluster and can be broad
  6. Is the channel on and working? Toasts, OS notifications, sound and webhooks are each off or on independently — use the test buttons in Settings, which report what actually happened

The Alerts tab shows the current state of every alert including the suppressed ones, and the history records what fired even while you were not being told. If something is firing and you did not hear about it, the answer is between steps 5 and 6.

Nothing appears at all

Check swarm.enabled, and that the cluster itself is enabled in the cluster editor. A cluster with no usable candidate managers is logged at startup and then simply never reports anything — which is the quietest possible failure, and why the editor refuses to save one with no candidates.