Alerting¶
The concept worth understanding before you trust any of this: a rolling update looks exactly like an outage. In both cases replicas drop below desired. Too little suppression and every deploy pages you; too much and a broken deploy is invisible.
Everything below is that bargain.
The state machine¶
An alert moves through four states:
| State | Meaning |
|---|---|
| Pending | The condition is true, but not for long enough yet |
| Firing | It has held for the rule's for duration. This is when you are told |
| Suppressed | It is true and firing, but something excuses it — see below |
| Resolved | It stopped being true, and stayed stopped for resolve after |
for is dwell, and it is why the seeded rules do not shout during a deploy. A service
with no running replicas has to stay that way for 60 seconds before anyone is told,
because a redeploy passes through zero legitimately.
resolve after is the same idea on the way out: a flapping service that recovers for
two seconds has not recovered.
Why a rolling update doesn't page you and a failed one does¶
A rule can be marked suppressed during update or suppressed during node drain. While Docker reports the service as updating, those alerts are held back.
Suppression is bounded. It expires after swarm.alerts.update_grace_sec (15 minutes
by default), measured from when the update started — not from when the alert fired.
Unbounded suppression is the failure mode of every naive implementation, because a
wedged deploy then stays invisible for exactly as long as it stays wedged.
Two rules are deliberately not suppressed by updates, because suppressing an update alert during an update would be circular:
- Rolling update paused — Docker stopped the rollout because tasks kept dying, and
is waiting for a human.
docker service lsrenders this as a bland2/3that looks like progress. This is the alert the whole feature exists for, and it is always raised as Critical regardless of what the rule says - Rolling update rolled back — Swarm gave up and put the previous version back
Grouping and rate¶
One dead node takes every service on it with it, and forty separate toasts teach an operator to ignore toasts. Simultaneous alerts are combined into one notification.
swarm.alerts.group_by:
severity(default) — one notification per severityseverity_node— also split by the node an alert is about, so "this node died" reads differently from "three unrelated services are unhappy". A service failing across several nodes is not attributed to any one of themnone— one notification per alert
swarm.alerts.max_notifications_per_minute (6 by default) is counted over a real
sixty-second window, per cluster. Anything over the cap is folded into a single summary
rather than dropped silently.
The seeded rules¶
Cleat ships 15 rules enabled, because a monitoring tool that needs configuring before it monitors anything mostly does not get configured. Edit them in Settings → Swarm → Alert rules, and use Reset to defaults to get back here.
| Condition | for |
Severity | Suppressed by |
|---|---|---|---|
| Service has no running replicas | 60s | Critical | update, drain |
| Service degraded (below desired) | 180s | Warning | update, drain |
| Rolling update paused | 0s | Critical | never |
| Rolling update rolled back | 0s | Critical | never |
| Tasks rejected | 0s | Warning | — |
| Restart storm (5 in 300s) | 0s | Warning | update |
| Tasks stuck pending (300s) | 0s | Warning | update |
| Image drift | 600s | Warning | update |
| Node down | 60s | Critical | — |
| Manager unreachable | 60s | Critical | — |
| Node drained | 0s | Info | — |
| Quorum lost | 15s | Critical | never |
| No leader | 15s | Critical | never |
| Capacity committed (90%) | 300s | Warning | — |
| Monitoring unreachable (180s) | 0s | Warning | never |
That last one is the watchdog, and it is the reason the others can be trusted. Without it, silence is ambiguous: a healthy cluster and a dead monitor look identical.