Alerts

The box tells someone what happens to it: a WAN going down, the pair switching, an update rolled back. Alerts go out by email, to an ntfy topic (phone notifications) or to a webhook; the last 200 are kept on the box, sent or not, for the System tab and stronghold alerts.

Config

# stronghold.yaml
alerts:
  level: warning               # the least severe sent: info, warning (default), critical
  events: [wan, ha, updates]   # the kinds sent; none: all
  email:
    to: [admin@example.com]
    server: smtp.example.com:587   # STARTTLS on 587 and 25, TLS on 465
    username: alerts@example.com   # its password is a secret on the box
    from: gw@example.com           # default: stronghold@<host name>
  ntfy:
    url: https://ntfy.sh/a-topic-nobody-guesses   # an access token is a secret
  webhook:
    url: https://hooks.example.com/stronghold     # a JSON POST per alert

The SMTP password and the ntfy token are secrets (alerts/smtp-password, alerts/ntfy-token), set with the settings (PUT /v1/config/alerts takes smtp-password and ntfy-token) and kept out of git. A login is only sent over TLS.

What is reported

Kind	Severity	When
wan	warning / info	a WAN goes down (its probes or link) or comes back; traffic moves to another WAN
ha	warning	this node took over as active (the other failed or stepped back); the peer doesn't answer over the sync network
ha	critical	both nodes are active (in an active-active pair: both hold the same group); a node is in fault
ha	info	a handover; the peer answers again
updates	critical / warning / info	an update is rolled back (the new image didn't work), fails, or is done
backups	warning / info	the scheduled backup fails; pushing the config to git fails, and works again
config	warning	a change is rolled back (not confirmed within 60 s, or it failed)
disk	warning / critical	/var or /sysroot is 90 % (97 %) full, and has room again
certificates	warning / critical	a certificate of the box's expires within 14 days, or has: those of certificates.md, and an IPsec tunnel's or an OpenVPN server's (vpn.md)
system	info	strongholdd started (the box restarted)
ids		intrusion detection (docs/ids.md)

• Repeats: the same alert (kind, severity, title) within 15 minutes of one that went out isn't sent again; it is counted (repeated).
• In a pair each box sends its own alerts, with its node name in the subject; WAN alerts come from the active node only (the standby's links follow it), and not in the minute after it became active, when it brings up its mac-move WAN.
• Delivery is tried once; a failure is logged and shown with the alert ("failed", with why). "Send a test" in the System tab (or stronghold alerts test, POST /v1/alerts/test) sends one on every channel and says how each went.

Message formats

• Email: subject [WARNING] <site>/<node>: <title>, the text below.
• ntfy: that subject as Title, Priority high for warnings and urgent for critical, the kind as tag, the text as message.
• Webhook: {"time", "host", "site", "node", "kind", "severity", "title", "text", "summary"}.

In the lab

A webhook and SMTP sink on the host got the test alert by all three channels, and then: a change nobody confirmed rolled back (gw1), the peer stopping and answering again (gw2).