Skip to content

High availability: two boxes as one gateway

Design for D7–D10 in design.md. Multi-WAN on one box is separate (wan.md); a pair can use both.

Nodes and identity

A pair shares one config (one git repository). What differs between the two boxes is written per node inside it, and each box knows which node it is from a file outside the repository: /var/lib/stronghold/node (one line, the node id). The HA tab writes it when a box joins a pair.

# ha.yaml
nodes:
  - { id: gw1, priority: 150 }   # higher wins the election
  - { id: gw2, priority: 100 }
sync: sync                       # network for conntrackd, config sync and the peer check
preempt: false                   # a recovered node doesn't take over again by itself

Networks gain an ha block:

networks:
  - id: lan
    ipv4: { address: 10.71.1.1/24 }        # the virtual address clients use
    ha: { addresses: { gw1: 10.71.1.2, gw2: 10.71.1.3 } }
  - id: sync
    ipv4: { address: 10.71.255.1/29 }      # no virtual address: per node only
    ha: { addresses: { gw1: 10.71.255.1, gw2: 10.71.255.2 }, virtual: false }
  - id: wan                                 # a single DHCP address
    ipv4: dhcp
    ha: { mode: mac-move, mac: "02:53:48:00:00:01" }
  - id: wan2                                # a static block
    ipv4: { address: 203.0.113.2/29 }
    ha: { mode: vrrp, addresses: { gw1: 203.0.113.3, gw2: 203.0.113.4 } }

Validation: every node has an address on every network with a virtual address, all inside its subnet and different from it; a MAC-move WAN has a locally administered MAC; the sync network has no virtual address.

Rendering, per node

  • NetworkManager: each network gets the node's own address. Virtual addresses are not in NetworkManager; keepalived adds them on the active node.
  • keepalived: one vrrp_sync_group with a VRRP instance per network that has a virtual address (unicast to the peer's address on that network, so no multicast is needed), plus the VRRP-mode WANs. The group moves together. The virtual address is added as a /32: the node's own address already brings the subnet route. Extra addresses (1:1 NAT, outbound NAT to an address, load balancer) move with it but are left out of the adverts (virtual_ipaddress_excluded), so a new one doesn't make the nodes reject each other's adverts until the standby has the config. The VRID is the network's position in the config.
  • IPv6: see "IPv6 in a pair" below.
  • Who is active: strongholdd watches which virtual addresses this node holds (every 0.5 s); holding them is being active. On a change it runs conntrackd's handover and moves the MAC-move WANs.
  • MAC-move WANs: the WAN connection has cloned-mac-address set to the shared MAC, dhcp-client-id=mac and autoconnect off on both nodes. strongholdd brings it up on the active node and down on the other. Same MAC and DHCP client ID, so the ISP gives back the same lease and the NAT state synced by conntrackd still matches. Nothing else may be on its port.
  • VRRP WANs: connections from the networks are SNATed to the virtual address (in inet stronghold_wan, before firewalld's masquerade), so they survive a failover; the box's own traffic keeps the node's address.
  • conntrackd: FTFW sync over the sync network (UDP 3780). On becoming active: commit the external cache, flush, resync, bulk send; on standing by: shorten timers and ask for a resync (as primary-backup.sh does).
  • Kea: libdhcp_ha.so in hot-standby with its own listener on the sync address (port 8001, 8002 for DHCPv6); the node with the higher priority is primary.
  • Unbound, Knot: run on both; they listen on the virtual addresses with ip-freebind, so the standby is ready. The local zones are shared, see "DNS names from DHCP".
  • Firewall: system rules allow VRRP (protocol 112) from the peer's own address on each network with a virtual address, and the peer API (TCP 7480), conntrackd (UDP 3780), Kea HA (TCP 8001, 8002), the Knots (TCP and UDP 8053) and SSH from the peer on the sync network.

DNS names from DHCP

Each box's Kea registers names with its own Knot, but DNS is answered by the active box, and Kea's serving box is the higher-priority one whatever the role. So the local zones are shared:

  • One Knot is the zones' primary, the other a secondary (internal/dhcpdns/pair.go). The primary sends every change to the secondary (NOTIFY, then a transfer); the secondary forwards the DNS updates it gets (its Kea's, or a record deleted in the DNS tab there, sent with knsupdate) to the primary. They talk over the sync network on port 8053 (Unbound has 53 there), signed with a TSIG key made from the pairing key.
  • The primary is the higher-priority node while the peer answers; when the peer stops answering, the box left becomes the primary (so new names go on being registered). When the peer is back, roles go back: the former primary takes the zones over from scratch (knotc zone-retransfer, its serial may be the newer), and the primary has Kea register its active leases' names again (lease4-resend-ddns, lease6-resend-ddns), so names registered meanwhile on the other box come back.
  • In the lab: gw1 (primary) powered off; gw2 took over the zones and registered a client's new name; gw1 back: both had the name within a minute of gw1's strongholdd starting.

Split brain (D10)

The VRRP adverts on the internal networks are one heartbeat path; the sync link is the other. A node must take over only when it hears its peer on no path:

  • strongholdd asks the peer's strongholdd for its state every second over the sync network (the peer API, GET /v1/state on port 7480). Every VRRP instance tracks a file strongholdd writes (/run/keepalived/stronghold-peer-active, keepalived's track_file with weight 0): while it says 1, this node stays in FAULT instead of becoming MASTER, even though the adverts stopped (say a LAN switch between the nodes failed). A file rather than a track_script, so keepalived doesn't run anything every second. The file is in keepalived's own runtime directory because SELinux lets keepalived read nothing else under /run (and doesn't log the denial). keepalived ignores a track file that is missing when it starts, so it is made at boot (tmpfiles) with 1: a booting node holds back until strongholdd has asked the peer. strongholdd writes it only once it knows the peer's state, so restarting strongholdd never moves the gateway. keepalived and conntrackd start after strongholdd at boot, once its first apply has brought up the interfaces: keepalived never retries a bind to a unicast source address that was missing when it started, and one instance in FAULT holds the whole sync group there.
  • The rule for 1 (ha.HoldBack): the peer is active, and either it has the higher priority, or this node isn't active and doesn't preempt. If both are active (they couldn't hear each other at all), the lower priority gives way as soon as the sync link works. A node also holds back while its config differs from the active peer's, so a preempting node doesn't take over with an old config. Maintenance (the HA tab) holds a node back too.
  • If the peer misses two answers in a row, the file goes to 0 and the node takes over after VRRP's master-down interval (about 3.6 s).
  • Losing only the sync link changes nothing: the adverts still arrive on the LAN. The HA tab shows the sync link as down, and conntrackd and Kea can't sync until it's back.

Verified in the lab (D9), see "Lab" below.

Config sync

Changes are made on the active node. The standby compares the config version in the active peer's state (a hash of the config files) with its own; when they differ and no change is waiting for confirmation on either, it fetches the config (GET /v1/config on the peer API), applies it with its own node's addresses and commits it ("Synced from gw1"), without a confirm window: it was confirmed on the active node.

  • The newer config wins. Each node's state says when its config was made (the commit's date; a synced commit keeps the date it was made on the peer). A standby whose config is newer than the active node's keeps it ("this box's config is newer than the active peer's"), and the active node takes it from the standby instead. So a change made just before a failover (or one that causes it) isn't lost to the new active node's older config. A peer that doesn't say (an older strongholdd, during an upgrade) gets the old rule.
  • A change strongholdd made on its own (at start, with a new version) doesn't hold sync up: the peer's config is applied with this strongholdd anyway, so it is kept.
  • Interfaces take changes in place. A changed profile that is up is reapplied (D-Bus Reapply with preserve-external-ip, which nmcli lacks) rather than brought up again: that would take the link down, and keepalived would fail over; plain nmcli device reapply removes keepalived's virtual addresses, which keepalived takes as a fault. A change NetworkManager can't reapply (a bond's mode) still brings the interface up again.

The peer API is authenticated with the pairing key (below), and reachable only from the peer's sync address (firewall). With a connected git repository (D14), both nodes could pull from it instead.

IPv6 in a pair

A network inside with IPv6 (from-wan-prefix or address) needs the network's virtual address, as for IPv4; a WAN with prefix-delegation must be a mac-move WAN.

  • One client to the ISP. The mac-move WAN takes the pair's MAC, and with it the DHCPv6 identity (dhcp-duid=ll, dhcp-iaid=mac) and the link-local address (addr-gen-mode=eui64): whichever node is active, the ISP sees the same client, gives it the same delegated prefix and routes it to the same next hop.
  • Addresses: the network's ULA ::1 and static address are virtual (keepalived, virtual_ipaddress_excluded); each node has its own ULA address, ::2 for the first node and ::3 for the second. Clients use ::1 as DNS server.
  • Router advertisements: both nodes advertise the network as on a single box (NetworkManager's shared mode), so the interface's profile doesn't change with the role (NetworkManager could only take that by bringing the interface up again). On the standby an nftables table (inet stronghold_ha) drops its advertisements, all but strongholdd's withdrawal (router lifetime 0, no prefixes).
  • Each node is its own router (its link-local address), so clients are told when that changes: a node that steps back withdraws itself (three advertisements with router lifetime 0), and a node that takes over withdraws the peer too, from the peer's link-local address (in the peer's state): a failed node can't, and Linux clients would keep sending part of their connections to it (both routers are their default routes) until its lifetime runs out.
  • A node that takes over keeps a network with a /64 of the delegated prefix quiet until the prefix is back on it (its WAN comes up only now), up to 30 s: until then NetworkManager advertises the prefix as deprecated (left from when the node last lost it), and clients would stop using their addresses in it for new connections. Then it asks its NetworkManager to advertise (a router solicitation, every second for a while: NetworkManager ignores them for 3 s after an advertisement).
  • The order matters on stepping back: the advertisements are quieted before the WAN goes down, which makes NetworkManager deprecate the delegated prefix.

The HA tab

  • Pair: this box and the peer side by side: node id, role (active, standby, fault), maintenance, config version (in sync or not), when this box last heard the peer over sync, and per network who holds the virtual address.
  • The peer's view comes from the peer itself: this box's strongholdd asks it over the sync network (GET /v1/status on the peer API; GET /v1/ha/peer/status), so no login to the peer is needed. If the peer doesn't answer, the tab says so.
  • Open the peer switches Cockpit to the peer (cockpit.jump to its sync address), Cockpit's own multi-host: it asks for the login there. Each box writes the peer's SSH host keys (from its state over sync) to /etc/ssh/ssh_known_hosts2, so Cockpit (OpenSSH) doesn't ask about an unknown host key. The keys come over the direct sync link: trusted on first use, as answering "yes" to ssh.
  • Switch which box is active at will, from either box: "Make gw2 active" on the active box (stronghold ha handover, POST /v1/ha/handover) or "Make this box active" on the standby (stronghold ha takeover, which asks the active box over the peer API, POST /v1/handover). The CLI waits until the gateway has moved. stronghold ha shows the pair. A switch is one-off: once the other box is active, the old one is a normal standby again and takes over if the new one fails. With preempt on, the box with the higher priority is always active, so the switch is refused; change the priorities instead.
  • Maintenance keeps a box out of the pair until it is ended (the peer takes over if the box was active); for working on a box. Also set up or edit the pair, and see recent events.
  • A planned handover loses nothing: a node switching or going into maintenance keeps the virtual addresses until the peer reports that it is ready (out of FAULT, in the election), then steps back; keepalived hands over at once. If the peer never gets ready, it steps back after 10 s anyway.
  • The peer API answers only the peer's sync address (checked by the handler as well as the firewall).
  • The MAC-move WAN counts as up as soon as the new active node has brought it up (multi-WAN uses it at once); its probes take it down again if it doesn't work.

Active-active

Both boxes forward at once, each for the networks and WANs it is home to; either takes the other's over when that one fails or steps back. With two ISPs both are used; with one, both boxes' CPUs still are.

# ha.yaml
nodes:
  - { id: gw1, priority: 150 }
  - { id: gw2, priority: 100 }
sync: sync
active-active: true

# interfaces.yaml: a network's home is the node that holds it while both
# are up (default: the node with the higher priority)
networks:
  - id: lan
    ha: { addresses: { gw1: 10.71.1.2, gw2: 10.71.1.3 } }            # home gw1
  - id: guest
    ha: { addresses: { gw1: 10.71.30.2, gw2: 10.71.30.3 }, home: gw2 }
  - id: wan
    ha: { mode: mac-move, mac: "02:53:48:00:00:01" }                  # home gw1
  - id: wan2
    ha: { mode: vrrp, addresses: { gw1: 203.0.113.3, gw2: 203.0.113.4 }, home: gw2 }

Groups

  • A group per node: the networks (and WANs) whose home it is. Each group is a keepalived sync group of its own (stronghold-gw1, stronghold-gw2): it moves as a whole. The home node has the higher VRRP priority in its group.
  • Holding: a node holds a group when it has its virtual addresses. Normally each holds its own; one node holds both when the other failed, is in maintenance, or handed over.
  • Coming home: a group goes back to its home node once that node is ready (it answers over sync, its config is the same, it isn't in maintenance) and has been for 30 s: the holder steps back from that group, as in a handover, and the home node takes it. Not by VRRP preemption, which the split-brain check would block, and which would move the group before the home node is ready.
  • Split brain is checked per group: a track file per group (/run/keepalived/stronghold-peer-holds-<group>), 1 while the peer answers over sync that it holds the group and should keep it. If both hold a group, the one that isn't its home gives way.
  • The primary is the node that holds the group of the node with the higher priority: normally that node, the other after a failover. What runs once per pair runs on the primary (the tailnet's routes, dynamic DNS for names that follow the active WAN), and config changes are made there (config sync as above).

Forwarding: every connection through one box, or both both ways

A box forwards a connection itself when it holds both the network it comes from and where it goes (a network, or a WAN for the internet). Otherwise it sends it to the peer over the sync network, and the peer forwards it on and sends the replies back over the sync network. So each box sees both directions of every connection it forwards: no asymmetric routing, conntrack's state stays right on both, and NAT happens where the WAN is.

  • To the peer: a routing table of its own (99: a default route via the peer's sync address), for the connections the WAN manager's nftables table marks 99: new connections from a network this box holds to a network the peer holds, and to the internet when this box holds no working WAN (the WAN manager only uses the WANs this box holds).
  • Back to the peer: connections that arrive from the sync network are marked 98; their replies take table 99 too.
  • Each group's own WAN: a group's networks go to the internet through the group's own WAN (its first by priority, while it works), also on a node that holds both groups: a policy route per group (as in routing.md). A connection's NAT address then moves with its group, so a group coming home cuts nothing. (A connection can't be relayed instead: the relaying node would have its NAT too, from conntrackd, and translate it before handing it on.)
  • When the peer goes away table 99 is emptied, so marked connections are routed here as any other: this box holds everything now.
  • The cost: traffic between the groups (a network of one node and a network or WAN of the other) crosses the sync link, so it should be as fast as the WAN; a direct cable is best. Put networks that talk to each other a lot in the same group.

Firewall

The first box filters, the second trusts the sync link (the peer already filtered):

  • Each zone's policy to the sync zone gets, before the user's rules, the rules of that zone to every zone the peer may forward to, limited to that zone's destinations: its networks' subnets, and for a WAN zone everything that isn't one of the box's networks (an ipset of them, negated). Port forwards and 1:1 NAT are accepted where they are translated (firewalld's ct status dnat), on the WAN's box.
  • The sync zone's policies to the other zones accept: what comes over the sync link was filtered by the peer.

What changes with groups

  • conntrackd syncs both ways, as it does now. A node that takes a group over commits the peer's connections from its external cache (connections it has itself, the relayed ones, collide and are left alone, so its own state with its NAT wins) and asks the peer for a resync; it doesn't flush, since it still forwards its own group. Right after the commit it drops the committed connections marked for a WAN whose group the peer keeps: the peer moved them there when it let go of the group's WAN, so their NAT is the peer's address, and here they would leave by this node's own interface of that WAN and get no answer, each packet keeping them alive (found in the lab: a client without internet for 30 to 90 s after its node came back from an update). The next packet starts again on this node's WAN.
  • TCP tracking in a pair (both kinds): no picking up connections in the middle (nf_conntrack_tcp_loose=0), no window checks (nf_conntrack_tcp_be_liberal=1), as conntrackd's documentation advises. A packet that reaches a node taking over before conntrackd has committed the peer's connections would otherwise start a new connection without its NAT, and that one's reset would end the real one; dropped, it is sent again a moment later.
  • A node that starts asks the peer for all its connections (conntrackd -n) and is ready for a group only after: conntrackd sends only changes, and an established connection has none, so a node that started after a connection would never know it.
  • MAC-move WANs come up on the node holding their group.
  • IPv6 router advertisements: each node advertises the networks it holds and is quiet on the others (inet stronghold_ha, per network).
  • Kea stays hot-standby: the primary DHCP server answers on every network, with each network's virtual address as its router.
  • Intrusion detection runs on both, each on the WANs it holds.
  • Updates one box at a time, as now: the node that isn't primary first; its groups move to the primary meanwhile.
  • Handover and maintenance move all of a node's groups to the peer; "take over" takes all of them.

Limits

  • Cross-group traffic costs a hop over the sync link.
  • A connection that left through the other group's WAN (its own was down) is cut when its group goes home.
  • IPv6 isn't relayed yet: IPv6 between the groups, and to a WAN of the other node's, doesn't work while both forward.
  • With WANs in several zones, the policies to the sync zone use the first WAN zone's rules for the internet.

Limits

  • A name removed (lease released or expired) on the box left alone while the primary was away can come back with the primary until that lease's expiry is processed there.
  • Maintenance doesn't survive a restart of strongholdd.
  • The virtual addresses' VRRP is IPv4; IPv6 addresses move with it.
  • IPv6 has no second WAN to fall back on while the mac-move WAN moves: a handover loses about 5 s of IPv6, a failure about 12 s (IPv4: none, and about 4 s, with the VRRP WAN carrying the traffic meanwhile).
  • The pairing key's 30 s limit needs the clocks in sync. chrony steps the clock whenever it is off by more than a second (makestep 1.0 -1 in the image, not only at boot), and when the peer refuses a request for the clocks, strongholdd has chrony measure at once (chronyc burst). A VM paused for a minute came back in sync about three minutes after it resumed (chrony wants a few samples before a big step).

Lab

gw2 is a second VM from the same image, with NICs on the same libvirt networks as gw1: WAN on default (MAC-move test) and stronghold-wan2, LAN on stronghold-lan, SYNC on stronghold-sync.

Results (gw1 and gw2, October 2026):

  • Power off the active box: the standby took over in about 4 s (two missed peer answers, then VRRP's master-down interval). A client pinging every 0.2 s lost 18 pings; its TCP download kept going (conntrackd), and the MAC-move WAN got the same lease back.
  • No adverts, sync up (the standby's LAN and WAN2 links cut): the standby stayed in FAULT; no split brain.
  • Only sync cut: the standby left FAULT for BACKUP but kept hearing the adverts, so nothing moved; back to FAULT when sync returned.
  • Restarting strongholdd on either box moves nothing.
  • Maintenance on the active box handed over to the peer, including the WAN lease and a 1:1 NAT address; port forwards worked through the new active box. With the peer getting ready first, a client pinging every 0.2 s lost nothing (it lost 16 pings before). Switching to gw2 and back from the HA tab: no lost pings from a client, nor to the boxes' own addresses.
  • Found on the way: SELinux keeps keepalived from reading a file under /run/stronghold without logging it, so the track file lives in /run/keepalived; and a page can't log in to another host through Cockpit ("Login failed" even with the host key known), so the peer's view goes through strongholdd and "Open the peer" uses Cockpit's own host switch.

Active-active (gw1 and gw2, 2026-10-18; lan, iot, wifi and the mac-move wan at home on gw1, guest and wan2 on gw2; a guest host is a network namespace with VLAN 30 on client2):

  • Each forwards its own: client1 (lan) to the internet through gw1's wan, the guest host through gw2's wan2 (NATed to its virtual address); client1 to an HTTP server on the guest host relayed gw1 → gw2 over sync and back (the connection ASSURED on both, mark 98 on gw2).
  • No working WAN: gw2's wan2 upstream cut (its link up, so its group stayed): its active-WAN rule pointed at table 99 and the guest host's internet went out through gw1's wan (mark 98 there); back on wan2 when restored.
  • Hard failure: gw2 powered off during a download through wan2: gw1 took guest and wan2 in about 5 s (27 pings at 0.2 s from client1 to the guest host lost: two missed peer answers before gw1 may leave FAULT for that group, then VRRP's master-down interval); the download went on.
  • Coming home: gw2 back, its group returned 30 s after it was ready, the same download going on at full rate, one ping lost. The same from maintenance.
  • Found on the way: a node that started after a connection never heard of it (conntrackd sends changes; now it asks for a resync at start); a packet reaching the new holder before conntrackd's commit started a connection without NAT whose reset ended the real one (now no loose pickup in a pair); a connection that left through the other group's WAN couldn't be relayed back (its NAT went with it; now each group uses its own WAN); and Suricata in prevent mode dropped every connection it hadn't seen start, about 15 s after a handover, in any pair (now it picks them up, ids.md).

Pairing key

The peer API between the two boxes (on the sync network, port 7480: status and the state the split-brain check relies on, the config sync, the VPN keys, maintenance and handover) is authenticated with a key both boxes hold and nobody else:

  • Every request carries an HMAC-SHA256 over its method, path, time, a random nonce and its body; the box refuses a wrong MAC, a time more than 30 seconds off (keep the clocks in sync: chrony) and a nonce it has seen (a replay). Every answer carries one over the request's nonce, its status and its body, and the asker refuses an answer not made with the key: so nobody else on the sync network can ask for the config or the VPN keys, move the active role, or feed the standby a config or a peer state.
  • The key is a secret on each box (/var/lib/stronghold/secrets/ha/pair-key, 32 random bytes), not in the config and not synced. Make it on the first box and set it on the second: the HA tab ("Pairing key"), stronghold pair-key (makes one, shown once) and stronghold pair-key KEY, the API (POST /v1/ha/pair-key), or pair-key in the ha section (Ansible's pair_key).
  • Without a key, or with different keys, the boxes don't talk to each other: the HA status says so (key_problem), and the box can't tell whether the peer is active, so the split-brain check falls back to VRRP alone. A new key on one box stops the pair until the other has it too.
  • The secrets endpoint still seals the VPN keys for a fresh X25519 key of the asker's, so they aren't readable on the wire even with the key.