Skip to content

WAN: health and multi-WAN failover

Every network in a WAN zone is a WAN. strongholdd watches each one with health probes and sends traffic out of the working WAN with the lowest priority. With one WAN the same machinery only reports its health. High availability across two boxes is a separate feature (its own tab; see design.md, D7–D10).

Config

networks:
  - id: wan
    zone: wan
    untagged: [port1]
    ipv4: dhcp
    wan:
      priority: 1                # lowest working one carries the traffic
      monitor:                   # default: ping 9.9.9.9 and 1.1.1.1
        - ping: 9.9.9.9
        - dns: { server: 1.1.1.1, name: fedoraproject.org }
        - http: https://fedoraproject.org/
      interval: 2s               # default 2s
      down-loss: 50              # % of the last 10 rounds; default 50
      down-latency: 800ms        # average round trip; default off
      up-after: 5                # good rounds in a row; default 5
  - id: wan2
    zone: wan
    vlan: 200                    # a WAN can be a VLAN too
    tagged: [port1]
    ipv4: { address: 203.0.113.2/29 }
    wan: { priority: 2, gateway: 203.0.113.1 }

A round passes if any probe answers. A WAN is down when at least down-loss % of the last 10 rounds failed (after at least 4 rounds), or the average round trip is above down-latency; it is up again after up-after good rounds in a row. A WAN whose interface is missing is down. If every WAN is down, the active one stays.

How it routes

  • The default route is on the WAN, in the main table too. NetworkManager keeps each WAN's routes (its default route and its subnet) in the main table, the default routes with a metric by the WAN's priority (100 for the first, 101 for the next): NetworkManager then says the box is connected, and so PackageKit, Cockpit and anything else that asks it see it online (with the routes only in the WANs' tables, PackageKit 1.4.0 thought the box offline and wouldn't download).
  • A table per WAN. strongholdd copies each WAN's routes into its own routing table, every 5 seconds and when a WAN comes up: 100 for the first WAN in the config, 101 for the next, and so on. It removes only what it copied there (NetworkManager's own routes in a table stay: the profile of an older Stronghold, put back when its change wasn't kept). Rules at priority 1000 send packets marked with the table's number to that table, and at 1001 make them unreachable if the WAN has no route, so a probe never leaves through another WAN. These rules are strongholdd's, not NetworkManager's: NetworkManager removes its own rules when a connection goes down, and in the lab that let a dead WAN's probes leak out of the other WAN, so it looked healthy and traffic flapped back to it every 16 seconds.
  • No link means down at once. A WAN whose interface has no carrier is down in the first round, without waiting for its loss to add up.
  • The inside first. At priority 32000 the main table is looked up without its default route (suppress_prefixlength 0): the inside networks and static routes.
  • Answers from a WAN's address use that WAN. At priority 32100 a rule per address of each WAN sends packets from it to its WAN's table. strongholdd follows the addresses every 5 seconds (DHCP, and the pair's virtual addresses move). Most of the box's answers carry the connection's mark anyway; this is for those routed without it, like WireGuard's (docs/vpn.md).
  • The active WAN is one rule at priority 32200, pointing at the active WAN's table. Switching is adding the new rule and then deleting the old one. The main table's own default route (the kernel's rule, 32766) is only reached when that table has no route. A box of before (rules at 39000, 40000 and 5240) has them removed.
  • Connections keep their WAN. An nftables table of Stronghold's own, inet stronghold_wan (outside firewalld's), marks each connection with the WAN it first leaves or arrives on, and puts that mark on its later packets. Connections to the box or forwarded in through a standby WAN are answered through that WAN, and connections on the backup stay there when the primary comes back. Packets arriving from a WAN are not marked, so they are routed to the networks by the main table.
  • Answers from the box itself follow the WAN their connection came in on: packets from a WAN that the box keeps for itself (the input hook, after NAT) get their connection's mark, and with tcp_fwmark_accept and fwmark_reflect (dist/sysctl) the box's sockets and ICMP answers use that mark. Without it they are first routed by the active WAN's table, and are lost while it has no route: in the lab, an HA box letting go of its WAN couldn't be reached on its other WAN for 10 seconds.
  • On failover, connections of the WAN that went down are deleted from conntrack, so clients' next packets start new connections on the new WAN instead of waiting for timeouts.
  • Probes (ICMP echo, a DNS query, an HTTP HEAD) are sent with the WAN's firewall mark (SO_MARK), so they test that WAN even on standby.

Limits

  • Failover only; no load balancing across WANs yet.
  • IPv6: a WAN's IPv6 routes are in the main table too, and NetworkManager asks for a delegated prefix only from the WAN with the main table's default route (the first, by metric); the WAN manager copies them into the WAN's own table (every 5 s, like the source rules), which its marked connections use. IPv6 looks up the main table whole at 32000, default route included, as before: the prefix comes from that WAN, and its addresses can't leave through another. With several WANs firewalld's IPv6 reverse-path filter is set to loose, so answers coming in on a standby WAN aren't dropped. The prefix comes from one WAN. Probes are IPv4.
  • PPPoE WANs: not yet.

Traffic shaping

A line's modem or the ISP queues far more than it should: under load (a backup, a download) everything else waits behind it, and calls and games lag. With shaping the box sends a little slower than the line, so the queue is the box's, kept short by CAKE (sch_cake), and shared fairly: per inside host first (CAKE looks through the NAT), then per connection, with DSCP classes (diffserv3).

# interfaces.yaml, a WAN network
wan:
  shaping:
    download: 450mbit     # about 90-95% of what the line really does
    upload: 45mbit
    overhead: docsis      # docsis, ethernet, pppoe-ptm, bridged-ptm, conservative, or bytes
  • Upload is CAKE on the WAN itself (its root queue).
  • Download can only be shaped as it arrives: the WAN's incoming traffic goes through an ifb device (shifb<n>) with CAKE on it (ingress: drops count against the sender; wash clears the ISP's DSCP marks).
  • Either direction alone works; without a rate it isn't shaped.
  • A loop puts it back within 10 s when an interface is made again (a pair's mac-move WAN comes up on the active box, a profile is brought up again). Shaping off takes Stronghold's queues away (only CAKE, which is Stronghold's).
  • stronghold shaping (and GET /v1/shaping) shows each direction's rate, packets, drops and queue.

Pick the rates from a speed test with shaping off, then lower them until a test under load shows little extra latency (bufferbloat).