Skip to content

Load balancer

Virtual servers: an address of the box (an extra address on one of its networks) that spreads connections over servers inside, checking them and leaving out the ones that fail. It is layer 4 (TCP and UDP): keepalived runs the kernel's IPVS, the same keepalived HA uses, both in Fedora. HTTP-aware balancing (TLS termination, routing by host name) is the reverse proxy's, HAProxy (proxy.md).

Config

# loadbalancer.yaml
virtual-servers:
  - id: web
    network: wan2              # the address is added to this network
    address: 203.0.113.20
    protocol: tcp              # default; or udp
    port: 443
    scheduler: wlc             # default; lc, rr, wrr, sh (by client), mh (consistent hash)
    persistence: 5m            # a client keeps its server; default off
    allow-from: office         # a network, address object or address; default anyone
    check: { type: https, path: /health, status: 200, interval: 5s, timeout: 3s, retries: 2 }
    servers:
      - { address: 10.71.1.10, port: 8443, weight: 2 }
      - { address: 10.71.1.11, port: 8443 }
      - { address: 10.71.1.12, port: 8443, weight: 0 }   # drained: no new connections
  • Health checks: tcp (connect, the default), http/https (a request and the expected status) or none (UDP services default to none). A server that fails retries checks in a row is out until it passes again; servers start out until their first check passes.
  • The address must be in the network's subnet, outside its DHCP pools, and not the box's own. On a single box it is added to the interface; in an HA pair the network needs a virtual address (vrrp), and keepalived moves the address with it (virtual_ipaddress_excluded, so the adverts don't change with it).
  • The servers answer through the box (IPVS NAT mode): they are on one of its networks, with the box as their gateway, and not on the virtual server's own network (their answers would go straight to clients there).
  • Firewall: Stronghold adds the rule that lets clients reach the address and port (from allow-from only, if set). IPVS keeps conntrack entries (net.ipv4.vs.conntrack), so the answers are established traffic.

How it runs

  • strongholdd renders keepalived's virtual_server blocks (internal/lb) into keepalived.conf (internal/ha): with the pair's VRRP config, or on their own on a box without a pair.
  • SELinux lets keepalived's checks connect to a few ports only; the stronghold policy module lets them connect to any (selinux.md).
  • In a pair, keepalived's lvs_sync_daemon sends the balanced connections to the standby over the sync network (UDP 8848), so they survive a failover; conntrackd syncs their conntrack entries too.
  • Packets IPVS forwards to the servers are routed by the main table: multi-WAN's marks are kept off them (internal/wan: the input marking runs after IPVS, and the output hook skips the inside networks), or they would go back out of the WAN they came in on.
  • The UI's Load balancer tab lists the virtual servers with each server's state and connections (GET /v1/lb/status, from /proc/net/ip_vs) and edits them (SetLoadBalancer, with the confirm window).

Lab

web on 10.71.200.20:80 (wan2), round robin over client1 and client2 (Python HTTP servers on 8080), HTTP check every 2 s:

  • requests from the host alternate between the two;
  • stopping client2's server took it out within seconds (all requests to client1), starting it brought it back;
  • switching the pair to gw2 during a request every 0.25 s: 40 of 40 answered.

Found on the way: - keepalived's init_file on the HA track file rewrote it to 1 when the health checker process first started (the first virtual server), which put the active box in FAULT. - The VRRP adverts listed every address, so a new one made the boxes reject each other's adverts until the standby synced. Extra addresses are now left out of the adverts. - HA's Apply stopped the peer API while it ran, so the standby stopped holding back. The API now keeps answering.