Functions and containers

Stronghold is a set of functions: each does one job (resolve DNS, hand out DHCP leases, watch traffic) and gets its own section of the config. This note says where each function runs, how it gets its ports and how the same pieces work on a Stronghold box (image mode), on any podman host and, later, on Kubernetes or OpenShift.

Status (19 October 2026): phases 1 to 6 and most of 8 are done (the end of this note): every service function runs in a container of its own, carried by the host image, or, on a box from packages, as the host's RPM or in its container; on a podman host strongholdd runs in a container too ("Podman hosts"). The operator (phase 7) is next.

The rule

A function runs in a container of its own, unless there is a performance or security reason not to. The reasons that exist are written down in the table below; a function moves out of the host only when the table says so. A box installed from packages may run the service functions as the host's own RPMs instead, configured by strongholdd all the same ("Enabling a function", D20).

Containers are worth it for the functions that parse what the network sends (DNS queries, DHCP packets, the traffic Suricata inspects): a bug in one of them is confined to that container (its own files, its own SELinux label, only the capabilities it needs) instead of the whole box.

1. Host functions and service functions

There are two kinds of function, and one control plane.

Host functions set up the kernel: they forward and filter the traffic itself. They run in the host's network namespace, always.

Function	Runs	Why not a container
Interfaces, bonds, bridges, VLANs, Wi-Fi (NetworkManager)	host, native	NetworkManager is the host's; a second one in a container would fight it over the same interfaces.
Firewall zones and policies (firewalld)	host, native	firewalld is the host's firewall; podman and libvirt already plug into it.
NAT (Stronghold's nftables table), routing, sysctls	host, native	Kernel settings of the host's namespace: a container would need NET_ADMIN there, which confines nothing.
Traffic shaping (CAKE, tc)	host, native	The same.
WireGuard tunnels (by NetworkManager)	host, native	Interfaces of the host, made by NetworkManager.
OpenVPN servers	host, native	openvpn-server@ units; the tun interfaces and the kernel's routing of the host; openvpn_t confines it (vpn.md).
IPsec tunnels (libreswan)	host, native	XFRM interfaces and the kernel's IPsec state of the host; ipsec_t confines it already (vpn.md).
Tailscale (tailscaled)	host network; in image mode a container of its own	Its interface is the host's, so it keeps the host's network; but it talks to the internet, and Fedora runs it unconfined: a container confines it ("Enabling a function", SELinux).
HA: keepalived (VRRP), conntrackd, IPVS	host, native	They move addresses and connection state of the host.
Identity: SSSD, realmd (users from FreeIPA / IdM or AD)	host, native	PAM, NSS and sudo for the whole box: Cockpit, SSH and the console use them.

Service functions are programs in user space that answer or watch traffic. Each runs in a container image of its own.

Function	Image	Network	Capabilities
DNS resolver (Unbound)	stronghold-dns-resolver	host	NET_BIND_SERVICE
Authoritative DNS (Knot: local zones, public DNS)	stronghold-dns-auth	host	NET_BIND_SERVICE
DHCPv4 and DHCPv6 (Kea, with its HA hook and DDNS)	stronghold-dhcp	host, or macvlan on the segment	NET_BIND_SERVICE, NET_RAW (DHCPv4 answers clients without an address)
Router advertisements (radvd)	stronghold-ra	host	NET_RAW
IDS/IPS (Suricata)	stronghold-ids	host	NET_RAW (af-packet), NET_ADMIN (nfqueue, prevent mode)
headscale	stronghold-headscale	host, or a bridge with its port published	none
Dynamic DNS (ddclient)	stronghold-dyndns	host	none
ACME certificates (lego, run when one is due)	stronghold-acme	host	NET_BIND_SERVICE (HTTP-01)
Routing daemons (FRR: BGP)	stronghold-routing	host	NET_ADMIN (zebra's routes), NET_RAW, NET_BIND_SERVICE, SYS_ADMIN (FRR asks for it, for VRFs)
Web proxy (Squid: caching, transparent for HTTP)	stronghold-web-proxy	host	none above the user's own (SETUID, SETGID, CHOWN, KILL: Squid runs as squid)
Reverse proxy (HAProxy: sites by name, TLS)	stronghold-reverse-proxy	host	NET_BIND_SERVICE (80, 443)
UPnP and NAT-PMP (miniupnpd: the devices' own port mappings)	stronghold-upnp	host	NET_ADMIN (its rules in the host's nftables, in a table strongholdd makes for it; upnp.md)

The control plane is strongholdd: the HTTP API, the standalone UI, commit-with-confirm and the config's git repository.

• On a Stronghold box (image mode) it runs native: it drives the host functions over D-Bus (NetworkManager, firewalld, systemd) and netlink, which in a container would need nearly everything the host has. Cockpit stays native too: it is the host's admin UI.
• On a podman host and on Kubernetes it runs in a container, stronghold-controller, and drives only the service functions, unless the host functions are turned on for that host (below).

Each service function container runs its program and stronghold-agent <function>: the agent gets the function's section of the config from strongholdd, renders the program's files, reloads it and reports back whether it works, with its health and metrics. A change counts as applied only when every agent it touches says so; a rollback (commit-with-confirm) reverts all of them.

Performance

The traffic through the box never passes through a container: forwarding, NAT, flowtables, IPVS, CAKE and WireGuard are done by the kernel, and the host functions that set them up are native. A container only changes how the service functions are started, so the rules for them are:

• No extra hop. Service functions use the host's network (or, for DHCP on another host, a macvlan on the segment): no veth pair, no NAT and no user-space port proxy (rootlessport, pasta) between a client and the program. Podman's published ports are not used for DNS or DHCP.
• No conntrack for DNS. The firewall marks DNS to and from the box's own resolvers notrack, so a busy resolver doesn't fill the connection table with one entry per query.
• CPUs for the IDS. Suricata can be given CPUs (ids.cpus): its unit kept to them (AllowedCPUs, a drop-in that the quadlet's generated unit takes too) and its af-packet or nfqueue workers pinned to them (ids.md, "CPUs"). The other functions get CPU and memory limits so none can starve the rest.
• Measured, not assumed. In the lab, before a function moves into a container: queries per second with dnsperf, leases per second with Kea's perfdhcp, and Suricata's drops at a given rate, native against container (both tools are in Fedora). A function whose numbers drop stays native until that is fixed.

Kernel modules (WireGuard, IPVS, sch_cake, nfnetlink_queue) are loaded by the host image (modules-load.d): a container can't load them.

2. Ports: by address, not by port

Clients find a DNS server on port 53 and a DHCP server on port 67; that can't be changed for them. So:

• Every service function listens on address:port, and the port is the standard one unless the config says otherwise. Another port is allowed (for a lab, for a resolver only other services use, or inside a cluster), never required.
• Two instances of the same kind share a host by address. The resolver listens on the LAN addresses, Knot's public view on the WAN address, both on 53. The config validation refuses two functions on the same address and port.
• No more privilege for the port. Binding to 53 needs only NET_BIND_SERVICE, nothing else, in its container.
• DHCP needs the segment. Kea has to be on the link its clients are on (host network or a macvlan), or be reached through a DHCP relay (Kea serves relayed requests), for example when it runs in a cluster.
• On Kubernetes the Service has port 53 and the pod may listen on another one (targetPort), so the pod runs under OpenShift's restricted SCC without NET_BIND_SERVICE.

3. Image mode

The host image carries no function (D23; D18 before it carried them all): it is Fedora's minimal bootc base, what a gateway's host needs (all Fedora's: NetworkManager, firewalld, keepalived, podman, tailscale, …) and dnf install stronghold, and nothing else. A function's container is installed when the function is turned on, as on a box from packages ("Installing"): its quadlet from the stronghold package, its image pulled at the version that quadlet names, or loaded from the box's cache.

• Updated apart. The host image has its own version, the Fedora release and its build (44.20261020.1342); it is rebuilt when Fedora updates (scripts/host-image.sh), with the same Stronghold, and boxes follow its stream (stronghold:44) with bootc upgrade, in the update window, rolled back by greenboot as before (updates.md). Stronghold's own updates are new packages in the next host image, whose quadlets name the functions' next images: after the upgrade strongholdd puts them in place and the containers pull them. A host image rolled back takes strongholdd back, and its quadlets with it at the next start; the functions' images are kept until pruned, so going back needs no network. Each agent reports its version and strongholdd flags one of another release line (design.md, "Versions").
• The disconnected ISO installs the offline flavour of the image (bootc/Containerfile.offline): the same image with the functions' images as a cache in /usr/lib/stronghold/images (stronghold-<function>_<version>.tar), which the package's quadlets load from before they would pull (stronghold-cached-image), and DHCP's and the resolver's quadlets installed, so first boot runs them with no network. Other functions are installed from the cache when turned on.
• A box moved from an image that carried the functions (or given a config that uses one it hasn't): strongholdd installs what the config uses, and applies it again once it is.

What follows is how it was until D23, kept for the reasons it records:

• Quadlets in the image. Each service function is a quadlet in /usr/share/containers/systemd/ (stronghold-dns-resolver.container, …), so systemd runs it like any other service.
• The images in the image. Each function's image is in the host image as an OCI archive, /usr/lib/stronghold/images/<function>.tar, with its image ID beside it. Before the function's container starts, a drop-in of its quadlet (load-function-image) makes localhost/stronghold-<function>:<version> that image: the same ID, it does nothing; another one (an update, a rollback), it loads this one from /usr and removes the one it replaces. So bootc upgrade and bootc rollback move the host and every function to the same version: no function is ever newer or older than the strongholdd that configures it. The update window, health check and rollback (updates.md) keep working as they are.
• Nothing pulled at run time. The quadlets have Pull=never; the box never fetches containers on its own, and needs no registry.
• Offline installs. The ISO and disk images carry the function images in /usr like everything else, so an install needs no network.
• Why not bootc's bound images. Logically bound images (/usr/lib/bootc/bound-images.d) are pulled from a registry when bootc stages an update: a registry for every box, and no install without the network. An image store of their own in /usr (/usr/lib/containers/storage, physically bound) can't be written in the build (podman's overlay store can't be made on the build's own overlay), and its whiteouts would be device files in the deployment, which ostree doesn't keep. The archive is a plain file; the cost is one copy of each function's image in /var (about 200 MB for the resolver, for now) and a load of a few seconds after an update.
• The bootc rules. Nothing writes to /usr; a function's config is in /etc/stronghold/, its state in /var/lib/stronghold/<function>/ (leases, zones, rules), mounted into its container. Containers run as container_t under SELinux, enforcing; bootc container lint passes.
• The same quadlets on any podman host. Fedora, Fedora CoreOS or RHEL run the same quadlet files from /etc/containers/systemd/, with the images loaded or pulled by the admin (the drop-in that loads them is the host image's only). strongholdd runs as stronghold-controller there, and the host functions are off ("Podman hosts", below).

Podman hosts (phase 6)

A host that isn't a Stronghold box (Fedora, Fedora CoreOS, RHEL with podman) runs the service functions for its networks, and strongholdd itself, in containers; its network, firewall and units stay its own.

• The controller is strongholdd in a container of its own, stronghold-controller (stronghold-images/controller: Fedora's stronghold and stronghold-ui on fedora-minimal), run as strongholdd --controller by its quadlet beside the functions' in /etc/containers/systemd. Host network (the HTTPS API on the host's addresses) and the host's name; NET_BIND_SERVICE, CHOWN and FOWNER only, no new privileges; ready when its API listens (Notify=true).
• It drives only the service functions, each in its container, through its agent's socket (/run/stronghold-functions, mounted as it is: each function's directory is labelled for sharing by its own container). It never asks the host's systemd: the containers are systemd's, as their quadlets say; a function off, or on the other side (Kea while DHCP is elsewhere), is stopped by its agent, which keeps it so across restarts, and started again by the applier that configures it. A function whose quadlet isn't installed can't run: a change that uses it fails and is rolled back (function.Missing); GET /v1/functions says where: none.
• Turning a function on installs its containers, as on a box from packages (D21), never all at once; the host does it, since the controller never asks its systemd. The role sets the functions' quadlets aside in /etc/stronghold/quadlets (the image from the functions' registry, pulled by the first start, or loaded with the install: stronghold_images), and installs stronghold-function-install.path. install (the UI's Here) leaves a request per container in /run/stronghold-functions/install/requests (install.AskHost); the host's stronghold-function-install puts that quadlet in place, by name and only from there, starts it, and answers in results (.done, or .failed with why). The controller then has its agent, and the enable that follows configures it. Those directories are container_file_t (the controller's quadlet labels them), the controller's to write.
• What it doesn't apply: the firewall, the interfaces, routes, WireGuard, identity (SSSD), HA, the WANs, updates, Tailscale, traffic shaping. The config still describes the host's ports and networks (by MAC or PCI address, as on a box): the functions take their interfaces and addresses from it, through sysfs, which a container on the host's network sees. The host's nftables aren't its own either: no rules for radvd's DHCPv6 networks, and the IDS in detect mode only (prevent needs the nfqueue rules; it is refused). Certificates: ACME by DNS-01 (or HTTP-01 with port 80 opened by the host's admin), OpenBao or Vault; not FreeIPA (certmonger is the host's), and none for Cockpit. Password logins to the standalone UI need the host's PAM: tokens and OIDC only.
• Its paths: its state (the config's repository, history, secrets) in the host's /var/lib/stronghold/controller, /var/lib/stronghold inside; its API's socket the host's /run/stronghold/api.sock, so the host's stronghold CLI, Cockpit's page and Ansible find it where they look on a box; a config given with the install in the host's /etc/stronghold/config (first boot takes only that: the host's ports aren't its to set up); the installed quadlets (/etc/containers/systemd, read only) say which functions it has. Suricata's rules and alerts are mounted read only: the IDS quadlet labels them for sharing (:z).
• GET /v1/status says install: controller; the UI then leaves out the load balancer, HA and the host areas of Features, and says who updates the host.
• Installing: stronghold.gateway.install with stronghold_controller (a directory with the controller's image and quadlet: scripts/container.sh's out/controller), and the functions' quadlets (stronghold_quadlets), set aside; no Stronghold package on the host. In the workspace, scripts/setup-podman-host.sh. Updating is the host admin's: new images loaded, the containers restarted.
• Its image has stronghold-daemon (strongholdd and the CLI) and the UI, not the stronghold package: NetworkManager and firewalld, which that package requires for a gateway, stay out.
• Not yet: a host given to Stronghold (host_functions: true, the controller with the host's D-Bus and NET_ADMIN, applying the host functions too).

Every function image is built from Fedora RPMs (Fedora's container builds require that), so the RPM packages of strongholdd and the UI come first.

4. Code

strongholdd applies each part through core.Applier. A service function is split in two:

• Render: its config section in, files out, nothing else. Testable without a box, the same wherever the program runs (dhcpdns.Render, headscale.Render, ids.Render).
• Runtime (internal/function): puts the files in place, runs the program's own commands (its config check, knotc, suricata-update, the headscale CLI) and starts, restarts, reloads, pauses or stops it. function.Host is the runtime on a Stronghold box: the files on the box, the program's systemd unit (what strongholdd always did). function.Fake is one in memory, for tests. The container runtime is next: it sends the files and commands to stronghold-agent in the function's container.

What belongs to the host stays outside the runtime even for a service function: nftables rules (radvd's advertisements, Suricata's nfqueue), the systemd drop-in that says how Suricata reads the traffic, and the interfaces' addresses.

Done (phase 2): DHCP and DNS (Kea, Knot, Unbound, radvd), Suricata and headscale apply through the runtime, and so do the API's tools for a lease or a record: knotc runs where Knot runs (Exec), the lease files are read there (Read), and Kea's control sockets are shared from its containers (KeaRunDir).

The agent (phase 3)

stronghold-agent (internal/agent, cmd/stronghold-agent, the stronghold-agent RPM) runs in each function's container and runs its program in the foreground:

• Its API is HTTP with JSON on a Unix socket the container shares with the host, /run/stronghold-functions/<function>/agent.sock (root only): write files, read and remove them, run a command, restart, reload, pause or stop the program, its status. agent.Client is the function.Runtime that uses it; function.Mux gives each program its runtime. The socket isn't under /run/stronghold: that is strongholdd's RuntimeDirectory, which systemd empties when strongholdd stops.
• What it may touch is given on its command line, in the image: directories (--path) and commands (--command, NAME:USER to run one as the program's user). Anything else is refused.
• It remembers the files it wrote and whether the program is wanted, in a volume (/var/lib/stronghold/functions/<function>): a new container (an update, a reboot) puts them back and starts the program before strongholdd is there. A program that ends on its own is started again, waiting longer each time.
• strongholdd picks the container for a function when its quadlet is installed (/usr/share/containers/systemd: the image decides; /etc/containers/systemd: an admin did), and then keeps the host's own unit for that program off. Commands of the program that other parts run (unbound-control for the blocklists, the overview and the troubleshooting tools) go to where it runs; the service and log tools show the container's unit.
• A function whose container isn't running doesn't stop the rest: its runtime answers ErrUnavailable (after waiting 30 s for an agent that is starting), that program keeps what it had, the others are applied, and the change reports it.

The DNS resolver (stronghold-dns-resolver, stronghold-images' functions/dns-resolver): Fedora's unbound and stronghold-agent on fedora-minimal; host network; the capabilities NET_BIND_SERVICE, SETUID, SETGID, CHOWN and KILL only (fewer than podman's default), no new privileges. In the container Unbound logs to stderr (the unit's journal) and binds its addresses with ip-freebind instead of Fedora's ip-transparent, which needs NET_ADMIN. unbound-control runs as unbound: Fedora's control socket is unbound's, and the container's root has no DAC_OVERRIDE. The quadlet pulls its image when the box hasn't it (Pull=missing, the box's cache first) and isn't ordered before strongholdd (see "Installing": a pull at boot needs the network strongholdd brings up).

Router advertisements (stronghold-ra, radvd): host network, NET_RAW, SETUID, SETGID and KILL; radvd drops to its user, its privileged helper keeps those. Its pid file is /run/radvd.pid (root can't write in /run/radvd, the radvd user's, without DAC_OVERRIDE). It can't set the interface's own hop limit (/proc/sys is read-only in a container): the kernel's default, 64, is what it advertises anyway.

Authoritative DNS (stronghold-dns-auth, Knot): one image, two containers, stronghold-dns-auth (the local zones) and stronghold-dns-public (the public view); the image's entrypoint is the agent and each quadlet's Exec= says what it runs and may touch. Host network; NET_BIND_SERVICE, SETUID, SETGID, CHOWN, KILL: knotd binds 53 and drops to knot itself (its config's user: knot:knot), with no capabilities left. Its journal (the names DHCP registered) is in the container's state volume, not the host's /var/lib/knot: the container's knot user isn't the host's (another uid), and the agent makes /run/knot and /var/lib/knot the container's knot's before it starts. A box that moves its Knot into the container keeps the static records (strongholdd writes them again) and loses the names DHCP had registered until the clients renew. knotc runs as knot (its control socket is knot's), for strongholdd and for the API's record tools, which go through the runtime too. Knot logs to stderr now, on the box and in the container: the unit's journal either way.

DHCP (stronghold-dhcp, Kea with its hooks): one image, three containers, stronghold-dhcp4, stronghold-dhcp6 and stronghold-dhcp-ddns. Kea doesn't drop privileges itself: the agent runs it as kea with the ambient capabilities Fedora's units give it (--user kea --ambient NET_BIND_SERVICE,NET_RAW for DHCPv4, NET_BIND_SERVICE for the others), and its config checks as kea too. Its leases are in the container's state volume; strongholdd reads them through the agent, which reads a file it may not open as root as the program's user instead (no more privilege for the agent). Its control sockets (deleting a lease, the pair's DNS updates again) are in /run/stronghold-functions/kea on the host, /run/kea inside, 0750 (Kea refuses looser): not the host's /run/kea, which Fedora's kea makes the host's kea user's at every boot (tmpfiles), a user the containers don't have. strongholdd's dhcpdns.KeaRunDir is that directory when Kea runs in containers.

headscale (stronghold-headscale): runs as headscale with NET_BIND_SERVICE (--user, --ambient), as Fedora's unit does. Its state is the tailnet (the server's key, the nodes), so the container uses the host's own /var/lib/headscale: moving headscale into its container, or back, keeps every node. The host's tmpfiles make that directory the host's headscale's at every boot; the agent makes it the container's before headscale starts, with DAC_READ_SEARCH (the agent only: headscale keeps NET_BIND_SERVICE alone). Its config directory is root's in the image (Fedora's package makes it headscale's; strongholdd writes it).

Intrusion detection (stronghold-ids, Suricata and suricata-update): Suricata runs as suricata from the start, keeping NET_RAW (af-packet) and NET_ADMIN (the nfqueue): never root (its own privilege drop, libcap-ng's, fails in a container with only these capabilities). Whether it reads the traffic with af-packet (detect) or the nfqueue (prevent) is a command-line choice: strongholdd writes it to /etc/suricata/stronghold-capture, which the agent reads at each start (--args-file); on the box the unit's drop-in says it. The nftables rules that send the WANs' traffic to the queue (fail-open) stay the host's. The rules (suricata-update, as suricata) and the alerts (eve.json) are in the function's directories, /var/lib/stronghold/functions/ids/lib and …/log, root's with suricata's group (2770); strongholdd reads them there (ids.HostRulesFile, ids.HostEveFile). Suricata on CPUs of its own (ids.cpus, AllowedCPUs) is done and measured on four CPUs (ids.md): leave some to the kernel.

The web proxy (stronghold-web-proxy, Squid): one process (squid -N): Squid's master puts /dev/null in place of its output, -N keeps it, and Squid, as squid, opens /dev/stdout and /dev/stderr again for its logs, which a pipe of root's refuses it, so the agent gives it a pipe that is squid's and copies it on (--output-user squid). No pid file (root without DAC_OVERRIDE can't write squid's directory; nothing needs it: the agent signals Squid itself). The cache is a volume, made squid's and its directories made (squid -z) before each start; where it is on the box (cache.path) is a drop-in of the quadlet that strongholdd writes, restating the quadlet's volumes (an empty Volume= clears them) with the cache's at the path (proxy.md, "Where the cache is"). On the host Squid logs as Fedora's package has it, to /var/log/squid.

The reverse proxy (stronghold-reverse-proxy, HAProxy): master-worker (-W -db); the master, root with NET_BIND_SERVICE, binds the ports and reads the certificates, the workers run as haproxy; --reload-signal USR2 starts new workers on the new config, the old ones finishing their connections. It logs to its output (log stdout).

Routing (stronghold-routing, FRR): the agent runs watchfrr, which starts zebra, mgmtd, staticd and bgpd and loads frr.conf once they are up; strongholdd applies a changed frr.conf with frr-reload.py, so the sessions that didn't change stay up. The daemons outlive watchfrr, so the agent stops them with FRR's own frrinit.sh stop (its --stop), and zebra takes its routes away. They fork into the background, so the container has podman's init (RunInit=true, catatonit) as PID 1 to reap them: with the agent as PID 1 a stopped daemon stayed a zombie, and frrinit.sh stop waited two minutes for it. FRR's daemons won't start without SYS_ADMIN (network namespaces, for VRFs), so the container has it: more than the other functions, still less than on the host (frr.service runs them as root with every capability) and confined by SELinux. The agent and watchfrr run as root without DAC_OVERRIDE: FRR's directories are root's with frr's group, and root is in frr and frrvty (the daemons' vty sockets). Routing as such (the kernel's tables, rules, NetworkManager's routes) stays the host's: zebra writes to it over netlink.

The agent runs its prestart steps before every start of the program, also when it starts it again after it ended on its own. That covers programs that need the WAN when they start: headscale fetches its DERP map at start, and at boot, before the WAN is up, it ends once and the agent starts it again.

Where each program runs and its state: stronghold functions, the API's GET /v1/functions, and the System tab's "Service functions".

The host's own units: each function's quadlet Conflicts= the unit it replaces (kea-dhcp4.service, …), and the host image masks those, so they never start, even where a box had them enabled before it ran its functions in containers (on an update, bootc's merge of /etc brings the masks). strongholdd keeps them off too.

Images and containers: one image per program family, one container (quadlet) per program, since the agent runs one program: Knot's image will run twice (local zones and public DNS), Kea's three times (DHCPv4, DHCPv6, DDNS).

Measured on stronghold-iso (2 vCPUs; lab.md): about 173,000 queries per second answered from the cache either way, the same latency (under 1 ms on average); the container costs nothing measurable.

cmd/stronghold-agent is the agent, from the same Go module as strongholdd: the config model, the renderers and the API contract stay versioned together.

5. Enabling a function

Each service function is off, elsewhere or here (D20). The host functions are always the box's own: they aren't enabled, they are configured.

Where

• Off: nothing runs and nothing points anywhere. Whoever started its programs, strongholdd stops them: the host's units disabled and stopped, the containers masked (unmasked when the function is here again and runs there).
• Elsewhere: a server the box doesn't run, given by address. What that means depends on the function:

Function	Elsewhere
DNS resolver	Its addresses go to clients: its IPv4 ones by DHCP, its IPv6 ones by DHCPv6 and the router advertisements (radvd's, NetworkManager's); Unbound doesn't run. The resolver's settings in dns.yaml (forwarders, blocklists, views) are kept, unused, for when it is here again. The local zones (Knot) stay on the box, but clients only find them if the resolver elsewhere forwards their domains to it. Off: clients get no DNS servers from the box.
Authoritative DNS (local zones)	A server that takes dynamic updates (FreeIPA / IdM, AD, another Knot), signed with the box's DDNS key (TSIG stronghold-ddns, HMAC-SHA256; on the box in /var/lib/stronghold/secrets/ddns-tsig), which its admin adds there: stronghold functions key (--bind, --knot, --nsupdate), the local zones' card in Features, or GET /v1/functions/dns-auth/key (admin) give it with its config for BIND and Knot. zones are the zones on it that the leases' names go into (default: the networks' DHCP domains); reverse zones only when listed, since the server may not have them. Kea's DDNS sends the names there, the box's resolver asks it for those zones (stub zones), Knot doesn't run, and dns.yaml holds no zones (they are on the server). When the place changes (here, elsewhere, other servers or zones), Kea sends every active lease's name again. Records are changed on that server, not from the box.
DHCP	A server on the segment (servers only: the box runs no DHCP there), or one reached through the box's relay: networks are the inside networks it relays, their requests forwarded from the box's address there, which tells the server the subnet. DHCPv4 goes to the IPv4 servers, from a network's static IPv4 address; DHCPv6 to the IPv6 ones, from a network whose ra is slaac+dhcpv6 or dhcpv6, from its static address (else its ULA ::1: one address, which names the subnet to the server), and radvd advertises it with the managed and other flags as when Kea serves it. The firewall lets those networks reach the relay and the servers answer it (UDP 67 and 547, from their addresses only). Kea doesn't run; networks with DHCP of their own, or with DHCPv6 that isn't relayed to an IPv6 server, are refused.
ACME certificates	Already: ACME, OpenBao or Vault PKI, FreeIPA (certificates.md).
headscale	Already: Tailscale's, or a headscale elsewhere (vpn.md).
Router advertisements, IDS, BGP, dynamic DNS	No elsewhere: off or here.

• The DHCP relay is a program of the DHCP function that runs while it is elsewhere: dnsmasq with relaying only (dhcp-relay=, port=0: no DNS, no ranges of its own) from /etc/stronghold/dhcp-relay.conf, on the host as stronghold-dnsmasq-relay.service (in the stronghold package; dnsmasq is suggested), in its container as stronghold-dhcp-relay. Its runtime follows functions.prefer (a function elsewhere has no runtime of its own), and the box offers it apart from Kea's: a box without dnsmasq still runs DHCP here. Kea serves relayed requests but doesn't relay. dnsmasq relays DHCPv6 too (dhcp-relay= with the network's IPv6 address and the server's). Fedora still ships ISC's dhcp-relay, but ISC ended it in 2022 and Fedora may retire it; systemd-networkd's RelayTarget= is DHCPv4 only and needs networkd to manage the interface, which is NetworkManager's.
• Here: Stronghold runs it. Enabling it writes its section of the config with defaults from the box's networks (pools from a network's prefix, the box's addresses), through commit-with-confirm like any change.

How it runs here

Where a function runs is the route's, and where there is a choice, the box's:

Route	Here is	Who installs the program	prefer: auto
Image mode (bootc)	its container	the host image carries it	container
Package mode (dnf install stronghold)	the host's RPM and unit, or its container	strongholdd when it is turned on: its packages (PackageKit) or its quadlet and image (the functions' registry)	host
Podman host (stronghold-controller)	its container	images pulled or loaded by the admin	container
OpenShift, Kubernetes	its pod	the operator	container

# stronghold.yaml
functions:
  prefer: auto          # auto | container | host
  dns-resolver:
    where: here         # off | elsewhere | here
    runtime: host       # container | host: overrides prefer
  dns-auth:
    where: elsewhere
    server: 10.71.1.5

The functions by name: dns-resolver, dns-auth (Knot, and its public view), dhcp (Kea's DHCPv4, DHCPv6 and DDNS), ra, ids, headscale, dyndns, acme, routing, web-proxy (Squid), reverse-proxy (HAProxy; proxy.md).

• auto is the route's default above, and keeps what strongholdd does now: an installed quadlet wins, else the host's unit.
• prefer is a preference: a function that can't run that way runs the other way. A function's runtime is not: a change that asks for one the box can't offer is refused when it is applied, and rolled back (host in image mode, where /usr is read-only and the image doesn't carry the RPMs; host without the program's package; container without its quadlet). Nothing switches unless every function can run as asked. GET /v1/functions says, per program, where it runs, which runtimes this box offers and why not the host, so the UI offers only what works.
• Switching stops one and starts the other: to the host, the container's unit is masked (a quadlet's unit can't be disabled) and the host's unit started by its applier; to the container, it is unmasked and started, and the host's unit disabled. The program's state stays where it was (below).
• A quadlet added later counts: strongholdd looks for a function's quadlet whenever it asks how the box can run it, not only when it starts, so a package-mode box that is given a container needs no restart. A container masked because the config ran its program on the host stays off when strongholdd starts.
• Package mode starts firewalld: Fedora's presets may leave it off and the package can't enable it; strongholdd's firewall applier enables and starts it.
• The paths a program shares with its container keep the right labels: on every apply strongholdd checks them (the resolver's blocklist directory, the only one) against where the program runs, whoever changed them (a policy package's relabel, a restorecon): not container_file_t under a running container, it restarts the container, which podman relabels; container_file_t on the host, it runs restorecon -F (a customizable type, which plain restorecon leaves).
• Done: prefer and runtime (strongholdd's runtimes, first of its appliers; function.Pick), switching both ways, tested on the package-mode VM (lab.md, "Package mode"); where and servers, with the resolver elsewhere (ClientDNS4/ClientDNS6 in the config: what clients are told), DHCP elsewhere with the relay (DHCPRelay, DHCPRelay6), the local zones elsewhere (LeaseZone, DNSAuthServers), the DDNS key in the CLI, the UI and the API, and off stopping a function's programs (off in strongholdd's runtimes), tried on the Fedora VM (lab.md).
• strongholdd alone renders the program's files (function.Host or the agent), whichever route installed the program. Nothing else writes unbound.conf or frr.conf: two writers would fight, and commit-with-confirm couldn't roll back what it didn't write.

Installing

A box from packages has only Stronghold: the stronghold package doesn't pull in the functions' programs (Kea, Unbound, Knot, radvd, …), and nothing installs them until a function is turned on. The disconnected ISO is the exception: its image has the functions' images as a cache, and DHCP and the resolver installed (D23). In image mode it is the same: the host image has no function in it.

• Turning a function on installs it. Choosing Here in the UI's Features (or the function module) installs the function first when the box can't run it, then enables it. Underneath, two steps, the UI's and the CLI's alike:
• Install (POST /v1/functions/{name}/install, stronghold functions install NAME [--elsewhere], an optional runtime), in the background; GET /v1/functions shows the progress and how it ended (install), and then that the box can run it. Refused in image mode.
  • On the host (functions.prefer host or auto, or its runtime): the function's packages (its programs', and Kea's hooks for DHCP; --elsewhere: the DHCP relay's dnsmasq) by PackageKit: strongholdd runs pkcon, which asks packagekitd over D-Bus, as Cockpit installs its pages; it doesn't run dnf itself. The package recommends PackageKit and polkit (which PackageKit asks, and needs only the library of).
  • In its containers (functions.prefer: container, or its runtime): podman by PackageKit when it isn't there, then each of the function's quadlets from /usr/share/stronghold/quadlets (the stronghold package's) to /etc/containers/systemd, and each container started once, which pulls its image. Those quadlets are the image's (stronghold's dist/quadlets) with the image from the functions' registry (registry.fedoraproject.org/stronghold-<function>, make REGISTRY= changes it), of the exact version the quadlets name (design.md, "Versions"), and Pull=missing; the image's own have localhost and never pull. Until the images are published, the lab mirrors that registry (lab.md, "The lab's repository and registry").
• Enable (POST /v1/functions/{name}/enable, stronghold functions enable NAME [here|elsewhere|off] [--servers …] [--networks …] [--zones …] [--dry-run]): one change through commit-with-confirm (config.EnableFunction). Here fills in defaults: a DHCP pool on each inside network with an address and no DHCP (.100-.199 of a /24), a recursive resolver. Elsewhere and off remove what the function can't keep: the networks' DHCP (and their DHCPv6: ra slaac), the local zones, the function's own section (IDS, headscale, dynamic DNS, BGP, ACME certificates). ?dry_run=true shows that diff first: the UI's "asking first". The resolver's settings stay, unused.
• From Ansible: the collection's function module (name, where, servers, networks, zones, runtime) installs the packages through the box's API (PackageKit, as the UI does, so it works over HTTPS too) when the box can't run the function yet, then enables it and pins its runtime, each a change it confirms. A function already where it is asked to be isn't enabled again, which would put pools back on networks left without DHCP. The box's config and history are the same whichever way it was done. The role stronghold.gateway.install puts Stronghold itself on a Fedora host (from its repositories, or one given: stronghold_repo), no function: its first config uses none, and stronghold_functions_prefer says where they will run.
• A config that uses a function the box hasn't (DHCP on a network, the resolver's settings, with neither its packages nor its container) is refused, saying so; first boot on a box from packages leaves DHCP and the resolver out of the config it makes when they aren't there.
• A container installed after boot (a box from packages given its quadlet) is started by strongholdd when it starts, as boot would have, unless it is masked: its agent must answer before strongholdd can configure it.
• A container stopped by hand (or one that failed) is started again when strongholdd needs its agent, unless it is masked: the config uses the function, and while it was down every change failed on it and was rolled back (found in the lab). To keep a function off, turn it off, or mask its container.
• Nothing waits on a pull. The containers aren't ordered before strongholdd, and strongholdd answers (to systemd, greenboot) before its first apply, which then doesn't wait for them: a container whose new image is pulled after an update needs the network that strongholdd brings up (a standby node has no route of its own until then), and ordered before it, it held strongholdd back past its start timeout until greenboot gave up (found in the lab). The containers it finds starting, or restarts on new images, get the config again once their agents answer.
• Upgrading keeps them in step. The package's quadlets name the images that go with it, by their exact version, so a function is never newer or older than strongholdd, as in image mode. The quadlets strongholdd installed (headed by a comment that says so) are kept as the package has them: after an upgrade (dnf restarts strongholdd), those that differ are replaced and their running containers restarted, pulling the new version's images. A quadlet without that comment is an admin's and left as it is.
• In image mode the same, containers only (the image has no packages to add): a function's container is installed when it is turned on (D23).

First boot

A new box from the image still has DHCP and DNS on its LAN without a click (a box from packages has them once they are turned on). First boot already looks for a DHCP server on the LAN (first-boot.md): none, and DHCP and the DNS resolver are here; one, and DHCP is elsewhere, pointing at it, with "run it here instead" offered.

Changing the runtime

Moving a running function between its RPM and its container moves its state too (Kea's leases, Knot's zones and journal, headscale's database, Suricata's rules): stop one, copy the state, start the other, through commit-with-confirm so a failed move goes back to what ran. Not at first: the runtime is chosen when a function is enabled, and changing it is disable, then enable.

SELinux

Both runtimes are confined (D3): the containers as container_t, the host's programs in their own Fedora domains, and strongholdd in stronghold_t writing either. Package mode needs the module to let it write the host programs' configs (selinux.md, "The functions on the host"). tailscaled runs on the host as unconfined_service_t, an internet-facing daemon with no confinement: in image mode it moves into a container of its own, stronghold-tailscale (host network, NET_ADMIN, /dev/net/tun, its state in /var/lib/stronghold/functions/tailscale), a host function that is a container all the same; in package mode it stays Fedora's RPM. Either is installed when the config first puts the box in a tailnet (D21), and the node comes up once it is; the host image has no tailscale (D23). The node's keys stay in /var/lib/tailscale, so a box whose tailscaled ran on the host keeps its node when it moves into the container.

6. Layout

• stronghold: cmd/stronghold-agent, and the functions in internal/functions/<function>/. One Go module.
• stronghold-images:

bootc/Containerfile           the host image
functions/<function>/Containerfile

- stronghold's dist/quadlets/: the functions' quadlets, used as they are by the host image and podman hosts; the stronghold package carries them with the registry's images for a box from packages. - stronghold-operator: Kubernetes and OpenShift, notes only for now. - No repository per function: they share one config model and one version.

7. Phases

1. RPM packages of strongholdd and the UI. Done (stronghold, stronghold-daemon, stronghold-agent, stronghold-selinux, stronghold-ui; built in mock by the workspace's scripts/rpms.sh).
2. Render and apply split for every service function; no change in behaviour. Done.
3. The DNS resolver (Unbound) in a container: the agent, the image, its quadlet; measured against native in the lab. Done.
4. Kea, Knot, radvd, Suricata, headscale, the same way. Done, with Suricata's CPUs (ids.cpus).
5. The host image carries them and runs them; the ISO carries them. Done for every service function (DNS, DHCP and its relay, router advertisements, IDS, headscale, dynamic DNS, ACME, routing).
6. Podman hosts: the controller in a container, quadlets for any host. Done ("Podman hosts"; the host functions on such a host not yet).
7. Kubernetes and OpenShift: the operator (stronghold-operator).
8. Enabling functions (D20): where and runtime in the config and the API, GET /v1/functions, the host-mode SELinux policy, PackageKit installs, the Ansible function module, the UI's Features section of System and the tabs that follow the config (ui.md), the DHCP relay (dnsmasq), tailscaled in a container in image mode. Done (lego's HTTP-01 on the host not tried: selinux.md, "Not yet").