Updates¶

The box is an image (bootc). An update is a newer image from where the box follows (bootc status: a registry, or in the lab containers-storage): it is downloaded alongside the running one and started on the next boot, and the previous one stays, to go back to. strongholdd drives it (internal/updates); bootc's own bootc-fetch-apply-updates.timer stays off, because in a pair the two boxes must not restart together.

By hand¶

stronghold update            # what runs, what's newer, what's downloaded, the previous
stronghold update check      # ask the source (metadata only)
stronghold update download   # download now, start at the next reboot
stronghold update apply      # restart into it (downloading first if needed)
stronghold update rollback   # restart into the previous image

The same is in the UI (System tab) and the API (/v1/updates); the UI shows the host image, Stronghold and the functions' images apart, since they are versioned apart (D23): a new host image may bring Fedora's updates with the same Stronghold. A restart is a soft reboot (userspace only) when the kernel is the same, else a reboot; one downloaded already starts with a reboot, nothing fetched again.

A download fetches the functions' images too: those the new image's stronghold package names for the functions the box runs in containers (stronghold-prefetch-images@.service, from the new image's cache when it has one, else pulled), so that after the restart each container starts at once; without them it pulls its own first, the function down until it has (an IDS image took three minutes in the lab). The window always downloads first; an apply with nothing downloaded doesn't, and its containers pull after the restart. If they can't be fetched the update goes on; they are pulled after it, as before. A box from packages pulls them after the upgrade (its new quadlets are only there then).

In a pair, apply and rollback on the active box hand over to the standby first, so the gateway stays up; without a peer that can take over they refuse (--force: go ahead, the gateway goes down). On the standby they just restart it.

Automatically¶

# stronghold.yaml
updates:
  automatic: true
  days: [sun]        # default every day
  at: "03:30"        # local time, the window's start (default 03:30)

In the window (two hours) the box checks every ten minutes until it has either found no newer image or updated:

A box alone restarts into the newer image.
In a pair the standby goes first (only while its peer is active and answers, so the gateway stays up). The active box waits until its peer runs the new image (the peer's state says which image it booted), then hands over and updates; so the pair updates one box at a time, and ends with the roles swapped.
A backup first, if the backup settings ask for one (backups.before-update): without it, the update doesn't go ahead.

After an update¶

A box that restarts into a new image checks that it works (a marker, written before the restart, says it is one; strongholdd stopped by its own soft reboot, or a reboot that returns at once, is the restart, not a failure, and leaves the marker): within five minutes its config must be active and applied without error. If not, it goes back to the previous image (bootc rollback) and says so in the updates' events; it doesn't try that image again on its own until a newer one comes (the next check finds the same one, and the window is over by then).

That check runs inside strongholdd, so a new image whose strongholdd doesn't start is caught by greenboot instead (Fedora's greenboot, greenboot-rs, in the image): its required health check (/usr/lib/greenboot/check/required.d/50-strongholdd.sh) wants strongholdd running and its API answering within three minutes of boot. A boot where it doesn't is red: greenboot reboots, and after three red boots (GREENBOOT_MAX_BOOT_ATTEMPTS) GRUB boots the previous image and greenboot keeps it. Its red.d script leaves /var/lib/stronghold/greenboot-red; the previous image's strongholdd finds it, says which image didn't start (an event, and a critical updates alert) and ends the update it was doing. greenboot's own default checks (greenboot-default-health-checks: the repositories resolve, the update server answers) aren't installed: a gateway whose WAN is down is still a healthy box.

Limits¶

The source is whatever the box was installed or switched to (bootc switch); changing it isn't in the UI yet.
Images aren't signature-checked by Stronghold; that is the registry's and the container policy's (/etc/containers/policy.json).
A box installed from packages has no image: its updates are dnf's ("A box installed from packages", below). PackageKit asks NetworkManager whether the box is online, so the WANs' default routes are in the main table too (wan.md, "How it routes"); with them only in the WANs' tables, PackageKit 1.4.0 thought the box offline and its downloads failed ("cacheonly option is activated").

A box installed from packages¶

Its updates are its packages', from the system's dnf repositories, through PackageKit as the functions' installs are: strongholdd runs pkcon, never dnf itself (internal/updates/packages.go).

Check refreshes the repositories and lists the updates (pkcon refresh, get-updates): the status shows how many, how many are security updates, and their packages.
Download prepares them as an offline update (pkcon --only-download update).
Apply triggers the offline update and reboots (pkcon offline-trigger): the packages are installed while the box boots (system-update.target, Fedora's offline updates, as GNOME Software does them), so nothing is replaced under the running services, and it boots again into them.
After it the box says how PackageKit's update ended (pkcon offline-status) and checks itself as after an image update; but packages have no previous version to go back to, so a box that doesn't work is reported (a critical alert), not rolled back. rollback is refused.
The window and the pair work as for images: the standby goes first; for the pair, a box's "image" is its pending updates (packages:up-to-date once there are none), and the active box follows once its peer, the standby, has none pending.
The same commands, API and UI; GET /v1/updates says mode: packages. Cockpit's Software Updates page and dnf upgrade still work too: Stronghold just does it in its window, one box at a time.

Going back¶

An older image (after bootc rollback, or an update rolled back) may find a config with settings that came with the newer Stronghold. The config is parsed strictly (D13), so it wouldn't load, and the box would run without its config. Instead (internal/core/fallback.go):

Each build records the commit it last applied (/var/lib/stronghold/applied.json, by version).
A config that needs a newer Stronghold (a key this build doesn't know, or a newer schema version) isn't an error: the box runs the commit this build last applied, or if it has none, the newest commit in the history it can read.
The config directory isn't touched: the newer image runs the current config again. Meanwhile changes (UI, CLI, API, the peer's sync) are refused, since saving would drop the newer settings; the status says why, and a critical config alert goes out.
A key that is only a typo looks the same: the box runs its last applied config, and the status names the key.
In the lab: an unknown key added to gw2's config; after a restart it ran its last applied commit, all its firewall zones in place.