futhark

A GitOps-driven homelab. Three Fedora nodes joined over a NetBird mesh. Two of them run one k3s cluster, reconciled by Flux from this repository; the third runs the git forge under Podman, deliberately outside it. Everything that cannot live inside a cluster at all is managed by OpenTofu.

The tree splits along four planes, and almost every question about the repository resolves to “which plane owns this?”:

The table says who owns what. What it cannot say is how the planes meet: each hands off to the next exactly once, and nothing reaches back the other way. Green is this repository, the one source of truth. Amber is a third party, the things this repository can call but never own. Purple is the standalone plane, which reads the same repository through a mechanism of its own.

The repository has two readers rather than one, which is the only place the “hands off exactly once” shape does not hold. Flux is one. The forge is the other, and it is drawn apart deliberately: it reconciles from git without Flux and without a cluster, so a cluster outage cannot take it down. That is the entire reason it exists. See The standalone Podman plane.

Where to go next:

• Setting up from nothing: Cold bootstrap.
• Something is broken: Troubleshooting.
• Adding to the tree: Conventions.
• Replacing a credential: Credential rotation.
• Looking for a command: Recipe reference.

What runs where

kenaz runs the k3s server, so it is both controller and worker, and it carries the tenant apps under nodes/kenaz.k8s/. ogma is an agent and the cluster’s entrypoint: it is the only node with public ingress, so both Traefiks and Pocket ID are pinned to it with a nodeSelector. brokkr is in no cluster and serves git.$DOMAIN and ci.$DOMAIN itself. All three are on the mesh and are addressed by their mesh DNS name, never by a stored address. See Nodes.

The pieces, roughly in dependency order: the Infisical operator, which syncs runtime secrets into Kubernetes Secrets; Pocket ID for OIDC; cert-manager for Let’s Encrypt over DNS-01; two Traefiks, one public and one mesh-only; and a VictoriaMetrics, VictoriaLogs and Grafana stack. Each is described in Cluster infrastructure, and the order they must come up in is Startup ordering.

Secrets, in one paragraph

No credential is ever committed in the clear, and neither is any identifying value, because this repository is public. Values that identify but grant nothing, such as node addresses and the domain, are committed SOPS-encrypted. Anything that could bootstrap or re-key the system lives in Proton Pass and is never committed at all. The cluster holds no credential for it, so a cluster compromise cannot reach the keys that rebuild it. Per-app runtime secrets live in Infisical and reach pods through the Infisical operator. brokkr reads no store at all: its secrets are files Ansible pushed. The full rule, and what to do when you need a new one, is in Secrets. Replacing one is Credential rotation.

Cold bootstrap

Build the whole thing from nothing, in order. At the end you have provisioned hosts on a NetBird mesh, a k3s cluster reconciling from this repository through Flux, every runtime secret resolving from Infisical, nightly backups reaching Backblaze B2, and optionally a git forge on a node outside that cluster.

Every step is re-runnable. Expect a few hours, most of it waiting on reconciliation and on DNS-01 propagation.

Prerequisites

Before step 1, have:

• A GPG key whose encryption subkey is on a smartcard, and the card. It is the only thing that opens the encrypted files in this repository.
• Accounts you will create along the way: Proton Pass, Infisical (EU region), NetBird Cloud, Backblaze B2, Bunny, and a GCP project if you want the Google Drive storage class.
• One or more hosts you can reach over SSH as root or as a sudo-capable user.
• Fedora or another dnf distribution on the operator machine, plus uv. just ops deps requires both, and installs everything else itself.

Five values cannot exist until something else is running, so they are filled in twice: the edge node’s mesh address (step 7), the backup B2 application key and the forge’s own B2 key (step 8), the Pocket ID API token (step 10) and Forgejo’s OIDC client pair (step 11). Each is called out where it lands. Blue runs forward through the thirteen steps, green marks the first and last, and each red dashed arrow reaches back to a step that has to be revisited once the value it needed finally exists.

The four red edges are the only backward ones, and they are why the phases are not simply a list. Each of those values is produced by a step that needs a file written several steps earlier.

Step 11 is skippable in full. Nothing after it depends on the forge, and nothing in the cluster does either, which is the property that node exists to have.

1. Proton Pass

Create a vault, named after the Infisical project so the two remote stores are named alike. Then mint a personal access token in the Proton Pass web app. That token, plus your GPG smartcard, is everything a new operator machine needs out of band.

PROTON_PASS_PERSONAL_ACCESS_TOKEN=pst_… pass-cli login
pass-cli info                        # session persists from here on

Replace pst_… with the token you just minted.

Generate the Flux deploy key now, since it is stored here:

ssh-keygen -t ed25519 -f /tmp/flux-deploy -N '' -C futhark-flux

Add /tmp/flux-deploy.pub to the repository’s Deploy Keys on GitHub. Read-only is enough. Put the private half in the vault as below, then shred -u /tmp/flux-deploy*.

Now the items. Nothing is matched by name here. Every consumer addresses a pass://<vault>/<item>/<field> path, and the committed files that hold those paths are what this table has to agree with. Item and field names are lowercase-with-spaces, per Naming.

Two NetBird PATs, on two service users of different roles, both created in the dashboard before either token. The roles, and why they differ, are Bootstrap step 2. They expire, at most a year out. See Checks and CI for what each lapse breaks and Credential rotation for the replacement procedure.

Create the NetBird account itself now, and delete its shipped Default policy. The full sequence is Bootstrap. Leave that policy in place and the rules you apply at step 8 describe an access model nothing is enforcing.

One healthchecks item holds every node’s ping URL, one field per node named for the host, so adding a node is adding a field. Create each check in healthchecks.io first, with period 2 minutes and grace 15 minutes, which is past the ladder’s restart and re-up rungs and short of its reboot rung. A node whose field is missing still runs its watchdog, it just reports nothing.

The three brokkr-* items are only needed if you are building a workflow: podman node. Skip them otherwise, and skip that node’s field on healthchecks too. Four of their ten fields you generate now, with a password manager or openssl rand -base64 48:

The admin’s username goes in config/sops/ops.sops.yaml rather than here, as ansible.secrets.brokkr.FORGEJO_ADMIN_USER. It grants nothing on its own but is identifying, so it cannot be an Environment= line in a committed unit file.

The remaining six cannot exist yet, because nothing has minted them: tofu/b2 mints the two brokkr-restic B2 fields at step 8, tofu/oidc mints the two brokkr-forgejo oidc fields at step 10, and Forgejo itself issues the two forge fields on brokkr-woodpecker at step 11. Leave all six as placeholders and finish them at step 11, which lists the command that prints each.

brokkr-restic/repository password deserves the same treatment as the cluster’s restic password: it is the one credential here with no second copy anywhere, by construction, and there is no recovery path if it is lost. See Backup and recovery.

storagebox and rclone-crypt are the items in the table no committed pass:// reference points at. Their consumers read Infisical, not the vault, so these copies exist only so the credentials are recoverable, and you retype them into /infra/csi-rclone. For the crypt passwords that is not a convenience: lose one and the data it wrapped is ciphertext forever.

Three Infisical identities, not one, for the same reason, and a sharper one. cluster-reader is seeded into the cluster and tofu-writer never leaves this machine, so collapsing them would hand the cluster a write credential. backup-reader is split off for a different reason again: the cluster’s read credential must not also be the one that decrypts B2.

The admin SSH private key goes in no store at all. Ansible authenticates with your own ~/.ssh identity rather than reading it, and only its public half is committed, SOPS-encrypted, at step 4.

Nothing here is committed. What the repository holds is the map: pass://<vault>/<item>/<field> references, already written into config/sops/ops.sops.yaml.example, so unless you named the vault something other than the name the templates assume there is nothing to edit in that half. You seal the file at step 5, sealed to your GPG key only and never the cluster age key. The reasoning is in Secrets.

2. Infisical

Sign up at eu.infisical.com. That is a separate data region, not a mirror of app.infisical.com, and an account on one is invisible to the other. Create a project futhark with a prod environment.

Create three Universal Auth machine identities. With yourself that is 4 of the 5 the free tier allows:

• cluster-reader: read-only on the whole project, denied /infra/k8up. Leave accessTokenTrustedIps alone for now. You set it at step 13, once the cluster has an egress address.
• tofu-writer: write on the node folders the two writing modules target, and nothing else. Today that is /nodes/kenaz/actual, /nodes/kenaz/open-webui, /nodes/kenaz/linkwarden, /nodes/kenaz/bifrost, /nodes/kenaz/vane and /nodes/kenaz/kvasir. See oidc and bifrost. Widen it as a module gains a folder, never to the whole project.
• backup-reader: read on /infra/k8up and nothing else. The split is deliberate. Losing the cluster’s read credential must not also mean losing the ability to decrypt B2. See Secrets.

Copy all three client ID and secret pairs into Proton Pass per the table above.

Then create the folders and secrets. Names are SCREAMING_SNAKE_CASE throughout, per Naming:

Every database password appears twice on purpose: once in /infra/postgres, where CloudNativePG creates the role, and once in the consuming app’s own folder, where it is assembled into a connection string. The admission policy confines each namespace to its own tier’s folder, so neither side can read the other’s. Type the same value into both. Generate them from letters and digits only, because each is interpolated into a URL. See The shared database.

B2_KEY_ID and B2_APPLICATION_KEY are placeholders for now. tofu/b2 mints that key at step 8. RESTIC_PASSWORD is yours to generate, from openssl rand -base64 32, and it must exist before the first backup runs: it is baked into the repository at creation. Lose it and the backups are ciphertext forever, which is the point of it. The durability table is in Backup and recovery.

That table goes stale as apps are added. The authoritative version is the tree itself: every InfisicalStaticSecret names its secretPath, and any that remaps a key names the Infisical secret in its template block.

grep -rl 'kind: InfisicalStaticSecret' infra nodes

One NetBird credential appears in that table, and exactly one may. NETBIRD_API_KEY is read by Glance’s peers widget and by nothing else, so it belongs to its own service user with no write capability: the worst an attacker who reads it can do is list peers. The two PATs that can change the mesh, netbird-policy and netbird-enrollment, stay on the operator machine and never enter the cluster. See Credential rotation.

POCKETID_ENCRYPTION_KEY and SSO_COOKIE_SECRET are both new material, each openssl rand -base64 32. The cookie secret additionally has to be URL-safe, so pipe it through tr -- '+/' '-_'. Both must exist before their Deployment first reconciles.

/infra/csi-rclone is the one row you cannot fill in yet. Two of its values are minted by rclone and four more are generated with it, and rclone arrives with just ops setup at step 3, so create the folder now, leave it empty, and come back after that step. It also needs a Hetzner Storage Box and a Google OAuth client that nothing else in this bootstrap creates. The rclone remotes is the whole procedure, end to end.

3. The operator machine

just runs everything else here, so it has to come first. Nothing can install its own runner:

sudo dnf install just
just ops setup

That installs ansible-core, ansible-lint, yamllint, kubectl, helm, kustomize, flux, rclone, b2, tofu, netbird, pre-commit, sops, age, pass-cli, the GPG smartcard stack, and mdbook with d2 and mdbook-d2 for the diagrams. It also installs the pre-commit hooks, runs tofu init in every module but b2, and checks you have a Proton Pass session and a place on the mesh. b2 is skipped with a message, since its remote backend authenticates against secrets step 5 has not written yet. Step 8 initialises it.

Most of those are pinned in mise.toml and installed by mise, which puts them in ~/.local/share/mise/shims rather than on PATH. Put that directory on PATH in your login profile, not only in an interactive shell configuration. The kustomize build, just --fmt and tofu validate pre-commit hooks run outside an interactive shell and call those binaries by name:

echo 'export PATH="$HOME/.local/share/mise/shims:$PATH"' >> ~/.profile

Verify from a shell that has read the new profile:

kustomize version

It prints the kustomize version in mise.toml. If the command is not found, PATH has not picked up the shims, and pre-commit will fail on the kustomize build hook.

rclone and b2 are the two there purely for bootstrap. Nothing in just calls either. rclone exists so you can go back and finish The rclone remotes from step 2, and b2 so you can create the state bucket and the two application keys that b2 needs before step 8 can run.

You need two things of your own: the GPG smartcard plugged in, and the Proton Pass session from step 1. The third, this machine’s own membership of the mesh, is what just ops mesh checks. k8s_cluster resolves each node’s mesh address through NetBird’s DNS from here. The client arrives with just ops deps, but joining does not. Run the netbird up that just ops mesh prints, since it carries the --interface-name this repository uses, log in over SSO, then add the peer to the admin group from the dashboard, per netbird.

On a cold bootstrap that check is expected to fail here. The NetBird account has an admin group only after step 8, so there is nothing to join yet. It is the last thing ops setup runs, so everything above it has already happened. Come back and re-run just ops mesh once step 8 is done. Nothing between here and step 9 needs the mesh.

Then generate the cluster age key:

just ops age-key

Put the printed age1… recipient into .sops.yaml, replacing AGE_CLUSTER_RECIPIENT_PLACEHOLDER. Store the private key in Proton Pass as the age key field of the sops item, then shred the temporary file the recipe names. Why this key is separate from your GPG key, and what it can and cannot open, is in Secrets.

Verify: .sops.yaml no longer contains AGE_CLUSTER_RECIPIENT_PLACEHOLDER, and the temporary key file is gone.

4. Node definitions

One ansible/nodes/<hostname>/host.yml per machine, symlinked into ansible/inventory/host_vars/<hostname>/, and the hostname listed in ansible/inventory/hosts.yml. Each machine’s address goes in config/sops/ops.sops.yaml at step 5. The schema and exact commands are in Nodes.

5. The encrypted files

Two encrypted files, each shipping as a .example template. Ask which are still missing, then work through them:

just ops sops                    # lists what has no real file yet
just ops sops <file>             # copies the template, opens it, encrypts on save

Re-run the same command later to edit one. It decrypts and re-encrypts around your editor, and it fails closed: an aborted edit or a failed encrypt removes the plaintext rather than leaving it at a *.sops.* path.

Both templates arrive with every pass:// reference already written, so only the identifying half needs filling in.

config/sops/ops.sops.yaml, sealed to your GPG key only:

• ansible.admin: the admin user’s name and SSH public key.
• ansible.secrets: nothing to change beyond the vault name, per step 1.
• ansible.secrets.brokkr: the forge node’s runtime secrets, if you are building one. Eleven keys, ten of them pass:// references into the three brokkr-* items from step 1 and one, FORGEJO_ADMIN_USER, a literal. Six of those references point at fields nothing has minted yet: the two FORGEJO_OIDC_*, the two WOODPECKER_FORGE_* and the two B2_*. The references are still correct, so write them all now; it is the Proton Pass fields behind six of them that stay placeholders until step 11.
• nodes.<hostname>.ip: that node’s public address. Leave mesh_ip empty. The node has not joined the mesh yet, and roles/netbird writes it in at step 7.
• brokkr.B2_BUCKET: the forge node’s own restic bucket, separate from the cluster’s. Read by both Ansible and tofu/b2, which is why it is top level rather than inside either plane’s section. Omit the key entirely if you are not building that node.
• tofu.<module>: each module’s own credentials and identifying values. netbird needs its PAT, bunny its API key, oidc the Pocket ID base URL and the Infisical project ID, b2 its two key pairs, the state passphrase and the state bucket.

config/sops/cluster.sops.yaml, sealed to the cluster age key as well, holding one cluster-values Secret:

• DOMAIN, SUB_INTERNAL, SUB_NODES: the base domain and its subdomain labels. Flux, all four tofu modules and Ansible read these, and nothing else spells a domain out. See Domains.
• PUBLIC_IP, MESH_IP: the edge node’s addresses. Put a placeholder in MESH_IP for now, because the node has not joined the mesh yet. Step 7 fills it.
• B2_BUCKET, B2_REGION: where the restic repository lives.

No tofu.<module> section carries a node address or a domain. Each module declares those in its refs.env and reads them from whichever of the two files owns them, at plan and apply time. See Values another plane owns.

Nothing builds until config/sops/cluster.sops.yaml exists. config/kustomization.yaml references it, so kustomize build fails without it. That is deliberate: better a loud failure than a cluster reconciling with half its inputs missing.

6. Verify and push

pre-commit run --all-files

Everything must pass, including kustomize build and sops-encrypted. The latter is the one that catches a plaintext committed by mistake. gitleaks will not, because a node address matches no credential pattern.

Then commit and push. Flux reconciles from the remote, not from your working tree, and an unpushed commit is invisible to the cluster.

7. Host provisioning

just ans setup '' --skip-tags podman

Update, admin user, SSH hardening, mesh join, firewall. Pass a hostname as the first argument to limit it to one machine; '' means all of them, and anything after it goes to ansible-playbook. Safe to re-run.

--skip-tags podman matters only if you are building a workflow: podman node. That plane needs credentials no step before this one has minted, so it is deferred to step 11. On a cluster-only fleet the flag is a no-op and plain just ans setup is equivalent. ssh_identity picks whichever login currently answers, and after the first run each host answers only as the admin user on the hardened port. Provisioning nodes one at a time is fine, because the mesh-peer resolution in roles/netbird retries while the new peer’s DNS record propagates.

Each host’s mesh address is read back out of netbird status --json and written into nodes.<hostname>.mesh_ip in config/sops/ops.sops.yaml by the same run, so just ans setup leaves that file modified. Commit it. playbooks/k8s.yml reads the value from there, and tofu/bunny gets the public address from the same map.

MESH_IP in config/sops/cluster.sops.yaml is the edge node’s copy of that address, and Flux cannot read the operator store, so it still needs filling by hand:

ssh <edge host> netbird status --json | jq -r .netbirdIp
just ops sops config/sops/cluster.sops.yaml     # set MESH_IP
git commit -am 'fix(cluster-values): real mesh address' && git push

Verify: each host answers as the admin user on the hardened port, and ssh <host> netbird status reports the peer connected.

8. Mesh policy, and the backup bucket

Both before the cluster, not after. Cross-node pod networking needs the all-protocol node-to-node rule, and without it the cluster fails in ways that look like anything but a network fault.

just tf init netbird
just tf plan netbird && just tf apply netbird

Read netbird first. Three things this step needs that it cannot check for you:

• The account’s shipped Default policy has to be gone, from step 1.
• The netbird-policy service user has to be at Admin for this apply, because it creates netbird_account_settings and Network Admin cannot write account settings. Promote it, apply, then demote it again. The procedure is Applying account settings.
• Nothing validates a policy server-side. NetBird has no policy tests, so a wrong rule applies cleanly and fails later, in traffic.

This also sets the account’s peer DNS domain and network range, so do it before any node joins. A peer registered under the old ones has to re-register.

The DNS zone it creates resolves *.$SUB_INTERNAL.$DOMAIN to the ingress node’s mesh address, read from ["nodes"]["<host>"]["mesh_ip"] per tofu/netbird/refs.env. That is why this step comes after step 7: the value is empty until roles/netbird records it, and an empty one fails the apply. Nothing answers on that address yet, which is fine. The record is static, and the name starts answering usefully once step 9 brings traefik-internal up on it.

Then the bucket the backups live in, and the key K8up uses:

just tf init b2
just tf plan b2 && just tf apply b2

Read b2 first. It needs a state bucket and a state passphrase created by hand, and if a bucket already exists it has to be imported rather than created. File the two outputs into Infisical /infra/k8up as B2_KEY_ID and B2_APPLICATION_KEY, replacing the placeholders from step 2. Do this before the cluster, because infra/backup reconciles at step 9 and reads them there. Get it wrong and the symptom is every K8up job failing against a repository it cannot open.

If you declared brokkr.B2_BUCKET at step 5, this apply also created the forge’s bucket and its scoped key. That pair goes to Proton Pass, not Infisical, because the node that reads it holds no store credential:

just tf output b2 -raw brokkr_b2_key_id           # -> brokkr-restic/key id
just tf output b2 -raw brokkr_b2_application_key  # -> brokkr-restic/application key

Filing them now completes two of the four placeholders left at step 5.

Verify:

just tf plan netbird     # no changes
just tf plan b2          # no changes
just ops mesh            # this machine is on the mesh now

9. Cluster and Flux

just ans k8s

Installs k3s from inventory, the controller first and then the workers, and bootstraps Flux, including the Secrets that cannot come from Flux because Flux needs them to resolve anything else. The local-path StorageClass comes up with k3s itself. The full sequence is in Bootstrap and reconciliation.

The kubeconfig lands at ansible/.generated/kubeconfig, mode 0600, gitignored. Every ks and fx recipe points at it automatically.

just ks status
just fx failing

Expect several minutes. Certificate issuance in particular waits on DNS-01 propagation.

Verify: just fx failing is empty and just ks certs shows every certificate Ready. Anything still failing after that, start at Troubleshooting.

10. The cloud plane

Pocket ID is running now, so create its admin API key at Settings → Admin → API Keys on auth.$DOMAIN, and replace the POCKETID_API_TOKEN placeholder from step 1.

tofu/oidc and tofu/bifrost write into Infisical at paths the cluster is already watching, which is why they go last.

just tf plan bunny   && just tf apply bunny
just tf plan oidc    && just tf apply oidc
just tf plan bifrost && just tf apply bifrost

Each has its own prerequisites. See bunny, oidc and bifrost.

Verify: all three plans are no-ops on a second run, the Actual app picks up its OIDC client from /nodes/kenaz/actual without further edits, and VK_OPEN_WEBUI in /nodes/kenaz/bifrost holds the same value as OPENAI_API_KEYS in /nodes/kenaz/open-webui.

cli-proxy-api serves no model until an account is linked, which is a browser flow rather than an apply. See CLI proxy login.

11. The forge

Skip this step if you are not building a workflow: podman node.

It comes last because it depends on almost everything before it, and on nothing after. tofu/bunny at step 10 published git.$DOMAIN and ci.$DOMAIN; tofu/oidc at the same step minted Forgejo’s Pocket ID client, which needed Pocket ID running from step 9; tofu/b2 at step 8 created the bucket. Nothing in the cluster depends on this node in return, which is the whole point of it.

Six Proton Pass fields were left as placeholders at step 1. Four can be filled now, and the last two only after Forgejo is running, which is why this step doubles back on itself.

If you already filed the two brokkr-restic fields at step 8, only the two oidc ones are outstanding here:

just tf output oidc -raw forgejo_oidc_client_id      # -> brokkr-forgejo/oidc client id
just tf output oidc -raw forgejo_oidc_client_secret  # -> brokkr-forgejo/oidc client secret

Then converge the plane that was skipped at step 7:

just ans setup brokkr --tags podman

That installs Podman, opens 443 and 22, writes the env files, initialises the restic repository, clones this repository, starts every container from the units in nodes/brokkr.podman/units/, and creates the local admin plus the Pocket ID login source.

The two forge fields on brokkr-woodpecker are still placeholders at this point, and they are the one credential here no plane in this repository can mint: Woodpecker authenticates against Forgejo, and Forgejo issues it. Forgejo has to be running first, which it now is. Woodpecker comes up unable to complete a login until you create them:

1. Log in to https://git.$DOMAIN as the local admin.
2. Settings → Applications → Create OAuth2 application, redirect URI https://ci.$DOMAIN/authorize.
3. File the id and secret into the Proton Pass brokkr-woodpecker item.
4. Re-run just ans setup brokkr --tags podman.

Verify, and the fourth item is the one that matters:

ssh brokkr podman ps                                 # five containers, all Up
curl -sI https://git.$DOMAIN | head -1               # 200, Let's Encrypt certificate
ssh brokkr systemctl start futhark-forge-backup.service
ssh brokkr journalctl -u futhark-forge-backup -n 20  # a restic snapshot id

Then, in a browser: git.$DOMAIN offers “Sign in with pocketid” and no sign-up form, the local admin password logs in, ci.$DOMAIN completes the Forgejo OAuth round trip, and a trivial .woodpecker.yaml in a test repository runs a step to completion.

Last, prove the property the node exists for. Scale Pocket ID to zero and confirm the local admin still logs in:

flux suspend kustomization auth && kubectl -n auth scale deploy/pocketid --replicas=0
# log in at git.$DOMAIN as the local admin
kubectl -n auth scale deploy/pocketid --replicas=1 && flux resume kustomization auth

If that fails, the node is decoration. Everything else about it is in The standalone Podman plane.

Set the GIT_MIRROR_BROKKR repository variable and the GIT_MIRROR_BROKKR_KEY secret on GitHub to have .github/workflows/mirror.yml push a mirror here nightly. Until they are set that job skips.

12. Prove the isolation holds

The tier boundary is enforced by RBAC and an admission policy rather than by a secret store’s own namespaces, so it is worth confirming rather than assuming. Both checks are in Checks and CI. The admission one is the important half: an InfisicalStaticSecret in a node namespace asking for an /infra path must be rejected at admission, not merely fail to sync.

Delete the probe objects afterwards.

13. Aftercare

Set accessTokenTrustedIps on the cluster-reader identity to the cluster’s egress address. It is the only server-side constraint available on a single shared credential, and Secrets explains why there is only one.

Then decommission whatever store these values came from, and confirm nothing still points at it:

grep -rn 'pass://' --exclude-dir=.git .

Install the Kvasir pipe function in Open WebUI. Nothing reconciles this: Open WebUI keeps its Functions in its own database, and ENABLE_PERSISTENT_CONFIG=False governs its settings, not these. A rebuilt cluster therefore has a running kvasir that no client can reach until this is repeated. What does cover it is the open-webui backup Schedule, since the function is a row in that database.

In Admin Panel then Functions, open the menu beside +, choose Import From Link, and paste https://github.com/brewcoua/kvasir/releases/latest/download/pipe.py. Importing runs that file on the server, so read it first, or verify it came from that repository’s build:

gh release download --pattern pipe.py --repo brewcoua/kvasir
gh attestation verify pipe.py --repo brewcoua/kvasir

Set the function id to kvasir, which is what prefixes its models, then set its KVASIR_URL valve to http://kvasir.kvasir.svc.cluster.local:8080. The shipped default is http://kvasir:8080, which does not resolve from another namespace. Enable the function: kvasir.storm and kvasir.co-storm then appear in the model picker.

Use the storm half. Co-STORM works, but this cluster gives Kvasir an emptyDir for its sessions, so a round table lasts only until the pod restarts. See Node apps.

Finally, put the two NetBird PAT expiry dates in a calendar. Nothing here tracks them. See Credential rotation.

The rclone remotes

Fill Infisical /infra/csi-rclone with the eleven values the two rclone-backed StorageClasses need, so they provision and mount. At the end, a PVC against each class binds and a pod mounts it.

These are the only Infisical secrets in Cold bootstrap that are neither generated by a recipe nor copied from a provider’s console. You collect them from two web consoles, from a key file you generate, and from rclone output on the operator machine.

Prerequisites

• rclone on the operator machine. just ops setup installs it at bootstrap step 3. You use it three times: to obscure the four crypt passwords, to mint the Google Drive token, and to create the Drive folder and read its id. You never write a config file for the cluster.
• A Hetzner account, for the Storage Box.
• A Google account and a GCP project you can create an OAuth client in.
• Access to write Infisical /infra/csi-rclone, and to the Proton Pass vault.

Expect an hour, most of it in two web consoles.

The artifact

The rclone INI is not a file you own. It is committed in infra/storage/app/secret.yaml, with the credentials as {{ .KEY.Value }} placeholders, and the Infisical operator renders it into the configData key csi-driver-rclone reads. That manifest is the authority on the config’s shape. This page is the authority on where the values in it come from.

What is committed there is one INI with four sections: a backend, and a crypt wrapping it, for each of the two StorageClasses in infra/storage/app/. The crypt section headers are fixed. Each must match the remote: parameter of the class naming it (storageclass-storagebox.yaml, storageclass-gdrive.yaml), or that class provisions nothing.

What you supply is eleven secrets under /infra/csi-rclone:

The shapes are illustrative, truncated with …, and no value in that column is real. Each rotates on its own. The rotation procedures are in Credential rotation.

Every value is a single line except STORAGEBOX_KEY_PEM, which is stored as the multi-line file it is. rclone reads key_pem as a Go string literal, so the INI needs the newlines escaped as \n; the manifest does that at render time with {{ .STORAGEBOX_KEY_PEM.Value | replace "\n" "\\n" }}. Paste the key as it comes out of ssh-keygen and change nothing about it.

The Storage Box

Everything here is in the Hetzner Console, under Storage Boxes.

1. Create Storage Box. Pick a location and a size.

2. Create a subaccount for the cluster, so the credential the cluster holds is not the box’s main account. It gets its own home directory, and both a username and a hostname of the form uXXXXX-subN. Those are the user and the host in the INI. The box’s own uXXXXX hostname is not used anywhere in this procedure.

3. Enable SSH support and External Reachability on the subaccount, and leave both off on the box itself. Port 22 is always active but carries SCP and SFTP only; port 23 is the one you enable, and it is the port rclone uses. External Reachability is not optional here, because the cluster is not inside Hetzner’s network. Both settings take a few minutes to apply.

   Both toggles exist at the box level too, and setting them there is what the Hetzner documentation describes. Do not. The box-level toggles reach the main account, whose directory is the parent of every subaccount, and every subaccount you add later. Scoped to this subaccount, the port 23 that is exposed to the internet reaches one directory, and a subaccount added later for something else starts with no external access at all.

4. Generate a dedicated key pair. Not your admin identity. This one ends up in Infisical and in every node’s driver container.

   ssh-keygen -t ed25519 -f ~/.ssh/futhark-storagebox -N '' -C futhark-csi
   

5. Install the public half. One command, and it handles both ports:

   cat ~/.ssh/futhark-storagebox.pub | ssh -p23 uXXXXX-subN@uXXXXX-subN.your-storagebox.de install-ssh-key
   

   The manual path is fiddlier than it looks, which is why install-ssh-key is the one to use: port 23 accepts only one-line OpenSSH keys and port 22 only RFC4716, so hand-uploading an authorized_keys means converting the key for whichever port you skipped.

6. Check it. No password prompt:

   sftp -P 23 -i ~/.ssh/futhark-storagebox uXXXXX-subN@uXXXXX-subN.your-storagebox.de
   

7. Collect the three values. Nothing here needs rclone. STORAGEBOX_HOST and STORAGEBOX_USER are the subaccount’s hostname and username from step 2, both uXXXXX-subN. The port is not a secret and is fixed at 23 in the manifest. STORAGEBOX_KEY_PEM is the contents of ~/.ssh/futhark-storagebox:

   cat ~/.ssh/futhark-storagebox
   

   The config carries the key itself rather than a path to it. The CSI driver mounts the config Secret and nothing else, so a key_file path would not exist inside the driver container and every mount would fail at authentication.

One option you may need and should not set pre-emptively. Port 23 gives a restricted shell, and rclone probes it for a hash command on first use. Whether that probe succeeds against a Storage Box is unverified here. If a mount logs errors about md5sum, add disable_hashcheck = true to [storagebox] in infra/storage/app/secret.yaml. It is a setting, not a credential, so it belongs in the manifest rather than in Infisical. Crypt does not use the remote’s hashes anyway.

The Google Drive credential

[gdrive] needs its own GCP project and OAuth client. rclone’s built-in client ID is shared, heavily rate limited, and being retired during 2026.

Two settings on that client decide whether the mount survives:

• Publishing status must be In production. An external app left in Testing is issued refresh tokens that expire after 7 days. The CSI driver mounts the config Secret read-only, so rclone cannot write a rotated token back. A stable refresh token is a hard requirement, and the failure is silent until the eighth day. See what cannot be rotated.
• Scope drive.file, not drive. It grants access only to files the app itself created, which is the folder boundary the cluster is supposed to have, enforced by Google rather than by convention. It is also not a sensitive scope, so publishing needs no verification review. drive is restricted and triggers one.

A personal Google account rules out a service account, which is the usual answer for unattended access: a service account has no Drive quota of its own, and files it creates in a shared folder fail. So this is a user OAuth token, minted once, interactively.

In the Google Cloud console

The pages moved. What used to be “APIs & Services → OAuth consent screen” is now Google Auth Platform, split across Branding, Audience, Data Access and Clients. Instructions written against the old UI, including rclone’s own, no longer match.

1. At console.cloud.google.com, create a project. It exists only to hold this client.
2. APIs & Services → Enable APIs and services, search for Google Drive API, enable it. Without this the token mints fine and every call 403s.
3. Google Auth Platform → Get started. Fill in Branding (an app name and a support email), then set the audience to External and give a contact email.
4. Data Access → Add or remove scopes. Add https://www.googleapis.com/auth/drive.file and nothing else. rclone’s own documentation tells you to add drive, docs and drive.metadata.readonly. Do not. drive is restricted, and asking for it puts the app into a verification review it will not pass for personal use.
5. Audience → Add users. Add yourself. You need to be a test user to authorize before the app is published.
6. Clients → Create client. Application type Desktop app, even though the operator machine may be headless. The headless flow below still uses a desktop client. Record the client ID and secret.
7. Audience → Publish app. Confirm the status now reads In production. This is the step whose omission is invisible for a week.

Minting the token

This is the one place a local rclone remote is unavoidable: GDRIVE_TOKEN only exists once rclone has completed the OAuth flow, and it writes the result to its own config file. The remote you make here is a means of producing GDRIVE_TOKEN and GDRIVE_ROOT_FOLDER_ID, not a draft of the cluster’s config.

Run rclone config: n, name it gdrive, storage type drive, then the client ID and secret from step 6. At the scope prompt, answer drive.file by value rather than by menu number, because the numbering shifts between rclone versions. Leave service_account_file blank. Say yes to the advanced config so you can reach root_folder_id, but leave it blank on this pass; you do not have the id yet. Answer n to the Shared Drive question.

Then the browser. On a machine with one, take the default and let rclone open it. Headless, answer n to “Use web browser to automatically authenticate rclone with remote?”, run the rclone authorize "drive" "…" command it prints on a machine that has a browser, and paste the token back. Either way Google shows an unverified-app warning. That is expected for a personal client, and you continue past it.

The folder

With drive.file, rclone sees only what rclone created. A folder made in the Drive web UI is invisible to the cluster no matter what you name it, so create it here:

rclone mkdir gdrive:futhark
rclone lsf --dirs-only --format pi --separator ' ' gdrive:

That id is GDRIVE_ROOT_FOLDER_ID. Every path is then relative to that folder, which is what puts the CSI driver’s <namespace>/<pvc> inside it rather than at the root of your Drive.

The crypt passwords

The four crypt passwords are the only values with no console and no file behind them. You generate them. They are not plaintext: rclone stores them obscured, and a plaintext value fails at mount time on the base64 decode rather than being read as the password. Generate each with:

rclone obscure "$(openssl rand -base64 32)"

Four values, all distinct. The two crypts do not share a password, so a leak of one does not read the other’s data. Copy all four into the Proton Pass vault before they go into Infisical, for the same reason as the restic repository password in Backup and recovery: lose one and its data is ciphertext forever, and Infisical is not a backup of itself. These are among the values that cannot be rotated.

Verifying locally first

Optional. Checking it worked catches the same failures a sync cycle later. Do this if you would rather find a bad credential on the operator machine than in a driver log.

The local config already has a working gdrive remote. Add the two crypt sections and a storagebox section to it, taking the section headers and remote = values from infra/storage/app/secret.yaml. Use key_file = ~/.ssh/futhark-storagebox in the local [storagebox], not key_pem: an INI value cannot span lines, which is exactly why the cluster’s config escapes the key instead.

Both crypts must then resolve. An empty listing is a pass, an error is not:

rclone lsd storagebox-crypt:
rclone lsd gdrive-crypt:

This exercises the credentials, the crypt passwords and the Drive folder id. It does not exercise the escaping, which exists only in the rendered config, so check that after the sync.

Upload and clean up

Put the eleven values into Infisical /infra/csi-rclone. Nine come from the two consoles, from ssh-keygen and from rclone obscure above. The remaining two, GDRIVE_TOKEN and GDRIVE_ROOT_FOLDER_ID, are read out of the local rclone config; rclone config file prints its path.

Then destroy the local copies. They hold the Drive refresh token and the Storage Box private key in the clear, and nothing on the operator machine reads them again:

shred -u "$(rclone config file | tail -1)" ~/.ssh/futhark-storagebox

Keep ~/.ssh/futhark-storagebox.pub if you want to recognise the key on the box later; the private half now lives only in Infisical and Proton Pass.

Checking it worked

Confirm the operator rendered the INI and that no placeholder was left unfilled. A missing Infisical secret does not fail the sync; it renders as <no value>:

kubectl -n csi-rclone get secret storagebox-secret -o jsonpath='{.data.configData}' | base64 -d

key_pem is the exception, because it is the one value the template passes through a function. A missing STORAGEBOX_KEY_PEM fails the render with invalid value; expected string, and the InfisicalStaticSecret reports it:

kubectl -n csi-rclone describe infisicalstaticsecret storagebox-secret

In the rendered INI, key_pem is one line, and its \n are two characters each. If you see the key spanning several lines, the value in Infisical was escaped by hand before it was stored. Store the file verbatim and let the manifest escape it.

The end-to-end test is a PVC. Create one against each class, watch it bind, and delete it:

kubectl -n default create -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rclone-smoke
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: storagebox-crypt
  resources:
    requests:
      storage: 1Gi
EOF
kubectl -n default get pvc rclone-smoke -w

Binding proves provisioning, not mounting. The mount happens when a pod attaches. If a pod stays in ContainerCreating, the reason is in the node driver’s rclone container:

kubectl -n csi-rclone get pods -o wide
kubectl -n csi-rclone logs <the node pod on the pod's node> -c rclone --tail=50

The node DaemonSet is what mounts, so the pod you want is the one on the same node as the stuck workload. That is what -o wide is for.

Both classes are reclaimPolicy: Retain, so deleting the PVC leaves the directory on the remote. Remove it with rclone purge if you do not want the smoke test’s leftovers.

Reading the remotes afterwards

Once both classes work, files.$SUB_INTERNAL.$DOMAIN browses each remote at its root, decrypted, in a browser. That is the faster check on any later day: a directory listing there exercises the credentials, the crypt passwords and the mount in one page load. It is also a full read-write view of everything on the Storage Box, so read Copyparty before using it.

Backup and recovery

Git holds every declaration and Infisical holds every runtime secret, so Cold bootstrap rebuilds the cluster from both. What neither covers is PVC data. That is what K8up is for, and it is the only part of this cluster that cannot be reconstructed by re-running something.

Use this page to decide whether a volume is protected, to restore one namespace, or to rebuild a node or the whole cluster.

Prerequisites

• kubectl access. Every just bak, just ks and just fx recipe sets KUBECONFIG itself.
• For a full rebuild: the GPG smartcard, a Proton Pass session, and Proton Pass still holding the restic repository password. Without it the bucket is noise.

What is backed up, and what is not

Five tiers, decided by where the data already lives rather than by how important it is.

A local-path volume is on one node’s disk. Lose that disk and the data is gone, which is why that tier is the one K8up carries. storagebox-crypt is already offsite and already snapshotted. Copying it to B2 would be a second offsite copy of the same bytes, paid for twice.

gdrive-crypt is excluded for a different reason, and the distinction matters because the two classes look alike from the cluster. Google Drive has trash and per-file version history. It has nothing that restores a directory tree to a point in time. A deletion that propagates through rclone is gone once trash expires, and drive-use-trash=false on that class means it does not even reach trash. So the class is for data you can re-fetch or afford to lose. Put anything irreplaceable on it and you must opt the PVC in explicitly, with the annotation below, which then pulls every byte back through rclone and up to B2. That is usually the sign it belonged on a different class.

The last row is the one this page’s machinery does not cover at all. brokkr is not in the cluster, so K8up does not see it and no Schedule names it; it runs its own restic timer against its own bucket. Everything below is about the cluster’s repository, and brokkr’s is in brokkr.

Deciding what gets copied

Two independent switches, and a volume needs both.

A Schedule puts the namespace in scope. They all live in infra/backup/config/schedules.yaml, one per namespace, sharing every field except the namespace itself — the cron expression and the job history limits are patched in by infra/backup/config/kustomization.yaml. A namespace with no Schedule is never visited.

They are central rather than in each app’s own directory because a Schedule needs the k8up.io CRDs, so an app shipping its own would need dependsOn: backup on its Kustomization — which would stop Pocket ID reconciling whenever the backup stack was unhealthy.

A PVC annotation opts the volume in. The operator runs with skipWithoutAnnotation: true, so nothing is inferred:

metadata:
  annotations:
    k8up.io/backup: "true"

That annotation lives next to the PVC it names, because what to keep is a property of the app. It is why nodes/kenaz.k8s/actual annotates actual-server-files and not actual-user-files. The omission is the tiering. VictoriaMetrics and VictoriaLogs already expire their own data, and metrics are not worth the egress, so neither is annotated. That left the monitoring namespace with nothing annotated at all once Grafana’s state moved to PostgreSQL, which is why it no longer has a Schedule either.

Currently annotated: Actual’s budget SQLite, Linkwarden’s page archives, Open WebUI’s uploads and vector store, Pocket ID’s uploaded images, Vane’s settings and history. Open WebUI’s and Pocket ID’s are the remainder after a migration: the databases behind them moved to PostgreSQL and what stays on the volume is files.

A third switch, for PostgreSQL

The shared PostgreSQL is the one workload whose data volume is deliberately left unannotated. restic copying a running data directory produces a snapshot that looks fine and fails at restore time, so what gets backed up there is a dump taken at backup time instead:

inheritedMetadata:
  annotations:
    k8up.io/backupcommand: /bin/sh -c "pg_dumpall --clean --if-exists -U postgres"
    k8up.io/backupcommand-container: postgres
    k8up.io/file-extension: .sql

K8up execs that command inside the running pod and streams its stdout into restic as a single .sql object. Nothing is written to disk first, so the dump is never stale, and every tenant database is in it. inheritedMetadata is how a CloudNativePG Cluster puts annotations on its pods, since it has no pod template of its own.

The recovery point is the last nightly run. Point-in-time recovery would mean continuous WAL archiving through CloudNativePG’s own barman-cloud plugin, which is a second backup system with its own bucket and retention. Not adopted.

Excluding paths inside a volume

A volume is rarely all state. Pocket ID’s is 68 MB, 62 MB of it MaxMind’s GeoLite2 database, which the app downloads and refreshes on its own.

restic arguments pass through a second annotation on the same PVC:

k8up.io/backup-restic-args: '["--exclude=/data/pocketid-data/GeoLite2-City.mmdb"]'

K8up mounts each PVC at /data/<claim-name>, which is where those paths come from. The value is a JSON array. A malformed one does not fail the backup — K8up logs failed to parse restic backup args from the annotation and skips that PVC entirely, so check the snapshot after changing it.

The other way to shrink a volume is to stop keeping state on it. Grafana’s was 257 MB, of which grafana.db was 1.3 MB and the rest a plugins directory the chart reinstalls; moving that 1.3 MB into PostgreSQL removed the volume from the backup set entirely rather than excluding most of it.

What is not backed up at all

Kubernetes resource manifests. Velero collected them cluster-wide; K8up backs up volume data and nothing else. Nothing is lost by it: git owns every declaration and Infisical owns every runtime secret, so no manifest in a backup would ever be the only copy.

SQLite is copied live

The nightly job copies pocket-id.db and Actual’s budget databases as files, alongside whatever -wal and -shm sit next to them. That is a crash-consistent copy, not a quiesced one: restoring it is equivalent to recovering from a power cut, which SQLite handles, but it is not the same as a dump. k8up.io/backupcommand would take a real sqlite3 .backup instead, and needs sqlite3 to exist in each image. Not adopted for these, though it is what the postgres namespace uses.

brokkr does take the quiesced copy, because it is not going through K8up at all and can run sqlite3 .backup on the host against the container’s file. Its databases are Forgejo’s and Woodpecker’s, and both hold every user’s credential, so a crash-consistent copy was not worth the saving there. See brokkr.

Encryption

restic encrypts the repository client-side with AES-256 before anything leaves the cluster, so Backblaze stores ciphertext it cannot read. One password does it, RESTIC_PASSWORD in Infisical /infra/k8up, handed to the operator as BACKUP_GLOBALREPOPASSWORD.

The password is baked into the repository at creation and cannot be changed afterwards without starting a new one. Losing it loses every backup, irrecoverably. There is no recovery path, by design — that is the same property that keeps Backblaze from reading them. It belongs in Proton Pass, and it must be there before the first backup runs.

It, and the B2 application key, are read with a machine identity of their own rather than the cluster-reader every other namespace uses. See Secrets.

Of the two, only the B2 application key can be replaced. b2 is what mints it, along with the bucket, its lifecycle rules and the reason there is no object lock on it. The rotation procedure is Credential rotation, and the same page states plainly which credentials cannot be rotated at all.

Repository maintenance

infra/backup/config/schedule-maintenance.yaml runs weekly in the k8up namespace, and is the only place check and prune run. There is one restic repository behind every Schedule, so a prune from auth would forget snapshots belonging to monitoring just the same; running it once also avoids three jobs contending for restic’s exclusive prune lock.

• check reads every pack file and verifies it against the index. This is the only thing that catches bit rot or a truncated upload, and it catches it while there is still an older snapshot to fall back on rather than at restore time.
• prune applies the retention policy: 14 daily, 8 weekly, 6 monthly.

A prune reclaims no billed storage for a further 30 days, because the bucket’s days_from_hiding_to_deleting rule keeps hidden versions that long.

Inspect what actually happened

just bak schedules          # every Schedule and its most recent jobs
just bak snapshots          # every restic snapshot, the volume it holds and its size
just bak ls <id> [<depth>]  # look inside one, without restoring it
just bak jobs               # backup, check and prune jobs, newest last
just bak logs <job> <ns>    # restic's summary of what that run copied

The Snapshot object carries no size, so just bak snapshots reads one off restic itself and joins it on the id. A 0 there is a snapshot that holds nothing, which is what a run that copied nothing looks like, and is the check that matters after any change to an annotation. It comes from the summary restic wrote at backup time, so a snapshot taken by a restic older than 0.17 has an empty size rather than a wrong one.

just bak ls lists a snapshot’s contents with the size of each entry, eza -T -L<depth> style. The depth counts from the volume the snapshot holds rather than from /, and defaults to 2:

just bak ls <id>       # the volume's contents and one level under them
just bak ls <id> 4     # deeper

Back up a namespace immediately instead of waiting for 03:00:

just bak now monitoring

Restore a namespace

This deletes data. The recipe removes the namespace’s local-path PVCs, recreates them empty and restores the newest snapshot of each. It prints what it will destroy and what it will leave alone, then makes you type the namespace back before it proceeds.

Look at what you have first:

just bak snapshots

Then restore:

just bak restore actual

Verify: the namespace’s pods return to Running with their data present, and just fx failing is empty once Flux resumes.

What the recipe does that a hand-written Restore does not:

• Suspends the Flux Kustomization that owns the namespace first. Flux recreates a deleted PVC within its reconcile interval, which races the restore and wins often enough to matter. Any HelmRelease targeting the namespace is suspended too — a chart-created PVC, like Grafana’s, appears in no Kustomization inventory at all.
• Scales the namespace’s workloads to zero. A PVC with a running pod on it stays Terminating forever.
• Recreates each PVC from its own live spec rather than letting Flux do it. K8up restores into an existing claim, and resuming Flux to get one back would also scale the workloads up again, straight onto a volume that is still empty.
• Sets spec.paths on each Restore. A restic snapshot holds exactly one path, so there is no snapshot that carries a whole namespace; without that field a restore of one volume can land another volume’s contents.
• Runs flux resume at the end, which is what puts the replica counts back.

If the recipe fails part way through, the namespace can be left suspended and scaled to zero. Resume it by hand:

just fx reconcile <kustomization>

Restore the PostgreSQL dump

Partly verified. just bak pg-dump has been run against a real snapshot and produces the .sql file. The replay half of just bak pg-restore has not yet completed a drill: the first attempt stopped before reaching psql, on a wait that could never finish. Run it once and correct this section from what actually happens, before relying on it in an incident.

just bak restore postgres is the wrong tool here and will not help: it restores local-path PVCs, and this namespace’s data is a .sql object rather than a volume snapshot.

A Restore CR is not an option here, and this is the one place K8up’s two halves are not symmetric. Its own documentation: “You can’t restore from backups that were done from stdin (PreBackupPod or backup command annotation). In those cases, use the manual restore option described below using the restic dump or restic mount commands.” So the dump comes back through restic, which is why restic is pinned in mise.toml.

Write it to a local file. This touches no database and no cluster state, so it is safe at any time, and it is also how the drill is done:

just bak pg-dump                        # newest .sql snapshot
just bak pg-dump <id>                   # a specific one, from just bak snapshots
just bak pg-dump <id> ./somewhere.sql   # somewhere other than .tmp/

The B2 key, the restic password, the bucket and the endpoint are read back off the cluster rather than out of Infisical by hand, so there is one procedure rather than two. Read the head of the file to see the roles and databases it would recreate.

Then the destructive half. pg_dumpall --clean drops and recreates every database and role in the dump, so anything written since the snapshot is gone:

just bak pg-restore
just bak pg-restore <id>

It fetches through pg-dump to a file rather than a pipe, so what was applied is still on disk afterwards. It prints what it will replace and makes you type the namespace back first, then suspends the tenants’ Flux Kustomizations, scales them to zero, replays into the primary over its local socket, and resumes Flux. Which namespaces count as tenants is read from the postgres policy overlay: whatever is allowed to reach port 5432, minus the operator.

The scale-down is per namespace, not per workload, because nothing declares which pods hold a database connection. monitoring is a tenant, so a replay also stops VictoriaMetrics and VictoriaLogs for its duration. Expect a gap in metrics and logs across the restore, including the metrics you would use to judge how it went. DaemonSets are left running throughout; they are never scaled by this and hold no connection.

Resuming Flux is a shell trap rather than the last line, so an abort anywhere after the suspension still puts the tenants back. If the process is killed hard enough to skip the trap, SIGKILL or a closed terminal, the cluster is left suspended and at zero replicas. Recover with:

just fx get                                  # SUSPENDED column shows what is still held
flux resume kustomization <name> [<name>…]

Flux restores every replica count from git, so nothing needs scaling back by hand.

Verify: each tenant shows its own data rather than an empty install, and just fx failing is empty.

Restore by hand, with restic

Worth knowing because it does not depend on the cluster being up. Take B2_KEY_ID, B2_APPLICATION_KEY and RESTIC_PASSWORD from Infisical /infra/k8up (or Proton Pass, for the last one), then:

export RESTIC_REPOSITORY="s3:https://s3.<region>.backblazeb2.com/<bucket>"
export RESTIC_PASSWORD=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

restic snapshots
restic restore <id> --target /restore

<region> and <bucket> are B2_REGION and B2_BUCKET in config/sops/cluster.sops.yaml.

brokkr

brokkr is outside the cluster and backs itself up, so nothing above applies to it. It writes to its own restic repository in its own B2 bucket, with a key that cannot read the cluster’s. That separation is the whole reason there are two repositories rather than one with two prefixes: restic has no per-path access control, so a key that can write into a repository can read every snapshot in it. See The standalone Podman plane.

Its credentials are already on the node, in /etc/futhark/restic.env, which is the fastest path when the node itself is what you are recovering:

ssh brokkr
sudo -i
set -a; . /etc/futhark/restic.env; set +a

restic snapshots --tag forge

Off the node, take RESTIC_PASSWORD, B2_KEY_ID and B2_APPLICATION_KEY from the Proton Pass brokkr-restic item, the bucket from brokkr.B2_BUCKET in config/sops/ops.sops.yaml and the region from B2_REGION in config/sops/cluster.sops.yaml, then export them as above.

Each snapshot holds the three data directories and the SQLite staging directory together, so one snapshot is the whole node’s state. To restore it:

# Restore beside the live data first and read it, rather than over the top of it.
restic restore latest --target /restore

systemctl stop forgejo woodpecker-server woodpecker-agent
cp -a /restore/srv/futhark/forgejo/. /srv/futhark/forgejo/
cp -a /restore/srv/futhark/woodpecker-server/. /srv/futhark/woodpecker-server/
# The live .db files are the ones the containers wrote; the consistent copies are in backup/.
cp /restore/srv/futhark/backup/forgejo.db /srv/futhark/forgejo/forgejo.db
cp /restore/srv/futhark/backup/woodpecker.db /srv/futhark/woodpecker-server/woodpecker.sqlite
systemctl start forgejo woodpecker-server woodpecker-agent

The database copies from backup/ are what to use, not the ones inside the data directories. Those were captured live and may hold a partial transaction; the ones in backup/ came from sqlite3 .backup and are consistent.

/srv/futhark/traefik is in the snapshot too but is not worth restoring. It holds the ACME account key and the issued certificates, and Traefik re-issues both on start.

Rehearse this before relying on it. Restoring to /restore and confirming git -C /restore/srv/futhark/forgejo/git/repositories/<owner>/<repo>.git log reads and sqlite3 /restore/srv/futhark/backup/forgejo.db .tables opens is the whole rehearsal, and it touches nothing live.

Rebuild brokkr from scratch

1. Reinstall the OS, then just ans setup brokkr. That converges the runtime, pushes the secrets, clones the repository and starts every container from the units in git. The forge comes up empty.
2. Commit the new nodes.brokkr.mesh_ip the netbird role wrote, and delete the stale peer from the NetBird dashboard, exactly as for a cluster node.
3. Restore as above. The bootstrap tasks in ansible/roles/forge_bootstrap recreate the local admin and the Pocket ID login source against the empty instance, and restoring over them replaces both with what the snapshot holds, which is the intended outcome.

Nothing here needs the cluster, Flux or Infisical at any point. That is the property being tested.

Rebuild a wiped node

A corrupted host gets reinstalled, which destroys every local-path volume on it. Both cluster nodes hold some: Pocket ID is pinned to ogma, Actual to kenaz, and the monitoring stack floats. For brokkr, which has no local-path volume and no cluster to rejoin, the procedure is Rebuild brokkr from scratch instead.

1. Reinstall the OS, then:

   just ans setup <host>
   

   The netbird role registers the node again and writes the new mesh IP back into nodes.<host>.mesh_ip in config/sops/ops.sops.yaml. Commit that change. If the node is the edge one, update MESH_IP in config/sops/cluster.sops.yaml to match.

2. Delete the node’s old peer from the NetBird dashboard. The rejoin creates a new one, and the stale peer otherwise sits in the groups holding an address nothing answers on.

3. Rejoin the cluster, reinstall the local-path provisioner, and re-seed the Flux and Infisical credentials:

   just ans k8s
   

4. Wait for Flux. just fx failing must come back empty. The apps return with empty PVCs.

5. Restore each namespace that had data on that node:

   just bak restore <namespace>
   

Step 5 is per namespace on purpose. There is no restore-everything task, because a restore is a destructive operation and the set of namespaces that actually lost data is a judgement the operator makes, not one a script should guess.

Rebuild the whole cluster

Cold bootstrap gets you a running, empty cluster. K8up comes back with it, since the Schedule resources are in git and the credentials are in Infisical. Snapshots reappear in just bak snapshots once the operator has synced them from the repository. Then restore per namespace as above.

What you need out of band is unchanged from the cold bootstrap: the GPG smartcard that opens SOPS, and a Proton Pass session. The backup-specific addition is that Proton Pass must still hold the restic repository password.

Where the pieces are

The alerts are what makes the rest trustworthy. One fires on any failed K8up job, backup, check or prune alike. One fires when a namespace has had no successful backup in 26 hours, which is the only rule that catches a deleted Schedule or an expired B2 key, where nothing fails because nothing ran.

Failure modes

A volume silently stops being backed up. A typo in k8up.io/backup-restic-args makes K8up skip the PVC and carry on, and the remaining volumes still produce a successful job. The symptom is a missing entry in just bak snapshots; the cause is in the job log, as failed to parse restic backup args from the annotation.

Every job fails immediately. The B2 credentials in /infra/k8up are wrong or expired, or B2_REGION does not match the account. tofu/b2’s check block catches the second at plan time.

A restore hangs with a PVC Terminating. A pod still has it mounted. just bak restore scales workloads to zero for this reason; if you wrote the Restore by hand, scale them down yourself.

Storage keeps growing after a prune. Expected for 30 days. The bucket keeps hidden object versions that long, deliberately, so an accidental or malicious repository wipe stays undoable.

Credential rotation

Replace any credential this repository depends on, prove the replacement works, then retire the old value. Each section below is self-contained: read the one you need.

The reader this page assumes is an operator with a Proton Pass session and a GPG smartcard, part way through an incident or a scheduled rotation, who wants to know what breaks, what to run, and how to tell it worked.

Prerequisites

• A Proton Pass session on this machine. pass-cli info must succeed. If it does not, see Secrets.
• The GPG smartcard, for anything that re-encrypts a *.sops.* file.
• kubectl pointed at the cluster, for anything that reseeds a Secret. Every just ks and just fx recipe sets KUBECONFIG itself.

The rule that makes most of this cheap

Nothing in git holds a credential. Committed files hold pass://<vault>/<item>/<field> references, and pass-cli resolves them at run time. Update a Proton Pass item in place and every reference keeps resolving, so most rotations need no commit at all.

Two families of exception, and both are called out in their own sections below:

• Values sealed into a *.sops.* file rather than referenced, such as admin.ssh_pubkey.
• Values consumed from Infisical by the cluster rather than from Proton Pass by the operator. Proton Pass holds a copy for recovery, so both stores move.

Order of operations

Every procedure here follows the same shape, and the order is what makes it safe:

1. Create the new credential. The old one stays live.
2. File the new value where its consumers read it.
3. Prove a consumer works with the new value.
4. Only then revoke or delete the old one.
5. Prove the same consumer still works after the revoke.

Step 5 is not ceremony. It is the only evidence that nothing else was quietly using the old credential. just bak jobs after a B2 key rotation is the worked example.

Because the old value stays live until step 4, rollback for steps 1 through 3 is always the same: put the old value back where you got it and re-run the verification. After step 4 there is no rollback, which is why the proof comes first.

Standing schedule

Expiry dates are not tracked anywhere in this repository. Put both NetBird dates in a calendar when you issue the tokens. What breaks when each lapses is in Checks and CI.

Operator identity

GPG key

The .sops.yaml recipient is the primary key’s full fingerprint, so renewing or replacing the encryption subkey alone changes no file. gpg selects the current encryption-capable subkey at encrypt time. Rotate the subkey freely and stop here.

Replacing the primary key is the expensive case, because every encrypted file in the repository is sealed to it.

Blast radius: none while both keys are recipients. Between removing the old recipient and having the new key available on every operator machine, no *.sops.* file can be decrypted, which blocks Ansible, OpenTofu and just ops sops. Flux is unaffected: it decrypts with the cluster age key, which is a separate recipient.

1. Generate the new key and move its encryption subkey to the smartcard.

2. Add the new fingerprint alongside the old one under every creation_rules entry in .sops.yaml. All three rules name the same PGP recipient today.

3. Re-encrypt the data key of both files to both recipients:

   just ops rekey
   

   The recipe runs sops updatekeys over every *.sops.* file in the repository. It rewrites the recipient list only. It does not change any value, so the diff is confined to the SOPS metadata block.

4. Verify by decrypting the file matched by each .sops.yaml rule, using the new key:

   sops -d config/sops/ops.sops.yaml >/dev/null
   sops -d config/sops/cluster.sops.yaml >/dev/null
   

   Both must exit 0. A rule you forget in step 2 fails here, not later.

5. Remove the old fingerprint from .sops.yaml, run just ops rekey again, and repeat both decrypts.

6. Commit. The diff touches .sops.yaml and the metadata block of both encrypted files.

7. Revoke the old key and publish the revocation.

Rollback: until step 5, the old key is still a recipient and still decrypts everything. After step 5, restore the previous commit and re-run updatekeys while you still hold the old key.

Proton Pass personal access token

Blast radius: pass-cli stops resolving references, so just ans render-secrets, just ans setup, just ans k8s and every just tf recipe fail. The cluster is unaffected. It holds no Proton Pass credential, by design.

1. Mint a new personal access token in the Proton Pass web app.

2. Log in with it:

   PROTON_PASS_PERSONAL_ACCESS_TOKEN=pst_… pass-cli login
   pass-cli info
   

   Replace pst_… with the new token. pass-cli persists a session after this, so it is a once-per-machine step.

3. Verify a real resolution rather than the session alone:

   just ans render-secrets
   

   It writes ansible/.generated/secrets.yml. No pass:// string may survive in it:

   grep -c 'pass://' ansible/.generated/secrets.yml    # expect 0
   

4. Revoke the old token in the web app.

Do this on every operator machine. The session is per machine.

Admin SSH key

This key is in no store. It is your own identity in ~/.ssh, and Ansible authenticates with it rather than reading it. Only the public half is committed, as ansible.admin.ssh_pubkey in config/sops/ops.sops.yaml.

Blast radius: this is the path back into a node. Getting it wrong locks you out of every host at once.

1. Generate the new keypair and load it into your agent.

2. Add the new public key alongside the old one. admin.ssh_pubkey holds a single value today, so the safest sequence is to append the new key to the admin user’s ~/.ssh/authorized_keys on each host by hand first:

   ssh-copy-id -i ~/.ssh/<new key>.pub -p <hardened port> <admin user>@<host>
   

3. Verify you can log in with the new key on every host before touching anything else.

4. Set admin.ssh_pubkey to the new key and re-encrypt:

   just ops sops config/sops/ops.sops.yaml
   

5. Apply it, which rewrites authorized_keys and drops the old key:

   just ans setup
   

6. Verify a fresh connection on each host, in a new terminal, keeping the working session open until it succeeds.

7. Delete the old private key locally.

Rollback: while the session from step 6 is still open, put the old public key back in the SOPS file and re-run just ans setup.

Proton Pass, the crown-jewel vault

The item and field names are in the table at Cold bootstrap. Rotation updates the item in place, so the committed pass:// references do not change.

The cluster age key

Flux mounts this as flux-system/sops-age and it is the only key that opens the cluster-plane files. It cannot open anything under ansible/ or tofu/.

Blast radius: every Flux Kustomization with spec.decryption fails to decrypt, so nothing reconciles. Running workloads keep running.

1. Generate the new keypair:

   just ops age-key
   

   It prints the age1… public recipient and the path to a temporary private key file.

2. Add the new recipient alongside the old one on the config/sops/cluster.sops.yaml rule in .sops.yaml, then re-encrypt that file to both:

   just ops rekey
   

3. Store the new private key in the age key field of the sops item, then shred the temporary file. Keep a copy of the old key until step 6.

4. Reseed the cluster:

   just ans k8s
   

   ansible/roles/flux_bootstrap recreates flux-system/sops-age.

5. Verify Flux decrypts with it:

   just fx failing
   

   Empty output. A decryption failure surfaces on the Kustomization, not the HelmRelease, so also check just fx get shows every Kustomization Ready.

6. Remove the old recipient from .sops.yaml, re-run just ops rekey from step 2, commit, push, and confirm just fx failing is still empty after Flux has pulled the new commit.

Rollback: before step 6 the old key is still a recipient. Restore the previous flux-system/sops-age by putting the old private key back in Proton Pass and re-running just ans k8s.

The Flux deploy key

Blast radius: Flux cannot fetch the repository. Running workloads keep running, and nothing new reconciles.

1. Generate a keypair:

   ssh-keygen -t ed25519 -f /tmp/flux-deploy -N '' -C futhark-flux
   

2. Add /tmp/flux-deploy.pub to the repository’s Deploy Keys on GitHub. Read-only is enough. Leave the old deploy key in place.

3. Put the private half in the deploy key field of the flux item base64-encoded on a single line, then shred -u /tmp/flux-deploy*:

   base64 -w0 < /tmp/flux-deploy
   

   The field must hold that one line and nothing else. A multi-line private key is silently flattened on the way to the cluster, and step 5 then fails with ssh: no key found on the GitRepository. See Multi-line values.

4. Reseed flux-system/git-deploy-key:

   just ans k8s
   

5. Verify Flux fetches with it. Push a trivial commit, then:

   just fx reconcile
   just fx sources
   

   The GitRepository must report the new commit as Ready, with its revision matching what you just pushed.

6. Delete the old deploy key on GitHub, then repeat step 5. A fetch that still succeeds is the evidence nothing else was using it.

The NetBird tokens

Three Personal Access Tokens, one per service user. All expire, 365 days out at most. A PAT inherits the role of the user it belongs to, so issue the replacement on the same service user and no role changes.

Two of them, netbird-policy and netbird-enrollment, live only on the operator machine. The third belongs to a read-only service user, is stored in Infisical at /infra/glance, and is the only NetBird credential that enters the cluster. Keep it that way: it is safe in there because it cannot change the mesh.

Blast radius: none takes the mesh down. Peers keep their configuration and keep connecting. What stops is changing anything: no policy applies, and no new node joins. Losing the read-only one costs one widget. The full table of what breaks is Checks and CI.

1. In the NetBird dashboard, Team → Users, open the service user, then Access Tokens.

2. Create a new token with the same name. The plaintext is shown once and stored hashed, so file it before closing the dialog.

3. Update the token field of the matching Proton Pass item in place.

4. Verify, and the check differs per token:

   netbird-policy, used by tofu/netbird:

   just tf plan netbird
   

   A clean plan. A dead token returns 401 rather than a plan.

   netbird-enrollment, used by ansible/roles/netbird:

   just ans setup <host>
   

   The task named “Mint a single-use setup key” must succeed. An already-connected peer skips both the lookup and the mint, so this only exercises the token on a peer that is not currently joined. If every peer is up, the honest check is to re-enrol one deliberately, or to accept that the token is unverified until the next join.

   The read-only one, used by Glance: file it into Infisical at /infra/glance as NETBIRD_API_KEY instead of into Proton Pass alone, wait for the InfisicalStaticSecret interval, then restart the pod and confirm the NetBird peers widget on home.$SUB_INTERNAL.$DOMAIN lists peers again.

   just ks restart glance glance
   

5. Delete the old token in the dashboard.

If a token was minted ad hoc for a one-off task, delete it as soon as the task is verified rather than leaving it to expire. The elevation procedure in netbird is the case where this comes up.

The Infisical machine identities

Three Universal Auth identities, each a client ID and client secret pair: infisical-cluster-reader, infisical-tofu-writer, infisical-backup-reader. What each is scoped to is Cold bootstrap.

Blast radius: cluster-reader and backup-reader are seeded into the cluster, so rotating either stops every InfisicalStaticSecret in its tier from syncing. Existing Kubernetes Secrets are not deleted, so running pods keep their values until they restart. tofu-writer never leaves the operator machine and only affects just tf apply oidc.

1. In the Infisical console, add a new client secret to the identity. An identity can hold more than one, so the old secret stays valid.

2. Update the client secret field of the matching Proton Pass item in place.

3. For cluster-reader and backup-reader, reseed infisical-universal-auth into every tier namespace:

   just ans k8s
   

4. Verify. The observable condition is an InfisicalStaticSecret that resyncs:

   kubectl get infisicalstaticsecrets -A
   

   Every one must report ready. InfisicalAuth is not ready here means the seed Secret is missing or stale, not that the operator is broken. See Secrets.

   For tofu-writer:

   just tf plan oidc
   

5. Delete the old client secret in the console, then repeat step 4.

Rotating the secret does not touch the identity’s project membership, role, paths, or accessTokenTrustedIps. Those live only in the console and are unaffected. If you replace the whole identity rather than its secret, all four have to be set again, and a missing project membership surfaces as Unauthorized access: status 403.

The Bunny API key

Bunny keys are account-wide, not zone-scoped, and two consumers read this one: tofu/bunny from Proton Pass, and cert-manager’s DNS-01 webhook from Infisical /infra/cert-manager as BUNNY_API_KEY. Both move together.

Blast radius: DNS records stop being managed and certificate issuance fails at the DNS-01 challenge. Existing certificates keep working until renewal.

1. Create the new key in the Bunny console.

2. Update the api key field of the bunny item in Proton Pass.

3. Update BUNNY_API_KEY under /infra/cert-manager in Infisical.

4. Verify both consumers. A clean plan proves the first:

   just tf plan bunny
   

   The second needs a real challenge, because the webhook only uses the key at issuance:

   just ks certs
   

   Delete one certificate’s Secret to force a renewal and watch it reissue, or wait for a scheduled renewal. Until an issuance succeeds, the webhook half is unverified.

5. Revoke the old key.

The Pocket ID API token

Used by tofu/oidc only.

Blast radius: just tf apply oidc fails. Existing OIDC clients keep working.

1. On auth.$DOMAIN, Settings → Admin → API Keys, create a key.

2. Update the api token field of the pocketid item.

3. Verify:

   just tf plan oidc
   

4. Delete the old key.

The Backblaze keys

Three distinct credentials, and they are easy to confuse:

For backblaze-tofu:

1. Create a new application key in the Backblaze console with the same capabilities.

2. Update key id and application key on the item.

3. Verify:

   just tf plan b2
   

4. Delete the old key.

For backblaze-tofu-state, the same loop applies to key id and application key, verified by any just tf plan b2, which reads and locks the remote state.

The state passphrase field on that item is different. It derives the key OpenTofu encrypts the state with, client-side. It rotates through the same fallback mechanism the migration off SSE-C used: in tofu/b2/backend.tf, add a second key_provider holding the old passphrase, point the fallback at a method keyed to it, run any just tf plan b2 to read the old state and any just tf apply b2 to rewrite it under the new one, then delete the fallback. Verify by fetching the object and confirming it is not JSON.

The healthchecks.io ping URLs

One field per node on the healthchecks item, named for the host. The URL is the credential: anyone holding it can ping the check.

Blast radius: that node’s watchdog stops reporting. The node itself is unaffected, and the check goes red after its grace period, which reads as a node fault and is not one.

1. In healthchecks.io, regenerate the check’s ping URL.

2. Update the matching field on the healthchecks item.

3. Apply it to the node:

   just ans setup <host>
   

4. Verify the check reports within its 2 minute period and returns to green.

brokkr’s secrets

Ten fields across three items, all of them read by ansible/roles/forge and none of them by anything in the cluster. They are grouped here because they share one rotation step and one caveat.

The caveat, and it applies to every row below: brokkr holds no secret-store credential, so nothing there reconciles a rotated value. It reaches the node only when Ansible runs. The git reconciler on the node will not do it, and neither will Flux. See The standalone Podman plane.

The common step, after updating any field:

just ans setup brokkr --tags podman

That rewrites the env files, restarts the containers that read them, and re-runs forgejo admin auth update-oauth, which is what makes a new OIDC client secret reach the running Forgejo rather than sitting in a file it already read.

Four of them need something first.

oidc client id and oidc client secret. Replace the Pocket ID client, then read the new pair out of the module. The client ID changes too, because Pocket ID mints it and the provider only honours an explicit one at create time:

just tf apply oidc -replace=pocketid_client.forgejo
just tf output oidc -raw forgejo_oidc_client_id
just tf output oidc -raw forgejo_oidc_client_secret

forge client id and forge client secret. No plane in this repository can mint these: Forgejo does. In Forgejo, Settings → Applications, edit the Woodpecker application and regenerate the secret. Existing Woodpecker sessions survive; a new login fails until the node is re-converged.

repository password. This cannot be rotated in place. restic key add adds a second key to the repository rather than replacing the first, so rotating it means adding the new key, updating the field, re-converging, then restic key remove for the old one. Losing it with no copy means losing every forge snapshot, exactly as for the cluster’s. See What cannot be rotated.

key id and application key. Same destroy-then-create semantics as K8up’s, so backups fail between the apply and the re-converge:

just tf apply b2 -replace=b2_application_key.brokkr
just tf output b2 -raw brokkr_b2_key_id
just tf output b2 -raw brokkr_b2_application_key

Verify, after the re-converge:

ssh brokkr systemctl start futhark-forge-backup.service
ssh brokkr journalctl -u futhark-forge-backup -n 20

Expect a snapshot id in the log. Then delete the old Backblaze key.

The Storage Box SSH key

The private half is in the storagebox item and in STORAGEBOX_KEY_PEM in Infisical /infra/csi-rclone, from where infra/storage/app/secret.yaml templates it in as key_pem, escaping its newlines on the way. Both move together. Store the key file as it is; nothing about it is reformatted by hand.

Blast radius: every storagebox-crypt mount fails at authentication. Pods with an existing mount keep it until they are rescheduled.

1. Generate the new keypair and install the public half, which covers both ports:

   ssh-keygen -t ed25519 -f ~/.ssh/futhark-storagebox-new -N '' -C futhark-csi
   cat ~/.ssh/futhark-storagebox-new.pub | ssh -p23 uXXXXX-subN@uXXXXX-subN.your-storagebox.de install-ssh-key
   

2. Verify the new key on its own before changing anything:

   sftp -P 23 -i ~/.ssh/futhark-storagebox-new uXXXXX-subN@uXXXXX-subN.your-storagebox.de
   

3. Update STORAGEBOX_KEY_PEM under /infra/csi-rclone with the contents of ~/.ssh/futhark-storagebox-new, verbatim, and the ssh key field of the storagebox item. Nothing else in the config changes, and no other credential is touched.

4. Confirm the operator synced the change through, and that key_pem came out as one line with \n between the key’s lines:

   kubectl -n csi-rclone get secret storagebox-secret -o jsonpath='{.data.configData}' | base64 -d
   

5. Verify a real mount. Create a PVC against storagebox-crypt and attach a pod to it. Binding alone proves provisioning, not mounting. The procedure is in The rclone remotes.

6. Remove the old public key from the subaccount’s authorized_keys, then repeat step 5.

Why key_pem and not key_file: the CSI driver mounts the config Secret and nothing else, so a filesystem path in the config does not exist inside the driver container.

Infisical, per app

The backup B2 key

The only credential in the backup path that can be replaced at all. tofu/b2 mints it.

Blast radius: backups stop. Every K8up job fails, which the “K8up job failed” alert in infra/monitoring/app/grafana/alerting/backup.yaml catches within the hour.

Both capabilities and bucket_ids force replacement, so rotation is the create loop, forced:

1. Mint the replacement. This destroys the old key first — b2_application_key.k8up has no create_before_destroy, so from here until step 3 lands no valid key exists and backups fail. There is no rollback past this point, because the old key is already gone:

   just tf apply b2 -replace=b2_application_key.k8up
   

2. Read the two sensitive outputs:

   just tf output b2 -raw k8up_b2_key_id           # -> B2_KEY_ID
   just tf output b2 -raw k8up_b2_application_key  # -> B2_APPLICATION_KEY
   

3. File both into Infisical /infra/k8up.

4. Verify with a real backup, not with a plan:

   just bak now monitoring
   just bak jobs
   

   The new job must reach Complete.

5. Confirm the old key is gone, rather than revoking it by hand — the apply in step 1 deleted it:

   b2 key list --long    # one `k8up` entry, the new id from step 2
   

No rollback. The old key is destroyed in step 1, before anything has been verified. If step 4 fails you go forward, not back: re-read the outputs and re-file them. Giving the resource a create_before_destroy lifecycle would make a rollback possible and close the outage window, at the cost of two live keys mid-rotation.

The Bifrost virtual keys

VK_OPEN_WEBUI, VK_CLI, VK_VANE, VK_KVASIR, and the consumer-side copy of three of them: OPENAI_API_KEYS for Open WebUI and OPENAI_API_KEY for each of Vane and Kvasir. The generic loop below does not apply: these are not edited in Infisical, and each pair lives in two folders that must agree. tofu/bifrost owns all seven. The procedure, including the 401 window between the two restarts and the extra Settings edit Vane needs, is in bifrost.

BIFROST_ENCRYPTION_KEY is different again. It encrypts the provider keys stored in config.db, so replacing it makes every stored row unreadable and Bifrost starts from an empty config store. Almost everything in there is reapplied from nodes/kenaz.k8s/bifrost/app/config.json on the next start, because source_of_truth is config.json. What is genuinely lost is the per-virtual-key usage the store has accumulated. Rotate it by updating Infisical, deleting the PVC’s contents, and restarting, in that order.

Everything else under /infra and /nodes

The generic loop for any per-app runtime secret, such as Grafana’s ADMIN_PASSWORD or MAXMIND_LICENSE_KEY:

1. Change the value at the provider, if it has one.

2. Update it in Infisical at its path.

3. Wait for the owning InfisicalStaticSecret’s refreshInterval, then confirm the Kubernetes Secret changed:

   kubectl -n <namespace> get secret <name> -o jsonpath='{.data.<KEY>}' | base64 -d
   

4. Restart the consumer if it reads its configuration only at startup:

   just ks restart <namespace> <deployment>
   

5. Verify the app works, then revoke the old value at the provider.

A PostgreSQL role password

The only values in the tree filed in two Infisical folders each. /infra/postgres is where CloudNativePG reads the role’s password; the app’s own folder is where it is assembled into a connection string. An InfisicalStaticSecret may only name a path inside its namespace’s own tier, and the two namespaces never share a folder even when both are infra tier, so neither can read the other’s copy. Changing one of the two leaves the app unable to log in.

One row per tenant. The second column is the key in /infra/postgres; the third is the key in the app’s own folder, and where it ends up:

Grafana is the one that does not assemble a URL: grafana.ini names the host, database and user in git and reads only the password from the environment.

1. Generate a replacement from letters and digits only. Every one but Grafana’s is interpolated into a connection URL, and anything needing percent-encoding produces a string that parses wrong.

2. Set it at both paths from the row above.

3. Within a refresh interval, CloudNativePG picks the new password up from the reloaded basic-auth Secret and applies it to the role. The app’s own Secret is rewritten on the same interval.

4. Restart the consumer, which read its connection details at startup:

   just ks restart <namespace> <deployment>
   

5. Verify the app still reads its own data. There is nothing to revoke: the old password stops working the moment the role is altered.

Rotating Pocket ID’s is the one worth scheduling rather than doing casually. Its restart is a cluster-wide login outage for as long as the pod takes to come back.

Which paths exist and who reads them is in Cold bootstrap. The authoritative list is the tree:

grep -rl 'kind: InfisicalStaticSecret' infra nodes

What cannot be rotated

Three values. Each one is unrotatable for a different reason, and in two cases the data is lost with the key.

RESTIC_PASSWORD. Baked into the restic repository when K8up’s first backup job initialises it, and unchangeable afterwards without starting a new repository. A new repository means the existing backups stay readable only with the old password. See Backup and recovery.

The same holds for brokkr-restic’s repository password, with one difference worth knowing: the node’s repository is initialised by restic init from Ansible rather than by an operator inside a job, so restic key add and restic key remove are available to add a second key and drop the first. That changes the password without abandoning the repository, but it is a two-step rotation against live data rather than a field edit. See brokkr’s secrets.

The four rclone-crypt passwords. password and password2 on each crypt remote derive the keys that wrapped the data. Change one and the data it wrapped is unreadable. They are stored obscured rather than plaintext, so treat the obscured string as the value.

The Google Drive OAuth refresh token. Rotatable at Google, but the cluster cannot accept a rotated one: the CSI driver mounts the config Secret read-only, so rclone cannot write a refreshed token back. A stable refresh token is a hard requirement, which is why the OAuth client must stay at publishing status In production. Left in Testing, Google issues tokens that expire after 7 days and the mount fails silently on the eighth. See The rclone remotes.

Losing either of the first two loses the data it protects, with no recovery path. That is the same property that keeps the provider from reading it. The restic password has a copy in Proton Pass for exactly this reason, and that copy is the one deliberate duplication in the secrets scheme.

Operator offboarding

Rotate in this order, so no step locks you out of the next one:

1. Every Proton Pass item the person could read, using the procedures above.
2. The Flux deploy key.
3. The cluster age key.
4. The admin SSH key.
5. The GPG primary key, last, because every other step needs a working sops.
6. Remove the person’s Proton Pass vault access and their NetBird peers from the admin group.

Confirm nothing still points at a decommissioned store:

grep -rn 'pass://' --exclude-dir=.git .

CLI proxy login

Seed an OAuth credential into cli-proxy-api so bifrost can serve a subscription model, and prove the model answers through the gateway.

This is the one credential in the cluster no apply can create. Every provider cli-proxy-api fronts is a CLI subscription rather than an API key, and the only way to obtain a token is the vendor’s browser flow. The result is a file under auth-dir, on the PVC, refreshed in place while the pod runs.

Run this once per provider account, and again after a restore that predates an account being added.

Prerequisites

• The generated kubeconfig at ansible/.generated/kubeconfig. Every command below goes through a just recipe, which exports KUBECONFIG itself, so nothing here depends on the shell’s own cluster context. just ks kctl is a plain kubectl passthrough for the steps no recipe covers.

• A browser on this machine, signed in to the account being linked.

• nodes/kenaz.k8s/cli-proxy-api already reconciled:

  just fx get
  

  Verify: the cli-proxy-api Kustomization reads Ready.

Why the Deployment is scaled down first

auth-dir is a single-writer volume on a ReadWriteOnce PVC, and the running pod holds it. The login pod mounts the same claim, so the Deployment has to let go of it first. Scaling to 0 also takes bifrost’s cli-proxy provider offline for the duration; requests to ollama are unaffected.

That is the reason this is a recipe rather than a list of commands. A flow abandoned halfway still has to put the replica back, so the cleanup runs from a trap on every exit path.

Procedure

just ks cli-proxy-login claude

The provider argument is claude, codex, or antigravity, and defaults to claude. Each vendor fixes its own OAuth callback port, so the recipe forwards the right one per provider.

The recipe scales the Deployment to 0, starts a login pod on the same volume with the app’s ConfigMap mounted, forwards the callback port, and follows the pod’s log. It reads the image from the Deployment, so the login always runs the build the server runs.

1. Wait for the authorization URL to appear in the log, then open it and sign in. The vendor redirects to http://localhost:<port>/..., the forward carries it to the pod, and the pod writes the token.

2. Press Ctrl-C once the log reports the login succeeded. The recipe deletes the login pod, scales the Deployment back to 1, and waits for the rollout.

Verify: one pod, Running and ready.

just ks pods cli-proxy-api

Verify end to end

From a host on the mesh, ask the gateway what it can serve. Read <virtual key> with just tf output bifrost -raw vk_cli. See bifrost.

curl -sS https://llm.$SUB_INTERNAL.$DOMAIN/v1/models \
  -H "Authorization: Bearer <virtual key>" | jq -r '.data[].id'

Success is at least one cli-proxy/ model in the list. If only ollama/ models appear, the token did not land: cli-proxy-api starts and answers /healthz with an empty auth-dir, so a healthy pod is not evidence of a credential.

Then send one request through it:

curl -sS https://llm.$SUB_INTERNAL.$DOMAIN/anthropic/v1/messages \
  -H "x-api-key: <virtual key>" \
  -H "content-type: application/json" \
  -d '{"model":"cli-proxy/<model id>","max_tokens":16,"messages":[{"role":"user","content":"ping"}]}'

Success is a content array in the response rather than an error object.

If the login fails partway

Nothing is written until the token file is, so a failed attempt leaves no state to undo. Press Ctrl-C and run the recipe again.

If the recipe was killed in a way that skipped its trap, SIGKILL rather than Ctrl-C, the login pod may still exist and the Deployment may still be at 0. Undo both by hand:

just ks kctl -n cli-proxy-api delete pod cli-proxy-login --ignore-not-found
just ks kctl -n cli-proxy-api scale deployment/cli-proxy-api --replicas=1

bifrost serves ollama either way, so nothing else in the cluster is waiting on this.

Recovery

The tokens are on cli-proxy-api-auth, which carries k8up.io/backup: "true" and is covered by the cli-proxy-api entry in infra/backup/config/schedules.yaml. Restoring the PVC restores every linked account. See Backup and recovery.

A restore is worth trying before repeating this procedure, because some vendors invalidate the previous token when a new login succeeds, and re-linking every account is the slower path.

Recipe reference

Every recipe in this repository, what it does, and which ones need a credential. Use it to find the command; the pages linked from each section explain the procedure around it.

Everything an operator runs goes through just. The root justfile is only a set of mod declarations. The real definitions are one file per namespace under .just/.

just help          # every recipe, namespace by namespace
just ks help       # one namespace's recipes

Arguments are positional: just ks logs media sonarr. A recipe documented as <x> requires that argument, and [<x>] takes an optional one.

ops, the operator machine

just itself is the one thing ops deps cannot install for you. It has to be there to run the recipe. sudo dnf install just first.

Every tool that is only a binary on PATH is pinned in mise.toml and installed with mise install; the dnf packages, their daemons and the uv tools are the rest of the recipe. The binaries land in ~/.local/share/mise/shims, which Cold bootstrap covers putting on PATH.

The tf init inside ops setup skips any module with a backend.tf while config/sops/ops.sops.yaml does not exist yet, and says so. That is the cold-bootstrap case: setup runs at step 3, the encrypted files land at step 5. Run just tf init <module> for it afterwards.

ops mesh only reports: netbird up is an interactive SSO login, and the admin group it has to land in is filled from the dashboard, not from tofu/netbird. ops deps installs the client either way.

ops age-key is not part of ops setup: it mints key material, so it is deliberately explicit. It prints the public recipient for .sops.yaml and leaves the private key in a temporary file for you to store in Proton Pass and shred.

ops sops takes either name, foo.sops.yaml or foo.sops.yaml.example, and always edits the real file. If it does not exist yet, the template is copied, opened, and encrypted on save. It fails closed: an aborted edit or a failed encrypt deletes the plaintext rather than leaving it at a *.sops.* path.

just ops sops                                 # what is still missing
just ops sops config/sops/ops.sops.yaml           # create it, or edit it

ops rekey re-seals every *.sops.* file to the recipients .sops.yaml currently names, and prints the resulting diff. Run it after editing .sops.yaml, since a rotation that changes the recipients there leaves the files themselves sealed to the old ones.

ans, hosts

setup and k8s both depend on render-secrets, so you rarely run it by hand.

setup is variadic after the hostname, so playbooks/setup.yml’s tags are reachable. Pass '' as the hostname to mean all of them:

just ans setup brokkr --tags podman        # one node's container plane
just ans setup '' --skip-tags podman       # the fleet's base layers only
just ans setup kenaz --tags mesh           # re-converge one node's mesh join

fx, Flux

ks, the cluster

Every ks, fx and bak recipe points KUBECONFIG at ansible/.generated/kubeconfig itself, so you do not need it in your environment. Re-converging the cluster is just ans k8s.

bak, backups

A size of 0 in bak snapshots is a run that copied nothing, which is otherwise indistinguishable from one that worked. bak ls shows what such a snapshot does and does not hold; bak logs is where the file and byte counts for a single run are.

bak restore deletes data. It prints which PVCs it will destroy and which it will leave alone, and requires you to type the namespace back before it proceeds. Only local-path PVCs are ever wiped, so the rclone-backed classes are never touched. It is deliberately not reachable from any other recipe, and restores the newest snapshot of each volume. Backup and recovery covers what it does behind that prompt, and why a hand-written Restore is not equivalent.

bak restore postgres does nothing useful: that namespace’s data is a pg_dumpall object rather than a volume snapshot, and K8up cannot restore anything it took from stdin. The two pg-* recipes are its equivalent, going through restic dump and split so the reading half is safe to run on its own. bak pg-restore prompts the same way, then suspends the tenants’ Flux Kustomizations, scales them to zero, replays the dump, and resumes them.

tf, the cloud plane

plan, apply and adopt need a Proton Pass session and the GPG smartcard plugged in, and so does init for a module that ships a backend.tf, because initialising a remote backend authenticates against it. There is no editing secrets from here. Secret values live in Proton Pass, and each module’s tofu.<module> section of config/sops/ops.sops.yaml holds only identifying values and the pass:// references that point at them. Edit it with just ops sops.

docs, this book

Checks and CI

What is checked automatically, what is not, and the manual checks that have no automation behind them. Read this before adding a hook or wondering why CI passed something it should have caught.

Pre-commit

Install the hooks once per clone:

just ops hooks

Run everything by hand:

pre-commit run --all-files

The hooks, from .pre-commit-config.yaml:

Five have a wrinkle worth knowing.

just-fmt checks each file as its own root. just --fmt formats one file at a time and does not follow mod, so the hook passes every justfile explicitly. It is still gated behind --unstable, and validate.yml installs the JUST_VERSION pin because the runner image ships no just.

sops-encrypted greps, it does not decrypt. It looks for an ENC[ marker and nothing else, so it needs no key and runs identically on a runner and on your laptop. It exists for one failure mode: writing a *.sops.yaml and committing it before running sops -e -i. gitleaks will not reliably catch that, because a node address or a domain matches no credential pattern, which is exactly the class of value those files hold. .gitleaks.toml allowlists the same paths, since SOPS ciphertext otherwise trips the entropy rules on every commit. The root .sops.yaml is excluded: it holds recipients, not ciphertext, and matches the filename pattern by accident.

tofu-validate deliberately does not run tofu init. init can touch .terraform.lock.hcl, and pre-commit treats a hook that modifies a tracked file as a failure. Run just tf init once locally before committing a .tf change, or validate fails on an uninitialized module. CI runs init as its own step instead, with -backend=false, which is also why a module with a remote backend needs no credentials in CI.

ansible-lint needs always_run: false set explicitly. Upstream’s own hook manifest sets always_run: true, which overrides the files: ^ansible/ scoping. Without the override it runs on every commit regardless of what changed.

The gitleaks hook only scans the staged index (--staged), which is empty under pre-commit run --all-files. Locally that is the right behaviour. In CI it silently scans nothing, which is why validate.yml runs a separate full-tree gitleaks dir ..

CI

Five workflows, all in .github/workflows/.

validate.yml runs on every pull request and every push to master, with cancel-in-progress concurrency. Three jobs:

• pre-commit installs kustomize and helm explicitly, since neither ships on the runner image and both are language: system hooks. It then runs tofu init -backend=false per module, pre-commit run --all-files, and the full-tree gitleaks scan described above.
• renovate-config runs renovate-config-validator --strict. A malformed renovate.json5 is otherwise silent: Renovate skips the repository at 03:00 and updates just stop arriving.
• ansible-syntax installs the galaxy collections and runs ansible-playbook --syntax-check playbooks/*.yml.

docs.yml builds this book with mdbook on every pull request and push, and deploys it to GitHub Pages only from master. It then asserts that every ```d2 fence in docs/src became a rendered diagram. That check is not redundant: if d2 or mdbook-d2 is missing, mdbook leaves the fence as a code block and still exits 0, so the diagrams would silently stop shipping.

trivy.yml scans container images for known vulnerabilities on every pull request touching infra/, nodes/ or flux/, weekly on a cron, and on demand. It is described in Vulnerability scanning below.

renovate.yml runs Renovate at 03:00, on a change to its own config, and on demand. That is an hour before the mirror cron, so an automerged update reaches Codeberg the same night. What it covers and what it may merge is Dependency updates.

mirror.yml pushes a full mirror to Codeberg on every push to master, daily on a cron, and on demand. It is disaster recovery, not a second remote you push to. It uses ssh-keyscan for the host key, which is trust-on-first-use, because Codeberg publishes no authenticated equivalent to GitHub’s api.github.com/meta. The blast radius is this mirror push only, not the live Flux deploy-key channel.

All five workflows check out with persist-credentials: false, and every tool they install is pinned in mise.toml, the same file just ops deps reads. A local just docs build therefore renders with the mdbook CI publishes, and pre-commit runs the kustomize CI installs. Neither side constructs a download URL: validate.yml and docs.yml run jdx/mise-action, just ops deps runs mise install, and both get the same checksummed binaries.

CI holds no decryption key and must never need one. Nothing above decrypts: kustomize build parses SOPS output fine, because SOPS encrypts values and leaves keys alone, and sops-encrypted only greps. ansible-playbook --syntax-check parses playbooks without templating inventory variables, so the file lookups in group_vars/all/ are never evaluated and the missing ansible/.generated/ files do not fail it. Keep it that way rather than putting a key into GitHub Actions.

Vulnerability scanning

Trivy runs in two places, answering two different questions.

What a commit would run

.github/workflows/trivy.yml renders the manifests, collects every image reference, and scans each one. It renders twice: kustomize build for what this repository writes, then helm template per HelmRelease for what the charts bring with them, and it adds Flux’s own controllers, whose tags flux/cluster.yaml does not name directly. Findings land in this repository’s Security tab under Code scanning, one entry per image repository, and a pull request gets a comment linking to its own results.

The entry is keyed on the repository rather than on the full reference so that it follows an image across a tag bump: the next scan of that repository supersedes the previous one and closes what the bump fixed. Keyed on the tag, the old findings would stay open under a name nothing scans again.

Rendering rather than diffing is deliberate. A diff shows the one image a commit renamed, and the question is what the cluster would then be running. Rendering is also the only view that reaches the chart-internal images (Traefik, Grafana, cert-manager, K8up, VictoriaMetrics), which Version pins does not digest-pin and no commit here ever names.

The scan never fails the check. A HIGH in an upstream base image is not something a pull request against this repository can fix, and a check nobody can turn green is a check people learn to click past.

To reproduce one image locally before opening a pull request:

just sec scan ghcr.io/open-webui/open-webui:v0.11.0

To scan an image the workflow does not discover, run it on demand:

gh workflow run trivy.yml -f image=docker.io/library/nginx:1.29

The run logs discovered N images before the matrix expands. A count that dropped without a manifest change means a helm template call started failing, which the same log records as a ::warning:: naming the file and chart.

What the cluster is running now

infra/trivy-operator watches workloads and writes a report per image, plus config audits, RBAC assessments, node hardening checks and CIS/NSA compliance. There is no web interface. The reports are Kubernetes objects, and just sec reads them:

just sec           # the recipes in this module
just sec reports   # vulnerability counts per workload, worst first
just sec compliance

Start at just sec reports, then open the row that stands out. The report name is <kind>-<hash>, which is why the recipe prints the workload name alongside it:

just sec show replicaset-open-webui-7d9c4f8b5 open-webui

Expect an empty list for the first few minutes after install, while the operator downloads its vulnerability database. kubectl -n trivy-system get jobs shows the scans as they run, one at a time.

This is not a duplicate of CI. A merged commit stops describing reality the moment a CVE is published against an image that has not changed, and the operator is the only thing here that notices.

To confirm a finding is gone after merging the bump, force the reports to be rebuilt rather than waiting for the operator’s own interval:

just sec rescan open-webui

Three alerts reach Slack, from infra/monitoring/app/grafana/alerting/security.yaml: a fixable CRITICAL in any workload, a cluster-wide CRITICAL and HIGH count above its 24-hour floor, and a compliance spec failing more controls than its 7-day baseline. The last two are regression alarms rather than presence checks, because CIS fails controls on a single-node k3s that no change here will satisfy, and an alert that is always firing gets muted. The Glance cluster page carries the same counts as a widget.

Suppressing a finding

.trivyignore at the repository root is the only place a suppression is written. All three scanners read that one file: .github/workflows/trivy.yml passes it to the action, just sec scan passes it to the CLI, and infra/trivy-operator/app/kustomization.yaml copies it into the operator’s HelmRelease values as it builds. The copy is a kustomize replacement rather than a valuesFrom reference, because Flux resolves a valuesFrom key through Helm’s --set parser, which reads a comma in a comment as a value separator and fails the release.

An entry belongs there only when the scanner cannot decide the finding, such as an advisory it cannot match against the installed version. A finding that is merely inconvenient does not qualify: suppress it and the alerts in infra/monitoring/app/grafana/alerting/security.yaml go quiet without anything being fixed. Give every entry a comment saying which image it covers and why the scanner is wrong, because nothing else records it.

Write one ID per line. The chart’s own trivy.ignoreFile list syntax is not used, and should not be: it renders each ID as a YAML list item, which the scanner then reads as the literal -.

An edit reaches the cluster on the next Flux reconcile, but existing reports are not rebuilt for it. Run just sec rescan <namespace> to see the effect.

What neither scanner covers

Both run with ignore-unfixed, so a vulnerability with no published fix never reaches you. That is the intent: it leaves nothing to merge. It also means every finding that does arrive corresponds to a Renovate pull request worth merging out of schedule.

Neither scanner reads this repository’s own tree for misconfiguration. gitleaks and GitGuardian cover secrets, and the operator’s config-audit scanner covers workload hardening from the cluster side, which is where the objects Helm generates are visible at all.

Infisical tier isolation

Two manual checks. Neither is automated, and both are worth re-running after any change to the operator HelmReleases or the admission policy. They are the only evidence the tier boundary is real rather than merely intended. Run them once at the end of Cold bootstrap, and delete the objects afterwards.

RBAC

Create an InfisicalStaticSecret in actual, a futk.eu/tier: node namespace, whose targets[0].namespace is monitoring. The object is admitted, because the admission policy below pins targets to the object’s own namespace and this one violates that, so admission catches it first.

To test RBAC on its own, add a second namespace to the node tier’s scopedNamespaces, point a target at it, and confirm the operator writes there. Then remove it and confirm the write starts failing with a forbidden error in the operator’s logs.

Admission

In actual, create one with sources[0].secretPath: /infra/cert-manager:

kubectl apply -f - <<'EOF'
apiVersion: secrets.infisical.com/v1beta1
kind: InfisicalStaticSecret
metadata:
  name: isolation-probe
  namespace: actual
spec:
  infisicalAuthRef: {name: infisical, namespace: infisical-node-kenaz}
  sources:
    # Spelled out because this bypasses kustomize; a committed manifest gets both
    # from config/infisical instead.
    - projectSlug: futharkd
      environmentSlug: prod
      secretPath: /infra/cert-manager
  targets:
    - kind: Secret
      name: isolation-probe
      namespace: actual
      creationPolicy: Owner
  syncOptions:
    refreshInterval: 1m
EOF

The API server must reject this at admission with secretPath must lie within the namespace's own tier. If it is created instead, the ValidatingAdmissionPolicyBinding is not selecting the namespace. Check that actual still carries futk.eu/tier and futk.eu/node.

Clean up:

kubectl -n actual delete infisicalstaticsecret isolation-probe --ignore-not-found

The per-node mesh checks

One healthchecks.io check per mesh node, pinged directly by that node’s watchdog rather than by anything in the cluster. See The mesh watchdog. This is the only mesh-health signal that does not travel over the mesh, which is the point: everything scraped into Grafana goes dark exactly when the thing being watched breaks.

Read a red check as “this node has been unhealthy for longer than the grace period, and its own repairs have not fixed it yet”. The /log events on the check’s timeline say which rungs fired. A /fail means the ladder is exhausted and the node needs re-enrolling by hand.

This is distinct from the cluster-wide dead-man’s switch in infra/monitoring/app/grafana/alerting/watchdog.yaml, which pings from inside Grafana and so reports on the cluster, not on the mesh underneath it.

NetBird token expiry

NetBird Personal Access Tokens expire, 365 days out at most. Three are in use, one on each of the three service users:

None takes the mesh down when it lapses. Peers keep their configuration and keep connecting. What stops is changing anything: no policy applies, and no new node joins. The read-only one costs one widget and nothing else.

Nothing in this repository tracks the expiry dates. Put them in a calendar when you issue the tokens. The replacement procedure is Credential rotation.

The forge node’s own signals

brokkr is outside the cluster, so nothing above reaches it and it reports on itself. Both signals are node-exporter textfile metrics, written by the units that produce them and scraped over the mesh:

futhark_quadlet_revision_info carries the revision the node currently has applied, as a label, which is the fastest way to answer “did that commit land”. Read it with ssh brokkr git -C /var/lib/futhark-gitops rev-parse --short HEAD, or from the metric.

Two Gatus endpoints cover it from the cluster side, git.$DOMAIN and ci.$DOMAIN, in a Forge group of their own. The direction of dependency is deliberate: the cluster watches the fallback, so you learn the fallback is gone before you need it. Their [CERTIFICATE_EXPIRATION] assertion does more work than anywhere else on that page, because these two certificates are renewed by a Traefik process on a machine nothing else watches, not by cert-manager. See The standalone Podman plane.

What has no check at all

Worth stating, because the absence is easy to mistake for coverage:

• NetBird policy. There are no server-side policy tests, so a wrong rule applies cleanly and fails later, in traffic. The isolating test is the substitute.
• The forge’s Quadlet units. kustomize-build never sees them, because they are systemd unit files rather than manifests and their directory holds no kustomization.yaml. Nothing validates them before they reach the node, so a broken unit is caught by the reconciler failing on the node, not by a hook. Recovery is another commit; see Changing what runs.
• The break-glass admin. Nothing proves it still works, and it is the one credential whose whole purpose is being available during an outage. Test it deliberately, by logging in to git.$DOMAIN with Pocket ID scaled to zero.
• Credential expiry. Nothing watches it. See above.
• Backups that copied nothing. A malformed exclude annotation makes K8up skip that PVC and the job still succeeds. The size column in just bak snapshots is the check, and the 26h alert in infra/monitoring/app/grafana/alerting/backup.yaml is the backstop only for a namespace that stopped entirely. See Backup and recovery.

Troubleshooting

Failure shapes that have actually happened here, what each one really means, and how to confirm it. Work top-down: the first three sections share one root cause and account for the worst outage this cluster has had.

Start here:

just ks status     # nodes, unhealthy pods, Flux sync state
just fx failing    # only what isn't Ready

Then narrow down:

just ks describe <ns> <pod>    # events, usually says why it won't schedule or mount
just ks previous <ns> <pod>    # a crashlooper's last words
just ks warnings               # Warning events cluster-wide
just fx logs                   # kustomize-controller, or name another

Half of everything times out, intermittently

Symptom: some DNS lookups resolve and some hang; anything that resolves at startup crashloops; direct-to-pod-IP works. Cross-node pod-to-pod traffic is being dropped.

CoreDNS runs one pod per node and kube-proxy load-balances 50/50, so exactly the remote endpoint is unreachable and the failure looks intermittent rather than total.

# from a pod, against a pod on the *other* node
ping -c2 <remote pod IP>

If that fails, the mesh policy is almost certainly not passing every protocol, which the IPIP overlay needs. Full explanation and the fix: Pod to mesh networking.

This is the one that took the cluster down and presented as a storage fault three layers away. Check it before believing a cross-node bug is node-local.

no route to host from a pod dialing a node

This is a source-address fault, not a routing one, despite what the message says. The WireGuard peer drops packets whose source is not the node’s own mesh address, so pod-sourced packets die on egress even when the route is correct.

Isolate it on the node:

ping -c2 -I <this node's mesh IP> <peer mesh IP>        # should succeed
ping -c2 -I <this node's pod-bridge IP> <peer mesh IP>  # fails without the SNAT rule

Then check the rules are actually installed, and at a priority that is not being shadowed:

ip rule
systemctl status futhark-mesh-routes
iptables -t nat -L POSTROUTING -n | grep netbird0

See Pod to mesh networking. If ip rule shows kube-router at a priority below 10, the whole script is a silent no-op.

kubectl exec, logs and port-forward are down cluster-wide

Same root cause as the two above. konnectivity-agent dials the controller’s konnectivity-server over the mesh, so it is the first thing to break when pod → mesh networking is broken, and it takes the whole exec path with it.

Flux can’t decrypt a manifest

kustomize-controller reporting failed to decrypt or an unparseable manifest means the sops-age Secret is missing, holds the wrong key, or the file was encrypted to a recipient the cluster key is not among.

kubectl get secret sops-age -n flux-system
sops -d <the file>          # works for you via the GPG card, independently of the cluster key

If sops -d succeeds locally but Flux fails, the file was sealed without the cluster age recipient. Compare the recipient the file carries against the one the rule names:

grep recipient: config/sops/cluster.sops.yaml
grep age: .sops.yaml

Two different age1… values mean the key was rotated but the file was never re-sealed. Fix it with just ops rekey, then commit and push: Flux only sees the pushed file. config/sops/ops.sops.yaml is supposed to fail this way in cluster context; it is deliberately not sealed to the cluster key, and Flux is never given it.

An InfisicalStaticSecret never syncs

kubectl get infisicalstaticsecret -A
kubectl describe infisicalstaticsecret -n <ns> <name>
kubectl logs -n infisical-<tier> deploy/... -f

Check in this order:

1. Is the namespace in the right tier’s scopedNamespaces? If not, the operator has no RBAC there and will not act at all. This is the most common cause after adding an app, and it looks like nothing happening rather than an error.
2. Is infisicalAuthRef pointing at that same tier’s namespace? A node app must reference infisical-node-<hostname>, not infisical-infra. The node operator cannot read the infra tier’s InfisicalAuth.
3. Was the object rejected at admission? A secretPath outside the namespace’s tier never gets created, and kubectl apply fails loudly, so check your shell history rather than the cluster.
4. Does the path actually hold the secret? Check the Infisical UI at that folder and environment.

A Kustomization is stuck, and its dependency is fine

Check for a namespace that does not exist yet. Every overlay under infra/policies/ sets kustomize’s top-level namespace: field, so the whole overlay fails to apply if the target Namespace is missing, as does any chart that writes into a namespace it does not create, which is how infisical-operator fails when scopedNamespaces names one. Add it to infra/namespaces/app/namespaces.yaml; nothing else declares namespaces. See Startup ordering.

A HelmRelease is Ready but running the old thing

flux reconcile will not reinstall a release whose chart version has not changed. Force it:

just fx redeploy <name>

A certificate never issues

just ks certs
kubectl describe certificaterequest -n <ns> <name>

Issuance goes through Let’s Encrypt DNS-01 against the Bunny zone, so it waits on DNS propagation and can legitimately take minutes. If it never completes, check the cert-manager-config Kustomization is Ready. The ClusterIssuer lives there, behind infisical-operator-config, because the webhook’s API key is an InfisicalStaticSecret.

just tf apply netbird returns 403 on netbird_account_settings

The netbird-policy service user is at Network Admin, which reads account settings and cannot write them. Promote it to Admin for the apply, then demote it again: Applying account settings.

A just tf plan returns 401, or Ansible fails to mint a setup key

The credential expired or was revoked. Which token, and what each lapse breaks, is NetBird token expiry. Replacements are in Credential rotation.

If instead a pass:// reference comes through as a literal string, the Proton Pass session is gone rather than the credential. pass-cli info says which.

tofu validate fails in pre-commit on a module you didn’t touch

The module was never initialized locally. The hook deliberately does not run init:

just tf init

See Checks and CI.

A commit changed nodes/brokkr.podman/ and nothing happened

That node is not reconciled by Flux, so just fx failing will never mention it and neither will anything else in the cluster. It polls this repository itself, every five minutes, and the only place the outcome is recorded is the node:

ssh brokkr systemctl status futhark-quadlet.timer
ssh brokkr journalctl -u futhark-quadlet -n 30
ssh brokkr git -C /var/lib/futhark-gitops rev-parse --short HEAD

Three causes, in the order they happen. The commit is not pushed, since the node clones the remote and not your working tree, exactly as Flux does. The branch was rewritten, which the reconciler refuses with a merge --ff-only failure rather than resolving. The reconcile failed, in which case the containers keep serving the previous revision and the only signal is futhark_quadlet_last_run_success at 0.

Recovery is another commit. Nothing on the node keeps a previous revision to roll back to, which is the same contract Flux has. See The standalone Podman plane.

A container on brokkr fails with Failed to load environment files

The EnvironmentFile that ansible/roles/forge writes is missing. Config on that node reconciles from git; secrets do not, and never will. Push them:

just ans setup brokkr --tags podman

The same command is the fix for a rotated secret that has not taken effect, and for The requested OAuth 2.0 Client does not exist on the Pocket ID login button, which means Forgejo holds a client id that tofu/oidc has since replaced. That run re-runs forgejo admin auth update-oauth as well as rewriting the files.

brokkr serves a self-signed or expired certificate

Traefik on that node issues its own over ACME TLS-ALPN-01, which is answered on 443 and requires the challenge to reach the node directly. Anything terminating TLS in front of it breaks issuance silently.

ssh brokkr journalctl -u traefik -n 50 | grep -i acme

Check git.$DOMAIN and ci.$DOMAIN still resolve to that node’s address and not the edge node’s; just tf plan bunny being a no-op confirms the records are what the repository says. cert-manager is not involved here at any point, so nothing in just ks certs is relevant.

Layout and naming

Rules for the GitOps tree: flux/, infra/ and nodes/. Follow them when adding a component or a node app, so kustomize build passes and Flux reconciles in the right order.

File names

Only two file names exist for Kubernetes YAML in this repo:

• ks.yaml, a Flux Kustomization CR, one per directory, named for what it reconciles.
• kustomization.yaml, a plain kustomize resource list, which is the name kustomize build itself requires.

If you are naming a Kubernetes YAML file something else, you are naming it wrong.

Directory layout

• infra/<component>/{ks.yaml, app/}: one Flux Kustomization per component.
• infra/<component>/{config-ks.yaml, config/}: only when that component’s CRs need CRDs its own ks.yaml cannot guarantee exist yet. This is a chicken-and-egg on first apply, and it is why cert-manager’s ClusterIssuer and infisical-operator’s InfisicalConnection each sit behind a second Kustomization.
• infra/<component>/app/<workload>/: subdivide app/ when a component reconciles several distinct workloads. One directory per workload, each a plain kustomization.yaml resource list. The component still has exactly one Flux Kustomization, one dependsOn set and one postBuild. monitoring is the case here, with five workloads that start together and share a namespace but are read and edited one at a time. Anything genuinely shared by all of them, such as its HelmRepository list, stays flat in app/. Each subdirectory has to kustomize build on its own, because pre-commit builds every directory holding a kustomization.yaml. That is also why a reusable Component cannot live under infra/. config/ is where those go.
• nodes/<hostname>.k8s/<app>/{ks.yaml, app/}: one directory per node app. See Node apps.
• nodes/<hostname>.podman/{units/, config/}: a node that runs containers without Kubernetes. No ks.yaml and no kustomization.yaml, because Flux reconciles none of it: a timer on the node rsyncs units/ into /etc/containers/systemd/ and config/ into /etc/futhark/. The file-name rule above does not apply, since nothing in it is Kubernetes YAML. Add the directory to .sourceignore, not to nodes/kustomization.yaml. See The standalone Podman plane.

Cluster-wide infra that happens to be pinned to a specific node is not a nodes/ entry. Pocket ID runs only on ogma, and expresses that as a nodeSelector under infra/. nodes/ is for tenant workloads, not infra controllers.

Flux Kustomization boilerplate

interval, prune, sourceRef, and the flux-system namespace are shared by every Flux Kustomization CR, so they are patched in once, from infra/kustomization.yaml and nodes/kustomization.yaml. A ks.yaml itself declares only metadata.name, spec.path, and spec.dependsOn.

flux/infra/ks.yaml and flux/nodes/ks.yaml are the exception and keep their full spec. flux/ has no kustomization.yaml of its own to patch from, because Flux auto-generates one from flux/cluster.yaml’s sync.path: flux, and adding a real one would pull cluster.yaml itself into reconciliation.

Version pins

One rule, everywhere: nothing floats. Every chart version, image, provider constraint, collection and release binary names an exact version, so the commit is the record of what runs. A range is a version the repo cannot state.

• Container images pin tag@sha256:…. The tag stays for readability; the digest is what actually resolves. A tag alone can be repointed at a different binary, and Flux would never reconcile, because nothing it watches changed. This holds for a Quadlet Image= line too, where the reasoning is the same and the reconciler is a timer rather than Flux.
• Helm charts pin MAJOR.MINOR.PATCH. A chart patch is still a template change reaching the cluster, and under a MAJOR.MINOR.* range it arrived with no commit behind it.

Each chart pin carries a comment recording the chart-to-app mapping, such as # chart 41.0.2 -> Traefik v3.7.6, so what a bump changes is readable without opening the chart. Where a chart’s image tag defaults to .Chart.AppVersion, as csi-driver-rclone does, that mapping is the only place the app version appears at all.

Keeping this many exact pins current by hand is not the intent. Renovate opens the bumps. See Dependency updates.

Adding a node app

1. Create nodes/<hostname>.k8s/<app>/{ks.yaml, app/}. The ks.yaml needs dependsOn: [infra-policies], plus whatever the app itself needs, usually traefik-internal and infisical-operator-config. nodes/kenaz.k8s/actual/ks.yaml is the worked example.
2. Add the directory to the sibling kustomization.yaml’s resources:.
3. Add the namespace to infra/namespaces/app/namespaces.yaml, labeled futk.eu/tier: node and futk.eu/node: <hostname>. Then create infra/policies/namespaces/<app>/ with the default-deny, same-namespace and from-monitoring network policy templates.
4. Add the ingress-bridge template only if app/ ships an Ingress.
5. If app/ ships an InfisicalStaticSecret, add its namespace to the right tier’s scopedNamespaces in infra/infisical-operator/app/. The operator has no RBAC there otherwise.
6. Add the new overlay directory to infra/policies/kustomization.yaml.
7. If the app needs a database, it gets one on the shared PostgreSQL rather than one of its own. That is four more files and a dependsOn: postgres-config, all listed in Giving a service a database.

An app’s own configuration, whether an env list or a settings file, ships through configMapGenerator rather than a ConfigMap manifest. The generated name carries a content hash, so editing the file rolls the Deployment. A fixed name does not: the new values reach the cluster and the running pod keeps the ones it started with, silently, until something else restarts it. The generator also needs the overlay to set namespace:, because kustomize only rewrites a reference when both sides agree on one.

Verify: pre-commit run kustomize-build --all-files passes, then after pushing, just fx failing is empty and the app’s pods reach Running.

Namespaces

Where namespaces are declared, and what their labels decide. Read this before adding a component or a node app, because a namespace declared in the wrong place deadlocks and a namespace labelled wrongly reads the wrong tier’s secrets.

Every Namespace CR lives in infra/namespaces/app/namespaces.yaml, owned by the namespaces Kustomization, which depends on nothing. Components do not declare their own. A controller whose chart writes into a namespace it does not own would otherwise deadlock against the component that does. See Startup ordering.

What stays per-namespace is the policy attached to it: infra/policies/namespaces/<namespace>/ holds that namespace’s NetworkPolicy and RBAC, not its Namespace.

Every non-control-plane namespace carries futk.eu/tier: infra or futk.eu/tier: node, and node namespaces add futk.eu/node: <hostname>.

Those labels are load-bearing, not documentation. The ValidatingAdmissionPolicy in infra/infisical-operator/config/ reads them to decide which Infisical path a namespace may pull from, so a namespace with the wrong label reads the wrong tier’s secrets, and one with no label cannot host an InfisicalStaticSecret at all. See Secrets.

Network policy

Which templates every namespace composes, what they add up to, and the one path they cannot govern. Read this when adding a namespace, so its overlay allows exactly what the app needs.

Every non-control-plane namespace gets a default-deny baseline plus explicit opt-in bridges, assembled per namespace from shared templates in infra/policies/namespaces/_templates/:

What that composes to, for one namespace: a wall with named holes in it, and one path that goes around the wall entirely. Each edge is labelled with the template that opens it, minus the netpol- prefix every template name carries. Red is the traffic the baseline drops. The ingress-edge bridge is the loose one: traefik-edge runs on hostNetwork, so the rule that admits it matches the mesh CIDR rather than a pod identity.

Kubernetes has no cluster-wide NetworkPolicy, so this is one overlay per namespace rather than one file. Egress is left open everywhere: the secret operators call out to their APIs, cert-manager calls ACME, and apps call whatever they call. In a single-tenant homelab the risk that matters is inbound.

One thing the baseline cannot cover: both Traefiks run with hostNetwork: true, so each shares the node’s network namespace and CNI policy enforcement never sees its sockets. The ingress-edge and ingress-internal overlays exist and are correct, but they do not govern that traffic. What actually governs it is firewalld (ansible/roles/firewall_ingress, public zone limited to 443 and the hardened SSH port), the fact that the internal ingress binds a mesh address only peers can reach, and Traefik’s own rate limiting.

The same hostNetwork is why both netpol-allow-from-ingress-edge and netpol-allow-from-ingress-internal are an ipBlock and not a namespaceSelector: those connections arrive as the ingress node’s own address, so a selector rule never matches and every request through that ingress returns 502. The block is the whole mesh CIDR from ansible/inventory/group_vars/all/network.yml, not the node’s /32. ipBlock accepts only a literal, and a /32 meant substituting an address that had to be maintained by hand in a second place. The rules therefore trust every mesh peer, not only the ingress; what bounds that is the NetBird policy in tofu/netbird, which decides which peers reach the cluster at all.

The two templates now hold the same rule, and stay separate anyway. They are two decisions that happen to agree: which one a namespace composes still records which ingress it expects traffic from, and either can change without dragging the other with it.

Pod-to-pod across namespaces

The templates above cover the two directions that recur: monitoring scraping everything, and an ingress reaching one namespace. A pod in one namespace calling a pod in another is neither, and six callers do it today. Each namespace they reach admits them with its own file. Every one of these files names the port the pod listens on, not the port its Service publishes.

Glance reaches into two namespaces.

infra/policies/namespaces/monitoring/netpol-allow-from-glance.yaml admits Glance to vmsingle on 8428 and to vlsingle on 9428, because most widgets on Glance’s cluster and network pages are an API query against one of them, and Glance is an ordinary pod that no mesh ipBlock covers. infra/policies/namespaces/gatus/netpol-allow-from-glance.yaml does the same for the Gatus API on 8080, which is where the apps page gets service health.

Open WebUI is the second caller, and reaches into three namespaces.

infra/policies/namespaces/searxng/netpol-allow-from-open-webui.yaml admits it to SearXNG on 8080, which is where its web search runs its queries. It could have gone through search.$SUB_INTERNAL.$DOMAIN instead, but that host sits behind auth-sso and a pod carries no session cookie. infra/policies/namespaces/bifrost/netpol-allow-from-open-webui.yaml admits it to Bifrost on 8080, which is its only model backend. infra/policies/namespaces/kvasir/netpol-allow-from-open-webui.yaml admits it to Kvasir on 8080, which its pipe function calls. Kvasir does compose netpol-allow-from-ingress-internal, but only for its runs page, and that path is behind auth-sso; a pod carries no session cookie, so this file is how the work gets in.

Vane is the third, and reaches the same two namespaces Open WebUI does, for the same two reasons: searxng on 8080 for results, bifrost on 8080 for the model. That is two more files, named after the caller in each of those two overlays.

Kvasir is the fourth, and reaches those same two namespaces for those same two reasons. Two more files again, in the same two overlays.

Bifrost is the fifth caller, and reaches its two in-cluster backends. infra/policies/namespaces/cli-proxy-api/netpol-allow-from-bifrost.yaml admits it to cli-proxy-api on 8317, and infra/policies/namespaces/munin/netpol-allow-from-bifrost.yaml to munin on 11434. Those two matter more than the others: neither app has an Ingress and neither composes netpol-allow-from-ingress-internal, so with the default deny in place each of these holes is the whole of that app’s reachability. That is what lets cli-proxy-api’s config.yaml ship an empty api-keys list and munin run with no credential at all.

Each is a file in its own overlay rather than a template in _templates/: each names one namespace, and a further caller should get its own file rather than a selector wide enough to hide who reads what.

The alternative was to point Glance at metrics.$SUB_INTERNAL.$DOMAIN, which needs no policy at all since the request then arrives from traefik-internal. That routes every widget out to the mesh interface and back, and leaves the dependency written down nowhere.

Rate limiting

Every namespace with an Ingress also composes the middleware-ratelimit template, a Traefik Middleware at average: 100, burst: 200, per source IP. It is basic DoS protection, not a precise budget.

Composing the template alone does nothing. Traefik only applies a Middleware to routers that name it, so the Ingress must reference it explicitly:

annotations:
  traefik.ingress.kubernetes.io/router.middlewares: <namespace>-ratelimit@kubernetescrd

Same-namespace reference only. traefik-edge’s kubernetesCRD provider does not set allowCrossNamespace, unlike traefik-internal, so a shared cross-namespace Middleware would not resolve there.

Verify a new namespace’s policy the way Pod to mesh networking does: from a pod in another namespace, confirm the connection is refused, then confirm the intended bridge works.

Secrets

Which store a value belongs in, how each plane reads it, and why the cluster cannot reach the keys that rebuild it. Read this before adding any secret, and Adding a new secret is the checklist at the end.

This repository is public. Nothing in it is a credential in the clear, and nothing in it is an identifying value in the clear either: no real addresses, endpoints, account or key identifiers.

Three stores, chosen by what a value can do rather than by who consumes it. Only two of them hold credentials. SOPS is a file format rather than a service.

The dividing line: publishing a node’s IP would tie this repository to a machine, but the IP grants nothing on its own, so that is SOPS. The Flux deploy key or the cluster age key grants everything, so that is Proton Pass. A Grafana admin password is neither. It is one app’s operational secret, rotatable without touching anything else, so that is Infisical.

The cluster holds no Proton Pass credential at all. That is the whole tier boundary, and it rests on absence rather than on a console-side path grant that could be misconfigured or drift. A compromise of the cluster cannot reach the keys that rebuild it, because there is nothing in the cluster to reach them with.

Drawn out, with the reads the table above already covers left off, so that the one edge that matters is the only one competing for attention. The boundary is the arrow that is not there: the red one, barred rather than pointed. Green is Proton Pass, the tier everything else is rebuilt from. Amber is a third party this repository can write to but does not own. What the cluster gets, it gets seeded once by Ansible, never by holding a credential of its own.

SOPS

Everything encrypted lives in config/sops/, in two files. The split is the tier boundary and nothing else: what the cluster may read, and what it may not.

.sops.yaml at the repository root names both paths exactly. A *.sops.* file anywhere else matches no rule and fails to encrypt, which is the intended answer: adding a third file is a decision, not an accident.

The cluster key opens cluster.sops.yaml alone. A compromise of flux-system yields the values Flux already substitutes into running manifests and nothing more. See Why the operator store is separate.

The operator recipient is the GPG primary key’s full fingerprint. Primary, so rotating or renewing the encryption subkey never edits .sops.yaml, because gpg picks the current encryption-capable subkey itself. Full fingerprint rather than the 16-hex long ID, because 64-bit key IDs are collision-generatable and this file decides who can read every secret here. The encryption subkey lives on a smartcard, so every sops -d needs the token and its PIN.

The cluster recipient is a separate age keypair, generated by just ops age-key. It has to be separate: a card-resident private key cannot be exported, and Flux needs one mounted in-cluster. It is never a personal identity.

Each file is committed alongside a .example template showing its shape. The real file is created once, at bootstrap, and encrypted before it is committed. Neither carries comments: SOPS encrypts comments along with values, so anything written there is invisible until decrypted and rewrites the whole ciphertext on every edit. Explanation belongs in the template or on this page. SOPS encrypts values and leaves keys alone, so kustomize build still parses an encrypted manifest. Only Flux needs the key.

Three resolvers, one per plane:

• Ansible reads decrypted output rather than the encrypted file. just ans render-secrets writes ansible/.generated/{secrets,nodes,cluster}.yml, and the playbooks and inventory/group_vars/all/ load those. No lookup against SOPS appears in any task.
• OpenTofu extracts its module’s section into the environment. See Rules for every module.
• Flux uses spec.decryption on every Kustomization, patched in once via infra/kustomization.yaml and nodes/kustomization.yaml. flux/infra/ks.yaml and flux/nodes/ks.yaml state it in full, since flux/ has no kustomization of its own.

Rotating either recipient re-encrypts files rather than values. Both procedures are in Credential rotation.

Proton Pass

The crown-jewel tier, in one vault. Its name is yours to choose and is written down nowhere in this repository: every committed reference spells it <vault>, and pass-cli resolves against whichever vault your session holds. Naming it after the Infisical project keeps the two remote stores alike. It holds the Flux git deploy key, the cluster age private key, all three Infisical machine identities, the two NetBird PATs, the Bunny API key and the Pocket ID admin token.

It also holds the restic repository password that encrypts the backups. That is the one deliberate duplication in this scheme. It is a runtime secret, so K8up reads it from Infisical like everything else, but a copy lives here because losing access to Infisical must not also mean losing the ability to decrypt B2. There is no recovery path if both go, by construction. See Backup and recovery.

The admin SSH private key is not in it, and never will be. It is the operator’s own identity, it already lives in ~/.ssh on the machine doing the connecting, and Ansible never reads it. It authenticates with it, which SSH does on its own. Its public half stays in config/sops/ops.sops.yaml as ansible.admin.ssh_pubkey: identifying, granting nothing.

Nothing in the repository holds a Proton Pass credential either. What a new operator machine needs out of band is exactly two things, the GPG smartcard that opens SOPS and a Proton Pass personal access token:

pass-cli info || PROTON_PASS_PERSONAL_ACCESS_TOKEN=pst_… pass-cli login

Replace pst_… with the token from the Proton Pass web app. pass-cli persists a session after that, so it is a once-per-machine step and not a per-command prompt. just ops pass-session checks it and prints this if it is missing.

Two commands, and the brace rule that separates them

Both planes compose SOPS with Proton Pass, and both resolve SOPS first, pass-cli second. The committed file holds references, and only the second step turns them into values. They use different pass-cli subcommands, which take opposite syntax:

Ansible reads the ansible subtree of config/sops/ops.sops.yaml, which maps the crown jewels this plane needs to pass://<vault>/<item>/<field> references under a nested secrets: key. The tofu subtree is deliberately not read here: resolving it would write secrets to disk that Ansible never uses. just ans render-secrets does the work, and ans setup and ans k8s depend on it:

sops -d --extract '["ansible"]' config/sops/ops.sops.yaml | pass-cli inject -f -o ansible/.generated/secrets.yml

--extract yields that subtree’s own keys at the top level, so the result is admin: and secrets:. --out-file rather than a shell redirect, because it applies pass-cli’s default 0600 where > would use the umask. ansible/.generated/ is gitignored. The playbooks load the result with an explicit vars_files, and the roles then reference plain variables such as secrets.flux.deploy_key. Keep no_log: true on those tasks.

Rendering first is not optional. {{ }} is Jinja syntax as well as pass-cli’s, so Ansible loading the decrypted-but-uninjected document would try to evaluate pass://<vault>/flux/deploy key as an expression. Nothing under group_vars/ or host_vars/ is encrypted, and no vars plugin decrypts anything, precisely so no braces survive to be misread.

OpenTofu reads its own module’s section of the same file. Each tofu.<module> map carries that module’s credentials as bare pass:// URIs alongside its identifying values, so --output-type dotenv flattens the map into the environment and pass-cli run rewrites the URI-valued variables in place:

sops -d --output-type dotenv --extract '["tofu"]["bunny"]' config/sops/ops.sops.yaml

Getting the braces backwards fails quietly in both directions. An unresolved reference is passed through as a literal string, which surfaces as a bad credential rather than as a template error.

Because the committed files hold references and not values, updating a Proton Pass item in place rotates a credential with no commit at all. That is what makes most of Credential rotation cheap.

Multi-line values

A Proton Pass field that holds more than one line cannot be referenced directly. pass-cli inject splices the resolved value into the document as text, and the reference sits inside a double-quoted YAML scalar, so YAML folds the value’s newlines into single spaces. The reference resolves, the document parses, and the variable reaches the role as one line with its structure gone. An SSH private key mangled this way is seeded without error and surfaces later as ssh: no key found on the GitRepository.

Store such a value base64-encoded on a single line, and decode it at the point of use:

base64 -w0 < <file holding the value>

identity: "{{ secrets.flux.deploy_key | b64decode }}"

That is why the flux item’s deploy key field holds base64. Single-line values, including the cluster age key, need none of this.

Why the operator store is separate

Vault, item and field names are identifying, and this repository commits nothing identifying in the clear. More than that, config/sops/ops.sops.yaml is a map to the keys that rebuild the cluster, so it is sealed to the operator GPG key only. Sealing it to the cluster age key as well would hand flux-system a map to the Flux deploy key and to the age key that decrypts it. That is the whole reason the two files exist rather than one.

The cost is one deliberate duplication. PUBLIC_IP and MESH_IP in cluster.sops.yaml are the edge node’s addresses, currently ogma, which nodes.<edge host> in ops.sops.yaml also records. They cannot be shared: Flux needs them and cannot read the operator store. ansible/roles/netbird writes the mesh address back into ops.sops.yaml after every join, so treat that as the canonical one and keep cluster.sops.yaml in step by hand. Nothing detects drift between them.

The node that reads no store

brokkr is the exception to the table above, and it is deliberate. It runs outside the cluster, so there is no Infisical operator to read for it, and giving it a machine identity would spend one of the five the free tier allows on a node that would then hold a credential able to read almost the whole project.

So it holds none. ansible/roles/forge writes its runtime secrets into /etc/futhark/*.env at 0600, resolved out of Proton Pass through the same just ans render-secrets path every other role uses, and the reconciler on the node excludes *.env from the rsync so git never touches them. config/sops/ops.sops.yaml carries the references under ansible.secrets.brokkr.

This keeps the boundary above intact in both directions. The node cannot fetch a secret it was not given, and nothing in the cluster can read what the node has. Two values reach it by hand rather than through a store, because their minter cannot write to one the node can read: tofu/oidc emits Forgejo’s OIDC client pair as outputs, and tofu/b2 emits its Backblaze key, both filed into Proton Pass the same way K8up’s key is filed into Infisical.

The cost is that a rotation there does not reconcile. It reaches the node when someone runs just ans setup brokkr --tags podman and not before. See The standalone Podman plane.

Infisical, and how tier isolation is enforced

Apps read their secrets through the Infisical operator, with an InfisicalConnection and an InfisicalAuth per tier and one InfisicalStaticSecret per app. The project is laid out as /infra/<component> and /nodes/<hostname>/<app>.

An InfisicalStaticSecret names only its secretPath. The project and environment slugs are global, so they come from the config/infisical Kustomize Component, referenced as components: [../../../config/infisical] in the overlay, and the slugs are written once. Change the project there and nowhere else.

What only exists in the Infisical console

Three things the repository cannot express, in the order they bite:

1. The identity’s project membership. An identity can exist org-wide, authenticate fine, and still be refused every read. That surfaces as Unauthorized access: status 403 from the operator, and as ProjectMembershipNotFound from the API, a membership problem wearing a permissions error’s clothes. Granting an org-level role does not fix it. The identity has to be assigned to the project.
2. Its role and paths. cluster-reader gets read on /infra/* and /nodes/* in the prod environment, minus /infra/k8up. backup-reader gets /infra/k8up and nothing else. Both halves matter: the admission policy below already lets any infra namespace name an /infra/* path, so if cluster-reader keeps /infra/k8up the backup tier’s separate identity buys nothing.
3. accessTokenTrustedIps, the third isolation layer below. Set on both identities, scoped to the cluster’s egress address, so it fails whenever that changes: after a node rebuild, or when a pod is rescheduled onto a node whose address was never listed.

None of this is reconciled. Nothing drifts back. When secrets stop resolving and the manifests look right, check these before reading any more YAML. Rotating an identity’s client secret leaves all three untouched; replacing the identity itself does not. See Credential rotation.

Infisical’s free tier caps identities at five, humans included, so identities are rationed rather than minted per tier. cluster-reader is shared by the infra tier and every node tier. Kubernetes auth is not an option either: Infisical would have to reach the k3s API server for a TokenReview, and that API server is mesh-only. So one credential can, by itself, read almost the entire project, and the isolation OpenBao used to enforce server-side has to be rebuilt in the cluster.

Three layers do that, none of them relying on a controller’s good behaviour:

1. Kubernetes RBAC. The operator is installed once per tier, each with scopedRBAC: true and its own scopedNamespaces. The chart then emits a Role and RoleBinding per scoped namespace and no cluster-wide secrets ClusterRole at all. The node tier’s ServiceAccount therefore has zero permissions in infisical-infra or any infra namespace. It cannot read the infra tier’s InfisicalAuth, let alone write a Secret next to it. Each tier keeps its own copy of the credential in its own namespace for the same reason.
2. A ValidatingAdmissionPolicy. RBAC governs where a Secret may land, not which path may be read, so infra/infisical-operator/config/validatingadmissionpolicy.yaml pins spec.sources[].secretPath to the namespace’s own tier, using the futk.eu/tier and futk.eu/node labels from Namespaces, and pins every target to the object’s own namespace. It is evaluated by the API server, so a violating object is never persisted. No extra controller, no Kyverno.
3. Trusted IPs. Both identities have accessTokenTrustedIps set to the cluster’s egress address in Infisical itself.

Both layers are testable, and Checks and CI has the probes.

The one path that gets its own identity

/infra/k8up holds the Backblaze application key and the restic repository password. Between them, that is everything needed to read every backup this cluster has ever taken. Layer 2 is not enough for it: the admission policy pins an InfisicalStaticSecret to its namespace’s tier, so any infra namespace may legitimately name any /infra/* path, /infra/k8up included.

So the backup tier spends one of the five identities. infisical-backup is its own operator install scoped to itself and k8up, authenticating as backup-reader, which is granted /infra/k8up and nothing else, and cluster-reader is denied that path in return. The second half is what makes it real, and it exists only in the Infisical console. Skip it and the tier is decoration. ansible/roles/flux_bootstrap seeds this credential in its own task, separate from the loop that seeds the shared one.

What still defeats it: a cluster-admin who can edit the policy or the HelmReleases. Flux reconciles and prunes all three every 10 minutes, so drift reverts within one interval, but that is the honest boundary. sources[].tagSlugs is a selection filter, not an authorization one. Do not treat it as isolation.

Naming

Every secret name, in every store, is SCREAMING_SNAKE_CASE. BUNNY_API_KEY, ADMIN_PASSWORD, NB_PAT. No hyphens, no camelCase, no lowercase.

One rule, for three reasons. It is the intersection of what every store accepts, since Infisical’s key charset is narrower than Kubernetes’ and a hyphen that works in one may not in the other. Most of these values end up as environment variables anyway, where the shape is not a choice. And a single rule means you never have to remember which store spells a thing which way.

The exception is Proton Pass, whose items and fields are lowercase-with-spaces, as in pass://<vault>/netbird-policy/token. Nothing there is matched by name or becomes an environment variable under that name. Every reference is a path, and the environment variable it lands in is named by the consumer, not by the store. NB_PAT under tofu.netbird and ansible.secrets.netbird.api_token are two different fields of two different items, and the names say nothing about that. The paths do.

Kubernetes Secret keys are a separate question, because the consumer often dictates them and upstream charts do not follow this convention. Where the consumer is configurable, point it at the conforming name: infra/monitoring/app/grafana/helmrelease.yaml sets userKey: ADMIN_USER rather than the chart’s default. Where it is not, remap with a template block on the InfisicalStaticSecret target:

targets:
  - kind: Secret
    name: bunny-api
    namespace: cert-manager
    creationPolicy: Owner
    template:
      engineVersion: v1
      data:
        api-key: "{{ .BUNNY_API_KEY.Value }}"

Two manifests need this today: cert-manager (api-key) and storage (configData). Remap in the manifest rather than bending the name in Infisical. The constraint belongs to the chart, so it should be visible next to the chart, not encoded as a mystery in a remote UI.

Write .Value, not the bare {{ .KEY }}. The template’s data is a map of structs, and the bare form only prints the value because that struct has a String() method. As soon as a key is piped into a function that takes a string, as storage pipes the Storage Box key into replace, the bare form is a render error. One spelling everywhere is cheaper than remembering which keys are plain. Neither form makes a missing key fail: both render <no value> and report a successful sync.

The same block also handles a consumer that wants a file rather than a value. storage (configData) builds an INI in the template and interpolates one Infisical secret per credential. Store the credentials, not the file: a config blob in Infisical hides its own structure from review, cannot be rotated a field at a time, and grants everything it contains at once.

Where a consumer’s file format cannot hold the value as stored, escape it in the template rather than in the store. storage holds the Storage Box private key as the multi-line file ssh-keygen wrote and renders it with {{ .STORAGEBOX_KEY_PEM.Value | replace "\n" "\\n" }}, because rclone reads key_pem as a Go string literal and an INI value cannot span lines. The engine is text/template with the Sprig functions, minus env and expandEnv. A value mangled before storage is a transformation nobody reviews and everybody has to repeat correctly at rotation.

The secrets outside GitOps

These exist to seed what Flux resolves for itself, so neither can come from Flux. ansible/roles/flux_bootstrap creates both:

• flux-system/sops-age. Without it Flux cannot decrypt anything.
• infisical-universal-auth, one copy per tier namespace.

Plus flux-system/git-deploy-key, which is how Flux reaches the repository at all.

Being outside GitOps cuts both ways. Delete one of these namespaces and its seed Secret goes with it, with nothing in the Flux tree to put it back. Every InfisicalStaticSecret in the cluster then fails with InfisicalAuth is not ready, which reads like an operator problem and is not one. Re-run the role:

just ans k8s

Verify: just fx failing is empty, and kubectl get infisicalstaticsecrets -A shows every object ready.

Adding a new secret

1. Decide the tier. Identifying but harmless goes in config/sops/: cluster.sops.yaml if Flux substitutes it, ops.sops.yaml otherwise. Anything that bootstraps or re-keys the system goes in Proton Pass, referenced from ansible.secrets for Ansible or from tofu.<module> for OpenTofu. One app’s operational secret goes in Infisical. Name it SCREAMING_SNAKE_CASE, per Naming.

2. For Infisical, put it under /infra/<component> or /nodes/<hostname>/<app> and add an InfisicalStaticSecret in the app’s namespace, with secretPath only. Add components: [<relative>/config/infisical] to the overlay for the project and environment. If that namespace is new, declare it in infra/namespaces/app/namespaces.yaml and add it to the right tier’s scopedNamespaces in infra/infisical-operator/app/. The operator cannot write there otherwise, and the chart’s install fails outright if a scoped namespace does not exist.

3. For SOPS, add the key with just ops sops config/sops/ops.sops.yaml or just ops sops config/sops/cluster.sops.yaml, which decrypts, opens $EDITOR and re-encrypts in place. Add no comment: the file is ciphertext, and the explanation belongs in the matching .example template. A new key in cluster.sops.yaml also needs the consumer’s postBuild.substituteFrom to name the cluster-values Secret.

4. Confirm both checks pass:

   pre-commit run sops-encrypted --all-files
   pre-commit run gitleaks --all-files
   

5. If the new value can ever need replacing, add it to Credential rotation, including the case where it cannot be rotated at all.

Domains

Where the base domain is declared, how each plane reads it, and what to re-apply when it changes.

One base domain, one file: config/sops/cluster.sops.yaml, a SOPS-encrypted Secret whose three relevant keys are:

The labels are bare (in, n), not domains: a consumer always composes $SUB_INTERNAL.$DOMAIN. Nothing else in the repo spells a domain out. Never hardcode one in an app.

Reading it

Flux. infra/substitutions publishes the Secret as cluster-values. A consumer declares the dependency and the source in its ks.yaml, then uses ${DOMAIN} and ${SUB_INTERNAL} directly in its manifests:

spec:
  dependsOn:
    - name: substitutions
  postBuild:
    substituteFrom:
      - kind: Secret
        name: cluster-values

infra/monitoring/ks.yaml is the reference implementation. dependsOn names the Kustomization that publishes the values, and substituteFrom names the object this manifest reads.

replacements is not used for domains. It rewrites one delimiter-separated segment of a field, which cannot reach a domain sitting mid-string. An OIDC discovery URL has a path after the host. Mixing both mechanisms would also mean two ways to spell one value.

Tofu. bunny, oidc and netbird each declare the keys they need in their refs.env, and sops --extract them at plan time. No module keeps a copy. See Values another plane owns.

Ansible. just ans render-secrets decrypts the same file into ansible/.generated/cluster.yml, and inventory/group_vars/all/dns.yml gives it a nested shape. Use {{ dns.domain }}, {{ dns.sub.nodes }} and {{ dns.sub.internal }}. The flat capitals exist only because a Kubernetes Secret cannot nest.

Who resolves the internal subdomain

Two resolvers answer $SUB_INTERNAL.$DOMAIN, and they are configured in different planes.

Your devices. tofu/netbird publishes a wildcard A record in NetBird’s own DNS zone, pointing at the mesh address traefik-internal listens on. Its distribution_groups is the admin group, so only the operator’s peers receive it. The nodes join node and k8s and do not.

Pods. infra/coredns writes a coredns-custom ConfigMap holding a stub zone that answers the same subdomain with the same address. Without it every internal hostname is NXDOMAIN inside the cluster: k3s strips loopback resolvers out of the kubelet resolv.conf, so CoreDNS forwards to the public Bunny zone, which holds nothing under the internal label. That is what broke every healthcheck Glance ran against an internal host.

k3s’s bundled Corefile ends with import /etc/coredns/custom/*.server and carries reload, so the ConfigMap lands without restarting CoreDNS. Verify from any pod:

kubectl -n glance exec deploy/glance -- nslookup metrics.$SUB_INTERNAL.$DOMAIN

Expect the mesh address. If it is still NXDOMAIN after about 30 seconds, force the reload with kubectl -n kube-system rollout restart deploy/coredns.

Both resolvers answer for every name under the subdomain, including ones with no Ingress behind them. Such a name resolves and then fails at the Traefik router, not at DNS.

Changing the domain

Edit the one file, then re-apply everything that resolved it:

just ops sops config/sops/cluster.sops.yaml
just tf apply bunny
just tf apply netbird
just tf apply oidc
just ans setup

netbird moves the peer domain, so every node’s ansible_host changes with it. Ansible reaches the nodes by mesh name.

just tf apply netbird here changes netbird_account_settings, which needs the policy service user promoted to Admin for the duration. See Applying account settings.

Verify: just fx failing is empty, just ks certs shows every certificate Ready under the new domain, and just ops mesh still resolves.

Internal ingresses are unauthenticated by default

An internal-class Ingress is reachable by anything on the mesh, with no login in front of it. That is still the default, and it is what metrics.$SUB_INTERNAL.$DOMAIN and logs.$SUB_INTERNAL.$DOMAIN rely on today. Mesh membership is their only access control.

There are three ways an internal host gets a login, in order of preference.

The app speaks OIDC. Best case, and no proxy is involved. Register a client in tofu/oidc and point the app at Pocket ID. Grafana does this through auth.generic_oauth, mapping the administrators and users groups to Admin and Viewer. Actual does it too. Only this option can express per-user roles, because only the app knows what a role means.

The SSO middleware. For an app with no OIDC support, add auth-sso@kubernetescrd to the Ingress after the namespace’s own rate limit:

annotations:
  traefik.ingress.kubernetes.io/router.middlewares: <namespace>-ratelimit@kubernetescrd,auth-sso@kubernetescrd

That is a Traefik forwardAuth pointing at the oauth2-proxy in infra/auth, which is registered with Pocket ID as one shared client. infra/glance is the reference implementation. The gate is binary: anyone in administrators or users gets in. The session cookie is scoped to .$SUB_INTERNAL.$DOMAIN, so one login covers every host that opts in. See Cluster infrastructure.

An app behind that middleware can still tell who the reader is, without becoming an OIDC client itself, by reading the X-Auth-Request-* headers the middleware forwards. infra/copyparty is the reference implementation: it maps the groups header onto per-volume permissions. This is weaker than the first option, because the app is trusting a header rather than a signed token, and it only holds while the app is unreachable except through the middleware.

Neither. Mesh membership only, which is a deliberate choice for a host whose readers are already trusted with the mesh.

Cross-namespace Middleware references work because infra/traefik-internal sets allowCrossNamespace: true on its Kubernetes CRD provider.

One thing that will bite you

In a plain manifest reconciled by a Flux Kustomization, escape a literal $ as $$, or envsubst will eat it. Both files under infra/storage/app/storageclass-*.yaml do this for rclone’s own ${pvc.metadata.*} template variables.

Startup ordering

The dependsOn graph the whole tree reconciles in, why each non-obvious edge exists, and where a new component belongs in it.

Two Kustomizations dependsOn nothing: namespaces, which is every Namespace CR and no controller, and substitutions, which is every postBuild.substituteFrom source and no controller either. A substitution target has to exist before any consumer reconciles, so it cannot wait on anything. The controllers that need nothing else from the cluster, infisical-operator, cert-manager and postgres, sit directly behind namespaces.

The real graph, as declared in each ks.yaml. Green marks the boundary: the two roots and the two sinks. Purple dashed marks a config-ks.yaml. Blue is the ordering spine, the chain that actually has to reconcile in sequence. The two grey bundles are not part of that chain and are drawn back so it reads through them: dashed grey is a substitution source that only has to exist, and solid grey is the fan-in onto infra-policies.

Only those four name namespaces in their dependsOn. Everything else reaches it transitively.

They need nothing from the cluster but a namespace to land in. For three of them the config-ks.yaml siblings are where the ordering actually bites, because those apply CRs the controller must already have registered CRDs for. trivy-operator has no such sibling: it reads no secret and no cluster value, so a namespace is genuinely all it waits for.

Six edges are less obvious than they look:

• auth depends on postgres-config, and it is the most expensive edge in the tree. Pocket ID runs on ogma and keeps its users, passkeys and OIDC clients in the shared instance, which is a single Cluster pinned to kenaz. Every login in the cluster now waits on that node and on that one pod. gatus carries the same edge for the same reason. Both are deliberate; the trade they buy is in Cluster infrastructure.

• namespaces is a root of its own rather than a file next to each component, because infisical-operator installs its chart with scopedRBAC: true. Helm emits a Role and RoleBinding inside every scopedNamespaces entry at install time, and those namespaces belong to components that are downstream of infisical-operator-config. With the Namespace CRs held by their consumers, the install failed outright on namespaces "auth" not found.

• A config-ks.yaml does not always belong downstream of its controller. The rule is what the dependency is for: what a chart mounts goes upstream of it, what needs the chart’s CRDs goes downstream. A config Kustomization producing a Secret the chart’s own Deployment mounts has to run first, or the Helm install waits on a pod that waits on a Secret that waits on the install. Nothing in the tree currently inverts it, since every config-ks.yaml here applies CRs, but the inversion is legitimate and is why the rule is stated rather than the pattern.

• glance-config is a third kind: a split made for neither CRDs nor mounts, but to keep postBuild substitution away from files that spell their own variables the same way Flux does. It has no dependsOn at all, because there is nothing it could need, and glance names it so the ConfigMap exists before the pod tries to mount it. Cluster infrastructure has the reasoning.

• gatus depends on coredns, which is the only dependency in the tree on a DNS record rather than on an object. Gatus probes every service by its internal hostname, and those names do not resolve inside the cluster until coredns has applied its stub zone. Without the edge, Gatus reconciles green with every check failing.

• substitutions has no dependencies, and holds every postBuild.substituteFrom source in the cluster: the cluster-values Secret and the monitoring-sizing ConfigMap. A substitution target must exist before the Kustomization that substitutes from it reconciles, and traefik-edge, one of those consumers, is upstream of infra-policies, the otherwise obvious home for them.

infra-policies sits behind every infra controller. Its overlays attach policy to namespaces that are already there, so the ordering it needs is the controllers’. middleware-ratelimit wants Traefik’s Middleware CRD registered, and the point of the edges as a whole is that a namespace’s default-deny policy lands before anything worth denying. nodes then depends on infra-policies.

What wait: true already buys, and why there are no healthChecks

Every Kustomization in the tree sets wait: true, patched in once by infra/kustomization.yaml, and none sets healthChecks. That is deliberate, and the two are alternatives rather than complements. wait: true health-checks every resource the Kustomization applied, and Flux ignores healthChecks entirely when it is set. Adding a healthChecks list would be config that never runs. Getting it to run means wait: false, which checks only the resources you remembered to name.

There is a third field, and it is the one that applies to custom resources: healthCheckExprs evaluates a CEL expression per kind, and unlike healthChecks it is evaluated only when wait: true is set. postgres-config is the one Kustomization that needs it. CloudNativePG’s Database and DatabaseRole report status.applied and no conditions array, so kstatus treats them as healthy the moment they reach the API server, and any of the five tenants gated on that edge starts against a database the instance manager has not created yet. Reach for it whenever a config-ks.yaml applies a CR whose controller reports progress somewhere other than conditions.

Write current and leave failed out, unless the controller has a state it genuinely never leaves. failed makes Flux give up the moment the expression matches, and a controller that retries passes through the failing state on its way to the working one. CloudNativePG applies a Database before the DatabaseRole that owns it exists, reports applied: false once, and succeeds on its own retry. With failed set on that, postgres-config stopped at failed early due to stalled resources and held every tenant behind it down. Without it, an object that truly never converges runs out the Kustomization’s timeout instead, which is the same outcome an hour later and the correct one.

The gap healthChecks would seem to close, “the HelmRelease is Ready but its pods are still starting”, is closed further upstream. helm-controller’s install.disableWait and upgrade.disableWait both default to false, so it polls the chart’s workloads with kstatus and only then reports the HelmRelease Ready. So a dependsOn edge onto a chart-based component already means that chart’s Deployments and DaemonSets are up.

local-path is the one piece that cannot come from Flux at all: monitoring, auth and nodes/kenaz.k8s/actual bind PVCs on their first reconcile, and nothing in the Flux-managed tree can provision a StorageClass for itself. It arrives with k3s, whose bundled provisioner ansible/roles/k8s_cluster deliberately leaves enabled, during just ans k8s, Cold bootstrap step 9.

Where a new component goes

• Has an InfisicalStaticSecret: downstream of infisical-operator-config, same as storage and monitoring. Also add its namespace to the right tier’s scopedNamespaces. See Cluster infrastructure.
• Needs a certificate: downstream of cert-manager-config.
• Needs ingress: downstream of traefik-internal, or traefik-edge if it is public-facing.
• Needs none of the above: it can be another root. Check first. Most things eventually need a cert or an ingress, and both have prerequisites.

Whatever you pick, add the component’s namespace to infra/namespaces/app/namespaces.yaml. Nothing else declares it. If anything under infra/policies/namespaces/ targets that namespace, add the component to infra/policies-ks.yaml’s dependsOn too, so the policy lands with the workload rather than ahead of it.

Verify the edge you added: just fx get shows the new Kustomization Ready, and nothing upstream of it moved to Reconciling and stayed there.

Dependency updates

How pinned versions get bumped, what Renovate can and cannot see, and what you must do by hand for a new pin to be tracked at all.

Every version in this repository is pinned exactly (Version pins), which is only sustainable if something else does the watching. Renovate reads the whole tree once a night and opens a pull request per update. The Dependency Dashboard issue lists everything it knows about, including updates it has not opened a PR for yet.

Config lives in .github/renovate.json5, and it runs from .github/workflows/renovate.yml. This is a self-hosted Renovate, not the Mend-hosted GitHub App, so no third party holds write access to the repository.

What it covers

The built-in managers are scoped to the directories they belong to, because a manager with no file patterns walks the whole tree. SOPS files are excluded outright: their version: field records the sops binary that encrypted them, which is history, not a dependency.

The annotation convention

Renovate cannot guess the upstream of a YAML value no built-in manager understands. Those pins carry a comment on the line directly above:

# renovate: datasource=github-releases depName=fluxcd/flux2 extractVersion=^v(?<version>.+)$
version: "2.9.4"

versioning= and extractVersion= are both optional and go after depName=, in that order. k3s needs versioning because v1.36.3+k3s1 is not plain semver.

The comment must stay immediately above its pin. Insert a line between them, or move the pin without its comment, and it silently stops being tracked. There is no error, just an update that never arrives. .github/workflows/validate.yml validates the config itself on every pull request, but nothing can validate an annotation that is simply absent.

The Quadlet manager needs no annotation, because a .container file’s Image= key is already an unambiguous shape: Image=<repo>:<tag>@sha256:<digest>, the same tag@digest pin every manifest in this repository uses. It exists at all because no built-in manager reads a systemd unit file, so without it those four images would never move and “nothing floats” would quietly become “nothing moves”. Those pins reach a node outside the cluster; how they get applied is The standalone Podman plane.

Group rules exist for the pins that are one decision written twice: the gitleaks version in both mise.toml and .pre-commit-config.yaml, the VictoriaMetrics stack, mdbook and its d2 preprocessor, the Flux operator and the distribution it installs.

What merges itself

GitHub Actions only, and only minor, patch and digest, after the release has been public for three days and validate.yml has passed. An action bump cannot reach the cluster. Everything else can, so everything else waits for a human.

Two of those PRs deserve a closer read than the rest:

• k3s (ansible/roles/k8s_cluster/defaults/main.yml) is a control-plane upgrade. Merging it changes nothing on its own. It takes effect on the next just ans k8s, which drains and restarts each node in turn.
• Chart majors carry values-schema changes. kustomize build --enable-helm in CI renders the chart, so a values key that no longer exists fails the PR, but a key that changed meaning does not.

Merging one out of schedule

Renovate decides when an update is available. Trivy decides when one is urgent.

A Slack alert for a fixable CRITICAL, or a finding in the Security tab, names an image that has a published fix. Both scanners run with ignore-unfixed, so nothing else reaches you. The corresponding pull request is already open on the Dependency Dashboard. Merge that one rather than waiting for the next pass over the list.

Chart-internal images have no pull request of their own. Bump the chart version instead, which is the only pin this repository holds for them, and confirm the new one carries the fix:

just sec scan docker.io/traefik:v3.7.9

Vulnerability scanning covers what each scanner sees.

Running it by hand

gh workflow run renovate.yml -f logLevel=debug -f dryRun=true

dryRun extracts and logs without creating branches, PRs or issues, which is how to check that a new annotation or file pattern is picked up. The log line to look for is packageFiles with updates, which lists every dependency found per manager. A manager showing zero has a file pattern that no longer matches.

Locally, without any token:

npx --yes --package renovate -- renovate --platform=local

Repository prerequisites

Renovate authenticates as a GitHub App of ours, minted per run and expiring within the hour. The default GITHUB_TOKEN cannot be used: pull requests it opens do not trigger on: pull_request, so validate.yml would never run and nothing would gate an automerge.

The App needs read/write on contents, pull requests, issues and workflows, installed on this repository, with its ID in the RENOVATE_APP_ID variable and its private key in the RENOVATE_APP_PRIVATE_KEY secret.

Its commits are signed by GitHub, not by a key of ours. platformCommit makes Renovate write through the GitHub API instead of git push, and GitHub signs what it writes with the App’s identity. A key we held would have to live in a repository secret and could not be verified anyway: a GitHub App has no user account to register a public key against, so the signature would show as unverified.

Three settings outside the repo tree matter as much as the config:

• Allow auto-merge must be on, and master must require the validate checks. Without a required check, an auto-merge lands the moment the PR opens.
• Allow squash merging must be on, and master must require signed commits. platformCommit signs what Renovate pushes; automergeStrategy: "squash" makes GitHub author and sign the commit that lands, where a rebase merge would replay the branch commits and drop their signatures. Enable the signed-commit rule only after a run has confirmed the branch commits are signed, or Renovate’s pushes are rejected and every pull request stalls. Afterwards, rebasing a bot branch locally and pushing it is refused unless you re-sign the rewritten commits.
• Labels must already exist. Renovate applies type/{major,minor,patch,digest} and renovate/{container,helm,terraform,ansible,github-action,tool}; it does not create them, and a label it cannot find is dropped without an error. just ops labels creates or updates them, reading the names straight out of packageRules so the two cannot disagree. Colour and description follow from the prefix. Run it once, and again after adding a label to the config. It never deletes, so anything you added by hand survives.

Inventory and roles

Where node facts live, which playbook and role does what, and how a secret reaches a task. Read this before editing inventory or adding a role.

Ansible owns everything below Kubernetes: the user you log in as, the SSH configuration, the firewall, the mesh join, and the k3s install itself. Once Flux is running, Ansible’s job is done. The only reasons to come back are adding a node and re-converging the cluster.

Run everything through just ans rather than ansible-playbook directly, because the recipes set the working directory ansible.cfg expects. See Recipe reference.

Inventory

ansible/inventory/hosts.yml is a bare list of node names. Everything about a node lives in ansible/nodes/<hostname>/host.yml, surfaced to Ansible by a symlink in ansible/inventory/host_vars/<hostname>/. Its one identifying value, the address, comes from config/sops/ops.sops.yaml. See Nodes for the schema and how to add one.

ansible/inventory/group_vars/all/ holds what is shared, all of it in the clear:

network.yml beside it holds every constant more than one role has an opinion about:

brokkr.yml beside these loads the forge node’s own bucket name the same way nodes.yml loads the addresses, out of a top-level key both Ansible and tofu/b2 read. See Secrets.

These are collected rather than written at each use site because they had drifted into different spellings of the same fact: the mesh range was a literal in fail2ban’s ignoreip, a literal again in the mesh role’s firewalld loop, and a hand-expanded regex in k8s_cluster’s assert. A constant with three spellings is three constants. The CIDRs specifically need three consumers to agree: k8s_cluster writes them into k3s’s config.yaml, netbird trusts both in firewalld, and the same role needs the pod CIDR to scope its pod-to-mesh SNAT rule. tofu/netbird reads mesh_cidr out of this file for the account’s network range. k8s_pod_cidr is the cluster-wide /16, the CNI carves a /24 out of it per node, and the SNAT rule must match the /16 or a peer’s pods are not covered. All of these are private or RFC6598 ranges, not identifying, so they are plain literals.

ansible_host resolves through NetBird’s DNS for mesh nodes, as <hostname>.<mesh_dns_domain>, and falls back to node.ip otherwise. There is no stored mesh IP anywhere in this repository. NetBird’s own resolver keeps the name correct across re-keys and reassignments, so there is nothing to update when an address changes. The expression is guarded on node is defined because flux_bootstrap runs against hosts: localhost, which is implicit, not in inventory, and has no node var.

setup.yml still overrides these three with set_fact mid-play, for first-time provisioning where the host does not yet answer as the admin user. Facts beat inventory vars, so that dance is unaffected by the defaults above.

repo_root is derived from playbook_dir, not inventory_dir, for the same hosts: localhost reason. localhost has no inventory_dir, and playbook_dir is play-scoped rather than host-scoped.

Playbooks

setup.yml runs per host and is gated by the node’s own flags. The netbird role only runs when node.mesh is true, firewall_ingress and egress_exporter only when node.public_ingress is, and the four Podman roles only when node.workflow is podman. Nothing in any role branches on a hostname, so a future node opts into any of them by setting the flag.

setup is variadic after the hostname, so the playbook’s own flags reach it. Pass '' as the hostname to mean all of them: just ans setup '' --skip-tags podman.

It also carries tags, so a single concern can be re-converged without running the whole thing:

admin_user and ssh_harden share one tag on purpose: ssh_harden disables root and password login, so running it without admin_user locks the host out permanently. ssh_identity and the post-play set_fact are tagged always, because they decide which login and port every other task connects with. Skipping them would have --tags mesh dial a fresh host as an admin user that does not exist yet.

The four roles under podman share one tag because they are one sequence, and the order in setup.yml is load-bearing: podman_host provides the runtime and the directories, forge writes the env files a .container unit needs before it can start, quadlet_gitops brings the units in from git and starts them, and forge_bootstrap talks to the container that results. Splitting the tag would let an operator run the fourth without the second. Skipping it is the supported case, and it is what a cold bootstrap does until the credentials that plane needs exist; see Cold bootstrap.

k8s.yml is tagged the same way: k8s, flux.

k8s.yml is two plays over one role, the controller and then the workers, because an agent’s config needs the join token the server only mints on its first start, and Ansible runs a play host-by-host in parallel. Its Flux play runs against hosts: localhost and reaches the cluster over the network with the fetched kubeconfig, with no SSH and no become.

Roles

Ordering between them is declared in each role’s meta/main.yml, not left to the order of the playbook’s role list. ssh_harden depends on admin_user for the lockout reason above, and the four firewall-writing roles depend on firewalld. That last one used to be two tasks inside ssh_harden, which made “the mesh role needs a running firewalld” an ordering fact you could only learn by reading setup.yml top to bottom. Ansible runs a role once per play regardless of how many times it is reached, so the dependencies cost nothing at runtime, though --list-tasks prints the pre-deduplication list and will show them repeated.

podman_host, forge, quadlet_gitops and forge_bootstrap cannot express their order that way. Each needs the previous one’s effects rather than its tasks, and a meta dependency would run the role rather than order it, so the sequence lives in setup.yml’s role list with a comment saying it is load-bearing. They do share variables across that boundary: forge and quadlet_gitops compose their paths from podman_host’s defaults, which resolve because a statically listed role’s defaults are merged into the play’s variables at play start, regardless of role order.

Five roles are worth knowing in more detail. The four Podman ones are covered together in The standalone Podman plane, because what is worth knowing about them is the plane they build rather than any one of them.

fedora_common

dnf upgrade on "*", then a reboot if needs-restarting -r asks for one. Both are gated: fedora_common_upgrade and fedora_common_reboot, defaulting true. Without them, asking for an unrelated change on a node carrying live workloads, such as re-converging the mesh, would upgrade and reboot it as a side effect. -e fedora_common_upgrade=false converges hostname and base tooling only. -e fedora_common_reboot=false upgrades now and reboots in a window, and the run still reports whether one is pending.

fail2ban

Two jails, both in jail.d/10-futhark.local: sshd on ssh_port, and recidive, which re-bans anything the first jail catches repeatedly. Bans are enforced by firewalld, from the firewallcmd-rich-rules action Fedora’s own jail.d/00-firewalld.conf already sets, which is the same firewall every other role touches.

ignoreip covers loopback, the mesh’s CGNAT range and both cluster CIDRs. Ops SSH and Ansible arrive over the mesh, so without that line a misfiring jail could lock the operator out of every node at once.

The role also redirects fail2ban’s own logging from the journal to /var/log/fail2ban.log, in fail2ban.d/10-futhark.conf. recidive needs a readable log to count bans in, and the file is what carries ban events into VictoriaLogs. See Cluster infrastructure. That is why recidive overrides the default systemd backend with polling: it reads the file, not the journal.

ssh_identity

A fresh host answers as node.initial_user on node.initial_port; once admin_user and ssh_harden have run, that login is gone. ssh_identity probes the hardened port and picks accordingly. Probing only the hardened port is sufficient because admin_user installs the key before ssh_harden closes the initial one, so there is no window where neither works.

netbird

The join itself, the firewalld zoning and the pod-to-mesh routing fix are covered in Pod to mesh networking and Mesh watchdog. One thing lives only here.

The daemon writes /etc/ssh/ssh_config.d/99-netbird.conf unless told not to, and that file sets StrictHostKeyChecking no, UserKnownHostsFile /dev/null and PasswordAuthentication yes for every mesh peer, on a fleet whose sshd ssh_harden deliberately locks down. NetBird’s own SSH server is disabled on these peers, so the file buys nothing. The role sets NB_DISABLE_SSH_CONFIG=true through a systemd drop-in before the daemon’s first start, so a fresh node never has the file, and removes any copy an earlier install left behind.

The removal is deliberately not followed by a restart. This play reaches the node over the mesh, so bouncing the daemon would cut the connection running it. The environment variable stops the file being regenerated; until the daemon next restarts on its own, a regenerated copy is simply removed again on the next converge.

just ops mesh checks the same file on the operator’s own machine, which Ansible does not manage, and prints the three commands to fix it.

k8s_cluster

Unverified. This role replaced a k0s and k0sctl setup, and has not yet been run against the fleet. The configuration below is what it declares, not observed behaviour. Verify each claim on the first converge.

kubelet’s node-ip is pinned to each node’s NetBird mesh IP, deliberately: the Kubernetes API, etcd and kubelet then bind only to mesh addresses and are never publicly exposed. Several consequences follow.

k3s would otherwise self-detect advertise-address from the default-route interface, which on these hosts is the public IP, so join tokens would carry an address workers cannot reach. The role pins it to the controller’s mesh address instead, read from inventory as node.mesh_ip. flannel-iface is pinned to netbird0 for the same reason: left alone, flannel would build its VXLAN overlay over the public internet rather than the mesh.

That value is not resolved at converge time. The management server assigns a mesh address at registration and it cannot be chosen in advance, so roles/netbird reads it back out of netbird status --json after the join and records it into the nodes map in config/sops/ops.sops.yaml, which group_vars/all/nodes.yml loads back. Writing it down rather than looking it up each run is what lets playbooks/k8s.yml run as a separate invocation from setup.yml, and lets a worker read the controller’s address straight out of hostvars. The write is guarded by a compare, because SOPS re-encryption changes the ciphertext even when the plaintext has not.

The role then asserts the recorded address is non-empty and inside NetBird’s CGNAT range (mesh_cidr). That assertion is not paranoia: a node deleted and re-registered picks up a different address, and a stale value would otherwise be baked silently into advertise-address and this kubelet’s node-ip.

The public IP still has to reach Kubernetes somehow, since kubelet can only ever register node-ip as InternalIP. It arrives as node-external-ip, which k3s’s own cloud controller turns into the node’s ExternalIP. Under k0s this needed a separate kubectl annotate pass after every converge. node.ip comes from the encrypted config/sops/ops.sops.yaml, so the public IP is never committed in the clear.

This decision is also what makes cross-node pod networking non-trivial. See Pod to mesh networking.

Three notes on how the role stays honest. The join token is read off the controller’s /var/lib/rancher/k3s/server/node-token at converge time rather than pre-shared through Proton Pass, so there is no second copy to go stale. /etc/rancher/k3s/config.yaml is rendered before the installer runs, so the service comes up configured on its very first start instead of joining on defaults and being corrected a moment later; a later change to it is a diff and a handler restart, not a reinstall. And the install itself is guarded by a version comparison rather than creates:, because re-running the vendor installer with a new INSTALL_K3S_VERSION is exactly how an upgrade happens.

The kubeconfig k3s writes says https://127.0.0.1:6443, which is true on the node and useless to the operator. The role fetches it and rewrites the address to the controller’s mesh IP, rather than pointing it at a tls-san name, so it keeps working when mesh DNS is the thing that broke.

The file lands in ansible/.generated/, which is gitignored, so a fresh clone or a wiped .generated/ has no kubeconfig and every just ks recipe fails. just ans kubeconfig re-runs that fetch on its own, through the kubeconfig tag on k8s.yml, without converging the cluster.

How secrets reach a play

Two mechanisms, no custom code in this repo any more.

Both go through just ans render-secrets, which decrypts into ansible/.generated/. Nothing is decrypted at load time, and no task mentions SOPS.

Identifying values land in nodes.yml, cluster.yml and the admin subtree of secrets.yml, which group_vars/all/nodes.yml, group_vars/all/dns.yml and group_vars/all/admin.yml read with a file lookup. They are inventory-level rather than a playbook’s vars_files because playbooks/k8s.yml reaches hostvars[<other node>].node.mesh_ip and, from its localhost Flux play, hostvars[<node>].admin.user, which resolve only if every host carries the variable itself. So admin.user and node.ip are ordinary variables at the point of use.

Crown-jewel values come from Proton Pass. just ans render-secrets decrypts the ansible subtree of config/sops/ops.sops.yaml and pipes it through pass-cli inject into ansible/.generated/secrets.yml. The playbooks load that with vars_files, so the roles that need one, flux_bootstrap and netbird, both no_log: true, reference an ordinary variable like secrets.flux.deploy_key. ans setup and ans k8s depend on that render, so it is not a step you run by hand. It needs a Proton Pass session, and pass-cli info checks for one.

Which store a given value belongs in is Secrets.

Nodes

The per-host schema, what each field decides, and how to add a node end to end. At the end of Adding a node the host is provisioned, on the mesh, and part of the cluster.

ansible/nodes/<hostname>/host.yml is the source of truth for a host, and carries everything reviewable in a diff. Its one identifying value, the address, lives in the nodes map of config/sops/ops.sops.yaml. ansible/inventory/host_vars/<hostname>/ holds a symlink to host.yml, which is how Ansible picks it up.

Do not confuse ansible/nodes/ with the repository-root nodes/. This one is provisioning data: identity, address, and how to reach and bootstrap the host. The other is workload definition, what runs once the host exists. See Node apps.

Schema

node:
  hostname: <hostname>
  os: fedora
  workflow: k8s
  k8s_role: controller
  mesh: true
  public_ingress: true
  ip: "{{ nodes[inventory_hostname].ip }}"
  initial_user: fedora
  initial_port: 22

ip stays a reference, never a literal, because a real address is an identifying value and this repository is public. The nodes map comes from config/sops/ops.sops.yaml:

nodes:
  kenaz:
    ip: 203.0.113.10
    mesh_ip: ""

Keeping the reference inside the node: dict rather than moving the field out of it is deliberate: every hostvars[x].node.ip in the roles and the k3s config template keeps working unchanged, and Jinja resolves the indirection at use time. See Secrets.

Exactly one host should be a controller. A second controller makes etcd a two-member cluster with quorum two, which is worse for availability than a single controller, not better.

workflow decides two things beyond which playbook touches the host: which four roles playbooks/setup.yml runs under its podman tag, and which NetBird group the peer joins. ansible/roles/netbird derives that group name from this field, so a new workflow value needs a matching netbird_group in tofu/netbird/groups.tf before the first join, or the peer enrols into a group that does not exist and the setup key is rejected. podman is brokkr; see The standalone Podman plane.

mesh is orthogonal to workflow: opt in for any node, cloud or local, that needs mesh reachability. There is no mesh IP to store. Once joined, the node is addressed as <hostname>.<mesh_dns_domain>, and NetBird’s own resolver keeps that correct across re-keys. The node’s Kubernetes InternalIP comes from k8s_cluster’s node-ip, which is the address recorded in node.mesh_ip, not from that name.

public_ingress and mesh are both read generically. Nothing in roles/netbird, roles/firewall_ingress or roles/flux_bootstrap branches on a hostname, so on this plane moving public ingress is a one-line inventory change.

The other planes are not so lucky. public_ingress gates the firewall_ingress role and nothing else. No tofu module and nothing in the Flux tree reads it, so the edge node is named independently in five more places, with nothing detecting disagreement between them. Moving public ingress means changing all six together:

Miss a nodeSelector and that Traefik pod sits Pending. Miss PUBLIC_IP and the edge never binds. Miss tofu/bunny/refs.env and the public A record points at the wrong host; miss tofu/netbird/refs.env and every internal hostname does.

Adding a node

mkdir -p ansible/nodes/<hostname> ansible/inventory/host_vars/<hostname>
$EDITOR ansible/nodes/<hostname>/host.yml

# add a `<hostname>:` entry under `nodes`, with its `ip` and an empty `mesh_ip`
just ops sops config/sops/ops.sops.yaml

ln -s ../../../nodes/<hostname>/host.yml ansible/inventory/host_vars/<hostname>/host.yml
# then add `<hostname>: {}` under all.hosts in ansible/inventory/hosts.yml
just ans setup <hostname>
just ans k8s   # workflow: k8s only

just ans setup writes the node’s mesh address back into nodes.<hostname>.mesh_ip. Commit that change before running just ans k8s, which reads it.

Verify: just ans ping reaches the new host, ssh <hostname> netbird status reports connected, and for a workflow: k8s node, just ks nodes lists it Ready.

A workflow: podman node skips just ans k8s entirely, and just ans setup is the whole procedure: it converges the runtime, pushes the secrets, installs the git reconciler and starts the containers. Verify with ssh <hostname> podman ps instead. Its own prerequisites, the NetBird group and the ansible.secrets.<hostname> references, are in The standalone Podman plane.

If the node will run its own tenant apps under nodes/<hostname>.k8s/, it also needs its own Infisical operator tier, which is three files and one defaults entry, listed in Cluster infrastructure.

Pod to mesh networking

Why a pod cannot reach a peer node’s own mesh address without help, the two separate fixes that make it work, and how to tell which one has broken. Read this before touching ansible/roles/netbird.

Written against k0s and kube-router. The cluster now runs k3s, whose default CNI is flannel with a VXLAN backend, not kube-router with IPIP. The decision this page starts from is unchanged and so is fix 2; fix 1 depends on the CNI installing a policy routing rule of its own, which flannel may not do at all. Check ip rule on a running node before trusting or removing anything under “1. Route lookup”.

kubelet’s node-ip is pinned to each node’s NetBird mesh IP, on purpose: the Kubernetes API, etcd and kubelet are then only ever bound to mesh addresses and are never publicly exposed. Everything below follows from that one decision.

Because the nodes share no L2 segment, the CNI builds its own overlay between their mesh IPs to carry cross-node pod-to-pod traffic: IPIP under kube-router, VXLAN under flannel. That has two consequences, both of which break a pod dialing a peer node’s own mesh address.

This is not a hypothetical case. konnectivity-agent does exactly that, dialing the controller’s konnectivity-server over the mesh, so while it is broken kubectl exec, logs and port-forward are down cluster-wide.

Two paths leave a pod for another node, and they fail for different reasons, which is why the two fixes below are not interchangeable:

1. Routing

kube-router installs from <node's pod /24> lookup 77 and puts the peer’s mesh IP into table 77 pointing at its tunnel, so the packet is routed into the very tunnel whose transport endpoint is that address.

The fix is one ip rule per peer at mesh_route_priority, matching only that peer’s /32 and never the pod CIDR, so pod-to-pod overlay routing is untouched. It looks the peer up in mesh_route_table, which the same script populates with a <peer>/32 dev netbird0 route. That table is ours and holds exactly what is written into it, rather than depending on where the VPN client happens to install its own routes. NetBird keeps those in table 7120, with rules at priorities 105 and 110, well clear of mesh_route_priority.

The priority is load-bearing whenever the CNI installs a rule of its own. kube-router’s moved before: 5209, then 99, then priority 9 as of kube-router v2.10.0. That last move silently shadowed this fix at its old priority 10 and broke every mesh-peer-initiated connection into a pod, meaning backends behind traefik-edge, which is hostNetwork and reaches them as the node’s own mesh IP. It surfaced as 502 Bad Gateway with no error anywhere else in the stack. Check ip rule on the node rather than assuming, and re-check after a CNI bump.

2. Source address

The WireGuard peer drops packets whose source is not the node’s own mesh address, as anti-spoofing, so pod-sourced packets still die on egress even once the route is correct. They fail with no route to host, which reads like a routing fault and is not one.

They have to be masqueraded to the node’s mesh IP. Nothing else does this: kube-proxy’s KUBE-POSTROUTING only masquerades service traffic carrying the 0x4000 mark, and NetBird only masquerades traffic for the network routes it advertises, which pod-to-peer traffic is not.

The SNAT is scoped to peer /32s rather than the whole mesh_cidr on purpose. Pods get to reach cluster nodes, such as konnectivity-agent to konnectivity-server and vmagent to traefik on the mesh IP, without inheriting the node’s own reach across the entire mesh.

All three rules live in ansible/roles/netbird/templates/futhark-mesh-routes.sh.j2, re-applied by a systemd oneshot because neither ip rule, a route in a custom table, nor iptables state survives a reboot.

The script reads the peer set out of netbird status --json each time it runs, rather than having Ansible resolve peer addresses and bake them in. A peer re-registering onto a different mesh address would otherwise strand these rules until someone ran Ansible again, over the very mesh that is now half broken. It tears down everything it owns and rebuilds from the current peer set, so a peer that has gone away leaves nothing behind. The SNAT rules sit in their own FUTHARK-MESH chain so they can be flushed as a set. The mesh watchdog re-runs the unit on every healthy probe, which is what makes drift correct itself.

The isolating test

Run on a node:

ping -c2 -I <this node's mesh IP> <peer mesh IP>        # succeeds
ping -c2 -I <this node's pod-bridge IP> <peer mesh IP>  # fails without the SNAT rule

Both are locally generated (OUTPUT path, and netbird0 is in firewalld’s trusted zone), so neither traverses FORWARD. That rules out every firewall hypothesis at once and isolates the drop to the mesh client itself.

The all-protocol policy rule

IPIP solves the source-address half for cross-node pod-to-pod traffic on its own. The outer header is sourced from the node’s own mesh IP, so it never presents a foreign source to the mesh client. But the overlay only carries traffic at all if the mesh policy permits it.

NetBird policy rules name a protocol: tcp, udp, icmp, netbird-ssh, or all. IPIP is IP protocol 4, so only all passes it; flannel’s VXLAN is UDP 8472, which a narrower rule could carry but only by writing the CNI’s port into the access model. Anything narrower and cross-node pod-to-pod blackholes: the route is correct, the tunnel is UP, nothing is logged, and every packet vanishes.

That is what netbird_policy.k8s in tofu/netbird is for: a rule from k8s to k8s with protocol = "all" and no ports.

rule {
  name          = "k8s to k8s, every protocol"
  action        = "accept"
  protocol      = "all"
  bidirectional = true
  sources       = [netbird_group.k8s.id]
  destinations  = [netbird_group.k8s.id]
}

Nothing asserts it server-side: NetBird has no policy tests, so a rule narrowed by mistake applies cleanly and only fails in traffic. Re-check the live policy before believing a cross-node networking bug is a node-local fault, and re-check it in particular after any dashboard edit, which tofu will not know about until the next plan.

The failure is worth recognising by shape, because it does not look like a policy problem. Half of all ClusterIP DNS answers time out while direct-to-pod-IP works, because CoreDNS runs one pod per node and kube-proxy load-balances 50/50, so only the remote endpoint is unreachable. Anything that resolves DNS at startup then fails intermittently or crashloops. The test:

# from a pod, against a pod on the *other* node
ping -c2 <remote pod IP>   # fails => overlay is dropping, check the all-protocol rule

The mesh watchdog

What repairs a node’s mesh client when nothing outside the node can reach it, how far the repair ladder goes, and where it stops. Read this when a healthchecks.io check goes red, or before tuning any netbird_watchdog_* default.

A mesh node is reached over the mesh. ansible_host in ansible/inventory/group_vars/all/main.yml resolves to <hostname>.<mesh_dns_domain> for any node with node.mesh, and k8s_cluster pins kubelet’s node-ip and k3s’s advertise-address to the mesh address. So a NetBird client that breaks takes down the cluster’s view of the node and Ansible’s route to it, in the same instant. There is no plane left to fix it from.

That is what the watchdog is for. Every part of it runs on the node, decides locally, and needs nothing from outside, including the reporting.

ansible/roles/netbird installs it on every node the role runs on. Nothing is per-node: a future node opts in by setting node.mesh: true, exactly as it does for the join itself.

What runs

The drop-in is not part of the ladder and does the most common repair on its own. Plain process death is systemd’s job. What needed overriding is the start limit, which by default gives up after repeated fast crashes and leaves netbird.service permanently failed, on a node whose only route back in is the mesh that daemon carries.

The probe

One netbird status --json, read with jq:

• .management.connected and .signal.connected, the control plane.
• .netbirdIp non-empty, meaning the interface is actually addressed.
• Peers whose .lastWireguardHandshake falls inside netbird_watchdog_handshake_max_age, the dataplane. A peer can report Connected with a long-dead tunnel, so freshness is read off the handshake rather than off the status field. NetBird emits that timestamp as RFC3339 with nanosecond precision and a numeric UTC offset, so it is parsed with date -d. jq’s fromdateiso8601 accepts only whole-second Z timestamps and throws on it, which once counted every peer stale and put the whole fleet in degraded at the same time.

That yields one of three states. A node with no peers at all is ok, not degraded, because there is nothing for it to hold a tunnel to.

The ladder

Rungs are counted in consecutive non-ok probes, so each is also a duration. At the default 2min interval:

Only one rung fires per netbird_watchdog_action_cooldown (10min), so a repair gets several probes to take effect before the next escalates on top of it. A single ok probe resets the counter to zero.

The re-up rung deliberately carries no --setup-key. It re-applies the configuration already on disk and cannot re-enrol a peer the account has forgotten.

The reboot guards

All four must pass:

• netbird_watchdog_reboot is true. It defaults to true, because the nodes this exists for are the ones nobody can reach to reboot by hand. Set it false in ansible/nodes/<host>/host.yml to cap the ladder at re-up, which is worth doing while first observing a node.
• Uptime is above netbird_watchdog_reboot_min_uptime (30min). If the mesh is broken this soon after boot, another reboot is not what fixes it.
• No reboot within netbird_watchdog_reboot_cooldown (24h). The timestamp lives under /var/lib, so it survives the reboot it is guarding.
• The node has a default route. Without one, NetBird is a symptom and not the fault, and rebooting would take the node down without addressing it.

The reboot is the last thing the script does, after the state file, the metrics and the ping are all written or delivered. Otherwise the reason for the reboot dies with the boot.

After a reboot the failure counter resets, so the ladder gets a fresh restart and re-up run. If that also fails it climbs back to the reboot rung, the 24h cooldown blocks it, and the node goes stuck.

Stuck, and why no credential lives on a node

stuck is terminal by construction. The watchdog logs at ERROR, publishes futhark_mesh_state 3, fails its healthchecks.io check, and then does nothing further.

What is left at that point is a peer the account no longer has, or a local configuration too broken to re-up, and both need a NetBird credential to fix. None is on the node, deliberately. roles/netbird mints a one-off, group-scoped setup key that expires in 300 seconds and hands it to a single netbird up; storing a reusable key or the enrollment PAT on every node would make each node a peer-enrollment credential at rest, permanently, to close a failure mode that ends in a human either way. The recovery is a converge from the operator machine, over the node’s other route in.

That is the trade this design accepts: a node that loses its registration entirely stays down until someone reaches it.

Reporting

Three channels, in descending order of how much you can trust them during an actual outage.

healthchecks.io. Each node pings its own check over the public internet, the one path that does not go through the thing being watched. A healthy probe pings the base URL, a repair posts to /log, and stuck posts to /fail. The two failure pings carry a plain-text body, kept by healthchecks.io with the ping and readable from the check’s page:

host=kenaz state=3 action=none failures=13
management=0 signal=0 peers=0/3 ip=none
last_action=reboot at=2026-08-10T19:07:09Z last_reboot=2026-08-09T04:12:55Z
management_error=dial tcp: i/o timeout
signal_error=context deadline exceeded

It is the only copy of the reason that survives a node that then reboots or stays unreachable, and it is what tells a stuck check apart from a node that simply went quiet. The URL comes from secrets.healthchecks[<hostname>], one item with one field per node, resolved by just ans render-secrets and written to /etc/futhark/mesh-watchdog.env at mode 0600. A node with no field still runs the ladder. It just reports nothing. Replacing a ping URL is Credential rotation.

The journal. journalctl -u futhark-mesh-watchdog on the node, with one line per unhealthy probe and per action taken.

Metrics. futhark_mesh_*, written atomically to /var/lib/node_exporter/textfile_collector/futhark_mesh.prom and picked up by node-exporter’s textfile collector, through the hostPath mount in infra/monitoring/app/exporters/node-exporter.yaml. Rules in infra/monitoring/app/grafana/alerting/mesh.yaml cover unhealthy, stuck, self-healed, and a watchdog that has stopped running. These are second-class on purpose: they are scraped over the mesh, so a hard failure takes this path down with it. What they add is the shape of a failure, and notice of the repairs that worked, which otherwise leave no trace anywhere.

futhark_mesh_repairs_total is the one to watch over time. A node repairing itself is the system working. A node repairing itself every day has an underlying fault the watchdog is papering over.

Tuning and testing

Every threshold is a default in ansible/roles/netbird/defaults/main.yml, overridable per node in ansible/nodes/<host>/host.yml.

To exercise the ladder on a node, with a shell already open on a route that is not the mesh:

This can reboot the node. Set netbird_watchdog_reboot: false for that node first, and do it on a worker before a controller.

systemctl stop netbird                    # the drop-in alone brings it back in ~10s
systemctl mask netbird; systemctl stop netbird   # the drop-in cannot act, so the ladder climbs
journalctl -fu futhark-mesh-watchdog
systemctl unmask netbird                  # before the reboot rung's window

Verify the recovery: netbird status reports connected, futhark_mesh_state returns to 0, and the node’s healthchecks.io check goes green within one probe interval.

The egress exporter

What publishes the cluster’s public address as a metric, and why the lookup runs on the node rather than in a pod. Read this before changing the Egress widget on Glance’s network page, or if that widget goes blank.

ansible/roles/egress_exporter runs on any node with node.public_ingress, under the metrics tag. Nothing in it branches on a hostname.

Why it is not a pod

The dashboard’s Egress widget used to call ifconfig.co from the Glance pod. That answers with the public address of whichever node the pod is scheduled on, which is not the node traffic arrives at: both Traefik releases are pinned to the ingress node with a nodeSelector, and Glance is not. The number displayed was real and belonged to the wrong host.

Asking from the ingress node itself removes the ambiguity, and there is already a channel for a node to publish a fact about itself: the textfile collector, which ansible/roles/netbird uses for the mesh watchdog’s metrics.

What runs

The script writes futhark_egress_ip_info{ip,asn_org,country} 1 and futhark_egress_last_run_timestamp_seconds to /var/lib/node_exporter/textfile_collector/futhark_egress.prom. That directory is the hostPath mounted into the node-exporter DaemonSet in infra/monitoring/app/exporters/node-exporter.yaml; the path is spelled in egress_exporter_textfile_dir and both spellings have to agree.

It writes to a temporary file in the same directory and moves it into place, because node-exporter reads that directory on every scrape and would otherwise be able to read a half-written file.

A failed lookup exits non-zero and leaves the previous file untouched, rather than publishing an empty one. A transient outage of the lookup service therefore reads as a stale address, not as the address having gone away.

Verify

On the node:

systemctl list-timers futhark-egress.timer
sudo /usr/local/bin/futhark-egress.sh
cat /var/lib/node_exporter/textfile_collector/futhark_egress.prom

The file should hold one futhark_egress_ip_info line with a non-empty ip label.

From a mesh device, confirm the series arrived:

curl -s "https://metrics.$SUB_INTERNAL.$DOMAIN/api/v1/query?query=futhark_egress_ip_info"

The address is a label value, not a sample value, so the sample is always 1 and the reading is in the labels. That is also why the metric carries no address in this repository: it is produced at runtime and never committed.

Bootstrap and reconciliation

How Flux gets installed, what it takes over, and where the handoff from Ansible sits. Read this when a bootstrap fails part way, or when you need to know why something is applied rather than reconciled.

flux/ is the GitOps entrypoint. flux/cluster.yaml, the FluxInstance CR, is applied once by ansible/roles/flux_bootstrap, and Flux does not reconcile the flux/ directory itself, to avoid watching its own bootstrap.

Everything else under flux/ is reconciled. flux/infra/ks.yaml and flux/nodes/ks.yaml are the two Flux Kustomization CRs that point Flux at the matching repo-root directories, ./infra and ./nodes. Those two keep their full spec rather than being patched like every other ks.yaml, because flux/ has no kustomization.yaml of its own. Flux auto-generates one from cluster.yaml’s sync.path: flux, and adding a real one would pull cluster.yaml itself into reconciliation. The rest of the naming and layout rules are in Layout and naming.

Bootstrap sequence

Run by just ans k8s (ansible/playbooks/k8s.yml):

1. k8s_cluster: render /etc/rancher/k3s/config.yaml from inventory, install k3s on the controller and then the workers, and fetch the kubeconfig into ansible/.generated/. The local-path StorageClass comes up with it, from k3s’s own bundled provisioner: monitoring, auth and nodes/kenaz.k8s/actual all bind PVCs on their first reconcile, and nothing in the Flux-managed tree can provision a StorageClass for itself. See Startup ordering.
2. flux_bootstrap:
   1. Install the Flux Operator via Helm.
   2. Apply the flux-system/git-deploy-key Secret, with known_hosts built from GitHub’s published host keys rather than a blind ssh-keyscan.
   3. Create the namespaces the next step writes into, since Flux does not exist yet to declare them. Flux takes ownership of all of them on its first reconcile.
   4. Apply the two Secrets that exist to seed what Flux resolves for itself, and so cannot come from Flux: flux-system/sops-age and infisical-universal-auth in each tier namespace. See Secrets.
   5. Wait for the Flux Operator to be ready.
   6. Apply flux/cluster.yaml. Flux takes over from here.

There is no follow-up step. Everything past flux/cluster.yaml is Flux reconciling git.

Verify the handoff:

just fx sources    # the GitRepository is Ready at the pushed revision
just fx failing    # empty

The handoff, and the one line it never crosses back over. Green is the handoff point itself, the moment Ansible stops and Flux owns the cluster. The two Kustomization labels are paths under flux/.

Every Kustomization carries a decryption block naming flux-system/sops-age, patched in once via infra/kustomization.yaml and nodes/kustomization.yaml. flux/infra/ks.yaml and flux/nodes/ks.yaml state it in full for the same reason they state everything else in full.

Binary assets and .sourceignore

source-controller applies a built-in exclude list when it packs the artifact, and that list drops *.png along with the other image and archive extensions. A PNG committed to the repository is therefore absent from the tree kustomize-controller builds, even though kustomize build and the pre-commit hooks pass locally. The failure surfaces as a build error naming a file that plainly exists:

kustomize build failed: ... open .../web-app-manifest-192x192.png: no such file or directory

.sourceignore in the repository root is read after those defaults, so a ! line re-includes what they excluded. It currently re-includes the Glance app icon under config/branding/. Add a line there when a component starts mounting any other binary, and confirm with just fx get that the consuming Kustomization reconciles.

Day-to-day

just fx get        # every Kustomization and its sync status
just fx failing    # only what isn't Ready
just fx reconcile  # force-reconcile everything, or `<name>` for one
just fx logs       # tail kustomize-controller

The full list is in Recipe reference.

Cluster infrastructure

What each cluster-wide component is, and the four that behave unlike the rest: the two ingresses, monitoring, and the per-tier Infisical operator. Layout rules are in Layout and naming, and the order they come up in is Startup ordering.

infra/ holds one directory per component, each with its own Flux Kustomization.

The two ingresses

Two releases, not one, because they answer on different addresses under different trust assumptions. Both run hostNetwork: true on the same node, and that is why they are the odd ones out everywhere else in the tree. There is no LoadBalancer to hand either a Service address. MetalLB was considered and rejected, since it can only manage a real L2 or BGP-announced IP, not a mesh one.

traefik-internal binds 443 on ${MESH_IP}, the ingress node’s mesh address. Give an internal Ingress the class internal. It never carries its own tls: block, because the wildcard is served as the default certificate. It has no Service at all, and tofu/netbird points the internal wildcard record straight at that address. A peer address needs no route, which is the whole reason for the shape: the mesh reaches it natively.

It used to be a pinned ClusterIP reached over a NetBird route that advertised the entire service CIDR. That path failed silently, with the route reporting Selected on the client and no packet ever crossing, and it cost a routing peer, masquerade, and a pinned address that had to stay inside k8s_service_cidr. None of that exists now.

traefik-edge binds 443 on ${PUBLIC_IP}, and its dashboard and metrics entryPoints on ${MESH_IP}, so those stay off the public interface. It cannot bind 80: the ingress node’s net.ipv4.ip_unprivileged_port_start is lowered only as far as 443 (ansible/roles/firewall_ingress), and neither release binds a plaintext port. traefik-internal takes 8082 for its ping entryPoint because traefik-edge already holds 8081 on the mesh address. They share one network namespace, so every port either one binds is a port the other cannot.

hostNetwork means CNI NetworkPolicy enforcement never sees either release’s sockets, so neither the ingress-edge nor the ingress-internal baseline governs that traffic. What governs it is firewalld, the fact that the mesh address is only reachable from the mesh, and Traefik’s own rate limiting. It is also why both netpol-allow-from-ingress-* templates are an ipBlock and not a namespaceSelector. See Network policy.

Both addresses are substituted by postBuild.substituteFrom from the SOPS-encrypted cluster-values Secret in config/sops/. That Kustomization has no dependsOn on purpose: a substitution target has to exist before its consumers reconcile, and infra-policies, the obvious home for it, depends on traefik-edge.

These two releases are the Secret’s only consumers of those keys, and the reason they have to be: hostIP in a pod spec takes no fieldRef, so an address bound there has to be written down. Where the same address is needed as data rather than as a bind target, it is discovered instead. Both Traefik scrape jobs in infra/monitoring read it off the API server, since a hostNetwork pod’s status.podIP is the kubelet’s node-ip, which config.yaml.j2 sets to the mesh address.

Auth

Two Deployments in one namespace, doing two different jobs.

Pocket ID is the OIDC provider, edge-exposed at auth.$DOMAIN because it is the login page for everything. It is pinned to ogma on single-writer SQLite, so its Deployment uses strategy: Recreate and never runs two pods at once. Apps that speak OIDC talk to it directly and need nothing else.

oauth2-proxy exists for the apps that do not. It is a single relying party registered as one Pocket ID client, exposed internally at sso.$SUB_INTERNAL.$DOMAIN, and it publishes the Traefik Middleware auth-sso@kubernetescrd. Any internal Ingress that names that middleware gets a login. See Internal ingresses are unauthenticated by default for how to opt an app in.

Four settings in app/oauth2-proxy-configmap.yaml carry the design, and changing any of them changes what the reader sees:

• OAUTH2_PROXY_UPSTREAMS: static://202 with OAUTH2_PROXY_SKIP_PROVIDER_BUTTON: "true" is oauth2-proxy’s documented Traefik static-upstream setup. The middleware calls the proxy’s root path, which answers 202 on a valid session and 302 to Pocket ID otherwise. Because the provider button is skipped, no oauth2-proxy-branded page is ever rendered: the only login UI is Pocket ID’s own.
• OAUTH2_PROXY_COOKIE_DOMAINS: .$SUB_INTERNAL.$DOMAIN is what makes one login cover every protected host. OAUTH2_PROXY_WHITELIST_DOMAINS has to match, since it bounds where the post-login redirect may send the browser.
• OAUTH2_PROXY_COOKIE_NAME uses the __Secure- prefix, not __Host-. The __Host- prefix forbids a Domain attribute, and a Domain attribute is exactly what shares the session across subdomains.
• OAUTH2_PROXY_CODE_CHALLENGE_METHOD: S256 has to be set, because every client in tofu/oidc/clients.tf is minted with pkce_enabled. oauth2-proxy sends no code_challenge unless asked, and Pocket ID answers an authorize request without one with invalid_request. That surfaces as a 403 on the protected host reading Login Failed: The upstream identity provider returned an error, not as a startup failure, so it appears only on the first login.

The gate itself is binary: OAUTH2_PROXY_ALLOWED_GROUPS admits administrators and users, and which of the two the reader is in changes nothing about whether the request gets through.

The identity is still available to a backend that wants it. OAUTH2_PROXY_SET_XAUTHREQUEST is on, so oauth2-proxy returns X-Auth-Request-User, -Email, -Preferred-Username and -Groups, and app/middleware-sso.yaml forwards all four. Traefik strips each of those headers from the incoming request before writing the auth response’s value, which is what stops a client from forging one. infra/copyparty is the only backend reading them today. An app that needs roles enforced by the proxy rather than by itself still has to speak OIDC directly.

Three secrets at /infra/auth feed it. SSO_OIDC_CLIENT_ID and SSO_OIDC_CLIENT_SECRET are minted by tofu/oidc (see oidc). SSO_COOKIE_SECRET is seeded by hand, once:

openssl rand -base64 32 | tr -- '+/' '-_'

Store that in Infisical at /infra/auth before the Deployment first reconciles. Without it the InfisicalStaticSecret template renders an empty value and oauth2-proxy refuses to start.

Monitoring

One Flux Kustomization, four workloads, one directory each under infra/monitoring/app/: metrics/ (vmsingle + vmagent), logs/ (vlsingle + fluent-bit), exporters/ and grafana/. Only helmrepositories.yaml stays flat, since every one of them draws on it.

Alert rules are in grafana/alerting/, one file per group (watchdog.yaml, node-health.yaml, kubernetes.yaml, mesh.yaml, backup.yaml) plus contactpoints.yaml and policies.yaml. They are ordinary Grafana provisioning YAML: write {{ $labels.instance }} as you would in the UI. Nothing escapes it, because nothing templates it. kustomization.yaml generates them into one ConfigMap labelled grafana_alert: "1", and the chart’s alerts sidecar copies it into /etc/grafana/provisioning/alerting/ and POSTs Grafana’s reload endpoint, so an edit lands on the next reconcile without restarting Grafana.

They used to live in the HelmRelease values, where the chart ran the whole block through Helm’s tpl and every {{ $labels.x }} had to be written {{ "{{" }} $labels.x {{ "}}" }} to survive it. If you ever put alerting back into values, that escaping comes back with it.

The sidecar is also why the release sets rbac.namespaced: true with an explicit extraRoleRules. The chart’s default hands it configmap and secret read across the whole cluster, and the chart’s own Role omits the rule for the alerts sidecar specifically.

The one thing that still needs the double-$ form is $${SLACK_WEBHOOK_URL} in contactpoints.yaml. That guards against Flux, not Helm: postBuild substitution runs over the generated ConfigMap and would blank an undefined ${VAR}. $$ escapes it to the literal that Grafana’s own env expansion then reads, out of the secret named in Secrets.

watchdog.yaml is the dead-man’s switch, and it sets its own group_wait, group_interval and repeat_interval for a reason. A rule’s interval only decides how often the alert is evaluated; how often a still-firing alert re-notifies is policy-side, and the inherited defaults (5m and 4h) mean the healthchecks.io ping lands about every four hours. Both go to 1m to match the rule, because Alertmanager flushes on group_interval ticks and requires repeat_interval to be at least as long.

Dashboards are in grafana/dashboards/, one JSON file each, generated into one ConfigMap per dashboard labelled grafana_dashboard: "1" and annotated grafana_folder with the Grafana folder to file it under. The chart’s dashboards sidecar writes each into /var/lib/grafana/dashboards/<folder>/, and foldersFromFilesStructure turns that directory back into the folder name. allowUiUpdates: false, so Grafana refuses a UI edit rather than accepting one the next sync would overwrite.

They are reconciled by their own Flux Kustomization, infra/monitoring/dashboards-ks.yaml, whose only distinguishing feature is that it has no postBuild. Dashboard JSON is full of Grafana’s own ${namespace}-style interpolations. postBuild substitution would blank the ones it reads as undefined cluster variables, and on ${__field.labels.node} it does not get that far: the build fails with envsubst error: variable substitution failed: missing closing brace. Do not fold this directory back into the monitoring Kustomization, and do not escape the JSON to make that possible.

Adding a dashboard is three steps: drop the JSON in grafana/dashboards/, add a configMapGenerator entry for it, and point grafana_folder at a folder. Pin the datasource by its provisioned uid (victoriametrics or victorialogs) instead of shipping a datasource template variable, so a dashboard cannot be pointed at the wrong store by a stray dropdown.

pod-logs.json is the dashboard to reach for when reading a workload’s logs. Its namespace, pod and container dropdowns come from kube_pod_info and kube_pod_container_info in VictoriaMetrics, so they list every pod rather than only the ones that logged in the window. Its Level dropdown holds a regexp alternation, Error being error|fatal, matched against the level field that the collector sets on every line — see Log levels. Anything typed into Search is ANDed onto every panel’s query.

vmagent’s scrape targets are in metrics/scrape-configs.yaml, merged into the release with valuesFrom rather than kept inline, so adding a target does not mean editing a HelmRelease. One file, not one per job: Flux merges valuesFrom entries with arrays replaced, so two ConfigMaps each holding scrape_configs would silently clobber each other.

The cadvisor and kubelet jobs scrape each node’s kubelet directly on port 10250, not through the API server’s /api/v1/nodes/<node>/proxy/ path. The proxy path authorizes against nodes/proxy, which the vmagent chart’s ClusterRole does not grant, so it answered 403 and collected no container_* metrics at all. A direct scrape authorizes against nodes/metrics, which the chart does grant. Both jobs relabel node and instance from the discovered node name, because cadvisor carries neither and the dashboards select on node. The kubelet job keeps only kubelet_volume_stats_*; the full endpoint is around 76,000 samples per node per scrape.

config.global.external_labels stamps cluster on every series. It exists because the Kubernetes dashboards select cluster="$cluster" on every query and nothing else here emits that label.

Retention and volume size for both stores are in infra/substitutions/app/monitoring-sizing.yaml, reaching the releases as ${VM_RETENTION} and friends. They sit together because they are one decision against one local-path disk. CPU and memory are not there. Those are per-workload and stay next to the release that sets them. The file cannot live under infra/monitoring/: a substituteFrom source has to exist before its consumer reconciles.

Log levels

Every line reaching VictoriaLogs carries a level field, one of trace/debug/info/warn/error/fatal, set by the Fluent Bit collector in infra/monitoring/app/logs/fluent-bit.yaml. A workload that logs JSON has its own level (or lvl, or severity) normalised; for the rest, a Lua filter reads the severity out of the front of the message — klog’s I0812, zap’s tab-delimited INFO, zerolog’s WRN, colour codes stripped first — and defaults to info when there is none.

That derivation is deliberately at ingest rather than in each query. Matching severity words against message text is what made gatus’ healthy errors=0 heartbeat read as an error, and every dashboard and widget had to repeat the same expression to be wrong in the same way. The collector is Fluent Bit rather than VictoriaLogs’ own vlagent only because vlagent cannot transform what it ships.

Host logs

The collector also tails one file from the host itself, /var/log/fail2ban.log. It works with no shipper on the node and no route from the host into the cluster, because the DaemonSet already mounts each node’s /var/log read-only.

Query the bans in Grafana against the VictoriaLogs datasource as app:fail2ban. A record_modifier filter attaches app and hostname, so events stay attributable per node.

The other half, the jails and why fail2ban logs to a file at all, is in Inventory and roles.

Glance

The dashboard at home.$SUB_INTERNAL.$DOMAIN, behind auth-sso@kubernetescrd. Four pages: home, apps, cluster, network. It holds no state, so there is no PVC and no K8up Schedule entry: the widget cache is in memory and the todo widget’s items are in the reader’s browser.

It is two Flux Kustomizations, and the split is the one thing to understand before editing it.

glance reconciles app/ with postBuild substitution, the way every other component does. glance-config reconciles config/ without it, for the same reason infra/monitoring/dashboards-ks.yaml does: Glance’s own environment variable syntax is ${VAR}, byte for byte what Flux’s envsubst consumes, and every API token and hostname in those files is one. Under substitution they would all be read as unset cluster variables.

So the values reach Glance as real environment variables instead. app/deployment.yaml is in the substituted Kustomization and spells ${SUB_INTERNAL} and ${DOMAIN} there once; the config files read them back at runtime. The API tokens arrive the same way, from the glance-secrets Secret.

Three consequences worth knowing before a config edit fails in an unhelpful way:

• Glance exits on a variable that does not resolve. A new ${SOMETHING} in a config file means adding SOMETHING to Infisical at /infra/glance first, or the pod crash-loops.
• Substitution runs over comments too. Writing the literal string ${VAR} in a YAML comment is enough to fail startup with parsing variable: environment variable VAR not found.
• An $included file must open with its own list marker and indent the rest under it, whether it is a page (- name: Home) or a single widget (- type: custom-api). Glance splices the file at the $include line and drops the - that was there, so a file written as a bare mapping silently merges into the item before it.

Config layout

glance.yml is the entrypoint and $includes one file per page. Each page file then $includes one file per custom-api widget, named widget-<thing>.yml, so a query change is a one-file diff. Built-in widgets that need no template (clock, calendar, weather, markets, bookmarks, rss, group, releases, repository) stay inline in the page file: a page file reads as layout, a widget file reads as logic.

Two constraints shape that:

• The files are flat, not in a widgets/ subdirectory. They all become keys of one ConfigMap, a ConfigMap key cannot contain a slash, and $include only resolves against the single directory they mount into.
• Kustomize does not glob. A new widget file has to be listed in configMapGenerator.files in infra/glance/config/kustomization.yaml or it does not reach the pod at all, and the $include fails at startup.

The logo and icons come from config/branding, pulled in as a Kustomize Component that generates the glance-assets ConfigMap. It is mounted at /app/assets and served at /assets/ through server.assets-path.

Validate a config change without a cluster:

podman run --rm \
  -e SUB_INTERNAL=in -e DOMAIN=example.eu \
  -e WAQI_TOKEN=x -e GITHUB_TOKEN=x -e NETBIRD_API_KEY=x \
  -e GATUS_URL=http://gatus.gatus.svc.cluster.local:8080 \
  -v ./infra/glance/config:/app/config:ro,Z \
  -v ./config/branding/logo:/app/assets:ro,Z \
  docker.io/glanceapp/glance:v0.8.5 config:validate

The assets mount is required: Glance refuses to start when assets-path points at a directory that does not exist, and config:validate enforces that too. Exit status 0 means the YAML parses, every $include resolved and every widget’s options are valid. It does not fetch anything, so a broken PromQL query or a wrong metric label still only shows up in the browser. config:print with the same arguments prints the spliced result, which is where a widget file missing its leading - becomes obvious.

Where the widgets get their data

Most widgets on the apps, cluster and network pages are a custom-api query against vmsingle over the cluster network, which is what infra/policies/namespaces/monitoring/netpol-allow-from-glance.yaml opens — on 8428 for vmsingle, and on 9428 for the VictoriaLogs error-count widget. Two of them need scrape jobs that exist only for them: flux for gotk_reconcile_condition, and cert-manager for certmanager_certificate_expiration_timestamp_seconds. Both are in infra/monitoring/app/metrics/scrape-configs.yaml.

The error-count widget lists the eight pods that logged the most error lines in the last fifteen minutes, and each row links into the pod-logs Grafana dashboard with that namespace and pod, the Error level and the same fifteen-minute window already selected.

The two widgets on the apps page that show service health read Gatus instead, at http://gatus.gatus.svc.cluster.local:8080, admitted by infra/policies/namespaces/gatus/netpol-allow-from-glance.yaml. Glance probes nothing itself. See Gatus.

The egress widget on the network page reads futhark_egress_ip_info, which the ingress node publishes through node-exporter’s textfile collector. It used to call ifconfig.co from the Glance pod, which answered with whatever node Glance happened to be scheduled on rather than the one traffic actually arrives at. See The egress exporter.

The backups widget shows job outcomes and nothing else. K8up’s per-repository gauges, including k8up_backup_restic_available_snapshots, are only ever pushed to a Prometheus Pushgateway by the backup job itself, and there is no Pushgateway here. Ask the repository directly with just bak snapshots.

Gatus

Every healthcheck in the cluster, and the status page at status.$SUB_INTERNAL.$DOMAIN, behind the same auth-sso@kubernetescrd as Glance. Plain manifests in infra/gatus/app, storage.type: postgres against the shared instance, so there is still no PVC and no K8up Schedule entry. It was memory until the migration and lost every result on restart.

Add a check by adding an endpoint to infra/gatus/app/config.yaml. That file is substituted by Flux, unlike Glance’s config, so ${SUB_INTERNAL} and ${DOMAIN} in it are filled in.

Two expansions run over that one file, and the difference matters when you write a new value into it. Flux’s postBuild goes first, over the generated ConfigMap; Gatus runs os.ExpandEnv over the same bytes when it loads them. A placeholder meant for Gatus has to survive the first pass, so double the dollar: $${GATUS_DB_URL} reaches the ConfigMap as ${GATUS_DB_URL} and Gatus fills it from the environment. That is how the database password gets into storage.path without being committed.

The cost is a rule about the whole file, comments included: never write a dollar sign followed by an empty brace pair. envsubst reads every byte it is given, reads that as a variable with no name, and fails the Kustomization with envsubst error: variable substitution failed: unable to parse variable name. The build stops there, so the symptom is the whole component going False, not one bad endpoint.

Each endpoint is probed over its public hostname through traefik-internal rather than over a cluster Service. That is deliberate: the path being tested then includes DNS, the mesh route, the Traefik router and the certificate, which is where failures actually are. Glance’s own endpoint expects a 302, since an unauthenticated probe of a host behind the SSO middleware is redirected to Pocket ID; a 200 there would mean the login had stopped being enforced.

This is why infra/coredns exists and why gatus dependsOn it. Without the stub zone, none of those hostnames resolve from inside the cluster and every check fails at DNS. See Who resolves the internal subdomain.

Copyparty

The file manager at files.$SUB_INTERNAL.$DOMAIN, and the only way to read what is on the two rclone remotes without an rclone config on your own machine. Both are crypt-wrapped, so the bytes are meaningless anywhere else.

It mounts each remote at its root, not at the per-PVC subdirectory infra/storage’s StorageClasses hand out. A StorageClass cannot express a fixed path, since its remotePath is a template over the claim’s namespace and name, so app/pv-storagebox.yaml and app/pv-gdrive.yaml are static PersistentVolumes naming the same driver and the same csi-rclone/storagebox-secret, with remotePath: "". Each sets storageClassName: "" and a claimRef, which is what binds it to its claim and keeps a provisioner out.

Understand the blast radius before logging in. At the crypt root, every other app’s remote directory is visible and writable, including actual/actual-user-files. Those directories are reclaimPolicy: Retain but not in K8up’s backup set, because the Storage Box snapshots itself. The Drive volume is narrower by accident of its credential: the OAuth client is scoped to drive.file, so Google hides every file the cluster did not create, and nothing in the cluster writes there yet.

Copyparty speaks no OIDC. It reads the identity out of the headers auth-sso@kubernetescrd forwards, which is what idp-h-usr and idp-h-grp in app/configmap.yaml name, and maps the two fleet-wide Pocket ID groups onto volume permissions: administrators may write and delete, users may read. Two settings make that safe rather than decorative. xff-src: lan tells copyparty which source addresses may assert those headers at all, and Traefik drops any client-supplied copy before setting its own. Remove auth-sso from the Ingress and every visitor is anonymous with no access to either volume.

Its SQLite index and thumbnail cache sit on a local-path PVC, redirected there by hist. That is the same rule as actual-server-files: SQLite does not belong on a network filesystem. It also pins the pod to a node, which the rclone mounts follow. Nothing enables e2dsa or e2ts — both walk every file, and the media scan reads every byte back down through the remote.

The config is a plain ConfigMap the process reads once at startup, so an edit needs a restart:

kubectl -n copyparty rollout restart deploy/copyparty

The shared database

One PostgreSQL instance serves every service that needs one, rather than each app bringing its own. infra/postgres/app installs the CloudNativePG operator into cnpg-system; infra/postgres/config holds the Cluster in postgres, and one pair of CRs per tenant.

It runs instances: 1, pinned to kenaz. local-path is node-local storage, so a replica means a second volume on ogma and every tenant’s database following the edge node’s uptime. The cost is that restarting or upgrading this Cluster is a short outage for every tenant at once.

Five tenants: Linkwarden, Grafana, Open WebUI, Pocket ID and Gatus. The first four were on SQLite on their own PVC, which k8up copied nightly while the process was writing to it, and that is why they moved: infra/backup/config/schedules.yaml already warned that copying a live data directory with restic produces a snapshot that fails at restore time, where a pg_dumpall is consistent by construction. Gatus is the exception and gained something instead of trading it. It was storage.type: memory, with no persistence at all, so it now keeps its history across a restart.

Read the last of those five twice before changing anything about it. Pocket ID is the cluster’s identity provider, and it now depends on a single-instance database on the other node. It runs on ogma and used to survive kenaz being down entirely; it no longer does, and while this Cluster is restarting nobody can log in to anything. Gatus is in the same position and is worse placed to be, since the status page is unavailable in exactly the outage it exists to report. Alerting does not run through Gatus, so a broken cluster still pages. Both were accepted knowingly, in exchange for a backup that restores.

There is no superuser password anywhere, in the cluster or in Infisical. enableSuperuserAccess is false, every tenant authenticates as its own role, and the two things that do need superuser rights, the nightly dump and its replay, run inside the pod over the local socket where peer authentication already identifies them.

Giving a service a database

Four things, and all four are needed:

1. A DatabaseRole and a Database in infra/postgres/config/database-<app>.yaml. The role owns that database and nothing else, and holds no createdb, createrole or superuser, so a leaked credential reaches one tenant’s rows. Both carry a retain reclaim policy: removing the manifest stops managing the object rather than dropping it. Give the role and the database a name with no hyphen in it, whatever the namespace is called: the name goes into CREATE ROLE and into a connection URL, and a hyphen is legal in neither unquoted. open-webui is openwebui here.
2. A target in infra/postgres/config/infisicalsecret.yaml producing a kubernetes.io/basic-auth Secret whose username matches the role. Label it cnpg.io/reload: "true" or a rotated password only lands at the next reconciliation.
3. infra/policies/namespaces/postgres/netpol-allow-from-<app>.yaml, opening port 5432 to that namespace. Without it the app resolves the Service and hangs. That file is also what just bak pg-restore reads to decide whose workloads to scale down, so write it even when the traffic is already allowed by something broader. monitoring is the case that proves it: the shared allow-from-monitoring template names no ports at all, so Grafana could always reach 5432, but without a file saying 5432 explicitly the restore would have left it writing through a pg_dumpall --clean replay.
4. The app’s own connection string, assembled in its InfisicalStaticSecret template from the password. nodes/kenaz.k8s/linkwarden/app/infisicalsecret.yaml is the worked example. What the variable is called is the app’s business: DATABASE_URL for Linkwarden and Open WebUI, DB_CONNECTION_STRING for Pocket ID, GATUS_DB_URL for Gatus. Grafana is the exception and needs no template, because grafana.ini spells the host, database and user in git and reads only the password from the environment.

The password is filed twice, once in /infra/postgres and once in the app’s own folder, and that is the admission policy working rather than an oversight: an InfisicalStaticSecret may only name a path inside its own namespace’s tier, and no two of these namespaces share a folder even where both are infra tier. Rotating means changing both, and Rotating a credential has the per-tenant key names. Generate it from letters and digits only, because it is interpolated into a URL and anything needing percent-encoding parses wrong.

Infisical operator

The operator is installed once per tier, and that is the isolation mechanism rather than a deployment detail. Each install sets scopedRBAC: true with its own scopedNamespaces, so the chart emits a Role/RoleBinding per listed namespace and no cluster-wide secrets ClusterRole. A tier’s ServiceAccount has no permissions anywhere outside its own list.

• infra tier: infra/infisical-operator/app/helmrelease-infra.yaml, release namespace infisical-infra. Owns the secrets.infisical.com CRDs (installCRDs: true).
• node tier: helmrelease-node-<hostname>.yaml, release namespace infisical-node-<hostname>, installCRDs: false and dependsOn the infra release, because two installs racing to own the same CRDs is the documented failure mode.
• backup tier: helmrelease-backup.yaml, release namespace infisical-backup, scoped to itself and k8up. Not per-host, and the only tier with a machine identity of its own.

Each tier’s namespace appears first in its own scopedNamespaces, and not by accident: that is what lets the operator read the InfisicalAuth and credential Secret it authenticates with. The infra and node tiers share one Infisical machine identity, because the free tier caps identities at five, so what separates them is RBAC plus the ValidatingAdmissionPolicy in config/, which pins each InfisicalStaticSecret’s secretPath to its namespace’s tier. The backup tier goes further and authenticates as a second identity, because the admission policy alone would let any infra namespace read /infra/k8up and that is the password that decrypts every backup. The reasoning, and what still defeats it, is in Secrets.

Adding a node tier

1. Add infra/namespaces/app/namespaces.yaml entries for the tier’s own namespace and that node’s app namespaces. The chart’s scoped Roles are written into namespaces it does not create, so the install fails outright if any of them is missing.
2. Add infra/infisical-operator/app/helmrelease-node-<hostname>.yaml, copying the kenaz one, swapping the hostname and listing those app namespaces in scopedNamespaces, then register it in the sibling kustomization.yaml.
3. Add infra/infisical-operator/config/nodes/<hostname>.yaml for the tier’s InfisicalConnection and InfisicalAuth, and list it in that kustomization.yaml.
4. Copy infra/policies/namespaces/infisical-node-kenaz/ to the new tier’s namespace and register it in infra/policies/kustomization.yaml. The tier namespace holds that tier’s copy of the universal-auth credential, so it is the last place to leave without a baseline.
5. Set app_tier: true in ansible/nodes/<hostname>/host.yml, so flux_bootstrap seeds the credential into the new namespace. The list of tiers is derived from that flag rather than written out, so there is nothing to keep in step with step 1.

Verify: after just ans k8s and a push, just fx failing is empty and kubectl -n infisical-node-<hostname> get infisicalauth reports ready.

Node apps

What lives under nodes/, how it differs from ansible/nodes/, and which secrets a node app may read. The procedure for adding one is Adding a node app.

nodes/ holds one directory per node that runs its own tenant apps, named <hostname>.k8s to match the workflow field in ansible/nodes/<hostname>/host.yml. Inside it, one subdirectory per app: ks.yaml, the Flux Kustomization CR, plus app/, the manifests.

This is a different nodes/ from ansible/nodes/. Ansible’s copy is provisioning data, meaning how to reach and bootstrap the host. This one is what runs once the host exists. See Nodes.

A node’s apps read their secrets from Infisical under /nodes/<hostname>/, through that node’s own operator tier in infisical-node-<hostname>, never the infra tier. That separation is enforced by RBAC and an admission policy rather than by convention. See Cluster infrastructure.

Not every cluster node gets a directory here. ogma runs no tenant apps. It is the cluster’s entrypoint, so what it carries is cluster-wide infra rather than per-node workloads: both Traefiks and Pocket ID, all under infra/ and all pinned with a nodeSelector.

And not every directory here is reconciled by Flux. nodes/brokkr.podman/ is Podman Quadlet units for a host outside the cluster, pulled by a timer on that host rather than pushed by Flux. It is named to match its workflow field the same way, holds no kustomization.yaml, and is excluded from the artifact Flux packs by .sourceignore. See The standalone Podman plane.

kenaz.k8s

kenaz runs the k3s server, so it is both controller and worker, plus Flux and most of infra/. The exceptions are the pieces pinned to ogma: both Traefiks and Pocket ID. It runs nine apps, each in nodes/kenaz.k8s/<app>/{ks.yaml,app/} and each reading /nodes/kenaz/<app>:

actual, open-webui and linkwarden are OIDC clients of Pocket ID, so tofu/oidc writes their client ID and secret. searxng, vane and kvasir speak no OIDC and are gated by the auth-sso middleware instead, though kvasir exposes only a runs page that way; the work reaches it by Service, from the one namespace its network policy admits. Every remaining key is either seeded by hand or minted by tofu/bifrost: WEBUI_SECRET_KEY signs Open WebUI’s JWTs and NEXTAUTH_SECRET signs Linkwarden’s, both by hand, while OPENAI_API_KEYS and the two OPENAI_API_KEY entries are the virtual keys bifrost issues its three in-cluster clients. See bifrost.

linkwarden and open-webui keep their data outside their own namespace, in the shared PostgreSQL under infra/postgres. What is left on each PVC is files rather than a database: Linkwarden’s page archives, and Open WebUI’s uploads, vector store and model cache. actual is the one that cannot follow them, because Actual Budget supports no backend but SQLite. See The shared database.

The model gateway

bifrost is where every model request in the cluster goes. It holds the provider credentials, so an app that wants a model needs a virtual key rather than a provider key, and adding a provider changes one ConfigMap instead of every app that would have called it. open-webui reaches it by Service, on its OpenAI-compatible surface, and OLLAMA_API_CONFIGS moved out of /nodes/kenaz/open-webui when it stopped calling Ollama Cloud directly.

client.allowed_origins in its config.json lists one entry, its own ingress host. Bifrost defaults that to *, which would let any page the operator has open drive the dashboard and management API from the browser on the governance.auth_config session. Nothing else here looks at Origin: the netpols match namespaces and the Traefik middleware counts requests. One entry is enough because no other caller is a browser — open-webui and gatus call by Service and by health probe, CLI clients send no Origin, and glance loads the favicon as an image, which CORS does not gate. The cost is that the dashboard no longer answers a browser pointed at a kubectl port-forward.

cli-proxy-api is the second provider behind it, turning subscription CLI logins into an API. It is one of the two apps here with no Ingress, no host, and no Infisical path: its whole configuration is non-secret and ships in git, and the credentials it does hold are OAuth tokens on a PVC, seeded by the browser flow in CLI proxy login. bifrost registers it as an Anthropic-shaped custom provider, so a Claude request keeps its wire format the whole way rather than round-tripping through the OpenAI schema.

That provider’s key in config.json is the literal unauthenticated, which is not a credential and grants nothing. cli-proxy-api serves an empty api-keys list and ignores whatever arrives, but bifrost drops a key with an empty value and then reports no valid keys found for provider without ever calling it. allow_private_network on the same provider is the other half: bifrost refuses RFC 1918 destinations by default, which is every Service in the cluster.

That provider’s models are spelled out rather than left as *. A wildcard makes bifrost discover the catalog at startup and rewrite anything its Anthropic model list does not recognise, so the Gemini ids arrive scrambled and unusable. The list is the one thing here that goes stale: linking another account in cli-proxy-api adds models that stay invisible until they are added to config.json too.

munin is the third provider, and the other app with no Ingress, no host and no Infisical path. It is an Ollama serving one embedding model, embeddinggemma:300m, on CPU, registered as an OpenAI-shaped custom provider with the same literal unauthenticated key and the same allow_private_network. It exists because kvasir embeds every source it collects, both providers above serve chat and nothing else, and Ollama Cloud publishes no embedding model. Its PVC holds the weights and carries no k8up.io/backup annotation: the init container refetches them, so there is nothing there worth a nightly snapshot. vk-kvasir is the only virtual key that names it.

The search surface

vane answers questions from the web, and sits on two backends: bifrost for the model and searxng for the results. It returns one cited answer in seconds.

It pins no model. It exposes a picker, and pinning one by environment variable would remove it, so the choice is made per question: ollama/deepseek-v4-flash:cloud for most of them, and ollama/deepseek-v4-pro:cloud when the reasoning matters more than the latency. Bifrost resolves the provider/model prefix, so that is how the names are spelled in the app.

Its virtual key names the ollama provider only. The app has no use for cli-proxy, and a key that cannot reach it cannot spend the subscription quota behind it on a search loop that does not stop.

Its configuration is not held in git. vane reads its environment only on the boot that creates data/config.json and owns that file afterwards, so its models, its embedding model and any later key change are set in its Settings page and live on the PVC. That is also why nodes/kenaz.k8s/searxng/app/settings.yml overrides the wolframalpha engine: vane routes factual questions through it, and upstream ships it disabled.

vane has no login at all, so auth-sso is the only thing in front of it.

kvasir sits on the same two backends and answers the opposite kind of question. Where vane returns one cited answer in seconds, kvasir runs STORM against a topic for minutes to tens of minutes and returns a cited article. It is our own service, built in brewcoua/kvasir, because upstream ships a library and a demo but no server and no image.

Its Ingress serves one page and nothing that costs anything: what is running, which stage it is in, and what each run has spent. The work does not arrive that way. The Open WebUI pipe function calls it by Service, because that host sits behind auth-sso and a pod carries no session cookie, so infra/policies/namespaces/kvasir/netpol-allow-from-open-webui.yaml is still what admits the research itself.

/healthz and /readyz are the exception to that login, carved out by a second Ingress on the same host in app/ingress.yaml. Traefik applies its middleware annotation to every router an Ingress creates, so exempting two paths means a second object rather than a second path. That is what gives it a Gatus check asserting a body, where every other app behind auth-sso can only assert the redirect.

Its models are pinned rather than picked, one fast and one strong, and its virtual key names ollama and munin and no more. That scoping matters more here than for vane: a run is unattended and spends minutes issuing calls, so a key that cannot reach cli-proxy cannot drain the subscription quota behind it when one goes wrong.

Every source it collects is embedded, to rank passages against each section, which is why it needs munin and why a broken embeddings provider fails a run in article generation rather than at startup.

It keeps nothing. Both writable paths are emptyDir, so it has no PVC and no backup Schedule. The one thing it would keep is a Co-STORM session, and Co-STORM is deliberately not wired. Nothing blocks it any more, now that munin serves the embeddings it needs; what it costs is a PVC and an entry in infra/backup/config/schedules.yaml, and a round table nobody is holding is not worth a nightly snapshot. The pipe function exposes both models regardless, so a Co-STORM session started here survives only until the pod restarts.

The Open WebUI half is one Pipe function, installed by hand and living in Open WebUI’s database rather than in git. See Cold bootstrap.

open-webui reaches searxng for web search, bifrost for models and kvasir for research, vane and kvasir each reach searxng and bifrost, and bifrost reaches cli-proxy-api and munin. None of them go through an ingress host, so nine files open those holes. See Pod-to-pod across namespaces.

New apps land the same way. The step-by-step is Adding a node app.

brokkr.podman

brokkr is in no Kubernetes cluster. It runs Forgejo at git.$DOMAIN and Woodpecker CI at ci.$DOMAIN as rootful Podman containers, behind a Traefik of its own that issues its own certificates, and it reconciles them by pulling this repository on a timer.

Nothing here reads Infisical, and that is the point rather than an omission: the node holds no credential for any secret store, so its secrets are pushed as 0600 env files by ansible/roles/forge. Everything about how that works, what it costs, and what to verify is in The standalone Podman plane.

The standalone Podman plane

How brokkr runs and reconciles workloads without Kubernetes, what it holds and what it deliberately does not, and how to change what runs on it. At the end of Changing what runs a commit has reached the node and restarted the container it affects.

brokkr hosts the forge: Forgejo at git.$DOMAIN and Woodpecker CI at ci.$DOMAIN, both on the public internet. It is a Fedora host provisioned by the same Ansible plane as the cluster nodes, on the same NetBird mesh, and it runs no Kubernetes at all.

That is the whole design constraint. A forge holding a mirror of this repository is what you reach for when the cluster is broken, so nothing on it may depend on the cluster. Pocket ID, Flux, cert-manager, the Infisical operator and both Traefiks are cluster workloads; brokkr uses none of them. It logs in against Pocket ID for convenience and keeps one local admin account for when that is unavailable.

Green is the one thing on the node that git never sees. Amber is a third party. The dashed red edge is the property the whole node exists for: the cluster provides brokkr with nothing, so a cluster outage cannot take the forge with it. The grey edge is the one direction of dependency that does exist, and it degrades to the local admin account rather than to no access.

Two halves, and why the split

Config is pulled from git. Secrets are pushed by Ansible. Nothing crosses.

The split is forced. Infisical’s free tier caps machine identities at five and three are already spent, and giving brokkr one would also mean the node holding a credential that reads almost the whole project. Instead it holds no store credential at all, which keeps the tier boundary in Secrets intact: the node’s secrets are files that Ansible wrote, and nothing on the node can fetch a secret it was not given.

The cost is real and worth stating: a rotated secret does not reconcile. It reaches the node only when someone runs Ansible. See Credential rotation.

The reconciler

ansible/roles/quadlet_gitops installs a oneshot service and a 5-minute timer. /usr/local/bin/futhark-quadlet.sh does five things:

1. git fetch in /var/lib/futhark-gitops, then compare HEAD to origin/master. Unchanged is the common case and exits immediately, so a tight interval costs one fetch.
2. git merge --ff-only. A rewritten branch fails loudly here rather than merging into whatever the node holds.
3. Render every file under nodes/brokkr.podman/ through envsubst, resolving ${DOMAIN} and ${SUB_INTERNAL} from /etc/futhark/substitutions.env.
4. rsync --delete the rendered units/ into /etc/containers/systemd/ and config/ into /etc/futhark/, then systemctl daemon-reload.
5. Restart what changed, and publish the applied revision as a node-exporter textfile metric.

The clone is anonymous HTTPS against the public remote. That is deliberate: no deploy key, no credential on the node, nothing to rotate, and the reconciliation works with Proton Pass and the cluster both unavailable.

Four properties follow from this and are the ones to hold in mind:

• rsync --delete is what makes it reconciliation rather than a copy. Delete a unit file in git and the unit is gone from the node; daemon-reload then stops and reaps its container. There is no separate teardown step and no orphan.
• A bad commit rolls forward, not back. Nothing keeps a previous revision to return to, so recovery is another commit. This is the same contract Flux has.
• *.env is excluded from the config rsync. Those files are Ansible’s, they hold the node’s secrets, and an unguarded --delete would remove them on every run. substitutions.env survives for the same reason.
• A config change restarts everything. Traefik’s static configuration is read once at process start, and which containers read a given config file is not derivable from its name, so the reconciler does not guess. A changed .container file restarts only that one service.

Why envsubst

The domain is an identifying value and this repository is public, so no file here spells it out. In the cluster, Flux resolves ${DOMAIN} from the cluster-values Secret with postBuild.substituteFrom. Off-cluster there is no Flux, so the reconciler does the same substitution from a file Ansible writes. The variable names are the same ones on purpose, so a unit file reads like a manifest. See Domains.

Only named variables are substituted, so a literal $ elsewhere survives. The $$ escaping a Flux-reconciled manifest needs does not apply here.

Quadlet, and the units

Quadlet is a systemd generator. A *.container file in /etc/containers/systemd/ becomes a *.service unit at daemon-reload time, so forgejo.container yields forgejo.service. Until that reload runs, a new file is not a unit and systemctl start cannot find it.

Podman rather than Docker because Fedora ships it, needs no third-party repository, and generates systemd units natively, which this repository already uses for the mesh watchdog and the egress exporter. Rootful rather than rootless, which is the simpler choice and not the safer one: it binds 443 and 22 with no port-floor sysctl and needs no lingering user session.

Non-secret configuration is Environment= lines in the unit, so it is reviewable in a diff. Secrets are EnvironmentFile= only. A local username counts as identifying, which is why WOODPECKER_ADMIN is in the env file rather than the unit.

Image pins are tag@sha256:…, the same rule as everywhere else in the repository, and Renovate opens the bumps through a custom manager in .github/renovate.json5, since no built-in manager reads a systemd unit file. See Dependency updates.

TLS without cert-manager

Traefik issues its own certificates over ACME TLS-ALPN-01, answered on the 443 listener that already exists. The two alternatives were both worse here:

• DNS-01, which the cluster uses, needs the Bunny API key. That is a crown-jewel credential, and putting it on a node whose secrets are files on disk widens its blast radius for nothing.
• HTTP-01 needs port 80 open, which roles/podman_host deliberately does not open, for the same reason roles/firewall_ingress closes it on the edge node.

The cost is that TLS-ALPN-01 cannot issue a wildcard, so each hostname needs its own record and its own certificate. With two hostnames that is the cheaper trade. tofu/bunny publishes the two A records; nothing writes a challenge record into that zone for these two.

The Woodpecker agent holds the Podman socket

The agent runs each pipeline step as a container, which means speaking the container runtime’s API, which means holding its socket. On a rootful host that socket is root: a pipeline step can mount any path on the node.

This is inherent to containerised CI rather than a shortcut taken here, but it should be stated plainly. Anyone who can author a pipeline on brokkr effectively owns brokkr. What bounds it:

• WOODPECKER_OPEN=false, so a Forgejo user cannot enrol themselves into Woodpecker.
• FORGEJO__service__ALLOW_ONLY_EXTERNAL_REGISTRATION=true, so Forgejo accounts come from Pocket ID or from the admin, never from a sign-up form.
• WOODPECKER_AUTHENTICATE_PUBLIC_REPOS=false, so a public repository’s pipelines are not run for anonymous events.

Traefik is on the same host and gets no socket at all, which is why it uses the file provider rather than Podman service discovery.

Authentication

Two paths, and the second exists because the first depends on the cluster.

Pocket ID, for everyone. tofu/oidc registers a pocketid_client and ansible/roles/forge_bootstrap registers it in Forgejo as an OAuth login source named pocketid. The callback path is /user/oauth2/pocketid/callback, where the middle segment is that source name, so the two have to spell the same word. Forgejo has no app.ini section for a login source, so forgejo admin auth add-oauth is the only handle there is; the role runs update-oauth on every converge after the first, which is what makes a rotated client secret reach the running Forgejo.

One local admin, for when Pocket ID is down. A username and password in Proton Pass, created by the same role. This is the account that makes the node worth having, so verifying it works is not optional. Woodpecker’s own admin is the same username, which is why WOODPECKER_ADMIN is set.

Woodpecker authenticates against Forgejo, not against Pocket ID. A Woodpecker user is a Forgejo user who granted it access, so the chain is browser to Woodpecker to Forgejo to Pocket ID. That OAuth application is created inside Forgejo and is the one credential here that no plane in this repository can mint.

Backups

brokkr writes to its own restic repository in its own B2 bucket, provisioned by tofu/b2 with a key scoped to that bucket alone. It is deliberately not the cluster’s repository: restic has no per-path access control, so a key that can write there can read every cluster snapshot ever taken.

futhark-forge-backup.timer runs nightly at 03:00. The script takes a sqlite3 .backup of each database into /srv/futhark/backup, then restic backup over the data directories and that staging directory, tagged forge. futhark-forge-prune.timer runs restic check and then restic forget --prune weekly, in that order: pruning an already-damaged repository writes the damage in.