futhark

A GitOps-driven homelab. Two Fedora nodes joined over a Tailscale mesh run one k0s cluster; everything inside that cluster is reconciled by Flux from this repository, and everything that cannot live inside it is managed by OpenTofu.

The tree splits along three planes, and almost every question about the repo resolves to “which plane owns this?”:

If you are setting a machine up from nothing, start at Cold bootstrap. If something is broken, start at Troubleshooting. If you are adding to the tree, the rules you have to follow are under Conventions.

Everything an operator runs goes through task:

task --list

What runs where

kenaz is the k0s controller+worker and the only node with public ingress. ogma is a worker; OpenBao and Pocket ID are pinned to it with a nodeSelector. Both are on the mesh and are addressed by their MagicDNS name, never by a stored address — see Nodes.

The pieces, roughly in dependency order: OpenBao (secrets, sealed by an external KMIP service), External Secrets Operator (which pulls from OpenBao into Kubernetes Secrets), Pocket ID (OIDC), cert-manager (Let’s Encrypt over DNS-01), two Traefiks — one public, one mesh-only — and a VictoriaMetrics/VictoriaLogs/Grafana stack. Each is described in Cluster infrastructure, and the order they must come up in is Startup ordering.

Secrets, in one paragraph

No credential is ever committed. Bootstrap secrets are pass:// pointers to a Proton Pass vault, resolved at runtime; runtime secrets live in OpenBao and reach pods through External Secrets Operator. The same rule covers identifying non-credentials — a public IP, the tailnet name — because this repository is public. The full rule, and what to do when you need a new one, is in Secrets.

Cold bootstrap

Building the whole thing from nothing, in order. Every step is re-runnable; none of them are one-shot except the OpenBao init in step 5, which stops the play on purpose.

0. The operator machine

task ops:setup

That installs everything the rest of this page assumes — ansible-core, ansible-lint, yamllint, kubectl, helm, kustomize, k0sctl, flux, tofu, pre-commit, Proton’s pass-cli, and mdbook — installs the repo’s pre-commit hooks, and runs tofu init in every module. It needs dnf and uv already present.

You must be logged into pass-cli and on the tailnet yourself: k0s_cluster resolves each node’s mesh address through MagicDNS from this machine.

1. Proton Pass items

Nothing below works until the vault holds the values the repo’s pass:// pointers name. Find them by grepping for the pointers rather than working from a list here — a list would go stale:

grep -rho 'pass://[^"'"'"']*' --include='*.yml' --include='*.yaml' --include='*.env' . | sort -u

They fall into four groups: the admin SSH identity, the Tailscale credentials and MagicDNS suffix, each node’s IP address, and the OVH KMS material for OpenBao’s seal. The Flux git deploy key and the per-module tofu credentials come later, at steps 4 and 7.

Getting the KMIP seal key right is the one genuinely fiddly part — see The OpenBao seal below.

2. Node definitions

One ansible/nodes/<hostname>/host.yml per machine, symlinked into ansible/inventory/host_vars/ and listed in ansible/inventory/hosts.yml. The schema and the exact commands are in Nodes.

3. Host provisioning

task ans:setup

Update, admin user, SSH hardening, mesh join, firewall. Add -- <host> to limit it to one machine. Safe to re-run; ssh_identity picks whichever login currently answers.

After the first run each host answers only as the admin user on the hardened port. If you are provisioning several nodes one at a time, the mesh-peer resolution in roles/tailscale retries while MagicDNS catches up, so ordering between them does not matter.

4. Tailnet policy

Do this before the cluster comes up, not after. Cross-node pod networking needs the ip-in-ip rule, and without it the cluster fails in a way that looks like an OpenBao seal fault rather than a network one.

task tf:plan -- tailscale

The first apply needs an import first — read tailscale in full before running it, because the resource owns the entire policy document.

Also required at this step: the Flux git deploy key. Generate an SSH keypair, add the public half as a deploy key on the repository, and store the private half in the Proton Pass item ansible/roles/flux_bootstrap points at.

5. Cluster and Flux

task ans:k0s

This renders k0sctl.yaml from inventory, converges the cluster, installs the local-path StorageClass, creates OpenBao’s namespace and seal Secret, then bootstraps Flux. The full sequence is in Bootstrap and reconciliation.

The kubeconfig lands at ansible/.generated/kubeconfig, mode 0600, gitignored. Every k0s:* and fx:* task points at it automatically.

Then initialize OpenBao, once:

task bao:policy-sync

On a fresh OpenBao this runs bao operator init, prints the recovery keys and root token, and stops the play. Those values are never written to disk. Copy the root token into the Proton Pass item the role reads back — futharkd/openbao/root token — before doing anything else. The recovery keys are only needed to mint a new root token later; the KMIP seal handles day-to-day unsealing, so there is no Shamir key to manage.

Then run it again:

task bao:policy-sync

The second run does the actual work: it creates the infra namespace plus one node-<hostname> namespace per nodes/*.k0s/ directory, each with its own secret/ kv-v2 mount, kubernetes auth mount, and reader/oidc-writer policies. It is idempotent — run it again any time you add a node.

6. Watch it converge

task k0s:status
task fx:failing

Flux reconciles openbao first and everything else behind it. Expect several minutes; the certificate issuance in particular waits on DNS-01 propagation. Anything still failing after that, start at Troubleshooting.

7. The cloud plane

task tf:plan -- bunny && task tf:apply -- bunny
task tf:plan -- oidc  && task tf:apply -- oidc

Each has its own prerequisites — see bunny and oidc. oidc in particular needs an OpenBao token minted from step 5’s root token, which is why it comes last.

The OpenBao seal

OpenBao auto-unseals against an OVH KMS KMIP endpoint. Two things about the setup are easy to get wrong, and both fail at startup in misleading ways.

The key id must be the KMIP object UID, not the OVH service key id and not its URN. okms keys and okms kmip are separate namespaces, and OpenBao never creates key material itself. Create and activate the object explicitly — OpenBao rejects a key that is not active:

okms kmip create symmetric --alg AES --size 256 --usage encrypt,decrypt
okms kmip activate <id>

Feeding it a service key id instead fails with a KMIP ItemNotFound on GetAttributes, which reads as “the key was deleted” and is not that.

The credential needs exactly three IAM actions. The seal wrapper only ever issues three KMIP operations — GetAttributes at startup to verify the object, then Encrypt and Decrypt to wrap the root key:

okms:kmip:getAttributes   (READ)
okms:kmip:encrypt         (OPERATE)
okms:kmip:decrypt         (OPERATE)

Deliberately not granted: okms:kmip:get returns the raw key material, which defeats the point of a seal whose wrapping key never leaves the HSM. okms:kmip:destroy and okms:kmip:rekey can render every existing ciphertext — the whole OpenBao store — permanently unreadable, and the recovery keys do not help, since they only mint a new root token. activate is only needed when creating the key, so use an admin credential for that step, not this one.

A missing action surfaces as Operation is not authorized on GetAttributes. That one is not about the key id at all; fix the policy, not the key.

okms kmip locate is not a valid way to check any of this: OVH exposes no IAM action for Locate, so it returns “not authorized” even for a correctly permissioned credential.

Task reference

Everything an operator runs goes through go-task. The root Taskfile.yaml is only a set of includes; the real definitions are one file per namespace under .taskfiles/.

task --list

Arguments come after --. A task documented as -- <x> requires one; [-- <x>] takes an optional one.

ops: — the operator machine

ans: — hosts

bao: — OpenBao

task bao:kv -- get -namespace=node-kenaz secret/actual
task bao:kv -- put -namespace=infra secret/foo key=value

fx: — Flux

k0s: — the cluster

Every k0s:* and fx:* task points KUBECONFIG at ansible/.generated/kubeconfig itself — you do not need it in your environment.

tf: — the cloud plane

docs: — this book

Checks and CI

Pre-commit

task ops:hooks installs them; pre-commit run --all-files runs everything by hand.

Three of these have a wrinkle worth knowing.

tofu-validate deliberately does not run tofu init. init can touch .terraform.lock.hcl, and pre-commit treats a hook that modifies a tracked file as a failure. So run task tf:init once locally before committing a .tf change, or validate fails on an uninitialized module. CI runs init as its own step instead.

ansible-lint needs always_run: false set explicitly. Upstream’s own hook manifest sets always_run: true, which overrides the files: ^ansible/ scoping — without the override it runs on every commit regardless of what changed.

The gitleaks hook only scans the staged index (--staged), which is empty under pre-commit run --all-files. Locally that is the right behaviour; in CI it silently scans nothing, which is why validate.yml runs a separate full-tree gitleaks dir ..

CI

Three workflows, all in .github/workflows/.

validate.yml runs on every pull request and every push to master, with cancel-in-progress concurrency. Two jobs:

• pre-commit — installs kustomize and helm explicitly (neither ships on the runner image, and both are language: system hooks), runs tofu init -backend=false per module, then pre-commit run --all-files, then the full-tree gitleaks scan described above.
• ansible-syntax — installs the galaxy collections and runs ansible-playbook --syntax-check playbooks/*.yml.

docs.yml builds this book with mdbook on every pull request and push, and deploys it to GitHub Pages only from master.

mirror.yml pushes a full mirror to Codeberg on every push to master, daily on a cron, and on demand. It is disaster recovery, not a second remote you push to. It uses ssh-keyscan for the host key, which is trust-on-first-use — Codeberg publishes no authenticated equivalent to GitHub’s api.github.com/meta. The blast radius is this mirror push only, not the live Flux deploy-key channel.

All three workflows check out with persist-credentials: false, and every version they install is pinned.

Troubleshooting

Start here:

task k0s:status    # nodes, unhealthy pods, Flux sync state
task fx:failing    # only what isn't Ready

Then narrow down:

task k0s:describe -- <ns>/<pod>    # events, usually says why it won't schedule or mount
task k0s:previous -- <ns>/<pod>    # a crashlooper's last words
task k0s:warnings                  # Warning events cluster-wide
task fx:logs                       # kustomize-controller, or -- helm-controller

The rest of this page is the failure shapes that have actually happened here, and what each one really means.

Half of everything times out, intermittently

Symptom: some DNS lookups resolve and some hang; anything that resolves at startup crashloops; direct-to-pod-IP works. Cross-node pod-to-pod traffic is being dropped.

CoreDNS runs one pod per node and kube-proxy load-balances 50/50, so exactly the remote endpoint is unreachable and the failure looks intermittent rather than total.

# from a pod, against a pod on the *other* node
ping -c2 <remote pod IP>

If that fails, the tailnet ACL is almost certainly missing the ip-in-ip rule the IPIP overlay needs. Full explanation and the fix: Pod to mesh networking.

This is the one that took the cluster down and presented as an OpenBao seal fault three layers away. Check it before believing a cross-node bug is node-local.

no route to host from a pod dialing a node

This is a source-address fault, not a routing one, despite what the message says. tailscaled drops packets whose source is not the node’s own tailnet IP, so pod-sourced packets die on egress even when the route is correct.

Isolate it on the node:

ping -c2 -I <this node's mesh IP> <peer mesh IP>        # should succeed
ping -c2 -I <this node's pod-bridge IP> <peer mesh IP>  # fails without the SNAT rule

Then check the rules are actually installed, and at a priority that is not being shadowed:

ip rule
systemctl status futhark-mesh-routes
iptables -t nat -L POSTROUTING -n | grep tailscale0

See Pod to mesh networking. If ip rule shows kube-router at a priority below 10, the whole script is a silent no-op.

kubectl exec, logs and port-forward are down cluster-wide

Same root cause as the two above. konnectivity-agent dials the controller’s konnectivity-server over the mesh, so it is the first thing to break when pod → mesh networking is broken, and it takes the whole exec path with it.

OpenBao won’t come up

task bao:policy-sync waits for the pod, and fails with the container’s own last 30 log lines if it never answers — read those first, rather than assuming the sync is stuck.

Almost always it is the KMIP seal refusing to configure. Two distinct failures:

• ItemNotFound on GetAttributes — wrong key id. It must be the KMIP object UID, not the OVH service key id.
• Operation is not authorized on GetAttributes — the credential is missing an IAM action.

Both are covered in The OpenBao seal.

Note that bao status exiting 2 means sealed-but-reachable, which should not linger — the KMIP seal auto-unseals on every start, so a pod that stays sealed is a seal configuration problem, not something to unseal by hand.

An ExternalSecret never syncs

kubectl get externalsecret -A
kubectl describe externalsecret -n <ns> <name>

Check in this order:

1. Is the right store referenced? A node app must use bao-node-<hostname>, not bao-infra. See Cluster infrastructure.
2. Does the OpenBao namespace exist? task bao:policy-sync creates one per nodes/*.k0s/ directory, so a newly added node needs a re-run.
3. Does ESO have write access to Secrets in that namespace? That is opt-in per namespace via the rbac-eso-writer template — see Secrets.
4. Does the path actually hold the key? task bao:kv -- get -namespace=<ns> secret/<name>.

A Kustomization is stuck, and its dependency is fine

Check for a namespace that does not exist yet. Every overlay under infra/configs/ sets kustomize’s top-level namespace: field, so the whole overlay fails to apply if the target Namespace is missing. A new component that declares its own namespace has to be added to infra/configs-ks.yaml’s dependsOn. See Startup ordering.

A HelmRelease is Ready but running the old thing

flux reconcile will not reinstall a release whose chart version has not changed. Force it:

task fx:redeploy -- <name>

A certificate never issues

task k0s:certs
kubectl describe certificaterequest -n <ns> <name>

Issuance goes through Let’s Encrypt DNS-01 against the Bunny zone, so it waits on DNS propagation and can legitimately take minutes. If it never completes, check the cert-manager-config Kustomization is Ready — the ClusterIssuer lives there, behind external-secrets-config, because the webhook’s API key is an ExternalSecret.

tofu apply fails with test(s) failed (400)

A tailnet policy test caught a regression. The apply aborted and the tailnet kept its previous policy, which is the intended behaviour. Plan does not catch this — it never submits the document. See tailscale.

tofu validate fails in pre-commit on a module you didn’t touch

The module was never initialized locally. The hook deliberately does not run init:

task tf:init

See Checks and CI.

Layout and naming

Rules for the GitOps tree — flux/, infra/, nodes/.

File names

Only two file names exist for Kubernetes YAML in this repo:

• ks.yaml — a Flux Kustomization CR, one per directory, named for what it reconciles.
• kustomization.yaml — a plain kustomize resource list (the name kustomize build itself requires).

If you are naming a Kubernetes YAML file something else, you are naming it wrong.

Directory layout

• infra/<component>/{ks.yaml, app/} — one Flux Kustomization per component.
• infra/<component>/{config-ks.yaml, config/} — only when that component’s CRs need CRDs its own ks.yaml cannot guarantee exist yet. This is a chicken-and-egg on first apply, and it is why cert-manager’s ClusterIssuer and external-secrets’s ClusterSecretStore each sit behind a second Kustomization.
• nodes/<hostname>.k0s/<app>/{ks.yaml, app/} — one directory per node app. See Node apps.

Cluster-wide infra that happens to be pinned to a specific node is not a nodes/ entry. OpenBao and Pocket ID both run only on ogma, and both express that as a nodeSelector under infra/. nodes/ is for tenant workloads, not infra controllers.

Flux Kustomization boilerplate

interval, prune, sourceRef, and the flux-system namespace are shared by every Flux Kustomization CR, so they are patched in once — from infra/kustomization.yaml and nodes/kustomization.yaml. A ks.yaml itself declares only metadata.name, spec.path, and spec.dependsOn.

flux/infra/ks.yaml and flux/nodes/ks.yaml are the exception and keep their full spec. flux/ has no kustomization.yaml of its own to patch from — Flux auto-generates one from flux/cluster.yaml’s sync.path: flux — and adding a real one would pull cluster.yaml itself into reconciliation.

Adding a node app

1. Create nodes/<hostname>.k0s/<app>/{ks.yaml, app/}. The ks.yaml needs dependsOn: [infra-configs], plus whatever the app itself needs — usually traefik-internal and external-secrets-config. nodes/kenaz.k0s/actual/ks.yaml is the worked example.
2. Add the directory to the sibling kustomization.yaml’s resources:.
3. Create infra/configs/namespaces/<app>/ with a namespace.yaml labeled futk.eu/tier: node and futk.eu/node: <hostname>, plus the default-deny, same-namespace and from-monitoring network policy templates.
4. Add the ingress-bridge template only if app/ ships an Ingress.
5. Add the rbac-eso-writer template only if app/ ships an ExternalSecret.
6. Add the new overlay directory to infra/configs/kustomization.yaml.

Namespaces

Infra controllers declare their own namespace, in infra/<component>/app/namespace.yaml.

Node and tenant namespaces are centralized instead, under infra/configs/namespaces/<namespace>/, alongside that namespace’s NetworkPolicy and RBAC. The split exists because a tenant namespace needs policy attached to it that the app itself should not own.

Every non-control-plane namespace carries futk.eu/tier: infra or futk.eu/tier: node, and node namespaces add futk.eu/node: <hostname>. Those labels mirror OpenBao’s own namespace split into infra and node-<hostname>, created by ansible/roles/openbao/tasks/namespaces.yml — the two need to stay in step, since a node’s apps read from the OpenBao namespace named after the node.

Network policy

Every non-control-plane namespace gets a default-deny baseline plus explicit opt-in bridges, assembled per namespace from shared templates in infra/configs/namespaces/_templates/:

Kubernetes has no cluster-wide NetworkPolicy, so this is one overlay per namespace rather than one file. Egress is left open everywhere — ESO calls OpenBao, cert-manager calls ACME, apps call whatever they call. In a single-tenant homelab the risk that matters is inbound.

One thing the baseline cannot cover: traefik-edge runs with hostNetwork: true, so it shares the node’s network namespace and CNI policy enforcement never sees its sockets. The ingress-edge overlay exists and is correct, but it does not govern that traffic. What actually governs it is firewalld (ansible/roles/firewall_ingress, public zone limited to 80/443 and the hardened SSH port) and Traefik’s own rate limiting.

Rate limiting

Every namespace with an Ingress also composes the middleware-ratelimit template — a Traefik Middleware at average: 100, burst: 200, per source IP. It is basic DoS protection, not a precise budget.

Composing the template alone does nothing. Traefik only applies a Middleware to routers that name it, so the Ingress must reference it explicitly:

annotations:
  traefik.ingress.kubernetes.io/router.middlewares: <namespace>-ratelimit@kubernetescrd

Same-namespace reference only. traefik-edge’s kubernetesCRD provider does not set allowCrossNamespace (unlike traefik-internal), so a shared cross-namespace Middleware would not resolve there.

Secrets

This repository is public. Nothing in it is a credential, and nothing in it is an identifying value either — no real addresses, endpoints, account or key identifiers. Files refer to where a value lives; they never inline it.

There are two stores, used at different times.

Proton Pass — bootstrap

Anything needed before the cluster can resolve secrets for itself comes from a Proton Pass vault, addressed as pass://<vault>/<item>/<field>. The pointer is committed; the value is resolved at runtime and never written to disk.

Two resolvers, one per plane:

• Ansible — the protonpass lookup plugin (ansible/plugins/lookup/protonpass.py, registered by lookup_plugins = plugins/lookup in ansible/ansible.cfg). It shells out to pass-cli and rejects any reference that does not start with pass://. Used for the admin SSH identity, the tailnet MagicDNS suffix, each node’s IP, the Flux git deploy key, the OpenBao KMIP seal material, and the OpenBao root token.
• OpenTofu — pass-cli run --env-file secrets.env -- tofu <cmd>. Each module commits a secrets.env holding pointers only. See Rules for every module.

.taskfiles/bao/Taskfile.yaml’s bao:kv uses a third form, pass-cli item view piped into kubectl exec over stdin — never argv, which would put the root token in the process table.

OpenBao — runtime

Apps read their secrets from OpenBao through External Secrets Operator, never from a static credential committed to the repo. The store an app uses depends on its tier:

• bao-infra (OpenBao namespace infra) — every node-agnostic component.
• bao-node-<hostname> (OpenBao namespace node-<hostname>) — that node’s own apps.

Each OpenBao namespace has its own secret/ kv-v2 mount, its own kubernetes auth mount and its own reader policy, so a compromised binding cannot read outside its namespace by construction. An app needing both scopes uses two ExternalSecrets, one per secretStoreRef — a store’s reach is never widened to cover both.

No long-lived credential lives in-cluster: every ClusterSecretStore authenticates via OpenBao’s kubernetes auth method as the external-secrets ServiceAccount, and the k0s API validates each request with a TokenReview.

Two secrets are deliberately outside GitOps reach, because they exist to seed values Flux needs before it can resolve anything: the git deploy key and OpenBao’s KMIP seal material. Both are (re)created by the relevant task ans:* run, not by Flux.

ESO RBAC

The external-secrets chart’s own cluster-wide RBAC is disabled — values.rbac.create: false in infra/external-secrets/app/helmrelease.yaml. In its place:

• infra/external-secrets/app/{clusterrole,role}.yaml grant ESO only what is genuinely cluster-scoped (ClusterSecretStore) or scoped to its own namespace.
• The rbac-eso-writer template in infra/configs/namespaces/_templates/ grants ESO write access to Secrets in one namespace at a time, and is added to a namespace’s overlay only if that namespace actually has an ExternalSecret.

ESO’s access to secrets is never cluster-wide.

Values that are identifying but not secret

A public IP is not a credential, but publishing it here would still tie this repo to a machine. The rule extends to every file type:

• In a tofu module, set it as TF_VAR_<name>=pass://... in secrets.env.
• In Ansible, use a protonpass lookup — that is why ansible/nodes/<hostname>/host.yml carries ip: "{{ lookup('protonpass', ...) }}" rather than a literal.
• In the cluster, use a placeholder. infra/traefik-edge and infra/monitoring reference ${PUBLIC_IP} and ${MESH_IP}, substituted by postBuild.substituteFrom from an edge-ips ConfigMap that ansible/roles/flux_bootstrap pushes straight into flux-system — into the cluster, never into git.

A genuinely non-identifying constant that is shared across the repo is read from its committed source instead of duplicated. The base domain is the only one, and it lives in config/domain/domain.env — see Domains.

Adding a new secret

1. Put the value in the Proton Pass vault if something needs it before the cluster is up; otherwise put it in OpenBao under the right namespace (task bao:kv -- put -namespace=...).
2. Reference it by pointer — pass:// in Ansible or secrets.env, an ExternalSecret in the cluster.
3. If it is an ExternalSecret in a namespace that had none, add the rbac-eso-writer template to that namespace’s overlay.
4. Confirm gitleaks is happy: pre-commit run gitleaks --all-files.

Domains

Base domains live in exactly one file, config/domain/domain.env, as DOMAIN and INT_DOMAIN. Ansible reads the same file through ansible/inventory/group_vars/all.yml. Never hardcode a base domain in an app.

config/domain/kustomization.yaml wraps it as a reusable Kustomize Component that generates a domain ConfigMap. It lives under config/ rather than infra/ precisely because it is not infra-only.

Flux’s postBuild.substitute and substituteFrom cannot reach HelmRelease.spec.values, so domain injection goes through plain kustomize instead. An app/ that needs a hostname adds the component and a replacements block splicing a key into the target field:

components:
  - ../../../config/domain
replacements:
  - source:
      kind: ConfigMap
      name: domain
      fieldPath: data.INT_DOMAIN
    targets: [...]

infra/monitoring/app/kustomization.yaml is the reference implementation.

Two things that will bite you:

• A Component cannot set a top-level namespace: — that field is reserved to the root Kustomization — so the generated ConfigMap would carry none and fail to apply. It is patched into flux-system once, inside config/domain/kustomization.yaml, because every consumer needs it there regardless of its own target namespace.
• In a plain manifest reconciled by a Flux Kustomization, escape a literal $ as $$, or envsubst will eat it. infra/storage/app/storageclass.yaml does this for rclone’s own ${pvc.metadata.*} template variables.

Startup ordering

infra/openbao is the root of the dependency graph. It dependsOn nothing, because nothing it needs — raft storage, its KMIP seal — lives in the cluster. Everything that reads secrets needs it, directly or transitively.

The real graph, as declared in each ks.yaml:

openbao ─┐
         ├─> external-secrets-config ─┬─> cert-manager-config ──┐
external-secrets ─┘                   │                         ├─> traefik-internal ─> traefik-edge ─> auth
cert-manager ─────────────────────────┘                         │
tailscale-operator ─> tailscale-operator-config ────────────────┘
                                      ├─> storage
                                      └─> monitoring (also needs traefik-internal)

everything above ──> infra-configs ──> nodes

openbao, external-secrets, cert-manager and tailscale-operator are the four roots — each installs a controller that needs nothing from the cluster. Their config-ks.yaml siblings are where the ordering actually bites, because those apply CRs the controller must already have registered CRDs for.

infra-configs sits behind every infra controller, not for secrets but for namespaces: each overlay under infra/configs/ sets kustomize’s top-level namespace: field, so the target Namespace must already exist or the entire overlay fails to apply. nodes then depends on infra-configs, which is what guarantees a node app’s namespace and its default-deny policy land before the app does.

Where a new component goes

• Reads secrets from OpenBao (has an ExternalSecret): downstream of external-secrets-config, same as storage and monitoring.
• Needs a certificate: downstream of cert-manager-config.
• Needs ingress: downstream of traefik-internal, or traefik-edge if it is public-facing.
• Needs none of the above: it can be a fifth root next to openbao. Check first — most things eventually need a cert or an ingress, and both have prerequisites.

Whatever you pick, if the component declares its own namespace and anything under infra/configs/namespaces/ targets it, add it to infra/configs-ks.yaml’s dependsOn too.

Inventory and roles

Ansible owns everything below Kubernetes: the user you log in as, the SSH configuration, the firewall, the mesh join, and the k0s install itself. Once Flux is running, Ansible’s job is done — the only reasons to come back are adding a node, re-converging the cluster, and OpenBao day-2 work.

Run everything through task ans:* rather than ansible-playbook directly; the tasks set the working directory ansible.cfg expects. See Task reference.

Inventory

ansible/inventory/hosts.yml is a bare list of node names. Everything about a node lives in ansible/nodes/<hostname>/host.yml, surfaced to Ansible by a symlink in ansible/inventory/host_vars/. See Nodes for the schema and how to add one.

ansible/inventory/group_vars/all.yml holds what is shared:

Three of those deserve explanation.

The CIDRs are pinned rather than left implicit because two things need to agree on them: k0s_cluster writes them into the k0s ClusterConfig, and the tailscale role needs the pod CIDR to scope its pod → mesh SNAT rule. k0s_pod_cidr is the cluster-wide /16; kube-router carves a /24 out of it per node, and the SNAT rule must match the /16 or a peer’s pods are not covered. They are private RFC1918 ranges, not identifying, so they are plain literals.

ansible_host resolves through MagicDNS for mesh nodes — <hostname>.<tailnet_domain> — and falls back to node.ip otherwise. There is no stored mesh IP anywhere in this repo: Tailscale’s own resolver keeps the name correct across re-keys and reassignments, so there is nothing to update when an address changes. The expression is guarded on node is defined because k0s_cluster, flux_bootstrap and the OpenBao plays run against hosts: localhost, which is implicit, not in inventory, and has no node var.

setup.yml still overrides these three with set_fact mid-play, for first-time provisioning where the host does not yet answer as the admin user. Facts beat inventory vars, so that dance is unaffected by the defaults above.

repo_root is derived from playbook_dir, not inventory_dir, for the same hosts: localhost reason — localhost has no inventory_dir, and playbook_dir is play-scoped rather than host-scoped.

Playbooks

setup.yml runs per host and is gated by the node’s own flags — the tailscale role only runs when node.mesh is true, firewall_ingress only when node.public_ingress is. Nothing in any role branches on a hostname, so a future node opts into either by setting the flag.

k0s.yml is fleet-wide, not per-host: one k0sctl apply converges every workflow: k0s node in inventory at once. It runs against hosts: localhost and reaches the cluster over the network with the k0sctl-fetched kubeconfig — no SSH, no become.

bao.yml reaches OpenBao by kubectl exec into the openbao-0 pod, not over SSH.

Roles

Two roles are worth knowing in more detail.

ssh_identity

A fresh host answers as node.initial_user on node.initial_port; once admin_user and ssh_harden have run, that login is gone. ssh_identity probes the hardened port and picks accordingly. Probing only the hardened port is sufficient because admin_user installs the key before ssh_harden closes the initial one — there is no window where neither works.

k0s_cluster

kubelet’s --node-ip is pinned to each node’s Tailscale mesh IP, deliberately: the Kubernetes API, etcd and kubelet then bind only to mesh addresses and are never publicly exposed. Several consequences follow.

k0s would otherwise self-detect spec.api.address from the default-route interface, which on these hosts is the public IP — join tokens would carry an address workers cannot reach. So the role resolves the controller’s mesh address through MagicDNS with getent (not a dig lookup, which would pull in dnspython), and asserts the result is inside Tailscale’s CGNAT range 100.64.0.0/10. That assertion is not paranoia: a stale MagicDNS entry from a deleted or re-keyed node resolves to something else, and would otherwise be baked silently into spec.api.address and every kubelet’s --node-ip.

The public IP still has to reach Kubernetes somehow, since kubelet can only ever register --node-ip as InternalIP. It arrives as the k0sproject.io/node-ip-external annotation, which the k0s cloud provider reads. node.ip is a pass:// lookup, so the public IP is never committed.

This decision is also what makes cross-node pod networking non-trivial. See Pod to mesh networking.

The protonpass lookup

ansible/plugins/lookup/protonpass.py resolves pass://<vault>/<item>/<field> references by shelling out to pass-cli. It rejects anything not starting with pass://, so a literal cannot slip through by accident. ansible/ansible.cfg registers it with lookup_plugins = plugins/lookup.

The rule about what belongs behind a pointer is in Secrets.

Nodes

One ansible/nodes/<hostname>/host.yml per node — the source of truth for that host. ansible/inventory/host_vars/<hostname>.yml is a symlink to it, which is how Ansible picks it up.

Do not confuse ansible/nodes/ with the repo-root nodes/. This one is provisioning data: identity, address, how to reach and bootstrap the host. The other is workload definition — what runs once the host exists. See Node apps.

Schema

node:
  hostname: kenaz
  os: fedora
  workflow: k0s
  k0s_role: controller+worker
  mesh: true
  public_ingress: true
  ip: "{{ lookup('protonpass', 'pass://futharkd/kenaz/ip address') }}"
  initial_user: fedora
  initial_port: 22

ip is a pass:// lookup, never a literal — a real address is an identifying value and this repository is public. See Secrets.

Exactly one host should be a controller. A second controller makes etcd a two-member cluster with quorum two, which is worse for availability than a single controller, not better.

mesh is orthogonal to workflow: opt in for any node, cloud or local, that needs mesh reachability. There is no mesh IP to store — once joined, the node is addressed as <hostname>.<tailnet_domain>, and Tailscale’s own resolver keeps that correct across re-keys. The node’s Kubernetes InternalIP comes from k0s_cluster’s privateInterface, which is interface-based (tailscale0), not from that name.

public_ingress and mesh are both read generically — nothing in roles/tailscale, roles/firewall_ingress or roles/flux_bootstrap branches on a hostname, so moving public ingress to a different node is a one-line inventory change.

Adding a node

mkdir -p ansible/nodes/<hostname>
$EDITOR ansible/nodes/<hostname>/host.yml
ln -s ../../nodes/<hostname>/host.yml ansible/inventory/host_vars/<hostname>.yml
# then add `<hostname>: {}` under all.hosts in ansible/inventory/hosts.yml
task ans:setup -- <hostname>
task ans:k0s

If the node will run its own tenant apps under nodes/<hostname>.k0s/, it also needs its own OpenBao namespace:

task bao:policy-sync

That loops every nodes/*.k0s/ directory and bootstraps any node-<hostname> namespace that does not exist yet. Then add infra/external-secrets/config/nodes/<hostname>.yaml — copy kenaz.yaml, swap the hostname — and list it in the sibling kustomization.yaml.

Pod to mesh networking

Read this before touching ansible/roles/tailscale. It was diagnosed the hard way twice.

kubelet’s --node-ip is pinned to each node’s Tailscale mesh IP, on purpose: the Kubernetes API, etcd and kubelet are then only ever bound to mesh addresses and are never publicly exposed. Everything below follows from that one decision.

Because the nodes share no L2 segment, k0s’s default CNI (kube-router) builds its own IPIP overlay between their mesh IPs to carry cross-node pod-to-pod traffic. That has two consequences, both of which break a pod dialing a peer node’s own mesh address.

This is not a hypothetical case. konnectivity-agent does exactly that — it dials the controller’s konnectivity-server over the mesh — so while it is broken, kubectl exec, logs and port-forward are down cluster-wide.

1. Routing

kube-router installs from <node's pod /24> lookup 77 and puts the peer’s mesh IP into table 77 pointing at its tunnel. That outranks Tailscale’s own pref 5270 (lookup 52), so the packet is routed into the very tunnel whose transport endpoint is that address.

The fix is one ip rule per peer at priority 10, matching only that peer’s /32 — never the pod CIDR, so pod-to-pod overlay routing is untouched.

The priority is load-bearing. kube-router currently installs its rule at pref 99; an earlier release used 5209. Anything numerically above kube-router’s is silently shadowed, and the whole script becomes a no-op with no error anywhere. Check ip rule on the node rather than assuming, and re-check after a kube-router bump.

2. Source address

tailscaled drops packets whose source is not the node’s own tailnet IP, as anti-spoofing, so pod-sourced packets still die on egress even once the route is correct. They fail with no route to host, which reads like a routing fault and is not one.

They have to be masqueraded to the node’s mesh IP. Nothing else does this: kube-proxy’s KUBE-POSTROUTING only masquerades service traffic carrying the 0x4000 mark, and tailscaled only installs its own ts-postrouting chain when acting as a subnet router or exit node.

The SNAT is scoped to peer /32s rather than the whole 100.64.0.0/10 tailnet on purpose: pods get to reach cluster nodes (konnectivity-agent → konnectivity-server, Prometheus → traefik on the mesh IP) without inheriting the node’s tag:futhark-node reach across the entire tailnet.

Both rules live in ansible/roles/tailscale/templates/futhark-mesh-routes.sh.j2, re-applied by a systemd oneshot because neither ip rule nor iptables state survives a reboot.

The isolating test

Run on a node:

ping -c2 -I <this node's mesh IP> <peer mesh IP>        # succeeds
ping -c2 -I <this node's pod-bridge IP> <peer mesh IP>  # fails without the SNAT rule

Both are locally generated (OUTPUT path, and tailscale0 is in firewalld’s trusted zone), so neither traverses FORWARD. That rules out every firewall hypothesis at once and isolates the drop to tailscaled itself.

Tailnet ACL prerequisite: ip-in-ip

IPIP solves the source-address half for cross-node pod-to-pod traffic on its own — the outer header is sourced from the node’s own mesh IP, so it never presents a foreign source to tailscaled. But the overlay only carries traffic at all if the tailnet ACL permits it.

A Tailscale rule that does not name a protocol matches TCP, UDP, ICMP and SCTP only. IPIP is IP protocol 4 and is silently dropped. Cross-node pod-to-pod then blackholes: the route is correct, the tunnel is UP, nothing is logged, and every packet vanishes.

In the legacy acls syntax proto is a single string, so this is its own rule rather than an extra entry on an existing one. Protocol 4 is named ip-in-ip (ipv4 is an accepted alias), and only TCP, UDP and SCTP may name ports, so the destination port must be *:

{
  "acls": [
    {
      "action": "accept",
      "src": ["tag:futhark-node"],
      "proto": "ip-in-ip",
      "dst": ["tag:futhark-node:*"]
    }
  ]
}

If the policy file uses the newer grants syntax there is no action field, and pasting the rule above yields json: unknown field "action". The equivalent grant expresses the protocol in ip:

{
  "grants": [
    {
      "src": ["tag:futhark-node"],
      "dst": ["tag:futhark-node"],
      "ip": ["ip-in-ip:*"]
    }
  ]
}

acls and grants may coexist in one policy file, so adding an acls block to a grants-based file is a valid way to do this.

The policy file is managed in tofu/tailscale, which also carries a tests entry asserting this rule. Tests are evaluated when the policy is written, so a regression aborts tofu apply and the tailnet keeps its previous policy rather than losing the overlay. Re-check the live policy before believing a cross-node networking bug is a node-local fault — if the tailnet was edited in the admin console rather than through tofu, the two can still have drifted.

The failure is worth recognising by shape, because it does not look like an ACL problem. Half of all ClusterIP DNS answers time out while direct-to-pod-IP works, because CoreDNS runs one pod per node and kube-proxy load-balances 50/50 — only the remote endpoint is unreachable. Anything that resolves DNS at startup then fails intermittently or crashloops. The test:

# from a pod, against a pod on the *other* node
ping -c2 <remote pod IP>   # fails => overlay is dropping, check ip-in-ip in the ACL

Bootstrap and reconciliation

flux/ is the GitOps entrypoint. flux/cluster.yaml — the FluxInstance CR — is applied once by ansible/roles/flux_bootstrap, and Flux does not reconcile the flux/ directory itself, to avoid watching its own bootstrap.

Everything else under flux/ is reconciled. flux/infra/ks.yaml and flux/nodes/ks.yaml are the two Flux Kustomization CRs that point Flux at the matching repo-root directories, ./infra and ./nodes. Those two keep their full spec rather than being patched like every other ks.yaml, because flux/ has no kustomization.yaml of its own — Flux auto-generates one from cluster.yaml’s sync.path: flux, and adding a real one would pull cluster.yaml itself into reconciliation. The rest of the naming and layout rules are in Layout and naming.

Bootstrap sequence

Run by task ans:k0s (ansible/playbooks/k0s.yml):

1. k0s_cluster — render k0sctl.yaml from inventory, k0sctl apply, fetch the kubeconfig into ansible/.generated/.
2. local_path_provisioner — install the local-path StorageClass. OpenBao’s StatefulSet needs it to bind its PVC on the very first reconcile, and nothing in the Flux-managed tree can provision it, since OpenBao is the root of that graph.
3. openbao’s prep.yml — create the openbao Namespace and the seal Secret (KMIP endpoint, key id, mTLS material, all from Proton Pass) before Flux exists. infra/openbao’s StatefulSet needs both mounted on its first reconcile. See Startup ordering.
4. flux_bootstrap:
   1. Install the Flux Operator via Helm.
   2. Apply the flux-system/git-deploy-key Secret, with known_hosts built from GitHub’s published host keys rather than a blind ssh-keyscan.
   3. Wait for the Flux Operator to be ready.
   4. Push the edge-ips ConfigMap — the public-ingress node’s public and mesh addresses, straight into the cluster and never into git. See Secrets.
   5. Apply flux/cluster.yaml. Flux takes over from here, reconciling openbao first and everything else behind it.

Once OpenBao is up and auto-unsealed, run task bao:policy-sync to finish the namespace, mount, auth and policy bootstrap. It is idempotent and safe to re-run. It is not part of the automated sequence above because it needs the OpenBao pod already running.

Day-to-day

task fx:get        # every Kustomization and its sync status
task fx:failing    # only what isn't Ready
task fx:reconcile  # force-reconcile everything, or `-- <name>` for one
task fx:logs       # tail kustomize-controller

The full list is in Task reference.

Cluster infrastructure

infra/ holds one directory per cluster-wide component, each with its own Flux Kustomization. Layout rules are in Layout and naming; the order they come up in is Startup ordering.

The two ingresses

They are split because they solve different problems, and the split is why traefik-edge is the odd one out everywhere else in the tree.

traefik-internal is exposed through the Tailscale operator, so the Service gets its own tailnet identity. Ordinary NetworkPolicy still governs its traffic and there is no node IP to know or inject anywhere. Give an internal Ingress the class internal; it never carries its own tls: block, because the wildcard is served as the default certificate.

traefik-edge binds 80/443 directly on the edge node with hostNetwork: true. There is no LoadBalancer to hand it a Service address — MetalLB was considered and rejected, since it can only manage a real L2/BGP-announced IP, not a tailnet one. hostNetwork means it shares the node’s network namespace, so CNI NetworkPolicy enforcement never sees its sockets and the ingress-edge baseline cannot govern it. What actually governs it is firewalld (ansible/roles/firewall_ingress) and Traefik’s own rate limiting. See Network policy.

The addresses it binds are ${PUBLIC_IP} and ${MESH_IP}, substituted by postBuild.substituteFrom from the edge-ips ConfigMap that ansible/roles/flux_bootstrap pushes into the cluster.

External Secrets Operator

ESO is pointed at infra/openbao through its vault provider, reached in-cluster at openbao.openbao.svc.cluster.local:8200. Isolation is enforced by OpenBao namespaces, not path-prefix policies: each namespace has its own secret/ kv-v2 mount, its own kubernetes auth mount and its own reader policy, so a compromised binding cannot read outside its own namespace by construction.

• infra namespace — infra/external-secrets/config/clustersecretstore-infra.yaml, store bao-infra. Used by every node-agnostic component: cert-manager, storage, monitoring and the rest.
• node-<hostname> namespace — infra/external-secrets/config/nodes/<hostname>.yaml, one per k0s node, store bao-node-<hostname>. Used by that node’s own apps under nodes/<hostname>.k0s/.

Every ClusterSecretStore authenticates via OpenBao’s kubernetes auth method as the external-secrets ServiceAccount in the external-secrets namespace, so no long-lived credential lives in-cluster at all — the k0s API validates each request with a TokenReview. An app needing secrets from both scopes uses two ExternalSecrets, one per secretStoreRef; a store’s reach is never widened to cover both.

Namespace, mount and auth-role bootstrap is ansible/roles/openbao’s tasks/namespaces.yml, run by task bao:policy-sync against the in-cluster OpenBao over kubectl exec. It is not managed by tofu — this repo has no OpenBao provider outside the one write exception described in Rules for every module.

The RBAC ESO itself runs under is deliberately narrow; see Secrets.

Adding a node to ESO

1. task bao:policy-sync — it loops every nodes/*.k0s/ directory and bootstraps any node-<hostname> namespace that does not exist yet.
2. Add infra/external-secrets/config/nodes/<hostname>.yaml (copy kenaz.yaml, swap the hostname) and list it in the sibling kustomization.yaml.

Node apps

nodes/ holds one directory per node that runs its own tenant apps, named <hostname>.k0s to match the workflow field in ansible/nodes/<hostname>/host.yml. Inside it, one subdirectory per app: ks.yaml (the Flux Kustomization CR) plus app/ (the manifests).

This is a different nodes/ from ansible/nodes/. Ansible’s copy is provisioning data — how to reach and bootstrap the host. This one is what runs once the host exists. See Nodes.

A node’s apps read their secrets from the OpenBao namespace node-<hostname> through that node’s own ClusterSecretStore, bao-node-<hostname> — not the shared bao-infra one. See Cluster infrastructure.

Not every k0s node gets a directory here. ogma runs no tenant apps: OpenBao and Pocket ID are cluster-wide infra rather than per-node workloads, so they live under infra/openbao/ and infra/auth/, each pinned to ogma with a nodeSelector.

kenaz.k0s

kenaz runs k0s as controller+worker, plus Flux and everything under infra/. Its first and so far only app is actual (nodes/kenaz.k0s/actual/{ks.yaml,app/}), reading from OpenBao namespace node-kenaz.

New apps land the same way — the step-by-step is Adding a node app.

Rules for every module

tofu/ covers provider-API resources Flux and Kustomize cannot own, because they live outside the cluster: a DNS record, a registrar account, a tailnet policy.

It is not used for anything Flux can reconcile. Pocket ID itself is a Flux-managed workload under infra/auth/, not a tofu resource — the oidc module only registers its OIDC clients, an operation against Pocket ID’s own API that no Kustomization can express.

Rules

Read-only against OpenBao. Never let a module write to OpenBao. Anything a module mints becomes a sensitive output, pasted into OpenBao by hand.

Exception: oidc. It mints OIDC client secrets in Pocket ID, and the whole point of the module is removing that hand-paste step for this one round trip, so it writes those secrets straight to OpenBao via the vault provider. It is scoped to the oidc-writer policy — write-only on secret/data/* in one namespace, created by ansible/roles/openbao/tasks/namespaces.yml — never the root token. Every other module stays read-only.

Provider tokens are never committed. A module’s secrets.env holds only Proton Pass pass:// pointers, which are safe to commit, resolved at runtime by pass-cli run --env-file. This also covers identifying values that are not credentials but still should not sit in git — a real public IP, the tailnet name. Set those as TF_VAR_<name>=pass://....

A genuinely non-identifying constant that is also shared with other parts of the repo — the domain — is read straight from its committed source, config/domain/domain.env, as a local rather than duplicated into terraform.tfvars. See Domains.

State stays local and gitignored. tofu/**/.terraform/, tofu/**/*.tfstate* and tofu/**/crash.log are all excluded. A module’s minted credentials can sit in state in plaintext even when marked sensitive — that only suppresses console and plan output. Keep state on the operator machine. .terraform.lock.hcl is the provider version lockfile and is committed.

Verify provider resource and attribute names against current provider docs before the first apply.

Running a module

task tf:init [-- <module>]   # no secrets needed, provider download only
task tf:plan -- <module>
task tf:apply -- <module>

task tf:init with no module argument inits every module under tofu/, and runs as part of task ops:setup. plan and apply wrap pass-cli run --env-file secrets.env -- tofu <cmd>.

The pre-commit tofu-validate hook only runs fmt and validate, never init — a hook that touches .terraform.lock.hcl fails pre-commit’s own “did this hook modify a file” check. Run task tf:init once locally before committing. CI runs init as its own step first; see Checks and CI.

Modules

bunny

Manages public DNS records against the existing Bunny DNS zone for config/domain/domain.env’s DOMAIN. The zone is looked up via a data source, not created — cert-manager’s DNS-01 webhook already points at that same zone.

Two records, both defined in dns.tf:

• auth.$DOMAIN → the edge node’s public IP. traefik-edge routes it to infra/auth, which is pinned to ogma.
• vault.$INT_DOMAIN → the edge node’s Tailscale mesh IP. traefik-internal routes it to infra/openbao, also pinned to ogma. It resolves publicly to a CGNAT address (100.64.0.0/10), so it is only reachable from the tailnet.

task tf:init -- bunny
task tf:plan -- bunny
task tf:apply -- bunny

Before the first apply

Populate the Proton Pass items secrets.env points at: the edge node’s IP address, its mesh IP, and a Bunny API key. The API key needs the same permissions as the one already used by infra/cert-manager’s DNS-01 webhook — Bunny API keys are account-wide, not zone-scoped.

oidc

Registers OIDC clients in Pocket ID and writes the minted client secret straight into OpenBao. It is the one module allowed to write to OpenBao — see the write exception in Rules for every module. It replaces the manual “register in the Pocket ID admin UI, hand-paste the secret into OpenBao” runbook.

What it manages

One pocketid_client + vault_kv_secret_v2 pair per app, in clients.tf. Add a new pair following the existing block’s shape as each app adopts OIDC login.

Each app owns its own non-secret OIDC config — client ID, discovery URL, hostname — in its own ConfigMap. This module only produces the one thing that cannot be committed: the client secret.

task tf:init -- oidc
task tf:plan -- oidc
task tf:apply -- oidc

Prerequisites

A Pocket ID admin API key, created at Settings → Admin → API Keys on auth.$DOMAIN, stored in the Proton Pass item secrets.env points at.

A namespace-scoped OpenBao token bound to the oidc-writer policy — write-only on secret/data/* plus list and read on secret/metadata/* in that namespace. ansible/roles/openbao/tasks/namespaces.yml creates this policy in every namespace it bootstraps. Mint one per target namespace with the root token:

bao token create -namespace=<namespace> -policy=oidc-writer -period=768h

Store it in the Proton Pass item VAULT_TOKEN in secrets.env points at. Not the root token — this module only ever needs to write one app’s client secret into one namespace.

There is no CA to fetch. OpenBao is served through traefik-internal’s Ingress with the same publicly-trusted wildcard certificate every other internal app uses, not a self-signed listener cert — see infra/openbao/app/configmap.yaml.

Verifying

• The Pocket ID admin UI shows the new client under Applications.
• bao kv get -namespace=<namespace> secret/<name> confirms the secret landed. The oidc-writer token has read access for exactly this.
• The app’s ExternalSecret syncs on its next poll interval: kubectl get externalsecret -n <app> shows SecretSynced.

tailscale

Manages the tailnet policy file, policy.hujson — the mesh’s access control, including the ip-in-ip rule the pod overlay depends on.

This is the one piece of the mesh that used to live nowhere but the Tailscale admin console, so it could not be reproduced from this repo and drifted silently. Pod to mesh networking documents why the ip-in-ip rule exists.

The policy is held in a file rather than a heredoc so it stays reviewable with its comments intact — HuJSON survives the round trip — and so the admin console’s External reference can point straight at it.

Why the policy is safe to commit

It is keyed on tags and autogroups, not on people or addresses: architecture, which this repo already publishes in far more detail, rather than values. Keep it that way — no user emails, no 100.x node addresses, no tailnet name. The tailnet name and the OAuth credentials are identifying, so they stay in secrets.env as pass:// pointers. If a rule ever genuinely needs a literal user, switch acl.tf to templatefile() and pass it as a TF_VAR pointer, the same pattern as bunny.

First apply

tailscale_acl owns the entire policy document. Anything the file omits is deleted from the tailnet on apply, including the rules granting your own access.

policy.hujson was seeded from the live policy as of the module’s first commit and matches it structurally, except for the added tests block. Confirm that is still true rather than assuming it. overwrite_existing_content is deliberately unset, so the provider refuses to apply until the live policy has been imported:

cd tofu/tailscale
tofu init
pass-cli run --env-file secrets.env -- tofu import tailscale_acl.this acl
pass-cli run --env-file secrets.env -- tofu plan

Read the plan before applying. If the tailnet has been edited in the admin console since, the diff shows it — reconcile the change into policy.hujson rather than letting the apply revert it. Then:

task tf:apply -- tailscale

prevent_destroy is set on the resource, because overwrite_existing_content only guards the first apply — nothing else would stop a later tofu destroy, or a dropped state file, from wiping the policy.

Keep an SSH path to the nodes open the first time. A policy that locks you out of the tailnet also locks you out of fixing it.

Tests

The tests block is evaluated server-side when the policy document is written, so a failing test aborts tofu apply and the tailnet keeps its previous policy — the regression never lands.

It is not caught at plan time. Plan does not submit the document, so a green plan says nothing about the tests. Expect a failure to surface as Error: Failed to update ACL — test(s) failed (400).

That is the main reason this module exists: the missing ip-in-ip rule took the cluster down and presented as an OpenBao seal fault three layers away. Add a test whenever a rule turns out to be load-bearing.

Protocols that do not use ports are tested against port 0, not * — the same rule the Tailscale docs give for icmp. A test written tag:futhark-node:* is rejected with invalid port "*".

To check a policy without writing it:

curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  --data @policy.json \
  "https://api.tailscale.com/api/v2/tailnet/$TAILSCALE_TAILNET/acl/validate"

{} means valid; a failing test returns {"message":"test(s) failed", ...}. The endpoint takes strict JSON, so strip the comments and trailing commas from policy.hujson first.

Before the first apply

Populate the Proton Pass items secrets.env points at: the tailnet name, and an OAuth client created in the admin console under Settings → OAuth clients with write access to the policy file.

Then, in the admin console’s policy file management page, set External reference to this directory’s URL and enable Prevent edits in the admin console. The former is only a link for other admins; the latter is what actually stops console edits from drifting from git. API writes — this module — keep working with the lock on.