Operations¶

Day-to-day running of the fleet: adding and removing nodes, watching health, and understanding what the convergence daemon does on every tick.

The convergence daemon¶

exitfleetd runs on the gateway every 30 seconds via a systemd timer (a oneshot service, not a long-running loop - the timer owns the cadence, so a slow tick never overlaps the next). Each run takes a lock and calls one function, ef_converge, which:

Preflight. Confirms the kernel prerequisites and repairs the safe ones - the wg-nodes interface exists, the ipip module is loaded, the exitfleet routing table is registered, the operator-subnet policy rule is present, and ip_forward / fib_multipath_hash_policy are set. If a genuine prerequisite is missing (e.g. wg-nodes isn't up yet at early boot), the tick is skipped and retried in 30s rather than crashing.
Reconcile tunnels. For every wg-nodes peer it derives the tunnel name from the last octet of the peer's IP (10.100.0.42 → ipip-42), creates the tunnel if missing, and recreates it if it's down or points at the wrong remote. It records which peers handshook within the last 30 seconds as healthy, and refreshes the node's ASN/network cache from whois.
Reap orphans. Any ipip-* tunnel (and its metadata cache) whose peer no longer exists is deleted.
Rebuild ECMP. If the set of healthy tunnels changed, the default multipath route in the exitfleet table is atomically replaced (ip route replace) so in-flight connections through other nodes are never dropped. If every node is dead the route is removed entirely, so operator traffic fails fast with ENETUNREACH instead of being black-holed.

The daemon reads all of its truth from the kernel and WireGuard each tick and applies only the diff. It is idempotent: running it a hundred times against an unchanged world changes nothing.

# Watch it
journalctl -u exitfleet.service -f
systemctl list-timers exitfleet.timer

Adding a node¶

The one-shot path is exitfleet deploy. If you built or recovered a node out of band (e.g. after a gateway rebuild), register it manually - the daemon does the rest:

exitfleet add-peer <node_pubkey> <wg_ip> <node_public_ip>
# e.g.
exitfleet add-peer 'Abc...=' 10.100.0.12 203.0.113.12

add-peer only adds the WireGuard peer (with the node's wg0 port appended to the endpoint and a 5-second keepalive). The next convergence tick builds the tunnel, resolves the ASN, and joins it to ECMP.

Removing a node¶

exitfleet remove-peer <node_pubkey>

The peer is removed immediately; within 30 seconds the daemon deletes the orphaned ipip-* tunnel, rebuilds ECMP without it, and drops its metadata. Then destroy the VPS with vpsctl node destroy.

Checking status¶

exitfleet status

Gateway:
  wg-operator: 2 operator(s)
  wg-nodes:    3 node(s)

Exit Nodes:
  TUNNEL     WG IP            ENDPOINT                 ASN        NETWORK                HEALTH
  ipip-10    10.100.0.10      203.0.113.50:51820       AS53667    BUYVM-LV               healthy (2s ago)
  ipip-11    10.100.0.11      198.51.100.22:51820      AS24940    HETZNER-FSN            healthy (8s ago)
  ipip-13    10.100.0.13      192.0.2.15:51820         AS20473    The Constant Company   DEAD

Routing:
  Mode:  ECMP rotation
  Table: default proto static ...

The ASN and network columns come from a whois cache the daemon maintains at /var/lib/exitfleet/<octet>. It is display-only - nothing operational depends on it, and a whois failure just shows unknown.

Health, automatically¶

With PersistentKeepalive = 5 on every peer, a node that dies stops handshaking and is dropped from ECMP within 30 seconds (six missed keepalives). When it recovers and handshakes again, the next tick re-adds it. There is nothing to run and no list to edit by hand - liveness is derived from the WireGuard handshake timestamp.

SSH access¶

You never SSH to an exit node through the exit path. From the gateway, reach a node directly over the management mesh (ssh root@10.100.0.10) or its public IP. From an operator station, SSH to the gateway's public IP goes direct (it uses the operator's real source IP, so the split-tunnel rule doesn't capture it); hop from there to the nodes.