Networking Internals¶
A short tour of why exitfleet is built the way it is. The exhaustive, kernel-source-referenced version - full packet-flow traces in both directions, every AllowedIPs check, and the 78-scenario robustness analysis - is in DESIGN.md and IMPLEMENTATION.md.
Two subnets, two trust domains¶
| Subnet | Interfaces | Purpose |
|---|---|---|
10.100.0.0/24 |
wg-nodes (gateway .1), wg0 (nodes .10-.253) |
Management mesh: gateway ↔ exit nodes, SSH, IPIP outer headers |
10.100.1.0/24 |
wg-operator (gateway .1), wg-exitfleet (operators .2-.9) |
Operator traffic: operator ↔ gateway, inner payload forwarded to targets |
The gateway runs two WireGuard interfaces with separate keypairs. Operators only ever learn the operator public key; exit nodes only ever learn the nodes public key. Compromise of one domain doesn't touch the other, and the firewall keys on interface names (iifname "wg-operator") rather than subnets, so the interface is the trust boundary.
Why IPIP inside WireGuard¶
WireGuard uses AllowedIPs as both a route and a packet filter. On wg-nodes, each exit-node peer is scoped to 10.100.0.X/32 - a management address. A packet addressed to a real target (say 198.51.100.10) matches no peer's AllowedIPs and is dropped.
IPIP tunnels get around this by wrapping the inner packet (arbitrary destination) inside an outer IP header with management addresses (10.100.0.1 → 10.100.0.10). WireGuard only inspects the outer header, which passes AllowedIPs; the inner packet is opaque payload. IPIP is chosen over GRE because it adds only 20 bytes, is kernel-native, and needs none of GRE's extra features - WireGuard already provides authentication, integrity, and encryption.
The critical consequence: an exit node's wg0 AllowedIPs is 10.100.0.0/24 only, never 10.100.1.0/24. Return traffic (after conntrack un-MASQUERADEs the destination back to 10.100.1.X) is routed via an explicit ipip-exit route, then re-encapsulated with an outer 10.100.0.1 destination that WireGuard accepts. If 10.100.1.0/24 were in AllowedIPs, wg-quick would add a conflicting route and all return traffic would be silently dropped.
ECMP: connections, not packets¶
The gateway steers the operator subnet into a dedicated routing table whose default route has one nexthop per exit tunnel. With net.ipv4.fib_multipath_hash_policy=1, the kernel hashes the L4 tuple (src_ip, src_port, dst_ip, dst_port), so each connection is pinned to one exit for its lifetime while different connections spread across the fleet. Route changes use ip route replace, which is atomic - adding or removing a node never disturbs connections flowing through the others.
MTU¶
WireGuard adds 60 bytes (IPv4 outer) or 80 bytes (IPv6 outer); IPIP adds 20. The operator config sets MTU = 1420, correct for an IPv4 WireGuard endpoint (1500 − 60 − 20). If your gateway endpoint is reachable only over IPv6, set MTU = 1400. Most scanning tools send small probes, so MTU rarely matters in practice; for large payloads the kernel handles PMTUD or fragments.
Split-tunnel routing on the operator station¶
The generated operator config uses Table = off and installs its own rules, in priority order:
| Priority | Rule | Effect |
|---|---|---|
| 50 | to 10.0.0.0/8 → main |
RFC1918 10.x (incl. the gateway's 10.100.1.1) stays direct - SSH to the gateway is not tunneled |
| 50 | to 172.16.0.0/12 → main |
RFC1918 stays local |
| 50 | to 192.168.0.0/16 → main |
RFC1918 stays local |
| 100 | from 10.100.1.2 → table 200 |
Everything else sourced from the tunnel IP goes through the fleet |
| (default) | main table | Traffic on the operator's real source IP goes direct |
SSH to the gateway's public IP works because it leaves on the operator's real source IP, so the from 10.100.1.2 rule never matches it - it hits the main table and goes direct, never entering the exit path.