In-place v2 -> v3 upgrade takes cluster offline: control plane private IPs reassigned, and eth1 lost after network re-attachment

[robsumoza]

Summary

Upgrading an existing v2 cluster in place to v3.2.0 took our 3 control plane + 3 agent cluster fully offline. Two independent issues, both specific to in-place v2 -> v3 upgrades (fresh v3 clusters are unaffected).

Environment: k3s, MicroOS, flannel + wireguard-native, HA control plane LB, network_ipv4_cidr = 10.0.0.0/8, Terraform 1.16.1, hcloud provider 1.68.0. Upgrading from v2.15.2.

---

1. Control plane private IPs are reassigned (private_ipv4 = null)

control_planes.tf passes private_ipv4 = null to the host module:

# We leave some room so 100 eventual Hetzner LBs that can be created perfectly safely
# It leaves the subnet with 254 x 254 - 100 = 64416 IPs to use, so probably enough.
private_ipv4 = null

v2.15.2 derived it:

private_ipv4 = cidrhost(hcloud_network_subnet.control_plane[...].ip_range, each.value.index + 101)

In v3 the attachment moved from the standalone hcloud_server_network resource into an inline network {} block on hcloud_server. There is no removed block for the old resource, so an in-place upgrade destroys the attachment and recreates it through the block. With private_ipv4 = null the block omits ip, so Hetzner auto-assigns on re-attach.

Plan output (v3.2.0, unmodified):

agents:         after.network = [{ip: '10.0.0.101',  network_id: 10299801}]
control_planes: after.network = [{                   network_id: 10299801}]

Agents pin their IP; control planes do not. Those addresses back kube-apiserver --advertise-address (control_planes.tf:384,443) and the k3s node-ip, and control_plane_config rewrites /etc/rancher/k3s/config.yaml and restarts k3s in the same apply — so an existing cluster can lose etcd quorum.

This also appears to contradict the module's own contract. variables.tf:1083 still validates control plane node keys as < 154:

# 154 because the private ip is derived from tonumber(key) + 101. See private_ipv4 in control_planes.tf

Restoring the v2 derivation produced a correct plan for us:

control_planes["0-0-..."]  after.network=[{'ip': '10.255.0.101', ...}]
control_planes["0-1-..."]  after.network=[{'ip': '10.255.0.102', ...}]
control_planes["0-2-..."]  after.network=[{'ip': '10.255.0.103', ...}]

2. Re-attachment changes the NIC MAC, so eth1 never comes back

Even with IPs pinned, the apply left all 6 nodes with k3s failing:

server is not ready: unable to find interface eth1: route ip+net: no such network interface

Detaching and re-attaching issues a new MAC. /etc/udev/rules.d/70-persistent-net.rules pins eth1 by MAC, so the rule stops matching and the NIC comes up under predictable naming (enp7s0). locals.tf:2855 hardcodes flannel_iface = "eth1", so k3s cannot bind.

Every node changed, e.g.:

86:00:00:95:9a:e6 -> 86:00:00:31:5c:65
86:00:00:87:43:a1 -> 86:00:00:31:5c:7a

v3.2.0 ships kh-rename-interface.service + rename_interface_boot.sh, which addresses exactly this failure mode. But it is delivered through cloud-init write_files, so it only lands on newly created or replaced nodes. Nodes upgraded in place are updated (hcloud_server in-place, labels only) and never receive it.

Recovery was to rewrite the udev rule with the current MAC on each node and reboot.

3. Ordering note

With -parallelism=1, Terraform still batched all 6 hcloud_server_network destroys ahead of all 6 hcloud_server updates, rather than interleaving per node. So the whole cluster was detached simultaneously rather than one node at a time. Worth calling out in the migration guide, since it means there is no safe rolling path for this change.

-target is not a workaround: Terraform rejects targeted plans that exclude the moved blocks (which span every node), and a targeted plan adds the inline network block to control planes whose hcloud_server_network is not in scope to be destroyed, double-managing the same attachment.

Suggestions

• Restore a derived private_ipv4 for control planes, or otherwise carry the existing private IP through the attachment migration.
• Deliver the interface-rename fix to existing nodes too (e.g. via the config-update provisioner) rather than cloud-init only.
• Document the attachment migration in MIGRATION.md / docs/v2-to-v3-migration.md. Neither currently mentions hcloud_server_network, the MAC change, or the resulting eth1 failure.