Migrate a K3s Master Node (VMware Fusion NAT on macOS)
Scenario: You’re running a single K3s server ("master") in a Linux VM on macOS via VMware Fusion, using the default vmnet8 NAT network with selective port‑forwards from the host Mac to the VM. You want to move that master to a new VM with minimal downtime.
This post walks through a safe, practical migration that preserves your cluster state. It assumes the default embedded SQLite datastore. If you run embedded etcd (multi‑server), use the K3s snapshot/restore flow instead of copying files (see Notes).
At a Glance
- We stop the old master to avoid split‑brain.
- We rsync the K3s server data directory to the new master.
- We set up NAT port‑forwards on macOS so users/agents reach the API via the Mac’s LAN IP.
- We start K3s on the new master, then repoint agents.
Downtime is limited to the stop/copy/start window.
Prerequisites
- macOS with VMware Fusion (vmnet8 NAT enabled)
- Old & new Ubuntu VMs (or similar) with SSH access
- K3s version you intend to run on the new master (e.g.
v1.33.3+k3s1
) - Admin rights on the Mac (to edit
nat.conf
)
Terminology
- Host = your Mac (on your LAN)
- VM = the Linux guest running K3s
- NEW_MASTER_VM_IP = the IP of the new master on vmnet8 (NAT) network
- MAC_HOST_IP = your Mac’s LAN IP that agents/users can reach
0) Variables
Set these once in your terminal (on the old master or your laptop, wherever you’ll run commands**):
export NEW_MASTER_VM_IP="your-new-vm-ip" # IP of the new K3s VM inside VMware NAT (vmnet8)
export MAC_HOST_IP="your-mac-lan-ip" # Your Mac's LAN IP (reachable by agents/users)
export NEW_MASTER_SSH="ubuntu@${NEW_MASTER_VM_IP}"
export K3S_VERSION="v1.33.3+k3s1"
We’ll forward ports from MAC_HOST_IP → NEW_MASTER_VM_IP later so the API is reachable from your LAN.
1) Back up the current cluster (old master)
- Longhorn volumes: temporarily set replica count to ≥2 for critical volumes to reduce risk during the move.
- Optional but recommended: export any app‑level backups (e.g., database dumps) and take a host‑level snapshot if feasible.
2) Prepare the new master VM (do not start K3s yet)
On the new master VM:
sudo mkdir -m 700 -p /var/lib/rancher/k3s/server/
export INSTALL_K3S_VERSION="${K3S_VERSION}"
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -
Allow passwordless rsync
as root for the copy step:
echo 'ubuntu ALL=(root) NOPASSWD:/usr/bin/rsync' | sudo tee /etc/sudoers.d/ubuntu-rsync
sudo visudo -cf /etc/sudoers.d/ubuntu-rsync
3) Stop the old master (avoid split‑brain)
On the old master:
sudo systemctl stop k3s
sudo systemctl disable k3s
Keep it off until the migration is complete.
4) Copy K3s server data to the new master
From the old master (push to the new VM):
sudo rsync -aHAX --numeric-ids --partial --inplace --rsync-path="sudo -n rsync" \
/var/lib/rancher/k3s/server/ \
${NEW_MASTER_SSH}:/var/lib/rancher/k3s/server/
Afterwards on the new master, confirm ownership & perms:
sudo chown -R root:root /var/lib/rancher/k3s/server
sudo chmod 700 /var/lib/rancher/k3s/server
5) Configure K3s on the new master
You have two options for how the API advertises itself. Option A is recommended.
Option A (Recommended): Advertise the VM IP, trust the Mac in TLS SAN
This keeps node networking simple; the API is still reachable from your LAN via host port‑forwarding.
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml >/dev/null <<'YAML'
# Advertise the VM's own NAT IP
node-ip: <NEW_MASTER_VM_IP>
advertise-address: <NEW_MASTER_VM_IP>
# Present the Mac's LAN IP as a valid TLS SAN so external clients can talk to https://MAC_HOST_IP:6443
tls-san:
- <MAC_HOST_IP>
# If you use Flannel and want it to prefer ExternalIP when present
flannel-external-ip: true
YAML
Option B (Advanced): Advertise the Mac IP (requires a dummy interface in the VM)
Only choose this if you must present MAC_HOST_IP
as the node’s IP. You’ll create a loopback/dummy address inside the VM so K3s can bind/validate certificates.
# Create a dummy interface that holds the Mac's IP as /32 so binding works inside the VM
sudo modprobe dummy
sudo ip link add dummy0 type dummy
sudo ip addr add <MAC_HOST_IP>/32 dev dummy0
sudo ip link set dummy0 up
sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml >/dev/null <<'YAML'
node-ip: <MAC_HOST_IP>
node-external-ip: <MAC_HOST_IP>
advertise-address: <MAC_HOST_IP>
tls-san:
- <MAC_HOST_IP>
flannel-external-ip: true
YAML
Note: If you pick Option B, agents running in VMs may also need special routing or dummy interfaces. For most setups, Option A is simpler and more reliable.
6) macOS (VMware Fusion) NAT port‑forwards (host → VM)
On the Mac host:
sudo nano /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf
Under [incomingtcp]
/ [incomingudp]
, map host ports → the new master VM:
[incomingtcp]
22 = <NEW_MASTER_VM_IP>:22
80 = <NEW_MASTER_VM_IP>:80
443 = <NEW_MASTER_VM_IP>:443
6443 = <NEW_MASTER_VM_IP>:6443 # Kubernetes API
10250 = <NEW_MASTER_VM_IP>:10250 # Kubelet metrics (optional)
[incomingudp]
# VXLAN (Flannel)
8472 = <NEW_MASTER_VM_IP>:8472
Apply changes (briefly interrupts host‑only/NAT networks):
sudo "/Applications/VMware Fusion.app/Contents/Library/vmnet-cli" --stop
sudo "/Applications/VMware Fusion.app/Contents/Library/vmnet-cli" --start
7) Start K3s on the new master
On the new master:
sudo systemctl enable k3s
sudo systemctl start k3s
sudo systemctl status k3s --no-pager
Validate:
sudo k3s kubectl get nodes -o wide
sudo k3s kubectl get pods -A
If you manage ~/.kube/config
on your laptop, copy /etc/rancher/k3s/k3s.yaml
from the new master and replace its server line with:
https://<MAC_HOST_IP>:6443
8) Point agents to the new master
On each agent node, update the API URL:
sudo nano /etc/systemd/system/k3s-agent.service.env
# Update value:
K3S_URL='https://<MAC_HOST_IP>:6443'
sudo systemctl daemon-reload
sudo systemctl restart k3s-agent
VM agents behind the same NAT? Usually no extra steps are needed if you used Option A (master advertises VM IP; API exposed via host forward). If you chose Option B (advertise
MAC_HOST_IP
inside the VM), some agents might require a dummy address or routing trick to reach/bind that IP internally—avoid unless necessary.
Confirm they rejoin:
kubectl get nodes -o wide
9) Private registry credentials (nodes that pull from it) - optional
Create or update /etc/rancher/k3s/registries.yaml
on the new master and any nodes that need it:
mirrors:
<your-private-registry-name>:
endpoint:
- "https://<your-private-registry-domain>"
configs:
"<your-private-registry-name>":
auth:
username: <USER>
password: <PASS>
Restart K3s / containerd:
# On the new master
sudo systemctl restart k3s
# On agents (if changed there)
sudo systemctl restart k3s-agent
10) Post‑migration checks
# Core health
kubectl get nodes -o wide
kubectl get pods -A
# Ingress/controller and services
kubectl get svc -A
kubectl describe nodes <new-master-node>
# Longhorn
kubectl -n longhorn-system get pods
# Validate volume attachments & UI
# API access from outside (from your laptop)
kubectl --kubeconfig ~/.kube/config cluster-info
11) Cleanup (old master)
- Keep the old master powered off until you’re confident.
- Archive your backups off the machine.
- When satisfied, wipe or repurpose the old VM.
Notes & Gotchas
- Where
config.yaml
lives: on the node that runsk3s
(the new master), not the old one. - Datastore type matters: This guide assumes embedded SQLite. For embedded etcd, use
k3s etcd-snapshot
and--cluster-init
/restore workflows instead of copying SQLite files. - TLS SANs: If clients connect via
https://MAC_HOST_IP:6443
, includeMAC_HOST_IP
intls-san
or they’ll fail certificate validation. - Time sync: Ensure NTP on all nodes—TLS and datastore can break with clock skew.
- Flannel:
flannel-external-ip: true
makes Flannel prefer a node’sExternalIP
when present, which can help in NAT/port‑forward scenarios. - Rsync flags:
--numeric-ids
preserves uids/gids;--inplace
avoids double space;--rsync-path
runs rsync as root on the remote via sudoers.
FAQ
Why not just change the VM’s MAC/IP and reuse the old VM entry? You can, but it’s brittle and often collides with VMware/Fusion networking caches and DHCP leases. A clean VM and explicit copy/forward is more repeatable.
Do I need to forward UDP 8472 (VXLAN) on the Mac? Only if workloads outside vmnet8 must reach VXLAN encapsulation on the master. For nodes within the same NAT network, VXLAN stays internal.
Agents can’t reach the API after the move. What now?
Double‑check nat.conf
forwards and that your laptop/agents use https://MAC_HOST_IP:6443
. Confirm the TLS SAN includes MAC_HOST_IP
.
Reference
- K3s migration discussion: https://github.com/k3s-io/k3s/discussions/5311
All rights reserved