Đã đăng vào thg 8 28, 4:56 SA 4 phút đọc

Migrate a K3s Master Node (VMware Fusion NAT on macOS)

Scenario: You’re running a single K3s server ("master") in a Linux VM on macOS via VMware Fusion, using the default vmnet8 NAT network with selective port‑forwards from the host Mac to the VM. You want to move that master to a new VM with minimal downtime.

This post walks through a safe, practical migration that preserves your cluster state. It assumes the default embedded SQLite datastore. If you run embedded etcd (multi‑server), use the K3s snapshot/restore flow instead of copying files (see Notes).

At a Glance

We stop the old master to avoid split‑brain.
We rsync the K3s server data directory to the new master.
We set up NAT port‑forwards on macOS so users/agents reach the API via the Mac’s LAN IP.
We start K3s on the new master, then repoint agents.

Downtime is limited to the stop/copy/start window.

Prerequisites

macOS with VMware Fusion (vmnet8 NAT enabled)
Old & new Ubuntu VMs (or similar) with SSH access
K3s version you intend to run on the new master (e.g. v1.33.3+k3s1)
Admin rights on the Mac (to edit nat.conf)

Terminology

Host = your Mac (on your LAN)

VM = the Linux guest running K3s

NEW_MASTER_VM_IP = the IP of the new master on vmnet8 (NAT) network

MAC_HOST_IP = your Mac’s LAN IP that agents/users can reach

0) Variables

Set these once in your terminal (on the old master or your laptop, wherever you’ll run commands**):

export NEW_MASTER_VM_IP="your-new-vm-ip"      # IP of the new K3s VM inside VMware NAT (vmnet8)
export MAC_HOST_IP="your-mac-lan-ip"          # Your Mac's LAN IP (reachable by agents/users)
export NEW_MASTER_SSH="ubuntu@${NEW_MASTER_VM_IP}"
export K3S_VERSION="v1.33.3+k3s1"

We’ll forward ports from MAC_HOST_IP → NEW_MASTER_VM_IP later so the API is reachable from your LAN.

1) Back up the current cluster (old master)

Longhorn volumes: temporarily set replica count to ≥2 for critical volumes to reduce risk during the move.
Optional but recommended: export any app‑level backups (e.g., database dumps) and take a host‑level snapshot if feasible.

2) Prepare the new master VM (do not start K3s yet)

On the new master VM:

sudo mkdir -m 700 -p /var/lib/rancher/k3s/server/
export INSTALL_K3S_VERSION="${K3S_VERSION}"
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -

Allow passwordless rsync as root for the copy step:

echo 'ubuntu ALL=(root) NOPASSWD:/usr/bin/rsync' | sudo tee /etc/sudoers.d/ubuntu-rsync
sudo visudo -cf /etc/sudoers.d/ubuntu-rsync

3) Stop the old master (avoid split‑brain)

On the old master:

sudo systemctl stop k3s
sudo systemctl disable k3s

Keep it off until the migration is complete.

4) Copy K3s server data to the new master

From the old master (push to the new VM):

sudo rsync -aHAX --numeric-ids --partial --inplace --rsync-path="sudo -n rsync" \
  /var/lib/rancher/k3s/server/ \
  ${NEW_MASTER_SSH}:/var/lib/rancher/k3s/server/

Afterwards on the new master, confirm ownership & perms:

sudo chown -R root:root /var/lib/rancher/k3s/server
sudo chmod 700 /var/lib/rancher/k3s/server

5) Configure K3s on the new master

You have two options for how the API advertises itself. Option A is recommended.

Option A (Recommended): Advertise the VM IP, trust the Mac in TLS SAN

This keeps node networking simple; the API is still reachable from your LAN via host port‑forwarding.

sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml >/dev/null <<'YAML'
# Advertise the VM's own NAT IP
node-ip: <NEW_MASTER_VM_IP>
advertise-address: <NEW_MASTER_VM_IP>
# Present the Mac's LAN IP as a valid TLS SAN so external clients can talk to https://MAC_HOST_IP:6443
tls-san:
  - <MAC_HOST_IP>
# If you use Flannel and want it to prefer ExternalIP when present
flannel-external-ip: true
YAML

Option B (Advanced): Advertise the Mac IP (requires a dummy interface in the VM)

Only choose this if you must present MAC_HOST_IP as the node’s IP. You’ll create a loopback/dummy address inside the VM so K3s can bind/validate certificates.

# Create a dummy interface that holds the Mac's IP as /32 so binding works inside the VM
sudo modprobe dummy
sudo ip link add dummy0 type dummy
sudo ip addr add <MAC_HOST_IP>/32 dev dummy0
sudo ip link set dummy0 up

sudo mkdir -p /etc/rancher/k3s
sudo tee /etc/rancher/k3s/config.yaml >/dev/null <<'YAML'
node-ip: <MAC_HOST_IP>
node-external-ip: <MAC_HOST_IP>
advertise-address: <MAC_HOST_IP>
tls-san:
  - <MAC_HOST_IP>
flannel-external-ip: true
YAML

Note: If you pick Option B, agents running in VMs may also need special routing or dummy interfaces. For most setups, Option A is simpler and more reliable.

6) macOS (VMware Fusion) NAT port‑forwards (host → VM)

On the Mac host:

sudo nano /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf

Under [incomingtcp] / [incomingudp], map host ports → the new master VM:

[incomingtcp]
22    = <NEW_MASTER_VM_IP>:22
80    = <NEW_MASTER_VM_IP>:80
443   = <NEW_MASTER_VM_IP>:443
6443  = <NEW_MASTER_VM_IP>:6443   # Kubernetes API
10250 = <NEW_MASTER_VM_IP>:10250  # Kubelet metrics (optional)

[incomingudp]
# VXLAN (Flannel)
8472  = <NEW_MASTER_VM_IP>:8472

Apply changes (briefly interrupts host‑only/NAT networks):

sudo "/Applications/VMware Fusion.app/Contents/Library/vmnet-cli" --stop
sudo "/Applications/VMware Fusion.app/Contents/Library/vmnet-cli" --start

7) Start K3s on the new master

On the new master:

sudo systemctl enable k3s
sudo systemctl start k3s
sudo systemctl status k3s --no-pager

Validate:

sudo k3s kubectl get nodes -o wide
sudo k3s kubectl get pods -A

If you manage ~/.kube/config on your laptop, copy /etc/rancher/k3s/k3s.yaml from the new master and replace its server line with:

https://<MAC_HOST_IP>:6443

8) Point agents to the new master

On each agent node, update the API URL:

sudo nano /etc/systemd/system/k3s-agent.service.env
# Update value:
K3S_URL='https://<MAC_HOST_IP>:6443'

sudo systemctl daemon-reload
sudo systemctl restart k3s-agent

VM agents behind the same NAT? Usually no extra steps are needed if you used Option A (master advertises VM IP; API exposed via host forward). If you chose Option B (advertise MAC_HOST_IP inside the VM), some agents might require a dummy address or routing trick to reach/bind that IP internally—avoid unless necessary.

Confirm they rejoin:

kubectl get nodes -o wide

9) Private registry credentials (nodes that pull from it) - optional

Create or update /etc/rancher/k3s/registries.yaml on the new master and any nodes that need it:

mirrors:
  <your-private-registry-name>:
    endpoint:
      - "https://<your-private-registry-domain>"
configs:
  "<your-private-registry-name>":
    auth:
      username: <USER>
      password: <PASS>

Restart K3s / containerd:

# On the new master
sudo systemctl restart k3s

# On agents (if changed there)
sudo systemctl restart k3s-agent

10) Post‑migration checks

# Core health
kubectl get nodes -o wide
kubectl get pods -A

# Ingress/controller and services
kubectl get svc -A
kubectl describe nodes <new-master-node>

# Longhorn
kubectl -n longhorn-system get pods
# Validate volume attachments & UI

# API access from outside (from your laptop)
kubectl --kubeconfig ~/.kube/config cluster-info

11) Cleanup (old master)

Keep the old master powered off until you’re confident.
Archive your backups off the machine.
When satisfied, wipe or repurpose the old VM.

Notes & Gotchas

Where config.yaml lives: on the node that runs k3s (the new master), not the old one.
Datastore type matters: This guide assumes embedded SQLite. For embedded etcd, use k3s etcd-snapshot and --cluster-init/restore workflows instead of copying SQLite files.
TLS SANs: If clients connect via https://MAC_HOST_IP:6443, include MAC_HOST_IP in tls-san or they’ll fail certificate validation.
Time sync: Ensure NTP on all nodes—TLS and datastore can break with clock skew.
Flannel: flannel-external-ip: true makes Flannel prefer a node’s ExternalIP when present, which can help in NAT/port‑forward scenarios.
Rsync flags: --numeric-ids preserves uids/gids; --inplace avoids double space; --rsync-path runs rsync as root on the remote via sudoers.

FAQ

Why not just change the VM’s MAC/IP and reuse the old VM entry? You can, but it’s brittle and often collides with VMware/Fusion networking caches and DHCP leases. A clean VM and explicit copy/forward is more repeatable.

Do I need to forward UDP 8472 (VXLAN) on the Mac? Only if workloads outside vmnet8 must reach VXLAN encapsulation on the master. For nodes within the same NAT network, VXLAN stays internal.

Agents can’t reach the API after the move. What now? Double‑check nat.conf forwards and that your laptop/agents use https://MAC_HOST_IP:6443. Confirm the TLS SAN includes MAC_HOST_IP.

Reference

K3s migration discussion: https://github.com/k3s-io/k3s/discussions/5311