Etcd backups proved their usefulness faster than I thought. I screwed up the configuration so badly, I didn’t even recognize it anymore. I did not want to waste more time, and just restored the DB from the backup. Here are the commands:
list backups
k3s etcd-snapshot ls
(...)
k3s_etcd_snapshot-fedora-1.home-1732186802.zip file:///var/lib/rancher/k3s/server/db/snapshots/k3s_etcd_snapshot-fedora-1.home-1732186802.zip 12785806 2024-11-21T12:00:02+01:00
k3s_etcd_snapshot-fedora-1.home-1732208402.zip file:///var/lib/rancher/k3s/server/db/snapshots/k3s_etcd_snapshot-fedora-1.home-1732208402.zip 12729498 2024-11-21T18:00:02+01:00
stop nodes
execute on all of the nodes
systemctl stop k3s
restore backup
on the node where backup is located
k3s server \
--cluster-reset \
--cluster-reset-restore-path=<PATH-TO-SNAPSHOT>
Start the control plane
systemctl start k3s
REMOVE DB ON the remaining nodes
rm -rf /var/lib/rancher/k3s/server/db/
start rest of the nodes
systemctl start k3s
I hope I will not be in this situation soon. But it is good to know, there is reliable restore mechanism in case of any other issues.
Leave a Reply