Etcd backups proved their usefulness faster than I thought. I screwed up the configuration so badly, I didn’t even recognize it anymore. I did not want to waste more time, and just restored the DB from the backup. Here are the commands:

list backups

k3s etcd-snapshot ls                                                                                                    
(...)
k3s_etcd_snapshot-fedora-1.home-1732186802.zip file:///var/lib/rancher/k3s/server/db/snapshots/k3s_etcd_snapshot-fedora-1.home-1732186802.zip 12785806 2024-11-21T12:00:02+01:00
k3s_etcd_snapshot-fedora-1.home-1732208402.zip file:///var/lib/rancher/k3s/server/db/snapshots/k3s_etcd_snapshot-fedora-1.home-1732208402.zip 12729498 2024-11-21T18:00:02+01:00

stop nodes

execute on all of the nodes

systemctl stop k3s

restore backup

on the node where backup is located

k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=<PATH-TO-SNAPSHOT>

Start the control plane

systemctl start k3s

REMOVE DB ON the remaining nodes

rm -rf /var/lib/rancher/k3s/server/db/

start rest of the nodes

systemctl start k3s

I hope I will not be in this situation soon. But it is good to know, there is reliable restore mechanism in case of any other issues.

Leave a Reply

Your email address will not be published. Required fields are marked *

+