It is just wonderful when you start the morning with your system booting into a kernel panic.
Loading, please wait...
/init: conf/conf.d/zz-resume-auto: line 1: syntax error: unexpected "("
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200
It looks like my initramfs has been corrupted. First, I try to boot into an old kernel. No dice. Looks like I have to regenerate my initramfs.
I boot up a live CD. Of course, update-initramfs will not work directly in the live CD environment - it won’t work on the live, read-only file system.
I realize I have a LVM encrypted partition setup (LUKS) and I can’t simply walk in there, chroot, fix the initramfs, and leave. So, I have to mount the LVM encrypted partitions first…
Find these partitions:
# blkid
...
/dev/nvme0n1p3: UUID="xxxxxxxx-yyyy" TYPE="crypto_LUKS" ...
/dev/nvme0n1p2: UUID="yyyyyyyy-xxxx" TYPE="crypto_LUKS" PARTLABEL="root" ...
I know for a fact that my nvme0n1p3 is the encrypted swap and nvme0n1p2 is my root but it will vary.
Mount using cryptsetup. The logical volume name must match the ones mounted by the original system. In my case, they were in the format of luks-<uuid> but I have seen <devicename>_crypt around as well depending on distro.
If you are unsure, just mount it as whatever, poke into /etc/crypttab to see, then reboot and start fresh.
Tale #1: Do not attempt to use dmsetup rename /dev/mapper/old new_name to save time, it will not work and you will get an error later saying cryptsetup: ERROR: Couldn't resolve device /dev/mapper/old and cryptsetup: WARNING: Couldn't determine root device.
Again, mount using cryptsetup with the right logical volume names…
cryptsetup luksOpen /dev/nvme0n1p2 luks-yyyyyyyy-xxxx
cryptsetup luksOpen /dev/nvme0n1p3 luks-xxxxxxxx-yyyy
Activate the logical volumes:
vgchange -ay
Create a rescue location and mount root on it:
mkdir /rescue
mount /dev/mapper/luks-yyyyyyyy-xxxx /rescue
Check that it is indeed root by ls /rescue.
Bind mount to prepare for chroot:
mount --bind /dev /rescue/dev
mount --bind /dev/pts /rescue/dev/pts
mount --bind /proc /rescue/proc
mount --bind /sys /rescue/sys
Now chroot in:
chroot /rescue
Update all initramfs entries:
update-initramfs -c -k all
Make sure there are absolutely no errors. Now reboot the system. The fastest way to get out of chroot cleanly is just to reboot clean, and try not to make a mess out of it any further.
Tale #2: If you see Warning: couldn't identify filesystem type for fsck hook, ignoring. You did not mount the logical volumes with the correct labels matching /etc/crypttab. You must make sure /dev/mapper in the chroot matches exactly what will be in the live installation.
Tale #3: If you drop to (initramfs) busybox shell after reboot, you may or may not see an error trying to find /dev/mapper/luks-xxx. See Tale #2. Labels must match. But while you are in busybox it does not hurt to check that cryptsetup is working (try cryptsetup luksOpen /dev/nvme0n1p2 root).
It is notoriously difficult to try to boot the system forcefully from the rescue shell in this state. Thus, figure out why it isn’t booting (check cryptsetup is present; then check /dev to see if drives are there; mount them with LVM and check /dev/mapper and match with /etc/crypttab and /etc/fstab). Check what is incorrectly configured, fix them, and regenerate initramfs.
Hopefully the system boots properly now.