It is just wonderful when you start the morning with your system booting into a kernel panic.
Loading, please wait... /init: conf/conf.d/zz-resume-auto: line 1: syntax error: unexpected "(" Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200
It looks like my initramfs has been corrupted. First, I try to boot into an old kernel. No dice. Looks like I have to regenerate my initramfs.
I boot up a live CD. Of course,
update-initramfs will not work directly in the live CD environment - it won’t work on the live, read-only file system.
I realize I have a LVM encrypted partition setup (LUKS) and I can’t simply walk in there,
chroot, fix the initramfs, and leave. So, I have to mount the LVM encrypted partitions first…
Find these partitions:
# blkid ... /dev/nvme0n1p3: UUID="xxxxxxxx-yyyy" TYPE="crypto_LUKS" ... /dev/nvme0n1p2: UUID="yyyyyyyy-xxxx" TYPE="crypto_LUKS" PARTLABEL="root" ...
I know for a fact that my
nvme0n1p3 is the encrypted swap and
nvme0n1p2 is my root but it will vary.
cryptsetup. The logical volume name must match the ones mounted by the original system. In my case, they were in the format of
luks-<uuid> but I have seen
<devicename>_crypt around as well depending on distro.
If you are unsure, just mount it as whatever, poke into
/etc/crypttab to see, then reboot and start fresh.
Tale #1: Do not attempt to use
dmsetup rename /dev/mapper/old new_name to save time, it will not work and you will get an error later saying
cryptsetup: ERROR: Couldn't resolve device /dev/mapper/old and
cryptsetup: WARNING: Couldn't determine root device.
Again, mount using
cryptsetup with the right logical volume names…
cryptsetup luksOpen /dev/nvme0n1p2 luks-yyyyyyyy-xxxx cryptsetup luksOpen /dev/nvme0n1p3 luks-xxxxxxxx-yyyy
Activate the logical volumes:
Create a rescue location and mount root on it:
mkdir /rescue mount /dev/mapper/luks-yyyyyyyy-xxxx /rescue
Check that it is indeed root by
Bind mount to prepare for
mount --bind /dev /rescue/dev mount --bind /dev/pts /rescue/dev/pts mount --bind /proc /rescue/proc mount --bind /sys /rescue/sys
Now chroot in:
update-initramfs -c -k all
Make sure there are absolutely no errors. Now reboot the system. The fastest way to get out of
chroot cleanly is just to reboot clean, and try not to make a mess out of it any further.
Tale #2: If you see
Warning: couldn't identify filesystem type for fsck hook, ignoring. You did not mount the logical volumes with the correct labels matching
/etc/crypttab. You must make sure /dev/mapper in the chroot matches exactly what will be in the live installation.
Tale #3: If you drop to (initramfs) busybox shell after reboot, you may or may not see an error trying to find
/dev/mapper/luks-xxx. See Tale #2. Labels must match. But while you are in busybox it does not hurt to check that
cryptsetup is working (try
cryptsetup luksOpen /dev/nvme0n1p2 root).
It is notoriously difficult to try to boot the system forcefully from the rescue shell in this state. Thus, figure out why it isn’t booting (check
cryptsetup is present; then check
/dev to see if drives are there; mount them with LVM and check
/dev/mapper and match with
/etc/fstab). Check what is incorrectly configured, fix them, and regenerate initramfs.
Hopefully the system boots properly now.