My work laptop got a distro update today (well yesterday now) so I decided to go ahead and do it...
During the update it asked about updating some cryptsetup config. Since I didn't modfy anything I figured that would be fine.
Well after the update finished, and I finished my call, and I went to reboot it dropped into an initramfs shell.
Several of the startup scripts were saying "Volume luks not found"... and the encrypted root filesystem could not be mounted. The rest of my day was spent doing a backup of the disk from a live image and figuring out what went wrong...
So I booted a live image, mounted the disk using the password, and copied the initramfs file from the boot folder.
Since looking at the files is difficult from the shell, I extracted it using sudo unmkinitfs initrd.img-5-15-52-generic out and looked at the files.
The folder cryptroot/crypttab has this
luks-06f5fccb-9a5a-4a8f-907e-1398520eb297 UUID=06f5fccb-9a5a-4a8f-907e-1398520eb297 /FIXME-initramfs-rootmnt/crypto_keyfile.bin luks,discard
The update commented out the KEYFILE_PATTERN=/crypto_keyfile.bin line in the /etc/cryptsetup-initramfs/conf-hook file. Since the config script modifies the initramfs simply changing it back (it was nice enough to create a backup) did not do anything because the changes need to be processed using the update-initramfs script on a working system.
There is a nice comment saying:
# If KEYFILE_PATTERN if null or unset (default) then no key file is
# copied to the initramfs image.
Unfortunately that is not at all helpful after the fact... so the key file was not being copied and for whatever reason it was not asking for a password as a fallback.
After much searching I found the encrypted disk could be mounted from the initramfs shell with cryptsetup and then mounting it.
The disks are shown from the live image here:
Note down the uuid of the patition that has the /boot folder. There may be a second one if you have an encrypted swap.
From the initramfs shell you can open it with:
cryptsetup luksOpen /dev/nvme0n1p2 luks-<uuid>
then type the password. You can mount the disk then with
mount /dev/mapper/luks-<uuid> /root
Then the /root file has all the real "rootfs" disk contents.
Unfortunately after mounting the disk, I was not sure how to start the os. The initramfs docs show all the scripts it goes through.
After scanning through them I decided to try rewriting the cryptab and re-running the init script.
I overwrote the crypttab using. echo 'luks-<uuid> UUID=<uuid> none luks,discard' > cryptroot/crypttab.
Then re-ran the ./init script and this time it didn't give any more errors that the root UUID=<uuid>.. disk was not found but it failed at run-init because of the pid 1 check.
If I remember correctly I manually ran the run-init command using:
exec run-init /root /sbin/init /root/dev/console
And to my suprise it booted into ubuntu!
Note: See the difference on using
exec <the-script>vs./<the-script>here .
After it loaded the desktop I promplty made sure the cryptsetup conf hook was correct and re-generated the initramfs using sudo update-initramfs and now it boots normally.
Hopefully this helps anyone else that runs into this! I will not get the time spent back but learned a bit more about how ubuntu starts up.