LVM filters and initrd

Another “don’t make the same mistake I did” post, you say? Yippee! I seem to be running into quite a few of these (semi) complex gotchas lately but I suppose they at least fuel techslaves with a bit of content which I can’t be too angry about these days.

Today’s gotcha is all about LVM filters and initrd and really this wouldn’t be a big problem but because I’ve never sat down to appreciate the initrd process in any great depth, it took me two days to figure out just exactly what went wrong (recovery however, was much faster).

As it turns out, if you have broken LVM filters in /etc/lvm/lvm.conf (RHEL) that accidentally reject your system disk/RAID device from LVM and your root filesystem is in LVM on that device then you update your kernel or create a new initrd and boot the new kernel/initrd, your system isn’t going to boot!

Ok that was an annoying sentence, my apologies. Let me recap:

1. Bad filters in /etc/lvm/lvm.conf that accidentally reject the device with your root filesystem
2. Install new kernel or rebuild initrd
3. Boot new kernel/initrd
4. Kernel panic
5. Fail

It’s really as simple as that. What I didn’t quite realize about the initrd build process is that it actually includes your lvm.conf file in the new initrd and a new initrd is built at each new kernel update/installation. Not fully understanding this until now is really just an exposure of my short commings as a sysadmin. But we all need to learn and so there it is, lesson learned and we, well I, can move on and be better.

What I did find interesting however is that my previous initrd built with a good lvm.conf would boot the system just fine, even though the current on disk /etc/lvm/lvm.conf had broken filters rejecting the device with my root filesystem! This of course makes sense now that I’ve dissected the problem but it really was quite strange to see my system booted on the good initrd but unable to see the volume group which contains the root filesystem logical volume with a quick “vgscan”…

Notes to self:

1. Every new kernel installation or update builds a new initrd
2. The mkinitrd process includes your CURRENT live configurations so if they are broken, your initrd configurations will be broken
3. I don’t know everything

That’s that.

UPDATE: Holy crizzap! I’m not the only person that ran into this problem! Apparently Mintos over on blogspot (dunno him) has a post about the exact same issue!