[ISSUE] ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[ISSUE] ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[adrian-bl]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

New issue by adrian-bl on void-packages repository

https://github.com/void-linux/void-packages/issues/37667

Description:
### Is this a new report?

Yes

### System Info

Void 5.18.4_1 x86_64 GenuineIntel uptodate rFFF

### Package(s) Affected

zfs-2.1.5_1

### Does a report exist for this bug with the project's home (upstream) and/or another distro?

_No response_

### Expected behaviour

The system boots

### Actual behaviour

After upgrading 2 systems from ZFS 2.1.4 to 2.1.5, both systems fail to boot as dracut complains about not being able to mount the rootfs.

Note that ZFS is *not* used as rootfs: one system uses ext4, the other btrfs.

### Steps to reproduce

1. Have a non root-on-zfs system
2. Install zfs-2.1.5_1
3. Reboot -> system will fail to startup

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[adrian-bl]

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

New comment by adrian-bl on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1165450907

Comment:
Digging around, the issue seems to be in zfs-lib.sh overwriting `root` with an old value.

I hacked up the following patch which fixes the issue for me:

```
--- zfs-lib.sh.orig	2022-06-24 12:46:00.250658147 +0200
+++ zfs-lib.sh	2022-06-24 12:39:42.823358632 +0200
@@ -87,11 +87,11 @@
         return
     fi
 
-    root=$(getarg root=)
+    xroot=$(getarg root=)
     rootfstype=$(getarg rootfstype=)
 
     # shellcheck disable=SC2249
-    case "$root" in
+    case "$xroot" in
         ""|zfs|zfs:|zfs:AUTO)
             root=zfs:AUTO
             rootfstype=zfs
@@ -99,7 +99,7 @@
             ;;
 
         ZFS=*|zfs:*)
-            root="${root#zfs:}"
+            root="${xroot#zfs:}"
             root="${root#ZFS=}"
             root=$(echo "$root" | tr '+' ' ')
             rootfstype=zfs
```

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[zdykstra]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

New comment by zdykstra on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1165599150

Comment:
If you don't have root-on-ZFS, there's no need for `zfs` to be in your initramfs. You can add:

`omit_dracutmodules+=" zfs "` to `/etc/dracut.conf.d/zfs.conf` to exclude it completely on your next initramfs rebuild.

What kernel commandline do you boot your system with?

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1165600300

Comment:
The real question is why this never bit you before.

This doesn't really seem to be a bug; I suspect you aren't specifying any `root=` argument in your kernel command line, which means the ZFS module here assumes control and tries to auto-detect a pool. If you specify a root argument that doesn't specify ZFS stuff, this function should set `root=` to whatever the value is that you specified and leave it alone.

Also, as zdykstra mentioned, you shouldn't put the ZFS dracut module in your initramfs if you aren't booting a ZFS root.

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[adrian-bl]

[-- Attachment #1: Type: text/plain, Size: 1955 bytes --]

New comment by adrian-bl on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1165622712

Comment:
@zdykstra 

> What kernel commandline do you boot your system with?

```
[root@abyss adrian]# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-5.18.4_1 root=UUID=976f527a-5bf9-442c-be36-f57d98cd86d3 ro loglevel=4 rd.debug
```

@ahesford 

No, i have `root` specified (see above)


> Also, as zdykstra mentioned, you shouldn't put the ZFS dracut module in your initramfs if you aren't booting a 
> ZFS root.

Well, both systems are a pretty vanilla void installation - just with the ZFS package installed (because my /home is on ZFS). I don't think it is expected that i'd have to blocklist ZFS in dracut in such a configuration.


> this function should set root= to whatever the value is that you specified and leave it alone.

The problem here is that dracut already changed this value before:

So my system sets `root=UUID=976...`, let's look at dracut's `parse-block.sh`:

```
case "$root" in
    block:LABEL=*|LABEL=*)
        root="${root#block:}"
        root="$(echo $root | sed 's,/,\\x2f,g')"
        root="block:/dev/disk/by-label/${root#LABEL=}"
        rootok=1 ;;
    block:UUID=*|UUID=*)
        root="${root#block:}"
        root="${root#UUID=}"
        root="$(echo $root | tr "[:upper:]" "[:lower:]")"
        root="block:/dev/disk/by-uuid/${root#UUID=}"
        rootok=1 ;;
 ```

Once this has run, `root` is set to `block:/dev/disk/by-uid/976...` (i verified that this is the case by booting with `rd.debug`)

Now later (after parse-block.sh but before the rootfs mount)`zfs-lib.sh` executes

```
root=$(getarg root=)
```

which will reset `root` with the value specified on the kernel command line, undoing whatever `parse-block.sh` did and ultimatively confusing the rest of dracut by having `root` set to a value it doesn't understand/expect.

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[adrian-bl]

[-- Attachment #1: Type: text/plain, Size: 2228 bytes --]

New comment by adrian-bl on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1165622712

Comment:
@zdykstra 

> What kernel commandline do you boot your system with?

```
[root@abyss adrian]# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-5.18.4_1 root=UUID=976f527a-5bf9-442c-be36-f57d98cd86d3 ro loglevel=4 rd.debug
```

@ahesford 

No, i have `root` specified (see above)


> Also, as zdykstra mentioned, you shouldn't put the ZFS dracut module in your initramfs if you aren't booting a 
> ZFS root.

Well, both systems are a pretty vanilla void installation - just with the ZFS package installed (because my /home is on ZFS). I don't think it is expected that i'd have to blocklist ZFS in dracut in such a configuration.


> this function should set root= to whatever the value is that you specified and leave it alone.

The problem here is that dracut already changed this value before:

So my system sets `root=UUID=976...`, let's look at dracut's `parse-block.sh`:

```
case "$root" in
    block:LABEL=*|LABEL=*)
        root="${root#block:}"
        root="$(echo $root | sed 's,/,\\x2f,g')"
        root="block:/dev/disk/by-label/${root#LABEL=}"
        rootok=1 ;;
    block:UUID=*|UUID=*)
        root="${root#block:}"
        root="${root#UUID=}"
        root="$(echo $root | tr "[:upper:]" "[:lower:]")"
        root="block:/dev/disk/by-uuid/${root#UUID=}"
        rootok=1 ;;
 ```

Once this has run, `root` is set to `block:/dev/disk/by-uid/976...` (i verified that this is the case by booting with `rd.debug`)

Now later (after parse-block.sh but before the rootfs mount)`zfs-lib.sh` executes

```
root=$(getarg root=)
```

which will reset `root` with the value specified on the kernel command line, undoing whatever `parse-block.sh` did and ultimatively confusing the rest of dracut by having `root` set to a value it doesn't understand/expect.

(Also see here for the rd.debug output on how `root` is set/updated (note that this includes the patch i posted above (with xroot)) https://gist.githubusercontent.com/adrian-bl/49f25982d68b2e9ca4907b19ad4ce57b/raw/f72cbd3eb2d0fa461aa14279cf3b1fcb7c76866c/gistfile1.txt)

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1165688484

Comment:
Thanks for the clarification. This is most definitely a bug and should be fixed upstream (see https://github.com/openzfs/zfs/pull/13589 for my proposal). Once we have some feedback from upstream, we can pull a patch back here. In the meantime, disabling the ZFS dracut module will get you what you need.

Also, cc: @Vaelatern 

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[Vaelatern]

[-- Attachment #1: Type: text/plain, Size: 241 bytes --]

New comment by Vaelatern on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1165734529

Comment:
> The real question is why this never bit you before.

It wasn't there on an earlier release.

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[Vaelatern]

[-- Attachment #1: Type: text/plain, Size: 159 bytes --]

New comment by Vaelatern on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1165747486

Comment:
My approach ^

Re: [ISSUE] [CLOSED] ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

Closed issue by adrian-bl on void-packages repository

https://github.com/void-linux/void-packages/issues/37667

Description:
### Is this a new report?

Yes

### System Info

Void 5.18.4_1 x86_64 GenuineIntel uptodate rFFF

### Package(s) Affected

zfs-2.1.5_1

### Does a report exist for this bug with the project's home (upstream) and/or another distro?

_No response_

### Expected behaviour

The system boots

### Actual behaviour

After upgrading 2 systems from ZFS 2.1.4 to 2.1.5, both systems fail to boot as dracut complains about not being able to mount the rootfs.

Note that ZFS is *not* used as rootfs: one system uses ext4, the other btrfs.

### Steps to reproduce

1. Have a non root-on-zfs system
2. Install zfs-2.1.5_1
3. Reboot -> system will fail to startup

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 251 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1171697460

Comment:
Should be fixed by d6d1cc4c403623e0e6d977b3eb1e59fad098fa2f, sorry we didn't catch this before the update.

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[adrian-bl]

[-- Attachment #1: Type: text/plain, Size: 208 bytes --]

New comment by adrian-bl on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1171936684

Comment:
Thanks! I can confirm that the updated package fixes the issue

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[aaFn]

[-- Attachment #1: Type: text/plain, Size: 388 bytes --]

New comment by aaFn on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1172787686

Comment:
I got hit by that bug, and now my system doesn't boot anymore because of that, and drops to dracut shell.
How can I boot to a working state from dracut ? Is there a way to regenerate the initramfs from inside dracut, applying the patch by hand ??

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[RunningDroid]

[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

New comment by RunningDroid on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1172817546

Comment:
@aaFn You should be able to run `lsblkid` to locate your root partition and do `mount $root /sysroot` then `exit` or control-d to boot your system.

Re: ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[aaFn]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

New comment by aaFn on void-packages repository

https://github.com/void-linux/void-packages/issues/37667#issuecomment-1172873625

Comment:
Hello @RunningDroid , that worked, thank you (I was not aware that exiting the shell was continuing the process after that .. I learnt something today).