[ISSUE] newer releases of various kernels cause boot to hang with ZFS

[ISSUE] newer releases of various kernels cause boot to hang with ZFS

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 2356 bytes --]

New issue by emacsomancer on void-packages repository

https://github.com/void-linux/void-packages/issues/12519
Description: ### System

* xuname:  
`Void 4.19.50_1 x86_64 GenuineIntel uptodate rrrmFFFF`
* package:  
`linux-4.19.51`, `linux-4.14.126`, `linux-5.1.10` (and I think `linux-5.1.9`) and both `zfs-0.8.0` and `zfs-0.8.1`.

### Expected behavior
The kernel versions which boot —  including `linux-4.19.50`, `linux-4.14.125` — proceed loading after enabling the ZFS kernel module, and successfully boot, e.g.:

```
.....
[    3.908527] dracut: luksOpen /dev/sda1 cryptz /rootkey.bin luks
[    6.366046] NET: Registered protocol family 38
[   11.486374] spl: loading out-of-tree module taints kernel.
[   11.492145] icp: module license 'CDDL' taints kernel.
[   11.492147] Disabling lock debugging due to kernel taint
[   12.410694] ZFS: Loaded module v0.8.0-1, ZFS pool version 5000, ZFS filesystem version 5
[   12.441836] dracut: ZFS: No sysroot.mount exists or zfs-generator did not extend it.
[   12.441902] dracut: ZFS: Mounting root with the traditional mount-zfs.sh instead.
[   12.457191] dracut: ZFS: Importing pool dozer...
[   13.388974] dracut: ZFS: Mounting dataset dozer/ROOT/system...
[   13.462031] dracut: Mounted root filesystem dozer/ROOT/system
[   13.482632] dracut: ZFS: There is an active pool, will export it.
[   13.530312] dracut: Switching root
[   13.715111] fuse init (API version 7.27)
[   13.742556] raid6: sse2x1   gen()  8789 MB/s
[   13.759555] raid6: sse2x1   xor()  7705 MB/s
[   13.776556] raid6: sse2x2   gen() 12730 MB/s
.....
```

### Actual behavior
All of the newer versions of kernels hang during boot, only proceeding as far as:
```
....
[    3.908527] dracut: luksOpen /dev/sda1 cryptz /rootkey.bin luks
[    6.366046] NET: Registered protocol family 38
[   11.486374] spl: loading out-of-tree module taints kernel.
[   11.492145] icp: module license 'CDDL' taints kernel.
[   11.492147] Disabling lock debugging due to kernel taint
[   12.410694] ZFS: Loaded module v0.8.0-1, ZFS pool version 5000, ZFS filesystem version 5

```
and then hangs indefinitely.


### Steps to reproduce the behavior
Choose one of the newer kernel releases  (e.g. anything more recent than `linux-4.19.50`, `linux-4.14.125`, or (I think) `linux-5.1.8`) in GRUB.

Re: newer releases of various kernels cause boot to hang with ZFS

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 400 bytes --]

New comment by Hoshpak on void-packages repository

https://github.com/void-linux/void-packages/issues/12519#issuecomment-503039114
Comment:
It would be interesting to get reports from other zfs users to verify it this is a general issue or just happens under certain circumstances we don't know yet. You could also ask upstream since they know better how their code interacts with the linux kernel.

Re: newer releases of various kernels cause boot to hang with ZFS

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 451 bytes --]

New comment by emacsomancer on void-packages repository

https://github.com/void-linux/void-packages/issues/12519#issuecomment-503243537
Comment:
Thanks, @Hoshpak . I checked upstream and found https://zfsonlinux.topicbox.com/groups/zfs-discuss/Tf5892c78870b27c6/zfs-root-with-0-8-1-no-longer-working , which seems similar but non-identical, but I wonder if the problem doesn't lie somewhere with dracut in any case. I'll explore it further upstream.

Re: newer releases of various kernels cause boot to hang with ZFS

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 1696 bytes --]

New comment by emacsomancer on void-packages repository

https://github.com/void-linux/void-packages/issues/12519#issuecomment-503410468
Comment:
Ok, so debugging this was a bit tricky. The gist is this: the recentness of the kernels was a red herring, the crucial factor was whether the kernel was installed before or after the zvol swap file was created. All kernels created after swap was created end up with a dracut configuration which (apparently by default) assumes that swap will be used for resume, and when booting any of these kernel results in dracut waiting about 2 minutes trying to finding the swap partition (so these kernels do *eventually* boot). 

So I'll close this issue (which wasn't what it appeared) but document a possible resolution:

One Potential Fix
==========
I don't really use resume, so at least one easy fix for this issue is to create a dracut conf file, e.g.:

/etc/dracut.conf.d/omit-resume-for-zvol-swap.conf containing:

`omit_dracutmodules+=" resume "`

and call `xbps-reconfigure -f` on the relevant kernels.

I'm not sure if a zvol swap can be used for resume/seen at the appropriate time by dracut or not.

Using Swap on ZFS root?
==============
It's possible that the location of the swap partition could be specified in a way that dracut could locate it (though, again, not using resume, I'm not sure about swap size requirements for using resume). zvol-type swap ends up being mounted on `/dev/zvol/<POOL-NAME>/swap`. I think dracut looks for the resume/swap to be located at `/dev/zd0`.

 I'll investigate further. But for now there's at least a clear fix for this issue, which turns out not to be at all what it first appeared.

Re: [ISSUE] [CLOSED] newer releases of various kernels cause boot to hang with ZFS

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 2359 bytes --]

Closed issue by emacsomancer on void-packages repository

https://github.com/void-linux/void-packages/issues/12519
Description: ### System

* xuname:  
`Void 4.19.50_1 x86_64 GenuineIntel uptodate rrrmFFFF`
* package:  
`linux-4.19.51`, `linux-4.14.126`, `linux-5.1.10` (and I think `linux-5.1.9`) and both `zfs-0.8.0` and `zfs-0.8.1`.

### Expected behavior
The kernel versions which boot —  including `linux-4.19.50`, `linux-4.14.125` — proceed loading after enabling the ZFS kernel module, and successfully boot, e.g.:

```
.....
[    3.908527] dracut: luksOpen /dev/sda1 cryptz /rootkey.bin luks
[    6.366046] NET: Registered protocol family 38
[   11.486374] spl: loading out-of-tree module taints kernel.
[   11.492145] icp: module license 'CDDL' taints kernel.
[   11.492147] Disabling lock debugging due to kernel taint
[   12.410694] ZFS: Loaded module v0.8.0-1, ZFS pool version 5000, ZFS filesystem version 5
[   12.441836] dracut: ZFS: No sysroot.mount exists or zfs-generator did not extend it.
[   12.441902] dracut: ZFS: Mounting root with the traditional mount-zfs.sh instead.
[   12.457191] dracut: ZFS: Importing pool dozer...
[   13.388974] dracut: ZFS: Mounting dataset dozer/ROOT/system...
[   13.462031] dracut: Mounted root filesystem dozer/ROOT/system
[   13.482632] dracut: ZFS: There is an active pool, will export it.
[   13.530312] dracut: Switching root
[   13.715111] fuse init (API version 7.27)
[   13.742556] raid6: sse2x1   gen()  8789 MB/s
[   13.759555] raid6: sse2x1   xor()  7705 MB/s
[   13.776556] raid6: sse2x2   gen() 12730 MB/s
.....
```

### Actual behavior
All of the newer versions of kernels hang during boot, only proceeding as far as:
```
....
[    3.908527] dracut: luksOpen /dev/sda1 cryptz /rootkey.bin luks
[    6.366046] NET: Registered protocol family 38
[   11.486374] spl: loading out-of-tree module taints kernel.
[   11.492145] icp: module license 'CDDL' taints kernel.
[   11.492147] Disabling lock debugging due to kernel taint
[   12.410694] ZFS: Loaded module v0.8.0-1, ZFS pool version 5000, ZFS filesystem version 5

```
and then hangs indefinitely.


### Steps to reproduce the behavior
Choose one of the newer kernel releases  (e.g. anything more recent than `linux-4.19.50`, `linux-4.14.125`, or (I think) `linux-5.1.8`) in GRUB.

Re: newer releases of various kernels cause boot to hang with ZFS

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 1722 bytes --]

New comment by emacsomancer on void-packages repository

https://github.com/void-linux/void-packages/issues/12519#issuecomment-503410468
Comment:
Ok, so debugging this was a bit tricky. The gist is this: the recentness of the kernels was a red herring, the crucial factor was whether the kernel was installed before or after the zvol swap file was created. All kernels created after swap was created end up with a dracut configuration which (apparently by default) assumes that swap will be used for resume, and when booting any of these kernel results in dracut waiting about 2 minutes trying to finding the swap partition (so these kernels do *eventually* boot). 

So I'll close this issue (which wasn't what it appeared) but document a possible resolution:

One Potential Fix
==========
I don't really use resume, so at least one easy fix for this issue is to create a dracut conf file, e.g.:

/etc/dracut.conf.d/omit-resume-for-zvol-swap.conf containing:

`omit_dracutmodules+=" resume "`

and call `xbps-reconfigure -f` on the relevant kernels.

I'm not sure if a zvol swap can be used for resume/seen at the appropriate time by dracut or not.

Using Swap on ZFS root?
==============
It's possible that the location of the swap partition could be specified in a way that dracut could locate it (though, again, resume not being something I typically use, I'm not sure about swap size requirements for using resume). zvol-type swap ends up being mounted on `/dev/zvol/<POOL-NAME>/swap`. I think dracut looks for the resume/swap to be located at `/dev/zd0`.

 I'll investigate further. But for now there's at least a clear fix for this issue, which turns out not to be at all what it first appeared.