[ISSUE] XBPS ordering problem with ZFS (root on zfs)

[ISSUE] XBPS ordering problem with ZFS (root on zfs)

[aNTDwcDqvXR4h2PH]

[-- Attachment #1: Type: text/plain, Size: 2662 bytes --]

New issue by aNTDwcDqvXR4h2PH on void-packages repository

https://github.com/void-linux/void-packages/issues/50859

Description:
### Is this a new report?

Yes

### System Info

Void 6.6.33_1 x86_64 AuthenticAMD notuptodate rF

### Package(s) Affected

zfs-2.2.4_2, xbps-0.59.2_3, linux-6.6_1

### Does a report exist for this bug with the project's home (upstream) and/or another distro?

I am aware that the issue was discussed in #22817 however this issue is related to the same problem but different.

### Expected behaviour

If I remember correctly on fresh install xbps does the following:
1. Download/verify packages
2. Extract packages
3. Build DKMS
4. Include DKMS in kernel
5. Generate initramfs

### Actual behaviour

Issue begins when you have 2 kernel versions and update system with `xbps-install -Su`

What happens:
1. `xbps` downloads/verifies packages
2. `xbps` extracts some packages
2.1. When zfs package is extracted `xbps` doesn't build DKMS and runs `dracut`
2.2. `dracut` shows error about missing zfs module
2.3. Rest of the packages is extracted
3. DKMS is built.
4. initramfs is generated without zfs kernel module.

Note: I'm not entirely sure if zfs module is missing in newest kernel or all of them.

### Steps to reproduce

1. Follow zfsbootmenu installation guide: https://docs.zfsbootmenu.org/en/v2.3.x/guides/void-linux/uefi.html
1.1. Set up: Single SATA Disk, Encrypted (I only omitted "-o compatibility"), zfsbootmenu package, refind in `/boot/efi/EFI/BOOT`
2. Setup Xorg, dwm, startx, zsh, apparmor (complain mode) and some other packages.
2.1. Right now there should be only one kernel version.
3. Wait for new kernel version.
4. Update system with `xbps-install -Su`
4.1. Now there should be 2 kernel versions.
5. See step 3.
6. See step 4.
6.1. Now there should be 3 kernel versions and the isssue should occur.
6.2. Dracut will run too soon and not include zfs module.

You can however mitigate this issue if you run `xbps-reconfigure -f zfs` after step 6.2. 

## System information:
### Users:
  - root
  - user

### ZFS pool
```
NAME                     USED  AVAIL  REFER  MOUNTPOINT
zroot                   7.39G   892G   192K  none
zroot/DATA              1.32G   892G   192K  none
zroot/DATA/home         1.32G   892G  1.32G  /home
zroot/DATA/root          480K   892G   480K  /root
zroot/ROOT              6.07G   892G   192K  none
zroot/ROOT/void         4.22G   892G  4.22G  /
zroot/ROOT/void-backup  1.85G   892G  1.85G  /
```
Note: This affects all my installations that have multiple kernel versions

### Kernel
- zbm-kcl: quiet apparmor=1 security=apparmor

Re: XBPS ordering problem with ZFS and multiple kernel versions (root on zfs)

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 391 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/50859#issuecomment-2173442330

Comment:
this isn't really solvable until xbps gets a proper hook system. for now you could xbps-reconfigure -f each kernel after updating

see:
- https://github.com/void-linux/void-packages/issues/22817
- https://github.com/void-linux/xbps/issues/304

Re: XBPS ordering problem with ZFS and multiple kernel versions (root on zfs)

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/50859#issuecomment-2173715176

Comment:
If you update a kernel and there is no ZFS update, the kernel package should trigger the scripts, in order,
1. `/etc/kernel.d/post-install/10-dkms` to build all DKMS modules for the newly installed kernel, which should include ZFS; and
2. `/etc/kernel.d/post-install/20-initramfs` to build the initramfs for the newly installed kernel.

The initramfs build happens after the DKMS build, so the ZFS modules should be available. No previously installed kernels or their initramfs images will be touched in this case.

If you update ZFS and there is no kernel update, the DKMS hooks in the ZFS package will trigger, in order:
1. Removal of any old ZFS modules for each installed kernel;
2. A build of the new ZFS modules for each installed kernel; and
3. A regeneration of the initramfs image for every kernel.

Note that every installed kernel must have its headers available, or the build will be skipped, and the regeneration in the final step will most likely result in the creation of an initramfs image without ZFS or other DKMS modules.

If you update ZFS and any kernel series simultaneously:
1. The pre-remove DKMS trigger in the ZFS package should trigger the removal of existing ZFS modules from all kernel versions.
2. All packages are unpacked. (Your diagnosis that only some packages are extracted is wrong; all packages are extracted before any is configured, but the pre-remove trigger was run before this stage.)
3. Packages are configured in alphabetical order:
    a. Any updated Linux package will trigger execution of `/etc/kernel.d/post-install` scripts: (i) the DKMS script will rebuild all DKMS modules, but ZFS has not yet been registered, so the rebuild will not include ZFS; and (ii) the initramfs script will indeed build an initramfs without ZFS modules (which may cause a dracut failure).
    b. The updated ZFS package will trigger a rebuild of ZFS for all installed kernels.
    c. The updated ZFS package will trigger a regeneration of an initramfs for all installed kernels.

Although you will have a broken initramfs image for each updated kernel series in an intermediate stage, the final ZFS configuration should correct the issue for all kernel versions. Of course, all kernel versions must have their headers available, or DKMS will fail.

Which of these scenarios is not working correctly for you? Are you sure you have kernel headers available for every version of the kernel affected by your issue?
 

Re: XBPS ordering problem with ZFS and multiple kernel versions (root on zfs)

[aNTDwcDqvXR4h2PH]

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

New comment by aNTDwcDqvXR4h2PH on void-packages repository

https://github.com/void-linux/void-packages/issues/50859#issuecomment-2174018324

Comment:
> 2. All packages are unpacked. (Your diagnosis that only some packages are extracted is wrong; all packages are extracted before any is configured, but the pre-remove trigger was run before this stage.)

Oh, you're right it's removing DKMS and at that point I can still see dracut error: module zfs not found.

Also there should be headers for each kernel since I haven't seen any errors with DKMS only dracut.
It's weird that this issue only occurs when there are multiple kernel versions.

I've already removed old kernels with `vkpurge rm all`, will have to wait for new kernel or zfs module and see what happens -_-
Here is a list of all installed packages: [packages.txt](https://github.com/user-attachments/files/15875059/packages.txt)

Re: XBPS ordering problem with ZFS and multiple kernel versions (root on zfs)

[aNTDwcDqvXR4h2PH]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

New comment by aNTDwcDqvXR4h2PH on void-packages repository

https://github.com/void-linux/void-packages/issues/50859#issuecomment-2174018324

Comment:
> 2. All packages are unpacked. (Your diagnosis that only some packages are extracted is wrong; all packages are extracted before any is configured, but the pre-remove trigger was run before this stage.)

Oh, you're right it's removing DKMS and at that point I saw dracut error: module zfs not found.

Also there should be headers for each kernel since I haven't seen any errors with DKMS only dracut.
It's weird that this issue only occurs when there are multiple kernel versions.

I've already removed old kernels with `vkpurge rm all`, will have to wait for new kernel or zfs module and see what happens -_-
Here is a list of all installed packages: [packages.txt](https://github.com/user-attachments/files/15875059/packages.txt)

Re: XBPS ordering problem with ZFS and multiple kernel versions (root on zfs)

[aNTDwcDqvXR4h2PH]

[-- Attachment #1: Type: text/plain, Size: 374 bytes --]

New comment by aNTDwcDqvXR4h2PH on void-packages repository

https://github.com/void-linux/void-packages/issues/50859#issuecomment-2198580863

Comment:
Closing the issue and marking it as duplicate since it's the same thing as classabbyamp pointed out.

Current "solution":
1. Reconfigure kernel after update with `xbps-reconfigure -f`
2. Reboot and run `vkpuge rm all`

Re: [ISSUE] [CLOSED] XBPS ordering problem with ZFS and multiple kernel versions (root on zfs)

[aNTDwcDqvXR4h2PH]

[-- Attachment #1: Type: text/plain, Size: 3488 bytes --]

Closed issue by aNTDwcDqvXR4h2PH on void-packages repository

https://github.com/void-linux/void-packages/issues/50859

Description:
### Is this a new report?

No

### System Info

Void 6.6.33_1 x86_64 AuthenticAMD notuptodate rF

### Package(s) Affected

xbps-0.59.2_3, zfs-2.2.4_2

### Does a report exist for this bug with the project's home (upstream) and/or another distro?

I am aware that the issue was discussed in #22817 however this issue is related to the same problem but different.

### Expected behaviour

If I remember correctly on fresh install xbps does the following:
1. Download/verify packages
2. Extract packages
3. Build DKMS
4. Include DKMS in kernel
5. Generate initramfs

### Actual behaviour

Issue begins when you have 2 kernel versions and update system with `xbps-install -Su` and there is new kernel or new zfs module.

What happens:
1. `xbps` downloads/verifies packages
2. `xbps` extracts some packages
2.1. ~When zfs package is extracted `xbps` doesn't build DKMS and runs `dracut`~ DKMS removes old module.
2.2. `dracut` shows error about missing zfs module
2.3. Rest of the packages is extracted
3. DKMS is built.
4. initramfs is generated without zfs kernel module.

Note: I'm not entirely sure if zfs module is missing in newest kernel or all of them.

### Steps to reproduce

1. Follow zfsbootmenu installation guide: https://docs.zfsbootmenu.org/en/v2.3.x/guides/void-linux/uefi.html
1.1. Set up: Single SATA Disk, Encrypted (I only omitted "-o compatibility"), zfsbootmenu package, refind in `/boot/efi/EFI/BOOT`
2. Setup Xorg, dwm, startx, zsh, apparmor (complain mode) and some other packages.
2.1. Right now there should be only one kernel version.
3. Wait for new kernel version.
4. Update system with `xbps-install -Su`
4.1. Now there should be 2 kernel versions.
5. See step 3.
6. See step 4.
6.1. Now there should be 3 kernel versions and the isssue should occur.
6.2. Dracut will run too soon and not include zfs module.

You can however mitigate this issue if you run `xbps-reconfigure -f zfs` after step 6.2. 

## System information:
### Users:
  - root
  - user

### ZFS pool
```
NAME                     USED  AVAIL  REFER  MOUNTPOINT
zroot                   7.39G   892G   192K  none
zroot/DATA              1.32G   892G   192K  none
zroot/DATA/home         1.32G   892G  1.32G  /home
zroot/DATA/root          480K   892G   480K  /root
zroot/ROOT              6.07G   892G   192K  none
zroot/ROOT/void         4.22G   892G  4.22G  /
zroot/ROOT/void-backup  1.85G   892G  1.85G  /
```
Note: This issue affects all my installations that have multiple kernel versions

### Kernel
- zbm-kcl: quiet apparmor=1 security=apparmor
- `ls -lah /boot`
```
total 284M
drwxr-xr-x  3 root root   12 Jun 16 23:38 ./
drwxr-xr-x 16 root root   21 May 20 18:33 ../
drwxr-xr-x  3 root root 4.0K Jan  1  1970 efi/
-rw-r--r--  1 root root 262K May 18 03:07 config-6.6.31_1
-rw-r--r--  1 root root 262K May 29 01:00 config-6.6.32_1
-rw-r--r--  1 root root 262K Jun 16 02:15 config-6.6.33_1
-rw-------  1 root root  83M Jun 16 23:49 initramfs-6.6.31_1.img
-rw-------  1 root root  83M Jun 16 23:50 initramfs-6.6.32_1.img
-rw-------  1 root root  82M Jun 16 23:50 initramfs-6.6.33_1.img
-rw-r--r--  1 root root  13M May 18 03:07 vmlinuz-6.6.31_1
-rw-r--r--  1 root root  13M May 29 01:00 vmlinuz-6.6.32_1
-rw-r--r--  1 root root  13M Jun 16 02:15 vmlinuz-6.6.33_1
```

Re: XBPS ordering problem with ZFS and multiple kernel versions (root on zfs)

[aNTDwcDqvXR4h2PH]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]

New comment by aNTDwcDqvXR4h2PH on void-packages repository

https://github.com/void-linux/void-packages/issues/50859#issuecomment-2198580863

Comment:
Closing the issue and marking it as duplicate since it's the same thing as classabbyamp pointed out.

Current "solution":
1. Reconfigure kernel after update with `xbps-reconfigure -f linuxX.Y`
2. Reboot and run `vkpuge rm all`