[ISSUE] kexec + hibernation doesn't work

[ISSUE] kexec + hibernation doesn't work

[r-ricci]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

New issue by r-ricci on void-packages repository

https://github.com/void-linux/void-packages/issues/54140

Description:
I'm trying to find the cause of this bug: https://github.com/zbm-dev/zfsbootmenu/issues/702.
I've reported it [here](https://lore.kernel.org/all/Z4WFjBVHpndct7br@desktop0a/), but others didn't have luck reproducing it.
So I'm asking here if someone else is able to reproduce or if I'm just hallucinating. I also think some kernel config used by Void is playing a role, but I don't know which one.

First, comment out `kernel.kexec_load_disabled=1` in /usr/lib/sysctl.d/10-void-user.conf and reboot (uncomment it again when done testing).
Then do something like this:

```
# kexec -l /boot/vmlinuz-6.12.11_1 --initrd=/boot/initramfs-6.12.11_1.img --reuse-cmdline
# reboot  #this calls kexec -e
# printf reboot >/sys/power/disk
# printf disk >/sys/power/state
```

A kernel oops should occur when the system resumes.

If possible, you should try this on a system without ZFS, so the kernel doesn't get tainted.

Please, let me know your suggestions.

Re: kexec + hibernation doesn't work

[mamba2410]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

New comment by mamba2410 on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2726992611

Comment:
Hi, I followed your steps in a fresh install in a qemu VM on ext4 and I end up rebooting and getting stuck in a black screen, not sure how to check if that's a kernel oops but it certainly doesn't feel like intended behaviour! One thing to note is that running `zzz -R` seems to work as intended. What's the reason for doing it the kexec way?

That being said, on my laptop which is on a ZFS root, both your instructions and `zzz -R` result in a frozen screen on boot, I'm assuming its a kernel oops.

I'm not familiar with this area of Linux, but happy to help test things since I can recreate the issue and the lack of hibernation on my laptop has been annoying me lol.

Re: kexec + hibernation doesn't work

[r-ricci]

[-- Attachment #1: Type: text/plain, Size: 1508 bytes --]

New comment by r-ricci on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727127421

Comment:
Thanks for your reply @mamba2410

The ZfsBootMenu bootloader is implemented as a Linux system which uses kexec to boot the final system. The prebuilt ZBM image downloaded from their website works fine. But when using a generated ZBM image based on Void's kernel (which is another supported setup), hibernation doesn't work. More info in the link in my original post and in ZBM docs.

I believe this is a kernel bug which only occurs when some of the configs (don't know which ones) used by our kernel builds are set. The reason I'm testing this on a plain Void install is to make sure the bug isn't caused by ZFS or ZBM.
A person on the kernel mailing list said they tried my config and got crashes, but nothing related kexec + hibernation.

***@***.***/>

***@***.***/>

On the other hand, I'm quite sure that this problem is in some way related to kexec + hibernation and I can reproduce it, so I opened this issue to see if others could too.

BTW, you can confirm it was an oops by looking at the output of the `dmesg` command. Or by looking at the text that gets printed on the screen. But if the oops messed up other parts of the system (including graphics) you can't do that.
So add `oops=panic` to the kernel command line and install and enable runit-kdump. The dmesg should get saved somewhere under /var/crash.

Do you use ZfsBootMenu on your ZFS laptop?

Re: kexec + hibernation doesn't work

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 194 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727301103

Comment:
the pre-built zbm images use void kernels too

Re: kexec + hibernation doesn't work

[mamba2410]

[-- Attachment #1: Type: text/plain, Size: 3931 bytes --]

New comment by mamba2410 on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727325198

Comment:
Thanks for the info, that's really useful. Seems like the prebuilt ZBM images use the Void kernel package versions 5.10, 5.15 and 6.1 according to the [Dockerfile](https://github.com/zbm-dev/zfsbootmenu/blob/v3.0.1/releng/docker/Dockerfile). This seems to line up with what you mentioned in the other discussions about it breaking after kernel 6.8, though I don't know if they use custom kernel configs or use the default Void ones.

Not sure how to tackle this problem aside from looking at the diffs between kernel configs, which would include a lot of noise since loads of options are changed. Looking at the diff between Void's 6.6 and 6.12 defconfigs and searching for kexec, I noticed these options were changed

```diff
@@ -304,10 +310,13 @@                                                                    
 #                                                                                       
# Kexec and crash features                                                              
#                                                                                       
-CONFIG_CRASH_CORE=y                                                                     
+CONFIG_CRASH_RESERVE=y                                                                  
+CONFIG_VMCORE_INFO=y                                                                    
 CONFIG_KEXEC_CORE=y                                                                     
 CONFIG_KEXEC=y                                                                          
-# CONFIG_KEXEC_FILE is not set                                                          
+CONFIG_KEXEC_FILE=y                                                                     
+CONFIG_KEXEC_SIG=y                                                                      
+# CONFIG_KEXEC_SIG_FORCE is not set                                                     
 # CONFIG_KEXEC_JUMP is not set                                                          
 CONFIG_CRASH_DUMP=y                                                                     
 CONFIG_CRASH_HOTPLUG=y
```

Looking at the diff between vanilla 6.12 and Void's 6.12 

```diff
@@ -285,7 +314,9 @@                                                                     
 CONFIG_VMCORE_INFO=y                                                                   
 CONFIG_KEXEC_CORE=y                                                                    
 CONFIG_KEXEC=y                                                                         
-# CONFIG_KEXEC_FILE is not set                                                         
+CONFIG_KEXEC_FILE=y                                                                    
+CONFIG_KEXEC_SIG=y                                                                     
+# CONFIG_KEXEC_SIG_FORCE is not set                                                    
 # CONFIG_KEXEC_JUMP is not set                                                         
 CONFIG_CRASH_DUMP=y                                                                    
 CONFIG_CRASH_HOTPLUG=y
```

the options were also changed, so I think this means that this is a Void-specific config option which has been added since the last known working kernel version. So, not knowing what this does, I reverted the change, recompiled and installed the new kernel but it doesn't seem to have fixed anything. Could have just been me doing it wrong though.

Yes I use ZBM on my laptop, made with `generate-zbm` so its using my host kernel of Void's 6.12. I installed `runit-kdump` and added `oops=panic crashkernel=256M` to my kernel commandline but I don't get a crash file. When I boot normally I do see `Loading crash kernel... OK` in the startup logs so I think its working? Sorry I'm not much help.

Re: kexec + hibernation doesn't work

[mamba2410]

[-- Attachment #1: Type: text/plain, Size: 4330 bytes --]

New comment by mamba2410 on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727325198

Comment:
Thanks for the info, that's really useful. Seems like the prebuilt ZBM images use the Void kernel package versions 5.10, 5.15 and 6.1 according to the [Dockerfile](https://github.com/zbm-dev/zfsbootmenu/blob/v3.0.1/releng/docker/Dockerfile). This seems to line up with what you mentioned in the other discussions about it breaking after kernel 6.8, though I don't know if they use custom kernel configs or use the default Void ones.

Not sure how to tackle this problem aside from looking at the diffs between kernel configs, which would include a lot of noise since loads of options are changed. Looking at the diff between Void's 6.6 and 6.12 defconfigs and searching for kexec, I noticed these options were changed

```diff
@@ -304,10 +310,13 @@                                                                    
 #                                                                                       
# Kexec and crash features                                                              
#                                                                                       
-CONFIG_CRASH_CORE=y                                                                     
+CONFIG_CRASH_RESERVE=y                                                                  
+CONFIG_VMCORE_INFO=y                                                                    
 CONFIG_KEXEC_CORE=y                                                                     
 CONFIG_KEXEC=y                                                                          
-# CONFIG_KEXEC_FILE is not set                                                          
+CONFIG_KEXEC_FILE=y                                                                     
+CONFIG_KEXEC_SIG=y                                                                      
+# CONFIG_KEXEC_SIG_FORCE is not set                                                     
 # CONFIG_KEXEC_JUMP is not set                                                          
 CONFIG_CRASH_DUMP=y                                                                     
 CONFIG_CRASH_HOTPLUG=y
```

Looking at the diff between vanilla 6.12 and Void's 6.12 

```diff
@@ -285,7 +314,9 @@                                                                     
 CONFIG_VMCORE_INFO=y                                                                   
 CONFIG_KEXEC_CORE=y                                                                    
 CONFIG_KEXEC=y                                                                         
-# CONFIG_KEXEC_FILE is not set                                                         
+CONFIG_KEXEC_FILE=y                                                                    
+CONFIG_KEXEC_SIG=y                                                                     
+# CONFIG_KEXEC_SIG_FORCE is not set                                                    
 # CONFIG_KEXEC_JUMP is not set                                                         
 CONFIG_CRASH_DUMP=y                                                                    
 CONFIG_CRASH_HOTPLUG=y
```

the options were also changed, so I think this means that this is a Void-specific config option which has been added since the last known working kernel version. So, not knowing what this does, I reverted the change, recompiled and installed the new kernel ~~but it doesn't seem to have fixed anything. Could have just been me doing it wrong though.~~ Upon *actually using* the custom dotconfig for the kernel, this does appear to fix the problem - installing the new kernel with `# CONFIG_KEXEC_FILE is not set` and running `generate-zbm` again  allows me to run `zzz -R` and it resumes perfectly fine.

I have no idea what impact unsetting `CONFIG_KEXEC_FILE` has on the wider system, but at least on my machine it fixed the hibernation issue. 

Yes I use ZBM on my laptop, made with `generate-zbm` so its using my host kernel of Void's 6.12. I installed `runit-kdump` and added `oops=panic crashkernel=256M` to my kernel commandline but I don't get a crash file. When I boot normally I do see `Loading crash kernel... OK` in the startup logs so I think its working? Sorry I'm not much help.

Re: kexec + hibernation doesn't work

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 287 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727369107

Comment:
the zbm dockerfile is not used, the latest zbm release used 6.1, 6.6, and 6.12: https://github.com/zbm-dev/zfsbootmenu/releases/tag/v3.0.1

Re: kexec + hibernation doesn't work

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 336 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727369107

Comment:
the zbm dockerfile is not used, the latest zbm release used 6.1, 6.6, and 6.12: https://github.com/zbm-dev/zfsbootmenu/releases/tag/v3.0.1

zbm uses the void kernels with no custom config

Re: kexec + hibernation doesn't work

[mamba2410]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

New comment by mamba2410 on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727386425

Comment:
Just tested it and the ZBM EFI image based on 6.6 allows me to run `zzz -R` and it wakes up fine, using the image based off of 6.12 causes my system to freeze then reset upon waking up. Unfortunately I'm not catching the logs in `/var/crash/` or in socklog logs. Sorry, I should have tested this earlier before assuming the release versions did work.

Re: kexec + hibernation doesn't work

[r-ricci]

[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]

New comment by r-ricci on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727127421

Comment:
Thanks for your reply @mamba2410

The ZfsBootMenu bootloader is implemented as a Linux system which uses kexec to boot the final system. The prebuilt ZBM image downloaded from their website works fine. But when using a generated ZBM image based on Void's kernel (which is another supported setup), hibernation doesn't work. More info in the link in my original post and in ZBM docs.

I believe this is a kernel bug which only occurs when some of the configs (don't know which ones) used by our kernel builds are set. The reason I'm testing this on a plain Void install is to make sure the bug isn't caused by ZFS or ZBM.
A person on the kernel mailing list said they tried my config and got crashes, but nothing related kexec + hibernation.

<https://lore.kernel.org/all/Z5bx7ZHNcyc5fM_L@darkstar.users.ipa.redhat.com/>

<https://lore.kernel.org/all/CALu+AoSSKh=5ELgQyzDrGEDm5fm2XKteH1ZC70mm89pNSSPMHw@mail.gmail.com/>

On the other hand, I'm quite sure that this problem is in some way related to kexec + hibernation and I can reproduce it, so I opened this issue to see if others could too.

BTW, you can confirm it was an oops by looking at the output of the `dmesg` command. Or by looking at the text that gets printed on the screen. But if the oops messed up other parts of the system (including graphics) you can't do that.
So add `oops=panic` to the kernel command line and install and enable runit-kdump. The dmesg should get saved somewhere under /var/crash.

Do you use ZfsBootMenu on your ZFS laptop?

Re: kexec + hibernation doesn't work

[r-ricci]

[-- Attachment #1: Type: text/plain, Size: 1652 bytes --]

New comment by r-ricci on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727127421

Comment:
Thanks for your reply @mamba2410

The ZfsBootMenu bootloader is implemented as a Linux system which uses kexec to boot the final system. The prebuilt ZBM image downloaded from their website works fine. But when using a generated ZBM image based on Void's kernel (which is another supported setup), hibernation doesn't work. More info in the link in my original post and in ZBM docs.

I believe this is a kernel bug which only occurs when some of the configs (don't know which ones) used by our kernel builds are set. The reason I'm testing this on a plain Void install is to make sure the bug isn't caused by ZFS or ZBM.
A person on the kernel mailing list said they tried my config and got crashes, but nothing related kexec + hibernation.

https://lore.kernel.org/all/Z5bx7ZHNcyc5fM_L@darkstar.users.ipa.redhat.com/

https://lore.kernel.org/all/CALu+AoSSKh=5ELgQyzDrGEDm5fm2XKteH1ZC70mm89pNSSPMHw@mail.gmail.com/

On the other hand, I'm quite sure that this problem is in some way related to kexec + hibernation and I can reproduce it, so I opened this issue to see if others could too.

BTW, you can confirm it was an oops by looking at the output of the `dmesg` command. Or by looking at the text that gets printed on the screen. But if the oops messed up other parts of the system (including graphics) you can't do that.
So add `oops=panic` to the kernel command line and install and enable runit-kdump. The dmesg should get saved somewhere under /var/crash.

Do you use ZfsBootMenu on your ZFS laptop?

Re: kexec + hibernation doesn't work

[r-ricci]

[-- Attachment #1: Type: text/plain, Size: 456 bytes --]

New comment by r-ricci on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727437428

Comment:
> I installed `runit-kdump` and added `oops=panic crashkernel=256M` to my kernel commandline but I don't get a crash file. When I boot normally I do see `Loading crash kernel... OK` in the startup logs so I think its working? Sorry I'm not much help.

IIRC, I had to reserve 512M, instead of 256, for it to work.

Re: kexec + hibernation doesn't work

[r-ricci]

[-- Attachment #1: Type: text/plain, Size: 405 bytes --]

New comment by r-ricci on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727448065

Comment:
> the zbm dockerfile is not used, the latest zbm release used 6.1, 6.6, and 6.12: https://github.com/zbm-dev/zfsbootmenu/releases/tag/v3.0.1

When I first reported the issue the latest ZBM was 2.3.0. I guess the prebuilt image didn't use 6.12 back then, did it?

Re: kexec + hibernation doesn't work

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 172 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727457338

Comment:
no, it was probably 6.1

Re: kexec + hibernation doesn't work

[mamba2410]

[-- Attachment #1: Type: text/plain, Size: 282 bytes --]

New comment by mamba2410 on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2727495661

Comment:
> IIRC, I had to reserve 512M, instead of 256, for it to work.

Thanks that worked, can confirm I get a kernel oops doing it on on 6.12.

Re: kexec + hibernation doesn't work

[r-ricci]

[-- Attachment #1: Type: text/plain, Size: 2612 bytes --]

New comment by r-ricci on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2734941052

Comment:
> Looking at the diff between vanilla 6.12 and Void's 6.12
>
> ```diff
> @@ -285,7 +314,9 @@                                                                     
>  CONFIG_VMCORE_INFO=y                                                                   
>  CONFIG_KEXEC_CORE=y                                                                    
>  CONFIG_KEXEC=y                                                                         
> -# CONFIG_KEXEC_FILE is not set                                                         
> +CONFIG_KEXEC_FILE=y                                                                    
> +CONFIG_KEXEC_SIG=y                                                                     
> +# CONFIG_KEXEC_SIG_FORCE is not set                                                    
>  # CONFIG_KEXEC_JUMP is not set                                                         
>  CONFIG_CRASH_DUMP=y                                                                    
>  CONFIG_CRASH_HOTPLUG=y
> ```
> 
> the options were also changed, so I think this means that this is a Void-specific config option which has been added since the last known working kernel version. So, not knowing what this does, I reverted the change, recompiled and installed the new kernel ~but it doesn't seem to have fixed anything. Could have just been me doing it wrong though.~ Upon _actually using_ the custom dotconfig for the kernel, this does appear to fix the problem - installing the new kernel with `# CONFIG_KEXEC_FILE is not set` and running `generate-zbm` again allows me to run `zzz -R` and it resumes perfectly fine.
> 
> I have no idea what impact unsetting `CONFIG_KEXEC_FILE` has on the wider system, but at least on my machine it fixed the hibernation issue.

`CONFIG_KEXEC_FILE` enables `kexec_file_load`, the newer kexec syscall (the older is `kexec_load`). `kexec_file_load` is now the default for kexec-tools.
I didn't notice that the kernel had this config not enabled by default, thank you for having suggested it.

Removing `CONFIG_KEXEC_FILE` from Void's config, despite not leading to a crash, doesn't make hibernation work for me. `zzz -R` just reboots without resuming. Same thing with plain defconfig, and on both v6.6 and latest -rc.
But using defconfig + `CONFIG_KEXEC_FILE=y` is enough to get the oops.

Anyway, the config I sent to the kernel mailing list *did* have `CONFIG_KEXEC_FILE` enabled. I still can't explain why they couldn't reproduce.

Re: [ISSUE] [CLOSED] kexec + hibernation doesn't work

[r-ricci]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

Closed issue by r-ricci on void-packages repository

https://github.com/void-linux/void-packages/issues/54140

Description:
I'm trying to find the cause of this bug: https://github.com/zbm-dev/zfsbootmenu/issues/702.
I've reported it [here](https://lore.kernel.org/all/Z4WFjBVHpndct7br@desktop0a/), but others didn't have luck reproducing it.
So I'm asking here if someone else is able to reproduce or if I'm just hallucinating. I also think some kernel config used by Void is playing a role, but I don't know which one.

First, comment out `kernel.kexec_load_disabled=1` in /usr/lib/sysctl.d/10-void-user.conf and reboot (uncomment it again when done testing).
Then do something like this:

```
# kexec -l /boot/vmlinuz-6.12.11_1 --initrd=/boot/initramfs-6.12.11_1.img --reuse-cmdline
# reboot  #this calls kexec -e
# printf reboot >/sys/power/disk
# printf disk >/sys/power/state
```

A kernel oops should occur when the system resumes.

If possible, you should try this on a system without ZFS, so the kernel doesn't get tainted.

Please, let me know your suggestions.

Re: kexec + hibernation doesn't work

[r-ricci]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

New comment by r-ricci on void-packages repository

https://github.com/void-linux/void-packages/issues/54140#issuecomment-2780728025

Comment:
https://lore.kernel.org/all/20250405-fix-e820-nosave-v2-1-d40dbe457c95@qtmlabs.xyz/