* [developer] Panic during pkg update
@ 2024-10-30 18:57 Gary Mills
2024-10-30 19:10 ` Peter Tribble
2024-11-11 18:36 ` Toomas Soome via illumos-developer
0 siblings, 2 replies; 7+ messages in thread
From: Gary Mills @ 2024-10-30 18:57 UTC (permalink / raw)
To: developer
I'm not sure if this is a bug or just ZFS being careful, but I got a
panic and reboot while I was doing a "pkg update". The system
has an AMD 6-core CPU with B550 support hardware. The next
"pkg update" completed normally, without a panic. Here's what
I found in /var/adm/messages. Does it look familiar?
Oct 30 09:14:31 b550 unix: [ID 836849 kern.notice]
Oct 30 09:14:31 b550 ^Mpanic[cpu4]/thread=fffffe2cc9e88780:
Oct 30 09:14:31 b550 genunix: [ID 129249 kern.notice] checksum of cached data doesn't match BP err=50 hdr=fffffe3d478f51c0 bp=fffffe0040433988 abd=fffffe3d478f7cc0 buf=fffffe3b5a6f9000
Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433760 zfs:zfs_nfsshare_inited+378b87f0 ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433890 zfs:arc_read+de1 ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe00404338e0 zfs:dbuf_issue_final_prefetch+77 ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433a70 zfs:dbuf_prefetch_impl+502 ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433b20 zfs:dmu_zfetch+2ed ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433bd0 zfs:dmu_buf_hold_array_by_dnode+321 ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433c70 zfs:dmu_read_uio_dnode+54 ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433cc0 zfs:dmu_read_uio_dbuf+51 ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433d60 zfs:zfs_read+19c ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433de0 genunix:fop_read+60 ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f00 genunix:read+2b5 ()
Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f10 unix:brand_sys_syscall+1fe ()
Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
Oct 30 09:14:31 b550 genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
Oct 30 09:14:31 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 0 reset port
Oct 30 09:14:32 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 1 reset port
Oct 30 09:14:50 b550 genunix: [ID 100000 kern.notice]
Oct 30 09:14:50 b550 genunix: [ID 665016 kern.notice] ^M100% done: 859875 pages dumped,
Oct 30 09:14:50 b550 genunix: [ID 851671 kern.notice] dump succeeded
Oct 30 09:15:34 b550 genunix: [ID 107833 kern.notice] ^MOpenIndiana Hipster 2022.10 Version illumos-806838751b 64-bit
--
-Gary Mills- -refurb- -Winnipeg, Manitoba, Canada-
------------------------------------------
illumos: illumos-developer
Permalink: https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-M70e6e59ba1570ed5ad3b1843
Delivery options: https://illumos.topicbox.com/groups/developer/subscription
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [developer] Panic during pkg update
2024-10-30 18:57 [developer] Panic during pkg update Gary Mills
@ 2024-10-30 19:10 ` Peter Tribble
2024-10-30 20:57 ` Gary Mills
2024-11-11 18:36 ` Toomas Soome via illumos-developer
1 sibling, 1 reply; 7+ messages in thread
From: Peter Tribble @ 2024-10-30 19:10 UTC (permalink / raw)
To: illumos-developer
[-- Attachment #1: Type: text/plain, Size: 3557 bytes --]
On Wed, Oct 30, 2024 at 6:57 PM Gary Mills <gary_mills@fastmail.fm> wrote:
> I'm not sure if this is a bug or just ZFS being careful, but I got a
> panic and reboot while I was doing a "pkg update". The system
> has an AMD 6-core CPU with B550 support hardware. The next
> "pkg update" completed normally, without a panic. Here's what
> I found in /var/adm/messages. Does it look familiar?
>
Looks similar to 12242
https://www.illumos.org/issues/12242
> Oct 30 09:14:31 b550 unix: [ID 836849 kern.notice]
> Oct 30 09:14:31 b550 ^Mpanic[cpu4]/thread=fffffe2cc9e88780:
> Oct 30 09:14:31 b550 genunix: [ID 129249 kern.notice] checksum of cached
> data doesn't match BP err=50 hdr=fffffe3d478f51c0 bp=fffffe0040433988
> abd=fffffe3d478f7cc0 buf=fffffe3b5a6f9000
> Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433760
> zfs:zfs_nfsshare_inited+378b87f0 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433890
> zfs:arc_read+de1 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe00404338e0
> zfs:dbuf_issue_final_prefetch+77 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433a70
> zfs:dbuf_prefetch_impl+502 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433b20
> zfs:dmu_zfetch+2ed ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433bd0
> zfs:dmu_buf_hold_array_by_dnode+321 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433c70
> zfs:dmu_read_uio_dnode+54 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433cc0
> zfs:dmu_read_uio_dbuf+51 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433d60
> zfs:zfs_read+19c ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433de0
> genunix:fop_read+60 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f00
> genunix:read+2b5 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f10
> unix:brand_sys_syscall+1fe ()
> Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
> Oct 30 09:14:31 b550 genunix: [ID 111219 kern.notice] dumping to
> /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
> Oct 30 09:14:31 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0:
> ahci_tran_reset_dport port 0 reset port
> Oct 30 09:14:32 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0:
> ahci_tran_reset_dport port 1 reset port
> Oct 30 09:14:50 b550 genunix: [ID 100000 kern.notice]
> Oct 30 09:14:50 b550 genunix: [ID 665016 kern.notice] ^M100% done: 859875
> pages dumped,
> Oct 30 09:14:50 b550 genunix: [ID 851671 kern.notice] dump succeeded
> Oct 30 09:15:34 b550 genunix: [ID 107833 kern.notice] ^MOpenIndiana
> Hipster 2022.10 Version illumos-806838751b 64-bit
>
>
> --
> -Gary Mills- -refurb- -Winnipeg, Manitoba,
> Canada-
>
> ------------------------------------------
> illumos: illumos-developer
> Permalink:
> https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-M70e6e59ba1570ed5ad3b1843
> Delivery options:
> https://illumos.topicbox.com/groups/developer/subscription
>
--
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
------------------------------------------
illumos: illumos-developer
Permalink: https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-Mb3a8e17dbdbd92d15f5db628
Delivery options: https://illumos.topicbox.com/groups/developer/subscription
[-- Attachment #2: Type: text/html, Size: 5321 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [developer] Panic during pkg update
2024-10-30 19:10 ` Peter Tribble
@ 2024-10-30 20:57 ` Gary Mills
2024-11-11 17:19 ` Gergő Doma
0 siblings, 1 reply; 7+ messages in thread
From: Gary Mills @ 2024-10-30 20:57 UTC (permalink / raw)
To: illumos-developer
On Wed, Oct 30, 2024 at 07:10:47PM +0000, Peter Tribble wrote:
> Looks similar to 12242
> [2]https://www.illumos.org/issues/12242
Thanks. Yes, it does. So, it's been fixed. I was upgrading from
hipster-20230306 to hipster-20241030. It was the kernel from
2023-03-06 that panicked.
--
-Gary Mills- -refurb- -Winnipeg, Manitoba, Canada-
------------------------------------------
illumos: illumos-developer
Permalink: https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-M148f55cde38926233be0b045
Delivery options: https://illumos.topicbox.com/groups/developer/subscription
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [developer] Panic during pkg update
2024-10-30 20:57 ` Gary Mills
@ 2024-11-11 17:19 ` Gergő Doma
0 siblings, 0 replies; 7+ messages in thread
From: Gergő Doma @ 2024-11-11 17:19 UTC (permalink / raw)
To: illumos-developer
[-- Attachment #1: Type: text/plain, Size: 346 bytes --]
>
> So, it's been fixed.
>
It seems to me that this is still an open issue - not fixed.
------------------------------------------
illumos: illumos-developer
Permalink: https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-Ma339ab77916f5339647f367a
Delivery options: https://illumos.topicbox.com/groups/developer/subscription
[-- Attachment #2: Type: text/html, Size: 995 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [developer] Panic during pkg update
2024-10-30 18:57 [developer] Panic during pkg update Gary Mills
2024-10-30 19:10 ` Peter Tribble
@ 2024-11-11 18:36 ` Toomas Soome via illumos-developer
2024-11-11 21:11 ` gusev.vitaliy via illumos-developer
1 sibling, 1 reply; 7+ messages in thread
From: Toomas Soome via illumos-developer @ 2024-11-11 18:36 UTC (permalink / raw)
To: illumos-developer
[-- Attachment #1: Type: text/plain, Size: 3690 bytes --]
> On 30. Oct 2024, at 20:57, Gary Mills <gary_mills@fastmail.fm> wrote:
>
> I'm not sure if this is a bug or just ZFS being careful, but I got a
> panic and reboot while I was doing a "pkg update". The system
> has an AMD 6-core CPU with B550 support hardware. The next
> "pkg update" completed normally, without a panic. Here's what
> I found in /var/adm/messages. Does it look familiar?
>
> Oct 30 09:14:31 b550 unix: [ID 836849 kern.notice]
> Oct 30 09:14:31 b550 ^Mpanic[cpu4]/thread=fffffe2cc9e88780:
> Oct 30 09:14:31 b550 genunix: [ID 129249 kern.notice] checksum of cached data doesn't match BP err=50 hdr=fffffe3d478f51c0 bp=fffffe0040433988 abd=fffffe3d478f7cc0 buf=fffffe3b5a6f9000
> Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433760 zfs:zfs_nfsshare_inited+378b87f0 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433890 zfs:arc_read+de1 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe00404338e0 zfs:dbuf_issue_final_prefetch+77 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433a70 zfs:dbuf_prefetch_impl+502 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433b20 zfs:dmu_zfetch+2ed ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433bd0 zfs:dmu_buf_hold_array_by_dnode+321 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433c70 zfs:dmu_read_uio_dnode+54 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433cc0 zfs:dmu_read_uio_dbuf+51 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433d60 zfs:zfs_read+19c ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433de0 genunix:fop_read+60 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f00 genunix:read+2b5 ()
> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f10 unix:brand_sys_syscall+1fe ()
> Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
> Oct 30 09:14:31 b550 genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
> Oct 30 09:14:31 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 0 reset port
> Oct 30 09:14:32 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 1 reset port
> Oct 30 09:14:50 b550 genunix: [ID 100000 kern.notice]
> Oct 30 09:14:50 b550 genunix: [ID 665016 kern.notice] ^M100% done: 859875 pages dumped,
> Oct 30 09:14:50 b550 genunix: [ID 851671 kern.notice] dump succeeded
> Oct 30 09:15:34 b550 genunix: [ID 107833 kern.notice] ^MOpenIndiana Hipster 2022.10 Version illumos-806838751b 64-bit
>
>
Dan got blown up while running zfs-tests (rsend), and that resulted on me picking one series of updates from OpenZFS concerning dbuf and dmu. There are still few XXX notes for myself, but so far both debug and non-debug builds have been behaving nicely (debug build used to run zfs-tests). I have seen myself also panic from arc (ASSERT fired while running zfs-tests on debuilg build — that was before the work mentioned above). Most likely need to pick some arc bits as well.
The current wip branch is: https://github.com/tsoome/illumos-gate/tree/rsend if you like to test. The problem about those panics is that they seem to be random, or at least not easily repeatable.
rgds,
toomas
------------------------------------------
illumos: illumos-developer
Permalink: https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-Mb8af562876078fc2e3f7037c
Delivery options: https://illumos.topicbox.com/groups/developer/subscription
[-- Attachment #2: Type: text/html, Size: 4768 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [developer] Panic during pkg update
2024-11-11 18:36 ` Toomas Soome via illumos-developer
@ 2024-11-11 21:11 ` gusev.vitaliy via illumos-developer
2024-11-12 6:18 ` gusev.vitaliy via illumos-developer
0 siblings, 1 reply; 7+ messages in thread
From: gusev.vitaliy via illumos-developer @ 2024-11-11 21:11 UTC (permalink / raw)
To: illumos-developer, Toomas Soome via illumos-developer; +Cc: Tom Caputi
[-- Attachment #1: Type: text/plain, Size: 7309 bytes --]
Panic comes from this function:
usr/src/uts/common/fs/zfs/arc.c
5369 /*
5370 * XXX this should be changed to return an error, and callers
5371 * re-read from disk on failure (on nondebug bits).
5372 */
5373 static void
5374 arc_hdr_verify_checksum(spa_t *spa, arc_buf_hdr_t *hdr, const blkptr_t *bp)
…
5395 err = zio_checksum_error_impl(spa, bp,
5396 BP_GET_CHECKSUM(bp), abd, psize, 0, NULL);
5397 if (err != 0) {
5398 /*
5399 * Use abd_copy_to_buf() rather than
5400 * abd_borrow_buf_copy() so that we are sure to
5401 * include the buf in crash dumps.
5402 */
5403 void *buf = kmem_alloc(psize, KM_SLEEP);
5404 abd_copy_to_buf(buf, abd, psize);
5405 panic("checksum of cached data doesn't match BP "
5406 "err=%u hdr=%p bp=%p abd=%p buf=%p",
5407 err, (void *)hdr, (void *)bp, (void *)abd, buf);
OpenZFS though doesn’t have this piece of code and generally returns error as written in comment XXX.
Tom, do you think the panic() call should be replaced with returning an error?
—
Vitaliy Gusev
> On 11 Nov 2024, at 21:36, Toomas Soome via illumos-developer <developer@lists.illumos.org> wrote:
>
>
>
>> On 30. Oct 2024, at 20:57, Gary Mills <gary_mills@fastmail.fm> wrote:
>>
>> I'm not sure if this is a bug or just ZFS being careful, but I got a
>> panic and reboot while I was doing a "pkg update". The system
>> has an AMD 6-core CPU with B550 support hardware. The next
>> "pkg update" completed normally, without a panic. Here's what
>> I found in /var/adm/messages. Does it look familiar?
>>
>> Oct 30 09:14:31 b550 unix: [ID 836849 kern.notice]
>> Oct 30 09:14:31 b550 ^Mpanic[cpu4]/thread=fffffe2cc9e88780:
>> Oct 30 09:14:31 b550 genunix: [ID 129249 kern.notice] checksum of cached data doesn't match BP err=50 hdr=fffffe3d478f51c0 bp=fffffe0040433988 abd=fffffe3d478f7cc0 buf=fffffe3b5a6f9000
>> Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433760 zfs:zfs_nfsshare_inited+378b87f0 ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433890 zfs:arc_read+de1 ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe00404338e0 zfs:dbuf_issue_final_prefetch+77 ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433a70 zfs:dbuf_prefetch_impl+502 ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433b20 zfs:dmu_zfetch+2ed ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433bd0 zfs:dmu_buf_hold_array_by_dnode+321 ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433c70 zfs:dmu_read_uio_dnode+54 ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433cc0 zfs:dmu_read_uio_dbuf+51 ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433d60 zfs:zfs_read+19c ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433de0 genunix:fop_read+60 ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f00 genunix:read+2b5 ()
>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f10 unix:brand_sys_syscall+1fe ()
>> Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
>> Oct 30 09:14:31 b550 genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
>> Oct 30 09:14:31 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 0 reset port
>> Oct 30 09:14:32 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 1 reset port
>> Oct 30 09:14:50 b550 genunix: [ID 100000 kern.notice]
>> Oct 30 09:14:50 b550 genunix: [ID 665016 kern.notice] ^M100% done: 859875 pages dumped,
>> Oct 30 09:14:50 b550 genunix: [ID 851671 kern.notice] dump succeeded
>> Oct 30 09:15:34 b550 genunix: [ID 107833 kern.notice] ^MOpenIndiana Hipster 2022.10 Version illumos-806838751b 64-bit
>>
>>
>
> Dan got blown up while running zfs-tests (rsend), and that resulted on me picking one series of updates from OpenZFS concerning dbuf and dmu. There are still few XXX notes for myself, but so far both debug and non-debug builds have been behaving nicely (debug build used to run zfs-tests). I have seen myself also panic from arc (ASSERT fired while running zfs-tests on debuilg build — that was before the work mentioned above). Most likely need to pick some arc bits as well.
>
> The current wip branch is: https://github.com/tsoome/illumos-gate/tree/rsend if you like to test. The problem about those panics is that they seem to be random, or at least not easily repeatable.
>
> rgds,
> toomas
>
> illumos <https://illumos.topicbox.com/latest> / illumos-developer / see discussions <https://illumos.topicbox.com/groups/developer> + participants <https://illumos.topicbox.com/groups/developer/members> + delivery options <https://illumos.topicbox.com/groups/developer/subscription>Permalink <https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-Mb8af562876078fc2e3f7037c>
------------------------------------------
illumos: illumos-developer
Permalink: https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-M5484aceef4b12b181bfff6f0
Delivery options: https://illumos.topicbox.com/groups/developer/subscription
[-- Attachment #2: Type: text/html, Size: 33143 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [developer] Panic during pkg update
2024-11-11 21:11 ` gusev.vitaliy via illumos-developer
@ 2024-11-12 6:18 ` gusev.vitaliy via illumos-developer
0 siblings, 0 replies; 7+ messages in thread
From: gusev.vitaliy via illumos-developer @ 2024-11-12 6:18 UTC (permalink / raw)
To: illumos-developer, Tom Caputi
[-- Attachment #1: Type: text/plain, Size: 7851 bytes --]
> On 12 Nov 2024, at 00:11, gusev.vitaliy via illumos-developer <developer@lists.illumos.org> wrote:
>
> Panic comes from this function:
>
> usr/src/uts/common/fs/zfs/arc.c
>
> 5369 /*
> 5370 * XXX this should be changed to return an error, and callers
> 5371 * re-read from disk on failure (on nondebug bits).
> 5372 */
> 5373 static void
> 5374 arc_hdr_verify_checksum(spa_t *spa, arc_buf_hdr_t *hdr, const blkptr_t *bp)
>
> …
>
> 5395 err = zio_checksum_error_impl(spa, bp,
> 5396 BP_GET_CHECKSUM(bp), abd, psize, 0, NULL);
> 5397 if (err != 0) {
> 5398 /*
> 5399 * Use abd_copy_to_buf() rather than
> 5400 * abd_borrow_buf_copy() so that we are sure to
> 5401 * include the buf in crash dumps.
> 5402 */
> 5403 void *buf = kmem_alloc(psize, KM_SLEEP);
> 5404 abd_copy_to_buf(buf, abd, psize);
> 5405 panic("checksum of cached data doesn't match BP "
> 5406 "err=%u hdr=%p bp=%p abd=%p buf=%p",
> 5407 err, (void *)hdr, (void *)bp, (void *)abd, buf);
>
> OpenZFS though doesn’t have this piece of code and generally returns error as written in comment XXX.
>
> Tom, do you think the panic() call should be replaced with returning an error?
>
I mean Tom Caputi :) Since this code was added in "8727 Native data and metadata encryption for zfs”.
Tom, could you clarify, in which cases zio_checksum_error_impl() can return error here: arc_read -> arc_hdr_verify_checksum -> zio_checksum_error_impl ? Is is so critical to panic?
—
Vitaliy Gusev
> —
> Vitaliy Gusev
>
>> On 11 Nov 2024, at 21:36, Toomas Soome via illumos-developer <developer@lists.illumos.org> wrote:
>>
>>
>>
>>> On 30. Oct 2024, at 20:57, Gary Mills <gary_mills@fastmail.fm> wrote:
>>>
>>> I'm not sure if this is a bug or just ZFS being careful, but I got a
>>> panic and reboot while I was doing a "pkg update". The system
>>> has an AMD 6-core CPU with B550 support hardware. The next
>>> "pkg update" completed normally, without a panic. Here's what
>>> I found in /var/adm/messages. Does it look familiar?
>>>
>>> Oct 30 09:14:31 b550 unix: [ID 836849 kern.notice]
>>> Oct 30 09:14:31 b550 ^Mpanic[cpu4]/thread=fffffe2cc9e88780:
>>> Oct 30 09:14:31 b550 genunix: [ID 129249 kern.notice] checksum of cached data doesn't match BP err=50 hdr=fffffe3d478f51c0 bp=fffffe0040433988 abd=fffffe3d478f7cc0 buf=fffffe3b5a6f9000
>>> Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433760 zfs:zfs_nfsshare_inited+378b87f0 ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433890 zfs:arc_read+de1 ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe00404338e0 zfs:dbuf_issue_final_prefetch+77 ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433a70 zfs:dbuf_prefetch_impl+502 ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433b20 zfs:dmu_zfetch+2ed ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433bd0 zfs:dmu_buf_hold_array_by_dnode+321 ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433c70 zfs:dmu_read_uio_dnode+54 ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433cc0 zfs:dmu_read_uio_dbuf+51 ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433d60 zfs:zfs_read+19c ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433de0 genunix:fop_read+60 ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f00 genunix:read+2b5 ()
>>> Oct 30 09:14:31 b550 genunix: [ID 655072 kern.notice] fffffe0040433f10 unix:brand_sys_syscall+1fe ()
>>> Oct 30 09:14:31 b550 unix: [ID 100000 kern.notice]
>>> Oct 30 09:14:31 b550 genunix: [ID 111219 kern.notice] dumping to /dev/zvol/dsk/rpool/dump, offset 65536, content: kernel
>>> Oct 30 09:14:31 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 0 reset port
>>> Oct 30 09:14:32 b550 ahci: [ID 405573 kern.info] NOTICE: ahci0: ahci_tran_reset_dport port 1 reset port
>>> Oct 30 09:14:50 b550 genunix: [ID 100000 kern.notice]
>>> Oct 30 09:14:50 b550 genunix: [ID 665016 kern.notice] ^M100% done: 859875 pages dumped,
>>> Oct 30 09:14:50 b550 genunix: [ID 851671 kern.notice] dump succeeded
>>> Oct 30 09:15:34 b550 genunix: [ID 107833 kern.notice] ^MOpenIndiana Hipster 2022.10 Version illumos-806838751b 64-bit
>>>
>>>
>>
>> Dan got blown up while running zfs-tests (rsend), and that resulted on me picking one series of updates from OpenZFS concerning dbuf and dmu. There are still few XXX notes for myself, but so far both debug and non-debug builds have been behaving nicely (debug build used to run zfs-tests). I have seen myself also panic from arc (ASSERT fired while running zfs-tests on debuilg build — that was before the work mentioned above). Most likely need to pick some arc bits as well.
>>
>> The current wip branch is: https://github.com/tsoome/illumos-gate/tree/rsend if you like to test. The problem about those panics is that they seem to be random, or at least not easily repeatable.
>>
>> rgds,
>> toomas
>>
>
> illumos <https://illumos.topicbox.com/latest> / illumos-developer / see discussions <https://illumos.topicbox.com/groups/developer> + participants <https://illumos.topicbox.com/groups/developer/members> + delivery options <https://illumos.topicbox.com/groups/developer/subscription>Permalink <https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-M5484aceef4b12b181bfff6f0>
------------------------------------------
illumos: illumos-developer
Permalink: https://illumos.topicbox.com/groups/developer/Te1153c7aaa3e05c7-Mc08c98af25e077799cc2d612
Delivery options: https://illumos.topicbox.com/groups/developer/subscription
[-- Attachment #2: Type: text/html, Size: 34920 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2024-11-12 6:21 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-10-30 18:57 [developer] Panic during pkg update Gary Mills
2024-10-30 19:10 ` Peter Tribble
2024-10-30 20:57 ` Gary Mills
2024-11-11 17:19 ` Gergő Doma
2024-11-11 18:36 ` Toomas Soome via illumos-developer
2024-11-11 21:11 ` gusev.vitaliy via illumos-developer
2024-11-12 6:18 ` gusev.vitaliy via illumos-developer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).