From: Gabriele Bulfon <gbulfon@sonicle.com>
To: illumos-developer <developer@lists.illumos.org>
Subject: fmd core dump
Date: Mon, 22 Jul 2024 16:01:29 +0200 (CEST) [thread overview]
Message-ID: <1321589141.1162.1721656889320@www> (raw)
In-Reply-To:
[-- Attachment #1.1: Type: text/plain, Size: 639 bytes --]
Hi, I have a couple of systems, installed in 2012 and updated up to illumos 2019 (will have to update to 2024 later).
They periodically (every 3-4 months, sometimes earlier) create a core dump under /var/fm/fmd.
Looks like fmd core dumped, so no email notice is sent, and we end up filling the rpool.
I found this link: https://support.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html
So here I attach the pstack of one of the dumps.
Any idea?
Gabriele
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
[-- Attachment #1.2: Type: text/html, Size: 2289 bytes --]
[-- Attachment #2: core.fmd.dump.pstack.txt --]
[-- Type: text/plain, Size: 13959 bytes --]
sonicle@xstreamserver2:/var/fm/fmd$ pfexec pstack core.fmd.25573
core 'core.fmd.25573' of 25573: /usr/lib/fm/fmd/fmd
--------------------- thread# 1 / lwp# 1 ---------------------
feed6715 __sigsuspend (8047d78, 4, 8047dc8, 807353d, fef4cc40, 0) + 15
080735a4 main (8047dcc, fef52308) + 39b
0805ffd8 _start_crt (1, 8047e2c, fefd0b60, 0, 0, 0) + 97
0805feaa _start (1, 8047eec, 0, 8047f00, 8047f19, 8047f2a) + 1a
--------------------- thread# 2 / lwp# 2 ---------------------
feed6335 __pollsys (fdadeb50, 1, fdadeae8, 0, fdca22e0, 0) + 15
fee65906 poll (fdadeb50, 1, 2710) + 66
fdc89c51 ses_contract_thread (0) + 103
feed1a3b _thrp_setup (fed80a40) + 88
feed1bd0 _lwp_start (fed80a40, 0, 0, 0, 0, 0)
--------------------- thread# 3 / lwp# 3 ---------------------
feed1c29 __lwp_park (feac9818, feac9838, fd94ef18, feed6da6, fefb0000, fd94eeb8) + 19
feecb958 cond_wait_queue (feac9818, feac9838, fd94ef18) + 6a
feecbc2c cond_wait_common (feac9818, feac9838, fd94ef18) + 27b
feecbe9a __cond_timedwait (feac9818, feac9838, fd94ef9c) + 111
feecbed4 cond_timedwait (feac9818, feac9838, fd94ef9c) + 35
fea84ab3 umem_update_thread (0) + 1f9
feed1a3b _thrp_setup (fed81a40) + 88
feed1bd0 _lwp_start (fed81a40, 0, 0, 0, 0, 0)
--------------------- thread# 4 / lwp# 4 ---------------------
feed71ee __door_return (fd82edb8, 4, 0, 0, 0, feed720b) + 2e
fdbd4d1d event_deliver_service (9017988, fd82edfc, 4, 0, 0) + 18b
feed720b __door_return () + 4b
--------------------- thread# 5 / lwp# 5 ---------------------
feed1c29 __lwp_park (83cee90, 83ceea0, 0, 0, 0, 0) + 19
feecb958 cond_wait_queue (83cee90, 83ceea0, 0) + 6a
feecbfd0 __cond_wait (83cee90, 83ceea0) + 8f
feecc024 cond_wait (83cee90, 83ceea0) + 2e
fdbd49f3 subscriber_event_handler (9017988) + 51
feed1a3b _thrp_setup (fed82a40) + 88
feed1bd0 _lwp_start (fed82a40, 0, 0, 0, 0, 0)
--------------------- thread# 6 / lwp# 6 ---------------------
feed68a5 _lwp_kill (6, 6, 3621, fef45000, fef45000, b) + 15
fee68a7b raise (6) + 2b
fee41cde abort () + 10e
08079939 fmd_panic (8081400)
0807994b fmd_panic (8081400) + 12
08065394 fmd_alloc (50, 1) + 81
0806f6a5 fmd_event_create (1, 328de12f, 1d365c3, 0) + 18
08073ae3 fmd_module_timeout (9a71300, 3f7, 328de12f) + 20
0807bd21 fmd_timerq_exec (9168a00) + 127
0807b299 fmd_thread_start (82a5fb0) + 5b
feed1a3b _thrp_setup (fed83240) + 88
feed1bd0 _lwp_start (fed83240, 0, 0, 0, 0, 0)
--------------------- thread# 7 / lwp# 7 ---------------------
feed1c29 __lwp_park (91694a0, 9169488, 0, fea8a743, 8f48194, 80c9340) + 19
feecb958 cond_wait_queue (91694a0, 9169488, 0) + 6a
feecbfd0 __cond_wait (91694a0, 9169488) + 8f
feecc024 cond_wait (91694a0, 9169488) + 2e
feecc06d pthread_cond_wait (91694a0, 9169488) + 24
0806fcbe fmd_eventq_delete (9169488) + 3f
080753f5 fmd_module_start (8f48180) + 13e
0807b299 fmd_thread_start (90c5198) + 5b
feed1a3b _thrp_setup (fed83a40) + 88
feed1bd0 _lwp_start (fed83a40, 0, 0, 0, 0, 0)
--------------------- thread# 8 / lwp# 8 ---------------------
feed1c29 __lwp_park (9169360, 9169348, 0, fea8a743, 8f487d4, 80c9380) + 19
feecb958 cond_wait_queue (9169360, 9169348, 0) + 6a
feecbfd0 __cond_wait (9169360, 9169348) + 8f
feecc024 cond_wait (9169360, 9169348) + 2e
feecc06d pthread_cond_wait (9169360, 9169348) + 24
0806fcbe fmd_eventq_delete (9169348) + 3f
080753f5 fmd_module_start (8f487c0) + 13e
0807b299 fmd_thread_start (90c5148) + 5b
feed1a3b _thrp_setup (fed84240) + 88
feed1bd0 _lwp_start (fed84240, 0, 0, 0, 0, 0)
--------------------- thread# 9 / lwp# 9 ---------------------
feed71ee __door_return (0, 0, 0, 0, 90c50d0, 0) + 2e
feebd60c door_xcreate_startf (fd40ec80) + 17b
0807b299 fmd_thread_start (90c50d0) + 5b
feed1a3b _thrp_setup (fed84a40) + 88
feed1bd0 _lwp_start (fed84a40, 0, 0, 0, 0, 0)
-------------------- thread# 10 / lwp# 10 --------------------
feed71ee __door_return (fd1fed88, 4, 0, 0, 6874, 0) + 2e
fdbd4d1d event_deliver_service (9168048, fd1fedfc, 4, 0, 0) + 18b
feed720b __door_return () + 4b
-------------------- thread# 11 / lwp# 11 --------------------
feed1c29 __lwp_park (fed85a40, 0, 8f788ac, 0, fd0ffc08, 1) + 19
feecb805 mutex_lock_impl (8f788ac, 0) + 291
feecc11f mutex_lock (8f788ac) + 19
08071d80 fmd_log_append (8f787c0, 9a94590, 0) + 46f
0807e93d fmd_xprt_recv (90ac178, 9902630, fa948298, 1cf5ce0) + 48d
080628d7 fmd_xprt_post (8f487c0, 90ac178, 9902630, fa948298) + c5
0807adf4 sysev_legacy (9ad8008) + 100
fdbd4a1b subscriber_event_handler (9168048) + 79
0807b299 fmd_thread_start (90c5058) + 5b
feed1a3b _thrp_setup (fed85a40) + 88
feed1bd0 _lwp_start (fed85a40, 0, 0, 0, 0, 0)
-------------------- thread# 12 / lwp# 12 --------------------
feed6335 __pollsys (9294d88, 4, 0, 0, 0, 800) + 15
fee65906 poll (9294d88, 4, ffffffff) + 66
feb1c7ef _svc_run_mt () + 2da
feb1cbf6 svc_run () + 48
0807b299 fmd_thread_start (90c5008) + 5b
feed1a3b _thrp_setup (fed86240) + 88
feed1bd0 _lwp_start (fed86240, 0, 0, 0, 0, 0)
-------------------- thread# 13 / lwp# 13 --------------------
feed1c29 __lwp_park (9280ef8, 9280ee0, 0, fea8a743, 8f48054, 80c94c0) + 19
feecb958 cond_wait_queue (9280ef8, 9280ee0, 0) + 6a
feecbfd0 __cond_wait (9280ef8, 9280ee0) + 8f
feecc024 cond_wait (9280ef8, 9280ee0) + 2e
feecc06d pthread_cond_wait (9280ef8, 9280ee0) + 24
0806fcbe fmd_eventq_delete (9280ee0) + 3f
080753f5 fmd_module_start (8f48040) + 13e
0807b299 fmd_thread_start (927dad8) + 5b
feed1a3b _thrp_setup (fed86a40) + 88
feed1bd0 _lwp_start (fed86a40, 0, 0, 0, 0, 0)
-------------------- thread# 14 / lwp# 14 --------------------
feed1c29 __lwp_park (9280e08, 9280df0, 0, fea8a743, 8fe9e54, 80c9500) + 19
feecb958 cond_wait_queue (9280e08, 9280df0, 0) + 6a
feecbfd0 __cond_wait (9280e08, 9280df0) + 8f
feecc024 cond_wait (9280e08, 9280df0) + 2e
feecc06d pthread_cond_wait (9280e08, 9280df0) + 24
0806fcbe fmd_eventq_delete (9280df0) + 3f
080753f5 fmd_module_start (8fe9e40) + 13e
0807b299 fmd_thread_start (927da10) + 5b
feed1a3b _thrp_setup (fed87240) + 88
feed1bd0 _lwp_start (fed87240, 0, 0, 0, 0, 0)
-------------------- thread# 15 / lwp# 15 --------------------
feed1c29 __lwp_park (9280ea8, 9280e90, 0, fea8a743, 8fe9954, 80c9540) + 19
feecb958 cond_wait_queue (9280ea8, 9280e90, 0) + 6a
feecbfd0 __cond_wait (9280ea8, 9280e90) + 8f
feecc024 cond_wait (9280ea8, 9280e90) + 2e
feecc06d pthread_cond_wait (9280ea8, 9280e90) + 24
0806fcbe fmd_eventq_delete (9280e90) + 3f
080753f5 fmd_module_start (8fe9940) + 13e
0807b299 fmd_thread_start (9473010) + 5b
feed1a3b _thrp_setup (fed87a40) + 88
feed1bd0 _lwp_start (fed87a40, 0, 0, 0, 0, 0)
-------------------- thread# 16 / lwp# 16 --------------------
0807b23e fmd_thread_start(), exit value = 0x00000000
** zombie (exited, not detached, not yet joined) **
-------------------- thread# 17 / lwp# 17 --------------------
feed1c29 __lwp_park (928af00, 928aee8, 0, fea8a743, 8fe96d4, 80c95c0) + 19
feecb958 cond_wait_queue (928af00, 928aee8, 0) + 6a
feecbfd0 __cond_wait (928af00, 928aee8) + 8f
feecc024 cond_wait (928af00, 928aee8) + 2e
feecc06d pthread_cond_wait (928af00, 928aee8) + 24
0806fcbe fmd_eventq_delete (928aee8) + 3f
080753f5 fmd_module_start (8fe96c0) + 13e
0807b299 fmd_thread_start (94dff40) + 5b
feed1a3b _thrp_setup (fed88a40) + 88
feed1bd0 _lwp_start (fed88a40, 0, 0, 0, 0, 0)
-------------------- thread# 18 / lwp# 18 --------------------
feed1c29 __lwp_park (9280e58, 9280e40, 0, fea8a743, 909b554, 80c9600) + 19
feecb958 cond_wait_queue (9280e58, 9280e40, 0) + 6a
feecbfd0 __cond_wait (9280e58, 9280e40) + 8f
feecc024 cond_wait (9280e58, 9280e40) + 2e
feecc06d pthread_cond_wait (9280e58, 9280e40) + 24
0806fcbe fmd_eventq_delete (9280e40) + 3f
080753f5 fmd_module_start (909b540) + 13e
0807b299 fmd_thread_start (950cbe0) + 5b
feed1a3b _thrp_setup (fed89a40) + 88
feed1bd0 _lwp_start (fed89a40, 0, 0, 0, 0, 0)
-------------------- thread# 19 / lwp# 19 --------------------
feed1c29 __lwp_park (9280638, 9280620, 0, fea8a743, 909b694, 80c9640) + 19
feecb958 cond_wait_queue (9280638, 9280620, 0) + 6a
feecbfd0 __cond_wait (9280638, 9280620) + 8f
feecc024 cond_wait (9280638, 9280620) + 2e
feecc06d pthread_cond_wait (9280638, 9280620) + 24
0806fcbe fmd_eventq_delete (9280620) + 3f
080753f5 fmd_module_start (909b680) + 13e
0807b299 fmd_thread_start (950cb40) + 5b
feed1a3b _thrp_setup (fed8a240) + 88
feed1bd0 _lwp_start (fed8a240, 0, 0, 0, 0, 0)
-------------------- thread# 20 / lwp# 20 --------------------
feed1c29 __lwp_park (928aeb0, 928ae98, 0, fea8a743, 909b7d4, 80c9680) + 19
feecb958 cond_wait_queue (928aeb0, 928ae98, 0) + 6a
feecbfd0 __cond_wait (928aeb0, 928ae98) + 8f
feecc024 cond_wait (928aeb0, 928ae98) + 2e
feecc06d pthread_cond_wait (928aeb0, 928ae98) + 24
0806fcbe fmd_eventq_delete (928ae98) + 3f
080753f5 fmd_module_start (909b7c0) + 13e
0807b299 fmd_thread_start (950ca78) + 5b
feed1a3b _thrp_setup (fed8aa40) + 88
feed1bd0 _lwp_start (fed8aa40, 0, 0, 0, 0, 0)
-------------------- thread# 22 / lwp# 22 --------------------
feed1c29 __lwp_park (928a8c0, 928a8a8, 0, fea8a743, 9094154, 80c9700) + 19
feecb958 cond_wait_queue (928a8c0, 928a8a8, 0) + 6a
feecbfd0 __cond_wait (928a8c0, 928a8a8) + 8f
feecc024 cond_wait (928a8c0, 928a8a8) + 2e
feecc06d pthread_cond_wait (928a8c0, 928a8a8) + 24
0806fcbe fmd_eventq_delete (928a8a8) + 3f
080753f5 fmd_module_start (9094140) + 13e
0807b299 fmd_thread_start (950ca28) + 5b
feed1a3b _thrp_setup (fed8b240) + 88
feed1bd0 _lwp_start (fed8b240, 0, 0, 0, 0, 0)
-------------------- thread# 23 / lwp# 23 --------------------
feed71ee __door_return (0, 0, 0, 0, 9571760, 0) + 2e
feebd60c door_xcreate_startf (fc40ec40) + 17b
0807b299 fmd_thread_start (9571760) + 5b
feed1a3b _thrp_setup (fed8c240) + 88
feed1bd0 _lwp_start (fed8c240, 0, 0, 0, 0, 0)
-------------------- thread# 24 / lwp# 24 --------------------
feed71ee __door_return (0, 0, 0, 0, 9571738, 0) + 2e
feebd60c door_xcreate_startf (fc40ec40) + 17b
0807b299 fmd_thread_start (9571738) + 5b
feed1a3b _thrp_setup (fed8ca40) + 88
feed1bd0 _lwp_start (fed8ca40, 0, 0, 0, 0, 0)
-------------------- thread# 25 / lwp# 25 --------------------
feed1c29 __lwp_park (9298418, 9298400, 0, fea8a743, 9094294, 80c97c0) + 19
feecb958 cond_wait_queue (9298418, 9298400, 0) + 6a
feecbfd0 __cond_wait (9298418, 9298400) + 8f
feecc024 cond_wait (9298418, 9298400) + 2e
feecc06d pthread_cond_wait (9298418, 9298400) + 24
0806fcbe fmd_eventq_delete (9298400) + 3f
080753f5 fmd_module_start (9094280) + 13e
0807b299 fmd_thread_start (95716e8) + 5b
feed1a3b _thrp_setup (fed8d240) + 88
feed1bd0 _lwp_start (fed8d240, 0, 0, 0, 0, 0)
-------------------- thread# 27 / lwp# 27 --------------------
feed1c29 __lwp_park (9298558, 9298540, 0, fea8a743, 90943d4, 80c9840) + 19
feecb958 cond_wait_queue (9298558, 9298540, 0) + 6a
feecbfd0 __cond_wait (9298558, 9298540) + 8f
feecc024 cond_wait (9298558, 9298540) + 2e
feecc06d pthread_cond_wait (9298558, 9298540) + 24
0806fcbe fmd_eventq_delete (9298540) + 3f
080753f5 fmd_module_start (90943c0) + 13e
0807b299 fmd_thread_start (95716c0) + 5b
feed1a3b _thrp_setup (fed8da40) + 88
feed1bd0 _lwp_start (fed8da40, 0, 0, 0, 0, 0)
-------------------- thread# 28 / lwp# 28 --------------------
feed1c29 __lwp_park (92985f8, 92985e0, 0, fea8a743, 9518814, 80c9880) + 19
feecb958 cond_wait_queue (92985f8, 92985e0, 0) + 6a
feecbfd0 __cond_wait (92985f8, 92985e0) + 8f
feecc024 cond_wait (92985f8, 92985e0) + 2e
feecc06d pthread_cond_wait (92985f8, 92985e0) + 24
0806fcbe fmd_eventq_delete (92985e0) + 3f
080753f5 fmd_module_start (9518800) + 13e
0807b299 fmd_thread_start (9571620) + 5b
feed1a3b _thrp_setup (fed8e240) + 88
feed1bd0 _lwp_start (fed8e240, 0, 0, 0, 0, 0)
-------------------- thread# 29 / lwp# 29 --------------------
feed1c29 __lwp_park (95a00c0, 95a00a8, 0, fea8a743, 9518954, 80c98c0) + 19
feecb958 cond_wait_queue (95a00c0, 95a00a8, 0) + 6a
feecbfd0 __cond_wait (95a00c0, 95a00a8) + 8f
feecc024 cond_wait (95a00c0, 95a00a8) + 2e
feecc06d pthread_cond_wait (95a00c0, 95a00a8) + 24
0806fcbe fmd_eventq_delete (95a00a8) + 3f
080753f5 fmd_module_start (9518940) + 13e
0807b299 fmd_thread_start (95715f8) + 5b
feed1a3b _thrp_setup (fed8ea40) + 88
feed1bd0 _lwp_start (fed8ea40, 0, 0, 0, 0, 0)
-------------------- thread# 30 / lwp# 30 --------------------
feed1c29 __lwp_park (96087b8, 96087a0, 0, fea8a743, 9518a94, 80c9900) + 19
feecb958 cond_wait_queue (96087b8, 96087a0, 0) + 6a
feecbfd0 __cond_wait (96087b8, 96087a0) + 8f
feecc024 cond_wait (96087b8, 96087a0) + 2e
feecc06d pthread_cond_wait (96087b8, 96087a0) + 24
0806fcbe fmd_eventq_delete (96087a0) + 3f
080753f5 fmd_module_start (9518a80) + 13e
0807b299 fmd_thread_start (9609e48) + 5b
feed1a3b _thrp_setup (fed8f240) + 88
feed1bd0 _lwp_start (fed8f240, 0, 0, 0, 0, 0)
-------------------- thread# 32 / lwp# 32 --------------------
feed1c29 __lwp_park (9608218, 9608200, 0, fea8a743, 909b054, 80c9980) + 19
feecb958 cond_wait_queue (9608218, 9608200, 0) + 6a
feecbfd0 __cond_wait (9608218, 9608200) + 8f
feecc024 cond_wait (9608218, 9608200) + 2e
feecc06d pthread_cond_wait (9608218, 9608200) + 24
0806fcbe fmd_eventq_delete (9608200) + 3f
080753f5 fmd_module_start (909b040) + 13e
0807b299 fmd_thread_start (9609da8) + 5b
feed1a3b _thrp_setup (fed8fa40) + 88
feed1bd0 _lwp_start (fed8fa40, 0, 0, 0, 0, 0)
-------------------- thread# 33 / lwp# 33 --------------------
feed71ee __door_return (0, 0, 0, 0, 1, 0) + 2e
08060a31 fmd_door_server (0) + 2e
0807b299 fmd_thread_start (98f1370) + 5b
feed1a3b _thrp_setup (fed88240) + 88
feed1bd0 _lwp_start (fed88240, 0, 0, 0, 0, 0)
next reply other threads:[~2024-07-22 14:01 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-07-22 14:01 Gabriele Bulfon [this message]
2024-07-22 14:10 ` [developer] " Toomas Soome
2024-07-22 14:21 ` Gabriele Bulfon
2024-08-09 14:43 ` Gabriele Bulfon
2024-08-09 16:28 ` Peter Tribble
2024-08-09 16:54 ` Toomas Soome
2024-08-09 16:47 ` Pramod Batni
-- strict thread matches above, loose matches on Subject: below --
2020-03-20 14:27 Gabriele Bulfon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1321589141.1162.1721656889320@www \
--to=gbulfon@sonicle.com \
--cc=developer@lists.illumos.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).