fmd core dump

fmd core dump

[Gabriele Bulfon]

[-- Attachment #1.1: Type: text/plain, Size: 1388 bytes --]

Hi, I have a system (not really a recent illumos kernel, probably around 2012) that recently caused a couple of core.fmd.xxx dumps, here's what mdb is saying:
 
bash-4.2# mdb core.fmd.1104
Loading modules: [ fmd libumem.so.1 libc.so.1 libnvpair.so.1 libtopo.so.1 libuut il.so.1 libavl.so.1 libsysevent.so.1 eft.so ld.so.1 ]
$C
fd98e988 libc.so.1`_lwp_kill+0x15(4, 6, 120a0, fef58000, fef58000, 4)
fd98e9a8 libc.so.1`raise+0x2b(6, 0, fd98e9c0, feed83e9, 0, 0)
fd98e9f8 libc.so.1`abort+0x10e(3a646d66, 4f424120, 203a5452, 75736e69, 63696666
, 746e6569)
fd98ee18 fmd_panic(8080ec0, fd98ee44, 1, 0)
fd98ee38 fmd_panic+0x12(8080ec0, c, 3e8, ffb3dd87)
fd98ee78 fmd_alloc+0x81(c, 1, 1dca2110, 0, 893c688, 84fd718)
fd98eeb8 fmd_eventq_insert_at_head+0x43(890bb48, 91ec5b8, 0, 92f1ab2d)
fd98eed8 fmd_module_gc+0x66(893c680, 0, 0, fd98eef8)
fd98ef18 fmd_modhash_apply+0x3e(84fd718, 8073d50, 0, 0, 6c275b0e, 30cef3)
fd98ef48 fmd_gc+0x28(80998c0, d, ff19063b, 30ceff, 84f8a48)
fd98efa8 fmd_timerq_exec+0x127(84f8a40, 0, feda22a0, fef58000)
fd98efc8 fmd_thread_start+0x5b(826cfb8, 0, 0, 0)
fd98efe8 libc.so.1`_thrp_setup+0x88(feda2240)
fd98eff8 libc.so.1`_lwp_start(feda2240, 0, 0, 0, 0, 0)
 
Any idea?
 
Gabriele
 
 
Sonicle S.r.l. 
: 
http://www.sonicle.com
Music: 
http://www.gabrielebulfon.com
Quantum Mechanics : 
http://www.cdbaby.com/cd/gabrielebulfon

[-- Attachment #1.2: Type: text/html, Size: 2564 bytes --]

Re: [developer] fmd core dump

[Joerg Schilling]

Gabriele Bulfon <gbulfon@sonicle.com> wrote:

> Hi, I have a system (not really a recent illumos kernel, probably around 2012) that recently caused a couple of core.fmd.xxx dumps, here's what mdb is saying:
>
> bash-4.2# mdb core.fmd.1104
> Loading modules: [ fmd libumem.so.1 libc.so.1 libnvpair.so.1 libtopo.so.1 libuut il.so.1 libavl.so.1 libsysevent.so.1 eft.so ld.so.1 ]
> > $C
> fd98e988 libc.so.1`_lwp_kill+0x15(4, 6, 120a0, fef58000, fef58000, 4)
> fd98e9a8 libc.so.1`raise+0x2b(6, 0, fd98e9c0, feed83e9, 0, 0)
> fd98e9f8 libc.so.1`abort+0x10e(3a646d66, 4f424120, 203a5452, 75736e69, 63696666
> , 746e6569)
> fd98ee18 fmd_panic(8080ec0, fd98ee44, 1, 0)
> fd98ee38 fmd_panic+0x12(8080ec0, c, 3e8, ffb3dd87)
> fd98ee78 fmd_alloc+0x81(c, 1, 1dca2110, 0, 893c688, 84fd718)
> fd98eeb8 fmd_eventq_insert_at_head+0x43(890bb48, 91ec5b8, 0, 92f1ab2d)
> fd98eed8 fmd_module_gc+0x66(893c680, 0, 0, fd98eef8)
> fd98ef18 fmd_modhash_apply+0x3e(84fd718, 8073d50, 0, 0, 6c275b0e, 30cef3)
> fd98ef48 fmd_gc+0x28(80998c0, d, ff19063b, 30ceff, 84f8a48)
> fd98efa8 fmd_timerq_exec+0x127(84f8a40, 0, feda22a0, fef58000)
> fd98efc8 fmd_thread_start+0x5b(826cfb8, 0, 0, 0)
> fd98efe8 libc.so.1`_thrp_setup+0x88(feda2240)
> fd98eff8 libc.so.1`_lwp_start(feda2240, 0, 0, 0, 0, 0)

I had a similar problem half a year ago and it turned out that the reason was a 
bug in "dmake" that caused target specific macro assignements to be done twice 
(when in parallel mode) and as a result, the linker map file was used two times 
on the linker command line.

That caused shared libraries loaded by fmd to become currupted...

Jörg

-- 
 EMail:joerg@schily.net                    (home) Jörg Schilling D-13353 Berlin
    joerg.schilling@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: [developer] fmd core dump

[RomanS]

Isn't simple OOM?

On Fri, Mar 20, 2020 at 5:27 PM Gabriele Bulfon <gbulfon@sonicle.com> wrote:
>
> Hi, I have a system (not really a recent illumos kernel, probably around 2012) that recently caused a couple of core.fmd.xxx dumps, here's what mdb is saying:
>
> bash-4.2# mdb core.fmd.1104
> Loading modules: [ fmd libumem.so.1 libc.so.1 libnvpair.so.1 libtopo.so.1 libuut il.so.1 libavl.so.1 libsysevent.so.1 eft.so ld.so.1 ]
> > $C
> fd98e988 libc.so.1`_lwp_kill+0x15(4, 6, 120a0, fef58000, fef58000, 4)
> fd98e9a8 libc.so.1`raise+0x2b(6, 0, fd98e9c0, feed83e9, 0, 0)
> fd98e9f8 libc.so.1`abort+0x10e(3a646d66, 4f424120, 203a5452, 75736e69, 63696666
> , 746e6569)
> fd98ee18 fmd_panic(8080ec0, fd98ee44, 1, 0)
> fd98ee38 fmd_panic+0x12(8080ec0, c, 3e8, ffb3dd87)
> fd98ee78 fmd_alloc+0x81(c, 1, 1dca2110, 0, 893c688, 84fd718)
> fd98eeb8 fmd_eventq_insert_at_head+0x43(890bb48, 91ec5b8, 0, 92f1ab2d)
> fd98eed8 fmd_module_gc+0x66(893c680, 0, 0, fd98eef8)
> fd98ef18 fmd_modhash_apply+0x3e(84fd718, 8073d50, 0, 0, 6c275b0e, 30cef3)
> fd98ef48 fmd_gc+0x28(80998c0, d, ff19063b, 30ceff, 84f8a48)
> fd98efa8 fmd_timerq_exec+0x127(84f8a40, 0, feda22a0, fef58000)
> fd98efc8 fmd_thread_start+0x5b(826cfb8, 0, 0, 0)
> fd98efe8 libc.so.1`_thrp_setup+0x88(feda2240)
> fd98eff8 libc.so.1`_lwp_start(feda2240, 0, 0, 0, 0, 0)
>
> Any idea?
>
> Gabriele
>
>
>
>
> Sonicle S.r.l. : http://www.sonicle.com
> Music: http://www.gabrielebulfon.com
> Quantum Mechanics : http://www.cdbaby.com/cd/gabrielebulfon
> illumos / illumos-developer / see discussions + participants + delivery options Permalink

Re: [developer] fmd core dump

[Toomas Soome]

[-- Attachment #1: Type: text/plain, Size: 3768 bytes --]

Well, fmd_alloc is taking two arguments, size and flags, so we are trying to allocate 50 bytes there, but failing.

What does pmap -x core tell? Or pmap -S core? It is possible that you are not out of memory, but out of swap (to make swap reservations).

rgds,
toomas

> On 9. Aug 2024, at 17:43, Gabriele Bulfon via illumos-developer <developer@lists.illumos.org> wrote:
> 
> The problem happened again, but this time the rpool was not yet full.
> The pstack output shows again the same problem:
> 
>  feed68a5 _lwp_kill (5, 6, 22c4, fef45000, fef45000, c) + 15
>  fee68a7b raise    (6) + 2b
>  fee41cde abort    () + 10e
>  08079939 fmd_panic (8081400)
>  0807994b fmd_panic (8081400) + 12
>  08065394 fmd_alloc (50, 1) + 81
>  0806f6a5 fmd_event_create (1, d1da323a, 1bd4e8f, 0) + 18
>  08073ae3 fmd_module_timeout (fb8ef100, 2a1, d1da323a) + 20
>  0807bd21 fmd_timerq_exec (915db80) + 127
>  0807b299 fmd_thread_start (8131030) + 5b
>  feed1a3b _thrp_setup (fed82a40) + 88
>  feed1bd0 _lwp_start (fed82a40, 0, 0, 0, 0, 0)
>  
> I can't believe this global zone is out of virtual memory, it's running various zones with a lot of processes and they all goes fine.
> Only fmd here is going panic.
> What I found is an old issue I even forgot about: an infolog_hival file is being produced continuously.
> Running a tail -f on it I get a continuous output like:
> 
> port_address        w500304801d0a8808LH
> PhyIdentifier88 %/pci@0,0/pci8086,2f02@1/pci15d9,808@0((
> event_type      port_broadcast_sesTPclass       3resource.sysevent.EC_hba.ESC_sas_hba_port_broadcast  version  __ttl0(__todf▒'|▒,▒▒,^C
>  
> As I remember, this may go on for some time then it will stop.
> 
> Any idea?
> G
>  
>  
> Sonicle S.r.l. : http://www.sonicle.com <https://www.sonicle.com/>
> Music: http://www.gabrielebulfon.com <http://www.gabrielebulfon.com/>
> eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
>  
>  
> 
> 
> Da: Toomas Soome via illumos-developer <developer@lists.illumos.org <mailto:developer@lists.illumos.org>>
> A: illumos-developer <developer@lists.illumos.org <mailto:developer@lists.illumos.org>>
> Data: 22 luglio 2024 16.10.42 CEST
> Oggetto: Re: [developer] fmd core dump
> 
> 
> 
> 
> On 22. Jul 2024, at 17:01, Gabriele Bulfon via illumos-developer <developer@lists.illumos.org> wrote:
> Hi, I have a couple of systems, installed in 2012 and updated up to illumos 2019 (will have to update to 2024 later).
> They periodically (every 3-4 months, sometimes earlier) create a core dump under /var/fm/fmd.
> Looks like fmd core dumped, so no email notice is sent, and we end up filling the rpool.
> I found  this link: https://support.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html
> So here I attach the pstack of one of the dumps.
>  
> Any idea?
> 
>  
> fmd_alloc() does panic when we are out of memory:
>  
>         if (data == NULL)
>                 fmd_panic("insufficient memory (%u bytes needed)\n", size);
> You can try adding some more swap space perhaps?
>  
> rgds,
> toomas
> 
> Gabriele
>  
>  
> Sonicle S.r.l. : http://www.sonicle.com <https://www.sonicle.com/>
> Music: http://www.gabrielebulfon.com <http://www.gabrielebulfon.com/>
> eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
>  
> <core.fmd.dump.pstack.txt>
> 
> illumos <https://illumos.topicbox.com/latest> / illumos-developer / see discussions <https://illumos.topicbox.com/groups/developer> + participants <https://illumos.topicbox.com/groups/developer/members> + delivery options <https://illumos.topicbox.com/groups/developer/subscription>Permalink <https://illumos.topicbox.com/groups/developer/Tde096911559aa716-M77a6e0329454caf1b3e91297>

[-- Attachment #2: Type: text/html, Size: 15155 bytes --]

Re: [developer] fmd core dump

[Pramod Batni]

[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]

The stack of one of the threads shows a call to
umem_update_thread() implying that libumem
is being used.

I am not sure if ‘fmd’ by default uses libumem.


If not, then were you using libumem to
debug a memory related issue (perhaps  a memory leak or memory corruption?)
The XML manifest file for the fmd service
will have information as to how the
‘fmd’ process is launched — LD_PRELOAD
is set to libumem and UMEM_DEBUG is set before invoking the ‘fmd’
executable in case libumem is being used.


If so, You might want to check the value of UMEM_DEBUG environment variable.

Please keep in mind that using the debug features of libumem has a cost
overhead in terms of memory (virtual address space and physical memory)
used depending on the
value of the UMEM_DEBUG variable.

Given that there are core files of  the ‘fmd’
process in your system, you might want
to check if there are any leaks detected by
mdb’s ::findleaks dcmd.

http://technopark02.blogspot.com/2016/08/solaris-memory-leak-checking-with.html?m=1

You can look at the above website to
get informed about how to use mdb’s
dcmds to get information about
libumem data structures.

Hope this helps,

Pramod





On Mon, 22 Jul 2024 at 19:33, Gabriele Bulfon via illumos-developer <
developer@lists.illumos.org> wrote:

> Hi, I have a couple of systems, installed in 2012 and updated up to
> illumos 2019 (will have to update to 2024 later).
> They periodically (every 3-4 months, sometimes earlier) create a core dump
> under /var/fm/fmd.
> Looks like fmd core dumped, so no email notice is sent, and we end up
> filling the rpool.
> I found  this link:
> https://support.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html
> So here I attach the pstack of one of the dumps.
>
> Any idea?
>
> Gabriele
>
>
> *Sonicle S.r.l. *: http://www.sonicle.com <https://www.sonicle.com/>
> *Music: *http://www.gabrielebulfon.com
> *eXoplanets : *https://gabrielebulfon.bandcamp.com/album/exoplanets
>
> *illumos <https://illumos.topicbox.com/latest>* / illumos-developer / see
> discussions <https://illumos.topicbox.com/groups/developer> + participants
> <https://illumos.topicbox.com/groups/developer/members> + delivery options
> <https://illumos.topicbox.com/groups/developer/subscription> Permalink
> <https://illumos.topicbox.com/groups/developer/Tde096911559aa716-M4ffab0f05ef3ac046ce9bf36>
>

[-- Attachment #2: Type: text/html, Size: 6829 bytes --]

Re: [developer] fmd core dump

[Peter Tribble]

[-- Attachment #1: Type: text/plain, Size: 3662 bytes --]

On Fri, Aug 9, 2024 at 3:43 PM Gabriele Bulfon via illumos-developer <
developer@lists.illumos.org> wrote:

> The problem happened again, but this time the rpool was not yet full.
> The pstack output shows again the same problem:
>
>  feed68a5 _lwp_kill (5, 6, 22c4, fef45000, fef45000, c) + 15
>  fee68a7b raise    (6) + 2b
>  fee41cde abort    () + 10e
>  08079939 fmd_panic (8081400)
>  0807994b fmd_panic (8081400) + 12
>  08065394 fmd_alloc (50, 1) + 81
>  0806f6a5 fmd_event_create (1, d1da323a, 1bd4e8f, 0) + 18
>  08073ae3 fmd_module_timeout (fb8ef100, 2a1, d1da323a) + 20
>  0807bd21 fmd_timerq_exec (915db80) + 127
>  0807b299 fmd_thread_start (8131030) + 5b
>  feed1a3b _thrp_setup (fed82a40) + 88
>  feed1bd0 _lwp_start (fed82a40, 0, 0, 0, 0, 0)
>
> I can't believe this global zone is out of virtual memory, it's running
> various zones with a lot of processes and they all goes fine.
>

One thing that occurs to me - how big is the fmd process? As it's 32-bit,
it can
only grow to 4G before it can't grow any further.


> Only fmd here is going panic.
> What I found is an old issue I even forgot about: an infolog_hival file is
> being produced continuously.
> Running a tail -f on it I get a continuous output like:
>
> port_address        w500304801d0a8808LH
> PhyIdentifier88 %/pci@0,0/pci8086,2f02@1/pci15d9,808@0((
> event_type      port_broadcast_sesTPclass
> 3resource.sysevent.EC_hba.ESC_sas_hba_port_broadcast  version
>  __ttl0(__todf▒'|▒,▒▒,^C
>
> As I remember, this may go on for some time then it will stop.
>
> Any idea?
> G
>
>
> *Sonicle S.r.l. *: http://www.sonicle.com <https://www.sonicle.com/>
> *Music: *http://www.gabrielebulfon.com
> *eXoplanets : *https://gabrielebulfon.bandcamp.com/album/exoplanets
>
>
> ------------------------------
>
>
> *Da:* Toomas Soome via illumos-developer <developer@lists.illumos.org>
> *A:* illumos-developer <developer@lists.illumos.org>
> *Data:* 22 luglio 2024 16.10.42 CEST
> *Oggetto:* Re: [developer] fmd core dump
>
>
>
>
> On 22. Jul 2024, at 17:01, Gabriele Bulfon via illumos-developer <
> developer@lists.illumos.org> wrote:
> Hi, I have a couple of systems, installed in 2012 and updated up to
> illumos 2019 (will have to update to 2024 later).
> They periodically (every 3-4 months, sometimes earlier) create a core dump
> under /var/fm/fmd.
> Looks like fmd core dumped, so no email notice is sent, and we end up
> filling the rpool.
> I found  this link:
> https://support.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html
> So here I attach the pstack of one of the dumps.
>
> Any idea?
>
>
> fmd_alloc() does panic when we are out of memory:
>
>
>         if (data == NULL)
>
>                 fmd_panic("insufficient memory (%u bytes needed)\n",
> size);
>
> You can try adding some more swap space perhaps?
>
>
> rgds,
> toomas
>
> Gabriele
>
>
> *Sonicle S.r.l. *: http://www.sonicle.com <https://www.sonicle.com/>
> *Music: *http://www.gabrielebulfon.com
> *eXoplanets : *https://gabrielebulfon.bandcamp.com/album/exoplanets
>
> <core.fmd.dump.pstack.txt>
>
>
> *illumos <https://illumos.topicbox.com/latest>* / illumos-developer / see
> discussions <https://illumos.topicbox.com/groups/developer> + participants
> <https://illumos.topicbox.com/groups/developer/members> + delivery options
> <https://illumos.topicbox.com/groups/developer/subscription> Permalink
> <https://illumos.topicbox.com/groups/developer/Tde096911559aa716-M77a6e0329454caf1b3e91297>
>


-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

[-- Attachment #2: Type: text/html, Size: 10274 bytes --]

Re: [developer] fmd core dump

[Gabriele Bulfon]

[-- Attachment #1.1: Type: text/plain, Size: 2829 bytes --]

The problem happened again, but this time the rpool was not yet full.
The pstack output shows again the same problem:

 feed68a5 _lwp_kill (5, 6, 22c4, fef45000, fef45000, c) + 15
 fee68a7b raise    (6) + 2b
 fee41cde abort    () + 10e
 08079939 fmd_panic (8081400)
 0807994b fmd_panic (8081400) + 12
 08065394 fmd_alloc (50, 1) + 81
 0806f6a5 fmd_event_create (1, d1da323a, 1bd4e8f, 0) + 18
 08073ae3 fmd_module_timeout (fb8ef100, 2a1, d1da323a) + 20
 0807bd21 fmd_timerq_exec (915db80) + 127
 0807b299 fmd_thread_start (8131030) + 5b
 feed1a3b _thrp_setup (fed82a40) + 88
 feed1bd0 _lwp_start (fed82a40, 0, 0, 0, 0, 0)
 
I can't believe this global zone is out of virtual memory, it's running various zones with a lot of processes and they all goes fine.
Only fmd here is going panic.
What I found is an old issue I even forgot about: an infolog_hival file is being produced continuously.
Running a tail -f on it I get a continuous output like:

port_address        w500304801d0a8808LH
PhyIdentifier88 %/pci@0,0/pci8086,2f02@1/pci15d9,808@0((
event_type      port_broadcast_sesTPclass       3resource.sysevent.EC_hba.ESC_sas_hba_port_broadcast  version  __ttl0(__todf▒'|▒,▒▒,^C
 
As I remember, this may go on for some time then it will stop.

Any idea?
G
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 

 


Da: Toomas Soome via illumos-developer <developer@lists.illumos.org>
A: illumos-developer <developer@lists.illumos.org>
Data: 22 luglio 2024 16.10.42 CEST
Oggetto: Re: [developer] fmd core dump




On 22. Jul 2024, at 17:01, Gabriele Bulfon via illumos-developer <developer@lists.illumos.org> wrote:
Hi, I have a couple of systems, installed in 2012 and updated up to illumos 2019 (will have to update to 2024 later).
They periodically (every 3-4 months, sometimes earlier) create a core dump under /var/fm/fmd.
Looks like fmd core dumped, so no email notice is sent, and we end up filling the rpool.
I found  this link: https://support.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html
So here I attach the pstack of one of the dumps.
 
Any idea?




 
fmd_alloc() does panic when we are out of memory:
 
        if (data == NULL)
                fmd_panic("insufficient memory (%u bytes needed)\n", size);

You can try adding some more swap space perhaps?

 
rgds,
toomas

Gabriele
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 


<core.fmd.dump.pstack.txt>


illumos / illumos-developer / see discussions + participants + delivery options Permalink

[-- Attachment #1.2: Type: text/html, Size: 10249 bytes --]

Re: [developer] fmd core dump

[Gabriele Bulfon]

[-- Attachment #1.1: Type: text/plain, Size: 2557 bytes --]

Here are some outputs.

top:
CPU states: 90.7% idle,  5.7% user,  3.6% kernel,  0.0% iowait,  0.0% swap
Kernel: 17637 ctxsw, 19873 trap, 8926 intr, 216414 syscall, 3 fork, 16781 flt
Memory: 128G phys mem, 16G free mem, 40G total swap, 40G free swap
 
swap -lh:
swapfile             dev    swaplo   blocks     free
/dev/zvol/dsk/rpool/swap 301,2        4K    4.00G    4.00G
/dev/zvol/dsk/rpool/swap2 301,3        4K    4.00G    4.00G
/dev/zvol/dsk/data/swap4 301,4        4K    32.0G    32.0G
 
swap -sh:
total: 30.8G allocated + 10.8G reserved = 41.6G used, 29.4G available
 
prstat -Z:
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE
     4     1489   29G   22G    17%  15:25:38 5.8% cloudserver
     5      185 3319M 2147M   1.6%   4:05:07 2.1% encoserver
     0       54 1036M 1044M   0.8%  15:17:20 0.8% global
     1       71 1271M  636M   0.5%   0:03:24 0.0% mlp
     3      232 7557M 5834M   4.5%   2:48:54 0.0% wp
 
G.
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 

 


Da: Toomas Soome via illumos-developer <developer@lists.illumos.org>
A: illumos-developer <developer@lists.illumos.org>
Data: 22 luglio 2024 16.10.42 CEST
Oggetto: Re: [developer] fmd core dump




On 22. Jul 2024, at 17:01, Gabriele Bulfon via illumos-developer <developer@lists.illumos.org> wrote:
Hi, I have a couple of systems, installed in 2012 and updated up to illumos 2019 (will have to update to 2024 later).
They periodically (every 3-4 months, sometimes earlier) create a core dump under /var/fm/fmd.
Looks like fmd core dumped, so no email notice is sent, and we end up filling the rpool.
I found  this link: https://support.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html
So here I attach the pstack of one of the dumps.
 
Any idea?




 
fmd_alloc() does panic when we are out of memory:
 
        if (data == NULL)
                fmd_panic("insufficient memory (%u bytes needed)\n", size);

You can try adding some more swap space perhaps?

 
rgds,
toomas

Gabriele
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 


<core.fmd.dump.pstack.txt>


illumos / illumos-developer / see discussions + participants + delivery options Permalink

[-- Attachment #1.2: Type: text/html, Size: 11961 bytes --]

Re: [developer] fmd core dump

[Toomas Soome]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]



> On 22. Jul 2024, at 17:01, Gabriele Bulfon via illumos-developer <developer@lists.illumos.org> wrote:
> 
> Hi, I have a couple of systems, installed in 2012 and updated up to illumos 2019 (will have to update to 2024 later).
> They periodically (every 3-4 months, sometimes earlier) create a core dump under /var/fm/fmd.
> Looks like fmd core dumped, so no email notice is sent, and we end up filling the rpool.
> I found  this link: https://support.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html
> So here I attach the pstack of one of the dumps.
>  
> Any idea?
> 

fmd_alloc() does panic when we are out of memory:

        if (data == NULL)
                fmd_panic("insufficient memory (%u bytes needed)\n", size);

You can try adding some more swap space perhaps?

rgds,
toomas

> Gabriele
>  
>  
> Sonicle S.r.l. : http://www.sonicle.com <https://www.sonicle.com/>
> Music: http://www.gabrielebulfon.com <http://www.gabrielebulfon.com/>
> eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
>  
> illumos <https://illumos.topicbox.com/latest> / illumos-developer / see discussions <https://illumos.topicbox.com/groups/developer> + participants <https://illumos.topicbox.com/groups/developer/members> + delivery options <https://illumos.topicbox.com/groups/developer/subscription>Permalink <https://illumos.topicbox.com/groups/developer/Tde096911559aa716-M4ffab0f05ef3ac046ce9bf36><core.fmd.dump.pstack.txt>


[-- Attachment #2: Type: text/html, Size: 6774 bytes --]