From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from tb-mx1.topicbox.com (localhost.local [127.0.0.1]) by tb-mx1.topicbox.com (Postfix) with ESMTP id 6F60312F28EE for ; Mon, 22 Jul 2024 10:21:46 -0400 (EDT) (envelope-from gbulfon@sonicle.com) Received: from tb-mx1.topicbox.com (localhost [127.0.0.1]) by tb-mx1.topicbox.com (Authentication Milter) with ESMTP id 89542BDD466; Mon, 22 Jul 2024 10:21:46 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1721658106; b=tX+k5QTKqRbolaHOi0hSe8WjRZQSBgkrSaVfV867Cb0LStXrmw IvFhp1b7AZHCSo3lGN7mc2CGTmaJXH+ZojJP6JnqMudBAEJ7mQX7BGSBmzoKpkKq saxL+4ITG5exuyxLTlv4gQU+amyHx0+q2+GQ4ADNWFF35kmpkab9DGJFg3l8akL4 AbPk5AgEfax8XIdEpNuThdX6y+eR2PUQlRvsz058payF1FSZskkXgM8yMW4hGGWX cCx4T4zPiSTSECSOC79pC2gB5fBXDY985ZEa/NpU+86PddIpnZLBrVqjgKC3Iw1D YUCmZ5vufVE5Dpt5XIFhOa786QhQ9hssmLiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=date:from:to:message-id:in-reply-to:references :subject:mime-version:content-type; s=arcseal; t=1721658106; bh= grJadQDBY8D0sX44Tk3TIHHVT3e07EKjZyp9D6Tfgkg=; b=g9yGjYi0IA2jgYWt 59HWbyPkJBsPr69H5MRaCUeLM32B8ldG8dKkCSytNyqUU1536gY0p+EhXs9vMd4P YjC7huz8aFc/3OUjqO4toOxkl3QXUulKfLURaTBw+hmttxvyQgoNJa6/fmDOEQSa 9LAuHgbMX1hgvB1i8whSDaIowv7SnorK/gKMJxZKoscGyLwaPWoXJ7W4Z6Whh+XE PdtHfhqXUaKz0afz5pgRW4p62rLF2Vc7crBk1VVLyJAnU3TTgqZ8bF231RLunnZ+ xljoOgBYpSTxQN1z/kEVK4ovVAqq4YiSY4+X/fWFoDQJXvldZQSQ2vyFjf3oQLO3 R24CAw== ARC-Authentication-Results: i=1; tb-mx1.topicbox.com; arc=none (no signatures found); bimi=none (No BIMI records found); dkim=pass (1024-bit rsa key sha256) header.d=sonicle.com header.i=@sonicle.com header.b=QQxf5Fp2 header.a=rsa-sha256 header.s=dkim x-bits=1024; dmarc=pass policy.published-domain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=sonicle.com; iprev=pass smtp.remote-ip=109.168.117.71 (mail.sonicle.com); spf=pass smtp.mailfrom=gbulfon@sonicle.com smtp.helo=mail.sonicle.com; x-aligned-from=pass (Address match); x-me-sender=none; x-ptr=pass smtp.helo=mail.sonicle.com policy.ptr=mail.sonicle.com; x-return-mx=pass header.domain=sonicle.com policy.is_org=yes (MX Records found: mail.sonicle.com); x-return-mx=pass smtp.domain=sonicle.com policy.is_org=yes (MX Records found: mail.sonicle.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-100 state=0 Authentication-Results: tb-mx1.topicbox.com; arc=none (no signatures found); bimi=none (No BIMI records found); dkim=pass (1024-bit rsa key sha256) header.d=sonicle.com header.i=@sonicle.com header.b=QQxf5Fp2 header.a=rsa-sha256 header.s=dkim x-bits=1024; dmarc=pass policy.published-domain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=sonicle.com; iprev=pass smtp.remote-ip=109.168.117.71 (mail.sonicle.com); spf=pass smtp.mailfrom=gbulfon@sonicle.com smtp.helo=mail.sonicle.com; x-aligned-from=pass (Address match); x-me-sender=none; x-ptr=pass smtp.helo=mail.sonicle.com policy.ptr=mail.sonicle.com; x-return-mx=pass header.domain=sonicle.com policy.is_org=yes (MX Records found: mail.sonicle.com); x-return-mx=pass smtp.domain=sonicle.com policy.is_org=yes (MX Records found: mail.sonicle.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-100 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgeeftddrheejgdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpeffhffvkfgjfhfugggtsehmtdersgdttdejnecuhfhr ohhmpefirggsrhhivghlvgcuuehulhhfohhnuceoghgsuhhlfhhonhesshhonhhitghlvg drtghomheqnecuggftrfgrthhtvghrnhepffduhffgteekheeitdetgffhheehfeelgfeg heejteeuveelveetgfffieejtefhnecuffhomhgrihhnpehsohhnihgtlhgvrdgtohhmpd hgrggsrhhivghlvggsuhhlfhhonhdrtghomhdpsggrnhgutggrmhhprdgtohhmpdhorhgr tghlvgdrtghomhdpthhophhitggsohigrdgtohhmnecukfhppedutdelrdduieekrdduud ejrdejudenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepihhnvghtpedutdel rdduieekrdduudejrdejuddphhgvlhhopehmrghilhdrshhonhhitghlvgdrtghomhdpmh grihhlfhhrohhmpeeoghgsuhhlfhhonhesshhonhhitghlvgdrtghomheqpdhnsggprhgt phhtthhopedupdhrtghpthhtohepoeguvghvvghlohhpvghrsehlihhsthhsrdhilhhluh hmohhsrdhorhhgqe X-ME-VSScore: -100 X-ME-VSCategory: clean Received-SPF: pass (sonicle.com: 109.168.117.71 is authorized to use 'gbulfon@sonicle.com' in 'mfrom' identity (mechanism 'a' matched)) receiver=tb-mx1.topicbox.com; identity=mailfrom; envelope-from="gbulfon@sonicle.com"; helo=mail.sonicle.com; client-ip=109.168.117.71 Received: from mail.sonicle.com (mail.sonicle.com [109.168.117.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx1.topicbox.com (Postfix) with ESMTPS for ; Mon, 22 Jul 2024 10:21:45 -0400 (EDT) (envelope-from gbulfon@sonicle.com) Received: from www (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.sonicle.com (Postfix) with ESMTPS id E4DF35CD893 for ; Mon, 22 Jul 2024 16:21:43 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.sonicle.com E4DF35CD893 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sonicle.com; s=dkim; t=1721658103; bh=grJadQDBY8D0sX44Tk3TIHHVT3e07EKjZyp9D6Tfgkg=; h=Date:From:To:In-Reply-To:References:Subject; b=QQxf5Fp2dPyf2/AG4clnEnxWeVbxhAwNCJhfFgRtj59AhWSymPGjBzeOr+akM4uwd BZjmw9AvvW7ZDSK1ofADzeeYcmfhxewusMgnqbw2vjM2eWI2wHDf7whZgAxy12twYd 8K9EGi2cegN2wedQlUoKb41epsHpw0ken7skAOC0= Received: from www (www [192.168.222.200]) by www with SMTP (SubEthaSMTP 3.1.7) id LYX2S6YC for developer@lists.illumos.org; Mon, 22 Jul 2024 16:21:43 +0200 (CEST) Date: Mon, 22 Jul 2024 16:21:43 +0200 (CEST) From: Gabriele Bulfon To: illumos-developer Message-ID: <1835527942.1250.1721658103859@www> In-Reply-To: References: <1321589141.1162.1721656889320@www> Subject: Re: [developer] fmd core dump MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_1248_1522779632.1721658103859" Forwarded-From: Topicbox-Policy-Reasoning: allow: sender is a member Topicbox-Message-UUID: bd2fe80c-4835-11ef-ba1b-e30c209ff758 ------=_Part_1248_1522779632.1721658103859 Content-Type: multipart/alternative; boundary="----=_Part_1249_660955338.1721658103859" ------=_Part_1249_660955338.1721658103859 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Here are some outputs. top: CPU states: 90.7% idle, =C2=A05.7% user, =C2=A03.6% kernel, =C2=A00.0% iowa= it, =C2=A00.0% swap Kernel: 17637 ctxsw, 19873 trap, 8926 intr, 216414 syscall, 3 fork, 16781 f= lt Memory: 128G phys mem, 16G free mem, 40G total swap, 40G free swap =C2=A0 swap -lh: swapfile =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 dev =C2=A0 =C2=A0swaplo = =C2=A0 blocks =C2=A0 =C2=A0 free /dev/zvol/dsk/rpool/swap 301,2 =C2=A0 =C2=A0 =C2=A0 =C2=A04K =C2=A0 =C2=A04= .00G =C2=A0 =C2=A04.00G /dev/zvol/dsk/rpool/swap2 301,3 =C2=A0 =C2=A0 =C2=A0 =C2=A04K =C2=A0 =C2=A0= 4.00G =C2=A0 =C2=A04.00G /dev/zvol/dsk/data/swap4 301,4 =C2=A0 =C2=A0 =C2=A0 =C2=A04K =C2=A0 =C2=A03= 2.0G =C2=A0 =C2=A032.0G =C2=A0 swap -sh: total: 30.8G allocated + 10.8G reserved =3D 41.6G used, 29.4G available =C2=A0 prstat -Z: ZONEID =C2=A0 =C2=A0NPROC =C2=A0SWAP =C2=A0 RSS MEMORY =C2=A0 =C2=A0 =C2=A0= TIME =C2=A0CPU ZONE =C2=A0 =C2=A0 =C2=A04 =C2=A0 =C2=A0 1489 =C2=A0 29G =C2=A0 22G =C2=A0 =C2= =A017% =C2=A015:25:38 5.8% cloudserver =C2=A0 =C2=A0 =C2=A05 =C2=A0 =C2=A0 =C2=A0185 3319M 2147M =C2=A0 1.6% =C2= =A0 4:05:07 2.1% encoserver =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 54 1036M 1044M =C2=A0 0.8% =C2= =A015:17:20 0.8% global =C2=A0 =C2=A0 =C2=A01 =C2=A0 =C2=A0 =C2=A0 71 1271M =C2=A0636M =C2=A0 0.5% = =C2=A0 0:03:24 0.0% mlp =C2=A0 =C2=A0 =C2=A03 =C2=A0 =C2=A0 =C2=A0232 7557M 5834M =C2=A0 4.5% =C2= =A0 2:48:54 0.0% wp =C2=A0 G. =C2=A0 =C2=A0 Sonicle S.r.l.=C2=A0:=C2=A0http://www.sonicle.com Music:=C2=A0http://www.gabrielebulfon.com eXoplanets=C2=A0:=C2=A0https://gabrielebulfon.bandcamp.com/album/exoplanets =C2=A0 =C2=A0 Da: Toomas Soome via illumos-developer A: illumos-developer Data: 22 luglio 2024 16.10.42 CEST Oggetto: Re: [developer] fmd core dump On 22. Jul 2024, at 17:01, Gabriele Bulfon via illumos-developer wrote: Hi, I have a couple of systems, installed in 2012 and updated up to illumos= 2019 (will have to update to 2024 later). They periodically (every 3-4 months, sometimes earlier) create a core dump = under /var/fm/fmd. Looks like fmd core dumped, so no email notice is sent, and we end up filli= ng the rpool. I found=C2=A0 this link: https://support.oracle.com/knowledge/Sun%20Microsy= stems/1020519_1.html So here I attach the pstack of one of the dumps. =C2=A0 Any idea? =C2=A0 fmd_alloc() does panic when we are out of memory: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0if (data =3D=3D NULL) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 fmd_panic("insuffic= ient memory (%u bytes needed)\n", size); You can try adding some more swap space perhaps? =C2=A0 rgds, toomas Gabriele =C2=A0 =C2=A0 Sonicle S.r.l.=C2=A0:=C2=A0http://www.sonicle.com Music:=C2=A0http://www.gabrielebulfon.com eXoplanets=C2=A0:=C2=A0https://gabrielebulfon.bandcamp.com/album/exoplanets =C2=A0 illumos / illumos-developer / see discussions + participants + delivery=C2= =A0options Permalink ------=_Part_1249_660955338.1721658103859 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Here are some outputs.

top:
<= span style=3D"font-family: 'courier new', courier;">CPU states: 90.7% idle,=  5.7% user,  3.6% kernel,  0.0% iowait,  0.0% swap
Kernel: 17637 c= txsw, 19873 trap, 8926 intr, 216414 syscall, 3 fork, 16781 flt
Memory: 128G phys mem, 1= 6G free mem, 40G total swap, 40G free swap
 
swap -lh:
swapfile           =   dev    swaplo   blocks     free
<= span style=3D"font-family: 'courier new', courier;">/dev/zvol/dsk/rpool/swa= p 301,2        4K    4.00G    4.00G=
/dev/zvol/d= sk/rpool/swap2 301,3        4K    4.00G  = ;  4.00G
 
swap -sh:
total: 30.8G allocated + 10.8= G reserved =3D 41.6G used, 29.4G available
 
prstat -Z:
ZONEID    NPROC &#= 160;SWAP   RSS MEMORY      TIME  CPU ZONE     = 4     1489   29G   22G    17%  15:25:38 = 5.8% cloudserver
     5      185 3319M 2147M   1.6% =   4:05:07 2.1% encoserver
     0       54 1036M 1044= M   0.8%  15:17:20 0.8% global
     1       71 = 1271M  636M   0.5%   0:03:24 0.0% mlp
     3   = 0;  232 7557M 5834M   4.5%   2:48:54 0.0% wp
 
G.
 
 


Da:= Toomas Soome via illumos-developer <developer@lists.illumos.org>
= A: illumos-developer <developer@lists.illumos.org>Data: 22 luglio 2024 16.10.42 CEST
Oggetto: Re: [developer] fmd core dump




On 22. Jul 2024, at 17:01, Gabriele Bulfon via illumos-developer <d= eveloper@lists.illumos.org> wrote:
Hi, I have a couple of systems, installed in 2012 and updated up to il= lumos 2019 (will have to update to 2024 later).
They periodically (every 3-4 months, someti= mes earlier) create a core dump under /var/fm/fmd.
Looks like fmd core dumped, so no email not= ice is sent, and we end up filling the rpool.
I found  this link: https://support.oracle.com/knowledge/Sun%20Microsy= stems/1020519_1.html
So here I attach the pstack of one of the d= umps.
 
Any idea?

 
fmd_alloc() does panic when we are out of memory:
 

        if= (data = =3D=3D NULL)

= 60;               fmd_panic("insufficient memor= y (%u bytes needed)\n", size);

You can try adding some more swap space perhaps?

 
rgds,
toomas

Gabriele
 
<core.fmd.dump.pstack.txt>

------=_Part_1249_660955338.1721658103859-- ------=_Part_1248_1522779632.1721658103859--