From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from tb-mx0.topicbox.com (localhost.local [127.0.0.1]) by tb-mx0.topicbox.com (Postfix) with ESMTP id 385F0214B822 for ; Fri, 9 Aug 2024 12:28:55 -0400 (EDT) (envelope-from peter.tribble@gmail.com) Received: from tb-mx0.topicbox.com (localhost [127.0.0.1]) by tb-mx0.topicbox.com (Authentication Milter) with ESMTP id F589E90C653; Fri, 9 Aug 2024 12:28:55 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1723220935; b=hL5rPaHIcW3aVTsBsvlem1babiAW8B75Zavgb/r4vsUoNVMdRk byO5UOA9DYm7ZqkOyhGoSNef0AugNK2gio/1ESs0rMD4cVUVRS+5pgGaoTpCunfs 7qPzOnIvoje84vQPWQUDO/qcGMHH18Vd8AIl0yws23um5bate7OFakOqVHjQYBdM 7sLAsd+X0yyx4VvQUVZCvIUjWcEBN4OyLvWuSmgRdnRVz6pnQGrVoVW3DFBGVFkY DfDTD+A7SgaS/nHmzM6xPgw4WRsRd8W0hBxFThklGUs//5VUpKGeyEpaxyUGvtg7 O/2NFboE8B2VBVj9mJORXXuSMMEaShkzZ3Ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=mime-version:references:in-reply-to:from:date :message-id:subject:to:content-type; s=arcseal; t=1723220935; bh=ytR4nlo/JuHcSxrZG15wRCgovJqDaBamYSqxOEUNABo=; b=e14yO99EX5x3 pyb/Fp+ExGtNFEpTxP3BfILbVzBhM+SuHcgQUolzlFdk8uMqsJIQgJKRQWPCigvn 6Ri/0FAdlF4sK30zPxfMpmraPbd6yaaTWJELtHVqg76OiCPvmCEspEQnKE1X8knB kMtPQfnoJPbpqdU8lFBeJqH+emVkX9W5eFePZIPZRxr7OaYtOhPbceSFGVZAF4Xi R2ehHLRQwuLru1WHtpVdTpoAdnz+LaNs7AFYqB/1SzziAg1rVU5fYKxTrahTxKvV OUkqPmkqbTfFikGoWHJXPj5RoC00+VqoRUPP69bu6Y/LWcCMftc6FiP8Btfwb0Zz vvYctbK2ug== ARC-Authentication-Results: i=1; tb-mx0.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC Policy is not at enforcement); dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=I/lzrkAF header.a=rsa-sha256 header.s=20230601 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; iprev=pass smtp.remote-ip=209.85.160.52 (mail-oa1-f52.google.com); spf=pass smtp.mailfrom=peter.tribble@gmail.com smtp.helo=mail-oa1-f52.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=Ea9ZCHbG; x-me-sender=none; x-ptr=pass smtp.helo=mail-oa1-f52.google.com policy.ptr=mail-oa1-f52.google.com; x-return-mx=pass header.domain=gmail.com policy.is_org=yes (MX Records found: alt1.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com); x-return-mx=pass smtp.domain=gmail.com policy.is_org=yes (MX Records found: alt1.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-51 state=0 Authentication-Results: tb-mx0.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC Policy is not at enforcement); dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=I/lzrkAF header.a=rsa-sha256 header.s=20230601 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; iprev=pass smtp.remote-ip=209.85.160.52 (mail-oa1-f52.google.com); spf=pass smtp.mailfrom=peter.tribble@gmail.com smtp.helo=mail-oa1-f52.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=Ea9ZCHbG; x-me-sender=none; x-ptr=pass smtp.helo=mail-oa1-f52.google.com policy.ptr=mail-oa1-f52.google.com; x-return-mx=pass header.domain=gmail.com policy.is_org=yes (MX Records found: alt1.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com); x-return-mx=pass smtp.domain=gmail.com policy.is_org=yes (MX Records found: alt1.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-51 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgeeftddrleeggdellecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenogfuuhhsphgvtghtffhomhgrihhnucdlgeelmdenucfjughrpegg fhgjhfffkffuvfgtsegrtderredttdejnecuhfhrohhmpefrvghtvghrucfvrhhisggslh gvuceophgvthgvrhdrthhrihgssghlvgesghhmrghilhdrtghomheqnecuggftrfgrthht vghrnhephefhuddujeekffehvddtleeuffejueeftddvheetudfftdevfeehheefkeetff einecuffhomhgrihhnpehshihsvghvvghnthdrvggtpdhsohhnihgtlhgvrdgtohhmpdhg rggsrhhivghlvggsuhhlfhhonhdrtghomhdpsggrnhgutggrmhhprdgtohhmpdhorhgrtg hlvgdrtghomhdpthhophhitggsohigrdgtohhmpdhpvghtvghrthhrihgssghlvgdrtgho rdhukhdpsghlohhgshhpohhtrdgtohhmnecukfhppedvtdelrdekhedrudeitddrhedvne cuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepvddtledrkeehrddu iedtrdehvddphhgvlhhopehmrghilhdqohgruddqfhehvddrghhoohhglhgvrdgtohhmpd hmrghilhhfrhhomhepoehpvghtvghrrdhtrhhisggslhgvsehgmhgrihhlrdgtohhmqedp nhgspghrtghpthhtohepuddprhgtphhtthhopeeouggvvhgvlhhophgvrheslhhishhtsh drihhllhhumhhoshdrohhrgheq X-ME-VSScore: -51 X-ME-VSCategory: clean Received-SPF: pass (gmail.com ... _spf.google.com: Sender is authorized to use 'peter.tribble@gmail.com' in 'mfrom' identity (mechanism 'include:_netblocks.google.com' matched)) receiver=tb-mx0.topicbox.com; identity=mailfrom; envelope-from="peter.tribble@gmail.com"; helo=mail-oa1-f52.google.com; client-ip=209.85.160.52 Received: from mail-oa1-f52.google.com (mail-oa1-f52.google.com [209.85.160.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx0.topicbox.com (Postfix) with ESMTPS for ; Fri, 9 Aug 2024 12:28:54 -0400 (EDT) (envelope-from peter.tribble@gmail.com) Received: by mail-oa1-f52.google.com with SMTP id 586e51a60fabf-26827ec5235so1179976fac.2 for ; Fri, 09 Aug 2024 09:28:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723220934; x=1723825734; darn=lists.illumos.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=ytR4nlo/JuHcSxrZG15wRCgovJqDaBamYSqxOEUNABo=; b=I/lzrkAFN9a8NTTx30aM20JGDixpgw69ASzS1nHqpfULctHhxqjOLpOOmDzaMxhtTP p24Zg7z4Q8bmDAjaxWMRRJIxKjXOXVWquUxYrgn7rWARXOcqBAjbw9I885DtbAEaxEAV Hz4FC/fND8cUU7R26OAoMRJVlando0GSTMs487MnrgQ13Haw0ghwfF78F8jkUXihIHJ/ 4Zp4xZ+Acmc5S/FO7COyxxWboEsVhrVz6WpaNGWEU/JV1UvVXUzEvNazQLz7D/aytZFq +E4ax19NyfJpFDTrdLtwqWPBgacO0XRU8VOamgNnUAB3oJtx1U4Uce255bwnEluBtnSj oReA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723220934; x=1723825734; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ytR4nlo/JuHcSxrZG15wRCgovJqDaBamYSqxOEUNABo=; b=Ea9ZCHbGzKAyLAf/5KSIqVLngc35dxqXZG7O+DLZv0C3t2hVxcx0EO57I7i7U8CNOe L4oyfPp4OF/maQifMk4AxcqXPIRXjceSFOKuGBl4jj4cgh1rApUe6aRKUv9jH/7KT+4+ lEaTyM0LgGbqDyO6w3peGgG/LPwznAe6Q2JrizgQL3ND267KY2VRrZzG6VHsq+E4zQ5j 21ewz0LminRCxQAwQohuGNuaYkluqEl9PrHlV482zwb9Qqw3dhKAb4DyKG1nMckk4ALw H+Yq+gQ8EjTLASBByXJWvDPr0TZi9dG/JSt0pQCCi/MXA1ZDF0KCr0vCZTuG4ADTlTMz Xt5A== X-Gm-Message-State: AOJu0Yz5xFs/Ixfp8Ws/rQRsDE6V3wPPHbGR6v6EX4oi8wLWKej2J6ul fIOhRV4mKdfT6wCuvARA8uAYFMFd55++bDMdSyRq8axQLnfKRqS3zV1vCL7vZ7SGs1SZhjEoLkK kytMcIiWmXC7zRd2t37DYdLCKWMPy X-Google-Smtp-Source: AGHT+IGy98GsyzvbtUN4+N3Ze7i4VZNvc27IhWn3+zEa1PbMvD9thUq315F6THdRj/LHVWSNiJCBHQ5XT/nKp1QG3eU= X-Received: by 2002:a05:6870:c14f:b0:261:236c:2bc0 with SMTP id 586e51a60fabf-26c62c5db33mr2410665fac.13.1723220933388; Fri, 09 Aug 2024 09:28:53 -0700 (PDT) MIME-Version: 1.0 References: <1321589141.1162.1721656889320@www> <148564749.1666.1723214608129@www> In-Reply-To: <148564749.1666.1723214608129@www> From: Peter Tribble Date: Fri, 9 Aug 2024 17:28:42 +0100 Message-ID: Subject: Re: [developer] fmd core dump To: illumos-developer Content-Type: multipart/alternative; boundary="0000000000009aaf91061f42a344" Topicbox-Policy-Reasoning: allow: sender is a member Topicbox-Message-UUID: 7e35db44-566c-11ef-afa1-9fd0018c7b06 --0000000000009aaf91061f42a344 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Aug 9, 2024 at 3:43=E2=80=AFPM Gabriele Bulfon via illumos-develope= r < developer@lists.illumos.org> wrote: > The problem happened again, but this time the rpool was not yet full. > The pstack output shows again the same problem: > > feed68a5 _lwp_kill (5, 6, 22c4, fef45000, fef45000, c) + 15 > fee68a7b raise (6) + 2b > fee41cde abort () + 10e > 08079939 fmd_panic (8081400) > 0807994b fmd_panic (8081400) + 12 > 08065394 fmd_alloc (50, 1) + 81 > 0806f6a5 fmd_event_create (1, d1da323a, 1bd4e8f, 0) + 18 > 08073ae3 fmd_module_timeout (fb8ef100, 2a1, d1da323a) + 20 > 0807bd21 fmd_timerq_exec (915db80) + 127 > 0807b299 fmd_thread_start (8131030) + 5b > feed1a3b _thrp_setup (fed82a40) + 88 > feed1bd0 _lwp_start (fed82a40, 0, 0, 0, 0, 0) > > I can't believe this global zone is out of virtual memory, it's running > various zones with a lot of processes and they all goes fine. > One thing that occurs to me - how big is the fmd process? As it's 32-bit, it can only grow to 4G before it can't grow any further. > Only fmd here is going panic. > What I found is an old issue I even forgot about: an infolog_hival file i= s > being produced continuously. > Running a tail -f on it I get a continuous output like: > > port_address w500304801d0a8808LH > PhyIdentifier88 %/pci@0,0/pci8086,2f02@1/pci15d9,808@0(( > event_type port_broadcast_sesTPclass > 3resource.sysevent.EC_hba.ESC_sas_hba_port_broadcast version > __ttl0(__todf=E2=96=92'|=E2=96=92,=E2=96=92=E2=96=92,^C > > As I remember, this may go on for some time then it will stop. > > Any idea? > G > > > *Sonicle S.r.l. *: http://www.sonicle.com > *Music: *http://www.gabrielebulfon.com > *eXoplanets : *https://gabrielebulfon.bandcamp.com/album/exoplanets > > > ------------------------------ > > > *Da:* Toomas Soome via illumos-developer > *A:* illumos-developer > *Data:* 22 luglio 2024 16.10.42 CEST > *Oggetto:* Re: [developer] fmd core dump > > > > > On 22. Jul 2024, at 17:01, Gabriele Bulfon via illumos-developer < > developer@lists.illumos.org> wrote: > Hi, I have a couple of systems, installed in 2012 and updated up to > illumos 2019 (will have to update to 2024 later). > They periodically (every 3-4 months, sometimes earlier) create a core dum= p > under /var/fm/fmd. > Looks like fmd core dumped, so no email notice is sent, and we end up > filling the rpool. > I found this link: > https://support.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html > So here I attach the pstack of one of the dumps. > > Any idea? > > > fmd_alloc() does panic when we are out of memory: > > > if (data =3D=3D NULL) > > fmd_panic("insufficient memory (%u bytes needed)\n", > size); > > You can try adding some more swap space perhaps? > > > rgds, > toomas > > Gabriele > > > *Sonicle S.r.l. *: http://www.sonicle.com > *Music: *http://www.gabrielebulfon.com > *eXoplanets : *https://gabrielebulfon.bandcamp.com/album/exoplanets > > > > > *illumos * / illumos-developer / see > discussions + participant= s > + delivery option= s > Permalink > > --=20 -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/ --0000000000009aaf91061f42a344 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Fri, Aug 9, 2024 at 3:43=E2=80=AFPM Ga= briele Bulfon via illumos-developer <developer@lists.illumos.org> wrote:
Th= e problem happened again, but this time the rpool was not yet full.
The = pstack output shows again the same problem:

=C2=A0feed68a5 _lwp_kill= (5, 6, 22c4, fef45000, fef45000, c) + 15
=C2=A0fee68a7b raise =C2=A0 = =C2=A0(6) + 2b
=C2=A0fee41cde abort =C2=A0 =C2=A0() + 10e
=C2=A008079= 939 fmd_panic (8081400)
=C2=A00807994b fmd_panic (8081400) + 12
=C2= =A008065394 fmd_alloc (50, 1) + 81
=C2=A00806f6a5 fmd_event_create (1, d= 1da323a, 1bd4e8f, 0) + 18
=C2=A008073ae3 fmd_module_timeout (fb8ef100, 2= a1, d1da323a) + 20
=C2=A00807bd21 fmd_timerq_exec (915db80) + 127
=C2= =A00807b299 fmd_thread_start (8131030) + 5b
=C2=A0feed1a3b _thrp_setup (= fed82a40) + 88
=C2=A0feed1bd0 _lwp_start (fed82a40, 0, 0, 0, 0, 0)
=
=C2=A0
I can't believe this gl= obal zone is out of virtual memory, it's running various zones with a l= ot of processes and they all goes fine.
One thing that occurs to me - how big is the fmd process? As i= t's 32-bit, it can
only grow to 4G before it can't gr= ow any further.
=C2=A0
Only fmd here is going panic.
Wh= at I found is an old issue I even forgot about: an infolog_hival file is be= ing produced continuously.
Running a tail -f on it I get a continuous ou= tput like:

port_address =C2=A0 =C2=A0 =C2=A0 =C2=A0w500304801d0a8808= LH
PhyIdentifier88 %/pci@0,0/pci8086,2f02@1/pci15d9,808@0((
event_typ= e =C2=A0 =C2=A0 =C2=A0port_broadcast_sesTPclass =C2=A0 =C2=A0 =C2=A0 3resou= rce.sysevent.EC_hba.ESC_sas_hba_port_broadcast =C2=A0version =C2=A0__ttl0(_= _todf=E2=96=92'|=E2=96=92,=E2=96=92=E2=96=92,^C
= =C2=A0
As I remember, this may go on for some time t= hen it will stop.

Any idea?
G
=C2=A0
=C2=A0
Sonicle S.r.l.=C2=A0:=C2=A0http://www.sonicle.com=
=C2=A0
<= /div>
=C2=A0


Da: Too= mas Soome via illumos-developer <developer@lists.illumos.org>
A: = illumos-developer <developer@lists.illumos.org>
Data: 22 luglio 2= 024 16.10.42 CEST
Oggetto: Re: [developer] fmd core dump




On 22. Jul 2024, at 17:01, Gabr= iele Bulfon via illumos-developer <developer@lists.illumos.org> wrote:
Hi, I have a couple of systems, installed in 2012 and updated u= p to illumos 2019 (will have to update to 2024 later).
They periodica= lly (every 3-4 months, sometimes earlier) create a core dump under /var/fm/= fmd.
Looks like fmd core dumped, so no email notice is sent, and we e= nd up filling the rpool.
So here I attach the pstack of one of the dum= ps.
=C2=A0
Any idea?

=C2=A0
fmd_alloc() does panic when we are out of memory:
=C2=A0

=C2=A0 =C2=A0 =C2=A0 =C2= =A0=C2=A0if (data =3D=3D NULL)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 fmd_panic("insufficient memory (%u bytes needed)\n", size);

You can try adding some more swap space perhaps?

<= /div>
=C2=A0
rgds,
toomas

Gabriele
=C2=A0
=C2=A0
Sonicle S.r.l.=C2=A0:=C2=A0http://www.sonicle.com
=C2=A0
<cor= e.fmd.dump.pstack.txt>

<= div id=3D"m_8332374497280099843topicbox-footer" style=3D"margin:10px 0px 0p= x;border-top:1px solid rgba(0,0,0,0.15);border-color:rgba(0,0,0,0.15);paddi= ng:7px 0px"> illumos / illumos-developer / see discussions + participants + delivery=C2=A0options Permalin= k

--
-Peter = Tribble
htt= p://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
--0000000000009aaf91061f42a344--