From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from tb-mx0.topicbox.com (localhost.local [127.0.0.1]) by tb-mx0.topicbox.com (Postfix) with ESMTP id 8EE6E214B9FF for ; Fri, 9 Aug 2024 12:47:51 -0400 (EDT) (envelope-from pramod.batni@gmail.com) Received: from tb-mx0.topicbox.com (localhost [127.0.0.1]) by tb-mx0.topicbox.com (Authentication Milter) with ESMTP id 9E57CE40F3B; Fri, 9 Aug 2024 12:47:51 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1723222071; b=wTEyN/+EKvAvddtxKLSU8izd++XEphG37z5b8TcthTQrQkKTz3 xJfEzwfPkSq5NmdZk1QEx2s//d9323Vvuan8DZkK30S1ORJEf3lSBO29VWhdaNNa bOuax7HcUwjPf8IHu7+07Lm9Ku/jggv0ZnZ9/hwKnudAqCvR5wfRazlCx6JabfKb du7PGeE9edxejOh60LI8e67SyJNAXH9y0gM+bftrcYkScCuJ0qtjRpibAyCaG8JT n6ItG7NDa5zsIVOtQ9oygEplE8Ge8OvY7EbyNI47gQxY4nFAjoWN8mU5yMua8zNs jMfBkAZcnO5seKr8f6pk1I7h9liPkVttutow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=mime-version:references:in-reply-to:from:date :message-id:subject:to:content-type; s=arcseal; t=1723222071; bh=3kT874goXDQT/CQ6MtaB3IVb57kka1X/KFUJiEMMlWI=; b=SJ8WQ/X0owFT CoiqnSVXM2dLWr8WrmGvSOyJKHyCCHziD9DNEDPa+R1x4E3WIRruGN+ojCtRAAZj HGgJTd2ogqZlo/ohQINiOaqBxc7yT0ZvRKsCZE533sMsVTkUVaU7WdBOzH8wEhg3 WROiFJrnRBurmE6ow78xS0UvoRnYPBUH1faEf/14SqLvR/vzVM5q3WOZJM5cwseq ZZ1MShbm20rQb8HTRMpD+dN9OYr1XT6lqotmbYmKR2LcE1eyJkAcXUzEGw+tvK+E cbAbBBODZzNFf6Zfyqz+0WBnd6mV87B3AiNgyryYV0pdlPpnZvcc+A0596xyr8yi NrVD4uKnRw== ARC-Authentication-Results: i=1; tb-mx0.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC Policy is not at enforcement); dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=EUcOkEiC header.a=rsa-sha256 header.s=20230601 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; iprev=pass smtp.remote-ip=209.85.215.182 (mail-pg1-f182.google.com); spf=pass smtp.mailfrom=pramod.batni@gmail.com smtp.helo=mail-pg1-f182.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=FWJR5f6Q; x-me-sender=none; x-ptr=pass smtp.helo=mail-pg1-f182.google.com policy.ptr=mail-pg1-f182.google.com; x-return-mx=pass header.domain=gmail.com policy.is_org=yes (MX Records found: gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com); x-return-mx=pass smtp.domain=gmail.com policy.is_org=yes (MX Records found: gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-51 state=0 Authentication-Results: tb-mx0.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC Policy is not at enforcement); dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=EUcOkEiC header.a=rsa-sha256 header.s=20230601 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; iprev=pass smtp.remote-ip=209.85.215.182 (mail-pg1-f182.google.com); spf=pass smtp.mailfrom=pramod.batni@gmail.com smtp.helo=mail-pg1-f182.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=FWJR5f6Q; x-me-sender=none; x-ptr=pass smtp.helo=mail-pg1-f182.google.com policy.ptr=mail-pg1-f182.google.com; x-return-mx=pass header.domain=gmail.com policy.is_org=yes (MX Records found: gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com); x-return-mx=pass smtp.domain=gmail.com policy.is_org=yes (MX Records found: gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=-51 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgeeftddrleeggddutdefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnh htshculddquddttddmnegoufhushhpvggtthffohhmrghinhculdegledmnecujfgurhep gghfjgfhfffkuffvtgesrgdtreertddtjeenucfhrhhomheprfhrrghmohguuceurghtnh hiuceophhrrghmohgurdgsrghtnhhisehgmhgrihhlrdgtohhmqeenucggtffrrghtthgv rhhnpeejtdffkeehtdffieeujedvteefvddtueehheejleekfeelvdfhvdefueegvdehue enucffohhmrghinhepsghlohhgshhpohhtrdgtohhmpdhorhgrtghlvgdrtghomhdpshho nhhitghlvgdrtghomhdpghgrsghrihgvlhgvsghulhhfohhnrdgtohhmpdgsrghnuggtrg hmphdrtghomhdpthhophhitggsohigrdgtohhmnecukfhppedvtdelrdekhedrvdduhedr udekvdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepihhnvghtpedvtdelrd ekhedrvdduhedrudekvddphhgvlhhopehmrghilhdqphhguddqfhdukedvrdhgohhoghhl vgdrtghomhdpmhgrihhlfhhrohhmpeeophhrrghmohgurdgsrghtnhhisehgmhgrihhlrd gtohhmqedpnhgspghrtghpthhtohepuddprhgtphhtthhopeeouggvvhgvlhhophgvrhes lhhishhtshdrihhllhhumhhoshdrohhrgheq X-ME-VSScore: -51 X-ME-VSCategory: clean Received-SPF: pass (gmail.com ... _spf.google.com: Sender is authorized to use 'pramod.batni@gmail.com' in 'mfrom' identity (mechanism 'include:_netblocks.google.com' matched)) receiver=tb-mx0.topicbox.com; identity=mailfrom; envelope-from="pramod.batni@gmail.com"; helo=mail-pg1-f182.google.com; client-ip=209.85.215.182 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx0.topicbox.com (Postfix) with ESMTPS for ; Fri, 9 Aug 2024 12:47:51 -0400 (EDT) (envelope-from pramod.batni@gmail.com) Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-7a264a24ea7so1715233a12.3 for ; Fri, 09 Aug 2024 09:47:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723222070; x=1723826870; darn=lists.illumos.org; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=3kT874goXDQT/CQ6MtaB3IVb57kka1X/KFUJiEMMlWI=; b=EUcOkEiCuhxTD+h847x8HUtvR8LfFPsOGQxqL2o3L2T8QqZQBnnXM1oCoDvdXGzowY hp1xjz+5Xpeo5ST658c7smC+FuCMEfAntBW1TrsuoXuc4E5kZqoFM8hYGRD+J4+HLgP4 XV4yzAlmzM42eamzMe85e+BoKA67zhqu1LkB31RvxmDTw6oWS6pyGymiJkmnEa8Iu6gU 6EoJttjA9FlvgrEZy/wCMl/T5Bgy0KewQKGsm1whL+EN9rRyO4FXvFf+0goCB6sin4R3 7MtVJ1tJmD4rMewL0ELJXQZs56ieNYRd1VaIcFHdoFy0In1i73XEXQc2QI7RoLOolFRB w0cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723222070; x=1723826870; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3kT874goXDQT/CQ6MtaB3IVb57kka1X/KFUJiEMMlWI=; b=FWJR5f6QJCi7nfuaxlY8E43X8nZ1KUiyrygs6uSNwtITUVAqV+popInhSk+eAnfXcu cf1RaDqDokQzpDd8aS9LswyKY2n08zrsXek5sQ5+TaO5B7L//jNofqwR5BqWAnv8yf3h FkCbrFHah0JC7NIQ3GkRgh8VmAHAGo6ZRyGGVvLMwvNFTDSmpHl4D1C8n+dQ7cQ+KjKh +sge5cDGBVKuCTGom2L7Oos2WgEnr0JMNSmrMJC58v6/6S/cVF7xR7a3wLRqVjQ4tsJh mO+x4gsZg9rzhdWHyoJsTopvcEX36FUFtjSWZWT7htDeus0SdS5ZfLd3Qg+B0jNgQxoA XCmQ== X-Gm-Message-State: AOJu0YzrWsxSoKizOVImCTauWvQe3b+aIwpN7+cFIa5RtToaGE1Oxdqr AnjzDV/ukH2EIynewfjGhc6U1e9rNx3mIDMrRu/bLadneFsxu9qx4XUf/3EsPUfKCggAf1lASn5 Yxx7uotZb7SJgqGmaE/0LCyEQK3f9bQ== X-Google-Smtp-Source: AGHT+IElG7yAb4do5y1wANzRjh4b5d78hDcV+EC0GaI0hnADXyVTwSSh2pk7uydsvXsSaCOhADdw7f1sONmMqLHEG+E= X-Received: by 2002:a17:902:dac2:b0:1ff:56b8:2c02 with SMTP id d9443c01a7336-200ae543bc8mr21568255ad.29.1723222069277; Fri, 09 Aug 2024 09:47:49 -0700 (PDT) MIME-Version: 1.0 References: <1321589141.1162.1721656889320@www> In-Reply-To: <1321589141.1162.1721656889320@www> From: Pramod Batni Date: Fri, 9 Aug 2024 22:17:38 +0530 Message-ID: Subject: Re: [developer] fmd core dump To: illumos-developer Content-Type: multipart/alternative; boundary="0000000000004efe72061f42e7e2" Topicbox-Policy-Reasoning: allow: sender is a member Topicbox-Message-UUID: 21d4d74e-566f-11ef-b73c-2e53038c7b06 --0000000000004efe72061f42e7e2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable The stack of one of the threads shows a call to umem_update_thread() implying that libumem is being used. I am not sure if =E2=80=98fmd=E2=80=99 by default uses libumem. If not, then were you using libumem to debug a memory related issue (perhaps a memory leak or memory corruption?) The XML manifest file for the fmd service will have information as to how the =E2=80=98fmd=E2=80=99 process is launched =E2=80=94 LD_PRELOAD is set to libumem and UMEM_DEBUG is set before invoking the =E2=80=98fmd=E2= =80=99 executable in case libumem is being used. If so, You might want to check the value of UMEM_DEBUG environment variable= . Please keep in mind that using the debug features of libumem has a cost overhead in terms of memory (virtual address space and physical memory) used depending on the value of the UMEM_DEBUG variable. Given that there are core files of the =E2=80=98fmd=E2=80=99 process in your system, you might want to check if there are any leaks detected by mdb=E2=80=99s ::findleaks dcmd. http://technopark02.blogspot.com/2016/08/solaris-memory-leak-checking-with.= html?m=3D1 You can look at the above website to get informed about how to use mdb=E2=80=99s dcmds to get information about libumem data structures. Hope this helps, Pramod On Mon, 22 Jul 2024 at 19:33, Gabriele Bulfon via illumos-developer < developer@lists.illumos.org> wrote: > Hi, I have a couple of systems, installed in 2012 and updated up to > illumos 2019 (will have to update to 2024 later). > They periodically (every 3-4 months, sometimes earlier) create a core dum= p > under /var/fm/fmd. > Looks like fmd core dumped, so no email notice is sent, and we end up > filling the rpool. > I found this link: > https://support.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html > So here I attach the pstack of one of the dumps. > > Any idea? > > Gabriele > > > *Sonicle S.r.l. *: http://www.sonicle.com > *Music: *http://www.gabrielebulfon.com > *eXoplanets : *https://gabrielebulfon.bandcamp.com/album/exoplanets > > *illumos * / illumos-developer / see > discussions + participant= s > + delivery option= s > Permalink > > --0000000000004efe72061f42e7e2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
The stack of one of the threads shows a call to
umem_update_thread() implying that libumem
is being used.
=C2=A0
I = am not sure if =E2=80=98fmd=E2=80=99 by default uses libumem.


If not, t= hen were you using libumem to=C2=A0
debug a memory r= elated issue (perhaps =C2=A0a memory leak or memory corruption?)
The XML manifest file for the fmd service=C2=A0
will have information as to how the=C2=A0
=E2=80=98fmd= =E2=80=99 process is launched =E2=80=94 LD_PRELOAD
is set to lib= umem and UMEM_DEBUG is set before invoking the =E2=80=98fmd=E2=80=99 execut= able in case libumem is being used.=C2=A0


If so, You might want to check the value = of UMEM_DEBUG environment variable.
=C2=A0
Please keep in mind that using the debug features of libumem = has a cost overhead in terms of memory (virtual address space and physical = memory) used depending on the
value of the UMEM_DEBU= G variable.=C2=A0

Given = that there are core files of =C2=A0the =E2=80=98fmd=E2=80=99
process in your system, you might want
to = check if there are any leaks detected by
mdb=E2=80= =99s ::findleaks dcmd.=C2=A0


=
You can look at the above website to=C2=A0
get informed about how to use mdb=E2=80=99s=C2=A0
dcmds to get information about=C2=A0
libumem data structu= res.=C2=A0

Hope this hel= ps,

Pramod





On Mon, 22 Jul 2024 at = 19:33, Gabriele Bulfon via illumos-developer <developer@lists.illumos.org> wrote:
Hi, I have a couple of systems, installed in 2012 and = updated up to illumos 2019 (will have to update to 2024 later).
They periodically (every 3-4 months, sometimes earlier) create = a core dump under /var/fm/fmd.
Looks like fmd core d= umped, so no email notice is sent, and we end up filling the rpool.
I found=C2=A0 this link: https://su= pport.oracle.com/knowledge/Sun%20Microsystems/1020519_1.html
So here I attach the pstack of one of the dumps.
=C2=A0
Any idea?

Gabriele
=C2= =A0
=C2=A0
Sonicle S.r.l.=C2=A0:=C2=A0http://w= ww.sonicle.com
--0000000000004efe72061f42e7e2--