From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from tb-mx1.topicbox.com (localhost.local [127.0.0.1]) by tb-mx1.topicbox.com (Postfix) with ESMTP id E55CBC58400 for ; Tue, 13 Feb 2024 14:32:28 -0500 (EST) (envelope-from rosenfeld@grumpf.hope-2000.org) Received: from tb-mx1.topicbox.com (localhost [127.0.0.1]) by tb-mx1.topicbox.com (Authentication Milter) with ESMTP id F39873892DB; Tue, 13 Feb 2024 14:32:28 -0500 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1707852748; b=oJwkAJ2+RKEe5w3ZE85wXNehOJCgGh5Zcn7CAoxG8AkCj424XG m5SOLeGSAsTGAyrsaUcuIfqDDgBsjBNQtqwwqmjGkbh2okWKdU41mMG7h+xpnks0 9ekU729JJ9weDMXq54l/a06+xw9LlY+1zWaZh/RHjXXcD2/1WLGQ/JGCEdtGJYyY cYxAhP3TZRPgldiTODmrOzami/lZk4doesuS7B66dJSuvldMdVKFuodmXTJZyJLv kn2A2NR2Ar+rASlf7pRi93o1aq2XMeEQOZw2wNsUv+iMV0gpdfEELFw5TpYi/3dx fm4e1EDbuFtNUDyQ0Rh/zCDCVCnzoCSWQ1sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=date:from:to:subject:message-id:references :mime-version:content-type:content-transfer-encoding :in-reply-to; s=arcseal; t=1707852748; bh=rAfSVxK5vE+bn645Kn3HbZ MhWdhF0+HCvrf0eLqhZxw=; b=Vb8Fl9dvGgb4sZXQ45TD5HkG0RJ70xplx+bwW/ X7if2zqObwcs2oE8a6uR7Uk7Ng5osJTEMrbhvwbX++WrMCuyAmyxmJJqzhZFP8tM SS1WULoyxdyIVXHQC0CqjuUqtPH7jIGz3aioRbvz2cPrP1mp294tYvHsC7XXCfBE 8yBkAreBm8ttCSMygyZvXAKhnj0VyY4fpIZ7TPAFbBgi5TmdrkWpdcUlNiBn8YmO SpmaDb11KXjnXmLykhAqcyN+XDQxzKzUw9c5HpzUfvWTDTDyfnzp3nObSyUgynyv Fw5Q8aHkO/6IzEZ5V1KaztfaApfEHAJG0nhCKjh/O61lMqOw== ARC-Authentication-Results: i=1; tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=none (no signatures found); dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=grumpf.hope-2000.org; iprev=pass smtp.remote-ip=78.46.35.119 (smtp.hope-2000.org); spf=pass smtp.mailfrom=rosenfeld@grumpf.hope-2000.org smtp.helo=smtp.hope-2000.org; x-aligned-from=pass (Address match); x-me-sender=none; x-ptr=pass smtp.helo=smtp.hope-2000.org policy.ptr=smtp.hope-2000.org; x-return-mx=pass header.domain=grumpf.hope-2000.org policy.org_domain=hope-2000.org policy.is_org=no (MX Records found: smtp.hope-2000.org); x-return-mx=pass smtp.domain=grumpf.hope-2000.org policy.org_domain=hope-2000.org policy.is_org=no (MX Records found: smtp.hope-2000.org); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 Authentication-Results: tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=none (no signatures found); dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=grumpf.hope-2000.org; iprev=pass smtp.remote-ip=78.46.35.119 (smtp.hope-2000.org); spf=pass smtp.mailfrom=rosenfeld@grumpf.hope-2000.org smtp.helo=smtp.hope-2000.org; x-aligned-from=pass (Address match); x-me-sender=none; x-ptr=pass smtp.helo=smtp.hope-2000.org policy.ptr=smtp.hope-2000.org; x-return-mx=pass header.domain=grumpf.hope-2000.org policy.org_domain=hope-2000.org policy.is_org=no (MX Records found: smtp.hope-2000.org); x-return-mx=pass smtp.domain=grumpf.hope-2000.org policy.org_domain=hope-2000.org policy.is_org=no (MX Records found: smtp.hope-2000.org); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgedvledrudehgdduvdeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuf fkfhggtggugfgjsehtkeertddttddunecuhfhrohhmpefjrghnshcutfhoshgvnhhfvghl ugcuoehrohhsvghnfhgvlhgusehgrhhumhhpfhdrhhhophgvqddvtddttddrohhrgheqne cuggftrfgrthhtvghrnhepfeehvdejvefhffdvkeehtddujeegueegjeejtedtgfetveej tdeuvdfgtdekfeeunecuffhomhgrihhnpehrvghsphhonhguihhnghdrihhnpdhilhhluh hmohhsrdhorhhgnecukfhppeejkedrgeeirdefhedrudduledpvdgrtddumeegfhekmedv vddumehfvgdttdemvddvgeemugejfhhfmehfvgduheemkedujegtnecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehinhgvthepjeekrdegiedrfeehrdduudelpdhhvghl ohepshhmthhprdhhohhpvgdqvddttddtrdhorhhgpdhmrghilhhfrhhomhepoehrohhsvg hnfhgvlhgusehgrhhumhhpfhdrhhhophgvqddvtddttddrohhrgheq X-ME-VSScore: 0 X-ME-VSCategory: clean Received-SPF: pass (grumpf.hope-2000.org: 78.46.35.119 is authorized to use 'rosenfeld@grumpf.hope-2000.org' in 'mfrom' identity (mechanism 'mx' matched)) receiver=tb-mx1.topicbox.com; identity=mailfrom; envelope-from="rosenfeld@grumpf.hope-2000.org"; helo=smtp.hope-2000.org; client-ip=78.46.35.119 Received: from smtp.hope-2000.org (smtp.hope-2000.org [78.46.35.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx1.topicbox.com (Postfix) with ESMTPS for ; Tue, 13 Feb 2024 14:32:28 -0500 (EST) (envelope-from rosenfeld@grumpf.hope-2000.org) Received: from executor.headcrashers.org (executor.headcrashers.org [IPv6:2a01:4f8:221:fe00:224:d7ff:fe15:817c]) (authenticated bits=0) by smtp.hope-2000.org (8.16.1/8.16.1) with ESMTPSA id 41DJWPPt023715 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Tue, 13 Feb 2024 20:32:26 +0100 (CET) Received: from executor.headcrashers.org (executor.local [127.0.0.1]) by executor.headcrashers.org (8.17.2+Sun/8.17.2) with ESMTP id 41DJWPKp026460 for ; Tue, 13 Feb 2024 20:32:25 +0100 (CET) Received: (from woodstoc@localhost) by executor.headcrashers.org (8.17.2+Sun/8.17.2/Submit) id 41DJWP5Z026459 for developer@lists.illumos.org; Tue, 13 Feb 2024 20:32:25 +0100 (CET) Date: Tue, 13 Feb 2024 20:32:25 +0100 From: Hans Rosenfeld To: developer@lists.illumos.org Subject: Re: [developer] panic inside lmrc driver Message-ID: Mail-Followup-To: developer@lists.illumos.org References: <17073109090.B28cc8.93356@composer.illumos.topicbox-beta.com> <17073244220.2fca.655057@composer.illumos.topicbox.com> <17073791720.CaDA97B5a.43725@composer.illumos.topicbox-beta.com> <17073900900.c244b2.25526@composer.illumos.topicbox-beta.com> <17074660100.55FC1.17713@composer.illumos.topicbox-beta.com> <17075110160.6eBE.15095@composer.illumos.topicbox.com> <17077255120.0c8E98De.61832@composer.illumos.topicbox-beta.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <17077255120.0c8E98De.61832@composer.illumos.topicbox-beta.com> Topicbox-Policy-Reasoning: allow: sender is a member Topicbox-Message-UUID: a24769b4-caa6-11ee-ac2f-9298ff880d3f On Mon, Feb 12, 2024 at 03:11:52AM -0500, maurilio.longo via illumos-developer wrote: > Hi Hans, > I'm sorry to report that I got a new problem, I was executing  a zfs recv of a few GBs of data while, at the same time, executing a dd if=/dev/zero of=/... onto my test pool, which right now has 4 disks in two mirror vdevs, when the pool stopped responding. > > In /var/adm/messages this is what I have  > > Feb  9 19:38:54 pg-1 ipmi: [ID 183295 kern.info] SMBIOS type 0x1, addr 0xca2 > Feb  9 19:38:54 pg-1 ipmi: [ID 306142 kern.info] device rev. 3, firmware rev. 2.65, version 2.0 > Feb  9 19:38:54 pg-1 ipmi: [ID 935091 kern.info] number of channels 2 > Feb  9 19:38:54 pg-1 ipmi: [ID 699450 kern.info] watchdog supported > Feb  9 19:38:55 pg-1 scsi: [ID 583861 kern.info] ses0 at lmrc2: target-port w300162b20dde3640 lun 0 > Feb  9 19:38:55 pg-1 genunix: [ID 936769 kern.info] ses0 is /pci@0,0/pci8086,43b8@1c/pci1590,32b@0/iport@p0/enclosure@w300162b20dde3> > Feb  9 19:42:05 pg-1 fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-GH, TYPE: Fault, VER: 1, SEVERITY: Major#012EVENT-TIME: Fri> > Feb  9 19:51:46 pg-1 rootnex: [ID 349649 kern.info] xsvc0 at root: space 0 offset 0 > Feb  9 19:51:46 pg-1 genunix: [ID 936769 kern.info] xsvc0 is /xsvc@0,0 > Feb 10 08:24:08 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 08:24:16 pg-1 lmrc: [ID 998901 kern.warning] WARNING: lmrc0: AEN failed, status = 255 > Feb 10 08:24:16 pg-1 lmrc: [ID 831201 kern.warning] WARNING: lmrc0: PD map sync failed, status = 255 > Feb 10 08:24:17 pg-1 zfs: [ID 961531 kern.warning] WARNING: Pool 'dati' has encountered an uncorrectable I/O failure and has been su> > Feb 10 08:24:22 pg-1 lmrc: [ID 864919 kern.notice] NOTICE: lmrc0: FW is in fault state! > Feb 10 08:24:22 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 08:28:02 pg-1 fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major#012EVENT-TIME: Sat> > Feb 10 08:28:02 pg-1 fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major#012EVENT-TIME: Sat> > Feb 10 08:28:03 pg-1 fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major#012EVENT-TIME: Sat> > Feb 10 08:28:03 pg-1 fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major#012EVENT-TIME: Sat> > Feb 10 08:28:10 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 08:28:18 pg-1 lmrc: [ID 380853 kern.warning] WARNING: lmrc0: LD target map sync failed, status = 255 > Feb 10 08:34:49 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 08:41:27 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 08:48:06 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 08:54:45 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 09:01:24 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 09:08:03 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 09:14:42 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... > Feb 10 09:21:21 pg-1 lmrc: [ID 408335 kern.warning] WARNING: lmrc0: resetting... This looks vaguely similar to what we've seen on DELL H755 controllers, where the controller just drops dead after a while, needing a full power cycle to come back to life. (See https://www.illumos.org/issues/15935) If you just reset the system (not power cycling), does it come up correctly again after this? Hans -- %SYSTEM-F-ANARCHISM, The operating system has been overthrown