From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HTML_MESSAGE,MAILING_LIST_MULTI,MIME_HTML_ONLY,MIME_HTML_ONLY_MULTI, MPART_ALT_DIFF,RCVD_IN_DNSWL_NONE autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 29394 invoked from network); 4 Apr 2023 18:04:19 -0000 Received: from tb-ob1.topicbox.com (64.147.108.173) by inbox.vuxu.org with ESMTPUTF8; 4 Apr 2023 18:04:19 -0000 Received: from tb-mx1.topicbox.com (tb-mx1.nyi.icgroup.com [10.90.30.61]) by tb-ob1.topicbox.com (Postfix) with ESMTP id 43AD02A537 for ; Tue, 4 Apr 2023 14:04:18 -0400 (EDT) (envelope-from bounce.mM04d378e70813bc1e9f21b260.r522be890-2105-11eb-b15e-8d699134e1fa@9fans.bounce.topicbox.com) Received: by tb-mx1.topicbox.com (Postfix, from userid 1132) id 616F217A8F0E; Tue, 4 Apr 2023 14:04:18 -0400 (EDT) ARC-Authentication-Results: i=2; topicbox.com; arc=pass; dkim=pass (2048-bit rsa key sha256) header.d=quintile.net header.i=@quintile.net header.b=Z5f5C8JW header.a=rsa-sha256 header.s=mythic-beasts-k1 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=quintile.net; spf=pass smtp.mailfrom=steve@quintile.net smtp.helo=mx2.mythic-beasts.com; x-internal-arc=fail (as.1.topicbox.com=pass, ams.1.topicbox.com=fail (body has been altered)) (Message modified while forwarding at Topicbox) ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=content-type:content-transfer-encoding:from :mime-version:subject:date:message-id:references:in-reply-to:to :list-help:list-id:list-post:list-subscribe:reply-to :list-unsubscribe; s=sysmsg-1; t=1680631458; bh=suw4eFK6flk6ZOq6 8p1a26uyaXlcRJDrJV5cQArU4IU=; b=grFerB5scqX4hZzTKY2UVVRZNjplEXdX X5SB9dyvx8JdMjbG7CR+UBquM+THSphvasgSySKRQ/H6TqLrRkw/cx6MlwwTIvTm 9vI2I60SAg2TTlHITwepSiaQOjeBBf+VZ0iwrgNmH5t2FWRmI0EY970Od9RB/MLU So+u9/6ZhsY= ARC-Seal: i=2; a=rsa-sha256; cv=pass; d=topicbox.com; s=sysmsg-1; t= 1680631458; b=T/m+k9d8kMiTZawJ2uqrHgqG7CztrXIC4+tR1xe2TPK9P/RknZ nZyRhaOK+0p4okbY58QIm8UElXswWr9ZDLoaFv75+3lWLy3uKsgijBDkfN5IDZkN Yt2P8jU1+ocheZdnpGrvOAV8LeKspS57ambMsQv2NUMQjHcJtpT9d6uds= Authentication-Results: topicbox.com; arc=pass; dkim=pass (2048-bit rsa key sha256) header.d=quintile.net header.i=@quintile.net header.b=Z5f5C8JW header.a=rsa-sha256 header.s=mythic-beasts-k1 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=quintile.net; spf=pass smtp.mailfrom=steve@quintile.net smtp.helo=mx2.mythic-beasts.com; x-internal-arc=fail (as.1.topicbox.com=pass, ams.1.topicbox.com=fail (body has been altered)) (Message modified while forwarding at Topicbox) X-Received-Authentication-Results: tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=pass (2048-bit rsa key sha256) header.d=quintile.net header.i=@quintile.net header.b=Z5f5C8JW header.a=rsa-sha256 header.s=mythic-beasts-k1 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=quintile.net; iprev=pass smtp.remote-ip=46.235.227.24 (mx2.mythic-beasts.com); spf=pass smtp.mailfrom=steve@quintile.net smtp.helo=mx2.mythic-beasts.com; x-aligned-from=pass (Address match); x-me-sender=none; x-ptr=pass smtp.helo=mx2.mythic-beasts.com policy.ptr=mx2.mythic-beasts.com; x-return-mx=pass header.domain=quintile.net policy.is_org=yes (MX Records found: mx.mythic-beasts.com); x-return-mx=pass smtp.domain=quintile.net policy.is_org=yes (MX Records found: mx.mythic-beasts.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=51 state=0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=9fans.net; h= content-type:content-transfer-encoding:from:mime-version:subject :date:message-id:references:in-reply-to:to:list-help:list-id :list-post:list-subscribe:reply-to:list-unsubscribe; s=dkim-1; t=1680631458; x=1680717858; bh=eTBWnpV64aYGxjvo7I/iEgXgnLaieGHC KRNFqZSyrNI=; b=f2QSfpr0accXFpY3dCOwTxYJ5gncBiwXviudU+TfhUGEIMqY HRWIRsnLdpWh99IN7fSWqhYqA12qLmneUUnqc/pzMu65v1fqLPx5sHvgedl6US4+ FalRjx6Yym2CbRoZDuVay+ddD5VXqlFeAVKxzTqMSRKsqQ/rUyl6OxcvUjQ= Received: from tb-mx1.topicbox.com (localhost.local [127.0.0.1]) by tb-mx1.topicbox.com (Postfix) with ESMTP id A5B6817A8A98 for <9fans@9fans.net>; Tue, 4 Apr 2023 14:03:59 -0400 (EDT) (envelope-from steve@quintile.net) Received: from tb-mx1.topicbox.com (localhost [127.0.0.1]) by tb-mx1.topicbox.com (Authentication Milter) with ESMTP id 636F4743B18; Tue, 4 Apr 2023 14:03:59 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1680631439; b=DUu5B2Mfh0E8iDZQ9UqleQQONtEfxYbaxoxMHq72i02dxpKnO/ eT9atVMbDJSDlubD2CQJy7zELMIRtWAz+0shDjfI/dlSSr6xcdFRjMvN/oL7qTjE 5zBx2KICWz9zEetfkGitFDtCl2juOKFnb4rFsYeIUNhpGad3fWvEGnTKpyane0rk fHBZAA0A87yMzYLb61na4b710zXkYTTI5Xvs2x9Ur618Ru2rX+8tnj045QTr3iAU eeJ0Ozat3zeIHDUd0Uk60CtROAP/1HXMz8R8GEGOJRqOgmcOXEfzIUgqRPhcjfKC y1tcaE3wlh004QtEE/475RjyRpPhDLyXHzrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=content-type:content-transfer-encoding:from :mime-version:subject:date:message-id:references:in-reply-to:to; s=arcseal; t=1680631439; bh=KMiEFOUC/gORhERYzo7LErxSXa6j0GKhnC/ fiOanfoE=; b=cDZJR78V4VeW0ZKSxs9T9Pz39dcKGEt+vnbFut/AXyUD/k+cjb1 pg8BEgd6WZFYI2dL4z5u0SjUTbI3vGMFReQmf01jOF6JJGrLB6yfrMMy6MqZGcES NH/gkFXagb9QJa0V1fTsCXtI06Hf2+wNTrKjayqrq05SHRo7Nlz5UvhQp5ZOFH1A kkhy8rwnfUnoYpUNe6y9tLO2BobiNdNPmW8x1fT0Abn0PBMWT6XFdp0GYveo9bTF HPeof08kn/Dg+tE0E0EfRZa2B6+qtBjnKaAYPzRGz98NOhKoXUlO4PtK2abJ43ma igTDx6gO6xTPrlipCrJE6zaBL9zGsyo/raQ== ARC-Authentication-Results: i=1; tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC did not pass); dkim=pass (2048-bit rsa key sha256) header.d=quintile.net header.i=@quintile.net header.b=Z5f5C8JW header.a=rsa-sha256 header.s=mythic-beasts-k1 x-bits=2048; dmarc=none policy.published-domain-policy=none policy.applied-disposition=none policy.evaluated-disposition=none (p=none,d=none,d.eval=none) policy.policy-from=p header.from=quintile.net; iprev=pass smtp.remote-ip=46.235.227.24 (mx2.mythic-beasts.com); spf=pass smtp.mailfrom=steve@quintile.net smtp.helo=mx2.mythic-beasts.com; x-aligned-from=pass (Address match); x-me-sender=none; x-ptr=pass smtp.helo=mx2.mythic-beasts.com policy.ptr=mx2.mythic-beasts.com; x-return-mx=pass header.domain=quintile.net policy.is_org=yes (MX Records found: mx.mythic-beasts.com); x-return-mx=pass smtp.domain=quintile.net policy.is_org=yes (MX Records found: mx.mythic-beasts.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=51 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeiledguddvtdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecuqfhnlhihuchonhgvuc hprghrthculdehuddmnecujfgurheptgfghfggufffkfhfjgfvofesrgejmherhhdtjeen ucfhrhhomhepufhtvghvvgcuufhimhhonhcuoehsthgvvhgvsehquhhinhhtihhlvgdrnh gvtheqnecuggftrfgrthhtvghrnhepgeekkeeiieefgfdvveeujeevuedvjeeivedugfeu udfhjeehvedtjedvleegudevnecuffhomhgrihhnpehpihigvghlhhgvrhhordguvghvpd hsrhdrhhhtpdhtohhpihgtsghogidrtghomhenucfkphepgeeirddvfeehrddvvdejrddv geenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepihhnvghtpeegiedrvdefhe drvddvjedrvdegpdhhvghlohepmhigvddrmhihthhhihgtqdgsvggrshhtshdrtghomhdp mhgrihhlfhhrohhmpeeoshhtvghvvgesqhhuihhnthhilhgvrdhnvghtqe X-ME-VSScore: 51 X-ME-VSCategory: clean Received-SPF: pass (quintile.net: Sender is authorized to use 'steve@quintile.net' in 'mfrom' identity (mechanism 'include:_spf.mythic-beasts.com' matched)) receiver=tb-mx1.topicbox.com; identity=mailfrom; envelope-from="steve@quintile.net"; helo=mx2.mythic-beasts.com; client-ip=46.235.227.24 Received: from mx2.mythic-beasts.com (mx2.mythic-beasts.com [46.235.227.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx1.topicbox.com (Postfix) with ESMTPS for <9fans@9fans.net>; Tue, 4 Apr 2023 14:03:58 -0400 (EDT) (envelope-from steve@quintile.net) Received: by mailhub-hex-d.mythic-beasts.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pjl0f-00C3A4-Ew for 9fans@9fans.net; Tue, 04 Apr 2023 19:03:57 +0100 Content-Type: multipart/alternative; boundary=Apple-Mail-271A77DE-7338-4E68-86AB-4F396DEE34D0 Content-Transfer-Encoding: 7bit From: Steve Simon Mime-Version: 1.0 (1.0) Subject: Re: [9fans] [PATCH] fossil: fix a deadlock in the caching logic Date: Tue, 4 Apr 2023 19:03:44 +0100 Message-Id: References: <16806285560.EA879D18D.575584@composer.9fans.topicbox.com> In-Reply-To: <16806285560.EA879D18D.575584@composer.9fans.topicbox.com> To: 9fans <9fans@9fans.net> X-Mailer: iPhone Mail (20D67) X-BlackCat-Spam-Score: 26 Topicbox-Policy-Reasoning: allow: sender is a member Topicbox-Message-UUID: 1a2a6b66-d313-11ed-a146-e2ec8c120819 Archived-At: =?UTF-8?B?PGh0dHBzOi8vOWZhbnMudG9waWNib3guY29tL2dyb3Vwcy85?= =?UTF-8?B?ZmFucy9UMzU0ZmU3MDJlMWU5ZDVlOS1NMDRkMzc4ZTcwODEzYmMxZTlmMjFi?= =?UTF-8?B?MjYwPg==?= List-Help: List-Id: "9fans" <9fans.9fans.net> List-Post: List-Software: Topicbox v0 List-Subscribe: Precedence: list Reply-To: 9fans <9fans@9fans.net> List-Unsubscribe: , Topicbox-Delivery-ID: 2:9fans:437d30aa-c441-11e9-8a57-d036212d11b0:522be890-2105-11eb-b15e-8d699134e1fa:M04d378e70813bc1e9f21b260:1:USb1_qAw2U5CSqkOfs5TXuwLcvOno9GeFuxNdky8_ok --Apple-Mail-271A77DE-7338-4E68-86AB-4F396DEE34D0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

was this hard to reproduce?

i have not seen fossil dead= locking and have used it since i installed my first home server in 2004.

there definitely _was_ a p= roblem in the snapshot code which was finally resolved around 2015 (roughly= ), i think perhaps skip, or forsyth found it - i apologise if i have the at= tribution wrong.

fossil = is also unhelpful if it runs out of space - i don’t believe brucee ev= er forgave it for that.
this is less of a problem whe= n it is run with venti of course.

-Steve


<= blockquote type=3D"cite">On 4 Apr 2023, at 6:16 pm, noam@pixelhero.dev wrot= e:


I've sporadically encountered a deadlock in fossil. Naturally, whe= n your root file system crashes, it can be hard to debug. My solution: stop= having a root file system. Was able to attach acid using mycroft's too= ling from ANTS, and get a clean stack trace (https://pixelhero.dev/notebook/foss= il/stacks/2023-04-03.1).

After a few h= ours yesterday (ht= tps://pixelhero.dev/notebook/fossil/2023-04-03.html), I eventually trac= ked down the deadlock. When blockWrite is told to flush a clean block to di= sk - i.e. one which is already flushed - it removes the block from the cach= e's free list, locks the block, detects that it's clean, and then..= . drops the reference. While keeping the block locked. And in the cache.

This leak of the lock, of course, means that= the *next* access to the block - which is still in the cache! - hangs inde= finitely. This is seen exactly in the stack trace:

_cacheLocal grabs the block from the cache, tries to lock it, and = hangs indefinitely. Worse, it does so under a call to fileWalk, which holds= a different lock, so the effect spreads out and makes even more of the fil= e system inaccessible as well (the fileMetaFlush proc hangs waiting on this= file lock).

This patch just ensures we ca= ll blockPut on the BioClean path as well, thus unlocking the block and read= ding it to the cache's free lists.

The= patch is on my branch - https://git.sr.ht/~pixelher= odev/plan9/commit/1bf8bd4f44e058261da7e89d87527b12073c9e0f - but I figu= red I should probably post it here as well.

If anyone has any other patches that weren't in the 9legacy download = as of ~2018, please let me know! :)

---
sys/src/cmd/fossil/cache.c | 4 +++-
1 file ch= anged, 3 insertions(+), 1 deletion(-)

diff= --git a/sys/src/cmd/fossil/cache.c b/sys/src/cmd/fossil/cache.c
index f473d211e..2fec44949 100644
--- a/sys/src/cmd/f= ossil/cache.c
+++ b/sys/src/cmd/fossil/cache.c
<= div>@@ -1203,8 +1203,10 @@ blockWrite(Block *b, int waitlock)
fprint(2, "%s: %d:%x:%d iostate is %d in blockWrite\n",
<= /div>
argv0, bb->part, bb->addr, bb->l.type, bb->iostate);<= br />
/* probably BioWriting if it happens? */
- = if(bb->iostate =3D=3D BioClean)
+ if(bb->iostate = =3D=3D BioClean){
+ blockPut(bb);
goto i= gnblock;
+ }
}

blockPut(bb);
--

<= /blockquote> = --Apple-Mail-271A77DE-7338-4E68-86AB-4F396DEE34D0--