From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FROM,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2 autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 418 invoked from network); 4 Apr 2023 18:34:35 -0000 Received: from tb-ob0.topicbox.com (64.147.108.117) by inbox.vuxu.org with ESMTPUTF8; 4 Apr 2023 18:34:35 -0000 Received: from tb-mx1.topicbox.com (tb-mx1.nyi.icgroup.com [10.90.30.61]) by tb-ob0.topicbox.com (Postfix) with ESMTP id 579D32A2A7 for ; Tue, 4 Apr 2023 14:34:33 -0400 (EDT) (envelope-from bounce.mMc25a40069de1a1f118f53839.r522be890-2105-11eb-b15e-8d699134e1fa@9fans.bounce.topicbox.com) Received: by tb-mx1.topicbox.com (Postfix, from userid 1132) id 7506017AC653; Tue, 4 Apr 2023 14:34:33 -0400 (EDT) ARC-Authentication-Results: i=2; topicbox.com; arc=pass; dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=aXRT9PPd header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; spf=pass smtp.mailfrom=skip.tavakkolian@gmail.com smtp.helo=mail-pj1-f47.google.com; x-internal-arc=fail (as.1.topicbox.com=pass, ams.1.topicbox.com=fail (message has been altered)) (Message modified while forwarding at Topicbox) ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=mime-version:references:in-reply-to:from:date :message-id:subject:to:content-type:content-transfer-encoding :list-help:list-id:list-post:list-subscribe:reply-to :list-unsubscribe; s=sysmsg-1; t=1680633273; bh=JQ+nM98N2XJNN/5/ e23A9tZWo2syKXjdebJC1URdn/M=; b=czPcdi7pvXINfjFoqkN05i0+XVzEezAN MSlYRPnc58Q2qZrh/XaIRDH9Qw797J50wUdqnH3RNyx9PFKJcxiBQd+bptsrSNQI NbYoY4tV44WPbLK5nHRgbvqmU9JuITBU4hdH/qAm6GFr4E90dQhvCT0PHe1KeOys GnkTouEvvXY= ARC-Seal: i=2; a=rsa-sha256; cv=pass; d=topicbox.com; s=sysmsg-1; t= 1680633273; b=VW5BhrWsjYOM45QgSQ2U6H2S+H1fE/x6ardkOPU8qIoQ9xwU5R tL1YPKTPJlSWBT3GZVpMpV1zBFqcHmOZRefzxfJIBid2hPSNFRGUlZApatfr1PMm eOSZQX6GyqqrXGJboajhmLYWUuB3evYrHLa8vwZUcbvKOF68cOMqawNSk= Authentication-Results: topicbox.com; arc=pass; dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=aXRT9PPd header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; spf=pass smtp.mailfrom=skip.tavakkolian@gmail.com smtp.helo=mail-pj1-f47.google.com; x-internal-arc=fail (as.1.topicbox.com=pass, ams.1.topicbox.com=fail (message has been altered)) (Message modified while forwarding at Topicbox) X-Received-Authentication-Results: tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC Policy is not at enforcement); dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=aXRT9PPd header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; iprev=pass smtp.remote-ip=209.85.216.47 (mail-pj1-f47.google.com); spf=pass smtp.mailfrom=skip.tavakkolian@gmail.com smtp.helo=mail-pj1-f47.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=wvyNLFFW; x-me-sender=none; x-ptr=pass smtp.helo=mail-pj1-f47.google.com policy.ptr=mail-pj1-f47.google.com; x-return-mx=pass header.domain=gmail.com policy.is_org=yes (MX Records found: alt2.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com); x-return-mx=pass smtp.domain=gmail.com policy.is_org=yes (MX Records found: alt2.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=9fans.net; h= mime-version:references:in-reply-to:from:date:message-id:subject :to:content-type:content-transfer-encoding:list-help:list-id :list-post:list-subscribe:reply-to:list-unsubscribe; s=dkim-1; t=1680633273; x=1680719673; bh=JQ+nM98N2XJNN/5/e23A9tZWo2syKXjd ebJC1URdn/M=; b=HQyK+Y0GpYcMHR/222K+JtDJbKj5owyOQFvzgSmWDbcrEApu 9rUT2rJh8ozoYWcqM8qnkoM/yLAVMLibqMsvGNsi3zRlLGwnC+iClpaQaGTzYESA qzs4ZQtCbz6YmTxLodqENVqjrhlXgbo4068UDCYyGzn4uxSWot30CNqxEFY= Received: from tb-mx1.topicbox.com (localhost.local [127.0.0.1]) by tb-mx1.topicbox.com (Postfix) with ESMTP id 0516B17AC212 for <9fans@9fans.net>; Tue, 4 Apr 2023 14:34:23 -0400 (EDT) (envelope-from skip.tavakkolian@gmail.com) Received: from tb-mx1.topicbox.com (localhost [127.0.0.1]) by tb-mx1.topicbox.com (Authentication Milter) with ESMTP id A78E5F445DF; Tue, 4 Apr 2023 14:34:23 -0400 ARC-Seal: i=1; a=rsa-sha256; cv=none; d=topicbox.com; s=arcseal; t= 1680633262; b=hYcs8+NddwSDJaKPsMSIEw2N+PbI4FMlc2e+o8VLG+uwrAvt5k EXVu6EuFuLuA19nNeHhF+LqTFamh/1y4KfXK1EmQDlXiZzu/MnevEXyblIp74ld9 kGmeu5elMdRfoNRh5+vRMuCeQqRFxLW5lJANhvp+crYsrrRLX1IfVyY7ROszV2S8 LvBr2Zh0d1jpjFqrsmGqPz4S+FXWQ1h3RjxHRmYEzLhWg+7Bfx2Ksd+MA0ZYvYQr s3iJYgzceHbGGHUcL6VWQ3MUPck47aQ4s/z4ADKJ7ez6U3A6BoAXqxF++rDOJSfp XeDLvgZ3Dxp6jJzc4p4GxY+Aci6xnzCHHusA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= topicbox.com; h=mime-version:references:in-reply-to:from:date :message-id:subject:to:content-type:content-transfer-encoding; s=arcseal; t=1680633262; bh=xQ+bB1tq6PmzufOxjNJqdLTeIwtbMHr8pLY F5r5poNA=; b=DTJv2eta1GCgf1eGPBo6zKFRt30TSCtFOZoBZgPwH0aVbeyldBG FwC3n74waCGRm2AYoS8Hn4IBRRkIlnnqiQkewxf4u91wubNvMx6ksGY3u52AIcjq Z2ta/XJDRD7tdcD5gSsIME2EHNihHf5uPoVuqSZGhXqy7riv4vz9e5nmPdtoT8vm 1mEOumiJQkiUvDH65JLwH2kRsXaxBa+FaUR8MhXz9VjQ10ZhpsdSk2ljFx4wYcsn denainwgcr+W/0yTRSc9UwbeytJQSOvguOMIFdwt02PT9QzN7eWppFxn6c/xu45o 7ddwgZHyZ9QhQ7XnOh2ZnjxYpMqBZXz77Vw== ARC-Authentication-Results: i=1; tb-mx1.topicbox.com; arc=none (no signatures found); bimi=skipped (DMARC Policy is not at enforcement); dkim=pass (2048-bit rsa key sha256) header.d=gmail.com header.i=@gmail.com header.b=aXRT9PPd header.a=rsa-sha256 header.s=20210112 x-bits=2048; dmarc=pass policy.published-domain-policy=none policy.published-subdomain-policy=quarantine policy.applied-disposition=none policy.evaluated-disposition=none (p=none,sp=quarantine,d=none,d.eval=none) policy.policy-from=p header.from=gmail.com; iprev=pass smtp.remote-ip=209.85.216.47 (mail-pj1-f47.google.com); spf=pass smtp.mailfrom=skip.tavakkolian@gmail.com smtp.helo=mail-pj1-f47.google.com; x-aligned-from=pass (Address match); x-google-dkim=pass (2048-bit rsa key) header.d=1e100.net header.i=@1e100.net header.b=wvyNLFFW; x-me-sender=none; x-ptr=pass smtp.helo=mail-pj1-f47.google.com policy.ptr=mail-pj1-f47.google.com; x-return-mx=pass header.domain=gmail.com policy.is_org=yes (MX Records found: alt2.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com); x-return-mx=pass smtp.domain=gmail.com policy.is_org=yes (MX Records found: alt2.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,alt3.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com); x-tls=pass smtp.version=TLSv1.2 smtp.cipher=ECDHE-RSA-AES256-GCM-SHA384 smtp.bits=256/256; x-vs=clean score=0 state=0 X-ME-VSCause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeiledguddviecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepgghfjg fhfffkuffvtgfgsehtqhertddttdejnecuhfhrohhmpefukhhiphcuvfgrvhgrkhhkohhl ihgrnhcuoehskhhiphdrthgrvhgrkhhkohhlihgrnhesghhmrghilhdrtghomheqnecugg ftrfgrthhtvghrnhepfeffheefgefgheevtddtudfhtefgledvueeiteegfeeuteeggeev feekvdeufeetnecuffhomhgrihhnpehpihigvghlhhgvrhhordguvghvpdhsrhdrhhhtne cukfhppedvtdelrdekhedrvdduiedrgeejnecuvehluhhsthgvrhfuihiivgeptdenucfr rghrrghmpehinhgvthepvddtledrkeehrddvudeirdegjedphhgvlhhopehmrghilhdqph hjuddqfhegjedrghhoohhglhgvrdgtohhmpdhmrghilhhfrhhomhepoehskhhiphdrthgr vhgrkhhkohhlihgrnhesghhmrghilhdrtghomheq X-ME-VSScore: 0 X-ME-VSCategory: clean Received-SPF: pass (gmail.com ... _spf.google.com: Sender is authorized to use 'skip.tavakkolian@gmail.com' in 'mfrom' identity (mechanism 'include:_netblocks.google.com' matched)) receiver=tb-mx1.topicbox.com; identity=mailfrom; envelope-from="skip.tavakkolian@gmail.com"; helo=mail-pj1-f47.google.com; client-ip=209.85.216.47 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tb-mx1.topicbox.com (Postfix) with ESMTPS for <9fans@9fans.net>; Tue, 4 Apr 2023 14:34:22 -0400 (EDT) (envelope-from skip.tavakkolian@gmail.com) Received: by mail-pj1-f47.google.com with SMTP id r7-20020a17090b050700b002404be7920aso33048966pjz.5 for <9fans@9fans.net>; Tue, 04 Apr 2023 11:34:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680633261; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xQ+bB1tq6PmzufOxjNJqdLTeIwtbMHr8pLYF5r5poNA=; b=wvyNLFFWUhQRwP1rR873l0Jy/qA+vNIBvaClRqaZ+drTAc9hWIUVDHwLFaRK4wwhVD zjcv4SR3Aw8bKS3wLfy3rE3hi0vXxqmgJ+R7rl2k3rH7RG4ykP7L0TnnlJuAs0mwWvWA OnklpEAqxZTH+X3MhJPIwo4c6tDl2OLR4DFRVMsSpHZ6sccNrrNUuhGYMJMb6Zm567nF zSrBmVlfAVHX/1iuyNkF5AGcodGT6689CP2YFNdimFuYtvEZpGF1b0aJuDRw8GAm7pzT 5viEvl5wqBBZrWoD7Cm+7R3ZRlEyfIHzwiUXgoyUYwd3h96K7NY1LQW8ABds1dyDb7h8 4/Kg== X-Gm-Message-State: AAQBX9cJJFkJWv2gyUybZjmSteEUyBzrvQlO5Y3kYwy1cFRTlmvmSmE4 OB9zza3XmR1ReobL/hTn3F+gGW1eNXzpNApbnHcseCNzWgc= X-Google-Smtp-Source: AKy350Z4R+prW5iPA5KbIoamjigCUlHNIEHzc+vUJVkAsvxLz188yOpKOcB9Zp/0lxfnemvbYqeWNl0FCAb2dQ8tEXs= X-Received: by 2002:a17:902:bd07:b0:19f:239a:d80c with SMTP id p7-20020a170902bd0700b0019f239ad80cmr1370550pls.7.1680633260844; Tue, 04 Apr 2023 11:34:20 -0700 (PDT) MIME-Version: 1.0 References: <16806285560.EA879D18D.575584@composer.9fans.topicbox.com> In-Reply-To: From: Skip Tavakkolian Date: Tue, 4 Apr 2023 11:34:09 -0700 Message-ID: Subject: Re: [9fans] [PATCH] fossil: fix a deadlock in the caching logic To: 9fans <9fans@9fans.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Topicbox-Policy-Reasoning: allow: sender is a member Topicbox-Message-UUID: 54655526-d317-11ed-acb0-ec797882298c Archived-At: =?UTF-8?B?PGh0dHBzOi8vOWZhbnMudG9waWNib3guY29tL2dyb3Vwcy85?= =?UTF-8?B?ZmFucy9UMzU0ZmU3MDJlMWU5ZDVlOS1NYzI1YTQwMDY5ZGUxYTFmMTE4ZjUz?= =?UTF-8?B?ODM5Pg==?= List-Help: List-Id: "9fans" <9fans.9fans.net> List-Post: List-Software: Topicbox v0 List-Subscribe: Precedence: list Reply-To: 9fans <9fans@9fans.net> List-Unsubscribe: , Topicbox-Delivery-ID: 2:9fans:437d30aa-c441-11e9-8a57-d036212d11b0:522be890-2105-11eb-b15e-8d699134e1fa:Mc25a40069de1a1f118f53839:1:kcS5YUUycfcVTEiZnlBQ4RzP8DTtWK2kWy02S_E5exE it definitely was not me. My bet would be on rsc, geoff, richard, forsyth, quanstrom or djc. On Tue, Apr 4, 2023 at 11:05=E2=80=AFAM Steve Simon wr= ote: > > > was this hard to reproduce? > > i have not seen fossil deadlocking and have used it since i installed my = first home server in 2004. > > there definitely _was_ a problem in the snapshot code which was finally r= esolved around 2015 (roughly), i think perhaps skip, or forsyth found it - = i apologise if i have the attribution wrong. > > fossil is also unhelpful if it runs out of space - i don=E2=80=99t believ= e brucee ever forgave it for that. > this is less of a problem when it is run with venti of course. > > -Steve > > > On 4 Apr 2023, at 6:16 pm, noam@pixelhero.dev wrote: > > =EF=BB=BF > I've sporadically encountered a deadlock in fossil. Naturally, when your = root file system crashes, it can be hard to debug. My solution: stop having= a root file system. Was able to attach acid using mycroft's tooling from A= NTS, and get a clean stack trace (https://pixelhero.dev/notebook/fossil/sta= cks/2023-04-03.1). > > After a few hours yesterday (https://pixelhero.dev/notebook/fossil/2023-0= 4-03.html), I eventually tracked down the deadlock. When blockWrite is told= to flush a clean block to disk - i.e. one which is already flushed - it re= moves the block from the cache's free list, locks the block, detects that i= t's clean, and then... drops the reference. While keeping the block locked.= And in the cache. > > This leak of the lock, of course, means that the *next* access to the blo= ck - which is still in the cache! - hangs indefinitely. This is seen exactl= y in the stack trace: > > _cacheLocal grabs the block from the cache, tries to lock it, and hangs i= ndefinitely. Worse, it does so under a call to fileWalk, which holds a diff= erent lock, so the effect spreads out and makes even more of the file syste= m inaccessible as well (the fileMetaFlush proc hangs waiting on this file l= ock). > > This patch just ensures we call blockPut on the BioClean path as well, th= us unlocking the block and readding it to the cache's free lists. > > The patch is on my branch - https://git.sr.ht/~pixelherodev/plan9/commit/= 1bf8bd4f44e058261da7e89d87527b12073c9e0f - but I figured I should probably = post it here as well. > > If anyone has any other patches that weren't in the 9legacy download as o= f ~2018, please let me know! :) > > --- > sys/src/cmd/fossil/cache.c | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/sys/src/cmd/fossil/cache.c b/sys/src/cmd/fossil/cache.c > index f473d211e..2fec44949 100644 > --- a/sys/src/cmd/fossil/cache.c > +++ b/sys/src/cmd/fossil/cache.c > @@ -1203,8 +1203,10 @@ blockWrite(Block *b, int waitlock) > fprint(2, "%s: %d:%x:%d iostate is %d in blockWrite\n", > argv0, bb->part, bb->addr, bb->l.type, bb->iostate); > /* probably BioWriting if it happens? */ > - if(bb->iostate =3D=3D BioClean) > + if(bb->iostate =3D=3D BioClean){ > + blockPut(bb); > goto ignblock; > + } > } > > blockPut(bb); > -- > > 9fans / 9fans / see discussions + participants + delivery options Permali= nk ------------------------------------------ 9fans: 9fans Permalink: https://9fans.topicbox.com/groups/9fans/T354fe702e1e9d5e9-Mc25a4= 0069de1a1f118f53839 Delivery options: https://9fans.topicbox.com/groups/9fans/subscription