From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 29065 invoked from network); 17 Jan 2021 22:23:31 -0000 Received: from mx1.math.uh.edu (129.7.128.32) by inbox.vuxu.org with ESMTPUTF8; 17 Jan 2021 22:23:31 -0000 Received: from lists1.math.uh.edu ([129.7.128.208]) by mx1.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1l1GSG-00GT9C-2c for ml@inbox.vuxu.org; Sun, 17 Jan 2021 16:23:28 -0600 Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by lists1.math.uh.edu with smtp (Exim 4.94) (envelope-from ) id 1l1GSF-006kME-GE for ml@inbox.vuxu.org; Sun, 17 Jan 2021 16:23:27 -0600 Received: from mx1.math.uh.edu ([129.7.128.32]) by lists1.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1l1GSD-006kM6-CK for ding@lists.math.uh.edu; Sun, 17 Jan 2021 16:23:25 -0600 Received: from quimby.gnus.org ([95.216.78.240]) by mx1.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1l1GSA-00GT8v-Uv for ding@lists.math.uh.edu; Sun, 17 Jan 2021 16:23:25 -0600 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=gpsy67MksQKgM6X+Kg2FuPXhXA4oxH0Es5S2iXCGtPg=; b=RU2wL70P8TUTawgdFEzAUfmgni jC+JB50jjklfvmE90xWOHRlTk6zh3Tqdv+tR9tsxeRKltg5xKzEfAmu7IcseON012HBrrtNMqWBa5 UCOpU1ZRR3cl1e+vcEDXQF9U/Eeo1csT5lFl7Dt/kwuZSzEy8B8OVKQcDj5H3s/ml7ik=; Received: from ericabrahamsen.net ([52.70.2.18] helo=mail.ericabrahamsen.net) by quimby.gnus.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l1GS0-0002TB-NL for ding@gnus.org; Sun, 17 Jan 2021 23:23:18 +0100 Received: from localhost (c-73-254-86-141.hsd1.wa.comcast.net [73.254.86.141]) (Authenticated sender: eric@ericabrahamsen.net) by mail.ericabrahamsen.net (Postfix) with ESMTPSA id A3E0DFA07E; Sun, 17 Jan 2021 22:23:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericabrahamsen.net; s=mail; t=1610922189; bh=gpsy67MksQKgM6X+Kg2FuPXhXA4oxH0Es5S2iXCGtPg=; h=From:To:Cc:Subject:References:Date:In-Reply-To:From; b=jhL4ykwDzt6jYv84/MvC4ZKcyJjxfHGYIRtS8X9xLYYFkre53BQjjXhk0DaZYp1Nt HbvrteHLLX4LTvb/ZqHxp/6pJY14SbO4HBhpT5aqVyvJSV74/UCXw9kIOjxx5VW40r kh9wYejxmFHg+Xpp1BpPv11mCPQ4a0O7cNV7zy8A= From: Eric Abrahamsen To: Eric S Fraga Cc: ding@gnus.org Subject: Re: Experimental new Maildir backend References: <86h7ngen1y.fsf@csic.es> <875z3w9nhm.fsf@ericabrahamsen.net> <87im7v8jbn.fsf@ucl.ac.uk> Date: Sun, 17 Jan 2021 14:23:03 -0800 In-Reply-To: <87im7v8jbn.fsf@ucl.ac.uk> (Eric S. Fraga's message of "Sun, 17 Jan 2021 19:17:00 +0000") Message-ID: <87bldn8apk.fsf@ericabrahamsen.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain List-ID: Precedence: bulk Eric S Fraga writes: > On Saturday, 16 Jan 2021 at 20:49, Eric Abrahamsen wrote: >> I've been toying with the idea of using sqlite as a store for Gnus' >> caches and data: it seems like that would get us the biggest speedup >> possible. > > This sounds interesting. > > I used to use nnmaildir all the time, primarily because I frequently > access my emails from three different devices which I keep in sync using > unison. The maildir format is ideal for this as there is no chance of > conflict arising if I forget to sync any given device. > > However, the performance of the current nnmaildir is atrocious in such a > scenario because the time stamp on the .overview file (I believe -- it's > been a while since I did the analysis) leads to the whole mailbox being > scanned again to build up the numerical indices used by gnus. The scan > is O(n^3) which becomes intractable when you have 1000s of emails in the > mailbox. > > Anything that enables quick scanning/updating of the index for maildir > groups would be a major improvement. I think there are a couple of overlapping issues -- actual bugs vs design problems -- that might need to be unpicked: it's obvious that first-time nov database building when adopting lots of old mail has terrible performance, and it shouldn't need to be that way. Then it sounds like there might be a real bug in that nov databases are getting rebuilt when they don't need to be? Lastly, perhaps there's further inefficiency just when retrieving nov headers on a run-of-the-mill group opening. Does that sound right? Is simply opening a group slow, even when all the nov databases are built? Anyway, no need to go digging up past research, I'm mostly just trying to clarify things in my own head.