From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 1019 invoked from network); 17 Jan 2021 09:02:06 -0000 Received: from mx1.math.uh.edu (129.7.128.32) by inbox.vuxu.org with ESMTPUTF8; 17 Jan 2021 09:02:06 -0000 Received: from lists1.math.uh.edu ([129.7.128.208]) by mx1.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1l13wi-00G33f-FQ for ml@inbox.vuxu.org; Sun, 17 Jan 2021 03:02:04 -0600 Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by lists1.math.uh.edu with smtp (Exim 4.94) (envelope-from ) id 1l13wh-006dMu-SG for ml@inbox.vuxu.org; Sun, 17 Jan 2021 03:02:03 -0600 Received: from mx2.math.uh.edu ([129.7.128.33]) by lists1.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1l13wf-006dMk-G7 for ding@lists.math.uh.edu; Sun, 17 Jan 2021 03:02:01 -0600 Received: from quimby.gnus.org ([95.216.78.240]) by mx2.math.uh.edu with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1l13wa-004Vmf-NX for ding@lists.math.uh.edu; Sun, 17 Jan 2021 03:02:01 -0600 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnus.org; s=20200322; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References :Message-ID:Date:Subject:From:To:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=QB2jbRjj1dJcjN7+oFwFoDGTetaeN8dUn9DFIX1Trqg=; b=BHXVKFwZVTepidaFqatDlrgpUh G3SjN9+tRjgNvuPEvvGhZP3F9anhuLHuUjbwKOIxj8OUXrFnHrWZDsPRTNWAo5PiQaCBWMLNsd7ah meBGufEqKZmRvRoVv//D1RYNRaNh6yCh1aSBYPrbYTyJJxjAeK1qATboVoDi13NlJ97s=; Received: from ciao.gmane.io ([116.202.254.214]) by quimby.gnus.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1l13wQ-0001xX-BK for ding@gnus.org; Sun, 17 Jan 2021 10:01:51 +0100 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1l13wP-0007bM-8f for ding@gnus.org; Sun, 17 Jan 2021 10:01:45 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: ding@gnus.org From: Juan =?utf-8?Q?Jos=C3=A9_Garc=C3=ADa-Ripoll?= Subject: Re: Experimental new Maildir backend Date: Sun, 17 Jan 2021 10:01:41 +0100 Message-ID: <868s8sdjii.fsf@csic.es> References: <86h7ngen1y.fsf@csic.es> <875z3w9nhm.fsf@ericabrahamsen.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1.90 (windows-nt) Cancel-Lock: sha1:gVd6W6iKL8GRo4qW67XhxU/HXng= List-ID: Precedence: bulk Eric Abrahamsen writes: > Very cool! It's great to see work on more backends. I'm also in the > process of putting together fixes for nnmaildir, so there might be a > little bit of redundancy in our work, but it looks like your approach is > a more drastic rethinking -- my changes are mostly incremental tweaks. Hi Eric, nice to see you also are interested in this format. I also started tweaking nnmaildir. I got to a point where I fixed the separator, introducing a configuration parameter that starts with ":" but can be configured to other values. Unfortunately the result was extremely slow. I gather Maildir creates one NOV file for every message, it also creates a folder for each flag and creates links (which I am not even sure whether it does correctly on Windows, where links are not always possible), for every article that has a mark. That made a folder with ~900 emails take minutes to load at all. I know that this is a Windows limitations but I think it points out how wasteful it is. In contrast, the archives I have based on nnml are very snappy. > I saw on the github page that this is based off nnml code, and there are > several functions (moving messages, creating groups) that haven't been > implemented yet. I'm curious if you were able to use any of Gnus' > backend inheritance features -- are you having to write everything from > scratch? I haven't read the code yet... I started using inheritance, but it did not work. The parent backend did not get its variables properly assigned and there were lots of confusing group names: i.e. Archives.2011 would have to be renamed Archives.2011.cur so that nnml tries to first find that directory and then attempts Archives.2011/cur. However, at some point I also was not satisfied with nnml's approach, which only stores the NOV files and enforces that all file names must be a number (the article number) and I gave up. The result is not that bad, ~500 lines. The biggest hurdle was figuring out which backend functions need to be created for the backend to tell Gnus the attributes of a message. That is not at all clear in the manual and nnmaildir, nnimap and others seem to follow different not well documented paths. > I've been toying with the idea of using sqlite as a store for Gnus' > caches and data: it seems like that would get us the biggest speedup > possible. I don't think it's feasible for Gnus' built-in backends, since > vanilla Emacs doesn't come with anything for talking to sqlite, but if > you made this an installable package, it could require the "sqlite" > package and do it that way... Just a thought! It is a hurdle to install sqlite in Emacs on Windows. I had to create my own "build-from-source" distribution for packaging Emacs with other dependencies that are not standard (https://github.com/juanjosegarciaripoll/emacs-build). However, the good news is that the hashtable cache approach very good. I have folders with 10000's of files and loading the hash table is not that bad. Plus, this can be done once and only updated later based on times, or dropping the NOVs unless they are needed -- the current code is not optimal in my implementation. I believe there is an Emacs database package that offers a hashtable backend as default. Maybe that would be a more reasonable approach to general caching, and it would benefit other stakeholders, such as org-roam. Thanks for the questions and feedback. You brought up very good points that would need to be addressed for Maildir to be more functional -- which I believe is relevant, given that Maildir is gaining traction again due to mu4e and equivalent (albeit, once more, Windows-buggy) packages. Cheers, -- Juan José García Ripoll http://juanjose.garciaripoll.com http://quinfog.hbar.es