Re: Experimental new Maildir backend

[Eric Abrahamsen <eric@ericabrahamsen.net>]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Saturday, 16 Jan 2021 at 20:49, Eric Abrahamsen wrote:
>> I've been toying with the idea of using sqlite as a store for Gnus'
>> caches and data: it seems like that would get us the biggest speedup
>> possible. 
>
> This sounds interesting.
>
> I used to use nnmaildir all the time, primarily because I frequently
> access my emails from three different devices which I keep in sync using
> unison.  The maildir format is ideal for this as there is no chance of
> conflict arising if I forget to sync any given device.
>
> However, the performance of the current nnmaildir is atrocious in such a
> scenario because the time stamp on the .overview file (I believe -- it's
> been a while since I did the analysis) leads to the whole mailbox being
> scanned again to build up the numerical indices used by gnus.  The scan
> is O(n^3) which becomes intractable when you have 1000s of emails in the
> mailbox.
>
> Anything that enables quick scanning/updating of the index for maildir
> groups would be a major improvement.

I think there are a couple of overlapping issues -- actual bugs vs
design problems -- that might need to be unpicked: it's obvious that
first-time nov database building when adopting lots of old mail has
terrible performance, and it shouldn't need to be that way. Then it
sounds like there might be a real bug in that nov databases are getting
rebuilt when they don't need to be? Lastly, perhaps there's further
inefficiency just when retrieving nov headers on a run-of-the-mill group
opening. Does that sound right? Is simply opening a group slow, even
when all the nov databases are built?

Anyway, no need to go digging up past research, I'm mostly just trying
to clarify things in my own head.