Re: Experimental new Maildir backend

["Juan José García-Ripoll" <juanjose.garciaripoll@gmail.com>]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:
> Very cool! It's great to see work on more backends. I'm also in the
> process of putting together fixes for nnmaildir, so there might be a
> little bit of redundancy in our work, but it looks like your approach is
> a more drastic rethinking -- my changes are mostly incremental tweaks.

Hi Eric, nice to see you also are interested in this format. I also
started tweaking nnmaildir. I got to a point where I fixed the
separator, introducing a configuration parameter that starts with ":"
but can be configured to other values. Unfortunately the result was
extremely slow.

I gather Maildir creates one NOV file for every message, it also creates
a folder for each flag and creates links (which I am not even sure
whether it does correctly on Windows, where links are not always
possible), for every article that has a mark. That made a folder with
~900 emails take minutes to load at all. I know that this is a Windows
limitations but I think it points out how wasteful it is. In contrast,
the archives I have based on nnml are very snappy.

> I saw on the github page that this is based off nnml code, and there are
> several functions (moving messages, creating groups) that haven't been
> implemented yet. I'm curious if you were able to use any of Gnus'
> backend inheritance features -- are you having to write everything from
> scratch? I haven't read the code yet...

I started using inheritance, but it did not work. The parent backend did
not get its variables properly assigned and there were lots of confusing
group names: i.e. Archives.2011 would have to be renamed
Archives.2011.cur so that nnml tries to first find that directory and
then attempts Archives.2011/cur. However, at some point I also was not
satisfied with nnml's approach, which only stores the NOV files and
enforces that all file names must be a number (the article number) and I
gave up. The result is not that bad, ~500 lines. The biggest hurdle was
figuring out which backend functions need to be created for the backend
to tell Gnus the attributes of a message. That is not at all clear in
the manual and nnmaildir, nnimap and others seem to follow different not
well documented paths.

> I've been toying with the idea of using sqlite as a store for Gnus'
> caches and data: it seems like that would get us the biggest speedup
> possible. I don't think it's feasible for Gnus' built-in backends, since
> vanilla Emacs doesn't come with anything for talking to sqlite, but if
> you made this an installable package, it could require the "sqlite"
> package and do it that way... Just a thought!

It is a hurdle to install sqlite in Emacs on Windows. I had to create my
own "build-from-source" distribution for packaging Emacs with other
dependencies that are not standard
(https://github.com/juanjosegarciaripoll/emacs-build).

However, the good news is that the hashtable cache approach very good. I
have folders with 10000's of files and loading the hash table is not
that bad. Plus, this can be done once and only updated later based on
times, or dropping the NOVs unless they are needed -- the current code
is not optimal in my implementation. I believe there is an Emacs
database package that offers a hashtable backend as default. Maybe that
would be a more reasonable approach to general caching, and it would
benefit other stakeholders, such as org-roam.

Thanks for the questions and feedback. You brought up very good points
that would need to be addressed for Maildir to be more functional --
which I believe is relevant, given that Maildir is gaining traction
again due to mu4e and equivalent (albeit, once more, Windows-buggy)
packages.

Cheers,

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es