Re: is nnmaildir very slow?

[George Clemmer <myglc2@gmail.com>]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Hello all,
>
> For a number of reasons, I have settled on nnmaildir for my main group
> server.  Unfortunately, however, it seems to be very slow at the initial
> scan, taking up to several minutes within the gnus-request-scan function
> (I added some gnus-message lines in gnus-int.el).  I haven't figured out
> yet why it is so slow.
>
> Does anybody else use maildir and see this?
>
> I am using emacs from git, updated yesterday.  The group that takes
> several minutes to scan has 8000 emails in it.
>
> Thanks.

Hi Eric,

In 2016, when I used both emacs/mu4e and emacs/gnus/notmuch against
Maildir I found nnmaildir painfully slow to opening large
Maildirs. AIUI, it is a known issue with nnmaildir.

You may be interested in my post to gmane.emacs.gnus.general at the time
re relative performance.  I can't find it online, so I am attaching in
below.

FWIW, due to this issue, I have since abandoned emacs/gnus/nnmaildir and
now read all my Maildir mail using emacs/mu4e.

- George

***

From: myglc2 <myglc2@gmail.com>
Newsgroups: gmane.emacs.gnus.general
Subject: Re: searching local mail stores
Date: Tue, 06 Sep 2016 18:56:56 -0400

I received many helpful comments on my earlier post under this
subject. I have since spent a few months off-and-on trying various mail
search setups. Along the way I performed a couple benchmarks that might
be of interest to list readers, so I am reporting them below.

First, to get a handle on comments to the effect that nnmaildir is slow
when there are many messages I put ~132,000 messages into 9 Maildir sub
directories occupying 2.1G. I configured gnus to treat the top level
maildir as a single store.  On a 3.4 Ghz machine with a SSD, gnus took
~25 seconds to open the maildir. Initial notmuch indexing took a few
minutes. Search performance depended on the number of search hits
generated (see discussion below).  For comparison, I deleted messages
until the maildir had ~15,000 messages in 5 sub-directories occupying
1.1G, at which point gnus opened the maildir in ~5 seconds.

Second, I wanted to compare gnus/notmuch with mu4e. notmuch and mu4e
both use the xapian search/index engine and have emacs mail search UIs
that are independent of gnus. I expected these to perform similarly and
in casual comparison this seemed to be the case, so I did not compare
the mu4e and notmuch UIs any further.

However notmuch also supports the gnus nnir search interface, which
allows a gnus 'G G' search to deliver notmuch search results to gnus
summary buffers. To me, the appeal of notmuch was the possibility of
fast search while otherwise continuing to read messages in gnus.  So I
compared gnus/notmuch with the mu4e UI, as shown in Table 1, below.

Table 1: seconds to operate on a maildir directory containing 15,000 messages

| operation        | gnus/notmuch |       mu4e | gnus/notmuch |  mu4e (1) |
|                  |   first line | first line |    All lines | All lines |
+------------------+--------------+------------+--------------+-----------|
| open             |            5 |          1 |            5 |        10 |
| (re)sort date    |            1 |          1 |            1 |        10 |
| (re)sort subject |            1 |          1 |            1 |        10 |

Note: mu4e normally limits display to, at most, the first 500 search
hits. In the mu4e (1) All lines results above, mu4e was forced to
display all search hits by setting 'M-x mu4e-headers-toggle-full-search’.

So, to generalize, mu4e is snappier than gnus/notmuch, unless we force
mu4e to display a lot of lines.  The biggest difference is the "open"
time, which, as demonstrated above, becomes significant when the maildir
contains a lot of messages.

SEARCH PERFORMANCE:

When searching, if the number of search hits is modest (<100), the
search/display time is similar for gnus/notmuch and mu4e. However, when
a search produces many hits (e.g., ~15,000), gnus takes about 10 sec to
display all of the results and mu4e takes 23 (once again setting M-x
mu4e-headers-toggle-full-search).

So the most noticeable oveall difference between gnus/notmuch and mu4e
is the time taken to initially "open" the maildir.

Regarding setup, I found mu4e easier with its single point of
configuration. In comparison, gnus/notmuch requires a compatible
"parallel" configuration of notmuch and gnus.

Bottom line:

1) gnus/notmuch/maildir works pretty well (at least on my computer)
   for 15,000 or so messages.

2) At 100,000 or messages, the gnus maildir startup delay is so tiresome
   that mu4e becomes truly compelling.

FWIW, I am currently a happy user of both mu4e and gnus/notmuch. I use
mu4e+mbsync to read multiple Gmail accounts. I use gnus/notmuch to
search and read mailing list archives that I have mirrored locally into
maildir.

- George