is nnmaildir very slow?

is nnmaildir very slow?

[Eric S Fraga]

Hello all,

For a number of reasons, I have settled on nnmaildir for my main group
server.  Unfortunately, however, it seems to be very slow at the initial
scan, taking up to several minutes within the gnus-request-scan function
(I added some gnus-message lines in gnus-int.el).  I haven't figured out
yet why it is so slow.  

Does anybody else use maildir and see this?

I am using emacs from git, updated yesterday.  The group that takes
several minutes to scan has 8000 emails in it.

Thanks.
-- 
Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid

Re: is nnmaildir very slow?

[George Clemmer]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Hello all,
>
> For a number of reasons, I have settled on nnmaildir for my main group
> server.  Unfortunately, however, it seems to be very slow at the initial
> scan, taking up to several minutes within the gnus-request-scan function
> (I added some gnus-message lines in gnus-int.el).  I haven't figured out
> yet why it is so slow.
>
> Does anybody else use maildir and see this?
>
> I am using emacs from git, updated yesterday.  The group that takes
> several minutes to scan has 8000 emails in it.
>
> Thanks.

Hi Eric,

In 2016, when I used both emacs/mu4e and emacs/gnus/notmuch against
Maildir I found nnmaildir painfully slow to opening large
Maildirs. AIUI, it is a known issue with nnmaildir.

You may be interested in my post to gmane.emacs.gnus.general at the time
re relative performance.  I can't find it online, so I am attaching in
below.

FWIW, due to this issue, I have since abandoned emacs/gnus/nnmaildir and
now read all my Maildir mail using emacs/mu4e.

- George

***

From: myglc2 <myglc2@gmail.com>
Newsgroups: gmane.emacs.gnus.general
Subject: Re: searching local mail stores
Date: Tue, 06 Sep 2016 18:56:56 -0400

I received many helpful comments on my earlier post under this
subject. I have since spent a few months off-and-on trying various mail
search setups. Along the way I performed a couple benchmarks that might
be of interest to list readers, so I am reporting them below.

First, to get a handle on comments to the effect that nnmaildir is slow
when there are many messages I put ~132,000 messages into 9 Maildir sub
directories occupying 2.1G. I configured gnus to treat the top level
maildir as a single store.  On a 3.4 Ghz machine with a SSD, gnus took
~25 seconds to open the maildir. Initial notmuch indexing took a few
minutes. Search performance depended on the number of search hits
generated (see discussion below).  For comparison, I deleted messages
until the maildir had ~15,000 messages in 5 sub-directories occupying
1.1G, at which point gnus opened the maildir in ~5 seconds.

Second, I wanted to compare gnus/notmuch with mu4e. notmuch and mu4e
both use the xapian search/index engine and have emacs mail search UIs
that are independent of gnus. I expected these to perform similarly and
in casual comparison this seemed to be the case, so I did not compare
the mu4e and notmuch UIs any further.

However notmuch also supports the gnus nnir search interface, which
allows a gnus 'G G' search to deliver notmuch search results to gnus
summary buffers. To me, the appeal of notmuch was the possibility of
fast search while otherwise continuing to read messages in gnus.  So I
compared gnus/notmuch with the mu4e UI, as shown in Table 1, below.

Table 1: seconds to operate on a maildir directory containing 15,000 messages

| operation        | gnus/notmuch |       mu4e | gnus/notmuch |  mu4e (1) |
|                  |   first line | first line |    All lines | All lines |
+------------------+--------------+------------+--------------+-----------|
| open             |            5 |          1 |            5 |        10 |
| (re)sort date    |            1 |          1 |            1 |        10 |
| (re)sort subject |            1 |          1 |            1 |        10 |

Note: mu4e normally limits display to, at most, the first 500 search
hits. In the mu4e (1) All lines results above, mu4e was forced to
display all search hits by setting 'M-x mu4e-headers-toggle-full-search’.

So, to generalize, mu4e is snappier than gnus/notmuch, unless we force
mu4e to display a lot of lines.  The biggest difference is the "open"
time, which, as demonstrated above, becomes significant when the maildir
contains a lot of messages.

SEARCH PERFORMANCE:

When searching, if the number of search hits is modest (<100), the
search/display time is similar for gnus/notmuch and mu4e. However, when
a search produces many hits (e.g., ~15,000), gnus takes about 10 sec to
display all of the results and mu4e takes 23 (once again setting M-x
mu4e-headers-toggle-full-search).

So the most noticeable oveall difference between gnus/notmuch and mu4e
is the time taken to initially "open" the maildir.

Regarding setup, I found mu4e easier with its single point of
configuration. In comparison, gnus/notmuch requires a compatible
"parallel" configuration of notmuch and gnus.

Bottom line:

1) gnus/notmuch/maildir works pretty well (at least on my computer)
   for 15,000 or so messages.

2) At 100,000 or messages, the gnus maildir startup delay is so tiresome
   that mu4e becomes truly compelling.

FWIW, I am currently a happy user of both mu4e and gnus/notmuch. I use
mu4e+mbsync to read multiple Gmail accounts. I use gnus/notmuch to
search and read mailing list archives that I have mirrored locally into
maildir.

- George

Re: is nnmaildir very slow?

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Hello all,
>
> For a number of reasons, I have settled on nnmaildir for my main group
> server.  Unfortunately, however, it seems to be very slow at the initial
> scan, taking up to several minutes within the gnus-request-scan function
> (I added some gnus-message lines in gnus-int.el).  I haven't figured out
> yet why it is so slow.  
>
> Does anybody else use maildir and see this?
>
> I am using emacs from git, updated yesterday.  The group that takes
> several minutes to scan has 8000 emails in it.

Is it only the initial scan that's that slow? I'd expect that once the
NOV files are generated, opening the groups would be pretty quick. Is
that true?

Re: is nnmaildir very slow?

[James Cloos]

>>>>> "EF" == Eric S Fraga <e.fraga@ucl.ac.uk> writes:

EF> server.  Unfortunately, however, [nnmaildir] seems to be very slow
EF> at the initial scan, taking up to several minutes within the
EF> gnus-request-scan function (I added some gnus-message lines in
EF> gnus-int.el).  I haven't figured out yet why it is so slow.

I switched some years back to nnimap on a sql-backed imap daemon for
that very reason.

I don't know whether it still does, but it used to scan through the
groups three times every time I started.

Which took *horribly* long back on that laptop's ide drive and shared
ram/io bus.

On top of that bug, part of the issue is using the filesystem as a
database.  Anything using that style of storage suffers a superlinear
delay as the size grows.

I'd start by watching emacs with something like strace -e open to see
whether it still loops through all of the groups multiple times.

That bug still requires a fix if so.

Otherwise a general profile of a startup probably would help.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: is nnmaildir very slow?

[Eric S Fraga]

On Thursday, 16 May 2019 at 11:04, George Clemmer wrote:
> In 2016, when I used both emacs/mu4e and emacs/gnus/notmuch against
> Maildir I found nnmaildir painfully slow to opening large
> Maildirs. AIUI, it is a known issue with nnmaildir.
>
> You may be interested in my post to gmane.emacs.gnus.general at the time
> re relative performance.  I can't find it online, so I am attaching in
> below.

Thanks.  I would have thought that 8000 messages (unlike your 132k)
would not be too large.  But I guess I'm wrong.  Will have to rethink my
setup.
-- 
Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid

Re: is nnmaildir very slow?

[Eric S Fraga]

On Thursday, 16 May 2019 at 09:22, Eric Abrahamsen wrote:
> Is it only the initial scan that's that slow? I'd expect that once the
> NOV files are generated, opening the groups would be pretty quick. Is
> that true?

yes, it is, and I was indeed thinking it is the generation of NOV files
causing the delay.
-- 
Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid

Re: is nnmaildir very slow?

[Eric S Fraga]

On Thursday, 16 May 2019 at 13:27, James Cloos wrote:
> I don't know whether it still does, but it used to scan through the
> groups three times every time I started.
>
> Which took *horribly* long back on that laptop's ide drive and shared
> ram/io bus.
>
> On top of that bug, part of the issue is using the filesystem as a
> database.  Anything using that style of storage suffers a superlinear
> delay as the size grows.
>
> I'd start by watching emacs with something like strace -e open to see
> whether it still loops through all of the groups multiple times.
>
> That bug still requires a fix if so.

Thanks.  I'll investigate further.

-- 
Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid

Re: is nnmaildir very slow?

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Thursday, 16 May 2019 at 09:22, Eric Abrahamsen wrote:
>> Is it only the initial scan that's that slow? I'd expect that once the
>> NOV files are generated, opening the groups would be pretty quick. Is
>> that true?
>
> yes, it is, and I was indeed thinking it is the generation of NOV files
> causing the delay.

But after that is it quick? I can't imagine it's regenerating the NOV
files at every startup...

Re: is nnmaildir very slow?

[Andreas Schwab]

On Mai 16 2019, James Cloos <cloos@jhcloos.com> wrote:

> On top of that bug, part of the issue is using the filesystem as a
> database.  Anything using that style of storage suffers a superlinear
> delay as the size grows.

Not necessarily.  Modern file systems have sublinear directory lookups.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: is nnmaildir very slow?

[Eric S Fraga]

On Thursday, 16 May 2019 at 14:29, Eric Abrahamsen wrote:
> But after that is it quick? I can't imagine it's regenerating the NOV
> files at every startup...

It is quick after the initial startup.  I don't know if it is slow at
every startup.  

I use gnus on three different systems: home, mobile device (Gemini PDA),
and office, using unison to synchronize all files between the three
systems.  Every time I start up gnus on a different device, it is
definitely slow.  As I don't tend to re-start on a given device before
using another one, as I start emacs and leave it running while I'm using
that system, I don't know if subsequent start-ups are slow as well.

So maybe it's something to do with how my files are
synchronized.  Unison is being asked to synchronize .newsrc.eld and the
Maildir (& Mail & News) directories.

-- 
Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid

Re: is nnmaildir very slow?

[James Cloos]

>>>>> "AS" == Andreas Schwab <schwab@linux-m68k.org> writes:

AS> On Mai 16 2019, James Cloos <cloos@jhcloos.com> wrote:
>> On top of that bug, part of the issue is using the filesystem as a
>> database.  Anything using that style of storage suffers a superlinear
>> delay as the size grows.

AS> Not necessarily.  Modern file systems have sublinear directory lookups.

That is not the only issue.

Cache presure is often a huge issue.

nnmaildir triggered emacs' gc much more often.

Of course with ssds and 32g ram my current workstation is vastly better.

But gnus is still *slow* running gnus-group-get-new-news.


-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

Re: is nnmaildir very slow?

[George Clemmer]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Thursday, 16 May 2019 at 14:29, Eric Abrahamsen wrote:
>> But after that is it quick? I can't imagine it's regenerating the NOV
>> files at every startup...
>
> It is quick after the initial startup.  I don't know if it is slow at
> every startup.

In my experience, as long as I kept Emacs running the performance was
great and ~ to Mu4E. The hit was only associated with an Emacs restart.

> I use gnus on three different systems: home, mobile device (Gemini PDA),
> and office, using unison to synchronize all files between the three
> systems.

Cool! Would you recommend the Gemini?

> Every time I start up gnus on a different device, it is
> definitely slow.  As I don't tend to re-start on a given device before
> using another one, as I start emacs and leave it running while I'm using
> that system, I don't know if subsequent start-ups are slow as well.
>
> So maybe it's something to do with how my files are
> synchronized.  Unison is being asked to synchronize .newsrc.eld and the
> Maildir (& Mail & News) directories.

How well does that work? Are all your Gnus marks really keep in sync? I
would think marks would get jumbled if the systems receive mail
independently.

Thanks, - George

Re: is nnmaildir very slow?

[Eric S Fraga]

On Saturday, 18 May 2019 at 06:53, George Clemmer wrote:
> In my experience, as long as I kept Emacs running the performance was
> great and ~ to Mu4E. The hit was only associated with an Emacs restart.

Yes, this agrees with my experience.  But as I do use three systems each
day, I do have to start up Emacs several times and the delay each time
is rather annoying.

I've solved the problem by expiring more (archiving) and splitting more
creatively so as to keep my inbox small.

>> I use gnus on three different systems: home, mobile device (Gemini PDA),
>> and office, using unison to synchronize all files between the three
>> systems.
>
> Cool! Would you recommend the Gemini?

Difficult to say.  I really like it but I'm waiting impatiently for the
Pyra.  The issue is that the Gemini is fragile when it comes to Debian
as it was really intended as an Android device.  I do have the Linux-fu
necessary to keep it going okay but I do have some wariness in
recommending it more generally.  The Pyra, however, will be Debian from
top to bottom so should be robust as was the OpenPandora it is replacing
(and which I used for years).

>> So maybe it's something to do with how my files are
>> synchronized.  Unison is being asked to synchronize .newsrc.eld and the
>> Maildir (& Mail & News) directories.
>
> How well does that work? Are all your Gnus marks really keep in sync? I
> would think marks would get jumbled if the systems receive mail
> independently.

That's one of the reasons for using nnmaildir: downloading emails on
different systems does not cause problems with synchronization.  I've
not had gnus get confused (yet).  But that may be why nnmaildir is so
slow on start-up.

thanks,
eric

-- 
Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid

Re: is nnmaildir very slow?

[Eric S Fraga]

On Saturday, 18 May 2019 at 06:53, George Clemmer wrote:
> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> On Thursday, 16 May 2019 at 14:29, Eric Abrahamsen wrote:
>>> But after that is it quick? I can't imagine it's regenerating the NOV
>>> files at every startup...
>>
>> It is quick after the initial startup.  I don't know if it is slow at
>> every startup.
>
> In my experience, as long as I kept Emacs running the performance was
> great and ~ to Mu4E. The hit was only associated with an Emacs restart.

I've done more experimenting.  The problem is definitely at startup and
*only* after I have synchronized systems.  The maildir scan is a "deep"
one, it seems, based on whether the directory has been modified.  An
indication of modification is taken purely from the time attribute.  So,
my synchronization makes gnus think the contents have changed.

I don't know how to get around this...

-- 
Eric S Fraga via Emacs 27.0.50 & org 9.2.2 on Debian buster/sid

Re: is nnmaildir very slow?

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Saturday, 18 May 2019 at 06:53, George Clemmer wrote:
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> On Thursday, 16 May 2019 at 14:29, Eric Abrahamsen wrote:
>>>> But after that is it quick? I can't imagine it's regenerating the NOV
>>>> files at every startup...
>>>
>>> It is quick after the initial startup.  I don't know if it is slow at
>>> every startup.
>>
>> In my experience, as long as I kept Emacs running the performance was
>> great and ~ to Mu4E. The hit was only associated with an Emacs restart.
>
> I've done more experimenting.  The problem is definitely at startup and
> *only* after I have synchronized systems.  The maildir scan is a "deep"
> one, it seems, based on whether the directory has been modified.  An
> indication of modification is taken purely from the time attribute.  So,
> my synchronization makes gnus think the contents have changed.
>
> I don't know how to get around this...

I can't think of any way to get around it, except maybe by saving a
"last modified" time along with the server information in the files
you're syncing...

Re: is nnmaildir very slow?

[Eric S Fraga]

On Friday, 24 May 2019 at 14:26, Eric Abrahamsen wrote:
> I can't think of any way to get around it, except maybe by saving a
> "last modified" time along with the server information in the files
> you're syncing...

So it turns out that unison has a -times option that will update the
modification time when synchronizing [1].  This should have done the job
but it didn't.  There's something being checked by gnus (in
nnmaildir--scan) that I cannot quite catch.

I've given up and moved back to nnml.  The problem with nnml is the lack
of robustness when working on multiple systems.  Just have to be more
careful...

Thanks all for suggestions etc.

Footnotes:

[1] took well over an hour to do the initial re-synchronization given
    that my synchronized file hierarchy contains 220k files... gulp.

-- 
Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid