* is nnmaildir very slow? @ 2019-05-16 5:32 Eric S Fraga 2019-05-16 15:04 ` George Clemmer ` (2 more replies) 0 siblings, 3 replies; 16+ messages in thread From: Eric S Fraga @ 2019-05-16 5:32 UTC (permalink / raw) To: ding Hello all, For a number of reasons, I have settled on nnmaildir for my main group server. Unfortunately, however, it seems to be very slow at the initial scan, taking up to several minutes within the gnus-request-scan function (I added some gnus-message lines in gnus-int.el). I haven't figured out yet why it is so slow. Does anybody else use maildir and see this? I am using emacs from git, updated yesterday. The group that takes several minutes to scan has 8000 emails in it. Thanks. -- Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 5:32 is nnmaildir very slow? Eric S Fraga @ 2019-05-16 15:04 ` George Clemmer 2019-05-16 21:00 ` Eric S Fraga 2019-05-16 16:22 ` Eric Abrahamsen 2019-05-16 17:27 ` James Cloos 2 siblings, 1 reply; 16+ messages in thread From: George Clemmer @ 2019-05-16 15:04 UTC (permalink / raw) To: Eric S Fraga; +Cc: ding Eric S Fraga <e.fraga@ucl.ac.uk> writes: > Hello all, > > For a number of reasons, I have settled on nnmaildir for my main group > server. Unfortunately, however, it seems to be very slow at the initial > scan, taking up to several minutes within the gnus-request-scan function > (I added some gnus-message lines in gnus-int.el). I haven't figured out > yet why it is so slow. > > Does anybody else use maildir and see this? > > I am using emacs from git, updated yesterday. The group that takes > several minutes to scan has 8000 emails in it. > > Thanks. Hi Eric, In 2016, when I used both emacs/mu4e and emacs/gnus/notmuch against Maildir I found nnmaildir painfully slow to opening large Maildirs. AIUI, it is a known issue with nnmaildir. You may be interested in my post to gmane.emacs.gnus.general at the time re relative performance. I can't find it online, so I am attaching in below. FWIW, due to this issue, I have since abandoned emacs/gnus/nnmaildir and now read all my Maildir mail using emacs/mu4e. - George *** From: myglc2 <myglc2@gmail.com> Newsgroups: gmane.emacs.gnus.general Subject: Re: searching local mail stores Date: Tue, 06 Sep 2016 18:56:56 -0400 I received many helpful comments on my earlier post under this subject. I have since spent a few months off-and-on trying various mail search setups. Along the way I performed a couple benchmarks that might be of interest to list readers, so I am reporting them below. First, to get a handle on comments to the effect that nnmaildir is slow when there are many messages I put ~132,000 messages into 9 Maildir sub directories occupying 2.1G. I configured gnus to treat the top level maildir as a single store. On a 3.4 Ghz machine with a SSD, gnus took ~25 seconds to open the maildir. Initial notmuch indexing took a few minutes. Search performance depended on the number of search hits generated (see discussion below). For comparison, I deleted messages until the maildir had ~15,000 messages in 5 sub-directories occupying 1.1G, at which point gnus opened the maildir in ~5 seconds. Second, I wanted to compare gnus/notmuch with mu4e. notmuch and mu4e both use the xapian search/index engine and have emacs mail search UIs that are independent of gnus. I expected these to perform similarly and in casual comparison this seemed to be the case, so I did not compare the mu4e and notmuch UIs any further. However notmuch also supports the gnus nnir search interface, which allows a gnus 'G G' search to deliver notmuch search results to gnus summary buffers. To me, the appeal of notmuch was the possibility of fast search while otherwise continuing to read messages in gnus. So I compared gnus/notmuch with the mu4e UI, as shown in Table 1, below. Table 1: seconds to operate on a maildir directory containing 15,000 messages | operation | gnus/notmuch | mu4e | gnus/notmuch | mu4e (1) | | | first line | first line | All lines | All lines | +------------------+--------------+------------+--------------+-----------| | open | 5 | 1 | 5 | 10 | | (re)sort date | 1 | 1 | 1 | 10 | | (re)sort subject | 1 | 1 | 1 | 10 | Note: mu4e normally limits display to, at most, the first 500 search hits. In the mu4e (1) All lines results above, mu4e was forced to display all search hits by setting 'M-x mu4e-headers-toggle-full-search’. So, to generalize, mu4e is snappier than gnus/notmuch, unless we force mu4e to display a lot of lines. The biggest difference is the "open" time, which, as demonstrated above, becomes significant when the maildir contains a lot of messages. SEARCH PERFORMANCE: When searching, if the number of search hits is modest (<100), the search/display time is similar for gnus/notmuch and mu4e. However, when a search produces many hits (e.g., ~15,000), gnus takes about 10 sec to display all of the results and mu4e takes 23 (once again setting M-x mu4e-headers-toggle-full-search). So the most noticeable oveall difference between gnus/notmuch and mu4e is the time taken to initially "open" the maildir. Regarding setup, I found mu4e easier with its single point of configuration. In comparison, gnus/notmuch requires a compatible "parallel" configuration of notmuch and gnus. Bottom line: 1) gnus/notmuch/maildir works pretty well (at least on my computer) for 15,000 or so messages. 2) At 100,000 or messages, the gnus maildir startup delay is so tiresome that mu4e becomes truly compelling. FWIW, I am currently a happy user of both mu4e and gnus/notmuch. I use mu4e+mbsync to read multiple Gmail accounts. I use gnus/notmuch to search and read mailing list archives that I have mirrored locally into maildir. - George ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 15:04 ` George Clemmer @ 2019-05-16 21:00 ` Eric S Fraga 0 siblings, 0 replies; 16+ messages in thread From: Eric S Fraga @ 2019-05-16 21:00 UTC (permalink / raw) To: ding On Thursday, 16 May 2019 at 11:04, George Clemmer wrote: > In 2016, when I used both emacs/mu4e and emacs/gnus/notmuch against > Maildir I found nnmaildir painfully slow to opening large > Maildirs. AIUI, it is a known issue with nnmaildir. > > You may be interested in my post to gmane.emacs.gnus.general at the time > re relative performance. I can't find it online, so I am attaching in > below. Thanks. I would have thought that 8000 messages (unlike your 132k) would not be too large. But I guess I'm wrong. Will have to rethink my setup. -- Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 5:32 is nnmaildir very slow? Eric S Fraga 2019-05-16 15:04 ` George Clemmer @ 2019-05-16 16:22 ` Eric Abrahamsen 2019-05-16 21:01 ` Eric S Fraga 2019-05-16 17:27 ` James Cloos 2 siblings, 1 reply; 16+ messages in thread From: Eric Abrahamsen @ 2019-05-16 16:22 UTC (permalink / raw) To: ding Eric S Fraga <e.fraga@ucl.ac.uk> writes: > Hello all, > > For a number of reasons, I have settled on nnmaildir for my main group > server. Unfortunately, however, it seems to be very slow at the initial > scan, taking up to several minutes within the gnus-request-scan function > (I added some gnus-message lines in gnus-int.el). I haven't figured out > yet why it is so slow. > > Does anybody else use maildir and see this? > > I am using emacs from git, updated yesterday. The group that takes > several minutes to scan has 8000 emails in it. Is it only the initial scan that's that slow? I'd expect that once the NOV files are generated, opening the groups would be pretty quick. Is that true? ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 16:22 ` Eric Abrahamsen @ 2019-05-16 21:01 ` Eric S Fraga 2019-05-16 21:29 ` Eric Abrahamsen 0 siblings, 1 reply; 16+ messages in thread From: Eric S Fraga @ 2019-05-16 21:01 UTC (permalink / raw) To: ding On Thursday, 16 May 2019 at 09:22, Eric Abrahamsen wrote: > Is it only the initial scan that's that slow? I'd expect that once the > NOV files are generated, opening the groups would be pretty quick. Is > that true? yes, it is, and I was indeed thinking it is the generation of NOV files causing the delay. -- Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 21:01 ` Eric S Fraga @ 2019-05-16 21:29 ` Eric Abrahamsen 2019-05-17 6:02 ` Eric S Fraga 0 siblings, 1 reply; 16+ messages in thread From: Eric Abrahamsen @ 2019-05-16 21:29 UTC (permalink / raw) To: ding Eric S Fraga <e.fraga@ucl.ac.uk> writes: > On Thursday, 16 May 2019 at 09:22, Eric Abrahamsen wrote: >> Is it only the initial scan that's that slow? I'd expect that once the >> NOV files are generated, opening the groups would be pretty quick. Is >> that true? > > yes, it is, and I was indeed thinking it is the generation of NOV files > causing the delay. But after that is it quick? I can't imagine it's regenerating the NOV files at every startup... ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 21:29 ` Eric Abrahamsen @ 2019-05-17 6:02 ` Eric S Fraga 2019-05-18 10:53 ` George Clemmer 0 siblings, 1 reply; 16+ messages in thread From: Eric S Fraga @ 2019-05-17 6:02 UTC (permalink / raw) To: ding On Thursday, 16 May 2019 at 14:29, Eric Abrahamsen wrote: > But after that is it quick? I can't imagine it's regenerating the NOV > files at every startup... It is quick after the initial startup. I don't know if it is slow at every startup. I use gnus on three different systems: home, mobile device (Gemini PDA), and office, using unison to synchronize all files between the three systems. Every time I start up gnus on a different device, it is definitely slow. As I don't tend to re-start on a given device before using another one, as I start emacs and leave it running while I'm using that system, I don't know if subsequent start-ups are slow as well. So maybe it's something to do with how my files are synchronized. Unison is being asked to synchronize .newsrc.eld and the Maildir (& Mail & News) directories. -- Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-17 6:02 ` Eric S Fraga @ 2019-05-18 10:53 ` George Clemmer 2019-05-19 9:20 ` Eric S Fraga 2019-05-20 9:53 ` Eric S Fraga 0 siblings, 2 replies; 16+ messages in thread From: George Clemmer @ 2019-05-18 10:53 UTC (permalink / raw) To: Eric S Fraga; +Cc: ding Eric S Fraga <e.fraga@ucl.ac.uk> writes: > On Thursday, 16 May 2019 at 14:29, Eric Abrahamsen wrote: >> But after that is it quick? I can't imagine it's regenerating the NOV >> files at every startup... > > It is quick after the initial startup. I don't know if it is slow at > every startup. In my experience, as long as I kept Emacs running the performance was great and ~ to Mu4E. The hit was only associated with an Emacs restart. > I use gnus on three different systems: home, mobile device (Gemini PDA), > and office, using unison to synchronize all files between the three > systems. Cool! Would you recommend the Gemini? > Every time I start up gnus on a different device, it is > definitely slow. As I don't tend to re-start on a given device before > using another one, as I start emacs and leave it running while I'm using > that system, I don't know if subsequent start-ups are slow as well. > > So maybe it's something to do with how my files are > synchronized. Unison is being asked to synchronize .newsrc.eld and the > Maildir (& Mail & News) directories. How well does that work? Are all your Gnus marks really keep in sync? I would think marks would get jumbled if the systems receive mail independently. Thanks, - George ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-18 10:53 ` George Clemmer @ 2019-05-19 9:20 ` Eric S Fraga 2019-05-20 9:53 ` Eric S Fraga 1 sibling, 0 replies; 16+ messages in thread From: Eric S Fraga @ 2019-05-19 9:20 UTC (permalink / raw) To: ding On Saturday, 18 May 2019 at 06:53, George Clemmer wrote: > In my experience, as long as I kept Emacs running the performance was > great and ~ to Mu4E. The hit was only associated with an Emacs restart. Yes, this agrees with my experience. But as I do use three systems each day, I do have to start up Emacs several times and the delay each time is rather annoying. I've solved the problem by expiring more (archiving) and splitting more creatively so as to keep my inbox small. >> I use gnus on three different systems: home, mobile device (Gemini PDA), >> and office, using unison to synchronize all files between the three >> systems. > > Cool! Would you recommend the Gemini? Difficult to say. I really like it but I'm waiting impatiently for the Pyra. The issue is that the Gemini is fragile when it comes to Debian as it was really intended as an Android device. I do have the Linux-fu necessary to keep it going okay but I do have some wariness in recommending it more generally. The Pyra, however, will be Debian from top to bottom so should be robust as was the OpenPandora it is replacing (and which I used for years). >> So maybe it's something to do with how my files are >> synchronized. Unison is being asked to synchronize .newsrc.eld and the >> Maildir (& Mail & News) directories. > > How well does that work? Are all your Gnus marks really keep in sync? I > would think marks would get jumbled if the systems receive mail > independently. That's one of the reasons for using nnmaildir: downloading emails on different systems does not cause problems with synchronization. I've not had gnus get confused (yet). But that may be why nnmaildir is so slow on start-up. thanks, eric -- Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-18 10:53 ` George Clemmer 2019-05-19 9:20 ` Eric S Fraga @ 2019-05-20 9:53 ` Eric S Fraga 2019-05-24 21:26 ` Eric Abrahamsen 1 sibling, 1 reply; 16+ messages in thread From: Eric S Fraga @ 2019-05-20 9:53 UTC (permalink / raw) To: ding On Saturday, 18 May 2019 at 06:53, George Clemmer wrote: > Eric S Fraga <e.fraga@ucl.ac.uk> writes: > >> On Thursday, 16 May 2019 at 14:29, Eric Abrahamsen wrote: >>> But after that is it quick? I can't imagine it's regenerating the NOV >>> files at every startup... >> >> It is quick after the initial startup. I don't know if it is slow at >> every startup. > > In my experience, as long as I kept Emacs running the performance was > great and ~ to Mu4E. The hit was only associated with an Emacs restart. I've done more experimenting. The problem is definitely at startup and *only* after I have synchronized systems. The maildir scan is a "deep" one, it seems, based on whether the directory has been modified. An indication of modification is taken purely from the time attribute. So, my synchronization makes gnus think the contents have changed. I don't know how to get around this... -- Eric S Fraga via Emacs 27.0.50 & org 9.2.2 on Debian buster/sid ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-20 9:53 ` Eric S Fraga @ 2019-05-24 21:26 ` Eric Abrahamsen 2019-05-25 9:23 ` Eric S Fraga 0 siblings, 1 reply; 16+ messages in thread From: Eric Abrahamsen @ 2019-05-24 21:26 UTC (permalink / raw) To: ding Eric S Fraga <e.fraga@ucl.ac.uk> writes: > On Saturday, 18 May 2019 at 06:53, George Clemmer wrote: >> Eric S Fraga <e.fraga@ucl.ac.uk> writes: >> >>> On Thursday, 16 May 2019 at 14:29, Eric Abrahamsen wrote: >>>> But after that is it quick? I can't imagine it's regenerating the NOV >>>> files at every startup... >>> >>> It is quick after the initial startup. I don't know if it is slow at >>> every startup. >> >> In my experience, as long as I kept Emacs running the performance was >> great and ~ to Mu4E. The hit was only associated with an Emacs restart. > > I've done more experimenting. The problem is definitely at startup and > *only* after I have synchronized systems. The maildir scan is a "deep" > one, it seems, based on whether the directory has been modified. An > indication of modification is taken purely from the time attribute. So, > my synchronization makes gnus think the contents have changed. > > I don't know how to get around this... I can't think of any way to get around it, except maybe by saving a "last modified" time along with the server information in the files you're syncing... ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-24 21:26 ` Eric Abrahamsen @ 2019-05-25 9:23 ` Eric S Fraga 0 siblings, 0 replies; 16+ messages in thread From: Eric S Fraga @ 2019-05-25 9:23 UTC (permalink / raw) To: ding On Friday, 24 May 2019 at 14:26, Eric Abrahamsen wrote: > I can't think of any way to get around it, except maybe by saving a > "last modified" time along with the server information in the files > you're syncing... So it turns out that unison has a -times option that will update the modification time when synchronizing [1]. This should have done the job but it didn't. There's something being checked by gnus (in nnmaildir--scan) that I cannot quite catch. I've given up and moved back to nnml. The problem with nnml is the lack of robustness when working on multiple systems. Just have to be more careful... Thanks all for suggestions etc. Footnotes: [1] took well over an hour to do the initial re-synchronization given that my synchronized file hierarchy contains 220k files... gulp. -- Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 5:32 is nnmaildir very slow? Eric S Fraga 2019-05-16 15:04 ` George Clemmer 2019-05-16 16:22 ` Eric Abrahamsen @ 2019-05-16 17:27 ` James Cloos 2019-05-16 21:02 ` Eric S Fraga 2019-05-16 21:40 ` Andreas Schwab 2 siblings, 2 replies; 16+ messages in thread From: James Cloos @ 2019-05-16 17:27 UTC (permalink / raw) To: Eric S Fraga; +Cc: ding >>>>> "EF" == Eric S Fraga <e.fraga@ucl.ac.uk> writes: EF> server. Unfortunately, however, [nnmaildir] seems to be very slow EF> at the initial scan, taking up to several minutes within the EF> gnus-request-scan function (I added some gnus-message lines in EF> gnus-int.el). I haven't figured out yet why it is so slow. I switched some years back to nnimap on a sql-backed imap daemon for that very reason. I don't know whether it still does, but it used to scan through the groups three times every time I started. Which took *horribly* long back on that laptop's ide drive and shared ram/io bus. On top of that bug, part of the issue is using the filesystem as a database. Anything using that style of storage suffers a superlinear delay as the size grows. I'd start by watching emacs with something like strace -e open to see whether it still loops through all of the groups multiple times. That bug still requires a fix if so. Otherwise a general profile of a startup probably would help. -JimC -- James Cloos <cloos@jhcloos.com> OpenPGP: 0x997A9F17ED7DAEA6 ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 17:27 ` James Cloos @ 2019-05-16 21:02 ` Eric S Fraga 2019-05-16 21:40 ` Andreas Schwab 1 sibling, 0 replies; 16+ messages in thread From: Eric S Fraga @ 2019-05-16 21:02 UTC (permalink / raw) To: ding On Thursday, 16 May 2019 at 13:27, James Cloos wrote: > I don't know whether it still does, but it used to scan through the > groups three times every time I started. > > Which took *horribly* long back on that laptop's ide drive and shared > ram/io bus. > > On top of that bug, part of the issue is using the filesystem as a > database. Anything using that style of storage suffers a superlinear > delay as the size grows. > > I'd start by watching emacs with something like strace -e open to see > whether it still loops through all of the groups multiple times. > > That bug still requires a fix if so. Thanks. I'll investigate further. -- Eric S Fraga via Emacs 27.0.50 & org 9.2.3 on Debian buster/sid ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 17:27 ` James Cloos 2019-05-16 21:02 ` Eric S Fraga @ 2019-05-16 21:40 ` Andreas Schwab 2019-05-17 17:46 ` James Cloos 1 sibling, 1 reply; 16+ messages in thread From: Andreas Schwab @ 2019-05-16 21:40 UTC (permalink / raw) To: James Cloos; +Cc: Eric S Fraga, ding On Mai 16 2019, James Cloos <cloos@jhcloos.com> wrote: > On top of that bug, part of the issue is using the filesystem as a > database. Anything using that style of storage suffers a superlinear > delay as the size grows. Not necessarily. Modern file systems have sublinear directory lookups. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1 "And now for something completely different." ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: is nnmaildir very slow? 2019-05-16 21:40 ` Andreas Schwab @ 2019-05-17 17:46 ` James Cloos 0 siblings, 0 replies; 16+ messages in thread From: James Cloos @ 2019-05-17 17:46 UTC (permalink / raw) To: Andreas Schwab; +Cc: Eric S Fraga, ding >>>>> "AS" == Andreas Schwab <schwab@linux-m68k.org> writes: AS> On Mai 16 2019, James Cloos <cloos@jhcloos.com> wrote: >> On top of that bug, part of the issue is using the filesystem as a >> database. Anything using that style of storage suffers a superlinear >> delay as the size grows. AS> Not necessarily. Modern file systems have sublinear directory lookups. That is not the only issue. Cache presure is often a huge issue. nnmaildir triggered emacs' gc much more often. Of course with ssds and 32g ram my current workstation is vastly better. But gnus is still *slow* running gnus-group-get-new-news. -JimC -- James Cloos <cloos@jhcloos.com> OpenPGP: 0x997A9F17ED7DAEA6 ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2019-05-25 9:23 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-05-16 5:32 is nnmaildir very slow? Eric S Fraga 2019-05-16 15:04 ` George Clemmer 2019-05-16 21:00 ` Eric S Fraga 2019-05-16 16:22 ` Eric Abrahamsen 2019-05-16 21:01 ` Eric S Fraga 2019-05-16 21:29 ` Eric Abrahamsen 2019-05-17 6:02 ` Eric S Fraga 2019-05-18 10:53 ` George Clemmer 2019-05-19 9:20 ` Eric S Fraga 2019-05-20 9:53 ` Eric S Fraga 2019-05-24 21:26 ` Eric Abrahamsen 2019-05-25 9:23 ` Eric S Fraga 2019-05-16 17:27 ` James Cloos 2019-05-16 21:02 ` Eric S Fraga 2019-05-16 21:40 ` Andreas Schwab 2019-05-17 17:46 ` James Cloos
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).