From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/87275 Path: news.gmane.org!.POSTED!not-for-mail From: myglc2 Newsgroups: gmane.emacs.gnus.general Subject: Re: searching local mail stores Date: Tue, 27 Sep 2016 13:29:19 -0400 Message-ID: <867f9x45g0.fsf@gmail.com> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Trace: blaine.gmane.org 1474998514 12250 195.159.176.226 (27 Sep 2016 17:48:34 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 27 Sep 2016 17:48:34 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) To: ding@gnus.org Original-X-From: ding-owner+M35495@lists.math.uh.edu Tue Sep 27 19:48:30 2016 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from lists1.math.uh.edu ([129.7.128.208]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bowUI-0000TI-1Y for ding-account@gmane.org; Tue, 27 Sep 2016 19:48:14 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by lists1.math.uh.edu with smtp (Exim 4.86_2) (envelope-from ) id 1bowTD-0002t0-Cq; Tue, 27 Sep 2016 12:47:07 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by lists1.math.uh.edu with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.86_2) (envelope-from ) id 1bowD4-0002hr-Ip for ding@lists.math.uh.edu; Tue, 27 Sep 2016 12:30:26 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtps (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.86_2) (envelope-from ) id 1bowD3-0003Ps-JT for ding@lists.math.uh.edu; Tue, 27 Sep 2016 12:30:26 -0500 Original-Received: from [195.159.176.226] (helo=blaine.gmane.org) by quimby.gnus.org with esmtps (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from ) id 1bowD2-0007Nu-72 for ding@gnus.org; Tue, 27 Sep 2016 19:30:24 +0200 Original-Received: from list by blaine.gmane.org with local (Exim 4.84_2) (envelope-from ) id 1bowCw-0004jq-0I for ding@gnus.org; Tue, 27 Sep 2016 19:30:18 +0200 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 74 Original-X-Complaints-To: usenet@blaine.gmane.org Cancel-Lock: sha1:e30+0LMDmWadvVAc4eubC6hvSAk= List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:87275 Archived-At: I received many helpful comments on my earlier post under this subject. I have since spent a few months off-and-on trying various mail search setups. Along the way I performed a couple benchmarks that might be of interest to list readers, so I am reporting them below. First, to get a handle on comments to the effect that nnmaildir is slow when there are many messages I put ~132,000 messages into 9 Maildir sub directories occupying 2.1G. I configured gnus to treat the top level maildir as a single store. On a 3.4 Ghz machine with a SSD, gnus took ~25 seconds to open the maildir. Initial notmuch indexing took a few minutes. Search performance depended on the number of search hits generated (see discussion below). For comparison, I deleted messages until the maildir had ~15,000 messages in 5 sub-directories occupying 1.1G, at which point gnus opened the maildir in ~5 seconds. Second, I wanted to compare gnus/notmuch with mu4e. notmuch and mu4e both use the xapian search/index engine and have emacs mail search UIs that are independent of gnus. I expected these to perform similarly and in casual comparison this seemed to be the case, so I did not compare the mu4e and notmuch UIs any further. However notmuch also supports the gnus nnir search interface, which allows a gnus 'G G' search to deliver notmuch search results to gnus summary buffers. To me, the appeal of notmuch was the possibility of fast search while otherwise continuing to read messages in gnus. So I compared gnus/notmuch with the mu4e UI, as shown in Table 1, below. Table 1: seconds to operate on a maildir directory containing 15,000 messages | operation | gnus/notmuch | mu4e | gnus/notmuch | mu4e (1) | | | first line | first line | All lines | All lines | +------------------+--------------+------------+--------------+-----------| | open | 5 | 1 | 5 | 10 | | (re)sort date | 1 | 1 | 1 | 10 | | (re)sort subject | 1 | 1 | 1 | 10 | Note: mu4e normally limits display to, at most, the first 500 search hits. In the "mu4e (1) All lines" results above, mu4e was forced to display all search hits by setting 'M-x mu4e-headers-toggle-full-search’. So, to generalize, mu4e is snappier than gnus/notmuch, unless we force mu4e to display a lot of lines. The biggest difference is the "open" time, which, as demonstrated above, becomes significant when the maildir contains a lot of messages. SEARCH PERFORMANCE: When searching, if the number of search hits is modest (<100), the search/display time is similar for gnus/notmuch and mu4e. However, when a search produces many hits (e.g., ~15,000), gnus takes about 10 sec to display all of the results and mu4e takes 23 (once again setting M-x mu4e-headers-toggle-full-search). So the most noticeable oveall difference between gnus/notmuch and mu4e is the time taken to initially "open" the maildir. Regarding setup, I found mu4e easier with its single point of configuration. In comparison, gnus/notmuch requires a compatible "parallel" configuration of notmuch and gnus. Bottom line: 1) gnus/notmuch/maildir works pretty well (at least on my computer) for 15,000 or so messages. 2) At 100,000 or messages, the gnus maildir startup delay is so tiresome that mu4e becomes truly compelling. FWIW, I am currently a happy user of both mu4e and gnus/notmuch. I use mu4e+mbsync to read multiple Gmail accounts. I use gnus/notmuch to search and read mailing list archives that I have mirrored locally into maildir. - George