Re: searching local mail stores

Re: searching local mail stores

[myglc2]

I received many helpful comments on my earlier post under this
subject. I have since spent a few months off-and-on trying various mail
search setups. Along the way I performed a couple benchmarks that might
be of interest to list readers, so I am reporting them below.

First, to get a handle on comments to the effect that nnmaildir is slow
when there are many messages I put ~132,000 messages into 9 Maildir sub
directories occupying 2.1G. I configured gnus to treat the top level
maildir as a single store.  On a 3.4 Ghz machine with a SSD, gnus took
~25 seconds to open the maildir. Initial notmuch indexing took a few
minutes. Search performance depended on the number of search hits
generated (see discussion below).  For comparison, I deleted messages
until the maildir had ~15,000 messages in 5 sub-directories occupying
1.1G, at which point gnus opened the maildir in ~5 seconds.

Second, I wanted to compare gnus/notmuch with mu4e. notmuch and mu4e
both use the xapian search/index engine and have emacs mail search UIs
that are independent of gnus. I expected these to perform similarly and
in casual comparison this seemed to be the case, so I did not compare
the mu4e and notmuch UIs any further.

However notmuch also supports the gnus nnir search interface, which
allows a gnus 'G G' search to deliver notmuch search results to gnus
summary buffers. To me, the appeal of notmuch was the possibility of
fast search while otherwise continuing to read messages in gnus.  So I
compared gnus/notmuch with the mu4e UI, as shown in Table 1, below.

Table 1: seconds to operate on a maildir directory containing 15,000 messages

| operation        | gnus/notmuch |       mu4e | gnus/notmuch |  mu4e (1) |
|                  |   first line | first line |    All lines | All lines |
+------------------+--------------+------------+--------------+-----------|
| open             |            5 |          1 |            5 |        10 |
| (re)sort date    |            1 |          1 |            1 |        10 |
| (re)sort subject |            1 |          1 |            1 |        10 |

Note: mu4e normally limits display to, at most, the first 500 search
hits. In the "mu4e (1) All lines" results above, mu4e was forced to
display all search hits by setting 'M-x mu4e-headers-toggle-full-search’.

So, to generalize, mu4e is snappier than gnus/notmuch, unless we force
mu4e to display a lot of lines.  The biggest difference is the "open"
time, which, as demonstrated above, becomes significant when the maildir
contains a lot of messages.

SEARCH PERFORMANCE:

When searching, if the number of search hits is modest (<100), the
search/display time is similar for gnus/notmuch and mu4e. However, when
a search produces many hits (e.g., ~15,000), gnus takes about 10 sec to
display all of the results and mu4e takes 23 (once again setting M-x
mu4e-headers-toggle-full-search).

So the most noticeable oveall difference between gnus/notmuch and mu4e
is the time taken to initially "open" the maildir.

Regarding setup, I found mu4e easier with its single point of
configuration. In comparison, gnus/notmuch requires a compatible
"parallel" configuration of notmuch and gnus.

Bottom line:

1) gnus/notmuch/maildir works pretty well (at least on my computer)
   for 15,000 or so messages.

2) At 100,000 or messages, the gnus maildir startup delay is so tiresome
   that mu4e becomes truly compelling.

FWIW, I am currently a happy user of both mu4e and gnus/notmuch. I use
mu4e+mbsync to read multiple Gmail accounts. I use gnus/notmuch to
search and read mailing list archives that I have mirrored locally into
maildir.

- George

Re: searching local mail stores

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

Erik Colson <eco@ecocode.net> writes:

> Rainer M Krug <Rainer@krugs.de> writes:
>
>> I tried, but decided to stick with notmuch as I liked the possibility
>> to search from the commandline as well. Also, and please correct me if I
>> am wrong, the FTS plugin only searches in the IMAP folder and not all
>> folder (as notmuch does) - which is sometimes a disadvantage.
>
> you can 'G' from the server buffer to search all mailboxes from a
> dovecot server.

Hm - probably should try it again...

Thanks,

Rainer


>
> best

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]

Re: searching local mail stores

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1739 bytes --]

On 2016-01-05 17:15, Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Alan Schmitt <alan.schmitt@polytechnique.org> writes:
>
>> On 2016-01-05 09:43, Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> I think the problem is that you can't actually use notmuch with the
>>> nnimap backend.
>>
>> Ah, I see. I could switch to the nnmaildir backend, but I prefer to let
>> dovecot handle all the mail manipulation.
>>
>>> But I think the idea is: if you're using an IMAP server, you're supposed
>>> to be letting the server do the searching, and give you the results.
>>> Searching the filesystem directly is kind of "going behind the server's
>>> back". I don't actually know if that would cause any real problems, but
>>> the code is definitely set up with that assumption in mind.
>>>
>>> Maybe try the FTS dovecot plugin?
>>
>> I should really do that (it's in my long todo list). I first need to
>> convert my installation to use a single dovecot instance, though. (A
>> couple hours later …) OK, done. I'll look into FTS (probably Lucene)
>> a little bit later.
>>
>> Thanks for the suggestion,
>
> Well I'm not saying you've got to do it this way! I'm not trying to
> advocate one method over the other, as I really don't have a
> comprehensive understanding of how all this works. I don't want to lead
> anyone astray...

I want to try it at some point, to compare to what I've been doing
(using either notmuch or mu4e). And I really needed to get my dovecot
config straightened out, which has just happened. So do not worry, I'm
going astray all by myself ;)

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Athmospheric CO₂ (Updated December 28, 2015, Mauna Loa Obs.): 402.60 ppm

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: searching local mail stores

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1252 bytes --]

On 2016-01-05 09:38, Rainer M Krug <Rainer@krugs.de> writes:

>> But I think the idea is: if you're using an IMAP server, you're supposed
>> to be letting the server do the searching, and give you the results.
>> Searching the filesystem directly is kind of "going behind the server's
>> back". I don't actually know if that would cause any real problems, but
>> the code is definitely set up with that assumption in mind.
>
> That is exactly why I left GG as it is, and use notmuch search simply by
> using the command notmuch-search. You y=could also define a new shortcut
> for the function (which I think I did - but I have forgotten the
> shortcut. M-x notmuch-search is much easier to remember as I do not
> search that often).

I have also used notmuch-search quite a bit, but there is one aspect
I dislike: it's hard to jump back to gnus from a notmuch search. I have
found a function that tries to do it (similar to the gnus links of
orgmode) but sometimes it does not work (it does not find the message),
and if there are unread messages in the target folders, it makes the
most recent one read.

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Athmospheric CO₂ (Updated December 28, 2015, Mauna Loa Obs.): 402.60 ppm

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: searching local mail stores

[Eric Abrahamsen]

Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> On 2016-01-05 09:43, Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I think the problem is that you can't actually use notmuch with the
>> nnimap backend.
>
> Ah, I see. I could switch to the nnmaildir backend, but I prefer to let
> dovecot handle all the mail manipulation.
>
>> But I think the idea is: if you're using an IMAP server, you're supposed
>> to be letting the server do the searching, and give you the results.
>> Searching the filesystem directly is kind of "going behind the server's
>> back". I don't actually know if that would cause any real problems, but
>> the code is definitely set up with that assumption in mind.
>>
>> Maybe try the FTS dovecot plugin?
>
> I should really do that (it's in my long todo list). I first need to
> convert my installation to use a single dovecot instance, though. (A
> couple hours later …) OK, done. I'll look into FTS (probably Lucene)
> a little bit later.
>
> Thanks for the suggestion,

Well I'm not saying you've got to do it this way! I'm not trying to
advocate one method over the other, as I really don't have a
comprehensive understanding of how all this works. I don't want to lead
anyone astray...

Re: searching local mail stores

[Eric Abrahamsen]

Rainer M Krug <Rainer@krugs.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Alan Schmitt <alan.schmitt@polytechnique.org> writes:
>>
>>> Hello,
>>>
>>> (Sorry if you are seeing this as a duplicate. I tried to post it
>>> through the gmane group but I'm not seeing it, so here it is again
>>> directly to the mailing list.)
>>>
>>> On 2016-01-03 17:04, Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>>>> In .gnus I have added:
>>>>>>
>>>>>>   (nnir-search-engine notmuch)
>>>>>>
>>>>>> in my mail-method ((nnml "")).
>>>>>>
>>>>>> I search by going to my email-topic and pressing GG.
>>>>>
>>>>> What else needs to be setup to use notmuch for search?
>>>>>
>>>>> (There's some info on setup here for instance:
>>>>> http://roland.entierement.nu/blog/2010/09/08/gnus-dovecot-offlineimap-search-a-howto.html
>>>>> )
>>>>>
>>>>> Is that the sort of setup that needs doing first?
>>>>
>>>> This, but also ‘nnir-notmuch-remove-prefix’. Notmuch knows nothing about
>>>> Gnus, it just knows where matching files are on your filesystem. Gnus
>>>> only wants to know the filename relative to where it *already knows how
>>>> to find the message*. You customize that option in order to match the
>>>> two up. The option is basically copied from nnir-namazu-remove-prefix,
>>>> which has a more useful docstring...
>>>
>>> I'm giving this a try, and I'm having a problem: whatever I try, I get
>>> an error:
>>>
>>> gnus-select-newsgroup: Couldn't request group nnir:nnir-m2bn91ihar.fsf:
>>> Search produced empty results.
>>>
>>> If I run the notmuch command on the command line, I see many results.
>>> I tried to run edebug on the nnir-run-notmuch function, and it does
>>> something strange. It actually populates a temporary buffer with the
>>> results I see when running the command line, but then it seems to go
>>> through these results discarding all of them.
>>>
>>> #+begin_src emacs-lisp
>>> (while (not (eobp))
>>>   (setq filenam (buffer-substring-no-properties (line-beginning-position)
>>>                                                 (line-end-position))
>>>         artno (file-name-nondirectory filenam)
>>>         dirnam (file-name-directory filenam))
>>>   (forward-line 1)
>>>
>>>   ;; don't match directories
>>>   (when (string-match article-pattern artno)
>>>     (when (not (null dirnam))
>>>
>>>     ;; maybe limit results to matching groups.
>>>       (when (or (not groupspec)
>>>             (string-match groupspec dirnam))
>>>         (nnir-add-result dirnam artno "" prefix server artlist)))))
>>> #+end_src
>>>
>>> The problem is in the matching of the name of the file (something like
>>> "1451911994.M307856P59885.charm-ecran.irisa.fr,S=3270,W=3347:2,S")
>>> against the pattern "^[0-9]+$", which fails and the found message is
>>> ignored.
>>>
>>> Is this a bug in this function, or do I need additional configuration?
>>>
>>> Here are my server settings:
>>>
>>> (setq gnus-select-method 
>>>       '(nnimap 
>>>         "zimbra"
>>>         (nnimap-shell-program
>>>          "~/bin/dovecot_bin/imap -c ~/.dovecotzimbra.rc")
>>>         (nnimap-stream shell)
>>>         (nnimap-logout-timeout 1.0)
>>>         (nnir-search-engine notmuch)
>>>         (nnir-notmuch-remove-prefix "/Users/schmitta/.Maildir/zimbra/")))
>>>
>>> (setq gnus-secondary-select-methods 
>>>       '((nnimap 
>>>          "local"
>>>          (nnimap-address "localhost")
>>>          (nnimap-server-port 143)
>>>          (nnimap-stream network)
>>>          (nnir-search-engine notmuch)
>>>          (nnir-notmuch-remove-prefix "/Users/schmitta/.Maildir/local"))
>>>         ))
>>>
>>
>> I think the problem is that you can't actually use notmuch with the
>> nnimap backend. Earlier in that function, you see that the
>> "article-pattern" regexp is different depending on whether you're using
>> an nnmaildir backend or not. In fact, if it uses the regexp associated
>> with nnmaildir, it looks like your articles would probably match, and
>> come out in the results.
>>
>> I don't really understand the reasoning behind this -- I suppose you
>> could alter the function to always use the other regexp, and see if
>> anything else breaks.
>>
>> But I think the idea is: if you're using an IMAP server, you're supposed
>> to be letting the server do the searching, and give you the results.
>> Searching the filesystem directly is kind of "going behind the server's
>> back". I don't actually know if that would cause any real problems, but
>> the code is definitely set up with that assumption in mind.
>
> That is exactly why I left GG as it is, and use notmuch search simply by
> using the command notmuch-search. You y=could also define a new shortcut
> for the function (which I think I did - but I have forgotten the
> shortcut. M-x notmuch-search is much easier to remember as I do not
> search that often).
>
>>
>> Maybe try the FTS dovecot plugin?
>
> I tried, but decided to stick with notmuch as I liked the possibility
> to search from the commandline as well. Also, and please correct me if I
> am wrong, the FTS plugin only searches in the IMAP folder and not all
> folder (as notmuch does) - which is sometimes a disadvantage.

If you let FTS search your IMAP folders, and notmuch search your
nnmaildir folders, you can still search them together at the same time.
That's the nice thing about nnir. So for instance say you had a topic
that contained some groups from IMAP and some groups from nnmaildir, if
you hit GG on the topic name, the respective backends would each find
their messages, then nnir would put them all in one search group for
you.

But you're right that each backend would be using a separate search engine.

Re: searching local mail stores

[Alan Schmitt]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

On 2016-01-05 09:43, Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I think the problem is that you can't actually use notmuch with the
> nnimap backend.

Ah, I see. I could switch to the nnmaildir backend, but I prefer to let
dovecot handle all the mail manipulation.

> But I think the idea is: if you're using an IMAP server, you're supposed
> to be letting the server do the searching, and give you the results.
> Searching the filesystem directly is kind of "going behind the server's
> back". I don't actually know if that would cause any real problems, but
> the code is definitely set up with that assumption in mind.
>
> Maybe try the FTS dovecot plugin?

I should really do that (it's in my long todo list). I first need to
convert my installation to use a single dovecot instance, though. (A
couple hours later …) OK, done. I'll look into FTS (probably Lucene)
a little bit later.

Thanks for the suggestion,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Athmospheric CO₂ (Updated December 28, 2015, Mauna Loa Obs.): 402.60 ppm

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: searching local mail stores

[Erik Colson]

Rainer M Krug <Rainer@krugs.de> writes:

> I tried, but decided to stick with notmuch as I liked the possibility
> to search from the commandline as well. Also, and please correct me if I
> am wrong, the FTS plugin only searches in the IMAP folder and not all
> folder (as notmuch does) - which is sometimes a disadvantage.

you can 'G' from the server buffer to search all mailboxes from a
dovecot server.

best
-- 
erik colson

Re: searching local mail stores

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 5248 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Alan Schmitt <alan.schmitt@polytechnique.org> writes:
>
>> Hello,
>>
>> (Sorry if you are seeing this as a duplicate. I tried to post it
>> through the gmane group but I'm not seeing it, so here it is again
>> directly to the mailing list.)
>>
>> On 2016-01-03 17:04, Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>>>> In .gnus I have added:
>>>>>
>>>>>   (nnir-search-engine notmuch)
>>>>>
>>>>> in my mail-method ((nnml "")).
>>>>>
>>>>> I search by going to my email-topic and pressing GG.
>>>>
>>>> What else needs to be setup to use notmuch for search?
>>>>
>>>> (There's some info on setup here for instance:
>>>> http://roland.entierement.nu/blog/2010/09/08/gnus-dovecot-offlineimap-search-a-howto.html
>>>> )
>>>>
>>>> Is that the sort of setup that needs doing first?
>>>
>>> This, but also ‘nnir-notmuch-remove-prefix’. Notmuch knows nothing about
>>> Gnus, it just knows where matching files are on your filesystem. Gnus
>>> only wants to know the filename relative to where it *already knows how
>>> to find the message*. You customize that option in order to match the
>>> two up. The option is basically copied from nnir-namazu-remove-prefix,
>>> which has a more useful docstring...
>>
>> I'm giving this a try, and I'm having a problem: whatever I try, I get
>> an error:
>>
>> gnus-select-newsgroup: Couldn't request group nnir:nnir-m2bn91ihar.fsf:
>> Search produced empty results.
>>
>> If I run the notmuch command on the command line, I see many results.
>> I tried to run edebug on the nnir-run-notmuch function, and it does
>> something strange. It actually populates a temporary buffer with the
>> results I see when running the command line, but then it seems to go
>> through these results discarding all of them.
>>
>> #+begin_src emacs-lisp
>> (while (not (eobp))
>>   (setq filenam (buffer-substring-no-properties (line-beginning-position)
>>                                                 (line-end-position))
>>         artno (file-name-nondirectory filenam)
>>         dirnam (file-name-directory filenam))
>>   (forward-line 1)
>>
>>   ;; don't match directories
>>   (when (string-match article-pattern artno)
>>     (when (not (null dirnam))
>>
>>     ;; maybe limit results to matching groups.
>>       (when (or (not groupspec)
>>             (string-match groupspec dirnam))
>>         (nnir-add-result dirnam artno "" prefix server artlist)))))
>> #+end_src
>>
>> The problem is in the matching of the name of the file (something like
>> "1451911994.M307856P59885.charm-ecran.irisa.fr,S=3270,W=3347:2,S")
>> against the pattern "^[0-9]+$", which fails and the found message is
>> ignored.
>>
>> Is this a bug in this function, or do I need additional configuration?
>>
>> Here are my server settings:
>>
>> (setq gnus-select-method 
>>       '(nnimap 
>>         "zimbra"
>>         (nnimap-shell-program
>>          "~/bin/dovecot_bin/imap -c ~/.dovecotzimbra.rc")
>>         (nnimap-stream shell)
>>         (nnimap-logout-timeout 1.0)
>>         (nnir-search-engine notmuch)
>>         (nnir-notmuch-remove-prefix "/Users/schmitta/.Maildir/zimbra/")))
>>
>> (setq gnus-secondary-select-methods 
>>       '((nnimap 
>>          "local"
>>          (nnimap-address "localhost")
>>          (nnimap-server-port 143)
>>          (nnimap-stream network)
>>          (nnir-search-engine notmuch)
>>          (nnir-notmuch-remove-prefix "/Users/schmitta/.Maildir/local"))
>>         ))
>>
>
> I think the problem is that you can't actually use notmuch with the
> nnimap backend. Earlier in that function, you see that the
> "article-pattern" regexp is different depending on whether you're using
> an nnmaildir backend or not. In fact, if it uses the regexp associated
> with nnmaildir, it looks like your articles would probably match, and
> come out in the results.
>
> I don't really understand the reasoning behind this -- I suppose you
> could alter the function to always use the other regexp, and see if
> anything else breaks.
>
> But I think the idea is: if you're using an IMAP server, you're supposed
> to be letting the server do the searching, and give you the results.
> Searching the filesystem directly is kind of "going behind the server's
> back". I don't actually know if that would cause any real problems, but
> the code is definitely set up with that assumption in mind.

That is exactly why I left GG as it is, and use notmuch search simply by
using the command notmuch-search. You y=could also define a new shortcut
for the function (which I think I did - but I have forgotten the
shortcut. M-x notmuch-search is much easier to remember as I do not
search that often).

>
> Maybe try the FTS dovecot plugin?

I tried, but decided to stick with notmuch as I liked the possibility
to search from the commandline as well. Also, and please correct me if I
am wrong, the FTS plugin only searches in the IMAP folder and not all
folder (as notmuch does) - which is sometimes a disadvantage.

Cheers,

Rainer

>
> Eric
>
>
>

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 454 bytes --]

Re: searching local mail stores

[Eric Abrahamsen]

Alan Schmitt <alan.schmitt@polytechnique.org> writes:

> Hello,
>
> (Sorry if you are seeing this as a duplicate. I tried to post it
> through the gmane group but I'm not seeing it, so here it is again
> directly to the mailing list.)
>
> On 2016-01-03 17:04, Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>>> In .gnus I have added:
>>>>
>>>>   (nnir-search-engine notmuch)
>>>>
>>>> in my mail-method ((nnml "")).
>>>>
>>>> I search by going to my email-topic and pressing GG.
>>>
>>> What else needs to be setup to use notmuch for search?
>>>
>>> (There's some info on setup here for instance:
>>> http://roland.entierement.nu/blog/2010/09/08/gnus-dovecot-offlineimap-search-a-howto.html
>>> )
>>>
>>> Is that the sort of setup that needs doing first?
>>
>> This, but also ‘nnir-notmuch-remove-prefix’. Notmuch knows nothing about
>> Gnus, it just knows where matching files are on your filesystem. Gnus
>> only wants to know the filename relative to where it *already knows how
>> to find the message*. You customize that option in order to match the
>> two up. The option is basically copied from nnir-namazu-remove-prefix,
>> which has a more useful docstring...
>
> I'm giving this a try, and I'm having a problem: whatever I try, I get
> an error:
>
> gnus-select-newsgroup: Couldn't request group nnir:nnir-m2bn91ihar.fsf:
> Search produced empty results.
>
> If I run the notmuch command on the command line, I see many results.
> I tried to run edebug on the nnir-run-notmuch function, and it does
> something strange. It actually populates a temporary buffer with the
> results I see when running the command line, but then it seems to go
> through these results discarding all of them.
>
> #+begin_src emacs-lisp
> (while (not (eobp))
>   (setq filenam (buffer-substring-no-properties (line-beginning-position)
>                                                 (line-end-position))
>         artno (file-name-nondirectory filenam)
>         dirnam (file-name-directory filenam))
>   (forward-line 1)
>
>   ;; don't match directories
>   (when (string-match article-pattern artno)
>     (when (not (null dirnam))
>
>     ;; maybe limit results to matching groups.
>       (when (or (not groupspec)
>             (string-match groupspec dirnam))
>         (nnir-add-result dirnam artno "" prefix server artlist)))))
> #+end_src
>
> The problem is in the matching of the name of the file (something like
> "1451911994.M307856P59885.charm-ecran.irisa.fr,S=3270,W=3347:2,S")
> against the pattern "^[0-9]+$", which fails and the found message is
> ignored.
>
> Is this a bug in this function, or do I need additional configuration?
>
> Here are my server settings:
>
> (setq gnus-select-method 
>       '(nnimap 
>         "zimbra"
>         (nnimap-shell-program
>          "~/bin/dovecot_bin/imap -c ~/.dovecotzimbra.rc")
>         (nnimap-stream shell)
>         (nnimap-logout-timeout 1.0)
>         (nnir-search-engine notmuch)
>         (nnir-notmuch-remove-prefix "/Users/schmitta/.Maildir/zimbra/")))
>
> (setq gnus-secondary-select-methods 
>       '((nnimap 
>          "local"
>          (nnimap-address "localhost")
>          (nnimap-server-port 143)
>          (nnimap-stream network)
>          (nnir-search-engine notmuch)
>          (nnir-notmuch-remove-prefix "/Users/schmitta/.Maildir/local"))
>         ))
>

I think the problem is that you can't actually use notmuch with the
nnimap backend. Earlier in that function, you see that the
"article-pattern" regexp is different depending on whether you're using
an nnmaildir backend or not. In fact, if it uses the regexp associated
with nnmaildir, it looks like your articles would probably match, and
come out in the results.

I don't really understand the reasoning behind this -- I suppose you
could alter the function to always use the other regexp, and see if
anything else breaks.

But I think the idea is: if you're using an IMAP server, you're supposed
to be letting the server do the searching, and give you the results.
Searching the filesystem directly is kind of "going behind the server's
back". I don't actually know if that would cause any real problems, but
the code is definitely set up with that assumption in mind.

Maybe try the FTS dovecot plugin?

Eric

Re: searching local mail stores

[Alan Schmitt]

[-- Attachment #1.1.1: Type: text/plain, Size: 3373 bytes --]

Hello,

(Sorry if you are seeing this as a duplicate. I tried to post it
through the gmane group but I'm not seeing it, so here it is again
directly to the mailing list.)

On 2016-01-03 17:04, Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>>> In .gnus I have added:
>>>
>>>   (nnir-search-engine notmuch)
>>>
>>> in my mail-method ((nnml "")).
>>>
>>> I search by going to my email-topic and pressing GG.
>>
>> What else needs to be setup to use notmuch for search?
>>
>> (There's some info on setup here for instance:
>> http://roland.entierement.nu/blog/2010/09/08/gnus-dovecot-offlineimap-search-a-howto.html
>> )
>>
>> Is that the sort of setup that needs doing first?
>
> This, but also ‘nnir-notmuch-remove-prefix’. Notmuch knows nothing about
> Gnus, it just knows where matching files are on your filesystem. Gnus
> only wants to know the filename relative to where it *already knows how
> to find the message*. You customize that option in order to match the
> two up. The option is basically copied from nnir-namazu-remove-prefix,
> which has a more useful docstring...

I'm giving this a try, and I'm having a problem: whatever I try, I get
an error:

gnus-select-newsgroup: Couldn't request group nnir:nnir-m2bn91ihar.fsf:
Search produced empty results.

If I run the notmuch command on the command line, I see many results.
I tried to run edebug on the nnir-run-notmuch function, and it does
something strange. It actually populates a temporary buffer with the
results I see when running the command line, but then it seems to go
through these results discarding all of them.

#+begin_src emacs-lisp
(while (not (eobp))
  (setq filenam (buffer-substring-no-properties (line-beginning-position)
                                                (line-end-position))
        artno (file-name-nondirectory filenam)
        dirnam (file-name-directory filenam))
  (forward-line 1)

  ;; don't match directories
  (when (string-match article-pattern artno)
    (when (not (null dirnam))

    ;; maybe limit results to matching groups.
      (when (or (not groupspec)
            (string-match groupspec dirnam))
        (nnir-add-result dirnam artno "" prefix server artlist)))))
#+end_src

The problem is in the matching of the name of the file (something like
"1451911994.M307856P59885.charm-ecran.irisa.fr,S=3270,W=3347:2,S")
against the pattern "^[0-9]+$", which fails and the found message is
ignored.

Is this a bug in this function, or do I need additional configuration?

Here are my server settings:

(setq gnus-select-method 
      '(nnimap 
        "zimbra"
        (nnimap-shell-program
         "~/bin/dovecot_bin/imap -c ~/.dovecotzimbra.rc")
        (nnimap-stream shell)
        (nnimap-logout-timeout 1.0)
        (nnir-search-engine notmuch)
        (nnir-notmuch-remove-prefix "/Users/schmitta/.Maildir/zimbra/")))

(setq gnus-secondary-select-methods 
      '((nnimap 
         "local"
         (nnimap-address "localhost")
         (nnimap-server-port 143)
         (nnimap-stream network)
         (nnir-search-engine notmuch)
         (nnir-notmuch-remove-prefix "/Users/schmitta/.Maildir/local"))
        ))

Thanks,

Alan

-- 
OpenPGP Key ID : 040D0A3B4ED2E5C7
Athmospheric CO₂ (Updated December 28, 2015, Mauna Loa Obs.): 402.60 ppm

[-- Attachment #1.1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 472 bytes --]

Re: searching local mail stores

[myglc2]

asjo@koldfront.dk (Adam Sjøgren) writes:

> myglc2 writes:
>
>> OTOH, gnus' notmuch INFO section doesn't mention it either. LOL LOL
>
> Uhm, the notmuch sub-section of the Gnus manual is *in* the nnir
> section?
>
>  · http://gnus.org/manual/gnus_108.html#Setting-up-nnir

Thanks and Oops.

Sorry this is long, but I have been looking at gnus search for a couple
weeks and obviously there is something fundamental I am missing.

When I read the top levels of gnus search INFO ...

+---------------------
| 8.1.1 What is nnir?
| -------------------
| 
| ‘nnir’ is a Gnus interface to a number of tools for searching through
| mail and news repositories.  Different backends (like ‘nnimap’ and
| ‘nntp’) work with different tools (called "engines" in ‘nnir’ lingo),
| but all use the same basic search interface.
| 
|    The ‘nnimap’ and ‘gmane’ search engines should work with no
| configuration.  Other engines require a local index that needs to be
| created and maintained outside of Gnus.
+---------------------
+---------------------
| 8.1.3 Setting up nnir
| ---------------------
| 
| To set up nnir you may need to do some prep work.  Firstly, you may need
| to configure the search engines you plan to use.  Some of them, like
| ‘imap’ and ‘gmane’, need no special configuration.  Others, like
| ‘namazu’ and ‘swish’, require configuration as described below.
| Secondly, you need to associate a search engine with a server or a
| backend.
| 
|    If you just want to use the ‘imap’ engine to search ‘nnimap’ servers,
| and the ‘gmane’ engine to search ‘gmane’ then you don’t have to do
| anything.  But you might want to read the details of the query language
| anyway.
+---------------------
+---------------------
| 
| 8.2 nnmairix
| ============
| 
| This paragraph describes how to set up mairix and the back end
| ‘nnmairix’ for indexing and searching your mail from within Gnus.
| Additionally, you can create permanent “smart” groups which are bound to
| mairix searches and are automatically updated.
+---------------------
+---------------------
| 8.2.1 About mairix
| ------------------
| 
| Mairix is a tool for indexing and searching words in locally stored
| mail.  It was written by Richard Curnow and is licensed under the GPL.
| Mairix comes with most popular GNU/Linux distributions, but it also runs
| under Windows (with cygwin), Mac OS X and Solaris.  The homepage can be
| found at <http://www.rpcurnow.force9.co.uk/mairix/index.html>
| 
|    Though mairix might not be as flexible as other search tools like
| swish++ or namazu, which you can use via the ‘nnir’ back end, it has the
| prime advantage of being incredibly fast.  On current systems, it can
| easily search through headers and message bodies of thousands and
| thousands of mails in well under a second.  Building the database
| necessary for searching might take a minute or two, but only has to be
| done once fully.  Afterwards, the updates are done incrementally and
| therefore are really fast, too.  Additionally, mairix is very easy to
| set up.
| 
+---------------------

... I get these impressions:

1) mairix, naamzu, notmuch, swish++ and swish-e all search local mail
   stores by building and searching an index

   Pretty sure this is correct. 

2) mairix is faster and available on windows, but the others aren't

   namazu, swish++ and swish-e have been ported to windows. So is this
   point true, or is the doc out of date?
   
3) mairix is not as flexible

   What does this mean?

3) mairix is faster

   Since they all use indexes, why does speed differ significantly?

4) mairix creates permanent "smart" groups

   "smart" sounds good but I can't tell what it means. Further down in
   the doc it says that nnir search results are ephemeral.

   So is mairix "smart" because the results are permanent, or is there
   something else that makes it more smart?

5) mairix is easier to set up

   Is this really true? 

Are there any other fundamental differences I am missing?

Many thanks, - george

Re: searching local mail stores

[Adam Sjøgren]

Benjamin writes:

> asjo@koldfront.dk (Adam Sjøgren) writes:

>>> Do you use the nnir-integration?

>> In .gnus I have added:

>>   (nnir-search-engine notmuch)

>> in my mail-method ((nnml "")).

>> I search by going to my email-topic and pressing GG.

> What else needs to be setup to use notmuch for search?

Ah, sorry - notmuch needs to have indexed your emails, as well. I store
emails locally (in nnml:), so that is very easy; my ~/.notmuch-config
contains:

  [database]
  path=/home/asjo/Mail
  [user]
  name=Adam Sjøgren
  primary_email=asjo@koldfront.dk
  other_email=asjo@asjo.org;asjo@kammeratadam.dk;adam@kammeratadam.dk;asjo@diku.dk
  [new]
  tags=
  ignore=incoming;abusereply;crashplan;drafts;gmane-autoauthorizer;google-alerts;gratisdns;littlebrother;maradns;mon;namazu;norwegian;spam;spam-archive;spamcop;virus
  [maildir]
  synchronize_flags=false

(I don't use maildir).


  Best regards,

    Adam

-- 
 "You don't seem very afraid."                                Adam Sjøgren
 "You don't seem very frightening."                      asjo@koldfront.dk

Re: searching local mail stores

[Eric Abrahamsen]

Benjamin Slade <slade@jnanam.net> writes:

> asjo@koldfront.dk (Adam Sjøgren) writes:
>>> Do you use the nnir-integration?
>>
>> In .gnus I have added:
>>
>>   (nnir-search-engine notmuch)
>>
>> in my mail-method ((nnml "")).
>>
>> I search by going to my email-topic and pressing GG.
>
> What else needs to be setup to use notmuch for search?
>
> (There's some info on setup here for instance:
> http://roland.entierement.nu/blog/2010/09/08/gnus-dovecot-offlineimap-search-a-howto.html
> )
>
> Is that the sort of setup that needs doing first?

This, but also ‘nnir-notmuch-remove-prefix’. Notmuch knows nothing about
Gnus, it just knows where matching files are on your filesystem. Gnus
only wants to know the filename relative to where it *already knows how
to find the message*. You customize that option in order to match the
two up. The option is basically copied from nnir-namazu-remove-prefix,
which has a more useful docstring...

Re: searching local mail stores

[Benjamin Slade]

asjo@koldfront.dk (Adam Sjøgren) writes:
>> Do you use the nnir-integration?
>
> In .gnus I have added:
>
>   (nnir-search-engine notmuch)
>
> in my mail-method ((nnml "")).
>
> I search by going to my email-topic and pressing GG.

What else needs to be setup to use notmuch for search?

(There's some info on setup here for instance:
http://roland.entierement.nu/blog/2010/09/08/gnus-dovecot-offlineimap-search-a-howto.html
)

Is that the sort of setup that needs doing first?

Re: searching local mail stores

[Adam Sjøgren]

myglc2 writes:

> OTOH, gnus' notmuch INFO section doesn't mention it either. LOL LOL

Uhm, the notmuch sub-section of the Gnus manual is *in* the nnir
section?

 · http://gnus.org/manual/gnus_108.html#Setting-up-nnir


  Best regards,

    Adam

-- 
 "Accept failure gracefully."                                 Adam Sjøgren
                                                         asjo@koldfront.dk

Re: searching local mail stores

[myglc2]

asjo@koldfront.dk (Adam Sjøgren) writes:

> myglc2 writes:
>
>> When I said newest I meant development started more recently, but this
>> is not necessarily a good thing.
>
> Agreed.
>
>> Since hey both are actively developed, and use xapian (notmuch also
>> supports gmime),
>
> But mu4e is a MUA, right?

While it does look like mu4e 0.9.13 has an experimental MUA, in my
setup, as suggested by the current mu4e doc, mail is sent using ...

(setq message-send-mail-function 'smtpmail-send-it)

> I think of notmuch as primarily a search tool. I guess I am slightly
> misaligned here :-)

It seems to me that mu and notmuch are both primarily mail search tools.
Each has a variety of interfaces. Each has an emacs mode, respectively
named "mu4e" and "notmuch emacs interface". So far, so good.

>> I thought the performance would be similar and trying one would be
>> representative of the other.
>
> I didn't even think of using mu4e as a search backend for Gnus. Is
> anyone using it like that?

I would guess not. The mu4e 0.9.13 INFO makes no mention of nnir. The mu
general list has a July 2015 'nnir' post proposing that the notmuch nnir
backend be adapted for mu with no follow-up.

>> Do you use the nnir-integration?
>
> In .gnus I have added:
>
>   (nnir-search-engine notmuch)
>
> in my mail-method ((nnml "")).
>
> I search by going to my email-topic and pressing GG.

Great! I have got to try this.

The big surprise for me, after spending a few days reading about mu and
notmuch, is that you can use notmuch as a gnus search backend. This is
huge! Thank you for pointing it out.

It could be notmuch's best kept "gnus secret". The notmuch emacs page

https://notmuchmail.org/emacstips/

makes no mention of nnir or gnus search. The only mention of gnus is
quite a nice section on how to install and configure gnus-alias.

I looked on the notmuch general news list and found a 'nnir' post in
2013 saying it works great and would be nice to add to the notmuch emacs
tips page. LOL

OTOH, gnus' notmuch INFO section doesn't mention it either. LOL LOL

Many thanks,  - George

Re: searching local mail stores

[Adam Sjøgren]

myglc2 writes:

> When I said newest I meant development started more recently, but this
> is not necessarily a good thing.

Agreed.

> Since hey both are actively developed, and use xapian (notmuch also
> supports gmime),

But mu4e is a MUA, right?

I think of notmuch as primarily a search tool. I guess I am slightly
misaligned here :-)

> I thought the performance would be similar and trying one would be
> representative of the other.

I didn't even think of using mu4e as a search backend for Gnus. Is
anyone using it like that?

> Do you use the nnir-integration?

In .gnus I have added:

  (nnir-search-engine notmuch)

in my mail-method ((nnml "")).

I search by going to my email-topic and pressing GG.


  Best regards,

    Adam

-- 
 "It was called Tinderbox so we could make the joke,          Adam Sjøgren
  'the tree is on fire'. Puns are very important in      asjo@koldfront.dk
  naming tools."

Re: searching local mail stores

[myglc2]

asjo@koldfront.dk (Adam Sjøgren) writes:

> myglc2 writes:
>
>> I plan to set aside notmuch because it seems pretty similar to mu4e. 
>
> Why? It is the newest, most actively developed, based on Xapian, and
> works pretty well, at least in my experience.

When I said newest I meant development started more recently, but this
is not necessarily a good thing. Since hey both are actively developed,
and use xapian (notmuch also supports gmime), I thought the performance
would be similar and trying one would be representative of the other.

> If you use Gnus, you wouldn't use the Emacs-interface that notmuch comes
> with, just use the nnir-integration and the indexing and search
> capabilities of notmuch.

Oh, I didn't realize it has nnir-integration. Now I will have to check
notmuch out.

Do you use the nnir-integration?

> I'm not saying it fits your wishes the best, but I wouldn't dismiss it
> out of hand - it is worth checking out.
>
>
>   Best regards,
>
>     Adam, who used Namazu before, which also worked quite well.

Many thanks, George

Re: searching local mail stores

[Adam Sjøgren]

myglc2 writes:

> I plan to set aside notmuch because it seems pretty similar to mu4e. 

Why? It is the newest, most actively developed, based on Xapian, and
works pretty well, at least in my experience.

If you use Gnus, you wouldn't use the Emacs-interface that notmuch comes
with, just use the nnir-integration and the indexing and search
capabilities of notmuch.

I'm not saying it fits your wishes the best, but I wouldn't dismiss it
out of hand - it is worth checking out.


  Best regards,

    Adam, who used Namazu before, which also worked quite well.

-- 
 "So this is what being a morning person is like, I           Adam Sjøgren
  thought. It's like being 80 years old."                asjo@koldfront.dk

searching local mail stores

[myglc2]

I would like to index and search across my local mail stores, and
eventually all my other files. gnus is my primary mail and news
reader. I have used ...

gmail -- nnimap -- gnus

... for some years. I now plan to move all my mail to my local server so
that I can index and search it more quickly. I have recently set up ...

gmail -- mbsync -- dovecot+lucene -- nnimap -- gnus 

gmail -- mbsync -- Maildir -- mu4e

I am using mu4e (part of mu) as a stalking horse + benchmark because it
is new and actively developed. mu uses the xapian search engine which
could be a good search solution for my other files since it handles many
document types. mu4e's primary downside (for me) is that it is not
integrated with gnus.

I want to compare the performance of mu4e with some of the gnus search
schemes.  Looking at gnus INFO ...

8 Searching
8.1 nnir Searching with various engines.
8.1.3 Setting up nnir
8.1.3.2 The imap Engine
8.1.3.3 The gmane Engine
8.1.3.4 The swish++ Engine
8.1.3.5 The swish-e Engine
8.1.3.6 The namazu Engine
8.1.3.7 The notmuch Engine
8.2 nnmairix

... it was not obvious which I should try. So I searched the gmane
gnus.user and gnus.general news groups. I found these hit counts and
dates:

| search  | gnus.user | gnus.user | gnus.general | gnus.general | total |
| term    |      hits |     dates |         hits |        dates |       |
|---------+-----------+-----------+--------------+--------------+-------|
| swish++ |        11 |     02-06 |           73 |        01-13 |    84 |
| swishe  |        33 |     02-10 |           40 |        97-14 |    73 |
| namazu  |        56 |     02-15 |          196 |        01-15 |   252 |
| notmuch |        33 |     09-15 |          101 |        10-15 |   134 |
| mairix  |       100 |     04-12 |          140 |        06-15 |   240 |
| mu4e    |         4 |     12-15 |            6 |        12-15 |    10 |
|---------+-----------+-----------+--------------+--------------+-------|

I plan to set aside notmuch because it seems pretty similar to mu4e. 

It looks like mairix and namazu are in the lead and the swishes are
pretty far behind. Or maybe the swishes work flawlessly:)

So maybe I should get namazu and/or mairix working?

Would anyone care to comment on which of these I should try and why?

- george