nnimap backend performances ?

nnimap backend performances ?

[Xavier Maillard]

Hi,

Currently I am fetching my incoming mails using a fantastic tool:
offlineimap. Mails are then deserved by a local imap server (dovecot) on which
Gnus is connected. So far so good.

As far as I can remeber, I do this from the age of stone principally because,
nnimap was considered pretty slow and also because I was nomade and it was
comfortable to do this.

Today, is it considered harmful to fetch my mail in "direct connect" to my
remote imap server ?

-- Xavier.

Re: nnimap backend performances ?

[Eric Abrahamsen]

Xavier Maillard <lists.emacs.gnus@xavier.maillard.im> writes:

> Hi,
>
> Currently I am fetching my incoming mails using a fantastic tool:
> offlineimap. Mails are then deserved by a local imap server (dovecot) on which
> Gnus is connected. So far so good.
>
> As far as I can remeber, I do this from the age of stone principally because,
> nnimap was considered pretty slow and also because I was nomade and it was
> comfortable to do this.
>
> Today, is it considered harmful to fetch my mail in "direct connect" to my
> remote imap server ?

My understanding is that the nnimap backend was re-written quite
significantly two or three years ago (?). At any rate, the people who
used the first iteration and were horrified now seem mostly happy with
the new version, YMMV. I used to use isync and local dovecot, now use
the direct connection, and it doesn't kill me. It isn't great (I'm in
China), but it doesn't kill me.

eric

Re: nnimap backend performances ?

[Xavier Maillard]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Xavier Maillard <lists.emacs.gnus@xavier.maillard.im> writes:
>
>> Hi,
>>
>> Currently I am fetching my incoming mails using a fantastic tool:
>> offlineimap. Mails are then deserved by a local imap server (dovecot) on which
>> Gnus is connected. So far so good.
>>
>> As far as I can remeber, I do this from the age of stone principally because,
>> nnimap was considered pretty slow and also because I was nomade and it was
>> comfortable to do this.
>>
>> Today, is it considered harmful to fetch my mail in "direct connect" to my
>> remote imap server ?
>
> My understanding is that the nnimap backend was re-written quite
> significantly two or three years ago (?). At any rate, the people who
> used the first iteration and were horrified now seem mostly happy with
> the new version, YMMV. I used to use isync and local dovecot, now use
> the direct connection, and it doesn't kill me. It isn't great (I'm in
> China), but it doesn't kill me.

Understand. SO this is definetely something I should test again and
reconsider.

THank you for the feedback.

-- Xavier.

Re: nnimap backend performances ?

[myglc2]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> My understanding is that the nnimap backend was re-written quite
> significantly two or three years ago (?). At any rate, the people who
> used the first iteration and were horrified now seem mostly happy with
> the new version, YMMV. I used to use isync and local dovecot, now use
> the direct connection, and it doesn't kill me. It isn't great (I'm in
> China), but it doesn't kill me.
>
> eric

So Eric, it sounds like you abandoned ...

https://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html

... which I was following & hit the "plain" auth issue you discussed
~1yr ago on the isync list.

Could you say more about your isycn-dovecot experience and why you moved
back to direct.

Many thanks --george

Re: nnimap backend performances ?

[Eric Abrahamsen]

On 12/29/15 14:25 PM, myglc2 wrote:
> The following message is a courtesy copy of an article
> that has been posted to gmane.emacs.gnus.general as well.
>
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> My understanding is that the nnimap backend was re-written quite
>> significantly two or three years ago (?). At any rate, the people who
>> used the first iteration and were horrified now seem mostly happy with
>> the new version, YMMV. I used to use isync and local dovecot, now use
>> the direct connection, and it doesn't kill me. It isn't great (I'm in
>> China), but it doesn't kill me.
>>
>> eric
>
> So Eric, it sounds like you abandoned ...
>
> https://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html

(I didn't see this message show up on the mailing list, but I'm
responding to the list anyway, maybe I just missed it.)

No, I haven't abandoned that setup! Or rather, I've moved halfway away
from it, but I'm going back.

I recently switched my two main email accounts from Google Apps to
self-hosted, and for the moment I'm having Gnus talk directly to the
remote server for the new accounts.

Basically I deleted the two Google Apps accounts, but left the
corresponding imap accounts on my local Dovecot. I didn't want to move
all the old mail into the new accounts, because there's a lot of it.

That means I've got two "legacy" accounts for two addresses, which only
exist locally, and two "new" accounts for the same two addresses, which
only exist on my remote server. At some point I'll explain to my local
Dovecot that these are actually four separate imap stores, and start
syncing the two new accounts locally, as well.

For the time being the two "legacy" accounts are still working just fine
with the local Dovecot arrangement, and I should say are much easier to
search than the new remote accounts. I want to move back mostly because
the connections are still too slow/unstable for my taste. It's not Gnus'
fault.

> ... which I was following & hit the "plain" auth issue you discussed
> ~1yr ago on the isync list.

Ugh, I went through a dark period of confusion and despair for a bit
there, when I was trying to learn too many new things all at the same
time. In fact, I believe the problem was partially solved by upgrading
isync (see the later patch posted to that isync thread). I'm now using
1.3.0, what are you using?

I need to update the blog post: the ~/.mbsyncrc file syntax changed for
the 1.3 version. There's another possibly important difference:

1) /etc/dovecot/dovecot.conf: In the userdb clause, on the "args" line,
I've added at the end:

username_format=%u

2) /etc/dovecot/passwd: The lines that used to look like

eric@ericabrahamsen.net:{PLAIN}passwurd

Now look like

eric@ericabrahamsen.net:{PLAIN}passwurd:eric@ericabrahamsen.net

I think the crux of the problem is that _authentication account_ and
_user account_ are actually separate concerns. Either the previous
version of isync was doing it wrong, or my dovecot config had to be
fixed to separate the two things correctly, or maybe both.

To be honest I've forgotten how it works, a bit, and that's the main
reason I haven't synced my two new accounts to the local dovecot.

I'll try to do that this weekend, and update that blog post! In the
meantime, I hope something in here is helpful to you.

Eric

Re: nnimap backend performances ?

[myglc2]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> On 12/29/15 14:25 PM, myglc2 wrote:
>>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> My understanding is that the nnimap backend was re-written quite
>>> significantly two or three years ago (?). At any rate, the people who
>>> used the first iteration and were horrified now seem mostly happy with
>>> the new version, YMMV. I used to use isync and local dovecot, now use
>>> the direct connection, and it doesn't kill me. It isn't great (I'm in
>>> China), but it doesn't kill me.
>>>
>>> eric
>>
>> So Eric, it sounds like you abandoned ...
>>
>> https://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html
>
>
> No, I haven't abandoned that setup! Or rather, I've moved halfway away
> from it, but I'm going back.
>
> I recently switched my two main email accounts from Google Apps to
> self-hosted, and for the moment I'm having Gnus talk directly to the
> remote server for the new accounts.
>
> Basically I deleted the two Google Apps accounts, but left the
> corresponding imap accounts on my local Dovecot. I didn't want to move
> all the old mail into the new accounts, because there's a lot of it.
>
> That means I've got two "legacy" accounts for two addresses, which only
> exist locally, and two "new" accounts for the same two addresses, which
> only exist on my remote server. At some point I'll explain to my local
> Dovecot that these are actually four separate imap stores, and start
> syncing the two new accounts locally, as well.
>
> For the time being the two "legacy" accounts are still working just fine
> with the local Dovecot arrangement, and I should say are much easier to
> search than the new remote accounts. I want to move back mostly because
> the connections are still too slow/unstable for my taste. It's not Gnus'
> fault.

This is really good to hear. Your "legacy" situation closely parallels
the archive I need for a work account that I am loosing.

>> ... which I was following & hit the "plain" auth issue you discussed
>> ~1yr ago on the isync list.
>
> Ugh, I went through a dark period of confusion and despair for a bit
> there, when I was trying to learn too many new things all at the same
> time. In fact, I believe the problem was partially solved by upgrading
> isync (see the later patch posted to that isync thread). I'm now using
> 1.3.0, what are you using?

isync 1.3.0 built from git://git.code.sf.net/p/isync/isync master 46e792c

> I need to update the blog post: the ~/.mbsyncrc file syntax changed for
> the 1.3 version. There's another possibly important difference:
>
> 1) /etc/dovecot/dovecot.conf: In the userdb clause, on the "args" line,
> I've added at the end:
>
> username_format=%u

Thanks, I had that from your reply on a later thread (quoted below)
showing your more recent dovecot config ...

***
From: Eric Abrahamsen <eric@ericabrahamsen.net>
Subject: Re: mail vs news
Newsgroups: gmane.emacs.gnus.general
To: ding@gnus.org
Date: Thu, 17 Dec 2015 19:13:40 +0800

Here's my output of "doveconf -n". This is all the configuration I've
got, apart from /etc/dovecot/passwd.

# 2.2.21 (5345f22): /etc/dovecot/dovecot.conf
# OS: Linux 4.2.5-1-ARCH i686  
auth_verbose = yes
disable_plaintext_auth = no
info_log_path = /var/log/dovecot-info.log
log_path = /var/log/dovecot.log
mail_plugins = "fts fts_lucene"
passdb {
  args = username_format=%u /etc/dovecot/passwd
  driver = passwd-file
}
plugin {
  fts = lucene
  fts_autoindex = yes
  fts_lucene = whitespace_chars=@.
}
protocols = imap
ssl = no
userdb {
  args = uid=eric gid=users home=/home/eric/.mail/%d/%n username_format=%u
  default_fields = mail=maildir:/home/eric/.mail/%d/%n/mail
  driver = static
}
***

That "doveconf -n" command is a beautiful thing! This also work for me
on dovecot 2.2.13 installed on debian 8 this way:

apt-get install dovecot-core dovecot-imapd dovecot-lucene

> 2) /etc/dovecot/passwd: The lines that used to look like
>
> eric@ericabrahamsen.net:{PLAIN}passwurd
>
> Now look like
>
> eric@ericabrahamsen.net:{PLAIN}passwurd:eric@ericabrahamsen.net

Nice. This is the puzzle piece I was missing. Now it works!

> I think the crux of the problem is that _authentication account_ and
> _user account_ are actually separate concerns. Either the previous
> version of isync was doing it wrong, or my dovecot config had to be
> fixed to separate the two things correctly, or maybe both.
>
> To be honest I've forgotten how it works, a bit, and that's the main
> reason I haven't synced my two new accounts to the local dovecot.
>
> I'll try to do that this weekend, and update that blog post! In the
> meantime, I hope something in here is helpful to you.

A huge help. Thank You.

FWIW, My goal is to sweep multiple gmail accounts into bullet-proof
archives with tools for searching and clustering several decades of
work. To get a handle on what is doable, I am experimenting with gnus
backends and search schemes, mu/mu4e, dovecot, and mbsync.

I would love to hear suggestions of other emacs-compatible solutions I
should try.

George

Re: nnimap backend performances ?

[Peter Davis]

Xavier Maillard <lists.emacs.gnus@xavier.maillard.im> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Xavier Maillard <lists.emacs.gnus@xavier.maillard.im> writes:
>>
>>> Hi,
>>>
>>> Currently I am fetching my incoming mails using a fantastic tool:
>>> offlineimap. Mails are then deserved by a local imap server (dovecot) on which
>>> Gnus is connected. So far so good.
>>>
>>> As far as I can remeber, I do this from the age of stone principally because,
>>> nnimap was considered pretty slow and also because I was nomade and it was
>>> comfortable to do this.
>>>
>>> Today, is it considered harmful to fetch my mail in "direct connect" to my
>>> remote imap server ?
>>
>> My understanding is that the nnimap backend was re-written quite
>> significantly two or three years ago (?). At any rate, the people who
>> used the first iteration and were horrified now seem mostly happy with
>> the new version, YMMV. I used to use isync and local dovecot, now use
>> the direct connection, and it doesn't kill me. It isn't great (I'm in
>> China), but it doesn't kill me.
>
> Understand. SO this is definetely something I should test again and
> reconsider.

I never got back to this, but I'm curious. I'm running Gnus 5.13. Does that mean I've got the re-written NNIMAP back end?

Thanks!
-pd

Re: nnimap backend performances ?

[Eric Abrahamsen]

Peter Davis <pfd@pfdstudio.com> writes:

> Xavier Maillard <lists.emacs.gnus@xavier.maillard.im> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Xavier Maillard <lists.emacs.gnus@xavier.maillard.im> writes:
>>>
>>>> Hi,
>>>>
>>>> Currently I am fetching my incoming mails using a fantastic tool:
>>>> offlineimap. Mails are then deserved by a local imap server (dovecot) on which
>>>> Gnus is connected. So far so good.
>>>>
>>>> As far as I can remeber, I do this from the age of stone principally because,
>>>> nnimap was considered pretty slow and also because I was nomade and it was
>>>> comfortable to do this.
>>>>
>>>> Today, is it considered harmful to fetch my mail in "direct connect" to my
>>>> remote imap server ?
>>>
>>> My understanding is that the nnimap backend was re-written quite
>>> significantly two or three years ago (?). At any rate, the people who
>>> used the first iteration and were horrified now seem mostly happy with
>>> the new version, YMMV. I used to use isync and local dovecot, now use
>>> the direct connection, and it doesn't kill me. It isn't great (I'm in
>>> China), but it doesn't kill me.
>>
>> Understand. SO this is definetely something I should test again and
>> reconsider.
>
> I never got back to this, but I'm curious. I'm running Gnus 5.13. Does that mean I've got the re-written NNIMAP back end?

It looks like the big re-write happened in Sep 2010, so earlier than I
thought, but there was also a pile of significant improvements in 2012
or thereabouts.

I don't actually know what version 5.13 signifies. Wikipedia says it was
bundled with Emacs 23.1 in 2009, but "wrapped up in early 2012 with
version 0.19". Do you have a minor version number, or some other way to
actually date the code?

Re: nnimap backend performances ?

[Eric Abrahamsen]

myglc2 <myglc2@gmail.com> writes:

[...]

>> 2) /etc/dovecot/passwd: The lines that used to look like
>>
>> eric@ericabrahamsen.net:{PLAIN}passwurd
>>
>> Now look like
>>
>> eric@ericabrahamsen.net:{PLAIN}passwurd:eric@ericabrahamsen.net
>
> Nice. This is the puzzle piece I was missing. Now it works!

Ah, I'm glad to hear that was the key -- that confirms my dim
recollection of what was going on. I'm still a bit hazy on what's
happening, but I think a good half hour with the docs will clear things
up, and I'll update the blog post once I've done that. Dovecot really is
very flexible, once you get your head around the principles.

>> I think the crux of the problem is that _authentication account_ and
>> _user account_ are actually separate concerns. Either the previous
>> version of isync was doing it wrong, or my dovecot config had to be
>> fixed to separate the two things correctly, or maybe both.
>>
>> To be honest I've forgotten how it works, a bit, and that's the main
>> reason I haven't synced my two new accounts to the local dovecot.
>>
>> I'll try to do that this weekend, and update that blog post! In the
>> meantime, I hope something in here is helpful to you.
>
> A huge help. Thank You.
>
> FWIW, My goal is to sweep multiple gmail accounts into bullet-proof
> archives with tools for searching and clustering several decades of
> work. To get a handle on what is doable, I am experimenting with gnus
> backends and search schemes, mu/mu4e, dovecot, and mbsync.
>
> I would love to hear suggestions of other emacs-compatible solutions I
> should try.

I might not be too helpful here, unfortunately. I've only ever used
gnus/dovecot/isync. As I said in that post, I think dovecot plus a fts
plugin works great, but I really don't like the search syntax on the
Gnus/nnir side -- I find it very cumbersome. If there were a nicer
syntax for searching, I think I'd be happy with this.

I could also imagine using nnmaildir -- it seems simpler, at least
conceptually, if all you want is to keep mails archived. In that case
you could probably use notmuch to index the mails, and set nnir to use
that notmuch installation as the search backend. If you don't need
accounts and authentication and everything for a simple local archive,
that should be enough. I'll bet the searching would be even faster, as
well.

E

Re: nnimap backend performances ?

[Peter Davis]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Peter Davis <pfd@pfdstudio.com> writes:
>
>> Xavier Maillard <lists.emacs.gnus@xavier.maillard.im> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> Xavier Maillard <lists.emacs.gnus@xavier.maillard.im> writes:
>>>>
>>>>> Hi,
>>>>>
>>>>> Currently I am fetching my incoming mails using a fantastic tool:
>>>>> offlineimap. Mails are then deserved by a local imap server (dovecot) on which
>>>>> Gnus is connected. So far so good.
>>>>>
>>>>> As far as I can remeber, I do this from the age of stone principally because,
>>>>> nnimap was considered pretty slow and also because I was nomade and it was
>>>>> comfortable to do this.
>>>>>
>>>>> Today, is it considered harmful to fetch my mail in "direct connect" to my
>>>>> remote imap server ?
>>>>
>>>> My understanding is that the nnimap backend was re-written quite
>>>> significantly two or three years ago (?). At any rate, the people who
>>>> used the first iteration and were horrified now seem mostly happy with
>>>> the new version, YMMV. I used to use isync and local dovecot, now use
>>>> the direct connection, and it doesn't kill me. It isn't great (I'm in
>>>> China), but it doesn't kill me.
>>>
>>> Understand. SO this is definetely something I should test again and
>>> reconsider.
>>
>> I never got back to this, but I'm curious. I'm running Gnus 5.13. Does that mean I've got the re-written NNIMAP back end?
>
> It looks like the big re-write happened in Sep 2010, so earlier than I
> thought, but there was also a pile of significant improvements in 2012
> or thereabouts.
>
> I don't actually know what version 5.13 signifies. Wikipedia says it was
> bundled with Emacs 23.1 in 2009, but "wrapped up in early 2012 with
> version 0.19". Do you have a minor version number, or some other way to
> actually date the code?

Not that I'm aware of, though I'd be happy to be corrected. I don't even see an nnimap.el or .elc on this machine, though it's certainly
working.

Thanks,
-pd

Re: nnimap backend performances ?

[Peter Davis]

Peter Davis <pfd@pfdstudio.com> writes:
>
> Not that I'm aware of, though I'd be happy to be corrected. I don't even see an nnimap.el or .elc on this machine, though it's certainly
> working.

Hmmm. I take it back. Spotlight on the Mac couldn't find it, but I did
with locate-library.

The .el says:

;; Copyright (C) 2010-2015 Free Software Foundation, Inc.

So I guess it's up to date.

Thanks!

-pd

Re: nnimap backend performances ?

[Eric Abrahamsen]

Peter Davis <pfd@pfdstudio.com> writes:

> Peter Davis <pfd@pfdstudio.com> writes:
>>
>> Not that I'm aware of, though I'd be happy to be corrected. I don't even see an nnimap.el or .elc on this machine, though it's certainly
>> working.
>
> Hmmm. I take it back. Spotlight on the Mac couldn't find it, but I did
> with locate-library.
>
> The .el says:
>
> ;; Copyright (C) 2010-2015 Free Software Foundation, Inc.
>
> So I guess it's up to date.

Ha, I should have thought of that!

Re: nnimap backend performances ?

[myglc2]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>> myglc2 <myglc2@gmail.com> writes:
>>
>> FWIW, My goal is to sweep multiple gmail accounts into bullet-proof
>> archives with tools for searching and clustering several decades of
>> work. To get a handle on what is doable, I am experimenting with gnus
>> backends and search schemes, mu/mu4e, dovecot, and mbsync.
>>
>> I would love to hear suggestions of other emacs-compatible solutions I
>> should try.
>
> I might not be too helpful here, unfortunately. I've only ever used
> gnus/dovecot/isync. As I said in that post, I think dovecot plus a fts
> plugin works great, but I really don't like the search syntax on the
> Gnus/nnir side -- I find it very cumbersome. If there were a nicer
> syntax for searching, I think I'd be happy with this.

Thanks, that is helpful Search syntax is important to me. Lord knows I
don't need any new obscure stuff to learn. Doesn't nnir simply pass the
search string to the backend.  If so, is the syntax you don't like
associated with nnimap, IMAP spec, or lucene?

I have a toy installation of dovecot running with a small number of
messages.  Search is very fast but article fetching is surprisingly
slow. Have you experienced anything like this?

> I could also imagine using nnmaildir -- it seems simpler, at least
> conceptually, if all you want is to keep mails archived. In that case
> you could probably use notmuch to index the mails, and set nnir to use
> that notmuch installation as the search backend. If you don't need
> accounts and authentication and everything for a simple local archive,
> that should be enough. I'll bet the searching would be even faster, as
> well.

I currently have mu4e set up, which uses the same indexer (xapian) as
notmuch with the same set of messages in Maildir. The search response
seems as fast, and article fetch seems much faster.

I plan to also set up notmuch and mairix. Then I will have:


gnus --    nnir  --   nnimap  --  dovecot  --  maildir
                      lucene                   messages
                      (index)

gnus --    nnir  --   notmuch --  maildir
                      xapian      messages
                      (index)

mu4e --     mu   --   maildir
          xapian      messages
          (index)

gnus --  mairix  --   maildir
          flex        messages
         (index)     smart group
                   (search cache)


Then I can inflate the size of the message stores and compare. Frankly,
it is hard to imagine why performance shouldn't be pretty similar across
the board.

As far as I can tell from reading and other helpful exchanges on the
list, the only major difference is that mairix caches a search result in
a so-called "smart" permanent group. This is implemented by a maildir
folder containing symbolic links to the actual messages.

In theory, the second time we preform the exact same search, it should
be faster. If we search for the same thing over and over it probably
will be faster. If most or our searches are unique, making a cache
probably will be slower.

If in practice we search for the same term over an over and marix lets
us just click on a group containing the result instead of typing the
search term, this could indeed be nicer even if getting the search
result is slower.

As far as syntax, I am assume that mu4e and notmuch use a xapian
syntax. Mairix uses the Flex lexical analyzer, so I assume it is
different.

Meanwhile I am trying to determine if there are other meaningful mairix
functional differences.  - George

Re: nnimap backend performances ?

[Eric Abrahamsen]

myglc2 <myglc2@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>> myglc2 <myglc2@gmail.com> writes:
>>>
>>> FWIW, My goal is to sweep multiple gmail accounts into bullet-proof
>>> archives with tools for searching and clustering several decades of
>>> work. To get a handle on what is doable, I am experimenting with gnus
>>> backends and search schemes, mu/mu4e, dovecot, and mbsync.
>>>
>>> I would love to hear suggestions of other emacs-compatible solutions I
>>> should try.
>>
>> I might not be too helpful here, unfortunately. I've only ever used
>> gnus/dovecot/isync. As I said in that post, I think dovecot plus a fts
>> plugin works great, but I really don't like the search syntax on the
>> Gnus/nnir side -- I find it very cumbersome. If there were a nicer
>> syntax for searching, I think I'd be happy with this.
>
> Thanks, that is helpful Search syntax is important to me. Lord knows I
> don't need any new obscure stuff to learn. Doesn't nnir simply pass the
> search string to the backend.  If so, is the syntax you don't like
> associated with nnimap, IMAP spec, or lucene?

Both nnimap and IMAP. Date-based IMAP searching in particular is hard to
use: I can never remember the keywords, or the proper date format, and
the results are often just wrong. It doesn't find messages that clearly
fall within the specified date range.

Nnimap is a little weird because it only lets you search on one header
by default. If you want to search on multiple headers, that's an extra
layer of querying. It's not a disaster, but it does make the process
more difficult.

> I have a toy installation of dovecot running with a small number of
> messages.  Search is very fast but article fetching is surprisingly
> slow. Have you experienced anything like this?

Yes, getting messages out of the server isn't super quick. That's why I
think plain old nnmaildir might be preferable for email archives.

The more I think about it, the more I'm leaning towards pulling my
archives out of Dovecot and just using nnmaildir/nnir/notmuch.

>> I could also imagine using nnmaildir -- it seems simpler, at least
>> conceptually, if all you want is to keep mails archived. In that case
>> you could probably use notmuch to index the mails, and set nnir to use
>> that notmuch installation as the search backend. If you don't need
>> accounts and authentication and everything for a simple local archive,
>> that should be enough. I'll bet the searching would be even faster, as
>> well.
>
> I currently have mu4e set up, which uses the same indexer (xapian) as
> notmuch with the same set of messages in Maildir. The search response
> seems as fast, and article fetch seems much faster.
>
> I plan to also set up notmuch and mairix. Then I will have:
>
>
> gnus --    nnir  --   nnimap  --  dovecot  --  maildir
>                       lucene                   messages
>                       (index)
>
> gnus --    nnir  --   notmuch --  maildir
>                       xapian      messages
>                       (index)
>
> mu4e --     mu   --   maildir
>           xapian      messages
>           (index)
>
> gnus --  mairix  --   maildir
>           flex        messages
>          (index)     smart group
>                    (search cache)
>
>
> Then I can inflate the size of the message stores and compare. Frankly,
> it is hard to imagine why performance shouldn't be pretty similar across
> the board.
>
> As far as I can tell from reading and other helpful exchanges on the
> list, the only major difference is that mairix caches a search result in
> a so-called "smart" permanent group. This is implemented by a maildir
> folder containing symbolic links to the actual messages.
>
> In theory, the second time we preform the exact same search, it should
> be faster. If we search for the same thing over and over it probably
> will be faster. If most or our searches are unique, making a cache
> probably will be slower.
>
> If in practice we search for the same term over an over and marix lets
> us just click on a group containing the result instead of typing the
> search term, this could indeed be nicer even if getting the search
> result is slower.
>
> As far as syntax, I am assume that mu4e and notmuch use a xapian
> syntax. Mairix uses the Flex lexical analyzer, so I assume it is
> different.
>
> Meanwhile I am trying to determine if there are other meaningful mairix
> functional differences.  - George

Notmuch also caches search results somehow (it's transparent, I'm not
sure how it works), though it doesn't have mairix's smart search groups.
I used mairix for a bit, several years ago, and thought the way it
creates groups was a bit awkward, but it's probably not a big deal.

Another consideration is non-ASCII searches. Not all the search backends
let you search for strings in funny encodings. IMAP requires you put the
strings in quotes, and at some point namazu didn't do non-ASCII searches
at all (or was it Mairix?). It's been a while now, so I don't want to
slander packages that may have upgraded in the meantime, but if that's
important to you it's something to look into.

Otherwise I think you're right -- there shouldn't be too much difference
in performance between the various backends.

E