blog post on gnus, dovecot, and lucene

blog post on gnus, dovecot, and lucene

[Eric Abrahamsen]

Hi,

I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
would blog about it. I finally did that:

http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html

The gist of it is how to move to an always-running dovecot daemon, fed
by isync, and incorporating lucene searches into that.

Enjoy!

Eric

Re: blog post on gnus, dovecot, and lucene

[Igor Sosa Mayor]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Hi,
>
> I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
> would blog about it. I finally did that:
>
> http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html
>
> The gist of it is how to move to an always-running dovecot daemon, fed
> by isync, and incorporating lucene searches into that.

interesting! Thanks!

-- 
:: Igor Sosa Mayor     :: joseleopoldo1792@gmail.com ::
:: GnuPG: 0x1C1E2890   :: http://www.gnupg.org/      ::
:: jabberid: rogorido  ::                            ::

Re: blog post on gnus, dovecot, and lucene

[Feng Shu]

Igor Sosa Mayor <joseleopoldo1792@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Hi,
>>
>> I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
>> would blog about it. I finally did that:
>>
>> http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html>
>> The gist of it is how to move to an always-running dovecot daemon, fed
>> by isync, and incorporating lucene searches into that.

I use gnus+dovecot+offlineimap+leafnod+rss2email

This is my configure: [[https://github.com/tumashu/emacs-helper]]

>
> interesting! Thanks!

-- 

Re: blog post on gnus, dovecot, and lucene

[Rasmus]

Hi,

Feng Shu <tumashu@gmail.com> writes:

> Igor Sosa Mayor <joseleopoldo1792@gmail.com> writes:
>>> I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
>>> would blog about it. I finally did that:
>>>
>>> http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html>
>>> The gist of it is how to move to an always-running dovecot daemon, fed
>>> by isync, and incorporating lucene searches into that.
>
> I use gnus+dovecot+offlineimap+leafnod+rss2email
>
> This is my configure: [[https://github.com/tumashu/emacs-helper]]

Thanks for sharing Eric and Feng.  I personally run Dovecot the way
Vincent suggests.  

The one thing I'm lacking is proper IMAP IDLE support to automatically
download new emails (not as a chronjob).  The one built into
offlineimap is okay-ish, but doesn't always see all changes so I still
have to run it manually.  I'm not sure how reliable it is when the
network changes or when I suspend my machine.

Does any of you have a reliable IDLE solution?

Cheers,
Rasmus

-- 
If you can mix business and politics wonderful things can happen!

Re: blog post on gnus, dovecot, and lucene

[Vincent Bernat]

 ❦  7 octobre 2014 13:43 +0800, Eric Abrahamsen <eric@ericabrahamsen.net> :

> I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
> would blog about it. I finally did that:
>
> http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html
>
> The gist of it is how to move to an always-running dovecot daemon, fed
> by isync, and incorporating lucene searches into that.

Note that you don't have to run dovecot as a server. You can run it as a
process. In mbsyncrc: 

Tunnel "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX"

In Gnus:

#+BEGIN_SRC elisp
(nnimap-stream shell)
(nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX")
#+END_SRC
-- 
Each module should do one thing well.
            - The Elements of Programming Style (Kernighan & Plauger)

Re: blog post on gnus, dovecot, and lucene

[Eric Abrahamsen]

Vincent Bernat <bernat@luffy.cx> writes:

>  ❦  7 octobre 2014 13:43 +0800, Eric Abrahamsen <eric@ericabrahamsen.net> :
>
>> I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
>> would blog about it. I finally did that:
>>
>> http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html
>>
>> The gist of it is how to move to an always-running dovecot daemon, fed
>> by isync, and incorporating lucene searches into that.
>
> Note that you don't have to run dovecot as a server. You can run it as a
> process. In mbsyncrc: 
>
> Tunnel "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX"
>
> In Gnus:
>
> #+BEGIN_SRC elisp
> (nnimap-stream shell)
> (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX")
> #+END_SRC

Yes, hence the section titled "The Problem" in that blog post :) Perhaps
I could have made that clearer!

So far as I can tell, if you're calling dovecot as a process, you can't
integrate lucene text indexing -- that only works when running dovecot
as a daemon.

In fact that was the whole impetus for the shift, and the blog post:
going from dovecot-the-process to dovecot-the-daemon, because of the
need for search indexing.

E

Re: blog post on gnus, dovecot, and lucene

[Vincent Bernat]

 ❦  8 octobre 2014 00:30 +0800, Eric Abrahamsen <eric@ericabrahamsen.net> :

>> Note that you don't have to run dovecot as a server. You can run it as a
>> process. In mbsyncrc: 
>>
>> Tunnel "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX"
>>
>> In Gnus:
>>
>> #+BEGIN_SRC elisp
>> (nnimap-stream shell)
>> (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX")
>> #+END_SRC
>
> Yes, hence the section titled "The Problem" in that blog post :) Perhaps
> I could have made that clearer!

Oh, it was pretty clear but I didn't read from the beginning because I
thought this was a post about reading mails offline with full text
search available! ;-)

> So far as I can tell, if you're calling dovecot as a process, you can't
> integrate lucene text indexing -- that only works when running dovecot
> as a daemon.
>
> In fact that was the whole impetus for the shift, and the blog post:
> going from dovecot-the-process to dovecot-the-daemon, because of the
> need for search indexing.

I didn't notice that. I added that to my dovecot.conf:

plugin {
  fts = lucene
  fts_lucene = whitespace_chars=@.
}

And the search became faster. However, I usually don't use FTS (only
search on recipient or title). If I check right now, I notice that the
indexes are not up-to-date. So, you may be right.
-- 
Keep it simple to make it faster.
            - The Elements of Programming Style (Kernighan & Plauger)

Re: blog post on gnus, dovecot, and lucene

[Eric Abrahamsen]

Vincent Bernat <bernat@luffy.cx> writes:

>  ❦  8 octobre 2014 00:30 +0800, Eric Abrahamsen <eric@ericabrahamsen.net> :
>
>>> Note that you don't have to run dovecot as a server. You can run it as a
>>> process. In mbsyncrc: 
>>>
>>> Tunnel "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX"
>>>
>>> In Gnus:
>>>
>>> #+BEGIN_SRC elisp
>>> (nnimap-stream shell)
>>> (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/XXXXX")
>>> #+END_SRC
>>
>> Yes, hence the section titled "The Problem" in that blog post :) Perhaps
>> I could have made that clearer!
>
> Oh, it was pretty clear but I didn't read from the beginning because I
> thought this was a post about reading mails offline with full text
> search available! ;-)
>
>> So far as I can tell, if you're calling dovecot as a process, you can't
>> integrate lucene text indexing -- that only works when running dovecot
>> as a daemon.
>>
>> In fact that was the whole impetus for the shift, and the blog post:
>> going from dovecot-the-process to dovecot-the-daemon, because of the
>> need for search indexing.
>
> I didn't notice that. I added that to my dovecot.conf:
>
> plugin {
>   fts = lucene
>   fts_lucene = whitespace_chars=@.
> }
>
> And the search became faster. However, I usually don't use FTS (only
> search on recipient or title). If I check right now, I notice that the
> indexes are not up-to-date. So, you may be right.

But it's pretty interesting that that did something -- I wasn't sure
that calling dovecot as a process would even invoke the configuration
files at all. But you don't think the indexes were updated, huh?

It might be worth amending the blog post at some point. I've asked about
this stuff on the dovecot mailing list, but no one seemed to know.

E

Re: blog post on gnus, dovecot, and lucene

[Vincent Bernat]

 ❦  8 octobre 2014 22:15 +0800, Eric Abrahamsen <eric@ericabrahamsen.net> :

>> I didn't notice that. I added that to my dovecot.conf:
>>
>> plugin {
>>   fts = lucene
>>   fts_lucene = whitespace_chars=@.
>> }
>>
>> And the search became faster. However, I usually don't use FTS (only
>> search on recipient or title). If I check right now, I notice that the
>> indexes are not up-to-date. So, you may be right.
>
> But it's pretty interesting that that did something -- I wasn't sure
> that calling dovecot as a process would even invoke the configuration
> files at all. But you don't think the indexes were updated, huh?
>
> It might be worth amending the blog post at some point. I've asked about
> this stuff on the dovecot mailing list, but no one seemed to know.

I have done a doveadm fts rescan to get those indexes updated. So, maybe
the solution could be a cronjob?
-- 
 /* Thanks to Rob `CmdrTaco' Malda for not influencing this code in any
  * way.
  */
        2.4.3 linux/net/core/netfilter.c

Re: blog post on gnus, dovecot, and lucene

[Eric Abrahamsen]

Vincent Bernat <bernat@luffy.cx> writes:

>  ❦  8 octobre 2014 22:15 +0800, Eric Abrahamsen <eric@ericabrahamsen.net> :
>
>>> I didn't notice that. I added that to my dovecot.conf:
>>>
>>> plugin {
>>>   fts = lucene
>>>   fts_lucene = whitespace_chars=@.
>>> }
>>>
>>> And the search became faster. However, I usually don't use FTS (only
>>> search on recipient or title). If I check right now, I notice that the
>>> indexes are not up-to-date. So, you may be right.
>>
>> But it's pretty interesting that that did something -- I wasn't sure
>> that calling dovecot as a process would even invoke the configuration
>> files at all. But you don't think the indexes were updated, huh?
>>
>> It might be worth amending the blog post at some point. I've asked about
>> this stuff on the dovecot mailing list, but no one seemed to know.
>
> I have done a doveadm fts rescan to get those indexes updated. So, maybe
> the solution could be a cronjob?

Right, that should work fine. At that point, though, I'm inclined just
to have dovecot run as a daemon...

Re: blog post on gnus, dovecot, and lucene

[Rainer M Krug]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Vincent Bernat <bernat@luffy.cx> writes:
>
>>  ❦  8 octobre 2014 22:15 +0800, Eric Abrahamsen <eric@ericabrahamsen.net> :
>>
>>>> I didn't notice that. I added that to my dovecot.conf:
>>>>
>>>> plugin {
>>>>   fts = lucene
>>>>   fts_lucene = whitespace_chars=@.
>>>> }
>>>>
>>>> And the search became faster. However, I usually don't use FTS (only
>>>> search on recipient or title). If I check right now, I notice that the
>>>> indexes are not up-to-date. So, you may be right.
>>>
>>> But it's pretty interesting that that did something -- I wasn't sure
>>> that calling dovecot as a process would even invoke the configuration
>>> files at all. But you don't think the indexes were updated, huh?
>>>
>>> It might be worth amending the blog post at some point. I've asked about
>>> this stuff on the dovecot mailing list, but no one seemed to know.
>>
>> I have done a doveadm fts rescan to get those indexes updated. So, maybe
>> the solution could be a cronjob?

Instead of a cronjob, one could run it after the sync with offlinesync
(I am sure mbsync can do it as well) as a hook which is executed
automatically. But there is still the issue of updating after moving
messages from gnus. 

>
> Right, that should work fine. At that point, though, I'm inclined just
> to have dovecot run as a daemon...

I think the daemon is the better way to do it.

Cheers,

Rainer

>
>
>

-- 
Rainer M. Krug
email: Rainer<at>krugs<dot>de
PGP: 0x0F52F982

[-- Attachment #2: Type: application/pgp-signature, Size: 494 bytes --]

Re: blog post on gnus, dovecot, and lucene

[Eric Abrahamsen]

On 10/07/14 09:18 AM, Rainer M Krug wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Hi,
>>
>> I mentioned a bit ago that I had shifted my gnus/dovecot setup, and
>> would blog about it. I finally did that:
>>
>> http://ericabrahamsen.net/tech/2014/oct/gnus-dovecot-lucene.html
>>
>> The gist of it is how to move to an always-running dovecot daemon, fed
>> by isync, and incorporating lucene searches into that.
>
> Nice - thanks.
>
> The idea of using virtual users under dovecot is intriguing - it makes
> all the name translations I am doing in offlineimap to keep my accounts
> in different folder structures unneccessary. Could you please elaborrate
> a little bit on your setup in this regard and where the emails are
> actually stored? I might be implementing th same approach because it
> sounds so simple!

I can't claim to understand the entire configuration procedure, as I was
just diving "deep enough" into dovecot, but the trick is this bit from
the dovecot.conf file:

passdb {
       driver = passwd-file
       args = /etc/dovecot/passwd
}

userdb {
       driver = static
       args = uid=eric gid=users home=/home/eric/.mail/%d/%n
       default_fields = mail=maildir:/home/eric/.mail/%d/%n/mail
}


Particularly the userdb section. I understand this to be saying: "each
virtual user should be run as uid "eric" and gid "users", and each user
will have their home directory under /home/eric/.mail, in a directory
that looks like domainname/username. Furthermore, the mail for each user
will be stored in maildir format, in a mail/ subdirectory in that user's
home directory." It's like making viritual home directories for each
virtual user.

So now my directory structure looks like:

~/.mail/ericabrahamsen.net/
  └── eric
      └── mail
          ├── cur
          ├── dovecot.index
          [ more dovecot stuff]
          ├── lucene-indexes
          ├── new
          ├── subscriptions
          └── tmp
~/.mail/paper-republic.org/
  ├── eric
  │  └── mail
  │      ├── cur
  │      ├── dovecot.index
  │       [ etc ]
  │      ├── lucene-indexes
  │      ├── new
  │      ├── subscriptions
  │      └── tmp
  └── info
      └── mail
          ├── cur
          ├── dovecot.index
          [ more dovecot stuff]
          ├── lucene-indexes
          ├── new
          ├── subscriptions
          └── tmp

So in cases where I have more than one user at the same domain name,
both users are nested under the domain name directory.

In hindsight the extra mail/ directory was probably unnecessary, I had
some idea that other files might go above that mail/ directory.

Hope that helps,
Eric