isync plus dovecot?

isync plus dovecot?

[Eric Abrahamsen]

I'm finally trying to ditch nnml for mail reading, and move to imap.
Offlineimap plus dovecot seems to be a popular solution (I am offline a
lot), but I also saw a lot of complaints about offlineimap, and the
suggestion of isync/mbsync as a replacement.

Has anyone done that? I realize this isn't quite a gnus question, but
I'm having a bit of trouble wrapping my head around this, and google
isn't helping any.

Specifically, you tell offlineimap to deliver to dovecot with a line
like:

preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir

I can't find the equivalent command for mbsync. I don't know if it's
possible, and worse I don't really understand what's going on well
enough to really figure it out logically.

Is it enough just to have mbsync dump to a Maildir directly, and then
tell gnus to access that maildir via dovecot?

Any pointers very welcome!

Eric

Re: isync plus dovecot?

[Rainer M. Krug]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I'm finally trying to ditch nnml for mail reading, and move to imap.
> Offlineimap plus dovecot seems to be a popular solution (I am offline a
> lot), but I also saw a lot of complaints about offlineimap, and the
> suggestion of isync/mbsync as a replacement.

I am using offlineimap, and I also saw many complains (and had a few
problems myself and looked at isync briefly), but it is now running solidly, after I reduced the
number of emails in a folder (or tag, as I am syncing from  gmail to
localhost, also dovecot).

One reason which kept me at offlineimap is that it seems to be more
widely used then isync.

So I can only say it works for me.

Cheers,

Rainer


>
> Has anyone done that? I realize this isn't quite a gnus question, but
> I'm having a bit of trouble wrapping my head around this, and google
> isn't helping any.
>
> Specifically, you tell offlineimap to deliver to dovecot with a line
> like:
>
> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>
> I can't find the equivalent command for mbsync. I don't know if it's
> possible, and worse I don't really understand what's going on well
> enough to really figure it out logically.
>
> Is it enough just to have mbsync dump to a Maildir directly, and then
> tell gnus to access that maildir via dovecot?
>
> Any pointers very welcome!
>
> Eric
>
>
>
<#secure method=pgpmime mode=sign>

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

Re: isync plus dovecot?

[Eric Abrahamsen]

Rainer@krugs.de (Rainer M. Krug) writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I'm finally trying to ditch nnml for mail reading, and move to imap.
>> Offlineimap plus dovecot seems to be a popular solution (I am offline a
>> lot), but I also saw a lot of complaints about offlineimap, and the
>> suggestion of isync/mbsync as a replacement.
>
> I am using offlineimap, and I also saw many complains (and had a few
> problems myself and looked at isync briefly), but it is now running solidly, after I reduced the
> number of emails in a folder (or tag, as I am syncing from  gmail to
> localhost, also dovecot).
>
> One reason which kept me at offlineimap is that it seems to be more
> widely used then isync.
>
> So I can only say it works for me.

Thanks! Anecdotal evidence is definitely important. I'm also using
multiple gmail addresses, and have never bothered much with tags/folders
as I've always used a mail client. I'm currently pulling down 55,000
messages in one account's INBOX, do you think that's likely to cause me
problems?

Thanks again,
Eric

>
> Cheers,
>
> Rainer
>
>
>>
>> Has anyone done that? I realize this isn't quite a gnus question, but
>> I'm having a bit of trouble wrapping my head around this, and google
>> isn't helping any.
>>
>> Specifically, you tell offlineimap to deliver to dovecot with a line
>> like:
>>
>> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>>
>> I can't find the equivalent command for mbsync. I don't know if it's
>> possible, and worse I don't really understand what's going on well
>> enough to really figure it out logically.
>>
>> Is it enough just to have mbsync dump to a Maildir directly, and then
>> tell gnus to access that maildir via dovecot?
>>
>> Any pointers very welcome!
>>
>> Eric
>>
>>
>>
> <#secure method=pgpmime mode=sign>

Re: isync plus dovecot?

[Rainer M. Krug]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Rainer@krugs.de (Rainer M. Krug) writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> I'm finally trying to ditch nnml for mail reading, and move to imap.
>>> Offlineimap plus dovecot seems to be a popular solution (I am offline a
>>> lot), but I also saw a lot of complaints about offlineimap, and the
>>> suggestion of isync/mbsync as a replacement.
>>
>> I am using offlineimap, and I also saw many complains (and had a few
>> problems myself and looked at isync briefly), but it is now running
>> solidly, after I reduced the
>> number of emails in a folder (or tag, as I am syncing from  gmail to
>> localhost, also dovecot).
>>
>> One reason which kept me at offlineimap is that it seems to be more
>> widely used then isync.
>>
>> So I can only say it works for me.
>
> Thanks! Anecdotal evidence is definitely important. I'm also using
> multiple gmail addresses, and have never bothered much with tags/folders
> as I've always used a mail client. I'm currently pulling down 55,000
> messages in one account's INBOX, do you think that's likely to cause me
> problems?

I experienced problems with my All Mail folder which was more then
200,000 messages, but, if I remember correctly,around 50,000 was
fine. So I would say: try it.

By the way: I am using offlineimap at the moment as daemon (not via
cron) and it seems to be working a) faster and b) more stable.

Cheers,

Rainer


>
> Thanks again,
> Eric
>
>>
>> Cheers,
>>
>> Rainer
>>
>>
>>>
>>> Has anyone done that? I realize this isn't quite a gnus question, but
>>> I'm having a bit of trouble wrapping my head around this, and google
>>> isn't helping any.
>>>
>>> Specifically, you tell offlineimap to deliver to dovecot with a line
>>> like:
>>>
>>> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>>>
>>> I can't find the equivalent command for mbsync. I don't know if it's
>>> possible, and worse I don't really understand what's going on well
>>> enough to really figure it out logically.
>>>
>>> Is it enough just to have mbsync dump to a Maildir directly, and then
>>> tell gnus to access that maildir via dovecot?
>>>
>>> Any pointers very welcome!
>>>
>>> Eric
>>>
>>>
>>>
>> <#secure method=pgpmime mode=sign>
>
>
>
<#secure method=pgpmime mode=sign>

Re: isync plus dovecot?

[Dave Abrahams]

Well, I have extensive experience with both offlineimap and mbsync, and
the offlineimap experience was essentially a neverending series of
problems and hacking.  Ever since I set up mbsync (from the isync
project), and added my idle script wrapper, I don't think about it at
all.  It Just Works.

>
> So I can only say it works for me.
>
> Cheers,
>
> Rainer
>
>>
>> Has anyone done that? I realize this isn't quite a gnus question, but
>> I'm having a bit of trouble wrapping my head around this, and google
>> isn't helping any.
>>
>> Specifically, you tell offlineimap to deliver to dovecot with a line
>> like:
>>
>> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>>
>> I can't find the equivalent command for mbsync. I don't know if it's
>> possible, and worse I don't really understand what's going on well
>> enough to really figure it out logically.
>>
>> Is it enough just to have mbsync dump to a Maildir directly, and then
>> tell gnus to access that maildir via dovecot?
>>
>> Any pointers very welcome!
>>
>> Eric
>>
>>
>>
> <#secure method=pgpmime mode=sign>
>

-- 
Dave Abrahams

Re: isync plus dovecot?

[Dan Christensen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I'm finally trying to ditch nnml for mail reading, and move to imap.
> Offlineimap plus dovecot seems to be a popular solution (I am offline a
> lot), but I also saw a lot of complaints about offlineimap, and the
> suggestion of isync/mbsync as a replacement.

I've used offlineimap for a couple of years, and I find it works well.
I've posted on the offlineimap list a way to get it to sync some of
the Gnus specific flags, like gnus-expire.  If you can't find the
message, let me know and I'll dig it up.

> Specifically, you tell offlineimap to deliver to dovecot with a line
> like:
>
> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>
> I can't find the equivalent command for mbsync.

Nothing in my .offlineimaprc mentions dovecot, and I don't use a
preauthtunnel.  I just tell offlineimap to connect to an IMAP server
at 127.0.0.1 and to another IMAP server at a certain host name.

> Is it enough just to have mbsync dump to a Maildir directly, and then
> tell gnus to access that maildir via dovecot?

That might work too, but I decided to let dovecot be in charge of the
Maildir directory, and make both Gnus and offlineimap interact with that
directory via the IMAP protocol.  My method of syncing the non-standard
flags relies on this.

I'm also curious to hear about alternatives to offlineimap, so I hope
some people reply with their experience.

Dan

Re: isync plus dovecot?

[Eric Abrahamsen]

Dan Christensen <jdc@uwo.ca> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I'm finally trying to ditch nnml for mail reading, and move to imap.
>> Offlineimap plus dovecot seems to be a popular solution (I am offline a
>> lot), but I also saw a lot of complaints about offlineimap, and the
>> suggestion of isync/mbsync as a replacement.
>
> I've used offlineimap for a couple of years, and I find it works well.
> I've posted on the offlineimap list a way to get it to sync some of
> the Gnus specific flags, like gnus-expire.  If you can't find the
> message, let me know and I'll dig it up.
>
>> Specifically, you tell offlineimap to deliver to dovecot with a line
>> like:
>>
>> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>>
>> I can't find the equivalent command for mbsync.
>
> Nothing in my .offlineimaprc mentions dovecot, and I don't use a
> preauthtunnel.  I just tell offlineimap to connect to an IMAP server
> at 127.0.0.1 and to another IMAP server at a certain host name.
>
>> Is it enough just to have mbsync dump to a Maildir directly, and then
>> tell gnus to access that maildir via dovecot?
>
> That might work too, but I decided to let dovecot be in charge of the
> Maildir directory, and make both Gnus and offlineimap interact with that
> directory via the IMAP protocol.  My method of syncing the non-standard
> flags relies on this.
>
> I'm also curious to hear about alternatives to offlineimap, so I hope
> some people reply with their experience.

Ugh, clearly I just don't know enough about how this all actually works
to make an informed decision. I don't even really get what letting
dovecot "be in charge of the Maildir directory" really means. And if the
other Eric is able to use the native Gnus agent and it works fine
offline, why is everyone jumping through all these hoops?

Maybe I'll just spend some time with google until I actually understand
what's happening...

Thanks,
Eric

PS I'm also using gmail. Actually I have four addresses, all of them
google, three of them google apps domains. Yeesh

Re: isync plus dovecot?

[Eric S Fraga]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:


[...]

> dovecot "be in charge of the Maildir directory" really means. And if the
> other Eric is able to use the native Gnus agent and it works fine
> offline, why is everyone jumping through all these hoops?

One of the reasons for using offlineimap (& co.) is to provide much
quicker access to your emails.  Given the non-threaded nature of Emacs,
every time you try to access emails to/from an external (IMAP) server,
Emacs will hang.  For those that have a single Emacs instance running,
this can be quite annoying.  Having emails stored locally via
offlineimap or mbsync gives you direct and fast access.

My solution to this is to have 2 Emacs instances running always, one for
email and the other for everything else.

-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 + Ma Gnus v0.8 + evil 1.0-dev
: BBDB version 3.02 ($Date: 2013/04/13 13:39:40 $)

Re: isync plus dovecot?

[Dan Christensen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>>> Is it enough just to have mbsync dump to a Maildir directly, and then
>>> tell gnus to access that maildir via dovecot?
>>
>> That might work too, but I decided to let dovecot be in charge of the
>> Maildir directory, and make both Gnus and offlineimap interact with that
>> directory via the IMAP protocol.  My method of syncing the non-standard
>> flags relies on this.
>
> Ugh, clearly I just don't know enough about how this all actually works
> to make an informed decision. I don't even really get what letting
> dovecot "be in charge of the Maildir directory" really means. 

Offlineimap has two modes of operation.  The original way it worked
was to connect to a remote IMAP server using the IMAP protocol and
to write directly to a Maildir in the local filesystem.  In this
mode of operation, you don't need a local IMAP server like dovecot,
it's just a Maildir <--> remote IMAP sync.

An alternate way is for offlineimap to synchronize between two IMAP
servers located anywhere on the internet.  Offlineimap uses the IMAP
protocol to do this, and has no knowledge of how the servers store
the messages on disk.  I use this way.

> And if the other Eric is able to use the native Gnus agent and it
> works fine offline, why is everyone jumping through all these hoops?

It seems to me that there are very frequent complaints about the agent /
IMAP interaction on this mailing list, and I believe that Lars doesn't
use the agent (at least not for IMAP), so I don't trust that set-up.

Dan

Re: isync plus dovecot?

[Rainer M. Krug]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> I'm finally trying to ditch nnml for mail reading, and move to imap.
>>> Offlineimap plus dovecot seems to be a popular solution (I am offline a
>>> lot), but I also saw a lot of complaints about offlineimap, and the
>>> suggestion of isync/mbsync as a replacement.
>>
>> I've used offlineimap for a couple of years, and I find it works well.
>> I've posted on the offlineimap list a way to get it to sync some of
>> the Gnus specific flags, like gnus-expire.  If you can't find the
>> message, let me know and I'll dig it up.
>>
>>> Specifically, you tell offlineimap to deliver to dovecot with a line
>>> like:
>>>
>>> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>>>
>>> I can't find the equivalent command for mbsync.
>>
>> Nothing in my .offlineimaprc mentions dovecot, and I don't use a
>> preauthtunnel.  I just tell offlineimap to connect to an IMAP server
>> at 127.0.0.1 and to another IMAP server at a certain host name.
>>
>>> Is it enough just to have mbsync dump to a Maildir directly, and then
>>> tell gnus to access that maildir via dovecot?
>>
>> That might work too, but I decided to let dovecot be in charge of the
>> Maildir directory, and make both Gnus and offlineimap interact with that
>> directory via the IMAP protocol.  My method of syncing the non-standard
>> flags relies on this.
>>
>> I'm also curious to hear about alternatives to offlineimap, so I hope
>> some people reply with their experience.
>
> Ugh, clearly I just don't know enough about how this all actually works
> to make an informed decision. I don't even really get what letting
> dovecot "be in charge of the Maildir directory" really means. And if the
> other Eric is able to use the native Gnus agent and it works fine
> offline, why is everyone jumping through all these hoops?

preauthtunnel simply means (from my ignorant understanding of dovecot et
al) that you don't have to supply a password to connect to the local
imap server, but that you (the user) use dovecot (imap) directly to
access you maildir. It is *still* via docecot and even Erics setup
should work with the usage of the preauth tunnel instead of localhost. 

Direct Maildir access is a different story.

Rainer

>
> Maybe I'll just spend some time with google until I actually understand
> what's happening...
>
> Thanks,
> Eric
>
> PS I'm also using gmail. Actually I have four addresses, all of them
> google, three of them google apps domains. Yeesh
>
>
>

Re: isync plus dovecot?

[Rainer M. Krug]

Dan Christensen <jdc@uwo.ca> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I'm finally trying to ditch nnml for mail reading, and move to imap.
>> Offlineimap plus dovecot seems to be a popular solution (I am offline a
>> lot), but I also saw a lot of complaints about offlineimap, and the
>> suggestion of isync/mbsync as a replacement.
>
> I've used offlineimap for a couple of years, and I find it works well.
> I've posted on the offlineimap list a way to get it to sync some of
> the Gnus specific flags, like gnus-expire.  If you can't find the
> message, let me know and I'll dig it up.

Yes please - I would be interested as well. Could you provide some
keywords to search for?

Rainer


>
>> Specifically, you tell offlineimap to deliver to dovecot with a line
>> like:
>>
>> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>>
>> I can't find the equivalent command for mbsync.
>
> Nothing in my .offlineimaprc mentions dovecot, and I don't use a
> preauthtunnel.  I just tell offlineimap to connect to an IMAP server
> at 127.0.0.1 and to another IMAP server at a certain host name.
>
>> Is it enough just to have mbsync dump to a Maildir directly, and then
>> tell gnus to access that maildir via dovecot?
>
> That might work too, but I decided to let dovecot be in charge of the
> Maildir directory, and make both Gnus and offlineimap interact with that
> directory via the IMAP protocol.  My method of syncing the non-standard
> flags relies on this.
>
> I'm also curious to hear about alternatives to offlineimap, so I hope
> some people reply with their experience.
>
> Dan
>
>
>

Re: isync plus dovecot?

[Dan Christensen]

Rainer@krugs.de (Rainer M. Krug) writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>> I've used offlineimap for a couple of years, and I find it works well.
>> I've posted on the offlineimap list a way to get it to sync some of
>> the Gnus specific flags, like gnus-expire.  If you can't find the
>> message, let me know and I'll dig it up.
>
> Yes please - I would be interested as well. Could you provide some
> keywords to search for?

Here's one place I described it:

  http://lists.gnu.org/archive/html/info-gnus-english/2011-10/msg00052.html

For convenience, I've copied the body of that message below.  I'm using
offlineimap 6.3.5-rc3.

Dan

offlineimap doesn't sync custom flags by default.  The hack I use just
monkeypatches the flagmap.  Put the following in .offlineimap.py:

import offlineimap.imaputil as IU
if not hasattr(IU, 'monkeypatchdone'):
    IU.flagmap += [('gnus-expire','E'),
                   ('gnus-dormant', 'Q'),
                   ('gnus-save', 'V'),
                   ('gnus-forward', 'W')]
    IU.monkeypatchdone = True

To activate this, you put

pythonfile = ~/.offlineimap.py

in your .offlineimaprc

Re: isync plus dovecot?

[Eric Abrahamsen]

Dan Christensen <jdc@uwo.ca> writes:

> Rainer@krugs.de (Rainer M. Krug) writes:
>
>> Dan Christensen <jdc@uwo.ca> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>> I've used offlineimap for a couple of years, and I find it works well.
>>> I've posted on the offlineimap list a way to get it to sync some of
>>> the Gnus specific flags, like gnus-expire.  If you can't find the
>>> message, let me know and I'll dig it up.
>>
>> Yes please - I would be interested as well. Could you provide some
>> keywords to search for?
>
> Here's one place I described it:
>
>   http://lists.gnu.org/archive/html/info-gnus-english/2011-10/msg00052.html
>
> For convenience, I've copied the body of that message below.  I'm using
> offlineimap 6.3.5-rc3.
>
> Dan
>
> offlineimap doesn't sync custom flags by default.  The hack I use just
> monkeypatches the flagmap.  Put the following in .offlineimap.py:
>
> import offlineimap.imaputil as IU
> if not hasattr(IU, 'monkeypatchdone'):
>     IU.flagmap += [('gnus-expire','E'),
>                    ('gnus-dormant', 'Q'),
>                    ('gnus-save', 'V'),
>                    ('gnus-forward', 'W')]
>     IU.monkeypatchdone = True
>
> To activate this, you put
>
> pythonfile = ~/.offlineimap.py
>
> in your .offlineimaprc

This looks a bit like the instructions I started off following online:

http://roland.entierement.nu/blog/2010/09/08/gnus-dovecot-offlineimap-search-a-howto.html

I only realized after I failing to get it to work several times that
there's an update at the bottom of the blog post for versions of dovecot
over 2.0.

Re: isync plus dovecot?

[Erik Colson]

Hi,

Since this thread is mentioning dovecot server, I wanted to know if anyone tried dovecot's own dsync. dsync synchronizes two dovecot IMAP mailboxes. It can therefor be a substitution for offlineimap IMAP <-> IMAP synchronisation when both servers are dovecot...

I'd be happy to hear comments on this as I didn't try it myself but hence, I removed my offlineimap setup when it did corrupt my mail when I accessed them from both gnus and Mail.app on macosx.

best
--
erik

Re: isync plus dovecot?

[Eric S Fraga]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I'm finally trying to ditch nnml for mail reading, and move to imap.
> Offlineimap plus dovecot seems to be a popular solution (I am offline a
> lot), but I also saw a lot of complaints about offlineimap, and the
> suggestion of isync/mbsync as a replacement.

I used to use offlineimap but gave up.  It just wasn't robust enough for
me.

I now use the gnus agent, and have been doing so for a couple of years
at least.  It also has some issues of robustness but not as many as I
found with offlineimap (YMMV).  I also spend considerable amount of time
offline every day and in fact process the majority of my emails offline
(like right now).

A lot will depend on what IMAP server you are using.  I, for my sins,
have to use an Outlook Exchange server which is very annoyingly not
standard compliant.  Google's gmail is also not quite with the standards
but seems to cause me less problems.

My various email folders (aka groups) have up to 5000 emails typically,
although largest is about 15000, so not quite your levels but also not
out of line completely.  Total email content is about 3GB.  However, my
actual "inbox" is always empty as I split all my incoming emails to
specific folders on retrieval.

HTH,
eric
-- 
: Eric S Fraga, GnuPG: 0xC89193D8FFFCF67D
: in Emacs 24.3.50.1 + Ma Gnus v0.8 + evil 1.0-dev
: BBDB version 3.02 ($Date: 2013/04/13 13:39:40 $)

Re: isync plus dovecot?

[Drew Raines]

Eric Abrahamsen wrote:

> I'm finally trying to ditch nnml for mail reading, and move to
> imap.  Offlineimap plus dovecot seems to be a popular solution 
> (I
> am offline a lot), but I also saw a lot of complaints about
> offlineimap, and the suggestion of isync/mbsync as a 
> replacement.
>
> Has anyone done that? I realize this isn't quite a gnus 
> question,
> but I'm having a bit of trouble wrapping my head around this, 
> and
> google isn't helping any.

I'm using mbsync.  I finally switched from offlineimap a year ago
after it got into a state that I couldn't fix.

My setup is:

   Gmail <-> mbsync <-> dovecot <-> nnimap

dovecot essentially serves as a cache, with mbsync running every 
few
minutes in a separate process.  Even though g-g-get-new-news is
pretty speedy (5-30s), I still use a dedicated Emacs for Gnus.  It
still talks over the Internet to Gmane.  I used to use gnus-agent
with nnimap and it worked fine.  Now I only use it with nntp.

I haven't had any major issues with mbsync.  A couple of times it 
has
gotten wedged from an unreliable network connection but I was able 
to
recover by deleting its local state.  It hasn't caused any 
headaches
for me.  This is isync 1.0.4 on Debian Wheezy.

Happy to share more details if it would help!

Drew

Re: isync plus dovecot?

[Eric Abrahamsen]

Drew Raines <aaraines@gmail.com> writes:

> Eric Abrahamsen wrote:
>
>> I'm finally trying to ditch nnml for mail reading, and move to
>> imap.  Offlineimap plus dovecot seems to be a popular solution (I
>> am offline a lot), but I also saw a lot of complaints about
>> offlineimap, and the suggestion of isync/mbsync as a replacement.
>>
>> Has anyone done that? I realize this isn't quite a gnus question,
>> but I'm having a bit of trouble wrapping my head around this, and
>> google isn't helping any.
>
> I'm using mbsync.  I finally switched from offlineimap a year ago
> after it got into a state that I couldn't fix.
>
> My setup is:
>
>   Gmail <-> mbsync <-> dovecot <-> nnimap
>
> dovecot essentially serves as a cache, with mbsync running every few
> minutes in a separate process.  Even though g-g-get-new-news is
> pretty speedy (5-30s), I still use a dedicated Emacs for Gnus.  It
> still talks over the Internet to Gmane.  I used to use gnus-agent
> with nnimap and it worked fine.  Now I only use it with nntp.
>
> I haven't had any major issues with mbsync.  A couple of times it has
> gotten wedged from an unreliable network connection but I was able to
> recover by deleting its local state.  It hasn't caused any headaches
> for me.  This is isync 1.0.4 on Debian Wheezy.
>
> Happy to share more details if it would help!

Thanks for this! So your .mbsyncrc has something like this, right?

MaildirStore account-local
Path = ~/Maildir/account/
Inbox ~/Maildir/account/Inbox

and then in your gnus select methods:

(nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir/")

Is that how you do it? The offlineimap examples I've seen have
offlineimap delivering to dovecot, and gnus reading from dovecot. The
above would have mbsync delivering to a maildir directly, and then
dovecot reading from that. I'm not familiar enough with how this works
to know if that's right. And three days of mail syncing seems like a
long time just to test a setup that's wrong to begin with!

The other question I've got regards multiple accounts, either
offlineimap or mbsync. Multi-account examples I've seen have offlineimap
using a separate local repository for each account, so you end up with:

$HOME/Maildir/account1/{cur,new,tmp,etc,etc}
$HOME/Maildir/account2/{cur,new,tmp,etc,etc}
$HOME/Maildir/account3/{cur,new,tmp,etc,etc}

But in my gnus configuration, I only seem to be able to pass one value
to the "-o mail_location" invocation, and it doesn't recurse into
subdirectories. Can I dump multiple accounts into a single local
repository? Or somehow tell dovecot to check multiple directories?

Almost done, I swear...

Thanks!
Eric

Re: isync plus dovecot?

[Drew Raines]

Eric Abrahamsen wrote:

> Thanks for this! So your .mbsyncrc has something like this, 
> right?

[...]

I use dovecot as a real IMAP server.  My mbsync config looks like
this.  It's a little obfuscated by my hack that keeps my IMAP
password out of it.

  https://gist.github.com/drewr/c6e213f0b7ccd712e136

mbsync syncs Gmail IMAP <-> dovecot IMAP and nnimap connects to
dovecot for the different users.  I set up userdb authentication 
in
dovecot to link to different directories in ~/Mail depending on 
who
connects.

  userdb {
    driver = static
    args = uid=1000 gid=1000 home=/home/aar/Mail/dovecot/%n
  }

The maildir approach, however, sounds nice.  Not having to 
maintain a
dovecot instance and bear the cost of Gnus reconnecting to it
periodically is attractive.

Drew

Re: isync plus dovecot?

[Eric Abrahamsen]

Drew Raines <aaraines@gmail.com> writes:

> Eric Abrahamsen wrote:
>
>> Thanks for this! So your .mbsyncrc has something like this, right?
>
> [...]
>
> I use dovecot as a real IMAP server.  My mbsync config looks like
> this.  It's a little obfuscated by my hack that keeps my IMAP
> password out of it.
>
>  https://gist.github.com/drewr/c6e213f0b7ccd712e136
>
> mbsync syncs Gmail IMAP <-> dovecot IMAP and nnimap connects to
> dovecot for the different users.  I set up userdb authentication in
> dovecot to link to different directories in ~/Mail depending on who
> connects.
>
>  userdb {
>    driver = static
>    args = uid=1000 gid=1000 home=/home/aar/Mail/dovecot/%n
>  }

Okay, I think I'm finally wrapping my head around the different possible
ways of using dovecot locally. I'm still going to stick with the maildir
approach though: as you mention it means one less process on the
machine, and I guess I feel more comfortable with it. I'm also still not
quite clear on the distinction between Dovecot's userdb users, system
users, and multiple email accounts.

> The maildir approach, however, sounds nice.  Not having to maintain a
> dovecot instance and bear the cost of Gnus reconnecting to it
> periodically is attractive.

To anyone and everyone:

I'm still trying to solve the problem of offlineimap or mbsync
delivering to a separate maildir for each email account, but dovecot
only reading from one mail_location. The next thing I'm going to try is
the following in offlineimaprc:

[Repository local-acc1]
preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail:LAYOUT=fs:DIRNAME=acc1

[Repository local-acc2]
preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail:LAYOUT=fs:DIRNAME=acc2

[Repository local-acc3]
preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail:LAYOUT=fs:DIRNAME=acc3

And hope that the accounts don't clobber each other. I'll report back
if it works, but please stop me if you already know it won't!

Thanks,
Eric

Re: isync plus dovecot?

[Dave Abrahams]

on Wed May 15 2013, Eric Abrahamsen <eric-AT-ericabrahamsen.net> wrote:

> I'm finally trying to ditch nnml for mail reading, and move to imap.
> Offlineimap plus dovecot seems to be a popular solution (I am offline a
> lot), but I also saw a lot of complaints about offlineimap, and the
> suggestion of isync/mbsync as a replacement.
>
> Has anyone done that? I realize this isn't quite a gnus question, but
> I'm having a bit of trouble wrapping my head around this, and google
> isn't helping any.
>
> Specifically, you tell offlineimap to deliver to dovecot with a line
> like:
>
> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir
>
> I can't find the equivalent command for mbsync. I don't know if it's
> possible, and worse I don't really understand what's going on well
> enough to really figure it out logically.
>
> Is it enough just to have mbsync dump to a Maildir directly, and then
> tell gnus to access that maildir via dovecot?
>
> Any pointers very welcome!

This may be cominig very late, but: http://onmsg.github.com might still
be of some use to you.  I haven't touched the stuff in some time now,
but it still works great for me.

Here's my fairly-complicated .mbsyncrc, including the preauth tunneling
thing:

--8<---------------cut here---------------start------------->8---
IMAPAccount boostpro-gmail-account
Host imap.gmail.com
User dave@boostpro.com
UseIMAPS yes
# RequireSSL yes
CertificateFile ~/.certs/imap.gmail.com.pem
Pass "################"

IMAPStore boostpro-gmail-store
Account boostpro-gmail-account

IMAPAccount local-dovecot
Host localhost
User dave
Tunnel /opt/local/libexec/dovecot/imap

IMAPStore local-dovecot-store
Account local-dovecot

Channel boostpro-active-channel
Master :boostpro-gmail-store:
Slave :local-dovecot-store:
Patterns "INBOX" "travel" "apple"
Create Both
# Sync ReNew Delete Flags
Expunge Both
SyncState ~/Library/Data/mbsync/boostpro-gmail-sync-state/

Channel boostpro-list-channel
Master :boostpro-gmail-store:
Slave :local-dovecot-store:
Patterns "committee"
Create Slave
Sync New Flags
SyncState ~/Library/Data/mbsync/boostpro-gmail-sync-state/

Channel boostpro-gmail-channel
Master :boostpro-gmail-store:
Slave :local-dovecot-store:
Patterns "[Gmail]/All Mail" "committee"
Create Slave
Sync New
SyncState ~/Library/Data/mbsync/boostpro-gmail-sync-state/

Channel boostpro-spam-channel
Master :boostpro-gmail-store:
Slave :local-dovecot-store:
Patterns "[Gmail]/Spam"
Create Master
Sync Push
SyncState ~/Library/Data/mbsync/boostpro-gmail-sync-state/

--8<---------------cut here---------------end--------------->8---


-- 
Dave Abrahams

Re: isync plus dovecot?

[Eric Abrahamsen]

Dave Abrahams <dave@boostpro.com> writes:

> on Wed May 15 2013, Eric Abrahamsen <eric-AT-ericabrahamsen.net> wrote:
>

[...]

> This may be cominig very late, but: http://onmsg.github.com might still
> be of some use to you.  I haven't touched the stuff in some time now,
> but it still works great for me.
>
> Here's my fairly-complicated .mbsyncrc, including the preauth tunneling
> thing:

Brilliant! This is great, thank you very much. I had begun to despair
slightly -- I knew it was possible, but wasn't making any headway. Your
set up is complicated indeed, but just knowing that the Tunnel directive
can work like that is enough to get me back on the road.

Do you only have the one account?

I'm sure I'll be back with more questions, but this is enough to get me
going. Sometimes I think it's an incredible amount of work just to get
email working the way you want; other times I think email is probably
60% of what I do with the computer, so...

Thanks again

> IMAPAccount boostpro-gmail-account
> Host imap.gmail.com
> User dave@boostpro.com
> UseIMAPS yes
> # RequireSSL yes
> CertificateFile ~/.certs/imap.gmail.com.pem
> Pass "################"
>
> IMAPStore boostpro-gmail-store
> Account boostpro-gmail-account
>
> IMAPAccount local-dovecot
> Host localhost
> User dave
> Tunnel /opt/local/libexec/dovecot/imap
>
> IMAPStore local-dovecot-store
> Account local-dovecot
>
> Channel boostpro-active-channel
> Master :boostpro-gmail-store:
> Slave :local-dovecot-store:
> Patterns "INBOX" "travel" "apple"
> Create Both
> # Sync ReNew Delete Flags
> Expunge Both
> SyncState ~/Library/Data/mbsync/boostpro-gmail-sync-state/
>
> Channel boostpro-list-channel
> Master :boostpro-gmail-store:
> Slave :local-dovecot-store:
> Patterns "committee"
> Create Slave
> Sync New Flags
> SyncState ~/Library/Data/mbsync/boostpro-gmail-sync-state/
>
> Channel boostpro-gmail-channel
> Master :boostpro-gmail-store:
> Slave :local-dovecot-store:
> Patterns "[Gmail]/All Mail" "committee"
> Create Slave
> Sync New
> SyncState ~/Library/Data/mbsync/boostpro-gmail-sync-state/
>
> Channel boostpro-spam-channel
> Master :boostpro-gmail-store:
> Slave :local-dovecot-store:
> Patterns "[Gmail]/Spam"
> Create Master
> Sync Push
> SyncState ~/Library/Data/mbsync/boostpro-gmail-sync-state/

Re: isync plus dovecot?

[Dave Abrahams]

on Mon Jun 17 2013, Eric Abrahamsen <eric-AT-ericabrahamsen.net> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>> on Wed May 15 2013, Eric Abrahamsen <eric-AT-ericabrahamsen.net> wrote:
>>
>
> [...]
>
>> This may be cominig very late, but: http://onmsg.github.com might still
>> be of some use to you.  I haven't touched the stuff in some time now,
>> but it still works great for me.
>>
>> Here's my fairly-complicated .mbsyncrc, including the preauth tunneling
>> thing:
>
> Brilliant! This is great, thank you very much. I had begun to despair
> slightly -- I knew it was possible, but wasn't making any headway. Your
> set up is complicated indeed, but just knowing that the Tunnel directive
> can work like that is enough to get me back on the road.
>
> Do you only have the one account?

Actually there's another account in that config, but it's private---and
commented out at the moment.

> I'm sure I'll be back with more questions, but this is enough to get me
> going. Sometimes I think it's an incredible amount of work just to get
> email working the way you want; other times I think email is probably
> 60% of what I do with the computer, so...

I known what you mean.  Please email me directly if you have questions
about onmsg.  I'm not likely to watch this list closely.

-- 
Dave Abrahams

Re: isync plus dovecot?

[Eric Abrahamsen]

Dave Abrahams <dave@boostpro.com> writes:

> on Wed May 15 2013, Eric Abrahamsen <eric-AT-ericabrahamsen.net> wrote:
>
>> I'm finally trying to ditch nnml for mail reading, and move to imap.
>> Offlineimap plus dovecot seems to be a popular solution (I am offline a
>> lot), but I also saw a lot of complaints about offlineimap, and the
>> suggestion of isync/mbsync as a replacement.

[...]

>> Specifically, you tell offlineimap to deliver to dovecot with a line
>> like:
>>
>> preauthtunnel = /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/Maildir

[...]

> Here's my fairly-complicated .mbsyncrc, including the preauth tunneling
> thing:

Okay, with help from Dave I'm now up and running with
mbsync/dovecot/gnus. It wasn't actually all that hard in the end,
the offlineimap and mbsync invocations were very similar.

I'm now able to access most mail through gnus, with one strange
exception: I've got several imap accounts, with a separate server per
account, and when I go into the individual servers via the *Server*
buffer they all look something like this:

22879: INBOX
    0: [Gmail]
    0: [Gmail]
    0: [Gmail]
    0: [Gmail]
    0: [Gmail]
    0: [Gmail]
8127: [Gmail]/Sent Mail

I'm guessing the empty entries represent Drafts, Important, Sent Mail,
Spam, Starred, and Trash, because each account only has Gmail's standard
folders (I haven't used imap before).  I suspect it has something to do
with the fact that the dovecot invocation in both mbsync and gnus looks
like:

/usr/lib/dovecot/imap -o
mail_location=maildir:$HOME/.mail/account1/:LAYOUT=fs:LAYOUT=fs:INDEX=$HOME/.mail/indexes/account1

Is it likely that separating the indexes from the mailboxes would
confuse gnus (or the gnus-dovecot conversation)?

If so, I assume I could remove this separation, and just carefully move
everything inside ~/.mail/indexes/account1 to ~/.mail/account1 ?

Thanks for any pointers (and for getting me this far),

Eric

Re: isync plus dovecot?

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:


[...]

> I'm now able to access most mail through gnus, with one strange
> exception: I've got several imap accounts, with a separate server per
> account, and when I go into the individual servers via the *Server*
> buffer they all look something like this:
>
> 22879: INBOX
>     0: [Gmail]
>     0: [Gmail]
>     0: [Gmail]
>     0: [Gmail]
>     0: [Gmail]
>     0: [Gmail]
> 8127: [Gmail]/Sent Mail
>
> I'm guessing the empty entries represent Drafts, Important, Sent Mail,
> Spam, Starred, and Trash, because each account only has Gmail's standard
> folders (I haven't used imap before).  I suspect it has something to do
> with the fact that the dovecot invocation in both mbsync and gnus looks
> like:
>
> /usr/lib/dovecot/imap -o
> mail_location=maildir:$HOME/.mail/account1/:LAYOUT=fs:LAYOUT=fs:INDEX=$HOME/.mail/indexes/account1
>
> Is it likely that separating the indexes from the mailboxes would
> confuse gnus (or the gnus-dovecot conversation)?

I tried moving the indexes for one of the accounts back under the main
directory for that account, with no effect. I also changed
gnus-ignored-newsgroups as per
http://www.emacswiki.org/emacs/GnusGmail#toc3, but that hasn't changed
anything (and anyway, the "[Gmail]/Sent Mail" group would have been
blocked out if that were the issue).

Still all the empty [Gmail] groups, and in the *Group* buffer each
account only shows three groups: INBOX, and [Gmail]/Sent Mail, which
behave normally, and [Gmail], which is empty.

I also noticed that the value of nnimap-split-methods was being ignored.
I tried respooling a random message, and after choosing the backend and
server, it gave me this error:

Debugger entered--Lisp error: (wrong-type-argument stringp nil)
  decode-coding-string(nil utf-8)
  nnimap-decode-gnus-group(nil)
  nnimap-request-accept-article(nil "Girzel" t)
  gnus-request-accept-article(nil (nnimap "Girzel" (nnimap-stream shell) (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail/girzel/:LAYOUT=fs") (nnimap-split-methods nnimap-split-fancy)) t t)
  eval((gnus-request-accept-article nil (quote (nnimap "Girzel" (nnimap-stream shell) (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail/girzel/:LAYOUT=fs") (nnimap-split-methods nnimap-split-fancy))) t t))
  nnimap-request-move-article(9208 "INBOX" "Girzel" (gnus-request-accept-article nil (quote (nnimap "Girzel" (nnimap-stream shell) (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail/girzel/:LAYOUT=fs") (nnimap-split-methods nnimap-split-fancy))) t t) t nil)
  gnus-request-move-article(9208 "nnimap+Girzel:INBOX" "Girzel" (gnus-request-accept-article nil (quote (nnimap "Girzel" (nnimap-stream shell) (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail/girzel/:LAYOUT=fs") (nnimap-split-methods nnimap-split-fancy))) t t) t nil)
  gnus-summary-move-article(nil nil (nnimap "Girzel" (nnimap-stream shell) (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail/girzel/:LAYOUT=fs") (nnimap-split-methods nnimap-split-fancy)))
  gnus-summary-respool-article(nil (nnimap "Girzel" (nnimap-stream shell) (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail/girzel/:LAYOUT=fs") (nnimap-split-methods nnimap-split-fancy)))
  call-interactively(gnus-summary-respool-article nil nil)

This was from the INBOX group, so I'm not sure why the group name was
nil, but hopefully it's all part of the same problem...

E