emac gnus-centric gmail-phobic multi-host mail management

emac gnus-centric gmail-phobic multi-host mail management

[regcl]

Sorry for the length of this. Please let me know if I am posting to the
wrong list.

I use a lot of different machines including mac os x desktops/notebooks
at work and home, Grid engine cluster submit hosts reached by ssh and
compute hosts reached by the grid engine qrsh commend at various work
sites, and debian servers at home.  I use emacs 24.3/tty everywhere,
which I build from source on most machines.

I have been using gnus for about a year as my mail client, and it works
everywhere. I am just starting to use newgroups.  

I want/need to modify my mail approach with these objectives:


A) Sweep mail off of the servers

Currently my mail is mostly served by gmail + an an exchange server,
either from work accounts administered by my employer/institution or
personal accounts and accessed by IMAP. I want to stop using these
servers as my master mail repository and reduce the amount of mail left
there to a minimum. So, I need to sweep mail off of accounts into my own
mail repos. The approach needs to support compartmentalization, by which
I mean that at home I need all the repos (personal and work) and at work
I need only work related mail repos.


B) Sync the gnus view

I want to sync my gnus view of mail and newsgroups so it is consistent
and up to date on all machines. Once again, this needs to be
compartmentalized. At work I want to see only the work-related stuff, at
home I want to see everything.


C) Compatible mail repo/backup scheme

I need to park the mail I am sweeping somewhere, once again with the
compartmentalization issues. While I use emacs as my primary email
client, it would probably be wise not to lock myself in to emacs, so
maybe I want the mail in a form compatible with other clients on my mac
os x desktops, most likely mac mail.app. What is the lightest weight
approach here? Is it maildir with dovecot/localhost?


D) Replication of the mail repos.

I don't think I want to put up a pop or imap server on the net, at least
not now. So I am thinking the problem here is primarily one of
replication of my  mail directories across machines. Am I right?

Based on my limited reading about mail, it seems like this could be done
with IMAP, but I know zilch about IMAP and I don't have a place to park
a personal IMAP server on the net.  

On the other hand I use git for all of the other replication/syncing
that I ever do. So here is the question: Can I park my mail in maildir
with sub-repos for work and play, and use git for replication between
machines? What would be the pros and cons of this approach vs learning
using IMAP?

Thanks in advance,
regcl

Re: emac gnus-centric gmail-phobic multi-host mail management

[W. Greenhouse]

Taking your last two questions:

regcl <regcl-layyI3BE+OXOHkwriC13pfZ8FUJU4vz8@public.gmane.org> writes:

[...]

> C) Compatible mail repo/backup scheme
>
> I need to park the mail I am sweeping somewhere, once again with the
> compartmentalization issues. While I use emacs as my primary email
> client, it would probably be wise not to lock myself in to emacs, so
> maybe I want the mail in a form compatible with other clients on my mac
> os x desktops, most likely mac mail.app. What is the lightest weight
> approach here? Is it maildir with dovecot/localhost?
>
>
> D) Replication of the mail repos.
>
> I don't think I want to put up a pop or imap server on the net, at least
> not now. So I am thinking the problem here is primarily one of
> replication of my  mail directories across machines. Am I right?
>
> Based on my limited reading about mail, it seems like this could be done
> with IMAP, but I know zilch about IMAP and I don't have a place to park
> a personal IMAP server on the net.  
>
> On the other hand I use git for all of the other replication/syncing
> that I ever do. So here is the question: Can I park my mail in maildir
> with sub-repos for work and play, and use git for replication between
> machines? What would be the pros and cons of this approach vs learning
> using IMAP?

Git would be sub-optimal for syncing Maildirs around, as would a
standard file-moving tool like rsync, because a fundamental idea of
Maildir is that mail states are expressed by renaming/moving the file.
Whenever Gnus flags something or marks it read, therefore, git or rsync
would treat this like a whole new file, leading to inefficiencies.
Fortunately, there are tools to parse the Maildir format and figure out
which messages are in fact the same on both ends, so we can quickly
rename them to express a change in "flaggedness", instead of resending
the whole thing and deleting the file on the other end (yuck).

Here are the two implementations I know of to efficiently sync maildirs
over SSH, without exposing a new network service such as IMAP.  (They
also bypass the system MTAs on both ends, as they operate on the end
user's maildirs directly):

1. Maildirsync <http://search.cpan.org/dist/maildirsync/maildirsync.pod>:
   Single perl script.  Requires you to write your own short shell script
   or cron job to handle synchronization.

2. http://syncmaildir.sourceforge.net/: a complete rewrite of
   maildirsync.  Evidently meant to "just work" without writing own shell
   hax; I had trouble getting it to install on Slackware, however, so I've
   been happily using maildirsync.  Syncmaildir seems to be packaged for
   Debian-like distros, so that's a consideration.

My configuration looks something like this:

1. My almost-always-on homebox slurps various mail sites via fetchmail
   using POP3.  On landing at the system MTA, the mail gets delivered
   (along with any local/daemon-generated mail) to various Maildirs
   according to rules in my ~/.procmailrc.

2. I read some mail on Gnus at the homebox.  When fetchmail->procmail
   delivers some new mail to my priority inbox, `display-time-mode'
   notifies me by putting a little picture of an envelope on my mode
   line. :) Otherwise I don't get prodded about new mail until I poll
   Gnus again myself.

3. I go on the road with my netbook.  On the netbook, I run my maildirsync
   script when I have reliable net, and then M-x gnus to read the mail.

4. Gnus nnmaildir is pretty self-contained (flags/unread state is all
   inside the maildir itself), so if you are only using maildir,
   maildirsync will be enough to share state between the two machines.
   Since I also read NNTP news with Gnus, however, and also use niceties
   like adaptive scoring, I also sync my emacs configuration dir (which
   contains the gnus score files and newsrc* files) using Bzr.

TODO: be able to sync just one thing, instead of the sync
mailidr-then-sync-bzr-repo dance.  Maybe
http://lars.ingebrigtsen.no/2014/02/01/emacs-cloud/ will help with this
someday.  Also, cronify the maildirsync end on the netbook.

Hope this helps.  Please don't hesitate to ask for clarification; I'd be
happy to share any of the configuration files (fetchmail, procmail,
gnus/message/smtpmail/bbdb, maildirsync) that might be interesting.

Re: emac gnus-centric gmail-phobic multi-host mail management

[regcl]

W. Greenhouse <wgreenhouse-sGOZH3hwPm2sTnJN9+BGXg@public.gmane.org>
writes:

> Taking your last two questions:
>
> regcl <regcl-layyI3BE+OXOHkwriC13pfZ8FUJU4vz8@public.gmane.org> writes:
>
> [...]
>
>> C) Compatible mail repo/backup scheme
>>
>> I need to park the mail I am sweeping somewhere, once again with the
>> compartmentalization issues. While I use emacs as my primary email
>> client, it would probably be wise not to lock myself in to emacs, so
>> maybe I want the mail in a form compatible with other clients on my mac
>> os x desktops, most likely mac mail.app. What is the lightest weight
>> approach here? Is it maildir with dovecot/localhost?
>>
>>
>> D) Replication of the mail repos.
>>
>> I don't think I want to put up a pop or imap server on the net, at least
>> not now. So I am thinking the problem here is primarily one of
>> replication of my  mail directories across machines. Am I right?
>>
>> Based on my limited reading about mail, it seems like this could be done
>> with IMAP, but I know zilch about IMAP and I don't have a place to park
>> a personal IMAP server on the net.  
>>
>> On the other hand I use git for all of the other replication/syncing
>> that I ever do. So here is the question: Can I park my mail in maildir
>> with sub-repos for work and play, and use git for replication between
>> machines? What would be the pros and cons of this approach vs learning
>> using IMAP?
>
> Git would be sub-optimal for syncing Maildirs around, as would a
> standard file-moving tool like rsync, because a fundamental idea of
> Maildir is that mail states are expressed by renaming/moving the file.
> Whenever Gnus flags something or marks it read, therefore, git or rsync
> would treat this like a whole new file, leading to inefficiencies.
> Fortunately, there are tools to parse the Maildir format and figure out
> which messages are in fact the same on both ends, so we can quickly
> rename them to express a change in "flaggedness", instead of resending
> the whole thing and deleting the file on the other end (yuck).
>

Actually, if the contents of a file don't change, git does not create
another object, since it recognizes it as the same file because the
file's SHA-1 does not change when you move it or rename it. The only
overhead of moving or renaming a file is to store its new name and
directory, which has to happen anyway for each commit. Since we don't
edit incoming mail messages, each mail message will create at most one
git storage object. This is why I was thinking git might work well for
Maildir.

> Here are the two implementations I know of to efficiently sync maildirs
> over SSH, without exposing a new network service such as IMAP.  (They
> also bypass the system MTAs on both ends, as they operate on the end
> user's maildirs directly):
>
> 1. Maildirsync <http://search.cpan.org/dist/maildirsync/maildirsync.pod>:
>    Single perl script.  Requires you to write your own short shell script
>    or cron job to handle synchronization.
>
> 2. http://syncmaildir.sourceforge.net/: a complete rewrite of
>    maildirsync.  Evidently meant to "just work" without writing own shell
>    hax; I had trouble getting it to install on Slackware, however, so I've
>    been happily using maildirsync.  Syncmaildir seems to be packaged for
>    Debian-like distros, so that's a consideration.
>

Thank you. I will compare the features of these to git to try to
understand what the relative merits might be.

> My configuration looks something like this:
>
> 1. My almost-always-on homebox slurps various mail sites via fetchmail
>    using POP3.  On landing at the system MTA, the mail gets delivered
>    (along with any local/daemon-generated mail) to various Maildirs
>    according to rules in my ~/.procmailrc.
>
> 2. I read some mail on Gnus at the homebox.  When fetchmail->procmail
>    delivers some new mail to my priority inbox, `display-time-mode'
>    notifies me by putting a little picture of an envelope on my mode
>    line. :) Otherwise I don't get prodded about new mail until I poll
>    Gnus again myself.
>
> 3. I go on the road with my netbook.  On the netbook, I run my maildirsync
>    script when I have reliable net, and then M-x gnus to read the mail.
>
> 4. Gnus nnmaildir is pretty self-contained (flags/unread state is all
>    inside the maildir itself), so if you are only using maildir,
>    maildirsync will be enough to share state between the two machines.
>    Since I also read NNTP news with Gnus, however, and also use niceties
>    like adaptive scoring, I also sync my emacs configuration dir (which
>    contains the gnus score files and newsrc* files) using Bzr.
>

Thank you for laying out these details.

> TODO: be able to sync just one thing, instead of the sync
> mailidr-then-sync-bzr-repo dance.  Maybe
> http://lars.ingebrigtsen.no/2014/02/01/emacs-cloud/ will help with this
> someday.  Also, cronify the maildirsync end on the netbook.

If we can do both of these functions with git, this issues may be resolved.
>
> Hope this helps.  Please don't hesitate to ask for clarification; I'd be
> happy to share any of the configuration files (fetchmail, procmail,
> gnus/message/smtpmail/bbdb, maildirsync) that might be interesting.

Yes, this is all very helpful, thank you. Once I select an approach I
may well want more details.

Best,
regcl