Setting up gnus: read on local, store remote (but not on mailserver)

Setting up gnus: read on local, store remote (but not on mailserver)

[supermiri123]

Hi gnurus!

Since I have a new job, I need to modify my gnus setup, and don't know how.  I have the following:

1) mail is coming from some server with limited storage space (in fact, not 
enough)

2) I have enough storage space on a machine I can ssh to, but where I do not have a reasonably recent emacs (No root rights.  Emacs version there is 21.)  However, this is in a local network (at a university), so ssh should not be too slow.

3) My email archive (which I depend on for every day work) has just about 2GB currently.

I would like to use my local emacs with gnus to read email, but have it stored on the other computer, so I can always access it, even if I'm not in my office.

Is there an effective solution known, or do I have to experiment?

Many thanks,

Martin

Re: Setting up gnus: read on local, store remote (but not on mailserver)

[W. Greenhouse]

Hi Martin,

supermiri123-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org writes:

> Hi gnurus!
>
> Since I have a new job, I need to modify my gnus setup, and don't know
> how.  I have the following:
>
> 1) mail is coming from some server with limited storage space (in
> fact, not enough)
>
> 2) I have enough storage space on a machine I can ssh to, but where I
> do not have a reasonably recent emacs (No root rights.  Emacs version
> there is 21.)  However, this is in a local network (at a university),
> so ssh should not be too slow.
>
> 3) My email archive (which I depend on for every day work) has just
> about 2GB currently.
>
> I would like to use my local emacs with gnus to read email, but have
> it stored on the other computer, so I can always access it, even if
> I'm not in my office.
>
> Is there an effective solution known, or do I have to experiment?
>
> Many thanks,
>
> Martin

A maildir might be a good choice.  The `nnmaildir' backend (info "(gnus)
Maildir") can use a remote filesystem, e.g.

(setq gnus-select-method
      '(nnmaildir "mail"
        (directory "/rsyncc:user-3I8OyKyBCfCmcwpypvFeBw@public.gmane.org:/path/to/maildir/")
        (get-new-mail t)))

The above would use Gnus to incorporate mail using whatever methods you
picked in `mail-sources', and store them in a remote maildir accessed
with rsync (you can use any TRAMP filename, e.g. ssh:, scpc:, etc.).

If you have the ability to install packages on the remote host, you
could also install offlineimap or something, and skip the `get-new-mail
t' in this example.

--
Regards,
WGG

Re: Setting up gnus: read on local, store remote (but not on mailserver)

[supermiri123]

Thanks for the quick response!

> A maildir might be a good choice.  The `nnmaildir' backend (info "(gnus)
> Maildir") can use a remote filesystem, e.g.
> 
> (setq gnus-select-method
>       '(nnmaildir "mail"
>         (directory "/rsyncc:user@remotehost.org:/path/to/maildir/")
>         (get-new-mail t)))
> 
> The above would use Gnus to incorporate mail using whatever methods you
> picked in `mail-sources', and store them in a remote maildir accessed
> with rsync (you can use any TRAMP filename, e.g. ssh:, scpc:, etc.).

Reading this, it appears to me that I was confused about what gnus-select-method does :-) I now checked my .emacs and learn that I use the customized variable mail-sources to tell gnus where to get my mail from, gnus-select-method to tell gnus where to get my news from, and gnus-secondary-select-method to tell gnus where to store my mail.  The last currently just reads ((nnml "")).

In the documentation I see that nnml allows a parameter nnml-directory.  Could I set this to a tramp path also?  This would have the advantage that I do not have to respool my old mail.  Also, the doc says that nnmaildir uses many many inodes (while nnml only uses many).  Should I worry about this?

Is there a particular reason to prefer nnmaildir over nnml in my setting?  (I was happy with nnml for some 10 years now, but of course, the setting is slightly different now.)


Many many thanks,

Martin

Re: Setting up gnus: read on local, store remote (but not on mailserver)

[W. Greenhouse]

Hi Martin,

supermiri123-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org writes:

> Thanks for the quick response!
>
>> A maildir might be a good choice.  The `nnmaildir' backend (info "(gnus)
>> Maildir") can use a remote filesystem, e.g.
>> 
>> (setq gnus-select-method
>>       '(nnmaildir "mail"
>>         (directory "/rsyncc:user-3I8OyKyBCfCmcwpypvFeBw@public.gmane.org:/path/to/maildir/")
>>         (get-new-mail t)))
>> 
>> The above would use Gnus to incorporate mail using whatever methods you
>> picked in `mail-sources', and store them in a remote maildir accessed
>> with rsync (you can use any TRAMP filename, e.g. ssh:, scpc:, etc.).
>
> Reading this, it appears to me that I was confused about what
> gnus-select-method does :-) I now checked my .emacs and learn that I
> use the customized variable mail-sources to tell gnus where to get my
> mail from, gnus-select-method to tell gnus where to get my news from,
> and gnus-secondary-select-method to tell gnus where to store my mail.
> The last currently just reads ((nnml "")).

Your usage is entirely sensible.  You can use either a mail or a news
backend as select-method or as one of the secondary select methods.  So
that was just an editorial decision on my part. :) I have sometimes used
Gnus in a mail-only configuration.

> In the documentation I see that nnml allows a parameter
> nnml-directory.  Could I set this to a tramp path also?  This would
> have the advantage that I do not have to respool my old mail.  Also,
> the doc says that nnmaildir uses many many inodes (while nnml only
> uses many).  Should I worry about this?
>
> Is there a particular reason to prefer nnmaildir over nnml in my
> setting?  (I was happy with nnml for some 10 years now, but of course,
> the setting is slightly different now.)

To answer your last question first: If only one Emacs machine will ever
be the client in this setup, and you are not using an external utility
like offlineimap or fetchmail directly on the mailbox, you could get
away with using nnml over TRAMP.  The advantage of nnmaildir is that,
unlike MH-style mailboxes like nnml, the maildir layout is designed to
be suitable for non-locking use, i.e. reading mail even as another
program is putting stuff into the inbox.  Therefore it's an excellent
choice for a mailbox hosted on a central server for multiple clients to
use.

Re: the inodes question, it depends on the filesystem on your university
server and how likely your sysadmin there is to complain :) You can
check out how many inodes you currently have available by using `df -i'.
I think it is not as much of a concern for many modern systems as it was
when the Gnus manual was first written (much like the reference to
dialup modems in (info "(gnus) Gnus Unplugged")).  nnmaildir uses up an
inode for each message and also has three NOV files for each mailbox
(one each for the new/ cur/ and tmp/ subdirs of the mailbox), so I guess
that accounts for the reference to it using slightly more inodes.

>
> Many many thanks,
>
> Martin

Hope this helps.

--
Regards,
WGG

Re: Setting up gnus: read on local, store remote (but not on mailserver)

[supermiri123]

> To answer your last question first: If only one Emacs machine will ever
> be the client in this setup, and you are not using an external utility
> like offlineimap or fetchmail directly on the mailbox, you could get
> away with using nnml over TRAMP.  The advantage of nnmaildir is that,
> unlike MH-style mailboxes like nnml, the maildir layout is designed to
> be suitable for non-locking use, i.e. reading mail even as another 
> program is putting stuff into the inbox.  Therefore it's an excellent
> choice for a mailbox hosted on a central server for multiple clients to
> use.

OK, since it is indeed the case that only one emacs at a time will be client, I will stick with nnml.  At least this has the advantage that I can simply copy the old mail archive then.

I guess that one drawback of the whole setup will be that regexp-searching mail articles will become much slower, right?  After all, emacs will have to fetch each message / file separately via ssh, check whether it matches...  Well, I'll see...

Many thanks for your help!

Martin

Re: Setting up gnus: read on local, store remote (but not on mailserver)

[W. Greenhouse]

supermiri123-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org writes:

>> To answer your last question first: If only one Emacs machine will ever
>> be the client in this setup, and you are not using an external utility
>> like offlineimap or fetchmail directly on the mailbox, you could get
>> away with using nnml over TRAMP.  The advantage of nnmaildir is that,
>> unlike MH-style mailboxes like nnml, the maildir layout is designed to
>> be suitable for non-locking use, i.e. reading mail even as another 
>> program is putting stuff into the inbox.  Therefore it's an excellent
>> choice for a mailbox hosted on a central server for multiple clients to
>> use.
>
> OK, since it is indeed the case that only one emacs at a time will be
> client, I will stick with nnml.  At least this has the advantage that
> I can simply copy the old mail archive then.
>
> I guess that one drawback of the whole setup will be that
> regexp-searching mail articles will become much slower, right?  After
> all, emacs will have to fetch each message / file separately via ssh,
> check whether it matches...  Well, I'll see...

Yeah, for searches within the message body, anyway.  Header searches
like / a should be close to the same speed, because Gnus loads up the
news overview file for the mailbox when you enter the group.

>
> Many thanks for your help!
>
> Martin

--
Regards,
WGG

Re: Setting up gnus: read on local, store remote (but not on mailserver)

[supermiri123]

OK, I've *almost* got it working.  I did the following via customize:

(gnus-home-directory "/scpc:YYY@XXX.ZZ:~")
(message-directory "/scpc:YYY@XXX.ZZ:~/Mail/")
(gnus-secondary-select-methods (quote ((nnml "" (nnml-marks-is-evil t)))))

This let me read my old mail which is stored with nnml.  I had to set nnml-marks-is-evil because otherwise tramp would encode and decode every .marks file, which was annoying.

However, the above does not let me read the archived mail stored with nnfolder. For these, the path is explicitely in my .newsrc.eld file.  I tried to do

(setq gnus-message-archive-method
      '(nnfolder "archive"
		 (nnfolder-directory   "/scpc:YYY@XXX.ZZ:~/Mail/archive")
		 (nnfolder-active-file "/scpc:YYY@XXX.ZZ:~/Mail/archive/active")
		 (nnfolder-get-new-mail nil)
		 (nnfolder-inhibit-expiry t)
		 (nnfolder-marks-is-evil t)))

but this had no effect (I guess it will only affect newly created archive folders).

In fact, I'd rather have no explicit paths in .newsrc.eld at all, because I do want to be able to use gnus by just taking the .emacs with me.

Any ideas?

Martin

Re: Setting up gnus: read on local, store remote (but not on mailserver)

[supermiri123]

I should add:

1) I'm willing to switch the nnfolder groups to nnml, and I don't care about marks or any other information currently stored.

2) There are 60 such groups, all named misc-mail-YYYY-MM

I'm thinking of moving them temporarily to my local machine and then enter each group and copy all articles to a new nnml group.

Martin

Re: Setting up gnus: read on local, store remote (but not on mailserver)

[W. Greenhouse]

Hi Martin,

supermiri123-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org writes:

> OK, I've *almost* got it working.  I did the following via customize:
>
> (gnus-home-directory "/scpc:YYY-UAZjL2UjQio@public.gmane.org:~")
> (message-directory "/scpc:YYY-UAZjL2UjQio@public.gmane.org:~/Mail/")
> (gnus-secondary-select-methods (quote ((nnml "" (nnml-marks-is-evil t)))))
>
> This let me read my old mail which is stored with nnml.  I had to set
> nnml-marks-is-evil because otherwise tramp would encode and decode
> every .marks file, which was annoying.
>
> However, the above does not let me read the archived mail stored with
> nnfolder. For these, the path is explicitely in my .newsrc.eld file.
> I tried to do
>
> (setq gnus-message-archive-method
>       '(nnfolder "archive"
> 		 (nnfolder-directory   "/scpc:YYY-UAZjL2UjQio@public.gmane.org:~/Mail/archive")
> 		 (nnfolder-active-file "/scpc:YYY-UAZjL2UjQio@public.gmane.org:~/Mail/archive/active")
> 		 (nnfolder-get-new-mail nil)
> 		 (nnfolder-inhibit-expiry t)
> 		 (nnfolder-marks-is-evil t)))
>
> but this had no effect (I guess it will only affect newly created
> archive folders).

I would say (despite the warnings in the ~/.newsrc.eld file) that you can
safely delete the line in your ~/.newsrc.eld where the nnfolder group is
defined.  For me it is a (setq gnus-server-alist ...) cell.  The new
nnfolder hierarchy should be detected from your settings.

Also, you might want to make sure that those old archive folders aren't
present but inactive.  Enter A A to see your entire list of available
groups from all backends (including nnfolder).  You might want to do
that to check before you try my idea above.

> In fact, I'd rather have no explicit paths in .newsrc.eld at all,
> because I do want to be able to use gnus by just taking the .emacs
> with me.
>
> Any ideas?

The mail-type backends including nnml and nnmaildir are *reasonably*
portable in that the NOV and marks are always included within the folder
itself, so that's all you really need to back up when changing systems.
But you can't eliminate .newsrc.eld completely, AFAIK, as it holds both
your group subscriptions and your marks for news-type backends
(including some mail systems that Gnus implements in a net-news-esque
way, like nnimap).  One idea you might think about is that all Gnus
directory variables inherit the `gnus-home-directory' variables, and
most backends inherit `gnus-directory'.  By setting these, you could
just move everything to the rsync/scpc address and hopefully everything
would just work.

I'm excited to here that remote nnml is mostly workable, though. :)  Is
the performance acceptable?

--
Regards,
WGG