No Gnus feature request: simpler file structure for synching between two machines

No Gnus feature request: simpler file structure for synching between two machines

[John Owens]

I'd like to use gnus both in the office (on a desktop) and at home/on
the road (on a laptop). This requires that I keep state on both
machines. I'd like a simple way to synch the state files on the two
machines. Ideally this could be handled within gnus (using the
excellent "unison" to synch or something equivalent). But even doing
it manually wouldn't be so bad.

Unfortunately, the file structure for gnus is fairly complex. Kai
points out that he synchs the following files to move state:

"~/.newsrc*, ~/Mail/, ~/News/, ~/.nnmail-cache, and ~/.authinfo"

He also mentioned porting the ".marks" code from nnml.el to nntp.el,
which would "enable you to transfer a file that contains the marks for
a particular server (or group). That way, different servers would be
separated, and so you can transfer the state regarding a particular
server." And another suggestion he had was to "set gnus-home-directory
near the beginning of ~/.emacs (not ~/.gnus!) to a subdir of your home
directory. Then you can just transfer one directory."

In any case, this multitude of files seems like a great deal of
complexity. And using synched gnus on multiple computers seems like an
eminently useful feature. So, I would like to request:

- What's the canonical list of files that belong to gnus that need to
  be synched to duplicate state?
  - What are the files for (only) reading news?
  - What are the files for the different mail backends (particularly
    nnimap)? 
  - How could gnus (today) be configured to make this synch as easy as
    possible (as per Kai's gnus-home-directory suggestion above)?

- How could gnus be changed in No Gnus to make a synch easier /
  painless / faster / transparent between multiple computers? Could
  the files be amalgamated into a single file or in a single
  directory? Could news and mail synch be separated (or should they
  be)? 

As this seems like a topic for a redesign of the file structure,
perhaps it would be a worthy feature for No Gnus.

Thanks for listening -

JDO

Re: No Gnus feature request: simpler file structure for synching

[Matt Armstrong]

John Owens <jowens@ucdavis.edu> writes:

> Unfortunately, the file structure for gnus is fairly complex. Kai
> points out that he synchs the following files to move state:
>
> "~/.newsrc*, ~/Mail/, ~/News/, ~/.nnmail-cache, and ~/.authinfo"

To solve similar problems, I have subdued Gnus into sticking all files
under my ~/g/ directory with the following in my .emacs:

----------------------------------------------------------------------
(setq
 gnus-home-directory (expand-file-name "~/g/")
 nnmail-message-id-cache-file (concat gnus-home-directory
				      ".nnmail-cache")
 mail-source-directory (concat gnus-home-directory "Mail/")
 mail-source-crash-box (concat mail-source-directory
			       ".emacs-mail-crash-box")
 message-directory (concat gnus-home-directory "Mail/")
 gnus-startup-file (concat gnus-home-directory ".newsrc"))

(setq-default gnus-init-file (expand-file-name "~/env/dot/gnus"))

And the following in my .gnus (which is actually ~/env/dot/gnus, see
above)
----------------------------------------------------------------------
(setq gnus-secondary-select-methods
      '((nnml ""
	      (nnml-directory "/home/matt/g/nnml")
	      (nnml-active-file "/home/matt/g/nnml/active")
	      (nnml-newsgroupsfile "/home/matt/g/nnml/newsgroups"))))


I haven't figured out the lispy magic required to use "~/g/nnml"
etc. in the gnus-secondary-select-methods setq above.

-- 
matt (UCD grad)

Re: No Gnus feature request: simpler file structure for synching

[Steve Youngs]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

|--==> "MA" == Matt Armstrong <matt@lickey.com> writes:

  MA> (setq gnus-secondary-select-methods
  MA>       '((nnml ""
  MA> 	      (nnml-directory "/home/matt/g/nnml")
  MA> 	      (nnml-active-file "/home/matt/g/nnml/active")
  MA> 	      (nnml-newsgroupsfile "/home/matt/g/nnml/newsgroups"))))

  MA> I haven't figured out the lispy magic required to use "~/g/nnml"
  MA> etc. in the gnus-secondary-select-methods setq above.

How about...

(defconst my-nnml-dir
  (file-name-as-directory
   (expand-file-name "g/nnml" (getenv "HOME")))
  "My nnml directory")

(setq gnus-secondary-select-methods
      '((nnml ""
	      (nnml-directory my-nnml-dir)
	      (nnml-active-file (expand-file-name "active" my-nnml-dir))
	      (nnml-newsgroupsfile (expand-file-name "newsgroups" my-nnml-dir)))))


-- 
|---<Steve Youngs>---------------<GnuPG KeyID: A94B3003>---|
|              Ashes to ashes, dust to dust.               |
|      The proof of the pudding, is under the crust.       |
|------------------------------<sryoungs@bigpond.net.au>---|

[-- Attachment #2: Type: application/pgp-signature, Size: 256 bytes --]

Re: No Gnus feature request: simpler file structure for synching

[Kai Grossjohann]

Matt Armstrong <matt@lickey.com> writes:

> (setq gnus-secondary-select-methods
>       '((nnml ""
> 	      (nnml-directory "/home/matt/g/nnml")
> 	      (nnml-active-file "/home/matt/g/nnml/active")
> 	      (nnml-newsgroupsfile "/home/matt/g/nnml/newsgroups"))))
>
>
> I haven't figured out the lispy magic required to use "~/g/nnml"
> etc. in the gnus-secondary-select-methods setq above.

(setq gnus-secondary-select-methods
      `((nnml ""
         (nnml-directory ,(expand-file-name "~/g/nnml"))
         (nnml-active-file ,(expand-file-name "~/g/nnml/active"))
         (nnml-newsgroupsfile ,(expand-file-name "~/g/nnml/newsgroups")))))

The trick is in the ` (backtick) and , (comma).

But maybe it works to just say (nnml-directory "~/g/nnml")?  Maybe
Gnus groks the tilde.

Kai

Re: No Gnus feature request: simpler file structure for synching

[Reiner Steib]

On Wed, Jan 07 2004, Kai Grossjohann wrote:

> But maybe it works to just say (nnml-directory "~/g/nnml")?  Maybe
> Gnus groks the tilde.

IIRC Gnus does but `nnir.el' insists on expanded names instead of
doing `expand-file-name' itself.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: No Gnus feature request: simpler file structure for synching

[Kai Grossjohann]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> IIRC Gnus does but `nnir.el' insists on expanded names instead of
> doing `expand-file-name' itself.

*blush*  Point taken.

Re: No Gnus feature request: simpler file structure for synching

[Steven E. Harris]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

John Owens <jowens@ucdavis.edu> writes:

> Unfortunately, the file structure for gnus is fairly complex. Kai
> points out that he synchs the following files to move state:

I agree that the set of files is daunting. In particular, the split
between having a few files littered in the home directory versus
entire trees of relevant files makes using rsync difficult. It would
be much easier if all settings were rooted under one directory, and
better partitioned therein by server.

After fiddling around with rsync for an entire day trying to
understand inclusion and exclusion lists, I still wound up finding
success only by running it twice: once to exchange a few files in my
home directory, again to exchange most of the files under the main
Gnus directory. Find the program attached, called 'gnusync.'


[-- Attachment #2: gnusync --]
[-- Type: text/plain, Size: 548 bytes --]

#!/bin/sh

CMD='rsync -e ssh -lptD -vuzb --exclude='*~''
REMOTE=Panix:~/temp/sync/gnus

HOME_FILES=".newsrc .newsrc.eld .bbdb .diary"
NEWS_DIR="$HOME/doc/news"


OP=$1

case $OP in
get)
    shift 1
    cd $HOME
    $CMD -r $* $REMOTE/home/ .

    cd $NEWS_DIR
    $CMD -r $* $REMOTE/news/ .
    ;;

put)
    shift 1
    cd $HOME
    $CMD $* $HOME_FILES $REMOTE/home/

    cd $NEWS_DIR
    $CMD -r --delete \
    --exclude='/agent/' \
    $* . $REMOTE/news/
    ;;

*)
    PROG=`basename $0`
    echo "Usage: $PROG (get|put)"
    exit 1
    ;;
esac

[-- Attachment #3: Type: text/plain, Size: 22 bytes --]


-- 
Steven E. Harris

Re: No Gnus feature request: simpler file structure for synching

[John Owens]

Some of the suggestions here advocate synching the entire Mail
directory. I don't know about you guys, but my Mail directory is
enormous. It's my understanding that imap is supposed to keep state on
the server and thus ease the burden of synching state across
clients. 

Let's say I read news from the same server from my two clients, and
read mail ONLY using the nnimap back end. What needs to be synched for

- the news part of my use; and
- the nnimap part of my use?

JDO

nnir problems (was: No Gnus feature request: simpler file structure for synching)

[Reiner Steib]

On Wed, Jan 07 2004, Kai Grossjohann wrote:

> Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:
>
>> IIRC Gnus does but `nnir.el' insists on expanded names instead of
>> doing `expand-file-name' itself.
>
> *blush*  Point taken.

I wanted to fix this item some time ago (after you put it in
contrib/), but I couldn't get it to work, because I use (and want to
index/search) three different nnml servers (mail, outgoing news, saved
interesting news articles) with different nnml server parameters.
IIRC that's not possible up to now and a known problem.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: No Gnus feature request: simpler file structure for synching

[Kai Grossjohann]

John Owens <jowens@ucdavis.edu> writes:

> Let's say I read news from the same server from my two clients, and
> read mail ONLY using the nnimap back end. What needs to be synched for
>
> - the news part of my use; and
> - the nnimap part of my use?

That's the problem: in ~/.newsrc.eld you find information about all
servers.  But the .newsrc.eld file is the only file that records which
NNTP groups have been read.

That's why I suggested to port the .marks code to nntp.el; then you
would have a file for each NNTP server which records only which
articles have been read on that NNTP server.  That file could then be
synced.

Then you'd sync nothing for the nnimap part (all state is kept on the
server, except for the list of groups you subscribe to -- but that
doesn't change so often), and the single file for the nntp part.

Kai

Re: No Gnus feature request: simpler file structure for synching

[Josh Huber]

Kai Grossjohann <kai@emptydomain.de> writes:

> Then you'd sync nothing for the nnimap part (all state is kept on
> the server, except for the list of groups you subscribe to -- but
> that doesn't change so often), and the single file for the nntp
> part.

Doesn't IMAP also support storing subscription information on the
server?  Does Gnus use this?  I guess not, since the subscribed groups
in Gnus were not subscribed via Squirrelmail. (and vice-versa)

I wonder if we could use IMSP to store ALL gnus configuration?  now,
wouldn't that be cool?  Does it have enough features?

http://asg.web.cmu.edu/cyrus/rfc/imsp.html

Then, no syncing would be needed -- all information would be stored on
the IMAP/IMSP server.

-- 
Josh Huber

Re: No Gnus feature request: simpler file structure for synching

[Simon Josefsson]

Josh Huber <huber+news@alum.wpi.edu> writes:

> Kai Grossjohann <kai@emptydomain.de> writes:
>
>> Then you'd sync nothing for the nnimap part (all state is kept on
>> the server, except for the list of groups you subscribe to -- but
>> that doesn't change so often), and the single file for the nntp
>> part.
>
> Doesn't IMAP also support storing subscription information on the
> server?  Does Gnus use this?  I guess not, since the subscribed groups
> in Gnus were not subscribed via Squirrelmail. (and vice-versa)

Gnus add IMAP-subscribed mailboxes to *Group* automatically, I
believe, but it doesn't IMAP-subscribe groups.  I'm not sure how it
could, it seems wrong to IMAP-subscribe any group the user opens.

> I wonder if we could use IMSP to store ALL gnus configuration?  now,
> wouldn't that be cool?  Does it have enough features?
>
> http://asg.web.cmu.edu/cyrus/rfc/imsp.html
>
> Then, no syncing would be needed -- all information would be stored on
> the IMAP/IMSP server.

I haven't seen many life signs from either IMSP nor ACAP recently.

Re: No Gnus feature request: simpler file structure for synching

[Steinar Bang]

>>>>> Kai Grossjohann <kai@emptydomain.de>:

> That's why I suggested to port the .marks code to nntp.el; then you
> would have a file for each NNTP server which records only which
> articles have been read on that NNTP server.  That file could then
> be synced.

I would like that very much.  Eg. I read news.gmane.org both at home
and and work, while the USENET servers are different.  I would like to
avoid having to skim through the same gmane groups twice.

A problem though, is that the set of groups I see at home is different
than the set of groups I see at work (though there is a non-empty
intersection of the two).

Is there some way to cater for this?  Or will I have to subscribe to
the same set of gmane groups both places?

Re: No Gnus feature request: simpler file structure for synching

[Reiner Steib]

On Wed, Jan 07 2004, Steven E. Harris wrote:

[ gnusync shell script ]

Do you allow some off-topic remarks?

> #!/bin/sh
>
> CMD='rsync -e ssh -lptD -vuzb --exclude='*~''
      ^                                   ^  ^^
      o                                   c  oc

The »*~« part is not properly quoted (o=opening, c=closing quotes).
You may use one of the following:

|     "                                   '*~'"
|     '                                   "*~"'
|     '                                   \'*~\''

> REMOTE=Panix:~/temp/sync/gnus

Some Unix-Standard (SUS2, SYS3?  I don't remember exactly) suggest to
use lower case variables. [1]

[...]
>     $CMD -r $* $REMOTE/home/ .
              ^^ better use "$@" (including the quotes)

Bye, Reiner.

[1] http://www.ee.surrey.ac.uk/Teaching/Unix/unix8.html
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: No Gnus feature request: simpler file structure for synching

[Steven E. Harris]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> The »*~« part is not properly quoted (o=opening, c=closing quotes).

Yes, I had noticed that while fiddling around with the exclude
settings, which rsync was ignoring. I tried various quoting schemes,
including this odd one, and it finally seemed to work, so I left
it. The real problem had been the omission of the '=' character
between "--exclude" and "*~" tokens.

> You may use one of the following:
> |     "                                   '*~'"
> |     '                                   "*~"'
> |     '                                   \'*~\''

While these all look good, and I tried them all again in good faith,
it turns out none of these three alternatives work properly on my
system (Cygwin, Windows 2000). This one works like the first I posted
(second-level quotes removed):

cmd='rsync -e ssh -lptD -vuzb --exclude=*~'


>> REMOTE=Panix:~/temp/sync/gnus
>
> Some Unix-Standard (SUS2, SYS3?  I don't remember exactly) suggest to
> use lower case variables. [1]

I was just copying some other shell script idioms I've seen. The
uppercase names help make the variable names stand out. Enclosing them
in braces helps too. The argument you offer makes sense, though, so
I'll change the case.

>>     $CMD -r $* $REMOTE/home/ .
>               ^^ better use "$@" (including the quotes)

Why is one preferable to the other in this case? I read the Bash man
page on $* and $@ and can't really see how one should choose.

-- 
Steven E. Harris

Re: No Gnus feature request: simpler file structure for synching

[Paul Jarc]

"Steven E. Harris" <seh@panix.com> wrote:
> Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:
>>>     $CMD -r $* $REMOTE/home/ .
>>               ^^ better use "$@" (including the quotes)
>
> Why is one preferable to the other in this case? I read the Bash man
> page on $* and $@ and can't really see how one should choose.

If any of the arguments contain whitespace, then $* or $@, unquoted,
will re-split all the arguments, which is probably not what you want.
"$*", quoted, gives you one word containing all the arguments,
separated by spaces - also probably not what you want.  "$@", quoted,
gives you each argument exactly as-is, in its own word.


paul

Re: No Gnus feature request: simpler file structure for synching

[Kai Grossjohann]

Steinar Bang <sb@dod.no> writes:

> A problem though, is that the set of groups I see at home is different
> than the set of groups I see at work (though there is a non-empty
> intersection of the two).
>
> Is there some way to cater for this?  Or will I have to subscribe to
> the same set of gmane groups both places?

Well, the nnml.el code uses one .marks file per group.  So you could
just rsync the right files.

I wonder when/whether I'll get to it.  I'm not promising anything!

Kai

Re: No Gnus feature request: simpler file structure for synching

[Steinar Bang]

>>>>> Kai Grossjohann <kai@emptydomain.de>:

> Well, the nnml.el code uses one .marks file per group.  So you could
> just rsync the right files.

I need synching for nntp.  Is that being worked on?