which gnus files used to hold gnus state? (for manual)

which gnus files used to hold gnus state? (for manual)

[John Owens]

I'd like to get a list of the files used to hold gnus state. This is
for potential inclusion in the manual (after discussion with Lars)
and would be important for users who are interested in synching the
gnus state of two machines (say, a laptop and a desktop).

I think this should be done in three parts and I hope gnus developers
can specify the relevant files for each of the parts. Also, the
default location for these files should be specified, and also the
gnus variables that can be used to change the defaults.

(I'm also putting some questions in - if you can answer my questions
I'd certainly appreciate it.)

1) general gnus state files that apply to any backend - .newsrc,    
.newsrc.eld, .emacs, .gnus. Is that the complete list? This could go
in manual section 1.8, "Startup Files".

2) gnus state files that apply to each backend. I am particularly 
concerned with nnimap, but I'm not sure there are any state files in
gnus that are associated with it (?) because maybe the imap server
keeps track of that internally? Evidently nnmail uses .nnmail-cache?
 
I'm aware there are News and Mail folders but I'm not sure what is 
supposed to live in them. How does this interact with nnimap - where 
are imap folders located in the directory structure and how does this 
work with gnus? 

This could go in section 6.3 where the backends are discussed.

3) gnus state files that apply to the Agent. Are there any?
(~/News/agent should contain ... ?) How about the cache? Where do
cached files live? (~/News/cache, but what files?) If I delete all the
cached files will they be refetched by the Agent? I'm, again,
particularly concerned with the imap interaction with the agent, and
want to know how files work with it. (Ideally I can have
semi-up-to-date IMAP mail folders on my laptop and can work with
those, and when I plug in they will be auto-updated manually? or
automatically?)

This could go in section 6.8 which deals with the Agent.

The overall goal would be to know exactly what files to sync if I want
to mirror my gnus setup on computer B from computer A. Hopefully in
the next generation of gnus there will be a list of state-containing
files and an automated way to sync them (gnus-sync-state-files with a
configurable sync command like "rsync -e ssh ..."). This could be
useful for users who use a laptop and want to sync before going on
the road.

JDO

Re: which gnus files used to hold gnus state? (for manual)

[Kai Großjohann]

John Owens <jowens@ece.ucdavis.edu> writes:

> I'd like to get a list of the files used to hold gnus state. This is
> for potential inclusion in the manual (after discussion with Lars)
> and would be important for users who are interested in synching the
> gnus state of two machines (say, a laptop and a desktop).

Sounds like a real cool project.  Alas, my knowledge is rather
limited.  But I'll share what I know.

> 1) general gnus state files that apply to any backend - .newsrc,    
> .newsrc.eld, .emacs, .gnus. Is that the complete list? This could go
> in manual section 1.8, "Startup Files".

You could add site-start.el and default.el, if you mention .emacs.
But those are general Emacs files, not Gnus-specific files.  Maybe
you should mention which files are Gnus specific and which aren't.

.newsrc, .newsrc.eld, .gnus are Gnus specific.  The rest isn't.

> 2) gnus state files that apply to each backend. I am particularly 
> concerned with nnimap, but I'm not sure there are any state files in
> gnus that are associated with it (?) because maybe the imap server
> keeps track of that internally? Evidently nnmail uses .nnmail-cache?

Maybe you should group the backends in some way -- nnmail is not a
backend per se, it's a group of backends.

There is also gnus-registry.el which comes into play.  But maybe Ted
should say something about it.

> I'm aware there are News and Mail folders but I'm not sure what is 
> supposed to live in them. How does this interact with nnimap - where 
> are imap folders located in the directory structure and how does this 
> work with gnus? 
>
> This could go in section 6.3 where the backends are discussed.

We have ~/Mail.  It contains the file ~/Mail/active for an
nnmail-based backend.  It also contains files and/or directories for
the groups from nnmail-based backends.  So people who are using nnml
will see the list of nnml groups in ~/Mail/active, and the
nnml:foo.bar group lives in a directory called ~/Mail/foo/bar or
~/Mail/foo.bar, depending on nnmail-use-long-file-names.

Then ~/Mail contains Incoming* files which are used by the
mail-sources mechanism to fetch mail.  All mails fetched by
mail-sources are first dumped into such an ~/Incoming* file, and from
there they are distributed into the groups (this is called splitting).
It seems that these files are usually deleted right after use, but
mail-source-delete-incoming can be frobbed to change this.

Then we have ~/.gnus-mail-crashbox or similar (forget the exact name)
which plays a role similar to the ~/Mail/Incoming* files, but I don't
know details.  Any experts out there who can help?

Now let's talk about the ~/News directory.  It contains a number of
files and subdirs, some related to news, some related to mail.  It's
rather confusing, I'm afraid.

First of all, we have ~/News/drafts which contains the nndraft:drafts
and nndraft:queue groups, and also the nndraft:delayed group if you
use that.  (Some people are surprised to see a directory called
~/News/drafts/drafts -- that's just the nndraft:drafts group.)

Secondly, we have a lot of *.SCORE and *.ADAPT files which are used
for scoring and adaptive scoring, you guessed it.

~/News/agent and ~/News/cache will be discussed below.

> 3) gnus state files that apply to the Agent. Are there any?
> (~/News/agent should contain ... ?) How about the cache? Where do
> cached files live? (~/News/cache, but what files?) If I delete all the
> cached files will they be refetched by the Agent? I'm, again,
> particularly concerned with the imap interaction with the agent, and
> want to know how files work with it. (Ideally I can have
> semi-up-to-date IMAP mail folders on my laptop and can work with
> those, and when I plug in they will be auto-updated manually? or
> automatically?)
>
> This could go in section 6.8 which deals with the Agent.

Please note that ~/News/cache has nothing to do with the agent,
though there has been talk about unifying the cache and the agent.
The talk comes from the fact that they have a similar purpose.

The cache lives in ~/News/cache and the agent lives in ~/News/agent.

The cache is the older part of Gnus.  The cache is meant to be used
manually by the user while browsing: the user hits some key and the
article is entered into the cache.  This means that the article will
stay around even after the server has expired it.  So, normally, the
cache is only useful for nntp, and perhaps nnimap.  But of course you
can count on creative abuses of the cache even for nnml and its ilk
;-)

~/News/cache contains an active file (listing all cached groups) and
a directory for each group.  (My directories have names like
~/News/cache/nntp+dfn:comp.emacs, but I gather that it might be
~/News/cache/nntp/dfn/comp/emacs or the like if
gnus-use-long-file-name is nil.)  Each group directory contains a
.overview file and the articles.

Now, for the agent.  There is a subdir ~/News/agent/lib which
contains the categories and the servers files.  Further subdirs are
for servers/groups.  (Presumably the exact directory layout depends
on some *-use-long-file-name(s) variable.)  A directory for a server
contains a subdir agent.lib with the file agent.lib/active.  All
other subdirs are groups (in foo.bar or foo/bar format).  Each group
contains a file .agentview and a file .overview and the messages.

What does .agentview contain? --> Anyone?

If you delete cached files, they will NOT be refetched by the agent:
you need to cache a message to insert it into the cache.  The agent
has nothing to do with this.

Yes, you can have an ~/News/agent directory which contains all the
messages from your IMAP server.  Well, probably not ALL messages.
And as long as you're unplugged, Gnus will use those local files.

There are a number of commands for updating the state.  Some commands
read information from the server and change the local directory to
match the situation on the server, other commands work in the other
direction.

In particular, `J s' fetches messages from the server (according to
criteria you specify), and `J S' sends queued outgoing messages, and
`J Y' (?) sends marks changes to the server.  Marks changes means it
tells the server which messages you have read or ticked etc.
Actually, `J Y' happens semi-automatically -- after connecting to the
server Gnus asks you whether to send the marks changes.

> The overall goal would be to know exactly what files to sync if I want
> to mirror my gnus setup on computer B from computer A. Hopefully in
> the next generation of gnus there will be a list of state-containing
> files and an automated way to sync them (gnus-sync-state-files with a
> configurable sync command like "rsync -e ssh ..."). This could be
> useful for users who use a laptop and want to sync before going on
> the road.

Nifty.


If you want to replicate the cache, replicate ~/News/cache.  Similar
for the agent.  And for mail, replicate ~/Mail.

It is difficult (but doable) to replicate just *parts* of the cache,
or parts of the agent, or parts of your mail.

For the cache and the agent, the problem is the active files.  You
need to have the right lines from the active files.  (I'm not sure if
it works to replicate the whole active file but only some groups.
Anyone?)

For mail, there are some backends where it is fairly painless to
replicate just some groups.  This is true for nnml and nnfolder: you
just replicate the right directory, plus the .marks file (nnml) or
the *.mrk file (nnfolder), and Bob's your uncle.  It is also probably
true for nnmaildir, but I'm no expert.  Anyone?  (Actually, the
.marks file lives inside the directory for the nnml group, so just
the directory for the nnml group is good enough.  For nnfolder
groups, you replicate two files -- the group file itself, and the
*.mrk file.)

But in these cases, you also have the active file problem.  Hm.

But it is very difficult to replicate just some groups for other mail
backends.  For example, nnmh does not have a .marks file and thus the
read/ticked/dormant/... marks are ONLY stored in the .newsrc.eld file,
and if you transfer that without also transferring all the groups --
hm, strange things may happen.  Is there anyone who has tried?

Hm.  Now that I think of it, the .newsrc.eld problem also may happen
with nnml and nnfolder: if you transfer .newsrc.eld, then you're in
trouble.

Hm.

Hm.

Hm.

So, it boils down to this: if you transfer .newsrc.eld, you got to
transfer the whole hog.  If you abstain from .newsrc.eld, then you
can do the some-groups trick alluded to for nnml and nnfolder (and
probably nnmaildir).

Stuff from cache and agent aren't mentioned in .newsrc.eld, so there
is no problem.

Above, when I say .newsrc.eld, I always also mean .newsrc.
-- 
This line is not blank.

Re: which gnus files used to hold gnus state? (for manual)

[Andreas Fuchs]

Today, Kai Großjohann <kai.grossjohann@gmx.net> wrote:
> Then we have ~/.gnus-mail-crashbox or similar (forget the exact name)
> which plays a role similar to the ~/Mail/Incoming* files, but I don't
> know details.  Any experts out there who can help?

Not an expert, but had sufficient exposure to ~/.emacs-crash-box to be
able to comment on it.

It's a file in which a mail or mailbox gets saved which makes gnus wet
its pants - e.g. sometimes procmail, delivering to a maildir would open
a file, write "F" to it (for "From ..."), and at the same time, gnus
would open the file and try to rip it out from under procmail's
feet. Now, you have two parts: one ~/.emacs-crash-box with "F" in it,
and if you delete it, you get another with the rest of the mail,
starting with "rom ...". Happened to me all the time until I switched to
gnus-centric splitting via bbdb and split-fancy (-:

> Now let's talk about the ~/News directory.  It contains a number of
> files and subdirs, some related to news, some related to mail.  It's
> rather confusing, I'm afraid.

Right. it is. Seems to be artifacts from older days of gnusing.

> The cache is the older part of Gnus.  The cache is meant to be used
> manually by the user while browsing: the user hits some key and the
> article is entered into the cache.  This means that the article will
> stay around even after the server has expired it.  So, normally, the
> cache is only useful for nntp, and perhaps nnimap.  But of course you
> can count on creative abuses of the cache even for nnml and its ilk
> ;-)

The cache has always struck me as a very strange facility. Why would I
want an article around after I have read it (and not marked it as
ticked or dormant) anyway?

>> The overall goal would be to know exactly what files to sync if I
>> want to mirror my gnus setup on computer B from computer A. Hopefully
>> in the next generation of gnus there will be a list of
>> state-containing files and an automated way to sync them
>> (gnus-sync-state-files with a configurable sync command like "rsync
>> -e ssh ..."). This could be useful for users who use a laptop and
>> want to sync before going on the road.
> 
> Nifty.
> 
> 
> If you want to replicate the cache, replicate ~/News/cache.  Similar
> for the agent.  And for mail, replicate ~/Mail.

If I might suggest here the use (but first, implementation for gnus) of
RMS, the Robust Mail Store. It seemed like a great idea when I first
read the paper. Maybe you can snarf some ideas from it, too.

http://www.informatik.uni-freiburg.de/~thiemann/papers/mailstore.pdf

> But in these cases, you also have the active file problem.  Hm.

See the URL for a possibly complete solution to that problem.

HTH,
-- 
Andreas Fuchs, <asf@acm.org>, asf@jabber.at, antifuchs
irc.freenode.net's #emacs - online emacs advice from IRC addicts

Re: which gnus files used to hold gnus state? (for manual)

[Kevin Greiner]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> John Owens <jowens@ece.ucdavis.edu> writes:
>
>> I'd like to get a list of the files used to hold gnus state. This is
>> for potential inclusion in the manual (after discussion with Lars)
>> and would be important for users who are interested in synching the
>> gnus state of two machines (say, a laptop and a desktop).
>

snip...

>
> Now, for the agent.  There is a subdir ~/News/agent/lib which
> contains the categories and the servers files.  Further subdirs are
> for servers/groups.  (Presumably the exact directory layout depends
> on some *-use-long-file-name(s) variable.)  A directory for a server
> contains a subdir agent.lib with the file agent.lib/active.  All
> other subdirs are groups (in foo.bar or foo/bar format).  Each group
> contains a file .agentview and a file .overview and the messages.
>
> What does .agentview contain? --> Anyone?

The .agentview contains per-article timestamps that are used by the
agent's expiration mechanism.

The .overview file contains the article headers.

>> The overall goal would be to know exactly what files to sync if I want
>> to mirror my gnus setup on computer B from computer A. Hopefully in
>> the next generation of gnus there will be a list of state-containing
>> files and an automated way to sync them (gnus-sync-state-files with a
>> configurable sync command like "rsync -e ssh ..."). This could be
>> useful for users who use a laptop and want to sync before going on
>> the road.
>
> Nifty.
>

snip ...

> So, it boils down to this: if you transfer .newsrc.eld, you got to
> transfer the whole hog.  If you abstain from .newsrc.eld, then you
> can do the some-groups trick alluded to for nnml and nnfolder (and
> probably nnmaildir).

But your seen marks are in .newsrc.eld.  If you don't include it, how
will you know which articles have been read?

> Stuff from cache and agent aren't mentioned in .newsrc.eld, so there
> is no problem.

There's no problem so long as the two machines are sharing the same
servers.  The article files in both the agent and the cache are named
after the article number.  You'll get garbage if you try to use these
files with any server except the one from which they were fetched (I
realize that this isn't what you proposed.  I just want to make sure
that anyone reading this understands).

> Above, when I say .newsrc.eld, I always also mean .newsrc.

Kevin

Re: which gnus files used to hold gnus state? (for manual)

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> kai.grossjohann@gmx.net (Kai Großjohann) writes:
>
>> So, it boils down to this: if you transfer .newsrc.eld, you got to
>> transfer the whole hog.  If you abstain from .newsrc.eld, then you
>> can do the some-groups trick alluded to for nnml and nnfolder (and
>> probably nnmaildir).
>
> But your seen marks are in .newsrc.eld.  If you don't include it, how
> will you know which articles have been read?

Whoops.  I guess that means that the seen marks should be written to
the .marks/*.mrk files.  WDYT?

>> Stuff from cache and agent aren't mentioned in .newsrc.eld, so there
>> is no problem.
>
> There's no problem so long as the two machines are sharing the same
> servers.  The article files in both the agent and the cache are named
> after the article number.  You'll get garbage if you try to use these
> files with any server except the one from which they were fetched (I
> realize that this isn't what you proposed.  I just want to make sure
> that anyone reading this understands).

Righty-oh.  Thanks for pointing this out.
-- 
This line is not blank.

Re: which gnus files used to hold gnus state? (for manual)

[Ted Zlatanov]

On Fri, 16 May 2003, kai.grossjohann@gmx.net wrote:
> John Owens <jowens@ece.ucdavis.edu> writes:
> 
>> I'd like to get a list of the files used to hold gnus state. This
>> is for potential inclusion in the manual (after discussion with
>> Lars) and would be important for users who are interested in
>> synching the gnus state of two machines (say, a laptop and a
>> desktop).
> 
> There is also gnus-registry.el which comes into play.  But maybe Ted
> should say something about it.

The registry basically stores data that would normally go into the
newrs.eld file, into ~/.gnus.registry.eld.  You have to enable and
load the gnus-registry.el package, though.  Also, a missing/corrupted
registry would not impact anything, except send articles to the
default/wrong mailbox with parent splitting, or register messages as
spam or ham incorrectly or not at all.  So synchronization of the
registry, as it's used now and as it will be used in my plans, is not
necessary to preserve the state of Gnus.

Ted