Re: which gnus files used to hold gnus state? (for manual)

[kai.grossjohann@gmx.net (Kai Großjohann)]

John Owens <jowens@ece.ucdavis.edu> writes:

> I'd like to get a list of the files used to hold gnus state. This is
> for potential inclusion in the manual (after discussion with Lars)
> and would be important for users who are interested in synching the
> gnus state of two machines (say, a laptop and a desktop).

Sounds like a real cool project.  Alas, my knowledge is rather
limited.  But I'll share what I know.

> 1) general gnus state files that apply to any backend - .newsrc,    
> .newsrc.eld, .emacs, .gnus. Is that the complete list? This could go
> in manual section 1.8, "Startup Files".

You could add site-start.el and default.el, if you mention .emacs.
But those are general Emacs files, not Gnus-specific files.  Maybe
you should mention which files are Gnus specific and which aren't.

.newsrc, .newsrc.eld, .gnus are Gnus specific.  The rest isn't.

> 2) gnus state files that apply to each backend. I am particularly 
> concerned with nnimap, but I'm not sure there are any state files in
> gnus that are associated with it (?) because maybe the imap server
> keeps track of that internally? Evidently nnmail uses .nnmail-cache?

Maybe you should group the backends in some way -- nnmail is not a
backend per se, it's a group of backends.

There is also gnus-registry.el which comes into play.  But maybe Ted
should say something about it.

> I'm aware there are News and Mail folders but I'm not sure what is 
> supposed to live in them. How does this interact with nnimap - where 
> are imap folders located in the directory structure and how does this 
> work with gnus? 
>
> This could go in section 6.3 where the backends are discussed.

We have ~/Mail.  It contains the file ~/Mail/active for an
nnmail-based backend.  It also contains files and/or directories for
the groups from nnmail-based backends.  So people who are using nnml
will see the list of nnml groups in ~/Mail/active, and the
nnml:foo.bar group lives in a directory called ~/Mail/foo/bar or
~/Mail/foo.bar, depending on nnmail-use-long-file-names.

Then ~/Mail contains Incoming* files which are used by the
mail-sources mechanism to fetch mail.  All mails fetched by
mail-sources are first dumped into such an ~/Incoming* file, and from
there they are distributed into the groups (this is called splitting).
It seems that these files are usually deleted right after use, but
mail-source-delete-incoming can be frobbed to change this.

Then we have ~/.gnus-mail-crashbox or similar (forget the exact name)
which plays a role similar to the ~/Mail/Incoming* files, but I don't
know details.  Any experts out there who can help?

Now let's talk about the ~/News directory.  It contains a number of
files and subdirs, some related to news, some related to mail.  It's
rather confusing, I'm afraid.

First of all, we have ~/News/drafts which contains the nndraft:drafts
and nndraft:queue groups, and also the nndraft:delayed group if you
use that.  (Some people are surprised to see a directory called
~/News/drafts/drafts -- that's just the nndraft:drafts group.)

Secondly, we have a lot of *.SCORE and *.ADAPT files which are used
for scoring and adaptive scoring, you guessed it.

~/News/agent and ~/News/cache will be discussed below.

> 3) gnus state files that apply to the Agent. Are there any?
> (~/News/agent should contain ... ?) How about the cache? Where do
> cached files live? (~/News/cache, but what files?) If I delete all the
> cached files will they be refetched by the Agent? I'm, again,
> particularly concerned with the imap interaction with the agent, and
> want to know how files work with it. (Ideally I can have
> semi-up-to-date IMAP mail folders on my laptop and can work with
> those, and when I plug in they will be auto-updated manually? or
> automatically?)
>
> This could go in section 6.8 which deals with the Agent.

Please note that ~/News/cache has nothing to do with the agent,
though there has been talk about unifying the cache and the agent.
The talk comes from the fact that they have a similar purpose.

The cache lives in ~/News/cache and the agent lives in ~/News/agent.

The cache is the older part of Gnus.  The cache is meant to be used
manually by the user while browsing: the user hits some key and the
article is entered into the cache.  This means that the article will
stay around even after the server has expired it.  So, normally, the
cache is only useful for nntp, and perhaps nnimap.  But of course you
can count on creative abuses of the cache even for nnml and its ilk
;-)

~/News/cache contains an active file (listing all cached groups) and
a directory for each group.  (My directories have names like
~/News/cache/nntp+dfn:comp.emacs, but I gather that it might be
~/News/cache/nntp/dfn/comp/emacs or the like if
gnus-use-long-file-name is nil.)  Each group directory contains a
.overview file and the articles.

Now, for the agent.  There is a subdir ~/News/agent/lib which
contains the categories and the servers files.  Further subdirs are
for servers/groups.  (Presumably the exact directory layout depends
on some *-use-long-file-name(s) variable.)  A directory for a server
contains a subdir agent.lib with the file agent.lib/active.  All
other subdirs are groups (in foo.bar or foo/bar format).  Each group
contains a file .agentview and a file .overview and the messages.

What does .agentview contain? --> Anyone?

If you delete cached files, they will NOT be refetched by the agent:
you need to cache a message to insert it into the cache.  The agent
has nothing to do with this.

Yes, you can have an ~/News/agent directory which contains all the
messages from your IMAP server.  Well, probably not ALL messages.
And as long as you're unplugged, Gnus will use those local files.

There are a number of commands for updating the state.  Some commands
read information from the server and change the local directory to
match the situation on the server, other commands work in the other
direction.

In particular, `J s' fetches messages from the server (according to
criteria you specify), and `J S' sends queued outgoing messages, and
`J Y' (?) sends marks changes to the server.  Marks changes means it
tells the server which messages you have read or ticked etc.
Actually, `J Y' happens semi-automatically -- after connecting to the
server Gnus asks you whether to send the marks changes.

> The overall goal would be to know exactly what files to sync if I want
> to mirror my gnus setup on computer B from computer A. Hopefully in
> the next generation of gnus there will be a list of state-containing
> files and an automated way to sync them (gnus-sync-state-files with a
> configurable sync command like "rsync -e ssh ..."). This could be
> useful for users who use a laptop and want to sync before going on
> the road.

Nifty.


If you want to replicate the cache, replicate ~/News/cache.  Similar
for the agent.  And for mail, replicate ~/Mail.

It is difficult (but doable) to replicate just *parts* of the cache,
or parts of the agent, or parts of your mail.

For the cache and the agent, the problem is the active files.  You
need to have the right lines from the active files.  (I'm not sure if
it works to replicate the whole active file but only some groups.
Anyone?)

For mail, there are some backends where it is fairly painless to
replicate just some groups.  This is true for nnml and nnfolder: you
just replicate the right directory, plus the .marks file (nnml) or
the *.mrk file (nnfolder), and Bob's your uncle.  It is also probably
true for nnmaildir, but I'm no expert.  Anyone?  (Actually, the
.marks file lives inside the directory for the nnml group, so just
the directory for the nnml group is good enough.  For nnfolder
groups, you replicate two files -- the group file itself, and the
*.mrk file.)

But in these cases, you also have the active file problem.  Hm.

But it is very difficult to replicate just some groups for other mail
backends.  For example, nnmh does not have a .marks file and thus the
read/ticked/dormant/... marks are ONLY stored in the .newsrc.eld file,
and if you transfer that without also transferring all the groups --
hm, strange things may happen.  Is there anyone who has tried?

Hm.  Now that I think of it, the .newsrc.eld problem also may happen
with nnml and nnfolder: if you transfer .newsrc.eld, then you're in
trouble.

Hm.

Hm.

Hm.

So, it boils down to this: if you transfer .newsrc.eld, you got to
transfer the whole hog.  If you abstain from .newsrc.eld, then you
can do the some-groups trick alluded to for nnml and nnfolder (and
probably nnmaildir).

Stuff from cache and agent aren't mentioned in .newsrc.eld, so there
is no problem.

Above, when I say .newsrc.eld, I always also mean .newsrc.
-- 
This line is not blank.