Cache/Agent unification

Cache/Agent unification

[Simon Josefsson]

After reading some gnus-cache.el code I have some ideas, namely:

* Gnus Cache should cache all NOV headers.  (Unless disabled.)

  Right now it only cache NOV headers for cached articles.  But
  caching all NOV headers would increase speed when entering groups,
  which I think would be nice.  Header caching should only be enabled
  (per default) for backends that doesn't have their own local NOV
  files, i.e nntp.  An interface should be added which lets gnus-cache
  use the backends' NOV file if it has one, this would be used for
  nnml, nnfolder, nnimap.  (The nnimap NOV cache might be removed once
  gnus-cache handles this though.)

  Implementation detail: gnus-cache currently "braids" NOV lines.
  This turned out to be too inefficient in nnimap, but you can take
  the same approach nnimap did -- always maintain a contiguous NOV
  file. So if you've cached NOV for article 50-100 and Gnus wants NOV
  for article 1000 you fetch NOV for 101-1000 and saves that.

* Gnus Agent should use the NOV file from the cache/backend instead of
  maintaining its own.

  Enabling the agent thus enables NOV caching of gnus-cache.  This
  removes all the (duplicte) NOV stuff in gnus-agent.  If someone see
  a problem with this, please tell me.

* The Agent stores its articles in the cache.

  I see one problem with this change.  Currently, someone may have
  their cache on a remote filesystem to preserve space in the client.
  The separation between the cache and the agent makes this posssible.
  I'm not sure how serious this objection is.  Please comment.  Also
  please think of other things this might break.

* Articles should be inserted into the Agent (cache) when you read (or
  prefetch) them.

  Of course subject to any agent predicates.

* Articles should be retrieved from the cache if available by default.

  Currently you need to frob `gnus-use-cache' to get this behaviour.
  Summary/article commands which jog the cache should be available.
  Some indication in the article modeline should be present to
  indicate where the article comes from (cache or backend).

* Finally apply my old patches to make the Agent consider read
  articles as well.

  Currently the agent is hardcoded to never download read articles.
  When you use Gnus Agent as a disconnected mail client, you kind of
  expect everything in the group to be available (perhaps except very
  long articles).

The scope of all these changes are, I think, to make the agent/cache a
integrated part of Gnus.  It shouldn't require any configuration to
take advantage of the Agent.  All backends should be "part" of the
agent/cache program per default.  For remote backends, articles should
be saved per default (perhaps not? It requires lots of disk space).

Re: Cache/Agent unification

[Kai Großjohann]

I like your suggestions.  A couple of things come to mind, though:

* Currently, the cache is being used to keep articles around even
  after they've expired from the server.  But for disconnected
  operation, you expect the local cache to be a more-or-less exact
  mirror of the server state.  Hence, for disconnected operation you
  want local copies to disappear when they disappear from the server.

* You're right about the disk space requirements.  Maybe really old
  articles shouldn't be in the local cache.  Hm.

  Maybe it would be enough to provide a couple of commands to add
  articles to the cache and to remove them.  Examples:

  - Add each article as it is read.
  - Add all articles visible in the current summary buffer.
  - Add all unread articles.
  - Remove old, read, articles.
  - Never cache this group.
  - Always cache all articles from that group.

Thoughts?

kai
-- 
~/.signature: No such file or directory

Re: Cache/Agent unification

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> * Currently, the cache is being used to keep articles around even
>   after they've expired from the server.  But for disconnected
>   operation, you expect the local cache to be a more-or-less exact
>   mirror of the server state.  Hence, for disconnected operation you
>   want local copies to disappear when they disappear from the server.

I think the cache should store everything ever entered to it (modulo
some expiring mechanism to free disk space).  The Agent use marks and
a active range to identify which articles exists on the server or not.
Really-cached articles (that possibly doesn't exist on the server) are
marked as such as well.  The point is that storing everything in the
same directory is OK and saves space comparing to having two
directories with almost the same content.

All this assumes backends never re-use article numbers, of course.
I think we could introduce a uidvalidity-like concept in the cache
system fairly easily, if some backends wants to renumber articles.
(Nnimap could invalidate the cache directory when uidvalidity
changes.) OTOH backends that renumber articles are hard to cache in
any useful way so maybe we shouldn't bother.

> * You're right about the disk space requirements.  Maybe really old
>   articles shouldn't be in the local cache.  Hm.
> 
>   Maybe it would be enough to provide a couple of commands to add
>   articles to the cache and to remove them.  Examples:
> 
>   - Add each article as it is read.
>   - Add all articles visible in the current summary buffer.
>   - Add all unread articles.
>   - Remove old, read, articles.
>   - Never cache this group.
>   - Always cache all articles from that group.

Yes, some of these are good.  But it shouldn't be too manual, it would
be nice if the agent just did things automatically based on what
you've told it.  It should be possible to indicate how much disk space
you want to waste on a general level.  Three levels seems like a good
start:

 * Cache nothing
 * Cache headers, and manually entered (cache or agent) articles
 * Cache everything

Maybe the variable could support regexp's as well, indicating which
level should apply to each group.  Group parameters could be used as
well.

IMHO and just thinking out loud, of course.

Re: Cache/Agent unification

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> I think the cache should store everything ever entered to it (modulo
> some expiring mechanism to free disk space).

Hm.  I'm wondering if that fits my usage pattern.  (The one I intend
to develop, that is.)  I normally read email at work, but sometimes I
read it from home (especially on the weekends).  So I normally don't
need the cache.  But when I connect from home, I want the cache to do
like RET does on each group, then download all the articles visible
there.

Hm.

In fact, I might wish to write a special function to do this for me on
the machine at work, then use rsync to transfer ~/News, ~/Mail and
friends to my home machine, then run Gnus there.  This has the
advantage that I can read stuff from my normal at-work news server
from home, too.

Hm.  Hmmmmm....

kai
-- 
~/.signature: No such file or directory

Re: Cache/Agent unification

[ShengHuo ZHU]

Simon Josefsson <jas@extundo.com> writes:

>> * You're right about the disk space requirements.  Maybe really old
>>   articles shouldn't be in the local cache.  Hm.
>> 
>>   Maybe it would be enough to provide a couple of commands to add
>>   articles to the cache and to remove them.  Examples:
>> 
>>   - Add each article as it is read.
>>   - Add all articles visible in the current summary buffer.
>>   - Add all unread articles.
>>   - Remove old, read, articles.
>>   - Never cache this group.
>>   - Always cache all articles from that group.
> 
> Yes, some of these are good.  But it shouldn't be too manual, it would
> be nice if the agent just did things automatically based on what
> you've told it.  It should be possible to indicate how much disk space
> you want to waste on a general level.  Three levels seems like a good
> start:
> 
>  * Cache nothing
>  * Cache headers, and manually entered (cache or agent) articles
>  * Cache everything

The idea sounds good to me.  But I'd like to think Agent as the second
level cache instead of Agent/Cache unification.  Cache and Agent is
for different purposes.  Currently, agentization requires too many
manual operations, e.g.  users have to manually *fetch* headers and
articles and Agent doesn't save read articles while plugged.  Maybe we
should make Agent more automatic and more comprehensive.  My proposal
is simple: calling Agent functions in gnus-retrieve-headers and its
friends; saving headers and articles into nnagent when users read news
in plugged mode.  Of course, users can use `J s' to fetch all news.

Any thought?

ShengHuo

Re: Cache/Agent unification

[Kai Großjohann]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> The idea sounds good to me.  But I'd like to think Agent as the
> second level cache instead of Agent/Cache unification.  Cache and
> Agent is for different purposes.

Which different purposes?

The only difference I can see is that I use the cache to have a local
copy even after the server expired something.  But is that enough to
warrant a whole new module?

kai
-- 
~/.signature: No such file or directory

Re: Cache/Agent unification

[Christoph Conrad]

Hello ShengHuo,

you wrote:

    > Cache and Agent is for different purposes.

I agree with that, my backup behaviour relys on having both. I backup
the cache, but not the agentized articles. The cache should save in a
different (customizable) directory (as is) than the normal agent saves
his articles.

Best regards,
cu, -cc-
-- 
=> GNU Emacs Webring @ <http://www.gnusoftware.com/WebRing/> <=
Look Ma, this man can twist his fingers as if they were made of rubber,
isn't that amazing? -- Not really, he's been using emacs for years...!

Re: Cache/Agent unification

[Simon Josefsson]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> The idea sounds good to me.  But I'd like to think Agent as the second
> level cache instead of Agent/Cache unification.

Ah, yes, this is probably a better view of things.

> Cache and Agent is for different purposes.  Currently, agentization
> requires too many manual operations, e.g.  users have to manually
> *fetch* headers and articles and Agent doesn't save read articles
> while plugged.  Maybe we should make Agent more automatic and more
> comprehensive.  My proposal is simple: calling Agent functions in
> gnus-retrieve-headers and its friends; saving headers and articles
> into nnagent when users read news in plugged mode.  Of course, users
> can use `J s' to fetch all news.
>
> Any thought?

Something like this is what I was thinking about.  The Agent should
just work, it shouldn't require endless amount of configuration.  But
while where at it we'd might as well fix the NOV cache situation also
-- right now you have three local NOV files that contains roughly the
same stuff (if you're using nnimap, cache and agent at least).

Maybe I could apply the allow-agent-to-download-read-articles patch as
well.

Re: Cache/Agent unification

[Paul Jarc]

Simon Josefsson <jas@extundo.com> writes:
> All this assumes backends never re-use article numbers, of course.
> I think we could introduce a uidvalidity-like concept in the cache
> system fairly easily, if some backends wants to renumber articles.

Or, if the backand interface overhaul ever happens, we could let
backends identify articles by arbitrary opaque-to-Gnus Lisp forms.
Gnus could still use numbers internally, if that's convenient, but the
cache/agent stuff would have to use the forms from the backends for
persistent identification.

> OTOH backends that renumber articles are hard to cache in any useful
> way so maybe we shouldn't bother.

Well, nnimap is hard to cache, because there is no permanent
identification of an article.  But for nnmaildir, there is.


paul

Re: Cache/Agent unification

[Kai Großjohann]

prj@po.cwru.edu (Paul Jarc) writes:

> Simon Josefsson <jas@extundo.com> writes:
> > All this assumes backends never re-use article numbers, of course.
>> I think we could introduce a uidvalidity-like concept in the cache
>> system fairly easily, if some backends wants to renumber articles.
>
> Or, if the backand interface overhaul ever happens, we could let
> backends identify articles by arbitrary opaque-to-Gnus Lisp forms.

Way to go!

I suggested this many moons ago, but Lars declined.

kai
-- 
~/.signature: No such file or directory

Re: Cache/Agent unification

[Simon Josefsson]

prj@po.cwru.edu (Paul Jarc) writes:

> Simon Josefsson <jas@extundo.com> writes:
> > All this assumes backends never re-use article numbers, of course.
>> I think we could introduce a uidvalidity-like concept in the cache
>> system fairly easily, if some backends wants to renumber articles.

Quoting.. err.

> Or, if the backand interface overhaul ever happens, we could let
> backends identify articles by arbitrary opaque-to-Gnus Lisp forms.
> Gnus could still use numbers internally, if that's convenient, but the
> cache/agent stuff would have to use the forms from the backends for
> persistent identification.

I think using integers is slow enough today, using opaque types would
be significantly slower.

Re: Cache/Agent unification

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> I think using integers is slow enough today, using opaque types would
> be significantly slower.

Really?  What takes so long?

kai
-- 
~/.signature: No such file or directory

Re: Cache/Agent unification

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

>> I think using integers is slow enough today, using opaque types would
>> be significantly slower.
>
> Really?  What takes so long?

Try profiling gnus-range-* and enter a large group, last time I did
this quite some time was spent expanding/collapsing/merging ranges.
(And ranges were supposedly added to speed things up plain integer
lists as well.)

Re: Cache/Agent unification

[Paul Jarc]

Simon Josefsson <jas@extundo.com> writes:
> Try profiling gnus-range-* and enter a large group, last time I did
> this quite some time was spent expanding/collapsing/merging ranges.
> (And ranges were supposedly added to speed things up plain integer
> lists as well.)

They were supposed to speed things up, sure.  But if Gnus is spending
so much time on them, is it clear that they *do* speed things up?
Expanding and collapsing would be gone if we stopped using integers;
merging could still be done quickly with obarrays, I think.
prin1-to-string is helpful - or else we could say that articles must
be identified by strings, which probably wouldn't be much of a burden
on backends.


paul

Re: Cache/Agent unification

[Per Abrahamsen]

Simon Josefsson <jas@extundo.com> writes:

> * Gnus Cache should cache all NOV headers.  (Unless disabled.)

Yes!  Fetching NOV is often the largest time consuming phase when
entering groups with many already read messages on the server.  Even
with a fast connection.

Re: Cache/Agent unification

[ShengHuo ZHU]

Simon Josefsson <jas@extundo.com> writes:

> Something like this is what I was thinking about.  The Agent should
> just work, it shouldn't require endless amount of configuration.  But
> while where at it we'd might as well fix the NOV cache situation also
> -- right now you have three local NOV files that contains roughly the
> same stuff (if you're using nnimap, cache and agent at least).

The NOV files and articles in Gnus cache are maintained by the user.
We should not change it.  The NOV cache of Gnus agent can inhibit the
one of nnimap.  We could use the NOV cache for nntp and other backends
too.

ShengHuo

Re: Cache/Agent unification

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

>> The idea sounds good to me.  But I'd like to think Agent as the
>> second level cache instead of Agent/Cache unification.  Cache and
>> Agent is for different purposes.
>
> Which different purposes?

The cache already conflates two separate functions -- persistent
articles and, er, the cache.  I think that Simon is right in that the
second of these functions should be conflated with the Agent instead.

So instead of making the present cache do all these things, I think
the caching behavior should perhaps be moved to the Agent instead, and
integrate the Agent more closely with the rest of Gnus.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Cache/Agent unification

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> The scope of all these changes are, I think, to make the agent/cache a
> integrated part of Gnus.

I think all the proposed changes sound great, modulo my reply to Kai. 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Cache/Agent unification

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>>> The idea sounds good to me.  But I'd like to think Agent as the
>>> second level cache instead of Agent/Cache unification.  Cache and
>>> Agent is for different purposes.
>>
>> Which different purposes?
>
> The cache already conflates two separate functions -- persistent
> articles and, er, the cache.  I think that Simon is right in that the
> second of these functions should be conflated with the Agent instead.

Ah, formulated like that, it is much clearer.  Keeping persistant
articles in the ~/News/cache/ directory and moving cached articles
into ~/News/agent/ seems like a good approach.

Christoph, would you be happy with that?  (I belive you took backups
of your cache but not the agent, presumably because you use the
persistant article feature of the cache and not the cache itself?)

Hm.  Maybe there still is a gap though -- cached articles (not
persistant ones) are kept around (and shown in summary buffers) even
when they have expired on the server, if `gnus-cache-remove-articles'
is nil (or something).  But this is probably a bug.  A cache should
cache things, not change semantics.  Those who rely on that behaviour
could use persistant articles instead, I think.

Re: Cache/Agent unification

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> Hm.  Maybe there still is a gap though -- cached articles (not
> persistant ones) are kept around (and shown in summary buffers) even
> when they have expired on the server, if `gnus-cache-remove-articles'
> is nil (or something).  But this is probably a bug.  A cache should
> cache things, not change semantics.

Says who?  :-)

It's definitely a feature, and it's a feature that's shared by the
Agent.  Neither the Agent nor the cache cares one whit whether
articles are expired on the server.  They'll keep articles around
until the user makes them go away.

(The same goes with canceled/superseded articles, but that might be
considered a bug.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Cache/Agent unification

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Hm.  Maybe there still is a gap though -- cached articles (not
>> persistant ones) are kept around (and shown in summary buffers) even
>> when they have expired on the server, if `gnus-cache-remove-articles'
>> is nil (or something).  But this is probably a bug.  A cache should
>> cache things, not change semantics.
>
> Says who?  :-)
>
> It's definitely a feature, and it's a feature that's shared by the
> Agent.  Neither the Agent nor the cache cares one whit whether
> articles are expired on the server.  They'll keep articles around
> until the user makes them go away.
>
> (The same goes with canceled/superseded articles, but that might be
> considered a bug.)

Hm.  Many people use the cache for persistent articles (at least I
think of it like that -- I hit `!' to let a message stick around
forever).  So that part of the cache is misnamed, IMHO.

Maybe we need to think of two (or three?) new
packages/functionalities, and then we can choose new names and then we
can think of how to move old functionality to the new names.

kai
-- 
~/.signature: No such file or directory

Re: Cache/Agent unification

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Hm.  Many people use the cache for persistent articles (at least I
> think of it like that -- I hit `!' to let a message stick around
> forever).  So that part of the cache is misnamed, IMHO.
>
> Maybe we need to think of two (or three?) new
> packages/functionalities, and then we can choose new names and then we
> can think of how to move old functionality to the new names.

I think there are basically two functions that people want to have.

  1) Persistent articles
     These are articles that are important for people,
     and they want them to remain there forever.

  2) A cache
     These are articles that are stored locally to make
     Gnus work faster, or work when not connected to
     the net.

So I think that the cache should be renamed and do its persistent
article thing, and the cache should be moved to the Agent, which
should be more aggressive.

(Hey!  I really like the new TAB action.  Makes everything all
pretty.) 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Cache/Agent unification

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> Hm.  Maybe there still is a gap though -- cached articles (not
>> persistant ones) are kept around (and shown in summary buffers) even
>> when they have expired on the server, if `gnus-cache-remove-articles'
>> is nil (or something).  But this is probably a bug.  A cache should
>> cache things, not change semantics.
>
> Says who?  :-)
>
> It's definitely a feature, and it's a feature that's shared by the
> Agent.  Neither the Agent nor the cache cares one whit whether
> articles are expired on the server.  They'll keep articles around
> until the user makes them go away.

If the cache also contain persistent articles we'll have a problem
when people want to backup the cache directory (to have backups of
server-expired articles) but not their agent directory.  Unifying the
agent and cache is no good then.  Admittedly, the articles could still
be in separate directories and the agent (or the cache) could look
into the cache's (or the agent's) directory before storing/fetching
articles, but that's sort of yucky (and also a problem when articles
are removed in one of the cache/agent but not the other).  I thought
it would be nice if we could have ~/News/persistant/ and ~/News/agent/
only.

But I'm not sure what the usage pattern is here.  Maybe it's enough to
separate persistent articles into a directory that people can backup.

Hm. I'm assuming that you want a agent mode that synched the agent
with what's on the server.  If you read mail on a remote server (IMAP)
you kind of expect that articles you've deleted from the server (with
another client or Gnus instance) are deleted from the agent.  Then you
could use the cache to keep expired articles around.  OTOH you could
also use persistent articles for that.  If the latter is OK, we could
remove the persistentness thing from the cache, which would make it
possible to merge the cache and agent directories.

Maybe there the persistent article store could be a `expiry-target'.
So when cache/agent articles expire from the server, the local copies
are run through this process and people that want to keep them around
can make them end up in the persistent article store.

Err. Or something.

Re: Cache/Agent unification

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> If the cache also contain persistent articles we'll have a problem
> when people want to backup the cache directory (to have backups of
> server-expired articles) but not their agent directory.

Well, unifying the persistent articles and the agent is not a good
idea, I think.  And I don't think people really want to back up the
cached articles.  (As opposed to the persistent articles.) 

> But I'm not sure what the usage pattern is here.  Maybe it's enough to
> separate persistent articles into a directory that people can backup.

I think so.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen