can back end keep track of *all* group info?

can back end keep track of *all* group info?

[Paul Grayson]

I'm writing a simple server that will allow me to access a huge
maildir (10,000+ messages).  I split the messages virtually - my
server applies splitting rules *while* you browse through your mail. 

My server keeps track of marks, and could report the number of
unread/flagged messages in each group for display in the *Group*
buffer.  I don't want my server to count the total number of messages
in a group - that will take too long - and I want to avoid the nnimap
problem of always having the unread/flagged count out of sync with the
server.

I love everything Gnus does, and really want to use it as a front end
to this server - is there any way to write it a back end?  I've
experimented a little, and it seems that Gnus generally requests much
more information than it uses, but checks too rarely (with, for
example, nnchoke-request-update-info) to keep the information
up-to-date.  Is there any way to trick Gnus into letting the server
keep track of everything, or would it mean rewriting a significant
part of the front end?  Are there any plans for allowing this in the
future?

If I don't want to rewrite parts of Gnus, is it possible to write my
own group-browsing mode and connect it to the Gnus summary mode?

If you think this is not worth the work, is there any other email
back end that lets you instantly re-split tens of thousands of
messages?

Thanks!
-Paul

Re: can back end keep track of *all* group info?

[Simon Josefsson]

Paul Grayson <pgrayson+ding@ks.uiuc.edu> writes:

> My server keeps track of marks, and could report the number of
> unread/flagged messages in each group for display in the *Group*
> buffer.  I don't want my server to count the total number of messages
> in a group - that will take too long - and I want to avoid the nnimap
> problem of always having the unread/flagged count out of sync with the
> server.

Which problem is this?  Do you mean the one that unread/flagged count
in Gnus is only an estimate?  That applies to all backends, if it is
noticable or not depends on the backend (and in nnimap's case, on the
remote server).

M-g on the group makes Gnus update the estimate with real information
though.

Making the Group buffer use the real values from the backend, when
they are able to provide that data, should fix this problem and be
very useful.  Shouldn't be too difficult to do, I think.

> I love everything Gnus does, and really want to use it as a front end
> to this server - is there any way to write it a back end?  I've
> experimented a little, and it seems that Gnus generally requests much
> more information than it uses, but checks too rarely (with, for
> example, nnchoke-request-update-info) to keep the information
> up-to-date.  Is there any way to trick Gnus into letting the server
> keep track of everything, or would it mean rewriting a significant
> part of the front end?  Are there any plans for allowing this in the
> future?
>
> If I don't want to rewrite parts of Gnus, is it possible to write my
> own group-browsing mode and connect it to the Gnus summary mode?

If you implement `gnus-request-set-mark' your backend should at least
receive all flags as set by Gnus.  The other way is handled via
`nnchoke-request-update-info', but I'd agree that this is called too
many times and still not enough times. (This is way nnimap doesn't
have a `nnimap-request-update-info' but rather calls this function
internally upon group entry, in `nnimap-request-group' -- which
doesn't seem to cause any problems.)

> If you think this is not worth the work, is there any other email
> back end that lets you instantly re-split tens of thousands of
> messages?

You could have a look at Wanderlust, maybe their backend interface is
cleaner.

Re: can back end keep track of *all* group info?

[Kai Großjohann]

Paul Grayson <pgrayson+ding@ks.uiuc.edu> writes:

> My server keeps track of marks, and could report the number of
> unread/flagged messages in each group for display in the *Group*
> buffer.  I don't want my server to count the total number of messages
> in a group - that will take too long - and I want to avoid the nnimap
> problem of always having the unread/flagged count out of sync with the
> server.

I think that there is already a feature in Gnus where the backend may
report the number of unread messages for each group.  But the
frontend part of Gnus doesn't use it in the group buffer.

I think if you just change the code for the %y spec in
gnus-group-line-format to see if the backend has told you the right
number and to use that number in lieu of the current estimate based
on min and max article number, then you should be fine.

Alas, I don't know what exactly is that mechanism which allows the
backend to report the %y number directly.  Simon?  I think you were
the one telling me this.

kai
-- 
Simplification good!  Oversimplification bad!  (Larry Wall)

Re: can back end keep track of *all* group info?

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> I think if you just change the code for the %y spec in
> gnus-group-line-format to see if the backend has told you the right
> number and to use that number in lieu of the current estimate based
> on min and max article number, then you should be fine.

Yes.  I don't think this is very difficult.

> Alas, I don't know what exactly is that mechanism which allows the
> backend to report the %y number directly.  Simon?  I think you were
> the one telling me this.

I think I was thinking of nnchoke-request-group, but that isn't really
called until the group is entered.  Hm.  Could we extend
nnchoke-request-list perhaps?  If the backend knows the total number
of articles in the group it can report it.  It should be a optional
parameter.

Re: can back end keep track of *all* group info?

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> I think I was thinking of nnchoke-request-group, but that isn't really
> called until the group is entered.  Hm.  Could we extend
> nnchoke-request-list perhaps?  If the backend knows the total number
> of articles in the group it can report it.  It should be a optional
> parameter.

Right.

kai
-- 
Simplification good!  Oversimplification bad!  (Larry Wall)

Re: can back end keep track of *all* group info?

[Paul Jarc]

Simon Josefsson <jas@extundo.com> wrote:
> I think I was thinking of nnchoke-request-group, but that isn't really
> called until the group is entered.  Hm.  Could we extend
> nnchoke-request-list perhaps?  If the backend knows the total number
> of articles in the group it can report it.  It should be a optional
> parameter.

We could use -retrieve-groups.  There, the backend already has the
option of providing this information, or not.  So Gnus could use that
count, if it's given, or estimate if it's not.


paul

Re: can back end keep track of *all* group info?

[Paul Jarc]

Regarding the subject: backends can store articles and marks, but not
group parameters.  At least, not as well.

Paul Grayson <pgrayson+ding@ks.uiuc.edu> wrote:
> I'm writing a simple server that will allow me to access a huge
> maildir (10,000+ messages).

You might have a look at nnmaildir.  It's in CVS now and is documented
(for now) at <URL:http://multivac.cwru.edu./nnmaildir/>.

> I split the messages virtually - my server applies splitting rules
> *while* you browse through your mail.

nnmaildir doesn't mind if you do that.

> My server keeps track of marks,

nnmaildir also does that.  Without knowing how your server does it,
it's tough to say how hard it would be to hook them together.

> I don't want my server to count the total number of messages in a
> group - that will take too long -

nnmaildir keeps a list of articles (not their contents, though) in
memory for each group.  I suppose that might be a problem for you.

> and I want to avoid the nnimap problem of always having the
> unread/flagged count out of sync with the server.

nnmaildir renumbers its articles when you close and reopen the server,
so those counts aren't so far off.  The price of this is that
nnmaildir doesn't work with the Gnus Agent or cache.

> I love everything Gnus does, and really want to use it as a front end
> to this server - is there any way to write it a back end?

There's some material in the manual about the backend interface.
That's where I started when writing nnmaildir.

> I've experimented a little, and it seems that Gnus generally
> requests much more information than it uses, but checks too rarely
> (with, for example, nnchoke-request-update-info) to keep the
> information up-to-date.

Yes, it's noticeably skewed toward NNTP.


paul

Re: can back end keep track of *all* group info?

[Paul Grayson]

Simon Josefsson <jas@extundo.com> writes:
> ...
> M-g on the group makes Gnus update the estimate with real information
> though.
> Making the Group buffer use the real values from the backend, when
> they are able to provide that data, should fix this problem and be
> very useful.  Shouldn't be too difficult to do, I think.

Okay, I think that I'm starting to see what I need to do to trick gnus
into working correctly with my server.  M-g now updates my groups
quite well.  However, 'g' still displays only the groups with ticked
articles, and never thinks any groups have unread articles.

Is there an easy way to replace whatever 'g' uses currently with the
M-g update?  I'd like it to only display the groups that have new
messages, and I want 'n' at the end of one group to jump correctly to
the next unread group, so changing %y (as Kai proposed) isn't going to
be good enough, right?

Is the only difference between M-g and whatever 'g' does to each group
that -request-group is called with the 'dont-check' option in the
latter?

> If you implement `gnus-request-set-mark' your backend should at least
> receive all flags as set by Gnus.  The other way is handled via
> `nnchoke-request-update-info', but I'd agree that this is called too
> many times and still not enough times. (This is way nnimap doesn't
> have a `nnimap-request-update-info' but rather calls this function
> internally upon group entry, in `nnimap-request-group' -- which
> doesn't seem to cause any problems.)

Yes, I'll do that.  I think that my main problem is displaying the
correct groups and unread counts.

Thanks, and have a merry Christmas!
-Paul

Re: can back end keep track of *all* group info?

[Simon Josefsson]

Paul Grayson <pgrayson+ding@ks.uiuc.edu> writes:

> Okay, I think that I'm starting to see what I need to do to trick gnus
> into working correctly with my server.  M-g now updates my groups
> quite well.  However, 'g' still displays only the groups with ticked
> articles, and never thinks any groups have unread articles.

If so, does your `nnchoke-retrieve-groups' really return correct
information?  Guns certainly should notice new unread articles.

> Is there an easy way to replace whatever 'g' uses currently with the
> M-g update?  I'd like it to only display the groups that have new
> messages, and I want 'n' at the end of one group to jump correctly to
> the next unread group, so changing %y (as Kai proposed) isn't going to
> be good enough, right?
>
> Is the only difference between M-g and whatever 'g' does to each group
> that -request-group is called with the 'dont-check' option in the
> latter?

Replacing `g' with `M-g' stuff becomes slow for most backends, so this
isn't a good solution.  Ideally the `g' stuff should use additional
information from backends which are able to provide it, for display in
the *Group* buffer.

`n' should work since Gnus over-estimates the number of unread
articles, it never under-estimates them.  Maybe this is related to the
problem above.

Re: can back end keep track of *all* group info?

[Paul Grayson]

> If so, does your `nnchoke-retrieve-groups' really return correct
> information?  Guns certainly should notice new unread articles.

Okay, it is starting to work quite well for me now.  Maybe
'nnchoke-retrieve-groups' shouldn't be listed as an optional function?
I'll send some more mail if I run into more problems.  Thanks for the
help!

-Paul

Re: can back end keep track of *all* group info?

[Simon Josefsson]

Paul Grayson <pgrayson+ding@ks.uiuc.edu> writes:

>> If so, does your `nnchoke-retrieve-groups' really return correct
>> information?  Guns certainly should notice new unread articles.
>
> Okay, it is starting to work quite well for me now.  Maybe
> 'nnchoke-retrieve-groups' shouldn't be listed as an optional function?

Well, it is optional. :-) Some backends can't provide the information
that is needed to implement it.