Exact article count for nnml groups

Exact article count for nnml groups

[David Hanak]

[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]

Hi,

Here's a patch I submitted last June which allows the group buffer to show
exact article counts for nnml groups instead of the estimate based on the
difference of the greatest and smallest article numbers.  Kai then helped
me to improve it a bit, and then suggested to wait until the feature freeze
is over.  I guess it is over now, and it would be nice if it could be
integrated into No Gnus.  (I have all the paperwork done.)

It is fairly simple, and perhaps a little bit out of date (after all, the
lisp files I patched were probably modified since then), but since they're
only additions, it shouldn't be too hard to merge them.  Also, if anyone
writes functions to retrieve the list of existing article numbers for any
other backend than nnml, i.e., nnimap, then it should be able to display
exact article counts for those backends, too.

Cheers,

-- 
David Hanak - Research Engineer
Institute for Software Integrated Systems  |  http://www.isis.vanderbilt.edu
Vanderbilt University                      |      Work phone: (615) 343 1319
Box 1829, Station B, Nashville, TN 37235   |            PGP key ID: 266BC45F

P.S.: I tried to post this message two times before, first to the
newsgroup, second time to the ding@gnus.org list address.  However, my
posting didn't appear on the newsgroup, while other postings kept coming.
Is it because I used a spamified sender address?  I try to post with my
real address this time, hope it works.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnus.patch --]
[-- Type: text/x-patch, Size: 3844 bytes --]

*** /usr/share/emacs/site-lisp/gnus/lisp/gnus-sum.el	2003-06-07 19:40:22.000000000 +0200
--- ./gnus-sum.el	2003-06-10 15:07:41.000000000 +0200
***************
*** 1092,1097 ****
--- 1092,1106 ----
    :type 'boolean
    :group 'gnus-article-mime)
  
+ (defcustom gnus-count-articles-in-groups nil
+ "*Regexp to match groups whose article count should be determined
+ accurately upon entering the group.  Non-matching groups are conventionally
+ looked up in the active info.
+ 
+ If t, matches all groups, if nil, matches none."
+   :type '(choice regexp (const t) (const nil))
+   :group 'gnus-summary)
+ 
  ;;; Internal variables
  
  (defvar gnus-summary-display-cache nil)
***************
*** 5226,5231 ****
--- 5235,5273 ----
        (memq article gnus-newsgroup-recent))
       (t t))))
  
+ (defun gnus-get-group-articles-list (group)
+   "Return a list of group articles if `gnus-count-articles-in-groups'
+ matches GROUP as a regexp.  Otherwise return nil."
+   (let ((group-real-name (gnus-group-real-name group)))
+     (when (and gnus-count-articles-in-groups
+ 	       (or (eq gnus-count-articles-in-groups t)
+ 		   (string-match gnus-count-articles-in-groups group)))
+       (let ((gnus-command-method (gnus-find-method-for-group group))
+ 	    (func1 'get-group-articles-list)
+ 	    (func2 'request-group-articles))
+ 	(or (and (gnus-check-backend-function func1 group)
+ 		 (funcall (gnus-get-function gnus-command-method func1)
+ 			  group-real-name (nth 1 gnus-command-method)))
+ 	    (and (gnus-check-backend-function func2 group)
+ 		 (funcall (gnus-get-function gnus-command-method func2)
+ 			  group-real-name (nth 1 gnus-command-method))
+ 		 (gnus-parse-group-articles-list)))))))
+ 
+ (defun gnus-parse-group-articles-list ()
+   "Parse article list from `nntp-server-buffer'."
+   (condition-case ()
+       (save-excursion
+ 	(set-buffer nntp-server-buffer)
+ 	(goto-char (point-min))		; first line contains comment
+ 	(delete-region (point) (line-end-position))
+ 	(insert "(")
+ 	(goto-char (point-max))
+ 	(forward-line -1)		; last line contains "."
+ 	(delete-region (point) (line-end-position))
+ 	(insert ")")
+ 	(read (point-min-marker)))
+     (error nil)))
+ 
  (defun gnus-articles-to-read (group &optional read-all)
    "Find out what articles the user wants to read."
    (let* ((display (gnus-group-find-parameter group 'display))
***************
*** 5242,5247 ****
--- 5284,5290 ----
  	      ;; articles in the group, or (if that's nil), the
  	      ;; articles in the cache.
  	      (or
+ 	       (gnus-get-group-articles-list group)
  	       (gnus-uncompress-range (gnus-active group))
  	       (gnus-cache-articles-in-group group))
  	    ;; Select only the "normal" subset of articles.
*** /usr/share/emacs/site-lisp/gnus/lisp/nnml.el	2003-05-01 16:23:15.000000000 +0200
--- ./nnml.el	2003-06-10 15:11:41.000000000 +0200
***************
*** 1007,1012 ****
--- 1007,1033 ----
  	(nnml-save-marks group server)
  	(nnheader-message 7 "Bootstrapping marks for %s...done" group)))))
  
+ (deffoo nnml-get-group-articles-list (group &optional server)
+   "Return the list of existing articles in GROUP directly.  See also
+ `nnml-request-group-articles'."
+   (sort (nnheader-directory-articles
+ 	 (nnheader-group-pathname group nnml-directory))
+ 	'<))
+ 
+ (deffoo nnml-request-group-articles (group &optional server)
+   "Return the list of existing articles in GROUP in `nntp-server-buffer'."
+   (let ((article-list (nnml-get-group-articles-list group server)))
+     (condition-case ()
+ 	(save-excursion
+ 	  (set-buffer nntp-server-buffer)
+ 	  (erase-buffer)
+ 	  (insert "211 Article list follows\n")
+ 	  (mapcar '(lambda (x) (insert (number-to-string x) "\n"))
+ 		  article-list)
+ 	  (insert ".\n")
+ 	  t)
+       (error nil))))
+ 
  (provide 'nnml)
  
  ;;; nnml.el ends here

Re: Exact article count for nnml groups

[Jesper Harder]

David Hanak <dhanak@isis.vanderbilt.edu> writes:

A few comments on the code:

> + (defcustom gnus-count-articles-in-groups nil
> + "*Regexp to match groups whose article count should be determined
> + accurately upon entering the group.  Non-matching groups are conventionally
> + looked up in the active info.

If possible try to follow the style guide for docstrings in
<info://elisp/Documentation+Tips>.

> +   (condition-case ()

It's harder to debug code inside `condition-case'.  It's better if you
can restrict to just the form(s) that you know might signal an error
(and you know that the error can safely be ignored, of course).

> + 	(delete-region (point) (line-end-position))

You need to use `point-at-bol' to be compatible with XEmacs.

> + 	(save-excursion
> + 	  (set-buffer nntp-server-buffer)

A slightly more concise way of doing this is

       (with-current-buffer nntp-server-buffer
         ...)

Re: Exact article count for nnml groups

[Michael Schierl]

Jesper Harder <harder@ifa.au.dk> writes:

> David Hanak <dhanak@isis.vanderbilt.edu> writes:
>
>> + 	(delete-region (point) (line-end-position))
>
> You need to use `point-at-bol' to be compatible with XEmacs.

I guess you mean `point-at-eol' ;)

Michael

Re: Exact article count for nnml groups

[David Hanak]

Jesper, Kevin, Kai,

thanks for your verbose replies and comments.  I wish I had time to work on
the "project"...  This whole thing started out as a very simple
modification to display exact article count for my nnml groups, and then I
thought why couldn't others profit from it, too.  I never planned to spend
this much time with it, though.

In short, I will try to incorporate your suggestions when I get to it.  For
more thoughts, read on.

On Fri, 12 Mar 2004, Jesper Harder wrote:

> If possible try to follow the style guide for docstrings in
> <info://elisp/Documentation+Tips>.

I will.

>> +   (condition-case ()
>
> It's harder to debug code inside `condition-case'.  It's better if you
> can restrict to just the form(s) that you know might signal an error
> (and you know that the error can safely be ignored, of course).

I think I copied this part from somewhere else in the Gnus code, but I
don't really remember the details.  Anyway, I will try to follow your
advice. 

>> + 	(delete-region (point) (line-end-position))
>
> You need to use `point-at-bol' to be compatible with XEmacs.

Grin.  I've never used XEmacs, let alone program it.


On Fri, 12 Mar 2004, Kevin Greiner wrote:

> * I'm not really happy with the use of a regexp to select groups.
>   Your gnus-get-group-articles-list is making calls to the backend
>   (i.e. server-specific code).  Shouldn't the on/off switch be in the
>   server's method?

If I only knew anything about servers and server variables in practice. :-)
But I will try to think of a better way...

> * The previous points had me looking in gnus-int where I found
>   gnus-request-group-articles.  Your new backend
>   'get-group-articles-list is simply the existing
>   'request-group-articles returning the articles as a list rather than
>   in a buffer.  From looking at your implementation, I can see why you
>   want to do this as it certainly optimizes the API for nnml.

:-))  We had the very same discussion with Kai last June, when the patch
was developed.  It was he who pointed me out the request-group-articles
server methods, and IIRC, it was he who suggested that if I'm unsatisfied
with the performance, than it should be used as a fallback function to a
quicker, more direct one, thus get-group-articles-list was created.

>   start a separate thread to discuss updating 'request-group-articles to
>   return a list rather than a buffer.

I don't think that would be good.  It would break the rule that request
functions always manipulate the server buffer, and never return their
results directly.

> * From what I can see, there's only one call to your new code and
>   that's in gnus-articles-to-read.  Are you sure that this one
>   function in gnus-sum is all that is necessary to change the article
>   counts in the GROUP buffer? Also, which article count are you trying
>   to correct? By my reading, the group buffer can display eight
>   different counts for each group.

Eight?  Wow, I'm impressed.  Ok, it is the count shown by the %y formatter,
AND the count respected by the test which checks group size before entering
a group and optionally asks for the number of articles to be requested.
I'm pretty sure it works, since I've been using the same patch for two
years now, and I never had any problems.

On Sat, 13 Mar 2004, Kai Grossjohann wrote:

> It's good stuff.

If it is, it's your influence! :-)

> I'd suggest to do away with the gnus-count-articles-in-groups variable
> -- just always try to invoke the new function if available.

What if someone wants to turn this functionality off, say, because he/she
has a slow machine, or when someone implements the required funciton for
nnimap, too, which can be really slow for bad IMAP connections?  I think we
need some way to control it other than the mere existence of the required
functions.

Thanks,

-- 
David

Re: Exact article count for nnml groups

[Kai Grossjohann]

David Hanak <dhanak@isis.vanderbilt.edu> writes:

> On Sat, 13 Mar 2004, Kai Grossjohann wrote:
>
>> I'd suggest to do away with the gnus-count-articles-in-groups variable
>> -- just always try to invoke the new function if available.
>
> What if someone wants to turn this functionality off, say, because
> he/she has a slow machine, or when someone implements the required
> funciton for nnimap, too, which can be really slow for bad IMAP
> connections?  I think we need some way to control it other than the
> mere existence of the required functions.

I suggested to do like nnml-nov-is-evil.  WDYT?

It would allow said person to turn it off for nnimap, but on for
another backend.

Kai

Re: Exact article count for nnml groups

[Kevin Greiner]

David Hanak <dhanak@isis.vanderbilt.edu> writes:

> Hi,
>
> Here's a patch I submitted last June which allows the group buffer to show
> exact article counts for nnml groups instead of the estimate based on the
> difference of the greatest and smallest article numbers.  Kai then helped
> me to improve it a bit, and then suggested to wait until the feature freeze
> is over.  I guess it is over now, and it would be nice if it could be
> integrated into No Gnus.  (I have all the paperwork done.)
>
> It is fairly simple, and perhaps a little bit out of date (after all, the
> lisp files I patched were probably modified since then), but since they're
> only additions, it shouldn't be too hard to merge them.  Also, if anyone
> writes functions to retrieve the list of existing article numbers for any
> other backend than nnml, i.e., nnimap, then it should be able to display
> exact article counts for those backends, too.

David,
I suspect that, since your proposing changing the back-end API,
everyone is waiting to see how Lars responds to your proposal.

There are a few points that I'd like to see addressed.

* I'm not really happy with the use of a regexp to select groups.
  Your gnus-get-group-articles-list is making calls to the backend
  (i.e. server-specific code).  Shouldn't the on/off switch be in the
  server's method?

* The gnus-get-group-articles-list, gnsu-parse-group-articles-list
  aren't specific to the summary.  They should be put in gnus-int.

* I don't find the name gnus-get-group-articles-list nor its
  description that informative.  From its use, and your text, I would
  say that you are returning the list of active articles.  That got me
  thinking that the name should be something like
  gnus-get-active-articles.

* The previous points had me looking in gnus-int where I found
  gnus-request-group-articles.  Your new backend
  'get-group-articles-list is simply the existing
  'request-group-articles returning the articles as a list rather than
  in a buffer.  From looking at your implementation, I can see why you
  want to do this as it certainly optimizes the API for nnml.
  However, I don't think that that is the proper way to approach the
  problem.  You should either use the existing 'request-group-articles
  function, even if it is inefficient for nnml, or start a separate
  thread to discuss updating 'request-group-articles to return a list
  rather than a buffer.

* From what I can see, there's only one call to your new code and
  that's in gnus-articles-to-read.  Are you sure that this one
  function in gnus-sum is all that is necessary to change the article
  counts in the GROUP buffer? Also, which article count are you trying
  to correct? By my reading, the group buffer can display eight
  different counts for each group.

Kevin

Re: Exact article count for nnml groups

[Lars Magne Ingebrigtsen]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> I suspect that, since your proposing changing the back-end API,
> everyone is waiting to see how Lars responds to your proposal.

I think it's a useful feature, and I agree with your comments about
it.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Exact article count for nnml groups

[Kai Grossjohann]

It's good stuff.  In addition to the other comments, I have one other
issue:

If I understand correctly, then your code never tries to invoke the
new backend function unless the gnus-count-articles-in-groups variable
allows it.  If the variable allows it, then you look whether the new
function is available.  If so, you call it.  Otherwise, you fall back
to the old backend function.

I'd suggest to do away with the gnus-count-articles-in-groups variable
-- just always try to invoke the new function if available.

If you really want to allow people to turn it off, you could do like
nnml-nov-is-evil and similar variables, which also allow to turn off
backend features.  (Following Kevin's unhappiness with matching on
group names.)

Kai