[patch] `gnus-agent-fetch-heades' (again)

[patch] `gnus-agent-fetch-heades' (again)

[Mike McEwan]

  From my previous mailing on the subject:

  "Lars, I don't understand the method that is used in
`gnus-agent-fetch-headers' to work out what we want to 
download. It asks for everything that is unread, but not in
`gnus-agent-group-alist'. Surely we're only interested in those
headers *after* the last one in this alist?"

  Well I'm coming to understand a little more now :-). Following my
last patch (Lars, at least you added a check to see that we actually
have a `gnus-agent-article-alist') I attempted to subscribe to a new
newsgroup, but not having a .agentview initially `~fetch-headers' does 
not select any articles :-(.

  Having thought a little more about the possible situations, just
fetching all headers after the last one in .agentview does not cut
it. Apart from the situation above, *all* headers in .agentview could
have been expired. It could be that all headers bar a few recently
downloaded via `gnus-downloadable-mark' had been expired. In this case 
headers after the last one in .agentview may have already been
downloaded. 

  The following *two* patches tackle the problem in slightly different 
ways. 

  The first, my preferred option, sets the last header fetched to in a
group's parameters. Headers can be safely downloaded from this point
on, the next time we connect to the news-server. The only slight
problem with this is that, being in a group's parameters, it is
visible to the user. Do you trust the user user not to meddle with
this strange new parameter :-P. I do. Indeed this where I'm currently
storing this information for the `new articles' functions I'm tweaking
with. This is where, if enabled, the timestamp of a group's last entry
is kept, and this provides a similar function.  I don't suppose it'd
be easy to tuck this away in the main body of a group's info though?

The second option is nearly a return to what I couldn't understand
above. Headers are selected for download if they are after the last
header in .agentview *and* the aren't in the list of read
articles. This could still be a bit dodgy I reckon. You have to make
sure that *all* expired articles are marked read - see below. I'm
still not happy with this. When calculating unread articles, gnus
could still re-download some headers because of an unlucky spread of
cancelled articles (not being read, they are unread no?). As mentioned
in my previous mailing, the mechanism in:

`nntp-retrieve-headers-with-xover'

 assumes we're always downloading a contiguous series of headers.

  Sorry to be so big below. That's what happens when you change the
indentation. 

-- 
Mike.

--- ChangeLog~	Thu Aug 27 10:03:55 1998
+++ ChangeLog	Sat Aug 29 13:00:51 1998
@@ -1,3 +1,10 @@
+1998-08-29  Mike McEwan  <mike@lotusland.demon.co.uk>
+
+	* gnus-agent.el (gnus-agent-fetch-headers): Cater for when there's 
+	no .agentview, all articles have been expired, or everything bar a 
+	few downloaded arts have been expired.
+	(gnus-agent-expire): Mark *all* expired articles as read.
+
 Thu Aug 27 11:03:59 1998  Lars Magne Ingebrigtsen  <larsi@menja.ifi.uio.no>
 
 	* gnus.el: Gnus v5.6.41 is released.
;;;
;;; Patch 1
;;;

--- gnus-agent.el.orig	Thu Aug 27 10:03:56 1998
+++ gnus-agent.el	Sat Aug 29 12:51:39 1998
@@ -751,32 +751,41 @@
       (pop gnus-agent-group-alist))))
 
 (defun gnus-agent-fetch-headers (group &optional force)
-  (when (gnus-agent-load-alist group)
-    (let ((articles (gnus-uncompress-range 
-		     (cons (1+ (caar (last (gnus-agent-load-alist group))))
-			   (cdr (gnus-active group))))))
-      ;; Fetch them.
-      (when articles
-	(gnus-message 7 "Fetching headers for %s..." group)
-	(save-excursion
-	  (set-buffer nntp-server-buffer)
-	  (unless (eq 'nov (gnus-retrieve-headers articles group))
-	    (nnvirtual-convert-headers))
-	  ;; Save these headers for later processing.
-	  (copy-to-buffer gnus-agent-overview-buffer (point-min) (point-max))
-	  (let (file)
-	    (when (file-exists-p
-		   (setq file (gnus-agent-article-name ".overview" group)))
-	      (gnus-agent-braid-nov group articles file))
-	    (gnus-make-directory (nnheader-translate-file-chars
-				  (file-name-directory file)))
-	    (write-region (point-min) (point-max) file nil 'silent)
-	    (gnus-agent-save-alist group articles nil)
-	    (gnus-agent-enter-history
-	     "last-header-fetched-for-session"
-	     (list (cons group (nth (- (length  articles) 1) articles)))
-	     (gnus-time-to-day (current-time)))
-	    articles))))))
+  (let* ((last (gnus-group-get-parameter group 'last))
+	 (max (cdr (gnus-active group)))
+	 (articles (if last
+		       (if (> max last)
+			   (gnus-uncompress-range
+			    (cons (1+ last) (cdr (gnus-active group))))
+			 nil)
+		     (gnus-list-of-unread-articles group))))
+    ;; Fetch them.
+    (when articles
+      (gnus-message 7 "Fetching headers for %s..." group)
+      (save-excursion
+	(set-buffer nntp-server-buffer)
+	(unless (eq 'nov (gnus-retrieve-headers articles group))
+	  (nnvirtual-convert-headers))
+	;; Save these headers for later processing.
+	(copy-to-buffer gnus-agent-overview-buffer (point-min) (point-max))
+	(let (file)
+	  (when (file-exists-p
+		 (setq file (gnus-agent-article-name ".overview" group)))
+	    (gnus-agent-braid-nov group articles file))
+	  (gnus-make-directory (nnheader-translate-file-chars
+				(file-name-directory file)))
+	  (write-region (point-min) (point-max) file nil 'silent)
+	  (gnus-agent-load-alist group)
+	  (gnus-agent-save-alist group articles nil)
+	  ;; Mark where we've fetched to.
+	  (gnus-group-set-parameter group 'last (cdr (gnus-active group))) 
+	  (gnus-agent-enter-history
+	   ;; The last header/article may have been cancelled, but
+	   ;; put an entry in the history for expiry processing.
+	   "headers-fetched-to-here-this-session"
+	   (list (cons group (cdr (gnus-active group))))
+	   (gnus-time-to-day (current-time)))
+	  articles)))))
 
 (defsubst gnus-agent-copy-nov-line (article)
   (let (b e)
@@ -1346,7 +1355,8 @@
 		   (sort gnus-agent-article-alist 'car-less-than-car))
 	     (let* ((alist gnus-agent-article-alist)
 		    (prev (cons nil alist))
-		    (first prev))
+		    (first prev)
+		    expired)
 	       (while (and alist
 			   (<= (caar alist) article))
 		 (if (or (not (cdar alist))
@@ -1355,22 +1365,33 @@
 				(number-to-string
 				 (caar alist))
 				group))))
-		     (setcdr prev (setq alist (cdr alist)))
+		     (progn
+		       (push (caar alist) expired)
+		       (setcdr prev (setq alist (cdr alist))))
 		   (setq prev alist
 			 alist (cdr alist))))
 	       (setq gnus-agent-article-alist (cdr first))
-	       ;;; Mark all articles up to the first article
-	       ;;; in `gnus-article-alist' as read.
+	       (gnus-agent-save-alist group)
+               ;; Mark all articles up to the first article
+	       ;; in `gnus-article-alist' as read.
 	       (when (caar gnus-agent-article-alist)
 		 (setcar (nthcdr 2 info)
 			 (gnus-range-add
 			  (nth 2 info)
 			  (cons 1 (- (caar gnus-agent-article-alist) 1)))))
+	       ;; Maybe everything has been expired from `gnus-article-alist'
+	       ;; and so the above marking as read could not be conducted,
+	       ;; or there are expired article within the range of the alist.
+	       (when (or (not (caar gnus-agent-article-alist))
+			 (> (car expired) (caar gnus-agent-article-alist)))  
+	       (setcar (nthcdr 2 info)
+		       (gnus-add-to-range
+			(nth 2 info)
+			(nreverse expired))))
 	       (gnus-dribble-enter
 		(concat "(gnus-group-set-info '"
 			(gnus-prin1-to-string info)
-			")"))
-	       (gnus-agent-save-alist group)))
+			")"))))
 	   expiry-hashtb)
 	  (set-buffer history)
 	  (setq histories (nreverse (sort histories '<)))

;;;
;;; Patch 2
;;;

--- gnus-agent.el.orig	Thu Aug 27 10:03:56 1998
+++ gnus-agent.el	Sat Aug 29 12:37:40 1998
@@ -751,32 +751,35 @@
       (pop gnus-agent-group-alist))))
 
 (defun gnus-agent-fetch-headers (group &optional force)
-  (when (gnus-agent-load-alist group)
-    (let ((articles (gnus-uncompress-range 
-		     (cons (1+ (caar (last (gnus-agent-load-alist group))))
-			   (cdr (gnus-active group))))))
-      ;; Fetch them.
-      (when articles
-	(gnus-message 7 "Fetching headers for %s..." group)
-	(save-excursion
-	  (set-buffer nntp-server-buffer)
-	  (unless (eq 'nov (gnus-retrieve-headers articles group))
-	    (nnvirtual-convert-headers))
-	  ;; Save these headers for later processing.
-	  (copy-to-buffer gnus-agent-overview-buffer (point-min) (point-max))
-	  (let (file)
-	    (when (file-exists-p
-		   (setq file (gnus-agent-article-name ".overview" group)))
-	      (gnus-agent-braid-nov group articles file))
-	    (gnus-make-directory (nnheader-translate-file-chars
-				  (file-name-directory file)))
-	    (write-region (point-min) (point-max) file nil 'silent)
-	    (gnus-agent-save-alist group articles nil)
-	    (gnus-agent-enter-history
-	     "last-header-fetched-for-session"
-	     (list (cons group (nth (- (length  articles) 1) articles)))
-	     (gnus-time-to-day (current-time)))
-	    articles))))))
+  (let ((articles (if (gnus-agent-load-alist group)   
+		      (gnus-sorted-intersection
+		       (gnus-list-of-unread-articles group)
+		       (gnus-uncompress-range
+			(cons (1+ (caar (last gnus-agent-article-alist)))
+			      (cdr (gnus-active group)))))
+		    (gnus-list-of-unread-articles group))))
+    ;; Fetch them.
+    (when articles
+      (gnus-message 7 "Fetching headers for %s..." group)
+      (save-excursion
+	(set-buffer nntp-server-buffer)
+	(unless (eq 'nov (gnus-retrieve-headers articles group))
+	  (nnvirtual-convert-headers))
+	;; Save these headers for later processing.
+	(copy-to-buffer gnus-agent-overview-buffer (point-min) (point-max))
+	(let (file)
+	  (when (file-exists-p
+		 (setq file (gnus-agent-article-name ".overview" group)))
+	    (gnus-agent-braid-nov group articles file))
+	  (gnus-make-directory (nnheader-translate-file-chars
+				(file-name-directory file)))
+	  (write-region (point-min) (point-max) file nil 'silent)
+	  (gnus-agent-save-alist group articles nil)
+	  (gnus-agent-enter-history
+	   "last-header-fetched-for-session"
+	   (list (cons group (nth (- (length  articles) 1) articles)))
+	   (gnus-time-to-day (current-time)))
+	  articles)))))
 
 (defsubst gnus-agent-copy-nov-line (article)
   (let (b e)
@@ -1346,7 +1349,8 @@
 		   (sort gnus-agent-article-alist 'car-less-than-car))
 	     (let* ((alist gnus-agent-article-alist)
 		    (prev (cons nil alist))
-		    (first prev))
+		    (first prev)
+		    expired)
 	       (while (and alist
 			   (<= (caar alist) article))
 		 (if (or (not (cdar alist))
@@ -1355,22 +1359,33 @@
 				(number-to-string
 				 (caar alist))
 				group))))
-		     (setcdr prev (setq alist (cdr alist)))
+		     (progn
+		       (push (caar alist) expired)
+		       (setcdr prev (setq alist (cdr alist))))
 		   (setq prev alist
 			 alist (cdr alist))))
 	       (setq gnus-agent-article-alist (cdr first))
-	       ;;; Mark all articles up to the first article
-	       ;;; in `gnus-article-alist' as read.
+	       (gnus-agent-save-alist group)
+               ;; Mark all articles up to the first article
+	       ;; in `gnus-article-alist' as read.
 	       (when (caar gnus-agent-article-alist)
 		 (setcar (nthcdr 2 info)
 			 (gnus-range-add
 			  (nth 2 info)
 			  (cons 1 (- (caar gnus-agent-article-alist) 1)))))
+	       ;; Maybe everything has been expired from `gnus-article-alist'
+	       ;; and so the above marking as read could not be conducted,
+	       ;; or there are expired article within the range of the alist.
+	       (when (or (not (caar gnus-agent-article-alist))
+			 (> (car expired) (caar gnus-agent-article-alist)))  
+	       (setcar (nthcdr 2 info)
+		       (gnus-add-to-range
+			(nth 2 info)
+			(nreverse expired))))
 	       (gnus-dribble-enter
 		(concat "(gnus-group-set-info '"
 			(gnus-prin1-to-string info)
-			")"))
-	       (gnus-agent-save-alist group)))
+			")"))))
 	   expiry-hashtb)
 	  (set-buffer history)
 	  (setq histories (nreverse (sort histories '<)))

Re: [patch] `gnus-agent-fetch-heades' (again)

[Bruce Stephens]

Mike McEwan <mike@lotusland.demon.co.uk> writes:

> Well I'm coming to understand a little more now :-). Following my
> last patch (Lars, at least you added a check to see that we actually
> have a `gnus-agent-article-alist') I attempted to subscribe to a new
> newsgroup, but not having a .agentview initially `~fetch-headers'
> does not select any articles :-(.

Whew!  I thought it might just be me having some problems.  Mostly,
agent works just fine, but just recently (5.6.40 or so) it seems to be
ignoring a newsgroup altogether.  

This newsgroup is moderated, so it's intermittent; in fact, when I
checked the directory, all of the articles had expired---but there
were new ones yesterday.  The agent just ignored them.  So, I thought
maybe just deleting the relevant directory, unsubscribing and
resubscribing, sacrificing the right number of chickens, and similar
things would work.  Clearly not, and stepping through the code leaves
me utterly confused as to how the agent is ever supposed to notice new
articles to download.

Also, killing groups hasn't worked for some time.  If you have a group
which has had articles in it (retrieved by the agent), and you kill
the group, then the next time you expire articles, gnus crashes.
Removing all knowledge of the group (by deleting lines from
agent.lib/history) seems to fix it.

Re: [patch] `gnus-agent-fetch-heades' (again)

[Bruce Stephens]

Bruce Stephens <bruce@cenderis.demon.co.uk> writes:

> Whew!  I thought it might just be me having some problems.  Mostly,
> agent works just fine, but just recently (5.6.40 or so) it seems to be
> ignoring a newsgroup altogether.  

And, of course, just now it correctly downloaded the group.  It may
just be paranoia, but I think Gnus reads my mail.