Garbled .overviews with 5.6.11?

Garbled .overviews with 5.6.11?

[Mike McEwan]

  I'm not sure of the above, but since upgrading to 5.6.11, and
applying Hrvoje Niksic's `procmail' patch and Kim-Minh Kaplan's
`fetch-old-headers' patch, I'm getting a number of `Strange nov line'
messages and `Invalid read syntax' when entering groups and
downloading news via the `agent'.

  It may be something to do with expiry processing, I can't fathom it
out yet. My .overview files appear to have a number of partial header
lines and the sequencing of article numbers, top to bottom, appears to 
have gone a little to pot. I may have compounded matters by attempting 
to sort out these files manually, but I'd be curious to know if anyone 
else has been seeing similar things?

  For the time being I've gone back to 5.6.10 - I'll see how it goes.

-- 
Mike.

Re: Garbled .overviews with 5.6.11?

[Lars Magne Ingebrigtsen]

Mike McEwan <mike@lotusland.demon.co.uk> writes:

>   I'm not sure of the above, but since upgrading to 5.6.11, and
> applying Hrvoje Niksic's `procmail' patch and Kim-Minh Kaplan's
> `fetch-old-headers' patch, I'm getting a number of `Strange nov line'
> messages and `Invalid read syntax' when entering groups and
> downloading news via the `agent'.
> 
>   It may be something to do with expiry processing, I can't fathom it
> out yet. My .overview files appear to have a number of partial header
> lines and the sequencing of article numbers, top to bottom, appears to 
> have gone a little to pot.

I just expired a bunch of articles, and I got one of those "Strange
nov lines" messages when fetching more articles later.  So there does
indeed seem to be something wonky in the code that's deleting lines
from the .overview files.

Let's see...  This is from alt.folklore.computers:

192316	Re: Y2K	"Michael A. Covington" <mc@ai.uga.edu>	23 Jun 1998 18:12:04 -0700	<6mp5bp$mch$1@cronkite.cc.uga.edu>	<6mmb2v$cob$1@nntp1.ba.best.com> <6mmb4i$cob$2@nntp1.ba.best.com> <6mmb7a$cob$3@nntp1.ba.best.com> <6mmbau$cob$4@nntp1.ba.best.com> <6mmbeo$cob$6@nntp1.ba.best.com>	1081	11	Xref: nntp.uio.no alt.folklore.suburban:4354 alt.folklore.computers:192316 
 Wilbur <cwilbur@vengeance.gwi.net>	11 Jun 1998 02:23:58 -0400	<m2ra0wjw1t.fsf@vengeance.gwi.net>	<6lm9uj$8ep$6@mawar.singnet.com.sg>	4386	128	Xref: nntp.uio.no alt.computer.consultants:84370 alt.folklore.computers:191417

It looks like something cut off the number at the start of the line.
Hm.  I've looked the code over, and the only function used to delete
lines is `gnus-delete-line', which means that this should be
impossible.  (Unless there's narrowing in effect, and there is none.)

I'm stumped.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Garbled .overviews with 5.6.11?

[Mike McEwan]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I just expired a bunch of articles, and I got one of those "Strange
> nov lines" messages when fetching more articles later.  So there does
> indeed seem to be something wonky in the code that's deleting lines
> from the .overview files.
> 
> Let's see...  This is from alt.folklore.computers:
> 
> 192316	Re: Y2K	"Michael A. Covington" <mc@ai.uga.edu>	23 Jun 1998 18:12:04 -0700	<6mp5bp$mch$1@cronkite.cc.uga.edu>	<6mmb2v$cob$1@nntp1.ba.best.com> <6mmb4i$cob$2@nntp1.ba.best.com> <6mmb7a$cob$3@nntp1.ba.best.com> <6mmbau$cob$4@nntp1.ba.best.com> <6mmbeo$cob$6@nntp1.ba.best.com>	1081	11	Xref: nntp.uio.no alt.folklore.suburban:4354 alt.folklore.computers:192316 
>  Wilbur <cwilbur@vengeance.gwi.net>	11 Jun 1998 02:23:58 -0400	<m2ra0wjw1t.fsf@vengeance.gwi.net>	<6lm9uj$8ep$6@mawar.singnet.com.sg>	4386	128	Xref: nntp.uio.no alt.computer.consultants:84370 alt.folklore.computers:191417
> 
> It looks like something cut off the number at the start of the line.
> Hm.  I've looked the code over, and the only function used to delete
> lines is `gnus-delete-line', which means that this should be
> impossible.  (Unless there's narrowing in effect, and there is none.)
> 
> I'm stumped.

  Me too, although I would still regard my elisp as extremely
rudimentary. Your above example is exactly what I'm seeing.

  How is expiry processing supposed to work now in respect of nov
lines that do not have downloaded articles? Is it that all nov lines
with an article number less than the lowest numbered expire-able
history line (for a given group) are/should be deleted? I guess I need
to watch a little more closely, but I'm sure that *not* all my old nov
lines are being deleted, and, as previously mentioned, I have
sometimes seen a glut of low numbered nov lines appear some-way down my
.overview files, seemingly out of sequence???

  I'm currently cleaning up my .agentview and .overview files by hand
when required. I do a `sort-numeric-fields' on my .overview, this
invariably brings the errant nov line to the top (the line number in
the `strange nov line message' can be way out), and delete all those
up to the article number that corresponds with the first article
number in my .agentview.

  I hope someone can figure out what's happening here - expiry is a
good thing if you want to get any reasonable performance back into
summary preparation.

-- 
Mike. 

Re: Garbled .overviews with 5.6.11?

[Lars Magne Ingebrigtsen]

Mike McEwan <mike@lotusland.demon.co.uk> writes:

>   How is expiry processing supposed to work now in respect of nov
> lines that do not have downloaded articles? Is it that all nov lines
> with an article number less than the lowest numbered expire-able
> history line (for a given group) are/should be deleted?

Yes -- all NOV lines that are older than the newest expirable article
and have no downloaded article are removed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Garbled .overviews with 5.6.11?

[Mike McEwan]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yes -- all NOV lines that are older than the newest expirable
> article and have no downloaded article are removed.

Well my `old' NOV lines still didn't appear to be getting deleted. I
think I may have discovered the problem. At least I reckon I'm
getting something like the desired effect now, but, with my elisp
still being a little on the simple side, I'm not entirely sure I'm
tackling things in the right way.

Here's the text of an earlier posting I made to the list:

"I have a secondary server `(nntp "sunsite.auc.dk")' covered under
the agent. However, when conducting a `gnus-agent-expire' I get:

File error: "Opening output file", "No such file or directory",
"/home/mike/News/agent/nntp/sunsite.auc.dk/gnu/emacs/gnus/.overview"

It would appear that, having finished expiring articles for my
primary select method (via which I subscribe to gnu.emacs.gnus), a
variable's value is perhaps not being reset to an appropriate value.

I've tried to track it down, but I can only verify that `sym'
evaluates to `gnu\.emacs\.gnus' at the time of the error in
`gnus-agent-expire'. I only subscribe to this group on my primary
server, which, at the time of error, has already had its articles
successfully expired."

I think what I'm accomplishing in the patch below is to create a
fresh `expiry-hashtb' when (and if) `(setq gnus-command-method (pop
methods))' moves on to another select method as is the case with
me. Gnus certainly doesn't barf as before when moving on to the
secondary server I have covered by the Agent.

A second problem, for me at least, was that the NOV lines in my
.overview files were not always in sequence (not really sure why yet,
apart from whatever headers are downloaded appear to simply be
appended to the end of a .overview file, regardless of their article
numbers). This means that `gnus-agent-expire' could not always delete
"all NOV lines that are older than the newest expirable article" as it
was not encountering old articles before new. Anyways, to remedy this
I've stuck in a `sort-numeric-fields' before the NOV lines are
expired. This is what I'm least sure about. could this be accomplished
in a more efficient way? Do I indeed, want to do it?

Despite the above, old NOV lines without articles were still *not*
being deleted, and so, after a spell with edebug, I reckoned the the
line:
 
(if (file-exists-p                              
     (gnus-agent-article-name                   		 
      (number-to-string article) group))        
    (forward-line 1)                            
  ;; Remove old NOV lines that have no articles.
  (gnus-delete-line)))                          

was never resulting in a non-existent article file. Shouldn't it
perhaps be `(number-to-string art) group))'. Seems to work for me.

Anyways, see what you think of the patch below. Still get one or two
of those `partial' NOV lines though :-(.

P.S. What's with this new paragraph fill mode - `M-q' doesn't seem to
let me indent the first line of a paragraph anymore?
    
-- 
Mike.

--- /usr/lib/xemacs/site-lisp/gnus/lisp/gnus-agent.el.orig	Mon Jun  1 03:30:14 1998
+++ /usr/lib/xemacs/site-lisp/gnus/lisp/gnus-agent.el	Mon Jun 29 23:31:49 1998
@@ -1254,13 +1254,13 @@
   (interactive)
   (let ((methods gnus-agent-covered-methods)
 	(day (- (gnus-time-to-day (current-time)) gnus-agent-expire-days))
-	(expiry-hashtb (gnus-make-hashtable 1023))
 	gnus-command-method sym group articles
 	history overview file histories elem art nov-file low info
 	unreads marked article)
     (save-excursion
       (setq overview (get-buffer-create " *expire overview*"))
       (while (setq gnus-command-method (pop methods))
+	(let ((expiry-hashtb (gnus-make-hashtable 1023)))
 	(gnus-agent-open-history)
 	(set-buffer
 	 (setq gnus-agent-current-history
@@ -1303,6 +1303,8 @@
 	     (erase-buffer)
 	     (when (file-exists-p nov-file)
 	       (insert-file-contents nov-file))
+	     ;;; Make sure article nov lines are in sequence
+	     (sort-numeric-fields 1 1 (buffer-size))
 	     (goto-char (point-min))
 	     (setq article 0)
 	     (while (setq elem (pop articles))
@@ -1319,7 +1321,7 @@
 				 (< art article)))
 		   (if (file-exists-p
 			(gnus-agent-article-name
-			 (number-to-string article) group))
+			 (number-to-string art) group))
 		       (forward-line 1)
 		     ;; Remove old NOV lines that have no articles.
 		     (gnus-delete-line)))
@@ -1362,7 +1364,7 @@
 	    (gnus-delete-line))
 	  (gnus-agent-save-history)
 	  (gnus-agent-close-history))
-	(gnus-message 4 "Expiry...done")))))
+	(gnus-message 4 "Expiry...done"))))))
 
 ;;;###autoload
 (defun gnus-agent-batch ()