Something very wrong here

Something very wrong here

[Harry Putnam]

I have a group that recieves all new mail for scanning.  Slurped from
a procmail spool.  Imagine I delete this group after using it and
respooling all its messages to other groups.

C-u G del  <prinb>  <gone without a trace>

Or is it?

Close gnus then emacs, restart.  Create a group with `G m' from nnml
backend naming it prinb attempt to enter the group:

Now reporting from *Messages* after restarting

Creating mail directory /home/reader/Mail/prinb/ <= create the group

    Retrieving newsgroup: nnml:prinb...
    Loading gnus-ml (source)...done
    Fetching headers for nnml:prinb...
    nnml: Receiving headers...done
    Fetching headers for nnml:prinb...done
    Scoring...done
    Loading gnus-async (source)...done
    Bootstrapping marks for prinb...

What is all the frenetic activity about? It even shows a percent count
as the non-existent headers are downloaded.
Fictitious group count is generated something like 3000+
Non existent marks are generated for non existent messages.

Finally reporting 
        No unread news

Do we need all this overhead, in an empty newly created group?
Why is there a fictitious message count generated?  Where is the false
info coming from?  Can't be .overview, since it is rm'ed. 

When  a group is C-u G del'ed and gnus is closed, shouldn't that
groups info disappear from newsrc.eld, as well as any `active' file?

Re: Something very wrong here

[Kai Großjohann]

Harry Putnam <reader@newsguy.com> writes:

> When  a group is C-u G del'ed and gnus is closed, shouldn't that
> groups info disappear from newsrc.eld, as well as any `active' file?

That's right.  Can you please check that this really happens?

Places which might contain traces of the group even after deleting:

.newsrc.eld
active file
cache active file
cache history file
some other cache file

When you find the culprit, somebody will surely fix the bug.

kai
-- 
Symbol's function definition is void: signature

Re: Something very wrong here

[Simon Josefsson]

On Mon, 3 Sep 2001, Harry Putnam wrote:

> C-u G del  <prinb>  <gone without a trace>
>
> Or is it?

Is the directory still there?  Does it contain anything?

>     Loading gnus-ml (source)...done

(Maybe Gnus should be byte compiled.)

> What is all the frenetic activity about? It even shows a percent count
> as the non-existent headers are downloaded.
> Fictitious group count is generated something like 3000+
> Non existent marks are generated for non existent messages.

Is the group cached?  Is the group agentized?  Maybe the cache or agent
directory wasn't emptied, and it is still around creating bogus articles.

Re: Something very wrong here

[Harry Putnam]

First off, I guess setting nnml-marks-is-evil is no way to find out
what is causing my problems.  So dropping that for now.

> Here lately I find already respooled messages still in there only not
> really, the files are gone but .overview or newsrc.eld still sees
> them.

This just happened again here.

Simon asks:

> How does your respooling setup look like?  Is the respooling commands
> invoked in the prescan summary buffer?  What does *Messages* contain
> after you have done all this?

My split looks like:

  (setq nnmail-split-methods 
    '(("tests" "^Subject:.*test.*labridge") 
      ("cron" "^From:.*\\(Cron Daemon\\)") 

[...] snipped a massive list of splits

    ("Edited"  "^Subject: \\[ed\\]\\|^Subject:  \\[ed\\]") 
      ("misc" ""))) 

[Not really part of split below but related]

  (setq gnus-message-archive-group
        '((let ((current-archive-date-string
	         (format-time-string "%m%y" (current-time))))
	    (concat (if (message-news-p)
		        "news-"
		      "mail-")
		    current-archive-date-string))))

The split is invoked via (respool) `B r' in the `prinb' group summary
buffer.

What I noticed just now is after respooling, prinb is presumably
empty, but when I open it I see the same messages as if they were
still there.  Attempting to open them gives the message that the the
article may have been cancelled or expunged.

Now checking the actual directory C-x d Mail/prinb  I see the .marks
file:  ((read (1 . 3031)))

And the .overview which contains the headers from some 19 messages,
but no messages.  So closing the group and reopen lets see what
*Messages* says (seems normal):

  Retrieving newsgroup: nnml:prinb...
  Fetching headers for nnml:prinb...done
  Scoring...done
  Generating summary...done

Only thing is, all the ones shown in summary buffer are ghosts..

Now looking a little further... C-u G del prinb  and watching the
record in *Messages*:

  Deleting group nnml:prinb...
  Deleting article /home/reader/Mail/prinb/.marks in prinb...
  Deleting article /home/reader/Mail/prinb/.overview in prinb...
  Deleting group nnml:prinb...done

Closing gnus, then emacs

grep prinb ~/.newsrc.eld Mail/active News/cache/active

prinb appears in .newsrc.eld in two places:

/home/reader/.newsrc.eld:(setq gnus-killed-list ==> prinb
/home/reader/.newsrc.eld:(setq gnus-topic-alist ==> prinb

Nothing in Mail/active but does appear in 
        News/cache/active: nnml:prinb 3031 3031 y
The directory is not actually in the cache though.

So removing the entry from News/cache/active, and restart

When a batch fetch is run I see a number of messages pulled from
procmail_dir/prinb.in  to Mail/prinb  (at the command line) 

No prinb group in evidence so pressing F  <no new groups>
Trying j nnml:prinb.... Yup there it is with 17 new messages including
another reply from Simon.  Respool the lot with B r and close the new
prinb.   Reopen it and there are the same messages only they are
ghosts.


   $ ls -al Mail/prinb
  total 20
  drwxrwxr-x    2 reader   reader       4096 Sep  4 08:57 .
  drwxr-xr-x   96 reader   reader       4096 Sep  4 08:57 ..
  -rw-r--r--    1 reader   reader         29 Sep  4 08:57 .marks
  -rw-------    1 reader   reader       4419 Sep  4 08:57 .overview

No messages but as you see, the .overview and marks files contains a
fare bit of data.

News/cache/active  has remained unchanged ... prinb is not in it now.

Sorry this is something of a ramble, but I couldn't see a tidy way to
express what is happening.

Re: Something very wrong here

[Kai Großjohann]

Harry Putnam <reader@newsguy.com> writes:

> First off, I guess setting nnml-marks-is-evil is no way to find out
> what is causing my problems.  So dropping that for now.

I think setting nnml-marks-is-evil would be worth a shot.  Setting it
effectively disables the .marks file handling, after all.

kai
-- 
Symbol's function definition is void: signature

Re: Something very wrong here

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

>    $ ls -al Mail/prinb
>   total 20
>   drwxrwxr-x    2 reader   reader       4096 Sep  4 08:57 .
>   drwxr-xr-x   96 reader   reader       4096 Sep  4 08:57 ..
>   -rw-r--r--    1 reader   reader         29 Sep  4 08:57 .marks
>   -rw-------    1 reader   reader       4419 Sep  4 08:57 .overview
>
> No messages but as you see, the .overview and marks files contains a
> fare bit of data.

Thanks for the detailed information.  The .overview file should be
empty. (The .marks file looked fine.)  I think the problem is that
`nnml-nov-delete-article' does not do what it should, because the
respool eval form changed some internal state.

Does the following patch make a difference?  The .overview file should
shrink to 0 bytes when you respool all articles.

FWIW I couldn't reproduce it.  I respooled all articles in a nnml
buffer, but the .overview file was empty after that.

If the patch doesn't do anything, maybe you could edebug
`nnml-request-move-article' and `nnml-nov-delete-article' to see that
the article file is really deleted (by `nnmail-delete-file-function')
and that `nnml-nov-delete-article' really removes the article from the
NOV file.  The latter doesn't seem to happen for you, maybe you can
find out why.

--- nnml.el.~6.22.~	Sun Sep  2 11:47:31 2001
+++ nnml.el	Tue Sep  4 21:01:51 2001
@@ -352,6 +352,7 @@
      (nnml-request-article article group server)
      (let (nnml-current-directory
 	   nnml-current-group
+	   nnml-nov-buffer-alist
 	   nnml-article-file-alist)
        (save-excursion
 	 (set-buffer buf)
@@ -361,6 +362,7 @@
 	 result))
      (progn
        (nnml-possibly-change-directory group server)
+       (nnmail-activate 'nnml t)
        (condition-case ()
 	   (funcall nnmail-delete-file-function
 		    (nnml-article-to-file  article))

Re: Something very wrong here

[Harry Putnam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> First off, I guess setting nnml-marks-is-evil is no way to find out
>> what is causing my problems.  So dropping that for now.
>
> I think setting nnml-marks-is-evil would be worth a shot.  Setting it
> effectively disables the .marks file handling, after all.

How is that done on a server?  When I attempt to edit my nnml server
with ^ in group buffer and then `e' on the nnml server.  I'm told the
server cannot be edited.

Re: Something very wrong here

[Kai Großjohann]

Harry Putnam <reader@newsguy.com> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> Harry Putnam <reader@newsguy.com> writes:
>>
>>> First off, I guess setting nnml-marks-is-evil is no way to find out
>>> what is causing my problems.  So dropping that for now.
>>
>> I think setting nnml-marks-is-evil would be worth a shot.  Setting it
>> effectively disables the .marks file handling, after all.
>
> How is that done on a server?  When I attempt to edit my nnml server
> with ^ in group buffer and then `e' on the nnml server.  I'm told the
> server cannot be edited.

You can't edit secondary servers.  For them, you have to change the
value of gnus-secondary-select-methods in ~/.gnus or wherever.

(This applies only if you use a list like (nnml "foo" ...) in
gnus-secondary-select-methods.  If you have a string like "nnml:foo"
or "foo" or something like this there, then `e' ought to work, I
guess.)

kai
-- 
Symbol's function definition is void: signature

Re: Something very wrong here

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:

[...]

> Thanks for the detailed information.  The .overview file should be
> empty. (The .marks file looked fine.)  I think the problem is that
> `nnml-nov-delete-article' does not do what it should, because the
> respool eval form changed some internal state.
>
> Does the following patch make a difference?  The .overview file should
> shrink to 0 bytes when you respool all articles.

I didn't get to try the patch but I see it is part of todays cvs so
instead I reupped my cvs and byte compiled the new el files.

I don't see any difference in the behavior.  Still after I respool
everthing, if I reopen that group before new stuff is added, I still
see all the old stuff listed in summary buffer.  

However, an additional point here.  Once new stuff is pulled into
prinb it seems the ghosts disappear, and the summary buffer shows only
the new stuff.

> FWIW I couldn't reproduce it.  I respooled all articles in a nnml
> buffer, but the .overview file was empty after that.

Seems to indicate a local config thing causing the problem then.   Do
you think the fact that this nnml group is generated from a procmail
spool has any baring?

> If the patch doesn't do anything, maybe you could edebug
> `nnml-request-move-article' and `nnml-nov-delete-article' to see that
> the article file is really deleted (by `nnmail-delete-file-function')
> and that `nnml-nov-delete-article' really removes the article from the
> NOV file.  The latter doesn't seem to happen for you, maybe you can
> find out why.

OK, now you've touched on a sore spot here.  I'm still too ignorant to
run edebug with any skill, and don't really know even how to get
started.

Looking it up in the info pages, it looks like I have a long ways to
go before I can just dance through stuff like you suggest.  Maybe you
can give me a couple of heavy clues to get started, and I'll fumble
with the info and help etc to get the rest.

What I don't know is how to start the debugger on those particular things.

Re: Something very wrong here

[Simon Josefsson]

Harry Putnam <reader@newsguy.com> writes:

>> Thanks for the detailed information.  The .overview file should be
>> empty. (The .marks file looked fine.)  I think the problem is that
>> `nnml-nov-delete-article' does not do what it should, because the
>> respool eval form changed some internal state.
>>
>> Does the following patch make a difference?  The .overview file should
>> shrink to 0 bytes when you respool all articles.
>
> I didn't get to try the patch but I see it is part of todays cvs so
> instead I reupped my cvs and byte compiled the new el files.

No, I don't think the patch is part of CVS.

>> FWIW I couldn't reproduce it.  I respooled all articles in a nnml
>> buffer, but the .overview file was empty after that.
>
> Seems to indicate a local config thing causing the problem then.   Do
> you think the fact that this nnml group is generated from a procmail
> spool has any baring?

How do procmail generate a nnml group?  Does it do everything Gnus
does to create a nnml group?  This may very well be the cause of the
problem.

> Looking it up in the info pages, it looks like I have a long ways to
> go before I can just dance through stuff like you suggest.  Maybe you
> can give me a couple of heavy clues to get started, and I'll fumble
> with the info and help etc to get the rest.

Try the patch first.

The simple steps to edebugging is to type M-x edebug-defun RET when
standing on the function to debug (by loading the corresponding .el
file), then use Gnus, and then press SPC to step through the code once
the edebug window is shown.  Press `e' to evaluate variables.

Re: Something very wrong here

[Kai Großjohann]

Harry Putnam <reader@newsguy.com> writes:

> I don't see any difference in the behavior.  Still after I respool
> everthing, if I reopen that group before new stuff is added, I still
> see all the old stuff listed in summary buffer.  

I hadn't really paid any attention up to here, but this sounds like
something I've seen before.  What happens when you reopen the group,
see the ghosts, and then select any of the ghosts for viewing?

In a totally unrelated circumstance (which therefore is probably a
symptom of the same bug ;-) I saw the ghost turn yellow on black which
indicates canceled messages.  What was the letter again?  G?  I
forget.

kai
-- 
Symbol's function definition is void: signature

Re: Something very wrong here

[Harry Putnam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> I don't see any difference in the behavior.  Still after I respool
>> everthing, if I reopen that group before new stuff is added, I still
>> see all the old stuff listed in summary buffer.  
>
> I hadn't really paid any attention up to here, but this sounds like
> something I've seen before.  What happens when you reopen the group,
> see the ghosts, and then select any of the ghosts for viewing?

he he... or you would have know that I get the cancelled or expunged
message and the `G'.  It was in the first post of the thread.

Na na na na na.  I'm telling everybody. I caught Kai napping... he he.

Re: Something very wrong here

[Harry Putnam]

Simon Josefsson <jas@extundo.com> writes:


>> I didn't get to try the patch but I see it is part of todays cvs so
>> instead I reupped my cvs and byte compiled the new el files.
>
> No, I don't think the patch is part of CVS.

Yikes, I should have looked a little closer but the nnml.el line in
todays cvs are almost identical.  Of course that is what patches are
all about.  So duly patched byte compiled and restarted.

The bad news is I see no difference.  The ghosts are still there.

> How do procmail generate a nnml group?  Does it do everything Gnus
> does to create a nnml group?  This may very well be the cause of the
> problem.

Nothing tricky... I just meant in the normal way.  That is, I have a
procmail spool directory where all incoming messages go to a file
named `prinb.in'.  Gnus slurps from that file and puts the messages in
nnml group `prinb'.  That is where the difficulty is occuring.  

So that nnml group is not made by hand nor is it a product of the
nnmail split.  It is generated from a procmail spool file.  From this
code:

(setq nnmail-use-procmail t) 
(setq mail-sources
        '((directory :path "/home/reader/spool/" 
                   :suffix ".in"))) 

> Try the patch first.

Didn't seem to help.

> The simple steps to edebugging is to type M-x edebug-defun RET when
> standing on the function to debug (by loading the corresponding .el
> file), then use Gnus, and then press SPC to step through the code once
> the edebug window is shown.  Press `e' to evaluate variables.

OK, thanks... working on that now.

Re: Something very wrong here

[Harry Putnam]

Harry Putnam <reader@newsguy.com> writes:

>> Try the patch first.
>
> Didn't seem to help.
>
>> The simple steps to edebugging is to type M-x edebug-defun RET when
>> standing on the function to debug (by loading the corresponding .el
>> file), then use Gnus, and then press SPC to step through the code once
>> the edebug window is shown.  Press `e' to evaluate variables.
>
> OK, thanks... working on that now.

A first take on this shows 
Pressing `B r' on a single message

Then run edebug and step through the
code the message is deleted and does not appear on reopening.

Close gnus restart emacs,gnus and try a buffer of 4 messages with no
edebug and the same ghost phenomena occurs.  I'm checking now what
happens to the actual .overview.