Debug a mail hiding problem

Debug a mail hiding problem

[Harry Putnam]

The following message is a courtesy copy of an article
that has been posted to gnu.emacs.gnus as well.


This report is something of a ramble since its a bit hard to describe
what is happening here.

I recently switched to using slrnpull to get a small newsfeed on disk.
Also changing a few scripts etc that fetch mail and news in batch
mode.

The basic batch command looks like:

 emacs   -batch -l ~/.emacs-dir/batch-fetch.el -f lars-fetch-news 2>&1\
 |awk -v"DATE=`date +"%b %d %T "`" '{print DATE $0}' >>$LOG

(The awk part just prepends the date to batch command output)
`lars-fetch-news' looks like:

	 (defun lars-fetch-news ()
	   (interactive)
	   ;(push "/usr/local/pgnus-0.96/lisp" load-path)
	   (let ((init-file-user "")
	        (gnus-always-read-dribble-file t))
	     (gnus))
	   (gnus-group-send-drafts)
	   (gnus-agent-fetch-session)
	   (gnus-group-quit))

(above code has been in place for months) 

Ever since the slrnpull changeover, I have this recurring problem
where something happens intermittantly during a mail fetch that causes
a few bytes (newlines) to be written to .emacs-mail-crash-box.

When the next batch fetch discovers the newlines in
.emacs-mail-crash-box it stops, thinking there is corrupt mail there.
Removing the .emacs-mail-crash-box file allows things to proceed
normally.

However the mail that was being collected disappears from gnus view.
Its still on disk, but gnus can't see it.  The .overview file is there
but incorrect.  

Deleting the .overview and running `nnml-generate-nov-databases-1' on
that directory resurrects the .overview but gnus still cannot see the
mail.  Even an M-c `remove all marks' from group buffer.  And a 
C-u <RET> will not display it.

Sometimes the above proceedure does fix things, sometimes not.  I
haven't been able to tell why.

My setup is somewhat unusual in this way:  I have procmail writing to
several spools in ~/spool.  Any mail not written to a specific spool
is delivered by procmail to ~/spool/prinb.in.  No mail is left on 
/var/spool/mail/$USER.

Gnus does all its collecting from that spool directory. This code is
in place for mail-sources:

  (setq nnmail-use-procmail t)
  (setq mail-sources
  ;;; dont need the file rule when procmail is delivering to prinb.in
        ;'((file      :path "/var/spool/mail/reader")
          '((directory :path "/home/reader/spool/"
                     :suffix ".in")))
  
Once gnus laps up the mail from ~/spool/*.in, then any mail not dropped
by procmail into special `~/spool/whatever.in' files shows up in
`nnml:prinb'.  Where it is scanned by me and then respooled with `B r'
and run through my split rules.

This basic setup has been in place for months, without problems.  Its
only since the slrnpull changeover that things have been breaking.  I
don't believe slrnpull has any bearing on this but something else I've
changed and forgotten or something similar.

The problem described seems to happen at the point gnus is writing to
`nnml:prinb'.  The outward manifestation can be seen while `tailing'
the output of the batch fetch.  I see the write to `nnml:prinb' stop in mid
stride.  Then the appearance of ~/.emacs-mail-crash-box.

At that point any number of messages may in fact be in ~/Mail/prinb
but are somehow rendered invisile to gnus.
 
Several times now, I could only get at that mail by `G d' (nndir) on
that directory and respool.

I have a hunch that what may be happening is procmail is writing to
~/spool/prinb.in at the same time gnus is collecting from there.

But I guess there is something in gnus to protect against that.. isn't
there? 

My procmail recipe looks like:

          :0
        prinb.in

I haven't tried putting the lock on it like :0: but I don't think that
is what that does any way.  And that recipe has been in place for months.

I've tried running a little looping script that looks for
.emacs-mail-crash-box and takes a quick snap shot of ps output.
Hoping to be able to identify something as a problem like procmail
running at the same time as the fetch command.  The loop is on a one
second cycle it writes the output of `ps waux' to a file every second
until it sees .emacs-mail-crash-box.  Then writes a long `ls' on that to
the file for date comparison.

I haven't really been able to discover anything usefull this way.

Can any of you experienced gnus users see something in this
description that might be a clue to solving the problem?