POP/fancy-split bug?

POP/fancy-split bug?

[Norman Walsh]

Hello world,

I've discovered that either POP or mail splitting is doing
something truly evil with mail that goes to a list that I
moderate. (I'm using 5.8.3, btw.)

Messages sent to me (the moderator) for approval look like this:

  From owner-docbook@lists.oasis-open.org Tue Jan 11 06:48:58 2000
  Received: from lists.unicomp.net (lists.unicomp.net [209.41.64.203])
  	by nexus.berkshire.net (8.9.2/8.9.2) with ESMTP id GAA04422
  	for <ndw@nwalsh.com>; Tue, 11 Jan 2000 06:48:57 -0500 (EST)
  From: owner-docbook@lists.oasis-open.org
  Received: (from majordomo@localhost)
  	by lists.unicomp.net (8.8.7/8.8.7) id GAA17370;
  	Tue, 11 Jan 2000 06:38:41 -0600
  Date: Tue, 11 Jan 2000 06:38:41 -0600
  Message-Id: <200001111238.GAA17370@lists.unicomp.net>
  X-Authentication-Warning: lists.unicomp.net: majordomo set sender to owner-docbook@lists.oasis-open.org using -f
  To: owner-docbook@lists.unicomp.net
  Subject: BOUNCE docbook@lists.oasis-open.org: Approval required:     
  Content-Length: 1565

  From owner-docbook@lists.unicomp.net  Tue Jan 11 06:38:40 2000
  Received: from nexus.berkshire.net (nexus.berkshire.net [206.72.196.10])
  	by lists.unicomp.net (8.8.7/8.8.7) with ESMTP id GAA17367
  	for <docbook@lists.oasis-open.org>; Tue, 11 Jan 2000 06:38:39 -0600
  Received: from nwalshpc.berkshire.net (nwalshpc.nwalsh.com [140.186.114.234])
  	by nexus.berkshire.net (8.9.2/8.9.2) with ESMTP id GAA04415
  	for <docbook@lists.oasis-open.org>; Tue, 11 Jan 2000 06:48:53 -0500 (EST)
  Date: Tue, 11 Jan 2000 06:49:02 -0500
  Message-ID: <7196-Tue11Jan2000064902-0500-ndw@nwalsh.com>
  X-Mailer: emacs 20.4.1 (via feedmail 7 I) 
  To: docbook@lists.oasis-open.org
  Subject: This is another test
  Reply-to: Norman Walsh <ndw@nwalsh.com>
  From: Norman Walsh <ndw@nwalsh.com>
  X-URL: http://nwalsh.com/
  X-Millennium: T-minus 50 weeks, 5 days, 17 hours, 11 minutes, 16 seconds
  Lines: 12
  User-Agent: Gnus/5.0802 (Gnus v5.8.2) Emacs/20.4
  MIME-Version: 1.0
  Content-Type: text/plain; charset=us-ascii
  
  this is another test, unapproved it should not be posted to the list.
  
                                          Cheers,
                                            norm
  
  -- 
  Norman Walsh <ndw@nwalsh.com>      | The universe that we observe has
  http://www.oasis-open.org/docbook/ | precisely the properties we should
  Member, DocBook Editorial Board    | expect if there is, at bottom, no
                                     | design, no purpose, no evil and no
                                     | good, nothing but pitiless
                                     | indifference.--Richard Dawkins

When my mail-source was (file ...), this always worked just fine
(and has for months and months): I get a single mail message in
the "approval" folder with the body set to the headers and body
of the message that I'm supposed to approve. 

But now that I'm using POP, I've noticed that I sometimes get a
completely empty message in the "approval" folder and the
unapproved message in the "list" folder, as if it had been
approved. I haven't been able to nail down what's different
between messages that succeed and messages that fail, but most
seem to fail.

Any thoughts?
                                        Cheers,
                                          norm

-- 
Norman Walsh <ndw@nwalsh.com>      | Life is a great bundle of little
http://nwalsh.com/                 | things.--Oliver Wendell Holmes

Re: POP/fancy-split bug?

[Kai Großjohann]

Norman Walsh <ndw@nwalsh.com> writes:

> But now that I'm using POP, I've noticed that I sometimes get a
> completely empty message in the "approval" folder and the
> unapproved message in the "list" folder, as if it had been
> approved. I haven't been able to nail down what's different
> between messages that succeed and messages that fail, but most
> seem to fail.

Depending on how you fetch mail, the output of the fetching process
may be a babyl file (default with movemail.c) or an mbox file (can
happen with pop3.el but don't know if that always happens).  If it is
an mbox file, a message is considered to start with a From_ line and
ends with an empty line.

The unapproved message starts with a From_ line, hence it is
conceivable that this From_ line is considered start of a message.
But why does this not always happen?  Well, I think Gnus also looks at
the Content-Length (maybe also Lines) header.  Gnus then moves forward
that many bytes or lines to see if a new message begins there.  If so,
the header information is considered to be correct.  If, however,
there is no From_ line after the number of bytes/lines given by
Content-Length, Gnus reverts to searching for From_ lines for
beginnings of messages.

Maybe everything works if the Content-Length header is good, and the
message is split in two if the Content-Length header is wrong?

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: POP/fancy-split bug?

[Norman Walsh]

/ Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) was heard to say:
| Depending on how you fetch mail, the output of the fetching process
| may be a babyl file (default with movemail.c) or an mbox file (can
| happen with pop3.el but don't know if that always happens).  If it is
| an mbox file, a message is considered to start with a From_ line and
| ends with an empty line.

That does seem to be the problem, though it doesn't appear to be
related to Content-Length. Many of the corrupted messages
include a content-length that appears to be valid.

I took a quick peek at epop3mail and it would appear to have the
same problems. At least, I don't see anything that makes me
think it works around the "From " problem.

As you suggested, the external movemail program seems to create
BABYL boxes (and therefore wouldn't suffer from this bug?), but
using it means I don't get any feedback as messages are
downloaded. Since I can't seem to prevent very big messages from
being downloaded, that's unfortunate.

I'm just mortified. Does everyone just live with these problems?
Is POP3 not much used among the gnusers? (Clearly, I have to get
my other system working again :-)

                                        Cheers,
                                          norm

-- 
Norman Walsh <ndw@nwalsh.com>      | It is necessary to try to surpass
http://nwalsh.com/                 | oneself always; this occupation
                                   | ought to last as long as
                                   | life.--Christina, Queen of Sweden

Re: POP/fancy-split bug?

[Kai Großjohann]

Norman Walsh <ndw@nwalsh.com> writes:

> As you suggested, the external movemail program seems to create
> BABYL boxes (and therefore wouldn't suffer from this bug?), but
> using it means I don't get any feedback as messages are
> downloaded. Since I can't seem to prevent very big messages from
> being downloaded, that's unfortunate.

But it ought to be simple to use pop3.el to create a babyl file.  You
could take pop3-movemail (which only was intended to be an example,
anyway!) and adapt it to produce a babyl file.  Gnus will
automagically grok this and DTRT.

kai
-- 
A large number of young women don't trust men with beards.  (BFBS Radio)

Re: POP/fancy-split bug?

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Norman Walsh <ndw@nwalsh.com>  on Tue, 11 Jan 2000
| That does seem to be the problem, though it doesn't appear to be
| related to Content-Length. Many of the corrupted messages
| include a content-length that appears to be valid.

Well, FWIW, that Content-Length is valid, as far as I can tell.  What I
think we need to see is where the next message in the Incoming* file abuts
the one that is split incorrectly.  If there is no blank line before the
next message delimter (some POP servers are notorious for stripping
whitespace) then the Content-Length split check will fail and Gnus will
fall back to looking for mbox separators.

If you are going to rewrite pop3.el functions, pop3-munge-message-separator
is the best place to do it.  I probably should have forced the MMDF format
instead of mbox; I really do not know why I did not.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.1 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE4e98Zgl+vIlSVSNkRAojlAJ0fHwqwMfkAiB2HdZ5gIVA8kr7CdwCfRKXG
GTyknpoTuPhqSr/z/9rDFqA=
=/u4H
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Warning: pregnant women, the elderly, and
Minion of Nathan - Nathan says Hi! \ children under 10 should avoid prolonged
PGP Key: at a key server near you!  \ exposure to Happy Fun Ball.