Re: Gnus, movemail, POP3, trailing empty lines

Re: Gnus, movemail, POP3, trailing empty lines

[St. Suika Fenderson Roberts]

visigoth@naiad.fac.cs.cmu.edu wrote:
> Rich Pieri <rich.pieri@PrescientTech.com> writes:

> > Then the mail server is messing up, because there *must* be one.  The
> > Unix mail separator is "\nFrom ...".  The newline must be there to
> > distinguish one message from the next.

> Actually, I read recently that "^From ..." is the separator.  Any
> reasonable mail agent ought to produce only "\nFrom ", but should
> accept any line beginning with "From " as beginning a new message,
> regardless of whether or not it was preceeded by a blank.

On the subject, a while back I was playing with my .forward file, and
found that gnus would properly handle mails split by `^From ...' instead
of the more standard `\n\nFrom ...' (I found this out by munging things
until I found a way to echo a bare `\n' onto the end of each mail :/ )

Has the code changed since?

	Suika
-- 
                      wrobert2@mailhost.tcs.tulane.edu
An effective way to deal with predators is to taste terrible.
                                                        --/usr/games/fortune
  <a href="http://studentweb.tulane.edu/~wrobert2/">Suika no homepage</a>

Re: Gnus, movemail, POP3, trailing empty lines

[Lars Magne Ingebrigtsen]

"St. Suika Fenderson Roberts" <wrobert2@mailhost.tcs.tulane.edu> writes:

> On the subject, a while back I was playing with my .forward file, and
> found that gnus would properly handle mails split by `^From ...' instead
> of the more standard `\n\nFrom ...' (I found this out by munging things
> until I found a way to echo a bare `\n' onto the end of each mail :/ )
> 
> Has the code changed since?

Yes.  The Emacs regexp handler overflowed on the complex regexp used
to identify the "From " headers.  (And it didn't recognize some of
them.)

nnmail now uses a simpler (ie., more complicated) rule; involving
"\n\nFrom". 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Gnus, movemail, POP3, trailing empty lines

[Jason L Tibbitts III]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

LMI> nnmail now uses a simpler (ie., more complicated) rule; involving
LMI> "\n\nFrom".

Sometimes Lars just has to show everyone that he operates on a completely
different plane...

 - J<

Re: Gnus, movemail, POP3, trailing empty lines

[Lars Magne Ingebrigtsen]

visigoth@naiad.fac.cs.cmu.edu writes:

> > Uhm...  But the problem is that "From " lines in some mboxes *aren't*
> > quoted.
> 
> Yes--but those mailboxes should be using a Content-Length header to
> show how long the message is.

They should, but some don't.

> You do it one way or the other, not both.

You can do one, the other, both and neither.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Gnus, movemail, POP3, trailing empty lines

[visigoth]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

> Uhm...  But the problem is that "From " lines in some mboxes *aren't*
> quoted.

Yes--but those mailboxes should be using a Content-Length header to
show how long the message is.  You do it one way or the other, not
both.  If the mboxes are CL-style, you shouldn't be looking at the
>From lines at all.  If From-lines are being looked at, all lines
starting with "From " are from-lines and should be interpreted as the
beginning of new messages.

BEGIN Rathole-Protocol v1
IHAVE From lines
IDONTWANT From lines
IHAVE From lines
IDONTWANT From lines
...

<grin>

John.

Re: Gnus, movemail, POP3, trailing empty lines

[Lars Magne Ingebrigtsen]

visigoth@naiad.fac.cs.cmu.edu writes:

> It's my belief, that regardless of what any standard says, Gnus should
> accept messages in this broken format.  Remember: the goal is to
> accept as many formats as possible, even if they're broken, and to
> produce only correct output.
> 
> >From ...
> some stuff
> >From this line is in the message, not a new one
> some more stuff
> 
> should never be generated as an mbox file, unless Content-Length is
> included in the file, because From lines are quoted with >.  This is
> *not* ambiguous.  The above can only be generated by two sequential
> messages.

Uhm...  But the problem is that "From " lines in some mboxes *aren't*
quoted.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Gnus, movemail, POP3, trailing empty lines

[Lars Magne Ingebrigtsen]

Mark Eichin <eichin@cygnus.com> writes:

> xxFrom
> xx>From
> xx>>From
> 
> in an actual *message* gets stored as
> 
> xx>From
> xx>>From
> xx>>>From
> 
> and then the MTA before displaying it strips one > from any >*From
> line (that's a regexp "kleene" * not a glob *) yielding
> 
> xxFrom
> xx>From
> xx>>From
> 
> That's the Right Way(tm) to quote From lines.

This seems reasonable to me, and it's easy to implement.  I've put it
on the Quassia Gnus todo list.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Gnus, movemail, POP3, trailing empty lines

[Mark Eichin]

> Let's try that again: tell me, how does this algorithm know that a given
> quoted "From " line in the body of the message is one that should be

I think what you're missing is that you *don't* just quote ^From
lines, you also quote ^>From, iteratively... you have to, it's the
only way to do it reliably. So, (xx inserted so the example doesn't
get mangled)

xxFrom
xx>From
xx>>From

in an actual *message* gets stored as

xx>From
xx>>From
xx>>>From

and then the MTA before displaying it strips one > from any >*From
line (that's a regexp "kleene" * not a glob *) yielding

xxFrom
xx>From
xx>>From

That's the Right Way(tm) to quote From lines.  I've *never* seen a
mailer actually do that; they all corrupt messages instead, taking
only any xxFrom and making it xx>From, then any xx>From and making
xxFrom even if it wasn't one it converted.

Re: Gnus, movemail, POP3, trailing empty lines

[Rich Pieri]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "PA" == Per Abrahamsen <abraham@dina.kvl.dk> writes:

PA> Rich Pieri <rich.pieri@PrescientTech.com> writes:
>> So tell me, how does this algorithm know that a given quoted "From "
>> line in the body of the message is one that should be dequoted?

PA> Yes.  They all should, if created by the algorithm above.  This leads
PA> to zero loss of information, unlike the traditional algorithm.

Let's try that again: tell me, how does this algorithm know that a given
quoted "From " line in the body of the message is one that should be
dequoted?  How does it distinguish a "From " line it quoted from one
that happens to have an identical "quoting" that it did not perform?

Put it another way, if the algorithm does not work 100% of the time it
should not be used.  99.8% of the time is simply not good enough.  At
the least it is going to screw up citations.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3a
Charset: noconv

iQCVAwUBMv35Dp6VRH7BJMxHAQHLgQP/f6ODPWngqg7Ag1gT2T/OLbqWoWTdZwx1
mF6tOZ2n/3l9ofoqIYuhd9/fpehR92QeMYPk5v2L2pOWssMRrJd8BjTyANHqcPEm
bEWiXUGoWF4zK0Aas5moHEjyjxYFIMyW6lu08iQHz4oLn4vvW+DMTlLPqa4gEXY2
Dg+E5MomVNQ=
=Ys8i
-----END PGP SIGNATURE-----
-- 
Rich Pieri <rich.pieri@prescienttech.com> | When not in use, Happy Fun Ball
Prescient Technologies, Inc.              | should be returned to its special
A Stone & Webster Company                 | container and kept under
I speak for myself, not PTI or SWEC       | refrigeration.

Re: Gnus, movemail, POP3, trailing empty lines

[Randal Schwartz]

>>>>> "Per" == Per Abrahamsen <abraham@dina.kvl.dk> writes:

Per> Rich Pieri <rich.pieri@PrescientTech.com> writes:

>> So tell me, how does this algorithm know that a given quoted "From "
>> line in the body of the message is one that should be dequoted?

Per> Yes.  They all should, if created by the algorithm above.  This leads
Per> to zero loss of information, unlike the traditional algorithm.

Huh?  What if my *original* message just happened to have ">From " in one
of the lines?  You've now turned it into "From ".

There is *no* way to get both "From " and ">From " safely from the
original message all the way to the end user, if "From " is also the
intermessage delimiter.  So, you must compromise on one or the other.

Most MUAs that I've seen turn "From " into ">From ", and then leave it
alone.  That seems reasonable, but it may just be that it's
particularly familiar.

-- 
Name: Randal L. Schwartz / Stonehenge Consulting Services (503)777-0095
Keywords: Perl training, UNIX[tm] consulting, video production, skiing, flying
Email: <merlyn@stonehenge.com> Snail: (Call) PGP-Key: (finger merlyn@ora.com)
Web: <A HREF="http://www.stonehenge.com/merlyn/">My Home Page!</A>
Quote: "I'm telling you, if I could have five lines in my .sig, I would!" -- me

Re: Gnus, movemail, POP3, trailing empty lines

[Per Abrahamsen]

Rich Pieri <rich.pieri@PrescientTech.com> writes:

> So tell me, how does this algorithm know that a given quoted "From "
> line in the body of the message is one that should be dequoted?

Yes.  They all should, if created by the algorithm above.  This leads
to zero loss of information, unlike the traditional algorithm.

Re: Gnus, movemail, POP3, trailing empty lines

[Rich Pieri]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "PA" == Per Abrahamsen <abraham@dina.kvl.dk> writes:

PA> Apropos: I think it would be nice if Gnus supported the less brain
PA> damaged mbox format DJB advocates: When a plain text message is
PA> imported to an mbox, all lines matching "^>*From " are quoted.  When
PA> a mbox message is exported (displayed, saved to plain text), all
PA> lines staring with `^>+From " are unquoted.

So tell me, how does this algorithm know that a given quoted "From "
line in the body of the message is one that should be dequoted?  No, the
MUA should not touch anything in the body of the message if there is any
way at all to avoid it.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3a
Charset: noconv

iQCVAwUBMv09QZ6VRH7BJMxHAQFUkgP/cOGIUZPFXBaYmvpRZnfOqwVKKkgr87+g
gKwgwk6mOCdhSFHdpJ4qi70MlV2BUTd/gWL2uBvQukrP0nX6mh02TVtEZCII0pyV
L3Idm/7vro67aynNEjOY9sZPvSyX88SXbCPK12bYo3fwgbxbJfci/lAIGr1F65O6
EhPmYRDXXGI=
=WvTX
-----END PGP SIGNATURE-----
-- 
Rich Pieri <rich.pieri@prescienttech.com> | Do not use Happy Fun Ball on
Prescient Technologies, Inc.              | concrete.
A Stone & Webster Company                 | 
I speak for myself, not PTI or SWEC       | 

Re: Gnus, movemail, POP3, trailing empty lines

[Steven L Baur]

Frank D Cringle writes:

> Per Abrahamsen <abraham@dina.kvl.dk> writes:
> [ about mbox From_ quoting ]

>> JWZ hates it, but he is wrong and DJB is right.

> I agree.

As do I.  Isn't this what Gnus is doing now?  Or was it that only
archiving outgoing messages was changed?
-- 
steve@miranova.com baur
Unsolicited commercial e-mail will be billed at $250/message.

Re: Gnus, movemail, POP3, trailing empty lines

[Frank D. Cringle]

Per Abrahamsen <abraham@dina.kvl.dk> writes:
[ about mbox From_ quoting ]

>JWZ hates it, but he is wrong and DJB is right.

I agree.

-- 
Frank Cringle,      fdc@cliwe.ping.de
voice: (+49 2304) 467101; fax: 943357

Re: Gnus, movemail, POP3, trailing empty lines

[Per Abrahamsen]

visigoth@naiad.fac.cs.cmu.edu writes:

> Okay--here's a quote from the document I found which talks about mbox
> format.  It's from the man pages that come with qmail (yes,
> D.J. Bernstein is a flaming, uh, maniac, but I think this is
> reasonable):

Apropos: I think it would be nice if Gnus supported the less brain
damaged mbox format DJB advocates: When a plain text message is
imported to an mbox, all lines matching "^>*From " are quoted.  When
a mbox message is exported (displayed, saved to plain text), all
lines staring with `^>+From " are unquoted.

It has two advantages:

1) No loss of information occurs when used on new messages.  Ever.

2) When the algorithm is used on old messages already in an mbox
   folder, in 99.8% percent of the cases, the results are more
   correct. 

JWZ hates it, but he is wrong and DJB is right.

Re: Gnus, movemail, POP3, trailing empty lines

[visigoth]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

> Yes; the change was made around 0.70, I think.  The new version is
> more correct than the last version, but it doesn't deal with these
> types of misconstructed spool files.  The "From " header is supposed
> to have a blank line before it, from what I understand.

Okay--here's a quote from the document I found which talks about mbox
format.  It's from the man pages that come with qmail (yes,
D.J. Bernstein is a flaming, uh, maniac, but I think this is
reasonable):

       A message encoded in mbox format begins with a From_ line,
       continues with a series of non-From_ lines, and ends  with
       a  blank  line.   {...}

HOW A MESSAGE IS READ
       
       A reader scans through an  mbox  file  looking  for  From_
       lines.   Any  From_ line marks the beginning of a message.
       The reader should not attempt to  take  advantage  of  the
       fact  that  every  From_  line  (past the beginning of the
       file) is preceded by a blank line.

{...}

It's my belief, that regardless of what any standard says, Gnus should
accept messages in this broken format.  Remember: the goal is to
accept as many formats as possible, even if they're broken, and to
produce only correct output.

>From ...
some stuff
>From this line is in the message, not a new one
some more stuff

should never be generated as an mbox file, unless Content-Length is
included in the file, because From lines are quoted with >.  This is
*not* ambiguous.  The above can only be generated by two sequential
messages.

The fact that we know of at least one place that the above can happen
for two messages and no places where it can occur for a single message
leads me to a rather obvious conclusion.

Sorry to rant, but I don't see why this is being argued about.

John.

Re: Gnus, movemail, POP3, trailing empty lines

[Lars Magne Ingebrigtsen]

Kai Grossjohann <grossjohann@ls6.informatik.uni-dortmund.de> writes:

> OK, it seems that movemail creates a babyl file if reading from a POP3
> server.  I have already asked Michael to please run movemail manually
> to show you the babyl file that's produced when a message doesn't end
> with an empty line.
> 
> Anyhow, it worked for him with Gnus 5.2.40 and Rgnus 0.50 (!) and
> discontinued working in rgnus 0.70 (or 0.71 or so).  It still doesn't
> work with 5.4.8.

Yes; the change was made around 0.70, I think.  The new version is
more correct than the last version, but it doesn't deal with these
types of misconstructed spool files.  The "From " header is supposed
to have a blank line before it, from what I understand.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Gnus, movemail, POP3, trailing empty lines

[Michael Huehne]

>>>>> "dave" == dave edmondson <dme@sco.com> schreibt:

 dave> surely discussions about delivery agent are secondary.  the
 dave> first thing i'd check is that the pop server is actually
 dave> dishing the messages out in distinct pieces.  if this is the
 dave> case, then movemail should be coerced into generating a correct
 dave> mailbox file (ie. it should be told to ensure that there is a
 dave> \n\nFrom seperator).  if the pop server _doesn't_ give out
 dave> these two as seperate messages, then perhaps it and the
 dave> delivery agent on the pop serving machine should be examined.

Hello Dave,

I  need to say  thanks  to all you  friendly   folks trying  to help
me.  But, as far as I  could see this  must  be a problem  of a gnus
later then rgnus-0.50.

I   have never   had this  kind   of problems  using   rguns-0.50 or
gnus-5.2.40 or much older gnus.  I have not changed my emacs-19.34.1
or movemailbinary since many month.

I first saw this problem in  gnus-0.71 or so.  After loosing several
Mails I gave  up testing beta gnus  versions later than .50. 

But now the problem  occurs again with  a  official gnus-5.4.8.   In
this  moment  I told  Kai   (my  friend an   Lisp  GURU)  about  the
problem. He was so kind  to suport me with his  initial mail in this
thread.

All I have changed is gnus ! So there must be the problem.

By  the way ... I love  to use gnus. But  the  sgnus t-shirts in xxl
from Lars are *not* fitting for 3 meter giants !

-- 
   Freundliche Grüße,
   Best regards,

   Michael Hühne

 ==================================== Michael Hühne
 =    ____                          = Lt. technische Vertriebsunterstützung
 =   /      /   /   ___    ___  _/_ = Email: Michael.Huehne@Germany.EU.net
 =  /----  /   /  /   /  /___/  /   = EUnet Deutschland GmbH
 = /____  /___/  /   /  /___   /    = Emil-Figge-Str. 80
 ===                              === D-44227 Dortmund Germany
 === Connecting Europe since 1982 === Tel.: +49/231/972-1071 Fax.: -1122

 #include <disclaimer.h>

Re: Gnus, movemail, POP3, trailing empty lines

[Kai Grossjohann]

>>>>> Kai Grossjohann writes:

  Kai> Hi all,
  Kai> Michael uses movemail to read mail from a POP3 server.  If a mail
  Kai> doesn't end with an empty line, the following mail isn't split
  Kai> correctly.

Rehi,

in the last thread I said some strange things.  I must've been out of
my mind (or haven't thought long enough about what happens).

OK, it seems that movemail creates a babyl file if reading from a POP3
server.  I have already asked Michael to please run movemail manually
to show you the babyl file that's produced when a message doesn't end
with an empty line.

Anyhow, it worked for him with Gnus 5.2.40 and Rgnus 0.50 (!) and
discontinued working in rgnus 0.70 (or 0.71 or so).  It still doesn't
work with 5.4.8.

I did a cursory search of nnmail-process-babyl-mail-format in 5.2.40's
nnmail.el and in 5.4.11's nnmail.el but couldn't find anything
offhand.  These functions *are* different, though, but I can't
immediately see the effects.

You'll be hearing from us later, I presume.

kai
-- 
My heart is down, my head is turning around;
I had to leave a little girl in Kingston town. -- Harry Belafonte

Re: Gnus, movemail, POP3, trailing empty lines

[Kai Grossjohann]

>>>>> Rich Pieri writes:

  Rich> I am inclined to agree; it is the delivery agent this guy is
  Rich> using that is the problem.  It is failing to create proper
  Rich> delimiters in the system mailbox, resulting in the POP server
  Rich> seeing the entire mailbox as a single message.

I know for sure that the POP3 server works well in that respect:
talking POP3 directly to the server reveals that it groks the
different messages all right.  (I use the same POP3 server and have
talked to it like that.)

kai
-- 
My heart is down, my head is turning around;
I had to leave a little girl in Kingston town. -- Harry Belafonte

Re: Gnus, movemail, POP3, trailing empty lines

[Rich Pieri]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "KG" == Kai Grossjohann
>>>>> <grossjohann@charly.informatik.uni-dortmund.de> writes:

KG> I do not know whether a POP3 server should be required to tack on an
KG> extra CRLF to the end of each message, thereby ensuring the twice-CRLF
KG> case always, or whether movemail should tack on the extra CRLF, or
KG> whether Gnus should be taught to do without the extra CRLF.

The "end of message" marker that the POP3 server spits out is
"\r\n.\r\n".  The first "\r\n" pair will start on a new line (it will
not be attached to the end of the last line of the message), and will be
written out to the local mail file; the ".\r\n" will not be written.
Then the next message is tacked on after that:

======
...
A Stone & Webster Company                 | and cover head.
I speak for myself, not PTI or SWEC       | 
\r\n
- From ...
...
======

KG> But somehow I think that the POP3 server is *not* the problem here.

I am inclined to agree; it is the delivery agent this guy is using that
is the problem.  It is failing to create proper delimiters in the system
mailbox, resulting in the POP server seeing the entire mailbox as a
single message.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3a
Charset: noconv

iQCVAwUBMvdUcZ6VRH7BJMxHAQE1HwP7B8rD8mBXH9c/jzdPDVnMBnHzifbQygM7
SObU54H8bl/qU5PC1vM0lIueJWmFGTz4I1O5mlFxqqIMAPSoVvuYGiT+lcRMOAFm
pm8FI11WhQLk3XpZ5EjyaxMjEyhQLOBuf05/P29ZhfPPZ/IDhpWhGvZ527wMOTrt
jVMM3/qTYio=
=n66P
-----END PGP SIGNATURE-----
-- 
Rich Pieri <rich.pieri@prescienttech.com> | Caution: Happy Fun Ball may
Prescient Technologies, Inc.              | suddenly accelerate to dangerous
A Stone & Webster Company                 | speeds.
I speak for myself, not PTI or SWEC       | 

Re: Gnus, movemail, POP3, trailing empty lines

[Rich Pieri]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "de" == dave edmondson <dme@sco.com> writes:

de> surely discussions about delivery agent are secondary.  the first
de> thing i'd check is that the pop server is actually dishing the
de> messages out in distinct pieces.

The POP server has no other choice than to do this.  There is no POP
command to retrieve an entire mailbox as a single chunk; each message
must be explicitly downloaded with a `RETR <msgid>' command.

I still say that it is the MTA (actually the delivery agent responsible
for appending to the mailbox) that is failing to insert the requisite
blank line before the new "\nFrom " line.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3a
Charset: noconv

iQCVAwUBMvdNSJ6VRH7BJMxHAQEYMQP+PwVhW0A+sQBctGVMpVnCbkb3tf3fXZqy
qan4P50xSuLimtfOQwOI9/AB/JhbTuGwj+f9QXj2qw98TnSxeveeGfM2+wsfQdmN
MkgUhwTSP3zdrGodzlKwMTNGfbwH5OcGSHrttz26R7GORRv3sDYra/VTAwan22S/
EkyDyjk9GEc=
=n+vl
-----END PGP SIGNATURE-----
-- 
Rich Pieri <rich.pieri@prescienttech.com> | Happy Fun Ball may stick to certain
Prescient Technologies, Inc.              | types of skin.
A Stone & Webster Company                 | 
I speak for myself, not PTI or SWEC       | 

Re: Gnus, movemail, POP3, trailing empty lines

[Bernard.Steiner]

     The POP3 server spits out a message at a time.  The last line of some
     messages is empty, or some others it is non-empty.  movemail concats
     all messages.  This results in a file that contains several messages,
     one after the other, and in some cases, a message begins with
     "\r\n\r\nFrom " (twice CRLF) whereas in other cases, a message begins
     with "\r\nFrom " (just one CRLF).  Gnus tries to read this file and
     does something strange when it sees the just-one-CRLF case.

Not exactly. You should be aware of the fact that all occourances (sp ?) of
"\r\n\r\nFrom " result from a non-initial mail in the mailbox, where the
preceding mail happens to contain a trailing empty line.
If you then interpret the empty trailing line (with the "\r\nFrom ") as a
separator, this may turn out OK. However, there's no reason why a mail should
contain a trailing empty line, and such a thing is certainly not required by
RFC 1939 (quoted below)

   Responses to certain commands are multi-line.  In these cases, which
   are clearly indicated below, after sending the first line of the
   response and a CRLF, any additional lines are sent, each terminated
   by a CRLF pair.  When all lines of the response have been sent, a
   final line is sent, consisting of a termination octet (decimal code
   046, ".") and a CRLF pair.

BTW the whole business about multiple mails with separators utterly fails
to produce recoverable results: Just consider PGP-signed mails that contain
an empty line, followed by a single "From " such as in

>From what I can see, this is a legal string iside a mail.

Now, you can say that your clever little mail-sorer will escape that to yield
a ">From " line, but then, the unpacking side will (correct me if I'm wrong)
never do a delete-one-greater-sign-from-a-string-matching-"^>>>*From ".
This is likely to produce checksum failures.
     
     I do not know whether a POP3 server should be required to tack on an
     extra CRLF to the end of each message, thereby ensuring the twice-CRLF
     case always, or whether movemail should tack on the extra CRLF, or
     whether Gnus should be taught to do without the extra CRLF.

     But somehow I think that the POP3 server is *not* the problem here.

Nope, it just is required to make sure there's a \r\n.\r\n sequence to signify
end of mail, where the CRLF pair before the dot belongs to the message.
See RFC 1939.

Bernard

Re: Gnus, movemail, POP3, trailing empty lines

[Kai Grossjohann]

>>>>> Per Abrahamsen writes:

  Per> If I read these correct, you have a seriously broken POP server.

See Bernard's mail: I was all wrong, of course.

As I see it now, the following is what happens:

The POP3 server spits out a message at a time.  The last line of some
messages is empty, or some others it is non-empty.  movemail concats
all messages.  This results in a file that contains several messages,
one after the other, and in some cases, a message begins with
"\r\n\r\nFrom " (twice CRLF) whereas in other cases, a message begins
with "\r\nFrom " (just one CRLF).  Gnus tries to read this file and
does something strange when it sees the just-one-CRLF case.

I do not know whether a POP3 server should be required to tack on an
extra CRLF to the end of each message, thereby ensuring the twice-CRLF
case always, or whether movemail should tack on the extra CRLF, or
whether Gnus should be taught to do without the extra CRLF.

But somehow I think that the POP3 server is *not* the problem here.

kai
-- 
My heart is down, my head is turning around;
I had to leave a little girl in Kingston town. -- Harry Belafonte

Re: Gnus, movemail, POP3, trailing empty lines

[Per Abrahamsen]

Kai Grossjohann <grossjohann@charly.informatik.uni-dortmund.de> writes:

> The POP3 server spits out mails separated by "\nFrom " but not
> "\n\nFrom ".  If the standard sez one "\n" is enough then Gnus should
> grok that, too, shouldn't it?

As far as I can see from RFC 1939 and movemail.c:

A pop server always spit out a single message a time, so it doesn't
have any use for unix mbox format.  In fact, it would be a violation
of the protocol if a messages began with `From '.  It may contain as
many "From " lines as wants, with or without a preceding empty line.
They are still part of a single message.

Movemail creates its output in BABYL format, and furthermore quotes
all "From " lines it sees just to be sure.

If I read these correct, you have a seriously broken POP server.

Re: Gnus, movemail, POP3, trailing empty lines

[Bernard.Steiner]

Hi,
     
       KG> Note that there is no empty line between the first "bla bla" and
       KG> the second mail.

That is completely beside the point, since the server sends all mails
seperately.
     
       Rich> Then the mail server is messing up, because there *must* be
       Rich> one.  The Unix mail separator is "\nFrom ...".  The newline
       Rich> must be there to distinguish one message from the next.
     
     The POP3 server spits out mails separated by "\nFrom " but not
     "\n\nFrom ".  If the standard sez one "\n" is enough then Gnus should
     grok that, too, shouldn't it?

The POP3 server spits out messages *exactly* like
+OK\r\nFrom ....\r\nRest of header\r\n\r\nbody with trailing\r\n.\r\n

If the client side is able to receive single pieces of mail and then munges
them up so as lateron not to be able to separate them again, this has nothing
whatsoever to do with the server.

Just for the record - there's no way in hell apart from a broken filesystem
that the server mistakes two mails as one.

Bernard

Re: Gnus, movemail, POP3, trailing empty lines

[Kai Grossjohann]

>>>>> "KG" == Kai Grossjohann
>>>>> <grossjohann@charly.informatik.uni-dortmund.de> writes:

  KG> Note that there is no empty line between the first "bla bla" and
  KG> the second mail.

>>>>> Rich Pieri writes:

  Rich> Then the mail server is messing up, because there *must* be
  Rich> one.  The Unix mail separator is "\nFrom ...".  The newline
  Rich> must be there to distinguish one message from the next.

The POP3 server spits out mails separated by "\nFrom " but not
"\n\nFrom ".  If the standard sez one "\n" is enough then Gnus should
grok that, too, shouldn't it?

Sadly, I haven't read the RFC (which one is it?) myself so I'm not
really sure whether the standard requires "\n\n" or not.

kai
-- 
My heart is down, my head is turning around;
I had to leave a little girl in Kingston town. -- Harry Belafonte

Re: Gnus, movemail, POP3, trailing empty lines

[dave edmondson]

surely discussions about delivery agent are secondary.  the first
thing i'd check is that the pop server is actually dishing the
messages out in distinct pieces.  if this is the case, then movemail
should be coerced into generating a correct mailbox file (ie. it
should be told to ensure that there is a \n\nFrom seperator).  if the
pop server _doesn't_ give out these two as seperate messages, then
perhaps it and the delivery agent on the pop serving machine should be
examined.

Re: Gnus, movemail, POP3, trailing empty lines

[Lars Magne Ingebrigtsen]

visigoth@naiad.fac.cs.cmu.edu writes:

> Actually, I read recently that "^From ..." is the separator.  Any
> reasonable mail agent ought to produce only "\nFrom ", but should
> accept any line beginning with "From " as beginning a new message,
> regardless of whether or not it was preceeded by a blank.

No; the mail delivery agent should always put a blank line before the
"From " line.  That's the de facto standard in this area.  If a
delivery agent doesn't do that, it's broken.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: Gnus, movemail, POP3, trailing empty lines

[visigoth]

Rich Pieri <rich.pieri@PrescientTech.com> writes:

> Then the mail server is messing up, because there *must* be one.  The
> Unix mail separator is "\nFrom ...".  The newline must be there to
> distinguish one message from the next.

Actually, I read recently that "^From ..." is the separator.  Any
reasonable mail agent ought to produce only "\nFrom ", but should
accept any line beginning with "From " as beginning a new message,
regardless of whether or not it was preceeded by a blank.

John.

Re: Gnus, movemail, POP3, trailing empty lines

[Rich Pieri]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "KG" == Kai Grossjohann
>>>>> <grossjohann@charly.informatik.uni-dortmund.de> writes:

KG> Note that there is no empty line between the first "bla bla" and the
KG> second mail.

Then the mail server is messing up, because there *must* be one.  The
Unix mail separator is "\nFrom ...".  The newline must be there to
distinguish one message from the next.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.3a
Charset: noconv

iQCVAwUBMvYUKZ6VRH7BJMxHAQFJWgP/fF52k59stPRb7m6vzQT5c2Nd8i7eHsWw
Vp3TSucjVWDhX5uHJB6p3puyfAxPLEE24Vh1xGT+zPxOkaolc0AxdjuXaxSUaFP0
teiTl12MdWn+wX3fF4FIG8vaaba84sU7spG6P3Mc0ITcOBxcL2whExu7iOEpekqm
E51ZXBmecbc=
=H+VD
-----END PGP SIGNATURE-----
-- 
Rich Pieri <rich.pieri@prescienttech.com> | Warning: pregnant women, the
Prescient Technologies, Inc.              | elderly, and children under 10
A Stone & Webster Company                 | should avoid prolonged exposure to
I speak for myself, not PTI or SWEC       | Happy Fun Ball.

Gnus, movemail, POP3, trailing empty lines

[Kai Grossjohann]

Hi all,

Michael uses movemail to read mail from a POP3 server.  If a mail
doesn't end with an empty line, the following mail isn't split
correctly.

Example:

,-----
| From foo@frob.org
| From: Foo <foo@frob.org>
| Subject: The Gnats and Gnus society
|                
| bla bla bla bla
| From pimpel@pampel.de
| From: Pimpel <pimpel@pampel.de>
| Subject: Pimpelnde Pampeln pampeln Pimpel
|                
| bla bla bla bla
`-----

Note that there is no empty line between the first "bla bla" and the
second mail.

These two mails are not split correctly, they're saved in a single
file in the nnml folder.

Has anyone seen this, too?
kai
-- 
My heart is down, my head is turning around;
I had to leave a little girl in Kingston town. -- Harry Belafonte