bad (i.e. serious) mail problems

bad (i.e. serious) mail problems

[Dmitry Yaitskov]

[-- Attachment #1: Type: text/plain, Size: 1156 bytes --]

Hi,

A few days ago I posted a message about a problem with new mail being
marked as read, and being unable to access a mail group. Here it goes
again... I have a mail group with the following group parametes:

((total-expire . t)
 (display . all)
 (comment . "personal mail")
 (visible . t)
 (charset . windows-1251))

Just now, 2 new messages arrived in this group. I saw it in the
*Group* buffer. When pressing Enter on that group, the new messages
counter went to 0, "No unread news" appeared in the minibuffer, and
the cursor jumped to the next group with some messages. According to
Lars:

>This usually means that the .overview file for the group is messed
>up...

Whatever. I certainly did not mess up my .overview files by hand... :)
If anybody's interested, attached is my .overview file for the group
currently screwed up (i.e. I cannot enter it at all). Last time
removing the "charset" parameter from the group pars fixed the
situation. I am sure I can fix it now, somehow. But IMHO the bug is
bad enough to justify some attention, just in case somebody (Lars?)
wants more info about the current state of my mailboxes...


-- 
Cheers,
-Dima.


[-- Attachment #2: screwed up overview --]
[-- Type: application/octet-stream, Size: 5910 bytes --]

4	Attention all Shaw@Home customers!	internet.inquiries@shaw.ca (Internet Inquiries)	Mon, 8 Feb 1999 19:23:56 -0700 (MST)	<199902090223.TAA01421@toc.shaw.home.com>		411	15	Xref: LUCY other:4	
9	Majordomo results: subscribe	Majordomo@cs.washington.edu	Mon, 8 Feb 1999 22:18:21 -0800	<199902090618.WAA22900@june.cs.washington.edu>		60	5	Xref: LUCY other:9	
11	Welcome to ntemacs-users	Majordomo@cs.washington.edu	Mon, 8 Feb 1999 22:18:26 -0800	<199902090618.WAA22901@june.cs.washington.edu>		1983	56	Xref: LUCY other:11	
15	Re: Package warnings when starting up 21.2.10	John Mignault <jbm@panix.com>	09 Feb 1999 12:04:17 -0500	<m37ltrse3y.fsf@jbm.dialup.access.net>	<m3aeyss0ok.fsf@jbm.dialup.access.net> <14015.15134.623000.932684@SLOTH> <uhfswmujw.fsf@home.com> <m3aeynspp3.fsf@jbm.dialup.access.net> <usocfllhf.fsf@home.com>	274	10	Xref: LUCY other:15	
16	Re: wingrid.hpp Q.	"Yuhan Ma" <yuhan@apexsc.com>	Tue, 9 Feb 1999 12:43:09 -0500	<000e01be5453$a8801e60$e149d2ce@nanjing.apexsc.com>		918	43	Xref: LUCY other:16	
17	Majordomo results: unsubscribe epson-inkjet dima@interlog.c	Majordomo@leben.com	Tue, 9 Feb 1999 17:33:26 -0600	<199902092333.RAA16542@spock.leben.com>		729	27	Xref: LUCY other:17	
20	Majordomo results: subscribe	Majordomo@hpc.uh.edu	Thu, 11 Feb 1999 00:33:47 -0600 (CST)	<199902110633.AAA27986@sina.hpc.uh.edu>		1069	31	Xref: LUCY Mail.dima:20	
21	Confirmation for subscribe ding	Majordomo@hpc.uh.edu	Thu, 11 Feb 1999 00:33:48 -0600 (CST)	<199902110633.AAA27987@sina.hpc.uh.edu>		777	26	Xref: LUCY Mail.dima:21	
22	Majordomo results: Re: Confirmation for subscribe ding	Majordomo@hpc.uh.edu	Thu, 11 Feb 1999 09:16:21 -0600 (CST)	<199902111516.JAA02389@sina.hpc.uh.edu>		89	6	Xref: LUCY Mail.dima:22	
23	Welcome to ding	Majordomo@hpc.uh.edu	Thu, 11 Feb 1999 09:16:22 -0600 (CST)	<199902111516.JAA02390@sina.hpc.uh.edu>		617	19	Xref: LUCY Mail.dima:23	
24	Changing C(++) style and tab size *on-the-fly*?	Tom Lyons <tbl@atria.com>	Thu, 11 Feb 1999 10:40:48 -0500 (EST)	<199902111540.KAA29682@gw.atria.com>		1026	27	Xref: LUCY Mail.dima:24	
26	Re: Got rid of m_pItems	"Michael Eisenstein" <eisenm@home.com>	Thu, 11 Feb 1999 11:28:47 -0500	<001201be55db$9e2869c0$685d4118@misha.apexsc.com>		703	29	Xref: LUCY Mail.dima:26	
27	Killed DEBUG_NEW	"Michael Eisenstein" <eisenm@home.com>	Thu, 11 Feb 1999 13:03:39 -0500	<005801be55e8$e139fc80$685d4118@misha.apexsc.com>		70	7	Xref: LUCY Mail.dima:27	
28	Re: Temp files left in ~/Mail?	Michael Cook <cook@sightpath.com>	11 Feb 1999 14:11:44 -0500	<80emnwwya7.fsf@tma-1.sightpath.com>	<u4sosbwc8.fsf@home.com>	451	12	Xref: LUCY List.ding:10 Mail.dima:28	
29	Re: Temp files left in ~/Mail?	Michael Cook <cook@sightpath.com>	11 Feb 1999 14:11:44 -0500	<80emnwwya7.fsf@tma-1.sightpath.com>	<u4sosbwc8.fsf@home.com>	452	13	Xref: LUCY List.ding:11 Mail.dima:29	
30	(none)	EmmA@dragon.mephi.ru (Emma Atamalian)	Fri, 12 Feb 1999 11:14:52 +0300 (MSK)	<ABiMyXwO9J@dragon.mephi.ru>	<00f601be3018$f06bd660$8b5d4118@lucy.apexsc.com>	260	11	Xref: LUCY Mail.dima:30	
31	test again	Dmitry Yaitskov <dimas@home.com>	12 Feb 1999 14:28:40 -0500	<uvhh78lqv.fsf@home.com>		27	7	Xref: LUCY Mail.dima:31	
32	Re: Background mail scannig?	Karel Sprenger <cjas@xs4all.nl>	13 Feb 1999 14:57:17 +0100	<uvhh6ieyq.fsf@xs4all.nl>	<upv7f8ky5.fsf@home.com>	1355	38	Xref: LUCY Mail.dima:32 List.ding:47	
37	New questions	"stas" <skomarovsky@geocities.com>	Sun, 14 Feb 1999 09:44:13 +0200	<000e01be57ed$d3124c50$680a010a@sts_s.newx.co.il>		7755	158	Xref: LUCY Mail.dima:37	
40	Re: building xemacs-21.0.63 under Cygwin B20.1	Andy Piper <andy@xemacs.org>	Sun, 14 Feb 1999 15:06:18 +0000	<3.0.5.32.19990214150618.009333d0@london.beasys.com>	<"David Starks-Browning"'s message of "Sat, 13 Feb 1999 20:01:32 +0000 (GMT Standard Time)"> <14021.54254.918000.958838@gargle.gargle.HOWL> <14021.55836.132000.750995@gargle.gargle.HOWL>	492	15	Xref: LUCY Mail.dima:40 List.xemacs-nt:29	
43	Re: Advice on K2 threes please	"S. Deem" <sdeem@u.washington.edu>	Wed, 17 Feb 1999 11:40:03 -0800 (PST)	<Pine.A41.4.05.9902171137380.130718-100000@homer24.u.washington.edu>	<ur9ro3lmu.fsf@home.com>	717	18	Xref: LUCY Mail.dima:43	
44	Re: Unable to access your site	Rob Henderson <robh@cs.indiana.edu>	Wed, 17 Feb 1999 18:08:12 -0500 (EST)	<199902172308.SAA25992@megamouth.cs.indiana.edu>	<7aceim$130$1@jetsam.uits.indiana.edu>	1947	48	Xref: LUCY Mail.dima:44	
45	RE: Ski, Bike, Canoe & Kayak Swap in Toronto	Bohdan Valentovic <bvalentovic@InSystems.com>	Thu, 18 Feb 1999 08:06:45 -0500	<4160E6FC08ABD21191F000805F857E9301CFC0@mail.insystems.ca>		658	29	Xref: LUCY Mail.dima:45	
46	RE: Tax form for 1998	Cathy Serre <cserre@InSystems.com>	Thu, 18 Feb 1999 09:18:14 -0500	<4160E6FC08ABD21191F000805F857E93019F71@mail.insystems.ca>		696	32	Xref: LUCY Mail.dima:46	
47	RE: Ski, Bike, Canoe & Kayak Swap in Toronto	Bohdan Valentovic <bvalentovic@InSystems.com>	Thu, 18 Feb 1999 10:02:09 -0500	<4160E6FC08ABD21191F000805F857E9301CFC2@mail.insystems.ca>		1434	53	Xref: LUCY Mail.dima:47	
48	Re: Advice on K2 threes please	"S. Deem" <sdeem@u.washington.edu>	Thu, 18 Feb 1999 10:59:52 -0800 (PST)	<Pine.A41.4.05.9902181058550.145818-100000@homer32.u.washington.edu>	<ud8378v0c.fsf@home.com>	491	15	Xref: LUCY Mail.dima:48	
49	True DBGrid Print	"York, Devin" <dyork@coair.com>	Thu, 18 Feb 1999 15:47:29 -0600	<199902182147.NAA11842@mx2-w.mail.home.com>		366	12	Xref: LUCY Mail.dima:49	
50	RE: True DBGrid Print	"York, Devin" <dyork@coair.com>	Thu, 18 Feb 1999 17:30:29 -0600	<199902182331.PAA03385@mx1-e.mail.home.com>		1345	38	Xref: LUCY Mail.dima:50	
51	RE: Tax form for 1998	Cathy Serre <cserre@InSystems.com>	Fri, 19 Feb 1999 07:49:18 -0500	<4160E6FC08ABD21191F000805F857E93019F7E@mail.insystems.ca>		393	19	Xref: LUCY Mail.dima:51	
53	RE: change of address	Address <address@airmiles.ca>	Tue, 23 Feb 1999 18:12:09 -0500	<D088E30F8154D211B1AB00A0C9D5E22C7AD3C8@exchtor1.loyalty.com>		692	37	Xref: LUCY Mail.dima:53	

Re: bad (i.e. serious) mail problems

[Lars Magne Ingebrigtsen]

Dmitry Yaitskov <dimas@home.com> writes:

> >This usually means that the .overview file for the group is messed
> >up...
> 
> Whatever. I certainly did not mess up my .overview files by hand... :)

Does `M-x nnml-generate-nov-databases RET' do anything interesting?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Lars Magne Ingebrigtsen <larsi@gnus.org> spake thusly:

> Dmitry Yaitskov <dimas@home.com> writes:
> 
> > >This usually means that the .overview file for the group is messed
> > >up...
> > 
> > Whatever. I certainly did not mess up my .overview files by hand... :)
> 
> Does `M-x nnml-generate-nov-databases RET' do anything interesting?

Nope. An attempt to enter that group still fails with a "no unread
news" message.

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Neil Crellin]

Dmitry Yaitskov <dimas@home.com> writes:
> Lars Magne Ingebrigtsen <larsi@gnus.org> spake thusly:
> > Dmitry Yaitskov <dimas@home.com> writes:
> > 
> > > >This usually means that the .overview file for the group is messed
> > > >up...
> > > 
> > > Whatever. I certainly did not mess up my .overview files by hand... :)
> > 
> > Does `M-x nnml-generate-nov-databases RET' do anything interesting?
> 
> Nope. An attempt to enter that group still fails with a "no unread
> news" message.

Even trying to enter with C-u RET?

-- 
Neil Crellin <neilc@wallaby.cc>

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Neil Crellin <neilc@Tesserae.COM> spake thusly:

> Dmitry Yaitskov <dimas@home.com> writes:
> > Lars Magne Ingebrigtsen <larsi@gnus.org> spake thusly:
> > > Dmitry Yaitskov <dimas@home.com> writes:
> > > 
> > > > >This usually means that the .overview file for the group is messed
> > > > >up...
> > > > 
> > > > Whatever. I certainly did not mess up my .overview files by hand... :)
> > > 
> > > Does `M-x nnml-generate-nov-databases RET' do anything interesting?
> > 
> > Nope. An attempt to enter that group still fails with a "no unread
> > news" message.
> 
> Even trying to enter with C-u RET?

Tried it now. Without any success.

The "fix" was, as it happened once already, in removing the default
charset from the group parameters. Can't imagine how it can be
related...

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Lars Magne Ingebrigtsen]

Dmitry Yaitskov <dimas@home.com> writes:

> Tried it now. Without any success.

Does the line for the group in the active file look ok?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Lars Magne Ingebrigtsen <larsi@gnus.org> spake thusly:

> Dmitry Yaitskov <dimas@home.com> writes:
> 
> > Tried it now. Without any success.
> 
> Does the line for the group in the active file look ok?

Mail.dima 56 4 y

Which is about right as far as I can see. BUt that's how it looks now,
when I "fixed" the prob. Ok, next time it happens (if it does), I will
check and post it.

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Lars Magne Ingebrigtsen <larsi@gnus.org> spake thusly:

> Dmitry Yaitskov <dimas@home.com> writes:
> 
> > Tried it now. Without any success.
> 
> Does the line for the group in the active file look ok?

Well, strange as it may seem, the line that does it is:

 (charset . windows-1251)

in the group's parameters. After I add this line, when mail arrives in 
the group, pressing Enter on the group resets the new messages counter 
to 0 in the *Group* buffer, and I cannot enter the group at all
despite the display all. Removing the charset line returns the
situation to normal, although the new mail remains marked as read of
course. I am willing to try some other experiments if need be. The
group parameters are:

((total-expire . t)
 (display . all)
 (comment . "personal mail")
 (visible . t)
 (charset . windows-1251))

The line for the broken group in the active file does not change, it is:

Mail.dima 57 4 y

and looks ok to me. The .overview also does not change, and is ok
(because after I took away the charset line, I can read all the
messages). Weird...

All this on XEmacs 21.2 under NT, Gnus v0.79.

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Lars Magne Ingebrigtsen]

Dmitry Yaitskov <dimas@home.com> writes:

> Well, strange as it may seem, the line that does it is:
> 
>  (charset . windows-1251)

Well, that's certainly odd.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bad (i.e. serious) mail problems

[Shane Holder]

I just had this same experience, but under nnfolder.  I had updated my
group buffer by typing `G' and Gnus showed that I had 1 piece of mail
in my mail group, but when I entered that group `<space>' the
num. articles in the group buffer just went to 0.  I then tried
C-<space> and the summary didn't show any new articles, but when I
edited the file (nnfolder backend) an e-mail exists that doesn't show
up in the summary buffer when Gnus is used.

My active file shows what I think should be the proper number of
articles.

perl 37 1 y

and here are the headers for the article.

> From nobody Mon Mar 01 09:36:03 1999
> > From mike.smith@ActiveState.com  Fri Feb 26 17:32:5 1999
> Received: from mailhost.rsn.hp.com (root@idiot.rsn.hp.com [15.99.200.11]) by dmail2.rsn.hp.com with ESMTP (8.7.1/8.7.1) id UAA25276 for <holder@dmail2.rsn.hp.com>; Fri, 26 Feb 1999 20:04:17 -0600 (CST)
> Received: from palrel3.hp.com (palrel3.hp.com [15.81.184.10])
>         by mailhost.rsn.hp.com (8.8.6 (PHNE_14041)/8.8.6) with ESMTP id UAA23367
>         for <holder@rsn.hp.com>; Fri, 26 Feb 1999 20:04:11 -0600 (CST)
> Received: from growl.pobox.com (growl.pobox.com [208.210.124.27])
>         by palrel3.hp.com (8.8.6 (PHNE_14041)/8.8.5tis) with ESMTP id SAA07776
>         for <holder@rsn.hp.com>; Fri, 26 Feb 1999 18:04:28 -0800 (PST)
> Received: from unknown (unknown [199.60.48.81])
>         by growl.pobox.com (Postfix) with SMTP for <shane.holder@pobox.com>
>         id E4BC35E04; Fri, 26 Feb 1999 21:02:13 -0500 (EST)
> Date: Fri, 26 Feb 1999 17:32:5
> Subject: ANNOUNCE: Perl Development Kit 1.1 Release Candidate 2
> To: "Perl-Win32-Announce Mailing List" <perl-win32-announce@listserv.activestate.com>
> From: "Michael Smith" <mike.smith@ActiveState.com>
> List-Unsubscribe: <mailto:leave-perl-win32-announce-22324N@lyris.activestate.com>
> List-Software: Lyris Server version 3.0
> List-Subscribe: <mailto:subscribe-perl-win32-announce@lyris.activestate.com>
> List-Owner: <mailto:owner-perl-win32-announce@lyris.activestate.com>
> X-List-Host: ActiveState Tool Corp. <http://www.activestate.com>
> Reply-To: "Michael Smith" <mike.smith@ActiveState.com>
> Sender: bounce-perl-win32-announce-22324@listserv.activestate.com
> X-Gnus-Mail-Source: pop:holder@dmail2
> Message-Id: <LYR22324-71011-1999.02.26-17.35.17--shane.holder#pobox.com@lyris.activestate.com>
> Precedence: bulk
> X-listname: perl-win32-announce 
> X-ListMember: [shane.holder@pobox.com]
> Lines: 48
> Xref: PANIC perl:37
> X-Gnus-Article-Number: 37   Mon Mar 01 09:36:03 1999

No other articles in this nnfolder share the same article number, so
it's not the same problem I've reported under the "pop wierdness"
thread.


Shane

Re: bad (i.e. serious) mail problems

[Lars Magne Ingebrigtsen]

Shane Holder <holder@rsn.hp.com> writes:

> I just had this same experience, but under nnfolder.  I had updated my
> group buffer by typing `G' and Gnus showed that I had 1 piece of mail
> in my mail group, but when I entered that group `<space>' the
> num. articles in the group buffer just went to 0.  

This sounds like the nnfolder server is looking in the wrong place
for the folder.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bad (i.e. serious) mail problems

[Shane Holder]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

  Lars> Shane Holder <holder@rsn.hp.com> writes:
  >> I just had this same experience, but under nnfolder.  I had
  >> updated my group buffer by typing `G' and Gnus showed that I had
  >> 1 piece of mail in my mail group, but when I entered that group
  >> `<space>' the num. articles in the group buffer just went to 0.

  Lars> This sounds like the nnfolder server is looking in the wrong
  Lars> place for the folder.

I don't think so because if I do C-u <space> I get articles, but still
missing the new article.  And I just got another article in that
group, and it shows up.  So, I've got articles before and after an
article that doesn't show up in the summary buffer.

Shane

Re: bad (i.e. serious) mail problems

[Lars Magne Ingebrigtsen]

Shane Holder <holder@rsn.hp.com> writes:

> I don't think so because if I do C-u <space> I get articles, but still
> missing the new article.  And I just got another article in that
> group, and it shows up.  So, I've got articles before and after an
> article that doesn't show up in the summary buffer.

Hm.  Is there anything odd about the headers of the article that
doesn't show up?  Is it a duplicate, for instance?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bad (i.e. serious) mail problems

[Shane Holder]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

  Lars> Shane Holder <holder@rsn.hp.com> writes:

  >> I don't think so because if I do C-u <space> I get articles, but still
  >> missing the new article.  And I just got another article in that
  >> group, and it shows up.  So, I've got articles before and after an
  >> article that doesn't show up in the summary buffer.

  Lars> Hm.  Is there anything odd about the headers of the article
  Lars> that doesn't show up?  Is it a duplicate, for instance?

Not a duplicate, but a `>' got inserted in front of one of the From
headers.  When I deleted this character the article showed up in the
summary.  When I looked at the Incoming... file it did not contain
this character.  Now, I suppose you're thinking well when I edited the
file somehow this got inserted, which is possible, but I didn't edit
the nnfolder file until I realized that something strange was going
on.

Shane

Re: bad (i.e. serious) mail problems

[Shane Holder]

>>>>> "Shane" == Shane Holder <holder@rsn.hp.com> writes:

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
  Lars> Shane Holder <holder@rsn.hp.com> writes:

  >>> I don't think so because if I do C-u <space> I get articles, but still
  >>> missing the new article.  And I just got another article in that
  >>> group, and it shows up.  So, I've got articles before and after an
  >>> article that doesn't show up in the summary buffer.

  Lars> Hm.  Is there anything odd about the headers of the article
  Lars> that doesn't show up?  Is it a duplicate, for instance?

  Shane> Not a duplicate, but a `>' got inserted in front of one of
  Shane> the From headers.  When I deleted this character the article
  Shane> showed up in the summary.  When I looked at the
  Shane> Incoming... file it did not contain this character.  Now, I
  Shane> suppose you're thinking well when I edited the file somehow
  Shane> this got inserted, which is possible, but I didn't edit the
  Shane> nnfolder file until I realized that something strange was
  Shane> going on.

I just had this happen again, still under .79.  I searched through my
nnfolder mail.misc file and noticed that there were about 3 other
articles that had a > in front of the From header.

Shane

Re: bad (i.e. serious) mail problems

[Harry Putnam]

Shane Holder <holder@rsn.hp.com> writes:

> >>>>> "Shane" == Shane Holder <holder@rsn.hp.com> writes:
> 
> >>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>   Lars> Shane Holder <holder@rsn.hp.com> writes:
> 
>   >>> I don't think so because if I do C-u <space> I get articles, but still
>   >>> missing the new article.  And I just got another article in that
>   >>> group, and it shows up.  So, I've got articles before and after an
>   >>> article that doesn't show up in the summary buffer.
> 
>   Lars> Hm.  Is there anything odd about the headers of the article
>   Lars> that doesn't show up?  Is it a duplicate, for instance?
> 
>   Shane> Not a duplicate, but a `>' got inserted in front of one of
>   Shane> the From headers.  When I deleted this character the article
>   Shane> showed up in the summary.  When I looked at the
>   Shane> Incoming... file it did not contain this character.  Now, I
>   Shane> suppose you're thinking well when I edited the file somehow
>   Shane> this got inserted, which is possible, but I didn't edit the
>   Shane> nnfolder file until I realized that something strange was
>   Shane> going on.
> 
> I just had this happen again, still under .79.  I searched through my
> nnfolder mail.misc file and noticed that there were about 3 other
> articles that had a > in front of the From header.

Hope this isn't off the wall but, could this be done by what ever
application is retrieving your mail?  (assuming Gnus is not) Mail
handling apps put a (>) in front of all lines beginning with "From" to
avoid confusing them with the Unix format line beginning with "From"
but no colon.

-- 
Harry Putnam <reader@newsguy.com>
Running Redhat Linux 5.2

Re: bad (i.e. serious) mail problems

[Shane Holder]

>>>>> "Harry" == Harry Putnam <reader@newsguy.com> writes:

  Shane> Not a duplicate, but a `>' got inserted in front of one of
  Shane> the From headers.  When I deleted this character the article
  Shane> showed up in the summary.  When I looked at the
  Shane> Incoming... file it did not contain this character.  Now, I
  Shane> suppose you're thinking well when I edited the file somehow
  Shane> this got inserted, which is possible, but I didn't edit the
  Shane> nnfolder file until I realized that something strange was
  Shane> going on.
   
  >> I just had this happen again, still under .79.  I searched through my
  >> nnfolder mail.misc file and noticed that there were about 3 other
  >> articles that had a > in front of the From header.

  Harry> Hope this isn't off the wall but, could this be done by what
  Harry> ever application is retrieving your mail?  (assuming Gnus is
  Harry> not) Mail handling apps put a (>) in front of all lines
  Harry> beginning with "From" to avoid confusing them with the Unix
  Harry> format line beginning with "From" but no colon.

Gnus is doing the retrival.  And the > insertion is coming between the
creation of the Incoming... file, and when the article gets inserted
into nnfolder.

Shane

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Shane Holder <holder@rsn.hp.com> writes:

  > Gnus is doing the retrival.  And the > insertion is coming between the
  > creation of the Incoming... file, and when the article gets inserted
  > into nnfolder.

A message should begin with "From " at the beginning of a line, and
end with an empty line.

Maybe the > are inserted in messages where there is no empty line
before "From "?

kai
-- 
I like _\bb_\bo_\bt_\bh kinds of music.

Re: bad (i.e. serious) mail problems

[Shane Holder]

OK, this just happened to me again.  But this time I've got some more
info.  I'm not sure about the number of messages being different, but
I've got 2 messages that have been merged into one, and the 2nd
message has a > in front of the From header.  All of the headers have
been retrieved within Gnus without editing the nnfolder file
containing the messages.

Below are the headers for both of the messages.  The second message
appeared in the text of the first.  There's something seriously funky
going on.  I would consider this a show stopper for a  product I was
shipping.  Consider a message from an advertiser and after that
message an urgent message from your boss, you delete the message from
the advertiser without reading it and consequently delete the message
from your boss and get fired. :)

Shane

X-From-Line: <deleted>  Fri Mar 5 18:53:50 1999
Date: Fri, 5 Mar 1999 18:53:50 -0600 (CST)
From: <deleted>
X-Gnus-Mail-Source: pop:holder@foo
Message-Id: <199903060053.SAA05634@foo.com>
To: foo@bar.com
Subject: goobers
Mime-Version: 1.0
Content-Type: text/plain; charset=X-roman8
Content-Transfer-Encoding: 7bit
Lines: 7
Xref: PANIC mail.misc:8842
X-Gnus-Article-Number: 8842   Mon Mar 08 09:47:36 1999

Text deleted

>From nobody Mon Mar 08 09:47:37 1999
> From BloomTimes@garden.m0.net  Fri 5  1999 Mar
Received: from mailhost.rsn.hp.com (root@idiot.rsn.hp.com [15.99.200.11]) by dmail2.rsn.hp.com with ESMTP (8.7.1/8.7.1) id TAA10491 for <holder@dmail2.rsn.hp.com>; Fri, 5 Mar 1999 19:31:41 -0600 (CST)
Received: from palrel3.hp.com (palrel3.hp.com [15.81.184.10])
	by mailhost.rsn.hp.com (8.8.6 (PHNE_14041)/8.8.6) with ESMTP id TAA26195
	for <holder@rsn.hp.com>; Fri, 5 Mar 1999 19:31:34 -0600 (CST)
Received: from growl.pobox.com (growl.pobox.com [208.210.124.27])
	by palrel3.hp.com (8.8.6 (PHNE_14041)/8.8.5tis) with ESMTP id RAA02259
	for <holder@rsn.hp.com>; Fri, 5 Mar 1999 17:31:47 -0800 (PST)
Received: from app27.merchantmail.net (app27.merchantmail.net [216.32.126.218])
	by growl.pobox.com (Postfix) with SMTP for <shane.holder@pobox.com>
	id 96F366566; Fri,  5 Mar 1999 20:29:09 -0500 (EST)
From: Bloom Times <BloomTimes@garden.m0.net>
Reply-To: BloomTimes@garden.m0.net
To: shane.holder@pobox.com
Subject: Garden.com's March Bloom Times
Errors-To: garden1@app1.merchantmail.net
Mime-version: 1.0
Content-type: multipart/alternative; boundary="139600885.920683300687.root@app31.merchantmail.net"
X-cid: 51867819
X-Gnus-Mail-Source: pop:holder@dmail2
Message-Id: <19990306012924.96F366566@growl.pobox.com>
Date: Fri,  5 Mar 1999 20:29:24 -0500 (EST)
Sender: garden1@app1.merchantmail.net
Lines: 279
Xref: PANIC mail.misc:8843
X-Gnus-Article-Number: 8843   Mon Mar 08 09:47:37 1999

Re: bad (i.e. serious) mail problems

[Shane Holder]

Just noticed an error in the headers.  There isn't a > on the From
line for the nobody address.  Corrected headers below.

X-From-Line: <deleted>  Fri Mar 5 18:53:50 1999
Date: Fri, 5 Mar 1999 18:53:50 -0600 (CST)
From: <deleted>
X-Gnus-Mail-Source: pop:holder@foo
Message-Id: <199903060053.SAA05634@foo.com>
To: foo@bar.com
Subject: goobers
Mime-Version: 1.0
Content-Type: text/plain; charset=X-roman8
Content-Transfer-Encoding: 7bit
Lines: 7
Xref: PANIC mail.misc:8842
X-Gnus-Article-Number: 8842   Mon Mar 08 09:47:36 1999

Text deleted

>From nobody Mon Mar 08 09:47:37 1999
> From BloomTimes@garden.m0.net  Fri 5  1999 Mar
Received: from mailhost.rsn.hp.com (root@idiot.rsn.hp.com [15.99.200.11]) by dmail2.rsn.hp.com with ESMTP (8.7.1/8.7.1) id TAA10491 for <holder@dmail2.rsn.hp.com>; Fri, 5 Mar 1999 19:31:41 -0600 (CST)
Received: from palrel3.hp.com (palrel3.hp.com [15.81.184.10])
	by mailhost.rsn.hp.com (8.8.6 (PHNE_14041)/8.8.6) with ESMTP id TAA26195
	for <holder@rsn.hp.com>; Fri, 5 Mar 1999 19:31:34 -0600 (CST)
Received: from growl.pobox.com (growl.pobox.com [208.210.124.27])
	by palrel3.hp.com (8.8.6 (PHNE_14041)/8.8.5tis) with ESMTP id RAA02259
	for <holder@rsn.hp.com>; Fri, 5 Mar 1999 17:31:47 -0800 (PST)
Received: from app27.merchantmail.net (app27.merchantmail.net [216.32.126.218])
	by growl.pobox.com (Postfix) with SMTP for <shane.holder@pobox.com>
	id 96F366566; Fri,  5 Mar 1999 20:29:09 -0500 (EST)
From: Bloom Times <BloomTimes@garden.m0.net>
Reply-To: BloomTimes@garden.m0.net
To: shane.holder@pobox.com
Subject: Garden.com's March Bloom Times
Errors-To: garden1@app1.merchantmail.net
Mime-version: 1.0
Content-type: multipart/alternative; boundary="139600885.920683300687.root@app31.merchantmail.net"
X-cid: 51867819
X-Gnus-Mail-Source: pop:holder@dmail2
Message-Id: <19990306012924.96F366566@growl.pobox.com>
Date: Fri,  5 Mar 1999 20:29:24 -0500 (EST)
Sender: garden1@app1.merchantmail.net
Lines: 279
Xref: PANIC mail.misc:8843
X-Gnus-Article-Number: 8843   Mon Mar 08 09:47:37 1999

Re: bad (i.e. serious) mail problems

[Shane Holder]

>>>>> "Shane" == Shane Holder <holder@rsn.hp.com> writes:

>>>>> "Shane" == Shane Holder <holder@rsn.hp.com> writes:
>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
  Lars> Shane Holder <holder@rsn.hp.com> writes:

  >>>> I don't think so because if I do C-u <space> I get articles, but still
  >>>> missing the new article.  And I just got another article in that
  >>>> group, and it shows up.  So, I've got articles before and after an
  >>>> article that doesn't show up in the summary buffer.

  Lars> Hm.  Is there anything odd about the headers of the article
  Lars> that doesn't show up?  Is it a duplicate, for instance?

  Shane> Not a duplicate, but a `>' got inserted in front of one of
  Shane> the From headers.  When I deleted this character the article
  Shane> showed up in the summary.  When I looked at the
  Shane> Incoming... file it did not contain this character.  Now, I
  Shane> suppose you're thinking well when I edited the file somehow
  Shane> this got inserted, which is possible, but I didn't edit the
  Shane> nnfolder file until I realized that something strange was
  Shane> going on.

  Shane> I just had this happen again, still under .79.  I searched
  Shane> through my nnfolder mail.misc file and noticed that there
  Shane> were about 3 other articles that had a > in front of the From
  Shane> header.

This is still happening under .80.  I'm "loosing" about 3 articles a
day.  I would like to debug the problem.  If someone can tell me how
to get gnus to re-read an Incoming file I might be able to make some
headway.

Shane

Re: bad (i.e. serious) mail problems (POSSIBLE culprit found)

[Shane Holder]

>>>>> "Shane" == Shane Holder <holder@rsn.hp.com> writes:

  Shane> through my nnfolder mail.misc file and noticed that there
  Shane> were about 3 other articles that had a > in front of the From
  Shane> header.

  Shane> This is still happening under .80.  I'm "loosing" about 3
  Shane> articles a day.  I would like to debug the problem.  If
  Shane> someone can tell me how to get gnus to re-read an Incoming
  Shane> file I might be able to make some headway.

While trying to figure this problem out, I noticed that an additional
header was getting inserted in my mail messages headers.  A "From
nobody <date>" header was getting pre-pended to the messages.  I
searched for nobody and found a few instances where From nobody gets
inserted into the message headers, but I think there is one in
particular that is causing the problem.

nnfolder-save-mail inserts this header, but it also searches through
the message and inserts a > on any "From " line, which kills the
original "From " header so I end up with:

>From nobody "Tue Mar 09 14:39:02 1999"
> From true-sender@foo.bar.com "Tue Mar 09 14:39:02 1999"
To: holder@rsn.hp.com
Subject:
etc:

This functionality doesn't exist in the nnml-save-mail but does exist
in nnmbox-save-mail and gnus-output-to-mail.  So, I don't know which
one is the culprit.  I've changed the nobody to nobody-<unique text>
so I can identify who inserts the From nobody, but I would appreciate
some direction.

Thanks,
Shane

Re: bad (i.e. serious) mail problems

[Harry Putnam]

Shane Holder <holder@rsn.hp.com> writes:

> This is still happening under .80.  I'm "loosing" about 3 articles a
> day.  I would like to debug the problem.  If someone can tell me how
> to get gnus to re-read an Incoming file I might be able to make some
> headway.

There a couple of ways to recycle the Incoming* files:
1) Do `G f' in Group buffer and give the address of Incoming file 
   (Mail/Incoming*what-ever)
2) Do `cat Mail/Incoming* /var/spool/mail/<user-name> To zap them all
   back onto the mail spool.  (assumes root priviledges)

The nicest way: 

3) Do `G m' in group buffer give "Mail" as the address and "nneething"
   as the method.  When you enter the "nneething" group you'll see the
   directory structure in "Mail" the Incoming files will be shown as
   regular messages (i.e. as Subject: lines) You can run them through
   the split routine with `B q' to see where they'd go or `B r' to
   send them there. (choose your regular secondary select backend as
   the method to respool with)  The origs will stay in ~/Mail

Learn more about "nneething" by doing `C-c C-i' from group buffer then
press `s' for regexp search and enter "nneething" (no quotes) as the string.

-- 
Harry Putnam reader@newsguy.com  
Running Redhat Linux-5.1
See http://www.jtan.com/~reader  for a brief pictorial 
saga of construction work in the trade of "Boilermaker"

Re: bad (i.e. serious) mail problems

[Shane Holder]

>>>>> "Harry" == Harry Putnam <reader@newsguy.com> writes:

  Harry> Shane Holder <holder@rsn.hp.com> writes:
  >> This is still happening under .80.  I'm "loosing" about 3 articles a
  >> day.  I would like to debug the problem.  If someone can tell me how
  >> to get gnus to re-read an Incoming file I might be able to make some
  >> headway.

  Harry> There a couple of ways to recycle the Incoming* files:
  Harry> 1) Do `G f' in Group buffer and give the address of Incoming file 
  Harry>    (Mail/Incoming*what-ever)

This produces something interesting.  I have an Incoming file that has
4 articles in it, and the summary for the group that gets created only
shows 3 articles.  :(

  Harry> 2) Do `cat Mail/Incoming* /var/spool/mail/<user-name> To zap them all
  Harry>    back onto the mail spool.  (assumes root priviledges)

Running NT so no can do, pop access only. :(

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Shane Holder <holder@rsn.hp.com> writes:

  > Harry> 2) Do `cat Mail/Incoming* /var/spool/mail/<user-name> To
  >   Harry>    zap them all back onto the mail spool.  (assumes root
  >   Harry>    priviledges)
  > 
  > Running NT so no can do, pop access only. :(

After `G f', use `B r' to respool the messages.

kai
-- 
I like _\bb_\bo_\bt_\bh kinds of music.

Re: bad (i.e. serious) mail problems

[Shane Holder]

>>>>> "Kai" == Kai Grossjohann <Kai.Grossjohann@CS.Uni-Dortmund.DE> writes:

  Kai> Shane Holder <holder@rsn.hp.com> writes:
  Harry> 2) Do `cat Mail/Incoming* /var/spool/mail/<user-name> To
  Harry> zap them all back onto the mail spool.  (assumes root
  Harry> priviledges)
  >> 
  >> Running NT so no can do, pop access only. :(

  Kai> After `G f', use `B r' to respool the messages.

Doesn't work because the funky article shows up as text in the article
sequentially previous to it, so the funky article doesn't go through
the header munging stuff as an article, but as the text of the
previous article.

Shane

Re: bad (i.e. serious) mail problems (POSSIBLE culprit found)

[Shane Holder]

Ok, it finally happened with the debug code in place.
Nnfolder-save-mail is where the From header is getting munged.  I
understand the code, but not the purpose.  Someone with understanding
of why it was put together this way needs to check into this.

BTW, I never figured out how to respool the incomming files.

Shane

Re: bad (i.e. serious) mail problems

[Simon Michael]

Harry Putnam <reader@newsguy.com> writes:
> Hope this isn't off the wall but, could this be done by what ever
> application is retrieving your mail?  (assuming Gnus is not) Mail

FWIW, I have seen fetchmail doing this, the version that comes with
redhat 5.1. Both times I upgraded it and the problem seemed to go
away.

> shipping.  Consider a message from an advertiser and after that
> message an urgent message from your boss, you delete the message from
> the advertiser without reading it and consequently delete the message
> from your boss and get fired. :)

Or as happened to a friend who's just switched to gnus: 
he replied to a work message, citing the original, which unfortunately
had a porn advertisement merged with it due to a >From header.

Could gnus prevent or warn about such inadvertently merged messages ?
It happens more often than one might expect. Maybe it would be
practical only in this special case.
.

Re: bad (i.e. serious) mail problems

[Shane Holder]

>>>>> "Simon" == Simon Michael <simon@joyful.com> writes:

  Simon> Harry Putnam <reader@newsguy.com> writes:

  >> Hope this isn't off the wall but, could this be done by what ever
  >> application is retrieving your mail?  (assuming Gnus is not) Mail

  Simon> FWIW, I have seen fetchmail doing this, the version that
  Simon> comes with redhat 5.1. Both times I upgraded it and the
  Simon> problem seemed to go away.

After looking around a bit, I think I found the problem.

Here are 2 sample headers from e-mails that get screwed up.

   From bmgoffer-owner@lists.mpath.com  Fri 05 Mar PST 14:55:22

   From blh@hpuerca.atl.hp.com  Wed Mar 17 8:57:24 1999

and one that works

   From bart@grafikon.com  Mon Mar 8 09:00:14 1999

Note that the date formats are different

It's trying to match against message-unix-mail-delimiter and it
doesn't seem to be comprehensive enough.  Now it could be that they
are going against RFC 822 but we can't control what people send.  Now
I still want to know why nnml doesn't do this.


Shane

Re: bad (i.e. serious) mail problems (POSSIBLE culprit found)

[Lars Magne Ingebrigtsen]

Shane Holder <holder@rsn.hp.com> writes:

> Ok, it finally happened with the debug code in place.
> Nnfolder-save-mail is where the From header is getting munged.  I
> understand the code, but not the purpose.  Someone with understanding
> of why it was put together this way needs to check into this.

The "From " mail evenlope is inserted to make the file remain a valid
Unix mbox file.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bad (i.e. serious) mail problems

[Lars Magne Ingebrigtsen]

Shane Holder <holder@rsn.hp.com> writes:

> After looking around a bit, I think I found the problem.
> 
> Here are 2 sample headers from e-mails that get screwed up.
> 
>    From bmgoffer-owner@lists.mpath.com  Fri 05 Mar PST 14:55:22
> 
>    From blh@hpuerca.atl.hp.com  Wed Mar 17 8:57:24 1999
> 
> and one that works
> 
>    From bart@grafikon.com  Mon Mar 8 09:00:14 1999
> 
> Note that the date formats are different

That shouldn't matter, since nnmail only looks for "\n\nFrom ".  It
shouldn't care about the date format.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> Shane Holder <holder@rsn.hp.com> writes:
> 
> > After looking around a bit, I think I found the problem.
> > 
> > Here are 2 sample headers from e-mails that get screwed up.
> > 
> >    From bmgoffer-owner@lists.mpath.com  Fri 05 Mar PST 14:55:22
> > 
> >    From blh@hpuerca.atl.hp.com  Wed Mar 17 8:57:24 1999
> > 
> > and one that works
> > 
> >    From bart@grafikon.com  Mon Mar 8 09:00:14 1999
> > 
> > Note that the date formats are different
> 
> That shouldn't matter, since nnmail only looks for "\n\nFrom ".  It
> shouldn't care about the date format.

May I remind the original problem that I encountered that started that
thread? I am using nnmail fetching messages from several pop3 servers.
One of the messages had that magical sequence (perfectly legal in a
mail message I think) - "\n\nFrom". That was *not* the start of the
message, it was just a part of the message text. And gnus screwed up
that message - it split it in the middle, and put junk in my mail
group as a result. If I understand that situation correctly (please
tell me that I'm wrong) this is kind of by design. Or am I missing
something here?

(I still have the IncomingXXX file and the FUBAR split message - can
email them if needed.)

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Dmitry Yaitskov <dimas@home.com> writes:

  > One of the messages had that magical sequence (perfectly legal in a
  > mail message I think) - "\n\nFrom".

While "\n\nFrom " (note the space) may appear in a mail message, some
programs use the Unix mbox format, which says that "From " at the
beginning of a line is the start of a message, and a message ends in
an empty line ("\n\n") -- so it is natural that "\n\nFrom " was
interpreted as the end of one and the beginning of the next message.

The movemail program outputs babyl format when reading from a POP3
server -- maybe you hacked your movemail or you are using the wrong
movemail or you are not using movemail at all?

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Shane Holder]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 802 bytes --]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:


  Lars> That shouldn't matter, since nnmail only looks for "\n\nFrom ".  It
  Lars> shouldn't care about the date format.

Not quite true.
>From nnfolder-save-mail:

    (unless (looking-at message-unix-mail-delimiter)
      (insert "From nobody-nnfolder-2 " (current-time-string) "\n")
      (goto-char (point-min)))

and message-unix-mail-delimiter is:

 "From \\([^ÿ\b\n-ÿ].*\\)? \\([^ÿÿ\x7f]+\\) +\\([^ÿÿ\x7f]+\\) +\\([0-3]?[0-9]\\) +\\([0-2][0-9]:[0-5][0-9]\\(:[0-6][0-9]\\)?\\) *\\([A-Z]?[A-Z]?[A-Z][A-Z]\\( DST\\)?\\|[-+]?[0-9][0-9][0-9][0-9]\\|\\) * \\([0-9][0-9]+\\) *\\([A-Z]?[A-Z]?[A-Z][A-Z]\\( DST\\)?\\|[-+]?[0-9][0-9][0-9][0-9]\\|\\) *\\(remote from .*\\)?\n"

I'd say that that is looking for much more than \n\nFrom.

Shane

Re: bad (i.e. serious) mail problems - solution, kind of

[Dmitry Yaitskov]

Kai.Grossjohann@CS.Uni-Dortmund.DE wrote:

> Dmitry Yaitskov <dimas@home.com> writes:
> 
>   > One of the messages had that magical sequence (perfectly legal in a
>   > mail message I think) - "\n\nFrom".
> 
> While "\n\nFrom " (note the space) may appear in a mail message, some
> programs use the Unix mbox format, which says that "From " at the
> beginning of a line is the start of a message, and a message ends in
> an empty line ("\n\n") -- so it is natural that "\n\nFrom " was
> interpreted as the end of one and the beginning of the next message.
> 
> The movemail program outputs babyl format when reading from a POP3
> server -- maybe you hacked your movemail or you are using the wrong
> movemail or you are not using movemail at all?

Thanks Kai & Jaap-Henk. Sorry, my ignorance - I did not know about the
special meaning of ^From in mbox format. Still, the problem was real,
and if I understand the situation correctly, it was in pop3-movemail
from pop3.el which I am using as my movemail. I have 2 versions of
pop3.el - 1.3q that comes with pgnus, and 2.03 from the XEmacs'
mail-lib package. Both screw things up. Below is my attempt to fix it,
the patch is against 1.3q. It does "work for me" (tm), but I am not
too sure I put the fix in the right place.


-------------------cut here--------------------
--- pop3.el.old	Wed Mar 03 20:38:16 1999
+++ pop3.el	Mon Mar 29 12:16:53 1999
@@ -200,6 +200,13 @@
 (defun pop3-clean-region (start end)
   (setq end (set-marker (make-marker) end))
   (save-excursion
	;; unix mbox format treats "^From " as message start. Escape it.
+    (goto-char start)
+	(if (re-search-forward "^From " end t)
+		(progn
+		  (goto-char (match-beginning 0))
+		  (insert ">")))
+	;;
     (goto-char start)
     (while (and (< (point) end) (search-forward "\r\n" end t))
       (replace-match "\n" t t))
-------------------cut here--------------------

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems - solution, kind of

[Dmitry Yaitskov]

Dmitry Yaitskov <dimas@home.com> wrote:

<snip-snip>
> 
> 
> -------------------cut here--------------------
> --- pop3.el.old	Wed Mar 03 20:38:16 1999
> +++ pop3.el	Mon Mar 29 12:16:53 1999
> @@ -200,6 +200,13 @@
>  (defun pop3-clean-region (start end)
>    (setq end (set-marker (make-marker) end))
>    (save-excursion
> 	;; unix mbox format treats "^From " as message start. Escape it.
> +    (goto-char start)
> +	(if (re-search-forward "^From " end t)
     ^^
Oopsie. That if should be while.

> +		(progn
> +		  (goto-char (match-beginning 0))
> +		  (insert ">")))
> +	;;
>      (goto-char start)
>      (while (and (< (point) end) (search-forward "\r\n" end t))
>        (replace-match "\n" t t))
> -------------------cut here--------------------

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems - solution, kind of

[Dmitry Yaitskov]

Dmitry Yaitskov <dimas@home.com> wrote:

> Dmitry Yaitskov <dimas@home.com> wrote:
> 
> <snip-snip>
<snip-snip more>

Sorry. Here's a somewhat better version, against pop3.el version 2.03:

----------------------------cut here-------------------------------
--- e:\bin\XEmacs\xemacs-packages\lisp\mail-lib\pop3.el	Mon Mar 08 22:36:12 1999
+++ pop3.el	Mon Mar 29 15:22:04 1999
@@ -310,12 +310,18 @@
 
 (defun pop3-clean-region (start end)
   "Convert MS-DOG style line endings to normal line endings.
-Also remove '.' from the beginning of lines."
+Also remove '.' from the beginning of lines.
+Also escape 'From ' at the beginning of lines with '>'."
   (setq end (set-marker (make-marker) end))
   (save-excursion
     (goto-char start)
     (while (and (< (point) end) (search-forward "\r\n" end t))
       (replace-match "\n" t t))
+	;; Escape "\n\nFrom " (for mbox).
+    (goto-char start)
+	(while (re-search-forward "\n\n\\(From \\)" end t)
+	  (replace-match "\n\n>\\1" t nil))
+	;;
     (goto-char start)
     (while (and (< (point) end) (re-search-forward "^\\." end t))
       (replace-match "" t t)
----------------------------cut here-------------------------------

I think this is what movemail that comes with XEmacs does (I mean it
doesn't seem to look for more than just a "\n\nFrom ". And the
message that got screwed up in my mail had the following
innocent-looking line:

>From that webpage:

that got treated like the start of a new message... FWIW.

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've gone back to determine where the problem might be.  It is not Gnus,
and it is not pop3.

* Dmitry Yaitskov <dimas@home.com>  on Sun, 28 Mar 1999
| [...]  If I understand that situation correctly (please tell me that I'm
| wrong) this is kind of by design. Or am I missing something here?

Your MTA is.

Gnus uses Content-Length headers, when they exist, to determine where to
split messages.  When they do not exist, it uses SMTP envelopes (what you
mistakenly call 'headers') just like Berkeley mail and every POP server
known to me to delimit messages.  If your MTA does not generate a
Content-Length header, it is required to escape lines that look like
envelopes.  All in all, Gnus has been very good at Doing the Right Thing
with properly formed messages.

Your MTA is not generating Content-Length headers, nor is it escaping lines
that look like envelopes.  In short, it is broken -- at least that is my
supposition.

I do not want to add your patch to pop3.el.  Message transports are not
allowed to modify message bodies in any way.  Less (or perhaps more,
depending on one's perspective) importantly, your patch is a workaround for
what appears to be a really badly broken MTA, one that I have no means (or
intention, sorry) of testing against.  I cannot support code that I cannot
(or for some reason will not) test against.  Sorry.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3ACMIgl+vIlSVSNkRAjTWAKDcWyd8xrrJKbutOARwwYvyspYFbwCgwAHC
n8rJL/i8RprKAnxr/j4dQ0U=
=WYu5
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Warning: pregnant women, the elderly, and
Minion of Nathan - Nathan says Hi! \ children under 10 should avoid prolonged
PGP Key: at a key server near you!  \ exposure to Happy Fun Ball.

Re: bad (i.e. serious) mail problems - solution, kind of

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Dmitry Yaitskov <dimas@home.com>  on Mon, 29 Mar 1999
| Sorry. Here's a somewhat better version, against pop3.el version 2.03:

2.03???

That is not my code.  Use at your own risk.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3ACNigl+vIlSVSNkRAg/aAJkBeR/XfQc9KfVdFgtcMp/XtdeQGACeL82k
TsMjSv2crhXs4PZeOyI2AQM=
=BZPB
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not use Happy Fun Ball on concrete.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> I've gone back to determine where the problem might be.  It is not Gnus,
> and it is not pop3.
> 
> * Dmitry Yaitskov <dimas@home.com>  on Sun, 28 Mar 1999
> | [...]  If I understand that situation correctly (please tell me that I'm
> | wrong) this is kind of by design. Or am I missing something here?
> 
> Your MTA is.
> 
> Gnus uses Content-Length headers, when they exist, to determine where to
> split messages.  When they do not exist, it uses SMTP envelopes (what you
> mistakenly call 'headers')

Where did I talk about headers?

> just like Berkeley mail and every POP server
> known to me to delimit messages.  If your MTA does not generate a
> Content-Length header, it is required to escape lines that look like
> envelopes.  All in all, Gnus has been very good at Doing the Right Thing
> with properly formed messages.
> 
> Your MTA is not generating Content-Length headers, nor is it escaping lines
> that look like envelopes.  In short, it is broken -- at least that is my
> supposition.

Eh.... sorry, what *is* my MTA? I'm afraid I don't have one :). Or I
am again missing something, in which - quite probable - case I'd
appreciate if you explained in more detail what you mean. I am on
WinNT, and my mail is fetched from pop3 servers by, if I get this
right, pop3-movemail. pop3-movemail gets messages from the servers one
by one, so I do not see why it would be necessary for the "\n\nFrom "
lines in messages' bodies to be escaped in any way - there is no
boundaries between messages to determine at this stage. (I just looked
up rfc1725 - it says that the message should comply with rfc822 -
which I think doesn't say anything about "\n\nFrom ". So I don't think
it is required by anything in this chain although of course I may be
wrong.) pop3-movemail puts the messages in its crashbox (BTW, could it
perhaps add Content-Length to messages missing it at this stage?),
from where on it is Gnus (nnmail) who transports the messages to
appropriate groups. What is the broken MTA here?

> I do not want to add your patch to pop3.el.  Message transports are not
> allowed to modify message bodies in any way.  Less (or perhaps more,
> depending on one's perspective) importantly, your patch is a workaround for
> what appears to be a really badly broken MTA, one that I have no means (or
> intention, sorry) of testing against.  I cannot support code that I cannot
> (or for some reason will not) test against.  Sorry.

Is ok. :)

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems - solution, kind of

[Dmitry Yaitskov]

Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> * Dmitry Yaitskov <dimas@home.com>  on Mon, 29 Mar 1999
> | Sorry. Here's a somewhat better version, against pop3.el version 2.03:
> 
> 2.03???
> 
> That is not my code.  Use at your own risk.

This is the pop3.el included in mail-lib XEmacs package:

--------------top of file---------------
;;; pop3.el --- Post Office Protocol (RFC 1460) interface

;; Copyright (C) 1996,1997,1998 Free Software Foundation, Inc.
;; Copyright (C) 1997 Franklin Lee

;; Author: Richard L. Pieri <ratinox@peorth.gweep.net>
;; Author: Franklin Lee <flee@lehman.com>
;; Author: Andy Piper <andy@xemacs.org>

;; Keywords: mail, pop3
;; Version: 2.03

;; This file is part of XEmacs.
<snip the rest of the file>

A fake??? ;^|

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > Gnus uses Content-Length headers, when they exist, to determine where to
  > split messages.  When they do not exist, it uses SMTP envelopes (what you
  > mistakenly call 'headers') just like Berkeley mail and every POP server
  > known to me to delimit messages.

Why should a POP server delimit messages in mbox style?  The POP
protocol says (I think) that a message is terminated by a "." line,
and internally, the POP server could well use any mechanism for
storing messages, such as a file per message, or an RDBMS.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Dmitry Yaitskov <dima@apexsc.com>  on Mon, 29 Mar 1999
| Eh.... sorry, what *is* my MTA? I'm afraid I don't have one :).

Yes, you do.  MTA: Mail Transport Agent.  Something like sendmail or qmail
or exim.  You have to have one, or you would not be receiving any mail at
all.  It is runinng on the machine from which you retrieve your mail.

| pop3-movemail gets messages from the servers one by one, so I do not see
| why it would be necessary for the "\n\nFrom " lines in messages' bodies
| to be escaped in any way

It is not for pop3-movemail, but for the POP server it talks to.  The
server needs to know where one message ends and the next begins.  If a
Content-Length header exists, it will be used; otherwise, the server will
scan through the spool file looking for RFC 821 (SMTP) envelopes (the
'\n\nFrom ' lines).  Escaping envelope-like lines is an artifact of RFC
821, not 822.

Thus, if escaping of envelop-like lines is required on your system, it
should have been done long before pop3-movemail ever comes into play.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AOnSgl+vIlSVSNkRAvWgAJ98nhcxvMuf23+sjTu/Xxv2oO3CqwCfWWJ5
yFJ7vcBOdsI5kbrghSl252Q=
=qzvZ
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Caution: Happy Fun Ball may suddenly
Minion of Nathan - Nathan says Hi! \ accelerate to dangerous speeds.
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > It is not for pop3-movemail, but for the POP server it talks to.  The
  > server needs to know where one message ends and the next begins.  If a
  > Content-Length header exists, it will be used; otherwise, the server will
  > scan through the spool file looking for RFC 821 (SMTP) envelopes (the
  > '\n\nFrom ' lines).  Escaping envelope-like lines is an artifact of RFC
  > 821, not 822.

So the whole thing is only needed if a `spool file' (mbox format) is
involved -- what if there is a setup where no such file is involved at
all?

Then pop3.el assumes such a file has been involved and just
concatenates the messages retrieved from the POP server.  IMHO this is
wrong -- the program converting to mbox format should take care of
escaping From_ lines if necessary.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Tue, 30 Mar 1999
| Why should a POP server delimit messages in mbox style?

It does not.  The POP server uses the whatever delimiters have been created
by the MTA to determine where messages begin.  What as this is usually an
RFC 821 envelope, RFC 821 envelopes are used as delimiters.

| The POP protocol says (I think) that a message is terminated by a "."
| line,

The '.' is just that, a terminator for the RETR command.  It tells the POP
client that the server has finished transmitting a message.  The dot
terminator is stripped by the client.

| and internally, the POP server could well use any mechanism for storing
| messages, such as a file per message, or an RDBMS.

POP servers do not store anything anywhere.  IMAP does, but IMAP is so
different from POP that comparing them is like comparing Gnus to a ball of
yarn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AOukgl+vIlSVSNkRAm6VAKD5MQBuP/MO5g6cbJjJaBlo9vXCtwCeIS9V
PkBo6I8xWYaTBOsdbBic6I8=
=D0FS
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ When not in use, Happy Fun Ball should be
Minion of Nathan - Nathan says Hi! \ returned to its special container and
PGP Key: at a key server near you!  \ kept under refrigeration.

Re: bad (i.e. serious) mail problems - solution, kind of

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Dmitry Yaitskov <dima@apexsc.com>  on Mon, 29 Mar 1999
| This is the pop3.el included in mail-lib XEmacs package:
[...]
| A fake??? ;^|

No, just not my code.  And I've asked them to change the name because it is 
not my code.  Either they have not, or you have an older version.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AOzxgl+vIlSVSNkRAmdpAKCudB/X6pAGfTqnpvsO1rUYIQuu4gCgl21X
R/QYk9jdlfbj4NOIdEX5lII=
=wOFM
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > * Kai.Grossjohann@CS.Uni-Dortmund.DE  on Tue, 30 Mar 1999
  > | Why should a POP server delimit messages in mbox style?
  > 
  > It does not.  The POP server uses the whatever delimiters have
  > been created by the MTA to determine where messages begin.  What
  > as this is usually an RFC 821 envelope, RFC 821 envelopes are used
  > as delimiters.

A message comes in via SMTP.  How it gets from there to the POP server
is not defined.  Maybe there are no delimiters (other than
end-of-file, or the closing of a socket) at all?

  > | The POP protocol says (I think) that a message is terminated by a "."
  > | line,
  > 
  > The '.' is just that, a terminator for the RETR command.  It tells the POP
  > client that the server has finished transmitting a message.  The dot
  > terminator is stripped by the client.

Right.  So, the client (including pop3.el) knows that the end of the
message must be right before the dot.  pop3.el, however, does not make
sure that these are the endings of messages in its output file.

  > | and internally, the POP server could well use any mechanism for storing
  > | messages, such as a file per message, or an RDBMS.
  > 
  > POP servers do not store anything anywhere.  IMAP does, but IMAP is so
  > different from POP that comparing them is like comparing Gnus to a ball of
  > yarn.

Hm?  Neither protocol says anything about storage.

But certainly, the messages must be stored somewhere, for the POP and
IMAP servers to be able to output them.  Nobody says that this has got
to be a file in /var/spool/mail for POP servers and something else for
IMAP servers.

Let me rephrase my sentence: ... and the message store accessed by the
POP server could use any method at all, such as a file per message, or
an RDBMS.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> * Dmitry Yaitskov <dima@apexsc.com>  on Mon, 29 Mar 1999
> | Eh.... sorry, what *is* my MTA? I'm afraid I don't have one :).
> 
> Yes, you do. MTA: Mail Transport Agent. Something like sendmail or
> qmail or exim. You have to have one, or you would not be receiving
> any mail at all. It is runinng on the machine from which you
> retrieve your mail.

I know what MTA stands for, thank you. I do not have one running on my
machine. I am saying that the problem happens on my machine, not on
the machine I retrieve my mail from. I retrieve my mail from 3(!)
different pop3 servers, running on 3 different machines - 2 of them
are unixes, one I dunno what. Without my patch, the problem shows with
2 of the 3 pop3 servers, *when I retrieve my mail from them via pop3*.
If I telnet to them and use pine there, the bloody From gets escaped
with a > on *both* unixes, whereas when fetching the same message via
pop3 only one of their pop3 servers passes me the escaped From,
whereas the other does not. And IMO it certainly does *not* have to.
It (or rather they) is *not* broken. The "\n\nFrom " has special
meaning *only* for mbox format - not for pop3. When I retrieve mail
via pop3 no mbox needs to be involved at all. Again, pop3 protocol
does *not* require "\n\nFrom " lines to be escaped in any way, they do
not have any special meaning for it. So your reasoning is wrong in
this case. The pop3 server knows perfectly well where one message ends
and another begins, it does not have to store messages in mbox format,
and when it gives messages to you (pop3-movemail) it splits them
correctly, one by bloody one. It is the job of whoever builds the mbox
to escape things having special meaning for mbox (and for nothing
else). So, it is gnus (or rather, pop3-movemail) who screws things up
in this case, because it (pop3-movemail) builds an invalid mbox out of
perfectly valid separate messages. For crissake, even M$ bloody
outlook express did *not* screw up this message although it received
it from a server that does *not* escape the From. (And BTW, it of
course did not escape the From - because it does not use mbox at all.)

> | pop3-movemail gets messages from the servers one by one, so I do not see
> | why it would be necessary for the "\n\nFrom " lines in messages' bodies
> | to be escaped in any way
> 
> It is not for pop3-movemail, but for the POP server it talks to.  The
> server needs to know where one message ends and the next begins.

See above. The server knows perfectly well where one message ends and
another begins. It is gnus who gets confused by the bad mbox built
locally on my MTA-less machine.

> If a
> Content-Length header exists, it will be used; otherwise, the server will
> scan through the spool file looking for RFC 821 (SMTP) envelopes (the
> '\n\nFrom ' lines).  Escaping envelope-like lines is an artifact of RFC
> 821, not 822.
> 
> Thus, if escaping of envelop-like lines is required on your system, it
> should have been done long before pop3-movemail ever comes into play.

No. See above.

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> * Dmitry Yaitskov <dima@apexsc.com>  on Mon, 29 Mar 1999
> | Eh.... sorry, what *is* my MTA? I'm afraid I don't have one :).
> 
> Yes, you do.  MTA: Mail Transport Agent.  Something like sendmail or qmail
> or exim.  You have to have one, or you would not be receiving any mail at
> all.  It is runinng on the machine from which you retrieve your mail.
> 
> | pop3-movemail gets messages from the servers one by one, so I do not see
> | why it would be necessary for the "\n\nFrom " lines in messages' bodies
> | to be escaped in any way
> 
> It is not for pop3-movemail, but for the POP server it talks to.  The
> server needs to know where one message ends and the next begins.  If a
> Content-Length header exists, it will be used; otherwise, the server will
> scan through the spool file looking for RFC 821 (SMTP) envelopes (the
> '\n\nFrom ' lines).  Escaping envelope-like lines is an artifact of RFC
> 821, not 822.

Could you please specify what you mean? I just looked up rfc821 and
could not find where it talks about "\n\nFrom " lines. Probably I just 
missed it though.

> 
> Thus, if escaping of envelop-like lines is required on your system, it
> should have been done long before pop3-movemail ever comes into play.

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Tue, 30 Mar 1999
| Then pop3.el assumes such a file has been involved and just concatenates
| the messages retrieved from the POP server.  IMHO this is wrong -- the
| program converting to mbox format should take care of escaping From_
| lines if necessary.

Read the POP3 specifications.  POP clients perform no modification to the
mail spool or messages retrieved from it.  Or if they do, they do as little
as possible.  IN the case of pop3, I construct an RFC 821-style envelope
based on RFC 822 headers only if the POP server is stupid enough to strip
it *AND* there is no MMDF or Babyl delimiter (yes, pop3 is smart enough to
grok MMDF and Babyl messages).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AUtcgl+vIlSVSNkRAiZEAKCjNyYR7qhA8Yzp6SJ4nbKadwf3TwCeIbjJ
5TYlNYw17uAgjz5OHKGfMMc=
=4BIQ
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Caution: Happy Fun Ball may suddenly
Minion of Nathan - Nathan says Hi! \ accelerate to dangerous speeds.
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Tue, 30 Mar 1999
| A message comes in via SMTP.  How it gets from there to the POP server is
| not defined.  Maybe there are no delimiters (other than end-of-file, or
| the closing of a socket) at all?

There must be some means of delimiting messages in a mail spool.  Using
SMTP envelopes (mbox) is just one of several in common use.

[...]

| Right.  So, the client (including pop3.el) knows that the end of the
| message must be right before the dot.  pop3.el, however, does not make
| sure that these are the endings of messages in its output file.

The '.' line is a terminator for the *protocol* (and is in fact the same
terminator used in other mail-related areas).  It signals the POP client
that the RETR command has completed.  That is all it does.  It is not part
of the message body.  It is not a spool delimiter.  It must be removed
before the message is permanently stored on the local system because it is
not part of the message.

[...]
| Let me rephrase my sentence: ... and the message store accessed by the
| POP server could use any method at all, such as a file per message, or
| an RDBMS.

Agreed.  Which is why pop3 groks the three mail spool file delimiters: SMTP 
envelope (mbox), MMDF, and Babyl.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AWhlgl+vIlSVSNkRAuLSAJ477ISyVkPvVNUypbjCzUPlPEczHgCeLulG
w7gOGs+lA/Bbx1sZ6nZjq4s=
=15st
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not use Happy Fun Ball on concrete.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
| And IMO it certainly does *not* have to.

Not just your opinion: POP is not allowed to muck with message bodies at
all.

| It (or rather they) is *not* broken. The "\n\nFrom " has special meaning
| *only* for mbox format - not for pop3.

Clarification: the SMTP envelope is of critical importance to SMTP (duh :).
It just so happens that 'mbox' uses the envelope for something else, since
it exist.  It was a stupid decision, but the world is stuck with it.

| When I retrieve mail via pop3 no mbox needs to be involved at all. Again,
| pop3 protocol does *not* require "\n\nFrom " lines to be escaped in any
| way, they do not have any special meaning for it.

Not entirely true.  If the mail host stores mail in mbox-style spool files,
and the MTA or delivery agent does not generate Content-Length headers,
then message body lines that resemble SMTP envelopes should be escaped by
the MTA or local delivery agent on the mail host.  If that does not happen,
the POP server or local mail clients could become confused.

The POP retrieveal mechanism cares not a whit about delimiters on the
server, since the POP server has already done that work.

[...]
| It is the job of whoever builds the mbox to escape things having special
| meaning for mbox (and for nothing else). So, it is gnus (or rather,
| pop3-movemail) who screws things up in this case, because it
| (pop3-movemail) builds an invalid mbox out of perfectly valid separate
| messages.

FYI, pop3-movemail (and the rest of the innards of the code) generate a
crashbox that is identical to the spool file on the server (or as close as
I can get given a certain university's widely used POP server that has a
tendency to strip trailing whitespace from messages).  If the mail host
uses MMDF, an MMDF crashbox is generated.  If the mail host uses Babyl, a
Babyl crashbox is generated.  If the mail host uses mbox, an mbox crashbox
is generated.

The only time pop3 generates an mbox-style crashbox is if the mail host
uses none of these formats and/or the POP server is broken and strips away
the message delimiters.  Then, and only then, will pop3 generate an SMTP
envelope style delimiter derrived from RFC 822 headers.  And I have put
much effort into making sure that the envelope-like delimiter is 100%
accurate in format.  Look at pop3-munge-message-separator for specifics.

If after all that Gnus' spool file parsing code still barfs on a particular
message, it means that the MTA or delivery agent on the mail host failed to
either generate a Content-Length header or escape the envelope-like line.

If that is not the case (show us the Content-Length header in that
message), then you have stumbled onto a Genuine Bug in Gnus.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AWyhgl+vIlSVSNkRAnwtAKCIU0h6DNSgP4YtScH26moR3NFyRgCg3gA1
gDaPZ6EcHqr5X4frqDwwJ2Q=
=cSD9
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
| Could you please specify what you mean? I just looked up rfc821 and
| could not find where it talks about "\n\nFrom " lines. Probably I just
| missed it though.

It is called the 'envelope'.  Look for that instead of '\n\nFrom line'.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AWzQgl+vIlSVSNkRAlTQAJ47BYyAUCKGhzo8iZsq1KXbypiJTACeOMqk
A2QDWPAqj6vQgcKHlD0hDso=
=Ofra
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Ingredients of Happy Fun Ball include an
Minion of Nathan - Nathan says Hi! \ unknown glowing substance which fell to
PGP Key: at a key server near you!  \ Earth, presumably from outer space.

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> * Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
<snip>
> | When I retrieve mail via pop3 no mbox needs to be involved at all. Again,
> | pop3 protocol does *not* require "\n\nFrom " lines to be escaped in any
> | way, they do not have any special meaning for it.
> 
> Not entirely true.  If the mail host stores mail in mbox-style spool files,
> and the MTA or delivery agent does not generate Content-Length headers,
> then message body lines that resemble SMTP envelopes should be escaped by
> the MTA or local delivery agent on the mail host.  If that does not happen,
> the POP server or local mail clients could become confused.

Where (rfc#) does it say so? Note, the pop3 server (or rather, 3
different pop servers that I tested) did not become confused. It was
the local mail client (gnus) that got confused.

> The POP retrieveal mechanism cares not a whit about delimiters on the
> server, since the POP server has already done that work.

So?...

> [...]
> | It is the job of whoever builds the mbox to escape things having special
> | meaning for mbox (and for nothing else). So, it is gnus (or rather,
> | pop3-movemail) who screws things up in this case, because it
> | (pop3-movemail) builds an invalid mbox out of perfectly valid separate
> | messages.
> 
> FYI, pop3-movemail (and the rest of the innards of the code) generate a
> crashbox that is identical to the spool file on the server (or as close as
> I can get given a certain university's widely used POP server that has a
> tendency to strip trailing whitespace from messages).  If the mail host
> uses MMDF, an MMDF crashbox is generated.  If the mail host uses Babyl, a
> Babyl crashbox is generated.  If the mail host uses mbox, an mbox crashbox
> is generated.

Could you please explain - how on Earth do you *know* what the "spool
file on the server" (if by the server you mean the machine running the
pop3 you got the mail from) looks like? This info is certainly not
included in the pop3 protocol, is it?

> The only time pop3 generates an mbox-style crashbox is if the mail host
> uses none of these formats and/or the POP server is broken and strips away
> the message delimiters.  Then, and only then, will pop3 generate an SMTP
> envelope style delimiter derrived from RFC 822 headers.  And I have put
> much effort into making sure that the envelope-like delimiter is 100%
> accurate in format.  Look at pop3-munge-message-separator for specifics.
> 
> If after all that Gnus' spool file parsing code still barfs on a particular
> message, it means that the MTA or delivery agent on the mail host failed to
> either generate a Content-Length header or escape the envelope-like line.
> 
> If that is not the case (show us the Content-Length header in that
> message), then you have stumbled onto a Genuine Bug in Gnus.

Thank you. I did stumble upon a bug, that's for sure. Again, just in
case you missed my point. I tried sending and receiving the message
that got FUBAR thru *3* different machines, running in 3 different
organizations. Retrieving that message with pop3-movemail/gnus from 2
of them produced the error, while the third did effectively what my
patch does. Reading that same message directly on those hosts was ok.
Getting that same message from same pop3 hosts with MS outlook express
did *not* produce the error, obviously because it (MSOE) was smart
enough not to assume things about the servers it was getting mail from
it had no way of knowing. Tell me that's not a bug.

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> * Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
> | Could you please specify what you mean? I just looked up rfc821 and
> | could not find where it talks about "\n\nFrom " lines. Probably I just
> | missed it though.
> 
> It is called the 'envelope'.  Look for that instead of '\n\nFrom line'.

Could you please quote the part of the rfc821 that requires lines
looking like "\n\nFrom " to be treated specially, even if they are
called "envelopes"?

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
| Where (rfc#) does it say so? Note, the pop3 server (or rather, 3
| different pop servers that I tested) did not become confused. It was
| the local mail client (gnus) that got confused.

Then the bug is in Gnus.  QED.

[...]
| Could you please explain - how on Earth do you *know* what the "spool
| file on the server" (if by the server you mean the machine running the
| pop3 you got the mail from) looks like? This info is certainly not
| included in the pop3 protocol, is it?

      (narrow-to-region start end)
      (goto-char (point-min))
      (if (not (or (looking-at "From .?") ; Unix mail
		   (looking-at "\001\001\001\001\n") ; MMDF
		   (looking-at "BABYL OPTIONS:") ; Babyl
		   ))

That is how I know.  If any of those regexps are there at the beginning of
the buffer, the spool is mbox, MMDF, or Babyl.  If none of these are there,
the POP server has (incorrectly) stripped that information from the
message, and something must be inserted to replace it.  The choice of
generating faux SMTP envelopes was arbitrary.  I probably had a good reason
for that.

| [...] Tell me that's not a bug.

Show me the offending message in its entirety, including all of the headers 
and the envelope, and I will tell you exactly where the cause is.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AXsjgl+vIlSVSNkRAs+/AKDqYr8ord8XqM7B62bUTHcyT+rUgQCg16aS
o3XSmKM/IaKIUNZGyICu0nE=
=+1DP
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not use Happy Fun Ball on concrete.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> * Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
> | Where (rfc#) does it say so? Note, the pop3 server (or rather, 3
> | different pop servers that I tested) did not become confused. It was
> | the local mail client (gnus) that got confused.
> 
> Then the bug is in Gnus.  QED.
> 
> [...]
> | Could you please explain - how on Earth do you *know* what the "spool
> | file on the server" (if by the server you mean the machine running the
> | pop3 you got the mail from) looks like? This info is certainly not
> | included in the pop3 protocol, is it?
> 
>       (narrow-to-region start end)
>       (goto-char (point-min))
>       (if (not (or (looking-at "From .?") ; Unix mail
> 		   (looking-at "\001\001\001\001\n") ; MMDF
> 		   (looking-at "BABYL OPTIONS:") ; Babyl
> 		   ))
> 
> That is how I know.  If any of those regexps are there at the beginning of
> the buffer, the spool is mbox, MMDF, or Babyl.  If none of these are there,
> the POP server has (incorrectly) stripped that information from the
> message, and something must be inserted to replace it.

Why incorrectly? This info is not part of the message, why should it
not be stripped, even if it was present while the server has been
storing the message? What rfc specifies that a message retrieved via
pop3 must have one of those things as the first line of the message?
Looks like a wrong assumption to me. And now I kinda see your logic in
not trying to escape the you-know-what :) lines, but it seems faulty
to me. You try to guess the format the server stored its messages in
before it gave them to you via pop3. If you cannot guess (i.e. none of
the 3 regexps above match) you generate SMTP envelopes yourself, but
you do *not* escape "\n\nFrom " as if you were dealing with a valid
mbox format whereas what you're dealing with are just separate
messages, and for separate messages, the "\n\nFrom " does not have to
mean anything, and having such lines for them is perfectly legal. But
if you are building the mbox (as is the case), *you* (I mean
pop3-movemail) is the missing :) MTA, and you should make sure that
what you build is valid. Whereas it is not.

BTW, I just rechecked - neither of my 3 pop3 servers supplies any of
the magic sequences you're using to second-guess its original storage
mode.

> The choice of
> generating faux SMTP envelopes was arbitrary.  I probably had a good reason
> for that.
> 
> | [...] Tell me that's not a bug.
> 
> Show me the offending message in its entirety, including all of the headers 
> and the envelope, and I will tell you exactly where the cause is.

Enjoy :) although I don't see how it can be useful. The 'envelope'
(the first line) has been generated by pop3-movemail anyway.

-------------------------cut here----------------------------------------
>From Robertson.Davenport@bailey.com  Fri Mar 12 08:45:40 1999
Return-Path: <owner-ntemacs-users@cs.washington.edu>
Received: from h1.mail.home.com ([24.0.0.50]) by mail.rdc1.on.home.com
          (InterMail v4.00.03 201-229-104) with ESMTP
          id <19990312152831.EGNX8107.mail.rdc1.on.home.com@h1.mail.home.com>
          for <dimas@mail.mtth1.on.wave.home.com>;
          Fri, 12 Mar 1999 07:28:31 -0800
Received: from mx3-w.mail.home.com (mx3-w.mail.home.com [24.0.0.53])
	by h1.mail.home.com (8.9.0/8.9.0) with ESMTP id HAA15729;
	Fri, 12 Mar 1999 07:28:29 -0800 (PST)
Received: from trout.cs.washington.edu (trout.cs.washington.edu [128.95.1.178])
	by mx3-w.mail.home.com (8.9.1/8.9.1) with ESMTP id HAA14336;
	Fri, 12 Mar 1999 07:28:29 -0800 (PST)
Received: (majordom@localhost) by trout.cs.washington.edu (8.8.5+CS/7.2trout) id FAA06845 for ntemacs-users-outgoing; Fri, 12 Mar 1999 05:46:21 -0800 (PST)
Received: from june.cs.washington.edu (june.cs.washington.edu [128.95.1.4]) by trout.cs.washington.edu (8.8.5+CS/7.2trout) with ESMTP id FAA06841 for <ntemacs-users@trout.cs.washington.edu>; Fri, 12 Mar 1999 05:46:19 -0800 (PST)
Received: from mail.bailey.com (mail.bailey.com [192.231.53.2]) by june.cs.washington.edu (8.8.7+CS/7.2ju) with SMTP id FAA13909 for <ntemacs-users@cs.washington.edu>; Fri, 12 Mar 1999 05:46:18 -0800
Received: from mail.bailey.com [192.231.53.2]
	(HELO localhost)
	by mail.bailey.com (AltaVista Mail V2.0J/2.0J BL25J listener)
	id 0000_03c2_36e9_1af1_8ba8;
	Fri, 12 Mar 1999 08:47:29 -0500
Message-ID: <557E1B259579D211AF3E0008C7B14C8A3C21D8@exchcle5.bailey.com>
From: "Davenport, Robertson G" <Robertson.Davenport@bailey.com>
To: Jack Vinson <jvinson@chevax.ecs.umass.edu>,
        ntemacs-users
	 <ntemacs-users@cs.washington.edu>
Subject: RE: command line telnet for NT?
Date: Fri, 12 Mar 1999 08:45:40 -0500
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2448.0)
Content-Type: text/plain;
	charset="iso-8859-1"
Sender: owner-ntemacs-users@cs.washington.edu
Precedence: bulk
X-FAQ: http://www.cs.washington.edu/homes/voelker/ntemacs.html

I just ran into a new telnet client (well, it's been around for 
os/2 since 95 apparently) from Dima Malof:
  http://www.corbina.net/~maloff/tn/
I haven't tested it with Emacs yet, and thought someone else 
might want to try it and see how it fares. It's an 80KB download.

>From that webpage:
"Features:
  Emulation of mansi.sys, nansi.sys, nnansi.sys drivers. 
     Browse your Unix's termcap -- you have an entry for one 
     of this drivers for sure. Tn can (at least) emulate ANSI, 
     AT386, Interactive UNIX, Linux or FreeBSD console. 
     (Default is AT386.) 
  Charset translation. KOI8-r table is included in the d
     istribution. 
  Scroll back buffer ("Terminal History"). 
  Built-in telnet-specific "mouse actions": copy or paste to 
     the System Clipboard using your mouse, or scroll terminal's 
     history. 
  Multi-console a la screen. Run up to 10 sessions, switch 
     them with alt+F1 ... alt+F10 (default). 
  Italic and underline support (Specify a color to emulate 
     these ANSI states). 
  Support for ENVIRON, NEW-ENVIRON (Partial auto-login and 
     environment vars), TTYPE, NAWS(Transfer screen's Cols and 
     Rows to the remote) telnet protocol options. 
  Key mappings. 

  You will need Winsock 2.0 to run it. (It's included with 
     NT 4.0. If you want to run it under Windows 95 download 
     and install Winsock 2.0 for Windows 95 -- 1.4M. 
     Windows'95 has slow and buggy TCP/IP, so don't be 
     surprised.) 

  You can use NT's keyboard driver to switch to Russian under 
     NT, and you should run a DOS keyboard driver under 
     Windows'95. 

Rob Davenport

> -----Original Message-----
> From: Jack Vinson [mailto:jvinson@chevax.ecs.umass.edu]
> Sent: Tuesday, March 09, 1999 11:13 AM
> 
> I am looking for a decent version of telnet that will work 
> from the command
> line.  This should behave like the current versions of telnet 
> on Unix, for
> compatibility with the functions that call telnet.
-------------------------cut here----------------------------------------


-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Greg Stark]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=NIL, Size: 1145 bytes --]


Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
>
> | Where (rfc#) does it say so? Note, the pop3 server (or rather, 3
> | different pop servers that I tested) did not become confused. It was
> | the local mail client (gnus) that got confused.
> 
> Then the bug is in Gnus.  QED.

Indeed SSR, if POP requires you not to mangle the message and you intend to
put it in a mbox format you're going to have to pick one standard or the other
to violate. Picking the mbox format to violate means generating a broken mbox,
so that's right out. Basically if you want to use an mbox spool to transport
the mail then you have a design limitation that precludes you following the
POP spec in certain circumstances.

I know it turns out your code wasn't involved, but it sounds like you seem
convinced you should be trusting on network data to preserve the integrity of
Gnus's spool. If that's the case you're asking for all kidns of trouble.
Security principles dictate that under no condition should data from the
network be trusted such that invalid data would break the client.

greg

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Greg Stark <gsstark@mit.edu> wrote:

> Stainless Steel Rat <ratinox@peorth.gweep.net> writes:
> 
> > * Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
> >
> > | Where (rfc#) does it say so? Note, the pop3 server (or rather, 3
> > | different pop servers that I tested) did not become confused. It was
> > | the local mail client (gnus) that got confused.
> > 
> > Then the bug is in Gnus.  QED.
> 
> Indeed SSR, if POP requires you not to mangle the message and you intend to
> put it in a mbox format you're going to have to pick one standard or the other
> to violate. Picking the mbox format to violate means generating a broken mbox,
> so that's right out. Basically if you want to use an mbox spool to transport
> the mail then you have a design limitation that precludes you following the
> POP spec in certain circumstances.
> 
> I know it turns out your code wasn't involved,

I think it was. It was pop3-movemail that built an invalid mbox from
perfectly valid *separate* messages retrieved via pop3 one by one.

> but it sounds like you seem
> convinced you should be trusting on network data to preserve the integrity of
> Gnus's spool. If that's the case you're asking for all kidns of trouble.
> Security principles dictate that under no condition should data from the
> network be trusted such that invalid data would break the client.
> 
> greg
> 
> 
> 

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > Not entirely true.  If the mail host stores mail in mbox-style
  > spool files, and the MTA or delivery agent does not generate
  > Content-Length headers, then message body lines that resemble SMTP
  > envelopes should be escaped by the MTA or local delivery agent on
  > the mail host.  If that does not happen, the POP server or local
  > mail clients could become confused.

The problem occurs in the file pop3-movemail is _writing_ to!  And
that does not depend at all on whether or not the mail host stores
mails in mbox-style files.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  >       (narrow-to-region start end)
  >       (goto-char (point-min))
  >       (if (not (or (looking-at "From .?") ; Unix mail
  > 		   (looking-at "\001\001\001\001\n") ; MMDF
  > 		   (looking-at "BABYL OPTIONS:") ; Babyl
  > 		   ))
  > 
  > That is how I know.

That a file starts with "From " is not sufficient indication that it
is mbox format.  It might be one message stored in one file.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > * Kai.Grossjohann@CS.Uni-Dortmund.DE  on Tue, 30 Mar 1999
  > 
  > | Right.  So, the client (including pop3.el) knows that the end of the
  > | message must be right before the dot.  pop3.el, however, does not make
  > | sure that these are the endings of messages in its output file.
  > 
  > The '.' line is a terminator for the *protocol* (and is in fact the same
  > terminator used in other mail-related areas).  It signals the POP client
  > that the RETR command has completed. [...]

It also indicates the end of the message.  In fact, it is the *only*
end of message present in the response of the RETR command.

Hm.  I have now read RFC 1939 (I hope this is the right one), and it
contains the following two snippets:

,-----
|    Responses to certain commands are multi-line.  In these cases, which
|    are clearly indicated below, after sending the first line of the
|    response and a CRLF, any additional lines are sent, each terminated
|    by a CRLF pair.  When all lines of the response have been sent, a
|    final line is sent, consisting of a termination octet (decimal code
|    046, ".") and a CRLF pair.  If any line of the multi-line response
|    begins with the termination octet, the line is "byte-stuffed" by
|    pre-pending the termination octet to that line of the response.
|    Hence a multi-line response is terminated with the five octets
|    "CRLF.CRLF".  When examining a multi-line response, the client checks
|    to see if the line begins with the termination octet.  If so and if
|    octets other than CRLF follow, the first octet of the line (the
|    termination octet) is stripped away.  If so and if CRLF immediately
|    follows the termination character, then the response from the POP
|    server is ended and the line containing ".CRLF" is not considered
|    part of the multi-line response.
`-----

The above snippet indicates that the dot-line and *only* the dot-line
indicates the end of the response.  It also indicates that some lines
in the response might be changed -- the ones beginning with a dot.

It does not impose any other restrictions whatsoever on multi-line
responses.  In particular, it does not require that there be only one
From_ line in the multi-line response.

,-----
|       RETR msg
| 
|          Arguments:
|              a message-number (required) which may NOT refer to a
|              message marked as deleted
| 
|          Restrictions:
|              may only be given in the TRANSACTION state
| 
|          Discussion:
|              If the POP3 server issues a positive response, then the
|              response given is multi-line.  After the initial +OK, the
|              POP3 server sends the message corresponding to the given
|              message-number, being careful to byte-stuff the termination
|              character (as with all multi-line responses).
| 
|          Possible Responses:
|              +OK message follows
|              -ERR no such message
| 
|          Examples:
|              C: RETR 1
|              S: +OK 120 octets
|              S: <the POP3 server sends the entire message here>
|              S: .
| 
`-----

The above snippet indicates that the response to a RETR command is
exactly one message (or an error).

Therefore, even if the response of a RETR command were to look like
this:

,-----
| +OK 109 octets
| From foo
| From: foo
| To: bar
| Subject: msg 1
| 
| msg 1 text
| 
| From bar
| From: bar
| To: foo
| Subject: msg 2
| 
| msg 2 text
| .
`-----

... this would still be just one message, right?  You are assuming
this is two messages, which is wrong.

If the RFC tells us that the above is an invalid response, please
quote the relevant portion.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > * Kai.Grossjohann@CS.Uni-Dortmund.DE  on Tue, 30 Mar 1999
  > | Then pop3.el assumes such a file has been involved and just concatenates
  > | the messages retrieved from the POP server.  IMHO this is wrong -- the
  > | program converting to mbox format should take care of escaping From_
  > | lines if necessary.
  > 
  > Read the POP3 specifications.  POP clients perform no modification to the
  > mail spool or messages retrieved from it.  Or if they do, they do as little
  > as possible.

Again: there is nothing that says that the POP server should output
escaped From_ lines when given the RETR command, right?  The From_
lines must be escaped when pop3.el outputs mbox format, though.  So
pop3.el must escape the From_ lines.

(Modulo the presence of Content-Length headers or stuff like that.)

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Frank D. Cringle]

Kai.Grossjohann@CS.Uni-Dortmund.DE writes:
> That a file starts with "From " is not sufficient indication that it
> is mbox format.  It might be one message stored in one file.

Right.  That is the case for qmail, where Mail destined to be
collected via POP is typically delivered to a Maildir (one file per
message).  The envelope is stored in the first 2 lines:

  From <envelope-sender>
  Delivered-To: recipient
  Received: ... blah...
  ... etc ...

The pop server, qmail-pop3d, sends the contents of this file to the
client as-is (with '.' escaping and CRLF endings).  It does not care
whether any body lines begin with "From ".

-- 
Frank Cringle,      fdc@cliwe.ping.de
voice: (+49 2304) 467101; fax: 943357

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Wed, 31 Mar 1999
| That a file starts with "From " is not sufficient indication that it
| is mbox format.  It might be one message stored in one file.

That a *MESSGE* starts with "From " *IS* sufficient indication that the
spoolis in mbox format.  The first line retrieved by the POP RETR command
will be either a message delimiter or an RFC 822 header.  It cannot be
anything else.  If it is an MMDF delimiter, it is an MMDF spool.  If it is
a Babyl delimiter, it is a Babyl spool.  If it is an RFC 822 header, the
POP server is broken.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3Ajsigl+vIlSVSNkRAhQHAJ4wtrLygPJWNo/tSkUdHqN9LB612QCgybUd
gLeetTy0u60BZKKw0oJQcgw=
=eDn1
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Caution: Happy Fun Ball may suddenly
Minion of Nathan - Nathan says Hi! \ accelerate to dangerous speeds.
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Wed, 31 Mar 1999
| | .
| `-----

| ... this would still be just one message, right?  You are assuming
| this is two messages, which is wrong.

I do no such thing.  The dot line is escaped by the server, meaning that
pop3-retr does not see it as a terminator, and it sees only one message.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AjwPgl+vIlSVSNkRAt7xAKC0hDfNPYzGWfMsDM027uvjU2xDqwCfYJZq
WUN+KrlQz//ZGiBstziVohQ=
=+xjy
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Happy Fun Ball may stick to certain types
Minion of Nathan - Nathan says Hi! \ of skin.
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Wed, 31 Mar 1999
| Again: there is nothing that says that the POP server should output
| escaped From_ lines when given the RETR command, right?  The From_ lines
| must be escaped when pop3.el outputs mbox format, though.

POP is not allowed to modify message bodies in any way (modulo properly
escaped and then de-escaped 'dot lines').  The delivery agent on the mail
host must either escape the envelope-like lines or generate a
Content-Length header if one does not exist.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3Aj1Dgl+vIlSVSNkRAkrdAJ90JoBOaPAX3jPskEe3IMySg8oLhwCbBeON
p3e9DBYtBq2zSxznhnPBR7k=
=mSlK
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ When not in use, Happy Fun Ball should be
Minion of Nathan - Nathan says Hi! \ returned to its special container and
PGP Key: at a key server near you!  \ kept under refrigeration.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > * Kai.Grossjohann@CS.Uni-Dortmund.DE  on Wed, 31 Mar 1999
  > | | .
  > | `-----
  > 
  > | ... this would still be just one message, right?  You are assuming
  > | this is two messages, which is wrong.
  > 
  > I do no such thing.  The dot line is escaped by the server, meaning that
  > pop3-retr does not see it as a terminator, and it sees only one message.

pop3-movemail writes this message to its output file as-is.  As the
output file (~/.crashbox) is mbox format, the From_ line in the middle
will be interpreted as a beginning-of-message by whatever is reading
the output file of pop3-movemail.

So, pop3-movemail needs to be fixed.

Btw, the mbox format says that a message should be terminated by an
empty line.  But the output of the RETR command is not required to end
in an empty line.  Thus, pop3-movemail should add the empty line (if
it is writing mbox format).

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
| Why incorrectly? This info is not part of the message, why should it not
| be stripped, even if it was present while the server has been storing the
| message?

Because the envelope *IS* part of the message.

| What rfc specifies that a message retrieved via pop3 must have one of
| those things as the first line of the message?

None; this information is defined elsewhere (notably RFC 821).

[...]
| BTW, I just rechecked - neither of my 3 pop3 servers supplies any of
| the magic sequences you're using to second-guess its original storage
| mode.

I'm not trying to second-guess storage modes.  I am trying to generate a
valid message delimiter when one does not already exist.

| Enjoy :) although I don't see how it can be useful. The 'envelope'
| (the first line) has been generated by pop3-movemail anyway.

Thank you.  The mail delivery agent on (I'm assuming) home.com's mail host
is failing to generate Content-Length headers and it is failing to properly
escape lines in message bodies, *AND* your POP server is stripping
envelopes or whatever other message delimiters it might use.  IOW, it looks
like you are being hammered by a broken implementation of SMTP *and* a
broken implementation of POP.

And FWIW, what pop3-movemail puts in your crashbox is identical to what
fetchmail would put there if you used it instead, modulo its version of
faux envelopes.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AkCJgl+vIlSVSNkRAiLiAKCw5GsUx0tZ/N6XliENYi2EM4gFGwCggDpc
WcyW1fQbjggqQuCnLbhtzmA=
=iRZD
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ When not in use, Happy Fun Ball should be
Minion of Nathan - Nathan says Hi! \ returned to its special container and
PGP Key: at a key server near you!  \ kept under refrigeration.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > POP is not allowed to modify message bodies in any way (modulo properly
  > escaped and then de-escaped 'dot lines').  The delivery agent on the mail
  > host must either escape the envelope-like lines or generate a
  > Content-Length header if one does not exist.

Why on earth should the delivery agent on the mail host do that?  It
is not needed by SMTP, it is not needed by POP3, it is *only* needed
by mbox format.  For all pop3-movemail knows, no mbox format has ever
been involved.  So pop3-movemail must escape From_ lines or add the
Content-Length header.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > That a *MESSGE* starts with "From " *IS* sufficient indication
  > that the spoolis in mbox format.  The first line retrieved by the
  > POP RETR command will be either a message delimiter or an RFC 822
  > header.

The first line retrieved by the RETR command is the first line of a
message.  How do you know whether the message comes from
/var/spool/mail/jrl, or from ~jrl/Maildir/42, or from Informix, or
from the Oracle of Delphi?  You don't know that.  It is only a single
message, no more, no less.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
| I think it was. It was pop3-movemail that built an invalid mbox from
| perfectly valid *separate* messages retrieved via pop3 one by one.

I obviously disagree.  It did the best it could with data that was invalid
to begin with.  Garbage in, garbage out.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AkEVgl+vIlSVSNkRAtJfAKDNvfvDceLpNhvwZbNWLPR7/CJMSQCfZlAB
gpg30RUTpzluKKstwaQP3wU=
=t7nc
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Ingredients of Happy Fun Ball include an
Minion of Nathan - Nathan says Hi! \ unknown glowing substance which fell to
PGP Key: at a key server near you!  \ Earth, presumably from outer space.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > I'm not trying to second-guess storage modes.  I am trying to
  > generate a valid message delimiter when one does not already
  > exist.

You should also be escaping superfluous message delimiters when
present.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Wed, 31 Mar 1999
| pop3-movemail writes this message to its output file as-is.

Agreed.  This is an aspect of a later RFC (padding does not exist in
'stock' POP3).  However, not un-escaping the padding is non-destructive.

| As the output file (~/.crashbox) is mbox format, the From_ line in the
| middle will be interpreted as a beginning-of-message by whatever is
| reading the output file of pop3-movemail.

| So, pop3-movemail needs to be fixed.

For the umpteenth time, SMTP requires that a Content-Length header be
generated if envelope-like lines in the message body are not escaped.  The
escaping, if it is required, should be performed long before pop3-movemail
is ever invoked.

| Btw, the mbox format says that a message should be terminated by an
| empty line.  But the output of the RETR command is not required to end
| in an empty line.  Thus, pop3-movemail should add the empty line (if
| it is writing mbox format).

Because pop3-clean-region transforms the POP termination sequence into a
blank line, making it unnecessary to add one.  It used to remove the
terminator entirely, but that caused problems with certain stupid POP
servers that have a penchant for stripping trailing whitespace from
messages.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AkOPgl+vIlSVSNkRAlvXAJ455WiV9RGY/5R6HpRtHDiWIJbTbwCg8/5j
F33fESbKxT+7AxMEi55rUH8=
=xEzW
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ If Happy Fun Ball begins to smoke, get
Minion of Nathan - Nathan says Hi! \ away immediately. Seek shelter and cover
PGP Key: at a key server near you!  \ head.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > For the umpteenth time, SMTP requires that a Content-Length header
  > be generated if envelope-like lines in the message body are not
  > escaped.  The escaping, if it is required, should be performed
  > long before pop3-movemail is ever invoked.

Hm.  This is not in RFC 821, which claims to describe SMTP.  Where is
it?

I went to
http://sunsite.cnlab-switch.ch/cgi-bin/search/standard/nph-findstd?scope=rfc&pattern=content-length&matches=3
and only RFC 2076 seems to be interesting...

It mentions Content-Length as `not Internet Standard' on page 23 or 24.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Christopher K Davis]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> That a *MESSGE* starts with "From " *IS* sufficient indication that the
> spoolis in mbox format.  The first line retrieved by the POP RETR command
> will be either a message delimiter or an RFC 822 header.  It cannot be
> anything else.  If it is an MMDF delimiter, it is an MMDF spool.  If it is
> a Babyl delimiter, it is a Babyl spool.  If it is an RFC 822 header, the
> POP server is broken.

Au contraire.

RFC1939:
11. Message Format

   All messages transmitted during a POP3 session are assumed to conform
   to the standard for the format of Internet text messages [RFC822].

Note that RFC 822 does *not* include delimeters, whether they be Unix
mbox format, MMDF, Babyl, or CrUnChYbItSoFsTuDlYcApS.  So, if any POP
servers are "broken" according to the spec, the ones that are leaving
the delimeters *in* are!

Note that SMTP and POP use dot-stuffing to create protocol-level
delimeters.  Note that maildir format (and, for that matter, nnml ;-)
delimit messages by using the filesystem.  "From ", "From " with a
Content-Length header, ^A^A^A^A, or Babyl delimeters are not the only
possibilities.  Depending on them to be there is probably unwise.

-- 
Christopher Davis * <ckd-sig@ckdhr.com> * <URL:http://www.ckdhr.com/ckd/>
Put location information in your DNS! <URL:http://www.ckdhr.com/dns-loc/>

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> * Kai.Grossjohann@CS.Uni-Dortmund.DE  on Wed, 31 Mar 1999
> | pop3-movemail writes this message to its output file as-is.
> 
> Agreed.  This is an aspect of a later RFC (padding does not exist in
> 'stock' POP3).  However, not un-escaping the padding is non-destructive.
> 
> | As the output file (~/.crashbox) is mbox format, the From_ line in the
> | middle will be interpreted as a beginning-of-message by whatever is
> | reading the output file of pop3-movemail.
> 
> | So, pop3-movemail needs to be fixed.
> 
> For the umpteenth time, SMTP requires that a Content-Length header be
> generated if envelope-like lines in the message body are not escaped.  The
> escaping, if it is required, should be performed long before pop3-movemail
> is ever invoked.

Could you at least once of those umpteenth times quote a relevant
portion of at least one rfc?

> 
> | Btw, the mbox format says that a message should be terminated by an
> | empty line.  But the output of the RETR command is not required to end
> | in an empty line.  Thus, pop3-movemail should add the empty line (if
> | it is writing mbox format).
> 
> Because pop3-clean-region transforms the POP termination sequence into a
> blank line, making it unnecessary to add one.  It used to remove the
> terminator entirely, but that caused problems with certain stupid POP
> servers that have a penchant for stripping trailing whitespace from
> messages.

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Dmitry Yaitskov]

Stainless Steel Rat <ratinox@peorth.gweep.net> wrote:

> * Dmitry Yaitskov <dimas@home.com>  on Tue, 30 Mar 1999
> | I think it was. It was pop3-movemail that built an invalid mbox from
> | perfectly valid *separate* messages retrieved via pop3 one by one.
> 
> I obviously disagree.  It did the best it could with data that was invalid
> to begin with.  Garbage in, garbage out.

Ok. All this argument has taken too much of my (and probably others')
time already. This thread has become riduculously long, so I'm posting
one last time on this topic. No more posts won't mean I agreed with
you, because I do not. Hopefully the xemacs maintainers will
incorporate my patch, otherwise I'll just apply it as new versions
come out. Still, one last time I'll argue with you.

First, your "Garbage in, garbage out" motto is, pardon my Frehcn,
complete bullshit as applied to email. Users shouldn't suffer because
a program reading their mail has its own notion of what is right and
what is wrong, and doesn't care if other programs have different such
notions. It would be at least understandable if it refused the problem
message *and* indicated to the uset that there was a "bad" message,
but it just plain silently screws things up. This is bad.

Also, 3 different pop3 servers that I get mail from provide mail in a
format that *you* consider to be broken in one respect or another.
Even if they were, no reasonable email program should ignore what 3
random unrelated pop3 servers out of 3 do.

Last, you still have not provided a single quote from a single rfc
which says that a message retrieved via pop3 *must* have one of the
magic sequences you're looking for as its first line, and *must* have
"\n\nFrom " lines quoted.

Thank you.

-- 
Cheers,
-Dima.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Wed, 31 Mar 1999
| The first line retrieved by the RETR command is the first line of a
| message.  How do you know whether the message comes from
| /var/spool/mail/jrl, or from ~jrl/Maildir/42, or from Informix, or
| from the Oracle of Delphi?  You don't know that.  It is only a single
| message, no more, no less.

If the first line of what RETR spits out is an SMTP envelope, the mail is
stored remotely in mbox format.  If the first line of what RETR spits out
is an MMDF delimiter, the mail is stored remotely in MMDF format.  If the
first line of what RETR spits out is a Babyl delimiter, the mail is stored
remotely in Babyl format.

If you know of other common storage formats, please let me know what they
are and exactly what they look like so I can include them in pop3's
separator munging exclusion.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AoW1gl+vIlSVSNkRAvn4AKCP6pvPEKZmKtUnbBSiI+wpv3kNPACg4uYT
hFQCzYTlvBDrzy6F0vymyfo=
=BYuU
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Ingredients of Happy Fun Ball include an
Minion of Nathan - Nathan says Hi! \ unknown glowing substance which fell to
PGP Key: at a key server near you!  \ Earth, presumably from outer space.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Christopher K Davis <ckd@ckdhr.com>  on Wed, 31 Mar 1999
| Note that RFC 822 does *not* include delimeters, whether they be Unix
| mbox format, MMDF, Babyl, or CrUnChYbItSoFsTuDlYcApS.

I never said RFC 822 said anything of the sort.  RFC 822 -- the format of
individual messages -- cares not a bit about how messages are stored after
delivery.  That is a requirement of the delivery mechanism.

| So, if any POP servers are "broken" according to the spec, the ones that
| are leaving the delimeters *in* are!

POP falls under delivery mechanism in the grand scheme of things.

| Note that SMTP and POP use dot-stuffing to create protocol-level
| delimeters.

And again, I said as much and never otherwise.  Escaping envelope-like
lines is not a protocol-level frob, which is why I resist that idea.

| Note that maildir format (and, for that matter, nnml ;-) delimit messages
| by using the filesystem.  "From ", "From " with a Content-Length header,
| ^A^A^A^A, or Babyl delimeters are not the only possibilities.  Depending
| on them to be there is probably unwise.

Which is why pop3-munge-message-separator exists at all.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AocLgl+vIlSVSNkRAnrCAJ9LjURXQcyQHfjDn1RVmDnQhKKsgACgm7mH
pPq02MVlf0q33SW/mEyJ7gg=
=gaji
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Warning: pregnant women, the elderly, and
Minion of Nathan - Nathan says Hi! \ children under 10 should avoid prolonged
PGP Key: at a key server near you!  \ exposure to Happy Fun Ball.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Wed, 31 Mar 1999
| It mentions Content-Length as `not Internet Standard' on page 23 or 24.

On the one hand, it is a defacto standard.  On the other, because it is not 
an official standard, SMTP software should escape anything that might break 
SMTP software, such as lines in the body of a message that look like they
might be envelopes.

In neither case is it a POP server or client's responsibility.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3Aoddgl+vIlSVSNkRAl7DAKDGg59CEK/uR3T8eFiC3d74kJz73wCdEOpa
WoXF54+ZyJdTkD3YwPnHp1k=
=aFqd
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Christopher K Davis]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:
> Christopher K Davis <ckd@ckdhr.com>  on Wed, 31 Mar 1999
>> Note that RFC 822 does *not* include delimeters, whether they be Unix
>> mbox format, MMDF, Babyl, or CrUnChYbItSoFsTuDlYcApS.

> I never said RFC 822 said anything of the sort.

You said, and I quote:
> The first line retrieved by the POP RETR command will be either a
> message delimiter or an RFC 822 header.  It cannot be anything else.
> If it is an MMDF delimiter, it is an MMDF spool.  If it is a Babyl
> delimiter, it is a Babyl spool.  If it is an RFC 822 header, the POP
> server is broken.

According to the POP3 specification in RFC 1939, RETR returns a message
(page 8).  Also according to the POP3 specification in RFC 1939, messages
transferred are assumed to conform to RFC 822 (page 19).  This language
does not appear to have changed since at least RFC 1460.

RFC 822 format does not include message separators.  Therefore, any
message returned by the RETR command containing a message separator is
not in compliance with RFC 822, and therefore violates the POP spec;
messages *not* containing a separator, which means their first line will
be an RFC 822 header, are *in compliance with the POP specification*.

A POP server which sends messages in compliance with the POP
specification is not, to most people, considered 'broken' (at least in
that respect :-).

-- 
Christopher Davis * <ckd-sig@ckdhr.com> * <URL:http://www.ckdhr.com/ckd/>
Put location information in your DNS! <URL:http://www.ckdhr.com/dns-loc/>

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Christopher K Davis <ckd@ckdhr.com>  on Wed, 31 Mar 1999
| You said, and I quote:
| > The first line retrieved by the POP RETR command will be either a
| > message delimiter or an RFC 822 header.  It cannot be anything else.
| > If it is an MMDF delimiter, it is an MMDF spool.  If it is a Babyl
| > delimiter, it is a Babyl spool.  If it is an RFC 822 header, the POP
| > server is broken.

And where in that do I say that the RFC 822 header is a delimiter of any
sort?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3AsbWgl+vIlSVSNkRAtvKAKDapYcqps+/zU54bj1+t0fVB7xGoACg4eUF
hr1X3GKba5gC/iIjJs18igU=
=eAs+
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Happy Fun Ball contains a liquid core,
Minion of Nathan - Nathan says Hi! \ which, if exposed due to rupture, should
PGP Key: at a key server near you!  \ not be touched, inhaled, or looked at.

Re: bad (i.e. serious) mail problems

[Russ Allbery]

Frank D Cringle <fdc@cliwe.ping.de> writes:

> Right.  That is the case for qmail, where Mail destined to be collected
> via POP is typically delivered to a Maildir (one file per message).  The
> envelope is stored in the first 2 lines:

>   From <envelope-sender>
>   Delivered-To: recipient
>   Received: ... blah...
>   ... etc ...

The maildir format does not add a "From " line.  From maildir(5):

     Each file in new is a newly delivered mail message.  The modification
     time of the file is the delivery date of the message.  The message is
     delivered without an extra UUCP-style From_ line, without any >From
     quoting, and without an extra blank line at the end.  The message is
     normally in RFC 822 format, starting with a Return-Path line and a
     Delivered-To line, but it could contain arbitrary binary data.  It
     might not even end with a newline.

-- 
Russ Allbery (rra@stanford.edu)         <URL:http://www.eyrie.org/~eagle/>

Re: bad (i.e. serious) mail problems

[Russ Allbery]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> On the one hand, it is a defacto standard.  On the other, because it is
> not an official standard, SMTP software should escape anything that
> might break SMTP software, such as lines in the body of a message that
> look like they might be envelopes.

I take great exception to the contention that my MTA should arbitrarily
munge messages just because some SMTP software uses a poorly designed and
badly implemented message delimiter structure.  The only reason why my
mail messages ever touch mbox format currently is because I'm still using
an old version of Gnus that can't read maildir native.

Both SMTP and POP have well-defined, unambiguous, and *reversible*
protocols for indicating to the remote system where the end of the message
is.  Adding on top of that an ambiguous protocol requiring an irreversible
escaping to not break, that is then parsed using a lot of really touchy
parsing that's not consistent across different MUAs, is broken.
Fundamentally and totally broken.  The fact that it's been broken in that
way for many years and that a lot of different software implements that
breakage is not an excuse.

Lots of mail software can now deliver to maildirs (including, IIRC, Exim
and Postfix).  Cyrus IMAP servers use another mail spool format that
similarly avoids escaping issues.  The number of servers that can handle
mail without doing broken >From escaping is growing and will continue to
grow.

-- 
Russ Allbery (rra@stanford.edu)         <URL:http://www.eyrie.org/~eagle/>

Re: bad (i.e. serious) mail problems

[Christopher K Davis]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

>>> The first line retrieved by the POP RETR command will be either a
>>> message delimiter or an RFC 822 header.  It cannot be anything else.
>>> If it is an MMDF delimiter, it is an MMDF spool.  If it is a Babyl
>>> delimiter, it is a Babyl spool.  If it is an RFC 822 header, the POP
>>> server is broken.

> And where in that do I say that the RFC 822 header is a delimiter of any
> sort?

You don't, and that wasn't what I said you said either.

Let's go through it again.

>>> The first line retrieved by the POP RETR command will be either a
>>> message delimiter or an RFC 822 header.  It cannot be anything else.

(Well, it could be, but I think we both agree that that would be either
a munged message or a broken server, so we'll ignore that possibility.)

>>> If it is an MMDF delimiter, it is an MMDF spool.  If it is a Babyl
>>> delimiter, it is a Babyl spool.

(I could postulate a server that stores messages in maildirs and has a
bug which generates for 0x01 bytes before each message, but again, that
would be silly.)

>>> If it is an RFC 822 header, the POP server is broken.

This sentence is the one which I disagree with.

Once again:

1. The POP RFCs (1939 and, for that matter, 1460, which you cite in
   pop3.el) state, quite clearly, that messages transferred are assumed
   to conform to RFC 822.

2. RETR transfers a message.

3. (1 + 2) Therefore, what RETR transfers is assumed to conform to RFC 822.

4. Messages conforming with RFC 822 do not include a delimeter.

5. (3 + 4) Therefore, what RETR transfers is assumed to not include a
   delimeter.

6. (follows from 5) If the first line retrieved from the POP RETR
   command is not a delimeter, the implementation is conformant with
   that portion of the specification.

-- 
Christopher Davis * <ckd-sig@ckdhr.com> * <URL:http://www.ckdhr.com/ckd/>
Put location information in your DNS! <URL:http://www.ckdhr.com/dns-loc/>

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > On the one hand, it is a defacto standard.  On the other, because it is not 
  > an official standard, SMTP software should escape anything that might break 
  > SMTP software, such as lines in the body of a message that look like they
  > might be envelopes.

Escaping From_ lines is necessary only for software which uses mbox
format.  Thus, software which uses mbox format should take care to
escape From_ lines.

pop3-movemail is a function which uses mbox format (as its output), so
it should escape From_ lines.

pop3-movemail should not depend on the fact that the SMTP
implementation or the POP3 server use mbox format anywhere -- that is
not at all a given.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Christopher K Davis <ckd@ckdhr.com> writes:

  > RFC 822 format does not include message separators.  Therefore, any
  > message returned by the RETR command containing a message separator is
  > not in compliance with RFC 822, and therefore violates the POP spec;
  > messages *not* containing a separator, which means their first line will
  > be an RFC 822 header, are *in compliance with the POP specification*.

RFC 821 has something to say on the format of message bodies.  It says
a message body is an arbitrary sequence of ASCII characters, except
that CRLF is forbidden (CR alone or LF alone are allowed, though).  As
far as I understand, the sequence <LF> F r o m <SPC> is not forbidden
according to this rule.

The POP3 RFC explicitly says that the output of a RETR command is one
message.  Thus, it cannot contain a message separator -- it is just
one message, after all.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > If the first line of what RETR spits out is an SMTP envelope, the
  > mail is stored remotely in mbox format.

No.  This is just guessing and may be right or wrong.

Anyway, POP3 says that messages are in RFC 822 format, and something
which begins with From_ is not conforming to RFC 822.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Christopher K Davis <ckd@ckdhr.com> writes:

  > According to the POP3 specification in RFC 1939, RETR returns a message
  > (page 8).

So any string in the message must not be interpreted as the end of the
message.  So From_ must not be interpreted as the end of the message.

What pop3-movemail does amounts to interpreting From_ as the end of
the message (the beginning of a new one, actually, but you know what I
mean).

In particular, RFC 821 allows any sequence of ASCII characters in the
body of a message, except <CR><LF> (<CR> alone or <LF> alone are
allowed).  Thus, <LF> F r o m <SPC> is allowed to be part of the body
of a message.

When such a message is written to an mbox format file, conversion must
be done because mbox format says that such a line is not allowed to be
part of the body of a message.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Thu, 01 Apr 1999
| What pop3-movemail does amounts to interpreting From_ as the end of
| the message (the beginning of a new one, actually, but you know what I
| mean).

This is nonsense, as you would see if you looked at the source code.  pop3
sees a message as a message, exactly as the server presents it.  pop3
generates a delimiter at the *beginning* of the message it is currently
dealing with only when no delimiter exists.

Now, if you want to argue the choce of an mbox-style delimiter being a bad
one, I would agree -- mbox is lousy.  The only reason I chose to do things
that way is because at the time I wrote it I was using a modified version
of GNUS that did not comprehend anything but mbox-style mailboxes.  Broken
though it might be, it is still the most widely used mail storage scheme.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3A+3Zgl+vIlSVSNkRAhwKAKC876jvqM0kzfOriUaA8VBhP25OSgCdHSe5
W4JCladlGp8682jnY/LS3Rw=
=NVEg
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Do not taunt Happy Fun Ball.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Hans de Graaff]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> * Kai.Grossjohann@CS.Uni-Dortmund.DE  on Thu, 01 Apr 1999
> | What pop3-movemail does amounts to interpreting From_ as the end of
> | the message (the beginning of a new one, actually, but you know what I
> | mean).
> 
> This is nonsense, as you would see if you looked at the source code.  pop3
> sees a message as a message, exactly as the server presents it.  pop3
> generates a delimiter at the *beginning* of the message it is currently
> dealing with only when no delimiter exists.

I agree, and this also makes sense. What I perceive as the issue that
people are discussing is that you should in this case also make sure
that there are no other From_ lines in the message you are writing
out, because those will be interpreted by any mbox-reading application 
as additional, if erroneous, message delimiters. 

For example (and I know I'm not getting all the details and formats of 
the headers right, but that doesn't matter for the argument):

Message 1 is received from POP:

----
From: someone
Subject: none

This is a message that has a line that starts with
>From .
And an extra line to boot.
----

The message contains no From_ line at the start, so you add one, and
write the complete message to the mbox. Then message 2 comes in.

----
>From someone at some date
From: xyzzy
Subject: all

Another message
----

This message does contain a From_ line, so you just write the full
message to the mbox file. Let's now look at the mbox file:

----
*From* someone at some date
From: someone
Subject: none

This is a message that has a line that starts with
*From* .
And an extra line to boot.
*From* someone at some date
From: xyzzy
Subject: all

Another message
----

Instead of having two message delimiters of the form From_, there are
three. This happens because the From_ line in the body of the first
message is not escaped when it is written into an mbox style folder.

> Now, if you want to argue the choce of an mbox-style delimiter being a bad
> one, I would agree -- mbox is lousy.

:-) I don't think anybody would want to argue about /that/.

Hans

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > * Kai.Grossjohann@CS.Uni-Dortmund.DE  on Thu, 01 Apr 1999
  > | What pop3-movemail does amounts to interpreting From_ as the end of
  > | the message (the beginning of a new one, actually, but you know what I
  > | mean).
  > 
  > This is nonsense, as you would see if you looked at the source code.  pop3
  > sees a message as a message, exactly as the server presents it.  pop3
  > generates a delimiter at the *beginning* of the message it is currently
  > dealing with only when no delimiter exists.

pop3-movemail does not interpret the internals of a message in any
way.  In this sense, what I wrote was nonsense.  But pop3-movemail
assumes that From_ lines are escaped (or the Content-Length header is
present) _in its input_, even though there is *no* reason whatsoever
to think that the input is in mbox format.

kai
-- 
I'd like to welcome you and an orange juice, please.

Re: bad (i.e. serious) mail problems

[Lars Magne Ingebrigtsen]

Shane Holder <holder@rsn.hp.com> writes:

> Not quite true.
> >From nnfolder-save-mail:
> 
>     (unless (looking-at message-unix-mail-delimiter)
>       (insert "From nobody-nnfolder-2 " (current-time-string) "\n")
>       (goto-char (point-min)))

Yup.  Fix in Pterodactyl Gnus v0.81.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bad (i.e. serious) mail problems

[Lars Magne Ingebrigtsen]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> This is nonsense, as you would see if you looked at the source code.  pop3
> sees a message as a message, exactly as the server presents it.  pop3
> generates a delimiter at the *beginning* of the message it is currently
> dealing with only when no delimiter exists.

Richard -- pop3 clearly does the wrong thing here.  The pop server is
not honor-bound (or even supposed to) quote Unix mail folder
delimiters in the body of the message, and when you are generating a
Unix mail folder based on the output of RETR, you have to quote those
lines.  Or generate something else than Unix mail folders.  MMDF, for
instance. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Okay, so...

Modifying the message body is a Bad Thing, so there is no way I am going to 
destructively modify it (ie, no escaping of envelope-like lines).

Is generating a Content-Length header if one does not exist a bad idea?

Or is there a better choice for delimiting mesages in the crashbox, one
that is universally grokked by Emacs mail clients?  MMDF-style delimiters
would be my choice.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3BQO5gl+vIlSVSNkRArEqAJ48KKoYyZDsx+7lLJ6uha5vtEHq0wCgwQ73
ogBbnxA0gqA3wceAfSP1IgE=
=jHpo
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Caution: Happy Fun Ball may suddenly
Minion of Nathan - Nathan says Hi! \ accelerate to dangerous speeds.
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Kai.Grossjohann]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

  > Is generating a Content-Length header if one does not exist a bad idea?

I think it's a wonderful idea!  Can't imagine what would go wrong with
that.  I'm not sure if all Emacs mail clients grok the header,
though.  Hm.  I guess mh-e has its own POP mechanism (the MH one),
right?  So we're left with mew, VM, Rmail and Gnus.  I think.

kai
-- 
Abort this operation?   [Abort]  [Cancel]

Re: bad (i.e. serious) mail problems

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

* Stainless Steel Rat <ratinox@peorth.gweep.net> writes:
| Is generating a Content-Length header if one does not exist a bad idea?

* Kai.Grossjohann@CS.Uni-Dortmund.DE  on Sat, 03 Apr 1999
| I think it's a wonderful idea!

Just one of those solutions that reared up and bit me on the behind
yesterday morning.  Shouldn't be too hard to do it, right? :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v0.9.5 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE3BhmYgl+vIlSVSNkRApGiAKDJqG8+p1mOizKGF1RhSafHjVoqoQCg6Whr
kcnTswYhTDXwWT28Y7erq/w=
=qFNJ
-----END PGP SIGNATURE-----

-- 
Rat <ratinox@peorth.gweep.net>    \ Happy Fun Ball may stick to certain types
Minion of Nathan - Nathan says Hi! \ of skin.
PGP Key: at a key server near you!  \ 

Re: bad (i.e. serious) mail problems

[Lars Magne Ingebrigtsen]

Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

> Is generating a Content-Length header if one does not exist a bad idea?

No, that would be OK.

> Or is there a better choice for delimiting mesages in the crashbox, one
> that is universally grokked by Emacs mail clients?  MMDF-style delimiters
> would be my choice.

MMDF seems like the nicest choice, but I don't think all Emacs mail
clients grok that one...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen