digests and MIME

digests and MIME

[Colin Rafferty]

Howdy-

It seems that MIME encoded messages that are read as digests are not
being decoded before they are split.  I think that they should be.

An example is the FAQ that Steven L. Baur (RIP) just posted.  It is
formatted incorrectly if it has not been decoded.

What do people think?

-- 
Colin Rafferty

Re: digests and MIME

[Steven L Baur]

>>>>> "Colin" == Colin Rafferty <craffert@spspme.ml.com> writes:

Colin> Howdy-

Colin> It seems that MIME encoded messages that are read as digests
Colin> are not being decoded before they are split.  I think that they
Colin> should be.

Colin> An example is the FAQ that Steven L. Baur (RIP) just posted.

I'm not dead yet. It's getting better. :-)

Colin> It is formatted incorrectly if it has not been decoded.

You're getting it transmogrified into QP I presume?

Colin> What do people think?

A better solution is to bug administrators to fix their news and mail
relays so they don't make the broken conversion 8bit -> QP.  I don't
think there's a `correct' thing to do when the message has been
corrupted in that fashion, except perhaps to post the FAQ as a MIME
digest in the first place.

-- 
steve@miranova.com baur
Unsolicited commercial e-mail will be billed at $250/message.
"Bill Clinton is a bore.  He doesn't have a creative bone in his
body."  -- David Brinkley

Re: digests and MIME

[Colin Rafferty]

>>>>>> "Steven" == Steven L Baur <steve@miranova.com> writes:
>>>>>> "Colin" == Colin Rafferty <craffert@spspme.ml.com> writes:

Colin> It seems that MIME encoded messages that are read as digests
Colin> are not being decoded before they are split.  I think that they
Colin> should be.

Colin> An example is the FAQ that Steven L. Baur (RIP) just posted.

Steven> I'm not dead yet. It's getting better. :-)

Colin> It is formatted incorrectly if it has not been decoded.

Steven> You're getting it transmogrified into QP I presume?

Yes I am.  See firther down for the relevent headers.

Colin> What do people think?

Steven> A better solution is to bug administrators to fix their news and
Steven> mail relays so they don't make the broken conversion 8bit -> QP.
Steven> I don't think there's a `correct' thing to do when the message
Steven> has been corrupted in that fashion, except perhaps to post the
Steven> FAQ as a MIME digest in the first place.

I have to disagree with this.  Some people will intentionally post
digests as MIME messages, the same way that they would post normal
message as MIME.  We should be able to handle this correctly.

As an example, you posted yours that way.  I received it here through
ding@ifi.  Here are the relevent headers from the message:

FAQ> Sender: steve@deanna.miranova.com
FAQ> Subject: Gnus (Emacs Newsreader) FAQ
FAQ> Message-ID: <m2hgmlphdj.fsf@deanna.miranova.com>
FAQ> Mime-Version: 1.0 (generated by tm-edit 7.94)
FAQ> X-Newsreader: Red Gnus v0.65/XEmacs 19.15
FAQ> Content-Transfer-Encoding: quoted-printable
FAQ> Content-Type: text/plain; charset=ISO-8859-1
FAQ> X-Content-Length: 47855

By the way, you have done a great job with the FAQ.  I think that it
probably should be sent in MIME.  It is just that Gnus should insert the
decoded article into " *gnus digest buffer*" rather than the original
article.

-- 
Colin

Re: digests and MIME

[Lars Magne Ingebrigtsen]

Colin Rafferty <craffert@spspme.ml.com> writes:

>  It is just that Gnus should insert the decoded article into " *gnus
> digest buffer*" rather than the original article.

Some of the article treatments are destructive, so I don't know
whether that would help much in general.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: digests and MIME

[Steven L Baur]

>>>>> "Colin" == Colin Rafferty <craffert@spspme.ml.com> writes:

>>>>> "Steven" == Steven L Baur <steve@miranova.com> writes:

Colin> It is formatted incorrectly if it has not been decoded.

Steven> You're getting it transmogrified into QP I presume?

Colin> Yes I am.  See firther down for the relevent headers.

 ...

Colin> I have to disagree with this.  Some people will intentionally post
Colin> digests as MIME messages, the same way that they would post normal
Colin> message as MIME.  We should be able to handle this correctly.

Colin> As an example, you posted yours that way.  I received it here through
Colin> ding@ifi.  Here are the relevent headers from the message:

FAQ> Mime-Version: 1.0 (generated by tm-edit 7.94)
FAQ> Content-Transfer-Encoding: quoted-printable
FAQ> Content-Type: text/plain; charset=ISO-8859-1

It's a problem with your mailer, or something between you and
ifi.uio.no. 

The way I posted it *and* the way I received it:

Mime-Version: 1.0 (generated by tm-edit 7.94)
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

-- 
steve@miranova.com baur
Unsolicited commercial e-mail will be billed at $250/message.
"Bill Clinton is a bore.  He doesn't have a creative bone in his
body."  -- David Brinkley

Re: digests and MIME

[Colin Rafferty]

Lars Magne Ingebrigtsen writes:
> Colin Rafferty <craffert@spspme.ml.com> writes:

>> It is just that Gnus should insert the decoded article into " *gnus
>> digest buffer*" rather than the original article.

> Some of the article treatments are destructive, so I don't know
> whether that would help much in general.

I agree about the other article treatments, but MIME is special.

While all other treatments are things we do because we like to make our
computers work hard, MIME is something that we do because the mail
sender tells us to.

Now that I think about it, I would even go as far as saying that nnmail
(the final delivery agent) should store the articles in transmogrified
form, since that is the intention of the sender.

-- 
Colin

Re: digests and MIME

[Colin Rafferty]

>>>>>> "Colin" == Colin Rafferty <craffert@spspme.ml.com> writes:
>>>>>> "Steven" == Steven L Baur <steve@miranova.com> writes:

Colin> It is formatted incorrectly if it has not been decoded.

Steven> You're getting it transmogrified into QP I presume?

Colin> Yes I am.  See firther down for the relevent headers.

Colin> As an example, you posted yours that way.  I received it here through
Colin> ding@ifi.  Here are the relevent headers from the message:

FAQ> Mime-Version: 1.0 (generated by tm-edit 7.94)
FAQ> Content-Transfer-Encoding: quoted-printable
FAQ> Content-Type: text/plain; charset=ISO-8859-1

Steven> It's a problem with your mailer, or something between you and
Steven> ifi.uio.no. 

Steven> The way I posted it *and* the way I received it:

Steven> Mime-Version: 1.0 (generated by tm-edit 7.94)
Steven> Content-Type: text/plain; charset=ISO-8859-1
Steven> Content-Transfer-Encoding: 8bit

I will have to try to find the offending (offensive?) relay site.
Looking through my archive, I have seen headers of the form:

HDR> X-MIME-Autoconverted: from 8bit to quoted-printable by relay.domain.com

At least these relays are telling us that they have done something evil.

On the other hand, I still think that my points about de-miming before
undigestifying are valid. :-)

-- 
Colin

Re: digests and MIME

[Steven L Baur]

>>>>> "Colin" == Colin Rafferty <craffert@spspme.ml.com> writes:

 ...

Colin> I will have to try to find the offending (offensive?) relay site.
Colin> Looking through my archive, I have seen headers of the form:

HDR> X-MIME-Autoconverted: from 8bit to quoted-printable by relay.domain.com

I've seen some stuff from Lars' new machine come through like that :-(.

What I like about sendmail 8.8 configuration is this:

Received: (from slist@localhost) by riker.miranova.com (8.8.3/8.8.3) id MAA19791 for steve@miranova.com; Tue, 19 Nov 1996 12:04:45 -0800
 ...
X-MIME-Autoconverted: from quoted-printable to 8bit by riker.miranova.com id LAA18920

That's the way I prefer to see this problem solved :-).

Colin> At least these relays are telling us that they have done
Colin> something evil.

Yup.

Colin> On the other hand, I still think that my points about de-miming
Colin> before undigestifying are valid. :-)

They are indeed valid, we just disagree on where it should be done.
-- 
steve@miranova.com baur
Unsolicited commercial e-mail will be billed at $250/message.
"Bill Clinton is a bore.  He doesn't have a creative bone in his
body."  -- David Brinkley

Re: digests and MIME

[Lars Magne Ingebrigtsen]

Colin Rafferty <craffert@spspme.ml.com> writes:

> I agree about the other article treatments, but MIME is special.
> 
> While all other treatments are things we do because we like to make our
> computers work hard, MIME is something that we do because the mail
> sender tells us to.

Yes, but.  There are many other kinds of encoding -- PEM, PGP, uhm,
well -- things that will not only add headers and footers, but which
will mangle the text in between as well.  Perhaps what we need is to
divide `gnus-article-display-hook' into two parts -- one which does
unencoding, and one which makes the article look all nice an pretty?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: digests and MIME

[Lars Magne Ingebrigtsen]

Steven L Baur <steve@miranova.com> writes:

> HDR> X-MIME-Autoconverted: from 8bit to quoted-printable by relay.domain.com
> 
> I've seen some stuff from Lars' new machine come through like that :-(.

For all I know, it may be my screwed-up sendmail config that messes
things up...  No, wait, it can't be, because my machine doesn't touch
mail that goes through ifi.  Let's see...  The sendmail at ifi *will*
encode things that "need" encoding, unless there already is a
"Mime-version: 1.0" header in there.  Perhaps it doesn't like those
"Mime-Version: 1.0 (generated by tm-edit 7.94)" headers?  Does the
MIME standard allow that?

> What I like about sendmail 8.8 configuration is this:
> 
> Received: (from slist@localhost) by riker.miranova.com (8.8.3/8.8.3) id MAA19791 for steve@miranova.com; Tue, 19 Nov 1996 12:04:45 -0800
>  ...
> X-MIME-Autoconverted: from quoted-printable to 8bit by riker.miranova.com id LAA18920
> 
> That's the way I prefer to see this problem solved :-).

Indeed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: digests and MIME

[Steven L Baur]

>>>>> "Colin" == Colin Rafferty <craffert@spspme.ml.com> writes:

Colin> Now that I think about it, I would even go as far as saying
Colin> that nnmail (the final delivery agent) should store the
Colin> articles in transmogrified form, since that is the intention of
Colin> the sender.

We already have ample hooks available to do transformations on
incoming messages.  My favorite is `nnml-prepare-save-mail-hook',
which I use to strip mailing list tags out of Subject: lines.

Other than debugging MIME software, I can't think of any reason why
you would want to store a message in quoted-printable though.
-- 
steve@miranova.com baur
Unsolicited commercial e-mail will be billed at $250/message.
"Bill Clinton is a bore.  He doesn't have a creative bone in his
body."  -- David Brinkley

Re: digests and MIME

[Hrvoje Niksic]

Steven L. Baur (steve@miranova.com) wrote:
> We already have ample hooks available to do transformations on
> incoming messages.  My favorite is `nnml-prepare-save-mail-hook',
> which I use to strip mailing list tags out of Subject: lines.
> 
> Other than debugging MIME software, I can't think of any reason why
> you would want to store a message in quoted-printable though.

This is the case only of quoted-printable (which is an example of
transfer-encoding).  Other than that, I *do* want to have MIME
messages saved as MIME messages, without additional processing.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Hocemo 101-icu!
--------------------------------+--------------------------------
I'm a Lisp variable -- bind me!

Re: digests and MIME

[Jacob Morzinski]

Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
> For all I know, it may be my screwed-up sendmail config that messes
> things up...  No, wait, it can't be, because my machine doesn't touch
> mail that goes through ifi.  Let's see...  The sendmail at ifi *will*
> encode things that "need" encoding, unless there already is a
> "Mime-version: 1.0" header in there.  Perhaps it doesn't like those
> "Mime-Version: 1.0 (generated by tm-edit 7.94)" headers?  Does the
> MIME standard allow that?


My reading of the RFC's suggests that it does allow it.  Here is an
excerpt from RFC 1521, which seems to be the one that applies in this
case:

[begin excerpt]

   Therefore, this document defines a new header field, "MIME-Version",
   which is to be used to declare the version of the Internet message
   body format standard in use.

   Messages composed in accordance with this document MUST include such
   a header field, with the following verbatim text:

   MIME-Version: 1.0

   The presence of this header field is an assertion that the message
   has been composed in compliance with this document.

   Since it is possible that a future document might extend the message
   format standard again, a formal BNF is given for the content of the
   MIME-Version field:

   version := "MIME-Version" ":" 1*DIGIT "." 1*DIGIT

[..snip..]

   NOTE TO IMPLEMENTORS: All header fields defined in this document,
   including MIME-Version, Content-type, etc., are subject to the
   general syntactic rules for header fields specified in RFC 822.  In
   particular, all can include comments, which means that the following
   two MIME-Version fields are equivalent:

                    MIME-Version: 1.0
                    MIME-Version: 1.0 (Generated by GBD-killer 3.7)

[end excerpt]


The Mime-version field was first defined in RFC 1341.  I didn't find
any more recent RFC's, overriding the header definition given here.

(Of couse, the sendmail at ifi might not have been taught to ignore
comments in the Mime-version header.)

-- 
 Jacob Morzinski                                jmorzins@mit.edu

Re: digests and MIME

[François Pinard]

Steven L Baur <steve@miranova.com> writes:

| Steven> You're getting it transmogrified into QP I presume?

| It's a problem with your mailer, or something between you and
| ifi.uio.no. 

I do not want to be heavy telling things that everyone knows.  But just
in case, maybe :-).  I apologise to the others.

Transmogrification might occur whenever a modern sendmail tries to
establish an SMTP connection with some sendmail daemon which refuses to
reply to the ESMTP protocol (defined in RFC 1652, I think).  But if the
ESMTP handshake completes, transfer will occur in full 8bits.

So, the solution might be that the receiving end of a transmogrified
transfer update their sendmail to *say* that it can handle 8bits.

-- 
François Pinard         ``Vivement GNU!''        pinard@iro.umontreal.ca
Support Programming Freedom, join our League!  Ask lpf@lpf.org for info!

Re: digests and MIME

[Lars Magne Ingebrigtsen]

pinard@progiciels-bpi.ca (François Pinard) writes:

> Transmogrification might occur whenever a modern sendmail tries to
> establish an SMTP connection with some sendmail daemon which refuses to
> reply to the ESMTP protocol (defined in RFC 1652, I think).  But if the
> ESMTP handshake completes, transfer will occur in full 8bits.

Yes...  This is from the sendmail man page:

     EightBitMode=mode
                 Set the handling of eight bit input to seven bit destinations
                 to mode: m (mimefy) will convert to seven-bit MIME format, p
                 (pass) will pass it as eight bits (but violates protocols),
                 and s (strict) will bounce the message.

And my sendmail.cf (by "default") says:

# 8-bit data handling
O EightBitMode=pass8

But it still transmogrifies into qp when talking to ifi.uio.no.
Anybody have any ideas why?  Or perhaps is should be "p" verbatim?
Hm, I'll try to change it to "p"...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: digests and MIME

[Steven L Baur]

>>>>> "Lars" == Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:
X-MIME-Autoconverted: from 8bit to quoted-printable by proletcult.slip.ifi.uio.no id RAA08333
 ** blast **  Take that sendmail.

Lars> And my sendmail.cf (by "default") says:

Lars> # 8-bit data handling
Lars> O EightBitMode=pass8

Lars> But it still transmogrifies into qp when talking to ifi.uio.no.
Lars> Anybody have any ideas why?  Or perhaps is should be "p" verbatim?
Lars> Hm, I'll try to change it to "p"...

What about the option SevenBitInput?  I see looking at sendmail.cf
(from 8.8.3):

# strip message body to 7 bits on input?
O SevenBitInput=False

# 8-bit data handling
O EightBitMode=pass8

-- 
steve@miranova.com baur
Unsolicited commercial e-mail will be billed at $250/message.
"Bill Clinton is a bore.  He doesn't have a creative bone in his
body."  -- David Brinkley

Re: digests and MIME

[Lars Magne Ingebrigtsen]

Steven L Baur <steve@miranova.com> writes:

> What about the option SevenBitInput?

# strip message body to 7 bits on input?
O SevenBitInput=False

Looks like we have the same sendmail.cf.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen