Re: [9fans] mime

Re: [9fans] mime

[forsyth]

>>nope.
>>that's the problem.  somewhere a gateway stripped out 'invalid' chars
>>(didn't even turn the e-acute into an 'i' which is what i would have
>>expected)

i think it's demon's smtp mailer that does it.  i lose utf-8 in messages sent
between here and home and both are on demon.

220 finch-post-10.mail.demon.net ESMTP Wed, 22 Nov 2000 15:12:40 +0000

Re: [9fans] mime

[Boyd Roberts]

From: <forsyth@vitanuova.com>

> i think it's demon's smtp mailer that does it.  i lose utf-8 in messages sent
> between here and home and both are on demon.

i got it back fine from 9fans.

Re: [9fans] mime

[Boyd Roberts]

my reliable source says:

    rfc821 says strip the 8th bit.
    rfc822 says strip the 8th bit.

    to many people, this was inconvenient
    so they didn't implement it.

    i implemented it.

brian reid supported the decision.

btw: i'm ex-digital [research].  i worked with the gateway guys,
     even though i was in the paris research lab.

Re: [9fans] mime

[rog]

> nah, i've done some good work in that pub [The Black Swan].

or was it the old white swan (goodramgate) or the swan (bishy
road, my fave), or the other white swan (blossom st) ?

lots of swans to choose from in york, those that haven't been
swept away by the floods....

Re: [9fans] mime

[Boyd Roberts]

From: <rog@vitanuova.com>
>
> > nah, i've done some good work in that pub [The Black Swan].
> 
> or was it the old white swan (goodramgate) or the swan (bishy
> road, my fave), or the other white swan (blossom st) ?

must be the old one.  it's a tudor building, near a gate.

now if they had been in The White Swan we may never have
had mime :-)

Re: [9fans] mime

[presotto]

	However, I'm still perplexed as to what to put in the header
	of a multipart message.  Does the transfer encoding specified
	there pertain to the whole message or just to the

		nyah nyah, if you only had a MIME mailer you wouldn't see this

	message that starts every multipart message?

Just answered my own question.  Rfc2045 says:

	6.4.  Interpretation and Use

	If a Content-Transfer-Encoding header field appears as part of a
	message header, it applies to the entire body of that message.  If a
	Content-Transfer-Encoding header field appears as part of an entity's
	headers, it applies only to the body of that entity.  If an entity is
	of type "multipart" the Content-Transfer-Encoding is not permitted to
	have any value other than "7bit", "8bit" or "binary".

That makes my life much easier.  Someone must have been thinking.

Re: [9fans] mime

[presotto]

[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]

You don't have to know the transfer encoding when parsing the header.
According to 822,

	Each header field can be viewed as a single, logical  line  of
	ASCII  characters,  comprising  a field-name and a field-body.

Of course 821 states

	The maximum total length of a text line including the
	<CRLF> is 1000 characters (but not counting the leading
	dot duplicated for transparency).

To stay consistent with these, headers need to be in the
7bit domain.  However, rfc2047 provides the amazing
=?b?gobbledeygook?= syntax to allow non ascii tokens in the
header.

Therefore, the 'Content-Transfer-Encoding: {78}bit' header field
isn't telling you anything per se about the header since the
headers are predefined to be US-ASCII with the =???=
encoding providing an escape in case you want what's there
to be convertible to non-ASCII.

When upas/marshal says that a message (or message portion)
is either 7 or 8 bit, it's just stating a fact.  However, if
smtp sees an 8bit message and doesn't get an esmtp connection
to the remote host, it should encode the message (or message
part) in base64 (or quoted printable).

We either have to bite the bullet and encode all 8-bit data
presented to marshal OR change smtp to do the recoding OR
both.  However, if we're also willing to relay messages (which
I do) than the marshal change is insufficient since mail may
pass through us from an 8bit to a 7bit channel without passing
marshal.

Therefore, I'm going to make smtp (or a front end to it)
convert and you can all stop your whining.

However, I'm still perplexed as to what to put in the header
of a multipart message.  Does the transfer encoding specified
there pertain to the whole message or just to the

	nyah nyah, if you only had a MIME mailer you wouldn't see this

message that starts every multipart message?

[-- Attachment #2: Type: message/rfc822, Size: 2026 bytes --]

From: rog@vitanuova.com
To: 9fans@cse.psu.edu
Subject: Re: [9fans] mime
Date: Wed, 22 Nov 2000 13:12:38 0000
Message-ID: <20001122121028.38BD3199E1@mail.cse.psu.edu>

> What brain damaged piece of dog
> excrement program would throw away the 8th bit of every
> byte in today's world?

annoying thing is that you don't know what the 8th bit means until
you've parsed the Content-transfer-encoding header.  it would be nice
if we could convert the whole thing from utf initially, but some
people have a habit of sending latin1 which of course translates as
lots of broken utf runes (and presumably utf translates as broken
latin1 on many machines).

i don't think that the header is allowed 8 bit stuff no matter what.

is the idea that if we just send utf regardless, sooner or later
mailers will assume that 8bit chars in mail headers are utf?

i might hack up a version of marshal that conforms to the standards,
at least superficially.  it would be nice to be able to send mail that
people on other platforms could read. we're not so far away.

  rog.

Re: [9fans] mime

[Boyd Roberts]

From: <presotto@plan9.bell-labs.com>
>
> Therefore, I'm going to make smtp (or a front end to it)
> convert and you can all stop your whining.
> 

i'm not whining.  i just think it's a horrible problem
that no-one should have to address, but unfortunately
it has be done.

Re: [9fans] mime

[Lyndon Nerenberg]

>>>>> "presotto" == presotto  <presotto@plan9.bell-labs.com> writes:
    presotto> However, I'm still perplexed as to what to put in the
    presotto> header of a multipart message.  Does the transfer
    presotto> encoding specified there pertain to the whole message or
    presotto> just to the

    presotto> 	nyah nyah, if you only had a MIME mailer you wouldn't
    presotto> see this

    presotto> message that starts every multipart message?

If the message contains a C-T-E header in the 822 headers, it applies
to the _entire_ message body (i.e. everything after the CRLFCRLF 822
header/body separator).

If any of the mime body parts contain a C-T-E header the 822 headers
should *not* contain a C-T-E header. Instead, each mime body part
should have an explicit C-T-E header (or an implicit "C-T-E: 7bit"
indicated by the absence of a C-T-E header on a particular body part).

As for the "nyah nyah" blob at the beginning of a mime multipart,
consider that that text will only be displayed by non-mime-aware
software, therefore it only makes sense to include only US-ASCII
characters in that text. You could also argue that in a multipart
message the "nyah nyah" part isn't really a mime body part (since
you can't reference it via mime), and therefore falls outside the
mime domain. If it's not mime, it is by definition 7bit content.

If you really want a definitive "ruling" on this you should talk
to Ned Freed.

--lyndon

Re: [9fans] mime

[rog]

> that's how it goes. je dconne pas...
>
> did you see the e acute?

nope.
that's the problem.  somewhere a gateway stripped out 'invalid' chars
(didn't even turn the e-acute into an 'i' which is what i would have
expected)

> i'm not sure you want to do that.  it's a mimefield.

yeah, but we're already neck deep in the big muddy.
so the damn fool might might as well keep yellin' to push on.

  rog.

Re: [9fans] mime

[Boyd Roberts]

From: <rog@vitanuova.com>
>
> yeah, but we're already neck deep in the big muddy.
> so the damn fool might might as well keep yellin' to push on.

yeah, well if i was gonna do it, i'd get the docs
and head off down to The White? Swan and ponder
the problem over several pints of taunton dry
blackthorn.  at least it'd kill some of the pain.

it's a horrible problem.

Re: [9fans] mime

[Nigel Roles]

> and head off down to The White? Swan and ponder

Black, assuming you're on Peasholme Green.

Re: [9fans] mime

[Steve Kilbane]

> yeah, well if i was gonna do it, i'd get the docs
> and head off down to The White? Swan and ponder
> the problem over several pints of taunton dry
> blackthorn.  at least it'd kill some of the pain.
> 
> it's a horrible problem.

actually, that sounds more like a description of how
it came about...

Re: [9fans] mime

[Boyd Roberts]

From: Steve Kilbane <steve@whitecrow.demon.co.uk>

> > yeah, well if i was gonna do it, i'd get the docs
> > and head off down to The White? Swan and ponder
> > the problem over several pints of taunton dry
> > blackthorn.  at least it'd kill some of the pain.
> > 
> > it's a horrible problem.
> 
> actually, that sounds more like a description of how
> it came about...

nah, i've done some good work in that pub [The Black Swan].

Re: [9fans] mime

[Boyd Roberts]

From: <rog@vitanuova.com>

> yeah, but we're already neck deep in the big muddy.
> so the damn fool might might as well keep yellin' to push on.

maybe you should grab the sendmail code from sendmail.org
to get a quick overview of the full horror.  it does a
pretty good job and is fairly clean.

i've read quite a bit of it in the last year :-(

Re: [9fans] mime

[rog]

> What brain damaged piece of dog
> excrement program would throw away the 8th bit of every
> byte in today's world?

annoying thing is that you don't know what the 8th bit means until
you've parsed the Content-transfer-encoding header.  it would be nice
if we could convert the whole thing from utf initially, but some
people have a habit of sending latin1 which of course translates as
lots of broken utf runes (and presumably utf translates as broken
latin1 on many machines).

i don't think that the header is allowed 8 bit stuff no matter what.

is the idea that if we just send utf regardless, sooner or later
mailers will assume that 8bit chars in mail headers are utf?

i might hack up a version of marshal that conforms to the standards,
at least superficially.  it would be nice to be able to send mail that
people on other platforms could read. we're not so far away.

  rog.

Re: [9fans] mime

[Boyd Roberts]

From: <rog@vitanuova.com>

> ...it would be nice
> if we could convert the whole thing from utf initially, but some
> people have a habit of sending latin1 which of course translates as
> lots of broken utf runes (and presumably utf translates as broken
> latin1 on many machines).

that's how it goes. je déconne pas...

did you see the e acute?

> i don't think that the header is allowed 8 bit stuff no matter what.

yes, hence that hideous ?...? encoding crap.

> is the idea that if we just send utf regardless, sooner or later
> mailers will assume that 8bit chars in mail headers are utf?

bit of a circular problem; the headers specify the encoding
which you don't know until you've read the headers.  it's
been hard enough to get 1 extra bit through [the 8th bit]
without stomping all over rfc 822.

> i might hack up a version of marshal that conforms to the standards,
> at least superficially.  it would be nice to be able to send mail that
> people on other platforms could read. we're not so far away.

i'm not sure you want to do that.  it's a mimefield.

[9fans] mime

[presotto]

If I were willing to do it correctly, I should have the
smtp program (my only MTA) determine the transfer-encoding
and not marshal.  It only needs to convert if it can't get
an esmtp conection.

Seriously though.  What brain damaged piece of dog
excrement program would throw away the 8th bit of every
byte in today's world?  Smtp's requirement for
7 bit clean was acknowledgement of systems of the time
that really did push messages across serial lines with
parity and/or used the 8th bit in mail files as a signalling
channel.  However, I really don't know of any such these
days.  Every smtp I call up seems perfectly happy to pass
on all 8 bits despite what the RFC says.

Re: [9fans] mime

[Boyd Roberts]

From: <presotto@plan9.bell-labs.com>

> Seriously though.  What brain damaged piece of dog
> excrement program would throw away the 8th bit of every
> byte in today's world?

not even sendmail, which qualifies, smashes the 8th bit
off on the receive side.  on the send side it does if
the mailer flag 7 is used.  mailer flag 9 (undocumented)
turns whatever encoding you've got back into an 8 bit
stream (in essense).

i have seen lotus notes V4? ack an EHLO but refuse to
offer 8BITMIME -- useless.

Re: [9fans] mime

[Steve Kilbane]

Boyd:
> i have seen lotus notes V4? ack an EHLO but refuse to
> offer 8BITMIME -- useless.

Nooo, not Notes - don't go there, please!

Re: [9fans] mime

[Boyd Roberts]

From: Steve Kilbane <steve@whitecrow.demon.co.uk>

> Boyd:
> > i have seen lotus notes V4? ack an EHLO but refuse to
> > offer 8BITMIME -- useless.
> 
> Nooo, not Notes - don't go there, please!

how do you think i picked up all this mime knowledge?

because of notes' appalling implementation of [e]smtp.

Re: [9fans] mime

[rob pike]

> Seriously though.  What brain damaged piece of dog
> excrement program would throw away the 8th bit of every
> byte in today's world?  Smtp's requirement for
> 7 bit clean was acknowledgement of systems of the time
> that really did push messages across serial lines with
> parity and/or used the 8th bit in mail files as a signalling
> channel.  However, I really don't know of any such these
> days.  Every smtp I call up seems perfectly happy to pass
> on all 8 bits despite what the RFC says.


Brian Ried is the source of the brain damage. Maybe it's
Brian damage.  He insisted on making the DEC gateways
clear the 8th bit in a stupefying bid to make everyone
honest.  He thought, I guess, that the computing world
didn't have enough complexity yet.  I hope he's happy
now.

-rob

Re: [9fans] mime

[Boyd Roberts]

From: rob pike <rob@plan9.bell-labs.com>
> 
> Brian Ried is the source of the brain damage. Maybe it's
> Brian damage.  He insisted on making the DEC gateways
> clear the 8th bit in a stupefying bid to make everyone
> honest.

i could believe he would do that but back while paul
vixie was with digital (till '93 or so) KJS didn't
clear the 8th bit on input.

then again, you could be talking about an earlier
or later time and over the past few years i've seen
him lose his mind.  there is one story that says he'd
smear fertiliser on his hands before flying to trigger
nitrate sniffers.

i could have asked the question on a list he's on but
i got myself kicked off it 'cos i wouldn't apologise
for a flame i sent -- as if no flames were ever sent
to this list.

Re: [9fans] mime

[matt]

----- Original Message -----
From: "Boyd Roberts" <boyd@planete.net>
> i could have asked the question on a list he's on but
> i got myself kicked off it 'cos i wouldn't apologise
> for a flame i sent -- as if no flames were ever sent
> to this list.

I'm shocked Boyd, that really doesn't sound like you at all, soembody really
must've wound you up for you to step out of character so.

Re: [9fans] mime

[Boyd Roberts]

From: matt <matt@proweb.co.uk>

> I'm shocked Boyd, that really doesn't sound like you at all, soembody really
> must've wound you up for you to step out of character so.

yeah, yeah, yeah... :-)

Re: [9fans] mime

[Boyd Roberts]

From: rob pike <rob@plan9.bell-labs.com>
> 
> Brian Ried is the source of the brain damage...

i've asked another source for confirmation.  i'd
forgotten that he's not on said list either.  he
asked to be removed.