Re: [9fans] problems with marshal

Re: [9fans] problems with marshal

[rog]

boyd wrote:
> by the end of the 4th mime doc either the authors had totally
> lost it, you have totally lost it or a combination of both.

i agree.
i'm very close to losing it totally today.

... (muffled shrieks) ...

  rog.

Re: [9fans] problems with marshal

[Lyndon Nerenberg]

>>>>> "rob" == rob pike <rob@plan9.bell-labs.com> writes:

    rob> Yeah, but doesn't the S in SMTP mean simple?

Yup. That's why SMTP doesn't define MIME :-)

--lyndon

Re: [9fans] problems with marshal

[Boyd Roberts]

given that we all know that X.400/500 are doomed, i'd say
that the X means eXtremely complicated.

Re: [9fans] problems with marshal

[forsyth]

[-- Attachment #1: Type: text/plain, Size: 151 bytes --]

yes, indeed, but they don't specify which sense of `simple'.
in much the same way, i might regard the book `Core Jini' as
`thick' in every sense.


[-- Attachment #2: Type: message/rfc822, Size: 1421 bytes --]

From: "rob pike" <rob@plan9.bell-labs.com>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] problems with marshal
Date: Thu, 23 Nov 2000 15:57:04 -0500
Message-ID: <20001123205722.99626199E1@mail.cse.psu.edu>

Yeah, but doesn't the S in SMTP mean simple?

-rob

Re: [9fans] problems with marshal

[rob pike]

Yeah, but doesn't the S in SMTP mean simple?

-rob

Re: [9fans] problems with marshal

[Boyd Roberts]

From: Lyndon Nerenberg <lyndon@messagingdirect.com>
> 
> The transports might be, however the MTAs are *not*. Try throwing
> 8bit content with NULs at an SMTP server that doesn't advertise (or
> with which you haven't negotiated) 8BITMINE.

the discussion was about 8 bit characters.  admittedly, nul is
part of the set of 8 bit characters, but it was never part of
the discussion.  the plan 9 utf encoding will never send a nul
as a multi-byte sequence.

anyway, we're talking about text messages.  as i have told
numerous people (but not in this group): the M in SMTP means
mail, the F in FTP means file.  grafting generalised file
transfer onto 821/822 was a heinous error.

Re: [9fans] problems with marshal

[Lyndon Nerenberg]

>>>>> "Boyd" == Boyd Roberts <boyd@planete.net> writes:

    Boyd> he said that MIME would take care of all of that.  i didn't
    Boyd> agree.  even in '92 i think it was pretty safe to say that
    Boyd> most/all? transports were 8 bit.

The transports might be, however the MTAs are *not*. Try throwing
8bit content with NULs at an SMTP server that doesn't advertise (or
with which you haven't negotiated) 8BITMINE. There is an awful lot
of mail software deployed that manipulates [E]SMTP data as C strings,
and throwing a (non-encoded) NUL in there pretty well does it for reliable
end-to-end delivery.

--lyndon

Re: [9fans] problems with marshal

[rog]

> I'll try changing it to always say
> 	Content-Transfer-Encoding: 7bit
> in the main header and see how it goes.

actually, that's the default anyway.  and even if it wasn't, it
wouldn't really help, as that header would be lying if utf8 characters
were included.

the whole thing's a crock of shit, but if it's to be a working crock
of shit i can't really see much alternative to converting utf8
plain/text attachments to quoted-printable.

of course, then there's the issue of utf8 in header lines.  =?.....?=,
rfc2047, aargh.

as you might be able to tell, i've had some dealings with
this stuff recently, hence i was plundering upas for ideas
on "nice" ways of dealing with the crud, hence i realised these
problems...

  cheers,
    rog.

Re: [9fans] problems with marshal

[Boyd Roberts]

From: <rog@vitanuova.com>
> 
> the whole thing's a crock of shit, but if it's to be a working crock
> of shit i can't really see much alternative to converting utf8
> plain/text attachments to quoted-printable.
> 

it is a _very nasty_ problem indeed.

being in france i've had to deal with ISO Latin 1 for a long
time.  oh, god, how many arguments have i had over this.

> of course, then there's the issue of utf8 in header lines.  =?.....?=,
> rfc2047, aargh.

read that one too.

i had to do some corporate mail snooping so that's why
understand the horror of it all.  i understand the horror,
but not all of MIME.  it's 150 pages of pure trash.

saying it's 8bit doesn't really help 'cos it's a 7 bit
transport unless it's ESMTP _and_ the ESMTP implementation
returns 8BITMIME as one of the supported options (IIRC).

doesn't 7 bit come from the IMPs?

a long time ago i suggested to paul vixie just to add
a new SMTP command 'will you do 8 bit transport'?  if
the response was no the message would be returned.  this
was based on the assumption that all messages potentially
had 8 bit chars and no analysis of the message body.

he said that MIME would take care of all of that.  i
didn't agree.  even in '92 i think it was pretty safe
to say that most/all? transports were 8 bit.

i wanted to force people who had 7 bit transports to
chuck 'em, if such things still existed.

Re: [9fans] problems with marshal

[Alex Bochannek]

Keep in mind that 8bit content-transfer-encoding is MTA specific. If
your MTA supports ESMTP, it works fine, but otherwise, that's what
transfer encodings like quoted-printable[1] or base64 are for. See
also UTR #17 at http://www.unicode.org/unicode/reports/tr17/

Alex.

[1] 8bit character encoding schemes like UTF-8 (nee UTF-2) for example
lend themselves nicely to quoted-printable since they preserve the
7bit ASCII character encoding.

Re: [9fans] problems with marshal

[presotto]

Sorry about that last message... I can't cut and paste anymore, too many
neurons gone.  I meant:

I'll try changing it to always say
	Content-Transfer-Encoding: 8bit
in the main header and see how it goes.

Re: [9fans] problems with marshal

[presotto]

I originally just said it was UTF-8 and 8bit encoding but backed off because
lots of mailers threw their hands up at UTF-8.  Perhaps I should have kept the
8bit part though.  I'll try changing it to always say
	Content-Transfer-Encoding: 7bit
in the main header and see how it goes.

[9fans] problems with marshal

[rog]

i came across a few problems with upas/marshal; i'm afraid i haven't
got time to fix all of them right now.

most minor problem is that in scanning for 8 bit characters, a
"Content-transfer-encoding: 8bit" header can be output several times.

that's fixed easily with:
diff /n/dump/2000/1121/sys/src/cmd/upas/ned/marshal.c marshal.c
370c370
< 		for(;;){
---
> 		while (docontenttype){
385c385
< 				if(*p++ & 0x80){
---
> 				if(*p++ & 0x80 && docontenttype){

(which also saves some buffer space on files containing lots of utf)

the other problem is that text attachments containing 8 bit utf don't
propagate the "Content-transfer-encoding: 8bit" into the main header.
fixing this would require reading all the attachments before spitting
out the principal header, which is not currently done.

the alternative (which would likely be preferable, as it would make
the resulting email smtp compatible) would be to encode any 8-bit
plain text attachments in quoted-printable.  then i might be able to
see the utf chars sent to 9fans!

one last thing:
diff /n/dump/2000/1121/sys/src/cmd/file.c file.c
345c345
< 	if (nbuf < 100)
---
> 	if (nbuf < 100 && !mime)

which stops "file -m" printing two type lines for short files.

  cheers,
    rog.

Re: [9fans] problems with marshal

[Boyd Roberts]

From: <rog@vitanuova.com>
> 
> the other problem is that text attachments containing 8 bit utf don't
> propagate the "Content-transfer-encoding: 8bit" into the main header.
> fixing this would require reading all the attachments before spitting
> out the principal header, which is not currently done.
> 

i reckon that a lot of careful thought would be required before
stepping into that mimefield [sic].

by the end of the 4th mime doc either the authors had totally
lost it, you have totally lost it or a combination of both.

at least with text messages from plan 9 they are obviously
8bit, 'cos it's UTF.  but when you talk smtp you then have to
decide on whether conversion is required depending on who
you're talking to at the other end.

it's a very ugly problem.  attatchments complicate it no end.

i agree with what Bill McLean, director of the AIM-9 sidewinder
project said:

    it is easy to build something that is complicated but
    it is hard to build something so that it's simple

that may not be exact.

during the trials between the sidewinder and what was to
become the AIM-54 phoenix, the phoenix team had a hanger
full of test equipment.  when asked for what sort of test
equipment the sidewinder team required the reply was:

    oh, a flashlight and a screwdriver