mx.gnus.org being very strick

mx.gnus.org being very strick

[Lars Magne Ingebrigtsen]

The recent release patch was rejected by Gnus' MX with the following
message: 
 
  ding-patches@gnus.org
    SMTP error from remote mailer after end of data:
    host a.mx.gnus.org [66.139.78.221]: 550 Error:
    improper use of 8-bit data in message body

Isn't that kinda too strict?    
    
-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mx.gnus.org being very strick

[Matthias Andree]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> The recent release patch was rejected by Gnus' MX with the following
> message: 
>  
>   ding-patches@gnus.org
>     SMTP error from remote mailer after end of data:
>     host a.mx.gnus.org [66.139.78.221]: 550 Error:
>     improper use of 8-bit data in message body
>
> Isn't that kinda too strict?    

Nope. Your mailer is broken and Justine (the canonical name of that
server) decided she didn't want malformatted mail.

You MUST use proper MIME encoding.

You can also assume that the operator of that machine set the
corresponding option it on purpose, because the Postfix default is to
accept such garbage as non-MIME mail with ISO national characters or UTF 8.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Lars Magne Ingebrigtsen]

Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:

> You MUST use proper MIME encoding.

No, I mustn't.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mx.gnus.org being very strict

[Matthias Andree]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:
>
>> You MUST use proper MIME encoding.
>
> No, I mustn't.

Who says you mustn't? Nobody.

Who says you needn't? Show me.

You can't expect other sites to accept broken mail.

Sending non-US-ASCII in a non-MIME mail is "broken", see RFC-2822:
mail is 7bit US-ASCII. The extension is MIME, RFC-2045..2049. How does
the recipient know what character set the mail is in?

If you don't care if your mail arrives, then of course you needn't send
technically correct mail... but you're not in the position to complain
about "very strict" sites.

They just chose not to send garbage out by not allowing garbage in.

No reason to complain.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Lars Magne Ingebrigtsen]

Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:

> Sending non-US-ASCII in a non-MIME mail is "broken", see RFC-2822:
> mail is 7bit US-ASCII. The extension is MIME, RFC-2045..2049. How does
> the recipient know what character set the mail is in?

*gasp*  The horror.

> If you don't care if your mail arrives, then of course you needn't send
> technically correct mail... but you're not in the position to complain
> about "very strict" sites.

Of course I am.

In this case it was the Gnus diff that didn't go out, which was
generated in the time-honored tradition of saying
"cvs rdiff -r -u | mail".  This no longer works, which is a bore. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mx.gnus.org being very strick

[Reiner Steib]

On Sat, May 01 2004, Lars Magne Ingebrigtsen wrote:

> The recent release patch was rejected by Gnus' MX with the following
> message: 
>
>   ding-patches@gnus.org
>     SMTP error from remote mailer after end of data:
>     host a.mx.gnus.org [66.139.78.221]: 550 Error:
>     improper use of 8-bit data in message body
>
> Isn't that kinda too strict?    

I don't know if it's related, but for `cvslog@quimby.gnus.org'
(gmane.emacs.gnus.cvs) you added
"Content-Type: text/plain; charset=iso-2022-jp" some time ago.
iso-2022-jp is the coding of the ChangeLog files.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo--- PGP key available via WWW   http://rsteib.home.pages.de/

Re: mx.gnus.org being very strict

[Lars Magne Ingebrigtsen]

Reiner Steib <4.uce.03.r.s@nurfuerspam.de> writes:

> I don't know if it's related, but for `cvslog@quimby.gnus.org'
> (gmane.emacs.gnus.cvs) you added
> "Content-Type: text/plain; charset=iso-2022-jp" some time ago.
> iso-2022-jp is the coding of the ChangeLog files.

Hm...  so I did.  :-)

I hacked it into the cvslog summarizer...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mx.gnus.org being very strict

[Matthias Andree]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> In this case it was the Gnus diff that didn't go out, which was
> generated in the time-honored tradition of saying
> "cvs rdiff -r -u | mail".  This no longer works, which is a bore. 

"No longer works?" It has never been guaranteed nor supposed to work for
non-US-ASCII content. With the advent of 8-bit-clean sendmail everyone
forgot he was doing something wrong.

Evidently, some non-ASCII characters were contained in the diff, for
instance, a Japanese or European national character in the ChangeLog or
documentation.

Is there any difficulty with creating a custom header such as:

To: destination@dd.re.ss
Subject: my subject
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/x-patch; charset="utf-8"

prepending that to the cvs output and using sendmail rather than mail?

I don't see any.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Miles Bader]

Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:
> Is there any difficulty with creating a custom header such as:
> 
> To: destination@dd.re.ss
> Subject: my subject
> MIME-Version: 1.0
> Content-Transfer-Encoding: 8bit
> Content-Type: text/x-patch; charset="utf-8"

I suspect that there is no coherent `charset' for the patch; it's just
bytes.

Anyway, you know the old rule: be conservative in what you produce,
and liberal in what you accept.  So yeah, it'd be nice if everyone
would use lots of headers, but rejecting mail outright because there
are no mime-headers on 8-bit mail is a bit overly strict.

It's the MUA and the recipient that are going to have a problem with the
email, if there is one -- and for the same reason, they're also the best
placed to judge whether there _is_ a problem.

-Miles
-- 
"1971 pickup truck; will trade for guns"

Re: mx.gnus.org being very strict

[Wes Hardaker]

>>>>> On Mon, 03 May 2004 13:51:11 +0200, Matthias Andree <ma@dt.e-technik.uni-dortmund.de> said:

Matthias> prepending that to the cvs output and using sendmail rather than mail?

Or piping it through mmencode would likely help.

-- 
"In the bathtub of history the truth is harder to hold than the soap,
 and much more difficult to find."  -- Terry Pratchett

Re: mx.gnus.org being very strict

[Per Abrahamsen]

Miles Bader <miles@gnu.org> writes:

> I suspect that there is no coherent `charset' for the patch; it's just
> bytes.

RFC 1428 suggests the charset "unknown-8bit" for that purpose.

Re: mx.gnus.org being very strict

[Matthias Andree]

Miles Bader <miles@gnu.org> writes:

> Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:
>> Is there any difficulty with creating a custom header such as:
>> 
>> To: destination@dd.re.ss
>> Subject: my subject
>> MIME-Version: 1.0
>> Content-Transfer-Encoding: 8bit
>> Content-Type: text/x-patch; charset="utf-8"
>
> I suspect that there is no coherent `charset' for the patch; it's just
> bytes.

Make the Content-Type: application/octet-stream when no information is
available, or text/x-patch; charset="x-unknown".

> Anyway, you know the old rule: be conservative in what you produce,
> and liberal in what you accept.  So yeah, it'd be nice if everyone
> would use lots of headers, but rejecting mail outright because there
> are no mime-headers on 8-bit mail is a bit overly strict.

Well, if they are going to emit ("produce" doesn't seem right) the stuff
they have accepted verbatim, accepting junk means producing junk,
breaking the "conservative in what you produce" part. Gateways, Relays
always face this problem. Accept liberally and emit garbage, or accept
only conforming mail and emit conservatively formatted data.

> It's the MUA and the recipient that are going to have a problem with the
> email, if there is one -- and for the same reason, they're also the best
> placed to judge whether there _is_ a problem.

The standards "mail is 7-bit" have been in effect since 1982 and haven't
changed, so you must not generate a patch that doesn't fit into 7-bit
encodings in the first place (that's what uuencode used to be good for).

If you want to overcome the 7-bit restriction, you MUST follow the
standards, i. e. RFC-2045..2049. Anything outside this is certainly not
"conservative". Only applying the robustness princible to the transports
and not to the source and sink does not count.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Matthias Andree]

Wes Hardaker <wes@hardakers.net> writes:

> Or piping it through mmencode would likely help.

No, it won't: mmencode does only the bare encoding, but does not add the
necessary information the recipient needs to have the mail automatically
decoded. Piping the cvs result into "nail" instead of "mail" will also
solve the problem. nail is MIME aware (and incidentally, it is the
application SuSE use for "mail").

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Miles Bader]

On Mon, May 03, 2004 at 08:13:32PM +0200, Matthias Andree wrote:
> > Anyway, you know the old rule: be conservative in what you produce,
> > and liberal in what you accept.  So yeah, it'd be nice if everyone
> > would use lots of headers, but rejecting mail outright because there
> > are no mime-headers on 8-bit mail is a bit overly strict.
> 
> Well, if they are going to emit ("produce" doesn't seem right) the stuff
> they have accepted verbatim, accepting junk means producing junk,
> breaking the "conservative in what you produce" part. Gateways, Relays
> always face this problem. Accept liberally and emit garbage, or accept
> only conforming mail and emit conservatively formatted data.

If you are a gateway, you should probably just be as transparent as you
can...

It's the endpoints that know what's going on; when gateways try to interpret
things, they as often as not just muck things up.

There are cases where you can't be transparent, and have to interpret, of
course, as when you're converting from one form to another.

-Miles
-- 
/\ /\
(^.^)
(")")
*This is the cute kitty virus, please copy this into your sig so it can spread.

Re: mx.gnus.org being very strict

[Matthias Andree]

Miles Bader <miles@gnu.org> writes:

> If you are a gateway, you should probably just be as transparent as you
> can...

Garbage in - Garbage out principle? No thanks, we've had enough of that.

> It's the endpoints that know what's going on; when gateways try to interpret
> things, they as often as not just muck things up.

The relay in question does not muck up, it accepts (if properly
formatted) or rejects.

RFC-2822, RFC-2045..2049 are quite clear.

As said before, the site in question has deliberately enabled this
option, i. e. bounce defective mail rather than relay it, the default of
the software in question, Postfix, is to ignore encoding defects.

Still, Lars's script sent out garbage, and he simply had to take into
account that such garbage is rejected somewhere. Better rejected than
dropped by a spam killer, methinks.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Per Abrahamsen]

Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:

> Miles Bader <miles@gnu.org> writes:
>
>> If you are a gateway, you should probably just be as transparent as you
>> can...
>
> Garbage in - Garbage out principle? 

Yes.  Gateways should just pass bytes along, not try to be smart about
it. 

> No thanks, we've had enough of that.

We haven't had nearly enough of that.  All the fucking mess we went
through until we reached the current fragile and complex state of
sometimes-working i18n in mail, is due to the utter incompetence of
the mail standard guys, who thought that instead of simplifying
thousands of mail servers running a dozen different code bases to be
transparent, we should put highly complex workarounds in millions
clients from hundreds of different code bases.

This being the polite interpretation.  Ask Dan Bernstein for the
unsweetened version.

Re: mx.gnus.org being very strict

[Matthias Andree]

Per Abrahamsen <abraham@dina.kvl.dk> writes:

> Yes.  Gateways should just pass bytes along, not try to be smart about
> it. 

Per, MIME is well defined. The gate sees "no Content-Transfer-Encoding,
but 8 bit data -> RFC-2822 violated, admin wants this refused".

It sees a malformatted mail and rejects it.
It sees a well-formatted mail and passes it on.
The default in Postfix is to pass on even junk
 - the admin decided otherwise.

> We haven't had nearly enough of that.  All the fucking mess we went
> through until we reached the current fragile and complex state of
> sometimes-working i18n in mail, is due to the utter incompetence of
> the mail standard guys, who thought that instead of simplifying
> thousands of mail servers running a dozen different code bases to be
> transparent, we should put highly complex workarounds in millions
> clients from hundreds of different code bases.

If you don't like workarounds,
send MIME with quoted-printable or base64 encoding, then no-one will
touch your mail in transport except to add Received: headers.

> This being the polite interpretation.  Ask Dan Bernstein for the
> unsweetened version.

DJB is not a reference, NIMBY (not in my back yard) principle WRT
security and fixes and apologies to the own users, deliberate standard
violation, this doesn't at all help his reputation.

The gateway demands that 8-bit stuff be advertised at 8-bit stuff in the
header. And it has every right to do that, and I cannot find anything
wrong with that. Impractical maybe...

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Steinar Bang]

>>>>> Matthias Andree <ma@dt.e-technik.uni-dortmund.de>:

> Per Abrahamsen <abraham@dina.kvl.dk> writes:

>> Yes.  Gateways should just pass bytes along, not try to be smart
>> about it.
[snip!]
>> This being the polite interpretation.  Ask Dan Bernstein for the
>> unsweetened version.

> DJB is not a reference, NIMBY (not in my back yard) principle WRT
> security and fixes and apologies to the own users, deliberate
> standard violation, this doesn't at all help his reputation.

While I do not always agree with Dan Bernstein (or with Per, for that
matter), I have to say I do in this case.  Though I do not agree about
the incompetence of the mail standardization guys, mentioned in the
part that was snipped away.

This goes back to the summer of 1993 when I read the MIME standard,
and started using quoted-printable in emails, on an infrastructure that
consisted of sendmails universally patched to do just-send-8, and a
silently agreement that the charset used should be latin-1.

Suffice to say the feedback I got from people receiving
quoted-unreadable was so _universally_ negative, that I quickly
stopped doing it.  In the eyes of others, things just worked, and I
broke them.

So one thing I never would do, would be to set up an MTA to reject
messages that has 8 bit, but no MIME.  I can't imaging what kind of
standards-obsessed narrowmindedness would make anyone do so.

Wouldn't it be better to take that MX out of the gnus.org zone
altogether? 

Re: mx.gnus.org being very strict

[Josh Huber]

Steinar Bang <sb@dod.no> writes:

> So one thing I never would do, would be to set up an MTA to reject
> messages that has 8 bit, but no MIME.  I can't imaging what kind of
> standards-obsessed narrowmindedness would make anyone do so.

I bet it cuts down on spam quite a bit. (**wild conjecture** ;)

-- 
Josh Huber

Re: mx.gnus.org being very strict

[Miles Bader]

Josh Huber <huber+news@alum.wpi.edu> writes:
> I bet it cuts down on spam quite a bit. (**wild conjecture** ;)

Even better, just reject _all email_!  Spam be-gone!

-Miles
-- 
We are all lying in the gutter, but some of us are looking at the stars.
-Oscar Wilde

Re: mx.gnus.org being very strict

[Matthias Andree]

Steinar Bang <sb@dod.no> writes:

> This goes back to the summer of 1993 when I read the MIME standard,
> and started using quoted-printable in emails, on an infrastructure that
> consisted of sendmails universally patched to do just-send-8, and a
> silently agreement that the charset used should be latin-1.
>
> Suffice to say the feedback I got from people receiving
> quoted-unreadable was so _universally_ negative, that I quickly
> stopped doing it.  In the eyes of others, things just worked, and I
> broke them.

Virtually every MUA talks MIME today, several MTAs (among them sendmail)
and MDAs (among them maildrop) and filters have options to convert
quoted-printable text/* to 8bit text/*.

What may have been an issue in the time when everyone was using Berkeley
"mail" and perhaps elm, is not an issue today. 8BITMIME set off as a
standard to smooth the transition from 7bit to 8bit, the ultimate goal
would have been that every MTA supported 8BITMIME (which may, BTW, be
implemented as returning to the sender an 8BITMIME mail that would have
been to be sent to a 7bit destination).

DJB was the first [name lacks here] to deliberately throw his sabot into
the gears, breaking the whole idea of having 8BITMIME
transitional. qmail as a gateway will happily trash mail. I still
occasionally receive mail with d or | where a German Umlaut should have
been, which proves that there are still 7bit systems in use with systems
that just-send-eight.

> So one thing I never would do, would be to set up an MTA to reject
> messages that has 8 bit, but no MIME.  I can't imaging what kind of
> standards-obsessed narrowmindedness would make anyone do so.

Yay. How does the recipient know the character set?

Norway may not face the ambiguity because virtually everything fits, but
for instance France (cœur is ISO-8859-15), Germany, Finland (Z charon,
€) suffer from the confusion whether an 8bit text is ISO-8859-1, -15,
Windows-1252 or UTF-8. Heuristics or "conventions" don't apply. The
introduction of the Euro symbol about five years ago broke the
assumption that a mail or news was in ISO-8859-1 character set.

There are means to transport that character set information.
These means are called MIME.

If the admin determines he doesn't want to relay mail lacking such
information, or information that it is 8bit content counter to what
RFC-2822 says, then that is a wise one, is a perfectly legal one (he has
the power to set his system's policies) and has my sympathy.

Given that the relay was pretty early in the delivery chain, it was the
sender who got his mail back right away - and that's where strictness
and being conservative have to apply.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Matthias Andree]

Josh Huber <huber+news@alum.wpi.edu> writes:

> I bet it cuts down on spam quite a bit. (**wild conjecture** ;)

Particularly given that most spam originates in the US, and the spammers
happily try to use accented characters to bypass trivial blacklists, you
guess that it has reasonable impact on the spam volume. Until spammers
are the first to have their MTA fixed, even before the ecommerce sites
such as ebay are fixed (I reported they were not using MIME and
regularly getting trapped by spam filters years ago, literally, to no
avail to date).

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Steinar Bang]

>>>>> Matthias Andree <ma@dt.e-technik.uni-dortmund.de>:

> Virtually every MUA talks MIME today, several MTAs (among them
> sendmail) and MDAs (among them maildrop) and filters have options to
> convert quoted-printable text/* to 8bit text/*.

That wasn't the point.  My lesson learnt, was: if you try doing things
strictly by the standard, you risk being the one that breaks something
that "just works".

Re: mx.gnus.org being very strict

[Matthias Andree]

Steinar Bang <sb@dod.no> writes:

> That wasn't the point.  My lesson learnt, was: if you try doing things
> strictly by the standard, you risk being the one that breaks something
> that "just works".

If the destination mailer doesn't adhere to the standard, it will have a
hard time displaying conformant mail.

Given that this issue is not about changing content but lack of
declaration of what the content actually is, we've digressed. Adding
MIME headers will not make matters worse, but omitting them may harm
presentation. A question mark or X is the friendliest hint something
went wrong.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Steinar Bang]

>>>>> Matthias Andree <ma@dt.e-technik.uni-dortmund.de>:

> If the destination mailer doesn't adhere to the standard, it will
> have a hard time displaying conformant mail.

I don't understand what you mean by "destination mailer".  If it is
the final MTA, then the statement doesn't make sense to me.

If it's the destination MUA, then there are two ways of adhering to
the standard:
 1. strictly adhering to the standard, and refusing all mesasages that
    don't fit
 2. adhering to the standard, but put some configurable best guess on
    the messages that don't (in Norway that would typically be to
    display messages with 8bit characters and no MIME markup, as 
     text/plain; charset=iso-8859-1)

It sounds to me that you're advocating alternative 1.

To me, alternative 2 is the only sensible approach.

Re: mx.gnus.org being very strict

[Matthias Andree]

Steinar Bang <sb@dod.no> writes:

>>>>>> Matthias Andree <ma@dt.e-technik.uni-dortmund.de>:
>
>> If the destination mailer doesn't adhere to the standard, it will
>> have a hard time displaying conformant mail.
>
> I don't understand what you mean by "destination mailer".  If it is
> the final MTA, then the statement doesn't make sense to me.

I meant the MUA.

> If it's the destination MUA, then there are two ways of adhering to
> the standard:
>  1. strictly adhering to the standard, and refusing all mesasages that
>     don't fit

The injecting agent has to reject this crap.

This is what happened in this case, no reason to complain. Fix the generator.

>  2. adhering to the standard, but put some configurable best guess on
>     the messages that don't (in Norway that would typically be to
>     display messages with 8bit characters and no MIME markup, as 
>      text/plain; charset=iso-8859-1)

> It sounds to me that you're advocating alternative 1.

Yup. I am still under the uncontradicted assumption that mx.gnus.org was
the first hop, and hence the proper place to apply conformance constraints.

German-language newsgroups are a nice proof that guess-work does not
quite work, postings are sprinkled with undecodable characters.

If the originator extends an existing format (8-bit national characters
in the originally 7bit medium) and that extension is standardized, it
must be properly marked. You send 8bit, you declare 8bit. Period.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Steinar Bang]

>>>>> Matthias Andree <ma@dt.e-technik.uni-dortmund.de>:

> The injecting agent has to reject this crap.

> This is what happened in this case, no reason to complain. Fix the
> generator.

IMO with attitudes like this, changing the MX sounds like a better
approach.

Re: mx.gnus.org being very strict

[Matthias Andree]

Steinar Bang <sb@dod.no> writes:

> IMO with attitudes like this, changing the MX sounds like a better
> approach.

As written before, early rejection of malformatted mail prevents it from
disappearing in black holes like antivirus, firewall or antispam
software later in the transport.

The _robust_ approach is not to send out such malformatted mail, and
that's what mx.gnus.org did. You may see it as inconvenient, but it's a
sane approach.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95

Re: mx.gnus.org being very strict

[Lars Magne Ingebrigtsen]

Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:

> The _robust_ approach is not to send out such malformatted mail, and
> that's what mx.gnus.org did. 

mx.gnus.org wasn't the originating MX.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: mx.gnus.org being very strict

[Matthias Andree]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Matthias Andree <ma@dt.e-technik.uni-dortmund.de> writes:
>
>> The _robust_ approach is not to send out such malformatted mail, and
>> that's what mx.gnus.org did. 
>
> mx.gnus.org wasn't the originating MX.

Heck. I knew there was a snare.

-- 
Matthias Andree

Encrypted mail welcome: my GnuPG key ID is 0x052E7D95