something decodes my headers...

something decodes my headers...

[Jan Rychter]

... and I don't know what it is.

I get lots of mail with RFC2047-encoded From: fields. Mostly
ISO-8859-2. Now, when mail-extr gets to those headers, they are already
in 8-bit form, which causes mail-extr to fail tragically. I get chopped
names, annoying BBDB behavior, and all sorts of breakage. People who get
ISO-8859-1 E-mail don't see this, because mail-extr.el is "extended" to
handle ISO-8859-1 in a particularly ugly way.

I've tried to trace what it is that decodes the headers to 8bit before
mail-extr sees them -- but I have no idea. They are RFC2047-encoded on
the disk. Looking through my init files also did not produce any clues.

Before I go diving into the code to trace the actual execution path,
perhaps somebody has suggestions?

--J.

Re: something decodes my headers...

[Simon Josefsson]

Jan Rychter <jan@rychter.com> writes:

> ... and I don't know what it is.
>
> I get lots of mail with RFC2047-encoded From: fields. Mostly
> ISO-8859-2. Now, when mail-extr gets to those headers, they are already
> in 8-bit form, which causes mail-extr to fail tragically. I get chopped
> names, annoying BBDB behavior, and all sorts of breakage. People who get
> ISO-8859-1 E-mail don't see this, because mail-extr.el is "extended" to
> handle ISO-8859-1 in a particularly ugly way.

Do you have a backtrace that says where mail-extr is invoked?  I think
it might be a bug to invoke mail-extr on 8bit data.

> I've tried to trace what it is that decodes the headers to 8bit before
> mail-extr sees them -- but I have no idea. They are RFC2047-encoded on
> the disk. Looking through my init files also did not produce any clues.

Note that if it is bbdb that calls mail-extr, it is probably a bug in
bbdb.  Chopped names could indicate a bug in rfc2047.el though, there
where bugs like this recently (perhaps not fixed?  I recall emacs bugs
caused them).

Re: something decodes my headers...

[Jan Rychter]

>>>>> "Simon" == Simon Josefsson <jas@extundo.com> writes:
 Simon> Jan Rychter <jan@rychter.com> writes:
 >> ... and I don't know what it is.
 >>
 >> I get lots of mail with RFC2047-encoded From: fields. Mostly
 >> ISO-8859-2. Now, when mail-extr gets to those headers, they are
 >> already in 8-bit form, which causes mail-extr to fail tragically. I
 >> get chopped names, annoying BBDB behavior, and all sorts of
 >> breakage. People who get ISO-8859-1 E-mail don't see this, because
 >> mail-extr.el is "extended" to handle ISO-8859-1 in a particularly
 >> ugly way.

 Simon> Do you have a backtrace that says where mail-extr is invoked?  

Silly me. I should have prepared it right away instead of wasting your
time. I don't know why I didn't think of it. Sorry about that.

 Simon> I think it might be a bug to invoke mail-extr on 8bit data.

It is. It is really not designed to do that, and it only works for a
subset of ISO-8859-1. Actually, I have the distinct impression that if
we removed the 8859-1 support from mail-extr.el, the breakage would be
noticed by many more people.

 >> I've tried to trace what it is that decodes the headers to 8bit
 >> before mail-extr sees them -- but I have no idea. They are
 >> RFC2047-encoded on the disk. Looking through my init files also did
 >> not produce any clues.

 Simon> Note that if it is bbdb that calls mail-extr, it is probably a
 Simon> bug in bbdb.  Chopped names could indicate a bug in rfc2047.el
 Simon> though, there where bugs like this recently (perhaps not fixed?
 Simon> I recall emacs bugs caused them).

Well, interestingly enough, mail-extract-address-components is being
called TWICE whenever I enter a message. Once by Gnus, once by BBDB. In
both cases, an 8-bit string is being passed. Backtraces below, with the
name of the person in the From: field changed to my spamtrap:

  mail-extract-address-components("full 8bit name <spam@rychter.com>")
  mm-dissect-buffer(nil nil)
  gnus-display-mime()
  gnus-article-prepare-display()
  gnus-article-prepare(96804 nil)
  gnus-summary-display-article(96804)
  gnus-summary-next-article(nil nil t)
  gnus-summary-prev-article(nil)
  call-interactively(gnus-summary-prev-article)


  mail-extract-address-components("full 8bit name <spam@rychter.com>")
  bbdb-rfc822-addresses("full 8bit name <spam@rychter.com>")
  bbdb-get-addresses(t "Jan Rychter" mail-fetch-field)
  bbdb/gnus-update-records(nil)
  bbdb/gnus-pop-up-bbdb-buffer()
  run-hooks(bbdb/gnus-pop-up-bbdb-buffer)
  apply(run-hooks bbdb/gnus-pop-up-bbdb-buffer)
  gnus-run-hooks(gnus-article-prepare-hook)
  gnus-article-prepare-display()
  gnus-article-prepare(96804 nil)
  gnus-summary-display-article(96804)
  gnus-summary-next-article(nil nil t)
  gnus-summary-prev-article(nil)
  call-interactively(gnus-summary-prev-article)

As a reminder, this results in mail-extract-address-components returning
a "chopped" partial name, because of an 8-bit character in the middle.

Does this help to narrow the problem?

thanks,
--J.

Re: something decodes my headers...

[Simon Josefsson]

Jan Rychter <jan@rychter.com> writes:

>  Simon> I think it might be a bug to invoke mail-extr on 8bit data.
>
> It is. It is really not designed to do that, and it only works for a
> subset of ISO-8859-1. Actually, I have the distinct impression that if
> we removed the 8859-1 support from mail-extr.el, the breakage would be
> noticed by many more people.

Yes.  OTOH, I doubt the Right Thing wrt mail-extr is to patch it.  It
should be rewritten.  But if there is a short patch that solve
problems, I guess it doesn't hurt.  Removing the iso-8859-1 support is
probably not a good thing in practice though, even if I agree in
theory.

>  Simon> Note that if it is bbdb that calls mail-extr, it is probably a
>  Simon> bug in bbdb.  Chopped names could indicate a bug in rfc2047.el
>  Simon> though, there where bugs like this recently (perhaps not fixed?
>  Simon> I recall emacs bugs caused them).
>
> Well, interestingly enough, mail-extract-address-components is being
> called TWICE whenever I enter a message. Once by Gnus, once by BBDB. In
> both cases, an 8-bit string is being passed. Backtraces below, with the
> name of the person in the From: field changed to my spamtrap:
>
>   mail-extract-address-components("full 8bit name <spam@rychter.com>")
>   mm-dissect-buffer(nil nil)

I don't see where the MIME decoding occurs before m-e-a-c is invoked
in that function.  It just seems to invoke (mail-fetch-field "from")
and then m-e-a-c on that string.  Could it be that the incoming
message had raw non-ASCII characters in it?  Does this patch help?
Assuming you leave g-e-a-c to its default value.

--- mm-decode.el.~6.91.~	Fri Apr 25 01:07:04 2003
+++ mm-decode.el	Wed Jun 11 06:37:37 2003
@@ -493,7 +493,8 @@
 	  ;; creates unibyte buffers. This `if', though not a perfect
 	  ;; solution, avoids most of them.
 	  (if from
-	      (setq from (cadr (mail-extract-address-components from))))))
+	      (setq from (cadr (funcall gnus-extract-address-components
+					from))))))
       (when cte
 	(setq cte (mail-header-strip cte)))
       (if (or (not ctl)

Re: something decodes my headers...

[Jan Rychter]

>>>>> "Simon" == Simon Josefsson <jas@extundo.com> writes:
 Simon> I think it might be a bug to invoke mail-extr on 8bit data.
 Simon> Jan Rychter <jan@rychter.com> writes: 
 >> It is. It is really not designed to do that, and it only works for a
 >> subset of ISO-8859-1. Actually, I have the distinct impression that
 >> if we removed the 8859-1 support from mail-extr.el, the breakage
 >> would be noticed by many more people.

 Simon> Yes.  OTOH, I doubt the Right Thing wrt mail-extr is to patch
 Simon> it.  It should be rewritten.  But if there is a short patch that
 Simon> solve problems, I guess it doesn't hurt.  Removing the
 Simon> iso-8859-1 support is probably not a good thing in practice
 Simon> though, even if I agree in theory.

I fully agree, although I'd love to see a "testing period" with the
ISO-8859-1 support in mail-extr.el removed. Just so that people notice
the problems.

 Simon> Note that if it is bbdb that calls mail-extr, it is probably a
 Simon> bug in bbdb.  Chopped names could indicate a bug in rfc2047.el
 Simon> though, there where bugs like this recently (perhaps not fixed?
 Simon> I recall emacs bugs caused them).
 >> Well, interestingly enough, mail-extract-address-components is being
 >> called TWICE whenever I enter a message. Once by Gnus, once by
 >> BBDB. In both cases, an 8-bit string is being passed. Backtraces
 >> below, with the name of the person in the From: field changed to my
 >> spamtrap:
 >>
 >> mail-extract-address-components("full 8bit name <spam@rychter.com>")
 >> mm-dissect-buffer(nil nil)

 Simon> I don't see where the MIME decoding occurs before m-e-a-c is
 Simon> invoked in that function.  It just seems to invoke
 Simon> (mail-fetch-field "from") and then m-e-a-c on that string.
 Simon> Could it be that the incoming message had raw non-ASCII
 Simon> characters in it?  Does this patch help?  Assuming you leave
 Simon> g-e-a-c to its default value.

[...]

No, the patch does not help. I do not mess with
gnus-extract-address-components, it is left at default.

Also, the messages in my nnml spool definitely do not have 8bit
characters:

From: =?iso-8859-2?Q?Pawe=B3_lastnamesnipped?= <spam@rychter.com>

So, it seems that something decodes the From: field.

--J.

Re: something decodes my headers...

[Simon Josefsson]

Jan Rychter <jan@rychter.com> writes:

>>>>>> "Simon" == Simon Josefsson <jas@extundo.com> writes:
>  Simon> I think it might be a bug to invoke mail-extr on 8bit data.
>  Simon> Jan Rychter <jan@rychter.com> writes: 
>  >> It is. It is really not designed to do that, and it only works for a
>  >> subset of ISO-8859-1. Actually, I have the distinct impression that
>  >> if we removed the 8859-1 support from mail-extr.el, the breakage
>  >> would be noticed by many more people.
>
>  Simon> Yes.  OTOH, I doubt the Right Thing wrt mail-extr is to patch
>  Simon> it.  It should be rewritten.  But if there is a short patch that
>  Simon> solve problems, I guess it doesn't hurt.  Removing the
>  Simon> iso-8859-1 support is probably not a good thing in practice
>  Simon> though, even if I agree in theory.
>
> I fully agree, although I'd love to see a "testing period" with the
> ISO-8859-1 support in mail-extr.el removed. Just so that people notice
> the problems.

Since mail-extr.el isn't part of Gnus, this is hard to do here.  Maybe
suggest it on emacs-devel?  OTOH, I'm not sure what the result of the
test period would be, it doesn't help if people just notice the
problem, someone has to do something about it too, and they can do
something about it without a test period, can't they?

> No, the patch does not help. I do not mess with
> gnus-extract-address-components, it is left at default.
>
> Also, the messages in my nnml spool definitely do not have 8bit
> characters:
>
> From: =?iso-8859-2?Q?Pawe=B3_lastnamesnipped?= <spam@rychter.com>
>
> So, it seems that something decodes the From: field.

I guess I missed something...

Re: something decodes my headers...

[Jan Rychter]

>>>>> "Simon" == Simon Josefsson <jas@extundo.com> writes:
 Simon> Jan Rychter <jan@rychter.com> writes:
 > "Simon" == Simon Josefsson <jas@extundo.com> writes:
 Simon> I think it might be a bug to invoke mail-extr on 8bit data.  Jan
 Simon> Rychter <jan@rychter.com> writes:
 > It is. It is really not designed to do that, and it only works for a
 > subset of ISO-8859-1. Actually, I have the distinct impression that
 > if we removed the 8859-1 support from mail-extr.el, the breakage
 > would be noticed by many more people.
 >>
 Simon> Yes.  OTOH, I doubt the Right Thing wrt mail-extr is to patch
 Simon> it.  It should be rewritten.  But if there is a short patch that
 Simon> solve problems, I guess it doesn't hurt.  Removing the
 Simon> iso-8859-1 support is probably not a good thing in practice
 Simon> though, even if I agree in theory.
 >>
 >> I fully agree, although I'd love to see a "testing period" with the
 >> ISO-8859-1 support in mail-extr.el removed. Just so that people
 >> notice the problems.

 Simon> Since mail-extr.el isn't part of Gnus, this is hard to do here.
 Simon> Maybe suggest it on emacs-devel?  OTOH, I'm not sure what the
 Simon> result of the test period would be, it doesn't help if people
 Simon> just notice the problem, someone has to do something about it
 Simon> too, and they can do something about it without a test period,
 Simon> can't they?

mail-extr.el looks way too scary to me.

 >> No, the patch does not help. I do not mess with
 >> gnus-extract-address-components, it is left at default.
 >>
 >> Also, the messages in my nnml spool definitely do not have 8bit
 >> characters:
 >>
 >> From: =?iso-8859-2?Q?Pawe=B3_lastnamesnipped?= <spam@rychter.com>
 >>
 >> So, it seems that something decodes the From: field.

 Simon> I guess I missed something...

Found it.

  rfc2047-parse-and-decode("=?iso-8859-2?Q?Pawe=B3_lastname?=")
  mail-decode-encoded-word-region(1 944)
  article-decode-encoded-words()
  run-hooks(article-decode-encoded-words)
  gnus-request-article-this-buffer(45728 "nnml+private:in")
  gnus-article-prepare(45728 nil)
  gnus-summary-display-article(45728 nil)
  gnus-summary-select-article(nil force)
  gnus-summary-show-article(nil)
  call-interactively(gnus-summary-show-article)

And, just as a reminder, this results in:

  mail-extract-address-components("Paweł lastname <spam@rychter.com>")
  mm-dissect-buffer(nil nil)
  gnus-display-mime()
  gnus-article-prepare-display()
  gnus-article-prepare(96804 nil)
  gnus-summary-display-article(96804)
  gnus-summary-next-article(nil nil t)
  gnus-summary-prev-article(nil)
  call-interactively(gnus-summary-prev-article)

... which fails miserably with mail-extract-address-components returning
"Pawe" as the sender's name.

I think that gnus-request-article-this-buffer should not rfc2047-decode
headers before mail-extract-address-components sees them?

Can I hope for a fix from someone more knowledgeable?

--J.

Re: something decodes my headers...

[Dave Love]

Simon Josefsson <jas@extundo.com> writes:

> Note that if it is bbdb that calls mail-extr, it is probably a bug in
> bbdb.

I don't think so.  If it didn't use decoded data, bbdb & al would have
to mess with a lot of decoding themselves, and I doubt that's
intended.  (I think mail-extr predates MIME handling anyhow.)

Re: something decodes my headers...

[Dave Love]

Jan Rychter <jan@rychter.com> writes:

> People who get
> ISO-8859-1 E-mail don't see this, because mail-extr.el is "extended" to
> handle ISO-8859-1 in a particularly ugly way.

That must be XEmacs-specific.  mail-extr in Emacs (at least Emacs 21)
works with multilingual text.

> I've tried to trace what it is that decodes the headers to 8bit before
> mail-extr sees them -- but I have no idea.

I think that's the wrong thing to look at.  mail-extr should just cope
with multilingual names.  [If it's really `8-bit' I assume it's a
non-Mule XEmacs, so you can't deal with multilingual text anyhow.]

Re: something decodes my headers...

[Simon Josefsson]

Dave Love <d.love@dl.ac.uk> writes:

> Jan Rychter <jan@rychter.com> writes:
>
>> People who get
>> ISO-8859-1 E-mail don't see this, because mail-extr.el is "extended" to
>> handle ISO-8859-1 in a particularly ugly way.
>
> That must be XEmacs-specific.  mail-extr in Emacs (at least Emacs 21)
> works with multilingual text.

What does "works" mean?  I'm not aware of any standards that define
how non-ASCII email address phrases should work (RFC (2)822 even
forbid them) except MIME, so any implementation claiming to support it
if it doesn't use MIME is ad-hoc at best.

>> I've tried to trace what it is that decodes the headers to 8bit before
>> mail-extr sees them -- but I have no idea.
>
> I think that's the wrong thing to look at.  mail-extr should just cope
> with multilingual names.

In that case, I wish the way it works was documented.  It is not clear
to me how to go from the RFC 822 address phrase to multilingual names,
if you are not using MIME.  E.g., is the BNF "atom" modified to
include non-ASCII?  The "qtext"?  The "ctext"?  Even "CHAR"?  All of
those may need to be modified if all kind of RFC 822 addresses are to
be parsed correctly (consider the three distinct kind of addresses
used in the To/Cc/From in this message).  Of course, mail-extr doesn't
understand the RFC 822 BNF at all, so it is not easy to realize that
whatever mail-extr implement is according to any standard derived from
the BNF.

Re: something decodes my headers...

[Jan Rychter]

>>>>> "Simon" == Simon Josefsson <jas@extundo.com> writes:
 Simon> Dave Love <d.love@dl.ac.uk> writes:
 >> Jan Rychter <jan@rychter.com> writes:
 >>
 > People who get ISO-8859-1 E-mail don't see this, because mail-extr.el
 > is "extended" to handle ISO-8859-1 in a particularly ugly way.
 >>
 >> That must be XEmacs-specific.  mail-extr in Emacs (at least Emacs
 >> 21) works with multilingual text.

 Simon> What does "works" mean?  I'm not aware of any standards that
 Simon> define how non-ASCII email address phrases should work (RFC
 Simon> (2)822 even forbid them) except MIME, so any implementation
 Simon> claiming to support it if it doesn't use MIME is ad-hoc at best.

In general, I'd agree with Simon, but I think what Dave meant was that
mail-extract-address-components is supposed to do its hairy thing
independent of whether it gets an rfc2047-encoded string or not. Perhaps
that is indeed the case.

In any case, I've reported the problem to XEmacs developers as it indeed
seems that mail-extr.el is different in GNU Emacs (it uses [:alnum:] in
regexes specifying the fullname part). The ISO-8859-1-specific hack
seems to be XEmacs-only.

I just hope someone will fix this, I hope we won't end up just pointing
fingers here and there.

--J.

Re: something decodes my headers...

[Dave Love]

Jan Rychter <jan@rychter.com> writes:

> In general, I'd agree with Simon, but I think what Dave meant was that
> mail-extract-address-components is supposed to do its hairy thing
> independent of whether it gets an rfc2047-encoded string or not.

No, it must be fully decoded.

> I just hope someone will fix this, I hope we won't end up just pointing
> fingers here and there.

I hope you don't expect fingers to be pointed from the direction of
Emacs.

[If you want to know the mail-extr author's intention, I think he's
now in Computer Science at Heriot-Watt University.]

Re: something decodes my headers...

[Jan Rychter]

>>>>> "Dave" == Dave Love <d.love@dl.ac.uk>:
 Dave> Jan Rychter <jan@rychter.com> writes:
 >> In general, I'd agree with Simon, but I think what Dave meant was
 >> that mail-extract-address-components is supposed to do its hairy
 >> thing independent of whether it gets an rfc2047-encoded string or
 >> not.

 Dave> No, it must be fully decoded.

 >> I just hope someone will fix this, I hope we won't end up just
 >> pointing fingers here and there.

 Dave> I hope you don't expect fingers to be pointed from the direction
 Dave> of Emacs.

No, I had actually meant Gnus vs XEmacs. But in any case, I've dug
deeper and indeed the problem seems to be XEmacs-specific. I've reported
it to XEmacs developers. Unfortunately, the fix is not obvious (without
regexp classes).

--J.

Re: something decodes my headers...

[Dave Love]

jas@extundo.com (Simon Josefsson) writes:

> What does "works" mean?

It doesn't fail in the way that's being complained about.

> I'm not aware of any standards that define
> how non-ASCII email address phrases should work (RFC (2)822 even
> forbid them) except MIME, so any implementation claiming to support it
> if it doesn't use MIME is ad-hoc at best.

It's obviously intended for use with decoded addresses, which is how
it's being used.

> In that case, I wish the way it works was documented.

You'll need to make a bug report if you're not happy.  I haven't
actually touched it, though I'd have guessed I made the relevant
change.

> It is not clear to me how to go from the RFC 822 address phrase to
> multilingual names, if you are not using MIME.

It's clear you can't pick apart a single MIME encoded word in the way
mail-extr tries to.  It must be decoded, and the exact (2)822 grammar
is irrelevant.

> Of course, mail-extr doesn't
> understand the RFC 822 BNF at all, so it is not easy to realize that
> whatever mail-extr implement is according to any standard derived from
> the BNF.

I don't understand that.  The BNF is largely irrelevant for extracting
personal names from addresses as far as I can tell.