Re: 21.5 crash in Gnus

Re: 21.5 crash in Gnus

[Reiner Steib]

On Tue, Mar 14 2006, Stephen J. Turnbull wrote:

> However, Gnus's I18N misfeatures are legion;

Care to explain?  Or give us some URLs or MIDs?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: 21.5 crash in Gnus

[Stephen J. Turnbull]

>>>>> "Reiner" == Reiner Steib <reinersteib+gmane@imap.cc> writes:

    Reiner> On Tue, Mar 14 2006, Stephen J. Turnbull wrote:

    >> However, Gnus's I18N misfeatures are legion;

    Reiner> Care to explain?  Or give us some URLs or MIDs?

Here are the reasons that bit me this week:

1.  Something isn't checking for the coding-system `gnus-decoded',
    which gets propagated as a buffer-local value (typically the
    *Article* buffer) of some coding system variable used by nnmail,
    thus making the error essentially permanent until you kill the
    buffer.

2.  In Japanese mail, ASCII gets RFC2231-encoded under at least some
    circumstances in references to local files in external bodies.
    (All the time for me, but what usually happens is I do
    gnus-mime-save-part-and-strip, then change the name from an
    ISO-2022-JP-encoded abomination -- Gnus didn't seem to pay
    attention to `file-name-coding-system' last I checked -- to an
    ASCII name for safety.  So Gnus may be "remembering" that this
    used to be a name that requires RFC2231, and then does it to ASCII
    even though it's unnecessary and stupid.)

3.  Every time you edit a mail, the RFC2231-encoded parameters get
    re-encoded when you save.  This is particularly annoying when
    somebody sends you a 15-attachment mail.

4.  The code for viewing MIME parts doesn't seem to grok RFC2231;
    clicking on an encoded external body gives a "I can't find any
    file with a name that seems to be RFC2231-encoded" error.

5.  There's no proper way to set the charset parameter when editing a
    message.

6.  Address headers don't get RFC2047-encoded in Japanese mail.  (This
    is a multiple regression, sometimes it works, sometimes it don't,
    lather, rinse, repeat.)

7.  Gnus doesn't seem to provide a useful way to "wash" mail with a
    broken Content-Type header.  I haven't actually checked that in a
    couple years -- I've had a helper function to do that since 1998,
    so it wasn't really worth searching.  And I don't recall exactly
    what my definition of "useful wash" was, I seem to recall there
    was a "view as charset" option but that didn't do what I wanted.

Gnus version 5.10.7, XEmacs package 1.89.

-- 
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: 21.5 crash in Gnus

[Reiner Steib]

On Wed, Mar 15 2006, Stephen J. Turnbull wrote:

>>>>>> "Reiner" == Reiner Steib <reinersteib+gmane@imap.cc> writes:
>
>     Reiner> On Tue, Mar 14 2006, Stephen J. Turnbull wrote:
>
>     >> However, Gnus's I18N misfeatures are legion;
>
>     Reiner> Care to explain?  Or give us some URLs or MIDs?

Is anyone interested in investigating these problems?

> Here are the reasons that bit me this week:
>
> 1.  Something isn't checking for the coding-system `gnus-decoded',
>     which gets propagated as a buffer-local value (typically the
>     *Article* buffer) of some coding system variable used by nnmail,
>     thus making the error essentially permanent until you kill the
>     buffer.
>
> 2.  In Japanese mail, ASCII gets RFC2231-encoded under at least some
>     circumstances in references to local files in external bodies.
>     (All the time for me, but what usually happens is I do
>     gnus-mime-save-part-and-strip, then change the name from an
>     ISO-2022-JP-encoded abomination -- Gnus didn't seem to pay
>     attention to `file-name-coding-system' last I checked -- to an
>     ASCII name for safety.  So Gnus may be "remembering" that this
>     used to be a name that requires RFC2231, and then does it to ASCII
>     even though it's unnecessary and stupid.)
>
> 3.  Every time you edit a mail, the RFC2231-encoded parameters get
>     re-encoded when you save.  This is particularly annoying when
>     somebody sends you a 15-attachment mail.
>
> 4.  The code for viewing MIME parts doesn't seem to grok RFC2231;
>     clicking on an encoded external body gives a "I can't find any
>     file with a name that seems to be RFC2231-encoded" error.
>
> 5.  There's no proper way to set the charset parameter when editing a
>     message.

You can set mm-coding-system-priorities locally, at least in Emacs. 

> 6.  Address headers don't get RFC2047-encoded in Japanese mail.  (This
>     is a multiple regression, sometimes it works, sometimes it don't,
>     lather, rinse, repeat.)
>
> 7.  Gnus doesn't seem to provide a useful way to "wash" mail with a
>     broken Content-Type header.  I haven't actually checked that in a
>     couple years -- I've had a helper function to do that since 1998,
>     so it wasn't really worth searching.  And I don't recall exactly
>     what my definition of "useful wash" was, I seem to recall there
>     was a "view as charset" option but that didn't do what I wanted.
>
> Gnus version 5.10.7, XEmacs package 1.89.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: 21.5 crash in Gnus

[Stephen J. Turnbull]

>>>>> "Reiner" == Reiner Steib <reinersteib+gmane@imap.cc> writes:

    Reiner> On Wed, Mar 15 2006, Stephen J. Turnbull wrote:
    >>>>>>> "Reiner" == Reiner Steib <reinersteib+gmane@imap.cc>
    >>>>>>> writes:

    Reiner> On Tue, Mar 14 2006, Stephen J. Turnbull wrote:

    >>>> However, Gnus's I18N misfeatures are legion;

    Reiner> Care to explain?  Or give us some URLs or MIDs?

    Reiner> Is anyone interested in investigating these problems?

I'm not.  These bugs generally bite me while I'm doing something
moderately destructive such as a save-and-strip-mime, which of course
makes replication impossible.  I've had great trouble understanding
the Gnus code, so it's extremely difficult to localize bugs.

> Here are the reasons that bit me this week:
>
> 1.  Something isn't checking for the coding-system `gnus-decoded',
>     which gets propagated as a buffer-local value (typically the
>     *Article* buffer) of some coding system variable used by nnmail,
>     thus making the error essentially permanent until you kill the
>     buffer.

I haven't seen this recently.

> 3.  Every time you edit a mail, the RFC2231-encoded parameters get
>     re-encoded when you save.  This is particularly annoying when
>     somebody sends you a 15-attachment mail.

Still the case.  Should be easily reproduced.

> 4.  The code for viewing MIME parts doesn't seem to grok RFC2231;
>     clicking on an encoded external body gives a "I can't find any
>     file with a name that seems to be RFC2231-encoded" error.

Still the case.  Should be easily reproduced.

    >> 5.  There's no proper way to set the charset parameter when
    >> editing a message.

    Reiner> You can set mm-coding-system-priorities locally, at least
    Reiner> in Emacs.

You call that "proper"?  I'll try it the next time it's relevant, but
it is not a reasonable interface.  Editing the mail in a separate
buffer in fundamental mode is far more convenient and reliable.

> 6.  Address headers don't get RFC2047-encoded in Japanese mail.  (This
>     is a multiple regression, sometimes it works, sometimes it don't,
>     lather, rinse, repeat.)

Haven't had a problem with this recently.

> 7.  Gnus doesn't seem to provide a useful way to "wash" mail with a
>     broken Content-Type header.

Haven't run into this in a while.

New one:

8.  Message mode handles headers of the form

    To: taro@tsukuba.net (筑波太郎)

    improperly, where the header was automatically generated as a
    reply, and the content of the comment was ISO-2022-JP-encoded
    Japanese.  It produces this:

    To: taro@tsukuba.net =?ISO-2022-JP?B?KBskQkNeR0hCQE86GyhCKQ==?=

    which is of course an erroneous address according to RFC 2822.

-- 
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: 21.5 crash in Gnus

[TSUCHIYA Masatoshi]

Hi,

>> On Wed, 19 Jul 2006 18:47:04 +0200
>> reinersteib+gmane@imap.cc (Reiner Steib) said as follows:

>> 5.  There's no proper way to set the charset parameter when editing a
>>     message.

Insert the following MML tag at the top of the editing message, and
you can write a message encoded in your favorite encoding.

    <#part type="text/plain" disposition=inline charset="example">
                                                          ^^^^^^^
If you edit a message in the buffer where `mml-mode' is activated, you
can use various helper commands to insert MML tags.

Of course, it should be discussed whether this method is proper or not.

Regards,

-- 
TSUCHIYA Masatoshi

Re: 21.5 crash in Gnus

[Stephen J. Turnbull]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

>>>>> "TSUCHIYA" == TSUCHIYA Masatoshi <tsuchiya@namazu.org> writes:

    TSUCHIYA> Insert the following MML tag at the top of the editing
    TSUCHIYA> message, and you can write a message encoded in your
    TSUCHIYA> favorite encoding.

    TSUCHIYA>     

[-- Attachment #2: Type: text/plain, Size: 385 bytes --]


I don't want to do that.  I want to fix a bogus global charset, not
create a new MIME part.

-- 
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Re: 21.5 crash in Gnus

[TSUCHIYA Masatoshi]

Hi,

>> On Thu, 20 Jul 2006 12:56:19 +0900
>> "sjt" == stephen@xemacs.org ("Stephen J. Turnbull") said as follows:

sjt> I don't want to do that.  I want to fix a bogus global charset,
sjt> not create a new MIME part.

Is a new MIME part required?  As this message shows, we can create a
message which consists of a single MIME part encoded in our favorite
encoding, if we prepare a message buffer looks like:

  To: foo
  Subject: test
  From: bar
  --text follows this line--
  <#part type="text/plain" disposition=inline charset="iso-8859-1">

`charset' parameter of this MML tag is used to specify the `charset'
parameter of the top level `Content-Type' header.

I hope that this is what you want to do.

Regards,

-- 
TSUCHIYA Masatoshi

Re: 21.5 crash in Gnus

[Miles Bader]

TSUCHIYA Masatoshi <tsuchiya@namazu.org> writes:
>   <#part type="text/plain" disposition=inline charset="iso-8859-1">
>
> `charset' parameter of this MML tag is used to specify the `charset'
> parameter of the top level `Content-Type' header.

Yes that works quite well, no mime parts created, correct mime headers
generated...

Maybe there should be a key-binding to prompt for the charset and insert
the tag though?  [At the beginning of the buffer?]

-Miles

-- 
Yo mama's so fat when she gets on an elevator it HAS to go down.

Re: 21.5 crash in Gnus

[Maciej Matysiak]

[-- Attachment #1: Type: text/plain, Size: 505 bytes --]

On the 20th of July 2006 at 14:23, Miles Bader <miles#gnu.org> wrote:

>> <#part type="text/plain" disposition=inline charset="iso-8859-1">
> Maybe there should be a key-binding to prompt for the charset and insert
> the tag though?  [At the beginning of the buffer?]

I use the following function to add proper header to messages, which should
be sent as unicode:

(defun mm:add-tags-with-utf ()
  "Insert tags with utf-8 encoding."
  (interactive)
  (save-excursion
    (message-goto-body)
    (insert "

[-- Attachment #2: Type: text/plain, Size: 38 bytes --]

\n")
    (end-of-buffer)
    (insert "

[-- Attachment #3: Type: text/plain, Size: 130 bytes --]

")
    ))

(define-key message-mode-map [(f5)] 'mm:add-tags-with-utf)

Modify to your needs.

hth,
 m.m.
-- 
 use gnus, not guns!

Re: 21.5 crash in Gnus

[Reiner Steib]

On Thu, Jul 20 2006, Miles Bader wrote:

> TSUCHIYA Masatoshi <tsuchiya@namazu.org> writes:
>>   <#part type="text/plain" disposition=inline charset="iso-8859-1">
>>
>> `charset' parameter of this MML tag is used to specify the `charset'
>> parameter of the top level `Content-Type' header.
>
> Yes that works quite well, no mime parts created, correct mime headers
> generated...

But if you include characters that are not present in this charset,
those will be replaced by question marks, see
<news:v9y7uo3xed.fsf@marauder.physik.uni-ulm.de> in gmane.test.

> Maybe there should be a key-binding to prompt for the charset and insert
> the tag though?  [At the beginning of the buffer?]

Except for testing, I never had the need to modify the charset beyond
setting `mm-coding-system-priorities'.  I wonder why people have the
need to specify the charset in the first place.

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: 21.5 crash in Gnus

[TSUCHIYA Masatoshi]

>> On Thu, 20 Jul 2006 21:18:43 +0200
>> reinersteib+gmane@imap.cc (Reiner Steib) said as follows:

>>> `charset' parameter of this MML tag is used to specify the `charset'
>>> parameter of the top level `Content-Type' header.
>>
>> Yes that works quite well, no mime parts created, correct mime headers
>> generated...

>But if you include characters that are not present in this charset,
>those will be replaced by question marks, see
><news:v9y7uo3xed.fsf@marauder.physik.uni-ulm.de> in gmane.test.

Yes, of course.

If a user specify his/her favorite encoding, it is his/her job to
select an encoding that can represent all characters in his/her
message, I think.  If a user specify no encoding, it is Gnus's job to
select an appropriate one according to `mm-coding-system-priorities'.

>I wonder why people have the need to specify the charset in the first
>place.

I show an example.  Because almost of all messages written in Japanese
are encoded in ISO-2022-JP, mail users in Japan will select
ISO-2022-JP as an normal-use encoding as Gnus also selects.
Unfortunately, a broken mailer cannot process messages encoded in
ISO-2022-JP, but can process messages in UTF-8.  When a user has to
send a message to another user using that broken mailer, he/she will
want to select UTF-8 temporally.

-- 
TSUCHIYA Masatoshi

Re: 21.5 crash in Gnus

[Stephen J. Turnbull]

>>>>> "Reiner" == Reiner Steib <reinersteib+gmane@imap.cc> writes:

    Reiner> Except for testing, I never had the need to modify the
    Reiner> charset beyond setting `mm-coding-system-priorities'.  I
    Reiner> wonder why people have the need to specify the charset in
    Reiner> the first place.

I've probably misspoken.

The problem that I run into regularly (not always, but a few times a
month) is that when I gnus-mime-save-part-and-strip, I get a message
edit buffer after the part is saved.  It looks OK, so I C-c C-c to
save it.  Then Gnus gets confused about the charset of the text
content of the message and misencodes it.  The net result is that the
message is unreadable.  Any attempt to edit the message using "e"
gives the same (or worse) results.

I don't have a test case to hand, so I can't be more precise.


-- 
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.

Specifying a charset (per group) (was: 21.5 crash in Gnus)

[Reiner Steib]

On Fri, Jul 21 2006, TSUCHIYA Masatoshi wrote:

>>> On Thu, 20 Jul 2006 21:18:43 +0200
>>> reinersteib+gmane@imap.cc (Reiner Steib) said as follows:

>> On Thu, Jul 20 2006, Miles Bader wrote:
> Maybe there should be a key-binding to prompt for the charset and insert
> the tag though?  [At the beginning of the buffer?]

>>> I wonder why people have the need to specify the charset in the first
>>> place.

> I show an example.  Because almost of all messages written in Japanese
> are encoded in ISO-2022-JP, mail users in Japan will select
> ISO-2022-JP as an normal-use encoding as Gnus also selects.
> Unfortunately, a broken mailer cannot process messages encoded in
> ISO-2022-JP, but can process messages in UTF-8.  When a user has to
> send a message to another user using that broken mailer, he/she will
> want to select UTF-8 temporally.

Would it make sense to introduce a `mm-group-coding-system-alist'
(similar to `gnus-group-posting-charset-alist', cf. (info
"(gnus)Charsets")) which takes precedence over the entries in
`mm-coding-system-priorities'.

Example:

(setq mm-coding-system-priorities '(iso-8859-1 iso-2022-jp utf-8))
(setq mm-group-coding-system-alist '(("not-japanese" utf-8)))

(let ((mm-coding-system-priorities
       ;; entry from `mm-group-coding-system-alist' if non-nil -> utf-8
       (push 'utf-8 mm-coding-system-priorities)))
  do-stuff)

What do people think?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Unreadable messages after gnus-mime-save-part-and-strip in XEmacs (was: 21.5 crash in Gnus)

[Reiner Steib]

On Fri, Jul 21 2006, Stephen J. Turnbull wrote:

> The problem that I run into regularly (not always, but a few times a
> month) is that when I gnus-mime-save-part-and-strip, I get a message
> edit buffer after the part is saved.  It looks OK, so I C-c C-c to
> save it.  Then Gnus gets confused about the charset of the text
> content of the message and misencodes it.  The net result is that the
> message is unreadable.  

Hm, to my knowledge Gnus runs the buffer through the same MIME
encoding mechanisms as when composing a regular message.

> Any attempt to edit the message using "e" gives the same (or worse)
> results.
>
> I don't have a test case to hand, so I can't be more precise.

Maybe you'll be able to save the raw buffer some day so we might be
able to reproduce it.

Does anyone else see such problems?

Bye, Reiner.
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

Re: Unreadable messages after gnus-mime-save-part-and-strip in

[Stephen J. Turnbull]

>>>>> "Reiner" == Reiner Steib <reinersteib+gmane@imap.cc> writes:

    Reiner> Hm, to my knowledge Gnus runs the [message edit] buffer
    Reiner> through the same MIME encoding mechanisms as when
    Reiner> composing a regular message.

It's not reasonable to assume any regularity at all when dealing with
Japanese email (unless you've constructed it yourself).  Of course 99%
of it is OK, but that last 1% is almost as bad as spam, except that if
you simply decode all the base64 and MIME words, Emacs autodetection
in a Japanese language environment will give sensible results (but
sometimes conflicting with any explicit charset parameters!)

I don't expect Gnus to deal with this crap automatically, but it's
very frustrating to have Gnus corrupt my edits because it thinks it
knows better than me.  Especially since both the presentation and the
message edit buffer look sane---until Gnus saves the edited message.

    Reiner> Maybe you'll be able to save the raw buffer some day so we
    Reiner> might be able to reproduce it.

Maybe, but often these mails contain sensitive internal documents
(that's why I'm save-and-stripping them).

    Reiner> Does anyone else see such problems?

For me it's always connected with a save-and-strip operation, but that
may be because I almost never edit a received message for any other
reason.

-- 
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba                    Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
               Ask not how you can "do" free software business;
              ask what your business can "do for" free software.