\201 irritation! :-)

\201 irritation! :-)

[François Pinard]

Hi, people.  Let me share a quick hack which will make my day happier! :-)


(defun fp-gnus-article-hide-pesky-201s ()
  "Hide these pesky \201, which pop all over in French Babyl email."
  (interactive)
  (save-excursion
    (set-buffer gnus-article-buffer)
    (let ((buffer-read-only nil)
	  (inhibit-point-motion-hooks t))
      (goto-char (point-min))
      (set-buffer-multibyte nil)
      (while (search-forward "\201" nil t)
	(delete-char (- (skip-chars-forward "\201"))))
      (set-buffer-multibyte t))))
(add-hook 'gnus-article-prepare-hook 'fp-gnus-article-hide-pesky-201s)

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

Re: \201 irritation! :-)

[François Pinard]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

[François Pinard]

> Hi, people.  Let me share a quick hack which will make my day happier! :-)
> (defun fp-gnus-article-hide-pesky-201s () [...]

Hey, hey.  I also wrote a Python script to clean out those \201s in
my existing message files, and guess what, the script found 45478 such
characters.

No wonder why I was seeing these regularly.  They multiply like rabbits on my
disks, probably in the dark, at night, while my home computer is shut down.

Here is the little script, who knows, who might be useful to others as well.
I just `locate' or `find' message files, through `... | xargs un201'.


[-- Attachment #2: un201 --]
[-- Type: text/plain, Size: 2548 bytes --]

#!/usr/bin/env python
# Copyright © 2000 Progiciels Bourbeau-Pinard inc.
# François Pinard <pinard@iro.umontreal.ca>, 2000.

"""\
Remove all Mule-ish \201 characters, escaped from Mule, in given files.

Usage: un201 [OPTION]... [FILE]...

   -k  Keep the first \201 in a string of those
   -v  Be verbose about deleted characters

If no file is given, act as a filter.
"""

import getopt, os, string, sys

class run:
    keep_one = 0
    verbose = 0

def main(*arguments):
    if not arguments:
        sys.stdout.write(__doc__)
        sys.exit(0)
    options, arguments = getopt.getopt(arguments, 'kv')
    for option, value in options:
        if option == '-k':
            run.keep_one = 1
        elif option == '-v':
            run.verbose = 1
    if not arguments:
        write = sys.stderr.write
        input = sys.stdin.read()
        output = process(input)
        count = len(input) - len(output)
        if run.verbose and count > 0:
            write("Deleted %d Mule-ish \\201 characters\n" % count)
        sys.stdout.write(output)
    else:
        write = sys.stdout.write
        for argument in arguments:
            if os.path.isdir(argument):
                write("%s: Directory, skipped\n" % argument)
                continue
            if not os.path.isfile(argument):
                write("%s: File does not exist, skipped\n" % argument)
                continue
            input = open(argument).read()
            output = process(input)
            count = len(input) - len(output)
            if run.verbose and count > 0:
                write("%s: Deleted %d Mule-ish \\201 characters\n"
                      % (argument, count))
                backup = '%s~' % argument
                work = '%s-tmp' % argument
                open(work, 'w').write(output)
                try:
                    os.remove(backup)
                except OSError:
                    pass
                os.rename(argument, backup)
                os.rename(work, argument)

def process(buffer):
    fragments = []
    start = 0
    while start >= 0:
        position = string.find(buffer, '\201', start)
        if position < 0:
            fragments.append(buffer[start:])
            break
        if run.keep_one:
            position = position + 1
        fragments.append(buffer[start:position])
        start = position
        while start < len(buffer) and buffer[start] == '\201':
            start = start + 1
    return string.join(fragments, '')

if __name__ == '__main__':
    apply(main, sys.argv[1:])

[-- Attachment #3: Type: text/plain, Size: 59 bytes --]


-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

Re: \201 irritation! :-)

[Kai Großjohann]

On 27 Aug 2000, François Pinard wrote:

> No wonder why I was seeing these regularly.  They multiply like
> rabbits on my disks, probably in the dark, at night, while my home
> computer is shut down.

And Saint IGNUtius said unto them: Go forth, be fruitful, and multiply!

kai
-- 
I like BOTH kinds of music.

Re: \201 irritation! :-)

[Stainless Steel Rat]

* François Pinard <pinard@iro.umontreal.ca>  on Sun, 27 Aug 2000
| Hi, people.  Let me share a quick hack which will make my day happier! :-)

*sigh*

$ ./configure --with-mule=no
$ make
$ make install

Haven't seen a \201 yet.
-- 
Rat <ratinox@peorth.gweep.net>    \ Happy Fun Ball contains a liquid core,
Minion of Nathan - Nathan says Hi! \ which, if exposed due to rupture, should
PGP Key: at a key server near you!  \ not be touched, inhaled, or looked at.

Re: \201 irritation! :-)

[François Pinard]

[Stainless Steel Rat]

> * François Pinard <pinard@iro.umontreal.ca>  on Sun, 27 Aug 2000
> | Hi, people.  Let me share a quick hack which will make my day happier! :-)

> *sigh*

> $ ./configure --with-mule=no
> $ make
> $ make install

> Haven't seen a \201 yet.

Maybe, but you probably haven't seen Korean, Polish or Japanese either...

I cannot honestly be interested in internationalisation of software, without
taking the opportunities Mule gives me.  I confess that Mule is sometimes
irritating, and some days more than others :-), but I still prefer to live
with the irritation, and contribute in some way to get the problem corrected.

Not that I have much personal need for international characters, besides
French.  But I really think people should be entitled, in the long run, in
writing at least their own name using the appropriate script, in all cases.
I also like naming people using the proper script when writing to them or
speaking about them to others, it looks like elementary politeness to me.

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

Re: \201 irritation! :-)

[Stainless Steel Rat]

* François Pinard <pinard@iro.umontreal.ca>  on Mon, 28 Aug 2000
| Maybe, but you probably haven't seen Korean, Polish or Japanese either...

True, but I don't need to :).

| I cannot honestly be interested in internationalisation of software, without
| taking the opportunities Mule gives me.  I confess that Mule is sometimes
| irritating, and some days more than others :-), but I still prefer to live
| with the irritation, and contribute in some way to get the problem corrected.

Then get on RMS's case to fix the bugs in MULE once and for all.
-- 
Rat <ratinox@peorth.gweep.net>    \ If Happy Fun Ball begins to smoke, get
Minion of Nathan - Nathan says Hi! \ away immediately. Seek shelter and cover
PGP Key: at a key server near you!  \ head.

Re: \201 irritation! :-)

[François Pinard]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 676 bytes --]

[Stainless Steel Rat]

> Then get on RMS's case to fix the bugs in MULE once and for all.

Richard does not take direct care of Emacs anymore.  Gerd does it,
and I presume Handa-san (åç぀‚“) still takes care of Mule matters.

My main difficulty in reporting bugs is extracting them from my heavily
customised Emacs set-up, so they may be reproduced by me, and at the
other end.  But with enough time and courage, I'll report everything I see.
We should be more at doing this, it will help things to be corrected faster.
In my experience, involved people are responsive, and much more than I am!

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

Re: \201 irritation! :-)

[Dave Love]

Come on!  Why do you feel the need to mangle a multibyte buffer like
that, apart from the fact that the best way to not see characters
would be via the display table?  (And what about correspondents with
Latin-2 (&c) names?)  If there's some bug in Emacs or Gnus, please
address it!

However, nnbabyl works OK for me with the only multilingual rmail file
I have (one that came from rms), though your messages in it had a
bogus signature then, obviously inserted in emacs-mule encoding.

Babyl files are emacs-mule encoded so that they work with messages in
different character sets.  Gnus displays them as such for me with the
contents re-encoded on display.  I assume you've managed to mangle the
file somehow or the displayed article (using TM, perhaps?) by multiple
decoding.  Use M-x find-file-literally to check file contents and
inspect Gnus' article-munging hooks.

Re: \201 irritation! :-)

[Dave Love]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

>>>>> "FP" == François Pinard <pinard@iro.umontreal.ca> writes:

 FP> [Stainless Steel Rat]
 >> Then get on RMS's case to fix the bugs in MULE once and for all.

This continued spouting of rubbish about Mule is tiresome, but I
suppose evidence mustn't intrude.  (It's rubbish about XEmacs as well
as Emacs, by the way.)

 FP> Richard does not take direct care of Emacs anymore.  Gerd does it,
 FP> and I presume Handa-san (å

[-- Attachment #2: Type: text/plain, Size: 1 bytes --]

ç

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: Type: text/plain, Size: 275 bytes --]

぀‚“) still takes
                              ^^^^^^^^
Some mishtake shurely?  The raw bytes there didn't appear to be
UTF-8.  Do others with Mule-UCS see them as such?

 FP> care of Mule matters.

Correct, but there's no evidence yet of an Emacs bug to fix here.

Re: \201 irritation! :-)

[Stainless Steel Rat]

* Dave Love <d.love@dl.ac.uk>  on Tue, 05 Sep 2000
| This continued spouting of rubbish about Mule is tiresome,

What, fixing bugs is rubbish?

| but I suppose evidence mustn't intrude.  (It's rubbish about XEmacs as
| well as Emacs, by the way.)

I have yet to hear of any \201 bugs in any XEmacs built with MULE, and an
XEmacs built without MULE cannot exhibit these bugs.  Nothing about XEmacs
throwing \201 bugs in gnu.emacs.gnus, nothing about \201 that I can recall
in comp.emacs.xemacs.

[...]
| Correct, but there's no evidence yet of an Emacs bug to fix here.

If that were true then there would be no appearance of \201.  The fact that
it did appear is proof that there are bugs in the FSF Emacs implementation
of MULE, and I will continue to offer XEmacs as a solution to those bugs
until such time as they are fixed.  And if your attitude accurately
reflects that of the FSF then you'd better get used to me doing so, because
while free software is important, software that works correctly is more
important.
-- 
Rat <ratinox@peorth.gweep.net>    \ Do not use Happy Fun Ball on concrete.
Minion of Nathan - Nathan says Hi! \ 
PGP Key: at a key server near you!  \ 

Re: \201 irritation! :-)

[François Pinard]

[Dave Love]

> Why do you feel the need to mangle a multibyte buffer like that, apart
> from the fact that the best way to not see characters would be via the
> display table?

Because I'm not always using Emacs to process these files, and eradicating
spurious \201 early slow down their multiplication at save time.

> If there's some bug in Emacs or Gnus, please address it!

I'm not competent enough to do so.

> I assume you've managed to mangle the file somehow or the displayed article
> (using TM, perhaps?) by multiple decoding.

I did not manage to do that.  But I managed to solve the problems I had,
and it might be a side effect of my ways of doing it.

When I inspect a file and see six or more \201 in a row, before the accented
letter one of them applies to, it hardly looks like sanity to me :-).
I may stumble in my methods, of course, but that is a big stumble then,
which I do not know how to avoid.

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

Re: \201 irritation! :-)

[François Pinard]

[Dave Love]

> This continued spouting of rubbish about Mule is tiresome,

Tiresome, just like the continued spouting of rubbish _from_ Mule. :-)

>  FP> Richard does not take direct care of Emacs anymore.  Gerd does it,
>  FP> and I presume Handa-san (åçã\201\200\202\223 still takes
>                               ^^^^^^^^

(I replaced quoted dangerous characters with their explicit \NNN display).

> Some mishtake shurely?  The raw bytes there didn't appear to be
> UTF-8.

Yes, I noticed this.  After having installed Otfried UTF-8 support, Gnus
apparently started to use it automatically, and something appears to be
wrong somewhere in the process.  What I manage to do is installing that
UTF-8 support, but I do not feel like being _the_ bug! :-)

>  FP> care of Mule matters.

> Correct, but there's no evidence yet of an Emacs bug to fix here.

We agree on that evidence that there are a few problems in this area.
At least one evident Emacs bug, to me, is insufficient documentation about
Mule matters -- and documentation bugs are bugs.  I hope to be wrong and
that documentation does exist, with me just not knowing where to look at.

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

Re: \201 irritation! :-)

[Kai Großjohann]

On 06 Sep 2000, François Pinard wrote:

> [Dave Love]
> 
>> If there's some bug in Emacs or Gnus, please address it!
> 
> I'm not competent enough to do so.

It would be very helpful, though, if you could find out some way to
reproduce the problem.  For instance, I wonder when does a \201
appear?  You might take note of a \201 appearing in a new message.
Maybe you know what you did with that message before?

Or you could try to take some of those \201 infected messages and
write them to a file, then use `G f' on that file to see what
happens.  I'd think that up to one \201 before a Latin-1 character is
okay -- Emacs might have used the emacs-mule coding system for writing
the file.

Since you have some automatic stuff to deal with \201, could you
enhance that to log the \201s it sees?  Especially in fresh files?
Does that turn up anything useful?

kai
-- 
I like BOTH kinds of music.

Re: \201 irritation! :-)

[Yair Friedman (Jerusalem)]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> On 06 Sep 2000, François Pinard wrote:
> 
> > [Dave Love]
> > 
> >> If there's some bug in Emacs or Gnus, please address it!
> > 
> > I'm not competent enough to do so.
> 
> It would be very helpful, though, if you could find out some way to
> reproduce the problem.

I seem to have a reliable way to reproduce the 201 problem.  Attached
is one of François's recent mail.

[-- Attachment #2: 29640.gz --]
[-- Type: application/zip, Size: 1655 bytes --]

[-- Attachment #3: Type: text/plain, Size: 278 bytes --]

The last line contains his name in QP although there is no clue for it in
the headers (Maybe M$-exchange is doing something to the mail?).

It looks like this : Fran=C3=A7ois or Fran<equals>C3<equals>A7ois

Pressing Wq (gnus-article-de-quoted-unreadable) creates the following:

[-- Attachment #4: with201.txt.gz --]
[-- Type: application/zip, Size: 85 bytes --]

[-- Attachment #5: Type: text/plain, Size: 403 bytes --]


It looks like this: Fran\201Ã\201§ois or Fran<201><A-tilde><201><section>ois

Although the \201 are created, they are not saved.  This is a bug
nevertheless.

My emacs-version is GNU Emacs 20.6.1 (i386-*-nt4.0.1381) The NT-Emacs
Pre-compiled version.  I can't compile it over here.
Gnus v5.8.7 snapshot of May 17th.

Please let me know how can I proceed tracing the bug.
--
Yair Friedman.

Re: \201 irritation! :-)

[Janne Rinta-Manty]

Another \201 report: I get \201s in the Summary buffer if the
message's headers have unencoded iso-8859-1 characters (with and
without "correct" Content-Type etc. headers). Article buffer has no
\201s.

For example, subject "n\201äkyyk\201ö 201?" in the summary is
displayed as "näkyykö 201?" in the article buffer.

The \201s appear only in nnimap summary buffers, nntp is OK. (I have
only nnimap and nntp groups here, so don't know about other backends.)

nnimap-stream is shell, gnus-group-charset-alist is the default.
GNU emacs 20.3.1, Gnus 5.8.8 (2000-08-21)

-- 
Janne Rinta-Mänty

Re: \201 irritation! :-)

[Simon Josefsson]

Janne Rinta-Manty <rintaman@cs.helsinki.fi> writes:

> Another \201 report: I get \201s in the Summary buffer if the
> message's headers have unencoded iso-8859-1 characters (with and
> without "correct" Content-Type etc. headers). Article buffer has no
> \201s.

Fixed, thanks.

(Unencoded non-ascii characters in headers are invalid.  But now you
shouldn't get \201's at least.  What symbols the character codes will
show up as will depend on what Mule implementation and configuration
you use, but that is as you would expect.)

Re: \201 irritation! :-)

[Dave Love]

>>>>> "FP" == François Pinard <pinard@iro.umontreal.ca> writes:

 FP> Because I'm not always using Emacs to process these files, and
 FP> eradicating spurious \201 early slow down their multiplication at
 FP> save time.

I suppose you can sacrifice multilingualization by setting
`rmail-file-coding-system' if you must.  Presumably you could maintain
multilingualization by using utf-8 if that works properly.

 >> If there's some bug in Emacs or Gnus, please address it!

 FP> I'm not competent enough to do so.

I mean help to get it fixed.  I would be expected to do that with,
say, recode even if I wasn't familiar with the code.

 >> I assume you've managed to mangle the file somehow or the
 >> displayed article (using TM, perhaps?) by multiple decoding.

 FP> I did not manage to do that.  But I managed to solve the problems
 FP> I had, and it might be a side effect of my ways of doing it.

 FP> When I inspect a file and see six or more \201 in a row, before
 FP> the accented letter one of them applies to,

So the file _has_ been mangled by multiple decoding somewhere.  For a
start, has Gnus even been writing to it, or only reading?  That would
help to narrow it down.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "FP" == François Pinard <pinard@iro.umontreal.ca> writes:

 FP> Tiresome, just like the continued spouting of rubbish _from_ Mule. :-)

You mean it's your application producing rubbish.  Bugs where Mule
doesn't do what you tell it seem to be rather rare.  Of course the
software can be fixed, unlike the black propaganda merchants.

 FP> (I replaced quoted dangerous characters with their explicit \NNN
 FP> display).

[They may have got perturbed further when Gnus tried to prevent me
posting with bogus characters.]

 FP> Yes, I noticed this.  After having installed Otfried UTF-8
 FP> support, Gnus apparently started to use it automatically, and
 FP> something appears to be wrong somewhere in the process.

It would presumably be helpful to know where.  Perhaps it doesn't
happen with the normal Mule-UCS, for instance.

 FP> At least one evident Emacs bug, to me, is insufficient
 FP> documentation about Mule matters -- and documentation bugs are
 FP> bugs.

Even if it's not actually a bug in rms's terms, it's doubtless a
deficiency and is taken seriously.  Please make a bug report about
specific things that are missing.  [rms is currently agitating
particularly for doc on input methods, and after people to help.]

Re: \201 irritation! :-)

[Dave Love]

It's probably not really worth replying to Proof by Disingenuous Non
Sequitur per textbooks on Generalized Logic, but...

>>>>> "Rat" == Stainless Steel Rat <ratinox@peorth.gweep.net> writes:

 Rat> I have yet to hear of any \201 bugs in any XEmacs built with
 Rat> MULE, 

Try paying more attention.

 Rat> and an XEmacs built without MULE cannot exhibit these bugs.

Presumably there's some problem with XEmacs/Mule that requires doing
that.  A unibyte Emacs doesn't generate emacs-mule coded text unless
there's a broken Mule-aware application running in it.  Anyone stating
otherwise should produce an evaluation in Emacs to back that up.

 Rat> Nothing about XEmacs throwing \201 bugs in gnu.emacs.gnus,
 Rat> nothing about \201 that I can recall in comp.emacs.xemacs.

Yes, people with a clue know that coding conversion errors in XEmacs
normally will result in ISO 2022 escapes -- as reported here -- or
just trashed non-ASCII characters.  There is no difference in
principle.  The vast majority of problems with Gnus are due to people
mistakenly running TM and possibly similar broken applications, as is
usually verifiable from their headers.  We see precious few bug
reports otherwise.  XEmacs users see \201 and other junk in Gnus for
various reasons, as in this thread.

 Rat> If that were true then there would be no appearance of \201.
 Rat> The fact that it did appear is proof that there are bugs in the
 Rat> FSF Emacs implementation of MULE, 

It's difficult to see how frobbing Gnus' coding conversion would fix
things if that wasn't rubbish.  It's there in the development history.
Genuine coding conversion bugs occur occasionally in quite obscure
circumstances and are fixed quickly, usually by Handa-san as they are
reported.  The record of bug reports and change logs shows so.

 Rat> and I will continue to offer XEmacs as a solution to those bugs
 Rat> until such time as they are fixed.

Do report on a single unfixed coding conversion bug in Emacs itself.
XEmacs/Mule isn't even available on the most widespread platform and
certainly has major known outstanding bugs which I've no wish to bash.
Hrvoje has written eloquently on the topic.

 Rat> And if your attitude accurately reflects that of the FSF
 Rat> then you'd better get used to me doing so, because while free
 Rat> software is important, software that works correctly is more
 Rat> important.

Lying about things isn't going to make any software work better.  I'm
baffled by the implication that XEmacs isn't free and anyone clueful
knows it has advantages over Emacs _and vice versa_.  As far as I
know, the attitude of the XEmacs maintainers is similar to that of the
Emacs maintainers -- soliciting reports of bugs and fixing them;
otherwise they make improvements, like importing Emacs Mule code and
techniques, so I suppose XEmacs is getting worse in Pieri's terms.
They don't appear to agree with him.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "YF" == Yair Friedman (Jerusalem) <YAIRFR@Amdocs.com> writes:

 YF> I seem to have a reliable way to reproduce the 201 problem.  Attached
 YF> is one of François's recent mail.

As I pointed out, that had garbled utf-8.

 YF> The last line contains his name in QP although there is no clue
 YF> for it in the headers (Maybe M$-exchange is doing something to
 YF> the mail?).

Yes.  I guess all bets are off if it's Exchange...

 YF> It looks like this: Fran\201Ã\201§ois or
 YF> Fran<201><A-tilde><201><section>ois

Presumably you aren't using Mule-UCS.

 YF> Although the \201 are created, they are not saved.

I'm not sure what that means.

 YF> This is a bug nevertheless.

It's likely Gnus should be more careful here -- I think UTF-meister
Florian has some pending fixes which might be relevant -- but it isn't
an Emacs bug as such.  The raw article is stored in emacs-mule coding
and then explicitly encoded, which can't be done properly if you don't
have a utf coding system.

 YF> Please let me know how can I proceed tracing the bug.

Thanks for the positive contribution.

Re: \201 irritation! :-)

[Stainless Steel Rat]

* Dave Love <d.love@dl.ac.uk>  on Fri, 08 Sep 2000
| It's probably not really worth replying to Proof by Disingenuous Non
| Sequitur per textbooks on Generalized Logic, but...

Pot.  Kettle.  Black.
-- 
Rat <ratinox@peorth.gweep.net>    \ Happy Fun Ball contains a liquid core,
Minion of Nathan - Nathan says Hi! \ which, if exposed due to rupture, should
PGP Key: at a key server near you!  \ not be touched, inhaled, or looked at.

Re: \201 irritation! :-)

[Yair Friedman (Jerusalem)]

Dave Love <d.love@dl.ac.uk> writes:

> >>>>> "YF" == Yair Friedman (Jerusalem) <YAIRFR@Amdocs.com> writes:
> 
>  YF> I seem to have a reliable way to reproduce the 201 problem.  Attached
>  YF> is one of François's recent mail.
> 
> As I pointed out, that had garbled utf-8.

Hmm.. Here is another example:
Harry Putnam's message "Re: Attached pgp and mailcrypt"
<m2vgw2cskc.fsf@gnus-5.8.8-cvs.now.playing>

has Kai Großjohann's name QP-coded (Exchange again?) as Gro=DFjohann W q
gives Gro\201ßjohann.

I'm not sure if it's the same problem I thought that you said the
previous problem related to Mule-UCS?  I don't see anything indicating
Harry is using Mule-UCS, maybe I don't know where to look.

Re: \201 irritation! :-)

[Kai Großjohann]

On 12 Sep 2000, Yair Friedman wrote:

> Hmm.. Here is another example:
> Harry Putnam's message "Re: Attached pgp and mailcrypt"
> <m2vgw2cskc.fsf@gnus-5.8.8-cvs.now.playing>
> 
> has Kai Großjohann's name QP-coded (Exchange again?) as Gro=DFjohann
> W q gives Gro\201ßjohann.

Fascinating.  I don't have to do anything at all to see Harry's
message correctly.  Hm.

I wonder if it's that I use 5.8.8 from CVS, or the different Emacs
version?  I'd be surprised if it was the latter, though.

kai
-- 
I like BOTH kinds of music.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "YF" == Yair Friedman (Jerusalem) <YAIRFR@Amdocs.com> writes:

 YF> has Kai Großjohann's name QP-coded (Exchange again?) 

Must be.

 YF> as Gro=DFjohann W q gives Gro\201ßjohann.

I can't reproduce that in Emacs 20.7 with W q on your article.  I
doubt it's a DozeN'T-specific Emacs problem, but it could be.  You're
not looking at the raw article buffer by any chance?

 YF> I'm not sure if it's the same problem I thought that you said the
 YF> previous problem related to Mule-UCS?

It's something different.

Re: \201 irritation! :-)

[Yair Friedman (Jerusalem)]

Dave Love <d.love@dl.ac.uk> writes:

> >>>>> "YF" == Yair Friedman (Jerusalem) <YAIRFR@Amdocs.com> writes:
> 
>  YF> has Kai Großjohann's name QP-coded (Exchange again?) 
> 
> Must be.
> 
>  YF> as Gro=DFjohann W q gives Gro\201ßjohann.
> 
> I can't reproduce that in Emacs 20.7 with W q on your article.  I
> doubt it's a DozeN'T-specific Emacs problem, but it could be.  You're
> not looking at the raw article buffer by any chance?

With the help of Kai (Thanks!) I found out that the crippled software
omitted or forgot to inset CTE header.  After manually inserting it,
everything is displayed well *before* and after Wq.

As for the malformed message, WIBNI gnus warn me about it and possibly
let me edit the message?

Nevertheless, gnus-article-de-quoted-unreadable should know how to handle
this malformed massage -- this is the whole purpose of it.

I'm not sure that the possible misbehavior of Wq triggers an Emacs bug
over here, or maybe it's just a Gnus bug only.
--
Yair Friedman.

Re: \201 irritation! :-)

[ShengHuo ZHU]

"Yair Friedman (Jerusalem)" <YAIRFR@Amdocs.com> writes:

> As for the malformed message, WIBNI gnus warn me about it and possibly

In this case, you'd better use your own human intelligence instead of
the artificial one.

> let me edit the message?

You could use `1 e' to edit raw messages.

> Nevertheless, gnus-article-de-quoted-unreadable should know how to handle
> this malformed massage -- this is the whole purpose of it.
> I'm not sure that the possible misbehavior of Wq triggers an Emacs bug
> over here, or maybe it's just a Gnus bug only.

Wq is supposed to do so and, AFAIK, no bug is found for Emacs build-in
coding-system.  I know there is some problems with copying utf-8
encoded texts between buffers in Emacs.  It could a bug of Emacs or
Mule-UCS.  Hope this won't happen in the future Unicode Emacs.

ShengHuo

Re: \201 irritation! :-)

[Dave Love]

>>>>> "YF" == Yair Friedman (Jerusalem) <YAIRFR@Amdocs.com> writes:

 YF> With the help of Kai (Thanks!) I found out that the crippled
 YF> software omitted or forgot to inset CTE header.  After manually
 YF> inserting it, everything is displayed well *before* and after Wq.

I've no idea why that should make a difference.  There was no CTE when
I tried it.  Can you debug it or find a way for us to reproduce it?

Re: \201 irritation! :-)

[Dave Love]

>>>>> "ZSH" == ShengHuo ZHU <zsh@cs.rochester.edu> writes:

 ZSH> Wq is supposed to do so and, AFAIK, no bug is found for Emacs
 ZSH> build-in coding-system.  I know there is some problems with
 ZSH> copying utf-8 encoded texts between buffers in Emacs.

That doesn't seem to be the case here, but do you have details that we
can investigate?

 ZSH> It could a bug of Emacs or Mule-UCS.

Auto-detection of utf-8 doesn't work too well in Emacs 20; perhaps
that's the problem (or, maybe, spurious byte combination, addressed
for Emacs 21).  I hope it doesn't require debugging Mule-UCS :-/.

 ZSH> Hope this won't happen in the future Unicode Emacs.

I don't think it should need a change in the internal representation
to make utf work.

Re: \201 irritation! :-)

[ShengHuo ZHU]

Dave Love <d.love@dl.ac.uk> writes:

> >>>>> "ZSH" == ShengHuo ZHU <zsh@cs.rochester.edu> writes:
> 
>  ZSH> Wq is supposed to do so and, AFAIK, no bug is found for Emacs
>  ZSH> build-in coding-system.  I know there is some problems with
>  ZSH> copying utf-8 encoded texts between buffers in Emacs.
> 
> That doesn't seem to be the case here, but do you have details that we
> can investigate?

False alarm! Anyway, by investigating the problem, I found out a bug
bothering me for a few months.  It was because of copying strings
containing control characters such as \210 between unibyte and
multibyte buffers. Such kind of strings only exist in utf-8 not other
coding-system.

ShengHuo

Re: \201 irritation! :-)

[ShengHuo ZHU]

"Yair Friedman (Jerusalem)" <YAIRFR@Amdocs.com> writes:

> With the help of Kai (Thanks!) I found out that the crippled software
> omitted or forgot to inset CTE header.  After manually inserting it,
> everything is displayed well *before* and after Wq.
> 
> As for the malformed message, WIBNI gnus warn me about it and possibly
> let me edit the message?
> 
> Nevertheless, gnus-article-de-quoted-unreadable should know how to handle
> this malformed massage -- this is the whole purpose of it.
> 
> I'm not sure that the possible misbehavior of Wq triggers an Emacs bug
> over here, or maybe it's just a Gnus bug only.

I got some clues about the problem.

First of all, please check gnus-newsgroup-charset in the summary
buffer.  If it isn't iso-8859-1, don't blame
gnus-article-de-quoted-unreadable.

Second, I just commit a patch, which enables Gnus to properly fetch
charset from the broken headers, where the content-type indicates the
charset (e.g. charset=utf-8 in François' case).

ShengHuo

Re: \201 irritation! :-)

[Florian Weimer]

Dave Love <d.love@dl.ac.uk> writes:

>  FP> Yes, I noticed this.  After having installed Otfried UTF-8
>  FP> support, Gnus apparently started to use it automatically, and
>  FP> something appears to be wrong somewhere in the process.
> 
> It would presumably be helpful to know where. 

See the Gnus ChangeLog. François' article would require splitting into
several text/plain parts with different charsets, so Gnus uses UTF-8
automatically if it's available to avoid this. In the past, UTF-8 used
to work pretty well. Unfortunately, UTF-8 is likely to contain some
of the quoting characters used by GNU Emacs for in-band signalling of
some MULE-related information (if I understand this problem correctly,
I'm no MULE wizard). That's why UTF-8 support is so fragile at the
moment, I think.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "ZSH" == ShengHuo ZHU <zsh@cs.rochester.edu> writes:

 ZSH> containing control characters such as \210 between unibyte and
 ZSH> multibyte buffers. Such kind of strings only exist in utf-8 not other
 ZSH> coding-system.

For what it's worth, that's changed in Mule 5.0, i.e. Emacs 21.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "FW" == Florian Weimer <fw@deneb.enyo.de> writes:

 FW> Dave Love <d.love@dl.ac.uk> writes:
 FP> Yes, I noticed this.  After having installed Otfried UTF-8
 FP> support, Gnus apparently started to use it automatically, and
 FP> something appears to be wrong somewhere in the process.
 >> 
 >> It would presumably be helpful to know where. 

 FW> See the Gnus ChangeLog. François' article would require splitting into
 FW> several text/plain parts with different charsets, so Gnus uses UTF-8
 FW> automatically if it's available to avoid this. 

Yes, `where' was supposed to mean where in Mule-UCS.

 FW> In the past, UTF-8 used to work pretty well. 

It would be nice to know how it broke if it's not what ShengHuo just
fixed.

 FW> Unfortunately, UTF-8 is likely to contain some of the quoting
 FW> characters used by GNU Emacs for in-band signalling of some
 FW> MULE-related information 

I guess you mean you can get spurious byte-combination, as I think
ShengHuo was talking about.  As far as I know, this shouldn't be a
problem simply decoding the external utf-8 coding system, but I don't
grok Mule-UCS and I don't know where the problem is occurring.

Anyway, the `eight-bit-control' charset in Mule 5.0 should fix that
sort of thing in the future, as well as allowing better auto-detection
of utf-8.

Re: \201 irritation! :-)

[ShengHuo ZHU]

Dave Love <d.love@dl.ac.uk> writes:

> >>>>> "ZSH" == ShengHuo ZHU <zsh@cs.rochester.edu> writes:
> 
>  ZSH> containing control characters such as \210 between unibyte and
>  ZSH> multibyte buffers. Such kind of strings only exist in utf-8 not other
>  ZSH> coding-system.
> 
> For what it's worth, that's changed in Mule 5.0, i.e. Emacs 21.

Maybe it is OK for characters between \200-\237, but not for those
between \240-\377, i.e. eight-bit-graphic. unibyte-char-to-multibyte
converts them to latin-iso8859-1.  Suppose to insert a unibyte string
with \337 to a multibyte buffer, I always get ß, unless using the
number (insert "0123" 223 "ABCD").  See the following examples

(insert "0123\337ABCD")

(let (buf)
  (with-temp-buffer
    (set-buffer-multibyte nil)
    (insert "0123\337ABCD")
    (setq buf (current-buffer))
    (with-temp-buffer
      (insert-buffer buf)
      (decode-coding-region (point-min) (point-max) 'iso-8859-1)
      (buffer-string))))

Maybe unibyte-char is converted to latin-iso8859-1 on purpose for
compatibility, but in the second case it would be ugly.  It
illustrates some cases in Gnus, getting data from network (in a
unibyte buffer) then copying them to multibyte buffer for decoding.

Maybe using unibyte-buffer in Gnus is not a `right thing' to do at
all.  I guess it was introduced to work with some buggy versions of
Emacs 20.  I suggest to remove it from oGnus.

ShengHuo

Re: \201 irritation! :-)

[Yair Friedman (Jerusalem)]

Today after refreshing gnus from the snapshot (gnus-CURRENT-20000920)
the bug still appears.
This is the pre-compiled GNU Emacs 20.6.1.

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> I got some clues about the problem.
> 
> First of all, please check gnus-newsgroup-charset in the summary
> buffer.  If it isn't iso-8859-1, don't blame
> gnus-article-de-quoted-unreadable.

,----
| gnus-newsgroup-charset's value is 
| iso-8859-1
| Local in buffer *Summary nnml:ding*; global value is 
| iso-8859-1
`----

> 
> Second, I just commit a patch, which enables Gnus to properly fetch
> charset from the broken headers, where the content-type indicates the
> charset (e.g. charset=utf-8 in François' case).

Same results -- when CTE is missing Wq shows \201 before the "ß".
What now?

Yair.

Re: \201 irritation! :-)

[ShengHuo ZHU]

"Yair Friedman (Jerusalem)" <YAIRFR@Amdocs.com> writes:

> > Second, I just commit a patch, which enables Gnus to properly fetch
> > charset from the broken headers, where the content-type indicates the
> > charset (e.g. charset=utf-8 in François' case).
> 
> Same results -- when CTE is missing Wq shows \201 before the "ß".
> What now?

It doesn't happen to me (I use Emacs 21.0.90 and 20.7).  It could be a
bug of Emacs.  What is the result of evaluating (insert 223) in a
multibyte buffer?  It should be \337.

ShengHuo

Re: \201 irritation! :-)

[ShengHuo ZHU]

"Yair Friedman (Jerusalem)" <YAIRFR@Amdocs.com> writes:

> ShengHuo ZHU <zsh@cs.rochester.edu> writes:
> 
> > It doesn't happen to me (I use Emacs 21.0.90 and 20.7).  It could be a
> > bug of Emacs.  What is the result of evaluating (insert 223) in a
> > multibyte buffer?  It should be \337.
> > 
> > ShengHuo
> 
> Attached is the offending article without the CTE.
> 
> Evaluating (insert 223) in the *scratch* buffer indeed insert \337. I
> don't think I did something special to make in unibyte, but I do have
> (require 'iso-transl) and (standard-display-european 1) in my .emacs
> buffer (I know the setup should be changed in recent versions, but I'm
> lazy).

(standard-display-european 1) is it! 

Here is a comment from standard-display-european:

    ;; If the user does this explicitly from Lisp (as in .emacs),
    ;; turn off multibyte chars for more compatibility.

ShengHuo

Re: \201 irritation! :-)

[Yair Friedman (Jerusalem)]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> It doesn't happen to me (I use Emacs 21.0.90 and 20.7).  It could be a
> bug of Emacs.  What is the result of evaluating (insert 223) in a
> multibyte buffer?  It should be \337.
> 
> ShengHuo

Attached is the offending article without the CTE.


[-- Attachment #2: 29758.gz --]
[-- Type: application/zip, Size: 2259 bytes --]

[-- Attachment #3: Type: text/plain, Size: 295 bytes --]


Evaluating (insert 223) in the *scratch* buffer indeed insert \337. I
don't think I did something special to make in unibyte, but I do have
(require 'iso-transl) and (standard-display-european 1) in my .emacs
buffer (I know the setup should be changed in recent versions, but I'm
lazy).

Yair.

Re: \201 irritation! :-)

[Florian Weimer]

Dave Love <d.love@dl.ac.uk> writes:

>  FW> Unfortunately, UTF-8 is likely to contain some of the quoting
>  FW> characters used by GNU Emacs for in-band signalling of some
>  FW> MULE-related information 
> 
> I guess you mean you can get spurious byte-combination, as I think
> ShengHuo was talking about.  

Byte-combination kills the quoted-printable encoder (or used to do it
in the past, at least).

> As far as I know, this shouldn't be a
> problem simply decoding the external utf-8 coding system, but I don't
> grok Mule-UCS and I don't know where the problem is occurring.

Copying byte-combinations between unibyte and multibyte buffers
results in some weird effects.  Sometimes, Emacs is not 8-bit
clean. :-(

> Anyway, the `eight-bit-control' charset in Mule 5.0 should fix that
> sort of thing in the future, as well as allowing better auto-detection
> of utf-8.

Does "encode-coding-region" produce characters in this encoding?

BTW: Please do not abuse GRAVE ACCENT (U+0060) as quotation mark.
Thanks.

Re: \201 irritation! :-)

[Kai Großjohann]

On 20 Sep 2000, Florian Weimer wrote:

> BTW: Please do not abuse GRAVE ACCENT (U+0060) as quotation mark.
> Thanks.

It is not clear to me how to get around this.  Using `foo' to quote
stuff is strongly engrained in Emacs.  For example in doc strings.
Switching to a different pair of characters (gotta switch both since '
is used as a right quote but isn't directional) would mean that all
*.el files have to be changed (for practical purposes, I don't think
there is a Lisp file without these characters in doc strings).

Also, maybe the tick and backtick characters should be changed from '
and `, respectively.  But of course that means changing the Lisp
syntax!

Thoughts?

kai
-- 
I like BOTH kinds of music.

Re: \201 irritation! :-)

[Yair Friedman (Jerusalem)]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> "Yair Friedman (Jerusalem)" <YAIRFR@Amdocs.com> writes:
> 
> > Evaluating (insert 223) in the *scratch* buffer indeed insert \337. I
> > don't think I did something special to make in unibyte, but I do have
> > (require 'iso-transl) and (standard-display-european 1) in my .emacs
> > buffer (I know the setup should be changed in recent versions, but I'm
> > lazy).
> 
> (standard-display-european 1) is it! 
> 
> Here is a comment from standard-display-european:
> 
>     ;; If the user does this explicitly from Lisp (as in .emacs),
>     ;; turn off multibyte chars for more compatibility.
> 
> ShengHuo

I replaced the call to standard-display-european with:
(set-language-environment "Latin-1")
(setq unibyte-display-via-language-environment t)

I'm not sure if this is really what I need, the documentation is quite
unclear.  After this, Wq works as expected.
--
Yair Friedman.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "YF" == Yair Friedman (Jerusalem) <YAIRFR@Amdocs.com> writes:

 YF> I'm not sure if this is really what I need, the documentation is
 YF> quite unclear.

Please make a bug report about exactly what you find unclear.  Thanks.

[If you set your locale to an 8859-1 one, e.g. with LANG, you don't
even need to set a language environment explicitly.]

Can someone debug why the quoted printable code loses in unibyte mode?
It worked OK for me in unibyte in the past.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "FW" == Florian Weimer <fw@deneb.enyo.de> writes:

 FW> Byte-combination kills the quoted-printable encoder (or used to
 FW> do it in the past, at least).

I don't see why it should, at least for a sane charset.  Do you know
exactly how it fails?  I assume Mule 5 will solve such problems, but I
want to make sure.

 FW> Copying byte-combinations between unibyte and multibyte buffers
 FW> results in some weird effects.  Sometimes, Emacs is not 8-bit
 FW> clean. :-(

I don't know what that means.  Just spurious combination of leading
bytes stuffed raw into a multibyte buffer or something else?
(Obviously going multibyte->unibyte is generally lossy.)  `8-bit
clean' doesn't seem meaningful in a multibyte setting, but Emacs is so
if you operate purely in unibyte (modulo `set-input-mode', perhaps).

 >> Anyway, the `eight-bit-control' charset in Mule 5.0 should fix that
 >> sort of thing in the future, as well as allowing better auto-detection
 >> of utf-8.

 FW> Does "encode-coding-region" produce characters in this encoding?

Do you mean _de_code-coding-region?  `encode-coding-region' would
produce raw bytes.

(find-charset-string (decode-coding-string "\344b\205" 'latin-9))
  => (ascii latin-iso8859-15 eight-bit-control)

 FW> BTW: Please do not abuse GRAVE ACCENT (U+0060) as quotation mark.
 FW> Thanks.

It's too late for that, but this will be ASCII.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "ZSH" == ShengHuo ZHU <zsh@cs.rochester.edu> writes:

 ZSH> Maybe it is OK for characters between \200-\237, but not for those
 ZSH> between \240-\377, i.e. eight-bit-graphic. unibyte-char-to-multibyte
 ZSH> converts them to latin-iso8859-1.  

Of course.

 ZSH> Suppose to insert a unibyte string with \337 to a multibyte
 ZSH> buffer, I always get ß, unless using the number (insert "0123"
 ZSH> 223 "ABCD").

Yes, text is meant to be converted that way.

 ZSH> Maybe unibyte-char is converted to latin-iso8859-1 

The conversion depend on `nonascii-insert-offset', though I think that
will default to Latin-1/

 ZSH> on purpose for compatibility, but in the second case it would be
 ZSH> ugly.  

I'm not sure I understand that.

 ZSH> It illustrates some cases in Gnus, getting data from network (in
 ZSH> a unibyte buffer) then copying them to multibyte buffer for
 ZSH> decoding.

That seems to me like the wrong thing to do.

Re: \201 irritation! :-)

[Karl Eichwalder]

"Yair Friedman (Jerusalem)" <YAIRFR@Amdocs.com> writes:

> I replaced the call to standard-display-european with:
> (set-language-environment "Latin-1")

Trying a 21.0.90 (July/August 2000) this setting didn't work for me
anymore.

Dave Love <d.love@dl.ac.uk> writes:

> [If you set your locale to an 8859-1 one, e.g. with LANG, you don't
> even need to set a language environment explicitly.]

Yes, I now happily use

    LANG=de_DE.ISO-8859-1

-- 
work : ke@suse.de                          |          ------    ,__o
     : http://www.suse.de/~ke/             |         ------   _-\_<,
home : keichwa@gmx.net                     |        ------   (*)/'(*)

Re: \201 irritation! :-)

[Florian Weimer]

Dave Love <d.love@dl.ac.uk> writes:

> >>>>> "FW" == Florian Weimer <fw@deneb.enyo.de> writes:
> 
>  FW> Byte-combination kills the quoted-printable encoder (or used to
>  FW> do it in the past, at least).
> 
> I don't see why it should, at least for a sane charset.  Do you know
> exactly how it fails?

If there is a byte-combination, "(char-after)" returns values outside
the usual 0 .. 255 range, and "quoted-printable-encode-region" doesn't
handle this.  The best thing probably is to switch the buffer to
uni-byte mode during quoted-printable encoding.  Perhaps adding
"mm-with-unibyte-current-buffer" could do the trick?

>  FW> Copying byte-combinations between unibyte and multibyte buffers
>  FW> results in some weird effects.  Sometimes, Emacs is not 8-bit
>  FW> clean. :-(
> 
> I don't know what that means.  Just spurious combination of leading
> bytes stuffed raw into a multibyte buffer or something else?

Take the "Chinese" line from the HELLO file, copy it to a multi-byte
buffer, do "encode-coding-region" on it and specify "utf-8" as
encoding.  Copy the result to a unibyte buffer, and paste it back into
the first (multi-byte) buffer.  Now the \201s are there, you can see
them if you switch the first buffer to uni-byte mode.

The trouble with UTF-8 is that it tends to generate more
byte-combinations than other encodings, as it seems.

>  >> Anyway, the `eight-bit-control' charset in Mule 5.0 should fix that
>  >> sort of thing in the future, as well as allowing better auto-detection
>  >> of utf-8.
> 
>  FW> Does "encode-coding-region" produce characters in this encoding?
> 
> Do you mean _de_code-coding-region?  `encode-coding-region' would
> produce raw bytes.

And these raw bytes are not properly dealt with in multi-byte buffers.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "KE" == Karl Eichwalder <keichwa@gmx.net> writes:

 KE> Trying a 21.0.90 (July/August 2000) this setting didn't work for
 KE> me anymore.

I don't remember any bug report about that and I'm surprised it is or
was broken.  I think you have a responsibility to report problems in
prerelease versions properly if you're going to use them at all.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "FW" == Florian Weimer <fw@deneb.enyo.de> writes:

 FW> If there is a byte-combination, "(char-after)" returns values
 FW> outside the usual 0 .. 255 range, and
 FW> "quoted-printable-encode-region" doesn't handle this.

Then it's either intrinsically broken or not being used appropriately
since it doesn't make sense for multibyte characters.  I suppose I'll
have to check it if no-one else can.

 FW> The best thing probably is to switch the buffer to
 FW> uni-byte mode during quoted-printable encoding.

Whatever you do, you need a buffer with encoded contents.  Making it
unibyte isn't right otherwise or you're dealing with the emacs-mule
charset.

 >> I don't know what that means.  Just spurious combination of leading
 >> bytes stuffed raw into a multibyte buffer or something else?

 FW> Take the "Chinese" line from the HELLO file, copy it to a
 FW> multi-byte buffer, do "encode-coding-region" on it and specify
 FW> "utf-8" as encoding.  Copy the result to a unibyte buffer, and
 FW> paste it back into the first (multi-byte) buffer.  

This seems to be just confusing several issues.  I don't see how it's
related to spurious byte combination, unless it refers to some problem
in Mule-UCS itself.  Can you give a code snippet demonstrating the
problems you're concerned about without using utf-8 as an example?  It
shouldn't be relevant.  Be careful to include the relevant context
(C-h C, at least) since you're concerned with code conversion.

 FW> Now the \201s are there, you can see them if you switch the first
 FW> buffer to uni-byte mode.

Of course you see the internal encoding if you switch it to unibyte
mode anyway.  You may also have pasted in raw bytes.

 FW> The trouble with UTF-8 is that it tends to generate more
 FW> byte-combinations than other encodings, as it seems.

 >> Do you mean _de_code-coding-region?  `encode-coding-region' would
 >> produce raw bytes.

 FW> And these raw bytes are not properly dealt with in multi-byte buffers.

So don't do that.  That's the point.

I'm not getting any indication that there is an issue here that needs
to be addressed in Emacs in future.

Re: \201 irritation! :-)

[Karl Eichwalder]

Dave Love <d.love@dl.ac.uk> writes:

> I don't remember any bug report about that and I'm surprised it is or
> was broken.  I think you have a responsibility to report problems in
> prerelease versions properly if you're going to use them at all.

I was testing a completely different thing and marking up Texinfo files
(on Gerd's kind request).  Nobody has the responsibility to resolve all
problems at once.

If you're interested in this issue just give it a try -- maybe, it was a
temporary glitch only.

-- 
work : ke@suse.de                          |          ------    ,__o
     : http://www.suse.de/~ke/             |         ------   _-\_<,
home : keichwa@gmx.net                     |        ------   (*)/'(*)

Re: \201 irritation! :-)

[Dave Love]

>>>>> "KE" == Karl Eichwalder <keichwa@gmx.net> writes:

 KE> Nobody has the responsibility to resolve all problems at once.

I said `report', not `resolve'.

 KE> If you're interested in this issue just give it a try

Without a bug report I don't know exactly what I have to try.  I use
Latin-1 most of the time and set-language-environment is invoked by
the locale code.  Please help us by making a proper bug report about
something apparently so fundamental.  Thanks.

Re: \201 irritation! :-)

[Florian Weimer]

Dave Love <d.love@dl.ac.uk> writes:

> >>>>> "FW" == Florian Weimer <fw@deneb.enyo.de> writes:
> 
>  FW> If there is a byte-combination, "(char-after)" returns values
>  FW> outside the usual 0 .. 255 range, and
>  FW> "quoted-printable-encode-region" doesn't handle this.
> 
> Then it's either intrinsically broken or not being used appropriately
> since it doesn't make sense for multibyte characters.  I suppose I'll
> have to check it if no-one else can.

The following code illustrates the problem:

(let ((multi (get-buffer-create "*Multibyte")) chars)
  (with-current-buffer multi
    (erase-buffer)
    (set-buffer-multibyte t)
    (insert 41813)
    (encode-coding-region (point-min) (point-max) 'utf-8)
    (goto-char (point-min))
    (while (not (eobp))
      (setq chars (cons (char-after) chars))
      (forward-char))
    (setq chars (nreverse chars))
    (insert "\n" 
	    (format "%S" chars)
	    "\n"))
  (switch-to-buffer multi))

I'm going to add the "mm-with-unibyte-current-buffer" stuff as soon as
I've left hospital permanently, so that I can deal with the
consequences in a more timely manner. ;-)

>  FW> The best thing probably is to switch the buffer to
>  FW> uni-byte mode during quoted-printable encoding.
> 
> Whatever you do, you need a buffer with encoded contents.  Making it
> unibyte isn't right otherwise or you're dealing with the emacs-mule
> charset.

When qp encoding takes place, Gnus has already invoked
"encode-coding-region" at some point, so it should be safe to look at
the raw bytes.

>  FW> Now the \201s are there, you can see them if you switch the first
>  FW> buffer to uni-byte mode.
> 
> Of course you see the internal encoding if you switch it to unibyte
> mode anyway.  You may also have pasted in raw bytes.

The problem is that raw bytes turn in to characters of the default
encoding (Latin-1 with my setup) when copying them around.  It would
be very helpful if Emacs could optionally generate an error message in
this case, so we could track those \201 stuff down more easily.

>  FW> The trouble with UTF-8 is that it tends to generate more
>  FW> byte-combinations than other encodings, as it seems.
> 
>  >> Do you mean _de_code-coding-region?  `encode-coding-region' would
>  >> produce raw bytes.
> 
>  FW> And these raw bytes are not properly dealt with in multi-byte buffers.
> 
> So don't do that.  That's the point.

The problem is the Gnus approach of gradually converting an article
with multiple parts, I think.  It's extremely fragile.

QP encoding is now unibyte (was: Re: \201 irritation! :-)

[Florian Weimer]

Florian Weimer <fw@deneb.enyo.de> writes:

> I'm going to add the "mm-with-unibyte-current-buffer" stuff as soon as
> I've left hospital permanently, so that I can deal with the
> consequences in a more timely manner. ;-)

2000-09-29  Florian Weimer  <fw@deneb.enyo.de>

	* qp.el (quoted-printable-encode-region): Wrap with
	`mm-with-unibyte-current-buffer'.

The change is now in place.  Please report any breakage whatsoever.

Re: \201 irritation! :-)

[Dave Love]

>>>>> "FW" == Florian Weimer <fw@deneb.enyo.de> writes:

 >> Whatever you do, you need a buffer with encoded contents.  Making it
 >> unibyte isn't right otherwise or you're dealing with the emacs-mule
 >> charset.

 FW> When qp encoding takes place, Gnus has already invoked
 FW> "encode-coding-region" at some point, so it should be safe to
 FW> look at the raw bytes.

So you want unibyte.

 >> Of course you see the internal encoding if you switch it to unibyte
 >> mode anyway.  You may also have pasted in raw bytes.

 FW> The problem is that raw bytes turn in to characters of the default
 FW> encoding (Latin-1 with my setup) when copying them around.

Characters in unibyte text will only be converted if they're in the
valid range.  As far as I know, you could prevent any conversion if
you really needed to.  I don't know what the problem is that ShengHuo
mentioned.

 FW> It would be very helpful if Emacs could optionally generate an
 FW> error message in this case, 

I don't think so.

 FW> so we could track those \201 stuff down more easily.

It's unclear to me that inserting unibyte text per se is the root of
the spurious coding conversions.

Re: \201 irritation! :-)

[Florian Weimer]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> > BTW: Please do not abuse GRAVE ACCENT (U+0060) as quotation mark.
> > Thanks.
> 
> It is not clear to me how to get around this.  Using `foo' to quote
> stuff is strongly engrained in Emacs.  For example in doc strings.
> Switching to a different pair of characters (gotta switch both since '
> is used as a right quote but isn't directional) would mean that all
> *.el files have to be changed (for practical purposes, I don't think
> there is a Lisp file without these characters in doc strings).

The code which renders the help messages would have to be changed,
nothing else.

> Also, maybe the tick and backtick characters should be changed from '
> and `, respectively.  But of course that means changing the Lisp
> syntax!

I think using these characters in source code (TeX, Texinfo, Lisp,
Shell) is acceptable, but presenting them to the user should be
reconsidered (at least in a few years, when Unicode fonts are in more
widespread use, so you can use the quotation mark characters from
the,).

Re: \201 irritation! :-)

[Karl Eichwalder]

Florian Weimer <fw@deneb.enyo.de> writes:

> I think using these characters in source code (TeX, Texinfo, Lisp,
> Shell) is acceptable, but presenting them to the user should be
> reconsidered

Yes, that's a good point.

> (at least in a few years, when Unicode fonts are in more widespread
> use, so you can use the quotation mark characters from the,).

In the meantime, it sould be possible for Latin-1 users to use « ... »
(french) or »...« (german) -- I don't know wheter other encodings/fonts
have something similar to offer.

-- 
work : ke@suse.de                          |                   ,__o
     : http://www.suse.de/~ke/             |                 _-\_<,
home : keichwa@gmx.net                     |                (*)/'(*)

Re: \201 irritation! :-)

[François Pinard]

[Florian Weimer]

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> > > BTW: Please do not abuse GRAVE ACCENT (U+0060) as quotation mark.
> > > Thanks.

Where did you get that this is an abuse?  ASCII meant it as symmetric to
acute accent, and so does Unicode, by leaving the grave accent and especially
the apostrophe as ambiguous as either an apostrophe or an acute accent.
There are separate, more specific Unicode points for non-ambiguous usage.

My feeling is that people are tempted to consider it as an abuse, because
they use fonts in which the apostrophe stands right, instead of being
symmetrical to U+0060.  They should rather consider the fonts as wrong, since
the symmetry is not preserved.  The fonts should be corrected, not the usage.

> The code which renders the help messages would have to be changed,
> nothing else.

This is being done in some places.  Some make a great fuss around all this.

> I think using these characters in source code (TeX, Texinfo, Lisp,
> Shell) is acceptable, but presenting them to the user should be
> reconsidered (at least in a few years, when Unicode fonts are in more
> widespread use, so you can use the quotation mark characters from
> the,).

So called Unicode fonts should be ASCII fonts for that little part where
Unicode and ASCII coincide.  Of course, if you make Unicode to differ from
ASCII in that region, you create yourself, and everybody, a lot of problems.

Many people frantically want to go Unicode.  But strangely, some are
reluctant to Unicode outside the first 127 characters, and want things
to be changed consequently.  It would better if Unicode people were using
Unicode, while leaving ASCII alone.  Moreover, it was meant that way.

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

tick and backtick in unicode

[Alan Shutko]

François Pinard <pinard@iro.umontreal.ca> writes:

> My feeling is that people are tempted to consider it as an abuse, because
> they use fonts in which the apostrophe stands right, instead of being
> symmetrical to U+0060.  They should rather consider the fonts as wrong, since
> the symmetry is not preserved.  The fonts should be corrected, not the usage.

Perhaps you should speak to the XFree86 people, then.  They recently
"fixed" the standard X fonts to make ` and ' assymetrical.

-- 
Alan Shutko <ats@acm.org> - In a variety of flavors!
Don't worry if you're a kleptomaniac; you can always take something for it.

Re: tick and backtick in unicode

[François Pinard]

[Alan Shutko]

> Perhaps you should speak to the XFree86 people, then.  They recently
> "fixed" the standard X fonts to make ` and ' assymetrical.

Maybe! :-)

I take your sentence as a figure of style, and hope you do not really imply
that I'm responsible, alone, to save the Earth!  I tried a few times already,
and it quickly gets very exhausting. :-)

Better each of us do our share.  My little contribution, in this particular
case, is to spread the word that some common sense is needed about fonts.
ASCII has not been recently re-invented, just because Unicode gets in the
picture, or because people feel that acute accent over apostrophe ambiguity,
which is indeed part of ASCII, lasted long enough.  Unicode is, by design,
coincident with ASCII for the first code points, and people pushing for
Unicode should remember this, in all simplicity.  Let them select some
proper Unicode apostrophe, outside ASCII, if they want it clear cut,
non ambiguous, and vertical.

I know that many people, and even very competent ones, are throwing of
confusion in this field, trying to "repair" ASCII.  This is unfortunate.

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

Re: tick and backtick in unicode

[Karl Eichwalder]

François Pinard <pinard@iro.umontreal.ca> writes:

> Let them select some proper Unicode apostrophe, outside ASCII, if they
> want it clear cut, non ambiguous, and vertical.

On my US keyboard both keys showing "accents" are not symmetrical:
`...'.

Using ISO-8859-1 these are the symmetrical accents: `...´ (where ´ is
8bit).  Using a recent xterm with recents fonts this is visible.

> I know that many people, and even very competent ones, are throwing of
> confusion in this field, trying to "repair" ASCII.  This is
> unfortunate.

At first, this was what I thought, too.  Then I read Markus Kuhn's
article on this subject and I fear he's right:

    http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html

;)

-- 
work : ke@suse.de                          |                   ,__o
     : http://www.suse.de/~ke/             |                 _-\_<,
home : keichwa@gmx.net                     |                (*)/'(*)

Re: tick and backtick in unicode

[Stainless Steel Rat]

* Karl Eichwalder <keichwa@gmx.net>  on Mon, 27 Nov 2000
| Using ISO-8859-1 these are the symmetrical accents: `...´ (where ´ is
| 8bit).  Using a recent xterm with recents fonts this is visible.

Yes, this is true.  And the apostrophe character is the "neutral" vertical
mark.

That said, the apostrophe characters on the two keyboards I have handy, one
of which is one of those "extended" Windows keyboards, are asymmetrical.
-- 
Rat <ratinox@peorth.gweep.net>    \ Caution: Happy Fun Ball may suddenly
Minion of Nathan - Nathan says Hi! \ accelerate to dangerous speeds.
PGP Key: at a key server near you!  \ 

Re: tick and backtick in unicode

[François Pinard]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

[Karl Eichwalder]

> On my US keyboard both keys showing "accents" are not symmetrical:
> `...'.

Neither on mine, actually.  So, the keycaps are not showing ASCII.  Anyway,
I would not take the keyboard, your or mine, as a reference.

> Using ISO-8859-1 these are the symmetrical accents: `...´ (where ´
> is 8bit).  Using a recent xterm with recents fonts this is visible.

I do not remember the text of ISO 8859-1, nor if it addresses that problem.
I'm using ETL fonts, which do something sensible.  I'm including a picture
of a small part of this message composition buffer...


[-- Attachment #2: quotes.png --]
[-- Type: image/png, Size: 1400 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1347 bytes --]


> > I know that many people, and even very competent ones, are throwing of
> > confusion in this field, trying to "repair" ASCII.  This is unfortunate.

> At first, this was what I thought, too.  Then I read Markus Kuhn's
> article on this subject and I fear he's right:

>     http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html

> ;)

Yes.  I was silently referring to Markus, above.  Markus is usually very
right, and quite dynamic as a guy, but I'm not following him on this one,
even if I was much tempted to.  I had to calm down and think a bit more,
before deciding that his proposal is a confusing compromise, however serious
it may look like.  In fact, Markus invited me to look at the Unicode
reference, and this is what really convinced me that Unicode is really
made to compatible with ASCII, including the ambiguous usage of the quote.

I might be wrong, of course.  That would not have been the first time,
nor the last.  Yet, I still honestly think that the problem has been
invented by font makers and later users, but is away from both Unicode and
ASCII standards.  Many active people are busy at producing complex software
solutions around a problem which should rather be corrected by font makers,
which did propagate the problem by imposing debatable, non-standard views.

-- 
François Pinard   http://www.iro.umontreal.ca/~pinard

Re: tick and backtick in unicode

[Karl Eichwalder]

François Pinard <pinard@iro.umontreal.ca> writes:

> I do not remember the text of ISO 8859-1, nor if it addresses that
> problem.

Markus explains that neither ASCII (ISO 646) nor ISO 8859-1 has left or
right quote characters (neither double nor single); it has a single and
a double quote you've to use to start and to end the quoted string.

ASCII has the grave accent ("back tick") and ISO 8859-1 has the cute
accent in addition.

Displaying the grave accent as a proper left single quote looks silly if
you've to read shell scripts -- okay, you better use $(COMMAND)...

> I'm using ETL fonts, which do something sensible.  I'm including a
> picture of a small part of this message composition buffer...

Me, too.  I was very pleased as I encountered these fonts the first
time (and still I am)!  It's nice but wrong, sadly ;-(

> In fact, Markus invited me to look at the Unicode
> reference, and this is what really convinced me that Unicode is really
> made to compatible with ASCII, including the ambiguous usage of the quote.

Yes, ASCII is ambiguous.  But the ETL fonts removing this ambiguity and
displaying the grave accent as a left single quote -- reading Emacs Lisp
code, it looks nice -- but it will confuse people in the long run!

> Yet, I still honestly think that the problem has been invented by font
> makers and later users, but is away from both Unicode and ASCII
> standards.

Yes, by the ETL font makers.  With ETL fonts I'm not able to display the
grave accent in an 8859-1 environment.

Id' say it's still feasible to use these `quotation technique' for Emacs
Lisp comments, but not for shell scripts nor for program message
considered to be read by the user.

-- 
work : ke@suse.de                          |                   ,__o
     : http://www.suse.de/~ke/             |                 _-\_<,
home : keichwa@gmx.net                     |                (*)/'(*)

Re: tick and backtick in unicode

[Kai Großjohann]

On Wed, 29 Nov 2000, Karl Eichwalder wrote:

> Markus explains that neither ASCII (ISO 646) nor ISO 8859-1 has left
> or right quote characters (neither double nor single); it has a
> single and a double quote you've to use to start and to end the
> quoted string.
> 
> ASCII has the grave accent ("back tick") and ISO 8859-1 has the cute

Are you saying that under your interpretation there is no way in
Unicode to display left and right single quotes?  IMVHO, such an
interpretation is silly, since the English speaking people have been
using left and right single (or double) quotes for ages to mark direct
speach and the like.  Removing these characters from Ascii or Latin-1
(or Unicode) means that it is not possible to express normal English
text using that encoding.  And surely we do not want to dumb down the
encoding in such a way that it becomes impossible to do that?

FWIW, TeX also uses `single quotes' like this.  By saying this, I mean
that when you provide TeX with an input that has `single quotes', then
the output will contain a single left and a single right quote.

I think it is not useful to make fonts such that TeX input looks
asymmetrical, nor such that Emacs Lisp code looks asymmetrical.

And I don't believe that shell code looks silly with left and right
quotes at all, no more silly than with grave and acute accents.

kai
-- 
The arms should be held in a natural and unaffected way and never
be conspicuous. -- Revised Technique of Latin American Dancing

Re: tick and backtick in unicode

[Karl Eichwalder]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Are you saying that under your interpretation there is no way in
> Unicode to display left and right single quotes?

No.  Unicode has left and right single quotes.

> IMVHO, such an interpretation is silly, since the English speaking
> people have been using left and right single (or double) quotes for
> ages to mark direct speach and the like.

I guess you're talking about computer usage ;)  In previous times, there
was a typewriter offering apostrophe (w/o direction) and accents only.
Then people started to misuse the grave accent as left single quote.

> Removing these characters from Ascii or Latin-1
> (or Unicode) means that it is not possible to express normal English
> text using that encoding.

That's a silly conclusion ;) There's no left double quote neither...

> FWIW, TeX also uses `single quotes' like this.  By saying this, I mean
> that when you provide TeX with an input that has `single quotes', then
> the output will contain a single left and a single right quote.

Please, just read Markus Kuhn's article.  He quotes Donald E. Knuth:

    Donald Knuth's <A HREF=
    "http://www-cs-staff.Stanford.EDU/~knuth/abcde.html#texbk">TeXbook</A>
    (chapter 2, page 3, end of second paragraph) has actually warned TeX
    users already since 1986 that the apostrophe and grave accent shapes
    can show up as required by ISO and Unicode and not as used in the
    rest of the TeXbook.

Of course, Knuth doesn't mention ISO and Unicode but mentions the
problem in general.

> I think it is not useful to make fonts such that TeX input looks
> asymmetrical,

We'll have to learn to live with this hassle.

> nor such that Emacs Lisp code looks asymmetrical.

Emacs code isn't affected; only quoted strings inside comments.
According to Markus only m4 is affected -- he recommends to make use of
'changequote'.

> And I don't believe that shell code looks silly with left and right
> quotes at all,

It does.  And: using ETL fonts it's hard to distinguish both.  Proper
back ticks (grave accent) are much better to recognize.  Recent X fonts
are already changed.

Please, don't forget to read (it isn't that long):

    http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html

-- 
work : ke@suse.de                          |                   ,__o
     : http://www.suse.de/~ke/             |                 _-\_<,
home : keichwa@gmx.net                     |                (*)/'(*)

Re: tick and backtick in unicode

[Graham Murray]

Karl Eichwalder <keichwa@gmx.net> writes:

> I guess you're talking about computer usage ;)  In previous times, there
> was a typewriter offering apostrophe (w/o direction) and accents only.
> Then people started to misuse the grave accent as left single quote.

No. This usage pre-dates the typewriter, the convention is used both
in manuscript and printing presses.  

Re: tick and backtick in unicode

[Toby Speight]

0> In article <20001129054556.6773DDB0FB@tux.gnu.franken.de>,
0> Karl Eichwalder <URL:mailto:keichwa@gmx.net> ("Karl") wrote:

Karl> Markus explains that neither ASCII (ISO 646) nor ISO 8859-1 has
Karl> left or right quote characters (neither double nor single); it
Karl> has a single and a double quote you've to use to start and to
Karl> end the quoted string.
Karl>
Karl> ASCII has the grave accent ("back tick") and ISO 8859-1 has the
Karl> [acute] accent in addition.

0> In article <vaflmu2rzlt.fsf@lucy.cs.uni-dortmund.de>,
0> Kai Großjohann <URL:mailto:Kai.Grossjohann@CS.Uni-Dortmund.DE> ("Kai") wrote:

Kai> Are you saying that under your interpretation there is no way in
Kai> Unicode to display left and right single quotes?

There's no way in Latin-1; there exist paired quote characters in
Unicode (outside the Latin-1 range).


Kai> FWIW, TeX also uses `single quotes' like this.  By saying this, I
Kai> mean that when you provide TeX with an input that has `single
Kai> quotes', then the output will contain a single left and a single
Kai> right quote.
Kai>
Kai> I think it is not useful to make fonts such that TeX input looks
Kai> asymmetrical, nor such that Emacs Lisp code looks asymmetrical.
Kai>
Kai> And I don't believe that shell code looks silly with left and
Kai> right quotes at all, no more silly than with grave and acute
Kai> accents.

The way around this is probably to use a display-table to map (unpaired)
` and ' to the paired quote characters in a mode-specific display-table
used by (e.g.) Help mode and TeX mode.  Caveat: I've only played with
display-tables in unibyte emacsen; I don't know how Mule changes display
tables.

Disclaimer: I use the GNU fonts for my Emacs, but I'm aware that WISINWOG.

Re: tick and backtick in unicode

[Per Abrahamsen]

Karl Eichwalder <keichwa@gmx.net> writes:

> Emacs code isn't affected; only quoted strings inside comments.

And documentation strings, and error messages, and code for
highlighting and buttonization of the above.

Howevr, what *really* bothers me is having to unlearn 15 years
consistent use of `this' quotation style.  But

        <URL: http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html>

is very convincing, better adopt now, rather than fight another
"Backspace vs. Delete vs. Help" battle against the rest of the world.