[9fans] UTF-8 criticism?

[9fans] UTF-8 criticism?

[Jack Johnson]

I've always appreciated Plan 9's native UTF-8 support, but while
searching for reasons for the crappy multilingual support in Squeak I
came across the following message.

I'm just wondering how Plan 9 deals with this, or if it really matters?

-Jack

------

UTF-8 is what is known as an output transformation. It is used to put
whatever is in memory into some other form that is more readily
digestible by other devices that expect 7-bit ASCII and its associated
zero-null-byte convention. The UTF-8 format specifies ways of storing
up to 4-byte characters without any nulls aligned on bytes.

UTF-8 is also more compact for the European languages, but it is very
lengthy for traditional Chinese, as all characters require 2 bytes and
some characters inevitably require 3 bytes.

The problem with UTF-8 is that it is non-indexable. If you have a
string of characters, you can't make an assumption about where the nth
character is. To find out, you have to do a linear search. That makes
string indexing O(n) instead of O(1), which is unacceptable. If you
were to sort a UTF-8 string for some reason, the bubble sort would
actually have a lower order of magnitude than the quicksort.

So UTF-8 is not a very good memory format for characters. NT uses
UCS-2 (the 2-octet character set) in its native encoding, and UTF-8
for a lot of transfers to disk and the network. I'm not so sure that
Be uses UTF-8 in memory. I think you'd actually find they use UCS-2.

So I don't think it'd be good for someone to go through the hassle of
implementing a UTF-8 set of string methods. I like the idea of
bringing Unicode into Squeak. But there's a lot more involved than
just adding 2-byte arrays.

For example, you will want to store method string in UTF-8, because
they aren't allowed to carry characters larger than 7 bits. But you'd
have to make sure that they get transformed properly for other
purposes. You will have to provide alternate input/output routines for
files because you shouldn't store text files in UCS-2. There are many
considerations and I recommend that you read the standard, and all,
before going ahead and doing it.

Re: [9fans] UTF-8 criticism?

[Rob Pike]

utf-8 is an exchange format and that's mostly how it's used.
the few programs that do any analysis on utf-8 in memory
do so usually as a side effect of whatever else is going on.
for instance, they might call cleanname() or some other
such file name-processing routine.

his criticisms are on target but building utf-8 routines is
not intellectually challenging, nor is it a big job.

in short, it doesn't really matter.

-rob

Re: [9fans] UTF-8 criticism?

[boyd, rounin]

> If you were to sort a UTF-8 string for some reason, the bubble
> sort would actually have a lower order of magnitude than the
> quicksort.

that's not the real problem.  it's implementing the collation
sequences.  the internal representation as 16 bit unsigneds
is not a problem.

Re: [9fans] UTF-8 criticism?

[Rob Pike]

> that's not the real problem.  it's implementing the collation
> sequences.  the internal representation as 16 bit unsigneds
> is not a problem.

actually it is, now that surrogates are well-established.

-rob

Re: [9fans] UTF-8 criticism?

[boyd, rounin]

> actually it is, now that surrogates are well-established.

surrogates?

Re: [9fans] UTF-8 criticism?

[boyd, rounin]

> actually it is, now that surrogates are well-established.

ahh, i see.  ick.

Re: [9fans] UTF-8 criticism?

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 79 bytes --]

yes, there is no standard so useful that standards committees cannot ruin it.

[-- Attachment #2: Type: message/rfc822, Size: 2636 bytes --]

From: "boyd, rounin" <boyd@insultant.net>
To: "Fans of the OS Plan 9 from Bell Labs" <9fans@cse.psu.edu>
Subject: Re: [9fans] UTF-8 criticism?
Date: Sun, 18 Jul 2004 21:34:01 +0200
Message-ID: <002f01c46cfe$32fcfc70$92ec7d50@SOMA>

> actually it is, now that surrogates are well-established.

ahh, i see.  ick.

Re: [9fans] UTF-8 criticism?

[Geoff Collyer]

A sitting standards committee is like a sitting legislature: unless
they pass new standards (resp.  laws), they have nothing to do.

Re: [9fans] UTF-8 criticism?

[Douglas A. Gwyn]

boyd, rounin wrote:
> surrogates?

Yeah, that is what happens when people settle on
too small a code space.

Re: [9fans] UTF-8 criticism?

[Skip Tavakkolian]

>> surrogates?
>
> Yeah, that is what happens when people settle on
> too small a code space.

The Klingon Language Institute probably agrees.  Far out man!

http://www.kli.org/tlh/pIqaD.html

Re: [9fans] UTF-8 criticism?

[Joel Salomon]

>> that's not the real problem.  it's implementing the collation
>> sequences.  the internal representation as 16 bit unsigneds
>> is not a problem.
>
> actually it is, now that surrogates are well-established.
>
> -rob
>

Would moving to 32 bit signed (and only 0 -- 2^21 allowed, plus -1 for
EOF) as in the more recent revisions of Unicode take care of the
surrogates problem?

--Joel

p.s.

>>> surrogates?
>>
>> Yeah, that is what happens when people settle on
>> too small a code space.
>
> The Klingon Language Institute probably agrees.  Far out man!
>

And I want native tengwar support... Go Geeks!!

Re: [9fans] UTF-8 criticism?

[boyd, rounin]

> Would moving to 32 bit signed (and only 0 -- 2^21 allowed, plus -1 for
> EOF) as in the more recent revisions of Unicode take care of the
> surrogates problem?

this has nothing to do with EOF.

Re: [9fans] UTF-8 criticism?

[rog]

you do have to be a bit careful with utf-8, as many possible byte
sequences map down to the same rune (error), so if you
do your comparisons too early, you run the risk of inconsistency.

for instance, you can exploit this (at least, i *think* this is the
cause) to create a file that can never be removed on ken's fileserver:

#include <u.h>
#include <libc.h>
void
main(void)
{
	char f[] = {0xc0, 0xb0, 0};
	create(f, OWRITE, 0666)
}

don't try this at home...

Re: [9fans] UTF-8 criticism?

[Joel Salomon]

>> Would moving to 32 bit signed (and only 0 -- 2^21 allowed, plus -1 for
>> EOF) as in the more recent revisions of Unicode take care of the
>> surrogates problem?
>
> this has nothing to do with EOF.

Sorry if I was unclear - let me try again. Would moving to 32 bit signed
(and only 0 -- 2^21 allowed), thus including all surrogates in the
directly accessible character set solve the problem?

Yes, this does open a new can of worms, but how much more difficult would
it be to move from 16 bit Runes to 21/32 bit wide Runes then it was to
move from 7 bit ASCII to Unicode in the first place?

As an aside, the way I've understood the Unicode standard (4.0), 21 bit
characters can be encoded in 1, 2, 3, or 4 bytes in UTF-8 and if text is
internally represented by int32, some out-of-band information (like EOF,
or bad UTF (but preserving the original bytes)) can be carried along.

--Joel

Re: [9fans] UTF-8 criticism?

[andrey mirtchovski]

> And I want native tengwar support... Go Geeks!!

sure thing. at plan9.ucalgary.ca

	% font=tengwar.20.font rio

gives you:

	http://pages.cpsc.ucalgary.ca/~mirtchov/screenshots/tengwar.gif

(acme mail displaying your message second from top :)

Tengwar Re: [9fans] UTF-8 criticism?

[Joel Salomon]

>> And I want native tengwar support... Go Geeks!!
>
> sure thing. at plan9.ucalgary.ca
> 	% font=tengwar.20.font rio
> gives you:
> 	http://pages.cpsc.ucalgary.ca/~mirtchov/screenshots/tengwar.gif
> (acme mail displaying your message second from top :)
>
Thanks. Still, it'd be nice to have the code spaces (tentatively)
allocated for tengwar available - and they're not in the first 65535
characters in Unicode (old unicode, now the BMP).

Of course, there's more to Unicode than the character set, but most of
that is in the text processing domain (troff, TeX).

--Joel

Re: [9fans] UTF-8 criticism?

[Joel Salomon]

Joel Salomon said:
> As an aside, the way I've understood the Unicode standard (4.0), 21 bit
> characters can be encoded in 1, 2, 3, or 4 bytes in UTF-8 and if text is
> internally represented by int32, some out-of-band information (like EOF,
> or bad UTF (but preserving the original bytes)) can be carried along.
>

And here's where the out-of-band encoding might come in useful:

rog@vitanuova.com said:
> you do have to be a bit careful with utf-8, as many possible byte
> sequences map down to the same rune (error), so if you
> do your comparisons too early, you run the risk of inconsistency.
>
> for instance, you can exploit this (at least, i *think* this is the
> cause) to create a file that can never be removed on ken's fileserver:
<snip>

but if "error" becomes 0x80000000 & XX, where XX is the original (bad, or
out-of-place) byte, we never lose the ability to retrieve/delete the file.
This would be an extension to Unicode, possibly a dangerous one, but maybe
worth considering.

--Joel

Re: [9fans] UTF-8 criticism?

[Douglas A. Gwyn]

Joel Salomon wrote:
> Would moving to 32 bit signed (and only 0 -- 2^21 allowed, plus -1 for
> EOF) as in the more recent revisions of Unicode take care of the
> surrogates problem?

Sure.  Actually it's not "more recent", as that's the
original ISO-10646 code points, predating Unicode.