[9fans] About The Codes Beyond Unicode-BMP

[9fans] About The Codes Beyond Unicode-BMP

[Hongzheng Wang]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

Hi,

I did an experiment to test if the programs in Plan9 could support the
codes beyond Unicode-BMP.
The result is not so good.

Let's repeat it:

Take U+01000 code for example.  Create a file and fill it with only
one character U+010000 encoded
in UTF-8.

Note that It is could be done with Vim on Linux since Vim has a good
support.  Double check could
be done by Nvi.  The internal representation of U+010000's with UTF-8
is F0908080 [1].

Open the file by ed or sam or acme.  Of course, it could not be
displayed correctly since no fonts in
system could coverage such a code yet.  Then, just re-write the file
again.  Then open it again by
non-Plan9 program, say, Nvi on Linux.  The internal representation became
EFBFBDEFBFBDEFBFBDEFBFBD.  That is, both ed and sam (also acme) failed
to recognize
U+010000 encoded by UTF-8, and destroyed it when writing.

So, does Plan9 acctually supports only the codes in Unicode-BMP?

BTW: the attachment is the gzipped test file containing only U+010000
encoded by UTF-8.

[1] http://en.wikipedia.org/wiki/UTF-8

--
HZ

[-- Attachment #2: test.gz --]
[-- Type: application/x-gzip, Size: 30 bytes --]

Re: [9fans] About The Codes Beyond Unicode-BMP

[erik quanstrom]

plan 9 supports utf16.  that is codpoints u+0000 — u+fffff.  there is no
support for 32bit characters.  to support larger characters, the starting point
would be changing Rune from ushort to ulong and changing constants like
UTFmax and fixing chartorune and runetochar.  (and finding all the places
that assume that UTFmax really is 3.)

it's all very doable, but it would be a very invasive change.

- erik

Re: [9fans] About The Codes Beyond Unicode-BMP

[Hongzheng Wang]

I see.  Thanks.

On Thu, Mar 13, 2008 at 10:55 PM, erik quanstrom <quanstro@coraid.com> wrote:
> plan 9 supports utf16.  that is codpoints u+0000 — u+fffff.  there is no
>  support for 32bit characters.  to support larger characters, the starting point
>  would be changing Rune from ushort to ulong and changing constants like
>  UTFmax and fixing chartorune and runetochar.  (and finding all the places
>  that assume that UTFmax really is 3.)
>
>  it's all very doable, but it would be a very invasive change.
>
>  - erik
>
>



-- 
HZ

Re: [9fans] About The Codes Beyond Unicode-BMP

[Russ Cox]

> plan 9 supports utf16.  that is codpoints u+0000 — u+fffff.  there is no
> support for 32bit characters. 

this is correct except for the use of the term utf16,
which is a character encoding, not a character set.
the subject line is correct - plan 9 doesn't support
codes beyond the BMP.

> to support larger characters, the starting point
> would be changing Rune from ushort to ulong and changing constants like
> UTFmax and fixing chartorune and runetochar.  (and finding all the places
> that assume that UTFmax really is 3.)
> it's all very doable, but it would be a very invasive change.

it would require recompiling everything, 
but i don't believe it would require changes
to code beyond the utf routines in the c library.
i do not believe there are many places (if any)
that presume to know the value of UTFmax.

russ

Re: [9fans] About The Codes Beyond Unicode-BMP

[erik quanstrom]

> it would require recompiling everything,
> but i don't believe it would require changes
> to code beyond the utf routines in the c library.
> i do not believe there are many places (if any)
> that presume to know the value of UTFmax.

you just pointed one out yesterday -- in devatach().

- erik

Re: [9fans] About The Codes Beyond Unicode-BMP

[Joel C. Salomon]

On Thu, Mar 13, 2008 at 10:55 AM, erik quanstrom <quanstro@coraid.com> wrote:
> plan 9 supports utf16.  that is codpoints u+0000 — u+fffff.

To be pedantic, UTF-16 has the ability to represent characters in the
'astral planes' via surrogate pairs (pairs of character in the range
U+D800–U+DFFF); Plan 9's charset is approximately UCS-2.

Java has the same trouble; its astral plane characters are first
encoded as UTF-16 surrogate pairs, then those 16-bit values are
encoded as UTF-8.

> to support larger characters, the starting point would be changing Rune
> from ushort to ulong and changing constants like UTFmax and fixing
> chartorune and runetochar.  (and finding all the places that assume that
> UTFmax really is 3.)
>
> it's all very doable, but it would be a very invasive change.

Not really, since only the 2²⁰+2¹⁶ values from 0–0x10FFFF are needed
and UTFmax only needs to go up to 4.  An advantage would be that
out-of-band symbols like EOF and yacc terminals could be represented
in the same data type as the characters

On the other hand, there are more useful bits of Unicode that are
unimplemented in Plan 9.  Mañana (as in /sys/doc/utf.{html,ps,pdf}
never did come.

--Joel