[9fans] strangely typed functions in standard library

[9fans] strangely typed functions in standard library

[Matt Stewart]

The following functions are described as accepting a Rune, but instead
the parameters are of type long.  Why?

int runelen(long);
char *utfrune(char *, long);
char *utfrrune(char *, long);

Re: [9fans] strangely typed functions in standard library

[Martin Neubauer]

* Matt Stewart (rotaerk1@gmail.com) wrote:
> The following functions are described as accepting a Rune, but instead
> the parameters are of type long.  Why?
> 
> int runelen(long);
> char *utfrune(char *, long);
> char *utfrrune(char *, long);
> 

Though I'm far from being an expert on that matter, I would assume
it's for the same reason putchar() takes an int.

Re: [9fans] strangely typed functions in standard library

[R]

On 5/16/06, Matt Stewart <rotaerk1@gmail.com> wrote:
> The following functions are described as accepting a Rune, but instead
> the parameters are of type long.  Why?
>
> int runelen(long);
> char *utfrune(char *, long);
> char *utfrrune(char *, long);

full unicode is 32 bit, even if plan9 (afaik)
supports only characters in the BMP.

Re: [9fans] strangely typed functions in standard library

[quanstro]

while true, this is not the reason for the function signature.  this is for
type promotion reasons.  Rune is still an unsigned short.  plan 9 does not
support ucs-4.

- erik

On Fri May 19 17:51:07 CDT 2006, viriketo@gmail.com wrote:

> Matt Stewart wrote:
> > The following functions are described as accepting a Rune, but instead
> > the parameters are of type long.  Why?
> > 
> > int runelen(long);
> > char *utfrune(char *, long);
> > char *utfrrune(char *, long);
> > 
>  From History in this wikipedia page (http://en.wikipedia.org/wiki/UTF-32):
> 
> UCS-4 is sufficient to represent all of the Unicode code space, which 
> has 1114112 (= 2^20+2^16) code points and therefore requires only up to 
> hexadecimal 10FFFF. Some people consider it wasteful to reserve such a 
> large code space for mapping a relatively small set of code points, so a 
> new encoding form, UTF-32, was proposed.

Re: [9fans] strangely typed functions in standard library

[Lluís Batlle i Rossell]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

Matt Stewart wrote:
> The following functions are described as accepting a Rune, but instead
> the parameters are of type long.  Why?
> 
> int runelen(long);
> char *utfrune(char *, long);
> char *utfrrune(char *, long);
> 
 From History in this wikipedia page (http://en.wikipedia.org/wiki/UTF-32):

UCS-4 is sufficient to represent all of the Unicode code space, which 
has 1114112 (= 2^20+2^16) code points and therefore requires only up to 
hexadecimal 10FFFF. Some people consider it wasteful to reserve such a 
large code space for mapping a relatively small set of code points, so a 
new encoding form, UTF-32, was proposed.

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3311 bytes --]

Re: [9fans] strangely typed functions in standard library

[Victor Nazarov]

Joel Salomon wrote:

> On 5/18/06, Bruce Ellis <bruce.ellis@gmail.com> wrote:
>
>> 32 bit unicode is not Rune friendly
>
>
> The "other" standard, ISO 10646, has promised that 21 bits will always
> be sufficient to represent characters.
>
> Making Rune a 32 bit type allows all characters to be represented and
> leaves room for out-of-band information; for example, the end of a
> utf8 text stream (EOF) can be (Rune32)-1, with no need for a wider
> type.
>
> --Joel

Oh, I think it is resonable. I thought that ISO10656 is 32 bit, so EOF 
detection
and so on will break expanding Rune to 32 bit...

Composing seems to me a better solution than encoding everything to one
alphabet. I mean sorting, transformations and so on (convertion to ASCII 
at least).
But I really not competent in this question.

--
Victor

Re: [9fans] strangely typed functions in standard library

[Joel Salomon]

On 5/18/06, Bruce Ellis <bruce.ellis@gmail.com> wrote:
> 32 bit unicode is not Rune friendly

The "other" standard, ISO 10646, has promised that 21 bits will always
be sufficient to represent characters.

Making Rune a 32 bit type allows all characters to be represented and
leaves room for out-of-band information; for example, the end of a
utf8 text stream (EOF) can be (Rune32)-1, with no need for a wider
type.

--Joel

Re: [9fans] strangely typed functions in standard library

[geoff]

I feel that the Unicode consortium bungled the job.  People warned
them from the outset that 16 bits was a largish space, but still
finite.  Once Han unification was done, the Unicode people felt that
they had all the room in the world, and assigned generous portions of
the space to each alphabet.  Others warned them that they should be
more conservative in their allocations, but were waved off.

It may be that 16 bits are not enough to express all the necessary
characters in the world, but I don't think that the Unicode consortium
have proven that case.

Re: [9fans] strangely typed functions in standard library

[quanstro]

of course, how an array of Runes is stored is a different issue from how a
Rune argument is passed on the stack (or in a register). 

don't consider this advocacy, but what would break if runes were expanded
to 32 bits?  i think that font handling would need some limits changed,
chartorune and runetochar would need to be modified and some enum
constants in libc.h would need to be changed.  this seems to me to be small
potatoes in comparison to dealing with issues lurking within the basic plane
like character composition.

rob has suggested passing uncomposed characters to libdraw and handling
the problem there.  but there's one problem with that.  how do you stick
a nonspacing horn onto an arbitrary letter?  how do you put a grave accent
on top of that?  (transliterations of cryllic to the roman alphabet use some 
double- and triple- accented letters which do not exist in precombined form
within unicode.)

i modified p9p libdraw at one point to draw combining characters.  (i.e. compose
a zero-width character on top of the previous character.) the results
were not legible.  a+diaresis may have been marginal buit A + " was mud.

this is really annoying.  characters should not combine.  the character-composition
algorithims of tex+metafont shouldn't come hidden within a character set.  

(well that's my 2¢, anyway.  maybe somebody has an idea on how to manage these
issues.)

- erik

p.s. how would you do this?  

> i'd like to map them to RFat ... something unassigned in
> 0xFF.. space.

the problem is that there's not enough free space in the basic plane. and as usual 
with these things invention is the mother of all necessity.


On Thu May 18 04:22:47 CDT 2006, bruce.ellis@gmail.com wrote:
> 32 bit unicode is not Rune friendly ... i hope Runes don't
> get fatter.  it will break many things.  rob has had something
> to say about this, do a search on the list.
> 
> i'd like to map them to RFat ... something unassigned in
> 0xFF.. space.
> 
> use them at your peril.
> 
> brucee
> 
> On 5/18/06, erik quanstrom <quanstro@quanstro.net> wrote:
> > while this is true, i believe that the real reason for this is that
> > on a >=32-bit machine, an ushort can just be declared
> > to be a long by the compiler whereas the compiler must emit
> > instructions to convert a long to an unsigned short.
> >
> > - erik
> >
> > On Tue May 16 10:10:37 CDT 2006, 0xef967c36@gmail.com wrote:
> > > On 5/16/06, Matt Stewart <rotaerk1@gmail.com> wrote:
> > > > The following functions are described as accepting a Rune, but instead
> > > > the parameters are of type long.  Why?
> > > >
> > > > int runelen(long);
> > > > char *utfrune(char *, long);
> > > > char *utfrrune(char *, long);
> > >
> > > full unicode is 32 bit, even if plan9 (afaik)
> > > supports only characters in the BMP.
> >
> 

Re: [9fans] strangely typed functions in standard library

[Bruce Ellis]

32 bit unicode is not Rune friendly ... i hope Runes don't
get fatter.  it will break many things.  rob has had something
to say about this, do a search on the list.

i'd like to map them to RFat ... something unassigned in
0xFF.. space.

use them at your peril.

brucee

On 5/18/06, erik quanstrom <quanstro@quanstro.net> wrote:
> while this is true, i believe that the real reason for this is that
> on a >=32-bit machine, an ushort can just be declared
> to be a long by the compiler whereas the compiler must emit
> instructions to convert a long to an unsigned short.
>
> - erik
>
> On Tue May 16 10:10:37 CDT 2006, 0xef967c36@gmail.com wrote:
> > On 5/16/06, Matt Stewart <rotaerk1@gmail.com> wrote:
> > > The following functions are described as accepting a Rune, but instead
> > > the parameters are of type long.  Why?
> > >
> > > int runelen(long);
> > > char *utfrune(char *, long);
> > > char *utfrrune(char *, long);
> >
> > full unicode is 32 bit, even if plan9 (afaik)
> > supports only characters in the BMP.
>

Re: [9fans] strangely typed functions in standard library

[erik quanstrom]

while this is true, i believe that the real reason for this is that
on a >=32-bit machine, an ushort can just be declared
to be a long by the compiler whereas the compiler must emit 
instructions to convert a long to an unsigned short.

- erik

On Tue May 16 10:10:37 CDT 2006, 0xef967c36@gmail.com wrote:
> On 5/16/06, Matt Stewart <rotaerk1@gmail.com> wrote:
> > The following functions are described as accepting a Rune, but instead
> > the parameters are of type long.  Why?
> >
> > int runelen(long);
> > char *utfrune(char *, long);
> > char *utfrrune(char *, long);
> 
> full unicode is 32 bit, even if plan9 (afaik)
> supports only characters in the BMP.