Re: [Caml-list] GSoC: better UTF-8 support

["Daniel Bünzli" <daniel.buenzli@erratique.ch>]

> Thinking more about this, one could introduce a new type (say “utf8”
> or “ustring”) for these UTF-8 strings.  It should be compatible with
> the way UTF-8 strings are handled on the C side for interoperability
> but “optimized” — e.g. should they contain their length (number of
> unicode chars)?
>
> Another thing: it could be a nice way to transition to *immutable*
> unicode strings.  This is not possible for (standard) strings because,
> as you all know, they are both used as strings and as buffers.  The
> introduction of unicode strings may be the right opportunity to
> distinguish both [1].

Frankly I see no benefit of introducing this half-baked UTF-8 support
into the standard library (which is what this proposal is about).

This will just bring in more noise in the interfaces. Even worse,
developers will think they handle unicode properly while they do in
fact not, bringing more confusion on already confusing topic (I'm
always surprised the little programmers know about unicode). Again,
pretending supporting unicode character level processing by replacing
latin1 character level processing the way you suggest is just plain
wrong.

For me either you :

1) Provide full unicode support in the standard library with at least
normal form and collation support in a new API, separate from the
current, existing String and Char modules.

2) Leave full unicode support to a third party library and keep the
current state with some improvements for coping with UTF-8 encoded
string literals and for interacting with file systems correctly.

Given signals given in the past by the ocaml dev team 2) seems more
likely to be accepted.

Best,

Daniel