RE: [Caml-list] GSoC: better UTF-8 support

[David Allsopp <dra-news@metastack.com>]

Daniel Bünzli wrote:
> > If it's to go into the standard library then yes, it should exactly
> > replicate the interface of the Char and String modules, that way
> > within the standard library UTF8 can be used as a drop-in replacement
> [...]
> 
> I'm not sure many programs would actually benefit from that. At a certain
> point if you really want to process unicode at the character level you'll
> need a proper library.

Yes, and at that point you'd use one. Not all programs need to analyse strings at a character level. For example, it would be nice for this to work within just the standard library:

D:\>md "Paweł Łukaszewski"
D:\>cd "Paweł Łukaszewski"
D:\Paweł Łukaszewski>ocaml
        Objective Caml version 3.11.2

# Sys.getcwd();;
- : string = "D:\\Pawel Lukaszewski"

The reason for this is because an internal conversion is done to hack around Unicode filenames - but having any representation of Unicode in the standard library would mean that these and other functions would no longer have to return incorrect answers, as in this case, but the standard library would have a default mechanism for dealing with Unicode. At the moment, there's nothing the standard library can do with this Unicode filename because there's no standardised (within the library) representation available for it.

Consider it as being similar to something like the Digest module. It's only really there because digests are used in .cmi files within the compiler. If you're serious about using digests in an application then you have to use an external library because MD5 is not usually enough and you'll need other algorithms. But the module is still potentially useful so it's better that it's publically visible in the standard library and not just an internal module of the compiler. Same would apply to this simple native support for UTF-8 - it would allow other parts of the standard library to work properly for simple applications would allow Unicode strings to be handled.

> Using these ascii/latin1 oriented interfaces to
> process unicode at the character level would be debilitating and
> frustrating for your final users (e.g. no treatement of normal forms, you
> do realize that in unicode there's more than one way of representing the
> character 'é').

Fully aware - but just because you need to work with strings does *not* imply you ever need even to compare them. Granted, the documentation may note that to perform a canonical comparison you'll need a third party library but I still maintain that having basic support is better and more usable than having none. The above issue, for example, means I can't even manipulate a directory tree in OCaml on my system which shouldn't even be related to string/text processing!


David