RE: [Caml-list] GSoC: better UTF-8 support

[David Allsopp <dra-news@metastack.com>]

Christophe TROESTLER wrote:
> - UTF8.Char and UTF8.String modules should be written with the same
>   interface as Char and String.  [Camomile should be adapted
>   consequently.]

Thinking of conventions like Unix/Pervasives.LargeFile, Bigarray.Genarray, Bigarray.Array1, etc. wouldn't it be better for these to be Char.UTF8 and String.UTF8? 

> - Printf/Scanf: %U of %cu for UTF8.Char.t
> 
> - Graphics: UTF-8 text printing
> 
> - Str: (character ranges)

If UTF-8 support is added to the standard library then it should be added everywhere where strings are manipulated or used - which rears the potentially ugly prospect of the Unix module?

> The questions are: would such changes be beneficial to you?

Personally, yes - it's an annoying limitation that you have to pull in a 3rd party library when all you want to do is handle a couple of accented characters accurately (my point being that not every application which needs UTF-8 needs it as a priority feature and isn't necessarily manipulating terabytes of data so requires completely optimised processing).

IMO it'd be better to have a standard library only supporting one particular Unicode encoding with a perhaps imperfect interface over a non-optimal storage representation than to have no support whatsoever, especially given that there are very good 3rd party libraries which provide the optimal (and with it, slightly more complex) implementations.

> Are there other issues to address?

I found this very old archive thread but it still poses some potentially relevant points: http://caml.inria.fr/pub/old_caml_site/caml-list/1224.html

> Is this enough for a GSoc proposal (seems a little light to me)?

I would posit that if this included the Unix module then it's a very big proposal!

> If it is done, is there a chance to have this work included in the standard distribution?

If the patches themselves are as potentially small as suggested (so maintenance issues aren't vastly increased) and the interfaces remain compatible (so nothing breaks) then it seems reasonable to hope, doesn't it?


David