Re: [Caml-list] Immutable strings

["Daniel Bünzli" <daniel.buenzli@erratique.ch>]

Le mardi, 8 juillet 2014 à 19:15, mattiasw@gmail.com a écrit :
> ocaml will be that last language that doesn't have standardize unicode
> support. Even old languages like Erlang has gone the UTF-8 way, and that
> includes program code.

For the fun I just had a look what python does.  

So in python basically they have a Unicode string which is a string made of Unicode *code points*. Fail, end of discussion. Should have been: *scalar values* (for those who don't understand why, I suggest reading my minimal Unicode introduction [1]).

(both in 2 and 3, apparently 2 used to be messier for reason I didn't bother to understand, they seem to be highly confused)

Sample code. U+D800 is the first surrogate, i.e. something you should never see in concrete Unicode textual processing, only in UTF-16 encoded bytes and paired with an appropriate low surrogate.

Python2:

>>> u'\uD800'.encode('utf-8')
'\xed\xa0\x80'

Congratulations, you just produced an invalid UTF-8 sequence (serialized a surrogate).  

Python3 is a *little* better with *UTF-8* (but wait…) encoding stuff

>>> "\uD800".encode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 0: surrogates not allowed


So now let's try UTF-16:

>>> "\uD800".encode("utf-16")
b'\xff\xfe\x00\xd8'


Congratulations you just produced an invalid UTF-16 sequence hi-surrogate without a corresponding low surrogate (which together would define an Unicode scalar value).

Why on earth do they allow to represent surrogates *at all* in their Unicode text data structure ? Basically they don't understand Unicode.  

The old camel should not be ashamed of its *outsanding* (absolutely) unicode support — this is not to say that nothing can be improved, I do have some proposal in the works — but the situation is not bad either.

Best,

Daniel

P.S. Skimming through these articles about python unicode strings I gather why people find unicode hard, there seem to be a high level of both technical and conceptual confusion. Again have a read at [1] if you'd like to clear (I hope) your mind about these things.


[1] http://erratique.ch/software/uucp/doc/Uucp.html#uminimal