[Caml-list] Supporting unicode in ocaml...

[Caml-list] Supporting unicode in ocaml...

[Jonathan Roewen]

Hi,

Does the ocaml team ever plan on supporting unicode to some degree?

What about being able to parse utf-8 encoded files, but keeping the
ascii only grammar? Then with the only change that if it's a utf-8
file, that the utf-8 encoding of string constants are maintained. With
this scheme, you could theoretically bail on non-ascii characters
everywhere else. And a 3rd-party library like camomile could be used
for higher-level processing of the utf8-encoded string constants (from
the camomile docs, utf8 strings use the ocaml string type too).

I must admit that I have no idea how complex even a seemingly small
change like this may be, but at least the detection of the byte order
mark should make it a compatible change...

Jonathan

Re: [Caml-list] Supporting unicode in ocaml...

[Richard Jones]

On Sat, Aug 19, 2006 at 11:37:47AM +1200, Jonathan Roewen wrote:
> Does the ocaml team ever plan on supporting unicode to some degree?
> 
> What about being able to parse utf-8 encoded files, but keeping the
> ascii only grammar? Then with the only change that if it's a utf-8
> file, that the utf-8 encoding of string constants are maintained. With
> this scheme, you could theoretically bail on non-ascii characters
> everywhere else. And a 3rd-party library like camomile could be used
> for higher-level processing of the utf8-encoded string constants (from
> the camomile docs, utf8 strings use the ocaml string type too).

Have a look at Camomile:

http://camomile.sourceforge.net/

Generally speaking, though, I just always use string == UTF-8 string
and avoid using some of the unsafe functions from the standard
library, such as String.lowercase.

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

Re: [Caml-list] Supporting unicode in ocaml...

[Jonathan Roewen]

> Have a look at Camomile:
>
> http://camomile.sourceforge.net/
>
> Generally speaking, though, I just always use string == UTF-8 string
> and avoid using some of the unsafe functions from the standard
> library, such as String.lowercase.

Yes I did (as if you read closer, I did mention). But being able to
use actual utf8 rather than manually encoding characters in the string
is what would be nice. Maybe I just need a script to preprocess
sources in utf8...

Re: [Caml-list] Supporting unicode in ocaml...

[Peter Jolly]

Jonathan Roewen wrote:
> Yes I did (as if you read closer, I did mention). But being able to
> use actual utf8 rather than manually encoding characters in the string
> is what would be nice. Maybe I just need a script to preprocess
> sources in utf8...

AFAIK, all you need to do is get rid of the BOM, which is redundant in
UTF-8 anyway.  I have no problems at all using UTF-8 encoded string
constants in OCaml, and Camomile's UTF8 module then handles them
seamlessly.  YMMV.

(There is one minor issue, which is that there are legal OCaml
identifiers which become illegal when the file is encoded in UTF-8 --
namely those that use accented letters.)