Re: [Caml-list] Re: Rope is the new string

["Loup Vaillant" <loup.vaillant@gmail.com>]

2007/10/9, Vincent Hanquez <tab@snarc.org>:
> On Tue, Oct 09, 2007 at 07:32:25PM +0200, Loup Vaillant wrote:
> > > definitely we also need some UTFstring type library (which can use rope,
> > > string, whatever internally), with all common type of operations
> > > (appending, finding, ...), but it's a just a specific sub case and also
> > > a different type not compatible with strings (in OCaml terminology).
> >
> > Then, we should have both byte arrays (the native Ocaml strings), and
> > unicode strings. We will also need proper syntactic sugar for unicode
> > strings. Operators, and literal values (like #"example"). Only then,
> > ropes could feel like native strings --and be useful as such.
>
> not sure If i see your point here, since your are mixing rope and
> unicode. however I think we are missing some other type of string
> implementation (maybe rope) *along* the current implementation of
> string.

Sorry. I just saw ropes as a possible implementation for unicode
strings (because there is no standard yet, we have the choice). By the
way, there is a good chance they are better than flat string for most
purposes.

> while we also miss unicode support somehow integrated, what
> implementation of the underlaying basic byte string is used, is
> irrevelant.

I agree.

> > > [...] it's a just a specific sub case [...]
> >
> > Internationalization is, mere text crunching is not. (You meant that,
> > right?) With properly interfaced unicode strings, I can do my text
> > crunching without worrying about internationalization, and with no
> > programming overhead. Then, when (if) I have to internationalize, it
> > is much easier.
>
> Absolutely. What I meant basicly resume into, that unicode strings are
> just a subset of strings (as array of bytes). you can store a unicode
> string in a byte string, whereas you can't store a byte string into a
> unicode string.
>
> i want a UTF library to be able to do something like:
>
> type ustring = unicode_type * string
> of_string: string -> ustring (* raise if not unicode compliant *)
> to_string: ustring -> string
> append: ustring -> ustring -> ustring
> ...etc

Err, I would prefer this:

type ustring (* the type should be abstract, I think *)
of_string_utf8: string -> ustring (* raise if not utf8 compliant *)
to_string_utf8: ustring -> string
of_string_utf16: string -> ustring (* raise if not utf16 compliant *)
to_string_utf16: ustring -> string
of_string_latin1: string -> ustring (* raise if not latin1 compliant *)
to_string_latin1: ustring -> string (* raise if characters not encoded
in Latin1 (the exeption should contain a usefull result) *)
(* etc for each encoding *)
scan_utf8 (* raise if not utf8 compliant (but do not lose a possible
partial result) *)
print_utf8
(* etc for each encoding *)

append: ustring -> ustring -> ustring
(* etc *) (* I agree on these ones *)

So I can mix-up different encodings:

print
  (append
    scan_Latin1
    (of_string text))
(* this is not Lisp *)

Just that the sample code you wrote suggest you could have different
types of unicode strings. I want only one type, so I don't mind the
encoding, except when reading and printing (to files and native
strings). To mind even less, you could have a general scan and
from_string functions which guess which encoding is used (not very
safe, but cool)

> that way when I'm manipulating unicode string, i won't try to append a
> binary string to a unicode string. I can code safely with my unicode
> string (whatever the format utf-{8..32}), and certainly expect the type
> system to complain loudly when doing something that might break unicode.

The exception system can do that, but how could the type system?
(Would be better if it could.)

Loup Vaillant