Unicode, update

Unicode, update

[Paul Steckler]

A couple of weeks ago or so, I asked about using OCaml file primitives
with the Camomile library for Unicode
on Windows.  I thought I'd update people on the list about my
resolution of these issues.

I decided to make the application UTF-8 throughout, so that the string
type always means UTF-8 -- OK, there
are a few exceptions to that rule.  The SQLite3 library already deals
with UTF-8 in a graceful way,   The same is
true for the C/C++ parsing library I'm using.  That leaves the OCaml
library procedures, like open_in and open_out,
which definitely don't handle Unicode filenames on Windows.

I took the OCaml sources and made modified versions of functions, like
file_exists, open_in, and so on,
that convert filenames from UTF-8 to UTF-16 and then used "wide"
versions of the underlying Win32
primitives.  In some cases, I had to convert UTF-16 back to UTF-8.
The Win32 functions MultiByteToWideChar
and WideCharToMultiByte handle those conversions nicely.  I link in
these new functions, named
file_exists_win32, open_in_win32, etc., and everything works a treat.

-- Paul

Re: Unicode, update

[Sylvain Le Gall]

Hello,

On 14-10-2010, Paul Steckler <steck@stecksoft.com> wrote:
> A couple of weeks ago or so, I asked about using OCaml file primitives
> with the Camomile library for Unicode
> on Windows.  I thought I'd update people on the list about my
> resolution of these issues.
>
> I decided to make the application UTF-8 throughout, so that the string
> type always means UTF-8 -- OK, there
> are a few exceptions to that rule.  The SQLite3 library already deals
> with UTF-8 in a graceful way,   The same is
> true for the C/C++ parsing library I'm using.  That leaves the OCaml
> library procedures, like open_in and open_out,
> which definitely don't handle Unicode filenames on Windows.
>
> I took the OCaml sources and made modified versions of functions, like
> file_exists, open_in, and so on,
> that convert filenames from UTF-8 to UTF-16 and then used "wide"
> versions of the underlying Win32
> primitives.  In some cases, I had to convert UTF-16 back to UTF-8.
> The Win32 functions MultiByteToWideChar
> and WideCharToMultiByte handle those conversions nicely.  I link in
> these new functions, named
> file_exists_win32, open_in_win32, etc., and everything works a treat.
>

Would it be possible to publish them as an external library? 

Thanks for the update
Sylvain Le Gall

Re: Unicode, update

[Paul Steckler]

Sylvain Le Gall <sylvain@le-gall.net> wrote:
> Would it be possible to publish them as an external library?

What I did isn't really complete enough to constitute a library, I'd say.

Here's what I did:

>From Pervasives:

  open_out_win32
  open_out_bin_win32
  open_out_gen_win32
  open_in_win32
  open_in_bin_win32
  open_in_gen_win32

>From Sys:

  file_exists_win32
  getcwd_win32
  chdir_win32
  missing: is_directory_win32, readdir_win32

I did not code up Win32/UTF8 equivalents of anything in the Unix
module.  A complete
library of Win32/UTF8 file functions would include a number of items
from that module.

If anyone's interested in finishing off what I've done, I could send along
what I have.

-- Paul