Batteries and portability

Batteries and portability

[Tiphaine Turpin]

Dear list,

When trying to execute a compiled program on another machine, I get the
following error:

Fatal error: exception
Sys_error("/usr/share/camomile/database/general_category.mar: No such
file or directory")

I don't think I'm using camomille and I'm not compiling with -linkall,
however linking takes a lot of time in my opinion (~15 seconds for about
2000loc). Is it the expected behavior of batteries to always expect this
file (and maybe others) to be present ?


Tiphaine

Re: [Caml-list] Batteries and portability

[Edgar Friendly]

Tiphaine Turpin wrote:
> Dear list,
> 
> When trying to execute a compiled program on another machine, I get the
> following error:
> 
> Fatal error: exception
> Sys_error("/usr/share/camomile/database/general_category.mar: No such
> file or directory")
> 
> I don't think I'm using camomille and I'm not compiling with -linkall,
> however linking takes a lot of time in my opinion (~15 seconds for about
> 2000loc). Is it the expected behavior of batteries to always expect this
> file (and maybe others) to be present ?
> 
> 
> Tiphaine
> 
Batteries uses camomile (one L) for its UTF8 ropes.  It is the expected
behavior for it to load those files on startup even if one doesn't use
ropes.  We'd love for this kind of thing to be optimized out by the
compiler, but this might be one case where the compiler can't tell
whether the side-effects of module initialization (reading from files)
can be elided.

If someone has a solution for this, we'd love to solve this problem in a
not-too-hackish way, but I worry that a non-camomile build of batteries
is the only way to eliminate this dependency.  Of course it's possible
to depend on files stored elsewhere, but camomile is quite set up to use
external files for its unicode conversion tables.

E

Re: [Caml-list] Batteries and portability

[Tiphaine Turpin]

Edgar Friendly a écrit :
> If someone has a solution for this, we'd love to solve this problem in a
> not-too-hackish way, but I worry that a non-camomile build of batteries
> is the only way to eliminate this dependency.
Is it possible ? I'm not able to run the configure script without Camomile.

Re: [Caml-list] Batteries and portability

[David MENTRE]

Hello,

On Thu, Jun 11, 2009 at 16:10, Edgar Friendly<thelema314@gmail.com> wrote:
> If someone has a solution for this, we'd love to solve this problem in a
> not-too-hackish way, but I worry that a non-camomile build of batteries
> is the only way to eliminate this dependency.  Of course it's possible
> to depend on files stored elsewhere, but camomile is quite set up to use
> external files for its unicode conversion tables.

Wouldn't it be possible to convert those files into OCaml strings and
use them instead? They would be included in the compiled module but
not incorporated in the final binary except if needed.

Yours,
david

Re: [Caml-list] Batteries and portability

[dmitry grebeniuk]

> When trying to execute a compiled program on another machine, I get the
> following error:
>
> Fatal error: exception
> Sys_error("/usr/share/camomile/database/general_category.mar: No such
> file or directory")
>
> I don't think I'm using camomille and I'm not compiling with -linkall

  This error appears because of camomile's
module initialization (module Default with default config
values).  If you really don't use camomile and it is not used
in batteries indirectly, you can defer initialization.
Apply the patch to camomile:
http://overbld.abcname.net/files/camomile-noprecomp.html
  But the patch has a [small] runtime cost.

Re: [Caml-list] Batteries and portability

[dmitry grebeniuk]

> When trying to execute a compiled program on another machine, I get the
> following error:
>
> Fatal error: exception
> Sys_error("/usr/share/camomile/database/general_category.mar: No such
> file or directory")
>
> I don't think I'm using camomille and I'm not compiling with -linkall

  This error appears because of camomile's
module initialization (module Default with default config
values).  If you really don't use camomile and it is not used
in batteries indirectly, you can defer initialization.
Apply the patch to camomile:
http://overbld.abcname.net/files/camomile-noprecomp.html
  But the patch has a [small] runtime cost.

Re: [Caml-list] Batteries and portability

[Tiphaine Turpin]

I'll try that. thanks a lot.

Tiphaine


dmitry grebeniuk a écrit :
>> When trying to execute a compiled program on another machine, I get the
>> following error:
>>
>> Fatal error: exception
>> Sys_error("/usr/share/camomile/database/general_category.mar: No such
>> file or directory")
>>
>> I don't think I'm using camomille and I'm not compiling with -linkall
>>     
>
>   This error appears because of camomile's
> module initialization (module Default with default config
> values).  If you really don't use camomile and it is not used
> in batteries indirectly, you can defer initialization.
> Apply the patch to camomile:
> http://overbld.abcname.net/files/camomile-noprecomp.html
>   But the patch has a [small] runtime cost.
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>   

Re: [Caml-list] Batteries and portability

[David Rajchenbach-Teller]

We could try that, but that looks to me like something that would break
more things than it would fix.

Cheers,
 David

On Thu, 2009-06-11 at 17:42 +0200, David MENTRE wrote:
> Wouldn't it be possible to convert those files into OCaml strings and
> use them instead? They would be included in the compiled module but
> not incorporated in the final binary except if needed.
> 
> Yours,
> david

Re: [Caml-list] Batteries and portability

[Tiphaine Turpin]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

The following completed patch works for me.

Tiphaine

dmitry grebeniuk a écrit :
> If you really don't use camomile and it is not used
> in batteries indirectly, you can defer initialization.
> Apply the patch to camomile:
> http://overbld.abcname.net/files/camomile-noprecomp.html
>   But the patch has a [small] runtime cost.
>   

[-- Attachment #2: camomile.patch --]
[-- Type: text/plain, Size: 6210 bytes --]

diff -aur camomile.orig/camomile-0.7.1/public/caseMap.ml camomile-0.7.1/public/caseMap.ml
--- camomile.orig/camomile-0.7.1/public/caseMap.ml	2006-08-14 13:35:03.000000000 +0200
+++ camomile-0.7.1/public/caseMap.ml	2009-06-12 14:45:22.000000000 +0200
@@ -17,35 +17,35 @@
   module UCharInfo = UCharInfo.Make(Config)
   open UCharInfo
 
-let uppercase_tbl = load_property_tbl `Uppercase
+let uppercase_tbl = lazy (load_property_tbl `Uppercase)
 
-let is_uppercase u = UCharTbl.Bool.get uppercase_tbl u 
+let is_uppercase u = UCharTbl.Bool.get (Lazy.force uppercase_tbl) u 
 
-let lowercase_tbl = load_property_tbl `Lowercase
-let is_lowercase u = UCharTbl.Bool.get lowercase_tbl u 
+let lowercase_tbl = lazy (load_property_tbl `Lowercase)
+let is_lowercase u = UCharTbl.Bool.get (Lazy.force lowercase_tbl) u 
 
-let conditional_casing_tbl = load_conditional_casing_tbl ()
-let conditional_casing u = UCharTbl.get conditional_casing_tbl u
+let conditional_casing_tbl = lazy (load_conditional_casing_tbl ())
+let conditional_casing u = UCharTbl.get (Lazy.force conditional_casing_tbl) u
 
-let casefolding_tbl = load_casefolding_tbl ()
-let casefolding_char u = UCharTbl.get casefolding_tbl u
+let casefolding_tbl = lazy (load_casefolding_tbl ())
+let casefolding_char u = UCharTbl.get (Lazy.force casefolding_tbl) u
 
 let is_null u = UChar.uint_code u = 0
 
-let to_lower1_tbl = load_to_lower1_tbl ()
+let to_lower1_tbl = lazy (load_to_lower1_tbl ())
 
 let to_lower1 u = 
-  let u' = UCharTbl.get to_lower1_tbl u in
+  let u' = UCharTbl.get (Lazy.force to_lower1_tbl) u in
   if is_null u' then u else u'
 
-let to_upper1_tbl = load_to_upper1_tbl ()
+let to_upper1_tbl = lazy (load_to_upper1_tbl ())
 let to_upper1 u = 
-  let u' = UCharTbl.get to_upper1_tbl u in
+  let u' = UCharTbl.get (Lazy.force to_upper1_tbl) u in
   if is_null u' then u else u'
 
-let to_title1_tbl = load_to_title1_tbl ()
+let to_title1_tbl = lazy (load_to_title1_tbl ())
 let to_title1 u = 
-  let u' = UCharTbl.get to_title1_tbl u in
+  let u' = UCharTbl.get (Lazy.force to_title1_tbl) u in
   if is_null u' then u else u'
 
 let is_case_ignorable u =
@@ -92,8 +92,8 @@
       in
       search (Text.next t i)
 
-    let soft_dotted_tbl = UCharInfo.load_property_tbl `Soft_Dotted
-    let is_soft_dotted u = UCharTbl.Bool.get soft_dotted_tbl u
+    let soft_dotted_tbl = lazy (UCharInfo.load_property_tbl `Soft_Dotted)
+    let is_soft_dotted u = UCharTbl.Bool.get (Lazy.force soft_dotted_tbl) u
 
     let is_after_soft_dotted t i =
       let rec search i =
diff -aur camomile.orig/camomile-0.7.1/public/uCharInfo.ml camomile-0.7.1/public/uCharInfo.ml
--- camomile.orig/camomile-0.7.1/public/uCharInfo.ml	2006-08-14 13:35:03.000000000 +0200
+++ camomile-0.7.1/public/uCharInfo.ml	2009-06-11 18:46:19.000000000 +0200
@@ -251,11 +251,11 @@
 
 (* General category *)
 
-let general_category_tbl : UCharTbl.Bits.t = 
-  read_data "general_category"
+let general_category_tbl : UCharTbl.Bits.t Lazy.t = 
+  lazy (read_data "general_category")
 
 let general_category u =
-  match UCharTbl.Bits.get general_category_tbl u with
+  match UCharTbl.Bits.get (Lazy.force general_category_tbl) u with
     0 ->
       let n = UChar.uint_code u in
       if n >= 0x0f0000 && n <= 0x100000 then `Co else
@@ -426,9 +426,9 @@
 
 (* Scripts *)
 
-let script_tbl : UCharTbl.Bits.t = read_data  "scripts"
+let script_tbl : UCharTbl.Bits.t Lazy.t = lazy (read_data "scripts")
 
-let script u = script_of_num (UCharTbl.Bits.get script_tbl u)
+let script u = script_of_num (UCharTbl.Bits.get (Lazy.force script_tbl) u)
 let load_script_map () = read_data  "scripts_map"
 
 (* Casing *)
@@ -495,10 +495,10 @@
 
 (* Combined class *)
 
-let combined_class_tbl : UCharTbl.Char.t =
-  read_data  "combined_class"
+let combined_class_tbl : UCharTbl.Char.t Lazy.t =
+  lazy (read_data  "combined_class")
 
-let combined_class u = Char.code (UCharTbl.Char.get combined_class_tbl u)
+let combined_class u = Char.code (UCharTbl.Char.get (Lazy.force combined_class_tbl) u)
 
 (* Decomposition *)
 
diff -aur camomile.orig/camomile-0.7.1/public/uCol.ml camomile-0.7.1/public/uCol.ml
--- camomile.orig/camomile-0.7.1/public/uCol.ml	2006-08-14 13:35:03.000000000 +0200
+++ camomile-0.7.1/public/uCol.ml	2009-06-12 14:47:16.000000000 +0200
@@ -58,10 +58,10 @@
 open UCharInfo
 
 let logical_order_exception_tbl = 
-  UCharInfo.load_property_tbl `Logical_Order_Exception
+  lazy (UCharInfo.load_property_tbl `Logical_Order_Exception)
 
 let is_logical_order_exception u = 
-  UCharTbl.Bool.get logical_order_exception_tbl u
+  UCharTbl.Bool.get (Lazy.force logical_order_exception_tbl) u
 
 let rec rearrange_aux x pos =
   if pos > XString.length x - 2 then () else
@@ -97,10 +97,10 @@
   loop0 0
 
 let noncharacter_code_point_tbl = 
-  UCharInfo.load_property_tbl `Noncharacter_Code_Point
+  lazy (UCharInfo.load_property_tbl `Noncharacter_Code_Point)
 
 let is_noncharacter_code_point u = 
-  UCharTbl.Bool.get noncharacter_code_point_tbl u
+  UCharTbl.Bool.get (Lazy.force noncharacter_code_point_tbl) u
 
 let reverse s =
   if String.length s = 0 then () else
diff -aur camomile.orig/camomile-0.7.1/public/uNF.ml camomile-0.7.1/public/uNF.ml
--- camomile.orig/camomile-0.7.1/public/uNF.ml	2006-08-14 13:35:03.000000000 +0200
+++ camomile-0.7.1/public/uNF.ml	2009-06-12 14:49:42.000000000 +0200
@@ -64,14 +64,14 @@
 
 let null = UChar.chr_of_uint 0
 
-let decomposition_tbl = load_decomposition_tbl ()
-let decomposition u = UCharTbl.get decomposition_tbl u
+let decomposition_tbl = lazy (load_decomposition_tbl ())
+let decomposition u = UCharTbl.get (Lazy.force decomposition_tbl) u
 
-let composition_exclusion_tbl = load_composition_exclusion_tbl ()
-let composition_exclusion u = UCharTbl.Bool.get composition_exclusion_tbl u
+let composition_exclusion_tbl = lazy (load_composition_exclusion_tbl ())
+let composition_exclusion u = UCharTbl.Bool.get (Lazy.force composition_exclusion_tbl) u
 
-let composition_tbl = load_composition_tbl ()
-let composition u = UCharTbl.get composition_tbl u
+let composition_tbl = lazy (load_composition_tbl ())
+let composition u = UCharTbl.get (Lazy.force composition_tbl) u
 
 let rec add_list x = function
     [] -> ()

Re: [Caml-list] Batteries and portability

[Peng Zang]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Has anyone pushed this patch upstream to the camomile author(s)?

It seems like a good change that should be made in the original sources to 
make it more portable...

Peng

On Friday 12 June 2009 09:08:47 am Tiphaine Turpin wrote:
> The following completed patch works for me.
>
> Tiphaine
>
> dmitry grebeniuk a écrit :
> > If you really don't use camomile and it is not used
> > in batteries indirectly, you can defer initialization.
> > Apply the patch to camomile:
> > http://overbld.abcname.net/files/camomile-noprecomp.html
> >   But the patch has a [small] runtime cost.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFKM+bRfIRcEFL/JewRAvjYAJsG79fJRftdZCt+wiSyFqub0HJjDwCgs+VJ
zKzqnoayxrloWHtMBCzyfx4=
=qAaj
-----END PGP SIGNATURE-----

Re: [Caml-list] Batteries and portability

[dmitry grebeniuk]

2009/6/13 Peng Zang <peng.zang@gmail.com>:

> Has anyone pushed this patch upstream to the
> camomile author(s)?
>
> It seems like a good change that should be made
> in the original sources to make it more portable...

  This patch is very useful (at least for me; thanks, Tiphaine!),
but it's not performance-wise to make each camomile library
copy behave like this. With this patch, every character
that passes patched function will cause runtime penalty -- one
check for lazy value and one pointer dereference.
  The right fix is to remove the construction of module "Default"
(and its initialization, consequently) from the camomile library.
But this will break api.
  Maybe it will be better to add this patch as a default behavior,
and make a configure option for "new camomile without
Default", so that old software will use slow camomile with lazy
accesses, and new software will use camomile without module
"Default".