[Caml-list] Utop Difficulties with C callbacks

[Caml-list] Utop Difficulties with C callbacks

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]

So, I've been working for a while with a library and I've made a good
iteration. My code compiles and is nice, it does what is expected, but it
depends on a library, Piqi, that doesn't want to play nice in the
interpreter. I want to be a good citizen and do my homework and due
diligence, but nothing I do seems to make utop want to play nice with piqi.

Here goes:

When I execute utop -require mypackage repository with the newly built
package mypackage, but I get this error:

File "_none_", line 1:
Error: The external function `camlidl_piqi_c_piqi_strtoull' is not available

I checked out piqi locally, and pinned the package to my local build, where
I tried to ensure that the appropriate .c file is getting compiled and
linked into the cma and cmxa. They are without a shadow of a doubt. I then
thought maybe I would add an include directive to utop to tell it to pick
up the mli where the the external func : type = "name" is. Nope.

I've done everything I can, from adding all combinations of additional
piqirun/piqilib/piqirun.pb/piqirun.ext combinations to the _oasis file that
specifies the serialization to trying to build my own utop to adding
additional require statements and include statements in the utop script. So
after trying to add the dependency to BuildDepends for myouterlib, I had
backed off to just adding the mylibrary to the specific binary that
consumes it, which is where my changes are (in that binary).

I even went so far as to just try and get the changes compiled into a
custom utop. Nope. I got errors and errors; I built it with:
ocamlbuild -use-ocamlfind -pkg threads -pkg utop -pkg myouterlib -pkg
camlp4 -pkg core_kernel customtop.top

(At the time I ran this, I had bap package's _oasis updated with the
appropriate compile BuildDepends so that it would link mylibrary too)

and I had the let () = Utop_main.main ();; in "utopmain.ml" and customary
customtop.mltop containing just "Utopmain". I got:

(Unbound value <everything I need *and* that's contained in mylibrary which
is specified in -pkg!>)

I don't know what else to do besides ship code that is watered down without
an interactive component. :/

[-- Attachment #2: Type: text/html, Size: 2560 bytes --]

Re: [Caml-list] Utop Difficulties with C callbacks

[Jeremie Dimino]

I have to run, but think the problem is that piqilib is not building a
shared library for its C stubs: if you want to be able to load a
library with C stubs in the toplevel, the stubs must be packed into a
shared library. You normally do this with ocamlmklib and pass a -dllib
option when building the .cma. oasis does all that automatically.

This other option is to build the bytecode executable with -custom,
which will link the C libraries, the ocamlrun executable and the
bytecode into a single native executable.

So simply adding -custom when building a custom utop should work. But
the proper fix is to update piqilib to build a .so for its C stubs.



On Thu, Mar 19, 2015 at 6:19 PM, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
> So, I've been working for a while with a library and I've made a good
> iteration. My code compiles and is nice, it does what is expected, but it
> depends on a library, Piqi, that doesn't want to play nice in the
> interpreter. I want to be a good citizen and do my homework and due
> diligence, but nothing I do seems to make utop want to play nice with piqi.
>
> Here goes:
>
> When I execute utop -require mypackage repository with the newly built
> package mypackage, but I get this error:
>
> File "_none_", line 1:
> Error: The external function `camlidl_piqi_c_piqi_strtoull' is not available
>
> I checked out piqi locally, and pinned the package to my local build, where
> I tried to ensure that the appropriate .c file is getting compiled and
> linked into the cma and cmxa. They are without a shadow of a doubt. I then
> thought maybe I would add an include directive to utop to tell it to pick up
> the mli where the the external func : type = "name" is. Nope.
>
> I've done everything I can, from adding all combinations of additional
> piqirun/piqilib/piqirun.pb/piqirun.ext combinations to the _oasis file that
> specifies the serialization to trying to build my own utop to adding
> additional require statements and include statements in the utop script. So
> after trying to add the dependency to BuildDepends for myouterlib, I had
> backed off to just adding the mylibrary to the specific binary that consumes
> it, which is where my changes are (in that binary).
>
> I even went so far as to just try and get the changes compiled into a custom
> utop. Nope. I got errors and errors; I built it with:
> ocamlbuild -use-ocamlfind -pkg threads -pkg utop -pkg myouterlib -pkg camlp4
> -pkg core_kernel customtop.top
>
> (At the time I ran this, I had bap package's _oasis updated with the
> appropriate compile BuildDepends so that it would link mylibrary too)
>
> and I had the let () = Utop_main.main ();; in "utopmain.ml" and customary
> customtop.mltop containing just "Utopmain". I got:
>
> (Unbound value <everything I need *and* that's contained in mylibrary which
> is specified in -pkg!>)
>
> I don't know what else to do besides ship code that is watered down without
> an interactive component. :/



-- 
Jeremie

Re: [Caml-list] Utop Difficulties with C callbacks

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 3722 bytes --]

Thanks! I appreciate this so much!

Gah, I skipped sleep in my determination to fix this.

I will hit it with -dllib iteration. I love oasis, and I'm an avid user.
Piqi doesn't use oasis :( sadness.

It does, however, already do the -custom approach, but only when compiling
the cma (I differentiate you said when building the executable). So I don't
know why it wouldn't work with that. In any case, I will take your advice
about the proper route. I'm all about quality and doing things right!

On Thu, Mar 19, 2015 at 3:01 PM, Jeremie Dimino <jdimino@janestreet.com>
wrote:

> I have to run, but think the problem is that piqilib is not building a
> shared library for its C stubs: if you want to be able to load a
> library with C stubs in the toplevel, the stubs must be packed into a
> shared library. You normally do this with ocamlmklib and pass a -dllib
> option when building the .cma. oasis does all that automatically.
>
> This other option is to build the bytecode executable with -custom,
> which will link the C libraries, the ocamlrun executable and the
> bytecode into a single native executable.
>
> So simply adding -custom when building a custom utop should work. But
> the proper fix is to update piqilib to build a .so for its C stubs.
>
>
>
> On Thu, Mar 19, 2015 at 6:19 PM, Kenneth Adam Miller
> <kennethadammiller@gmail.com> wrote:
> > So, I've been working for a while with a library and I've made a good
> > iteration. My code compiles and is nice, it does what is expected, but it
> > depends on a library, Piqi, that doesn't want to play nice in the
> > interpreter. I want to be a good citizen and do my homework and due
> > diligence, but nothing I do seems to make utop want to play nice with
> piqi.
> >
> > Here goes:
> >
> > When I execute utop -require mypackage repository with the newly built
> > package mypackage, but I get this error:
> >
> > File "_none_", line 1:
> > Error: The external function `camlidl_piqi_c_piqi_strtoull' is not
> available
> >
> > I checked out piqi locally, and pinned the package to my local build,
> where
> > I tried to ensure that the appropriate .c file is getting compiled and
> > linked into the cma and cmxa. They are without a shadow of a doubt. I
> then
> > thought maybe I would add an include directive to utop to tell it to
> pick up
> > the mli where the the external func : type = "name" is. Nope.
> >
> > I've done everything I can, from adding all combinations of additional
> > piqirun/piqilib/piqirun.pb/piqirun.ext combinations to the _oasis file
> that
> > specifies the serialization to trying to build my own utop to adding
> > additional require statements and include statements in the utop script.
> So
> > after trying to add the dependency to BuildDepends for myouterlib, I had
> > backed off to just adding the mylibrary to the specific binary that
> consumes
> > it, which is where my changes are (in that binary).
> >
> > I even went so far as to just try and get the changes compiled into a
> custom
> > utop. Nope. I got errors and errors; I built it with:
> > ocamlbuild -use-ocamlfind -pkg threads -pkg utop -pkg myouterlib -pkg
> camlp4
> > -pkg core_kernel customtop.top
> >
> > (At the time I ran this, I had bap package's _oasis updated with the
> > appropriate compile BuildDepends so that it would link mylibrary too)
> >
> > and I had the let () = Utop_main.main ();; in "utopmain.ml" and
> customary
> > customtop.mltop containing just "Utopmain". I got:
> >
> > (Unbound value <everything I need *and* that's contained in mylibrary
> which
> > is specified in -pkg!>)
> >
> > I don't know what else to do besides ship code that is watered down
> without
> > an interactive component. :/
>
>
>
> --
> Jeremie
>

[-- Attachment #2: Type: text/html, Size: 4710 bytes --]

Re: [Caml-list] Utop Difficulties with C callbacks

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 5363 bytes --]

Well, I gave this a shot, and got more errors. I'm going to list what I
did, and then copy the errors, and then I'm out for the day.

git clone https://github.com/alavrik/piqi.git
cd piqi/piqilib
make && make ocaml
ocamlfind ocamlc -ccopt "-fPIC -DPIC -shared -DNATIVE_CODE  -o
piqi_c_impl.so " piqi_c_impl.c
ocamlfind ocamlc -a -custom             -dllib piqilib_stubs    -o
piqilib.cma piqi_version.cmo piqi_piqirun.cmo piqi_piqi.cmo piqloc.cmo
piqi_util.cmo piq_ast.cmo piqi_impl_piqi.cmo piqi_boot.cmo piqi_c.cmo
piqi_config.cmo piqi_iolist.cmo piqi_name.cmo piqi_common.cmo piqi_file.cmo
piqi_command.cmo piqi_protobuf.cmo piqi_db.cmo piq_lexer.cmo piq_parser.cmo
piq_gen.cmo piqi_objstore.cmo piqobj.cmo piqobj_common.cmo
piqobj_to_protobuf.cmo piqobj_of_protobuf.cmo piqobj_to_piq.cmo
piqobj_of_piq.cmo piq.cmo piqi.cmo piqi_pp.cmo piqi_json_parser.cmo
piqi_json_gen.cmo piqi_json.cmo piqi_base64.cmo piqobj_to_json.cmo
piqobj_of_json.cmo piqi_xml.cmo piqobj_to_xml.cmo piqobj_of_xml.cmo
piqi_convert.cmo piqi_compile.cmo piqi_light.cmo piqi_getopt.cmo
cp piqi_c_impl.so piqilib_stubs.so
utop -require mylib

(mylib being the bap package that was compiled with oasis where a
BuildDepends: piqilib specified a need).

-----------
Cannot load required shared library dllpiqilib_stubs.
Reason: ./dllpiqilib_stubs.so: ./dllpiqilib_stubs.so: only ET_DYN and
ET_EXEC can be loaded.

Error: Reference to undefined global `Piqi_convert'

On Thu, Mar 19, 2015 at 3:33 PM, Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> Thanks! I appreciate this so much!
>
> Gah, I skipped sleep in my determination to fix this.
>
> I will hit it with -dllib iteration. I love oasis, and I'm an avid user.
> Piqi doesn't use oasis :( sadness.
>
> It does, however, already do the -custom approach, but only when compiling
> the cma (I differentiate you said when building the executable). So I don't
> know why it wouldn't work with that. In any case, I will take your advice
> about the proper route. I'm all about quality and doing things right!
>
> On Thu, Mar 19, 2015 at 3:01 PM, Jeremie Dimino <jdimino@janestreet.com>
> wrote:
>
>> I have to run, but think the problem is that piqilib is not building a
>> shared library for its C stubs: if you want to be able to load a
>> library with C stubs in the toplevel, the stubs must be packed into a
>> shared library. You normally do this with ocamlmklib and pass a -dllib
>> option when building the .cma. oasis does all that automatically.
>>
>> This other option is to build the bytecode executable with -custom,
>> which will link the C libraries, the ocamlrun executable and the
>> bytecode into a single native executable.
>>
>> So simply adding -custom when building a custom utop should work. But
>> the proper fix is to update piqilib to build a .so for its C stubs.
>>
>>
>>
>> On Thu, Mar 19, 2015 at 6:19 PM, Kenneth Adam Miller
>> <kennethadammiller@gmail.com> wrote:
>> > So, I've been working for a while with a library and I've made a good
>> > iteration. My code compiles and is nice, it does what is expected, but
>> it
>> > depends on a library, Piqi, that doesn't want to play nice in the
>> > interpreter. I want to be a good citizen and do my homework and due
>> > diligence, but nothing I do seems to make utop want to play nice with
>> piqi.
>> >
>> > Here goes:
>> >
>> > When I execute utop -require mypackage repository with the newly built
>> > package mypackage, but I get this error:
>> >
>> > File "_none_", line 1:
>> > Error: The external function `camlidl_piqi_c_piqi_strtoull' is not
>> available
>> >
>> > I checked out piqi locally, and pinned the package to my local build,
>> where
>> > I tried to ensure that the appropriate .c file is getting compiled and
>> > linked into the cma and cmxa. They are without a shadow of a doubt. I
>> then
>> > thought maybe I would add an include directive to utop to tell it to
>> pick up
>> > the mli where the the external func : type = "name" is. Nope.
>> >
>> > I've done everything I can, from adding all combinations of additional
>> > piqirun/piqilib/piqirun.pb/piqirun.ext combinations to the _oasis file
>> that
>> > specifies the serialization to trying to build my own utop to adding
>> > additional require statements and include statements in the utop
>> script. So
>> > after trying to add the dependency to BuildDepends for myouterlib, I had
>> > backed off to just adding the mylibrary to the specific binary that
>> consumes
>> > it, which is where my changes are (in that binary).
>> >
>> > I even went so far as to just try and get the changes compiled into a
>> custom
>> > utop. Nope. I got errors and errors; I built it with:
>> > ocamlbuild -use-ocamlfind -pkg threads -pkg utop -pkg myouterlib -pkg
>> camlp4
>> > -pkg core_kernel customtop.top
>> >
>> > (At the time I ran this, I had bap package's _oasis updated with the
>> > appropriate compile BuildDepends so that it would link mylibrary too)
>> >
>> > and I had the let () = Utop_main.main ();; in "utopmain.ml" and
>> customary
>> > customtop.mltop containing just "Utopmain". I got:
>> >
>> > (Unbound value <everything I need *and* that's contained in mylibrary
>> which
>> > is specified in -pkg!>)
>> >
>> > I don't know what else to do besides ship code that is watered down
>> without
>> > an interactive component. :/
>>
>>
>>
>> --
>> Jeremie
>>
>
>

[-- Attachment #2: Type: text/html, Size: 6981 bytes --]

Re: [Caml-list] Utop Difficulties with C callbacks

[Jeremie Dimino]

On Thu, Mar 19, 2015 at 9:36 PM, Kenneth Adam Miller
<kennethadammiller@gmail.com> wrote:
> ocamlfind ocamlc -ccopt "-fPIC -DPIC -shared -DNATIVE_CODE  -o
> piqi_c_impl.so " piqi_c_impl.c

I don't think this command will do what you expect. You can use
ocamlmklib instead:

  ocamlfind ocamlmklib -o piqilib_stubs piqi_c_impl.o

This will create dllpiqilib_stubs.so and libpiqilib_stubs.a. You can
use -verbose to see what it is running. You need to install the
dllpiqilib_stubs.so file with ocamlfind as well, it will put it where
ocamlrun can find it.

> ocamlfind ocamlc -a -custom             -dllib piqilib_stubs    -o
> piqilib.cma piqi_version.cmo piqi_piqirun.cmo piqi_piqi.cmo piqloc.cmo
> piqi_util.cmo piq_ast.cmo piqi_impl_piqi.cmo piqi_boot.cmo piqi_c.cmo
> piqi_config.cmo piqi_iolist.cmo piqi_name.cmo piqi_common.cmo piqi_file.cmo
> piqi_command.cmo piqi_protobuf.cmo piqi_db.cmo piq_lexer.cmo piq_parser.cmo
> piq_gen.cmo piqi_objstore.cmo piqobj.cmo piqobj_common.cmo
> piqobj_to_protobuf.cmo piqobj_of_protobuf.cmo piqobj_to_piq.cmo
> piqobj_of_piq.cmo piq.cmo piqi.cmo piqi_pp.cmo piqi_json_parser.cmo
> piqi_json_gen.cmo piqi_json.cmo piqi_base64.cmo piqobj_to_json.cmo
> piqobj_of_json.cmo piqi_xml.cmo piqobj_to_xml.cmo piqobj_of_xml.cmo
> piqi_convert.cmo piqi_compile.cmo piqi_light.cmo piqi_getopt.cmo
> cp piqi_c_impl.so piqilib_stubs.so
> utop -require mylib

AFAIK the only effect of -custom on a library is to require -custom
linking when linking an executable, so you probably don't want it
here. You should also keep the -cclib for when you do want to link a
custom executable. In the end it should look like this:

  ocamlfind ocamlc -a -cclib -lpiqilib_stubs -dllib -lpiqilib_stubs -o
piqilib.cma <cmo files>

-- 
Jeremie

Re: [Caml-list] Utop Difficulties with C callbacks

[Daniel Bünzli]

Btw. it is easy to forget that all this is quite well documented in OCaml's manual and is always worth a (re)read. See sections 19.1.[3-6]

http://caml.inria.fr/pub/docs/manual-ocaml/intfc.html#sec414

Best,

Daniel

Re: [Caml-list] Utop Difficulties with C callbacks

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 1461 bytes --]

Sorry Daniel, I should have done even more homework. I got tired after
working so hard at it. I thought I had tried everything.

But sometimes the amount of information and required reading can be
somewhat overwhelming. Somewhere between Intel PIN, SCADA systems, SANS
courses, Zyre, ZeroMQ, OCaml, CMU's BAP, and other things I'm interested
in, I have to know when it's faster to just stop and ask or to keep trying.

In any case, I notice that Google's forums is really great because they're
searchable, which means I can save other people time by searching for
things that have already been asked. I subscribed myself through that
medium rather than the caml-list that's available at the ocaml site :)

By the way -- that worked!!

I'm editing the Makefile to augment the normal build process. I should have
some merge requests out today for each of the projects :)

On Fri, Mar 20, 2015 at 5:57 AM, Daniel Bünzli <daniel.buenzli@erratique.ch>
wrote:

> Btw. it is easy to forget that all this is quite well documented in
> OCaml's manual and is always worth a (re)read. See sections 19.1.[3-6]
>
> http://caml.inria.fr/pub/docs/manual-ocaml/intfc.html#sec414
>
> Best,
>
> Daniel
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2269 bytes --]

Re: [Caml-list] Utop Difficulties with C callbacks

[Daniel Bünzli]

Le vendredi, 20 mars 2015 à 15:51, Kenneth Adam Miller a écrit :
> Sorry Daniel, I should have done even more homework.  

No problem, this was not a criticism. It's just that sometimes being insulated from the details by our build systems, meta-build systems and other ocamlfinds we forget about the underlying basic utilities and the fact that those are quite well documented.  

Best,

Daniel