[Caml-list] Determining what code is depending on `Unix`

[Caml-list] Determining what code is depending on `Unix`

[Tim Cuthbertson]

[-- Attachment #1: Type: text/plain, Size: 2251 bytes --]

Hi all,

I'm attempting to convert an existing (reasonably small) HTTP server
written in OCaml to run on mirage's xen backend. I've gotten the compiler
reasonably happy by replacing uses of Unix / Lwt_unix with appropriate
mirage-compatible modules. But when it comes to link time, it seems that my
application is still indirectly depending on Unix:

+ ocamlbuild -quiet -cflag -warn-error -cflag +a-3 -build-dir
[...]/_build/mirage-xen -use-ocamlfind -lflags -g,-linkpkg,-dontlink,unix
main.native.o
+ ocamlfind ocamlopt -g -linkpkg -dontlink unix -output-obj -syntax camlp4o
-package 'fat-filesystem, io-page, mirage-block-xen.front, batteries,
yojson, safepass, sha, lwt, lwt.syntax, mirage-console.xen, mirage-types,
mirage-types.lwt, mirage-net-xen, mirage-types,cohttp,cohttp.lwt,
mirage-http, lwt, mirage.runtime,
tcpip.stack-direct,tcpip.stack-socket,mirage-clock-xen,conduit.mirage'
src/passe.cmx src/passe_server.cmx src/unikernel.cmx main.cmx -o
main.native.o
File "_none_", line 1:
Error: No implementations provided for the following modules:
         Unix referenced from
/home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_engine),
           /home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_unix),

/home/tim/.opam/mirage_4_01_0/lib/ipaddr/ipaddr_unix.cmxa(Ipaddr_unix)
Command exited with code 2.

I can't figure out why those modules are being linked in, though. I have
grepped all files in _build/mirage-xen/**/*.ml.depends (as well as my own
source code), and nothing in there matches "unix", nor mentions any of the
modules listed above. The only lwt_* matches are:

Cohttp_lwt_body
Lwt_mutex
Lwt_stream

Which all seem like they shouldn't depend on unix either directly or
indirectly.

Is there some way to see a tree / list of "what module in my source code is
transitively depending on Unix, and via what modules is that dependency
inherited?" Or even if there's no way to see a tree, is there a way to do
it one step at a time (e.g "what is depending directly on Lwt_unix", then
repeat the process for those modules listed until I see something that
surprises me)

Failing that, does anyone have tricks for weeding indirect `Unix` usage out
of a codebase which is not directly using it?

Thanks,
 - Tim Cuthbertson

[-- Attachment #2: Type: text/html, Size: 2565 bytes --]

Re: [Caml-list] Determining what code is depending on `Unix`

[Török Edwin]

On 06/13/2015 10:22 AM, Tim Cuthbertson wrote:
> Hi all,
> 
> I'm attempting to convert an existing (reasonably small) HTTP server written in OCaml to run on mirage's xen backend. I've gotten the compiler reasonably happy by replacing uses of Unix / Lwt_unix with appropriate mirage-compatible modules. But when it comes to link time, it seems that my application is still indirectly depending on Unix:
> 
> + ocamlbuild -quiet -cflag -warn-error -cflag +a-3 -build-dir [...]/_build/mirage-xen -use-ocamlfind -lflags -g,-linkpkg,-dontlink,unix main.native.o
> + ocamlfind ocamlopt -g -linkpkg -dontlink unix -output-obj -syntax camlp4o -package 'fat-filesystem, io-page, mirage-block-xen.front, batteries, yojson, safepass, sha, lwt, lwt.syntax, mirage-console.xen, mirage-types, mirage-types.lwt, mirage-net-xen, mirage-types,cohttp,cohttp.lwt, mirage-http, lwt, mirage.runtime, tcpip.stack-direct,tcpip.stack-socket,mirage-clock-xen,conduit.mirage' src/passe.cmx src/passe_server.cmx src/unikernel.cmx main.cmx -o main.native.o
> File "_none_", line 1:
> Error: No implementations provided for the following modules:
>          Unix referenced from /home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_engine),
>            /home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_unix),
>            /home/tim/.opam/mirage_4_01_0/lib/ipaddr/ipaddr_unix.cmxa(Ipaddr_unix)
> Command exited with code 2.
> 
> I can't figure out why those modules are being linked in, though. I have grepped all files in _build/mirage-xen/**/*.ml.depends (as well as my own source code), and nothing in there matches "unix", nor mentions any of the modules listed above. The only lwt_* matches are:
> 
> Cohttp_lwt_body
> Lwt_mutex
> Lwt_stream
> 
> Which all seem like they shouldn't depend on unix either directly or indirectly.
> 
> Is there some way to see a tree / list of "what module in my source code is transitively depending on Unix, and via what modules is that dependency inherited?" Or even if there's no way to see a tree, is there a way to do it one step at a time (e.g "what is depending directly on Lwt_unix", then repeat the process for those modules listed until I see something that surprises me)
> 
> Failing that, does anyone have tricks for weeding indirect `Unix` usage out of a codebase which is not directly using it?

'ocamlfind query -r' can be useful, in your case:

$ ocamlfind query -r -p-format fat-filesystem io-page mirage-block-xen.front batteries yojson safepass sha lwt lwt.syntax mirage-console.xen mirage-types mirage-types.lwt mirage-net-xen mirage-types cohttp cohttp.lwt mirage-http lwt mirage.runtime tcpip.stack-direct tcpip.stack-socket mirage-clock-xen conduit.mirage | grep unix
unix
lwt.unix
ipaddr.unix
conduit.lwt-unix
io-page.unix

Drop cohttp.lwt, tcpip.stack-direct, tcpip.stack-socket (use tcpip.xen? and you already have mirage-http and cohttp.lwt-core), and then:

$ ocamlfind query -r -p-format fat-filesystem io-page mirage-block-xen.front batteries yojson safepass sha lwt lwt.syntax mirage-console.xen mirage-types mirage-types.lwt mirage-net-xen mirage-types cohttp mirage-http lwt mirage.runtime mirage-clock-xen conduit.mirage | grep unix
unix

Best regards,
--Edwin

Re: [Caml-list] Determining what code is depending on `Unix`

[Simon Cruanes]

[-- Attachment #1: Type: text/plain, Size: 2778 bytes --]

I don't have a general answer for finding such dependencies, but in the list you have it seems there is at least batteries that depends on Unix.

Le 13 juin 2015 09:22:28 UTC+02:00, Tim Cuthbertson <tim@gfxmonk.net> a écrit :
>Hi all,
>
>I'm attempting to convert an existing (reasonably small) HTTP server
>written in OCaml to run on mirage's xen backend. I've gotten the
>compiler
>reasonably happy by replacing uses of Unix / Lwt_unix with appropriate
>mirage-compatible modules. But when it comes to link time, it seems
>that my
>application is still indirectly depending on Unix:
>
>+ ocamlbuild -quiet -cflag -warn-error -cflag +a-3 -build-dir
>[...]/_build/mirage-xen -use-ocamlfind -lflags
>-g,-linkpkg,-dontlink,unix
>main.native.o
>+ ocamlfind ocamlopt -g -linkpkg -dontlink unix -output-obj -syntax
>camlp4o
>-package 'fat-filesystem, io-page, mirage-block-xen.front, batteries,
>yojson, safepass, sha, lwt, lwt.syntax, mirage-console.xen,
>mirage-types,
>mirage-types.lwt, mirage-net-xen, mirage-types,cohttp,cohttp.lwt,
>mirage-http, lwt, mirage.runtime,
>tcpip.stack-direct,tcpip.stack-socket,mirage-clock-xen,conduit.mirage'
>src/passe.cmx src/passe_server.cmx src/unikernel.cmx main.cmx -o
>main.native.o
>File "_none_", line 1:
>Error: No implementations provided for the following modules:
>         Unix referenced from
>/home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_engine),
>         /home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_unix),
>
>/home/tim/.opam/mirage_4_01_0/lib/ipaddr/ipaddr_unix.cmxa(Ipaddr_unix)
>Command exited with code 2.
>
>I can't figure out why those modules are being linked in, though. I
>have
>grepped all files in _build/mirage-xen/**/*.ml.depends (as well as my
>own
>source code), and nothing in there matches "unix", nor mentions any of
>the
>modules listed above. The only lwt_* matches are:
>
>Cohttp_lwt_body
>Lwt_mutex
>Lwt_stream
>
>Which all seem like they shouldn't depend on unix either directly or
>indirectly.
>
>Is there some way to see a tree / list of "what module in my source
>code is
>transitively depending on Unix, and via what modules is that dependency
>inherited?" Or even if there's no way to see a tree, is there a way to
>do
>it one step at a time (e.g "what is depending directly on Lwt_unix",
>then
>repeat the process for those modules listed until I see something that
>surprises me)
>
>Failing that, does anyone have tricks for weeding indirect `Unix` usage
>out
>of a codebase which is not directly using it?
>
>Thanks,
> - Tim Cuthbertson
>
>-- 
>Caml-list mailing list.  Subscription management and archives:
>https://sympa.inria.fr/sympa/arc/caml-list
>Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>Bug reports: http://caml.inria.fr/bin/caml-bugs

-- 
Simon

[-- Attachment #2: Type: text/html, Size: 3091 bytes --]

Re: [Caml-list] Determining what code is depending on `Unix`

[Tim Cuthbertson]

On Sat, Jun 13, 2015 at 10:07 PM, Török Edwin <edwin+ml-ocaml@etorok.net> wrote:
>
> On 06/13/2015 10:22 AM, Tim Cuthbertson wrote:
> > Hi all,
> >
> > I'm attempting to convert an existing (reasonably small) HTTP server written in OCaml to run on mirage's xen backend. I've gotten the compiler reasonably happy by replacing uses of Unix / Lwt_unix with appropriate mirage-compatible modules. But when it comes to link time, it seems that my application is still indirectly depending on Unix:
> >
> > + ocamlbuild -quiet -cflag -warn-error -cflag +a-3 -build-dir [...]/_build/mirage-xen -use-ocamlfind -lflags -g,-linkpkg,-dontlink,unix main.native.o
> > + ocamlfind ocamlopt -g -linkpkg -dontlink unix -output-obj -syntax camlp4o -package 'fat-filesystem, io-page, mirage-block-xen.front, batteries, yojson, safepass, sha, lwt, lwt.syntax, mirage-console.xen, mirage-types, mirage-types.lwt, mirage-net-xen, mirage-types,cohttp,cohttp.lwt, mirage-http, lwt, mirage.runtime, tcpip.stack-direct,tcpip.stack-socket,mirage-clock-xen,conduit.mirage' src/passe.cmx src/passe_server.cmx src/unikernel.cmx main.cmx -o main.native.o
> > File "_none_", line 1:
> > Error: No implementations provided for the following modules:
> >          Unix referenced from /home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_engine),
> >            /home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_unix),
> >            /home/tim/.opam/mirage_4_01_0/lib/ipaddr/ipaddr_unix.cmxa(Ipaddr_unix)
> > Command exited with code 2.
> >
> > I can't figure out why those modules are being linked in, though. I have grepped all files in _build/mirage-xen/**/*.ml.depends (as well as my own source code), and nothing in there matches "unix", nor mentions any of the modules listed above. The only lwt_* matches are:
> >
> > Cohttp_lwt_body
> > Lwt_mutex
> > Lwt_stream
> >
> > Which all seem like they shouldn't depend on unix either directly or indirectly.
> >
> > Is there some way to see a tree / list of "what module in my source code is transitively depending on Unix, and via what modules is that dependency inherited?" Or even if there's no way to see a tree, is there a way to do it one step at a time (e.g "what is depending directly on Lwt_unix", then repeat the process for those modules listed until I see something that surprises me)
> >
> > Failing that, does anyone have tricks for weeding indirect `Unix` usage out of a codebase which is not directly using it?
>
> 'ocamlfind query -r' can be useful, in your case:
>
> $ ocamlfind query -r -p-format fat-filesystem io-page mirage-block-xen.front batteries yojson safepass sha lwt lwt.syntax mirage-console.xen mirage-types mirage-types.lwt mirage-net-xen mirage-types cohttp cohttp.lwt mirage-http lwt mirage.runtime tcpip.stack-direct tcpip.stack-socket mirage-clock-xen conduit.mirage | grep unix
> unix
> lwt.unix
> ipaddr.unix
> conduit.lwt-unix
> io-page.unix
>
> Drop cohttp.lwt, tcpip.stack-direct, tcpip.stack-socket (use tcpip.xen? and you already have mirage-http and cohttp.lwt-core), and then:
>
> $ ocamlfind query -r -p-format fat-filesystem io-page mirage-block-xen.front batteries yojson safepass sha lwt lwt.syntax mirage-console.xen mirage-types mirage-types.lwt mirage-net-xen mirage-types cohttp mirage-http lwt mirage.runtime mirage-clock-xen conduit.mirage | grep unix
> unix
>
> Best regards,
> --Edwin


Thanks for the advice, that has gotten me much further.
tcpip-stack-direct still seems necessary, but *-socket isn't. I notice
`unix` is still in the output after removing these modules (which
seems like a bad thing), and yet it successfully compiles
main.native.o now. Perhaps that is related to:



On Sat, Jun 13, 2015 at 11:06 PM, Simon Cruanes
<simon.cruanes.2007@m4x.org> wrote:
>
> I don't have a general answer for finding such dependencies, but in the list you have it seems there is at least batteries that depends on Unix.
>
>

I thought so too, but I was hoping that I can use toplevel modules
(specifically BatList and BatString), so that even though the
"batteries" package depends on Unix, those individual modules I'm
using don't. In which case the Unix dependency won't actually be
needed or referenced. But perhaps the dependencies aren't that
fine-grained?

(I should probably drop the Batteries dependency since I'm using so
little, but it's used more heavily in non-server parts of the same
codebase)

Re: [Caml-list] Determining what code is depending on `Unix`

[Simon Cruanes]

[-- Attachment #1: Type: text/plain, Size: 5313 bytes --]

If you do not use the module Batteries itself (a pack of all modules in the library, including those that depend on Unix) but instead directly use BatList, BatString you might avoid the dependency. I'm not sure though, it can be not the case if the dependency is enforced through ocamlfind or if {BatIO, BatEnum} use Unix (almost all modules depend on them).

The batteries team values retro compatibility more than being lightweight at link time, which is why it still depends on Unix.

Le 13 juin 2015 16:36:33 UTC+02:00, Tim Cuthbertson <tim@gfxmonk.net> a écrit :
>On Sat, Jun 13, 2015 at 10:07 PM, Török Edwin
><edwin+ml-ocaml@etorok.net> wrote:
>>
>> On 06/13/2015 10:22 AM, Tim Cuthbertson wrote:
>> > Hi all,
>> >
>> > I'm attempting to convert an existing (reasonably small) HTTP
>server written in OCaml to run on mirage's xen backend. I've gotten the
>compiler reasonably happy by replacing uses of Unix / Lwt_unix with
>appropriate mirage-compatible modules. But when it comes to link time,
>it seems that my application is still indirectly depending on Unix:
>> >
>> > + ocamlbuild -quiet -cflag -warn-error -cflag +a-3 -build-dir
>[...]/_build/mirage-xen -use-ocamlfind -lflags
>-g,-linkpkg,-dontlink,unix main.native.o
>> > + ocamlfind ocamlopt -g -linkpkg -dontlink unix -output-obj -syntax
>camlp4o -package 'fat-filesystem, io-page, mirage-block-xen.front,
>batteries, yojson, safepass, sha, lwt, lwt.syntax, mirage-console.xen,
>mirage-types, mirage-types.lwt, mirage-net-xen,
>mirage-types,cohttp,cohttp.lwt, mirage-http, lwt, mirage.runtime,
>tcpip.stack-direct,tcpip.stack-socket,mirage-clock-xen,conduit.mirage'
>src/passe.cmx src/passe_server.cmx src/unikernel.cmx main.cmx -o
>main.native.o
>> > File "_none_", line 1:
>> > Error: No implementations provided for the following modules:
>> >          Unix referenced from
>/home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_engine),
>> >           
>/home/tim/.opam/mirage_4_01_0/lib/lwt/lwt-unix.cmxa(Lwt_unix),
>> >           
>/home/tim/.opam/mirage_4_01_0/lib/ipaddr/ipaddr_unix.cmxa(Ipaddr_unix)
>> > Command exited with code 2.
>> >
>> > I can't figure out why those modules are being linked in, though. I
>have grepped all files in _build/mirage-xen/**/*.ml.depends (as well as
>my own source code), and nothing in there matches "unix", nor mentions
>any of the modules listed above. The only lwt_* matches are:
>> >
>> > Cohttp_lwt_body
>> > Lwt_mutex
>> > Lwt_stream
>> >
>> > Which all seem like they shouldn't depend on unix either directly
>or indirectly.
>> >
>> > Is there some way to see a tree / list of "what module in my source
>code is transitively depending on Unix, and via what modules is that
>dependency inherited?" Or even if there's no way to see a tree, is
>there a way to do it one step at a time (e.g "what is depending
>directly on Lwt_unix", then repeat the process for those modules listed
>until I see something that surprises me)
>> >
>> > Failing that, does anyone have tricks for weeding indirect `Unix`
>usage out of a codebase which is not directly using it?
>>
>> 'ocamlfind query -r' can be useful, in your case:
>>
>> $ ocamlfind query -r -p-format fat-filesystem io-page
>mirage-block-xen.front batteries yojson safepass sha lwt lwt.syntax
>mirage-console.xen mirage-types mirage-types.lwt mirage-net-xen
>mirage-types cohttp cohttp.lwt mirage-http lwt mirage.runtime
>tcpip.stack-direct tcpip.stack-socket mirage-clock-xen conduit.mirage |
>grep unix
>> unix
>> lwt.unix
>> ipaddr.unix
>> conduit.lwt-unix
>> io-page.unix
>>
>> Drop cohttp.lwt, tcpip.stack-direct, tcpip.stack-socket (use
>tcpip.xen? and you already have mirage-http and cohttp.lwt-core), and
>then:
>>
>> $ ocamlfind query -r -p-format fat-filesystem io-page
>mirage-block-xen.front batteries yojson safepass sha lwt lwt.syntax
>mirage-console.xen mirage-types mirage-types.lwt mirage-net-xen
>mirage-types cohttp mirage-http lwt mirage.runtime mirage-clock-xen
>conduit.mirage | grep unix
>> unix
>>
>> Best regards,
>> --Edwin
>
>
>Thanks for the advice, that has gotten me much further.
>tcpip-stack-direct still seems necessary, but *-socket isn't. I notice
>`unix` is still in the output after removing these modules (which
>seems like a bad thing), and yet it successfully compiles
>main.native.o now. Perhaps that is related to:
>
>
>
>On Sat, Jun 13, 2015 at 11:06 PM, Simon Cruanes
><simon.cruanes.2007@m4x.org> wrote:
>>
>> I don't have a general answer for finding such dependencies, but in
>the list you have it seems there is at least batteries that depends on
>Unix.
>>
>>
>
>I thought so too, but I was hoping that I can use toplevel modules
>(specifically BatList and BatString), so that even though the
>"batteries" package depends on Unix, those individual modules I'm
>using don't. In which case the Unix dependency won't actually be
>needed or referenced. But perhaps the dependencies aren't that
>fine-grained?
>
>(I should probably drop the Batteries dependency since I'm using so
>little, but it's used more heavily in non-server parts of the same
>codebase)
>
>-- 
>Caml-list mailing list.  Subscription management and archives:
>https://sympa.inria.fr/sympa/arc/caml-list
>Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>Bug reports: http://caml.inria.fr/bin/caml-bugs

-- 
Simon

[-- Attachment #2: Type: text/html, Size: 5903 bytes --]