From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by c5ff346549e7 (Postfix) with ESMTPS id 3D1721E6 for ; Thu, 29 Mar 2018 22:45:20 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.48,378,1517871600"; d="asc'?scan'208,217";a="320628272" Received: from sympa.inria.fr ([193.51.193.213]) by mail2-relais-roc.national.inria.fr with ESMTP; 30 Mar 2018 00:45:17 +0200 Received: by sympa.inria.fr (Postfix, from userid 20132) id D3F7C8242A; Fri, 30 Mar 2018 00:45:17 +0200 (CEST) Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id AD0D08239C for ; Fri, 30 Mar 2018 00:45:12 +0200 (CEST) Authentication-Results: mail3-smtp-sop.national.inria.fr; spf=None smtp.pra=info@gerd-stolpmann.de; spf=None smtp.mailfrom=info@gerd-stolpmann.de; spf=None smtp.helo=postmaster@mout.kundenserver.de IronPort-PHdr: =?us-ascii?q?9a23=3AhI9noBBhCXqmxaGsEfs6UyQJP3N1i/DPJgcQr6Af?= =?us-ascii?q?oPdwSPX4o8bcNUDSrc9gkEXOFd2Cra4c0KyO6+jJYi8p2d65qncMcZhBBVcuqP?= =?us-ascii?q?49uEgeOvODElDxN/XwbiY3T4xoXV5h+GynYwAOQJ6tL1LdrWev4jEMBx7xKRR6?= =?us-ascii?q?JvjvGo7Vks+7y/2+94fcbglUijexe69+IAmrpgjNq8cahpdvJLwswRXTuHtIfO?= =?us-ascii?q?pWxWJsJV2Nmhv3+9m98p1+/SlOovwt78FPX7n0cKQ+VrxYES8pM3sp683xtBnM?= =?us-ascii?q?VhWA630BWWgLiBVIAgzF7BbnXpfttybxq+Rw1DWGMcDwULs5Qiqp4bt1RxD0iS?= =?us-ascii?q?cHLz85/3/Risxsl6JQvRatqwViz4LIfI2ZMfxzdb7fc9wHX2pMRttfWTJPAo28?= =?us-ascii?q?bIUBAeQOMulaoIbhvFYOogeyCBOrCu/zxDJFhHn71rA63eQ7FgHG2RQtEdMQv3?= =?us-ascii?q?TOsdr+KbocUfyyzKnM1TXDaulZ2TH76IPVdR0hoPeMXbx3ccfKxkkgCRnFgUiJ?= =?us-ascii?q?pYz+IzOV0eANs3Ce7+V6U+KvjXQrpBxvrTW2wMonl4fHhoUQyl/e9CV5xp44Jd?= =?us-ascii?q?KmR05hfNGoCYBQtz2dN4tyWsMiRntnuCA8yrEcpJG0YjAGx4g9yB7ab/yIb4mI?= =?us-ascii?q?4g7jVeqLJjd4nn1ldKqwhxaz7UigyvD8WdKu3FlWqSpFl9/BvW0O2RzL8sWLVO?= =?us-ascii?q?Zx80W71TuNyQzf8PxILEQqmabBKZMswqY8m5kTvEjZAyP7m0v7gLWXe0gg4OSk?= =?us-ascii?q?9uXqbqjgq5SBLYF7kBv+Pb4rmsGnAeQ3LAwOX2+D9OS5073j4FH5TK9Jjv0rlK?= =?us-ascii?q?nWrY7VJcIApqGnGQ9VzJgs6wylAzi7ztsUhXgHLFRbdxKbl4XlJUzCLfLiAfq9?= =?us-ascii?q?nVigiipnyvTYMrH7AJjALmDPkLL7crZ8705cxhAzzdda559MC7EBJvzzWkjqud?= =?us-ascii?q?zXEBA5KRG7z//8CNVh0YMeXXqPAqqDP6PUt1+E/P8gLPeUaI8PpDn9M+Ql5+Lp?= =?us-ascii?q?jXIhhVASZ6yp3Z8OZHC8H/RmOFmZbGH3gtYBFGcKphAxQPbriF2ESz5TZmy9U7?= =?us-ascii?q?gy5jEhW8qaCtLjT4a2yJuF0W+BF5Nfa2RdB1bERXfyfouNVvMXZSS6LcpokzhC?= =?us-ascii?q?Xr+kHctp+RiwtQOy8LFgMvacrjYTso7/ktR3z+LWnBA2szdzCpLO/XuKSjRbk3?= =?us-ascii?q?kEDwQq26RnpEV7zB/X0K52hNRXGMZfouhVVQMiMJfayap2BoahCUr6Yt6VRQP+?= =?us-ascii?q?EZ2dCjYrQ4d0modWOhcvK5CZlhnGmhGSLfoQnr2PCoYz9/uNjXf8Pco40GrB0L?= =?us-ascii?q?Ekhl8gBMdCZzX/2vxPsjPLDouMqH230r6wfP1NjiHA7mvG1naJuFldWQh2F6nI?= =?us-ascii?q?DyhGOxnm6O/h70aHdIeATLQqNgwbk5yHLbZWLN70i1lLQuzuIsrTeSS9ljXoCA?= =?us-ascii?q?=3D=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0CyAABza71ahkvASNldGgEBAQEBAgEBA?= =?us-ascii?q?QEIAQEBAYMXgQxvKINciACNAoFTIQ9mGoIghECLcYF6CAMjhGEChBoZBgYwGAE?= =?us-ascii?q?CAQEBAQEBAQEBEgEBAQgLCwgoL4I4JAGCSAEBAQMBIzIZCwULCxgnAwICISURB?= =?us-ascii?q?gEMBgIBAQWEbAMNDAEKrW2CHB+ENoIyDYEsgh8PhUSCGlSBP4EMIgyCIgYugk8?= =?us-ascii?q?3CwIBgWeCdIJUAocLF49lLAh0RgOBdoFQToVfZ4IPgTWGCCeEVIkUO4YrgSUcX?= =?us-ascii?q?oEsTSQUOoJDCYFnMA4JegEIjRVtAY9GAQE?= X-IPAS-Result: =?us-ascii?q?A0CyAABza71ahkvASNldGgEBAQEBAgEBAQEIAQEBAYMXgQx?= =?us-ascii?q?vKINciACNAoFTIQ9mGoIghECLcYF6CAMjhGEChBoZBgYwGAECAQEBAQEBAQEBE?= =?us-ascii?q?gEBAQgLCwgoL4I4JAGCSAEBAQMBIzIZCwULCxgnAwICISURBgEMBgIBAQWEbAM?= =?us-ascii?q?NDAEKrW2CHB+ENoIyDYEsgh8PhUSCGlSBP4EMIgyCIgYugk83CwIBgWeCdIJUA?= =?us-ascii?q?ocLF49lLAh0RgOBdoFQToVfZ4IPgTWGCCeEVIkUO4YrgSUcXoEsTSQUOoJDCYF?= =?us-ascii?q?nMA4JegEIjRVtAY9GAQE?= X-IronPort-AV: E=Sophos;i="5.48,378,1517871600"; d="asc'?scan'208,217";a="260337620" Received: from mout.kundenserver.de ([217.72.192.75]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES128-GCM-SHA256; 30 Mar 2018 00:45:11 +0200 Received: from office1.lan.sumadev.de ([85.183.69.161]) by mrelayeu.kundenserver.de (mreue105 [212.227.15.183]) with ESMTPSA (Nemesis) id 0MY6Rk-1f5ziX1h3I-00UofA; Fri, 30 Mar 2018 00:45:09 +0200 Received: from gerdbook.fritz.box (unknown [192.168.65.10]) by office1.lan.sumadev.de (Postfix) with ESMTPSA id 6514CDC05D; Fri, 30 Mar 2018 00:45:08 +0200 (CEST) To: Hans Ole Rafaelsen , David Scott Cc: caml-list@inria.fr References: From: Gerd Stolpmann Message-ID: Date: Fri, 30 Mar 2018 00:44:58 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="bR6EVtmQy3u9XfnJdPDXZdW5MWDJeYZoy" X-Provags-ID: V03:K0:Fyf40WkrX6VAPD8ZcYQhtNZPtOuVNf4Gq//K0qebSZHlTuTlZ7b wSKjGfT0sg7UQlgHJLOrasixVxAlVaFdUKCf0ek4CZBHccr3prthvKoVxqfsO63kL7M7pui dpzqrSkDHu6bcYfplzUIZBQiIlYxVsMltN4M1lVY59bDA+/Rel09YxVhWoXhsJQlWGkAZLt 3ZWNdqrV9gLURO52305+g== X-UI-Out-Filterresults: notjunk:1;V01:K0:xouVEAblgbY=:eMsmbMIuhMEgso98uVRq1r O0IKG4dtGMu08oZigMAcmNXoFDE6aLh4/+GEXqvkkm9+uPQzIOhWvPq1xyfYZd1k7kjIWvlvt RVHwTh+ICmP1g3RMiYlTYyLBiZmACZ0hvsf6EPK0unWqwQM9IKzZbuZT3rS2ipCY0I0S4AxNT 3v8RZg4n6WHEdGTg6AQdD82F0ezpb6+TKDP/vC4gZea+9hYbTt4fEy1Lzx3JllxoWp25tc00X /7MXHxZEzV2hIeB56xOhFaxyq2O382cdoc3cKrYGcyciPMpXfQ5CD8Qv76+HyWZgGcuXMvqyK aOBNxgH9NTxRf2qO+V1srmAfaOaUaFcxZIS3QXaGVpfYeB1sdAwt6YLW3UOcvK9pRG0x90dhP KDnY9x83/sC3IgCJUl5nvMeq4uqneR4xj/uQ4WepNT77VlTEUvTriijepzGgwpy5wdCjCoFJR WRcRQFwC/Kg32ICC1R62lfbSkJ1FHAvQjpGIpSqIFhE25FfhilHvsexsFT/fWBqmPSYJsaBMk +iGOxTCQ1tEdtsgug12e9lLQb+GxfKb4Peq4a1WqZpK6FY68DKC0aXgWH7KjM3DC8KGWlOqIg 3gKSuCKDRkX9z/7cEnkVeuuOFVG6wGBVZDwYNDdC6DQj8dvA2cU1MURJ1hRBtWyUK8Zth/MEC QsVgckXrqO16XIQNthiDoKWXo+46pA5AMV8LSoB4VRLLsJ1NbiLie9YfkQcW12a8Hfq/4cOX/ 9v5wzCI9BOVGJHiAxtlaa4vEcW6riEHrnW+Wcg== Subject: Re: [Caml-list] Create a deployable application unit from OPAM? Reply-To: Gerd Stolpmann X-Loop: caml-list@inria.fr X-Sequence: 16760 Errors-to: caml-list-owner@inria.fr Precedence: list Precedence: bulk Sender: caml-list-request@inria.fr X-no-archive: yes List-Id: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --bR6EVtmQy3u9XfnJdPDXZdW5MWDJeYZoy Content-Type: multipart/mixed; boundary="OsrDZ89vu8ULKhqRudUtPkD2iNNR9giXq"; protected-headers="v1" From: Gerd Stolpmann To: Hans Ole Rafaelsen , David Scott Cc: caml-list@inria.fr Message-ID: Subject: Re: [Caml-list] Create a deployable application unit from OPAM? References: In-Reply-To: --OsrDZ89vu8ULKhqRudUtPkD2iNNR9giXq Content-Type: multipart/alternative; boundary="------------FA6807F676D8EA98DD34BD12" Content-Language: en-US This is a multi-part message in MIME format. --------------FA6807F676D8EA98DD34BD12 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 29.03.18 20:50, Hans Ole Rafaelsen wrote: > Hi David, > > It helps a bit. But my biggest problem is that the Ocaml application > (main.exe in your example) has dependencies to other Ocaml libraries > built using OPAM. I need to find a way to find those dependencies and > make sure that the paths are resolved correctly once libraries have > have been copied to the image. As long as the files are in the findlib tree (i.e. .opam/lib), you can try to create a copy of the tree with the required files, and set a few environment variables before running your code: =C2=A0- CAML_LD_LIBRARY_PATH: set this to the stublibs directory =C2=A0- OCAMLPATH set this to the root of the tree. AFAIK ocsigen respects OCAMLPATH at runtime when dynamically loading libraries. However, I've never tried. The question is then which files you need. I guess at least: =C2=A0- the .cmi and .cmxs files from the subdirectories in .opam/lib You definitely do not need: =C2=A0- any bytecode files (.cmo, cma, .so in stublibs) =C2=A0- .cmx, .cmxa For figuring out the minimal set of ocaml packages for running, you could look into: ocamlfind query -recursive As you are mostly interested in deploying, and not necessarily relocating, another strategy might be to keep the file tree at the place where the build happens, and to just remove files you don't need. (Or just pick the right files in your Dockerfile, and ensure that the files end up at the same place in the image.) These are just some wild ideas. Ocsigen is in particular complicated because of the dynamic loading. Gerd > > --=20 > Hans Ole > > On Thu, Mar 29, 2018 at 8:13 PM, David Scott > wrote: > > Hi, > > I had a quick go at creating something with Docker: (and alpine > since I was more familiar with it, but I suspect a similar > technique would work with ubuntu) > > https://github.com/djs55/experiments/pull/1 > > > It uses `opam` to build a simple OCaml example (taken from > cryptokit) which needs `libgmp` at runtime. The `Dockerfile` has a > slightly hacky attempt to detect the runtime dependency: > > opam depext -n cryptokit | grep -v '#' | grep '\-dev' | while read > in; do basename "$in" -dev; done > needed-packages.txt > > It assumes that > > - all external C libraries have an alpine depext called foo-dev > - foo-dev has a corresponding runtime package called foo > > and installs these `foo` in an output minimal alpine image (i.e. > one without the development tools or other unnecessary stuff) > > Another approach would be to use `ldd` or `objdump` recursively to > find libraries and copy only those binaries into the output > (possibly using `FROM scratch`?) > > I don't know if this will help -- I realise my test case is a bit > minimal :-) > > Cheers, > Dave > > > On Thu, Mar 29, 2018 at 5:31 PM, Hans Ole Rafaelsen > > wrote: > > Hi, > > I have small application written using Ocsigen. Ocaml and the > rest of the development environment is installed using OPAM. > > Now I want to install this application on a few other nodes, > without having to bring bring along the whole development > process for deployment on each node. What I want to end up > with, is a single unit, e.g a tar-file, a docker container or > similar, that can be copied and deployed to the target nodes. > This unit should only contain the binaries needed to run the > application. > > I have tried to ask on the Ocsigen mailing-list how do do this > for Ocsigen applications, but without success. But I guess the > problem is more related to how to deploy applications built > using OPAM. I did not find any way to do this with OPAM, but I > might have missed something. I have tried to copy all the > binary files (.so, .cmxa .cmxs, etc) from the OPAM directory > and setting the environment variables to point the this new > location, but without much success. When starting > ocsigenserver it was not able to solve all symbols. Looking > more into the problem, it looks like binaries such as > ocsigenserver has hard coded the location of the OPAM > development environment. Might be possible to override this > using environment variables, but I'm not sure how. So I gave > up on this track. > > > Is there a way to generate such a deployable unit from OPAM > development environment? > > Do anyone have some tips / best practice on how to deploy > applications built using OPAM? > > (I'm developing and want to deploy on Ubuntu 16.04.) > > Regards, > > Hans Ole Rafaelsen > > > > > --=20 > Dave Scott > > --=20 ------------------------------------------------------------ Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de My OCaml site: http://www.camlcity.org Contact details: http://www.camlcity.org/contact.html Company homepage: http://www.gerd-stolpmann.de ------------------------------------------------------------ --------------FA6807F676D8EA98DD34BD12 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

On 29.03.18 20:50, Hans Ole Rafaelsen wrote:

Hi David,

It helps a bit. But my biggest problem is that the Ocaml application (main.exe in your example) has dependencies to other Ocaml libraries built using OPAM. I need to find a way to find those dependencies and make sure that the paths are resolved correctly once libraries have have been copied to the image.
As long as the files are in the findlib tree (i.e. .opam/lib), you can try to create a copy of the tree with the required files, and set a few environment variables before running your code:

=C2=A0- CAML_LD_LIBRARY_PATH: set this to the stublibs directory
=C2=A0- OCAMLPATH set this to the root of the tree.

AFAIK ocsigen respects OCAMLPATH at runtime when dynamically loading libraries. However, I've never tried.

The question is then which files you need. I guess at least:
=C2=A0- the .cmi and .cmxs files from the subdirectories in .opam/lib=

You definitely do not need:
=C2=A0- any bytecode files (.cmo, cma, .so in stublibs)
=C2=A0- .cmx, .cmxa

For figuring out the minimal set of ocaml packages for running, you could look into: ocamlfind query -recursive

As you are mostly interested in deploying, and not necessarily relocating, another strategy might be to keep the file tree at the place where the build happens, and to just remove files you don't need. (Or just pick the right files in your Dockerfile, and ensure that the files end up at the same place in the image.)

These are just some wild ideas. Ocsigen is in particular complicated because of the dynamic loading.

Gerd

--
Hans Ole

On Thu, Mar 29, 2018 at 8:13 PM, David= Scott <scott.dj@gmail.c= om> wrote:
Hi,

I had a quick go at creating something with Docker: (and alpine since I was more familiar with it, but I suspect a similar technique would work with ubuntu)
=


It uses `opam` to build a simple OCaml example (taken from cryptokit) which needs `libgmp` at runtime. The `Dockerfile` has a slightly hacky attempt to detect the runtime dependency:

opam depext -n cryptokit | grep -v '#' | grep '\-dev' | while read in; do basename "$in" -dev; done > needed-packages.txt

It assumes that

- all external C libraries have an alpine depext called foo-dev
- foo-dev has a corresponding runtime package called foo

and installs these `foo` in an output minimal alpine image (i.e. one without the development tools or other unnecessary stuff)

Another approach would be to use `ldd` or `objdump` recursively to find libraries and copy only those binaries into the output (possibly using `FROM scratch`?)

I don't know if this will help -- I realise my test case is a bit minimal :-)

Cheers,
Dave


On Thu, Mar 29, 2018 at 5:31= PM, Hans Ole Rafaelsen <hrafae= lsen@gmail.com> wrote:
Hi,

I have small application written using Ocsigen. Ocaml and the rest of the development environment is installed using OPAM.

Now I want to install this application on a few other nodes, without having to bring bring along the whole development process for deployment on each node. What I want to end up with, is a single unit, e.g a tar-file, a docker container or similar, that can be copied and deployed to the target nodes. This unit should only contain the binaries needed to run the application.

I have tried to ask on the Ocsigen mailing-list how do do this for Ocsigen applications, but without success. But I guess the problem is more related to how to deploy applications built using OPAM. I did not find any way to do this with OPAM, but I might have missed something. I have tried to copy all the binary files (.so, .cmxa .cmxs, etc) from the OPAM directory and setting the environment variables to point the this new location, but without much success. When starting ocsigenserver it was not able to solve all symbols. Looking more into the problem, it looks like binaries such as ocsigenserver has hard coded the location of the OPAM development environment. Might be possible to override this using environment variables, but I'm not sure how. So I gave up on this track.


Is there a way to generate such a deployable unit from OPAM development environment?

Do anyone have some tips / best practice on how to deploy applications built using OPAM?
(I'm developing and want to deploy on Ubuntu 16.04.)

Regards,

Hans Ole Rafaelsen



--
Dave Scott


--=20
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------
--------------FA6807F676D8EA98DD34BD12-- --OsrDZ89vu8ULKhqRudUtPkD2iNNR9giXq-- --bR6EVtmQy3u9XfnJdPDXZdW5MWDJeYZoy Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEDD3jJ6xcCmoYHoi4Bozhv1ksHlMFAlq9bGoACgkQBozhv1ks HlMbjwf/a7dEJDMZ0Uz4Qzx+/TN3qzWpI1rTFYZ9BHLmbgKEKtJgkJpLM6d+lpOM 4G88b3Vq09sGT327A2veBGR53D1qLjSvb7UH3ZsN3ZfjDcDM0Q+IOzsimtsI+18M zFg4BLInrNTD1rUQ1vvsNyd/pxyIoCijfQH9ezGVA4tybCJ6lfQDYmlxC5Brr2ih 80uy6u0sGTe5/Gh+NtTvoo85zXYyphfK6/UnywAI500/nrmj3rMGkmRkiCZYWELe oUgCcZPqEh9g6CHzkjicZIWLeOQEmiBOyszJ5dElpMvvcXEFB0qBJFlHrtiDTJ/O gr221XrkNSTYaJ/ybFPdenX7b7oZ8w== =s/KX -----END PGP SIGNATURE----- --bR6EVtmQy3u9XfnJdPDXZdW5MWDJeYZoy--