From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 2FF3E7EE7A for ; Thu, 28 Mar 2013 10:19:16 +0100 (CET) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of philippe.veber@gmail.com) identity=pra; client-ip=209.85.210.172; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="philippe.veber@gmail.com"; x-sender="philippe.veber@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of philippe.veber@gmail.com designates 209.85.210.172 as permitted sender) identity=mailfrom; client-ip=209.85.210.172; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="philippe.veber@gmail.com"; x-sender="philippe.veber@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-ia0-f172.google.com) identity=helo; client-ip=209.85.210.172; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="philippe.veber@gmail.com"; x-sender="postmaster@mail-ia0-f172.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhcBAOIKVFHRVdKsm2dsb2JhbABDgzq2OgGIKnsIFg4BAQEBAQYLCwkUKIIfAQEEAScZARsQCgMBAwELBgULAwQ0IgERAQUBDg4GEwkSh2YBAwkGDKBWjDKCe4QwChknDVmIfAEFDI5mIgQHg0ADkyGDRoEfjgYWKYQvOw X-IPAS-Result: AhcBAOIKVFHRVdKsm2dsb2JhbABDgzq2OgGIKnsIFg4BAQEBAQYLCwkUKIIfAQEEAScZARsQCgMBAwELBgULAwQ0IgERAQUBDg4GEwkSh2YBAwkGDKBWjDKCe4QwChknDVmIfAEFDI5mIgQHg0ADkyGDRoEfjgYWKYQvOw X-IronPort-AV: E=Sophos;i="4.84,924,1355094000"; d="scan'208";a="10811914" Received: from mail-ia0-f172.google.com ([209.85.210.172]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/RC4-SHA; 28 Mar 2013 10:18:49 +0100 Received: by mail-ia0-f172.google.com with SMTP id l29so8178457iag.3 for ; Thu, 28 Mar 2013 02:18:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=ej807iVUXmlPIt7ZtCrulMujX0cFlWTdKq4LmkM2dZ0=; b=lSWZWPhRcEqXIPVyOxdTS5x4qRni60ykMxqo+o1j0LnN+AkUAcQD52w9xUUN3FQj2J 1WfR44k0iQHBZucwfSxWmQC4tBx43eSwtU4SQQ0KwV7nMiJuPV1jH9ipmmUnzBrVqLe4 4miutY7PNYuOXyqXvdlbNcj75mbQPGveFD0GsRbZkflGh5WPMbAw5V0sQUU6JaZ76qsj FkCZ90SmiCe5TC2KHCxNYI5gfamoAGGNJeLxMf3mnmukWroo0xLq3iGbUf8Vra3T0Rtt oo/gVhnqfzM9MX5IXSK3muWRw8w1yb7Zau/n9E2rX7L4AE7KCzFTxGDlhjWgVIRE6wer TUTg== X-Received: by 10.50.42.168 with SMTP id p8mr6816802igl.106.1364462328218; Thu, 28 Mar 2013 02:18:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.136.8 with HTTP; Thu, 28 Mar 2013 02:18:28 -0700 (PDT) In-Reply-To: <1364400454.3218.54.camel@thinkpad> References: <1364400454.3218.54.camel@thinkpad> From: Philippe Veber Date: Thu, 28 Mar 2013 10:18:28 +0100 Message-ID: To: Gerd Stolpmann Cc: caml users Content-Type: multipart/alternative; boundary=14dae934108d620e5b04d8f8a40b X-Validation-by: philippe.veber@gmail.com Subject: Re: [Caml-list] Master-slave architecture behind an ocsigen server. --14dae934108d620e5b04d8f8a40b Content-Type: text/plain; charset=ISO-8859-1 Thanks for your input Gerd! As I understand it, your suggestion is to have an RPC server (based on netplex) doing the actual calculations. That RPC server would be called by the ocsigen server when needed (the ocsigen server is the client in that scheme). So in that schema, only the RPC call should be lwt-friendly. Digging in ocamlnet documentation, it seems that could be achieved using [Rpc_simple_client.call] wrapped inside a [Lwt_preemptive.detach]. This would be more complex than using nproc, but if it is effectively dangerous/impossible to fork the ocsigen server process (as required to use nproc), that seems a feasible solution for me --> I might ask a couple more questions later then. Danke sehr! ph. 2013/3/27 Gerd Stolpmann > Am Dienstag, den 26.03.2013, 15:29 +0100 schrieb Philippe Veber: > > Dear all, > > > > I'm developping an ocsigen website doing some scientific calculations. > > Up to now, the calculations were done in the same process that runs > > the server. In order to gain in scalability (and maybe stability too), > > I would like to run those calculations in a separate (pool of) > > process(es). As this is a pretty typical setup, I guess quite a few > > people have already done that. So I'd like to hear some suggestions on > > what library to use in this particular context. It seems to me that > > the release library [1] should do the job and is lwt-friendly, but > > there are maybe other good options? > > Well, I don't know whether this is an option for Ocsigen users, but > Ocamlnet includes fairly good multiprocessing support. You can run > servers that dynamically start subprocesses on demand. Look for Netplex: > > > http://projects.camlcity.org/projects/dl/ocamlnet-3.6.3/doc/html-main/Intro.html#netplex > > I've no good recipe, though, how to plug in service processors that base > on lwt (well, there is an adaptor in Ocamlnet for lwt - Uw_lwt - but I > wouldn't know what to do on the Ocsigen side, but maybe worth > exploring). > > Ocamlnet also includes other mechanisms that are generally interesting > for compute stuff, namely Netmulticore for exploiting several cores on > the same machine with fast shared memory architecture, and RPC for > distributing computations in a network. Both are extensions of Netplex, > so it is easy to integrate into a single program. > > Gerd > > > Thanks for any hint, cheers! > > > > Philippe. > > > > [1] https://github.com/andrenth/release > > > > -- > ------------------------------------------------------------ > Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de > Creator of GODI and camlcity.org. > Contact details: http://www.camlcity.org/contact.html > Company homepage: http://www.gerd-stolpmann.de > *** Searching for new projects! Need consulting for system > *** programming in Ocaml? Gerd Stolpmann can help you. > ------------------------------------------------------------ > > --14dae934108d620e5b04d8f8a40b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks for your input Gerd! As I understand it, your sugge= stion is to have an RPC server (based on netplex) doing the actual calculat= ions. That RPC server would be called by the ocsigen server when needed (th= e ocsigen server is the client in that scheme). So in that schema, only the= RPC call should be lwt-friendly. Digging in ocamlnet documentation, it see= ms that could be achieved using [Rpc_simple_client.call] wrapped inside a [= Lwt_preemptive.detach].

This would be more complex than using nproc, but if it is effectively d= angerous/impossible to fork the ocsigen server process (as required to use = nproc), that seems a feasible solution for me --> I might ask a couple m= ore questions later then.

Danke sehr!
ph.



2013/3/27 Gerd Stolpmann <info@gerd-stolpmann.= de>
Am Dienstag, den 26.03.2013, 15:29 +0100 sch= rieb Philippe Veber:
> Dear all,
>
> I'm developping an ocsigen website doing some scientific calculati= ons.
> Up to now, the calculations were done in the same process that runs
> the server. In order to gain in scalability (and maybe stability too),=
> I would like to run those calculations in a separate (pool of)
> process(es). As this is a pretty typical setup, I guess quite a few
> people have already done that. So I'd like to hear some suggestion= s on
> what library to use in this particular context. It seems to me that
> the release library [1] should do the job and is lwt-friendly, but
> there are maybe other good options?

Well, I don't know whether this is an option for Ocsigen users, b= ut
Ocamlnet includes fairly good multiprocessing support. You can run
servers that dynamically start subprocesses on demand. Look for Netplex:

http://projects.camlcity.org/pr= ojects/dl/ocamlnet-3.6.3/doc/html-main/Intro.html#netplex

I've no good recipe, though, how to plug in service processors that bas= e
on lwt (well, there is an adaptor in Ocamlnet for lwt - Uw_lwt - but I
wouldn't know what to do on the Ocsigen side, but maybe worth
exploring).

Ocamlnet also includes other mechanisms that are generally interesting
for compute stuff, namely Netmulticore for exploiting several cores on
the same machine with fast shared memory architecture, and RPC for
distributing computations in a network. Both are extensions of Netplex,
so it is easy to integrate into a single program.

Gerd

> Thanks for any hint, cheers!
>
> Philippe.
>
> [1] = https://github.com/andrenth/release
>

--
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany =A0 =A0gerd@gerd-stolpmann.de
Creator of GODI and camlc= ity.org.
Contact details: =A0 =A0 =A0 =A0http://www.camlcity.org/contact.html
Company homepage: =A0 =A0 =A0 http://www.gerd-stolpmann.de
*** Searching for new projects! Need consulting for system
*** programming in Ocaml? Gerd Stolpmann can help you.
------------------------------------------------------------


--14dae934108d620e5b04d8f8a40b--