[Caml-list] Question about Lwt/Async

[Caml-list] Question about Lwt/Async

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 1490 bytes --]

Hi all

I'm thinking about my next project in OCaml, and I'm wondering how many
users of OCaml currently use Lwt or Async regularly.

One of the advantages of OCaml over Haskell (which I'm not crazy about) is
the fact that you don't have to constantly be stuck inside a monad.
However, once you want to use these user-level threading libraries, you're
essentially tied to a monad. It also means that the usage of any other
monad from Lwt/Async code is out -- OCaml doesn't have the monad
transformer infrastructure to layer monads easily as far as I can tell (am
I wrong?). I mean, even in Haskell using Monad Transformers is a pain (IMO).

Also, what happens to general utility functions that aren't rewritten for
Async/Lwt -- as far as I can tell, being in non-monadic code, they will
always starve other threads, since they cannot yield to another Async/Lwt
thread. Is this perception correct? If so, this seems to imply that you
either write your code to cooperate within these frameworks and suffer the
monad, or don't, and make it near-impossible for Lwt/Async users to make
use of your code.

I would like to get an idea of the usage level of these libraries, as well
as the burden of writing compatible code, any difficulties etc. Also, I'd
like to get a sense of the domains that benefit from these libraries. Some
domains (such as gaming) traditionally involve a continuous main loop, and
would thus only suffer from the additional overhead of queuing in these
libraries.

-Yotam

[-- Attachment #2: Type: text/html, Size: 1668 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Malcolm Matalka]

Yotam Barnoy <yotambarnoy@gmail.com> writes:

> Hi all
>
> I'm thinking about my next project in OCaml, and I'm wondering how many
> users of OCaml currently use Lwt or Async regularly.
>
> One of the advantages of OCaml over Haskell (which I'm not crazy about) is
> the fact that you don't have to constantly be stuck inside a monad.
> However, once you want to use these user-level threading libraries, you're
> essentially tied to a monad. It also means that the usage of any other
> monad from Lwt/Async code is out -- OCaml doesn't have the monad
> transformer infrastructure to layer monads easily as far as I can tell (am
> I wrong?). I mean, even in Haskell using Monad Transformers is a pain (IMO).
>
> Also, what happens to general utility functions that aren't rewritten for
> Async/Lwt -- as far as I can tell, being in non-monadic code, they will
> always starve other threads, since they cannot yield to another Async/Lwt
> thread. Is this perception correct? If so, this seems to imply that you
> either write your code to cooperate within these frameworks and suffer the
> monad, or don't, and make it near-impossible for Lwt/Async users to make
> use of your code.
>
> I would like to get an idea of the usage level of these libraries, as well
> as the burden of writing compatible code, any difficulties etc. Also, I'd
> like to get a sense of the domains that benefit from these libraries. Some
> domains (such as gaming) traditionally involve a continuous main loop, and
> would thus only suffer from the additional overhead of queuing in these
> libraries.
>
> -Yotam

I mostly use Async.  However, I think most usage of Lwt or Async
requires doing as little as possible in these frameworks and using them
to orchestrate other functions.  For example, I usually try to separate
parsing of a network protocol from the reading and writing of the bytes.

Re: [Caml-list] Question about Lwt/Async

[Simon Cruanes]

[-- Attachment #1: Type: text/plain, Size: 3130 bytes --]

Hi,

It seems that Lwt is more popular in the community
outside JaneStreet than Async (if only by looking at its reverse
dependencies on opam.ocaml.org). There has been posts about this, for
instance http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .
However, if you're writing a library, it is good taste (if possible) to
parametrize you code over an "IO" monad that will be easy to instantiate
with either Async or Lwt (or the trivial blocking monad where 'a t = 'a
and (>>=) x f = f x) along with the required IO primitives.

Regarding general utility functions, if they do not perform IO or depend
on (blocking) IO they can be used directly with Async/Lwt (unless they
really take a very long time to complete).

Le Mon, 07 Mar 2016, Malcolm Matalka a écrit :
> Yotam Barnoy <yotambarnoy@gmail.com> writes:
> > Hi all
> >
> > I'm thinking about my next project in OCaml, and I'm wondering how many
> > users of OCaml currently use Lwt or Async regularly.
> >
> > One of the advantages of OCaml over Haskell (which I'm not crazy about) is
> > the fact that you don't have to constantly be stuck inside a monad.
> > However, once you want to use these user-level threading libraries, you're
> > essentially tied to a monad. It also means that the usage of any other
> > monad from Lwt/Async code is out -- OCaml doesn't have the monad
> > transformer infrastructure to layer monads easily as far as I can tell (am
> > I wrong?). I mean, even in Haskell using Monad Transformers is a pain (IMO).
> >
> > Also, what happens to general utility functions that aren't rewritten for
> > Async/Lwt -- as far as I can tell, being in non-monadic code, they will
> > always starve other threads, since they cannot yield to another Async/Lwt
> > thread. Is this perception correct? If so, this seems to imply that you
> > either write your code to cooperate within these frameworks and suffer the
> > monad, or don't, and make it near-impossible for Lwt/Async users to make
> > use of your code.
> >
> > I would like to get an idea of the usage level of these libraries, as well
> > as the burden of writing compatible code, any difficulties etc. Also, I'd
> > like to get a sense of the domains that benefit from these libraries. Some
> > domains (such as gaming) traditionally involve a continuous main loop, and
> > would thus only suffer from the additional overhead of queuing in these
> > libraries.
> >
> > -Yotam
> 
> I mostly use Async.  However, I think most usage of Lwt or Async
> requires doing as little as possible in these frameworks and using them
> to orchestrate other functions.  For example, I usually try to separate
> parsing of a network protocol from the reading and writing of the bytes.
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs


-- 
Simon Cruanes

http://weusepgp.info/
key 49AA62B6, fingerprint 949F EB87 8F06 59C6 D7D3  7D8D 4AC0 1D08 49AA 62B6

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 3532 bytes --]

Is there an existing compatibility library functorized over the
intersection of Async and Lwt? That would make being compatible with both
much easier.

On Mon, Mar 7, 2016 at 4:08 AM, Simon Cruanes <simon.cruanes.2007@m4x.org>
wrote:

> Hi,
>
> It seems that Lwt is more popular in the community
> outside JaneStreet than Async (if only by looking at its reverse
> dependencies on opam.ocaml.org). There has been posts about this, for
> instance http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .
> However, if you're writing a library, it is good taste (if possible) to
> parametrize you code over an "IO" monad that will be easy to instantiate
> with either Async or Lwt (or the trivial blocking monad where 'a t = 'a
> and (>>=) x f = f x) along with the required IO primitives.
>
> Regarding general utility functions, if they do not perform IO or depend
> on (blocking) IO they can be used directly with Async/Lwt (unless they
> really take a very long time to complete).
>
> Le Mon, 07 Mar 2016, Malcolm Matalka a écrit :
> > Yotam Barnoy <yotambarnoy@gmail.com> writes:
> > > Hi all
> > >
> > > I'm thinking about my next project in OCaml, and I'm wondering how many
> > > users of OCaml currently use Lwt or Async regularly.
> > >
> > > One of the advantages of OCaml over Haskell (which I'm not crazy
> about) is
> > > the fact that you don't have to constantly be stuck inside a monad.
> > > However, once you want to use these user-level threading libraries,
> you're
> > > essentially tied to a monad. It also means that the usage of any other
> > > monad from Lwt/Async code is out -- OCaml doesn't have the monad
> > > transformer infrastructure to layer monads easily as far as I can tell
> (am
> > > I wrong?). I mean, even in Haskell using Monad Transformers is a pain
> (IMO).
> > >
> > > Also, what happens to general utility functions that aren't rewritten
> for
> > > Async/Lwt -- as far as I can tell, being in non-monadic code, they will
> > > always starve other threads, since they cannot yield to another
> Async/Lwt
> > > thread. Is this perception correct? If so, this seems to imply that you
> > > either write your code to cooperate within these frameworks and suffer
> the
> > > monad, or don't, and make it near-impossible for Lwt/Async users to
> make
> > > use of your code.
> > >
> > > I would like to get an idea of the usage level of these libraries, as
> well
> > > as the burden of writing compatible code, any difficulties etc. Also,
> I'd
> > > like to get a sense of the domains that benefit from these libraries.
> Some
> > > domains (such as gaming) traditionally involve a continuous main loop,
> and
> > > would thus only suffer from the additional overhead of queuing in these
> > > libraries.
> > >
> > > -Yotam
> >
> > I mostly use Async.  However, I think most usage of Lwt or Async
> > requires doing as little as possible in these frameworks and using them
> > to orchestrate other functions.  For example, I usually try to separate
> > parsing of a network protocol from the reading and writing of the bytes.
> >
> > --
> > Caml-list mailing list.  Subscription management and archives:
> > https://sympa.inria.fr/sympa/arc/caml-list
> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
> --
> Simon Cruanes
>
> http://weusepgp.info/
> key 49AA62B6, fingerprint 949F EB87 8F06 59C6 D7D3  7D8D 4AC0 1D08 49AA
> 62B6
>

[-- Attachment #2: Type: text/html, Size: 4859 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Ashish Agarwal]

[-- Attachment #1: Type: text/plain, Size: 5141 bytes --]

> Also, what happens to general utility functions that aren't rewritten for
Async/Lwt -- as far as I can tell, being in non-monadic code, they will
always starve other threads, since they cannot yield to another Async/Lwt
thread.

There is Lwt_preemptive.detach and Async's In_thread.run to get around this.


> It seems that Lwt is more popular in the community outside JaneStreet
than Async (if only by looking at its reverse dependencies on opam.ocaml.org).
There has been posts about this, for instance
http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .

I'd be wary of drawing conclusions from one blog post and even from opam. I
think the answer is: both are used a lot. Also depends on what you mean by
"a user". It's not too useful to count Jane Street's packages and one
barely used package on opam both as 1. A lot of code is not on opam.


> Is there an existing compatibility library functorized over the
intersection of Async and Lwt? That would make being compatible with both
much easier.

Most people provide this internally for each of their projects, e.g. Cohttp's
IO signature <https://github.com/mirage/ocaml-cohttp/blob/master/lib/s.mli>.
However, we have quite a few projects that needed this abstraction, so
duplicating this code in each repo seemed wrong. Thus we developed future
<https://github.com/solvuu/future>, which was recently released in opam.



On Mon, Mar 7, 2016 at 9:06 AM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:

> Is there an existing compatibility library functorized over the
> intersection of Async and Lwt? That would make being compatible with both
> much easier.
>
> On Mon, Mar 7, 2016 at 4:08 AM, Simon Cruanes <simon.cruanes.2007@m4x.org>
> wrote:
>
>> Hi,
>>
>> It seems that Lwt is more popular in the community
>> outside JaneStreet than Async (if only by looking at its reverse
>> dependencies on opam.ocaml.org). There has been posts about this, for
>> instance http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .
>> However, if you're writing a library, it is good taste (if possible) to
>> parametrize you code over an "IO" monad that will be easy to instantiate
>> with either Async or Lwt (or the trivial blocking monad where 'a t = 'a
>> and (>>=) x f = f x) along with the required IO primitives.
>>
>> Regarding general utility functions, if they do not perform IO or depend
>> on (blocking) IO they can be used directly with Async/Lwt (unless they
>> really take a very long time to complete).
>>
>> Le Mon, 07 Mar 2016, Malcolm Matalka a écrit :
>> > Yotam Barnoy <yotambarnoy@gmail.com> writes:
>> > > Hi all
>> > >
>> > > I'm thinking about my next project in OCaml, and I'm wondering how
>> many
>> > > users of OCaml currently use Lwt or Async regularly.
>> > >
>> > > One of the advantages of OCaml over Haskell (which I'm not crazy
>> about) is
>> > > the fact that you don't have to constantly be stuck inside a monad.
>> > > However, once you want to use these user-level threading libraries,
>> you're
>> > > essentially tied to a monad. It also means that the usage of any other
>> > > monad from Lwt/Async code is out -- OCaml doesn't have the monad
>> > > transformer infrastructure to layer monads easily as far as I can
>> tell (am
>> > > I wrong?). I mean, even in Haskell using Monad Transformers is a pain
>> (IMO).
>> > >
>> > > Also, what happens to general utility functions that aren't rewritten
>> for
>> > > Async/Lwt -- as far as I can tell, being in non-monadic code, they
>> will
>> > > always starve other threads, since they cannot yield to another
>> Async/Lwt
>> > > thread. Is this perception correct? If so, this seems to imply that
>> you
>> > > either write your code to cooperate within these frameworks and
>> suffer the
>> > > monad, or don't, and make it near-impossible for Lwt/Async users to
>> make
>> > > use of your code.
>> > >
>> > > I would like to get an idea of the usage level of these libraries, as
>> well
>> > > as the burden of writing compatible code, any difficulties etc. Also,
>> I'd
>> > > like to get a sense of the domains that benefit from these libraries.
>> Some
>> > > domains (such as gaming) traditionally involve a continuous main
>> loop, and
>> > > would thus only suffer from the additional overhead of queuing in
>> these
>> > > libraries.
>> > >
>> > > -Yotam
>> >
>> > I mostly use Async.  However, I think most usage of Lwt or Async
>> > requires doing as little as possible in these frameworks and using them
>> > to orchestrate other functions.  For example, I usually try to separate
>> > parsing of a network protocol from the reading and writing of the bytes.
>> >
>> > --
>> > Caml-list mailing list.  Subscription management and archives:
>> > https://sympa.inria.fr/sympa/arc/caml-list
>> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>>
>> --
>> Simon Cruanes
>>
>> http://weusepgp.info/
>> key 49AA62B6, fingerprint 949F EB87 8F06 59C6 D7D3  7D8D 4AC0 1D08 49AA
>> 62B6
>>
>
>

[-- Attachment #2: Type: text/html, Size: 7193 bytes --]

Re: [Caml-list] Question about Lwt/Async

[rudi.grinberg]

[-- Attachment #1: Type: text/plain, Size: 5324 bytes --]

Since my post was mentioned, I thought I’d chime in.

I’ve used both libraries and I’ve found little practical difference between the two. I think porting a codebase from Lwt to Async (and vice versa) is mostly mechanical work.

While I still prefer Async’s interface a little more but I think the two main points in my blog post still stand. If portability and maximum interoperability with the community are important to you then the decision is already made in my eyes.

On March 7, 2016 at 9:26:41 AM, Ashish Agarwal (agarwal1975@gmail.com) wrote:

> Also, what happens to general utility functions that aren't rewritten for Async/Lwt -- as far as I can tell, being in non-monadic code, they will always starve other threads, since they cannot yield to another Async/Lwt thread.

There is Lwt_preemptive.detach and Async's In_thread.run to get around this.


> It seems that Lwt is more popular in the community outside JaneStreet than Async (if only by looking at its reverse dependencies on opam.ocaml.org). There has been posts about this, for instance http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .

I'd be wary of drawing conclusions from one blog post and even from opam. I think the answer is: both are used a lot. Also depends on what you mean by "a user". It's not too useful to count Jane Street's packages and one barely used package on opam both as 1. A lot of code is not on opam.


> Is there an existing compatibility library functorized over the intersection of Async and Lwt? That would make being compatible with both much easier.

Most people provide this internally for each of their projects, e.g. Cohttp's IO signature. However, we have quite a few projects that needed this abstraction, so duplicating this code in each repo seemed wrong. Thus we developed future, which was recently released in opam.



On Mon, Mar 7, 2016 at 9:06 AM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
Is there an existing compatibility library functorized over the intersection of Async and Lwt? That would make being compatible with both much easier.

On Mon, Mar 7, 2016 at 4:08 AM, Simon Cruanes <simon.cruanes.2007@m4x.org> wrote:
Hi,

It seems that Lwt is more popular in the community
outside JaneStreet than Async (if only by looking at its reverse
dependencies on opam.ocaml.org). There has been posts about this, for
instance http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .
However, if you're writing a library, it is good taste (if possible) to
parametrize you code over an "IO" monad that will be easy to instantiate
with either Async or Lwt (or the trivial blocking monad where 'a t = 'a
and (>>=) x f = f x) along with the required IO primitives.

Regarding general utility functions, if they do not perform IO or depend
on (blocking) IO they can be used directly with Async/Lwt (unless they
really take a very long time to complete).

Le Mon, 07 Mar 2016, Malcolm Matalka a écrit :
> Yotam Barnoy <yotambarnoy@gmail.com> writes:
> > Hi all
> >
> > I'm thinking about my next project in OCaml, and I'm wondering how many
> > users of OCaml currently use Lwt or Async regularly.
> >
> > One of the advantages of OCaml over Haskell (which I'm not crazy about) is
> > the fact that you don't have to constantly be stuck inside a monad.
> > However, once you want to use these user-level threading libraries, you're
> > essentially tied to a monad. It also means that the usage of any other
> > monad from Lwt/Async code is out -- OCaml doesn't have the monad
> > transformer infrastructure to layer monads easily as far as I can tell (am
> > I wrong?). I mean, even in Haskell using Monad Transformers is a pain (IMO).
> >
> > Also, what happens to general utility functions that aren't rewritten for
> > Async/Lwt -- as far as I can tell, being in non-monadic code, they will
> > always starve other threads, since they cannot yield to another Async/Lwt
> > thread. Is this perception correct? If so, this seems to imply that you
> > either write your code to cooperate within these frameworks and suffer the
> > monad, or don't, and make it near-impossible for Lwt/Async users to make
> > use of your code.
> >
> > I would like to get an idea of the usage level of these libraries, as well
> > as the burden of writing compatible code, any difficulties etc. Also, I'd
> > like to get a sense of the domains that benefit from these libraries. Some
> > domains (such as gaming) traditionally involve a continuous main loop, and
> > would thus only suffer from the additional overhead of queuing in these
> > libraries.
> >
> > -Yotam
>
> I mostly use Async.  However, I think most usage of Lwt or Async
> requires doing as little as possible in these frameworks and using them
> to orchestrate other functions.  For example, I usually try to separate
> parsing of a network protocol from the reading and writing of the bytes.
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs


--
Simon Cruanes

http://weusepgp.info/
key 49AA62B6, fingerprint 949F EB87 8F06 59C6 D7D3  7D8D 4AC0 1D08 49AA 62B6



[-- Attachment #2: Type: text/html, Size: 8986 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 6257 bytes --]

> While I still prefer Async’s interface

Me too, that's why we created an [overlay][1] over Lwt, that provides an
interface in the style of Core library.
I'm currently working on releasing this library to opam.

[1]: https://github.com/BinaryAnalysisPlatform/core-lwt

On Mon, Mar 7, 2016 at 9:55 AM, <rudi.grinberg@gmail.com> wrote:

> Since my post was mentioned, I thought I’d chime in.
>
> I’ve used both libraries and I’ve found little practical difference
> between the two. I think porting a codebase from Lwt to Async (and vice
> versa) is mostly mechanical work.
>
> While I still prefer Async’s interface a little more but I think the two
> main points in my blog post still stand. If portability and maximum
> interoperability with the community are important to you then the decision
> is already made in my eyes.
>
> On March 7, 2016 at 9:26:41 AM, Ashish Agarwal (agarwal1975@gmail.com)
> wrote:
>
> > Also, what happens to general utility functions that aren't rewritten
> for Async/Lwt -- as far as I can tell, being in non-monadic code, they will
> always starve other threads, since they cannot yield to another Async/Lwt
> thread.
>
> There is Lwt_preemptive.detach and Async's In_thread.run to get around
> this.
>
>
> > It seems that Lwt is more popular in the community outside JaneStreet
> than Async (if only by looking at its reverse dependencies on
> opam.ocaml.org). There has been posts about this, for instance
> http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .
>
> I'd be wary of drawing conclusions from one blog post and even from opam.
> I think the answer is: both are used a lot. Also depends on what you mean
> by "a user". It's not too useful to count Jane Street's packages and one
> barely used package on opam both as 1. A lot of code is not on opam.
>
>
> > Is there an existing compatibility library functorized over the
> intersection of Async and Lwt? That would make being compatible with both
> much easier.
>
> Most people provide this internally for each of their projects, e.g. Cohttp's
> IO signature
> <https://github.com/mirage/ocaml-cohttp/blob/master/lib/s.mli>. However,
> we have quite a few projects that needed this abstraction, so duplicating
> this code in each repo seemed wrong. Thus we developed future
> <https://github.com/solvuu/future>, which was recently released in opam.
>
>
>
> On Mon, Mar 7, 2016 at 9:06 AM, Yotam Barnoy <yotambarnoy@gmail.com>
> wrote:
>
>> Is there an existing compatibility library functorized over the
>> intersection of Async and Lwt? That would make being compatible with both
>> much easier.
>>
>> On Mon, Mar 7, 2016 at 4:08 AM, Simon Cruanes <simon.cruanes.2007@m4x.org
>> > wrote:
>>
>>> Hi,
>>>
>>> It seems that Lwt is more popular in the community
>>> outside JaneStreet than Async (if only by looking at its reverse
>>> dependencies on opam.ocaml.org). There has been posts about this, for
>>> instance http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .
>>> However, if you're writing a library, it is good taste (if possible) to
>>> parametrize you code over an "IO" monad that will be easy to instantiate
>>> with either Async or Lwt (or the trivial blocking monad where 'a t = 'a
>>> and (>>=) x f = f x) along with the required IO primitives.
>>>
>>> Regarding general utility functions, if they do not perform IO or depend
>>> on (blocking) IO they can be used directly with Async/Lwt (unless they
>>> really take a very long time to complete).
>>>
>>> Le Mon, 07 Mar 2016, Malcolm Matalka a écrit :
>>> > Yotam Barnoy <yotambarnoy@gmail.com> writes:
>>> > > Hi all
>>> > >
>>> > > I'm thinking about my next project in OCaml, and I'm wondering how
>>> many
>>> > > users of OCaml currently use Lwt or Async regularly.
>>> > >
>>> > > One of the advantages of OCaml over Haskell (which I'm not crazy
>>> about) is
>>> > > the fact that you don't have to constantly be stuck inside a monad.
>>> > > However, once you want to use these user-level threading libraries,
>>> you're
>>> > > essentially tied to a monad. It also means that the usage of any
>>> other
>>> > > monad from Lwt/Async code is out -- OCaml doesn't have the monad
>>> > > transformer infrastructure to layer monads easily as far as I can
>>> tell (am
>>> > > I wrong?). I mean, even in Haskell using Monad Transformers is a
>>> pain (IMO).
>>> > >
>>> > > Also, what happens to general utility functions that aren't
>>> rewritten for
>>> > > Async/Lwt -- as far as I can tell, being in non-monadic code, they
>>> will
>>> > > always starve other threads, since they cannot yield to another
>>> Async/Lwt
>>> > > thread. Is this perception correct? If so, this seems to imply that
>>> you
>>> > > either write your code to cooperate within these frameworks and
>>> suffer the
>>> > > monad, or don't, and make it near-impossible for Lwt/Async users to
>>> make
>>> > > use of your code.
>>> > >
>>> > > I would like to get an idea of the usage level of these libraries,
>>> as well
>>> > > as the burden of writing compatible code, any difficulties etc.
>>> Also, I'd
>>> > > like to get a sense of the domains that benefit from these
>>> libraries. Some
>>> > > domains (such as gaming) traditionally involve a continuous main
>>> loop, and
>>> > > would thus only suffer from the additional overhead of queuing in
>>> these
>>> > > libraries.
>>> > >
>>> > > -Yotam
>>> >
>>> > I mostly use Async.  However, I think most usage of Lwt or Async
>>> > requires doing as little as possible in these frameworks and using them
>>> > to orchestrate other functions.  For example, I usually try to separate
>>> > parsing of a network protocol from the reading and writing of the
>>> bytes.
>>> >
>>> > --
>>> > Caml-list mailing list.  Subscription management and archives:
>>> > https://sympa.inria.fr/sympa/arc/caml-list
>>> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>
>>>
>>> --
>>> Simon Cruanes
>>>
>>> http://weusepgp.info/
>>> key 49AA62B6, fingerprint 949F EB87 8F06 59C6 D7D3  7D8D 4AC0 1D08 49AA
>>> 62B6
>>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 9525 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 7011 bytes --]

We actually have another work, that heavily intersects with  `Lwt` and
`Async`. It is code named `bap-future`[1], and in short this
is a library to work with asynchronous values. This can be seen, as `Lwt.t`
or `Ivar.t` cleaned up of everything unnecessary.
In the ideal future, the `Future.t` can be used as a common denominator
between `Lwt` and `Async`.

[1]:
https://github.com/BinaryAnalysisPlatform/bap/blob/master/lib/bap_future/bap_future.mli

P.S. It will be also released very soon.

On Mon, Mar 7, 2016 at 9:59 AM, Ivan Gotovchits <ivg@ieee.org> wrote:

> > While I still prefer Async’s interface
>
> Me too, that's why we created an [overlay][1] over Lwt, that provides an
> interface in the style of Core library.
> I'm currently working on releasing this library to opam.
>
> [1]: https://github.com/BinaryAnalysisPlatform/core-lwt
>
> On Mon, Mar 7, 2016 at 9:55 AM, <rudi.grinberg@gmail.com> wrote:
>
>> Since my post was mentioned, I thought I’d chime in.
>>
>> I’ve used both libraries and I’ve found little practical difference
>> between the two. I think porting a codebase from Lwt to Async (and vice
>> versa) is mostly mechanical work.
>>
>> While I still prefer Async’s interface a little more but I think the two
>> main points in my blog post still stand. If portability and maximum
>> interoperability with the community are important to you then the decision
>> is already made in my eyes.
>>
>> On March 7, 2016 at 9:26:41 AM, Ashish Agarwal (agarwal1975@gmail.com)
>> wrote:
>>
>> > Also, what happens to general utility functions that aren't rewritten
>> for Async/Lwt -- as far as I can tell, being in non-monadic code, they will
>> always starve other threads, since they cannot yield to another Async/Lwt
>> thread.
>>
>> There is Lwt_preemptive.detach and Async's In_thread.run to get around
>> this.
>>
>>
>> > It seems that Lwt is more popular in the community outside JaneStreet
>> than Async (if only by looking at its reverse dependencies on
>> opam.ocaml.org). There has been posts about this, for instance
>> http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .
>>
>> I'd be wary of drawing conclusions from one blog post and even from opam.
>> I think the answer is: both are used a lot. Also depends on what you mean
>> by "a user". It's not too useful to count Jane Street's packages and one
>> barely used package on opam both as 1. A lot of code is not on opam.
>>
>>
>> > Is there an existing compatibility library functorized over the
>> intersection of Async and Lwt? That would make being compatible with both
>> much easier.
>>
>> Most people provide this internally for each of their projects, e.g. Cohttp's
>> IO signature
>> <https://github.com/mirage/ocaml-cohttp/blob/master/lib/s.mli>. However,
>> we have quite a few projects that needed this abstraction, so duplicating
>> this code in each repo seemed wrong. Thus we developed future
>> <https://github.com/solvuu/future>, which was recently released in opam.
>>
>>
>>
>> On Mon, Mar 7, 2016 at 9:06 AM, Yotam Barnoy <yotambarnoy@gmail.com>
>> wrote:
>>
>>> Is there an existing compatibility library functorized over the
>>> intersection of Async and Lwt? That would make being compatible with both
>>> much easier.
>>>
>>> On Mon, Mar 7, 2016 at 4:08 AM, Simon Cruanes <
>>> simon.cruanes.2007@m4x.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> It seems that Lwt is more popular in the community
>>>> outside JaneStreet than Async (if only by looking at its reverse
>>>> dependencies on opam.ocaml.org). There has been posts about this, for
>>>> instance http://rgrinberg.com/blog/2014/12/11/abandoning-async/ .
>>>> However, if you're writing a library, it is good taste (if possible) to
>>>> parametrize you code over an "IO" monad that will be easy to instantiate
>>>> with either Async or Lwt (or the trivial blocking monad where 'a t = 'a
>>>> and (>>=) x f = f x) along with the required IO primitives.
>>>>
>>>> Regarding general utility functions, if they do not perform IO or depend
>>>> on (blocking) IO they can be used directly with Async/Lwt (unless they
>>>> really take a very long time to complete).
>>>>
>>>> Le Mon, 07 Mar 2016, Malcolm Matalka a écrit :
>>>> > Yotam Barnoy <yotambarnoy@gmail.com> writes:
>>>> > > Hi all
>>>> > >
>>>> > > I'm thinking about my next project in OCaml, and I'm wondering how
>>>> many
>>>> > > users of OCaml currently use Lwt or Async regularly.
>>>> > >
>>>> > > One of the advantages of OCaml over Haskell (which I'm not crazy
>>>> about) is
>>>> > > the fact that you don't have to constantly be stuck inside a monad.
>>>> > > However, once you want to use these user-level threading libraries,
>>>> you're
>>>> > > essentially tied to a monad. It also means that the usage of any
>>>> other
>>>> > > monad from Lwt/Async code is out -- OCaml doesn't have the monad
>>>> > > transformer infrastructure to layer monads easily as far as I can
>>>> tell (am
>>>> > > I wrong?). I mean, even in Haskell using Monad Transformers is a
>>>> pain (IMO).
>>>> > >
>>>> > > Also, what happens to general utility functions that aren't
>>>> rewritten for
>>>> > > Async/Lwt -- as far as I can tell, being in non-monadic code, they
>>>> will
>>>> > > always starve other threads, since they cannot yield to another
>>>> Async/Lwt
>>>> > > thread. Is this perception correct? If so, this seems to imply that
>>>> you
>>>> > > either write your code to cooperate within these frameworks and
>>>> suffer the
>>>> > > monad, or don't, and make it near-impossible for Lwt/Async users to
>>>> make
>>>> > > use of your code.
>>>> > >
>>>> > > I would like to get an idea of the usage level of these libraries,
>>>> as well
>>>> > > as the burden of writing compatible code, any difficulties etc.
>>>> Also, I'd
>>>> > > like to get a sense of the domains that benefit from these
>>>> libraries. Some
>>>> > > domains (such as gaming) traditionally involve a continuous main
>>>> loop, and
>>>> > > would thus only suffer from the additional overhead of queuing in
>>>> these
>>>> > > libraries.
>>>> > >
>>>> > > -Yotam
>>>> >
>>>> > I mostly use Async.  However, I think most usage of Lwt or Async
>>>> > requires doing as little as possible in these frameworks and using
>>>> them
>>>> > to orchestrate other functions.  For example, I usually try to
>>>> separate
>>>> > parsing of a network protocol from the reading and writing of the
>>>> bytes.
>>>> >
>>>> > --
>>>> > Caml-list mailing list.  Subscription management and archives:
>>>> > https://sympa.inria.fr/sympa/arc/caml-list
>>>> > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>> > Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>>
>>>>
>>>> --
>>>> Simon Cruanes
>>>>
>>>> http://weusepgp.info/
>>>> key 49AA62B6, fingerprint 949F EB87 8F06 59C6 D7D3  7D8D 4AC0 1D08 49AA
>>>> 62B6
>>>>
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 10631 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Jesper Louis Andersen]

[-- Attachment #1: Type: text/plain, Size: 2550 bytes --]

On Mon, Mar 7, 2016 at 2:38 AM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:

> Also, what happens to general utility functions that aren't rewritten for
> Async/Lwt -- as far as I can tell, being in non-monadic code, they will
> always starve other threads, since they cannot yield to another Async/Lwt
> thread. Is this perception correct?


Yes.

On one hand, your observation is negative in the sense that now your code
has "color" in the sense that it is written for one library only. And you
have to transform code to having the right color before it can be used.
This is not the case if the concurrency model is at a lower level[0].

On the other hand, your observation is positive: cooperative scheduling
makes the points in which the code can switch explicit. This gives the
programmer far more control over when you are done with a task and start to
process the next task. You can also avoid the preemption check in the code
all the time. If your code manipulates lots of shared data, it also
simplifies things since you don't usually have to protect data with a mutex
in a single-threaded context as much[1]. Cooperative models, if carefully
managed, can exploit structure in the problem domain, whereas a preemptive
model needs to fit all.

My personal opinion is that the preemptive model eventually wins over the
cooperative model, much like it has in most (all popular) operating
systems. It is simply more productive to take an up-front performance hit
as a sacrifice for a system which is more robust against stray code
misbehaving. If a cooperative system fails, it is fails catastrophically.
If a preemptive system fails, it degrades in performance.

But given I have more than 10 years of Erlang programming behind me by now,
I'm obviously biased toward certain computational models :)

[0] Erlang would be one such example, where the system is preemptively
scheduling for you and you can use any code in any place without having to
worry about blocking for latency. Go is quasi-preemptive because it checks
on function calls, but in contrast to Erlang a loop is not forced to factor
through a recursion, so it can in principle run indefinitely. Haskell (GHC)
is quasi-preemptive as well, checking on memory allocation boundaries. So
the thing to look out for in GHC is latency from processing large arrays
with no allocation, say.

[1] Erlang has two VM runtimes for this reason. One is single-threaded and
can avoid lots of locks which is far faster for certain workloads, or on
embedded devices with a single core only.

-- 
J.

[-- Attachment #2: Type: text/html, Size: 3262 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Yaron Minsky]

This is definitely a fraught topic, and it's unfortunate that there's
no clear solution.

To add a bit more information:

- Async is more portable than it once was.  There's now Core_kernel,
  Async_kernel and Async_rpc_kernel, which allows us to do things like
  run Async applications in the browser.  I would think Windows
  support would be pretty doable by someone who understands that world
  well.

  That said, the chain of dependencies brought in by Async is still
  quite big.  This is something that could perhaps be improved, either
  with better dead code analysis in OCaml, or some tweaks to
  Async_kernel and Core_kernel themselves.

- There are things we could contemplate to make it easier to bridge
  the divide.  Jeremie Dimino did a proof of concept rewrite of lwt to
  use async as its implementation, where an Lwt.t and a Deferred.t are
  equal at the type level.

    https://github.com/janestreet/lwt-async

  Another possibility, and one that might be easier to write, would be
  to allow Lwt code to run using the Async scheduler as another
  possible back-end.  This would allow one to have programs that used
  both Async and Lwt together in one program, without running on
  different threads.

It's worth mentioning if that there is interest in making Async more
suitable for a wider variety of goals, we're happy to work with
outside contributors on it.  For example, if someone wanted to work on
Windows support for Async, we'd be happy to help out on integrating
that work.

Probably the biggest issue is executable size.  That will get better
when we release an unpacked version of our external libraries.  But
even then, the module-level granularity captures more things than
would be ideal.

y

On Mon, Mar 7, 2016 at 10:16 AM, Jesper Louis Andersen
<jesper.louis.andersen@gmail.com> wrote:
>
> On Mon, Mar 7, 2016 at 2:38 AM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
>>
>> Also, what happens to general utility functions that aren't rewritten for
>> Async/Lwt -- as far as I can tell, being in non-monadic code, they will
>> always starve other threads, since they cannot yield to another Async/Lwt
>> thread. Is this perception correct?
>
>
> Yes.
>
> On one hand, your observation is negative in the sense that now your code
> has "color" in the sense that it is written for one library only. And you
> have to transform code to having the right color before it can be used. This
> is not the case if the concurrency model is at a lower level[0].
>
> On the other hand, your observation is positive: cooperative scheduling
> makes the points in which the code can switch explicit. This gives the
> programmer far more control over when you are done with a task and start to
> process the next task. You can also avoid the preemption check in the code
> all the time. If your code manipulates lots of shared data, it also
> simplifies things since you don't usually have to protect data with a mutex
> in a single-threaded context as much[1]. Cooperative models, if carefully
> managed, can exploit structure in the problem domain, whereas a preemptive
> model needs to fit all.
>
> My personal opinion is that the preemptive model eventually wins over the
> cooperative model, much like it has in most (all popular) operating systems.
> It is simply more productive to take an up-front performance hit as a
> sacrifice for a system which is more robust against stray code misbehaving.
> If a cooperative system fails, it is fails catastrophically. If a preemptive
> system fails, it degrades in performance.
>
> But given I have more than 10 years of Erlang programming behind me by now,
> I'm obviously biased toward certain computational models :)
>
> [0] Erlang would be one such example, where the system is preemptively
> scheduling for you and you can use any code in any place without having to
> worry about blocking for latency. Go is quasi-preemptive because it checks
> on function calls, but in contrast to Erlang a loop is not forced to factor
> through a recursion, so it can in principle run indefinitely. Haskell (GHC)
> is quasi-preemptive as well, checking on memory allocation boundaries. So
> the thing to look out for in GHC is latency from processing large arrays
> with no allocation, say.
>
> [1] Erlang has two VM runtimes for this reason. One is single-threaded and
> can avoid lots of locks which is far faster for certain workloads, or on
> embedded devices with a single core only.
>
> --
> J.

Re: [Caml-list] Question about Lwt/Async

[Malcolm Matalka]

Yaron Minsky <yminsky@janestreet.com> writes:

> This is definitely a fraught topic, and it's unfortunate that there's
> no clear solution.
>
> To add a bit more information:
>
> - Async is more portable than it once was.  There's now Core_kernel,
>   Async_kernel and Async_rpc_kernel, which allows us to do things like
>   run Async applications in the browser.  I would think Windows
>   support would be pretty doable by someone who understands that world
>   well.
>
>   That said, the chain of dependencies brought in by Async is still
>   quite big.  This is something that could perhaps be improved, either
>   with better dead code analysis in OCaml, or some tweaks to
>   Async_kernel and Core_kernel themselves.

When I last looked at the scheduler it was limited to [select] or
[epoll], is this still the case?  How difficult would it be to expand on
those?

>
> - There are things we could contemplate to make it easier to bridge
>   the divide.  Jeremie Dimino did a proof of concept rewrite of lwt to
>   use async as its implementation, where an Lwt.t and a Deferred.t are
>   equal at the type level.
>
>     https://github.com/janestreet/lwt-async
>
>   Another possibility, and one that might be easier to write, would be
>   to allow Lwt code to run using the Async scheduler as another
>   possible back-end.  This would allow one to have programs that used
>   both Async and Lwt together in one program, without running on
>   different threads.
>
> It's worth mentioning if that there is interest in making Async more
> suitable for a wider variety of goals, we're happy to work with
> outside contributors on it.  For example, if someone wanted to work on
> Windows support for Async, we'd be happy to help out on integrating
> that work.
>
> Probably the biggest issue is executable size.  That will get better
> when we release an unpacked version of our external libraries.  But
> even then, the module-level granularity captures more things than
> would be ideal.
>
> y
>
> On Mon, Mar 7, 2016 at 10:16 AM, Jesper Louis Andersen
> <jesper.louis.andersen@gmail.com> wrote:
>>
>> On Mon, Mar 7, 2016 at 2:38 AM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
>>>
>>> Also, what happens to general utility functions that aren't rewritten for
>>> Async/Lwt -- as far as I can tell, being in non-monadic code, they will
>>> always starve other threads, since they cannot yield to another Async/Lwt
>>> thread. Is this perception correct?
>>
>>
>> Yes.
>>
>> On one hand, your observation is negative in the sense that now your code
>> has "color" in the sense that it is written for one library only. And you
>> have to transform code to having the right color before it can be used. This
>> is not the case if the concurrency model is at a lower level[0].
>>
>> On the other hand, your observation is positive: cooperative scheduling
>> makes the points in which the code can switch explicit. This gives the
>> programmer far more control over when you are done with a task and start to
>> process the next task. You can also avoid the preemption check in the code
>> all the time. If your code manipulates lots of shared data, it also
>> simplifies things since you don't usually have to protect data with a mutex
>> in a single-threaded context as much[1]. Cooperative models, if carefully
>> managed, can exploit structure in the problem domain, whereas a preemptive
>> model needs to fit all.
>>
>> My personal opinion is that the preemptive model eventually wins over the
>> cooperative model, much like it has in most (all popular) operating systems.
>> It is simply more productive to take an up-front performance hit as a
>> sacrifice for a system which is more robust against stray code misbehaving.
>> If a cooperative system fails, it is fails catastrophically. If a preemptive
>> system fails, it degrades in performance.
>>
>> But given I have more than 10 years of Erlang programming behind me by now,
>> I'm obviously biased toward certain computational models :)
>>
>> [0] Erlang would be one such example, where the system is preemptively
>> scheduling for you and you can use any code in any place without having to
>> worry about blocking for latency. Go is quasi-preemptive because it checks
>> on function calls, but in contrast to Erlang a loop is not forced to factor
>> through a recursion, so it can in principle run indefinitely. Haskell (GHC)
>> is quasi-preemptive as well, checking on memory allocation boundaries. So
>> the thing to look out for in GHC is latency from processing large arrays
>> with no allocation, say.
>>
>> [1] Erlang has two VM runtimes for this reason. One is single-threaded and
>> can avoid lots of locks which is far faster for certain workloads, or on
>> embedded devices with a single core only.
>>
>> --
>> J.

Re: [Caml-list] Question about Lwt/Async

[Yaron Minsky]

Right now, only select and epoll are supported, but adding support for
something else isn't hard.  The Async_unix library has an interface
called File_descr_watcher_intf.S, which both select and epoll go
through.  Adding support for another shouldn't be difficult if someone
with the right OS expertise wants to do it.

Is there a particular kernel API you want support for?

y

On Mon, Mar 7, 2016 at 1:16 PM, Malcolm Matalka <mmatalka@gmail.com> wrote:
> Yaron Minsky <yminsky@janestreet.com> writes:
>
>> This is definitely a fraught topic, and it's unfortunate that there's
>> no clear solution.
>>
>> To add a bit more information:
>>
>> - Async is more portable than it once was.  There's now Core_kernel,
>>   Async_kernel and Async_rpc_kernel, which allows us to do things like
>>   run Async applications in the browser.  I would think Windows
>>   support would be pretty doable by someone who understands that world
>>   well.
>>
>>   That said, the chain of dependencies brought in by Async is still
>>   quite big.  This is something that could perhaps be improved, either
>>   with better dead code analysis in OCaml, or some tweaks to
>>   Async_kernel and Core_kernel themselves.
>
> When I last looked at the scheduler it was limited to [select] or
> [epoll], is this still the case?  How difficult would it be to expand on
> those?
>
>>
>> - There are things we could contemplate to make it easier to bridge
>>   the divide.  Jeremie Dimino did a proof of concept rewrite of lwt to
>>   use async as its implementation, where an Lwt.t and a Deferred.t are
>>   equal at the type level.
>>
>>     https://github.com/janestreet/lwt-async
>>
>>   Another possibility, and one that might be easier to write, would be
>>   to allow Lwt code to run using the Async scheduler as another
>>   possible back-end.  This would allow one to have programs that used
>>   both Async and Lwt together in one program, without running on
>>   different threads.
>>
>> It's worth mentioning if that there is interest in making Async more
>> suitable for a wider variety of goals, we're happy to work with
>> outside contributors on it.  For example, if someone wanted to work on
>> Windows support for Async, we'd be happy to help out on integrating
>> that work.
>>
>> Probably the biggest issue is executable size.  That will get better
>> when we release an unpacked version of our external libraries.  But
>> even then, the module-level granularity captures more things than
>> would be ideal.
>>
>> y
>>
>> On Mon, Mar 7, 2016 at 10:16 AM, Jesper Louis Andersen
>> <jesper.louis.andersen@gmail.com> wrote:
>>>
>>> On Mon, Mar 7, 2016 at 2:38 AM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
>>>>
>>>> Also, what happens to general utility functions that aren't rewritten for
>>>> Async/Lwt -- as far as I can tell, being in non-monadic code, they will
>>>> always starve other threads, since they cannot yield to another Async/Lwt
>>>> thread. Is this perception correct?
>>>
>>>
>>> Yes.
>>>
>>> On one hand, your observation is negative in the sense that now your code
>>> has "color" in the sense that it is written for one library only. And you
>>> have to transform code to having the right color before it can be used. This
>>> is not the case if the concurrency model is at a lower level[0].
>>>
>>> On the other hand, your observation is positive: cooperative scheduling
>>> makes the points in which the code can switch explicit. This gives the
>>> programmer far more control over when you are done with a task and start to
>>> process the next task. You can also avoid the preemption check in the code
>>> all the time. If your code manipulates lots of shared data, it also
>>> simplifies things since you don't usually have to protect data with a mutex
>>> in a single-threaded context as much[1]. Cooperative models, if carefully
>>> managed, can exploit structure in the problem domain, whereas a preemptive
>>> model needs to fit all.
>>>
>>> My personal opinion is that the preemptive model eventually wins over the
>>> cooperative model, much like it has in most (all popular) operating systems.
>>> It is simply more productive to take an up-front performance hit as a
>>> sacrifice for a system which is more robust against stray code misbehaving.
>>> If a cooperative system fails, it is fails catastrophically. If a preemptive
>>> system fails, it degrades in performance.
>>>
>>> But given I have more than 10 years of Erlang programming behind me by now,
>>> I'm obviously biased toward certain computational models :)
>>>
>>> [0] Erlang would be one such example, where the system is preemptively
>>> scheduling for you and you can use any code in any place without having to
>>> worry about blocking for latency. Go is quasi-preemptive because it checks
>>> on function calls, but in contrast to Erlang a loop is not forced to factor
>>> through a recursion, so it can in principle run indefinitely. Haskell (GHC)
>>> is quasi-preemptive as well, checking on memory allocation boundaries. So
>>> the thing to look out for in GHC is latency from processing large arrays
>>> with no allocation, say.
>>>
>>> [1] Erlang has two VM runtimes for this reason. One is single-threaded and
>>> can avoid lots of locks which is far faster for certain workloads, or on
>>> embedded devices with a single core only.
>>>
>>> --
>>> J.

Re: [Caml-list] Question about Lwt/Async

[Malcolm Matalka]

Yaron Minsky <yminsky@janestreet.com> writes:

> Right now, only select and epoll are supported, but adding support for
> something else isn't hard.  The Async_unix library has an interface
> called File_descr_watcher_intf.S, which both select and epoll go
> through.  Adding support for another shouldn't be difficult if someone
> with the right OS expertise wants to do it.
>
> Is there a particular kernel API you want support for?

kqueue, I run most things on FreeBSD and select is sadly mostly useless
for anything serious.  I've played with the idea of adding kqueue
support myself but haven't had the time.

>
> y
>
> On Mon, Mar 7, 2016 at 1:16 PM, Malcolm Matalka <mmatalka@gmail.com> wrote:
>> Yaron Minsky <yminsky@janestreet.com> writes:
>>
>>> This is definitely a fraught topic, and it's unfortunate that there's
>>> no clear solution.
>>>
>>> To add a bit more information:
>>>
>>> - Async is more portable than it once was.  There's now Core_kernel,
>>>   Async_kernel and Async_rpc_kernel, which allows us to do things like
>>>   run Async applications in the browser.  I would think Windows
>>>   support would be pretty doable by someone who understands that world
>>>   well.
>>>
>>>   That said, the chain of dependencies brought in by Async is still
>>>   quite big.  This is something that could perhaps be improved, either
>>>   with better dead code analysis in OCaml, or some tweaks to
>>>   Async_kernel and Core_kernel themselves.
>>
>> When I last looked at the scheduler it was limited to [select] or
>> [epoll], is this still the case?  How difficult would it be to expand on
>> those?
>>
>>>
>>> - There are things we could contemplate to make it easier to bridge
>>>   the divide.  Jeremie Dimino did a proof of concept rewrite of lwt to
>>>   use async as its implementation, where an Lwt.t and a Deferred.t are
>>>   equal at the type level.
>>>
>>>     https://github.com/janestreet/lwt-async
>>>
>>>   Another possibility, and one that might be easier to write, would be
>>>   to allow Lwt code to run using the Async scheduler as another
>>>   possible back-end.  This would allow one to have programs that used
>>>   both Async and Lwt together in one program, without running on
>>>   different threads.
>>>
>>> It's worth mentioning if that there is interest in making Async more
>>> suitable for a wider variety of goals, we're happy to work with
>>> outside contributors on it.  For example, if someone wanted to work on
>>> Windows support for Async, we'd be happy to help out on integrating
>>> that work.
>>>
>>> Probably the biggest issue is executable size.  That will get better
>>> when we release an unpacked version of our external libraries.  But
>>> even then, the module-level granularity captures more things than
>>> would be ideal.
>>>
>>> y
>>>
>>> On Mon, Mar 7, 2016 at 10:16 AM, Jesper Louis Andersen
>>> <jesper.louis.andersen@gmail.com> wrote:
>>>>
>>>> On Mon, Mar 7, 2016 at 2:38 AM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:
>>>>>
>>>>> Also, what happens to general utility functions that aren't rewritten for
>>>>> Async/Lwt -- as far as I can tell, being in non-monadic code, they will
>>>>> always starve other threads, since they cannot yield to another Async/Lwt
>>>>> thread. Is this perception correct?
>>>>
>>>>
>>>> Yes.
>>>>
>>>> On one hand, your observation is negative in the sense that now your code
>>>> has "color" in the sense that it is written for one library only. And you
>>>> have to transform code to having the right color before it can be used. This
>>>> is not the case if the concurrency model is at a lower level[0].
>>>>
>>>> On the other hand, your observation is positive: cooperative scheduling
>>>> makes the points in which the code can switch explicit. This gives the
>>>> programmer far more control over when you are done with a task and start to
>>>> process the next task. You can also avoid the preemption check in the code
>>>> all the time. If your code manipulates lots of shared data, it also
>>>> simplifies things since you don't usually have to protect data with a mutex
>>>> in a single-threaded context as much[1]. Cooperative models, if carefully
>>>> managed, can exploit structure in the problem domain, whereas a preemptive
>>>> model needs to fit all.
>>>>
>>>> My personal opinion is that the preemptive model eventually wins over the
>>>> cooperative model, much like it has in most (all popular) operating systems.
>>>> It is simply more productive to take an up-front performance hit as a
>>>> sacrifice for a system which is more robust against stray code misbehaving.
>>>> If a cooperative system fails, it is fails catastrophically. If a preemptive
>>>> system fails, it degrades in performance.
>>>>
>>>> But given I have more than 10 years of Erlang programming behind me by now,
>>>> I'm obviously biased toward certain computational models :)
>>>>
>>>> [0] Erlang would be one such example, where the system is preemptively
>>>> scheduling for you and you can use any code in any place without having to
>>>> worry about blocking for latency. Go is quasi-preemptive because it checks
>>>> on function calls, but in contrast to Erlang a loop is not forced to factor
>>>> through a recursion, so it can in principle run indefinitely. Haskell (GHC)
>>>> is quasi-preemptive as well, checking on memory allocation boundaries. So
>>>> the thing to look out for in GHC is latency from processing large arrays
>>>> with no allocation, say.
>>>>
>>>> [1] Erlang has two VM runtimes for this reason. One is single-threaded and
>>>> can avoid lots of locks which is far faster for certain workloads, or on
>>>> embedded devices with a single core only.
>>>>
>>>> --
>>>> J.

Re: [Caml-list] Question about Lwt/Async

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 6142 bytes --]

Out of curiosity, what polling mechanism is available on the lwt side?

On Mon, Mar 7, 2016 at 3:06 PM, Malcolm Matalka <mmatalka@gmail.com> wrote:

> Yaron Minsky <yminsky@janestreet.com> writes:
>
> > Right now, only select and epoll are supported, but adding support for
> > something else isn't hard.  The Async_unix library has an interface
> > called File_descr_watcher_intf.S, which both select and epoll go
> > through.  Adding support for another shouldn't be difficult if someone
> > with the right OS expertise wants to do it.
> >
> > Is there a particular kernel API you want support for?
>
> kqueue, I run most things on FreeBSD and select is sadly mostly useless
> for anything serious.  I've played with the idea of adding kqueue
> support myself but haven't had the time.
>
> >
> > y
> >
> > On Mon, Mar 7, 2016 at 1:16 PM, Malcolm Matalka <mmatalka@gmail.com>
> wrote:
> >> Yaron Minsky <yminsky@janestreet.com> writes:
> >>
> >>> This is definitely a fraught topic, and it's unfortunate that there's
> >>> no clear solution.
> >>>
> >>> To add a bit more information:
> >>>
> >>> - Async is more portable than it once was.  There's now Core_kernel,
> >>>   Async_kernel and Async_rpc_kernel, which allows us to do things like
> >>>   run Async applications in the browser.  I would think Windows
> >>>   support would be pretty doable by someone who understands that world
> >>>   well.
> >>>
> >>>   That said, the chain of dependencies brought in by Async is still
> >>>   quite big.  This is something that could perhaps be improved, either
> >>>   with better dead code analysis in OCaml, or some tweaks to
> >>>   Async_kernel and Core_kernel themselves.
> >>
> >> When I last looked at the scheduler it was limited to [select] or
> >> [epoll], is this still the case?  How difficult would it be to expand on
> >> those?
> >>
> >>>
> >>> - There are things we could contemplate to make it easier to bridge
> >>>   the divide.  Jeremie Dimino did a proof of concept rewrite of lwt to
> >>>   use async as its implementation, where an Lwt.t and a Deferred.t are
> >>>   equal at the type level.
> >>>
> >>>     https://github.com/janestreet/lwt-async
> >>>
> >>>   Another possibility, and one that might be easier to write, would be
> >>>   to allow Lwt code to run using the Async scheduler as another
> >>>   possible back-end.  This would allow one to have programs that used
> >>>   both Async and Lwt together in one program, without running on
> >>>   different threads.
> >>>
> >>> It's worth mentioning if that there is interest in making Async more
> >>> suitable for a wider variety of goals, we're happy to work with
> >>> outside contributors on it.  For example, if someone wanted to work on
> >>> Windows support for Async, we'd be happy to help out on integrating
> >>> that work.
> >>>
> >>> Probably the biggest issue is executable size.  That will get better
> >>> when we release an unpacked version of our external libraries.  But
> >>> even then, the module-level granularity captures more things than
> >>> would be ideal.
> >>>
> >>> y
> >>>
> >>> On Mon, Mar 7, 2016 at 10:16 AM, Jesper Louis Andersen
> >>> <jesper.louis.andersen@gmail.com> wrote:
> >>>>
> >>>> On Mon, Mar 7, 2016 at 2:38 AM, Yotam Barnoy <yotambarnoy@gmail.com>
> wrote:
> >>>>>
> >>>>> Also, what happens to general utility functions that aren't
> rewritten for
> >>>>> Async/Lwt -- as far as I can tell, being in non-monadic code, they
> will
> >>>>> always starve other threads, since they cannot yield to another
> Async/Lwt
> >>>>> thread. Is this perception correct?
> >>>>
> >>>>
> >>>> Yes.
> >>>>
> >>>> On one hand, your observation is negative in the sense that now your
> code
> >>>> has "color" in the sense that it is written for one library only. And
> you
> >>>> have to transform code to having the right color before it can be
> used. This
> >>>> is not the case if the concurrency model is at a lower level[0].
> >>>>
> >>>> On the other hand, your observation is positive: cooperative
> scheduling
> >>>> makes the points in which the code can switch explicit. This gives the
> >>>> programmer far more control over when you are done with a task and
> start to
> >>>> process the next task. You can also avoid the preemption check in the
> code
> >>>> all the time. If your code manipulates lots of shared data, it also
> >>>> simplifies things since you don't usually have to protect data with a
> mutex
> >>>> in a single-threaded context as much[1]. Cooperative models, if
> carefully
> >>>> managed, can exploit structure in the problem domain, whereas a
> preemptive
> >>>> model needs to fit all.
> >>>>
> >>>> My personal opinion is that the preemptive model eventually wins over
> the
> >>>> cooperative model, much like it has in most (all popular) operating
> systems.
> >>>> It is simply more productive to take an up-front performance hit as a
> >>>> sacrifice for a system which is more robust against stray code
> misbehaving.
> >>>> If a cooperative system fails, it is fails catastrophically. If a
> preemptive
> >>>> system fails, it degrades in performance.
> >>>>
> >>>> But given I have more than 10 years of Erlang programming behind me
> by now,
> >>>> I'm obviously biased toward certain computational models :)
> >>>>
> >>>> [0] Erlang would be one such example, where the system is preemptively
> >>>> scheduling for you and you can use any code in any place without
> having to
> >>>> worry about blocking for latency. Go is quasi-preemptive because it
> checks
> >>>> on function calls, but in contrast to Erlang a loop is not forced to
> factor
> >>>> through a recursion, so it can in principle run indefinitely. Haskell
> (GHC)
> >>>> is quasi-preemptive as well, checking on memory allocation
> boundaries. So
> >>>> the thing to look out for in GHC is latency from processing large
> arrays
> >>>> with no allocation, say.
> >>>>
> >>>> [1] Erlang has two VM runtimes for this reason. One is
> single-threaded and
> >>>> can avoid lots of locks which is far faster for certain workloads, or
> on
> >>>> embedded devices with a single core only.
> >>>>
> >>>> --
> >>>> J.
>

[-- Attachment #2: Type: text/html, Size: 8360 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Milan Stanojević]

> One of the advantages of OCaml over Haskell (which I'm not crazy about) is
> the fact that you don't have to constantly be stuck inside a monad. However,
> once you want to use these user-level threading libraries, you're
> essentially tied to a monad. It also means that the usage of any other monad
> from Lwt/Async code is out -- OCaml doesn't have the monad transformer
> infrastructure to layer monads easily as far as I can tell (am I wrong?). I
> mean, even in Haskell using Monad Transformers is a pain (IMO).

Are there any particular monads you have in mind?

My experience is that it's not a big problem in practice with Async (I
don't have experience with lwt but I guess it is similar) but I don't
really use a lot of monads (mostly just Deferred and Or_error)

With async, a good programming style is to keep asynchronous code at
the fringes of your application and make as much of your program's
logic synchronous. This is good for performance but more importantly
makes your code easier to reason about because you don't have to worry
about any interleaving and races since async runs code without binds
and maps as an atomic job.

This means that large parts of your code are just usual synchronous
ocaml code (you might not even open Async in a module) and you are
free to easily use any monad you want.

Of course, there are cases when you will run into issues, usually
combining Or_error and Deferred monads. Common case is when handling
rpcs in async. There indeed you are stuck for a bit in
Deferrred.Or_error monad but at some point you need to handle two
cases (Error and Ok) and you escape it.

Re: [Caml-list] Question about Lwt/Async

[Milan Stanojević]

> Most people provide this internally for each of their projects, e.g.
> Cohttp's IO signature. However, we have quite a few projects that needed
> this abstraction, so duplicating this code in each repo seemed wrong. Thus
> we developed future, which was recently released in opam.

I didn't check opam but I guess there are libraries that are just
Async or (more likely) just Lwt. If I want to use them both I'm out of
luck, right?

I'd rather try to do something like Yaron said and have Lwt run using
Async backend or vice versa. Both are probably possible. Converting
between Lwt.t and Deferred.t in that case is trivial.

This would allow us then to mix Async and Lwt libraries in the same
application, and programmers would be free to choose to have their
interface be Async or Lwt,  or both if abstracting layer works
for them.

Of course, devil is in the details. I hope that this is indeed
possible and that changes necessary to Lwt and Async to make this work
are not huge.

Re: [Caml-list] Question about Lwt/Async

[Malcolm Matalka]

Yotam Barnoy <yotambarnoy@gmail.com> writes:

> Out of curiosity, what polling mechanism is available on the lwt side?

I believe Lwt supports multiple event loops underneath, including libuv
to get whatever polling the OS supports.

Re: [Caml-list] Question about Lwt/Async

[Adrien Nader]

Unless that has changed recently, it's using libev, not libev. But the
crux remains the same: it relies on a C library that provides a
compatibility layer across event loops. I see little reason to not do
the same for Core rather than trying to handle properly each and every
mechanism around. The reason to also (still) support select() is that
it's everywhere (except that on Windows it's not really usable but there
you know you have libev or libuv).

-- 
Adrien Nader

Re: [Caml-list] Question about Lwt/Async

[Francois Berenger]

On 03/07/2016 06:03 PM, Yaron Minsky wrote:
> This is definitely a fraught topic, and it's unfortunate that there's
> no clear solution.
>
> To add a bit more information:
>
> - Async is more portable than it once was.  There's now Core_kernel,
>    Async_kernel and Async_rpc_kernel, which allows us to do things like
>    run Async applications in the browser.  I would think Windows
>    support would be pretty doable by someone who understands that world
>    well.

Just for information: with the ZeroMQ bindings that are in opam
(package zmq), it is possible to do synchronous and/or asynchronous 
distributed programming in OCaml.
That's not exactly RPC, but with some translation layer you can do some 
kind of RPC.
However, unlike async, zmq is not a pure OCaml library so
that's not as portable. You need the right version of the ZeroMQ
C++ library to be installed.

But, the good news, some asynchronous programming can be done
in OCaml without pulling in neither async nor lwt. :)

-- 
Regards,
Francois.
"When in doubt, use more types"

Re: [Caml-list] Question about Lwt/Async

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

On Tue, Mar 8, 2016 at 6:55 AM, Milan Stanojević <milanst@gmail.com> wrote:

> Of course, devil is in the details. I hope that this is indeed
> possible and that changes necessary to Lwt and Async to make this work
> are not huge.


​Right now we just need a few modifications to Async to make this work (Lwt
using Async backend or vice versa).​

It's relatively easy to get into a state with the limitation that you
couldn't use the same file descriptor with Lwt and Async, which seems
acceptable. For stdout/stderr it'd be fine as long as you don't print at
the same time, in which case the output might be interleaved. But with a
bit more work on Lwt and Async this could be sorted

-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 1610 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

On Tue, Mar 8, 2016 at 6:56 AM, Malcolm Matalka <mmatalka@gmail.com> wrote:

> Yotam Barnoy <yotambarnoy@gmail.com> writes:
>
> > Out of curiosity, what polling mechanism is available on the lwt side?
>
> I believe Lwt supports multiple event loops underneath, including libuv
> to get whatever polling the OS supports.


​Yes, by default Lwt has a backend using Unix.select, and a libev one if
 available at compile time (`opam install conf-libev`)
​. At runtime Lwt will use the Unix.select backend by default, except if it
was configured to choose the libev one (the default on Linux) [1].

There is also a backend using the glib main loop, for GTK applications

 [1]
https://github.com/ocsigen/lwt/blob/master/_oasis#L88

-- 
Jeremie

[-- Attachment #2: Type: text/html, Size: 2509 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Yaron Minsky]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

Jeremie, other than having some different back-ends available (e.g., glib
main loop), how different are the approaches to backend management between
Async and Lwt?

y

On Tue, Mar 8, 2016 at 6:04 AM, Jeremie Dimino <jdimino@janestreet.com>
wrote:

> On Tue, Mar 8, 2016 at 6:56 AM, Malcolm Matalka <mmatalka@gmail.com>
> wrote:
>
>> Yotam Barnoy <yotambarnoy@gmail.com> writes:
>>
>> > Out of curiosity, what polling mechanism is available on the lwt side?
>>
>> I believe Lwt supports multiple event loops underneath, including libuv
>> to get whatever polling the OS supports.
>
>
> ​Yes, by default Lwt has a backend using Unix.select, and a libev one if
>  available at compile time (`opam install conf-libev`)
> ​. At runtime Lwt will use the Unix.select backend by default, except if
> it was configured to choose the libev one (the default on Linux) [1].
>
> There is also a backend using the glib main loop, for GTK applications
>
>  [1]
> https://github.com/ocsigen/lwt/blob/master/_oasis#L88
>
> --
> Jeremie
>

[-- Attachment #2: Type: text/html, Size: 3200 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Jeremie Dimino]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

On Tue, Mar 8, 2016 at 12:47 PM, Yaron Minsky <yminsky@janestreet.com>
wrote:

> Jeremie, other than having some different back-ends available (e.g., glib
> main loop), how different are the approaches to backend management between
> Async and Lwt?
>

​The backend interfaces are slightly different​, but we just need a bit of
glue in the middle. Essentially the difference is that with Lwt you provide
one callback per fd and watch (read or write), while with Async you have a
global callback.

​Right now what we need to change in Async to make this work is:

- allow to provide a backend ​programmatically; right now you can only
choose between the predefined epoll and select ones
- make the scheduler ignore fds returned by the backend that are not
handled by async

[-- Attachment #2: Type: text/html, Size: 1756 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Malcolm Matalka]

Jeremie Dimino <jdimino@janestreet.com> writes:

> On Tue, Mar 8, 2016 at 12:47 PM, Yaron Minsky <yminsky@janestreet.com>
> wrote:
>
>> Jeremie, other than having some different back-ends available (e.g., glib
>> main loop), how different are the approaches to backend management between
>> Async and Lwt?
>>
>
> ​The backend interfaces are slightly different​, but we just need a bit of
> glue in the middle. Essentially the difference is that with Lwt you provide
> one callback per fd and watch (read or write), while with Async you have a
> global callback.
>
> ​Right now what we need to change in Async to make this work is:
>
> - allow to provide a backend ​programmatically; right now you can only
> choose between the predefined epoll and select ones
> - make the scheduler ignore fds returned by the backend that are not
> handled by async

For what it's worth, which isn't much right now, I've been slowly
developing an interface point for event loops and user facing code.  The
rough idea is to present "asynchronous system calls" like an OS would,
so user facing code has an interface to program against and the
underlying event loop can change as someone wants, libev, libuv, direct
epoll or kqueue, etc.  So Async and Lwt libraries could be implemented
in terms of this interface and share the same event loop, to cooperate
nicely.  So far I haven't implemented anything using the interface
except for a barely functional test to demonstrate that it even works,
so it's quite raw.  And it's clearly deficient on a few things, but I
think the idea is sound and would alleviate some of the pain of deciding
to use Lwt or Async and if it works on JS or Windows or My Favorite OS
(just flip out the underlying scheduler implementation).

The work in progress around the interface can be found below, any
constructive feedback would be appreciated.

https://bitbucket.org/acslab/abb_scheduler_inf/src

Re: [Caml-list] Question about Lwt/Async

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 3510 bytes --]

Am Mittwoch, den 09.03.2016, 07:35 +0000 schrieb Malcolm Matalka:
> Jeremie Dimino <jdimino@janestreet.com> writes:
> 
> > On Tue, Mar 8, 2016 at 12:47 PM, Yaron Minsky <yminsky@janestreet.com>
> > wrote:
> >
> >> Jeremie, other than having some different back-ends available (e.g., glib
> >> main loop), how different are the approaches to backend management between
> >> Async and Lwt?
> >>
> >
> > ​The backend interfaces are slightly different​, but we just need a bit of
> > glue in the middle. Essentially the difference is that with Lwt you provide
> > one callback per fd and watch (read or write), while with Async you have a
> > global callback.
> >
> > ​Right now what we need to change in Async to make this work is:
> >
> > - allow to provide a backend ​programmatically; right now you can only
> > choose between the predefined epoll and select ones
> > - make the scheduler ignore fds returned by the backend that are not
> > handled by async
> 
> For what it's worth, which isn't much right now, I've been slowly
> developing an interface point for event loops and user facing code.  The
> rough idea is to present "asynchronous system calls" like an OS would,
> so user facing code has an interface to program against and the
> underlying event loop can change as someone wants, libev, libuv, direct
> epoll or kqueue, etc.  So Async and Lwt libraries could be implemented
> in terms of this interface and share the same event loop, to cooperate
> nicely.  So far I haven't implemented anything using the interface
> except for a barely functional test to demonstrate that it even works,
> so it's quite raw.  And it's clearly deficient on a few things, but I
> think the idea is sound and would alleviate some of the pain of deciding
> to use Lwt or Async and if it works on JS or Windows or My Favorite OS
> (just flip out the underlying scheduler implementation).
> 
> The work in progress around the interface can be found below, any
> constructive feedback would be appreciated.
> 
> https://bitbucket.org/acslab/abb_scheduler_inf/src
> 

Very academic. The reality is different. Most of these operations are
only provided as synchronous calls anyway in all OS I know (and you can
only provide a non-blocking version by using helper threads). The only
operations you can do something about are those reading/writing a file
descriptor, but even here there is a strong OS dependency, e.g. on
Windows async operations are very restricted, limiting implementation
options drastically. The truth is that you cannot abstract the OS away.

And what if I need linkat and not link? And what about calling my
favorite C library that uses blocking I/O? E.g. I'm often preferring a
variant of Unix.read/write using bigarrays as buffer.

I'd prefer a reduced approach for interoperability: Focus on event loops
and ways to read/write, and accept that everything else must be dealt
with using helper threads.

Sorry for not being constructive. I don't like the approach (and I also
don't like Lwt and Async, and by the way these are not the only kids on
the block).

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] Question about Lwt/Async

[Malcolm Matalka]

Gerd Stolpmann <info@gerd-stolpmann.de> writes:

> Am Mittwoch, den 09.03.2016, 07:35 +0000 schrieb Malcolm Matalka:
>> Jeremie Dimino <jdimino@janestreet.com> writes:
>> 
>> > On Tue, Mar 8, 2016 at 12:47 PM, Yaron Minsky <yminsky@janestreet.com>
>> > wrote:
>> >
>> >> Jeremie, other than having some different back-ends available (e.g., glib
>> >> main loop), how different are the approaches to backend management between
>> >> Async and Lwt?
>> >>
>> >
>> > ​The backend interfaces are slightly different​, but we just need a bit of
>> > glue in the middle. Essentially the difference is that with Lwt you provide
>> > one callback per fd and watch (read or write), while with Async you have a
>> > global callback.
>> >
>> > ​Right now what we need to change in Async to make this work is:
>> >
>> > - allow to provide a backend ​programmatically; right now you can only
>> > choose between the predefined epoll and select ones
>> > - make the scheduler ignore fds returned by the backend that are not
>> > handled by async
>> 
>> For what it's worth, which isn't much right now, I've been slowly
>> developing an interface point for event loops and user facing code.  The
>> rough idea is to present "asynchronous system calls" like an OS would,
>> so user facing code has an interface to program against and the
>> underlying event loop can change as someone wants, libev, libuv, direct
>> epoll or kqueue, etc.  So Async and Lwt libraries could be implemented
>> in terms of this interface and share the same event loop, to cooperate
>> nicely.  So far I haven't implemented anything using the interface
>> except for a barely functional test to demonstrate that it even works,
>> so it's quite raw.  And it's clearly deficient on a few things, but I
>> think the idea is sound and would alleviate some of the pain of deciding
>> to use Lwt or Async and if it works on JS or Windows or My Favorite OS
>> (just flip out the underlying scheduler implementation).
>> 
>> The work in progress around the interface can be found below, any
>> constructive feedback would be appreciated.
>> 
>> https://bitbucket.org/acslab/abb_scheduler_inf/src
>> 
>
> Very academic. The reality is different. Most of these operations are
> only provided as synchronous calls anyway in all OS I know (and you can
> only provide a non-blocking version by using helper threads). The only
> operations you can do something about are those reading/writing a file
> descriptor, but even here there is a strong OS dependency, e.g. on
> Windows async operations are very restricted, limiting implementation
> options drastically. The truth is that you cannot abstract the OS
> away.

I'm not sure how thoroughly you read the link, but each call takes a
callback to indicate when the operation is complete.  I put "system
call" in quotes because it's attempting to define a what an asynchronous
set of system calls would look like.  It also defines a "Scheduler"
which maps to an event loop.

>
> And what if I need linkat and not link? And what about calling my
> favorite C library that uses blocking I/O? E.g. I'm often preferring a
> variant of Unix.read/write using bigarrays as buffer.

Deferring something to a thread is lacking currently, one of the
"obvious deficiencies" I mentioned.  I'm not sure how this is any
different from the given state of event loops in Ocaml.

>
> I'd prefer a reduced approach for interoperability: Focus on event loops
> and ways to read/write, and accept that everything else must be dealt
> with using helper threads.

This is entirely possible given the interface I linked to, I don't see
this as being a reason not to like the interface.  In fact, the demo
implementation of the interface in the link I gave does this, even
dirtier.  It runs everything in a thread and uses some queues and cond
variables to kick off callbacks.

>
> Sorry for not being constructive. I don't like the approach (and I also
> don't like Lwt and Async, and by the way these are not the only kids on
> the block).

Definitely, I'd like more options, the hope is that an interface like
this would provide a meeting point for the various libraries.

>
> Gerd

Re: [Caml-list] Question about Lwt/Async

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]

Am Mittwoch, den 09.03.2016, 14:37 +0000 schrieb Malcolm Matalka:
> > Sorry for not being constructive. I don't like the approach (and I also
> > don't like Lwt and Async, and by the way these are not the only kids on
> > the block).
> 
> Definitely, I'd like more options, the hope is that an interface like
> this would provide a meeting point for the various libraries.

I'm very much in favor of this. However, earlier attempts were not so
successful, so my thinking that it is better to reduce this to the bare
minimum - the luxury could be defined elsewhere.

So why not simply focus on, say poll, read, write?

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] Question about Lwt/Async

[François Bobot]

On 07/03/2016 18:03, Yaron Minsky wrote:
> - Async is more portable than it once was.  There's now Core_kernel,
>    Async_kernel and Async_rpc_kernel, which allows us to do things like
>    run Async applications in the browser.  I would think Windows
>    support would be pretty doable by someone who understands that world
>    well.
>
>    That said, the chain of dependencies brought in by Async is still
>    quite big.  This is something that could perhaps be improved, either
>    with better dead code analysis in OCaml, or some tweaks to
>    Async_kernel and Core_kernel themselves.
>

Is it possible and simple to write a Lwt_Async_rpc based on Async_rpc_kernel which would expose 
Lwt.t functions that dispatch or serve Async_rpc rpcs?

Thanks,

-- 
François

Re: [Caml-list] Question about Lwt/Async

[Yaron Minsky]

Not easily at present, I think. Obviously. if you run two runtimes in
two threads, you can do it, but it's ugly.  If you had
lwt-on-async-runtime available, as was discussed in the thread, then
you could do it, but you'd need some extra overhead to create matching
pairs of Lwt.t's and Deferred.t's, since those two types would not be
equal.

So, with some overhead, yeah, I think you could do it.

y

On Fri, Mar 11, 2016 at 8:21 AM, François Bobot <francois.bobot@cea.fr> wrote:
> On 07/03/2016 18:03, Yaron Minsky wrote:
>>
>> - Async is more portable than it once was.  There's now Core_kernel,
>>    Async_kernel and Async_rpc_kernel, which allows us to do things like
>>    run Async applications in the browser.  I would think Windows
>>    support would be pretty doable by someone who understands that world
>>    well.
>>
>>    That said, the chain of dependencies brought in by Async is still
>>    quite big.  This is something that could perhaps be improved, either
>>    with better dead code analysis in OCaml, or some tweaks to
>>    Async_kernel and Core_kernel themselves.
>>
>
> Is it possible and simple to write a Lwt_Async_rpc based on Async_rpc_kernel
> which would expose Lwt.t functions that dispatch or serve Async_rpc rpcs?
>
> Thanks,
>
> --
> François
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Question about Lwt/Async

[François Bobot]

On 11/03/2016 16:22, Yaron Minsky wrote:
> Not easily at present, I think. Obviously. if you run two runtimes in
> two threads, you can do it, but it's ugly.  If you had
> lwt-on-async-runtime available, as was discussed in the thread, then
> you could do it, but you'd need some extra overhead to create matching
> pairs of Lwt.t's and Deferred.t's, since those two types would not be
> equal.
>

Sorry I don't know if have been clear that I was thinking to make two ocaml process communicate 
together one using Async and the other Lwt. The use case for example, is to write a job server with 
Async and to make a web interface for it using ocsigen (Lwt); so the webserver (Lwt) would connect 
to the job server (Async) using the rpc protocol defined by Async_rpc_kernel.


> So, with some overhead, yeah, I think you could do it.

Does you answer is applicable to this use case?

Thank you already for this answer.

Best,

-- 
François

Re: [Caml-list] Question about Lwt/Async

[Yaron Minsky]

Well, Async_rpc_kernel is tied to the Deferred type, so it's not so
general as to abstract over which IO monad is being used.  So I'm
afraid it doesn't give you a nice way of interoperating with Async
programs.

That said, we're hopefully going to do a rewrite of the protocol in
the next few months.  It's not crazy that we could try to make some of
the low level protocol work shareable, at least.

y

On Fri, Mar 11, 2016 at 11:15 AM, François Bobot <francois.bobot@cea.fr> wrote:
> On 11/03/2016 16:22, Yaron Minsky wrote:
>>
>> Not easily at present, I think. Obviously. if you run two runtimes in
>> two threads, you can do it, but it's ugly.  If you had
>> lwt-on-async-runtime available, as was discussed in the thread, then
>> you could do it, but you'd need some extra overhead to create matching
>> pairs of Lwt.t's and Deferred.t's, since those two types would not be
>> equal.
>>
>
> Sorry I don't know if have been clear that I was thinking to make two ocaml
> process communicate together one using Async and the other Lwt. The use case
> for example, is to write a job server with Async and to make a web interface
> for it using ocsigen (Lwt); so the webserver (Lwt) would connect to the job
> server (Async) using the rpc protocol defined by Async_rpc_kernel.
>
>
>> So, with some overhead, yeah, I think you could do it.
>
>
> Does you answer is applicable to this use case?
>
> Thank you already for this answer.
>
> Best,
>
> --
> François