From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id 09FEF7FF9F for ; Mon, 7 Mar 2016 19:41:45 +0100 (CET) IronPort-PHdr: 9a23:ME1F6xeBEJUIz84gTLxJ0zDglGMj4u6mDksu8pMizoh2WeGdxc6yZR7h7PlgxGXEQZ/co6odzbGG7OawACdZuMzJmUtBWaIPfidNsd8RkQ0kDZzNImzAB9muURYHGt9fXkRu5XCxPBsdMs//Y1rPvi/6tmZKSV3BPAZ4bt74BpTVx5zukbvipNuDOk4R3WD1SIgxBSv1hD2ZjtMRj4pmJ/R54TryiVwMRd5rw3h1L0mYhRf265T41pdi9yNNp6BprJYYAu3SNp41Rr1ADTkgL3t9pIiy7UGCHkOz4S45W2EdlR5NSy3M8Bj+XZ655i7/v/Z03CqTFcLzRLEwHz+l6vE4ZgXvjXI2PiQ+9inyi8prj7MT9AOkphpkwJ/8YoiTOeFiZK7QYZURQm8XDZUZbDBIHo7pN9hHNOEGJ+sN6tCl/1Y= Authentication-Results: mail3-smtp-sop.national.inria.fr; spf=None smtp.pra=yminsky@janestreet.com; spf=Pass smtp.mailfrom=yminsky@janestreet.com; spf=None smtp.helo=postmaster@mxout1.mail.janestreet.com Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of yminsky@janestreet.com) identity=pra; client-ip=38.105.200.112; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="yminsky@janestreet.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of yminsky@janestreet.com designates 38.105.200.112 as permitted sender) identity=mailfrom; client-ip=38.105.200.112; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="yminsky@janestreet.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mxout1.mail.janestreet.com) identity=helo; client-ip=38.105.200.112; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="postmaster@mxout1.mail.janestreet.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0BYAACLyt1WknDIaSZchAxtBro6AQ2BaR2FcgKBJQc4FAEBAQEBAQEBEAEBAQEHFglQgi2CFQEBBBIRHQEBKQ4BDwsLAwoCAiYCAiEBEgEFAQ4BDQYTCBMHh20DEgMLoWqBMT4xik9nhEEBBIV1DYRCAQEBAQEBAQEBAQEBAQEBAQEBARECBApxhRyEPYI9gVeDJoE6jimJBoVjhhWBdY56hwyGCxEegQ8eAQGCOA0RCIFmTAGIAYE7AQEB X-IPAS-Result: A0BYAACLyt1WknDIaSZchAxtBro6AQ2BaR2FcgKBJQc4FAEBAQEBAQEBEAEBAQEHFglQgi2CFQEBBBIRHQEBKQ4BDwsLAwoCAiYCAiEBEgEFAQ4BDQYTCBMHh20DEgMLoWqBMT4xik9nhEEBBIV1DYRCAQEBAQEBAQEBAQEBAQEBAQEBARECBApxhRyEPYI9gVeDJoE6jimJBoVjhhWBdY56hwyGCxEegQ8eAQGCOA0RCIFmTAGIAYE7AQEB X-IronPort-AV: E=Sophos;i="5.22,552,1449529200"; d="scan'208";a="167498222" Received: from mxout1.mail.janestreet.com ([38.105.200.112]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 Mar 2016 19:41:43 +0100 Received: from tot-qpr-mailcore2.delacy.com ([172.27.56.106] helo=tot-qpr-mailcore2) by mxout1.mail.janestreet.com with esmtps (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.82) (envelope-from ) id 1ad06A-0008Sy-Gj for caml-list@inria.fr; Mon, 07 Mar 2016 13:41:42 -0500 X-JS-Flow: external Received: by tot-qpr-mailcore2 with JS-mailcore (0.1) (envelope-from ) id BW3ctm-AAAEq1-OZ; 2016-03-07 13:41:42.462200-05:00 Received: from mail-yk0-f181.google.com ([209.85.160.181]) by mxgoog2.mail.janestreet.com with esmtps (UNKNOWN:AES128-GCM-SHA256:128) (Exim 4.72) (envelope-from ) id 1ad06A-0000oz-9b for caml-list@inria.fr; Mon, 07 Mar 2016 13:41:42 -0500 Received: by mail-yk0-f181.google.com with SMTP id x17so16984769ykd.1 for ; Mon, 07 Mar 2016 10:41:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=4ENy6NaKwvJvNDTVtZ0oYgVVIL+wGCtRQpRJfIcvqOo=; b=UbW/w7hCfaN2xKTjr2t5xYzh9h3FrZ2Zzm2MPvQLN1t39FJ1w60BJrHprhS4G2ZipZ Mhoi50DtKKHQpIflpA/hT3RUWiHt0uoAljwLNySpKJlDtcEymY7HUR6tX56zaWDEN7bS SzOc3LbO7jamLW9Rp9OhsRv4fCpyl2lB/I0Mk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=4ENy6NaKwvJvNDTVtZ0oYgVVIL+wGCtRQpRJfIcvqOo=; b=SroSJ94uSAnCPExNhO3IRT7hdxnMpIzQhrhCcqkBJj62i+ExUxcJ7bwf5lCnCEs31W 294Av65O4DN3nvfUIQoBTM29JuwFbxUiV0nm3OMH1Myb/2tKnEkhXVHVPWIW2fKNK8hn xfOOgdkhwDphKEF1Y0hD6SVu8JvDYaHxrEzXGgXr6mY04lCol1vCy9bPvj9lK2e7+scr QDo+OYNeCp/sEdYU/dnunkmyO2odd6i4ZVaWdObGAtNrr7nA462DDyQ1pSfTjERzEpuJ y22stF/dI8RsrH/JWKCdBX1TQMDOkylSu5Q9GjsimfbhFg+qBM72lv/ZeDjXmxOlMgGe /BTw== X-Gm-Message-State: AD7BkJKqFgKFzT/dPJIWVZIB88wWLZytS7CQWYvmIitaxMo0mJtOIYF/ftNlfC4L/yWM37nkZHTgZOyAsnOdzmxKXQxpF5YN0z2cuh+r0/BEk76ZCBEeXE7ut+MCwUWxclzzRF3On3gvLHzOn32c X-Received: by 10.37.28.193 with SMTP id c184mr14042014ybc.88.1457376101978; Mon, 07 Mar 2016 10:41:41 -0800 (PST) X-Received: by 10.37.28.193 with SMTP id c184mr14042009ybc.88.1457376101852; Mon, 07 Mar 2016 10:41:41 -0800 (PST) MIME-Version: 1.0 Received: by 10.13.226.65 with HTTP; Mon, 7 Mar 2016 10:41:22 -0800 (PST) In-Reply-To: <86h9gi9msc.fsf@gmail.com> References: <86h9gi9msc.fsf@gmail.com> From:Yaron Minsky Date: Mon, 7 Mar 2016 13:41:22 -0500 Message-ID: To:Malcolm Matalka Cc:Jesper Louis Andersen , Yotam Barnoy , Ocaml Mailing List Content-Type: text/plain; charset=UTF-8 X-JS-Processed-by: mailcore Subject: Re: [Caml-list] Question about Lwt/Async Right now, only select and epoll are supported, but adding support for something else isn't hard. The Async_unix library has an interface called File_descr_watcher_intf.S, which both select and epoll go through. Adding support for another shouldn't be difficult if someone with the right OS expertise wants to do it. Is there a particular kernel API you want support for? y On Mon, Mar 7, 2016 at 1:16 PM, Malcolm Matalka wrote: > Yaron Minsky writes: > >> This is definitely a fraught topic, and it's unfortunate that there's >> no clear solution. >> >> To add a bit more information: >> >> - Async is more portable than it once was. There's now Core_kernel, >> Async_kernel and Async_rpc_kernel, which allows us to do things like >> run Async applications in the browser. I would think Windows >> support would be pretty doable by someone who understands that world >> well. >> >> That said, the chain of dependencies brought in by Async is still >> quite big. This is something that could perhaps be improved, either >> with better dead code analysis in OCaml, or some tweaks to >> Async_kernel and Core_kernel themselves. > > When I last looked at the scheduler it was limited to [select] or > [epoll], is this still the case? How difficult would it be to expand on > those? > >> >> - There are things we could contemplate to make it easier to bridge >> the divide. Jeremie Dimino did a proof of concept rewrite of lwt to >> use async as its implementation, where an Lwt.t and a Deferred.t are >> equal at the type level. >> >> https://github.com/janestreet/lwt-async >> >> Another possibility, and one that might be easier to write, would be >> to allow Lwt code to run using the Async scheduler as another >> possible back-end. This would allow one to have programs that used >> both Async and Lwt together in one program, without running on >> different threads. >> >> It's worth mentioning if that there is interest in making Async more >> suitable for a wider variety of goals, we're happy to work with >> outside contributors on it. For example, if someone wanted to work on >> Windows support for Async, we'd be happy to help out on integrating >> that work. >> >> Probably the biggest issue is executable size. That will get better >> when we release an unpacked version of our external libraries. But >> even then, the module-level granularity captures more things than >> would be ideal. >> >> y >> >> On Mon, Mar 7, 2016 at 10:16 AM, Jesper Louis Andersen >> wrote: >>> >>> On Mon, Mar 7, 2016 at 2:38 AM, Yotam Barnoy wrote: >>>> >>>> Also, what happens to general utility functions that aren't rewritten for >>>> Async/Lwt -- as far as I can tell, being in non-monadic code, they will >>>> always starve other threads, since they cannot yield to another Async/Lwt >>>> thread. Is this perception correct? >>> >>> >>> Yes. >>> >>> On one hand, your observation is negative in the sense that now your code >>> has "color" in the sense that it is written for one library only. And you >>> have to transform code to having the right color before it can be used. This >>> is not the case if the concurrency model is at a lower level[0]. >>> >>> On the other hand, your observation is positive: cooperative scheduling >>> makes the points in which the code can switch explicit. This gives the >>> programmer far more control over when you are done with a task and start to >>> process the next task. You can also avoid the preemption check in the code >>> all the time. If your code manipulates lots of shared data, it also >>> simplifies things since you don't usually have to protect data with a mutex >>> in a single-threaded context as much[1]. Cooperative models, if carefully >>> managed, can exploit structure in the problem domain, whereas a preemptive >>> model needs to fit all. >>> >>> My personal opinion is that the preemptive model eventually wins over the >>> cooperative model, much like it has in most (all popular) operating systems. >>> It is simply more productive to take an up-front performance hit as a >>> sacrifice for a system which is more robust against stray code misbehaving. >>> If a cooperative system fails, it is fails catastrophically. If a preemptive >>> system fails, it degrades in performance. >>> >>> But given I have more than 10 years of Erlang programming behind me by now, >>> I'm obviously biased toward certain computational models :) >>> >>> [0] Erlang would be one such example, where the system is preemptively >>> scheduling for you and you can use any code in any place without having to >>> worry about blocking for latency. Go is quasi-preemptive because it checks >>> on function calls, but in contrast to Erlang a loop is not forced to factor >>> through a recursion, so it can in principle run indefinitely. Haskell (GHC) >>> is quasi-preemptive as well, checking on memory allocation boundaries. So >>> the thing to look out for in GHC is latency from processing large arrays >>> with no allocation, say. >>> >>> [1] Erlang has two VM runtimes for this reason. One is single-threaded and >>> can avoid lots of locks which is far faster for certain workloads, or on >>> embedded devices with a single core only. >>> >>> -- >>> J.