From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 62D337FF9F for ; Mon, 7 Mar 2016 18:04:06 +0100 (CET) IronPort-PHdr: 9a23:RLtCohJXktv5EIekLNmcpTZWNBhigK39O0sv0rFitYgUI/TxwZ3uMQTl6Ol3ixeRBMOAu60C27Od4viocFdDyKjCmUhKSIZLWR4BhJdetC0bK+nBN3fGKuX3ZTcxBsVIWQwt1Xi6NU9IBJS2PAWK8TWM5DIfUi/yKRBybrysXNWC0ILnjqvjo9X6WEZhunmUWftKNhK4rAHc5IE9oLBJDeIP8CbPuWZCYO9MxGlldhq5lhf44dqsrtY4q3wD89pozcNLUL37cqIkVvQYSW1+ayFmrPHs4DvOVhOC/DM4VXgXiVJhBQTI9gr3WN+lsCbhrudnni2dIMztC7kyVTm49KptYBDtgSYDcTU+9TeEpNZ3ifdqqQimoVRawojPY5DdYOt7f6XGfsIyR2NHU91NTSFMHsW3aI5ZXLlJBvpRs4So/whGlhC5HwT5Qbq3kjI= Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=yminsky@janestreet.com; spf=Pass smtp.mailfrom=yminsky@janestreet.com; spf=None smtp.helo=postmaster@mxout1.mail.janestreet.com Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of yminsky@janestreet.com) identity=pra; client-ip=38.105.200.112; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="yminsky@janestreet.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of yminsky@janestreet.com designates 38.105.200.112 as permitted sender) identity=mailfrom; client-ip=38.105.200.112; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="yminsky@janestreet.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mxout1.mail.janestreet.com) identity=helo; client-ip=38.105.200.112; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="yminsky@janestreet.com"; x-sender="postmaster@mxout1.mail.janestreet.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0BYAACds91WknDIaSZchAxtBro6AQ2BaR2FcgKBIgc4FAEBAQEBAQEBEAEBAQEHFglQgi2CFQEBBBIRHQEBNwEPCwsNAgImAgIhARIBBQEOAQ0GExsHh20DEgMLohaBMT4xik9nhEEBBIVsDYRCAQEBAQEBAQEBAQEBAQEBAQEBARECBApxhRyEPYI9hH2BOo4piQaFY4YVgXWOeocMhgsRHoEPHgEBgjgNEQiBZkwBiTwBAQE X-IPAS-Result: A0BYAACds91WknDIaSZchAxtBro6AQ2BaR2FcgKBIgc4FAEBAQEBAQEBEAEBAQEHFglQgi2CFQEBBBIRHQEBNwEPCwsNAgImAgIhARIBBQEOAQ0GExsHh20DEgMLohaBMT4xik9nhEEBBIVsDYRCAQEBAQEBAQEBAQEBAQEBAQEBARECBApxhRyEPYI9hH2BOo4piQaFY4YVgXWOeocMhgsRHoEPHgEBgjgNEQiBZkwBiTwBAQE X-IronPort-AV: E=Sophos;i="5.22,552,1449529200"; d="scan'208";a="206420697" Received: from mxout1.mail.janestreet.com ([38.105.200.112]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 07 Mar 2016 18:04:05 +0100 Received: from tot-qpr-mailcore1.delacy.com ([172.27.56.68] helo=tot-qpr-mailcore1) by mxout1.mail.janestreet.com with esmtps (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.82) (envelope-from ) id 1acyZg-0001iu-90 for caml-list@inria.fr; Mon, 07 Mar 2016 12:04:04 -0500 X-JS-Flow: external Received: by tot-qpr-mailcore1 with JS-mailcore (0.1) (envelope-from ) id BW3bSE-AAAAl6-HS; 2016-03-07 12:04:04.234086-05:00 Received: from mail-yw0-f178.google.com ([209.85.161.178]) by mxgoog1.mail.janestreet.com with esmtps (UNKNOWN:AES128-GCM-SHA256:128) (Exim 4.72) (envelope-from ) id 1acyZg-0007Vn-5J for caml-list@inria.fr; Mon, 07 Mar 2016 12:04:04 -0500 Received: by mail-yw0-f178.google.com with SMTP id d65so28713649ywb.0 for ; Mon, 07 Mar 2016 09:04:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=wuTGrPhT4h60JAvmi4ueI/yM71r/kPcjJwBsw1JQnPY=; b=w13elFfA80BhE0EgipLGZITEuxyKUU2aKB6fPhuG1Hl6m+J0m+m4VnFX5rpeXhdrBn K2RTaRw6lrmjvCs4XYe2GDscUNOuLozpr2a5gzAi7Th3ZzVLWX6GNkHLyMYyHLXG0jfp NqS2fGjojYsWzBm54UBnuXNKOkmmH4SXU2h0s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=wuTGrPhT4h60JAvmi4ueI/yM71r/kPcjJwBsw1JQnPY=; b=nGFbrDFiZJzYXLuGvDiHUStAjUImZvuapLZqT0xDhxCl/rft6SjmLmgArapzN/ER0v 0N828+knpxQ+8tgs3vwqWulL8MJxdkGWRuhorV7P00e/2Xqm5pEvQ2vXg82opBom7vH6 1ItIua74PRNpWXbaGQ9fMT0LLOd+VEj5gPAq6knCLP4r/cdP84bUJ/4CE9yyuGKyB4qq l8/+B+vPoqQ4KAVqlGkt7YkrPBxH9YSX6k4ZnTRbJ9o3153EPy778kNpXIPQCKmJD7lm 904iTIL3ioKlR9IX37ZSkwdPhBKEGh04CPA72ZwNxmlgtzkOKekOXhjX/q5geGhugbWp mg6A== X-Gm-Message-State: AD7BkJKZdUq6fiAnu6IYY0zw+fGBeUhiYWiB/x/29/bTYqNRmIqtLYLDkx+Cvc0VhKszmzLo5LlNg/bsZLCsLsFW6BOXlbRRMca0u8y/jAeZoJtPxDQhf23Kva2mloNsdG+wGiObqAIbp6Zp6Nfk X-Received: by 10.129.48.12 with SMTP id w12mr866526yww.82.1457370243851; Mon, 07 Mar 2016 09:04:03 -0800 (PST) X-Received: by 10.129.48.12 with SMTP id w12mr866504yww.82.1457370243616; Mon, 07 Mar 2016 09:04:03 -0800 (PST) MIME-Version: 1.0 Received: by 10.13.226.65 with HTTP; Mon, 7 Mar 2016 09:03:44 -0800 (PST) In-Reply-To: References: From:Yaron Minsky Date: Mon, 7 Mar 2016 12:03:44 -0500 Message-ID: To:Jesper Louis Andersen Cc:Yotam Barnoy , Ocaml Mailing List Content-Type: text/plain; charset=UTF-8 X-JS-Processed-by: mailcore Subject: Re: [Caml-list] Question about Lwt/Async This is definitely a fraught topic, and it's unfortunate that there's no clear solution. To add a bit more information: - Async is more portable than it once was. There's now Core_kernel, Async_kernel and Async_rpc_kernel, which allows us to do things like run Async applications in the browser. I would think Windows support would be pretty doable by someone who understands that world well. That said, the chain of dependencies brought in by Async is still quite big. This is something that could perhaps be improved, either with better dead code analysis in OCaml, or some tweaks to Async_kernel and Core_kernel themselves. - There are things we could contemplate to make it easier to bridge the divide. Jeremie Dimino did a proof of concept rewrite of lwt to use async as its implementation, where an Lwt.t and a Deferred.t are equal at the type level. https://github.com/janestreet/lwt-async Another possibility, and one that might be easier to write, would be to allow Lwt code to run using the Async scheduler as another possible back-end. This would allow one to have programs that used both Async and Lwt together in one program, without running on different threads. It's worth mentioning if that there is interest in making Async more suitable for a wider variety of goals, we're happy to work with outside contributors on it. For example, if someone wanted to work on Windows support for Async, we'd be happy to help out on integrating that work. Probably the biggest issue is executable size. That will get better when we release an unpacked version of our external libraries. But even then, the module-level granularity captures more things than would be ideal. y On Mon, Mar 7, 2016 at 10:16 AM, Jesper Louis Andersen wrote: > > On Mon, Mar 7, 2016 at 2:38 AM, Yotam Barnoy wrote: >> >> Also, what happens to general utility functions that aren't rewritten for >> Async/Lwt -- as far as I can tell, being in non-monadic code, they will >> always starve other threads, since they cannot yield to another Async/Lwt >> thread. Is this perception correct? > > > Yes. > > On one hand, your observation is negative in the sense that now your code > has "color" in the sense that it is written for one library only. And you > have to transform code to having the right color before it can be used. This > is not the case if the concurrency model is at a lower level[0]. > > On the other hand, your observation is positive: cooperative scheduling > makes the points in which the code can switch explicit. This gives the > programmer far more control over when you are done with a task and start to > process the next task. You can also avoid the preemption check in the code > all the time. If your code manipulates lots of shared data, it also > simplifies things since you don't usually have to protect data with a mutex > in a single-threaded context as much[1]. Cooperative models, if carefully > managed, can exploit structure in the problem domain, whereas a preemptive > model needs to fit all. > > My personal opinion is that the preemptive model eventually wins over the > cooperative model, much like it has in most (all popular) operating systems. > It is simply more productive to take an up-front performance hit as a > sacrifice for a system which is more robust against stray code misbehaving. > If a cooperative system fails, it is fails catastrophically. If a preemptive > system fails, it degrades in performance. > > But given I have more than 10 years of Erlang programming behind me by now, > I'm obviously biased toward certain computational models :) > > [0] Erlang would be one such example, where the system is preemptively > scheduling for you and you can use any code in any place without having to > worry about blocking for latency. Go is quasi-preemptive because it checks > on function calls, but in contrast to Erlang a loop is not forced to factor > through a recursion, so it can in principle run indefinitely. Haskell (GHC) > is quasi-preemptive as well, checking on memory allocation boundaries. So > the thing to look out for in GHC is latency from processing large arrays > with no allocation, say. > > [1] Erlang has two VM runtimes for this reason. One is single-threaded and > can avoid lots of locks which is far faster for certain workloads, or on > embedded devices with a single core only. > > -- > J.