From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 9BE877FE07 for ; Thu, 4 Feb 2016 01:14:58 +0100 (CET) IronPort-PHdr: 9a23:WQ14lBEZZjS1I707cPsmtJ1GYnF86YWxBRYc798ds5kLTJ75oMiwAkXT6L1XgUPTWs2DsrQf27WQ6vGrAzRIyK3CmU5BWaQEbwUCh8QSkl5oK+++Imq/EsTXaTcnFt9JTl5v8iLzG0FUHMHjew+a+SXqvnYsExnyfTB4Ov7yUtaLyZ/niKbrqtaJO01hv3mUX/BbFF2OtwLft80b08NJC50a7V/3mEZOYPlc3mhyJFiezF7W78a0+4N/oWwL46pyv+YJa6jxfrw5QLpEF3xmdjltvIy4/SXEGC6G4nAbVmBetxNUCgzG5VmuW5L4riL+teNV1yyTPMmwRrcxD2eM9aBuHT3lkioCJnYI+WXTjdQ42LNSpBamvzRwxofVZMeeM/8oLfCVRs8TWWcUBpUZbCdGGI7pKtJXV+c= Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=yallop@gmail.com; spf=Pass smtp.mailfrom=yallop@gmail.com; spf=None smtp.helo=postmaster@mail-qg0-f54.google.com Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of yallop@gmail.com) identity=pra; client-ip=209.85.192.54; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="yallop@gmail.com"; x-sender="yallop@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of yallop@gmail.com designates 209.85.192.54 as permitted sender) identity=mailfrom; client-ip=209.85.192.54; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="yallop@gmail.com"; x-sender="yallop@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-qg0-f54.google.com) identity=helo; client-ip=209.85.192.54; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="yallop@gmail.com"; x-sender="postmaster@mail-qg0-f54.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0DLAAAJl7JWlDbAVdFehAxeDwaIVbEaAQ2BZiOFIEoCgToHOBQBAQEBAQEBARABAQEBBwsLCR8wQRABgVuCFAEBAQMBEhEdARsdAQMBCwYFCwMKAgImAgIhAQERAQUBHAYTGweHYwEDCggOpAmBMT4xizSBaYJXhT8KGScNUYNuAQEBAQEBAQMBAQEBAQEUAQEECgRtiU6CN4FBgzqBOgWOGYhYhUmCbIMlgXOBW40Whn+Db4IWER6BDR4BAYI4HoFxGy6DDYUugTABAQE X-IPAS-Result: A0DLAAAJl7JWlDbAVdFehAxeDwaIVbEaAQ2BZiOFIEoCgToHOBQBAQEBAQEBARABAQEBBwsLCR8wQRABgVuCFAEBAQMBEhEdARsdAQMBCwYFCwMKAgImAgIhAQERAQUBHAYTGweHYwEDCggOpAmBMT4xizSBaYJXhT8KGScNUYNuAQEBAQEBAQMBAQEBAQEUAQEECgRtiU6CN4FBgzqBOgWOGYhYhUmCbIMlgXOBW40Whn+Db4IWER6BDR4BAYI4HoFxGy6DDYUugTABAQE X-IronPort-AV: E=Sophos;i="5.22,392,1449529200"; d="scan'208";a="201191832" Received: from mail-qg0-f54.google.com ([209.85.192.54]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/AES128-GCM-SHA256; 04 Feb 2016 01:14:57 +0100 Received: by mail-qg0-f54.google.com with SMTP id b35so29406746qge.0 for ; Wed, 03 Feb 2016 16:14:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=4JuMndAOnHksBivb6mixcp/QK4Pvd/0uD5OeSSGKtO0=; b=sggRVoPmWbZ2SxvcTrZDGN+tKk5B80wwjR1fYcbqEyXQ8NjM55He2iNVtl4TBd/uzP 2FF0WUF37FUKFmb5VZn5WUZbQISYb5u+LJNh6RgfiVfzfgAxWntUi+Rc43HA182j3mIk hXZ3yizCg6tb/XdI1hzD1eEHMWYpmwjYTJXLd8vWgjWMdGa0ro7ablMSUoDdP/C9RHuT UGz/1GuIo8Vq6P8uRdwSan0QgIL/2t6D6sQ4jabndsrF6ZfUNPdVfmWhxC0Bfw+An/J3 S46cMGyb/tlaSq4NzvuVcB15LbInHVUyInoyD9Dnk2BcTfqmA6oy1bsuoDxd4/wRArWQ wSjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=4JuMndAOnHksBivb6mixcp/QK4Pvd/0uD5OeSSGKtO0=; b=cqHKjiRyjHh3MS/tudLB0vrKW9YMW2EvxLq+lsyrSDs4aXig7n51lWIlN57So8mzp6 7pUE4W5R4pZIpV5mAzkKy9H+SeAYqvnzntHeP4jvha8UCyevyk0B14PH3RKIlD/MJmUl VAycqMyo7Rzn7YUv+6s8vdo5EjK8SaGT29cZrnuOpcYKwHnz2eqjzRk0HDfmwQdrzb7N qlaUpNPU4IlhZhO+MK0LuCFq0epA7/TBxuzLrSGO+iU2/g6UmQUspasugpsO3g9GtWUi +60QVyd6ZpplDHb6SlbmTtmZmkqrsHPfKJg7Whd1gnKtMYkiU9dNEumCHHxB33HKFRra Ww6g== X-Gm-Message-State: AG10YOQtO2H64CU36gpOYsnBXsyaaUydze8ozifDa3Q82qvh0dRTH/FiUM6Qh4TsWOVrF8+IFgotIXd0XzaeQg== MIME-Version: 1.0 X-Received: by 10.140.99.5 with SMTP id p5mr5445167qge.76.1454544895592; Wed, 03 Feb 2016 16:14:55 -0800 (PST) Received: by 10.55.192.129 with HTTP; Wed, 3 Feb 2016 16:14:55 -0800 (PST) In-Reply-To: <86mvrhr1o1.fsf@gmail.com> References: <56B1DC54.1060109@tu-berlin.de> <86r3guq8t4.fsf@gmail.com> <86mvrhr1o1.fsf@gmail.com> Date: Wed, 3 Feb 2016 16:14:55 -0800 Message-ID: From: Jeremy Yallop To: Malcolm Matalka Cc: David Sheets , Jeremie Dimino , =?UTF-8?Q?Christoph_H=C3=B6ger?= , caml users Content-Type: text/plain; charset=UTF-8 Subject: Re: [Caml-list] Save callbacks from OCaml to C On 3 February 2016 at 12:15, Malcolm Matalka wrote: > Jeremy Yallop writes: > >> On 3 February 2016 at 05:44, David Sheets wrote: >>> On Wed, Feb 3, 2016 at 12:26 PM, Malcolm Matalka wrote: >>>> Jeremie Dimino writes: >>>>> You need to register [ml_t], [ml_x] and [ml_g >>>>> ] as GC roots. Otherwise if the GC runs in caml_ba_alloc for instance, >>>>> [ml_t] might ends up containing garbage even before reaching >>>>> [caml_callback3]. You can use the normal macros for that: >>>>> >>>> If one is using ctypes, is all of this taken care of? I have a library >>>> that registers a bunch of Ocaml functions in C code, which the C code >>>> calls. I haven't experienced anything bad happening yet, but that >>>> doesn't mean much... >>> >>> If you use ctypes and pass OCaml closures to C, you *must* retain a >>> reference to the closure to avoid it being GCed. If you do not, you >>> may experience the exception CallToExpiredClosure sporadically. >> >> Besides David's caveat, the answer is yes: ctypes will take care of >> registering arguments as GC roots as necessary. > > Can you clarify this a bit? I'm not that familiar with how the C FFI > works. If I pass in a closure to a C function and it is registered as a > GC root, doesn't that mean it won't be GCd if my Ocaml program forgets > about it or? That's how roots behave, yes: while a value is registered as a root, the value won't be collected. There are (roughly speaking) two types of root in OCaml: local roots, which persist for the duration of a function call, and global roots, which persist until explicitly released. A C function binding written by hand must ensure that OCaml values passed to it as arguments are registered as local roots, so that if a collection occurs while the function is running the values won't be prematurely collected. A C binding written using ctypes can generally ignore the matter of roots. That's partly because ctypes takes care of root registration, but also because most types passed between OCaml and C in a ctypes binding are C values, not OCaml values. For example, if you want to pass a structure with several fields between OCaml and C there are two approaches. One approach is to represent the structure as an OCaml record, which involves accessing the fields of the value in your C binding using various macros, taking care to register values as roots to protect them from the GC. The other approach is to represent the structure as a C struct, which involves accessing its fields in OCaml using the functions ctypes provides. (If you enjoy programming in an untyped dialect of C with ubiquitous concurrency, you'll probably favour the first approach. If you prefer programming in OCaml then the second approach might have some appeal.) Using the C value representation for values that cross the C-OCaml boundary generally works well, but when things become higher-order, the situation changes a bit. When a C library expects to be given a first-order value such as a struct we have to give it a struct with the appropriate layout, since C functions can directly access the representation of values. However, when the library expects a function pointer we have a bit more freedom, since the representation of functions isn't accessible -- in fact, the only thing that can be done with a function pointer, besides passing it from place to place, is calling it. This freedom means that we can pass an OCaml function, suitably packaged up, where a C function pointer is expected. Passing OCaml functions to C as function pointers raises some interesting issues relating to object lifetime and the garbage collector. The main difficulty arises from the fact that once you pass a function pointer to a C library there's no way of knowing how long the library holds on to it: for example, the library might discard the function pointer when the call into the library returns, or it might store the function pointer in a global variable to be retrieved and called later. In order to prevent the associated function from being collected prematurely, some kind of action is needed on the OCaml side, whether registering a global root, or ensuring that the function is reachable from the OCaml program. > Also, David and I were talking about how to solve this on IRC. In my > specific case, callbacks are one-shot, which means I know they need to > be remembered until they are called then they can (possibly) be freed. > Is there a nice solution here? I'd prefer not to store them in some > other data structure and remove them later just to keep a reference > alive, if possible. Storing some kind of references to the functions in a place that the collector can see is essential to prevent the functions from being collected prematurely. The situation is the same whether you use ctypes or write bindings by hand. Storing the functions in a table, and removing them automatically after they're called is one approach. An alternative is to use the new Ctypes.Roots module, which will be available in the next release: https://github.com/ocamllabs/ocaml-ctypes/blob/182a9e64src/ctypes/ctypes.mli#L419-L435 > That is overhead I'd prefer to avoid, if possible. > I plan on having possibly hundreds of thousands of these callbacks alive > at any point in time. In that case it sounds like there'll be an overhead of up to a few megabytes.