From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id D65487FE07 for ; Thu, 4 Feb 2016 08:26:22 +0100 (CET) IronPort-PHdr: 9a23:li7UeBYkO1T9MA5l7gh97ZH/LSx+4OfEezUN459isYplN5qZpcu8bnLW6fgltlLVR4KTs6sC0LqJ9fmxEjVYvd6oizMrTt9lb1c9k8IYnggtUoauKHbQC7rUVRE8B9lIT1R//nu2YgB/Ecf6YEDO8DXptWZBUiv2OQc9HOnpAIma153xjLDtvcSCKFwQ2HKUWvBbElaflU3prM4YgI9veO4a6yDihT92QdlQ3n5iPlmJnhzxtY+a9Z9n9DlM6bp6r5YTGY2zRakzTKRZATI6KCh1oZSz7ViQBTeIs1kRSGgTg1J5CgzB6wmyCob4ti/9rsJy3SCbOYv9SrViChq46KI+bRbsgyADMnYc+X3ejs95xPZepRu9rhh8yqbbZYiUMLx1eaaLLoBSfnZIQssED38JOYi7dYZaSrNZZes= Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=mmatalka@gmail.com; spf=Pass smtp.mailfrom=mmatalka@gmail.com; spf=None smtp.helo=postmaster@mail-wm0-f52.google.com Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of mmatalka@gmail.com) identity=pra; client-ip=74.125.82.52; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="mmatalka@gmail.com"; x-sender="mmatalka@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of mmatalka@gmail.com designates 74.125.82.52 as permitted sender) identity=mailfrom; client-ip=74.125.82.52; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="mmatalka@gmail.com"; x-sender="mmatalka@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-wm0-f52.google.com) identity=helo; client-ip=74.125.82.52; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="mmatalka@gmail.com"; x-sender="postmaster@mail-wm0-f52.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0CYAADx+7JWlDRSfUpehAxeD4hbsQUBDYFmFwyFIEoCgTo4FAEBAQEBAQEBEAEBAQEHCwsJHzCCLYIUAQEBAwESFRkBGx0BAwELBgULDQklDwEEDQIRAQUBIhMbB4djAQMKCAQKo3WBMT4xjR2CV4VHChknDVGDbgEBAQEBAQEBAQEBAQEBAQEBARIBAQQKBIQBggOEN4I3gUGDAkuBJwWWcYVJgmyDJYFzgVuHN4Vfhn+Db4IWL4ENHgEBgjgegVBqhnqBMAEBAQ X-IPAS-Result: A0CYAADx+7JWlDRSfUpehAxeD4hbsQUBDYFmFwyFIEoCgTo4FAEBAQEBAQEBEAEBAQEHCwsJHzCCLYIUAQEBAwESFRkBGx0BAwELBgULDQklDwEEDQIRAQUBIhMbB4djAQMKCAQKo3WBMT4xjR2CV4VHChknDVGDbgEBAQEBAQEBAQEBAQEBAQEBARIBAQQKBIQBggOEN4I3gUGDAkuBJwWWcYVJgmyDJYFzgVuHN4Vfhn+Db4IWL4ENHgEBgjgegVBqhnqBMAEBAQ X-IronPort-AV: E=Sophos;i="5.22,393,1449529200"; d="scan'208";a="201230166" Received: from mail-wm0-f52.google.com ([74.125.82.52]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/AES128-GCM-SHA256; 04 Feb 2016 08:26:22 +0100 Received: by mail-wm0-f52.google.com with SMTP id l66so12971510wml.0 for ; Wed, 03 Feb 2016 23:26:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-type; bh=lk7wEJJRG/Pigcl/QA/lOzLemvG7SkL6JfpGzcAPON0=; b=BjYx32A67Y6Wm9LoXWga1+jICtCzy9GZBXWykBx5phjt/RRi2YU1Rus6Uy6E59UFIN w594axME//IqtG5eRhHlqj1//yz+Vwk3J1AyPMt6AQcAS31HppWqVTSYQHS7Fi7gS6i4 jhxL6TmIybu0pFtyM+LbkqnOD0N858osuyhENdIaDTlzi8WihnTLBH00TydIYepYtUsR ZYwMnrI8Y634WL0wD9Xuagwa83FWqQon4EPt9cCw3n2hJWBbbEkWJeZwQy5O5eLywMuP a9natZJ4Qmw7SU0Bx53S/mq+Acy4ouoZCMwGOEhs+sClZH2s+hoBjl2zoM6PERiWrjZi sbcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-type; bh=lk7wEJJRG/Pigcl/QA/lOzLemvG7SkL6JfpGzcAPON0=; b=Gvh26v/06KIryKzRcy+tc9U9j+pdt2Tn+IbVBqgNux+pizium5aaCJOm/POjD6UUkF hGL99nc/pq0J5B0TPJmgLeUSFsWHcKcC4NMAR7ipPLyaCArfbfEz0JMkdSkPJqpEyuCO orguBum6/YLihdPSKVNrPs/0CwcFu19FqCaIdedGB/WOACAua4LaSLejIc9LLM4y2EYz mn9dfcs2Hl4BYEF5mgwTce9S/duNcZ0cpbEezNIcfp6iURVtchIUG++ldBEcYXpA9sUq pI16Mx9IPaUdDNDO45JPDQgd1Go/6oEHO0fTe1Wpvoyo/aHBYuHeEv/dYacwBuBZWmGW 3ATA== X-Gm-Message-State: AG10YOTI7Q7YebtMz+ifIyZ0nz220xrfHz4DTmDf7PCGXm6DFFmhDHM1xm9lcAofQturhA== X-Received: by 10.28.10.7 with SMTP id 7mr8194326wmk.57.1454570781757; Wed, 03 Feb 2016 23:26:21 -0800 (PST) Received: from localhost ([37.153.108.22]) by smtp.gmail.com with ESMTPSA id w66sm11334859wmd.2.2016.02.03.23.26.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Feb 2016 23:26:20 -0800 (PST) From: Malcolm Matalka To: Jeremy Yallop Cc: David Sheets , Jeremie Dimino , Christoph =?utf-8?Q?H=C3=B6ger?= , caml users References: <56B1DC54.1060109@tu-berlin.de> <86r3guq8t4.fsf@gmail.com> <86mvrhr1o1.fsf@gmail.com> Date: Thu, 04 Feb 2016 07:26:06 +0000 In-Reply-To: (Jeremy Yallop's message of "Wed, 3 Feb 2016 16:14:55 -0800") Message-ID: <86io25q6lt.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain Subject: Re: [Caml-list] Save callbacks from OCaml to C Jeremy Yallop writes: > On 3 February 2016 at 12:15, Malcolm Matalka wrote: >> Jeremy Yallop writes: >> >>> On 3 February 2016 at 05:44, David Sheets wrote: >>>> On Wed, Feb 3, 2016 at 12:26 PM, Malcolm Matalka wrote: >>>>> Jeremie Dimino writes: >>>>>> You need to register [ml_t], [ml_x] and [ml_g >>>>>> ] as GC roots. Otherwise if the GC runs in caml_ba_alloc for instance, >>>>>> [ml_t] might ends up containing garbage even before reaching >>>>>> [caml_callback3]. You can use the normal macros for that: >>>>>> >>>>> If one is using ctypes, is all of this taken care of? I have a library >>>>> that registers a bunch of Ocaml functions in C code, which the C code >>>>> calls. I haven't experienced anything bad happening yet, but that >>>>> doesn't mean much... >>>> >>>> If you use ctypes and pass OCaml closures to C, you *must* retain a >>>> reference to the closure to avoid it being GCed. If you do not, you >>>> may experience the exception CallToExpiredClosure sporadically. >>> >>> Besides David's caveat, the answer is yes: ctypes will take care of >>> registering arguments as GC roots as necessary. >> >> Can you clarify this a bit? I'm not that familiar with how the C FFI >> works. If I pass in a closure to a C function and it is registered as a >> GC root, doesn't that mean it won't be GCd if my Ocaml program forgets >> about it or? > > That's how roots behave, yes: while a value is registered as a root, > the value won't be collected. There are (roughly speaking) two types > of root in OCaml: local roots, which persist for the duration of a > function call, and global roots, which persist until explicitly > released. A C function binding written by hand must ensure that OCaml > values passed to it as arguments are registered as local roots, so > that if a collection occurs while the function is running the values > won't be prematurely collected. > > A C binding written using ctypes can generally ignore the matter of > roots. That's partly because ctypes takes care of root registration, > but also because most types passed between OCaml and C in a ctypes > binding are C values, not OCaml values. For example, if you want to > pass a structure with several fields between OCaml and C there are two > approaches. One approach is to represent the structure as an OCaml > record, which involves accessing the fields of the value in your C > binding using various macros, taking care to register values as roots > to protect them from the GC. The other approach is to represent the > structure as a C struct, which involves accessing its fields in OCaml > using the functions ctypes provides. (If you enjoy programming in an > untyped dialect of C with ubiquitous concurrency, you'll probably > favour the first approach. If you prefer programming in OCaml then > the second approach might have some appeal.) > > Using the C value representation for values that cross the C-OCaml > boundary generally works well, but when things become higher-order, > the situation changes a bit. When a C library expects to be given a > first-order value such as a struct we have to give it a struct with > the appropriate layout, since C functions can directly access the > representation of values. However, when the library expects a > function pointer we have a bit more freedom, since the representation > of functions isn't accessible -- in fact, the only thing that can be > done with a function pointer, besides passing it from place to place, > is calling it. This freedom means that we can pass an OCaml function, > suitably packaged up, where a C function pointer is expected. > > Passing OCaml functions to C as function pointers raises some > interesting issues relating to object lifetime and the garbage > collector. The main difficulty arises from the fact that once you > pass a function pointer to a C library there's no way of knowing how > long the library holds on to it: for example, the library might > discard the function pointer when the call into the library returns, > or it might store the function pointer in a global variable to be > retrieved and called later. In order to prevent the associated > function from being collected prematurely, some kind of action is > needed on the OCaml side, whether registering a global root, or > ensuring that the function is reachable from the OCaml program. > >> Also, David and I were talking about how to solve this on IRC. In my >> specific case, callbacks are one-shot, which means I know they need to >> be remembered until they are called then they can (possibly) be freed. >> Is there a nice solution here? I'd prefer not to store them in some >> other data structure and remove them later just to keep a reference >> alive, if possible. > > Storing some kind of references to the functions in a place that the > collector can see is essential to prevent the functions from being > collected prematurely. The situation is the same whether you use > ctypes or write bindings by hand. > > Storing the functions in a table, and removing them automatically > after they're called is one approach. An alternative is to use the > new Ctypes.Roots module, which will be available in the next release: > > https://github.com/ocamllabs/ocaml-ctypes/blob/182a9e64src/ctypes/ctypes.mli#L419-L435 Thank you for the thorough response. It seems like Ctypes.Roots might solve my problem, although the URL gives me a 404. Do you have an estimation of when this will be released (or anything someone like myself can do to help?) > >> That is overhead I'd prefer to avoid, if possible. >> I plan on having possibly hundreds of thousands of these callbacks alive >> at any point in time. > > In that case it sounds like there'll be an overhead of up to a few megabytes. Any suggestions for a datatype to use here? I do have an object that is long lived that represents the event loop I'm integrating against, so I can store anything I want in there. Last night I was really concerned about storing this extra information in the loop, just seemed like a waste, but in the morning light I'm less worried about it. I could just use a Hashtbl I guess with some reference to the closure. My current idea is to make some integer value and wrap the closure up in something like: (fun () -> Hashtbl.remove t id; closure ()) What kind of sucks about that is the wrapper needs to be unique to each type of closure that gets called, there doesn't seem like a really generic way to do this wrapping. Am I on the wrong track? Thanks again, /Malcolm