From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id CA5957FB1E for ; Thu, 4 Dec 2014 17:39:48 +0100 (CET) Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of kennethadammiller@gmail.com) identity=pra; client-ip=209.85.218.43; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="kennethadammiller@gmail.com"; x-sender="kennethadammiller@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of kennethadammiller@gmail.com designates 209.85.218.43 as permitted sender) identity=mailfrom; client-ip=209.85.218.43; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="kennethadammiller@gmail.com"; x-sender="kennethadammiller@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-oi0-f43.google.com) identity=helo; client-ip=209.85.218.43; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="kennethadammiller@gmail.com"; x-sender="postmaster@mail-oi0-f43.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av4BAPGNgFTRVdorm2dsb2JhbABZg1hYBIMBw0+GFgKBFgcWAQEBAQERAQEBAQEGCwsJFC6EAgEBAQMBEhEdARsRDQMBCwYDAgsNJwMCAiEBAREBBQELEQYTIogGAQMJCQ2jRZBNPTGLMIFsgweLOAoZJw1ZhQUBAQEBBgEBAQEBFwEBBA6NIYEBgjILgnGBUQWKU4lTgiKBYz2BcoEjEIxIgjSBdhIlgRCCCYI6ITCCRQEBAQ X-IPAS-Result: Av4BAPGNgFTRVdorm2dsb2JhbABZg1hYBIMBw0+GFgKBFgcWAQEBAQERAQEBAQEGCwsJFC6EAgEBAQMBEhEdARsRDQMBCwYDAgsNJwMCAiEBAREBBQELEQYTIogGAQMJCQ2jRZBNPTGLMIFsgweLOAoZJw1ZhQUBAQEBBgEBAQEBFwEBBA6NIYEBgjILgnGBUQWKU4lTgiKBYz2BcoEjEIxIgjSBdhIlgRCCCYI6ITCCRQEBAQ X-IronPort-AV: E=Sophos;i="5.07,516,1413237600"; d="scan'208";a="91684548" Received: from mail-oi0-f43.google.com ([209.85.218.43]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/RC4-SHA; 04 Dec 2014 17:39:29 +0100 Received: by mail-oi0-f43.google.com with SMTP id a3so12699280oib.30 for ; Thu, 04 Dec 2014 08:39:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=8X7AIwiSTsKBGFzoR7kTKnrHQdipoB3ZVfFSs2uw4Bo=; b=cy4/4iidfgQUSCOAZbuZTyE5c+nnEN+wRsdxc6sOlWiXFkmoCaRLxqgrIoWVC2J+E7 NUalifXc9QI5yB48DTYzWU90VFUqfeFOFP8uT9uTVgr0h9A6KLCWVyYzYxQZUPWytHhW dvQ0uathNWrVuMtkBO5AgIeQMijfurM0MLr7EJiGVHBedrudMqI3oTnp+LHZXcv07oay wEJEXZuCFngj0Yl/MNLKR2aTM5zYUEpGT63MWuuYS8/36IKeHncF253R6tDqOZuCfJe+ oBjxjdDXdpvvgmoWkr0zJ5n5pow6ytrUcVeRMSc+V1jy6d7SUCNiY428fcHtGGZw8vO/ 0QUQ== MIME-Version: 1.0 X-Received: by 10.202.69.137 with SMTP id s131mr2010531oia.103.1417711168200; Thu, 04 Dec 2014 08:39:28 -0800 (PST) Received: by 10.182.197.1 with HTTP; Thu, 4 Dec 2014 08:39:28 -0800 (PST) In-Reply-To: References: <54801373.3010506@fugmann.net> <5480309B.3020402@fugmann.net> Date: Thu, 4 Dec 2014 11:39:28 -0500 Message-ID: From: Kenneth Adam Miller To: caml users Content-Type: multipart/alternative; boundary=001a113dae32932a600509669af3 Subject: Re: [Caml-list] Potential OCaml-ZMQ memory management problems --001a113dae32932a600509669af3 Content-Type: text/plain; charset=UTF-8 ok, I tried it with Gc.major() and Gc.minor(); and that didn't eliminate the segfault. There's an issue with debugging too; when I compile my unit tests for debugging or byte code, it reports parameters not being found (as in the string returned from ZMQ is empty), even though a simple recompile with native code makes that specific mysterious error go away. In any case, because it does not receive the parameters while compiled for debug or byte code, it cannot proceed to the point where the segfault occurs. Also, just removing some of the preceeding tests in order that they not be in the way when compiled for byte or debug code causes the segfault to not occur... How do I debug this in binary mode? On Thu, Dec 4, 2014 at 5:04 AM, Kenneth Adam Miller < kennethadammiller@gmail.com> wrote: > Ok I'll give that a shot when I get a chance to edit it. Thanks, > gc.major... I'm new to ocaml and I'm still reading through RWO, on chapter > 9 learning about how modules compose nicely :) > > On Thu, Dec 4, 2014 at 4:59 AM, Anders Fugmann wrote: > >> The garbage collector settings seems quite sane to me. You could try to >> force a major collection every call to iltrans. >> >> Gc.major () >> >> It could be that the segfault is really an out of memory situation. Maybe >> you have a leak in your code somewhere (Not releasing references to large >> objects). >> >> /Anders >> >> >> On 12/04/2014 09:02 AM, Kenneth Adam Miller wrote: >> >>> I'm using zmq and ocaml-zmq as installed by opam; I'm not at my work >>> computer or I would provide those details. >>> >>> Yes, I will try to work on creating a reproducible script >>> >>> Could it possibly be a call to Tunegc.set_gc () from >>> https://github.com/argp/bap/blob/master/ocaml/tunegc.ml ? >>> >>> I notice that this error didn't appear in single calls to my utility; >>> basically, I'm using iltrans from bap and I'm using it in a long living >>> process, unlike the current implementation which expects a few >>> transformations and then process death. I did notice that when I run it, >>> if I'm watching system monitor that it begins consuming vast amounts of >>> memory (quickly grows from a few megabytes to 800+) before it hits >>> segfault. Would there be any way to restore the aggressiveness of the GC >>> between calls to iltrans? >>> >>> On Thu, Dec 4, 2014 at 2:55 AM, Anders Fugmann >> > wrote: >>> >>> Hi Kenneth, >>> >>> The Ocaml-zmq code copies data from received messages into memory >>> under >>> the control of the ocaml garbage collector, and immediatly frees the >>> ZMQ buffers. >>> >>> You do not need to explicitly free the data received from call to >>> recv. >>> >>> Can you produce a small sample code that exposes the problem? We are >>> using the ocaml-zmq binding extensively, and have not seen any >>> problems. >>> >>> If I were to take a wild guess, it either a mismatch between library >>> versions or the garbage-collector collecting the socket or zmq >>> context. >>> >>> What version of the ocaml-zmq bindings and libzmq (c impl) are you >>> using? >>> >>> /Anders >>> >>> >>> >>> On 12/04/2014 07:09 AM, Kenneth Adam Miller wrote: >>> >>> I'm using ocaml-zmq (https://github.com/issuu/__ocaml-zmq >>> ) and I think I'm >>> encountering a memory management issue. I could be wrong >>> however, but >>> basically the issue (I think) is I have a rather large set of >>> messages >>> to send via zmq, and I'm getting a segfault. >>> >>> Does anybody know if I need to free the strings received from >>> zmq recv >>> functions in ocaml? If so how do I do that from ocaml? >>> >>> There's no code because this is just a general novice ocaml >>> questions. >>> >>> >>> >>> >> > --001a113dae32932a600509669af3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
ok, I tried it with Gc.major() and Gc.minor(); and that di= dn't eliminate the segfault.

There's an issue wi= th debugging too; when I compile my unit tests for debugging or byte code, = it reports parameters not being found (as in the string returned from ZMQ i= s empty), even though a simple recompile with native code makes that specif= ic mysterious error go away. In any case, because it does not receive the p= arameters while compiled for debug or byte code, it cannot proceed to the p= oint where the segfault occurs. Also, just removing some of the preceeding = tests in order that they not be in the way when compiled for byte or debug = code causes the segfault to not occur...

How do I de= bug this in binary mode?=C2=A0
=
On Thu, Dec 4, 2014 at 5:04 AM, Kenneth Adam= Miller <kennethadammiller@gmail.com> wrote:
Ok I'll give that a shot = when I get a chance to edit it. Thanks, gc.major... I'm new to ocaml an= d I'm still reading through RWO, on chapter 9 learning about how module= s compose nicely :)

On Thu, Dec 4, 2014 at 4:59= AM, Anders Fugmann <anders@fugmann.net> wrote:
The garbage collector settings seems quite sane to m= e. You could try to force a major collection every call to iltrans.

Gc.major ()

It could be that the segfault is really an out of memory situation. Maybe y= ou have a leak in your code somewhere (Not releasing references to large ob= jects).

/Anders


On 12/04/2014 09:02 AM, Kenneth Adam Miller wrote:
I'm using zmq and ocaml-zmq as installed by opam; I'm not at my wor= k
computer or I would provide those details.

Yes, I will try to work on creating a reproducible script

Could it possibly be a call to Tunegc.set_gc () from
https://github.com/argp/bap/blob/master/ocaml/tunegc.ml<= /a> ?

I notice that this error didn't appear in single calls to my utility; basically, I'm using iltrans from bap and I'm using it in a long li= ving
process, unlike the current implementation which expects a few
transformations and then process death. I did notice that when I run it,
if I'm watching system monitor that it begins consuming vast amounts of=
memory (quickly grows from a few megabytes to 800+) before it hits
segfault. Would there be any way to restore the aggressiveness of the GC
between calls to iltrans?

On Thu, Dec 4, 2014 at 2:55 AM, Anders Fugmann <anders@fugmann.net
 <mailto:anders@f= ugmann.net>> wrote:

=C2=A0 =C2=A0 Hi Kenneth,

=C2=A0 =C2=A0 The Ocaml-zmq code copies data from received messages into me= mory under
=C2=A0 =C2=A0 the control of the ocaml garbage collector, and immediatly fr= ees the
=C2=A0 =C2=A0 ZMQ buffers.

=C2=A0 =C2=A0 You do not need to explicitly free the data received from cal= l to recv.

=C2=A0 =C2=A0 Can you produce a small sample code that exposes the problem?= We are
=C2=A0 =C2=A0 using the ocaml-zmq binding extensively, and have not seen an= y problems.

=C2=A0 =C2=A0 If I were to take a wild guess, it either a mismatch between = library
=C2=A0 =C2=A0 versions or the garbage-collector collecting the socket or zm= q context.

=C2=A0 =C2=A0 What version of the ocaml-zmq bindings and libzmq (c impl) ar= e you
=C2=A0 =C2=A0 using?

=C2=A0 =C2=A0 /Anders



=C2=A0 =C2=A0 On 12/04/2014 07:09 AM, Kenneth Adam Miller wrote:

 =C2=A0 =C2=A0 =C2=A0 =C2=A0 I'm using ocaml-zmq (https://github.com/issuu/__= ocaml-zmq
=C2=A0 =C2=A0 =C2=A0 =C2=A0 <https://github.com/issuu/ocaml-zmq>) an= d I think I'm
=C2=A0 =C2=A0 =C2=A0 =C2=A0 encountering a memory management issue. I could= be wrong
=C2=A0 =C2=A0 =C2=A0 =C2=A0 however, but
=C2=A0 =C2=A0 =C2=A0 =C2=A0 basically the issue (I think) is I have a rathe= r large set of
=C2=A0 =C2=A0 =C2=A0 =C2=A0 messages
=C2=A0 =C2=A0 =C2=A0 =C2=A0 to send via zmq, and I'm getting a segfault= .

=C2=A0 =C2=A0 =C2=A0 =C2=A0 Does anybody know if I need to free the strings= received from
=C2=A0 =C2=A0 =C2=A0 =C2=A0 zmq recv
=C2=A0 =C2=A0 =C2=A0 =C2=A0 functions in ocaml? If so how do I do that from= ocaml?

=C2=A0 =C2=A0 =C2=A0 =C2=A0 There's no code because this is just a gene= ral novice ocaml
=C2=A0 =C2=A0 =C2=A0 =C2=A0 questions.






--001a113dae32932a600509669af3--