[Caml-list] Potential OCaml-ZMQ memory management problems

[Caml-list] Potential OCaml-ZMQ memory management problems

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 458 bytes --]

I'm using ocaml-zmq (https://github.com/issuu/ocaml-zmq) and I think I'm
encountering a memory management issue. I could be wrong however, but
basically the issue (I think) is I have a rather large set of messages to
send via zmq, and I'm getting a segfault.

Does anybody know if I need to free the strings received from zmq recv
functions in ocaml? If so how do I do that from ocaml?

There's no code because this is just a general novice ocaml questions.

[-- Attachment #2: Type: text/html, Size: 1059 bytes --]

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Anders Fugmann]

Hi Kenneth,

The Ocaml-zmq code copies data from received messages into memory under
the control of the ocaml garbage collector, and immediatly frees the ZMQ 
buffers.

You do not need to explicitly free the data received from call to recv.

Can you produce a small sample code that exposes the problem? We are 
using the ocaml-zmq binding extensively, and have not seen any problems.

If I were to take a wild guess, it either a mismatch between library 
versions or the garbage-collector collecting the socket or zmq context.

What version of the ocaml-zmq bindings and libzmq (c impl) are you using?

/Anders


On 12/04/2014 07:09 AM, Kenneth Adam Miller wrote:
> I'm using ocaml-zmq (https://github.com/issuu/ocaml-zmq) and I think I'm
> encountering a memory management issue. I could be wrong however, but
> basically the issue (I think) is I have a rather large set of messages
> to send via zmq, and I'm getting a segfault.
>
> Does anybody know if I need to free the strings received from zmq recv
> functions in ocaml? If so how do I do that from ocaml?
>
> There's no code because this is just a general novice ocaml questions.

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2065 bytes --]

I'm using zmq and ocaml-zmq as installed by opam; I'm not at my work
computer or I would provide those details.

Yes, I will try to work on creating a reproducible script

Could it possibly be a call to Tunegc.set_gc () from
https://github.com/argp/bap/blob/master/ocaml/tunegc.ml ?

I notice that this error didn't appear in single calls to my utility;
basically, I'm using iltrans from bap and I'm using it in a long living
process, unlike the current implementation which expects a few
transformations and then process death. I did notice that when I run it, if
I'm watching system monitor that it begins consuming vast amounts of memory
(quickly grows from a few megabytes to 800+) before it hits segfault. Would
there be any way to restore the aggressiveness of the GC between calls to
iltrans?

On Thu, Dec 4, 2014 at 2:55 AM, Anders Fugmann <anders@fugmann.net> wrote:

> Hi Kenneth,
>
> The Ocaml-zmq code copies data from received messages into memory under
> the control of the ocaml garbage collector, and immediatly frees the ZMQ
> buffers.
>
> You do not need to explicitly free the data received from call to recv.
>
> Can you produce a small sample code that exposes the problem? We are using
> the ocaml-zmq binding extensively, and have not seen any problems.
>
> If I were to take a wild guess, it either a mismatch between library
> versions or the garbage-collector collecting the socket or zmq context.
>
> What version of the ocaml-zmq bindings and libzmq (c impl) are you using?
>
> /Anders
>
>
>
> On 12/04/2014 07:09 AM, Kenneth Adam Miller wrote:
>
>> I'm using ocaml-zmq (https://github.com/issuu/ocaml-zmq) and I think I'm
>> encountering a memory management issue. I could be wrong however, but
>> basically the issue (I think) is I have a rather large set of messages
>> to send via zmq, and I'm getting a segfault.
>>
>> Does anybody know if I need to free the strings received from zmq recv
>> functions in ocaml? If so how do I do that from ocaml?
>>
>> There's no code because this is just a general novice ocaml questions.
>>
>
>

[-- Attachment #2: Type: text/html, Size: 2905 bytes --]

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Anders Fugmann]

The garbage collector settings seems quite sane to me. You could try to 
force a major collection every call to iltrans.

Gc.major ()

It could be that the segfault is really an out of memory situation. 
Maybe you have a leak in your code somewhere (Not releasing references 
to large objects).

/Anders


On 12/04/2014 09:02 AM, Kenneth Adam Miller wrote:
> I'm using zmq and ocaml-zmq as installed by opam; I'm not at my work
> computer or I would provide those details.
>
> Yes, I will try to work on creating a reproducible script
>
> Could it possibly be a call to Tunegc.set_gc () from
> https://github.com/argp/bap/blob/master/ocaml/tunegc.ml ?
>
> I notice that this error didn't appear in single calls to my utility;
> basically, I'm using iltrans from bap and I'm using it in a long living
> process, unlike the current implementation which expects a few
> transformations and then process death. I did notice that when I run it,
> if I'm watching system monitor that it begins consuming vast amounts of
> memory (quickly grows from a few megabytes to 800+) before it hits
> segfault. Would there be any way to restore the aggressiveness of the GC
> between calls to iltrans?
>
> On Thu, Dec 4, 2014 at 2:55 AM, Anders Fugmann <anders@fugmann.net
> <mailto:anders@fugmann.net>> wrote:
>
>     Hi Kenneth,
>
>     The Ocaml-zmq code copies data from received messages into memory under
>     the control of the ocaml garbage collector, and immediatly frees the
>     ZMQ buffers.
>
>     You do not need to explicitly free the data received from call to recv.
>
>     Can you produce a small sample code that exposes the problem? We are
>     using the ocaml-zmq binding extensively, and have not seen any problems.
>
>     If I were to take a wild guess, it either a mismatch between library
>     versions or the garbage-collector collecting the socket or zmq context.
>
>     What version of the ocaml-zmq bindings and libzmq (c impl) are you
>     using?
>
>     /Anders
>
>
>
>     On 12/04/2014 07:09 AM, Kenneth Adam Miller wrote:
>
>         I'm using ocaml-zmq (https://github.com/issuu/__ocaml-zmq
>         <https://github.com/issuu/ocaml-zmq>) and I think I'm
>         encountering a memory management issue. I could be wrong
>         however, but
>         basically the issue (I think) is I have a rather large set of
>         messages
>         to send via zmq, and I'm getting a segfault.
>
>         Does anybody know if I need to free the strings received from
>         zmq recv
>         functions in ocaml? If so how do I do that from ocaml?
>
>         There's no code because this is just a general novice ocaml
>         questions.
>
>
>

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 3035 bytes --]

Ok I'll give that a shot when I get a chance to edit it. Thanks,
gc.major... I'm new to ocaml and I'm still reading through RWO, on chapter
9 learning about how modules compose nicely :)

On Thu, Dec 4, 2014 at 4:59 AM, Anders Fugmann <anders@fugmann.net> wrote:

> The garbage collector settings seems quite sane to me. You could try to
> force a major collection every call to iltrans.
>
> Gc.major ()
>
> It could be that the segfault is really an out of memory situation. Maybe
> you have a leak in your code somewhere (Not releasing references to large
> objects).
>
> /Anders
>
>
> On 12/04/2014 09:02 AM, Kenneth Adam Miller wrote:
>
>> I'm using zmq and ocaml-zmq as installed by opam; I'm not at my work
>> computer or I would provide those details.
>>
>> Yes, I will try to work on creating a reproducible script
>>
>> Could it possibly be a call to Tunegc.set_gc () from
>> https://github.com/argp/bap/blob/master/ocaml/tunegc.ml ?
>>
>> I notice that this error didn't appear in single calls to my utility;
>> basically, I'm using iltrans from bap and I'm using it in a long living
>> process, unlike the current implementation which expects a few
>> transformations and then process death. I did notice that when I run it,
>> if I'm watching system monitor that it begins consuming vast amounts of
>> memory (quickly grows from a few megabytes to 800+) before it hits
>> segfault. Would there be any way to restore the aggressiveness of the GC
>> between calls to iltrans?
>>
>> On Thu, Dec 4, 2014 at 2:55 AM, Anders Fugmann <anders@fugmann.net
>> <mailto:anders@fugmann.net>> wrote:
>>
>>     Hi Kenneth,
>>
>>     The Ocaml-zmq code copies data from received messages into memory
>> under
>>     the control of the ocaml garbage collector, and immediatly frees the
>>     ZMQ buffers.
>>
>>     You do not need to explicitly free the data received from call to
>> recv.
>>
>>     Can you produce a small sample code that exposes the problem? We are
>>     using the ocaml-zmq binding extensively, and have not seen any
>> problems.
>>
>>     If I were to take a wild guess, it either a mismatch between library
>>     versions or the garbage-collector collecting the socket or zmq
>> context.
>>
>>     What version of the ocaml-zmq bindings and libzmq (c impl) are you
>>     using?
>>
>>     /Anders
>>
>>
>>
>>     On 12/04/2014 07:09 AM, Kenneth Adam Miller wrote:
>>
>>         I'm using ocaml-zmq (https://github.com/issuu/__ocaml-zmq
>>         <https://github.com/issuu/ocaml-zmq>) and I think I'm
>>         encountering a memory management issue. I could be wrong
>>         however, but
>>         basically the issue (I think) is I have a rather large set of
>>         messages
>>         to send via zmq, and I'm getting a segfault.
>>
>>         Does anybody know if I need to free the strings received from
>>         zmq recv
>>         functions in ocaml? If so how do I do that from ocaml?
>>
>>         There's no code because this is just a general novice ocaml
>>         questions.
>>
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4186 bytes --]

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 3929 bytes --]

ok, I tried it with Gc.major() and Gc.minor(); and that didn't eliminate
the segfault.

There's an issue with debugging too; when I compile my unit tests for
debugging or byte code, it reports parameters not being found (as in the
string returned from ZMQ is empty), even though a simple recompile with
native code makes that specific mysterious error go away. In any case,
because it does not receive the parameters while compiled for debug or byte
code, it cannot proceed to the point where the segfault occurs. Also, just
removing some of the preceeding tests in order that they not be in the way
when compiled for byte or debug code causes the segfault to not occur...

How do I debug this in binary mode?

On Thu, Dec 4, 2014 at 5:04 AM, Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> Ok I'll give that a shot when I get a chance to edit it. Thanks,
> gc.major... I'm new to ocaml and I'm still reading through RWO, on chapter
> 9 learning about how modules compose nicely :)
>
> On Thu, Dec 4, 2014 at 4:59 AM, Anders Fugmann <anders@fugmann.net> wrote:
>
>> The garbage collector settings seems quite sane to me. You could try to
>> force a major collection every call to iltrans.
>>
>> Gc.major ()
>>
>> It could be that the segfault is really an out of memory situation. Maybe
>> you have a leak in your code somewhere (Not releasing references to large
>> objects).
>>
>> /Anders
>>
>>
>> On 12/04/2014 09:02 AM, Kenneth Adam Miller wrote:
>>
>>> I'm using zmq and ocaml-zmq as installed by opam; I'm not at my work
>>> computer or I would provide those details.
>>>
>>> Yes, I will try to work on creating a reproducible script
>>>
>>> Could it possibly be a call to Tunegc.set_gc () from
>>> https://github.com/argp/bap/blob/master/ocaml/tunegc.ml ?
>>>
>>> I notice that this error didn't appear in single calls to my utility;
>>> basically, I'm using iltrans from bap and I'm using it in a long living
>>> process, unlike the current implementation which expects a few
>>> transformations and then process death. I did notice that when I run it,
>>> if I'm watching system monitor that it begins consuming vast amounts of
>>> memory (quickly grows from a few megabytes to 800+) before it hits
>>> segfault. Would there be any way to restore the aggressiveness of the GC
>>> between calls to iltrans?
>>>
>>> On Thu, Dec 4, 2014 at 2:55 AM, Anders Fugmann <anders@fugmann.net
>>> <mailto:anders@fugmann.net>> wrote:
>>>
>>>     Hi Kenneth,
>>>
>>>     The Ocaml-zmq code copies data from received messages into memory
>>> under
>>>     the control of the ocaml garbage collector, and immediatly frees the
>>>     ZMQ buffers.
>>>
>>>     You do not need to explicitly free the data received from call to
>>> recv.
>>>
>>>     Can you produce a small sample code that exposes the problem? We are
>>>     using the ocaml-zmq binding extensively, and have not seen any
>>> problems.
>>>
>>>     If I were to take a wild guess, it either a mismatch between library
>>>     versions or the garbage-collector collecting the socket or zmq
>>> context.
>>>
>>>     What version of the ocaml-zmq bindings and libzmq (c impl) are you
>>>     using?
>>>
>>>     /Anders
>>>
>>>
>>>
>>>     On 12/04/2014 07:09 AM, Kenneth Adam Miller wrote:
>>>
>>>         I'm using ocaml-zmq (https://github.com/issuu/__ocaml-zmq
>>>         <https://github.com/issuu/ocaml-zmq>) and I think I'm
>>>         encountering a memory management issue. I could be wrong
>>>         however, but
>>>         basically the issue (I think) is I have a rather large set of
>>>         messages
>>>         to send via zmq, and I'm getting a segfault.
>>>
>>>         Does anybody know if I need to free the strings received from
>>>         zmq recv
>>>         functions in ocaml? If so how do I do that from ocaml?
>>>
>>>         There's no code because this is just a general novice ocaml
>>>         questions.
>>>
>>>
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 5363 bytes --]

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 4281 bytes --]

Actually, I literally just found the location of the error... I'm trying to
send some data from ocaml that is causing ZMQ to segfault...

On Thu, Dec 4, 2014 at 11:39 AM, Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> ok, I tried it with Gc.major() and Gc.minor(); and that didn't eliminate
> the segfault.
>
> There's an issue with debugging too; when I compile my unit tests for
> debugging or byte code, it reports parameters not being found (as in the
> string returned from ZMQ is empty), even though a simple recompile with
> native code makes that specific mysterious error go away. In any case,
> because it does not receive the parameters while compiled for debug or byte
> code, it cannot proceed to the point where the segfault occurs. Also, just
> removing some of the preceeding tests in order that they not be in the way
> when compiled for byte or debug code causes the segfault to not occur...
>
> How do I debug this in binary mode?
>
> On Thu, Dec 4, 2014 at 5:04 AM, Kenneth Adam Miller <
> kennethadammiller@gmail.com> wrote:
>
>> Ok I'll give that a shot when I get a chance to edit it. Thanks,
>> gc.major... I'm new to ocaml and I'm still reading through RWO, on chapter
>> 9 learning about how modules compose nicely :)
>>
>> On Thu, Dec 4, 2014 at 4:59 AM, Anders Fugmann <anders@fugmann.net>
>> wrote:
>>
>>> The garbage collector settings seems quite sane to me. You could try to
>>> force a major collection every call to iltrans.
>>>
>>> Gc.major ()
>>>
>>> It could be that the segfault is really an out of memory situation.
>>> Maybe you have a leak in your code somewhere (Not releasing references to
>>> large objects).
>>>
>>> /Anders
>>>
>>>
>>> On 12/04/2014 09:02 AM, Kenneth Adam Miller wrote:
>>>
>>>> I'm using zmq and ocaml-zmq as installed by opam; I'm not at my work
>>>> computer or I would provide those details.
>>>>
>>>> Yes, I will try to work on creating a reproducible script
>>>>
>>>> Could it possibly be a call to Tunegc.set_gc () from
>>>> https://github.com/argp/bap/blob/master/ocaml/tunegc.ml ?
>>>>
>>>> I notice that this error didn't appear in single calls to my utility;
>>>> basically, I'm using iltrans from bap and I'm using it in a long living
>>>> process, unlike the current implementation which expects a few
>>>> transformations and then process death. I did notice that when I run it,
>>>> if I'm watching system monitor that it begins consuming vast amounts of
>>>> memory (quickly grows from a few megabytes to 800+) before it hits
>>>> segfault. Would there be any way to restore the aggressiveness of the GC
>>>> between calls to iltrans?
>>>>
>>>> On Thu, Dec 4, 2014 at 2:55 AM, Anders Fugmann <anders@fugmann.net
>>>> <mailto:anders@fugmann.net>> wrote:
>>>>
>>>>     Hi Kenneth,
>>>>
>>>>     The Ocaml-zmq code copies data from received messages into memory
>>>> under
>>>>     the control of the ocaml garbage collector, and immediatly frees the
>>>>     ZMQ buffers.
>>>>
>>>>     You do not need to explicitly free the data received from call to
>>>> recv.
>>>>
>>>>     Can you produce a small sample code that exposes the problem? We are
>>>>     using the ocaml-zmq binding extensively, and have not seen any
>>>> problems.
>>>>
>>>>     If I were to take a wild guess, it either a mismatch between library
>>>>     versions or the garbage-collector collecting the socket or zmq
>>>> context.
>>>>
>>>>     What version of the ocaml-zmq bindings and libzmq (c impl) are you
>>>>     using?
>>>>
>>>>     /Anders
>>>>
>>>>
>>>>
>>>>     On 12/04/2014 07:09 AM, Kenneth Adam Miller wrote:
>>>>
>>>>         I'm using ocaml-zmq (https://github.com/issuu/__ocaml-zmq
>>>>         <https://github.com/issuu/ocaml-zmq>) and I think I'm
>>>>         encountering a memory management issue. I could be wrong
>>>>         however, but
>>>>         basically the issue (I think) is I have a rather large set of
>>>>         messages
>>>>         to send via zmq, and I'm getting a segfault.
>>>>
>>>>         Does anybody know if I need to free the strings received from
>>>>         zmq recv
>>>>         functions in ocaml? If so how do I do that from ocaml?
>>>>
>>>>         There's no code because this is just a general novice ocaml
>>>>         questions.
>>>>
>>>>
>>>>
>>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 5920 bytes --]

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Anders Peter Fugmann]

Sending data over a zmq socket should not cause a segfault.

I suggest that you open an issue on
https://github.com/issuu/ocaml-zmq
with exact details of the segfault and we can take a stab at it.

/Anders

On 04/12/14 17:45, Kenneth Adam Miller wrote:
> Actually, I literally just found the location of the error... I'm trying
> to send some data from ocaml that is causing ZMQ to segfault...

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 1474 bytes --]

Well I am just no thorough and you are correct.

The sending of data over a zmq socket and the conversion of that data from
string to protobuf encoded string all occurred in one line. One I added a
print statement and then segregated them more cleanly, I can see that it is
most certainly the line that converts to protobuf.

The exact function that fails (on my end, could be deeper within this) is
to_pb from here:

https://github.com/argp/bap/blob/master/ocaml/piqi/ast_piqi.ml#L186

In any case, I did a test, and in my first function when to_pb gets called
the first time and succeeds, I added an additional call to it... which also
succeeded. But then in a subsequent unit test, the one that has been
failing, still segfaults.

If I turn off the tests prior to the segfaulting test, to_pb works in this
particular run. But if the tests run before hand, something goes awry
between the tests. Is it possible that to_pb is using some shared state
between calls?


On Thu, Dec 4, 2014 at 2:36 PM, Anders Peter Fugmann <anders@fugmann.net>
wrote:

> Sending data over a zmq socket should not cause a segfault.
>
> I suggest that you open an issue on
> https://github.com/issuu/ocaml-zmq
> with exact details of the segfault and we can take a stab at it.
>
> /Anders
>
>
> On 04/12/14 17:45, Kenneth Adam Miller wrote:
>
>> Actually, I literally just found the location of the error... I'm trying
>> to send some data from ocaml that is causing ZMQ to segfault...
>>
>
>
>

[-- Attachment #2: Type: text/html, Size: 2418 bytes --]

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Anders Fugmann]

On 12/04/2014 10:48 PM, Kenneth Adam Miller wrote:
> Well I am just no thorough and you are correct.
>
> The sending of data over a zmq socket and the conversion of that data
> from string to protobuf encoded string all occurred in one line. One I
> added a print statement and then segregated them more cleanly, I can see
> that it is most certainly the line that converts to protobuf.
>
> The exact function that fails (on my end, could be deeper within this)
> is to_pb from here:
>
> https://github.com/argp/bap/blob/master/ocaml/piqi/ast_piqi.ml#L186
>
> In any case, I did a test, and in my first function when to_pb gets
> called the first time and succeeds, I added an additional call to it...
> which also succeeded. But then in a subsequent unit test, the one that
> has been failing, still segfaults.
>
> If I turn off the tests prior to the segfaulting test, to_pb works in
> this particular run. But if the tests run before hand, something goes
> awry between the tests. Is it possible that to_pb is using some shared
> state between calls?

I would not expect so.

If you create a failing unittest that I could try?

Also, does the segfault contain a usable back trace (using gdb)? That 
might give some insights into which code is failing.

/Anders

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

Yes, I'll try and recreate it for you.

No, the backtrace in gdb is useless. All it says is:
#0  0x0000000000843033  in caml_c_call ()
#1  0x0000000000000000  in ?? ()

On Fri, Dec 5, 2014 at 4:14 AM, Anders Fugmann <anders@fugmann.net> wrote:

> On 12/04/2014 10:48 PM, Kenneth Adam Miller wrote:
>
>> Well I am just no thorough and you are correct.
>>
>> The sending of data over a zmq socket and the conversion of that data
>> from string to protobuf encoded string all occurred in one line. One I
>> added a print statement and then segregated them more cleanly, I can see
>> that it is most certainly the line that converts to protobuf.
>>
>> The exact function that fails (on my end, could be deeper within this)
>> is to_pb from here:
>>
>> https://github.com/argp/bap/blob/master/ocaml/piqi/ast_piqi.ml#L186
>>
>> In any case, I did a test, and in my first function when to_pb gets
>> called the first time and succeeds, I added an additional call to it...
>> which also succeeded. But then in a subsequent unit test, the one that
>> has been failing, still segfaults.
>>
>> If I turn off the tests prior to the segfaulting test, to_pb works in
>> this particular run. But if the tests run before hand, something goes
>> awry between the tests. Is it possible that to_pb is using some shared
>> state between calls?
>>
>
> I would not expect so.
>
> If you create a failing unittest that I could try?
>
> Also, does the segfault contain a usable back trace (using gdb)? That
> might give some insights into which code is failing.
>
> /Anders
>
>
>

[-- Attachment #2: Type: text/html, Size: 2288 bytes --]

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2315 bytes --]

While reproducing it, I found that in the bap/ocaml directory's input.ml,
there is a mutable list that is being updated by functors in speclist when
parse_argv or parse is called; it retains the old list between calls to my
function. So I need to reset it.
(line 6 at https://github.com/argp/bap/blob/master/ocaml/input.ml)

But now I get a strange compiler error! I don't know how ocaml could be
such a hard language to use...

Input.inputs:=ref [];

Error: Unbound value Input.inputs

But you can know that I have included the ocaml directory and linked it
correct, since using Input.get_program already worked...

On Fri, Dec 5, 2014 at 9:38 AM, Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> Yes, I'll try and recreate it for you.
>
> No, the backtrace in gdb is useless. All it says is:
> #0  0x0000000000843033  in caml_c_call ()
> #1  0x0000000000000000  in ?? ()
>
> On Fri, Dec 5, 2014 at 4:14 AM, Anders Fugmann <anders@fugmann.net> wrote:
>
>> On 12/04/2014 10:48 PM, Kenneth Adam Miller wrote:
>>
>>> Well I am just no thorough and you are correct.
>>>
>>> The sending of data over a zmq socket and the conversion of that data
>>> from string to protobuf encoded string all occurred in one line. One I
>>> added a print statement and then segregated them more cleanly, I can see
>>> that it is most certainly the line that converts to protobuf.
>>>
>>> The exact function that fails (on my end, could be deeper within this)
>>> is to_pb from here:
>>>
>>> https://github.com/argp/bap/blob/master/ocaml/piqi/ast_piqi.ml#L186
>>>
>>> In any case, I did a test, and in my first function when to_pb gets
>>> called the first time and succeeds, I added an additional call to it...
>>> which also succeeded. But then in a subsequent unit test, the one that
>>> has been failing, still segfaults.
>>>
>>> If I turn off the tests prior to the segfaulting test, to_pb works in
>>> this particular run. But if the tests run before hand, something goes
>>> awry between the tests. Is it possible that to_pb is using some shared
>>> state between calls?
>>>
>>
>> I would not expect so.
>>
>> If you create a failing unittest that I could try?
>>
>> Also, does the segfault contain a usable back trace (using gdb)? That
>> might give some insights into which code is failing.
>>
>> /Anders
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 3535 bytes --]

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Yotam Barnoy]

[-- Attachment #1: Type: text/plain, Size: 2536 bytes --]

You didn't export inputs in Input.mli.

-Yotam

On Mon, Dec 8, 2014 at 1:11 PM, Kenneth Adam Miller <
kennethadammiller@gmail.com> wrote:

> While reproducing it, I found that in the bap/ocaml directory's input.ml,
> there is a mutable list that is being updated by functors in speclist when
> parse_argv or parse is called; it retains the old list between calls to my
> function. So I need to reset it.
> (line 6 at https://github.com/argp/bap/blob/master/ocaml/input.ml)
>
> But now I get a strange compiler error! I don't know how ocaml could be
> such a hard language to use...
>
> Input.inputs:=ref [];
>
> Error: Unbound value Input.inputs
>
> But you can know that I have included the ocaml directory and linked it
> correct, since using Input.get_program already worked...
>
> On Fri, Dec 5, 2014 at 9:38 AM, Kenneth Adam Miller <
> kennethadammiller@gmail.com> wrote:
>
>> Yes, I'll try and recreate it for you.
>>
>> No, the backtrace in gdb is useless. All it says is:
>> #0  0x0000000000843033  in caml_c_call ()
>> #1  0x0000000000000000  in ?? ()
>>
>> On Fri, Dec 5, 2014 at 4:14 AM, Anders Fugmann <anders@fugmann.net>
>> wrote:
>>
>>> On 12/04/2014 10:48 PM, Kenneth Adam Miller wrote:
>>>
>>>> Well I am just no thorough and you are correct.
>>>>
>>>> The sending of data over a zmq socket and the conversion of that data
>>>> from string to protobuf encoded string all occurred in one line. One I
>>>> added a print statement and then segregated them more cleanly, I can see
>>>> that it is most certainly the line that converts to protobuf.
>>>>
>>>> The exact function that fails (on my end, could be deeper within this)
>>>> is to_pb from here:
>>>>
>>>> https://github.com/argp/bap/blob/master/ocaml/piqi/ast_piqi.ml#L186
>>>>
>>>> In any case, I did a test, and in my first function when to_pb gets
>>>> called the first time and succeeds, I added an additional call to it...
>>>> which also succeeded. But then in a subsequent unit test, the one that
>>>> has been failing, still segfaults.
>>>>
>>>> If I turn off the tests prior to the segfaulting test, to_pb works in
>>>> this particular run. But if the tests run before hand, something goes
>>>> awry between the tests. Is it possible that to_pb is using some shared
>>>> state between calls?
>>>>
>>>
>>> I would not expect so.
>>>
>>> If you create a failing unittest that I could try?
>>>
>>> Also, does the segfault contain a usable back trace (using gdb)? That
>>> might give some insights into which code is failing.
>>>
>>> /Anders
>>>
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4057 bytes --]

Re: [Caml-list] Potential OCaml-ZMQ memory management problems

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2919 bytes --]

Excellent call. I find it is better to just update inputs in get_program
and do a push request. It doesn't affect any existing program adversely and
will only help others in the instance they reuse the function as I am.

On Mon, Dec 8, 2014 at 1:16 PM, Yotam Barnoy <yotambarnoy@gmail.com> wrote:

> You didn't export inputs in Input.mli.
>
> -Yotam
>
> On Mon, Dec 8, 2014 at 1:11 PM, Kenneth Adam Miller <
> kennethadammiller@gmail.com> wrote:
>
>> While reproducing it, I found that in the bap/ocaml directory's input.ml,
>> there is a mutable list that is being updated by functors in speclist when
>> parse_argv or parse is called; it retains the old list between calls to my
>> function. So I need to reset it.
>> (line 6 at https://github.com/argp/bap/blob/master/ocaml/input.ml)
>>
>> But now I get a strange compiler error! I don't know how ocaml could be
>> such a hard language to use...
>>
>> Input.inputs:=ref [];
>>
>> Error: Unbound value Input.inputs
>>
>> But you can know that I have included the ocaml directory and linked it
>> correct, since using Input.get_program already worked...
>>
>> On Fri, Dec 5, 2014 at 9:38 AM, Kenneth Adam Miller <
>> kennethadammiller@gmail.com> wrote:
>>
>>> Yes, I'll try and recreate it for you.
>>>
>>> No, the backtrace in gdb is useless. All it says is:
>>> #0  0x0000000000843033  in caml_c_call ()
>>> #1  0x0000000000000000  in ?? ()
>>>
>>> On Fri, Dec 5, 2014 at 4:14 AM, Anders Fugmann <anders@fugmann.net>
>>> wrote:
>>>
>>>> On 12/04/2014 10:48 PM, Kenneth Adam Miller wrote:
>>>>
>>>>> Well I am just no thorough and you are correct.
>>>>>
>>>>> The sending of data over a zmq socket and the conversion of that data
>>>>> from string to protobuf encoded string all occurred in one line. One I
>>>>> added a print statement and then segregated them more cleanly, I can
>>>>> see
>>>>> that it is most certainly the line that converts to protobuf.
>>>>>
>>>>> The exact function that fails (on my end, could be deeper within this)
>>>>> is to_pb from here:
>>>>>
>>>>> https://github.com/argp/bap/blob/master/ocaml/piqi/ast_piqi.ml#L186
>>>>>
>>>>> In any case, I did a test, and in my first function when to_pb gets
>>>>> called the first time and succeeds, I added an additional call to it...
>>>>> which also succeeded. But then in a subsequent unit test, the one that
>>>>> has been failing, still segfaults.
>>>>>
>>>>> If I turn off the tests prior to the segfaulting test, to_pb works in
>>>>> this particular run. But if the tests run before hand, something goes
>>>>> awry between the tests. Is it possible that to_pb is using some shared
>>>>> state between calls?
>>>>>
>>>>
>>>> I would not expect so.
>>>>
>>>> If you create a failing unittest that I could try?
>>>>
>>>> Also, does the segfault contain a usable back trace (using gdb)? That
>>>> might give some insights into which code is failing.
>>>>
>>>> /Anders
>>>>
>>>>
>>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4734 bytes --]