[Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[Goswin von Brederlow]

Hi,

I'm stuck with a bug in the Tetrix example for my QT5 bindings:

  https://github.com/mrvn/ocaml-qt5

The segfault happens when you click start and the first piece is moved
one tile down in caml_mrvn_QT5_OPainter_fillRect. The arguments to the
call all look ok but something must corrupt the painter. The segfault
goes away when I force a Gc.full_major before creating a new OPainter
in TetrixBoard:148.

Any help would be welcome. I kinow it's a wild goose chase but when I
try to narrow it down to a smaller problem the segfault goes away.

MfG
	Goswin

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[Anatoly Zaretsky]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

Hello.

On Wed, Mar 23, 2016 at 12:50 PM, Goswin von Brederlow <goswin-v-b@web.de>
wrote:

> I'm stuck with a bug in the Tetrix example for my QT5 bindings:
>
>   https://github.com/mrvn/ocaml-qt5
>
> The segfault happens when you click start and the first piece is moved
> one tile down in caml_mrvn_QT5_OPainter_fillRect. The arguments to the
> call all look ok but something must corrupt the painter. The segfault
> goes away when I force a Gc.full_major before creating a new OPainter
> in TetrixBoard:148.
>

Just a wild guess: there are a lot of raw c++ pointer casts to ocaml values
in the code, and to quote
http://caml.inria.fr/pub/docs/manual-ocaml/intfc.html#sec424 "this can
crash the garbage collector" in some non-obvious circumstances.

[-- Attachment #2: Type: text/html, Size: 1326 bytes --]

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[François Bobot]

On 23/03/2016 16:18, Anatoly Zaretsky wrote:
> On Wed, Mar 23, 2016 at 12:50 PM, Goswin von Brederlow <goswin-v-b@web.de
> <mailto:goswin-v-b@web.de>> wrote:
>
>     I'm stuck with a bug in the Tetrix example for my QT5 bindings:
>
>     https://github.com/mrvn/ocaml-qt5
>
>     The segfault happens when you click start and the first piece is moved
>     one tile down in caml_mrvn_QT5_OPainter_fillRect. The arguments to the
>     call all look ok but something must corrupt the painter. The segfault
>     goes away when I force a Gc.full_major before creating a new OPainter
>     in TetrixBoard:148.
>
>
> Just a wild guess: there are a lot of raw c++ pointer casts to ocaml values in the code, and to
> quote http://caml.inria.fr/pub/docs/manual-ocaml/intfc.html#sec424 "this can crash the garbage
> collector" in some non-obvious circumstances.

And in the future of no-naked pointer it will be forbidden. Perhaps it should be added to the ocaml 
documentation with the precise future rules [1].

The segfault is in C++ code, so I think it is not this problem. Moreover no asserts of the debug 
runtime are broken. On debian I don't know how to install qt5 in debug mode, so I'm not able to see 
in gdb which pointer is at fault.

NB: the INSTALL file forget to ask to install qt5 development file, even if it is obviously needed ;).

[1]: https://github.com/ocaml/ocaml/pull/297#issuecomment-159233967

-- 
François

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[Goswin von Brederlow]

On Wed, Mar 23, 2016 at 06:18:12PM +0100, François Bobot wrote:
> On 23/03/2016 16:18, Anatoly Zaretsky wrote:
> >On Wed, Mar 23, 2016 at 12:50 PM, Goswin von Brederlow <goswin-v-b@web.de
> ><mailto:goswin-v-b@web.de>> wrote:
> >
> >    I'm stuck with a bug in the Tetrix example for my QT5 bindings:
> >
> >    https://github.com/mrvn/ocaml-qt5
> >
> >    The segfault happens when you click start and the first piece is moved
> >    one tile down in caml_mrvn_QT5_OPainter_fillRect. The arguments to the
> >    call all look ok but something must corrupt the painter. The segfault
> >    goes away when I force a Gc.full_major before creating a new OPainter
> >    in TetrixBoard:148.
> >
> >
> >Just a wild guess: there are a lot of raw c++ pointer casts to ocaml values in the code, and to
> >quote http://caml.inria.fr/pub/docs/manual-ocaml/intfc.html#sec424 "this can crash the garbage
> >collector" in some non-obvious circumstances.

I know about that and all c++ pointers are paired with an ocaml object
that holds them. The c++ object also holds a weak refrence to the
ocaml object. When the c++ object is destructed it removes the pointer
from the paired ocaml object before being freed. Therefore there can
never be a reachable danling pointer on the ocaml heap that could
potentially point the next ocaml heap. 

It also seems like the heap and c++ allocations are at opposite ends
of the address space, e.g. from the debug output:

  value caml_mrvn_Qt5_OClass_register_obj(OClass*, value)(0x113bbe8, 0x7f4cfac1cfa8)

The c++ object is at 0x113bbe8, the lower end of the heap and the
ocaml object at 0x7f4cfac1cfa8, the upper end of the heap.

> And in the future of no-naked pointer it will be forbidden. Perhaps it
> should be added to the ocaml documentation with the precise future rules
> [1].

That will break a lot of code and will be rather anyoing. I guess for
me it's time to overload operator new() to add a black colored ocaml
header before every allocation.

More generally what I often wish I had would be a block that holds
both ocaml values and at least one C pointer. Could we have a tag(s) for
blocks where the Gc skips the first field(s) and only scans the rest?

> The segfault is in C++ code, so I think it is not this problem. Moreover no
> asserts of the debug runtime are broken. On debian I don't know how to
> install qt5 in debug mode, so I'm not able to see in gdb which pointer is at
> fault.
> 
> NB: the INSTALL file forget to ask to install qt5 development file, even if it is obviously needed ;).
> 
> [1]: https://github.com/ocaml/ocaml/pull/297#issuecomment-159233967

Yeah, it's still at the proof-of-concept stage. The INSTALL is oasis
generated and I haven't even looked at what oasis puts in there. No
opam package yet either.

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[Goswin von Brederlow]

On Thu, Mar 24, 2016 at 11:25:59AM +0100, Goswin von Brederlow wrote:
> On Wed, Mar 23, 2016 at 06:18:12PM +0100, François Bobot wrote:
> > On 23/03/2016 16:18, Anatoly Zaretsky wrote:
> > >On Wed, Mar 23, 2016 at 12:50 PM, Goswin von Brederlow <goswin-v-b@web.de
> > ><mailto:goswin-v-b@web.de>> wrote:
> > >
> > >    I'm stuck with a bug in the Tetrix example for my QT5 bindings:
> > >
> > >    https://github.com/mrvn/ocaml-qt5
> > >
> > >    The segfault happens when you click start and the first piece is moved
> > >    one tile down in caml_mrvn_QT5_OPainter_fillRect. The arguments to the
> > >    call all look ok but something must corrupt the painter. The segfault
> > >    goes away when I force a Gc.full_major before creating a new OPainter
> > >    in TetrixBoard:148.

Turns out the problem has nothing directly to do with the ocaml
bindings. The problem is that Qt only allows one active painter per
paint device at a time and has explicit begin()/end() functions to
start and top painting where begin() does error checking. The QPainter
constructor also allows passing the paint device directly, which
automatically calls begin(device) but reports no errors. The
destructor then also automatically calls end().

Problem in ocaml now is that the destructor no longer runs when the
painter goes out of scope but only when the Gc gets around to it. And
for some reason creating a second painter for a device kind of works
(in disregard of what the API says) as long as you don't destroy the
first one half way through, which the Gc does.

I've converted to explicit painter#begin_ and painter#end_ calls and
added some checks in the bindings to enforce them and now everything
works. Now off to adding KeyEvent parsing and one can actually play.

MfG
	Goswin

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[SP]

On Fri, Mar 25, 2016 at 12:28:39PM +0100, Goswin von Brederlow wrote:
>I've converted to explicit painter#begin_ and painter#end_ calls and
>added some checks in the bindings to enforce them and now everything
>works. Now off to adding KeyEvent parsing and one can actually play.

I think having Qt usable from OCaml would be great.

But I'm not sure the best way to do it to map the object oriented system
"one-to-one" with OCaml objects. It will be as tedious to use as it is
with C++ and keeping track of any changes in the API might be a pain.

I'd like to follow the progress on this. Will you be posting on the
mailing list or have you got a blog about this?

-- 
	SP

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[Goswin von Brederlow]

On Tue, Mar 29, 2016 at 11:29:12PM +0100, SP wrote:
> On Fri, Mar 25, 2016 at 12:28:39PM +0100, Goswin von Brederlow wrote:
> >I've converted to explicit painter#begin_ and painter#end_ calls and
> >added some checks in the bindings to enforce them and now everything
> >works. Now off to adding KeyEvent parsing and one can actually play.
> 
> I think having Qt usable from OCaml would be great.
> 
> But I'm not sure the best way to do it to map the object oriented system
> "one-to-one" with OCaml objects. It will be as tedious to use as it is
> with C++ and keeping track of any changes in the API might be a pain.

I think it is the only sane way. GUI widgets are full of inheritance
and you have tons of functions that can take any widget. Without
objects you would have to use recursive types like

    type t = unit pushButton AbstarctButton.abstractButton Widget.widget Object.oObject OCLass.oClass

(and how do I say that a widget is both an oObject and paintDevice?)

or phantom types like

    type t = [`PushButton | `AbstractButton | `Widget | `Object | `PaintDevice ] OClass.oClass

And that is just for a simple button. Other types would get a lot
longer, not to mention the type for methods taking other widgets where
you have multiple of those types.


From what I see the API doesn't change much. Even between QT4 and QT5
there is only a handfull of API changes. So I don't think tracking
them will be that hard.

Getting the API right and playing ball with the Gc the first time
around is a pain though. One has to be carefull about Qt taking
ownership of the arguments or not, which requires a human reading the
API docs. So I'm afraid automatic stub generation from the header
files is mostly not possible.

> I'd like to follow the progress on this. Will you be posting on the
> mailing list or have you got a blog about this?

You can subscribe to the github repository to see commits, issues,
pull requests etc. But I will probably announce larger milestones on
the mailing list too.  So far it's a proof of concept and I think
having the Tetrix example work proofs the concept viable. So I'm gonna
keep working on this.

MfG
	Goswin

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[Jonas Jensen]

On 31 March 2016 at 12:21, Goswin von Brederlow <goswin-v-b@web.de> wrote:
>
> I think it is the only sane way. GUI widgets are full of inheritance
> and you have tons of functions that can take any widget. Without
> objects you would have to use recursive types like
>
>     type t = unit pushButton AbstarctButton.abstractButton Widget.widget Object.oObject OCLass.oClass
>
> (and how do I say that a widget is both an oObject and paintDevice?)
>
> or phantom types like
>
>     type t = [`PushButton | `AbstractButton | `Widget | `Object | `PaintDevice ] OClass.oClass
>
> And that is just for a simple button. Other types would get a lot
> longer, not to mention the type for methods taking other widgets where
> you have multiple of those types.

The phantom-types approach looks attractive. What would be the problem
with it? Obviously, those long types should not be written by hand but
could be aliases, like

type widget = [ `Widget | `Object | `PaintDevice ]
type abstractButton = [ `AbstractButton | widget ]
type pushButton = [ `PushButton | abstractButton ]

And then a handle to a push button would have type "pushButton
oClass". A function taking any widget would have type "[> widget ]
oClass -> ...".

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[Goswin von Brederlow]

On Thu, Mar 31, 2016 at 01:00:31PM +0200, Jonas Jensen wrote:
> On 31 March 2016 at 12:21, Goswin von Brederlow <goswin-v-b@web.de> wrote:
> >
> > I think it is the only sane way. GUI widgets are full of inheritance
> > and you have tons of functions that can take any widget. Without
> > objects you would have to use recursive types like
> >
> >     type t = unit pushButton AbstarctButton.abstractButton Widget.widget Object.oObject OCLass.oClass
> >
> > (and how do I say that a widget is both an oObject and paintDevice?)
> >
> > or phantom types like
> >
> >     type t = [`PushButton | `AbstractButton | `Widget | `Object | `PaintDevice ] OClass.oClass
> >
> > And that is just for a simple button. Other types would get a lot
> > longer, not to mention the type for methods taking other widgets where
> > you have multiple of those types.
> 
> The phantom-types approach looks attractive. What would be the problem
> with it? Obviously, those long types should not be written by hand but
> could be aliases, like
> 
> type widget = [ `Widget | `Object | `PaintDevice ]
> type abstractButton = [ `AbstractButton | widget ]
> type pushButton = [ `PushButton | abstractButton ]

Didn't remember that. :)
 
> And then a handle to a push button would have type "pushButton
> oClass". A function taking any widget would have type "[> widget ]
> oClass -> ...".

At the start I tried a modular based binding and I didn't like it.

   win#show

becomes

   Qt5.OWidget.show win

where you have to know in which subclass the show function was defined. Or

   Qt5.OMainWindow.show win

when OMainWindow includes all the functions from OWidget (which needs
some more submodules in each module for inheriting purposes).

One can open Qt5 to shorten that a bit but different Qt5 classes have
the same methods with different argument types. So the module name for
the class has to be typed in a lot. Even local open is of limited use
because you never know when 2 modules will have diverging functions.
But yeah, if you like typing more then you could use this.


On the technical side you have the problem that I think needs true
inheritance at least internally. A pushButton is not just an abstract
type pointing to the Qt5 c++ class. It needs additional data tracking
of other objects. You can't see much of that yet but I want to do the
tracking on the ocaml side if possible and not in the c stubs.

The most common example would be signals and slots. Every time you
connect to a signal something has to keep the closure called by the
signal alive. At least until you disconnect from the signal or the
object dies. And since you can connect many closures to a single
signal I was thinking of tracking them in a Hashtbl in the ocaml
object, one per signal.

With objects each class inherits all the data of the parent while
alowing the child to be used with methods of the parent. You can't do
that with records or tuples. And if I have one class definition per
Qt5 class anyway I might as well expose that to the user instead of
hiding it in modules.

The other thing, which you can already see in the event handler, is
overloading virtual functions. The TetrixBoard class inherits the
paintEvent, keyPressEvent and timerEvent mixins. There are a lot more
virtual functions that can be overloaded in Qt5 widgets and the list
grows and grows as you go down the inheritance tree. Would be harder
to keep track of them with a record data structure.


Now that I figured out enough to have the tetrix example working I
should try again to make a modular interface for comparison. Writing
my arguments above I can already think of ways to work around them. My
goal is to have generated code from a API description (basically the
C++ header file with some more annotations thrown in about ownership
and in a simpler to parse format [ocaml code?]). Maybe it wouldn't be
to hard to generate both objects and modules from that.

MfG
	Goswin

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[SP]

On Sat, Apr 02, 2016 at 01:38:41PM +0200, Goswin von Brederlow wrote:
> With objects each class inherits all the data of the parent while
> alowing the child to be used with methods of the parent.

This is part of the problem I'm referring to when I say "tracking the
changes in the API".

> You can't do that with records or tuples. And if I have one class
> definition per Qt5 class anyway I might as well expose that to the
> user instead of hiding it in modules.

Yes, it would have to be new approach all together.

There is still merit in having a one-to-one ocaml wrapper for Qt. If
you can do the wrapping using some generator operating on the headers,
even better.

But I would encourage that a higher level, functional wrapper is used
for applications. It would be a shame if heavy object orientation
affected user interface codebases because of such necessity.

-- 
    SP

Re: [Caml-list] RFH: can't figure out why my QT5 widget bindings segfault

[Goswin von Brederlow]

On Wed, Apr 06, 2016 at 11:56:21PM +0100, SP wrote:
> On Sat, Apr 02, 2016 at 01:38:41PM +0200, Goswin von Brederlow wrote:
> >With objects each class inherits all the data of the parent while
> >alowing the child to be used with methods of the parent.
> 
> This is part of the problem I'm referring to when I say "tracking the
> changes in the API".
> 
> >You can't do that with records or tuples. And if I have one class
> >definition per Qt5 class anyway I might as well expose that to the
> >user instead of hiding it in modules.
> 
> Yes, it would have to be new approach all together.

Then I'm afraid you have to rewrite Qt. You have to overload every
single Qt class to use the many virtual functions.
 
> There is still merit in having a one-to-one ocaml wrapper for Qt. If
> you can do the wrapping using some generator operating on the headers,
> even better.
> 
> But I would encourage that a higher level, functional wrapper is used
> for applications. It would be a shame if heavy object orientation
> affected user interface codebases because of such necessity.
> 
> -- 
>    SP

Feel free to do that. Personally I think a GUI is the worst fit for
functional programming and the best fit for object oriented. A GUI
screams side effect and mutability to me and a purely functional
approach I think is entirely pointless beyond the academical.

Luckily ocaml allows one to use both functional and object oriented
programming where it fits best.

MfG
	Goswin