Re: [Caml-list] What does Jane Street use/want for an IDE? What about you?

[Jon Harrop <jon@ffconsultancy.com>]

On Tuesday 04 November 2008 18:35:45 Kuba Ober wrote:
> On Monday 03 November 2008, Jon Harrop wrote:
> > On Monday 03 November 2008 14:15:38 Kuba Ober wrote:
> > > I could port Camelia to OCaml if
> > > someone would first develop Qt bindings for OCaml that would allow me
> > > to do in OCaml what I'm doing in C++ so far ;)
> >
> > That may already be possible if you go via more mainstream dynamic
> > languages like Python. However, python-qt4-dev has fewer installs on
> > Debian and Ubuntu than OCaml does, so you may well find that the Python
> > bindings are inadequate as well.
>
> Python Qt bindings are pretty much feature-complete, but C++'s class system
> maps perhaps better to Python/LISP than to OCaml?

You'll just be invoking autogenerated Python code using OCaml so OCaml's class 
system is only relevant if you want to do some fancy statically-typed shim. 
I'd forget about that and just focus on making the whole of Qt4 available 
safely from OCaml in any form first. Even this is an unsolved problem in the 
OCaml world!

> > > That's an undertaking bigger than Camelia itself. I will not develop
> > > for gtk-anything as lost feature parity with Qt right around the time
> > > when Qt 3 came around, IIRC.
> >
> > I can well believe that. Another option is to create a new GUI toolkit
> > from scratch in OCaml.
>
> Since Camelia uses so much of Qt's functionality, this is a task on par
> with reimplementing Qt. Good luck with that ;)

From Qt4's architecture diagram:

  http://trolltech.com/images/template/product-architecture-diagram-collapsed

  Core, GUI, database, scripting, network, OpenGL, XML, Multimedia, Font 
Engine and Webkit

Of those, only core and GUI might be relevant here but the core lacks 
first-class functions for events and callbacks and the GUI is unable to 
express integrated graphics (e.g. OpenGL layers).

So the amount of Qt that is objectively beneficial is tiny and the cost of 
using it is very high.

> > > Qt 4 is leaps and bounds above anything gtk provides, in
> > > terms of integrated functionality. This is not meant as a flamebait, I
> > > believe it to be an accurate statement of fact.
> >
> > And WPF is leaps and bounds above anything Qt 4 provides, in terms of
> > functionality, integration and performance. Combined with the fact that
> > using Qt4 from a decent language is very hard and basically completely
> > pointless by design anyway, I think there is a strong argument for
> > starting from scratch.
>
> WPF is just another thing, and it's not really portable in any decent sense
> of the word.

Sure. My point is that you're building on antique technology when you could be 
building modern technology instead.

> Camelia could be run on a 486 system with perhaps 64MB of 
> memory when compiled using Qt/Embedded. While one can question whether it's
> useful or not, Qt has the benefit of targeting all major computing
> platforms out there.

That sales hype for Qt is just a reflection of the fact that Qt was built on 
sand using a dying technology (C++). In practice, the world moved on to 
garbage collected virtual machines for GUI programming years ago. Qt was left 
behind and is desperately trying to catch up by migrating to the JVM but the 
JVM is now dying...

> Using Qt4 in a decent language will be easy once there is a C++ parser in
> said decent language.

You'll be Greenspunning managed C++ from the .NET world. That's a fine idea if 
you regard Qt as useful legacy code but I don't see the point myself. Qt will 
always have a stone-age API and reimplementing managed C++ is a huge 
undertaking.

In contrast, you can implement a GUI toolkit in OCaml that far exceeds the 
relevant limitations of Qt4 with quite easily.

> I don't know if I mentioned it, but some time ago I 
> did a proof-of-concept of machine-translating QObject.cpp and friends to
> LISP and it not only worked, but required no moc ;) So I'd say that as soon
> as there's a good-enough C++ parser written in OCaml, and given
> availability of a basic AI package (to implement a KB and searches), it
> should be perfectly doable to translate Qt4 into human-readable,
> not-too-bad OCaml.

Then the question is simply whether it is easier to write a C++ parser in 
OCaml along with the translation code and a sane shim over the whole of Qt4 
or just replace it wholesale.

> > > I wish I could use gtk
> > > since it's easier to bind with, but I'd end up having to either
> > > reimplement parts of Qt, or have to deal with lots of 3rd party
> > > libraries, each from a different author who had a different API design
> > > mindset. Qt takes care of those integration issues internally and
> > > provides a relatively
> > > self-consistent API -- this saves a metric crapton of time (I use the
> > > darn thing).
> >
> > Swings and roundabouts. The metric crapton of time you save with Qt4's
> > functionality is paid by having to use an awful language.
>
> Let's not flame about it,

I would not regard that as flame bait here. I'm sure if OCaml had decent Qt 
bindings they would be very heavily used indeed (the OCaml language would 
excel at GUI programming, which is precisely why so many people here are 
yearning for such bindings) but Qt4 is nowhere near useful enough to make me 
want to go back to C++ or any language that does have Qt bindings.

From what I've heard, a lot of OCaml programming use OpenGL for GUI 
programming instead of GUI toolkits.

> but as a pure exercise I have implemented 
> everything (as in all presented algorithms/approacheds) for three "major"
> courses from www.osu.edu (MATH707/727, CSE630 and CSE680), in LISP, OCaml
> and C++. The code in each case has similar readability and uses similarly
> high-level concepts. I'd say that if you think in OCaml or LISP and
> implement in C++, you can still get pretty good code ;) Of course some
> LISP/OCaml-isms are unavailable in C++, but then some other ones can be
> implemented with relative cleanliness using available template
> metaprogramming.

I've no idea what those exercises where but get your OCaml code peer reviewed 
here before drawing a conclusion. There are very few tasks where C++ is 
competitively comprehensible.

> > > > . Graphical throwback of the documentation related to what is under
> > > > the mouse pointer.
> > >
> > > Easy to do -- goes on my to-do list.
> >
> > How would you implement it?
>
> 1. Everything has type annotations.
> 2. I know scope of symbols from my simple parser.
>
> It's easy enough to look things up by name+context and type, right?

You require the programmer to write type annotations everywhere?

> > > > . Graphical throwback of errors and, in the case of type errors,
> > > > optional highlighting of previous unification points.
> > >
> > > Camelia does that.
> >
> > Cool. AFAIK, OCaml does not export information about unification points.
> > So how do you get hold of that data in Camelia?
>
> I was probably wrong about unification points. What are they?

The points in the program where the type checker unified two or more types in 
order to check type correctness, e.g. the two branches of an "if" expression.

When the programmer makes a mistake, the type checker can be misled early in 
the code but report an actual type error later in the code. So it can be 
helpful (particularly for newbies) to see where the type checker unified 
because the error will almost certainly have been at one of those points.

For example, a function to sum a list of floats:

  let rec sum = function
    | [] -> 0
    | h::t -> h +. sum t

The type error is reported at the "+." operator because the type was 
incorrectly inferred to be "int" when the type checker came across the "0" in 
the first clause. Highlighting the unification points of that type 
immediately leads the programmer to the point of error. Without this, the 
programmer has no idea where the error really is.

> > > > . Parallel so seven of my cores aren't idle while I'm waiting.
> > >
> > > Everything is done in one thread right now, but over time it can be
> > > spread out. Implementing this properly requires refectoring of the code
> > > first: right now Camelia is in no shape to attempt that. After 2.0
> > > release there will be time to attack that.
> >
> > It is trivial if you can pull the sources of the OCaml compiler into your
> > IDE and if OCaml has a parallel GC. ;-)
>
> Doing even that requires a lot of refactoring. Camelia 1.1 had its OCaml-
> awareness spread about everywhere, so before anything major happens I had
> to get all language-specific stuff abstracted out anyway. I'm in the
> process of doing that.

If you want your IDE to support multiple languages, yes. My preference would 
be to write an IDE specifically for OCaml first.

> > It is a logical progression from building your own GUI toolkit. You
> > represent graphics using scene graphs. Everything in an interactive
> > session is converted into scene graphs for rendering.
> >
> > So:
> >
> >   val pretty_printer : Expr.t -> string
> >
> > becomes:
> >
> >   val pretty_printer : Expr.t -> Scene.t
> >
> > and the resulting Scene.t is fed through a renderer like Smoke.
>
> 1. How exactly do you see the pretty_printer rendering things?

Easiest for me to answer by just posting our code. Here is the code from the 
current version of Smoke that typesets paragraphs of text by overflowing 
words onto new lines:

let rec charsmap = function
    | 'f' :: 'f' :: 'i' :: t -> (`CMR, 177) :: charsmap t
    | 'f' :: 'f' :: 'l' :: t -> (`CMR, 178) :: charsmap t
    | 'f' :: 'f' :: t -> (`CMR, 174) :: charsmap t
    | 'f' :: 'i' :: t -> (`CMR, 175) :: charsmap t
    | 'f' :: 'l' :: t -> (`CMR, 176) :: charsmap t
    | c :: t -> charmap c :: charsmap t
    | [] -> []

let word string =
  let string = charsmap(list_of_string string) in
  let aux x (font1, c1) (font2, c2) =
    let contours = Cmfont.glyph font1 c1 in
    let contours = List.map Contour.make contours in
    let g = List.map (Contour.trans (translate x 0.)) contours in
    let advance =
      if font1=font2 then Cmfont.advance font1 c1 c2 else
        Cmfont.advance `CMR 32 32 in
    x +. advance, g in
  let rec aux2 x = function
    | [] -> []
    | h1::t ->
        let h2 = match t with h2::_ -> h2 | _ -> `CMR, 32 in
        let x, g = aux x h1 h2 in
        g :: aux2 x t in
  aux2 0. string |> List.concat |> ContourGeometry.of_contours

let word = memoize (fun _ -> word)

let paragraph ~skip ~justify ~width ~string =
  let words = List.map word (String.split ' ' string) in

> 2. How easy it is to replace pretty_printer built into OCaml?

Already trivial:

$ ocaml
        Objective Caml version 3.10.2

# 3.4;;
- : float = 3.4
# open Format;;
# let print_float ff x = fprintf ff "%0.20f" x;;
val print_float : Format.formatter -> float -> unit = <fun>
# #install_printer print_float;;
# 3.4;;
- : float = 3.39999999999999991118

However, I'd rather OCaml carried run-time type information and the pretty 
printer associated run-time types with typesetting functions.

> I presume that the running OCaml instance (toplevel) would
> dump scene graphs, and then a separate OCaml instance would
> take those and render them?

I would just replace the backend of the current top-level to generate graphics 
instead of text.

> > > Processing toplevel output is an issue nicely orthogonal to editor and
> > > knowledgebase, so this doesn't block on the major refactoring that has
> > > to happen on the editor end.
> >
> > Yes and no. I assume your editor is completely hard-wired from the ground
> > up for editing plain text and maybe even unicode but is completely
> > incapable of rendering arbitrary vector graphics. If so, it will need to
> > be completely gutted.
>
> Well, the thing about toplevel is that it's completely separate from the
> actual source editor.

In the current design, yes.

> So I can actually replace toplevel at any time by simply substituting a
> different widget for it. The toplevel widget talks to a wrapper around the
> real OCaml process, which would be re-used.

You really want to preserve information (e.g. types) in the graphical output 
for throwback to work so I would go for tighter integration of the top-level 
and IDE.

-- 
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e