[Caml-list] Win32 API

[Caml-list] Win32 API

[Harry Chomsky]

I've embarked on a project to enable writing full-fledged Win32 applications
using OCaml.  I'm doing this mostly for fun as I take a break between jobs.
Of course I hope some people may find it useful in the end.  There's a long
way to go still, but I wanted to let the community know that I'm doing this
and see if you have any advice.  Most of all, if this has already been done,
please tell me so I can stop wasting my time!  As far as I can tell from the
mailing list archives, it has not been done yet, and a few people have
requested it.

If anyone wants to see where I am with it, and offer help or feedback, I've
posted the code at:

http://www.speakeasy.org/~hchomsky/ocaml-win32.html

Much of the work is tedious, but it's been interesting trying to compensate
for Win32's tendency to use C's weak typing.  I've tried to present the API
to OCaml programmers in a strongly typed fashion consistent with the OCaml
philosophy, but still provide all the capabilities that Win32 C programmers
are used to.

The main area I'm uncomfortable with is memory management and the section of
the OCaml manual entitled "Living in harmony with the garbage collector".  I
can't tell from the manual whether I need to pay attention to all these
rules and special C macros.  I do use the C functions malloc and free
occasionally, but only in a very local fashion -- I never pass the resulting
pointers back to the OCaml program.  My C code also makes extensive use of
the functions alloc, alloc_tuple, copy_string, etc. and the macros
Store_field, etc. that are provided by the OCaml runtime library.  So far
everything "just works", but I'd like to make sure I'm not overlooking some
kind of lurking problem here.  Can someone give me a clear explanation of
when I can safely ignore rules 1 and 2?

One place where I most likely *do* have a problem is the array of "message
handler" functions that the C code stores when it registers a window class.
I think I have to register these as global roots, since in theory the OCaml
code could provide dynamically-generated closures which are at risk of being
garbage-collected.  Again, my understanding is fuzzy and I would appreciate
any advice.

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Daniel de Rauglaudre]

Hi,

On Thu, May 31, 2001 at 01:34:29PM -0700, Harry Chomsky wrote:

> I've embarked on a project to enable writing full-fledged Win32 applications
> using OCaml.  I'm doing this mostly for fun as I take a break between jobs.
> [...]
> http://www.speakeasy.org/~hchomsky/ocaml-win32.html
> [...]
> The main area I'm uncomfortable with is memory management and the section of
> [...]

As far as I read your C code, you are likely to have problems with the
OCaml GC: you risk unexpected Memory Faults or Bus Errors. When you
call an OCaml alloc function, all your variables of type "value" may
point to wrong places. E.g. if you write:

some_function (value x)
{
   value y;
   y = alloc_tuple(2);
   ...
}

The "alloc_tuple" may call the GC: and the GC may move the other OCaml
allocated values. If the parameter x corresponds to a pointer (i.e. not
an integer), its value may become false after the allocation of y (may
or may not). The correct way is to previously inform the GC that it
has to change the value of the variable x if it changes the location
of the values it refers to.

This can (must) be done by the macros defined in the module memory.h.
The correct way (if I am not wrong) to write this function like this:

some_function (value x)
{
   CAMLparam1(x);
   value y;
   y = alloc_tuple(2);
   ...
}

-- 
Daniel de RAUGLAUDRE
daniel.de_rauglaudre@inria.fr
http://cristal.inria.fr/~ddr/
-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Harry Chomsky]

> When you
> call an OCaml alloc function, all your variables of type "value" may
> point to wrong places.

OK, I think I'm understanding this better now.  So if a C function does all
of its work using simple macros like Val_int then it's ok, but if it does
anything that might cause the OCaml runtime to allocate memory, then it has
to use the CAMLparam and CAMLreturn macros as described in the
documentation.  Is that right?

Now I understand also why the naming conventions differ so radically between
e.g. Val_int and copy_int32.  Clearly the former just manipulates bits while
the latter allocates memory.  I had thought this was an insignificant
distinction, so when I wrote functions like copy_int32 I named them
Val_handle etc.  I thought this made the code read more easily.  But I see
now why the distinction matters.

I will go through my existing code to make it use the CAML* macros where
necessary and rename the Val_* functions to copy_*.  I'll write to the list
again when I have posted these modifications.

(To the writers of the [mostly superb] documentation: The rules listed in
"Living in harmony with the garbage collector" are very clear, but most of
the sample C code in the other sections of the chapter don't follow the
rules.  Perhaps the documentation could explain not just how to follow the
rules but also *when* to follow them, and why they are ignored in those
samples.  Thanks!)

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Daniel de Rauglaudre]

Hi,

On Thu, May 31, 2001 at 03:59:55PM -0700, Harry Chomsky wrote:

> OK, I think I'm understanding this better now.  So if a C function does all
> of its work using simple macros like Val_int then it's ok, but if it does
> anything that might cause the OCaml runtime to allocate memory, then it has
> to use the CAMLparam and CAMLreturn macros as described in the
> documentation.  Is that right?

Right. And CAMLlocal also.

-- 
Daniel de RAUGLAUDRE
daniel.de_rauglaudre@inria.fr
http://cristal.inria.fr/~ddr/
-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Harry Chomsky]

> > OK, I think I'm understanding this better now.  So if a C function does
all
> > of its work using simple macros like Val_int then it's ok, but if it
does
> > anything that might cause the OCaml runtime to allocate memory, then it
has
> > to use the CAMLparam and CAMLreturn macros as described in the
> > documentation.  Is that right?
>
> Right. And CAMLlocal also.

Now, what happens in the following situation?

  some_function(alloc_something(), alloc_something_else());

I haven't declared any C variables here, so we might think there are no GC
problems.  But there are two allocations here, and it looks like the second
one might invalidate the result of the first one.  So maybe this is unsafe
after all.  Here's an improved version:

  CAMLlocal1(v);
  v = alloc_something();
  some_function(v, alloc_something_else());

Now if the second allocation invalidates the result of the first, the
runtime will update the variable v.  But what if the value of v was already
pushed onto the stack before the second allocation occurred?  (I can't
remember how the C calling convention works, so this might make more sense
if you reverse the order of the parameters.)  The runtime updates the
*variable* v, but it doesn't know about the second copy of that value that's
on the stack, so it doesn't update that, and the function still gets called
with an invalid parameter.

So the only safe solution is to go all the way:

  CAMLlocal2(v1, v2);
  v1 = alloc_something();
  v2 = alloc_something_else();
  some_function(v1, v2);

Is this correct?

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Daniel de Rauglaudre]

Hi,

On Sun, Jun 03, 2001 at 09:01:49PM -0700, Harry Chomsky wrote:

> Now, what happens in the following situation?
>   some_function(alloc_something(), alloc_something_else());

It is unsafe. In C, you cannot call a function of two parameters if one
can modify the evaluation of the other:
      some_function(i, i++);
      some_function(a[i], i++);
In these examples, for the first parameter, you don't know which i is
concerned: the initial value of i or its ending value i + 1.

>   CAMLlocal1(v);
>   v = alloc_something();
>   some_function(v, alloc_something_else());

The rules in C are clear: the evaluation order of the parameters is
unspecified. All these calls are then illegal. You even cannot hope
that a CAMLlocal1(v) would save the value of v pushed one the stack
since in C, all parameters are transmitted by value. Therefore, you
can fix the value of v as many times you want by the CAMLlocal
feature, the pushed value will not change.

Your second example would work with a C compiler evaluating the last
parameters before the first ones.

> So the only safe solution is to go all the way:
> 
>   CAMLlocal2(v1, v2);
>   v1 = alloc_something();
>   v2 = alloc_something_else();
>   some_function(v1, v2);

Right.

-- 
Daniel de RAUGLAUDRE
daniel.de_rauglaudre@inria.fr
http://cristal.inria.fr/~ddr/
-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Stefan Monnier]

>>>>> "Daniel" == Daniel de Rauglaudre <daniel.de_rauglaudre@inria.fr> writes:
> On Sun, Jun 03, 2001 at 09:01:49PM -0700, Harry Chomsky wrote:
>> Now, what happens in the following situation?
>> some_function(alloc_something(), alloc_something_else());
> It is unsafe. In C, you cannot call a function of two parameters if one
> can modify the evaluation of the other.

It is unsafe in the case of O'Caml because of the GC, but as far as C
is concerned, this is not unsafe but only undefined since the
evaluation order is undefined.  And since in this case you don't care
which value you get, the undefinedness would be irrelevant.


	Stefan
-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Dmitry Bely]

"Harry Chomsky" <harry@chomsky.net> writes:

> I've embarked on a project to enable writing full-fledged Win32 applications
> using OCaml.  I'm doing this mostly for fun as I take a break between jobs.
> Of course I hope some people may find it useful in the end.  There's a long
> way to go still, but I wanted to let the community know that I'm doing this
> and see if you have any advice.  Most of all, if this has already been done,
> please tell me so I can stop wasting my time!  As far as I can tell from the
> mailing list archives, it has not been done yet, and a few people have
> requested it.
> 
> If anyone wants to see where I am with it, and offer help or feedback, I've
> posted the code at:
> 
> http://www.speakeasy.org/~hchomsky/ocaml-win32.html

Excuse me, but why not to use Camlidl to define all interface C
funtions/data types (Camlidl user's manual claims that you can even use
.idl files from Windows SDK with some modifications) and then generate
necessary stubs and C<->ocaml conversion functions automatically? This will
make your life a way easier ...

Hope to hear from you soon,
Dmitry


-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

[Caml-list] Win32 API

[Harry Chomsky]

For several months I've been putting together an OCaml library providing
direct access to the Win32 API.  It's a huge project that I'm sure I will
never totally finish... but I've got a good chunk of it done, and I've just
posted it at the following URL:

http://www.speakeasy.org/~hchomsky/code/ocaml-win32.zip

It requires OCaml 3.04, because it uses the module include feature in
combination with "external" declarations.

If anybody has questions or comments about this code or would like to use
it, please let me know.  I'm planning to use it in the near future to build
a tool for interactive OCaml programming, similar to an IDE in some ways.

Last time I posted this, in May, I had some concerns about memory
management.  The discussion that followed convinced me that I was doing it
wrong, and I fixed the problems according to my new understanding.  I'm
pretty sure I've gotten it right this time.

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Warp]

> For several months I've been putting together an OCaml library providing
> direct access to the Win32 API.  It's a huge project that I'm sure I will
> never totally finish... but I've got a good chunk of it done, and I've
just
> posted it at the following URL:
>
> http://www.speakeasy.org/~hchomsky/code/ocaml-win32.zip

Hello Harry !
That's quite a big work, and it's works fine.

You can add theses lines to the Makefile

dllwin32_c.dll : $(MODULES:+=_c.obj)
    -del $@
    link user32.lib kernel32.lib gdi32.lib ocamlrun.lib advapi32.lib
shell32.lib /dll /out:$@ $**

Then , it'll try to build the Dynamic Link Library version of Win32_c, which
is easier to use
with bytecode ocaml.

But I get theses two unresolved externals :

local_roots ( again ! )
atom_table

... which are not referenced in the ocamlrun.lib... have they been forgotten
?

I would also like to share your ideas about a future OCaml IDE on Win32.

Bye
Warp

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Harry Chomsky]

> Then , it'll try to build the Dynamic Link Library version of Win32_c,
which
> is easier to use
> with bytecode ocaml.

I was thinking it would be nice to do this, but I haven't had a chance to
look at the new dynamic-linking features in 3.04 so I haven't tried to
implement it yet.  It would be great if you could figure out how to make
this work.

> I would also like to share your ideas about a future OCaml IDE on Win32.

I've been tossing ideas around in my head for a long time.  I'm not
interested in writing a standard IDE based on the traditional
edit/compile/run cycle.  I'm very fond of the Smalltalk style of interactive
programming, and I want to see how close I can come to it in a static
language with a traditional compilation model like OCaml.  Basically, I want
to blur the line between writing code in files and compiling it, on the one
hand, and entering expressions into the toplevel for evaluation, on the
other.

I started thinking about this when playing with Hugs, which is a cute
interactive system with the unfortunate severe limitation that you can't
bind new variables at the toplevel.  You can define a function, but you
can't give it a name and call it by that name in subsequent expressions you
enter.  The limitation actually makes a lot of sense, given the nature of
the Haskell language: it's much more declarative than OCaml, and a module is
meant to appear "all at once" rather than "one command at a time" as in
OCaml.  Just as function calls have no side effects, so also you can't enter
an expression that has the "side effect" of binding a new name.  It became
clear to me that a good Haskell IDE should have a window for each module
you're working with, with an editable display of the module's code in the
main panel, and a separate little panel at the bottom where you can enter
expressions to be evaluated in the context of the module.  That would give
you the interactivity of Hugs along with the ability to do real programming.

But I decided to focus on OCaml first, both as an implementation language
and as a target language.  It wasn't quite as clear to me how to build a
tool like this for OCaml, because things you do at the toplevel have effects
that accumulate.  I think I've finally got my ideas worked out though, with
some help from Dan Pless, whom I've cc'ed.  Again, each module should have a
window where you edit its code; then there will be a separate window (or
perhaps several) where you enter commands to execute interactively in the
context of all the modules.  This "session" window will feel like the
toplevel in some ways, but less line-oriented.  It will keep a record of the
commands you've entered and the results they've yielded, and when you change
something in one of the modules you will have the option of automatically
reexecuting all the commands in the new context.  There should also probably
be a way to capture the sequence of commands from the session into a new
module which you can then edit normally.

The Smalltalker in me insists that the code you're editing be stored on disk
in a temporary location for compilation and crash protection, but not be
accessible as a normal set of files until you decide to "save" it.  In other
words, you can make changes to your modules and try out the new code without
saving your changes, then if you like the way they work you save them all at
once.  This gives the programmer the freedom to play around with new ideas
without bothering to make backups and keep track of them.  Once the code is
saved, it looks just like a traditional program -- you can distribute it and
grep it and all that.

Looking at your message from last week, I'm not sure if what I'm going to
work on matches your goals very well.  Perhaps the ultimate IDE could have
all the Emacs-like features you're interested in as well as the
Smalltalk-like approach that I'm interested in... or perhaps my ideas are
too far outside the mainstream of most programmers' tastes.  I'm curious to
hear what you think.
-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Warp]

> > Then , it'll try to build the Dynamic Link Library version of Win32_c,
> which
> > is easier to use
> > with bytecode ocaml.
>
> I was thinking it would be nice to do this, but I haven't had a chance to
> look at the new dynamic-linking features in 3.04 so I haven't tried to
> implement it yet.  It would be great if you could figure out how to make
> this work.

yes... I manage to make my own DLL use local_roots using a custom modified
interpreter.
But I think it will be better building a modified ocamllib including missing
globals,
then, you only need to name your lib dll[name].dll and add it in files when
building with
ocamlc / opt.

> > I would also like to share your ideas about a future OCaml IDE on Win32.
>
> I've been tossing ideas around in my head for a long time.  I'm not
> interested in writing a standard IDE based on the traditional
> edit/compile/run cycle.  I'm very fond of the Smalltalk style of
interactive
> programming, and I want to see how close I can come to it in a static
> language with a traditional compilation model like OCaml.  Basically, I
want
> to blur the line between writing code in files and compiling it, on the
one
> hand, and entering expressions into the toplevel for evaluation, on the
> other.

I don't know smaltalk, but I think if you have a background compiling
process running and "showing" you errors and another parsing process
which show you tips on arguments names / types , it'll be good enough.

> I started thinking about this when playing with Hugs, which is a cute
> interactive system with the unfortunate severe limitation that you can't
> bind new variables at the toplevel.  You can define a function, but you
> can't give it a name and call it by that name in subsequent expressions
you
> enter.  The limitation actually makes a lot of sense, given the nature of
> the Haskell language: it's much more declarative than OCaml, and a module
is
> meant to appear "all at once" rather than "one command at a time" as in
> OCaml.  Just as function calls have no side effects, so also you can't
enter
> an expression that has the "side effect" of binding a new name.  It became
> clear to me that a good Haskell IDE should have a window for each module
> you're working with, with an editable display of the module's code in the
> main panel, and a separate little panel at the bottom where you can enter
> expressions to be evaluated in the context of the module.  That would give
> you the interactivity of Hugs along with the ability to do real
programming.

yes. Having a interactive buffer to test your code in realtime is a good
feature.

> But I decided to focus on OCaml first, both as an implementation language
> and as a target language.  It wasn't quite as clear to me how to build a
> tool like this for OCaml, because things you do at the toplevel have
effects
> that accumulate.  I think I've finally got my ideas worked out though,
with
> some help from Dan Pless, whom I've cc'ed.  Again, each module should have
a
> window where you edit its code; then there will be a separate window (or
> perhaps several) where you enter commands to execute interactively in the
> context of all the modules.  This "session" window will feel like the
> toplevel in some ways, but less line-oriented.  It will keep a record of
the
> commands you've entered and the results they've yielded, and when you
change
> something in one of the modules you will have the option of automatically
> reexecuting all the commands in the new context.  There should also
probably
> be a way to capture the sequence of commands from the session into a new
> module which you can then edit normally.

( having a window/buffer for each separate module.... )
When you're writting module-oriented ocaml code, you're using different
files for each module. So different buffers....

About the "session" window, a background compiling process will do a great
job.

> The Smalltalker in me insists that the code you're editing be stored on
disk
> in a temporary location for compilation and crash protection, but not be
> accessible as a normal set of files until you decide to "save" it.  In
other
> words, you can make changes to your modules and try out the new code
without
> saving your changes, then if you like the way they work you save them all
at
> once.  This gives the programmer the freedom to play around with new ideas
> without bothering to make backups and keep track of them.  Once the code
is
> saved, it looks just like a traditional program -- you can distribute it
and
> grep it and all that.

I think a good IDE either have to have an integrated CVS access , or
something like a custom versionning system.
Like this, even when you save and quit , you can always edit again your file
and 'undo changes you have made when you where drunk last nite :)
Borland C++ Builder IDE have for example the poor idea of erazing to undo
buffer when you're saving your file....

> Looking at your message from last week, I'm not sure if what I'm going to
> work on matches your goals very well.  Perhaps the ultimate IDE could have
> all the Emacs-like features you're interested in as well as the
> Smalltalk-like approach that I'm interested in... or perhaps my ideas are
> too far outside the mainstream of most programmers' tastes.  I'm curious
to
> hear what you think.

I think that both are good and can be done in the same project.

Here's my features :
1 - syntax highlightning modes
2 - multiple undo
3 - full customizable shortkeys a la Emacs (conf file written in ocaml)
4 - full buffer/point access API (needed by 3) : search/regexp
5 - indent modes
6 - typing/arguments tips ( using background compiling )
7 - integrated shell
8 - custom versionning

Here's yours (if I have understand) :
1 - "session" buffer for module testing
2 - background compiling not to have the edit/compile style
3 - file backup and versionning

With your win32 api, all theses goals seems reasonable to accomplish.
I think that most of programmers will agree to have theses features enabled,
and if not when can always disable them. The *keyword* to a good IDE is
customization !

Waiting for comments.
Bye
Warp

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr

Re: [Caml-list] Win32 API

[Harry Chomsky]

I've now put together a more proper release of this library, including an
LGPL license.  The new URL is:

http://www.speakeasy.org/~hchomsky/ocaml-win32.html

----- Original Message -----
From: "Harry Chomsky" <harry@chomsky.net>
To: "Caml-list" <caml-list@inria.fr>
Sent: Monday, December 31, 2001 12:52 PM
Subject: [Caml-list] Win32 API


> For several months I've been putting together an OCaml library providing
> direct access to the Win32 API.  It's a huge project that I'm sure I will
> never totally finish... but I've got a good chunk of it done, and I've
just
> posted it at the following URL:
>
> http://www.speakeasy.org/~hchomsky/code/ocaml-win32.zip
>
> It requires OCaml 3.04, because it uses the module include feature in
> combination with "external" declarations.
>
> If anybody has questions or comments about this code or would like to use
> it, please let me know.  I'm planning to use it in the near future to
build
> a tool for interactive OCaml programming, similar to an IDE in some ways.
>
> Last time I posted this, in May, I had some concerns about memory
> management.  The discussion that followed convinced me that I was doing it
> wrong, and I fixed the problems according to my new understanding.  I'm
> pretty sure I've gotten it right this time.
>
> -------------------
> Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ:
http://caml.inria.fr/FAQ/
> To unsubscribe, mail caml-list-request@inria.fr  Archives:
http://caml.inria.fr
>

-------------------
Bug reports: http://caml.inria.fr/bin/caml-bugs  FAQ: http://caml.inria.fr/FAQ/
To unsubscribe, mail caml-list-request@inria.fr  Archives: http://caml.inria.fr