exception safety / RAII ?

exception safety / RAII ?

[Michael Benfield]

I'm looking at OCaml coming from sort of a C++ background and I'm 
finding it really exciting.

There's one thing that worries me though. C++ programmers have been 
dealing with issues of exception safety for years - it's a complicated 
problem because coding in the presence of exceptions for all intents 
and purposes means your function could end at any point, so how can you 
make sure resources are deallocated? The C++ solution to this problem 
is a technique called Resource Acquisition Is Initialization. C++ 
objects have destructors, which are simply functions that will always 
be called on exit from a scope - including if the exit is caused by an 
exception coming up through your function. You make resource release 
(whether the resource is memory, a socket, whatever) happen in a 
destructor, and then you are set. This is very handy even disregarding 
exceptions.

So I'm just wondering what facilities OCaml has to either implement 
this concept, or other concepts to help with exception safety? The 
OCaml manual says: "Also, finalization can be performed by trapping all 
exceptions, performing the finalization, then raising again the 
exception". This makes me cringe.

Re: [Caml-list] exception safety / RAII ?

[Gerd Stolpmann]

Am Samstag, den 05.03.2005, 13:16 -0500 schrieb Michael Benfield:
> I'm looking at OCaml coming from sort of a C++ background and I'm 
> finding it really exciting.
> 
> There's one thing that worries me though. C++ programmers have been 
> dealing with issues of exception safety for years - it's a complicated 
> problem because coding in the presence of exceptions for all intents 
> and purposes means your function could end at any point, so how can you 
> make sure resources are deallocated? 

This is not as complicated as for C++. Memory is deallocated
automatically by the garbage collector. The remaining resources are
usually handled manually, e.g. for files

let f = open_in ... in
try
  ...  (* code may raise arbitrary exception *)
with
  any_exception ->
    close_in f;
    raise any_exception

If that happens frequently, one can also define higher-order functions
for that purpose, e.g.

let with_in_file f fn =
  try 
    let r = fn f in close_in f; r
  with
    any_exception -> 
      close_in f; 
      raise any_exception

and then

with_in_file 
  (open_in ...)
  (fun f -> ...)

> The C++ solution to this problem 
> is a technique called Resource Acquisition Is Initialization. C++ 
> objects have destructors, which are simply functions that will always 
> be called on exit from a scope - including if the exit is caused by an 
> exception coming up through your function. You make resource release 
> (whether the resource is memory, a socket, whatever) happen in a 
> destructor, and then you are set. This is very handy even disregarding 
> exceptions.
> 
> So I'm just wondering what facilities OCaml has to either implement 
> this concept, or other concepts to help with exception safety? The 
> OCaml manual says: "Also, finalization can be performed by trapping all 
> exceptions, performing the finalization, then raising again the 
> exception". This makes me cringe.

Using finalization functions is often a bad idea. You cannot predict
when they are called. For example, if one tried to close files by
finalization it would be very likely to run out of file descriptors
because finalization is deferred to some unknown point in time in the
future.

The main purpose of finalization is to synchronize O'Caml's memory
management with some foreign mechanism, e.g. because one is calling a
foreign library.

So the simplified answer to your question: There is nothing like RAII,
and one does not miss it, because O'Caml has much better concepts.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------

Re: [Caml-list] exception safety / RAII ?

[Jon Harrop]

On Saturday 05 March 2005 18:16, Michael Benfield wrote:
> I'm looking at OCaml coming from sort of a C++ background and I'm
> finding it really exciting.

Yes, it is an excellent language, far better than C++ for virtually all tasks. 
I do suggest that you try to shed most of the C++ ways of thinking though, as 
the vast majority of them do not apply to OCaml. In particular, forget about 
iterators and OOP.

> There's one thing that worries me though. C++ programmers have been
> dealing with issues of exception safety for years - it's a complicated
> problem because coding in the presence of exceptions for all intents
> and purposes means your function could end at any point, so how can you
> make sure resources are deallocated?

This is only a complicated problem if you do not have a garbage collector, 
which OCaml does.

> The C++ solution to this problem 
> is a technique called Resource Acquisition Is Initialization.

This is a poor man's alternative to garbage collection. The principal problem 
is the inability to determine what object owns which resources at any given 
point in the code.

> C++ 
> objects have destructors, which are simply functions that will always
> be called on exit from a scope - including if the exit is caused by an
> exception coming up through your function. You make resource release
> (whether the resource is memory, a socket, whatever) happen in a
> destructor, and then you are set. This is very handy even disregarding
> exceptions.

In OCaml, any internal resources are transparently deallocated by the garbage 
collector so you do not need to worry. In the relatively unlikely event of an 
external resource, you can either explicitly deallocate yourself or you can 
wrap the resource in an OCaml object and set the finaliser of the object to 
deallocate the resource for you. In theory, this could take forever to 
deallocate. In practice, resources are deallocated extremely quickly.

> So I'm just wondering what facilities OCaml has to either implement
> this concept, or other concepts to help with exception safety? The
> OCaml manual says: "Also, finalization can be performed by trapping all
> exceptions, performing the finalization, then raising again the
> exception". This makes me cringe.

This can be done much more elegantly in OCaml than in C++. Do you know how 
this is done in OCaml?

In most cases you probably won't care when a file is closed after writing to 
it, so you can just rely on the garbage collector. In the few cases that you 
do mind, this is neither difficult nor complicated to implement.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: exception safety / RAII ?

[Stefan Monnier]

> In most cases you probably won't care when a file is closed after writing to
> it, so you can just rely on the garbage collector.

Very bad practice in (e.g.) an NFS world where the filesystem makes no
guarantee about file writes until you actually close the file.


        Stefan

Re: [Caml-list] Re: exception safety / RAII ?

[Brian Hurt]

On Sun, 6 Mar 2005, Stefan Monnier wrote:

> > In most cases you probably won't care when a file is closed after writing to
> > it, so you can just rely on the garbage collector.
> 
> Very bad practice in (e.g.) an NFS world where the filesystem makes no
> guarantee about file writes until you actually close the file.

Ocaml does give you the close_out function to explicitly close an I/O 
stream.  Closing on GC collection just catches the odd cases, like exiting 
the function unexpected (via an exception, for example).

Brian

Re: [Caml-list] exception safety / RAII ?

[Michael Walter]

On Mon, 7 Mar 2005 00:03:52 +0000, Jon Harrop <jon@jdh30.plus.com> wrote:
> > The C++ solution to this problem
> > is a technique called Resource Acquisition Is Initialization.
> 
> This is a poor man's alternative to garbage collection. The principal problem
> is the inability to determine what object owns which resources at any given
> point in the code.

I'm not sure one should think of it as a "poor man's" solution or an
"alternative to GC". An indication to that could be the inclusion of
"using" into C#. Another indication could be the inelegance of
finalizers (or as well the trouble people seem to have using them -- I
wouldn't blame a wrong (read: C++) context as a cause for that).

Michael

Re: [Caml-list] Re: exception safety / RAII ?

[Jon Harrop]

On Monday 07 March 2005 01:32, Stefan Monnier wrote:
> > In most cases you probably won't care when a file is closed after writing
> > to it, so you can just rely on the garbage collector.
>
> Very bad practice in (e.g.) an NFS world where the filesystem makes no
> guarantee about file writes until you actually close the file.

Yes, this may go wrong under certain circumstances (it is not a guarantee) but 
I very rarely have problems with this.

On Monday 07 March 2005 03:31, Michael Walter wrote:
> I'm not sure one should think of it as a "poor man's" solution or an
> "alternative to GC". An indication to that could be the inclusion of
> "using" into C#.

I don't know C#, but when would you want to deallocate a resource before all 
references to it have disappeared?

> Another indication could be the inelegance of 
> finalizers (or as well the trouble people seem to have using them -- I
> wouldn't blame a wrong (read: C++) context as a cause for that).

How are finalisers inelegant? Apart from referencing an object from its own 
finaliser, what problems do people have?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: exception safety / RAII ?

[Stefan Monnier]

>> Very bad practice [...]
[...]
> I very rarely have problems with this.

Very rarely having problems with something can't save it from being
a very bad practice.  Not explicitly closing your files is (in 99% of the
cases) just sloppy coding.  Kinda like letting a GC finalizer close
your windows: when the effect is visible from outside the process it
shouldn't be done in a finalizer.


        Stefan

Re: [Caml-list] Re: exception safety / RAII ?

[Michael Walter]

On Mon, 7 Mar 2005 13:30:48 +0000, Jon Harrop <jon@jdh30.plus.com> wrote:
> > I'm not sure one should think of it as a "poor man's" solution or an
> > "alternative to GC". An indication to that could be the inclusion of
> > "using" into C#.
> 
> I don't know C#, but when would you want to deallocate a resource before all
> references to it have disappeared?

I never had such a desire.

> > Another indication could be the inelegance of
> > finalizers (or as well the trouble people seem to have using them -- I
> > wouldn't blame a wrong (read: C++) context as a cause for that).
> 
> How are finalisers inelegant? Apart from referencing an object from its own
> finaliser, what problems do people have?

Referencing other objects from a finalizer.

Michael

Re: [Caml-list] Re: exception safety / RAII ?

[Jon Harrop]

On Monday 07 March 2005 14:37, Stefan Monnier wrote:
> > I very rarely have problems with this.
>
> Very rarely having problems with something can't save it from being
> a very bad practice.  Not explicitly closing your files is (in 99% of the
> cases) just sloppy coding.

If we're talking about programs which are expected to run for an arbitrary 
amount of time (servers, the top-level etc.) then yes.

However, many programs run for a short time and, in these cases, I believe 
that OCaml guarantees to close your files at least upon program termination, 
if not before. Therefore, I would say that implicitly deallocating external 
resources is not sloppy coding in general.

> Kinda like letting a GC finalizer close 
> your windows: when the effect is visible from outside the process it
> shouldn't be done in a finalizer.

The term "visible" in this context is subjective. People could look to see 
when you close your file, or they could make other measurements to determine 
whether or not you had deallocated an external resource, but unless this is 
likely to cause a problem I wouldn't worry.

In the case of closing windows, I agree that would be sloppy coding. If you 
were talking about resources which the window had required and which the 
window manager is not likely to run out of, I wouldn't call it sloppy coding.

In the case of lablGL not guaranteeing that OpenGL and context resources are 
deallocated in the proper order, this has never caused a problem for anyone 
AFAIK and it would be tricky to fix so I don't worry about. I'd like to fix 
it, yes, but it isn't at the top of my priority list because it doesn't cause 
a problem.

In the case of me implicitly deallocating OpenGL display lists, this is never 
likely to cause a problem in practice so I'll probably never have to worry 
about. A user could still determine that I don't deallocate them immediately 
though, so you could still say that it is "visible".

I also think that this discussion ties in with OCaml and soft real-time tasks. 
In theory, OCaml should not be good for such tasks because the GC could act 
unpredictably. In practice, I find that OCaml is perfectly good for soft 
real-time.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] Re: exception safety / RAII ?

[Michael Walter]

On Mon, 7 Mar 2005 17:29:19 +0000, Jon Harrop <jon@jdh30.plus.com> wrote:
> On Monday 07 March 2005 15:21, Michael Walter wrote:
> > On Mon, 7 Mar 2005 13:30:48 +0000, Jon Harrop <jon@jdh30.plus.com> wrote:
> > > I don't know C#, but when would you want to deallocate a resource before
> > > all references to it have disappeared?
> >
> > I never had such a desire.
> 
> Would you mind elaborating a little on what you do desire, i.e. what does
> "using" do in C#, when would you use it and how does this relate to OCaml?

Sure. I hope the following answers all three questions at once:

let using resource thunk =
  try
    let
      result = thunk resource
    in
      dispose resource;
      result
  with
   any_exception ->
     dispose resource;
     raise any_exception

My O'Caml is not very fluent, possible "dispose resource" should read
"resource # dispose". Basically, the idea is to deterministically
clean up resources, as early as possible (yes, "but it's not as early
as possible" is besides the point :-). In my experience this
simplifies resource management and reasoning about it.

> Presumably this is only difficult in the more complicated case of a general
> dependency graph between objects? In particular, one which has cycles. What
> kinds of programs require such sophistication?
I have no idea about with finalizers in O'Caml (hence my more
broad/general statement), but in other languages I've worked with
there are several limitations which all basically origin in the fact
that finalization order in these languages was non-deterministic.

Greetings,
Michael

Re: [Caml-list] Re: exception safety / RAII ?

[Jon Harrop]

On Monday 07 March 2005 18:47, Michael Walter wrote:
> On Mon, 7 Mar 2005 17:29:19 +0000, Jon Harrop <jon@jdh30.plus.com> wrote:
> > Would you mind elaborating a little on what you do desire, i.e. what does
> > "using" do in C#, when would you use it and how does this relate to
> > OCaml?
>
> Sure. I hope the following answers all three questions at once:
>
> let using resource thunk =
>   try
>     let
>       result = thunk resource
>     in
>       dispose resource;
>       result
>   with
>    any_exception ->
>      dispose resource;
>      raise any_exception
>
> My O'Caml is not very fluent, possible "dispose resource" should read
> "resource # dispose".

Either is clear enough.

> Basically, the idea is to deterministically 
> clean up resources, as early as possible (yes, "but it's not as early
> as possible" is besides the point :-). In my experience this
> simplifies resource management and reasoning about it.

Yes, so you can use your code to do this in OCaml. My concerns with such code 
basically revolve around the possibility of incorrectly deallocating an 
external resource and then using it again. This is not possible if you rely 
on the GC but, as you say, the problem is then the lack of guarantees about 
when resources have been deallocated by what point in the code.

Perhaps explicitly asking the GC to deallocate all such resources is a better 
solution? Depending on the circumstances, this could give a big performance 
hit though...

> > Presumably this is only difficult in the more complicated case of a
> > general dependency graph between objects? In particular, one which has
> > cycles. What kinds of programs require such sophistication?
>
> I have no idea about with finalizers in O'Caml (hence my more
> broad/general statement), but in other languages I've worked with
> there are several limitations which all basically origin in the fact
> that finalization order in these languages was non-deterministic.

Ok, I have found that, with a little thought and careful design beforehand, 
this is not a problem in OCaml. In the case of DAG dependency graphs, my 
solution is to represent the dependency graph for the GC by maintaining a 
list of references to dependees in each object. This forces the GC to respect 
dependencies when collecting.

If the dependency graph is allowed to contain cycles then this might not work. 
I'm not sure though.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] Re: exception safety / RAII ?

[Ville-Pertti Keinonen]

On Mon, 2005-03-07 at 13:47 -0500, Michael Walter wrote:

> My O'Caml is not very fluent, possible "dispose resource" should read
> "resource # dispose". Basically, the idea is to deterministically
> clean up resources, as early as possible (yes, "but it's not as early
> as possible" is besides the point :-). In my experience this
> simplifies resource management and reasoning about it.

You're right, what you describe is often a good way of managing
resources and also standard practice in e.g. Common Lisp (your "using"
function looks a bit like unwind-protect).

In OCaml you need to know what function to use for disposal of the
specific kind of resource.  For the most general approach, you can pass
a function to call in order to dispose of the resource (making it even
more like unwind-protect).

> I have no idea about with finalizers in O'Caml (hence my more
> broad/general statement), but in other languages I've worked with
> there are several limitations which all basically origin in the fact
> that finalization order in these languages was non-deterministic.

This is also true in OCaml.

The current languages that I'm aware of in which people rely on
deterministic finalization are Python and Visual Basic.  In both cases,
it places nasty limitations on the implementation.

IMHO currently the closest to the "best of both worlds" is a good
garbage collector (with non-deterministic finalization) and explicit
management for those resources that need them (file handles are a good
example; in addition to the reasons already mentioned in this thread,
also because they may refer to sockets and to avoid running into file
descriptor limits).

I'm not sure I understand Jon Harrop's concern about using resources
after they've been deallocated.  This has been addressed in the obvious
way (return errors for operations on remaining references) in various
languages for decades, and unlike memory management and type errors,
AFAIK hasn't been a major source of bugs or complaints.

Re: [Caml-list] Re: exception safety / RAII ?

[Richard Jones]

On Tue, Mar 08, 2005 at 01:33:12PM +0200, Ville-Pertti Keinonen wrote:
> IMHO currently the closest to the "best of both worlds" is a good
> garbage collector (with non-deterministic finalization) and explicit
> management for those resources that need them (file handles are a good
> example; in addition to the reasons already mentioned in this thread,
> also because they may refer to sockets and to avoid running into file
> descriptor limits).

I think it'd be nice to have both finalisation and destructors when a
local variable goes out of scope.  You might need to mark variables
specially to indicate that those (and only those) should be reference
counted.  I understand that the implementation would be non-trivial.
Perhaps:

  val open_in : string -> in_channel refcounted

There was a great paper I read about a year ago all about how
finalisation and destruction are completely separate concepts.
Unfortunately I can't find it now ...

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

Re: [Caml-list] Re: exception safety / RAII ?

[Damien Doligez]

On Mar 7, 2005, at 18:10, Jon Harrop wrote:

> However, many programs run for a short time and, in these cases, I 
> believe
> that OCaml guarantees to close your files at least upon program 
> termination,

OCaml does no such thing.  The OS does.

-- Damien

Re: [Caml-list] Re: exception safety / RAII ?

[Michael Walter]

On Tue, 8 Mar 2005 12:32:11 +0000, Richard Jones <rich@annexia.org> wrote:
> [...]
> 
> There was a great paper I read about a year ago all about how
> finalisation and destruction are completely separate concepts.
> Unfortunately I can't find it now ...

Might it be "Destructors, Finalizers, and Synchronization" by Hans
Boehm? IIRC that was really a nice paper.

Michael

Re: [Caml-list] Re: exception safety / RAII ?

[Jon Harrop]

On Tuesday 08 March 2005 11:33, Ville-Pertti Keinonen wrote:
> I'm not sure I understand Jon Harrop's concern about using resources
> after they've been deallocated.  This has been addressed in the obvious
> way (return errors for operations on remaining references) in various
> languages for decades, and unlike memory management and type errors,
> AFAIK hasn't been a major source of bugs or complaints.

A great deal of effort has been put into writing static verifiers to ensure 
correct use, in order to remove this class of run-time errors. So I think 
this is unquestionably a source of bugs.

Could it not be said that having a GC is a way to avoid such errors in the 
context of memory allocation and deallocation?

On Tuesday 08 March 2005 13:07, Damien Doligez wrote:
> On Mar 7, 2005, at 18:10, Jon Harrop wrote:
> > I believe that OCaml guarantees to close your files at least upon program
> > termination,
>
> OCaml does no such thing.  The OS does.

Yipes! In that case I take back what I said before and agree with Stefan - it 
is sloppy coding style to not explicitly close a file.

That seems most odd though. Is this for historical reasons - closing files was 
added long before object finalisers?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] Re: exception safety / RAII ?

[Oliver Bandel]

On Mon, Mar 07, 2005 at 01:30:48PM +0000, Jon Harrop wrote:
> On Monday 07 March 2005 01:32, Stefan Monnier wrote:
> > > In most cases you probably won't care when a file is closed after writing
> > > to it, so you can just rely on the garbage collector.
> >
> > Very bad practice in (e.g.) an NFS world where the filesystem makes no
> > guarantee about file writes until you actually close the file.
> 
> Yes, this may go wrong under certain circumstances (it is not a guarantee) but 
> I very rarely have problems with this.

Building reliable programs on unreliable assumptions
will definitely yield to problems.
Not during development, not during tests, but when the software
finally is used. (Murphey....)
So, buying that piece of code.... and your customers will wish
they better had bought the product from the other company...

Ciao,
   Oliver

Re: [Caml-list] Re: exception safety / RAII ?

[Damien Doligez]

On Mar 8, 2005, at 19:28, Jon Harrop wrote:

> That seems most odd though. Is this for historical reasons - closing 
> files was
> added long before object finalisers?

It was done after thinking hard about the question.  The answer is that
the "close" system call has several effects.  Resource deallocation is
one of these effects, but it's not the only one.  So when to close is
almost always a decision that must be made by the programmer, and we
don't want to encourage people to think otherwise.

And it would be ugly to have both explicit (close function) and implicit
(finalizer) closing of a file.

-- Damien

Re: [Caml-list] Re: exception safety / RAII ?

[Oliver Bandel]

On Mon, Mar 07, 2005 at 05:10:52PM +0000, Jon Harrop wrote:
> On Monday 07 March 2005 14:37, Stefan Monnier wrote:
> > > I very rarely have problems with this.
> >
> > Very rarely having problems with something can't save it from being
> > a very bad practice.  Not explicitly closing your files is (in 99% of the
> > cases) just sloppy coding.
> 
> If we're talking about programs which are expected to run for an arbitrary 
> amount of time (servers, the top-level etc.) then yes.
> 
> However, many programs run for a short time and, in these cases, I believe 
> that OCaml guarantees to close your files at least upon program termination, 
> if not before. Therefore, I would say that implicitly deallocating external 
> resources is not sloppy coding in general.
> 
> > Kinda like letting a GC finalizer close 
> > your windows: when the effect is visible from outside the process it
> > shouldn't be done in a finalizer.
> 
> The term "visible" in this context is subjective.
[...]

If we say "visible from an outer environment", then it is clear,
that this is really unfunctional programming style.

What you are suggesting here?
... => It's how to write programs that introduce side-effects in a statistical
manner and hoping on some subjective likeliehoods that all works well
for all the time.

So, why are you using and promoting functional programming style
(which yields to more safeness in programs), when on the other
side you are layzy in programming?

That is, where the difference is between lazy evaluation of a
programming language and lazy programmer's habbits!

Some people rely on Perl's reference counting and are happy without
"use strict", and others rely on OCaml's Garbage Collector and
are happy with "-unsafe"-option....


But be sure: to write good and reliable software means to rely
on *nothing*!

You can write crap software in *every* language, if you want to...

...and crap begins, where the programmer is to lazy to write
good code.

Good code menas: good readable/understandable/good documented,
  but also reliable because of "knowing what's going on",
which means: prefer functional style, not imperative (and the reason is:
imperative code has side-effects you can not always oversee and which yields
to *stochastical* behaviour ---> that's the same problem, what you
are introducing, when relying on the Garbage Collector instead
of writing good code!).

The advantages of FP will be diminished by your style of programming!


It's like

let const = ref 1 (* no likeliehood here *)
let baviour_of_your_program = ..... (* something wired*)

  let _ = Random.self_init ();
          
    baviour_of_your_program  const (Random.float 10000.0)

Ciao,
   Oliver


P.S.: I tried Random.float max_float and it seems to be always between
       1.<something> * 10^307 ...  1.<something> * 10^308
      Is this a problem of initialization, or a problem of the Random-algorithm,
      or a kind of strange (?) behaviour of a random walk?

Re: [Caml-list] Re: exception safety / RAII ?

[Oliver Bandel]

On Tue, Mar 08, 2005 at 01:10:21AM +0000, Jon Harrop wrote:
> On Monday 07 March 2005 18:47, Michael Walter wrote:
> > On Mon, 7 Mar 2005 17:29:19 +0000, Jon Harrop <jon@jdh30.plus.com> wrote:
> > > Would you mind elaborating a little on what you do desire, i.e. what does
> > > "using" do in C#, when would you use it and how does this relate to
> > > OCaml?
> >
> > Sure. I hope the following answers all three questions at once:
> >
> > let using resource thunk =
> >   try
> >     let
> >       result = thunk resource
> >     in
> >       dispose resource;
> >       result
> >   with
> >    any_exception ->
> >      dispose resource;
> >      raise any_exception
> >
> > My O'Caml is not very fluent, possible "dispose resource" should read
> > "resource # dispose".
> 
> Either is clear enough.
> 
> > Basically, the idea is to deterministically 
> > clean up resources, as early as possible (yes, "but it's not as early
> > as possible" is besides the point :-). In my experience this
> > simplifies resource management and reasoning about it.
> 
> Yes, so you can use your code to do this in OCaml. My concerns with such code 
> basically revolve around the possibility of incorrectly deallocating an 
> external resource and then using it again. This is not possible if you rely 
> on the GC but, as you say, the problem is then the lack of guarantees about 
> when resources have been deallocated by what point in the code.

Allocating ressources (here: "variables" or name-content-bindings)
is not a problem in  functional programming.

Such problems occur when using imperative style.

When using I/O then you work on imperative system's stuff,
like filehandles or simething, when e.g. using Unix-module.
Then you have to be careful, what you are doing.

But even when you use the buffered I/O then it is
linked to system ressources, which are imperative in nature.

This means: do not rely on the GC then!
Free ressources as soon as possible!

If you reuse a "variable" again in a functional
language then this does not yield to problems like
dangling pointers in C.

But nevertheless for example opening a directory to read it's contents
in a recursive function, that traverses a very long path with many subdirectories
(and maybe opening filhandles to all files in a directory)
may cause to problems.... even if the GC and the memory does not say
something abot it, the system may say, that there are no more filedescriptors
available...

Relying on the GC doe not help you to prevent problrms here.

Testing on directories with only a handful of files, nothing
goes wrong... but then used the software in "realworld",
it crashes... which in C means: writing coredump,
and in OCaml (if no internal bug yields to C-like bhaviour ;-))
menas an uncatched exception.


> 
> Perhaps explicitly asking the GC to deallocate all such resources is a better 
> solution? Depending on the circumstances, this could give a big performance 
> hit though...

Best solution, when regarding the filehandle-excample is:
open only if necessary and as late as possible
 (maybe think again about your algorithm in use)
and close as far as possible.

All, what you have to do in C or *ANY* other language
to write good code also applies to OCaml.

It may be a good style of programming to wrap every
imperative stuff (like file I/O) in an environment
that automatically allocates/deallocates the resources.

Something like Postscript's 
gsave/grestore
or that a tag like <mytag> must be closed with </mytag>
in XML-like code...
...so it also makes sense to do I/O only in an
environment of something like

open_ressource()
   (do_something()) 
close_ressource()


Writing wrappers seems not to be a big problem in a functional
language.

But maybe this Could be part oc OCaml++ or something like that ;-)


Maybe Haskell's I/O seems to be like that, but I'm not
such a Haskell expert and didn't explore Hashell's
I/O in detail.... (because I switched to OCaml before ;-))


Ciao,
   Oliver

Re: Re: exception safety / RAII ?

[Daniel Yokomizo]

"Jon Harrop" <jon@jdh30.plus.com> escreveu na mensagem
news:200503080110.21839.jon@jdh30.plus.com...
> On Monday 07 March 2005 18:47, Michael Walter wrote:

[snip]

> > I have no idea about with finalizers in O'Caml (hence my more
> > broad/general statement), but in other languages I've worked with
> > there are several limitations which all basically origin in the fact
> > that finalization order in these languages was non-deterministic.
>
> Ok, I have found that, with a little thought and careful design
beforehand,
> this is not a problem in OCaml. In the case of DAG dependency graphs, my
> solution is to represent the dependency graph for the GC by maintaining a
> list of references to dependees in each object. This forces the GC to
respect
> dependencies when collecting.

AFAIK it doesn't force the GC to respect the dependencies. A object is
garbage if it can't be reached from any root references, it doesn't matter
if other garbage objects still reference it. So if we have:


ROOT -> A -> B -> C

D -> E -> F

G -> B

H -> C


the GC can collect any of [D, E, F, G, H], in any order it wants, because
they're all garbage. An incremental collector could collect first [F, G, H]
because they are (say) large objects, and don't recycle the memory for [D,
E] until the next collection.

IIUC the current OCaml GC implementation may exhibit such properties (i.e.
respect dependencies) but it isn't required to do so.

> If the dependency graph is allowed to contain cycles then this might not
work.
> I'm not sure though.
>
> -- 
> Dr Jon D Harrop, Flying Frog Consultancy Ltd.
> Objective CAML for Scientists
> http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] Re: Re: exception safety / RAII ?

[Jon Harrop]

On Tuesday 08 March 2005 22:53, Daniel Yokomizo wrote:
> "Jon Harrop" <jon@jdh30.plus.com> escreveu na mensagem
> news:200503080110.21839.jon@jdh30.plus.com...
> > Ok, I have found that, with a little thought and careful design
> > beforehand, 
> > this is not a problem in OCaml. In the case of DAG dependency graphs, my
> > solution is to represent the dependency graph for the GC by maintaining a
> > list of references to dependees in each object. This forces the GC to
> > respect 
> > dependencies when collecting.
>
> AFAIK it doesn't force the GC to respect the dependencies. An object is 
> garbage if it can't be reached from any root references, it doesn't matter
> if other garbage objects still reference it.

Yes, thank you for the more formal description. :-)

Provided people want exactly this behaviour (which I do) then I believe that 
the approach I am using will work. You are quite right that this only 
respects dependencies between live objects and not between garbage objects.

I haven't come across any tasks which require more sophisticated GC trickery, 
but I've no doubt that such tasks exist. I can't think of a good general 
solution which doesn't basically require you to implement your own GC in 
OCaml (weak pointers, incremental etc.).

> IIUC the current OCaml GC implementation may exhibit such properties (i.e.
> respect dependencies) but it isn't required to do so.

Yes, the current GC may happen to deallocate things in the reverse of the 
order they were allocated in (this would explain why some things currently 
happen to work) but I've no idea if this really is the case.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] Re: Re: exception safety / RAII ?

[Damien Doligez]

On Mar 8, 2005, at 23:53, Daniel Yokomizo wrote:

> AFAIK it doesn't force the GC to respect the dependencies. A object is
> garbage if it can't be reached from any root references, it doesn't 
> matter
> if other garbage objects still reference it. So if we have:
>
>
> ROOT -> A -> B -> C
>
> D -> E -> F
>
> G -> B
>
> H -> C
>
>
> the GC can collect any of [D, E, F, G, H], in any order it wants, 
> because
> they're all garbage. An incremental collector could collect first [F, 
> G, H]
> because they are (say) large objects, and don't recycle the memory for 
> [D,
> E] until the next collection.

As far as _collecting_ is concerned, the GC can do it in any order it 
wants,
and the current implementation is likely to do it in order of increasing
addresses (i.e. in some unpredictable order).

> IIUC the current OCaml GC implementation may exhibit such properties 
> (i.e.
> respect dependencies) but it isn't required to do so.

What I did specify and implement for 3.08 is the following:

If you call Gc.finalise on your values in the same order as they are
allocated, and if you don't introduce new depedencies afterward (with
assignments), then the finalisation functions will be called in the
right order (i.e. D before E before F, etc).

On the other hand, it means that a non-terminating finalisation function
must call Gc.finalise_release in order to let the GC run other
finalisation functions.

-- Damien

Re: [Caml-list] Re: exception safety / RAII ?

[Damien Doligez]

On Mar 8, 2005, at 22:56, Oliver Bandel wrote:

> P.S.: I tried Random.float max_float and it seems to be always between
>        1.<something> * 10^307 ...  1.<something> * 10^308
>       Is this a problem of initialization, or a problem of the 
> Random-algorithm,
>       or a kind of strange (?) behaviour of a random walk?

I've seen some 10^306 too.  If you think about it, 99% of the reals
between 0 and 10^308 are in the 10^306..10^308 range.

But it's true that there is a "problem": roughly speaking, Random.float
simply takes a (uniformly distributed) real between 0 and 1, rounds it
to the nearest float, and multiplies by its argument.  So in your case
it will never return a number greater than 0 and smaller than
epsilon * max_float.

It's not clear what the spec should be anyway.  If you want a uniform
distribution on the real interval and rounding to the nearest float,
then the missing result range has a vanishingly small probability
anyway.  If you want a uniform distribution on the (finite) set of
floats in the interval, you will get really counter-intuitive results
and a function that cannot be used for physical simulations.

-- Damien

Re: exception safety / RAII ?

[Stefan Monnier]

>> > I very rarely have problems with this.
>> Very rarely having problems with something can't save it from being
>> a very bad practice.  Not explicitly closing your files is (in 99% of the
>> cases) just sloppy coding.
> If we're talking about programs which are expected to run for an arbitrary 
> amount of time (servers, the top-level etc.) then yes.

This logic is routinely used in C to simply never call `free' because they
only run for a short time.  That's a textbook example of "sloppy coding".

>> Kinda like letting a GC finalizer close  your windows: when the effect is
>> visible from outside the process it shouldn't be done in a finalizer.
> The term "visible" in this context is subjective.

I wouldn't call it "subjective", but it is indeed relative to a set of
assumptions about what is "visible" and what isn't.  My assumptions here are
generally that CPU-time and swap-space use are "invisible".  The CPU-time
part is mostly intrinsic in the fact that I write in a high-level language
on top of a multiuser OS and don't have any direct control over the actual
assembly code generated, nor over the scheduling decisions.  The swap-space
part is a basic assumption of a tracing GC.  They're not ideal assumptions,
but without them life is a lot more difficult.

In contrast "the content of my file when another NFS client tries to read it
20 seconds after I generated it (and got an ocamltop prompt back indicating
it'd been generated)" is something that I do consider as "externally
visible" because it doesn't make life much more difficult: just close the
file when you're done with it.  It's extremely rare that the point in the
code where a file can be closed is not trivial to find.


        Stefan

Re: exception safety / RAII ?

[Stefan Monnier]

>> I'm not sure I understand Jon Harrop's concern about using resources
>> after they've been deallocated.  This has been addressed in the obvious
>> way (return errors for operations on remaining references) in various
>> languages for decades, and unlike memory management and type errors,
>> AFAIK hasn't been a major source of bugs or complaints.

> A great deal of effort has been put into writing static verifiers to ensure
> correct use, in order to remove this class of run-time errors.  So I think
> this is unquestionably a source of bugs.

There are different kinds of bugs.  They can be common/rare, serious/benign,
hard/easy to find, hard/easy to fix, ...
Errors that have to do with things like "close" are generally easy to fix.
It's important to catch them, so we have things like Vault that try to catch
them statically.

> Could it not be said that having a GC is a way to avoid such errors in the
> context of memory allocation and deallocation?

In contrast, errors that have to do with leakage and/or dangling pointers,
are often hard to fix, requiring a considerable rework of the code.

Having a GC gives you a freedom when designing an API that is completely
incomparable to the minor convenience of not having to explicitly
close files.  Compare the success of C++ destructors to deal with
file-closing and the insufficiency of those same destructors to deal with
object deallocation (leading to the never ending use of reference counting
and/or explicit copying).


        Stefan

Re: [Caml-list] Re: exception safety / RAII ?

[Jon Harrop]

On Wednesday 09 March 2005 14:48, you wrote:
> >> Very rarely having problems with something can't save it from being
> >> a very bad practice.  Not explicitly closing your files is (in 99% of
> >> the cases) just sloppy coding.
> >
> > If we're talking about programs which are expected to run for an
> > arbitrary amount of time (servers, the top-level etc.) then yes.

My statements were based on the incorrect assumption that the OCaml GC closes 
files when it collects file handles. As this is not the case, I definitely 
agree with you that not explicitly closing files in OCaml is sloppy coding 
because they will not be closed implicitly.

However, provided you don't need to make any guarantees about when the file is 
closed during the running of the program, I still think that implicitly 
closing a file (or deallocating an external resource) via the GC is not 
sloppy coding. Indeed, this facility can be very useful and can eliminate an 
important class of run-time errors.

> This logic is routinely used in C to simply never call `free' because they
> only run for a short time.  That's a textbook example of "sloppy coding".

I wouldn't advocate never calling free() in a C program, but what is the 
difference between calling free at some unspecified point in the future and 
relying on a GC?

> It's extremely rare that the point in the code where a file can be closed is
> not trivial to find. 

In the case of files, yes. More generally, this can be applied to all sorts of 
external resources where that is not true.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] Re: exception safety / RAII ?

[Marcin 'Qrczak' Kowalczyk]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Having a GC gives you a freedom when designing an API that is completely
> incomparable to the minor convenience of not having to explicitly
> close files.  Compare the success of C++ destructors to deal with
> file-closing and the insufficiency of those same destructors to deal with
> object deallocation (leading to the never ending use of reference counting
> and/or explicit copying).

The fact that GC is useful for freeing memory doesn't imply that we
should have no convenient way for closing files.

IMHO there should be something similar to C# 'using', especially given
that the GC will not close dropped files.

The current semantics of C# 'using' is not enough in the presence
of asynchronous exceptions. If OCaml wants to ever have reliable
asynchronous exceptions, its 'using' should have three distinguished
parts: acquire the resource, use it, and release it.

-- 
   __("<         Marcin Kowalczyk
   \__/       qrczak@knm.org.pl
    ^^     http://qrnik.knm.org.pl/~qrczak/

Re: [Caml-list] Re: exception safety / RAII

[Oliver Bandel]

On Wed, Mar 09, 2005 at 04:19:40PM +0000, Jon Harrop wrote:
> On Wednesday 09 March 2005 14:48, you wrote:
> > >> Very rarely having problems with something can't save it from being
> > >> a very bad practice.  Not explicitly closing your files is (in 99% of
> > >> the cases) just sloppy coding.
> > >
> > > If we're talking about programs which are expected to run for an
> > > arbitrary amount of time (servers, the top-level etc.) then yes.
> 
> My statements were based on the incorrect assumption that the OCaml GC closes 
> files when it collects file handles. As this is not the case, I definitely 
> agree with you that not explicitly closing files in OCaml is sloppy coding 
> because they will not be closed implicitly.
> 
> However, provided you don't need to make any guarantees about when the file is 
> closed during the running of the program, I still think that implicitly 
> closing a file (or deallocating an external resource) via the GC is not 
> sloppy coding. Indeed, this facility can be very useful and can eliminate an 
> important class of run-time errors.
> 
> > This logic is routinely used in C to simply never call `free' because they
> > only run for a short time.  That's a textbook example of "sloppy coding".
> 
> I wouldn't advocate never calling free() in a C program, but what is the 
> difference between calling free at some unspecified point in the future and 
> relying on a GC?

I'm not a GC-expert, but it seems obvious to me, that a GC frees
ressources at unspecified time and that also means at unspecified
time in respect to the code that is executed.

But calling a free() at a certain point in a program means that the
deallocation is done at a certain time (in respect to code
that is executed, even when not in certain real time in seconds).

So, a free() in C is not unspecified, as it is called at a certain
section of code.

The GC frees ressources not as long as they are in use, but
that they are not in use does not mean that they are *immediately* freed
by the GC.
It depends on statistical/stochastical things (not to determine).
And that's the difference.

To say a GC that it has to free ressources immediately,
something like a GC-free-flush or something like that
seems a littlebid like going back to free().
(Well, it's not the same, because the GC handles the
"references" of which values are needed and which are not needed
anymore, so a trigger to the GC to free ressources is not the same
as writing code in C that does a free()-call. But normally
it should be easier programming when the programmer does not necessarily
have to think about freeing of GC-ressources.... normally the
GC should be intelligent enough to handle it by itself.)

(Rare cases may be there, where it is necessary to invoke
 GC-specific functions.)

Ciao,
   Oliver

Re: [Caml-list] Re: exception safety / RAII

[Charles Forsyth]

>>I'm not a GC-expert, but it seems obvious to me, that a GC frees
>>ressources at unspecified time and that also means at unspecified
>>time in respect to the code that is executed.

i was going to keep out of this, but i suppose i won't.
	it seems obvious to me, that a GC frees
	resources at unspecified time
not necessarily: it is perfectly possible to define a language so as
to define as predictable those things that could be made predictable,
and then rely on that when programming.  that places the onus
on the implementation to do that, of course, and i suppose that
might result in inefficiency, but perhaps that can be traded off
against other things.  garbage collectors exist that can do that,
with reasonable efficiency in many contexts.
i've used for years a system that behaves
in just this way, and it works well, particularly when faults occur.

i'm not saying this approach is a universal panacea,
or that it's appropriate for caml, just that it is certainly feasible,
and it can work well in practice.  ``but will it work in theory?''
good question.

Re: exception safety / RAII ?

[Stefan Monnier]

> The fact that GC is useful for freeing memory doesn't imply that we
> should have no convenient way for closing files.

Agreed, I was only talking specifically about misuse of finalizers.


        Stefan

Re: exception safety / RAII ?

[Stefan Monnier]

> My statements were based on the incorrect assumption that the OCaml GC
> closes files when it collects file handles. As this is not the case,
> I definitely agree with you that not explicitly closing files in OCaml is
> sloppy coding because they will not be closed implicitly.

My argument stays the same either way.

> However, provided you don't need to make any guarantees about when the
> file is  closed during the running of the program, I still think that
> implicitly  closing a file (or deallocating an external resource) via the
> GC is not  sloppy coding. Indeed, this facility can be very useful and can
> eliminate an important class of run-time errors.

Can you give examples where it's useful?  I don't consider "not needing to
call `close' on line NNNN of my code" to be useful.

Can you describe the "important class of runtime errors" that would be
supposedly eliminated?

>> This logic is routinely used in C to simply never call `free' because they
>> only run for a short time.  That's a textbook example of "sloppy coding".
> I wouldn't advocate never calling free() in a C program, but what is the
> difference between calling free at some unspecified point in the future and
> relying on a GC?

When you rely on a GC, you don't need to keep the pointer around somewhere
in order to be able to call "free" on it and you don't have to keep track of
who might still be using the object.  This means that a library for
a collection data structure can have a very different interface since it
doesn't have to worry about ownership of the contents of the
data structure.

>> It's extremely rare that the point in the code where a file can be closed
>> is not trivial to find.
> In the case of files, yes.  More generally, this can be applied to all
> sorts of external resources where that is not true.

Might be: there's no way we can generalize.  I'm talking about files here.
Finalizers are great, but they shouldn't be used for files.

I'd also argue that they generally shouldn't be used for any "external
resource".  Though I wouldn't be surprised if there's a counter example
somewhere where I'd agree that finalizers are good solution for the
management of some particular kind of external resource in some
particular circumstance.


        Stefan

Re: [Caml-list] Re: exception safety / RAII ?

[Jon Harrop]

On Thursday 10 March 2005 14:33, Stefan Monnier wrote:
> > However, provided you don't need to make any guarantees about when the
> > file is  closed during the running of the program, I still think that
> > implicitly closing a file (or deallocating an external resource) via the
> > GC is not  sloppy coding. Indeed, this facility can be very useful and
> > can eliminate an important class of run-time errors.
>
> Can you give examples where it's useful?  I don't consider "not needing to
> call `close' on line NNNN of my code" to be useful.

OpenGL display lists from earlier in this thread.

> Can you describe the "important class of runtime errors" that would be
> supposedly eliminated?

In that case, undefined rendering behaviour when a display list is executed 
after it has been incorrectly destroyed. This could also wreck the OpenGL 
state and break rendering for the rest of the frame or even for the rest of 
the program.

Unless you go to great lengths to catch such errors in all cases (which could 
require arbitrarily complicated tests, slowing down your program) such 
mistakes could make an OCaml program segfault with more fragile external 
resources.

> > In the case of files, yes.  More generally, this can be applied to all
> > sorts of external resources where that is not true.
>
> Might be: there's no way we can generalize.  I'm talking about files here.
> Finalizers are great, but they shouldn't be used for files.

Let me try another file-specific example then: Programs which save output at 
some point during their execution. From the users point of view, there is no 
"visible" difference between having the program close the file as soon as 
writing is complete and leaving it up to the GC to close the file a short 
time afterwards (and before it completes). I have many such programs.

> I'd also argue that they generally shouldn't be used for any "external
> resource".  Though I wouldn't be surprised if there's a counter example
> somewhere where I'd agree that finalizers are good solution for the
> management of some particular kind of external resource in some
> particular circumstance.

Even if the external resource is memory?

Generally, I'd always consider exploiting the GC because of the safety it 
provides...

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] Re: exception safety / RAII ?

[Michael Walter]

On Thu, 10 Mar 2005 16:52:01 +0000, Jon Harrop <jon@ffconsultancy.com> wrote:
> > Can you give examples where it's useful?  I don't consider "not needing to
> > call `close' on line NNNN of my code" to be useful.
> 
> OpenGL display lists from earlier in this thread.

Typically you manage your rendering resources at well-defined points
in the life time of a program (for performance reasons -- for
performance reasons you probably also wouldn't use display lists, but
that is besides the point), so I don't see how this is a problem

> > > In the case of files, yes.  More generally, this can be applied to all
> > > sorts of external resources where that is not true.
> >
> > Might be: there's no way we can generalize.  I'm talking about files here.
> > Finalizers are great, but they shouldn't be used for files.
> 
> Let me try another file-specific example then: Programs which save output at
> some point during their execution. From the users point of view, there is no
> "visible" difference between having the program close the file as soon as
> writing is complete and leaving it up to the GC to close the file a short
> time afterwards (and before it completes). I have many such programs.

You might want to be able to process the output file as soon as it's
written, not as soon as the GC decides to close the file. Also,
wouldn't it be trivial to just implement it "the nice way" (maybe I
think of different cases where "programs [..] save output")?

Michael

Re: exception safety / RAII ?

[Stefan Monnier]

>> > However, provided you don't need to make any guarantees about when the
>> > file is  closed during the running of the program, I still think that
>> > implicitly closing a file (or deallocating an external resource) via the
>> > GC is not  sloppy coding. Indeed, this facility can be very useful and
>> > can eliminate an important class of run-time errors.
>> 
>> Can you give examples where it's useful?  I don't consider "not needing to
>> call `close' on line NNNN of my code" to be useful.

> OpenGL display lists from earlier in this thread.

Two problems with this example: it doesn't involve files AFAICT, and it's
not an example in any useful sense.  You'd need to describe the skeleton of
the code to explain why it's useful (rather than merely convenient).

>> Can you describe the "important class of runtime errors" that would be
>> supposedly eliminated?

> In that case, undefined rendering behaviour when a display list is executed
> after it has been incorrectly destroyed.  This could also wreck the OpenGL
> state and break rendering for the rest of the frame or even for the rest of
> the program.

[ Ignoring the fact that it still doesn't seem to involve files ] Are such
errors common (or difficult to diagnose/fix) enough that they deserve the
name "important class of runtime errors"? (genuine question: I know next to
nothing about OpenGL).

>> > In the case of files, yes.  More generally, this can be applied to all
>> > sorts of external resources where that is not true.
>> 
>> Might be: there's no way we can generalize.  I'm talking about files here.
>> Finalizers are great, but they shouldn't be used for files.

> Let me try another file-specific example then: Programs which save output
> at  some point during their execution.  From the users point of view,
> there is no  "visible" difference between having the program close the
> file as soon as  writing is complete and leaving it up to the GC to close
> the file a short  time afterwards (and before it completes).  I have many
> such programs.

You're arguing beside my point: I don't care whether there are cases where
using finalizers to close files doesn't hurt.  My point is both that there
are cases where they do hurt (even sometimes for the exact same code where
they didn't hurt in other circumstances), and that the added convenience of
not needing to explicitly close the file is insignificant.

So I'd like to see an example where the lack of a "file-close finalizer"
would make the code harder to write.

>> I'd also argue that they generally shouldn't be used for any "external
>> resource".  Though I wouldn't be surprised if there's a counter example
>> somewhere where I'd agree that finalizers are good solution for the
>> management of some particular kind of external resource in some
>> particular circumstance.
> Even if the external resource is memory?

I've already replied to this and even gave another example (CPU), so please
try something else.


        Stefan