[Caml-list] Channels not closed on gc?

[Caml-list] Channels not closed on gc?

[Lauri Alanko]

I only recently noticed that ocaml does not close open channels when  
they are garbage collected. This is evidently intentional behavior,  
but it was quite unexpected.

To be clear, I do think it's bad style to rely on GC for releasing OS  
resources, but that doesn't explain why GC shouldn't do this if the  
programmer has failed to explicitly close the channel. And if the  
intention were to _enforce_ good style, the channel finaliser would  
spout out an error or warning upon detecting that the channel hasn't  
yet been closed, instead of just silently leaking file handles like it  
does currently.

It is of course trivial to "fix" this by attaching a simple finaliser,  
but the fact that this is not done by default makes me suspect that  
there would be something fishy with this approach. So, what's the  
rationale for the current behavior?

Thanks,


Lauri

AW: [Caml-list] Channels not closed on gc?

[Gerd Stolpmann]

Am 21.05.2012 13:23:36 schrieb(en) Lauri Alanko:
> I only recently noticed that ocaml does not close open channels when  
> they are garbage collected. This is evidently intentional behavior,  
> but it was quite unexpected.
> 
> To be clear, I do think it's bad style to rely on GC for releasing OS  
> resources, but that doesn't explain why GC shouldn't do this if the  
> programmer has failed to explicitly close the channel. And if the  
> intention were to _enforce_ good style, the channel finaliser would  
> spout out an error or warning upon detecting that the channel hasn't  
> yet been closed, instead of just silently leaking file handles like  
> it does currently.
> 
> It is of course trivial to "fix" this by attaching a simple  
> finaliser, but the fact that this is not done by default makes me  
> suspect that there would be something fishy with this approach. So,  
> what's the rationale for the current behavior?

It's predictable.

Closing a channel is not only about releasing OS resources. Imagine the  
channel is actually a pipe - closing it means to signal EOF, i.e. it's  
a way of notifying another program about an event. This should not  
happen behind one's back.

Closing a regular file would in deed be harmless, but there is no  
generic way to identify such channels (in the OS).

Gerd

> Thanks,
> 
> 
> Lauri
> 
> 
> 
>-- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
> 


-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

Re: [Caml-list] Channels not closed on gc?

[Philippe Wang]

On Mon, May 21, 2012 at 1:57 PM, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> Am 21.05.2012 13:23:36 schrieb(en) Lauri Alanko:
>
>> I only recently noticed that ocaml does not close open channels when they
>> are garbage collected. This is evidently intentional behavior, but it was
>> quite unexpected.
>>
>> To be clear, I do think it's bad style to rely on GC for releasing OS
>> resources, but that doesn't explain why GC shouldn't do this if the
>> programmer has failed to explicitly close the channel. And if the intention
>> were to _enforce_ good style, the channel finaliser would spout out an error
>> or warning upon detecting that the channel hasn't yet been closed, instead
>> of just silently leaking file handles like it does currently.
>>
>> It is of course trivial to "fix" this by attaching a simple finaliser, but
>> the fact that this is not done by default makes me suspect that there would
>> be something fishy with this approach. So, what's the rationale for the
>> current behavior?
>
>
> It's predictable.
>
> Closing a channel is not only about releasing OS resources. Imagine the
> channel is actually a pipe - closing it means to signal EOF, i.e. it's a way
> of notifying another program about an event. This should not happen behind
> one's back.
>
> Closing a regular file would in deed be harmless, but there is no generic
> way to identify such channels (in the OS).
>
> Gerd
>
>
>> Thanks,
>>
>>
>> Lauri

Wouldn't it be a good idea to provide both ways?
i.e., one that collects automatically, in addition to the current one
that doesn't.

-- 
Philippe Wang
   mail@philippewang.info

Re: [Caml-list] Channels not closed on gc?

[Gerd Stolpmann]

Am Montag, den 21.05.2012, 14:53 +0200 schrieb Philippe Wang:
> On Mon, May 21, 2012 at 1:57 PM, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> > Am 21.05.2012 13:23:36 schrieb(en) Lauri Alanko:
> >
> >> I only recently noticed that ocaml does not close open channels when they
> >> are garbage collected. This is evidently intentional behavior, but it was
> >> quite unexpected.
> >>
> >> To be clear, I do think it's bad style to rely on GC for releasing OS
> >> resources, but that doesn't explain why GC shouldn't do this if the
> >> programmer has failed to explicitly close the channel. And if the intention
> >> were to _enforce_ good style, the channel finaliser would spout out an error
> >> or warning upon detecting that the channel hasn't yet been closed, instead
> >> of just silently leaking file handles like it does currently.
> >>
> >> It is of course trivial to "fix" this by attaching a simple finaliser, but
> >> the fact that this is not done by default makes me suspect that there would
> >> be something fishy with this approach. So, what's the rationale for the
> >> current behavior?
> >
> >
> > It's predictable.
> >
> > Closing a channel is not only about releasing OS resources. Imagine the
> > channel is actually a pipe - closing it means to signal EOF, i.e. it's a way
> > of notifying another program about an event. This should not happen behind
> > one's back.
> >
> > Closing a regular file would in deed be harmless, but there is no generic
> > way to identify such channels (in the OS).
> >
> > Gerd
> >
> >
> >> Thanks,
> >>
> >>
> >> Lauri
> 
> Wouldn't it be a good idea to provide both ways?
> i.e., one that collects automatically, in addition to the current one
> that doesn't.

You can easily do this yourself:

Gc.finalise close_in ch

(if ch is an in_channel; for out_channels use close_out_noerr). We
exploit here that channels track whether they are already closed. If
there is the danger of getting an exception, catch it:

let close_out_catch ch =
  try close_out ch
  with error ->
    prerr_endline("Exception from attempt to close channel: " ^ Printexc.to_string error)

This points, btw, to another problem: Exceptions can no longer be
reasonably handled, but only be printed. You cannot do much in a
finaliser.

Gerd

> -- 
> Philippe Wang
>    mail@philippewang.info
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
*** Searching for new projects! Need consulting for system
*** programming in Ocaml? Gerd Stolpmann can help you.
------------------------------------------------------------

Re: [Caml-list] Channels not closed on gc?

[Philippe Wang]

On Mon, May 21, 2012 at 3:31 PM, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:

>> Wouldn't it be a good idea to provide both ways?
>> i.e., one that collects automatically, in addition to the current one
>> that doesn't.
>
> You can easily do this yourself:
>
> Gc.finalise close_in ch

Certainly, but I meant on a high-level way, without the need to
explicitly use OCaml's internal API, for the sake of "non ocaml
hackers".  :)

--
Philippe Wang

Re: [Caml-list] Channels not closed on gc?

[Mehdi Dogguy]

On 21/05/12 16:18, Philippe Wang wrote:
> On Mon, May 21, 2012 at 3:31 PM, Gerd
> Stolpmann<info@gerd-stolpmann.de>  wrote:
>
>>> Wouldn't it be a good idea to provide both ways? i.e., one that
>>> collects automatically, in addition to the current one that
>>> doesn't.
>>
>> You can easily do this yourself:
>>
>> Gc.finalise close_in ch
>
> Certainly, but I meant on a high-level way, without the need to
> explicitly use OCaml's internal API, for the sake of "non ocaml
> hackers".  :)
>

IIRC, OCaml core team has always been reluctant to add
non-essential-to-the-compiler features/helpers in OCaml's stdlib. This
proposal fits the "non-essential-to-the-compiler" description :)

Besides, if I'm not mistaken, Batteries already provide such mechanism
(See module BatIO). Some "non ocaml hackers" can already use it!

Regards,

-- 
Mehdi Dogguy مهدي الدڤي
http://dogguy.org/

Re: AW: [Caml-list] Channels not closed on gc?

[Goswin von Brederlow]

Gerd Stolpmann <info@gerd-stolpmann.de> writes:

> Am 21.05.2012 13:23:36 schrieb(en) Lauri Alanko:
>> I only recently noticed that ocaml does not close open channels when  
>> they are garbage collected. This is evidently intentional behavior,  
>> but it was quite unexpected.
>> 
>> To be clear, I do think it's bad style to rely on GC for releasing OS  
>> resources, but that doesn't explain why GC shouldn't do this if the  
>> programmer has failed to explicitly close the channel. And if the  
>> intention were to _enforce_ good style, the channel finaliser would  
>> spout out an error or warning upon detecting that the channel hasn't  
>> yet been closed, instead of just silently leaking file handles like  
>> it does currently.
>> 
>> It is of course trivial to "fix" this by attaching a simple  
>> finaliser, but the fact that this is not done by default makes me  
>> suspect that there would be something fishy with this approach. So,  
>> what's the rationale for the current behavior?
>
> It's predictable.

Predictably buggy.

> Closing a channel is not only about releasing OS resources. Imagine the  
> channel is actually a pipe - closing it means to signal EOF, i.e. it's  
> a way of notifying another program about an event. This should not  
> happen behind one's back.

If your channel is a pipe and you need it to stay open then keep it
alive. If you need to control when it is closed than close it when you
want it to be close, which you can only do as long as it stays alive.

So this argument is realy not an argument.

Note that nobody suggested restricting closing of channels to just the
GC. Only to have the GC double check and close them before they leak. If
you do it right then nothing changes. But if you did it wrong the GC
will catch that.

> Closing a regular file would in deed be harmless, but there is no  
> generic way to identify such channels (in the OS).
>
> Gerd


Lets compare this feature to boundary checks for strings and arrays. In
correct code they are completly and utterly pointless since we know all
our access remains within bounds. Still we do check because we know that
not all code is correct.

Same with channels (and Unix.file_descr). If your code is correct and
you properly close them all correctly it is pointless for the GC to
check and close them. But still this should be done to catch those cases
where the programmer forgot about it. And this check would be a lot
cheaper than boundary checks. It only checks once on destruction.

And I agree that such a case should output a warning because people
should not start to assume the GC will close the channel, it doesn't do
so determnistically nor when the program terminates. Not closing a
channel is a resource mismanagement in the code so te code should be
fixed. The GC should just help to spot such cases.

MfG
        Goswin

Re: AW: [Caml-list] Channels not closed on gc?

[Gerd Stolpmann]

Am Dienstag, den 29.05.2012, 14:08 +0200 schrieb Goswin von Brederlow:
> Gerd Stolpmann <info@gerd-stolpmann.de> writes:
> 
> > Am 21.05.2012 13:23:36 schrieb(en) Lauri Alanko:
> >> I only recently noticed that ocaml does not close open channels when  
> >> they are garbage collected. This is evidently intentional behavior,  
> >> but it was quite unexpected.
> >> 
> >> To be clear, I do think it's bad style to rely on GC for releasing OS  
> >> resources, but that doesn't explain why GC shouldn't do this if the  
> >> programmer has failed to explicitly close the channel. And if the  
> >> intention were to _enforce_ good style, the channel finaliser would  
> >> spout out an error or warning upon detecting that the channel hasn't  
> >> yet been closed, instead of just silently leaking file handles like  
> >> it does currently.
> >> 
> >> It is of course trivial to "fix" this by attaching a simple  
> >> finaliser, but the fact that this is not done by default makes me  
> >> suspect that there would be something fishy with this approach. So,  
> >> what's the rationale for the current behavior?
> >
> > It's predictable.
> 
> Predictably buggy.

My argument is that closing the channel would be worse than current
behavior, because the close action can cause additional effects, like
that the program at the other end of the pipe interprets the EOF it sees
then. If the close is unintentional (by forgetting to keep the channel
alive), this will be surprising. If the close is intentional (by a
missing explicit close), the close will happen at an unpredictable point
in time. Both effects make it difficult to diagnose the problem, and
will probably be recognized as "random effects" by the programmer.

> > Closing a channel is not only about releasing OS resources. Imagine the  
> > channel is actually a pipe - closing it means to signal EOF, i.e. it's  
> > a way of notifying another program about an event. This should not  
> > happen behind one's back.
> 
> If your channel is a pipe and you need it to stay open then keep it
> alive. If you need to control when it is closed than close it when you
> want it to be close, which you can only do as long as it stays alive.
> 
> So this argument is realy not an argument.

I cannot follow. We discuss here how to handle a programming error, like
forgetting to keep the channel open, or forgetting to close the channel
explicitly, and the question whether the GC is helpful handling this.

> Note that nobody suggested restricting closing of channels to just the
> GC. Only to have the GC double check and close them before they leak. If
> you do it right then nothing changes. But if you did it wrong the GC
> will catch that.
> 
> > Closing a regular file would in deed be harmless, but there is no  
> > generic way to identify such channels (in the OS).
> >
> > Gerd
> 
> 
> Lets compare this feature to boundary checks for strings and arrays. In
> correct code they are completly and utterly pointless since we know all
> our access remains within bounds. Still we do check because we know that
> not all code is correct.
> 
> Same with channels (and Unix.file_descr). If your code is correct and
> you properly close them all correctly it is pointless for the GC to
> check and close them. But still this should be done to catch those cases
> where the programmer forgot about it. And this check would be a lot
> cheaper than boundary checks. It only checks once on destruction.
> 
> And I agree that such a case should output a warning because people
> should not start to assume the GC will close the channel, it doesn't do
> so determnistically nor when the program terminates. Not closing a
> channel is a resource mismanagement in the code so te code should be
> fixed. The GC should just help to spot such cases.

This is almost my position :-) You can output a warning in such cases
using Gc.finalise (which is well-documented official function, btw, so
nothing "internal" or so as I read here on the list). We could now only
argue whether this should be the default or not.

Gerd

> MfG
>         Goswin
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
*** Searching for new projects! Need consulting for system
*** programming in Ocaml? Gerd Stolpmann can help you.
------------------------------------------------------------

Re: [Caml-list] Channels not closed on gc?

[Jérémie Dimino]

Le Tue, 29 May 2012 14:08:55 +0200,
Goswin von Brederlow <goswin-v-b@web.de> a écrit :

> If your channel is a pipe and you need it to stay open then keep it
> alive. If you need to control when it is closed than close it when you
> want it to be close, which you can only do as long as it stays alive.
> 
> So this argument is realy not an argument.
> 
> Note that nobody suggested restricting closing of channels to just the
> GC. Only to have the GC double check and close them before they leak.
> If you do it right then nothing changes. But if you did it wrong the
> GC will catch that.

Sometimes you have a file descriptor and you temporary create a
channel for it. Once you are done with the channel you flush it (if it
is an output channel) but you do not close it and continue to use the
file descriptor. In this case automatically closing the channel (and
so the underlying file descriptor) would be wrong.

> Lets compare this feature to boundary checks for strings and arrays.
> In correct code they are completly and utterly pointless since we
> know all our access remains within bounds. Still we do check because
> we know that not all code is correct.
> 
> Same with channels (and Unix.file_descr). If your code is correct and
> you properly close them all correctly it is pointless for the GC to
> check and close them. But still this should be done to catch those
> cases where the programmer forgot about it. And this check would be a
> lot cheaper than boundary checks. It only checks once on destruction.
> 
> And I agree that such a case should output a warning because people
> should not start to assume the GC will close the channel, it doesn't
> do so determnistically nor when the program terminates. Not closing a
> channel is a resource mismanagement in the code so te code should be
> fixed. The GC should just help to spot such cases.

Not if you consider that a buffered channel is just an helper data
structure on top of a read/write function.

Cheers,

-- 
Jérémie

Re: AW: [Caml-list] Channels not closed on gc?

[oliver]

On Tue, May 29, 2012 at 02:46:04PM +0200, Gerd Stolpmann wrote:
> Am Dienstag, den 29.05.2012, 14:08 +0200 schrieb Goswin von Brederlow:
> > Gerd Stolpmann <info@gerd-stolpmann.de> writes:
> > 
> > > Am 21.05.2012 13:23:36 schrieb(en) Lauri Alanko:
> > >> I only recently noticed that ocaml does not close open channels when  
> > >> they are garbage collected. This is evidently intentional behavior,  
> > >> but it was quite unexpected.
[...]



[...]
> > Lets compare this feature to boundary checks for strings and arrays. In
> > correct code they are completly and utterly pointless since we know all
> > our access remains within bounds. Still we do check because we know that
> > not all code is correct.
> > 
> > Same with channels (and Unix.file_descr). If your code is correct and
> > you properly close them all correctly it is pointless for the GC to
> > check and close them. But still this should be done to catch those cases
> > where the programmer forgot about it. And this check would be a lot
> > cheaper than boundary checks. It only checks once on destruction.
> > 
> > And I agree that such a case should output a warning because people
> > should not start to assume the GC will close the channel, it doesn't do
> > so determnistically nor when the program terminates. Not closing a
> > channel is a resource mismanagement in the code so te code should be
> > fixed. The GC should just help to spot such cases.
> 
> This is almost my position :-) You can output a warning in such cases
> using Gc.finalise (which is well-documented official function, btw, so
> nothing "internal" or so as I read here on the list). We could now only
> argue whether this should be the default or not.
[...]

Because the filehandles are ressources from the system,
not just something like a value inside the users code,
I think this should not be done implicitly.

The already mentioned side effects with Pipes are one
reason. I remember the discussion was on this list
many years before, and there were mentioned more reasons
on why this should be done explicitly.


So, doing it explicitly is fine.

Other languages with GC also handle this case explicitly,
and surely not because the designers just forgot about it.


Ciao,
   Oliver

Re: AW: [Caml-list] Channels not closed on gc?

[Török Edwin]

On 2012-05-29 17:13, oliver wrote:
> On Tue, May 29, 2012 at 02:46:04PM +0200, Gerd Stolpmann wrote:
>> Am Dienstag, den 29.05.2012, 14:08 +0200 schrieb Goswin von Brederlow:
>>> Gerd Stolpmann <info@gerd-stolpmann.de> writes:
>>>
>>>> Am 21.05.2012 13:23:36 schrieb(en) Lauri Alanko:
>>>>> I only recently noticed that ocaml does not close open channels when  
>>>>> they are garbage collected. This is evidently intentional behavior,  
>>>>> but it was quite unexpected.
> [...]
> 
> 
> 
> [...]
>>> Lets compare this feature to boundary checks for strings and arrays. In
>>> correct code they are completly and utterly pointless since we know all
>>> our access remains within bounds. Still we do check because we know that
>>> not all code is correct.
>>>
>>> Same with channels (and Unix.file_descr). If your code is correct and
>>> you properly close them all correctly it is pointless for the GC to
>>> check and close them. But still this should be done to catch those cases
>>> where the programmer forgot about it. And this check would be a lot
>>> cheaper than boundary checks. It only checks once on destruction.
>>>
>>> And I agree that such a case should output a warning because people
>>> should not start to assume the GC will close the channel, it doesn't do
>>> so determnistically nor when the program terminates. Not closing a
>>> channel is a resource mismanagement in the code so te code should be
>>> fixed. The GC should just help to spot such cases.
>>
>> This is almost my position :-) You can output a warning in such cases
>> using Gc.finalise (which is well-documented official function, btw, so
>> nothing "internal" or so as I read here on the list). We could now only
>> argue whether this should be the default or not.
> [...]
> 
> Because the filehandles are ressources from the system,
> not just something like a value inside the users code,
> I think this should not be done implicitly.
> 
> The already mentioned side effects with Pipes are one
> reason. I remember the discussion was on this list
> many years before, and there were mentioned more reasons
> on why this should be done explicitly.
> 
> 
> So, doing it explicitly is fine.
> 
> Other languages with GC also handle this case explicitly,
> and surely not because the designers just forgot about it.
> 

To make sure that filehandles are closed properly I've lately written
with_file functions that take a filename and a function.
It then opens the file, invokes the function (in a try block), and closes the file (even if the try block threw an exception).
That way no file handles leak, and the GC is not needed to clean up.

This is in some ways similar to RAII in C++ where destructors automatically clean up resources when you leave a scope.
Maybe there should be a small library that provides these convenience functions (other resources can be handled similarly,
opendir/closedir comes to mind, but there are probably other cases where an explicit close-like functionality is needed).

Of course this can only be done if you deal with the file in a well-contained location, it would be harder to use
if you open the file in one place, and need to use it in several different places throughout your program.

Best regards,
--Edwin

Re: AW: [Caml-list] Channels not closed on gc?

[Philippe Veber]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

To make sure that filehandles are closed properly I've lately written
> with_file functions that take a filename and a function.
> It then opens the file, invokes the function (in a try block), and closes
> the file (even if the try block threw an exception).
> That way no file handles leak, and the GC is not needed to clean up.
>
> This is in some ways similar to RAII in C++ where destructors
> automatically clean up resources when you leave a scope.
> Maybe there should be a small library that provides these convenience
> functions (other resources can be handled similarly,
> opendir/closedir comes to mind, but there are probably other cases where
> an explicit close-like functionality is needed).
>

Those functions are defined in Batteries, see BatPervasives.with_dispose (
http://ocaml-batteries-team.github.com/batteries-included/hdoc/BatPervasives.html),
BatFile.with_file_in and BatFile.with_file_out (
http://ocaml-batteries-team.github.com/batteries-included/hdoc/BatFile.html
).

They are indeed very useful !
Cheers,
ph.

[-- Attachment #2: Type: text/html, Size: 1446 bytes --]