Closing all open file descriptors

Closing all open file descriptors

[Dave Benjamin]

I'm writing a daemon and I would like to be able to close all open file 
descriptors. As far as I can tell, there is no way to determine the 
maximum number of file descriptors available, and neither is there a way 
to get a file descriptor from an integer. The best I've come up with is 
the following:

   for fd = 0 to 1024 do
     try Unix.close (Obj.magic fd : Unix.file_descr)
     with Unix.Unix_error _ -> ()
   done;

Is there a better way?

Thanks,
Dave

Re: [Caml-list] Closing all open file descriptors

[Erik de Castro Lopo]

Dave Benjamin wrote:

> I'm writing a daemon and I would like to be able to close all open file 
> descriptors. As far as I can tell, there is no way to determine the 
> maximum number of file descriptors available, and neither is there a way 
> to get a file descriptor from an integer. The best I've come up with is 
> the following:
> 
>    for fd = 0 to 1024 do
>      try Unix.close (Obj.magic fd : Unix.file_descr)
>      with Unix.Unix_error _ -> ()
>    done;

I remember doing this in a daemon I wrote years ago in C but didn't
do it for a daemon I wrote recently in Ocaml (much nicer for daemons
than *any* other language).

My curiosity was piqued, so I did a bit of a search and found the
rational for this here:

    http://cloud9.hedgee.com/scribbles/daemon

in particular the fghack tool which prevents a daemon that doesn't
close all file descriptors from going into the background :

    http://cr.yp.to/daemontools/fghack.html

One possible solution to this would be a function written in C and
wrapped in Ocaml that does the same as the Obj.magic hack above.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"I have long argued that the entertainment industry doesn't want
people to have computers. Computers give users too much capability,
too much flexibility, too much freedom. The entertainment industry
wants users to sit back and consume things. They are trying to
turn a computer into an Internet Entertainment Platform, along the
lines of a television or VCR. This bill is a large step in that
direction."  -- Bruce Schneier

Re: [Caml-list] Closing all open file descriptors

[Dave Benjamin]

Erik de Castro Lopo wrote:
> Dave Benjamin wrote:
> 
>> I'm writing a daemon and I would like to be able to close all open file 
>> descriptors. As far as I can tell, there is no way to determine the 
>> maximum number of file descriptors available, and neither is there a way 
>> to get a file descriptor from an integer. The best I've come up with is 
>> the following:
>>
>>    for fd = 0 to 1024 do
>>      try Unix.close (Obj.magic fd : Unix.file_descr)
>>      with Unix.Unix_error _ -> ()
>>    done;
> 
> I remember doing this in a daemon I wrote years ago in C but didn't
> do it for a daemon I wrote recently in Ocaml (much nicer for daemons
> than *any* other language).

Did your program "daemonize" at all, then? Or did you use daemontools?

> My curiosity was piqued, so I did a bit of a search and found the
> rational for this here:
> 
>     http://cloud9.hedgee.com/scribbles/daemon
> 
> in particular the fghack tool which prevents a daemon that doesn't
> close all file descriptors from going into the background :
> 
>     http://cr.yp.to/daemontools/fghack.html

I never really thought all that much about closing file descriptors 
until recently, when I discovered that Apache leaks a bunch of 
descriptors to its modules, one of which is the listening socket which 
can prevent Apache from restarting. So, I've been reading Advanced 
Programming in the UNIX Environment to try to fill in the many gaps in 
my understanding of what's going on here.

I can see how it might be annoying to try to make a daemon run on stdio 
instead, and not being able to keep the file handles open... but it 
seems like daemons could be written to work either way if designed with 
some care.

> One possible solution to this would be a function written in C and
> wrapped in Ocaml that does the same as the Obj.magic hack above.

Yeah, the whole operation could be done in a C function too, and 
probably wouldn't be much more code to write.

Thanks for the interesting links,
Dave

Re: [Caml-list] Closing all open file descriptors

[Erik de Castro Lopo]

Dave Benjamin wrote:

> Did your program "daemonize" at all, then? Or did you use daemontools?

Yes, it called Unix.fork and then in the child called Unix.setsid.

Err, what's daemontools?

> I can see how it might be annoying to try to make a daemon run on stdio 
> instead, and not being able to keep the file handles open... but it 
> seems like daemons could be written to work either way if designed with 
> some care.

My program has passes around a logging function of type "string -> unit".
If it not a daemon it uses print_endline for logging and if it is a daemon 
it uses a wrapper around syslog (via the debian/ubuntu libsyslog-ocaml-dev)
package.

> Yeah, the whole operation could be done in a C function too, and 
> probably wouldn't be much more code to write.

My daemon is only tiny, but I'm, still glad its in Ocaml.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"War is deceit."
-- Islam's prophet Mohammed (Hadith 4:268)

Re: [Caml-list] Closing all open file descriptors

[Dave Benjamin]

Erik de Castro Lopo wrote:
> Dave Benjamin wrote:
> 
>> Did your program "daemonize" at all, then? Or did you use daemontools?
> 
> Yes, it called Unix.fork and then in the child called Unix.setsid.
> 
> Err, what's daemontools?

daemontools is a service management framework by D. J. Bernstein which 
those pages you linked were partially in reference to. The basic 
philosophy is that you don't write daemons at all; you write regular 
programs that read from stdin and write to stdout. You don't fork, you 
don't setsid, you don't worry about the environment and you don't keep 
track of PID files.

(IMHO, that's why they're so worked up about closing file descriptors. 
They're trying to get daemons to stop being daemons so they can be 
controlled by daemontools, and this is one of the sticking points.)

The daemontools style is actually a little bit like functional 
programming in its use of wrapper programs. A typical daemontools 
solution to the file descriptor closing problem would be to write a tiny 
C program that closes the file descriptors and then forks and execs its 
argument, like this:

closefds mydaemon arg1 arg2 ...

where "closefds" closes the file descriptors and then runs "mydaemon" 
with the supplied arguments. This way, "closefds" would just be another 
reusable tool.

Incidentally, Linux comes with a "setsid" program that does the same 
think with fork and setsid. It doesn't seem to be available for BSD, 
though (though "nohup" is, which is similar as well).

> My program has passes around a logging function of type "string -> unit".
> If it not a daemon it uses print_endline for logging and if it is a daemon 
> it uses a wrapper around syslog (via the debian/ubuntu libsyslog-ocaml-dev)
> package.

Makes sense to me.

>> Yeah, the whole operation could be done in a C function too, and 
>> probably wouldn't be much more code to write.
> 
> My daemon is only tiny, but I'm, still glad its in Ocaml.

I agree, overall the Unix module is pretty nice to use.

Here's a port of the "daemonize" C function described in APUE to OCaml, 
and it's where my earlier code example came from:

(* Based on Adv. Programming in the UNIX Environment, 2nd Ed. *)

#use "topfind";;
#require "unix";;
#require "syslog";;

open Printf

let daemonize cmd =
   (* Clear file creation mask. *)
   ignore (Unix.umask 0);

   (* Become a session leader to lose controlling TTY. *)
   if Unix.fork () > 0 then exit 0;
   ignore (Unix.setsid ());

   (* Ensure future opens won't allocate controlling TTYs. *)
   Sys.set_signal Sys.sighup Sys.Signal_ignore;
   if Unix.fork () > 0 then exit 0;

   (* Change the current working directory to the root so
      we won't prevent file systems from being unmounted. *)
   Unix.chdir "/";

   (* Close all open file descriptors. *)
   for fd = 0 to 1024 do
     try Unix.close (Obj.magic fd : Unix.file_descr)
     with Unix.Unix_error _ -> ()
   done;

   (* Attach file descriptors 0, 1, and 2 to /dev/null. *)
   let fd0 = Unix.openfile "/dev/null" [Unix.O_RDWR] 0o666 in
   let fd1 = Unix.dup fd0 in
   let fd2 = Unix.dup fd0 in

   (* Initialize the log file. *)
   let log = Syslog.openlog cmd in
   if fd0 <> Unix.stdin || fd1 <> Unix.stdout || fd2 <> U
     begin
       Syslog.syslog log `LOG_ERR
         (sprintf "unexpected file descriptors %d %d %d"
            (Obj.magic fd0 : int)
            (Obj.magic fd1 : int)
            (Obj.magic fd2 : int));
       exit 1
     end;
   log

let () =
   let log = daemonize "daemonize" in
   Syslog.syslog log `LOG_INFO "Starting daemon";
   Unix.sleep 10;
   Syslog.syslog log `LOG_INFO "Farewell"

Cheers,
Dave

RE: [Caml-list] Closing all open file descriptors

[David Allsopp]

> I'm writing a daemon and I would like to be able to close all open file 
> descriptors. As far as I can tell, there is no way to determine the 
> maximum number of file descriptors available, and neither is there a way 
> to get a file descriptor from an integer. The best I've come up with is 
> the following:
>
>    for fd = 0 to 1024 do
>      try Unix.close (Obj.magic fd : Unix.file_descr)
>      with Unix.Unix_error _ -> ()
>    done;
>
> Is there a better way?

Probably irrelevant here, but this approach wouldn't work under Windows
(Unix.file_descr is the Win32 file handle at that point which is often
larger than 1024). More relevantly, Unix can be reconfigured to allow for
more than 1024 open files.

Wouldn't an easier, portable way be to wrap Unix.openfile and Unix.close to
maintain a set of open file_descrs or is your concern the risk of files left
open by third party code?


David

Re: [Caml-list] Closing all open file descriptors

[Dave Benjamin]

David Allsopp wrote:
> Probably irrelevant here, but this approach wouldn't work under Windows
> (Unix.file_descr is the Win32 file handle at that point which is often
> larger than 1024). More relevantly, Unix can be reconfigured to allow for
> more than 1024 open files.

Yes, I assumed that this would be non-portable, and that is yet another 
problem. It seems that dropping down to C is the only way to really do 
this right.

> Wouldn't an easier, portable way be to wrap Unix.openfile and Unix.close to
> maintain a set of open file_descrs or is your concern the risk of files left
> open by third party code?

The latter. The simple solution is to use Unix.set_close_on_exec from 
the parent process, but that only works if it's under my control.

Dave

Re: [Caml-list] Closing all open file descriptors

[Andre Nathan]

On Thu, 2007-09-13 at 23:41 -0700, Dave Benjamin wrote:
> Yes, I assumed that this would be non-portable, and that is yet another 
> problem. It seems that dropping down to C is the only way to really do 
> this right.

You could check for the availability of the closefrom() system call,
which closes descriptors starting from the one passed as its argument. I
believe the BSDs and Solaris have it.

HTH,
Andre

Re: [Caml-list] Closing all open file descriptors

[Mattias Engdegård]

>Probably irrelevant here, but this approach wouldn't work under Windows
>(Unix.file_descr is the Win32 file handle at that point which is often
>larger than 1024). More relevantly, Unix can be reconfigured to allow for
>more than 1024 open files.

I think platform-dependent code is required here. The common way of
doing this under Linux (and Solaris, probably) is to readdir /proc/PID/fd/.
Windows is of course very different.

Re: [Caml-list] Closing all open file descriptors

[Dave Benjamin]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed, Size: 803 bytes --]

On Fri, 14 Sep 2007, Mattias Engdegård wrote:

>> Probably irrelevant here, but this approach wouldn't work under Windows
>> (Unix.file_descr is the Win32 file handle at that point which is often
>> larger than 1024). More relevantly, Unix can be reconfigured to allow for
>> more than 1024 open files.
>
> I think platform-dependent code is required here. The common way of 
> doing this under Linux (and Solaris, probably) is to readdir 
> /proc/PID/fd/. Windows is of course very different.

I like this approach. It doesn't eliminate the need to convert ints to 
descrs, but at least you don't have a bunch of unnecessary system calls or 
a hard-coded maximum (assuming getrlimit is unavailable, as it is with 
OCaml currently). I wonder if /dev/fd is more or less portable...

Dave

Re: [Caml-list] Closing all open file descriptors

[Oliver Bandel]

Zitat von Dave Benjamin <dave@ramenlabs.com>:

> On Fri, 14 Sep 2007, Mattias Engdegård wrote:
>
> >> Probably irrelevant here, but this approach wouldn't work under Windows
> >> (Unix.file_descr is the Win32 file handle at that point which is often
> >> larger than 1024). More relevantly, Unix can be reconfigured to allow for
> >> more than 1024 open files.
> >
> > I think platform-dependent code is required here. The common way of
> > doing this under Linux (and Solaris, probably) is to readdir
> > /proc/PID/fd/. Windows is of course very different.
>
> I like this approach. It doesn't eliminate the need to convert ints to
> descrs,

On Unix, file-descriptors are identified by integer-values.

In OCaml, you have abstract types for file-descriptors.
So, how to convert them? I think, officially there is no way.

But if you like the concept of completely using files,
Plan9 as operating system would make sense to you. :)



> but at least you don't have a bunch of unnecessary system calls or
> a hard-coded maximum

Something like Sys.max_open_fd would be fine,
and I think there should be a way to make it cross-platform-available.



> (assuming getrlimit is unavailable, as it is with
> OCaml currently).

getrlimit seems to be a Unix-only thing.
I don't know if this is also available on other systems, like Windows or so.

So, if it is Unix-specific, this only would make sense in the Unix-modlue.

But something like the maximum number of open files / max-num of
used filedescriptors would also make sense on unix.

So, I would ask for something like a Sys.max_open_fd.
This owuld be a cross-platform thing.


> I wonder if /dev/fd is more or less portable...

Will this be available on non-Unix and non-Linux systems?

You also mentioned Win-systems?
If you also need that portability, you should not look for
Unix-only features.

Ciao,
   Oliver

Re: [Caml-list] Closing all open file descriptors

[Markus E L]

Oliver Bandel wrote:

> Zitat von Dave Benjamin <dave@ramenlabs.com>:
>
>> On Fri, 14 Sep 2007, Mattias Engdegård wrote:
>>
>> >> Probably irrelevant here, but this approach wouldn't work under Windows
>> >> (Unix.file_descr is the Win32 file handle at that point which is often
>> >> larger than 1024). More relevantly, Unix can be reconfigured to allow for
>> >> more than 1024 open files.
>> >
>> > I think platform-dependent code is required here. The common way of
>> > doing this under Linux (and Solaris, probably) is to readdir
>> > /proc/PID/fd/. Windows is of course very different.
>>
>> I like this approach. It doesn't eliminate the need to convert ints to
>> descrs,
>
> On Unix, file-descriptors are identified by integer-values.
>
> In OCaml, you have abstract types for file-descriptors.
> So, how to convert them? I think, officially there is no way.

By writing a C-function which does the conversion. This, of course
could only exist under Linux. 

Generally a function enumerate_my_open_files would not be such a bad
idea.

Regards -- Markus

Re: [Caml-list] Closing all open file descriptors

[Oliver Bandel]

Zitat von Markus E L <ls-ocaml-2006@m-e-leypold.de>:

>
> Oliver Bandel wrote:
>
> > Zitat von Dave Benjamin <dave@ramenlabs.com>:
> >
> >> On Fri, 14 Sep 2007, Mattias Engdegård wrote:
> >>
> >> >> Probably irrelevant here, but this approach wouldn't work under Windows
> >> >> (Unix.file_descr is the Win32 file handle at that point which is often
> >> >> larger than 1024). More relevantly, Unix can be reconfigured to allow
> for
> >> >> more than 1024 open files.
> >> >
> >> > I think platform-dependent code is required here. The common way of
> >> > doing this under Linux (and Solaris, probably) is to readdir
> >> > /proc/PID/fd/. Windows is of course very different.
> >>
> >> I like this approach. It doesn't eliminate the need to convert ints to
> >> descrs,
> >
> > On Unix, file-descriptors are identified by integer-values.
> >
> > In OCaml, you have abstract types for file-descriptors.
> > So, how to convert them? I think, officially there is no way.
>
> By writing a C-function which does the conversion. This, of course
> could only exist under Linux.
[...]

I think this would need to hack in the internals of OCaml.
And I don't see that it really would be necessary.
But to have the number of max_open_files IMHO would be fine.
I mentioned that it might be called Sys.max_open_descr or similarly.


>
> Generally a function enumerate_my_open_files would not be such a bad
> idea.

You can gather together the files (names or descriptors)
you have opened, if you want. After the work is done,
close the channels/files.

If you think you need all this for your daemon,
I ask: why do you think you need it? If you write it, you have
full control over all files you open, or if you are using
sockets, there also should be no problem.

Why do you want to have numbers for your opened files?
You can have a filename and get a file_descr or a channel.
You can use a Map, if you need to enumerate them.
But you also could use a Map that has filenames as keys
and file_descr or channels as value of the map.
So, if you want to close one certain channel, you
have your association.
You also could usa a PID of a child as key, or both:
a PID and a filename, or connection-ID or something like that.

Instead of looking for libraries for all kind of possible
applications, IMHO it's better to build the needed things
by your own, using the already available libraries like Map,
Set and so on.
Or if you need FIFOs for the connections, there also is a module for that
(Queue-module).

Rethink your design and you possibly will not have a necessity
for the functions you asked for.

If you would say a littlebid more about the tasks of your
daemon, one could give a more detailed answer.

Ciao,
   Oliver

Re: [Caml-list] Closing all open file descriptors

[Erik de Castro Lopo]

Oliver Bandel wrote:

> If you think you need all this for your daemon,
> I ask: why do you think you need it? If you write it, you have
> full control over all files you open

No, thats not right. I can write a program, that opens a bunch
of file descriptors, and then exec his program and his program
will inherit all open file descritors.

For instance:

    http://cr.yp.to/daemontools/fghack.html

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"The happiness you have demanded is now mandatory."
    -- Jello Biafra

Re: [Caml-list] Closing all open file descriptors

[Oliver Bandel]

Zitat von Erik de Castro Lopo <mle+ocaml@mega-nerd.com>:

> Oliver Bandel wrote:
>
> > If you think you need all this for your daemon,
> > I ask: why do you think you need it? If you write it, you have
> > full control over all files you open
>
> No, thats not right. I can write a program, that opens a bunch
> of file descriptors, and then exec his program and his program
> will inherit all open file descritors.

Yes, that's correct.
But you (as the programmer) have the control (by design)
about how to handle that case.

If you have all your open descriptors in a list,
you can close them after a fork.

In C, with fcntl, there is a possibility to close files
on an exec automatically with the close-on-exec flag.

Ciao,
   Oliver

Re: [Caml-list] Closing all open file descriptors

[Mattias Engdegård]

>If you have all your open descriptors in a list,
>you can close them after a fork.

That approach does not compose well --- descriptors opened by
a library will be excluded. The only good way is to ask the system.

Re: [Caml-list] Closing all open file descriptors

[Oliver Bandel]

Zitat von Mattias Engdegå   rd <mattias@virtutech.se>:

> >If you have all your open descriptors in a list,
> >you can close them after a fork.
>
> That approach does not compose well --- descriptors opened by
> a library will be excluded. The only good way is to ask the system.
>


What do you mean with "does not compose well"?
Do you see OCaml-specific problems here, or what?
In C, there is a function fileno(3).
Possibly this is, what people discussing
in this thread are looking for.

This would be a thing that could be included in
Ocaml, but the result would not be an integer-value,
because OCaml uses abstract types for file descriptors.
And to have buffered-io streams together with
abstract file descriptor-types and additionally having a
integer-aequivalent of it, IMHO would be an emerging mess.


When it is intended to make low-level handling
of things with the Unix-functions, one would use
Unix.open and not open_in or open_out, IMHO.
And then the file-descriptors are already available.

I'm a littlebid sceptical on using high-level and low-level
stuff together. In C it's also not recommended to mix buffered
and non-buffered IO, and so in OCaml, which is quite more
high-level stuff, I would be even more sceptical on mixing
these things together.

As a general rule I would recommend, not to mix
the different IO-possibilities.
And that's a reason, why C-functions like fileno(3)
are rarely used by myself.

Ciao,
   Oliver

Re: [Caml-list] Closing all open file descriptors

[Markus E L]

Oliver Bandel wrote:

> Zitat von Mattias Engdegå   rd <mattias@virtutech.se>:
>
>> >If you have all your open descriptors in a list,
>> >you can close them after a fork.
>>
>> That approach does not compose well --- descriptors opened by
>> a library will be excluded. The only good way is to ask the system.
>>
>
>
> What do you mean with "does not compose well"?
> Do you see OCaml-specific problems here, or what?

No.

> In C, there is a function fileno(3).
> Possibly this is, what people discussing
> in this thread are looking for.

No. You're barking up the completely wrong tree. We are looking for a
way to iterate over ALL open filedescriptors, wether inherited from
the parent process (which we don't control / haven't written
ourselves, so no recurrence to set_closeon_exec), or opened by a
library procedure or opened by our own program.

> This would be a thing that could be included in
> Ocaml, but the result would not be an integer-value,

Bah. Nobody asked for integer values.

> because OCaml uses abstract types for file descriptors.

Yes, yes, we know.

> I'm a littlebid sceptical on using high-level and low-level
> stuff together. In C it's also not recommended to mix buffered
> and non-buffered IO, 

But you _have_ to do it, how else could you do buffered IO on sockets?

> and so in OCaml, which is quite more
> high-level stuff, I would be even more sceptical on mixing
> these things together.

No, it's very similar to the relationship between stdio and unix file
descriptors.

> As a general rule I would recommend, not to mix the different
> IO-possibilities.

It's not so rare that you would have to do a select() on some I/O
channel you'd on the other side rather prefer to do buffered I/O with
instead of writing (again) your own buffering. Fortunately this
works. 

So I rate "I would not recommend" rather too strong. It works and it
is AFAI understand guaranteed to work. One just has to be prepared to
understand the interaction between buffered and unbuffered I/O, which
are a _little_ bit more complex than one type of I/O alone, but --
nonetheless -- managable.

Regards -- Markus

Re: [Caml-list] Closing all open file descriptors

[Oliver Bandel]

Zitat von Mattias Engdegå   rd <mattias@virtutech.se>:

> >If you have all your open descriptors in a list,
> >you can close them after a fork.
>
> That approach does not compose well --- descriptors opened by
> a library will be excluded.
[...]

Oh, I have not seen this last sentence.

Well, wouldn't the library offer possibilities
to close those files?

Can you give an example on where you see problems?

Ciao,
   Oliver

Re: [Caml-list] Closing all open file descriptors

[Markus E L]

Oliver Bandel wrote:

> Zitat von Mattias Engdegå   rd <mattias@virtutech.se>:
>
>> >If you have all your open descriptors in a list,
>> >you can close them after a fork.
>>
>> That approach does not compose well --- descriptors opened by
>> a library will be excluded.
> [...]
>
> Oh, I have not seen this last sentence.
>
> Well, wouldn't the library offer possibilities
> to close those files?
>
> Can you give an example on where you see problems?

GTK and the X11 library without doubt open a plethora of file
descriptors. If now you fork() in a gtk event handler -- can you tell
me what happens to those file descriptors? 

Regards -- Markus

Re: [Caml-list] Closing all open file descriptors

[skaller]

On Sat, 2007-09-15 at 16:29 +0200, Markus E L wrote:

> GTK and the X11 library without doubt open a plethora of file
> descriptors. If now you fork() in a gtk event handler -- can you tell
> me what happens to those file descriptors? 

No, so don't do it. Instead, start a separate process,
and send it a message from the GTK event handler.
Then THAT process does the fork/exec to start the daemon.

If you want to spawn a demon, get another demon to do it 
for you:)

I suggest OP examine:

DAEMON.C(1)  

and

libslack(3)

[which was written by one of my friends <raf at raf dot org>]



-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Closing all open file descriptors

[Markus E L]

"Mattias Engdegård" wrote:

>>If you have all your open descriptors in a list,
>>you can close them after a fork.
>
> That approach does not compose well --- descriptors opened by
> a library will be excluded. 

And this too.

> The only good way is to ask the system.

Absolutely. Full ACK.

Regards -- Markus

Re: [Caml-list] Closing all open file descriptors

[Markus E L]

Oliver Bandel wrote:

> Zitat von Erik de Castro Lopo <mle+ocaml@mega-nerd.com>:
>
>> Oliver Bandel wrote:
>>
>> > If you think you need all this for your daemon,
>> > I ask: why do you think you need it? If you write it, you have
>> > full control over all files you open
>>
>> No, thats not right. I can write a program, that opens a bunch
>> of file descriptors, and then exec his program and his program
>> will inherit all open file descritors.
>
> Yes, that's correct.
> But you (as the programmer) have the control (by design)
> about how to handle that case.

Let's assume (as an example) I'm starting a demon / background process
by clicking on an icon in a desktop environment. Which file
descriptors are open when the demon starts executing? Would I be
expected to patch the desktop to ensure that they aren't? Or will I
have to rely on a rather undocumented bit of the desktop software's
behaviour (perhaps subject to change) to guarantee my demons security.

This sucks on so many levels. My position is, if there is something
you inherit or can create in your process there must also be a way to
browse/index/enumerate what you inherited/created. So, yes, there
should be a enumerate_my_open_descriptors() even in POSIX. If it it
isn't it has to be reinvented in a platform specific way on every
platfrom where this is possible (e.g. under Linux using
/proc/self/fd), to get a better programming environment.

> If you have all your open descriptors in a list,
> you can close them after a fork.

And if you inherit them unkknowingly, you're lost.


> In C, with fcntl, there is a possibility to close files
> on an exec automatically with the close-on-exec flag.

This is aboout what a parent does whom one possibly doesn't control.

Regards -- Markus

Re: [Caml-list] Closing all open file descriptors

[Eric Cooper]

I think the right interface for this should be a "fold" over open file
descriptors:

    open Unix

    let descr_of_int n = (Obj.magic (n : int) : file_descr)

    let valid_descr fd =
      try ignore (fstat fd); true
      with Unix_error (EBADF, "fstat", _) -> false

    let fold_open_fds f init =
      let f' x s =
	let fd = descr_of_int (int_of_string s) in
	if valid_descr fd then f x fd else x
      in
      Array.fold_left f' init (Sys.readdir "/proc/self/fd")

Note that the readdir itself opens a file descriptor, which is
included in the result but then closed. So there's a little extra
complexity to filter it out so that the caller's function doesn't get
applied to a bad descriptor.

-- 
Eric Cooper             e c c @ c m u . e d u

Re: [Caml-list] Closing all open file descriptors

[Markus E L]

Eric Cooper wrote:

> I think the right interface for this should be a "fold" over open file
> descriptors:
>
>     open Unix
>
>     let descr_of_int n = (Obj.magic (n : int) : file_descr)
>
>     let valid_descr fd =
>       try ignore (fstat fd); true
>       with Unix_error (EBADF, "fstat", _) -> false
>
>     let fold_open_fds f init =
>       let f' x s =
> 	let fd = descr_of_int (int_of_string s) in
> 	if valid_descr fd then f x fd else x
>       in
>       Array.fold_left f' init (Sys.readdir "/proc/self/fd")
>
> Note that the readdir itself opens a file descriptor, which is
> included in the result but then closed. So there's a little extra
> complexity to filter it out so that the caller's function doesn't get
> applied to a bad descriptor.

Yes, that's a good idea. This can be used to retrieve a list where
needed or to operate on the descriptors immediately. On systems which
absolutely don't have a (system dependend) method to pull the
filedescriptors from the system, one could just iterate over a
predefined range. So this interface could be implemented on a larger
range of systems with varying efficiency.

Now the price question: Does anyone here have some insights how one
gets at the open descriptors under windows? :-)

Regards -- Markus

Re: [Caml-list] Closing all open file descriptors

[skaller]

On Sat, 2007-09-15 at 18:17 +0200, Markus E L wrote:

> Now the price question: Does anyone here have some insights how one
> gets at the open descriptors under windows? :-)

Why do you need to?

BOOL WINAPI CreateProcess(
  __in          LPCTSTR lpApplicationName,
  __in_out      LPTSTR lpCommandLine,
  __in          LPSECURITY_ATTRIBUTES lpProcessAttributes,
  __in          LPSECURITY_ATTRIBUTES lpThreadAttributes,
  __in          BOOL bInheritHandles,
  __in          DWORD dwCreationFlags,
  __in          LPVOID lpEnvironment,
  __in          LPCTSTR lpCurrentDirectory,
  __in          LPSTARTUPINFO lpStartupInfo,
  __out         LPPROCESS_INFORMATION lpProcessInformation
);

bInheritHandles
        
        If this parameter TRUE, each inheritable handle in the calling
        process is inherited by the new process. If the parameter is
        FALSE, the handles are not inherited. Note that inherited
        handles have the same value and access rights as the original
        handles.
        
-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Closing all open file descriptors

[skaller]

On Sat, 2007-09-15 at 16:16 +0200, Markus E L wrote:

> Let's assume (as an example) I'm starting a demon / background process
> by clicking on an icon in a desktop environment. Which file
> descriptors are open when the demon starts executing?

The solution to all problems in Computer Science is already known:
"add another level of indirection". [Tcl and Perl version ..
double the quotes and escape characters .. :]

In this case:

	a) the Desktop must keep track of its file descriptors F
	b) it starts Helper which is passed F and closes them
	c) Helper then starts Plugin with no open file descriptors:
	--> Plugin doesn't need to close inherited file descriptors

If Desktop doesn't do that, Desktop is screwed.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] Closing all open file descriptors

[Gerd Stolpmann]

Am Donnerstag, den 13.09.2007, 17:56 -0500 schrieb Dave Benjamin:
> I'm writing a daemon and I would like to be able to close all open file 
> descriptors. As far as I can tell, there is no way to determine the 
> maximum number of file descriptors available, and neither is there a way 
> to get a file descriptor from an integer. The best I've come up with is 
> the following:
> 
>    for fd = 0 to 1024 do
>      try Unix.close (Obj.magic fd : Unix.file_descr)
>      with Unix.Unix_error _ -> ()
>    done;
> 
> Is there a better way?

Not really. Unix wants it this way.

ocamlnet's netsys library has functions to deal with that:
Netsys.file_descr_of_int and Netsys.sysconf_open_max. Look here for the
code:

https://godirepo.camlcity.org/wwwsvn/trunk/code/src/netsys/?root=lib-ocamlnet2

This does not work under Win32, of course, but I bet this wasn't part of
your question. The Netsys functions fail in this case.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Closing all open file descriptors

[Oliver Bandel]

Zitat von Dave Benjamin <dave@ramenlabs.com>:

> I'm writing a daemon and I would like to be able to close all open file
> descriptors. As far as I can tell, there is no way to determine the
> maximum number of file descriptors available, and neither is there a way
> to get a file descriptor from an integer. The best I've come up with is
> the following:
>
>    for fd = 0 to 1024 do
>      try Unix.close (Obj.magic fd : Unix.file_descr)
>      with Unix.Unix_error _ -> ()
>    done;
>
> Is there a better way?
[...]

Yes: Do not start your daemon from a big program,
start it from scratch.
Normally only stdin, stdout and stderr are opened.
So, when your daemon starts as a separate program,
you only need to close these descriptors.

If you want to know, how many open files are allowed,
on Unix-systems you can use the MAX_OPEN value in C programs.

As a general rule you could remember: fork as early as possible.
If you have a huge memory usage and the usage of the different processes
is varied in a wide range, forking before allocating mem means to
save ressources.
If you first allocate huge memory portions and then fork,
you have wasted memory. It normally does not matter
until the moment you change your data, because modern Unix's and Linux
use COW (Copy On Write). But when you then change your data,
a lot of ressources will be needed.

So: fork early, allocate ressources afterwards.

There is no reason to do it diiferently in respect to usage of filedescriptors.

If you write a daemon, make it a separate program,
not making a fork from a huge application.

So, why should it be necessary then, to close file descriptors?
Why should they be opened, if you start a daemon as a stand-alone?

(Why should there be a need for a daemon to be forked off from
a big app, so that you could assume that many descriptors are open?)

Ciao,
   Oliver

Re: [Caml-list] Closing all open file descriptors

[Ville-Pertti Keinonen]

On Sep 14, 2007, at 1:56 AM, Dave Benjamin wrote:

>   for fd = 0 to 1024 do
>     try Unix.close (Obj.magic fd : Unix.file_descr)
>     with Unix.Unix_error _ -> ()
>   done;
>
> Is there a better way?

While I don't have a solution for your general problem (other than  
having your program launched by a known mechanism), I recommend that  
you never close the file descriptors for stdin, stdout and stderr.   
Subsequent opens will end up reusing those file descriptor numbers,  
but system libraries will still consider them valid (via libc).  This  
can cause some very strange bugs.

The safe way to detach stdio is to open /dev/null for writing, dup2  
that over stdout and stderr, open /dev/null for reading and dup2 that  
over stdin (obviously you can close the newly opened descriptors),  
thus you never run anything without valid stdio file descriptors.