* File descriptor madness @ 2002-05-09 18:15 Peter Stephenson 2002-05-09 18:47 ` Danek Duvall 0 siblings, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2002-05-09 18:15 UTC (permalink / raw) To: Zsh hackers list Playing with ztcp and zselect has raised my awareness of file descriptor usage to a higher plane. In particular, getpwuid() on Solaris (2.8) is opening an fd to its new (so new it's not properly documented) `doors' system as an interface to NIS. If I remember right, we've always had some sort of problem with NIS and its use of file descriptors. In this case it's tying up fd 3, right in the paltry set of 9 fd's we have available. It never releases it. (endpwent() doesn't help, by the way, in case you thought it might.) One solution to this problem is to open (or dup something such as fd 0 onto) all fd's 3 to 9 during initialisation and then close them again before any user code is executed (keeping careful track of any that were already open). It's not particularly efficient, but not *that* inefficient either. Any comments? -- Peter Stephenson <pws@csr.com> Software Engineer CSR Ltd., Science Park, Milton Road, Cambridge, CB4 0WH, UK Tel: +44 (0)1223 392070 ********************************************************************** The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. ********************************************************************** ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: File descriptor madness 2002-05-09 18:15 File descriptor madness Peter Stephenson @ 2002-05-09 18:47 ` Danek Duvall 2002-05-09 19:08 ` Bart Schaefer 0 siblings, 1 reply; 9+ messages in thread From: Danek Duvall @ 2002-05-09 18:47 UTC (permalink / raw) To: Zsh hackers list Just checked the source for this, and it doesn't look like this is scheduled for a change any time soon; this behavior is on purpose. That said, you may be able to get away with closing the newly opened file descriptor afterwards, but I think it would be better to artificially force getpwuid() to use a higher fd, or perhaps dup2() to move it after the call. I'm sure it's been explained before, but why does the limit of 9 fds exist? Is it just because of the limitations in redirection constructs? Danek ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: File descriptor madness 2002-05-09 18:47 ` Danek Duvall @ 2002-05-09 19:08 ` Bart Schaefer 2002-05-09 18:57 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2002-05-09 19:08 UTC (permalink / raw) To: Zsh hackers list On Thu, 9 May 2002, Peter Stephenson wrote: > In this case it's tying up fd 3, right in the paltry set of 9 fd's we > have available. [...] > One solution to this problem is to open (or dup something such as fd 0 > onto) all fd's 3 to 9 during initialisation and then close them again > before any user code is executed It'd have to be an actual open() or the equivalent, because there's no way to guarantee that any particular descriptor (even 0) is valid when the shell starts up. What's the cheapest/fastest way to create descriptors -- pipe(), perhaps? On Thu, 9 May 2002, Danek Duvall wrote: > That said, you may be able to get away with closing the newly opened > file descriptor afterwards, but I think it would be better to > artificially force getpwuid() to use a higher fd, or perhaps dup2() to > move it after the call. Moving it after the call won't help, as something in the library almost certainly will still be referring to the old number. > I'm sure it's been explained before, but why does the limit of 9 fds > exist? Is it just because of the limitations in redirection constructs? It's entirely because of limitations in the redirection syntax. Really, there isn't any limit of 10 (0 through 9) -- this works fine if fd 12 is already open for writing: echo twelve >&12 However, only a single digit to the *left* of the redirect operator can be parsed as an fd number without breaking sh syntax. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: File descriptor madness 2002-05-09 19:08 ` Bart Schaefer @ 2002-05-09 18:57 ` Peter Stephenson 2002-05-09 20:36 ` Bart Schaefer 0 siblings, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2002-05-09 18:57 UTC (permalink / raw) To: Zsh hackers list Bart Schaefer wrote: > On Thu, 9 May 2002, Peter Stephenson wrote: > > > In this case it's tying up fd 3, right in the paltry set of 9 fd's we > > have available. [...] > > One solution to this problem is to open (or dup something such as fd 0 > > onto) all fd's 3 to 9 during initialisation and then close them again > > before any user code is executed > > It'd have to be an actual open() or the equivalent, because there's no way > to guarantee that any particular descriptor (even 0) is valid when the > shell starts up. Apart from 0, which I suppose we need to test with open(), I had in mind just using dup() to dup 0 and see where the fd ended up. This would in one go show open fd's (avoided) and open new fd's to block them. Or doesn't that work? -- Peter Stephenson <pws@pwstephenson.fsnet.co.uk> Work: pws@csr.com Web: http://www.pwstephenson.fsnet.co.uk ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: File descriptor madness 2002-05-09 18:57 ` Peter Stephenson @ 2002-05-09 20:36 ` Bart Schaefer 2002-05-10 17:24 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2002-05-09 20:36 UTC (permalink / raw) To: Peter Stephenson; +Cc: Zsh hackers list On Thu, 9 May 2002, Peter Stephenson wrote: > Apart from 0, which I suppose we need to test with open(), I had in mind > just using dup() to dup 0 and see where the fd ended up. This would in > one go show open fd's (avoided) and open new fd's to block them. Or > doesn't that work? Just one dup() will only show you where the first hole in the sequence of fd numbers is, not whether there are any more descriptors open above that. So you'd have to do (psuedo-code) if (!is_a_valid_fd(0)) { was_closed[0] = open(...); } while ((i = dup(0)) < 10) { was_closed[i] = 1; } /* ... initialize the shell ... */ for (i = 0; i < 10; ++i) { if (was_closed[i]) close(i); } I was merely wondering aloud whether it would be faster to do for (i = 0; !pipe(&just_opened[i]); ++i) { if (just_opened[++i]) >= 9) break; } /* ... initialize the shell ... */ while (i >= 0) { close(just_opened[i--]); } Replace pipe() with something else if there's a faster way to create new descriptors. It's possible/probable that dup() is a lot faster, but it'd be nice not to need to hit the filesystem for that initial open(...) in the first example. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: File descriptor madness 2002-05-09 20:36 ` Bart Schaefer @ 2002-05-10 17:24 ` Peter Stephenson 2002-05-10 17:41 ` Peter Stephenson 2002-05-10 17:57 ` Slightly OT " Borsenkow Andrej 0 siblings, 2 replies; 9+ messages in thread From: Peter Stephenson @ 2002-05-10 17:24 UTC (permalink / raw) To: Zsh hackers list Bart Schaefer wrote: > On Thu, 9 May 2002, Peter Stephenson wrote: > > > Apart from 0, which I suppose we need to test with open(), I had in mind > > just using dup() to dup 0 and see where the fd ended up. This would in > > one go show open fd's (avoided) and open new fd's to block them. Or > > doesn't that work? > > Just one dup() will only show you where the first hole in the sequence of > fd numbers is, not whether there are any more descriptors open above that. Yes, I thought of that. Here's an attempt. It seems to work, according to lsof. I can't think of a good test of whether it's necessary, though. Perhaps there are enough systems potentially using NIS or something similar that it's always reasonable. Index: Src/init.c =================================================================== RCS file: /cvsroot/zsh/zsh/Src/init.c,v retrieving revision 1.22 diff -u -r1.22 init.c --- Src/init.c 6 May 2002 14:50:13 -0000 1.22 +++ Src/init.c 10 May 2002 17:16:11 -0000 @@ -613,14 +613,11 @@ #endif struct timezone dummy_tz; char *ptr; -#ifdef HAVE_GETRLIMIT - int i; -#endif + int i, j; #if defined(SITEFPATH_DIR) || defined(FPATH_DIR) char **fpathptr; # if defined(FPATH_DIR) && defined(FPATH_SUBDIRS) char *fpath_subdirs[] = FPATH_SUBDIRS; - int j; # endif # ifdef SITEFPATH_DIR int fpathlen = 1; @@ -628,6 +625,43 @@ int fpathlen = 0; # endif #endif + int close_fds[10], tmppipe[2]; + + /* + * Workaround a problem with NIS (in one guise or another) which + * grabs file descriptors and keeps them for future reference. + * We don't want these to be in the range where the user can + * open fd's, i.e. 0 to 9 inclusive. So we make sure all + * fd's in that range are in use. + */ + memset(close_fds, 0, 10*sizeof(int)); + if (pipe(tmppipe) == 0) { + /* + * Strategy: Make sure we have at least fd 0 open (hence + * the pipe). From then on, keep dup'ing until we are + * up to 9. If we go over the top, close immediately, else + * mark for later closure. + */ + i = -1; /* max fd we have checked */ + while (i < 9) { + /* j is current fd */ + if (i < tmppipe[0]) + j = tmppipe[0]; + else if (i < tmppipe[1]) + j = tmppipe[1]; + else { + j = dup(0); + if (j == -1) + break; + } + if (j < 10) + close_fds[j] = 1; + else + close(j); + if (i < j) + i = j; + } + } addhookdefs(argzero, zshhooks, sizeof(zshhooks)/sizeof(*zshhooks)); @@ -813,6 +847,11 @@ } times(&shtms); + + /* Close the file descriptors we opened to block off 0 to 9 */ + for (i = 0; i < 10; i++) + if (close_fds[i]) + close(i); } /* Initialize signal handling */ -- Peter Stephenson <pws@csr.com> Software Engineer CSR Ltd., Science Park, Milton Road, Cambridge, CB4 0WH, UK Tel: +44 (0)1223 392070 ********************************************************************** The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. ********************************************************************** ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: File descriptor madness 2002-05-10 17:24 ` Peter Stephenson @ 2002-05-10 17:41 ` Peter Stephenson 2002-05-13 9:38 ` Peter Stephenson 2002-05-10 17:57 ` Slightly OT " Borsenkow Andrej 1 sibling, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2002-05-10 17:41 UTC (permalink / raw) To: Zsh hackers list Peter Stephenson wrote: > Yes, I thought of that. Here's an attempt. It seems to work, according > to lsof. I missed the case where all 10 fd's are already used, in which case the pipe's descriptors leak. Add if (i < tmppipe[0]) close(tmppipe[0]); if (i < tmppipe[1]) close(tmppipe[1]); and stir. -- Peter Stephenson <pws@csr.com> Software Engineer CSR Ltd., Science Park, Milton Road, Cambridge, CB4 0WH, UK Tel: +44 (0)1223 392070 ********************************************************************** The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. ********************************************************************** ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: File descriptor madness 2002-05-10 17:41 ` Peter Stephenson @ 2002-05-13 9:38 ` Peter Stephenson 0 siblings, 0 replies; 9+ messages in thread From: Peter Stephenson @ 2002-05-13 9:38 UTC (permalink / raw) To: Zsh hackers list Peter Stephenson wrote: > Peter Stephenson wrote: > > Yes, I thought of that. Here's an attempt. It seems to work, according > > to lsof. > > I missed the case where all 10 fd's are already used, in which case the > pipe's descriptors leak. I've committed these two changes. If anyone notices odd behaviour on start-up or with file descriptors this can be reviewed. -- Peter Stephenson <pws@csr.com> Software Engineer CSR Ltd., Science Park, Milton Road, Cambridge, CB4 0WH, UK Tel: +44 (0)1223 392070 ********************************************************************** The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. ********************************************************************** ^ permalink raw reply [flat|nested] 9+ messages in thread
* Slightly OT Re: File descriptor madness 2002-05-10 17:24 ` Peter Stephenson 2002-05-10 17:41 ` Peter Stephenson @ 2002-05-10 17:57 ` Borsenkow Andrej 1 sibling, 0 replies; 9+ messages in thread From: Borsenkow Andrej @ 2002-05-10 17:57 UTC (permalink / raw) To: Peter Stephenson; +Cc: Zsh hackers list В Птн, 10.05.2002, в 21:24, Peter Stephenson написал: > + memset(close_fds, 0, 10*sizeof(int)); Do we really assume memset is unconditionally available? Well, judging by the fact there were no bug reports it must be on all modern systems. -andrej ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2002-05-13 9:39 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2002-05-09 18:15 File descriptor madness Peter Stephenson 2002-05-09 18:47 ` Danek Duvall 2002-05-09 19:08 ` Bart Schaefer 2002-05-09 18:57 ` Peter Stephenson 2002-05-09 20:36 ` Bart Schaefer 2002-05-10 17:24 ` Peter Stephenson 2002-05-10 17:41 ` Peter Stephenson 2002-05-13 9:38 ` Peter Stephenson 2002-05-10 17:57 ` Slightly OT " Borsenkow Andrej
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).