Hello, I have observed warning messages from dns server: dns 30792: warning process exceeded 100 file descriptors dns 30888: warning process exceeded 200 file descriptors … probably the file descriptor leak comes from dnresolve.c udpquery(Query *qp, char *mntpt, int depth, int patient, int inns) { … msg = system(open("/dev/null", ORDWR), "outside"); … } char * system(int fd, char *cmd) { int pid, p, i; static Waitmsg msg; if((pid = fork()) == -1) sysfatal("fork failed: %r"); else if(pid == 0){ dup(fd, 0); close(fd); for (i = 3; i < 200; i++) close(i); /* don't leak fds */ execl("/bin/rc", "rc", "-c", cmd, nil); sysfatal("exec rc: %r"); } for(p = waitpid(); p >= 0; p = waitpid()) if(p == pid) return msg.msg; return "lost child"; } fd is lost if pid > 0 my server is running on 9front. however both 9atom and bell-labs use same routine. Kenji Arisawa
[-- Attachment #1: Type: text/plain, Size: 1406 bytes --] I've had this problem too, I have yet to resolve it. On Feb 16, 2016 10:54 AM, "arisawa" <arisawa@ar.aichi-u.ac.jp> wrote: > Hello, > > I have observed warning messages from dns server: > dns 30792: warning process exceeded 100 file descriptors > dns 30888: warning process exceeded 200 file descriptors > … > > probably the file descriptor leak comes from dnresolve.c > > udpquery(Query *qp, char *mntpt, int depth, int patient, int inns) > { > … > msg = system(open("/dev/null", ORDWR), "outside"); > … > } > > char * > system(int fd, char *cmd) > { > int pid, p, i; > static Waitmsg msg; > > if((pid = fork()) == -1) > sysfatal("fork failed: %r"); > else if(pid == 0){ > dup(fd, 0); > close(fd); > for (i = 3; i < 200; i++) > close(i); /* don't leak fds */ > execl("/bin/rc", "rc", "-c", cmd, nil); > sysfatal("exec rc: %r"); > } > for(p = waitpid(); p >= 0; p = waitpid()) > if(p == pid) > return msg.msg; > return "lost child"; > } > > fd is lost if pid > 0 > > my server is running on 9front. however both 9atom and bell-labs use same > routine. > > Kenji Arisawa > > > > [-- Attachment #2: Type: text/html, Size: 1982 bytes --]
> I've had this problem too, I have yet to resolve it.
You could start by tossing the very first and totally superfluous
"else". to simplify things.
Then, it would be tempting to take the
dup(fd,0); close(fd);
out to before the if(pid==0)...
Well spotted, Kenji. Let's hope my shot in the dark is half as good :-)
Lucio.
[-- Attachment #1: Type: text/plain, Size: 30 bytes --] what's your fucking problem? [-- Attachment #2: Type: text/html, Size: 51 bytes --]
[-- Attachment #1: Type: text/plain, Size: 1278 bytes --] On 16 February 2016 at 16:42, <lucio@proxima.alt.za> wrote: > Then, it would be tempting to take the > > dup(fd,0); close(fd); > > out to before the if(pid==0)... > the idea is to have fd (/dev/null in this case) be standard input in the new process, so it needs to follow the pid==0 test. the "outside" command is one that wasn't distributed (I assume it bound a separate #I interface on /net.alt and set it up), so the code currently probably doesn't do anything useful elsewhere. > probably the file descriptor leak comes from dnresolve.c you can cat /proc/$dnspid/fd where dnspid is the process id one or more of the active dns processes, to see which files are open, after the message appears. if there are many /dev/null open, that suggests your idea was right. i think you're right that it leaks an fd to /dev/null in that system call, so it should instead open /dev/null separately and assign fd before the call and close it afterwards. even so, i wonder if that's really what's happening in every case of "more than N fds", because the call to outside is only needed in the case that the udp under /net.alt is being used and an open there has failed. still, looking at the /proc/N/fd file should help decide that. [-- Attachment #2: Type: text/html, Size: 2588 bytes --]
its worse. the static msg.msg is useless, its never set anywhere. the interface of system() makes no sense. its only used for running "outside" and the parent proc doesnt need the fd to /dev/null, it could as well just open it in the child like: close(0); open("/dev/null", OREAD); the caller goes at some lengths queueing processes waiting for the remount to complete, but does it in two levels. thers a sleep and a qlock. limitation on the number of processes that can be queued on a qlock()? also, when fork() returns -1, it kills the parent. the whole thing about remounting /net.alt seems wrong. it probably fixed some issue at the labs with /net.alt being imported from some other machine that kept rebooting or something. after spending 5 minutes writing the code fixing all these issues mentiond above, i'll just throw it all away and delete the whole remounting logic for /net.alt in 9front. -- cinap
> limitation on the number of processes that can be queued on a qlock()?
in user space, yes.
- erik
[-- Attachment #1: Type: text/plain, Size: 1164 bytes --] On 16 February 2016 at 18:01, <cinap_lenrek@felloff.net> wrote: > and the parent proc doesnt need the fd to /dev/null, it could as well just > open it in the child like: > > close(0); open("/dev/null", OREAD); > > There's no harm in making and using a more general function, even in a specific way, so that part's ok. The caller just needs to play its part properly. after spending 5 minutes writing the code fixing all these issues mentiond > above, i'll just throw it all away and delete the whole remounting logic > for /net.alt in 9front. It's often better to use the Erlang fail-fast ("just fail") and restart approach for persistent services. More important would be to look at /proc/N/fd on a failing system. I've a feeling that the system/outside stuff isn't actually the problem, since I've seen the diagnostic on a system that wasn't using /net.alt. In that case, the problem (as I remember it) was that an Internet link further on was down, so no messages got through to remote DNS, and file descriptors were building up in slave processes waiting for replies on /net/udp. Once the link was up, it went back to normal. [-- Attachment #2: Type: text/html, Size: 2339 bytes --]
[-- Attachment #1: Type: text/plain, Size: 1210 bytes --] On 16 February 2016 at 15:52, arisawa <arisawa@ar.aichi-u.ac.jp> wrote: > > I have observed warning messages from dns server: > dns 30792: warning process exceeded 100 file descriptors > dns 30888: warning process exceeded 200 file descriptors > It's worth noting that this message doesn't necessarily mean you've got a file descriptor leak. It might, but at the start it just means that a process (or process group sharing a file descriptor group) has many file descriptors open. That could happen if a server with many clients has a file descriptor per client and then client requests open some more. ndb/dns in particular will try user-level requests in parallel, and those in turn can lead to several concurrent queries to various name servers at a given level (root itself has 13). Web browser clients will fetch page elements concurrently. That's why it's useful to check the /proc/N/fd file to try to see what they are. (Not just for ndb/dns, but for other applications that provoke the message.) Arguably, if you're using the system in a real, Internet-facing application, the warning message might be obsolete, compared to the time when even 100 clients was a big network. [-- Attachment #2: Type: text/html, Size: 1791 bytes --]
the logic (code? usage?) of system(int fd, char *cmd) is ugly.
thanks for fixing, cinap.
charies may be speaking right point, because the dns server is running only for my home use.
only my family uses the dns service. my home network is very simple.I have never used /net.alt for that.
the dns had been running on file server and sometimes (once a month or so) failed into trouble.
a few weeks ago, I separated it from file server so that I can see the trouble really comes from dns.
the contents of /proc/N/fs are shown below.
it seems /dev/null is not the criminal as charies suggested.
how to find fd leakage?
thanks in advance
io% ps|grep dns
arisawa 248 0:00 0:00 348K Await dns
arisawa 249 0:00 0:00 14584K Pread dns
arisawa 250 0:12 0:12 14576K Pread dns
io% cat /proc/^(248 249 250)^/fd
/usr/arisawa
0 r M 8 (0000000000000005 0 00) 8192 1 /dev/cons
1 w c 0 (0000000000000002 0 00) 0 1 #c/cons
2 w c 0 (0000000000000002 0 00) 0 193 #c/cons
3 r c 0 (000000000000000f 0 00) 0 4 /dev/random
4 w M 22 (000000000000ce7e 54050 40) 8192 1098023 /sys/log/dns
5 r c 0 (0000000000000001 0 00) 0 6873616 /dev/bintime
6 r M 22 (000000000000929e 3098 00) 8192 10949 /lib/ndb/local
7 r M 22 (000000000000929b 47 00) 8192 10242 /lib/ndb/common
8 r I 0 (0000000000000004 3 00) 0 127 /net/ndb
9 rw | 0 (0000000000000fc1 0 00) 65536 3954 #|/data
11 w s 0 (0000000000000009 0 00) 0 2 #s/dns
/usr/arisawa
0 r M 8 (0000000000000005 0 00) 8192 1 /dev/cons
1 w c 0 (0000000000000002 0 00) 0 1 #c/cons
2 w c 0 (0000000000000002 0 00) 0 193 #c/cons
3 r c 0 (000000000000000f 0 00) 0 4 /dev/random
4 w M 22 (000000000000ce7e 54050 40) 8192 1098023 /sys/log/dns
5 r c 0 (0000000000000001 0 00) 0 6873816 /dev/bintime
6 r M 22 (000000000000929e 3098 00) 8192 10949 /lib/ndb/local
7 r M 22 (000000000000929b 47 00) 8192 10242 /lib/ndb/common
8 r I 0 (0000000000000004 3 00) 0 127 /net/ndb
9 rw | 0 (0000000000000fc1 0 00) 65536 3954 #|/data
11 w s 0 (0000000000000009 0 00) 0 2 #s/dns
12 rw I 0 (000000000002002d 0 00) 0 3362 /net/udp/1/data
/usr/arisawa
0 r M 8 (0000000000000005 0 00) 8192 1 /dev/cons
1 w c 0 (0000000000000002 0 00) 0 1 #c/cons
2 w c 0 (0000000000000002 0 00) 0 193 #c/cons
3 r c 0 (000000000000000f 0 00) 0 4 /dev/random
4 w M 22 (000000000000ce7e 54050 40) 8192 1098023 /sys/log/dns
5 r c 0 (0000000000000001 0 00) 0 6873824 /dev/bintime
6 r M 22 (000000000000929e 3098 00) 8192 10949 /lib/ndb/local
7 r M 22 (000000000000929b 47 00) 8192 10242 /lib/ndb/common
8 r I 0 (0000000000000004 3 00) 0 127 /net/ndb
9 rw | 0 (0000000000000fc1 0 00) 65536 3954 #|/data
11 w s 0 (0000000000000009 0 00) 0 2 #s/dns
12 rw I 0 (000000000002002d 0 00) 0 3362 /net/udp/1/data
io%
> 2016/02/17 7:24、Charles Forsyth <charles.forsyth@gmail.com> のメール:
>
>
> On 16 February 2016 at 15:52, arisawa <arisawa@ar.aichi-u.ac.jp> wrote:
>
> I have observed warning messages from dns server:
> dns 30792: warning process exceeded 100 file descriptors
> dns 30888: warning process exceeded 200 file descriptors
>
> It's worth noting that this message doesn't necessarily mean you've got a file descriptor leak.
> It might, but at the start it just means that a process (or process group sharing a file descriptor group) has many file descriptors open.
> That could happen if a server with many clients has a file descriptor per client and then client requests open some more.
>
> ndb/dns in particular will try user-level requests in parallel, and those in turn can lead to several concurrent
> queries to various name servers at a given level (root itself has 13). Web browser clients will fetch page
> elements concurrently. That's why it's useful to check the /proc/N/fd file to try to see what they are.
> (Not just for ndb/dns, but for other applications that provoke the message.)
> Arguably, if you're using the system in a real, Internet-facing application, the warning message might
> be obsolete, compared to the time when even 100 clients was a big network.
looks good. thanks for reporting mr. arisawa. -- cinap
> It's often better to use the Erlang fail-fast ("just fail") and restart
> approach for persistent services.
indeed. ndb/dns also has this "restart" ctl message where it restarts the
server part but keeps the 9p pipe posted. tho that kills all the fid's and
new fids starts as "dummy" user owned file, not a directory, so after restart
/net/dns vanishes.
doing some further experiments, it appears that when i make new fids start
out as directories it works. :-)
ndb/cs also has some logic where it remounts /srv/dns when opening /net/dns
fails.
--
cinap
On Tue Feb 16 13:19:29 PST 2016, charles.forsyth@gmail.com wrote:
> On 16 February 2016 at 18:01, <cinap_lenrek@felloff.net> wrote:
>
> > and the parent proc doesnt need the fd to /dev/null, it could as well just
> > open it in the child like:
> >
> > close(0); open("/dev/null", OREAD);
> >
> >
> There's no harm in making and using a more general function, even in a
> specific way, so that part's ok.
> The caller just needs to play its part properly.
>
> after spending 5 minutes writing the code fixing all these issues mentiond
> > above, i'll just throw it all away and delete the whole remounting logic
> > for /net.alt in 9front.
>
>
> It's often better to use the Erlang fail-fast ("just fail") and restart
> approach for persistent services.
>
> More important would be to look at /proc/N/fd on a failing system.
> I've a feeling that the system/outside stuff isn't actually the problem,
> since I've seen the diagnostic on a system that wasn't using /net.alt.
> In that case, the problem (as I remember it) was that an Internet link
> further on was down,
> so no messages got through to remote DNS, and file descriptors were
> building up in slave processes
> waiting for replies on /net/udp. Once the link was up, it went back to
> normal.
we saw this a lot at coraid, but never did catch a smoking-gun process.
i don't recall a perfect correlation to internet down.
- erik