file descriptor usage?

file descriptor usage?

[Scott Blachowicz]

Hi-

Our news server has finally been reconstructed and I'm playing with later 
versions of gnus and started getting wierd problems with emacs failing to load 
various files, my visiting a file only to get an error about the 
file exists, but is unreadable and such.  All of the weirdness happened after 
I did `M-x gnus' (triggering the autoload of stuff).  Finally I saw an error 
message that led me to check my 'limit' command in my shell and sure enough, 
it said that I had a limit of 64 file descriptors.  I bumped it up to 256 and 
now I can run gnus.

So, my question is...has something happened that that would cause gnus/emacs 
(19.33 & xemacs 19.14) to chew up a whole lot more file descriptors?  Like I 
said, it's been a while...this might be the first time that I've gotten any 
5.4.x release going...

Scott Blachowicz  Ph: 206/283-8802x240   Mathsoft (Data Analysis Products Div)
                                         1700 Westlake Ave N #500
scott@statsci.com                        Seattle, WA USA   98109
Scott.Blachowicz@seaslug.org

Re: file descriptor usage?

[Lars Magne Ingebrigtsen]

Scott Blachowicz <scott@bloke.statsci.com> writes:

> Our news server has finally been reconstructed and I'm playing with
> later versions of gnus and started getting wierd problems with emacs
> failing to load various files, my visiting a file only to get an
> error about the file exists, but is unreadable and such.  All of the
> weirdness happened after I did `M-x gnus' (triggering the autoload
> of stuff).  Finally I saw an error message that led me to check my
> 'limit' command in my shell and sure enough, it said that I had a
> limit of 64 file descriptors.  I bumped it up to 256 and now I can
> run gnus.

What was the exact error message?

> So, my question is...has something happened that that would cause
> gnus/emacs (19.33 & xemacs 19.14) to chew up a whole lot more file
> descriptors?  Like I said, it's been a while...this might be the
> first time that I've gotten any 5.4.x release going...

Gnus is now a gazillion small files instead of one biiig file, and I
guess it might be possible that the chain of `require's and
`autoload's would keep a big number of files open at once (while
loading). 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@ifi.uio.no * Lars Ingebrigtsen

Re: file descriptor usage?

[Sudish Joseph]

Lars Magne Ingebrigtsen writes:
> Gnus is now a gazillion small files instead of one biiig file, and I
> guess it might be possible that the chain of `require's and
> `autoload's would keep a big number of files open at once (while
> loading). 

Also try "netstat -nt | grep :119" and see how many connection gnus
has open to the news server.  Something changed in 5.4 in how gnus
compares two `implicit' servers (foreign groups not associated with an
explicitly defined virtual server) -- gnus now opens one connection
per such server.  I worked around by editing my .newsrc.eld.  You
might need to do the same, depending upon how many open connections
you see with netsat (I had 12 to the same server, where rgnus-0.82
used to open 2).

-Sudish

Re: file descriptor usage?

[Scott Blachowicz]

Sudish Joseph <sj@eng.mindspring.net> wrote:

> Also try "netstat -nt | grep :119"
Actually on this system (SunOS 5.x), it'd be more like

	netstat -n | grep '\.119'

At any rate, I haven't been able to catch it doing that with netstat, but I 
was perusing some of the messages generated by our local INN setup and found a 
LOT of connects from my system:

    Mar 24 08:47:26 junior nnrpd[17665]: bloke.statsci.com connect
    Mar 24 08:53:56 junior nnrpd[17665]: bloke.statsci.com group news.lists.filters 243
    Mar 24 08:54:33 junior nnrpd[17665]: bloke.statsci.com group news.admin.net-abuse.bulletins 25
    Mar 24 08:54:56 junior nnrpd[17895]: bloke.statsci.com connect
    Mar 24 08:55:15 junior nnrpd[17898]: bloke.statsci.com connect
    Mar 24 08:55:16 junior nnrpd[17899]: bloke.statsci.com connect
    Mar 24 08:55:18 junior nnrpd[17900]: bloke.statsci.com connect
    Mar 24 08:55:20 junior nnrpd[17901]: bloke.statsci.com connect
    Mar 24 08:55:22 junior nnrpd[17902]: bloke.statsci.com connect
    Mar 24 08:55:22 junior nnrpd[17903]: bloke.statsci.com connect
    Mar 24 08:55:23 junior nnrpd[17904]: bloke.statsci.com connect
    Mar 24 08:55:23 junior nnrpd[17905]: bloke.statsci.com connect
    Mar 24 08:55:24 junior nnrpd[17906]: bloke.statsci.com connect
    Mar 24 08:55:25 junior nnrpd[17907]: bloke.statsci.com connect
    Mar 24 08:55:29 junior nnrpd[17908]: bloke.statsci.com connect
    ...etc...
    Mar 24 08:58:13 junior nnrpd[17967]: bloke.statsci.com connect
    Mar 24 08:58:14 junior nnrpd[17968]: bloke.statsci.com connect
    Mar 24 08:58:15 junior nnrpd[17969]: bloke.statsci.com connect
    Mar 24 08:58:17 junior nnrpd[17970]: bloke.statsci.com connect
    Mar 24 09:04:56 junior nnrpd[17895]: bloke.statsci.com timeout
    Mar 24 09:04:56 junior nnrpd[17895]: bloke.statsci.com exit articles 0 groups 0
    Mar 24 09:04:56 junior nnrpd[17895]: bloke.statsci.com times user 0.100 system 0.070 elapsed 601.076
    Mar 24 09:05:16 junior nnrpd[17898]: bloke.statsci.com timeout
    Mar 24 09:05:16 junior nnrpd[17898]: bloke.statsci.com exit articles 0 groups 1
    Mar 24 09:05:16 junior nnrpd[17898]: bloke.statsci.com times user 0.100 system 0.030 elapsed 600.565
    Mar 24 09:05:16 junior nnrpd[17899]: bloke.statsci.com timeout
    Mar 24 09:05:16 junior nnrpd[17899]: bloke.statsci.com exit articles 0 groups 1
    Mar 24 09:05:16 junior nnrpd[17899]: bloke.statsci.com times user 0.090 system 0.030 elapsed 600.511
    ...etc...
    Mar 24 09:08:18 junior nnrpd[17665]: bloke.statsci.com timeout
    Mar 24 09:08:18 junior nnrpd[17665]: bloke.statsci.com exit articles 268 groups 812
    Mar 24 09:08:18 junior nnrpd[17665]: bloke.statsci.com times user 8.560 system 15.050 elapsed 1252.228

I think I might have the nocem stuff enabled...would it be doing a
bazillion separate connections for THAT?  Should I just do a `M-x
gnus-bug' to the list for diagnostic purposes?

Hmmm...my problems could be more related to the long time lag in my news
reading (news server crashed a while ago...has been re-setup...had to
delete all the article numbers from my .newsrc file).

Scott Blachowicz  Ph: 206/283-8802x240   Mathsoft (Data Analysis Products Div)
                                         1700 Westlake Ave N #500
scott@statsci.com                        Seattle, WA USA   98109
Scott.Blachowicz@seaslug.org

Re: file descriptor usage?

[Lars Magne Ingebrigtsen]

Scott Blachowicz <scott@bloke.statsci.com> writes:

> I think I might have the nocem stuff enabled...would it be doing a
> bazillion separate connections for THAT?

No, it's an, uhm, feature of how virtual servers are handled.  For
some reason, Gnus believes that you have several (similar) virtual
servers, so you get these results.  I have yet to come up with a fix. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: file descriptor usage?

[David Moore]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Scott Blachowicz <scott@bloke.statsci.com> writes:
> 
> > I think I might have the nocem stuff enabled...would it be doing a
> > bazillion separate connections for THAT?
> 
> No, it's an, uhm, feature of how virtual servers are handled.  For
> some reason, Gnus believes that you have several (similar) virtual
> servers, so you get these results.  I have yet to come up with a fix. 

What about yet another layer of indirection.  That is, something in nntp
might use the same connection for two virtual servers to the same
host/port.  Although, I guess this doesn't work if they have different
nntp-nov-is-evil settings or something.


Isn't there a function to test if two virtual servers are the same?  If
so, does it not work in these cases for some reason?  Or is it not being
used at the right places?  Or should such a thing be written?


-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | In a cloud bones of steel.

Re: file descriptor usage?

[Lars Magne Ingebrigtsen]

David Moore <dmoore@ucsd.edu> writes:

> What about yet another layer of indirection.  That is, something in nntp
> might use the same connection for two virtual servers to the same
> host/port.  Although, I guess this doesn't work if they have different
> nntp-nov-is-evil settings or something.

Actually, I think you're totally right here.  If I just un-ooize the
`nntp-connection-alist', all virtual nntp servers will use share the
same nntp connection.  Argh.  I hate it when the solution to a
difficult problem is so trivial.  :-)

> Isn't there a function to test if two virtual servers are the same?  If
> so, does it not work in these cases for some reason?

It seems not to be working from the reports I've seen, but I have been
unable to reproduce this.

-- 
(domestic pets only, the antidote for overdose, milk.)
  lmi@gnus.org * Lars Magne Ingebrigtsen

Re: file descriptor usage?

[Scott Blachowicz]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:

> Scott Blachowicz <scott@bloke.statsci.com> writes:
> 
> > I think I might have the nocem stuff enabled...would it be doing a
> > bazillion separate connections for THAT?
> 
> No, it's an, uhm, feature of how virtual servers are handled.  For
> some reason, Gnus believes that you have several (similar) virtual
> servers, so you get these results.  I have yet to come up with a fix. 

Let's see...I've got a few...

Scott Blachowicz  Ph: 206/283-8802x240   Mathsoft (Data Analysis Products Div)
                                         1700 Westlake Ave N #500
scott@statsci.com                        Seattle, WA USA   98109
Scott.Blachowicz@seaslug.org

Re: file descriptor usage?

[Scott Blachowicz]

Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> No, it's an, uhm, feature of how virtual servers are handled.  For
> some reason, Gnus believes that you have several (similar) virtual
> servers, so you get these results.  I have yet to come up with a fix. 

Let's see...I've got a few...I just counted 11 nnvirtual groups that
are various cross sections of the gnu.*, comp.* and
nntp+msnews.microsoft.com:microsoft.* hierarchies.  How do you define
"similar" above?  I suppose I should go clean out the nnvirtual groups
that I don't really read that often any more, huh?

Scott Blachowicz  Ph: 206/283-8802x240   Mathsoft (Data Analysis Products Div)
                                         1700 Westlake Ave N #500
scott@statsci.com                        Seattle, WA USA   98109
Scott.Blachowicz@seaslug.org

Re: file descriptor usage?

[David Moore]

Scott Blachowicz <scott@apple.statsci.com> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> wrote:
> > No, it's an, uhm, feature of how virtual servers are handled.  For
> > some reason, Gnus believes that you have several (similar) virtual
> > servers, so you get these results.  I have yet to come up with a fix. 
> 
> Let's see...I've got a few...I just counted 11 nnvirtual groups that
> are various cross sections of the gnu.*, comp.* and
> nntp+msnews.microsoft.com:microsoft.* hierarchies.  How do you define
> "similar" above?  I suppose I should go clean out the nnvirtual groups
> that I don't really read that often any more, huh?

nnvirtual != virtual server.  There is a confusing name conflict
here. :) 

The 'virtual servers' are the "nntp+msnews.microsoft.com" entries, which
is technically unrelated to nnvirtual.  Other than that you seem to have
placed a lot of 'virtual servers' inside of your nnvirtual groups.

The problem is that gnus sees two things of the form "nntp+msnews..."
and opens a separate connection for each.

-- 
David Moore <dmoore@ucsd.edu>       | Computer Systems Lab      __o
UCSD Dept. Computer Science - 0114  | Work: (619) 534-8604    _ \<,_
La Jolla, CA 92093-0114             | Fax:  (619) 534-1445   (_)/ (_)
<URL:http://oj.egbt.org/dmoore/>    | In a cloud bones of steel.

Re: file descriptor usage?

[Scott Blachowicz]

David Moore <dmoore@ucsd.edu> wrote:

> nnvirtual != virtual server.  There is a confusing name conflict
> here. :) 

Granted. :-)

> The problem is that gnus sees two things of the form "nntp+msnews..."
> and opens a separate connection for each.

When does it do that?  Would it be doing that at startup? Just for groups
whose level is right (i.e. it needs to generate an unread message count)?
Would a workaround be to make sure that all of those groups on the 'virtual
server' have a high enough level to not need a message count on startup?
Hmmm...I thought it was called 'foreign server'?  Maybe it's time for some
RTFM work on my part :-)...or is a better solution to update to 5.4.39 or
later (by the time I actually get around to it)?

Thanx,
Scott Blachowicz  Ph: 206/283-8802x240   Mathsoft (Data Analysis Products Div)
                                         1700 Westlake Ave N #500
scott@statsci.com                        Seattle, WA USA   98109
Scott.Blachowicz@seaslug.org