From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/55367 Path: main.gmane.org!not-for-mail From: Kevin Greiner Newsgroups: gmane.emacs.gnus.general Subject: Re: Profiling Date: Mon, 29 Dec 2003 23:06:59 -0600 Sender: ding-owner@lists.math.uh.edu Message-ID: References: NNTP-Posting-Host: deer.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: sea.gmane.org 1072760871 30446 80.91.224.253 (30 Dec 2003 05:07:51 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Tue, 30 Dec 2003 05:07:51 +0000 (UTC) Original-X-From: ding-owner+M3907@lists.math.uh.edu Tue Dec 30 06:07:41 2003 Return-path: Original-Received: from malifon.math.uh.edu ([129.7.128.13]) by deer.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 1AbC6W-0001Ln-00 for ; Tue, 30 Dec 2003 06:07:41 +0100 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by malifon.math.uh.edu with smtp (Exim 3.20 #1) id 1AbC69-00050b-00; Mon, 29 Dec 2003 23:07:17 -0600 Original-Received: from justine.libertine.org ([66.139.78.221] ident=postfix) by malifon.math.uh.edu with esmtp (Exim 3.20 #1) id 1AbC60-0004zZ-00 for ding@lists.math.uh.edu; Mon, 29 Dec 2003 23:07:08 -0600 Original-Received: from quimby.gnus.org (quimby.gnus.org [80.91.224.244]) by justine.libertine.org (Postfix) with ESMTP id B10C23A004E for ; Mon, 29 Dec 2003 23:07:07 -0600 (CST) Original-Received: from news by quimby.gnus.org with local (Exim 3.35 #1 (Debian)) id 1AbC5z-00017E-00 for ; Tue, 30 Dec 2003 06:07:07 +0100 Original-To: ding@gnus.org Original-Path: not-for-mail Original-Newsgroups: gnus.ding Original-Lines: 61 Original-NNTP-Posting-Host: dialup-216-12-206-172.ev1.net Original-X-Trace: quimby.gnus.org 1072760827 4291 216.12.206.172 (30 Dec 2003 05:07:07 GMT) Original-X-Complaints-To: usenet@quimby.gnus.org Original-NNTP-Posting-Date: Tue, 30 Dec 2003 05:07:07 +0000 (UTC) User-Agent: Gnus/5.1003 (Gnus v5.10.3) Emacs/21.3 (windows-nt) Cancel-Lock: sha1:ipB1j72n20fs2cPySi38ChCO2TY= Precedence: bulk Xref: main.gmane.org gmane.emacs.gnus.general:55367 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:55367 Lars Magne Ingebrigtsen writes: > Gnus was feeling a bit more sluggish than usual, so I've spent a > couple of hours profiling and fixing obvious things. (I've cached > some of the server-to-method calls and stuff like that.) > > The main remaining suspicious function now is > `gnus-agent-possibly-alter-active', which is called for every > (covered) group. Is that really necessary? It takes about one > quarter of the total time spent when pressing `g'... Necessary yes. The server for the gnus.ding group recently changed its active range to start at 51065. When this happened, every agentized article below that cut-off (for me, about 30K articles) disappeared. The gnus-agent-possibly-alter-active has to be called where it is to correctly set the number of unread articles in the Group buffer. What can be improved is the implementation. Right now, it opens each group's alist file. In my original implementation, I wanted to use the agent's private copy of the server's active file. That way agent files are opened once per server rather than once per group. I abandoned that implementation as it required that I add code to maintain the agent's active file as articles are added/removed from the alist. While it may be more complex to design, its implementation should not be too difficult, and the performance should be much better. I'll look into it tomorrow. On a related topic (scalability rather than performance), my primary nntp server was recently replaced. As part of the change over, the implementor set the article numbers about 6M above each group's active range (I suspect so that they didn't have to worry about the number of articles being posted while they performed the change over). The gnus-agent-possibly-alter-active function overrode the new active range to produce a range spanning the old and new article ranges (about 14M). This killed gnus with memory exhausted in gnus-list-of-unread-articles. I could not even run catchup as it also calls gnus-list-of-unread-articles. I've written two fixes so far (but have not checked them in due to a lack of testing). 1) Added gnus-sequence-of-unread-articles to return a compressed sequence rather than an uncompressed list. I then modified gnus-group-catchup to use it. This means that you can catchup any group irregardless of size (it's also a good deal faster). 2) I modified gnus-agent-possibly-alter-active to assume that all articles between the end of the alist and the start of the new real active range are read. That's not technically true. However, these articles are no longer available on the server and they are not stored in the agent so they are lost (hence read). This change partly solved my memory exhausted problem as the number of unread articles is a more manageable 8M (Foolish server admins, I've never seen more than 100K articles due to retention limits). Any suggestions as to what else may be done? On a long-term perspective, should we consider making gnus-newsgroup-unreads a sequence rather than a list? Kevin