From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/11969 Path: main.gmane.org!not-for-mail From: Ken Raeburn Newsgroups: gmane.emacs.gnus.general Subject: Re: opening speed for nnml groups Date: 03 Sep 1997 16:33:43 -0400 Sender: raeburn@cygnus.com Message-ID: References: NNTP-Posting-Host: coloc-standby.netfonds.no X-Trace: main.gmane.org 1035151590 1144 80.91.224.250 (20 Oct 2002 22:06:30 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Sun, 20 Oct 2002 22:06:30 +0000 (UTC) Return-Path: Original-Received: from xemacs.org (xemacs.cs.uiuc.edu [128.174.252.16]) by altair.xemacs.org (8.8.7/8.8.7) with ESMTP id QAA16864 for ; Wed, 3 Sep 1997 16:43:59 -0700 Original-Received: from ifi.uio.no (0@ifi.uio.no [129.240.64.2]) by xemacs.org (8.8.5/8.8.5) with SMTP id SAA29377 for ; Wed, 3 Sep 1997 18:39:54 -0500 (CDT) Original-Received: from claymore.vcinet.com (claymore.vcinet.com [208.205.12.23]) by ifi.uio.no with SMTP (8.6.11/ifi2.4) id for ; Thu, 4 Sep 1997 00:20:36 +0200 Original-Received: (qmail 5437 invoked by uid 504); 3 Sep 1997 20:33:49 -0000 Original-Received: (qmail 5434 invoked from network); 3 Sep 1997 20:33:46 -0000 Original-Received: from tweedledumb.cygnus.com (192.80.44.1) by claymore.vcinet.com with SMTP; 3 Sep 1997 20:33:46 -0000 Original-Received: from cujo.cygnus.com (cujo.cygnus.com [192.80.44.10]) by tweedledumb.cygnus.com (8.8.5/8.8.5) with SMTP id QAA21175; Wed, 3 Sep 1997 16:33:45 -0400 (EDT) Original-Received: by cujo.cygnus.com; (5.65v3.2/1.1.8.2/20Sep95-0235PM) id AA24362; Wed, 3 Sep 1997 16:33:45 -0400 Original-To: ding@gnus.org In-Reply-To: Steinar Bang's message of 02 Sep 1997 14:01:31 +0200 Original-Lines: 82 X-Mailer: Gnus v5.4.37/Emacs 19.34 Xref: main.gmane.org gmane.emacs.gnus.general:11969 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:11969 Steinar Bang writes: > I haven't done any timing of it yet, but it seems like opening nnml > groups get noticably slower, as the groups get bigger (with more > ticked or read articles in them). > > I was sort of idly wondering where the speed hit was? Is it in the > parsing of the .overview file? (I forget who pointed this out to me...) If you use the message date in the summary line, that's probably a big piece of it. See my messages of some time ago about regexp ordering in timezone-parse-date. Getting the order right (since all email I tried matched one pattern, and it wasn't the first one) saves a large fraction of the time spent in that routine. In a test I just did with ELP and a mail group of some 7800 messages, timezone-parse-date (probably without fixed regexp ordering, I've been lame) took 46.9 seconds out of the total of 194.3. gnus-summary-read-group-1 took 194.3 seconds gnus-select-newsgroup took 32 gnus-get-newsgroup-headers-xover took 29.6 (nov-parse-line done inline) gnus-summary-prepare took 152 gnus-make-threads took 4 gnus-sort-threads took 12.8 gnus-summary-prepare-threads took 135.6 formatting and inserting summary line took ...?? Since my line-format-spec uses the date, let's be generous and say that gnus-summary-prepare-threads would've taken only 88 seconds without it, and the total time would've been 147 seconds. Then gnus-summary-prepare-threads still looks like the dominant factor at 60%. But parsing the overview data is still 20%, and could perhaps be optimized. Hm. Interesting. I ran gnus-compile to compile the specs, and ran my test again. Now timezone-parse-date supposedly only took 25.7 seconds, but the total time was still 190, and gnus-summary-prepare-threads still took 132.5. Something seems fishy here... Oh, and gnus-simplify-subject-fuzzy used about 26 seconds in almost 9800 calls, but I haven't traced down where in the call graph that would be located. I wonder if the hacks I did for gnus-summary-kill-same-subject (more generally, for matching of simplified subjects) could help here. (I sent them in 24 May 1996, and Lars said that same day that he'd include them in Red Gnus. Still hasn't happened... :-) gnus-summary-highlight-line took 14 seconds, in 7873 calls gnus-sort-threads took 12 seconds; gnus-thread-sort-by-number took 4.5 seconds in 20703 calls gnus-put-text-property took 9 seconds in 15812 calls That looks like it for functions taking over 5% of the time. > I remember that slow exit on big or sparse nnml group, was a problem > when you did expiry on exiting a group, so I went for demonical > expiry, as well as demonical fetch and sort. Me too (for expiry), but it's still a pain to have to wait if I come back to Emacs while it's running. I can't even interrupt it with ^G. > If the reading and parsing of the .overview file is the problem, would > it be an idea to cache the parsed info between visits? And update it > during fetch and sort operations? Or maybe store the data as s-expressions, possibly with extra fields for caching references links and parsed dates, rather than the standard news overview format? > Or is that too expensive memory-wise? The overview file for this group I was working with is 1.8M. I doubt the lisp version would be much smaller. I wouldn't want to cache that much info in memory just on the chance that I might come back to that newsgroup eventually. Maybe if it got dumped the moment I entered some other newsgroup; that would permit leaving, fetching new mail, and re-entering.