From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/65328 Path: news.gmane.org!not-for-mail From: Ted Zlatanov Newsgroups: gmane.emacs.gnus.general Subject: Re: Huge memory consumption on accessing large newsgroup Date: Tue, 02 Oct 2007 06:11:40 -0500 Organization: =?utf-8?B?0KLQtdC+0LTQvtGAINCX0LvQsNGC0LDQvdC+0LI=?= @ Cienfuegos Message-ID: References: <87wsw4u21m.fsf@gmx.de> <87sl6rh886.fsf@gmx.de> <87hcldp4j1.fsf@srcf.ucam.org> <87odfjprux.fsf@enki.rimspace.net> <874phatd0z.fsf@enki.rimspace.net> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: sea.gmane.org 1191323604 5384 80.91.229.12 (2 Oct 2007 11:13:24 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Tue, 2 Oct 2007 11:13:24 +0000 (UTC) Cc: ding@gnus.org To: Daniel Pittman Original-X-From: ding-owner+M13840@lists.math.uh.edu Tue Oct 02 13:13:21 2007 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1Icfgk-000646-PA for ding-account@gmane.org; Tue, 02 Oct 2007 13:13:19 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1Icffr-0002ky-Hw; Tue, 02 Oct 2007 06:12:23 -0500 Original-Received: from mx2.math.uh.edu ([129.7.128.33]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1Icffq-0002kl-8R for ding@lists.math.uh.edu; Tue, 02 Oct 2007 06:12:22 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx2.math.uh.edu with esmtp (Exim 4.67) (envelope-from ) id 1Icffk-00045f-1r for ding@lists.math.uh.edu; Tue, 02 Oct 2007 06:12:22 -0500 Original-Received: from blockstar.com ([170.224.69.95] helo=mail.blockstar.com) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1Icffb-0003pq-00 for ; Tue, 02 Oct 2007 13:12:07 +0200 Original-Received: from tzz (c-24-14-57-89.hsd1.il.comcast.net [24.14.57.89]) by mail.blockstar.com (Postfix) with ESMTP id 1F2113F8117; Tue, 2 Oct 2007 04:27:56 -0700 (PDT) X-Face: bd.DQ~'29fIs`T_%O%C\g%6jW)yi[zuz6;d4V0`@y-~$#3P_Ng{@m+e4o<4P'#(_GJQ%TT= D}[Ep*b!\e,fBZ'j_+#"Ps?s2!4H2-Y"sx" Mail-Followup-To: Daniel Pittman , ding@gnus.org In-Reply-To: <874phatd0z.fsf@enki.rimspace.net> (Daniel Pittman's message of "Tue, 02 Oct 2007 13:23:56 +1000") User-Agent: Gnus/5.110007 (No Gnus v0.7) Emacs/22.1.50 (darwin) X-Spam-Score: -2.4 (--) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:65328 Archived-At: On Tue, 02 Oct 2007 13:23:56 +1000 Daniel Pittman wrote: DP> The issue with large memory consumption, as far as I could see from the DP> thread, was that the compressed range data type was expanded to a flat DP> list. This caused, no surprise, huge memory use. DP> The original problem is that the code calls `gnus-uncompress-range' on DP> the data *at all* -- and, so, turns a nicely brief data structure into a DP> vast bloated million-number list. DP> The *solution* is to rebuild the algorithm to operate on the compressed DP> version (regardless of the internal representation), not to change the DP> representation. You're right, I didn't know this. I thought the memory problems were caused by the original list. In light of your explanation, I completely agree that the right way is to find all instances of `gnus-uncompress-range' and fix their consequences. Thank you for explaining. DP> (And because this has been a stupidly annoying couple of week in other DP> areas, and because this is nice simple and essentially stress-free work DP> I am getting tempted to fix it myself. DP> So, maybe inversion lists were the way to get the code fixed after all, DP> if not quite so directly as expected. ;) The `gnus-uncompress-range' function is not called in too many places: gnus-agent.el:7 gnus-draft.el:1 gnus-group.el:2 gnus-move.el:4 gnus-nocem.el:1 gnus-range.el:2 gnus-start.el:2 gnus-sum.el:9 nnimap.el:1 nnmaildir.el:3 nnsoup.el:1 nnvirtual.el:1 In gnus-number-of-unseen-articles-in-group for example it's called only to find the length of the list, and in gnus-move-group-to-server only to see if the list is not nil. There's room for improvement. Maybe there should be a group API that abstracts the data structures, so subordinate code doesn't have to know if it's a list or a compressed range? That would be a good first step to cleaning things up. Ted