From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/65034 Path: news.gmane.org!not-for-mail From: Katsumi Yamaoka Newsgroups: gmane.emacs.gnus.general Subject: Re: Huge memory consumption on accessing large newsgroup Date: Fri, 10 Aug 2007 18:08:10 +0900 Organization: Emacsen advocacy group Message-ID: References: <87wsw4u21m.fsf@gmx.de> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: sea.gmane.org 1186737008 5555 80.91.229.12 (10 Aug 2007 09:10:08 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Fri, 10 Aug 2007 09:10:08 +0000 (UTC) Cc: bugs@gnus.org, ding@gnus.org To: Sven Joachim Original-X-From: ding-owner+M13544@lists.math.uh.edu Fri Aug 10 11:10:01 2007 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1IJQVM-0004JO-Ph for ding-account@gmane.org; Fri, 10 Aug 2007 11:10:01 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1IJQUT-0001DK-TI; Fri, 10 Aug 2007 04:09:05 -0500 Original-Received: from mx2.math.uh.edu ([129.7.128.33]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1IJQUS-0001Cx-5z for ding@lists.math.uh.edu; Fri, 10 Aug 2007 04:09:04 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx2.math.uh.edu with esmtp (Exim 4.67) (envelope-from ) id 1IJQUO-0003eM-LB for ding@lists.math.uh.edu; Fri, 10 Aug 2007 04:09:04 -0500 Original-Received: from orlando.hostforweb.net ([216.246.45.90]) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1IJQUN-0006Q7-00; Fri, 10 Aug 2007 11:08:59 +0200 Original-Received: from [66.225.201.151] (port=44969 helo=mail.jpl.org) by orlando.hostforweb.net with esmtpa (Exim 4.63) (envelope-from ) id 1IJQTm-0003Ri-UU; Fri, 10 Aug 2007 04:08:24 -0500 X-Hashcash: 1:20:070810:svenjoac@gmx.de::58czstjSwxhHsQ2e:0009xz X-Hashcash: 1:20:070810:bugs@gnus.org::k6lw1sQeSCHN5rt+:0000BLR0 X-Hashcash: 1:20:070810:ding@gnus.org::Stc3Lj/2ZNYX3YAd:00007T7B X-Face: #kKnN,xUnmKia.'[pp`;Omh}odZK)?7wQSl"4o04=EixTF+V[""w~iNbM9ZL+.b*_CxUmFk B#Fu[*?MZZH@IkN:!"\w%I_zt>[$nm7nQosZ<3eu;B:$Q_:p!',P.c0-_Cy[dz4oIpw0ESA^D*1Lw= L&i*6&( User-Agent: Gnus/5.110007 (No Gnus v0.7) Emacs/22.1.50 (gnu/linux) Cancel-Lock: sha1:Rvf5bGb9IwMjjnohCNaQKJMmYAM= X-Antivirus-Scanner: Clean mail though you should still use an Antivirus X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - orlando.hostforweb.net X-AntiAbuse: Original Domain - gnus.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - jpl.org X-Source: X-Source-Args: X-Source-Dir: X-Spam-Score: -2.4 (--) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:65034 Archived-At: --=-=-= (I added the ding list to Cc.) >>>>> Sven Joachim wrote: > Gnus v5.11 > GNU Emacs 22.1.50.1 (i486-pc-linux-gnu, GTK+ Version 2.10.13) > of 2007-08-06 on debian, modified by Debian > 200 news.motzarella.org InterNetNews NNRP server INN 2.4.4 (20060818 snapshot) ready (posting ok). I visited http://news.motzarella.org/ and got an account on Motzarella out of curiosity. ;-) > When accessing comp.os.linux.misc on news.motzarella.org, a _very_ > large newsgroup with more than 30,000,000 articles, Emacs' memory > footprint grew heavily. Yes, now the ACTIVE of that group is: --=-=-= Content-Type: application/emacs-lisp Content-Disposition: inline (gnus-active "nntp+motzarella:comp.os.linux.misc") => (3437 . 30538699) --=-=-= Content-Disposition: inline However, I verified there are actually no more than less than 3,000 articles. It should not be a cause of this problem. > It took ~2 minutes to display its question > "How many articles...?" and the memory usage was at 555 MB (RSS). > When I answered "500" and the summary buffer finally appeared, it grew > up to 893 MB. Which is a bit scary, since my computer has "only" 1 GB > of RAM and is now already paging quite a bit. > Will this become better if I subscribe to the newsgroup and catch up? > In any case I probably will have to kill my current Emacs session soon. The real cause is that Gnus first expands this ACTIVE data into: (3437 3438 3439 3440 ...... 30538696 30538697 30538698 30538699) If you run Emacs on a super computer, try this: --=-=-= Content-Type: application/emacs-lisp Content-Disposition: inline (gnus-uncompress-range '(3437 . 30538699)) --=-=-= Content-Disposition: inline I think the possible solution is to narrow the range into the one with which Emacs can work lightly. Here it is: --=-=-= Content-Type: application/emacs-lisp Content-Disposition: inline (defadvice gnus-uncompress-range (before narrow-range (ranges) activate) "Narrow the range if it is unreasonably wide." (setcar ranges (max (car ranges) (- (cdr ranges) 10000)))) --=-=-= Content-Disposition: inline The 10000 will probably need to be a customizable variable. I'm going to do it next week. BTW, I needed to set the server variable `nntp-authinfo-force' to t in order to let No Gnus v0.7 send the AUTHINFO data to the Motzarella server as follows: (nntp "motzarella" (nntp-address "news.motzarella.org") (nntp-authinfo-user "yamaoka") (nntp-authinfo-password "********") (nntp-authinfo-force t) ...) I guess you have the FORCE element in the ~/.authinfo file since such a server variable has not been implemented yet in Gnus v5.11. Regards, --=-=-=--