From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/65042 Path: news.gmane.org!not-for-mail From: Katsumi Yamaoka Newsgroups: gmane.emacs.gnus.general Subject: Re: Huge memory consumption on accessing large newsgroup Date: Mon, 13 Aug 2007 20:44:37 +0900 Organization: Emacsen advocacy group Message-ID: References: <87wsw4u21m.fsf@gmx.de> <87ps1vh85n.fsf@gmx.de> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: sea.gmane.org 1187005599 19637 80.91.229.12 (13 Aug 2007 11:46:39 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Mon, 13 Aug 2007 11:46:39 +0000 (UTC) Cc: bugs@gnus.org, ding@gnus.org To: Sven Joachim Original-X-From: ding-owner+M13552@lists.math.uh.edu Mon Aug 13 13:46:36 2007 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1IKYNW-0007ir-D2 for ding-account@gmane.org; Mon, 13 Aug 2007 13:46:34 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1IKYMP-0002xq-IZ; Mon, 13 Aug 2007 06:45:25 -0500 Original-Received: from mx2.math.uh.edu ([129.7.128.33]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1IKYMO-0002xc-1M for ding@lists.math.uh.edu; Mon, 13 Aug 2007 06:45:24 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx2.math.uh.edu with esmtp (Exim 4.67) (envelope-from ) id 1IKYMK-0002z0-03 for ding@lists.math.uh.edu; Mon, 13 Aug 2007 06:45:23 -0500 Original-Received: from orlando.hostforweb.net ([216.246.45.90]) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1IKYMI-0000is-00; Mon, 13 Aug 2007 13:45:18 +0200 Original-Received: from [66.225.201.151] (port=48917 helo=mail.jpl.org) by orlando.hostforweb.net with esmtpa (Exim 4.63) (envelope-from ) id 1IKYLk-00066h-7A; Mon, 13 Aug 2007 06:44:45 -0500 X-Hashcash: 1:20:070813:svenjoac@gmx.de::1b3FbFzyLyheI/Um:0005na X-Hashcash: 1:20:070813:bugs@gnus.org::GZMQkU0MEIyW70fe:00001W4F X-Hashcash: 1:20:070813:ding@gnus.org::OVwpa/SsreWmwgte:00000Uft X-Face: #kKnN,xUnmKia.'[pp`;Omh}odZK)?7wQSl"4o04=EixTF+V[""w~iNbM9ZL+.b*_CxUmFk B#Fu[*?MZZH@IkN:!"\w%I_zt>[$nm7nQosZ<3eu;B:$Q_:p!',P.c0-_Cy[dz4oIpw0ESA^D*1Lw= L&i*6&( User-Agent: Gnus/5.1008 (Gnus v5.10.8) Emacs/22.1.50 (gnu/linux) Cancel-Lock: sha1:lVriRjRFG+1fq8jkVDyoH+I3KpU= X-Antivirus-Scanner: Clean mail though you should still use an Antivirus X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - orlando.hostforweb.net X-AntiAbuse: Original Domain - gnus.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - jpl.org X-Source: X-Source-Args: X-Source-Dir: X-Spam-Score: -2.4 (--) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:65042 Archived-At: --=-=-= >>>>> Sven Joachim wrote: >> (defadvice gnus-uncompress-range (before narrow-range (ranges) activate) >> "Narrow the range if it is unreasonably wide." >> (let ((ttl 10000) [...] > Does not seem to help much, it seems. :-) With this advice, Emacs' > memory footprint grew to 303 MB on accessing be.politics at > news.motzarella.org, another random example of a group with supposedly > 30m+ articles. And fetching 50 headers increased it to 574 MB. I found the other things to be fixed. Could you try the patch (for Gnus v5.11) attached below? It still might not be complete, so I set the default value of the new variable gnus-maximum-newsgroup to nil so that it might not change the present behavior of Gnus. Now I have (setq gnus-maximum-newsgroup 100000) and Gnus runs smoothly with be.politics and comp.os.linux.misc on news.motzarella.org (though the value 100000 is too much for those groups, see a note in the docstring of the variable). --=-=-= Content-Type: text/x-patch Content-Disposition: inline --- gnus.el~ 2007-07-26 07:41:56 +0000 +++ gnus.el 2007-08-13 11:43:02 +0000 @@ -1501,6 +1501,17 @@ :type '(choice (const :tag "No limit" nil) integer)) +(defcustom gnus-maximum-newsgroup nil + "The maximum number of articles a newsgroup. +If this is a number, old articles in a newsgroup exceeding this number +are silently ignored. If it is nil, no article is ignored. Note that +setting this variable to a number might prevent you from reading very +old articles." + :group 'gnus-group-select + :version "22.2" + :type '(choice (const :tag "No limit" nil) + integer)) + (defcustom gnus-use-long-file-name (not (memq system-type '(usg-unix-v xenix))) "*Non-nil means that the default name of a file to save articles in is the group name. If it's nil, the directory form of the group name is used instead. --- gnus-agent.el~ 2007-07-26 07:41:55 +0000 +++ gnus-agent.el 2007-08-13 11:43:02 +0000 @@ -1765,7 +1765,14 @@ (gnus-agent-find-parameter group 'agent-predicate))))) (articles (if fetch-all - (gnus-uncompress-range (gnus-active group)) + (if gnus-maximum-newsgroup + (let ((active (gnus-active group))) + (gnus-uncompress-range + (cons (max (car active) + (- (cdr active) + gnus-maximum-newsgroup)) + (cdr active)))) + (gnus-uncompress-range (gnus-active group))) (gnus-list-of-unread-articles group))) (gnus-decode-encoded-word-function 'identity) (gnus-decode-encoded-address-function 'identity) --- gnus-sum.el~ 2007-07-26 07:41:56 +0000 +++ gnus-sum.el 2007-08-13 11:43:02 +0000 @@ -5472,7 +5472,13 @@ ;; articles in the group, or (if that's nil), the ;; articles in the cache. (or - (gnus-uncompress-range (gnus-active group)) + (if gnus-maximum-newsgroup + (let ((active (gnus-active group))) + (gnus-uncompress-range + (cons (max (car active) + (- (cdr active) gnus-maximum-newsgroup)) + (cdr active)))) + (gnus-uncompress-range (gnus-active group))) (gnus-cache-articles-in-group group)) ;; Select only the "normal" subset of articles. (gnus-sorted-nunion @@ -6534,23 +6540,26 @@ (let* ((read (gnus-info-read (gnus-get-info group))) (active (or (gnus-active group) (gnus-activate-group group))) (last (cdr active)) + (bottom (if gnus-maximum-newsgroup + (max (car active) (- last gnus-maximum-newsgroup)) + (car active))) first nlast unread) ;; If none are read, then all are unread. (if (not read) - (setq first (car active)) + (setq first bottom) ;; If the range of read articles is a single range, then the ;; first unread article is the article after the last read ;; article. Sounds logical, doesn't it? (if (and (not (listp (cdr read))) - (or (< (car read) (car active)) + (or (< (car read) bottom) (progn (setq read (list read)) nil))) - (setq first (max (car active) (1+ (cdr read)))) + (setq first (max bottom (1+ (cdr read)))) ;; `read' is a list of ranges. (when (/= (setq nlast (or (and (numberp (car read)) (car read)) (caar read))) 1) - (setq first (car active))) + (setq first bottom)) (while read (when first (while (< first nlast) @@ -6575,7 +6584,12 @@ (gnus-list-range-difference (gnus-list-range-difference (gnus-sorted-complement - (gnus-uncompress-range active) + (gnus-uncompress-range + (if gnus-maximum-newsgroup + (cons (max (car active) + (- (cdr active) gnus-maximum-newsgroup)) + (cdr active)) + active)) (gnus-list-of-unread-articles group)) (cdr (assq 'dormant marked))) (cdr (assq 'tick marked)))))) @@ -6587,23 +6601,26 @@ (let* ((read (gnus-info-read (gnus-get-info group))) (active (or (gnus-active group) (gnus-activate-group group))) (last (cdr active)) + (bottom (if gnus-maximum-newsgroup + (max (car active) (- last gnus-maximum-newsgroup)) + (car active))) first nlast unread) ;; If none are read, then all are unread. (if (not read) - (setq first (car active)) + (setq first bottom) ;; If the range of read articles is a single range, then the ;; first unread article is the article after the last read ;; article. Sounds logical, doesn't it? (if (and (not (listp (cdr read))) - (or (< (car read) (car active)) + (or (< (car read) bottom) (progn (setq read (list read)) nil))) - (setq first (max (car active) (1+ (cdr read)))) + (setq first (max bottom (1+ (cdr read)))) ;; `read' is a list of ranges. (when (/= (setq nlast (or (and (numberp (car read)) (car read)) (caar read))) 1) - (setq first (car active))) + (setq first bottom)) (while read (when first (push (cons first nlast) unread)) --=-=-=--