From: Katsumi Yamaoka <yamaoka@jpl.org>
To: Sven Joachim <svenjoac@gmx.de>
Cc: bugs@gnus.org, ding@gnus.org
Subject: Re: Huge memory consumption on accessing large newsgroup
Date: Mon, 13 Aug 2007 20:44:37 +0900 [thread overview]
Message-ID: <b4md4xrfyl6.fsf@jpl.org> (raw)
In-Reply-To: <87ps1vh85n.fsf@gmx.de>
[-- Attachment #1: Type: text/plain, Size: 962 bytes --]
>>>>> Sven Joachim wrote:
>> (defadvice gnus-uncompress-range (before narrow-range (ranges) activate)
>> "Narrow the range if it is unreasonably wide."
>> (let ((ttl 10000)
[...]
> Does not seem to help much, it seems. :-) With this advice, Emacs'
> memory footprint grew to 303 MB on accessing be.politics at
> news.motzarella.org, another random example of a group with supposedly
> 30m+ articles. And fetching 50 headers increased it to 574 MB.
I found the other things to be fixed. Could you try the patch
(for Gnus v5.11) attached below? It still might not be complete,
so I set the default value of the new variable
gnus-maximum-newsgroup
to nil so that it might not change the present behavior of Gnus.
Now I have
(setq gnus-maximum-newsgroup 100000)
and Gnus runs smoothly with be.politics and comp.os.linux.misc on
news.motzarella.org (though the value 100000 is too much for those
groups, see a note in the docstring of the variable).
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 4988 bytes --]
--- gnus.el~ 2007-07-26 07:41:56 +0000
+++ gnus.el 2007-08-13 11:43:02 +0000
@@ -1501,6 +1501,17 @@
:type '(choice (const :tag "No limit" nil)
integer))
+(defcustom gnus-maximum-newsgroup nil
+ "The maximum number of articles a newsgroup.
+If this is a number, old articles in a newsgroup exceeding this number
+are silently ignored. If it is nil, no article is ignored. Note that
+setting this variable to a number might prevent you from reading very
+old articles."
+ :group 'gnus-group-select
+ :version "22.2"
+ :type '(choice (const :tag "No limit" nil)
+ integer))
+
(defcustom gnus-use-long-file-name (not (memq system-type '(usg-unix-v xenix)))
"*Non-nil means that the default name of a file to save articles in is the group name.
If it's nil, the directory form of the group name is used instead.
--- gnus-agent.el~ 2007-07-26 07:41:55 +0000
+++ gnus-agent.el 2007-08-13 11:43:02 +0000
@@ -1765,7 +1765,14 @@
(gnus-agent-find-parameter group
'agent-predicate)))))
(articles (if fetch-all
- (gnus-uncompress-range (gnus-active group))
+ (if gnus-maximum-newsgroup
+ (let ((active (gnus-active group)))
+ (gnus-uncompress-range
+ (cons (max (car active)
+ (- (cdr active)
+ gnus-maximum-newsgroup))
+ (cdr active))))
+ (gnus-uncompress-range (gnus-active group)))
(gnus-list-of-unread-articles group)))
(gnus-decode-encoded-word-function 'identity)
(gnus-decode-encoded-address-function 'identity)
--- gnus-sum.el~ 2007-07-26 07:41:56 +0000
+++ gnus-sum.el 2007-08-13 11:43:02 +0000
@@ -5472,7 +5472,13 @@
;; articles in the group, or (if that's nil), the
;; articles in the cache.
(or
- (gnus-uncompress-range (gnus-active group))
+ (if gnus-maximum-newsgroup
+ (let ((active (gnus-active group)))
+ (gnus-uncompress-range
+ (cons (max (car active)
+ (- (cdr active) gnus-maximum-newsgroup))
+ (cdr active))))
+ (gnus-uncompress-range (gnus-active group)))
(gnus-cache-articles-in-group group))
;; Select only the "normal" subset of articles.
(gnus-sorted-nunion
@@ -6534,23 +6540,26 @@
(let* ((read (gnus-info-read (gnus-get-info group)))
(active (or (gnus-active group) (gnus-activate-group group)))
(last (cdr active))
+ (bottom (if gnus-maximum-newsgroup
+ (max (car active) (- last gnus-maximum-newsgroup))
+ (car active)))
first nlast unread)
;; If none are read, then all are unread.
(if (not read)
- (setq first (car active))
+ (setq first bottom)
;; If the range of read articles is a single range, then the
;; first unread article is the article after the last read
;; article. Sounds logical, doesn't it?
(if (and (not (listp (cdr read)))
- (or (< (car read) (car active))
+ (or (< (car read) bottom)
(progn (setq read (list read))
nil)))
- (setq first (max (car active) (1+ (cdr read))))
+ (setq first (max bottom (1+ (cdr read))))
;; `read' is a list of ranges.
(when (/= (setq nlast (or (and (numberp (car read)) (car read))
(caar read)))
1)
- (setq first (car active)))
+ (setq first bottom))
(while read
(when first
(while (< first nlast)
@@ -6575,7 +6584,12 @@
(gnus-list-range-difference
(gnus-list-range-difference
(gnus-sorted-complement
- (gnus-uncompress-range active)
+ (gnus-uncompress-range
+ (if gnus-maximum-newsgroup
+ (cons (max (car active)
+ (- (cdr active) gnus-maximum-newsgroup))
+ (cdr active))
+ active))
(gnus-list-of-unread-articles group))
(cdr (assq 'dormant marked)))
(cdr (assq 'tick marked))))))
@@ -6587,23 +6601,26 @@
(let* ((read (gnus-info-read (gnus-get-info group)))
(active (or (gnus-active group) (gnus-activate-group group)))
(last (cdr active))
+ (bottom (if gnus-maximum-newsgroup
+ (max (car active) (- last gnus-maximum-newsgroup))
+ (car active)))
first nlast unread)
;; If none are read, then all are unread.
(if (not read)
- (setq first (car active))
+ (setq first bottom)
;; If the range of read articles is a single range, then the
;; first unread article is the article after the last read
;; article. Sounds logical, doesn't it?
(if (and (not (listp (cdr read)))
- (or (< (car read) (car active))
+ (or (< (car read) bottom)
(progn (setq read (list read))
nil)))
- (setq first (max (car active) (1+ (cdr read))))
+ (setq first (max bottom (1+ (cdr read))))
;; `read' is a list of ranges.
(when (/= (setq nlast (or (and (numberp (car read)) (car read))
(caar read)))
1)
- (setq first (car active)))
+ (setq first bottom))
(while read
(when first
(push (cons first nlast) unread))
next prev parent reply other threads:[~2007-08-13 11:44 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <87wsw4u21m.fsf@gmx.de>
2007-08-10 9:08 ` Katsumi Yamaoka
2007-08-10 11:39 ` Katsumi Yamaoka
2007-08-10 12:43 ` Sven Joachim
2007-08-13 11:44 ` Katsumi Yamaoka [this message]
2007-08-13 17:30 ` Sven Joachim
2007-08-14 11:46 ` Katsumi Yamaoka
2007-09-13 10:27 ` Katsumi Yamaoka
2007-08-10 12:42 ` Sven Joachim
2007-09-29 21:04 ` Gaute Strokkenes
2007-09-30 22:11 ` Ted Zlatanov
2007-10-01 0:29 ` Katsumi Yamaoka
2007-10-01 1:04 ` Daniel Pittman
2007-10-02 2:13 ` Ted Zlatanov
2007-10-02 3:23 ` Daniel Pittman
2007-10-02 11:11 ` Ted Zlatanov
2007-10-02 12:17 ` Daniel Pittman
2007-10-02 16:08 ` Ted Zlatanov
2007-10-03 0:19 ` Daniel Pittman
2007-10-02 13:33 ` Daniel Pittman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b4md4xrfyl6.fsf@jpl.org \
--to=yamaoka@jpl.org \
--cc=bugs@gnus.org \
--cc=ding@gnus.org \
--cc=svenjoac@gmx.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).