From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/65335 Path: news.gmane.org!not-for-mail From: Daniel Pittman Newsgroups: gmane.emacs.gnus.general Subject: Re: Huge memory consumption on accessing large newsgroup Date: Tue, 02 Oct 2007 23:33:52 +1000 Organization: Cybersource: Australia's Leading Linux and Open Source Solutions Company Message-ID: <87sl4tvdxb.fsf@enki.rimspace.net> References: <87wsw4u21m.fsf@gmx.de> <87sl6rh886.fsf@gmx.de> <87hcldp4j1.fsf@srcf.ucam.org> <87odfjprux.fsf@enki.rimspace.net> <874phatd0z.fsf@enki.rimspace.net> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: sea.gmane.org 1191370214 4068 80.91.229.12 (3 Oct 2007 00:10:14 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Wed, 3 Oct 2007 00:10:14 +0000 (UTC) To: ding@gnus.org Original-X-From: ding-owner+M13846@lists.math.uh.edu Wed Oct 03 02:10:10 2007 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from util0.math.uh.edu ([129.7.128.18]) by lo.gmane.org with esmtp (Exim 4.50) id 1IcroT-0004S5-4b for ding-account@gmane.org; Wed, 03 Oct 2007 02:10:05 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by util0.math.uh.edu with smtp (Exim 4.63) (envelope-from ) id 1Icrnb-00070g-I1; Tue, 02 Oct 2007 19:09:11 -0500 Original-Received: from mx1.math.uh.edu ([129.7.128.32]) by util0.math.uh.edu with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1IcrnZ-00070O-Cj for ding@lists.math.uh.edu; Tue, 02 Oct 2007 19:09:09 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx1.math.uh.edu with esmtp (Exim 4.67) (envelope-from ) id 1IcrnS-0007zl-Cj for ding@lists.math.uh.edu; Tue, 02 Oct 2007 19:09:09 -0500 Original-Received: from main.gmane.org ([80.91.229.2] helo=ciao.gmane.org) by quimby.gnus.org with esmtp (Exim 3.35 #1 (Debian)) id 1IcrnJ-0006hq-00 for ; Wed, 03 Oct 2007 02:08:53 +0200 Original-Received: from list by ciao.gmane.org with local (Exim 4.43) id 1IcrnK-000182-0e for ding@gnus.org; Wed, 03 Oct 2007 00:08:54 +0000 Original-Received: from nat20.cyber.com.au ([203.7.155.20]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 03 Oct 2007 00:08:54 +0000 Original-Received: from daniel by nat20.cyber.com.au with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 03 Oct 2007 00:08:54 +0000 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 195 Original-X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: nat20.cyber.com.au User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/23.0.0 (gnu/linux) Cancel-Lock: sha1:yy5pM2LN30kFXCXt7vwxju7Dmms= X-Spam-Score: -1.8 (-) List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:65335 Archived-At: --=-=-= Daniel Pittman writes: [...] > (And because this has been a stupidly annoying couple of week in other > areas, and because this is nice simple and essentially stress-free work > I am getting tempted to fix it myself. > > So, maybe inversion lists were the way to get the code fixed after all, > if not quite so directly as expected. ;) ...and that turns out to be true. Here is a patch against current CVS that addresses the low hanging fruit in the area. I will tackle the more complex examples some time soon, or someone else can, and we will see where we go next. This has only been very briefly looked at and isn't thoroughly tested at this stage. It should all be correct but could really do with a third party review for correctness before committing to CVS or anything. Daniel -- Daniel Pittman Phone: 03 9621 2377 Level 4, 10 Queen St, Melbourne Web: http://www.cyber.com.au Cybersource: Australia's Leading Linux and Open Source Solutions Company --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=keep-ranges-compressed-1.patch ? keep-ranges-compressed-1.patch Index: lisp/gnus-draft.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/gnus-draft.el,v retrieving revision 7.16 diff -u -u -r7.16 gnus-draft.el --- lisp/gnus-draft.el 24 Mar 2007 19:50:00 -0000 7.16 +++ lisp/gnus-draft.el 2 Oct 2007 13:32:14 -0000 @@ -212,15 +212,14 @@ (gnus-activate-group "nndraft:queue") (save-excursion (let* ((articles (nndraft-articles)) - (unsendable (gnus-uncompress-range - (cdr (assq 'unsend - (gnus-info-marks - (gnus-get-info "nndraft:queue")))))) + (unsendable (cdr (assq 'unsend + (gnus-info-marks + (gnus-get-info "nndraft:queue")))))) (gnus-posting-styles nil) (total (length articles)) article) (while (setq article (pop articles)) - (unless (memq article unsendable) + (unless (gnus-member-of-range article unsendable) (let ((message-sending-message (format "Sending message %d of %d..." (- total (length articles)) total))) Index: lisp/gnus-group.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/gnus-group.el,v retrieving revision 7.97 diff -u -u -r7.97 gnus-group.el --- lisp/gnus-group.el 3 Aug 2007 06:20:58 -0000 7.97 +++ lisp/gnus-group.el 2 Oct 2007 13:32:16 -0000 @@ -1496,10 +1496,10 @@ (active (gnus-active group))) (if (not active) 0 - (length (gnus-uncompress-range - (gnus-range-difference - (gnus-range-difference (list active) (gnus-info-read info)) - seen)))))) + (gnus-range-length + (gnus-range-difference + (gnus-range-difference (list active) (gnus-info-read info)) + seen))))) ;; Moving through the Group buffer (in topic mode) e.g. with C-n doesn't ;; update the state (enabled/disabled) of the icon `gnus-group-describe-group' @@ -4406,9 +4406,8 @@ (setcar (nthcdr 3 info) (gnus-delete-alist type (car marked))) (setcdr m (gnus-compress-sequence articles t))) - (setcdr m (gnus-compress-sequence - (sort (nconc (gnus-uncompress-range (cdr m)) - (copy-sequence articles)) '<) t)))))) + (setcdr m (gnus-range-add (cdr m) + (sort (copy-sequence articles) '<))))))) (defun gnus-add-mark (group mark article) "Mark ARTICLE in GROUP with MARK, whether the group is displayed or not." Index: lisp/gnus-move.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/gnus-move.el,v retrieving revision 7.9 diff -u -u -r7.9 gnus-move.el --- lisp/gnus-move.el 24 Jan 2007 07:15:37 -0000 7.9 +++ lisp/gnus-move.el 2 Oct 2007 13:32:18 -0000 @@ -91,8 +91,8 @@ ;; Then we read the headers from the `from-server'. (when (and (gnus-request-group group nil from-server) (gnus-active group) - (gnus-uncompress-range - (gnus-active group)) + ;; Should this simply test for an empty or nil range? + (gnus-range-length (gnus-active group)) (setq type (gnus-retrieve-headers (gnus-uncompress-range (gnus-active group)) Index: lisp/gnus-start.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/gnus-start.el,v retrieving revision 7.54 diff -u -u -r7.54 gnus-start.el --- lisp/gnus-start.el 7 Sep 2007 02:52:27 -0000 7.54 +++ lisp/gnus-start.el 2 Oct 2007 13:32:18 -0000 @@ -2376,8 +2376,7 @@ info (gnus-add-to-range (gnus-info-read info) - (nconc (gnus-uncompress-range dormant) - (gnus-uncompress-range ticked))))))))) + (gnus-range-add dormant ticked)))))))) (defun gnus-load (file) "Load FILE, but in such a way that read errors can be reported." Index: lisp/gnus-sum.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/gnus-sum.el,v retrieving revision 7.201 diff -u -u -r7.201 gnus-sum.el --- lisp/gnus-sum.el 27 Sep 2007 21:22:18 -0000 7.201 +++ lisp/gnus-sum.el 2 Oct 2007 13:32:24 -0000 @@ -5747,6 +5747,8 @@ (setq articles (cdr articles))) out)) +;; REVISIT: Never used and, given the other memory use issues, probably best +;; if it stays that way. Should, I think, be removed. --dp, 2007-10-02 (defun gnus-uncompress-marks (marks) "Uncompress the mark ranges in MARKS." (let ((uncompressed '(score bookmark)) @@ -6765,6 +6767,8 @@ (gnus-list-range-difference (gnus-list-range-difference (gnus-sorted-complement + ;; REVISIT: This needs `gnus-range-complement' implemented + ;; and tested, then we can drop the expansion of lists here. (gnus-uncompress-range (if gnus-newsgroup-maximum-articles (cons (max (car active) @@ -6772,7 +6776,7 @@ gnus-newsgroup-maximum-articles -1)) (cdr active)) - active)) + active) (gnus-list-of-unread-articles group)) (cdr (assq 'dormant marked))) (cdr (assq 'tick marked)))))) Index: lisp/nnimap.el =================================================================== RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v retrieving revision 7.36 diff -u -u -r7.36 nnimap.el --- lisp/nnimap.el 17 Aug 2007 11:09:00 -0000 7.36 +++ lisp/nnimap.el 2 Oct 2007 13:32:27 -0000 @@ -634,14 +634,13 @@ (imap-search (concat "UID " (imap-range-to-message-set - (gnus-compress-sequence - (append (gnus-uncompress-sequence - (and fetch-old - (cons (if (numberp fetch-old) - (max 1 (- (car articles) fetch-old)) - 1) - (1- (car articles))))) - articles))))) + (gnus-range-add + (and fetch-old + (cons (if (numberp fetch-old) + (max 1 (- (car articles) fetch-old)) + 1) + (1- (car articles)))) + articles)))) (mapcar (lambda (msgid) (imap-search (format "HEADER Message-Id \"%s\"" msgid))) --=-=-=--