* [PATCH] nnimap performance improvement for large or old groups
@ 2006-03-03 1:38 Daniel Pittman
2006-03-03 9:34 ` Simon Josefsson
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Pittman @ 2006-03-03 1:38 UTC (permalink / raw)
Cc: Simon Josefsson
[-- Attachment #1: Type: text/plain, Size: 941 bytes --]
One of the nagging problems with using nnimap as my primary mail store
is that it takes absolutely forever to enter my INBOX, as well as
causing XEmacs to allocate around 150MB of memory.
This is because I have been using the same INBOX for a long while now,
and the 'read' info range starts with '(1 . 695705)'
The code in `nnimap-request-update-info-internal' called
`gnus-uncompress-range' on this, resulting in a list containing around
seven million numbers -- an awful lot of memory, and time spent working
through it.
To address this I rewrote the code in that routine to work with
compressed ranges, rather than uncompressed, which gives me a huge
performance improvement (almost instant entry, vs ten to fifteen
seconds) and reduces the memory use significantly.
I don't think this is too performance-inefficient to include as is, and
it doesn't seem to adversely effect entry into small groups or anything
like that.
Daniel
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap-performance.patch --]
[-- Type: text/x-patch, Size: 1454 bytes --]
? build.info
? build.sh
? nnimap-performance.patch
? contrib/auto-autoloads.el
? contrib/semantic.cache
Index: lisp/nnimap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v
retrieving revision 7.30
diff -u -u -r7.30 nnimap.el
--- lisp/nnimap.el 21 Feb 2006 07:14:23 -0000 7.30
+++ lisp/nnimap.el 3 Mar 2006 01:37:36 -0000
@@ -1183,18 +1183,12 @@
(let (seen unseen)
;; read info could contain articles marked unread by other
;; imap clients! we correct this
- (setq seen (gnus-uncompress-range (gnus-info-read info))
- unseen (imap-search "UNSEEN UNDELETED")
- seen (gnus-set-difference seen unseen)
- ;; seen might lack articles marked as read by other
- ;; imap clients! we correct this
- seen (append seen (imap-search "SEEN"))
- ;; remove dupes
- seen (sort seen '<)
- seen (gnus-compress-sequence seen t)
- ;; we can't return '(1) since this isn't a "list of ranges",
- ;; and we can't return '((1)) since g-list-of-unread-articles
- ;; is buggy so we return '((1 . 1)).
+ (setq unseen (gnus-compress-sequence
+ (imap-search "UNSEEN UNDELETED"))
+ seen (gnus-range-difference (gnus-info-read info) unseen)
+ seen (gnus-range-add seen
+ (gnus-compress-sequence
+ (imap-search "SEEN")))
seen (if (and (integerp (car seen))
(null (cdr seen)))
(list (cons (car seen) (car seen)))
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] nnimap performance improvement for large or old groups
2006-03-03 1:38 [PATCH] nnimap performance improvement for large or old groups Daniel Pittman
@ 2006-03-03 9:34 ` Simon Josefsson
2006-03-04 0:00 ` Daniel Pittman
0 siblings, 1 reply; 6+ messages in thread
From: Simon Josefsson @ 2006-03-03 9:34 UTC (permalink / raw)
Cc: ding
Daniel Pittman <daniel@rimspace.net> writes:
> One of the nagging problems with using nnimap as my primary mail store
> is that it takes absolutely forever to enter my INBOX, as well as
> causing XEmacs to allocate around 150MB of memory.
>
> This is because I have been using the same INBOX for a long while now,
> and the 'read' info range starts with '(1 . 695705)'
>
> The code in `nnimap-request-update-info-internal' called
> `gnus-uncompress-range' on this, resulting in a list containing around
> seven million numbers -- an awful lot of memory, and time spent working
> through it.
>
> To address this I rewrote the code in that routine to work with
> compressed ranges, rather than uncompressed, which gives me a huge
> performance improvement (almost instant entry, vs ten to fifteen
> seconds) and reduces the memory use significantly.
>
>
> I don't think this is too performance-inefficient to include as is, and
> it doesn't seem to adversely effect entry into small groups or anything
> like that.
Your patch seem correct. Installed, thanks!
It should probably be tested for a while before being included in
v5-10 though, the logic seem somewhat complicated and while I
convinced myself that it does the same thing, I might be mistaken.
> Daniel
>
> ? build.info
> ? build.sh
> ? nnimap-performance.patch
> ? contrib/auto-autoloads.el
> ? contrib/semantic.cache
> Index: lisp/nnimap.el
> ===================================================================
> RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v
> retrieving revision 7.30
> diff -u -u -r7.30 nnimap.el
> --- lisp/nnimap.el 21 Feb 2006 07:14:23 -0000 7.30
> +++ lisp/nnimap.el 3 Mar 2006 01:37:36 -0000
> @@ -1183,18 +1183,12 @@
> (let (seen unseen)
> ;; read info could contain articles marked unread by other
> ;; imap clients! we correct this
> - (setq seen (gnus-uncompress-range (gnus-info-read info))
> - unseen (imap-search "UNSEEN UNDELETED")
> - seen (gnus-set-difference seen unseen)
> - ;; seen might lack articles marked as read by other
> - ;; imap clients! we correct this
> - seen (append seen (imap-search "SEEN"))
> - ;; remove dupes
> - seen (sort seen '<)
> - seen (gnus-compress-sequence seen t)
> - ;; we can't return '(1) since this isn't a "list of ranges",
> - ;; and we can't return '((1)) since g-list-of-unread-articles
> - ;; is buggy so we return '((1 . 1)).
> + (setq unseen (gnus-compress-sequence
> + (imap-search "UNSEEN UNDELETED"))
> + seen (gnus-range-difference (gnus-info-read info) unseen)
> + seen (gnus-range-add seen
> + (gnus-compress-sequence
> + (imap-search "SEEN")))
> seen (if (and (integerp (car seen))
> (null (cdr seen)))
> (list (cons (car seen) (car seen)))
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] nnimap performance improvement for large or old groups
2006-03-03 9:34 ` Simon Josefsson
@ 2006-03-04 0:00 ` Daniel Pittman
2006-03-05 22:10 ` Simon Josefsson
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Pittman @ 2006-03-04 0:00 UTC (permalink / raw)
Simon Josefsson <jas@extundo.com> writes:
> Daniel Pittman <daniel@rimspace.net> writes:
>
>> One of the nagging problems with using nnimap as my primary mail store
>> is that it takes absolutely forever to enter my INBOX, as well as
>> causing XEmacs to allocate around 150MB of memory.
>>
>> This is because I have been using the same INBOX for a long while now,
>> and the 'read' info range starts with '(1 . 695705)'
>>
>> The code in `nnimap-request-update-info-internal' called
>> `gnus-uncompress-range' on this, resulting in a list containing around
>> seven million numbers -- an awful lot of memory, and time spent working
>> through it.
>>
>> To address this I rewrote the code in that routine to work with
>> compressed ranges, rather than uncompressed, which gives me a huge
>> performance improvement (almost instant entry, vs ten to fifteen
>> seconds) and reduces the memory use significantly.
>>
>>
>> I don't think this is too performance-inefficient to include as is, and
>> it doesn't seem to adversely effect entry into small groups or anything
>> like that.
>
> Your patch seem correct. Installed, thanks!
>
> It should probably be tested for a while before being included in
> v5-10 though, the logic seem somewhat complicated and while I
> convinced myself that it does the same thing, I might be mistaken.
I completely agree that this should have long and thorough testing, for
all that I did very carefully check the way that it worked, and made
sure to verify every step of the new computation manually.
Oh, but it surely is *nice* to have that in there, though. :)
Daniel
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] nnimap performance improvement for large or old groups
2006-03-04 0:00 ` Daniel Pittman
@ 2006-03-05 22:10 ` Simon Josefsson
2006-03-15 10:32 ` Frank Schmitt
0 siblings, 1 reply; 6+ messages in thread
From: Simon Josefsson @ 2006-03-05 22:10 UTC (permalink / raw)
Cc: ding
Daniel Pittman <daniel@rimspace.net> writes:
> Simon Josefsson <jas@extundo.com> writes:
>> Daniel Pittman <daniel@rimspace.net> writes:
>>
>>> One of the nagging problems with using nnimap as my primary mail store
>>> is that it takes absolutely forever to enter my INBOX, as well as
>>> causing XEmacs to allocate around 150MB of memory.
>>>
>>> This is because I have been using the same INBOX for a long while now,
>>> and the 'read' info range starts with '(1 . 695705)'
>>>
>>> The code in `nnimap-request-update-info-internal' called
>>> `gnus-uncompress-range' on this, resulting in a list containing around
>>> seven million numbers -- an awful lot of memory, and time spent working
>>> through it.
>>>
>>> To address this I rewrote the code in that routine to work with
>>> compressed ranges, rather than uncompressed, which gives me a huge
>>> performance improvement (almost instant entry, vs ten to fifteen
>>> seconds) and reduces the memory use significantly.
>>>
>>>
>>> I don't think this is too performance-inefficient to include as is, and
>>> it doesn't seem to adversely effect entry into small groups or anything
>>> like that.
>>
>> Your patch seem correct. Installed, thanks!
>>
>> It should probably be tested for a while before being included in
>> v5-10 though, the logic seem somewhat complicated and while I
>> convinced myself that it does the same thing, I might be mistaken.
>
> I completely agree that this should have long and thorough testing, for
> all that I did very carefully check the way that it worked, and made
> sure to verify every step of the new computation manually.
There are some subtle situations where a message is marked unseen on
the server (by another client), which should be propagated back into
gnus, and also when an article is removed.
But it looks OK to me. So if nobody complains, let's move it to v5-10
in 3 months.
> Oh, but it surely is *nice* to have that in there, though. :)
Yup.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] nnimap performance improvement for large or old groups
2006-03-05 22:10 ` Simon Josefsson
@ 2006-03-15 10:32 ` Frank Schmitt
2006-03-30 18:37 ` Wes Hardaker
0 siblings, 1 reply; 6+ messages in thread
From: Frank Schmitt @ 2006-03-15 10:32 UTC (permalink / raw)
Simon Josefsson <jas@extundo.com> writes:
> There are some subtle situations where a message is marked unseen on
> the server (by another client), which should be propagated back into
> gnus, and also when an article is removed.
>
> But it looks OK to me. So if nobody complains, let's move it to v5-10
> in 3 months.
I don't complain, I praise. I love it!
--
Did you ever realize how much text fits in eighty columns? If you now consider
that a signature usually consists of up to four lines, this gives you enough
space to spread a tremendous amount of information with your messages. So seize
this opportunity and don't waste your signature with bullshit nobody will read.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] nnimap performance improvement for large or old groups
2006-03-15 10:32 ` Frank Schmitt
@ 2006-03-30 18:37 ` Wes Hardaker
0 siblings, 0 replies; 6+ messages in thread
From: Wes Hardaker @ 2006-03-30 18:37 UTC (permalink / raw)
Oh my gosh I'm in love. Thank you thank you thank you!
Something like that patch has been needed for years, and I've been
solving the problem by entering groups specifying the rough number of
articles I need to see (C-u 50 ENTER) to speed up loading. With that
patch the fetching is much much faster (though summary generation of
lots of articles is still slow, but that's another battle).
--
"In the bathtub of history the truth is harder to hold than the soap,
and much more difficult to find." -- Terry Pratchett
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2006-03-30 18:37 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2006-03-03 1:38 [PATCH] nnimap performance improvement for large or old groups Daniel Pittman
2006-03-03 9:34 ` Simon Josefsson
2006-03-04 0:00 ` Daniel Pittman
2006-03-05 22:10 ` Simon Josefsson
2006-03-15 10:32 ` Frank Schmitt
2006-03-30 18:37 ` Wes Hardaker
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).