From: "Bjørn Mork" <bjorn@mork.no>
To: ding@gnus.org
Subject: Re: expiry taking "forever" in both cpu time and real time when the IMAP UID is high
Date: Thu, 02 Dec 2010 14:28:32 +0100 [thread overview]
Message-ID: <87wrns8fsv.fsf@nemi.mork.no> (raw)
In-Reply-To: <m28w08misw.fsf@rimspace.net>
Daniel Pittman <daniel@rimspace.net> writes:
> Bjørn Mork <bjorn@mork.no> writes:
>
>> And then after having implemented the IMAP internal copy speedup, I noticed
>> that the "fetch all articles which are about to be expired one by one" code
>> isn't the worst part of the expiration process...
>>
>> After copying the articles, Gnus does "something" which evidently depends on
>> the UID value. When expiring from my work INBOX, which now is using UIDs in
>> the 280000s although it usually only has a few hundred messages, emacs will
>> eat 100% of one CPU core for a loooong time (several minutes).
>
> [...]
>
>> Any hints? Anyone else seen this?
>
> Oh, yeah. Gnus is expanding a range that includes that high number, so it can
> walk through it, then compress it back into a range. I /think/ this is the
> stuff that Larsi was just talking about fixing up with the "range tree" stuff
> and all.
>
> I mostly ran into it on Zimbra which used a sparse UID space, so could
> legitimately have gaps of 100 or even 1000 between consecutive message UIDs.
OK, then I hope it will be handled. I managed to trace my particular
problem down to gnus-summary-expire-articles() in gnus-sum.el which will
(dolist (article expirable)
where "expirable" is the result of gnus-list-of-read-articles() in the
if total-expire is true.
I find this very odd:
(gnus-compress-sequence (gnus-list-of-read-articles "nnimap+mail.mork.no:INBOX.test.source"))
((2 . 93) (95 . 199))
when I have:
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2010 Double Precision, Inc. See COPYING for distribution information.
a login test secret
a OK LOGIN Ok.
a select INBOX.test.source
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1285784686] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
a OK [READ-WRITE] Ok
a uid fetch 1:* flags
* 1 FETCH (UID 2 FLAGS (\Seen))
* 2 FETCH (UID 90 FLAGS (\Seen))
* 3 FETCH (UID 94 FLAGS ())
* 4 FETCH (UID 200 FLAGS ())
a OK FETCH completed.
And if I change the UID of the last message to 200000 I get:
(gnus-compress-sequence (gnus-list-of-read-articles "nnimap+mail.mork.no:INBOX.test.source"))
((2 . 93) (95 . 199999))
and
a select INBOX.test.source
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1285784686] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
a OK [READ-WRITE] Ok
a uid fetch 1:* flags
* 1 FETCH (UID 2 FLAGS (\Seen))
* 2 FETCH (UID 90 FLAGS (\Seen))
* 3 FETCH (UID 94 FLAGS ())
* 4 FETCH (UID 200000 FLAGS ())
a OK FETCH completed.
I really do not understand why the list-of-read-articles isn't just (2 90)
in both cases... Those are the only two articles with a \Seen flag. Why
care about the holes?
Anyway, you don't have to be a rocket scientist to see the problem with
the dolist and the 199996 non-existing UIDs. Make that a large number
and you'll *really* be into trouble.
Bjørn
next prev parent reply other threads:[~2010-12-02 13:28 UTC|newest]
Thread overview: 219+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-09-05 1:04 That newfangled IMAP thing Lars Magne Ingebrigtsen
2010-09-05 1:12 ` Steinar Bang
2010-09-05 11:20 ` Lars Magne Ingebrigtsen
2010-09-05 11:44 ` Sebastian Krause
2010-09-05 12:00 ` Frank Schmitt
2010-09-05 13:16 ` Steinar Bang
2010-09-05 19:58 ` James Cloos
2010-09-06 7:17 ` Bjørn Mork
2010-09-07 14:15 ` Wes Hardaker
2010-09-07 14:52 ` Richard Riley
2010-09-07 17:56 ` Wes Hardaker
2010-12-05 12:33 ` Matthias Andree
2010-09-05 1:56 ` Daniel Pittman
2010-09-05 2:17 ` Tibor Simko
2010-09-05 2:27 ` Ted Zlatanov
2010-09-05 7:18 ` Steinar Bang
2010-09-05 11:00 ` Bjørn Mork
2010-09-05 13:31 ` Steinar Bang
2010-09-05 11:23 ` Lars Magne Ingebrigtsen
2010-09-05 13:22 ` Steinar Bang
2010-09-05 13:25 ` Lars Magne Ingebrigtsen
2010-09-05 13:43 ` Steinar Bang
2010-09-05 17:26 ` Bjørn Mork
2010-09-05 17:30 ` Bjørn Mork
2010-09-06 9:05 ` Bjørn Mork
2010-09-06 15:02 ` Lars Magne Ingebrigtsen
2010-09-06 15:28 ` Lars Magne Ingebrigtsen
2010-09-09 14:54 ` Ted Zlatanov
2010-09-09 14:58 ` Lars Magne Ingebrigtsen
2010-09-09 15:07 ` Ted Zlatanov
2010-09-05 17:30 ` Lars Magne Ingebrigtsen
2010-09-05 10:55 ` Bjørn Mork
2010-09-05 13:28 ` Lars Magne Ingebrigtsen
2010-09-05 17:07 ` Bjørn Mork
2010-09-05 17:10 ` Lars Magne Ingebrigtsen
2010-09-05 17:30 ` Memnon Anon
2010-09-06 7:41 ` Bjørn Mork
2010-09-27 23:16 ` Bjørn Mork
2010-09-29 14:43 ` Lars Magne Ingebrigtsen
2010-09-29 19:19 ` Bjørn Mork
2010-12-02 12:03 ` [PATCH] nnimap.el (nnimap-process-expiry-targets): avoid downloading articles unless necessary Bjørn Mork
2010-12-02 12:17 ` expiry taking "forever" in both cpu time and real time when the IMAP UID is high Bjørn Mork
2010-12-02 12:59 ` Daniel Pittman
2010-12-02 13:28 ` Bjørn Mork [this message]
2010-12-05 12:21 ` Lars Magne Ingebrigtsen
2011-11-06 22:24 ` slow nnimap expiry (dubious patch) John Magorrian
2012-01-03 22:53 ` Lars Magne Ingebrigtsen
2012-02-13 8:56 ` Bjørn Mork
2012-02-13 19:27 ` Lars Ingebrigtsen
2012-02-13 20:48 ` Bjørn Mork
2012-02-26 14:27 ` Dave Abrahams
2012-03-10 0:51 ` Lars Magne Ingebrigtsen
2012-03-10 18:16 ` Dave Abrahams
2012-03-10 19:57 ` John Magorrian
2010-12-05 12:16 ` [PATCH] nnimap.el (nnimap-process-expiry-targets): avoid downloading articles unless necessary Lars Magne Ingebrigtsen
2010-09-05 12:43 ` That newfangled IMAP thing Greg Troxel
2010-09-05 12:56 ` Lars Magne Ingebrigtsen
2010-09-05 13:16 ` Lars Magne Ingebrigtsen
2010-09-05 13:38 ` Lars Magne Ingebrigtsen
2010-09-05 13:40 ` Lars Magne Ingebrigtsen
2010-09-05 13:46 ` Lars Magne Ingebrigtsen
2010-09-05 14:12 ` Lars Magne Ingebrigtsen
2010-09-05 14:43 ` Lars Magne Ingebrigtsen
2010-09-05 14:49 ` Steinar Bang
2010-09-05 16:59 ` Lars Magne Ingebrigtsen
2010-09-05 17:00 ` Lars Magne Ingebrigtsen
2010-09-05 17:47 ` Steinar Bang
2010-09-05 17:49 ` Steinar Bang
2010-09-06 12:50 ` Ted Zlatanov
2010-09-09 12:55 ` Steinar Bang
2010-09-09 13:55 ` Bjørn Mork
2010-09-09 13:59 ` Lars Magne Ingebrigtsen
2010-09-09 14:26 ` Daniel Pittman
2010-09-09 17:14 ` Sebastian Krause
2010-09-10 23:33 ` Lars Magne Ingebrigtsen
2010-09-10 23:53 ` Jason L Tibbitts III
2010-09-11 0:02 ` Lars Magne Ingebrigtsen
2010-09-11 0:16 ` Greg Troxel
2010-09-11 0:31 ` Lars Magne Ingebrigtsen
2010-09-11 3:42 ` Daniel Pittman
2010-09-11 11:07 ` Ted Zlatanov
2010-09-11 11:47 ` Julien Danjou
2010-09-11 15:44 ` Lars Magne Ingebrigtsen
2010-09-11 17:20 ` Lars Magne Ingebrigtsen
2010-09-11 17:30 ` Julien Danjou
2010-09-11 17:31 ` Lars Magne Ingebrigtsen
2010-09-12 8:38 ` Daniel Pittman
2010-09-12 9:24 ` Julien Danjou
2010-09-12 9:53 ` Daniel Pittman
2010-09-12 11:12 ` Richard Riley
2010-09-12 13:15 ` Lars Magne Ingebrigtsen
2010-09-12 15:12 ` Lars Magne Ingebrigtsen
2010-09-12 15:40 ` Steinar Bang
2010-09-12 16:03 ` Lars Magne Ingebrigtsen
2010-09-12 16:56 ` Frank Schmitt
2010-09-12 17:03 ` Lars Magne Ingebrigtsen
2010-09-12 17:26 ` Steinar Bang
2010-09-12 17:29 ` Lars Magne Ingebrigtsen
2010-09-12 19:47 ` Lars Magne Ingebrigtsen
2010-09-12 20:19 ` Lars Magne Ingebrigtsen
2010-09-13 4:28 ` Daniel Pittman
2010-09-12 17:24 ` Richard Riley
2010-09-13 10:54 ` Ted Zlatanov
2010-09-13 4:26 ` Daniel Pittman
2010-09-13 9:28 ` Lars Magne Ingebrigtsen
2010-09-13 9:49 ` Steinar Bang
2010-09-13 9:57 ` David Edmondson
2010-09-13 10:20 ` Lars Magne Ingebrigtsen
2010-09-13 9:58 ` Lars Magne Ingebrigtsen
2010-09-13 10:07 ` Julien Danjou
2010-09-13 10:24 ` Steinar Bang
2010-09-13 11:06 ` Lars Magne Ingebrigtsen
2010-09-13 11:54 ` Daniel Pittman
2010-09-13 12:20 ` Steinar Bang
2010-09-11 20:52 ` Lars Magne Ingebrigtsen
2010-09-12 2:17 ` Lars Magne Ingebrigtsen
2010-09-12 8:32 ` Daniel Pittman
2010-09-12 13:08 ` Lars Magne Ingebrigtsen
2010-09-13 4:24 ` Daniel Pittman
2010-09-13 10:51 ` Ted Zlatanov
2010-09-13 11:09 ` Lars Magne Ingebrigtsen
2010-09-13 12:24 ` Ted Zlatanov
2010-09-13 12:58 ` Lars Magne Ingebrigtsen
2010-09-13 14:22 ` Lars Magne Ingebrigtsen
2010-09-13 20:36 ` Lars Magne Ingebrigtsen
2010-09-13 21:43 ` Knut Anders Hatlen
2010-09-13 21:55 ` Lars Magne Ingebrigtsen
2010-09-13 22:12 ` Knut Anders Hatlen
2010-09-13 22:14 ` Steinar Bang
2010-09-13 22:54 ` Dan Christensen
2010-09-14 7:19 ` Steinar Bang
2010-09-14 15:10 ` Lars Magne Ingebrigtsen
2010-09-14 15:18 ` Julien Danjou
2010-09-14 15:29 ` Steinar Bang
2010-09-14 15:30 ` Ted Zlatanov
2010-09-14 15:44 ` Lars Magne Ingebrigtsen
2010-09-14 16:21 ` Lars Magne Ingebrigtsen
2010-09-14 18:51 ` Steinar Bang
2010-09-14 19:01 ` Lars Magne Ingebrigtsen
2010-09-14 19:23 ` Steinar Bang
2010-09-14 19:26 ` Lars Magne Ingebrigtsen
2010-09-14 19:47 ` Steinar Bang
2010-09-14 19:57 ` Lars Magne Ingebrigtsen
2010-09-14 20:07 ` Steinar Bang
2010-09-14 19:19 ` Ted Zlatanov
2010-09-14 19:24 ` Lars Magne Ingebrigtsen
2010-09-14 19:35 ` Ted Zlatanov
2010-09-14 19:40 ` Lars Magne Ingebrigtsen
2010-09-14 19:40 ` Julien Danjou
2010-09-14 19:43 ` Lars Magne Ingebrigtsen
2010-09-14 19:46 ` Ted Zlatanov
2010-09-15 8:37 ` Julien Danjou
2010-09-14 19:26 ` Ted Zlatanov
2010-09-14 19:28 ` Lars Magne Ingebrigtsen
2010-09-14 19:44 ` Ted Zlatanov
2010-09-15 10:24 ` Steinar Bang
2010-09-14 19:35 ` Julien Danjou
2010-09-14 19:44 ` Dan Christensen
2010-09-14 19:50 ` Ted Zlatanov
2010-09-14 20:07 ` Lars Magne Ingebrigtsen
2010-09-14 20:23 ` Ted Zlatanov
2010-09-14 20:49 ` Frank Schmitt
2010-09-15 6:05 ` Steinar Bang
2010-09-15 7:33 ` Lars Magne Ingebrigtsen
2010-09-13 23:56 ` Lars Magne Ingebrigtsen
2010-09-14 0:10 ` Lars Magne Ingebrigtsen
2010-09-14 0:35 ` Lars Magne Ingebrigtsen
2010-09-14 10:58 ` Ted Zlatanov
2010-09-12 2:18 ` Ted Zlatanov
2010-09-12 2:32 ` Lars Magne Ingebrigtsen
2010-09-06 15:23 ` Dan Christensen
2010-09-06 20:14 ` Lars Magne Ingebrigtsen
2010-09-06 21:01 ` Steinar Bang
2010-09-06 21:05 ` Lars Magne Ingebrigtsen
2010-09-06 21:27 ` Steinar Bang
2010-09-06 21:29 ` Lars Magne Ingebrigtsen
2010-09-06 21:39 ` Steinar Bang
2010-09-07 0:57 ` Daniel Pittman
2010-09-07 1:41 ` Lars Magne Ingebrigtsen
2010-09-07 2:49 ` Daniel Pittman
2010-09-07 6:14 ` Gijs Hillenius
2010-09-07 7:17 ` Steinar Bang
2010-09-07 14:57 ` Bjørn Mork
2010-09-07 17:59 ` James Cloos
2010-09-07 18:16 ` Steinar Bang
2010-09-08 8:34 ` Vegard Vesterheim
2010-09-08 9:06 ` Steinar Bang
2010-09-08 10:00 ` Vegard Vesterheim
2010-09-08 10:07 ` Julien Danjou
2010-09-08 12:06 ` Steinar Bang
2010-09-08 12:18 ` Julien Danjou
2010-09-08 14:29 ` Vegard Vesterheim
2010-09-08 14:43 ` Steinar Bang
2010-09-08 11:50 ` Steinar Bang
2010-09-08 21:26 ` Wes Hardaker
2010-09-05 15:02 ` Mark Plaksin
2010-09-05 16:01 ` David Engster
2010-09-05 16:06 ` David Engster
2010-09-05 16:15 ` Lars Magne Ingebrigtsen
2010-09-05 18:15 ` Steinar Bang
2010-09-05 18:21 ` Lars Magne Ingebrigtsen
2010-09-05 19:46 ` Steinar Bang
2010-09-05 19:58 ` Lars Magne Ingebrigtsen
2010-09-06 12:57 ` Ted Zlatanov
2010-09-06 13:01 ` Lars Magne Ingebrigtsen
2010-09-06 13:12 ` Ted Zlatanov
2010-09-05 20:04 ` Julien Danjou
2010-09-05 20:10 ` Lars Magne Ingebrigtsen
2010-09-05 14:44 ` Tom Rauchenwald
2010-09-06 3:51 ` Daniel Pittman
2010-09-06 8:45 ` Julien Danjou
2010-09-06 13:39 ` Romain Francoise
2010-09-06 14:23 ` Julien Danjou
2010-09-06 21:01 ` Daniel Dehennin
2010-09-06 21:04 ` Lars Magne Ingebrigtsen
2010-09-06 16:26 ` Gijs Hillenius
2010-09-06 16:40 ` Lars Magne Ingebrigtsen
2010-09-05 13:27 ` Steinar Bang
2010-09-05 17:25 ` Frank Schmitt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87wrns8fsv.fsf@nemi.mork.no \
--to=bjorn@mork.no \
--cc=ding@gnus.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).