slow nnimap moves due do many SELECTs

slow nnimap moves due do many SELECTs

[Ted Zlatanov]

Moving N articles results in:

09:00:37 190 SELECT "INBOX"
09:00:37 191 UID FETCH 274385 BODY.PEEK[]
09:00:37 192 UID COPY 274385 "INBOX.ding"
09:00:37 193 UID STORE 274385 +FLAGS.SILENT (\Deleted)
09:00:37 194 UID EXPUNGE 274385
09:00:37 195 SELECT "INBOX.ding"
09:00:37 196 UID SEARCH HEADER Message-Id "<87tylhhpie.fsf@lifelogs.com>"

09:00:52 197 SELECT "INBOX"
09:00:52 198 UID FETCH 274377 BODY.PEEK[]
09:00:52 199 UID COPY 274377 "INBOX.ding"
09:00:52 200 UID STORE 274377 +FLAGS.SILENT (\Deleted)
09:00:52 201 UID EXPUNGE 274377
09:00:52 202 SELECT "INBOX.ding"
09:00:53 203 UID SEARCH HEADER Message-Id "<87r5gmoriu.fsf@andy.bu.edu>"

etc. for each article.  Can than be optimized to switch mailboxes only
once?

Thanks
Ted

Re: slow nnimap moves due do many SELECTs

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Moving N articles results in:
>
> 09:00:37 190 SELECT "INBOX"
> 09:00:37 191 UID FETCH 274385 BODY.PEEK[]
> 09:00:37 192 UID COPY 274385 "INBOX.ding"
> 09:00:37 193 UID STORE 274385 +FLAGS.SILENT (\Deleted)
> 09:00:37 194 UID EXPUNGE 274385
> 09:00:37 195 SELECT "INBOX.ding"
> 09:00:37 196 UID SEARCH HEADER Message-Id "<87tylhhpie.fsf@lifelogs.com>"

[...]

> etc. for each article.  Can than be optimized to switch mailboxes only
> once?

Hm.  Well, the thing is that we need to select the group we moved to to
find out what the article number we got was.  Because Gnus needs that to
copy over the marks and other stuff.  So copying an article from group A
to group B involves at least to SELECTs.

However, it's possible to imagine an interface where you say "here's a
list of articles to move, and where they are going to be moved, and then
you return the list of article numbers".  Then nnimap could issue one
SELECT for the group you're moving from, and one SELECT per mailbox it
moved things to.  So if you moved 400 messages to INBOX.ding, there'd be
only two SELECTs.

However, this would require quite a bit of fiddling with the interface
functions...

But there may be other ways to fix this that I'm not seeing?  Is there a
way to get the UID of a message without SELECTing the mailbox it's in
first?  Or a way to get the new UID from the UID COPY command?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow nnimap moves due do many SELECTs

[Michael Welsh Duggan]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> Moving N articles results in:
>>
>> 09:00:37 190 SELECT "INBOX"
>> 09:00:37 191 UID FETCH 274385 BODY.PEEK[]
>> 09:00:37 192 UID COPY 274385 "INBOX.ding"
>> 09:00:37 193 UID STORE 274385 +FLAGS.SILENT (\Deleted)
>> 09:00:37 194 UID EXPUNGE 274385
>> 09:00:37 195 SELECT "INBOX.ding"
>> 09:00:37 196 UID SEARCH HEADER Message-Id "<87tylhhpie.fsf@lifelogs.com>"
>
> [...]
>
>> etc. for each article.  Can than be optimized to switch mailboxes only
>> once?
>
> Hm.  Well, the thing is that we need to select the group we moved to to
> find out what the article number we got was.  Because Gnus needs that to
> copy over the marks and other stuff.  So copying an article from group A
> to group B involves at least to SELECTs.

You may not need to copy over the marks.  The spec says: "The flags and
internal date of the message(s) SHOULD be preserved, and the Recent flag
SHOULD be set, in the copy."  If the SHOULD is bothersome, it would be
nice to have a per-server variable we can set if we know that that
server handles this correctly. 

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: slow nnimap moves due do many SELECTs

[Lars Magne Ingebrigtsen]

Michael Welsh Duggan <md5i@md5i.com> writes:

> You may not need to copy over the marks.  The spec says: "The flags and
> internal date of the message(s) SHOULD be preserved, and the Recent flag
> SHOULD be set, in the copy."  If the SHOULD is bothersome, it would be
> nice to have a per-server variable we can set if we know that that
> server handles this correctly. 

Gnus copies over marks on articles on `B m' if the backend supports it.
IMAP does, so Gnus does.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow nnimap moves due do many SELECTs

[Michael Welsh Duggan]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <md5i@md5i.com> writes:
>
>> You may not need to copy over the marks.  The spec says: "The flags and
>> internal date of the message(s) SHOULD be preserved, and the Recent flag
>> SHOULD be set, in the copy."  If the SHOULD is bothersome, it would be
>> nice to have a per-server variable we can set if we know that that
>> server handles this correctly. 
>
> Gnus copies over marks on articles on `B m' if the backend supports it.
> IMAP does, so Gnus does.

I think I may have not provided enough context, or I have misunderstood
your response.  The above applies to the IMAP COPY command, which SHOULD
preserve marks.  Which would mean that Gnus may not have to do this
manually.  Which would mean that it does not need the ID of the copied
message in the destination folder.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: slow nnimap moves due do many SELECTs

[Lars Magne Ingebrigtsen]

Michael Welsh Duggan <md5i@md5i.com> writes:

> I think I may have not provided enough context, or I have misunderstood
> your response.  The above applies to the IMAP COPY command, which SHOULD
> preserve marks.  Which would mean that Gnus may not have to do this
> manually.  Which would mean that it does not need the ID of the copied
> message in the destination folder.

Sorry; I misunderstood.

Well, that should make things much faster.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow nnimap moves due do many SELECTs

[Ted Zlatanov]

On Wed, 22 Sep 2010 19:57:12 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Michael Welsh Duggan <md5i@md5i.com> writes:
>> I think I may have not provided enough context, or I have misunderstood
>> your response.  The above applies to the IMAP COPY command, which SHOULD
>> preserve marks.  Which would mean that Gnus may not have to do this
>> manually.  Which would mean that it does not need the ID of the copied
>> message in the destination folder.

LMI> Sorry; I misunderstood.

LMI> Well, that should make things much faster.

Have you had a chance to try turning off the mark copy on IMAP
copy/move?  It's making article moves and copies REALLY slow with large
IMAP groups and it seems like a simple (negative lines of code) fix.

Ted

Re: slow nnimap moves due do many SELECTs

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Have you had a chance to try turning off the mark copy on IMAP
> copy/move?  It's making article moves and copies REALLY slow with large
> IMAP groups and it seems like a simple (negative lines of code) fix.

I've looked over the code, and Gnus needs to get the article ID back
from `nnimap-request-move-article' not only for marks copying (which, as
you point out, can be done on the IMAP level), but also for updating
active info and stuff.  I'm not 100% sure that all that useful,
though...

The scenario is that if we copy a message from group A to group B, then
Gnus (at present) knows immediately that group B has gotten the new
article, and adjusts the active ranges and the group buffer
automatically.  I think.  Let's test...  Yes.  So that wouldn't work any
more with the IMAP COPY thing.

So if we want that to work, we have to SELECT the group we're moving to,
unfortunately.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow nnimap moves due do many SELECTs

[Ted Zlatanov]

On Wed, 29 Sep 2010 23:46:21 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> Have you had a chance to try turning off the mark copy on IMAP
>> copy/move?  It's making article moves and copies REALLY slow with large
>> IMAP groups and it seems like a simple (negative lines of code) fix.

LMI> I've looked over the code, and Gnus needs to get the article ID back
LMI> from `nnimap-request-move-article' not only for marks copying (which, as
LMI> you point out, can be done on the IMAP level), but also for updating
LMI> active info and stuff.  I'm not 100% sure that all that useful,
LMI> though...

LMI> The scenario is that if we copy a message from group A to group B, then
LMI> Gnus (at present) knows immediately that group B has gotten the new
LMI> article, and adjusts the active ranges and the group buffer
LMI> automatically.  I think.  Let's test...  Yes.  So that wouldn't work any
LMI> more with the IMAP COPY thing.

LMI> So if we want that to work, we have to SELECT the group we're moving to,
LMI> unfortunately.

OK, I can live with the slowness if it happens once.  But currently it
SELECTs the destination group for *every* article move instead of
batching the moves.

Ted

Re: slow nnimap moves due do many SELECTs

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> OK, I can live with the slowness if it happens once.  But currently it
> SELECTs the destination group for *every* article move instead of
> batching the moves.

That would mean changing the calling interface rather drastically.

So we have three options:

1) have nnimap moves be slow

2) have them be fast, but not have the same features as other moves
(i.e., updating groups immediately)

3) rework the entire moving/accepting interface for all backends

I'm leaning towards 2)...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow nnimap moves due do many SELECTs

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm leaning towards 2)...

Or option pi), which I've now implemented: Use EXAMINE instead of
SELECT.  On some servers EXAMINE is a lot faster.  Could you try it and
see whether this helps at all?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow nnimap moves due do many SELECTs

[Ted Zlatanov]

On Thu, 30 Sep 2010 18:43:31 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> I'm leaning towards 2)...

LMI> Or option pi), which I've now implemented: Use EXAMINE instead of
LMI> SELECT.  On some servers EXAMINE is a lot faster.  Could you try it and
LMI> see whether this helps at all?

Yes, that's a bit faster.  SELECT does a lot more so that makes sense;
is there any downside to EXAMINE?

On Thu, 30 Sep 2010 19:02:48 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:

>> Why not use the LAST parameter to nnimap-request-move-article to
>> trigger the group update?  It's only set for the last article in the
>> list AFAICT.

LMI> I don't really want to rewrite all the backend -move and -accept
LMI> methods.  There must be at least ten of them...

I don't understand.  The LAST parameter is already there and given to
you by Gnus (it's t only for the last article) when the user runs `B m',
right?

Ted

Re: slow nnimap moves due do many SELECTs

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Yes, that's a bit faster.  SELECT does a lot more so that makes sense;
> is there any downside to EXAMINE?

Hm...  no, not really.  Well, if you want to read the group you've just
moved the articles to later, you need to SELECT it then, but that's
rather ... not a big deal.

> I don't understand.  The LAST parameter is already there and given to
> you by Gnus (it's t only for the last article) when the user runs `B m',
> right?

So I could just rewrite the gnus-summary-move-article function to expect
back a list of articles on LAST?  Yes, that should work...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow nnimap moves due do many SELECTs

[Ted Zlatanov]

On Thu, 30 Sep 2010 18:35:31 +0200 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Ted Zlatanov <tzz@lifelogs.com> writes:
>> OK, I can live with the slowness if it happens once.  But currently it
>> SELECTs the destination group for *every* article move instead of
>> batching the moves.

LMI> That would mean changing the calling interface rather drastically.

LMI> So we have three options:

LMI> 1) have nnimap moves be slow

LMI> 2) have them be fast, but not have the same features as other moves
LMI> (i.e., updating groups immediately)

LMI> 3) rework the entire moving/accepting interface for all backends

Why not use the LAST parameter to nnimap-request-move-article to trigger
the group update?  It's only set for the last article in the list AFAICT.

Ted

Re: slow nnimap moves due do many SELECTs

[Lars Magne Ingebrigtsen]

Ted Zlatanov <tzz@lifelogs.com> writes:

> Why not use the LAST parameter to nnimap-request-move-article to
> trigger the group update?  It's only set for the last article in the
> list AFAICT.

I don't really want to rewrite all the backend -move and -accept
methods.  There must be at least ten of them...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: slow nnimap moves due do many SELECTs

[Austin F. Frank]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

On Tue, Sep 28 2010, Ted Zlatanov wrote:

> Have you had a chance to try turning off the mark copy on IMAP
> copy/move?  It's making article moves and copies REALLY slow with
> large IMAP groups and it seems like a simple (negative lines of code)
> fix.

Just a quick followup on this.  I'm also seeing very slow moves, and
wondered whether the resolution of this is basically that it's a "won't
fix"/"not broken", or whether there's still work to do.  If the former,
I'll deal.  If the latter, how can I help speed moves up?

Thanks,
/au

-- 
Austin Frank
http://aufrank.net
GPG Public Key (D7398C2F): http://aufrank.net/personal.asc

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: slow nnimap moves due do many SELECTs

[Ted Zlatanov]

On Wed, 22 Sep 2010 13:09:39 -0400 Michael Welsh Duggan <md5i@md5i.com> wrote: 

MWD> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>> 
>>> Moving N articles results in:
>>> 
>>> 09:00:37 190 SELECT "INBOX"
...
>>> 09:00:37 195 SELECT "INBOX.ding"

>> Hm.  Well, the thing is that we need to select the group we moved to to
>> find out what the article number we got was.  Because Gnus needs that to
>> copy over the marks and other stuff.  So copying an article from group A
>> to group B involves at least to SELECTs.

MWD> You may not need to copy over the marks.  The spec says: "The flags and
MWD> internal date of the message(s) SHOULD be preserved, and the Recent flag
MWD> SHOULD be set, in the copy."  If the SHOULD is bothersome, it would be
MWD> nice to have a per-server variable we can set if we know that that
MWD> server handles this correctly. 

I hope that's well-supported, because SELECTs are very expensive on at
least Courier IMAP and probably most other IMAP servers.

Ted