IMAP article move

IMAP article move

[Ted Zlatanov]

Can we optimize IMAP article moves so they use the server
capabilities?  Right now, I think, we just copy and then delete the
old article.  Across various backends that makes sense, but if it's
within the same IMAP backend, the server move can be much faster.

I looked at gnus-sum.el, gnus-int.el, and nnimap.el.  It seems
possible, although the changes would have to go all the way to the
top, because gnus-summary-move-article does the copy & delete action.

If this works, Gnus will be much faster and more reliable in the
common case of article moves in IMAP.

Ted

Re: IMAP article move

[Raymond Scholz]

· "Ted Zlatanov" <tzz@lifelogs.com> wrote:

> Can we optimize IMAP article moves so they use the server
> capabilities?  Right now, I think, we just copy and then delete the
> old article.  Across various backends that makes sense, but if it's
> within the same IMAP backend, the server move can be much faster.

Frank Schmitt has dome some work.

;; From:  Frank Schmitt <ich@Frank-Schmitt.net>
;; Subject: [Code] Copying/Moving mails on imap server
;; Newsgroups: gmane.emacs.gnus.general
;; Date: Sat Feb 28 21:44:18 2004 +0100
;; Message-ID: <m38yin0w9p.fsf@fed1.frank-schmitt.net>

You can find it on Gmane:

http://thread.gmane.org/gmane.emacs.gnus.general/56455

Cheers, Ray
-- 
/* Halley */ (Halley's comment.)

Re: IMAP article move

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> Can we optimize IMAP article moves so they use the server
> capabilities?  Right now, I think, we just copy and then delete the
> old article.  Across various backends that makes sense, but if it's
> within the same IMAP backend, the server move can be much faster.
>
> I looked at gnus-sum.el, gnus-int.el, and nnimap.el.  It seems
> possible, although the changes would have to go all the way to the
> top, because gnus-summary-move-article does the copy & delete action.
>
> If this works, Gnus will be much faster and more reliable in the
> common case of article moves in IMAP.

What are you proposing?  There is some optimization in
nnimap-request-move-article and nnimap-request-accept-article to make
moving more efficient, and I thought it was pretty fast.  But I rarely
copy/move articles, so I wouldn't know.  Maybe you could illustrate by
comparing which IMAP commands are sent now, and which commands you
want to be sent?

Re: IMAP article move

[Ted Zlatanov]

On Mon, 08 Nov 2004, jas@extundo.com wrote:

> What are you proposing?  There is some optimization in
> nnimap-request-move-article and nnimap-request-accept-article to make
> moving more efficient, and I thought it was pretty fast.  But I rarely
> copy/move articles, so I wouldn't know.  Maybe you could illustrate by
> comparing which IMAP commands are sent now, and which commands you
> want to be sent?

I apologize.  Reading the source code, I thought that the IMAP COPY
(imap-message-copy) was not used, but it is.  There does seem to be an
unnecessary full-body FETCH before the COPY, though, which is what
slows things down (I think).  I just entered the group and did a `B m'
command, and the log showed that the article was fetched in its full
glory before it was moved.

I think this is something that Gnus does for every backend when
moving, because it's needed in some cases (where you don't do the
internal backend move but a fetch+store instead).  So the extra fetch
just needs to be eliminated when moving within the same IMAP server.

nnml and nnmaildir could probably benefit from the same optimization,
if they do a file move internally as well.

Ted

Re: IMAP article move

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On Mon, 08 Nov 2004, jas@extundo.com wrote:
>
>> What are you proposing?  There is some optimization in
>> nnimap-request-move-article and nnimap-request-accept-article to make
>> moving more efficient, and I thought it was pretty fast.  But I rarely
>> copy/move articles, so I wouldn't know.  Maybe you could illustrate by
>> comparing which IMAP commands are sent now, and which commands you
>> want to be sent?
>
> I apologize.  Reading the source code, I thought that the IMAP COPY
> (imap-message-copy) was not used, but it is.  There does seem to be an
> unnecessary full-body FETCH before the COPY, though, which is what
> slows things down (I think).  I just entered the group and did a `B m'
> command, and the log showed that the article was fetched in its full
> glory before it was moved.
>
> I think this is something that Gnus does for every backend when
> moving, because it's needed in some cases (where you don't do the
> internal backend move but a fetch+store instead).  So the extra fetch
> just needs to be eliminated when moving within the same IMAP server.
>
> nnml and nnmaildir could probably benefit from the same optimization,
> if they do a file move internally as well.

I agree.  The problem is that nnimap-request-move-article doesn't know
what the article will be used for, so it doesn't know whether to
optimize away the FETCH or not.

One kludge is to make B m bind a variable, say
gnus-move-destination-group, that nnimap-request-move-article could
look at, and if set it could simply refuse to return the article data,
but perform the article move directly, to the g-m-d-g group.

Hopefully there are cleaner solutions as well...

IMAP article move optimization! (was: IMAP article move)

[Ted Zlatanov]

On Tue, 09 Nov 2004, jas@extundo.com wrote:

> I agree.  The problem is that nnimap-request-move-article doesn't know
> what the article will be used for, so it doesn't know whether to
> optimize away the FETCH or not.
> 
> One kludge is to make B m bind a variable, say
> gnus-move-destination-group, that nnimap-request-move-article could
> look at, and if set it could simply refuse to return the article data,
> but perform the article move directly, to the g-m-d-g group.
> 
> Hopefully there are cleaner solutions as well...

I added the kludge you described to gnus-sum.el and nnimap.el, it's in
CVS now.  When the source and destination servers are the same,
nnimap.el will NOT do an extra request.

Please test extensively!  This change, if buggy, could be very
annoying.  It looks OK to me and I tested it, but there's no
substitute for real-world testing under a variety of setups.

Thanks
Ted

Re: IMAP article move optimization!

[Katsumi Yamaoka]

>>>>> In <4n4qg0csx4.fsf_-_@lifelogs.com> Ted Zlatanov wrote:

> On Tue, 09 Nov 2004, jas@extundo.com wrote:

>> Hopefully there are cleaner solutions as well...

> I added the kludge you described to gnus-sum.el and nnimap.el, it's in
> CVS now.  When the source and destination servers are the same,
> nnimap.el will NOT do an extra request.

Isn't it better to use defvar to gnus-sum-hint-move-is-internal?

While compiling nnimap-request-move-article in file gnus/lisp/nnimap.el:
  ** reference to free variable gnus-sum-hint-move-is-internal

Re: IMAP article move optimization!

[Ted Zlatanov]

On Mon, 28 Feb 2005, yamaoka@jpl.org wrote:

>>>>> In <4n4qg0csx4.fsf_-_@lifelogs.com> Ted Zlatanov wrote: > On Tue, 09 Nov 2004, jas@extundo.com wrote:
> 
>>> Hopefully there are cleaner solutions as well...
> 
>> I added the kludge you described to gnus-sum.el and nnimap.el, it's in
>> CVS now.  When the source and destination servers are the same,
>> nnimap.el will NOT do an extra request.
> 
> Isn't it better to use defvar to gnus-sum-hint-move-is-internal?
> 
> While compiling nnimap-request-move-article in file gnus/lisp/nnimap.el: **
> 	reference to free variable gnus-sum-hint-move-is-internal

I wanted it to be a local variable without definition or effect
outside its scope.  It's a kludge anyhow, the real solution is to have
nnimap-request-move-article check the source and destination server.
Do you agree?

Ted

Re: IMAP article move optimization!

[Katsumi Yamaoka]

>>>>> In <4nhdjwg1nn.fsf@lifelogs.com> Ted Zlatanov wrote:

>> Isn't it better to use defvar to gnus-sum-hint-move-is-internal?

> I wanted it to be a local variable without definition or effect
> outside its scope.  It's a kludge anyhow, the real solution is to have
> nnimap-request-move-article check the source and destination server.
> Do you agree?

I see.  Sorry to have written rubbish.

Re: IMAP article move optimization!

[Ted Zlatanov]

On Tue, 01 Mar 2005, yamaoka@jpl.org wrote:

>>>>> In <4nhdjwg1nn.fsf@lifelogs.com> Ted Zlatanov wrote:
> 
>>> Isn't it better to use defvar to gnus-sum-hint-move-is-internal?
> 
>> I wanted it to be a local variable without definition or effect
>> outside its scope.  It's a kludge anyhow, the real solution is to have
>> nnimap-request-move-article check the source and destination server.
>> Do you agree?
> 
> I see.  Sorry to have written rubbish.

Your point was valid!  If it was rubbish I would have told you I
disagree :)

I am just asking you and Simon if you agree that
nnimap-request-move-article should check the servers for equality.  I
can also put that as a parameter to gnus-request-move-article and have
it be passed to ALL move commands - nnmaildir, nnml, etc.  Is that
better?  Do other backends benefit?  I would imagine nnml and nnfolder
won't, but maybe nnmaildir would benefit.

Ted

Re: IMAP article move optimization!

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On Tue, 01 Mar 2005, yamaoka@jpl.org wrote:
>
>>>>>> In <4nhdjwg1nn.fsf@lifelogs.com> Ted Zlatanov wrote:
>> 
>>>> Isn't it better to use defvar to gnus-sum-hint-move-is-internal?
>> 
>>> I wanted it to be a local variable without definition or effect
>>> outside its scope.  It's a kludge anyhow, the real solution is to have
>>> nnimap-request-move-article check the source and destination server.
>>> Do you agree?
>> 
>> I see.  Sorry to have written rubbish.
>
> Your point was valid!  If it was rubbish I would have told you I
> disagree :)
>
> I am just asking you and Simon if you agree that
> nnimap-request-move-article should check the servers for equality.  I
> can also put that as a parameter to gnus-request-move-article and have
> it be passed to ALL move commands - nnmaildir, nnml, etc.  Is that
> better?  Do other backends benefit?  I would imagine nnml and nnfolder
> won't, but maybe nnmaildir would benefit.

Would this mean changing the back end interface?  I'm not sure it is
clean enough to warrant that, but I haven't really thought about it.

I haven't really read your patches, but why would n-r-m-a have to
check server equality if g-s-h-m-i-i is non-nil?

Btw, at least internal move appear to work here, although it flashes
something about "internal move" in the echo area.  Moving several
messages appear to be possible to optimize further, though, since it
select the new and old mailbox for each flag set, and the flags could
be set finally.  But let's not think about that more until this
approach has settled down.

Perhaps I'm not of much help here, I wish I had more time to tinker
with this.  Feel free to install anything you think is good.

Re: IMAP article move optimization!

[Ted Zlatanov]

On Tue, 01 Mar 2005, jas@extundo.com wrote:

> I haven't really read your patches, but why would n-r-m-a have to
> check server equality if g-s-h-m-i-i is non-nil?

It shouldn't have to, and now it doesn't.  The backend will get a
move-is-internal parameter passed to the request-move function.  I
implemented this in all the backends as parameters, although only
nnimap uses it currently.

> Btw, at least internal move appear to work here, although it flashes
> something about "internal move" in the echo area.  Moving several
> messages appear to be possible to optimize further, though, since it
> select the new and old mailbox for each flag set, and the flags could
> be set finally.  But let's not think about that more until this
> approach has settled down.

I took the opportunity to update the marks and summary only at the end
of the move.  It seems to have produced a speed-up.  It can still be
optimized, perhaps - can you take a look?  For every article marks
update, Gnus also goes to that article, shows the thread, then goes to
that article again.  I'm not sure I understand that process.

The "move is internal/external" message is level 10, so it should only
show up for people who have gnus-verbosity turned way up.  If you
think I should change it, let me know.

Ted

Re: IMAP article move optimization!

[Katsumi Yamaoka]

>>>>> In <4nsm3fpc1a.fsf@lifelogs.com> Ted Zlatanov wrote:

>> I see.  Sorry to have written rubbish.

> Your point was valid!  If it was rubbish I would have told you I
> disagree :)

Please never mind.  I'm not a user of nnimap and am not familiar
with it.  It is that the warning only attracted my attention. ;-)

Re: IMAP article move

[David Abrahams]

Simon Josefsson wrote:
> "Ted Zlatanov" <tzz@lifelogs.com> writes:
> 
>> Can we optimize IMAP article moves so they use the server
>> capabilities?  Right now, I think, we just copy and then delete the
>> old article.  Across various backends that makes sense, but if it's
>> within the same IMAP backend, the server move can be much faster.
>>
>> I looked at gnus-sum.el, gnus-int.el, and nnimap.el.  It seems
>> possible, although the changes would have to go all the way to the
>> top, because gnus-summary-move-article does the copy & delete action.
>>
>> If this works, Gnus will be much faster and more reliable in the
>> common case of article moves in IMAP.
> 
> What are you proposing?  There is some optimization in
> nnimap-request-move-article and nnimap-request-accept-article to make
> moving more efficient, and I thought it was pretty fast.  But I rarely
> copy/move articles, so I wouldn't know.  Maybe you could illustrate by
> comparing which IMAP commands are sent now, and which commands you
> want to be sent?

The thing that seems to hurt usability the most for me is that Gnus
rebuilds the summary buffer after each message is moved in a
multiple-message move.

-- 
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com

Re: IMAP article move

[Ted Zlatanov]

On Mon, 10 Jan 2005, dave@boost-consulting.com wrote:

> The thing that seems to hurt usability the most for me is that Gnus
> rebuilds the summary buffer after each message is moved in a
> multiple-message move.

I see where this happens in gnus-sum.el.  Let's give the change I just
made, which eliminates the extra IMAP FETCH, a few days to settle
down, and then I can look at optimizing this summary buffer rebuild
process.  Let me know if you notice a performance boost.

If anyone knows why this code is called for every article
move/copy/crosspost:

	(set-buffer gnus-summary-buffer)

	(gnus-summary-goto-subject article)

in gnus-sum.el, I would appreciate the explanation.  I don't know all
the subtleties of the Gnus internals.  It seems like it would be
better to either call this just once, after the end of the (while
articles...) loop, or at least every N articles.

Ted

Re: IMAP article move

[Ted Zlatanov]

On 25 Feb 2005, tzz@lifelogs.com wrote:

On Mon, 10 Jan 2005, dave@boost-consulting.com wrote:
> 
>> The thing that seems to hurt usability the most for me is that Gnus
>> rebuilds the summary buffer after each message is moved in a
>> multiple-message move.
> 
> If anyone knows why this code is called for every article
> move/copy/crosspost:
> 
> 	(set-buffer gnus-summary-buffer)
> 
> 	(gnus-summary-goto-subject article)
> 
> in gnus-sum.el, I would appreciate the explanation.  I don't know all
> the subtleties of the Gnus internals.  It seems like it would be
> better to either call this just once, after the end of the (while
> articles...) loop, or at least every N articles.

David,

I added some code that will only do the summary updates at the end of
the article moves.  Let me know if it works well for you.

Ted

Re: IMAP article move

[David Abrahams]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On 25 Feb 2005, tzz@lifelogs.com wrote:
>
> On Mon, 10 Jan 2005, dave@boost-consulting.com wrote:
>> 
>>> The thing that seems to hurt usability the most for me is that Gnus
>>> rebuilds the summary buffer after each message is moved in a
>>> multiple-message move.
>> 
>> If anyone knows why this code is called for every article
>> move/copy/crosspost:
>> 
>> 	(set-buffer gnus-summary-buffer)
>> 
>> 	(gnus-summary-goto-subject article)
>> 
>> in gnus-sum.el, I would appreciate the explanation.  I don't know all
>> the subtleties of the Gnus internals.  It seems like it would be
>> better to either call this just once, after the end of the (while
>> articles...) loop, or at least every N articles.
>
> David,
>
> I added some code that will only do the summary updates at the end of
> the article moves.  Let me know if it works well for you.

Thanks!

I'm a bit swamped now but will update my Gnus as soon as I get a
chance and let you know how it works out.

Any chance you know how fix the "IMAP connection times out and then
the next operation fails instead of re-establishing it automatically"
issue easily?

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: IMAP article move

[Ted Zlatanov]

On Thu, 03 Mar 2005, dave@boost-consulting.com wrote:

> Any chance you know how fix the "IMAP connection times out and then
> the next operation fails instead of re-establishing it automatically"
> issue easily?

I am not the nnimap.el expert, unfortunately, but if you describe the
problem in detail (try using M-x gnus-bug so I can see your settings
too) I can look at it.  If it can be replicated, it's easier to fix of
course, but with IMAP that's often hard to do.

Ted

Re: IMAP article move

[David Abrahams]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On 25 Feb 2005, tzz@lifelogs.com wrote:
>
> David,
>
> I added some code that will only do the summary updates at the end of
> the article moves.  Let me know if it works well for you.

Haven't noticed any problems yet!  Also haven't had to do any bulk
moves, so I haven't seen vast speedups.  I'm sure they're there,
though.

Thanks!

-- 
Dave Abrahams
Boost Consulting
www.boost-consulting.com

Re: IMAP article move

[Ted Zlatanov]

On Mon, 21 Mar 2005, dave@boost-consulting.com wrote:

> "Ted Zlatanov" <tzz@lifelogs.com> writes:
> 
>> On 25 Feb 2005, tzz@lifelogs.com wrote:
>>
>> David,
>>
>> I added some code that will only do the summary updates at the end of
>> the article moves.  Let me know if it works well for you.
> 
> Haven't noticed any problems yet!  Also haven't had to do any bulk
> moves, so I haven't seen vast speedups.  I'm sure they're there,
> though.

I still see a lot of summary updates here.  This may be a part of the
marks update in IMAP.  Occasionally I get "No more newsgroups" which
makes no sense - could the summary updates be moving to the next
newsgroup instead of staying on the current one?  Simon, any idea?

Ted

Re: IMAP article move

[Simon Josefsson]

"Ted Zlatanov" <tzz@lifelogs.com> writes:

> On Mon, 21 Mar 2005, dave@boost-consulting.com wrote:
>
>> "Ted Zlatanov" <tzz@lifelogs.com> writes:
>> 
>>> On 25 Feb 2005, tzz@lifelogs.com wrote:
>>>
>>> David,
>>>
>>> I added some code that will only do the summary updates at the end of
>>> the article moves.  Let me know if it works well for you.
>> 
>> Haven't noticed any problems yet!  Also haven't had to do any bulk
>> moves, so I haven't seen vast speedups.  I'm sure they're there,
>> though.
>
> I still see a lot of summary updates here.  This may be a part of the
> marks update in IMAP.  Occasionally I get "No more newsgroups" which
> makes no sense - could the summary updates be moving to the next
> newsgroup instead of staying on the current one?  Simon, any idea?

Summary updates are to be expected, I think, because of the mark
updates.  Could be optimized as well, but it might be better to
stabilize the current approach first.  (Or not.)  I don't have any
idea on the "No more newsgroups" issue though.

Re: IMAP article move

[Ted Zlatanov]

On Tue, 22 Mar 2005, jas@extundo.com wrote:

> Summary updates are to be expected, I think, because of the mark
> updates.  Could be optimized as well, but it might be better to
> stabilize the current approach first.

I am happy with the stability of the current moves.  I would be happy
to see more optimizations of the IMAP moves.  Compared to Mozilla,
Gnus is still REALLY slow moving IMAP articles.

Ted

Re: IMAP article move

[Bjørn Mork]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

Simon Josefsson <jas@extundo.com> writes:

> What are you proposing?  There is some optimization in
> nnimap-request-move-article and nnimap-request-accept-article to make
> moving more efficient, and I thought it was pretty fast.  But I rarely
> copy/move articles, so I wouldn't know.  Maybe you could illustrate by
> comparing which IMAP commands are sent now, and which commands you
> want to be sent?

On a related issue: nnimap expiry with an expiry-target on the same
IMAP server is terribly slow.  To the degree that's entirely unusable.

I just got a new laptop and installed a fresh copy of CVS Gnus, and
that's when I noticed that this hasn't been fixed.  The attached patch
has fixed the problem for me on the old laptop (and now also on the
new one).

nnimap expiry will still be unecessarily slow with other expiry-
targets, but I don't know how to fix that.



Bjørn
-- 
How can you say that you wish you were a mental problem?  


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: nnimap-fix.patch --]
[-- Type: text/x-patch, Size: 1763 bytes --]

Index: lisp/nnimap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v
retrieving revision 7.13
diff -u -u -r7.13 nnimap.el
--- lisp/nnimap.el	26 Oct 2004 17:40:05 -0000	7.13
+++ lisp/nnimap.el	11 Jan 2005 08:53:24 -0000
@@ -1452,14 +1452,23 @@
 
 (defun nnimap-expiry-target (arts group server)
   (unless (eq nnmail-expiry-target 'delete)
-    (with-temp-buffer
-      (dolist (art arts)
-	(nnimap-request-article art group server (current-buffer))
-	;; hints for optimization in `nnimap-request-accept-article'
-	(let ((nnimap-current-move-article art)
-	      (nnimap-current-move-group group)
-	      (nnimap-current-move-server server))
-	  (nnmail-expiry-target-group nnmail-expiry-target group))))
+    (if (and (not (functionp nnmail-expiry-target)) 
+	     (gnus-server-equal (gnus-group-method nnmail-expiry-target) 
+				(gnus-server-to-method 
+				 (format "nnimap:%s" server))))
+	;; moving article within same server, speed it up...
+	(and (nnimap-possibly-change-group group)
+	     (imap-message-copy (imap-range-to-message-set arts)
+				(gnus-group-short-name nnmail-expiry-target) 
+				'dontcreate nil nnimap-server-buffer))
+      (with-temp-buffer
+	(dolist (art arts)
+	  (nnimap-request-article art group server (current-buffer))
+	  ;; hints for optimization in `nnimap-request-accept-article'
+	  (let ((nnimap-current-move-article art)
+		(nnimap-current-move-group group)
+		(nnimap-current-move-server server))
+	    (nnmail-expiry-target-group nnmail-expiry-target group)))))
     ;; It is not clear if `nnmail-expiry-target' somehow cause the
     ;; current group to be changed or not, so we make sure here.
     (nnimap-possibly-change-group group server)))