nnimap move and UID SEARCH HEADER

nnimap move and UID SEARCH HEADER

[Michael Welsh Duggan]

So, UID SEARCH HEADER is extremely slow in the benighted piece of blight
that is Exchange, even more so on large groups.  As such, moving an
article to a large group is tantamount to locking up Gnus for 10 minutes
or so.  I don't remember why gnus needs the article number within the
destination group, but if an option could be added to turn off this
search, I would be very grateful.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: nnimap move and UID SEARCH HEADER

[Lars Magne Ingebrigtsen]

Michael Welsh Duggan <md5i@md5i.com> writes:

> I don't remember why gnus needs the article number within the
> destination group, but if an option could be added to turn off this
> search, I would be very grateful.

Gnus needs the article number to do Agent/registry/cache stuff.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap move and UID SEARCH HEADER

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Michael Welsh Duggan <md5i@md5i.com> writes:
>
>> I don't remember why gnus needs the article number within the
>> destination group, but if an option could be added to turn off this
>> search, I would be very grateful.
>
> Gnus needs the article number to do Agent/registry/cache stuff.

This reminds me of a question I had:  if I understand correctly, Gnus
does a search on Message-ID to figure out the UID an article gets after
moving.  But what if the target group already has an article with that
Message-ID?  E.g. maybe the article was copied there already?

Also, when editing an article, do we request the UID before or after
expunging the old version?  If it's before, we might get the wrong
UID.

The way offlineimap handles this is by inserting a unique header
when adding a message to an IMAP folder, and then searching for 
that header.  This would be easy for Gnus to do, but it would
eliminate the optimization of doing the copy server-side when
that is possible.

Dan

Re: nnimap move and UID SEARCH HEADER

[Ted Zlatanov]

On Sun, 21 Nov 2010 06:16:34 +0100 Lars Magne Ingebrigtsen <larsi@gnus.org> wrote: 

LMI> Michael Welsh Duggan <md5i@md5i.com> writes:
>> I don't remember why gnus needs the article number within the
>> destination group, but if an option could be added to turn off this
>> search, I would be very grateful.

LMI> Gnus needs the article number to do Agent/registry/cache stuff.

At least for the registry we could work around that if it improved Gnus
speed significantly.  The registry only really needs the message ID IIRC.

Ted

Re: nnimap move and UID SEARCH HEADER

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> This reminds me of a question I had:  if I understand correctly, Gnus
> does a search on Message-ID to figure out the UID an article gets after
> moving.  But what if the target group already has an article with that
> Message-ID?  E.g. maybe the article was copied there already?

That's a good question.  I don't have an answer.  :-)

> Also, when editing an article, do we request the UID before or after
> expunging the old version?  If it's before, we might get the wrong
> UID.

Hm...  true...

> The way offlineimap handles this is by inserting a unique header
> when adding a message to an IMAP folder, and then searching for 
> that header.  This would be easy for Gnus to do, but it would
> eliminate the optimization of doing the copy server-side when
> that is possible.

Yup.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap move and UID SEARCH HEADER

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> This reminds me of a question I had:  if I understand correctly, Gnus
>> does a search on Message-ID to figure out the UID an article gets after
>> moving.  But what if the target group already has an article with that
>> Message-ID?  E.g. maybe the article was copied there already?
>
> That's a good question.  I don't have an answer.  :-)

In the one case I tested, dovecot returned the right UID.  Maybe it
defaults to the highest UID when there are multiple matches?

>> Also, when editing an article, do we request the UID before or after
>> expunging the old version?  If it's before, we might get the wrong
>> UID.
>
> Hm...  true...

In my one test, Gnus expunges before doing the search, so at least
this won't be a problem.

I also noticed the following:  dovecot responds to the COPY command with
the new UID!

3 UID COPY 10722 "test"
* 139 EXISTS
* 1 RECENT
3 OK [COPYUID 1254248544 10722 10731] Copy completed.

The new UID is 10731.

This is part of UIDPLUS:

  http://www.faqs.org/rfcs/rfc4315.html

The long number is the UIDVALIDITY, then the list of source UIDs, then
a list of destination UIDs.

Gnus could just look for "COPYUID" in the output, and only do the search
when it's not present.  This would be faster and more reliable, I think,
especially when moving or copying lots of articles.

Dan

Re: nnimap move and UID SEARCH HEADER

[Michael Welsh Duggan]

Dan Christensen <jdc@uwo.ca> writes:

> I also noticed the following:  dovecot responds to the COPY command with
> the new UID!
>
> 3 UID COPY 10722 "test"
> * 139 EXISTS
> * 1 RECENT
> 3 OK [COPYUID 1254248544 10722 10731] Copy completed.
>
> The new UID is 10731.
>
> This is part of UIDPLUS:
>
>   http://www.faqs.org/rfcs/rfc4315.html
>
> The long number is the UIDVALIDITY, then the list of source UIDs, then
> a list of destination UIDs.
>
> Gnus could just look for "COPYUID" in the output, and only do the search
> when it's not present.  This would be faster and more reliable, I think,
> especially when moving or copying lots of articles.

I can verify that on Cyrus as well.

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: nnimap move and UID SEARCH HEADER

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> I also noticed the following:  dovecot responds to the COPY command with
> the new UID!
>
> 3 UID COPY 10722 "test"
> * 139 EXISTS
> * 1 RECENT
> 3 OK [COPYUID 1254248544 10722 10731] Copy completed.

Wow!  That solves all the problems for servers that support this.  I'll
implement it now.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap move and UID SEARCH HEADER

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Wow!  That solves all the problems for servers that support this.  I'll
> implement it now.

It's now implemented.  Doesn't look like Gmail implements UIDPLUS,
unfortunately. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap move and UID SEARCH HEADER

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Wow!  That solves all the problems for servers that support this.  I'll
>> implement it now.
>
> It's now implemented.  Doesn't look like Gmail implements UIDPLUS,
> unfortunately. 

Looks like the wrong number is being used.  See command 32 below.  
It uses the old article number 2369, but the new one is 2370.

20:15:36 27 UID STORE 2369 -FLAGS.SILENT (\Seen \Flagged \Answered gnus-expire gnus-dormant gnus-save gnus-download gnus-forward)
20:15:36 28 UID FETCH 2369 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
20:15:36 29 UID COPY 2369 "test"
20:15:36 30 UID STORE 2369 +FLAGS.SILENT (\Deleted)
20:15:36 31 UID EXPUNGE 2369
20:15:36 32 UID STORE 2369 +FLAGS.SILENT (\Seen)
20:15:36 33 SELECT "test"
20:15:36 34 UID FETCH 1:* FLAGS

Dan

Re: nnimap move and UID SEARCH HEADER

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Looks like the wrong number is being used.  See command 32 below.  
> It uses the old article number 2369, but the new one is 2370.

Ah, the last element in the list.  Fixed and pushed.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap move and UID SEARCH HEADER

[Dan Christensen]

Thanks.  Now the right article is found.  But for some reason, when
I reload the group with `9 M-g', an expirable mark I set is lost.
See command 95 (where the expire flag is set on the original) and 
command 104 (where it is removed from the copy).

[And, as mentioned in another thread, there seem to be some unnecessary
commands sent when `B m' is done: 96, 101; 97?; 102 & 103?]

Dan

`B m'

20:31:06 95 UID STORE 2374 +FLAGS.SILENT (gnus-expire)
20:31:06 96 UID STORE 2374 -FLAGS.SILENT (\Seen \Flagged \Answered gnus-dormant gnus-save gnus-download gnus-forward)
20:31:06 97 UID FETCH 2374 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])
20:31:06 98 UID COPY 2374 "test"
20:31:06 99 UID STORE 2374 +FLAGS.SILENT (\Deleted)
20:31:06 100 UID EXPUNGE 2374
20:31:06 101 UID STORE 2375 +FLAGS.SILENT (\Seen)
20:31:06 102 SELECT "test"
20:31:06 103 UID FETCH 1:* FLAGS

`9 M-g'

20:31:14 104 UID STORE 2375 -FLAGS.SILENT (gnus-expire)
20:31:14 105 SELECT "test"
20:31:14 106 UID FETCH 1:* FLAGS
20:31:14 107 UID FETCH 2367:2375 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])

Re: nnimap move and UID SEARCH HEADER

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Thanks.  Now the right article is found.  But for some reason, when
> I reload the group with `9 M-g', an expirable mark I set is lost.

Is that in the group or the summary buffer?

> [And, as mentioned in another thread, there seem to be some unnecessary
> commands sent when `B m' is done: 96, 101; 97?; 102 & 103?]

Well, yes, some of them are unnecessary for IMAP, but necessary in
general in Gnus.

> 20:31:06 95 UID STORE 2374 +FLAGS.SILENT (gnus-expire)
> 20:31:06 96 UID STORE 2374 -FLAGS.SILENT (\Seen \Flagged \Answered
> gnus-dormant gnus-save gnus-download gnus-forward)

These two are the only way to sync flags totally from Gnus.  That is,
set the ones we know are set, and remove all the flags that we know
don't exist.  But I could extend -request-set-mark with a 'set in
addition to 'add and 'del, which would just use a FLAGS instead of
-/+FLAGS.

I've now done that for all the backends except nnmaildir.  Could someone
fix the marks code in nnmaildir to understand 'set?

> 20:31:06 97 UID FETCH 2374 (UID RFC822.SIZE BODYSTRUCTURE
> BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References
> In-Reply-To Xref X-Spam-Status To Cc Keywords Newsgroups Subject)])

Needed for other backends...

> 20:31:06 98 UID COPY 2374 "test"
> 20:31:06 99 UID STORE 2374 +FLAGS.SILENT (\Deleted)
> 20:31:06 100 UID EXPUNGE 2374

> 20:31:06 101 UID STORE 2375 +FLAGS.SILENT (\Seen)

Ditto.

> 20:31:06 102 SELECT "test"
> 20:31:06 103 UID FETCH 1:* FLAGS

I'm not sure what's that about.

> `9 M-g'
>
> 20:31:14 104 UID STORE 2375 -FLAGS.SILENT (gnus-expire)

Hm...  there's special-casing in the general sync-all-flags code to
exclude gnus-expire, so it's deleted again.  Can anybody remember why?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap move and UID SEARCH HEADER

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I've now done that for all the backends except nnmaildir.  Could someone
> fix the marks code in nnmaildir to understand 'set?

Oh, it already understands this.  Then I'll adjust the move thing to use
'set, and that'll save one command.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap move and UID SEARCH HEADER

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Oh, it already understands this.  Then I'll adjust the move thing to use
> 'set, and that'll save one command.

And after fixing a +/- mixup, it now seems to work.

Of course, it still does all that other superfluous stuff, but I still
haven't decided how to approach it, and to what degree the IMAP way of
doing this should deviate from how other backends do it.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: nnimap move and UID SEARCH HEADER

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Thanks.  Now the right article is found.  But for some reason, when
>> I reload the group with `9 M-g', an expirable mark I set is lost.
>
> Is that in the group or the summary buffer?

Summary.

>> 20:31:06 95 UID STORE 2374 +FLAGS.SILENT (gnus-expire)
>> 20:31:06 96 UID STORE 2374 -FLAGS.SILENT (\Seen \Flagged \Answered
>> gnus-dormant gnus-save gnus-download gnus-forward)
>
> These two are the only way to sync flags totally from Gnus.  That is,
> set the ones we know are set, and remove all the flags that we know
> don't exist.  

But couldn't Gnus compare what the backend marks are to what they should
be and just set what needs to be changed?  

On the other hand, maybe it's good to have some redundancy here,
especially now that you've reduced it to one command.

> Of course, it still does all that other superfluous stuff, but I still
> haven't decided how to approach it, and to what degree the IMAP way of
> doing this should deviate from how other backends do it.

Yeah, it's not clear.  Things are even more inefficient when you move or
copy multiple messages, as I think IMAP can do this with one command.

> Hm...  there's special-casing in the general sync-all-flags code to
> exclude gnus-expire, so it's deleted again.  Can anybody remember why?

I don't remember why.  When does that special-casing kick in?  Is it
just for articles that are moved/copied?  I vaguely remember that when
moving/copying an article, the expire flag was removed because you must
want to keep the article in that case.

Dan

IMAP flag setting

[Dan Christensen]

I mentioned before how when I move an article from an IMAP group, Gnus
now briefly clears the \Seen flag before moving the article.  This is
slightly annoying because my biff program pops up a message and beeps
whenever this happens.  But I just realized that it is more problematic
than that, because my cell phone monitors my INBOX using IMAP idle, and
so it also does an extra refresh whenever this happens, wasting
bandwidth and draining the battery.  Can we avoid clearing the \Seen
flag?  It doesn't seem to make any sense to clear it and then set it
again on the article after it is moved.

Dan

Re: nnimap move and UID SEARCH HEADER

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> But couldn't Gnus compare what the backend marks are to what they should
> be and just set what needs to be changed?  
>
> On the other hand, maybe it's good to have some redundancy here,
> especially now that you've reduced it to one command.

Yeah, I don't think it hurts... 

>> Hm...  there's special-casing in the general sync-all-flags code to
>> exclude gnus-expire, so it's deleted again.  Can anybody remember why?
>
> I don't remember why.  When does that special-casing kick in?  Is it
> just for articles that are moved/copied?  I vaguely remember that when
> moving/copying an article, the expire flag was removed because you must
> want to keep the article in that case.

Yeah, that sounds reasonable.  This is the code in the move-article
function:

	    (let* ((expirable (gnus-group-auto-expirable-p to-group))
		   (marks (if expirable
			      gnus-article-mark-lists
			    (delete '(expirable . expire)
				    (copy-sequence
				     gnus-article-mark-lists))))

So if the group we're moving to isn't auto-expirable, we ignore the
expirableness.  I don't know whether that really makes sense or not.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: IMAP flag setting

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> I mentioned before how when I move an article from an IMAP group, Gnus
> now briefly clears the \Seen flag before moving the article. 

It does?  I thought I fixed that.  When I do a move now, these are the
commands executed:

13:58:13 55268 UID FETCH 55 (UID RFC822.SIZE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref To Newsgroups)]).
13:58:13 55269 UID COPY 55 "ebay".
13:58:13 55270 UID STORE 55 +FLAGS.SILENT (\Deleted).
13:58:13 55271 EXPUNGE.
13:58:13 55272 EXAMINE "ebay".
13:58:13 55273 UID SEARCH HEADER Message-Id "<m3hbf4zvj4.fsf@quimbies.gnus.org>".
13:58:13 55274 SELECT "ebay".
13:58:13 55275 UID STORE 12 +FLAGS.SILENT (\Seen).

No clearing of the \Seen mark...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: IMAP flag setting

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> I mentioned before how when I move an article from an IMAP group, Gnus
>> now briefly clears the \Seen flag before moving the article. 
>
> It does?  I thought I fixed that.

Sorry, my fault.  I hadn't realize you made relevant commits, and hadn't
pulled them.  Now it looks very streamlined!  Thanks!

Dan