nnimap: article moving

nnimap: article moving

[NAGY Andras]

Been curious how efficiently nnimap works, and made the following
test: select an nnimap group, do not view any articles, but select one
in the summary buffer.  Press B m (gnus-summary-move-article), enter a
mailbox name on the same (physical and virtual) server, press ENTER
and leave the group.

I'd expect nnimap to do a server-side move (copy uid and mark as
deleted), without downloading the article.  Unfortunately, the article
is downloaded _two times_, but interestingly, not uploaded again, so
the server-side move actually takes place, but why the download then?
Consider this as a preformance-bug-report.

Oort Gnus v0.01.


Andras


IMAP transcript:

541 SELECT "INBOX.list.humor-l"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 977537919]  
* OK [UIDNEXT 11]  
541 OK [READ-WRITE] Completed
542 UID SEARCH UNSEEN UNDELETED
* SEARCH
542 OK Completed
543 UID SEARCH SEEN
* SEARCH 5 7 8 9
543 OK Completed
544 UID SEARCH FLAGGED
* SEARCH
544 OK Completed
545 UID SEARCH ANSWERED
* SEARCH
545 OK Completed
546 UNSELECT
546 OK Completed
547 SELECT "INBOX.list.humor-l"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 977537919]  
* OK [UIDNEXT 11]  
547 OK [READ-WRITE] Completed
548 UID SEARCH UID 5:9
* SEARCH 5 7 8 9
548 OK Completed
549 UID FETCH 5 BODY.PEEK[]
* 1 FETCH (UID 5 BODY[] {2671}

[...] Text of message.

)
549 OK Completed
550 UID FETCH 5 BODY.PEEK[]
* 1 FETCH (UID 5 BODY[] {2671}

[...] Text of message.

)
550 OK Completed
551 UID COPY 5 "INBOX.archive.list.humor-l"
551 OK [COPYUID 977547453 5 52] Completed
552 UID STORE 5 +FLAGS (\Deleted)
* 1 FETCH (FLAGS (\Deleted \Seen) UID 5)
552 OK Completed
553 SELECT "INBOX.archive.list.humor-l"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  
* 4 EXISTS
* 1 RECENT
* OK [UNSEEN 2]  
* OK [UIDVALIDITY 977547453]  
* OK [UIDNEXT 53]  
553 OK [READ-WRITE] Completed
554 UID STORE 52 FLAGS (\Seen)
* 4 FETCH (FLAGS (\Recent \Seen) UID 52)
554 OK Completed
555 UID SEARCH UNSEEN UNDELETED
* SEARCH 50
555 OK Completed
556 UID SEARCH SEEN
* SEARCH 1 51 52
556 OK Completed
557 UID SEARCH FLAGGED
* SEARCH
557 OK Completed
558 UID SEARCH ANSWERED
* SEARCH
558 OK Completed
559 UNSELECT
559 OK Completed
560 EXAMINE "INBOX.archive.list.humor-l"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  
* 4 EXISTS
* 0 RECENT
* OK [UNSEEN 2]  
* OK [UIDVALIDITY 977547453]  
* OK [UIDNEXT 53]  
560 OK [READ-ONLY] Completed
561 FETCH 1,* UID
* 1 FETCH (UID 1)
* 4 FETCH (UID 52)
561 OK Completed
562 SELECT "INBOX.archive.list.humor-l"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  
* 4 EXISTS
* 0 RECENT
* OK [UNSEEN 2]  
* OK [UIDVALIDITY 977547453]  
* OK [UIDNEXT 53]  
562 OK [READ-WRITE] Completed
563 EXPUNGE
* 4 EXISTS
* 0 RECENT
563 OK Completed
564 CLOSE
564 OK Completed
565 SELECT "INBOX.list.humor-l"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]  
* 4 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 977537919]  
* OK [UIDNEXT 11]  
565 OK [READ-WRITE] Completed
566 UID SEARCH UID 5:9 NOT SINCE 25-Dec-2000
* SEARCH
566 OK Completed
567 EXPUNGE
* 1 EXPUNGE
* 3 EXISTS
* 0 RECENT
567 OK Completed
568 CLOSE
568 OK Completed

Re: nnimap: article moving

[NAGY Andras]

NAGY Andras <nagya@inf.elte.hu> writes:

> do not view any articles, but select one
> in the summary buffer.

I mean place the cursor on it, but do not press enter/space/etc, of
course.


Andras

Re: nnimap: article moving

[Simon Josefsson]

NAGY Andras <nagya@inf.elte.hu> writes:

> Been curious how efficiently nnimap works

Nnimap is painfully slow.

At some point (or rather an interval) in time, I hope to extend the
Gnus<->backend interface to make IMAP groups work much faster.  Don't
blame IMAP for nnimap speed, IMAP can be very fast.

Two hints to speed things up considerably (at least if your reading
habits are close enough to mine):

(setq gnus-keep-backlog 10
      gnus-asynchronous t)

> and made the following test: select an nnimap group, do not view any
> articles, but select one in the summary buffer.  Press B m
> (gnus-summary-move-article), enter a mailbox name on the same
> (physical and virtual) server, press ENTER and leave the group.
> 
> I'd expect nnimap to do a server-side move (copy uid and mark as
> deleted), without downloading the article.  Unfortunately, the article
> is downloaded _two times_, but interestingly, not uploaded again, so
> the server-side move actually takes place, but why the download then?

Nnimap does not know that the target is a nnimap group at the time the
article is fetched.  So it download the article and save some
variables, and if nnimap is later invoked with an instruction to
"accept" a new article, it notices the variables and simply invoke a
"move" command rather than uploading the article.

As for downloading the article twice, that's simply stupid.  imap.el
actually cache stuff so nnimap should take advantage of that.
However, perhaps `gnus-keep-backlog' work too?

Perhaps the Gnus Move command can "hint" to the backend what the
target group is.  Hm.

nnimap bottlenecks? (Was: nnimap: article moving)

[Steinar Bang]

>>>>> Simon Josefsson <sj@extundo.com>:

> NAGY Andras <nagya@inf.elte.hu> writes:
>> Been curious how efficiently nnimap works

> Nnimap is painfully slow.

> At some point (or rather an interval) in time, I hope to extend the
> Gnus<->backend interface to make IMAP groups work much faster.
> Don't blame IMAP for nnimap speed, IMAP can be very fast.

I'm curious: what are the current bottleneck(s) of nnimap?

>From messages in the minibuffer, it seems that on a `g' it waits for
the completion of the check for new articles in each group, before
skipping to the next.  Is this one of the bottlenecks?

Re: nnimap bottlenecks? (Was: nnimap: article moving)

[Simon Josefsson]

Steinar Bang <sb@metis.no> writes:

> I'm curious: what are the current bottleneck(s) of nnimap?

Practical point of view: Finding out if there are new mail (nnimap
select each group and search for highest/lowest article number).
Synching flags when entering a group (it download all flags every
time).

Design point of view: The Gnus backend interface doesn't take
advantage of all advanced features IMAP provide that make things
"feel" fast (partial group completions, partial article fetching,
incremental summary buffer download, incremental flag download,
asynchronous operations, server-side article threading, server-side
article sorting, etc).  The current practical issues are simply
consequences of this.

IMHO the speed issue should be solved by attacking the second problem.
Speed can be improved by working on the first problem, but only to a
certain extent and then you'll find yourself with a "tweaked" Gnus
backend interface noone can comprehend that does things only slightly
faster.  In the end you've slowly messed up the backend interface.
It's much better to completely mess up the backend interface from the
beginning. :-)  But also quite difficult to do right...

> From messages in the minibuffer, it seems that on a `g' it waits for
> the completion of the check for new articles in each group, before
> skipping to the next.  Is this one of the bottlenecks?

Yes.  This illustrate the dichotomy above well, we have

Practical: Improve the situation by making the polling asynchronous
somehow.

Design: Improve backend interface to accept more flexible information
than the weird information of article number of earliest / latest
article.  Such as total number of articles, number of unread articles,
a binary "there is new mail" indicator, etc.  IMAP can provide the
latter information much much faster than the previous one.

Re: nnimap bottlenecks? (Was: nnimap: article moving)

[Soeren Laursen]

With another backend interface we would also solve the counting
problem with unread articles, right?

-- 
Søren Laursen http://www.tele.auc.dk/~slau/

Re: nnimap bottlenecks? (Was: nnimap: article moving)

[Simon Josefsson]

Soeren Laursen <slau@opel.kom.auc.dk> writes:

> With another backend interface we would also solve the counting
> problem with unread articles, right?

Yes.  Well, the backend interface change is the easy part, the hard
part would be to modify Gnus to use the new interface in code that
counts unread articles (and other places).