Faster nnimap: nnimap-retrieve-groups-asynchronous

Faster nnimap: nnimap-retrieve-groups-asynchronous

[Simon Josefsson]

I tried to speed up new mail checking (`g') on IMAP groups.  Startup
uses old technique, but subsequent `g' can be faster.  The speedup
depends on how many groups usually receives new mail each time you
press `g' (the p below).  It might depend on the server as well.  For
me, it is faster against Cyrus IMAPD -- results from other servers
would be interesting.

nnimap-retrieve-groups-asynchronous's value is t

Documentation:
Send asynchronous STATUS commands for each mailbox before checking mail.
If you have mailboxes that rarely receives mail, this speeds up new
mail checking.  It works by first sending STATUS commands for each
mailbox, and then only checking groups which has a modified UIDNEXT
more carefully for new mail.

In summary, the default is O((1-p)*k+p*n) and changing it to nil makes
it O(n).  If p is small, then the default is probably faster.

You can customize this variable.

Defined in `nnimap'.

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Nicolas Kowalski]

Simon Josefsson <jas@extundo.com> writes:


[...]

> me, it is faster against Cyrus IMAPD -- results from other servers
> would be interesting.

With UW IMAP server, this works faster also :-)

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[dme]

* jas@extundo.com [2002-04-29 17:41:18]
> For me, it is faster against Cyrus IMAPD -- results from other
> servers would be interesting.

It is significantly faster when used over a relatively high latency
link (150ms rtt) to the iPlanet Messaging Server 5.2.

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Wes Hardaker]

>>>>> On Mon, 29 Apr 2002 18:41:18 +0200, Simon Josefsson <jas@extundo.com> said:

Simon> I tried to speed up new mail checking (`g') on IMAP groups.

Excellent!

FYI, I get about a 2x improvement from a downtown coffee shop to my
house (through 2 dsl modems) using a UW server.
-- 
"The trouble with having an open mind, of course, is that people will
 insist on coming along and trying to put things in it."   -- Terry Pratchett

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Danny Siu]

'g' is now ~150% faster than before for my 60+ nnimap groups on UoW imapd.

cool stuff!

Simon Josefsson writes:

  Simon> I tried to speed up new mail checking (`g') on IMAP groups.
  Simon> Startup uses old technique, but subsequent `g' can be faster.  The
  Simon> speedup depends on how many groups usually receives new mail each
  Simon> time you press `g' (the p below).  It might depend on the server as
  Simon> well.  For me, it is faster against Cyrus IMAPD -- results from
  Simon> other servers would be interesting.

  Simon> nnimap-retrieve-groups-asynchronous's value is t

  Simon> Documentation: Send asynchronous STATUS commands for each mailbox
  Simon> before checking mail.  If you have mailboxes that rarely receives
  Simon> mail, this speeds up new mail checking.  It works by first sending
  Simon> STATUS commands for each mailbox, and then only checking groups
  Simon> which has a modified UIDNEXT more carefully for new mail.

  Simon> In summary, the default is O((1-p)*k+p*n) and changing it to nil
  Simon> makes it O(n).  If p is small, then the default is probably faster.

  Simon> You can customize this variable.

  Simon> Defined in `nnimap'.



-- 
Danny Siu

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Ilpo Nyyssönen]

Fresh Oort Gnus from CVS gives me this with my Courier IMAP:

Signaling: (wrong-type-argument number-char-or-marker-p nil)
  nnimap-retrieve-groups(("inbox.Trash" "inbox.news" "inbox.sent" "inbox") "ypy")
  gnus-retrieve-groups(("inbox.Trash" "inbox.news" "inbox.sent" "inbox") (nnimap "ypy" (nnimap-address "localhost") (nnimap-stream ssl) (nnimap-authenticator login) (nnimap-list-pattern ("inbox.*" "shared.*"))))
  gnus-read-active-file-2(("inbox.Trash" "inbox.news" "inbox.sent" "inbox") (nnimap "ypy" (nnimap-address "localhost") (nnimap-stream ssl) (nnimap-authenticator login) (nnimap-list-pattern ("inbox.*" "shared.*"))))
  gnus-read-active-file-1((nnimap "ypy" (nnimap-address "localhost") (nnimap-stream ssl) (nnimap-authenticator login) (nnimap-list-pattern ("inbox.*" "shared.*"))) nil)
  gnus-read-active-file()
  gnus-group-get-new-news(nil)
  call-interactively(gnus-group-get-new-news)

-- 
Ilpo Nyyssönen # biny # /* :-) */

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Simon Josefsson]

On Tue, 30 Apr 2002, Ilpo Nyyssönen wrote:

> Fresh Oort Gnus from CVS gives me this with my Courier IMAP:
> 
> Signaling: (wrong-type-argument number-char-or-marker-p nil)
>   nnimap-retrieve-groups(("inbox.Trash" "inbox.news" "inbox.sent" "inbox") "ypy")

Try evaluating (load "nnimap.el") to get a better backtrace.

Anyway, it seems as if Courier has some problems, probably caused by them 
using too large UIDNEXT values (larger than emacs lisp integer limit).  
Please try patch below.  Does anyone know if Elisp already has a 
string<-numerical?  My implementation is probably not the fastest.

Index: imap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/imap.el,v
retrieving revision 6.37
diff -u -p -r6.37 imap.el
--- imap.el	2002/04/29 16:42:29	6.37
+++ imap.el	2002/04/29 20:03:42
@@ -2128,8 +2128,8 @@ Return nil if no complete line has arriv
     (imap-forward)
     (cond ((search-forward "PERMANENTFLAGS " nil t)
 	   (imap-mailbox-put 'permanentflags (imap-parse-flag-list)))
-	  ((search-forward "UIDNEXT " nil t)
-	   (imap-mailbox-put 'uidnext (read (current-buffer))))
+	  ((search-forward "UIDNEXT \\([0-9]+\\)" nil t)
+	   (imap-mailbox-put 'uidnext (match-string 1)))
 	  ((search-forward "UNSEEN " nil t)
 	   (imap-mailbox-put 'unseen (read (current-buffer))))
 	  ((looking-at "UIDVALIDITY \\([0-9]+\\)")
@@ -2302,7 +2302,9 @@ Return nil if no complete line has arriv
 		((eq token 'RECENT)
 		 (imap-mailbox-put 'recent (read (current-buffer)) mailbox))
 		((eq token 'UIDNEXT)
-		 (imap-mailbox-put 'uidnext (read (current-buffer)) mailbox))
+		 (and (looking-at " \\([0-9]+\\)")
+		      (imap-mailbox-put 'uidnext (match-string 1) mailbox)
+		      (goto-char (match-end 1))))
 		((eq token 'UIDVALIDITY)
 		 (and (looking-at " \\([0-9]+\\)")
 		      (imap-mailbox-put 'uidvalidity (match-string 1) mailbox)
Index: nnimap.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/nnimap.el,v
retrieving revision 6.30
diff -u -p -r6.30 nnimap.el
--- nnimap.el	2002/04/29 19:28:02	6.30
+++ nnimap.el	2002/04/29 20:03:42
@@ -938,6 +938,23 @@ function is generally only called when G
 
 ;; Optional backend functions
 
+(defun nnimap-string-lessp-numerical (s1 s2)
+  "Return t if first arg string is less than second in numerical order."
+  (cond ((string= s1 s2)
+	 nil)
+	((> (length s1) (length s2))
+	 nil)
+	((< (length s1) (length s2))
+	 t)
+	((< (string-to-number (substring s1 0 1))
+	    (string-to-number (substring s2 0 1)))
+	 t)
+	((> (string-to-number (substring s1 0 1))
+	    (string-to-number (substring s2 0 1)))
+	 nil)
+	(t
+	 (nnimap-string-lessp-numerical (substring s1 1) (substring s2 1)))))
+
 (deffoo nnimap-retrieve-groups (groups &optional server)
   (when (nnimap-possibly-change-server server)
     (gnus-message 5 "nnimap: Checking mailboxes...")
@@ -959,8 +976,9 @@ function is generally only called when G
 		  (tag   (nth 1 asyncgroup))
 		  new old)
 	      (when (imap-ok-p (imap-wait-for-tag tag nnimap-server-buffer))
-		(if (< (car (gnus-gethash group nnimap-mailbox-info))
-		       (imap-mailbox-get 'uidnext group nnimap-server-buffer))
+		(if (nnimap-string-lessp-numerical
+		     (car (gnus-gethash group nnimap-mailbox-info))
+		     (imap-mailbox-get 'uidnext group nnimap-server-buffer))
 		    (push (list group) slowgroups)
 		  (insert (cdr (gnus-gethash group nnimap-mailbox-info))))))))
 	(dolist (group slowgroups)

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Ilpo Nyyssönen]

Simon Josefsson <jas@extundo.com> writes:

> On Tue, 30 Apr 2002, Ilpo Nyyssönen wrote:
>
>> Fresh Oort Gnus from CVS gives me this with my Courier IMAP:
>> 
>> Signaling: (wrong-type-argument number-char-or-marker-p nil)
>>   nnimap-retrieve-groups(("inbox.Trash" "inbox.news" "inbox.sent" "inbox") "ypy")
>
> Try evaluating (load "nnimap.el") to get a better backtrace.

I'll try that patch little later.

Signaling: (wrong-type-argument number-char-or-marker-p nil)
  <(nil nil)
  (if (< (car ...) (imap-mailbox-get ... group nnimap-server-buffer)) (push (list group) slowgroups) (insert (cdr ...)))
  (if (imap-ok-p (imap-wait-for-tag tag nnimap-server-buffer)) (if (< ... ...) (push ... slowgroups) (insert ...)))
  (when (imap-ok-p (imap-wait-for-tag tag nnimap-server-buffer)) (if (< ... ...) (push ... slowgroups) (insert ...)))
  (let ((group ...) (tag ...) new old) (when (imap-ok-p ...) (if ... ... ...)))
  (while --dolist-temp--20204 (setq asyncgroup (car --dolist-temp--20204)) (let (... ... new old) (when ... ...)) (setq --dolist-temp--20204 (cdr --dolist-temp--20204)))
  (let ((--dolist-temp--20204 asyncgroups) asyncgroup) (while --dolist-temp--20204 (setq asyncgroup ...) (let ... ...) (setq --dolist-temp--20204 ...)) nil)
  (catch (quote --cl-block-nil--) (let (... asyncgroup) (while --dolist-temp--20204 ... ... ...) nil))
  (cl-block-wrapper (catch (quote --cl-block-nil--) (let ... ... nil)))
  (block nil (let (... asyncgroup) (while --dolist-temp--20204 ... ... ...) nil))
  (dolist (asyncgroup asyncgroups) (let (... ... new old) (when ... ...)))
  (if (null nnimap-retrieve-groups-asynchronous) (setq slowgroups groups) (dolist (group groups) (gnus-message 7 "nnimap: Checking mailbox %s" group) (add-to-list ... ...)) (dolist (asyncgroup asyncgroups) (let ... ...)))
  (let (asyncgroups slowgroups) (if (null nnimap-retrieve-groups-asynchronous) (setq slowgroups groups) (dolist ... ... ...) (dolist ... ...)) (dolist (group slowgroups) (if nnimap-retrieve-groups-asynchronous ...) (gnus-message 7 "nnimap: Rechecking mailbox %s" group) (imap-mailbox-put ... nil group nnimap-server-buffer) (or ... ...)))
  (save-current-buffer (set-buffer nntp-server-buffer) (erase-buffer) (nnimap-before-find-minmax-bugworkaround) (let (asyncgroups slowgroups) (if ... ... ... ...) (dolist ... ... ... ... ...)))
  (with-current-buffer nntp-server-buffer (erase-buffer) (nnimap-before-find-minmax-bugworkaround) (let (asyncgroups slowgroups) (if ... ... ... ...) (dolist ... ... ... ... ...)))
  (progn (gnus-message 5 "nnimap: Checking mailboxes...") (with-current-buffer nntp-server-buffer (erase-buffer) (nnimap-before-find-minmax-bugworkaround) (let ... ... ...)) (gnus-message 5 "nnimap: Checking mailboxes...done") (quote active))
  (if (nnimap-possibly-change-server server) (progn (gnus-message 5 "nnimap: Checking mailboxes...") (with-current-buffer nntp-server-buffer ... ... ...) (gnus-message 5 "nnimap: Checking mailboxes...done") (quote active)))
  (when (nnimap-possibly-change-server server) (gnus-message 5 "nnimap: Checking mailboxes...") (with-current-buffer nntp-server-buffer (erase-buffer) (nnimap-before-find-minmax-bugworkaround) (let ... ... ...)) (gnus-message 5 "nnimap: Checking mailboxes...done") (quote active))
  nnimap-retrieve-groups(("inbox.Trash" "inbox.news" "inbox.sent" "inbox") "ypy")
  gnus-retrieve-groups(("inbox.Trash" "inbox.news" "inbox.sent" "inbox") (nnimap "ypy" (nnimap-address "localhost") (nnimap-stream ssl) (nnimap-authenticator login) (nnimap-list-pattern ("inbox.*" "shared.*"))))
  gnus-read-active-file-2(("inbox.Trash" "inbox.news" "inbox.sent" "inbox") (nnimap "ypy" (nnimap-address "localhost") (nnimap-stream ssl) (nnimap-authenticator login) (nnimap-list-pattern ("inbox.*" "shared.*"))))
  gnus-read-active-file-1((nnimap "ypy" (nnimap-address "localhost") (nnimap-stream ssl) (nnimap-authenticator login) (nnimap-list-pattern ("inbox.*" "shared.*"))) nil)
  gnus-read-active-file()
  gnus-group-get-new-news(nil)
  call-interactively(gnus-group-get-new-news)

-- 
Ilpo Nyyssönen # biny # /* :-) */

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Ilpo Nyyssönen]

Simon Josefsson <jas@extundo.com> writes:

> Anyway, it seems as if Courier has some problems, probably caused by them 
> using too large UIDNEXT values (larger than emacs lisp integer limit).  
> Please try patch below.  Does anyone know if Elisp already has a 
> string<-numerical?  My implementation is probably not the fastest.

Yes, that patch seems to fix it.

-- 
Ilpo Nyyssönen # biny # /* :-) */

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Simon Josefsson]

iny+news@iki.fi (Ilpo Nyyssönen) writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Anyway, it seems as if Courier has some problems, probably caused by them 
>> using too large UIDNEXT values (larger than emacs lisp integer limit).  
>> Please try patch below.  Does anyone know if Elisp already has a 
>> string<-numerical?  My implementation is probably not the fastest.
>
> Yes, that patch seems to fix it.

Added to CVS, thanks.

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Kai Großjohann]

Simon Josefsson <jas@extundo.com> writes:

> I tried to speed up new mail checking (`g') on IMAP groups.

You succeeded, at least on my machine.  (Talking to a Cyrus server.)
Great!

kai
-- 
Silence is foo!

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Amos Gouaux]

>>>>> On Tue, 30 Apr 2002 13:06:49 +0200,
>>>>> Kai Großjohann <Kai.Grossjohann@CS.Uni-Dortmund.DE> (kg) writes:

kg> You succeeded, at least on my machine.  (Talking to a Cyrus server.)
kg> Great!

Yeah he has.  This is great!

-- 
Amos

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Bjørn Mork]

Simon Josefsson <jas@extundo.com> writes:

> I tried to speed up new mail checking (`g') on IMAP groups.  Startup
> uses old technique, but subsequent `g' can be faster.  The speedup
> depends on how many groups usually receives new mail each time you
> press `g' (the p below).  It might depend on the server as well.  For
> me, it is faster against Cyrus IMAPD -- results from other servers
> would be interesting.

This speeded up mail checking noticably on all my imap servers, but
unfortunately it's causing a problem with one of them. I could of
course turn it off, but then I lose the performance gain... So I'm
hoping the problem can be fixed.

I currently use three imap servers:
a) IMAP4 server (InterMail vM.4.01.03.23 201-229-121-123-20010418)
b) Courier-IMAP ready. Copyright 1998-2001 Double Precision, Inc.
c) Courier-IMAP ready. Copyright 1998-2002 Double Precision, Inc.

I can't do server side splitting on a) and b), so I'm using a
nnimap-split-rule on them. c) is doing server side splitting using
procmail. 

The problem is that Gnus doesn't detect new mail split into a folder
on b) when pressing 'g'. Doing a 'M-g' on the folder will however show
the new mail. The funny thing is that everything works fine on server
a) and c). So it looks like a combination of a Courier server and
client side splitting is causing the problem, unless there is a
specific problem with the Courier version running on b).

This is a small extract of the imap-log on server b):

91 EXAMINE "INBOX"
* FLAGS (\Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS ()] No permanent flags permitted
* 131 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 979914089] Ok
91 OK [READ-ONLY] Ok
[..]
95 EXAMINE "INBOX.Trash"
* FLAGS (\Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS ()] No permanent flags permitted
* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 979914089] Ok
95 OK [READ-ONLY] Ok
[..]
128 STATUS "INBOX.Trash" (uidnext)
* STATUS "INBOX.Trash" (UIDNEXT 1)
128 OK STATUS Completed.
[..]
170 UID FETCH 370 BODY.PEEK[HEADER]
* 131 FETCH (UID 370 BODY[HEADER] {1628}
[snip mail]
170 OK FETCH completed.
171 UID COPY 370 "INBOX.Trash"
171 OK COPY completed.
172 UID STORE 370 +FLAGS (\Seen \Deleted)
* 131 FETCH (FLAGS (\Seen \Deleted))
172 OK STORE completed.
173 EXPUNGE
* 131 EXPUNGE
* 130 EXISTS
* 0 RECENT
173 OK EXPUNGE completed
174 CLOSE
174 OK mailbox closed.
175 STATUS "INBOX.Trash" (uidnext)
* STATUS "INBOX.Trash" (UIDNEXT 1)
175 OK STATUS Completed.


Hmm, definitely looks like a server problem when I'm looking at it
now. Shouldn't the UIDNEXT value change between command 128 and 175? 
Is this a known problem with the Courier server?

Maybe I should do some experiments with client side splitting on my c) 
server, too.


Bjørn
-- 
So, your ten-incher is great?  

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Simon Josefsson]

Bjørn Mork <bmork@dod.no> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> I tried to speed up new mail checking (`g') on IMAP groups.  Startup
>> uses old technique, but subsequent `g' can be faster.  The speedup
>> depends on how many groups usually receives new mail each time you
>> press `g' (the p below).  It might depend on the server as well.  For
>> me, it is faster against Cyrus IMAPD -- results from other servers
>> would be interesting.
>
> This speeded up mail checking noticably on all my imap servers, but
> unfortunately it's causing a problem with one of them. I could of
> course turn it off, but then I lose the performance gain... So I'm
> hoping the problem can be fixed.
>
> I currently use three imap servers:
> a) IMAP4 server (InterMail vM.4.01.03.23 201-229-121-123-20010418)
> b) Courier-IMAP ready. Copyright 1998-2001 Double Precision, Inc.
> c) Courier-IMAP ready. Copyright 1998-2002 Double Precision, Inc.

I suspect the same problem as Ilpo reported, also Courier IMAPD from
what I recall.  I haven't debug that yet though.

> The problem is that Gnus doesn't detect new mail split into a folder
> on b) when pressing 'g'. Doing a 'M-g' on the folder will however show
> the new mail. The funny thing is that everything works fine on server
> a) and c). So it looks like a combination of a Courier server and
> client side splitting is causing the problem, unless there is a
> specific problem with the Courier version running on b).

Can you find out the server versions used on server a and b?

Ilpo was not using client side splitting from what I recall.

> 95 EXAMINE "INBOX.Trash"
> * FLAGS (\Answered \Flagged \Deleted \Seen \Recent)
> * OK [PERMANENTFLAGS ()] No permanent flags permitted
> * 0 EXISTS
> * 0 RECENT
> * OK [UIDVALIDITY 979914089] Ok
> 95 OK [READ-ONLY] Ok
> [..]
> 128 STATUS "INBOX.Trash" (uidnext)
> * STATUS "INBOX.Trash" (UIDNEXT 1)
> 128 OK STATUS Completed.
> [..]
> 170 UID FETCH 370 BODY.PEEK[HEADER]
> * 131 FETCH (UID 370 BODY[HEADER] {1628}
> [snip mail]
> 170 OK FETCH completed.
> 171 UID COPY 370 "INBOX.Trash"
> 171 OK COPY completed.
> 172 UID STORE 370 +FLAGS (\Seen \Deleted)
> * 131 FETCH (FLAGS (\Seen \Deleted))
> 172 OK STORE completed.
> 173 EXPUNGE
> * 131 EXPUNGE
> * 130 EXISTS
> * 0 RECENT
> 173 OK EXPUNGE completed
> 174 CLOSE
> 174 OK mailbox closed.
> 175 STATUS "INBOX.Trash" (uidnext)
> * STATUS "INBOX.Trash" (UIDNEXT 1)
> 175 OK STATUS Completed.
>
> Hmm, definitely looks like a server problem when I'm looking at it
> now. Shouldn't the UIDNEXT value change between command 128 and 175? 

Hm.  At first glance, it looks like it, but it could be valid
behaviour if UIDVALIDITY changed. Could you try running with this
patch?  Does the UIDVALIDITY change between 128- and 175-like commands
then?  Nnimap is buggy, it should look at UIDVALIDITY as well here.
I'll fix it.

--- nnimap.el.~6.36.~	Sat May 25 16:16:03 2002
+++ nnimap.el	Thu Jun  6 15:05:15 2002
@@ -975,7 +975,8 @@
 			     'asyncgroups
 			   'slowgroups)
 			 (list group (imap-mailbox-status-asynch
-				      group 'uidnext nnimap-server-buffer))))
+				      group '(uidnext uidvalidity)
+				      nnimap-server-buffer))))
 	  (dolist (asyncgroup asyncgroups)
 	    (let ((group (nth 0 asyncgroup))
 		  (tag   (nth 1 asyncgroup))
@@ -984,7 +985,8 @@
 		(if (nnimap-string-lessp-numerical
 		     (car (gnus-gethash
 			   (concat server group) nnimap-mailbox-info))
-		     (imap-mailbox-get 'uidnext group nnimap-server-buffer))
+		     (imap-mailbox-get '(uidnext uidvalidity)
+				       group nnimap-server-buffer))
 		    (push (list group) slowgroups)
 		  (insert (cdr (gnus-gethash (concat server group)
 					     nnimap-mailbox-info))))))))


> Is this a known problem with the Courier server?

No, but there are other known problems with it.

The workaround is to disable nnimap-retrieve-groups-asynchronous for
the server.

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Bjørn Mork]

Simon Josefsson <jas@extundo.com> writes:
> Bjørn Mork <bmork@dod.no> writes:
>
>> I currently use three imap servers:
>> a) IMAP4 server (InterMail vM.4.01.03.23 201-229-121-123-20010418)
>> b) Courier-IMAP ready. Copyright 1998-2001 Double Precision, Inc.
>> c) Courier-IMAP ready. Copyright 1998-2002 Double Precision, Inc.
>
> I suspect the same problem as Ilpo reported, also Courier IMAPD from
> what I recall.  I haven't debug that yet though.
>
>> The problem is that Gnus doesn't detect new mail split into a folder
>> on b) when pressing 'g'. Doing a 'M-g' on the folder will however show
>> the new mail. The funny thing is that everything works fine on server
>> a) and c). So it looks like a combination of a Courier server and
>> client side splitting is causing the problem, unless there is a
>> specific problem with the Courier version running on b).
>
> Can you find out the server versions used on server a and b?

I assume you meant b and c, since a reports the version its the
greeting above. c is running a CVS version from 2002-03-14, which
should make it 1.4.3 + some fixes. I believe b is running 1.3.5 but
haven't been able to confirm this. Is there any way to check this
remotely, or do I really have to get hold of the guy who compiled it? 

> Hm.  At first glance, it looks like it, but it could be valid
> behaviour if UIDVALIDITY changed. Could you try running with this
> patch?  Does the UIDVALIDITY change between 128- and 175-like commands
> then?  Nnimap is buggy, it should look at UIDVALIDITY as well here.
> I'll fix it.

But it doesn't make any difference in this case. With your patch:

44 STATUS "INBOX.Trash" (uidnext uidvalidity)
* STATUS "INBOX.Trash" (UIDNEXT 2 UIDVALIDITY 979914089)
44 OK STATUS Completed.
[..]
57 SELECT "INBOX"
* FLAGS (\Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen)] Limited
* 135 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 979914089] Ok
57 OK [READ-WRITE] Ok
58 UID SEARCH UNSEEN UNDELETED
* SEARCH 376
58 OK SEARCH done.
59 UID FETCH 376 BODY.PEEK[HEADER]
* 135 FETCH (UID 376 BODY[HEADER] {1380}
[snipped mail header]
59 OK FETCH completed.
60 UID COPY 376 "INBOX.Trash"
60 OK COPY completed.
61 UID STORE 376 +FLAGS (\Seen \Deleted)
* 135 FETCH (FLAGS (\Seen \Deleted \Recent))
61 OK STORE completed.
62 EXPUNGE
* 135 EXPUNGE
* 134 EXISTS
* 0 RECENT
62 OK EXPUNGE completed
63 CLOSE
63 OK mailbox closed.
[..]
73 STATUS "INBOX.Trash" (uidnext uidvalidity)
* STATUS "INBOX.Trash" (UIDNEXT 2 UIDVALIDITY 979914089)
73 OK STATUS Completed.


I also tested client side splitting with server c, the Courier-IMAP
version 1.4.3, and it does indeed look like it's working better:

36 STATUS "INBOX.Trash" (uidnext uidvalidity)
* STATUS "INBOX.Trash" (UIDNEXT 4 UIDVALIDITY 980027355)
36 OK STATUS Completed.
[..]
56 SELECT "INBOX.mail.other"
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\Draft \Answered \Flagged \Deleted \Seen)] Limited
* 94 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1018363677] Ok
56 OK [READ-WRITE] Ok
57 UID SEARCH UNSEEN UNDELETED
* SEARCH 327
57 OK SEARCH done.
58 UID FETCH 327 BODY.PEEK[HEADER]
* 94 FETCH (UID 327 BODY[HEADER] {856}
[snipped mail header]
58 OK FETCH completed.
59 UID COPY 327 "INBOX.Trash"
59 OK COPY completed.
60 UID STORE 327 +FLAGS (\Seen \Deleted)
* 94 FETCH (UID 327 FLAGS (\Seen \Deleted \Recent))
60 OK STORE completed.
61 EXPUNGE
* 94 EXPUNGE
* 93 EXISTS
* 0 RECENT
61 OK EXPUNGE completed
62 CLOSE
62 OK mailbox closed.
[..]
77 STATUS "INBOX.Trash" (uidnext uidvalidity)
* STATUS "INBOX.Trash" (UIDNEXT 5 UIDVALIDITY 980027355)
77 OK STATUS Completed.


The only problem is that Gnus still didn't show me the new mail in the
INBOX.Thrash group. Now I'm lost again...

> The workaround is to disable nnimap-retrieve-groups-asynchronous for
> the server.

Yeah, I know. I just had a small hope I could avoid that :-)


Bjørn
-- 
You must be a real DAF driver to think that the Centre Bulletin is
always right.

Re: Faster nnimap: nnimap-retrieve-groups-asynchronous

[Bjørn Mork]

Bjørn Mork <bmork@dod.no> writes:
[..]

Why bother with all the testing when you could have just read the
Changelog:

2002-02-24  Mr. Sam  <mrsam@courier-mta.com>

        * imapscanclient.c: fix UIDNEXT in the STATUS response to take into
        account new mail.

Yes, it's a server bug and it's fixed in Courier-IMAP version 1.4.3
and later. I'll see if I can get the server in question upgraded.

> I also tested client side splitting with server c, the Courier-IMAP
> version 1.4.3, and it does indeed look like it's working better:
[..]
> The only problem is that Gnus still didn't show me the new mail in the
> INBOX.Thrash group. Now I'm lost again...

Which seems to be only an unfortunate effect of the patch in
<iluzny8zhbc.fsf@latte.josefsson.org> . Both server side and client
side splitting works fine with Courier-IMAP version 1.4.3 as long as
this patch isn't installed.


Bjørn
-- 
Let me tell you something, you cretin, Dennis Ritchie is Polish.