Unable to get all messages in an IMAP group

Unable to get all messages in an IMAP group

[Vincent Bernat]

Hey!

Since the latest IMAP changes, I am unable to enter some groups and on
some others, I don't get all the messages. For example, currently, I see
3 messages in my INBOX while there 10 when I use another IMAP client.

When I refresh the group, I get "nnimap read 0k from " with nothing else
after. What could be the problem?
-- 
As to the Adjective: when in doubt, strike it out.
		-- Mark Twain, "Pudd'nhead Wilson's Calendar"

Re: Unable to get all messages in an IMAP group

[Steinar Bang]

>>>>> Vincent Bernat <bernat@luffy.cx>:

> Since the latest IMAP changes, I am unable to enter some groups and on
> some others, I don't get all the messages. For example, currently, I see
> 3 messages in my INBOX while there 10 when I use another IMAP client.

> When I refresh the group, I get "nnimap read 0k from " with nothing else
> after. What could be the problem?

What server?  Dovecot? Gmail? Other?
Do you use the agent on the nnimap groups where you have the issue?

Re: Unable to get all messages in an IMAP group

[Vincent Bernat]

 ❦  3 février 2015 07:42 +0100, Steinar Bang <sb@dod.no> :

>> Since the latest IMAP changes, I am unable to enter some groups and on
>> some others, I don't get all the messages. For example, currently, I see
>> 3 messages in my INBOX while there 10 when I use another IMAP client.
>
>> When I refresh the group, I get "nnimap read 0k from " with nothing else
>> after. What could be the problem?
>
> What server?  Dovecot? Gmail? Other?
> Do you use the agent on the nnimap groups where you have the issue?

It's Dovecot 2.2.13. I don't use the agent. I see that nnimap-debug
doesn't exist anymore. How to get some traces?

Also, the content of .newsrc.eld may be inexact. I am synchronizing it
across my hosts and I am using isync to replicate IMAP mailboxes (served
locally by Dovecot). So, each host has its own view of the IMAP
server. Usually, after a sync, M-g is enough to get Gnus and Dovecot
agree again.
-- 
But, for my own part, it was Greek to me.
		-- William Shakespeare, "Julius Caesar"

Re: Unable to get all messages in an IMAP group

[Vincent Bernat]

 ❦  3 février 2015 08:40 +0100, Vincent Bernat <bernat@luffy.cx> :

>>> Since the latest IMAP changes, I am unable to enter some groups and on
>>> some others, I don't get all the messages. For example, currently, I see
>>> 3 messages in my INBOX while there 10 when I use another IMAP client.
>>
>>> When I refresh the group, I get "nnimap read 0k from " with nothing else
>>> after. What could be the problem?
>>
>> What server?  Dovecot? Gmail? Other?
>> Do you use the agent on the nnimap groups where you have the issue?
>
> It's Dovecot 2.2.13. I don't use the agent. I see that nnimap-debug
> doesn't exist anymore. How to get some traces?
>
> Also, the content of .newsrc.eld may be inexact. I am synchronizing it
> across my hosts and I am using isync to replicate IMAP mailboxes (served
> locally by Dovecot). So, each host has its own view of the IMAP
> server. Usually, after a sync, M-g is enough to get Gnus and Dovecot
> agree again.

I am using that in Gnus:

#v+
(setq gnus-select-method
      ;; Primary server: dovecot
      `(nnimap ""
               (nnimap-stream shell)
               (nnimap-shell-program "/usr/lib/dovecot/imap -o mail_location=maildir:~/.mbsync/mails/luffy")
               (nnir-search-engine imap)))
#v-

IMAP server is feeded through isync/mbsync. So, UUID or whatever IMAP
uses to identify messages may be off-sync. This never was a problem
before.
-- 
The Public is merely a multiplied "me."
		-- Mark Twain

Re: Unable to get all messages in an IMAP group

[Steinar Bang]

>>>>> Vincent Bernat <bernat@luffy.cx>:

> It's Dovecot 2.2.13. I don't use the agent.

Hm, ok... from your symptoms, I suspected the agent, but obviously that
wasn't the problem.

And dovecot is the best-behaved IMAP server nnimap is used
against.

> I see that nnimap-debug doesn't exist anymore.

So it is.

> How to get some traces?

I don't know, unfortunately.

> Also, the content of .newsrc.eld may be inexact. I am synchronizing it
> across my hosts and I am using isync to replicate IMAP mailboxes (served
> locally by Dovecot). So, each host has its own view of the IMAP
> server. Usually, after a sync, M-g is enough to get Gnus and Dovecot
> agree again.

What happens if you move your .newsrc.eld aside and create a new
.newsrc.eld for the nnimap groups?  Does it work initially and then get
corrupted?  Does it work initially and continue to work?

Re: Unable to get all messages in an IMAP group

[Tassilo Horn]

Steinar Bang <sb@dod.no> writes:

>> I see that nnimap-debug doesn't exist anymore.
>
> So it is.
>
>> How to get some traces?
>
> I don't know, unfortunately.

,----[ C-h v nnimap-record-commands RET ]
| nnimap-record-commands is a variable defined in `nnimap.el'.
| Its value is nil
| 
|   This variable may be risky if used as a file-local variable.
| 
| Documentation:
| If non-nil, log commands to the "*imap log*" buffer.
`----

Bye,
Tassilo

Re: Unable to get all messages in an IMAP group

[Vincent Bernat]

 ❦  3 février 2015 09:05 +0100, Steinar Bang <sb@dod.no> :

>> Also, the content of .newsrc.eld may be inexact. I am synchronizing it
>> across my hosts and I am using isync to replicate IMAP mailboxes (served
>> locally by Dovecot). So, each host has its own view of the IMAP
>> server. Usually, after a sync, M-g is enough to get Gnus and Dovecot
>> agree again.
>
> What happens if you move your .newsrc.eld aside and create a new
> .newsrc.eld for the nnimap groups?  Does it work initially and then get
> corrupted?  Does it work initially and continue to work?

I'll try tonight. On my laptop, there is no such problem (same
newsrc.eld). It should be related to those out-of-sync UID.
-- 
Take care to branch the right way on equality.
            - The Elements of Programming Style (Kernighan & Plauger)

Re: Unable to get all messages in an IMAP group

[Steinar Bang]

>>>>> Vincent Bernat <bernat@luffy.cx>:

> I'll try tonight. On my laptop, there is no such problem (same
> newsrc.eld). It should be related to those out-of-sync UID.

One other possibility:

Go back to the previous version of gnus in git, and if that works, use
git bisect to find the commit where the change happened.

(If going back to the previous version of gnus doesn't help, then this
approach won't work, obviously...)

Re: Unable to get all messages in an IMAP group

[Vincent Bernat]

 ❦  3 février 2015 11:54 +0100, Steinar Bang <sb@dod.no> :

>> I'll try tonight. On my laptop, there is no such problem (same
>> newsrc.eld). It should be related to those out-of-sync UID.
>
> One other possibility:
>
> Go back to the previous version of gnus in git, and if that works, use
> git bisect to find the commit where the change happened.
>
> (If going back to the previous version of gnus doesn't help, then this
> approach won't work, obviously...)

So, it's easy to fix by going back. I have bisected (and ensured to not
fix the situation by not saving .newsrc.eld) and found that the problem
is this commit:
 https://github.com/emacsmirror/gnus/commit/447c56af7cb2afac14a0f3ed23f85d907c443a5d

Before this commit, if I enter the group, I don't see all the messages
but if I hit M-g on the group, then enter it, everything is here (which
is what I expect). With this commit, doing M-g still gives an incomplete
view of the group.
-- 
Use uniform input formats.
            - The Elements of Programming Style (Kernighan & Plauger)

Re: Unable to get all messages in an IMAP group

[Steinar Bang]

>>>>> Vincent Bernat <bernat@luffy.cx>:

> So, it's easy to fix by going back. I have bisected (and ensured to not
> fix the situation by not saving .newsrc.eld) and found that the problem
> is this commit:
>  https://github.com/emacsmirror/gnus/commit/447c56af7cb2afac14a0f3ed23f85d907c443a5d

Ah, the speedup-on-large-imap-move commit, I think...?
 http://thread.gmane.org/gmane.emacs.gnus.general/85440

Re: Unable to get all messages in an IMAP group

[Vincent Bernat]

 ❦  4 février 2015 07:28 +0100, Steinar Bang <sb@dod.no> :

>> So, it's easy to fix by going back. I have bisected (and ensured to not
>> fix the situation by not saving .newsrc.eld) and found that the problem
>> is this commit:
>>  https://github.com/emacsmirror/gnus/commit/447c56af7cb2afac14a0f3ed23f85d907c443a5d
>
> Ah, the speedup-on-large-imap-move commit, I think...?
>  http://thread.gmane.org/gmane.emacs.gnus.general/85440

Yes.

Unfortunately, in my case, this is not fixed in today master. Before the
patch, M-g does:

#v+
21:46:17 [] 106 SELECT "INBOX"
21:46:17 [] 107 UID FETCH 1:* FLAGS
#v-

After the patch, it does:

#v+
21:44:23 [] 9626 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
#v-

What easy way do I have to restore M-g behaviour to issue a full
refresh?
-- 
Say what you mean, simply and directly.
            - The Elements of Programming Style (Kernighan & Plauger)

Re: Unable to get all messages in an IMAP group

[Steinar Bang]

>>>>> Vincent Bernat <bernat@luffy.cx>:

[snip!]
> After the patch, it does:

> #v+
> 21:44:23 [] 9626 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
> #v-

Well, at least that explains why 'M-g' stopped working...

> What easy way do I have to restore M-g behaviour to issue a full
> refresh?

I don't know, unfortunately... hope that Lars get back online soon, with
an urge to fix Gnus issues...?  (it sounded like he was going walkabout
in Australia the last time he posted)

Re: Unable to get all messages in an IMAP group

[Vincent Bernat]

[-- Attachment #1: Type: text/plain, Size: 632 bytes --]

 ❦  4 février 2015 21:48 +0100, Vincent Bernat <bernat@luffy.cx> :

>> Ah, the speedup-on-large-imap-move commit, I think...?
>>  http://thread.gmane.org/gmane.emacs.gnus.general/85440
>
> Yes.
>
> Unfortunately, in my case, this is not fixed in today master. Before the
> patch, M-g does:
>
> #v+
> 21:46:17 [] 106 SELECT "INBOX"
> 21:46:17 [] 107 UID FETCH 1:* FLAGS
> #v-
>
>
> After the patch, it does:
>
> #v+
> 21:44:23 [] 9626 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
> #v-
>
> What easy way do I have to restore M-g behaviour to issue a full
> refresh?

I am fixing that with this patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-nnimap-fix-group-scan-request.patch --]
[-- Type: text/x-diff, Size: 1506 bytes --]

From 025abeeebac0c196b8efa3aa538bcc69b7037374 Mon Sep 17 00:00:00 2001
From: Vincent Bernat <vincent@bernat.im>
Date: Wed, 4 Feb 2015 22:08:48 +0100
Subject: [PATCH] nnimap: fix group scan request

`gnus-group-get-new-news-this-group' was not refreshing the group with
the newly introduced `nnimap-request-group-scan' method due to two
naming problems.
---
 lisp/gnus-int.el | 2 +-
 lisp/nnimap.el   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/lisp/gnus-int.el b/lisp/gnus-int.el
index dd938ce07586..4e870bb84bb9 100644
--- a/lisp/gnus-int.el
+++ b/lisp/gnus-int.el
@@ -442,7 +442,7 @@ If it is down, start it up (again)."
 (defun gnus-request-group-scan (group info)
   "Request that GROUP get a complete rescan."
   (let ((gnus-command-method (gnus-find-method-for-group group))
-	(func 'request-group-description))
+	(func 'request-group-scan))
     (when (gnus-check-backend-function func group)
       (funcall (gnus-get-function gnus-command-method func)
 	       (gnus-group-real-name group) (nth 1 gnus-command-method) info))))
diff --git a/lisp/nnimap.el b/lisp/nnimap.el
index e619c0f13c29..cdbc2a68709b 100644
--- a/lisp/nnimap.el
+++ b/lisp/nnimap.el
@@ -820,7 +820,7 @@ textual parts.")
 			group))
 	t))))
 
-(deffoo nnimap-request-scan-group (group &optional server info)
+(deffoo nnimap-request-group-scan (group &optional server info)
   (setq group (nnimap-decode-gnus-group group))
   (let (marks high low)
     (with-current-buffer (nnimap-buffer)
-- 
2.1.4


[-- Attachment #3: Type: text/plain, Size: 102 bytes --]


-- 
It is a wise father that knows his own child.
		-- William Shakespeare, "The Merchant of Venice"

Re: Unable to get all messages in an IMAP group

[Vincent Bernat]

 ❦  4 février 2015 22:11 +0100, Vincent Bernat <bernat@luffy.cx> :

>> Yes.
>>
>> Unfortunately, in my case, this is not fixed in today master. Before the
>> patch, M-g does:
>>
>> #v+
>> 21:46:17 [] 106 SELECT "INBOX"
>> 21:46:17 [] 107 UID FETCH 1:* FLAGS
>> #v-
>>
>>
>> After the patch, it does:
>>
>> #v+
>> 21:44:23 [] 9626 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
>> #v-
>>
>> What easy way do I have to restore M-g behaviour to issue a full
>> refresh?
>
> I am fixing that with this patch:

This doesn't fix everything for me. I think there is another problem.

When I refresh my INBOX with M-g, then do g twice, I get this trace:

22:15:19 [] 4253 SELECT "INBOX"
22:15:19 [] 4254 UID FETCH 1:* FLAGS

Count is correct (all read).

22:15:28 [exoscale] 4260 EXAMINE "INBOX" (QRESYNC (1402209543 15465))

Count is correct (all read)

22:15:31 [exoscale] 4355 SELECT "INBOX"
22:15:31 [exoscale] 4356 UID FETCH 1:* FLAGS

Count is incorrect (many unread)

I suppose that M-g may not refresh the right server due to the empty
square brackets in contrast with what is happening with g. This doesn't
happen with my other INBOX (the primary one).
-- 
Use the good features of a language; avoid the bad ones.
            - The Elements of Programming Style (Kernighan & Plauger)

Re: Unable to get all messages in an IMAP group

[Vincent Bernat]

 ❦  4 février 2015 22:19 +0100, Vincent Bernat <vincent@bernat.im> :

> This doesn't fix everything for me. I think there is another problem.
>
> When I refresh my INBOX with M-g, then do g twice, I get this trace:
>
> 22:15:19 [] 4253 SELECT "INBOX"
> 22:15:19 [] 4254 UID FETCH 1:* FLAGS
>
> Count is correct (all read).
>
> 22:15:28 [exoscale] 4260 EXAMINE "INBOX" (QRESYNC (1402209543 15465))
>
> Count is correct (all read)
>
> 22:15:31 [exoscale] 4355 SELECT "INBOX"
> 22:15:31 [exoscale] 4356 UID FETCH 1:* FLAGS
>
> Count is incorrect (many unread)
>
> I suppose that M-g may not refresh the right server due to the empty
> square brackets in contrast with what is happening with g. This doesn't
> happen with my other INBOX (the primary one).

If I use this patch:

#v+
diff --git a/lisp/nnimap.el b/lisp/nnimap.el
index cdbc2a68709b..8f9afa634a87 100644
--- a/lisp/nnimap.el
+++ b/lisp/nnimap.el
@@ -822,6 +822,7 @@ textual parts.")
 
 (deffoo nnimap-request-group-scan (group &optional server info)
   (setq group (nnimap-decode-gnus-group group))
+  (nnimap-change-group group server)
   (let (marks high low)
     (with-current-buffer (nnimap-buffer)
       (erase-buffer)
#v-

I get this trace:

22:33:33 [exoscale] 5936 SELECT "INBOX"
22:33:33 [exoscale] 5937 SELECT "INBOX"
22:33:33 [exoscale] 5938 UID FETCH 1:* FLAGS

So, it seems to use the right IMAP server but the count is the wrong one
(a lot of unread messages). But maybe the group is just borked, I could
use M-c on it. I leave it as is if it needs more debugging.
-- 
Hell is empty and all the devils are here.
		-- Wm. Shakespeare, "The Tempest"

Re: Unable to get all messages in an IMAP group

[Lars Ingebrigtsen]

Vincent Bernat <bernat@luffy.cx> writes:

> I am fixing that with this patch:
>
> From 025abeeebac0c196b8efa3aa538bcc69b7037374 Mon Sep 17 00:00:00 2001
> From: Vincent Bernat <vincent@bernat.im>
> Date: Wed, 4 Feb 2015 22:08:48 +0100
> Subject: [PATCH] nnimap: fix group scan request
>
> `gnus-group-get-new-news-this-group' was not refreshing the group with
> the newly introduced `nnimap-request-group-scan' method due to two
> naming problems.

Thanks; applied.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Unable to get all messages in an IMAP group

[Lars Ingebrigtsen]

Vincent Bernat <vincent@bernat.im> writes:

> +  (nnimap-change-group group server)

I've installed a version of this in git Gnus.  It seems to be doing the
right thing for me now, at least...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: Unable to get all messages in an IMAP group

[Vincent Bernat]

 ❦  5 février 2015 14:10 +1100, Lars Ingebrigtsen <larsi@gnus.org> :

>> +  (nnimap-change-group group server)
>
> I've installed a version of this in git Gnus.  It seems to be doing the
> right thing for me now, at least...

Thanks! It works fine for me now.
-- 
Localise input and output in subroutines.
            - The Elements of Programming Style (Kernighan & Plauger)