Finding message by ID fails when message not visible

Finding message by ID fails when message not visible

[Kai von Fintel]

I'm puzzled by the following behavior: if I search in an nnimap group
for a message with a particular message-ID, the search fails unless the
message is visible. If for example, only new messages are shown, the
search for an old message with a particular ID fails but the same
message is found when "/o" has made the old messages appear.

This is relevant because I use links to gnus messages from org-mode and
more often than not the link fails: I get taken to the nnimap group that
contains the message, but I get the error "No such article (may have
expired or been canceled)", even though the message is in fact in the
group.

I suppose org-gnus.el might have to be amended to work around this, but
first I'd like to understand whether this is expected behavior in gnus.

Can anybody enlighten me or help me trouble-shoot this?

-- Kai.

Re: Finding message by ID fails when message not visible

[Eric Abrahamsen]

Kai von Fintel <fintel@mit.edu> writes:

> I'm puzzled by the following behavior: if I search in an nnimap group
> for a message with a particular message-ID, the search fails unless the
> message is visible. If for example, only new messages are shown, the
> search for an old message with a particular ID fails but the same
> message is found when "/o" has made the old messages appear.
>
> This is relevant because I use links to gnus messages from org-mode and
> more often than not the link fails: I get taken to the nnimap group that
> contains the message, but I get the error "No such article (may have
> expired or been canceled)", even though the message is in fact in the
> group.
>
> I suppose org-gnus.el might have to be amended to work around this, but
> first I'd like to understand whether this is expected behavior in gnus.
>
> Can anybody enlighten me or help me trouble-shoot this?

No, it's not expected, and I haven't seen it before -- sorry for the
unhelpful reply! I just tried it (dovecot imap server), and it worked
both with and without angle brackets on the ID (sometimes brackets can
be an issue).

To be sure everyone's on the same page, how exactly are you searching?
If you're in the *Group* buffer and you hit "G G" on the group, and
paste in the message ID, is the message found? If you're already in the
*Summary* buffer, there isn't really any way to "search" as such,
there's only "limiting" what messages are shown by some criteria -- in
those cases it's true you won't be able to get to unseen messages.

But that should never affect following links from org mode...

Eric

Re: Finding message by ID fails when message not visible

[Kai von Fintel]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Kai von Fintel <fintel@mit.edu> writes:
>
>> I'm puzzled by the following behavior: if I search in an nnimap group
>> for a message with a particular message-ID, the search fails unless the
>> message is visible. If for example, only new messages are shown, the
>> search for an old message with a particular ID fails but the same
>> message is found when "/o" has made the old messages appear.
>>
>> This is relevant because I use links to gnus messages from org-mode and
>> more often than not the link fails: I get taken to the nnimap group that
>> contains the message, but I get the error "No such article (may have
>> expired or been canceled)", even though the message is in fact in the
>> group.
>>
>> I suppose org-gnus.el might have to be amended to work around this, but
>> first I'd like to understand whether this is expected behavior in gnus.
>>
>> Can anybody enlighten me or help me trouble-shoot this?
>
> No, it's not expected, and I haven't seen it before -- sorry for the
> unhelpful reply! I just tried it (dovecot imap server), and it worked
> both with and without angle brackets on the ID (sometimes brackets can
> be an issue).
>
> To be sure everyone's on the same page, how exactly are you searching?
> If you're in the *Group* buffer and you hit "G G" on the group, and
> paste in the message ID, is the message found? If you're already in the
> *Summary* buffer, there isn't really any way to "search" as such,
> there's only "limiting" what messages are shown by some criteria -- in
> those cases it's true you won't be able to get to unseen messages.
>
> But that should never affect following links from org mode...
>
> Eric

Thanks for the reply, Eric. More details then.

1. GG in the group buffer works (after a brief delay, displaying
"Opening server FastmailLocal" and then "Searching
nnimap+FastmailLocal:Archive...done").

2. M-^ ("Fetch article with id ...) in the summary buffer (a) works when
the message is visible, (b) fails when the message isn't visible (with
"No such article (may have expired or been canceled)").

3. When I follow the org link, say
"gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", I do get
taken to the right nnimap group in gnus but the article is not displayed
(again, with "No such article (may have expired or been canceled)").

One funny thing is that when the "search" fails (in both cases 2b and
3), the article summary actually is visible (see screenshot:
https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)

If 1 and 2 are as expected, then I guess the issue is with org-gnus and
how it interacts with my IMAP set-up (offlineimap + dovecot + nnimap)?

-- Kai.

Re: Finding message by ID fails when message not visible

[Kai von Fintel]

Kai von Fintel <fintel@mit.edu> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Kai von Fintel <fintel@mit.edu> writes:
>>
>>> I'm puzzled by the following behavior: if I search in an nnimap group
>>> for a message with a particular message-ID, the search fails unless the
>>> message is visible. If for example, only new messages are shown, the
>>> search for an old message with a particular ID fails but the same
>>> message is found when "/o" has made the old messages appear.
>>>
>>> This is relevant because I use links to gnus messages from org-mode and
>>> more often than not the link fails: I get taken to the nnimap group that
>>> contains the message, but I get the error "No such article (may have
>>> expired or been canceled)", even though the message is in fact in the
>>> group.
>>>
>>> I suppose org-gnus.el might have to be amended to work around this, but
>>> first I'd like to understand whether this is expected behavior in gnus.
>>>
>>> Can anybody enlighten me or help me trouble-shoot this?
>>
>> No, it's not expected, and I haven't seen it before -- sorry for the
>> unhelpful reply! I just tried it (dovecot imap server), and it worked
>> both with and without angle brackets on the ID (sometimes brackets can
>> be an issue).
>>
>> To be sure everyone's on the same page, how exactly are you searching?
>> If you're in the *Group* buffer and you hit "G G" on the group, and
>> paste in the message ID, is the message found? If you're already in the
>> *Summary* buffer, there isn't really any way to "search" as such,
>> there's only "limiting" what messages are shown by some criteria -- in
>> those cases it's true you won't be able to get to unseen messages.
>>
>> But that should never affect following links from org mode...
>>
>> Eric
>
> Thanks for the reply, Eric. More details then.
>
> 1. GG in the group buffer works (after a brief delay, displaying
> "Opening server FastmailLocal" and then "Searching
> nnimap+FastmailLocal:Archive...done").
>
> 2. M-^ ("Fetch article with id ...) in the summary buffer (a) works when
> the message is visible, (b) fails when the message isn't visible (with
> "No such article (may have expired or been canceled)").
>
> 3. When I follow the org link, say
> "gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", I do get
> taken to the right nnimap group in gnus but the article is not displayed
> (again, with "No such article (may have expired or been canceled)").
>
> One funny thing is that when the "search" fails (in both cases 2b and
> 3), the article summary actually is visible (see screenshot:
> https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)
>
> If 1 and 2 are as expected, then I guess the issue is with org-gnus and
> how it interacts with my IMAP set-up (offlineimap + dovecot + nnimap)?
>
> -- Kai.

Some more follow-up. I tried to understand the code in org-gnus.el and
org-sum.el. It appears that the functions "org-gnus-open" and
"org-gnus-follow-link" call the function "gnus-summary-goto-article"
from gnus-sum.el. The latter is called with the optional arguments "nil"
and "t". The last one is supposed to force that all articles are loaded,
but that doesn't seem to happen.

When I call (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t)
in the summary buffer of the relevant group, it finds the message when
it is listed and doesn't find it if the buffer has only the first 200
articles (loading all articles with "/o" then makes the function
succeed).

I added (gnus-large-newsgroup nil) to the group parameters, which then
loads all articles when entering the group, and
(gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t) then
succeeds in the summary buffer.

However, the gnus link still doesn't work from outside gnus. It's as if
the FORCE optional argument that is given to "gnus-summary-goto-article"
has no effect.

Any thoughts?

-- Kai.

Re: Finding message by ID fails when message not visible

[Eric Abrahamsen]

Kai von Fintel <fintel@mit.edu> writes:

> Kai von Fintel <fintel@mit.edu> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Kai von Fintel <fintel@mit.edu> writes:
>>>
>>>> I'm puzzled by the following behavior: if I search in an nnimap group
>>>> for a message with a particular message-ID, the search fails unless the
>>>> message is visible. If for example, only new messages are shown, the
>>>> search for an old message with a particular ID fails but the same
>>>> message is found when "/o" has made the old messages appear.
>>>>
>>>> This is relevant because I use links to gnus messages from org-mode and
>>>> more often than not the link fails: I get taken to the nnimap group that
>>>> contains the message, but I get the error "No such article (may have
>>>> expired or been canceled)", even though the message is in fact in the
>>>> group.
>>>>
>>>> I suppose org-gnus.el might have to be amended to work around this, but
>>>> first I'd like to understand whether this is expected behavior in gnus.
>>>>
>>>> Can anybody enlighten me or help me trouble-shoot this?
>>>
>>> No, it's not expected, and I haven't seen it before -- sorry for the
>>> unhelpful reply! I just tried it (dovecot imap server), and it worked
>>> both with and without angle brackets on the ID (sometimes brackets can
>>> be an issue).
>>>
>>> To be sure everyone's on the same page, how exactly are you searching?
>>> If you're in the *Group* buffer and you hit "G G" on the group, and
>>> paste in the message ID, is the message found? If you're already in the
>>> *Summary* buffer, there isn't really any way to "search" as such,
>>> there's only "limiting" what messages are shown by some criteria -- in
>>> those cases it's true you won't be able to get to unseen messages.
>>>
>>> But that should never affect following links from org mode...
>>>
>>> Eric
>>
>> Thanks for the reply, Eric. More details then.
>>
>> 1. GG in the group buffer works (after a brief delay, displaying
>> "Opening server FastmailLocal" and then "Searching
>> nnimap+FastmailLocal:Archive...done").
>>
>> 2. M-^ ("Fetch article with id ...) in the summary buffer (a) works when
>> the message is visible, (b) fails when the message isn't visible (with
>> "No such article (may have expired or been canceled)").
>>
>> 3. When I follow the org link, say
>> "gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", I do get
>> taken to the right nnimap group in gnus but the article is not displayed
>> (again, with "No such article (may have expired or been canceled)").
>>
>> One funny thing is that when the "search" fails (in both cases 2b and
>> 3), the article summary actually is visible (see screenshot:
>> https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)
>>
>> If 1 and 2 are as expected, then I guess the issue is with org-gnus and
>> how it interacts with my IMAP set-up (offlineimap + dovecot + nnimap)?
>>
>> -- Kai.
>
> Some more follow-up. I tried to understand the code in org-gnus.el and
> org-sum.el. It appears that the functions "org-gnus-open" and
> "org-gnus-follow-link" call the function "gnus-summary-goto-article"
> from gnus-sum.el. The latter is called with the optional arguments "nil"
> and "t". The last one is supposed to force that all articles are loaded,
> but that doesn't seem to happen.
>
> When I call (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t)
> in the summary buffer of the relevant group, it finds the message when
> it is listed and doesn't find it if the buffer has only the first 200
> articles (loading all articles with "/o" then makes the function
> succeed).
>
> I added (gnus-large-newsgroup nil) to the group parameters, which then
> loads all articles when entering the group, and
> (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t) then
> succeeds in the summary buffer.
>
> However, the gnus link still doesn't work from outside gnus. It's as if
> the FORCE optional argument that is given to "gnus-summary-goto-article"
> has no effect.

I've got the same setup as you (Gnus and local dovecot), but perhaps the
versions are different? Can you tell us your versions for Gnus and Org?

When I step through `gnus-summary-goto-article', searching for a message
ID that is *not* currently display, the "force" argument never even
comes into play. I end up in this branch:

(if (and (stringp article)
	       (string-match "@\\|%40" article))
	  (gnus-summary-refer-article article)

So then on to `gnus-summary-refer-article' (no force argument used), and
in that function I end up in the "t" branch of the "cond" later on, and
eventually into "(gnus-summary-insert-subject message-id)". Then that
leads to "(gnus-read-header id)", and all is well.

I know this is tedious, but would you mind stepping through with edebug
and seeing where things fall apart? My off-the-cuff guess is that you're
using an older version of Gnus where the nnimap backend isn't quite all
put together...

If you aren't comfortable with edebug let me know and I can provide
exact instructions. It's a very useful thing to know!

Eric

Re: Finding message by ID fails when message not visible

[Kai von Fintel]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Kai von Fintel <fintel@mit.edu> writes:
>
>> Kai von Fintel <fintel@mit.edu> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>
>>>>> I'm puzzled by the following behavior: if I search in an nnimap group
>>>>> for a message with a particular message-ID, the search fails unless the
>>>>> message is visible. If for example, only new messages are shown, the
>>>>> search for an old message with a particular ID fails but the same
>>>>> message is found when "/o" has made the old messages appear.
>>>>>
>>>>> This is relevant because I use links to gnus messages from org-mode and
>>>>> more often than not the link fails: I get taken to the nnimap group that
>>>>> contains the message, but I get the error "No such article (may have
>>>>> expired or been canceled)", even though the message is in fact in the
>>>>> group.
>>>>>
>>>>> I suppose org-gnus.el might have to be amended to work around this, but
>>>>> first I'd like to understand whether this is expected behavior in gnus.
>>>>>
>>>>> Can anybody enlighten me or help me trouble-shoot this?
>>>>
>>>> No, it's not expected, and I haven't seen it before -- sorry for the
>>>> unhelpful reply! I just tried it (dovecot imap server), and it worked
>>>> both with and without angle brackets on the ID (sometimes brackets can
>>>> be an issue).
>>>>
>>>> To be sure everyone's on the same page, how exactly are you searching?
>>>> If you're in the *Group* buffer and you hit "G G" on the group, and
>>>> paste in the message ID, is the message found? If you're already in the
>>>> *Summary* buffer, there isn't really any way to "search" as such,
>>>> there's only "limiting" what messages are shown by some criteria -- in
>>>> those cases it's true you won't be able to get to unseen messages.
>>>>
>>>> But that should never affect following links from org mode...
>>>>
>>>> Eric
>>>
>>> Thanks for the reply, Eric. More details then.
>>>
>>> 1. GG in the group buffer works (after a brief delay, displaying
>>> "Opening server FastmailLocal" and then "Searching
>>> nnimap+FastmailLocal:Archive...done").
>>>
>>> 2. M-^ ("Fetch article with id ...) in the summary buffer (a) works when
>>> the message is visible, (b) fails when the message isn't visible (with
>>> "No such article (may have expired or been canceled)").
>>>
>>> 3. When I follow the org link, say
>>> "gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", I do get
>>> taken to the right nnimap group in gnus but the article is not displayed
>>> (again, with "No such article (may have expired or been canceled)").
>>>
>>> One funny thing is that when the "search" fails (in both cases 2b and
>>> 3), the article summary actually is visible (see screenshot:
>>> https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)
>>>
>>> If 1 and 2 are as expected, then I guess the issue is with org-gnus and
>>> how it interacts with my IMAP set-up (offlineimap + dovecot + nnimap)?
>>>
>>> -- Kai.
>>
>> Some more follow-up. I tried to understand the code in org-gnus.el and
>> org-sum.el. It appears that the functions "org-gnus-open" and
>> "org-gnus-follow-link" call the function "gnus-summary-goto-article"
>> from gnus-sum.el. The latter is called with the optional arguments "nil"
>> and "t". The last one is supposed to force that all articles are loaded,
>> but that doesn't seem to happen.
>>
>> When I call (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t)
>> in the summary buffer of the relevant group, it finds the message when
>> it is listed and doesn't find it if the buffer has only the first 200
>> articles (loading all articles with "/o" then makes the function
>> succeed).
>>
>> I added (gnus-large-newsgroup nil) to the group parameters, which then
>> loads all articles when entering the group, and
>> (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t) then
>> succeeds in the summary buffer.
>>
>> However, the gnus link still doesn't work from outside gnus. It's as if
>> the FORCE optional argument that is given to "gnus-summary-goto-article"
>> has no effect.
>
> I've got the same setup as you (Gnus and local dovecot), but perhaps the
> versions are different? Can you tell us your versions for Gnus and Org?
>
> When I step through `gnus-summary-goto-article', searching for a message
> ID that is *not* currently display, the "force" argument never even
> comes into play. I end up in this branch:
>
> (if (and (stringp article)
> 	       (string-match "@\\|%40" article))
> 	  (gnus-summary-refer-article article)
>
> So then on to `gnus-summary-refer-article' (no force argument used), and
> in that function I end up in the "t" branch of the "cond" later on, and
> eventually into "(gnus-summary-insert-subject message-id)". Then that
> leads to "(gnus-read-header id)", and all is well.
>
> I know this is tedious, but would you mind stepping through with edebug
> and seeing where things fall apart? My off-the-cuff guess is that you're
> using an older version of Gnus where the nnimap backend isn't quite all
> put together...
>
> If you aren't comfortable with edebug let me know and I can provide
> exact instructions. It's a very useful thing to know!
>
> Eric

#### System Info
- OS: darwin
- Emacs: 24.5.1
- Gnus: v5.13

Well, that was fun ... sort of. I found something I can tweak to make it
work, but I'm not clear on why this issue arises.

At the relevant point, the article id is "<56614DE9.5030300@upf.edu>"
(the <> having been added by gnus-summary-refer-article) and the group
is "nnimap+FastmailLocal:Archive".

In `gnus-request-head (article group)', a funcall to `nnimap-request-head'
is constructed. Because `gnus-override-method' at that point contains a long
construct `(nnimap "FastmailLocal" ...)', the group argument of the call
to nnimap-request-head is set to nil rather than to "Archive". I take
it, the idea is that nnimap should be able to return the correct group
on its own?

The relevant code (in gnus-int.el) is

(setq res (funcall head article
                         (and (not gnus-override-method)
                              (gnus-group-real-name group))
       (nth 1 gnus-command-method)))

Now, `nnimap-request-head (article nil "FastmailLocal")' returns "(nil .
277)". I believe the "nil" in that cons should be the group ("Archive").
Maybe that's a problem in nnimap.el but looking at the code I don't see
how it could ever return anything other than nil.

From that point on, the group info seems lost and the message can't be
targeted.

If I revert the commit to gnus-int.el that introduced the override
method thingy
(http://git.gnus.org/cgit/gnus.git/commit/?id=8ba1cd0b96466a96265ec5336728519aa6030d83),
to the following instead of what's above:

(setq res (funcall head article (gnus-group-real-name group)
			 (nth 1 gnus-command-method))))

my messages get found because the actual group is passed on to
nnimap-request-head.

So, I have things working to a certain degree, but I'm utterly confused
about why this is happening.

Can you see from this what the problem is?

-- Kai.

Re: Finding message by ID fails when message not visible

[Eric Abrahamsen]

Kai von Fintel <fintel@mit.edu> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Kai von Fintel <fintel@mit.edu> writes:
>>
>>> Kai von Fintel <fintel@mit.edu> writes:
>>>
>>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>>
>>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>>
>>>>>> I'm puzzled by the following behavior: if I search in an nnimap group
>>>>>> for a message with a particular message-ID, the search fails unless the
>>>>>> message is visible. If for example, only new messages are shown, the
>>>>>> search for an old message with a particular ID fails but the same
>>>>>> message is found when "/o" has made the old messages appear.
>>>>>>
>>>>>> This is relevant because I use links to gnus messages from org-mode and
>>>>>> more often than not the link fails: I get taken to the nnimap group that
>>>>>> contains the message, but I get the error "No such article (may have
>>>>>> expired or been canceled)", even though the message is in fact in the
>>>>>> group.
>>>>>>
>>>>>> I suppose org-gnus.el might have to be amended to work around this, but
>>>>>> first I'd like to understand whether this is expected behavior in gnus.
>>>>>>
>>>>>> Can anybody enlighten me or help me trouble-shoot this?
>>>>>
>>>>> No, it's not expected, and I haven't seen it before -- sorry for the
>>>>> unhelpful reply! I just tried it (dovecot imap server), and it worked
>>>>> both with and without angle brackets on the ID (sometimes brackets can
>>>>> be an issue).
>>>>>
>>>>> To be sure everyone's on the same page, how exactly are you searching?
>>>>> If you're in the *Group* buffer and you hit "G G" on the group, and
>>>>> paste in the message ID, is the message found? If you're already in the
>>>>> *Summary* buffer, there isn't really any way to "search" as such,
>>>>> there's only "limiting" what messages are shown by some criteria -- in
>>>>> those cases it's true you won't be able to get to unseen messages.
>>>>>
>>>>> But that should never affect following links from org mode...
>>>>>
>>>>> Eric
>>>>
>>>> Thanks for the reply, Eric. More details then.
>>>>
>>>> 1. GG in the group buffer works (after a brief delay, displaying
>>>> "Opening server FastmailLocal" and then "Searching
>>>> nnimap+FastmailLocal:Archive...done").
>>>>
>>>> 2. M-^ ("Fetch article with id ...) in the summary buffer (a) works when
>>>> the message is visible, (b) fails when the message isn't visible (with
>>>> "No such article (may have expired or been canceled)").
>>>>
>>>> 3. When I follow the org link, say
>>>> "gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", I do get
>>>> taken to the right nnimap group in gnus but the article is not displayed
>>>> (again, with "No such article (may have expired or been canceled)").
>>>>
>>>> One funny thing is that when the "search" fails (in both cases 2b and
>>>> 3), the article summary actually is visible (see screenshot:
>>>> https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)
>>>>
>>>> If 1 and 2 are as expected, then I guess the issue is with org-gnus and
>>>> how it interacts with my IMAP set-up (offlineimap + dovecot + nnimap)?
>>>>
>>>> -- Kai.
>>>
>>> Some more follow-up. I tried to understand the code in org-gnus.el and
>>> org-sum.el. It appears that the functions "org-gnus-open" and
>>> "org-gnus-follow-link" call the function "gnus-summary-goto-article"
>>> from gnus-sum.el. The latter is called with the optional arguments "nil"
>>> and "t". The last one is supposed to force that all articles are loaded,
>>> but that doesn't seem to happen.
>>>
>>> When I call (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t)
>>> in the summary buffer of the relevant group, it finds the message when
>>> it is listed and doesn't find it if the buffer has only the first 200
>>> articles (loading all articles with "/o" then makes the function
>>> succeed).
>>>
>>> I added (gnus-large-newsgroup nil) to the group parameters, which then
>>> loads all articles when entering the group, and
>>> (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t) then
>>> succeeds in the summary buffer.
>>>
>>> However, the gnus link still doesn't work from outside gnus. It's as if
>>> the FORCE optional argument that is given to "gnus-summary-goto-article"
>>> has no effect.
>>
>> I've got the same setup as you (Gnus and local dovecot), but perhaps the
>> versions are different? Can you tell us your versions for Gnus and Org?
>>
>> When I step through `gnus-summary-goto-article', searching for a message
>> ID that is *not* currently display, the "force" argument never even
>> comes into play. I end up in this branch:
>>
>> (if (and (stringp article)
>> 	       (string-match "@\\|%40" article))
>> 	  (gnus-summary-refer-article article)
>>
>> So then on to `gnus-summary-refer-article' (no force argument used), and
>> in that function I end up in the "t" branch of the "cond" later on, and
>> eventually into "(gnus-summary-insert-subject message-id)". Then that
>> leads to "(gnus-read-header id)", and all is well.
>>
>> I know this is tedious, but would you mind stepping through with edebug
>> and seeing where things fall apart? My off-the-cuff guess is that you're
>> using an older version of Gnus where the nnimap backend isn't quite all
>> put together...
>>
>> If you aren't comfortable with edebug let me know and I can provide
>> exact instructions. It's a very useful thing to know!
>>
>> Eric
>
> #### System Info
> - OS: darwin
> - Emacs: 24.5.1
> - Gnus: v5.13
>
> Well, that was fun ... sort of. I found something I can tweak to make it
> work, but I'm not clear on why this issue arises.
>
> At the relevant point, the article id is "<56614DE9.5030300@upf.edu>"
> (the <> having been added by gnus-summary-refer-article) and the group
> is "nnimap+FastmailLocal:Archive".
>
> In `gnus-request-head (article group)', a funcall to `nnimap-request-head'
> is constructed. Because `gnus-override-method' at that point contains a long
> construct `(nnimap "FastmailLocal" ...)', the group argument of the call
> to nnimap-request-head is set to nil rather than to "Archive". I take
> it, the idea is that nnimap should be able to return the correct group
> on its own?
>
> The relevant code (in gnus-int.el) is
>
> (setq res (funcall head article
>                          (and (not gnus-override-method)
>                               (gnus-group-real-name group))
>        (nth 1 gnus-command-method)))
>
> Now, `nnimap-request-head (article nil "FastmailLocal")' returns "(nil .
> 277)". I believe the "nil" in that cons should be the group ("Archive").
> Maybe that's a problem in nnimap.el but looking at the code I don't see
> how it could ever return anything other than nil.
>
> From that point on, the group info seems lost and the message can't be
> targeted.
>
> If I revert the commit to gnus-int.el that introduced the override
> method thingy
> (http://git.gnus.org/cgit/gnus.git/commit/?id=8ba1cd0b96466a96265ec5336728519aa6030d83),
> to the following instead of what's above:
>
> (setq res (funcall head article (gnus-group-real-name group)
> 			 (nth 1 gnus-command-method))))
>
> my messages get found because the actual group is passed on to
> nnimap-request-head.
>
> So, I have things working to a certain degree, but I'm utterly confused
> about why this is happening.
>
> Can you see from this what the problem is?

My guess is it might just be time to update Gnus. Nnimap has undergone a
lot of changes over the past couple of years, as has the rest of Gnus,
and while the versioning system has always confused me, I'm pretty sure
v5.13 is several years old. The code branches you seem to be following
don't match up with what I'm seeing (in git master, aka Ma Gnus), and
it's probably not worth trying to figure out exactly what's going on. I
find Ma Gnus pretty stable -- would you be willing to run from git?

Eric

Re: Finding message by ID fails when message not visible

[Kai von Fintel]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Kai von Fintel <fintel@mit.edu> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Kai von Fintel <fintel@mit.edu> writes:
>>>
>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>
>>>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>>>
>>>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>>>
>>>>>>> I'm puzzled by the following behavior: if I search in an 
>>>>>>> nnimap group
>>>>>>> for a message with a particular message-ID, the search 
>>>>>>> fails unless the
>>>>>>> message is visible. If for example, only new messages are 
>>>>>>> shown, the
>>>>>>> search for an old message with a particular ID fails but 
>>>>>>> the same
>>>>>>> message is found when "/o" has made the old messages 
>>>>>>> appear.
>>>>>>>
>>>>>>> This is relevant because I use links to gnus messages from 
>>>>>>> org-mode and
>>>>>>> more often than not the link fails: I get taken to the 
>>>>>>> nnimap group that
>>>>>>> contains the message, but I get the error "No such article 
>>>>>>> (may have
>>>>>>> expired or been canceled)", even though the message is in 
>>>>>>> fact in the
>>>>>>> group.
>>>>>>>
>>>>>>> I suppose org-gnus.el might have to be amended to work 
>>>>>>> around this, but
>>>>>>> first I'd like to understand whether this is expected 
>>>>>>> behavior in gnus.
>>>>>>>
>>>>>>> Can anybody enlighten me or help me trouble-shoot this?
>>>>>>
>>>>>> No, it's not expected, and I haven't seen it before -- 
>>>>>> sorry for the
>>>>>> unhelpful reply! I just tried it (dovecot imap server), and 
>>>>>> it worked
>>>>>> both with and without angle brackets on the ID (sometimes 
>>>>>> brackets can
>>>>>> be an issue).
>>>>>>
>>>>>> To be sure everyone's on the same page, how exactly are you 
>>>>>> searching?
>>>>>> If you're in the *Group* buffer and you hit "G G" on the 
>>>>>> group, and
>>>>>> paste in the message ID, is the message found? If you're 
>>>>>> already in the
>>>>>> *Summary* buffer, there isn't really any way to "search" as 
>>>>>> such,
>>>>>> there's only "limiting" what messages are shown by some 
>>>>>> criteria -- in
>>>>>> those cases it's true you won't be able to get to unseen 
>>>>>> messages.
>>>>>>
>>>>>> But that should never affect following links from org 
>>>>>> mode...
>>>>>>
>>>>>> Eric
>>>>>
>>>>> Thanks for the reply, Eric. More details then.
>>>>>
>>>>> 1. GG in the group buffer works (after a brief delay, 
>>>>> displaying
>>>>> "Opening server FastmailLocal" and then "Searching
>>>>> nnimap+FastmailLocal:Archive...done").
>>>>>
>>>>> 2. M-^ ("Fetch article with id ...) in the summary buffer 
>>>>> (a) works when
>>>>> the message is visible, (b) fails when the message isn't 
>>>>> visible (with
>>>>> "No such article (may have expired or been canceled)").
>>>>>
>>>>> 3. When I follow the org link, say
>>>>> "gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", 
>>>>> I do get
>>>>> taken to the right nnimap group in gnus but the article is 
>>>>> not displayed
>>>>> (again, with "No such article (may have expired or been 
>>>>> canceled)").
>>>>>
>>>>> One funny thing is that when the "search" fails (in both 
>>>>> cases 2b and
>>>>> 3), the article summary actually is visible (see screenshot:
>>>>> https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)
>>>>>
>>>>> If 1 and 2 are as expected, then I guess the issue is with 
>>>>> org-gnus and
>>>>> how it interacts with my IMAP set-up (offlineimap + dovecot 
>>>>> + nnimap)?
>>>>>
>>>>> -- Kai.
>>>>
>>>> Some more follow-up. I tried to understand the code in 
>>>> org-gnus.el and
>>>> org-sum.el. It appears that the functions "org-gnus-open" and
>>>> "org-gnus-follow-link" call the function 
>>>> "gnus-summary-goto-article"
>>>> from gnus-sum.el. The latter is called with the optional 
>>>> arguments "nil"
>>>> and "t". The last one is supposed to force that all articles 
>>>> are loaded,
>>>> but that doesn't seem to happen.
>>>>
>>>> When I call (gnus-summary-goto-article 
>>>> "56614DE9.5030300@upf.edu" nil t)
>>>> in the summary buffer of the relevant group, it finds the 
>>>> message when
>>>> it is listed and doesn't find it if the buffer has only the 
>>>> first 200
>>>> articles (loading all articles with "/o" then makes the 
>>>> function
>>>> succeed).
>>>>
>>>> I added (gnus-large-newsgroup nil) to the group parameters, 
>>>> which then
>>>> loads all articles when entering the group, and
>>>> (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t) 
>>>> then
>>>> succeeds in the summary buffer.
>>>>
>>>> However, the gnus link still doesn't work from outside gnus. 
>>>> It's as if
>>>> the FORCE optional argument that is given to 
>>>> "gnus-summary-goto-article"
>>>> has no effect.
>>>
>>> I've got the same setup as you (Gnus and local dovecot), but 
>>> perhaps the
>>> versions are different? Can you tell us your versions for Gnus 
>>> and Org?
>>>
>>> When I step through `gnus-summary-goto-article', searching for 
>>> a message
>>> ID that is *not* currently display, the "force" argument never 
>>> even
>>> comes into play. I end up in this branch:
>>>
>>> (if (and (stringp article)
>>> 	       (string-match "@\\|%40" article))
>>> 	  (gnus-summary-refer-article article)
>>>
>>> So then on to `gnus-summary-refer-article' (no force argument 
>>> used), and
>>> in that function I end up in the "t" branch of the "cond" 
>>> later on, and
>>> eventually into "(gnus-summary-insert-subject message-id)". 
>>> Then that
>>> leads to "(gnus-read-header id)", and all is well.
>>>
>>> I know this is tedious, but would you mind stepping through 
>>> with edebug
>>> and seeing where things fall apart? My off-the-cuff guess is 
>>> that you're
>>> using an older version of Gnus where the nnimap backend isn't 
>>> quite all
>>> put together...
>>>
>>> If you aren't comfortable with edebug let me know and I can 
>>> provide
>>> exact instructions. It's a very useful thing to know!
>>>
>>> Eric
>>
>> #### System Info
>> - OS: darwin
>> - Emacs: 24.5.1
>> - Gnus: v5.13
>>
>> Well, that was fun ... sort of. I found something I can tweak 
>> to make it
>> work, but I'm not clear on why this issue arises.
>>
>> At the relevant point, the article id is 
>> "<56614DE9.5030300@upf.edu>"
>> (the <> having been added by gnus-summary-refer-article) and 
>> the group
>> is "nnimap+FastmailLocal:Archive".
>>
>> In `gnus-request-head (article group)', a funcall to 
>> `nnimap-request-head'
>> is constructed. Because `gnus-override-method' at that point 
>> contains a long
>> construct `(nnimap "FastmailLocal" ...)', the group argument of 
>> the call
>> to nnimap-request-head is set to nil rather than to "Archive". 
>> I take
>> it, the idea is that nnimap should be able to return the 
>> correct group
>> on its own?
>>
>> The relevant code (in gnus-int.el) is
>>
>> (setq res (funcall head article
>>                          (and (not gnus-override-method)
>>                               (gnus-group-real-name group))
>>        (nth 1 gnus-command-method)))
>>
>> Now, `nnimap-request-head (article nil "FastmailLocal")' 
>> returns "(nil .
>> 277)". I believe the "nil" in that cons should be the group 
>> ("Archive").
>> Maybe that's a problem in nnimap.el but looking at the code I 
>> don't see
>> how it could ever return anything other than nil.
>>
>> From that point on, the group info seems lost and the message 
>> can't be
>> targeted.
>>
>> If I revert the commit to gnus-int.el that introduced the 
>> override
>> method thingy
>> (http://git.gnus.org/cgit/gnus.git/commit/?id=8ba1cd0b96466a96265ec5336728519aa6030d83),
>> to the following instead of what's above:
>>
>> (setq res (funcall head article (gnus-group-real-name group)
>> 			 (nth 1 gnus-command-method))))
>>
>> my messages get found because the actual group is passed on to
>> nnimap-request-head.
>>
>> So, I have things working to a certain degree, but I'm utterly 
>> confused
>> about why this is happening.
>>
>> Can you see from this what the problem is?
>
> My guess is it might just be time to update Gnus. Nnimap has 
> undergone a
> lot of changes over the past couple of years, as has the rest of 
> Gnus,
> and while the versioning system has always confused me, I'm 
> pretty sure
> v5.13 is several years old. The code branches you seem to be 
> following
> don't match up with what I'm seeing (in git master, aka Ma 
> Gnus), and
> it's probably not worth trying to figure out exactly what's 
> going on. I
> find Ma Gnus pretty stable -- would you be willing to run from 
> git?
>
> Eric

OK, I switched to Ma Gnus from git master (`gnu-version' now 
reports "Ma
Gnus v0.14)").

No luck, same issue. And reverting `gnus-request-head' in 
gnus-int.el to
the version without gnus-override-method makes the links work. I 
will
run through it all again with edebug (which unfortunately is 
super-slow
here) over the holiday break and see whether I can determine more
precisely where my installation fails. I'm very puzzled that your 
pretty
much identical set-up works.

-- Kai.

Re: Finding message by ID fails when message not visible

[Eric Abrahamsen]

Kai von Fintel <fintel@mit.edu> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Kai von Fintel <fintel@mit.edu> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>
>>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>>
>>>>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>>>>
>>>>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>>>>
>>>>>>>> I'm puzzled by the following behavior: if I search in an
>>>>>>>> nnimap group
>>>>>>>> for a message with a particular message-ID, the search fails
>>>>>>>> unless the
>>>>>>>> message is visible. If for example, only new messages are
>>>>>>>> shown, the
>>>>>>>> search for an old message with a particular ID fails but the
>>>>>>>> same
>>>>>>>> message is found when "/o" has made the old messages appear.
>>>>>>>>
>>>>>>>> This is relevant because I use links to gnus messages from
>>>>>>>> org-mode and
>>>>>>>> more often than not the link fails: I get taken to the nnimap
>>>>>>>> group that
>>>>>>>> contains the message, but I get the error "No such article
>>>>>>>> (may have
>>>>>>>> expired or been canceled)", even though the message is in fact
>>>>>>>> in the
>>>>>>>> group.
>>>>>>>>
>>>>>>>> I suppose org-gnus.el might have to be amended to work around
>>>>>>>> this, but
>>>>>>>> first I'd like to understand whether this is expected behavior
>>>>>>>> in gnus.
>>>>>>>>
>>>>>>>> Can anybody enlighten me or help me trouble-shoot this?
>>>>>>>
>>>>>>> No, it's not expected, and I haven't seen it before -- 
>>>>>>> sorry for the
>>>>>>> unhelpful reply! I just tried it (dovecot imap server), and it
>>>>>>> worked
>>>>>>> both with and without angle brackets on the ID (sometimes
>>>>>>> brackets can
>>>>>>> be an issue).
>>>>>>>
>>>>>>> To be sure everyone's on the same page, how exactly are you
>>>>>>> searching?
>>>>>>> If you're in the *Group* buffer and you hit "G G" on the group,
>>>>>>> and
>>>>>>> paste in the message ID, is the message found? If you're
>>>>>>> already in the
>>>>>>> *Summary* buffer, there isn't really any way to "search" as
>>>>>>> such,
>>>>>>> there's only "limiting" what messages are shown by some
>>>>>>> criteria -- in
>>>>>>> those cases it's true you won't be able to get to unseen
>>>>>>> messages.
>>>>>>>
>>>>>>> But that should never affect following links from org mode...
>>>>>>>
>>>>>>> Eric
>>>>>>
>>>>>> Thanks for the reply, Eric. More details then.
>>>>>>
>>>>>> 1. GG in the group buffer works (after a brief delay, displaying
>>>>>> "Opening server FastmailLocal" and then "Searching
>>>>>> nnimap+FastmailLocal:Archive...done").
>>>>>>
>>>>>> 2. M-^ ("Fetch article with id ...) in the summary buffer (a)
>>>>>> works when
>>>>>> the message is visible, (b) fails when the message isn't visible
>>>>>> (with
>>>>>> "No such article (may have expired or been canceled)").
>>>>>>
>>>>>> 3. When I follow the org link, say
>>>>>> "gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", I
>>>>>> do get
>>>>>> taken to the right nnimap group in gnus but the article is not
>>>>>> displayed
>>>>>> (again, with "No such article (may have expired or been
>>>>>> canceled)").
>>>>>>
>>>>>> One funny thing is that when the "search" fails (in both cases
>>>>>> 2b and
>>>>>> 3), the article summary actually is visible (see screenshot:
>>>>>> https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)
>>>>>>
>>>>>> If 1 and 2 are as expected, then I guess the issue is with
>>>>>> org-gnus and
>>>>>> how it interacts with my IMAP set-up (offlineimap + dovecot +
>>>>>> nnimap)?
>>>>>>
>>>>>> -- Kai.
>>>>>
>>>>> Some more follow-up. I tried to understand the code in
>>>>> org-gnus.el and
>>>>> org-sum.el. It appears that the functions "org-gnus-open" and
>>>>> "org-gnus-follow-link" call the function
>>>>> "gnus-summary-goto-article"
>>>>> from gnus-sum.el. The latter is called with the optional
>>>>> arguments "nil"
>>>>> and "t". The last one is supposed to force that all articles are
>>>>> loaded,
>>>>> but that doesn't seem to happen.
>>>>>
>>>>> When I call (gnus-summary-goto-article "56614DE9.5030300@upf.edu"
>>>>> nil t)
>>>>> in the summary buffer of the relevant group, it finds the message
>>>>> when
>>>>> it is listed and doesn't find it if the buffer has only the first
>>>>> 200
>>>>> articles (loading all articles with "/o" then makes the function
>>>>> succeed).
>>>>>
>>>>> I added (gnus-large-newsgroup nil) to the group parameters, which
>>>>> then
>>>>> loads all articles when entering the group, and
>>>>> (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t) then
>>>>> succeeds in the summary buffer.
>>>>>
>>>>> However, the gnus link still doesn't work from outside gnus. It's
>>>>> as if
>>>>> the FORCE optional argument that is given to
>>>>> "gnus-summary-goto-article"
>>>>> has no effect.
>>>>
>>>> I've got the same setup as you (Gnus and local dovecot), but
>>>> perhaps the
>>>> versions are different? Can you tell us your versions for Gnus and
>>>> Org?
>>>>
>>>> When I step through `gnus-summary-goto-article', searching for a
>>>> message
>>>> ID that is *not* currently display, the "force" argument never
>>>> even
>>>> comes into play. I end up in this branch:
>>>>
>>>> (if (and (stringp article)
>>>> 	       (string-match "@\\|%40" article))
>>>> 	  (gnus-summary-refer-article article)
>>>>
>>>> So then on to `gnus-summary-refer-article' (no force argument
>>>> used), and
>>>> in that function I end up in the "t" branch of the "cond" later
>>>> on, and
>>>> eventually into "(gnus-summary-insert-subject message-id)". Then
>>>> that
>>>> leads to "(gnus-read-header id)", and all is well.
>>>>
>>>> I know this is tedious, but would you mind stepping through with
>>>> edebug
>>>> and seeing where things fall apart? My off-the-cuff guess is that
>>>> you're
>>>> using an older version of Gnus where the nnimap backend isn't
>>>> quite all
>>>> put together...
>>>>
>>>> If you aren't comfortable with edebug let me know and I can
>>>> provide
>>>> exact instructions. It's a very useful thing to know!
>>>>
>>>> Eric
>>>
>>> #### System Info
>>> - OS: darwin
>>> - Emacs: 24.5.1
>>> - Gnus: v5.13
>>>
>>> Well, that was fun ... sort of. I found something I can tweak to
>>> make it
>>> work, but I'm not clear on why this issue arises.
>>>
>>> At the relevant point, the article id is
>>> "<56614DE9.5030300@upf.edu>"
>>> (the <> having been added by gnus-summary-refer-article) and the
>>> group
>>> is "nnimap+FastmailLocal:Archive".
>>>
>>> In `gnus-request-head (article group)', a funcall to
>>> `nnimap-request-head'
>>> is constructed. Because `gnus-override-method' at that point
>>> contains a long
>>> construct `(nnimap "FastmailLocal" ...)', the group argument of the
>>> call
>>> to nnimap-request-head is set to nil rather than to "Archive". I
>>> take
>>> it, the idea is that nnimap should be able to return the correct
>>> group
>>> on its own?
>>>
>>> The relevant code (in gnus-int.el) is
>>>
>>> (setq res (funcall head article
>>>                          (and (not gnus-override-method)
>>>                               (gnus-group-real-name group))
>>>        (nth 1 gnus-command-method)))
>>>
>>> Now, `nnimap-request-head (article nil "FastmailLocal")' returns
>>> "(nil .
>>> 277)". I believe the "nil" in that cons should be the group
>>> ("Archive").
>>> Maybe that's a problem in nnimap.el but looking at the code I don't
>>> see
>>> how it could ever return anything other than nil.
>>>
>>> From that point on, the group info seems lost and the message can't
>>> be
>>> targeted.
>>>
>>> If I revert the commit to gnus-int.el that introduced the override
>>> method thingy
>>> (http://git.gnus.org/cgit/gnus.git/commit/?id=8ba1cd0b96466a96265ec5336728519aa6030d83),
>>> to the following instead of what's above:
>>>
>>> (setq res (funcall head article (gnus-group-real-name group)
>>> 			 (nth 1 gnus-command-method))))
>>>
>>> my messages get found because the actual group is passed on to
>>> nnimap-request-head.
>>>
>>> So, I have things working to a certain degree, but I'm utterly
>>> confused
>>> about why this is happening.
>>>
>>> Can you see from this what the problem is?
>>
>> My guess is it might just be time to update Gnus. Nnimap has
>> undergone a
>> lot of changes over the past couple of years, as has the rest of
>> Gnus,
>> and while the versioning system has always confused me, I'm pretty
>> sure
>> v5.13 is several years old. The code branches you seem to be
>> following
>> don't match up with what I'm seeing (in git master, aka Ma Gnus),
>> and
>> it's probably not worth trying to figure out exactly what's going
>> on. I
>> find Ma Gnus pretty stable -- would you be willing to run from git?
>>
>> Eric
>
> OK, I switched to Ma Gnus from git master (`gnu-version' now reports
> "Ma
> Gnus v0.14)").
>
> No luck, same issue. And reverting `gnus-request-head' in gnus-int.el
> to
> the version without gnus-override-method makes the links work. I will
> run through it all again with edebug (which unfortunately is
> super-slow
> here) over the holiday break and see whether I can determine more
> precisely where my installation fails. I'm very puzzled that your
> pretty
> much identical set-up works.

Well, let me know how that goes. I just stepped through a couple of
link-following processes, and can tell you that the call to
gnus-request-head/nnimap-request-head also returns a cons of (nil
. 12345) here. By that point we're already in the proper Summary buffer,
so I don't think it matters that the group name isn't in there.

I notice that gnus-summary-insert-subject actually gets called twice,
with the same "id" argument, and only inserts the correct article on the
second pass, but I haven't gone so far as to figure out why that
happens. In the let* construct at the top of the function, this branch:

(t
  (gnus-read-header id))

Returns a full header vector (with article number, and the various
Subject/From headers) for the article I'm looking for.

Anyway, there are a few more points of data to work with, hope you can
sort it out...

Eric