Re: Finding message by ID fails when message not visible

[Kai von Fintel <fintel@mit.edu>]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Kai von Fintel <fintel@mit.edu> writes:
>
>> Kai von Fintel <fintel@mit.edu> writes:
>>
>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>
>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>
>>>>> I'm puzzled by the following behavior: if I search in an nnimap group
>>>>> for a message with a particular message-ID, the search fails unless the
>>>>> message is visible. If for example, only new messages are shown, the
>>>>> search for an old message with a particular ID fails but the same
>>>>> message is found when "/o" has made the old messages appear.
>>>>>
>>>>> This is relevant because I use links to gnus messages from org-mode and
>>>>> more often than not the link fails: I get taken to the nnimap group that
>>>>> contains the message, but I get the error "No such article (may have
>>>>> expired or been canceled)", even though the message is in fact in the
>>>>> group.
>>>>>
>>>>> I suppose org-gnus.el might have to be amended to work around this, but
>>>>> first I'd like to understand whether this is expected behavior in gnus.
>>>>>
>>>>> Can anybody enlighten me or help me trouble-shoot this?
>>>>
>>>> No, it's not expected, and I haven't seen it before -- sorry for the
>>>> unhelpful reply! I just tried it (dovecot imap server), and it worked
>>>> both with and without angle brackets on the ID (sometimes brackets can
>>>> be an issue).
>>>>
>>>> To be sure everyone's on the same page, how exactly are you searching?
>>>> If you're in the *Group* buffer and you hit "G G" on the group, and
>>>> paste in the message ID, is the message found? If you're already in the
>>>> *Summary* buffer, there isn't really any way to "search" as such,
>>>> there's only "limiting" what messages are shown by some criteria -- in
>>>> those cases it's true you won't be able to get to unseen messages.
>>>>
>>>> But that should never affect following links from org mode...
>>>>
>>>> Eric
>>>
>>> Thanks for the reply, Eric. More details then.
>>>
>>> 1. GG in the group buffer works (after a brief delay, displaying
>>> "Opening server FastmailLocal" and then "Searching
>>> nnimap+FastmailLocal:Archive...done").
>>>
>>> 2. M-^ ("Fetch article with id ...) in the summary buffer (a) works when
>>> the message is visible, (b) fails when the message isn't visible (with
>>> "No such article (may have expired or been canceled)").
>>>
>>> 3. When I follow the org link, say
>>> "gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", I do get
>>> taken to the right nnimap group in gnus but the article is not displayed
>>> (again, with "No such article (may have expired or been canceled)").
>>>
>>> One funny thing is that when the "search" fails (in both cases 2b and
>>> 3), the article summary actually is visible (see screenshot:
>>> https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)
>>>
>>> If 1 and 2 are as expected, then I guess the issue is with org-gnus and
>>> how it interacts with my IMAP set-up (offlineimap + dovecot + nnimap)?
>>>
>>> -- Kai.
>>
>> Some more follow-up. I tried to understand the code in org-gnus.el and
>> org-sum.el. It appears that the functions "org-gnus-open" and
>> "org-gnus-follow-link" call the function "gnus-summary-goto-article"
>> from gnus-sum.el. The latter is called with the optional arguments "nil"
>> and "t". The last one is supposed to force that all articles are loaded,
>> but that doesn't seem to happen.
>>
>> When I call (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t)
>> in the summary buffer of the relevant group, it finds the message when
>> it is listed and doesn't find it if the buffer has only the first 200
>> articles (loading all articles with "/o" then makes the function
>> succeed).
>>
>> I added (gnus-large-newsgroup nil) to the group parameters, which then
>> loads all articles when entering the group, and
>> (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t) then
>> succeeds in the summary buffer.
>>
>> However, the gnus link still doesn't work from outside gnus. It's as if
>> the FORCE optional argument that is given to "gnus-summary-goto-article"
>> has no effect.
>
> I've got the same setup as you (Gnus and local dovecot), but perhaps the
> versions are different? Can you tell us your versions for Gnus and Org?
>
> When I step through `gnus-summary-goto-article', searching for a message
> ID that is *not* currently display, the "force" argument never even
> comes into play. I end up in this branch:
>
> (if (and (stringp article)
> 	       (string-match "@\\|%40" article))
> 	  (gnus-summary-refer-article article)
>
> So then on to `gnus-summary-refer-article' (no force argument used), and
> in that function I end up in the "t" branch of the "cond" later on, and
> eventually into "(gnus-summary-insert-subject message-id)". Then that
> leads to "(gnus-read-header id)", and all is well.
>
> I know this is tedious, but would you mind stepping through with edebug
> and seeing where things fall apart? My off-the-cuff guess is that you're
> using an older version of Gnus where the nnimap backend isn't quite all
> put together...
>
> If you aren't comfortable with edebug let me know and I can provide
> exact instructions. It's a very useful thing to know!
>
> Eric

#### System Info
- OS: darwin
- Emacs: 24.5.1
- Gnus: v5.13

Well, that was fun ... sort of. I found something I can tweak to make it
work, but I'm not clear on why this issue arises.

At the relevant point, the article id is "<56614DE9.5030300@upf.edu>"
(the <> having been added by gnus-summary-refer-article) and the group
is "nnimap+FastmailLocal:Archive".

In `gnus-request-head (article group)', a funcall to `nnimap-request-head'
is constructed. Because `gnus-override-method' at that point contains a long
construct `(nnimap "FastmailLocal" ...)', the group argument of the call
to nnimap-request-head is set to nil rather than to "Archive". I take
it, the idea is that nnimap should be able to return the correct group
on its own?

The relevant code (in gnus-int.el) is

(setq res (funcall head article
                         (and (not gnus-override-method)
                              (gnus-group-real-name group))
       (nth 1 gnus-command-method)))

Now, `nnimap-request-head (article nil "FastmailLocal")' returns "(nil .
277)". I believe the "nil" in that cons should be the group ("Archive").
Maybe that's a problem in nnimap.el but looking at the code I don't see
how it could ever return anything other than nil.

From that point on, the group info seems lost and the message can't be
targeted.

If I revert the commit to gnus-int.el that introduced the override
method thingy
(http://git.gnus.org/cgit/gnus.git/commit/?id=8ba1cd0b96466a96265ec5336728519aa6030d83),
to the following instead of what's above:

(setq res (funcall head article (gnus-group-real-name group)
			 (nth 1 gnus-command-method))))

my messages get found because the actual group is passed on to
nnimap-request-head.

So, I have things working to a certain degree, but I'm utterly confused
about why this is happening.

Can you see from this what the problem is?

-- Kai.