Re: Finding message by ID fails when message not visible

[Kai von Fintel <fintel@mit.edu>]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Kai von Fintel <fintel@mit.edu> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Kai von Fintel <fintel@mit.edu> writes:
>>>
>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>
>>>>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>>>>
>>>>>> Kai von Fintel <fintel@mit.edu> writes:
>>>>>>
>>>>>>> I'm puzzled by the following behavior: if I search in an 
>>>>>>> nnimap group
>>>>>>> for a message with a particular message-ID, the search 
>>>>>>> fails unless the
>>>>>>> message is visible. If for example, only new messages are 
>>>>>>> shown, the
>>>>>>> search for an old message with a particular ID fails but 
>>>>>>> the same
>>>>>>> message is found when "/o" has made the old messages 
>>>>>>> appear.
>>>>>>>
>>>>>>> This is relevant because I use links to gnus messages from 
>>>>>>> org-mode and
>>>>>>> more often than not the link fails: I get taken to the 
>>>>>>> nnimap group that
>>>>>>> contains the message, but I get the error "No such article 
>>>>>>> (may have
>>>>>>> expired or been canceled)", even though the message is in 
>>>>>>> fact in the
>>>>>>> group.
>>>>>>>
>>>>>>> I suppose org-gnus.el might have to be amended to work 
>>>>>>> around this, but
>>>>>>> first I'd like to understand whether this is expected 
>>>>>>> behavior in gnus.
>>>>>>>
>>>>>>> Can anybody enlighten me or help me trouble-shoot this?
>>>>>>
>>>>>> No, it's not expected, and I haven't seen it before -- 
>>>>>> sorry for the
>>>>>> unhelpful reply! I just tried it (dovecot imap server), and 
>>>>>> it worked
>>>>>> both with and without angle brackets on the ID (sometimes 
>>>>>> brackets can
>>>>>> be an issue).
>>>>>>
>>>>>> To be sure everyone's on the same page, how exactly are you 
>>>>>> searching?
>>>>>> If you're in the *Group* buffer and you hit "G G" on the 
>>>>>> group, and
>>>>>> paste in the message ID, is the message found? If you're 
>>>>>> already in the
>>>>>> *Summary* buffer, there isn't really any way to "search" as 
>>>>>> such,
>>>>>> there's only "limiting" what messages are shown by some 
>>>>>> criteria -- in
>>>>>> those cases it's true you won't be able to get to unseen 
>>>>>> messages.
>>>>>>
>>>>>> But that should never affect following links from org 
>>>>>> mode...
>>>>>>
>>>>>> Eric
>>>>>
>>>>> Thanks for the reply, Eric. More details then.
>>>>>
>>>>> 1. GG in the group buffer works (after a brief delay, 
>>>>> displaying
>>>>> "Opening server FastmailLocal" and then "Searching
>>>>> nnimap+FastmailLocal:Archive...done").
>>>>>
>>>>> 2. M-^ ("Fetch article with id ...) in the summary buffer 
>>>>> (a) works when
>>>>> the message is visible, (b) fails when the message isn't 
>>>>> visible (with
>>>>> "No such article (may have expired or been canceled)").
>>>>>
>>>>> 3. When I follow the org link, say
>>>>> "gnus:nnimap+FastmailLocal:Archive#56614DE9.5030300@upf.edu", 
>>>>> I do get
>>>>> taken to the right nnimap group in gnus but the article is 
>>>>> not displayed
>>>>> (again, with "No such article (may have expired or been 
>>>>> canceled)").
>>>>>
>>>>> One funny thing is that when the "search" fails (in both 
>>>>> cases 2b and
>>>>> 3), the article summary actually is visible (see screenshot:
>>>>> https://www.dropbox.com/s/b1b9mxi7bl0a51j/missing-article.jpg?dl=0)
>>>>>
>>>>> If 1 and 2 are as expected, then I guess the issue is with 
>>>>> org-gnus and
>>>>> how it interacts with my IMAP set-up (offlineimap + dovecot 
>>>>> + nnimap)?
>>>>>
>>>>> -- Kai.
>>>>
>>>> Some more follow-up. I tried to understand the code in 
>>>> org-gnus.el and
>>>> org-sum.el. It appears that the functions "org-gnus-open" and
>>>> "org-gnus-follow-link" call the function 
>>>> "gnus-summary-goto-article"
>>>> from gnus-sum.el. The latter is called with the optional 
>>>> arguments "nil"
>>>> and "t". The last one is supposed to force that all articles 
>>>> are loaded,
>>>> but that doesn't seem to happen.
>>>>
>>>> When I call (gnus-summary-goto-article 
>>>> "56614DE9.5030300@upf.edu" nil t)
>>>> in the summary buffer of the relevant group, it finds the 
>>>> message when
>>>> it is listed and doesn't find it if the buffer has only the 
>>>> first 200
>>>> articles (loading all articles with "/o" then makes the 
>>>> function
>>>> succeed).
>>>>
>>>> I added (gnus-large-newsgroup nil) to the group parameters, 
>>>> which then
>>>> loads all articles when entering the group, and
>>>> (gnus-summary-goto-article "56614DE9.5030300@upf.edu" nil t) 
>>>> then
>>>> succeeds in the summary buffer.
>>>>
>>>> However, the gnus link still doesn't work from outside gnus. 
>>>> It's as if
>>>> the FORCE optional argument that is given to 
>>>> "gnus-summary-goto-article"
>>>> has no effect.
>>>
>>> I've got the same setup as you (Gnus and local dovecot), but 
>>> perhaps the
>>> versions are different? Can you tell us your versions for Gnus 
>>> and Org?
>>>
>>> When I step through `gnus-summary-goto-article', searching for 
>>> a message
>>> ID that is *not* currently display, the "force" argument never 
>>> even
>>> comes into play. I end up in this branch:
>>>
>>> (if (and (stringp article)
>>> 	       (string-match "@\\|%40" article))
>>> 	  (gnus-summary-refer-article article)
>>>
>>> So then on to `gnus-summary-refer-article' (no force argument 
>>> used), and
>>> in that function I end up in the "t" branch of the "cond" 
>>> later on, and
>>> eventually into "(gnus-summary-insert-subject message-id)". 
>>> Then that
>>> leads to "(gnus-read-header id)", and all is well.
>>>
>>> I know this is tedious, but would you mind stepping through 
>>> with edebug
>>> and seeing where things fall apart? My off-the-cuff guess is 
>>> that you're
>>> using an older version of Gnus where the nnimap backend isn't 
>>> quite all
>>> put together...
>>>
>>> If you aren't comfortable with edebug let me know and I can 
>>> provide
>>> exact instructions. It's a very useful thing to know!
>>>
>>> Eric
>>
>> #### System Info
>> - OS: darwin
>> - Emacs: 24.5.1
>> - Gnus: v5.13
>>
>> Well, that was fun ... sort of. I found something I can tweak 
>> to make it
>> work, but I'm not clear on why this issue arises.
>>
>> At the relevant point, the article id is 
>> "<56614DE9.5030300@upf.edu>"
>> (the <> having been added by gnus-summary-refer-article) and 
>> the group
>> is "nnimap+FastmailLocal:Archive".
>>
>> In `gnus-request-head (article group)', a funcall to 
>> `nnimap-request-head'
>> is constructed. Because `gnus-override-method' at that point 
>> contains a long
>> construct `(nnimap "FastmailLocal" ...)', the group argument of 
>> the call
>> to nnimap-request-head is set to nil rather than to "Archive". 
>> I take
>> it, the idea is that nnimap should be able to return the 
>> correct group
>> on its own?
>>
>> The relevant code (in gnus-int.el) is
>>
>> (setq res (funcall head article
>>                          (and (not gnus-override-method)
>>                               (gnus-group-real-name group))
>>        (nth 1 gnus-command-method)))
>>
>> Now, `nnimap-request-head (article nil "FastmailLocal")' 
>> returns "(nil .
>> 277)". I believe the "nil" in that cons should be the group 
>> ("Archive").
>> Maybe that's a problem in nnimap.el but looking at the code I 
>> don't see
>> how it could ever return anything other than nil.
>>
>> From that point on, the group info seems lost and the message 
>> can't be
>> targeted.
>>
>> If I revert the commit to gnus-int.el that introduced the 
>> override
>> method thingy
>> (http://git.gnus.org/cgit/gnus.git/commit/?id=8ba1cd0b96466a96265ec5336728519aa6030d83),
>> to the following instead of what's above:
>>
>> (setq res (funcall head article (gnus-group-real-name group)
>> 			 (nth 1 gnus-command-method))))
>>
>> my messages get found because the actual group is passed on to
>> nnimap-request-head.
>>
>> So, I have things working to a certain degree, but I'm utterly 
>> confused
>> about why this is happening.
>>
>> Can you see from this what the problem is?
>
> My guess is it might just be time to update Gnus. Nnimap has 
> undergone a
> lot of changes over the past couple of years, as has the rest of 
> Gnus,
> and while the versioning system has always confused me, I'm 
> pretty sure
> v5.13 is several years old. The code branches you seem to be 
> following
> don't match up with what I'm seeing (in git master, aka Ma 
> Gnus), and
> it's probably not worth trying to figure out exactly what's 
> going on. I
> find Ma Gnus pretty stable -- would you be willing to run from 
> git?
>
> Eric

OK, I switched to Ma Gnus from git master (`gnu-version' now 
reports "Ma
Gnus v0.14)").

No luck, same issue. And reverting `gnus-request-head' in 
gnus-int.el to
the version without gnus-override-method makes the links work. I 
will
run through it all again with edebug (which unfortunately is 
super-slow
here) over the holiday break and see whether I can determine more
precisely where my installation fails. I'm very puzzled that your 
pretty
much identical set-up works.

-- Kai.