Re: nnir, gnus-goto-article and such

[Dave Abrahams <dave@boostpro.com>]

on Thu Sep 22 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:

>>>>>> "John" == John Wiegley <jwiegley@gmail.com> writes:
>
> [...]
>
>     John> What doesn't work is that 'A T' doesn't do everything it can
>     John> to recursively find all referenced articles from all
>     John> refer-article sources.  
>
> The logic used in 'A T' is different from that of 'A R' and friends
> because its doing something different. That's not to say that it doesn't
> have bugs, and I'm sure that it can be improved. 

Andrew, I think there's a miscommunication here.  John is talking about
what he sees as a design/operational deficiency, while you're talking
about the implementation.

For example, `A T' starts out with a call to `gnus-warp-to-article'.
However, `gnus-warp-to-article' for some reason only works in nnir
groups.  There are several other backends that aggregate messages from
other groups, but none of them remembers where the messages came from.
Even if I happen to have the article in the registry, and *it* knows
which group it came from, I won't end up warping there.  And I think
that's the cause of a lot of the grief John and I are experiencing.

>     John> In contrast, if you hand-select each article that 'A R' finds,
>     John> and then press 'A R' on that, and so on recursively, you will
>     John> ultimately end up with the entire thread.  
>
> I don't think this gives the entire thread. It only gives the articles
> that precede the article you start with along the current branch. 

...and does `A T' always do *at least* that well?  It should, IMO.

> If
> there are subsequent articles along the current branch, or articles on
> another branch, this recursive procedure won't find them. That's why the
> 'A T' function tries to do a /search/ for articles rather than just
> recursive retrieval. This is complicated by the limitations of the
> different kinds of servers. 

Right... you know, I could swear that wanderlust lets me search in nntp
groups.  It has a single uniform interface to these searches across all
backends.  *looks it up* yeah, it's called a filter folder.  I wonder
how that works?

Aha, it looks like maybe they use the server for everything but body
searches (which we don't care about for thread-following)?  I'm not
entirely sure.

--8<---------------cut here---------------start------------->8---
(defun elmo-nntp-use-server-search-p (condition)
  (if (vectorp condition)
      (not (string= "body" (elmo-filter-key condition)))
    (and (elmo-nntp-use-server-search-p (nth 1 condition))
	 (elmo-nntp-use-server-search-p (nth 2 condition)))))
--8<---------------cut here---------------end--------------->8---


> Here is the complete `A T' logic for nntp and nnimap groups:
>
> 1. nntp: The nntp protocol has no searching capabilities at all, so
>    there is no way to search by message-id. Gmane has a Xapian-powered
>    search engine written by Olly Betts that works very well, but doesn't
>    allow searching by message-id. 

If it does full-text searches you could narrow the field by searching
for the message-id anyway.

> What gnus does in this case is very primitive: it retrieves the
> headers for a (configurable) number of articles before and after the
> current article (250 before and 250 after by default) in the current
> group, and then gnus itself searches within these headers for articles
> belonging to the thread. But there is no way to know if there are
> other articles in the thread in other groups or on other servers.
>
> 2. nnimap: The imap protocol allows searching by headers, but only
>    within a single group. So by default if you `A T' on an article in an
>    nnimap group a search will be performed for all the articles that
>    might be in the thread in the current group only. 
>
>    Recently I added the ability to replace this search with an nnir
>    search which will look for articles in the thread on the whole server
>    rather than just in the current group. But it won't look outside the
>    server. 

Right... we'd like it to use all the resources that could be at its
disposal; that is, everything in gnus-refer-article-method.

>    I have also recently re-written much of nnir, but I won't push it
>    until after the freeze is over. As part of this re-write, when
>    searching for a thread on the whole server the registry is consulted
>    to see what group the article on which you invoke `A T' came from,
>    and consult /both/ that server and the server containing the current
>    group. This is a very minor improvement, since most of the time these
>    two servers will be the same. 

Except in my case, where I'm creating an ephemeral group to hold the
referenced article.  I think that's exactly what I want right now.

> I guess we can consult the registry for all the articles mentioned in
> the references header and see if any of those articles came from
> different servers and search those servers as well. But if they are nntp
> servers we can't really search them effectively (since nntp doesn't have
> search facilities). 

Wow; we can't pull up a message by ID from NNTP?  too bad.

> We could also configure this to /always/ search a particular set of
> servers independently of where the proximate article came from.

IMO, use `gnus-refer-article-method'; it already answers the question
:-)

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com