`gnus-refer-article-methods' and `gnus-summary-refer-thread'

`gnus-refer-article-methods' and `gnus-summary-refer-thread'

[Dave Abrahams]

It's a bit curious that when I'm sitting on the root of a thread, `A T'
doesn't find the rest of it via nnir when I have that in
`gnus-refer-article-methods'.  I think I can guess why that is, but it
seems more like a programming artifact than a design decision.
Shouldn't it search all the same methods for the message-ID?

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: `gnus-refer-article-methods' and `gnus-summary-refer-thread'

[Andrew Cohen]

>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:

    Dave> It's a bit curious that when I'm sitting on the root of a
    Dave> thread, `A T' doesn't find the rest of it via nnir when I have
    Dave> that in `gnus-refer-article-methods'.  

`A T' (gnus-summary-refer-thread) doesn't use gnus-refer-article-methods
at all. This is distinct from `^` (gnus-summary-refer-parent-article)
which consults the entries in `gnus-refer-article-methods' in order
until the parent is found.

For nnimap groups `A T' searches /in the current group/ for any article
that is referenced in the primary article, or refers to the primary
article or any of its references. In rare cases where mailers have not
properly coordinated references it is possible for this to miss
articles, but it should be rare and avoiding it would require recursing
searches which would be very slow.

What it does /not/ do is search across multiple groups, and this is the
most likely reason for not finding all articles within a thread.  The
current function adds the found articles from the current group into the
summary buffer. Nnir creates an ephemeral group which can contain
articles from multiple groups which is substantively different.  Having
said that I am just about to check in a modification that will allow
searching across multiple groups through the use of a prefix-arg or
setting a variable. Rather than calling the current
`gnus-summary-refer-thread' this will construct the same imap search
pattern but use nnir to search across the entire server and create an
ephemeral group with the results.

HTH,
Andy

Re: `gnus-refer-article-methods' and `gnus-summary-refer-thread'

[Dave Abrahams]

on Fri Jul 22 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:

>>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:
>
>     Dave> It's a bit curious that when I'm sitting on the root of a
>     Dave> thread, `A T' doesn't find the rest of it via nnir when I have
>     Dave> that in `gnus-refer-article-methods'.  
>
> `A T' (gnus-summary-refer-thread) doesn't use gnus-refer-article-methods
> at all. This is distinct from `^` (gnus-summary-refer-parent-article)
> which consults the entries in `gnus-refer-article-methods' in order
> until the parent is found.

Right.  Like I said in the part of my post you didn't cite, I think I
understand why it's that way, but it seems like a programming artifact
rather than a design decision.

> For nnimap groups `A T' searches /in the current group/ for any article
> that is referenced in the primary article, or refers to the primary
> article or any of its references. In rare cases where mailers have not
> properly coordinated references it is possible for this to miss
> articles, but it should be rare and avoiding it would require recursing
> searches which would be very slow.

I'm not trying to deal with the case of improperly-coordinated
references.  I delete/expire things from my INBOX but sometimes want to
get to the parent article of a message, which is certainly in my "All
Mail" group (cool, your nnir features make that work for me!) or
reconstruct the thread attached to a message (bummer, no dice).  These
seem—from a user's point-of-view—like highly similar operations, and I
don't see why the article searches shouldn't proceed through all the
same means.

> What it does /not/ do is search across multiple groups, and this is the
> most likely reason for not finding all articles within a thread.  

Yes, I'm aware that's why it's not finding the article.

> The current function adds the found articles from the current group
> into the summary buffer. Nnir creates an ephemeral group which can
> contain articles from multiple groups which is substantively
> different.

Yes, but when I use `^' it can use nnir to insert articles that don't
exist in the current group into the current group's summary
buffer... which would is substantively similar to what I'm asking for ;-)

> Having said that I am just about to check in a
> modification that will allow searching across multiple groups through
> the use of a prefix-arg or setting a variable. Rather than calling the
> current `gnus-summary-refer-thread' this will construct the same imap
> search pattern but use nnir to search across the entire server and
> create an ephemeral group with the results.

I'm not yet sure I understand what you're describing.

For Wanderlust I co-wrote some code that, given a message, will pop up
that message's whole thread (with the source message selected) in an
ephemeral group
<https://github.com/dabrahams/dwamacs/blob/master/site-lisp/wl-conversation.el>.
That was very useful to me and would be happy to use something just like
that instead of the current `A T' functionality, if that's what you're
proposing.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: `gnus-refer-article-methods' and `gnus-summary-refer-thread'

[Andrew Cohen]

>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:

    Dave> on Fri Jul 22 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:
    >>>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:
    >> 
    Dave> It's a bit curious that when I'm sitting on the root of a
    Dave> thread, `A T' doesn't find the rest of it via nnir when I have
    Dave> that in `gnus-refer-article-methods'.
    >> 
    >> `A T' (gnus-summary-refer-thread) doesn't use
    >> gnus-refer-article-methods at all. This is distinct from `^`
    >> (gnus-summary-refer-parent-article) which consults the entries in
    >> `gnus-refer-article-methods' in order until the parent is found.

    Dave> Right.  Like I said in the part of my post you didn't cite, I
    Dave> think I understand why it's that way, but it seems like a
    Dave> programming artifact rather than a design decision.

Finding the parent is finding a single article with a given
Message-ID. Once found we know to stop. Since gnus has no idea how many
articles are in a thread it doesn't know when it should stop. So looping
through a set of methods doesn't make much sense to me. Instead, we just
try a single method, which is the search I described. Aside from broken
mailers, this will find the whole thread within the given group. I'm
still not entirely clear, but I think your only unhappiness is the
restriction to searching within the group rather than across all groups
on the server. Is this right?

    >> For nnimap groups `A T' searches /in the current group/ for any
    >> article that is referenced in the primary article, or refers to
    >> the primary article or any of its references. In rare cases where
    >> mailers have not properly coordinated references it is possible
    >> for this to miss articles, but it should be rare and avoiding it
    >> would require recursing searches which would be very slow.


[...]


    >> What it does /not/ do is search across multiple groups, and this
    >> is the most likely reason for not finding all articles within a
    >> thread.

    Dave> Yes, I'm aware that's why it's not finding the article.

OK, if this is the only case that's at issue, the change I am about to
check in should do what you want. 

    >> The current function adds the found articles from the current
    >> group into the summary buffer. Nnir creates an ephemeral group
    >> which can contain articles from multiple groups which is
    >> substantively different.

    Dave> Yes, but when I use `^' it can use nnir to insert articles
    Dave> that don't exist in the current group into the current group's
    Dave> summary buffer... which would is substantively similar to what
    Dave> I'm asking for ;-)

The problem is that when inserting an article header from a foreign
group into an existing summary buffer, gnus has no way to keep track of
where that article came from. It assigns it a phony (negative) article
number so it knows its not native, but thats it. So every time you try
to do something with it (like read it), it has to perform the search
again. This isn't too bad for one or two foreign articles (provided the
search is fast, which it usually is for retrieving a single article with
a known message-id), but is horrible if you have a thread with many
articles in it.

Nnir does two things, really: it performs queries to find articles
according to search criteria; and it creates ephemeral groups based on
these query results.  Searching for messages with a given Message-ID, or
for messages within the current group, only use the first capability.

    >> Having said that I am just about to check in a modification that
    >> will allow searching across multiple groups through the use of a
    >> prefix-arg or setting a variable. Rather than calling the current
    >> `gnus-summary-refer-thread' this will construct the same imap
    >> search pattern but use nnir to search across the entire server
    >> and create an ephemeral group with the results.

    Dave> I'm not yet sure I understand what you're describing.

I think I'm doing what you are asking for: the ability to return a group
containing all the articles in a thread, even if these articles are
spread across multiple groups. If I don't have this right, let me know.

Re: `gnus-refer-article-methods' and `gnus-summary-refer-thread'

[Dave Abrahams]

on Sat Jul 23 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:

>>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:
>
>     Dave> on Fri Jul 22 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:
>     >>>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:
>     >> 
>     Dave> It's a bit curious that when I'm sitting on the root of a
>     Dave> thread, `A T' doesn't find the rest of it via nnir when I have
>     Dave> that in `gnus-refer-article-methods'.
>     >> 
>     >> `A T' (gnus-summary-refer-thread) doesn't use
>     >> gnus-refer-article-methods at all. This is distinct from `^`
>     >> (gnus-summary-refer-parent-article) which consults the entries in
>     >> `gnus-refer-article-methods' in order until the parent is found.
>
>     Dave> Right.  Like I said in the part of my post you didn't cite, I
>     Dave> think I understand why it's that way, but it seems like a
>     Dave> programming artifact rather than a design decision.
>
> Finding the parent is finding a single article with a given
> Message-ID. Once found we know to stop. Since gnus has no idea how many
> articles are in a thread it doesn't know when it should stop. 

Oh, but it's an easy single search: the root of the thread is the first
message-ID in the `References' header.  Then you look for anything whose
`Message-ID' matches the root id or whose `References' contains the root
id.  [You could easily make it a bit more thorough in case of broken
threads by making use of more of the contents of `References', but I've
found this to work perfectly well in practice].

> So looping
> through a set of methods doesn't make much sense to me. Instead, we just
> try a single method, which is the search I described. Aside from broken
> mailers, this will find the whole thread within the given group. I'm
> still not entirely clear, but I think your only unhappiness is the
> restriction to searching within the group rather than across all groups
> on the server. Is this right?

Well, as mentioned earlier, I'm only interested in looking in *one*
other group on the server.  But I'd also like to find messages in the
registry and agent if possible (hey, why not ask for the moon, I
figure?).

>     >> The current function adds the found articles from the current
>     >> group into the summary buffer. Nnir creates an ephemeral group
>     >> which can contain articles from multiple groups which is
>     >> substantively different.
>
>     Dave> Yes, but when I use `^' it can use nnir to insert articles
>     Dave> that don't exist in the current group into the current group's
>     Dave> summary buffer... which would is substantively similar to what
>     Dave> I'm asking for ;-)
>
> The problem is that when inserting an article header from a foreign
> group into an existing summary buffer, gnus has no way to keep track of
> where that article came from. It assigns it a phony (negative) article
> number so it knows its not native, but thats it. So every time you try
> to do something with it (like read it), it has to perform the search
> again. 

Oh. I feel unclean. :-P

> This isn't too bad for one or two foreign articles (provided the
> search is fast, which it usually is for retrieving a single article
> with a known message-id), but is horrible if you have a thread with
> many articles in it.

Yuck, I see!

> Nnir does two things, really: it performs queries to find articles
> according to search criteria; and it creates ephemeral groups based on
> these query results.  Searching for messages with a given Message-ID, or
> for messages within the current group, only use the first capability.
>
>     >> Having said that I am just about to check in a modification that
>     >> will allow searching across multiple groups through the use of a
>     >> prefix-arg or setting a variable. Rather than calling the current
>     >> `gnus-summary-refer-thread' this will construct the same imap
>     >> search pattern but use nnir to search across the entire server
>     >> and create an ephemeral group with the results.
>
>     Dave> I'm not yet sure I understand what you're describing.
>
> I think I'm doing what you are asking for: the ability to return a group
> containing all the articles in a thread, even if these articles are
> spread across multiple groups. If I don't have this right, let me know.

I think you have this right.  Although, let me add, if there's no way to
limit which groups on a server are searched it's going to be way too
slow for me (because I have too many groups).

And thanks for all you're doing!

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: `gnus-refer-article-methods' and `gnus-summary-refer-thread'

[Andrew Cohen]

>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:

    Dave> on Sat Jul 23 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:
    >>>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:
    >> 

[...]


    Dave> Oh, but it's an easy single search: the root of the thread is
    Dave> the first message-ID in the `References' header.  Then you
    Dave> look for anything whose `Message-ID' matches the root id or
    Dave> whose `References' contains the root id.  [You could easily
    Dave> make it a bit more thorough in case of broken threads by
    Dave> making use of more of the contents of `References', but I've
    Dave> found this to work perfectly well in practice].

That's almost exactly what the current search does. (It tries to cope a
bit with broken mailers by being a bit more expansive and for, err,
everything that references anything, so to speak).

    >> So looping through a set of methods doesn't make much sense to
    >> me. Instead, we just try a single method, which is the search I
    >> described. Aside from broken mailers, this will find the whole
    >> thread within the given group. I'm still not entirely clear, but
    >> I think your only unhappiness is the restriction to searching
    >> within the group rather than across all groups on the server. Is
    >> this right?

    Dave> Well, as mentioned earlier, I'm only interested in looking in
    Dave> *one* other group on the server.  But I'd also like to find
    Dave> messages in the registry and agent if possible (hey, why not
    Dave> ask for the moon, I figure?).

I wouldn't think the registry or the agent have any info that isn't also
obtained from the search? And since we never know if we have found
everything we have to do the search in any case...


[...]


    Dave> I think you have this right.  Although, let me add, if there's
    Dave> no way to limit which groups on a server are searched it's
    Dave> going to be way too slow for me (because I have too many
    Dave> groups).

This will respect the `nnir-ignored-newsgroups' variable. Is that
adequate? If not we can always add (yet) another variable.

    Dave> And thanks for all you're doing!

No worries!

Re: `gnus-refer-article-methods' and `gnus-summary-refer-thread'

[Dave Abrahams]

on Sat Jul 23 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:

>>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:
>
>     Dave> on Sat Jul 23 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:
>     >>>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:
>     >> 
>     Dave> Well, as mentioned earlier, I'm only interested in looking in
>     Dave> *one* other group on the server.  But I'd also like to find
>     Dave> messages in the registry and agent if possible (hey, why not
>     Dave> ask for the moon, I figure?).
>
> I wouldn't think the registry or the agent have any info that isn't also
> obtained from the search? 

Well, no, but they can be accessed offline ;-)

> And since we never know if we have found
> everything we have to do the search in any case...
>
>
> [...]
>
>
>     Dave> I think you have this right.  Although, let me add, if there's
>     Dave> no way to limit which groups on a server are searched it's
>     Dave> going to be way too slow for me (because I have too many
>     Dave> groups).
>
> This will respect the `nnir-ignored-newsgroups' variable. Is that
> adequate? If not we can always add (yet) another variable.

Yes, it seems to be fast enough!

Thanks again.  There's another problem that I have more info on, but I
raised it in another thread so will continue there.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

nnir and agent (was: `gnus-refer-article-methods' and `gnus-summary-refer-thread')

[Dave Abrahams]

on Sat Jul 23 2011, Dave Abrahams <dave-AT-boostpro.com> wrote:

> on Sat Jul 23 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:
>
>>>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:
>>
>>     Dave> on Sat Jul 23 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:
>>     >>>>>>> "Dave" == Dave Abrahams <dave@boostpro.com> writes:
>>     >> 
>>     Dave> Well, as mentioned earlier, I'm only interested in looking in
>>     Dave> *one* other group on the server.  But I'd also like to find
>>     Dave> messages in the registry and agent if possible (hey, why not
>>     Dave> ask for the moon, I figure?).
>>
>> I wouldn't think the registry or the agent have any info that isn't also
>> obtained from the search? 
>
> Well, no, but they can be accessed offline ;-)

I notice that the agent *does* have an effect on nnir: I was unable to
find a thread with `G G <message-id>' until after I had done `M-x
gnus-agent-expire-group RET RET' on the group.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com