nnir, gnus-goto-article and such

nnir, gnus-goto-article and such

[Dave Abrahams]

I have this nifty function that lets me look up an article by
message-id, no matter which of my groups it's in:

--8<---------------cut here---------------start------------->8---
(defun gnus-goto-article (message-id)
  (with-current-buffer
      (get-buffer-create (concat "*goto-article " message-id "*"))
    (erase-buffer)
    ;; insert dummy article
    (insert (format "From nobody Tue Sep 13 22:05:34 2011\n\n"))
    (gnus-group-read-ephemeral-group 
     message-id
     `(nndoc ,message-id 
             (nndoc-address ,(current-buffer)) 
             (nndoc-article-type mbox))
     'activate 
     (not    'quit-config) 
     (not    'request-only) 
     '(-1) ; 'select-articles
     (not    'parameters)
     0     ; ' number
     )
    (gnus-summary-refer-article message-id)))
--8<---------------cut here---------------end--------------->8---

Now there are just two problems with this: 

1. (very minor): if I do `/ o' in the resulting buffer, I see the dummy
   article, which is needed if I want to activate the group.  How can I
   remove it from the group after activation?

2. (important) I'd like to see the whole thread (e.g. with `A T').  That
   works fine if it's an IMAP article because nnir finds it.  However,
   nnir doesn't work on nntp newsgroups.

   It seems to me what's missing is some way of recording the source of
   the message retrieved by gnus-summary-refer-article, and some way of
   asking for a thread search in an nntp server.  Ideas?

TIA,

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

gnus-warp-to-article (was: nnir, gnus-goto-article and such)

[Dave Abrahams]

on Thu Sep 15 2011, Dave Abrahams <dave-AT-boostpro.com> wrote:

> I have this nifty function that lets me look up an article by
> message-id, no matter which of my groups it's in:
> (defun gnus-goto-article (message-id)
>   (with-current-buffer
>       (get-buffer-create (concat "*goto-article " message-id "*"))
>     (erase-buffer)
>     ;; insert dummy article
>     (insert (format "From nobody Tue Sep 13 22:05:34 2011\n\n"))
>     (gnus-group-read-ephemeral-group 
>      message-id
>      `(nndoc ,message-id 
>              (nndoc-address ,(current-buffer)) 
>              (nndoc-article-type mbox))
>      'activate 
>      (not    'quit-config) 
>      (not    'request-only) 
>      '(-1) ; 'select-articles
>      (not    'parameters)
>      0     ; ' number
>      )
>     (gnus-summary-refer-article message-id)))
> Now there are just two problems with this: 
>
> 1. (very minor): if I do `/ o' in the resulting buffer, I see the dummy
>    article, which is needed if I want to activate the group.  How can I
>    remove it from the group after activation?
>
> 2. (important) I'd like to see the whole thread (e.g. with `A T').  That
>    works fine if it's an IMAP article because nnir finds it.  However,
>    nnir doesn't work on nntp newsgroups.
>
>    It seems to me what's missing is some way of recording the source of
>    the message retrieved by gnus-summary-refer-article, and some way of
>    asking for a thread search in an nntp server.  Ideas?

Further investigation reveals that groups supporting
gnus-warp-to-article (virtual groups and nnir groups) already seem to do
something like that, and in such a group, `A T' would "just work."  If
anyone is able to offer further guidance here it would be most welcome.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Lars Magne Ingebrigtsen]

Dave Abrahams <dave@boostpro.com> writes:

> 2. (important) I'd like to see the whole thread (e.g. with `A T').  That
>    works fine if it's an IMAP article because nnir finds it.  However,
>    nnir doesn't work on nntp newsgroups.
>
>    It seems to me what's missing is some way of recording the source of
>    the message retrieved by gnus-summary-refer-article, and some way of
>    asking for a thread search in an nntp server.  Ideas?

Hm...  `gnus-summary-refer-article' work presumably because of the
registry, I guess?  Then the registry should be able to tell you what
group it came from, too, I think?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: nnir, gnus-goto-article and such

[John Wiegley]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Hm...  `gnus-summary-refer-article' work presumably because of the registry,
> I guess?  Then the registry should be able to tell you what group it came
> from, too, I think?

The registry doesn't know about articles existing on the nntp server which
gnus-summary-refer-article can still query for.  So the registry is always a
subset of the total articles I can refer to.  (Which is a good thing, because
GigaNews has millions upon millions of articles, and that would cause the
registry become uselessly slow).

What doesn't work is that 'A T' doesn't do everything it can to recursively
find all referenced articles from all refer-article sources.  In contrast, if
you hand-select each article that 'A R' finds, and then press 'A R' on that,
and so on recursively, you will ultimately end up with the entire thread.  'A
T' seems to use a different logic -- which gets even more different if nnimap
is ever involved, *even if the referred-to article came from a non-IMAP
server*.

John

Re: nnir, gnus-goto-article and such

[Andrew Cohen]

>>>>> "John" == John Wiegley <jwiegley@gmail.com> writes:


[...]


    John> What doesn't work is that 'A T' doesn't do everything it can
    John> to recursively find all referenced articles from all
    John> refer-article sources.  

The logic used in 'A T' is different from that of 'A R' and friends
because its doing something different. That's not to say that it doesn't
have bugs, and I'm sure that it can be improved. 

    John> In contrast, if you hand-select each article that 'A R' finds,
    John> and then press 'A R' on that, and so on recursively, you will
    John> ultimately end up with the entire thread.  

I don't think this gives the entire thread. It only gives the articles
that precede the article you start with along the current branch. If
there are subsequent articles along the current branch, or articles on
another branch, this recursive procedure won't find them. That's why the
'A T' function tries to do a /search/ for articles rather than just
recursive retrieval. This is complicated by the limitations of the
different kinds of servers. 


Here is the complete `A T' logic for nntp and nnimap groups:

1. nntp: The nntp protocol has no searching capabilities at all, so
   there is no way to search by message-id. Gmane has a Xapian-powered
   search engine written by Olly Betts that works very well, but doesn't
   allow searching by message-id. What gnus does in this case is very
   primitive: it retrieves the headers for a (configurable) number of
   articles before and after the current article (250 before and 250 after
   by default) in the current group, and then gnus itself searches within
   these headers for articles belonging to the thread. But there is no way
   to know if there are other articles in the thread in other groups or on
   other servers.

2. nnimap: The imap protocol allows searching by headers, but only
   within a single group. So by default if you `A T' on an article in an
   nnimap group a search will be performed for all the articles that
   might be in the thread in the current group only. 

   Recently I added the ability to replace this search with an nnir
   search which will look for articles in the thread on the whole server
   rather than just in the current group. But it won't look outside the
   server. 

   I have also recently re-written much of nnir, but I won't push it
   until after the freeze is over. As part of this re-write, when
   searching for a thread on the whole server the registry is consulted
   to see what group the article on which you invoke `A T' came from,
   and consult /both/ that server and the server containing the current
   group. This is a very minor improvement, since most of the time these
   two servers will be the same. 

I guess we can consult the registry for all the articles mentioned in
the references header and see if any of those articles came from
different servers and search those servers as well. But if they are nntp
servers we can't really search them effectively (since nntp doesn't have
search facilities). We could also configure this to /always/ search a
particular set of servers independently of where the proximate article
came from.

Best,
Andy

Re: nnir, gnus-goto-article and such

[Lars Magne Ingebrigtsen]

Andrew Cohen <cohen@andy.bu.edu> writes:

> I guess we can consult the registry for all the articles mentioned in
> the references header and see if any of those articles came from
> different servers and search those servers as well.

Yes, that would be nice.

> But if they are nntp servers we can't really search them effectively
> (since nntp doesn't have search facilities).

No, but if we know the server/group, then we could try selecting the
last (say) 1000 articles...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Wed Sep 21 2011, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:

> Dave Abrahams <dave@boostpro.com> writes:
>
>> 2. (important) I'd like to see the whole thread (e.g. with `A T').  That
>>    works fine if it's an IMAP article because nnir finds it.  However,
>>    nnir doesn't work on nntp newsgroups.
>>
>>    It seems to me what's missing is some way of recording the source of
>>    the message retrieved by gnus-summary-refer-article, and some way of
>>    asking for a thread search in an nntp server.  Ideas?
>
> Hm...  `gnus-summary-refer-article' work presumably because of the
> registry, I guess?  

Well, as far as I know I don't have the registry turned on and I have
never been able to understand exactly how it could help me.  On the
surface it sounds exactly like something I'd want and I get very excited
but when I think about it carefully, it seems otherwise.  

> Then the registry should be able to tell you what group it came from,
> too, I think?

Maybe...  I'm still not sure how to use it.  But I'll take another look.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Thu Sep 22 2011, Andrew Cohen <cohen-AT-andy.bu.edu> wrote:

>>>>>> "John" == John Wiegley <jwiegley@gmail.com> writes:
>
> [...]
>
>     John> What doesn't work is that 'A T' doesn't do everything it can
>     John> to recursively find all referenced articles from all
>     John> refer-article sources.  
>
> The logic used in 'A T' is different from that of 'A R' and friends
> because its doing something different. That's not to say that it doesn't
> have bugs, and I'm sure that it can be improved. 

Andrew, I think there's a miscommunication here.  John is talking about
what he sees as a design/operational deficiency, while you're talking
about the implementation.

For example, `A T' starts out with a call to `gnus-warp-to-article'.
However, `gnus-warp-to-article' for some reason only works in nnir
groups.  There are several other backends that aggregate messages from
other groups, but none of them remembers where the messages came from.
Even if I happen to have the article in the registry, and *it* knows
which group it came from, I won't end up warping there.  And I think
that's the cause of a lot of the grief John and I are experiencing.

>     John> In contrast, if you hand-select each article that 'A R' finds,
>     John> and then press 'A R' on that, and so on recursively, you will
>     John> ultimately end up with the entire thread.  
>
> I don't think this gives the entire thread. It only gives the articles
> that precede the article you start with along the current branch. 

...and does `A T' always do *at least* that well?  It should, IMO.

> If
> there are subsequent articles along the current branch, or articles on
> another branch, this recursive procedure won't find them. That's why the
> 'A T' function tries to do a /search/ for articles rather than just
> recursive retrieval. This is complicated by the limitations of the
> different kinds of servers. 

Right... you know, I could swear that wanderlust lets me search in nntp
groups.  It has a single uniform interface to these searches across all
backends.  *looks it up* yeah, it's called a filter folder.  I wonder
how that works?

Aha, it looks like maybe they use the server for everything but body
searches (which we don't care about for thread-following)?  I'm not
entirely sure.

--8<---------------cut here---------------start------------->8---
(defun elmo-nntp-use-server-search-p (condition)
  (if (vectorp condition)
      (not (string= "body" (elmo-filter-key condition)))
    (and (elmo-nntp-use-server-search-p (nth 1 condition))
	 (elmo-nntp-use-server-search-p (nth 2 condition)))))
--8<---------------cut here---------------end--------------->8---


> Here is the complete `A T' logic for nntp and nnimap groups:
>
> 1. nntp: The nntp protocol has no searching capabilities at all, so
>    there is no way to search by message-id. Gmane has a Xapian-powered
>    search engine written by Olly Betts that works very well, but doesn't
>    allow searching by message-id. 

If it does full-text searches you could narrow the field by searching
for the message-id anyway.

> What gnus does in this case is very primitive: it retrieves the
> headers for a (configurable) number of articles before and after the
> current article (250 before and 250 after by default) in the current
> group, and then gnus itself searches within these headers for articles
> belonging to the thread. But there is no way to know if there are
> other articles in the thread in other groups or on other servers.
>
> 2. nnimap: The imap protocol allows searching by headers, but only
>    within a single group. So by default if you `A T' on an article in an
>    nnimap group a search will be performed for all the articles that
>    might be in the thread in the current group only. 
>
>    Recently I added the ability to replace this search with an nnir
>    search which will look for articles in the thread on the whole server
>    rather than just in the current group. But it won't look outside the
>    server. 

Right... we'd like it to use all the resources that could be at its
disposal; that is, everything in gnus-refer-article-method.

>    I have also recently re-written much of nnir, but I won't push it
>    until after the freeze is over. As part of this re-write, when
>    searching for a thread on the whole server the registry is consulted
>    to see what group the article on which you invoke `A T' came from,
>    and consult /both/ that server and the server containing the current
>    group. This is a very minor improvement, since most of the time these
>    two servers will be the same. 

Except in my case, where I'm creating an ephemeral group to hold the
referenced article.  I think that's exactly what I want right now.

> I guess we can consult the registry for all the articles mentioned in
> the references header and see if any of those articles came from
> different servers and search those servers as well. But if they are nntp
> servers we can't really search them effectively (since nntp doesn't have
> search facilities). 

Wow; we can't pull up a message by ID from NNTP?  too bad.

> We could also configure this to /always/ search a particular set of
> servers independently of where the proximate article came from.

IMO, use `gnus-refer-article-method'; it already answers the question
:-)

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Ted Zlatanov]

On Mon, 26 Sep 2011 16:01:52 -0400 Dave Abrahams <dave@boostpro.com> wrote: 

DA> on Wed Sep 21 2011, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:

>> Dave Abrahams <dave@boostpro.com> writes:
>> 
>>> 2. (important) I'd like to see the whole thread (e.g. with `A T').  That
>>> works fine if it's an IMAP article because nnir finds it.  However,
>>> nnir doesn't work on nntp newsgroups.
>>> 
>>> It seems to me what's missing is some way of recording the source of
>>> the message retrieved by gnus-summary-refer-article, and some way of
>>> asking for a thread search in an nntp server.  Ideas?
>> 
>> Hm...  `gnus-summary-refer-article' work presumably because of the
>> registry, I guess?  

DA> Well, as far as I know I don't have the registry turned on and I have
DA> never been able to understand exactly how it could help me.  On the
DA> surface it sounds exactly like something I'd want and I get very excited
DA> but when I think about it carefully, it seems otherwise.  

The registry is a place where we record interesting things (group,
subject, sender, recipients, registry marks) about articles, keyed by
the message ID.  But it's also a general key-val store with some
indexing.  For Lars' point, the registry is an opportunistic cache that
remember where you've seen a message-ID (server name and group name) and
where it's been split before.

>> Then the registry should be able to tell you what group it came from,
>> too, I think?

DA> Maybe...  I'm still not sure how to use it.  But I'll take another look.

Simply call it like this (for message-ID "34"):

(gnus-registry-get-id-key "34" 'group)
(gnus-registry-get-id-key "34" 'subject)

gnus-registry.el has most of the install info at the beginning if you
want to try it out.

Ted

Re: nnir, gnus-goto-article and such

[John Wiegley]

>>>>> Dave Abrahams <dave@boostpro.com> writes:

>> I guess we can consult the registry for all the articles mentioned in the
>> references header and see if any of those articles came from different
>> servers and search those servers as well. But if they are nntp servers we
>> can't really search them effectively (since nntp doesn't have search
>> facilities).

> Wow; we can't pull up a message by ID from NNTP?  too bad.

How, then, does 'A R' work for any Message-Id known to the NNTP server?  I've
tried this for ancient articles from 7 years ago, and Gnus pops the message
right into my summary buffer.  How is it finding it then?

John

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Mon Sep 26 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Mon, 26 Sep 2011 16:01:52 -0400 Dave Abrahams <dave@boostpro.com> wrote: 
>
> DA> on Wed Sep 21 2011, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:
>
>>> Dave Abrahams <dave@boostpro.com> writes:
>>> 
>>>> 2. (important) I'd like to see the whole thread (e.g. with `A T').  That
>>>> works fine if it's an IMAP article because nnir finds it.  However,
>>>> nnir doesn't work on nntp newsgroups.
>>>> 
>>>> It seems to me what's missing is some way of recording the source of
>>>> the message retrieved by gnus-summary-refer-article, and some way of
>>>> asking for a thread search in an nntp server.  Ideas?
>>> 
>>> Hm...  `gnus-summary-refer-article' work presumably because of the
>>> registry, I guess?  
>
> DA> Well, as far as I know I don't have the registry turned on and I have
> DA> never been able to understand exactly how it could help me.  On the
> DA> surface it sounds exactly like something I'd want and I get very excited
> DA> but when I think about it carefully, it seems otherwise.  
>
> The registry is a place where we record interesting things (group,
> subject, sender, recipients, registry marks) about articles, keyed by
> the message ID.  But it's also a general key-val store with some
> indexing.  For Lars' point, the registry is an opportunistic cache that
> remember where you've seen a message-ID (server name and group name) and
> where it's been split before.

Yes, I understand all that... I know what it is as a library component.
What I don't see is how *at the user level* it can easily and usefully
be exploited.

>>> Then the registry should be able to tell you what group it came from,
>>> too, I think?
>
> DA> Maybe...  I'm still not sure how to use it.  But I'll take another look.
>
> Simply call it like this (for message-ID "34"):
>
> (gnus-registry-get-id-key "34" 'group)
> (gnus-registry-get-id-key "34" 'subject)

"34" sounds like a message number, not a Message-ID (like what's found
in mail headers).  How is gnus-registry going to know which group's
number 34 I'm referencing?

> gnus-registry.el has most of the install info at the beginning if you
> want to try it out.

I re-installed it yesterday, thanks.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Tue Sep 27 2011, John Wiegley <jwiegley-AT-gmail.com> wrote:

>>>>>> Dave Abrahams <dave@boostpro.com> writes:
>
>
>>> I guess we can consult the registry for all the articles mentioned in the
>>> references header and see if any of those articles came from different
>>> servers and search those servers as well. But if they are nntp servers we
>>> can't really search them effectively (since nntp doesn't have search
>>> facilities).
>
>> Wow; we can't pull up a message by ID from NNTP?  too bad.
>
> How, then, does 'A R' work for any Message-Id known to the NNTP server?  I've
> tried this for ancient articles from 7 years ago, and Gnus pops the message
> right into my summary buffer.  How is it finding it then?

Right, if it couldn't do that, it wouldn't be able to handle `^'
either in a default configuration.  And it does.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Mon Sep 26 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:
DA> on Wed Sep 21 2011, Lars Magne Ingebrigtsen <larsi-AT-gnus.org> wrote:

>>> Then the registry should be able to tell you what group it came from,
>>> too, I think?
>
> DA> Maybe...  I'm still not sure how to use it.  But I'll take another look.
>
> Simply call it like this (for message-ID "34"):
>
> (gnus-registry-get-id-key "34" 'group)
> (gnus-registry-get-id-key "34" 'subject)

So here's an example where I tried it out.  

(defun gnus-goto-article (message-id)
  (with-temp-buffer
    (erase-buffer)
    ;; Insert dummy article
    (insert (format "From nobody Tue Sep 13 22:05:34 2011\n\n"))
    (gnus-group-read-ephemeral-group message-id
     `(nndoc ,message-id
             (nndoc-address ,(current-buffer))
             (nndoc-article-type mbox))
     'activate
     (not    'quit-config)
     (not    'request-only)
     '(-1) ; 'select-articles
     (not    'parameters)
     0     ; ' number
     )
    (gnus-summary-refer-article message-id)
    (message "group from registry: %s" (gnus-registry-get-id-key message-id 'group))
    ))

I queried the registry at the point where I'd like to know the group of
that article.  Unfortunately, I get back nil.  And if I wanted to
conjure up an article number, well, it's -1, so that's not going to be
much help, is it?

It seems like this shouldn't be so hard...

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Ted Zlatanov]

On Tue, 27 Sep 2011 11:35:42 -0400 Dave Abrahams <dave@boostpro.com> wrote: 

DA> on Mon Sep 26 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

>> The registry is a place where we record interesting things (group,
>> subject, sender, recipients, registry marks) about articles, keyed by
>> the message ID.  But it's also a general key-val store with some
>> indexing.  For Lars' point, the registry is an opportunistic cache that
>> remember where you've seen a message-ID (server name and group name) and
>> where it's been split before.

DA> Yes, I understand all that... I know what it is as a library component.
DA> What I don't see is how *at the user level* it can easily and usefully
DA> be exploited.

Er, you can split to the reference parent, or by subject, or by sender?
It lets the user keep registry marks?  Are those not exploitative enough?

>> Simply call it like this (for message-ID "34"):
>> 
>> (gnus-registry-get-id-key "34" 'group)
>> (gnus-registry-get-id-key "34" 'subject)

DA> "34" sounds like a message number, not a Message-ID (like what's found
DA> in mail headers).  How is gnus-registry going to know which group's
DA> number 34 I'm referencing?

No, I said message-ID was 34 so I didn't have to type
"<m2wrcu0zht.fsf@pluto.luannocracy.com>".  Sorry for the confusion.

DA> So here's an example where I tried it out.  

DA> (defun gnus-goto-article (message-id)
DA>   (with-temp-buffer
DA>     (erase-buffer)
DA>     ;; Insert dummy article
DA>     (insert (format "From nobody Tue Sep 13 22:05:34 2011\n\n"))
DA>     (gnus-group-read-ephemeral-group message-id
DA>      `(nndoc ,message-id
DA>              (nndoc-address ,(current-buffer))
DA>              (nndoc-article-type mbox))
DA>      'activate
DA>      (not    'quit-config)
DA>      (not    'request-only)
DA>      '(-1) ; 'select-articles
DA>      (not    'parameters)
DA>      0     ; ' number
DA>      )
DA>     (gnus-summary-refer-article message-id)
DA>     (message "group from registry: %s" (gnus-registry-get-id-key message-id 'group))
DA>     ))

DA> I queried the registry at the point where I'd like to know the group of
DA> that article.  Unfortunately, I get back nil.  And if I wanted to
DA> conjure up an article number, well, it's -1, so that's not going to be
DA> much help, is it?

The registry notices articles when you visit a group or when articles
are split.  To feed it artificial data, use:

(defun gnus-registry-handle-action (id from to subject sender
                                       &optional recipients)

I hope that helps...

Ted

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Tue Sep 27 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Tue, 27 Sep 2011 11:35:42 -0400 Dave Abrahams <dave@boostpro.com> wrote: 
>
> DA> on Mon Sep 26 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:
>
>>> The registry is a place where we record interesting things (group,
>>> subject, sender, recipients, registry marks) about articles, keyed by
>>> the message ID.  But it's also a general key-val store with some
>>> indexing.  For Lars' point, the registry is an opportunistic cache that
>>> remember where you've seen a message-ID (server name and group name) and
>>> where it's been split before.
>
> DA> Yes, I understand all that... I know what it is as a library component.
> DA> What I don't see is how *at the user level* it can easily and usefully
> DA> be exploited.
>
> Er, you can split to the reference parent, 

I don't use client-side splitting, so I guess that doesn't make much
difference to me, but I guess it might for others.

> or by subject, or by sender?  

You can't split by subject or sender without the registry?!  You're
kidding, right?  Like I said, I never ask Gnus to shuffle mail into
different folders for me but that seems pretty basic.

> It lets the user keep registry marks?  Are those not exploitative
> enough?

What's a user going to *do* with registry marks?  That's another tool
whose real utility escapes me.

>>> Simply call it like this (for message-ID "34"):
>>> 
>>> (gnus-registry-get-id-key "34" 'group)
>>> (gnus-registry-get-id-key "34" 'subject)
>
> DA> "34" sounds like a message number, not a Message-ID (like what's found
> DA> in mail headers).  How is gnus-registry going to know which group's
> DA> number 34 I'm referencing?
>
> No, I said message-ID was 34 so I didn't have to type
> "<m2wrcu0zht.fsf@pluto.luannocracy.com>".  Sorry for the confusion.

np, thanks for the clarification.

> DA> So here's an example where I tried it out.  
>
> DA> (defun gnus-goto-article (message-id)
> DA>   (with-temp-buffer
> DA>     (erase-buffer)
> DA>     ;; Insert dummy article
> DA>     (insert (format "From nobody Tue Sep 13 22:05:34 2011\n\n"))
> DA>     (gnus-group-read-ephemeral-group message-id
> DA>      `(nndoc ,message-id
> DA>              (nndoc-address ,(current-buffer))
> DA>              (nndoc-article-type mbox))
> DA>      'activate
> DA>      (not    'quit-config)
> DA>      (not    'request-only)
> DA>      '(-1) ; 'select-articles
> DA>      (not    'parameters)
> DA>      0     ; ' number
> DA>      )
> DA>     (gnus-summary-refer-article message-id)
> DA>     (message "group from registry: %s" (gnus-registry-get-id-key message-id 'group))
> DA>     ))
>
> DA> I queried the registry at the point where I'd like to know the group of
> DA> that article.  Unfortunately, I get back nil.  And if I wanted to
> DA> conjure up an article number, well, it's -1, so that's not going to be
> DA> much help, is it?
>
> The registry notices articles when you visit a group or when articles
> are split.  To feed it artificial data, use:
>
> (defun gnus-registry-handle-action (id from to subject sender
>                                        &optional recipients)

Umm, sorry.  That function isn't documented.  How do I use it?  How
would I get the group that gnus-summary-refer-article finds the message
in into the registry?  I don't even see a parameter for group.

> I hope that helps...

Getting there, but not quite yet.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Ted Zlatanov]

On Wed, 28 Sep 2011 10:34:45 -0400 Dave Abrahams <dave@boostpro.com> wrote: 

DA> on Tue Sep 27 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

>> Er, you can split to the reference parent, 

DA> I don't use client-side splitting, so I guess that doesn't make much
DA> difference to me, but I guess it might for others.

Yes, it makes a huge difference.

>> or by subject, or by sender?  

DA> You can't split by subject or sender without the registry?!  You're
DA> kidding, right?  Like I said, I never ask Gnus to shuffle mail into
DA> different folders for me but that seems pretty basic.

No, I mean it will split to the same group as other articles with the
same subject or sender.  So if I put your mails in "dave" it will figure
out it should keep putting new ones in there.

>> It lets the user keep registry marks?  Are those not exploitative
>> enough?

DA> What's a user going to *do* with registry marks?  That's another tool
DA> whose real utility escapes me.

It lets the user define any marks they want, without reconfiguring Gnus
and without storing them in the newsrc.eld, by message-ID and not
article number.

>> (defun gnus-registry-handle-action (id from to subject sender
>> &optional recipients)

DA> Umm, sorry.  That function isn't documented.  How do I use it?  How
DA> would I get the group that gnus-summary-refer-article finds the message
DA> in into the registry?  I don't even see a parameter for group.

id = message ID, a string
from = source group, nil in your case for a new article
to = destination group, your group
subject, sender, receipients = strings

The group name is fully qualified IIRC.

Ted

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Wed Sep 28 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

>>> or by subject, or by sender?  
>
> DA> You can't split by subject or sender without the registry?!  You're
> DA> kidding, right?  Like I said, I never ask Gnus to shuffle mail into
> DA> different folders for me but that seems pretty basic.
>
> No, I mean it will split to the same group as other articles with the
> same subject or sender.  So if I put your mails in "dave" it will figure
> out it should keep putting new ones in there.

...which always seemed wrong to me.  Some subject lines (e.g. "FYI") are
so common that it's not a good guide to threading.  Since the message
has References: and In-Reply-To: headers, why not use those?

>>> It lets the user keep registry marks?  Are those not exploitative
>>> enough?
>
> DA> What's a user going to *do* with registry marks?  That's another tool
> DA> whose real utility escapes me.
>
> It lets the user define any marks they want, without reconfiguring Gnus
> and without storing them in the newsrc.eld, by message-ID and not
> article number.

IIRC, IMAP lets you do that anyway?  But anyway, again, why would I want
to define new marks?  I guess I can think of a use-case, but marks that
don't follow me to other mail clients are pretty limited.

>>> (defun gnus-registry-handle-action (id from to subject sender
>>> &optional recipients)
>
> DA> Umm, sorry.  That function isn't documented.  How do I use it?  How
> DA> would I get the group that gnus-summary-refer-article finds the message
> DA> in into the registry?  I don't even see a parameter for group.
>
> id = message ID, a string
> from = source group, nil in your case for a new article

nil is not useful to me; the whole point of this exercise is to record
the group associated with an article.

What I'd want to do with the registry is get gnus-summary-refer-article
to put in the registry the group in which it finds the article, so
gnus-warp-to-article will go to *that* group when I try to get to the
whole thread.

> to = destination group, your group

I don't think I want gnus to remember that the article is in my group.
I want it to be like an nnir group, where Gnus remembers the group in
which the article was originally found.

Note: nnir doesn't use the registry, but has its *own* record of which
group was the source of each article.  Why?  Dunno.

> subject, sender, receipients = strings
>
> The group name is fully qualified IIRC.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]


on Wed Sep 28 2011, Dave Abrahams <dave-AT-boostpro.com> wrote:

> What I'd want to do with the registry is get gnus-summary-refer-article
> to put in the registry the group in which it finds the article, so
> gnus-warp-to-article will go to *that* group when I try to get to the
> whole thread.
>
>> to = destination group, your group
>
> I don't think I want gnus to remember that the article is in my group.
> I want it to be like an nnir group, where Gnus remembers the group in
> which the article was originally found.

So, thanks, you seem to have gotten me over one hump... here's my patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnus-sum.patch --]
[-- Type: text/x-patch, Size: 893 bytes --]

	Modified   lisp/gnus-sum.el
diff --git a/lisp/gnus-sum.el b/lisp/gnus-sum.el
index 423de35..8452bb9 100644
--- a/lisp/gnus-sum.el
+++ b/lisp/gnus-sum.el
@@ -9077,6 +9077,13 @@ non-numeric or nil fetch the number specified by the
 		       (setq number (gnus-summary-insert-subject message-id)))
 	      ;; and display the article.
 	      (gnus-summary-select-article nil nil nil number)
+              (when (bound-and-true-p gnus-registry-install)
+                (let ((header (gnus-summary-article-header (gnus-summary-article-number))))
+                  (gnus-registry-handle-action
+                   message-id nil
+                   (gnus-method-to-full-server-name gnus-override-method)
+                   (mail-header-subject header)
+                   (mail-header-from header))))
 	      (throw 'found t)))
 	  (gnus-message 3 "Couldn't fetch article %s" message-id)))))))

[-- Attachment #3: Type: text/plain, Size: 716 bytes --]


But now, of course, gnus-warp-to-article doesn't consult the registry to
find out the article's group, so `A T' still fails.  For some reason, it
goes directly to the backend, only one of which supports warping: nnir.

[Aside: it seems like this territory is full of components that could in
principle be nicely generalized, but whose implementation has only been
made to work for one specific usage scenario]

would it work if nnregistry supported warping?  I don't really
understand where/how that backend comes into play.

> Note: nnir doesn't use the registry, but has its *own* record of which
> group was the source of each article.  Why?  Dunno.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Ted Zlatanov]

On Wed, 28 Sep 2011 21:19:38 -0400 Dave Abrahams <dave@boostpro.com> wrote: 

DA> on Wed Sep 28 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

>>>> or by subject, or by sender?  
>> 
DA> You can't split by subject or sender without the registry?!  You're
DA> kidding, right?  Like I said, I never ask Gnus to shuffle mail into
DA> different folders for me but that seems pretty basic.
>> 
>> No, I mean it will split to the same group as other articles with the
>> same subject or sender.  So if I put your mails in "dave" it will figure
>> out it should keep putting new ones in there.

DA> ...which always seemed wrong to me.  Some subject lines (e.g. "FYI") are
DA> so common that it's not a good guide to threading.  Since the message
DA> has References: and In-Reply-To: headers, why not use those?

We do, that's what splitting by parent does.  But many clients don't do
it properly.

In my experience subject lines don't recur that often but yeah that's
one possible way to misfile a message.

Splitting by sender and recipient is less prone to such homonymous
accidents.

DA> What's a user going to *do* with registry marks?  That's another tool
DA> whose real utility escapes me.
>> 
>> It lets the user define any marks they want, without reconfiguring Gnus
>> and without storing them in the newsrc.eld, by message-ID and not
>> article number.

DA> IIRC, IMAP lets you do that anyway?  But anyway, again, why would I want
DA> to define new marks?  I guess I can think of a use-case, but marks that
DA> don't follow me to other mail clients are pretty limited.

Most backends, especially NNTP, don't let you define marks at all.  The
whole point of custom marks is that we (the Gnus developers) haven't
thought of them yet.  The example ones are To-Do, Later, Work, and
Personal.  I add Perl and a few work-related tags that would make no
sense for anyone else.

>>>> (defun gnus-registry-handle-action (id from to subject sender
>>>> &optional recipients)
>> 
DA> Umm, sorry.  That function isn't documented.  How do I use it?  How
DA> would I get the group that gnus-summary-refer-article finds the message
DA> in into the registry?  I don't even see a parameter for group.
>> 
>> id = message ID, a string
>> from = source group, nil in your case for a new article

DA> nil is not useful to me; the whole point of this exercise is to record
DA> the group associated with an article.

start with article groups = (A C) in each case below;

from = nil, to = B: article created/noticed in group B => (A B C)
from = A, to = B: article move from A to B => (B C)
from = A, to = nil: article registry deletion from group A => (C)

DA> What I'd want to do with the registry is get gnus-summary-refer-article
DA> to put in the registry the group in which it finds the article, so
DA> gnus-warp-to-article will go to *that* group when I try to get to the
DA> whole thread.

DA> I don't think I want gnus to remember that the article is in my group.
DA> I want it to be like an nnir group, where Gnus remembers the group in
DA> which the article was originally found.

from = nil, to = noticed_group

DA> Note: nnir doesn't use the registry, but has its *own* record of which
DA> group was the source of each article.  Why?  Dunno.

The registry is an optional package so nnir doesn't require it.  The
nnregistry backend tracks articles through the registry.

Ted

Re: nnir, gnus-goto-article and such

[Ted Zlatanov]

On Wed, 28 Sep 2011 22:06:37 -0400 Dave Abrahams <dave@boostpro.com> wrote: 

DA> So, thanks, you seem to have gotten me over one hump... here's my patch:
DA> 	Modified   lisp/gnus-sum.el
DA> diff --git a/lisp/gnus-sum.el b/lisp/gnus-sum.el
DA> index 423de35..8452bb9 100644
DA> --- a/lisp/gnus-sum.el
DA> +++ b/lisp/gnus-sum.el
DA> @@ -9077,6 +9077,13 @@ non-numeric or nil fetch the number specified by the
DA>  		       (setq number (gnus-summary-insert-subject message-id)))
DA>  	      ;; and display the article.
DA>  	      (gnus-summary-select-article nil nil nil number)
DA> +              (when (bound-and-true-p gnus-registry-install)
DA> +                (let ((header (gnus-summary-article-header (gnus-summary-article-number))))
DA> +                  (gnus-registry-handle-action
DA> +                   message-id nil
DA> +                   (gnus-method-to-full-server-name gnus-override-method)
DA> +                   (mail-header-subject header)
DA> +                   (mail-header-from header))))
DA>  	      (throw 'found t)))
DA>  	  (gnus-message 3 "Couldn't fetch article %s" message-id)))))))

DA> But now, of course, gnus-warp-to-article doesn't consult the registry to
DA> find out the article's group, so `A T' still fails.  For some reason, it
DA> goes directly to the backend, only one of which supports warping: nnir.

I think you need to explicitly enable nnregistry so that will work (see
the comments).  Perhaps it should be automatically loaded when the Gnus
registry is initialized.  Andy Cohen did that work IIRC so I don't know
if there are any reasons not to do it.  It seems reasonable to me.

DA> [Aside: it seems like this territory is full of components that could in
DA> principle be nicely generalized, but whose implementation has only been
DA> made to work for one specific usage scenario]

I can't tell, I was not involved with writing these components.  At
least for the registry I know it's optional so we can't depend on it.

Ted

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Thu Sep 29 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Wed, 28 Sep 2011 22:06:37 -0400 Dave Abrahams <dave@boostpro.com> wrote: 
>
> DA> So, thanks, you seem to have gotten me over one hump... here's my patch:
> DA> 	Modified   lisp/gnus-sum.el
> DA> diff --git a/lisp/gnus-sum.el b/lisp/gnus-sum.el
> DA> index 423de35..8452bb9 100644
> DA> --- a/lisp/gnus-sum.el
> DA> +++ b/lisp/gnus-sum.el
> DA> @@ -9077,6 +9077,13 @@ non-numeric or nil fetch the number specified by the
> DA>  		       (setq number (gnus-summary-insert-subject message-id)))
> DA>  	      ;; and display the article.
> DA>  	      (gnus-summary-select-article nil nil nil number)
> DA> +              (when (bound-and-true-p gnus-registry-install)
> DA> +                (let ((header (gnus-summary-article-header (gnus-summary-article-number))))
> DA> +                  (gnus-registry-handle-action
> DA> +                   message-id nil
> DA> +                   (gnus-method-to-full-server-name gnus-override-method)
> DA> +                   (mail-header-subject header)
> DA> +                   (mail-header-from header))))
> DA>  	      (throw 'found t)))
> DA>  	  (gnus-message 3 "Couldn't fetch article %s" message-id)))))))
>
> DA> But now, of course, gnus-warp-to-article doesn't consult the registry to
> DA> find out the article's group, so `A T' still fails.  For some reason, it
> DA> goes directly to the backend, only one of which supports warping: nnir.
>
> I think you need to explicitly enable nnregistry so that will work (see
> the comments).  

in nnregistry.el there are no comments.

in gnus-registry.el it says

  ;; You should also consider using the nnregistry backend to look up
  ;; articles.  See the Gnus manual for more information.

The Gnus manual says almost nothing useful about nnregistry.  It shows
how it can be used in gnus-refer-article-method, and I have done that.

What further enabling do you have in mind?

> Perhaps it should be automatically loaded when the Gnus registry is
> initialized.  Andy Cohen did that work IIRC so I don't know if there
> are any reasons not to do it.  It seems reasonable to me.
>
> DA> [Aside: it seems like this territory is full of components that
> DA> could in principle be nicely generalized, but whose implementation
> DA> has only been made to work for one specific usage scenario]
>
> I can't tell, I was not involved with writing these components.  

IIUC from reading elsewhere you rewrote the registry.

> At least for the registry I know it's optional so we can't depend on
> it.

I'm not sure what "optional" means in this context.  Is Gnus going to
ship without the registry code?

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Ted Zlatanov]

On Thu, 29 Sep 2011 09:40:54 -0400 Dave Abrahams <dave@boostpro.com> wrote: 

DA> in nnregistry.el there are no comments.

There are a few:

#+begin_src lisp
;; This file provides the `nnregistry' Gnus back-end.  It can be used
;; in `gnus-refer-article-method' to quickly search for a message by
;; id, regardless of the back-end that stores it.  See the Gnus manual
;; for usage examples and more information.
#+end_src

but yeah, it's a pretty new package.

DA> in gnus-registry.el it says

DA>   ;; You should also consider using the nnregistry backend to look up
DA>   ;; articles.  See the Gnus manual for more information.

DA> The Gnus manual says almost nothing useful about nnregistry.  It shows
DA> how it can be used in gnus-refer-article-method, and I have done that.

DA> What further enabling do you have in mind?

That was it.  

>> Perhaps it should be automatically loaded when the Gnus registry is
>> initialized.  Andy Cohen did that work IIRC so I don't know if there
>> are any reasons not to do it.  It seems reasonable to me.
>> 
DA> [Aside: it seems like this territory is full of components that
DA> could in principle be nicely generalized, but whose implementation
DA> has only been made to work for one specific usage scenario]
>> 
>> I can't tell, I was not involved with writing these components.  

DA> IIUC from reading elsewhere you rewrote the registry.

I wrote *registry.el but I was talking about nnir.el, nnregistry.el, and
other search functionality.  The registry is not a search engine so I
thought you didn't mean it.  I'm not sure how it could be generalized
and in what direction--tell me what you think.

>> At least for the registry I know it's optional so we can't depend on
>> it.

DA> I'm not sure what "optional" means in this context.  Is Gnus going to
DA> ship without the registry code?

Users should be able to use every aspect of Gnus without calling
`gnus-registru-initialize'.  The memory usage of the gnus-registry, if
we enabled it by default, would be unpleasant for many of our users on
older or less capable machines.  Thus any general Gnus functionality
should not depend on the gnus-registry's availability.

Furthermore the gnus-registry should be treated as a lossy cache.  If
you find something useful, great.  But don't rely on it.  That attitude
lets us expire it more aggressively.  There are "precious" properties
you can set, so for instance the registry marks are not lost, but from
our side we can't expect that.

Ted

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Thu Sep 29 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> On Thu, 29 Sep 2011 09:40:54 -0400 Dave Abrahams <dave@boostpro.com> wrote: 
>
> DA> [Aside: it seems like this territory is full of components that
> DA> could in principle be nicely generalized, but whose implementation
> DA> has only been made to work for one specific usage scenario]
>>> 
>>> I can't tell, I was not involved with writing these components.  
>
> DA> IIUC from reading elsewhere you rewrote the registry.
>
> I wrote *registry.el but I was talking about nnir.el, nnregistry.el, and
> other search functionality.  The registry is not a search engine so I
> thought you didn't mean it.  I'm not sure how it could be generalized
> and in what direction--tell me what you think.

Well, it's in the territory, but it's not one of those components.  I
don't have any complaints about the registry itself.  I was referring
more to things like gnus-warp-to-article, which only functions on a
single kind of group, or general usage of the registry, which only
happens in a very few circumstances even though the kind of queries the
registry can answer are all over the place.

>>> At least for the registry I know it's optional so we can't depend on
>>> it.
>
> DA> I'm not sure what "optional" means in this context.  Is Gnus going to
> DA> ship without the registry code?
>
> Users should be able to use every aspect of Gnus without calling
> `gnus-registru-initialize'.  The memory usage of the gnus-registry, if
> we enabled it by default, would be unpleasant for many of our users on
> older or less capable machines.  Thus any general Gnus functionality
> should not depend on the gnus-registry's availability.

Well, here's an example of a missing generalization then.  The registry
provides a general interface for looking up (information about) articles
based on their message IDs.  However, there's only one of them, and it's
persistent.  nnir would only need a small, special-purpose registry for
each of its groups.  Or, maybe better, one registry could be refcounted
and just discarded when you leave the last nnir group if you haven't
initialized the registry in the meantime.  It wouldn't grow if nobody
was putting things in it.

> Furthermore the gnus-registry should be treated as a lossy cache.  If
> you find something useful, great.  But don't rely on it.  That attitude
> lets us expire it more aggressively.  There are "precious" properties
> you can set, so for instance the registry marks are not lost, but from
> our side we can't expect that.

Sorry, don't know what you mean by "our side."  You can't make things
precious programmatically?

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Ted Zlatanov]

On Thu, 29 Sep 2011 10:40:50 -0400 Dave Abrahams <dave@boostpro.com> wrote: 

DA> general usage of the registry, which only happens in a very few
DA> circumstances even though the kind of queries the registry can
DA> answer are all over the place.

It's a fairly new component and I haven't pushed it hard on anyone.

>> Users should be able to use every aspect of Gnus without calling
>> `gnus-registru-initialize'.  The memory usage of the gnus-registry, if
>> we enabled it by default, would be unpleasant for many of our users on
>> older or less capable machines.  Thus any general Gnus functionality
>> should not depend on the gnus-registry's availability.

DA> Well, here's an example of a missing generalization then.  The registry
DA> provides a general interface for looking up (information about) articles
DA> based on their message IDs.  However, there's only one of them, and it's
DA> persistent.  nnir would only need a small, special-purpose registry for
DA> each of its groups.  Or, maybe better, one registry could be refcounted
DA> and just discarded when you leave the last nnir group if you haven't
DA> initialized the registry in the meantime.  It wouldn't grow if nobody
DA> was putting things in it.

That's not hard to do using registry.el, which is intended to be a
general-purpose storage facility with or without persistent.  But I
unfortunately don't have the time or need to do what you suggest.

>> Furthermore the gnus-registry should be treated as a lossy cache.  If
>> you find something useful, great.  But don't rely on it.  That attitude
>> lets us expire it more aggressively.  There are "precious" properties
>> you can set, so for instance the registry marks are not lost, but from
>> our side we can't expect that.

DA> Sorry, don't know what you mean by "our side."  You can't make things
DA> precious programmatically?

Sure, but choose not to.  I think it should be entirely up to the user
what he wants to keep around.

Ted

Re: nnir, gnus-goto-article and such

[Dave Abrahams]

on Thu Sep 29 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

> DA> Sorry, don't know what you mean by "our side."  You can't make things
> DA> precious programmatically?
>
> Sure, but choose not to.  I think it should be entirely up to the user
> what he wants to keep around.

My point was that I could, for example, make things precious for the
life of the ephemeral group that's using them, and the unmark them as
precious when the group goes away.  I'm just saying, there are plenty of
ways to avoid sticking the user with additional persistent storage.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com

Re: nnir, gnus-goto-article and such

[Ted Zlatanov]

On Thu, 29 Sep 2011 16:04:44 -0400 Dave Abrahams <dave@boostpro.com> wrote: 

DA> on Thu Sep 29 2011, Ted Zlatanov <tzz-AT-lifelogs.com> wrote:

DA> Sorry, don't know what you mean by "our side."  You can't make things
DA> precious programmatically?
>> 
>> Sure, but choose not to.  I think it should be entirely up to the user
>> what he wants to keep around.

DA> My point was that I could, for example, make things precious for the
DA> life of the ephemeral group that's using them, and the unmark them as
DA> precious when the group goes away.  I'm just saying, there are plenty of
DA> ways to avoid sticking the user with additional persistent storage.

I don't think you know the life of the ephemeral group.  But we can
definitely make the article precious artificially by giving it a
registry mark, those are always precious.

Just call `gnus-registry-set-article-mark-internal' with the registry
mark and a list of message *numbers* (or look inside the function for
the logic that works with message IDs).

Ted