Testers wanted: New gnus backend nnselect

Testers wanted: New gnus backend nnselect

[Andrew Cohen]

TLDR: Please test the new feature found on the git repository at
feature/gnus-select.

As you may have seen from some posts from me earlier, I have implemented
a new backend in gnus (nnselect) that allows an arbitrary list of
messages to form a group. Such groups can be ephemeral, or they can be
permanent (with all the rights and responsibilities of permanent
groups.) These messages can be from multiple "real" groups, on multiple
servers, and multiple backend types. (For example I have created groups
including messages from 2 different imap servers and several gmane
groups).  This works especially well (but is not limited to) searching.

Here are some examples of a few permanent groups that I use:

1. All messages received (on 3 different imap servers) in the last 2
   days.
2. All flagged messages (on 2 different imap servers)
3. All unseen messages (on 3 different imap servers)
4. All messages from some domain.
5. A list of "exceptional" messages that I just want to keep together in
   a prominent place.

and I'm sure you get the idea.

I have been using much of this code for several years, but only recently
put aside the time to clean it up a bit, and add some missing
functionality. Its a pretty invasive change to gnus so it needs
testing before pushing it to the main repository.

Why is it invasive? It is a generalization of the code I wrote many
years ago re-working the nnir backend. Consequently in this new format
nnir has CEASED to be a backend and is now just some code for searching
indices. Searching in gnus now works by creating an ephemeral nnselect
group where the list of articles is obtained from nnir.

I have so far mostly tested this with imap. Since it is essentially
backend independent it should just work, but this needs to be tested.

So please, please, give it some testing. The new code can be found in a
branch, feature/gnus-select, in the emacs git repository. Just check it
out, make, and give it a whirl.

Ideally if you just continue searching as you have always done you won't
notice much of a difference (except that referring threads is a bit
different---more on that later). This is the first thing that needs
testing.

Once you are confident that this is working you can try new
things. Firstly make some permanent nnselect groups. You do this as you
would for any group: start with "G m" (gnus-group-make-group) from the
Group buffer; give it a name; choose the method "nnselect"; it will then
query for a function; and then an argument. The function will be applied
to the argument (with funcall) to generate the list of articles. A good
choice for the function is nnir-run-query, and the argument can be an
appropriate search query (some examples below). It can also be any
function that returns a vector, each entry of which is a 3-vector formed
from [Group, Number, Score] (the group containing the actual article;
the number of the article in that group; a score, which is supposed to
help in sorting but is largely unused at this point---just set it to 1
if you are unsure). So for a silly example the function could be the
identity, and the arg this vector.

Here are some ready-made functions and args (sorry for the arcane
input---this will get simplified later):

function: nnir-run-query
args: ((nnir-query-spec
       (query . "FLAGGED")
       (criteria . ""))
      (nnir-group-spec
       (<server>)))

Replace <server> with the server you want to search in double quotes
(e.g. "nnimap:junk"). If you want to restrict to some groups or use
multiple servers see the front-matter in nnir.el for the syntax of
nnir-group-spec.

(NB: Inputting this from the minibuffer can be a pain. You can always
edit it later in the group parameters by using, e.g., "G p" on a group
line in the Group buffer.)

Here is a function for taking a number of days and performing a search
for articles from the past days on a couple of imap servers:

(defun my-recent-email (days)
  (let ((query-spec
         (list
          (cons 'query
                (format-time-string "SINCE %d-%b-%Y"
                                    (time-subtract (current-time)
                                                   (days-to-time days))))
          (cons 'criteria "")))
        (group-spec  (list (list "nnimap:server1") (list "nnimap:server2"))))
    (nnir-run-query   (cons 'nnir-specs
                            (list (cons 'nnir-query-spec query-spec)
                                  (cons 'nnir-group-spec group-spec))))))

To use it make an nnselect group with

Function: my-recent-email
Arg: 2

or change the 2 to however many days you prefer.


OK, this is long enough. I'll send a subsequent email with more info,
but let me know if you need assistance.

Best,
Andy

Re: Testers wanted: New gnus backend nnselect

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

> TLDR: Please test the new feature found on the git repository at
> feature/gnus-select.

Very cool! Later this week I'll make another copy of the search-language
branch that's rebased onto this one.

[...]

> Why is it invasive? It is a generalization of the code I wrote many
> years ago re-working the nnir backend. Consequently in this new format
> nnir has CEASED to be a backend and is now just some code for searching
> indices. Searching in gnus now works by creating an ephemeral nnselect
> group where the list of articles is obtained from nnir.

Once this has landed, I hope we can rename nnir.el to gnus-search.el,
and drop the "nnir" term altogether.

> I have so far mostly tested this with imap. Since it is essentially
> backend independent it should just work, but this needs to be tested.
>
> So please, please, give it some testing. The new code can be found in a
> branch, feature/gnus-select, in the emacs git repository. Just check it
> out, make, and give it a whirl.
>
> Ideally if you just continue searching as you have always done you won't
> notice much of a difference (except that referring threads is a bit
> different---more on that later). This is the first thing that needs
> testing.

I've started on a gnus-search-thread method for the search engines -- in
fact, the majority of them can already handle searching by thread.
Ideally `nnselect-request-thread' could then become engine-agnostic, and
only punt if the engine in question doesn't implement the
`gnus-search-thread' method.

> Once you are confident that this is working you can try new
> things. Firstly make some permanent nnselect groups. You do this as you
> would for any group: start with "G m" (gnus-group-make-group) from the
> Group buffer; give it a name; choose the method "nnselect"; it will then
> query for a function; and then an argument. The function will be applied
> to the argument (with funcall) to generate the list of articles. A good
> choice for the function is nnir-run-query, and the argument can be an
> appropriate search query (some examples below). It can also be any
> function that returns a vector, each entry of which is a 3-vector formed
> from [Group, Number, Score] (the group containing the actual article;
> the number of the article in that group; a score, which is supposed to
> help in sorting but is largely unused at this point---just set it to 1
> if you are unsure). So for a silly example the function could be the
> identity, and the arg this vector.

I think that, to maximize chances of people actually testing this, it
would be best to provide a ready-made
`gnus-group-make-persistent-search-group' function (or call it
whatever), and give it a keybinding. Say:

"G g": gnus-group-make-ephemeral-search-group
"G G": gnus-group-make-persistent-search-group

I know this or something like it is on the todo list, but I'd recommend
doing this first so people are more likely to give it a whirl...

Eric

Re: Testers wanted: New gnus backend nnselect

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    Eric> Andrew Cohen <cohen@bu.edu> writes:
    >> TLDR: Please test the new feature found on the git repository at
    >> feature/gnus-select.

    Eric> Very cool! Later this week I'll make another copy of the
    Eric> search-language branch that's rebased onto this one.

    Eric> [...]


[...]

    Eric> Once this has landed, I hope we can rename nnir.el to
    Eric> gnus-search.el, and drop the "nnir" term altogether.


Yes, that's what I meant when I said there was a lot of renaming to do:) 

[...]


    Eric> I've started on a gnus-search-thread method for the search
    Eric> engines -- in fact, the majority of them can already handle
    Eric> searching by thread.  Ideally `nnselect-request-thread' could
    Eric> then become engine-agnostic, and only punt if the engine in
    Eric> question doesn't implement the `gnus-search-thread' method.

Great! But there is one quirk to keep in mind. If you start in a
"normal" group (like one with an imap backend) you can't (easily) insert
articles from other groups into the same summary buffer. If you look at
nnimap-request-thread you will notice that if
gnus-refer-thread-use-search is nil then only the current group is
probed for thread articles. But if its non-nil then it defers to
nnselect-search-thread which creates a NEW nnselect ephemeral buffer to
allow the results to cross groups. You can probably implement this in 
new nnfoo-request-thread methods for all the backends.

The function nnselect-request-thread is called only when referring a
thread from an nnselect group. The only way it is engine specific right
now is in the construction of the search query string
(nnimap-make-thread-query). So you could provide a function
nnir-make-thread-query (or better yet encapsulate the whole nnir related
stuff in nnselect-request-thread into a function like nnir-refer-thread)
and this would make the function fully universal (and keep all the
search-related junk in nnir where it belongs)

Then this same function can be used in nnselect-search-thread,
easy-peasy.

Sorry, re-reading the above paragraphs I might not have been clear (but
maybe I was?) And I guess I'm suggesting the function should be called
gnus-refer-thread.


[...]

    Eric> I think that, to maximize chances of people actually testing
    Eric> this, it would be best to provide a ready-made
    Eric> `gnus-group-make-persistent-search-group' function (or call it
    Eric> whatever), and give it a keybinding. Say:

But it has a keybinding: the usual one!  "G m"
(gnus-group-make-group):)

OK, OK,  it will only take a second to make a dedicated function.
But at some point I hope people go beyond searching in the testing. 


[...]


Best,
Andy

Re: Testers wanted: New gnus backend nnselect

[Andrew Cohen]

>>>>> "Andrew" == Andrew Cohen <cohen@bu.edu> writes:

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]

    Andrew> Sorry, re-reading the above paragraphs I might not have been
    Andrew> clear (but maybe I was?) And I guess I'm suggesting the
    Andrew> function should be called gnus-refer-thread.

And of course I meant gnus-search-refer-thread :)

Re: Testers wanted: New gnus backend nnselect

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

[...]

> Great! But there is one quirk to keep in mind. If you start in a
> "normal" group (like one with an imap backend) you can't (easily) insert
> articles from other groups into the same summary buffer. If you look at
> nnimap-request-thread you will notice that if
> gnus-refer-thread-use-search is nil then only the current group is
> probed for thread articles. But if its non-nil then it defers to
> nnselect-search-thread which creates a NEW nnselect ephemeral buffer to
> allow the results to cross groups. You can probably implement this in 
> new nnfoo-request-thread methods for all the backends.
>
> The function nnselect-request-thread is called only when referring a
> thread from an nnselect group. The only way it is engine specific right
> now is in the construction of the search query string
> (nnimap-make-thread-query). So you could provide a function
> nnir-make-thread-query (or better yet encapsulate the whole nnir related
> stuff in nnselect-request-thread into a function like nnir-refer-thread)
> and this would make the function fully universal (and keep all the
> search-related junk in nnir where it belongs)
>
> Then this same function can be used in nnselect-search-thread,
> easy-peasy.
>
> Sorry, re-reading the above paragraphs I might not have been clear (but
> maybe I was?) And I guess I'm suggesting the function should be called
> gnus-refer-thread.
>
> And of course I meant gnus-search-refer-thread :)

My eyes did cross for a second there, reading the above :)

But I think I've got the basics, and will have a look at the details
over the next week. My goal is definitely to make the search code
completely backend agnostic, so that all nnselect-request-thread has to
do is call a single function.

What I'm thinking now is to make thread search a part of the basic query
language. In fact, there would be a few "meta search terms" that could
be specified along with the base query. "thread:t" would be one of them,
along with "count:t", "limit:NN", and "sort:<field>,<field>" (obviously
they can't all be used in conjunction).

The syntax for searching for a single message's thread would be
"thread:t id:<sdfsdf@sdf>". That would get parsed into:

('nnir-query-spec ('query ((id . "<sdfsdf@sdf>")))
                  ('thread t))

It would be up to the search engines to make sense of that.

Eric

Re: Testers wanted: New gnus backend nnselect

[Harry Putnam]

Andrew Cohen <cohen@bu.edu> writes:

> TLDR: Please test the new feature found on the git repository at
> feature/gnus-select.
>

First off. TLDR?

Sorry to blunder into something that is well over my head.  But I am
quite interested in searching inside gnus.

I've haven't attempted using nnir since a number of years ago and have
forgotten whatever I knew about it then.

I am a long time emacs/gnus user and have used gnus nearly exclusively
since early `Quassia' gnus. But have never really tried to learn enough
elisp etc to participate in dev.

Sorry if this next query is well known by everybody:

It is not clear (to me) how I might get this code incorporated into a
fresh version of emacs-dev (26.X) and try to figure out how to use it.

I normally (since gnus-dev was merged into main emacs-dev) just `git pull'
on my emacs branch for the newest stuff in gnus.

You mentioned its location on git repo... but I'm not sure of the
mechanics to get it.

What would I do differently to get to this work in progress the
discussion is about, feature/gnus-select?

Re: Testers wanted: New gnus backend nnselect

[Eric Abrahamsen]

Harry Putnam <reader@newsguy.com> writes:

> Andrew Cohen <cohen@bu.edu> writes:
>
>> TLDR: Please test the new feature found on the git repository at
>> feature/gnus-select.
>>
>
> First off. TLDR?
>
> Sorry to blunder into something that is well over my head.  But I am
> quite interested in searching inside gnus.
>
> I've haven't attempted using nnir since a number of years ago and have
> forgotten whatever I knew about it then.
>
> I am a long time emacs/gnus user and have used gnus nearly exclusively
> since early `Quassia' gnus. But have never really tried to learn enough
> elisp etc to participate in dev.
>
> Sorry if this next query is well known by everybody:
>
> It is not clear (to me) how I might get this code incorporated into a
> fresh version of emacs-dev (26.X) and try to figure out how to use it.
>
> I normally (since gnus-dev was merged into main emacs-dev) just `git pull'
> on my emacs branch for the newest stuff in gnus.
>
> You mentioned its location on git repo... but I'm not sure of the
> mechanics to get it.
>
> What would I do differently to get to this work in progress the
> discussion is about, feature/gnus-select?

Basically, go into your emacs repo, run "git checkout
feature/gnus-select", then run "make".

Probably you don't want to run "make install", so then from the
top-level dir of the emacs repo run "src/emacs" to start that version of
emacs. That ought to be it...

Happy testing!
Eric

Re: Testers wanted: New gnus backend nnselect

[Harry Putnam]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]

Harry wrote:
>> What would I do differently to get to this work in progress the
>> discussion is about, feature/gnus-select?

Eric replied:
> Basically, go into your emacs repo, run "git checkout
> feature/gnus-select", then run "make".

Thanks... I did that and got this response:

 $ git checkout feature/gnus-select

  Branch feature/gnus-select set up to track remote branch
  feature/gnus-select from origin.  Switched to a new branch
  'feature/gnus-select'

I didn't really understand what it meant but suspect it is just what I
wanted.

And now I guess for future I would go back to `git pull' and any changes
in feature/gnus-select would be pulled too?

Or do I need to do something different?

Re: Testers wanted: New gnus backend nnselect

[Eric Abrahamsen]

Harry Putnam <reader@newsguy.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> [...]
>
> Harry wrote:
>>> What would I do differently to get to this work in progress the
>>> discussion is about, feature/gnus-select?
>
> Eric replied:
>> Basically, go into your emacs repo, run "git checkout
>> feature/gnus-select", then run "make".
>
> Thanks... I did that and got this response:
>
>  $ git checkout feature/gnus-select
>
>   Branch feature/gnus-select set up to track remote branch
>   feature/gnus-select from origin.  Switched to a new branch
>   'feature/gnus-select'
>
> I didn't really understand what it meant but suspect it is just what I
> wanted.

It is!

> And now I guess for future I would go back to `git pull' and any changes
> in feature/gnus-select would be pulled too?
>
> Or do I need to do something different?

Nope, that's correct. "git pull" to fetch new changes, followed by
another "make". If you want to get back to master at any point, just
"git checkout master", and "make" again.

Thanks for testing!

Re: Testers wanted: New gnus backend nnselect

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Thursday, 27 Apr 2017 at 16:03, Harry Putnam wrote:
>> Andrew Cohen <cohen@bu.edu> writes:
>>
>>> TLDR: Please test the new feature found on the git repository at
>>> feature/gnus-select.
>>>
>>
>> First off. TLDR?
>
> Too long, didn't read.  (or don't read)  :-)

nnselect is basically a much more generalized version of nnvirtual: it
lets you create groups holding messages from other groups. That could
mean all the messages from other groups (the way nnvirtual works now),
or messages based on arbitrary criteria. It's also the new incarnation
of nnir, so it serves as the backend that holds search results.

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Friday, 28 Apr 2017 at 01:18, Andrew Cohen wrote:
>> It should work, but its totally untested (I don't use the agent). The
>> nnselect backend (which is independent of searching) should act like any
>> backend and use the agent as appropriate (although since it hasn't been
>> tested it is probably buggy). The searching part should behave as (the
>> other) Eric indicated: searches in local indices should work fine, and
>> searches that require a network connection should signal an error.
>
> I don't have any local index (as far as I know).  I simply enable the
> agent to ensure I can read my emails off-line (I spend 2-3 hours a day
> off-line while commuting).

This is IMAP, right? Anyway, no, the search won't work. I guess I'd
expect it to behave the same as it behaves now. I've toyed with the idea
of using the agent and the registry to create a sort of limited
ephemeral index of messages seen in this Gnus session, but that's mostly
daydreaming.

>> It would be helpful if you could checkout feature/gnus-select and see
>> what happens :)
>
> Is this from the emacs repository or is there a separate gnus one?  In
> either case, where is it located please?

Yes, it's in the base emacs repo, the feature/gnus-select branch.

Eric

Re: Testers wanted: New gnus backend nnselect

[Harry Putnam]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]

Harry wrote:
>> And now I guess for future I would go back to `git pull' and any changes
>> in feature/gnus-select would be pulled too?
>>
>> Or do I need to do something different?

Eric replied:
> Nope, that's correct. "git pull" to fetch new changes, followed by
> another "make". If you want to get back to master at any point, just
> "git checkout master", and "make" again.

All right! Thanks much, and this went miles toward a better grip on git
too!

Re: Testers wanted: New gnus backend nnselect

[Lars Ingebrigtsen]

Andrew Cohen <cohen@bu.edu> writes:

> Here are some examples of a few permanent groups that I use:
>
> 1. All messages received (on 3 different imap servers) in the last 2
>    days.
> 2. All flagged messages (on 2 different imap servers)
> 3. All unseen messages (on 3 different imap servers)
> 4. All messages from some domain.
> 5. A list of "exceptional" messages that I just want to keep together in
>    a prominent place.

Sounds really cool.  Will this get merged at any point?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no