any partial or rudamentary documentation of new nnselect work somewhere?

any partial or rudamentary documentation of new nnselect work somewhere?

[Harry Putnam]

I'd like to get my hands on some bit of docu for the new nnselect work
that is being done... I realize it is not done and won't be for some
time.

But I need a start of some kind.  Such things as the search syntax
and something explaining the basics of how the search terms need to
work with many backends.  That is, some big picture explanations and
possibly as many actual examples as anyone has time to write.

Something as rough as some of the experts working on nnselect might
save there test command lines off to a file ... even with no
explanatory remarks and make that available somewhere.

I've followed the discussion to the best of my ability... but mostly
its over my head.

I would still like to do some testing and I suspect the work needs to
have a few non-expert testers at some point to make sure the new work
is usable to some of us dim wits too.

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Andrew Cohen]

Dear Harry (and others)

>>>>> "Harry" == Harry Putnam <reader@newsguy.com> writes:

    Harry> I'd like to get my hands on some bit of docu for the new
    Harry> nnselect work that is being done... I realize it is not done
    Harry> and won't be for some time.

Actually, almost done :)

    Harry> But I need a start of some kind.  Such things as the search
    Harry> syntax and something explaining the basics of how the search
    Harry> terms need to work with many backends.  That is, some big
    Harry> picture explanations and possibly as many actual examples as
    Harry> anyone has time to write.

First, nnselect and the new universal search syntax are two different
things. The feature branch in git feature/gnus-select contains the
nnselect work but not the new search language. So if you check out this
branch you should be able to continue working with essentially no change
in anything: if all you are doing is searching that will work the same
way as before (it will use the nnselect backend behind the scenes but
shouldn't behave markedly differently from the way things behaved
earlier). It would be great if you could try this out and report any
bugs.

MAKE A BACKUP OF .newsrc.eld JUST IN CASE!

nnselect allows doing lots of other things as well, but you can worry
about those later once there is some documentation in place. Or you can
ask me and I can send you some recipes.

    Harry> Something as rough as some of the experts working on nnselect
    Harry> might save there test command lines off to a file ... even
    Harry> with no explanatory remarks and make that available
    Harry> somewhere.

That's easy---there are no special commands :) For basic search just "G
G" as usual. Or if you want to create a permanent search group (that is,
one that behaves like any other group but the selected articles are the
result of a search query) use "G g".

    Harry> I've followed the discussion to the best of my ability... but
    Harry> mostly its over my head.

    Harry> I would still like to do some testing and I suspect the work
    Harry> needs to have a few non-expert testers at some point to make
    Harry> sure the new work is usable to some of us dim wits too.

Definitely! First I would like to make sure nnselect isn't going to do
anything bad (not likely, but needs testing). This should be easy since
users don't need to learn anything new to test the basic structure.

Once that is working correctly we will merge in the universal search
language that Eric has developed. This will require users to change the
syntax of their searches and might have a bit of a learning curve (but
only a bit---the language is clear and simple so it shouldn't pose any
difficulties).

Best,
Andy

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Harry Putnam]

Andrew Cohen <cohen@bu.edu> writes:

>     Harry> Something as rough as some of the experts working on nnselect
>     Harry> might save there test command lines off to a file ... even
>     Harry> with no explanatory remarks and make that available
>     Harry> somewhere.

> That's easy---there are no special commands :) For basic search just "G
> G" as usual. Or if you want to create a permanent search group (that is,
> one that behaves like any other group but the selected articles are the
> result of a search query) use "G g".

[I really don't want to bog down or belabor something when such a
vigorous and productive spate of coding is underway, so maybe you can
just post urls or direct me to the proper place to get started]

In emacs from source of a few days ago (and with feature/gnus-select
pulled in too):

OK, I guess I'm thoroughly lost here... Your comments seem to indicate
there is existing nnselect doucumentation some where. Or at least that
the functionality of existing code is known to gnus.

At the top of gnus manual... (The one that comes with emacs sources
when one sets up an emacs branch from git repo)

  That manual says it belongs to version 5.13 ... but isn't that quite
  old?

Anyway:

Using regex search `s' and entering `nnselect', there are no hits at all.

In gnus...
Entering a group and pressing  `G G'... I am prompted for an article number.

Surely that is not where one enters search terms, since it is asking
for an article number.  But even if I were
to find something explaining how to set up and use nnselect... I'd
still need to know how to enter a search. That is,  the syntax.

I'm guessing one does not use a regex like in `M-s' in group buffer,
type searches.

If this information is readily available please just respond with RTFM
or the like depending on where it is.

I don't want to waste your time.

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Andrew Cohen]

Dear Harry:

>>>>> "Harry" == Harry Putnam <reader@newsguy.com> writes:

    Harry> Andrew Cohen <cohen@bu.edu> writes: Something as rough as
    Harry> some of the experts working on nnselect might save there test
    Harry> command lines off to a file ... even with no explanatory
    Harry> remarks and make that available somewhere.

    >> That's easy---there are no special commands :) For basic search
    >> just "G G" as usual. Or if you want to create a permanent search
    >> group (that is, one that behaves like any other group but the
    >> selected articles are the result of a search query) use "G g".


[...]

    Harry> In emacs from source of a few days ago (and with
    Harry> feature/gnus-select pulled in too):

    Harry> OK, I guess I'm thoroughly lost here... Your comments seem to
    Harry> indicate there is existing nnselect doucumentation some
    Harry> where. Or at least that the functionality of existing code is
    Harry> known to gnus.


[...]


Sorry, I hadn't realized that you weren't using search previously. First
there are two "kinds" of searching---in a summary buffer you can search
(in a variety of ways, including the regex search you were trying); or
you can query servers to find articles matching certain search terms. In
this thread we are only talking about the latter search facility. It is
usually referred to as "nnir" searching. It is documented in the gnus
manual under "Searching" (go to info, select gnus, and select the
searching node). This should give you the information you need to start
searching (you might have to do some configuration depending on what
backends you are using).

In the various threads going on currently on this mailing list we are
talking about changes and extensions to this facility. There are two
independent things:

1. Introduce a new backend, nnselect, that among other uses will replace
   the part of nnir that manages the "gnus" related handling of articles
   found as part of a search.

2. Introduce a new, universal, query language for searching for
   articles.

Item 1. can be tested without modifying anything (that is, if you have
nnir searching set up as in the manual then using the new code from
feature/gnus-select should mostly just work without any new
configuration).

Item 2. is in a separate feature branch---pulling in feature/gnus-select
will NOT currently alter the way searches are done.


So the first step in any testing is to make sure that nnir-based
searching is working for you PRIOR to testing the new stuff. I would
suggest reading the manual to get started.

Best,
Andy

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Harry Putnam]

Andrew Cohen <cohen@bu.edu> writes:

[...]

> Sorry, I hadn't realized that you weren't using search previously. First
> there are two "kinds" of searching---in a summary buffer you can search
> (in a variety of ways, including the regex search you were trying); or
> you can query servers to find articles matching certain search terms. In
> this thread we are only talking about the latter search facility. It is
> usually referred to as "nnir" searching. It is documented in the gnus
> manual under "Searching" (go to info, select gnus, and select the
> searching node). This should give you the information you need to start
> searching (you might have to do some configuration depending on what
> backends you are using).

[...]

Sorry, I should have explained the massive holes in my knowledge of gnus.

Some yrs ago now, I wrote my own search engine in perl... I mean just
a perl script that searched for regex thru thousands of gmane messages
that I had downloaded with the agent.  So much downloading that I was
kicked off gmane a few times... probably looked like something
sinister.

I could search on exact code in perl groups for example ... and return
exact bits of code with careful use of perl regex.

Later on I added more (perl) code that created a directory and filled
it with symlinks to the actual messages where my regex found hits. I
could then pull that directory into gnus as an ephemeral group or
nndoc inside gnus.

All very clunky and probably duplicating functionality already in gnus
but in a primitive and homeboy way.  Also very slow.

Hardcore regex searching of that many messages is rather slow.  And
I'm not a good enough perl coder to know how to speed it up. The
only upside is that it is very exact.

At the high point I had used the agent to collect well over 200,000
messages ... no longer remember how many, but the slowness of searches
eventually led me to fade out on all that search activity. Not to
mention that I quit getting barred on gmane too.

So haven't really worked at it for several years.  And never really
learned the gnus capabilities in nnir, namzu, swish, and  etc.

All the new efforts are pretty exciting and has gotten me interested
once again.

Many thanks for the nice summaries and big picture guidance.

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Eric Abrahamsen]

Harry Putnam <reader@newsguy.com> writes:

> Andrew Cohen <cohen@bu.edu> writes:
>
> [...]
>
>> Sorry, I hadn't realized that you weren't using search previously. First
>> there are two "kinds" of searching---in a summary buffer you can search
>> (in a variety of ways, including the regex search you were trying); or
>> you can query servers to find articles matching certain search terms. In
>> this thread we are only talking about the latter search facility. It is
>> usually referred to as "nnir" searching. It is documented in the gnus
>> manual under "Searching" (go to info, select gnus, and select the
>> searching node). This should give you the information you need to start
>> searching (you might have to do some configuration depending on what
>> backends you are using).
>
> [...]
>
> Sorry, I should have explained the massive holes in my knowledge of gnus.
>
> Some yrs ago now, I wrote my own search engine in perl... I mean just
> a perl script that searched for regex thru thousands of gmane messages
> that I had downloaded with the agent.  So much downloading that I was
> kicked off gmane a few times... probably looked like something
> sinister.
>
> I could search on exact code in perl groups for example ... and return
> exact bits of code with careful use of perl regex.
>
> Later on I added more (perl) code that created a directory and filled
> it with symlinks to the actual messages where my regex found hits. I
> could then pull that directory into gnus as an ephemeral group or
> nndoc inside gnus.
>
> All very clunky and probably duplicating functionality already in gnus
> but in a primitive and homeboy way.  Also very slow.

It does duplicate what's possible in Gnus (this could be done even with
Gnus as it stands now), using an external indexing program like namazu
or notmuch. The one big caveat would be that these programs almost
certainly wouldn't index perl code correctly -- I haven't tried it, but
I'd be very surprised if they did it right.

The generalized search functionality that I'm working on (built on
Andy's nnselect branch) would let you create all kinds of weird search
engine behavior. In theory it wouldn't be hard at all to create a "grep"
mixin class for the indexed search engines, that accepted an additional
"grep:" key, and ran the grep over the results of the initial search.
Ie, you could enter the search string:

subject:"having a problem" grep:"my ($smith, $jones) = @a;" since:3m

The subject: and since: keys would be passed to the underlying search
engine (notmuch or mairix, etc), which would return a first round of
results very quickly. The results are in the form of filenames, so the
grep: key could then be used to do a second pass over those files. A
best-of-both-worlds situation.

That sounds pretty useful. Maybe there's no need for a mixin class at
all; this could be a standard feature of the indexed search class. If
grep wasn't available on the system, you'd just get a polite note to
that effect.

Actually, there's already a regexp syntax: surrounding text with forward
slashes. Search engines that don't handle regexps could simply transform
"body:/(.*)/" into "grep:(.*)"...

Hmmm...

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Harry Putnam]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Harry Putnam <reader@newsguy.com> writes:

[...]

>> I could search on exact code in perl groups for example ... and return
>> exact bits of code with careful use of perl regex.
>>
>> Later on I added more (perl) code that created a directory and filled
>> it with symlinks to the actual messages where my regex found hits. I
>> could then pull that directory into gnus as an ephemeral group or
>> nndoc inside gnus.
>>
>> All very clunky and probably duplicating functionality already in gnus
>> but in a primitive and homeboy way.  Also very slow.
>
> It does duplicate what's possible in Gnus (this could be done even with
> Gnus as it stands now), using an external indexing program like namazu
> or notmuch. The one big caveat would be that these programs almost
> certainly wouldn't index perl code correctly -- I haven't tried it, but
> I'd be very surprised if they did it right.

That was roughly my thinking at the time...(about indexing) I
considered trying to incorporate one or another of the indexing search
tools available.

But, first off, I didn't know how to do that... but might have fumbled
my way thru.

But, second and more important... indexing perl or other kinds of
code; I new would be a loser. And that was my primary objective at
the time.

Back then there was a tool called glimpse that was still under
development (although, barely) that does have a near regex capability
and is also an indexing tool... but once again the indexing doesn't do
well with code.

So, decided to put up with slowness of full-fledged perl regex
searching.  Decided indexing just didn't go with the kind of searchs I
was interested in.

You would know vastly more than I about the capabilities of any of the
search tools available now. But I believe you are right in thinking
they would not index code very well at all.

> The generalized search functionality that I'm working on (built on
> Andy's nnselect branch) would let you create all kinds of weird search
> engine behavior. In theory it wouldn't be hard at all to create a "grep"
> mixin class for the indexed search engines, that accepted an additional
> "grep:" key, and ran the grep over the results of the initial search.
> Ie, you could enter the search string:
>
> subject:"having a problem" grep:"my ($smith, $jones) = @a;" since:3m
>
> The subject: and since: keys would be passed to the underlying search
> engine (notmuch or mairix, etc), which would return a first round of
> results very quickly. The results are in the form of filenames, so the
> grep: key could then be used to do a second pass over those files. A
> best-of-both-worlds situation.
>
> That sounds pretty useful. Maybe there's no need for a mixin class at
> all; this could be a standard feature of the indexed search class. If
> grep wasn't available on the system, you'd just get a polite note to
> that effect.

> Actually, there's already a regexp syntax: surrounding text with forward
> slashes. Search engines that don't handle regexps could simply transform
> "body:/(.*)/" into "grep:(.*)"...
>
> Hmmm...

Yes, yes, and away you go... That two phase bit sounds really
useful.  I am really sorry to be of no help at all.

If you fellows have any low level scut work that needs doing, and I do
mean `low level'... that is, no coding required.

I realize that is a piss poor offer, and there probably is nothing
like that involved... but still ... just in case .. the offer stands.

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Eric Abrahamsen]

Harry Putnam <reader@newsguy.com> writes:

> You would know vastly more than I about the capabilities of any of the
> search tools available now. But I believe you are right in thinking
> they would not index code very well at all.

"Vastly" is an overstatement :) But I think we can assume it's not going
to work.

[...]

>> That sounds pretty useful. Maybe there's no need for a mixin class at
>> all; this could be a standard feature of the indexed search class. If
>> grep wasn't available on the system, you'd just get a polite note to
>> that effect.
>
>> Actually, there's already a regexp syntax: surrounding text with forward
>> slashes. Search engines that don't handle regexps could simply transform
>> "body:/(.*)/" into "grep:(.*)"...
>>
>> Hmmm...
>
> Yes, yes, and away you go... That two phase bit sounds really
> useful.  I am really sorry to be of no help at all.
>
> If you fellows have any low level scut work that needs doing, and I do
> mean `low level'... that is, no coding required.
>
> I realize that is a piss poor offer, and there probably is nothing
> like that involved... but still ... just in case .. the offer stands.

Thinking about it further, I think automatically translating regexps
into "grep:" might be going overboard, and would introduce unnecessary
complications. But providing the grep functionality itself still sounds
like a good idea.

And thanks for the offer of help! It's true there's not a whole lot to
be done at this point, just a bit of leftover functionality, then tests
and docs. What would be most useful is if someone could explain git
better to me so that the rebase process was less bewildering :(

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Harry Putnam]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> What would be most useful is if someone could explain git
> better to me so that the rebase process was less bewildering :(

If no one responds to that... (and you've no doubt considered this)
But, you could take the query to gmane.comp.version-control.git

There are some heavy hitters there that can probably explain things
thouroughly.

The name of the group does not suggest it is a devel group but that
does appear to be mostly what is going on there.  None the less, I've
seen many detailed questions answered there.  Well out of my league.
But there are many true experts there that appear willing to answer
complicated question.

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Harry Putnam]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Thinking about it further, I think automatically translating regexps
> into "grep:" might be going overboard, and would introduce unnecessary
> complications. But providing the grep functionality itself still sounds
> like a good idea.

Anything that functions grep like for that second pass would really be
great.

I have no idea what coding that would look like, but hope it is not
too daunting or something that will bog things down.

Re: any partial or rudamentary documentation of new nnselect work somewhere?

[Eric Abrahamsen]

Harry Putnam <reader@newsguy.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Thinking about it further, I think automatically translating regexps
>> into "grep:" might be going overboard, and would introduce unnecessary
>> complications. But providing the grep functionality itself still sounds
>> like a good idea.
>
> Anything that functions grep like for that second pass would really be
> great.
>
> I have no idea what coding that would look like, but hope it is not
> too daunting or something that will bog things down.

No, it shouldn't be too difficult. The guts of the code already exist in
the find-grep engine, which I kind of doubt anyone's using, but which
can be refactored to use this same mechanism.