[RFC] Gnus generalized search, part II

[RFC] Gnus generalized search, part II

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 5782 bytes --]

I've been working on a generalized searching for Gnus, where a single
query language gets translated into different search-engine-appropriate
strings. This allows searching multiple backends at once. It's more or
less working, and I'm attaching the new version of the nnir.el file here
which can be used as a drop-in replacement for the existing file.
Ideally, if accepted, this would get rebased on top of Andy Cohen's
reworking of nnir/nnselect.

How it works:

The query entered by the user is parsed into a sexp structure, and then
each engine is responsible for interpreting that.

For instance, you mark one IMAP group, and one maildir group (indexed
with notmuch). Then you enter a query:

"from:john after:1w or -mark:!"

Internally, this becomes:
((from . "john") (or (since 14 4 2017) (not (mark . "flag"))))

The imap engine turns that into:
"FROM john OR SINCE 14-Apr-2017 UNFLAGGED"

And the notmuch engine turns it into:
"from:john date:4/14/2017.. or not tag:flag"

Results from both servers are put in the same summary buffer.

That's pretty much it, I hope people will be interested in this. I've
started writing tests, and will do documentation if this is accepted.
I've pasted the complete docstring of nnir-search-parse-query below.

---------------------
Notes for the curious:

The search engines are now implemented as classes. This allowed for
factoring out a bunch of common code.

I nearly set this up for running multiple searches each in their own
thread, allowing for limited search concurrency. I backed off at the
last minute because of weird IMAP behavior, but the code is pretty much
set up for threads, if IMAP can get sorted out.

I re-implemented a limited version of the IMAP LITERAL+ code I wrote
years ago. If the server advertises support, searches for non-ASCII
strings will make use the LITERAL+ mechanism. ¡¡Turning this on enforces
CHARSET UTF-8!! Ie, the assumption is that if a server can handle
LITERAL+, it can handle CHARSET UTF-8. This is probably totally wrong,
but it would be easy to shut off, or fix if I can figure out how to
DTRT.

So far as I can tell, Hyrex and Swish-e are defunct. They're still in
there, but their search transformation is lacking because there are no
good docs.

Namazu docs are also lacking: they give the examples of searching on
"message-id", "from", and "subject" headers, but are there more? I don't
know. I can't test because mknmz errors on my machine.

Things I'd like to add:

1. Support for IMAP MULTISEARCH and FUZZY
2. A command to automatically update all engine indexes.
3. Regular expression searches for engines that support them.
4. Engines for lucene, solr, raw xapian, sphinx... What else are people
   using? There's a base class for locally-indexed search engines, so
   these should be easy to add.
5. Create an offline index of gmane messages, to be updated monthly. The
   gmane search engine would search locally but request remotely (only
   partly joking).

------------------------------
nnir-search-parse-query is a Lisp closure.

(nnir-search-parse-query STRING)

Turn STRING into an s-expression based query.

The resulting query structure is passed to the various search
backends, each of which adapts it as needed.

The search "language" is essentially a series of key:value
expressions.  Key is most often a mail header, but there are
other keys.  Value is a string, quoted if it contains spaces.
Key and value are separated by a colon, no space.  Expressions
are implictly ANDed; the "or" keyword can be used to
OR. "not" will negate the following expression, or keys can be
prefixed with a "-".  The "near" operator will work for
engines that understand it; other engines will convert it to
"or".  Parenthetical groups work as expected.

A key that matches the name of a mail header will search that
header.

Search keys can be abbreviated so long as they remain
unambiguous, ie "f" will search the "from" header. "s" will raise an
error.

Other keys:

"address" will search all sender and recipient headers.

"recipient" will search "To", "Cc", and "Bcc".

"before" will search messages sent before the specified
date (date specifications to come later).  Date is exclusive.

"after" (or its synonym "since") will search messages sent
after the specified date.  Date is inclusive.

"mark" will search messages that have some sort of mark.
Likely values include "flag", "seen", "read", "replied".
It’s also possible to use Gnus’ internal marks, ie "mark:R"
will be interpreted as mark:read.

"tag" will search tags -- right now that’s translated to
"keyword" in IMAP, and left as "tag" for notmuch. At some
point this should also be used to search marks in the Gnus
registry.

"contact" will search messages to/from a contact.  Contact
management packages must push a function onto
‘nnir-search-contact-sources’, the docstring of which see, for
this to work.

"contact-from" does what you’d expect.

"contact-to" searches the same headers as "recipient".

Other keys can be specified, provided that the search backends
know how to interpret them.

Date values (any key in ‘nnir-search-date-keys’) can be provided
in any format that ‘parse-time-string’ can parse (note that this
can produce weird results).  Dates with missing bits will be
interpreted as the most recent occurance thereof (ie "march 03"
is the most recent March 3rd).  Lastly, relative specifications
such as 1d (one day ago) are understood.  This also accepts w, m,
and y.  m is assumed to be 30 days.

This function will accept pretty much anything as input. Its only job is
to parse the query into a sexp, and pass that on -- it is the job of the
search backends to make sense of the structured query. Malformed,
unusable or invalid queries will typically be silently ignored.

[-- Attachment #2: nnir.el --]
[-- Type: application/emacs-lisp, Size: 89882 bytes --]

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

Dear Eric:

This is great (although I haven't tested it yet:))

Some questions/comments:

The previous version of nnir that I contributed could search multiple
backends (indeed something I did routinely with messages threads that
were spread out across gmane and my personal email which in turn was a
combination of imap and namazu) but it was cumbersome since it worked
only with queries that were understood simultaneously by both
backends. This effectively reduced the searches to rather simple ones,
but it was nevertheless pretty effective. One aspect of the design that
was important was that C-g while searching wouldn't abort the whole
thing but just the current backend. So if the gmane search web site
wasn't responding, for example, a C-g would move things along and still
search the other backends. Can you make sure that your modifications do
the same? (They probably do, but I just want to make sure:))

I occasionally do some complicated searching which needs access to the
raw imap syntax (and indeed the imap thread referral uses the ability of
nnir-run-query to accept a raw imap search constructed in
'nnimap-make-thread-query) so I think it is important for nnir-run-query
to accept this format as well. I know I previously said it would be good
if the whole criteria thing goes away, but perhaps I was too
hasty. Right now a prefix arg to the gnus entry point to searching sets
the criteria in a complicated fashion; we could simplify this so that a
prefix arg skips all parsing of the query-spec and sends the raw query
directly to the query engine (as is the case now if you use a criterion
of 'imap). The user would presumably not try this with multiple backends
since the syntax wouldn't be common.

My limited window to work on this is rapidly closing. The good news is
that I think that all of the nnselect stuff is done (aside from some
renaming of things) and seems to be working. The bad news is that after
weeks of trying I still have no git access :( I have sent 2 emails to
this list, and on the advice of several people I have signed up for a
savannah account and requested membership in emacs, but its been almost
a week with no response.

The changes are pretty large and somewhat invasive, and since there are
likely to be bugs found in testing I expect some back and forth with
testers and subsequent modifications in the code. I doubt I have the
wherewithal to do this without using git. So if I can't get access soon
I may have to forgo pushing these changes.

Best,
Andy

Re: [RFC] Gnus generalized search, part II

[Eli Zaretskii]

> From: Andrew Cohen <cohen@bu.edu>
> Date: Sat, 22 Apr 2017 08:16:08 +0800
> Cc: ding@gnus.org
> 
> The bad news is that after weeks of trying I still have no git
> access :( I have sent 2 emails to this list, and on the advice of
> several people I have signed up for a savannah account and requested
> membership in emacs, but its been almost a week with no response.

I approved you now.  Sorry for the delay; John usually handles these
requests, but I guess he's still not out of the woods with his email
backlog.

(In the future, and for others' knowledge as well: if your schedule is
for some reason tight, please tell that explicitly in your request, so
that those who read the request could act in a timely manner.)

Thanks.

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

    >> From: Andrew Cohen <cohen@bu.edu> Date: Sat, 22 Apr 2017 08:16:08
    >> +0800 Cc: ding@gnus.org
    >> 
    >> The bad news is that after weeks of trying I still have no git
    >> access :( I have sent 2 emails to this list, and on the advice of
    >> several people I have signed up for a savannah account and
    >> requested membership in emacs, but its been almost a week with no
    >> response.

    Eli> I approved you now.  Sorry for the delay; John usually handles
    Eli> these requests, but I guess he's still not out of the woods
    Eli> with his email backlog.

Thanks very much.  I completely understand. I noticed that John hasn't
posted much recently on the list and assumed he must be busy. 

    Eli> (In the future, and for others' knowledge as well: if your
    Eli> schedule is for some reason tight, please tell that explicitly
    Eli> in your request, so that those who read the request could act
    Eli> in a timely manner.)

You are right, I should have mentioned it. But the upside is I've had
more time on my own to ensure that all my code is entirely bug free :)

Best,
Andy

Re: [RFC] Gnus generalized search, part II

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> The query entered by the user is parsed into a sexp structure, and then
> each engine is responsible for interpreting that.

I think this sounds like a good approach.  I haven't tried the code
myself, but I skimmed it briefly and it looks good to me.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> The query entered by the user is parsed into a sexp structure, and then
>> each engine is responsible for interpreting that.
>
> I think this sounds like a good approach.  I haven't tried the code
> myself, but I skimmed it briefly and it looks good to me.  :-)

Cool! Glad it's acceptable in theory. I am hoping that people who are
likely to care about this stuff will argue about the search language
syntax a bit. That will affect users the most, and also be the most
annoying to change subsequently. The rest is just bugs :)

Re: [RFC] Gnus generalized search, part II

[Dan Christensen]

Looks interesting.  Two questions:

Will mairix be supported?

How do you handle differing capabilities of the search backends?
E.g. mairix supports substring searches, and fuzzy matches, which
are handy if you are looking for a word that might be singular or
plural, or might be a noun or a verb, for example.

Dan

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Dan Christensen <jdc@uwo.ca> writes:

> Looks interesting.  Two questions:
>
> Will mairix be supported?

Sure! At least, I think so -- I'm not clear why nnmairix is its own
server, rather than being an nnir engine. I think if it's used in raw
mode it should work just like the other engines. At any rate, as many
engines as possible should be supported.

> How do you handle differing capabilities of the search backends?
> E.g. mairix supports substring searches, and fuzzy matches, which
> are handy if you are looking for a word that might be singular or
> plural, or might be a noun or a verb, for example.

This is something I haven't figured out yet, and wanted to get feedback
from people here. Obviously a lot of engines support
regexp/substring/fuzzy searching (even IMAP with the FUZZY capability),
but I haven't come up with a good way of indicating that yet.

I guess two distinct things need to be supported: regexps, and substring
matches. Off the top of my head we could do:

"subject:/cat(ch?)/" for regexp: forward slashes around the string.

"subject:cat*" for substrings: a leading or trailing asterisk.

Engines that can't handle regexp would be required to ignore them --
they would never make sense. Engines that can do substrings would
transform the asterisks appropriately; engines that can't would just
drop the asterisk and hope.

What do you think?
Eric

Re: [RFC] Gnus generalized search, part II

[Dan Christensen]

On Apr 23, 2017, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Will mairix be supported?
>
> Sure! At least, I think so -- I'm not clear why nnmairix is its own
> server, rather than being an nnir engine. I think if it's used in raw
> mode it should work just like the other engines. At any rate, as many
> engines as possible should be supported.

I don't know much about how things work in the background, but mairix
produces a maildir of results, so maybe that's why it is handled
specially.  One of the great things about it is that it can already
combine results from maildirs and mboxes, so it already works across
(some) servers.

But from a configuration perspective, I've often had trouble with
nnmairix, so it would be great if it could be fitted into a more
general, stable set-up.

Maybe mairix's raw mode that you mention would help with this?  But
then Gnus would have to work backwords from the maildir paths and
nnfolder paths + position in file to figure out the messages...

>> How do you handle differing capabilities of the search backends?

[...]

> What do you think?

Sounds reasonable, especially if it is documented and maybe displays
warnings/messages in some cases.

Dan

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Dan Christensen <jdc@uwo.ca> writes:

> On Apr 23, 2017, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> Dan Christensen <jdc@uwo.ca> writes:
>>
>>> Will mairix be supported?
>>
>> Sure! At least, I think so -- I'm not clear why nnmairix is its own
>> server, rather than being an nnir engine. I think if it's used in raw
>> mode it should work just like the other engines. At any rate, as many
>> engines as possible should be supported.
>
> I don't know much about how things work in the background, but mairix
> produces a maildir of results, so maybe that's why it is handled
> specially.  One of the great things about it is that it can already
> combine results from maildirs and mboxes, so it already works across
> (some) servers.
>
> But from a configuration perspective, I've often had trouble with
> nnmairix, so it would be great if it could be fitted into a more
> general, stable set-up.

Anything you can remember about these troubles would be helpful. And I
hope you'll be willing to test the mairix support when it's ready (I've
already added the backend, but the query transformation will take a bit
more work).

> Maybe mairix's raw mode that you mention would help with this?  But
> then Gnus would have to work backwords from the maildir paths and
> nnfolder paths + position in file to figure out the messages...

Gnus already does that for *all* of the other locally-indexed search
engines, so that's no big deal :)

>>> How do you handle differing capabilities of the search backends?
>
> [...]
>
>> What do you think?
>
> Sounds reasonable, especially if it is documented and maybe displays
> warnings/messages in some cases.

Yup, it might be nice to just drop a message saying that some search
terms were ignored, that's a good idea.

Re: [RFC] Gnus generalized search, part II

[Dan Christensen]

On Apr 23, 2017, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> But from a configuration perspective, I've often had trouble with
>> nnmairix, so it would be great if it could be fitted into a more
>> general, stable set-up.
>
> Anything you can remember about these troubles would be helpful.

I would get errors about the nnmairix group (or server?) not existing,
I believe.  Deleting and recreating the nnmairix group wouldn't help.
Restarting Gnus wouldn't help.  But I would fiddle around, enter the
server buffer and try things, Esc-g the group, and usually could get
it to work.  Very strange.  It was worse when I used a imap group to
store the search results, but still sometimes happens with an nnmaildir
group.

I also found that using this custom search wrapper helped:

(defun jdc-gnus-search nil
  (interactive)
  (pop-to-buffer "*Group*")
  (gnus-group-jump-to-group "nnmairix+mairixserver:nnmairixsearch")
  (ignore-errors (gnus-group-get-new-news-this-group))
  (call-interactively 'nnmairix-search))

> And I hope you'll be willing to test the mairix support when it's
> ready (I've already added the backend, but the query transformation
> will take a bit more work).

Sure!

Dan

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> The query entered by the user is parsed into a sexp structure, and then
>> each engine is responsible for interpreting that.
>
> I think this sounds like a good approach.  I haven't tried the code
> myself, but I skimmed it briefly and it looks good to me.  :-)

Okay, I've pushed these changes as the scratch/nnir-search branch.

This branch is mostly the same as the nnir.el file I posted last time,
but with more switches for turning query parsing on and off. In the
storied Gnus tradition of over-customization, there are now a grand
total of four ways of controlling whether queries are parsed or raw:

1. The big switch is `nnir-use-parsed-queries'. It is t by default,
   but if set to nil, Gnus will behave more or less the way it does
   now.
2. If a prefix argument is given to the nnir search command (ie, "C-u
   G G" in the *Group* buffer), that search query will not be parsed,
   and will be passed raw to all the marked servers/groups.
3. Individual search engines can be told never to parse search
   queries, by specifying the `raw-queries-p' parameter to engine
   creation. If multiple groups are marked for searching, the query
   will be parsed for groups with engines that allow it, and not for
   engines that don't.
4. Entire classes of engines can be marked never to parse queries, by
   setting variables like nnir-notmuch-raw-queries-p, with "notmuch"
   replaced by the various engine names. Again, queries to multiple
   engines will still be parsed by engines that allow it.

I do hope people will test this. Actually testing that search groups
behave correctly is of course important, but if you just want to fool
with the search language and see how it is parsed, and transformed, you
can use stuff like this:

#+BEGIN_SRC elisp
  (let* ((query-string "subject:gnus or since:1w")
	 (parsed-query (nnir-search-parse-query query-string))
	 (test-imap (make-instance 'gnus-search-imap))
	 (test-notmuch (make-instance 'gnus-search-notmuch)))
    (message "notmuch query: %s\nimap query: %s"
	     (nnir-search-transform-top-level test-imap parsed-query)
	     (nnir-search-transform-top-level test-notmuch parsed-query)))
#+END_SRC

`nnir-search-parse-query' turns strings into sexps, and
`nnir-search-transform-top-level' turns the sexps back into
engine-specific strings -- it requires an engine instance as the first
argument.

All the engines are named gnus-search-*. There are more on the way.

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Dan Christensen <jdc@uwo.ca> writes:

> On Apr 23, 2017, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> Dan Christensen <jdc@uwo.ca> writes:
>>
>>> But from a configuration perspective, I've often had trouble with
>>> nnmairix, so it would be great if it could be fitted into a more
>>> general, stable set-up.
>>
>> Anything you can remember about these troubles would be helpful.
>
> I would get errors about the nnmairix group (or server?) not existing,
> I believe.  Deleting and recreating the nnmairix group wouldn't help.
> Restarting Gnus wouldn't help.  But I would fiddle around, enter the
> server buffer and try things, Esc-g the group, and usually could get
> it to work.  Very strange.  It was worse when I used a imap group to
> store the search results, but still sometimes happens with an nnmaildir
> group.
>
> I also found that using this custom search wrapper helped:
>
> (defun jdc-gnus-search nil
>   (interactive)
>   (pop-to-buffer "*Group*")
>   (gnus-group-jump-to-group "nnmairix+mairixserver:nnmairixsearch")
>   (ignore-errors (gnus-group-get-new-news-this-group))
>   (call-interactively 'nnmairix-search))

Ugh, that gives me flashbacks to times I've had to "jiggle" groups to
get them to work. Anyway, thanks for the warning. I do expect that
making mairix a regular search engine would solve these problems.

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

Hi Eric:

I noticed you have eliminated the 'shortcut option in nnir. This was
important functionality which should probably be restored (stopping
after the first match rather than finding all matches). For example its
used in article referral (when the referral methods include 'nnir) where
we are just trying to find a particular message-id.

I originally implemented this as part of the criteria in the search
query. Since these are going away in the universal search form the
shortcut implementation will be different. Maybe just include another
search term, like "?:" and "*:" for match-one and match-any, with "*:"
the default?

And on the long list of things that would be nice: when inputting the
universal query string TAB should complete on the preset search keys.
If I want to search for an author the current nnir allows this with only
a few keystrokes: C-u G G; "author-name" RET; f TAB; RET. gnus-search
should allow something similar.

Best,
Andy

(Who is beginning to think that you and I are the only two who will ever
use this new stuff).

Re: [RFC] Gnus generalized search, part II

[Adam Sjøgren]

Andrew writes:

> (Who is beginning to think that you and I are the only two who will ever
> use this new stuff).

(I, for one, am waiting on the sidelines to hear how I can use notmuch
with all the new, improved stuff you're implementing. It sounds
exciting, but I don't have anything to contribute besides cheering you
both on :-))


  Best regards,

    Adam

-- 
 "My baby cu cu cuu                                           Adam Sjøgren
  Sweet lighting beam my"                                asjo@koldfront.dk

Re: [RFC] Gnus generalized search, part II

[Saša Janiška]

asjo@koldfront.dk (Adam Sjøgren) writes:

> (I, for one, am waiting on the sidelines to hear how I can use notmuch
> with all the new, improved stuff you're implementing. It sounds
> exciting, but I don't have anything to contribute besides cheering you
> both on :-))

Same here…using Gnus+notmuch for my imap email and using gmane for 99%
of mailing lists I follow…


Sincerely,
Gour

-- 
A person who has given up all desires for sense gratification,
who lives free from desires, who has given up all sense of
proprietorship and is devoid of false ego — he alone can
attain real peace.

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

I found a few minutes to try the generalized search but couldn't get the
file to load :( I suspect just some simple typos, but I haven't had the
time to track them down (and they all seem to be in search backends that
aren't functional anyway).

I made some a (random) guess about what changes to make and got things
to load, but I haven't tried running any functions yet:)

Eric, did I make the right changes?

--- nnir_search.el~	2017-04-22 17:03:10.509173535 +0800
+++ nnir_search.el	2017-04-26 16:13:25.806384881 +0800
@@ -1064,7 +1064,7 @@
 (eieio-oset-default 'gnus-search-swish-e 'prefix
 		    nnir-swish-e-remove-prefix)
 
-(eieio-oset-default 'gnus-search-swish-e 'config-file
+(eieio-oset-default 'gnus-search-swish-e 'index-files
 		    nnir-swish-e-index-files)
 
 (eieio-oset-default 'gnus-search-swish-e 'switches
@@ -1499,7 +1499,7 @@
 

 
-(cl-defmethod nnir-search-transform-expression ((engine gnus-engine-swish++)
+(cl-defmethod nnir-search-transform-expression ((engine gnus-search-swish++)
 						(expr (head near)))
   (format "%s near %s"
 	  (nnir-search-transform-expression engine (nth 1 expr))
@@ -1946,20 +1946,20 @@
 	(forward-line 1)))
     (apply #'vector (nreverse (delete-dups artlist)))))
 
-(cl-defmethod nnir-search-transform-expression ((_e gnus-engine-gmane)
+(cl-defmethod nnir-search-transform-expression ((_e gnus-search-gmane)
 						(_expr (head near)))
   nil)
 
 ;; Can Gmane handle OR or NOT keywords?
-(cl-defmethod nnir-search-transform-expression ((_e gnus-engine-gmane)
+(cl-defmethod nnir-search-transform-expression ((_e gnus-search-gmane)
 						(_expr (head or)))
   nil)
 
-(cl-defmethod nnir-search-transform-expression ((_e gnus-engine-gmane)
+(cl-defmethod nnir-search-transform-expression ((_e gnus-search-gmane)
 						(_expr (head not)))
   nil)
 
-(cl-defmethod nnir-search-transform-expression ((_e gnus-engine-gmane)
+(cl-defmethod nnir-search-transform-expression ((_e gnus-search-gmane)
 						(expr list))
   "The only keyword value gmane can handle is author, ie from."
   (when (memq (car expr) '(from sender author))

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

OK, those changes seemed to have worked and the example code works fine
(except that the notmuch and imap printout is reversed :))


I'll play around with more complex queries in a while.

Re: [RFC] Gnus generalized search, part II

[Joakim Jalap]

Andrew Cohen <cohen@bu.edu> writes:

> Andy
>
> (Who is beginning to think that you and I are the only two who will ever
> use this new stuff).

Just wanted to say that I will use it for sure :) It's just that gnus is
so complex I have trouble getting anything to work at all. Right now I'm
struggling with gnus-cloud. I've never really gotten search in gnus to
work for me at all, but I think this generalized search stuff might make
it much easier.

I will try to give it a spin during the weekend if I can.

-- Joakim

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

> Hi Eric:
>
> I noticed you have eliminated the 'shortcut option in nnir. This was
> important functionality which should probably be restored (stopping
> after the first match rather than finding all matches). For example its
> used in article referral (when the referral methods include 'nnir) where
> we are just trying to find a particular message-id.
>
> I originally implemented this as part of the criteria in the search
> query. Since these are going away in the universal search form the
> shortcut implementation will be different. Maybe just include another
> search term, like "?:" and "*:" for match-one and match-any, with "*:"
> the default?

Yup, I took that out, but only temporarily -- I'll put that or an
equivalent back in soon. I think I took it out before trying to
implement MULTISEARCH: servers with this capability won't need the group
loop at all, they can search many/all groups with a single command. That
should provide a nice speedup for all searches on servers that support
it (curse you, Dovecot!), but in particular for the message-id search.
Anyway, it will get put back in.

I've actually ended up restoring something like the criteria: I
mentioned in an earlier message allowing "meta" search terms like
thread, sort, limit, or count. Those will be parsed and stored outside
the query itself. A "single result" key could live there, but I'm not
sure it's necessary: a search on message-id implies single result, and
you could use "limit" in other cases.

> And on the long list of things that would be nice: when inputting the
> universal query string TAB should complete on the preset search keys.
> If I want to search for an author the current nnir allows this with only
> a few keystrokes: C-u G G; "author-name" RET; f TAB; RET. gnus-search
> should allow something similar.

Right now you can truncate search keys arbitrarily, so long as they
remain unambiguous. Ie, "subject:flights" can be given as
"subj:flights", or even "su:flights" (but "s" alone is ambiguous with
"since"). Is that what you meant? In your example above, what does the
"f" actually complete to? I would have thought that would be the search
string.

> (Who is beginning to think that you and I are the only two who will ever
> use this new stuff).

I was starting to think the same thing! Glad to see others chiming in.

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Saša Janiška <gour@atmarama.com> writes:

> asjo@koldfront.dk (Adam Sjøgren) writes:
>
>> (I, for one, am waiting on the sidelines to hear how I can use notmuch
>> with all the new, improved stuff you're implementing. It sounds
>> exciting, but I don't have anything to contribute besides cheering you
>> both on :-))
>
> Same here…using Gnus+notmuch for my imap email and using gmane for 99%
> of mailing lists I follow…

I'm curious about this: how do you use notmuch with imap? As I
understand things, the two won't be compatible: notmuch will return
results as a list of filenames, and those filenames can't be passed to
imap to retrieve the messages themselves. Are you currently using the
two together somehow? And where is your imap server?

Eric

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

> I found a few minutes to try the generalized search but couldn't get the
> file to load :( I suspect just some simple typos, but I haven't had the
> time to track them down (and they all seem to be in search backends that
> aren't functional anyway).

Bah, it looks like I pushed a slightly older version from the wrong
machine. Sorry about that. Those changes look correct, but there might
be a few more, I'll check in a second.

Actually, I decided it was silly to try to maintain two versions of the
same code, so I'm just going to rebase the whole thing on top of
features/gnus-select. I'll try to get that done today, and that will
include fixing these blunders.

Frankly I'm skeptical anyone's using the swish or hyrex backends. The
software doesn't even seem installable -- I wonder if we need to keep
them.

Eric

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

> I found a few minutes to try the generalized search but couldn't get the
> file to load :( I suspect just some simple typos, but I haven't had the
> time to track them down (and they all seem to be in search backends that
> aren't functional anyway).
>
> I made some a (random) guess about what changes to make and got things
> to load, but I haven't tried running any functions yet:)

This is weird, are you using the scratch/gnus-search branch? These
errors aren't there on the branch (no wonder I was confused). Perhaps
the file I sent last time is buggy...

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Andrew Cohen <cohen@bu.edu> writes:
>
>> I found a few minutes to try the generalized search but couldn't get the
>> file to load :( I suspect just some simple typos, but I haven't had the
>> time to track them down (and they all seem to be in search backends that
>> aren't functional anyway).
>
> Bah, it looks like I pushed a slightly older version from the wrong
> machine. Sorry about that. Those changes look correct, but there might
> be a few more, I'll check in a second.
>
> Actually, I decided it was silly to try to maintain two versions of the
> same code, so I'm just going to rebase the whole thing on top of
> features/gnus-select. I'll try to get that done today, and that will
> include fixing these blunders.

Okay, the scratch/gnus-search branch is now rebased onto
features/gnus-select. I've renamed everything from nnir to gnus-search,
and dropped all the unused code. I'll admit it was enjoyable.

I think you'll have to pull --force to make it work. It was a right mess
getting it rebased, and I hope I haven't screwed anything up.

Things:

1. I moved the -make-search-group functions out of nnselect and into
   gnus-search, since that seems like the right place for them.
2. I renamed the `nnir-mode' function to `nnselect-mode' and moved it
   into nnselect.el. That function should apply to all nnselect groups,
   not just search groups. But I don't know the right place to trigger
   it, and currently it isn't called at all. I wish groups had a
   parameter holding a group-specific bit of code to run on summary
   buffer creation.
3. I poked at nnselect here and there, to get it basically working with
   gnus-search. It shouldn't be anything too invasive.

Next:

1. I'd like to address the threading issue, and write
   `gnus-search-refer-thread'. I haven't really looked at this issue
   yet.
2. I'm curious when `nnselect-request-article' is ever called with a
   message-id string, rather than an article number. As I said, I'll
   re-instate the shortcut for message-id searching in the imap engine,
   but does this ever actually happen?
3. Figure out a proper way to run the nnselect-mode function in nnselect
   summary buffers. I tried making that a real major mode, derived from
   Gnus' summary-mode, but that made a mess of things and I backed off
   from it.

At some point, I guess when the threading issue is resolved, I'm hoping
you'll merge this scratch branch into features/gnus-select. Then
development/testing can continue in that branch (we can let it mellow
for a bit while docs are written), and then eventually that gets merged
into master.

Eric

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    Eric> Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]


    Eric> Okay, the scratch/gnus-search branch is now rebased onto
    Eric> features/gnus-select. I've renamed everything from nnir to
    Eric> gnus-search, and dropped all the unused code. I'll admit it
    Eric> was enjoyable.

    Eric> I think you'll have to pull --force to make it work. It was a
    Eric> right mess getting it rebased, and I hope I haven't screwed
    Eric> anything up.

I made significant changes to the file nnir.el as part of ripping out
all the gnusey stuff. I'll look at gnus-search and figure out how to
rebase properly. But I'd prefer that the nnselect changes, which are
more dangerous, get tested first with the current search language before
introducing the new universal search language. This should help keep
straight where the bugs are.

    Eric> Things:

    Eric> 1. I moved the -make-search-group functions out of nnselect
    Eric> and into gnus-search, since that seems like the right place
    Eric> for them.  

Actually I don't think so---I was planning on moving them out of both
nnselect and gnus-search. I was hoping to keep gnus-search to strictly
searching and not play with gnus group stuff at all, and nnselect isn't
about searching but is a more general backend.  I think this should be
in gnus-group (where it belongs given its name:))

    Eric> 2. I renamed the `nnir-mode' function to `nnselect-mode' and
    Eric> moved it into nnselect.el. That function should apply to all
    Eric> nnselect groups, not just search groups. But I don't know the
    Eric> right place to trigger it, and currently it isn't called at
    Eric> all. I wish groups had a parameter holding a group-specific
    Eric> bit of code to run on summary buffer creation.  

In fact this function has been removed entirely in my code; one of the
things the great nnselect refactoring has allowed is handling this kind
of stuff in a completely backend agnostic way (that is, now that
nnselect is a real, complete backend it can rely on more of the standard
gnus code to do things and doesn't need the special mode to handle it).

    Eric> 3. I poked at nnselect here and there, to get it basically
    Eric> working with gnus-search. It shouldn't be anything too
    Eric> invasive.

I'll take a look. The code on my end is still evolving (just clean-up
and doing things in a slightly better way but the functionality as
already all present).

    Eric> Next:

    Eric> 1. I'd like to address the threading issue, and write
    Eric> `gnus-search-refer-thread'. I haven't really looked at this
    Eric> issue yet.  

Actually I have thought alot about this and have a clear idea for how it
should be done. Basically I am hoping that everything related to the
mechanics of searching should be in gnus-search and everything related
to gnus should be outside of gnus-search. So the function
gnus-search-refer-thread (in gnus-search:)) should take as arguments an
article and a group-spec and return the select-list of articles in the
thread. The construction of the group/server list and everything else
should be done in the group functions. Currently the variable
gnus-refer-article-methods is a list of backends and the special symbol
'current. Including the 'nnir (now 'nnselect) backend allows searching
as part of article referral. I think we should change this to use a
special symbol 'search similar to what is done with 'current. This would
in fact continue to use the nnselect backend (which in turn calls the
function gnus-search-refer-thread in the request-thread function) but it
makes more sense to hide this implementation detail from the user
customization variable.

    Eric> 2. I'm curious when `nnselect-request-article' is ever called
    Eric> with a message-id string, rather than an article number. As I
    Eric> said, I'll re-instate the shortcut for message-id searching in
    Eric> the imap engine, but does this ever actually
    Eric> happen?  

This happens in two ways (and it happens all the time, at least for me):

1. When selecting a pseudo-article in the summary buffer (for any
   backend foo, not just nnselect) the nnfoo-request-article is passed
   the message-id rather than the (negative) article number.
2. When using nnselect in gnus-refer-article-methods. This list of
   methods is looped over in two places: referring an article (e.g. by
   finding a parent, or if its a pseudo-article as in case 1) or in
   referring a thread. In all cases when this hits nnselect it tries to
   request the article by id (since it doesn't know the number).

The strategy is rather simple: normally gnus-request-article passes the
article message-id and the group corresponding to the summary buffer
where the request came from. If this is an imap backend, for example,
this is enough information for nnimap to simply try to retrieve that id
from that group and no real searching is necessary. If instead search is
desired (by putting 'nnselect in the list of refer-article-methods)
nnselect-request-article instead does a SEARCH for this id in all groups
on the corresponding server. (The code restricts to imap since I never
bothered to special-case all the different search engines to find by
message-id---this is where the universal search comes in). In fact I use
a slightly better heuristic---if we are coming from an nnselect summary
buffer then we may already have articles from groups on different
servers. In this case we hunt through the thread that the originating id
comes from and see which servers all these articles come from. We then
do the search on all of these servers.


    Eric> 3. Figure out a proper way to run the nnselect-mode function
    Eric> in nnselect summary buffers. I tried making that a real major
    Eric> mode, derived from Gnus' summary-mode, but that made a mess of
    Eric> things and I backed off from it.

Now unnecessary. 


    Eric> At some point, I guess when the threading issue is resolved,

As I described above, I think the thread issue is essentially trivial
(for my part, anyway:)): all we need is a function
gnus-search-refer-thread (in gnus-search.el) that takes as arguments a
list of message-ids and a group-spec. It should then run a query on all
the groups/servers in the spec searching for the OR of all these ids in
either the message-id header or the references header, and return the
full list of articles found in the usual vectorized format. 

The construction of this list of ids and the group-spec is already
handled in the backend and doesn't require any further intervention from
gnus-search.

    Eric>  I'm hoping you'll merge this scratch branch into
    Eric> features/gnus-select. Then development/testing can continue in
    Eric> that branch (we can let it mellow for a bit while docs are
    Eric> written), and then eventually that gets merged into master.

My plan is to proceed in the following stages:

1. Have testers ensure that their usual workflow (e.g. searching with "G
   G") continues to function with the new nnselect backend. I am
   confident that this is the case, but want verification before
   proceeding.

Once this has been established we can proceed with the next two steps in
parallel:

2a: Have testers explore the more general functionality provided by
    nnselect. 

3a: Adopt the new universal search language.

So after 1. we can merge the universal stuff into the branch.

Best,
Andy

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     Eric> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> [...]
>
>
>     Eric> Okay, the scratch/gnus-search branch is now rebased onto
>     Eric> features/gnus-select. I've renamed everything from nnir to
>     Eric> gnus-search, and dropped all the unused code. I'll admit it
>     Eric> was enjoyable.
>
>     Eric> I think you'll have to pull --force to make it work. It was a
>     Eric> right mess getting it rebased, and I hope I haven't screwed
>     Eric> anything up.
>
> I made significant changes to the file nnir.el as part of ripping out
> all the gnusey stuff. I'll look at gnus-search and figure out how to
> rebase properly. But I'd prefer that the nnselect changes, which are
> more dangerous, get tested first with the current search language before
> introducing the new universal search language. This should help keep
> straight where the bugs are.

Ah, okay, I'll take a look at your version of nnir as well.

I've only rebased my branch, so the nnselect changes can be tested in
isolation just by using feature/gnus-select. It's true that someone
using scratch/gnus-search will get both my changes and yours, but at
least your changes are isolated. The new search language and all that is
actually easier to test as its relatively self-contained.

>     Eric> Things:
>
>     Eric> 1. I moved the -make-search-group functions out of nnselect
>     Eric> and into gnus-search, since that seems like the right place
>     Eric> for them.
>
> Actually I don't think so---I was planning on moving them out of both
> nnselect and gnus-search. I was hoping to keep gnus-search to strictly
> searching and not play with gnus group stuff at all, and nnselect isn't
> about searching but is a more general backend.  I think this should be
> in gnus-group (where it belongs given its name:))

Good point!

>     Eric> 2. I renamed the `nnir-mode' function to `nnselect-mode' and
>     Eric> moved it into nnselect.el. That function should apply to all
>     Eric> nnselect groups, not just search groups. But I don't know the
>     Eric> right place to trigger it, and currently it isn't called at
>     Eric> all. I wish groups had a parameter holding a group-specific
>     Eric> bit of code to run on summary buffer creation.
>
> In fact this function has been removed entirely in my code; one of the
> things the great nnselect refactoring has allowed is handling this kind
> of stuff in a completely backend agnostic way (that is, now that
> nnselect is a real, complete backend it can rely on more of the standard
> gnus code to do things and doesn't need the special mode to handle it).

Ah, okay. I don't see nnselect-summary-line-format getting applied
anywhere though -- the main thing I was interested in was seeing
original group names.

>     Eric> 3. I poked at nnselect here and there, to get it basically
>     Eric> working with gnus-search. It shouldn't be anything too
>     Eric> invasive.
>
> I'll take a look. The code on my end is still evolving (just clean-up
> and doing things in a slightly better way but the functionality as
> already all present).

That's fine. As you continue with nnselect, I'll keep my branch rebased
on top of your new work.

[...]

>     Eric> 2. I'm curious when `nnselect-request-article' is ever called
>     Eric> with a message-id string, rather than an article number. As I
>     Eric> said, I'll re-instate the shortcut for message-id searching in
>     Eric> the imap engine, but does this ever actually
>     Eric> happen?
>
> This happens in two ways (and it happens all the time, at least for me):
>
> 1. When selecting a pseudo-article in the summary buffer (for any
>    backend foo, not just nnselect) the nnfoo-request-article is passed
>    the message-id rather than the (negative) article number.
> 2. When using nnselect in gnus-refer-article-methods. This list of
>    methods is looped over in two places: referring an article (e.g. by
>    finding a parent, or if its a pseudo-article as in case 1) or in
>    referring a thread. In all cases when this hits nnselect it tries to
>    request the article by id (since it doesn't know the number).
>
> The strategy is rather simple: normally gnus-request-article passes the
> article message-id and the group corresponding to the summary buffer
> where the request came from. If this is an imap backend, for example,
> this is enough information for nnimap to simply try to retrieve that id
> from that group and no real searching is necessary. If instead search is
> desired (by putting 'nnselect in the list of refer-article-methods)
> nnselect-request-article instead does a SEARCH for this id in all groups
> on the corresponding server. (The code restricts to imap since I never
> bothered to special-case all the different search engines to find by
> message-id---this is where the universal search comes in). In fact I use
> a slightly better heuristic---if we are coming from an nnselect summary
> buffer then we may already have articles from groups on different
> servers. In this case we hunt through the thread that the originating id
> comes from and see which servers all these articles come from. We then
> do the search on all of these servers.

Okay, interesting -- thanks for the explanation.

>     Eric> At some point, I guess when the threading issue is resolved,
>
> As I described above, I think the thread issue is essentially trivial
> (for my part, anyway:)): all we need is a function
> gnus-search-refer-thread (in gnus-search.el) that takes as arguments a
> list of message-ids and a group-spec. It should then run a query on all
> the groups/servers in the spec searching for the OR of all these ids in
> either the message-id header or the references header, and return the
> full list of articles found in the usual vectorized format.
>
> The construction of this list of ids and the group-spec is already
> handled in the backend and doesn't require any further intervention from
> gnus-search.

Actually, if I'm understanding this correctly, all you'd need to do is
run a regular search with gnus-search-run-query. In the
search-query-spec, put a (thread t) key alongside the query key. So the
query spec would look like:

'((query "id:<msg-id>") (thread t))

That will be enough to tell the search engines to do whatever they have
to do to produce a whole thread. (Note this code isn't actually written
yet, but that's how it will work.)

>     Eric>  I'm hoping you'll merge this scratch branch into
>     Eric> features/gnus-select. Then development/testing can continue in
>     Eric> that branch (we can let it mellow for a bit while docs are
>     Eric> written), and then eventually that gets merged into master.
>
> My plan is to proceed in the following stages:
>
> 1. Have testers ensure that their usual workflow (e.g. searching with "G
>    G") continues to function with the new nnselect backend. I am
>    confident that this is the case, but want verification before
>    proceeding.
>
> Once this has been established we can proceed with the next two steps in
> parallel:
>
> 2a: Have testers explore the more general functionality provided by
>     nnselect.
>
> 3a: Adopt the new universal search language.
>
> So after 1. we can merge the universal stuff into the branch.

Sounds good.

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Wednesday, 26 Apr 2017 at 15:51, Eric Abrahamsen wrote:
>> Andrew Cohen <cohen@bu.edu> writes:
>
> [...]
>
>>> (Who is beginning to think that you and I are the only two who will ever
>>> use this new stuff).
>>
>> I was starting to think the same thing! Glad to see others chiming in.
>
> I know that I would likely use this and would definitely do so if the
> generalised search interface included off-line use with the
> agent...  not sure if that is planned or even possible but thought I
> would mention it!

Andy will know about any interaction between nnselect and the agent, but
the search engines are pretty discrete from the rest of Gnus, and won't
know about plugged/connected status. If your indexes are local (mairix,
notmuch, namazu, local IMAP, etc) everything will work fine. If the
indexes are remote (a remote IMAP server, for instance), then the search
would time out. Ideally, if Gnus is unplugged, it would refuse to search
remote servers at all. I don't actually know what happens right now.

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    Eric> Andrew Cohen <cohen@bu.edu> writes:
    >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
    >> 
    Eric> Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]


    Eric> Ah, okay. I don't see nnselect-summary-line-format getting
    Eric> applied anywhere though -- the main thing I was interested in
    Eric> was seeing original group names.

Hmm, working fine for me. Can you try with just a checkout of
feature/gnus-select and see if it works?

Ohh, uhm, forgot to mention:) You have to put the spec in the
gnus-summary-line-format variable instead of the old nnir specific one
(which has gone away). This spec will be happily ignored in normal
groups (%g for short group name and %G for the full name).



[...]


    Eric> Actually, if I'm understanding this correctly, all you'd need
    Eric> to do is run a regular search with gnus-search-run-query. In
    Eric> the search-query-spec, put a (thread t) key alongside the
    Eric> query key. So the query spec would look like:

    Eric> '((query "id:<msg-id>") (thread t))

    Eric> That will be enough to tell the search engines to do whatever
    Eric> they have to do to produce a whole thread. (Note this code
    Eric> isn't actually written yet, but that's how it will work.)

I don't think this is general enough. Some backends don't allow native
thread searching so we have to build the list of message-ids
ourselves. And for some backends the threads that are found aren't the
same ones that gnus thinks of as threads (you can read the ancient
discussion about why Lars chose not to use the imap thread command). So
what nnselect (formerly nnir) does is to make its choice as to the
message-ids in the thread and then search for these (the algorithm:
start with the primary article and get its message-id and all the
message-ids in its references header. Then search for all messages that
have any of these ids either in their message-id or references
header. Finally allow for similar subjects in threading as well. This
works quite well.)

If we are only passing the message-id for the primary article to
gnus-search then it will have to root around to get all the other
relevant info (references and subject headers of the primary
article). This I think breaks the paradigm of gnus-search as just
search. So my idea was to pass the full list of message-ids (and perhaps
the subject) to the gnus-search function and then let it do what it
wants: it could, depending on some configuration, use the native backend
thread search (and ignore everything passed but the primary article
message-id). Or, by default, it could do it the gnus-approved way.

Best,
Andy

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

>>>>> "Eric" == Eric S Fraga <e.fraga@ucl.ac.uk> writes:

    Eric> On Wednesday, 26 Apr 2017 at 15:51, Eric Abrahamsen wrote:
    >> Andrew Cohen <cohen@bu.edu> writes:

    Eric> [...]

    >>> (Who is beginning to think that you and I are the only two who
    >>> will ever use this new stuff).
    >> 
    >> I was starting to think the same thing! Glad to see others
    >> chiming in.

    Eric> I know that I would likely use this and would definitely do so
    Eric> if the generalised search interface included off-line use with
    Eric> the agent...  not sure if that is planned or even possible but
    Eric> thought I would mention it!


It should work, but its totally untested (I don't use the agent). The
nnselect backend (which is independent of searching) should act like any
backend and use the agent as appropriate (although since it hasn't been
tested it is probably buggy). The searching part should behave as (the
other) Eric indicated: searches in local indices should work fine, and
searches that require a network connection should signal an error.


It would be helpful if you could checkout feature/gnus-select and see
what happens :)

Best,
Andy

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

>>>>> "Eric" == Eric S Fraga <e.fraga@ucl.ac.uk> writes:


    Eric> I don't have any local index (as far as I know).  I simply
    Eric> enable the agent to ensure I can read my emails off-line (I
    Eric> spend 2-3 hours a day off-line while commuting).

Hmm, I am interested in what happens (I suspect that as long as your
nnselect groups don't involve searching everything just works; I can
give you some recipes to try).

What kind of backends are you using when you are online? imap?


    >> It would be helpful if you could checkout feature/gnus-select and
    >> see what happens :)

    Eric> Is this from the emacs repository or is there a separate gnus
    Eric> one?  In either case, where is it located please?


gnus is now developed from the central emacs git repository. You can
find this on a feature branch called

feature/gnus-select

Best,
Andy

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

[Oops, didn't realize I was duplicating your response to the other Eric.]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     Eric> Andrew Cohen <cohen@bu.edu> writes:
>     >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>     >>
>     Eric> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> [...]
>
>
>     Eric> Ah, okay. I don't see nnselect-summary-line-format getting
>     Eric> applied anywhere though -- the main thing I was interested in
>     Eric> was seeing original group names.
>
> Hmm, working fine for me. Can you try with just a checkout of
> feature/gnus-select and see if it works?
>
> Ohh, uhm, forgot to mention:) You have to put the spec in the
> gnus-summary-line-format variable instead of the old nnir specific one
> (which has gone away). This spec will be happily ignored in normal
> groups (%g for short group name and %G for the full name).

Aha! I was led astray by the presence of the
nnselect-summary-line-format variable. :)

>
> [...]
>
>
>     Eric> Actually, if I'm understanding this correctly, all you'd need
>     Eric> to do is run a regular search with gnus-search-run-query. In
>     Eric> the search-query-spec, put a (thread t) key alongside the
>     Eric> query key. So the query spec would look like:
>
>     Eric> '((query "id:<msg-id>") (thread t))
>
>     Eric> That will be enough to tell the search engines to do whatever
>     Eric> they have to do to produce a whole thread. (Note this code
>     Eric> isn't actually written yet, but that's how it will work.)
>
> I don't think this is general enough. Some backends don't allow native
> thread searching so we have to build the list of message-ids
> ourselves. And for some backends the threads that are found aren't the
> same ones that gnus thinks of as threads (you can read the ancient
> discussion about why Lars chose not to use the imap thread command). So
> what nnselect (formerly nnir) does is to make its choice as to the
> message-ids in the thread and then search for these (the algorithm:
> start with the primary article and get its message-id and all the
> message-ids in its references header. Then search for all messages that
> have any of these ids either in their message-id or references
> header. Finally allow for similar subjects in threading as well. This
> works quite well.)
>
> If we are only passing the message-id for the primary article to
> gnus-search then it will have to root around to get all the other
> relevant info (references and subject headers of the primary
> article). This I think breaks the paradigm of gnus-search as just
> search. So my idea was to pass the full list of message-ids (and perhaps
> the subject) to the gnus-search function and then let it do what it
> wants: it could, depending on some configuration, use the native backend
> thread search (and ignore everything passed but the primary article
> message-id). Or, by default, it could do it the gnus-approved way.

I guess I'd be fine with providing a customization option allowing
users to choose whether to use Gnus' concept of threading, or the search
engines'. Personally I'm much in favor of letting the search engines
handle it, for the sake of simplicity and efficiency, and I don't mind
at all if the results aren't exactly what Gnus would have returned. But
I can see the utility of a user option.

One more time slowly, as this is my first time looking into threading
and it's confusing: `gnus-search-refer-thread' may get called when the
user is in the Summary buffer and runs any of the `gnus-summary-refer-*'
commands, right?

If this only ever gets called from the summary buffer (possibly an
incorrect assumption), would it make sense to pass the value of
gnus-newsgroup-threads to `gnus-search-refer-thread' instead of bare
message ids? gnus-newsgroup-threads already has the reference headers
in it, and would get us halfway there already...

I could be misunderstanding the whole situation, of course.


BTW, I patched Gnorb to use the nnselect backend instead of its own
awful jury-rigged nngnorb backend. So far all is working great, both
with ephemeral and persistent groups. Very nice to have a simpler
solution to that!

Eric

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]

> If this only ever gets called from the summary buffer (possibly an
> incorrect assumption), would it make sense to pass the value of
> gnus-newsgroup-threads to `gnus-search-refer-thread' instead of bare
> message ids? gnus-newsgroup-threads already has the reference headers
> in it, and would get us halfway there already...

Of course I realized right afterwards that this or something like it is
exactly what you were talking about doing. I'm getting there, just slowly...

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    Eric> Eric Abrahamsen <eric@ericabrahamsen.net> writes: [...]

    >> If this only ever gets called from the summary buffer (possibly
    >> an incorrect assumption), would it make sense to pass the value
    >> of gnus-newsgroup-threads to `gnus-search-refer-thread' instead
    >> of bare message ids? gnus-newsgroup-threads already has the
    >> reference headers in it, and would get us halfway there
    >> already...

    Eric> Of course I realized right afterwards that this or something
    Eric> like it is exactly what you were talking about doing. I'm
    Eric> getting there, just slowly...


:)

The other reason for not using the search-engine based notion of
threading: my threads are often spread out over multiple servers (and
even multiple backend types, like imap and gmane). With nnselect gnus
has no problem making threads that span these things, and previously
with the old nnir and the current nnselect thread referral will 
work happily in this context (that is, gnus will create the thread with
all the articles that have been retrieved even crossing server and
backend types; then thread referral will attempt to consult ALL relevant
servers and backends to find as many thread related articles as
possible). Using the native thread searching won't be able to do this
well.

I do think the option should be available to do a native thread search,
but I strongly think the default should be the more general version that
we have now. It isn't particularly slow (and I doubt its significantly,
if at all, slower than native thread searching---it just lets gnus
construct the proper list of message ids rather than using a native
list).

I prefer passing the list of message-ids to gnus-search (rather than a
gnus construct like the dependencies hash), and then letting gnus-search
create the proper query and do the search (this is more or less how
things are now: nnselect calls nnimap-make-thread-query with the set of
message-ids to construct the query and then subsequently calls
nnir-run-query to do the search. We could keep the two steps or combine
into one function).  Aside from being simple and clean this has the
advantage of allowing the function to be called for any user-generated
(rather than gnus-generated) list of message-ids, not just a real
thread. That way you could use this function to make an nnselect group
of a bunch of related messages that weren't necessarily part of a
singlet thread. For example we have several threads related to this new
stuff going on in gmane.emacs.gnus.general and emacs-devel (and as soon
as I get around to posting my survey gmane.emacs.gnus.user :)) and this
would allow all of these things to appear as one group.

Best,
Andy

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:


[...]


    Eric> BTW, I patched Gnorb to use the nnselect backend instead of
    Eric> its own awful jury-rigged nngnorb backend. So far all is
    Eric> working great, both with ephemeral and persistent groups. Very
    Eric> nice to have a simpler solution to that!

Great! Did you try anything other than imap?

A (small) word of warning---I encountered a bug (not, in fact, a bug in
any of the new stuff, but rather something so ancient I can't even find
anything about the relevant code in the git logs:)) that can corrupt the
info-list for groups. It gets triggered only in a very rare
circumstance: if you are replying to a message from a summary buffer in
which the message resides in a group other than the summary buffer group
(this basically never happens unless you are using nnselect or nnir, in
which case it happens frequently) AND the setting up of the message
buffer and/or the sending of the message FAIL, then the
gnus-newsgroup-name gets the wrong value and this causes various info
structures to get set improperly. It won't delete any mail but it can
get the marks rather confused.

I have a fix that I will push shortly. In the meantime to avoid
complications: if you try to reply to a message and it fails for any
reason, leave the summary buffer with "Q" rather than "q" so it doesn't
try to update info.

Best,
Andy

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     Eric> Eric Abrahamsen <eric@ericabrahamsen.net> writes: [...]
>
>     >> If this only ever gets called from the summary buffer (possibly
>     >> an incorrect assumption), would it make sense to pass the value
>     >> of gnus-newsgroup-threads to `gnus-search-refer-thread' instead
>     >> of bare message ids? gnus-newsgroup-threads already has the
>     >> reference headers in it, and would get us halfway there
>     >> already...
>
>     Eric> Of course I realized right afterwards that this or something
>     Eric> like it is exactly what you were talking about doing. I'm
>     Eric> getting there, just slowly...
>
>
> :)
>
> The other reason for not using the search-engine based notion of
> threading: my threads are often spread out over multiple servers (and
> even multiple backend types, like imap and gmane). With nnselect gnus
> has no problem making threads that span these things, and previously
> with the old nnir and the current nnselect thread referral will
> work happily in this context (that is, gnus will create the thread with
> all the articles that have been retrieved even crossing server and
> backend types; then thread referral will attempt to consult ALL relevant
> servers and backends to find as many thread related articles as
> possible). Using the native thread searching won't be able to do this
> well.

I wonder if we have a misunderstanding... My original assumption was
that native thread searching would only be responsible for locating the
relevant messages, *not* for creating thread structures. Ie, engines
supply lists of messages from their respective backends that they
believe to be relevant to the original list of message-ids. All those
messages get dumped in flat list into a summary buffer, and Gnus does
its usual thread construction routines. "notmuch --thread" and "mairix
--threads" return flat lists to begin with. IMAP THREAD returns nested
lists, but I'd been assuming that I would flatten those results out
before passing them to nnselect.

Each engine pretty much does the same trick of
references/in-reply-to/maybe-subject, but I have to believe they do a
more efficient job of it in their own domains. But then we let Gnus do
the final threading before presenting the results to the user.

> I do think the option should be available to do a native thread search,
> but I strongly think the default should be the more general version that
> we have now. It isn't particularly slow (and I doubt its significantly,
> if at all, slower than native thread searching---it just lets gnus
> construct the proper list of message ids rather than using a native
> list).
>
> I prefer passing the list of message-ids to gnus-search (rather than a
> gnus construct like the dependencies hash), and then letting gnus-search
> create the proper query and do the search (this is more or less how
> things are now: nnselect calls nnimap-make-thread-query with the set of
> message-ids to construct the query and then subsequently calls
> nnir-run-query to do the search. We could keep the two steps or combine
> into one function).

I'm happy with this, I really don't have an opinion on where the
reference deconstruction should happen.

> Aside from being simple and clean this has the advantage of allowing
> the function to be called for any user-generated (rather than
> gnus-generated) list of message-ids, not just a real thread. That way
> you could use this function to make an nnselect group of a bunch of
> related messages that weren't necessarily part of a singlet thread.
> For example we have several threads related to this new stuff going on
> in gmane.emacs.gnus.general and emacs-devel (and as soon as I get
> around to posting my survey gmane.emacs.gnus.user :)) and this would
> allow all of these things to appear as one group.

If my assumptions above are correct, then this is how it would work,
even with native thread searching.

Does that make sense?

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>
> [...]
>
>
>     Eric> BTW, I patched Gnorb to use the nnselect backend instead of
>     Eric> its own awful jury-rigged nngnorb backend. So far all is
>     Eric> working great, both with ephemeral and persistent groups. Very
>     Eric> nice to have a simpler solution to that!
>
> Great! Did you try anything other than imap?

Only imap so far. Gnorb uses the registry to get group+msg-id pairs,
then calls gnus-request-head to get the article number, then makes the
vectors out of that. Theoretically it shouldn't matter what the backend
is, but...

At some point I'll bit the bullet and set up a proper Gnus testing
environment.

> A (small) word of warning---I encountered a bug (not, in fact, a bug in
> any of the new stuff, but rather something so ancient I can't even find
> anything about the relevant code in the git logs:)) that can corrupt the
> info-list for groups. It gets triggered only in a very rare
> circumstance: if you are replying to a message from a summary buffer in
> which the message resides in a group other than the summary buffer group
> (this basically never happens unless you are using nnselect or nnir, in
> which case it happens frequently) AND the setting up of the message
> buffer and/or the sending of the message FAIL, then the
> gnus-newsgroup-name gets the wrong value and this causes various info
> structures to get set improperly. It won't delete any mail but it can
> get the marks rather confused.
>
> I have a fix that I will push shortly. In the meantime to avoid
> complications: if you try to reply to a message and it fails for any
> reason, leave the summary buffer with "Q" rather than "q" so it doesn't
> try to update info.

Ugh, that sounds very much like Gnus. I've never hit that one in
particular, but I'm glad you're fixing it. There's still something
terribly fragile about marks; I wonder if more uses of gnus-atomic-* or
at least just unwind-protect wouldn't be helpful.

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    Eric> Eric Abrahamsen <eric@ericabrahamsen.net> writes: [...]

[...]


    Eric> I wonder if we have a misunderstanding... My original
    Eric> assumption was that native thread searching would only be
    Eric> responsible for locating the relevant messages, *not* for
    Eric> creating thread structures. Ie, engines supply lists of
    Eric> messages from their respective backends that they believe to
    Eric> be relevant to the original list of message-ids. All those
    Eric> messages get dumped in flat list into a summary buffer, and
    Eric> Gnus does its usual thread construction routines. "notmuch
    Eric> --thread" and "mairix --threads" return flat lists to begin
    Eric> with. IMAP THREAD returns nested lists, but I'd been assuming
    Eric> that I would flatten those results out before passing them to
    Eric> nnselect.

Yes, we are on the same page.

    Eric> Each engine pretty much does the same trick of
    Eric> references/in-reply-to/maybe-subject, but I have to believe
    Eric> they do a more efficient job of it in their own domains. But
    Eric> then we let Gnus do the final threading before presenting the
    Eric> results to the user.


But the issue is that with email threads that span multiple servers any
given server might not know enough about the thread to return all the
relevant ids.

Consider the current system: start with a message; find all the messages
in the current summary buffer that gnus thinks are part of a thread;
combine all of the servers that these messages come from. Now take the
initial message with its message-id and its references. Do an OR search
on all of the servers (at some time the algorithm collected all the
message-ids in all the references from all the articles in the current
summary buffer thread; I think I stopped doing that because of a bug in
gnus sparse article handling that I finally tracked down last week(!)
but I intend to put it back).

But now imagine the native search that you propose: you start with a
single message-id that you pass to gnus-search. Which servers does it
look at? To get everything you need to pass it the list of servers to
check. But the initial message-id is only on one of those servers, so
for the others you don't even have the full list of references to
search. You could consult the servers, see which one has the original
message, extract its references, etc. Or you could ask for the list of
references and the list of servers up front. This is what I am
suggesting.  Gnus has all the relevant info to begin with, so lets use
it. 


    >> I do think the option should be available to do a native thread
    >> search, but I strongly think the default should be the more
    >> general version that we have now. It isn't particularly slow (and
    >> I doubt its significantly, if at all, slower than native thread
    >> searching---it just lets gnus construct the proper list of
    >> message ids rather than using a native list).
    >> 
    >> I prefer passing the list of message-ids to gnus-search (rather
    >> than a gnus construct like the dependencies hash), and then
    >> letting gnus-search create the proper query and do the search
    >> (this is more or less how things are now: nnselect calls
    >> nnimap-make-thread-query with the set of message-ids to construct
    >> the query and then subsequently calls nnir-run-query to do the
    >> search. We could keep the two steps or combine into one
    >> function).

    Eric> I'm happy with this, I really don't have an opinion on where
    Eric> the reference deconstruction should happen.

    >> Aside from being simple and clean this has the advantage of
    >> allowing the function to be called for any user-generated (rather
    >> than gnus-generated) list of message-ids, not just a real
    >> thread. That way you could use this function to make an nnselect
    >> group of a bunch of related messages that weren't necessarily
    >> part of a singlet thread.  For example we have several threads
    >> related to this new stuff going on in gmane.emacs.gnus.general
    >> and emacs-devel (and as soon as I get around to posting my survey
    >> gmane.emacs.gnus.user :)) and this would allow all of these
    >> things to appear as one group.

    Eric> If my assumptions above are correct, then this is how it would
    Eric> work, even with native thread searching.

This case is different---I am thinking of a non-threaded
situation. Imagine that I have a bunch of messages about a conference
that I am organizing that are in different threads (and potentially on
different servers). I might want to view all the messages related to
these as a single (nnselect) group. Since they don't have even a common
identifying subject, searching for them directly can be a problem. But
imagine that I have a small set of them; I could then search for all the
threads that contain all of them and do a reasonable job of constructing
my conference group. (In a nutshell I am just saying a search for the OR
of a bunch of message-ids in the References headers on a server is a
pretty effective way to organize a virtual group).

Best,
Andy

Re: [RFC] Gnus generalized search, part II

[Harry Putnam]

Andrew Cohen <cohen@bu.edu> writes:

>> Ohh, uhm, forgot to mention:) You have to put the spec in the
>> gnus-summary-line-format variable instead of the old nnir specific one
>> (which has gone away). This spec will be happily ignored in normal
>> groups (%g for short group name and %G for the full name).

Eric A

> Aha! I was led astray by the presence of the
> nnselect-summary-line-format variable. :)

Sorry to bog things down to something probably trivial.  But could one
of you spell out a bit more what this `spec'  is?

My summary line format is a long string of char that I pieced together
about a hundred yrs ago, haven't thought much about since, and are now
mainly mystifying.

Is it upper or lower case `g' as would seem to be what Andrew C
suggests or something more?

(setq gnus-summary-line-format "%O%2t%U%R%7d%z%I%(%[%4L: %-20,20f%]%) %s\n")

Re: [RFC] Gnus generalized search, part II

[Andrew Cohen]

>>>>> "Harry" == Harry Putnam <reader@newsguy.com> writes:

    Harry> Sorry to bog things down to something probably trivial.  But
    Harry> could one of you spell out a bit more what this `spec' is?

    Harry> My summary line format is a long string of char that I pieced
    Harry> together about a hundred yrs ago, haven't thought much about
    Harry> since, and are now mainly mystifying.

    Harry> Is it upper or lower case `g' as would seem to be what Andrew
    Harry> C suggests or something more?

    Harry> (setq gnus-summary-line-format "%O%2t%U%R%7d%z%I%(%[%4L:
    Harry> %-20,20f%]%) %s\n")


The summary-line format is documented in the gnus manual in the section
Gnus->Summary Buffer->Summary Buffer Format->Summary Buffer Lines

There you can find the syntax of this thing and what all the different
possibilities are (except for the new nnselect part of the spec which
hasn't yet entered this part of the documentation:))

The only nnselect specific items are the specs for showing the "real"
information about an article in an nnselect buffer. They are

          %Z    Search retrieval score value (integer)
          %G    Article original full group name (string)
          %g    Article original short group name (string)

If you are using the new nnselect stuff from git, then you can just use
these in your spec and they will be blank in ordinary summary buffers,
and show the right thing in nnselect buffers.

So indeed %g is the right spec to get the originating group name (or %G
if you want the longer name).

And although I doubt it will be that helpful, here is the crazy one I
use (I have a strangely formatted summary buffer :))

(setq gnus-summary-line-format (concat 
				"%15{%U%R%}" 
				"%15{│ %}" 
				"%12{%-16&user-date;%}"
				"%15{ │%}"
				"%12{ %}%(%-18,18f%)"
				"%12{ %}%*%7{%B%}%«%14{%~(ignore 0)Z %}%11{%s%} %14{%g%}%»\n"))

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     Eric> Eric Abrahamsen <eric@ericabrahamsen.net> writes: [...]
>
> [...]
>
>
>     Eric> I wonder if we have a misunderstanding... My original
>     Eric> assumption was that native thread searching would only be
>     Eric> responsible for locating the relevant messages, *not* for
>     Eric> creating thread structures. Ie, engines supply lists of
>     Eric> messages from their respective backends that they believe to
>     Eric> be relevant to the original list of message-ids. All those
>     Eric> messages get dumped in flat list into a summary buffer, and
>     Eric> Gnus does its usual thread construction routines. "notmuch
>     Eric> --thread" and "mairix --threads" return flat lists to begin
>     Eric> with. IMAP THREAD returns nested lists, but I'd been assuming
>     Eric> that I would flatten those results out before passing them to
>     Eric> nnselect.
>
> Yes, we are on the same page.
>
>     Eric> Each engine pretty much does the same trick of
>     Eric> references/in-reply-to/maybe-subject, but I have to believe
>     Eric> they do a more efficient job of it in their own domains. But
>     Eric> then we let Gnus do the final threading before presenting the
>     Eric> results to the user.
>
>
> But the issue is that with email threads that span multiple servers any
> given server might not know enough about the thread to return all the
> relevant ids.
>
> Consider the current system: start with a message; find all the messages
> in the current summary buffer that gnus thinks are part of a thread;
> combine all of the servers that these messages come from. Now take the
> initial message with its message-id and its references. Do an OR search
> on all of the servers (at some time the algorithm collected all the
> message-ids in all the references from all the articles in the current
> summary buffer thread; I think I stopped doing that because of a bug in
> gnus sparse article handling that I finally tracked down last week(!)
> but I intend to put it back).

Actually, sorry, I had already come around to the idea that we need to
start out with a list of messages ids, not just a single one. At this
point I was only saying that I'd still prefer to be using the actual
"thread" commands on the various engines. I probably didn't make that
clear.

Rather than continuing to talk around this, maybe we should start
looking at some code. Would you be willing to make a commit on the
scratch/gnus-search branch, adding calls to the (still hypothetical)
`gnus-search-refer-thread' function where you'd like to see them, and
passing in the appropriate arguments? I can take a look at that and
start working on the function itself; I'll have a better sense of how it
needs to work, and can test it. What do you think?

Re: [RFC] Gnus generalized search, part II

[Eric Abrahamsen]

Okay, I've pushed a new version of this to scratch/gnus-search. It was a
rebase, so you'll have to force pull -- from here on out I'll use merges
instead, so that won't be necessary.

Things:

1. Hyrex is deleted.
2. Mairix is added.
3. Code for the local-index engines is largely refactored.
4. Second-pass grep searches are supported for local-index engines.
5. The loop-break behavior for single-message-id search in IMAP is
   reinstated.
6. I've decided to drop MULTISEARCH, RELEVANCY and other RETURN values
   in IMAP for now, because handling that would require reworking how
   IMAP server responses are parsed, and that's a job for another day.
7. I've added a bit of backward-compatibility protection, and obsoletion
   information, so the transition is less jarring. I'm leaning towards
   defaulting `gnus-search-use-parsed-queries' to nil, as well, to
   reduce early-adopter bafflement.
8. Thread searching is back in, mostly -- see below.

Next things:

1. Swish-e and swish++ have not been tested, and are looking very ripe
   for deletion.
2. Namazu creates an index on the Date field, but I don't understand how
   to search it. Is it just a string match?
3. Tests need to be expanded.
4. Documentation needs to be finished.
5. There are still problems with the Mairix query transformations.

Thread searching still has to be gone over and refactored. There's a bit
of hackiness in IMAP: `nnimap-make-thread-query' is still in there, but
only used in `nnimap-request-thread'. `nnselect-search-thread' and
`gnus-search-thread' now look nearly identical.

I didn't go farther in part because I'm not confident in how this is
*supposed* to look. My present interpretation is that there are two ways to do
search referral: either search in the present group and insert the
results in among the existing messages (not creating an nnselect group),
or else create an nnselect group and search *all* groups on the server
for thread-related messages. The `gnus-refer-thread-use-search' variable
decides which it is. But if the user's in an nnselect search group to
begin with, and refers the thread, then... do what?

Anyway, that's where it's at. Input on thread search behavior is
welcome, otherwise I'll do some more code reading and think about it.

Eric