Gnus development mailing list
 help / color / mirror / Atom feed
* New "gnus-search" syntax and interface
@ 2020-11-04 17:15 Eric Abrahamsen
  2020-11-04 18:14 ` Pankaj Jangid
                   ` (5 more replies)
  0 siblings, 6 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-04 17:15 UTC (permalink / raw)
  To: ding

[Resending via ding as the newsgroup seems to be unresponsive]

Hi all,

For those of you on Emacs master, I just pushed the gnus-search.el
library, which I've been sitting on for a while. It obsoletes nnir.el
(while borrowing some of its code), and uses nnselect to display search
results.

Under default settings, it should do nothing at all, and searches should
behave exactly as before. The nnir.el library has been moved to
lisp/obsolete, so code that requires it should not break, but it is no
longer needed.

If anything breaks for anyone out of the box (though it shouldn't),
please let me know right away!

Past that, the fun stuff comes if you set
`gnus-search-use-parsed-queries' to t. That will enable the "generalized
query syntax", a single search syntax that can be used across any
supported search engine. That means you can mark an imap group and a
maildir group indexed by notmuch (for instance), and issue a single
query against both. The query will be parsed and transformed into
strings that the engines accept, and you'll get the expected results.

It also opens the door to some nice conveniences. You can search by Gnus
mark, for instance. Or issue relative date queries like "since:4d" for
messages in the last four days (no need to remember the IMAP date
format!). Or "limit:20" for just some results. If you use EBDB, I've
already pushed a change so you can auto-complete contact names or email
addresses on keys like "from" or "to" or "cc". I'll propose a similar
addition to BBDB soon.

The syntax is fully documented in the new "Search Queries" section of
the Gnus manual, but basically it looks like this:

(from:bob or from:jane) -to:sam since:2w subject:"sam's surprise party"

Terms can be negated with a minus sign, as above, or with a preceding
"not". Terms can be auto-completed with TAB, or you can leave them
abbreviated so long as they're not ambiguous. So the query above could
be:

(f:bob or f:jane) -t:sam sin:2w su:"sam's surprise party"

You can shut off the parsing and use the native syntax of whatever group
you're searching on a per-engine, per-server, or per-search basis. See
the manual for details.

I hope this is a welcome addition!

Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 17:15 New "gnus-search" syntax and interface Eric Abrahamsen
@ 2020-11-04 18:14 ` Pankaj Jangid
  2020-11-04 18:45   ` Eric Abrahamsen
  2020-11-05  2:19 ` Eric Abrahamsen
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 49+ messages in thread
From: Pankaj Jangid @ 2020-11-04 18:14 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Under default settings, it should do nothing at all, and searches
> should behave exactly as before. The nnir.el library has been moved to
> lisp/obsolete, so code that requires it should not break, but it is no
> longer needed.
>
> If anything breaks for anyone out of the box (though it shouldn't),
> please let me know right away!
>
> Past that, the fun stuff comes if you set
> `gnus-search-use-parsed-queries' to t. That will enable the
> "generalized query syntax", a single search syntax that can be used
> across any supported search engine. That means you can mark an imap
> group and a maildir group indexed by notmuch (for instance), and issue
> a single query against both. The query will be parsed and transformed
> into strings that the engines accept, and you'll get the expected
> results.
>
> It also opens the door to some nice conveniences. You can search by
> Gnus mark, for instance. Or issue relative date queries like
> "since:4d" for messages in the last four days (no need to remember the
> IMAP date format!). Or "limit:20" for just some results. If you use
> EBDB, I've already pushed a change so you can auto-complete contact
> names or email addresses on keys like "from" or "to" or "cc". I'll
> propose a similar addition to BBDB soon.
>
> The syntax is fully documented in the new "Search Queries" section of
> the Gnus manual, but basically it looks like this:
>
> (from:bob or from:jane) -to:sam since:2w subject:"sam's surprise
> party"
>
> Terms can be negated with a minus sign, as above, or with a preceding
> "not". Terms can be auto-completed with TAB, or you can leave them
> abbreviated so long as they're not ambiguous. So the query above could
> be:
>
> (f:bob or f:jane) -t:sam sin:2w su:"sam's surprise party"


This is real convenience when you have lots of email
servers/groups. Thanks for this stuff. I have already played with it for
an hour or so.

> You can shut off the parsing and use the native syntax of whatever
> group you're searching on a per-engine, per-server, or per-search
> basis. See the manual for details.

Who would do that now.


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 18:14 ` Pankaj Jangid
@ 2020-11-04 18:45   ` Eric Abrahamsen
  2020-11-04 19:32     ` Pankaj Jangid
  2020-11-04 19:39     ` Pankaj Jangid
  0 siblings, 2 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-04 18:45 UTC (permalink / raw)
  To: ding

Pankaj Jangid <pankaj@codeisgreat.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>

[...]

>> (f:bob or f:jane) -t:sam sin:2w su:"sam's surprise party"
>
>
> This is real convenience when you have lots of email
> servers/groups. Thanks for this stuff. I have already played with it for
> an hour or so.

Glad it's useful! It's particularly nice with nnselect's persistent
search groups, because the queries are re-parsed every time the group is
updated, meaning that a relative date query like "since:5d" will always
indicate the most recent five days.

>> You can shut off the parsing and use the native syntax of whatever
>> group you're searching on a per-engine, per-server, or per-search
>> basis. See the manual for details.
>
> Who would do that now.

Well, if you've only got one server and it has a reasonable search
engine that you're already used to using, there might be no need for
this. Or you might find bugs and need to disable it! But otherwise, I
quite agree :)


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 18:45   ` Eric Abrahamsen
@ 2020-11-04 19:32     ` Pankaj Jangid
  2020-11-04 19:39     ` Pankaj Jangid
  1 sibling, 0 replies; 49+ messages in thread
From: Pankaj Jangid @ 2020-11-04 19:32 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>>> (f:bob or f:jane) -t:sam sin:2w su:"sam's surprise party"
>>
>>
>> This is real convenience when you have lots of email
>> servers/groups. Thanks for this stuff. I have already played with it for
>> an hour or so.
>
[...]

>>> You can shut off the parsing and use the native syntax of whatever
>>> group you're searching on a per-engine, per-server, or per-search
>>> basis. See the manual for details.
>>
>> Who would do that now.
>
> Well, if you've only got one server and it has a reasonable search
> engine that you're already used to using, there might be no need for
> this. Or you might find bugs and need to disable it! But otherwise, I
> quite agree :)

Yes. These are fair use-cases. Agree.


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 18:45   ` Eric Abrahamsen
  2020-11-04 19:32     ` Pankaj Jangid
@ 2020-11-04 19:39     ` Pankaj Jangid
  2020-11-04 19:49       ` Eric Abrahamsen
  1 sibling, 1 reply; 49+ messages in thread
From: Pankaj Jangid @ 2020-11-04 19:39 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>>> (f:bob or f:jane) -t:sam sin:2w su:"sam's surprise party"
>>
>>
>> This is real convenience when you have lots of email
>> servers/groups. Thanks for this stuff. I have already played with it
>> for an hour or so.
>
> Glad it's useful! It's particularly nice with nnselect's persistent
> search groups, because the queries are re-parsed every time the group
> is updated, meaning that a relative date query like "since:5d" will
> always indicate the most recent five days.
>

Need to read about this a bit. Are all the commands that are available
in normal summary buffer, available in nnselect persistent search
groups?

If yes then this could be one use case -> Instead of spliting, we can
use nnselect and save on Gnus startup time. I don't know. I have to read
more.


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 19:39     ` Pankaj Jangid
@ 2020-11-04 19:49       ` Eric Abrahamsen
  0 siblings, 0 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-04 19:49 UTC (permalink / raw)
  To: Pankaj Jangid; +Cc: ding



On November 4, 2020 11:39:21 AM PST, Pankaj Jangid <pankaj@codeisgreat.org> wrote:
>Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>>> (f:bob or f:jane) -t:sam sin:2w su:"sam's surprise party"
>>>
>>>
>>> This is real convenience when you have lots of email
>>> servers/groups. Thanks for this stuff. I have already played with it
>>> for an hour or so.
>>
>> Glad it's useful! It's particularly nice with nnselect's persistent
>> search groups, because the queries are re-parsed every time the group
>> is updated, meaning that a relative date query like "since:5d" will
>> always indicate the most recent five days.
>>
>
>Need to read about this a bit. Are all the commands that are available
>in normal summary buffer, available in nnselect persistent search
>groups?

That's right. In the Group buffer, "G G" makes an ephemeral search group, "G g" makes a permanent one. If you make an ephemeral group, enter it, and decide you want to make it permanent after the fact, you can do so with "C-c C-p".

>If yes then this could be one use case -> Instead of spliting, we can
>use nnselect and save on Gnus startup time. I don't know. I have to
>read
>more.

Andy Cohen, who wrote nnselect, says that it has totally changed how he uses Gnus, for exactly this reason. Permanent search groups are more flexible than splitting.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 17:15 New "gnus-search" syntax and interface Eric Abrahamsen
  2020-11-04 18:14 ` Pankaj Jangid
@ 2020-11-05  2:19 ` Eric Abrahamsen
  2020-11-05 11:58 ` Adam Sjøgren
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-05  2:19 UTC (permalink / raw)
  To: ding

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> [Resending via ding as the newsgroup seems to be unresponsive]
>
> Hi all,
>
> For those of you on Emacs master, I just pushed the gnus-search.el
> library, which I've been sitting on for a while. It obsoletes nnir.el
> (while borrowing some of its code), and uses nnselect to display search
> results.
>
> Under default settings, it should do nothing at all, and searches should
> behave exactly as before. The nnir.el library has been moved to
> lisp/obsolete, so code that requires it should not break, but it is no
> longer needed.
>
> If anything breaks for anyone out of the box (though it shouldn't),
> please let me know right away!

Of course, a day after I push this (and after a couple of years of
poking at this library), I realized that there is a
backwards-incompatible change in behavior, even if you don't turn query
parsing on.

nnir had `nnir-imap-default-search-key', which let you set a default
search key (like "body" or "subject"). If you wanted to use a different
key you had to give a prefix argument to the search command. Or you
could set that option to "imap" and explicitly enter the search keys.

The prefix argument now toggles "raw" searches vs parsed, and raw
searches of imap groups always behave as with the "imap" value above.
Ie, you can no longer specify a single default search key.

I've added an obsoletion note to this effect, but it could still be
surprising for people who were using that feature. If it bothers anyone,
please tell me and I'll think of a way to get the behavior back in
there.

Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 17:15 New "gnus-search" syntax and interface Eric Abrahamsen
  2020-11-04 18:14 ` Pankaj Jangid
  2020-11-05  2:19 ` Eric Abrahamsen
@ 2020-11-05 11:58 ` Adam Sjøgren
  2020-11-05 16:04   ` Eric Abrahamsen
  2020-11-06  6:23 ` Jose A. Ortega Ruiz
                   ` (2 subsequent siblings)
  5 siblings, 1 reply; 49+ messages in thread
From: Adam Sjøgren @ 2020-11-05 11:58 UTC (permalink / raw)
  To: ding

Eric writes:

> Past that, the fun stuff comes if you set
> `gnus-search-use-parsed-queries' to t. That will enable the "generalized
> query syntax", a single search syntax that can be used across any
> supported search engine.

I did that, placed point on nnml:normal and pressed G G as I usually do.
Entered at string to search for at the Query: prompt, and instead of the
expected results returned from notmuch, I got:

  No search engine defined for nnml:
  cl-no-applicable-method: No applicable method: gnus-search-run-search,
  nil, "nnml:", ((parsed-query "Leaf") (query . "Leaf") (raw)),
  ("nnml:normal")

My mail method is defined like this:

  (setq asjo-local-mail-method '(nnml ""
                                      (nnir-search-engine notmuch)))

    [...]

  (setq gnus-secondary-select-methods
        `(
          ,asjo-local-mail-method
      [...]

(My archive method is also nnml - nnml:archive, and also has
(nnir-search-engine notmuch).)

I get the same error if I switch gnus-search-use-parsed-queries to nil.

Do I need to tell Gnus that I want to use notmuch in a new way?


  Best regards,

    Adam

-- 
 "This is part of what makes them good hackers: when        Adam Sjøgren
  something's broken, they need to fix it."            asjo@koldfront.dk


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-05 11:58 ` Adam Sjøgren
@ 2020-11-05 16:04   ` Eric Abrahamsen
  2020-11-05 16:14     ` Eric Abrahamsen
  2020-11-05 16:41     ` Adam Sjøgren
  0 siblings, 2 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-05 16:04 UTC (permalink / raw)
  To: Adam Sjøgren; +Cc: ding

Adam Sjøgren <asjo@koldfront.dk> writes:

> Eric writes:
>
>> Past that, the fun stuff comes if you set
>> `gnus-search-use-parsed-queries' to t. That will enable the "generalized
>> query syntax", a single search syntax that can be used across any
>> supported search engine.
>
> I did that, placed point on nnml:normal and pressed G G as I usually do.
> Entered at string to search for at the Query: prompt, and instead of the
> expected results returned from notmuch, I got:
>
>   No search engine defined for nnml:
>   cl-no-applicable-method: No applicable method: gnus-search-run-search,
>   nil, "nnml:", ((parsed-query "Leaf") (query . "Leaf") (raw)),
>   ("nnml:normal")
>
> My mail method is defined like this:
>
>   (setq asjo-local-mail-method '(nnml ""
>                                       (nnir-search-engine notmuch)))
>
>     [...]
>
>   (setq gnus-secondary-select-methods
>         `(
>           ,asjo-local-mail-method
>       [...]
>
> (My archive method is also nnml - nnml:archive, and also has
> (nnir-search-engine notmuch).)
>
> I get the same error if I switch gnus-search-use-parsed-queries to nil.
>
> Do I need to tell Gnus that I want to use notmuch in a new way?

Apparently I borked the backward compatibility. The new way to define
the search engine (in your case) would be

(gnus-search-engine gnus-search-notmuch)

but old `nnir-search-engine' way should definitely be supported. Would
you please eval the below and confirm that it works correctly?

Thanks!

(defun gnus-search-server-to-engine (srv)
  (let* ((method (gnus-server-to-method srv))
	 (server
	  (or (assoc 'gnus-search-engine (cddr method))
	      (assoc (car method) gnus-search-default-engines)
	      (when-let ((old (assoc 'nnir-search-engine
				     (cddr method))))
		(nnheader-message
		 8 "\"nnir-search-engine\" is no longer a valid parameter")
		(pcase (nth 1 old)
		  ('notmuch 'gnus-search-notmuch)
		  ('namazu 'gnus-search-namazu)
		  ('find-grep 'gnus-search-find-grep)))))
	 (inst
	  (cond
	   ((null server) nil)
	   ((eieio-object-p (cadr server))
	    (cadr server))
	   ((class-p (cadr server))
	    (make-instance (cadr server)))
	   (t nil))))
    (if inst
	(when (cddr server)
	  (pcase-dolist (`(,key ,value) (cddr server))
	    (condition-case nil
		(setf (slot-value inst key) value)
	      ((invalid-slot-name invalid-slot-type)
	       (nnheader-message
		5 "Invalid search engine parameter: (%s %s)"
		key value)))))
      (nnheader-message 5 "No search engine defined for %s" srv))
    inst))


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-05 16:04   ` Eric Abrahamsen
@ 2020-11-05 16:14     ` Eric Abrahamsen
  2020-11-05 16:41     ` Adam Sjøgren
  1 sibling, 0 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-05 16:14 UTC (permalink / raw)
  To: Adam Sjøgren; +Cc: ding

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Adam Sjøgren <asjo@koldfront.dk> writes:
>
>> Eric writes:
>>
>>> Past that, the fun stuff comes if you set
>>> `gnus-search-use-parsed-queries' to t. That will enable the "generalized
>>> query syntax", a single search syntax that can be used across any
>>> supported search engine.
>>
>> I did that, placed point on nnml:normal and pressed G G as I usually do.
>> Entered at string to search for at the Query: prompt, and instead of the
>> expected results returned from notmuch, I got:
>>
>>   No search engine defined for nnml:
>>   cl-no-applicable-method: No applicable method: gnus-search-run-search,
>>   nil, "nnml:", ((parsed-query "Leaf") (query . "Leaf") (raw)),
>>   ("nnml:normal")
>>
>> My mail method is defined like this:
>>
>>   (setq asjo-local-mail-method '(nnml ""
>>                                       (nnir-search-engine notmuch)))
>>
>>     [...]
>>
>>   (setq gnus-secondary-select-methods
>>         `(
>>           ,asjo-local-mail-method
>>       [...]
>>
>> (My archive method is also nnml - nnml:archive, and also has
>> (nnir-search-engine notmuch).)
>>
>> I get the same error if I switch gnus-search-use-parsed-queries to nil.
>>
>> Do I need to tell Gnus that I want to use notmuch in a new way?
>
> Apparently I borked the backward compatibility. The new way to define
> the search engine (in your case) would be
>
> (gnus-search-engine gnus-search-notmuch)
>
> but old `nnir-search-engine' way should definitely be supported. Would
> you please eval the below and confirm that it works correctly?
>
> Thanks!
>
> (defun gnus-search-server-to-engine (srv)
>   (let* ((method (gnus-server-to-method srv))
> 	 (server
> 	  (or (assoc 'gnus-search-engine (cddr method))
> 	      (assoc (car method) gnus-search-default-engines)
> 	      (when-let ((old (assoc 'nnir-search-engine
> 				     (cddr method))))
> 		(nnheader-message
> 		 8 "\"nnir-search-engine\" is no longer a valid parameter")
> 		(pcase (nth 1 old)
> 		  ('notmuch 'gnus-search-notmuch)
> 		  ('namazu 'gnus-search-namazu)
> 		  ('find-grep 'gnus-search-find-grep)))))
> 	 (inst
> 	  (cond
> 	   ((null server) nil)
> 	   ((eieio-object-p (cadr server))
> 	    (cadr server))
> 	   ((class-p (cadr server))
> 	    (make-instance (cadr server)))
> 	   (t nil))))
>     (if inst
> 	(when (cddr server)
> 	  (pcase-dolist (`(,key ,value) (cddr server))
> 	    (condition-case nil
> 		(setf (slot-value inst key) value)
> 	      ((invalid-slot-name invalid-slot-type)
> 	       (nnheader-message
> 		5 "Invalid search engine parameter: (%s %s)"
> 		key value)))))
>       (nnheader-message 5 "No search engine defined for %s" srv))
>     inst))

This is wrong too, let me just take a deep breath and do this right...


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-05 16:04   ` Eric Abrahamsen
  2020-11-05 16:14     ` Eric Abrahamsen
@ 2020-11-05 16:41     ` Adam Sjøgren
  2020-11-05 17:18       ` Eric Abrahamsen
  1 sibling, 1 reply; 49+ messages in thread
From: Adam Sjøgren @ 2020-11-05 16:41 UTC (permalink / raw)
  To: ding

Eric writes:

> Apparently I borked the backward compatibility. The new way to define
> the search engine (in your case) would be
>
> (gnus-search-engine gnus-search-notmuch)

This works for me!


Eric also writes:

> This is wrong too, let me just take a deep breath and do this right...

Or document that an update is needed and leave it at that?


  Best regards,

    Adam

-- 
 "Didn't matter to me what I said. Still doesn't,           Adam Sjøgren
  really."                                             asjo@koldfront.dk



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-05 16:41     ` Adam Sjøgren
@ 2020-11-05 17:18       ` Eric Abrahamsen
  2020-11-05 17:34         ` Adam Sjøgren
  0 siblings, 1 reply; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-05 17:18 UTC (permalink / raw)
  To: Adam Sjøgren; +Cc: ding

Adam Sjøgren <asjo@koldfront.dk> writes:

> Eric writes:
>
>> Apparently I borked the backward compatibility. The new way to define
>> the search engine (in your case) would be
>>
>> (gnus-search-engine gnus-search-notmuch)
>
> This works for me!

Good!


> Eric also writes:
>
>> This is wrong too, let me just take a deep breath and do this right...
>
> Or document that an update is needed and leave it at that?

That would be a little mean, particularly since it's so easy to provide
for backwards compatibility (I mean, it *will* be easy once I've got my
cddr's pointed in the right direction).

More worryingly, I see I've changed the option type of
gnus-search-default-engines -- nnir-method-default-engines was a proper
alist, gnus-search-default-engines is a list of lists. I've always found
it a bit confusing in Gnus, how some places require cons cells and
others two-element lists, but there's no reason not to main consistency
with the past, and I'll change that, too.

Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-05 17:18       ` Eric Abrahamsen
@ 2020-11-05 17:34         ` Adam Sjøgren
  2020-11-06  3:50           ` Eric Abrahamsen
  0 siblings, 1 reply; 49+ messages in thread
From: Adam Sjøgren @ 2020-11-05 17:34 UTC (permalink / raw)
  To: ding

Eric writes:

>> Or document that an update is needed and leave it at that?

> That would be a little mean,

Maybe, but sometimes it makes for less clutter - especially if nnir no
longer is a thing.

> particularly since it's so easy to provide for backwards compatibility

Fair enough.

> (I mean, it *will* be easy once I've got my cddr's pointed in the
> right direction).

I can't even remember which is which of car and cdr, so no further
comments from me here... ;-)

> I've always found it a bit confusing in Gnus, how some places require
> cons cells and others two-element lists, but there's no reason not to
> main consistency with the past, and I'll change that, too.

That why we have to old folk-expression: "It's all lisp to me".


  :-),

   Adam

-- 
 "Remember, Robert, in life anything can happen."           Adam Sjøgren
                                                       asjo@koldfront.dk



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-05 17:34         ` Adam Sjøgren
@ 2020-11-06  3:50           ` Eric Abrahamsen
  0 siblings, 0 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-06  3:50 UTC (permalink / raw)
  To: Adam Sjøgren; +Cc: ding

Adam Sjøgren <asjo@koldfront.dk> writes:

> Eric writes:
>
>>> Or document that an update is needed and leave it at that?
>
>> That would be a little mean,
>
> Maybe, but sometimes it makes for less clutter - especially if nnir no
> longer is a thing.
>
>> particularly since it's so easy to provide for backwards compatibility
>
> Fair enough.
>
>> (I mean, it *will* be easy once I've got my cddr's pointed in the
>> right direction).
>
> I can't even remember which is which of car and cdr, so no further
> comments from me here... ;-)

The *scratch* buffer saves us all.

>> I've always found it a bit confusing in Gnus, how some places require
>> cons cells and others two-element lists, but there's no reason not to
>> main consistency with the past, and I'll change that, too.
>
> That why we have to old folk-expression: "It's all lisp to me".

We should have a mug!

I've pushed a fix for this, and changed the option type, and made a
couple of other small improvements (the larger/smaller search key in
imap now accepts KB/MB units).

Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 17:15 New "gnus-search" syntax and interface Eric Abrahamsen
                   ` (2 preceding siblings ...)
  2020-11-05 11:58 ` Adam Sjøgren
@ 2020-11-06  6:23 ` Jose A. Ortega Ruiz
       [not found]   ` <12MvvhYMQyipeZzkJx1ODwHAD4xQZo6qw1FSX6nvgZAyLZCPkEVFXXGxOQTuxL1zvwZC6BER4jnUFXNgIEjIZA==@protonmail.internalid>
  2020-11-07  4:59   ` Eric Abrahamsen
  2020-11-13 11:07 ` Eric S Fraga
  2020-11-18  9:21 ` yoctocell
  5 siblings, 2 replies; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-06  6:23 UTC (permalink / raw)
  To: ding


Hi,

Thanks for this, it looks very promising.

I'm trying it, and seems to be working, but with some unexpected
behaviour.  More concretely, if i put my cursor on a topic and try G G,
i don't always find results that are otherwise found when G G-ing on one
of its groups.

Also, sometimes some of my ephemeral queries seem to get "stick", and
they are repeated over and over every time i do 'g' to refresh my group,
even though i cannot see any group (it was ephemeral).

I don't have good recipes (yet) to reproduce, but i'll try.

Cheers,
jao
-- 
You don’t stop doing things because you get old.
You get old because you stop doing things.
  - Rosamunde Pilcher



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-06  6:23 ` Jose A. Ortega Ruiz
       [not found]   ` <12MvvhYMQyipeZzkJx1ODwHAD4xQZo6qw1FSX6nvgZAyLZCPkEVFXXGxOQTuxL1zvwZC6BER4jnUFXNgIEjIZA==@protonmail.internalid>
@ 2020-11-07  4:59   ` Eric Abrahamsen
  2020-11-08  1:23     ` Jose A. Ortega Ruiz
  2020-11-08  2:43     ` Jose A. Ortega Ruiz
  1 sibling, 2 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-07  4:59 UTC (permalink / raw)
  To: Jose A. Ortega Ruiz; +Cc: ding

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> Hi,
>
> Thanks for this, it looks very promising.
>
> I'm trying it, and seems to be working, but with some unexpected
> behaviour.  More concretely, if i put my cursor on a topic and try G G,
> i don't always find results that are otherwise found when G G-ing on one
> of its groups.

Please do give more details about this if you notice anything. The code
for collecting groups is relatively simple, and I don't quite see how
the results could be different. If you're willing to experiment, you
could run this code snippet while marking groups, or putting point on
topic headings, and see if anything looks different:

(seq-group-by
 (lambda (elt) (gnus-group-server elt))
 (or gnus-group-marked
     (if (gnus-group-group-name)
	 (list (gnus-group-group-name))
       (cdr
	(assoc (gnus-group-topic-name) gnus-topic-alist)))))

> Also, sometimes some of my ephemeral queries seem to get "stick", and
> they are repeated over and over every time i do 'g' to refresh my group,
> even though i cannot see any group (it was ephemeral).

Huh, that's odd. How do you know the queries are repeated if the group's
gone?

These issues both sound like they might be more of an nnselect thing
than a gnus-search thing -- did you notice any behavior like this after
the nnselect update (landed September 5 in ecfc13e), but before the
gnus-search update?

It's also possible that 206cff8 might have thrown something off, since
that commit also touches code in this area.

Thanks,
Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-07  4:59   ` Eric Abrahamsen
@ 2020-11-08  1:23     ` Jose A. Ortega Ruiz
       [not found]       ` <VEKLrJRKnbVIVztgsX0O5q0i9OwitXf-t5q2hcVN-ZDq0SRE1KS4DIpk7iNeQxIhD1_9AC4DWOdDJsJW2XCMlg==@protonmail.internalid>
  2020-11-08  2:38       ` Eric Abrahamsen
  2020-11-08  2:43     ` Jose A. Ortega Ruiz
  1 sibling, 2 replies; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-08  1:23 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

On Fri, Nov 06 2020, Eric Abrahamsen wrote:

>> Also, sometimes some of my ephemeral queries seem to get "stick", and
>> they are repeated over and over every time i do 'g' to refresh my group,
>> even though i cannot see any group (it was ephemeral).
>
> Huh, that's odd. How do you know the queries are repeated if the
> group's gone?

Because i see messages in Messages describing them, which mention both
the query and the (gone) ephemeral group.  Also because if one of them
takes a while, i see the update halting on it.

> These issues both sound like they might be more of an nnselect thing
> than a gnus-search thing -- did you notice any behavior like this after
> the nnselect update (landed September 5 in ecfc13e), but before the
> gnus-search update?

Hmmm, it could be.  I don't use search a lot, but i'm thinking now that
maybe i saw this problem once before.  

I am thinking that it could be that this only happens when there's an
error executing the query: maybe when an exception is raised in the
middle of the process, part of the code that cleans up the ephemeral
state is not called?

Sorry about the vagueness. I have bootstraped emacs again, and will try
to report something more sensible (i thought submitted a bug report
yesterday, but i discovered today that i hadn't sent it yet!).

Cheers,
jao
-- 
We shall do a much better programming job, provided we approach the
task with a full appreciation of its tremendous difficulty, provided
that we respect the intrinsic limitations of the human mind and
approach the task as very humble programmers. -Alan Turing


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-08  1:23     ` Jose A. Ortega Ruiz
       [not found]       ` <VEKLrJRKnbVIVztgsX0O5q0i9OwitXf-t5q2hcVN-ZDq0SRE1KS4DIpk7iNeQxIhD1_9AC4DWOdDJsJW2XCMlg==@protonmail.internalid>
@ 2020-11-08  2:38       ` Eric Abrahamsen
  2020-11-08  2:51         ` Jose A. Ortega Ruiz
  2020-11-08  2:55         ` Andrew Cohen
  1 sibling, 2 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-08  2:38 UTC (permalink / raw)
  To: Jose A. Ortega Ruiz; +Cc: ding, Andrew Cohen

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Fri, Nov 06 2020, Eric Abrahamsen wrote:
>
>>> Also, sometimes some of my ephemeral queries seem to get "stick", and
>>> they are repeated over and over every time i do 'g' to refresh my group,
>>> even though i cannot see any group (it was ephemeral).
>>
>> Huh, that's odd. How do you know the queries are repeated if the
>> group's gone?
>
> Because i see messages in Messages describing them, which mention both
> the query and the (gone) ephemeral group.  Also because if one of them
> takes a while, i see the update halting on it.
>
>> These issues both sound like they might be more of an nnselect thing
>> than a gnus-search thing -- did you notice any behavior like this after
>> the nnselect update (landed September 5 in ecfc13e), but before the
>> gnus-search update?
>
> Hmmm, it could be.  I don't use search a lot, but i'm thinking now that
> maybe i saw this problem once before.  
>
> I am thinking that it could be that this only happens when there's an
> error executing the query: maybe when an exception is raised in the
> middle of the process, part of the code that cleans up the ephemeral
> state is not called?

Aha! That does make sense. I just made an ephemeral nnselect group with
an `nnselect-function' that just raised an error, and the group does
hang around afterwards. I'm cc'ing Andy, the nnselect author, to see if
this is something specific to nnselect, or maybe just the way ephemeral
groups work in general.

Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-07  4:59   ` Eric Abrahamsen
  2020-11-08  1:23     ` Jose A. Ortega Ruiz
@ 2020-11-08  2:43     ` Jose A. Ortega Ruiz
  2020-11-08  5:03       ` Eric Abrahamsen
  1 sibling, 1 reply; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-08  2:43 UTC (permalink / raw)
  To: ding

On Fri, Nov 06 2020, Eric Abrahamsen wrote:

>> I'm trying it, and seems to be working, but with some unexpected
>> behaviour.  More concretely, if i put my cursor on a topic and try G G,
>> i don't always find results that are otherwise found when G G-ing on one
>> of its groups.
>
> Please do give more details about this if you notice anything. The code
> for collecting groups is relatively simple, and I don't quite see how
> the results could be different.

Hmmm, are nested topics supported?  That is, if i search on a topic that
contains subtopics, is the search including groups inside those
subtopics? 

I am trying for instance a query at the top Groups node, including all
my groups, to perform a global search, and it seems to never find any
results.

The same seems to happen with any nesting.  E.g.

    [Groups]
      [topic1]
        group1
        [subtopic1]
          group2

If i perform a successful query on [subtopic1] (which finds something in,
say group2), the same query on [topic1] will return an empty result.
Similarly, a query on [topic1] succeeding for group1 will fail when
performed on [Groups].

Regarding the queries performed over and over, i think i've found when
that happens.  If i issue a query with empty results (so i get a message
saying the nnselect ephemeral group is empty), from then on refreshing
the Groups buffer will repeat the query (or queries, they seem to
accumulate) again.  In Messages i see for instance the first time i
issue the query:

  Searching nnimap:group1...done
  Searching nnimap:group2...done
  Group nnselect:nnselect-877dqw8tq8.fsf contains no messages

And then from then on, a g in Groups will make the query be run again
(it takes as long as the first time), although in Messages i see only
the first two messages:

  Opening server
  Searching nnimap:group1...done
  Searching nnimap:group2...done

but no mention again of the ephemeral group. As you mentioned, this may
well be a problem in nnselect.

(By the way, all of this is searching nnimap groups from a local dovecot
server, using the IMAP engine.)

I hope this makes a bit more sense.  Please feel free to ask about any
further details!

Cheers,
jao



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-08  2:38       ` Eric Abrahamsen
@ 2020-11-08  2:51         ` Jose A. Ortega Ruiz
  2020-11-08  2:55         ` Andrew Cohen
  1 sibling, 0 replies; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-08  2:51 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding, Andrew Cohen

On Sat, Nov 07 2020, Eric Abrahamsen wrote:

[...]

>> I am thinking that it could be that this only happens when there's an
>> error executing the query: maybe when an exception is raised in the
>> middle of the process, part of the code that cleans up the ephemeral
>> state is not called?
>
> Aha! That does make sense. I just made an ephemeral nnselect group with
> an `nnselect-function' that just raised an error, and the group does
> hang around afterwards. I'm cc'ing Andy, the nnselect author, to see if
> this is something specific to nnselect, or maybe just the way ephemeral
> groups work in general.

I've observed that the "error" can be simply that the query returns no
results (nothing fancy like a misconfiguration or the like).  At the
very least, i can reproduce it reliably that way.

Cheers,
jao
-- 
Adding manpower to a late software project makes it later.
 -Fred Brooks


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-08  2:38       ` Eric Abrahamsen
  2020-11-08  2:51         ` Jose A. Ortega Ruiz
@ 2020-11-08  2:55         ` Andrew Cohen
  1 sibling, 0 replies; 49+ messages in thread
From: Andrew Cohen @ 2020-11-08  2:55 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: Jose A. Ortega Ruiz, ding, Andrew Cohen

Ephemeral groups don't clean up properly if the retrieval fails. I'll fix 
this in nnselect and then think about something more general for all 
ephemeral groups

Nov 8, 2020 10:38:41 AM Eric Abrahamsen <eric@ericabrahamsen.net>:

> "Jose A. Ortega Ruiz" <jao@gnu.org> writes:
>
>> On Fri, Nov 06 2020, Eric Abrahamsen wrote:
>>
>>>> Also, sometimes some of my ephemeral queries seem to get "stick", 
and
>>>> they are repeated over and over every time i do 'g' to refresh my 
group,
>>>> even though i cannot see any group (it was ephemeral).
>>>
>>> Huh, that's odd. How do you know the queries are repeated if the
>>> group's gone?
>>
>> Because i see messages in Messages describing them, which mention both
>> the query and the (gone) ephemeral group.  Also because if one of them
>> takes a while, i see the update halting on it.
>>
>>> These issues both sound like they might be more of an nnselect thing
>>> than a gnus-search thing -- did you notice any behavior like this 
after
>>> the nnselect update (landed September 5 in ecfc13e), but before the
>>> gnus-search update?
>>
>> Hmmm, it could be.  I don't use search a lot, but i'm thinking now 
that
>> maybe i saw this problem once before. 
>>
>> I am thinking that it could be that this only happens when there's an
>> error executing the query: maybe when an exception is raised in the
>> middle of the process, part of the code that cleans up the ephemeral
>> state is not called?
>
> Aha! That does make sense. I just made an ephemeral nnselect group with
> an `nnselect-function' that just raised an error, and the group does
> hang around afterwards. I'm cc'ing Andy, the nnselect author, to see if
> this is something specific to nnselect, or maybe just the way ephemeral
> groups work in general.
>
> Eric
>


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-08  2:43     ` Jose A. Ortega Ruiz
@ 2020-11-08  5:03       ` Eric Abrahamsen
  2020-11-08  7:16         ` Jose A. Ortega Ruiz
  0 siblings, 1 reply; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-08  5:03 UTC (permalink / raw)
  To: Jose A. Ortega Ruiz; +Cc: ding

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Fri, Nov 06 2020, Eric Abrahamsen wrote:
>
>>> I'm trying it, and seems to be working, but with some unexpected
>>> behaviour.  More concretely, if i put my cursor on a topic and try G G,
>>> i don't always find results that are otherwise found when G G-ing on one
>>> of its groups.
>>
>> Please do give more details about this if you notice anything. The code
>> for collecting groups is relatively simple, and I don't quite see how
>> the results could be different.
>
> Hmmm, are nested topics supported?  That is, if i search on a topic that
> contains subtopics, is the search including groups inside those
> subtopics? 

That's it! Nested topics are not supported. I'll put this on the list
and get it fixed in the next couple days.

Thanks,
Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-08  5:03       ` Eric Abrahamsen
@ 2020-11-08  7:16         ` Jose A. Ortega Ruiz
  2020-11-12 20:51           ` Eric Abrahamsen
  0 siblings, 1 reply; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-08  7:16 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

On Sat, Nov 07 2020, Eric Abrahamsen wrote:

>> Hmmm, are nested topics supported?  That is, if i search on a topic that
>> contains subtopics, is the search including groups inside those
>> subtopics?
>
> That's it! Nested topics are not supported. I'll put this on the list
> and get it fixed in the next couple days.

Excellent. Count on me to give it a test drive!

Thanks a lot,
jao
-- 
A ship in port is safe; but that is not what ships are built
for. -Grace Hopper (1906-1992)


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-08  7:16         ` Jose A. Ortega Ruiz
@ 2020-11-12 20:51           ` Eric Abrahamsen
  2020-11-13  3:17             ` Jose A. Ortega Ruiz
  0 siblings, 1 reply; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-12 20:51 UTC (permalink / raw)
  To: Jose A. Ortega Ruiz; +Cc: ding

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]


On 11/08/20 07:16 AM, Jose A. Ortega Ruiz wrote:
> On Sat, Nov 07 2020, Eric Abrahamsen wrote:
>
>>> Hmmm, are nested topics supported?  That is, if i search on a topic that
>>> contains subtopics, is the search including groups inside those
>>> subtopics?
>>
>> That's it! Nested topics are not supported. I'll put this on the list
>> and get it fixed in the next couple days.
>
> Excellent. Count on me to give it a test drive!

Okay here goes -- the attached patch should make topic searching work as
expected. Would you give it a try?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnus-search-topics.diff --]
[-- Type: text/x-patch, Size: 1040 bytes --]

diff --git a/lisp/gnus/gnus-group.el b/lisp/gnus/gnus-group.el
index c8b97c0852..ec3a49344e 100644
--- a/lisp/gnus/gnus-group.el
+++ b/lisp/gnus/gnus-group.el
@@ -3194,8 +3194,10 @@ gnus-group-make-search-group
 		  (or gnus-group-marked
 		      (if (gnus-group-group-name)
 			  (list (gnus-group-group-name))
-			(cdr
-			 (assoc (gnus-group-topic-name) gnus-topic-alist))))))))
+			(mapcar #'caadr
+				(gnus-topic-find-groups
+				 (gnus-group-topic-name)
+				 nil 'all nil 'recursive))))))))
 	     (query-spec
 	      (or
 	       (cdr (assq 'search-query-spec specs))
@@ -3243,8 +3245,10 @@ gnus-group-read-ephemeral-search-group
 		 (or gnus-group-marked
 		     (if (gnus-group-group-name)
 			 (list (gnus-group-group-name))
-		       (cdr
-			(assoc (gnus-group-topic-name) gnus-topic-alist))))))))
+		       (mapcar #'caadr
+				(gnus-topic-find-groups
+				 (gnus-group-topic-name)
+				 nil 'all nil 'recursive))))))))
 	 (query-spec
 	  (or (cdr (assq 'search-query-spec specs))
 	      (cdr (assq 'nnir-query-spec specs))

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-12 20:51           ` Eric Abrahamsen
@ 2020-11-13  3:17             ` Jose A. Ortega Ruiz
       [not found]               ` <-NlYHnQ3eprZs5vpzJzwiWpUHjyOUwbkarR4R4m8DK_5ik1XoE8SVsxNfQWJwgUWVIfjwxU5eCbwaWIzwZUJNQ==@protonmail.internalid>
  2020-11-13  6:38               ` Eric Abrahamsen
  0 siblings, 2 replies; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-13  3:17 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

On Thu, Nov 12 2020, Eric Abrahamsen wrote:

> On 11/08/20 07:16 AM, Jose A. Ortega Ruiz wrote:
>> On Sat, Nov 07 2020, Eric Abrahamsen wrote:
>>
>>>> Hmmm, are nested topics supported?  That is, if i search on a topic that
>>>> contains subtopics, is the search including groups inside those
>>>> subtopics?
>>>
>>> That's it! Nested topics are not supported. I'll put this on the list
>>> and get it fixed in the next couple days.
>>
>> Excellent. Count on me to give it a test drive!
>
> Okay here goes -- the attached patch should make topic searching work as
> expected. Would you give it a try?

hmmm, no.  with this patch applied, GG on a topic raises:

  gnus-topic-find-groups("inbox" nil all nil recursive)
  gnus-group-read-ephemeral-search-group(nil)
  funcall-interactively(gnus-group-read-ephemeral-search-group nil)
  call-interactively(gnus-group-read-ephemeral-search-group nil nil)
  command-execute(gnus-group-read-ephemeral-search-group)


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13  3:17             ` Jose A. Ortega Ruiz
       [not found]               ` <-NlYHnQ3eprZs5vpzJzwiWpUHjyOUwbkarR4R4m8DK_5ik1XoE8SVsxNfQWJwgUWVIfjwxU5eCbwaWIzwZUJNQ==@protonmail.internalid>
@ 2020-11-13  6:38               ` Eric Abrahamsen
  2020-11-13 19:15                 ` Jose A. Ortega Ruiz
  1 sibling, 1 reply; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-13  6:38 UTC (permalink / raw)
  To: Jose A. Ortega Ruiz; +Cc: ding

[-- Attachment #1: Type: text/plain, Size: 1159 bytes --]

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Thu, Nov 12 2020, Eric Abrahamsen wrote:
>
>> On 11/08/20 07:16 AM, Jose A. Ortega Ruiz wrote:
>>> On Sat, Nov 07 2020, Eric Abrahamsen wrote:
>>>
>>>>> Hmmm, are nested topics supported?  That is, if i search on a topic that
>>>>> contains subtopics, is the search including groups inside those
>>>>> subtopics?
>>>>
>>>> That's it! Nested topics are not supported. I'll put this on the list
>>>> and get it fixed in the next couple days.
>>>
>>> Excellent. Count on me to give it a test drive!
>>
>> Okay here goes -- the attached patch should make topic searching work as
>> expected. Would you give it a try?
>
> hmmm, no.  with this patch applied, GG on a topic raises:
>
>   gnus-topic-find-groups("inbox" nil all nil recursive)
>   gnus-group-read-ephemeral-search-group(nil)
>   funcall-interactively(gnus-group-read-ephemeral-search-group nil)
>   call-interactively(gnus-group-read-ephemeral-search-group nil nil)
>   command-execute(gnus-group-read-ephemeral-search-group)

Bah, it's always the last minute little "inconsequential" changes that
get you. This version should at least not explode.



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnus-search-topics.diff --]
[-- Type: text/x-patch, Size: 1022 bytes --]

diff --git a/lisp/gnus/gnus-group.el b/lisp/gnus/gnus-group.el
index c8b97c0852..ec3a49344e 100644
--- a/lisp/gnus/gnus-group.el
+++ b/lisp/gnus/gnus-group.el
@@ -3194,8 +3194,10 @@ gnus-group-make-search-group
 		  (or gnus-group-marked
 		      (if (gnus-group-group-name)
 			  (list (gnus-group-group-name))
-			(cdr
-			 (assoc (gnus-group-topic-name) gnus-topic-alist))))))))
+			(mapcar #'caadr
+				(gnus-topic-find-groups
+				 (gnus-group-topic-name)
+				 nil 'all nil t))))))))
 	     (query-spec
 	      (or
 	       (cdr (assq 'search-query-spec specs))
@@ -3243,8 +3245,10 @@ gnus-group-read-ephemeral-search-group
 		 (or gnus-group-marked
 		     (if (gnus-group-group-name)
 			 (list (gnus-group-group-name))
-		       (cdr
-			(assoc (gnus-group-topic-name) gnus-topic-alist))))))))
+		       (mapcar #'caadr
+				(gnus-topic-find-groups
+				 (gnus-group-topic-name)
+				 nil 'all nil t))))))))
 	 (query-spec
 	  (or (cdr (assq 'search-query-spec specs))
 	      (cdr (assq 'nnir-query-spec specs))

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 17:15 New "gnus-search" syntax and interface Eric Abrahamsen
                   ` (3 preceding siblings ...)
  2020-11-06  6:23 ` Jose A. Ortega Ruiz
@ 2020-11-13 11:07 ` Eric S Fraga
  2020-11-13 12:39   ` Gijs Hillenius
  2020-11-18  9:21 ` yoctocell
  5 siblings, 1 reply; 49+ messages in thread
From: Eric S Fraga @ 2020-11-13 11:07 UTC (permalink / raw)
  To: ding

Eric,

pardon my possibly silly question but: how do I get search working
again?

I am tracking Emacs from git (updated this morning).  When I try to
search, I get no results.  I use notmuch as my search engine and my
groups are nnml.

If I try notmuch directly (within Emacs, notmuch package from MELPA),
searches work just fine.

I'm not going to indicate what settings I have currently as my .gnus.el
is an organic monster that has evolved over decades now.  I am happy to
clean up significantly but need to know what the basic structure &
settings should be and where to put them.

Thank you,
eric
-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4 on Debian bullseye/sid



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13 11:07 ` Eric S Fraga
@ 2020-11-13 12:39   ` Gijs Hillenius
  2020-11-13 13:01     ` Eric S Fraga
  0 siblings, 1 reply; 49+ messages in thread
From: Gijs Hillenius @ 2020-11-13 12:39 UTC (permalink / raw)
  To: ding

On 13 November 2020 11:07 Eric S Fraga, wrote:

> Eric,
>
> pardon my possibly silly question but: how do I get search working
> again?

Hi Eric

I ran into this too.

An immediate fix might be to

set gnus-search-use-parsed-queries to t

over here that restores GG the way it worked. And then we get to use the
other features too.




^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13 12:39   ` Gijs Hillenius
@ 2020-11-13 13:01     ` Eric S Fraga
  2020-11-13 16:15       ` Eric Abrahamsen
  0 siblings, 1 reply; 49+ messages in thread
From: Eric S Fraga @ 2020-11-13 13:01 UTC (permalink / raw)
  To: ding

On Friday, 13 Nov 2020 at 13:39, Gijs Hillenius wrote:
> set gnus-search-use-parsed-queries to t

Thanks for the suggestion.  I have tried this.  Still get no results.

However, it seems like the results must be found as the amount of time
Emacs sits after requesting the query seems to be a function of the
number of emails I would expect notmuch to return.  That is, there is a
very quick response when only a handful of emails would match the query
but seconds when hundreds or thousands would.

This would suggest that the query is being performed but the subsequent
processing is going wrong somewhere.

How can I debug what the search is doing?

Thank you,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4 on Debian bullseye/sid



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13 13:01     ` Eric S Fraga
@ 2020-11-13 16:15       ` Eric Abrahamsen
  2020-11-13 16:56         ` Eric S Fraga
  0 siblings, 1 reply; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-13 16:15 UTC (permalink / raw)
  To: Eric S Fraga; +Cc: ding

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Friday, 13 Nov 2020 at 13:39, Gijs Hillenius wrote:
>> set gnus-search-use-parsed-queries to t
>
> Thanks for the suggestion.  I have tried this.  Still get no results.
>
> However, it seems like the results must be found as the amount of time
> Emacs sits after requesting the query seems to be a function of the
> number of emails I would expect notmuch to return.  That is, there is a
> very quick response when only a handful of emails would match the query
> but seconds when hundreds or thousands would.
>
> This would suggest that the query is being performed but the subsequent
> processing is going wrong somewhere.
>
> How can I debug what the search is doing?

Bummer, I'm sorry about this. Can you show me your server config, which
I assume looks like:

'(nnml "" (nnir-search-engine notmuch (<maybe more config>)))

And then any nnir-* configuration you have, maybe
`nnir-notmuch-filter-group-names-function'? Any advising of functions?

Deep configuration of Gnus is a wonderful thing, but it does make it
hard to make changes to the software!

Anyway, let's get things working again as soon as possible.

Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13 16:15       ` Eric Abrahamsen
@ 2020-11-13 16:56         ` Eric S Fraga
  2020-11-13 17:21           ` Eric Abrahamsen
  0 siblings, 1 reply; 49+ messages in thread
From: Eric S Fraga @ 2020-11-13 16:56 UTC (permalink / raw)
  To: ding

On Friday, 13 Nov 2020 at 08:15, Eric Abrahamsen wrote:
> Bummer, I'm sorry about this. 

Don't worry!  It's only a minor inconvenience as I can still use
"in-group" searches/filtering.

> Can you show me your server config, which I assume looks like:
>
> '(nnml "" (nnir-search-engine notmuch (<maybe more config>)))

I do have:

'(nnml "outlook" (nnir-search-engine notmuch) (get-new-mail t))

and I used to have:

  (setq nnir-notmuch-filter-group-names-function nil)

but I've deleted that for the moment with no effect (restarting emacs &
gnus).

Can you tell me the minimal settings I need to get nnml groups searched
with notmuch?  My server setup is in need of a deep clean so this is a
good excuse.

Thank you,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4 on Debian bullseye/sid



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13 16:56         ` Eric S Fraga
@ 2020-11-13 17:21           ` Eric Abrahamsen
  2020-11-13 20:06             ` Jose A. Ortega Ruiz
  0 siblings, 1 reply; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-13 17:21 UTC (permalink / raw)
  To: Eric S Fraga; +Cc: ding

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Friday, 13 Nov 2020 at 08:15, Eric Abrahamsen wrote:
>> Bummer, I'm sorry about this. 
>
> Don't worry!  It's only a minor inconvenience as I can still use
> "in-group" searches/filtering.
>
>> Can you show me your server config, which I assume looks like:
>>
>> '(nnml "" (nnir-search-engine notmuch (<maybe more config>)))
>
> I do have:
>
> '(nnml "outlook" (nnir-search-engine notmuch) (get-new-mail t))

Okay, simple enough.

> and I used to have:
>
>   (setq nnir-notmuch-filter-group-names-function nil)
>
> but I've deleted that for the moment with no effect (restarting emacs &
> gnus).

That used to be nil by default, so it would only come into play if you'd
set it to something.

> Can you tell me the minimal settings I need to get nnml groups searched
> with notmuch?  My server setup is in need of a deep clean so this is a
> good excuse.

The cleanest thing to do is probably keep all the config in the server
definition itself. So:

'(nnml "outlook"
       (gnus-search-engine
	gnus-search-notmuch
	(remove-prefix "/path/to/mail/toplevel/")
	(config-file "/path/to/notmuch/config")))

If you anticipate switching back to Emacs 27 you can keep using
'(nnir-search-engine notmuch), that will continue to be honored.

If you feel like the searches are being conducted correctly, then my
guess is that the 'remove-prefix is wrong, and Gnus is parsing notmuch's
output incorrectly.

Did you customize `nnir-notmuch-remove-prefix'? I also note that its
default value is:

(regexp-quote (or (getenv "MAILDIR") (expand-file-name "~/Mail")))

While the default value of the equivalent
`gnus-search-notmuch-remove-prefix' is:

(concat (getenv "HOME") "/Mail/")

I'm not sure why I did that differently (though on my machine the result
is the same). Can you try the config above (with the 'remove-prefix
clause) and confirm that the path in that clause really is the path to
the toplevel directory of your mail installation? Basically, removing
the 'remove-prefix should result in a relative file path where the
directory is the group name.

That's my first and best guess.

Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13  6:38               ` Eric Abrahamsen
@ 2020-11-13 19:15                 ` Jose A. Ortega Ruiz
  2020-11-14  1:02                   ` Eric Abrahamsen
  0 siblings, 1 reply; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-13 19:15 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

On Thu, Nov 12 2020, Eric Abrahamsen wrote:

[...]

> Bah, it's always the last minute little "inconsequential" changes that
> get you.

indeed :)

> This version should at least not explode.

it's not only not exploding, but working as expected.  thanks!

jao
-- 
If you could kick in the pants the person responsible for most of your
trouble, you wouldn't sit for a month. — Theodore Roosevelt


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13 17:21           ` Eric Abrahamsen
@ 2020-11-13 20:06             ` Jose A. Ortega Ruiz
  2020-11-16 10:44               ` Eric S Fraga
  2020-11-16 15:00               ` Eric S Fraga
  0 siblings, 2 replies; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-13 20:06 UTC (permalink / raw)
  To: ding


Hi Eric F,

This sounds suspiciously similar to a problem i had to overcome trying
to use gnus-search-notmuch in a slightly different context: the search
for groups in the results was using the group names using dots instead
of slashes.  In my case it was with an nntp group, but i was expecting
it to happen with nnml too.  if evaluating the following snippet fixes
it for you, then it's exactly the problem i encountered (Eric A. will
remember):

    (with-eval-after-load "gnus-search"
      (cl-defmethod gnus-search-indexed-parse-output :around
        ((e gnus-search-notmuch) s q groups)
        (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
                          groups)))
          (cl-call-next-method e s q gs))))

(Here, i'm assuming your search engine is defined as
gnus-search-notmuch).

HTH,
jao
-- 
It is a truism of escape plans that the problem with going anywhere is
that you take yourself with you -- Emma Brockes



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13 19:15                 ` Jose A. Ortega Ruiz
@ 2020-11-14  1:02                   ` Eric Abrahamsen
  0 siblings, 0 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-14  1:02 UTC (permalink / raw)
  To: Jose A. Ortega Ruiz; +Cc: ding

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Thu, Nov 12 2020, Eric Abrahamsen wrote:
>
> [...]
>
>> Bah, it's always the last minute little "inconsequential" changes that
>> get you.
>
> indeed :)
>
>> This version should at least not explode.
>
> it's not only not exploding, but working as expected.  thanks!

In it goes...


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13 20:06             ` Jose A. Ortega Ruiz
@ 2020-11-16 10:44               ` Eric S Fraga
  2020-11-16 15:00               ` Eric S Fraga
  1 sibling, 0 replies; 49+ messages in thread
From: Eric S Fraga @ 2020-11-16 10:44 UTC (permalink / raw)
  To: ding

Eric & jao,

thank you for the suggestions.  I'm swamped with work but will try these
out sometime this week and will get back to you with the outcome!

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4 on Debian bullseye/sid



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-13 20:06             ` Jose A. Ortega Ruiz
  2020-11-16 10:44               ` Eric S Fraga
@ 2020-11-16 15:00               ` Eric S Fraga
  2020-11-16 18:47                 ` Eric Abrahamsen
  1 sibling, 1 reply; 49+ messages in thread
From: Eric S Fraga @ 2020-11-16 15:00 UTC (permalink / raw)
  To: ding

On Friday, 13 Nov 2020 at 20:06, Jose A. Ortega Ruiz wrote:
> This sounds suspiciously similar to a problem i had to overcome trying
> to use gnus-search-notmuch in a slightly different context: the search
> for groups in the results was using the group names using dots instead
> of slashes.  

Your advice code did the job!  Thank you.  Search is now working as it
should.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4 on Debian bullseye/sid



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-16 15:00               ` Eric S Fraga
@ 2020-11-16 18:47                 ` Eric Abrahamsen
  2020-11-17 11:04                   ` Eric S Fraga
  0 siblings, 1 reply; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-16 18:47 UTC (permalink / raw)
  To: ding

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Friday, 13 Nov 2020 at 20:06, Jose A. Ortega Ruiz wrote:
>> This sounds suspiciously similar to a problem i had to overcome trying
>> to use gnus-search-notmuch in a slightly different context: the search
>> for groups in the results was using the group names using dots instead
>> of slashes.  
>
> Your advice code did the job!  Thank you.  Search is now working as it
> should.

Well that's pretty weird. I didn't think I changed any of the actual
parsing behavior, only reorganized code. I'll do a more careful
comparison of the original nnir code and the new gnus-search stuff, and
see what happened. You shouldn't have to advise anything just to get it
basically functioning.



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-16 18:47                 ` Eric Abrahamsen
@ 2020-11-17 11:04                   ` Eric S Fraga
  2020-11-17 23:58                     ` Eric Abrahamsen
  0 siblings, 1 reply; 49+ messages in thread
From: Eric S Fraga @ 2020-11-17 11:04 UTC (permalink / raw)
  To: ding

On Monday, 16 Nov 2020 at 10:47, Eric Abrahamsen wrote:
> Well that's pretty weird. I didn't think I changed any of the actual
> parsing behavior, only reorganized code. 

With these complex systems, the whole concept of emergent behaviour
raises its head! ;-)

Happy to test out anything you change etc., of course.

Thank you,
eric

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4 on Debian bullseye/sid



^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-17 11:04                   ` Eric S Fraga
@ 2020-11-17 23:58                     ` Eric Abrahamsen
  2020-11-18  0:46                       ` Jose A. Ortega Ruiz
  0 siblings, 1 reply; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-17 23:58 UTC (permalink / raw)
  To: Eric S Fraga; +Cc: ding

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Monday, 16 Nov 2020 at 10:47, Eric Abrahamsen wrote:
>> Well that's pretty weird. I didn't think I changed any of the actual
>> parsing behavior, only reorganized code. 
>
> With these complex systems, the whole concept of emergent behaviour
> raises its head! ;-)
>
> Happy to test out anything you change etc., of course.

Thanks very much. This is still baffling me. If you (and/or Jose, or
anyone else) have a moment, would you please send me the command-line
output of a successful notmuch search (ie absolute filenames) which
fails in Emacs master right now? Feel free to anonymize however
necessary, though obviously leaving the group-name part of the filepath
alone. Plus whatever notmuch-related config you've got...

Thanks!
Eri


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-17 23:58                     ` Eric Abrahamsen
@ 2020-11-18  0:46                       ` Jose A. Ortega Ruiz
  2020-11-18 20:40                         ` Eric Abrahamsen
       [not found]                         ` <1x7NOCTHudFuCvB0kEBCGDds7KKAdbu-tZRD41ue36qG8dPBUSj7W9lq7CK5WJwL50cGQKZcom2KMkW_2VZi8Q==@protonmail.internalid>
  0 siblings, 2 replies; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-18  0:46 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: Eric S Fraga, ding

On Tue, Nov 17 2020, Eric Abrahamsen wrote:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> On Monday, 16 Nov 2020 at 10:47, Eric Abrahamsen wrote:
>>> Well that's pretty weird. I didn't think I changed any of the actual
>>> parsing behavior, only reorganized code. 
>>
>> With these complex systems, the whole concept of emergent behaviour
>> raises its head! ;-)
>>
>> Happy to test out anything you change etc., of course.
>
> Thanks very much. This is still baffling me. If you (and/or Jose, or
> anyone else) have a moment, would you please send me the command-line
> output of a successful notmuch search (ie absolute filenames) which
> fails in Emacs master right now? Feel free to anonymize however
> necessary, though obviously leaving the group-name part of the filepath
> alone. Plus whatever notmuch-related config you've got...

If i perform a gnus-search for "baffled" in this group, which is called
by gnus "nntp:gmane.emacs.gnus.general", the buffer where
gnus-search-indexed-parse-output runs has the following output from
notmuch (which correctly inserts real path names):

   /home/jao/var/mail/gmane/emacs/gnus/general/65
   /home/jao/var/mail/feeds/news/cur/1605280382.M861830P724182.osgiliath,S=3292,W=3331:2,Sa
   /home/jao/var/mail/gwene/org/arxiv/computer/science/1102

where /home/jao/var/mail is my remove-prefix.  

That method (gnus-search.el:1364) is receiving as its 'groups' argument
the value '("gmane.emacs.gnus.general").  the method constructs a regexp
that it calls group-regexp using that value, and that regexp is wrong.

That is because it uses as its value:

    (when groups
      (regexp-opt (mapcar (lambda (x) (gnus-group-real-name x)) groups)))

which, for grous being '("gmane.emacs.gnus.general"), evaluates to:

    "\\(?:gmane\\.emacs\\.gnus\\.general\\)"

That is wrong because the method then tries to match in the results
buffer, using that regexp, things like:

   /home/jao/var/mail/gmane.emacs.gnus.general/65

instead of correct pathnames such as:

   /home/jao/var/mail/gmane/emacs/gnus/general/65

The end result is that, despite notmuch finding correct paths and
inserting them correctly in the temporary results buffer,
gnus-search-indexed-parse-output fails to recognise them when it is
filtering those belonging to the searched for groups.  And this is
because it's constructing a wrong regular expression in
gnus-search.el:1367, and using it in gnus-search.el:1381.

The workaround my advice implements is changing the value of the input
parameter groups from '("foo.bar") to '("foo/bar").

I'm most probably belabouring here many things obvious to you, sorry
about that :)

Hope this helps,
jao
-- 
Flags are bits of colored cloth that governments use first to
shrink-wrap people's brains and then as ceremonial shrouds to bury the
dead. -Arundhati Roy, writer and activist (b. 1961)


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-04 17:15 New "gnus-search" syntax and interface Eric Abrahamsen
                   ` (4 preceding siblings ...)
  2020-11-13 11:07 ` Eric S Fraga
@ 2020-11-18  9:21 ` yoctocell
  2020-11-18 20:53   ` Eric Abrahamsen
  5 siblings, 1 reply; 49+ messages in thread
From: yoctocell @ 2020-11-18  9:21 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

On 4 November 2020 09:15, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> [Resending via ding as the newsgroup seems to be unresponsive]
>
> Hi all,
>
> For those of you on Emacs master, I just pushed the gnus-search.el
> library, which I've been sitting on for a while. It obsoletes nnir.el
> (while borrowing some of its code), and uses nnselect to display search
> results.
>
> Under default settings, it should do nothing at all, and searches should
> behave exactly as before. The nnir.el library has been moved to
> lisp/obsolete, so code that requires it should not break, but it is no
> longer needed.
>
> If anything breaks for anyone out of the box (though it shouldn't),
> please let me know right away!

I am new to gnus and trying to setup search with mairix and nnmaildir, I have the
following in my config

  (setq gnus-select-method '(nnnil "")
	gnus-secondary-select-methods
	'((nntp "news.gwene.org")
	  (nnmaildir "yoctocell"
		     (directory "~/.local/share/mail/yoctocell@disroot.org")
                     (gnus-search-engine gnus-search-mairix))))

When I press G G and enter a query, I get an error saying

  Group nnselect:nnselect-86mtzfrr62.fsf contains no messages

Am I doing something wrong? Thanks in advance!

-- 
yoctocell


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-18  0:46                       ` Jose A. Ortega Ruiz
@ 2020-11-18 20:40                         ` Eric Abrahamsen
       [not found]                         ` <1x7NOCTHudFuCvB0kEBCGDds7KKAdbu-tZRD41ue36qG8dPBUSj7W9lq7CK5WJwL50cGQKZcom2KMkW_2VZi8Q==@protonmail.internalid>
  1 sibling, 0 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-18 20:40 UTC (permalink / raw)
  To: Jose A. Ortega Ruiz; +Cc: Eric S Fraga, ding

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Tue, Nov 17 2020, Eric Abrahamsen wrote:
>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> On Monday, 16 Nov 2020 at 10:47, Eric Abrahamsen wrote:
>>>> Well that's pretty weird. I didn't think I changed any of the actual
>>>> parsing behavior, only reorganized code. 
>>>
>>> With these complex systems, the whole concept of emergent behaviour
>>> raises its head! ;-)
>>>
>>> Happy to test out anything you change etc., of course.
>>
>> Thanks very much. This is still baffling me. If you (and/or Jose, or
>> anyone else) have a moment, would you please send me the command-line
>> output of a successful notmuch search (ie absolute filenames) which
>> fails in Emacs master right now? Feel free to anonymize however
>> necessary, though obviously leaving the group-name part of the filepath
>> alone. Plus whatever notmuch-related config you've got...
>
> If i perform a gnus-search for "baffled" in this group, which is called
> by gnus "nntp:gmane.emacs.gnus.general", the buffer where
> gnus-search-indexed-parse-output runs has the following output from
> notmuch (which correctly inserts real path names):
>
>    /home/jao/var/mail/gmane/emacs/gnus/general/65
>    /home/jao/var/mail/feeds/news/cur/1605280382.M861830P724182.osgiliath,S=3292,W=3331:2,Sa
>    /home/jao/var/mail/gwene/org/arxiv/computer/science/1102
>
> where /home/jao/var/mail is my remove-prefix.  
>
> That method (gnus-search.el:1364) is receiving as its 'groups' argument
> the value '("gmane.emacs.gnus.general").  the method constructs a regexp
> that it calls group-regexp using that value, and that regexp is wrong.
>
> That is because it uses as its value:
>
>     (when groups
>       (regexp-opt (mapcar (lambda (x) (gnus-group-real-name x)) groups)))
>
> which, for grous being '("gmane.emacs.gnus.general"), evaluates to:
>
>     "\\(?:gmane\\.emacs\\.gnus\\.general\\)"
>
> That is wrong because the method then tries to match in the results
> buffer, using that regexp, things like:
>
>    /home/jao/var/mail/gmane.emacs.gnus.general/65
>
> instead of correct pathnames such as:
>
>    /home/jao/var/mail/gmane/emacs/gnus/general/65
>
> The end result is that, despite notmuch finding correct paths and
> inserting them correctly in the temporary results buffer,
> gnus-search-indexed-parse-output fails to recognise them when it is
> filtering those belonging to the searched for groups.  And this is
> because it's constructing a wrong regular expression in
> gnus-search.el:1367, and using it in gnus-search.el:1381.
>
> The workaround my advice implements is changing the value of the input
> parameter groups from '("foo.bar") to '("foo/bar").
>
> I'm most probably belabouring here many things obvious to you, sorry
> about that :)

No, that's perfect, I'm trying to make completely sure I understand
what's going on here. The original nnir-run-* functions all did pretty
much the same thing but with slightly different code, and in combining
all those functions I didn't get everything exactly in place.

The original notmuch code used the GROUPS parameter to build a "--path"
option for the notmuch call, but even if I re-instate that, we'll still
want this filter here for the other engines.

Would you try this version of the function? It should just permit any
path separator when checking for group name matches.

Thanks,
Eric


(cl-defmethod gnus-search-indexed-parse-output ((engine gnus-search-indexed)
server query &optional groups)
(let ((prefix (slot-value engine 'remove-prefix))
 (group-regexp (when groups
			(mapconcat
			 (lambda (x)
			   (replace-regexp-in-string
			    ;; Accept any of [.\/] as path separators.
			    "[.\\/]" "[.\\\\/]"
			    (gnus-group-real-name x)))
			 groups "\\|")))
	artlist vectors article group)
    (goto-char (point-min))
    (while (not (eobp))
      (pcase-let ((`(,f-name ,score) (gnus-search-indexed-extract engine)))
	(when (and (file-readable-p f-name)
		   (null (file-directory-p f-name))
		   (or (null groups)
		       (and (gnus-search-single-p query)
			    (alist-get 'thread query))
		       (string-match-p group-regexp f-name)))
	  (push (list f-name score) artlist))))
    ;; Are we running an additional grep query?
    (when-let ((grep-reg (alist-get 'grep query)))
      (setq artlist (gnus-search-grep-search engine artlist grep-reg)))
    ;; Turn (file-name score) into [group article score].
    (pcase-dolist (`(,f-name ,score) artlist)
      (setq article (file-name-nondirectory f-name))
      ;; Remove prefix.
      (when (and prefix
		 (file-name-absolute-p prefix)
		 (string-match (concat "^"
				       (file-name-as-directory prefix))
			       f-name))
	(setq group (replace-match "" t t (file-name-directory f-name))))
      ;; Break the directory name down until it's something that
      ;; (probably) can be used as a group name.
      (setq group
	    (replace-regexp-in-string
	     "[/\\]" "."
	     (replace-regexp-in-string
	      "/?\\(cur\\|new\\|tmp\\)?/\\'" ""
	      (replace-regexp-in-string
	       "^[./\\]" ""
	       group nil t)
	      nil t)
	     nil t))

      (push (vector (gnus-group-full-name group server)
		    (if (string-match-p "\\`[[:digit:]]+\\'" article)
			(string-to-number article)
		      (nnmaildir-base-name-to-article-number
		       (substring article 0 (string-match ":" article))
		       group nil))
		    (if (numberp score)
			score
		      (string-to-number score)))
	    vectors))
    vectors))


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-18  9:21 ` yoctocell
@ 2020-11-18 20:53   ` Eric Abrahamsen
  2020-11-19 11:02     ` yoctocell
  2020-11-22 12:56     ` yoctocell
  0 siblings, 2 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-18 20:53 UTC (permalink / raw)
  To: yoctocell; +Cc: ding

yoctocell <yoctocell@disroot.org> writes:

> On 4 November 2020 09:15, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> [Resending via ding as the newsgroup seems to be unresponsive]
>>
>> Hi all,
>>
>> For those of you on Emacs master, I just pushed the gnus-search.el
>> library, which I've been sitting on for a while. It obsoletes nnir.el
>> (while borrowing some of its code), and uses nnselect to display search
>> results.
>>
>> Under default settings, it should do nothing at all, and searches should
>> behave exactly as before. The nnir.el library has been moved to
>> lisp/obsolete, so code that requires it should not break, but it is no
>> longer needed.
>>
>> If anything breaks for anyone out of the box (though it shouldn't),
>> please let me know right away!
>
> I am new to gnus and trying to setup search with mairix and nnmaildir, I have the
> following in my config
>
>   (setq gnus-select-method '(nnnil "")
> 	gnus-secondary-select-methods
> 	'((nntp "news.gwene.org")
> 	  (nnmaildir "yoctocell"
> 		     (directory "~/.local/share/mail/yoctocell@disroot.org")
>                      (gnus-search-engine gnus-search-mairix))))
>
> When I press G G and enter a query, I get an error saying
>
>   Group nnselect:nnselect-86mtzfrr62.fsf contains no messages
>
> Am I doing something wrong? Thanks in advance!

The only other required config is the "remove-prefix" key, which tells
Gnus how to strip off the leading part of the absolute filename of
search results. Right now it might require an explicit absolute path,
not a tilde expansion, so you'd probably need:

(setq gnus-select-method '(nnnil "")
      gnus-secondary-select-methods
      '((nntp "news.gwene.org")
	(nnmaildir "yoctocell"
		   (directory "~/.local/share/mail/yoctocell@disroot.org")
                   (gnus-search-engine gnus-search-mairix
                     (remove-prefix "/<absolute>/<path>/.local/share/mail/yoctocell@disroot.org")))))

That's not very reasonable, so I'll add a call to `expand-file-name' in
a bit. But specifying an absolute file name doesn't hurt, either.

Check first with `gnus-search-use-parsed-queries' to nil, and let me
know if it still doesn't work.

Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
       [not found]                           ` <871rgqz98k.fsf@ericabrahamsen.net>
@ 2020-11-18 21:05                             ` Jose A. Ortega Ruiz
  2020-11-18 21:31                               ` Eric Abrahamsen
  0 siblings, 1 reply; 49+ messages in thread
From: Jose A. Ortega Ruiz @ 2020-11-18 21:05 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: Eric S Fraga, ding

On Wed, Nov 18 2020, Eric Abrahamsen wrote:


[...]

> No, that's perfect, I'm trying to make completely sure I understand
> what's going on here. The original nnir-run-* functions all did pretty
> much the same thing but with slightly different code, and in combining
> all those functions I didn't get everything exactly in place.
>
> The original notmuch code used the GROUPS parameter to build a "--path"
> option for the notmuch call, but even if I re-instate that, we'll still
> want this filter here for the other engines.

As an aside, I cannot find --path in notmuch search man page, but
something that would speed things up considerably for some cases would
be to append to the generated notmuch query "folder:/<regexp>/"
(mandatory /'s, to say you're looking for a regexp, not a literal).  But
that won't fix the problem, either.

> Would you try this version of the function? It should just permit any
> path separator when checking for group name matches.

It works for me (and saves me having to define my own engine, so it's
nice!).

Thanks,
jao
-- 
It is hard enough to remember my opinions, without also remembering my
reasons for them.
   -Friedrich Wilhelm Nietzsche, philosopher (1844-1900)


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-18 21:05                             ` Jose A. Ortega Ruiz
@ 2020-11-18 21:31                               ` Eric Abrahamsen
  0 siblings, 0 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-18 21:31 UTC (permalink / raw)
  To: Jose A. Ortega Ruiz; +Cc: Eric S Fraga, ding

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Wed, Nov 18 2020, Eric Abrahamsen wrote:
>
>
> [...]
>
>> No, that's perfect, I'm trying to make completely sure I understand
>> what's going on here. The original nnir-run-* functions all did pretty
>> much the same thing but with slightly different code, and in combining
>> all those functions I didn't get everything exactly in place.
>>
>> The original notmuch code used the GROUPS parameter to build a "--path"
>> option for the notmuch call, but even if I re-instate that, we'll still
>> want this filter here for the other engines.
>
> As an aside, I cannot find --path in notmuch search man page, but
> something that would speed things up considerably for some cases would
> be to append to the generated notmuch query "folder:/<regexp>/"
> (mandatory /'s, to say you're looking for a regexp, not a literal).  But
> that won't fix the problem, either.

Oops, I meant path:, not --path. Looks like there's both path: and
folder:. In theory I would like to support this, as doing filtering will
be faster in notmuch than in Gnus, but it also looks like there are a
lot of possible different configurations we'd need to accommodate.

>> Would you try this version of the function? It should just permit any
>> path separator when checking for group name matches.
>
> It works for me (and saves me having to define my own engine, so it's
> nice!).

It should be basic functionality! Thanks for testing.

Eric


^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-18 20:53   ` Eric Abrahamsen
@ 2020-11-19 11:02     ` yoctocell
  2020-11-22 12:56     ` yoctocell
  1 sibling, 0 replies; 49+ messages in thread
From: yoctocell @ 2020-11-19 11:02 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]

On 18 November 2020 12:53, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> The only other required config is the "remove-prefix" key, which tells
> Gnus how to strip off the leading part of the absolute filename of
> search results. Right now it might require an explicit absolute path,
> not a tilde expansion, so you'd probably need:
>
> (setq gnus-select-method '(nnnil "")
>       gnus-secondary-select-methods
>       '((nntp "news.gwene.org")
> 	(nnmaildir "yoctocell"
> 		   (directory "~/.local/share/mail/yoctocell@disroot.org")
>                    (gnus-search-engine gnus-search-mairix
>                      (remove-prefix "/<absolute>/<path>/.local/share/mail/yoctocell@disroot.org")))))
>
> That's not very reasonable, so I'll add a call to `expand-file-name' in
> a bit. But specifying an absolute file name doesn't hurt, either.
>
> Check first with `gnus-search-use-parsed-queries' to nil, and let me
> know if it still doesn't work.
>
> Eric

Thanks, now it works as expected.

-- 
yoctocell
PGP fingerprint: 9DDD 8877 BC0C A9CE 5D8F  9ED6 0F9A 432E 305F DAFF

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 865 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-18 20:53   ` Eric Abrahamsen
  2020-11-19 11:02     ` yoctocell
@ 2020-11-22 12:56     ` yoctocell
  2020-11-22 16:31       ` Eric Abrahamsen
  1 sibling, 1 reply; 49+ messages in thread
From: yoctocell @ 2020-11-22 12:56 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: ding

[-- Attachment #1: Type: text/plain, Size: 1428 bytes --]

On 18 November 2020 12:53, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> The only other required config is the "remove-prefix" key, which tells
> Gnus how to strip off the leading part of the absolute filename of
> search results. Right now it might require an explicit absolute path,
> not a tilde expansion, so you'd probably need:
>
> (setq gnus-select-method '(nnnil "")
>       gnus-secondary-select-methods
>       '((nntp "news.gwene.org")
> 	(nnmaildir "yoctocell"
> 		   (directory "~/.local/share/mail/yoctocell@disroot.org")
>                    (gnus-search-engine gnus-search-mairix
>                      (remove-prefix "/<absolute>/<path>/.local/share/mail/yoctocell@disroot.org")))))
>
> That's not very reasonable, so I'll add a call to `expand-file-name' in
> a bit. But specifying an absolute file name doesn't hurt, either.
>
> Check first with `gnus-search-use-parsed-queries' to nil, and let me
> know if it still doesn't work.
>
> Eric
>
>

Sorry to bother you again. When I change the search engine to
gnus-search-notmuch and press G G it doesn't seem to work and it just
echoes

Doing notmuch query on (nnmaildir+yoctocell:Inbox)...
Group nnselect:nnselect-86y2it4kpw.fsf contains no messages

I've also tried changing the `config-file' variable but it gives the same results.

-- 
yoctocell
PGP fingerprint: 9DDD 8877 BC0C A9CE 5D8F  9ED6 0F9A 432E 305F DAFF

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 865 bytes --]

^ permalink raw reply	[flat|nested] 49+ messages in thread

* Re: New "gnus-search" syntax and interface
  2020-11-22 12:56     ` yoctocell
@ 2020-11-22 16:31       ` Eric Abrahamsen
  0 siblings, 0 replies; 49+ messages in thread
From: Eric Abrahamsen @ 2020-11-22 16:31 UTC (permalink / raw)
  To: ding

yoctocell <yoctocell@disroot.org> writes:

> On 18 November 2020 12:53, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> The only other required config is the "remove-prefix" key, which tells
>> Gnus how to strip off the leading part of the absolute filename of
>> search results. Right now it might require an explicit absolute path,
>> not a tilde expansion, so you'd probably need:
>>
>> (setq gnus-select-method '(nnnil "")
>>       gnus-secondary-select-methods
>>       '((nntp "news.gwene.org")
>> 	(nnmaildir "yoctocell"
>> 		   (directory "~/.local/share/mail/yoctocell@disroot.org")
>>                    (gnus-search-engine gnus-search-mairix
>>                      (remove-prefix "/<absolute>/<path>/.local/share/mail/yoctocell@disroot.org")))))
>>
>> That's not very reasonable, so I'll add a call to `expand-file-name' in
>> a bit. But specifying an absolute file name doesn't hurt, either.
>>
>> Check first with `gnus-search-use-parsed-queries' to nil, and let me
>> know if it still doesn't work.
>>
>> Eric
>>
>>
>
> Sorry to bother you again. When I change the search engine to
> gnus-search-notmuch and press G G it doesn't seem to work and it just
> echoes
>
> Doing notmuch query on (nnmaildir+yoctocell:Inbox)...
> Group nnselect:nnselect-86y2it4kpw.fsf contains no messages
>
> I've also tried changing the `config-file' variable but it gives the same results.

Is this the same maildir as you were testing with mairix? Can you show
me your full config? Remember that all of these local indexed search
engines will require the 'remove-prefix key.



^ permalink raw reply	[flat|nested] 49+ messages in thread

end of thread, other threads:[~2020-11-22 16:33 UTC | newest]

Thread overview: 49+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-04 17:15 New "gnus-search" syntax and interface Eric Abrahamsen
2020-11-04 18:14 ` Pankaj Jangid
2020-11-04 18:45   ` Eric Abrahamsen
2020-11-04 19:32     ` Pankaj Jangid
2020-11-04 19:39     ` Pankaj Jangid
2020-11-04 19:49       ` Eric Abrahamsen
2020-11-05  2:19 ` Eric Abrahamsen
2020-11-05 11:58 ` Adam Sjøgren
2020-11-05 16:04   ` Eric Abrahamsen
2020-11-05 16:14     ` Eric Abrahamsen
2020-11-05 16:41     ` Adam Sjøgren
2020-11-05 17:18       ` Eric Abrahamsen
2020-11-05 17:34         ` Adam Sjøgren
2020-11-06  3:50           ` Eric Abrahamsen
2020-11-06  6:23 ` Jose A. Ortega Ruiz
     [not found]   ` <12MvvhYMQyipeZzkJx1ODwHAD4xQZo6qw1FSX6nvgZAyLZCPkEVFXXGxOQTuxL1zvwZC6BER4jnUFXNgIEjIZA==@protonmail.internalid>
2020-11-07  4:59   ` Eric Abrahamsen
2020-11-08  1:23     ` Jose A. Ortega Ruiz
     [not found]       ` <VEKLrJRKnbVIVztgsX0O5q0i9OwitXf-t5q2hcVN-ZDq0SRE1KS4DIpk7iNeQxIhD1_9AC4DWOdDJsJW2XCMlg==@protonmail.internalid>
2020-11-08  2:38       ` Eric Abrahamsen
2020-11-08  2:51         ` Jose A. Ortega Ruiz
2020-11-08  2:55         ` Andrew Cohen
2020-11-08  2:43     ` Jose A. Ortega Ruiz
2020-11-08  5:03       ` Eric Abrahamsen
2020-11-08  7:16         ` Jose A. Ortega Ruiz
2020-11-12 20:51           ` Eric Abrahamsen
2020-11-13  3:17             ` Jose A. Ortega Ruiz
     [not found]               ` <-NlYHnQ3eprZs5vpzJzwiWpUHjyOUwbkarR4R4m8DK_5ik1XoE8SVsxNfQWJwgUWVIfjwxU5eCbwaWIzwZUJNQ==@protonmail.internalid>
2020-11-13  6:38               ` Eric Abrahamsen
2020-11-13 19:15                 ` Jose A. Ortega Ruiz
2020-11-14  1:02                   ` Eric Abrahamsen
2020-11-13 11:07 ` Eric S Fraga
2020-11-13 12:39   ` Gijs Hillenius
2020-11-13 13:01     ` Eric S Fraga
2020-11-13 16:15       ` Eric Abrahamsen
2020-11-13 16:56         ` Eric S Fraga
2020-11-13 17:21           ` Eric Abrahamsen
2020-11-13 20:06             ` Jose A. Ortega Ruiz
2020-11-16 10:44               ` Eric S Fraga
2020-11-16 15:00               ` Eric S Fraga
2020-11-16 18:47                 ` Eric Abrahamsen
2020-11-17 11:04                   ` Eric S Fraga
2020-11-17 23:58                     ` Eric Abrahamsen
2020-11-18  0:46                       ` Jose A. Ortega Ruiz
2020-11-18 20:40                         ` Eric Abrahamsen
     [not found]                         ` <1x7NOCTHudFuCvB0kEBCGDds7KKAdbu-tZRD41ue36qG8dPBUSj7W9lq7CK5WJwL50cGQKZcom2KMkW_2VZi8Q==@protonmail.internalid>
     [not found]                           ` <871rgqz98k.fsf@ericabrahamsen.net>
2020-11-18 21:05                             ` Jose A. Ortega Ruiz
2020-11-18 21:31                               ` Eric Abrahamsen
2020-11-18  9:21 ` yoctocell
2020-11-18 20:53   ` Eric Abrahamsen
2020-11-19 11:02     ` yoctocell
2020-11-22 12:56     ` yoctocell
2020-11-22 16:31       ` Eric Abrahamsen

Gnus development mailing list

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://inbox.vuxu.org/ding

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 ding ding/ http://inbox.vuxu.org/ding \
		ding@inbox.vuxu.org
	public-inbox-index ding

Example config snippet for mirrors.
Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.archive.emacs.gnus.general


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git