Announcements and discussions for Gnus, the GNU Emacs Usenet newsreader
 help / color / mirror / Atom feed
* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
       [not found] <87lf1k11ed.fsf@onenetbeyond.org>
@ 2021-11-19 19:05 ` jao
  2021-12-14 21:41   ` dal-blazej
  0 siblings, 1 reply; 25+ messages in thread
From: jao @ 2021-11-19 19:05 UTC (permalink / raw)
  To: dal-blazej; +Cc: info-gnus-english


hi,

i am not sure what's different in your setup, because i am using the
latest notmuch and a pretty much identical config (i put leafnode as my
primary method, but that's all), and it works out of the box, including
referring to threads.

maybe it's the emacs version? mine says:
   GNU Emacs 29.0.50 of 2021-11-17

so, it's from emacs master updated a couple of days ago...

cheers,
jao
-- 
I always pass on good advice. It's the only thing to do with it. It is
never any use to oneself.
 -Oscar Wilde


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-11-19 19:05 ` gnus-search-engine set to gnus-search-notmuch and refer threads jao
@ 2021-12-14 21:41   ` dal-blazej
  2021-12-15 17:41     ` Eric Abrahamsen
  2021-12-21  5:56     ` Andrew Cohen
  0 siblings, 2 replies; 25+ messages in thread
From: dal-blazej @ 2021-12-14 21:41 UTC (permalink / raw)
  To: jao; +Cc: info-gnus-english


To be sure I upgraded both softwares :

emacs-version 29.0.50
notmuch 0.34.2+35~ged03bab


Maybe my first mail was a bit confuse ?

To reproduce the issue :

1. In the *server* buffer, use `gnus-group-read-ephemeral-search-group'
with the query "from:jao@gnu.org" 

2. The first search succeed ; *in the ephemeral search buffer*, now use
`gnus-summary-refer-thread' on an article. I get: 
(wrong-type-argument listp "")


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-14 21:41   ` dal-blazej
@ 2021-12-15 17:41     ` Eric Abrahamsen
  2021-12-18 23:22       ` dal-blazej
  2021-12-21  5:56     ` Andrew Cohen
  1 sibling, 1 reply; 25+ messages in thread
From: Eric Abrahamsen @ 2021-12-15 17:41 UTC (permalink / raw)
  To: info-gnus-english

dal-blazej@onenetbeyond.org writes:

> To be sure I upgraded both softwares :
>
> emacs-version 29.0.50
> notmuch 0.34.2+35~ged03bab
>
>
> Maybe my first mail was a bit confuse ?
>
> To reproduce the issue :
>
> 1. In the *server* buffer, use `gnus-group-read-ephemeral-search-group'
> with the query "from:jao@gnu.org" 
>
> 2. The first search succeed ; *in the ephemeral search buffer*, now use
> `gnus-summary-refer-thread' on an article. I get: 
> (wrong-type-argument listp "")

Thanks for keeping up with this. I can't immediately reproduce this.
What's your value of `gnus-refer-thread-use-search'?



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-15 17:41     ` Eric Abrahamsen
@ 2021-12-18 23:22       ` dal-blazej
  0 siblings, 0 replies; 25+ messages in thread
From: dal-blazej @ 2021-12-18 23:22 UTC (permalink / raw)
  To: info-gnus-english


Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> Thanks for keeping up with this. I can't immediately reproduce this.
> What's your value of `gnus-refer-thread-use-search'?

It is `t'.

I remember to try something also with `gnus-refer-article-method', like:

(setq gnus-refer-article-method
      `(current
        (nnselect "nnimap:Dal")
        (nntp "leafnode"
              (gnus-search-engine
               gnus-search-notmuch
               (remove-prefix ,(expand-file-name "~/Maildir/news/"))
               (config-file ,(expand-file-name "~/.notmuch-config"))))))

Without success, now it is 'current.


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-14 21:41   ` dal-blazej
  2021-12-15 17:41     ` Eric Abrahamsen
@ 2021-12-21  5:56     ` Andrew Cohen
  2021-12-22 20:56       ` Jose A. Ortega Ruiz
  1 sibling, 1 reply; 25+ messages in thread
From: Andrew Cohen @ 2021-12-21  5:56 UTC (permalink / raw)
  To: info-gnus-english

>>>>> "dal-blazej" == dal-blazej  <dal-blazej@onenetbeyond.org> writes:

[...]

    dal-blazej> To reproduce the issue :

    dal-blazej> 1. In the *server* buffer, use
    dal-blazej> `gnus-group-read-ephemeral-search-group' with the query
    dal-blazej> "from:jao@gnu.org"

    dal-blazej> 2. The first search succeed ; *in the ephemeral search
    dal-blazej> buffer*, now use `gnus-summary-refer-thread' on an
    dal-blazej> article. I get: (wrong-type-argument listp "")

I don't know if this is related but I just fixed a rather rare bug in
thread referral on master (it happens when the subject of the message on
which you initiate the thread referral is nearly empty).

You might give this a try to see if it fixes your problem. 

-- 
Andrew Cohen



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-21  5:56     ` Andrew Cohen
@ 2021-12-22 20:56       ` Jose A. Ortega Ruiz
  2021-12-22 21:16         ` Eric Abrahamsen
  0 siblings, 1 reply; 25+ messages in thread
From: Jose A. Ortega Ruiz @ 2021-12-22 20:56 UTC (permalink / raw)
  To: info-gnus-english

On Tue, Dec 21 2021, Andrew Cohen wrote:

>>>>>> "dal-blazej" == dal-blazej  <dal-blazej@onenetbeyond.org> writes:
>
> [...]
>
>     dal-blazej> To reproduce the issue :
>
>     dal-blazej> 1. In the *server* buffer, use
>     dal-blazej> `gnus-group-read-ephemeral-search-group' with the query
>     dal-blazej> "from:jao@gnu.org"
>
>     dal-blazej> 2. The first search succeed ; *in the ephemeral search
>     dal-blazej> buffer*, now use `gnus-summary-refer-thread' on an
>     dal-blazej> article. I get: (wrong-type-argument listp "")
>
> I don't know if this is related but I just fixed a rather rare bug in
> thread referral on master (it happens when the subject of the message on
> which you initiate the thread referral is nearly empty).
>
> You might give this a try to see if it fixes your problem. 

I've just been able to reproduce the problem in latest master, so i am
afraid the answer is no :) This is the beginning of the backtrace I get
when A T in a nnselect buffer (with notmuch as its engine):

Debugger entered--Lisp error: (wrong-type-argument listp "")
  alist-get(parsed-query "")
  #f(compiled-function (engine query-spec) #<bytecode 0x99ea4e27204d6f9>)(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "")
  apply(#f(compiled-function (engine query-spec) #<bytecode 0x99ea4e27204d6f9>) #<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "")
  gnus-search-make-query-string(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "")
  #f(compiled-function (engine server query groups) "Run QUERY against SERVER using ENGINE.\nThis method is common to all indexed search engines.\n\nReturns a list of [group article score] vectors." #<bytecode 0x121b0d437dc8f9e7>)(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "nnml:" "" nil)
  apply(#f(compiled-function (engine server query groups) "Run QUERY against SERVER using ENGINE.\nThis method is common to all indexed search engines.\n\nReturns a list of [group article score] vectors." #<bytecode 0x121b0d437dc8f9e7>) (#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "nnml:" "" nil))
  #f(compiled-function (&rest cnm-args) #<bytecode 0x1b59543183000dee>)(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "nnml:" "" nil)
  #f(compiled-function (cl--cnm engine server query groups) "Handle notmuch's thread-search routine." #<bytecode 0x1d4ee8fb5ee95069>)(#f(compiled-function (&rest cnm-args) #<bytecode 0x1b59543183000dee>) #<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "nnml:" ((parsed-query (or (id . "CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAzoymm...") (or (id . "t7sEZR5CwAuextCqAQyuzZ4N-BAsbVMaKTaGiG6o56GNhGyz0U...") (or (id . "Ivxz0PU9N9S5LLCmCuI5-UE8pt2AVamHqzLc3LibJxLdYCvo1X...") (or (id . "a2wBK5wGXfG6kT2lSke4Mmqc14JC2SLMqP2nsMxZAizapFFBuH...") (id . "V00BOpL5XPuHuS_tIuCyc-9zxFq7mR1gLUgK-BhIUZjJOXoFdr...")))))) (query . "id:<CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAz...") (thread . t)) nil)
  apply(#f(compiled-function (cl--cnm engine server query groups) "Handle notmuch's thread-search routine." #<bytecode 0x1d4ee8fb5ee95069>) #f(compiled-function (&rest cnm-args) #<bytecode 0x1b59543183000dee>) (#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "nnml:" ((parsed-query (or (id . "CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAzoymm...") (or (id . "t7sEZR5CwAuextCqAQyuzZ4N-BAsbVMaKTaGiG6o56GNhGyz0U...") (or (id . "Ivxz0PU9N9S5LLCmCuI5-UE8pt2AVamHqzLc3LibJxLdYCvo1X...") (or ... ...))))) (query . "id:<CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAz...") (thread . t)) nil))
  #f(compiled-function (&rest args) #<bytecode -0x17376153736bae46>)(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "nnml:" ((parsed-query (or (id . "CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAzoymm...") (or (id . "t7sEZR5CwAuextCqAQyuzZ4N-BAsbVMaKTaGiG6o56GNhGyz0U...") (or (id . "Ivxz0PU9N9S5LLCmCuI5-UE8pt2AVamHqzLc3LibJxLdYCvo1X...") (or (id . "a2wBK5wGXfG6kT2lSke4Mmqc14JC2SLMqP2nsMxZAizapFFBuH...") (id . "V00BOpL5XPuHuS_tIuCyc-9zxFq7mR1gLUgK-BhIUZjJOXoFdr...")))))) (query . "id:<CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAz...") (thread . t)) nil)
  apply(#f(compiled-function (&rest args) #<bytecode -0x17376153736bae46>) #<gnus-search-notmuch gnus-search-notmuch-1556d745508e> ("nnml:" ((parsed-query (or (id . "CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAzoymm...") (or (id . "t7sEZR5CwAuextCqAQyuzZ4N-BAsbVMaKTaGiG6o56GNhGyz0U...") (or (id . "Ivxz0PU9N9S5LLCmCuI5-UE8pt2AVamHqzLc3LibJxLdYCvo1X...") (or ... ...))))) (query . "id:<CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAz...") (thread . t)) nil))
  gnus-search-run-search(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "nnml:" ((parsed-query (or (id . "CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAzoymm...") (or (id . "t7sEZR5CwAuextCqAQyuzZ4N-BAsbVMaKTaGiG6o56GNhGyz0U...") (or (id . "Ivxz0PU9N9S5LLCmCuI5-UE8pt2AVamHqzLc3LibJxLdYCvo1X...") (or (id . "a2wBK5wGXfG6kT2lSke4Mmqc14JC2SLMqP2nsMxZAizapFFBuH...") (id . "V00BOpL5XPuHuS_tIuCyc-9zxFq7mR1gLUgK-BhIUZjJOXoFdr...")))))) (query . "id:<CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAz...") (thread . t)) nil)
  #f(compiled-function (arg0) #<bytecode -0x78d3342791d6591>)(("nnml:"))
  mapc(#f(compiled-function (arg0) #<bytecode -0x78d3342791d6591>) (("nnml:")))


So, when we invoke A T, somewhere in the twisted maze we're trying to
construct an invalid query string... maybe that rings a bell to someone?

Curiously (or not), if i A W on the article instead, that works, and an
A T there (its original group) works fine too.

Cheers,
jao
-- 
One will rarely err if extreme actions be ascribed to vanity, ordinary
actions to habit, and mean actions to fear. -Friedrich Nietzsche,
philosopher (1844-1900)



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-22 20:56       ` Jose A. Ortega Ruiz
@ 2021-12-22 21:16         ` Eric Abrahamsen
  2021-12-22 21:19           ` Eric Abrahamsen
  0 siblings, 1 reply; 25+ messages in thread
From: Eric Abrahamsen @ 2021-12-22 21:16 UTC (permalink / raw)
  To: info-gnus-english

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Tue, Dec 21 2021, Andrew Cohen wrote:
>
>>>>>>> "dal-blazej" == dal-blazej  <dal-blazej@onenetbeyond.org> writes:
>>
>> [...]
>>
>>     dal-blazej> To reproduce the issue :
>>
>>     dal-blazej> 1. In the *server* buffer, use
>>     dal-blazej> `gnus-group-read-ephemeral-search-group' with the query
>>     dal-blazej> "from:jao@gnu.org"
>>
>>     dal-blazej> 2. The first search succeed ; *in the ephemeral search
>>     dal-blazej> buffer*, now use `gnus-summary-refer-thread' on an
>>     dal-blazej> article. I get: (wrong-type-argument listp "")
>>
>> I don't know if this is related but I just fixed a rather rare bug in
>> thread referral on master (it happens when the subject of the message on
>> which you initiate the thread referral is nearly empty).
>>
>> You might give this a try to see if it fixes your problem. 
>
> I've just been able to reproduce the problem in latest master, so i am
> afraid the answer is no :) This is the beginning of the backtrace I get
> when A T in a nnselect buffer (with notmuch as its engine):
>
> Debugger entered--Lisp error: (wrong-type-argument listp "")
>   alist-get(parsed-query "")
>   #f(compiled-function (engine query-spec) #<bytecode 0x99ea4e27204d6f9>)(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "")
>   apply(#f(compiled-function (engine query-spec) #<bytecode 0x99ea4e27204d6f9>) #<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "")
>   gnus-search-make-query-string(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "")
>   #f(compiled-function (engine server query groups) "Run QUERY against
> SERVER using ENGINE.\nThis method is common to all indexed search
> engines.\n\nReturns a list of [group article score] vectors."
> #<bytecode 0x121b0d437dc8f9e7>)(#<gnus-search-notmuch
> gnus-search-notmuch-1556d745508e> "nnml:" "" nil)
>   apply(#f(compiled-function (engine server query groups) "Run QUERY
> against SERVER using ENGINE.\nThis method is common to all indexed
> search engines.\n\nReturns a list of [group article score] vectors."
> #<bytecode 0x121b0d437dc8f9e7>) (#<gnus-search-notmuch
> gnus-search-notmuch-1556d745508e> "nnml:" "" nil))
>   #f(compiled-function (&rest cnm-args) #<bytecode
> 0x1b59543183000dee>)(#<gnus-search-notmuch
> gnus-search-notmuch-1556d745508e> "nnml:" "" nil)
>   #f(compiled-function (cl--cnm engine server query groups) "Handle
> notmuch's thread-search routine." #<bytecode
> 0x1d4ee8fb5ee95069>)(#f(compiled-function (&rest cnm-args) #<bytecode
> 0x1b59543183000dee>) #<gnus-search-notmuch
> gnus-search-notmuch-1556d745508e> "nnml:" ((parsed-query (or (id .
> "CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAzoymm...") (or (id .
> "t7sEZR5CwAuextCqAQyuzZ4N-BAsbVMaKTaGiG6o56GNhGyz0U...") (or (id .
> "Ivxz0PU9N9S5LLCmCuI5-UE8pt2AVamHqzLc3LibJxLdYCvo1X...") (or (id .
> "a2wBK5wGXfG6kT2lSke4Mmqc14JC2SLMqP2nsMxZAizapFFBuH...") (id .
> "V00BOpL5XPuHuS_tIuCyc-9zxFq7mR1gLUgK-BhIUZjJOXoFdr...")))))) (query .
> "id:<CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAz...") (thread . t))
> nil)

This is a bit hard to look at, but it seems like the problem is in
notmuch's thread-specific search handling. What is supposed to happen
is that notmuch first uses those message-ids to find the messages, then
finds the thread:<thread-id> statements for all of the search results, then does
*another* search using the thread ids from the first search. So
obviously something's going wrong with that. The relevant method is
currently line 1589 in gnus-search.el, I'm also pasting it below. Would
you mind edebugging it and stepping through to see which part exactly is
wrong?

Thanks.

(cl-defmethod gnus-search-run-search :around ((engine gnus-search-notmuch)
					      server query groups)
  "Handle notmuch's thread-search routine."
  ;; Notmuch allows for searching threads, but only using its own
  ;; thread ids.  That means a thread search is a \"double-bounce\":
  ;; once to find the relevant thread ids, and again to find the
  ;; actual messages.  This method performs the first \"bounce\".
  (if (alist-get 'thread query)
      (with-slots (program proc-buffer) engine
	(let* ((qstring
		(gnus-search-make-query-string engine query))
	       (cp-list (gnus-search-indexed-search-command
			 engine qstring query groups))
	       thread-ids proc)
	  (set-buffer proc-buffer)
	  (erase-buffer)
	  (setq proc (apply #'start-process (format "search-%s" server)
			    proc-buffer program cp-list))
	  (while (process-live-p proc)
	    (accept-process-output proc))
	  (while (re-search-forward "^thread:\\([^ ]+\\)" (point-max) t)
	    (push (match-string 1) thread-ids))
	  (cl-call-next-method
	   engine server
	   ;; Completely replace the query with our new thread-based one.
	   (mapconcat (lambda (thrd) (concat "thread:" thrd))
		      thread-ids " or ")
	   nil)))
    (cl-call-next-method engine server query groups)))



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-22 21:16         ` Eric Abrahamsen
@ 2021-12-22 21:19           ` Eric Abrahamsen
  2021-12-22 23:01             ` Jose A. Ortega Ruiz
  0 siblings, 1 reply; 25+ messages in thread
From: Eric Abrahamsen @ 2021-12-22 21:19 UTC (permalink / raw)
  To: info-gnus-english

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> "Jose A. Ortega Ruiz" <jao@gnu.org> writes:
>
>> On Tue, Dec 21 2021, Andrew Cohen wrote:
>>
>>>>>>>> "dal-blazej" == dal-blazej  <dal-blazej@onenetbeyond.org> writes:
>>>
>>> [...]
>>>
>>>     dal-blazej> To reproduce the issue :
>>>
>>>     dal-blazej> 1. In the *server* buffer, use
>>>     dal-blazej> `gnus-group-read-ephemeral-search-group' with the query
>>>     dal-blazej> "from:jao@gnu.org"
>>>
>>>     dal-blazej> 2. The first search succeed ; *in the ephemeral search
>>>     dal-blazej> buffer*, now use `gnus-summary-refer-thread' on an
>>>     dal-blazej> article. I get: (wrong-type-argument listp "")
>>>
>>> I don't know if this is related but I just fixed a rather rare bug in
>>> thread referral on master (it happens when the subject of the message on
>>> which you initiate the thread referral is nearly empty).
>>>
>>> You might give this a try to see if it fixes your problem. 
>>
>> I've just been able to reproduce the problem in latest master, so i am
>> afraid the answer is no :) This is the beginning of the backtrace I get
>> when A T in a nnselect buffer (with notmuch as its engine):
>>
>> Debugger entered--Lisp error: (wrong-type-argument listp "")
>>   alist-get(parsed-query "")
>>   #f(compiled-function (engine query-spec) #<bytecode 0x99ea4e27204d6f9>)(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "")
>>   apply(#f(compiled-function (engine query-spec) #<bytecode 0x99ea4e27204d6f9>) #<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "")
>>   gnus-search-make-query-string(#<gnus-search-notmuch gnus-search-notmuch-1556d745508e> "")
>>   #f(compiled-function (engine server query groups) "Run QUERY against
>> SERVER using ENGINE.\nThis method is common to all indexed search
>> engines.\n\nReturns a list of [group article score] vectors."
>> #<bytecode 0x121b0d437dc8f9e7>)(#<gnus-search-notmuch
>> gnus-search-notmuch-1556d745508e> "nnml:" "" nil)
>>   apply(#f(compiled-function (engine server query groups) "Run QUERY
>> against SERVER using ENGINE.\nThis method is common to all indexed
>> search engines.\n\nReturns a list of [group article score] vectors."
>> #<bytecode 0x121b0d437dc8f9e7>) (#<gnus-search-notmuch
>> gnus-search-notmuch-1556d745508e> "nnml:" "" nil))
>>   #f(compiled-function (&rest cnm-args) #<bytecode
>> 0x1b59543183000dee>)(#<gnus-search-notmuch
>> gnus-search-notmuch-1556d745508e> "nnml:" "" nil)
>>   #f(compiled-function (cl--cnm engine server query groups) "Handle
>> notmuch's thread-search routine." #<bytecode
>> 0x1d4ee8fb5ee95069>)(#f(compiled-function (&rest cnm-args) #<bytecode
>> 0x1b59543183000dee>) #<gnus-search-notmuch
>> gnus-search-notmuch-1556d745508e> "nnml:" ((parsed-query (or (id .
>> "CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAzoymm...") (or (id .
>> "t7sEZR5CwAuextCqAQyuzZ4N-BAsbVMaKTaGiG6o56GNhGyz0U...") (or (id .
>> "Ivxz0PU9N9S5LLCmCuI5-UE8pt2AVamHqzLc3LibJxLdYCvo1X...") (or (id .
>> "a2wBK5wGXfG6kT2lSke4Mmqc14JC2SLMqP2nsMxZAizapFFBuH...") (id .
>> "V00BOpL5XPuHuS_tIuCyc-9zxFq7mR1gLUgK-BhIUZjJOXoFdr...")))))) (query .
>> "id:<CAAwB6VDPO=pT-XU8cPNfZRtxXC48LD-OXepiGejb-+QAz...") (thread . t))
>> nil)
>
> This is a bit hard to look at, but it seems like the problem is in
> notmuch's thread-specific search handling. What is supposed to happen
> is that notmuch first uses those message-ids to find the messages, then
> finds the thread:<thread-id> statements for all of the search results, then does
> *another* search using the thread ids from the first search. So
> obviously something's going wrong with that. The relevant method is
> currently line 1589 in gnus-search.el, I'm also pasting it below. Would
> you mind edebugging it and stepping through to see which part exactly is
> wrong?
>
> Thanks.
>
> (cl-defmethod gnus-search-run-search :around ((engine gnus-search-notmuch)
> 					      server query groups)
>   "Handle notmuch's thread-search routine."
>   ;; Notmuch allows for searching threads, but only using its own
>   ;; thread ids.  That means a thread search is a \"double-bounce\":
>   ;; once to find the relevant thread ids, and again to find the
>   ;; actual messages.  This method performs the first \"bounce\".
>   (if (alist-get 'thread query)
>       (with-slots (program proc-buffer) engine
> 	(let* ((qstring
> 		(gnus-search-make-query-string engine query))
> 	       (cp-list (gnus-search-indexed-search-command
> 			 engine qstring query groups))
> 	       thread-ids proc)
> 	  (set-buffer proc-buffer)
> 	  (erase-buffer)
> 	  (setq proc (apply #'start-process (format "search-%s" server)
> 			    proc-buffer program cp-list))
> 	  (while (process-live-p proc)
> 	    (accept-process-output proc))

My guess is that we need to go to point-min right here.

> 	  (while (re-search-forward "^thread:\\([^ ]+\\)" (point-max) t)
> 	    (push (match-string 1) thread-ids))
> 	  (cl-call-next-method
> 	   engine server
> 	   ;; Completely replace the query with our new thread-based one.
> 	   (mapconcat (lambda (thrd) (concat "thread:" thrd))
> 		      thread-ids " or ")
> 	   nil)))
>     (cl-call-next-method engine server query groups)))



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-22 21:19           ` Eric Abrahamsen
@ 2021-12-22 23:01             ` Jose A. Ortega Ruiz
  2021-12-23  0:30               ` Eric Abrahamsen
  2021-12-23 20:55               ` Eric Abrahamsen
  0 siblings, 2 replies; 25+ messages in thread
From: Jose A. Ortega Ruiz @ 2021-12-22 23:01 UTC (permalink / raw)
  To: info-gnus-english

On Wed, Dec 22 2021, Eric Abrahamsen wrote:

[...]

> My guess is that we need to go to point-min right here.
>
>> 	  (while (re-search-forward "^thread:\\([^ ]+\\)" (point-max) t)
>> 	    (push (match-string 1) thread-ids))
>> 	  (cl-call-next-method
>> 	   engine server
>> 	   ;; Completely replace the query with our new thread-based one.
>> 	   (mapconcat (lambda (thrd) (concat "thread:" thrd))
>> 		      thread-ids " or ")
>> 	   nil)))
>>     (cl-call-next-method engine server query groups)))

it's not enough.  before that, there's a problem with invoking the
search for threads, because notmuch is passed the flag --duplicate=1,
and that's not accepted for format thread:

  Error: --duplicate=N is only supported with --output=files and --output=messages.

funny thing is that that flag is added for no apparent reason in line
1633 of gnus-search.el:

 	"--duplicate=1" ; I have found this necessary, I don't know why.

i don't know why either because searches seem to work without it :).
if one eliminates that flag and adds your suggested (goto-char
(point-min)) we are not out the woods yet: the thread id is read, but
including an eol, so one needs

  (while (re-search-forward "^thread:\\([^\n ]+\\)" (point-max) t)
                                         ^^^^

and then, we still fail, because that leads, somehow to an error of the
form:

  Debugger entered--Lisp error: (wrong-type-argument listp "thread:000000000001a830")
     alist-get(parsed-query "thread:000000000001a830")
     #f(compiled-function (engine query-spec) #<bytecode 0x1fd1b560df8c2360>)(#<gnus-search-notmuch gnus-search-notmuch-157d5a02af62> "thread:000000000001a830")
     apply(#f(compiled-function (engine query-spec) #<bytecode 0x1fd1b560df8c2360>) #<gnus-search-notmuch gnus-search-notmuch-157d5a02af62> "thread:000000000001a830")
     gnus-search-make-query-string(#<gnus-search-notmuch gnus-search-notmuch-157d5a02af62> "thread:000000000001a830")

which indicates that the notmuch method for
gnus-search-make-query-string is buggy: it doesn't know how to parse
"thread:000000000001a830" queries...  and here i ran out of steam, but
the fix seems nearby :)

hth,
jao



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-22 23:01             ` Jose A. Ortega Ruiz
@ 2021-12-23  0:30               ` Eric Abrahamsen
  2021-12-23  3:34                 ` Jose A. Ortega Ruiz
  2021-12-23 20:55               ` Eric Abrahamsen
  1 sibling, 1 reply; 25+ messages in thread
From: Eric Abrahamsen @ 2021-12-23  0:30 UTC (permalink / raw)
  To: info-gnus-english

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Wed, Dec 22 2021, Eric Abrahamsen wrote:
>
> [...]
>
>> My guess is that we need to go to point-min right here.
>>
>>> 	  (while (re-search-forward "^thread:\\([^ ]+\\)" (point-max) t)
>>> 	    (push (match-string 1) thread-ids))
>>> 	  (cl-call-next-method
>>> 	   engine server
>>> 	   ;; Completely replace the query with our new thread-based one.
>>> 	   (mapconcat (lambda (thrd) (concat "thread:" thrd))
>>> 		      thread-ids " or ")
>>> 	   nil)))
>>>     (cl-call-next-method engine server query groups)))
>
> it's not enough.  before that, there's a problem with invoking the
> search for threads, because notmuch is passed the flag --duplicate=1,
> and that's not accepted for format thread:
>
>   Error: --duplicate=N is only supported with --output=files and --output=messages.
>
> funny thing is that that flag is added for no apparent reason in line
> 1633 of gnus-search.el:
>
>  	"--duplicate=1" ; I have found this necessary, I don't know why.

Hmm, I inherited this code, and didn't know why it was necessary either,
so I didn't take it out. I think notmuch can return multiple search
results if the same message exists in multiple folders, but since we're
filtering folders to begin with, maybe we don't need this.

> i don't know why either because searches seem to work without it :).
> if one eliminates that flag and adds your suggested (goto-char
> (point-min)) we are not out the woods yet: the thread id is read, but
> including an eol, so one needs
>
>   (while (re-search-forward "^thread:\\([^\n ]+\\)" (point-max) t)
>                                          ^^^^

Obviously I was hoping this would Just Work™ without me having to
actually install notmuch and test it.

> and then, we still fail, because that leads, somehow to an error of the
> form:
>
>   Debugger entered--Lisp error: (wrong-type-argument listp "thread:000000000001a830")
>      alist-get(parsed-query "thread:000000000001a830")
>      #f(compiled-function (engine query-spec) #<bytecode 0x1fd1b560df8c2360>)(#<gnus-search-notmuch gnus-search-notmuch-157d5a02af62> "thread:000000000001a830")
>      apply(#f(compiled-function (engine query-spec) #<bytecode 0x1fd1b560df8c2360>) #<gnus-search-notmuch gnus-search-notmuch-157d5a02af62> "thread:000000000001a830")
>      gnus-search-make-query-string(#<gnus-search-notmuch gnus-search-notmuch-157d5a02af62> "thread:000000000001a830")
>
> which indicates that the notmuch method for
> gnus-search-make-query-string is buggy: it doesn't know how to parse
> "thread:000000000001a830" queries...  and here i ran out of steam, but
> the fix seems nearby :)

You have done heroic work, and I hope it may yet be unnecessary for me
to install and configure notmuch. Telling notmuch what to do with
"thread:xxx" queries is not difficult, and I should have a solution for
this in the next day or so.

Thanks for the extensive debugging,
Eric



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-23  0:30               ` Eric Abrahamsen
@ 2021-12-23  3:34                 ` Jose A. Ortega Ruiz
  0 siblings, 0 replies; 25+ messages in thread
From: Jose A. Ortega Ruiz @ 2021-12-23  3:34 UTC (permalink / raw)
  To: info-gnus-english

On Wed, Dec 22 2021, Eric Abrahamsen wrote:

[...]

> You have done heroic work, and I hope it may yet be unnecessary for me
> to install and configure notmuch. Telling notmuch what to do with
> "thread:xxx" queries is not difficult, and I should have a solution for
> this in the next day or so.

i fear you'll have to install it.  the naive fixes below
(properly set a query with (query . "thread:xxxxxxxxx") rather than the
raw string, and eliminating --duplicate for thread searches) end up
performing the query in nnselect group, rather than moving to the parent
group.

cheers,
jao


(cl-defmethod gnus-search-run-search :around ((engine gnus-search-notmuch)
					      server query groups)
  "Handle notmuch's thread-search routine."
  ;; Notmuch allows for searching threads, but only using its own
  ;; thread ids.  That means a thread search is a \"double-bounce\":
  ;; once to find the relevant thread ids, and again to find the
  ;; actual messages.  This method performs the first \"bounce\".
  (if (alist-get 'thread query)
      (with-slots (program proc-buffer) engine
	(let* ((qstring
		(gnus-search-make-query-string engine query))
	       (cp-list (gnus-search-indexed-search-command
			 engine qstring query groups))
	       thread-ids proc)
	  (set-buffer proc-buffer)
	  (erase-buffer)
	  (setq proc (apply #'start-process (format "search-%s" server)
			    proc-buffer program cp-list))
	  (while (process-live-p proc)
	    (accept-process-output proc))
          (goto-char (point-min))
	  (while (re-search-forward "^thread:\\([^\n ]+\\)" (point-max) t)
	    (push (match-string 0) thread-ids))
	  (cl-call-next-method
	   engine server
	   ;; Completely replace the query with our new thread-based one.
	   `((query . ,(mapconcat 'identity thread-ids " or ")))
;;;; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	   nil)))
    (cl-call-next-method engine server query groups)))

(cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch)
						  (qstring string)
						  query &optional _groups)
  ;; Theoretically we could use the GROUPS parameter to pass a
  ;; --folder switch to notmuch, but I'm not confident of getting the
  ;; format right.
  (let ((limit (alist-get 'limit query))
	(thread (alist-get 'thread query)))
    (with-slots (switches config-file) engine
      `(,(format "--config=%s" config-file)
	"search"
	,(if thread
	     "--output=threads"
	   "--output=files")
	,@(unless thread '("--duplicate=1"))
                                        ; I have found this necessary, I don't know why.
;;;; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	,@switches
	,(if limit (format "--limit=%d" limit) "")
	,qstring
	))))




^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-22 23:01             ` Jose A. Ortega Ruiz
  2021-12-23  0:30               ` Eric Abrahamsen
@ 2021-12-23 20:55               ` Eric Abrahamsen
  2021-12-24  3:08                 ` Jose A. Ortega Ruiz
  1 sibling, 1 reply; 25+ messages in thread
From: Eric Abrahamsen @ 2021-12-23 20:55 UTC (permalink / raw)
  To: info-gnus-english

[-- Attachment #1: Type: text/plain, Size: 2757 bytes --]

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Wed, Dec 22 2021, Eric Abrahamsen wrote:
>
> [...]
>
>> My guess is that we need to go to point-min right here.
>>
>>> 	  (while (re-search-forward "^thread:\\([^ ]+\\)" (point-max) t)
>>> 	    (push (match-string 1) thread-ids))
>>> 	  (cl-call-next-method
>>> 	   engine server
>>> 	   ;; Completely replace the query with our new thread-based one.
>>> 	   (mapconcat (lambda (thrd) (concat "thread:" thrd))
>>> 		      thread-ids " or ")
>>> 	   nil)))
>>>     (cl-call-next-method engine server query groups)))
>
> it's not enough.  before that, there's a problem with invoking the
> search for threads, because notmuch is passed the flag --duplicate=1,
> and that's not accepted for format thread:
>
>   Error: --duplicate=N is only supported with --output=files and --output=messages.
>
> funny thing is that that flag is added for no apparent reason in line
> 1633 of gnus-search.el:
>
>  	"--duplicate=1" ; I have found this necessary, I don't know why.
>
> i don't know why either because searches seem to work without it :).
> if one eliminates that flag and adds your suggested (goto-char
> (point-min)) we are not out the woods yet: the thread id is read, but
> including an eol, so one needs
>
>   (while (re-search-forward "^thread:\\([^\n ]+\\)" (point-max) t)
>                                          ^^^^
>
> and then, we still fail, because that leads, somehow to an error of the
> form:
>
>   Debugger entered--Lisp error: (wrong-type-argument listp "thread:000000000001a830")
>      alist-get(parsed-query "thread:000000000001a830")
>      #f(compiled-function (engine query-spec) #<bytecode 0x1fd1b560df8c2360>)(#<gnus-search-notmuch gnus-search-notmuch-157d5a02af62> "thread:000000000001a830")
>      apply(#f(compiled-function (engine query-spec) #<bytecode 0x1fd1b560df8c2360>) #<gnus-search-notmuch gnus-search-notmuch-157d5a02af62> "thread:000000000001a830")
>      gnus-search-make-query-string(#<gnus-search-notmuch gnus-search-notmuch-157d5a02af62> "thread:000000000001a830")
>
> which indicates that the notmuch method for
> gnus-search-make-query-string is buggy: it doesn't know how to parse
> "thread:000000000001a830" queries...  and here i ran out of steam, but
> the fix seems nearby :)

Here's what I've got so far, would you try it out? I'm not sure about
notmuch not knowing how to parse a "thread:000000000001a830" query, it
looks like it handles it fine to me:

(let* ((engine (make-instance 'gnus-search-notmuch))
       (query "thread:23434223455")
       (parsed-query (gnus-search-parse-query query)))
  (gnus-search-make-query-string engine `((query ,query)
					  (parsed-query . ,parsed-query))))

=> "thread:23434223455"

Am I misunderstanding something?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: notmuchthreads.diff --]
[-- Type: text/x-patch, Size: 1532 bytes --]

diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
index d64c0cb90c..bb7fd47eb6 100644
--- a/lisp/gnus/gnus-search.el
+++ b/lisp/gnus/gnus-search.el
@@ -1606,13 +1606,17 @@ gnus-search-run-search
 			    proc-buffer program cp-list))
 	  (while (process-live-p proc)
 	    (accept-process-output proc))
-	  (while (re-search-forward "^thread:\\([^ ]+\\)" (point-max) t)
-	    (push (match-string 1) thread-ids))
+          (goto-char (point-min))
+	  (while (re-search-forward "^thread:\\([^[:space:]\n]+\\)" (point-max) t)
+	    (cl-pushnew (match-string 1) thread-ids :test #'equal))
 	  (cl-call-next-method
 	   engine server
-	   ;; Completely replace the query with our new thread-based one.
-	   (mapconcat (lambda (thrd) (concat "thread:" thrd))
-		      thread-ids " or ")
+	   ;; If we found threads, completely replace the query with
+	   ;; our new thread-based one.
+           (if thread-ids
+	       (mapconcat (lambda (thrd) (concat "thread:" thrd))
+		          thread-ids " or ")
+             query)
 	   nil)))
     (cl-call-next-method engine server query groups)))
 
@@ -1629,8 +1633,9 @@ gnus-search-indexed-search-command
 	"search"
 	,(if thread
 	     "--output=threads"
-	   "--output=files")
-	"--duplicate=1" ; I have found this necessary, I don't know why.
+           ;; No one knows why this "duplicate" flag is necessary, but
+           ;; it doesn't work with a thread search.
+	   "--output=files --duplicate=1")
 	,@switches
 	,(if limit (format "--limit=%d" limit) "")
 	,qstring

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-23 20:55               ` Eric Abrahamsen
@ 2021-12-24  3:08                 ` Jose A. Ortega Ruiz
  2021-12-27 21:54                   ` Jose A. Ortega Ruiz
  0 siblings, 1 reply; 25+ messages in thread
From: Jose A. Ortega Ruiz @ 2021-12-24  3:08 UTC (permalink / raw)
  To: info-gnus-english

On Thu, Dec 23 2021, Eric Abrahamsen wrote:


[...]

> Here's what I've got so far, would you try it out? I'm not sure about

no, that doesn't work because this call:

> +           (if thread-ids
> +	       (mapconcat (lambda (thrd) (concat "thread:" thrd))
> +		          thread-ids " or ")
> +             query)

is passing a query *string* to gnus-search-run-search's next method, but
that method (for the indexed parent class in this case) expects a query
represented as an *alist*.  but passing an alist instead, gives rise to
the error i mentioned in the previous mail: the query is run, but on the
current group (which is nnselect), and that group (a search result)
might not contain all messages in the thread (because it contains only
messages found by the original thread).  or at least that's how i'm
understanding things...



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-24  3:08                 ` Jose A. Ortega Ruiz
@ 2021-12-27 21:54                   ` Jose A. Ortega Ruiz
  2021-12-30 23:51                     ` Eric Abrahamsen
  0 siblings, 1 reply; 25+ messages in thread
From: Jose A. Ortega Ruiz @ 2021-12-27 21:54 UTC (permalink / raw)
  To: info-gnus-english

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]


Hi again,

So, summing up what we have so far (patch attached), the effect of these
changes is to make the errors go away, but i'm not sure if the behaviour
is expected: with the patch applied, A T will "work" for me in an
nnselect group, with the caveat that, when gnus-refer-thread-use-search
is t (or when calling gnus-refer-thread with C-u otherwise), the result
is limited to the email already present in the nnselect group... that
might be expected, in which case i think the patch can be applied (it's
basically what you already propose, Eric, except it constructs the query
as an alist).

Cheers,
jao
-- 
There are two hard things in computer science: cache invalidation,
naming things, and off-by-one errors.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-fixes-for-threaded-notmuch-searches.patch --]
[-- Type: text/x-diff, Size: 1241 bytes --]

From b56af8eb9c708ae9b700a061908509e16bc5bab4 Mon Sep 17 00:00:00 2001
From: Jose A Ortega Ruiz <jao@gnu.org>
Date: Mon, 27 Dec 2021 21:42:32 +0000
Subject: [PATCH] fixes for threaded notmuch searches

---
 lisp/gnus/gnus-search.el | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
index d64c0cb90c..0743ee79ff 100644
--- a/lisp/gnus/gnus-search.el
+++ b/lisp/gnus/gnus-search.el
@@ -1611,8 +1611,8 @@ gnus-search-run-search
 	  (cl-call-next-method
 	   engine server
 	   ;; Completely replace the query with our new thread-based one.
-	   (mapconcat (lambda (thrd) (concat "thread:" thrd))
-		      thread-ids " or ")
+	   `((query . ,(mapconcat (lambda (thrd) (concat "thread:" thrd))
+		                  thread-ids " or ")))
 	   nil)))
     (cl-call-next-method engine server query groups)))
 
@@ -1630,7 +1630,7 @@ gnus-search-indexed-search-command
 	,(if thread
 	     "--output=threads"
 	   "--output=files")
-	"--duplicate=1" ; I have found this necessary, I don't know why.
+        ,@(unless thread '("--duplicate=1"))  ; I have found this necessary, I don't know why.
 	,@switches
 	,(if limit (format "--limit=%d" limit) "")
 	,qstring
-- 
2.34.1


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-27 21:54                   ` Jose A. Ortega Ruiz
@ 2021-12-30 23:51                     ` Eric Abrahamsen
  2021-12-31  0:07                       ` Andrew Cohen
  2021-12-31  1:39                       ` jao
  0 siblings, 2 replies; 25+ messages in thread
From: Eric Abrahamsen @ 2021-12-30 23:51 UTC (permalink / raw)
  To: Jose A. Ortega Ruiz; +Cc: info-gnus-english

[-- Attachment #1: Type: text/plain, Size: 1660 bytes --]

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> Hi again,
>
> So, summing up what we have so far (patch attached), the effect of these
> changes is to make the errors go away, but i'm not sure if the behaviour
> is expected: with the patch applied, A T will "work" for me in an
> nnselect group, with the caveat that, when gnus-refer-thread-use-search
> is t (or when calling gnus-refer-thread with C-u otherwise), the result
> is limited to the email already present in the nnselect group... that
> might be expected, in which case i think the patch can be applied (it's
> basically what you already propose, Eric, except it constructs the query
> as an alist).

Okay, here goes the next try. A few things to note:

- I realized notmuch already has a "thread:{<sub-query>}" syntax that
  does the double search I was doing in elisp, so now we just use that instead.
- In all my testing I couldn't see that having "duplicate=1" on thread
  searches causes any problems, so I've taken it off. Can you please
  doublecheck this? If it's still mucking it up for you, I'll put it
  back in. I wish I really understood what the problem is (I think it
  has to do with notmuch potentially storing the same message in
  multiple locations, using symlinks).
- The search result filtration now won't filter on group names if the
  search is a thread search. This should resolve the issue you were
  seeing where "A T" would only search within the group you had searched
  in to begin with. I guess I think that an explicit thread search by
  the user should result in a full scan of the server. We can see if
  that surprises/annoys anyone, though.

Thanks again,
Eric


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: notmuchthreads.diff --]
[-- Type: text/x-patch, Size: 3885 bytes --]

diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el
index d64c0cb90c..4fb1fa3ec3 100644
--- a/lisp/gnus/gnus-search.el
+++ b/lisp/gnus/gnus-search.el
@@ -1352,7 +1352,10 @@ gnus-search-indexed-parse-output
 						server query &optional groups)
   (let ((prefix (or (slot-value engine 'remove-prefix)
                     ""))
-        (groups (mapcar #'gnus-group-short-name groups))
+        (groups (unless (alist-get 'thread query)
+                  ;; If we're searching threads, return messages from
+                  ;; all groups.
+                  (mapcar #'gnus-group-short-name groups)))
 	artlist article group)
     (goto-char (point-min))
     ;; Prep prefix, we want to at least be removing the root
@@ -1529,6 +1532,17 @@ gnus-search-transform
 				     (_query null))
   "*")
 
+(cl-defmethod gnus-search-make-query-string :around ((_engine gnus-search-notmuch)
+                                                     query-spec)
+  "Use Notmuch's thread:{<sub-query>} syntax.
+This packs the whole original query into <sub-query>, then
+returns all threads containing all matched messages."
+  (let ((query-string (cl-call-next-method))
+        (threadp (alist-get 'thread query-spec)))
+    (if threadp
+        (format "thread:{%s}" query-string)
+      query-string)))
+
 (cl-defmethod gnus-search-transform-expression ((engine gnus-search-notmuch)
 						(expr (head near)))
   (format "%s near %s"
@@ -1586,55 +1600,22 @@ gnus-search-transform-expression
       (format "date:%s.." (notmuch-date (cdr expr))))
      (t (ignore-errors (cl-call-next-method))))))
 
-(cl-defmethod gnus-search-run-search :around ((engine gnus-search-notmuch)
-					      server query groups)
-  "Handle notmuch's thread-search routine."
-  ;; Notmuch allows for searching threads, but only using its own
-  ;; thread ids.  That means a thread search is a \"double-bounce\":
-  ;; once to find the relevant thread ids, and again to find the
-  ;; actual messages.  This method performs the first \"bounce\".
-  (if (alist-get 'thread query)
-      (with-slots (program proc-buffer) engine
-	(let* ((qstring
-		(gnus-search-make-query-string engine query))
-	       (cp-list (gnus-search-indexed-search-command
-			 engine qstring query groups))
-	       thread-ids proc)
-	  (set-buffer proc-buffer)
-	  (erase-buffer)
-	  (setq proc (apply #'start-process (format "search-%s" server)
-			    proc-buffer program cp-list))
-	  (while (process-live-p proc)
-	    (accept-process-output proc))
-	  (while (re-search-forward "^thread:\\([^ ]+\\)" (point-max) t)
-	    (push (match-string 1) thread-ids))
-	  (cl-call-next-method
-	   engine server
-	   ;; Completely replace the query with our new thread-based one.
-	   (mapconcat (lambda (thrd) (concat "thread:" thrd))
-		      thread-ids " or ")
-	   nil)))
-    (cl-call-next-method engine server query groups)))
-
 (cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch)
 						  (qstring string)
 						  query &optional _groups)
   ;; Theoretically we could use the GROUPS parameter to pass a
   ;; --folder switch to notmuch, but I'm not confident of getting the
   ;; format right.
-  (let ((limit (alist-get 'limit query))
-	(thread (alist-get 'thread query)))
+  (let ((limit (alist-get 'limit query)))
     (with-slots (switches config-file) engine
-      `(,(format "--config=%s" config-file)
-	"search"
-	,(if thread
-	     "--output=threads"
-	   "--output=files")
-	"--duplicate=1" ; I have found this necessary, I don't know why.
-	,@switches
-	,(if limit (format "--limit=%d" limit) "")
-	,qstring
-	))))
+      (append
+       (list (format "--config=%s" config-file)
+             "search"
+             "--output=files"
+             "--duplicate=1")
+       (when limit (list (format "--limit=%d" limit)))
+       switches
+       (list qstring)))))
 
 ;;; Mairix interface
 

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-30 23:51                     ` Eric Abrahamsen
@ 2021-12-31  0:07                       ` Andrew Cohen
  2021-12-31  0:20                         ` Eric Abrahamsen
  2021-12-31  2:57                         ` Jose A. Ortega Ruiz
  2021-12-31  1:39                       ` jao
  1 sibling, 2 replies; 25+ messages in thread
From: Andrew Cohen @ 2021-12-31  0:07 UTC (permalink / raw)
  To: info-gnus-english

>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    EA> "Jose A. Ortega Ruiz" <jao@gnu.org> writes:

[...]

    EA>  - The search result filtration now won't filter on group names
    EA> if the search is a thread search. This should resolve the issue
    EA> you were seeing where "A T" would only search within the group
    EA> you had searched in to begin with. I guess I think that an
    EA> explicit thread search by the user should result in a full scan
    EA> of the server. We can see if that surprises/annoys anyone,
    EA> though.

Umm, maybe I am misunderstanding this paragraph, but there is a variable
for explicit control of this:

(defcustom gnus-refer-thread-use-search nil
  "Search an entire server when referring threads.
A nil value will only search for thread-related articles in the
current group."
  :version "28.1"
  :group 'gnus-thread
  :type 'boolean)

I usually expect a full server search but not always. Hence this
variable. I hope that the methods will respect it so that the user can
control the result. Of course if some method is incapable of doing both
kinds of searches then there would be no choice.

Now that I am sending this, perhaps the original bug resulted from the
default nil value for this variable?


-- 
Andrew Cohen



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-31  0:07                       ` Andrew Cohen
@ 2021-12-31  0:20                         ` Eric Abrahamsen
  2021-12-31  0:37                           ` Andrew Cohen
  2021-12-31  2:57                         ` Jose A. Ortega Ruiz
  1 sibling, 1 reply; 25+ messages in thread
From: Eric Abrahamsen @ 2021-12-31  0:20 UTC (permalink / raw)
  To: info-gnus-english

Andrew Cohen <acohen@ust.hk> writes:

>>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     EA> "Jose A. Ortega Ruiz" <jao@gnu.org> writes:
>
> [...]
>
>     EA>  - The search result filtration now won't filter on group names
>     EA> if the search is a thread search. This should resolve the issue
>     EA> you were seeing where "A T" would only search within the group
>     EA> you had searched in to begin with. I guess I think that an
>     EA> explicit thread search by the user should result in a full scan
>     EA> of the server. We can see if that surprises/annoys anyone,
>     EA> though.
>
> Umm, maybe I am misunderstanding this paragraph, but there is a variable
> for explicit control of this:
>
> (defcustom gnus-refer-thread-use-search nil
>   "Search an entire server when referring threads.
> A nil value will only search for thread-related articles in the
> current group."
>   :version "28.1"
>   :group 'gnus-thread
>   :type 'boolean)
>
> I usually expect a full server search but not always. Hence this
> variable. I hope that the methods will respect it so that the user can
> control the result. Of course if some method is incapable of doing both
> kinds of searches then there would be no choice.
>
> Now that I am sending this, perhaps the original bug resulted from the
> default nil value for this variable?

Ugh, that's what I get for not re-reading the whole thread every time I
sit down to work on something like this.

Yes, the problem (and the only problem) was that notmuch was
constructing its thread queries incorrectly, and I'm pretty sure that
problem is now fixed.

`gnus-refer-thread-use-search' should definitely be respected. Let me
think about this. If that variable is nil, search isn't used at all, in
which case no code in gnus-search.el matters. If it is t, then search is
used, but the "groups" parameter is set to nil, in which case the edit
in the previously-attached patch doesn't matter.

In fact the only thing the patch changes is how gnus-search behaves if
the user enters a search query including "thread:t". So it was my
message that was wrong.

And my last confusion was thinking that I wasn't seeing a conflict
between --thread and --duplicate=1, when in fact the attached patch
simply never uses --thread.

So, despite much dumbness, I think the patch is still correct.

WDYT?

Eric



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-31  0:20                         ` Eric Abrahamsen
@ 2021-12-31  0:37                           ` Andrew Cohen
  2021-12-31  1:13                             ` Eric Abrahamsen
  0 siblings, 1 reply; 25+ messages in thread
From: Andrew Cohen @ 2021-12-31  0:37 UTC (permalink / raw)
  To: info-gnus-english

>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]


    EA> `gnus-refer-thread-use-search' should definitely be
    EA> respected. Let me think about this. If that variable is nil,
    EA> search isn't used at all, in which case no code in
    EA> gnus-search.el matters. 

I don't think this is right (at least for imap): search is still used
even if in a single group. (nnselect-request-thread warps to the
original group the article came from, and then calls
gnus-summary-refer-thread, which in turn will initiate a search in this
group). 


Haven't actually looked at the detailed patch yet. 

-- 
Andrew Cohen



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-31  0:37                           ` Andrew Cohen
@ 2021-12-31  1:13                             ` Eric Abrahamsen
  0 siblings, 0 replies; 25+ messages in thread
From: Eric Abrahamsen @ 2021-12-31  1:13 UTC (permalink / raw)
  To: info-gnus-english

Andrew Cohen <acohen@ust.hk> writes:

>>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> [...]
>
>
>     EA> `gnus-refer-thread-use-search' should definitely be
>     EA> respected. Let me think about this. If that variable is nil,
>     EA> search isn't used at all, in which case no code in
>     EA> gnus-search.el matters. 
>
> I don't think this is right (at least for imap): search is still used
> even if in a single group. (nnselect-request-thread warps to the
> original group the article came from, and then calls
> gnus-summary-refer-thread, which in turn will initiate a search in this
> group). 
>
>
> Haven't actually looked at the detailed patch yet. 

The patch doesn't touch imap code, only the indexed search engines.
Which, come to think of it, will mean that behavior diverges between
different engines, which is bad.

Bah, something is still wrong here, and I need to go out. I'll do
another round of stupid patches tomorrow.

Eric



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-30 23:51                     ` Eric Abrahamsen
  2021-12-31  0:07                       ` Andrew Cohen
@ 2021-12-31  1:39                       ` jao
  2022-02-17 21:11                         ` Eric Abrahamsen
  1 sibling, 1 reply; 25+ messages in thread
From: jao @ 2021-12-31  1:39 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: info-gnus-english

On Thu, Dec 30 2021, Eric Abrahamsen wrote:


[...]

> Okay, here goes the next try. A few things to note:
>
> - I realized notmuch already has a "thread:{<sub-query>}" syntax that
>   does the double search I was doing in elisp, so now we just use that instead.

makes sense.

> - In all my testing I couldn't see that having "duplicate=1" on thread
>   searches causes any problems, so I've taken it off. Can you please
>   doublecheck this? If it's still mucking it up for you, I'll put it
>   back in. I wish I really understood what the problem is (I think it
>   has to do with notmuch potentially storing the same message in
>   multiple locations, using symlinks).

hmm, are you sure you've removed it? i can see, after applying your
diff, at least for files searches:

    (cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch)
    						  (qstring string)
    						  query &optional _groups)
      ;; Theoretically we could use the GROUPS parameter to pass a
      ;; --folder switch to notmuch, but I'm not confident of getting the
      ;; format right.
      (let ((limit (alist-get 'limit query)))
        (with-slots (switches config-file) engine
          (append
           (list (format "--config=%s" config-file)
                 "search"
                 "--output=files"
                 "--duplicate=1")
           (when limit (list (format "--limit=%d" limit)))
           switches
           (list qstring)))))

at any rate, i had already tried searches without it in my patched
version and haven't seen any adverse effects.  my understanding is that
notmuch is clever enough to detect duplicate messages with different
filenames .


> - The search result filtration now won't filter on group names if the
>   search is a thread search. This should resolve the issue you were
>   seeing where "A T" would only search within the group you had searched
>   in to begin with. I guess I think that an explicit thread search by
>   the user should result in a full scan of the server. We can see if
>   that surprises/annoys anyone, though.

the behaviour for me is the same as with my previous patch: A T stays in
the nnselect group.  a thing to notice is that, in general, there is no
single "the group you had searched in to begin width" (pretty often, i
do searches accross all my nnml groups, of which i have plenty)... a
full scan of the server is, i think, precisely what a notmuch user would
expect :) (but i don't know if this is really supposed to work for
gnus-search: in general, collecting all messages of a thread will return
messages from a list of different gnus groups: should we be able to show
all of them in an ephemeral group then?).

be it as it may, even with the full original thread belonging to a
single nnml group, A T is leaving me in nnselect with only the messages
that were already there (i.e., it's not equivalent to A W followed by A
T... but then again, maybe it's not supposed to be?)

thanks a lot!
jao
-- 
Beauty is the purgation of superfluities.
  -Michelangelo Buonarroti (1475-1564)


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-31  0:07                       ` Andrew Cohen
  2021-12-31  0:20                         ` Eric Abrahamsen
@ 2021-12-31  2:57                         ` Jose A. Ortega Ruiz
  1 sibling, 0 replies; 25+ messages in thread
From: Jose A. Ortega Ruiz @ 2021-12-31  2:57 UTC (permalink / raw)
  To: info-gnus-english

On Fri, Dec 31 2021, Andrew Cohen wrote:

[...]

> Now that I am sending this, perhaps the original bug resulted from the
> default nil value for this variable?

nope, i've got it set to t.  what might be happening is that setting it
to its non-default (or calling A T with C-u) triggers the behaviour...


jao
-- 
I have lived in this world just long enough to look carefully the
second time into things that I am most certain of the first
time. -Josh Billings, columnist and humorist (21 Apr 1818-1885)



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2021-12-31  1:39                       ` jao
@ 2022-02-17 21:11                         ` Eric Abrahamsen
  2022-02-18  0:22                           ` Andrew Cohen
  2022-02-18  7:36                           ` Eric Abrahamsen
  0 siblings, 2 replies; 25+ messages in thread
From: Eric Abrahamsen @ 2022-02-17 21:11 UTC (permalink / raw)
  To: info-gnus-english; +Cc: Andrew Cohen

jao <jao@gnu.org> writes:

> On Thu, Dec 30 2021, Eric Abrahamsen wrote:
>
>
> [...]
>
>> Okay, here goes the next try. A few things to note:
>>
>> - I realized notmuch already has a "thread:{<sub-query>}" syntax that
>>   does the double search I was doing in elisp, so now we just use that instead.
>
> makes sense.
>
>> - In all my testing I couldn't see that having "duplicate=1" on thread
>>   searches causes any problems, so I've taken it off. Can you please
>>   doublecheck this? If it's still mucking it up for you, I'll put it
>>   back in. I wish I really understood what the problem is (I think it
>>   has to do with notmuch potentially storing the same message in
>>   multiple locations, using symlinks).
>
> hmm, are you sure you've removed it? i can see, after applying your
> diff, at least for files searches:
>
>     (cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch)
>     						  (qstring string)
>     						  query &optional _groups)
>       ;; Theoretically we could use the GROUPS parameter to pass a
>       ;; --folder switch to notmuch, but I'm not confident of getting the
>       ;; format right.
>       (let ((limit (alist-get 'limit query)))
>         (with-slots (switches config-file) engine
>           (append
>            (list (format "--config=%s" config-file)
>                  "search"
>                  "--output=files"
>                  "--duplicate=1")
>            (when limit (list (format "--limit=%d" limit)))
>            switches
>            (list qstring)))))
>
> at any rate, i had already tried searches without it in my patched
> version and haven't seen any adverse effects.  my understanding is that
> notmuch is clever enough to detect duplicate messages with different
> filenames .
>
>
>> - The search result filtration now won't filter on group names if the
>>   search is a thread search. This should resolve the issue you were
>>   seeing where "A T" would only search within the group you had searched
>>   in to begin with. I guess I think that an explicit thread search by
>>   the user should result in a full scan of the server. We can see if
>>   that surprises/annoys anyone, though.
>
> the behaviour for me is the same as with my previous patch: A T stays in
> the nnselect group.  a thing to notice is that, in general, there is no
> single "the group you had searched in to begin width" (pretty often, i
> do searches accross all my nnml groups, of which i have plenty)... a
> full scan of the server is, i think, precisely what a notmuch user would
> expect :) (but i don't know if this is really supposed to work for
> gnus-search: in general, collecting all messages of a thread will return
> messages from a list of different gnus groups: should we be able to show
> all of them in an ephemeral group then?).
>
> be it as it may, even with the full original thread belonging to a
> single nnml group, A T is leaving me in nnselect with only the messages
> that were already there (i.e., it's not equivalent to A W followed by A
> T... but then again, maybe it's not supposed to be?)

I can't believe how long this is taking me...

I was confusing myself because there are two separate problems: notmuch
thread searching was simply broken, and referring a thread from within
an nnselect group only finds messages already in that select group.

I've pushed a patch that should simply get thread searching into a
working state (notmuch's "thread{}" syntax turned out to be more
complicated and less useful than I thought, so I dropped it).

Next, I think it's a reasonable design decision to say that referring a
thread from within an nnselect group should search that group's
constituent groups, not the group itself. What I'm seeing is that
nnselect actually does that (if we're using search for thread-referral,
it nixes out the group argument and searches the whole original
server(s)), but then an error is raised later on, when we try to fetch
the headers (line 656). I can see that all the thread article numbers
are returned correctly, but then when we get to `gnus-fetch-headers' I
get an args-out-of-range error, it looks like because we're looking for
the index of an article number in the original (smaller) list of
newsgroup headers. This was referring a thread from a summary buffer
that contained only a single message.

Andy, does that sound familiar?

One other bug I'm not sure what to do with: notmuch only accepts
message-ids as search terms *without* the angle brackets. If the user is
using unparsed (raw) queries, presumably they know not to include angle
brackets. If they're using parsed queries, the parsing process strips
any brackets out. But if they're using raw queries *and* refer threads,
nnselect passes in the thread search query with the angle brackets
(reasonable, since that's how `mail-header-id' and friends return them).

But that causes failure in the subsequent notmuch search. I don't want
to special-case this, but on the other hand if all other search engines
are also able to handle the no-brackets case, maybe we could just always
strip the brackets?

Eric



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2022-02-17 21:11                         ` Eric Abrahamsen
@ 2022-02-18  0:22                           ` Andrew Cohen
  2022-02-18  7:36                           ` Eric Abrahamsen
  1 sibling, 0 replies; 25+ messages in thread
From: Andrew Cohen @ 2022-02-18  0:22 UTC (permalink / raw)
  To: info-gnus-english

>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:


[...]


    EA> Next, I think it's a reasonable design decision to say that
    EA> referring a thread from within an nnselect group should search
    EA> that group's constituent groups, not the group itself. What I'm
    EA> seeing is that nnselect actually does that (if we're using
    EA> search for thread-referral, it nixes out the group argument and
    EA> searches the whole original server(s)), but then an error is
    EA> raised later on, when we try to fetch the headers (line 656). I
    EA> can see that all the thread article numbers are returned
    EA> correctly, but then when we get to `gnus-fetch-headers' I get an
    EA> args-out-of-range error, it looks like because we're looking for
    EA> the index of an article number in the original (smaller) list of
    EA> newsgroup headers. This was referring a thread from a summary
    EA> buffer that contained only a single message.

    EA> Andy, does that sound familiar?

Err, no. I use this constantly in nnselect groups and have no
errors. And I just tried it with an nnselect summary buffer with a
single message with no trouble.

Maybe its backend specific? I am using imap... I suspect it is still
some kind of notmuch problem.

I admit the code for this is a bit hairy, but it tries to do the
following:

Identify the potential groups that might hold related articles (this is
a bit more than just searching all the groups; If
=gnus-refer-thread-use-search= is true, search ALL groups on the
corresponding servers in the current nnselect group, but this isn't
relevant here).

If any hits are found they are MERGED into the nnselect group (by
extending the selection). Then the headers are retrieved with
=gnus-fetch-headers=.

Gory details to help debugging:

The search results are held in new-nnselect-artlist, so you can inspect
this to see if the search result is correct---it should be the usual vector of
vectors e.g. [["nnimap+server:group" 47 100] ["nnimap+server:group" 49
100]]. 

The mapc merges these into the current selection (its a merge rather
than append because some of the searched-for articles are already in the
selection; the virtual group numbers of these articles are collected in
=old-arts=).  As a side effect it counts the number of genuinely new
articles, numbered =first= to =last=.

Then we fetch the headers of all the articles in the thread (both the
old and the new). That is the =gnus-fetch-headers= in the next code
block.  You can check this by comparing with the
=gnus-newsgroup-selection=; this is the full vector of articles (all the
original articles in the nnselect plus all the new ones). The virtual
group article number is just the location in the vector. So you can
compare old-arts, first, and last to see that it is all correct.



-- 
Andrew Cohen



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
  2022-02-17 21:11                         ` Eric Abrahamsen
  2022-02-18  0:22                           ` Andrew Cohen
@ 2022-02-18  7:36                           ` Eric Abrahamsen
  1 sibling, 0 replies; 25+ messages in thread
From: Eric Abrahamsen @ 2022-02-18  7:36 UTC (permalink / raw)
  To: info-gnus-english

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> jao <jao@gnu.org> writes:
>
>> On Thu, Dec 30 2021, Eric Abrahamsen wrote:
>>
>>
>> [...]
>>
>>> Okay, here goes the next try. A few things to note:
>>>
>>> - I realized notmuch already has a "thread:{<sub-query>}" syntax that
>>>   does the double search I was doing in elisp, so now we just use that instead.
>>
>> makes sense.
>>
>>> - In all my testing I couldn't see that having "duplicate=1" on thread
>>>   searches causes any problems, so I've taken it off. Can you please
>>>   doublecheck this? If it's still mucking it up for you, I'll put it
>>>   back in. I wish I really understood what the problem is (I think it
>>>   has to do with notmuch potentially storing the same message in
>>>   multiple locations, using symlinks).
>>
>> hmm, are you sure you've removed it? i can see, after applying your
>> diff, at least for files searches:
>>
>>     (cl-defmethod gnus-search-indexed-search-command ((engine gnus-search-notmuch)
>>     						  (qstring string)
>>     						  query &optional _groups)
>>       ;; Theoretically we could use the GROUPS parameter to pass a
>>       ;; --folder switch to notmuch, but I'm not confident of getting the
>>       ;; format right.
>>       (let ((limit (alist-get 'limit query)))
>>         (with-slots (switches config-file) engine
>>           (append
>>            (list (format "--config=%s" config-file)
>>                  "search"
>>                  "--output=files"
>>                  "--duplicate=1")
>>            (when limit (list (format "--limit=%d" limit)))
>>            switches
>>            (list qstring)))))
>>
>> at any rate, i had already tried searches without it in my patched
>> version and haven't seen any adverse effects.  my understanding is that
>> notmuch is clever enough to detect duplicate messages with different
>> filenames .
>>
>>
>>> - The search result filtration now won't filter on group names if the
>>>   search is a thread search. This should resolve the issue you were
>>>   seeing where "A T" would only search within the group you had searched
>>>   in to begin with. I guess I think that an explicit thread search by
>>>   the user should result in a full scan of the server. We can see if
>>>   that surprises/annoys anyone, though.
>>
>> the behaviour for me is the same as with my previous patch: A T stays in
>> the nnselect group.  a thing to notice is that, in general, there is no
>> single "the group you had searched in to begin width" (pretty often, i
>> do searches accross all my nnml groups, of which i have plenty)... a
>> full scan of the server is, i think, precisely what a notmuch user would
>> expect :) (but i don't know if this is really supposed to work for
>> gnus-search: in general, collecting all messages of a thread will return
>> messages from a list of different gnus groups: should we be able to show
>> all of them in an ephemeral group then?).
>>
>> be it as it may, even with the full original thread belonging to a
>> single nnml group, A T is leaving me in nnselect with only the messages
>> that were already there (i.e., it's not equivalent to A W followed by A
>> T... but then again, maybe it's not supposed to be?)
>
> I can't believe how long this is taking me...
>
> I was confusing myself because there are two separate problems: notmuch
> thread searching was simply broken, and referring a thread from within
> an nnselect group only finds messages already in that select group.
>
> I've pushed a patch that should simply get thread searching into a
> working state (notmuch's "thread{}" syntax turned out to be more
> complicated and less useful than I thought, so I dropped it).
>
> Next, I think it's a reasonable design decision to say that referring a
> thread from within an nnselect group should search that group's
> constituent groups, not the group itself. What I'm seeing is that
> nnselect actually does that (if we're using search for thread-referral,
> it nixes out the group argument and searches the whole original
> server(s)), but then an error is raised later on, when we try to fetch
> the headers (line 656). I can see that all the thread article numbers
> are returned correctly, but then when we get to `gnus-fetch-headers' I
> get an args-out-of-range error, it looks like because we're looking for
> the index of an article number in the original (smaller) list of
> newsgroup headers. This was referring a thread from a summary buffer
> that contained only a single message.
>
> Andy, does that sound familiar?
>
> One other bug I'm not sure what to do with: notmuch only accepts
> message-ids as search terms *without* the angle brackets. If the user is
> using unparsed (raw) queries, presumably they know not to include angle
> brackets. If they're using parsed queries, the parsing process strips
> any brackets out. But if they're using raw queries *and* refer threads,
> nnselect passes in the thread search query with the angle brackets
> (reasonable, since that's how `mail-header-id' and friends return them).
>
> But that causes failure in the subsequent notmuch search. I don't want
> to special-case this, but on the other hand if all other search engines
> are also able to handle the no-brackets case, maybe we could just always
> strip the brackets?

Okay, after some hairy off-list debugging, I believe everything should
be sorted and working okay. Jose, next time you update Emacs, would you
give it a whirl?

Eric



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: gnus-search-engine set to gnus-search-notmuch and refer threads
@ 2022-02-18  1:20 Eric Abrahamsen
  0 siblings, 0 replies; 25+ messages in thread
From: Eric Abrahamsen @ 2022-02-18  1:20 UTC (permalink / raw)
  To: info-gnus-english

Andrew Cohen <acohen@ust.hk> writes:

>>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>
> [...]
>
>
>     EA> Next, I think it's a reasonable design decision to say that
>     EA> referring a thread from within an nnselect group should search
>     EA> that group's constituent groups, not the group itself. What I'm
>     EA> seeing is that nnselect actually does that (if we're using
>     EA> search for thread-referral, it nixes out the group argument and
>     EA> searches the whole original server(s)), but then an error is
>     EA> raised later on, when we try to fetch the headers (line 656). I
>     EA> can see that all the thread article numbers are returned
>     EA> correctly, but then when we get to `gnus-fetch-headers' I get an
>     EA> args-out-of-range error, it looks like because we're looking for
>     EA> the index of an article number in the original (smaller) list of
>     EA> newsgroup headers. This was referring a thread from a summary
>     EA> buffer that contained only a single message.
>
>     EA> Andy, does that sound familiar?
>
> Err, no. I use this constantly in nnselect groups and have no
> errors. And I just tried it with an nnselect summary buffer with a
> single message with no trouble.

Thanks for the swift and thorough response!

> Maybe its backend specific? I am using imap... I suspect it is still
> some kind of notmuch problem.

I really, really hope not, as the whole point of all this was complete
code isolation. I don't see how it could be, as new-nnselect-artlist was
populated correctly, and after that notmuch doesn't seem to be involved.

> I admit the code for this is a bit hairy, but it tries to do the
> following:
>
> Identify the potential groups that might hold related articles (this is
> a bit more than just searching all the groups; If
> =gnus-refer-thread-use-search= is true, search ALL groups on the
> corresponding servers in the current nnselect group, but this isn't
> relevant here).
>
> If any hits are found they are MERGED into the nnselect group (by
> extending the selection). Then the headers are retrieved with
> =gnus-fetch-headers=.
>
> Gory details to help debugging:
>
> The search results are held in new-nnselect-artlist, so you can inspect
> this to see if the search result is correct---it should be the usual vector of
> vectors e.g. [["nnimap+server:group" 47 100] ["nnimap+server:group" 49
> 100]]. 
>
> The mapc merges these into the current selection (its a merge rather
> than append because some of the searched-for articles are already in the
> selection; the virtual group numbers of these articles are collected in
> =old-arts=).  As a side effect it counts the number of genuinely new
> articles, numbered =first= to =last=.
>
> Then we fetch the headers of all the articles in the thread (both the
> old and the new). That is the =gnus-fetch-headers= in the next code
> block.  You can check this by comparing with the
> =gnus-newsgroup-selection=; this is the full vector of articles (all the
> original articles in the nnselect plus all the new ones). The virtual
> group article number is just the location in the vector. So you can
> compare old-arts, first, and last to see that it is all correct.

In my edebugging test, "new-nnselect-artlist" held the correct
(newly-found) article vectors. Then when the code got here:

(append (sort old-arts #'<)
        (number-sequence first last))

old-arts held all the newly-found article numbers. So the mapc seemed to
add all articles in "new-nnselect-artlist".

Then "first" was 2 and "last" was 1, so `number-sequence' returned nil
-- that's expected, yes?

Then I stepped forward to the `gnus-fetch-headers' call, and got args
out of range, the sequence being a vector holding the single vector of
the _original_ message that was in the nnselect group, and the index
being 14, the highest of the newly-found article numbers.

Setting up testing takes a little bit, I'm going to do some automation
for gnus-mock, and then jump back into this.

Thanks again,
Eric



^ permalink raw reply	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2022-02-18  7:36 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <87lf1k11ed.fsf@onenetbeyond.org>
2021-11-19 19:05 ` gnus-search-engine set to gnus-search-notmuch and refer threads jao
2021-12-14 21:41   ` dal-blazej
2021-12-15 17:41     ` Eric Abrahamsen
2021-12-18 23:22       ` dal-blazej
2021-12-21  5:56     ` Andrew Cohen
2021-12-22 20:56       ` Jose A. Ortega Ruiz
2021-12-22 21:16         ` Eric Abrahamsen
2021-12-22 21:19           ` Eric Abrahamsen
2021-12-22 23:01             ` Jose A. Ortega Ruiz
2021-12-23  0:30               ` Eric Abrahamsen
2021-12-23  3:34                 ` Jose A. Ortega Ruiz
2021-12-23 20:55               ` Eric Abrahamsen
2021-12-24  3:08                 ` Jose A. Ortega Ruiz
2021-12-27 21:54                   ` Jose A. Ortega Ruiz
2021-12-30 23:51                     ` Eric Abrahamsen
2021-12-31  0:07                       ` Andrew Cohen
2021-12-31  0:20                         ` Eric Abrahamsen
2021-12-31  0:37                           ` Andrew Cohen
2021-12-31  1:13                             ` Eric Abrahamsen
2021-12-31  2:57                         ` Jose A. Ortega Ruiz
2021-12-31  1:39                       ` jao
2022-02-17 21:11                         ` Eric Abrahamsen
2022-02-18  0:22                           ` Andrew Cohen
2022-02-18  7:36                           ` Eric Abrahamsen
2022-02-18  1:20 Eric Abrahamsen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).