Re: New "gnus-search" syntax and interface

[Eric Abrahamsen <eric@ericabrahamsen.net>]

"Jose A. Ortega Ruiz" <jao@gnu.org> writes:

> On Tue, Nov 17 2020, Eric Abrahamsen wrote:
>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> On Monday, 16 Nov 2020 at 10:47, Eric Abrahamsen wrote:
>>>> Well that's pretty weird. I didn't think I changed any of the actual
>>>> parsing behavior, only reorganized code. 
>>>
>>> With these complex systems, the whole concept of emergent behaviour
>>> raises its head! ;-)
>>>
>>> Happy to test out anything you change etc., of course.
>>
>> Thanks very much. This is still baffling me. If you (and/or Jose, or
>> anyone else) have a moment, would you please send me the command-line
>> output of a successful notmuch search (ie absolute filenames) which
>> fails in Emacs master right now? Feel free to anonymize however
>> necessary, though obviously leaving the group-name part of the filepath
>> alone. Plus whatever notmuch-related config you've got...
>
> If i perform a gnus-search for "baffled" in this group, which is called
> by gnus "nntp:gmane.emacs.gnus.general", the buffer where
> gnus-search-indexed-parse-output runs has the following output from
> notmuch (which correctly inserts real path names):
>
>    /home/jao/var/mail/gmane/emacs/gnus/general/65
>    /home/jao/var/mail/feeds/news/cur/1605280382.M861830P724182.osgiliath,S=3292,W=3331:2,Sa
>    /home/jao/var/mail/gwene/org/arxiv/computer/science/1102
>
> where /home/jao/var/mail is my remove-prefix.  
>
> That method (gnus-search.el:1364) is receiving as its 'groups' argument
> the value '("gmane.emacs.gnus.general").  the method constructs a regexp
> that it calls group-regexp using that value, and that regexp is wrong.
>
> That is because it uses as its value:
>
>     (when groups
>       (regexp-opt (mapcar (lambda (x) (gnus-group-real-name x)) groups)))
>
> which, for grous being '("gmane.emacs.gnus.general"), evaluates to:
>
>     "\\(?:gmane\\.emacs\\.gnus\\.general\\)"
>
> That is wrong because the method then tries to match in the results
> buffer, using that regexp, things like:
>
>    /home/jao/var/mail/gmane.emacs.gnus.general/65
>
> instead of correct pathnames such as:
>
>    /home/jao/var/mail/gmane/emacs/gnus/general/65
>
> The end result is that, despite notmuch finding correct paths and
> inserting them correctly in the temporary results buffer,
> gnus-search-indexed-parse-output fails to recognise them when it is
> filtering those belonging to the searched for groups.  And this is
> because it's constructing a wrong regular expression in
> gnus-search.el:1367, and using it in gnus-search.el:1381.
>
> The workaround my advice implements is changing the value of the input
> parameter groups from '("foo.bar") to '("foo/bar").
>
> I'm most probably belabouring here many things obvious to you, sorry
> about that :)

No, that's perfect, I'm trying to make completely sure I understand
what's going on here. The original nnir-run-* functions all did pretty
much the same thing but with slightly different code, and in combining
all those functions I didn't get everything exactly in place.

The original notmuch code used the GROUPS parameter to build a "--path"
option for the notmuch call, but even if I re-instate that, we'll still
want this filter here for the other engines.

Would you try this version of the function? It should just permit any
path separator when checking for group name matches.

Thanks,
Eric


(cl-defmethod gnus-search-indexed-parse-output ((engine gnus-search-indexed)
server query &optional groups)
(let ((prefix (slot-value engine 'remove-prefix))
 (group-regexp (when groups
			(mapconcat
			 (lambda (x)
			   (replace-regexp-in-string
			    ;; Accept any of [.\/] as path separators.
			    "[.\\/]" "[.\\\\/]"
			    (gnus-group-real-name x)))
			 groups "\\|")))
	artlist vectors article group)
    (goto-char (point-min))
    (while (not (eobp))
      (pcase-let ((`(,f-name ,score) (gnus-search-indexed-extract engine)))
	(when (and (file-readable-p f-name)
		   (null (file-directory-p f-name))
		   (or (null groups)
		       (and (gnus-search-single-p query)
			    (alist-get 'thread query))
		       (string-match-p group-regexp f-name)))
	  (push (list f-name score) artlist))))
    ;; Are we running an additional grep query?
    (when-let ((grep-reg (alist-get 'grep query)))
      (setq artlist (gnus-search-grep-search engine artlist grep-reg)))
    ;; Turn (file-name score) into [group article score].
    (pcase-dolist (`(,f-name ,score) artlist)
      (setq article (file-name-nondirectory f-name))
      ;; Remove prefix.
      (when (and prefix
		 (file-name-absolute-p prefix)
		 (string-match (concat "^"
				       (file-name-as-directory prefix))
			       f-name))
	(setq group (replace-match "" t t (file-name-directory f-name))))
      ;; Break the directory name down until it's something that
      ;; (probably) can be used as a group name.
      (setq group
	    (replace-regexp-in-string
	     "[/\\]" "."
	     (replace-regexp-in-string
	      "/?\\(cur\\|new\\|tmp\\)?/\\'" ""
	      (replace-regexp-in-string
	       "^[./\\]" ""
	       group nil t)
	      nil t)
	     nil t))

      (push (vector (gnus-group-full-name group server)
		    (if (string-match-p "\\`[[:digit:]]+\\'" article)
			(string-to-number article)
		      (nnmaildir-base-name-to-article-number
		       (substring article 0 (string-match ":" article))
		       group nil))
		    (if (numberp score)
			score
		      (string-to-number score)))
	    vectors))
    vectors))