[-- Attachment #1: Type: text/plain, Size: 320 bytes --] Hi all, I've reworked the way gnus-search parses the output from search utilities like notmuch, which should eliminate some of the bugs and funny behavior that's been reported here. I'd be much happier if I had some more eyes on it -- if you're using notmuch or namazu, please help me give this a whirl! Thanks, Eric [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-Rework-gnus-search-indexed-parse-output.patch --] [-- Type: text/x-patch, Size: 5012 bytes --] From 7f9e962b890bfa0924ca17a7abd8cc83e1b8d2bf Mon Sep 17 00:00:00 2001 From: Eric Abrahamsen <eric@ericabrahamsen.net> Date: Sat, 26 Jun 2021 10:16:19 -0700 Subject: [PATCH 1/1] Rework gnus-search-indexed-parse-output * lisp/gnus/gnus-search.el (gnus-search-indexed-parse-output): Be more careful about matching filesystem paths to Gnus group names; make absolutely sure that we only return valid article numbers. --- lisp/gnus/gnus-search.el | 94 ++++++++++++++++++---------------------- 1 file changed, 42 insertions(+), 52 deletions(-) diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el index 70bde264c1..add34e7e02 100644 --- a/lisp/gnus/gnus-search.el +++ b/lisp/gnus/gnus-search.el @@ -1351,68 +1351,58 @@ gnus-search-run-search (cl-defmethod gnus-search-indexed-parse-output ((engine gnus-search-indexed) server query &optional groups) - (let ((prefix (slot-value engine 'remove-prefix)) - (group-regexp (when groups - (mapconcat - (lambda (group-name) - (mapconcat #'regexp-quote - (split-string - (gnus-group-real-name group-name) - "[.\\/]") - "[.\\\\/]")) - groups - "\\|"))) - artlist vectors article group) + (let ((prefix (or (slot-value engine 'remove-prefix) + "")) + artlist article group) (goto-char (point-min)) + ;; Prep prefix, we want to at least be removing the root + ;; filesystem separator. + (when (stringp prefix) + (setq prefix (file-name-as-directory + (expand-file-name prefix "/")))) (while (not (or (eobp) (looking-at-p "\\(?:[[:space:]\n]+\\)?Process .+ finished"))) (pcase-let ((`(,f-name ,score) (gnus-search-indexed-extract engine))) (when (and f-name (file-readable-p f-name) - (null (file-directory-p f-name)) - (or (null groups) - (and (gnus-search-single-p query) - (alist-get 'thread query)) - (string-match-p group-regexp f-name))) - (push (list f-name score) artlist)))) + (null (file-directory-p f-name))) + (setq group + (replace-regexp-in-string + "[/\\]" "." + (replace-regexp-in-string + "/?\\(cur\\|new\\|tmp\\)?/\\'" "" + (replace-regexp-in-string + "\\`\\." "" + (string-remove-prefix + prefix (file-name-directory f-name)) + nil t) + nil t) + nil t)) + (setq article (file-name-nondirectory f-name) + article + ;; TODO: Provide a cleaner way of producing final + ;; article numbers for the various backends. + (if (string-match-p "\\`[[:digit:]]+\\'" article) + (string-to-number article) + (nnmaildir-base-name-to-article-number + (substring article 0 (string-match ":" article)) + group (string-remove-prefix "nnmaildir:" server)))) + (when (and (numberp article) + (or (null groups) + (member group groups))) + (push (list f-name article group score) + artlist))))) ;; Are we running an additional grep query? (when-let ((grep-reg (alist-get 'grep query))) (setq artlist (gnus-search-grep-search engine artlist grep-reg))) - ;; Prep prefix. - (when (and prefix (null (string-empty-p prefix))) - (setq prefix (file-name-as-directory (expand-file-name prefix)))) - ;; Turn (file-name score) into [group article score]. - (pcase-dolist (`(,f-name ,score) artlist) - (setq article (file-name-nondirectory f-name) - group (file-name-directory f-name)) - ;; Remove prefix. - (when prefix - (setq group (string-remove-prefix prefix group))) - ;; Break the directory name down until it's something that - ;; (probably) can be used as a group name. - (setq group - (replace-regexp-in-string - "[/\\]" "." - (replace-regexp-in-string - "/?\\(cur\\|new\\|tmp\\)?/\\'" "" - (replace-regexp-in-string - "^[./\\]" "" - group nil t) - nil t) - nil t)) - - (push (vector (gnus-group-full-name group server) - (if (string-match-p "\\`[[:digit:]]+\\'" article) - (string-to-number article) - (nnmaildir-base-name-to-article-number - (substring article 0 (string-match ":" article)) - group (string-remove-prefix "nnmaildir:" server))) - (if (numberp score) - score - (string-to-number score))) - vectors)) - vectors)) + ;; Munge into the list of vectors expected by nnselect. + (mapcar (pcase-lambda (`(,_ ,article ,group ,score)) + (vector group article + (if (numberp score) + score + (string-to-number score)))) + artlist))) (cl-defmethod gnus-search-indexed-extract ((_engine gnus-search-indexed)) "Base implementation treats the whole line as a filename, and -- 2.32.0
Eric writes:
> if you're using notmuch or namazu, please help me give this a whirl!
Which version of Emacs is your patch against? I just tried applying it
to master@2a33fc8d19 and got:
checking file lisp/gnus/gnus-search.el
Hunk #1 FAILED at 1351.
1 out of 1 hunk FAILED
Best regards,
Adam
--
"Synge? Hvorfor skal vi altid synge, Adam Sjøgren
bare fordi det er en tegnefilm?" asjo@koldfront.dk
Adam Sjøgren <asjo@koldfront.dk> writes:
> Eric writes:
>
>> if you're using notmuch or namazu, please help me give this a whirl!
>
> Which version of Emacs is your patch against? I just tried applying it
> to master@2a33fc8d19 and got:
It applies to up-to-date master -- there's one other change that went
into this function since your current revision. The changes are confined
to the one function, so you could also just eval this definition:
(cl-defmethod gnus-search-indexed-parse-output ((engine gnus-search-indexed)
server query &optional groups)
(let ((prefix (or (slot-value engine 'remove-prefix)
""))
artlist article group)
(goto-char (point-min))
;; Prep prefix, we want to at least be removing the root
;; filesystem separator.
(when (stringp prefix)
(setq prefix (file-name-as-directory
(expand-file-name prefix "/"))))
(while (not (or (eobp)
(looking-at-p
"\\(?:[[:space:]\n]+\\)?Process .+ finished")))
(pcase-let ((`(,f-name ,score) (gnus-search-indexed-extract engine)))
(when (and f-name
(file-readable-p f-name)
(null (file-directory-p f-name)))
(setq group
(replace-regexp-in-string
"[/\\]" "."
(replace-regexp-in-string
"/?\\(cur\\|new\\|tmp\\)?/\\'" ""
(replace-regexp-in-string
"\\`\\." ""
(string-remove-prefix
prefix (file-name-directory f-name))
nil t)
nil t)
nil t))
(setq article (file-name-nondirectory f-name)
article
;; TODO: Provide a cleaner way of producing final
;; article numbers for the various backends.
(if (string-match-p "\\`[[:digit:]]+\\'" article)
(string-to-number article)
(nnmaildir-base-name-to-article-number
(substring article 0 (string-match ":" article))
group (string-remove-prefix "nnmaildir:" server))))
(when (and (numberp article)
(or (null groups)
(member group groups)))
(push (list f-name article group score)
artlist)))))
;; Are we running an additional grep query?
(when-let ((grep-reg (alist-get 'grep query)))
(setq artlist (gnus-search-grep-search engine artlist grep-reg)))
;; Munge into the list of vectors expected by nnselect.
(mapcar (pcase-lambda (`(,_ ,article ,group ,score))
(vector group article
(if (numberp score)
score
(string-to-number score))))
artlist)))
I see 2 issues:
1. you may need
(setq group (gnus-group-full-name group server))
before
(setq article (file-name-nondirectory f-name) ...
2. with the patch
(member group groups)
is nil when I use search-group-spec to '("nnml:" . ("nnml:")) to match
all nnml groups.
cheers
Eric Abrahamsen <eric@ericabrahamsen.net> writes:
> Adam Sjøgren <asjo@koldfront.dk> writes:
>
>> Eric writes:
>>
>>> if you're using notmuch or namazu, please help me give this a whirl!
>>
>> Which version of Emacs is your patch against? I just tried applying it
>> to master@2a33fc8d19 and got:
>
> It applies to up-to-date master -- there's one other change that went
> into this function since your current revision. The changes are confined
> to the one function, so you could also just eval this definition:
>
> (cl-defmethod gnus-search-indexed-parse-output ((engine gnus-search-indexed)
> server query &optional groups)
> (let ((prefix (or (slot-value engine 'remove-prefix)
> ""))
> artlist article group)
> (goto-char (point-min))
> ;; Prep prefix, we want to at least be removing the root
> ;; filesystem separator.
> (when (stringp prefix)
> (setq prefix (file-name-as-directory
> (expand-file-name prefix "/"))))
> (while (not (or (eobp)
> (looking-at-p
> "\\(?:[[:space:]\n]+\\)?Process .+ finished")))
> (pcase-let ((`(,f-name ,score) (gnus-search-indexed-extract engine)))
> (when (and f-name
> (file-readable-p f-name)
> (null (file-directory-p f-name)))
> (setq group
> (replace-regexp-in-string
> "[/\\]" "."
> (replace-regexp-in-string
> "/?\\(cur\\|new\\|tmp\\)?/\\'" ""
> (replace-regexp-in-string
> "\\`\\." ""
> (string-remove-prefix
> prefix (file-name-directory f-name))
> nil t)
> nil t)
> nil t))
> (setq article (file-name-nondirectory f-name)
> article
> ;; TODO: Provide a cleaner way of producing final
> ;; article numbers for the various backends.
> (if (string-match-p "\\`[[:digit:]]+\\'" article)
> (string-to-number article)
> (nnmaildir-base-name-to-article-number
> (substring article 0 (string-match ":" article))
> group (string-remove-prefix "nnmaildir:" server))))
> (when (and (numberp article)
> (or (null groups)
> (member group groups)))
> (push (list f-name article group score)
> artlist)))))
> ;; Are we running an additional grep query?
> (when-let ((grep-reg (alist-get 'grep query)))
> (setq artlist (gnus-search-grep-search engine artlist grep-reg)))
> ;; Munge into the list of vectors expected by nnselect.
> (mapcar (pcase-lambda (`(,_ ,article ,group ,score))
> (vector group article
> (if (numberp score)
> score
> (string-to-number score))))
> artlist)))
>
>
>
--
7LL-1 Make peace with your past so it doesn’t spoil your present.
Eric writes:
> Adam Sjøgren <asjo@koldfront.dk> writes:
>
>> Eric writes:
>>
>>> if you're using notmuch or namazu, please help me give this a whirl!
>>
>> Which version of Emacs is your patch against? I just tried applying it
>> to master@2a33fc8d19 and got:
>
> It applies to up-to-date master
Ok, I did a git pull just before trying - well, no ~25 minutes before
trying the patch, as I built in between. D'oh!
I will give it another go, thanks!
Best regards,
Adam
--
"Here comes my darling, saying hello you Adam Sjøgren
Why you look so worried, whats a-wrong with you?" asjo@koldfront.dk
Adam writes:
> I will give it another go, thanks!
It applied cleanly to master@6b4043833c. I am using notmuch with nnml,
by having (gnus-search-engine gnus-search-notmuch) on my nnml-method.
If I put point at nnml:normal in the *Group* buffer and press GG and
type "Honda" as the search term, with the patch, I get:
Doing notmuch query on (nnml:normal)...
Group nnselect:nnselect-87lf6tq9s1.fsf contains no messages
Without the patch, I get a group with search results:
O. [ 859: Honda Danmark ] [...]
O. [1028: Honda Danmark ]
O [ 819: Honda Danmark ]
O [1048: Honda Danmark ]
O [1002: Honda Danmark ]
O [1001: Honda Danmark ]
O [1006: Honda Danmark ]
O [ 855: Honda Danmark ]
O [1113: Honda Danmark ]
Doing the search with notmuch on the commandline:
$ notmuch search Honda | cut -d' ' -f1-5
thread:000000000000f081 2020-06-09 [1/1]
thread:000000000000ed7b 2020-06-03 [1/1]
thread:000000000000ee56 2020-05-26 [1/1]
thread:000000000000efe8 2020-05-18 [1/1]
thread:000000000000ed4e 2020-03-12 [1/1]
thread:000000000000ed4d 2020-03-12 [1/1]
thread:000000000000ef24 2020-01-14 [1/1]
thread:00000000000123a0 2019-09-05 [1/1]
thread:000000000001239f 2019-09-04 [1/1]
thread:000000000000c7cf 2018-10-22 [1/1]
thread:0000000000014de8 2018-03-04 [1/1]
thread:0000000000004e0d 2014-05-16 [1/1]
thread:0000000000009b6a 2000-04-12 [1/1(2)]
(I cut the output to remove information I don't want to share).
Best regards,
Adam
--
"the office is quiet now, i am at the desk, Adam Sjøgren
preparing to landscape for another day." asjo@koldfront.dk
[-- Attachment #1: Type: text/plain, Size: 1924 bytes --] the attached patch applied to the patched file fixes this for me. Adam Sjøgren <asjo@koldfront.dk> writes: > Adam writes: > >> I will give it another go, thanks! > > It applied cleanly to master@6b4043833c. I am using notmuch with nnml, > by having (gnus-search-engine gnus-search-notmuch) on my nnml-method. > > If I put point at nnml:normal in the *Group* buffer and press GG and > type "Honda" as the search term, with the patch, I get: > > Doing notmuch query on (nnml:normal)... > Group nnselect:nnselect-87lf6tq9s1.fsf contains no messages > > Without the patch, I get a group with search results: > > O. [ 859: Honda Danmark ] [...] > O. [1028: Honda Danmark ] > O [ 819: Honda Danmark ] > O [1048: Honda Danmark ] > O [1002: Honda Danmark ] > O [1001: Honda Danmark ] > O [1006: Honda Danmark ] > O [ 855: Honda Danmark ] > O [1113: Honda Danmark ] > > Doing the search with notmuch on the commandline: > > $ notmuch search Honda | cut -d' ' -f1-5 > thread:000000000000f081 2020-06-09 [1/1] > thread:000000000000ed7b 2020-06-03 [1/1] > thread:000000000000ee56 2020-05-26 [1/1] > thread:000000000000efe8 2020-05-18 [1/1] > thread:000000000000ed4e 2020-03-12 [1/1] > thread:000000000000ed4d 2020-03-12 [1/1] > thread:000000000000ef24 2020-01-14 [1/1] > thread:00000000000123a0 2019-09-05 [1/1] > thread:000000000001239f 2019-09-04 [1/1] > thread:000000000000c7cf 2018-10-22 [1/1] > thread:0000000000014de8 2018-03-04 [1/1] > thread:0000000000004e0d 2014-05-16 [1/1] > thread:0000000000009b6a 2000-04-12 [1/1(2)] > > (I cut the output to remove information I don't want to share). > > > Best regards, > > Adam -- 7LL-7 Smile, you don’t own all the problems in the world. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0002-Rework-gnus-search-indexed-parse-output.patch --] [-- Type: text/x-diff; name="0002-Rework-gnus-search-indexed-parse-output.patch", Size: 403 bytes --] --- gnus-search-old.el 2021-06-29 13:00:20.478849987 +0200 +++ gnus-search.el 2021-06-29 13:00:28.226895659 +0200 @@ -1373,6 +1373,7 @@ nil t) nil t) nil t)) + (setq group (gnus-group-full-name group server)) (setq article (file-name-nondirectory f-name) article ;; TODO: Provide a cleaner way of producing final
Thomas writes:
> the attached patch applied to the patched file fixes this for me.
That makes search work for me as well.
👍,
Adam
--
"This is not going exactly as planned." Adam Sjøgren
asjo@koldfront.dk
Adam writes: > Thomas writes: > >> the attached patch applied to the patched file fixes this for me. > > That makes search work for me as well. In case Thomas' email doesn't reach the mailing list, here is the combined diff I ended up with: diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el index 70bde264c1..1b7b38a1cc 100644 --- a/lisp/gnus/gnus-search.el +++ b/lisp/gnus/gnus-search.el @@ -1351,68 +1351,59 @@ gnus-search-run-search (cl-defmethod gnus-search-indexed-parse-output ((engine gnus-search-indexed) server query &optional groups) - (let ((prefix (slot-value engine 'remove-prefix)) - (group-regexp (when groups - (mapconcat - (lambda (group-name) - (mapconcat #'regexp-quote - (split-string - (gnus-group-real-name group-name) - "[.\\/]") - "[.\\\\/]")) - groups - "\\|"))) - artlist vectors article group) + (let ((prefix (or (slot-value engine 'remove-prefix) + "")) + artlist article group) (goto-char (point-min)) + ;; Prep prefix, we want to at least be removing the root + ;; filesystem separator. + (when (stringp prefix) + (setq prefix (file-name-as-directory + (expand-file-name prefix "/")))) (while (not (or (eobp) (looking-at-p "\\(?:[[:space:]\n]+\\)?Process .+ finished"))) (pcase-let ((`(,f-name ,score) (gnus-search-indexed-extract engine))) (when (and f-name (file-readable-p f-name) - (null (file-directory-p f-name)) - (or (null groups) - (and (gnus-search-single-p query) - (alist-get 'thread query)) - (string-match-p group-regexp f-name))) - (push (list f-name score) artlist)))) + (null (file-directory-p f-name))) + (setq group + (replace-regexp-in-string + "[/\\]" "." + (replace-regexp-in-string + "/?\\(cur\\|new\\|tmp\\)?/\\'" "" + (replace-regexp-in-string + "\\`\\." "" + (string-remove-prefix + prefix (file-name-directory f-name)) + nil t) + nil t) + nil t)) + (setq group (gnus-group-full-name group server)) + (setq article (file-name-nondirectory f-name) + article + ;; TODO: Provide a cleaner way of producing final + ;; article numbers for the various backends. + (if (string-match-p "\\`[[:digit:]]+\\'" article) + (string-to-number article) + (nnmaildir-base-name-to-article-number + (substring article 0 (string-match ":" article)) + group (string-remove-prefix "nnmaildir:" server)))) + (when (and (numberp article) + (or (null groups) + (member group groups))) + (push (list f-name article group score) + artlist))))) ;; Are we running an additional grep query? (when-let ((grep-reg (alist-get 'grep query))) (setq artlist (gnus-search-grep-search engine artlist grep-reg))) - ;; Prep prefix. - (when (and prefix (null (string-empty-p prefix))) - (setq prefix (file-name-as-directory (expand-file-name prefix)))) - ;; Turn (file-name score) into [group article score]. - (pcase-dolist (`(,f-name ,score) artlist) - (setq article (file-name-nondirectory f-name) - group (file-name-directory f-name)) - ;; Remove prefix. - (when prefix - (setq group (string-remove-prefix prefix group))) - ;; Break the directory name down until it's something that - ;; (probably) can be used as a group name. - (setq group - (replace-regexp-in-string - "[/\\]" "." - (replace-regexp-in-string - "/?\\(cur\\|new\\|tmp\\)?/\\'" "" - (replace-regexp-in-string - "^[./\\]" "" - group nil t) - nil t) - nil t)) - - (push (vector (gnus-group-full-name group server) - (if (string-match-p "\\`[[:digit:]]+\\'" article) - (string-to-number article) - (nnmaildir-base-name-to-article-number - (substring article 0 (string-match ":" article)) - group (string-remove-prefix "nnmaildir:" server))) - (if (numberp score) - score - (string-to-number score))) - vectors)) - vectors)) + ;; Munge into the list of vectors expected by nnselect. + (mapcar (pcase-lambda (`(,_ ,article ,group ,score)) + (vector group article + (if (numberp score) + score + (string-to-number score)))) + artlist))) (cl-defmethod gnus-search-indexed-extract ((_engine gnus-search-indexed)) "Base implementation treats the whole line as a filename, and -- "Wiggle, wiggle, wiggle like a bowl of soup, Adam Sjøgren Wiggle, wiggle, wiggle like a rolling hoop, asjo@koldfront.dk
Hi Eric, I'm very busy today and tomorrow but I'll try to give this a whirl soon. -- Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian bullseye/sid
[-- Attachment #1: Type: text/plain, Size: 589 bytes --] Adam Sjøgren <asjo@koldfront.dk> writes: > Adam writes: > >> Thomas writes: >> >>> the attached patch applied to the patched file fixes this for me. >> >> That makes search work for me as well. > > In case Thomas' email doesn't reach the mailing list, here is the > combined diff I ended up with: Thanks to you both! That happened to slip by the test cases I was using, and that's why we get extra testers! I've attached a new version of the patch with your change applied; I'll give it a few days in case anyone else wants to test it, then push. Thanks, Eric [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-Rework-gnus-search-indexed-parse-output.patch --] [-- Type: text/x-patch, Size: 5068 bytes --] From b15fd217335f6711560ef88205ccfc262b4d2f25 Mon Sep 17 00:00:00 2001 From: Eric Abrahamsen <eric@ericabrahamsen.net> Date: Sat, 26 Jun 2021 10:16:19 -0700 Subject: [PATCH] Rework gnus-search-indexed-parse-output * lisp/gnus/gnus-search.el (gnus-search-indexed-parse-output): Be more careful about matching filesystem paths to Gnus group names; make absolutely sure that we only return valid article numbers. --- lisp/gnus/gnus-search.el | 95 ++++++++++++++++++---------------------- 1 file changed, 43 insertions(+), 52 deletions(-) diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el index 70bde264c1..898b57bcef 100644 --- a/lisp/gnus/gnus-search.el +++ b/lisp/gnus/gnus-search.el @@ -1351,68 +1351,59 @@ gnus-search-run-search (cl-defmethod gnus-search-indexed-parse-output ((engine gnus-search-indexed) server query &optional groups) - (let ((prefix (slot-value engine 'remove-prefix)) - (group-regexp (when groups - (mapconcat - (lambda (group-name) - (mapconcat #'regexp-quote - (split-string - (gnus-group-real-name group-name) - "[.\\/]") - "[.\\\\/]")) - groups - "\\|"))) - artlist vectors article group) + (let ((prefix (or (slot-value engine 'remove-prefix) + "")) + artlist article group) (goto-char (point-min)) + ;; Prep prefix, we want to at least be removing the root + ;; filesystem separator. + (when (stringp prefix) + (setq prefix (file-name-as-directory + (expand-file-name prefix "/")))) (while (not (or (eobp) (looking-at-p "\\(?:[[:space:]\n]+\\)?Process .+ finished"))) (pcase-let ((`(,f-name ,score) (gnus-search-indexed-extract engine))) (when (and f-name (file-readable-p f-name) - (null (file-directory-p f-name)) - (or (null groups) - (and (gnus-search-single-p query) - (alist-get 'thread query)) - (string-match-p group-regexp f-name))) - (push (list f-name score) artlist)))) + (null (file-directory-p f-name))) + (setq group + (replace-regexp-in-string + "[/\\]" "." + (replace-regexp-in-string + "/?\\(cur\\|new\\|tmp\\)?/\\'" "" + (replace-regexp-in-string + "\\`\\." "" + (string-remove-prefix + prefix (file-name-directory f-name)) + nil t) + nil t) + nil t)) + (setq group (gnus-group-full-name group server)) + (setq article (file-name-nondirectory f-name) + article + ;; TODO: Provide a cleaner way of producing final + ;; article numbers for the various backends. + (if (string-match-p "\\`[[:digit:]]+\\'" article) + (string-to-number article) + (nnmaildir-base-name-to-article-number + (substring article 0 (string-match ":" article)) + group (string-remove-prefix "nnmaildir:" server)))) + (when (and (numberp article) + (or (null groups) + (member group groups))) + (push (list f-name article group score) + artlist))))) ;; Are we running an additional grep query? (when-let ((grep-reg (alist-get 'grep query))) (setq artlist (gnus-search-grep-search engine artlist grep-reg))) - ;; Prep prefix. - (when (and prefix (null (string-empty-p prefix))) - (setq prefix (file-name-as-directory (expand-file-name prefix)))) - ;; Turn (file-name score) into [group article score]. - (pcase-dolist (`(,f-name ,score) artlist) - (setq article (file-name-nondirectory f-name) - group (file-name-directory f-name)) - ;; Remove prefix. - (when prefix - (setq group (string-remove-prefix prefix group))) - ;; Break the directory name down until it's something that - ;; (probably) can be used as a group name. - (setq group - (replace-regexp-in-string - "[/\\]" "." - (replace-regexp-in-string - "/?\\(cur\\|new\\|tmp\\)?/\\'" "" - (replace-regexp-in-string - "^[./\\]" "" - group nil t) - nil t) - nil t)) - - (push (vector (gnus-group-full-name group server) - (if (string-match-p "\\`[[:digit:]]+\\'" article) - (string-to-number article) - (nnmaildir-base-name-to-article-number - (substring article 0 (string-match ":" article)) - group (string-remove-prefix "nnmaildir:" server))) - (if (numberp score) - score - (string-to-number score))) - vectors)) - vectors)) + ;; Munge into the list of vectors expected by nnselect. + (mapcar (pcase-lambda (`(,_ ,article ,group ,score)) + (vector group article + (if (numberp score) + score + (string-to-number score)))) + artlist))) (cl-defmethod gnus-search-indexed-extract ((_engine gnus-search-indexed)) "Base implementation treats the whole line as a filename, and -- 2.32.0
On 07/23/21 13:12 PM, Alexandr Vityazev wrote:
> Hi!
>
> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24,
> cairo version 1.16.0). Gnus v5.13.
>
> I tested the new version of gnus-search-indexed-parse-output yesterday
> and the search with notmuch completely fails. gnus-verbose is set to
> 10. press GG and type any search query, with the patch, I get:
>
> Doing /home/akagi/.guix-profile/bin/notmuch query on
> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
> nnmaildir+posteo:Trash nnmaildir+posteo:Notes)...
> Group nnselect:nnselect-87lf5xus2s.fsf contains no messages
Hi, thanks for the report. Would you mind running your query on the
command line (using the same syntax as gnus-search does, mostly
output=files), and sending me the resulting list of file names
(off-list, if you prefer). I'll try to find out why the parsing process
is failing.
On 07/23/21 18:52 PM, Alexandr Vityazev wrote:
> On 2021-07-23, 08:18 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> On 07/23/21 13:12 PM, Alexandr Vityazev wrote:
>>> Hi!
>>>
>>> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24,
>>> cairo version 1.16.0). Gnus v5.13.
>>>
>>> I tested the new version of gnus-search-indexed-parse-output yesterday
>>> and the search with notmuch completely fails. gnus-verbose is set to
>>> 10. press GG and type any search query, with the patch, I get:
>>>
>>> Doing /home/akagi/.guix-profile/bin/notmuch query on
>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes)...
>>> Group nnselect:nnselect-87lf5xus2s.fsf contains no messages
>>
>> Hi, thanks for the report. Would you mind running your query on the
>> command line (using the same syntax as gnus-search does, mostly
>> output=files), and sending me the resulting list of file names
>> (off-list, if you prefer). I'll try to find out why the parsing process
>> is failing.
>>
>>
> notmuch search --output=files test
>
>
>
> in *Group* GG with "test" query, I get:
>
> Doing /home/akagi/.guix-profile/bin/notmuch query on
> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
> nnmaildir+posteo:Trash nnmaildir+posteo:Notes nnmaildir+posteo:Junk)...
> Group nnselect:nnselect-87czr8uasp.fsf contains no messages
Hmm, I'm not able to reproduce this: it's a little difficult since I
don't actually have the files on my machine, but as far as I can tell
the search results are being parsed correctly. Are you familiar with
edebug at all? Is there any chance I could get you to step through the
function and see where things are going off?
Thanks,
Eric
[-- Attachment #1: Type: text/plain, Size: 2059 bytes --] On 2021-07-24, 15:16 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote: > On 07/23/21 18:52 PM, Alexandr Vityazev wrote: >> On 2021-07-23, 08:18 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote: >> >>> On 07/23/21 13:12 PM, Alexandr Vityazev wrote: >>>> Hi! >>>> >>>> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, >>>> cairo version 1.16.0). Gnus v5.13. >>>> >>>> I tested the new version of gnus-search-indexed-parse-output yesterday >>>> and the search with notmuch completely fails. gnus-verbose is set to >>>> 10. press GG and type any search query, with the patch, I get: >>>> >>>> Doing /home/akagi/.guix-profile/bin/notmuch query on >>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent >>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes)... >>>> Group nnselect:nnselect-87lf5xus2s.fsf contains no messages >>> >>> Hi, thanks for the report. Would you mind running your query on the >>> command line (using the same syntax as gnus-search does, mostly >>> output=files), and sending me the resulting list of file names >>> (off-list, if you prefer). I'll try to find out why the parsing process >>> is failing. >>> >>> >> notmuch search --output=files test >> >> >> >> in *Group* GG with "test" query, I get: >> >> Doing /home/akagi/.guix-profile/bin/notmuch query on >> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent >> nnmaildir+posteo:Trash nnmaildir+posteo:Notes nnmaildir+posteo:Junk)... >> Group nnselect:nnselect-87czr8uasp.fsf contains no messages > > Hmm, I'm not able to reproduce this: it's a little difficult since I > don't actually have the files on my machine, but as far as I can tell > the search results are being parsed correctly. Are you familiar with > edebug at all? Is there any chance I could get you to step through the > function and see where things are going off? > > Thanks, > Eric > > I tested gnus-search-indexed-parse-output with edebug and found where something is going on. The attached patch fixes the issue for me. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: gnus-search-indexed-parse-output --] [-- Type: text/x-patch, Size: 1383 bytes --] From bfba4f38289223c3e78b19a73ad9133d015f8420 Mon Sep 17 00:00:00 2001 From: Alexandr Vityazev <avityazev@posteo.org> Date: Sun, 25 Jul 2021 14:09:39 +0300 Subject: [PATCH] gnus-search: gnus-search-indexed-parse-output: Fix search issue. * lisp/gnus/gnus-search.el (gnus-search-indexed-parse-output): Setting the group variable after the article variable. --- lisp/gnus/gnus-search.el | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el index 39bde837b3..80c3ce62f1 100644 --- a/lisp/gnus/gnus-search.el +++ b/lisp/gnus/gnus-search.el @@ -1384,7 +1384,6 @@ Returns a list of [group article score] vectors." nil t) nil t) nil t)) - (setq group (gnus-group-full-name group server)) (setq article (file-name-nondirectory f-name) article ;; TODO: Provide a cleaner way of producing final @@ -1394,6 +1393,7 @@ Returns a list of [group article score] vectors." (nnmaildir-base-name-to-article-number (substring article 0 (string-match ":" article)) group (string-remove-prefix "nnmaildir:" server)))) + (setq group (gnus-group-full-name group server)) (when (and (numberp article) (or (null groups) (member group groups))) -- 2.32.0 [-- Attachment #3: Type: text/plain, Size: 37 bytes --] -- Best regards, Alexandr Vityazev
On 07/25/21 11:17 AM, Alexandr Vityazev wrote:
> On 2021-07-24, 15:16 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> On 07/23/21 18:52 PM, Alexandr Vityazev wrote:
>>> On 2021-07-23, 08:18 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>>>
>>>> On 07/23/21 13:12 PM, Alexandr Vityazev wrote:
>>>>> Hi!
>>>>>
>>>>> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24,
>>>>> cairo version 1.16.0). Gnus v5.13.
>>>>>
>>>>> I tested the new version of gnus-search-indexed-parse-output yesterday
>>>>> and the search with notmuch completely fails. gnus-verbose is set to
>>>>> 10. press GG and type any search query, with the patch, I get:
>>>>>
>>>>> Doing /home/akagi/.guix-profile/bin/notmuch query on
>>>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
>>>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes)...
>>>>> Group nnselect:nnselect-87lf5xus2s.fsf contains no messages
>>>>
>>>> Hi, thanks for the report. Would you mind running your query on the
>>>> command line (using the same syntax as gnus-search does, mostly
>>>> output=files), and sending me the resulting list of file names
>>>> (off-list, if you prefer). I'll try to find out why the parsing process
>>>> is failing.
>>>>
>>>>
>>> notmuch search --output=files test
>>>
>>>
>>>
>>> in *Group* GG with "test" query, I get:
>>>
>>> Doing /home/akagi/.guix-profile/bin/notmuch query on
>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes nnmaildir+posteo:Junk)...
>>> Group nnselect:nnselect-87czr8uasp.fsf contains no messages
>>
>> Hmm, I'm not able to reproduce this: it's a little difficult since I
>> don't actually have the files on my machine, but as far as I can tell
>> the search results are being parsed correctly. Are you familiar with
>> edebug at all? Is there any chance I could get you to step through the
>> function and see where things are going off?
>>
>> Thanks,
>> Eric
>>
>>
> I tested gnus-search-indexed-parse-output with edebug and found where
> something is going on. The attached patch fixes the issue for me.
Wonderful, thank you very much for doing that. That code was added in to
fix a different bug, and I think we're just playing the usual "is the
group prefixed or not" game. I need to more thoroughly test the
primary/secondary select method cases, but at least we know what the
issue is -- thanks again for your debugging!
Eric
On Sun, 2021-07-25 at 11:17, Alexandr Vityazev <avityazev@posteo.org> wrote: > On 2021-07-24, 15:16 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote: >> On 07/23/21 18:52 PM, Alexandr Vityazev wrote: >>> On 2021-07-23, 08:18 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote: >>>> On 07/23/21 13:12 PM, Alexandr Vityazev wrote: >>>>> Hi! >>>>> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, >>>>> cairo version 1.16.0). Gnus v5.13. >>>>> I tested the new version of gnus-search-indexed-parse-output yesterday >>>>> and the search with notmuch completely fails. gnus-verbose is set to >>>>> 10. press GG and type any search query, with the patch, I get: >>>>> Doing /home/akagi/.guix-profile/bin/notmuch query on >>>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent >>>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes)... >>>>> Group nnselect:nnselect-87lf5xus2s.fsf contains no messages >>>> Hi, thanks for the report. Would you mind running your query on the >>>> command line (using the same syntax as gnus-search does, mostly >>>> output=files), and sending me the resulting list of file names >>>> (off-list, if you prefer). I'll try to find out why the parsing process >>>> is failing. >>> notmuch search --output=files test >>> in *Group* GG with "test" query, I get: >>> Doing /home/akagi/.guix-profile/bin/notmuch query on >>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent >>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes nnmaildir+posteo:Junk)... >>> Group nnselect:nnselect-87czr8uasp.fsf contains no messages >> Hmm, I'm not able to reproduce this: it's a little difficult since I >> don't actually have the files on my machine, but as far as I can tell >> the search results are being parsed correctly. Are you familiar with >> edebug at all? Is there any chance I could get you to step through the >> function and see where things are going off? >> Thanks, >> Eric > I tested gnus-search-indexed-parse-output with edebug and found where > something is going on. The attached patch fixes the issue for me. Hello all, Alexandr, I applied this patch against the latest commit to master at the time of writing (55a9c17cef) and I still get no search results when there should be. With `gnus-verbose' and `gnus-verbose-backends', I just see Retrieving newsgroup: nnselect:nnselect-86im0r65gy.fsf... Doing notmuch query on (test)... Group nnselect:nnselect-86im0r65gy.fsf contains no messages in *Messages*. My search setup is simple. I just have '((nnimap . imap) (nntp . gmane) (nnml . notmuch)) as the value of `gnus-search-default-engines'. The only way I can get searching working again is to revert gnus-search.el to 0897ade8f9. Regards, Joe
On Fri, 2021-07-30 at 15:17, Joseph Mingrone <jrm@ftfl.ca> wrote: > On Sun, 2021-07-25 at 11:17, Alexandr Vityazev <avityazev@posteo.org> wrote: >> On 2021-07-24, 15:16 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote: >>> On 07/23/21 18:52 PM, Alexandr Vityazev wrote: >>>> On 2021-07-23, 08:18 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote: >>>>> On 07/23/21 13:12 PM, Alexandr Vityazev wrote: >>>>>> Hi! >>>>>> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, >>>>>> cairo version 1.16.0). Gnus v5.13. >>>>>> I tested the new version of gnus-search-indexed-parse-output yesterday >>>>>> and the search with notmuch completely fails. gnus-verbose is set to >>>>>> 10. press GG and type any search query, with the patch, I get: >>>>>> Doing /home/akagi/.guix-profile/bin/notmuch query on >>>>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent >>>>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes)... >>>>>> Group nnselect:nnselect-87lf5xus2s.fsf contains no messages >>>>> Hi, thanks for the report. Would you mind running your query on the >>>>> command line (using the same syntax as gnus-search does, mostly >>>>> output=files), and sending me the resulting list of file names >>>>> (off-list, if you prefer). I'll try to find out why the parsing process >>>>> is failing. >>>> notmuch search --output=files test >>>> in *Group* GG with "test" query, I get: >>>> Doing /home/akagi/.guix-profile/bin/notmuch query on >>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent >>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes nnmaildir+posteo:Junk)... >>>> Group nnselect:nnselect-87czr8uasp.fsf contains no messages >>> Hmm, I'm not able to reproduce this: it's a little difficult since I >>> don't actually have the files on my machine, but as far as I can tell >>> the search results are being parsed correctly. Are you familiar with >>> edebug at all? Is there any chance I could get you to step through the >>> function and see where things are going off? >>> Thanks, >>> Eric >> I tested gnus-search-indexed-parse-output with edebug and found where >> something is going on. The attached patch fixes the issue for me. > Hello all, > Alexandr, I applied this patch against the latest commit to master at > the time of writing (55a9c17cef) and I still get no search results when > there should be. With `gnus-verbose' and `gnus-verbose-backends', I > just see Sorry. That should have read with `gnus-verbose' and `gnus-verbose-backends' both set to 10. > Retrieving newsgroup: nnselect:nnselect-86im0r65gy.fsf... > Doing notmuch query on (test)... > Group nnselect:nnselect-86im0r65gy.fsf contains no messages > in *Messages*. > My search setup is simple. I just have > '((nnimap . imap) (nntp . gmane) (nnml . notmuch)) as the value of > `gnus-search-default-engines'. > The only way I can get searching working again is to revert > gnus-search.el to 0897ade8f9. > Regards, > Joe
Joseph Mingrone <jrm@ftfl.ca> writes:
> On Fri, 2021-07-30 at 15:17, Joseph Mingrone <jrm@ftfl.ca> wrote:
>
>> On Sun, 2021-07-25 at 11:17, Alexandr Vityazev <avityazev@posteo.org> wrote:
>
>>> On 2021-07-24, 15:16 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>>>> On 07/23/21 18:52 PM, Alexandr Vityazev wrote:
>>>>> On 2021-07-23, 08:18 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>>>>>> On 07/23/21 13:12 PM, Alexandr Vityazev wrote:
>>>>>>> Hi!
>
>>>>>>> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24,
>>>>>>> cairo version 1.16.0). Gnus v5.13.
>
>>>>>>> I tested the new version of gnus-search-indexed-parse-output yesterday
>>>>>>> and the search with notmuch completely fails. gnus-verbose is set to
>>>>>>> 10. press GG and type any search query, with the patch, I get:
>
>>>>>>> Doing /home/akagi/.guix-profile/bin/notmuch query on
>>>>>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
>>>>>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes)...
>>>>>>> Group nnselect:nnselect-87lf5xus2s.fsf contains no messages
>
>>>>>> Hi, thanks for the report. Would you mind running your query on the
>>>>>> command line (using the same syntax as gnus-search does, mostly
>>>>>> output=files), and sending me the resulting list of file names
>>>>>> (off-list, if you prefer). I'll try to find out why the parsing process
>>>>>> is failing.
>
>
>>>>> notmuch search --output=files test
>
>
>
>>>>> in *Group* GG with "test" query, I get:
>
>>>>> Doing /home/akagi/.guix-profile/bin/notmuch query on
>>>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
>>>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes nnmaildir+posteo:Junk)...
>>>>> Group nnselect:nnselect-87czr8uasp.fsf contains no messages
>
>>>> Hmm, I'm not able to reproduce this: it's a little difficult since I
>>>> don't actually have the files on my machine, but as far as I can tell
>>>> the search results are being parsed correctly. Are you familiar with
>>>> edebug at all? Is there any chance I could get you to step through the
>>>> function and see where things are going off?
>
>>>> Thanks,
>>>> Eric
>
>
>>> I tested gnus-search-indexed-parse-output with edebug and found where
>>> something is going on. The attached patch fixes the issue for me.
>
>> Hello all,
>
>> Alexandr, I applied this patch against the latest commit to master at
>> the time of writing (55a9c17cef) and I still get no search results when
>> there should be. With `gnus-verbose' and `gnus-verbose-backends', I
>> just see
>
> Sorry. That should have read with `gnus-verbose' and
> `gnus-verbose-backends' both set to 10.
Remind me which of your servers is primary and which secondary? I'm
still pretty sure that the issue is prefixed vs unprefixed group names.
On Fri, 2021-07-30 at 11:47, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
> Remind me which of your servers is primary and which secondary? I'm
> still pretty sure that the issue is prefixed vs unprefixed group names.
`gnus-message-archive-method' is '(nnml ""), both `gnus-select-method'
and `gnus-message-archive-method' are set to (nnml ""), and
`gnus-search-default-engines' is '((nnimap . imap) (nntp . gmane) (nnml
. notmuch))). Gnus stores mail in the standard location, ~/Mail.
Is that what you mean? I can step through the code if you can give me a
hint of what I should be looking for.
On 2021-07-30, 15:17 -0300, Joseph Mingrone <jrm@ftfl.ca> wrote:
> On Sun, 2021-07-25 at 11:17, Alexandr Vityazev <avityazev@posteo.org> wrote:
>
>> On 2021-07-24, 15:16 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>>> On 07/23/21 18:52 PM, Alexandr Vityazev wrote:
>>>> On 2021-07-23, 08:18 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>>>>> On 07/23/21 13:12 PM, Alexandr Vityazev wrote:
>>>>>> Hi!
>
>>>>>> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24,
>>>>>> cairo version 1.16.0). Gnus v5.13.
>
>>>>>> I tested the new version of gnus-search-indexed-parse-output yesterday
>>>>>> and the search with notmuch completely fails. gnus-verbose is set to
>>>>>> 10. press GG and type any search query, with the patch, I get:
>
>>>>>> Doing /home/akagi/.guix-profile/bin/notmuch query on
>>>>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
>>>>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes)...
>>>>>> Group nnselect:nnselect-87lf5xus2s.fsf contains no messages
>
>>>>> Hi, thanks for the report. Would you mind running your query on the
>>>>> command line (using the same syntax as gnus-search does, mostly
>>>>> output=files), and sending me the resulting list of file names
>>>>> (off-list, if you prefer). I'll try to find out why the parsing process
>>>>> is failing.
>
>
>>>> notmuch search --output=files test
>
>
>
>>>> in *Group* GG with "test" query, I get:
>
>>>> Doing /home/akagi/.guix-profile/bin/notmuch query on
>>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
>>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes nnmaildir+posteo:Junk)...
>>>> Group nnselect:nnselect-87czr8uasp.fsf contains no messages
>
>>> Hmm, I'm not able to reproduce this: it's a little difficult since I
>>> don't actually have the files on my machine, but as far as I can tell
>>> the search results are being parsed correctly. Are you familiar with
>>> edebug at all? Is there any chance I could get you to step through the
>>> function and see where things are going off?
>
>>> Thanks,
>>> Eric
>
>
>> I tested gnus-search-indexed-parse-output with edebug and found where
>> something is going on. The attached patch fixes the issue for me.
>
> Hello all,
>
> Alexandr, I applied this patch against the latest commit to master at
> the time of writing (55a9c17cef) and I still get no search results when
> there should be. With `gnus-verbose' and `gnus-verbose-backends', I
> just see
>
> Retrieving newsgroup: nnselect:nnselect-86im0r65gy.fsf...
> Doing notmuch query on (test)...
> Group nnselect:nnselect-86im0r65gy.fsf contains no messages
>
> in *Messages*.
>
> My search setup is simple. I just have
>
> '((nnimap . imap) (nntp . gmane) (nnml . notmuch)) as the value of
> `gnus-search-default-engines'.
>
> The only way I can get searching working again is to revert
> gnus-search.el to 0897ade8f9.
>
> Regards,
>
> Joe
>
>
Hi, Joseph!
All my methods are set in gnus-secondary-select-methods. Most likely the
patch solves the problem only with these settings. I did not research
whether it will work with other settings. To solve the problem with the
search it is necessary to conduct a more complete analysis, I still
cannot give an answer.
--
Best regards,
Alexandr Vityazev
Joseph Mingrone <jrm@ftfl.ca> writes:
> On Fri, 2021-07-30 at 11:47, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>> Remind me which of your servers is primary and which secondary? I'm
>> still pretty sure that the issue is prefixed vs unprefixed group names.
>
> `gnus-message-archive-method' is '(nnml ""), both `gnus-select-method'
> and `gnus-message-archive-method' are set to (nnml ""), and
> `gnus-search-default-engines' is '((nnimap . imap) (nntp . gmane) (nnml
> . notmuch))). Gnus stores mail in the standard location, ~/Mail.
>
> Is that what you mean? I can step through the code if you can give me a
> hint of what I should be looking for.
That's fine, it means your `gnus-select-method' is 'nnml, and the other
servers are all `gnus-secondary-select-methods'. That's all I was asking.
Alexandr Vityazev <avityazev@posteo.org> writes:
> On 2021-07-30, 15:17 -0300, Joseph Mingrone <jrm@ftfl.ca> wrote:
>
>> On Sun, 2021-07-25 at 11:17, Alexandr Vityazev <avityazev@posteo.org> wrote:
>>
>>> On 2021-07-24, 15:16 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>>
>>>> On 07/23/21 18:52 PM, Alexandr Vityazev wrote:
>>>>> On 2021-07-23, 08:18 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>>
>>>>>> On 07/23/21 13:12 PM, Alexandr Vityazev wrote:
>>>>>>> Hi!
>>
>>>>>>> GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.24,
>>>>>>> cairo version 1.16.0). Gnus v5.13.
>>
>>>>>>> I tested the new version of gnus-search-indexed-parse-output yesterday
>>>>>>> and the search with notmuch completely fails. gnus-verbose is set to
>>>>>>> 10. press GG and type any search query, with the patch, I get:
>>
>>>>>>> Doing /home/akagi/.guix-profile/bin/notmuch query on
>>>>>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
>>>>>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes)...
>>>>>>> Group nnselect:nnselect-87lf5xus2s.fsf contains no messages
>>
>>>>>> Hi, thanks for the report. Would you mind running your query on the
>>>>>> command line (using the same syntax as gnus-search does, mostly
>>>>>> output=files), and sending me the resulting list of file names
>>>>>> (off-list, if you prefer). I'll try to find out why the parsing process
>>>>>> is failing.
>>
>>
>>>>> notmuch search --output=files test
>>
>>
>>
>>>>> in *Group* GG with "test" query, I get:
>>
>>>>> Doing /home/akagi/.guix-profile/bin/notmuch query on
>>>>> (nnmaildir+posteo:Drafts nnmaildir+posteo:Inbox nnmaildir+posteo:Sent
>>>>> nnmaildir+posteo:Trash nnmaildir+posteo:Notes nnmaildir+posteo:Junk)...
>>>>> Group nnselect:nnselect-87czr8uasp.fsf contains no messages
>>
>>>> Hmm, I'm not able to reproduce this: it's a little difficult since I
>>>> don't actually have the files on my machine, but as far as I can tell
>>>> the search results are being parsed correctly. Are you familiar with
>>>> edebug at all? Is there any chance I could get you to step through the
>>>> function and see where things are going off?
>>
>>>> Thanks,
>>>> Eric
>>
>>
>>> I tested gnus-search-indexed-parse-output with edebug and found where
>>> something is going on. The attached patch fixes the issue for me.
>>
>> Hello all,
>>
>> Alexandr, I applied this patch against the latest commit to master at
>> the time of writing (55a9c17cef) and I still get no search results when
>> there should be. With `gnus-verbose' and `gnus-verbose-backends', I
>> just see
>>
>> Retrieving newsgroup: nnselect:nnselect-86im0r65gy.fsf...
>> Doing notmuch query on (test)...
>> Group nnselect:nnselect-86im0r65gy.fsf contains no messages
>>
>> in *Messages*.
>>
>> My search setup is simple. I just have
>>
>> '((nnimap . imap) (nntp . gmane) (nnml . notmuch)) as the value of
>> `gnus-search-default-engines'.
>>
>> The only way I can get searching working again is to revert
>> gnus-search.el to 0897ade8f9.
>>
>> Regards,
>>
>> Joe
>>
>>
>
> Hi, Joseph!
>
> All my methods are set in gnus-secondary-select-methods. Most likely the
> patch solves the problem only with these settings. I did not research
> whether it will work with other settings. To solve the problem with the
> search it is necessary to conduct a more complete analysis, I still
> cannot give an answer.
I'm getting to it! I just need to find a moment to sit down and set up
some dummy data and test the various possible configurations. It
shouldn't be to hard to unilaterally force prefixed group names where
that's needed, and force unprefixed names elsewhere, I just need to
find the time. I'd like to get this fixed once and for all.
[-- Attachment #1: Type: text/plain, Size: 788 bytes --] Eric Abrahamsen <eric@ericabrahamsen.net> writes: [...] > I'm getting to it! I just need to find a moment to sit down and set up > some dummy data and test the various possible configurations. It > shouldn't be to hard to unilaterally force prefixed group names where > that's needed, and force unprefixed names elsewhere, I just need to > find the time. I'd like to get this fixed once and for all. Okay, I think I've finally got it. The incoming group names are indeed prefixed or not depending on whether the server is primary or not (arguably this should be fixed to always unprefixed in nnselect, but I'll worry about that later), so this patch enforces unprefixed group names within the function body, and prefixed group names in the return value. Please give it a shot! Eric [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: gnus-search-groups-fix.diff --] [-- Type: text/x-patch, Size: 1172 bytes --] diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el index 53af2f6fe6..d92122f2bd 100644 --- a/lisp/gnus/gnus-search.el +++ b/lisp/gnus/gnus-search.el @@ -1358,6 +1358,7 @@ gnus-search-indexed-parse-output server query &optional groups) (let ((prefix (or (slot-value engine 'remove-prefix) "")) + (groups (mapcar #'gnus-group-short-name groups)) artlist article group) (goto-char (point-min)) ;; Prep prefix, we want to at least be removing the root @@ -1404,10 +1405,12 @@ gnus-search-indexed-parse-output (setq artlist (gnus-search-grep-search engine artlist grep-reg))) ;; Munge into the list of vectors expected by nnselect. (mapcar (pcase-lambda (`(,_ ,article ,group ,score)) - (vector group article - (if (numberp score) - score - (string-to-number score)))) + (vector + (gnus-group-full-name group server) + article + (if (numberp score) + score + (string-to-number score)))) artlist))) (cl-defmethod gnus-search-indexed-extract ((_engine gnus-search-indexed))
On 2021-08-05, 10:15 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>
> [...]
>
>> I'm getting to it! I just need to find a moment to sit down and set up
>> some dummy data and test the various possible configurations. It
>> shouldn't be to hard to unilaterally force prefixed group names where
>> that's needed, and force unprefixed names elsewhere, I just need to
>> find the time. I'd like to get this fixed once and for all.
>
> Okay, I think I've finally got it. The incoming group names are indeed
> prefixed or not depending on whether the server is primary or not
> (arguably this should be fixed to always unprefixed in nnselect, but
> I'll worry about that later), so this patch enforces unprefixed group
> names within the function body, and prefixed group names in the return
> value. Please give it a shot!
>
> Eric
>
>
The patch was applied to the master branch and it doesn't work for me.
In my case because of line 1388 with the code:
#+begin_src elisp
(setq group (gnus-group-full-name group server))
#+end_src
code on 1395 line:
#+begin_src elisp
(nnmaildir-base-name-to-article-number
(substring article 0 (string-match ":" article))
group (string-remove-prefix "nnmaildir:" server))
#+end_src
always return nil, because the group on line 1397 always gets something
like "nnmaildir+gmail:Inbox" and not "Inbox".
And in the code on 1400 line:
#+begin_src elisp
(member group groups)
#+end_src
also requires the group similar to "Inbox", so it can only be solved by
deleting line 1388:
#+begin_src elisp
(setq group (gnus-group-full-name group server))
#+end_src
--
Best regards,
Alexandr Vityazev
On Thu, 2021-08-05 at 19:54, Alexandr Vityazev <avityazev@posteo.org> wrote: > On 2021-08-05, 10:15 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote: >> Eric Abrahamsen <eric@ericabrahamsen.net> writes: >> [...] >>> I'm getting to it! I just need to find a moment to sit down and set up >>> some dummy data and test the various possible configurations. It >>> shouldn't be to hard to unilaterally force prefixed group names where >>> that's needed, and force unprefixed names elsewhere, I just need to >>> find the time. I'd like to get this fixed once and for all. >> Okay, I think I've finally got it. The incoming group names are indeed >> prefixed or not depending on whether the server is primary or not >> (arguably this should be fixed to always unprefixed in nnselect, but >> I'll worry about that later), so this patch enforces unprefixed group >> names within the function body, and prefixed group names in the return >> value. Please give it a shot! >> Eric > The patch was applied to the master branch and it doesn't work for me. > In my case because of line 1388 with the code: > #+begin_src elisp > (setq group (gnus-group-full-name group server)) > #+end_src > code on 1395 line: > #+begin_src elisp > (nnmaildir-base-name-to-article-number > (substring article 0 (string-match ":" article)) > group (string-remove-prefix "nnmaildir:" server)) > #+end_src > always return nil, because the group on line 1397 always gets something > like "nnmaildir+gmail:Inbox" and not "Inbox". > And in the code on 1400 line: > #+begin_src elisp > (member group groups) > #+end_src > also requires the group similar to "Inbox", so it can only be solved by > deleting line 1388: > #+begin_src elisp > (setq group (gnus-group-full-name group server)) > #+end_src I see the same as you Alexandr, i.e., with the patch applied to a recent commit from the master branch, still no search results, but after removing line 1388, results are returned. Thanks both, Joe
[-- Attachment #1: Type: text/plain, Size: 2255 bytes --] Joseph Mingrone <jrm@ftfl.ca> writes: > On Thu, 2021-08-05 at 19:54, Alexandr Vityazev <avityazev@posteo.org> wrote: > >> On 2021-08-05, 10:15 -0700, Eric Abrahamsen <eric@ericabrahamsen.net> wrote: > >>> Eric Abrahamsen <eric@ericabrahamsen.net> writes: > > >>> [...] > >>>> I'm getting to it! I just need to find a moment to sit down and set up >>>> some dummy data and test the various possible configurations. It >>>> shouldn't be to hard to unilaterally force prefixed group names where >>>> that's needed, and force unprefixed names elsewhere, I just need to >>>> find the time. I'd like to get this fixed once and for all. > >>> Okay, I think I've finally got it. The incoming group names are indeed >>> prefixed or not depending on whether the server is primary or not >>> (arguably this should be fixed to always unprefixed in nnselect, but >>> I'll worry about that later), so this patch enforces unprefixed group >>> names within the function body, and prefixed group names in the return >>> value. Please give it a shot! > >>> Eric > > > >> The patch was applied to the master branch and it doesn't work for me. >> In my case because of line 1388 with the code: > >> #+begin_src elisp >> (setq group (gnus-group-full-name group server)) >> #+end_src > > >> code on 1395 line: > >> #+begin_src elisp >> (nnmaildir-base-name-to-article-number >> (substring article 0 (string-match ":" article)) >> group (string-remove-prefix "nnmaildir:" server)) >> #+end_src > > >> always return nil, because the group on line 1397 always gets something >> like "nnmaildir+gmail:Inbox" and not "Inbox". > >> And in the code on 1400 line: > >> #+begin_src elisp >> (member group groups) >> #+end_src > > >> also requires the group similar to "Inbox", so it can only be solved by >> deleting line 1388: > >> #+begin_src elisp >> (setq group (gnus-group-full-name group server)) >> #+end_src > > I see the same as you Alexandr, i.e., with the patch applied to a recent > commit from the master branch, still no search results, but after > removing line 1388, results are returned. Grr, sorry, I sent the wrong diff -- the original call to `gnus-group-full-name' was still in there. I promise I'm sending the correct, freshly-tested diff this time! [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: gnus-search-groups.diff --] [-- Type: text/x-patch, Size: 1501 bytes --] diff --git a/lisp/gnus/gnus-search.el b/lisp/gnus/gnus-search.el index 53af2f6fe6..8182630dfe 100644 --- a/lisp/gnus/gnus-search.el +++ b/lisp/gnus/gnus-search.el @@ -1358,6 +1358,7 @@ gnus-search-indexed-parse-output server query &optional groups) (let ((prefix (or (slot-value engine 'remove-prefix) "")) + (groups (mapcar #'gnus-group-short-name groups)) artlist article group) (goto-char (point-min)) ;; Prep prefix, we want to at least be removing the root @@ -1384,7 +1385,6 @@ gnus-search-indexed-parse-output nil t) nil t) nil t)) - (setq group (gnus-group-full-name group server)) (setq article (file-name-nondirectory f-name) article ;; TODO: Provide a cleaner way of producing final @@ -1404,10 +1404,12 @@ gnus-search-indexed-parse-output (setq artlist (gnus-search-grep-search engine artlist grep-reg))) ;; Munge into the list of vectors expected by nnselect. (mapcar (pcase-lambda (`(,_ ,article ,group ,score)) - (vector group article - (if (numberp score) - score - (string-to-number score)))) + (vector + (gnus-group-full-name group server) + article + (if (numberp score) + score + (string-to-number score)))) artlist))) (cl-defmethod gnus-search-indexed-extract ((_engine gnus-search-indexed))
On Thu, 2021-08-05 at 14:24, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
> Grr, sorry, I sent the wrong diff -- the original call to
> `gnus-group-full-name' was still in there. I promise I'm sending the
> correct, freshly-tested diff this time!
Looks good.
Joe
Joseph Mingrone <jrm@ftfl.ca> writes:
> On Thu, 2021-08-05 at 14:24, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>> Grr, sorry, I sent the wrong diff -- the original call to
>> `gnus-group-full-name' was still in there. I promise I'm sending the
>> correct, freshly-tested diff this time!
>
> Looks good.
In it goes, thanks for testing!
Eric, I tried your latest gnus-search code, but it didn't work for me. I noticed that in gnus-search-indexed-parse-output the `group' variable was wrong, resulting in nnmaildir-base-name-to-article-number returning nil. This is my setup, having multiple secondary nnmaildir servers: (setq gnus-secondary-select-methods '((nnmaildir "One" (directory "~/Mail/one/")) (nnmaildir "Two" (directory "~/Mail/two/")) ...) gnus-search-default-engines '((nnmaildir . notmuch))) I found that I could make it work by giving `prefix' its value from the servers at the start of the function. Like so: (let ((prefix (or (nnmaildir--srv-dir (alist-get (string-remove-prefix "nnmaildir:" server) nnmaildir--servers nil nil #'equal)) "")) ...)) This uses the correct prefix "~/Mail/one/" and so on, not just "~/Mail/" from the engine. I have no idea what a general solution would look like. It could be my setup that is wrong, I don't know.
Daniel Jensen <daniel@bigwalter.net> writes:
> Eric,
>
> I tried your latest gnus-search code, but it didn't work for me. I
> noticed that in gnus-search-indexed-parse-output the `group' variable
> was wrong, resulting in nnmaildir-base-name-to-article-number returning
> nil. This is my setup, having multiple secondary nnmaildir servers:
>
> (setq gnus-secondary-select-methods
> '((nnmaildir "One" (directory "~/Mail/one/"))
> (nnmaildir "Two" (directory "~/Mail/two/"))
> ...)
> gnus-search-default-engines
> '((nnmaildir . notmuch)))
>
> I found that I could make it work by giving `prefix' its value from the
> servers at the start of the function. Like so:
>
> (let ((prefix (or (nnmaildir--srv-dir
> (alist-get (string-remove-prefix "nnmaildir:" server)
> nnmaildir--servers nil nil #'equal))
> ""))
> ...))
>
> This uses the correct prefix "~/Mail/one/" and so on, not just "~/Mail/"
> from the engine. I have no idea what a general solution would look like.
> It could be my setup that is wrong, I don't know.
Sorry for the slow response, I was camping...
If you're only using one notmuch index, it's enough to set the global
options like `gnus-search-notmuch-remove-prefix'. If you've got two
nnmaildir servers each with its own notmuch index, you'll have to
specific the different prefixes within the engine configuration:
((nnmaildir "One"
(directory "~/Mail/one/"
(gnus-search-engine gnus-search-notmuch
(remove-prefix "~/Mail/one/"))))
(nnmaildir "Two"
(directory "~/Mail/two/"
(gnus-search-engine gnus-search-notmuch
(remove-prefix "~/Mail/two/")))))
Note how the sexps are nested, it's important that the engine-related
variables go within the engine configuration sexp.
I realize that 'remove-prefix is redundant with the nnmaildir 'directory
setting here, and hope to eventually be able to get rid of that.
Hope that helps,
Eric
On 2021-08-13 at 09:50, Eric Abrahamsen wrote: > Sorry for the slow response, I was camping... No worries! > If you're only using one notmuch index, it's enough to set the global > options like `gnus-search-notmuch-remove-prefix'. If you've got two > nnmaildir servers each with its own notmuch index, you'll have to > specific the different prefixes within the engine configuration: > > ((nnmaildir "One" > (directory "~/Mail/one/" > (gnus-search-engine gnus-search-notmuch > (remove-prefix "~/Mail/one/")))) > (nnmaildir "Two" > (directory "~/Mail/two/" > (gnus-search-engine gnus-search-notmuch > (remove-prefix "~/Mail/two/"))))) > > Note how the sexps are nested, it's important that the engine-related > variables go within the engine configuration sexp. I had tried something like this before, but could never get it right. Actually your example above needed a small change: ((nnmaildir "One" (directory "~/Mail/one/") (gnus-search-engine gnus-search-notmuch (remove-prefix "~/Mail/one/"))) (nnmaildir "Two" (directory "~/Mail/two/") (gnus-search-engine gnus-search-notmuch (remove-prefix "~/Mail/two/")))) Notice the gnus-search-engine sexps had to move up for it to work. But now all is well. Thank you very much!
Daniel Jensen <daniel@bigwalter.net> writes:
> On 2021-08-13 at 09:50, Eric Abrahamsen wrote:
>
>> Sorry for the slow response, I was camping...
>
> No worries!
>
>> If you're only using one notmuch index, it's enough to set the global
>> options like `gnus-search-notmuch-remove-prefix'. If you've got two
>> nnmaildir servers each with its own notmuch index, you'll have to
>> specific the different prefixes within the engine configuration:
>>
>> ((nnmaildir "One"
>> (directory "~/Mail/one/"
>> (gnus-search-engine gnus-search-notmuch
>> (remove-prefix "~/Mail/one/"))))
>> (nnmaildir "Two"
>> (directory "~/Mail/two/"
>> (gnus-search-engine gnus-search-notmuch
>> (remove-prefix "~/Mail/two/")))))
>>
>> Note how the sexps are nested, it's important that the engine-related
>> variables go within the engine configuration sexp.
>
> I had tried something like this before, but could never get it right.
> Actually your example above needed a small change:
>
> ((nnmaildir "One"
> (directory "~/Mail/one/")
> (gnus-search-engine gnus-search-notmuch
> (remove-prefix "~/Mail/one/")))
> (nnmaildir "Two"
> (directory "~/Mail/two/")
> (gnus-search-engine gnus-search-notmuch
> (remove-prefix "~/Mail/two/"))))
>
> Notice the gnus-search-engine sexps had to move up for it to work. But
> now all is well. Thank you very much!
"Now watch carefully as I do this incorrectly..."
It really is easy to screw up! Glad everything's working now.
[-- Attachment #1: Type: text/plain, Size: 531 bytes --] Hello! I have a problem with gnus with notmuch search. When I open Gnus, I can see my mail, but when I try to press "G G" and search for example mail from Bandcamp with query "Bandcamp" I get a message "Group nnselect:nnselect-<some number>.fsf" contains no messages. I tried to repeat notmuch query, which Gnus sends, it renders some results, but they seem not to be parsed correctly. Maybe the reason is that search results contain encoded unicode characters so they cannot be parsed? Best wishes, Makarov Sergey [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 865 bytes --]
Sergey Makarov <setser200018@gmail.com> writes:
> Hello!
>
> I have a problem with gnus with notmuch search. When I open Gnus, I can
> see my mail, but when I try to press "G G" and search for example mail
> from Bandcamp with query "Bandcamp" I get a message "Group
> nnselect:nnselect-<some number>.fsf" contains no messages. I tried to
> repeat notmuch query, which Gnus sends, it renders some results, but
> they seem not to be parsed correctly. Maybe the reason is that search
> results contain encoded unicode characters so they cannot be parsed?
Hi there,
Can you run the equivalent notmuch search in a terminal, and paste the
output here? I'll see if the parsing thing is choking on something in
particular.
Thanks,
Eric