emacs 28 search not working with nnml

emacs 28 search not working with nnml

[Thomas Alexander Gerds]

hello

a recent upgrade to emacs 28.0.50 and my homegrown search functions
stopped working. Eric Abrahamsen kindly helped me to translate my
obsolete nnir-settings (thank you Eric!). but there is another problem
deeper in the system which I describe here.

I am using nnml and have all my email downloaded to my laptop using
notmuch for searching. now I am confident that notmuch is finding some
articles when I enter via GG or envoke this very simple function:

(defun test-search ()
  (interactive)
  (gnus-group-read-ephemeral-search-group
   nil
   `((search-query-spec (query . "from:Alessandra"))
     (search-group-spec ("nnml:" . ("nnml:research"))))))

some hours debugging showed that nnselect-retrieve-headers calls the
macro nnselect-get-artlist which returns a too long group name, i.e.,
"nnml:home.tag.mail.research" where I think that it should simply return
"research". at least this helps a little bit because

a) now nnmail-group-pathname does not produce the buggy result
/home/tag/mail/nnml/home/tag/mail

b) the following hack

,----
| (setq nnselect-retrieve-headers-override-function 'my-retrieve-headers)
| (defun my-retrieve-headers (sequence &optional group server fetch-old)
|   (setq group (replace-in-string group "nnml.home.tag.mail." ""))
|     (when (nnml-possibly-change-directory group server)
|     ...
`----

shows me a summary buffer with the search results. only the body of
articles are all empty. I guess this happens because
(replace-in-string group "nnml.home.tag.mail." "") should be applied not only when
retrieving headers but also when retrieving articles.

now I am hoping that this is a bug which can be fixed somewhere earlier
on in this complex chain of commands? but, any help is very much
appreciated.

cheers, Thomas



-- 
7LL-6 Stop thinking too much, it’s alright not to know all the answers.

Re: emacs 28 search not working with nnml

[Eric S Fraga]

On Wednesday, 23 Jun 2021 at 14:54, Thomas Alexander Gerds wrote:
> I am using nnml and have all my email downloaded to my laptop using

I have a similar configuration.  My settings look like this:

#+begin_src emacs-lisp
  (setq gnus-secondary-select-methods
        '((nnml "work"
                (gnus-search-engine gnus-search-notmuch
                                    (remove-prefix "/HOME/Mail")
                                    (config-file "/HOME/.notmuch-config")
                                    )
                (get-new-mail t))))

  (add-to-list 'gnus-search-default-engines '(nnml . notmuch))
  (setq gnus-search-notmuch-remove-prefix "/HOME/Mail")
  (with-eval-after-load "gnus-search"
    (cl-defmethod gnus-search-indexed-parse-output :around
      ((e gnus-search-notmuch) s q groups)
      (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
                        groups)))
        (cl-call-next-method e s q gs))))
#+end_src

where HOME in various places above should be replaced by your home
directory.  I have gnus splitting email into various sub-directories of
Mail/.

The last bit was suggested by Jose A. Ortega Ruiz on the gnus newsgroup
[1] sometime in late 2020 and proved to be necessary (at that time) to
get the new search engine to work with nnml.  It basically changes the
file paths from . to / separators.



Footnotes:
[1]  gnus:nntp+news.gwene.org:gmane.emacs.gnus.general#87pn4hc97h.fsf@gnus.jao.io

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian bullseye/sid

Re: emacs 28 search not working with nnml

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Wednesday, 23 Jun 2021 at 14:54, Thomas Alexander Gerds wrote:
>> I am using nnml and have all my email downloaded to my laptop using
>
> I have a similar configuration.  My settings look like this:
>
> #+begin_src emacs-lisp
>   (setq gnus-secondary-select-methods
>         '((nnml "work"
>                 (gnus-search-engine gnus-search-notmuch
>                                     (remove-prefix "/HOME/Mail")
>                                     (config-file "/HOME/.notmuch-config")
>                                     )
>                 (get-new-mail t))))
>
>   (add-to-list 'gnus-search-default-engines '(nnml . notmuch))
>   (setq gnus-search-notmuch-remove-prefix "/HOME/Mail")
>   (with-eval-after-load "gnus-search"
>     (cl-defmethod gnus-search-indexed-parse-output :around
>       ((e gnus-search-notmuch) s q groups)
>       (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
>                         groups)))
>         (cl-call-next-method e s q gs))))
> #+end_src
>
> where HOME in various places above should be replaced by your home
> directory.  I have gnus splitting email into various sub-directories of
> Mail/.
>
> The last bit was suggested by Jose A. Ortega Ruiz on the gnus newsgroup
> [1] sometime in late 2020 and proved to be necessary (at that time) to
> get the new search engine to work with nnml.  It basically changes the
> file paths from . to / separators.

FWIW, this hack isn't necessary any more, if you're running a recent
Emacs from master (looks like since early February of this year).

Thomas, the bug you're seeing is something I'm aware of, I just haven't
come up with a great solution for it yet (actually I'd forgotten about
it). I'll get that fixed soon, and report back here.

Eric

Re: emacs 28 search not working with nnml

[Thomas Alexander Gerds]

Eric(1): thank you so much, this helped and solved my problem!
Eric(2): you are right: the hack is not needed and my search functions work
         smoothly without it. not sure what remaining bug there might
         be, but you may want to consider to change the default value
         here for people like me who use "mail" instead of "Mail":

(defcustom gnus-search-notmuch-remove-prefix gnus-directory
  "The prefix to remove from each file name returned by notmuch
in order to get a group name (albeit with / instead of .).  This is a
regular expression.

This variable can also be set per-server."
  :type 'regexp
  :version "28.1")

cheers Thomas

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>
>> On Wednesday, 23 Jun 2021 at 14:54, Thomas Alexander Gerds wrote:
>>> I am using nnml and have all my email downloaded to my laptop using
>>
>> I have a similar configuration.  My settings look like this:
>>
>> #+begin_src emacs-lisp
>>   (setq gnus-secondary-select-methods
>>         '((nnml "work"
>>                 (gnus-search-engine gnus-search-notmuch
>>                                     (remove-prefix "/HOME/Mail")
>>                                     (config-file "/HOME/.notmuch-config")
>>                                     )
>>                 (get-new-mail t))))
>>
>>   (add-to-list 'gnus-search-default-engines '(nnml . notmuch))
>>   (setq gnus-search-notmuch-remove-prefix "/HOME/Mail")
>>   (with-eval-after-load "gnus-search"
>>     (cl-defmethod gnus-search-indexed-parse-output :around
>>       ((e gnus-search-notmuch) s q groups)
>>       (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
>>                         groups)))
>>         (cl-call-next-method e s q gs))))
>> #+end_src
>>
>> where HOME in various places above should be replaced by your home
>> directory.  I have gnus splitting email into various sub-directories of
>> Mail/.
>>
>> The last bit was suggested by Jose A. Ortega Ruiz on the gnus newsgroup
>> [1] sometime in late 2020 and proved to be necessary (at that time) to
>> get the new search engine to work with nnml.  It basically changes the
>> file paths from . to / separators.
>
> FWIW, this hack isn't necessary any more, if you're running a recent
> Emacs from master (looks like since early February of this year).
>
> Thomas, the bug you're seeing is something I'm aware of, I just haven't
> come up with a great solution for it yet (actually I'd forgotten about
> it). I'll get that fixed soon, and report back here.
>
> Eric
>
>
>
>

-- 
Thomas A. Gerds -- Department of Biostatistics Copenhagen
University of Copenhagen, Oester Farimagsgade 5, 1014 Copenhagen,
Denmark

Re: emacs 28 search not working with nnml

[Eric S Fraga]

On Wednesday, 23 Jun 2021 at 08:01, Eric Abrahamsen wrote:
> FWIW, this hack isn't necessary any more, if you're running a recent
> Emacs from master (looks like since early February of this year).

Thank you.  I will remove this hack.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.4.6 on Debian bullseye/sid

Re: emacs 28 search not working with nnml

[Eric Abrahamsen]

Thomas Alexander Gerds <grb615@ku.dk> writes:

> Eric(1): thank you so much, this helped and solved my problem!
> Eric(2): you are right: the hack is not needed and my search functions work
>          smoothly without it. not sure what remaining bug there might
>          be,

The remaining bug is subtler: if you search a group whose name is a
substring of a different group, the code will also attempt to return
results from that other group, which will fail.

> but you may want to consider to change the default value here for
> people like me who use "mail" instead of "Mail":

Wait, which default value are you referring to here? `gnus-directory'?

> (defcustom gnus-search-notmuch-remove-prefix gnus-directory
>   "The prefix to remove from each file name returned by notmuch
> in order to get a group name (albeit with / instead of .).  This is a
> regular expression.
>
> This variable can also be set per-server."
>   :type 'regexp
>   :version "28.1")
>
> cheers Thomas
>
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>
>>> On Wednesday, 23 Jun 2021 at 14:54, Thomas Alexander Gerds wrote:
>>>> I am using nnml and have all my email downloaded to my laptop using
>>>
>>> I have a similar configuration.  My settings look like this:
>>>
>>> #+begin_src emacs-lisp
>>>   (setq gnus-secondary-select-methods
>>>         '((nnml "work"
>>>                 (gnus-search-engine gnus-search-notmuch
>>>                                     (remove-prefix "/HOME/Mail")
>>>                                     (config-file "/HOME/.notmuch-config")
>>>                                     )
>>>                 (get-new-mail t))))
>>>
>>>   (add-to-list 'gnus-search-default-engines '(nnml . notmuch))
>>>   (setq gnus-search-notmuch-remove-prefix "/HOME/Mail")
>>>   (with-eval-after-load "gnus-search"
>>>     (cl-defmethod gnus-search-indexed-parse-output :around
>>>       ((e gnus-search-notmuch) s q groups)
>>>       (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
>>>                         groups)))
>>>         (cl-call-next-method e s q gs))))
>>> #+end_src
>>>
>>> where HOME in various places above should be replaced by your home
>>> directory.  I have gnus splitting email into various sub-directories of
>>> Mail/.
>>>
>>> The last bit was suggested by Jose A. Ortega Ruiz on the gnus newsgroup
>>> [1] sometime in late 2020 and proved to be necessary (at that time) to
>>> get the new search engine to work with nnml.  It basically changes the
>>> file paths from . to / separators.
>>
>> FWIW, this hack isn't necessary any more, if you're running a recent
>> Emacs from master (looks like since early February of this year).
>>
>> Thomas, the bug you're seeing is something I'm aware of, I just haven't
>> come up with a great solution for it yet (actually I'd forgotten about
>> it). I'll get that fixed soon, and report back here.
>>
>> Eric
>>
>>
>>
>>

Re: emacs 28 search not working with nnml

[Thomas Alexander Gerds]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Thomas Alexander Gerds <grb615@ku.dk> writes:
>
>> Eric(1): thank you so much, this helped and solved my problem!
>> Eric(2): you are right: the hack is not needed and my search functions work
>>          smoothly without it. not sure what remaining bug there might
>>          be,
>
> The remaining bug is subtler: if you search a group whose name is a
> substring of a different group, the code will also attempt to return
> results from that other group, which will fail.

aah. haha, and I thought it was a neat feature that in addition to
nnml:research the search was also applied to nnml:research.old

>
>> but you may want to consider to change the default value here for
>> people like me who use "mail" instead of "Mail":
>
> Wait, which default value are you referring to here? `gnus-directory'?

no the default value of gnus-search-notmuch-remove-prefix. I have
forgotten if there also is a variable that has a default value equal to
"~/Mail". it seems that gnus-directory defaults to "~/News".


>> (defcustom gnus-search-notmuch-remove-prefix gnus-directory
>>   "The prefix to remove from each file name returned by notmuch
>> in order to get a group name (albeit with / instead of .).  This is a
>> regular expression.
>>
>> This variable can also be set per-server."
>>   :type 'regexp
>>   :version "28.1")
>>
>> cheers Thomas
>>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>>>
>>>> On Wednesday, 23 Jun 2021 at 14:54, Thomas Alexander Gerds wrote:
>>>>> I am using nnml and have all my email downloaded to my laptop using
>>>>
>>>> I have a similar configuration.  My settings look like this:
>>>>
>>>> #+begin_src emacs-lisp
>>>>   (setq gnus-secondary-select-methods
>>>>         '((nnml "work"
>>>>                 (gnus-search-engine gnus-search-notmuch
>>>>                                     (remove-prefix "/HOME/Mail")
>>>>                                     (config-file "/HOME/.notmuch-config")
>>>>                                     )
>>>>                 (get-new-mail t))))
>>>>
>>>>   (add-to-list 'gnus-search-default-engines '(nnml . notmuch))
>>>>   (setq gnus-search-notmuch-remove-prefix "/HOME/Mail")
>>>>   (with-eval-after-load "gnus-search"
>>>>     (cl-defmethod gnus-search-indexed-parse-output :around
>>>>       ((e gnus-search-notmuch) s q groups)
>>>>       (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
>>>>                         groups)))
>>>>         (cl-call-next-method e s q gs))))
>>>> #+end_src
>>>>
>>>> where HOME in various places above should be replaced by your home
>>>> directory.  I have gnus splitting email into various sub-directories of
>>>> Mail/.
>>>>
>>>> The last bit was suggested by Jose A. Ortega Ruiz on the gnus newsgroup
>>>> [1] sometime in late 2020 and proved to be necessary (at that time) to
>>>> get the new search engine to work with nnml.  It basically changes the
>>>> file paths from . to / separators.
>>>
>>> FWIW, this hack isn't necessary any more, if you're running a recent
>>> Emacs from master (looks like since early February of this year).
>>>
>>> Thomas, the bug you're seeing is something I'm aware of, I just haven't
>>> come up with a great solution for it yet (actually I'd forgotten about
>>> it). I'll get that fixed soon, and report back here.
>>>
>>> Eric
>>>
>>>
>>>
>>>
>
>
>
>

-- 
Thomas A. Gerds -- Department of Biostatistics Copenhagen
University of Copenhagen, Oester Farimagsgade 5, 1014 Copenhagen,
Denmark

Re: emacs 28 search not working with nnml

[Eric Abrahamsen]

Thomas Alexander Gerds <grb615@ku.dk> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Thomas Alexander Gerds <grb615@ku.dk> writes:
>>
>>> Eric(1): thank you so much, this helped and solved my problem!
>>> Eric(2): you are right: the hack is not needed and my search functions work
>>>          smoothly without it. not sure what remaining bug there might
>>>          be,
>>
>> The remaining bug is subtler: if you search a group whose name is a
>> substring of a different group, the code will also attempt to return
>> results from that other group, which will fail.
>
> aah. haha, and I thought it was a neat feature that in addition to
> nnml:research the search was also applied to nnml:research.old

No, sorry, that's a bug :) But you can certainly use a mark-by-regexp
command to mark all your "*research*" groups and then search them at
once! Only fourteen more keystrokes.

>>
>>> but you may want to consider to change the default value here for
>>> people like me who use "mail" instead of "Mail":
>>
>> Wait, which default value are you referring to here? `gnus-directory'?
>
> no the default value of gnus-search-notmuch-remove-prefix. I have
> forgotten if there also is a variable that has a default value equal to
> "~/Mail". it seems that gnus-directory defaults to "~/News".

Oh, I see. That's just the way that's always been, it's not something I
set. It's hard to imagine any default value that would actually be of
use -- I imagine everyone's got something slightly different.