nnir notmuch search on specific groups?

nnir notmuch search on specific groups?

[Andreas Goesele]

Hi,

with quite some effort I was able to get nnir to work together with the
notmuch search engine. But then, differently from what I expected from
the documentation in the manual, I discovered that obviously with the
notmuch engine the searches are not restricted to the groups selected.

Looking at the code I found this error message, commented out:

;; (when group
;;   (error "The notmuch backend cannot search specific groups")

I guess this indeed means what it says, but as it is commented out and
not mentioned in the manual, I would like to get some confirmation:

Is it possible or not to search specific groups with the notmuch
backend?

Thanks a lot

Andreas

-- 
Omnis enim res, quae dando non deficit, dum habetur et non datur,
nondum habetur, quomodo habenda est.
                                      Augustinus, De doctrina christiana

Re: nnir notmuch search on specific groups?

[Eric Abrahamsen]

"Andreas Goesele" <goesele@hfph.mwn.de> writes:

> Hi,
>
> with quite some effort I was able to get nnir to work together with the
> notmuch search engine. But then, differently from what I expected from
> the documentation in the manual, I discovered that obviously with the
> notmuch engine the searches are not restricted to the groups selected.
>
> Looking at the code I found this error message, commented out:
>
> ;; (when group
> ;;   (error "The notmuch backend cannot search specific groups")
>
> I guess this indeed means what it says, but as it is commented out and
> not mentioned in the manual, I would like to get some confirmation:
>
> Is it possible or not to search specific groups with the notmuch
> backend?

People have been struggling with this for a while, here's the approach
that looks most likely to solve your problem:

https://lists.gnu.org/archive/html/info-gnus-english/2016-09/msg00065.html

What mail backend are you using for this, by the way?

Eric

Re: nnir notmuch search on specific groups?

[Adam Sjøgren]

Eric writes:

> People have been struggling with this for a while, here's the approach
> that looks most likely to solve your problem:
>
> https://lists.gnu.org/archive/html/info-gnus-english/2016-09/msg00065.html

That doesn't seem to have any hints on how to search only the email in a
specific group?

The nnir-integration built into Gnus, i.e. setting (nnir-search-engine
notmuch) on my local mail back end of choice (nnml), works for me, but
it doesn't allow searching in a specific group.

But maybe notmuch has some search term that can be used.

Yes, there is a path:<directory-path> term, which could be useful. There
is also a folder:<maildir-folder>, if you use maildir.

Maybe nnir could figure out the paths of selected groups and include
them in the search string for notmuch?


  Best regards,

    Adam

-- 
 "if we are questioned, we tell the observers, april          Adam Sjøgren
  fools, and leave the area before you are also fooled   asjo@koldfront.dk
  by this day with our bricks."


_______________________________________________
info-gnus-english mailing list
info-gnus-english@gnu.org
https://lists.gnu.org/mailman/listinfo/info-gnus-english

Re: nnir notmuch search on specific groups?

[Eric Abrahamsen]

Adam Sjøgren <asjo@koldfront.dk> writes:

> Eric writes:
>
>> People have been struggling with this for a while, here's the approach
>> that looks most likely to solve your problem:
>>
>> https://lists.gnu.org/archive/html/info-gnus-english/2016-09/msg00065.html
>
> That doesn't seem to have any hints on how to search only the email in a
> specific group?

You're right, I misremembered the point of that conversation.

> The nnir-integration built into Gnus, i.e. setting (nnir-search-engine
> notmuch) on my local mail back end of choice (nnml), works for me, but
> it doesn't allow searching in a specific group.
>
> But maybe notmuch has some search term that can be used.
>
> Yes, there is a path:<directory-path> term, which could be useful. There
> is also a folder:<maildir-folder>, if you use maildir.
>
> Maybe nnir could figure out the paths of selected groups and include
> them in the search string for notmuch?

I went back and looked at my old work on gnus-search (which will land
someday... sniff), and it looks like I bailed on making this work there,
too. I vaguely remember that I couldn't even make it work properly on
the command line, couldn't figure out why "folder" and "path" returned
the results they did.

But `nnir-run-notmuch' filters its results same as all the other search
routines, so I'm not sure why we can't do the filtering on group name,
after the fact. It has this:

 ;; maybe limit results to matching groups.
 (when (or (not groupspec)
           (string-match groupspec dirnam))
   (nnir-add-result dirnam artno "" prefix server artlist)))))

That should be usable for filtering, right?

I have the feeling that I've gone over all this before, then forgotten
what the conclusion was...

Eric


_______________________________________________
info-gnus-english mailing list
info-gnus-english@gnu.org
https://lists.gnu.org/mailman/listinfo/info-gnus-english

Re: nnir notmuch search on specific groups?

[Adam Sjøgren]

Eric writes:

> I vaguely remember that I couldn't even make it work properly on the
> command line, couldn't figure out why "folder" and "path" returned the
> results they did.

path:{folder} works for me, using notmuch on the command line.

My email is in nnml format in various folders in ~/Mail/{folder}, and my
.notmuch-config has:

  [database]
  path=/home/asjo/Mail

Searching for "Adam" gives me a lot of results:

  asjo@tullinup:~$ time notmuch search Adam | wc -l
  32911

  real    0m6.574s
  user    0m5.016s
  sys     0m1.577s

If I limit the search using path:, I get less, so it looks like it
works:

  asjo@tullinup:~$ time notmuch search "Adam path:cron" | wc -l
  507

  real    0m0.148s
  user    0m0.128s
  sys     0m0.024s

My notmuch index has ~119K entries:

  $ time notmuch count
  119018

> But `nnir-run-notmuch' filters its results same as all the other search
> routines, so I'm not sure why we can't do the filtering on group name,
> after the fact. It has this:
>
>  ;; maybe limit results to matching groups.
>  (when (or (not groupspec)
>            (string-match groupspec dirnam))
>    (nnir-add-result dirnam artno "" prefix server artlist)))))
>
> That should be usable for filtering, right?

Yeah! I guess it could matter when the results are excluded,
performance-wise, but in practise, I would guess it would be "fast
enough"™ to do it post hoc.

> I have the feeling that I've gone over all this before, then forgotten
> what the conclusion was...

I know that feeling all too well... :-)


  Best regards,

    Adam

-- 
 "Since it is completely undocumented, your faith that        Adam Sjøgren
  it will stick around until they invent v4 is           asjo@koldfront.dk
  charmingly quaint."


_______________________________________________
info-gnus-english mailing list
info-gnus-english@gnu.org
https://lists.gnu.org/mailman/listinfo/info-gnus-english

Re: nnir notmuch search on specific groups?

[Eric Abrahamsen]

Adam Sjøgren <asjo@koldfront.dk> writes:

> Eric writes:
>
>> I vaguely remember that I couldn't even make it work properly on the
>> command line, couldn't figure out why "folder" and "path" returned the
>> results they did.
>
> path:{folder} works for me, using notmuch on the command line.
>
> My email is in nnml format in various folders in ~/Mail/{folder}, and my
> .notmuch-config has:
>
>   [database]
>   path=/home/asjo/Mail
>
> Searching for "Adam" gives me a lot of results:
>
>   asjo@tullinup:~$ time notmuch search Adam | wc -l
>   32911
>
>   real    0m6.574s
>   user    0m5.016s
>   sys     0m1.577s
>
> If I limit the search using path:, I get less, so it looks like it
> works:
>
>   asjo@tullinup:~$ time notmuch search "Adam path:cron" | wc -l
>   507
>
>   real    0m0.148s
>   user    0m0.128s
>   sys     0m0.024s

Beats me what my problem was, then! But no matter.

>> But `nnir-run-notmuch' filters its results same as all the other search
>> routines, so I'm not sure why we can't do the filtering on group name,
>> after the fact. It has this:
>>
>>  ;; maybe limit results to matching groups.
>>  (when (or (not groupspec)
>>            (string-match groupspec dirnam))
>>    (nnir-add-result dirnam artno "" prefix server artlist)))))
>>
>> That should be usable for filtering, right?
>
> Yeah! I guess it could matter when the results are excluded,
> performance-wise, but in practise, I would guess it would be "fast
> enough"™ to do it post hoc.

Sure. It would be preferable to do it up front though!

This whole thing looks a little weird. The "groupspec" filter criteria
above is supposed to come from a "notmuch-group" query parameter (ie
nnir would promppt you for it), but as that parameter doesn't appear in
`nnir-engines', the prompt never happens, and we never get a "groupspec"
at all.

I don't see why we couldn't just use the GROUP argument. I think it
should be optional, because we can't always be sure the user doesn't
have some odd name transformation stuff going on, but for instance in
your case with nnml it seems like it ought to work fine.

Would you try this and see if it does what you'd expect?


(defun nnir-run-notmuch (query server &optional groups)
  "Run QUERY against notmuch.
Returns a vector of (group name, file name) pairs (also vectors,
actually).  If GROUPS is a list of group names, use them to
construct path: search terms."

  (save-excursion
    (let* ((qstring (cdr (assq 'query query)))
	   (prefix (nnir-read-server-parm 'nnir-notmuch-remove-prefix server))
           artlist
	   (article-pattern (if (string-match "\\`nnmaildir:"
					      (gnus-group-server server))
				":[0-9]+"
			      "^[0-9]+$"))
	   (groups (mapcar #'gnus-group-short-name groups))
	   (pathquery (when groups
			(mapconcat (lambda (g)
				     (format " path:%s" g))
				   groups " or")))
           artno dirnam filenam)

      (when (equal "" qstring)
        (error "notmuch: You didn't enter anything"))

      (set-buffer (get-buffer-create nnir-tmp-buffer))
      (erase-buffer)

      (if groups
          (message "Doing notmuch query %s on %s..."
		   qstring (mapconcat #'identity groups " "))
        (message "Doing notmuch query %s..." qstring))

      (when groups
	(setq qstring (concat qstring pathquery)))

      (let* ((cp-list `( ,nnir-notmuch-program
                         nil            ; input from /dev/null
                         t              ; output
                         nil            ; don't redisplay
                         "search"
                         "--format=text"
                         "--output=files"
                         ,@(nnir-read-server-parm 'nnir-notmuch-additional-switches server)
                         ,qstring       ; the query, in notmuch format
                         ))
             (exitstatus
              (progn
                (message "%s args: %s" nnir-notmuch-program
                         (mapconcat #'identity (nthcdr 4 cp-list) " ")) ;; ???
                (apply #'call-process cp-list))))
        (unless (or (null exitstatus)
                    (zerop exitstatus))
          (nnheader-report 'nnir "Couldn't run notmuch: %s" exitstatus)
          ;; notmuch failure reason is in this buffer, show it if
          ;; the user wants it.
          (when (> gnus-verbose 6)
            (display-buffer nnir-tmp-buffer))))

      ;; The results are output in the format of:
      ;; absolute-path-name
      (goto-char (point-min))
      (while (not (eobp))
        (setq filenam (buffer-substring-no-properties (line-beginning-position)
                                                      (line-end-position))
              artno (file-name-nondirectory filenam)
              dirnam (file-name-directory filenam))
        (forward-line 1)

        ;; don't match directories
        (when (string-match article-pattern artno)
          (when (not (null dirnam))

	    (nnir-add-result dirnam artno "" prefix server artlist))))

      (message "Massaging notmuch output...done")

      artlist)))


_______________________________________________
info-gnus-english mailing list
info-gnus-english@gnu.org
https://lists.gnu.org/mailman/listinfo/info-gnus-english

Re: nnir notmuch search on specific groups?

[Andreas Goesele]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Adam Sjøgren <asjo@koldfront.dk> writes:
>
> Would you try this and see if it does what you'd expect?

That's great: It works like charme with folders directly in the Mail
directory. For subfolders it only would be necessary to replace the "."
by a slash "/". So instead of searching "mail.subfolder" (as subfolders
are shown in the group buffer) it should search "mail/subfolder".

> (defun nnir-run-notmuch (query server &optional groups)
>   "Run QUERY against notmuch.
> Returns a vector of (group name, file name) pairs (also vectors,
> actually).  If GROUPS is a list of group names, use them to
> construct path: search terms."
>
>   (save-excursion
>     (let* ((qstring (cdr (assq 'query query)))
> 	   (prefix (nnir-read-server-parm 'nnir-notmuch-remove-prefix server))
>            artlist
> 	   (article-pattern (if (string-match "\\`nnmaildir:"
> 					      (gnus-group-server server))
> 				":[0-9]+"
> 			      "^[0-9]+$"))
> 	   (groups (mapcar #'gnus-group-short-name groups))
> 	   (pathquery (when groups
> 			(mapconcat (lambda (g)
> 				     (format " path:%s" g))
> 				   groups " or")))
>            artno dirnam filenam)
>
>       (when (equal "" qstring)
>         (error "notmuch: You didn't enter anything"))
>
>       (set-buffer (get-buffer-create nnir-tmp-buffer))
>       (erase-buffer)
>
>       (if groups
>           (message "Doing notmuch query %s on %s..."
> 		   qstring (mapconcat #'identity groups " "))
>         (message "Doing notmuch query %s..." qstring))
>
>       (when groups
> 	(setq qstring (concat qstring pathquery)))
>
>       (let* ((cp-list `( ,nnir-notmuch-program
>                          nil            ; input from /dev/null
>                          t              ; output
>                          nil            ; don't redisplay
>                          "search"
>                          "--format=text"
>                          "--output=files"
>                          ,@(nnir-read-server-parm 'nnir-notmuch-additional-switches server)
>                          ,qstring       ; the query, in notmuch format
>                          ))
>              (exitstatus
>               (progn
>                 (message "%s args: %s" nnir-notmuch-program
>                          (mapconcat #'identity (nthcdr 4 cp-list) " ")) ;; ???
>                 (apply #'call-process cp-list))))
>         (unless (or (null exitstatus)
>                     (zerop exitstatus))
>           (nnheader-report 'nnir "Couldn't run notmuch: %s" exitstatus)
>           ;; notmuch failure reason is in this buffer, show it if
>           ;; the user wants it.
>           (when (> gnus-verbose 6)
>             (display-buffer nnir-tmp-buffer))))
>
>       ;; The results are output in the format of:
>       ;; absolute-path-name
>       (goto-char (point-min))
>       (while (not (eobp))
>         (setq filenam (buffer-substring-no-properties (line-beginning-position)
>                                                       (line-end-position))
>               artno (file-name-nondirectory filenam)
>               dirnam (file-name-directory filenam))
>         (forward-line 1)
>
>         ;; don't match directories
>         (when (string-match article-pattern artno)
>           (when (not (null dirnam))
>
> 	    (nnir-add-result dirnam artno "" prefix server artlist))))
>
>       (message "Massaging notmuch output...done")
>
>       artlist)))
>
>

-- 
Omnis enim res, quae dando non deficit, dum habetur et non datur,
nondum habetur, quomodo habenda est.
                                      Augustinus, De doctrina christiana
_______________________________________________
info-gnus-english mailing list
info-gnus-english@gnu.org
https://lists.gnu.org/mailman/listinfo/info-gnus-english

Re: nnir notmuch search on specific groups?

[Eric Abrahamsen]

"Andreas Goesele" <goesele@hfph.mwn.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Adam Sjøgren <asjo@koldfront.dk> writes:
>>
>> Would you try this and see if it does what you'd expect?
>
> That's great: It works like charme with folders directly in the Mail
> directory. 

Great!

> For subfolders it only would be necessary to replace the "." by a
> slash "/". So instead of searching "mail.subfolder" (as subfolders are
> shown in the group buffer) it should search "mail/subfolder".

Well that's where I'm a little hesitant. I'm not really familiar with
how this path transformation stuff usually works, and don't want to

I suppose we could add an option that's either nil (don't use groups and
paths at all), t (use untransformed paths), or a function that munges
the group names appropriately. Then it would just be a matter of finding
a reasonable default behavior.

Eric


_______________________________________________
info-gnus-english mailing list
info-gnus-english@gnu.org
https://lists.gnu.org/mailman/listinfo/info-gnus-english

Re: nnir notmuch search on specific groups?

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> "Andreas Goesele" <goesele@hfph.mwn.de> writes:
>
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Adam Sjøgren <asjo@koldfront.dk> writes:
>>>
>>> Would you try this and see if it does what you'd expect?
>>
>> That's great: It works like charme with folders directly in the Mail
>> directory. 
>
> Great!
>
>> For subfolders it only would be necessary to replace the "." by a
>> slash "/". So instead of searching "mail.subfolder" (as subfolders are
>> shown in the group buffer) it should search "mail/subfolder".
>
> Well that's where I'm a little hesitant. I'm not really familiar with
> how this path transformation stuff usually works, and don't want to

Ahem... Don't want to come up with some random default that only works
for some users. For instance, we can't assume forward slash for Windows
users, right?


_______________________________________________
info-gnus-english mailing list
info-gnus-english@gnu.org
https://lists.gnu.org/mailman/listinfo/info-gnus-english

Re: nnir notmuch search on specific groups?

[Andreas Goesele]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>> For subfolders it only would be necessary to replace the "." by a
>> slash "/". So instead of searching "mail.subfolder" (as subfolders are
>> shown in the group buffer) it should search "mail/subfolder".
>
> Well that's where I'm a little hesitant. I'm not really familiar with
> how this path transformation stuff usually works, and don't want to
>
> I suppose we could add an option that's either nil (don't use groups and
> paths at all), t (use untransformed paths), or a function that munges
> the group names appropriately. Then it would just be a matter of finding
> a reasonable default behavior.

I think that would be a great solution, which could solve the backslash
vs. forward slash problem!

Andreas

-- 
Omnis enim res, quae dando non deficit, dum habetur et non datur,
nondum habetur, quomodo habenda est.
                                      Augustinus, De doctrina christiana

Re: nnir notmuch search on specific groups?

[Eric Abrahamsen]

"Andreas Goesele" <goesele@hfph.mwn.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>>> For subfolders it only would be necessary to replace the "." by a
>>> slash "/". So instead of searching "mail.subfolder" (as subfolders are
>>> shown in the group buffer) it should search "mail/subfolder".
>>
>> Well that's where I'm a little hesitant. I'm not really familiar with
>> how this path transformation stuff usually works, and don't want to
>>
>> I suppose we could add an option that's either nil (don't use groups and
>> paths at all), t (use untransformed paths), or a function that munges
>> the group names appropriately. Then it would just be a matter of finding
>> a reasonable default behavior.
>
> I think that would be a great solution, which could solve the backslash
> vs. forward slash problem!

Okay, I've reported this with a patch as bug#33122. The new option name
is called `nnir-notmuch-filter-group-names-function'.

In the end I decided a t value didn't make a lot of sense -- if users
want to pass the group names through unchanged (which is almost
certainly a bad idea), they can set the above option to #'identity.

By default (assuming this patch is accepted with no change), the
filtering *doesn't* do the transform you wanted. You can add your
desired behavior (the docstring mentions this) like so:

(add-function :filter-return nnir-notmuch-filter-group-names-function
  (lambda (g) (replace-regexp-in-string "\\." "/" g)))

Eric

Re: nnir notmuch search on specific groups?

[Andreas Goesele]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Okay, I've reported this with a patch as bug#33122. The new option name
> is called `nnir-notmuch-filter-group-names-function'.
>
> In the end I decided a t value didn't make a lot of sense -- if users
> want to pass the group names through unchanged (which is almost
> certainly a bad idea), they can set the above option to #'identity.
>
> By default (assuming this patch is accepted with no change), the
> filtering *doesn't* do the transform you wanted. You can add your
> desired behavior (the docstring mentions this) like so:
>
> (add-function :filter-return nnir-notmuch-filter-group-names-function
>   (lambda (g) (replace-regexp-in-string "\\." "/" g)))

Thanks, this basically works great. There is just one small glitch: If
one marks different groups (let say three) the search becomes:

search --format=text --output=files search-term path:<directory1> or path:<directory2> or path:<directory3>

This includes all mails in the directories 2 and 3

So it should be:

search --format=text --output=files search-term (path:<directory1> or path:<directory3> or path:<directory3>)

Thanks again!

Andreas

-- 
Omnis enim res, quae dando non deficit, dum habetur et non datur,
nondum habetur, quomodo habenda est.
                                      Augustinus, De doctrina christiana

Re: nnir notmuch search on specific groups?

[Eric Abrahamsen]

"Andreas Goesele" <goesele@hfph.mwn.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Okay, I've reported this with a patch as bug#33122. The new option name
>> is called `nnir-notmuch-filter-group-names-function'.
>>
>> In the end I decided a t value didn't make a lot of sense -- if users
>> want to pass the group names through unchanged (which is almost
>> certainly a bad idea), they can set the above option to #'identity.
>>
>> By default (assuming this patch is accepted with no change), the
>> filtering *doesn't* do the transform you wanted. You can add your
>> desired behavior (the docstring mentions this) like so:
>>
>> (add-function :filter-return nnir-notmuch-filter-group-names-function
>>   (lambda (g) (replace-regexp-in-string "\\." "/" g)))
>
> Thanks, this basically works great. There is just one small glitch: If
> one marks different groups (let say three) the search becomes:
>
> search --format=text --output=files search-term path:<directory1> or
> path:<directory2> or path:<directory3>
>
> This includes all mails in the directories 2 and 3
>
> So it should be:
>
> search --format=text --output=files search-term (path:<directory1> or
> path:<directory3> or path:<directory3>)

Aha! Thanks for the pointer. I've updated the patch on the bug report.

Eric

Re: nnir notmuch search on specific groups?

[Andreas Goesele]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Aha! Thanks for the pointer. I've updated the patch on the bug report.

I can't find any problem anymore. So thanks a lot!

Andreas

-- 
Omnis enim res, quae dando non deficit, dum habetur et non datur,
nondum habetur, quomodo habenda est.
                                      Augustinus, De doctrina christiana

Re: nnir notmuch search on specific groups?

[Andreas Goesele]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I went back and looked at my old work on gnus-search (which will land
> someday... sniff), and it looks like I bailed on making this work there,
> too. I vaguely remember that I couldn't even make it work properly on
> the command line, couldn't figure out why "folder" and "path" returned
> the results they did.

I find the results using "path" rather consistent. The only - pleasant -
surprise was "that terms with a common prefix will be implicitly combined
with OR".

> But `nnir-run-notmuch' filters its results same as all the other search
> routines, so I'm not sure why we can't do the filtering on group name,
> after the fact.

This would be a real improvement.

But as notmuch isn't *very* fast with a huge amount of hits I would
prefer a solution which restricts the search beforehand. (Unfortunately I
can't contribute.)

Andreas

-- 
Omnis enim res, quae dando non deficit, dum habetur et non datur,
nondum habetur, quomodo habenda est.
                                      Augustinus, De doctrina christiana

Re: nnir notmuch search on specific groups?

[Andreas Goesele]

Adam Sjøgren <asjo@koldfront.dk> writes:

> Eric writes:
>
>> People have been struggling with this for a while, here's the approach
>> that looks most likely to solve your problem:
>>
>> https://lists.gnu.org/archive/html/info-gnus-english/2016-09/msg00065.html
>
> That doesn't seem to have any hints on how to search only the email in a
> specific group?
>
> The nnir-integration built into Gnus, i.e. setting (nnir-search-engine
> notmuch) on my local mail back end of choice (nnml), works for me, but
> it doesn't allow searching in a specific group.

I finally got the built in nnir-notmuch integration to work, and I'm
using nnml too.

> But maybe notmuch has some search term that can be used.
>
> Yes, there is a path:<directory-path> term, which could be useful. There
> is also a folder:<maildir-folder>, if you use maildir.
>
> Maybe nnir could figure out the paths of selected groups and include
> them in the search string for notmuch?

Yes, I'm using this path:<directory-path> search term. But the
possibility to search the group on the current line or the groups marked
would be very handy.

Andreas

-- 
Omnis enim res, quae dando non deficit, dum habetur et non datur,
nondum habetur, quomodo habenda est.
                                      Augustinus, De doctrina christiana
_______________________________________________
info-gnus-english mailing list
info-gnus-english@gnu.org
https://lists.gnu.org/mailman/listinfo/info-gnus-english