Create a group from a list of message-ids

Create a group from a list of message-ids

[Alexander Baier]

Hello,

I use a tool called mu [1] to index and search my mail, which is stored
in a Maildir folder. mu is able to give me the message ids of all the
mails that match a given query. I want to use these ids to create a new
group in gnus that shows the search result. So it basically boils down
to this:

Is there a function or facility in gnus that takes a list of message-ids
(or even a list of local path names) and creates a new group containing
the corresponding messages?

I am fine with writing some elisp to call out to mu and converting its
output to a list, I just need above mentioned functionality.

Thanks in advance,

Alex

___
[1]  http://www.djcbsoftware.nl/code/mu/

Re: Create a group from a list of message-ids

[Dave Goldberg]

> Hello,
> I use a tool called mu [1] to index and search my mail, which is stored
> in a Maildir folder. mu is able to give me the message ids of all the
> mails that match a given query. I want to use these ids to create a new
> group in gnus that shows the search result. So it basically boils down
> to this:

> Is there a function or facility in gnus that takes a list of message-ids
> (or even a list of local path names) and creates a new group containing
> the corresponding messages?

> I am fine with writing some elisp to call out to mu and converting its
> output to a list, I just need above mentioned functionality.

nnir.el provides an interface to various search capabilities and although I can't speak to it as I don't use it, I see maildir mentioned for one of the backends.  Perhaps that is enough to get you started.

-- 
Dave Goldberg
david.goldberg6@verizon.net

Re: Create a group from a list of message-ids

[Alexander Baier]

Hello David,

On 13-11-25 03:02 Dave Goldberg wrote:
>> Hello,
>> I use a tool called mu [1] to index and search my mail, which is stored
>> in a Maildir folder. mu is able to give me the message ids of all the
>> mails that match a given query. I want to use these ids to create a new
>> group in gnus that shows the search result. So it basically boils down
>> to this:
>
>> Is there a function or facility in gnus that takes a list of message-ids
>> (or even a list of local path names) and creates a new group containing
>> the corresponding messages?
>
>> I am fine with writing some elisp to call out to mu and converting its
>> output to a list, I just need above mentioned functionality.
>
> nnir.el provides an interface to various search capabilities and
> although I can't speak to it as I don't use it, I see maildir
> mentioned for one of the backends.  Perhaps that is enough to get you
> started.

Thank you for that tipp. I just looked at a few backends for nnir and
found that the namazu backend provides similar if not the same
capabilities as the command line utility mu.

I will look into this and see if it meets my demands.

Regards,

--
Alex

Re: Create a group from a list of message-ids

[Frank Terbeck]

Alexander Baier wrote:
> On 13-11-25 03:02 Dave Goldberg wrote:
[...]
>> nnir.el provides an interface to various search capabilities and
>> although I can't speak to it as I don't use it, I see maildir
>> mentioned for one of the backends.  Perhaps that is enough to get you
>> started.
>
> Thank you for that tipp. I just looked at a few backends for nnir and
> found that the namazu backend provides similar if not the same
> capabilities as the command line utility mu.
>
> I will look into this and see if it meets my demands.

If you find a solution, I'd be interested in what you come up with. :-)

Regards, Frank

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Create a group from a list of message-ids

[Eric Abrahamsen]

Alexander Baier <lexi.baier@gmail.com> writes:

> Hello David,
>
> On 13-11-25 03:02 Dave Goldberg wrote:
>>> Hello,
>>> I use a tool called mu [1] to index and search my mail, which is stored
>>> in a Maildir folder. mu is able to give me the message ids of all the
>>> mails that match a given query. I want to use these ids to create a new
>>> group in gnus that shows the search result. So it basically boils down
>>> to this:
>>
>>> Is there a function or facility in gnus that takes a list of message-ids
>>> (or even a list of local path names) and creates a new group containing
>>> the corresponding messages?
>>
>>> I am fine with writing some elisp to call out to mu and converting its
>>> output to a list, I just need above mentioned functionality.
>>
>> nnir.el provides an interface to various search capabilities and
>> although I can't speak to it as I don't use it, I see maildir
>> mentioned for one of the backends.  Perhaps that is enough to get you
>> started.
>
> Thank you for that tipp. I just looked at a few backends for nnir and
> found that the namazu backend provides similar if not the same
> capabilities as the command line utility mu.
>
> I will look into this and see if it meets my demands.

If you're willing to consider switching indexing backends, I'd also
recommend notmuch, which I found more flexible and easier to use than
namazu or mairix.

Eric

Re: Create a group from a list of message-ids

[Alexander Baier]

Hello Frank,

On 13-11-25 09:58 Frank Terbeck wrote:
> Alexander Baier wrote:
>> On 13-11-25 03:02 Dave Goldberg wrote:
> [...]
>>> nnir.el provides an interface to various search capabilities and
>>> although I can't speak to it as I don't use it, I see maildir
>>> mentioned for one of the backends.  Perhaps that is enough to get you
>>> started.
>>
>> Thank you for that tipp. I just looked at a few backends for nnir and
>> found that the namazu backend provides similar if not the same
>> capabilities as the command line utility mu.
>>
>> I will look into this and see if it meets my demands.
>
> If you find a solution, I'd be interested in what you come up with. :-)
>
> Regards, Frank

I will follow up here, as soon as I configured a decent setup.

Regards,
--
Alex

Re: Create a group from a list of message-ids

[Alexander Baier]

Hello Eric,

On 13-11-25 10:01 Eric Abrahamsen wrote:
> Alexander Baier <lexi.baier@gmail.com> writes:
>
>> Hello David,
>>
>> On 13-11-25 03:02 Dave Goldberg wrote:
>>>> Hello,
>>>> I use a tool called mu [1] to index and search my mail, which is stored
>>>> in a Maildir folder. mu is able to give me the message ids of all the
>>>> mails that match a given query. I want to use these ids to create a new
>>>> group in gnus that shows the search result. So it basically boils down
>>>> to this:
>>>
>>>> Is there a function or facility in gnus that takes a list of message-ids
>>>> (or even a list of local path names) and creates a new group containing
>>>> the corresponding messages?
>>>
>>>> I am fine with writing some elisp to call out to mu and converting its
>>>> output to a list, I just need above mentioned functionality.
>>>
>>> nnir.el provides an interface to various search capabilities and
>>> although I can't speak to it as I don't use it, I see maildir
>>> mentioned for one of the backends.  Perhaps that is enough to get you
>>> started.
>>
>> Thank you for that tipp. I just looked at a few backends for nnir and
>> found that the namazu backend provides similar if not the same
>> capabilities as the command line utility mu.
>>
>> I will look into this and see if it meets my demands.
>
> If you're willing to consider switching indexing backends, I'd also
> recommend notmuch, which I found more flexible and easier to use than
> namazu or mairix.
>
> Eric

I did not yet give notmuch a look, could you point out what in your
opinion make notmuch more flexible than namazu?

Thanks,
--
Alex

Re: Create a group from a list of message-ids

[Pedro Silva]

[-- Attachment #1: Type: text/plain, Size: 3872 bytes --]

david.goldberg6@verizon.net (Dave Goldberg) writes:

>> Hello,
>> I use a tool called mu [1] to index and search my mail, which is stored
>> in a Maildir folder. mu is able to give me the message ids of all the
>> mails that match a given query. I want to use these ids to create a new
>> group in gnus that shows the search result. So it basically boils down
>> to this:
>
>> Is there a function or facility in gnus that takes a list of message-ids
>> (or even a list of local path names) and creates a new group containing
>> the corresponding messages?
>
>> I am fine with writing some elisp to call out to mu and converting its
>> output to a list, I just need above mentioned functionality.
>
> nnir.el provides an interface to various search capabilities and although I
> can't speak to it as I don't use it, I see maildir mentioned for one of the
> backends.  Perhaps that is enough to get you started.

I while back I wanted exactly this. It's a work in progress; to be frank
I don't remeber if I ever for it working, as I switched away from
maildir and consequently mu. Worth a shot!

--8<---------------cut here---------------start------------->8---
(require 'nnir)

(defcustom nnir-mu-program "mu"
  "*Name of mu indexing program."
  :type '(string)
  :group 'nnir)

(defcustom nnir-mu-additional-switches '()
  "*A list of strings, to be given as additional arguments to mu.

Note that this should be a list.  Ie, do NOT use the following:
    (setq nnir-mu-additional-switches \"-s date -z\") ; wrong
Instead, use this:
    (setq nnir-mu-additional-switches '(\"-s\" \"date\" \"-z\"))"
  :type '(repeat (string))
  :group 'nnir)

(defun nnir-run-mu (query server &optional group)
  "Run QUERY against mu.
Returns a vector of (group name, file name) pairs (also vectors,
actually)."
  (save-excursion
    (let ((qstring (cdr (assq 'query query)))
          (groupspec (car group))
          artlist)

      (when (equal "" qstring)
        (error "mu: You didn't enter anything"))

      (set-buffer (get-buffer-create nnir-tmp-buffer))
      (erase-buffer)

      (if groupspec
          (message "Doing mu query %s on %s..." qstring groupspec)
        (message "Doing mu query %s..." qstring))

      (let* ((cp-list `( ,nnir-mu-program
                         nil            ; input from /dev/null
                         t              ; output
                         nil            ; don't redisplay
                         "find"
                         "--format=sexp"
                         ,@(nnir-read-server-parm 'nnir-mu-additional-switches server)
                         ,@(split-string qstring "\\s-+")       ; the query, in mu format
                         ))
             (exitstatus
              (progn
                (message "%s args: %s" nnir-mu-program
                         (mapconcat 'identity (cddddr cp-list) " ")) ;; ???
                (apply 'call-process cp-list))))
        (unless (or (null exitstatus)
                    (zerop exitstatus))
          (nnheader-report 'nnir "Couldn't run mu: %s" exitstatus)
          ;; mu failure reason is in this buffer, show it if
          ;; the user wants it.
          (when (> gnus-verbose 6)
            (display-buffer nnir-tmp-buffer))))

      ;; The results are output in the format of:
      ;; plists
      (goto-char (point-min))
      (while (not (eobp))
        (ignore-errors
          (let* ((item (read (current-buffer)))
                 (group (concat server ":" (substring (plist-get item :maildir) 1)))
                 (artno (plist-get item :docid))
                 (score 1000))
            (push (vector group artno score) artlist))))

      (message "Massaging mu output...done")
      (print artlist)
      artlist)))

(add-to-list 'nnir-engines '(mu nnir-run-mu ()))
--8<---------------cut here---------------end--------------->8---
--
Pedro

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 800 bytes --]

Re: Create a group from a list of message-ids

[Alexander Baier]

Hello Pedro,

On 13-11-25 10:28 Pedro Silva wrote:
> david.goldberg6@verizon.net (Dave Goldberg) writes:
>
>>> Hello,
>>> I use a tool called mu [1] to index and search my mail, which is stored
>>> in a Maildir folder. mu is able to give me the message ids of all the
>>> mails that match a given query. I want to use these ids to create a new
>>> group in gnus that shows the search result. So it basically boils down
>>> to this:
>>
>>> Is there a function or facility in gnus that takes a list of message-ids
>>> (or even a list of local path names) and creates a new group containing
>>> the corresponding messages?
>>
>>> I am fine with writing some elisp to call out to mu and converting its
>>> output to a list, I just need above mentioned functionality.
>>
>> nnir.el provides an interface to various search capabilities and although I
>> can't speak to it as I don't use it, I see maildir mentioned for one of the
>> backends.  Perhaps that is enough to get you started.
>
> I while back I wanted exactly this. It's a work in progress; to be frank
> I don't remeber if I ever for it working, as I switched away from
> maildir and consequently mu. Worth a shot!
>

[...]

> --
> Pedro

This cerainly looks interesting. If I have time for this and I do not
find that any other backend has better things to offer than mu, I might
see if I get this working.

Regards,
--
Alex

Re: Create a group from a list of message-ids

[Eric Abrahamsen]

Alexander Baier <lexi.baier@gmail.com> writes:

> Hello Eric,
>
> On 13-11-25 10:01 Eric Abrahamsen wrote:
>> Alexander Baier <lexi.baier@gmail.com> writes:
>>
>>> Hello David,
>>>
>>> On 13-11-25 03:02 Dave Goldberg wrote:
>>>>> Hello,
>>>>> I use a tool called mu [1] to index and search my mail, which is stored
>>>>> in a Maildir folder. mu is able to give me the message ids of all the
>>>>> mails that match a given query. I want to use these ids to create a new
>>>>> group in gnus that shows the search result. So it basically boils down
>>>>> to this:
>>>>
>>>>> Is there a function or facility in gnus that takes a list of message-ids
>>>>> (or even a list of local path names) and creates a new group containing
>>>>> the corresponding messages?
>>>>
>>>>> I am fine with writing some elisp to call out to mu and converting its
>>>>> output to a list, I just need above mentioned functionality.
>>>>
>>>> nnir.el provides an interface to various search capabilities and
>>>> although I can't speak to it as I don't use it, I see maildir
>>>> mentioned for one of the backends.  Perhaps that is enough to get you
>>>> started.
>>>
>>> Thank you for that tipp. I just looked at a few backends for nnir and
>>> found that the namazu backend provides similar if not the same
>>> capabilities as the command line utility mu.
>>>
>>> I will look into this and see if it meets my demands.
>>
>> If you're willing to consider switching indexing backends, I'd also
>> recommend notmuch, which I found more flexible and easier to use than
>> namazu or mairix.
>>
>> Eric
>
> I did not yet give notmuch a look, could you point out what in your
> opinion make notmuch more flexible than namazu?

Hmm, I was afraid you'd ask that! :)

As far as I can recall, the interface to namazu felt very slow, and the
search syntax was much more awkward and limited than notmuch. Notmuch
just feels easier to use: it caches recent queries, lets you filter
previous queries more easily, and makes browsing the results quite easy
-- it actually acts as its own little MUA, with threading and
everything. (Actually that can be annoying when all you want is the
original message.) Mairix can't do unicode indexing, so that was right
out.

Test-installing notmuch isn't too onerous (though the initial indexing
is slow), it keeps all its business in one directory that you can nuke
if you don't like it. It also ships with all the necessary elisp files.
It's also under very active (borderline obsessive) development.

The notmuch developers paid me nothing for this message :)

Eric

Re: Create a group from a list of message-ids

[Alexander Baier]

On 13-11-25 14:20 Eric Abrahamsen wrote:

> Alexander Baier <lexi.baier@gmail.com> writes:

[...]

>>
>> I did not yet give notmuch a look, could you point out what in your
>> opinion make notmuch more flexible than namazu?
>
> Hmm, I was afraid you'd ask that! :)
>
> As far as I can recall, the interface to namazu felt very slow, and the
> search syntax was much more awkward and limited than notmuch. Notmuch
> just feels easier to use: it caches recent queries, lets you filter
> previous queries more easily, and makes browsing the results quite easy
> -- it actually acts as its own little MUA, with threading and
> everything. (Actually that can be annoying when all you want is the
> original message.) Mairix can't do unicode indexing, so that was right
> out.
>

This sounds very nice! You mention notmuch comes with its own
MUA. Although this might be usefull in a lot of situations, I am looking
for something that can integrate with gnus. This means I would like to
have something, that displays the search results as a group in gnus that
I can act upon like the other nnir backends. A search through the (gnus-)
manual didn't turn up anything about notmuch as a nnir backend. Is this
just something missing from the manual, or does a nnir notmuch backend
simply not exist?

If there is no notmuch backend, how do you integrate notmuch into to
gnus? Or do you just use it outside gnus?

>
> Test-installing notmuch isn't too onerous (though the initial indexing
> is slow), it keeps all its business in one directory that you can nuke
> if you don't like it. It also ships with all the necessary elisp files.
> It's also under very active (borderline obsessive) development.
>

An active project certainly is another nice point. All the other
alternatives seem rather inactive to me. (Well, except mu)

>
> The notmuch developers paid me nothing for this message :)
>

Well, even if they did, I am glad for every input I can get :)

>
> Eric

Regards,
-- 
Alexander Baier

Re: Create a group from a list of message-ids

[Eric Abrahamsen]

On 11/25/13 21:08 PM, Alexander Baier wrote:
> On 13-11-25 14:20 Eric Abrahamsen wrote:
>
>> Alexander Baier <lexi.baier@gmail.com> writes:
>
> [...]
>
>>>
>>> I did not yet give notmuch a look, could you point out what in your
>>> opinion make notmuch more flexible than namazu?
>>
>> Hmm, I was afraid you'd ask that! :)
>>
>> As far as I can recall, the interface to namazu felt very slow, and the
>> search syntax was much more awkward and limited than notmuch. Notmuch
>> just feels easier to use: it caches recent queries, lets you filter
>> previous queries more easily, and makes browsing the results quite easy
>> -- it actually acts as its own little MUA, with threading and
>> everything. (Actually that can be annoying when all you want is the
>> original message.) Mairix can't do unicode indexing, so that was right
>> out.
>>
>
> This sounds very nice! You mention notmuch comes with its own
> MUA. Although this might be usefull in a lot of situations, I am looking
> for something that can integrate with gnus. This means I would like to
> have something, that displays the search results as a group in gnus that
> I can act upon like the other nnir backends. A search through the (gnus-)
> manual didn't turn up anything about notmuch as a nnir backend. Is this
> just something missing from the manual, or does a nnir notmuch backend
> simply not exist?
>
> If there is no notmuch backend, how do you integrate notmuch into to
> gnus? Or do you just use it outside gnus?

It doesn't plug in to nnir, it has its own way of displaying search
results (hence my statement that it's a bit of a MUA unto itself). If
you're just looking to read/reply to messages, or save attachments, it
works great. If you were hoping to do something else with the message,
you'll want to use the "warp to original" snippet from here:

http://www.emacswiki.org/emacs/NotMuch

It also comes with its own tagging structure and all kinds of stuff I
don't use. The superfluity bugs me a little, but the searching is nice
enough that it doesn't matter. A nnir plugin for notmuch might actually
be nice.

Give it a shot!

>>
>> Test-installing notmuch isn't too onerous (though the initial indexing
>> is slow), it keeps all its business in one directory that you can nuke
>> if you don't like it. It also ships with all the necessary elisp files.
>> It's also under very active (borderline obsessive) development.
>>
>
> An active project certainly is another nice point. All the other
> alternatives seem rather inactive to me. (Well, except mu)
>
>>
>> The notmuch developers paid me nothing for this message :)
>>
>
> Well, even if they did, I am glad for every input I can get :)
>
>>
>> Eric
>
> Regards,
> -- 
> Alexander Baier

Re: Create a group from a list of message-ids

[Frank Terbeck]

Eric Abrahamsen wrote:
[...]
> It also comes with its own tagging structure and all kinds of stuff I
> don't use. The superfluity bugs me a little, but the searching is nice
> enough that it doesn't matter. A nnir plugin for notmuch might actually
> be nice.

Actually, at least in the git version of gnus, ‘nnir’ seems to feature
support for ‘notmuch’ (ie. there is a defcustom ‘nnir-notmuch-program’
and a defun ‘nnir-run-notmuch’).

Since ‘mu’ seems to work fairly similarly to notmuch, I would guess
that, this part would serve as a source for ideas for ‘mu’-integration.

Regards, Frank

Re: Create a group from a list of message-ids

[Alexander Baier]

On 13-11-25 16:26 Eric Abrahamsen wrote:
> On 11/25/13 21:08 PM, Alexander Baier wrote:

[...]

>> This sounds very nice! You mention notmuch comes with its own
>> MUA. Although this might be usefull in a lot of situations, I am looking
>> for something that can integrate with gnus. This means I would like to
>> have something, that displays the search results as a group in gnus that
>> I can act upon like the other nnir backends. A search through the (gnus-)
>> manual didn't turn up anything about notmuch as a nnir backend. Is this
>> just something missing from the manual, or does a nnir notmuch backend
>> simply not exist?
>>
>> If there is no notmuch backend, how do you integrate notmuch into to
>> gnus? Or do you just use it outside gnus?
>
> It doesn't plug in to nnir

Actually I think it might just do. I just looked through the code of
nnir.el and found an nnir implementation for notmuch. I did not yet have
time to try it out, but will do so as soon as I have some free time.

>, it has its own way of displaying search
> results (hence my statement that it's a bit of a MUA unto itself). If
> you're just looking to read/reply to messages, or save attachments, it
> works great. If you were hoping to do something else with the message,
> you'll want to use the "warp to original" snippet from here:
>
> http://www.emacswiki.org/emacs/NotMuch

I think this warp feature will actually be something that should meet
my most common needs. This will be a nice fallback if the nnir backend
turn out to be somewhat faulty.

> It also comes with its own tagging structure and all kinds of stuff I
> don't use. The superfluity bugs me a little, but the searching is nice
> enough that it doesn't matter. A nnir plugin for notmuch might actually
> be nice.
>
> Give it a shot!
>

[...]

Regards,
-- 
Alexander Baier

Re: Create a group from a list of message-ids

[Eric Abrahamsen]

Frank Terbeck <ft@bewatermyfriend.org> writes:

> Eric Abrahamsen wrote:
> [...]
>> It also comes with its own tagging structure and all kinds of stuff I
>> don't use. The superfluity bugs me a little, but the searching is nice
>> enough that it doesn't matter. A nnir plugin for notmuch might actually
>> be nice.
>
> Actually, at least in the git version of gnus, ‘nnir’ seems to feature
> support for ‘notmuch’ (ie. there is a defcustom ‘nnir-notmuch-program’
> and a defun ‘nnir-run-notmuch’).
>
> Since ‘mu’ seems to work fairly similarly to notmuch, I would guess
> that, this part would serve as a source for ideas for ‘mu’-integration.
>
> Regards, Frank

Thanks to both of you for pointing that out! I had no idea that was
there, but would definitely prefer using a more gnus-y interface. I set
it up like so:

(setq nnir-method-default-engines  '((nnimap . notmuch) (nntp . gmane)))

And gave it a shot.

It didn't work for me for two reasons. I have several different imap
servers, all run through dovecot with a maildir structure. They're all
under ~/.mail, like ~/.mail/acc1 ~/mail/acc2 etc. That means
nnir-notmuch-remove-prefix has to be set to an appropriate value for
each server, which you can do with server parameters. But that means you
can only search one server at a time (which I guess you'd do by putting
a notmuch-group key into the query? I don't even know how to do that).
That isn't really what I want -- I'd like to search all my mail at once.
And even if I had to do it server by server, it seems like
`nnir-run-notmuch' should be able to figure out how to filter to that
server by itself...

The second problem was the maildir structure. I specify maildir through
the nnimap-shell-program parameter, which looks like:

/usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail/acc1/:LAYOUT=fs

`nnir-compose-result' munges results differently depending on whether
they're maildir or not, but it only determines that by checking
(gnus-group-server server) for the nnmaildir string. I'd need to change
that, I guess by putting more thorough checking in `nnir-run-notmuch',
`nnir-run-namazu', and `nnir-run-swish++', which would then pass a flag
to `nnir-compose-results'.

Lastly, a notmuch result in my case looks like this:

/home/eric/.mail/pr/[Gmail]/Sent Mail/cur/1384510308.M437592P10993.pellet,S=3672,W=3728:2,S

Which would, if the above problems were fixed, turn into a result that looks like
["nnimap+PR:[Gmail].Sent Mail" "1384510308.M437592P10993.pellet"]. The problem
there is that the slash between [Gmail] and Sent Mail (which really is
a filesystem separator, because I've used LAYOUT=fs) gets turned into a
dot, when it's actually supposed to remain a slash. That's probably my
fault for using LAYOUT=fs, and it would be ugly to put a special-case
check for that in nnir.el

Ahem, so... Please consider that mostly notes to self. If you're not
using dovecot with a maildir layout, this probably won't be an issue.

I'd be happy to provide a patch for more thorough checking for maildir
structure, but I probably won't use this anyway, because I don't want to
be restricted to searching one server at a time.

All for now,
Eric

Re: Create a group from a list of message-ids

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Frank Terbeck <ft@bewatermyfriend.org> writes:

[...]

>> Actually, at least in the git version of gnus, ‘nnir’ seems to feature
>> support for ‘notmuch’ (ie. there is a defcustom ‘nnir-notmuch-program’
>> and a defun ‘nnir-run-notmuch’).

[...]

> Thanks to both of you for pointing that out! I had no idea that was
> there, but would definitely prefer using a more gnus-y interface. I set
> it up like so:
>
> (setq nnir-method-default-engines  '((nnimap . notmuch) (nntp . gmane)))
>
> And gave it a shot.

Bleagh, I'm giving up on this. I re-syncd some of my accounts via
dovecot, without the LAYOUT=fs flag, because that seemed to be causing
me more problems elsewhere, with no discernible benefit.

Now all my groups are prefixed with a ".", and there is no INBOX in the
filesystem at all -- the messages that appear to be in the INBOX group
from within gnus are located in the top-level "cur/" directory for the
account. So any messages found from that directory break nnir, because
there's no group name at all.

Back to the notmuch interface...

E

Re: Create a group from a list of message-ids

[Alexander Baier]

On 13-11-27 06:16 Eric Abrahamsen wrote:
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Frank Terbeck <ft@bewatermyfriend.org> writes:
>
> [...]
>
>>> Actually, at least in the git version of gnus, ‘nnir’ seems to feature
>>> support for ‘notmuch’ (ie. there is a defcustom ‘nnir-notmuch-program’
>>> and a defun ‘nnir-run-notmuch’).
>
> [...]
>
>> Thanks to both of you for pointing that out! I had no idea that was
>> there, but would definitely prefer using a more gnus-y interface. I set
>> it up like so:
>>
>> (setq nnir-method-default-engines  '((nnimap . notmuch) (nntp . gmane)))
>>
>> And gave it a shot.
>
> Bleagh, I'm giving up on this. I re-syncd some of my accounts via
> dovecot, without the LAYOUT=fs flag, because that seemed to be causing
> me more problems elsewhere, with no discernible benefit.
>
> Now all my groups are prefixed with a ".", and there is no INBOX in the
> filesystem at all -- the messages that appear to be in the INBOX group
> from within gnus are located in the top-level "cur/" directory for the
> account. So any messages found from that directory break nnir, because
> there's no group name at all.
>
> Back to the notmuch interface...
>
> E

I tried to get nnir and notmuch working last night, but didn't get
anywhere either.  Issuing a search query via "G G" gives me following
error:

"nnir-retrieve-headers: Unknown header type nil while requesting articles
of group nnmaildir+gmail:gmail.Inbox"


As it turns out my setup is quite similar to yours in some aspekts. I
also habe several accounts under one maildir or whatever this
"top-maildir" is called. Like this: mail/acc1, mail/acc2, ... .  The
gmail account also has folders with dots in its names.

For now I try to go for the notmuch interface as you do, as this seems
simpler to me.  But this actually poses me with a different problem:
Doing a (require 'notmuch) in my initfile seems to somehow invalidate my
posting styles.  Now, upon composing new mails or news, I always get the
same content in the message-buffer (that is the buffer in message-mode I
write my mail in) that is not related to anything in my
`gnus-posting-styles'. It adds something like Gcc: mymailbox/sent to the
headers, which is far from anything I configured in
`gnus-posting-styles'.
Did you experience something similar with your setup?

Regards,
-- 
 Alexander Baier

Re: Create a group from a list of message-ids

[Eric Abrahamsen]

Alexander Baier <lexi.baier@gmail.com> writes:

> On 13-11-27 06:16 Eric Abrahamsen wrote:
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>
>>> Frank Terbeck <ft@bewatermyfriend.org> writes:
>>
>> [...]
>>
>>>> Actually, at least in the git version of gnus, ‘nnir’ seems to feature
>>>> support for ‘notmuch’ (ie. there is a defcustom ‘nnir-notmuch-program’
>>>> and a defun ‘nnir-run-notmuch’).
>>
>> [...]
>>
>>> Thanks to both of you for pointing that out! I had no idea that was
>>> there, but would definitely prefer using a more gnus-y interface. I set
>>> it up like so:
>>>
>>> (setq nnir-method-default-engines  '((nnimap . notmuch) (nntp . gmane)))
>>>
>>> And gave it a shot.
>>
>> Bleagh, I'm giving up on this. I re-syncd some of my accounts via
>> dovecot, without the LAYOUT=fs flag, because that seemed to be causing
>> me more problems elsewhere, with no discernible benefit.
>>
>> Now all my groups are prefixed with a ".", and there is no INBOX in the
>> filesystem at all -- the messages that appear to be in the INBOX group
>> from within gnus are located in the top-level "cur/" directory for the
>> account. So any messages found from that directory break nnir, because
>> there's no group name at all.
>>
>> Back to the notmuch interface...
>>
>> E
>
> I tried to get nnir and notmuch working last night, but didn't get
> anywhere either.  Issuing a search query via "G G" gives me following
> error:
>
> "nnir-retrieve-headers: Unknown header type nil while requesting articles
> of group nnmaildir+gmail:gmail.Inbox"
>
>
> As it turns out my setup is quite similar to yours in some aspekts. I
> also habe several accounts under one maildir or whatever this
> "top-maildir" is called. Like this: mail/acc1, mail/acc2, ... .  The
> gmail account also has folders with dots in its names.
>
> For now I try to go for the notmuch interface as you do, as this seems
> simpler to me.  But this actually poses me with a different problem:
> Doing a (require 'notmuch) in my initfile seems to somehow invalidate my
> posting styles.  Now, upon composing new mails or news, I always get the
> same content in the message-buffer (that is the buffer in message-mode I
> write my mail in) that is not related to anything in my
> `gnus-posting-styles'. It adds something like Gcc: mymailbox/sent to the
> headers, which is far from anything I configured in
> `gnus-posting-styles'.
> Did you experience something similar with your setup?

The Gcc thing is something notmuch does automatically, and you can stop
it with:

(setq notmuch-fcc-dirs nil)

I haven't seen any other interference with posting styles, and I require
notmuch before setting gnus-posting-styles. Are you composing mails by
using "m" in gnus, or "C-x m" elsewhere? I just went and checked, and
group-specific posting styles still work for me, when I have point on a
group and hit "C-u m"...

Re: Create a group from a list of message-ids

[Alexander Baier]

On 13-11-27 09:10 Eric Abrahamsen wrote:
> Alexander Baier <lexi.baier@gmail.com> writes:

[...]

>> I tried to get nnir and notmuch working last night, but didn't get
>> anywhere either.  Issuing a search query via "G G" gives me following
>> error:
>>
>> "nnir-retrieve-headers: Unknown header type nil while requesting articles
>> of group nnmaildir+gmail:gmail.Inbox"
>>
>>
>> As it turns out my setup is quite similar to yours in some aspekts. I
>> also habe several accounts under one maildir or whatever this
>> "top-maildir" is called. Like this: mail/acc1, mail/acc2, ... .  The
>> gmail account also has folders with dots in its names.
>>
>> For now I try to go for the notmuch interface as you do, as this seems
>> simpler to me.  But this actually poses me with a different problem:
>> Doing a (require 'notmuch) in my initfile seems to somehow invalidate my
>> posting styles.  Now, upon composing new mails or news, I always get the
>> same content in the message-buffer (that is the buffer in message-mode I
>> write my mail in) that is not related to anything in my
>> `gnus-posting-styles'. It adds something like Gcc: mymailbox/sent to the
>> headers, which is far from anything I configured in
>> `gnus-posting-styles'.
>> Did you experience something similar with your setup?
>
> The Gcc thing is something notmuch does automatically, and you can stop
> it with:
>
> (setq notmuch-fcc-dirs nil)
>
> I haven't seen any other interference with posting styles, and I require
> notmuch before setting gnus-posting-styles. Are you composing mails by
> using "m" in gnus, or "C-x m" elsewhere? I just went and checked, and
> group-specific posting styles still work for me, when I have point on a
> group and hit "C-u m"...
>

Jepp, setting notmuch-fcc-dirs to nil solves this!

Thanks,
-- 
 Alexander Baier

Re: Create a group from a list of message-ids

[Alexander Baier]

On 13-11-26 05:06 Eric Abrahamsen wrote:
> Frank Terbeck <ft@bewatermyfriend.org> writes:
>
>> Eric Abrahamsen wrote:
>> [...]
>>> It also comes with its own tagging structure and all kinds of stuff I
>>> don't use. The superfluity bugs me a little, but the searching is nice
>>> enough that it doesn't matter. A nnir plugin for notmuch might actually
>>> be nice.
>>
>> Actually, at least in the git version of gnus, ‘nnir’ seems to feature
>> support for ‘notmuch’ (ie. there is a defcustom ‘nnir-notmuch-program’
>> and a defun ‘nnir-run-notmuch’).
>>
>> Since ‘mu’ seems to work fairly similarly to notmuch, I would guess
>> that, this part would serve as a source for ideas for ‘mu’-integration.
>>
>> Regards, Frank
>
> Thanks to both of you for pointing that out! I had no idea that was
> there, but would definitely prefer using a more gnus-y interface. I set
> it up like so:
>
> (setq nnir-method-default-engines  '((nnimap . notmuch) (nntp . gmane)))
>
> And gave it a shot.
>
> It didn't work for me for two reasons. I have several different imap
> servers, all run through dovecot with a maildir structure. They're all
> under ~/.mail, like ~/.mail/acc1 ~/mail/acc2 etc. That means
> nnir-notmuch-remove-prefix has to be set to an appropriate value for
> each server, which you can do with server parameters. But that means you
> can only search one server at a time (which I guess you'd do by putting
> a notmuch-group key into the query? I don't even know how to do that).
> That isn't really what I want -- I'd like to search all my mail at once.
> And even if I had to do it server by server, it seems like
> `nnir-run-notmuch' should be able to figure out how to filter to that
> server by itself...
>
> The second problem was the maildir structure. I specify maildir through
> the nnimap-shell-program parameter, which looks like:
>
> /usr/lib/dovecot/imap -o mail_location=maildir:$HOME/.mail/acc1/:LAYOUT=fs
>
> `nnir-compose-result' munges results differently depending on whether
> they're maildir or not, but it only determines that by checking
> (gnus-group-server server) for the nnmaildir string. I'd need to change
> that, I guess by putting more thorough checking in `nnir-run-notmuch',
> `nnir-run-namazu', and `nnir-run-swish++', which would then pass a flag
> to `nnir-compose-results'.
>
> Lastly, a notmuch result in my case looks like this:
>
> /home/eric/.mail/pr/[Gmail]/Sent Mail/cur/1384510308.M437592P10993.pellet,S=3672,W=3728:2,S
>
> Which would, if the above problems were fixed, turn into a result that looks like
> ["nnimap+PR:[Gmail].Sent Mail" "1384510308.M437592P10993.pellet"]. The problem
> there is that the slash between [Gmail] and Sent Mail (which really is
> a filesystem separator, because I've used LAYOUT=fs) gets turned into a
> dot, when it's actually supposed to remain a slash. That's probably my
> fault for using LAYOUT=fs, and it would be ugly to put a special-case
> check for that in nnir.el
>
> Ahem, so... Please consider that mostly notes to self. If you're not
> using dovecot with a maildir layout, this probably won't be an issue.
>
> I'd be happy to provide a patch for more thorough checking for maildir
> structure, but I probably won't use this anyway, because I don't want to
> be restricted to searching one server at a time.
>
> All for now,
> Eric
>
Hi Eric,

I am digging this up again, as I had some time to look at this myself
and maybe have an idea of how to tackle this problem.

Now, as I see it, the thing preventing us from searching multiple
servers is the fact that nnir-compose-result might be passed a server
that does not fit to the matched article as this may come from a
different server.  So I think, solving this problem is "just" a matter
of teaching nnir-compose-result to correct the server it is passed in
case it does not fit to the matched article.  For nnir-compose-result to
be able to so we need to tell it how to extract the server from the
dirnam parameter.  The question is, how do we tell nnir-compose-result
this?  Do you have any ideas on this?

Regards,
-- 
 Alexander Baier

PS: After writing this I am not exactly sure if I am just rehashing your
point(s), in case that is true I apologize.

Re: Create a group from a list of message-ids

[Eric Abrahamsen]

Alexander Baier <lexi.baier@gmail.com> writes:

> On 13-11-26 05:06 Eric Abrahamsen wrote:
>> Frank Terbeck <ft@bewatermyfriend.org> writes:
>>
>>> Eric Abrahamsen wrote:
>>> [...]
>>>> It also comes with its own tagging structure and all kinds of stuff I
>>>> don't use. The superfluity bugs me a little, but the searching is nice
>>>> enough that it doesn't matter. A nnir plugin for notmuch might actually
>>>> be nice.
>>>
>>> Actually, at least in the git version of gnus, ‘nnir’ seems to feature
>>> support for ‘notmuch’ (ie. there is a defcustom ‘nnir-notmuch-program’
>>> and a defun ‘nnir-run-notmuch’).
>>>
>>> Since ‘mu’ seems to work fairly similarly to notmuch, I would guess
>>> that, this part would serve as a source for ideas for ‘mu’-integration.
>>>
>>> Regards, Frank
>>
>> Thanks to both of you for pointing that out! I had no idea that was
>> there, but would definitely prefer using a more gnus-y interface. I set
>> it up like so:
>>
>> (setq nnir-method-default-engines  '((nnimap . notmuch) (nntp . gmane)))
>>
>> And gave it a shot.
>>
>> It didn't work for me for two reasons. I have several different imap
>> servers, all run through dovecot with a maildir structure. They're all
>> under ~/.mail, like ~/.mail/acc1 ~/mail/acc2 etc. That means
>> nnir-notmuch-remove-prefix has to be set to an appropriate value for
>> each server, which you can do with server parameters. But that means you
>> can only search one server at a time (which I guess you'd do by putting
>> a notmuch-group key into the query? I don't even know how to do that).
>> That isn't really what I want -- I'd like to search all my mail at once.
>> And even if I had to do it server by server, it seems like
>> `nnir-run-notmuch' should be able to figure out how to filter to that
>> server by itself...

[...]

> Hi Eric,
>
> I am digging this up again, as I had some time to look at this myself
> and maybe have an idea of how to tackle this problem.
>
> Now, as I see it, the thing preventing us from searching multiple
> servers is the fact that nnir-compose-result might be passed a server
> that does not fit to the matched article as this may come from a
> different server.  So I think, solving this problem is "just" a matter
> of teaching nnir-compose-result to correct the server it is passed in
> case it does not fit to the matched article.  For nnir-compose-result to
> be able to so we need to tell it how to extract the server from the
> dirnam parameter.  The question is, how do we tell nnir-compose-result
> this?  Do you have any ideas on this?

Right now, servers to search are chosen with
`gnus-group-make-nnir-group', which does one of three things:

1. Extracts the server from the group under point
2. Extracts server from the marked groups, if groups are marked
3. If we're in the *Server* buffer, use the server under point

Options one and three obviously only provide a single server, option two
results in something like this (I marked the INBOXs of three different accounts):

(("nnimap:acc1" ("nnimap+acc1:INBOX"))
 ("nnimap:acc2" ("nnimap+acc2:INBOX"))
 ("nnimap:acc3" ("nnimap+acc3:INBOX")))

What eventually happens is that `nnir-run-query' loops
over each server in the above sexp (ignoring the group), and calls
`nnir-run-notmuch' once for each server. Each loop passes the same
search results to `nnir-compose-results', which filters the results
based on the current server.

So theoretically if you'd marked one group from each of the accounts you
wanted to search, you'd get full results. We should be able to do the
same thing by wrapping `gnus-group-make-nnir-group' and force-feeding it
arguments:

(setq notmuch-servers-to-search '(("nnimap:acc1")
                                  ("nnimap:acc2")
                                  ("nnimap:acc3")))

(defun nnir-search-all-servers (&optional query-spec)
  (interactive)
  (setq query-spec (or query-spec
		       (list (cons 'query
			      (read-string "Query: " nil 'nnir-search-history)))))
  (gnus-group-make-nnir-group
   nil (list
	(cons 'nnir-query-spec query-spec)
	(cons 'nnir-group-spec notmuch-servers-to-search))))


Can you set notmuch-servers-to-search appropriately, and test that? It's
hard for me to test as I'm still having the "is it maildir or isn't it"
problem, as well as the "I don't have an INBOX" problem.

Thanks,
Eric

Re: Create a group from a list of message-ids

[Alexander Baier]

On 13-11-29 04:41 Eric Abrahamsen wrote:
> Alexander Baier <lexi.baier@gmail.com> writes:
>
>> On 13-11-26 05:06 Eric Abrahamsen wrote:
>>> Frank Terbeck <ft@bewatermyfriend.org> writes:
>>>
>>>> Eric Abrahamsen wrote:
>>>> [...]
>>>>> It also comes with its own tagging structure and all kinds of stuff I
>>>>> don't use. The superfluity bugs me a little, but the searching is nice
>>>>> enough that it doesn't matter. A nnir plugin for notmuch might actually
>>>>> be nice.
>>>>
>>>> Actually, at least in the git version of gnus, ‘nnir’ seems to feature
>>>> support for ‘notmuch’ (ie. there is a defcustom ‘nnir-notmuch-program’
>>>> and a defun ‘nnir-run-notmuch’).
>>>>
>>>> Since ‘mu’ seems to work fairly similarly to notmuch, I would guess
>>>> that, this part would serve as a source for ideas for ‘mu’-integration.
>>>>
>>>> Regards, Frank
>>>
>>> Thanks to both of you for pointing that out! I had no idea that was
>>> there, but would definitely prefer using a more gnus-y interface. I set
>>> it up like so:
>>>
>>> (setq nnir-method-default-engines  '((nnimap . notmuch) (nntp . gmane)))
>>>
>>> And gave it a shot.
>>>
>>> It didn't work for me for two reasons. I have several different imap
>>> servers, all run through dovecot with a maildir structure. They're all
>>> under ~/.mail, like ~/.mail/acc1 ~/mail/acc2 etc. That means
>>> nnir-notmuch-remove-prefix has to be set to an appropriate value for
>>> each server, which you can do with server parameters. But that means you
>>> can only search one server at a time (which I guess you'd do by putting
>>> a notmuch-group key into the query? I don't even know how to do that).
>>> That isn't really what I want -- I'd like to search all my mail at once.
>>> And even if I had to do it server by server, it seems like
>>> `nnir-run-notmuch' should be able to figure out how to filter to that
>>> server by itself...
>
> [...]
>
>> Hi Eric,
>>
>> I am digging this up again, as I had some time to look at this myself
>> and maybe have an idea of how to tackle this problem.
>>
>> Now, as I see it, the thing preventing us from searching multiple
>> servers is the fact that nnir-compose-result might be passed a server
>> that does not fit to the matched article as this may come from a
>> different server.  So I think, solving this problem is "just" a matter
>> of teaching nnir-compose-result to correct the server it is passed in
>> case it does not fit to the matched article.  For nnir-compose-result to
>> be able to so we need to tell it how to extract the server from the
>> dirnam parameter.  The question is, how do we tell nnir-compose-result
>> this?  Do you have any ideas on this?
>
> Right now, servers to search are chosen with
> `gnus-group-make-nnir-group', which does one of three things:
>
> 1. Extracts the server from the group under point
> 2. Extracts server from the marked groups, if groups are marked
> 3. If we're in the *Server* buffer, use the server under point
>
> Options one and three obviously only provide a single server, option two
> results in something like this (I marked the INBOXs of three different accounts):
>
> (("nnimap:acc1" ("nnimap+acc1:INBOX"))
>  ("nnimap:acc2" ("nnimap+acc2:INBOX"))
>  ("nnimap:acc3" ("nnimap+acc3:INBOX")))
>
> What eventually happens is that `nnir-run-query' loops
> over each server in the above sexp (ignoring the group), and calls
> `nnir-run-notmuch' once for each server. Each loop passes the same
> search results to `nnir-compose-results', which filters the results
> based on the current server.
>
> So theoretically if you'd marked one group from each of the accounts you
> wanted to search, you'd get full results. We should be able to do the
> same thing by wrapping `gnus-group-make-nnir-group' and force-feeding it
> arguments:
>
> (setq notmuch-servers-to-search '(("nnimap:acc1")
>                                   ("nnimap:acc2")
>                                   ("nnimap:acc3")))
>
> (defun nnir-search-all-servers (&optional query-spec)
>   (interactive)
>   (setq query-spec (or query-spec
> 		       (list (cons 'query
> 			      (read-string "Query: " nil 'nnir-search-history)))))
>   (gnus-group-make-nnir-group
>    nil (list
> 	(cons 'nnir-query-spec query-spec)
> 	(cons 'nnir-group-spec notmuch-servers-to-search))))
>
>
> Can you set notmuch-servers-to-search appropriately, and test that? It's
> hard for me to test as I'm still having the "is it maildir or isn't it"
> problem, as well as the "I don't have an INBOX" problem.
>
> Thanks,
> Eric
>
>
>
Hi Eric,

I tested your code and it seems to do exactly what you are describing.
There is, however, another problem I am facing, that produces a nil as
article number in some or all of the results (vectors with 3 elements)
returned by nnir-compose-result.  I have not figured out how to solve
this but will follow up as soon as I find something.

So long,
-- 
 Alexander Baier