gnus nnmaildir and notmuch

gnus nnmaildir and notmuch

[Erik Colson]

Hi,

I've setup gnus with nnmaildir as a secondary mail source and using
notmuch as indexer/search.
Searching with the notmuch nnir backend works very well, and I nicely
get a gnus summary buffer of the corresponding results. In that buffer I
can select and move messages to other maildir folders and the articles
are then effectively moved. I can also mark messages flagged or
expirable etc, but when I exit the buffer and relaunch the query, the
marks are _not_ saved.

How come? Is this expected behavior ? Can it be changed ?

thx
-- 
erik colson

Re: gnus nnmaildir and notmuch

[Eric Abrahamsen]

Erik Colson <eco@ecocode.net> writes:

> Hi,
>
> I've setup gnus with nnmaildir as a secondary mail source and using
> notmuch as indexer/search.
> Searching with the notmuch nnir backend works very well, and I nicely
> get a gnus summary buffer of the corresponding results. In that buffer I
> can select and move messages to other maildir folders and the articles
> are then effectively moved. I can also mark messages flagged or
> expirable etc, but when I exit the buffer and relaunch the query, the
> marks are _not_ saved.
>
> How come? Is this expected behavior ? Can it be changed ?

nnir sort of caches its results -- if the group hasn't been killed
between searches, you'll just see the same results again. nnir doesn't
support re-scanning, otherwise it seems like it would be reasonable to
use M-g in an nnir group to refresh the results. I don't know if it's
possible to implement `nnir-request-scan', or if it simply won't work
for any ephemeral groups.

To be honest, I just did a little testing, and I couldn't even get marks
to propagate: I flagged a message (the "real" backend was nnimap),
exited the group, and the flag wasn't there in the real group, nor in
the nnir group when I re-ran the search.

Sorry I don't have anything more helpful!

Eric

Re: gnus nnmaildir and notmuch

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Erik Colson <eco@ecocode.net> writes:
>
>> Hi,
>>
>> I've setup gnus with nnmaildir as a secondary mail source and using
>> notmuch as indexer/search.
>> Searching with the notmuch nnir backend works very well, and I nicely
>> get a gnus summary buffer of the corresponding results. In that buffer I
>> can select and move messages to other maildir folders and the articles
>> are then effectively moved. I can also mark messages flagged or
>> expirable etc, but when I exit the buffer and relaunch the query, the
>> marks are _not_ saved.
>>
>> How come? Is this expected behavior ? Can it be changed ?
>
> nnir sort of caches its results -- if the group hasn't been killed
> between searches, you'll just see the same results again. nnir doesn't
> support re-scanning, otherwise it seems like it would be reasonable to
> use M-g in an nnir group to refresh the results. I don't know if it's
> possible to implement `nnir-request-scan', or if it simply won't work
> for any ephemeral groups.
>
> To be honest, I just did a little testing, and I couldn't even get marks
> to propagate: I flagged a message (the "real" backend was nnimap),
> exited the group, and the flag wasn't there in the real group, nor in
> the nnir group when I re-ran the search.

Oh, that's because there's no nnimap-request-update-mark deffoo. There
is nnimap-request-set-mark, though -- I don't know why nnir doesn't use
that.

Re: gnus nnmaildir and notmuch

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Erik Colson <eco@ecocode.net> writes:
>>
>>> Hi,
>>>
>>> I've setup gnus with nnmaildir as a secondary mail source and using
>>> notmuch as indexer/search.
>>> Searching with the notmuch nnir backend works very well, and I nicely
>>> get a gnus summary buffer of the corresponding results. In that buffer I
>>> can select and move messages to other maildir folders and the articles
>>> are then effectively moved. I can also mark messages flagged or
>>> expirable etc, but when I exit the buffer and relaunch the query, the
>>> marks are _not_ saved.
>>>
>>> How come? Is this expected behavior ? Can it be changed ?
>>
>> nnir sort of caches its results -- if the group hasn't been killed
>> between searches, you'll just see the same results again. nnir doesn't
>> support re-scanning, otherwise it seems like it would be reasonable to
>> use M-g in an nnir group to refresh the results. I don't know if it's
>> possible to implement `nnir-request-scan', or if it simply won't work
>> for any ephemeral groups.
>>
>> To be honest, I just did a little testing, and I couldn't even get marks
>> to propagate: I flagged a message (the "real" backend was nnimap),
>> exited the group, and the flag wasn't there in the real group, nor in
>> the nnir group when I re-ran the search.
>
> Oh, that's because there's no nnimap-request-update-mark deffoo. There
> is nnimap-request-set-mark, though -- I don't know why nnir doesn't use
> that.

Bah, that's wrong as well. Never mind, rabbit hole is too deep, and this
isn't pertinent to the original issue.

Re: gnus nnmaildir and notmuch

[Eric Abrahamsen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Thursday, 23 Mar 2017 at 15:25, Eric Abrahamsen wrote:
>
> [...]
>
>> To be honest, I just did a little testing, and I couldn't even get marks
>> to propagate: I flagged a message (the "real" backend was nnimap),
>> exited the group, and the flag wasn't there in the real group, nor in
>> the nnir group when I re-ran the search.
>
> I tend to warp to the article in the original group (A W) which then
> allows one to do whatever you want with it.  Not ideal, mind you.

Especially not ideal when nnir is set up to do mark propagation, and
there's no reason (that I can think of) why nnimap would want to be
exempt from that.

Ten minutes of edebugging was inconclusive; I'll try to find more time
to mess with it.

Ideally we could find some sort of solution to the OP's problem as well:
it seems like nnir groups are too ephemeral to allow you to update marks
from the real backend(s), but not ephemeral enough to be recreated from
scratch when the search is run a second time.

Eric

Re: gnus nnmaildir and notmuch

[Andrew Cohen]

I'm responsible for the code but haven't had time to work on gnus for
several years. Updating of marks is supposed to work---indeed, it works
fine for me with imap. I haven't used notmuch but the marks code
shouldn't depend on that.

One reason I can't say something more useful is I am using a version of
nnir that has updates that I never got around to pushing. Its possible
that I fixed some bugs in the marks code and never pushed it. If I can
find some time to poke around I will.

Best,
Andy

Re: gnus nnmaildir and notmuch

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

> I'm responsible for the code but haven't had time to work on gnus for
> several years. Updating of marks is supposed to work---indeed, it works
> fine for me with imap. I haven't used notmuch but the marks code
> shouldn't depend on that.

I assume it won't be hard to get it working again (indeed, I'm not sure
the problem isn't mine). Probably just some code drift.

> One reason I can't say something more useful is I am using a version of
> nnir that has updates that I never got around to pushing. Its possible
> that I fixed some bugs in the marks code and never pushed it. If I can
> find some time to poke around I will.

I think you might have sent me something at some point -- what do you
think of pushing a scratch branch to the Emacs repo? That way it
wouldn't have to be polished, and more people could get a look at it.

E

Re: gnus nnmaildir and notmuch

[Erik Colson]

Andrew Cohen <cohen@bu.edu> writes:

> I'm responsible for the code but haven't had time to work on gnus for
> several years. Updating of marks is supposed to work---indeed, it works
> fine for me with imap. I haven't used notmuch but the marks code
> shouldn't depend on that.

Hi Andrew,

I think nnir flag support does work with nnimap when using imap as
search engine. In my particular case I'm using nnmaildir with notmuch
search engine. I'll try with nnimap and notmuch search engine, but that
configuration will break my other imap account so I have to be really
careful setting this up.

At this point, I guess the problem is probably not related to
notmuch. More precisely I think the cause is the nnmaildir backend. Does
this make sense ?

best
-- 
erik colson

Re: gnus nnmaildir and notmuch

[Erik Colson]

Erik Colson <eco@ecocode.net> writes:

> I think nnir flag support does work with nnimap when using imap as
> search engine.

Hi

Ignore this. flags are not committed when using nnimap backend with imap
search engine. So this is more of a enhancement request for nnir.

best
-- 
erik colson

Re: gnus nnmaildir and notmuch

[Erik Colson]

Erik Colson <eco@ecocode.net> writes:

> So this is more of a enhancement request for nnir.

And I should definitely RTFM

"The nnir group made in this way is an ephemeral group, and some changes
are not permanent: aside from reading, moving, and deleting, you can’t
act on the original article. But there is an alternative: you can warp
(i.e., jump) to the original group for the article on the current line
with A W, aka gnus-warp-to-article. Even better, the function
gnus-summary-refer-thread, bound by default in summary buffers to A T,
will first warp to the original group before it works its magic and
includes all the articles in the thread. From here you can read, move
and delete articles, but also copy them, alter article marks,
whatever. Go nuts."

It kinda makes the notmuch search engine loose its purpose, but at least
it is documented :)

best
-- 
erik colson

Re: gnus nnmaildir and notmuch

[Eric Abrahamsen]

Erik Colson <eco@ecocode.net> writes:

> Erik Colson <eco@ecocode.net> writes:
>
>> So this is more of a enhancement request for nnir.
>
> And I should definitely RTFM
>
> "The nnir group made in this way is an ephemeral group, and some changes
> are not permanent: aside from reading, moving, and deleting, you can’t
> act on the original article. But there is an alternative: you can warp
> (i.e., jump) to the original group for the article on the current line
> with A W, aka gnus-warp-to-article. Even better, the function
> gnus-summary-refer-thread, bound by default in summary buffers to A T,
> will first warp to the original group before it works its magic and
> includes all the articles in the thread. From here you can read, move
> and delete articles, but also copy them, alter article marks,
> whatever. Go nuts."
>
> It kinda makes the notmuch search engine loose its purpose, but at least
> it is documented :)

No, I think this is supposed to work -- at any rate, there is code in
nnir for handling this. I'll try to take a look later.

Actually there are two issues: do the marks propagate to the backend,
and when they do, does the nnir group update its display correctly? My
understanding had been that you were having the latter problem, but
perhaps it's all connected.

Eric

Re: gnus nnmaildir and notmuch

[Eric Abrahamsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Erik Colson <eco@ecocode.net> writes:
>
>> Erik Colson <eco@ecocode.net> writes:
>>
>>> So this is more of a enhancement request for nnir.
>>
>> And I should definitely RTFM
>>
>> "The nnir group made in this way is an ephemeral group, and some changes
>> are not permanent: aside from reading, moving, and deleting, you can’t
>> act on the original article. But there is an alternative: you can warp
>> (i.e., jump) to the original group for the article on the current line
>> with A W, aka gnus-warp-to-article. Even better, the function
>> gnus-summary-refer-thread, bound by default in summary buffers to A T,
>> will first warp to the original group before it works its magic and
>> includes all the articles in the thread. From here you can read, move
>> and delete articles, but also copy them, alter article marks,
>> whatever. Go nuts."
>>
>> It kinda makes the notmuch search engine loose its purpose, but at least
>> it is documented :)
>
> No, I think this is supposed to work -- at any rate, there is code in
> nnir for handling this. I'll try to take a look later.

Okay, I looked at it, and the basic problem is that marks from ephemeral
groups are explicitly not propagated.

The practical definition of an ephemeral group is, "does it have a
quit-config parameter", ie, will it be made to disappear somehow when
the user leaves the summary. All `gnus-group-ephemeral-p' does is checks
if a quit-config parameter is present.

`gnus-summary-exit' is responsible for making marks permanent. It checks
for a quit-config parameter, and if it's present prevents a whole lot of
stuff from happening -- including making the marks permanent. I opened
an nnir group and manually killed the quit-config parameter, and can
confirm marks were propagated when I exited the group.

So that's where we are. nnir groups are supposed to be ephemeral, but
that means marks won't be propagated. So long as
`gnus-group-make-nnir-group' uses `gnus-group-read-ephemeral-group', we
can't avoid this, as the latter function unconditionally installs a
quit-config parameter.

Andy, I wonder if any of your local changes affect this behavior?

Thanks,
Eric

Re: gnus nnmaildir and notmuch

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:


[...]

    Eric> Andy, I wonder if any of your local changes affect this
    Eric> behavior?

Sorry, I sent a reply to the other Erik but didn't cc the list.


The behavior is intentional. As you learned, ephemeral groups explicitly
do not propagate marks. And nnir searches are ephemeral.

But as you also discovered nnir has the necessary code to propagate
marks. This is because it allows for non-ephemeral search groups as
well. I sent a message to this list about this feature several years
ago.

The new feature (permanent search groups with full marks propagation)
can be explored as follows (I didn't add the code to make it
particularly user friendly when I introduced it and then got too busy to
do much more):

Do an nnir search as you normally would (e.g. "G G" in the Group
buffer). While in the nnir Summary buffer invoke the command
'gnus-summary-create-nnir-group. This will prompt for a name and create
a group using the search parameters that yielded the nnir summary
buffer. This Group should then show up in the Group buffer. You can
modify the search by editing the Group parameters (e.g. "G p" on the
group in the Group buffer). This group behaves (almost) like any other
Gnus group, and marks should propagate as you would expect.

For example I have a group that finds all of my "flagged" messages.
For reference the group parameters for this search include:

 (nnir-specs
  (nnir-query-spec
   (query . "FLAGGED")
   (criteria . ""))
  (nnir-group-spec
   ("nnimap:home")))

This searches for all the flagged message on my nnimap:home server and
creates a a group out of the results. (It currently has over a thousand
messages:()

I also have a bunch of groups for certain correspondents; and groups for
messages in certain date ranges. And a variety of others.

In fact these "virtual" search groups are probably the primary way I
have been using gnus for the past few years. Although I haven't done it
yet, I keep thinking I should stop moving my email around and keep it
all in a single inbox and then just use these permanent search groups
instead.

I made the decision at the time that regular searches should properly be
ephemeral and not propagate marks. (And while modifying the quit-config
parameter on an ephemeral group mostly works there are some gotchas that
make it not by itself a good solution). But "permanent" search  groups
should behave more, uhm,  permanently. 

There are a few wrinkles to be aware of:

1. Since searching can be slow I chose to not redo the search on every
   scan and use a cached search result. You can always force a hard
   rescan. (This could be managed by using levels to avoid scanning on
   groups that are slow to scan instead.)

2. These permanent groups don't do expiry. Expiry is tricky for groups
   whose messages are aggregated from different sources: some of the
   sources may not support expiry; should the search group have its own
   expiry parameters or rely on the expiry rules from each article's
   home group? Since expiry rules often rely on time information, whose
   time? The search group or the home group? Of course choices could be
   made but I think any choice would lead to some surprises so I chose
   to not use it. (I visit the home groups of most articles in my
   virtual groups not infrequently and these home groups have their own
   expiry rules that then get applied).

3. Gnus sometimes gets confused about how many articles are in the
   virtual group and how many are unread. I never found it annoying
   enough to track down but there are still bugs in this regard.

4. The interface isn't very user friendly. (Actually it works pretty
   well---if you are conversant in raw imap search format. To make a new
   group I just do a random search, create the group, and then edit the
   group parameter to put in my properly crafted imap search query. The
   formats of the nnir-query-spec and nnir-group-spec are documented in
   nnir.el)

5. Unfortunately "virtual" has already been used in Gnus to mean
   something else so I can't call these groups (created from messages
   spread out across real groups) virtual. I have been (in my head)
   calling them "select" groups.

Finally I should say that when I rewrote nnir from scratch I
deliberately separated out the "searching" part from the message and
group handling part to allow this. Everything is more or less factored
out and my plan was to create an "nnselect" backend that includes all of
the non-searching parts and just uses a group parameter to specify which
articles should be included in the group. This parameter could, for
example, be an nnir search query; or it could be any function that
returns a list of articles in an appropriate format (this format is also
described in nnir.el). Everything is in place to do this (its probably
an afternoon's work to rename and move the pieces around) but I just
haven't found time to do so.

Best,
Andy

   

Re: gnus nnmaildir and notmuch

[Andrew Cohen]

Oops, forgot an important detail. When choosing the name for the
"permanent" select group it must start with "nnir" (so for example I
call my group of flagged message "nnir-flaged"). This is a (temporary)
safety device I was using to keep track of changes but can be relaxed.

Best,
Andy

Re: gnus nnmaildir and notmuch

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:


[...]

> The new feature (permanent search groups with full marks propagation)
> can be explored as follows (I didn't add the code to make it
> particularly user friendly when I introduced it and then got too busy to
> do much more):
>
> Do an nnir search as you normally would (e.g. "G G" in the Group
> buffer). While in the nnir Summary buffer invoke the command
> 'gnus-summary-create-nnir-group. This will prompt for a name and create
> a group using the search parameters that yielded the nnir summary
> buffer. This Group should then show up in the Group buffer. You can
> modify the search by editing the Group parameters (e.g. "G p" on the
> group in the Group buffer). This group behaves (almost) like any other
> Gnus group, and marks should propagate as you would expect.

Very cool! I never noticed `gnus-summary-create-nnir-group' down at the
bottom of the file.

So far as I can see, the only remaining pieces would be to give that
command a gnus-group-mode keybinding, and have it prompt for a query,
rather than taking the query from the current group. If at present it's
necessary to prefix the group name with "nnir-", that can simply be done
unconditionally in the command.

That's it, isn't it? That and documentation, of course. I'd be happy to
help with completing this, and/or documenting it.

[...]

> 5. Unfortunately "virtual" has already been used in Gnus to mean
>    something else so I can't call these groups (created from messages
>    spread out across real groups) virtual. I have been (in my head)
>    calling them "select" groups.
>
> Finally I should say that when I rewrote nnir from scratch I
> deliberately separated out the "searching" part from the message and
> group handling part to allow this. Everything is more or less factored
> out and my plan was to create an "nnselect" backend that includes all of
> the non-searching parts and just uses a group parameter to specify which
> articles should be included in the group. This parameter could, for
> example, be an nnir search query; or it could be any function that
> returns a list of articles in an appropriate format (this format is also
> described in nnir.el). Everything is in place to do this (its probably
> an afternoon's work to rename and move the pieces around) but I just
> haven't found time to do so.

I dove pretty deeply into nnir a year or two ago when I was writing
Gnorb. I needed exactly what you describe here: a simple way to create
groups containing arbitrary messages. I ended up having to write a fake
"nngnorb" backend, and require users to add it to their server list,
because that was the only way nnir could find the correct query function
to run. I would love to just be able to specify a function as part of
the query parameter!

As for the backends... It seems like there are too many backends
already. Why don't we just hijack nnvirtual? nnselect sounds like it
would be exactly like nnvirtual, except with the ability to specify
articles in addition to groups. It would be more work to merge the two,
but I think it would make much more sense all around. Virtual groups
would be any kind of virtual group, and then there'd just be a search
interface laid on top of it.

What do you (or anyone else) think?

Eric

Re: gnus nnmaildir and notmuch

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:


[...]


    Eric> So far as I can see, the only remaining pieces would be to
    Eric> give that command a gnus-group-mode keybinding, and have it
    Eric> prompt for a query, rather than taking the query from the
    Eric> current group. If at present it's necessary to prefix the
    Eric> group name with "nnir-", that can simply be done
    Eric> unconditionally in the command.

    Eric> That's it, isn't it? That and documentation, of course. I'd be
    Eric> happy to help with completing this, and/or documenting it.

Already done in my private version (sans documentation). But I don't
want to do it now since I'd rather finish the factoring I mentioned
previously. This factoring would lead to incompatible changes in
people's setups (mostly renaming, but still...)


[...]


    Eric> I dove pretty deeply into nnir a year or two ago when I was
    Eric> writing Gnorb. I needed exactly what you describe here: a
    Eric> simple way to create groups containing arbitrary messages. I
    Eric> ended up having to write a fake "nngnorb" backend, and require
    Eric> users to add it to their server list, because that was the
    Eric> only way nnir could find the correct query function to run. I
    Eric> would love to just be able to specify a function as part of
    Eric> the query parameter!

This is also already done in my personal version. 


    Eric> As for the backends... It seems like there are too many
    Eric> backends already. Why don't we just hijack nnvirtual? nnselect
    Eric> sounds like it would be exactly like nnvirtual, except with
    Eric> the ability to specify articles in addition to groups. It
    Eric> would be more work to merge the two, but I think it would make
    Eric> much more sense all around. Virtual groups would be any kind
    Eric> of virtual group, and then there'd just be a search interface
    Eric> laid on top of it.

Merging with nnvirtual doesn't make any sense: the functionality of
nnvirtual (combining existing groups to act as a single group) can
already be done directly as part of nnselect. And the approach taken in
nnvirtual to doing this is radically different from the one I adopted
for nnir (which makes sense since most of the hard work in nnir doesn't
exist for merging entire groups). So the choice is to either eliminate
nnvirtual entirely and replace it with nnselect (possibly renaming in
the process) or let both co-exist. Since replacing it would break things
for anyone using it I favor leaving it alone (unless it turns out that
no one is using it). Over time we can then consider eliminating
nnvirtual. 

(And a small mea culpa---looks like I DID add expiry but just disabled
it for myself. It was so long ago I've mostly forgotten what I did :))

Andy

Re: gnus nnmaildir and notmuch

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

> Already done in my private version (sans documentation). But I don't
> want to do it now since I'd rather finish the factoring I mentioned
> previously. This factoring would lead to incompatible changes in
> people's setups (mostly renaming, but still...)

[...]

> This is also already done in my personal version. 

Okay, looking forward to it!

>     Eric> As for the backends... It seems like there are too many
>     Eric> backends already. Why don't we just hijack nnvirtual? nnselect
>     Eric> sounds like it would be exactly like nnvirtual, except with
>     Eric> the ability to specify articles in addition to groups. It
>     Eric> would be more work to merge the two, but I think it would make
>     Eric> much more sense all around. Virtual groups would be any kind
>     Eric> of virtual group, and then there'd just be a search interface
>     Eric> laid on top of it.
>
> Merging with nnvirtual doesn't make any sense: the functionality of
> nnvirtual (combining existing groups to act as a single group) can
> already be done directly as part of nnselect. And the approach taken in
> nnvirtual to doing this is radically different from the one I adopted
> for nnir (which makes sense since most of the hard work in nnir doesn't
> exist for merging entire groups). So the choice is to either eliminate
> nnvirtual entirely and replace it with nnselect (possibly renaming in
> the process) or let both co-exist. Since replacing it would break things
> for anyone using it I favor leaving it alone (unless it turns out that
> no one is using it). Over time we can then consider eliminating
> nnvirtual. 

Sure, by "merging" I just meant that we don't need both of them -- I was
also assuming that the code from nnir/nnselect would mostly replace
nnvirtual. Co-existence is fine, though it will be confusing for people
exploring Gnus. Perhaps if the docs emphasize nnselect, and only mention
nnvirtual in terms of backwards compatibility, it wouldn't be a big
deal.

Eric