Gnus development mailing list
 help / color / mirror / Atom feed
* useful things with nnselect
@ 2020-09-08  1:41 Andrew Cohen
  2020-09-09  3:05 ` Eric Abrahamsen
  2020-09-09  6:23 ` Eric S Fraga
  0 siblings, 2 replies; 9+ messages in thread
From: Andrew Cohen @ 2020-09-08  1:41 UTC (permalink / raw)
  To: ding


Now that I've finally found a few moments to push the nnselect backend
to master (with some unfortunate formatting and doc errors along the
way, and a failure to provide aliases for some obsolete variables and
functions :() I thought I would write a longer email to this group about
how I am using it and why it might be right for some of you too.

I unfortunately have 4 email accounts (3 work and 1 personal) spread out
on 3 servers, including outlook and gmail (all imap,
fortunately). Together these accounts have about 100k messages. I also
use gmane a bit. Handling all of this is not, uh,
straightforward. Previously I used gnus to sort things into many groups,
and relied heavily on searching to make things manageable. However this
still left me with groups that contained tens of thousands of messages,
and visiting a whole such group is too slow. It ended up being far too
cumbersome.

I have now switched to a different paradigm. I keep all of my messages
in a small number of groups. I do very little sorting (none, actually)
into these groups (for example, all of my gmail is in a single
group). But I have a large number of virtual nnselect groups for all
sorts of things. Since these are collecting messages from only a small
number of sources, they are very, very fast. And its easy to construct
new ones on the fly. Thread referral makes it easy to look at message
chains as needed. And since the number of messages in these groups is
not too large, I can enter the whole group and use limiting and
group-searching (not nnir-searching) very effectively, and its
practically instantaneous.

For example, I have groups which contain all messages from certain
people.  I have groups for the (small number) of mailing lists I
subscribe to (I used to sort these, but I no longer bother). I have a
group "Recent" that contains all email from the past 7 days from all 4
accounts (this is the group I use most often; its kind of like having
expiration without any actual expiring :)).  I have another group of all
flagged messages from my work accounts. And so on.

And I have a "Todo" group---I can add or remove any message with a
keystroke. My normal workflow then is to look at the Recent group, and
as messages come in I deal with some, leave some for later, completely
ignore some, and add some to the Todo group. As I clear tasks I remove
the associated message from the Todo group. And I am constantly
consulting my other virtual groups as I go along.

I will probably add the Todo thing to gnus sometime soon (nnselect makes
it pretty trivial to do). 

I have been working this way for a couple of years now, and its been a
big improvement. It may not work for everyone, but just wanted to share.




^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: useful things with nnselect
  2020-09-08  1:41 useful things with nnselect Andrew Cohen
@ 2020-09-09  3:05 ` Eric Abrahamsen
  2020-09-09  3:26   ` Andrew Cohen
  2020-09-09  6:23 ` Eric S Fraga
  1 sibling, 1 reply; 9+ messages in thread
From: Eric Abrahamsen @ 2020-09-09  3:05 UTC (permalink / raw)
  To: ding

Andrew Cohen <acohen@ust.hk> writes:

> Now that I've finally found a few moments to push the nnselect backend
> to master (with some unfortunate formatting and doc errors along the
> way, and a failure to provide aliases for some obsolete variables and
> functions :() I thought I would write a longer email to this group about
> how I am using it and why it might be right for some of you too.

Glad to see it in!

> I unfortunately have 4 email accounts (3 work and 1 personal) spread out
> on 3 servers, including outlook and gmail (all imap,
> fortunately). Together these accounts have about 100k messages. I also
> use gmane a bit. Handling all of this is not, uh,
> straightforward. Previously I used gnus to sort things into many groups,
> and relied heavily on searching to make things manageable. However this
> still left me with groups that contained tens of thousands of messages,
> and visiting a whole such group is too slow. It ended up being far too
> cumbersome.
>
> I have now switched to a different paradigm. I keep all of my messages
> in a small number of groups. I do very little sorting (none, actually)
> into these groups (for example, all of my gmail is in a single
> group). But I have a large number of virtual nnselect groups for all
> sorts of things. Since these are collecting messages from only a small
> number of sources, they are very, very fast. And its easy to construct
> new ones on the fly. Thread referral makes it easy to look at message
> chains as needed. And since the number of messages in these groups is
> not too large, I can enter the whole group and use limiting and
> group-searching (not nnir-searching) very effectively, and its
> practically instantaneous.
>
> For example, I have groups which contain all messages from certain
> people.  I have groups for the (small number) of mailing lists I
> subscribe to (I used to sort these, but I no longer bother). I have a
> group "Recent" that contains all email from the past 7 days from all 4
> accounts (this is the group I use most often; its kind of like having
> expiration without any actual expiring :)).  I have another group of all
> flagged messages from my work accounts. And so on.

I'd be curious to hear how this differs from nnir -- as far as I know,
these things were mostly already possible with nnir, right?

FWIW, I'm most excited about nnselect as a hacker, as it allows you to
decouple the search process from the servers altogether -- nnir required
all searches to *start* with one or more servers, while nnselect lets
you approach the thing in a more ad-hoc fashion.

One thing I'm noticing now (sorry I didn't see this earlier!) is that
ephemeral search groups don't seem 100% ephemeral: I've searched using
"G G" in the Groups buffer and "G" in the Server buffer, and while the
search groups are no longer visible when I leave them, they seem to be
updating when I hit "g" to update everything later on: I get a bunch of
Messages about re-searching the groups that were part of the earlier
search groups.

Eric



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: useful things with nnselect
  2020-09-09  3:05 ` Eric Abrahamsen
@ 2020-09-09  3:26   ` Andrew Cohen
  2020-09-09 18:12     ` Eric Abrahamsen
  0 siblings, 1 reply; 9+ messages in thread
From: Andrew Cohen @ 2020-09-09  3:26 UTC (permalink / raw)
  To: ding


>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

[...]

    EA> I'd be curious to hear how this differs from nnir -- as far as I
    EA> know, these things were mostly already possible with nnir,
    EA> right?

In fact, no, most of these things weren't possible with nnir. Firstly,
nnir is (or I mean, used to be :)) a combination of two things:
searching, and glue to make the result of a search work as a gnus
group. This glue didn't allow for persistent groups, only ephemeral ones
(well, a long time ago I hacked something in to kind of allow it, but
the resulting groups didn't work like regular gnus groups and had
significant limitations). Also, nnir didn't originally contain any of
the logic to make changes in these groups permanent (again, I have over
time added more and more of this functionality, but it was never
complete).

The point of nnselect was to cleanly separate the searching (which is in
nnir) from what to do with the list of matches that searching
returns. nnselect is now a first-class backend---nnselect groups should
behave like any other group (can be persistent or ephemeral; all marks,
changes, etc are supported). Also, nnselect groups don't have to come
from an nnir search. The Todo group I described has nothing to do with
searches and nnir isn't used. I have a variety of groups like this that
I didn't describe because they are kind of niche. But nnselect makes
this possible.

    EA> FWIW, I'm most excited about nnselect as a hacker, as it allows
    EA> you to decouple the search process from the servers altogether
    EA> -- nnir required all searches to *start* with one or more
    EA> servers, while nnselect lets you approach the thing in a more
    EA> ad-hoc fashion.

Yes, that is the point---decoupling the search from anything
gnus-specific, and allowing the search to be its own thing (which allows
it to be server-neutral). 

    EA> One thing I'm noticing now (sorry I didn't see this earlier!) is
    EA> that ephemeral search groups don't seem 100% ephemeral: I've
    EA> searched using "G G" in the Groups buffer and "G" in the Server
    EA> buffer, and while the search groups are no longer visible when I
    EA> leave them, they seem to be updating when I hit "g" to update
    EA> everything later on: I get a bunch of Messages about
    EA> re-searching the groups that were part of the earlier search
    EA> groups.

This sounds like a bug, and isn't happening for me. When closing one of
these groups this should happen:

    (when (gnus-ephemeral-group-p group)
      (gnus-kill-ephemeral-group group)
      (setq gnus-ephemeral-servers
	    (assq-delete-all 'nnselect gnus-ephemeral-servers)))

So the ephemeral group should be really, truly, gone after exit. 




^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: useful things with nnselect
  2020-09-08  1:41 useful things with nnselect Andrew Cohen
  2020-09-09  3:05 ` Eric Abrahamsen
@ 2020-09-09  6:23 ` Eric S Fraga
  2020-09-09  6:41   ` Andrew Cohen
  1 sibling, 1 reply; 9+ messages in thread
From: Eric S Fraga @ 2020-09-09  6:23 UTC (permalink / raw)
  To: ding

On Tuesday,  8 Sep 2020 at 09:41, Andrew Cohen wrote:
> Now that I've finally found a few moments to push the nnselect backend
> to master (with some unfortunate formatting and doc errors along the
> way, and a failure to provide aliases for some obsolete variables and
> functions :() I thought I would write a longer email to this group about
> how I am using it and why it might be right for some of you too.

[...]

> I have been working this way for a couple of years now, and its been a
> big improvement. It may not work for everyone, but just wanted to share.

Thank you for this.  Very interesting.  I have a setup that has some
similarity to yours, but obviously without using nnselect: using virtual
groups and some splitting.  I will check nnselect out.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.7 on Debian bullseye/sid



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: useful things with nnselect
  2020-09-09  6:23 ` Eric S Fraga
@ 2020-09-09  6:41   ` Andrew Cohen
  2020-09-09 11:31     ` Eric S Fraga
  0 siblings, 1 reply; 9+ messages in thread
From: Andrew Cohen @ 2020-09-09  6:41 UTC (permalink / raw)
  To: ding

>>>>> "ESF" == Eric S Fraga <e.fraga@ucl.ac.uk> writes:

[...]

    ESF> Thank you for this.  Very interesting.  I have a setup that has
    ESF> some similarity to yours, but obviously without using nnselect:
    ESF> using virtual groups and some splitting.  I will check nnselect
    ESF> out.

Before nnselect that is probably very similar to what I did. With
nnselect I can be much more fine-grained than I can with splitting and
combining, and I can do it on the fly. Let me know how it goes and if
you need any tips.

Oh, forgot to mention that (not surprisingly) nnselect can replace
nnvirtual. You just make an nnselect group with an nnselect-function
like the following (just hacked this together but it worked in one
simple test :)) and nnselect-args containing the groups you want to
combine:

(defun my-virtual (groups)
    (let ((limit 1000))
      (apply #'vconcat
	     (mapcar
	      #'(lambda (group)
		  (pcase-let ((`(,min . ,max) (gnus-active group))
			      (value))
		    (when (> (- max min) limit) (setq min (- max limit)))
		    (dotimes (number (1+ (- max min)) value)
		      (push  (vector group (+ min number) 1)
			     value))
		    (nreverse value)))
	      groups))))

(I have put an artificial limit on the size of the group since some of
my groups have hundreds of thousands of messages, and loading this many
headers is unbearably slow. But you can easily increase this beyond 1000
without too much trouble). 



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: useful things with nnselect
  2020-09-09  6:41   ` Andrew Cohen
@ 2020-09-09 11:31     ` Eric S Fraga
  2020-09-11  0:00       ` Andrew Cohen
  0 siblings, 1 reply; 9+ messages in thread
From: Eric S Fraga @ 2020-09-09 11:31 UTC (permalink / raw)
  To: ding

On Wednesday,  9 Sep 2020 at 14:41, Andrew Cohen wrote:
> Oh, forgot to mention that (not surprisingly) nnselect can replace
> nnvirtual. 
[...]
> (I have put an artificial limit on the size of the group since some of
> my groups have hundreds of thousands of messages, and loading this many
> headers is unbearably slow. 

Mine have tens to hundreds of thousands of emails.  Virtual groups seem
to be very fast when all I'm doing is combining groups.  But all of the
groups involved in my virtual groupings are nnml: will that make a
difference?

Thanks for the code, in any case.  I've made a note of it for when I get
around to trying nnselect (not soon as too busy with some high priority
work to muck about with email settings ;-)).
-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.7 on Debian bullseye/sid



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: useful things with nnselect
  2020-09-09  3:26   ` Andrew Cohen
@ 2020-09-09 18:12     ` Eric Abrahamsen
  0 siblings, 0 replies; 9+ messages in thread
From: Eric Abrahamsen @ 2020-09-09 18:12 UTC (permalink / raw)
  To: ding

Andrew Cohen <acohen@ust.hk> writes:

>>>>>> "EA" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> [...]
>
>     EA> I'd be curious to hear how this differs from nnir -- as far as I
>     EA> know, these things were mostly already possible with nnir,
>     EA> right?
>
> In fact, no, most of these things weren't possible with nnir. Firstly,
> nnir is (or I mean, used to be :)) a combination of two things:
> searching, and glue to make the result of a search work as a gnus
> group. This glue didn't allow for persistent groups, only ephemeral ones
> (well, a long time ago I hacked something in to kind of allow it, but
> the resulting groups didn't work like regular gnus groups and had
> significant limitations). Also, nnir didn't originally contain any of
> the logic to make changes in these groups permanent (again, I have over
> time added more and more of this functionality, but it was never
> complete).
>
> The point of nnselect was to cleanly separate the searching (which is in
> nnir) from what to do with the list of matches that searching
> returns. nnselect is now a first-class backend---nnselect groups should
> behave like any other group (can be persistent or ephemeral; all marks,
> changes, etc are supported). Also, nnselect groups don't have to come
> from an nnir search. The Todo group I described has nothing to do with
> searches and nnir isn't used. I have a variety of groups like this that
> I didn't describe because they are kind of niche. But nnselect makes
> this possible.

Cool, thanks for clarifying!



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: useful things with nnselect
  2020-09-09 11:31     ` Eric S Fraga
@ 2020-09-11  0:00       ` Andrew Cohen
  2020-09-11  8:25         ` Eric S Fraga
  0 siblings, 1 reply; 9+ messages in thread
From: Andrew Cohen @ 2020-09-11  0:00 UTC (permalink / raw)
  To: ding


>>>>> "ESF" == Eric S Fraga <e.fraga@ucl.ac.uk> writes:

    >> (I have put an artificial limit on the size of the group since
    >> some of my groups have hundreds of thousands of messages, and
    >> loading this many headers is unbearably slow.

    ESF> Mine have tens to hundreds of thousands of emails.  Virtual
    ESF> groups seem to be very fast when all I'm doing is combining
    ESF> groups.  But all of the groups involved in my virtual groupings
    ESF> are nnml: will that make a difference?

Sorry, I wasn't thinking clearly---this limit is unnecessary. The only
thing that is slow is retrieving a huge number of headers and processing
them (e.g. threading) to create the summary buffer. This is true for any
group in gnus, including nnvirtual. The combining itself is no problem
and doesn't need this limit; you may either remove it, or just set it to
some very large number.



^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: useful things with nnselect
  2020-09-11  0:00       ` Andrew Cohen
@ 2020-09-11  8:25         ` Eric S Fraga
  0 siblings, 0 replies; 9+ messages in thread
From: Eric S Fraga @ 2020-09-11  8:25 UTC (permalink / raw)
  To: ding

On Friday, 11 Sep 2020 at 08:00, Andrew Cohen wrote:
> The only thing that is slow is retrieving a huge number of headers and
> processing them (e.g. threading) to create the summary buffer. 

This makes sense.  Thank you.

-- 
Eric S Fraga via Emacs 28.0.50 & org 9.3.7 on Debian bullseye/sid



^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2020-09-11  8:26 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-08  1:41 useful things with nnselect Andrew Cohen
2020-09-09  3:05 ` Eric Abrahamsen
2020-09-09  3:26   ` Andrew Cohen
2020-09-09 18:12     ` Eric Abrahamsen
2020-09-09  6:23 ` Eric S Fraga
2020-09-09  6:41   ` Andrew Cohen
2020-09-09 11:31     ` Eric S Fraga
2020-09-11  0:00       ` Andrew Cohen
2020-09-11  8:25         ` Eric S Fraga

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).