nnvirtual

nnvirtual

[Andrew Cohen]

Gnus has had the nnvirtual backend, allowing several newsgroups to be
read as a single, combined newsgroup, for ages. But is anyone using it?
And if so, how and why?

Background (if you are interested): the new backend I have written,
nnselect, is a general backend that can take any set of articles and
treat them like a single group. This group behaves in all respects like
any regular gnus group (subject to bugs, and a few untested items like
article caching and the agent). One use (but not the only use) is to use
a search function to obtain the selection of articles.

So nnselect can be used as a full replacement for nnvirtual, in addition
to doing lots more. For reference, the function that you can find at the
end of this message can be used to combine groups (it takes a list of
groups to combine as argument).

So why my question? In part, because the way I am doing things has some
performance issues when you have a group with hundreds of thousands of
articles (which is often the case for gmane, since nothing ever
expires). I can fix all of the performance issues with one exception:
retrieving a hundred thousand headers and doing anything with them is
never going to be speedy. And this has nothing to do with combining but
just getting and dealing with all these headers is trouble. So to
optimize things well I want to know what people are really doing with
these combined groups. In a past life, when gmane searching was
available, I had a straightforward answer to this question: I just
searched for whatever articles were of interest and made my group out of
these. I never needed to combine whole groups.

But search is now defunct (at least for the moment). And I must say,
using gmane without it kind of sucks. So I rarely do anything more
complicated than looking at recent messages, and occasionally going back
to look through the past several hundred messages. Never more
than 500. Or maybe 1000.

So the really simple thing that I have done in the function that follows
is apply a (configurable) limit to how many articles are in the combined
group from any one component group. I have been using this for a week
now with the limit set at 1000 which has a noticeable but not annoying
impact on performance. But I've also set the limit to 100,000 and tried
to pull in gmane.emacs.gnus.general (which stands at around 85,000
messages). I never successfully retrieved a header list (either with
nnselect or with nntp) but the nnselect part of the code did fine,
albeit with a few seconds devoted to some avoidable bookkeeping. But I
don't want to spend time writing code to avoid the bookkeeping if it
isn't something that is relevant anyway.

Here is the promised function to use with nnselect (set limit to
whatever number you like. 500 probably has no detectable effect on
performance and should pull in all the articles you are likely to be
interested in. Then when creating the group use select-to-virtual as the
nnselect-function and a list of the groups you want to combine as the
nnselect-args. Note that you can combine any groups from any combination
of servers/backends. And the limit applies per group, not to the
aggregate.):

  (defun select-to-virtual (groups)
    (let ((limit 1000))
      (apply #'vconcat
	     (mapcar
	      #'(lambda (group)
		  (pcase-let ((`(,min . ,max) (gnus-active group))
			      (value))
		    (when (> (- max min) limit) (setq min (- max limit)))
		    (dotimes (number (1+ (- max min)) value)
		      (push  (vector group (+ min number) 1)
			     value))
		    (nreverse value)))
	      groups))))

Re: nnvirtual

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

> Gnus has had the nnvirtual backend, allowing several newsgroups to be
> read as a single, combined newsgroup, for ages. But is anyone using it?
> And if so, how and why?
>
> Background (if you are interested): the new backend I have written,
> nnselect, is a general backend that can take any set of articles and
> treat them like a single group. This group behaves in all respects like
> any regular gnus group (subject to bugs, and a few untested items like
> article caching and the agent). One use (but not the only use) is to use
> a search function to obtain the selection of articles.
>
> So nnselect can be used as a full replacement for nnvirtual, in addition
> to doing lots more. For reference, the function that you can find at the
> end of this message can be used to combine groups (it takes a list of
> groups to combine as argument).
>
> So why my question? In part, because the way I am doing things has some
> performance issues when you have a group with hundreds of thousands of
> articles (which is often the case for gmane, since nothing ever
> expires). I can fix all of the performance issues with one exception:
> retrieving a hundred thousand headers and doing anything with them is
> never going to be speedy. And this has nothing to do with combining but
> just getting and dealing with all these headers is trouble. So to
> optimize things well I want to know what people are really doing with
> these combined groups. In a past life, when gmane searching was
> available, I had a straightforward answer to this question: I just
> searched for whatever articles were of interest and made my group out of
> these. I never needed to combine whole groups.
>
> But search is now defunct (at least for the moment). And I must say,
> using gmane without it kind of sucks. So I rarely do anything more
> complicated than looking at recent messages, and occasionally going back
> to look through the past several hundred messages. Never more
> than 500. Or maybe 1000.
>
> So the really simple thing that I have done in the function that follows
> is apply a (configurable) limit to how many articles are in the combined
> group from any one component group. I have been using this for a week
> now with the limit set at 1000 which has a noticeable but not annoying
> impact on performance. But I've also set the limit to 100,000 and tried
> to pull in gmane.emacs.gnus.general (which stands at around 85,000
> messages). I never successfully retrieved a header list (either with
> nnselect or with nntp) but the nnselect part of the code did fine,
> albeit with a few seconds devoted to some avoidable bookkeeping. But I
> don't want to spend time writing code to avoid the bookkeeping if it
> isn't something that is relevant anyway.

(Me again.)

Is there a way to make use of Gnus' existing "big group" behavior?
Meaning, add the active numbers, and if it's more than
`gnus-large-newsgroup', prompt the user for the number of articles to
retrieve? Or is that happening at a different level of infrastructure?

Eric

Re: nnvirtual

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    Eric> Andrew Cohen <cohen@bu.edu> writes:
    >> Gnus has had the nnvirtual backend, allowing several newsgroups
    >> to be read as a single, combined newsgroup, for ages. But is
    >> anyone using it?  And if so, how and why?

[...]


    Eric> (Me again.)

    Eric> Is there a way to make use of Gnus' existing "big group"
    Eric> behavior?  Meaning, add the active numbers, and if it's more
    Eric> than `gnus-large-newsgroup', prompt the user for the number of
    Eric> articles to retrieve? Or is that happening at a different
    Eric> level of infrastructure?

Different level of infrastructure. But its not hard to solve this
problem (in fact the limit variable that I stuck in by hand does almost
exactly what the large newsgroup variable does, minus the prompting
which would be trivial to add). But I am trying to understand what
functionality would be served by any of this to decide how best to
handle it.

My real guess is that no one is using it and there it serves no
purpose. In which case the simplest solution is probably the best one:)

Re: nnvirtual

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     Eric> Andrew Cohen <cohen@bu.edu> writes:
>     >> Gnus has had the nnvirtual backend, allowing several newsgroups
>     >> to be read as a single, combined newsgroup, for ages. But is
>     >> anyone using it?  And if so, how and why?
>
> [...]
>
>
>     Eric> (Me again.)
>
>     Eric> Is there a way to make use of Gnus' existing "big group"
>     Eric> behavior?  Meaning, add the active numbers, and if it's more
>     Eric> than `gnus-large-newsgroup', prompt the user for the number of
>     Eric> articles to retrieve? Or is that happening at a different
>     Eric> level of infrastructure?
>
> Different level of infrastructure. But its not hard to solve this
> problem (in fact the limit variable that I stuck in by hand does almost
> exactly what the large newsgroup variable does, minus the prompting
> which would be trivial to add). But I am trying to understand what
> functionality would be served by any of this to decide how best to
> handle it.
>
> My real guess is that no one is using it and there it serves no
> purpose. In which case the simplest solution is probably the best one:)

I'd say we have to assume that, if the functionality is there, someone
is using it, and combining groups is definitely something people do --
see Dovecot's virtual groups, for instance.

As for the large newsgroup prompting, I think it would be desirable
simply because it makes select groups behave like other Gnus groups, and
means the limit isn't hardcoded. It short, it reduces users' mental
friction when using select groups.

Random other things:

How does sorting work in select groups? Right now gnus-search does a
pass in `gnus-search-run-query', sorting on rsv, but since nearly all
engines return a score of 100, it's fairly meaningless. Summary buffer
articles really are sorted according to the order returned by
`gnus-search-run-query', which I find odd: shouldn't the normal
`gnus-thread|article-sort-functions' routine kick in after group
creation? Anyway, I'd just as soon not have gnus-search be responsible
for sorting at all: it seems like passing on the rsv should be enough.
On the other hand, I'd sure like to let users specify a sort meta-key,
like "sort:score,-date", but then that information would have to get
passed down the line somehow. Perhaps something for later.

I've provided a count:t meta key in the search language, but it isn't
used right now. The idea is that, if the user includes this key, the
search returns a total number of hits per engine, rather than actually
displaying the results (hits per group would be nice, but that
information could be difficult extract from some engines). It will be
fairly easy to hot-wire `gnus-group-make-search-group' to return this
value, and permanent groups could simply strip it out.

If you like, it could also be a cheap way of updating message counts in
permanent search groups. I note now that, at start-up for instance, the
permanent groups do not show counts until they're entered. Sending a
count request ought to be significantly cheaper than actually retrieving
results, and might be useful.

Anyway, just a possibility.

E

Re: nnvirtual

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    Eric> Andrew Cohen <cohen@bu.edu> writes:
    >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
    >> 

[...]

    Eric> I'd say we have to assume that, if the functionality is there,
    Eric> someone is using it, and combining groups is definitely
    Eric> something people do -- see Dovecot's virtual groups, for
    Eric> instance.

Yes, I am not proposing eliminating it, I'm proposing making it more
useful (but I don't know what that means yet:()


    Eric> As for the large newsgroup prompting, I think it would be
    Eric> desirable simply because it makes select groups behave like
    Eric> other Gnus groups, and means the limit isn't hardcoded. It
    Eric> short, it reduces users' mental friction when using select
    Eric> groups.

Yes, I agree, and my intention is to add it (since its trivial) but
first I want to know if I can make the whole thing more useful.

    Eric> Random other things:

    Eric> How does sorting work in select groups? Right now gnus-search
    Eric> does a pass in `gnus-search-run-query', sorting on rsv, but
    Eric> since nearly all engines return a score of 100, it's fairly
    Eric> meaningless. 

I agree. I have gone back and forth about whether to eliminate it
entirely. Eliminating it would allow some compression in various
internal variables but this probably doesn't make much of a difference.
But nnselect is currently ignoring the RSV---it is still available so
that sorting functions can use it, however. nnselect is agnostic about
the whole thing.

    Eric> Summary buffer articles really are sorted according to the
    Eric> order returned by `gnus-search-run-query', which I find odd:
    Eric> shouldn't the normal `gnus-thread|article-sort-functions'
    Eric> routine kick in after group creation? 

Yes, and that is how it behaves for me---nnselect buffers are sorted
exactly like any other gnus buffer. Not for you? I wonder why...

(Aside: Various gnus functions require the article list to be
non-decreasing so in general the way nnselect works is to sort the
selection by group, and then ascending article number within each
group. This is how regular groups operate as well. Any subsequent
sorting in the summary buffer is "virtual" in the sense that the article
list itself isn't changed.).

    Eric> Anyway, I'd just as soon not have gnus-search be responsible
    Eric> for sorting at all: it seems like passing on the rsv should be
    Eric> enough.  

Any sorting done by nnir (and now gnus-search) is purely
incidental. nnselect mangles the order anyway. So I think you should
safely ignore any issues with regard to the return order of the
articles.

    Eric> On the other hand, I'd sure like to let users specify a sort
    Eric> meta-key, like "sort:score,-date", but then that information
    Eric> would have to get passed down the line somehow. Perhaps
    Eric> something for later.

All nnselect groups should be sorted exactly as any other gnus group,
and should be as flexible as the usual sorting system. If that isn't
working right its a bug.

    Eric> I've provided a count:t meta key in the search language, but
    Eric> it isn't used right now. The idea is that, if the user
    Eric> includes this key, the search returns a total number of hits
    Eric> per engine, rather than actually displaying the results (hits
    Eric> per group would be nice, but that information could be
    Eric> difficult extract from some engines). It will be fairly easy
    Eric> to hot-wire `gnus-group-make-search-group' to return this
    Eric> value, and permanent groups could simply strip it out.

Hmm, the old nnir reported this (it went by too fast to be useful, but
it was always available after the fact in the message buffer).

    Eric> If you like, it could also be a cheap way of updating message
    Eric> counts in permanent search groups. I note now that, at
    Eric> start-up for instance, the permanent groups do not show counts
    Eric> until they're entered. 

This is a temporary artifact of my not yet activating the groups at
startup. The counts are all present and ready to be reported (as soon as
the group is created), I just haven't gotten around to keeping the
proper list of nnselect groups to allow activation and re-scanning as a
matter of course.

    Eric> Sending a count request ought to be significantly cheaper than
    Eric> actually retrieving results, and might be useful.

The counts are currently updated without retrieving any headers, and is
therefore quite fast. For example I have a variety of groups with
thousands of messages that are part of the nnselection and the time it
takes to update the group is a fraction of a second.

Re: nnvirtual

[Andrew Cohen]

>>>>> "Andrew" == Andrew Cohen <cohen@bu.edu> writes:

    Eric> How does sorting work in select groups? Right now gnus-search
    Eric> does a pass in `gnus-search-run-query', sorting on rsv, but
    Eric> since nearly all engines return a score of 100, it's fairly
    Eric> meaningless.

    Eric> Summary buffer articles really are sorted according to the
    Eric> order returned by `gnus-search-run-query', which I find odd:
    Eric> shouldn't the normal `gnus-thread|article-sort-functions'
    Eric> routine kick in after group creation?

    Andrew> Yes, and that is how it behaves for me---nnselect buffers
    Andrew> are sorted exactly like any other gnus buffer. Not for you?
    Andrew> I wonder why...

    Andrew> (Aside: Various gnus functions require the article list to
    Andrew> be non-decreasing so in general the way nnselect works is to
    Andrew> sort the selection by group, and then ascending article
    Andrew> number within each group. This is how regular groups operate
    Andrew> as well. Any subsequent sorting in the summary buffer is
    Andrew> "virtual" in the sense that the article list itself isn't
    Andrew> changed.).

And just to be a bit clearer on sorting.

Gnus uses 2 different sorting functions: one for threaded display and
one for unthreaded display. For unthreaded display the default is to
sort by number, but you can sort any way you like (using either the
canned gnus sort predicates or by rolling your own).

As mentioned in my previous message gnus really wants article lists to
be in non-decreasing order internally, so this means that a default
unthreaded summary buffer will appear to be in the same order as the
article list. For nnselect we first sort the selected articles by group
and then ascending order within the group. This is usually the way they
are retrieved in nnir (more or less deliberately) so it gives the
appearance of using the order in which they are retrieved in nnir as the
sort order but this isn't true; just try a different setting for
gnus-article-sort-functions, for example this one:

(setq gnus-article-sort-functions
  '(gnus-article-sort-by-number
    gnus-article-sort-by-subject))

(and of course you can check similar things if you are using a threaded
display).

So currently to use the RSV for sorting purposes requires constructing
predicates that look at the RSV. This is trivial to do (and I'll
probably include them as part of nnselect or in gnus-sum with the other
sorting predicates at some point). But we can (and probably should)
ignore any sorting issues coming out of gnus-search.

Best,
Andy

Re: nnvirtual

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     Eric> Andrew Cohen <cohen@bu.edu> writes:
>     >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>     >> 
>
> [...]
>
>     Eric> I'd say we have to assume that, if the functionality is there,
>     Eric> someone is using it, and combining groups is definitely
>     Eric> something people do -- see Dovecot's virtual groups, for
>     Eric> instance.
>
> Yes, I am not proposing eliminating it, I'm proposing making it more
> useful (but I don't know what that means yet:()
>
>
>     Eric> As for the large newsgroup prompting, I think it would be
>     Eric> desirable simply because it makes select groups behave like
>     Eric> other Gnus groups, and means the limit isn't hardcoded. It
>     Eric> short, it reduces users' mental friction when using select
>     Eric> groups.
>
> Yes, I agree, and my intention is to add it (since its trivial) but
> first I want to know if I can make the whole thing more useful.

Yeah, I'm not sure either. The two general principles I'm after are:
give users the ability to set up commonly-used patterns, and at the same
time give them as much flexibility as possible. I mentioned earlier the
idea of search "presets", the code for which is pretty much done (and
very simple). Each preset is a list of groups to search, and a base
search query. The user selects a preset, adds additional query terms,
an ephemeral search group is created. I made this because it seemed like
the right balance between configuration and flexibility.

But if, for example, it were possible to start a search from within a
summary buffer, saying "run this search within this group", then presets
might not be necessary: users could create permanent select groups
incorporating several other groups, then run ad-hoc searches within
them.

All of which is only to say that I don't really know, either. But Gnus
users are a strange bunch, I'd say support as much as possible.

>     Eric> Random other things:
>
>     Eric> How does sorting work in select groups? Right now gnus-search
>     Eric> does a pass in `gnus-search-run-query', sorting on rsv, but
>     Eric> since nearly all engines return a score of 100, it's fairly
>     Eric> meaningless. 
>
> I agree. I have gone back and forth about whether to eliminate it
> entirely. Eliminating it would allow some compression in various
> internal variables but this probably doesn't make much of a difference.
> But nnselect is currently ignoring the RSV---it is still available so
> that sorting functions can use it, however. nnselect is agnostic about
> the whole thing.

I'd still like to keep it, and the infrastructure to support it. It
might become more useful later. Namazu hints that it supports scoring (I
really need to get a working installation), imap returns relevancy with
FUZZY, and future engines for solr or xapian-based stuff would provide
scoring.

> All nnselect groups should be sorted exactly as any other gnus group,
> and should be as flexible as the usual sorting system. If that isn't
> working right its a bug.

> So currently to use the RSV for sorting purposes requires constructing
> predicates that look at the RSV. This is trivial to do (and I'll
> probably include them as part of nnselect or in gnus-sum with the other
> sorting predicates at some point). But we can (and probably should)
> ignore any sorting issues coming out of gnus-search.

Responding to both messages here: if there were a new predicate for
sorting on score, I think we'd be fine. Part of me still would like to
let users specify sorting strategies on the fly, as part of the search
query. Sorting could also be passed to the backends, and combined with
limit:n could result in faster queries. At this point that's probably
more work than it's worth, though.

>     Eric> I've provided a count:t meta key in the search language, but
>     Eric> it isn't used right now. The idea is that, if the user
>     Eric> includes this key, the search returns a total number of hits
>     Eric> per engine, rather than actually displaying the results (hits
>     Eric> per group would be nice, but that information could be
>     Eric> difficult extract from some engines). It will be fairly easy
>     Eric> to hot-wire `gnus-group-make-search-group' to return this
>     Eric> value, and permanent groups could simply strip it out.
>
> Hmm, the old nnir reported this (it went by too fast to be useful, but
> it was always available after the fact in the message buffer).

Right, but the idea is that sometimes the count itself is all you want
to see.

>     Eric> If you like, it could also be a cheap way of updating message
>     Eric> counts in permanent search groups. I note now that, at
>     Eric> start-up for instance, the permanent groups do not show counts
>     Eric> until they're entered. 
>
> This is a temporary artifact of my not yet activating the groups at
> startup. The counts are all present and ready to be reported (as soon as
> the group is created), I just haven't gotten around to keeping the
> proper list of nnselect groups to allow activation and re-scanning as a
> matter of course.
>
>     Eric> Sending a count request ought to be significantly cheaper than
>     Eric> actually retrieving results, and might be useful.
>
> The counts are currently updated without retrieving any headers, and is
> therefore quite fast. For example I have a variety of groups with
> thousands of messages that are part of the nnselection and the time it
> takes to update the group is a fraction of a second.

Well, it's probably premature optimization. But it would be pretty
simple to implement...

Re: nnvirtual

[Andrew Cohen]

>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:

    >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
    >> 

[...]

    Eric> But if, for example, it were possible to start a search from
    Eric> within a summary buffer, saying "run this search within this
    Eric> group", then presets might not be necessary: users could
    Eric> create permanent select groups incorporating several other
    Eric> groups, then run ad-hoc searches within them.

Umm, this has always been possible with nnir. Does it not work for you?

[...]


And wrt RSV scoring.

    Eric> I'd still like to keep it, and the infrastructure to support
    Eric> it. It might become more useful later. Namazu hints that it
    Eric> supports scoring (I really need to get a working
    Eric> installation), imap returns relevancy with FUZZY, and future
    Eric> engines for solr or xapian-based stuff would provide scoring.

I used namazu with its scoring for many years. Aside from trying out the
scoring and seeing that it all worked I never used it in practice. I
find these scores (from search engines in general) very much not
useful. But I decided early on after my back and forth that there is
little cost in keeping it, and making the proper predicate is easy and
on my todo list.


[...]

    Eric> Responding to both messages here: if there were a new
    Eric> predicate for sorting on score, I think we'd be fine. Part of
    Eric> me still would like to let users specify sorting strategies on
    Eric> the fly, as part of the search query. Sorting could also be
    Eric> passed to the backends, and combined with limit:n could result
    Eric> in faster queries. At this point that's probably more work
    Eric> than it's worth, though.

I definitely think that gnus-search should stay away from any kind of
sorting. Modulo bugs the gnus strategy of sorting as an overlay on
summary buffers works great. 

    Eric> I've provided a count:t meta key in the search language, but
    Eric> it isn't used right now. The idea is that, if the user
    Eric> includes this key, the search returns a total number of hits
    Eric> per engine, rather than actually displaying the results (hits
    Eric> per group would be nice, but that information could be
    Eric> difficult extract from some engines). It will be fairly easy
    Eric> to hot-wire `gnus-group-make-search-group' to return this
    Eric> value, and permanent groups could simply strip it out.
    >> 
    >> Hmm, the old nnir reported this (it went by too fast to be
    >> useful, but it was always available after the fact in the message
    >> buffer).

    Eric> Right, but the idea is that sometimes the count itself is all
    Eric> you want to see.

    Eric> If you like, it could also be a cheap way of updating message
    Eric> counts in permanent search groups. I note now that, at
    Eric> start-up for instance, the permanent groups do not show counts
    Eric> until they're entered.
    >> 
    >> This is a temporary artifact of my not yet activating the groups
    >> at startup. The counts are all present and ready to be reported
    >> (as soon as the group is created), I just haven't gotten around
    >> to keeping the proper list of nnselect groups to allow activation
    >> and re-scanning as a matter of course.
    >> 
    Eric> Sending a count request ought to be significantly cheaper than
    Eric> actually retrieving results, and might be useful.
    >> 
    >> The counts are currently updated without retrieving any headers,
    >> and is therefore quite fast. For example I have a variety of
    >> groups with thousands of messages that are part of the
    >> nnselection and the time it takes to update the group is a
    >> fraction of a second.

    Eric> Well, it's probably premature optimization. But it would be
    Eric> pretty simple to implement...


I don't think its a real optimization. All the time now is taken in
setting up the connection; the search itself would take the same time,
the only difference being you might not be returning the list of
UIDs. But parsing the UIDs takes essentially no time. Unless you had
something else in mind?

Also nnselect isn't about searching---there are many other ways that
selection lists can be produced.

Re: nnvirtual

[Eric Abrahamsen]

Andrew Cohen <cohen@bu.edu> writes:

>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>     >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>     >> 
>
> [...]
>
>     Eric> But if, for example, it were possible to start a search from
>     Eric> within a summary buffer, saying "run this search within this
>     Eric> group", then presets might not be necessary: users could
>     Eric> create permanent select groups incorporating several other
>     Eric> groups, then run ad-hoc searches within them.
>
> Umm, this has always been possible with nnir. Does it not work for you?

Whoops, that function did not survive the nnir->gnus-search
transformation. I'll put it back in. I think that means I can just drop
search presets.

> And wrt RSV scoring.
>
>     Eric> I'd still like to keep it, and the infrastructure to support
>     Eric> it. It might become more useful later. Namazu hints that it
>     Eric> supports scoring (I really need to get a working
>     Eric> installation), imap returns relevancy with FUZZY, and future
>     Eric> engines for solr or xapian-based stuff would provide scoring.
>
> I used namazu with its scoring for many years. Aside from trying out the
> scoring and seeing that it all worked I never used it in practice. I
> find these scores (from search engines in general) very much not
> useful. But I decided early on after my back and forth that there is
> little cost in keeping it, and making the proper predicate is easy and
> on my todo list.

I see it's already in. BTW, that patch seems to have introduced a
circular dependency between gnus-sum->nnselect->gnus-art->gnus-sum. I
can't get it to compile.

On a related note, I'm having a hard time finding a place to put the
`gnus-group-make-search-group' functions: the call to
nnselect-categorize also tends to create circular dependencies.
Something to think about as this whole thing gets closer to completion.

>     Eric> Responding to both messages here: if there were a new
>     Eric> predicate for sorting on score, I think we'd be fine. Part of
>     Eric> me still would like to let users specify sorting strategies on
>     Eric> the fly, as part of the search query. Sorting could also be
>     Eric> passed to the backends, and combined with limit:n could result
>     Eric> in faster queries. At this point that's probably more work
>     Eric> than it's worth, though.
>
> I definitely think that gnus-search should stay away from any kind of
> sorting. Modulo bugs the gnus strategy of sorting as an overlay on
> summary buffers works great. 

That's fine, I'm happy to drop the sorting stuff.

> I don't think its a real optimization. All the time now is taken in
> setting up the connection; the search itself would take the same time,
> the only difference being you might not be returning the list of
> UIDs. But parsing the UIDs takes essentially no time. Unless you had
> something else in mind?

No, it was just the thought that the engines might have an easier time
returning a simple count than the results themselves (other engines
return more than just UIDs). But I think I was just getting caught up in
trying to expose and make use of everything the engines offer -- not
really necessary.

Re: nnvirtual

[Steinar Bang]

>>>>> Andrew Cohen <cohen@bu.edu>:

> But search is now defunct (at least for the moment). And I must say,
> using gmane without it kind of sucks.

The search functionality in gmane is indeed missed.

(But having gmane without this functionality is better than not having
gmane at all...)

Re: nnvirtual

[Adam Sjøgren]

Steinar writes:

> (But having gmane without this functionality is better than not having
> gmane at all...)

I could agree more!

It is kind of disheartening, though that, nothing [visible, anyway] has
happened since september last year.

I left a comment on the home.gmane.org blog a month ago, and it is still
awaiting moderation.


  Best regards,

    Adam

-- 
 "Calvin, would you set the table please?" "Mm... I           Adam Sjøgren
  don't think so. I'm not enthusiastic about setting     asjo@koldfront.dk
  the table."

Re: nnvirtual

[Harry Putnam]

asjo@koldfront.dk (Adam Sjøgren) writes:

> Steinar writes:
>
>> (But having gmane without this functionality is better than not having
>> gmane at all...)
>
> I could agree more!
>
> It is kind of disheartening, though that, nothing [visible, anyway] has
> happened since september last year.

gmane is and has been an institution for a while now.

Kind of scary like when the big insurance co's and banks started
failing.

If only gnus experts could get a government bailout .. to pay for coders.

> I left a comment on the home.gmane.org blog a month ago, and it is still
> awaiting moderation.

Just a thought on that... maybe way off in left field and not
appropriate but I wonder if stuff like this is welcome on
`gmane.discuss'... I don't see too much traffic there

Re: nnvirtual

[Lars Ingebrigtsen]

Andrew Cohen <cohen@bu.edu> writes:

> Gnus has had the nnvirtual backend, allowing several newsgroups to be
> read as a single, combined newsgroup, for ages. But is anyone using it?
> And if so, how and why?

I use it to combine all the really low-volume gnu.* newsgroups.
Because...  er...  uhm...  I'm not sure.  :-)  I just do.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no