Re: nnvirtual

[Eric Abrahamsen <eric@ericabrahamsen.net>]

Andrew Cohen <cohen@bu.edu> writes:

> Gnus has had the nnvirtual backend, allowing several newsgroups to be
> read as a single, combined newsgroup, for ages. But is anyone using it?
> And if so, how and why?
>
> Background (if you are interested): the new backend I have written,
> nnselect, is a general backend that can take any set of articles and
> treat them like a single group. This group behaves in all respects like
> any regular gnus group (subject to bugs, and a few untested items like
> article caching and the agent). One use (but not the only use) is to use
> a search function to obtain the selection of articles.
>
> So nnselect can be used as a full replacement for nnvirtual, in addition
> to doing lots more. For reference, the function that you can find at the
> end of this message can be used to combine groups (it takes a list of
> groups to combine as argument).
>
> So why my question? In part, because the way I am doing things has some
> performance issues when you have a group with hundreds of thousands of
> articles (which is often the case for gmane, since nothing ever
> expires). I can fix all of the performance issues with one exception:
> retrieving a hundred thousand headers and doing anything with them is
> never going to be speedy. And this has nothing to do with combining but
> just getting and dealing with all these headers is trouble. So to
> optimize things well I want to know what people are really doing with
> these combined groups. In a past life, when gmane searching was
> available, I had a straightforward answer to this question: I just
> searched for whatever articles were of interest and made my group out of
> these. I never needed to combine whole groups.
>
> But search is now defunct (at least for the moment). And I must say,
> using gmane without it kind of sucks. So I rarely do anything more
> complicated than looking at recent messages, and occasionally going back
> to look through the past several hundred messages. Never more
> than 500. Or maybe 1000.
>
> So the really simple thing that I have done in the function that follows
> is apply a (configurable) limit to how many articles are in the combined
> group from any one component group. I have been using this for a week
> now with the limit set at 1000 which has a noticeable but not annoying
> impact on performance. But I've also set the limit to 100,000 and tried
> to pull in gmane.emacs.gnus.general (which stands at around 85,000
> messages). I never successfully retrieved a header list (either with
> nnselect or with nntp) but the nnselect part of the code did fine,
> albeit with a few seconds devoted to some avoidable bookkeeping. But I
> don't want to spend time writing code to avoid the bookkeeping if it
> isn't something that is relevant anyway.

(Me again.)

Is there a way to make use of Gnus' existing "big group" behavior?
Meaning, add the active numbers, and if it's more than
`gnus-large-newsgroup', prompt the user for the number of articles to
retrieve? Or is that happening at a different level of infrastructure?

Eric