From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/87534 Path: news.gmane.org!.POSTED!not-for-mail From: Eric Abrahamsen Newsgroups: gmane.emacs.gnus.general Subject: Re: nnvirtual Date: Wed, 03 May 2017 11:00:50 -0700 Message-ID: <87y3udn7l9.fsf@ericabrahamsen.net> References: <87mvauxgt9.fsf@hanan> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1493834530 27661 195.159.176.226 (3 May 2017 18:02:10 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Wed, 3 May 2017 18:02:10 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) To: ding@gnus.org Original-X-From: ding-owner+m35749@lists.math.uh.edu Wed May 03 20:02:05 2017 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from mxfilter-048035.atla03.us.yomura.com ([107.189.48.35]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d5ybF-00075D-J0 for ding-account@gmane.org; Wed, 03 May 2017 20:02:05 +0200 X-Yomura-MXScrub: 1.0 Original-Received: from lists1.math.uh.edu (unknown [129.7.128.208]) by mxfilter-048035.atla03.us.yomura.com (Halon) with ESMTPS id 9f7b5a60-302a-11e7-b087-b499baabecb2; Wed, 03 May 2017 18:02:09 +0000 (UTC) Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by lists1.math.uh.edu with smtp (Exim 4.87) (envelope-from ) id 1d5yad-0007Ox-Ro; Wed, 03 May 2017 13:01:27 -0500 Original-Received: from mx2.math.uh.edu ([129.7.128.33]) by lists1.math.uh.edu with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.87) (envelope-from ) id 1d5yab-0007OS-C7 for ding@lists.math.uh.edu; Wed, 03 May 2017 13:01:25 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx2.math.uh.edu with esmtps (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.87) (envelope-from ) id 1d5yaa-0007jQ-5h for ding@lists.math.uh.edu; Wed, 03 May 2017 13:01:25 -0500 Original-Received: from [195.159.176.226] (helo=blaine.gmane.org) by quimby.gnus.org with esmtps (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from ) id 1d5yaY-00039n-PJ for ding@gnus.org; Wed, 03 May 2017 20:01:22 +0200 Original-Received: from list by blaine.gmane.org with local (Exim 4.84_2) (envelope-from ) id 1d5yaS-0006Ap-4x for ding@gnus.org; Wed, 03 May 2017 20:01:16 +0200 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 57 Original-X-Complaints-To: usenet@blaine.gmane.org Cancel-Lock: sha1:f2k8ER8xZ5B8WAvxNmpA0zQn8BY= List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:87534 Archived-At: Andrew Cohen writes: > Gnus has had the nnvirtual backend, allowing several newsgroups to be > read as a single, combined newsgroup, for ages. But is anyone using it? > And if so, how and why? > > Background (if you are interested): the new backend I have written, > nnselect, is a general backend that can take any set of articles and > treat them like a single group. This group behaves in all respects like > any regular gnus group (subject to bugs, and a few untested items like > article caching and the agent). One use (but not the only use) is to use > a search function to obtain the selection of articles. > > So nnselect can be used as a full replacement for nnvirtual, in addition > to doing lots more. For reference, the function that you can find at the > end of this message can be used to combine groups (it takes a list of > groups to combine as argument). > > So why my question? In part, because the way I am doing things has some > performance issues when you have a group with hundreds of thousands of > articles (which is often the case for gmane, since nothing ever > expires). I can fix all of the performance issues with one exception: > retrieving a hundred thousand headers and doing anything with them is > never going to be speedy. And this has nothing to do with combining but > just getting and dealing with all these headers is trouble. So to > optimize things well I want to know what people are really doing with > these combined groups. In a past life, when gmane searching was > available, I had a straightforward answer to this question: I just > searched for whatever articles were of interest and made my group out of > these. I never needed to combine whole groups. > > But search is now defunct (at least for the moment). And I must say, > using gmane without it kind of sucks. So I rarely do anything more > complicated than looking at recent messages, and occasionally going back > to look through the past several hundred messages. Never more > than 500. Or maybe 1000. > > So the really simple thing that I have done in the function that follows > is apply a (configurable) limit to how many articles are in the combined > group from any one component group. I have been using this for a week > now with the limit set at 1000 which has a noticeable but not annoying > impact on performance. But I've also set the limit to 100,000 and tried > to pull in gmane.emacs.gnus.general (which stands at around 85,000 > messages). I never successfully retrieved a header list (either with > nnselect or with nntp) but the nnselect part of the code did fine, > albeit with a few seconds devoted to some avoidable bookkeeping. But I > don't want to spend time writing code to avoid the bookkeeping if it > isn't something that is relevant anyway. (Me again.) Is there a way to make use of Gnus' existing "big group" behavior? Meaning, add the active numbers, and if it's more than `gnus-large-newsgroup', prompt the user for the number of articles to retrieve? Or is that happening at a different level of infrastructure? Eric