From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/87533 Path: news.gmane.org!.POSTED!not-for-mail From: Andrew Cohen Newsgroups: gmane.emacs.gnus.general Subject: nnvirtual Date: Wed, 03 May 2017 20:31:30 +0800 Organization: Boston University Message-ID: <87mvauxgt9.fsf@hanan> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain X-Trace: blaine.gmane.org 1493814775 7075 195.159.176.226 (3 May 2017 12:32:55 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Wed, 3 May 2017 12:32:55 +0000 (UTC) User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) To: ding@gnus.org Original-X-From: ding-owner+m35748@lists.math.uh.edu Wed May 03 14:32:51 2017 Return-path: Envelope-to: ding-account@gmane.org Original-Received: from mxfilter-048034.atla03.us.yomura.com ([107.189.48.34]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1d5tSc-0001gz-Af for ding-account@gmane.org; Wed, 03 May 2017 14:32:50 +0200 X-Yomura-MXScrub: 1.0 Original-Received: from lists1.math.uh.edu (unknown [129.7.128.208]) by mxfilter-048034.atla03.us.yomura.com (Halon) with ESMTPS id a02ea451-2ffc-11e7-8ed1-b499baa2b07a; Wed, 03 May 2017 12:32:53 +0000 (UTC) Original-Received: from localhost ([127.0.0.1] helo=lists.math.uh.edu) by lists1.math.uh.edu with smtp (Exim 4.87) (envelope-from ) id 1d5tRe-0004tO-Ok; Wed, 03 May 2017 07:31:50 -0500 Original-Received: from mx2.math.uh.edu ([129.7.128.33]) by lists1.math.uh.edu with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.87) (envelope-from ) id 1d5tRa-0004sg-KO for ding@lists.math.uh.edu; Wed, 03 May 2017 07:31:46 -0500 Original-Received: from quimby.gnus.org ([80.91.231.51]) by mx2.math.uh.edu with esmtps (TLSv1.2:DHE-RSA-AES128-SHA:128) (Exim 4.87) (envelope-from ) id 1d5tRZ-0004pa-8w for ding@lists.math.uh.edu; Wed, 03 May 2017 07:31:46 -0500 Original-Received: from [195.159.176.226] (helo=blaine.gmane.org) by quimby.gnus.org with esmtps (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from ) id 1d5tRX-0008Dz-SF for ding@gnus.org; Wed, 03 May 2017 14:31:43 +0200 Original-Received: from list by blaine.gmane.org with local (Exim 4.84_2) (envelope-from ) id 1d5tRP-0000GX-CE for ding@gnus.org; Wed, 03 May 2017 14:31:35 +0200 X-Injected-Via-Gmane: http://gmane.org/ Original-Lines: 70 Original-X-Complaints-To: usenet@blaine.gmane.org Cancel-Lock: sha1:SP2ki+HTL8D8rL+o6xGjMcpvckk= List-ID: Precedence: bulk Xref: news.gmane.org gmane.emacs.gnus.general:87533 Archived-At: Gnus has had the nnvirtual backend, allowing several newsgroups to be read as a single, combined newsgroup, for ages. But is anyone using it? And if so, how and why? Background (if you are interested): the new backend I have written, nnselect, is a general backend that can take any set of articles and treat them like a single group. This group behaves in all respects like any regular gnus group (subject to bugs, and a few untested items like article caching and the agent). One use (but not the only use) is to use a search function to obtain the selection of articles. So nnselect can be used as a full replacement for nnvirtual, in addition to doing lots more. For reference, the function that you can find at the end of this message can be used to combine groups (it takes a list of groups to combine as argument). So why my question? In part, because the way I am doing things has some performance issues when you have a group with hundreds of thousands of articles (which is often the case for gmane, since nothing ever expires). I can fix all of the performance issues with one exception: retrieving a hundred thousand headers and doing anything with them is never going to be speedy. And this has nothing to do with combining but just getting and dealing with all these headers is trouble. So to optimize things well I want to know what people are really doing with these combined groups. In a past life, when gmane searching was available, I had a straightforward answer to this question: I just searched for whatever articles were of interest and made my group out of these. I never needed to combine whole groups. But search is now defunct (at least for the moment). And I must say, using gmane without it kind of sucks. So I rarely do anything more complicated than looking at recent messages, and occasionally going back to look through the past several hundred messages. Never more than 500. Or maybe 1000. So the really simple thing that I have done in the function that follows is apply a (configurable) limit to how many articles are in the combined group from any one component group. I have been using this for a week now with the limit set at 1000 which has a noticeable but not annoying impact on performance. But I've also set the limit to 100,000 and tried to pull in gmane.emacs.gnus.general (which stands at around 85,000 messages). I never successfully retrieved a header list (either with nnselect or with nntp) but the nnselect part of the code did fine, albeit with a few seconds devoted to some avoidable bookkeeping. But I don't want to spend time writing code to avoid the bookkeeping if it isn't something that is relevant anyway. Here is the promised function to use with nnselect (set limit to whatever number you like. 500 probably has no detectable effect on performance and should pull in all the articles you are likely to be interested in. Then when creating the group use select-to-virtual as the nnselect-function and a list of the groups you want to combine as the nnselect-args. Note that you can combine any groups from any combination of servers/backends. And the limit applies per group, not to the aggregate.): (defun select-to-virtual (groups) (let ((limit 1000)) (apply #'vconcat (mapcar #'(lambda (group) (pcase-let ((`(,min . ,max) (gnus-active group)) (value)) (when (> (- max min) limit) (setq min (- max limit))) (dotimes (number (1+ (- max min)) value) (push (vector group (+ min number) 1) value)) (nreverse value))) groups))))