* nnvirtual @ 2017-05-03 12:31 Andrew Cohen 2017-05-03 18:00 ` nnvirtual Eric Abrahamsen ` (2 more replies) 0 siblings, 3 replies; 13+ messages in thread From: Andrew Cohen @ 2017-05-03 12:31 UTC (permalink / raw) To: ding Gnus has had the nnvirtual backend, allowing several newsgroups to be read as a single, combined newsgroup, for ages. But is anyone using it? And if so, how and why? Background (if you are interested): the new backend I have written, nnselect, is a general backend that can take any set of articles and treat them like a single group. This group behaves in all respects like any regular gnus group (subject to bugs, and a few untested items like article caching and the agent). One use (but not the only use) is to use a search function to obtain the selection of articles. So nnselect can be used as a full replacement for nnvirtual, in addition to doing lots more. For reference, the function that you can find at the end of this message can be used to combine groups (it takes a list of groups to combine as argument). So why my question? In part, because the way I am doing things has some performance issues when you have a group with hundreds of thousands of articles (which is often the case for gmane, since nothing ever expires). I can fix all of the performance issues with one exception: retrieving a hundred thousand headers and doing anything with them is never going to be speedy. And this has nothing to do with combining but just getting and dealing with all these headers is trouble. So to optimize things well I want to know what people are really doing with these combined groups. In a past life, when gmane searching was available, I had a straightforward answer to this question: I just searched for whatever articles were of interest and made my group out of these. I never needed to combine whole groups. But search is now defunct (at least for the moment). And I must say, using gmane without it kind of sucks. So I rarely do anything more complicated than looking at recent messages, and occasionally going back to look through the past several hundred messages. Never more than 500. Or maybe 1000. So the really simple thing that I have done in the function that follows is apply a (configurable) limit to how many articles are in the combined group from any one component group. I have been using this for a week now with the limit set at 1000 which has a noticeable but not annoying impact on performance. But I've also set the limit to 100,000 and tried to pull in gmane.emacs.gnus.general (which stands at around 85,000 messages). I never successfully retrieved a header list (either with nnselect or with nntp) but the nnselect part of the code did fine, albeit with a few seconds devoted to some avoidable bookkeeping. But I don't want to spend time writing code to avoid the bookkeeping if it isn't something that is relevant anyway. Here is the promised function to use with nnselect (set limit to whatever number you like. 500 probably has no detectable effect on performance and should pull in all the articles you are likely to be interested in. Then when creating the group use select-to-virtual as the nnselect-function and a list of the groups you want to combine as the nnselect-args. Note that you can combine any groups from any combination of servers/backends. And the limit applies per group, not to the aggregate.): (defun select-to-virtual (groups) (let ((limit 1000)) (apply #'vconcat (mapcar #'(lambda (group) (pcase-let ((`(,min . ,max) (gnus-active group)) (value)) (when (> (- max min) limit) (setq min (- max limit))) (dotimes (number (1+ (- max min)) value) (push (vector group (+ min number) 1) value)) (nreverse value))) groups)))) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-03 12:31 nnvirtual Andrew Cohen @ 2017-05-03 18:00 ` Eric Abrahamsen 2017-05-04 1:06 ` nnvirtual Andrew Cohen 2017-05-12 15:38 ` nnvirtual Steinar Bang 2018-04-11 19:42 ` nnvirtual Lars Ingebrigtsen 2 siblings, 1 reply; 13+ messages in thread From: Eric Abrahamsen @ 2017-05-03 18:00 UTC (permalink / raw) To: ding Andrew Cohen <cohen@bu.edu> writes: > Gnus has had the nnvirtual backend, allowing several newsgroups to be > read as a single, combined newsgroup, for ages. But is anyone using it? > And if so, how and why? > > Background (if you are interested): the new backend I have written, > nnselect, is a general backend that can take any set of articles and > treat them like a single group. This group behaves in all respects like > any regular gnus group (subject to bugs, and a few untested items like > article caching and the agent). One use (but not the only use) is to use > a search function to obtain the selection of articles. > > So nnselect can be used as a full replacement for nnvirtual, in addition > to doing lots more. For reference, the function that you can find at the > end of this message can be used to combine groups (it takes a list of > groups to combine as argument). > > So why my question? In part, because the way I am doing things has some > performance issues when you have a group with hundreds of thousands of > articles (which is often the case for gmane, since nothing ever > expires). I can fix all of the performance issues with one exception: > retrieving a hundred thousand headers and doing anything with them is > never going to be speedy. And this has nothing to do with combining but > just getting and dealing with all these headers is trouble. So to > optimize things well I want to know what people are really doing with > these combined groups. In a past life, when gmane searching was > available, I had a straightforward answer to this question: I just > searched for whatever articles were of interest and made my group out of > these. I never needed to combine whole groups. > > But search is now defunct (at least for the moment). And I must say, > using gmane without it kind of sucks. So I rarely do anything more > complicated than looking at recent messages, and occasionally going back > to look through the past several hundred messages. Never more > than 500. Or maybe 1000. > > So the really simple thing that I have done in the function that follows > is apply a (configurable) limit to how many articles are in the combined > group from any one component group. I have been using this for a week > now with the limit set at 1000 which has a noticeable but not annoying > impact on performance. But I've also set the limit to 100,000 and tried > to pull in gmane.emacs.gnus.general (which stands at around 85,000 > messages). I never successfully retrieved a header list (either with > nnselect or with nntp) but the nnselect part of the code did fine, > albeit with a few seconds devoted to some avoidable bookkeeping. But I > don't want to spend time writing code to avoid the bookkeeping if it > isn't something that is relevant anyway. (Me again.) Is there a way to make use of Gnus' existing "big group" behavior? Meaning, add the active numbers, and if it's more than `gnus-large-newsgroup', prompt the user for the number of articles to retrieve? Or is that happening at a different level of infrastructure? Eric ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-03 18:00 ` nnvirtual Eric Abrahamsen @ 2017-05-04 1:06 ` Andrew Cohen 2017-05-05 4:30 ` nnvirtual Eric Abrahamsen 0 siblings, 1 reply; 13+ messages in thread From: Andrew Cohen @ 2017-05-04 1:06 UTC (permalink / raw) To: ding >>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: Eric> Andrew Cohen <cohen@bu.edu> writes: >> Gnus has had the nnvirtual backend, allowing several newsgroups >> to be read as a single, combined newsgroup, for ages. But is >> anyone using it? And if so, how and why? [...] Eric> (Me again.) Eric> Is there a way to make use of Gnus' existing "big group" Eric> behavior? Meaning, add the active numbers, and if it's more Eric> than `gnus-large-newsgroup', prompt the user for the number of Eric> articles to retrieve? Or is that happening at a different Eric> level of infrastructure? Different level of infrastructure. But its not hard to solve this problem (in fact the limit variable that I stuck in by hand does almost exactly what the large newsgroup variable does, minus the prompting which would be trivial to add). But I am trying to understand what functionality would be served by any of this to decide how best to handle it. My real guess is that no one is using it and there it serves no purpose. In which case the simplest solution is probably the best one:) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-04 1:06 ` nnvirtual Andrew Cohen @ 2017-05-05 4:30 ` Eric Abrahamsen 2017-05-05 12:09 ` nnvirtual Andrew Cohen 0 siblings, 1 reply; 13+ messages in thread From: Eric Abrahamsen @ 2017-05-05 4:30 UTC (permalink / raw) To: ding Andrew Cohen <cohen@bu.edu> writes: >>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: > > Eric> Andrew Cohen <cohen@bu.edu> writes: > >> Gnus has had the nnvirtual backend, allowing several newsgroups > >> to be read as a single, combined newsgroup, for ages. But is > >> anyone using it? And if so, how and why? > > [...] > > > Eric> (Me again.) > > Eric> Is there a way to make use of Gnus' existing "big group" > Eric> behavior? Meaning, add the active numbers, and if it's more > Eric> than `gnus-large-newsgroup', prompt the user for the number of > Eric> articles to retrieve? Or is that happening at a different > Eric> level of infrastructure? > > Different level of infrastructure. But its not hard to solve this > problem (in fact the limit variable that I stuck in by hand does almost > exactly what the large newsgroup variable does, minus the prompting > which would be trivial to add). But I am trying to understand what > functionality would be served by any of this to decide how best to > handle it. > > My real guess is that no one is using it and there it serves no > purpose. In which case the simplest solution is probably the best one:) I'd say we have to assume that, if the functionality is there, someone is using it, and combining groups is definitely something people do -- see Dovecot's virtual groups, for instance. As for the large newsgroup prompting, I think it would be desirable simply because it makes select groups behave like other Gnus groups, and means the limit isn't hardcoded. It short, it reduces users' mental friction when using select groups. Random other things: How does sorting work in select groups? Right now gnus-search does a pass in `gnus-search-run-query', sorting on rsv, but since nearly all engines return a score of 100, it's fairly meaningless. Summary buffer articles really are sorted according to the order returned by `gnus-search-run-query', which I find odd: shouldn't the normal `gnus-thread|article-sort-functions' routine kick in after group creation? Anyway, I'd just as soon not have gnus-search be responsible for sorting at all: it seems like passing on the rsv should be enough. On the other hand, I'd sure like to let users specify a sort meta-key, like "sort:score,-date", but then that information would have to get passed down the line somehow. Perhaps something for later. I've provided a count:t meta key in the search language, but it isn't used right now. The idea is that, if the user includes this key, the search returns a total number of hits per engine, rather than actually displaying the results (hits per group would be nice, but that information could be difficult extract from some engines). It will be fairly easy to hot-wire `gnus-group-make-search-group' to return this value, and permanent groups could simply strip it out. If you like, it could also be a cheap way of updating message counts in permanent search groups. I note now that, at start-up for instance, the permanent groups do not show counts until they're entered. Sending a count request ought to be significantly cheaper than actually retrieving results, and might be useful. Anyway, just a possibility. E ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-05 4:30 ` nnvirtual Eric Abrahamsen @ 2017-05-05 12:09 ` Andrew Cohen 2017-05-05 12:51 ` nnvirtual Andrew Cohen 2017-05-05 15:46 ` nnvirtual Eric Abrahamsen 0 siblings, 2 replies; 13+ messages in thread From: Andrew Cohen @ 2017-05-05 12:09 UTC (permalink / raw) To: ding >>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: Eric> Andrew Cohen <cohen@bu.edu> writes: >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: >> [...] Eric> I'd say we have to assume that, if the functionality is there, Eric> someone is using it, and combining groups is definitely Eric> something people do -- see Dovecot's virtual groups, for Eric> instance. Yes, I am not proposing eliminating it, I'm proposing making it more useful (but I don't know what that means yet:() Eric> As for the large newsgroup prompting, I think it would be Eric> desirable simply because it makes select groups behave like Eric> other Gnus groups, and means the limit isn't hardcoded. It Eric> short, it reduces users' mental friction when using select Eric> groups. Yes, I agree, and my intention is to add it (since its trivial) but first I want to know if I can make the whole thing more useful. Eric> Random other things: Eric> How does sorting work in select groups? Right now gnus-search Eric> does a pass in `gnus-search-run-query', sorting on rsv, but Eric> since nearly all engines return a score of 100, it's fairly Eric> meaningless. I agree. I have gone back and forth about whether to eliminate it entirely. Eliminating it would allow some compression in various internal variables but this probably doesn't make much of a difference. But nnselect is currently ignoring the RSV---it is still available so that sorting functions can use it, however. nnselect is agnostic about the whole thing. Eric> Summary buffer articles really are sorted according to the Eric> order returned by `gnus-search-run-query', which I find odd: Eric> shouldn't the normal `gnus-thread|article-sort-functions' Eric> routine kick in after group creation? Yes, and that is how it behaves for me---nnselect buffers are sorted exactly like any other gnus buffer. Not for you? I wonder why... (Aside: Various gnus functions require the article list to be non-decreasing so in general the way nnselect works is to sort the selection by group, and then ascending article number within each group. This is how regular groups operate as well. Any subsequent sorting in the summary buffer is "virtual" in the sense that the article list itself isn't changed.). Eric> Anyway, I'd just as soon not have gnus-search be responsible Eric> for sorting at all: it seems like passing on the rsv should be Eric> enough. Any sorting done by nnir (and now gnus-search) is purely incidental. nnselect mangles the order anyway. So I think you should safely ignore any issues with regard to the return order of the articles. Eric> On the other hand, I'd sure like to let users specify a sort Eric> meta-key, like "sort:score,-date", but then that information Eric> would have to get passed down the line somehow. Perhaps Eric> something for later. All nnselect groups should be sorted exactly as any other gnus group, and should be as flexible as the usual sorting system. If that isn't working right its a bug. Eric> I've provided a count:t meta key in the search language, but Eric> it isn't used right now. The idea is that, if the user Eric> includes this key, the search returns a total number of hits Eric> per engine, rather than actually displaying the results (hits Eric> per group would be nice, but that information could be Eric> difficult extract from some engines). It will be fairly easy Eric> to hot-wire `gnus-group-make-search-group' to return this Eric> value, and permanent groups could simply strip it out. Hmm, the old nnir reported this (it went by too fast to be useful, but it was always available after the fact in the message buffer). Eric> If you like, it could also be a cheap way of updating message Eric> counts in permanent search groups. I note now that, at Eric> start-up for instance, the permanent groups do not show counts Eric> until they're entered. This is a temporary artifact of my not yet activating the groups at startup. The counts are all present and ready to be reported (as soon as the group is created), I just haven't gotten around to keeping the proper list of nnselect groups to allow activation and re-scanning as a matter of course. Eric> Sending a count request ought to be significantly cheaper than Eric> actually retrieving results, and might be useful. The counts are currently updated without retrieving any headers, and is therefore quite fast. For example I have a variety of groups with thousands of messages that are part of the nnselection and the time it takes to update the group is a fraction of a second. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-05 12:09 ` nnvirtual Andrew Cohen @ 2017-05-05 12:51 ` Andrew Cohen 2017-05-05 15:46 ` nnvirtual Eric Abrahamsen 1 sibling, 0 replies; 13+ messages in thread From: Andrew Cohen @ 2017-05-05 12:51 UTC (permalink / raw) To: ding >>>>> "Andrew" == Andrew Cohen <cohen@bu.edu> writes: Eric> How does sorting work in select groups? Right now gnus-search Eric> does a pass in `gnus-search-run-query', sorting on rsv, but Eric> since nearly all engines return a score of 100, it's fairly Eric> meaningless. Eric> Summary buffer articles really are sorted according to the Eric> order returned by `gnus-search-run-query', which I find odd: Eric> shouldn't the normal `gnus-thread|article-sort-functions' Eric> routine kick in after group creation? Andrew> Yes, and that is how it behaves for me---nnselect buffers Andrew> are sorted exactly like any other gnus buffer. Not for you? Andrew> I wonder why... Andrew> (Aside: Various gnus functions require the article list to Andrew> be non-decreasing so in general the way nnselect works is to Andrew> sort the selection by group, and then ascending article Andrew> number within each group. This is how regular groups operate Andrew> as well. Any subsequent sorting in the summary buffer is Andrew> "virtual" in the sense that the article list itself isn't Andrew> changed.). And just to be a bit clearer on sorting. Gnus uses 2 different sorting functions: one for threaded display and one for unthreaded display. For unthreaded display the default is to sort by number, but you can sort any way you like (using either the canned gnus sort predicates or by rolling your own). As mentioned in my previous message gnus really wants article lists to be in non-decreasing order internally, so this means that a default unthreaded summary buffer will appear to be in the same order as the article list. For nnselect we first sort the selected articles by group and then ascending order within the group. This is usually the way they are retrieved in nnir (more or less deliberately) so it gives the appearance of using the order in which they are retrieved in nnir as the sort order but this isn't true; just try a different setting for gnus-article-sort-functions, for example this one: (setq gnus-article-sort-functions '(gnus-article-sort-by-number gnus-article-sort-by-subject)) (and of course you can check similar things if you are using a threaded display). So currently to use the RSV for sorting purposes requires constructing predicates that look at the RSV. This is trivial to do (and I'll probably include them as part of nnselect or in gnus-sum with the other sorting predicates at some point). But we can (and probably should) ignore any sorting issues coming out of gnus-search. Best, Andy ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-05 12:09 ` nnvirtual Andrew Cohen 2017-05-05 12:51 ` nnvirtual Andrew Cohen @ 2017-05-05 15:46 ` Eric Abrahamsen 2017-05-06 0:20 ` nnvirtual Andrew Cohen 1 sibling, 1 reply; 13+ messages in thread From: Eric Abrahamsen @ 2017-05-05 15:46 UTC (permalink / raw) To: ding Andrew Cohen <cohen@bu.edu> writes: >>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: > > Eric> Andrew Cohen <cohen@bu.edu> writes: > >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: > >> > > [...] > > Eric> I'd say we have to assume that, if the functionality is there, > Eric> someone is using it, and combining groups is definitely > Eric> something people do -- see Dovecot's virtual groups, for > Eric> instance. > > Yes, I am not proposing eliminating it, I'm proposing making it more > useful (but I don't know what that means yet:() > > > Eric> As for the large newsgroup prompting, I think it would be > Eric> desirable simply because it makes select groups behave like > Eric> other Gnus groups, and means the limit isn't hardcoded. It > Eric> short, it reduces users' mental friction when using select > Eric> groups. > > Yes, I agree, and my intention is to add it (since its trivial) but > first I want to know if I can make the whole thing more useful. Yeah, I'm not sure either. The two general principles I'm after are: give users the ability to set up commonly-used patterns, and at the same time give them as much flexibility as possible. I mentioned earlier the idea of search "presets", the code for which is pretty much done (and very simple). Each preset is a list of groups to search, and a base search query. The user selects a preset, adds additional query terms, an ephemeral search group is created. I made this because it seemed like the right balance between configuration and flexibility. But if, for example, it were possible to start a search from within a summary buffer, saying "run this search within this group", then presets might not be necessary: users could create permanent select groups incorporating several other groups, then run ad-hoc searches within them. All of which is only to say that I don't really know, either. But Gnus users are a strange bunch, I'd say support as much as possible. > Eric> Random other things: > > Eric> How does sorting work in select groups? Right now gnus-search > Eric> does a pass in `gnus-search-run-query', sorting on rsv, but > Eric> since nearly all engines return a score of 100, it's fairly > Eric> meaningless. > > I agree. I have gone back and forth about whether to eliminate it > entirely. Eliminating it would allow some compression in various > internal variables but this probably doesn't make much of a difference. > But nnselect is currently ignoring the RSV---it is still available so > that sorting functions can use it, however. nnselect is agnostic about > the whole thing. I'd still like to keep it, and the infrastructure to support it. It might become more useful later. Namazu hints that it supports scoring (I really need to get a working installation), imap returns relevancy with FUZZY, and future engines for solr or xapian-based stuff would provide scoring. > All nnselect groups should be sorted exactly as any other gnus group, > and should be as flexible as the usual sorting system. If that isn't > working right its a bug. > So currently to use the RSV for sorting purposes requires constructing > predicates that look at the RSV. This is trivial to do (and I'll > probably include them as part of nnselect or in gnus-sum with the other > sorting predicates at some point). But we can (and probably should) > ignore any sorting issues coming out of gnus-search. Responding to both messages here: if there were a new predicate for sorting on score, I think we'd be fine. Part of me still would like to let users specify sorting strategies on the fly, as part of the search query. Sorting could also be passed to the backends, and combined with limit:n could result in faster queries. At this point that's probably more work than it's worth, though. > Eric> I've provided a count:t meta key in the search language, but > Eric> it isn't used right now. The idea is that, if the user > Eric> includes this key, the search returns a total number of hits > Eric> per engine, rather than actually displaying the results (hits > Eric> per group would be nice, but that information could be > Eric> difficult extract from some engines). It will be fairly easy > Eric> to hot-wire `gnus-group-make-search-group' to return this > Eric> value, and permanent groups could simply strip it out. > > Hmm, the old nnir reported this (it went by too fast to be useful, but > it was always available after the fact in the message buffer). Right, but the idea is that sometimes the count itself is all you want to see. > Eric> If you like, it could also be a cheap way of updating message > Eric> counts in permanent search groups. I note now that, at > Eric> start-up for instance, the permanent groups do not show counts > Eric> until they're entered. > > This is a temporary artifact of my not yet activating the groups at > startup. The counts are all present and ready to be reported (as soon as > the group is created), I just haven't gotten around to keeping the > proper list of nnselect groups to allow activation and re-scanning as a > matter of course. > > Eric> Sending a count request ought to be significantly cheaper than > Eric> actually retrieving results, and might be useful. > > The counts are currently updated without retrieving any headers, and is > therefore quite fast. For example I have a variety of groups with > thousands of messages that are part of the nnselection and the time it > takes to update the group is a fraction of a second. Well, it's probably premature optimization. But it would be pretty simple to implement... ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-05 15:46 ` nnvirtual Eric Abrahamsen @ 2017-05-06 0:20 ` Andrew Cohen 2017-05-07 2:39 ` nnvirtual Eric Abrahamsen 0 siblings, 1 reply; 13+ messages in thread From: Andrew Cohen @ 2017-05-06 0:20 UTC (permalink / raw) To: ding >>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: >> [...] Eric> But if, for example, it were possible to start a search from Eric> within a summary buffer, saying "run this search within this Eric> group", then presets might not be necessary: users could Eric> create permanent select groups incorporating several other Eric> groups, then run ad-hoc searches within them. Umm, this has always been possible with nnir. Does it not work for you? [...] And wrt RSV scoring. Eric> I'd still like to keep it, and the infrastructure to support Eric> it. It might become more useful later. Namazu hints that it Eric> supports scoring (I really need to get a working Eric> installation), imap returns relevancy with FUZZY, and future Eric> engines for solr or xapian-based stuff would provide scoring. I used namazu with its scoring for many years. Aside from trying out the scoring and seeing that it all worked I never used it in practice. I find these scores (from search engines in general) very much not useful. But I decided early on after my back and forth that there is little cost in keeping it, and making the proper predicate is easy and on my todo list. [...] Eric> Responding to both messages here: if there were a new Eric> predicate for sorting on score, I think we'd be fine. Part of Eric> me still would like to let users specify sorting strategies on Eric> the fly, as part of the search query. Sorting could also be Eric> passed to the backends, and combined with limit:n could result Eric> in faster queries. At this point that's probably more work Eric> than it's worth, though. I definitely think that gnus-search should stay away from any kind of sorting. Modulo bugs the gnus strategy of sorting as an overlay on summary buffers works great. Eric> I've provided a count:t meta key in the search language, but Eric> it isn't used right now. The idea is that, if the user Eric> includes this key, the search returns a total number of hits Eric> per engine, rather than actually displaying the results (hits Eric> per group would be nice, but that information could be Eric> difficult extract from some engines). It will be fairly easy Eric> to hot-wire `gnus-group-make-search-group' to return this Eric> value, and permanent groups could simply strip it out. >> >> Hmm, the old nnir reported this (it went by too fast to be >> useful, but it was always available after the fact in the message >> buffer). Eric> Right, but the idea is that sometimes the count itself is all Eric> you want to see. Eric> If you like, it could also be a cheap way of updating message Eric> counts in permanent search groups. I note now that, at Eric> start-up for instance, the permanent groups do not show counts Eric> until they're entered. >> >> This is a temporary artifact of my not yet activating the groups >> at startup. The counts are all present and ready to be reported >> (as soon as the group is created), I just haven't gotten around >> to keeping the proper list of nnselect groups to allow activation >> and re-scanning as a matter of course. >> Eric> Sending a count request ought to be significantly cheaper than Eric> actually retrieving results, and might be useful. >> >> The counts are currently updated without retrieving any headers, >> and is therefore quite fast. For example I have a variety of >> groups with thousands of messages that are part of the >> nnselection and the time it takes to update the group is a >> fraction of a second. Eric> Well, it's probably premature optimization. But it would be Eric> pretty simple to implement... I don't think its a real optimization. All the time now is taken in setting up the connection; the search itself would take the same time, the only difference being you might not be returning the list of UIDs. But parsing the UIDs takes essentially no time. Unless you had something else in mind? Also nnselect isn't about searching---there are many other ways that selection lists can be produced. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-06 0:20 ` nnvirtual Andrew Cohen @ 2017-05-07 2:39 ` Eric Abrahamsen 0 siblings, 0 replies; 13+ messages in thread From: Eric Abrahamsen @ 2017-05-07 2:39 UTC (permalink / raw) To: ding Andrew Cohen <cohen@bu.edu> writes: >>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: > > >>>>>>> "Eric" == Eric Abrahamsen <eric@ericabrahamsen.net> writes: > >> > > [...] > > Eric> But if, for example, it were possible to start a search from > Eric> within a summary buffer, saying "run this search within this > Eric> group", then presets might not be necessary: users could > Eric> create permanent select groups incorporating several other > Eric> groups, then run ad-hoc searches within them. > > Umm, this has always been possible with nnir. Does it not work for you? Whoops, that function did not survive the nnir->gnus-search transformation. I'll put it back in. I think that means I can just drop search presets. > And wrt RSV scoring. > > Eric> I'd still like to keep it, and the infrastructure to support > Eric> it. It might become more useful later. Namazu hints that it > Eric> supports scoring (I really need to get a working > Eric> installation), imap returns relevancy with FUZZY, and future > Eric> engines for solr or xapian-based stuff would provide scoring. > > I used namazu with its scoring for many years. Aside from trying out the > scoring and seeing that it all worked I never used it in practice. I > find these scores (from search engines in general) very much not > useful. But I decided early on after my back and forth that there is > little cost in keeping it, and making the proper predicate is easy and > on my todo list. I see it's already in. BTW, that patch seems to have introduced a circular dependency between gnus-sum->nnselect->gnus-art->gnus-sum. I can't get it to compile. On a related note, I'm having a hard time finding a place to put the `gnus-group-make-search-group' functions: the call to nnselect-categorize also tends to create circular dependencies. Something to think about as this whole thing gets closer to completion. > Eric> Responding to both messages here: if there were a new > Eric> predicate for sorting on score, I think we'd be fine. Part of > Eric> me still would like to let users specify sorting strategies on > Eric> the fly, as part of the search query. Sorting could also be > Eric> passed to the backends, and combined with limit:n could result > Eric> in faster queries. At this point that's probably more work > Eric> than it's worth, though. > > I definitely think that gnus-search should stay away from any kind of > sorting. Modulo bugs the gnus strategy of sorting as an overlay on > summary buffers works great. That's fine, I'm happy to drop the sorting stuff. > I don't think its a real optimization. All the time now is taken in > setting up the connection; the search itself would take the same time, > the only difference being you might not be returning the list of > UIDs. But parsing the UIDs takes essentially no time. Unless you had > something else in mind? No, it was just the thought that the engines might have an easier time returning a simple count than the results themselves (other engines return more than just UIDs). But I think I was just getting caught up in trying to expose and make use of everything the engines offer -- not really necessary. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-03 12:31 nnvirtual Andrew Cohen 2017-05-03 18:00 ` nnvirtual Eric Abrahamsen @ 2017-05-12 15:38 ` Steinar Bang 2017-05-12 17:42 ` nnvirtual Adam Sjøgren 2018-04-11 19:42 ` nnvirtual Lars Ingebrigtsen 2 siblings, 1 reply; 13+ messages in thread From: Steinar Bang @ 2017-05-12 15:38 UTC (permalink / raw) To: ding >>>>> Andrew Cohen <cohen@bu.edu>: > But search is now defunct (at least for the moment). And I must say, > using gmane without it kind of sucks. The search functionality in gmane is indeed missed. (But having gmane without this functionality is better than not having gmane at all...) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-12 15:38 ` nnvirtual Steinar Bang @ 2017-05-12 17:42 ` Adam Sjøgren 2017-05-13 0:17 ` nnvirtual Harry Putnam 0 siblings, 1 reply; 13+ messages in thread From: Adam Sjøgren @ 2017-05-12 17:42 UTC (permalink / raw) To: ding Steinar writes: > (But having gmane without this functionality is better than not having > gmane at all...) I could agree more! It is kind of disheartening, though that, nothing [visible, anyway] has happened since september last year. I left a comment on the home.gmane.org blog a month ago, and it is still awaiting moderation. Best regards, Adam -- "Calvin, would you set the table please?" "Mm... I Adam Sjøgren don't think so. I'm not enthusiastic about setting asjo@koldfront.dk the table." ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-12 17:42 ` nnvirtual Adam Sjøgren @ 2017-05-13 0:17 ` Harry Putnam 0 siblings, 0 replies; 13+ messages in thread From: Harry Putnam @ 2017-05-13 0:17 UTC (permalink / raw) To: ding asjo@koldfront.dk (Adam Sjøgren) writes: > Steinar writes: > >> (But having gmane without this functionality is better than not having >> gmane at all...) > > I could agree more! > > It is kind of disheartening, though that, nothing [visible, anyway] has > happened since september last year. gmane is and has been an institution for a while now. Kind of scary like when the big insurance co's and banks started failing. If only gnus experts could get a government bailout .. to pay for coders. > I left a comment on the home.gmane.org blog a month ago, and it is still > awaiting moderation. Just a thought on that... maybe way off in left field and not appropriate but I wonder if stuff like this is welcome on `gmane.discuss'... I don't see too much traffic there ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: nnvirtual 2017-05-03 12:31 nnvirtual Andrew Cohen 2017-05-03 18:00 ` nnvirtual Eric Abrahamsen 2017-05-12 15:38 ` nnvirtual Steinar Bang @ 2018-04-11 19:42 ` Lars Ingebrigtsen 2 siblings, 0 replies; 13+ messages in thread From: Lars Ingebrigtsen @ 2018-04-11 19:42 UTC (permalink / raw) To: Andrew Cohen; +Cc: ding Andrew Cohen <cohen@bu.edu> writes: > Gnus has had the nnvirtual backend, allowing several newsgroups to be > read as a single, combined newsgroup, for ages. But is anyone using it? > And if so, how and why? I use it to combine all the really low-volume gnu.* newsgroups. Because... er... uhm... I'm not sure. :-) I just do. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2018-04-11 19:42 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-05-03 12:31 nnvirtual Andrew Cohen 2017-05-03 18:00 ` nnvirtual Eric Abrahamsen 2017-05-04 1:06 ` nnvirtual Andrew Cohen 2017-05-05 4:30 ` nnvirtual Eric Abrahamsen 2017-05-05 12:09 ` nnvirtual Andrew Cohen 2017-05-05 12:51 ` nnvirtual Andrew Cohen 2017-05-05 15:46 ` nnvirtual Eric Abrahamsen 2017-05-06 0:20 ` nnvirtual Andrew Cohen 2017-05-07 2:39 ` nnvirtual Eric Abrahamsen 2017-05-12 15:38 ` nnvirtual Steinar Bang 2017-05-12 17:42 ` nnvirtual Adam Sjøgren 2017-05-13 0:17 ` nnvirtual Harry Putnam 2018-04-11 19:42 ` nnvirtual Lars Ingebrigtsen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).