* how can I keep gnus from Hiding All Threads when open a group @ 2017-06-02 20:39 Harry Putnam 2017-06-03 3:58 ` Eric Abrahamsen 0 siblings, 1 reply; 7+ messages in thread From: Harry Putnam @ 2017-06-02 20:39 UTC (permalink / raw) To: info-gnus-english I notice when I open a group, gnus hides all threads seemingly automatically. But when opening really large groups; 12,000 and up, even 50,000 msgs that process of hiding all threads can become a hefty time sink. Even when unplugged and all the messages are already downloaded. How could I go about telling gnus to leave the threads unhidden or would that have the effect I'm hoping for... i.e. speeding up opening really large groups. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: how can I keep gnus from Hiding All Threads when open a group 2017-06-02 20:39 how can I keep gnus from Hiding All Threads when open a group Harry Putnam @ 2017-06-03 3:58 ` Eric Abrahamsen 2017-06-03 17:37 ` Harry Putnam 0 siblings, 1 reply; 7+ messages in thread From: Eric Abrahamsen @ 2017-06-03 3:58 UTC (permalink / raw) To: info-gnus-english Harry Putnam <reader@newsguy.com> writes: > I notice when I open a group, gnus hides all threads seemingly > automatically. > > But when opening really large groups; 12,000 and up, even 50,000 msgs > that process of hiding all threads can become a hefty time sink. > > Even when unplugged and all the messages are already downloaded. > > How could I go about telling gnus to leave the threads unhidden or > would that have the effect I'm hoping for... i.e. speeding up opening > really large groups. I'm not quite sure what you mean by "hiding" -- do you mean the threads are folded? Like you only see the top message in the thread and have to expand it to see the rest? Anyway, I doubt this is the source of the slowdown -- asking Gnus to display 50,000 messages in a summary buffer is going to be slow no matter what. Turning off threading altogether should help some, but that's all I can say. Eric ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: how can I keep gnus from Hiding All Threads when open a group 2017-06-03 3:58 ` Eric Abrahamsen @ 2017-06-03 17:37 ` Harry Putnam 2017-06-04 1:31 ` Eric Abrahamsen 0 siblings, 1 reply; 7+ messages in thread From: Harry Putnam @ 2017-06-03 17:37 UTC (permalink / raw) To: info-gnus-english Eric Abrahamsen <eric@ericabrahamsen.net> writes: > Harry Putnam <reader@newsguy.com> writes: > >> I notice when I open a group, gnus hides all threads seemingly >> automatically. >> >> But when opening really large groups; 12,000 and up, even 50,000 msgs >> that process of hiding all threads can become a hefty time sink. >> >> Even when unplugged and all the messages are already downloaded. >> >> How could I go about telling gnus to leave the threads unhidden or >> would that have the effect I'm hoping for... i.e. speeding up opening >> really large groups. > > I'm not quite sure what you mean by "hiding" -- do you mean the threads > are folded? Like you only see the top message in the thread and have to > expand it to see the rest? Yes. I got the terminology from gnus itself.... when I open a large group After showing a message about generating the summary buffer, it shows a message of `Hiding all threads' and then a count usually only displays the thousands. > Anyway, I doubt this is the source of the slowdown -- asking Gnus to > display 50,000 messages in a summary buffer is going to be slow no > matter what. Turning off threading altogether should help some, but > that's all I can say. I do realize that groups of that size are really really slow. I have some experience in that area. However unless the `hiding all threads' message is misleading and doesn't really have anything to do with what is happening, I can see the wait growing longer by watching `hiding all threads'; 1000 ... ...2000 ....3000 the waits between are relatively short but after about 4000 the waits between ouput grow exponentially. Getting up to 7/8 thousand the wait between 7 and 8000 is quite lengthy. I haven't timed it or I'd post the times. That's what gave me the idea of cutting out the thread hiding.... Also, must of what I do in there means I have to SHOW threads anyway `T S' so I have to undo the Hiding anyway. Writing that made me think it might be a setting of my own doing. Can one set how threads are displayed when a group is opened? ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: how can I keep gnus from Hiding All Threads when open a group 2017-06-03 17:37 ` Harry Putnam @ 2017-06-04 1:31 ` Eric Abrahamsen 2017-06-16 21:28 ` Harry Putnam 0 siblings, 1 reply; 7+ messages in thread From: Eric Abrahamsen @ 2017-06-04 1:31 UTC (permalink / raw) To: info-gnus-english Harry Putnam <reader@newsguy.com> writes: > Eric Abrahamsen <eric@ericabrahamsen.net> writes: > >> Harry Putnam <reader@newsguy.com> writes: >> >>> I notice when I open a group, gnus hides all threads seemingly >>> automatically. >>> >>> But when opening really large groups; 12,000 and up, even 50,000 msgs >>> that process of hiding all threads can become a hefty time sink. >>> >>> Even when unplugged and all the messages are already downloaded. >>> >>> How could I go about telling gnus to leave the threads unhidden or >>> would that have the effect I'm hoping for... i.e. speeding up opening >>> really large groups. >> >> I'm not quite sure what you mean by "hiding" -- do you mean the threads >> are folded? Like you only see the top message in the thread and have to >> expand it to see the rest? > > Yes. > > I got the terminology from gnus itself.... when I open a large group > After showing a message about generating the summary buffer, it shows > a message of `Hiding all threads' and then a count usually only > displays the thousands. Ah I see. I don't use this and didn't realize it was the normal term. >> Anyway, I doubt this is the source of the slowdown -- asking Gnus to >> display 50,000 messages in a summary buffer is going to be slow no >> matter what. Turning off threading altogether should help some, but >> that's all I can say. > > I do realize that groups of that size are really really slow. > I have some experience in that area. > > However unless the `hiding all threads' message is misleading and > doesn't really have anything to do with what is happening, I can see > the wait growing longer by watching `hiding all threads'; 1000 ... > ...2000 ....3000 the waits between are relatively short but after about > 4000 the waits between ouput grow exponentially. Getting up to 7/8 > thousand the wait between 7 and 8000 is quite lengthy. I haven't > timed it or I'd post the times. > > That's what gave me the idea of cutting out the thread hiding.... Also, > must of what I do in there means I have to SHOW threads anyway `T S' > so I have to undo the Hiding anyway. > > Writing that made me think it might be a setting of my own doing. > > Can one set how threads are displayed when a group is opened? Okay, well that certainly makes it sound like thread hiding really is the source of the problem. It looks like setting `gnus-thread-hide-subtree' to nil will do what you want. But I'd definitely recommend playing with `gnus-show-threads' and `gnus-fetch-old-headers' (and maybe try `gnus-build-sparse-threads'), and see which combination of settings gets you the fastest load. No matter what, I think the main recommendation would be to try to open group buffers with fewer messages! Is it really necessary to display tens of thousands of messages each time? E ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: how can I keep gnus from Hiding All Threads when open a group 2017-06-04 1:31 ` Eric Abrahamsen @ 2017-06-16 21:28 ` Harry Putnam 2017-06-17 1:35 ` Eric Abrahamsen 0 siblings, 1 reply; 7+ messages in thread From: Harry Putnam @ 2017-06-16 21:28 UTC (permalink / raw) To: info-gnus-english Eric Abrahamsen <eric@ericabrahamsen.net> writes: >>> Anyway, I doubt this is the source of the slowdown -- asking Gnus to >>> display 50,000 messages in a summary buffer is going to be slow no >>> matter what. Turning off threading altogether should help some, but >>> that's all I can say. Harry replied: >> I do realize that groups of that size are really really slow. >> I have some experience in that area. >> >> However unless the `hiding all threads' message is misleading and >> doesn't really have anything to do with what is happening, I can see >> the wait growing longer by watching `hiding all threads'; 1000 ... >> ...2000 ....3000 the waits between are relatively short but after about >> 4000 the waits between ouput grow exponentially. Getting up to 7/8 >> thousand the wait between 7 and 8000 is quite lengthy. I haven't >> timed it or I'd post the times. >> >> That's what gave me the idea of cutting out the thread hiding.... Also, >> must of what I do in there means I have to SHOW threads anyway `T S' >> so I have to undo the Hiding anyway. [...] > Okay, well that certainly makes it sound like thread hiding really is > the source of the problem. It looks like setting > `gnus-thread-hide-subtree' to nil will do what you want. But I'd That change to `nil' seems not to have made much (if any ) difference.. In fact I still see that message about hiding threads and progressivly longer waits. > definitely recommend playing with `gnus-show-threads' and > `gnus-fetch-old-headers' (and maybe try `gnus-build-sparse-threads'), > and see which combination of settings gets you the fastest load. Haven't tried messing with those yet. As you might imagine and in line with your comment below, doing so means lots of wait time chking if a change in those vars helps. Plus I've kind of gotten past the necessity. > No matter what, I think the main recommendation would be to try to open > group buffers with fewer messages! Is it really necessary to display > tens of thousands of messages each time? Ahh yes, and so I have moved past the need to open so many messages at once, and thereby not facing the threading issues in such a massive setting. You see, it seems sort of required (opening many many msgs at once )to initially get significant amounts of postings accumulated into the agent to do what I'm really working on.. which is have a large database of technical information from usenet on disk. Some kinds of searching just cannot be done over the internet. For example; no online provider in there right mind can allow regex searching.. it would tie them up if used to any extent. (The effort to acquire postings is preparatory to try to learn how to use the new search possibilities that yourself, Andrew C and others have been working on currently as noted in several continuing threads on `gmane.emacs.gnus.general') The problem is that if I asked gnus to just download whatever is on the gmane or enews.newsguy.com spools for a group..., in the most interesting ones there may be several hundred thousand msgs going back to early 2000, in some cases I've gotten clear into the late 90s. So I set about acquiring around 50,000 or so where possible. Ended up with 8 grps or so, but most are between 10 and 40 thousand with a few above 70,000. Hence the spate of time spent opening many many headers to be able to mark and download 50,000 or so in bunches of 5,000 or so, and over time, so as not to get blackballed off gmane for abusive downloading. Maybe some kind of trickery with agent categories could have made this a bit or even a lot easier... I just set the predicate to `true' on all agentized groups and did the downloading manually I have 38 groups agentized over three servers .. quite minor league compared to a real news spool... I ended up with 1,033,195 posts but now only have to keep up with what is newly posted. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: how can I keep gnus from Hiding All Threads when open a group 2017-06-16 21:28 ` Harry Putnam @ 2017-06-17 1:35 ` Eric Abrahamsen 2017-06-17 18:24 ` Harry Putnam 0 siblings, 1 reply; 7+ messages in thread From: Eric Abrahamsen @ 2017-06-17 1:35 UTC (permalink / raw) To: info-gnus-english Harry Putnam <reader@newsguy.com> writes: [...] >> No matter what, I think the main recommendation would be to try to open >> group buffers with fewer messages! Is it really necessary to display >> tens of thousands of messages each time? [...] > (The effort to acquire postings is preparatory to try to learn how to > use the new search possibilities that yourself, Andrew C and others > have been working on currently as noted in several continuing threads > on `gmane.emacs.gnus.general') > > The problem is that if I asked gnus to just download whatever is on > the gmane or enews.newsguy.com spools for a group..., in the most > interesting ones there may be several hundred thousand msgs going back > to early 2000, in some cases I've gotten clear into the late 90s. > > So I set about acquiring around 50,000 or so where possible. Ended up > with 8 grps or so, but most are between 10 and 40 thousand with a few > above 70,000. > > Hence the spate of time spent opening many many headers to be able to > mark and download 50,000 or so in bunches of 5,000 or so, and over > time, so as not to get blackballed off gmane for abusive downloading. > > Maybe some kind of trickery with agent categories could have made this > a bit or even a lot easier... I just set the predicate to `true' on > all agentized groups and did the downloading manually > > I have 38 groups agentized over three servers .. quite minor league > compared to a real news spool... > > I ended up with 1,033,195 posts but now only have to keep up with what > is newly posted. Yeesh... At this point it might make more sense just to install your own local LeafNode server and do nntp yourself, don't you think? I'm sorry I can't speak to your specific questions about the agent -- I only use the agent with my gmane backend (all the other backends are local), and there I just agentize the whole server, I haven't tried anything finer-grained than that. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: how can I keep gnus from Hiding All Threads when open a group 2017-06-17 1:35 ` Eric Abrahamsen @ 2017-06-17 18:24 ` Harry Putnam 0 siblings, 0 replies; 7+ messages in thread From: Harry Putnam @ 2017-06-17 18:24 UTC (permalink / raw) To: info-gnus-english Eric Abrahamsen <eric@ericabrahamsen.net> writes: > Harry Putnam <reader@newsguy.com> writes: > > [...] > >>> No matter what, I think the main recommendation would be to try to open >>> group buffers with fewer messages! Is it really necessary to display >>> tens of thousands of messages each time? > > [...] > >> (The effort to acquire postings is preparatory to try to learn how to >> use the new search possibilities that yourself, Andrew C and others >> have been working on currently as noted in several continuing threads >> on `gmane.emacs.gnus.general') >> >> The problem is that if I asked gnus to just download whatever is on >> the gmane or enews.newsguy.com spools for a group..., in the most >> interesting ones there may be several hundred thousand msgs going back >> to early 2000, in some cases I've gotten clear into the late 90s. >> >> So I set about acquiring around 50,000 or so where possible. Ended up >> with 8 grps or so, but most are between 10 and 40 thousand with a few >> above 70,000. >> >> Hence the spate of time spent opening many many headers to be able to >> mark and download 50,000 or so in bunches of 5,000 or so, and over >> time, so as not to get blackballed off gmane for abusive downloading. >> >> Maybe some kind of trickery with agent categories could have made this >> a bit or even a lot easier... I just set the predicate to `true' on >> all agentized groups and did the downloading manually >> >> I have 38 groups agentized over three servers .. quite minor league >> compared to a real news spool... >> >> I ended up with 1,033,195 posts but now only have to keep up with what >> is newly posted. > > Yeesh... At this point it might make more sense just to install your own > local LeafNode server and do nntp yourself, don't you think? Yes, my thoughts exactly... I was just studying up on slrnpull. Something I used a bit long ago... never really fiddled with leafnode.. maybe its a little simpler? The stumbling block with either is probably how to slurp the database gnus has already generated. But also, how would I fit news pulled that way, into the search features being coded in gnus lately? > I'm sorry I can't speak to your specific questions about the agent -- I > only use the agent with my gmane backend (all the other backends are > local), and there I just agentize the whole server, I haven't tried > anything finer-grained than that. I'm sorry to be bending your ear so extensively but not getting much input on ... gnus.user Perhaps if you knew the depths of my ignorance you might know you probably can fill in a few things. Do you know if there is supposed to be (in orginal intent) a difference between having groups under an agentized server... and having not just the server agentized but also specific groups agentized under that server as well? I do know that the list of agentized groups (by category) in News/agent/lib/categories does not appear to include any of my gmane groups that have not been agentized, even though the gmane server is agentized. Not sure one could even use a category if the group itself was not agentized. For example... With the server agentized... Do you use either `J s' (gnus-agent-fetch-session) (Fetch all articles and headers that are eligible for fetching) OR `J u (gnus-agent-fetch-groups N) (Put all new articles in the current groups into the Agent) in group buffer, after a scan for new news, to download the new messages? The `J u' doc string is pretty easy to understand ... all NEW msgs in CURRENT group But for some reason Larsi (who writes really good documentation) or someone else felt it was necessary to use a slightly different description for `J s' 'articles and headers that are eligible for fetching' I'm guessing the description uses different language because the action is different in some way... That is, not just `New' msgs but maybe something more inclusive. Maybe something like all un-downloaded msgs that conform to a specific category, which might be something of a disaster in groups with well over 100.000 msgs that have not been downloaded yet. I ask because the times I tried `J s' or `J u' (Even on just the current group) it produces a massive lengthy read of something.. I'm not sure if the visible count being shown is characters or lines but it gets up to over 120,000 eventually but I've never let it finish... I was afraid it was trying to start from the very oldest posts on the gmane spool and start downloading from there. Which might be a good way to loose access to the new gmane. I'd like to know what is being read in that case above? You say, you do nothing finer grained with the agent than an agentized server.. so do you mean that you download all groups you have subscribed to on that server? Unless you only subscribe to a few groups, that seems like it would become a very lot of messages at some point. How do you handle the downloading? In terms of cmds used? Do you do batching to keep up with new posts and actually go unplugged at times? I noticed in my one attempt (recently) at using the batch command shown in the manual: You can run a complete batch command from the command line with the following incantation: #!/bin/sh emacs -batch -l ~/.emacs -l ~/.gnus.el -f gnus-agent-batch >/dev/null 2>&1 that gnus started downloading from groups under an agentized server that were not themselves agentized as well as groups that were. I ask all this because it is really time consuming and somewhat difficult to set up real experimenting to discover the basic actions one should expect when using the agent. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-06-17 18:24 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-06-02 20:39 how can I keep gnus from Hiding All Threads when open a group Harry Putnam 2017-06-03 3:58 ` Eric Abrahamsen 2017-06-03 17:37 ` Harry Putnam 2017-06-04 1:31 ` Eric Abrahamsen 2017-06-16 21:28 ` Harry Putnam 2017-06-17 1:35 ` Eric Abrahamsen 2017-06-17 18:24 ` Harry Putnam
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).