how can I keep gnus from Hiding All Threads when open a group

how can I keep gnus from Hiding All Threads when open a group

[Harry Putnam]

I notice when I open a group, gnus hides all threads seemingly
automatically.

But when opening really large groups; 12,000 and up, even 50,000 msgs
that process of hiding all threads can become a hefty time sink.

Even when unplugged and all the messages are already downloaded.

How could I go about telling gnus to leave the threads unhidden or
would that have the effect I'm hoping for... i.e. speeding up opening
really large groups.

Re: how can I keep gnus from Hiding All Threads when open a group

[Eric Abrahamsen]

Harry Putnam <reader@newsguy.com> writes:

> I notice when I open a group, gnus hides all threads seemingly
> automatically.
>
> But when opening really large groups; 12,000 and up, even 50,000 msgs
> that process of hiding all threads can become a hefty time sink.
>
> Even when unplugged and all the messages are already downloaded.
>
> How could I go about telling gnus to leave the threads unhidden or
> would that have the effect I'm hoping for... i.e. speeding up opening
> really large groups.

I'm not quite sure what you mean by "hiding" -- do you mean the threads
are folded? Like you only see the top message in the thread and have to
expand it to see the rest?

Anyway, I doubt this is the source of the slowdown -- asking Gnus to
display 50,000 messages in a summary buffer is going to be slow no
matter what. Turning off threading altogether should help some, but
that's all I can say.

Eric

Re: how can I keep gnus from Hiding All Threads when open a group

[Harry Putnam]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> I notice when I open a group, gnus hides all threads seemingly
>> automatically.
>>
>> But when opening really large groups; 12,000 and up, even 50,000 msgs
>> that process of hiding all threads can become a hefty time sink.
>>
>> Even when unplugged and all the messages are already downloaded.
>>
>> How could I go about telling gnus to leave the threads unhidden or
>> would that have the effect I'm hoping for... i.e. speeding up opening
>> really large groups.
>
> I'm not quite sure what you mean by "hiding" -- do you mean the threads
> are folded? Like you only see the top message in the thread and have to
> expand it to see the rest?

Yes.

I got the terminology from gnus itself.... when I open a large group
After showing a message about generating the summary buffer, it shows
a message of `Hiding all threads' and then a count usually only
displays the thousands.

> Anyway, I doubt this is the source of the slowdown -- asking Gnus to
> display 50,000 messages in a summary buffer is going to be slow no
> matter what. Turning off threading altogether should help some, but
> that's all I can say.

I do realize that groups of that size are really really slow.
I have some experience in that area.

However unless the `hiding all threads' message is misleading and
doesn't really have anything to do with what is happening, I can see
the wait growing longer by watching `hiding all threads'; 1000 ...
...2000 ....3000 the waits between are relatively short but after about
4000 the waits between ouput grow exponentially.  Getting up to 7/8
thousand the wait between 7 and 8000 is quite lengthy.  I haven't
timed it or I'd post the times.

That's what gave me the idea of cutting out the thread hiding.... Also,
must of what I do in there means I have to SHOW threads anyway `T S'
so I have to undo the Hiding anyway.

Writing that made me think it might be a setting of my own doing.

Can one set how threads are displayed when a group is opened?

Re: how can I keep gnus from Hiding All Threads when open a group

[Eric Abrahamsen]

Harry Putnam <reader@newsguy.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Harry Putnam <reader@newsguy.com> writes:
>>
>>> I notice when I open a group, gnus hides all threads seemingly
>>> automatically.
>>>
>>> But when opening really large groups; 12,000 and up, even 50,000 msgs
>>> that process of hiding all threads can become a hefty time sink.
>>>
>>> Even when unplugged and all the messages are already downloaded.
>>>
>>> How could I go about telling gnus to leave the threads unhidden or
>>> would that have the effect I'm hoping for... i.e. speeding up opening
>>> really large groups.
>>
>> I'm not quite sure what you mean by "hiding" -- do you mean the threads
>> are folded? Like you only see the top message in the thread and have to
>> expand it to see the rest?
>
> Yes.
>
> I got the terminology from gnus itself.... when I open a large group
> After showing a message about generating the summary buffer, it shows
> a message of `Hiding all threads' and then a count usually only
> displays the thousands.

Ah I see. I don't use this and didn't realize it was the normal term.

>> Anyway, I doubt this is the source of the slowdown -- asking Gnus to
>> display 50,000 messages in a summary buffer is going to be slow no
>> matter what. Turning off threading altogether should help some, but
>> that's all I can say.
>
> I do realize that groups of that size are really really slow.
> I have some experience in that area.
>
> However unless the `hiding all threads' message is misleading and
> doesn't really have anything to do with what is happening, I can see
> the wait growing longer by watching `hiding all threads'; 1000 ...
> ...2000 ....3000 the waits between are relatively short but after about
> 4000 the waits between ouput grow exponentially.  Getting up to 7/8
> thousand the wait between 7 and 8000 is quite lengthy.  I haven't
> timed it or I'd post the times.
>
> That's what gave me the idea of cutting out the thread hiding.... Also,
> must of what I do in there means I have to SHOW threads anyway `T S'
> so I have to undo the Hiding anyway.
>
> Writing that made me think it might be a setting of my own doing.
>
> Can one set how threads are displayed when a group is opened?

Okay, well that certainly makes it sound like thread hiding really is
the source of the problem. It looks like setting
`gnus-thread-hide-subtree' to nil will do what you want. But I'd
definitely recommend playing with `gnus-show-threads' and
`gnus-fetch-old-headers' (and maybe try `gnus-build-sparse-threads'),
and see which combination of settings gets you the fastest load.

No matter what, I think the main recommendation would be to try to open
group buffers with fewer messages! Is it really necessary to display
tens of thousands of messages each time?

E

Re: how can I keep gnus from Hiding All Threads when open a group

[Harry Putnam]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>>> Anyway, I doubt this is the source of the slowdown -- asking Gnus to
>>> display 50,000 messages in a summary buffer is going to be slow no
>>> matter what. Turning off threading altogether should help some, but
>>> that's all I can say.

Harry replied:
>> I do realize that groups of that size are really really slow.
>> I have some experience in that area.
>>
>> However unless the `hiding all threads' message is misleading and
>> doesn't really have anything to do with what is happening, I can see
>> the wait growing longer by watching `hiding all threads'; 1000 ...
>> ...2000 ....3000 the waits between are relatively short but after about
>> 4000 the waits between ouput grow exponentially.  Getting up to 7/8
>> thousand the wait between 7 and 8000 is quite lengthy.  I haven't
>> timed it or I'd post the times.
>>
>> That's what gave me the idea of cutting out the thread hiding.... Also,
>> must of what I do in there means I have to SHOW threads anyway `T S'
>> so I have to undo the Hiding anyway.

[...]

> Okay, well that certainly makes it sound like thread hiding really is
> the source of the problem. It looks like setting
> `gnus-thread-hide-subtree' to nil will do what you want. But I'd

That change to `nil' seems not to have made much (if any )
difference.. In fact I still see that message about hiding threads and
progressivly longer waits.

> definitely recommend playing with `gnus-show-threads' and
> `gnus-fetch-old-headers' (and maybe try `gnus-build-sparse-threads'),
> and see which combination of settings gets you the fastest load.

Haven't tried messing with those yet.  As you might imagine and in
line with your comment below, doing so means lots of wait time chking
if a change in those vars helps. Plus I've kind of gotten past the
necessity.

> No matter what, I think the main recommendation would be to try to open
> group buffers with fewer messages! Is it really necessary to display
> tens of thousands of messages each time?

Ahh yes, and so I have moved past the need to open so many messages at
once, and thereby not facing the threading issues in such a massive
setting.

You see, it seems sort of required (opening many many msgs at once )to
initially get significant amounts of postings accumulated into the
agent to do what I'm really working on.. which is have a large
database of technical information from usenet on disk.  Some kinds of
searching just cannot be done over the internet.

For example; no online provider in there right mind can allow regex
searching.. it would tie them up if used to any extent.

(The effort to acquire postings is preparatory to try to learn how to
use the new search possibilities that yourself, Andrew C and others
have been working on currently as noted in several continuing threads
on `gmane.emacs.gnus.general')

The problem is that if I asked gnus to just download whatever is on
the gmane or enews.newsguy.com spools for a group..., in the most
interesting ones there may be several hundred thousand msgs going back
to early 2000, in some cases I've gotten clear into the late 90s.

So I set about acquiring around 50,000 or so where possible. Ended up
with 8 grps or so, but most are between 10 and 40 thousand with a few
above 70,000.  

Hence the spate of time spent opening many many headers to be able to
mark and download 50,000 or so in bunches of 5,000 or so, and over
time, so as not to get blackballed off gmane for abusive downloading.

Maybe some kind of trickery with agent categories could have made this
a bit or even a lot easier... I just set the predicate to `true' on
all agentized groups and did the downloading manually

I have 38 groups agentized over three servers .. quite minor league
compared to a real news spool...

I ended up with 1,033,195 posts but now only have to keep up with what
is newly posted.

Re: how can I keep gnus from Hiding All Threads when open a group

[Eric Abrahamsen]

Harry Putnam <reader@newsguy.com> writes:

[...]

>> No matter what, I think the main recommendation would be to try to open
>> group buffers with fewer messages! Is it really necessary to display
>> tens of thousands of messages each time?

[...]

> (The effort to acquire postings is preparatory to try to learn how to
> use the new search possibilities that yourself, Andrew C and others
> have been working on currently as noted in several continuing threads
> on `gmane.emacs.gnus.general')
>
> The problem is that if I asked gnus to just download whatever is on
> the gmane or enews.newsguy.com spools for a group..., in the most
> interesting ones there may be several hundred thousand msgs going back
> to early 2000, in some cases I've gotten clear into the late 90s.
>
> So I set about acquiring around 50,000 or so where possible. Ended up
> with 8 grps or so, but most are between 10 and 40 thousand with a few
> above 70,000.  
>
> Hence the spate of time spent opening many many headers to be able to
> mark and download 50,000 or so in bunches of 5,000 or so, and over
> time, so as not to get blackballed off gmane for abusive downloading.
>
> Maybe some kind of trickery with agent categories could have made this
> a bit or even a lot easier... I just set the predicate to `true' on
> all agentized groups and did the downloading manually
>
> I have 38 groups agentized over three servers .. quite minor league
> compared to a real news spool...
>
> I ended up with 1,033,195 posts but now only have to keep up with what
> is newly posted.

Yeesh... At this point it might make more sense just to install your own
local LeafNode server and do nntp yourself, don't you think?

I'm sorry I can't speak to your specific questions about the agent -- I
only use the agent with my gmane backend (all the other backends are
local), and there I just agentize the whole server, I haven't tried
anything finer-grained than that.

Re: how can I keep gnus from Hiding All Threads when open a group

[Harry Putnam]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
> [...]
>

>>> No matter what, I think the main recommendation would be to try to open
>>> group buffers with fewer messages! Is it really necessary to display
>>> tens of thousands of messages each time?
>
> [...]
>
>> (The effort to acquire postings is preparatory to try to learn how to
>> use the new search possibilities that yourself, Andrew C and others
>> have been working on currently as noted in several continuing threads
>> on `gmane.emacs.gnus.general')
>>
>> The problem is that if I asked gnus to just download whatever is on
>> the gmane or enews.newsguy.com spools for a group..., in the most
>> interesting ones there may be several hundred thousand msgs going back
>> to early 2000, in some cases I've gotten clear into the late 90s.
>>
>> So I set about acquiring around 50,000 or so where possible. Ended up
>> with 8 grps or so, but most are between 10 and 40 thousand with a few
>> above 70,000.  
>>
>> Hence the spate of time spent opening many many headers to be able to
>> mark and download 50,000 or so in bunches of 5,000 or so, and over
>> time, so as not to get blackballed off gmane for abusive downloading.
>>
>> Maybe some kind of trickery with agent categories could have made this
>> a bit or even a lot easier... I just set the predicate to `true' on
>> all agentized groups and did the downloading manually
>>
>> I have 38 groups agentized over three servers .. quite minor league
>> compared to a real news spool...
>>
>> I ended up with 1,033,195 posts but now only have to keep up with what
>> is newly posted.
>
> Yeesh... At this point it might make more sense just to install your own
> local LeafNode server and do nntp yourself, don't you think?

Yes, my thoughts exactly... I was just studying up on slrnpull.
Something I used a bit long ago... never really fiddled with
leafnode.. maybe its a little simpler?

The stumbling block with either is probably how to slurp the database
gnus has already generated.

But also, how would I fit news pulled that way, into the search
features being coded in gnus lately?

> I'm sorry I can't speak to your specific questions about the agent -- I
> only use the agent with my gmane backend (all the other backends are
> local), and there I just agentize the whole server, I haven't tried
> anything finer-grained than that.

I'm sorry to be bending your ear so extensively but not getting much
input on ... gnus.user

Perhaps if you knew the depths of my ignorance you might know you
probably can fill in a few things.

Do you know if there is supposed to be (in orginal intent) a
difference between having groups under an agentized server... and
having not just the server agentized but also specific groups
agentized under that server as well?

I do know that the list of agentized groups (by category) in
News/agent/lib/categories does not appear to include any of my gmane
groups that have not been agentized, even though the gmane server is
agentized.  Not sure one could even use a category if the group itself
was not agentized.

For example... With the server agentized...

Do you use either `J s' (gnus-agent-fetch-session)
  (Fetch all articles and headers that are eligible for fetching)

  OR
                   `J u (gnus-agent-fetch-groups N)
   (Put all new articles in the current groups into the Agent)
   
in group buffer, after a scan for new news, to download the new
messages?

The `J u' doc string is pretty easy to understand ... all NEW msgs in
CURRENT group

But for some reason Larsi (who writes really good documentation) or
someone else felt it was necessary to use a slightly different
description for `J s'

   'articles and headers that are eligible for fetching'

I'm guessing the description uses different language because the action
is different in some way... That is, not just `New' msgs but maybe
something more inclusive.  Maybe something like all un-downloaded msgs
that conform to a specific category, which might be something of a
disaster in groups with well over 100.000 msgs that have not been
downloaded yet.

I ask because the times I tried `J s' or `J u' (Even on just the
current group) it produces a massive lengthy read of something.. I'm
not sure if the visible count being shown is characters or lines but
it gets up to over 120,000 eventually but I've never let it
finish... I was afraid it was trying to start from the very oldest
posts on the gmane spool and start downloading from there.

Which might be a good way to loose access to the new gmane.

I'd like to know what is being read in that case above?

You say, you do nothing finer grained with the agent than an agentized
server.. so do you mean that you download all groups you have
subscribed to on that server?

Unless you only subscribe to a few groups, that seems like it would
become a very lot of messages at some point.

How do you handle the downloading? In terms of cmds used?

Do you do batching to keep up with new posts and actually go unplugged
at times?

I noticed in my one attempt (recently) at using the batch command
shown in the manual:

     You can run a complete batch command from the command line with the
     following incantation:

     #!/bin/sh
     emacs -batch -l ~/.emacs -l ~/.gnus.el -f gnus-agent-batch >/dev/null 2>&1

that gnus started downloading from groups under an agentized server
that were not themselves agentized as well as groups that were.

I ask all this because it is really time consuming and somewhat
difficult to set up real experimenting to discover the basic actions
one should expect when using the agent.