Gnus is very slow in displaying headers.

Gnus is very slow in displaying headers.

[Daniel M.]

Hello everybody,

I noticed that even when I enter a group with a relatively small 
amount of headers - in my case, 12000 for comp.lang.lisp, it takes 
Gnus a very long time to thread/load them (headers are already in 
the spool - they were downloaded previously by the Gnus) - 3 minutes 
and 100% CPU for the duration of that period.

Forte Agent, for example, has no problem with groups which
contain hundreds of thousands of headers - it takes merely a couple 
of seconds to load them, so it is not a lack of memory/CPU cycles.

Am I doing something wrong, or is it a fundamental issue (elisp is too
slow? something in the core of X/Emacs?) which renders Gnus unusable
for anything more then a very small groups?

After searching the group for a similar questions, I saw an advice
to set nntp-marks-is-evil variable to 't', which I have done, but 
it made no noticable difference.

I use Gnus 5.10.6 on Debian (precompiled package from unstable).

BTW, listing of all active groups (zombie, killed groups - I am not 
sure how I am supposed to call them. In effect, it is the list of all 
groups carried by the server) also takes a lot of time - almost a full 
minute, although there are only ~32000 groups and the file containing
them resides on the HD. I even tried to sort that file, but it didn't 
help :(.

Daniel.

Re: Gnus is very slow in displaying headers.

[Jorge Godoy]

Daniel M. <xt@nm.ru> writes:

> I noticed that even when I enter a group with a relatively small 
> amount of headers - in my case, 12000 for comp.lang.lisp, it takes 
> Gnus a very long time to thread/load them (headers are already in 
> the spool - they were downloaded previously by the Gnus) - 3 minutes 
> and 100% CPU for the duration of that period.

It doesn't happen here, but I have an nntp server (leafnode) on my LAN.
Access here is almost instantaneous, except when I happen to get an
article in HTML or some not-found-on-keyservers PGP key use to sign
messages.

> Am I doing something wrong, or is it a fundamental issue (elisp is too
> slow? something in the core of X/Emacs?) which renders Gnus unusable
> for anything more then a very small groups?

I dunno what you're doing or how, but here the reality is different.  It
looks more like your Forte Agent. 

> BTW, listing of all active groups (zombie, killed groups - I am not 
> sure how I am supposed to call them. In effect, it is the list of all 
> groups carried by the server) also takes a lot of time - almost a full 
> minute, although there are only ~32000 groups and the file containing
> them resides on the HD. I even tried to sort that file, but it didn't 
> help :(.

Weird.

It seems you're not using the local file, then...  It is faster here and
the amount of groups is also big (Usenet + a brazilian network with a
few thousand groups).

Have you tried removing your configuration and redoing it? 



Be seeing you,
-- 
Godoy.     <godoy@ieee.org>

Re: Gnus is very slow in displaying headers.

[Jorge Godoy]

Daniel M. <xt@nm.ru> writes:

> Yes, I tried. Also, I tried to use Gnus both under Linux and 
> Windows 2000, with the same result. Starting Gnus with
> 'M-x gnus-unplugged' doesn't improve the situation, which means
> that Gnus is not so slow because of a network.
>
> Any ideas what can be wrong so that a performance degrades
> so much - or is it possible somehow to check what is wrong 
> (something like profiling)?

You'll need help from one of the gurus. ;-)

I'm just a simple user. :-)


Be seeing you,
-- 
Godoy.     <godoy@ieee.org>

Re: Gnus is very slow in displaying headers.

[Kevin Greiner]

Daniel M. <xt@nm.ru> writes:

> Hello everybody,
>
> I noticed that even when I enter a group with a relatively small 
> amount of headers - in my case, 12000 for comp.lang.lisp, it takes 
> Gnus a very long time to thread/load them (headers are already in 
> the spool - they were downloaded previously by the Gnus) - 3 minutes 
> and 100% CPU for the duration of that period.
>
> Forte Agent, for example, has no problem with groups which
> contain hundreds of thousands of headers - it takes merely a couple 
> of seconds to load them, so it is not a lack of memory/CPU cycles.
>
> Am I doing something wrong, or is it a fundamental issue (elisp is too
> slow? something in the core of X/Emacs?) which renders Gnus unusable
> for anything more then a very small groups?

The problem is mostly likely due to your setting for
gnus-summary-line-format.  This is a string that describes the fields
displayed for each line of the summary buffer.  Internally, gnus
compiles this string into an elisp expression that inserts an article
into the buffer.  The approach is infinitely customizable but not very
optimizable.  You might look in the manual under 'Summary Buffer
Lines' to see if can identify a simplier string; one that meets your
requirements yet offers better performance.


> After searching the group for a similar questions, I saw an advice
> to set nntp-marks-is-evil variable to 't', which I have done, but 
> it made no noticable difference.

Right.  Nothing to do with your problem.

> I use Gnus 5.10.6 on Debian (precompiled package from unstable).
>
> BTW, listing of all active groups (zombie, killed groups - I am not 
> sure how I am supposed to call them. In effect, it is the list of all 
> groups carried by the server) also takes a lot of time - almost a full 
> minute, although there are only ~32000 groups and the file containing
> them resides on the HD. I even tried to sort that file, but it didn't 
> help :(.

I see the same thing, I just don't display the active list often enough to
try to optimize it.

Kevin

Re: Gnus is very slow in displaying headers.

[Daniel M.]

>
>It seems you're not using the local file, then...  It is faster here and
>the amount of groups is also big (Usenet + a brazilian network with a
>few thousand groups).
>
>Have you tried removing your configuration and redoing it? 

Yes, I tried. Also, I tried to use Gnus both under Linux and 
Windows 2000, with the same result. Starting Gnus with
'M-x gnus-unplugged' doesn't improve the situation, which means
that Gnus is not so slow because of a network.

Any ideas what can be wrong so that a performance degrades
so much - or is it possible somehow to check what is wrong 
(something like profiling)?

Daniel.

Re: Gnus is very slow in displaying headers.

[Kai Grossjohann]

Daniel M. <xt@nm.ru> writes:

> Using any other formatting options pushes loading time still 
> higher. No matter what I tried, it doesn't take less then 2 minutes
> to load the group, even if the most basic usable summary formatting
> line was used.

Does it help to put (gnus-compile) after the (setq
gnus-summary-line-format ...) statement?

Kai

Re: Gnus is very slow in displaying headers.

[Kai Grossjohann]

Daniel M. <xt@nm.ru> writes:

>>Does it help to put (gnus-compile) after the (setq
>>gnus-summary-line-format ...) statement?
>
> Unfortunatley not at all - it still takes the same 1 min. 37 sec.
> to load the group. For 26 sec. the status line reads: "Fetching
> headers for comp.lang.lisp" (the group I ma loading), then
> for the rest of the time: "Generating summary".

I see.  I'm afraid that probably most people do not open groups with
that many articles.  The time used for summary generation depends on
the length of the summary buffer.

Some people like to keep lots of articles marked as unread, especially
for mail, so that they do not lose track of the context.  But I think
that it works well to (setq gnus-fetch-old-headers t): this leads to
Gnus showing the complete thread, if there is an unread article in
that thread.  This way, I get the complete context of new discussions,
without having to keep all old articles of this discussion unread.

So my question to you is: why are there 12,000 unread articles in that
group?  I'm suspecting that you don't intend to read them all...

Kai

Re: Gnus is very slow in displaying headers.

[Derrell.Lipman]

Kai Grossjohann <kai@emptydomain.de> writes:

> Some people like to keep lots of articles marked as unread, especially
> for mail, so that they do not lose track of the context.  But I think
> that it works well to (setq gnus-fetch-old-headers t): this leads to
> Gnus showing the complete thread, if there is an unread article in
> that thread.  This way, I get the complete context of new discussions,
> without having to keep all old articles of this discussion unread.

I used to do that.  Based on a previous thread in the ding list about slow
presentation of a group, I now have the following comment and setting in my
.gnus file:

    (setq
     ;; Show headers of previous messages in a thread
     ;; Don't do this; it greatly slows down presentation of the group
     gnus-fetch-old-headers			nil
    )

My recollection was that the difference in speed, disabling this, was
remarkable.

Derrell

Re: Gnus is very slow in displaying headers.

[Daniel M.]

>The problem is mostly likely due to your setting for
>gnus-summary-line-format.  This is a string that describes the fields
>displayed for each line of the summary buffer.  Internally, gnus
>compiles this string into an elisp expression that inserts an article
>into the buffer.  The approach is infinitely customizable but not very
>optimizable.  You might look in the manual under 'Summary Buffer
>Lines' to see if can identify a simplier string; one that meets your
>requirements yet offers better performance.

It indeed improves somewhat the performance but it is still far
from usable. Here are the settings I have used and the time to load
headers into a summary buffer (the group contained 12000
headers):

(setq gnus-summary-line-format "") - 1 min. and 8 sec.
(setq gnus-summary-line-format "\n") - 1 min. and 8 sec.
(setq gnus-summary-line-format "%s\n") - 1 min. 37 sec.

Setting gnus-show-threads to nil doesn't make any noticable
difference.

Using any other formatting options pushes loading time still 
higher. No matter what I tried, it doesn't take less then 2 minutes
to load the group, even if the most basic usable summary formatting
line was used.

Is there are other ways to speed up the process?

BTW, my computer's specs are:

CPU: Athlon 1333, RAM: 768MB.

Daniel.

Re: Gnus is very slow in displaying headers.

[Daniel M.]

>Does it help to put (gnus-compile) after the (setq
>gnus-summary-line-format ...) statement?

Unfortunatley not at all - it still takes the same 1 min. 37 sec.
to load the group. For 26 sec. the status line reads: "Fetching
headers for comp.lang.lisp" (the group I ma loading), then
for the rest of the time: "Generating summary".

Daniel.

Re: Gnus is very slow in displaying headers.

[Jesper Harder]

Daniel M. <xt@nm.ru> writes:

> I noticed that even when I enter a group with a relatively small 
> amount of headers - in my case, 12000 for comp.lang.lisp, it takes 
> Gnus a very long time to thread/load them (headers are already in 
> the spool - they were downloaded previously by the Gnus) - 3 minutes 
> and 100% CPU for the duration of that period.

You *are* running a byte compiled Gnus, right?  (i.e. you used the
Makefile to build it, ./configure && make).

-- 
Jesper Harder                                <http://purl.org/harder/>

Re: Gnus is very slow in displaying headers.

[Andrew A. Raines]

Daniel M. <xt@nm.ru> writes:

[...]

> Usually I don't follow daily discussions in newsgroups
> closely. Instead, I fetch all headers from a group and browse
> through them, reading old posts that catch my interest.

Wow.  Isn't that similar to entering random URLs in your web browser
to find interesting sites?

USENET doesn't work well like that.  I would suggest running through
groups you don't follow closely every week or two with your fingers
on C-k, stopping on threads that interest you.  If looking through
thousands of messages is your thing, take heart; debian-user can
easily provide 1000-1500 per week.

Leave finding the old, interesting content to Google.

-- 
    aaraines@pobox.com (Andrew A. Raines)

Re: Gnus is very slow in displaying headers.

[Daniel M.]

>I used to do that.  Based on a previous thread in the ding list about slow
>presentation of a group, I now have the following comment and setting in my
>.gnus file:
>
>    (setq
>     ;; Show headers of previous messages in a thread
>     ;; Don't do this; it greatly slows down presentation of the group
>     gnus-fetch-old-headers			nil
>    )
>
>My recollection was that the difference in speed, disabling this, was
>remarkable.

I tried setting that variable as you had advised, but it didn't have 
an impact on the speed. If I understand the documentation correctly,
this variable has an impact only if there are 'read' articles - usually 
Gnus will construct threads using old, read articles if there are some
unread articles in the thread. Setting that variable to 'nil' will prevent
Gnus from loading the read articles. But in my case, almost all
of the articles are unread - because of that, there is no noticeable 
difference.

Daniel.

Re: Gnus is very slow in displaying headers.

[Daniel M.]

>So my question to you is: why are there 12,000 unread articles in that
>group?  I'm suspecting that you don't intend to read them all...
>

Usually I don't follow daily discussions in newsgroups closely. Instead,
I fetch all headers from a group and browse through them,
reading old posts that catch my interest. But sooner or later I would
catch on those ones skipped previously, or get back to those already 
read  - this is like a 'cookbook' or a reference: you don't read everything 
from the beginning to the end, but use it to glean the immediately 
useful information or to refresh your memory.

For example, there are more then 160000 articles in linux.debian.user
(on gmane's newsserver). Obviously, with the current summary
generating speed it is impractical to try and read it with Gnus :(.

Daniel.

Re: Gnus is very slow in displaying headers.

[Daniel M.]

On Tue, 24 Aug 2004 22:07:58 +0200, Jesper Harder <harder@ifa.au.dk> wrote:

>You *are* running a byte compiled Gnus, right?  (i.e. you used the
>Makefile to build it, ./configure && make).

I suppose so. On Linux, I use Debian's package which includes 
Gnus (it comes with Xemacs, from an unstable distribution). On 
Windows it also included in Xemacs,  and comes bundled with a 
binary package from Xemacs' site. Both versions contain files with 
.elc extensions.

Daniel.

Re: Gnus is very slow in displaying headers.

[Kai Grossjohann]

Derrell.Lipman@UnwiredUniverse.com writes:

> Kai Grossjohann <kai@emptydomain.de> writes:
>
>> Some people like to keep lots of articles marked as unread, especially
>> for mail, so that they do not lose track of the context.  But I think
>> that it works well to (setq gnus-fetch-old-headers t): this leads to
>> Gnus showing the complete thread, if there is an unread article in
>> that thread.  This way, I get the complete context of new discussions,
>> without having to keep all old articles of this discussion unread.
>
> I used to do that.  Based on a previous thread in the ding list about slow
> presentation of a group, I now have the following comment and setting in my
> .gnus file:
>
>     (setq
>      ;; Show headers of previous messages in a thread
>      ;; Don't do this; it greatly slows down presentation of the group
>      gnus-fetch-old-headers			nil
>     )
>
> My recollection was that the difference in speed, disabling this, was
> remarkable.

Well, what do I say about this?

Many people have many articles marked as unread.  It is faster to mark
them as read and (setq gnus-fetch-old-headers t) -- this tends to show
the "interesting" articles that people would otherwise mark as unread.

Obviously, not showing the "interesting" articles is faster still.
But then, people might tend to mark interesting articles as unread,
leading to 12,000 unread articles in a group, which leads to the
slowdown.

Kai

Re: Gnus is very slow in displaying headers.

[Kai Grossjohann]

Daniel M. <xt@nm.ru> writes:

>> So my question to you is: why are there 12,000 unread articles in that
>> group?  I'm suspecting that you don't intend to read them all...
>
> Usually I don't follow daily discussions in newsgroups closely. Instead,
> I fetch all headers from a group and browse through them,
> reading old posts that catch my interest. But sooner or later I would
> catch on those ones skipped previously, or get back to those already 
> read  - this is like a 'cookbook' or a reference: you don't read everything 
> from the beginning to the end, but use it to glean the immediately 
> useful information or to refresh your memory.

I see.  This means that it is a one-time only thing, until you were
able to catch up on those articles.

Note that you can use scoring, or adaptive scoring, to keep down the
number of articles you read, once you have started reading the group.

But for the initial part, I don't know of a way to make it faster with
Gnus.  As far as I'm concerned, myself, I just hit c before I start
reading a group, because reading (even skimming) 12,000 articles is
out of the question for me.

Kai

Re: Gnus is very slow in displaying headers.

[ignotus]

>>>>> Daniel M. wrote:

> I noticed that even when I enter a group with a relatively small
> amount of headers - in my case, 12000 for comp.lang.lisp, it takes
> Gnus a very long time to thread/load them

[...]

Yes, the sad fact is that Gnus is that slow and it doesn't matter what
you do you can't bring it up to the ,,normal speed'' (which means under
1 second), not even close to it.

Heh, I still remember my horror when I first saw the speed of Gnus after
using Mutt / slrn.  With Mutt/slrn threading / displaying a message etc
takes no time, you have all the time for yourself to actually *read* the
messages and it feels good.

I tried to do that with Gnus, trimmed down all the features I thought I
didn't absolutely need etc but of course it was still extremely slow.
But I sticked with it because it was so much more configurable, I saw at
the time that with an Emacs based software the limit is my imagination,
while with Mutt / slrn / etc the limit is the configuration options the
developers decide to impelement.

So as my elisp knowledge grew I customized Gnus bit by bit to my tastes,
and of course Gnus became slower and slower but I was able to subscribe
to more and more newsgroups / mailing lists.

You may know already what happened to me.  While mutt / slrn maximizes
the time you have for reading by not ,,interrupting you'' with long
computations, Gnus minimizes the time you will need to read the messages
by having so much ,,time-saving'' features (adapting scoring is a
godsend).  And after you experienced both, I can assure you that the
latter feels better. ;)

So the moral of the story is that while Gnus is slow, following the same
volume of messages with mutt / slrn isn't possible for me because it
would take so much of my time.

And for those who don't understand why would someone in his right mind
read 12k messages: please never forget about the faceless, voiceless and
lifeless masses of ,,Usenet junkies''. :-)

-- 
...sutongi tti olleh
                        If you are smart enough to know that you're not smart
                           enough to be an Engineer, then you're in Business.

Re: Gnus is very slow in displaying headers.

[Daniel M.]

>Yes, the sad fact is that Gnus is that slow and it doesn't matter what
>you do you can't bring it up to the ,,normal speed'' (which means under
>1 second), not even close to it.

Maybe one day X/Emacs would be fitted with Common Lisp backend
having a native-code compiler - then probably the speed would improve
sufficiently for Gnus to be usable with bigger groups: for the time being,
though, it seems that Gnus is not for me :(.

I don't understand, though, why summary buffers have to be generated
afresh every time you enter a group - it seems to me that generating
it once, saving to HD and then adding/removing chunks from it when 
needed (for example, when new headers arrive, some articles are read 
etc.) should speed the things up immensely. Also, it looks to me like
a good idea to have a command line option which will cause regeneration
of summary buffers (if summary buffers are saved to HD). Then, if
user decides to change summary buffer's format line, he can say, for
example: 'xemacs --gnus-regenerate-summary-buffers', then go and
have a (long) lunch, after which he can come back and start reading
his dose of news without waiting for regeneration of summary buffers.


>I tried to do that with Gnus, trimmed down all the features I thought I
>didn't absolutely need etc but of course it was still extremely slow.
>But I sticked with it because it was so much more configurable, I saw at
>the time that with an Emacs based software the limit is my imagination,
>while with Mutt / slrn / etc the limit is the configuration options the
>developers decide to impelement.

Well, you can add whatever options you want by simply modifying 
slrn's source code - some people have done exactly that (see
http://www.xs4all.nl/~thunder7/, for example). Maybe it is easier 
to do in elisp, but if you are well familiar with GNU's development
chain (which I, at least, am not), it shouldn't be a big problem.

>You may know already what happened to me.  While mutt / slrn maximizes
>the time you have for reading by not ,,interrupting you'' with long
>computations, Gnus minimizes the time you will need to read the messages
>by having so much ,,time-saving'' features (adapting scoring is a
>godsend).  And after you experienced both, I can assure you that the
>latter feels better. ;)

Don't think it will work for me - by the time the group is loaded, the
time allocated for my news reading would probably be up :^).


Daniel.