speed of Gnus

speed of Gnus

[Peter Petersen]

Hello,

This time I need to voice some frustration, unfortunately. :-(

After having customized Gnus more and more to my needs, I now find that 
working with it is unbearable in some cases:

Entering groups with thousands of articles!

Recently I tried to open a huge group with more than 40000 posts: Gnus 
started to work and work around for hours (sic!) without ever letting me 
see the summary buffer! After 3 or 4 hours I had lost my patience and 
did a control-g, which immediately interrupted the crazy process. So 
Gnus was NOT hung!


I compared that to other newsreaders:

Forte Agent needed 7 minutes! (about 6 minutes to download all the 
headers, which was about the same time for Gnus, and then only a few 
seconds to actually allow me to see the list of articles)

Ditto for slrn and tin.


In the meantime I upgraded Gnus from 5.10.6 to 5.10.7 and the situation 
has improved considerably, BUT:

Now having to wait about 40 minutes until I can enter a group with 40000 
articles - while a huge step forward - is still unacceptable! Moreover, 
CPU load is still close to 100% - blocking other programs and making it 
impossible to play an online game, for instance (while Gnus is 
generating the summary, even after downloading the articles is over).

When does the huge delay happen?
Well, it happens after having downloaded the headers. It's the time 
until "generating summary" can be read AND (even more so, perhaps) the 
time until "generating summary... done" can be seen. But what seems to 
take most of the time actually is: the time between "generating 
summary... done" and seeing the headers list!

I already applied some hints and tips I found while googling:
- simplifying the summary line
- putting (gnus-compile) at the bottom of .gnus
- entering the corresponding groups with escape-enter instead of enter
- sorting by article number (instead of date)
- ....


There seems to be a _slight_ improvement, but all in all not much has 
changed.

Now you may ask why I have to open such huge groups (or rather: 
downloading thousands of headers)...
Well I always do it this way; it doesn't mean that I read all those 
articles (oh NO!), but I can search through them for keywords like "via 
ac97", "crypto file system" or whatever.

I actually like to get _all_ the available headers in a group when I am 
actively searching for a solution. Other newsreaders allow me to do 
this.


I now realize again why I had such a hard time (in the past) coming away 
from Forte Agent:
I can open the hugest of the huge groups and rapidly see its headers, 
mark them as I please, sort them as I please, easily switch between 
offline and online mode (or use them simultaneously: downloading headers 
in _one_ group, downloading bodies in other groups - while at the same 
time selectively opening articles in some groups via visual inspection 
and just because I am curious...), I can easily watch threads and "lock" 
articles which thusly are protected against deleting or "compacting" the 
database (or expiring).


All of the other newsreaders available for unix systems don't seem to 
offer those vital features (slrn, tin etc.), the only exception being 
Gnus!

Or am I wrong here?
- Do slrn or tin offer a way to save interesting articles in such a way 
that they are visible _inside_ the corresponding group? (like Gnus does 
with its caching or Forte Agent with its "lock symbol"). I don't think 
so.
But this is a vital feature to me. I want to archive posts this way and 
be able to always see them in the corresponding group and to be able to 
post a follow-up from inside the group.


What's the consequence?
Well, I wouldn't like to return to Forte Agent, because:
- I need to run it via "emulation" (Wine) (though it works very well, 
indeed!)
- launching urls, pictures and links is more trouble than worth it 
(because you have to launch Linux programs from inside a windows 
application)
- Gnus has so many nice additional features (limiting the display of 
articles according to a multitude of criteria; ticking articles... etc.)
- Agent doesn't have unlimited threading depth
- Agent doesn't give me all the different colours for quoted text

On the other hand scoring (and all the complex possibilities with Gnus) 
is totally unimportant to me.



Ideas anyone?
- Can slrn (or tin) save articles in the group?
  If so: I will switch from Gnus to slrn, most likely ;-)
- How to speed up Gnus (if at all possible)?
- How do you get along with Gnus and with huge groups?


sorry for the rant and
thanks again
Peter

Re: speed of Gnus

[Alexander Syrov]

Hello, Peter.

On Mon, 11 Apr 2005 10:44:56 +0200
Peter Petersen wrote:

2 years or so ago I too switched from Forte Agent to Gnus. At first I had
the same problems as you describe. After a while I realized that using a
newsreader the "Gnus way" is also quite comfortable. You can't make Gnus
generate summary for thousands of articles quickly enough, but possibly you
may live without seeing all those articles at once.

 PP> Well I always do it this way; it doesn't mean that I read all those 
 PP> articles (oh NO!), but I can search through them for keywords like "via 
 PP> ac97", "crypto file system" or whatever.

Searching for articles is much more effective with some indexing
software. I use namazu and gnus-namazu.el. Another alternative is
nnir.el. It takes a few seconds to process a query and build a virtual
group containing the matching articles.

 PP> easily switch between 
 PP> offline and online mode

J j (gnus-agent-toggle-plugged) works in summary and group buffers.

 PP> (or use them simultaneously: downloading headers 
 PP> in _one_ group, downloading bodies in other groups - while at the same 
 PP> time selectively opening articles in some groups via visual inspection 
 PP> and just because I am curious...),

Lack of asyncronous operation is an evident drawback of
Gnus. Unfortunately, this functionality can't be easily implemented in Emacs.

 PP> I can easily watch threads and "lock" 
 PP> articles which thusly are protected against deleting or "compacting" the 
 PP> database (or expiring).

You can do it with Gnus. When would you like to mark an article? If you read
it the first time or if you come back to it while searching. In the former
case, Gnus shows you all the recently fetched articles in default setup
(and the old articles in the same threads if you set gnus-fetch-old-headers
to t). In the latter case you may search for articles with indexing
software. It is much more productive than manually watching over thousands
of message headers.

 PP> - Do slrn or tin offer a way to save interesting articles in such a way 
 PP> that they are visible _inside_ the corresponding group? (like Gnus does 
 PP> with its caching or Forte Agent with its "lock symbol"). I don't think 
 PP> so.

You probably need a local newsserver. Gnus Agent eliminates the need for
it, but it is neccessary with other newsreaders if you want to create an
offline storage for articles. You may try e.g. leafnode.

 PP> On the other hand scoring (and all the complex possibilities with Gnus) 
 PP> is totally unimportant to me.

I recommend you to try using scoring. It helps to deal with lots of mail &
news much more quickly.

 PP> - Can slrn (or tin) save articles in the group?
 PP>  If so: I will switch from Gnus to slrn, most likely ;-)

If you definitely can't change you habits, it seems to be the best
solution. You may use slrn + local leafnode server. But I recommend to try
following the "Gnus way" of reading news for a week or two before taking
this decision. Maybe you'll like it better.

 PP> - How to speed up Gnus (if at all possible)?

Not to the extent you want.

 PP> - How do you get along with Gnus and with huge groups?

My news archive (Gnus Agent cache) is over 1 Gb large and contains over
230,000 messages. But I rarely have more than 300-500 articles in summary
buffer at once. Scoring helps me to sort out new articles I don't want to
read and gnus-namazu allows to search for old articles quickly and
effectively. 

-- 
Regards,
Alexander Syrov.

Re: speed of Gnus

[Peter Petersen]

* Alexander Syrov <asyrov@mail.ru> schrieb:
Hello, Alexander.


> newsreader the "Gnus way" is also quite comfortable. You can't make Gnus
> generate summary for thousands of articles quickly enough, but possibly you
> may live without seeing all those articles at once.

In most cases, there is no problem, because the group is not that
big. Up to 10000 or with a little bit more patience 15000 the speed of
Gnus is still tolerable for me, and fortunately, this is sufficient for
most groups, yes.

With larger groups I now try to avoid the incredible delay by
successively downloading _portions_ of 10000 articles. Of course, this is
suboptimal when deciding upon threads to tick or cache. But at least, it
is something like a work-around.


>  PP> Well I always do it this way; it doesn't mean that I read all those 
>  PP> articles (oh NO!), but I can search through them for keywords like "via 
>  PP> ac97", "crypto file system" or whatever.
>
> Searching for articles is much more effective with some indexing
> software. I use namazu and gnus-namazu.el. Another alternative is
> nnir.el. It takes a few seconds to process a query and build a virtual
> group containing the matching articles.

I would have to try this, though I always liked to see _all_ headers (=
subjects, topics) and just browse them (with Forte Agent). They may not
reflect the most reliable way to find interesting articles, but I still
think, visual inspection is unbeaten when compared to any fixed rule or
rule set. And searching through article _bodies_ of an entire group or
lots of groups was not that slow either with Forte Agent, and it was
comfortable. Oh yeah...


>  PP> easily switch between 
>  PP> offline and online mode
>
> J j (gnus-agent-toggle-plugged) works in summary and group buffers.

I know, but that was not what I was going to say.


>  PP> (or use them simultaneously: downloading headers 
>  PP> in _one_ group, downloading bodies in other groups - while at the same 
>  PP> time selectively opening articles in some groups via visual inspection 
>  PP> and just because I am curious...),
>
> Lack of asyncronous operation is an evident drawback of
> Gnus. Unfortunately, this functionality can't be easily implemented in Emacs.

Now that is what I was referring to... Looks like one has to live with
this limitation.



>  PP> I can easily watch threads and "lock" 
>  PP> articles which thusly are protected against deleting or "compacting" the 
>  PP> database (or expiring).
>
> You can do it with Gnus. When would you like to mark an article? If you read
> it the first time or if you come back to it while searching. In the former
> case, Gnus shows you all the recently fetched articles in default setup
> (and the old articles in the same threads if you set gnus-fetch-old-headers
> to t). In the latter case you may search for articles with indexing
> software. It is much more productive than manually watching over thousands
> of message headers.

I still trust my eyes and my "intuition" or curiosity way more than any
fixed rule set. But you are right, other methods _can_ be more
productive, especially if they are considerably less time consuming and
life doesn't leave you much leasure time any longer. :-(


>  PP> - Do slrn or tin offer a way to save interesting articles in such a way 
>  PP> that they are visible _inside_ the corresponding group? (like Gnus does 
>  PP> with its caching or Forte Agent with its "lock symbol"). I don't think 
>  PP> so.
>
> You probably need a local newsserver. Gnus Agent eliminates the need for
> it, but it is neccessary with other newsreaders if you want to create an
> offline storage for articles. You may try e.g. leafnode.

Hehe. :)
No, I am through with this chapter, local newsservers. I had tried them
years ago (leafnode, inn - and also things like slrnpull or noffle). It
proved to be totally impractical for a single user like me (of course,
it would be different if I shared my news depot with several or lots of
other people: then a newsserver is vital).

When I didn't have an adsl connection yet and had to pay per minute for
my modem connection, there was nothing, really nothing (none of the
often suggested and furiously proclaimed Linux/Unix methods, like
running your own news server) that came close to the speed and comfort
of running Forte Agent - and yes, I did extended tests and gave local
news servers a chance for many months. Agent always turned out to be
superior - measured by my own needs and interest, naturally!

It is just so much easier to have the flexibility to download all the
headers in _one_ specific group, to only download 1000 headers in a
couple of other groups, to download all the bodies in yet another group,
to download all the marked bodies and watched threads in all the groups,
while at the same time spending one's waiting time online picking this
or that maybe interesting article and also posting some questions or
replies at the same time. Those were the Forte Agent times. ;-)

No newsserver solution ever offers that degree of flexibility. Not for a
single user who likes to be subscribed to a large number of (partially)
large groups and who also likes to easily subscribe or unsubscribe some
groups or just have a look in a few other groups, depending on my
current interests or open questions...


But hey, with my modem times being over, basically (except when ADSL is
down), these are all things of the past. Furthermore, there is this
_one_ news reader (Gnus) with its Agent mode, something no other reader
like slrn, tin etc. offers.


>  PP> - Can slrn (or tin) save articles in the group?
>  PP>  If so: I will switch from Gnus to slrn, most likely ;-)
>
> If you definitely can't change you habits, it seems to be the best
> solution. You may use slrn + local leafnode server. But I recommend to try
> following the "Gnus way" of reading news for a week or two before taking
> this decision. Maybe you'll like it better.

Except for large groups, I actually like the "Gnus way".


>  PP> - How to speed up Gnus (if at all possible)?
>
> Not to the extent you want.

O.k., that's what I expected.


>  PP> - How do you get along with Gnus and with huge groups?
>
> My news archive (Gnus Agent cache) is over 1 Gb large and contains over
> 230,000 messages. But I rarely have more than 300-500 articles in summary
> buffer at once. Scoring helps me to sort out new articles I don't want to
> read and gnus-namazu allows to search for old articles quickly and
> effectively. 

Many thanks.


Regards
Peter