Gnus performance

Gnus performance

[Ding Lei]

Hello folks ...
	I am currently using mutt and everything seems to be fine..
	But now I need newsgroup function and doesn't want to switch back and force 
between email and news client.
    So .. now considering migrating to GNUS, but I am quite worried about the
performance	of GNUS, imho... most of the emacs stuff seems to be quite slow,
processing big maildirs, for ex, how long will it take for GNUS to load a
maildir with 1500+ emails(mutt takes about 2-3 seconds on my machine)?
   Thank you.



-- 
Yours,

   <<<:::::   D i n g    L e i   ::::::>>
 ||                                      ||
 || Ext: 8106                            ||
 || Email: <dinglei [A] ipanel [O] cn>   ||
 || Dept. Of Technology/Engineering      ||
 || Embedded Information Services Inc.   ||
 ||                                      ||
<((((((    =====================     )))))>>>
	
In a museum in Havana, there are two skulls of Christopher Columbus,
"one when he was a boy and one when he was a man."
		-- Mark Twain

Re: Gnus performance

[Kai Grossjohann]

Ding Lei <dinglei@ipanel.cn> writes:

>     So .. now considering migrating to GNUS, but I am quite worried
> about the performance of GNUS, imho... most of the emacs stuff seems
> to be quite slow, processing big maildirs, for ex, how long will it
> take for GNUS to load a maildir with 1500+ emails(mutt takes about
> 2-3 seconds on my machine)?

Yes, performance is a problem with Gnus.  The usual solution, I
think, is to work around them somehow.  For instance, I configured
Gnus in such a way that it shows me the messages I want to see,
instead of all of them.  This enables me to enter a large folder
quickly, because Gnus doesn't have to show all messages.

Are you willing to think about this?

Kai

Re: Gnus performance

[Miles Bader]

Kai Grossjohann <kai <at> emptydomain.de> writes:
> > to be quite slow, processing big maildirs, for ex, how long will it
> > take for GNUS to load a maildir with 1500+ emails(mutt takes about
> > 2-3 seconds on my machine)?
> 
> Yes, performance is a problem with Gnus.  The usual solution, I
> think, is to work around them somehow.  For instance, I configured
> Gnus in such a way that it shows me the messages I want to see,
> instead of all of them.  This enables me to enter a large folder
> quickly, because Gnus doesn't have to show all messages.

Gnus is probably slower than mutt for many things, but it's not absurdly
slow; I don't really find it to be a problem in practice.  Some of it is
using slightly different habits than you would with mutt, as Kai said, but
it can handle big mailboxes reasonably too.

For instance, I just timed entering an nnml mailbox (separate file per
message, cached overview file) with about 6500 messages, but I told it
to only show the last 1500 to match your example.  It took about 2 seconds.

[This directory is on an NFS server, which slows things about a bit; the
 machine is a 2.5 GHz P4 Celeron with 512 MB of RAM and a fairly slow disk
 (BTW, "P4 Celeron" == "Stupidly Slow for Its Clock-Speed"]

-Miles

Re: Gnus performance

[Paul Jarc]

Miles Bader <miles@gnu.org> wrote:
> For instance, I just timed entering an nnml mailbox (separate file per
> message, cached overview file) with about 6500 messages, but I told it
> to only show the last 1500 to match your example.  It took about 2 seconds.

Unfortunately, nnmaildir is currently quite a bit slower than that.
But I'm working on it.


paul

Re: Gnus performance

[James Leifer]

prj@po.cwru.edu (Paul Jarc) writes:

> Miles Bader <miles@gnu.org> wrote:
>> For instance, I just timed entering an nnml mailbox (separate file per
>> message, cached overview file) with about 6500 messages, but I told it
>> to only show the last 1500 to match your example.  It took about 2 seconds.
>
> Unfortunately, nnmaildir is currently quite a bit slower than that.
> But I'm working on it.
>
> paul

Hi Paul,

I mentioned a while back that I though nnmaildir could be greatly
optimised if it made the reasonable (at least to me) assumption that
messages were immutable, i.e. once a uniq was chosen in a maildir, the
contents of the uniq never changed.

At the time I said that this assumption might save a lot of stat'ing
and that nnmaildir could get away with just listing the maildir
directory.

Did you ever consider this angle?  Does it seem plausible?

Kind regards,
James

Re: Gnus performance

[Paul Jarc]

James Leifer <James.Leifer@inria.fr> wrote:
> I mentioned a while back that I though nnmaildir could be greatly
> optimised if it made the reasonable (at least to me) assumption that
> messages were immutable, i.e. once a uniq was chosen in a maildir, the
> contents of the uniq never changed.

Yep - I need to think it through a bit more, along with some other
ideas.  (Sorry, I haven't been spending much time on it.)  This could
be optional, but I wouldn't want to make it the default, since it
would report false information in the case where an article were
modified.


paul

Re: Gnus performance

[James Leifer]

prj@po.cwru.edu (Paul Jarc) writes:

> James Leifer <James.Leifer@inria.fr> wrote:
>> I mentioned a while back that I though nnmaildir could be greatly
>> optimised if it made the reasonable (at least to me) assumption that
>> messages were immutable, i.e. once a uniq was chosen in a maildir, the
>> contents of the uniq never changed.
>
> Yep - I need to think it through a bit more, along with some other
> ideas.  (Sorry, I haven't been spending much time on it.)  This could
> be optional, but I wouldn't want to make it the default, since it
> would report false information in the case where an article were
> modified.

If I understand correctly, the unsafe behaviour is as follows:

* nnmaildir caches the NOV data for a message;
* later the message's headers are modified without gnus knowing...
* ... thus making the cached NOV data stale
* ... thus causing the Summary buffer to contain a stale entry.

Yes?

Of course I agree that this unsafe behaviour shouldn't be the default.
However, in return for better performance (much less stat'ing), I
would gladly accept this possibility as I never change a message's
headers or body.

Might be nice to have a batch function that one can launch from cron
at 07.00 to clean up the NOV cache (and generate NOV entries for the
newly arrived message) so that it's completely up to date when I start
the day...

By the way, how *does* one modify a message in the usual course of
gnus usage? Even for drafts I do D E to edit old ones under a *new*
message name.

Regards,
-James

Re: Gnus performance

[Paul Jarc]

James Leifer <James.Leifer@inria.fr> wrote:
> * nnmaildir caches the NOV data for a message;
> * later the message's headers are modified without gnus knowing...
> * ... thus making the cached NOV data stale
> * ... thus causing the Summary buffer to contain a stale entry.

With the fast-but-dangerous behavior, yes.  The message body could
also be modified, invalidating the old message size.

> Of course I agree that this unsafe behaviour shouldn't be the default.
> However, in return for better performance (much less stat'ing), I
> would gladly accept this possibility as I never change a message's
> headers or body.

Right, so you would set the option - once it exists. :)

> Might be nice to have a batch function that one can launch from cron
> at 07.00 to clean up the NOV cache (and generate NOV entries for the
> newly arrived message) so that it's completely up to date when I start
> the day...

That should be easy, I think - just start Gnus with the default
setting for this option.

> By the way, how *does* one modify a message in the usual course of
> gnus usage?

e


paul

Re: Gnus performance

[James Leifer]

prj@po.cwru.edu (Paul Jarc) writes:

prj@po.cwru.edu (Paul Jarc) writes:

[snipping liberally]

> James Leifer <James.Leifer@inria.fr> wrote:
>> * nnmaildir caches the NOV data for a message;
>> * later the message's headers are modified without gnus knowing...
>> * ... thus making the cached NOV data stale
>> * ... thus causing the Summary buffer to contain a stale entry.
>
> With the fast-but-dangerous behavior, yes.  The message body could
> also be modified, invalidating the old message size.

Of course, I *would* like insertions and deletions of messages to be
handled in the fast-but-dangerous case.  I reckon this can be done
without resorting to any stat'ing: one can just open the maildir to
get the names of the uniqs.

>> By the way, how *does* one modify a message in the usual course of
>> gnus usage?
>
> e
>

Aha!

It might be nice to cause the NOV entry to be regenerated in this
case.  Then the unsafe behaviour would only be triggered when a
message were modified by a program other than gnus.  If I had any
tools that needed to do that (which I don't), I suppose they could
just delete the NOV entry in this case, thus also guaranteeing
safety.

The only dangerous case left would be an external program that doesn't
delete the NOV entry when it modifies a message...

Yes?

-James

Re: Gnus performance

[Paul Jarc]

James Leifer <James.Leifer@inria.fr> wrote:
> Of course, I *would* like insertions and deletions of messages to be
> handled in the fast-but-dangerous case.

Right.  Any time a change is done via nnmaildir, the NOV data can be
updated.

> Then the unsafe behaviour would only be triggered when a
> message were modified by a program other than gnus.  If I had any
> tools that needed to do that (which I don't), I suppose they could
> just delete the NOV entry in this case, thus also guaranteeing
> safety.

However, that would also assign a new article number for the message,
which would interfere with caching, the agent, and 'seen marks,
possibly more.


paul

Re: Gnus performance

[James Leifer]

prj@po.cwru.edu (Paul Jarc) writes:

> James Leifer <James.Leifer@inria.fr> wrote:

>> Then the unsafe behaviour would only be triggered when a
>> message were modified by a program other than gnus.  If I had any
>> tools that needed to do that (which I don't), I suppose they could
>> just delete the NOV entry in this case, thus also guaranteeing
>> safety.
>
> However, that would also assign a new article number for the message,
> which would interfere with caching, the agent, and 'seen marks,
> possibly more.

Right... Hadn't understood the side effects of deleting a NOV entry (I
thought it was semantically a nop).

This does point to the fact that gnus probably should leave more
control of such things to the backend: I would like the backend to
manage *all* marks.  I think this was mentioned on the list.  Probably
major architectural changes like that will take a long time to happen,
though.

A possible solution: define a kind of NOV entry that says that
everything is invalid *except* the article number.  Then it would
always be safe to put a NOV entry back in this state.  But, I don't
know enough here to tell whether that could really work.

-James