Improving Gnus speed

Improving Gnus speed

[Francis Moreau]

Hello,

I'd like to give some feedbacks about Gnus' speed.

First I just wrote this in order to improve Gnus _only_.

I use as a newsreader Gnus and Thunderbird, and I must admit that Gnus
is much slower specially to fetch header and to display the summary
buffer.

One example: I'm using article caching to save precious articles. Here's
my related config:

   (setq gnus-use-cache 'passive)
   (setq gnus-cacheable-groups "^nntp")
   (setq gnus-keep-backlog 50)

Even if articles are cached, Gnus takes some times to display the
summary buffer:

Entering in the group containing cached articles: 4 secs.

This group contains 230 unread articles.

Now display all articles in this group by doing '/ o':

Gnus asks me:

     How many articles from nnml+cache:nntp+news.free.fr:comp.lang.c (default 42555):

Type <RET>.

     Fetching headers for nnml+cache:nntp+news.free.fr:comp.lang.c...
     < 7 secs>
     done
     < 22 secs>
     Generating summary...
     < 3 secs>
     done

As you can see all these figures are pretty high, the biggest being 22
secs. I'm wondering what's Gnus is executing since it seems between 2
steps.

Corresponding group parameters:

     (display . all)
     (gnus-use-scoring t)
     (gnus-thread-sort-functions '((not gnus-thread-sort-by-number)
                                   gnus-thread-sort-by-total-score)))))

I'm interested in improving this, so any suggestions are welcome.

Otherwise I can have a look to the revelant part of the Gnus' code,
although I almost don't speak elisp but that would be a reason to start
learning this language, if some one can suggest me some functions or
some starting points to look at.

BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP
method only (not git protocol). Is that right ?

IIRC, git protocol is more effective, and I actually don't compile Git
with HTTP support since it adds some dependency like curl for no good
reason.

Out off topic: Why does Gnus development use a mailing list ? That's
pretty paradoxical, isn't it ?

Thanks
-- 
Francis

Re: Improving Gnus speed

[Didier Verna]

Francis Moreau <francis.moro@gmail.com> wrote:

> Out off topic: Why does Gnus development use a mailing list ? That's
> pretty paradoxical, isn't it ?

  Because if you think that Gnus is a newsreader, you're wrong...

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

Re: Improving Gnus speed

[Francis Moreau]

Didier Verna <didier@xemacs.org> writes:

> Francis Moreau <francis.moro@gmail.com> wrote:
>
>> Out off topic: Why does Gnus development use a mailing list ? That's
>> pretty paradoxical, isn't it ?
>
>   Because if you think that Gnus is a newsreader, you're wrong...

:)

Is that a joke ?

If so I would suggest you to use ':)' next time...

Otherwise just read (gnus)Top.

Bye.
-- 
Francis

Re: Improving Gnus speed

[Knut Anders Hatlen]

Francis Moreau <francis.moro@gmail.com> writes:

> Didier Verna <didier@xemacs.org> writes:
>
>> Francis Moreau <francis.moro@gmail.com> wrote:
>>
>>> Out off topic: Why does Gnus development use a mailing list ? That's
>>> pretty paradoxical, isn't it ?
>>
>>   Because if you think that Gnus is a newsreader, you're wrong...
>
> :)
>
> Is that a joke ?
>
> If so I would suggest you to use ':)' next time...
>
> Otherwise just read (gnus)Top.

Which says pretty clearly that it's a news *and* mail reader. :)

-- 
Knut Anders

Re: Improving Gnus speed

[Steinar Bang]

And in any case, I read this thread on a newsgroup on news.gmane.org...

Re: Improving Gnus speed

[Francis Moreau]

Knut Anders Hatlen <kahatlen@gmail.com> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Didier Verna <didier@xemacs.org> writes:
>>
>>> Francis Moreau <francis.moro@gmail.com> wrote:
>>>
>>>> Out off topic: Why does Gnus development use a mailing list ? That's
>>>> pretty paradoxical, isn't it ?
>>>
>>>   Because if you think that Gnus is a newsreader, you're wrong...
>>
>> :)
>>
>> Is that a joke ?
>>
>> If so I would suggest you to use ':)' next time...
>>
>> Otherwise just read (gnus)Top.
>
> Which says pretty clearly that it's a news *and* mail reader. :)

No it says that it's primarly a newsreader: "(gnus)Mail in a Newsreader"
is an example.

-- 
Francis

Re: Improving Gnus speed

[Richard Riley]

Francis Moreau <francis.moro@gmail.com> writes:

> Knut Anders Hatlen <kahatlen@gmail.com> writes:
>
>> Francis Moreau <francis.moro@gmail.com> writes:
>>
>>> Didier Verna <didier@xemacs.org> writes:
>>>
>>>> Francis Moreau <francis.moro@gmail.com> wrote:
>>>>
>>>>> Out off topic: Why does Gnus development use a mailing list ? That's
>>>>> pretty paradoxical, isn't it ?
>>>>
>>>>   Because if you think that Gnus is a newsreader, you're wrong...
>>>
>>> :)
>>>
>>> Is that a joke ?
>>>
>>> If so I would suggest you to use ':)' next time...
>>>
>>> Otherwise just read (gnus)Top.
>>
>> Which says pretty clearly that it's a news *and* mail reader. :)
>
> No it says that it's primarly a newsreader: "(gnus)Mail in a Newsreader"
> is an example.

On that note some of the messages could be made clearer. e.g "posting
article..." when you send to a usenet group and "sending email..." when
using a To address.

Re: Improving Gnus speed

[Didier Verna]

Richard Riley <rileyrg@googlemail.com> wrote:

> On that note some of the messages could be made clearer. e.g "posting
> article..." when you send to a usenet group and "sending email..." 
> when using a To address.

  That's because there is a distinction in your mind, whereas Gnus tries
hard to blur the difference between sending mails and posting news. ;-)

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

Re: Improving Gnus speed

[Richard Riley]

Didier Verna <didier@xemacs.org> writes:

> Richard Riley <rileyrg@googlemail.com> wrote:
>
>> On that note some of the messages could be made clearer. e.g "posting
>> article..." when you send to a usenet group and "sending email..." 
>> when using a To address.
>
>   That's because there is a distinction in your mind, whereas Gnus tries
> hard to blur the difference between sending mails and posting news. ;-)

Because there is a huge distinction when sending/posting.

I like to know in the combined UI that I am dealing with a group as
opposed to an email.

Re: Improving Gnus speed

[Steinar Bang]

>>>>> Richard Riley <rileyrg@googlemail.com>:

> Didier Verna <didier@xemacs.org> writes:

>> That's because there is a distinction in your mind, whereas Gnus
>> tries hard to blur the difference between sending mails and posting
>> news. ;-)

> Because there is a huge distinction when sending/posting.

There is...?  In what way?

Re: Improving Gnus speed

[Richard Riley]

Steinar Bang <sb@dod.no> writes:

>>>>>> Richard Riley <rileyrg@googlemail.com>:
>
>> Didier Verna <didier@xemacs.org> writes:
>
>>> That's because there is a distinction in your mind, whereas Gnus
>>> tries hard to blur the difference between sending mails and posting
>>> news. ;-)
>
>> Because there is a huge distinction when sending/posting.
>
> There is...?  In what way?
>

The intended audiences.

Frequently I spot that I am accidentally emailing rather than posting a
reply.

Obviously I can only speak for myself but one of the first things people
do/used to do is turn on the "really post" prompt on when it was an nntp
post since its going public.

Re: Improving Gnus speed

[Steinar Bang]

>>>>> Richard Riley <rileyrg@googlemail.com>:

> The intended audiences.

> Frequently I spot that I am accidentally emailing rather than posting a
> reply.

Hm... that has never happened to me.  The other way around: sending an
intended private reply to a mailing list _has_ happened.

What I do then is to set
 (broken-reply-to . t)
in the group parameters of the nnimap group represting the mailing
list...

> Obviously I can only speak for myself but one of the first things
> people do/used to do is turn on the "really post" prompt on when it
> was an nntp post since its going public.

I've sometimes thought that it would have been nice to have a posting
delay, where I could go in, say within 5 minutes and kill a posting.

But after a while, I figured that the best thing to do was to just live
with my malposts.  I've got expiry turned on when posting to USENEt, so
it means that google groups will stop showing them after a couple of
weeks time.

Re: Improving Gnus speed

[Adam Sjøgren]

On Tue, 09 Nov 2010 18:48:46 +0100, Steinar wrote:

> I've got expiry turned on when posting to USENEt, so it means that
> google groups will stop showing them after a couple of weeks time.

Hopefully you never post anything that could be useful to anyone beyond
a couple of weeks time.


  Best regards,

    Adam

-- 
 "Angels can fly because they take themselves lightly."       Adam Sjøgren
                                                         asjo@koldfront.dk

Re: Improving Gnus speed

[Steinar Bang]

>>>>> asjo@koldfront.dk (Adam Sjøgren):

> Hopefully you never post anything that could be useful to anyone
> beyond a couple of weeks time.

What I still post on USENET can safely be counted into that category.

(I don't count gmane as USENET in this context, and I don't expire on
gmane)

Re: Improving Gnus speed

[Richard Riley]

Steinar Bang <sb@dod.no> writes:

>>>>>> Richard Riley <rileyrg@googlemail.com>:
>
>> The intended audiences.
>
>> Frequently I spot that I am accidentally emailing rather than posting a
>> reply.
>
> Hm... that has never happened to me.  The other way around: sending an
> intended private reply to a mailing list _has_ happened.
>
> What I do then is to set
>  (broken-reply-to . t)
> in the group parameters of the nnimap group represting the mailing
> list...

I need to look that up.

>
>> Obviously I can only speak for myself but one of the first things
>> people do/used to do is turn on the "really post" prompt on when it
>> was an nntp post since its going public.
>
> I've sometimes thought that it would have been nice to have a posting
> delay, where I could go in, say within 5 minutes and kill a posting.
>
> But after a while, I figured that the best thing to do was to just live
> with my malposts.  I've got expiry turned on when posting to USENEt, so
> it means that google groups will stop showing them after a couple of
> weeks time.
>

Why would you do that? If you post to usenet then I think it should stay
for google completeness in threads.

But for me the bottom line is that both mentally and physically usenet
and mail are two different entities and while I like the common features
Gnus provides I still want that difference to be apparent to me. There
are a few rough edges in Gnus where "sending" and "posting" are
used. They mean different things and I like that they do.

Re: Improving Gnus speed

[Didier Verna]

Francis Moreau <francis.moro@gmail.com> wrote:

> Is that a joke ?

Nope.


> If so I would suggest you to use ':)' next time...
>
> Otherwise just read (gnus)Top.

Hmmm, here we are, lecturing 20 years old-gnus-timers about reading the
Gnus Manual. Nicely done :-)

Hint: read "Getting News", then "Getting Mail", then compare their
respective sizes, then come back and try again :-)

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

Re: Improving Gnus speed

[Richard Riley]

Didier Verna <didier@xemacs.org> writes:

> Francis Moreau <francis.moro@gmail.com> wrote:
>
>> Is that a joke ?
>
> Nope.
>
>> If so I would suggest you to use ':)' next time...
>>
>> Otherwise just read (gnus)Top.
>
> Hmmm, here we are, lecturing 20 years old-gnus-timers about reading the
> Gnus Manual. Nicely done :-)

Did you? ;)

>
> Hint: read "Getting News", then "Getting Mail", then compare their
> respective sizes, then come back and try again :-)

And that means what? Its just indicative of the complexity of the
product and the rather famous Gnus Manual in particular. Its not for
nothing #emacs and other resources are constantly full of new adopters
pleading for help in getting their email working with Gnus. Nothing to
do whatsoever with the relative functionality. It is not for the feint
of heart.

Gnus can most certainly be thought of as primarily a newsreader in my
mind and most certainly your claim that its NOT a news reader must be a
joke. It is isn't it??

regards

r.

Re: Improving Gnus speed

[Francis Moreau]

Richard Riley <rileyrg@googlemail.com> writes:

> Didier Verna <didier@xemacs.org> writes:
>
>> Francis Moreau <francis.moro@gmail.com> wrote:
>>
>>> Is that a joke ?
>>
>> Nope.
>>
>>> If so I would suggest you to use ':)' next time...
>>>
>>> Otherwise just read (gnus)Top.
>>
>> Hmmm, here we are, lecturing 20 years old-gnus-timers about reading the
>> Gnus Manual. Nicely done :-)
>
> Did you? ;)
>

Well, probably he still hasn't understood it after 20 years ;) (Please
note the smiley).

>> Hint: read "Getting News", then "Getting Mail", then compare their
>> respective sizes, then come back and try again :-)
>
> And that means what? Its just indicative of the complexity of the
> product and the rather famous Gnus Manual in particular. Its not for
> nothing #emacs and other resources are constantly full of new adopters
> pleading for help in getting their email working with Gnus. Nothing to
> do whatsoever with the relative functionality. It is not for the feint
> of heart.

Thanks Richard for responding.

I was going to respond something similar (well except the part related
to #emacs) but I wouldn't have done better.

-- 
Francis

Re: Improving Gnus speed

[Didier Verna]

Richard Riley <rileyrg@googlemail.com> wrote:

> And that means what? Its just indicative of the complexity of the
> product and the rather famous Gnus Manual in particular.

  No, it is indicative of the fact that its needs more energy to
implement mail reading in Gnus than only news reading (and also that it
is more complex to do it). And since this work has been done, as the
documentation proves), it is indicative of the fact that reading mail in
Gnus is in fact at least as important as reading news (since people are
willing to spend so much time and energy on it).

> Its not for nothing #emacs and other resources are constantly full of
> new adopters pleading for help in getting their email working with
> Gnus.

  Err, exactly my point, thanks for supporting me. New adopters indeed
want to get help reading mail with Gnus. This means that new adopters
want to read mail with Gnus. So Gnus is a mail reader (at least as much
as it is a news reader).


> Gnus can most certainly be thought of as primarily a newsreader in my
> mind 

  Not in mine. It is primarily a mail reader for me (but of course
you're entitled to think otherwise). The other poster was also right in
saying that Gnus is a way of life. More precisely, the way of life in
question is to handle mail in a news-like way (auto-expiry etc).

Just for the anecdote, I almost got to writing a Gnus book for O'Reilly
France some years ago. The editor contacted me about it and while we
were discussing the contents and organization, he said that in order to
reach the broadest possible audience, the quickstart had to show how to
use nnimap as the primary select method...

In fact, I would really like if we had serious statistics about Gnus
usage, and I wouldn't be surprised if people read more mails than news
with it.


> and most certainly your claim that its NOT a news reader must be a
> joke. It is isn't it??

  No, it's not a joke. It's a news/mail reader and you've got to
acknowledge the fact that reading mail with it is at least as important
as reading news with it. Guys, remember the original question ? The OP
found it paradoxical that Gnus development discussion happens on a
mailing list. I maintain that if you get that sort of feeling, it's
because your view of what Gnus really is wrong.

-- 
Resistance is futile. You will be jazzimilated.

Scientific site:   http://www.lrde.epita.fr/~didier
Music (Jazz) site: http://www.didierverna.com

Re: Improving Gnus speed

[Richard Riley]

Didier Verna <didier@xemacs.org> writes:

> Richard Riley <rileyrg@googlemail.com> wrote:
>
>> And that means what? Its just indicative of the complexity of the
>> product and the rather famous Gnus Manual in particular.
>
>   No, it is indicative of the fact that its needs more energy to
> implement mail reading in Gnus than only news reading (and also that it
> is more complex to do it). And since this work has been done, as the
> documentation proves), it is indicative of the fact that reading mail in
> Gnus is in fact at least as important as reading news (since people are
> willing to spend so much time and energy on it).

Gnus is still a news reader.

I assumed you were joking too. Apparently not.

The entire look and feel is that of a newsreader. The complexity of the
mail part is as a result of the newsreader approach  - something I
like. Many of course dont preferring more traditional mail clients.

Re: Improving Gnus speed

[Francis Moreau]

Didier Verna <didier@xemacs.org> writes:

[...]

>
> In fact, I would really like if we had serious statistics about Gnus
> usage, and I wouldn't be surprised if people read more mails than news
> with it.
>

I think you're still wrong. You would be surprised to see how much
people use Gnus for news and a seperate mail reader (like VM, Mew,
trn4...) for mails.

-- 
Francis

Re: Improving Gnus speed

[Richard Riley]

Francis Moreau <francis.moro@gmail.com> writes:

> Didier Verna <didier@xemacs.org> writes:
>
> [...]
>
>>
>> In fact, I would really like if we had serious statistics about Gnus
>> usage, and I wouldn't be surprised if people read more mails than news
>> with it.
>>
>
> I think you're still wrong. You would be surprised to see how much
> people use Gnus for news and a seperate mail reader (like VM, Mew,
> trn4...) for mails.

That's certainly my experience from talking to people in #emacs irc
channel. Most people there think Gnus is an unwieldy and slow monster
for mail and prefer to use Mutt or something similar - Despite adverts
to the contrary most still hang on to the belief that Gnus is useless
for IMAP too ;)

Re: Improving Gnus speed

[Adam Sjøgren]

On Tue, 09 Nov 2010 14:55:15 +0100, Francis wrote:

> Didier Verna <didier@xemacs.org> writes:

[...]
>> Because if you think that Gnus is a newsreader, you're wrong...

> :)

> Is that a joke ?

No Gnus is a way of life.


  (Ok, maybe that is a bit much, but Gnus is much more than a newsreader
  :-) <-- ^ not a joke, but a smile nevertheless),

    Adam

-- 
 "Angels can fly because they take themselves lightly."       Adam Sjøgren
                                                         asjo@koldfront.dk

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> Even if articles are cached, Gnus takes some times to display the
> summary buffer:
>
> Entering in the group containing cached articles: 4 secs.
>
> This group contains 230 unread articles.

That's obscenely slow.  Even if the 230 articles were fetched over the
net, it shouldn't take anywhere that long.

> Now display all articles in this group by doing '/ o':
>
> Gnus asks me:
>
>      How many articles from nnml+cache:nntp+news.free.fr:comp.lang.c (default 42555):
>
> Type <RET>.
>
>      Fetching headers for nnml+cache:nntp+news.free.fr:comp.lang.c...
>      < 7 secs>
>      done
>      < 22 secs>
>      Generating summary...
>      < 3 secs>
>      done

That seems rather slow, too, but Gnus is rather slow when rendering
large summary buffers, and I've never understood why, really.  Entering
a 10K group takes a couple of seconds, but entering a 100K group takes
several minutes.  So there's something exponential going on somewhere...

> I'm interested in improving this, so any suggestions are welcome.

(setq debug-on-quit t) and then `C-g'-ing a few times should tell you
what function is taking so long.  `M-x elp-instrument-package RET gnus
RET', doing something, and then `M-x elp-results' should give you a
detailed look.

> BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP
> method only (not git protocol). Is that right ?
>
> IIRC, git protocol is more effective, and I actually don't compile Git
> with HTTP support since it adds some dependency like curl for no good
> reason.

The Gnus sources are so small that just using HTTP seems OK to me,
speedwise.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Even if articles are cached, Gnus takes some times to display the
>> summary buffer:
>>
>> Entering in the group containing cached articles: 4 secs.
>>
>> This group contains 230 unread articles.
>
> That's obscenely slow.  Even if the 230 articles were fetched over the
> net, it shouldn't take anywhere that long.

That's reassuring.

>> Now display all articles in this group by doing '/ o':
>>
>> Gnus asks me:
>>
>>      How many articles from nnml+cache:nntp+news.free.fr:comp.lang.c (default 42555):
>>
>> Type <RET>.
>>
>>      Fetching headers for nnml+cache:nntp+news.free.fr:comp.lang.c...
>>      < 7 secs>
>>      done
>>      < 22 secs>
>>      Generating summary...
>>      < 3 secs>
>>      done
>
> That seems rather slow, too, but Gnus is rather slow when rendering

I forgot to mention that the number 42555 is wrong, there're actually
only 1298 articles in this buffer.

I don't know why it's wrong though.

> large summary buffers, and I've never understood why, really.  Entering
> a 10K group takes a couple of seconds, but entering a 100K group takes
> several minutes.  So there's something exponential going on somewhere...

That's my impression too.

>> I'm interested in improving this, so any suggestions are welcome.
>
> (setq debug-on-quit t) and then `C-g'-ing a few times should tell you
> what function is taking so long.  `M-x elp-instrument-package RET gnus
> RET', doing something, and then `M-x elp-results' should give you a
> detailed look.

ok, I'll try that and see I can find something.

>> BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP
>> method only (not git protocol). Is that right ?
>>
>> IIRC, git protocol is more effective, and I actually don't compile Git
>> with HTTP support since it adds some dependency like curl for no good
>> reason.
>
> The Gnus sources are so small that just using HTTP seems OK to me,
> speedwise.

Well even in that case, why not using the native protocol which is more
efficient, stable ... ?

-- 
Francis

Re: Improving Gnus speed

[Andreas Schwab]

Francis Moreau <francis.moro@gmail.com> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Francis Moreau <francis.moro@gmail.com> writes:
>>
>>> BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP
>>> method only (not git protocol). Is that right ?
>>>
>>> IIRC, git protocol is more effective, and I actually don't compile Git
>>> with HTTP support since it adds some dependency like curl for no good
>>> reason.
>>
>> The Gnus sources are so small that just using HTTP seems OK to me,
>> speedwise.
>
> Well even in that case, why not using the native protocol which is more
> efficient, stable ... ?

It _is_ using the native protocol, over HTTP.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Improving Gnus speed

[Ted Zlatanov]

On Tue, 09 Nov 2010 22:03:35 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 

AS> Francis Moreau <francis.moro@gmail.com> writes:
>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>> The Gnus sources are so small that just using HTTP seems OK to me,
>>> speedwise.
>> 
>> Well even in that case, why not using the native protocol which is more
>> efficient, stable ... ?

AS> It _is_ using the native protocol, over HTTP.

Yeah.  I set it up to use the smart server or something :)

Also, many corporate firewalls including mine don't allow the git
protocol.  So HTTP is a better transport than native in order to serve
the most people.

Ted

Re: Improving Gnus speed

[Richard Riley]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Tue, 09 Nov 2010 22:03:35 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>
> AS> Francis Moreau <francis.moro@gmail.com> writes:
>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>> The Gnus sources are so small that just using HTTP seems OK to me,
>>>> speedwise.
>>> 
>>> Well even in that case, why not using the native protocol which is more
>>> efficient, stable ... ?
>
> AS> It _is_ using the native protocol, over HTTP.
>
> Yeah.  I set it up to use the smart server or something :)
>
> Also, many corporate firewalls including mine don't allow the git
> protocol.  So HTTP is a better transport than native in order to serve
> the most people.
>
> Ted
>

Well, most of those that are limited by a firewall yes. Most people use
the git protocol. Which begs the question : can both not be enabled? The
transport protocol shouldn't affect the repo in any shape nor form
should it?

Re: Improving Gnus speed

[Francis Moreau]

Richard Riley <rileyrg@googlemail.com> writes:

> Ted Zlatanov <tzz@lifelogs.com> writes:
>
>> On Tue, 09 Nov 2010 22:03:35 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>>
>> AS> Francis Moreau <francis.moro@gmail.com> writes:
>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>> The Gnus sources are so small that just using HTTP seems OK to me,
>>>>> speedwise.
>>>> 
>>>> Well even in that case, why not using the native protocol which is more
>>>> efficient, stable ... ?
>>
>> AS> It _is_ using the native protocol, over HTTP.
>>
>> Yeah.  I set it up to use the smart server or something :)
>>
>> Also, many corporate firewalls including mine don't allow the git
>> protocol.  So HTTP is a better transport than native in order to serve
>> the most people.
>>
>
> Well, most of those that are limited by a firewall yes. Most people use
> the git protocol.

FWIW, Gnus is the first open source project that is using HTTP only.

-- 
Francis

Re: Improving Gnus speed

[Francis Moreau]

Francis Moreau <francis.moro@gmail.com> writes:

> Richard Riley <rileyrg@googlemail.com> writes:
>
>> Ted Zlatanov <tzz@lifelogs.com> writes:
>>
>>> On Tue, 09 Nov 2010 22:03:35 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: 
>>>
>>> AS> Francis Moreau <francis.moro@gmail.com> writes:
>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>>>>> The Gnus sources are so small that just using HTTP seems OK to me,
>>>>>> speedwise.
>>>>> 
>>>>> Well even in that case, why not using the native protocol which is more
>>>>> efficient, stable ... ?
>>>
>>> AS> It _is_ using the native protocol, over HTTP.
>>>
>>> Yeah.  I set it up to use the smart server or something :)
>>>
>>> Also, many corporate firewalls including mine don't allow the git
>>> protocol.  So HTTP is a better transport than native in order to serve
>>> the most people.
>>>
>>
>> Well, most of those that are limited by a firewall yes. Most people use
>> the git protocol.
>
> FWIW, Gnus is the first open source project that is using HTTP only.
>

I mean the fisrt open source project I encountered.

-- 
Francis

Re: Improving Gnus speed

[Julien Danjou]

On Tue, Nov 09 2010, Richard Riley wrote:

> Well, most of those that are limited by a firewall yes. Most people use
> the git protocol. Which begs the question : can both not be enabled? The
> transport protocol shouldn't affect the repo in any shape nor form
> should it?

I does not affect anything, it's just a sysadmin problem to install a
git-server.

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: Improving Gnus speed

[Ted Zlatanov]

On Wed, 10 Nov 2010 10:39:00 +0100 Julien Danjou <julien@danjou.info> wrote: 

JD> On Tue, Nov 09 2010, Richard Riley wrote:
>> Well, most of those that are limited by a firewall yes. Most people use
>> the git protocol. Which begs the question : can both not be enabled? The
>> transport protocol shouldn't affect the repo in any shape nor form
>> should it?

JD> I does not affect anything, it's just a sysadmin problem to install a
JD> git-server.

You're assuming there's a need for this.  Is there anyone that can't get
work done using the HTTP/HTTPS transport?

Ted

Re: Improving Gnus speed

[Julien Danjou]

On Wed, Nov 10 2010, Ted Zlatanov wrote:

> You're assuming there's a need for this.  Is there anyone that can't get
> work done using the HTTP/HTTPS transport?

I don't assume anything, I'm just technically answering a technical
question. :)

(My personal opinion is that HTTP is enough and that it's up to the
gnus.org sysadmin to decide if they want to set up the service or not.)

-- 
Julien Danjou
// ᐰ <julien@danjou.info>   http://julien.danjou.info

Re: Improving Gnus speed

[Richard Riley]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 10 Nov 2010 10:39:00 +0100 Julien Danjou <julien@danjou.info> wrote: 
>
> JD> On Tue, Nov 09 2010, Richard Riley wrote:
>>> Well, most of those that are limited by a firewall yes. Most people use
>>> the git protocol. Which begs the question : can both not be enabled? The
>>> transport protocol shouldn't affect the repo in any shape nor form
>>> should it?
>
> JD> I does not affect anything, it's just a sysadmin problem to install a
> JD> git-server.
>
> You're assuming there's a need for this.  Is there anyone that can't get
> work done using the HTTP/HTTPS transport?
>

The git protocol is so much more efficient. And with open source can be
provided free by places like github. Why use a slow and inefficient one
when both can be used? Seems a strange decision. OK, not as bad as bzr
for emacs but  ... ;)

Re: Improving Gnus speed

[Andreas Schwab]

Richard Riley <rileyrg@googlemail.com> writes:

> The git protocol is so much more efficient.

It _is_ using the git protocol.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Improving Gnus speed

[Francis Moreau]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> Francis Moreau <francis.moro@gmail.com> writes:
>>>
>>>> BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP
>>>> method only (not git protocol). Is that right ?
>>>>
>>>> IIRC, git protocol is more effective, and I actually don't compile Git
>>>> with HTTP support since it adds some dependency like curl for no good
>>>> reason.
>>>
>>> The Gnus sources are so small that just using HTTP seems OK to me,
>>> speedwise.
>>
>> Well even in that case, why not using the native protocol which is more
>> efficient, stable ... ?
>
> It _is_ using the native protocol, over HTTP.

Ah that's true, since git version 1.6.6, git has Smart HTTP Transport.

Hopefully this is the Smart protocol which is currently used.

-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

I've done some sloppy benchmarking with this:

(benchmark-elapse 1 (gnus-summary-limit-to-unread))

With varying numbers of articles:

10k  2.3s
20k  5.6s
30k  9.8s
40k 15.8s
50k 20.5s

So it's not linear, but it's not terribly exponential, either.

But I think I know what's probably causing the non-linearity -- it's
probably the article marks stuff.  It `memq's over these lists, and the
longer they are, the longer it takes.  `M P b'-ing the 50k buffer has
been running for a few minutes now, and that's mostly all list
manipulation...

I was going to rewrite all the marks stuff to use ranges directly, and
that should help a great deal here.  I'll get to it one of these
evenings.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Even if articles are cached, Gnus takes some times to display the
>> summary buffer:
>>
>> Entering in the group containing cached articles: 4 secs.
>>
>> This group contains 230 unread articles.
>
> That's obscenely slow.  Even if the 230 articles were fetched over the
> net, it shouldn't take anywhere that long.
>
>> Now display all articles in this group by doing '/ o':
>>
>> Gnus asks me:
>>
>>      How many articles from nnml+cache:nntp+news.free.fr:comp.lang.c (default 42555):
>>
>> Type <RET>.
>>
>>      Fetching headers for nnml+cache:nntp+news.free.fr:comp.lang.c...
>>      < 7 secs>
>>      done
>>      < 22 secs>
>>      Generating summary...
>>      < 3 secs>
>>      done
>
> That seems rather slow, too, but Gnus is rather slow when rendering
> large summary buffers, and I've never understood why, really.  Entering
> a 10K group takes a couple of seconds, but entering a 100K group takes
> several minutes.  So there's something exponential going on somewhere...
>
>> I'm interested in improving this, so any suggestions are welcome.
>
> (setq debug-on-quit t) and then `C-g'-ing a few times should tell you
> what function is taking so long.

Here's my first experiments:

While in the fetching headers... process I got the following backtrace:

  string-match("\\(<[^<]+>\\)[  ]*\\'" "...
  gnus-get-newsgroup-headers-xover
  gnus-fetch-headers
  gnus-summary-insert-articles
  gnus-summary-insert-old-articles


and while in the 22 secs processing, hitting C-g gives:

  parse-time-tokenize
  parse-time-string
  mail-header-parse-date
  gnus-thread-latest-date
  gnus-thread-sort-by-most-recent-date
  gnus-sort-threads-recursive
      .
  33 calls to gnus-sort-threads-recursive
      .
  gnus-sort-threads
  gnus-summary-prepare
  gnus-summary-limit
  gnus-summary-insert-old-articles


> `M-x elp-instrument-package RET gnus RET', doing something, and then
> `M-x elp-results' should give you a detailed look.

This fails with the following message:

     "Variable binding depth exceeds max-specpdl-size"

However doing 'M-x elp-results' still shows me:

  gnus-sort-threads-recursive             641         65.774695999  0.1026126302
  gnus-thread-sort-by-most-recent-date    4348        31.652889000  0.0072798732
  gnus-thread-latest-date                 8696        31.436723000  0.0036150785
  gnus-thread-total-score                 82494       17.555852999  0.0002128136
  gnus-thread-total-score-1               82494       17.134565999  0.0002077068
  gnus-summary-insert-articles            1           8.04944       8.04944
  gnus-fetch-headers                      1           8.019706      8.019706
  gnus-get-newsgroup-headers-xover        1           8.005099      8.005099
  gnus-thread-sort-by-total-score         8697        3.4791829999  0.0004000440
  gnus-id-to-thread                       82494       1.3472719999  1.633...e-05
  gnus-float-time                         41229       0.2444190000  5.928...e-06
  gnus-retrieve-headers                   2           0.0244269999  0.0122134999
  gnus-make-threads                       1           0.023105      0.023105
  gnus-sorted-nunion                      1           0.015179      0.015179
  gnus-cache-retrieve-headers             1           0.012292      0.012292
  [...]



-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

>   gnus-thread-total-score-1               82494       17.134565999  0.00020770

Geez.  How many articles was this?  

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>>   gnus-thread-total-score-1               82494       17.134565999  0.00020770
>
> Geez.  How many articles was this?  

As I said somewhere else I have 1298 articles although Gnus mentions
42555 when prompting how many articles it should fetch, I don't know why
there's such a difference.

Thanks
-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> As I said somewhere else I have 1298 articles although Gnus mentions
> 42555 when prompting how many articles it should fetch, I don't know why
> there's such a difference.

The prompt uses an estimate based on the LOW . HIGH marks in the group.

If you `/ w' after entering, you still only get 1298 articles?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> As I said somewhere else I have 1298 articles although Gnus mentions
>> 42555 when prompting how many articles it should fetch, I don't know why
>> there's such a difference.
>
> The prompt uses an estimate based on the LOW . HIGH marks in the group.
>
> If you `/ w' after entering, you still only get 1298 articles?

Nope, I get only 253 articles (same number after entering the group), it
shows me only threads which have unread articles I think.

If I do '/ o', then it shows me 1298 articles but I have to wait 25
secs.

-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> If I do '/ o', then it shows me 1298 articles but I have to wait 25
> secs.

If you do an `M-x elp-instrument-package RET gnus RET', `/ o', and then
`M-x elp-results', that should tell us what it's doing, I think.

`M-x elp-results' clears out all the counters and stuff, so it's
important that you don't do more than the single command between
instrumenting and showing the result, or between each result set.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> If I do '/ o', then it shows me 1298 articles but I have to wait 25
>> secs.
>
> If you do an `M-x elp-instrument-package RET gnus RET', `/ o', and then
> `M-x elp-results', that should tell us what it's doing, I think.
>
> `M-x elp-results' clears out all the counters and stuff, so it's
> important that you don't do more than the single command between
> instrumenting and showing the result, or between each result set.

I think it's what I did.

Ok just to be sure, we're talking about the same thing, I'm going to
describe what currently is my config and the associated elp results.

Here's my config for my group caches:

       ("^nnml\\+cache"
         (display . all)
         (gnus-use-scoring nil)
         (gnus-thread-sort-functions '((not gnus-thread-sort-by-number))))

So I disabled entirely the scoring stuff since it takes a lot of time
and this is not really important in these kind of groups since they
store only important things (with equal scores).

Therefore, when I'm entering such group, most of the time is now spent
in the fetching process (not the sorting one).

For example, I have a group which contains 1298 articles. Entering it
lasts 9 seconds:

    - 7 secs in the fetching process
    - 2 secs in sorting + summary buffer generation

As you can see, the fetching process is now the annoying step, specially
since all articles for this group are cached so they're on my disk.

I'm giving you the elp results when entering in the group containing
1298 articles, with all articles displayed:

gnus-thread-total-score                                 21809       18.607898999  0.0008532211
gnus-thread-total-score-1                               21805       18.246756999  0.0008368152
gnus-topic-select-group                                 1           10.660905     10.660905
gnus-group-select-group                                 1           10.66085      10.66085
gnus-group-read-group                                   1           10.66084      10.66084
gnus-summary-read-group                                 1           10.660791     10.660791
gnus-summary-read-group-1                               1           10.660778     10.660778
gnus-select-newsgroup                                   1           7.674363      7.674363
gnus-fetch-headers                                      1           6.927848      6.927848
gnus-get-newsgroup-headers-xover                        1           6.78624       6.78624
gnus-summary-number-of-articles-in-thread               22361       3.2177489999  0.0001439000
gnus-summary-prepare                                    1           2.901636      2.901636
gnus-summary-prepare-threads                            1           2.881676      2.881676
gnus-dd-mmm                                             1301        0.9418810000  0.0007239669
gnus-articles-to-read                                   1           0.586238      0.586238
gnus-sorted-difference                                  3           0.268062      0.089354
gnus-summary-limit-children                             1297        0.2439169999  0.0001880624
gnus-id-to-thread                                       21805       0.1913100000  8.773...e-06
gnus-retrieve-headers                                   2           0.153315      0.0766575
gnus-cache-retrieve-headers                             1           0.138898      0.138898
gnus-sort-threads-recursive                             820         0.1360629999  0.0001659304
gnus-run-hooks                                          1310        0.0821950000  6.274...e-05
gnus-summary-highlight-line                             1301        0.0734670000  5.646...e-05
gnus-add-text-properties                                10416       0.0530739999  5.095...e-06
gnus-summary-maybe-hide-threads                         1           0.04941       0.04941
gnus-summary-hide-all-threads                           1           0.0494        0.0494
gnus-put-text-property-excluding-characters-with-faces  1303        0.0478549999  3.672...e-05
gnus-summary-next-thread                                44          0.047207      0.0010728863
gnus-summary-go-to-next-thread                          44          0.046881      0.0010654772
gnus-put-text-property                                  6540        0.0405520000  6.200...e-06
gnus-map-function                                       1325        0.0323199999  2.439...e-05
gnus-summary-from-or-to-or-newsgroups                   1301        0.0305190000  2.345...e-05
gnus-summary-hide-thread                                22          0.025443      0.0011565
gnus-extract-address-components                         1301        0.0206770000  1.589...e-05
gnus-summary-initial-limit                              1           0.015415      0.015415
gnus-killed-articles                                    1           0.013221      0.013221
gnus-simplify-whitespace                                1325        0.0102530000  7.738...e-06
gnus-make-threads                                       1           0.009472      0.009472
gnus-simplify-subject-re                                1325        0.0092170000  6.956...e-06
gnus-sort-threads                                       1           0.008926      0.008926
gnus-build-old-threads                                  1           0.008742      0.008742
gnus-update-missing-marks                               1           0.007904      0.007904
gnus-build-get-header                                   6           0.0077429999  0.0012904999
gnus-summary-setup-buffer                               1           0.006748      0.006748
gnus-summary-mode                                       1           0.005939      0.005939
gnus-uncompress-range                                   1           0.005137      0.005137
gnus-group-update-group                                 1           0.005086      0.005086
gnus-mode-line-buffer-identification                    3           0.005086      0.0016953333
gnus-update-summary-mark-positions                      2           0.00403       0.002015
gnus-summary-insert-line                                4           0.0038380000  0.0009595000
gnus-message                                            7           0.003665      0.0005235714




-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> For example, I have a group which contains 1298 articles. Entering it
> lasts 9 seconds:
>
>     - 7 secs in the fetching process
>     - 2 secs in sorting + summary buffer generation

[...]

> I'm giving you the elp results when entering in the group containing
> 1298 articles, with all articles displayed:
>
> gnus-thread-total-score                                 21809       18.607898999  0.0008532211

elp is still claiming that this function is being called 21k times, and
that this takes 18 seconds.  Which sort of doesn't make much sense, since:

> gnus-group-select-group                                 1           10.66085      10.66085

This is surely the top-level function being called, so the 18 seconds
sorting should be counted here, too, but it isn't, which is why I was
wondering whether you were doing other things between each
elp-results... 

Anyway:

> gnus-get-newsgroup-headers-xover                        1           6.78624       6.78624

There's the seven seconds...

> gnus-summary-number-of-articles-in-thread               22361       3.2177489999  0.0001439000

And there's a totally crazy call number again.  That function should be
called (generally) once per displayed article in the summary buffer (if
you have that in your summary line format).  But it's called 10x the
number of times.

> gnus-dd-mmm                                             1301        0.9418810000  0.0007239669

I'm guessing you have a somewhat non-standard summary line format for
that function to show up, but at least it tallies up with the number of
articles in the group, somewhat.

> gnus-id-to-thread                                       21805       0.1913100000  8.773...e-06

Again, a totally ridiculous call number...

I'd suggest removing all Gnus customisations to find out what triggers
these weird call numbers.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> For example, I have a group which contains 1298 articles. Entering it
>> lasts 9 seconds:
>>
>>     - 7 secs in the fetching process
>>     - 2 secs in sorting + summary buffer generation
>
> [...]
>
>> I'm giving you the elp results when entering in the group containing
>> 1298 articles, with all articles displayed:
>>
>> gnus-thread-total-score                                 21809       18.607898999  0.0008532211
>
> elp is still claiming that this function is being called 21k times, and
> that this takes 18 seconds.  Which sort of doesn't make much sense, since:
>
>> gnus-group-select-group                                 1           10.66085      10.66085
>
> This is surely the top-level function being called, so the 18 seconds
> sorting should be counted here, too, but it isn't, which is why I was
> wondering whether you were doing other things between each
> elp-results... 

I agree with you, these numbers are crazy. Actually I was looking at the
time elapsed, and I can't interpret them at all. It says 18.6 seconds
where as the whole process lasts 10 seconds at most.

I re-did several times the same benchmarks and the figures seem stable.

>
> Anyway:
>
>> gnus-get-newsgroup-headers-xover                        1           6.78624       6.78624
>
> There's the seven seconds...
>

Yes the longuest one

>
>> gnus-summary-number-of-articles-in-thread               22361       3.2177489999  0.0001439000
>
> And there's a totally crazy call number again.  That function should be
> called (generally) once per displayed article in the summary buffer (if
> you have that in your summary line format).  But it's called 10x the
> number of times.
>
>> gnus-dd-mmm                                             1301        0.9418810000  0.0007239669
>
> I'm guessing you have a somewhat non-standard summary line format for
> that function to show up, but at least it tallies up with the number of
> articles in the group, somewhat.

Here's my summary line format if that can help:

  (setq gnus-summary-line-format
        (concat
         "%0{%U%R%z%}" "%10{│%}" "%3t" "%10{│%}"
         "%(%-20,20f %)" "%10{│%}" "%1{%d%}" "%10{│%}"
         "%6V" "%10{│%}" "%10{%B%}" "%s\n"))
  

>> gnus-id-to-thread                                       21805       0.1913100000  8.773...e-06
>
> Again, a totally ridiculous call number...
>
> I'd suggest removing all Gnus customisations to find out what triggers
> these weird call numbers.

Ok, I'm trying...

Let's remove my summary line format customisation... and the numbers are
now:

  gnus-topic-select-group                 1           8.071814      8.071814
  gnus-group-select-group                 1           8.071757      8.071757
  gnus-group-read-group                   1           8.071748      8.071748
  gnus-summary-read-group                 1           8.071693      8.071693
  gnus-summary-read-group-1               1           8.071681      8.071681
  gnus-select-newsgroup                   1           7.30483       7.30483
  gnus-summary-limit-children             1324        7.1840629999  0.0054260294
  gnus-fetch-headers                      1           6.445576      6.445576
  gnus-get-newsgroup-headers-xover        1           6.420683      6.420683
  gnus-articles-to-read                   1           0.818016      0.818016
  gnus-summary-prepare                    1           0.573297      0.573297
  gnus-summary-prepare-threads            1           0.553526      0.553526
  gnus-run-hooks                          1337        0.2399320000  0.0001794554
  gnus-summary-highlight-line             1326        0.2316510000  0.0001746990
  gnus-sort-threads-recursive             837         0.1313619999  0.0001569438
  gnus-summary-initial-limit              1           0.10973       0.10973
  gnus-summary-highlight-line-0           1326        0.0972370000  7.333...e-05
  gnus-summary-maybe-hide-threads         1           0.04301       0.04301
  gnus-summary-hide-all-threads           1           0.043         0.043
  gnus-summary-next-thread                44          0.0411179999  0.0009344999
  gnus-summary-go-to-next-thread          44          0.040834      0.0009280454
  gnus-retrieve-headers                   2           0.03551       0.017755
  gnus-map-function                       1352        0.0294150000  2.175...e-05

which seem much more sensible.

Now restoring my summary line format and removing only the "%6V" format
specifier gives almost the same sane results.

-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> Let's remove my summary line format customisation... and the numbers are
> now:
>
>   gnus-topic-select-group                 1           8.071814      8.071814

[...]

>   gnus-summary-limit-children             1324        7.1840629999  0.0054260294

This seems wrong...  I mean, the elapsed time.  I guess elp doesn't give
reliable total times for shorter functions...

>   gnus-get-newsgroup-headers-xover        1           6.420683      6.420683

This still takes almost seven seconds, which is weird.  Do you have any
xover-parsing customisations?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Let's remove my summary line format customisation... and the numbers are
>> now:
>>
>>   gnus-topic-select-group                 1           8.071814      8.071814
>
> [...]
>
>>   gnus-summary-limit-children             1324        7.1840629999  0.0054260294
>
> This seems wrong...  I mean, the elapsed time.  I guess elp doesn't give
> reliable total times for shorter functions...
>
>>   gnus-get-newsgroup-headers-xover        1           6.420683      6.420683
>
> This still takes almost seven seconds, which is weird.  Do you have any
> xover-parsing customisations?

I'm afraid I haven't:

   $ grep xover .gnus

shows nothing.

-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

>> This still takes almost seven seconds, which is weird.  Do you have any
>> xover-parsing customisations?
>
> I'm afraid I haven't:
>
>    $ grep xover .gnus
>
> shows nothing.

What about `gnus-decode-encoded-word-function'?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>>> This still takes almost seven seconds, which is weird.  Do you have any
>>> xover-parsing customisations?
>>
>> I'm afraid I haven't:
>>
>>    $ grep xover .gnus
>>
>> shows nothing.
>
> What about `gnus-decode-encoded-word-function'?

I haven't set this one.

The only occurence of the word 'decode' in my .gnus is for this:

  (eval-after-load "mm-decode"
    '(progn
       (add-to-list 'mm-discouraged-alternatives "text/html")
       (add-to-list 'mm-discouraged-alternatives "text/richtext")))

Thanks
-- 
Francis

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>>   gnus-thread-total-score-1               82494       17.134565999  0.00020770
>
> Geez.  How many articles was this?  

BTW, even if scoring is disabled in this group,
'gnus-thread-total-score' is still called a lot of time.

I can verify that scoring is disabled since all articles have a null
score in this group.

However looking at the backtrace below, it seems that the score is still
calculated...

Is it correct ?

The Backtrace is:

* gnus-thread-total-score(([8781 "Idees pour resoudre un probleme" "Francis Moreau <francis.moro@gmail.
  (format "%6d" (gnus-thread-total-score (and ... ...)))					       
  (insert (format "%6d" (gnus-thread-total-score ...)))						       
  (progn (gnus-add-text-properties (point) (progn ... ...) (quote ...)) (gnus-add-text-properties (poin
  eval((progn (gnus-add-text-properties (point) (progn ... ...) (quote ...)) (gnus-add-text-properties 
  (progn (eval gnus-summary-line-format-spec) (point))						       

  (gnus-put-text-property (point) (progn (eval gnus-summary-line-format-spec) (point)) (quote gnus-numb
  (progn (when (and gnus-tmp-dummy-line ...) (gnus-summary-insert-dummy-line gnus-tmp-dummy-line ...) (
  (if gnus-tmp-header (progn (when ... ... ...) (setq gnus-tmp-unread ...) (push ... gnus-newsgroup-dat
  (when gnus-tmp-header (when (and gnus-tmp-dummy-line ...) (gnus-summary-insert-dummy-line gnus-tmp-du
  (if (stringp gnus-tmp-header) (cond (... ... ... ... ...) (... ... ...) (... ... ...) (t ...)) (setq 
  (while (or threads stack gnus-tmp-new-adopts new-roots) (if (and ... ... ... ...) (if gnus-tmp-new-ad
  (if (vectorp (car threads)) (gnus-summary-prepare-unthreaded threads) (if gnus-summary-display-while-
  (let ((gnus-tmp-level 0) (default-score ...) (gnus-visual-p ...) (building-line-count gnus-summary-di

  gnus-summary-prepare-threads((([8781 "Idees pour resoudre un probleme" "Francis Moreau <francis.moro@
  (progn (gnus-summary-prepare-threads (if gnus-show-threads ... ...)))				       
  (if gnus-newsgroup-headers (progn (gnus-summary-prepare-threads ...)))			       
  (when gnus-newsgroup-headers (gnus-summary-prepare-threads (if gnus-show-threads ... ...)))	       
  (let ((inhibit-read-only t)) (erase-buffer) (setq gnus-newsgroup-data nil gnus-newsgroup-data-reverse

  gnus-summary-prepare()									       
  (if no-display nil (gnus-summary-prepare))							       
  (unless no-display (gnus-summary-prepare))							       
  (cond ((not new-group) (gnus-set-global-variables) (when kill-buffer ...) (gnus-configure-windows ...

Thanks
-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> BTW, even if scoring is disabled in this group,
> 'gnus-thread-total-score' is still called a lot of time.

This function isn't dependent on scoring being enabled -- it's a thread
sorting function that examines the total score of each thread.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> BTW, even if scoring is disabled in this group,
>> 'gnus-thread-total-score' is still called a lot of time.
>
> This function isn't dependent on scoring being enabled -- it's a thread
> sorting function that examines the total score of each thread.

ah ok, but what the point to examine the total score if scoring is
disabled ?

-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> ah ok, but what the point to examine the total score if scoring is
> disabled ?

If the user disables scoring, but still insists on using the total
scoring for the sorting, then, er, Gnus could discover that, but it
seems like a pretty odd situation.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> ah ok, but what the point to examine the total score if scoring is
>> disabled ?
>
> If the user disables scoring, but still insists on using the total
> scoring for the sorting, then, er, Gnus could discover that, but it
> seems like a pretty odd situation.

Well, this how this happens for my config.

I globaly set my thread sorting function to:

  (setq gnus-thread-sort-functions
        '((not gnus-thread-sort-by-number)
          gnus-thread-sort-by-total-score))

But it appears that for some groups (the cached one), scoring is not
really appropriate, so I just use group parameter for these groupe to
disable scoring.

I don't know, maybe it's just me, but I would have expected Gnus to
automatically not spend time in calculating scores specially since all
scores values are shown as 0 in the summary buffer.
-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

>   gnus-thread-total-score-1               82494       17.134565999  0.0002077068

And I guess that this means that you're sorting the threads by total
score?  If you try normal sorting, does group entry time become OK?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>>   gnus-thread-total-score-1               82494       17.134565999  0.0002077068
>
> And I guess that this means that you're sorting the threads by total
> score? 

Yes I have this in my .gnus:

  ("^nntp\\|gmane\\."
   (display . all)
    (gnus-use-scoring t)
     (gnus-thread-sort-functions '((not gnus-thread-sort-by-number)
                                    gnus-thread-sort-by-total-score)))))


> If you try normal sorting, does group entry time become OK?

Well the group entry still lasts 3 seconds.

And when showing all articles in this group '\ o', the 22 secs time
now lasts 17 seconds...

So there's an improvement, but I don't think that's ok.

Thanks
-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

>> If you try normal sorting, does group entry time become OK?
>
> Well the group entry still lasts 3 seconds.

So it goes from 27 seconds to 3 seconds?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>>> If you try normal sorting, does group entry time become OK?
>>
>> Well the group entry still lasts 3 seconds.
>
> So it goes from 27 seconds to 3 seconds?

Nope, entering the group was 4 seconds and if I remove total scoring it
decreases to 3 seconds.

-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> Nope, entering the group was 4 seconds and if I remove total scoring it
> decreases to 3 seconds.

Right.  Hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> and while in the 22 secs processing, hitting C-g gives:
>
>   parse-time-tokenize
>   parse-time-string
>   mail-header-parse-date
>   gnus-thread-latest-date
>   gnus-thread-sort-by-most-recent-date
>   gnus-sort-threads-recursive
>       .
>   33 calls to gnus-sort-threads-recursive
>       .
>   gnus-sort-threads

...

>   gnus-sort-threads-recursive             641         65.774695999  0.1026126302
>   gnus-thread-sort-by-most-recent-date    4348        31.652889000  0.0072798732
>   gnus-thread-latest-date                 8696        31.436723000  0.0036150785

...

> Yes I have this in my .gnus:
> 
>   ("^nntp\\|gmane\\."
>    (display . all)
>     (gnus-use-scoring t)
>      (gnus-thread-sort-functions '((not gnus-thread-sort-by-number)
>                                     gnus-thread-sort-by-total-score)))))

Why are there calls to gnus-thread-sort-by-most-recent-date above, even
though that's not listed as a sort function?  In my experience, it is
very slow, so eliminating it should help.  I did some work to speed it
up, but if anyone else can improve it further, that would be good.

Dan

Re: Improving Gnus speed

[Francis Moreau]

Dan Christensen <jdc@uwo.ca> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> and while in the 22 secs processing, hitting C-g gives:
>>
>>   parse-time-tokenize
>>   parse-time-string
>>   mail-header-parse-date
>>   gnus-thread-latest-date
>>   gnus-thread-sort-by-most-recent-date
>>   gnus-sort-threads-recursive
>>       .
>>   33 calls to gnus-sort-threads-recursive
>>       .
>>   gnus-sort-threads
>
> ...
>
>>   gnus-sort-threads-recursive             641         65.774695999  0.1026126302
>>   gnus-thread-sort-by-most-recent-date    4348        31.652889000  0.0072798732
>>   gnus-thread-latest-date                 8696        31.436723000  0.0036150785
>
> ...
>
>> Yes I have this in my .gnus:
>> 
>>   ("^nntp\\|gmane\\."
>>    (display . all)
>>     (gnus-use-scoring t)
>>      (gnus-thread-sort-functions '((not gnus-thread-sort-by-number)
>>                                     gnus-thread-sort-by-total-score)))))
>
> Why are there calls to gnus-thread-sort-by-most-recent-date above, even
> though that's not listed as a sort function? 

You're right the actual sort functions used were:

   (setq gnus-thread-sort-functions
         '(gnus-thread-sort-by-most-recent-date
           gnus-thread-sort-by-total-score))


> In my experience, it is very slow, so eliminating it should help.  I
> did some work to speed it up, but if anyone else can improve it
> further, that would be good.

Well, that's a crude workaround since this what I'd like to use.

But why are there so many calls to gnus-thread-total-score and
gnus-thread-sort-by-most-recent-date knowing that my group contains only
1298 articles ?

Do you have any hint to improve it further ?

Removing gnus-thread-sort-by-most-recent-date considerably accelerate
the thread sorting time.

Also do you know why the fetching process is so long (~ 7secs) specially
since the group contains only cached articles ?

Thanks
-- 
Francis

Re: Improving Gnus speed

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> But why are there so many calls to gnus-thread-total-score and
> gnus-thread-sort-by-most-recent-date knowing that my group contains only
> 1298 articles ?

The way thread sorting works is that not only are the threads sorted,
but the children of each article are also sorted using the same
function.  So it gets called recursively a lot of times.  That's
why a change I made to cache the parsed date/time info makes such
a bit difference.

>  gnus-thread-sort-by-most-recent-date    4348        31.652889000  0.0072798732
>  gnus-thread-total-score-1               82494       17.134565999  0.0002077068
>  gnus-thread-sort-by-total-score         8697        3.4791829999  0.0004000440

On the other hand, I'm still not sure why the above functions are called
that many times.  I wonder if the recursion is going one level deeper
than necessary?  Or if the overall sorting as well as the sorting of
children can be accomplished in a unified way, without repeating so
much work?  E.g., maybe first each thread could be traversed, and on
the way back up from the leaves to the root, you propagate to each
node the most recent date seen by any of the children, storing this
as some extra data with each article.  Then you just sort using this
stored data.

> Also do you know why the fetching process is so long (~ 7secs) specially
> since the group contains only cached articles ?

Sorry, I don't know anything about that.

Dan

Re: Improving Gnus speed

[Francis Moreau]

Dan Christensen <jdc@uwo.ca> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> But why are there so many calls to gnus-thread-total-score and
>> gnus-thread-sort-by-most-recent-date knowing that my group contains only
>> 1298 articles ?
>
> The way thread sorting works is that not only are the threads sorted,
> but the children of each article are also sorted using the same
> function.  So it gets called recursively a lot of times.  That's
> why a change I made to cache the parsed date/time info makes such
> a bit difference.

Hmm, I'm wondering if sorting the 'children' for a given thread is
really important. I would say that I could live without it I think
specially if it makes group entering faster.


Or better, when entering in a group, threads are not expanded, and
sorting children could happen when expanding threads. That would make
the group entering much faster and children could still be sorted.

-- 
Francis

Re: Improving Gnus speed

[Francis Moreau]

Francis Moreau <francis.moro@gmail.com> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Francis Moreau <francis.moro@gmail.com> writes:
>>
>>> But why are there so many calls to gnus-thread-total-score and
>>> gnus-thread-sort-by-most-recent-date knowing that my group contains only
>>> 1298 articles ?
>>
>> The way thread sorting works is that not only are the threads sorted,
>> but the children of each article are also sorted using the same
>> function.  So it gets called recursively a lot of times.  That's
>> why a change I made to cache the parsed date/time info makes such
>> a bit difference.
>
> Hmm, I'm wondering if sorting the 'children' for a given thread is
> really important. I would say that I could live without it I think
> specially if it makes group entering faster.

Indeed changing the definition of gnus-sort-threads-recursive like the
following:

	(defun gnus-sort-threads-recursive (threads func)
	  (sort threads func))

make the sorting time almost null.

And when looking at the threads, I even don't make any differences with
the previous method.

Now the time is totaly spent in the fetching process thing.


> Or better, when entering in a group, threads are not expanded, and
> sorting children could happen when expanding threads. That would make
> the group entering much faster and children could still be sorted.

-- 
Francis

Re: Improving Gnus speed

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Hmm, I'm wondering if sorting the 'children' for a given thread is
>> really important. I would say that I could live without it I think
>> specially if it makes group entering faster.
>
> Indeed changing the definition of gnus-sort-threads-recursive like the
> following:
>
> 	(defun gnus-sort-threads-recursive (threads func)
> 	  (sort threads func))
>
> make the sorting time almost null.

In my tests, if one of the sort functions is
gnus-thread-sort-by-most-recent-date, then the above takes almost as
long as sorting all of the children too, since it needs to convert
*all* time strings to emacs times, and that dominates.

As you pointed out, the real thing to investigate is the rest of
the time taken for summary buffer generation.  Any takers?

Dan

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> As you pointed out, the real thing to investigate is the rest of
> the time taken for summary buffer generation.  Any takers?

For large buffers, a lot of the time is spent on marks management.  The
various marks are list lists of numbers, and `memq' is being used to
check for their presence.

I've had fixing this on my agenda for a few months, but it hasn't
happened yet.  :-)  The plan is to not uncompress the ranges into lists,
and use `gnus-member-of-range' instead:

(setq range '((1 . 10000) (20000 . 30000)))
(length (setq list (gnus-uncompress-range range)))

(benchmark-run-compiled 20000 (memq 15000 list))
1.4s

(benchmark-run-compiled 20000 (gnus-member-of-range 15000 range))
0.1s

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Miles Bader]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> I've had fixing this on my agenda for a few months, but it hasn't
> happened yet.  :-)  The plan is to not uncompress the ranges into lists,
> and use `gnus-member-of-range' instead:

Hmm, so should vastly reduce consing too...

-miles

-- 
.Numeric stability is probably not all that important when you're guessing.

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Miles Bader <miles@gnu.org> writes:

> Hmm, so should vastly reduce consing too...

Yup.  I hope that it'll have a significant effect overall when entering
large groups, but I think the proof is hidden inside a pudding
somewhere...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

And if that doesn't help, the ranges can be rewritten to balanced binary
trees.  (I think that was Stefan's suggestion...)

That is, if you look at my list of replied articles in this group:

  (reply 56100 56109 56168 56244 56260 56335 56346 56368
	 (56402 . 56403)
	 (56462 . 56463)
	 56531 56536 56578
	 (56597 . 56598)
	 56670 56717 56747 56791
	 (56798 . 56799)
	 56825 56827 56829
	 (56839 . 56840)
	 56842 56850 56852 56856 56859 56862 56882 56909 56926 56977 57007 57019 57088 57093 57110 57127 57139 57146
	 (57152 . 57153)

[gazillions more entries dropped]

then doing `gnus-member-of-range' on those are probably not going to be
very fast.  But if they were arranged as a balanced binary tree, then
it'd be only O(log n) to find them.  And the conversion from list-based
ranges to balanced binary trees is probably fast.  I think.

Has anybody written such a function in Lisp, by any chance?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Daniel Pittman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Miles Bader <miles@gnu.org> writes:
>
>> Hmm, so should vastly reduce consing too...
>
> Yup.  I hope that it'll have a significant effect overall when entering
> large groups, but I think the proof is hidden inside a pudding somewhere...

FWIW, I started using offlineimap and Dovecot to sync from my Zimbra server
because NNIMAP / Gnus was expanding the entire range at a few points just to
do this.

Which would cons a range of ~ 100,000 integers, do stuff, and then free it
all.  Which was ... painful.  I never got around to identifying exactly where
it was, though.

Anyway, I would be very happy to see this change. :)
        Daniel
-- 
✣ Daniel Pittman            ✉ daniel@rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons

Re: Improving Gnus speed

[Francis Moreau]

Dan Christensen <jdc@uwo.ca> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Francis Moreau <francis.moro@gmail.com> writes:
>>
>>> Hmm, I'm wondering if sorting the 'children' for a given thread is
>>> really important. I would say that I could live without it I think
>>> specially if it makes group entering faster.
>>
>> Indeed changing the definition of gnus-sort-threads-recursive like the
>> following:
>>
>> 	(defun gnus-sort-threads-recursive (threads func)
>> 	  (sort threads func))
>>
>> make the sorting time almost null.
>
> In my tests, if one of the sort functions is
> gnus-thread-sort-by-most-recent-date, then the above takes almost as
> long as sorting all of the children too, since it needs to convert
> *all* time strings to emacs times, and that dominates.

Why are all time strings converted to emacs times in that case ?

Thanks
-- 
Francis

Re: Improving Gnus speed

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> In my tests, if one of the sort functions is
>> gnus-thread-sort-by-most-recent-date, then the above takes almost as
>> long as sorting all of the children too, since it needs to convert
>> *all* time strings to emacs times, and that dominates.
>
> Why are all time strings converted to emacs times in that case ?

Because to know the most recent date in a thread, you have to compare
all of the dates of articles in that thread.

If you only want to sort by the date of the root of the thread, then
you switch to a sort function with a slightly different name, and
you would then find that sorting just the top level threads is
much faster than also sorting all children as well.

(On the other hand, I think later in the summary buffer preparation
process, Gnus would take the time to convert all date strings to emacs
times *anyways*, for display in the summary buffer.  So the *overall*
time for Summary buffer generation would be about the same no matter
which of the sort methods you used.)

Dan

Re: Improving Gnus speed

[Francis Moreau]

Dan Christensen <jdc@uwo.ca> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Dan Christensen <jdc@uwo.ca> writes:
>>
>>> In my tests, if one of the sort functions is
>>> gnus-thread-sort-by-most-recent-date, then the above takes almost as
>>> long as sorting all of the children too, since it needs to convert
>>> *all* time strings to emacs times, and that dominates.
>>
>> Why are all time strings converted to emacs times in that case ?
>
> Because to know the most recent date in a thread, you have to compare
> all of the dates of articles in that thread.

Really ?

I would think that you just have to compare the date of all thread *leafs*
only.

No ?

-- 
Francis

Re: Improving Gnus speed

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> I would think that you just have to compare the date of all thread *leafs*
> only.

You can't always assume that the References headers are correct, or that
the Date headers provided are correct, etc.  

Moreover, it doesn't matter, since the dates will be parsed later
anyways.  Since the conversion is cached, all of this only affects
when the time is taken to do the conversion, not the total time.

On my laptop, with 6500 articles, we're only talking about 1.3 seconds,
so this is already pretty good.  But I suspect it can be sped up.

Dan

Re: Improving Gnus speed

[Francis Moreau]

Dan Christensen <jdc@uwo.ca> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> I would think that you just have to compare the date of all thread *leafs*
>> only.
>
> You can't always assume that the References headers are correct, or that
> the Date headers provided are correct, etc.  
>
> Moreover, it doesn't matter, since the dates will be parsed later
> anyways.  Since the conversion is cached, all of this only affects
> when the time is taken to do the conversion, not the total time.
>
> On my laptop, with 6500 articles, we're only talking about 1.3 seconds,
> so this is already pretty good.  But I suspect it can be sped up.

I agree this is good.

But that's no what I get with 1300 *cached* articles. I still have
almost 10 seconds to get the summary buffer. I initialy had more than 20
seconds and had to disable the scoring (okay that's pretty useless for
my cached groups but still).

-- 
Francis

Re: Improving Gnus speed

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> On my laptop, with 6500 articles, we're only talking about 1.3 seconds,
>> so this is already pretty good.  But I suspect it can be sped up.
>
> I agree this is good.
>
> But that's no what I get with 1300 *cached* articles. I still have
> almost 10 seconds to get the summary buffer. 

The 1.3 seconds I mentioned was just the time for date parsing.
It takes something like 13 or 15 seconds for the full summary buffer
generation in my test group.  

Dan

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

>>> On my laptop, with 6500 articles, we're only talking about 1.3 seconds,
>>> so this is already pretty good.  But I suspect it can be sped up.

[...]

> The 1.3 seconds I mentioned was just the time for date parsing.
> It takes something like 13 or 15 seconds for the full summary buffer
> generation in my test group.  

With only 6500 articles?  What kind of CPU would that be?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>>>> On my laptop, with 6500 articles, we're only talking about 1.3 seconds,
>>>> so this is already pretty good.  But I suspect it can be sped up.
>
> [...]
>
>> The 1.3 seconds I mentioned was just the time for date parsing.
>> It takes something like 13 or 15 seconds for the full summary buffer
>> generation in my test group.  
>
> With only 6500 articles?  What kind of CPU would that be?

Core 2 Duo 2.2GHz.  Here are the top elp-results from just entering this
group, which has 6556 articles.  Seems to be a lot of different things
all contributing to this time.  Sorting is more than I expected: 6.3s.
But I'm a bit confused:  I think times in the second-last column include
the time spent in called functions, but gnus-sort-threads takes only
3.3s while gnus-sort-threads-recursive takes 6.3s.  I thought g-s-t-r
was only called from g-s-t!?

And the other day, I posted the following timing info from a freshly
started emacs:

(benchmark-run 1 (gnus-sort-threads threads))
(1.638083 8 0.38992400000000016)

I'm also confused by the number of calls to gnus-thread-total-score-1 (22704).

First the data from a day-old emacs;  afterwards, the data for a fresh emacs.

gnus-topic-read-group                                                         1           14.067652     14.067652
gnus-group-read-group                                                         1           14.067596     14.067596
gnus-summary-read-group                                                       1           14.067559     14.067559
gnus-summary-read-group-1                                                     1           14.067547     14.067547
gnus-summary-prepare                                                          1           9.758998      9.758998
gnus-sort-threads-recursive                                                   3844        6.3334839999  0.0016476285
gnus-summary-prepare-threads                                                  1           5.421154      5.421154
gnus-select-newsgroup                                                         1           4.298248      4.298248
gnus-fetch-headers                                                            1           4.274015      4.274015
gnus-sort-threads                                                             1           3.306262      3.306262
gnus-thread-sort-by-most-recent-date                                          17741       3.1533499999  0.0001777436
gnus-thread-latest-date                                                       35482       3.0043529999  8.467...e-05
gnus-get-newsgroup-headers                                                    1           2.925865      2.925865
gnus-retrieve-headers                                                         2           2.6951549999  1.3475774999
gnus-user-format-function-s                                                   6558        1.6891390000  0.0002575692
gnus-replace-in-string                                                        6559        1.6063970000  0.0002449149
gnus-thread-total-score                                                       22704       1.4428329999  6.354...e-05
gnus-cache-retrieve-headers                                                   1           1.347705      1.347705
gnus-thread-total-score-1                                                     22704       1.3222239999  5.823...e-05
gnus-run-hooks                                                                11          0.914453      0.0831320909
gnus-registry-register-message-ids                                            1           0.9140429999  0.9140429999
gnus-registry-fetch-message-id-fast                                           6556        0.8349619999  0.0001273584
gnus-user-date                                                                6558        0.5879020000  8.964...e-05
gnus-simplify-subject-fuzzy                                                   8487        0.4746110000  5.592...e-05
gnus-user-format-function-t                                                   6558        0.4306520000  6.566...e-05
gnus-summary-from-or-to-or-newsgroups                                         6558        0.3486270000  5.316...e-05
gnus-summary-highlight-line                                                   6558        0.3438830000  5.243...e-05
gnus-float-time                                                               134926      0.3180409999  2.357...e-06
gnus-put-text-property                                                        26236       0.2860350000  1.090...e-05
gnus-seconds-year                                                             6537        0.1706889999  2.611...e-05
gnus-put-text-property-excluding-characters-with-faces                        6559        0.1672420000  2.549...e-05


After restarting emacs, the numbers look a bit better (especially
gnus-sort-threads-recursive): 

gnus-topic-read-group                                   1           13.214392     13.214392
gnus-group-read-group                                   1           13.214339     13.214339
gnus-summary-read-group                                 1           13.214303     13.214303
gnus-summary-read-group-1                               1           13.214291     13.214291
gnus-summary-prepare                                    1           9.061338      9.061338
gnus-summary-prepare-threads                            1           5.205671      5.205671
gnus-sort-threads-recursive                             3844        4.8261949999  0.0012555137
gnus-select-newsgroup                                   1           4.1157070000  4.1157070000
gnus-fetch-headers                                      1           4.095992      4.095992
gnus-get-newsgroup-headers                              1           2.736999      2.736999
gnus-retrieve-headers                                   2           2.7169679999  1.3584839999
gnus-sort-threads                                       1           2.71412       2.71412
gnus-thread-sort-by-most-recent-date                    17741       2.6269509999  0.0001480723
gnus-thread-latest-date                                 35482       2.4759769999  6.978...e-05
gnus-thread-total-score                                 22704       2.1373710000  9.414...e-05
gnus-thread-total-score-1                               22704       1.9588889999  8.627...e-05
gnus-user-format-function-s                             6558        1.7538890000  0.0002674426
gnus-replace-in-string                                  6559        1.6119780000  0.0002457658
gnus-cache-retrieve-headers                             1           1.358604      1.358604
gnus-run-hooks                                          13          0.972687      0.0748220769
gnus-registry-register-message-ids                      1           0.972067      0.972067
gnus-registry-fetch-message-id-fast                     6556        0.8265469999  0.0001260748
gnus-user-format-function-t                             6558        0.6889870000  0.0001050605
gnus-user-date                                          6558        0.5876640000  8.961...e-05
gnus-simplify-subject-fuzzy                             8487        0.4407050000  5.192...e-05
gnus-float-time                                         134926      0.3646349999  2.702...e-06
gnus-summary-from-or-to-or-newsgroups                   6558        0.2739099999  4.176...e-05
gnus-summary-highlight-line                             6558        0.2018580000  3.078...e-05
gnus-seconds-year                                       6537        0.1698009999  2.597...e-05
gnus-put-text-property                                  26236       0.1380570000  5.262...e-06
gnus-seconds-today                                      6558        0.1306740000  1.992...e-05
gnus-registry-fetch-groups                              6556        0.0946810000  1.444...e-05
gnus-put-text-property-excluding-characters-with-faces  6559        0.0879599999  1.341...e-05
gnus-make-threads                                       1           0.086339      0.086339
gnus-gather-threads-by-subject                          1           0.077483      0.077483
gnus-id-to-thread                                       22704       0.0766690000  3.376...e-06
gnus-extract-address-components                         6558        0.0762749999  1.163...e-05

Dan

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> gnus-topic-read-group            1           13.214392     13.214392
> gnus-group-read-group            1           13.214339     13.214339
> gnus-summary-read-group          1           13.214303     13.214303
> gnus-summary-read-group-1        1           13.214291     13.214291
> gnus-summary-prepare             1           9.061338      9.061338
> gnus-summary-prepare-threads     1           5.205671      5.205671
> gnus-sort-threads-recursive      3844        4.8261949999  0.0012555137
> gnus-select-newsgroup            1           4.1157070000  4.1157070000
> gnus-fetch-headers               1           4.095992      4.095992
> gnus-get-newsgroup-headers       1           2.736999      2.736999
> gnus-retrieve-headers            2           2.7169679999  1.3584839999
> gnus-sort-threads                1           2.71412       2.71412
> gnus-thread-sort-by-most-recen   17741       2.6269509999  0.0001480723
> gnus-thread-latest-date          35482       2.4759769999  6.978...e-05
> gnus-thread-total-score          22704       2.1373710000  9.414...e-05
> gnus-thread-total-score-1        22704       1.9588889999  8.627...e-05
> gnus-user-format-function-s      6558        1.7538890000  0.0002674426
> gnus-replace-in-string           6559        1.6119780000  0.0002457658
> gnus-cache-retrieve-headers      1           1.358604      1.358604
> gnus-run-hooks                   13          0.972687      0.0748220769
> gnus-registry-register-message   1           0.972067      0.972067
> gnus-registry-fetch-message-id   6556        0.8265469999  0.0001260748
> gnus-user-format-function-t      6558        0.6889870000  0.0001050605
> gnus-user-date                   6558        0.5876640000  8.961...e-05
> gnus-simplify-subject-fuzzy      8487        0.4407050000  5.192...e-05
> gnus-float-time                  134926      0.3646349999  2.702...e-06

Here's me entering a group with 6000 articles (3GHz Core 2):

gnus-group-select-group                1           3.819664      3.819664
gnus-group-read-group                  1           3.81966       3.81966
gnus-summary-read-group                1           3.819644      3.819644
gnus-summary-read-group-1              1           3.819639      3.819639
gnus-summary-prepare                   1           1.736278      1.736278
gnus-summary-prepare-threads           1           1.492577      1.492577
gnus-select-newsgroup                  1           1.308981      1.308981
gnus-fetch-headers                     1           1.178148      1.178148
gnus-get-newsgroup-headers-xover       1           1.041805      1.041805
gnus-possibly-score-headers            1           0.693136      0.693136
gnus-score-headers                     1           0.69119       0.69119
gnus-score-string                      2           0.6885730000  0.3442865000
gnus-retrieve-headers                  2           0.272322      0.136161
gnus-sort-threads-recursive            324         0.210677      0.0006502376
gnus-sort-threads                      1           0.209757      0.209757
gnus-put-text-property                 17794       0.1451210000  8.155...e-06
gnus-summary-highlight-line            5930        0.1418359999  2.391...e-05
gnus-cache-retrieve-headers            1           0.136199      0.136199
gnus-put-text-property-excluding-char  5931        0.0833610000  1.405...e-05
gnus-summary-initial-limit             1           0.074482      0.074482
gnus-summary-from-or-to-or-newsgroups  5930        0.0732839999  1.235...e-05
gnus-articles-to-read                  1           0.070914      0.070914
gnus-summary-limit-children            5928        0.0673060000  1.135...e-05
gnus-score-string<                     125555      0.0566680000  4.513...e-07

WTF?  125K calls to gnus-score-string<?  I'm starting to wonder whether
ELP works properly...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> Hmm, I'm wondering if sorting the 'children' for a given thread is
> really important. I would say that I could live without it I think
> specially if it makes group entering faster.

It's annoying when you get the wrong sorting for gathered threads.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Hmm, I'm wondering if sorting the 'children' for a given thread is
>> really important. I would say that I could live without it I think
>> specially if it makes group entering faster.
>
> It's annoying when you get the wrong sorting for gathered threads.

But you can still sort children of *gathered* thread although that you
probably won't sort anything since for this thread, each article has
only one children.

I was just wondering about 'normal' threads.
-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Francis Moreau <francis.moro@gmail.com> writes:

> But you can still sort children of *gathered* thread although that you
> probably won't sort anything since for this thread, each article has
> only one children.

Each gathered thread may have many children...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Francis Moreau]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> But you can still sort children of *gathered* thread although that you
>> probably won't sort anything since for this thread, each article has
>> only one children.
>
> Each gathered thread may have many children...

Well all gathered thread I saw have several children indeed but each
child had only one child:

   t0
     -> t1
          -> t2
               -> t3
                    -> ...

But gathered thread is not really important I think since it represents
only a minor part.

-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> That's why a change I made to cache the parsed date/time info makes
> such a bit difference.

Do you have a patch for this?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> That's why a change I made to cache the parsed date/time info makes
>> such a bit difference.
>
> Do you have a patch for this?

It was merged in June.

Dan

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> It was merged in June.

Ok.  :-)

It doesn't cache total-thread-score, then, I'm assuming...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> It doesn't cache total-thread-score, then, I'm assuming...

No, in fact, the only change was to make sure that gnus-date-get-time is
used everywhere, since that function caches the result of
safe-date-to-time as a text-property.

Separate from that low-level cache, the sorting function could
probably annotate the thread tree and re-use that information
for all of the recursive sorts.  Or use a decorate-sort-undecorate
pattern.  I haven't thought through the details, though, so I
pose this as a challenge for someone.  :-)

Dan

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> Separate from that low-level cache, the sorting function could
> probably annotate the thread tree and re-use that information
> for all of the recursive sorts.  Or use a decorate-sort-undecorate
> pattern. 

It sounds possible, but peeking at the code, it doesn't seem trivial to
do generally, since the user can compose their own composited sorting
functions.  So the cache would have to be per
article+sorting-function...  hm...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> Separate from that low-level cache, the sorting function could
>> probably annotate the thread tree and re-use that information
>> for all of the recursive sorts.  Or use a decorate-sort-undecorate
>> pattern. 
>
> It sounds possible, but peeking at the code, it doesn't seem trivial to
> do generally, since the user can compose their own composited sorting
> functions.  So the cache would have to be per
> article+sorting-function...  hm...

Re: caching:

I think caching mainly makes sense for the functions which require
traversing the thread tree: gnus-thread-sort-by-total-score and
gnus-thread-sort-by-most-recent-number and -date.  Are there
any others?  If these tucked away their result when called,
I imagine there would be a big savings.  Depending on the shape
of the tree, this would reduce sort key computation time from
somewhere between O(n log n) and O(n^2) to O(n).

Re: decorate-sort-undecorate (DSU):

Here for each article you create a key which is a list of the relevant
values.  For example, if the sort functions are

           '((not gnus-thread-sort-by-number)
             gnus-thread-sort-by-score))

then the key would be (-number score) [or maybe the reverse].  Then you
replace the list of threads with a list of pairs (key thread) and sort
on the keys.  This avoids recomputing the key each time an article needs
to be compared with another, which would reduce overhead a lot.  After
sorting, you make a pass through to extract the list of threads in
sorted order.

When computing the keys, if you traverse the threads from leaves to the
root, you can compute the keys quickly without needing to implement the
caching.

But I don't know what to do if the user implements their own sorting
function.  It would be easier if the interface was changed: instead
of providing a list of comparison functions, you provide a list of
functions which compute key values which are then compared using
the default emacs order.  Then we could always use DSU.  But maybe
it's not worth changing the interface for this?

Options:

(a) Try caching, and see if it is good enough.
(b) Use DSU when only built-in sort functions are used, and fall
back to existing behaviour?  (Complicates code to have a fall-back,
and also would have to assume that the user didn't redefine the
built-in functions.)
(c) Change the interface so DSU can always be used?

Dan

Re: Improving Gnus speed

[Francis Moreau]

Dan Christensen <jdc@uwo.ca> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Dan Christensen <jdc@uwo.ca> writes:
>>
>>> Separate from that low-level cache, the sorting function could
>>> probably annotate the thread tree and re-use that information
>>> for all of the recursive sorts.  Or use a decorate-sort-undecorate
>>> pattern. 
>>
>> It sounds possible, but peeking at the code, it doesn't seem trivial to
>> do generally, since the user can compose their own composited sorting
>> functions.  So the cache would have to be per
>> article+sorting-function...  hm...
>
> Re: caching:
>
> I think caching mainly makes sense for the functions which require
> traversing the thread tree: gnus-thread-sort-by-total-score and
> gnus-thread-sort-by-most-recent-number and -date.  Are there
> any others?  If these tucked away their result when called,
> I imagine there would be a big savings.  Depending on the shape
> of the tree, this would reduce sort key computation time from
> somewhere between O(n log n) and O(n^2) to O(n).

Yes but I still think that the wrong approach is currently done because
all this sorting is mostly done for nothing.

When I'm entering in the group, all threads are hidden. And I won't look
at 80% of the actual threads. So sorting them are useless.

I really think that sorting should happen only when threads are
expanded. And that would reduce a _lot_ the number of threads to sort.

-- 
Francis

Re: Improving Gnus speed

[Dan Christensen]

Francis Moreau <francis.moro@gmail.com> writes:

> Yes but I still think that the wrong approach is currently done because
> all this sorting is mostly done for nothing.
>
> When I'm entering in the group, all threads are hidden. And I won't look
> at 80% of the actual threads. So sorting them are useless.
>
> I really think that sorting should happen only when threads are
> expanded. And that would reduce a _lot_ the number of threads to sort.

I may be wrong, but I think that rearranging the articles in the summary
buffer is a little complicated in Gnus.  I don't think there's any
reason that the thread sorting needs to take a significant amount of
time if done correctly.

Moreover, I never hide threads, so I want them sorted correctly from the
start. 

Dan

Re: Improving Gnus speed

[Francis Moreau]

Dan Christensen <jdc@uwo.ca> writes:

> Francis Moreau <francis.moro@gmail.com> writes:
>
>> Yes but I still think that the wrong approach is currently done because
>> all this sorting is mostly done for nothing.
>>
>> When I'm entering in the group, all threads are hidden. And I won't look
>> at 80% of the actual threads. So sorting them are useless.
>>
>> I really think that sorting should happen only when threads are
>> expanded. And that would reduce a _lot_ the number of threads to sort.
>
> I may be wrong, but I think that rearranging the articles in the summary
> buffer is a little complicated in Gnus.  I don't think there's any
> reason that the thread sorting needs to take a significant amount of
> time if done correctly.
>
> Moreover, I never hide threads, so I want them sorted correctly from the
> start. 

Sure but in that case all threads are expanded so also sorted.

I don't want to pay the price to sort them all because I won't read most
of them, and when I don't read a thread I just want it to be hidden
(colapsed).

So it's really: "do the job when it's really needed."

-- 
Francis

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> But I don't know what to do if the user implements their own sorting
> function.  It would be easier if the interface was changed: instead
> of providing a list of comparison functions, you provide a list of
> functions which compute key values which are then compared using
> the default emacs order.  Then we could always use DSU.  But maybe
> it's not worth changing the interface for this?

I think it would be nicer if the interface was unchanged...

> (a) Try caching, and see if it is good enough.
> (b) Use DSU when only built-in sort functions are used, and fall
> back to existing behaviour?  (Complicates code to have a fall-back,
> and also would have to assume that the user didn't redefine the
> built-in functions.)
> (c) Change the interface so DSU can always be used?

... so I think (a) sounds best.  And, yeah, you're right, only the
recursive sorting functions really need caching...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Dan Christensen]

Dan Christensen <jdc@uwo.ca> writes:

> I think caching mainly makes sense for the functions which require
> traversing the thread tree: gnus-thread-sort-by-total-score and
> gnus-thread-sort-by-most-recent-number and -date.  Are there
> any others?  If these tucked away their result when called,
> I imagine there would be a big savings.  Depending on the shape
> of the tree, this would reduce sort key computation time from
> somewhere between O(n log n) and O(n^2) to O(n).

I've done some tests on the threads from a group I have with about 
6500 articles.  It takes about 1.6s to do the sorting step using
gnus-thread-sort-by-most-recent-date, and it turns out that the majority
of the time is taken by converting the dates from the string stored in
the header to an emacs time, i.e. the function gnus-date-get-time (and
what it calls).

Even though that function caches its result, I thought that it might be
worth caching the most-recent-date results too, since they currently
involve traversing all sub-threads of all threads.  But it turns out
that that would barely make any difference, at least for groups of
this size.

Here's how I determined this.  First, I saved the thread tree in a file,
so I could do isolated tests of just the sorting.  

Here's the total time required:

(benchmark-run 1 (gnus-sort-threads threads))
(1.638083 8 0.38992400000000016)  [The first number is the time in seconds.]

Now I get a fresh copy of threads, and do the latest-date computation on
all of the articles, using a maptree function I wrote:

(benchmark-run 1 (maptree (lambda (header) (gnus-date-get-time
					    (mail-header-date header))) 
			  threads))
(1.31202 4 0.2062689999999998)

That's 1.3 of the total 1.6 seconds right there.

If I do that again, without resetting threads, the time goes to near
zero, which shows how effective the caching of the date-conversion
results is:

(0.056382999999999996 0 0.0)

And now if we do the sort after having pre-computed the date-conversion,
it only takes about 0.3s!

(benchmark-run 1 (gnus-sort-threads threads))
(0.299614 1 0.0529029999999997)

The upshot is that about 1.3s is date conversion for each article, the
rest has to do with the latest-date traversal and the actual sorting.

In fact, further tests show that the latest-date sub-thread traversal
barely adds anything.

So there doesn't seem to be much point in adding caching for these
recursively defined sort functions.  To improve the speed here, what
needs to be done is to improve the speed of the date conversion.

Moreover, for me, other stages of preparing the summary buffer take
significantly longer, so it would be worth investigating these.

I should also note that I first did the timings yesterday, and got a
total time of 2.5s (reproducibly).  Then, 24 hours later, in the same
emacs, the total time was almost double that (reproducibly).  So I
started a fresh emacs, and got the times above (1.6s total).  Maybe this
has to do with emacs memory management?  In any case, the effect means
that the time varies by a factor of 3 or more, so tracking this down
would be another good way to speed up Gnus.

Dan

Re: Improving Gnus speed

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> The upshot is that about 1.3s is date conversion for each article, the
> rest has to do with the latest-date traversal and the actual sorting.

Ah, right.  Thanks for benchmarking this.

Perhaps the date parsing stuff should be pushed down into the C layer?
Surely one of the many libraries already included in a standard Emacs
build has a date parser already included...

> I should also note that I first did the timings yesterday, and got a
> total time of 2.5s (reproducibly).  Then, 24 hours later, in the same
> emacs, the total time was almost double that (reproducibly).  So I
> started a fresh emacs, and got the times above (1.6s total).

Yeah, benchmarking in Emacs can be kinda, er, non-reproducible.  I've
never understood why, though.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Dan Christensen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Dan Christensen <jdc@uwo.ca> writes:
>
>> The upshot is that about 1.3s is date conversion for each article, the
>> rest has to do with the latest-date traversal and the actual sorting.
>
> Ah, right.  Thanks for benchmarking this.
>
> Perhaps the date parsing stuff should be pushed down into the C layer?
> Surely one of the many libraries already included in a standard Emacs
> build has a date parser already included...

Could be.  The lisp parser that we fall back onto in the worst case
is very complicated, handling all sorts of weird date formats.  But
it seems to me that it should be possible to quickly parse the most
common formats, and use slower code only when completely necessary.
Could it be that 99% of current e-mail/news has only a small number
of possible date formats?

Another idea for some backends is to rewrite the Date: header as an
X-Gnus-Date:  header when the article arrives.  We take the time to
carefully parse it on arrival, storing the result, and then avoid
re-doing the slow parsing every time we want to show the article.
This could work for backends like nnfolder, nnml, etc, in which Gnus
can easily edit the article.

But this is probably not worth it.  Better to just make date parsing
lightning fast if possible.

Dan

Date parser in C (was: Improving Gnus speed)

[Lars Magne Ingebrigtsen]

Dan Christensen <jdc@uwo.ca> writes:

> But this is probably not worth it.  Better to just make date parsing
> lightning fast if possible.

Yes.  So if somebody could kindly look into which of the gazillion
libraries that Emacs 24 is linked with that has a date parser, and
expose that to the Emacs Lisp layer, that'd be nice.

(You'll have to argue with the emacs-devel people yourself, I'm afraid.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Date parser in C (was: Improving Gnus speed)

[Andreas Schwab]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yes.  So if somebody could kindly look into which of the gazillion
> libraries that Emacs 24 is linked with that has a date parser, and
> expose that to the Emacs Lisp layer, that'd be nice.

parse-datetime exists in gnulib.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Date parser in C

[Lars Magne Ingebrigtsen]

Andreas Schwab <schwab@linux-m68k.org> writes:

>> Yes.  So if somebody could kindly look into which of the gazillion
>> libraries that Emacs 24 is linked with that has a date parser, and
>> expose that to the Emacs Lisp layer, that'd be nice.
>
> parse-datetime exists in gnulib.

So whoever volunteers to update strftime and stuff from gnulib could
include parse-datetime, too.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Improving Gnus speed

[Ted Zlatanov]

On Tue, 09 Nov 2010 13:35:15 +0100 Francis Moreau <francis.moro@gmail.com> wrote: 

FM> Out off topic: Why does Gnus development use a mailing list ? That's
FM> pretty paradoxical, isn't it ?

I read it as a newsgroup on GMane.  But it's not a paradox in any case.
Many corporate firewalls block NNTP, for instance.

Ted

Re: Improving Gnus speed

[Steinar Bang]

>>>>> Ted Zlatanov <tzz@lifelogs.com>:

> I read it as a newsgroup on GMane.  But it's not a paradox in any
> case.  Many corporate firewalls block NNTP, for instance.

Actually I've used the existence of gmane to argue for opening corporate
firewalls for gmane, twice.  My argument was that if we were going to use
open source or free software, such as, news.gmane.org was neccessary as
a source for support.

Re: Improving Gnus speed

[Eden Cardim]

>>>>> "Francis" == Francis Moreau <francis.moro@gmail.com> writes:

Francis> Out off topic: Why does Gnus development use a mailing list ? That's
Francis> pretty paradoxical, isn't it ?

As a beginning gnus user, I wouldn't be able to reply as to why it
happens for gnus in particular, but I thought it would be worth the
mention that this "paradox" actually makes sense in most cases. It's
called bootstrapping, the reasoning behind it is that the building of a
new tool is often motivated by limitations in all the alternative tools,
which implies that the best tool for the development project is the very
thing you're developing. Another benefit is that the development effort
also doubles up as a sort of continuous alpha test. There are quite a
few notable cases:

- most linux kernel development happens on a running linux kernel
- gcc is (in most cases) compiled by gcc
- git development uses git for revision control
- subversion too
- perl's build/test process is controlled by a perl script
- trac development uses trac as SCM
- bugzilla too

just my $0.02

-- 
Eden Cardim
Software Engineer
+55 73 9986-3963
edencardim.com