* Improving Gnus speed @ 2010-11-09 12:35 Francis Moreau 2010-11-09 13:45 ` Didier Verna ` (3 more replies) 0 siblings, 4 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-09 12:35 UTC (permalink / raw) To: ding Hello, I'd like to give some feedbacks about Gnus' speed. First I just wrote this in order to improve Gnus _only_. I use as a newsreader Gnus and Thunderbird, and I must admit that Gnus is much slower specially to fetch header and to display the summary buffer. One example: I'm using article caching to save precious articles. Here's my related config: (setq gnus-use-cache 'passive) (setq gnus-cacheable-groups "^nntp") (setq gnus-keep-backlog 50) Even if articles are cached, Gnus takes some times to display the summary buffer: Entering in the group containing cached articles: 4 secs. This group contains 230 unread articles. Now display all articles in this group by doing '/ o': Gnus asks me: How many articles from nnml+cache:nntp+news.free.fr:comp.lang.c (default 42555): Type <RET>. Fetching headers for nnml+cache:nntp+news.free.fr:comp.lang.c... < 7 secs> done < 22 secs> Generating summary... < 3 secs> done As you can see all these figures are pretty high, the biggest being 22 secs. I'm wondering what's Gnus is executing since it seems between 2 steps. Corresponding group parameters: (display . all) (gnus-use-scoring t) (gnus-thread-sort-functions '((not gnus-thread-sort-by-number) gnus-thread-sort-by-total-score))))) I'm interested in improving this, so any suggestions are welcome. Otherwise I can have a look to the revelant part of the Gnus' code, although I almost don't speak elisp but that would be a reason to start learning this language, if some one can suggest me some functions or some starting points to look at. BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP method only (not git protocol). Is that right ? IIRC, git protocol is more effective, and I actually don't compile Git with HTTP support since it adds some dependency like curl for no good reason. Out off topic: Why does Gnus development use a mailing list ? That's pretty paradoxical, isn't it ? Thanks -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 12:35 Improving Gnus speed Francis Moreau @ 2010-11-09 13:45 ` Didier Verna 2010-11-09 13:55 ` Francis Moreau 2010-11-09 18:55 ` Lars Magne Ingebrigtsen ` (2 subsequent siblings) 3 siblings, 1 reply; 103+ messages in thread From: Didier Verna @ 2010-11-09 13:45 UTC (permalink / raw) To: Francis Moreau; +Cc: ding Francis Moreau <francis.moro@gmail.com> wrote: > Out off topic: Why does Gnus development use a mailing list ? That's > pretty paradoxical, isn't it ? Because if you think that Gnus is a newsreader, you're wrong... -- Resistance is futile. You will be jazzimilated. Scientific site: http://www.lrde.epita.fr/~didier Music (Jazz) site: http://www.didierverna.com ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 13:45 ` Didier Verna @ 2010-11-09 13:55 ` Francis Moreau 2010-11-09 14:00 ` Knut Anders Hatlen ` (2 more replies) 0 siblings, 3 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-09 13:55 UTC (permalink / raw) To: ding Didier Verna <didier@xemacs.org> writes: > Francis Moreau <francis.moro@gmail.com> wrote: > >> Out off topic: Why does Gnus development use a mailing list ? That's >> pretty paradoxical, isn't it ? > > Because if you think that Gnus is a newsreader, you're wrong... :) Is that a joke ? If so I would suggest you to use ':)' next time... Otherwise just read (gnus)Top. Bye. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 13:55 ` Francis Moreau @ 2010-11-09 14:00 ` Knut Anders Hatlen 2010-11-09 14:22 ` Steinar Bang 2010-11-09 14:29 ` Francis Moreau 2010-11-09 14:49 ` Didier Verna 2010-11-09 15:04 ` Adam Sjøgren 2 siblings, 2 replies; 103+ messages in thread From: Knut Anders Hatlen @ 2010-11-09 14:00 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Didier Verna <didier@xemacs.org> writes: > >> Francis Moreau <francis.moro@gmail.com> wrote: >> >>> Out off topic: Why does Gnus development use a mailing list ? That's >>> pretty paradoxical, isn't it ? >> >> Because if you think that Gnus is a newsreader, you're wrong... > > :) > > Is that a joke ? > > If so I would suggest you to use ':)' next time... > > Otherwise just read (gnus)Top. Which says pretty clearly that it's a news *and* mail reader. :) -- Knut Anders ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 14:00 ` Knut Anders Hatlen @ 2010-11-09 14:22 ` Steinar Bang 2010-11-09 14:29 ` Francis Moreau 1 sibling, 0 replies; 103+ messages in thread From: Steinar Bang @ 2010-11-09 14:22 UTC (permalink / raw) To: ding And in any case, I read this thread on a newsgroup on news.gmane.org... ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 14:00 ` Knut Anders Hatlen 2010-11-09 14:22 ` Steinar Bang @ 2010-11-09 14:29 ` Francis Moreau 2010-11-09 14:49 ` Richard Riley 1 sibling, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-09 14:29 UTC (permalink / raw) To: Knut Anders Hatlen; +Cc: ding Knut Anders Hatlen <kahatlen@gmail.com> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Didier Verna <didier@xemacs.org> writes: >> >>> Francis Moreau <francis.moro@gmail.com> wrote: >>> >>>> Out off topic: Why does Gnus development use a mailing list ? That's >>>> pretty paradoxical, isn't it ? >>> >>> Because if you think that Gnus is a newsreader, you're wrong... >> >> :) >> >> Is that a joke ? >> >> If so I would suggest you to use ':)' next time... >> >> Otherwise just read (gnus)Top. > > Which says pretty clearly that it's a news *and* mail reader. :) No it says that it's primarly a newsreader: "(gnus)Mail in a Newsreader" is an example. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 14:29 ` Francis Moreau @ 2010-11-09 14:49 ` Richard Riley 2010-11-09 16:37 ` Didier Verna 0 siblings, 1 reply; 103+ messages in thread From: Richard Riley @ 2010-11-09 14:49 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Knut Anders Hatlen <kahatlen@gmail.com> writes: > >> Francis Moreau <francis.moro@gmail.com> writes: >> >>> Didier Verna <didier@xemacs.org> writes: >>> >>>> Francis Moreau <francis.moro@gmail.com> wrote: >>>> >>>>> Out off topic: Why does Gnus development use a mailing list ? That's >>>>> pretty paradoxical, isn't it ? >>>> >>>> Because if you think that Gnus is a newsreader, you're wrong... >>> >>> :) >>> >>> Is that a joke ? >>> >>> If so I would suggest you to use ':)' next time... >>> >>> Otherwise just read (gnus)Top. >> >> Which says pretty clearly that it's a news *and* mail reader. :) > > No it says that it's primarly a newsreader: "(gnus)Mail in a Newsreader" > is an example. On that note some of the messages could be made clearer. e.g "posting article..." when you send to a usenet group and "sending email..." when using a To address. ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 14:49 ` Richard Riley @ 2010-11-09 16:37 ` Didier Verna 2010-11-09 16:51 ` Richard Riley 0 siblings, 1 reply; 103+ messages in thread From: Didier Verna @ 2010-11-09 16:37 UTC (permalink / raw) To: Richard Riley; +Cc: ding Richard Riley <rileyrg@googlemail.com> wrote: > On that note some of the messages could be made clearer. e.g "posting > article..." when you send to a usenet group and "sending email..." > when using a To address. That's because there is a distinction in your mind, whereas Gnus tries hard to blur the difference between sending mails and posting news. ;-) -- Resistance is futile. You will be jazzimilated. Scientific site: http://www.lrde.epita.fr/~didier Music (Jazz) site: http://www.didierverna.com ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 16:37 ` Didier Verna @ 2010-11-09 16:51 ` Richard Riley 2010-11-09 16:56 ` Steinar Bang 0 siblings, 1 reply; 103+ messages in thread From: Richard Riley @ 2010-11-09 16:51 UTC (permalink / raw) To: Richard Riley; +Cc: ding Didier Verna <didier@xemacs.org> writes: > Richard Riley <rileyrg@googlemail.com> wrote: > >> On that note some of the messages could be made clearer. e.g "posting >> article..." when you send to a usenet group and "sending email..." >> when using a To address. > > That's because there is a distinction in your mind, whereas Gnus tries > hard to blur the difference between sending mails and posting news. ;-) Because there is a huge distinction when sending/posting. I like to know in the combined UI that I am dealing with a group as opposed to an email. ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 16:51 ` Richard Riley @ 2010-11-09 16:56 ` Steinar Bang 2010-11-09 17:12 ` Richard Riley 0 siblings, 1 reply; 103+ messages in thread From: Steinar Bang @ 2010-11-09 16:56 UTC (permalink / raw) To: ding >>>>> Richard Riley <rileyrg@googlemail.com>: > Didier Verna <didier@xemacs.org> writes: >> That's because there is a distinction in your mind, whereas Gnus >> tries hard to blur the difference between sending mails and posting >> news. ;-) > Because there is a huge distinction when sending/posting. There is...? In what way? ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 16:56 ` Steinar Bang @ 2010-11-09 17:12 ` Richard Riley 2010-11-09 17:48 ` Steinar Bang 0 siblings, 1 reply; 103+ messages in thread From: Richard Riley @ 2010-11-09 17:12 UTC (permalink / raw) To: ding Steinar Bang <sb@dod.no> writes: >>>>>> Richard Riley <rileyrg@googlemail.com>: > >> Didier Verna <didier@xemacs.org> writes: > >>> That's because there is a distinction in your mind, whereas Gnus >>> tries hard to blur the difference between sending mails and posting >>> news. ;-) > >> Because there is a huge distinction when sending/posting. > > There is...? In what way? > The intended audiences. Frequently I spot that I am accidentally emailing rather than posting a reply. Obviously I can only speak for myself but one of the first things people do/used to do is turn on the "really post" prompt on when it was an nntp post since its going public. ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 17:12 ` Richard Riley @ 2010-11-09 17:48 ` Steinar Bang 2010-11-09 18:59 ` Adam Sjøgren 2010-11-09 19:06 ` Richard Riley 0 siblings, 2 replies; 103+ messages in thread From: Steinar Bang @ 2010-11-09 17:48 UTC (permalink / raw) To: ding >>>>> Richard Riley <rileyrg@googlemail.com>: > The intended audiences. > Frequently I spot that I am accidentally emailing rather than posting a > reply. Hm... that has never happened to me. The other way around: sending an intended private reply to a mailing list _has_ happened. What I do then is to set (broken-reply-to . t) in the group parameters of the nnimap group represting the mailing list... > Obviously I can only speak for myself but one of the first things > people do/used to do is turn on the "really post" prompt on when it > was an nntp post since its going public. I've sometimes thought that it would have been nice to have a posting delay, where I could go in, say within 5 minutes and kill a posting. But after a while, I figured that the best thing to do was to just live with my malposts. I've got expiry turned on when posting to USENEt, so it means that google groups will stop showing them after a couple of weeks time. ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 17:48 ` Steinar Bang @ 2010-11-09 18:59 ` Adam Sjøgren 2010-11-09 19:17 ` Steinar Bang 2010-11-09 19:06 ` Richard Riley 1 sibling, 1 reply; 103+ messages in thread From: Adam Sjøgren @ 2010-11-09 18:59 UTC (permalink / raw) To: ding On Tue, 09 Nov 2010 18:48:46 +0100, Steinar wrote: > I've got expiry turned on when posting to USENEt, so it means that > google groups will stop showing them after a couple of weeks time. Hopefully you never post anything that could be useful to anyone beyond a couple of weeks time. Best regards, Adam -- "Angels can fly because they take themselves lightly." Adam Sjøgren asjo@koldfront.dk ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 18:59 ` Adam Sjøgren @ 2010-11-09 19:17 ` Steinar Bang 0 siblings, 0 replies; 103+ messages in thread From: Steinar Bang @ 2010-11-09 19:17 UTC (permalink / raw) To: ding >>>>> asjo@koldfront.dk (Adam Sjøgren): > Hopefully you never post anything that could be useful to anyone > beyond a couple of weeks time. What I still post on USENET can safely be counted into that category. (I don't count gmane as USENET in this context, and I don't expire on gmane) ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 17:48 ` Steinar Bang 2010-11-09 18:59 ` Adam Sjøgren @ 2010-11-09 19:06 ` Richard Riley 1 sibling, 0 replies; 103+ messages in thread From: Richard Riley @ 2010-11-09 19:06 UTC (permalink / raw) To: ding Steinar Bang <sb@dod.no> writes: >>>>>> Richard Riley <rileyrg@googlemail.com>: > >> The intended audiences. > >> Frequently I spot that I am accidentally emailing rather than posting a >> reply. > > Hm... that has never happened to me. The other way around: sending an > intended private reply to a mailing list _has_ happened. > > What I do then is to set > (broken-reply-to . t) > in the group parameters of the nnimap group represting the mailing > list... I need to look that up. > >> Obviously I can only speak for myself but one of the first things >> people do/used to do is turn on the "really post" prompt on when it >> was an nntp post since its going public. > > I've sometimes thought that it would have been nice to have a posting > delay, where I could go in, say within 5 minutes and kill a posting. > > But after a while, I figured that the best thing to do was to just live > with my malposts. I've got expiry turned on when posting to USENEt, so > it means that google groups will stop showing them after a couple of > weeks time. > Why would you do that? If you post to usenet then I think it should stay for google completeness in threads. But for me the bottom line is that both mentally and physically usenet and mail are two different entities and while I like the common features Gnus provides I still want that difference to be apparent to me. There are a few rough edges in Gnus where "sending" and "posting" are used. They mean different things and I like that they do. ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 13:55 ` Francis Moreau 2010-11-09 14:00 ` Knut Anders Hatlen @ 2010-11-09 14:49 ` Didier Verna 2010-11-09 15:27 ` Richard Riley 2010-11-09 15:04 ` Adam Sjøgren 2 siblings, 1 reply; 103+ messages in thread From: Didier Verna @ 2010-11-09 14:49 UTC (permalink / raw) To: Francis Moreau; +Cc: ding Francis Moreau <francis.moro@gmail.com> wrote: > Is that a joke ? Nope. > If so I would suggest you to use ':)' next time... > > Otherwise just read (gnus)Top. Hmmm, here we are, lecturing 20 years old-gnus-timers about reading the Gnus Manual. Nicely done :-) Hint: read "Getting News", then "Getting Mail", then compare their respective sizes, then come back and try again :-) -- Resistance is futile. You will be jazzimilated. Scientific site: http://www.lrde.epita.fr/~didier Music (Jazz) site: http://www.didierverna.com ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 14:49 ` Didier Verna @ 2010-11-09 15:27 ` Richard Riley 2010-11-09 15:42 ` Francis Moreau 2010-11-09 16:35 ` Didier Verna 0 siblings, 2 replies; 103+ messages in thread From: Richard Riley @ 2010-11-09 15:27 UTC (permalink / raw) To: ding Didier Verna <didier@xemacs.org> writes: > Francis Moreau <francis.moro@gmail.com> wrote: > >> Is that a joke ? > > Nope. > >> If so I would suggest you to use ':)' next time... >> >> Otherwise just read (gnus)Top. > > Hmmm, here we are, lecturing 20 years old-gnus-timers about reading the > Gnus Manual. Nicely done :-) Did you? ;) > > Hint: read "Getting News", then "Getting Mail", then compare their > respective sizes, then come back and try again :-) And that means what? Its just indicative of the complexity of the product and the rather famous Gnus Manual in particular. Its not for nothing #emacs and other resources are constantly full of new adopters pleading for help in getting their email working with Gnus. Nothing to do whatsoever with the relative functionality. It is not for the feint of heart. Gnus can most certainly be thought of as primarily a newsreader in my mind and most certainly your claim that its NOT a news reader must be a joke. It is isn't it?? regards r. ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 15:27 ` Richard Riley @ 2010-11-09 15:42 ` Francis Moreau 2010-11-09 16:35 ` Didier Verna 1 sibling, 0 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-09 15:42 UTC (permalink / raw) To: Richard Riley; +Cc: ding Richard Riley <rileyrg@googlemail.com> writes: > Didier Verna <didier@xemacs.org> writes: > >> Francis Moreau <francis.moro@gmail.com> wrote: >> >>> Is that a joke ? >> >> Nope. >> >>> If so I would suggest you to use ':)' next time... >>> >>> Otherwise just read (gnus)Top. >> >> Hmmm, here we are, lecturing 20 years old-gnus-timers about reading the >> Gnus Manual. Nicely done :-) > > Did you? ;) > Well, probably he still hasn't understood it after 20 years ;) (Please note the smiley). >> Hint: read "Getting News", then "Getting Mail", then compare their >> respective sizes, then come back and try again :-) > > And that means what? Its just indicative of the complexity of the > product and the rather famous Gnus Manual in particular. Its not for > nothing #emacs and other resources are constantly full of new adopters > pleading for help in getting their email working with Gnus. Nothing to > do whatsoever with the relative functionality. It is not for the feint > of heart. Thanks Richard for responding. I was going to respond something similar (well except the part related to #emacs) but I wouldn't have done better. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 15:27 ` Richard Riley 2010-11-09 15:42 ` Francis Moreau @ 2010-11-09 16:35 ` Didier Verna 2010-11-09 16:49 ` Richard Riley 2010-11-09 19:52 ` Francis Moreau 1 sibling, 2 replies; 103+ messages in thread From: Didier Verna @ 2010-11-09 16:35 UTC (permalink / raw) To: Richard Riley; +Cc: ding Richard Riley <rileyrg@googlemail.com> wrote: > And that means what? Its just indicative of the complexity of the > product and the rather famous Gnus Manual in particular. No, it is indicative of the fact that its needs more energy to implement mail reading in Gnus than only news reading (and also that it is more complex to do it). And since this work has been done, as the documentation proves), it is indicative of the fact that reading mail in Gnus is in fact at least as important as reading news (since people are willing to spend so much time and energy on it). > Its not for nothing #emacs and other resources are constantly full of > new adopters pleading for help in getting their email working with > Gnus. Err, exactly my point, thanks for supporting me. New adopters indeed want to get help reading mail with Gnus. This means that new adopters want to read mail with Gnus. So Gnus is a mail reader (at least as much as it is a news reader). > Gnus can most certainly be thought of as primarily a newsreader in my > mind Not in mine. It is primarily a mail reader for me (but of course you're entitled to think otherwise). The other poster was also right in saying that Gnus is a way of life. More precisely, the way of life in question is to handle mail in a news-like way (auto-expiry etc). Just for the anecdote, I almost got to writing a Gnus book for O'Reilly France some years ago. The editor contacted me about it and while we were discussing the contents and organization, he said that in order to reach the broadest possible audience, the quickstart had to show how to use nnimap as the primary select method... In fact, I would really like if we had serious statistics about Gnus usage, and I wouldn't be surprised if people read more mails than news with it. > and most certainly your claim that its NOT a news reader must be a > joke. It is isn't it?? No, it's not a joke. It's a news/mail reader and you've got to acknowledge the fact that reading mail with it is at least as important as reading news with it. Guys, remember the original question ? The OP found it paradoxical that Gnus development discussion happens on a mailing list. I maintain that if you get that sort of feeling, it's because your view of what Gnus really is wrong. -- Resistance is futile. You will be jazzimilated. Scientific site: http://www.lrde.epita.fr/~didier Music (Jazz) site: http://www.didierverna.com ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 16:35 ` Didier Verna @ 2010-11-09 16:49 ` Richard Riley 2010-11-09 19:52 ` Francis Moreau 1 sibling, 0 replies; 103+ messages in thread From: Richard Riley @ 2010-11-09 16:49 UTC (permalink / raw) To: ding Didier Verna <didier@xemacs.org> writes: > Richard Riley <rileyrg@googlemail.com> wrote: > >> And that means what? Its just indicative of the complexity of the >> product and the rather famous Gnus Manual in particular. > > No, it is indicative of the fact that its needs more energy to > implement mail reading in Gnus than only news reading (and also that it > is more complex to do it). And since this work has been done, as the > documentation proves), it is indicative of the fact that reading mail in > Gnus is in fact at least as important as reading news (since people are > willing to spend so much time and energy on it). Gnus is still a news reader. I assumed you were joking too. Apparently not. The entire look and feel is that of a newsreader. The complexity of the mail part is as a result of the newsreader approach - something I like. Many of course dont preferring more traditional mail clients. ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 16:35 ` Didier Verna 2010-11-09 16:49 ` Richard Riley @ 2010-11-09 19:52 ` Francis Moreau 2010-11-09 20:48 ` Richard Riley 1 sibling, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-09 19:52 UTC (permalink / raw) To: Richard Riley; +Cc: ding Didier Verna <didier@xemacs.org> writes: [...] > > In fact, I would really like if we had serious statistics about Gnus > usage, and I wouldn't be surprised if people read more mails than news > with it. > I think you're still wrong. You would be surprised to see how much people use Gnus for news and a seperate mail reader (like VM, Mew, trn4...) for mails. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 19:52 ` Francis Moreau @ 2010-11-09 20:48 ` Richard Riley 0 siblings, 0 replies; 103+ messages in thread From: Richard Riley @ 2010-11-09 20:48 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Didier Verna <didier@xemacs.org> writes: > > [...] > >> >> In fact, I would really like if we had serious statistics about Gnus >> usage, and I wouldn't be surprised if people read more mails than news >> with it. >> > > I think you're still wrong. You would be surprised to see how much > people use Gnus for news and a seperate mail reader (like VM, Mew, > trn4...) for mails. That's certainly my experience from talking to people in #emacs irc channel. Most people there think Gnus is an unwieldy and slow monster for mail and prefer to use Mutt or something similar - Despite adverts to the contrary most still hang on to the belief that Gnus is useless for IMAP too ;) ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 13:55 ` Francis Moreau 2010-11-09 14:00 ` Knut Anders Hatlen 2010-11-09 14:49 ` Didier Verna @ 2010-11-09 15:04 ` Adam Sjøgren 2 siblings, 0 replies; 103+ messages in thread From: Adam Sjøgren @ 2010-11-09 15:04 UTC (permalink / raw) To: ding On Tue, 09 Nov 2010 14:55:15 +0100, Francis wrote: > Didier Verna <didier@xemacs.org> writes: [...] >> Because if you think that Gnus is a newsreader, you're wrong... > :) > Is that a joke ? No Gnus is a way of life. (Ok, maybe that is a bit much, but Gnus is much more than a newsreader :-) <-- ^ not a joke, but a smile nevertheless), Adam -- "Angels can fly because they take themselves lightly." Adam Sjøgren asjo@koldfront.dk ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 12:35 Improving Gnus speed Francis Moreau 2010-11-09 13:45 ` Didier Verna @ 2010-11-09 18:55 ` Lars Magne Ingebrigtsen 2010-11-09 19:58 ` Francis Moreau ` (2 more replies) 2010-11-09 21:47 ` Improving Gnus speed Ted Zlatanov 2010-11-10 4:27 ` Eden Cardim 3 siblings, 3 replies; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-09 18:55 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Even if articles are cached, Gnus takes some times to display the > summary buffer: > > Entering in the group containing cached articles: 4 secs. > > This group contains 230 unread articles. That's obscenely slow. Even if the 230 articles were fetched over the net, it shouldn't take anywhere that long. > Now display all articles in this group by doing '/ o': > > Gnus asks me: > > How many articles from nnml+cache:nntp+news.free.fr:comp.lang.c (default 42555): > > Type <RET>. > > Fetching headers for nnml+cache:nntp+news.free.fr:comp.lang.c... > < 7 secs> > done > < 22 secs> > Generating summary... > < 3 secs> > done That seems rather slow, too, but Gnus is rather slow when rendering large summary buffers, and I've never understood why, really. Entering a 10K group takes a couple of seconds, but entering a 100K group takes several minutes. So there's something exponential going on somewhere... > I'm interested in improving this, so any suggestions are welcome. (setq debug-on-quit t) and then `C-g'-ing a few times should tell you what function is taking so long. `M-x elp-instrument-package RET gnus RET', doing something, and then `M-x elp-results' should give you a detailed look. > BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP > method only (not git protocol). Is that right ? > > IIRC, git protocol is more effective, and I actually don't compile Git > with HTTP support since it adds some dependency like curl for no good > reason. The Gnus sources are so small that just using HTTP seems OK to me, speedwise. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 18:55 ` Lars Magne Ingebrigtsen @ 2010-11-09 19:58 ` Francis Moreau 2010-11-09 21:03 ` Andreas Schwab 2010-11-09 21:27 ` Lars Magne Ingebrigtsen 2010-11-10 14:16 ` Francis Moreau 2 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-09 19:58 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Even if articles are cached, Gnus takes some times to display the >> summary buffer: >> >> Entering in the group containing cached articles: 4 secs. >> >> This group contains 230 unread articles. > > That's obscenely slow. Even if the 230 articles were fetched over the > net, it shouldn't take anywhere that long. That's reassuring. >> Now display all articles in this group by doing '/ o': >> >> Gnus asks me: >> >> How many articles from nnml+cache:nntp+news.free.fr:comp.lang.c (default 42555): >> >> Type <RET>. >> >> Fetching headers for nnml+cache:nntp+news.free.fr:comp.lang.c... >> < 7 secs> >> done >> < 22 secs> >> Generating summary... >> < 3 secs> >> done > > That seems rather slow, too, but Gnus is rather slow when rendering I forgot to mention that the number 42555 is wrong, there're actually only 1298 articles in this buffer. I don't know why it's wrong though. > large summary buffers, and I've never understood why, really. Entering > a 10K group takes a couple of seconds, but entering a 100K group takes > several minutes. So there's something exponential going on somewhere... That's my impression too. >> I'm interested in improving this, so any suggestions are welcome. > > (setq debug-on-quit t) and then `C-g'-ing a few times should tell you > what function is taking so long. `M-x elp-instrument-package RET gnus > RET', doing something, and then `M-x elp-results' should give you a > detailed look. ok, I'll try that and see I can find something. >> BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP >> method only (not git protocol). Is that right ? >> >> IIRC, git protocol is more effective, and I actually don't compile Git >> with HTTP support since it adds some dependency like curl for no good >> reason. > > The Gnus sources are so small that just using HTTP seems OK to me, > speedwise. Well even in that case, why not using the native protocol which is more efficient, stable ... ? -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 19:58 ` Francis Moreau @ 2010-11-09 21:03 ` Andreas Schwab 2010-11-09 21:49 ` Ted Zlatanov 2010-11-10 13:38 ` Francis Moreau 0 siblings, 2 replies; 103+ messages in thread From: Andreas Schwab @ 2010-11-09 21:03 UTC (permalink / raw) To: Francis Moreau; +Cc: ding Francis Moreau <francis.moro@gmail.com> writes: > Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > >> Francis Moreau <francis.moro@gmail.com> writes: >> >>> BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP >>> method only (not git protocol). Is that right ? >>> >>> IIRC, git protocol is more effective, and I actually don't compile Git >>> with HTTP support since it adds some dependency like curl for no good >>> reason. >> >> The Gnus sources are so small that just using HTTP seems OK to me, >> speedwise. > > Well even in that case, why not using the native protocol which is more > efficient, stable ... ? It _is_ using the native protocol, over HTTP. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 21:03 ` Andreas Schwab @ 2010-11-09 21:49 ` Ted Zlatanov 2010-11-09 22:45 ` Richard Riley 2010-11-10 13:38 ` Francis Moreau 1 sibling, 1 reply; 103+ messages in thread From: Ted Zlatanov @ 2010-11-09 21:49 UTC (permalink / raw) To: ding On Tue, 09 Nov 2010 22:03:35 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: AS> Francis Moreau <francis.moro@gmail.com> writes: >> Lars Magne Ingebrigtsen <larsi@gnus.org> writes: >>> The Gnus sources are so small that just using HTTP seems OK to me, >>> speedwise. >> >> Well even in that case, why not using the native protocol which is more >> efficient, stable ... ? AS> It _is_ using the native protocol, over HTTP. Yeah. I set it up to use the smart server or something :) Also, many corporate firewalls including mine don't allow the git protocol. So HTTP is a better transport than native in order to serve the most people. Ted ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 21:49 ` Ted Zlatanov @ 2010-11-09 22:45 ` Richard Riley 2010-11-10 8:49 ` Francis Moreau 2010-11-10 9:39 ` Julien Danjou 0 siblings, 2 replies; 103+ messages in thread From: Richard Riley @ 2010-11-09 22:45 UTC (permalink / raw) To: ding Ted Zlatanov <tzz@lifelogs.com> writes: > On Tue, 09 Nov 2010 22:03:35 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: > > AS> Francis Moreau <francis.moro@gmail.com> writes: >>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes: >>>> The Gnus sources are so small that just using HTTP seems OK to me, >>>> speedwise. >>> >>> Well even in that case, why not using the native protocol which is more >>> efficient, stable ... ? > > AS> It _is_ using the native protocol, over HTTP. > > Yeah. I set it up to use the smart server or something :) > > Also, many corporate firewalls including mine don't allow the git > protocol. So HTTP is a better transport than native in order to serve > the most people. > > Ted > Well, most of those that are limited by a firewall yes. Most people use the git protocol. Which begs the question : can both not be enabled? The transport protocol shouldn't affect the repo in any shape nor form should it? ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 22:45 ` Richard Riley @ 2010-11-10 8:49 ` Francis Moreau 2010-11-10 10:31 ` Francis Moreau 2010-11-10 9:39 ` Julien Danjou 1 sibling, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-10 8:49 UTC (permalink / raw) To: Richard Riley; +Cc: ding Richard Riley <rileyrg@googlemail.com> writes: > Ted Zlatanov <tzz@lifelogs.com> writes: > >> On Tue, 09 Nov 2010 22:03:35 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: >> >> AS> Francis Moreau <francis.moro@gmail.com> writes: >>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes: >>>>> The Gnus sources are so small that just using HTTP seems OK to me, >>>>> speedwise. >>>> >>>> Well even in that case, why not using the native protocol which is more >>>> efficient, stable ... ? >> >> AS> It _is_ using the native protocol, over HTTP. >> >> Yeah. I set it up to use the smart server or something :) >> >> Also, many corporate firewalls including mine don't allow the git >> protocol. So HTTP is a better transport than native in order to serve >> the most people. >> > > Well, most of those that are limited by a firewall yes. Most people use > the git protocol. FWIW, Gnus is the first open source project that is using HTTP only. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 8:49 ` Francis Moreau @ 2010-11-10 10:31 ` Francis Moreau 0 siblings, 0 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-10 10:31 UTC (permalink / raw) To: Richard Riley; +Cc: ding Francis Moreau <francis.moro@gmail.com> writes: > Richard Riley <rileyrg@googlemail.com> writes: > >> Ted Zlatanov <tzz@lifelogs.com> writes: >> >>> On Tue, 09 Nov 2010 22:03:35 +0100 Andreas Schwab <schwab@linux-m68k.org> wrote: >>> >>> AS> Francis Moreau <francis.moro@gmail.com> writes: >>>>> Lars Magne Ingebrigtsen <larsi@gnus.org> writes: >>>>>> The Gnus sources are so small that just using HTTP seems OK to me, >>>>>> speedwise. >>>>> >>>>> Well even in that case, why not using the native protocol which is more >>>>> efficient, stable ... ? >>> >>> AS> It _is_ using the native protocol, over HTTP. >>> >>> Yeah. I set it up to use the smart server or something :) >>> >>> Also, many corporate firewalls including mine don't allow the git >>> protocol. So HTTP is a better transport than native in order to serve >>> the most people. >>> >> >> Well, most of those that are limited by a firewall yes. Most people use >> the git protocol. > > FWIW, Gnus is the first open source project that is using HTTP only. > I mean the fisrt open source project I encountered. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 22:45 ` Richard Riley 2010-11-10 8:49 ` Francis Moreau @ 2010-11-10 9:39 ` Julien Danjou 2010-11-10 13:26 ` Ted Zlatanov 1 sibling, 1 reply; 103+ messages in thread From: Julien Danjou @ 2010-11-10 9:39 UTC (permalink / raw) To: Richard Riley; +Cc: ding On Tue, Nov 09 2010, Richard Riley wrote: > Well, most of those that are limited by a firewall yes. Most people use > the git protocol. Which begs the question : can both not be enabled? The > transport protocol shouldn't affect the repo in any shape nor form > should it? I does not affect anything, it's just a sysadmin problem to install a git-server. -- Julien Danjou // ᐰ <julien@danjou.info> http://julien.danjou.info ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 9:39 ` Julien Danjou @ 2010-11-10 13:26 ` Ted Zlatanov 2010-11-10 13:28 ` Julien Danjou 2010-11-10 14:12 ` Richard Riley 0 siblings, 2 replies; 103+ messages in thread From: Ted Zlatanov @ 2010-11-10 13:26 UTC (permalink / raw) To: ding On Wed, 10 Nov 2010 10:39:00 +0100 Julien Danjou <julien@danjou.info> wrote: JD> On Tue, Nov 09 2010, Richard Riley wrote: >> Well, most of those that are limited by a firewall yes. Most people use >> the git protocol. Which begs the question : can both not be enabled? The >> transport protocol shouldn't affect the repo in any shape nor form >> should it? JD> I does not affect anything, it's just a sysadmin problem to install a JD> git-server. You're assuming there's a need for this. Is there anyone that can't get work done using the HTTP/HTTPS transport? Ted ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 13:26 ` Ted Zlatanov @ 2010-11-10 13:28 ` Julien Danjou 2010-11-10 14:12 ` Richard Riley 1 sibling, 0 replies; 103+ messages in thread From: Julien Danjou @ 2010-11-10 13:28 UTC (permalink / raw) To: Ted Zlatanov; +Cc: ding On Wed, Nov 10 2010, Ted Zlatanov wrote: > You're assuming there's a need for this. Is there anyone that can't get > work done using the HTTP/HTTPS transport? I don't assume anything, I'm just technically answering a technical question. :) (My personal opinion is that HTTP is enough and that it's up to the gnus.org sysadmin to decide if they want to set up the service or not.) -- Julien Danjou // ᐰ <julien@danjou.info> http://julien.danjou.info ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 13:26 ` Ted Zlatanov 2010-11-10 13:28 ` Julien Danjou @ 2010-11-10 14:12 ` Richard Riley 2010-11-10 17:40 ` Andreas Schwab 1 sibling, 1 reply; 103+ messages in thread From: Richard Riley @ 2010-11-10 14:12 UTC (permalink / raw) To: ding Ted Zlatanov <tzz@lifelogs.com> writes: > On Wed, 10 Nov 2010 10:39:00 +0100 Julien Danjou <julien@danjou.info> wrote: > > JD> On Tue, Nov 09 2010, Richard Riley wrote: >>> Well, most of those that are limited by a firewall yes. Most people use >>> the git protocol. Which begs the question : can both not be enabled? The >>> transport protocol shouldn't affect the repo in any shape nor form >>> should it? > > JD> I does not affect anything, it's just a sysadmin problem to install a > JD> git-server. > > You're assuming there's a need for this. Is there anyone that can't get > work done using the HTTP/HTTPS transport? > The git protocol is so much more efficient. And with open source can be provided free by places like github. Why use a slow and inefficient one when both can be used? Seems a strange decision. OK, not as bad as bzr for emacs but ... ;) ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 14:12 ` Richard Riley @ 2010-11-10 17:40 ` Andreas Schwab 0 siblings, 0 replies; 103+ messages in thread From: Andreas Schwab @ 2010-11-10 17:40 UTC (permalink / raw) To: Richard Riley; +Cc: ding Richard Riley <rileyrg@googlemail.com> writes: > The git protocol is so much more efficient. It _is_ using the git protocol. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 21:03 ` Andreas Schwab 2010-11-09 21:49 ` Ted Zlatanov @ 2010-11-10 13:38 ` Francis Moreau 1 sibling, 0 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-10 13:38 UTC (permalink / raw) To: Andreas Schwab; +Cc: ding Andreas Schwab <schwab@linux-m68k.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Lars Magne Ingebrigtsen <larsi@gnus.org> writes: >> >>> Francis Moreau <francis.moro@gmail.com> writes: >>> >>>> BTW, Gnus is using Git as SCM (cool) but it seems that it uses HTTP >>>> method only (not git protocol). Is that right ? >>>> >>>> IIRC, git protocol is more effective, and I actually don't compile Git >>>> with HTTP support since it adds some dependency like curl for no good >>>> reason. >>> >>> The Gnus sources are so small that just using HTTP seems OK to me, >>> speedwise. >> >> Well even in that case, why not using the native protocol which is more >> efficient, stable ... ? > > It _is_ using the native protocol, over HTTP. Ah that's true, since git version 1.6.6, git has Smart HTTP Transport. Hopefully this is the Smart protocol which is currently used. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 18:55 ` Lars Magne Ingebrigtsen 2010-11-09 19:58 ` Francis Moreau @ 2010-11-09 21:27 ` Lars Magne Ingebrigtsen 2010-11-10 14:16 ` Francis Moreau 2 siblings, 0 replies; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-09 21:27 UTC (permalink / raw) To: ding I've done some sloppy benchmarking with this: (benchmark-elapse 1 (gnus-summary-limit-to-unread)) With varying numbers of articles: 10k 2.3s 20k 5.6s 30k 9.8s 40k 15.8s 50k 20.5s So it's not linear, but it's not terribly exponential, either. But I think I know what's probably causing the non-linearity -- it's probably the article marks stuff. It `memq's over these lists, and the longer they are, the longer it takes. `M P b'-ing the 50k buffer has been running for a few minutes now, and that's mostly all list manipulation... I was going to rewrite all the marks stuff to use ranges directly, and that should help a great deal here. I'll get to it one of these evenings. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 18:55 ` Lars Magne Ingebrigtsen 2010-11-09 19:58 ` Francis Moreau 2010-11-09 21:27 ` Lars Magne Ingebrigtsen @ 2010-11-10 14:16 ` Francis Moreau 2010-11-10 18:20 ` Lars Magne Ingebrigtsen ` (2 more replies) 2 siblings, 3 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-10 14:16 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Even if articles are cached, Gnus takes some times to display the >> summary buffer: >> >> Entering in the group containing cached articles: 4 secs. >> >> This group contains 230 unread articles. > > That's obscenely slow. Even if the 230 articles were fetched over the > net, it shouldn't take anywhere that long. > >> Now display all articles in this group by doing '/ o': >> >> Gnus asks me: >> >> How many articles from nnml+cache:nntp+news.free.fr:comp.lang.c (default 42555): >> >> Type <RET>. >> >> Fetching headers for nnml+cache:nntp+news.free.fr:comp.lang.c... >> < 7 secs> >> done >> < 22 secs> >> Generating summary... >> < 3 secs> >> done > > That seems rather slow, too, but Gnus is rather slow when rendering > large summary buffers, and I've never understood why, really. Entering > a 10K group takes a couple of seconds, but entering a 100K group takes > several minutes. So there's something exponential going on somewhere... > >> I'm interested in improving this, so any suggestions are welcome. > > (setq debug-on-quit t) and then `C-g'-ing a few times should tell you > what function is taking so long. Here's my first experiments: While in the fetching headers... process I got the following backtrace: string-match("\\(<[^<]+>\\)[ ]*\\'" "... gnus-get-newsgroup-headers-xover gnus-fetch-headers gnus-summary-insert-articles gnus-summary-insert-old-articles and while in the 22 secs processing, hitting C-g gives: parse-time-tokenize parse-time-string mail-header-parse-date gnus-thread-latest-date gnus-thread-sort-by-most-recent-date gnus-sort-threads-recursive . 33 calls to gnus-sort-threads-recursive . gnus-sort-threads gnus-summary-prepare gnus-summary-limit gnus-summary-insert-old-articles > `M-x elp-instrument-package RET gnus RET', doing something, and then > `M-x elp-results' should give you a detailed look. This fails with the following message: "Variable binding depth exceeds max-specpdl-size" However doing 'M-x elp-results' still shows me: gnus-sort-threads-recursive 641 65.774695999 0.1026126302 gnus-thread-sort-by-most-recent-date 4348 31.652889000 0.0072798732 gnus-thread-latest-date 8696 31.436723000 0.0036150785 gnus-thread-total-score 82494 17.555852999 0.0002128136 gnus-thread-total-score-1 82494 17.134565999 0.0002077068 gnus-summary-insert-articles 1 8.04944 8.04944 gnus-fetch-headers 1 8.019706 8.019706 gnus-get-newsgroup-headers-xover 1 8.005099 8.005099 gnus-thread-sort-by-total-score 8697 3.4791829999 0.0004000440 gnus-id-to-thread 82494 1.3472719999 1.633...e-05 gnus-float-time 41229 0.2444190000 5.928...e-06 gnus-retrieve-headers 2 0.0244269999 0.0122134999 gnus-make-threads 1 0.023105 0.023105 gnus-sorted-nunion 1 0.015179 0.015179 gnus-cache-retrieve-headers 1 0.012292 0.012292 [...] -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 14:16 ` Francis Moreau @ 2010-11-10 18:20 ` Lars Magne Ingebrigtsen 2010-11-11 8:14 ` Francis Moreau 2010-11-12 11:11 ` Francis Moreau 2010-11-10 18:21 ` Lars Magne Ingebrigtsen 2010-11-12 18:55 ` Dan Christensen 2 siblings, 2 replies; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-10 18:20 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > gnus-thread-total-score-1 82494 17.134565999 0.00020770 Geez. How many articles was this? -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 18:20 ` Lars Magne Ingebrigtsen @ 2010-11-11 8:14 ` Francis Moreau 2010-11-11 18:34 ` Lars Magne Ingebrigtsen 2010-11-12 11:11 ` Francis Moreau 1 sibling, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-11 8:14 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> gnus-thread-total-score-1 82494 17.134565999 0.00020770 > > Geez. How many articles was this? As I said somewhere else I have 1298 articles although Gnus mentions 42555 when prompting how many articles it should fetch, I don't know why there's such a difference. Thanks -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-11 8:14 ` Francis Moreau @ 2010-11-11 18:34 ` Lars Magne Ingebrigtsen 2010-11-11 19:54 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-11 18:34 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > As I said somewhere else I have 1298 articles although Gnus mentions > 42555 when prompting how many articles it should fetch, I don't know why > there's such a difference. The prompt uses an estimate based on the LOW . HIGH marks in the group. If you `/ w' after entering, you still only get 1298 articles? -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-11 18:34 ` Lars Magne Ingebrigtsen @ 2010-11-11 19:54 ` Francis Moreau 2010-11-14 16:10 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-11 19:54 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> As I said somewhere else I have 1298 articles although Gnus mentions >> 42555 when prompting how many articles it should fetch, I don't know why >> there's such a difference. > > The prompt uses an estimate based on the LOW . HIGH marks in the group. > > If you `/ w' after entering, you still only get 1298 articles? Nope, I get only 253 articles (same number after entering the group), it shows me only threads which have unread articles I think. If I do '/ o', then it shows me 1298 articles but I have to wait 25 secs. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-11 19:54 ` Francis Moreau @ 2010-11-14 16:10 ` Lars Magne Ingebrigtsen 2010-11-15 9:40 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-14 16:10 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > If I do '/ o', then it shows me 1298 articles but I have to wait 25 > secs. If you do an `M-x elp-instrument-package RET gnus RET', `/ o', and then `M-x elp-results', that should tell us what it's doing, I think. `M-x elp-results' clears out all the counters and stuff, so it's important that you don't do more than the single command between instrumenting and showing the result, or between each result set. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-14 16:10 ` Lars Magne Ingebrigtsen @ 2010-11-15 9:40 ` Francis Moreau 2010-11-21 6:02 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-15 9:40 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> If I do '/ o', then it shows me 1298 articles but I have to wait 25 >> secs. > > If you do an `M-x elp-instrument-package RET gnus RET', `/ o', and then > `M-x elp-results', that should tell us what it's doing, I think. > > `M-x elp-results' clears out all the counters and stuff, so it's > important that you don't do more than the single command between > instrumenting and showing the result, or between each result set. I think it's what I did. Ok just to be sure, we're talking about the same thing, I'm going to describe what currently is my config and the associated elp results. Here's my config for my group caches: ("^nnml\\+cache" (display . all) (gnus-use-scoring nil) (gnus-thread-sort-functions '((not gnus-thread-sort-by-number)))) So I disabled entirely the scoring stuff since it takes a lot of time and this is not really important in these kind of groups since they store only important things (with equal scores). Therefore, when I'm entering such group, most of the time is now spent in the fetching process (not the sorting one). For example, I have a group which contains 1298 articles. Entering it lasts 9 seconds: - 7 secs in the fetching process - 2 secs in sorting + summary buffer generation As you can see, the fetching process is now the annoying step, specially since all articles for this group are cached so they're on my disk. I'm giving you the elp results when entering in the group containing 1298 articles, with all articles displayed: gnus-thread-total-score 21809 18.607898999 0.0008532211 gnus-thread-total-score-1 21805 18.246756999 0.0008368152 gnus-topic-select-group 1 10.660905 10.660905 gnus-group-select-group 1 10.66085 10.66085 gnus-group-read-group 1 10.66084 10.66084 gnus-summary-read-group 1 10.660791 10.660791 gnus-summary-read-group-1 1 10.660778 10.660778 gnus-select-newsgroup 1 7.674363 7.674363 gnus-fetch-headers 1 6.927848 6.927848 gnus-get-newsgroup-headers-xover 1 6.78624 6.78624 gnus-summary-number-of-articles-in-thread 22361 3.2177489999 0.0001439000 gnus-summary-prepare 1 2.901636 2.901636 gnus-summary-prepare-threads 1 2.881676 2.881676 gnus-dd-mmm 1301 0.9418810000 0.0007239669 gnus-articles-to-read 1 0.586238 0.586238 gnus-sorted-difference 3 0.268062 0.089354 gnus-summary-limit-children 1297 0.2439169999 0.0001880624 gnus-id-to-thread 21805 0.1913100000 8.773...e-06 gnus-retrieve-headers 2 0.153315 0.0766575 gnus-cache-retrieve-headers 1 0.138898 0.138898 gnus-sort-threads-recursive 820 0.1360629999 0.0001659304 gnus-run-hooks 1310 0.0821950000 6.274...e-05 gnus-summary-highlight-line 1301 0.0734670000 5.646...e-05 gnus-add-text-properties 10416 0.0530739999 5.095...e-06 gnus-summary-maybe-hide-threads 1 0.04941 0.04941 gnus-summary-hide-all-threads 1 0.0494 0.0494 gnus-put-text-property-excluding-characters-with-faces 1303 0.0478549999 3.672...e-05 gnus-summary-next-thread 44 0.047207 0.0010728863 gnus-summary-go-to-next-thread 44 0.046881 0.0010654772 gnus-put-text-property 6540 0.0405520000 6.200...e-06 gnus-map-function 1325 0.0323199999 2.439...e-05 gnus-summary-from-or-to-or-newsgroups 1301 0.0305190000 2.345...e-05 gnus-summary-hide-thread 22 0.025443 0.0011565 gnus-extract-address-components 1301 0.0206770000 1.589...e-05 gnus-summary-initial-limit 1 0.015415 0.015415 gnus-killed-articles 1 0.013221 0.013221 gnus-simplify-whitespace 1325 0.0102530000 7.738...e-06 gnus-make-threads 1 0.009472 0.009472 gnus-simplify-subject-re 1325 0.0092170000 6.956...e-06 gnus-sort-threads 1 0.008926 0.008926 gnus-build-old-threads 1 0.008742 0.008742 gnus-update-missing-marks 1 0.007904 0.007904 gnus-build-get-header 6 0.0077429999 0.0012904999 gnus-summary-setup-buffer 1 0.006748 0.006748 gnus-summary-mode 1 0.005939 0.005939 gnus-uncompress-range 1 0.005137 0.005137 gnus-group-update-group 1 0.005086 0.005086 gnus-mode-line-buffer-identification 3 0.005086 0.0016953333 gnus-update-summary-mark-positions 2 0.00403 0.002015 gnus-summary-insert-line 4 0.0038380000 0.0009595000 gnus-message 7 0.003665 0.0005235714 -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-15 9:40 ` Francis Moreau @ 2010-11-21 6:02 ` Lars Magne Ingebrigtsen 2010-11-21 13:54 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-21 6:02 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > For example, I have a group which contains 1298 articles. Entering it > lasts 9 seconds: > > - 7 secs in the fetching process > - 2 secs in sorting + summary buffer generation [...] > I'm giving you the elp results when entering in the group containing > 1298 articles, with all articles displayed: > > gnus-thread-total-score 21809 18.607898999 0.0008532211 elp is still claiming that this function is being called 21k times, and that this takes 18 seconds. Which sort of doesn't make much sense, since: > gnus-group-select-group 1 10.66085 10.66085 This is surely the top-level function being called, so the 18 seconds sorting should be counted here, too, but it isn't, which is why I was wondering whether you were doing other things between each elp-results... Anyway: > gnus-get-newsgroup-headers-xover 1 6.78624 6.78624 There's the seven seconds... > gnus-summary-number-of-articles-in-thread 22361 3.2177489999 0.0001439000 And there's a totally crazy call number again. That function should be called (generally) once per displayed article in the summary buffer (if you have that in your summary line format). But it's called 10x the number of times. > gnus-dd-mmm 1301 0.9418810000 0.0007239669 I'm guessing you have a somewhat non-standard summary line format for that function to show up, but at least it tallies up with the number of articles in the group, somewhat. > gnus-id-to-thread 21805 0.1913100000 8.773...e-06 Again, a totally ridiculous call number... I'd suggest removing all Gnus customisations to find out what triggers these weird call numbers. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 6:02 ` Lars Magne Ingebrigtsen @ 2010-11-21 13:54 ` Francis Moreau 2010-11-21 18:17 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-21 13:54 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> For example, I have a group which contains 1298 articles. Entering it >> lasts 9 seconds: >> >> - 7 secs in the fetching process >> - 2 secs in sorting + summary buffer generation > > [...] > >> I'm giving you the elp results when entering in the group containing >> 1298 articles, with all articles displayed: >> >> gnus-thread-total-score 21809 18.607898999 0.0008532211 > > elp is still claiming that this function is being called 21k times, and > that this takes 18 seconds. Which sort of doesn't make much sense, since: > >> gnus-group-select-group 1 10.66085 10.66085 > > This is surely the top-level function being called, so the 18 seconds > sorting should be counted here, too, but it isn't, which is why I was > wondering whether you were doing other things between each > elp-results... I agree with you, these numbers are crazy. Actually I was looking at the time elapsed, and I can't interpret them at all. It says 18.6 seconds where as the whole process lasts 10 seconds at most. I re-did several times the same benchmarks and the figures seem stable. > > Anyway: > >> gnus-get-newsgroup-headers-xover 1 6.78624 6.78624 > > There's the seven seconds... > Yes the longuest one > >> gnus-summary-number-of-articles-in-thread 22361 3.2177489999 0.0001439000 > > And there's a totally crazy call number again. That function should be > called (generally) once per displayed article in the summary buffer (if > you have that in your summary line format). But it's called 10x the > number of times. > >> gnus-dd-mmm 1301 0.9418810000 0.0007239669 > > I'm guessing you have a somewhat non-standard summary line format for > that function to show up, but at least it tallies up with the number of > articles in the group, somewhat. Here's my summary line format if that can help: (setq gnus-summary-line-format (concat "%0{%U%R%z%}" "%10{│%}" "%3t" "%10{│%}" "%(%-20,20f %)" "%10{│%}" "%1{%d%}" "%10{│%}" "%6V" "%10{│%}" "%10{%B%}" "%s\n")) >> gnus-id-to-thread 21805 0.1913100000 8.773...e-06 > > Again, a totally ridiculous call number... > > I'd suggest removing all Gnus customisations to find out what triggers > these weird call numbers. Ok, I'm trying... Let's remove my summary line format customisation... and the numbers are now: gnus-topic-select-group 1 8.071814 8.071814 gnus-group-select-group 1 8.071757 8.071757 gnus-group-read-group 1 8.071748 8.071748 gnus-summary-read-group 1 8.071693 8.071693 gnus-summary-read-group-1 1 8.071681 8.071681 gnus-select-newsgroup 1 7.30483 7.30483 gnus-summary-limit-children 1324 7.1840629999 0.0054260294 gnus-fetch-headers 1 6.445576 6.445576 gnus-get-newsgroup-headers-xover 1 6.420683 6.420683 gnus-articles-to-read 1 0.818016 0.818016 gnus-summary-prepare 1 0.573297 0.573297 gnus-summary-prepare-threads 1 0.553526 0.553526 gnus-run-hooks 1337 0.2399320000 0.0001794554 gnus-summary-highlight-line 1326 0.2316510000 0.0001746990 gnus-sort-threads-recursive 837 0.1313619999 0.0001569438 gnus-summary-initial-limit 1 0.10973 0.10973 gnus-summary-highlight-line-0 1326 0.0972370000 7.333...e-05 gnus-summary-maybe-hide-threads 1 0.04301 0.04301 gnus-summary-hide-all-threads 1 0.043 0.043 gnus-summary-next-thread 44 0.0411179999 0.0009344999 gnus-summary-go-to-next-thread 44 0.040834 0.0009280454 gnus-retrieve-headers 2 0.03551 0.017755 gnus-map-function 1352 0.0294150000 2.175...e-05 which seem much more sensible. Now restoring my summary line format and removing only the "%6V" format specifier gives almost the same sane results. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 13:54 ` Francis Moreau @ 2010-11-21 18:17 ` Lars Magne Ingebrigtsen 2010-11-21 19:37 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-21 18:17 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Let's remove my summary line format customisation... and the numbers are > now: > > gnus-topic-select-group 1 8.071814 8.071814 [...] > gnus-summary-limit-children 1324 7.1840629999 0.0054260294 This seems wrong... I mean, the elapsed time. I guess elp doesn't give reliable total times for shorter functions... > gnus-get-newsgroup-headers-xover 1 6.420683 6.420683 This still takes almost seven seconds, which is weird. Do you have any xover-parsing customisations? -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 18:17 ` Lars Magne Ingebrigtsen @ 2010-11-21 19:37 ` Francis Moreau 2010-11-24 22:09 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-21 19:37 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Let's remove my summary line format customisation... and the numbers are >> now: >> >> gnus-topic-select-group 1 8.071814 8.071814 > > [...] > >> gnus-summary-limit-children 1324 7.1840629999 0.0054260294 > > This seems wrong... I mean, the elapsed time. I guess elp doesn't give > reliable total times for shorter functions... > >> gnus-get-newsgroup-headers-xover 1 6.420683 6.420683 > > This still takes almost seven seconds, which is weird. Do you have any > xover-parsing customisations? I'm afraid I haven't: $ grep xover .gnus shows nothing. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 19:37 ` Francis Moreau @ 2010-11-24 22:09 ` Lars Magne Ingebrigtsen 2010-11-25 7:35 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-24 22:09 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: >> This still takes almost seven seconds, which is weird. Do you have any >> xover-parsing customisations? > > I'm afraid I haven't: > > $ grep xover .gnus > > shows nothing. What about `gnus-decode-encoded-word-function'? -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-24 22:09 ` Lars Magne Ingebrigtsen @ 2010-11-25 7:35 ` Francis Moreau 0 siblings, 0 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-25 7:35 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >>> This still takes almost seven seconds, which is weird. Do you have any >>> xover-parsing customisations? >> >> I'm afraid I haven't: >> >> $ grep xover .gnus >> >> shows nothing. > > What about `gnus-decode-encoded-word-function'? I haven't set this one. The only occurence of the word 'decode' in my .gnus is for this: (eval-after-load "mm-decode" '(progn (add-to-list 'mm-discouraged-alternatives "text/html") (add-to-list 'mm-discouraged-alternatives "text/richtext"))) Thanks -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 18:20 ` Lars Magne Ingebrigtsen 2010-11-11 8:14 ` Francis Moreau @ 2010-11-12 11:11 ` Francis Moreau 2010-11-14 16:11 ` Lars Magne Ingebrigtsen 1 sibling, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-12 11:11 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> gnus-thread-total-score-1 82494 17.134565999 0.00020770 > > Geez. How many articles was this? BTW, even if scoring is disabled in this group, 'gnus-thread-total-score' is still called a lot of time. I can verify that scoring is disabled since all articles have a null score in this group. However looking at the backtrace below, it seems that the score is still calculated... Is it correct ? The Backtrace is: * gnus-thread-total-score(([8781 "Idees pour resoudre un probleme" "Francis Moreau <francis.moro@gmail. (format "%6d" (gnus-thread-total-score (and ... ...))) (insert (format "%6d" (gnus-thread-total-score ...))) (progn (gnus-add-text-properties (point) (progn ... ...) (quote ...)) (gnus-add-text-properties (poin eval((progn (gnus-add-text-properties (point) (progn ... ...) (quote ...)) (gnus-add-text-properties (progn (eval gnus-summary-line-format-spec) (point)) (gnus-put-text-property (point) (progn (eval gnus-summary-line-format-spec) (point)) (quote gnus-numb (progn (when (and gnus-tmp-dummy-line ...) (gnus-summary-insert-dummy-line gnus-tmp-dummy-line ...) ( (if gnus-tmp-header (progn (when ... ... ...) (setq gnus-tmp-unread ...) (push ... gnus-newsgroup-dat (when gnus-tmp-header (when (and gnus-tmp-dummy-line ...) (gnus-summary-insert-dummy-line gnus-tmp-du (if (stringp gnus-tmp-header) (cond (... ... ... ... ...) (... ... ...) (... ... ...) (t ...)) (setq (while (or threads stack gnus-tmp-new-adopts new-roots) (if (and ... ... ... ...) (if gnus-tmp-new-ad (if (vectorp (car threads)) (gnus-summary-prepare-unthreaded threads) (if gnus-summary-display-while- (let ((gnus-tmp-level 0) (default-score ...) (gnus-visual-p ...) (building-line-count gnus-summary-di gnus-summary-prepare-threads((([8781 "Idees pour resoudre un probleme" "Francis Moreau <francis.moro@ (progn (gnus-summary-prepare-threads (if gnus-show-threads ... ...))) (if gnus-newsgroup-headers (progn (gnus-summary-prepare-threads ...))) (when gnus-newsgroup-headers (gnus-summary-prepare-threads (if gnus-show-threads ... ...))) (let ((inhibit-read-only t)) (erase-buffer) (setq gnus-newsgroup-data nil gnus-newsgroup-data-reverse gnus-summary-prepare() (if no-display nil (gnus-summary-prepare)) (unless no-display (gnus-summary-prepare)) (cond ((not new-group) (gnus-set-global-variables) (when kill-buffer ...) (gnus-configure-windows ... Thanks -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-12 11:11 ` Francis Moreau @ 2010-11-14 16:11 ` Lars Magne Ingebrigtsen 2010-11-15 9:07 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-14 16:11 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > BTW, even if scoring is disabled in this group, > 'gnus-thread-total-score' is still called a lot of time. This function isn't dependent on scoring being enabled -- it's a thread sorting function that examines the total score of each thread. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-14 16:11 ` Lars Magne Ingebrigtsen @ 2010-11-15 9:07 ` Francis Moreau 2010-11-21 5:44 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-15 9:07 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> BTW, even if scoring is disabled in this group, >> 'gnus-thread-total-score' is still called a lot of time. > > This function isn't dependent on scoring being enabled -- it's a thread > sorting function that examines the total score of each thread. ah ok, but what the point to examine the total score if scoring is disabled ? -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-15 9:07 ` Francis Moreau @ 2010-11-21 5:44 ` Lars Magne Ingebrigtsen 2010-11-21 14:07 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-21 5:44 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > ah ok, but what the point to examine the total score if scoring is > disabled ? If the user disables scoring, but still insists on using the total scoring for the sorting, then, er, Gnus could discover that, but it seems like a pretty odd situation. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 5:44 ` Lars Magne Ingebrigtsen @ 2010-11-21 14:07 ` Francis Moreau 0 siblings, 0 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-21 14:07 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> ah ok, but what the point to examine the total score if scoring is >> disabled ? > > If the user disables scoring, but still insists on using the total > scoring for the sorting, then, er, Gnus could discover that, but it > seems like a pretty odd situation. Well, this how this happens for my config. I globaly set my thread sorting function to: (setq gnus-thread-sort-functions '((not gnus-thread-sort-by-number) gnus-thread-sort-by-total-score)) But it appears that for some groups (the cached one), scoring is not really appropriate, so I just use group parameter for these groupe to disable scoring. I don't know, maybe it's just me, but I would have expected Gnus to automatically not spend time in calculating scores specially since all scores values are shown as 0 in the summary buffer. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 14:16 ` Francis Moreau 2010-11-10 18:20 ` Lars Magne Ingebrigtsen @ 2010-11-10 18:21 ` Lars Magne Ingebrigtsen 2010-11-11 8:22 ` Francis Moreau 2010-11-12 18:55 ` Dan Christensen 2 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-10 18:21 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > gnus-thread-total-score-1 82494 17.134565999 0.0002077068 And I guess that this means that you're sorting the threads by total score? If you try normal sorting, does group entry time become OK? -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 18:21 ` Lars Magne Ingebrigtsen @ 2010-11-11 8:22 ` Francis Moreau 2010-11-11 18:33 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-11 8:22 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> gnus-thread-total-score-1 82494 17.134565999 0.0002077068 > > And I guess that this means that you're sorting the threads by total > score? Yes I have this in my .gnus: ("^nntp\\|gmane\\." (display . all) (gnus-use-scoring t) (gnus-thread-sort-functions '((not gnus-thread-sort-by-number) gnus-thread-sort-by-total-score))))) > If you try normal sorting, does group entry time become OK? Well the group entry still lasts 3 seconds. And when showing all articles in this group '\ o', the 22 secs time now lasts 17 seconds... So there's an improvement, but I don't think that's ok. Thanks -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-11 8:22 ` Francis Moreau @ 2010-11-11 18:33 ` Lars Magne Ingebrigtsen 2010-11-11 19:56 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-11 18:33 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: >> If you try normal sorting, does group entry time become OK? > > Well the group entry still lasts 3 seconds. So it goes from 27 seconds to 3 seconds? -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-11 18:33 ` Lars Magne Ingebrigtsen @ 2010-11-11 19:56 ` Francis Moreau 2010-11-14 16:10 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-11 19:56 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >>> If you try normal sorting, does group entry time become OK? >> >> Well the group entry still lasts 3 seconds. > > So it goes from 27 seconds to 3 seconds? Nope, entering the group was 4 seconds and if I remove total scoring it decreases to 3 seconds. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-11 19:56 ` Francis Moreau @ 2010-11-14 16:10 ` Lars Magne Ingebrigtsen 0 siblings, 0 replies; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-14 16:10 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Nope, entering the group was 4 seconds and if I remove total scoring it > decreases to 3 seconds. Right. Hm... -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-10 14:16 ` Francis Moreau 2010-11-10 18:20 ` Lars Magne Ingebrigtsen 2010-11-10 18:21 ` Lars Magne Ingebrigtsen @ 2010-11-12 18:55 ` Dan Christensen 2010-11-12 20:07 ` Francis Moreau 2 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-11-12 18:55 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > and while in the 22 secs processing, hitting C-g gives: > > parse-time-tokenize > parse-time-string > mail-header-parse-date > gnus-thread-latest-date > gnus-thread-sort-by-most-recent-date > gnus-sort-threads-recursive > . > 33 calls to gnus-sort-threads-recursive > . > gnus-sort-threads ... > gnus-sort-threads-recursive 641 65.774695999 0.1026126302 > gnus-thread-sort-by-most-recent-date 4348 31.652889000 0.0072798732 > gnus-thread-latest-date 8696 31.436723000 0.0036150785 ... > Yes I have this in my .gnus: > > ("^nntp\\|gmane\\." > (display . all) > (gnus-use-scoring t) > (gnus-thread-sort-functions '((not gnus-thread-sort-by-number) > gnus-thread-sort-by-total-score))))) Why are there calls to gnus-thread-sort-by-most-recent-date above, even though that's not listed as a sort function? In my experience, it is very slow, so eliminating it should help. I did some work to speed it up, but if anyone else can improve it further, that would be good. Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-12 18:55 ` Dan Christensen @ 2010-11-12 20:07 ` Francis Moreau 2010-11-12 21:38 ` Dan Christensen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-12 20:07 UTC (permalink / raw) To: Dan Christensen; +Cc: ding Dan Christensen <jdc@uwo.ca> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> and while in the 22 secs processing, hitting C-g gives: >> >> parse-time-tokenize >> parse-time-string >> mail-header-parse-date >> gnus-thread-latest-date >> gnus-thread-sort-by-most-recent-date >> gnus-sort-threads-recursive >> . >> 33 calls to gnus-sort-threads-recursive >> . >> gnus-sort-threads > > ... > >> gnus-sort-threads-recursive 641 65.774695999 0.1026126302 >> gnus-thread-sort-by-most-recent-date 4348 31.652889000 0.0072798732 >> gnus-thread-latest-date 8696 31.436723000 0.0036150785 > > ... > >> Yes I have this in my .gnus: >> >> ("^nntp\\|gmane\\." >> (display . all) >> (gnus-use-scoring t) >> (gnus-thread-sort-functions '((not gnus-thread-sort-by-number) >> gnus-thread-sort-by-total-score))))) > > Why are there calls to gnus-thread-sort-by-most-recent-date above, even > though that's not listed as a sort function? You're right the actual sort functions used were: (setq gnus-thread-sort-functions '(gnus-thread-sort-by-most-recent-date gnus-thread-sort-by-total-score)) > In my experience, it is very slow, so eliminating it should help. I > did some work to speed it up, but if anyone else can improve it > further, that would be good. Well, that's a crude workaround since this what I'd like to use. But why are there so many calls to gnus-thread-total-score and gnus-thread-sort-by-most-recent-date knowing that my group contains only 1298 articles ? Do you have any hint to improve it further ? Removing gnus-thread-sort-by-most-recent-date considerably accelerate the thread sorting time. Also do you know why the fetching process is so long (~ 7secs) specially since the group contains only cached articles ? Thanks -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-12 20:07 ` Francis Moreau @ 2010-11-12 21:38 ` Dan Christensen 2010-11-13 20:46 ` Francis Moreau 2010-11-14 16:14 ` Lars Magne Ingebrigtsen 0 siblings, 2 replies; 103+ messages in thread From: Dan Christensen @ 2010-11-12 21:38 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > But why are there so many calls to gnus-thread-total-score and > gnus-thread-sort-by-most-recent-date knowing that my group contains only > 1298 articles ? The way thread sorting works is that not only are the threads sorted, but the children of each article are also sorted using the same function. So it gets called recursively a lot of times. That's why a change I made to cache the parsed date/time info makes such a bit difference. > gnus-thread-sort-by-most-recent-date 4348 31.652889000 0.0072798732 > gnus-thread-total-score-1 82494 17.134565999 0.0002077068 > gnus-thread-sort-by-total-score 8697 3.4791829999 0.0004000440 On the other hand, I'm still not sure why the above functions are called that many times. I wonder if the recursion is going one level deeper than necessary? Or if the overall sorting as well as the sorting of children can be accomplished in a unified way, without repeating so much work? E.g., maybe first each thread could be traversed, and on the way back up from the leaves to the root, you propagate to each node the most recent date seen by any of the children, storing this as some extra data with each article. Then you just sort using this stored data. > Also do you know why the fetching process is so long (~ 7secs) specially > since the group contains only cached articles ? Sorry, I don't know anything about that. Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-12 21:38 ` Dan Christensen @ 2010-11-13 20:46 ` Francis Moreau 2010-11-14 9:32 ` Francis Moreau 2010-11-14 16:13 ` Lars Magne Ingebrigtsen 2010-11-14 16:14 ` Lars Magne Ingebrigtsen 1 sibling, 2 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-13 20:46 UTC (permalink / raw) To: Dan Christensen; +Cc: ding Dan Christensen <jdc@uwo.ca> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> But why are there so many calls to gnus-thread-total-score and >> gnus-thread-sort-by-most-recent-date knowing that my group contains only >> 1298 articles ? > > The way thread sorting works is that not only are the threads sorted, > but the children of each article are also sorted using the same > function. So it gets called recursively a lot of times. That's > why a change I made to cache the parsed date/time info makes such > a bit difference. Hmm, I'm wondering if sorting the 'children' for a given thread is really important. I would say that I could live without it I think specially if it makes group entering faster. Or better, when entering in a group, threads are not expanded, and sorting children could happen when expanding threads. That would make the group entering much faster and children could still be sorted. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-13 20:46 ` Francis Moreau @ 2010-11-14 9:32 ` Francis Moreau 2010-11-29 0:40 ` Dan Christensen 2010-11-14 16:13 ` Lars Magne Ingebrigtsen 1 sibling, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-14 9:32 UTC (permalink / raw) To: Dan Christensen; +Cc: ding Francis Moreau <francis.moro@gmail.com> writes: > Dan Christensen <jdc@uwo.ca> writes: > >> Francis Moreau <francis.moro@gmail.com> writes: >> >>> But why are there so many calls to gnus-thread-total-score and >>> gnus-thread-sort-by-most-recent-date knowing that my group contains only >>> 1298 articles ? >> >> The way thread sorting works is that not only are the threads sorted, >> but the children of each article are also sorted using the same >> function. So it gets called recursively a lot of times. That's >> why a change I made to cache the parsed date/time info makes such >> a bit difference. > > Hmm, I'm wondering if sorting the 'children' for a given thread is > really important. I would say that I could live without it I think > specially if it makes group entering faster. Indeed changing the definition of gnus-sort-threads-recursive like the following: (defun gnus-sort-threads-recursive (threads func) (sort threads func)) make the sorting time almost null. And when looking at the threads, I even don't make any differences with the previous method. Now the time is totaly spent in the fetching process thing. > Or better, when entering in a group, threads are not expanded, and > sorting children could happen when expanding threads. That would make > the group entering much faster and children could still be sorted. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-14 9:32 ` Francis Moreau @ 2010-11-29 0:40 ` Dan Christensen 2010-11-29 4:47 ` Lars Magne Ingebrigtsen 2010-11-29 8:27 ` Francis Moreau 0 siblings, 2 replies; 103+ messages in thread From: Dan Christensen @ 2010-11-29 0:40 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Hmm, I'm wondering if sorting the 'children' for a given thread is >> really important. I would say that I could live without it I think >> specially if it makes group entering faster. > > Indeed changing the definition of gnus-sort-threads-recursive like the > following: > > (defun gnus-sort-threads-recursive (threads func) > (sort threads func)) > > make the sorting time almost null. In my tests, if one of the sort functions is gnus-thread-sort-by-most-recent-date, then the above takes almost as long as sorting all of the children too, since it needs to convert *all* time strings to emacs times, and that dominates. As you pointed out, the real thing to investigate is the rest of the time taken for summary buffer generation. Any takers? Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 0:40 ` Dan Christensen @ 2010-11-29 4:47 ` Lars Magne Ingebrigtsen 2010-11-29 6:01 ` Miles Bader 2010-11-29 8:27 ` Francis Moreau 1 sibling, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-29 4:47 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > As you pointed out, the real thing to investigate is the rest of > the time taken for summary buffer generation. Any takers? For large buffers, a lot of the time is spent on marks management. The various marks are list lists of numbers, and `memq' is being used to check for their presence. I've had fixing this on my agenda for a few months, but it hasn't happened yet. :-) The plan is to not uncompress the ranges into lists, and use `gnus-member-of-range' instead: (setq range '((1 . 10000) (20000 . 30000))) (length (setq list (gnus-uncompress-range range))) (benchmark-run-compiled 20000 (memq 15000 list)) 1.4s (benchmark-run-compiled 20000 (gnus-member-of-range 15000 range)) 0.1s -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 4:47 ` Lars Magne Ingebrigtsen @ 2010-11-29 6:01 ` Miles Bader 2010-11-29 6:18 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Miles Bader @ 2010-11-29 6:01 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > I've had fixing this on my agenda for a few months, but it hasn't > happened yet. :-) The plan is to not uncompress the ranges into lists, > and use `gnus-member-of-range' instead: Hmm, so should vastly reduce consing too... -miles -- .Numeric stability is probably not all that important when you're guessing. ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 6:01 ` Miles Bader @ 2010-11-29 6:18 ` Lars Magne Ingebrigtsen 2010-11-29 6:36 ` Lars Magne Ingebrigtsen 2010-11-29 6:40 ` Daniel Pittman 0 siblings, 2 replies; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-29 6:18 UTC (permalink / raw) To: ding Miles Bader <miles@gnu.org> writes: > Hmm, so should vastly reduce consing too... Yup. I hope that it'll have a significant effect overall when entering large groups, but I think the proof is hidden inside a pudding somewhere... -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 6:18 ` Lars Magne Ingebrigtsen @ 2010-11-29 6:36 ` Lars Magne Ingebrigtsen 2010-11-29 6:40 ` Daniel Pittman 1 sibling, 0 replies; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-29 6:36 UTC (permalink / raw) To: ding And if that doesn't help, the ranges can be rewritten to balanced binary trees. (I think that was Stefan's suggestion...) That is, if you look at my list of replied articles in this group: (reply 56100 56109 56168 56244 56260 56335 56346 56368 (56402 . 56403) (56462 . 56463) 56531 56536 56578 (56597 . 56598) 56670 56717 56747 56791 (56798 . 56799) 56825 56827 56829 (56839 . 56840) 56842 56850 56852 56856 56859 56862 56882 56909 56926 56977 57007 57019 57088 57093 57110 57127 57139 57146 (57152 . 57153) [gazillions more entries dropped] then doing `gnus-member-of-range' on those are probably not going to be very fast. But if they were arranged as a balanced binary tree, then it'd be only O(log n) to find them. And the conversion from list-based ranges to balanced binary trees is probably fast. I think. Has anybody written such a function in Lisp, by any chance? :-) -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 6:18 ` Lars Magne Ingebrigtsen 2010-11-29 6:36 ` Lars Magne Ingebrigtsen @ 2010-11-29 6:40 ` Daniel Pittman 1 sibling, 0 replies; 103+ messages in thread From: Daniel Pittman @ 2010-11-29 6:40 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Miles Bader <miles@gnu.org> writes: > >> Hmm, so should vastly reduce consing too... > > Yup. I hope that it'll have a significant effect overall when entering > large groups, but I think the proof is hidden inside a pudding somewhere... FWIW, I started using offlineimap and Dovecot to sync from my Zimbra server because NNIMAP / Gnus was expanding the entire range at a few points just to do this. Which would cons a range of ~ 100,000 integers, do stuff, and then free it all. Which was ... painful. I never got around to identifying exactly where it was, though. Anyway, I would be very happy to see this change. :) Daniel -- ✣ Daniel Pittman ✉ daniel@rimspace.net ☎ +61 401 155 707 ♽ made with 100 percent post-consumer electrons ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 0:40 ` Dan Christensen 2010-11-29 4:47 ` Lars Magne Ingebrigtsen @ 2010-11-29 8:27 ` Francis Moreau 2010-11-29 19:30 ` Dan Christensen 1 sibling, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-29 8:27 UTC (permalink / raw) To: Dan Christensen; +Cc: ding Dan Christensen <jdc@uwo.ca> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Francis Moreau <francis.moro@gmail.com> writes: >> >>> Hmm, I'm wondering if sorting the 'children' for a given thread is >>> really important. I would say that I could live without it I think >>> specially if it makes group entering faster. >> >> Indeed changing the definition of gnus-sort-threads-recursive like the >> following: >> >> (defun gnus-sort-threads-recursive (threads func) >> (sort threads func)) >> >> make the sorting time almost null. > > In my tests, if one of the sort functions is > gnus-thread-sort-by-most-recent-date, then the above takes almost as > long as sorting all of the children too, since it needs to convert > *all* time strings to emacs times, and that dominates. Why are all time strings converted to emacs times in that case ? Thanks -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 8:27 ` Francis Moreau @ 2010-11-29 19:30 ` Dan Christensen 2010-11-29 20:10 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-11-29 19:30 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Dan Christensen <jdc@uwo.ca> writes: > >> In my tests, if one of the sort functions is >> gnus-thread-sort-by-most-recent-date, then the above takes almost as >> long as sorting all of the children too, since it needs to convert >> *all* time strings to emacs times, and that dominates. > > Why are all time strings converted to emacs times in that case ? Because to know the most recent date in a thread, you have to compare all of the dates of articles in that thread. If you only want to sort by the date of the root of the thread, then you switch to a sort function with a slightly different name, and you would then find that sorting just the top level threads is much faster than also sorting all children as well. (On the other hand, I think later in the summary buffer preparation process, Gnus would take the time to convert all date strings to emacs times *anyways*, for display in the summary buffer. So the *overall* time for Summary buffer generation would be about the same no matter which of the sort methods you used.) Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 19:30 ` Dan Christensen @ 2010-11-29 20:10 ` Francis Moreau 2010-11-29 20:23 ` Dan Christensen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-29 20:10 UTC (permalink / raw) To: Dan Christensen; +Cc: ding Dan Christensen <jdc@uwo.ca> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Dan Christensen <jdc@uwo.ca> writes: >> >>> In my tests, if one of the sort functions is >>> gnus-thread-sort-by-most-recent-date, then the above takes almost as >>> long as sorting all of the children too, since it needs to convert >>> *all* time strings to emacs times, and that dominates. >> >> Why are all time strings converted to emacs times in that case ? > > Because to know the most recent date in a thread, you have to compare > all of the dates of articles in that thread. Really ? I would think that you just have to compare the date of all thread *leafs* only. No ? -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 20:10 ` Francis Moreau @ 2010-11-29 20:23 ` Dan Christensen 2010-11-29 20:56 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-11-29 20:23 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > I would think that you just have to compare the date of all thread *leafs* > only. You can't always assume that the References headers are correct, or that the Date headers provided are correct, etc. Moreover, it doesn't matter, since the dates will be parsed later anyways. Since the conversion is cached, all of this only affects when the time is taken to do the conversion, not the total time. On my laptop, with 6500 articles, we're only talking about 1.3 seconds, so this is already pretty good. But I suspect it can be sped up. Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 20:23 ` Dan Christensen @ 2010-11-29 20:56 ` Francis Moreau 2010-11-29 21:30 ` Dan Christensen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-29 20:56 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> I would think that you just have to compare the date of all thread *leafs* >> only. > > You can't always assume that the References headers are correct, or that > the Date headers provided are correct, etc. > > Moreover, it doesn't matter, since the dates will be parsed later > anyways. Since the conversion is cached, all of this only affects > when the time is taken to do the conversion, not the total time. > > On my laptop, with 6500 articles, we're only talking about 1.3 seconds, > so this is already pretty good. But I suspect it can be sped up. I agree this is good. But that's no what I get with 1300 *cached* articles. I still have almost 10 seconds to get the summary buffer. I initialy had more than 20 seconds and had to disable the scoring (okay that's pretty useless for my cached groups but still). -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 20:56 ` Francis Moreau @ 2010-11-29 21:30 ` Dan Christensen 2010-12-05 15:01 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-11-29 21:30 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Dan Christensen <jdc@uwo.ca> writes: > >> On my laptop, with 6500 articles, we're only talking about 1.3 seconds, >> so this is already pretty good. But I suspect it can be sped up. > > I agree this is good. > > But that's no what I get with 1300 *cached* articles. I still have > almost 10 seconds to get the summary buffer. The 1.3 seconds I mentioned was just the time for date parsing. It takes something like 13 or 15 seconds for the full summary buffer generation in my test group. Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 21:30 ` Dan Christensen @ 2010-12-05 15:01 ` Lars Magne Ingebrigtsen 2010-12-05 18:05 ` Dan Christensen 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-12-05 15:01 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: >>> On my laptop, with 6500 articles, we're only talking about 1.3 seconds, >>> so this is already pretty good. But I suspect it can be sped up. [...] > The 1.3 seconds I mentioned was just the time for date parsing. > It takes something like 13 or 15 seconds for the full summary buffer > generation in my test group. With only 6500 articles? What kind of CPU would that be? -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-12-05 15:01 ` Lars Magne Ingebrigtsen @ 2010-12-05 18:05 ` Dan Christensen 2010-12-05 18:46 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-12-05 18:05 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Dan Christensen <jdc@uwo.ca> writes: > >>>> On my laptop, with 6500 articles, we're only talking about 1.3 seconds, >>>> so this is already pretty good. But I suspect it can be sped up. > > [...] > >> The 1.3 seconds I mentioned was just the time for date parsing. >> It takes something like 13 or 15 seconds for the full summary buffer >> generation in my test group. > > With only 6500 articles? What kind of CPU would that be? Core 2 Duo 2.2GHz. Here are the top elp-results from just entering this group, which has 6556 articles. Seems to be a lot of different things all contributing to this time. Sorting is more than I expected: 6.3s. But I'm a bit confused: I think times in the second-last column include the time spent in called functions, but gnus-sort-threads takes only 3.3s while gnus-sort-threads-recursive takes 6.3s. I thought g-s-t-r was only called from g-s-t!? And the other day, I posted the following timing info from a freshly started emacs: (benchmark-run 1 (gnus-sort-threads threads)) (1.638083 8 0.38992400000000016) I'm also confused by the number of calls to gnus-thread-total-score-1 (22704). First the data from a day-old emacs; afterwards, the data for a fresh emacs. gnus-topic-read-group 1 14.067652 14.067652 gnus-group-read-group 1 14.067596 14.067596 gnus-summary-read-group 1 14.067559 14.067559 gnus-summary-read-group-1 1 14.067547 14.067547 gnus-summary-prepare 1 9.758998 9.758998 gnus-sort-threads-recursive 3844 6.3334839999 0.0016476285 gnus-summary-prepare-threads 1 5.421154 5.421154 gnus-select-newsgroup 1 4.298248 4.298248 gnus-fetch-headers 1 4.274015 4.274015 gnus-sort-threads 1 3.306262 3.306262 gnus-thread-sort-by-most-recent-date 17741 3.1533499999 0.0001777436 gnus-thread-latest-date 35482 3.0043529999 8.467...e-05 gnus-get-newsgroup-headers 1 2.925865 2.925865 gnus-retrieve-headers 2 2.6951549999 1.3475774999 gnus-user-format-function-s 6558 1.6891390000 0.0002575692 gnus-replace-in-string 6559 1.6063970000 0.0002449149 gnus-thread-total-score 22704 1.4428329999 6.354...e-05 gnus-cache-retrieve-headers 1 1.347705 1.347705 gnus-thread-total-score-1 22704 1.3222239999 5.823...e-05 gnus-run-hooks 11 0.914453 0.0831320909 gnus-registry-register-message-ids 1 0.9140429999 0.9140429999 gnus-registry-fetch-message-id-fast 6556 0.8349619999 0.0001273584 gnus-user-date 6558 0.5879020000 8.964...e-05 gnus-simplify-subject-fuzzy 8487 0.4746110000 5.592...e-05 gnus-user-format-function-t 6558 0.4306520000 6.566...e-05 gnus-summary-from-or-to-or-newsgroups 6558 0.3486270000 5.316...e-05 gnus-summary-highlight-line 6558 0.3438830000 5.243...e-05 gnus-float-time 134926 0.3180409999 2.357...e-06 gnus-put-text-property 26236 0.2860350000 1.090...e-05 gnus-seconds-year 6537 0.1706889999 2.611...e-05 gnus-put-text-property-excluding-characters-with-faces 6559 0.1672420000 2.549...e-05 After restarting emacs, the numbers look a bit better (especially gnus-sort-threads-recursive): gnus-topic-read-group 1 13.214392 13.214392 gnus-group-read-group 1 13.214339 13.214339 gnus-summary-read-group 1 13.214303 13.214303 gnus-summary-read-group-1 1 13.214291 13.214291 gnus-summary-prepare 1 9.061338 9.061338 gnus-summary-prepare-threads 1 5.205671 5.205671 gnus-sort-threads-recursive 3844 4.8261949999 0.0012555137 gnus-select-newsgroup 1 4.1157070000 4.1157070000 gnus-fetch-headers 1 4.095992 4.095992 gnus-get-newsgroup-headers 1 2.736999 2.736999 gnus-retrieve-headers 2 2.7169679999 1.3584839999 gnus-sort-threads 1 2.71412 2.71412 gnus-thread-sort-by-most-recent-date 17741 2.6269509999 0.0001480723 gnus-thread-latest-date 35482 2.4759769999 6.978...e-05 gnus-thread-total-score 22704 2.1373710000 9.414...e-05 gnus-thread-total-score-1 22704 1.9588889999 8.627...e-05 gnus-user-format-function-s 6558 1.7538890000 0.0002674426 gnus-replace-in-string 6559 1.6119780000 0.0002457658 gnus-cache-retrieve-headers 1 1.358604 1.358604 gnus-run-hooks 13 0.972687 0.0748220769 gnus-registry-register-message-ids 1 0.972067 0.972067 gnus-registry-fetch-message-id-fast 6556 0.8265469999 0.0001260748 gnus-user-format-function-t 6558 0.6889870000 0.0001050605 gnus-user-date 6558 0.5876640000 8.961...e-05 gnus-simplify-subject-fuzzy 8487 0.4407050000 5.192...e-05 gnus-float-time 134926 0.3646349999 2.702...e-06 gnus-summary-from-or-to-or-newsgroups 6558 0.2739099999 4.176...e-05 gnus-summary-highlight-line 6558 0.2018580000 3.078...e-05 gnus-seconds-year 6537 0.1698009999 2.597...e-05 gnus-put-text-property 26236 0.1380570000 5.262...e-06 gnus-seconds-today 6558 0.1306740000 1.992...e-05 gnus-registry-fetch-groups 6556 0.0946810000 1.444...e-05 gnus-put-text-property-excluding-characters-with-faces 6559 0.0879599999 1.341...e-05 gnus-make-threads 1 0.086339 0.086339 gnus-gather-threads-by-subject 1 0.077483 0.077483 gnus-id-to-thread 22704 0.0766690000 3.376...e-06 gnus-extract-address-components 6558 0.0762749999 1.163...e-05 Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-12-05 18:05 ` Dan Christensen @ 2010-12-05 18:46 ` Lars Magne Ingebrigtsen 0 siblings, 0 replies; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-12-05 18:46 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > gnus-topic-read-group 1 13.214392 13.214392 > gnus-group-read-group 1 13.214339 13.214339 > gnus-summary-read-group 1 13.214303 13.214303 > gnus-summary-read-group-1 1 13.214291 13.214291 > gnus-summary-prepare 1 9.061338 9.061338 > gnus-summary-prepare-threads 1 5.205671 5.205671 > gnus-sort-threads-recursive 3844 4.8261949999 0.0012555137 > gnus-select-newsgroup 1 4.1157070000 4.1157070000 > gnus-fetch-headers 1 4.095992 4.095992 > gnus-get-newsgroup-headers 1 2.736999 2.736999 > gnus-retrieve-headers 2 2.7169679999 1.3584839999 > gnus-sort-threads 1 2.71412 2.71412 > gnus-thread-sort-by-most-recen 17741 2.6269509999 0.0001480723 > gnus-thread-latest-date 35482 2.4759769999 6.978...e-05 > gnus-thread-total-score 22704 2.1373710000 9.414...e-05 > gnus-thread-total-score-1 22704 1.9588889999 8.627...e-05 > gnus-user-format-function-s 6558 1.7538890000 0.0002674426 > gnus-replace-in-string 6559 1.6119780000 0.0002457658 > gnus-cache-retrieve-headers 1 1.358604 1.358604 > gnus-run-hooks 13 0.972687 0.0748220769 > gnus-registry-register-message 1 0.972067 0.972067 > gnus-registry-fetch-message-id 6556 0.8265469999 0.0001260748 > gnus-user-format-function-t 6558 0.6889870000 0.0001050605 > gnus-user-date 6558 0.5876640000 8.961...e-05 > gnus-simplify-subject-fuzzy 8487 0.4407050000 5.192...e-05 > gnus-float-time 134926 0.3646349999 2.702...e-06 Here's me entering a group with 6000 articles (3GHz Core 2): gnus-group-select-group 1 3.819664 3.819664 gnus-group-read-group 1 3.81966 3.81966 gnus-summary-read-group 1 3.819644 3.819644 gnus-summary-read-group-1 1 3.819639 3.819639 gnus-summary-prepare 1 1.736278 1.736278 gnus-summary-prepare-threads 1 1.492577 1.492577 gnus-select-newsgroup 1 1.308981 1.308981 gnus-fetch-headers 1 1.178148 1.178148 gnus-get-newsgroup-headers-xover 1 1.041805 1.041805 gnus-possibly-score-headers 1 0.693136 0.693136 gnus-score-headers 1 0.69119 0.69119 gnus-score-string 2 0.6885730000 0.3442865000 gnus-retrieve-headers 2 0.272322 0.136161 gnus-sort-threads-recursive 324 0.210677 0.0006502376 gnus-sort-threads 1 0.209757 0.209757 gnus-put-text-property 17794 0.1451210000 8.155...e-06 gnus-summary-highlight-line 5930 0.1418359999 2.391...e-05 gnus-cache-retrieve-headers 1 0.136199 0.136199 gnus-put-text-property-excluding-char 5931 0.0833610000 1.405...e-05 gnus-summary-initial-limit 1 0.074482 0.074482 gnus-summary-from-or-to-or-newsgroups 5930 0.0732839999 1.235...e-05 gnus-articles-to-read 1 0.070914 0.070914 gnus-summary-limit-children 5928 0.0673060000 1.135...e-05 gnus-score-string< 125555 0.0566680000 4.513...e-07 WTF? 125K calls to gnus-score-string<? I'm starting to wonder whether ELP works properly... -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-13 20:46 ` Francis Moreau 2010-11-14 9:32 ` Francis Moreau @ 2010-11-14 16:13 ` Lars Magne Ingebrigtsen 2010-11-15 9:03 ` Francis Moreau 1 sibling, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-14 16:13 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Hmm, I'm wondering if sorting the 'children' for a given thread is > really important. I would say that I could live without it I think > specially if it makes group entering faster. It's annoying when you get the wrong sorting for gathered threads. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-14 16:13 ` Lars Magne Ingebrigtsen @ 2010-11-15 9:03 ` Francis Moreau 2010-11-21 5:43 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-15 9:03 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Hmm, I'm wondering if sorting the 'children' for a given thread is >> really important. I would say that I could live without it I think >> specially if it makes group entering faster. > > It's annoying when you get the wrong sorting for gathered threads. But you can still sort children of *gathered* thread although that you probably won't sort anything since for this thread, each article has only one children. I was just wondering about 'normal' threads. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-15 9:03 ` Francis Moreau @ 2010-11-21 5:43 ` Lars Magne Ingebrigtsen 2010-11-21 14:39 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-21 5:43 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > But you can still sort children of *gathered* thread although that you > probably won't sort anything since for this thread, each article has > only one children. Each gathered thread may have many children... -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 5:43 ` Lars Magne Ingebrigtsen @ 2010-11-21 14:39 ` Francis Moreau 0 siblings, 0 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-21 14:39 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> But you can still sort children of *gathered* thread although that you >> probably won't sort anything since for this thread, each article has >> only one children. > > Each gathered thread may have many children... Well all gathered thread I saw have several children indeed but each child had only one child: t0 -> t1 -> t2 -> t3 -> ... But gathered thread is not really important I think since it represents only a minor part. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-12 21:38 ` Dan Christensen 2010-11-13 20:46 ` Francis Moreau @ 2010-11-14 16:14 ` Lars Magne Ingebrigtsen 2010-11-14 18:10 ` Dan Christensen 1 sibling, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-14 16:14 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > That's why a change I made to cache the parsed date/time info makes > such a bit difference. Do you have a patch for this? -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-14 16:14 ` Lars Magne Ingebrigtsen @ 2010-11-14 18:10 ` Dan Christensen 2010-11-14 18:26 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-11-14 18:10 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Dan Christensen <jdc@uwo.ca> writes: > >> That's why a change I made to cache the parsed date/time info makes >> such a bit difference. > > Do you have a patch for this? It was merged in June. Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-14 18:10 ` Dan Christensen @ 2010-11-14 18:26 ` Lars Magne Ingebrigtsen 2010-11-14 21:27 ` Dan Christensen 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-14 18:26 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > It was merged in June. Ok. :-) It doesn't cache total-thread-score, then, I'm assuming... -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-14 18:26 ` Lars Magne Ingebrigtsen @ 2010-11-14 21:27 ` Dan Christensen 2010-11-21 5:42 ` Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-11-14 21:27 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > It doesn't cache total-thread-score, then, I'm assuming... No, in fact, the only change was to make sure that gnus-date-get-time is used everywhere, since that function caches the result of safe-date-to-time as a text-property. Separate from that low-level cache, the sorting function could probably annotate the thread tree and re-use that information for all of the recursive sorts. Or use a decorate-sort-undecorate pattern. I haven't thought through the details, though, so I pose this as a challenge for someone. :-) Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-14 21:27 ` Dan Christensen @ 2010-11-21 5:42 ` Lars Magne Ingebrigtsen 2010-11-21 20:42 ` Dan Christensen 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-21 5:42 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > Separate from that low-level cache, the sorting function could > probably annotate the thread tree and re-use that information > for all of the recursive sorts. Or use a decorate-sort-undecorate > pattern. It sounds possible, but peeking at the code, it doesn't seem trivial to do generally, since the user can compose their own composited sorting functions. So the cache would have to be per article+sorting-function... hm... -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 5:42 ` Lars Magne Ingebrigtsen @ 2010-11-21 20:42 ` Dan Christensen 2010-11-21 21:46 ` Francis Moreau ` (2 more replies) 0 siblings, 3 replies; 103+ messages in thread From: Dan Christensen @ 2010-11-21 20:42 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Dan Christensen <jdc@uwo.ca> writes: > >> Separate from that low-level cache, the sorting function could >> probably annotate the thread tree and re-use that information >> for all of the recursive sorts. Or use a decorate-sort-undecorate >> pattern. > > It sounds possible, but peeking at the code, it doesn't seem trivial to > do generally, since the user can compose their own composited sorting > functions. So the cache would have to be per > article+sorting-function... hm... Re: caching: I think caching mainly makes sense for the functions which require traversing the thread tree: gnus-thread-sort-by-total-score and gnus-thread-sort-by-most-recent-number and -date. Are there any others? If these tucked away their result when called, I imagine there would be a big savings. Depending on the shape of the tree, this would reduce sort key computation time from somewhere between O(n log n) and O(n^2) to O(n). Re: decorate-sort-undecorate (DSU): Here for each article you create a key which is a list of the relevant values. For example, if the sort functions are '((not gnus-thread-sort-by-number) gnus-thread-sort-by-score)) then the key would be (-number score) [or maybe the reverse]. Then you replace the list of threads with a list of pairs (key thread) and sort on the keys. This avoids recomputing the key each time an article needs to be compared with another, which would reduce overhead a lot. After sorting, you make a pass through to extract the list of threads in sorted order. When computing the keys, if you traverse the threads from leaves to the root, you can compute the keys quickly without needing to implement the caching. But I don't know what to do if the user implements their own sorting function. It would be easier if the interface was changed: instead of providing a list of comparison functions, you provide a list of functions which compute key values which are then compared using the default emacs order. Then we could always use DSU. But maybe it's not worth changing the interface for this? Options: (a) Try caching, and see if it is good enough. (b) Use DSU when only built-in sort functions are used, and fall back to existing behaviour? (Complicates code to have a fall-back, and also would have to assume that the user didn't redefine the built-in functions.) (c) Change the interface so DSU can always be used? Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 20:42 ` Dan Christensen @ 2010-11-21 21:46 ` Francis Moreau 2010-11-21 22:52 ` Dan Christensen 2010-11-24 22:11 ` Lars Magne Ingebrigtsen 2010-11-29 0:29 ` Dan Christensen 2 siblings, 1 reply; 103+ messages in thread From: Francis Moreau @ 2010-11-21 21:46 UTC (permalink / raw) To: Dan Christensen; +Cc: ding Dan Christensen <jdc@uwo.ca> writes: > Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > >> Dan Christensen <jdc@uwo.ca> writes: >> >>> Separate from that low-level cache, the sorting function could >>> probably annotate the thread tree and re-use that information >>> for all of the recursive sorts. Or use a decorate-sort-undecorate >>> pattern. >> >> It sounds possible, but peeking at the code, it doesn't seem trivial to >> do generally, since the user can compose their own composited sorting >> functions. So the cache would have to be per >> article+sorting-function... hm... > > Re: caching: > > I think caching mainly makes sense for the functions which require > traversing the thread tree: gnus-thread-sort-by-total-score and > gnus-thread-sort-by-most-recent-number and -date. Are there > any others? If these tucked away their result when called, > I imagine there would be a big savings. Depending on the shape > of the tree, this would reduce sort key computation time from > somewhere between O(n log n) and O(n^2) to O(n). Yes but I still think that the wrong approach is currently done because all this sorting is mostly done for nothing. When I'm entering in the group, all threads are hidden. And I won't look at 80% of the actual threads. So sorting them are useless. I really think that sorting should happen only when threads are expanded. And that would reduce a _lot_ the number of threads to sort. -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 21:46 ` Francis Moreau @ 2010-11-21 22:52 ` Dan Christensen 2010-11-22 7:57 ` Francis Moreau 0 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-11-21 22:52 UTC (permalink / raw) To: ding Francis Moreau <francis.moro@gmail.com> writes: > Yes but I still think that the wrong approach is currently done because > all this sorting is mostly done for nothing. > > When I'm entering in the group, all threads are hidden. And I won't look > at 80% of the actual threads. So sorting them are useless. > > I really think that sorting should happen only when threads are > expanded. And that would reduce a _lot_ the number of threads to sort. I may be wrong, but I think that rearranging the articles in the summary buffer is a little complicated in Gnus. I don't think there's any reason that the thread sorting needs to take a significant amount of time if done correctly. Moreover, I never hide threads, so I want them sorted correctly from the start. Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 22:52 ` Dan Christensen @ 2010-11-22 7:57 ` Francis Moreau 0 siblings, 0 replies; 103+ messages in thread From: Francis Moreau @ 2010-11-22 7:57 UTC (permalink / raw) To: Dan Christensen; +Cc: ding Dan Christensen <jdc@uwo.ca> writes: > Francis Moreau <francis.moro@gmail.com> writes: > >> Yes but I still think that the wrong approach is currently done because >> all this sorting is mostly done for nothing. >> >> When I'm entering in the group, all threads are hidden. And I won't look >> at 80% of the actual threads. So sorting them are useless. >> >> I really think that sorting should happen only when threads are >> expanded. And that would reduce a _lot_ the number of threads to sort. > > I may be wrong, but I think that rearranging the articles in the summary > buffer is a little complicated in Gnus. I don't think there's any > reason that the thread sorting needs to take a significant amount of > time if done correctly. > > Moreover, I never hide threads, so I want them sorted correctly from the > start. Sure but in that case all threads are expanded so also sorted. I don't want to pay the price to sort them all because I won't read most of them, and when I don't read a thread I just want it to be hidden (colapsed). So it's really: "do the job when it's really needed." -- Francis ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 20:42 ` Dan Christensen 2010-11-21 21:46 ` Francis Moreau @ 2010-11-24 22:11 ` Lars Magne Ingebrigtsen 2010-11-29 0:29 ` Dan Christensen 2 siblings, 0 replies; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-24 22:11 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > But I don't know what to do if the user implements their own sorting > function. It would be easier if the interface was changed: instead > of providing a list of comparison functions, you provide a list of > functions which compute key values which are then compared using > the default emacs order. Then we could always use DSU. But maybe > it's not worth changing the interface for this? I think it would be nicer if the interface was unchanged... > (a) Try caching, and see if it is good enough. > (b) Use DSU when only built-in sort functions are used, and fall > back to existing behaviour? (Complicates code to have a fall-back, > and also would have to assume that the user didn't redefine the > built-in functions.) > (c) Change the interface so DSU can always be used? ... so I think (a) sounds best. And, yeah, you're right, only the recursive sorting functions really need caching... -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-21 20:42 ` Dan Christensen 2010-11-21 21:46 ` Francis Moreau 2010-11-24 22:11 ` Lars Magne Ingebrigtsen @ 2010-11-29 0:29 ` Dan Christensen 2010-11-29 4:41 ` Lars Magne Ingebrigtsen 2 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-11-29 0:29 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > I think caching mainly makes sense for the functions which require > traversing the thread tree: gnus-thread-sort-by-total-score and > gnus-thread-sort-by-most-recent-number and -date. Are there > any others? If these tucked away their result when called, > I imagine there would be a big savings. Depending on the shape > of the tree, this would reduce sort key computation time from > somewhere between O(n log n) and O(n^2) to O(n). I've done some tests on the threads from a group I have with about 6500 articles. It takes about 1.6s to do the sorting step using gnus-thread-sort-by-most-recent-date, and it turns out that the majority of the time is taken by converting the dates from the string stored in the header to an emacs time, i.e. the function gnus-date-get-time (and what it calls). Even though that function caches its result, I thought that it might be worth caching the most-recent-date results too, since they currently involve traversing all sub-threads of all threads. But it turns out that that would barely make any difference, at least for groups of this size. Here's how I determined this. First, I saved the thread tree in a file, so I could do isolated tests of just the sorting. Here's the total time required: (benchmark-run 1 (gnus-sort-threads threads)) (1.638083 8 0.38992400000000016) [The first number is the time in seconds.] Now I get a fresh copy of threads, and do the latest-date computation on all of the articles, using a maptree function I wrote: (benchmark-run 1 (maptree (lambda (header) (gnus-date-get-time (mail-header-date header))) threads)) (1.31202 4 0.2062689999999998) That's 1.3 of the total 1.6 seconds right there. If I do that again, without resetting threads, the time goes to near zero, which shows how effective the caching of the date-conversion results is: (0.056382999999999996 0 0.0) And now if we do the sort after having pre-computed the date-conversion, it only takes about 0.3s! (benchmark-run 1 (gnus-sort-threads threads)) (0.299614 1 0.0529029999999997) The upshot is that about 1.3s is date conversion for each article, the rest has to do with the latest-date traversal and the actual sorting. In fact, further tests show that the latest-date sub-thread traversal barely adds anything. So there doesn't seem to be much point in adding caching for these recursively defined sort functions. To improve the speed here, what needs to be done is to improve the speed of the date conversion. Moreover, for me, other stages of preparing the summary buffer take significantly longer, so it would be worth investigating these. I should also note that I first did the timings yesterday, and got a total time of 2.5s (reproducibly). Then, 24 hours later, in the same emacs, the total time was almost double that (reproducibly). So I started a fresh emacs, and got the times above (1.6s total). Maybe this has to do with emacs memory management? In any case, the effect means that the time varies by a factor of 3 or more, so tracking this down would be another good way to speed up Gnus. Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 0:29 ` Dan Christensen @ 2010-11-29 4:41 ` Lars Magne Ingebrigtsen 2010-11-29 20:17 ` Dan Christensen 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-11-29 4:41 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > The upshot is that about 1.3s is date conversion for each article, the > rest has to do with the latest-date traversal and the actual sorting. Ah, right. Thanks for benchmarking this. Perhaps the date parsing stuff should be pushed down into the C layer? Surely one of the many libraries already included in a standard Emacs build has a date parser already included... > I should also note that I first did the timings yesterday, and got a > total time of 2.5s (reproducibly). Then, 24 hours later, in the same > emacs, the total time was almost double that (reproducibly). So I > started a fresh emacs, and got the times above (1.6s total). Yeah, benchmarking in Emacs can be kinda, er, non-reproducible. I've never understood why, though. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-29 4:41 ` Lars Magne Ingebrigtsen @ 2010-11-29 20:17 ` Dan Christensen 2010-12-05 15:00 ` Date parser in C (was: Improving Gnus speed) Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Dan Christensen @ 2010-11-29 20:17 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Dan Christensen <jdc@uwo.ca> writes: > >> The upshot is that about 1.3s is date conversion for each article, the >> rest has to do with the latest-date traversal and the actual sorting. > > Ah, right. Thanks for benchmarking this. > > Perhaps the date parsing stuff should be pushed down into the C layer? > Surely one of the many libraries already included in a standard Emacs > build has a date parser already included... Could be. The lisp parser that we fall back onto in the worst case is very complicated, handling all sorts of weird date formats. But it seems to me that it should be possible to quickly parse the most common formats, and use slower code only when completely necessary. Could it be that 99% of current e-mail/news has only a small number of possible date formats? Another idea for some backends is to rewrite the Date: header as an X-Gnus-Date: header when the article arrives. We take the time to carefully parse it on arrival, storing the result, and then avoid re-doing the slow parsing every time we want to show the article. This could work for backends like nnfolder, nnml, etc, in which Gnus can easily edit the article. But this is probably not worth it. Better to just make date parsing lightning fast if possible. Dan ^ permalink raw reply [flat|nested] 103+ messages in thread
* Date parser in C (was: Improving Gnus speed) 2010-11-29 20:17 ` Dan Christensen @ 2010-12-05 15:00 ` Lars Magne Ingebrigtsen 2010-12-05 17:27 ` Andreas Schwab 0 siblings, 1 reply; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-12-05 15:00 UTC (permalink / raw) To: ding Dan Christensen <jdc@uwo.ca> writes: > But this is probably not worth it. Better to just make date parsing > lightning fast if possible. Yes. So if somebody could kindly look into which of the gazillion libraries that Emacs 24 is linked with that has a date parser, and expose that to the Emacs Lisp layer, that'd be nice. (You'll have to argue with the emacs-devel people yourself, I'm afraid.) -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Date parser in C (was: Improving Gnus speed) 2010-12-05 15:00 ` Date parser in C (was: Improving Gnus speed) Lars Magne Ingebrigtsen @ 2010-12-05 17:27 ` Andreas Schwab 2010-12-05 17:38 ` Date parser in C Lars Magne Ingebrigtsen 0 siblings, 1 reply; 103+ messages in thread From: Andreas Schwab @ 2010-12-05 17:27 UTC (permalink / raw) To: ding Lars Magne Ingebrigtsen <larsi@gnus.org> writes: > Yes. So if somebody could kindly look into which of the gazillion > libraries that Emacs 24 is linked with that has a date parser, and > expose that to the Emacs Lisp layer, that'd be nice. parse-datetime exists in gnulib. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Date parser in C 2010-12-05 17:27 ` Andreas Schwab @ 2010-12-05 17:38 ` Lars Magne Ingebrigtsen 0 siblings, 0 replies; 103+ messages in thread From: Lars Magne Ingebrigtsen @ 2010-12-05 17:38 UTC (permalink / raw) To: ding Andreas Schwab <schwab@linux-m68k.org> writes: >> Yes. So if somebody could kindly look into which of the gazillion >> libraries that Emacs 24 is linked with that has a date parser, and >> expose that to the Emacs Lisp layer, that'd be nice. > > parse-datetime exists in gnulib. So whoever volunteers to update strftime and stuff from gnulib could include parse-datetime, too. :-) -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 12:35 Improving Gnus speed Francis Moreau 2010-11-09 13:45 ` Didier Verna 2010-11-09 18:55 ` Lars Magne Ingebrigtsen @ 2010-11-09 21:47 ` Ted Zlatanov 2010-11-09 22:55 ` Steinar Bang 2010-11-10 4:27 ` Eden Cardim 3 siblings, 1 reply; 103+ messages in thread From: Ted Zlatanov @ 2010-11-09 21:47 UTC (permalink / raw) To: ding On Tue, 09 Nov 2010 13:35:15 +0100 Francis Moreau <francis.moro@gmail.com> wrote: FM> Out off topic: Why does Gnus development use a mailing list ? That's FM> pretty paradoxical, isn't it ? I read it as a newsgroup on GMane. But it's not a paradox in any case. Many corporate firewalls block NNTP, for instance. Ted ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 21:47 ` Improving Gnus speed Ted Zlatanov @ 2010-11-09 22:55 ` Steinar Bang 0 siblings, 0 replies; 103+ messages in thread From: Steinar Bang @ 2010-11-09 22:55 UTC (permalink / raw) To: ding >>>>> Ted Zlatanov <tzz@lifelogs.com>: > I read it as a newsgroup on GMane. But it's not a paradox in any > case. Many corporate firewalls block NNTP, for instance. Actually I've used the existence of gmane to argue for opening corporate firewalls for gmane, twice. My argument was that if we were going to use open source or free software, such as, news.gmane.org was neccessary as a source for support. ^ permalink raw reply [flat|nested] 103+ messages in thread
* Re: Improving Gnus speed 2010-11-09 12:35 Improving Gnus speed Francis Moreau ` (2 preceding siblings ...) 2010-11-09 21:47 ` Improving Gnus speed Ted Zlatanov @ 2010-11-10 4:27 ` Eden Cardim 3 siblings, 0 replies; 103+ messages in thread From: Eden Cardim @ 2010-11-10 4:27 UTC (permalink / raw) To: ding >>>>> "Francis" == Francis Moreau <francis.moro@gmail.com> writes: Francis> Out off topic: Why does Gnus development use a mailing list ? That's Francis> pretty paradoxical, isn't it ? As a beginning gnus user, I wouldn't be able to reply as to why it happens for gnus in particular, but I thought it would be worth the mention that this "paradox" actually makes sense in most cases. It's called bootstrapping, the reasoning behind it is that the building of a new tool is often motivated by limitations in all the alternative tools, which implies that the best tool for the development project is the very thing you're developing. Another benefit is that the development effort also doubles up as a sort of continuous alpha test. There are quite a few notable cases: - most linux kernel development happens on a running linux kernel - gcc is (in most cases) compiled by gcc - git development uses git for revision control - subversion too - perl's build/test process is controlled by a perl script - trac development uses trac as SCM - bugzilla too just my $0.02 -- Eden Cardim Software Engineer +55 73 9986-3963 edencardim.com ^ permalink raw reply [flat|nested] 103+ messages in thread
end of thread, other threads:[~2010-12-05 18:46 UTC | newest] Thread overview: 103+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-11-09 12:35 Improving Gnus speed Francis Moreau 2010-11-09 13:45 ` Didier Verna 2010-11-09 13:55 ` Francis Moreau 2010-11-09 14:00 ` Knut Anders Hatlen 2010-11-09 14:22 ` Steinar Bang 2010-11-09 14:29 ` Francis Moreau 2010-11-09 14:49 ` Richard Riley 2010-11-09 16:37 ` Didier Verna 2010-11-09 16:51 ` Richard Riley 2010-11-09 16:56 ` Steinar Bang 2010-11-09 17:12 ` Richard Riley 2010-11-09 17:48 ` Steinar Bang 2010-11-09 18:59 ` Adam Sjøgren 2010-11-09 19:17 ` Steinar Bang 2010-11-09 19:06 ` Richard Riley 2010-11-09 14:49 ` Didier Verna 2010-11-09 15:27 ` Richard Riley 2010-11-09 15:42 ` Francis Moreau 2010-11-09 16:35 ` Didier Verna 2010-11-09 16:49 ` Richard Riley 2010-11-09 19:52 ` Francis Moreau 2010-11-09 20:48 ` Richard Riley 2010-11-09 15:04 ` Adam Sjøgren 2010-11-09 18:55 ` Lars Magne Ingebrigtsen 2010-11-09 19:58 ` Francis Moreau 2010-11-09 21:03 ` Andreas Schwab 2010-11-09 21:49 ` Ted Zlatanov 2010-11-09 22:45 ` Richard Riley 2010-11-10 8:49 ` Francis Moreau 2010-11-10 10:31 ` Francis Moreau 2010-11-10 9:39 ` Julien Danjou 2010-11-10 13:26 ` Ted Zlatanov 2010-11-10 13:28 ` Julien Danjou 2010-11-10 14:12 ` Richard Riley 2010-11-10 17:40 ` Andreas Schwab 2010-11-10 13:38 ` Francis Moreau 2010-11-09 21:27 ` Lars Magne Ingebrigtsen 2010-11-10 14:16 ` Francis Moreau 2010-11-10 18:20 ` Lars Magne Ingebrigtsen 2010-11-11 8:14 ` Francis Moreau 2010-11-11 18:34 ` Lars Magne Ingebrigtsen 2010-11-11 19:54 ` Francis Moreau 2010-11-14 16:10 ` Lars Magne Ingebrigtsen 2010-11-15 9:40 ` Francis Moreau 2010-11-21 6:02 ` Lars Magne Ingebrigtsen 2010-11-21 13:54 ` Francis Moreau 2010-11-21 18:17 ` Lars Magne Ingebrigtsen 2010-11-21 19:37 ` Francis Moreau 2010-11-24 22:09 ` Lars Magne Ingebrigtsen 2010-11-25 7:35 ` Francis Moreau 2010-11-12 11:11 ` Francis Moreau 2010-11-14 16:11 ` Lars Magne Ingebrigtsen 2010-11-15 9:07 ` Francis Moreau 2010-11-21 5:44 ` Lars Magne Ingebrigtsen 2010-11-21 14:07 ` Francis Moreau 2010-11-10 18:21 ` Lars Magne Ingebrigtsen 2010-11-11 8:22 ` Francis Moreau 2010-11-11 18:33 ` Lars Magne Ingebrigtsen 2010-11-11 19:56 ` Francis Moreau 2010-11-14 16:10 ` Lars Magne Ingebrigtsen 2010-11-12 18:55 ` Dan Christensen 2010-11-12 20:07 ` Francis Moreau 2010-11-12 21:38 ` Dan Christensen 2010-11-13 20:46 ` Francis Moreau 2010-11-14 9:32 ` Francis Moreau 2010-11-29 0:40 ` Dan Christensen 2010-11-29 4:47 ` Lars Magne Ingebrigtsen 2010-11-29 6:01 ` Miles Bader 2010-11-29 6:18 ` Lars Magne Ingebrigtsen 2010-11-29 6:36 ` Lars Magne Ingebrigtsen 2010-11-29 6:40 ` Daniel Pittman 2010-11-29 8:27 ` Francis Moreau 2010-11-29 19:30 ` Dan Christensen 2010-11-29 20:10 ` Francis Moreau 2010-11-29 20:23 ` Dan Christensen 2010-11-29 20:56 ` Francis Moreau 2010-11-29 21:30 ` Dan Christensen 2010-12-05 15:01 ` Lars Magne Ingebrigtsen 2010-12-05 18:05 ` Dan Christensen 2010-12-05 18:46 ` Lars Magne Ingebrigtsen 2010-11-14 16:13 ` Lars Magne Ingebrigtsen 2010-11-15 9:03 ` Francis Moreau 2010-11-21 5:43 ` Lars Magne Ingebrigtsen 2010-11-21 14:39 ` Francis Moreau 2010-11-14 16:14 ` Lars Magne Ingebrigtsen 2010-11-14 18:10 ` Dan Christensen 2010-11-14 18:26 ` Lars Magne Ingebrigtsen 2010-11-14 21:27 ` Dan Christensen 2010-11-21 5:42 ` Lars Magne Ingebrigtsen 2010-11-21 20:42 ` Dan Christensen 2010-11-21 21:46 ` Francis Moreau 2010-11-21 22:52 ` Dan Christensen 2010-11-22 7:57 ` Francis Moreau 2010-11-24 22:11 ` Lars Magne Ingebrigtsen 2010-11-29 0:29 ` Dan Christensen 2010-11-29 4:41 ` Lars Magne Ingebrigtsen 2010-11-29 20:17 ` Dan Christensen 2010-12-05 15:00 ` Date parser in C (was: Improving Gnus speed) Lars Magne Ingebrigtsen 2010-12-05 17:27 ` Andreas Schwab 2010-12-05 17:38 ` Date parser in C Lars Magne Ingebrigtsen 2010-11-09 21:47 ` Improving Gnus speed Ted Zlatanov 2010-11-09 22:55 ` Steinar Bang 2010-11-10 4:27 ` Eden Cardim
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).