Benchmarking Gnus

Benchmarking Gnus

[Lars Magne Ingebrigtsen]

Inspired by the recent discussion on news.software.readers, I've been
benchmarking group entry a bit.  I've used ELP, and it led me to find
some weaknesses in the manner format specs are used.  (The
pre-compiled functions were supposed to be used for default values,
but they weren't.)  After fixing that, and inlining some other
functions, I got group entry of a 1K group down from 2 seconds to 1.5
seconds.  (Yes, this is a new and spiffy machine.  :-)

There are now three things that take about the same time when entering
a group, and account for 95% of the time spent:

1) parsing NOV headers (this has gotten slower due to MIME stuff, I'm
   guessing)
2) scoring
3) creating and outputting the threads

If someone wants to look closer at this, then I'm all for it.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Benchmarking Gnus

[Russ Allbery]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Inspired by the recent discussion on news.software.readers, I've been
> benchmarking group entry a bit.  I've used ELP, and it led me to find
> some weaknesses in the manner format specs are used.  (The pre-compiled
> functions were supposed to be used for default values, but they
> weren't.)  After fixing that, and inlining some other functions, I got
> group entry of a 1K group down from 2 seconds to 1.5 seconds.  (Yes,
> this is a new and spiffy machine.  :-)

Any luck on the speed problems when pressing "g"?  I'm not sure there's
any other good profiling data that I can provide....

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: Benchmarking Gnus

[Lars Magne Ingebrigtsen]

Russ Allbery <rra@stanford.edu> writes:

> Any luck on the speed problems when pressing "g"?  I'm not sure there's
> any other good profiling data that I can provide....

I haven't looked into that, but, yes, it seems to have gotten slower
lately, too...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Benchmarking Gnus

[Jesper Harder]

larsi@gnus.org (Lars Magne Ingebrigtsen) writes:

> Inspired by the recent discussion on news.software.readers, I've been
> benchmarking group entry a bit. [..]  If someone wants to look closer
> at this, then I'm all for it.  :-)

Oort seems be getting getting just as fast as 5.8.8 now.  I tried
entering a 10K group with

  Emacs 21 + Oort Gnus, 
  Emacs 21 + Gnus 5.9, 
  Emacs 20.7 + Gnus 5.8.8.  

Oort was nearly twice as fast as 5.9 and around 10% quicker than 5.8.8

Re: Benchmarking Gnus

[Lars Magne Ingebrigtsen]

Jesper Harder <harder@ifa.au.dk> writes:

> Oort was nearly twice as fast as 5.9 and around 10% quicker than 5.8.8

Cool!

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Benchmarking Gnus

[Simon Josefsson]

On Wed, 16 Jan 2002, Lars Magne Ingebrigtsen wrote:

> There are now three things that take about the same time when entering
> a group, and account for 95% of the time spent:
> 
> 1) parsing NOV headers (this has gotten slower due to MIME stuff, I'm
>    guessing)
> 2) scoring
> 3) creating and outputting the threads

A short term solution would be to cache things.  Shenghuo's cache and 
agent unification probably makes this easier and more reliable.

A long term solution would be to redesign this stuff and make it run 
asynchronously.  The summary buffer would appear immediately containing as 
many empty lines as there are messages that should be shown, and then 
things are populated in the background.  This might even give us some more 
flexibility, like allowing backends to perform threading (IMAP supports 
this).

But long term solutions should perhaps be targetted at Guile..  gimme a 
guile emacs! :-)

Re: Benchmarking Gnus

[Josh Huber]

Simon Josefsson <jas@extundo.com> writes:

> A long term solution would be to redesign this stuff and make it run
> asynchronously.  The summary buffer would appear immediately
> containing as many empty lines as there are messages that should be
> shown, and then things are populated in the background.  This might
> even give us some more flexibility, like allowing backends to
> perform threading (IMAP supports this).

Er, doesn't this require some support from emacs for threads?  Isn't
there none?

> But long term solutions should perhaps be targetted at Guile..  gimme a 
> guile emacs! :-)

Hehe, yeah, but that's a while away, I think :)

-- 
Josh Huber

Re: Benchmarking Gnus

[Karl Kleinpaste]

Josh Huber <huber@alum.wpi.edu> writes:
> Er, doesn't this require some support from emacs for threads?  Isn't
> there none?

Not exactly.  See function gnus-date-start-timer and
date-treat-date-lapsed, to say nothing of gnus-asynchronous.

Re: Benchmarking Gnus

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> A long term solution would be to redesign this stuff and make it run
> asynchronously. 

Yes! Yes! Yes!

> The summary buffer would appear immediately containing as many empty
> lines as there are messages that should be shown, and then things
> are populated in the background.  This might even give us some more
> flexibility, like allowing backends to perform threading (IMAP
> supports this).

Sounds good to me!

> But long term solutions should perhaps be targetted at Guile..
> gimme a guile emacs! :-)

Aww... not _that_ long term, I hope!

Re: Benchmarking Gnus

[Daniel Pittman]

On Wed, 16 Jan 2002, Simon Josefsson wrote:
> On Wed, 16 Jan 2002, Lars Magne Ingebrigtsen wrote:
> 
>> There are now three things that take about the same time when
>> entering a group, and account for 95% of the time spent:

[...]

> A long term solution would be to redesign this stuff and make it run
> asynchronously. The summary buffer would appear immediately containing
> as many empty lines as there are messages that should be shown, and
> then things are populated in the background. This might even give us
> some more flexibility, like allowing backends to perform threading
> (IMAP supports this).
> 
> But long term solutions should perhaps be targetted at Guile.. gimme a
> guile emacs! :-)

The day they write it, and it's shown in beta, is the day to start
writing code for it, I think.

Because, you know, that means I never have to deal with a Scheme based
Emacs at all. ;)

        Daniel

-- 
In the present-day reconstruction of physics, fragments of the Newtonian
concepts are stubbornly retained. The result is to reduce modern physics to a
sort of mystic chant over an unintelligible universe.
        -- Alfred North Whitehead, _Modes of Thought_

Re: Benchmarking Gnus

[Ami Fischman]

Not quite a "benchmarking" issue, more concerned w/ speed/effeciency.  I
have several nntp servers I use (main one + several "vendor" servers for
specific programs, like gnus).  I noticed that since adding virtual groups
(which contain groups from multiple servers) the same server will be opened
several times in one 'g' operation.

For example, here is the *Messages* log:

Checking new news...
Opening nnvirtual server on ^nntp\+news\.ISP\.net:sci\.electronics\..*$...done
Opening nntp server on news.ISP.net...done
Opening nntp server on news.opera.com...done
Opening nntp server on news.gnus.org...done
Opening nnvirtual server on ^nntp\+news\.ISP\.net:comp\.dcom\.xdsl\|nntp\+news\.ISP\.net:comp\.dcom\.telecom.tech$...done
Opening nntp server on news.ISP.net...done
Opening nntp server on news.vmware.com...done
nnml: Reading incoming mail from file...
nnml: Reading incoming mail (no new mail)...done
Opening nnvirtual server on ^nntp\+news\.ISP\.net:gnu\.emacs\.gnus\|nntp\+news\.gnus\.org:gnus\.ding$...done
Opening nntp server on news.ISP.net...done
Opening nntp server on news.gnus.org...done
Opening nntp server on news.ISP.net...done
Checking new news...done

You'll note that news.ISP.net is opened several times, as is news.gnus.org.
This wouldn't bother me, except the "Opening nntp server on..." message
always takes a few seconds to scroll by, making me think that opening the
extra connection is taking time that needn't be taken (for instance, if the
first connection were kept open, or else, if _before_ checking new news,
gnus figured out all the info it needs to ask of each server, and do it at
the same time).

Any idea how to avoid this?
-- 
  Ami Fischman
  usenet@fischman.org

Re: Benchmarking Gnus

[Lars Magne Ingebrigtsen]

Ami Fischman <usenet@fischman.org> writes:

> Not quite a "benchmarking" issue, more concerned w/ speed/effeciency.  I
> have several nntp servers I use (main one + several "vendor" servers for
> specific programs, like gnus).  I noticed that since adding virtual groups
> (which contain groups from multiple servers) the same server will be opened
> several times in one 'g' operation.

[...]

> Opening nntp server on news.ISP.net...done

[...]

> Opening nnvirtual server on ^nntp\+news\.ISP\.net:comp\.dcom\.xdsl\|nntp\+news\.ISP\.net:comp\.dcom\.telecom.tech$...done
> Opening nntp server on news.ISP.net...done

[...]

> Opening nnvirtual server on ^nntp\+news\.ISP\.net:gnu\.emacs\.gnus\|nntp\+news\.gnus\.org:gnus\.ding$...done
> Opening nntp server on news.ISP.net...done

[...]

> You'll note that news.ISP.net is opened several times, as is news.gnus.org.
> This wouldn't bother me, except the "Opening nntp server on..." message
> always takes a few seconds to scroll by, making me think that opening the
> extra connection is taking time that needn't be taken (for instance, if the
> first connection were kept open, or else, if _before_ checking new news,
> gnus figured out all the info it needs to ask of each server, and do it at
> the same time).

This definitely sounds like a bug.  Gnus is supposed to keep the
connections open and re-use them as needed.  The most likely reason I
see for this not happening is that Gnus isn't able to recognize the
groups as coming from the same server.

What is the `G e' output on two of the component groups that are
included in the two different nnvirtual groups?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Benchmarking Gnus

[Ami Fischman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

[...]

> This definitely sounds like a bug.  Gnus is supposed to keep the
> connections open and re-use them as needed.  The most likely reason I
> see for this not happening is that Gnus isn't able to recognize the
> groups as coming from the same server.
>
> What is the `G e' output on two of the component groups that are
> included in the two different nnvirtual groups?

They all just say:

"nntp:news.ISP.net"

(same thing in each of the groups coming from this server).

Then the virtual (say for my all-gnus-all-the-time group) says:

(nnvirtual "^nntp\\+news\\.ISP\\.net:gnu\\.emacs\\.gnus\\|nntp\\+news\\.gnus\\.org:gnus\\.ding$")

(and the other virtuals say similar things).

Ideas?

-- 
  Ami Fischman
  usenet@fischman.org

Re: Benchmarking Gnus

[Lars Magne Ingebrigtsen]

Ami Fischman <usenet@fischman.org> writes:

> They all just say:
>
> "nntp:news.ISP.net"
>
> (same thing in each of the groups coming from this server).

Try evaluating the following:

(gnus-server-equal
 (gnus-find-method-for-group "comp.lang.lisp")
 (gnus-find-method-for-group "comp.emacs"))

(Use the group names that you have in the virtual group instead.)

If this returns t, then it shouldn't open more than one connection.
In theory.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Benchmarking Gnus

[Ami Fischman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> (gnus-server-equal
>  (gnus-find-method-for-group "comp.lang.lisp")
>  (gnus-find-method-for-group "comp.emacs"))
> (Use the group names that you have in the virtual group instead.)
> If this returns t, then it shouldn't open more than one connection.
> In theory.  :-)

Theory schmeory.  Indeed, it returns t...

-- 
  Ami Fischman
  usenet@fischman.org

Re: Benchmarking Gnus

[Lars Magne Ingebrigtsen]

Ami Fischman <usenet@fischman.org> writes:

> Theory schmeory.  Indeed, it returns t...

Try `M-x debug-on-entry RET nntp-open-connection RET' and see what the
parameters to `nntp-open-server' are in each of the nnvirtual-related
openings.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Benchmarking Gnus

[Ami Fischman]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> Try `M-x debug-on-entry RET nntp-open-connection RET' and see what the
> parameters to `nntp-open-server' are in each of the nnvirtual-related
> openings.

The servers (and they are spelled identically).  

Although, since at least this morning's CVS update, the subsequent "opening"
messages are very quick, so either my net cnxns are less loaded b/c it's
sunday, or it really is keeping the connections open.  Wait on further
debugging on this for tuesday (monday is a holiday in the US) and I'll see
if the delay is back.

-- 
  Ami Fischman
  usenet@fischman.org

Re: Benchmarking Gnus

[Jesper Harder]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

larsi@gnus.org (Lars Magne Ingebrigtsen) writes:

> There are now three things that take about the same time when entering
> a group, and account for 95% of the time spent:
>
> 1) parsing NOV headers (this has gotten slower due to MIME stuff, I'm
>    guessing)
> 2) scoring
> 3) creating and outputting the threads
>
> If someone wants to look closer at this, then I'm all for it.  :-)

Here's a small tweak, which speeds up threading a bit (around 5-10%):
`gnus-parent-id' is usually called without the optional argument -- in
that case we only want the last msg.id in References, so we don't really
need to call `gnus-split-references' to parse *all* the references and
build a list of them.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: gnus-util.el.diff --]
[-- Type: text/x-patch, Size: 679 bytes --]

--- ../../gnus/lisp/gnus-util.el	Wed Jan 16 05:14:26 2002
+++ gnus-util.el	Fri Jan 18 22:58:21 2002
@@ -479,10 +479,13 @@
 If N, return the Nth ancestor instead."
   (when (and references
 	     (not (zerop (length references))))
-    (let ((ids (inline (gnus-split-references references))))
-      (while (nthcdr (or n 1) ids)
-	(setq ids (cdr ids)))
-      (car ids))))
+    (if n
+	(let ((ids (inline (gnus-split-references references))))
+	  (while (nthcdr n ids)
+	    (setq ids (cdr ids)))
+	  (car ids))
+      (when (string-match "<[^> \t]+>\\'" references)
+	(match-string 0 references)))))
 
 (defun gnus-buffer-live-p (buffer)
   "Say whether BUFFER is alive or not."

Re: Benchmarking Gnus

[Simon Josefsson]

Jesper Harder <harder@ifa.au.dk> writes:

> larsi@gnus.org (Lars Magne Ingebrigtsen) writes:
>
>> There are now three things that take about the same time when 
>> entering a group, and account for 95% of the time spent:
>>
>> 1) parsing NOV headers (this has gotten slower due to MIME 
>>    stuff, I'm guessing) 2) scoring 3) creating and outputting 
>>    the threads
>>
>> If someone wants to look closer at this, then I'm all for it. 
   :-)
>
> Here's a small tweak, which speeds up threading a bit (around 
> 5-10%): `gnus-parent-id' is usually called without the optional 
> argument -- in that case we only want the last msg.id in 
> References, so we don't really need to call 
> `gnus-split-references' to parse *all* the references and build 
> a list of them.

I committed it, thanks.

Benchmarking Gnus

[Lars Magne Ingebrigtsen]

When entering large groups, Gnus generates quite a lot of garbage,
which results in time being spent collecting that garbage.  It would
be interesting to see where the garbage is being generated.

Wasn't there some version of, er, some package that would keep track
of how much memory each function uses?  Perhaps ELP, even?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Benchmarking Gnus

[IPmonger]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>    When entering large groups, Gnus generates quite a lot of garbage,
>  which results in time being spent collecting that garbage.  It would
>  be interesting to see where the garbage is being generated. 
>
>    Wasn't there some version of, er, some package that would keep
>  track of how much memory each function uses?  Perhaps ELP, even?

    How do you engage it?

    My largest group is the mailbox I use for the linux kernel mailing
  list.  Of course, it's now very sparse since I don't keep everything,
  but the article #s exceed 42000...  Is that big enough?

-IPmonger
-- 
------------------
IPmonger
ipmonger@delamancha.org
CCIE #8338

Re: Benchmarking Gnus

[Lars Magne Ingebrigtsen]

IPmonger <ipmonger@delamancha.org> writes:

>>    Wasn't there some version of, er, some package that would keep
>>  track of how much memory each function uses?  Perhaps ELP, even?

I found it -- it was a version of ELP I had hacked to keep track of
all that stuff.  I even had to hack the Emacs internals to get at the
info, and those internals were still there.  :-)

>     My largest group is the mailbox I use for the linux kernel mailing
>   list.  Of course, it's now very sparse since I don't keep everything,
>   but the article #s exceed 42000...  Is that big enough?

That's plenty big.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Benchmarking Gnus

[IPmonger]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> IPmonger <ipmonger@delamancha.org> writes:
>
>>>    Wasn't there some version of, er, some package that would keep
>>>  track of how much memory each function uses?  Perhaps ELP, even?
>
> I found it -- it was a version of ELP I had hacked to keep track of
> all that stuff.  I even had to hack the Emacs internals to get at the
> info, and those internals were still there.  :-)
>
>>     My largest group is the mailbox I use for the linux kernel mailing
>>   list.  Of course, it's now very sparse since I don't keep everything,
>>   but the article #s exceed 42000...  Is that big enough?
>
> That's plenty big.

    I'm ready to profile, but when I interactively try
  M-x elp-instrument-package RET gnus RET  I get errors indicating that
  it can't instrument autoloaded functions.

-IPmonger
-- 
------------------
IPmonger
ipmonger@delamancha.org
CCIE #8338

Re: Benchmarking Gnus

[Kai Großjohann]

IPmonger <ipmonger@delamancha.org> writes:

>     I'm ready to profile, but when I interactively try
>   M-x elp-instrument-package RET gnus RET  I get errors indicating that
>   it can't instrument autoloaded functions.

Load Gnus first.  For example, start Gnus, then enter a group, then
exit the group, then M-x elp-instrument-package RET, then enter the
group a second time for profiling entering groups.

kai
-- 
~/.signature is: umop 3p!sdn    (Frank Nobis)

Re: Benchmarking Gnus

[IPmonger]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> IPmonger <ipmonger@delamancha.org> writes:
>
>>    I'm ready to profile, but when I interactively try M-x
>>  elp-instrument-package RET gnus RET  I get errors indicating that it
>>  can't instrument autoloaded functions. 
>
>    Load Gnus first.  For example, start Gnus, then enter a group, then
>  exit the group, then M-x elp-instrument-package RET, then enter the
>  group a second time for profiling entering groups. 

  Same result, although the complaint is about a different function.
  :-)

  Here's the results of M-x elp-results RET:

  Function Name                         Call Count  Elapsed Time  Average Time
  ====================================  ==========  ============  ============
  gnus-short-group-name                 1           0.0003010000  0.0003010000


    Does that seem close to being correct?  Admittedly, it didn't take
  very long to enter the group, but I suppose that I expected more than
  1 function call to be profiled.

    Someone else suggested that I might need to upgrade my XEmacs from
  21.1 to a version closer to what they run (21.5), so that's what I'll
  try next.

-IPmonger
-- 
------------------
IPmonger
ipmonger@delamancha.org
CCIE #8338

Re: Benchmarking Gnus

[Kai Großjohann]

IPmonger <ipmonger@delamancha.org> writes:

> Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:
>
>> IPmonger <ipmonger@delamancha.org> writes:
>>
>>>    I'm ready to profile, but when I interactively try M-x
>>>  elp-instrument-package RET gnus RET  I get errors indicating that it
>>>  can't instrument autoloaded functions. 
>>
>>    Load Gnus first.  For example, start Gnus, then enter a group, then
>>  exit the group, then M-x elp-instrument-package RET, then enter the
>>  group a second time for profiling entering groups. 
>
>   Same result, although the complaint is about a different function.
>   :-)

Gah?

Well, go through the list of *.el files that are part of Gnus and do
M-x load-library RET on all of them...

(Or use dired and the `L' key.)

kai
-- 
~/.signature is: umop 3p!sdn    (Frank Nobis)

Re: Benchmarking Gnus

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Well, go through the list of *.el files that are part of Gnus and do
> M-x load-library RET on all of them...

I think more recent XEmacsen has this elp bug fixed...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Benchmarking Gnus

[Josh Huber]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I think more recent XEmacsen has this elp bug fixed...

Yes, it's at least fixed in 21.5...

-- 
Josh Huber

Re: Benchmarking Gnus

[IPmonger]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]


    Ok, I updated to a newer XEmacs (21.5.4) and ran ELP on gnus with my
  44000+ nnml folder for the Linux kernel, but I need some help
  understanding what these results indicate.

    To generate them, I:
     started XEmacs
     instrumented gnus (elp-instrument-package)
     loaded gnus (M-x gnus)
     hid empty topics
     listed all topics
     entered linux-kernel nnml folder
     exited back to *Group* buffer
    
   Are profiling results cumulative for an Emacs/XEmacs session?

-IPmonger
-- 
------------------
IPmonger
ipmonger@delamancha.org
CCIE #8338



[-- Attachment #2: ELP Profiling Results: Gnus + nnml --]
[-- Type: text/plain, Size: 2616 bytes --]

Function Name                                           Call Count  Elapsed Time  Average Time
======================================================  ==========  ============  ============
gnus-run-hooks                                          1147        1.542707      0.0013449930
gnus-remassoc                                           18861       1.2365829999  6.556...e-05
gnus-update-alist-soft                                  7757        1.1905260000  0.0001534776
gnus-put-text-property-excluding-characters-with-faces  1119        0.6554359999  0.0005857336
gnus-byte-compile                                       1           0.380348      0.380348
gnus-define-keys-1                                      47          0.285708      0.0060788936
gnus-extract-address-components                         1632        0.2381220000  0.0001459080
gnus-split-references                                   1346        0.112672      8.370...e-05
gnus-message                                            25          0.0536890000  0.00214756
gnus-point-at-eol                                       4381        0.0289090000  6.598...e-06
gnus-prin1                                              2           0.017061      0.0085305000
gnus-set-work-buffer                                    48          0.0059399999  0.00012375
gnus-point-at-bol                                       661         0.0057210000  8.655...e-06
gnus-write-buffer                                       1           0.005479      0.005479
gnus-make-hashtable                                     11          0.003488      0.0003170909
gnus-prin1-to-string                                    1           0.003236      0.003236
gnus-goto-colon                                         9           0.00223       0.0002477777
gnus-newsgroup-savable-name                             4           0.000636      0.000159
gnus-make-directory                                     4           0.000534      0.0001335
gnus-turn-off-edit-menu                                 2           0.000359      0.0001795
gnus-buffer-live-p                                      13          0.0002739999  2.107...e-05
gnus-create-hash-size                                   9           0.000148      1.644...e-05
gnus-alive-p                                            1           0.000122      0.000122
gnus-make-sort-function                                 1           5e-05         5e-05
gnus-delete-if                                          3           4.9e-05       1.633...e-05
gnus-delete-alist                                       1           4e-05         4e-05

Re: Benchmarking Gnus

[IPmonger]

IPmonger <ipmonger@delamancha.org> writes:

>    Are profiling results cumulative for an Emacs/XEmacs session?

    Please ignore this stupid question.  I see elp-reset-all. :-)

-IPmonger
------------------
IPmonger
ipmonger@delamancha.org
CCIE #8338

Re: Benchmarking Gnus

[Daniel Pittman]

On Mon, 04 Feb 2002, ipmonger@delamancha.org wrote:
> Ok, I updated to a newer XEmacs (21.5.4) and ran ELP on gnus with my
> 44000+ nnml folder for the Linux kernel, but I need some help
> understanding what these results indicate.

[...]

>    Are profiling results cumulative for an Emacs/XEmacs session?

Maybe. If you set `elp-reset-after-results' to non-nil, getting the
results discards them at the same time.

        Daniel

-- 
Among persons under 30, the inability to program is regarded as a form
of illiteracy.          -- James Martin 
(Comment predicting trends of the late 1990s, Made in the early 70's)

Re: Benchmarking Gnus

[Wes Hardaker]

>>>>> On Sun, 27 Jan 2002 01:30:29 +0100, Lars Magne Ingebrigtsen <larsi@gnus.org> said:

Lars> When entering large groups, Gnus generates quite a lot of
Lars> garbage, which results in time being spent collecting that
Lars> garbage.  It would be interesting to see where the garbage is
Lars> being generated.

Actually, I've recently discovered that I have a different profiling
problem I need to solve:

hardaker  3772  6.1 30.9 87020 78868 pts/2   S    13:15   2:43 xemacs
                         ^^^^^ ^^^^^

I really don't like a 80 meg resident application running on my
machine.  I think this was a recent change of some kind and I'm not
sure what's done it.  I'm pretty sure it was a recent update of Gnus.
I keep meaning to run xemacs for a while without running gnus to see
if Gnus really is the responsible party for the memory leak, but,
well, I just can't seem to run xemacs that long without starting up
Gnus...

Anyone else see problems like this?  cvs update from ~ Jan 25th.

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: Benchmarking Gnus

[IPmonger]

Wes Hardaker <wes@hardakers.net> writes:

>    Actually, I've recently discovered that I have a different
>  profiling problem I need to solve:
>
> hardaker  3772  6.1 30.9 87020 78868 pts/2   S    13:15   2:43 xemacs
>                          ^^^^^ ^^^^^
>
> Anyone else see problems like this?  cvs update from ~ Jan 25th.

    Um, I'm not seeing that problem.  Admittedly, I'm running gnus from
  ~ Jan 28th w/XEmacs 21.1 on RedHat 7.2 i686 Linux:

   2651 ipmonger  14   0 20180  19M 12248 S    11.2 10.7  49:49 xemacs

    If gnus is responsible, then it's either not nnml (which is the only
  back-end I use) or it's been corrected...

-IPmonger
-- 
------------------
IPmonger
ipmonger@delamancha.org
CCIE #8338

Re: Benchmarking Gnus

[Wes Hardaker]

>>>>> On Wed, 30 Jan 2002 17:09:56 -0500, IPmonger <ipmonger@delamancha.org> said:

ipmonger> Wes Hardaker <wes@hardakers.net> writes:
>> Actually, I've recently discovered that I have a different
>> profiling problem I need to solve:
>> 
>> hardaker  3772  6.1 30.9 87020 78868 pts/2   S    13:15   2:43 xemacs
>> ^^^^^ ^^^^^
>> 
>> Anyone else see problems like this?  cvs update from ~ Jan 25th.

ipmonger> Um, I'm not seeing that problem.  Admittedly, I'm running gnus from
ipmonger> ~ Jan 28th w/XEmacs 21.1 on RedHat 7.2 i686 Linux:

ipmonger> 2651 ipmonger  14   0 20180  19M 12248 S    11.2 10.7  49:49 xemacs

ipmonger> If gnus is responsible, then it's either not nnml (which is the only
ipmonger> back-end I use) or it's been corrected...

Gnus is definitely responsible.  I finally did run it for a while
without gnus and didn't get any major memory leaks.  Immediately after
starting gnus and entering a few groups, I'm back at huge memory
usages.  sigh...

However, these are imap folders.

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: Benchmarking Gnus

[ShengHuo ZHU]

Wes Hardaker <wes@hardakers.net> writes:

>>>>>> On Wed, 30 Jan 2002 17:09:56 -0500, IPmonger <ipmonger@delamancha.org> said:
>
> ipmonger> Wes Hardaker <wes@hardakers.net> writes:
>>> Actually, I've recently discovered that I have a different
>>> profiling problem I need to solve:
>>> 
>>> hardaker  3772  6.1 30.9 87020 78868 pts/2   S    13:15   2:43 xemacs
>>> ^^^^^ ^^^^^
>>> 
>>> Anyone else see problems like this?  cvs update from ~ Jan 25th.
>
> ipmonger> Um, I'm not seeing that problem.  Admittedly, I'm running gnus from
> ipmonger> ~ Jan 28th w/XEmacs 21.1 on RedHat 7.2 i686 Linux:
>
> ipmonger> 2651 ipmonger  14   0 20180  19M 12248 S    11.2 10.7  49:49 xemacs
>
> ipmonger> If gnus is responsible, then it's either not nnml (which
> ipmonger> is the only back-end I use) or it's been corrected...
>
> Gnus is definitely responsible.  I finally did run it for a while
> without gnus and didn't get any major memory leaks.  Immediately after
> starting gnus and entering a few groups, I'm back at huge memory
> usages.  sigh...

Mine is 45.5 meg right now.  It was reported that the image cache used
a lot of memory.  In Emacs, setting a small image-cache-eviction-delay
or running clear-image-cache may help a little. But I am not sure
about XEmacs.

Anyway, after adding 256 meg memory, I don't care much about that now.

ShengHuo

Re: Benchmarking Gnus

[Wes Hardaker]

>>>>> On Wed, 30 Jan 2002 22:22:14 -0500, ShengHuo ZHU <zsh@cs.rochester.edu> said:

ZSH> Mine is 45.5 meg right now.  It was reported that the image cache used
ZSH> a lot of memory.  In Emacs, setting a small image-cache-eviction-delay
ZSH> or running clear-image-cache may help a little. But I am not sure
ZSH> about XEmacs.

ZSH> Anyway, after adding 256 meg memory, I don't care much about that now.

I have 256 and I start caring when I start trying to run a lot of
things (like vmware NT with 128 for it), etc...

Is there any way to figure out where memory is going in emacs/xemacs?
The obvious things (a buffer) shows nothing.  The largest buffer is a
nnimap history buffer, but it's only 4 megs.

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."

Re: Benchmarking Gnus

[Steinar Bang]

>>>>> Wes Hardaker <wes@hardakers.net>:

[snip! 80MB xemacs with Gnus]
> Gnus is definitely responsible.  I finally did run it for a while
> without gnus and didn't get any major memory leaks.  Immediately
> after starting gnus and entering a few groups, I'm back at huge
> memory usages.  sigh...

> However, these are imap folders.

Do you have some old articles marked?  Are you using agent?  If so, I
have seen the "exploding process size on folder visit" problem as well
(search for the thread with the subject "any tips for how to tune
Gnus/emacs for low memory usage?") on a laptop with very little
physical memory by today's standards.

The problems were more severe with XEmacs 21.4, than they were with
GNU Emacs 20.7, though the problems were recognizably present there as
well. 

Didn't larsi do some tuning on opening Summary buffers recently, that
should avoid creating so much stuff to be garbage collected?  Or was
that just nntp?

Has anyone looked into the bug with agent fetching all articles
between the oldest ticked articles and the current (ie. re-fetching
articles that were previously expired from the agent)?  It was
reported by Bjørn Mork a couple of weeks back.

Re: Benchmarking Gnus

[Bjørn Mork]

Steinar Bang <sb@dod.no> writes:

> Has anyone looked into the bug with agent fetching all articles
> between the oldest ticked articles and the current (ie. re-fetching
> articles that were previously expired from the agent)?  It was
> reported by Bjørn Mork a couple of weeks back.

I believe this was fixed by larsi a week ago or so. Agent works more
or less as expected for me at the moment.


Bjørn
-- 
Save the sexist jokes!  

Re: Benchmarking Gnus

[Steinar Bang]

>>>>> Bjørn Mork <bmork@dod.no>:

> Steinar Bang <sb@dod.no> writes:

>> Has anyone looked into the bug with agent fetching all articles
>> between the oldest ticked articles and the current (ie. re-fetching
>> articles that were previously expired from the agent)?  It was
>> reported by Bjørn Mork a couple of weeks back.

> I believe this was fixed by larsi a week ago or so. Agent works more
> or less as expected for me at the moment.

Maybe it's time to upgrade then?  I saw some nnimap expiry problems
reported earlier.  Have you seen any indications of this?

Re: Benchmarking Gnus

[Bjørn Mork]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

Steinar Bang <sb@dod.no> writes:
>>>>>> Bjørn Mork <bmork@dod.no>:
>
>> I believe this was fixed by larsi a week ago or so. Agent works more
>> or less as expected for me at the moment.
>
> Maybe it's time to upgrade then?  I saw some nnimap expiry problems
> reported earlier.  Have you seen any indications of this?

Haven't noticed anything. What sort of problems? I am still running
with the attached patch (provided by Simon, I think) to avoid problems
with the combination nnimap, agent and expiry-target.


Bjørn
-- 
Man, I can't believe how sadistic you are.  

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: agent-expiry-fix.diff --]
[-- Type: text/x-patch, Size: 880 bytes --]

Index: lisp/gnus-int.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus-int.el,v
retrieving revision 6.9
diff -u -r6.9 gnus-int.el
--- lisp/gnus-int.el	2002/01/11 02:36:58	6.9
+++ lisp/gnus-int.el	2002/01/31 22:38:29
@@ -514,9 +514,9 @@
 	(let ((mail-parse-charset message-default-charset))
 	  (mail-encode-encoded-word-buffer)))
       (message-encode-message-body)))
-  (let ((func (car (or gnus-command-method
-		       (gnus-find-method-for-group group)))))
-    (funcall (intern (format "%s-request-accept-article" func))
+  (let ((gnus-command-method (or gnus-command-method
+				 (gnus-find-method-for-group group))))
+    (funcall (gnus-get-function gnus-command-method 'request-accept-article)
 	     (if (stringp group) (gnus-group-real-name group) group)
 	     (cadr gnus-command-method)
 	     last)))

Re: Benchmarking Gnus

[Steinar Bang]

>>>>> Bjørn Mork <bmork@dod.no>:

> Haven't noticed anything. What sort of problems? I am still running
> with the attached patch (provided by Simon, I think) to avoid
> problems with the combination nnimap, agent and expiry-target.

Is it stuff in the patch, and not the CVS trunk, that avoids
re-fetching articles that has already been expired in agent?

Re: Benchmarking Gnus

[Bjørn Mork]

Steinar Bang <sb@dod.no> writes:
>>>>>> Bjørn Mork <bmork@dod.no>:
>
>> Haven't noticed anything. What sort of problems? I am still running
>> with the attached patch (provided by Simon, I think) to avoid
>> problems with the combination nnimap, agent and expiry-target.
>
> Is it stuff in the patch, and not the CVS trunk, that avoids
> re-fetching articles that has already been expired in agent?

Nope.


Bjørn
-- 
You're always totally wrong.  

Re: Benchmarking Gnus

[Steinar Bang]

>>>>> Bjørn Mork <bmork@dod.no>:

> Steinar Bang <sb@dod.no> writes:
>>>>>>> Bjørn Mork <bmork@dod.no>:

>>> Haven't noticed anything. What sort of problems? I am still
>>> running with the attached patch (provided by Simon, I think) to
>>> avoid problems with the combination nnimap, agent and
>>> expiry-target.

>> Is it stuff in the patch, and not the CVS trunk, that avoids
>> re-fetching articles that has already been expired in agent?

> Nope.

What are the problems with the above combination?

One thing that has never worked for me, is that expiration of
agentized nnimap groups, run in unplugged mode, are made to be imap
delete operations in the folders on the server, when I go plugged
again. 

I'm not even sure if this is _supposed_ to be working...? :-)

Does the patch have anything to to with this problem?

Re: Benchmarking Gnus

[Bjørn Mork]

Steinar Bang <sb@dod.no> writes:

> What are the problems with the above combination?

I'm using total-expire and another imap mailbox as expiry-target in
one of my imap inboxes. This caused an error when I tried to exit from
the inbox while unplugged. The patch eliminates the error message. 
Don't ask me why and how - I don't know :-)

Expiry won't work with the patch either (since articles cannot be
moved while unplugged), but that doesn't matter to me since expiry
process will run again as soon as I'm plugged anyway. The only problem
with this is that it kind of assumes that offline deletion isn't
implemented...

> One thing that has never worked for me, is that expiration of
> agentized nnimap groups, run in unplugged mode, are made to be imap
> delete operations in the folders on the server, when I go plugged
> again. 

Really? I didn't think deleting worked at all when unplugged. Let me
try. No, it doesn't. That's good, or I would have lost a lot of mail
without noticing.

> I'm not even sure if this is _supposed_ to be working...? :-)

No, I don't think it is. 


Bjørn
-- 
You sound like a real coward.  

Re: Benchmarking Gnus

[Simon Josefsson]

On Fri, 1 Feb 2002, Steinar Bang wrote:

> One thing that has never worked for me, is that expiration of
> agentized nnimap groups, run in unplugged mode, are made to be imap
> delete operations in the folders on the server, when I go plugged
> again. 
> 
> I'm not even sure if this is _supposed_ to be working...? :-)

It sounds like a bug, expiring the agent nnml group should not delete 
anything in the original nnimap group.

> Does the patch have anything to to with this problem?

No, I don't think so.  I'll check in the patch as well, I think it is 
correct.

Re: Benchmarking Gnus

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> It sounds like a bug, expiring the agent nnml group should not
> delete anything in the original nnimap group.

No it's not a bug.  :-)
What happens is what you describe.

I expected unplugged expiry to be transferred to the original nnimap
group when I went plugged.  Ie. I expected what you term buggy
behaviour. :-)

Re: Benchmarking Gnus

[Simon Josefsson]

On Fri, 1 Feb 2002, Steinar Bang wrote:

> >>>>> Simon Josefsson <jas@extundo.com>:
> 
> > It sounds like a bug, expiring the agent nnml group should not
> > delete anything in the original nnimap group.
> 
> No it's not a bug.  :-)
> What happens is what you describe.
> 
> I expected unplugged expiry to be transferred to the original nnimap
> group when I went plugged.  Ie. I expected what you term buggy
> behaviour. :-)

Hm. There is two kinds of expiry in a agent group, one is the normal
expiring in the real backend via E marks (nnimap), which should work (does
it?), but you should also be able to locally expire the agent cache with
gnus-agent-expire-articles (does it work?).

The first should delete articles on the server, the second should not.

Hm.  I don't use the agent much, but does this make sense?

(Btw, wouldn't it be nice if it was possible to enable the agent via 
customize instead of forcing people to manually edit ~/.gnus?)

Re: Benchmarking Gnus

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> Hm. There is two kinds of expiry in a agent group, one is the normal
> expiring in the real backend via E marks (nnimap), which should work
> (does it?),

If you mean: "does it work if you run expiry when plugged?", the
answer is yes.

If you mean: "if you run expiry when unplugged, are the results
transferred to the nnimap group, when you go plugged?", the answer was
no, the last time I tried.

> but you should also be able to locally expire the agent cache with
> gnus-agent-expire-articles (does it work?).

I haven't tried recently, and I'm not in physical reach of my laptop
right now, but it worked the last time I tried.  Ie. it _did_ expire
articles older than 7 days, but if I had some old ticked articles,
the articles in between were re-fetched the next time I fetched
articles into the agent.

According to Bjørn Mork earlier in this thread, this should be fixed
now...? 

> The first should delete articles on the server, the second should
> not.

> Hm.  I don't use the agent much,

This means that you don't use a laptop with suspend/resume? :-)

> but does this make sense?

Sure.

> (Btw, wouldn't it be nice if it was possible to enable the agent via
> customize instead of forcing people to manually edit ~/.gnus?)

You can toggle server agent status interactively in the server buffer,
if that's what you're thinking about...?

Re: Benchmarking Gnus

[ShengHuo ZHU]

Steinar Bang <sb@dod.no> writes:

>>>>>> Simon Josefsson <jas@extundo.com>:
>
>> Hm. There is two kinds of expiry in a agent group, one is the normal
>> expiring in the real backend via E marks (nnimap), which should work
>> (does it?),
>
> If you mean: "does it work if you run expiry when plugged?", the
> answer is yes.
>
> If you mean: "if you run expiry when unplugged, are the results
> transferred to the nnimap group, when you go plugged?", the answer was
> no, the last time I tried.

Gnus should skip expiring offline servers, shouldn't it?

[...]

>> (Btw, wouldn't it be nice if it was possible to enable the agent via
>> customize instead of forcing people to manually edit ~/.gnus?)
>
> You can toggle server agent status interactively in the server buffer,
> if that's what you're thinking about...?

One possible solution is customizing the gnus-agent variable and
running gnus-agentize if gnus-agent is t after loading ~/.gnus.  

ShengHuo

Re: Benchmarking Gnus

[Simon Josefsson]

On Fri, 1 Feb 2002, ShengHuo ZHU wrote:

> > If you mean: "if you run expiry when unplugged, are the results
> > transferred to the nnimap group, when you go plugged?", the answer was
> > no, the last time I tried.
> 
> Gnus should skip expiring offline servers, shouldn't it?

IMHO it should remember the expiry command and run it when you get back
online.  Hm.  But the conditions that decides which articles should be
expired could have changed until then.  Nah, I think you are right, if you 
want to expire a group, you need to be online.  If you are offline, maybe 
it should expire the local agent directory instead.  Opinions?

Re: Benchmarking Gnus

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

> On Fri, 1 Feb 2002, ShengHuo ZHU wrote:

>>> If you mean: "if you run expiry when unplugged, are the results
>>> transferred to the nnimap group, when you go plugged?", the answer
>>> was no, the last time I tried.

>> Gnus should skip expiring offline servers, shouldn't it?

> IMHO it should remember the expiry command and run it when you get
> back online. 

That's what I would think.

> Hm.  But the conditions that decides which articles should be
> expired could have changed until then. 

Some articles may have been ticked from a different client, you mean?
If they have, Gnus should detect that, not delete the article, import
the flags for the changed articles into agent, fetching or re-fetching
copies of the articles if neccessary.

This may be more than a 5-liner...:-) but see below.

> Nah, I think you are right, if you want to expire a group, you need
> to be online.  If you are offline, maybe it should expire the local
> agent directory instead.  Opinions?

IMO an agentized group should reflect the underlying group as closely
as possible.  This means that all operations, including move and
expiry, should be available, and operate the same way as they do when
plugged.

The basic worldview is that the underlying group is the master, and
that the agent group will reflect the master as closely as possible.

This goes for nnimap groups.  Ie. if an article has been deleted on
the server, I would like the article to be deleted in the agent as
well, even if I had it ticked there.

However, for nntp, I have a different requirement.  On nntp, I would
like to keep a ticked article, even if the article have been expired
on the server.

Though I can live with an identical behaviour to what I see with
nnimap, if it is still possible to save local copies of articles, in
the way we did before agent.  Is it?

Re: Benchmarking Gnus

[Simon Josefsson]

On Fri, 1 Feb 2002, Steinar Bang wrote:

> > Hm. There is two kinds of expiry in a agent group, one is the normal
> > expiring in the real backend via E marks (nnimap), which should work
> > (does it?),
> 
> If you mean: "does it work if you run expiry when plugged?", the
> answer is yes.

Good.

> If you mean: "if you run expiry when unplugged, are the results
> transferred to the nnimap group, when you go plugged?", the answer was
> no, the last time I tried.

Bad.  I wonder why?  Hm.  Ah, nnagent-request-expire-articles needs to
save the invokation into a file that is evaluated when you plug in.  Much
like the "flags" file.  Maybe it can even use that file.  This looks like
a five-liner.

> > but you should also be able to locally expire the agent cache with
> > gnus-agent-expire-articles (does it work?).
> 
> I haven't tried recently, and I'm not in physical reach of my laptop
> right now, but it worked the last time I tried.  Ie. it _did_ expire
> articles older than 7 days, but if I had some old ticked articles,
> the articles in between were re-fetched the next time I fetched
> articles into the agent.
> 
> According to Bjørn Mork earlier in this thread, this should be fixed
> now...? 

Fixed in what sense?  I think the agent MUST have ticked (and generally 
marked articles) locally stored, otherwise the marks will be removed by 
code in gnus-sum.el that removes flags on articles that has disappeared. 
Hm. Maybe that code could be changed, instead of requiring the agent to 
download all marked articles though.

> > The first should delete articles on the server, the second should
> > not.
> 
> > Hm.  I don't use the agent much,
> 
> This means that you don't use a laptop with suspend/resume? :-)

My laptop always has a network connection. :-)

And I never expire anything anyway.  Maybe that can be useful for the
benchmark/design discussion, I have some IMAP groups with >500000 messages
or so.  (Which are completely unusable in Gnus).

> > (Btw, wouldn't it be nice if it was possible to enable the agent via
> > customize instead of forcing people to manually edit ~/.gnus?)
> 
> You can toggle server agent status interactively in the server buffer,
> if that's what you're thinking about...?

No, I was thinking about that you need to call (gnus-agentize) in ~/.gnus 
to enable that code (and other code as well).  I'd like a 
`gnus-use-agent' variable, much like the `gnus-use-cache' variable.

Re: Benchmarking Gnus

[Steinar Bang]

>>>>> Simon Josefsson <jas@extundo.com>:

[snip!]
>> If you mean: "if you run expiry when unplugged, are the results
>> transferred to the nnimap group, when you go plugged?", the answer
>> was no, the last time I tried.

> Bad.  I wonder why?  Hm.  Ah, nnagent-request-expire-articles needs
> to save the invokation into a file that is evaluated when you plug
> in.  Much like the "flags" file.  Maybe it can even use that file.
> This looks like a five-liner.

Kewl!

Re: Benchmarking Gnus

[Wes Hardaker]

>>>>> On Thu, 31 Jan 2002 17:34:29 +0100, Steinar Bang <sb@dod.no> said:

>>>>> Wes Hardaker <wes@hardakers.net>:
Steinar> [snip! 80MB xemacs with Gnus]

>> Gnus is definitely responsible.  I finally did run it for a while
>> without gnus and didn't get any major memory leaks.  Immediately
>> after starting gnus and entering a few groups, I'm back at huge
>> memory usages.  sigh...

>> However, these are imap folders.

Steinar> Do you have some old articles marked?  Are you using agent?

Yep.  Though I'm typically online most of the time.

I have done a recent update and the problem still persists so the
recent agent changes didn't help.

(I should try turning off agent support entirely and see if the
problem still exists...  Maybe I can narrow it down further to agent
vs not-agent)

Steinar> The problems were more severe with XEmacs 21.4, than they
Steinar> were with GNU Emacs 20.7, though the problems were
Steinar> recognizably present there as well.

I'm at 21.4.5, so...  (it's probably time to update it again).

-- 
"Ninjas aren't dangerous.  They're more afraid of you than you are of them."