Re: Profiling

[Kevin Greiner <kgreiner@xpediantsolutions.com>]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Gnus was feeling a bit more sluggish than usual, so I've spent a
> couple of hours profiling and fixing obvious things.  (I've cached
> some of the server-to-method calls and stuff like that.)
>
> The main remaining suspicious function now is
> `gnus-agent-possibly-alter-active', which is called for every
> (covered) group.  Is that really necessary?  It takes about one
> quarter of the total time spent when pressing `g'...

Necessary yes.  The server for the gnus.ding group recently changed
its active range to start at 51065.  When this happened, every
agentized article below that cut-off (for me, about 30K articles)
disappeared.

The gnus-agent-possibly-alter-active has to be called where it is to
correctly set the number of unread articles in the Group buffer.  What
can be improved is the implementation.  Right now, it opens each
group's alist file.  In my original implementation, I wanted to use
the agent's private copy of the server's active file.  That way agent
files are opened once per server rather than once per group.  I
abandoned that implementation as it required that I add code to
maintain the agent's active file as articles are added/removed from
the alist.  While it may be more complex to design, its implementation
should not be too difficult, and the performance should be much
better.  I'll look into it tomorrow.

On a related topic (scalability rather than performance), my primary
nntp server was recently replaced.  As part of the change over, the
implementor set the article numbers about 6M above each group's active
range (I suspect so that they didn't have to worry about the number of
articles being posted while they performed the change over).  The
gnus-agent-possibly-alter-active function overrode the new active
range to produce a range spanning the old and new article ranges
(about 14M).  This killed gnus with memory exhausted in
gnus-list-of-unread-articles.  I could not even run catchup as it also
calls gnus-list-of-unread-articles.  I've written two fixes so far
(but have not checked them in due to a lack of testing).

1) Added gnus-sequence-of-unread-articles to return a compressed
   sequence rather than an uncompressed list.  I then modified
   gnus-group-catchup to use it.  This means that you can catchup any
   group irregardless of size (it's also a good deal faster).

2) I modified gnus-agent-possibly-alter-active to assume that all
   articles between the end of the alist and the start of the new real
   active range are read.  That's not technically true.  However,
   these articles are no longer available on the server and they are
   not stored in the agent so they are lost (hence read).  This change
   partly solved my memory exhausted problem as the number of unread
   articles is a more manageable 8M (Foolish server admins, I've never
   seen more than 100K articles due to retention limits).

Any suggestions as to what else may be done?  On a long-term
perspective, should we consider making gnus-newsgroup-unreads a
sequence rather than a list?

Kevin