Re: opening speed for nnml groups

[Ken Raeburn <raeburn@cygnus.com>]

Steinar Bang <sb@metis.no> writes:

> I haven't done any timing of it yet, but it seems like opening nnml
> groups get noticably slower, as the groups get bigger (with more
> ticked or read articles in them).
> 
> I was sort of idly wondering where the speed hit was?  Is it in the
> parsing of the .overview file?

(I forget who pointed this out to me...)
If you use the message date in the summary line, that's probably a big
piece of it.  See my messages of some time ago about regexp ordering
in timezone-parse-date.  Getting the order right (since all email I
tried matched one pattern, and it wasn't the first one) saves a large
fraction of the time spent in that routine.

In a test I just did with ELP and a mail group of some 7800 messages,
timezone-parse-date (probably without fixed regexp ordering, I've been
lame) took 46.9 seconds out of the total of 194.3.

gnus-summary-read-group-1 took 194.3 seconds
  gnus-select-newsgroup took 32
    gnus-get-newsgroup-headers-xover took 29.6 (nov-parse-line done inline)
  gnus-summary-prepare took 152
    gnus-make-threads took 4
    gnus-sort-threads took 12.8
    gnus-summary-prepare-threads took 135.6
      formatting and inserting summary line took ...??

Since my line-format-spec uses the date, let's be generous and say
that gnus-summary-prepare-threads would've taken only 88 seconds
without it, and the total time would've been 147 seconds.  Then
gnus-summary-prepare-threads still looks like the dominant factor at
60%.  But parsing the overview data is still 20%, and could perhaps be
optimized.

Hm.  Interesting.  I ran gnus-compile to compile the specs, and ran my
test again.  Now timezone-parse-date supposedly only took 25.7
seconds, but the total time was still 190, and
gnus-summary-prepare-threads still took 132.5.  Something seems fishy
here...

Oh, and gnus-simplify-subject-fuzzy used about 26 seconds in almost
9800 calls, but I haven't traced down where in the call graph that
would be located.  I wonder if the hacks I did for
gnus-summary-kill-same-subject (more generally, for matching of
simplified subjects) could help here.  (I sent them in 24 May 1996,
and Lars said that same day that he'd include them in Red Gnus.  Still
hasn't happened... :-)

gnus-summary-highlight-line took 14 seconds, in 7873 calls

gnus-sort-threads took 12 seconds; gnus-thread-sort-by-number took 4.5
seconds in 20703 calls

gnus-put-text-property took 9 seconds in 15812 calls

That looks like it for functions taking over 5% of the time.

> I remember that slow exit on big or sparse nnml group, was a problem
> when you did expiry on exiting a group, so I went for demonical
> expiry, as well as demonical fetch and sort.

Me too (for expiry), but it's still a pain to have to wait if I come
back to Emacs while it's running.  I can't even interrupt it with ^G.

> If the reading and parsing of the .overview file is the problem, would
> it be an idea to cache the parsed info between visits? And update it
> during fetch and sort operations?

Or maybe store the data as s-expressions, possibly with extra fields
for caching references links and parsed dates, rather than the
standard news overview format?

> Or is that too expensive memory-wise?

The overview file for this group I was working with is 1.8M.  I doubt
the lisp version would be much smaller.  I wouldn't want to cache that
much info in memory just on the chance that I might come back to that
newsgroup eventually.  Maybe if it got dumped the moment I entered
some other newsgroup; that would permit leaving, fetching new mail,
and re-entering.