Re: Faster NOV braiding for large newsgroups with many cached articles

[Gareth McCaughan <gareth.mccaughan@pobox.com>]

On Saturday 19 April 2008, Reiner Steib wrote:

[me:]
> > Specifically, gnus-cache-braid-nov takes several minutes to run,
> > and much of this appears to be because all the insertions in the
> > nntp-server-buffer are kinda slow.
> >
> > By building up the new buffer contents in a list of strings,
> > assembling them into a single string, and then dumping that into
> > the buffer where it belongs in a single operation, I can (on my
> > machine, on one occasion -- I haven't tested this scientifically)
> > speed up gnus-cache-braid-nov by a factor of about 20; 30 seconds
> > instead of 10 minutes.
...
> With which (X)Emacs and Gnus versions?  Did you try other versions as
> well?

GNU Emacs (haven't tried XEmacs); several different versions
of Gnus, namely the ones distributed with many recent releases
of GNU Emacs. I'm afraid I don't have records of exactly which
ones, but what I have at the moment is:

    (emacs-version)
    "GNU Emacs 22.1.1 (i386-pc-freebsd, GTK+ Version 2.10.14)
     of 2007-07-25 on g.local"
    (gnus-version)
    "Gnus v5.11"

> It is easier for us if you don't post the modified function.  Instead,
> produce a diff (unified diff preferred: "-u") against the version you
> use (preferably HEAD revision of the CVS trunk, else tell us the
> version). [1]

Looks like you've made the diff yourself in this case; thanks.
(I posted the new code rather than the diff because I was
basically replacing an entire function and it seemed friendlier
to casual ding@-readers to have the code as readily readable
as possible.)

> Better use `gnus-message' here.

Oops, yes. Didn't even notice its existence.

> | +    ;; reverse chunks and concatenate
> | +    (let ((n 0) (records new-records))
> | +      (while records
> | +        (incf n (length (car records)))
> | +        (setq records (cdr records)))
> | +      (let ((new-content (make-string n ?.)))
> | +        (setq n 0)
> | +        (setq records (nreverse new-records))
> | +        (setf new-records nil) ; help the GC a little
> 
> Please explain why you use `setf' and why GC need help.

I use "setf" because I write much more Common Lisp than
elisp and occasionally forget myself. setq would be just
as good, of course.

I don't know whether the GC does in fact *need* help,
but it can't do any harm to give it more opportunity
to collect the garbage early if it wants.

> On Fri, Apr 18 2008, Gareth McCaughan wrote on bugs@gnus.org:
> [...]
> > I posted a version of gnus-cache-braid-nov that works that way
> > to ding@gnus.org. (No replies; fair enough.) 
> 
> Sorry, I didn't have time to look at your code.  It would be better to
> remind us by following-up to the original message instead of starting
> a new thread on a different list.

It occurred to me that perhaps I'd posted it to the wrong
place originally and bugs@ might have been preferable. I'm
sorry if that was less than maximally convenient.

> > It might be better to look at the size of the group and of the cache
> > and choose heuristically between the two implementations, so as not
> > to pay the memory cost for large groups with few cached articles
> > (where I think the speed should be OK with the old implementation,
> > though I haven't measured it).
> 
> Sounds useful.

OK; I'll have a play and put together something that
avoids gratuitously large time and memory costs in as
wide a range of situations as I can.

> > If there's any interest in improving this and my code is useful,
> > I am happy to sign whatever papers are necessary. 
> 
> I'll send you the form off-list.

Thanks.

-- 
g