Re: [ patch ] async stuff fix(?) (was Re: 0.3 and async pre-fetch)

[Ken Raeburn <raeburn@cygnus.com>]

Sudish Joseph <sudish@mindspring.com> writes:

> The downside to using strings is that we'll get -large- strings that
> will hog memory until garbage collection.

I think that'll be bad for performance.  Remember emacs does gc
roughly after each N bytes allocated.  So if you get a large string
per article, say, you'll be doing gc very often as you go through one
newsgroup.  (Just building a summary buffer for 1000 articles causes
me to run gc several times.  That's part of the reason for the
simplified-subject-caching speedups I sent mail about a while back.
At some point interning in a symbol table once is cheaper than
regenerating multiple copies of the same strings over and over.)

Also, the cost of doing a gc is, I think, dependent on the currently
allocated storage and the max-so-far allocated storage.  (Has anyone
looked at it more closely?  I think only buffers are allocated from
storage that can get freed, but I could be wrong.)  So allocating big
strings that can't be gc'ed right away means the price of future gc
passes probably goes up.

Can you return articles or other big stuff as buffers or buffer
portions?  (Carefully arranging to always have them cleaned up, of
course.)

>	  Is there any way to
> truncate the actual string to get around this?  (It has to be there,
> I'm too lazy to dig right now.)  Even better, any way to totally
> obliterate the in-memory string; i.e., gc a single string?

None I know of.

>	  I don't
> think the number of actual strings generated will increase gc costs.
> If so, we might keep a constant number of strings that we grow and
> truncate as needed.  Hmm, do buffers cost more/less than strings?  We
> might pass buffers as our result from the filter if that's better.
> Buffers aren't gc'ed I think, but we can manually maintain those quite
> easily.

A buffer should be more expensive than a string, but passing around
lots of mark in a single buffer may be cheaper than manipulating lots
of substrings.  The buffer contents don't go through the alloc/gc
interface that strings and other objects use.

> PS:  If anyone would care to explain: can buffers be gc'ed?  are
> lambda's gc'ed (if not, gnus currently leaks)?  can an arbitrary
> object be completely destroyed/gc'ed all by itself?

I think you have to explicitly kill buffers first; yes; no.