Re: Memory exhausted while saving .newsrc.eld

[Kevin Greiner <kgreiner@xpediantsolutions.com>]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> I just noticed that doing a 'J s' (gnus-agent-fetch-session) on
>> sufficient large number of groups can make the gnus-newsrc-alist so
>> large that you run out of memory while saving it.  I've traced the
>> problem to gnus-save-newsrc-file which writes everything to a buffer
>> then uses save-buffer to write it to a file.
>
> Wow.  Is the .newsrc.eld file really that big?

Yea, I subscribed to a huge number of groups then run fetch session
about once a week to see what happens.  So far, I've hit the
out-of-memory problem and a buffer overflow when an agent overview
file passed 108M.  I'm not sure how useful the testing is, but it's
nice to see that you can be fairly stupid and still have gnus
function.

> Anyway, I just found a bug in the .newsrc.eld code that made my file
> 7k long instead of 700k, as it apparently was supposed to be.  I've
> now bound print-length to nil the appropriate function.
>
>> I'd like to rewrite gnus-save-newsrc-file to write directly to a
>> temp file then, if no errors occur during writing, rename the temp
>> file as .newsrc.eld.
>
> I'm not sure it's possible to do any such thing in a reasonable
> fashion.  You'd have basically to implement the Lisp writer in a
> file-specific manner.

I agree.  I checked in the change on 3/2 but it's a dog executing. I
changed gnus-gnus-to-quick-newsrc-format to print to the
standard-output stream.  During normal execution, that's the temp
buffer and performance is as good as always.  If you clear the new
'gnus-save-startup-file-via-temp-buffer' variable, you can redefine
standard-output as a lambda that calls write-region.  The lambda does
some caching to avoid calling write-region on each single character,
still its slower than I'd like to see.  I haven't optimized it further
as this has a very low priority to me.

Kevin