Memory exhausted while saving .newsrc.eld

Memory exhausted while saving .newsrc.eld

[Kevin Greiner]

I just noticed that doing a 'J s' (gnus-agent-fetch-session) on
sufficient large number of groups can make the gnus-newsrc-alist so
large that you run out of memory while saving it.  I've traced the
problem to gnus-save-newsrc-file which writes everything to a buffer
then uses save-buffer to write it to a file.  I'd like to rewrite
gnus-save-newsrc-file to write directly to a temp file then, if no
errors occur during writing, rename the temp file as .newsrc.eld.  I
plan on handing version-control correctly so that the new
gnus-backup-startup-file will be respected.  The only problem that I
see is gnus-save-quick-newsrc-hook.  This hook is currently run while
in the temporary buffer.  By its documentation, you can use it to set
the buffer's version-control.  Does anyone use this hook for anything
other than setting version-control?  If so, I'd sure like to hear
about it.

Thanks,
Kevin

Re: Memory exhausted while saving .newsrc.eld

[Lars Magne Ingebrigtsen]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> I just noticed that doing a 'J s' (gnus-agent-fetch-session) on
> sufficient large number of groups can make the gnus-newsrc-alist so
> large that you run out of memory while saving it.  I've traced the
> problem to gnus-save-newsrc-file which writes everything to a buffer
> then uses save-buffer to write it to a file.

Wow.  Is the .newsrc.eld file really that big?

Anyway, I just found a bug in the .newsrc.eld code that made my file
7k long instead of 700k, as it apparently was supposed to be.  I've
now bound print-length to nil the appropriate function.

> I'd like to rewrite gnus-save-newsrc-file to write directly to a
> temp file then, if no errors occur during writing, rename the temp
> file as .newsrc.eld.

I'm not sure it's possible to do any such thing in a reasonable
fashion.  You'd have basically to implement the Lisp writer in a
file-specific manner.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Memory exhausted while saving .newsrc.eld

[Kevin Greiner]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> I just noticed that doing a 'J s' (gnus-agent-fetch-session) on
>> sufficient large number of groups can make the gnus-newsrc-alist so
>> large that you run out of memory while saving it.  I've traced the
>> problem to gnus-save-newsrc-file which writes everything to a buffer
>> then uses save-buffer to write it to a file.
>
> Wow.  Is the .newsrc.eld file really that big?

Yea, I subscribed to a huge number of groups then run fetch session
about once a week to see what happens.  So far, I've hit the
out-of-memory problem and a buffer overflow when an agent overview
file passed 108M.  I'm not sure how useful the testing is, but it's
nice to see that you can be fairly stupid and still have gnus
function.

> Anyway, I just found a bug in the .newsrc.eld code that made my file
> 7k long instead of 700k, as it apparently was supposed to be.  I've
> now bound print-length to nil the appropriate function.
>
>> I'd like to rewrite gnus-save-newsrc-file to write directly to a
>> temp file then, if no errors occur during writing, rename the temp
>> file as .newsrc.eld.
>
> I'm not sure it's possible to do any such thing in a reasonable
> fashion.  You'd have basically to implement the Lisp writer in a
> file-specific manner.

I agree.  I checked in the change on 3/2 but it's a dog executing. I
changed gnus-gnus-to-quick-newsrc-format to print to the
standard-output stream.  During normal execution, that's the temp
buffer and performance is as good as always.  If you clear the new
'gnus-save-startup-file-via-temp-buffer' variable, you can redefine
standard-output as a lambda that calls write-region.  The lambda does
some caching to avoid calling write-region on each single character,
still its slower than I'd like to see.  I haven't optimized it further
as this has a very low priority to me.

Kevin