From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/51114 Path: main.gmane.org!not-for-mail From: Kevin Greiner Newsgroups: gmane.emacs.gnus.general Subject: Re: Memory exhausted while saving .newsrc.eld Date: Mon, 31 Mar 2003 08:55:23 -0600 Sender: owner-ding@hpc.uh.edu Message-ID: References: NNTP-Posting-Host: main.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: main.gmane.org 1049122688 11464 80.91.224.249 (31 Mar 2003 14:58:08 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Mon, 31 Mar 2003 14:58:08 +0000 (UTC) Original-X-From: owner-ding@hpc.uh.edu Mon Mar 31 16:58:07 2003 Return-path: Original-Received: from malifon.math.uh.edu ([129.7.128.13]) by main.gmane.org with esmtp (Exim 3.35 #1 (Debian)) id 1900iJ-0002ox-00 for ; Mon, 31 Mar 2003 16:56:44 +0200 Original-Received: from sina.hpc.uh.edu ([129.7.128.10] ident=lists) by malifon.math.uh.edu with esmtp (Exim 3.20 #1) id 1900hP-0007bG-00; Mon, 31 Mar 2003 08:55:47 -0600 Original-Received: by sina.hpc.uh.edu (TLB v0.09a (1.20 tibbs 1996/10/09 22:03:07)); Mon, 31 Mar 2003 08:56:51 -0600 (CST) Original-Received: from sclp3.sclp.com (sclp3.sclp.com [66.230.238.2]) by sina.hpc.uh.edu (8.9.3/8.9.3) with SMTP id IAA12899 for ; Mon, 31 Mar 2003 08:56:38 -0600 (CST) Original-Received: (qmail 18311 invoked by alias); 31 Mar 2003 14:55:30 -0000 Original-Received: (qmail 18306 invoked from network); 31 Mar 2003 14:55:29 -0000 Original-Received: from quimby.gnus.org (80.91.224.244) by 66.230.238.6 with SMTP; 31 Mar 2003 14:55:29 -0000 Original-Received: from news by quimby.gnus.org with local (Exim 3.12 #1 (Debian)) id 1900kh-0005KL-00 for ; Mon, 31 Mar 2003 16:59:11 +0200 Original-To: ding@gnus.org Original-Path: not-for-mail Original-Newsgroups: gnus.ding Original-Lines: 43 Original-NNTP-Posting-Host: 65.241.87.5 Original-X-Trace: quimby.gnus.org 1049122751 20480 65.241.87.5 (31 Mar 2003 14:59:11 GMT) Original-X-Complaints-To: usenet@quimby.gnus.org Original-NNTP-Posting-Date: 31 Mar 2003 14:59:11 GMT User-Agent: Gnus/5.090017 (Oort Gnus v0.17) Emacs/21.2 (windows-nt) Cancel-Lock: sha1:a+Lzo34l5YiF0+RkJPVZlYyCTck= Precedence: list X-Majordomo: 1.94.jlt7 Xref: main.gmane.org gmane.emacs.gnus.general:51114 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:51114 Lars Magne Ingebrigtsen writes: > Kevin Greiner writes: > >> I just noticed that doing a 'J s' (gnus-agent-fetch-session) on >> sufficient large number of groups can make the gnus-newsrc-alist so >> large that you run out of memory while saving it. I've traced the >> problem to gnus-save-newsrc-file which writes everything to a buffer >> then uses save-buffer to write it to a file. > > Wow. Is the .newsrc.eld file really that big? Yea, I subscribed to a huge number of groups then run fetch session about once a week to see what happens. So far, I've hit the out-of-memory problem and a buffer overflow when an agent overview file passed 108M. I'm not sure how useful the testing is, but it's nice to see that you can be fairly stupid and still have gnus function. > Anyway, I just found a bug in the .newsrc.eld code that made my file > 7k long instead of 700k, as it apparently was supposed to be. I've > now bound print-length to nil the appropriate function. > >> I'd like to rewrite gnus-save-newsrc-file to write directly to a >> temp file then, if no errors occur during writing, rename the temp >> file as .newsrc.eld. > > I'm not sure it's possible to do any such thing in a reasonable > fashion. You'd have basically to implement the Lisp writer in a > file-specific manner. I agree. I checked in the change on 3/2 but it's a dog executing. I changed gnus-gnus-to-quick-newsrc-format to print to the standard-output stream. During normal execution, that's the temp buffer and performance is as good as always. If you clear the new 'gnus-save-startup-file-via-temp-buffer' variable, you can redefine standard-output as a lambda that calls write-region. The lambda does some caching to avoid calling write-region on each single character, still its slower than I'd like to see. I haven't optimized it further as this has a very low priority to me. Kevin