Gnus from CVS, Emacs from CVS --> out of memory?

Gnus from CVS, Emacs from CVS --> out of memory?

[Kai Großjohann]

From time to time, after telling the Agent `J s', I get an out of
memory error.  Emacs doesn't like it at all.  It started happening
maybe a week ago.

Is anyone else seeing this?

Sometimes, Emacs tells me it's out of memory and asks me whether to
continue.  Saying no gives me a working Emacs, but saying yes means
that Emacs is really hung.  Constantly beeping (or flashing, due to
visible-bell).

Sometimes, I think that Emacs isn't asking for permission, it just
constantly dings...
-- 
file-error; Data: (Opening input file no such file or directory ~/.signature)

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Xavier Maillard]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

On  9 mai 2003, Kai Großjohann wrote:

>  From time to time, after telling the Agent `J s', I get an out of
>  memory error.  Emacs doesn't like it at all.  It started happening
>  maybe a week ago.
>  
>  Is anyone else seeing this?

I don't have this problem here even if I use daily Emacs CVS and Gnus
CVS. The agent is working quite good for me.
  
>  Sometimes, Emacs tells me it's out of memory and asks me whether to
>  continue.  Saying no gives me a working Emacs, but saying yes means
>  that Emacs is really hung.  Constantly beeping (or flashing, due to
>  visible-bell).
>  
>  Sometimes, I think that Emacs isn't asking for permission, it just
>  constantly dings...

Dunno where the problem is exactly, but I will take care on my side to
see whether it happens too.

zeDek
-- 
http://www.gnusfr.org -- French Gnus user site

Anti-war disclaimer:
	"Bombing for peace is like fucking for virginity"

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Kevin Greiner]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> From time to time, after telling the Agent `J s', I get an out of
> memory error.  Emacs doesn't like it at all.  It started happening
> maybe a week ago.
>
> Is anyone else seeing this?
>
> Sometimes, Emacs tells me it's out of memory and asks me whether to
> continue.  Saying no gives me a working Emacs, but saying yes means
> that Emacs is really hung.  Constantly beeping (or flashing, due to
> visible-bell).
>
> Sometimes, I think that Emacs isn't asking for permission, it just
> constantly dings...

The constant dings is the result of running out of memory while trying
to generate the "out of memory" alert.  I used to get it when I evaled
a form on way too many buffers.


You might check the size of your group's .overview file.  I managed to
get one up to 108M by not running gnus-agent-expire for several
months.

That seems to be the most likely culpret.  The group's article lists
may also be growing but it's hard to believe that they'd be that
large.

Kevin

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> The constant dings is the result of running out of memory while trying
> to generate the "out of memory" alert.  I used to get it when I evaled
> a form on way too many buffers.

Hehe.

> You might check the size of your group's .overview file.  I managed to
> get one up to 108M by not running gnus-agent-expire for several
> months.
>
> That seems to be the most likely culpret.  The group's article lists
> may also be growing but it's hard to believe that they'd be that
> large.

Hm.  The sum of all .overview files is 111M, distributed over 300 of
them.  No file is larger than 10M.

(I just deleted some files of 20M each.  But I still get the same
symtoms.)

The article lists, is that the .agentview files?  They are all <100k.

-- 
This line is not blank.

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Kevin Greiner]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> The constant dings is the result of running out of memory while trying
>> to generate the "out of memory" alert.  I used to get it when I evaled
>> a form on way too many buffers.
>
> Hehe.
>
>> You might check the size of your group's .overview file.  I managed to
>> get one up to 108M by not running gnus-agent-expire for several
>> months.
>>
>> That seems to be the most likely culpret.  The group's article lists
>> may also be growing but it's hard to believe that they'd be that
>> large.
>
> Hm.  The sum of all .overview files is 111M, distributed over 300 of
> them.  No file is larger than 10M.
>
> (I just deleted some files of 20M each.  But I still get the same
> symtoms.)

What does list-buffers show after you kill the fetch session?  As an
example, I just opened two groups and my .newsrc-dribble file is
already over 3M.

> The article lists, is that the .agentview files?  They are all <100k.

Actually I was thinking of the seen, read, ticked lists in newsrc.eld

Kevin

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> What does list-buffers show after you kill the fetch session?  As an
> example, I just opened two groups and my .newsrc-dribble file is
> already over 3M.

ibuffer wasn't enlightening, but now I did this:

ELISP> (mapcar (lambda (b) (cons b (buffer-size b))) (buffer-list))
((#<buffer *ielm*> . 124)
 (#<buffer  *Minibuf-1*> . 0)
 (#<buffer *scratch*> . 581)
 (#<buffer *Group*> . 6551)
 (#<buffer drafts/*sent posting on gnu.emacs.help*> . 873)
 (#<buffer *sent posting on gnu.emacs.gnus*> . 2244)
 (#<buffer *Dead Summary nnimap+du:teach.infoa.mailto*> . 68)
 (#<buffer *sent reply to info dan*> . 621)
 (#<buffer *sent followup to Wang Yin on gnu.emacs.help*> . 841)
 (#<buffer *sent followup to Nacho on gnu.emacs.gnus*> . 2212)
 (#<buffer *Article*> . 241)
 (#<buffer *sent posting on gnu.emacs.help*> . 564)
 (#<buffer *sent followup to Paul Stevenson on gnu.emacs.gnus*> . 1153)
 (#<buffer *sent posting on teach.infoa*> . 527)
 (#<buffer  *Minibuf-0*> . 0)
 (#<buffer *Messages*> . 9607)
 (#<buffer  *Echo Area 0*> . 8)
 (#<buffer  *Echo Area 1*> . 19)
 (#<buffer  *nntpd*> . 2482)
 (#<buffer  *Gnus agent overview*> . 32349)
 (#<buffer  *gnus work*> . 28)
 (#<buffer  trace of POP session to mail.gmx.net> . 157)
 (#<buffer  trace of POP session to pop-serv1.uni-duisburg.de> . 173)
 (#<buffer  *nnimap* dtm> . 0)
 (#<buffer  *nnimap* du> . 0)
 (#<buffer  *server news.freshmeat.net nntp  *nntpd**> . 0)
 (#<buffer  *code-converting-work*> . 29)
 (#<buffer .bbdb> . 213373)
 (#<buffer  *Original Article*> . 1094)
 (#<buffer  *Async Prefetch Article*> . 0)
 (#<buffer  *canonical address*> . 16)
 (#<buffer  *extract address components*> . 24)
 (#<buffer  *split*> . 0)
 (#<buffer  *gnus article copy*> . 1094)
 (#<buffer  *Gnus Backlog*> . 2350948)
 (#<buffer *Completions*> . 183)
 (#<buffer  *server news.uni-duisburg.de nntp  *nntpd**> . 271)
 (#<buffer  *server news.cis.dfn.de nntp  *nntpd**> . 955)
 (#<buffer  *server news.gnus.org nntp  *nntpd**> . 2601)
 (#<buffer  *server news.gmane.org nntp  *nntpd**> . 1186)
 (#<buffer *Quail Completions*> . 0)
 (#<buffer  *nnmail message-id cache*> . 270109)
 (#<buffer .newsrc-dribble> . 586116)
 (#<buffer *Warnings*> . 68)
 (#<buffer *Help*> . 189))

I did `J s' in a Gnus.  It told me it's out of memory and asked me
whether to continue.  I said no, and then evaled the above snippet.

It doesn't look dangerous to me.  What is going on here?

How can I investigate further?
-- 
This line is not blank.

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Kevin Greiner]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> What does list-buffers show after you kill the fetch session?  As an
>> example, I just opened two groups and my .newsrc-dribble file is
>> already over 3M.
>
> ibuffer wasn't enlightening, but now I did this:
>
> ELISP> (mapcar (lambda (b) (cons b (buffer-size b))) (buffer-list))
> ((#<buffer *ielm*> . 124)
>  (#<buffer  *Minibuf-1*> . 0)
>  (#<buffer *scratch*> . 581)
>  (#<buffer *Group*> . 6551)
>  (#<buffer drafts/*sent posting on gnu.emacs.help*> . 873)
>  (#<buffer *sent posting on gnu.emacs.gnus*> . 2244)
>  (#<buffer *Dead Summary nnimap+du:teach.infoa.mailto*> . 68)
>  (#<buffer *sent reply to info dan*> . 621)
>  (#<buffer *sent followup to Wang Yin on gnu.emacs.help*> . 841)
>  (#<buffer *sent followup to Nacho on gnu.emacs.gnus*> . 2212)
>  (#<buffer *Article*> . 241)
>  (#<buffer *sent posting on gnu.emacs.help*> . 564)
>  (#<buffer *sent followup to Paul Stevenson on gnu.emacs.gnus*> . 1153)
>  (#<buffer *sent posting on teach.infoa*> . 527)
>  (#<buffer  *Minibuf-0*> . 0)
>  (#<buffer *Messages*> . 9607)
>  (#<buffer  *Echo Area 0*> . 8)
>  (#<buffer  *Echo Area 1*> . 19)
>  (#<buffer  *nntpd*> . 2482)
>  (#<buffer  *Gnus agent overview*> . 32349)
>  (#<buffer  *gnus work*> . 28)
>  (#<buffer  trace of POP session to mail.gmx.net> . 157)
>  (#<buffer  trace of POP session to pop-serv1.uni-duisburg.de> . 173)
>  (#<buffer  *nnimap* dtm> . 0)
>  (#<buffer  *nnimap* du> . 0)
>  (#<buffer  *server news.freshmeat.net nntp  *nntpd**> . 0)
>  (#<buffer  *code-converting-work*> . 29)
>  (#<buffer .bbdb> . 213373)
>  (#<buffer  *Original Article*> . 1094)
>  (#<buffer  *Async Prefetch Article*> . 0)
>  (#<buffer  *canonical address*> . 16)
>  (#<buffer  *extract address components*> . 24)
>  (#<buffer  *split*> . 0)
>  (#<buffer  *gnus article copy*> . 1094)
>  (#<buffer  *Gnus Backlog*> . 2350948)
>  (#<buffer *Completions*> . 183)
>  (#<buffer  *server news.uni-duisburg.de nntp  *nntpd**> . 271)
>  (#<buffer  *server news.cis.dfn.de nntp  *nntpd**> . 955)
>  (#<buffer  *server news.gnus.org nntp  *nntpd**> . 2601)
>  (#<buffer  *server news.gmane.org nntp  *nntpd**> . 1186)
>  (#<buffer *Quail Completions*> . 0)
>  (#<buffer  *nnmail message-id cache*> . 270109)
>  (#<buffer .newsrc-dribble> . 586116)
>  (#<buffer *Warnings*> . 68)
>  (#<buffer *Help*> . 189))
>
> I did `J s' in a Gnus.  It told me it's out of memory and asked me
> whether to continue.  I said no, and then evaled the above snippet.
>
> It doesn't look dangerous to me.  What is going on here?
>
> How can I investigate further?

First load-library debug.
Set debug-on-error or debug-on-signal (not sure which, if either, will
work with out-of-memory).
Then try 'J s'.

Alternatively, have you checked the message log?  Do you have any idea
how many groups were fetched before the error?  If the error always
occurs in one group, you might be able to trace the execution in that
group prior to out-of-memory.

Kevin

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> Alternatively, have you checked the message log?  Do you have any idea
> how many groups were fetched before the error?  If the error always
> occurs in one group, you might be able to trace the execution in that
> group prior to out-of-memory.

Ah, now I got an Emacs where I could do something after the OOM.  It
was fetching nnimap+du:INBOX.spamspam.  I've now increased the group
level to > gnus-agent-handle-level.  Seems to have done the trick.

Alas, I have no idea what might have been the problem, here.  Gnus
was also complaining about "Invalid line 42" or "Invalid entry 42" or
"Invalid overview data 42" or something like this, for values of 42
ranging from 1 to about 8,000.

I wish I had the patience to get to the bottom of this.  I still have
the ~/News/agent/nnimap/du/INBOX/spamspam directory, if that helps?

-- 
This line is not blank.

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Kevin Greiner]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> Kevin Greiner <kgreiner@xpediantsolutions.com> writes:
>
>> Alternatively, have you checked the message log?  Do you have any idea
>> how many groups were fetched before the error?  If the error always
>> occurs in one group, you might be able to trace the execution in that
>> group prior to out-of-memory.
>
> Ah, now I got an Emacs where I could do something after the OOM.  It
> was fetching nnimap+du:INBOX.spamspam.  I've now increased the group
> level to > gnus-agent-handle-level.  Seems to have done the trick.
>
> Alas, I have no idea what might have been the problem, here.  Gnus
> was also complaining about "Invalid line 42" or "Invalid entry 42" or
> "Invalid overview data 42" or something like this, for values of 42
> ranging from 1 to about 8,000.
>
> I wish I had the patience to get to the bottom of this.  I still have
> the ~/News/agent/nnimap/du/INBOX/spamspam directory, if that helps?

Take a look at the .overview file for articles 1 through 8,000.
What's wrong with these lines?  Would it cause read to consume memory?

Kevin

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Kai Großjohann]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> Take a look at the .overview file for articles 1 through 8,000.
> What's wrong with these lines?  Would it cause read to consume memory?

The .overview file looks normal, the headers are what you might
expect from spam.  The file has 264159 bytes and 885 lines, so each
line is 300 bytes long on average.  Sounds pretty normal to me.

(Actually, the first number is 3265 and the last number is 6353, and
so there are a lot of gaps.  But the message went by so quick, it was
hard to see.)
-- 
This line is not blank.

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Dave Love]

Kevin Greiner <kgreiner@xpediantsolutions.com> writes:

> The constant dings is the result of running out of memory while trying
> to generate the "out of memory" alert.

Then it's a bug which should be reported.  Emacs keeps some memory
free for such an eventuality.

> You might check the size of your group's .overview file.  I managed to
> get one up to 108M by not running gnus-agent-expire for several
> months.

For what it's worth, out-of-memory errors can arise from the total
heap getting too big or fragmented, regardless of buffer sizes or
available VM.  (See `VALIDATE_LISP_STORAGE' in alloc.c.)

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Harry Putnam]

kai.grossjohann@gmx.net (Kai Großjohann) writes:

> From time to time, after telling the Agent `J s', I get an out of
> memory error.  Emacs doesn't like it at all.  It started happening
> maybe a week ago.
>
> Is anyone else seeing this?

Here is something that seems likely to be related to Kai's reported
problem with J s.:


Starting with cvs 5.10.2  and with a more recent cvs that still says
5.10.2 (yesterday) (and just now including Lars just released 5.10.2
release)
I see the following phenomena:

Starting emacs/gnus from scratch with `emacs -f gnus'.  Everything
seems to proceed normally but once everything is fully loaded.  If I
run `J s' gnus/emacs completely crashs and the frame disappears
without a sound.  Some sort of silent kill happens.  Looks like what
one would expect from a `kill -9 emacs'.

(NOTE: Reverting to cvs version v0.20 cures the problem reported
below)


Recent changes I've done are:

1) Removed the 3 cons cells related to undownloaded faces with
   customize

2) Added `%O' specifier to gnus-summary-line-format (in .gnus.el)
  (setq gnus-summary-line-format "%O%2t%U%R%7d%z%I%(%[%4L: %-20,20f%]%) %s\n")

3) Added a handler someone posted to .gnus.el
  (setq gnus-use-demon t)
  (setq gnus-demon-timestep 1)
  (gnus-demon-add-handler 'gnus-demon-scan-news 120 60)

Re: Gnus from CVS, Emacs from CVS --> out of memory?

[Harry Putnam]

Harry Putnam <hgp@sbcglobal.net> writes:

TYPO ALERT
> (NOTE: Reverting to cvs version v0.20 cures the problem reported
> below)
  ^^^^^  
Should be `above'