extreme cpu usage with nndir (on specific files)

extreme cpu usage with nndir (on specific files)

[Harry Putnam]

I've run into a problem this morning that has me somewhat stymied.

Using gnus to create an nndir group of an archive I want to copy to a
second nnml server.
The directorys look like ~/00 ~/99 ~/98 and contain messages from
those years.

With file names like so:
awk00:

  00853
  00854
  00855
  00856
[...]

The biggest directory contains some 3000+ messages, the other
two have under 2000 each. 

Making an nndir group of anyone of them causes gnus to go into a spasm
of cpu usage that seems clear out of line with the job at hand.

Watching top display I see cpu pegged near 99 % and memory around 42%.
This lasts for a very long time, like 5 minutes.  Just to get to the
point where gnus asks how many messages I want to see.

Answering with <RET> and the cpu usage is pegged again for an even
longer time.  Eventually the group appears.  After sorting date and
marking with `M P a' (both seem to take unduly long as well).  Then
copy to the desired group.  The copying is also very very slow.
 
These files have the following format:

These two line are the first two headers in every post:
#! rnews 1596
Xref: news.demon.co.uk comp.lang.awk:8026
REST OF HEADERS
BLANK LINE
body
body

I wondered if the "#!" may be causing all the commotion?

I've also tried it with a vanilla emacs and gnus on  a dummy account, but 
still see the hefty cpu usage etc.  Seems it must be something about these 
particular files. 

Re: extreme cpu usage with nndir (on specific files)

[Kai Großjohann]

One way to find out what the heck Emacs is doing is to M-x
toggle-debug-on-quit RET, then hit C-g when it takes the long time.
Do this several times, and you might get an idea what takes so long.

Hm.

Or maybe M-x elp-instrument-package RET nndir RET?  Anybody out there
who groks elp?
kai
-- 
I like BOTH kinds of music.

Re: extreme cpu usage with nndir (on specific files)

[Harry Putnam]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> One way to find out what the heck Emacs is doing is to M-x
> toggle-debug-on-quit RET, then hit C-g when it takes the long time.
> Do this several times, and you might get an idea what takes so long.
> 
> Hm.
> 
> Or maybe M-x elp-instrument-package RET nndir RET?  Anybody out there
> who groks elp?

I found out a little by a different method.  First I stripped the 
`^#! rnews ' line replaced it with `From gnus@rnews'.  That didn't
help so it wasn't the header with the #!  causing the trouble.  Next I
stripped the Xref header.  That helped a lot so that header was
causing trouble.  Then I began to notice that gnus had created  a
number of messages from my archive with  `nobody' as sender and `no
subject'.

Checking those messages in the actual archive I found those particular
messages had the `From: ' line above the `Path: ' line.

Finally I made a small awk program that puts `X-' in front of:

X-#! rnews
X-Xref:
X-Path:

Adds my own Path: line with a blank line before it:

fix.awk:
---------
/^#! / {a=1}
/^$/ {a=0}
/^#! rnews/ && a=1 \
{print "\nPath: harrys.news.service!verygood!at.home.now\nX-"$0;next}
/^Xref: / && a=1 {print "X-"$0;next}
/^Path: / && a=1 {print "X-"$0;next}{print}

Gnus liked it... pulled the unsorted  complete archive in as a nndoc
and sorted it by date and copied to the desired nnml+n2m groups.

All handling of these files now take normal amounts of time and I was
able to preserver the original headers (sort of).

I think the main problem was the Xref: header.  Is that likely to give
gnus a problem?

This all took so long that I think I'll wait till next time to try the
approach you suggested.