opening speed for nnml groups

opening speed for nnml groups

[Steinar Bang]

I haven't done any timing of it yet, but it seems like opening nnml
groups get noticably slower, as the groups get bigger (with more
ticked or read articles in them).

I was sort of idly wondering where the speed hit was?  Is it in the
parsing of the .overview file?

I remember that slow exit on big or sparse nnml group, was a problem
when you did expiry on exiting a group, so I went for demonical
expiry, as well as demonical fetch and sort.

If the reading and parsing of the .overview file is the problem, would
it be an idea to cache the parsed info between visits? And update it
during fetch and sort operations?

Or is that too expensive memory-wise?

Re: opening speed for nnml groups

[Steve Folkes]

Steinar Bang <sb@metis.no> writes:

>>> I haven't done any timing of it yet, but it seems like opening nnml
>>> groups get noticably slower, as the groups get bigger (with more
>>> ticked or read articles in them).

I've noticed this to.

>>> I was sort of idly wondering where the speed hit was?  Is it in the
>>> parsing of the .overview file?

It may be in the Unix filesystem itself.  I had a problem
with our newsfeed a few years ago that put about 10,000
files in one directory; it took over a week to remove them
all (using "rm -rf").

Some of the NNML groups that I use for mailing lists have
become quite large (over 1,000 messages) and have started to
slow down.  They are also slow when I use "ls" on the
directory that stores them though.

I started to look at changing the file naming code for the
NNML method so that it would attempt to spread files across
several directories, but I did not have the time to get
anywhere with it.

			- Steve.
-- 
Steve Folkes, N137, DERA Malvern, St Andrews Rd, Malvern, Worcs. WR14 3PS. UK.
   Tel: +44 1684 895649; Fax: +44 1684 894303; Email: smf@rivers.dra.hmg.gb

Re: opening speed for nnml groups

[Lars Balker Rasmussen]

Steve Folkes <smf@rivers.dra.hmg.gb> writes:
> It may be in the Unix filesystem itself.  I had a problem
> with our newsfeed a few years ago that put about 10,000
> files in one directory; it took over a week to remove them
> all (using "rm -rf").

What Unix would that be?

% mkdir test
% perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }'
% time rm -rf test
0.60u 11.18s 1:55.40 10.2%
% uname -a
SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc

> Some of the NNML groups that I use for mailing lists have
> become quite large (over 1,000 messages) and have started to
> slow down.  They are also slow when I use "ls" on the
> directory that stores them though.

ls is primarily slow because it has to sort the files.

But of course, the Unix filesystem gets slower as more files are in a
directory, but it's not _that_ bad.
-- 
Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS
lbr@mjolner.dk

Re: opening speed for nnml groups

[Mark Eichin]

Indeed, 
+% mkdir test
+% time perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }'
1.490u 72.550s 1:31.65 80.7%    0+0k 0+0io 287pf+0w
+% time sync
0.000u 0.040s 0:00.21 19.0%     0+0k 0+0io 21pf+0w
+% time rm -rf test
0.200u 8.120s 0:09.26 89.8%     0+0k 0+0io 86pf+0w

But that's a p133 running linux, run of the mill 2G SCSI disks.

> Some of the NNML groups that I use for mailing lists have
> become quite large (over 1,000 messages) and have started to

Heh -- my mail.misc has 16000 files, mail.personal has 6500... and I
usually enter them with 33 or 99 (whatever is easiest to type) to read
new stuff, or maybe 300 or so if I want to search back a bit.  *That*
isn't slow... but when I hit $ to refile some spam, from mail.misc to
mail.spam (which only has 600 messages at the moment) it takes as long
as 5 to 10 seconds to recover.  I'd guess most of that has to do with
rewriting the 3.5M .overview file in mail.misc -- having 64M of RAM
does help, especially with linux aggressive caching...

Needless to say I'm pushing gnus quite a bit here.  All I really need
to do is write some scripts to refile all of mail.misc during 1996
into archive.1996.mail.misc, or something like that...

Re: opening speed for nnml groups

[Ken Raeburn]

Steinar Bang <sb@metis.no> writes:

> I haven't done any timing of it yet, but it seems like opening nnml
> groups get noticably slower, as the groups get bigger (with more
> ticked or read articles in them).
> 
> I was sort of idly wondering where the speed hit was?  Is it in the
> parsing of the .overview file?

(I forget who pointed this out to me...)
If you use the message date in the summary line, that's probably a big
piece of it.  See my messages of some time ago about regexp ordering
in timezone-parse-date.  Getting the order right (since all email I
tried matched one pattern, and it wasn't the first one) saves a large
fraction of the time spent in that routine.

In a test I just did with ELP and a mail group of some 7800 messages,
timezone-parse-date (probably without fixed regexp ordering, I've been
lame) took 46.9 seconds out of the total of 194.3.

gnus-summary-read-group-1 took 194.3 seconds
  gnus-select-newsgroup took 32
    gnus-get-newsgroup-headers-xover took 29.6 (nov-parse-line done inline)
  gnus-summary-prepare took 152
    gnus-make-threads took 4
    gnus-sort-threads took 12.8
    gnus-summary-prepare-threads took 135.6
      formatting and inserting summary line took ...??

Since my line-format-spec uses the date, let's be generous and say
that gnus-summary-prepare-threads would've taken only 88 seconds
without it, and the total time would've been 147 seconds.  Then
gnus-summary-prepare-threads still looks like the dominant factor at
60%.  But parsing the overview data is still 20%, and could perhaps be
optimized.

Hm.  Interesting.  I ran gnus-compile to compile the specs, and ran my
test again.  Now timezone-parse-date supposedly only took 25.7
seconds, but the total time was still 190, and
gnus-summary-prepare-threads still took 132.5.  Something seems fishy
here...

Oh, and gnus-simplify-subject-fuzzy used about 26 seconds in almost
9800 calls, but I haven't traced down where in the call graph that
would be located.  I wonder if the hacks I did for
gnus-summary-kill-same-subject (more generally, for matching of
simplified subjects) could help here.  (I sent them in 24 May 1996,
and Lars said that same day that he'd include them in Red Gnus.  Still
hasn't happened... :-)

gnus-summary-highlight-line took 14 seconds, in 7873 calls

gnus-sort-threads took 12 seconds; gnus-thread-sort-by-number took 4.5
seconds in 20703 calls

gnus-put-text-property took 9 seconds in 15812 calls

That looks like it for functions taking over 5% of the time.

> I remember that slow exit on big or sparse nnml group, was a problem
> when you did expiry on exiting a group, so I went for demonical
> expiry, as well as demonical fetch and sort.

Me too (for expiry), but it's still a pain to have to wait if I come
back to Emacs while it's running.  I can't even interrupt it with ^G.

> If the reading and parsing of the .overview file is the problem, would
> it be an idea to cache the parsed info between visits? And update it
> during fetch and sort operations?

Or maybe store the data as s-expressions, possibly with extra fields
for caching references links and parsed dates, rather than the
standard news overview format?

> Or is that too expensive memory-wise?

The overview file for this group I was working with is 1.8M.  I doubt
the lisp version would be much smaller.  I wouldn't want to cache that
much info in memory just on the chance that I might come back to that
newsgroup eventually.  Maybe if it got dumped the moment I entered
some other newsgroup; that would permit leaving, fetching new mail,
and re-entering.

Re: opening speed for nnml groups

[Russ Allbery]

Lars Balker Rasmussen <lbr@mjolner.dk> writes:
> Steve Folkes <smf@rivers.dra.hmg.gb> writes:

>> It may be in the Unix filesystem itself.  I had a problem with our
>> newsfeed a few years ago that put about 10,000 files in one directory;
>> it took over a week to remove them all (using "rm -rf").

> What Unix would that be?

> % mkdir test
> % perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }'
> % time rm -rf test
> 0.60u 11.18s 1:55.40 10.2%
> % uname -a
> SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc

A week is a bit long, but when I cut back on the expire time for
control.cancel on our news server from ten days to half a day, it took
fastrm (which is a good bit faster than rm -rf under most circumstances)
the majority of a day to remove the files in the directory, and that was
an UltraSPARC 2 on a RAID array.  The directory itself, not including its
contents, was over 1MB.

> But of course, the Unix filesystem gets slower as more files are in a
> directory, but it's not _that_ bad.

Unless I'm mistaken, Unix stores a directory as a linked list of files.
Each time you open or delete a file, it has to traverse the linked list to
find it, and each time you create a file, it has to traverse the linked
list to find a hole.  It should quickly become clear from that why large
directories cause a slowdown.

(There could also be internal Gnus or emacs performance issues that
overshadow that, such as the .overview handling -- on that score, I have
no idea.)

-- 
Russ Allbery (rra@stanford.edu)         <URL:http://www.eyrie.org/~eagle/>

Re: opening speed for nnml groups

[Sean Doran]

Lars Balker Rasmussen <lbr@mjolner.dk> writes:

> % mkdir test
> % perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }'
> % time rm -rf test
> 0.60u 11.18s 1:55.40 10.2%
> % uname -a
> SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc

Ich, perl is a pig.  This makes me feel good about my
choice for a scripting shell:

 : sean.ebone.net ; uname -a        
NetBSD sean.ebone.net 1.2G NetBSD 1.2G (EBONE) #0: Sun Aug 17 20:19:52 CEST 1997     smd@sean.ebone.net:/usr/src/sys/arch/i386/compile/EBONE i386
: sean.ebone.net ; scsh
Scsh 0.5
> (create-directory "test")
> (run (time scsh -c 
	"(let loop ((i 1))
		(close-output-port 
		 (open-output-file (string-append \"test/\"
				    (number->string i))))
		(if (< i 10001) 
		    (loop (+ i 1))))"))
      175.68 real         6.71 user        68.59 sys
> (run (time rm -rf test))
       94.45 real         0.12 user         3.38 sys
> (create-directory "test")
> (run (time perl -e "foreach $a (1..10000) { open FOO, \">test/$a\" }"))
      198.90 real	  5.81 user	   68.99 sys
> (run (time scsh -c "(begin (for-each (lambda (f)
                                        (delete-file f))
                                       (glob \"./test/*\"))
			     (delete-directory \"test\"))"))
       89.79 real         7.71 user         7.83 sys

Well I cheat in that I use a statically-linked scshvm,
which makes an impressive difference. :)

> ls is primarily slow because it has to sort the files.

Yes, quite likely.  The globbing is certainly the most
CPU-intensive part of the quick rm equivalent.

> But of course, the Unix filesystem gets slower as more files are in a
> directory, but it's not _that_ bad.

Ayup.  I can't see any reason to blame the filesystem for 
nnml's slow behaviour on large groups.   The obvious CPU
eaters are mostly in preparing the *Summary* buffer, 
in the work done upon exiting the group, and in anything
else that does significant marching through and manipulation of
the *Summary* buffer.

Copying, respooling or moving articles also has ugly time
consumption properties having to do with maintaining
.overview files.

	Sean.

Re: opening speed for nnml groups

[Lars Balker Rasmussen]

Sean Doran <smd@ebone.net> writes:
> Ich, perl is a pig.  This makes me feel good about my
> choice for a scripting shell:
[...]
> > (run (time scsh -c 
> 	"(let loop ((i 1))
> 		(close-output-port 
> 		 (open-output-file (string-append \"test/\"
> 				    (number->string i))))
> 		(if (< i 10001) 
> 		    (loop (+ i 1))))"))
>       175.68 real         6.71 user        68.59 sys
> > (run (time perl -e "foreach $a (1..10000) { open FOO, \">test/$a\" }"))
>       198.90 real	  5.81 user	   68.99 sys

Not quite fair, as perl constructs the 1..10000 list in my orignal code
snippet.  Try this instead:

(run (time perl -e "while ($i++ < 10000) { open FOO, \">test/$i\" }"))

BTW, thank you for all the "My Pentium is faster than your Ultra" mails :-)

ObGnus: Are anyone able to (e)dit an nnml message without surprising
	results?
-- 
Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS
lbr@mjolner.dk

Re: opening speed for nnml groups

[Colin Rafferty]

Lars Balker Rasmussen writes:

> ObGnus: Are anyone able to (e)dit an nnml message without surprising
> 	results?

I am.  What is your surprise?

-- 
Colin

Re: opening speed for nnml groups

[David S. Goldberg]

> Lars Balker Rasmussen writes:
>> ObGnus: Are anyone able to (e)dit an nnml message without
>> surprising results?

> I am.  What is your surprise?

So am I, but I suspect he's talking about the surprise one gets if one
uses tm.  In particular, the edit buffer contains tm's representation
of an attachment, not the attachment itself, so when you save it...

Anyway, if using tm, one should turn off Mime before editing.

--
Dave Goldberg
Post: The Mitre Corporation\MS B305\202 Burlington Rd.\Bedford, MA 01730
Phone: 781-271-3887
Email: dsg@mitre.org

Edit MIME messages (was: Re: opening speed for nnml groups)

[Kai Grossjohann]

>>>>> On 05 Sep 1997, David S Goldberg said:

  David> So am I, but I suspect he's talking about the surprise one
  David> gets if one uses tm.  In particular, the edit buffer contains
  David> tm's representation of an attachment, not the attachment
  David> itself, so when you save it...

There used to be a time when this was fixed.  I think.  Has it creeped
back in?

kai
-- 
I like both kinds of music.

Re: Edit MIME messages (was: Re: opening speed for nnml groups)

[David S. Goldberg]

>>>>> On 05 Sep 1997, David S Goldberg said:
[when using tm, editing the article buffer may lose attachments]

> There used to be a time when this was fixed.  I think.  Has it
> creeped back in?

It's repeatable.  If I edit an article without first turning off Mime
(W-m) or using C-u g (thanks to Matt Simmons for that suggestion,
which is a better option IMO), the article buffer will contain the
text of tm's representation of the attachment rather than the
attachment itself.  If I subsequently save the buffer, I lose the
attachment -- all that's left is the text that looks like

[2 97PLAYr11.DOC <application/msword>]

which is from a real example I decided I didn't care about :-)

--
Dave Goldberg
Post: The Mitre Corporation\MS B305\202 Burlington Rd.\Bedford, MA 01730
Phone: 781-271-3887
Email: dsg@mitre.org

Re: Edit MIME messages (was: Re: opening speed for nnml groups)

[Steinar Bang]

>>>>> dsg@linus.mitre.org (David S. Goldberg):

>>>>> On 05 Sep 1997, David S Goldberg said:
> [when using tm, editing the article buffer may lose attachments]

>> There used to be a time when this was fixed.  I think.  Has it
>> creeped back in?

> It's repeatable.  If I edit an article without first turning off Mime
> (W-m) or using C-u g (thanks to Matt Simmons for that suggestion,
> which is a better option IMO), the article buffer will contain the
> text of tm's representation of the attachment rather than the
> attachment itself.  If I subsequently save the buffer, I lose the
> attachment -- all that's left is the text that looks like

> [2 97PLAYr11.DOC <application/msword>]

> which is from a real example I decided I didn't care about :-)

Hmm... version 7.106 of tm... same version I'm using... better check...

Re: opening speed for nnml groups

[Lars Balker Rasmussen]

dsg@linus.mitre.org (David S. Goldberg) writes:
> > Lars Balker Rasmussen writes:
> >> ObGnus: Are anyone able to (e)dit an nnml message without
> >> surprising results?
> 
> > I am.  What is your surprise?
> 
> So am I, but I suspect he's talking about the surprise one gets if one
> uses tm. [...]

No, I'm not, as I don't use tm. (I'm sorry this took some time, but I've
been busy :) )

When I edit a message[1], and I review the message (after having viewed
some other message), the contents of the generated Article buffer
appears to be the backlog[3].

Playing around with this, it appears it's not just a backlog bug.  With
gnus-keep-backlog at 0, after editing one message, I press up and space
to view the former message and back down again to view the edited
message, the Article buffer will still show the former message!

I don't think Lars has been able to reproduce this one in the past, so I
hope someone else is :-)

[1] Particularly for deleting larger parts of it[2]
[2] Any other reason to edit other peoples messsages?
[3] I have gnus-keep-backlog set at 10
-- 
Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS
lbr@mjolner.dk

Re: opening speed for nnml groups

[Colin Rafferty]

Lars Balker Rasmussen writes:

> When I edit a message[1], and I review the message (after having viewed
> some other message), the contents of the generated Article buffer
> appears to be the backlog[3].

> Playing around with this, it appears it's not just a backlog bug.  With
> gnus-keep-backlog at 0, after editing one message, I press up and space
> to view the former message and back down again to view the edited
> message, the Article buffer will still show the former message!

Actually, that still sounds like a backlog bug.  Setting it to 0 should
still be activating the keep-backlog code.

I would suggest setting it to nil (not zero), and see if you have the
same problems.

-- 
Colin

Re: opening speed for nnml groups

[Lars Magne Ingebrigtsen]

Lars Balker Rasmussen <lbr@mjolner.dk> writes:

> When I edit a message[1], and I review the message (after having viewed
> some other message), the contents of the generated Article buffer
> appears to be the backlog[3].

Yup.  Fix in Quassia Gnus v0.3.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen