* opening speed for nnml groups @ 1997-09-02 12:01 Steinar Bang 1997-09-03 13:05 ` Steve Folkes 1997-09-03 20:33 ` Ken Raeburn 0 siblings, 2 replies; 16+ messages in thread From: Steinar Bang @ 1997-09-02 12:01 UTC (permalink / raw) I haven't done any timing of it yet, but it seems like opening nnml groups get noticably slower, as the groups get bigger (with more ticked or read articles in them). I was sort of idly wondering where the speed hit was? Is it in the parsing of the .overview file? I remember that slow exit on big or sparse nnml group, was a problem when you did expiry on exiting a group, so I went for demonical expiry, as well as demonical fetch and sort. If the reading and parsing of the .overview file is the problem, would it be an idea to cache the parsed info between visits? And update it during fetch and sort operations? Or is that too expensive memory-wise? ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-02 12:01 opening speed for nnml groups Steinar Bang @ 1997-09-03 13:05 ` Steve Folkes 1997-09-03 15:01 ` Lars Balker Rasmussen 1997-09-03 20:33 ` Ken Raeburn 1 sibling, 1 reply; 16+ messages in thread From: Steve Folkes @ 1997-09-03 13:05 UTC (permalink / raw) Steinar Bang <sb@metis.no> writes: >>> I haven't done any timing of it yet, but it seems like opening nnml >>> groups get noticably slower, as the groups get bigger (with more >>> ticked or read articles in them). I've noticed this to. >>> I was sort of idly wondering where the speed hit was? Is it in the >>> parsing of the .overview file? It may be in the Unix filesystem itself. I had a problem with our newsfeed a few years ago that put about 10,000 files in one directory; it took over a week to remove them all (using "rm -rf"). Some of the NNML groups that I use for mailing lists have become quite large (over 1,000 messages) and have started to slow down. They are also slow when I use "ls" on the directory that stores them though. I started to look at changing the file naming code for the NNML method so that it would attempt to spread files across several directories, but I did not have the time to get anywhere with it. - Steve. -- Steve Folkes, N137, DERA Malvern, St Andrews Rd, Malvern, Worcs. WR14 3PS. UK. Tel: +44 1684 895649; Fax: +44 1684 894303; Email: smf@rivers.dra.hmg.gb ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-03 13:05 ` Steve Folkes @ 1997-09-03 15:01 ` Lars Balker Rasmussen 1997-09-03 19:45 ` Mark Eichin ` (2 more replies) 0 siblings, 3 replies; 16+ messages in thread From: Lars Balker Rasmussen @ 1997-09-03 15:01 UTC (permalink / raw) Steve Folkes <smf@rivers.dra.hmg.gb> writes: > It may be in the Unix filesystem itself. I had a problem > with our newsfeed a few years ago that put about 10,000 > files in one directory; it took over a week to remove them > all (using "rm -rf"). What Unix would that be? % mkdir test % perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }' % time rm -rf test 0.60u 11.18s 1:55.40 10.2% % uname -a SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc > Some of the NNML groups that I use for mailing lists have > become quite large (over 1,000 messages) and have started to > slow down. They are also slow when I use "ls" on the > directory that stores them though. ls is primarily slow because it has to sort the files. But of course, the Unix filesystem gets slower as more files are in a directory, but it's not _that_ bad. -- Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS lbr@mjolner.dk ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-03 15:01 ` Lars Balker Rasmussen @ 1997-09-03 19:45 ` Mark Eichin 1997-09-03 21:26 ` Russ Allbery 1997-09-04 14:55 ` Sean Doran 2 siblings, 0 replies; 16+ messages in thread From: Mark Eichin @ 1997-09-03 19:45 UTC (permalink / raw) Cc: ding Indeed, +% mkdir test +% time perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }' 1.490u 72.550s 1:31.65 80.7% 0+0k 0+0io 287pf+0w +% time sync 0.000u 0.040s 0:00.21 19.0% 0+0k 0+0io 21pf+0w +% time rm -rf test 0.200u 8.120s 0:09.26 89.8% 0+0k 0+0io 86pf+0w But that's a p133 running linux, run of the mill 2G SCSI disks. > Some of the NNML groups that I use for mailing lists have > become quite large (over 1,000 messages) and have started to Heh -- my mail.misc has 16000 files, mail.personal has 6500... and I usually enter them with 33 or 99 (whatever is easiest to type) to read new stuff, or maybe 300 or so if I want to search back a bit. *That* isn't slow... but when I hit $ to refile some spam, from mail.misc to mail.spam (which only has 600 messages at the moment) it takes as long as 5 to 10 seconds to recover. I'd guess most of that has to do with rewriting the 3.5M .overview file in mail.misc -- having 64M of RAM does help, especially with linux aggressive caching... Needless to say I'm pushing gnus quite a bit here. All I really need to do is write some scripts to refile all of mail.misc during 1996 into archive.1996.mail.misc, or something like that... ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-03 15:01 ` Lars Balker Rasmussen 1997-09-03 19:45 ` Mark Eichin @ 1997-09-03 21:26 ` Russ Allbery 1997-09-04 14:55 ` Sean Doran 2 siblings, 0 replies; 16+ messages in thread From: Russ Allbery @ 1997-09-03 21:26 UTC (permalink / raw) Lars Balker Rasmussen <lbr@mjolner.dk> writes: > Steve Folkes <smf@rivers.dra.hmg.gb> writes: >> It may be in the Unix filesystem itself. I had a problem with our >> newsfeed a few years ago that put about 10,000 files in one directory; >> it took over a week to remove them all (using "rm -rf"). > What Unix would that be? > % mkdir test > % perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }' > % time rm -rf test > 0.60u 11.18s 1:55.40 10.2% > % uname -a > SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc A week is a bit long, but when I cut back on the expire time for control.cancel on our news server from ten days to half a day, it took fastrm (which is a good bit faster than rm -rf under most circumstances) the majority of a day to remove the files in the directory, and that was an UltraSPARC 2 on a RAID array. The directory itself, not including its contents, was over 1MB. > But of course, the Unix filesystem gets slower as more files are in a > directory, but it's not _that_ bad. Unless I'm mistaken, Unix stores a directory as a linked list of files. Each time you open or delete a file, it has to traverse the linked list to find it, and each time you create a file, it has to traverse the linked list to find a hole. It should quickly become clear from that why large directories cause a slowdown. (There could also be internal Gnus or emacs performance issues that overshadow that, such as the .overview handling -- on that score, I have no idea.) -- Russ Allbery (rra@stanford.edu) <URL:http://www.eyrie.org/~eagle/> ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-03 15:01 ` Lars Balker Rasmussen 1997-09-03 19:45 ` Mark Eichin 1997-09-03 21:26 ` Russ Allbery @ 1997-09-04 14:55 ` Sean Doran 1997-09-04 15:39 ` Lars Balker Rasmussen 2 siblings, 1 reply; 16+ messages in thread From: Sean Doran @ 1997-09-04 14:55 UTC (permalink / raw) Lars Balker Rasmussen <lbr@mjolner.dk> writes: > % mkdir test > % perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }' > % time rm -rf test > 0.60u 11.18s 1:55.40 10.2% > % uname -a > SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc Ich, perl is a pig. This makes me feel good about my choice for a scripting shell: : sean.ebone.net ; uname -a NetBSD sean.ebone.net 1.2G NetBSD 1.2G (EBONE) #0: Sun Aug 17 20:19:52 CEST 1997 smd@sean.ebone.net:/usr/src/sys/arch/i386/compile/EBONE i386 : sean.ebone.net ; scsh Scsh 0.5 > (create-directory "test") > (run (time scsh -c "(let loop ((i 1)) (close-output-port (open-output-file (string-append \"test/\" (number->string i)))) (if (< i 10001) (loop (+ i 1))))")) 175.68 real 6.71 user 68.59 sys > (run (time rm -rf test)) 94.45 real 0.12 user 3.38 sys > (create-directory "test") > (run (time perl -e "foreach $a (1..10000) { open FOO, \">test/$a\" }")) 198.90 real 5.81 user 68.99 sys > (run (time scsh -c "(begin (for-each (lambda (f) (delete-file f)) (glob \"./test/*\")) (delete-directory \"test\"))")) 89.79 real 7.71 user 7.83 sys Well I cheat in that I use a statically-linked scshvm, which makes an impressive difference. :) > ls is primarily slow because it has to sort the files. Yes, quite likely. The globbing is certainly the most CPU-intensive part of the quick rm equivalent. > But of course, the Unix filesystem gets slower as more files are in a > directory, but it's not _that_ bad. Ayup. I can't see any reason to blame the filesystem for nnml's slow behaviour on large groups. The obvious CPU eaters are mostly in preparing the *Summary* buffer, in the work done upon exiting the group, and in anything else that does significant marching through and manipulation of the *Summary* buffer. Copying, respooling or moving articles also has ugly time consumption properties having to do with maintaining .overview files. Sean. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-04 14:55 ` Sean Doran @ 1997-09-04 15:39 ` Lars Balker Rasmussen 1997-09-04 21:44 ` Colin Rafferty 0 siblings, 1 reply; 16+ messages in thread From: Lars Balker Rasmussen @ 1997-09-04 15:39 UTC (permalink / raw) Sean Doran <smd@ebone.net> writes: > Ich, perl is a pig. This makes me feel good about my > choice for a scripting shell: [...] > > (run (time scsh -c > "(let loop ((i 1)) > (close-output-port > (open-output-file (string-append \"test/\" > (number->string i)))) > (if (< i 10001) > (loop (+ i 1))))")) > 175.68 real 6.71 user 68.59 sys > > (run (time perl -e "foreach $a (1..10000) { open FOO, \">test/$a\" }")) > 198.90 real 5.81 user 68.99 sys Not quite fair, as perl constructs the 1..10000 list in my orignal code snippet. Try this instead: (run (time perl -e "while ($i++ < 10000) { open FOO, \">test/$i\" }")) BTW, thank you for all the "My Pentium is faster than your Ultra" mails :-) ObGnus: Are anyone able to (e)dit an nnml message without surprising results? -- Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS lbr@mjolner.dk ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-04 15:39 ` Lars Balker Rasmussen @ 1997-09-04 21:44 ` Colin Rafferty 1997-09-05 12:20 ` David S. Goldberg 0 siblings, 1 reply; 16+ messages in thread From: Colin Rafferty @ 1997-09-04 21:44 UTC (permalink / raw) Lars Balker Rasmussen writes: > ObGnus: Are anyone able to (e)dit an nnml message without surprising > results? I am. What is your surprise? -- Colin ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-04 21:44 ` Colin Rafferty @ 1997-09-05 12:20 ` David S. Goldberg 1997-09-05 15:21 ` Edit MIME messages (was: Re: opening speed for nnml groups) Kai Grossjohann 1997-09-10 8:43 ` opening speed for nnml groups Lars Balker Rasmussen 0 siblings, 2 replies; 16+ messages in thread From: David S. Goldberg @ 1997-09-05 12:20 UTC (permalink / raw) Cc: GNUS Mailing List > Lars Balker Rasmussen writes: >> ObGnus: Are anyone able to (e)dit an nnml message without >> surprising results? > I am. What is your surprise? So am I, but I suspect he's talking about the surprise one gets if one uses tm. In particular, the edit buffer contains tm's representation of an attachment, not the attachment itself, so when you save it... Anyway, if using tm, one should turn off Mime before editing. -- Dave Goldberg Post: The Mitre Corporation\MS B305\202 Burlington Rd.\Bedford, MA 01730 Phone: 781-271-3887 Email: dsg@mitre.org ^ permalink raw reply [flat|nested] 16+ messages in thread
* Edit MIME messages (was: Re: opening speed for nnml groups) 1997-09-05 12:20 ` David S. Goldberg @ 1997-09-05 15:21 ` Kai Grossjohann 1997-09-05 15:57 ` David S. Goldberg 1997-09-10 8:43 ` opening speed for nnml groups Lars Balker Rasmussen 1 sibling, 1 reply; 16+ messages in thread From: Kai Grossjohann @ 1997-09-05 15:21 UTC (permalink / raw) Cc: Colin Rafferty, GNUS Mailing List >>>>> On 05 Sep 1997, David S Goldberg said: David> So am I, but I suspect he's talking about the surprise one David> gets if one uses tm. In particular, the edit buffer contains David> tm's representation of an attachment, not the attachment David> itself, so when you save it... There used to be a time when this was fixed. I think. Has it creeped back in? kai -- I like both kinds of music. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Edit MIME messages (was: Re: opening speed for nnml groups) 1997-09-05 15:21 ` Edit MIME messages (was: Re: opening speed for nnml groups) Kai Grossjohann @ 1997-09-05 15:57 ` David S. Goldberg 1997-09-09 7:06 ` Steinar Bang 0 siblings, 1 reply; 16+ messages in thread From: David S. Goldberg @ 1997-09-05 15:57 UTC (permalink / raw) Cc: GNUS Mailing List >>>>> On 05 Sep 1997, David S Goldberg said: [when using tm, editing the article buffer may lose attachments] > There used to be a time when this was fixed. I think. Has it > creeped back in? It's repeatable. If I edit an article without first turning off Mime (W-m) or using C-u g (thanks to Matt Simmons for that suggestion, which is a better option IMO), the article buffer will contain the text of tm's representation of the attachment rather than the attachment itself. If I subsequently save the buffer, I lose the attachment -- all that's left is the text that looks like [2 97PLAYr11.DOC <application/msword>] which is from a real example I decided I didn't care about :-) -- Dave Goldberg Post: The Mitre Corporation\MS B305\202 Burlington Rd.\Bedford, MA 01730 Phone: 781-271-3887 Email: dsg@mitre.org ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Edit MIME messages (was: Re: opening speed for nnml groups) 1997-09-05 15:57 ` David S. Goldberg @ 1997-09-09 7:06 ` Steinar Bang 0 siblings, 0 replies; 16+ messages in thread From: Steinar Bang @ 1997-09-09 7:06 UTC (permalink / raw) >>>>> dsg@linus.mitre.org (David S. Goldberg): >>>>> On 05 Sep 1997, David S Goldberg said: > [when using tm, editing the article buffer may lose attachments] >> There used to be a time when this was fixed. I think. Has it >> creeped back in? > It's repeatable. If I edit an article without first turning off Mime > (W-m) or using C-u g (thanks to Matt Simmons for that suggestion, > which is a better option IMO), the article buffer will contain the > text of tm's representation of the attachment rather than the > attachment itself. If I subsequently save the buffer, I lose the > attachment -- all that's left is the text that looks like > [2 97PLAYr11.DOC <application/msword>] > which is from a real example I decided I didn't care about :-) Hmm... version 7.106 of tm... same version I'm using... better check... ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-05 12:20 ` David S. Goldberg 1997-09-05 15:21 ` Edit MIME messages (was: Re: opening speed for nnml groups) Kai Grossjohann @ 1997-09-10 8:43 ` Lars Balker Rasmussen 1997-09-10 15:07 ` Colin Rafferty 1997-09-13 23:54 ` Lars Magne Ingebrigtsen 1 sibling, 2 replies; 16+ messages in thread From: Lars Balker Rasmussen @ 1997-09-10 8:43 UTC (permalink / raw) dsg@linus.mitre.org (David S. Goldberg) writes: > > Lars Balker Rasmussen writes: > >> ObGnus: Are anyone able to (e)dit an nnml message without > >> surprising results? > > > I am. What is your surprise? > > So am I, but I suspect he's talking about the surprise one gets if one > uses tm. [...] No, I'm not, as I don't use tm. (I'm sorry this took some time, but I've been busy :) ) When I edit a message[1], and I review the message (after having viewed some other message), the contents of the generated Article buffer appears to be the backlog[3]. Playing around with this, it appears it's not just a backlog bug. With gnus-keep-backlog at 0, after editing one message, I press up and space to view the former message and back down again to view the edited message, the Article buffer will still show the former message! I don't think Lars has been able to reproduce this one in the past, so I hope someone else is :-) [1] Particularly for deleting larger parts of it[2] [2] Any other reason to edit other peoples messsages? [3] I have gnus-keep-backlog set at 10 -- Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS lbr@mjolner.dk ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-10 8:43 ` opening speed for nnml groups Lars Balker Rasmussen @ 1997-09-10 15:07 ` Colin Rafferty 1997-09-13 23:54 ` Lars Magne Ingebrigtsen 1 sibling, 0 replies; 16+ messages in thread From: Colin Rafferty @ 1997-09-10 15:07 UTC (permalink / raw) Lars Balker Rasmussen writes: > When I edit a message[1], and I review the message (after having viewed > some other message), the contents of the generated Article buffer > appears to be the backlog[3]. > Playing around with this, it appears it's not just a backlog bug. With > gnus-keep-backlog at 0, after editing one message, I press up and space > to view the former message and back down again to view the edited > message, the Article buffer will still show the former message! Actually, that still sounds like a backlog bug. Setting it to 0 should still be activating the keep-backlog code. I would suggest setting it to nil (not zero), and see if you have the same problems. -- Colin ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-10 8:43 ` opening speed for nnml groups Lars Balker Rasmussen 1997-09-10 15:07 ` Colin Rafferty @ 1997-09-13 23:54 ` Lars Magne Ingebrigtsen 1 sibling, 0 replies; 16+ messages in thread From: Lars Magne Ingebrigtsen @ 1997-09-13 23:54 UTC (permalink / raw) Lars Balker Rasmussen <lbr@mjolner.dk> writes: > When I edit a message[1], and I review the message (after having viewed > some other message), the contents of the generated Article buffer > appears to be the backlog[3]. Yup. Fix in Quassia Gnus v0.3. -- (domestic pets only, the antidote for overdose, milk.) larsi@gnus.org * Lars Magne Ingebrigtsen ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: opening speed for nnml groups 1997-09-02 12:01 opening speed for nnml groups Steinar Bang 1997-09-03 13:05 ` Steve Folkes @ 1997-09-03 20:33 ` Ken Raeburn 1 sibling, 0 replies; 16+ messages in thread From: Ken Raeburn @ 1997-09-03 20:33 UTC (permalink / raw) Steinar Bang <sb@metis.no> writes: > I haven't done any timing of it yet, but it seems like opening nnml > groups get noticably slower, as the groups get bigger (with more > ticked or read articles in them). > > I was sort of idly wondering where the speed hit was? Is it in the > parsing of the .overview file? (I forget who pointed this out to me...) If you use the message date in the summary line, that's probably a big piece of it. See my messages of some time ago about regexp ordering in timezone-parse-date. Getting the order right (since all email I tried matched one pattern, and it wasn't the first one) saves a large fraction of the time spent in that routine. In a test I just did with ELP and a mail group of some 7800 messages, timezone-parse-date (probably without fixed regexp ordering, I've been lame) took 46.9 seconds out of the total of 194.3. gnus-summary-read-group-1 took 194.3 seconds gnus-select-newsgroup took 32 gnus-get-newsgroup-headers-xover took 29.6 (nov-parse-line done inline) gnus-summary-prepare took 152 gnus-make-threads took 4 gnus-sort-threads took 12.8 gnus-summary-prepare-threads took 135.6 formatting and inserting summary line took ...?? Since my line-format-spec uses the date, let's be generous and say that gnus-summary-prepare-threads would've taken only 88 seconds without it, and the total time would've been 147 seconds. Then gnus-summary-prepare-threads still looks like the dominant factor at 60%. But parsing the overview data is still 20%, and could perhaps be optimized. Hm. Interesting. I ran gnus-compile to compile the specs, and ran my test again. Now timezone-parse-date supposedly only took 25.7 seconds, but the total time was still 190, and gnus-summary-prepare-threads still took 132.5. Something seems fishy here... Oh, and gnus-simplify-subject-fuzzy used about 26 seconds in almost 9800 calls, but I haven't traced down where in the call graph that would be located. I wonder if the hacks I did for gnus-summary-kill-same-subject (more generally, for matching of simplified subjects) could help here. (I sent them in 24 May 1996, and Lars said that same day that he'd include them in Red Gnus. Still hasn't happened... :-) gnus-summary-highlight-line took 14 seconds, in 7873 calls gnus-sort-threads took 12 seconds; gnus-thread-sort-by-number took 4.5 seconds in 20703 calls gnus-put-text-property took 9 seconds in 15812 calls That looks like it for functions taking over 5% of the time. > I remember that slow exit on big or sparse nnml group, was a problem > when you did expiry on exiting a group, so I went for demonical > expiry, as well as demonical fetch and sort. Me too (for expiry), but it's still a pain to have to wait if I come back to Emacs while it's running. I can't even interrupt it with ^G. > If the reading and parsing of the .overview file is the problem, would > it be an idea to cache the parsed info between visits? And update it > during fetch and sort operations? Or maybe store the data as s-expressions, possibly with extra fields for caching references links and parsed dates, rather than the standard news overview format? > Or is that too expensive memory-wise? The overview file for this group I was working with is 1.8M. I doubt the lisp version would be much smaller. I wouldn't want to cache that much info in memory just on the chance that I might come back to that newsgroup eventually. Maybe if it got dumped the moment I entered some other newsgroup; that would permit leaving, fetching new mail, and re-entering. ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~1997-09-13 23:54 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 1997-09-02 12:01 opening speed for nnml groups Steinar Bang 1997-09-03 13:05 ` Steve Folkes 1997-09-03 15:01 ` Lars Balker Rasmussen 1997-09-03 19:45 ` Mark Eichin 1997-09-03 21:26 ` Russ Allbery 1997-09-04 14:55 ` Sean Doran 1997-09-04 15:39 ` Lars Balker Rasmussen 1997-09-04 21:44 ` Colin Rafferty 1997-09-05 12:20 ` David S. Goldberg 1997-09-05 15:21 ` Edit MIME messages (was: Re: opening speed for nnml groups) Kai Grossjohann 1997-09-05 15:57 ` David S. Goldberg 1997-09-09 7:06 ` Steinar Bang 1997-09-10 8:43 ` opening speed for nnml groups Lars Balker Rasmussen 1997-09-10 15:07 ` Colin Rafferty 1997-09-13 23:54 ` Lars Magne Ingebrigtsen 1997-09-03 20:33 ` Ken Raeburn
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).