From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.emacs.gnus.general/11968 Path: main.gmane.org!not-for-mail From: Russ Allbery Newsgroups: gmane.emacs.gnus.general Subject: Re: opening speed for nnml groups Date: 03 Sep 1997 14:26:12 -0700 Sender: eagle@windlord.Stanford.EDU Message-ID: References: <0f4t82zbjt.fsf@odin.mjolner.dk> NNTP-Posting-Host: coloc-standby.netfonds.no X-Trace: main.gmane.org 1035151589 1139 80.91.224.250 (20 Oct 2002 22:06:29 GMT) X-Complaints-To: usenet@main.gmane.org NNTP-Posting-Date: Sun, 20 Oct 2002 22:06:29 +0000 (UTC) Return-Path: Original-Received: from xemacs.org (xemacs.cs.uiuc.edu [128.174.252.16]) by altair.xemacs.org (8.8.7/8.8.7) with ESMTP id QAA16449 for ; Wed, 3 Sep 1997 16:18:56 -0700 Original-Received: from ifi.uio.no (0@ifi.uio.no [129.240.64.2]) by xemacs.org (8.8.5/8.8.5) with SMTP id SAA28125 for ; Wed, 3 Sep 1997 18:14:52 -0500 (CDT) Original-Received: from maud.ifi.uio.no (0@maud.ifi.uio.no [129.240.64.19]) by ifi.uio.no with ESMTP (8.6.11/ifi2.4) id for ; Wed, 3 Sep 1997 23:57:02 +0200 Original-Received: from claymore.vcinet.com (claymore.vcinet.com [208.205.12.23]) by maud.ifi.uio.no ; Wed, 3 Sep 1997 23:54:04 +0200 Original-Received: (qmail 5611 invoked by uid 504); 3 Sep 1997 21:26:20 -0000 Original-Received: (qmail 5608 invoked from network); 3 Sep 1997 21:26:20 -0000 Original-Received: from windlord.stanford.edu (36.21.0.44) by claymore.vcinet.com with SMTP; 3 Sep 1997 21:26:20 -0000 Original-Received: (qmail 26720 invoked by uid 500); 3 Sep 1997 21:26:12 -0000 Original-To: ding@gnus.org In-Reply-To: Lars Balker Rasmussen's message of 03 Sep 1997 17:01:26 +0200 Original-Lines: 38 X-Mailer: Gnus v5.3/Emacs 19.34 Xref: main.gmane.org gmane.emacs.gnus.general:11968 X-Report-Spam: http://spam.gmane.org/gmane.emacs.gnus.general:11968 Lars Balker Rasmussen writes: > Steve Folkes writes: >> It may be in the Unix filesystem itself. I had a problem with our >> newsfeed a few years ago that put about 10,000 files in one directory; >> it took over a week to remove them all (using "rm -rf"). > What Unix would that be? > % mkdir test > % perl -e 'foreach $a (1..10000) { open FOO, ">test/$a" }' > % time rm -rf test > 0.60u 11.18s 1:55.40 10.2% > % uname -a > SunOS heimdal 5.5.1 Generic_103640-08 sun4u sparc A week is a bit long, but when I cut back on the expire time for control.cancel on our news server from ten days to half a day, it took fastrm (which is a good bit faster than rm -rf under most circumstances) the majority of a day to remove the files in the directory, and that was an UltraSPARC 2 on a RAID array. The directory itself, not including its contents, was over 1MB. > But of course, the Unix filesystem gets slower as more files are in a > directory, but it's not _that_ bad. Unless I'm mistaken, Unix stores a directory as a linked list of files. Each time you open or delete a file, it has to traverse the linked list to find it, and each time you create a file, it has to traverse the linked list to find a hole. It should quickly become clear from that why large directories cause a slowdown. (There could also be internal Gnus or emacs performance issues that overshadow that, such as the .overview handling -- on that score, I have no idea.) -- Russ Allbery (rra@stanford.edu)