Re: [9fans] Re: [sources] 20070410: % cat

Re: [9fans] Re: [sources] 20070410: % cat

[Lucio De Re]

> Then you need two different sets of policy files and to have your
> initialisation bind the right ones into place.  I'm doing this now for
> a set of diverse machines with different keys, policies, etc. all
> sharing a single file server.

That does not contradict my statement that it does not scale.  Much as
I appreciate the philosophical value of bind/mount, a trillion
instances of a configuration file are going to be unmanageable.  Ron
is right that there is no slick solution, but it's worth knowing that
the current approach deserves exploring further.

++L

Re: [9fans] Re: [sources] 20070410: % cat

[Devon H. O'Dell]

2007/4/13, Lucio De Re <lucio@proxima.alt.za>:
> > Then you need two different sets of policy files and to have your
> > initialisation bind the right ones into place.  I'm doing this now for
> > a set of diverse machines with different keys, policies, etc. all
> > sharing a single file server.
>
> That does not contradict my statement that it does not scale.  Much as
> I appreciate the philosophical value of bind/mount, a trillion
> instances of a configuration file are going to be unmanageable.  Ron
> is right that there is no slick solution, but it's worth knowing that
> the current approach deserves exploring further.
>
> ++L

Do you know anybody who has ever come up with a better idea? This is
very similar to Maildir replacing mbox. It sounds good, but when you
have 250k messages, it ends up being slower simply to traverse the
directory. In FreeBSD anyway.

However, it does scale. Much, much better than a case where you have
the same requirement for thousands of systems in a switch () statment.

--dho

Re: [9fans] Re: [sources] 20070410: % cat

[erik quanstrom]

i think mdir was done to solve nfs locking problems, not
for performance.

for 250k messages, either solution is poor.  deleting the
first of 250k messages requires rewriting the whole mbox.
dealing with 250k directory entries can be painful, too,
as many fs keep directores as arrays.

if you have that many messages, you might want an index. ;-)

- erik

Re: [9fans] Re: [sources] 20070410: % cat

[Lucio De Re]

> Do you know anybody who has ever come up with a better idea? This is
> very similar to Maildir replacing mbox. It sounds good, but when you
> have 250k messages, it ends up being slower simply to traverse the
> directory. In FreeBSD anyway.
> 
A valid comment, I must grant.  I guess I'm hoping the better idea
will not be overlooked by a NIH when it does present itself.

> However, it does scale. Much, much better than a case where you have
> the same requirement for thousands of systems in a switch () statment.

Also valid.  But a database lookup would be even better, would it not?
That has got to be kept in mind, like or not.  Thinking out of the box
is called for, will it ever happen?

++L

Re: [9fans] Re: [sources] 20070410: % cat

[Lyndon Nerenberg]

> for 250k messages, either solution is poor.  deleting the
> first of 250k messages requires rewriting the whole mbox.
> dealing with 250k directory entries can be painful, too,
> as many fs keep directores as arrays.
>
> if you have that many messages, you might want an index. ;-)

I've said it before, and I'll keep saying it until I'm proved wrong:

   nothing runs faster or scales better than the Cyrus IMAP mail store.

MH filesystem layout with per-folder index + header cache.  I have 
deployed mail servers with literally millions of user accounts using this 
layout, and it just works.

These days I don't find large directories to be a problem.  A few months 
ago I did some performance tests on large directory operations (create, 
unlink, readdir) on Linux and FreeBSD.  For both OSes, directory 
enumeration times were negligible for the tests I ran (in the vicinity of 
200K entries).  And with everyone caching directory entries these days, 
the only real difference between directory I/O and file I/O is the 
locking overhead for directory modifications.  (This all assumes local 
disk. Throw in NFS and everything implodes.)

But scalability aside, the real evilness in mbox is the quoting of 
'^From '.

--lyndon

Re: [9fans] Re: [sources] 20070410: % cat

[Paweł Lasek]

On 4/13/07, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:
> > for 250k messages, either solution is poor.  deleting the
> > first of 250k messages requires rewriting the whole mbox.
> > dealing with 250k directory entries can be painful, too,
> > as many fs keep directores as arrays.
> >
> > if you have that many messages, you might want an index. ;-)


> These days I don't find large directories to be a problem.  A few months
> ago I did some performance tests on large directory operations (create,
> unlink, readdir) on Linux and FreeBSD.  For both OSes, directory
> enumeration times were negligible for the tests I ran (in the vicinity of
> 200K entries).  And with everyone caching directory entries these days,
> the only real difference between directory I/O and file I/O is the
> locking overhead for directory modifications.  (This all assumes local
> disk. Throw in NFS and everything implodes.)

That's also thanks to improved filesystems. Modern ones (at least in
Linux, I don't know about *BSD as I didn't use them that extentively)
use weird/complicated/WTF trees to keep directory (and not only) data.
I am not sure if HFS+ doesn't use it also for several other things, as
at least one of the developers working on it came from Be where he
worked on BeFS, which was basically a database with hierarchical
interface...

I don't know about other fs'es, but I have never had problem with
directory entries on XFS. Or anything else, as I never managed to
break it despite being designed for UPS-backed servers (It caches very
aggressively, especially writes) while ReiserFS gave up very fast :-)

BTW, wouldn't XFS be a nice addition to Plan9? Plan9-specific things
can be easily put into  extended attributes, just make sure that
inodes will have larger-than-default size so that xattr access would
be blazing fast....

> --lyndon
>


-- 
Paul Lasek

Re: [9fans] Re: [sources] 20070410: % cat

[Russ Cox]

> That's also thanks to improved filesystems. 

This isn't true.

The savings are due to aggressive caching in the
operating system, which cuts the file system out
entirely.  The disks are slow no matter how you 
use them.

File I/O for things like Python and TeX is slower
on Plan 9 because it has to go to the file server,
which is in another process or another machine,
rather than answer out of a kernel cache like Linux
does. 

Lots of things get easier if you assume you are the
only machine in the universe.  See earlier discussion
on caching read EODs.

Russ

Re: [9fans] Re: [sources] 20070410: % cat

[geoff]

Then you need two different sets of policy files and to have your
initialisation bind the right ones into place.  I'm doing this now for
a set of diverse machines with different keys, policies, etc. all
sharing a single file server.

Re: [9fans] Re: [sources] 20070410: % cat

[geoff]

PXE booting is not likely to change much.  Dave Presotto did much of
that work and he's now gone.  /cfg/pxe is here to stay, as far as I
can see.

Re: [9fans] Re: [sources] 20070410: % cat

[Lucio De Re]

> /cfg is intended for machines with per-node configuration.
> If you have 1000 machines that are functionally identical,
> you're not supposed to write /cfg directories for any of them.

Still doesn't scale: what if you have *two* sets of 1024 machines with
slightly different configurations in the two groups?

++L

Re: [9fans] Re: [sources] 20070410: % cat

[Lucio De Re]

> We have also pushed essentially all files unique to a single machine
> into /cfg/$sysname here.  I pushed out an updated cpurc(8) a couple of
> weeks ago and am now bringing the start-up scripts on sources up to
> date, in stages, to match it, in part to minimise the differences
> between what we run and what's on sources.

I understood this was the role of /sys/lib/sysconfig, but I can easily
understand the desire to change.  Jim suggested, however, that PXE
handling wasn't cast in stone and that might have affected the
stability of /cfg/pxe.  I suppose this means we need a bull from Bell
Labs to pin this down for posterity?

++L

Re: [9fans] Re: [sources] 20070410: % cat

[Anthony Sorace]

//To start with, Plan 9 can handle as many things in the
//root as you like.  It is not constrained by the size
//of an rk05 or the time to fsck the root partition.

Of course; as I said, it's largely an aesthetic decision. But I do
believe it has an impact on *people* reading /, rather than programs
reading it. And /sys/lib feels like a very natural fit for exactly
this sort of information.

Yes, /cfg is provably shorter to type than /sys/lib/sysconfig... but
you really believe that matters here? I don't dispute the impact of
length on use generally, but this is seldom-changing config
information. Is it really true that nothing/nobody ever used sysconfig
(contrary to the cpurc man page)?

Personally, i think the argument of "so we don't have to edit
cpurc/termrc" is vacuous. They change infrequently enough, and in
small enough ways each time, that merging changes is simple. I think
the fear (as Russ pointed to elsewhere) of moving towards init.d is
much more legitimate.

Re: [9fans] Re: [sources] 20070410: % cat

[Devon H. O'Dell]

2007/4/12, geoff@plan9.bell-labs.com <geoff@plan9.bell-labs.com>:
> You only need files in /cfg for a machine if that machine is an
> exception to your normal policies (in cpurc, namespace, consoledb,
> etc.).  I would expect that a cluster would need zero or one such
> exceptions, as the machines are supposed to be interchangeable, right?

In my mind, this is not entirely the case. The other reason to use it
is to avoid actually touching cpurc, so that you don't have to
remember to type -s '/rc/bin/cpurc' when I pull, after backing it up
and merging any changes back in. Since pull is the primary means of
keeping the system up to date, it's a real pain to try to merge all
these changes.

--dho

Re: [9fans] Re: [sources] 20070410: % cat

[geoff]

You only need files in /cfg for a machine if that machine is an
exception to your normal policies (in cpurc, namespace, consoledb,
etc.).  I would expect that a cluster would need zero or one such
exceptions, as the machines are supposed to be interchangeable, right?

Re: [9fans] Re: [sources] 20070410: % cat

[Russ Cox]

On 4/12/07, Anthony Sorace <anothy@gmail.com> wrote:
> I'm having trouble understanding the motivation behind /cfg, as
> compared to /sys/lib/sysconfig. The later seems much more general
> (although i'm not clear on the pxe case specifically), as well as
> having the aesthetic benefit of not needlessly cluttering root.
>
> Am I missing something?

To start with, Plan 9 can handle as many things in the
root as you like.  It is not constrained by the size
of an rk05 or the time to fsck the root partition.

At the time, /cfg was deemed shorter to type.
There is no real difference, except that there
are programs that actually use /cfg and nothing
ever used /sys/lib/sysconfig.  It's too late to
turn back now.  I didn't really like /cfg at the
time, but I've gotten used to it, and I do believe
that part of why it gets used is the much
shorter name.

Russ

Re: [9fans] Re: [sources] 20070410: % cat

[Anthony Sorace]

I'm having trouble understanding the motivation behind /cfg, as
compared to /sys/lib/sysconfig. The later seems much more general
(although i'm not clear on the pxe case specifically), as well as
having the aesthetic benefit of not needlessly cluttering root.

Am I missing something?

Re: [9fans] Re: [sources] 20070410: % cat

[geoff]

/cfg/pxe is needed for PXE booting.

We have also pushed essentially all files unique to a single machine
into /cfg/$sysname here.  I pushed out an updated cpurc(8) a couple of
weeks ago and am now bringing the start-up scripts on sources up to
date, in stages, to match it, in part to minimise the differences
between what we run and what's on sources.