Re: [9fans] Re: [sources] 20070410: % cat

Re: [9fans] Re: [sources] 20070410: % cat

[Lucio De Re]

> Then you need two different sets of policy files and to have your
> initialisation bind the right ones into place.  I'm doing this now for
> a set of diverse machines with different keys, policies, etc. all
> sharing a single file server.

That does not contradict my statement that it does not scale.  Much as
I appreciate the philosophical value of bind/mount, a trillion
instances of a configuration file are going to be unmanageable.  Ron
is right that there is no slick solution, but it's worth knowing that
the current approach deserves exploring further.

++L

Re: [9fans] Re: [sources] 20070410: % cat

[Devon H. O'Dell]

2007/4/13, Lucio De Re <lucio@proxima.alt.za>:
> > Then you need two different sets of policy files and to have your
> > initialisation bind the right ones into place.  I'm doing this now for
> > a set of diverse machines with different keys, policies, etc. all
> > sharing a single file server.
>
> That does not contradict my statement that it does not scale.  Much as
> I appreciate the philosophical value of bind/mount, a trillion
> instances of a configuration file are going to be unmanageable.  Ron
> is right that there is no slick solution, but it's worth knowing that
> the current approach deserves exploring further.
>
> ++L

Do you know anybody who has ever come up with a better idea? This is
very similar to Maildir replacing mbox. It sounds good, but when you
have 250k messages, it ends up being slower simply to traverse the
directory. In FreeBSD anyway.

However, it does scale. Much, much better than a case where you have
the same requirement for thousands of systems in a switch () statment.

--dho

Re: [9fans] Re: [sources] 20070410: % cat

[erik quanstrom]

i think mdir was done to solve nfs locking problems, not
for performance.

for 250k messages, either solution is poor.  deleting the
first of 250k messages requires rewriting the whole mbox.
dealing with 250k directory entries can be painful, too,
as many fs keep directores as arrays.

if you have that many messages, you might want an index. ;-)

- erik

Re: [9fans] Re: [sources] 20070410: % cat

[Lyndon Nerenberg]

> for 250k messages, either solution is poor.  deleting the
> first of 250k messages requires rewriting the whole mbox.
> dealing with 250k directory entries can be painful, too,
> as many fs keep directores as arrays.
>
> if you have that many messages, you might want an index. ;-)

I've said it before, and I'll keep saying it until I'm proved wrong:

   nothing runs faster or scales better than the Cyrus IMAP mail store.

MH filesystem layout with per-folder index + header cache.  I have 
deployed mail servers with literally millions of user accounts using this 
layout, and it just works.

These days I don't find large directories to be a problem.  A few months 
ago I did some performance tests on large directory operations (create, 
unlink, readdir) on Linux and FreeBSD.  For both OSes, directory 
enumeration times were negligible for the tests I ran (in the vicinity of 
200K entries).  And with everyone caching directory entries these days, 
the only real difference between directory I/O and file I/O is the 
locking overhead for directory modifications.  (This all assumes local 
disk. Throw in NFS and everything implodes.)

But scalability aside, the real evilness in mbox is the quoting of 
'^From '.

--lyndon

Re: [9fans] Re: [sources] 20070410: % cat

[Paweł Lasek]

On 4/13/07, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:
> > for 250k messages, either solution is poor.  deleting the
> > first of 250k messages requires rewriting the whole mbox.
> > dealing with 250k directory entries can be painful, too,
> > as many fs keep directores as arrays.
> >
> > if you have that many messages, you might want an index. ;-)


> These days I don't find large directories to be a problem.  A few months
> ago I did some performance tests on large directory operations (create,
> unlink, readdir) on Linux and FreeBSD.  For both OSes, directory
> enumeration times were negligible for the tests I ran (in the vicinity of
> 200K entries).  And with everyone caching directory entries these days,
> the only real difference between directory I/O and file I/O is the
> locking overhead for directory modifications.  (This all assumes local
> disk. Throw in NFS and everything implodes.)

That's also thanks to improved filesystems. Modern ones (at least in
Linux, I don't know about *BSD as I didn't use them that extentively)
use weird/complicated/WTF trees to keep directory (and not only) data.
I am not sure if HFS+ doesn't use it also for several other things, as
at least one of the developers working on it came from Be where he
worked on BeFS, which was basically a database with hierarchical
interface...

I don't know about other fs'es, but I have never had problem with
directory entries on XFS. Or anything else, as I never managed to
break it despite being designed for UPS-backed servers (It caches very
aggressively, especially writes) while ReiserFS gave up very fast :-)

BTW, wouldn't XFS be a nice addition to Plan9? Plan9-specific things
can be easily put into  extended attributes, just make sure that
inodes will have larger-than-default size so that xattr access would
be blazing fast....

> --lyndon
>


-- 
Paul Lasek

Re: [9fans] Re: [sources] 20070410: % cat

[Russ Cox]

> That's also thanks to improved filesystems. 

This isn't true.

The savings are due to aggressive caching in the
operating system, which cuts the file system out
entirely.  The disks are slow no matter how you 
use them.

File I/O for things like Python and TeX is slower
on Plan 9 because it has to go to the file server,
which is in another process or another machine,
rather than answer out of a kernel cache like Linux
does. 

Lots of things get easier if you assume you are the
only machine in the universe.  See earlier discussion
on caching read EODs.

Russ

Re: [9fans] Re: [sources] 20070410: % cat

[Lucio De Re]

> Do you know anybody who has ever come up with a better idea? This is
> very similar to Maildir replacing mbox. It sounds good, but when you
> have 250k messages, it ends up being slower simply to traverse the
> directory. In FreeBSD anyway.
> 
A valid comment, I must grant.  I guess I'm hoping the better idea
will not be overlooked by a NIH when it does present itself.

> However, it does scale. Much, much better than a case where you have
> the same requirement for thousands of systems in a switch () statment.

Also valid.  But a database lookup would be even better, would it not?
That has got to be kept in mind, like or not.  Thinking out of the box
is called for, will it ever happen?

++L

Re: [9fans] Re: [sources] 20070410: % cat >/sys/lib/dist/changes/1176262206.1.txt << EOF

[Russ Cox]

/cfg is intended for machines with per-node configuration.
If you have 1000 machines that are functionally identical,
you're not supposed to write /cfg directories for any of them.

But if you have one machine that is supposed to do something
a little different (like run a venti server, or be an auth server),
then you can put those differences into /cfg/$sysname/cpurc.

What /cfg replaced was a giant

	switch($sysname){
	case this
		...
	case that
		...
	}

in /rc/bin/cpurc.  It's definitely a step forward for that
situation.

The Plan 9 distribution has always been "this works
well at Bell Labs; you might need to change it to suit you",
and that applies to /cfg as much as anything else.

/cfg is just a pragmatic solution for one use model.
If you are running large clusters, you will want something
different.  But most Plan 9 installations are not large clusters.

I for one am glad that people can edit cpurc instead of
having to agree on a general startup script mechanism.
Otherwise we might end up with something like /etc/init.d.

Russ

Re: [9fans] Re: [sources] 20070410: % cat

[Lucio De Re]

> /cfg is intended for machines with per-node configuration.
> If you have 1000 machines that are functionally identical,
> you're not supposed to write /cfg directories for any of them.

Still doesn't scale: what if you have *two* sets of 1024 machines with
slightly different configurations in the two groups?

++L

Re: [9fans] Re: [sources] 20070410: % cat

[geoff]

Then you need two different sets of policy files and to have your
initialisation bind the right ones into place.  I'm doing this now for
a set of diverse machines with different keys, policies, etc. all
sharing a single file server.

Re: [9fans] Re: [sources] 20070410: % cat >/sys/lib/dist/changes/1176262206.1.txt << EOF

[ron minnich]

On 4/12/07, Uriel <uriel99@gmail.com> wrote:

> Having /cfg by default might be a good idea, but I don't think it is a
> big deal as long as it is optional, I suspect most small networks can
> get by with cpurc.local and maybe a cpustart.local, and if you want
> /cfg, you can just create it and cpurc should take advantage of it
> automatically.

what's interesting about this file per node config stuff:
1. small sites don't need it
2. for big sites, it's just not practical at all. File per node on
1024 or more nodes? it just won't go. And 1024 is not a lot.
3. For medium sites, it's kind of ok, I guess.

So we've got at least three cases here. From my point of view, that
argues that we've done something wrong.

File per node for config info is reminiscent of what Sun used to do.
You had a directory, in which you put files like this:
0040506789AB
i.e. the MAC address, and that was a symlink to the correct bootstrap.
It was fine for networks of a 100-200 hundred nodes, at least in my
experience. It's totally unworkable at any reasonable scale.

For systems I work with, file per node is not going to work. A
configuration entry per node is marginally workable, to a point; but I
recently saw a system that had about 1Kbyte of XML per node
configuration, in one file; a 1 Mbyte file for a 1024 cluster, which
is not large nowadays; it's headache-inducing.

The onesis stuff (onesis.org) has introduced the concept of roles to
cluster computing, and it was used to bring up a 4096-node
IB-connected cluster. It is easy to set up (I've done it).

This note is written in the hope that we can do better. I don't
pretend I know how ... but per-node files doesn't really feel right.

thanks

ron

Re: [9fans] Re: [sources] 20070410: % cat

[geoff]

You only need files in /cfg for a machine if that machine is an
exception to your normal policies (in cpurc, namespace, consoledb,
etc.).  I would expect that a cluster would need zero or one such
exceptions, as the machines are supposed to be interchangeable, right?

Re: [9fans] Re: [sources] 20070410: % cat

[Devon H. O'Dell]

2007/4/12, geoff@plan9.bell-labs.com <geoff@plan9.bell-labs.com>:
> You only need files in /cfg for a machine if that machine is an
> exception to your normal policies (in cpurc, namespace, consoledb,
> etc.).  I would expect that a cluster would need zero or one such
> exceptions, as the machines are supposed to be interchangeable, right?

In my mind, this is not entirely the case. The other reason to use it
is to avoid actually touching cpurc, so that you don't have to
remember to type -s '/rc/bin/cpurc' when I pull, after backing it up
and merging any changes back in. Since pull is the primary means of
keeping the system up to date, it's a real pain to try to merge all
these changes.

--dho

Re: [9fans] Re: [sources] 20070410: % cat >/sys/lib/dist/changes/1176262206.1.txt << EOF

[Devon H. O'Dell]

2007/4/12, Uriel <uriel99@gmail.com>:
> On 4/12/07, Devon H. O'Dell <devon.odell@gmail.com> wrote:

[snip]

> > One thing I'm not sure of still is, if there is a cpurc.local, it may
> > want to implement some things dependent on e.g. what if you are
> > starting some client script that requires the network to already have
> > been initialized. This seems to be what you would end up putting in
> > your /cfg/$sysname/cpustart,
>
> We might want a /rc/bin/cpustart.local too?

No, it appears to serve the same purpose as cpurc.local --
cpu-specific startup, after initialization. (And it's not in the
default distribution). At what point do we stop and at what point do
we create cpustart.local.local.local?

> > We need to have something automatically create /cfg for peoples
> > fossils when they upgrade then. You can't just mkdir /cfg.
>
> Having /cfg by default might be a good idea, but I don't think it is a
> big deal as long as it is optional, I suspect most small networks can
> get by with cpurc.local and maybe a cpustart.local, and if you want
> /cfg, you can just create it and cpurc should take advantage of it
> automatically.

If we don't put /cfg in, I agree that cpustart.local is a good idea.

> > I could. How is your script splitting them up though? I was assuming
> > that it used those lines to split each change.
>
> For some unknown reason russ refused to have the changes split on his
> end, which would be the ideal. I have a script that splits the daily

Because it's a pain in the ass to go through all the changes individually.

> changes file and puts them in sources, see for example:
> /n/sources/contrib/uriel/changes/2007/0410/
> and then use that as input for the email notifications.
>
> If someone wants to get a single email per day, they can get a digest
> of plan9changes.
>
> To split I use some awk magic to detect text that is not indented,
> which is the comments that go over the diffs. See the end of
> /n/sources/contrib/uriel/plog for the (awful and fragile) details.

Ok. I'll start adding subjects then. But do please note, I'm not
intending to split them up past daily changelogs right now. Yes, the
makemail script does create separate changes, but editing them all
separately, figuring out what to name them, etc, would increase the
effort needed to do these things by a power of at least n*2.

--dho

Re: [9fans] Re: [sources] 20070410: % cat

[geoff]

/cfg/pxe is needed for PXE booting.

We have also pushed essentially all files unique to a single machine
into /cfg/$sysname here.  I pushed out an updated cpurc(8) a couple of
weeks ago and am now bringing the start-up scripts on sources up to
date, in stages, to match it, in part to minimise the differences
between what we run and what's on sources.

Re: [9fans] Re: [sources] 20070410: % cat

[Anthony Sorace]

I'm having trouble understanding the motivation behind /cfg, as
compared to /sys/lib/sysconfig. The later seems much more general
(although i'm not clear on the pxe case specifically), as well as
having the aesthetic benefit of not needlessly cluttering root.

Am I missing something?

Re: [9fans] Re: [sources] 20070410: % cat

[Russ Cox]

On 4/12/07, Anthony Sorace <anothy@gmail.com> wrote:
> I'm having trouble understanding the motivation behind /cfg, as
> compared to /sys/lib/sysconfig. The later seems much more general
> (although i'm not clear on the pxe case specifically), as well as
> having the aesthetic benefit of not needlessly cluttering root.
>
> Am I missing something?

To start with, Plan 9 can handle as many things in the
root as you like.  It is not constrained by the size
of an rk05 or the time to fsck the root partition.

At the time, /cfg was deemed shorter to type.
There is no real difference, except that there
are programs that actually use /cfg and nothing
ever used /sys/lib/sysconfig.  It's too late to
turn back now.  I didn't really like /cfg at the
time, but I've gotten used to it, and I do believe
that part of why it gets used is the much
shorter name.

Russ

Re: [9fans] Re: [sources] 20070410: % cat

[Anthony Sorace]

//To start with, Plan 9 can handle as many things in the
//root as you like.  It is not constrained by the size
//of an rk05 or the time to fsck the root partition.

Of course; as I said, it's largely an aesthetic decision. But I do
believe it has an impact on *people* reading /, rather than programs
reading it. And /sys/lib feels like a very natural fit for exactly
this sort of information.

Yes, /cfg is provably shorter to type than /sys/lib/sysconfig... but
you really believe that matters here? I don't dispute the impact of
length on use generally, but this is seldom-changing config
information. Is it really true that nothing/nobody ever used sysconfig
(contrary to the cpurc man page)?

Personally, i think the argument of "so we don't have to edit
cpurc/termrc" is vacuous. They change infrequently enough, and in
small enough ways each time, that merging changes is simple. I think
the fear (as Russ pointed to elsewhere) of moving towards init.d is
much more legitimate.

Re: [9fans] Re: [sources] 20070410: % cat

[Lucio De Re]

> We have also pushed essentially all files unique to a single machine
> into /cfg/$sysname here.  I pushed out an updated cpurc(8) a couple of
> weeks ago and am now bringing the start-up scripts on sources up to
> date, in stages, to match it, in part to minimise the differences
> between what we run and what's on sources.

I understood this was the role of /sys/lib/sysconfig, but I can easily
understand the desire to change.  Jim suggested, however, that PXE
handling wasn't cast in stone and that might have affected the
stability of /cfg/pxe.  I suppose this means we need a bull from Bell
Labs to pin this down for posterity?

++L

Re: [9fans] Re: [sources] 20070410: % cat

[geoff]

PXE booting is not likely to change much.  Dave Presotto did much of
that work and he's now gone.  /cfg/pxe is here to stay, as far as I
can see.