Re: [9fans] Re: [sources] 20070410: % cat >/sys/lib/dist/changes/1176262206.1.txt << EOF

["ron minnich" <rminnich@gmail.com>]

On 4/12/07, Uriel <uriel99@gmail.com> wrote:

> Having /cfg by default might be a good idea, but I don't think it is a
> big deal as long as it is optional, I suspect most small networks can
> get by with cpurc.local and maybe a cpustart.local, and if you want
> /cfg, you can just create it and cpurc should take advantage of it
> automatically.

what's interesting about this file per node config stuff:
1. small sites don't need it
2. for big sites, it's just not practical at all. File per node on
1024 or more nodes? it just won't go. And 1024 is not a lot.
3. For medium sites, it's kind of ok, I guess.

So we've got at least three cases here. From my point of view, that
argues that we've done something wrong.

File per node for config info is reminiscent of what Sun used to do.
You had a directory, in which you put files like this:
0040506789AB
i.e. the MAC address, and that was a symlink to the correct bootstrap.
It was fine for networks of a 100-200 hundred nodes, at least in my
experience. It's totally unworkable at any reasonable scale.

For systems I work with, file per node is not going to work. A
configuration entry per node is marginally workable, to a point; but I
recently saw a system that had about 1Kbyte of XML per node
configuration, in one file; a 1 Mbyte file for a 1024 cluster, which
is not large nowadays; it's headache-inducing.

The onesis stuff (onesis.org) has introduced the concept of roles to
cluster computing, and it was used to bring up a 4096-node
IB-connected cluster. It is easy to set up (I've done it).

This note is written in the hope that we can do better. I don't
pretend I know how ... but per-node files doesn't really feel right.

thanks

ron