[9fans] some #s

[9fans] some #s

[ron minnich]

The geode cluster Andrey mentioned has a clone that etherboots linux and
comes up running bproc etc. (see clustermatic.org for info on this). The
two clusters are identical in every way.

Time from power-on to full cluster ready to roll (i.e ready for bproc
equivalent of 'cpu'): 10 seconds

Some timing for same hardware with 9load/plan 9:

0->10 seconds: 9load is up and has loaded the plan 9 kernel over ether
10-20 seconds: no real output from plan 9
20-30 seconds (or so): until I get the 'boot from [il]' prompt

Obviously we'll try to speed this up just a bit, but still, it's not too
terrible. I need to get it to skip the prompt step but that doesn't seem
like a big deal (there is no console, really, on these machines, so a
prompt is rather unimportant, esp. given there is only one choice)

Overall there's lots more procs needed to make the plan 9 nodes work as
cluster nodes than the Linux bproc stuff -- about a factor of 5. bproc
does remote exec somewhat faster, I'm working on the #s there too. But
plan 9 does do some thing better than bproc. So it's a bit of a wash in
most ways. Nevertheless I'm going to see what else I can eliminate.

Anyway, this is just FYI, I'm hoping to get good ideas from people at
usenix on reducing some overheads on this system.

ron

Re: [9fans] some #s

[andrey mirtchovski]

On Tue, 3 Jun 2003, ron minnich wrote:

> 0->10 seconds: 9load is up and has loaded the plan 9 kernel over ether
> 10-20 seconds: no real output from plan 9
> 20-30 seconds (or so): until I get the 'boot from [il]' prompt
>

as far as i remember (it was a year ago, but should still hold) the problem
with booting speed came from the fact that you have no plan9.ini to feed
9load with. when it doesn't know what to boot it probes everything (and
sometimes probes it anyways -- just to be able to prompt you with choices)..

i just tried this -- a cpuserver under vmware boots to prompt in 17 seconds
with me hitting enter at the prompt and 27 seconds when the prompt timeouts.
that's with a local harddrive...

only my theory, though. i'd be happy to hear other opinions :)

andrey

Re: [9fans] some #s

[ron minnich]

On Tue, 3 Jun 2003, andrey mirtchovski wrote:

> > 0->10 seconds: 9load is up and has loaded the plan 9 kernel over ether
> > 10-20 seconds: no real output from plan 9
> > 20-30 seconds (or so): until I get the 'boot from [il]' prompt
> >
>
> as far as i remember (it was a year ago, but should still hold) the problem
> with booting speed came from the fact that you have no plan9.ini to feed
> 9load with. when it doesn't know what to boot it probes everything (and
> sometimes probes it anyways -- just to be able to prompt you with choices)..

I already commented out that table (I forget the name) in 9load such that
it only probes ether.

see you at usenix, we'll make it faster there :-)

ron

Re: [9fans] some #s

[C H Forsyth]

i suppose it's nice when booting is fast during development or on a
mobile device, and it's always good to avoid wasting time (eg, by
probing nonexistent drives) but in a production grid environment you
won't be rebooting that often will you?

just curious, really.

Re: [9fans] some #s

[ron minnich]

On Wed, 4 Jun 2003, C H Forsyth wrote:

> i suppose it's nice when booting is fast during development or on a
> mobile device, and it's always good to avoid wasting time (eg, by
> probing nonexistent drives) but in a production grid environment you
> won't be rebooting that often will you?

depends on the "grid" (grid being a very new term for an idea that is at
least 25 years old -- see RSEXEC on the early Arpanet on TOPS-10).

Yes, in many cases, CPU servers should reboot after each job, in many
environments -- not just US Gov't either, industry people can be quite
paranoid. So it matters.

In other cases, e.g. SETI@home, I guess rebooting your windows box every
time it stops would be bad :-)

ron

Re: [9fans] some #s

[andrey mirtchovski]

On Wed, 4 Jun 2003, ron minnich wrote:

> Yes, in many cases, CPU servers should reboot after each job, in many
> environments
>

in plan9 you could say that it suffices to get rid of the namespace of the
job that had just completed :)

there's another point that should be made (not that there's much interest in
this discussion, but anyway): 9load does a very good job at what it's
supposed to do. trying to accomplish a similar configuration under different
operating systems is a much bigger PITA than it is with 9load+plan9.ini...

A testament to that must be the amount of change to the source code that
was required to fit 9load into the booting model that Ron demanded --
it's minimal at best :)

andrey

Re: [9fans] some #s

[northern snowfall]

>
>
>Yes, in many cases, CPU servers should reboot after each job, in many
>environments -- not just US Gov't either, industry people can be quite
>paranoid. So it matters.
>
Ron, can you please elaborate on this? I'm not sure why a CPU should
reboot and foggy on when you deem a job 'over'.

>

[9fans] Re: some #s

[Jim Choate]

On Wed, 4 Jun 2003, northern snowfall wrote:

> >Yes, in many cases, CPU servers should reboot after each job, in many
> >environments -- not just US Gov't either, industry people can be quite
> >paranoid. So it matters.
> >
> Ron, can you please elaborate on this? I'm not sure why a CPU should
> reboot and foggy on when you deem a job 'over'.

Memory leakage, if all the memory isn't written to a known state then in
some environments a serious security risk may be opened. Swap is another
well known example.


 --
    ____________________________________________________________________

      We are all interested in the future for that is where you and I
      are going to spend the rest of our lives.

                              Criswell, "Plan 9 from Outer Space"

      ravage@ssz.com                            jchoate@open-forge.org
      www.ssz.com                               www.open-forge.org
    --------------------------------------------------------------------

Re: [9fans] Re: some #s

[northern snowfall]

>
>
>Memory leakage, if all the memory isn't written to a known state then in
>some environments a serious security risk may be opened. Swap is another
>well known example.
>
Rebooting doesn't eradicate that vulnerability. It only obfuscates its
scope.

>

Re: [9fans] Re: some #s

[Dan Cross]

> >Memory leakage, if all the memory isn't written to a known state then in
> >some environments a serious security risk may be opened. Swap is another
> >well known example.
>
> Rebooting doesn't eradicate that vulnerability. It only obfuscates its
> scope.

Most problems that run in massively parallel configurations (as on
`grid' style machines) don't swap; each individual job is sized to run
in the available memory of the processor it's running on.  Swapping is
just too slow.

I have no idea what Choate means by ``memory leakage.''  Most sytems
have well defined semantics for returning memory to a ``known state''
after a process exits, and it's reasonable to assume that `jobs' come
in units of processes.  Perhaps there are some that don't, but
considering that we're talking about Plan 9 here, that doesn't seem
relevant.

	- Dan C.

Re: [9fans] Re: some #s

[Jim Choate]

On Wed, 4 Jun 2003, Dan Cross wrote:

> I have no idea what Choate means by ``memory leakage.''

<sigh>

> Most sytems have well defined semantics for returning memory to a
> ``known state'' after a process exits,

And Unix and Microsoft (among many others) are known not to do that well.
Not to mention the many times programmers just don't do 'the right thing'.

Some time reading security literature about such holes will be quite
elucidating. Try starting with "Applied Cryptography".


 --
    ____________________________________________________________________

      We are all interested in the future for that is where you and I
      are going to spend the rest of our lives.

                              Criswell, "Plan 9 from Outer Space"

      ravage@ssz.com                            jchoate@open-forge.org
      www.ssz.com                               www.open-forge.org
    --------------------------------------------------------------------

Re: [9fans] Re: some #s

[Dan Cross]

Jim Choate <ravage@einstein.ssz.com> writes:
> > Most sytems have well defined semantics for returning memory to a
> > ``known state'' after a process exits,
>
> And Unix and Microsoft (among many others) are known not to do that well.

Which, as I stated, is irrelevant since we're talking about Plan 9.

> Not to mention the many times programmers just don't do 'the right thing'.

Which programmers are those?  The OS programmers, or the application
programmers?  My whole point was that what the application programmers
did was irrelevant.

> Some time reading security literature about such holes will be quite
> elucidating. Try starting with "Applied Cryptography".

You know, I don't know why you think you're such an expert on everything.

Some time spent reading a book on basic English grammar and spelling
might be quite elucidating.  Try starting with ``Dick and Jane.''

	- Dan C.

Re: [9fans] Re: some #s

[Jim Choate]

On Thu, 5 Jun 2003, Dan Cross wrote:

> Which, as I stated, is irrelevant since we're talking about Plan 9.

Which has yet to go through a professional level vetting. About the best
that can be said for it was what was published in "Maximum Security".

> Which programmers are those?  The OS programmers, or the application
> programmers?  My whole point was that what the application programmers
> did was irrelevant.

Really? Talk to some crypto programmers some time. I think they'd disagree
with you a great deal on that one. The reality is that memory leakage via
apps -is- a major concern for real world security issues. OS'es don't have
the best record in that regard either.

I'd suggest arXiv or SiteSeer as a start. You can also take a look at the
NSA version of Linux.


 --
    ____________________________________________________________________

      We are all interested in the future for that is where you and I
      are going to spend the rest of our lives.

                              Criswell, "Plan 9 from Outer Space"

      ravage@ssz.com                            jchoate@open-forge.org
      www.ssz.com                               www.open-forge.org
    --------------------------------------------------------------------

Re: [9fans] Re: some #s

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 210 bytes --]

We don't zero pages till we're going to give them to another process.
If you want to feel safe after doing something really secret, I'ld recommend
power cycling at the least.  Choate is completely right here.

[-- Attachment #2: Type: message/rfc822, Size: 2702 bytes --]

From: Jim Choate <ravage@einstein.ssz.com>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] Re: some #s
Date: Thu, 5 Jun 2003 18:06:04 -0500 (CDT)
Message-ID: <Pine.LNX.4.33.0306051802450.9676-100000@einstein.ssz.com>


On Thu, 5 Jun 2003, Dan Cross wrote:

> Which, as I stated, is irrelevant since we're talking about Plan 9.

Which has yet to go through a professional level vetting. About the best
that can be said for it was what was published in "Maximum Security".

> Which programmers are those?  The OS programmers, or the application
> programmers?  My whole point was that what the application programmers
> did was irrelevant.

Really? Talk to some crypto programmers some time. I think they'd disagree
with you a great deal on that one. The reality is that memory leakage via
apps -is- a major concern for real world security issues. OS'es don't have
the best record in that regard either.

I'd suggest arXiv or SiteSeer as a start. You can also take a look at the
NSA version of Linux.


 --
    ____________________________________________________________________

      We are all interested in the future for that is where you and I
      are going to spend the rest of our lives.

                              Criswell, "Plan 9 from Outer Space"

      ravage@ssz.com                            jchoate@open-forge.org
      www.ssz.com                               www.open-forge.org
    --------------------------------------------------------------------

Re: [9fans] Re: some #s

[Dan Cross]

> We don't zero pages till we're going to give them to another process.
> If you want to feel safe after doing something really secret, I'ld recommend
> power cycling at the least.

So, my factotum isn't safe if it exits?  Hmm....

> Choate is completely right here.

Perhaps by accident.

	- Dan C.

Re: [9fans] Re: some #s

[David Presotto]

The memory is cleaned before another process gets it but that could be
too late in the presence of bugs.  It wouldn't be too hard to zero them
immediately, I probably should.  Someone will notice that things get slower...

Also, when you reboot your machine, with ctl-alt-del, factotum's pages
are still sitting in memory somewhere.  Someone can load their own kernel
and look at the data.  I should also change ctl-alt-del to zero out process
memory before bringing the system down.  Then there are still crashes...

Even with all that, I'ld be happier if the BIOS or boot ROM zeroed
all memory.

Re: [9fans] Re: some #s

[northern snowfall]

>
>
>Also, when you reboot your machine, with ctl-alt-del, factotum's pages
>are still sitting in memory somewhere.  Someone can load their own kernel
>and look at the data.  I should also change ctl-alt-del to zero out process
>memory before bringing the system down.  Then there are still crashes...
>
This was my original point. Rebooting does nothing but obfuscate the issue
to a slight extent. Memory leaks are a huge problem, but, can't be evaded
by measures that dont cleanse memory in some fashion. As I'm sure we all
know here, memory zero'd is pointless in situations where the physical hard-
ware can be accessed (even remotely through the kernel or driver bugs, etc).

With some slick I/O techniques, nulled memory can still be read for
resonating
patterns. The only real solution in secure clusters (or other situations) is
to force the supervisor code to perform a NSA trusted random-pattern
cleanse,
or, something more paranoid. But, as is stated above, that doesn't eradicate
the problem of crashes.

I have some solutions I'm looking at in Autumn, but, the papers wont be out
for a while. Though, it probably isn't anything the NIPR/NSA spooks haven't
already done ;)

Don

http://deadchildren.org/~north_

>

Re: [9fans] Re: some #s

[ron minnich]

And, to add to the stew, you guys are all focusing on the 'is memory
clean' issue, and there are more general issues involved here.

So, back to the original issue, which is that sometimes you really do want
to reboot after every app :-)

ron

[9fans] Re: some #s

[Jim Choate]

On Thu, 5 Jun 2003, ron minnich wrote:

> And, to add to the stew, you guys are all focusing on the 'is memory
> clean' issue, and there are more general issues involved here.
>
> So, back to the original issue, which is that sometimes you really do want
> to reboot after every app :-)

Specious distinction.


 --
    ____________________________________________________________________

      We are all interested in the future for that is where you and I
      are going to spend the rest of our lives.

                              Criswell, "Plan 9 from Outer Space"

      ravage@ssz.com                            jchoate@open-forge.org
      www.ssz.com                               www.open-forge.org
    --------------------------------------------------------------------

Re: [9fans] Re: some #s

[northern snowfall]

>
>
>> Also, when you reboot your machine, with ctl-alt-del, factotum's pages
>> are still sitting in memory somewhere.  Someone can load their own
>> kernel
>> and look at the data.  I should also change ctl-alt-del to zero out
>> process
>> memory before bringing the system down.  Then there are still crashes...
>
Oh, I forgot to mention. Another thing is stale cluster connection
hijacking. Just as in NFS, this tends to be a problem in cluster
situations. Rebooting does a body good for this kind of issue, but,
it sure as hell isn't the best answer. The protocols should be more
wary, IM(not-so-humble)O.

Don

http://deadchildren.org/~north_

[9fans] Re: some #s

[Jim Choate]

On Thu, 5 Jun 2003, David Presotto wrote:

> Even with all that, I'ld be happier if the BIOS or boot ROM zeroed
> all memory.

Contact your BIOS vendor and ask about a security release that replaces the
simple memory check with a memory walker (ala Barber Pole) or burn your own
BIOS.


 --
    ____________________________________________________________________

      We are all interested in the future for that is where you and I
      are going to spend the rest of our lives.

                              Criswell, "Plan 9 from Outer Space"

      ravage@ssz.com                            jchoate@open-forge.org
      www.ssz.com                               www.open-forge.org
    --------------------------------------------------------------------

Re: [9fans] Re: some #s

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 285 bytes --]

I've already done that.  My meaning was only that I wished it were actually
a universal feature.  Most people, faced with the possibility of turning on
a feature might do it.  Make them burn a BIOS on a system and its unlikely
they'll ever bother; too scarey and too much of a pain.

[-- Attachment #2: Type: message/rfc822, Size: 2200 bytes --]

From: Jim Choate <ravage@einstein.ssz.com>
To: <9fans@cse.psu.edu>
Subject: [9fans] Re: some #s
Date: Fri, 6 Jun 2003 07:01:50 -0500 (CDT)
Message-ID: <Pine.LNX.4.33.0306060658360.9676-100000@einstein.ssz.com>


On Thu, 5 Jun 2003, David Presotto wrote:

> Even with all that, I'ld be happier if the BIOS or boot ROM zeroed
> all memory.

Contact your BIOS vendor and ask about a security release that replaces the
simple memory check with a memory walker (ala Barber Pole) or burn your own
BIOS.


 --
    ____________________________________________________________________

      We are all interested in the future for that is where you and I
      are going to spend the rest of our lives.

                              Criswell, "Plan 9 from Outer Space"

      ravage@ssz.com                            jchoate@open-forge.org
      www.ssz.com                               www.open-forge.org
    --------------------------------------------------------------------

Re: [9fans] Re: some #s

[boyd, rounin]

> So, my factotum isn't safe if it exits?  Hmm....

well when i wrote a pop client daemon on lunix i hid the
password in a pipe, that way it wouldn't turn up in a core
dump [ok, there was a small window].

but, yeah, for a buncha high security applications you don't
want the data on disk and you wanna blow its memory
when they thing exits.

security is always a trade off between what you want to protect
and how much you wanna pay.

obviously, a bunch of you know that ...

Re: [9fans] Re: some #s

[Russ Cox]

>> Choate is completely right here.
>
> Perhaps by accident.

You know, he's been reasonable recently.
You're the one egging him on.

Russ

Re: [9fans] Re: some #s

[Dan Cross]

"Russ Cox" <rsc@plan9.bell-labs.com> writes:
> >> Choate is completely right here.
> >
> > Perhaps by accident.
>
> You know, he's been reasonable recently.

Given his track record, I'm confident he'll shatter that record soon.

> You're the one egging him on.

Perhaps.  Maybe I just don't like being told to go read books I read
five years ago.

	- Dan C.

Re: [9fans] Re: some #s

[northern snowfall]

>
>
>Maybe I just don't like being told to go read books I read
>five years ago.
>
If you already read them, then, let your knowledge and actions
prove it :). Unnecessary aggression proves nothing and negates
everything...

"What do you care what other people think" - you know who

Don

>

Re: [9fans] Re: some #s

[Dan Cross]

northern snowfall <dbailey27@ameritech.net> writes:
> If you already read them, then, let your knowledge and actions
> prove it :). Unnecessary aggression proves nothing and negates
> everything...
>
> "What do you care what other people think" - you know who

All right, all right, all right; I get the picture.  I'll stop antagonizing
Choate.  What can I say?  It's been a rough few days.

	- Dan C.

Re: [9fans] Re: some #s

[northern snowfall]

>
>
>All right, all right, all right; I get the picture.  I'll stop antagonizing
>Choate.  What can I say?  It's been a rough few days.
>
Well I'll raise a glass to that, Dan. Cheers to a better tomorrow, and
many more to come.

Don

http://deadchildren.org/~north_

>

Re: [9fans] Re: some #s

[Dan Cross]

> Well I'll raise a glass to that, Dan. Cheers to a better tomorrow, and
> many more to come.

Thanks Don, that's very kind of you.

> http://deadchildren.org/~north_

However, I have to say, that's a rather morbid domainname....  :-)

	- Dan C.

Re: [9fans] Re: some #s

[northern snowfall]

>
>
>>http://deadchildren.org/~north_
>>
>However, I have to say, that's a rather morbid domainname....  :-)
>
Quite so, but with reason. It doesn't represent a somatic death , but
rather the death of spirit. The Dead Children's Society is a charity
organization that benefits people in my 'extended family' that have
made choices which alienate them from their concept of family.

Dead Children is a way of helping those that refuse to help themselves,
but in a very covert way. It doesn't give hand-outs, but, creates
situations for the target individual in which they are still given
a choice to help themselves, or to not. I believe the only true charity
is the charity that allows an individual to choose a more positive
road. I do not believe in feeding a man for a day.

The Blessed Children's Society, does not give hand-outs either.
Rather, it gives opportunity of a different kind. Funds from the extended
family is pooled in such a way as none of the donators may make a
withdrawal.
Individuals given donations are the youth of the extended family that
prove their interest in the family to be true with hard work and play,
either in their respective schooling or through independent study. Though,
recipients still do not *know* about the donations. They just happen.

All 'board' members of each society are silent members.

We are somewhat modeled after the John D. and Catherine T. MacArthur
Foundation[1] and, to a lesser extent, the Charles Stewart Mott
Foundation[2]; both of which, we believe, had a very solid hand in
shaping (and possibly saving) both our youth and our future.

Hopefully, with hard work and perseverance, we will make opportunities
for the youth to come that shapes their future for the better, helping
to perpetuate the cycle of strong family values and a universally equality
conscious society.

Don

[1] John D. and Catherine T. MacArthur Foundation http://www.macfound.org
[2] Charles Stewart Mott Foundation http://www.mott.org

http://deadchildren.org/~north_


>

Re: [9fans] Re: some #s

[boyd, rounin]

> "What do you care what other people think" - you know who

Feynman

Re: [9fans] Re: some #s

[Jack Johnson]

On Thu, 5 Jun 2003, David Presotto wrote:
> We don't zero pages till we're going to give them to another process.
> If you want to feel safe after doing something really secret, I'ld recommend
> power cycling at the least.  Choate is completely right here.

As long as we're playing the game....

Assuming you were sharing a host (knowingly or otherwise) with someone who
could accomplish the task, would there be a semi-convenient way to detect
someone combing through your dead pages?  Like an IDS for memory?

-Jack

Re: [9fans] some #s

[ron minnich]

On Wed, 4 Jun 2003, northern snowfall wrote:

> Ron, can you please elaborate on this? I'm not sure why a CPU should
> reboot and foggy on when you deem a job 'over'.

need to reboot: all jobs leave state on a machine in one form or another.

'over': user-defined

Which means: after a job is over, state can be left behind.

Depending on the context this can be "bad".

I had a long writeup here on things that can happen and what you need to
do but I'm going to leave it at 'sometimes you need to reboot after a job
is over' :-)

ron

Re: [9fans] some #s

[jmk]

On Wed Jun  4 00:22:20 EDT 2003, rminnich@lanl.gov wrote:
> ...
> Some timing for same hardware with 9load/plan 9:
>
> 0->10 seconds: 9load is up and has loaded the plan 9 kernel over ether
> 10-20 seconds: no real output from plan 9
> 20-30 seconds (or so): until I get the 'boot from [il]' prompt
> ...

i just tried some boots without a plan9.ini (9load tweaked to probe
suitable ether devices, but no other changes). the machine is a standard
asus k7m motherboard with a floppy and 2 ether cards, no other
peripherals:

 it takes 20s from power on to get 9load off the floppy, probe for peripherals
 and get to the 'boot from: ' prompt;
 it takes 3s after typing 'ether0!/386/9pccpu' to the above prompt to get the
 kernel over the ether and get the 'root is from (il, tcp)[il]: ' prompt.

perhaps the kernel is poking some other peripherals on your system (e.g. things
that look like discs) to create the delay you see after the kernel is loaded?

Re: [9fans] some #s

[ron minnich]

On Wed, 4 Jun 2003 jmk@plan9.bell-labs.com wrote:

> perhaps the kernel is poking some other peripherals on your system (e.g. things
> that look like discs) to create the delay you see after the kernel is loaded?

next on my list, that's a good thing to check.

ron