Re: [9fans] gcc on plan9

[Ronald G Minnich <rminnich@lanl.gov>]

Roman Shaposhnik wrote:
> One question that I still have, though, 
> is what
> makes you think that once you're done with porting gcc (big task) and 
> porting HPC apps to
> gcc/Plan9 (even bigger one!) they will *execute* faster than they do on 
> Linux ?

Excellent question.

It's all about parallel performance; making sure your 1000 nodes run 
1000 times as fast as 1 node, or, if they don't, that it's Somebody 
Else's Problem. The reason that the OS can impact parallel performance 
boils down to the kinds of tasks that go on in OSes that can be run at 
awkward times,and in turn interfere with parallel applications, and 
result in degraded performance. (for another approach, see Cray's 
synchronised scheduler work; make all nodes schedule the app at the same 
time).

Imagine you have one of these lovely apps, on a 1000-node cluster with a 
5-microsecond latency network. Let us further imagine (this stuff 
exists; see Quadrics) that you can do a broadcast/global sum op in 5 
microseconds. After 1 millisecond, they all need to talk to each other, 
and can not proceed until they're all agreed on (say) the value of a 
computed number -- e.g. some sort of global sum of a variable held by 
each of 1000 procs. The generic term for this type of thing is 'global 
reduction' -- you reduce a vector to a scalar of some sort.

The math is pretty easy to do, but it boils down to this: OS activities 
can interfere with, say, just one task, and kill the parallel 
performance of the app, making your 1000-node app run like a 750 node 
app -- or worse.

Pretend you're delayed one microsecond; do the math; it's depressing. 
One millisecond compute interval is a really extreme case, chosen for 
ease of illustration, but ...

In the clustering world, what a lot of people do is run real heavy nodes 
in clusters -- they have stuff like cron running, if you can believe it! 
They pretty much do a full desktop install, then turn off a few daemons, 
and away they go. Some really famous companies actually run clusters 
this way -- you'd be surprised at who. So do some famous gov't labs.

If they're lucky, interference never hits them. If they're not, they get 
less-than-ideal app performance. Then, they draw a conjecture from the 
OS interference that comes with such bad configuration: you can't run a 
cluster node with anything but a custom OS which has no clock 
interrupts, and, for that matter, no ability to run more than one 
process at a time. See the computational node kernel on the BG/L for one 
example, or the catamount kernel on Red Storm. Those kernels are really 
constrained; just running one proc at a time is only part of the story.

Here at LANL, we run pretty light cluster nodes.

Here is a cluster node running xcpu (under busybox, as you can see):
     1 ?        S      0:00 /bin/ash /linuxrc
     2 ?        S      0:00 [migration/0]
     3 ?        SN     0:00 [ksoftirqd/0]
     4 ?        S      0:00 [watchdog/0]
     5 ?        S      0:00 [migration/1]
     6 ?        SN     0:00 [ksoftirqd/1]
     7 ?        S      0:00 [watchdog/1]
     8 ?        S      0:00 [migration/2]
     9 ?        SN     0:00 [ksoftirqd/2]
    10 ?        S      0:00 [watchdog/2]
    11 ?        S      0:00 [migration/3]
    12 ?        SN     0:00 [ksoftirqd/3]
    13 ?        S      0:00 [watchdog/3]
    14 ?        S<     0:00 [events/0]
    15 ?        S<     0:00 [events/1]
    16 ?        S<     0:00 [events/2]
    17 ?        S<     0:00 [events/3]
    18 ?        S<     0:00 [khelper]
    19 ?        S<     0:00 [kthread]
    26 ?        S<     0:00 [kblockd/0]
    27 ?        S<     0:00 [kblockd/1]
    28 ?        S<     0:00 [kblockd/2]
    29 ?        S<     0:00 [kblockd/3]
   105 ?        S      0:00 [pdflush]
   106 ?        S      0:00 [pdflush]
   107 ?        S      0:00 [kswapd1]
   109 ?        S<     0:00 [aio/0]
   108 ?        S      0:00 [kswapd0]
   110 ?        S<     0:00 [aio/1]
   111 ?        S<     0:00 [aio/2]
   112 ?        S<     0:00 [aio/3]
   697 ?        S<     0:00 [kseriod]
   855 ?        S      0:00 xsrv -D 0 tcp!*!20001
   857 ?        S      0:00 9pserve -u tcp!*!20001
   864 ?        S      0:00 u9fs -a none -u root -m 65560 -p 564
   865 ?        S      0:00 /bin/ash

see how little we have running? Oh, but wait, what's all that stuff in 
[]? It's the stuff we can't turn off. Note there is per-cpu stuff, and 
other junk. Note that this node has been up for five hours, and this 
stuff is pretty quiet(0 run time); our nodes are the quietest (in the OS 
interference sense) Linux nodes I have yet seen. But, that said, all 
this can hit you.

And, in Linux, there's a lot of stuff people are finding you can't turn 
off. Lots of timers down there, lots of magic that goes on, and you just 
can't turn it off, or adjust it, try as you might.

Plan 9, our conjecture goes, is a small, tight, kernel, with lots of 
stuff moved to user mode (file systems); and, we believe that the Plan 9 
architecture is a good match to future HPC (High Performance Computing) 
systems, as typified by Red Storm and BG/L: small, fixed-configuration 
nodes with memory, network, CPU, and nothing else. The ability to not 
even have a file system on the node is a big plus. The ability to 
transparently have the file system remote/local puts the application 
into the driver's seat as to how the node is configured, and what 
tradeoffs are made; the system as a whole is incredibly flexible.

Our measurements, so far, do show that Plan 9 is "quieter" than Linux. A 
full Plan 9 desktop has less OS noise than a Linux box at the login 
prompt. This matters.

But it only matters if people can run their apps. Hence our concern 
about getting gcc-based cra-- er, applications code, running.

I'm not really trying to make Plan 9 look like Linux. I just want to run 
MPQC for a friend of mine :-)

thanks

ron