From: Roman Shaposhnick <rvs@sun.com>
To: Fans of the OS Plan 9 from Bell Labs <9fans@cse.psu.edu>
Subject: Re: [9fans] gcc on plan9
Date: Fri, 9 Jun 2006 12:56:58 -0700 [thread overview]
Message-ID: <20060609195658.GF1693@submarine> (raw)
In-Reply-To: <4487B351.8000808@lanl.gov>
Thanks for a thorough explanation! It all makes sense now.
I wish you luck with moving forward with this project.
Thanks,
Roman.
P.S. Keep us posted ;-)
On Wed, Jun 07, 2006 at 11:19:13PM -0600, Ronald G Minnich wrote:
> Roman Shaposhnik wrote:
> >One question that I still have, though,
> >is what
> >makes you think that once you're done with porting gcc (big task) and
> >porting HPC apps to
> >gcc/Plan9 (even bigger one!) they will *execute* faster than they do on
> >Linux ?
>
> Excellent question.
>
> It's all about parallel performance; making sure your 1000 nodes run
> 1000 times as fast as 1 node, or, if they don't, that it's Somebody
> Else's Problem. The reason that the OS can impact parallel performance
> boils down to the kinds of tasks that go on in OSes that can be run at
> awkward times,and in turn interfere with parallel applications, and
> result in degraded performance. (for another approach, see Cray's
> synchronised scheduler work; make all nodes schedule the app at the same
> time).
>
> Imagine you have one of these lovely apps, on a 1000-node cluster with a
> 5-microsecond latency network. Let us further imagine (this stuff
> exists; see Quadrics) that you can do a broadcast/global sum op in 5
> microseconds. After 1 millisecond, they all need to talk to each other,
> and can not proceed until they're all agreed on (say) the value of a
> computed number -- e.g. some sort of global sum of a variable held by
> each of 1000 procs. The generic term for this type of thing is 'global
> reduction' -- you reduce a vector to a scalar of some sort.
>
> The math is pretty easy to do, but it boils down to this: OS activities
> can interfere with, say, just one task, and kill the parallel
> performance of the app, making your 1000-node app run like a 750 node
> app -- or worse.
>
> Pretend you're delayed one microsecond; do the math; it's depressing.
> One millisecond compute interval is a really extreme case, chosen for
> ease of illustration, but ...
>
> In the clustering world, what a lot of people do is run real heavy nodes
> in clusters -- they have stuff like cron running, if you can believe it!
> They pretty much do a full desktop install, then turn off a few daemons,
> and away they go. Some really famous companies actually run clusters
> this way -- you'd be surprised at who. So do some famous gov't labs.
>
> If they're lucky, interference never hits them. If they're not, they get
> less-than-ideal app performance. Then, they draw a conjecture from the
> OS interference that comes with such bad configuration: you can't run a
> cluster node with anything but a custom OS which has no clock
> interrupts, and, for that matter, no ability to run more than one
> process at a time. See the computational node kernel on the BG/L for one
> example, or the catamount kernel on Red Storm. Those kernels are really
> constrained; just running one proc at a time is only part of the story.
>
> Here at LANL, we run pretty light cluster nodes.
>
> Here is a cluster node running xcpu (under busybox, as you can see):
> 1 ? S 0:00 /bin/ash /linuxrc
> 2 ? S 0:00 [migration/0]
> 3 ? SN 0:00 [ksoftirqd/0]
> 4 ? S 0:00 [watchdog/0]
> 5 ? S 0:00 [migration/1]
> 6 ? SN 0:00 [ksoftirqd/1]
> 7 ? S 0:00 [watchdog/1]
> 8 ? S 0:00 [migration/2]
> 9 ? SN 0:00 [ksoftirqd/2]
> 10 ? S 0:00 [watchdog/2]
> 11 ? S 0:00 [migration/3]
> 12 ? SN 0:00 [ksoftirqd/3]
> 13 ? S 0:00 [watchdog/3]
> 14 ? S< 0:00 [events/0]
> 15 ? S< 0:00 [events/1]
> 16 ? S< 0:00 [events/2]
> 17 ? S< 0:00 [events/3]
> 18 ? S< 0:00 [khelper]
> 19 ? S< 0:00 [kthread]
> 26 ? S< 0:00 [kblockd/0]
> 27 ? S< 0:00 [kblockd/1]
> 28 ? S< 0:00 [kblockd/2]
> 29 ? S< 0:00 [kblockd/3]
> 105 ? S 0:00 [pdflush]
> 106 ? S 0:00 [pdflush]
> 107 ? S 0:00 [kswapd1]
> 109 ? S< 0:00 [aio/0]
> 108 ? S 0:00 [kswapd0]
> 110 ? S< 0:00 [aio/1]
> 111 ? S< 0:00 [aio/2]
> 112 ? S< 0:00 [aio/3]
> 697 ? S< 0:00 [kseriod]
> 855 ? S 0:00 xsrv -D 0 tcp!*!20001
> 857 ? S 0:00 9pserve -u tcp!*!20001
> 864 ? S 0:00 u9fs -a none -u root -m 65560 -p 564
> 865 ? S 0:00 /bin/ash
>
> see how little we have running? Oh, but wait, what's all that stuff in
> []? It's the stuff we can't turn off. Note there is per-cpu stuff, and
> other junk. Note that this node has been up for five hours, and this
> stuff is pretty quiet(0 run time); our nodes are the quietest (in the OS
> interference sense) Linux nodes I have yet seen. But, that said, all
> this can hit you.
>
> And, in Linux, there's a lot of stuff people are finding you can't turn
> off. Lots of timers down there, lots of magic that goes on, and you just
> can't turn it off, or adjust it, try as you might.
>
> Plan 9, our conjecture goes, is a small, tight, kernel, with lots of
> stuff moved to user mode (file systems); and, we believe that the Plan 9
> architecture is a good match to future HPC (High Performance Computing)
> systems, as typified by Red Storm and BG/L: small, fixed-configuration
> nodes with memory, network, CPU, and nothing else. The ability to not
> even have a file system on the node is a big plus. The ability to
> transparently have the file system remote/local puts the application
> into the driver's seat as to how the node is configured, and what
> tradeoffs are made; the system as a whole is incredibly flexible.
>
> Our measurements, so far, do show that Plan 9 is "quieter" than Linux. A
> full Plan 9 desktop has less OS noise than a Linux box at the login
> prompt. This matters.
>
> But it only matters if people can run their apps. Hence our concern
> about getting gcc-based cra-- er, applications code, running.
>
> I'm not really trying to make Plan 9 look like Linux. I just want to run
> MPQC for a friend of mine :-)
>
> thanks
>
> ron
next prev parent reply other threads:[~2006-06-09 19:56 UTC|newest]
Thread overview: 243+ messages / expand[flat|nested] mbox.gz Atom feed top
2006-06-07 17:58 Corey
2006-06-07 18:24 ` Roman Shaposhnick
2006-06-07 18:50 ` Ronald G Minnich
2006-06-07 18:55 ` William Josephson
2006-06-07 20:12 ` Ronald G Minnich
2006-06-07 20:20 ` Roman Shaposhnick
2006-06-07 20:42 ` Ronald G Minnich
2006-06-07 20:44 ` William Josephson
2006-06-07 21:00 ` Ronald G Minnich
2006-06-07 19:05 ` Roman Shaposhnick
2006-06-07 20:13 ` Ronald G Minnich
2006-06-07 20:48 ` Roman Shaposhnick
2006-06-07 21:04 ` Ronald G Minnich
2006-06-08 6:42 ` Roman Shaposhnik
2006-06-08 7:00 ` Iruatã Souza (muzgo)
2006-06-07 19:48 ` Corey
2006-06-07 20:16 ` Ronald G Minnich
2006-06-07 20:26 ` Roman Shaposhnick
2006-06-07 20:43 ` Ronald G Minnich
2006-06-08 5:49 ` lucio
2006-06-07 18:57 ` Francisco J Ballesteros
2006-06-07 19:55 ` Corey
2006-06-07 20:46 ` rog
2006-06-07 21:07 ` Corey
2006-06-07 21:16 ` Christoph Lohmann
2006-06-07 21:25 ` Rodolfo (kix)
2006-06-07 21:50 ` Corey
2006-06-07 21:54 ` David Leimbach
2006-06-07 21:56 ` Ronald G Minnich
2006-06-07 22:06 ` Lluís Batlle i Rossell
2006-06-07 22:16 ` David Leimbach
2006-06-07 22:31 ` Francisco J Ballesteros
2006-06-07 22:33 ` Ronald G Minnich
2006-06-08 9:44 ` Victor Nazarov
2006-06-09 21:13 ` Roman Shaposhnick
2006-06-08 1:28 ` quanstro
2006-06-09 13:12 ` Lluís Batlle
2006-06-09 13:21 ` quanstro
2006-06-09 13:47 ` Lluís Batlle
2006-06-07 23:04 ` geoff
2006-06-07 23:10 ` Steve Simon
2006-06-08 0:22 ` Bakul Shah
2006-06-08 0:27 ` geoff
2006-06-08 3:31 ` Ronald G Minnich
2006-06-08 15:04 ` David Leimbach
2006-06-08 15:20 ` Ronald G Minnich
2006-06-07 23:15 ` andrey mirtchovski
2006-06-08 0:17 ` geoff
2006-06-09 21:15 ` Roman Shaposhnick
2006-06-09 21:26 ` andrey mirtchovski
2006-06-09 21:27 ` andrey mirtchovski
2006-06-09 21:34 ` Roman Shaposhnick
2006-06-09 21:38 ` andrey mirtchovski
2006-06-09 21:44 ` andrey mirtchovski
2006-06-10 6:57 ` "Nils O. Selåsdal"
2006-06-08 1:58 ` Latchesar Ionkov
2006-06-08 0:53 ` geoff
2006-06-08 1:07 ` Latchesar Ionkov
2006-06-08 1:15 ` quanstro
2006-06-08 15:02 ` David Leimbach
2006-06-09 4:04 ` quanstro
2006-06-08 1:22 ` geoff
2006-06-08 1:25 ` Latchesar Ionkov
2006-06-09 21:21 ` Roman Shaposhnick
2006-06-09 21:27 ` Roman Shaposhnick
2006-06-08 14:59 ` David Leimbach
2006-06-09 4:05 ` quanstro
2006-06-08 1:39 ` [9fans] quantity vs. quality Lyndon Nerenberg
2006-06-08 3:05 ` Dan Cross
2006-06-08 3:44 ` Joel Salomon
2006-06-08 7:03 ` Roman Shaposhnik
2006-06-09 22:03 ` Roman Shaposhnick
2006-06-09 22:44 ` David Leimbach
2006-06-09 22:46 ` quanstro
2006-06-09 22:51 ` Latchesar Ionkov
2006-06-09 22:55 ` quanstro
2006-06-09 23:19 ` Latchesar Ionkov
2006-06-09 23:29 ` quanstro
2006-06-10 1:57 ` Latchesar Ionkov
2006-06-09 23:51 ` quanstro
2006-06-10 0:10 ` Roman Shaposhnick
2006-06-10 2:31 ` Latchesar Ionkov
2006-06-10 0:45 ` Roman Shaposhnick
2006-06-10 3:01 ` Latchesar Ionkov
2006-06-10 0:52 ` quanstro
2006-06-10 1:04 ` Roman Shaposhnick
2006-06-10 23:13 ` Ronald G Minnich
2006-06-11 0:44 ` quanstro
2006-06-11 5:08 ` lucio
2006-06-11 10:09 ` quanstro
2006-06-11 12:00 ` lucio
2006-06-11 22:59 ` quanstro
2006-06-11 23:26 ` geoff
2006-06-12 3:45 ` Paul Lalonde
2006-06-12 20:16 ` Ronald G Minnich
2006-06-12 20:23 ` Roman Shaposhnick
2006-06-12 20:56 ` Ronald G Minnich
2006-06-12 21:09 ` Victor Nazarov
2006-06-13 0:05 ` Roman Shaposhnik
2006-06-12 21:15 ` Francisco J Ballesteros
2006-06-13 12:08 ` rog
2006-06-13 16:34 ` Skip Tavakkolian
2006-06-13 21:35 ` "Nils O. Selåsdal"
2006-06-14 22:09 ` Roman Shaposhnick
2006-06-15 15:46 ` Victor Nazarov
2006-06-11 5:42 ` Russ Cox
2006-06-11 10:08 ` quanstro
2006-06-12 1:03 ` Roman Shaposhnik
2006-06-10 23:05 ` Ronald G Minnich
2006-06-11 0:00 ` quanstro
2006-06-10 0:24 ` andrey mirtchovski
2006-06-10 0:36 ` quanstro
2006-06-10 2:27 ` Latchesar Ionkov
2006-06-10 0:23 ` quanstro
2006-06-10 0:41 ` Paul Lalonde
2006-06-10 0:59 ` quanstro
2006-06-10 1:15 ` Paul Lalonde
2006-06-10 5:19 ` Bruce Ellis
2006-06-10 2:51 ` Latchesar Ionkov
2006-06-10 0:45 ` quanstro
2006-06-10 3:10 ` Latchesar Ionkov
2006-06-10 0:53 ` quanstro
2006-06-10 23:06 ` Ronald G Minnich
2006-06-10 23:15 ` geoff
2006-06-11 2:58 ` jmk
2006-06-10 23:04 ` Ronald G Minnich
2006-06-11 0:05 ` quanstro
2006-06-10 23:03 ` Ronald G Minnich
2006-06-09 23:25 ` Ronald G Minnich
2006-06-09 23:38 ` David Leimbach
2006-06-09 23:45 ` andrey mirtchovski
2006-06-09 23:46 ` Paul Lalonde
2006-06-10 23:03 ` Ronald G Minnich
2006-06-10 23:02 ` Ronald G Minnich
2006-06-11 0:12 ` quanstro
2006-06-11 2:20 ` Ronald G Minnich
2006-06-11 22:31 ` David Leimbach
2006-06-10 0:28 ` Roman Shaposhnick
2006-06-09 21:29 ` Roman Shaposhnick
2006-06-09 21:34 ` andrey mirtchovski
2006-06-08 1:49 ` [9fans] gcc on plan9 Ignacio Torres Masdeu
2006-06-08 14:10 ` David Leimbach
2006-06-09 4:09 ` quanstro
2006-06-09 5:25 ` David Leimbach
2006-06-07 22:39 ` Corey
2006-06-07 22:49 ` Paul Lalonde
2006-06-07 23:17 ` Corey
2006-06-07 23:37 ` Paul Lalonde
2006-06-08 0:50 ` quanstro
2006-06-08 1:16 ` John Barham
2006-06-08 14:51 ` David Leimbach
2006-06-09 4:49 ` ems
2006-06-09 12:07 ` quanstro
2006-06-09 19:17 ` Roman Shaposhnick
2006-06-09 21:27 ` Bakul Shah
2006-06-09 21:33 ` Francisco J Ballesteros
2006-06-09 21:39 ` quanstro
2006-06-09 21:56 ` Francisco J Ballesteros
2006-06-09 22:01 ` quanstro
2006-06-09 22:17 ` Francisco J Ballesteros
2006-06-09 22:35 ` quanstro
2006-06-09 22:22 ` Bakul Shah
2006-06-09 22:30 ` quanstro
2006-06-09 23:22 ` Bakul Shah
2006-06-10 13:51 ` Francisco J Ballesteros
2006-06-10 23:21 ` Ronald G Minnich
2006-06-11 0:07 ` LiteStar numnums
2006-06-10 0:00 ` Harri Haataja
2006-06-10 0:03 ` Bruce Ellis
2006-06-10 1:00 ` Roman Shaposhnick
2006-06-10 4:37 ` Iruatã Souza (muzgo)
2006-06-12 0:35 ` Roman Shaposhnik
2006-06-09 21:43 ` quanstro
2006-06-10 12:43 ` rog
2006-06-07 22:56 ` Christoph Lohmann
2006-06-07 23:08 ` Dan Cross
2006-06-07 23:32 ` Federico G. Benavento
2006-06-07 23:48 ` Dan Cross
2006-06-08 0:32 ` Corey
2006-06-08 1:04 ` jmk
2006-06-08 3:35 ` Ronald G Minnich
2006-06-08 3:51 ` Ronald G Minnich
2006-06-08 4:07 ` Roman Shaposhnik
2006-06-08 5:19 ` Ronald G Minnich
2006-06-08 6:07 ` Federico G. Benavento
2006-06-08 17:45 ` Micah Stetson
2006-06-08 18:54 ` Ronald G Minnich
2006-06-08 18:55 ` Iruatã Souza (muzgo)
2006-06-08 18:56 ` andrey mirtchovski
2006-06-08 19:01 ` Iruatã Souza (muzgo)
2006-06-09 19:56 ` Roman Shaposhnick [this message]
2006-06-09 22:19 ` quanstro
2006-06-09 23:19 ` Ronald G Minnich
2006-06-08 4:11 ` Paul Lalonde
2006-06-08 5:25 ` ems
2006-06-08 5:43 ` Simon Williams
2006-06-08 6:09 ` Bruce Ellis
2006-07-16 15:17 ` ASCII Glenda (was Re: [9fans] gcc on plan9) csant
2006-07-16 22:17 ` e.c.sharpe
2006-07-16 22:37 ` Jack Johnson
2006-07-17 8:35 ` Rodolfo (kix)
2006-06-08 1:06 ` [9fans] gcc on plan9 geoff
2006-06-08 1:21 ` quanstro
2006-06-08 10:15 ` Victor Nazarov
2006-06-08 9:40 ` Lluís Batlle
2006-06-08 15:55 ` Corey
2006-06-08 22:48 ` Fred Wade
2006-06-08 23:08 ` Roman Shaposhnick
2006-06-08 23:51 ` Corey
2006-06-07 20:46 ` Latchesar Ionkov
2006-06-07 19:07 ` Roman Shaposhnick
2006-06-07 19:17 ` Latchesar Ionkov
2006-06-07 19:26 ` Roman Shaposhnick
2006-06-07 19:35 ` Brantley Coile
2006-06-07 20:15 ` Ronald G Minnich
2006-06-07 20:32 ` Roman Shaposhnick
2006-06-07 19:47 ` Latchesar Ionkov
2006-06-07 20:00 ` Corey
2006-06-07 20:13 ` Roman Shaposhnick
2006-06-07 20:40 ` Ronald G Minnich
2006-06-08 6:49 ` Roman Shaposhnik
2006-06-07 21:21 ` Christoph Lohmann
2006-06-07 21:49 ` Ronald G Minnich
2006-06-07 22:50 ` Dan Cross
2006-06-08 7:08 cej
2006-06-08 9:18 ` quanstro
2006-06-08 11:58 ` rog
2006-06-09 4:20 ` quanstro
2006-06-08 17:20 ` Corey
2006-06-09 4:00 ` quanstro
2006-06-09 12:40 ` Corey
2006-06-09 13:15 ` quanstro
2006-06-09 15:10 ` Corey
2006-06-09 15:27 ` C H Forsyth
2006-06-09 15:39 ` David Leimbach
2006-06-09 15:37 ` quanstro
2006-06-09 14:52 ` lucio
2006-06-09 18:42 ` quanstro
2006-06-09 21:12 ` David Leimbach
2006-06-10 5:28 ` lucio
2006-06-10 1:36 ` Roman Shaposhnick
2006-06-10 16:32 ` David Leimbach
2006-06-08 7:24 cej
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20060609195658.GF1693@submarine \
--to=rvs@sun.com \
--cc=9fans@cse.psu.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).