[9fans] Barrelfish

[9fans] Barrelfish

[Tim Newsham]

Rethinking multi-core systems as distributed heterogeneous
systems.  Thoughts?

http://www.sigops.org/sosp/sosp09/papers/baumann-sosp09.pdf

Tim Newsham
http://www.thenewsh.com/~newsham/

Re: [9fans] Barrelfish

[Roman Shaposhnik]

On Wed, Oct 14, 2009 at 12:09 PM, Tim Newsham <newsham@lava.net> wrote:
> Rethinking multi-core systems as distributed heterogeneous
> systems.  Thoughts?

Somehow this feels related to the work that came out of Berkeley a year
or so ago. I'm still not convinced what is the benefits of multiple
kernels. If you are managing a couple of 100s of cores a single kernel
would do just fine, once the industry is ready for a couple dozen of
thousands PUs -- the kernel is most likely to be dispensed with anyway.

Did you find any ideas there particularly engaging?

Thanks,
Roman.

Re: [9fans] Barrelfish

[Tim Newsham]

> Somehow this feels related to the work that came out of Berkeley a year
> or so ago. I'm still not convinced what is the benefits of multiple
> kernels. If you are managing a couple of 100s of cores a single kernel
> would do just fine, once the industry is ready for a couple dozen of
> thousands PUs -- the kernel is most likely to be dispensed with anyway.

I'm not familiar with the berkeley work.

> Did you find any ideas there particularly engaging?

I'm still digesting it.  My first thoughts were that if my pc is a
distributed heterogeneous computer, what lessons it can borrow from
earlier work on distributed heterogeneous computing (ie. plan9).

I found the discussion on cache coherency, message passing and
optimization to be enlightening.  The fact that you may want to
organize your core OS quite a bit differently depending on which
model cpus in the same family you use is kind of scary.

The mention that "... the overhead of cache coherence restricts the
ability to scale up to even 80 cores" is also eye openeing. If we're at
aprox 8 cores today, thats only 5 yrs away (if we double cores every
1.5 yrs).

> Roman.

Tim Newsham
http://www.thenewsh.com/~newsham/

Re: [9fans] Barrelfish

[Lyndon Nerenberg (VE6BBM/VE7TFX)]

> I'm not familiar with the berkeley work.

Me either. Any chance of some references to this?

Re: [9fans] Barrelfish

[Noah Evans]

http://ramp.eecs.berkeley.edu/

Tim: Andrew Baumann is aware of Plan 9 but their approach is quite a
bit different. They are consciously avoiding the networking issue as
well(they've been asked to extend their messaging model to the network
and have actively said they're not interested).

On Wed, Oct 14, 2009 at 11:33 PM, Lyndon Nerenberg (VE6BBM/VE7TFX)
<lyndon@orthanc.ca> wrote:
>> I'm not familiar with the berkeley work.
>
> Me either. Any chance of some references to this?
>
>
>

Re: [9fans] Barrelfish

[erik quanstrom]

> http://ramp.eecs.berkeley.edu/
>
> Tim: Andrew Baumann is aware of Plan 9 but their approach is quite a
> bit different. They are consciously avoiding the networking issue as
> well(they've been asked to extend their messaging model to the network
> and have actively said they're not interested).

every interconnect is a network.  sometimes we don't admit it.

- erik

Re: [9fans] Barrelfish

[Noah Evans]

Have you read the paper? I don't think you understand the difference
in scope or goals here.

On Wed, Oct 14, 2009 at 11:45 PM, erik quanstrom <quanstro@coraid.com> wrote:
>> http://ramp.eecs.berkeley.edu/
>>
>> Tim: Andrew Baumann is aware of Plan 9 but their approach is quite a
>> bit different. They are consciously avoiding the networking issue as
>> well(they've been asked to extend their messaging model to the network
>> and have actively said they're not interested).
>
> every interconnect is a network.  sometimes we don't admit it.
>
> - erik
>
>

Re: [9fans] Barrelfish

[Eric Van Hensbergen]

On Oct 14, 2009, at 3:42 PM, Noah Evans wrote:

> http://ramp.eecs.berkeley.edu/
>
> Tim: Andrew Baumann is aware of Plan 9 but their approach is quite a
> bit different. They are consciously avoiding the networking issue as
> well(they've been asked to extend their messaging model to the network
> and have actively said they're not interested).
>

While they may not be interested in implementing a network messaging
model, they don't oppose it.  Jonathan and I talked with Andrew about
porting Barrelfish to Blue Gene yesterday to test some of their
scalability claims at large scale.

        -eric

Re: [9fans] Barrelfish

[Noah Evans]

Do want.

On Thu, Oct 15, 2009 at 12:10 AM, Eric Van Hensbergen <ericvh@gmail.com> wrote:
> On Oct 14, 2009, at 3:42 PM, Noah Evans wrote:
>
>> http://ramp.eecs.berkeley.edu/
>>
>> Tim: Andrew Baumann is aware of Plan 9 but their approach is quite a
>> bit different. They are consciously avoiding the networking issue as
>> well(they've been asked to extend their messaging model to the network
>> and have actively said they're not interested).
>>
>
> While they may not be interested in implementing a network messaging model,
> they don't oppose it.  Jonathan and I talked with Andrew about porting
> Barrelfish to Blue Gene yesterday to test some of their scalability claims
> at large scale.
>
>       -eric
>
>
>

Re: [9fans] Barrelfish

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1074 bytes --]

>
>
>
>  Did you find any ideas there particularly engaging?
>>
>
> I'm still digesting it.  My first thoughts were that if my pc is a
> distributed heterogeneous computer, what lessons it can borrow from earlier
> work on distributed heterogeneous computing (ie. plan9).
>
> I found the discussion on cache coherency, message passing and optimization
> to be enlightening.  The fact that you may want to
> organize your core OS quite a bit differently depending on which
> model cpus in the same family you use is kind of scary.
>
> The mention that "... the overhead of cache coherence restricts the ability
> to scale up to even 80 cores" is also eye openeing. If we're at aprox 8
> cores today, thats only 5 yrs away (if we double cores every
> 1.5 yrs).
>


I personally thought the use of DSLs built on Haskell was rather clever, but
the other discoveries are the sort of feedback I suspect our CPU vendors
aren't going to think about on their own somehow :-)


>
>  Roman.
>>
>
> Tim Newsham
> http://www.thenewsh.com/~newsham/
>
>

[-- Attachment #2: Type: text/html, Size: 1788 bytes --]

Re: [9fans] Barrelfish

[Roman Shaposhnik]

On Wed, Oct 14, 2009 at 2:21 PM, Tim Newsham <newsham@lava.net> wrote:
> I'm not familiar with the berkeley work.

Sorry I can't readily find the paper (the URL is somewhere on IMAP @Sun :-()
But it got presented at the Birkeley ParLab overview given to us by
Dave Patterson.
They were talking thin hypervisors and that sort of stuff. More details here:
   http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-23.pdf
(page 10) but still no original paper in sight...

> I'm still digesting it.  My first thoughts were that if my pc is a
> distributed heterogeneous computer

It may very well be that, but why would you want to manage that complexity?
Your GPU is already heavily "multicore", yet you don't see it (and you really
don't want to see it!)

> The mention that "... the overhead of cache coherence restricts the ability
> to scale up to even 80 cores" is also eye openeing. If we're at aprox 8
> cores today, thats only 5 yrs away (if we double cores every
> 1.5 yrs).

A couple of years ago we had a Plan9 summit @Google campus and Ken was
there. I still remember the question he asked me: what exactly would you make
all those core do on your desktop?

Frankly, I still don't have a good answer.

Thanks,
Roman.

Re: [9fans] Barrelfish

[Eric Van Hensbergen]

On Oct 14, 2009, at 7:50 PM, Roman Shaposhnik wrote:

> On Wed, Oct 14, 2009 at 2:21 PM, Tim Newsham <newsham@lava.net> wrote:
>> I'm not familiar with the berkeley work.
>
> Sorry I can't readily find the paper (the URL is somewhere on IMAP
> @Sun :-()
> But it got presented at the Birkeley ParLab overview given to us by
> Dave Patterson.
> They were talking thin hypervisors and that sort of stuff. More
> details here:
>   http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-23.pdf
> (page 10) but still no original paper in sight...

You may be thinking about the Tesselation work from Berkley ParLab (http://parlab.eecs.berkeley.edu/publication/221
)

        -eric

Re: [9fans] Barrelfish

[Sam Watkins]

On Wed, Oct 14, 2009 at 06:50:28PM -0700, Roman Shaposhnik wrote:
> > The mention that "... the overhead of cache coherence restricts the ability
> > to scale up to even 80 cores" is also eye openeing. If we're at aprox 8
> > cores today, thats only 5 yrs away (if we double cores every
> > 1.5 yrs).

Sharing the memory between processes is a stupid approach to multi-processing /
multi-threading.  Modern popular computer architecture and software design is
fairly much uniformly stupid.

> A couple of years ago we had a Plan9 summit @Google campus and Ken was
> there. I still remember the question he asked me: what exactly would you make
> all those core do on your desktop?

It's easy to write good code that will take advantage of arbitrarily many
processors to run faster / smoother, if you have a proper language for the
task.  With respect to Ken, Bill Gates said something along the lines of "who
would need more than 640K?".  There is a vast range of applications that cannot
be managed in real time using existing single-core technology.

Sam

Re: [9fans] Barrelfish

[Richard Miller]

> It's easy to write good code that will take advantage of arbitrarily many
> processors to run faster / smoother, if you have a proper language for the
> task.

... and if you can find a way around Amdahl's law (qv).

Re: [9fans] Barrelfish

[W B Hacker]

Richard Miller wrote:
>> It's easy to write good code that will take advantage of arbitrarily many
>> processors to run faster / smoother, if you have a proper language for the
>> task.
>
> ... and if you can find a way around Amdahl's law (qv).
>
>
>

http://www.cis.temple.edu/~shi/docs/amdahl/amdahl.html

Re: [9fans] Barrelfish

[Sam Watkins]

On Thu, Oct 15, 2009 at 12:50:48PM +0100, Richard Miller wrote:
> > It's easy to write good code that will take advantage of arbitrarily many
> > processors to run faster / smoother, if you have a proper language for the
> > task.
>
> ... and if you can find a way around Amdahl's law (qv).

"The speedup of a program using multiple processors in parallel computing is
limited by the time needed for the sequential fraction of the program."

So it would only be a problem supposing that a significant part of the program
is unparallelizable.  I can think of many many tasks where "Amdahl's law" is
not going to be a problem at all, for a properly designed system.

For example if I had a thousand processors I might raytrace complex scenes for
an animated game at 100 fps, or do complex dsp over a 2 hour audio track in one
millisecond.

I suppose most difficult/interesting tasks can be parallelized effectively.
Seems that Amdahl's law is a minor issue.  Of course if you are trying to run
old-fashioned sequential programs on a parallel machine you will not benefit.
You would need to rewrite them.

Sam

Re: [9fans] Barrelfish

[ron minnich]

On Fri, Oct 16, 2009 at 10:03 AM, Sam Watkins <sam@nipl.net> wrote:

> So it would only be a problem supposing that a significant part of the program
> is unparallelizable.  I can think of many many tasks where "Amdahl's law" is
> not going to be a problem at all, for a properly designed system.
>
> For example if I had a thousand processors I might raytrace complex scenes for
> an animated game at 100 fps, or do complex dsp over a 2 hour audio track in one
> millisecond.

Yes, if you had that few processors, it might seem easy.

It gets somewhat harder when you have, say, 128,000. Insignificant
bits of code that were not even visible suddenly dominate the time.

ron

Re: [9fans] Barrelfish

[Wes Kussmaul]

ron minnich wrote:

> Insignificant
> bits of code that were not even visible suddenly dominate the time.

Reminds me of some project development teams.

Maybe Marvin Minsky was on to something.

Re: [9fans] Barrelfish

[Roman Shaposhnik]

On Fri, Oct 16, 2009 at 10:03 AM, Sam Watkins <sam@nipl.net> wrote:
> On Thu, Oct 15, 2009 at 12:50:48PM +0100, Richard Miller wrote:
>> > It's easy to write good code that will take advantage of arbitrarily many
>> > processors to run faster / smoother, if you have a proper language for the
>> > task.
>>
>> ... and if you can find a way around Amdahl's law (qv).
>
> "The speedup of a program using multiple processors in parallel computing is
> limited by the time needed for the sequential fraction of the program."
>
> So it would only be a problem supposing that a significant part of the program
> is unparallelizable.  I can think of many many tasks where "Amdahl's law" is
> not going to be a problem at all, for a properly designed system.

Lets do a little math, shall we? Better yet, lets graph it:
   http://en.wikipedia.org/wiki/File:AmdahlsLaw.svg

Now, do you see what's on the right side of X axis? That's
right 65536 cores. Pause and appreciate the measeleness
of speedup...

Thanks,
Roman.

Re: [9fans] Barrelfish

[Josh Wood]

On Oct 15, 2009, at 3:53 AM, Sam Watkins wrote:

> With respect to Ken, Bill Gates said something along the lines of "who
> would need more than 640K?"

With respect to Ken, from Roman's report, you only know that he asked
a question. Roman was the one without an answer, and no one echoed
Gates's arbitrary limit.

-Josh

Re: [9fans] Barrelfish

[hiro]

> There is a vast range of applications that cannot
> be managed in real time using existing single-core technology.

I'm sorry to interrupt your discussion, but what is real time?

Re: [9fans] Barrelfish

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

On Thu, Oct 15, 2009 at 6:11 AM, hiro <23hiro@googlemail.com> wrote:

> > There is a vast range of applications that cannot
> > be managed in real time using existing single-core technology.
>
> I'm sorry to interrupt your discussion, but what is real time?
>
>
Real time just means "fast enough to work properly".  You can throw all
kinds of other crap on top of that and say things about scheduling
requirements and timeslices within which a process must complete, and duty
cycles, but those are just things to look at to figure out if your system is
"fast enough to work properly"

Dave

[-- Attachment #2: Type: text/html, Size: 932 bytes --]

Re: [9fans] Barrelfish

[Roman Shaposhnik]

On Thu, Oct 15, 2009 at 10:53 AM, Sam Watkins <sam@nipl.net> wrote:
> On Wed, Oct 14, 2009 at 06:50:28PM -0700, Roman Shaposhnik wrote:
>> > The mention that "... the overhead of cache coherence restricts the ability
>> > to scale up to even 80 cores" is also eye openeing. If we're at aprox 8
>> > cores today, thats only 5 yrs away (if we double cores every
>> > 1.5 yrs).
>
> Sharing the memory between processes is a stupid approach to multi-processing /
> multi-threading.  Modern popular computer architecture and software design is
> fairly much uniformly stupid.

It is. But what's your proposal on code sharing? All those PC
registers belonging to
different cores have to point somewhere. Is that somewhere is not shared memory
the code has to be put there for every single core, right?

Thanks,
Roman.

Re: [9fans] Barrelfish

[Bakul Shah]

On Sun, 18 Oct 2009 01:15:45 -0000 Roman Shaposhnik <roman@shaposhnik.org>  wrote:
> On Thu, Oct 15, 2009 at 10:53 AM, Sam Watkins <sam@nipl.net> wrote:
> > On Wed, Oct 14, 2009 at 06:50:28PM -0700, Roman Shaposhnik wrote:
> >> > The mention that "... the overhead of cache coherence restricts the ab=
> ility
> >> > to scale up to even 80 cores" is also eye openeing. If we're at aprox =
> 8
> >> > cores today, thats only 5 yrs away (if we double cores every
> >> > 1.5 yrs).
> >
> > Sharing the memory between processes is a stupid approach to multi-proces=
> sing /
> > multi-threading. =A0Modern popular computer architecture and software des=
> ign is
> > fairly much uniformly stupid.

> It is. But what's your proposal on code sharing? All those PC
> registers belonging to
> different cores have to point somewhere. Is that somewhere is not shared me=
> mory
> the code has to be put there for every single core, right?

Different technoglogies/techniques make sense at different
levels of scaling and at different points in time so sharing
memory is not necessarily stupid -- unless one thinks that
any compromise (to produce usable solutions in a realistic
time frame) is stupid.

At the hardware level we do have message passing between a
processor and the memory controller -- this is exactly the
same as talking to a shared server and has the same issues of
scaling etc. If you have very few clients, a single shared
server is indeed a cost effective solution.

When you absolutely have to share state, somebody has to
mediate access to the shared state and you can't get around
the fact that it's going to cost you.  But if you know
something about the patterns of sharing, you can get away
from a single shared memory & increase concurrency.  A simple
example is a h/w fifo (to connect producer/consumer but you
also gave up some flexibility).

As the number of processors increases on a device, sharing
state between neighbors will be increasingly cheaper compared
any global sharing. Even if you use message passing, messages
between near neighbors will be far cheaper than between
processors in different neighboorhoods. So switching to
message passing is not going to fix things; you have to worry
about placement as well (just like in h/w design).

Re: [9fans] Barrelfish

[Roman Shaposhnik]

On Sun, Oct 18, 2009 at 6:06 AM, Roman Shaposhnik <shaposhnik@gmail.com> wrote
>> It is. But what's your proposal on code sharing? All those PC
>> registers belonging to
>> different cores have to point somewhere. Is that somewhere is not shared me=
>> mory
>> the code has to be put there for every single core, right?
>
> At the hardware level we do have message passing between a
> processor and the memory controller -- this is exactly the
> same as talking to a shared server and has the same issues of
> scaling etc. If you have very few clients, a single shared
> server is indeed a cost effective solution.

I guess I'm not following. My question to OP was strictly about
code sharing. Basically were do the cores get instructions from
if not from shared memory.

Thanks,
Roman.

Re: [9fans] Barrelfish

[Bakul Shah]

On Sun, 18 Oct 2009 06:22:33 PDT Roman Shaposhnik <roman@shaposhnik.org>  wrote:
> On Sun, Oct 18, 2009 at 6:06 AM, Roman Shaposhnik <shaposhnik@gmail.com> wrot
> e
> >> It is. But what's your proposal on code sharing? All those PC
> >> registers belonging to
> >> different cores have to point somewhere. Is that somewhere is not shared m
> e=
> >> mory
> >> the code has to be put there for every single core, right?
> >
> > At the hardware level we do have message passing between a
> > processor and the memory controller -- this is exactly the
> > same as talking to a shared server and has the same issues of
> > scaling etc. If you have very few clients, a single shared
> > server is indeed a cost effective solution.
>
> I guess I'm not following. My question to OP was strictly about
> code sharing. Basically were do the cores get instructions from
> if not from shared memory.

Sorry, I should've done a better job of editing.  I was
really responding to the OP's point that sharing memory
between processes is a stupid approach. My point was that
"sharing memory" is just a low level programming interface
(implemented by message passing in h/w) and it makes sense at
some scale.

Re: [9fans] Barrelfish

[ron minnich]

Since we seem to be having the parallel programming discussion again
please look at this:
https://asc.llnl.gov/sequoia/benchmarks/

The summaries are interesting.

ron

Re: [9fans] Parallelism is over a barrel(fish)?

[Lyndon Nerenberg (VE6BBM/VE7TFX)]

>From last week's ACM Technews ...

Why Desktop Multiprocessing Has Speed Limits
Computerworld (10/05/09) Vol. 43, No. 30, P. 24; Wood, Lamont

Despite the mainstreaming of multicore processors for desktops, not
every desktop application can be rewritten for multicore frameworks,
which means some bottlenecks will persist.  "If you have a task that
cannot be parallelized and you are currently on a plateau of
performance in a single-processor environment, you will not see that
task getting significantly faster in the future," says analyst Tom
Halfhill.  Adobe Systems' Russell Williams points out that performance
does not scale linearly even with parallelization on account of memory
bandwidth issues and delays dictated by interprocessor communications.
Analyst Jim Turley says that, overall, consumer operating systems
"don't do anything smart" with multicore architecture.  "We have to
reinvent computing, and get away from the fundamental premises we
inherited from von Neumann," says Microsoft technical fellow Burton
Smith.  "He assumed one instruction would be executed at a time, and
we are no longer even maintaining the appearance of one instruction at
a time." Analyst Rob Enderle notes that most applications will operate
on only a single core, which means that the benefits of a multicore
architecture only come when multiple applications are run.  "What we'd
all like is a magic compiler that takes yesterday's source code and
spreads it across multiple cores, and that is just not happening,"
says Turley.  Despite the performance issues, vendors prefer multicore
processors because they can facilitate a higher level of power
efficiency.  "Using multiple cores will let us get more performance
while staying within the power envelope," says Acer's Glenn Jystad.

http://www.computerworld.com/s/article/342870/The_Desktop_Traffic_Jam?intsrc=print_latest

Re: [9fans] Parallelism is over a barrel(fish)?

[W B Hacker]

Lyndon Nerenberg (VE6BBM/VE7TFX) wrote:
>>From last week's ACM Technews ...
>
> Why Desktop Multiprocessing Has Speed Limits
> Computerworld (10/05/09) Vol. 43, No. 30, P. 24; Wood, Lamont
>
> Despite the mainstreaming of multicore processors for desktops, not
> every desktop application can be rewritten for multicore frameworks,
> which means some bottlenecks will persist.  "If you have a task that
> cannot be parallelized and you are currently on a plateau of
> performance in a single-processor environment, you will not see that
> task getting significantly faster in the future," says analyst Tom
> Halfhill.  Adobe Systems' Russell Williams points out that performance
> does not scale linearly even with parallelization on account of memory
> bandwidth issues and delays dictated by interprocessor communications.
> Analyst Jim Turley says that, overall, consumer operating systems
> "don't do anything smart" with multicore architecture.  "We have to
> reinvent computing, and get away from the fundamental premises we
> inherited from von Neumann," says Microsoft technical fellow Burton
> Smith.  "He assumed one instruction would be executed at a time, and
> we are no longer even maintaining the appearance of one instruction at
> a time." Analyst Rob Enderle notes that most applications will operate
> on only a single core, which means that the benefits of a multicore
> architecture only come when multiple applications are run.  "What we'd
> all like is a magic compiler that takes yesterday's source code and
> spreads it across multiple cores, and that is just not happening,"
> says Turley.  Despite the performance issues, vendors prefer multicore
> processors because they can facilitate a higher level of power
> efficiency.  "Using multiple cores will let us get more performance
> while staying within the power envelope," says Acer's Glenn Jystad.
>
> http://www.computerworld.com/s/article/342870/The_Desktop_Traffic_Jam?intsrc=print_latest
>
>
>
'The usual <talking through their anatomy> suspects'....

But they miss the point. Work will always be foudn for 'faster' devices, but the
majority of the 'needed' benefit has been accomplished until entirely new
challenges surface. Computer games and digital video special-effects are just candy.

Even a dual-core allows moving the overhead, housekeeping, I/O, interrupt
servicing, et al out of the way of a single-core-bound application. OS/2 Hybrid
Multi-Procesing - even with unmodified Win 3X era apps.

Beyond that it matters little.

Given a 'decent' (not magical)[1] OS, and environment, the apps that actually
*matter* to 99% of the population are more than fast enough on the likes of a
VIA C6 --> nano/Geode/Atom [2], embedded Ppc [3], or even an ARMish single-core
[4]- with or without DSP etc. on-substrate.

Faster storage and networks now matter far more than faster local CPU.

The ratio of these 'goodies', and their benefits to the population in general
to the count of supercomputers [5] and near-real-time video-stream processors
[6] is - and will remain - extremely lopsided in favor of the small 'appliance'.

Those hyping multi-multi core for the consumer 'PC" market are locked ino an
obsolete behaviour pattern.

Lower power consumption, smaller form-factor, better display and input interface
faster networking is where the need lies.

Nothing yet shipped can match the effectiveness of an experienced Wife or
Personal Assistant (human) at the other end of an ordinary phone line when (s)he
has *anticipated* your needs and called you *before* you recognized the need
yourself.

Code THAT into silicon, teach it to cook, and you still have a lousy bed-partner...


Bill Hacker


[1] Anything not horribly wasteful (eg - not Windows), such as Plan9, any *BSD,
the leaner Linux (Vector/Slackware), Haiku - all make a more than fast enough
desktop on any single-core of 700 MHz or better, even if dragging X-Windows and
the like around as a boat-anchor.

[2] Laptops amd Netbooks

[3] Embedded high-end. Game boxen, Ford and other motor cars

[4] A large percentage of PDA's and telecoms handhelds

[5] Devilishly hard to substitute for, SETI et al notwithstanding, but needed in
relatively small numbers vs, for example, a mobile phone or automobile
fuel/pollution reduction system.

[6] Given the preponderance of dreck spewed from television and cinema,
civilization could well be better-off if all such devices on the planet went on
a long holiday and humans returned to actually paying attention to one another.

Re: [9fans] Barrelfish

[Eric Van Hensbergen]

And how does one deal with heterogeneous cores and complex on chip
interconnect topologies?  Barrelfish also gas a nice benefit in that
it could span coherence domains.

There's no real evdence that single kernels do well with hundreds of
real cores (as opposed to hw threads) - in fact most of the data I've
seen is to the contrary.

Sent from my iPhone

On Oct 14, 2009, at 1:54 PM, Roman Shaposhnik <roman@shaposhnik.org>
wrote:

> On Wed, Oct 14, 2009 at 12:09 PM, Tim Newsham <newsham@lava.net>
> wrote:
>> Rethinking multi-core systems as distributed heterogeneous
>> systems.  Thoughts?
>
> Somehow this feels related to the work that came out of Berkeley a
> year
> or so ago. I'm still not convinced what is the benefits of multiple
> kernels. If you are managing a couple of 100s of cores a single kernel
> would do just fine, once the industry is ready for a couple dozen of
> thousands PUs -- the kernel is most likely to be dispensed with
> anyway.
>
> Did you find any ideas there particularly engaging?
>
> Thanks,
> Roman.
>

Re: [9fans] Barrelfish

[Roman Shaposhnik]

> And how does one deal with heterogeneous cores and complex on chip
> interconnect topologies?

Good question. Do they have to be heterogeneous? My oppinion is that the
future of big multicore will be more Cell-like.

> There's no real evdence that single kernels do well with hundreds of real
> cores (as opposed to hw threads) - in fact most of the data I've seen is to
> the contrary.

Agreed. But then, again, you don't really want a kernel for anything but message
passing in such an architecture (the other function of the kernel --
multiplexing
I/O is only needed on selected few cores) at which point it really becomes a
misnomer to even call it a kernel -- a thin hypervisor perhaps...

Thanks,
Roman.

Re: [9fans] Barrelfish

[Eric Van Hensbergen]

On Oct 14, 2009, at 8:05 PM, Roman Shaposhnik wrote:

>> And how does one deal with heterogeneous cores and complex on chip
>> interconnect topologies?
>
> Good question. Do they have to be heterogeneous? My oppinion is that
> the
> future of big multicore will be more Cell-like.
>

They don't have to be, but that is part of both the multikernel and
satellite kernel vision.

>> There's no real evdence that single kernels do well with hundreds
>> of real
>> cores (as opposed to hw threads) - in fact most of the data I've
>> seen is to
>> the contrary.
>
> Agreed. But then, again, you don't really want a kernel for anything
> but message
> passing in such an architecture (the other function of the kernel --
> multiplexing
> I/O is only needed on selected few cores) at which point it really
> becomes a
> misnomer to even call it a kernel -- a thin hypervisor perhaps...
>

If you look at the core of Barrelfish, you'll see that this is
essentially what they are doing -- essentially using an extremely
small microkernel (like L4) that's very
efficient at various forms of message passing.  That's the only thing
that is duplicated on the various cores.  The services themselves can
be distributed
and/or replicated as appropriate (although their approach favors
replication) -- it all depends on the characteristics of the workload.

      -eric

Re: [9fans] Barrelfish

[Tim Newsham]

> If you look at the core of Barrelfish, you'll see that this is essentially
> what they are doing -- essentially using an extremely small microkernel (like
> L4) that's very
> efficient at various forms of message passing.  That's the only thing that is
> duplicated on the various cores.  The services themselves can be distributed
> and/or replicated as appropriate (although their approach favors replication)
> -- it all depends on the characteristics of the workload.

it sounds like the kernel (L4-like, supposedly tuned to the specific
hardware) and the "monitor" (userland, portable) are shared, from
the paper.  Btw, they have the source code up for free
(http://www.barrelfish.org/release_20090914.html) which I supposed
could be used to more definitively answer these questions with
some effort...

>    -eric

Tim Newsham
http://www.thenewsh.com/~newsham/

Re: [9fans] Barrelfish

[Eric Van Hensbergen]

On Oct 14, 2009, at 9:32 PM, Tim Newsham wrote:

>> If you look at the core of Barrelfish, you'll see that this is
>> essentially what they are doing -- essentially using an extremely
>> small microkernel (like L4) that's very
>> efficient at various forms of message passing.  That's the only
>> thing that is duplicated on the various cores.  The services
>> themselves can be distributed
>> and/or replicated as appropriate (although their approach favors
>> replication) -- it all depends on the characteristics of the
>> workload.
>
> it sounds like the kernel (L4-like, supposedly tuned to the specific
> hardware) and the "monitor" (userland, portable) are shared, from
> the paper.

I'm confused what you mean by "shared".
The monitor is replicated on every core as it is responsible for
coordination amongst the cores - some things are replicated while
others are coordinated.  They do choose to replicate most things as
part of their core scalability argument, in an effort to reduce lock
contention to centralized resources.

(from section 4.4): On each core, replicated data structures, such as
memory alloca- tion tables and address space mappings, are kept
globally consistent by means of an agreement protocol run by the
monitors. Application requests that access global state are handled by
the monitors, which mediate access to remote copies of state.

       -eric

Re: [9fans] Barrelfish

[Tim Newsham]

>> it sounds like the kernel (L4-like, supposedly tuned to the specific
>> hardware) and the "monitor" (userland, portable) are shared, from
>> the paper.
>
> I'm confused what you mean by "shared".

ugh, I completely botched that.. I meant "replicated" not "shared".

>     -eric

Tim Newsham
http://www.thenewsh.com/~newsham/

Re: [9fans] Barrelfish

[Christopher Nielsen]

I think this is an interesting approach.

There are several interesting ideas being pursued here. The focus of
the discussion has been on the multikernel approach, which I think has
merit.

Something that has not been discussed here is the wide use of DSLs for
systems programming, and using haskell to write a framework for
rapidly developing and proving correctness of DSLs. This is just as
significant as the multikernel ideas.

I downloaded the source, built the system, and will be playing with it.

Thoughts?

On Wed, Oct 14, 2009 at 12:09, Tim Newsham <newsham@lava.net> wrote:
> Rethinking multi-core systems as distributed heterogeneous
> systems.  Thoughts?
>
> http://www.sigops.org/sosp/sosp09/papers/baumann-sosp09.pdf
>
> Tim Newsham
> http://www.thenewsh.com/~newsham/
>
>



-- 
Christopher Nielsen
"They who can give up essential liberty for temporary
safety, deserve neither liberty nor safety." --Benjamin Franklin

Re: [9fans] Barrelfish

[W B Hacker]

Christopher Nielsen wrote:
> I think this is an interesting approach.
>
> There are several interesting ideas being pursued here. The focus of
> the discussion has been on the multikernel approach, which I think has
> merit.
>
> Something that has not been discussed here is the wide use of DSLs for
> systems programming, and using haskell to write a framework for
> rapidly developing and proving correctness of DSLs. This is just as
> significant as the multikernel ideas.
>
> I downloaded the source, built the system, and will be playing with it.
>
> Thoughts?

Their 'plan' for security needs a recce as well.

Message-passing-based 'creatures' - kernel-level or otherwise - have their own
challenges in this regard (Windows, to name one bad example). Likewise, though
I've only just started looking at it, if 'Haiku' even *has* a security model, I
am (still, yet) blissfuly unaware of it...

Bill Hacker

>
> On Wed, Oct 14, 2009 at 12:09, Tim Newsham <newsham@lava.net> wrote:
>> Rethinking multi-core systems as distributed heterogeneous
>> systems.  Thoughts?
>>
>> http://www.sigops.org/sosp/sosp09/papers/baumann-sosp09.pdf
>>
>> Tim Newsham
>> http://www.thenewsh.com/~newsham/
>>
>>
>
>
>