Re: [9fans] Plan 9 on Blue Gene

Re: [9fans] Plan 9 on Blue Gene

[erik quanstrom]

> I've been writing a lot of Erlang code lately, and I keep thinking about,
> but not having too much time to do much about, wanting to have a runtime for
> the libthread "threads" that could auto-schedule them to libthread "procs",
> in much the same way Haskell "sparks" may end up real threads, or Erlang
> processes, might run in parallel.

the model is that there may be any number of procs sharing memory,
channels, etc.  each proc has at least one thread.  threads have their
own stack and one one-at-a time.

since threads run one at a time and have a few, well-known calls that
implicitly schedule, one often needs no locking.  this is like the big
kernel lock in linux.  and so in general converting threadcreate to
proccreate will break programs which rely on the implicit mutex
between threads to keep memory accesses from overlapping.

my personal and uninformed opinion is that it's better to be explicit
about resource sharing and just lock critical sections—or better yet,
don't overlap data use.  use channels to transfer data.  if there's no
overlapping data access then proccreate and threadcreate may usually
be interchangable.

another personal opinion is that parallelism can be a
"performance hack".  however, when the speed differences
between various resources (e.g. disk drive seek vs anything else)
is great enough, the difference between working and broken
can be parallelism.

- erik

Re: [9fans] Plan 9 on Blue Gene

[Uriel]

On Thu, Jul 31, 2008 at 4:23 AM, ron minnich <rminnich@gmail.com> wrote:
> On Wed, Jul 30, 2008 at 6:48 PM, David Leimbach <leimy2k@gmail.com> wrote:
>
>> Does Plan 9 Port help?  I mean, libthread on Plan 9 Port alone could be
>> worth a ton to me in some situations.
>> Concurrent programming for the win?
>
> probably not for this community. When we had plan9port in xcpu we got
> nothing but complaints. This in spite of the fact that some things are
> impossible to scale with 5000 posix threads, and easy to scale with
> 5000 plan 9 style threads.

Why not use rsc's libtask instead? It would avoid most of the p9p
baggage (which certainly it is not designed to make it easy for people
to build apps that depend on it).

libtask is small enough that it could easily be distributed together with xcpu.

Just an ignorant suggestion by someone that is not even clear on what xcpu does.

uriel

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Thu, Jul 31, 2008 at 3:03 PM, Charles Forsyth <forsyth@terzarima.net> wrote:

> speaking of higher levels of abstraction:
> given some scientific code i've seen (before this, nothing to do with the things
> running on Blue Gene), i'd observe that fixing some of the algorithms used (which
> is compiler and platform independent activity) will often yield much bigger results
> than (say) compiling it with gcc, xlc, xlf, etc.  you'd be amazed (or perhaps not)
> how naive some of the code can be.

Excellent point. Some of the app code is terrible. I am hoping we'll
prove that we can "go native" and do as well as or better than the
existing apps. At the same time, having just gotten badly burned on
Clustermatic because people wanted to run xemacs on compute nodes and
got unhappy when we said no, I've gotten sensitized to the "wants" of
users. How you respond to stupid demands can make or break your
project. It's hard to take but we're a service industry.

ron

Re: [9fans] Plan 9 on Blue Gene

[Charles Forsyth]

you could change the subject line to continue this discussion
for other reasons, but for this particular work on plan 9,
it's not worth getting into a discussion of aspects of belief
in particular C implementations.  (just to add a contrasting data point,
at my previous employment we had examples of important scientific code on power where gcc
compiled faster(!) and compiled vastly faster code than xlc, for instance.)
it doesn't matter. it isn't relevant to this plan 9 project, so i think there's not much point
in spending time on it here, at least once we've satisfactorily explained why it doesn't matter.
compilers are programs. once you know or can guess what another program does
that's clever in your case, you can do it too (subject perhaps to patents).
in fact, you can often do it simpler and better because you can work out what
really made the difference in the earlier case, or (better) deal with it at a higher level
of abstraction.

improving qc/9c code on power/power64 is likely to happen in the course
of this project, where needed to make the kernel and plan 9-specific applications run better.
it probably isn't worth the effort (at least for the current project)
to do that for arbitrary scientific code, and is certainly
outside the scope of the agreed specification of the project.

speaking of higher levels of abstraction:
given some scientific code i've seen (before this, nothing to do with the things
running on Blue Gene), i'd observe that fixing some of the algorithms used (which
is compiler and platform independent activity) will often yield much bigger results
than (say) compiling it with gcc, xlc, xlf, etc.  you'd be amazed (or perhaps not)
how naive some of the code can be.

then there's the bigger question about which language to use to produce much
faster code easily, and i'd hazard a moderately informed guess that C is not the language
of choice for a lot of that.  again, that's outside the scope of our project.

Re: [9fans] Plan 9 on Blue Gene

[Charles Forsyth]

> Correct me if I'm wrong here, but don't these require extensive run-time
> support, in addition to compiler support?  Will the run-time libraries
> also be linux libraries running under a compatibility layer?

yes, but the final answer to the second question is unclear,
depending on the library.  the further up you go, the more likely
it will be left unchanged, if only because there are so many more of them.
if a higher-level library uses a lower-level one and cheats, by being written
to a particular implementation of that library not to a supposedly agreed interface,
then there is less flexibility for replacing the latter, too.

Re: [9fans] Plan 9 on Blue Gene

[Charles Forsyth]

> Just a dumb question, as i'm totally out of this business, it is easier to write an emulator than translate the applications to plan9 c ? (for example) or to write (or port) the C++ and Fortran compilers and related tools?

it is easier (in both cases), but it's also not the point. applications typically aren't compiled
on blue gene, and it's closer to an embedded environment from that point of view.
although it is possible to compile code on an iphone, it's typically silly.

you don't "port" the variants of the compilers that people used;
they weren't written to be "portable" in the sense required for plan 9 (even gcc).
trying to write new ones (eg, for Fortran) is one of those things
that's certainly possible in principle but in practice probably a complete waste of money (and probably time),
certainly in the context of the current project.

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Thu, Jul 31, 2008 at 5:53 AM, Philippe Anel <xigh@bouyapop.org> wrote:
> Can you tell us why some things are impossible to scale with 5000 posix
> threads (and easy to scale with 5000 plan 9 style threads) ?

the trivial stuff you deal with first, like the default 8MB thread
stack which makes the machine fall over.

But it still seems to very hard on a linux box to spawn 5k threads and
run them, one thread to a socket. The whole machine just starts to bog
down.

I did not write this particular code, was only a victim of it.

ron

Re: [9fans] Plan 9 on Blue Gene

[erik quanstrom]

> Obviously depending on the C calls involved, some are re-entrant and some
> are not, and that makes a rather large difference too.  You don't want to
> call strtok from two different threads (unless you have thread local storage
> available, then it might be safe).

malloc?  it should be available everywhere.  while the heap is not thread-local,
if the pointer to it is on the stack, functions like strtok may be used.

- erik

Re: [9fans] Plan 9 on Blue Gene

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]

On Thu, Jul 31, 2008 at 7:11 AM, Philippe Anel <xigh@bouyapop.org> wrote:

>  That's almost what they do with KSE in FreeBSD (or Scheduler Activation
> in NetBSD) right ?
>
> Phil;
>

KSEs were a way to make things that were not processes or threads in
particular cooperatively scheduled with user-space callbacks.  Basically the
kernel signaled user-space to make a decision on what the next context would
be.  This is not the same a cooperatively scheduled coroutines where the
re-scheduling happens as the result of making certain library calls to
certain I/O routines  (libtask does this, thanks again Russ!)

KSEs were a mechanism to implement true-pre-emptive M:N threading for Posix
threads... allowing M pthreads to be scheduled on N user processes. (I may
have my M's and N's backwards).

Usually what I've been seeing is the more popular thread implementations are
1:1, where each pthread is a user process as well.   I'm not sure if KSE
based threads or M:N in general, are a win in every situation people had
hoped (or any for that matter).  But then again I've not looked terribly
deeply at this.  1:1 seems easier to reason about and keep track of :-)

Dave


>
>
>
>
>  I've been writing a lot of Erlang code lately, and I keep thinking about,
> but not having too much time to do much about, wanting to have a runtime for
> the libthread "threads" that could auto-schedule them to libthread "procs",
> in much the same way Haskell "sparks" may end up real threads, or Erlang
> processes, might run in parallel.
>
> Dave
>
>
>>
>>
>> ----- Original Message ----- From: "ron minnich" <rminnich@gmail.com>
>> To: "Fans of the OS Plan 9 from Bell Labs" <9fans@9fans.net>
>> Sent: Thursday, July 31, 2008 4:23 AM
>> Subject: Re: [9fans] Plan 9 on Blue Gene
>>
>>
>>
>> On Wed, Jul 30, 2008 at 6:48 PM, David Leimbach <leimy2k@gmail.com>
>>> wrote:
>>>
>>> Does Plan 9 Port help?  I mean, libthread on Plan 9 Port alone could be
>>>> worth a ton to me in some situations.
>>>> Concurrent programming for the win?
>>>>
>>>
>>> probably not for this community. When we had plan9port in xcpu we got
>>> nothing but complaints. This in spite of the fact that some things are
>>> impossible to scale with 5000 posix threads, and easy to scale with
>>> 5000 plan 9 style threads.
>>>
>>> ron
>>>
>>>
>>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4249 bytes --]

Re: [9fans] Plan 9 on Blue Gene

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 2999 bytes --]

On Thu, Jul 31, 2008 at 7:01 AM, erik quanstrom <quanstro@quanstro.net>wrote:

> > I've been writing a lot of Erlang code lately, and I keep thinking about,
> > but not having too much time to do much about, wanting to have a runtime
> for
> > the libthread "threads" that could auto-schedule them to libthread
> "procs",
> > in much the same way Haskell "sparks" may end up real threads, or Erlang
> > processes, might run in parallel.
>
> the model is that there may be any number of procs sharing memory,
> channels, etc.  each proc has at least one thread.  threads have their
> own stack and one one-at-a time.
>
> since threads run one at a time and have a few, well-known calls that
> implicitly schedule, one often needs no locking.  this is like the big
> kernel lock in linux.  and so in general converting threadcreate to
> proccreate will break programs which rely on the implicit mutex

between threads to keep memory accesses from overlapping.


Channels are locked though right, implicitly, such that they can be used as
a "safe" communication mechanism between proccreate'd contexts and
threadcreate'd contexts in a safe way.

Therefore, if you're not sharing memory between threads/procs, the model
holds up just fine for the CSP style concurrency it was intended to
implement.  Basically "don't share", and "just do message passing".

Obviously depending on the C calls involved, some are re-entrant and some
are not, and that makes a rather large difference too.  You don't want to
call strtok from two different threads (unless you have thread local storage
available, then it might be safe).


>
>
> my personal and uninformed opinion is that it's better to be explicit
> about resource sharing and just lock critical sections—or better yet,
> don't overlap data use.  use channels to transfer data.  if there's no
> overlapping data access then proccreate and threadcreate may usually
> be interchangable.


This is exactly what I'm referring to.


>
>
> another personal opinion is that parallelism can be a
> "performance hack".  however, when the speed differences
> between various resources (e.g. disk drive seek vs anything else)
> is great enough, the difference between working and broken
> can be parallelism.


I think if you take the definition of concurrent processes, that is any set
of processes with no defined sequence or ordering to them running, and you
have them cooperating to produce a solution AND they can run at the same
time, you're talking about a form of parallelism.  For many people,
designing code that's concurrent is enough to express the solution in a sane
way, without tracking mutexes and locks and shared resources, and
parallelism is just "icing".

For some "pure parallelism" is the end goal.  And there's often a very
different approach to getting that (OpenMP, Parallel Haskell vs Concurrent
Haskell.  Erlang running with SMP enabled.)


Dave


>
>
> - erik
>

[-- Attachment #2: Type: text/html, Size: 4251 bytes --]

Re: [9fans] Plan 9 on Blue Gene

[Philippe Anel]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

That's almost what they do with KSE in FreeBSD (or Scheduler Activation in NetBSD) right ?

Phil;



  I've been writing a lot of Erlang code lately, and I keep thinking about, but not having too much time to do much about, wanting to have a runtime for the libthread "threads" that could auto-schedule them to libthread "procs", in much the same way Haskell "sparks" may end up real threads, or Erlang processes, might run in parallel.


  Dave



    ----- Original Message ----- From: "ron minnich" <rminnich@gmail.com>
    To: "Fans of the OS Plan 9 from Bell Labs" <9fans@9fans.net>
    Sent: Thursday, July 31, 2008 4:23 AM
    Subject: Re: [9fans] Plan 9 on Blue Gene




      On Wed, Jul 30, 2008 at 6:48 PM, David Leimbach <leimy2k@gmail.com> wrote:


        Does Plan 9 Port help?  I mean, libthread on Plan 9 Port alone could be
        worth a ton to me in some situations.
        Concurrent programming for the win?


      probably not for this community. When we had plan9port in xcpu we got
      nothing but complaints. This in spite of the fact that some things are
      impossible to scale with 5000 posix threads, and easy to scale with
      5000 plan 9 style threads.

      ron








[-- Attachment #2: Type: text/html, Size: 2772 bytes --]

Re: [9fans] Plan 9 on Blue Gene

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 2147 bytes --]

On Thu, Jul 31, 2008 at 5:53 AM, Philippe Anel <xigh@bouyapop.org> wrote:

> Can you tell us why some things are impossible to scale with 5000 posix
> threads (and easy to scale with 5000 plan 9 style threads) ?
> Is this specific to posix or linux ? Or maybe you will write a paper on
> this ?
>
> Phil;


I think it's because plan 9 style threads are more like coroutines, and
don't initiate a new kernel-schedulable process.

If you use libthread from plan 9 port on a unix deployed system the
threadcreate function doesn't make a pthread.  proccreate, on the other
hand, does. (on the platforms I've used).

As a result you get no chance at parallelism with threadcreate, which uses
more compute resources to schedule, and proccreate "might" cause two threads
to run in parallel, if the code is arranged such that the two libthread
processes can run independently or at least overlapping.

This duality is one of the reasons I find libthread interesting in general,
even outside of Plan 9.

I've been writing a lot of Erlang code lately, and I keep thinking about,
but not having too much time to do much about, wanting to have a runtime for
the libthread "threads" that could auto-schedule them to libthread "procs",
in much the same way Haskell "sparks" may end up real threads, or Erlang
processes, might run in parallel.

Dave


>
>
> ----- Original Message ----- From: "ron minnich" <rminnich@gmail.com>
> To: "Fans of the OS Plan 9 from Bell Labs" <9fans@9fans.net>
> Sent: Thursday, July 31, 2008 4:23 AM
> Subject: Re: [9fans] Plan 9 on Blue Gene
>
>
>
>  On Wed, Jul 30, 2008 at 6:48 PM, David Leimbach <leimy2k@gmail.com>
>> wrote:
>>
>>  Does Plan 9 Port help?  I mean, libthread on Plan 9 Port alone could be
>>> worth a ton to me in some situations.
>>> Concurrent programming for the win?
>>>
>>
>> probably not for this community. When we had plan9port in xcpu we got
>> nothing but complaints. This in spite of the fact that some things are
>> impossible to scale with 5000 posix threads, and easy to scale with
>> 5000 plan 9 style threads.
>>
>> ron
>>
>>
>>
>
>

[-- Attachment #2: Type: text/html, Size: 3241 bytes --]

Re: [9fans] Plan 9 on Blue Gene

[gdiaz]

hello

thanks for the clarifications Eric and Ron ☺,

btw, if you're planning to go to Greece to the 3rd. iwp9, i would love to see a real
sheet of these ones:

http://www.atkielski.com/PDF/data/fortran.pdf

☺

slds.

gabi


> On Wed, Jul 30, 2008 at 10:25 AM,  <gdiaz@9grid.es> wrote:
>>
>> i'm asking from a technical point of view, i suppose dealing with the current users and customers is the real issue, right?
>>
>
> and tens of millions of lines of fortran that no one understands anymore....
>
> Its not that we aren't promoting other paradigms, its just we also
> need to be able to support existing code bases.
>
>         -eric

Re: [9fans] Plan 9 on Blue Gene

[gdiaz]

hello

thanks for the clarifications Eric and Ron ☺,

btw, if you're planning to go to Greece to the 3rd. iwp9, i would love to see a real
sheet of these ones:

http://www.atkielski.com/PDF/data/fortran.pdf

☺

slds.

gabi


> On Wed, Jul 30, 2008 at 10:25 AM,  <gdiaz@9grid.es> wrote:
>>
>> i'm asking from a technical point of view, i suppose dealing with the current users and customers is the real issue, right?
>>
>
> and tens of millions of lines of fortran that no one understands anymore....
>
> Its not that we aren't promoting other paradigms, its just we also
> need to be able to support existing code bases.
>
>         -eric

Re: [9fans] Plan 9 on Blue Gene

[Philippe Anel]

Can you tell us why some things are impossible to scale with 5000 posix
threads (and easy to scale with 5000 plan 9 style threads) ?
Is this specific to posix or linux ? Or maybe you will write a paper on this
?

Phil;

----- Original Message -----
From: "ron minnich" <rminnich@gmail.com>
To: "Fans of the OS Plan 9 from Bell Labs" <9fans@9fans.net>
Sent: Thursday, July 31, 2008 4:23 AM
Subject: Re: [9fans] Plan 9 on Blue Gene


> On Wed, Jul 30, 2008 at 6:48 PM, David Leimbach <leimy2k@gmail.com> wrote:
>
>> Does Plan 9 Port help?  I mean, libthread on Plan 9 Port alone could be
>> worth a ton to me in some situations.
>> Concurrent programming for the win?
>
> probably not for this community. When we had plan9port in xcpu we got
> nothing but complaints. This in spite of the fact that some things are
> impossible to scale with 5000 posix threads, and easy to scale with
> 5000 plan 9 style threads.
>
> ron
>
>

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Wed, Jul 30, 2008 at 6:48 PM, David Leimbach <leimy2k@gmail.com> wrote:

> Does Plan 9 Port help?  I mean, libthread on Plan 9 Port alone could be
> worth a ton to me in some situations.
> Concurrent programming for the win?

probably not for this community. When we had plan9port in xcpu we got
nothing but complaints. This in spite of the fact that some things are
impossible to scale with 5000 posix threads, and easy to scale with
5000 plan 9 style threads.

ron

Re: [9fans] Plan 9 on Blue Gene

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Wed, Jul 30, 2008 at 5:02 PM, ron minnich <rminnich@gmail.com> wrote:

> On Wed, Jul 30, 2008 at 4:36 PM, David Leimbach <leimy2k@gmail.com> wrote:
>
> > So is there any traction to use the new platform, or is it mostly just
> > people running their familiar apps and writing new apps for their
> familiar
> > programming environment?
>
> There are always users who are adventurous. I'm counting on it. First
> we have to show "no penalty". Then, I hope to be able to say: "hey,
> here's a nifty feature" and get them hooked.
>
> I don't think that second step will be hard. The work is in the first
> step. The great thing about BG/L and /P is that they got people out of
> the mindset that "everything is a Linux". Once you cross that Rubicon
> life gets much easier.
>

Does Plan 9 Port help?  I mean, libthread on Plan 9 Port alone could be
worth a ton to me in some situations.

Concurrent programming for the win?


>
>
> ron
>
>

[-- Attachment #2: Type: text/html, Size: 1602 bytes --]

Re: [9fans] Plan 9 on Blue Gene

[erik quanstrom]

> the mindset that "everything is a Linux". Once you cross that Rubicon
> life gets much easier.

only if it's the rubicon and not the styx.

☺

- erik

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Wed, Jul 30, 2008 at 4:36 PM, David Leimbach <leimy2k@gmail.com> wrote:

> So is there any traction to use the new platform, or is it mostly just
> people running their familiar apps and writing new apps for their familiar
> programming environment?

There are always users who are adventurous. I'm counting on it. First
we have to show "no penalty". Then, I hope to be able to say: "hey,
here's a nifty feature" and get them hooked.

I don't think that second step will be hard. The work is in the first
step. The great thing about BG/L and /P is that they got people out of
the mindset that "everything is a Linux". Once you cross that Rubicon
life gets much easier.


ron

Re: [9fans] Plan 9 on Blue Gene

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

On Wed, Jul 30, 2008 at 8:36 AM, Eric Van Hensbergen <ericvh@gmail.com>wrote:

> On Wed, Jul 30, 2008 at 10:25 AM,  <gdiaz@9grid.es> wrote:
> >
> > i'm asking from a technical point of view, i suppose dealing with the
> current users and customers is the real issue, right?
> >
>
> and tens of millions of lines of fortran that no one understands
> anymore....
>
> Its not that we aren't promoting other paradigms, its just we also
> need to be able to support existing code bases.
>
>        -eric
>

So is there any traction to use the new platform, or is it mostly just
people running their familiar apps and writing new apps for their familiar
programming environment?

Dave

[-- Attachment #2: Type: text/html, Size: 1155 bytes --]

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Wed, Jul 30, 2008 at 11:21 AM, erik quanstrom <quanstro@coraid.com> wrote:
>> Obviously Plan 9's compiler
>> isn't optimal.. but what really are the requirements people
>
> really?  that depends on your definition of optimal.
> by my definition which heavily rates speed of compliation
> and correctness, it's sure closer than the competition.

for people whose programs run in time units measured in 1/4 years,
compilation speed as an issue is in the noise.

ron

Re: [9fans] Plan 9 on Blue Gene

[erik quanstrom]

> Obviously Plan 9's compiler
> isn't optimal.. but what really are the requirements people

really?  that depends on your definition of optimal.
by my definition which heavily rates speed of compliation
and correctness, it's sure closer than the competition.

- erik

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Wed, Jul 30, 2008 at 11:03 AM, don bailey <don.bailey@gmail.com> wrote:
>> Can you elaborate here? What tricks can the IBM compilers use
>> that the Plan 9 ones can't? Are we talking optimization?

yes. Quite impressive optimization. Which results in very high
measured performance. At least when I've looked at it. The XLF/XLC
compilers represent a few decades of very good work from IBM.

> No, really, that's not troll bait. I'm actually interested in
> understanding the project's basis for discriminating against
> specific compiler capability. Obviously Plan 9's compiler
> isn't optimal.. but what really are the requirements people
> want?
>

it's not taken as such.

The simplest requirement is "XLC version x.y.z". Why? Because these
programs are beasts. Once they are working people are not inclined to
change the compiler and runtime, as that is a frequent source of
breakage. Sometimes you really do want the compiler bug.

That said, if you look at the code XLF/XLC emit, it's pretty
impressive. I suggest you grab the released papers on them. There's an
ugly effect that can occur on big parallel machines. Very small
performance deltas on small numbers of nodes can result in  huge
deltas on the whole machine. So the kinds of things you don't care
about on, e.g., your desktop, become utterly crucial on lots of nodes.

ron

Re: [9fans] Plan 9 on Blue Gene

[Eric Van Hensbergen]

On Wed, Jul 30, 2008 at 1:03 PM, don bailey <don.bailey@gmail.com> wrote:
>> Can you elaborate here? What tricks can the IBM compilers use
>> that the Plan 9 ones can't? Are we talking optimization?
>
> No, really, that's not troll bait. I'm actually interested in
> understanding the project's basis for discriminating against
> specific compiler capability. Obviously Plan 9's compiler
> isn't optimal.. but what really are the requirements people
> want?
>

People want to run the same binaries they are currently running -
those are built with XLC and XLF.

            -eric

Re: [9fans] Plan 9 on Blue Gene

[don bailey]

> Can you elaborate here? What tricks can the IBM compilers use
> that the Plan 9 ones can't? Are we talking optimization?
 >

No, really, that's not troll bait. I'm actually interested in
understanding the project's basis for discriminating against
specific compiler capability. Obviously Plan 9's compiler
isn't optimal.. but what really are the requirements people
want?

D

Re: [9fans] Plan 9 on Blue Gene

[Steven D. Vormwald]

ron minnich wrote:
> On Wed, Jul 30, 2008 at 9:10 AM, Steven D. Vormwald <sdvormwa@mtu.edu> wrote:
>
>> Correct me if I'm wrong here, but don't these require extensive run-time
>> support, in addition to compiler support?  Will the run-time libraries also
>> be linux libraries running under a compatibility layer?
>
> binaries on the bg/l machine are static.
>
> The compatibilty is needed in two places: system call layer, and
> allowing CNK binaries direct access to network hardware.
>
> ron
>

Thanks for all your responses.  That clears things up quite a bit.

Steven Vormwald

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Wed, Jul 30, 2008 at 10:39 AM, Benjamin Huntsman
<BHuntsman@mail2.cu-portland.edu> wrote:

> So, does that mean that you guys have a version of XLC that can produce Plan 9 binaries, or are you using some other method to "convert" it's output?
>

I convert the ELF from the toolchain to a.out. It's very
straightforward as the toolchain produces an elf with 2 segments :-)

ron

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Wed, Jul 30, 2008 at 9:10 AM, Steven D. Vormwald <sdvormwa@mtu.edu> wrote:

> Correct me if I'm wrong here, but don't these require extensive run-time
> support, in addition to compiler support?  Will the run-time libraries also
> be linux libraries running under a compatibility layer?

binaries on the bg/l machine are static.

The compatibilty is needed in two places: system call layer, and
allowing CNK binaries direct access to network hardware.

ron

Re: [9fans] Plan 9 on Blue Gene

[Benjamin Huntsman]

[-- Attachment #1: Type: text/plain, Size: 316 bytes --]

>What is XLC and where can I find more information on the standard?

XLC is IBM's POWER/PowerPC compiler.  It produces great code, but is expensive.

So, does that mean that you guys have a version of XLC that can produce Plan 9 binaries, or are you using some other method to "convert" it's output?

Thanks!

[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 2944 bytes --]

Re: [9fans] Plan 9 on Blue Gene

[don bailey]

> 1. rewrite apps in plan 9 c. The Plan 9 C compiler is fine for what we
> do on Plan 9. For scientific apps, it's not that great a compiler. The
> IBM compilers know all the tricks. The effort to get Plan 9 C up to
> the standards of XLC is mind-boggling. And XLF? We're not going to
> write a Fortran compiler from scratch.

Can you elaborate here? What tricks can the IBM compilers use
that the Plan 9 ones can't? Are we talking optimization? What
is XLC and where can I find more information on the standard?

D

Re: [9fans] Plan 9 on Blue Gene

[Joel C. Salomon]

On Wed, Jul 30, 2008 at 11:41 AM, ron minnich <rminnich@gmail.com> wrote:
> Build on Linux, run on Plan 9.

So there's not so much a need for gcc & al. to run on Plan 9, as for
them to target it.

(Then we can put the gcc-p9 executables on a handy Linux box
(linuxemu?), and write shell-script wrappers like folks did for
vnc+Firefox.)

--Joel

Re: [9fans] Plan 9 on Blue Gene

[Steven D. Vormwald]

ron minnich wrote:
> On Wed, Jul 30, 2008 at 8:21 AM, Steven D. Vormwald <sdvormwa@mtu.edu> wrote:
>
>> So would developers on this platform be encouraged to use languages and
>> features currently in plan 9 for HPC development, or would they target
>> existing HPC languages and features, which would be added to plan 9, either
>> via native ports or some kind of compatibility layer?
>
> The latter. Build on Linux, run on Plan 9. For most users. Some users
> have the time and willingness to try something new like Plan 9, most
> have day jobs and just want to run their code. So we make it as easy
> for them as we can.
>

Since I still do most of my work on Linux, I can't really argue with
that...  ;)

>> I noticed that a
>> limited version of MPI was mentioned in one of the papers on IBM's website,
>> but what about other systems, such as Co-Array fortran or UPC?
>>
>
> Cross-compile.
>
> ron
>

Correct me if I'm wrong here, but don't these require extensive run-time
support, in addition to compiler support?  Will the run-time libraries
also be linux libraries running under a compatibility layer?

Steven Vormwald

Re: [9fans] Plan 9 on Blue Gene

[Jack Johnson]

On Wed, Jul 30, 2008 at 7:10 AM, ron minnich <rminnich@gmail.com> wrote:
> In the HPC world, there is lots of conservatism. There is an editor at
> LANL, named Fred, written in Fortran, that has been in use for longer
> than most of you have been alive. Until very recently, it was a
> required part of any HPC system.

Any guesses as to just how old Fred is?  Or better yet, when is Fred's birthday?

It seems like there should be a Ratfor to C translator in Plan 9, if
only for nostalgia.

-Jack

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Wed, Jul 30, 2008 at 8:21 AM, Steven D. Vormwald <sdvormwa@mtu.edu> wrote:

> So would developers on this platform be encouraged to use languages and
> features currently in plan 9 for HPC development, or would they target
> existing HPC languages and features, which would be added to plan 9, either
> via native ports or some kind of compatibility layer?

The latter. Build on Linux, run on Plan 9. For most users. Some users
have the time and willingness to try something new like Plan 9, most
have day jobs and just want to run their code. So we make it as easy
for them as we can.

> I noticed that a
> limited version of MPI was mentioned in one of the papers on IBM's website,
> but what about other systems, such as Co-Array fortran or UPC?
>

Cross-compile.

ron

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

On Wed, Jul 30, 2008 at 8:25 AM,  <gdiaz@9grid.es> wrote:
> Hello,
>
> Just a dumb question, as i'm totally out of this business, it is easier to write an emulator than translate the applications to plan9 c ? (for example) or to write (or port) the C++ and Fortran compilers and related tools?

yes. I looked at the app mess for a while.

1. rewrite apps in plan 9 c. The Plan 9 C compiler is fine for what we
do on Plan 9. For scientific apps, it's not that great a compiler. The
IBM compilers know all the tricks. The effort to get Plan 9 C up to
the standards of XLC is mind-boggling. And XLF? We're not going to
write a Fortran compiler from scratch.
2. Port the compilers. Why, exactly? None of the users would be
willing to use a Plan 9 desktop.

>
> i'm asking from a technical point of view, i suppose dealing with the current users and customers is the real issue, right?

partly.

What we are really doing is using the Plan 9 os on blue gene as a
target from a cross-compiler, running on Linux. The tack I'm taking
(for now) is a system call compatibility interface in the kernel.
Recall how apps set themselves up to be debugged: write hang to ctl.
In this case, they write 'CNK' to ctl. After the exec, the system
calls are handled differently: have a Linux/gcc parameter interface,
and do different operations (some new), and will allow direct access
to the network interfaces.

We're helped by the fact that Compute Node Kernel (the lightweight
kernel for bluegene) doesn't support fork.

ron

Re: [9fans] Plan 9 on Blue Gene

[Eric Van Hensbergen]

On Wed, Jul 30, 2008 at 10:21 AM, Steven D. Vormwald <sdvormwa@mtu.edu> wrote:
>
> So would developers on this platform be encouraged to use languages and
> features currently in plan 9 for HPC development,
>

It is unlikely that existing features in Plan 9 scale applications (or
system services) to 256,000 cores.  The project isn't just to use Plan
9, but to explore distributed system concepts which can be used to
reach such scale.  This will involve new system abstractions, new
programing abstractions, new run times, and potentially new
languages....  What ends up other the other end of the pipe may end up
looking very different from Plan 9.

            -eric

Re: [9fans] Plan 9 on Blue Gene

[Eric Van Hensbergen]

On Wed, Jul 30, 2008 at 10:25 AM,  <gdiaz@9grid.es> wrote:
>
> i'm asking from a technical point of view, i suppose dealing with the current users and customers is the real issue, right?
>

and tens of millions of lines of fortran that no one understands anymore....

Its not that we aren't promoting other paradigms, its just we also
need to be able to support existing code bases.

        -eric

Re: [9fans] Plan 9 on Blue Gene

[gdiaz]

Hello,

Just a dumb question, as i'm totally out of this business, it is easier to write an emulator than translate the applications to plan9 c ? (for example) or to write (or port) the C++ and Fortran compilers and related tools?

i'm asking from a technical point of view, i suppose dealing with the current users and customers is the real issue, right?

thanks

gabi

> In the HPC world, there is lots of conservatism. There is an editor at
> LANL, named Fred, written in Fortran, that has been in use for longer
> than most of you have been alive. Until very recently, it was a
> required part of any HPC system.
>
> So, we're doing a binary compatibility module so we can run code
> compiled with the hot IBM compilers like XLC and XLF.
>
> ron

Re: [9fans] Plan 9 on Blue Gene

[Steven D. Vormwald]

ron minnich wrote:
> In the HPC world, there is lots of conservatism. There is an editor at
> LANL, named Fred, written in Fortran, that has been in use for longer
> than most of you have been alive. Until very recently, it was a
> required part of any HPC system.
>
> So, we're doing a binary compatibility module so we can run code
> compiled with the hot IBM compilers like XLC and XLF.
>
> ron
>

So would developers on this platform be encouraged to use languages and
features currently in plan 9 for HPC development, or would they target
existing HPC languages and features, which would be added to plan 9,
either via native ports or some kind of compatibility layer?  I noticed
that a limited version of MPI was mentioned in one of the papers on
IBM's website, but what about other systems, such as Co-Array fortran or
UPC?

Steven Vormwald

Re: [9fans] Plan 9 on Blue Gene

[ron minnich]

In the HPC world, there is lots of conservatism. There is an editor at
LANL, named Fred, written in Fortran, that has been in use for longer
than most of you have been alive. Until very recently, it was a
required part of any HPC system.

So, we're doing a binary compatibility module so we can run code
compiled with the hot IBM compilers like XLC and XLF.

ron

Re: [9fans] Plan 9 on Blue Gene

[Eric Van Hensbergen]

On Wed, Jul 30, 2008 at 7:32 AM, Roman V. Shaposhnik <rvs@sun.com> wrote:
>
> Do you have any bits and pieces of the software ecosystem not
> readily available on Plan9 (dreadful things like a C++ compiler)
> covered by these funds or is your intention to use available
> Plan9 userland as-is?
>

There is effort ongoing to support certain legacy applications -- but
at the moment its not something any of you would find useful -- and at
best it'll be more akin to linuxemu than a port of a C++ compiler.

        -eric

Re: [9fans] Plan 9 on Blue Gene

[Roman V. Shaposhnik]

On Sat, 2008-07-26 at 13:47 -0500, Eric Van Hensbergen wrote:
> On Sat, Jul 26, 2008 at 1:37 PM, Steven Vormwald <sdvormwa@mtu.edu> wrote:
> > Is there any (public) information about how plan 9 is/was being used on Blue
> > Gene?  The only information I can find seems to be "press release"-type
> > papers that just say that it runs on Blue Gene, but not what it was used for
> > nor how it was setup and used.
> >
>
> The existing BG/L port was a prototype proof of concept.  It was not
> done in a clean-room fashion and so large portions of it are not
> releasable.  The proof of concept was done as part of a proposal to a
> DOE funded project which just got under way.  As part of that project
> which is funded for the next three years, we'll be doing a "clean"
> port to the BG/P (which will be open sourced) along with using Plan 9
> to explore large scale (tens of thousands of nodes) distributed
> systems.  High level details are available off of the IBM Research
> pages: http://domino.research.ibm.com/comm/research_projects.nsf/pages/hare.index.html
> -- more detailed information to come.

Do you have any bits and pieces of the software ecosystem not
readily available on Plan9 (dreadful things like a C++ compiler)
covered by these funds or is your intention to use available
Plan9 userland as-is?

Thanks,
Roman.

Re: [9fans] Plan 9 on Blue Gene

[Eric Van Hensbergen]

On Sat, Jul 26, 2008 at 1:37 PM, Steven Vormwald <sdvormwa@mtu.edu> wrote:
> Is there any (public) information about how plan 9 is/was being used on Blue
> Gene?  The only information I can find seems to be "press release"-type
> papers that just say that it runs on Blue Gene, but not what it was used for
> nor how it was setup and used.
>

The existing BG/L port was a prototype proof of concept.  It was not
done in a clean-room fashion and so large portions of it are not
releasable.  The proof of concept was done as part of a proposal to a
DOE funded project which just got under way.  As part of that project
which is funded for the next three years, we'll be doing a "clean"
port to the BG/P (which will be open sourced) along with using Plan 9
to explore large scale (tens of thousands of nodes) distributed
systems.  High level details are available off of the IBM Research
pages: http://domino.research.ibm.com/comm/research_projects.nsf/pages/hare.index.html
-- more detailed information to come.

                -eric

[9fans] Plan 9 on Blue Gene

[Steven Vormwald]

Is there any (public) information about how plan 9 is/was being used on
Blue Gene?  The only information I can find seems to be "press
release"-type papers that just say that it runs on Blue Gene, but not
what it was used for nor how it was setup and used.

Steven Vormwald