[9fans] MIPS LSB compiler

[9fans] MIPS LSB compiler

[lucio]

"Go" has added a cat amongst the pigeons :-)

I'm a language aficionado and could not resist finding out more, but
what I found turned out to be relevant to me in additional ways: no
one has pointed out that the go toolchain is based on kenCC and
produces Linux-elf executables.

Given the addition of this toolchain, one wonders how far we are from
being able to port all the P9 compilers to Linux and consequently to
all Posix platforms.  My beef is that I have a wide choice of cross-
and native toolchains with which to port Plan 9 to a MIPS platform
(LSB), but I really wish I could settle on something I am much more
comfortable and familiar with.

Considering how readily p9p ported to the MIPS platform (the
outstanding issue that stops it from being submitted for inclusion in
the release is my lack of understanding of the MIPS-64/MIPS-32
details, regarding the unused getcallerpc() - sigh!) it seems to me
that adding MIPS support to go should not be hard.

My grasp of Linux and ELF, as well as the P9 toolchain, is too flimsy
to embark on this alone, but if anybody wants me to complete any task
for which the documentation isn't too obscure, I'd love to help.  My
hope is that this will help me understand the underlying issues better
and help me progress with my own porting project.

Porting go both to Plan 9 and to Linux-MIPSEL looms high on my list of
immediate interests, feel free to contact me offline if you believe I
can assist with these.

++L

Re: [9fans] MIPS LSB compiler

[Roman Shaposhnik]

On Thu, Nov 12, 2009 at 8:34 PM,  <lucio@proxima.alt.za> wrote:
> "Go" has added a cat amongst the pigeons :-)
>
> I'm a language aficionado

Makes two of us and I wouldn't mind comparing notes (in fact,
it would be quite helpful if all of us here at 9fans did). I wish I had
more time to devote to it, but so far the following things seem to
be really worth looking into:
    * Full freaking closures, which are not all that outlandish for a
garbage-collected
      language, but are pretty cool from the implementation standpoint
    * A ducktyping of sorts with interfaces and such. On the surface
it just saves
      you a bunch of "extends XXX", but it actually seems to bridge
the gap between
      dynamically typed world and a statically typed one to an extent
that makes me
      rethink whether static typed languages are as devoid of fun as a
Principia Mathematica is.

And of course, the juiciest parts are in the implementation!

> Given the addition of this toolchain, one wonders how far we are from
> being able to port all the P9 compilers to Linux and consequently to
> all Posix platforms.  My beef is that I have a wide choice of cross-
> and native toolchains with which to port Plan 9 to a MIPS platform
> (LSB), but I really wish I could settle on something I am much more
> comfortable and familiar with.

I guess I'm not following your line of thought here.

Thanks,
Roman.

Re: [9fans] MIPS LSB compiler

[John Barham]

On Thu, Nov 12, 2009 at 9:12 PM, Roman Shaposhnik <roman@shaposhnik.org> wrote:
> On Thu, Nov 12, 2009 at 8:34 PM,  <lucio@proxima.alt.za> wrote:
>> "Go" has added a cat amongst the pigeons :-)
> ...
>    * A ducktyping of sorts with interfaces and such. On the surface
> it just saves
>      you a bunch of "extends XXX", but it actually seems to bridge
> the gap between
>      dynamically typed world and a statically typed one to an extent

As a long-time Python programmer, Go's interfaces are what I find most
immediately appealing as they give you the speed and safety of a
compiler w/ the flexibility of swapping in arbitrary types provided
they satisfy the interface.  It's trivially easy compared to C++
(inheriting from a common base class--hah, try doing that w/
std::fstream) or Java (explicitly implementing an interface).  This is
such a common idiom in dynamic languages that it's second nature to
talk about "file like" objects.

And goroutines w/ channels is just icing on the cake.  Although you
take a slight hit vs C or C++ in terms of single threaded performance,
as soon as you're on multi-core, and who isn't nowadays, Go wins.

I see Go pushing C usage down even further to the
embedded/kernel/bootstrap space and rapidly displacing C++ as the
language of choice for developing new high performance
systems--already its out of the box library is much richer than what
C++ gives you, especially for network programming where standard C++
is mute.

Go overlaps w/ a lot of what I've used Python for in systems and
network programming.  Python is very nice when it works but if you
have to escape out to C for speed it quickly becomes a drag.  SWIG
helps but machine generated code is not fun to debug.

And I can't be the first person to wonder when the Go ARM compiler
will be available on Android...

Interesting times indeed.

  John

Re: [9fans] MIPS LSB compiler

[Tim Newsham]

>    * A ducktyping of sorts with interfaces and such. On the surface
> it just saves
>      you a bunch of "extends XXX", but it actually seems to bridge
> the gap between
>      dynamically typed world and a statically typed one to an extent
> that makes me
>      rethink whether static typed languages are as devoid of fun as a
> Principia Mathematica is.

The type system is more restrictive than duck typing.  Thats sort
of the point of any static type system.  But there are useful constructs
that you can express in a dynamically typed language or a language
with a more complex type system that you cannot express in go.  A
good, simple example is "map".  Go would need generics to support it.

> Roman.

Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com

Re: [9fans] MIPS LSB compiler

[Iruata Souza]

On Fri, Nov 13, 2009 at 3:48 PM, Tim Newsham <newsham@lava.net> wrote:
>>   * A ducktyping of sorts with interfaces and such. On the surface
>> it just saves
>>     you a bunch of "extends XXX", but it actually seems to bridge
>> the gap between
>>     dynamically typed world and a statically typed one to an extent
>> that makes me
>>     rethink whether static typed languages are as devoid of fun as a
>> Principia Mathematica is.
>
> The type system is more restrictive than duck typing.  Thats sort
> of the point of any static type system.  But there are useful constructs
> that you can express in a dynamically typed language or a language
> with a more complex type system that you cannot express in go.  A
> good, simple example is "map".  Go would need generics to support it.
>


$GOOROOT/src/pkg/bytes/bytes.go:248 func ToLower(s []byte) []byte
  { return Map(unicode.ToLower, s) }

Re: [9fans] MIPS LSB compiler

[Tim Newsham]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 735 bytes --]

>> with a more complex type system that you cannot express in go.  A
>> good, simple example is "map".  Go would need generics to support it.
>
> $GOOROOT/src/pkg/bytes/bytes.go:248 func ToLower(s []byte) []byte
>  { return Map(unicode.ToLower, s) }

I should have been more clear.  I mean a generic map of over
a container of an arbitrary type.  You can definitely define
maps over specific types.  The bytes.Map function maps
a function from int to int over an array (or slice?  I'm
not yet that familiar with go) of bytes.  If you wanted a
function that maps a function from int to float over an array
of ints you would need to write another function.

Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com

Re: [9fans] MIPS LSB compiler

[lucio]

>> Given the addition of this toolchain, one wonders how far we are from
>> being able to port all the P9 compilers to Linux and consequently to
>> all Posix platforms.  My beef is that I have a wide choice of cross-
>> and native toolchains with which to port Plan 9 to a MIPS platform
>> (LSB), but I really wish I could settle on something I am much more
>> comfortable and familiar with.
>
> I guess I'm not following your line of thought here.

There are two almost orthogonal issues here: "go" itself and the
toolchain that it relies on.  Whereas I do care about "go" and would
dearly like to assimilate it into my toolkit, my more immediate
interest lies with porting native Plan 9 "C" facilities beyond the
Plan 9 boundaries, specifically into the various environments served
by the "ELF" object format.  Worse, my wish is for the Plan 9 kernel
to be able to cope with ELF executables so that Plan 9 can readily (*)
join the platforms that can be used to cross-develop software.

Now, the "go" toolchain is very close to the Plan 9 native toolchain,
its most obvious difference lies in its default object format.  Given
the desirability of being able to generate ELF executables from within
Plan 9 (the MIPS toolchain already does, but I had issues with that
before now and the "go" toolchain diverges from that model somewhat, I
believe from inspecting the source somewhat superficially) my hope is
that the differences in the toolchains provided by the "go"
development and Plan 9 can mostly be eliminated.  Besides providing
Plan 9 with a more modern toolchain, it also makes it more practical
to port "go" to Plan 9 and it makes it possible to cross-develop "go"
code on a Plan 9 platform; a small amount of effort in the direction
of the MIPS architecture will also facilitate porting "go" to it.  All
developments I would be keen to contribute to, but lack the confidence
more than the expertise to undertake on my own.

++L

(*) There are mutual benefits: cross-compiling on a GNU platform to a
Plan 9 target is too hard if one needs to produce Plan 9 native
executables.  Dave Hogan managed to twist GCC 3.0 (actually, binutils
2.11, if I remember right) to produce Plan 9 native executables, but I
have failed dismally to reproduce his results on later versions of the
GNU toolchain, it's just too dense for me.  I have configured a
cross-development toolchain under NetBSD that uses the Plan 9
libraries and syscall interface; it isn't yet possible for me to test
whether the generated ELF executable actually would work, too many
other chestnuts in the fire right now.  Were this cross-development
tool to be viable, one would use it to bootstrap the most recent
stable version of GCC/G++, with results some 9fans might welcome.

Re: [9fans] MIPS LSB compiler

[Paul Lalonde]

I'd be very interested in an ELF based cross-compilation to plan9.  I
have this many-core IA part that I would desperately love to boot a
nicer OS on than we currently have (memory footprint, scheduling, vm
architecture, syscall performance, remote exposure), but the principal
application that has to run on it is in C++.

If there was a clear path, I might even be able to shake loose some
resources for it.

Paul

On 2009-11-15, at 8:52 AM, lucio@proxima.alt.za wrote:

>>> Given the addition of this toolchain, one wonders how far we are
>>> from
>>> being able to port all the P9 compilers to Linux and consequently to
>>> all Posix platforms.  My beef is that I have a wide choice of cross-
>>> and native toolchains with which to port Plan 9 to a MIPS platform
>>> (LSB), but I really wish I could settle on something I am much more
>>> comfortable and familiar with.
>>
>> I guess I'm not following your line of thought here.
>
> There are two almost orthogonal issues here: "go" itself and the
> toolchain that it relies on.  Whereas I do care about "go" and would
> dearly like to assimilate it into my toolkit, my more immediate
> interest lies with porting native Plan 9 "C" facilities beyond the
> Plan 9 boundaries, specifically into the various environments served
> by the "ELF" object format.  Worse, my wish is for the Plan 9 kernel
> to be able to cope with ELF executables so that Plan 9 can readily (*)
> join the platforms that can be used to cross-develop software.
>
> Now, the "go" toolchain is very close to the Plan 9 native toolchain,
> its most obvious difference lies in its default object format.  Given
> the desirability of being able to generate ELF executables from within
> Plan 9 (the MIPS toolchain already does, but I had issues with that
> before now and the "go" toolchain diverges from that model somewhat, I
> believe from inspecting the source somewhat superficially) my hope is
> that the differences in the toolchains provided by the "go"
> development and Plan 9 can mostly be eliminated.  Besides providing
> Plan 9 with a more modern toolchain, it also makes it more practical
> to port "go" to Plan 9 and it makes it possible to cross-develop "go"
> code on a Plan 9 platform; a small amount of effort in the direction
> of the MIPS architecture will also facilitate porting "go" to it.  All
> developments I would be keen to contribute to, but lack the confidence
> more than the expertise to undertake on my own.
>
> ++L
>
> (*) There are mutual benefits: cross-compiling on a GNU platform to a
> Plan 9 target is too hard if one needs to produce Plan 9 native
> executables.  Dave Hogan managed to twist GCC 3.0 (actually, binutils
> 2.11, if I remember right) to produce Plan 9 native executables, but I
> have failed dismally to reproduce his results on later versions of the
> GNU toolchain, it's just too dense for me.  I have configured a
> cross-development toolchain under NetBSD that uses the Plan 9
> libraries and syscall interface; it isn't yet possible for me to test
> whether the generated ELF executable actually would work, too many
> other chestnuts in the fire right now.  Were this cross-development
> tool to be viable, one would use it to bootstrap the most recent
> stable version of GCC/G++, with results some 9fans might welcome.
>
>

Re: [9fans] MIPS LSB compiler

[ron minnich]

On Sun, Nov 15, 2009 at 9:26 AM, Paul Lalonde <plalonde@telus.net> wrote:
> I'd be very interested in an ELF based cross-compilation to plan9.  I have
> this many-core IA part that I would desperately love to boot a nicer OS on
> than we currently have (memory footprint, scheduling, vm architecture,
> syscall performance, remote exposure), but the principal application that
> has to run on it is in C++.
>
> If there was a clear path, I might even be able to shake loose some
> resources for it.

on blue gene I use the IBM toolchain on Linux to produce static elf
binaries. Since the non-Linux kernel on Blue Gene uses the Linux
system call interface, these are in fact just Linux binaries compiled
to the Linux 2.0 system call interface (plus some wacky system call
numbers in the 1024 and up range).  I start these Linux/ELF binaries
up on BG/P on Plan 9 using a support program that switches the process
between plan 9 mode and linux mode. The support program on plan 9 uses
libmach to crack the binaries and start them up. I have an in-kernel
linux system call emulation device that lets me run those binaries. I
support a subset of the system calls -- e.g., just enough of futex to
make it go on BG/P -- but extending it is easy.

I chose the in-kernel support as opposed to existing linuxemu support
on Plan 9 for the usual HPC reason: when you're r unning the app
nothing else should run.

What's more interesting: I also build Plan 9 binaries on Linux/ELF
that then run under the Linux emulation environment on Plan 9. Why the
emulation environment? Well, the linux toolchain insists on building
in the system calls. So I have a Linux/ELF, that does a few Linux
system calls, that I drectly map to their Plan 9 equivalent
(open/read/write/close -- pretty direct). Since pretty much everything
I need to do on Plan 9 involves those 4, I've really got a native Plan
9 binary that happens to want to talk Linux system call numbers ...
This way, I can run the *same* binary on Linux and Plan 9, and get
very direct performance comparisons. And, yes, in some cases, Plan 9
is winning :-)

Anyway all that code is available.

ron

Re: [9fans] MIPS LSB compiler

[Bruce Ellis]

Good work indeed, Ron. All those bits and they all fit together!

I guess it also solves the Fortran etc. for Plan9 issue.

brucee

On 11/16/09, ron minnich <rminnich@gmail.com> wrote:
> On Sun, Nov 15, 2009 at 9:26 AM, Paul Lalonde <plalonde@telus.net> wrote:
> > I'd be very interested in an ELF based cross-compilation to plan9.  I have
> > this many-core IA part that I would desperately love to boot a nicer OS on
> > than we currently have (memory footprint, scheduling, vm architecture,
> > syscall performance, remote exposure), but the principal application that
> > has to run on it is in C++.
> >
> > If there was a clear path, I might even be able to shake loose some
> > resources for it.
>
> on blue gene I use the IBM toolchain on Linux to produce static elf
> binaries. Since the non-Linux kernel on Blue Gene uses the Linux
> system call interface, these are in fact just Linux binaries compiled
> to the Linux 2.0 system call interface (plus some wacky system call
> numbers in the 1024 and up range).  I start these Linux/ELF binaries
> up on BG/P on Plan 9 using a support program that switches the process
> between plan 9 mode and linux mode. The support program on plan 9 uses
> libmach to crack the binaries and start them up. I have an in-kernel
> linux system call emulation device that lets me run those binaries. I
> support a subset of the system calls -- e.g., just enough of futex to
> make it go on BG/P -- but extending it is easy.
>
> I chose the in-kernel support as opposed to existing linuxemu support
> on Plan 9 for the usual HPC reason: when you're r unning the app
> nothing else should run.
>
> What's more interesting: I also build Plan 9 binaries on Linux/ELF
> that then run under the Linux emulation environment on Plan 9. Why the
> emulation environment? Well, the linux toolchain insists on building
> in the system calls. So I have a Linux/ELF, that does a few Linux
> system calls, that I drectly map to their Plan 9 equivalent
> (open/read/write/close -- pretty direct). Since pretty much everything
> I need to do on Plan 9 involves those 4, I've really got a native Plan
> 9 binary that happens to want to talk Linux system call numbers ...
> This way, I can run the *same* binary on Linux and Plan 9, and get
> very direct performance comparisons. And, yes, in some cases, Plan 9
> is winning :-)
>
> Anyway all that code is available.
>
> ron
>
>

Re: [9fans] MIPS LSB compiler

[lucio]

> Anyway all that code is available.

What I want is the knowledge behind this code, it is a huge amount of
information that gets distilled into that code :-(

++L

Re: [9fans] MIPS LSB compiler

[Bruce Ellis]

yep, molto bene.

On 11/16/09, lucio@proxima.alt.za <lucio@proxima.alt.za> wrote:
> > Anyway all that code is available.
>
> What I want is the knowledge behind this code, it is a huge amount of
> information that gets distilled into that code :-(
>
> ++L
>
>
>

Re: [9fans] MIPS LSB compiler

[lucio]

> If there was a clear path, I might even be able to shake loose some
> resources for it.

I'm thoroughly muddled up, but I'm not going to give up easily.
Between linuxemu, p9p and many bits from different directions, ELF
support and the eventual implementation of GCC/G++ is definitely
possible.  But the bits of informations required are scattered in many
different people, none of whom are likely to have the time or
opportunity to complete this seemingly unimportant task.

++L

Re: [9fans] MIPS LSB compiler

[Andre Guenther]

correct me if i am wrong. but isn't the usual MIPS calling convention to
store the return pc in R31?

you may just want to look at your compiler output. mips has various
"branch and link" commands that do a branch and put the returning pc
into R31, which is the usual calling convention.

lucio@proxima.alt.za wrote:
> "Go" has added a cat amongst the pigeons :-)
>
> I'm a language aficionado and could not resist finding out more, but
> what I found turned out to be relevant to me in additional ways: no
> one has pointed out that the go toolchain is based on kenCC and
> produces Linux-elf executables.
>
> Given the addition of this toolchain, one wonders how far we are from
> being able to port all the P9 compilers to Linux and consequently to
> all Posix platforms.  My beef is that I have a wide choice of cross-
> and native toolchains with which to port Plan 9 to a MIPS platform
> (LSB), but I really wish I could settle on something I am much more
> comfortable and familiar with.
>
> Considering how readily p9p ported to the MIPS platform (the
> outstanding issue that stops it from being submitted for inclusion in
> the release is my lack of understanding of the MIPS-64/MIPS-32
> details, regarding the unused getcallerpc() - sigh!) it seems to me
> that adding MIPS support to go should not be hard.
>
> My grasp of Linux and ELF, as well as the P9 toolchain, is too flimsy
> to embark on this alone, but if anybody wants me to complete any task
> for which the documentation isn't too obscure, I'd love to help.  My
> hope is that this will help me understand the underlying issues better
> and help me progress with my own porting project.
>
> Porting go both to Plan 9 and to Linux-MIPSEL looms high on my list of
> immediate interests, feel free to contact me offline if you believe I
> can assist with these.
>
> ++L
>
>
>

Re: [9fans] MIPS LSB compiler

[lucio]

> correct me if i am wrong. but isn't the usual MIPS calling convention to
> store the return pc in R31?
>
Almost certainly.

> you may just want to look at your compiler output. mips has various
> "branch and link" commands that do a branch and put the returning pc
> into R31, which is the usual calling convention.

As I mentioned, the function isn't even used, so its exact
implementation doesn't matter.  What matters is knowing that the
implementation is accurate and my problem is that I don't even know
whether the 64-bit version should be different from the 32-bit
version, I don't understand what differentiates the two alternatives
within the toolchain: how does one choose which version should be
used.  Or, if you feel brave, show me what I should submit to Russ for
release.  I will happily provide you with a set of patches that works
perfectly happily on my gNewSense, MIPS-64 notebook.

++L