mailing list of musl libc
 help / color / Atom feed
* [musl] ppc64le and 32-bit LE userland compatibility
@ 2020-05-29 19:03 Will Springer
  2020-05-29 19:24 ` Rich Felker
                   ` (3 more replies)
  0 siblings, 4 replies; 59+ messages in thread
From: Will Springer @ 2020-05-29 19:03 UTC (permalink / raw)
  To: linuxppc-dev; +Cc: binutils, libc-dev, libc-alpha, musl, daniel, eery

Hey all, a couple of us over in #talos-workstation on freenode have been
working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
designated for this (unless you count the patchset from a decade ago [1]), so
it's pretty much uncharted territory as far as Linux is concerned. We want to
sync up with libc and the relevant kernel folks to establish the best path
forward.

The practical application that drove these early developments (as you might
expect) is x86 emulation. The box86 project [2] implements a translation layer
for ia32 library calls to native architecture ones; this way, emulation
overhead is significantly reduced by relying on native libraries where
possible (libc, libGL, etc.) instead of emulating an entire x86 userspace.
box86 is primarily targeted at ARM, but it can be adapted to other
architectures—so long as they match ia32's 32-bit, little-endian nature. Hence
the need for a ppcle userland; modern POWER brought ppc64le as a supported
configuration, but without a 32-bit equivalent there is no option for a 32/64
multilib environment, as seen with ppc/ppc64 and arm/aarch64.

Surprisingly, beyond minor patching of gcc to get crosscompile going,
bootstrapping the initial userland was not much of a problem. The work has
been done on top of the Void Linux PowerPC project [3], and much of that is
now present in its source package tree [4].

The first issue with running the userland came from the ppc32 signal handler 
forcing BE in the MSR, causing any 32LE process receiving a signal (such as a 
shell receiving SIGCHLD) to terminate with SIGILL. This was trivially patched, 
along with enabling the 32-bit vDSO on ppc64le kernels [5]. (Given that this 
behavior has been in place since 2006, I don't think anyone has been using the 
kernel in this state to run ppcle userlands.)

The next problem concerns the ABI more directly. The failure mode was `file`
surfacing EINVAL from pread64 when invoked on an ELF; pread64 was passed a
garbage value for `pos`, which didn't appear to be caused by anything in 
`file`. Initially it seemed as though the 32-bit components of the arg were
getting swapped, and we made hacky fixes to glibc and musl to put them in the
"right order"; however, we weren't sure if that was the correct approach, or
if there were knock-on effects we didn't know about. So we found the relevant
compat code path in the kernel, at arch/powerpc/kernel/sys_ppc32.c, where
there exists this comment:

> /*
>  * long long munging:
>  * The 32 bit ABI passes long longs in an odd even register pair.
>  */

It seems that the opposite is true in LE mode, and something is expecting long
longs to start on an even register. I realized this after I tried swapping hi/
lo `u32`s here and didn't see an improvement. I whipped up a patch [6] that
switches which syscalls use padding arguments depending on endianness, while
hopefully remaining tidy enough to be unobtrusive. (I took some liberties with
variable names/types so that the macro could be consistent.)

This was enough to fix up the `file` bug. I'm no seasoned kernel hacker,
though, and there is still concern over the right way to approach this,
whether it should live in the kernel or libc, etc. Frankly, I don't know the
ABI structure enough to understand why the register padding has to be
different in this case, or what lower-level component is responsible for it. 
For comparison, I had a look at the mips tree, since it's bi-endian and has a 
similar 32/64 situation. There is a macro conditional upon endianness that is 
responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__ instead 
of an if/else on the generic __LITTLE_ENDIAN__. Not sure what to make of that. 
(It also simply swaps registers for LE, unlike what I did for ppc.)

Also worth noting is the one other outstanding bug, where the time-related
syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
endian bug to me, and it doesn't affect standard syscalls (which is why if you
run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm 
all ears.

Again, I'd appreciate feedback on the approach to take here, in order to 
touch/special-case only the minimum necessary, while keeping the kernel/libc 
folks happy.

Cheers,
Will [she/her]

(p.s. there is ancillary interest in a ppcle-native kernel as well; that's a 
good deal more work and not the focus of this message at all, but it is a 
topic of interest)

[1]: https://lwn.net/Articles/408845/
[2]: https://github.com/ptitSeb/box86
[3]: https://voidlinux-ppc.org/
[4]: https://github.com/void-ppc/void-packages
[5]: https://gist.github.com/eerykitty/01707dc6bca2be32b4c5e30d15d15dcf
[6]: https://gist.github.com/Skirmisher/02891c1a8cafa0ff18b2460933ef4f3c





^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] ppc64le and 32-bit LE userland compatibility
  2020-05-29 19:03 [musl] ppc64le and 32-bit LE userland compatibility Will Springer
@ 2020-05-29 19:24 ` Rich Felker
  2020-05-30 22:56   ` Will Springer
  2020-05-30 15:37 ` [musl] " Christophe Leroy
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 59+ messages in thread
From: Rich Felker @ 2020-05-29 19:24 UTC (permalink / raw)
  To: Will Springer
  Cc: linuxppc-dev, binutils, libc-dev, libc-alpha, musl, daniel, eery

On Fri, May 29, 2020 at 07:03:48PM +0000, Will Springer wrote:
> The next problem concerns the ABI more directly. The failure mode was `file`
> surfacing EINVAL from pread64 when invoked on an ELF; pread64 was passed a
> garbage value for `pos`, which didn't appear to be caused by anything in 
> `file`. Initially it seemed as though the 32-bit components of the arg were
> getting swapped, and we made hacky fixes to glibc and musl to put them in the
> "right order"; however, we weren't sure if that was the correct approach, or
> if there were knock-on effects we didn't know about. So we found the relevant
> compat code path in the kernel, at arch/powerpc/kernel/sys_ppc32.c, where
> there exists this comment:
> 
> > /*
> >  * long long munging:
> >  * The 32 bit ABI passes long longs in an odd even register pair.
> >  */
> 
> It seems that the opposite is true in LE mode, and something is expecting long
> longs to start on an even register. I realized this after I tried swapping hi/
> lo `u32`s here and didn't see an improvement. I whipped up a patch [6] that
> switches which syscalls use padding arguments depending on endianness, while
> hopefully remaining tidy enough to be unobtrusive. (I took some liberties with
> variable names/types so that the macro could be consistent.)

The argument passing for pread/pwrite is historically a mess and
differs between archs. musl has a dedicated macro that archs can
define to override it. But it looks like it should match regardless of
BE vs LE, and musl already defines it for powerpc with the default
definition, adding a zero arg to start on an even arg-slot index,
which is an odd register (since ppc32 args start with an odd one, r3).

> [6]: https://gist.github.com/Skirmisher/02891c1a8cafa0ff18b2460933ef4f3c

I don't think this is correct, but I'm confused about where it's
getting messed up because it looks like it should already be right.

> This was enough to fix up the `file` bug. I'm no seasoned kernel hacker,
> though, and there is still concern over the right way to approach this,
> whether it should live in the kernel or libc, etc. Frankly, I don't know the
> ABI structure enough to understand why the register padding has to be
> different in this case, or what lower-level component is responsible for it.. 
> For comparison, I had a look at the mips tree, since it's bi-endian and has a 
> similar 32/64 situation. There is a macro conditional upon endianness that is 
> responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__ instead 
> of an if/else on the generic __LITTLE_ENDIAN__. Not sure what to make of that. 
> (It also simply swaps registers for LE, unlike what I did for ppc.)

Indeed the problem is probably that you need to swap registers for LE,
not remove the padding slot. Did you check what happens if you pass a
value larger than 32 bits?

If so, the right way to fix this on the kernel side would be to
construct the value as a union rather than by bitwise ops so it's
endian-agnostic:

	(union { u32 parts[2]; u64 val; }){{ arg1, arg2 }}.val

But the kernel folks might prefer endian ifdefs for some odd reason...

> Also worth noting is the one other outstanding bug, where the time-related
> syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
> endian bug to me, and it doesn't affect standard syscalls (which is why if you
> run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
> functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
> gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm 
> all ears.

Not sure about this. Worst-case, just leave it disabled until someone
finds a fix.

Rich

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-05-29 19:03 [musl] ppc64le and 32-bit LE userland compatibility Will Springer
  2020-05-29 19:24 ` Rich Felker
@ 2020-05-30 15:37 ` Christophe Leroy
  2020-05-30 22:17   ` Will Springer
  2020-05-30 19:22 ` Segher Boessenkool
  2020-06-01 21:28 ` Joseph Myers
  3 siblings, 1 reply; 59+ messages in thread
From: Christophe Leroy @ 2020-05-30 15:37 UTC (permalink / raw)
  To: Will Springer, linuxppc-dev
  Cc: libc-alpha, eery, daniel, musl, binutils, libc-dev



Le 29/05/2020 à 21:03, Will Springer a écrit :

[...]

> 
> Also worth noting is the one other outstanding bug, where the time-related
> syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
> endian bug to me, and it doesn't affect standard syscalls (which is why if you
> run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
> functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
> gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm
> all ears.
> 

There is a series at 
https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231 to 
switch powerpc to the Generic C VDSO.

Can you try and see whether it fixes your issue ?

Christophe

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-05-29 19:03 [musl] ppc64le and 32-bit LE userland compatibility Will Springer
  2020-05-29 19:24 ` Rich Felker
  2020-05-30 15:37 ` [musl] " Christophe Leroy
@ 2020-05-30 19:22 ` Segher Boessenkool
  2020-05-31  0:57   ` Will Springer
  2020-06-01 21:28 ` Joseph Myers
  3 siblings, 1 reply; 59+ messages in thread
From: Segher Boessenkool @ 2020-05-30 19:22 UTC (permalink / raw)
  To: Will Springer
  Cc: linuxppc-dev, libc-alpha, eery, daniel, musl, binutils, libc-dev

Hi!

On Fri, May 29, 2020 at 07:03:48PM +0000, Will Springer wrote:
> Hey all, a couple of us over in #talos-workstation on freenode have been
> working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> designated for this

https://www.polyomino.org.uk/publications/2011/Power-Arch-32-bit-ABI-supp-1.0-Embedded.pdf

> (unless you count the patchset from a decade ago [1]), so

The original sysv PowerPC supplement
http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
supports LE as well, and most powerpcle ports use that.  But, the
big-endian Linux ABI differs in quite a few places, and it of course
makes a lot better sense if powerpcle-linux follows that.

> it's pretty much uncharted territory as far as Linux is concerned. We want to
> sync up with libc and the relevant kernel folks to establish the best path
> forward.
> 
> The practical application that drove these early developments (as you might
> expect) is x86 emulation. The box86 project [2] implements a translation layer
> for ia32 library calls to native architecture ones; this way, emulation
> overhead is significantly reduced by relying on native libraries where
> possible (libc, libGL, etc.) instead of emulating an entire x86 userspace.
> box86 is primarily targeted at ARM, but it can be adapted to other
> architectures—so long as they match ia32's 32-bit, little-endian nature. Hence
> the need for a ppcle userland; modern POWER brought ppc64le as a supported
> configuration, but without a 32-bit equivalent there is no option for a 32/64
> multilib environment, as seen with ppc/ppc64 and arm/aarch64.
> 
> Surprisingly, beyond minor patching of gcc to get crosscompile going,

What patches did you need?  I regularly build >30 cross compilers (on
both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
in the past those worked fine as well).  I also cross-built
powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
from various x86).

> bootstrapping the initial userland was not much of a problem. The work has
> been done on top of the Void Linux PowerPC project [3], and much of that is
> now present in its source package tree [4].
> 
> The first issue with running the userland came from the ppc32 signal handler 
> forcing BE in the MSR, causing any 32LE process receiving a signal (such as a 
> shell receiving SIGCHLD) to terminate with SIGILL. This was trivially patched, 

Heh :-)

> along with enabling the 32-bit vDSO on ppc64le kernels [5]. (Given that this 
> behavior has been in place since 2006, I don't think anyone has been using the 
> kernel in this state to run ppcle userlands.)

Almost no project that used 32-bit PowerPC in LE mode has sent patches
to the upstreams.

> The next problem concerns the ABI more directly. The failure mode was `file`
> surfacing EINVAL from pread64 when invoked on an ELF; pread64 was passed a
> garbage value for `pos`, which didn't appear to be caused by anything in 
> `file`. Initially it seemed as though the 32-bit components of the arg were
> getting swapped, and we made hacky fixes to glibc and musl to put them in the
> "right order"; however, we weren't sure if that was the correct approach, or
> if there were knock-on effects we didn't know about. So we found the relevant
> compat code path in the kernel, at arch/powerpc/kernel/sys_ppc32.c, where
> there exists this comment:
> 
> > /*
> >  * long long munging:
> >  * The 32 bit ABI passes long longs in an odd even register pair.
> >  */
> 
> It seems that the opposite is true in LE mode, and something is expecting long
> longs to start on an even register. I realized this after I tried swapping hi/
> lo `u32`s here and didn't see an improvement. I whipped up a patch [6] that
> switches which syscalls use padding arguments depending on endianness, while
> hopefully remaining tidy enough to be unobtrusive. (I took some liberties with
> variable names/types so that the macro could be consistent.)

The ABI says long longs are passed in the same order in registers as it
would be in memory; so the high part and the low part are swapped between
BE and LE.  Which registers make up a pair is exactly the same between
the two.  (You can verify this with an existing powerpcle-* compiler, too;
I did, and we implement it correctly as far as I can see).

> This was enough to fix up the `file` bug. I'm no seasoned kernel hacker,
> though, and there is still concern over the right way to approach this,
> whether it should live in the kernel or libc, etc. Frankly, I don't know the
> ABI structure enough to understand why the register padding has to be
> different in this case, or what lower-level component is responsible for it. 
> For comparison, I had a look at the mips tree, since it's bi-endian and has a 
> similar 32/64 situation. There is a macro conditional upon endianness that is 
> responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__ instead 
> of an if/else on the generic __LITTLE_ENDIAN__. Not sure what to make of that. 
> (It also simply swaps registers for LE, unlike what I did for ppc.)

But you should :-)

> Also worth noting is the one other outstanding bug, where the time-related
> syscalls in the 32-bit vDSO seem to return garbage. It doesn't look like an
> endian bug to me, and it doesn't affect standard syscalls (which is why if you
> run `date` on musl it prints the correct time, unlike on glibc). The vDSO time
> functions are implemented in ppc asm (arch/powerpc/kernel/vdso32/
> gettimeofday.S), and I've never touched the stuff, so if anyone has a clue I'm 
> all ears.
> 
> Again, I'd appreciate feedback on the approach to take here, in order to 
> touch/special-case only the minimum necessary, while keeping the kernel/libc 
> folks happy.

A huge factor in having good GCC support for powerpcle-linux (or anything
else) is someone needs to regularly test it, and share test results with
us (via gcc-testresults@).  Hint hint hint :-)

That way we know it is in good shape, know when we are regressing it,
know there is interest in it.

gl;hf,


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-05-30 15:37 ` [musl] " Christophe Leroy
@ 2020-05-30 22:17   ` Will Springer
  2020-06-05 23:54     ` Will Springer
  0 siblings, 1 reply; 59+ messages in thread
From: Will Springer @ 2020-05-30 22:17 UTC (permalink / raw)
  To: linuxppc-dev, Christophe Leroy
  Cc: libc-alpha, eery, daniel, musl, binutils, libc-dev

On Saturday, May 30, 2020 8:37:43 AM PDT Christophe Leroy wrote:
> There is a series at
> https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231 to
> switch powerpc to the Generic C VDSO.
> 
> Can you try and see whether it fixes your issue ?
> 
> Christophe

Sure thing, I spotted that after making the initial post. Will report back 
with results.

Will [she/her]





^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] ppc64le and 32-bit LE userland compatibility
  2020-05-29 19:24 ` Rich Felker
@ 2020-05-30 22:56   ` Will Springer
  0 siblings, 0 replies; 59+ messages in thread
From: Will Springer @ 2020-05-30 22:56 UTC (permalink / raw)
  To: Rich Felker, linuxppc-dev
  Cc: binutils, libc-dev, libc-alpha, musl, daniel, eery

On Friday, May 29, 2020 12:24:27 PM PDT Rich Felker wrote:
> The argument passing for pread/pwrite is historically a mess and
> differs between archs. musl has a dedicated macro that archs can
> define to override it. But it looks like it should match regardless of
> BE vs LE, and musl already defines it for powerpc with the default
> definition, adding a zero arg to start on an even arg-slot index,
> which is an odd register (since ppc32 args start with an odd one, r3).
> 
> > [6]:
> > https://gist.github.com/Skirmisher/02891c1a8cafa0ff18b2460933ef4f3c
> I don't think this is correct, but I'm confused about where it's
> getting messed up because it looks like it should already be right.

Hmm, interesting. Will have to go back to it I guess...

> > This was enough to fix up the `file` bug. I'm no seasoned kernel
> > hacker, though, and there is still concern over the right way to
> > approach this, whether it should live in the kernel or libc, etc.
> > Frankly, I don't know the ABI structure enough to understand why the
> > register padding has to be different in this case, or what
> > lower-level component is responsible for it.. For comparison, I had a
> > look at the mips tree, since it's bi-endian and has a similar 32/64
> > situation. There is a macro conditional upon endianness that is
> > responsible for munging long longs; it uses __MIPSEB__ and __MIPSEL__
> > instead of an if/else on the generic __LITTLE_ENDIAN__. Not sure what
> > to make of that. (It also simply swaps registers for LE, unlike what
> > I did for ppc.)
> Indeed the problem is probably that you need to swap registers for LE,
> not remove the padding slot. Did you check what happens if you pass a
> value larger than 32 bits?
> 
> If so, the right way to fix this on the kernel side would be to
> construct the value as a union rather than by bitwise ops so it's
> endian-agnostic:
> 
> 	(union { u32 parts[2]; u64 val; }){{ arg1, arg2 }}.val
> 
> But the kernel folks might prefer endian ifdefs for some odd reason...

You are right, this does seem odd considering what the other archs do. 
It's quite possible I made a silly mistake, of course...

I haven't tested with values outside the 32-bit range yet; again, this is 
new territory for me, so I haven't exactly done exhaustive tests on 
everything. I'll give it a closer look.

> > Also worth noting is the one other outstanding bug, where the
> > time-related syscalls in the 32-bit vDSO seem to return garbage. It
> > doesn't look like an endian bug to me, and it doesn't affect standard
> > syscalls (which is why if you run `date` on musl it prints the
> > correct time, unlike on glibc). The vDSO time functions are
> > implemented in ppc asm (arch/powerpc/kernel/vdso32/ gettimeofday.S),
> > and I've never touched the stuff, so if anyone has a clue I'm all
> > ears.
> Not sure about this. Worst-case, just leave it disabled until someone
> finds a fix.

Apparently these asm implementations are being replaced by the generic C 
ones [1], so it may be this fixes itself on its own.

Thanks,
Will [she/her]

[1]: https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231







^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-05-30 19:22 ` Segher Boessenkool
@ 2020-05-31  0:57   ` Will Springer
  2020-05-31 20:42     ` Segher Boessenkool
  0 siblings, 1 reply; 59+ messages in thread
From: Will Springer @ 2020-05-31  0:57 UTC (permalink / raw)
  To: Segher Boessenkool, linuxppc-dev
  Cc: libc-alpha, eery, daniel, musl, binutils, libc-dev

On Saturday, May 30, 2020 12:22:12 PM PDT Segher Boessenkool wrote:
> The original sysv PowerPC supplement
> http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
> supports LE as well, and most powerpcle ports use that.  But, the
> big-endian Linux ABI differs in quite a few places, and it of course
> makes a lot better sense if powerpcle-linux follows that.

Right, I should have clarified I was talking about Linux ABIs 
specifically.

> What patches did you need?  I regularly build >30 cross compilers (on
> both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
> in the past those worked fine as well).  I also cross-built
> powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
> from various x86).

There was just an assumption that LE == powerpc64le in libgo, spotted by 
q66 (daniel@ on the CC). I just pushed the patch to [1].

> Almost no project that used 32-bit PowerPC in LE mode has sent patches
> to the upstreams.

Right, but I have heard concerns from at least one person familiar with 
the ppc kernel about breaking existing users of this arch-endianness 
combo, if any. It seems likely that none of those use upstream, though ^^;

> The ABI says long longs are passed in the same order in registers as it
> would be in memory; so the high part and the low part are swapped
> between BE and LE.  Which registers make up a pair is exactly the same
> between the two.  (You can verify this with an existing powerpcle-*
> compiler, too; I did, and we implement it correctly as far as I can
> see).

I'll give it a closer look. This is my first time poking at this sort of 
thing in depth, so excuse my unfamiliarity!

> A huge factor in having good GCC support for powerpcle-linux (or
> anything else) is someone needs to regularly test it, and share test
> results with us (via gcc-testresults@).  Hint hint hint :-)
> 
> That way we know it is in good shape, know when we are regressing it,
> know there is interest in it.

Once I have more of a bootstrapped userland than a barely-functional 
cross chroot, I'll get back to you on that :)
 
> gl;hf,
> 
> 
> Segher

Thanks,
Will [she/her]

[1]: https://github.com/Skirmisher/void-packages/blob/master/srcpkgs/gcc/patches/libgo-ppcle.patch





^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-05-31  0:57   ` Will Springer
@ 2020-05-31 20:42     ` Segher Boessenkool
  2020-05-31 22:29       ` Daniel Kolesa
  0 siblings, 1 reply; 59+ messages in thread
From: Segher Boessenkool @ 2020-05-31 20:42 UTC (permalink / raw)
  To: Will Springer
  Cc: linuxppc-dev, libc-alpha, eery, daniel, musl, binutils, libc-dev

On Sun, May 31, 2020 at 12:57:12AM +0000, Will Springer wrote:
> On Saturday, May 30, 2020 12:22:12 PM PDT Segher Boessenkool wrote:
> > The original sysv PowerPC supplement
> > http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
> > supports LE as well, and most powerpcle ports use that.  But, the
> > big-endian Linux ABI differs in quite a few places, and it of course
> > makes a lot better sense if powerpcle-linux follows that.
> 
> Right, I should have clarified I was talking about Linux ABIs 
> specifically.

That was the link you deleted.

> > What patches did you need?  I regularly build >30 cross compilers (on
> > both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
> > in the past those worked fine as well).  I also cross-built
> > powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
> > from various x86).
> 
> There was just an assumption that LE == powerpc64le in libgo, spotted by 
> q66 (daniel@ on the CC). I just pushed the patch to [1].

Please send GCC patches to gcc-patches@ ?

> > Almost no project that used 32-bit PowerPC in LE mode has sent patches
> > to the upstreams.
> 
> Right, but I have heard concerns from at least one person familiar with 
> the ppc kernel about breaking existing users of this arch-endianness 
> combo, if any. It seems likely that none of those use upstream, though ^^;

So we don't care, because we *cannot* care.

> > A huge factor in having good GCC support for powerpcle-linux (or
> > anything else) is someone needs to regularly test it, and share test
> > results with us (via gcc-testresults@).  Hint hint hint :-)
> > 
> > That way we know it is in good shape, know when we are regressing it,
> > know there is interest in it.
> 
> Once I have more of a bootstrapped userland than a barely-functional 
> cross chroot, I'll get back to you on that :)

Cool!  Looking forward to it.

Thanks,


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-05-31 20:42     ` Segher Boessenkool
@ 2020-05-31 22:29       ` Daniel Kolesa
  2020-06-02  1:36         ` Segher Boessenkool
  0 siblings, 1 reply; 59+ messages in thread
From: Daniel Kolesa @ 2020-05-31 22:29 UTC (permalink / raw)
  To: Segher Boessenkool, musl, Will Springer
  Cc: linuxppc-dev, libc-alpha, eery, Palmer Dabbelt via binutils, libc-dev

On Sun, May 31, 2020, at 22:42, Segher Boessenkool wrote:
> On Sun, May 31, 2020 at 12:57:12AM +0000, Will Springer wrote:
> > On Saturday, May 30, 2020 12:22:12 PM PDT Segher Boessenkool wrote:
> > > The original sysv PowerPC supplement
> > > http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
> > > supports LE as well, and most powerpcle ports use that.  But, the
> > > big-endian Linux ABI differs in quite a few places, and it of course
> > > makes a lot better sense if powerpcle-linux follows that.
> > 
> > Right, I should have clarified I was talking about Linux ABIs 
> > specifically.
> 
> That was the link you deleted.
> 
> > > What patches did you need?  I regularly build >30 cross compilers (on
> > > both BE and LE hosts; I haven't used 32-bit hosts for a long time, but
> > > in the past those worked fine as well).  I also cross-built
> > > powerpcle-linux-gcc quite a few times (from powerpc64le, from powerpc64,
> > > from various x86).
> > 
> > There was just an assumption that LE == powerpc64le in libgo, spotted by 
> > q66 (daniel@ on the CC). I just pushed the patch to [1].
> 
> Please send GCC patches to gcc-patches@ ?

FWIW, that patch alone is not very useful, we'd need to otherwise patch libgo to recognize a new GOARCH (as right now it's likely to just use 'ppc' which is wrong).

That said, we'll get back to you with any patches we have. One I can already think of - we will need to update the dynamic linker name so that it uses ld-musl-powerpcle.so instead of powerpc (musl needs to be updated the same way by adding the subarch variable for the 'le' prefix).

> 
> > > Almost no project that used 32-bit PowerPC in LE mode has sent patches
> > > to the upstreams.
> > 
> > Right, but I have heard concerns from at least one person familiar with 
> > the ppc kernel about breaking existing users of this arch-endianness 
> > combo, if any. It seems likely that none of those use upstream, though ^^;
> 
> So we don't care, because we *cannot* care.

Well, that's the reason this thread was opened in the first place - to call out to any potential users, and synchronize with upstreams on a single way forward that all upstreams can agree on, since this effort requires changes in various parts of the stack. We don't want to hog changes locally or otherwise do any changes that would be in conflict with upstream projects, as that would mean needlessly diverging, which only means trouble later on.

> 
> > > A huge factor in having good GCC support for powerpcle-linux (or
> > > anything else) is someone needs to regularly test it, and share test
> > > results with us (via gcc-testresults@).  Hint hint hint :-)
> > > 
> > > That way we know it is in good shape, know when we are regressing it,
> > > know there is interest in it.
> > 
> > Once I have more of a bootstrapped userland than a barely-functional 
> > cross chroot, I'll get back to you on that :)
> 
> Cool!  Looking forward to it.
> 
> Thanks,

Either way, thanks for the hints so far.

> 
> 
> Segher
>

Daniel (q66)

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-05-29 19:03 [musl] ppc64le and 32-bit LE userland compatibility Will Springer
                   ` (2 preceding siblings ...)
  2020-05-30 19:22 ` Segher Boessenkool
@ 2020-06-01 21:28 ` Joseph Myers
  2020-06-01 21:36   ` Rich Felker
                     ` (2 more replies)
  3 siblings, 3 replies; 59+ messages in thread
From: Joseph Myers @ 2020-06-01 21:28 UTC (permalink / raw)
  To: Will Springer
  Cc: linuxppc-dev, libc-alpha, eery, daniel, musl, binutils, libc-dev

On Fri, 29 May 2020, Will Springer via Binutils wrote:

> Hey all, a couple of us over in #talos-workstation on freenode have been
> working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> designated for this (unless you count the patchset from a decade ago [1]), so
> it's pretty much uncharted territory as far as Linux is concerned. We want to
> sync up with libc and the relevant kernel folks to establish the best path
> forward.

As a general comment on the glibc side of things, if this is considered 
like a new port, and it probably is, the same principles that apply to new 
ports apply here.

There's a general discussion at 
<https://sourceware.org/glibc/wiki/NewPorts>, although much of that is 
only applicable when adding new CPU architecture support.  More specific 
points include that new 32-bit ports should default to 64-bit time and 
file offsets from the start, with no support for 32-bit time or offsets 
(meaning that if you want to use this with some kind of library call 
translation, the library call translation will need to deal with 
corresponding type size conversions).  And a new port should not be added 
that uses the IBM long double format.  You can use IEEE binary128 long 
double, possibly with an ABI similar to that used on powerpc64le, or can 
use long double = double, but should not support IBM long double, and 
preferably should only have one long double format rather than using the 
glibc support for building with different options resulting in functions 
for different long double formats being called.

-- 
Joseph S. Myers
joseph@codesourcery.com

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-01 21:28 ` Joseph Myers
@ 2020-06-01 21:36   ` Rich Felker
  2020-06-01 23:26   ` Daniel Kolesa
  2020-06-02  1:42   ` Segher Boessenkool
  2 siblings, 0 replies; 59+ messages in thread
From: Rich Felker @ 2020-06-01 21:36 UTC (permalink / raw)
  To: Joseph Myers
  Cc: Will Springer, linuxppc-dev, libc-alpha, eery, daniel, musl,
	binutils, libc-dev

On Mon, Jun 01, 2020 at 09:28:25PM +0000, Joseph Myers wrote:
> On Fri, 29 May 2020, Will Springer via Binutils wrote:
> 
> > Hey all, a couple of us over in #talos-workstation on freenode have been
> > working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> > little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> > designated for this (unless you count the patchset from a decade ago [1]), so
> > it's pretty much uncharted territory as far as Linux is concerned. We want to
> > sync up with libc and the relevant kernel folks to establish the best path
> > forward.
> 
> As a general comment on the glibc side of things, if this is considered 
> like a new port, and it probably is, the same principles that apply to new 
> ports apply here.
> 
> There's a general discussion at 
> <https://sourceware.org/glibc/wiki/NewPorts>, although much of that is 
> only applicable when adding new CPU architecture support.  More specific 
> points include that new 32-bit ports should default to 64-bit time and 
> file offsets from the start, with no support for 32-bit time or offsets 
> (meaning that if you want to use this with some kind of library call 
> translation, the library call translation will need to deal with 
> corresponding type size conversions).  And a new port should not be added 
> that uses the IBM long double format.  You can use IEEE binary128 long 
> double, possibly with an ABI similar to that used on powerpc64le, or can 
> use long double = double, but should not support IBM long double, and 
> preferably should only have one long double format rather than using the 
> glibc support for building with different options resulting in functions 
> for different long double formats being called.

Thanks, these are great points, and the same applies for musl I think.
We always have 64-bit off_t anyway, but new ports should have 64-bit
time_t to begin with rather than defining _REDIR_TIME64.

It's a little bit complicated by the fact that powerpcle would be a
"subarch" for powerpc, and we don't yet have any like that where the
subarchs' time64 statuses differ, but it doesn't look like that should
be hard to do. The arch-specific alltypes.h.in already has access to
endianness knowledge. src/ldso/powerpc/dlsym_time64.S would also need
added preprocessor conditionals.

Exemption from this would be open to discussion if there are existing
non-upstream users of powerpcle musl that otherwise complies with ABI
policy except for time64, but I'm not aware of any that aren't
experimental.

Rich

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-01 21:28 ` Joseph Myers
  2020-06-01 21:36   ` Rich Felker
@ 2020-06-01 23:26   ` Daniel Kolesa
  2020-06-01 23:45     ` Joseph Myers
  2020-06-02  1:58     ` Segher Boessenkool
  2020-06-02  1:42   ` Segher Boessenkool
  2 siblings, 2 replies; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-01 23:26 UTC (permalink / raw)
  To: Joseph Myers, Will Springer
  Cc: linuxppc-dev, libc-alpha, eery, musl,
	Palmer Dabbelt via binutils, via libc-dev

On Mon, Jun 1, 2020, at 23:28, Joseph Myers wrote:
> On Fri, 29 May 2020, Will Springer via Binutils wrote:
> 
> > Hey all, a couple of us over in #talos-workstation on freenode have been
> > working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> > little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> > designated for this (unless you count the patchset from a decade ago [1]), so
> > it's pretty much uncharted territory as far as Linux is concerned. We want to
> > sync up with libc and the relevant kernel folks to establish the best path
> > forward.
> 
> As a general comment on the glibc side of things, if this is considered 
> like a new port, and it probably is, the same principles that apply to new 
> ports apply here.
> 
> There's a general discussion at 
> <https://sourceware.org/glibc/wiki/NewPorts>, although much of that is 
> only applicable when adding new CPU architecture support.  More specific 
> points include that new 32-bit ports should default to 64-bit time and 
> file offsets from the start, with no support for 32-bit time or offsets 
> (meaning that if you want to use this with some kind of library call 
> translation, the library call translation will need to deal with 
> corresponding type size conversions).  And a new port should not be added 
> that uses the IBM long double format.  You can use IEEE binary128 long 
> double, possibly with an ABI similar to that used on powerpc64le, or can 
> use long double = double, but should not support IBM long double, and 
> preferably should only have one long double format rather than using the 
> glibc support for building with different options resulting in functions 
> for different long double formats being called.

Are you sure this would be a new port? Glibc already works in this combination, as it seems to me it'd be best if it was just a variant of the existing 32-bit PowerPC port, sharing most conventions besides endianness with the BE port.

128-bit IEEE long double would not work, since that relies on VSX being present (gcc will explicitly complain if it's not). I'd be all for using 64-bit long double, though (musl already does, on all ppc ports).

While we're at long double, I'd actually be interested in transitioning the existing big endian ports in Void (64-bit and 32-bit, neither has VSX baseline requirement in my case) to using 64-bit long double, abandoning the IBM format altogether (little endian will transition to 128-bit IEEE long double once it's ready on your side, as that assumes POWER8 baseline which includes VSX).

What would be the best way for me to proceed with that? I actually experimented with this, using the old glibc compat symbols from pre-ibm128 times, and I mostly had it working, except I haven't managed to find a way to switch the default symbols to 64-bit ones, which is problematic as linking everything against nldbl_nonshared is fragile and potentially quirky (breaks dlsym, function pointer equality across libraries, etc).

There is also one more thing while we're at this. The 64-bit big endian Void port uses the ELFv2 ABI, even on glibc. This is not officially supported on glibc as far as I can tell, but it does work out of box, without any patching (things in general match little endian then, i.e. ld64.so.2 etc, but they're big endian). Is there any chance of making that support official?

> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-01 23:26   ` Daniel Kolesa
@ 2020-06-01 23:45     ` Joseph Myers
  2020-06-01 23:55       ` Joseph Myers
                         ` (2 more replies)
  2020-06-02  1:58     ` Segher Boessenkool
  1 sibling, 3 replies; 59+ messages in thread
From: Joseph Myers @ 2020-06-01 23:45 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: Will Springer, libc-alpha, eery, musl,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, 2 Jun 2020, Daniel Kolesa wrote:

> Are you sure this would be a new port? Glibc already works in this 
> combination, as it seems to me it'd be best if it was just a variant of 
> the existing 32-bit PowerPC port, sharing most conventions besides 
> endianness with the BE port.

The supported glibc ABIs are listed at 
<https://sourceware.org/glibc/wiki/ABIList>.  This would be a new ABI, 
which should have a new ABI-and-architecture-specific dynamic linker name 
(all new ports are expected to have a unique dynamic linker name for each 
ABI, to support systems using multiarch directory arrangements), new 
symbol versions and avoid legacy features such as 32-bit time or offsets 
or IBM long double.

> 128-bit IEEE long double would not work, since that relies on VSX being 
> present (gcc will explicitly complain if it's not). I'd be all for using 

The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
understanding was that the suggestion was for 32-bit userspace to run 
under powerpc64le kernels running on POWER8 or later, meaning that such a 
32-bit LE port, and any ABI designed for such a port, can assume VSX is 
available.  Or does VSX not work, at the hardware level, for 32-bit 
POWER8?  (In which case you could pick another ABI for binary128 argument 
passing and return.)

> There is also one more thing while we're at this. The 64-bit big endian 
> Void port uses the ELFv2 ABI, even on glibc. This is not officially 
> supported on glibc as far as I can tell, but it does work out of box, 
> without any patching (things in general match little endian then, i.e. 
> ld64.so.2 etc, but they're big endian). Is there any chance of making 
> that support official?

If you want to support ELFv2 for 64-bit big endian in glibc, again that 
should have a unique dynamic linker name, new symbol versions, only 
binary128 long double, etc. - avoid all the legacy aspects of the existing 
ELFv1 port rather than selectively saying that "ELFv1" itself is the only 
legacy aspect and keeping the others (when it's the others that are 
actually more problematic in glibc).

-- 
Joseph S. Myers
joseph@codesourcery.com

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-01 23:45     ` Joseph Myers
@ 2020-06-01 23:55       ` Joseph Myers
  2020-06-02  0:13         ` Daniel Kolesa
  2020-06-02  0:11       ` Daniel Kolesa
  2020-06-02  2:12       ` Segher Boessenkool
  2 siblings, 1 reply; 59+ messages in thread
From: Joseph Myers @ 2020-06-01 23:55 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Mon, 1 Jun 2020, Joseph Myers wrote:

> The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> understanding was that the suggestion was for 32-bit userspace to run 
> under powerpc64le kernels running on POWER8 or later, meaning that such a 
> 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> available.  Or does VSX not work, at the hardware level, for 32-bit 
> POWER8?  (In which case you could pick another ABI for binary128 argument 
> passing and return.)

In fact, my understanding is that the ABI for passing binary128 values in 
vector registers is perfectly implementable for processors with just VMX 
(AltiVec) and not VSX.  So if you do want to support binary128 for a new 
ABI for either 32-bit LE or 32-bit or 64-bit BE, you don't need to require 
VSX for that ABI, you just need to change any GCC requirement for VSX for 
binary128 to allow it with VMX when building for your new ABI.

-- 
Joseph S. Myers
joseph@codesourcery.com

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-01 23:45     ` Joseph Myers
  2020-06-01 23:55       ` Joseph Myers
@ 2020-06-02  0:11       ` Daniel Kolesa
  2020-06-02 13:40         ` Joseph Myers
  2020-06-02  2:12       ` Segher Boessenkool
  2 siblings, 1 reply; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-02  0:11 UTC (permalink / raw)
  To: Joseph Myers
  Cc: Will Springer, libc-alpha, eery, musl,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 2, 2020, at 01:45, Joseph Myers wrote:
> On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> 
> > Are you sure this would be a new port? Glibc already works in this 
> > combination, as it seems to me it'd be best if it was just a variant of 
> > the existing 32-bit PowerPC port, sharing most conventions besides 
> > endianness with the BE port.
> 
> The supported glibc ABIs are listed at 
> <https://sourceware.org/glibc/wiki/ABIList>.  This would be a new ABI, 
> which should have a new ABI-and-architecture-specific dynamic linker name 
> (all new ports are expected to have a unique dynamic linker name for each 
> ABI, to support systems using multiarch directory arrangements), new 
> symbol versions and avoid legacy features such as 32-bit time or offsets 
> or IBM long double.
> 
> > 128-bit IEEE long double would not work, since that relies on VSX being 
> > present (gcc will explicitly complain if it's not). I'd be all for using 
> 
> The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> understanding was that the suggestion was for 32-bit userspace to run 
> under powerpc64le kernels running on POWER8 or later, meaning that such a 
> 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> available.  Or does VSX not work, at the hardware level, for 32-bit 
> POWER8?  (In which case you could pick another ABI for binary128 argument 
> passing and return.)

POWER8 may have VSX (well, actually POWER7 and newer has VSX and can run LE, but glibc does not support this, musl potentially does), but the overall assumption here is that the resulting binaries should eventually not be limited to being just userspace under ppc64le, but should be runnable on a native kernel as well, which should not be limited to any particular baseline other than just PowerPC.

While it should in theory be possible to do IEEE ldbl128 using a different ABI, I don't really see any benefit in this - for one, the baseline hardware doesn't support on any level, it would mean further complicating the ABI, and it would require explicit support in the compiler, which currently doesn't exist. Using 64-bit long doubles sounds like a much better way out to me.

> 
> > There is also one more thing while we're at this. The 64-bit big endian 
> > Void port uses the ELFv2 ABI, even on glibc. This is not officially 
> > supported on glibc as far as I can tell, but it does work out of box, 
> > without any patching (things in general match little endian then, i.e. 
> > ld64.so.2 etc, but they're big endian). Is there any chance of making 
> > that support official?
> 
> If you want to support ELFv2 for 64-bit big endian in glibc, again that 
> should have a unique dynamic linker name, new symbol versions, only 
> binary128 long double, etc. - avoid all the legacy aspects of the existing 
> ELFv1 port rather than selectively saying that "ELFv1" itself is the only 
> legacy aspect and keeping the others (when it's the others that are 
> actually more problematic in glibc).

Again, the BE port cannot use binary128 long double, at least not with the same ABI as on POWER8, since it runs on all 64-bit PowerPC systems starting with 970 (G5, and potentially even POWER4 if built without AltiVec). Unique dynamic linker names are complicated, since as it is, glibc uses ld64.so.1 for ELFv1, and ld64.so.2 for ELFv2. (on 32-bit PowerPC, it's ld.so.1, and uses the SVR4 ABI which is not related to either the AIX/ELFv1 nor the ELFv2 ABIs) If we were to introduce new ports, what would those use? ld64.so.3 for BE/v2? ld.so.2 for LE/32-bit? I can see the reason for a new dynamic linker name though (multi-arch setups).

However, the effective difference between the ports would be rather minimal, if any, as far as I can see. As I already said, we have a whole glibc/ELFv2/BE system, with nearly all of the existing Linux userland covered by the distro, and there haven't been any issues whatsoever.

> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-01 23:55       ` Joseph Myers
@ 2020-06-02  0:13         ` Daniel Kolesa
  0 siblings, 0 replies; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-02  0:13 UTC (permalink / raw)
  To: Joseph Myers
  Cc: libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 2, 2020, at 01:55, Joseph Myers wrote:
> On Mon, 1 Jun 2020, Joseph Myers wrote:
> 
> > The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> > understanding was that the suggestion was for 32-bit userspace to run 
> > under powerpc64le kernels running on POWER8 or later, meaning that such a 
> > 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> > available.  Or does VSX not work, at the hardware level, for 32-bit 
> > POWER8?  (In which case you could pick another ABI for binary128 argument 
> > passing and return.)
> 
> In fact, my understanding is that the ABI for passing binary128 values in 
> vector registers is perfectly implementable for processors with just VMX 
> (AltiVec) and not VSX.  So if you do want to support binary128 for a new 
> ABI for either 32-bit LE or 32-bit or 64-bit BE, you don't need to require 
> VSX for that ABI, you just need to change any GCC requirement for VSX for 
> binary128 to allow it with VMX when building for your new ABI.

Which still doesn't help us even if true, since we plan to support hardware that doesn't have any kind of vector functionality in the first place (PowerPC G3/G4, and for the ELFv2 64-bit BE port, the minimum for binary packages is 970/G5 which does have AltiVec, but it is also supported to build your userland from source without this, for e.g. POWER5 machines, or e5500 SoCs)

> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-05-31 22:29       ` Daniel Kolesa
@ 2020-06-02  1:36         ` Segher Boessenkool
  0 siblings, 0 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-02  1:36 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: musl, Will Springer, linuxppc-dev, libc-alpha, eery,
	Palmer Dabbelt via binutils, libc-dev

On Mon, Jun 01, 2020 at 12:29:56AM +0200, Daniel Kolesa wrote:
> On Sun, May 31, 2020, at 22:42, Segher Boessenkool wrote:
> > > There was just an assumption that LE == powerpc64le in libgo, spotted by 
> > > q66 (daniel@ on the CC). I just pushed the patch to [1].
> > 
> > Please send GCC patches to gcc-patches@ ?
> 
> FWIW, that patch alone is not very useful, we'd need to otherwise patch libgo to recognize a new GOARCH (as right now it's likely to just use 'ppc' which is wrong).

Gotcha.

> That said, we'll get back to you with any patches we have. One I can already think of - we will need to update the dynamic linker name so that it uses ld-musl-powerpcle.so instead of powerpc (musl needs to be updated the same way by adding the subarch variable for the 'le' prefix).

Thanks!  That would be good progress.

> > > > Almost no project that used 32-bit PowerPC in LE mode has sent patches
> > > > to the upstreams.
> > > 
> > > Right, but I have heard concerns from at least one person familiar with 
> > > the ppc kernel about breaking existing users of this arch-endianness 
> > > combo, if any. It seems likely that none of those use upstream, though ^^;
> > 
> > So we don't care, because we *cannot* care.
> 
> Well, that's the reason this thread was opened in the first place - to call out to any potential users, and synchronize with upstreams on a single way forward that all upstreams can agree on, since this effort requires changes in various parts of the stack. We don't want to hog changes locally or otherwise do any changes that would be in conflict with upstream projects, as that would mean needlessly diverging, which only means trouble later on.

Much appreciated!

I don't actually foresee any huge problems -- just lots of hard work ;-)


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-01 21:28 ` Joseph Myers
  2020-06-01 21:36   ` Rich Felker
  2020-06-01 23:26   ` Daniel Kolesa
@ 2020-06-02  1:42   ` Segher Boessenkool
  2020-06-02  2:03     ` Daniel Kolesa
  2 siblings, 1 reply; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-02  1:42 UTC (permalink / raw)
  To: Joseph Myers
  Cc: Will Springer, libc-alpha, eery, daniel, musl, binutils,
	libc-dev, linuxppc-dev

Hi Joseph,

On Mon, Jun 01, 2020 at 09:28:25PM +0000, Joseph Myers wrote:
> On Fri, 29 May 2020, Will Springer via Binutils wrote:
> 
> > Hey all, a couple of us over in #talos-workstation on freenode have been
> > working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> > little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> > designated for this (unless you count the patchset from a decade ago [1]), so
> > it's pretty much uncharted territory as far as Linux is concerned. We want to
> > sync up with libc and the relevant kernel folks to establish the best path
> > forward.
> 
> As a general comment on the glibc side of things, if this is considered 
> like a new port, and it probably is, the same principles that apply to new 
> ports apply here.
> 
> There's a general discussion at 
> <https://sourceware.org/glibc/wiki/NewPorts>, although much of that is 
> only applicable when adding new CPU architecture support.  More specific 
> points include that new 32-bit ports should default to 64-bit time and 
> file offsets from the start, with no support for 32-bit time or offsets 
> (meaning that if you want to use this with some kind of library call 
> translation, the library call translation will need to deal with 
> corresponding type size conversions).

Either that, or use the same as BE 32-bit PowerPC Linux, I'd say (it
won't make things worse, and if it is easier?)  But preferably the
newer, better, thing of course :-)

> And a new port should not be added 
> that uses the IBM long double format.  You can use IEEE binary128 long 
> double, possibly with an ABI similar to that used on powerpc64le, or can 
> use long double = double, but should not support IBM long double, and 
> preferably should only have one long double format rather than using the 
> glibc support for building with different options resulting in functions 
> for different long double formats being called.

You cannot use IEEE QP float ("binary128") here, but more on that in a
later post.

(I so very much agree about the problems having more than one long
double format -- on the other hand, you'll just share it with BE, and
with the existing powerpcle-linux (sup)port).


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-01 23:26   ` Daniel Kolesa
  2020-06-01 23:45     ` Joseph Myers
@ 2020-06-02  1:58     ` Segher Boessenkool
  2020-06-02  2:09       ` Jeffrey Walton
  2020-06-02  2:12       ` Daniel Kolesa
  1 sibling, 2 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-02  1:58 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: Joseph Myers, Will Springer, libc-alpha, eery, musl,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 02, 2020 at 01:26:37AM +0200, Daniel Kolesa wrote:
> On Mon, Jun 1, 2020, at 23:28, Joseph Myers wrote:
> Are you sure this would be a new port? Glibc already works in this combination, as it seems to me it'd be best if it was just a variant of the existing 32-bit PowerPC port, sharing most conventions besides endianness with the BE port.

That's right.  Except it isn't an "official" existing port, never has
been "officially" supported.

> 128-bit IEEE long double would not work, since that relies on VSX being present (gcc will explicitly complain if it's not). I'd be all for using 64-bit long double, though (musl already does, on all ppc ports).

The current IEEE QP float support requires VSX for its emulation, yes
(possibly even Power8?)  As Mike reminded me today, it also requires
__int128 support, which rules out anything 32-bit currently.  Without
that restriction, we could just make QP float passed in GPRs (use the
ABIs for any struct passed that way), and that'll just work out with
all ABIs, older or not.

> While we're at long double, I'd actually be interested in transitioning the existing big endian ports in Void (64-bit and 32-bit, neither has VSX baseline requirement in my case) to using 64-bit long double, abandoning the IBM format altogether (little endian will transition to 128-bit IEEE long double once it's ready on your side, as that assumes POWER8 baseline which includes VSX).

I recommend new ports that cannot jump to IEEE QP float directly to use
long double == double for the time being, avoiding the extra
complications that IBM double double would bring.  But you'll still have
a transition to IEEE 128 if you ever want to go there.

But if you already use double-double, I don't know if the cost changing
away from that is worth it now.

> What would be the best way for me to proceed with that? I actually experimented with this, using the old glibc compat symbols from pre-ibm128 times, and I mostly had it working, except I haven't managed to find a way to switch the default symbols to 64-bit ones, which is problematic as linking everything against nldbl_nonshared is fragile and potentially quirky (breaks dlsym, function pointer equality across libraries, etc).

Yup.  "Rebuild the world" works :-/  I don't have any  better advice,
nothing you cannot figure out yourself.

> There is also one more thing while we're at this. The 64-bit big endian Void port uses the ELFv2 ABI, even on glibc. This is not officially supported on glibc as far as I can tell, but it does work out of box, without any patching (things in general match little endian then, i.e. ld64.so.2 etc, but they're big endian). Is there any chance of making that support official?

(I don't talk for glibc).

The first thing needed is for "us" to have faith in it.  That starts
with seeing test results for the testsuites!

(Something similar goes for the GCC port -- there is no official support
for BE ELFv2, but of course it does work, and if we get test results we
may keep it that way, hint hint :-) )


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02  1:42   ` Segher Boessenkool
@ 2020-06-02  2:03     ` Daniel Kolesa
  0 siblings, 0 replies; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-02  2:03 UTC (permalink / raw)
  To: Segher Boessenkool, Joseph Myers
  Cc: Will Springer, libc-alpha, eery, musl,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 2, 2020, at 03:42, Segher Boessenkool wrote:
> Hi Joseph,
> 
> On Mon, Jun 01, 2020 at 09:28:25PM +0000, Joseph Myers wrote:
> > On Fri, 29 May 2020, Will Springer via Binutils wrote:
> > 
> > > Hey all, a couple of us over in #talos-workstation on freenode have been
> > > working on an effort to bring up a Linux PowerPC userland that runs in 32-bit
> > > little-endian mode, aka ppcle. As far as we can tell, no ABI has ever been
> > > designated for this (unless you count the patchset from a decade ago [1]), so
> > > it's pretty much uncharted territory as far as Linux is concerned. We want to
> > > sync up with libc and the relevant kernel folks to establish the best path
> > > forward.
> > 
> > As a general comment on the glibc side of things, if this is considered 
> > like a new port, and it probably is, the same principles that apply to new 
> > ports apply here.
> > 
> > There's a general discussion at 
> > <https://sourceware.org/glibc/wiki/NewPorts>, although much of that is 
> > only applicable when adding new CPU architecture support.  More specific 
> > points include that new 32-bit ports should default to 64-bit time and 
> > file offsets from the start, with no support for 32-bit time or offsets 
> > (meaning that if you want to use this with some kind of library call 
> > translation, the library call translation will need to deal with 
> > corresponding type size conversions).
> 
> Either that, or use the same as BE 32-bit PowerPC Linux, I'd say (it
> won't make things worse, and if it is easier?)  But preferably the
> newer, better, thing of course :-)
> 
> > And a new port should not be added 
> > that uses the IBM long double format.  You can use IEEE binary128 long 
> > double, possibly with an ABI similar to that used on powerpc64le, or can 
> > use long double = double, but should not support IBM long double, and 
> > preferably should only have one long double format rather than using the 
> > glibc support for building with different options resulting in functions 
> > for different long double formats being called.
> 
> You cannot use IEEE QP float ("binary128") here, but more on that in a
> later post.
> 
> (I so very much agree about the problems having more than one long
> double format -- on the other hand, you'll just share it with BE, and
> with the existing powerpcle-linux (sup)port).

Well, it'd be nice to use the opportunity to ditch the IBM long doubles altogether, since these get in the way for me in various places (for instance, GCC still can't constant-fold them, which breaks on constexpr in C++, as well as makes it impossible to e.g. enable the runtime/standard library for GDC in GCC as that heavily relies on the `real` type in D, which maps directly to `long double` and druntime/phobos heavily relies on constant folding of the `real` type being functional; there are also assorted libraries and applications in the common userland that don't like the IBM format for one reason or another)

That said, that's also problematic:

1) ppc64le is going to newly use IEEE754 binary128, so we're all good there - baseline mandates VSX, so at least passing them is fine, which is the important thing (actual binary128 instructions came with ISA 3.0 but that can be detected at runtime, at least in glibc) - ibm128 is then implemented via symvers/compat, which is fine.

2) ppc64 for now uses IBM 128-bit long double, so it's problematic. I don't care about ELFv1, as it's legacy and has tons of drawbacks on its own, and there's a whole legacy ecosystem relying on it; that said, a new ELFv2 port (let's say, with ld64.so.3 dynamic linker) could easily default to another long double format, without worrying about compat - I propose this format to be binary64, as it's already used by musl on BE/64 (allows projects such as `gcompat` to work) and thus is already implemented in all toolchains we care about and can easily be flipped on, and is fully compatible with all CPUs that can run ppc64 code, even without VMX/VSX

3) ppcle would be a new port (let's say ld.so.2), so it could default to a new format; binary64 would be good

4) that leaves ppc32/BE, which would be the only outlier - while I could probably implement compat in much the same way as ppc64le does with binary128 (bump symvers for math symbols, leave older symvers for existing binaries, change defaults), this would be divergent from the existing port; glibc/IBM probably won't want to switch this, and while I would definitely like to, maintaining a divergent downstream patchset seems non-ideal. There is some basis for this - glibc did use to use binary64 on ppc32 and ppc64 BE in the past, and the compatibility symbols are still there under the old symvers, but not quite sure what to do there.

> 
> 
> Segher
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02  1:58     ` Segher Boessenkool
@ 2020-06-02  2:09       ` Jeffrey Walton
  2020-06-02  2:12       ` Daniel Kolesa
  1 sibling, 0 replies; 59+ messages in thread
From: Jeffrey Walton @ 2020-06-02  2:09 UTC (permalink / raw)
  To: musl
  Cc: Daniel Kolesa, Joseph Myers, Will Springer, libc-alpha, eery,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Mon, Jun 1, 2020 at 9:58 PM Segher Boessenkool
<segher@kernel.crashing.org> wrote:
>
> On Tue, Jun 02, 2020 at 01:26:37AM +0200, Daniel Kolesa wrote:
> > On Mon, Jun 1, 2020, at 23:28, Joseph Myers wrote:
> > Are you sure this would be a new port? Glibc already works in this combination, as it seems to me it'd be best if it was just a variant of the existing 32-bit PowerPC port, sharing most conventions besides endianness with the BE port.
>
> That's right.  Except it isn't an "official" existing port, never has
> been "officially" supported.
>
> > 128-bit IEEE long double would not work, since that relies on VSX being present (gcc will explicitly complain if it's not). I'd be all for using 64-bit long double, though (musl already does, on all ppc ports).
>
> The current IEEE QP float support requires VSX for its emulation, yes
> (possibly even Power8?) ...

I believe Steven Munroe has an implementation that includes emulation
for lesser POWER architectures at
https://github.com/munroesj52/pveclib. If I am not mistaken, Munroe 's
library supports back to POWER7.

Modern GCC is fine. Clang has problems because it lacks support for
ISO/IEC TS 18661 or N2341
(http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2341.pdf).

Jeff

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02  1:58     ` Segher Boessenkool
  2020-06-02  2:09       ` Jeffrey Walton
@ 2020-06-02  2:12       ` Daniel Kolesa
  2020-06-02  2:36         ` Segher Boessenkool
  1 sibling, 1 reply; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-02  2:12 UTC (permalink / raw)
  To: Segher Boessenkool, musl
  Cc: Joseph Myers, Will Springer, libc-alpha, eery,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 2, 2020, at 03:58, Segher Boessenkool wrote:
> On Tue, Jun 02, 2020 at 01:26:37AM +0200, Daniel Kolesa wrote:
> > On Mon, Jun 1, 2020, at 23:28, Joseph Myers wrote:
> > Are you sure this would be a new port? Glibc already works in this combination, as it seems to me it'd be best if it was just a variant of the existing 32-bit PowerPC port, sharing most conventions besides endianness with the BE port.
> 
> That's right.  Except it isn't an "official" existing port, never has
> been "officially" supported.
> 
> > 128-bit IEEE long double would not work, since that relies on VSX being present (gcc will explicitly complain if it's not). I'd be all for using 64-bit long double, though (musl already does, on all ppc ports).
> 
> The current IEEE QP float support requires VSX for its emulation, yes
> (possibly even Power8?)  As Mike reminded me today, it also requires
> __int128 support, which rules out anything 32-bit currently.  Without
> that restriction, we could just make QP float passed in GPRs (use the
> ABIs for any struct passed that way), and that'll just work out with
> all ABIs, older or not.
> 
> > While we're at long double, I'd actually be interested in transitioning the existing big endian ports in Void (64-bit and 32-bit, neither has VSX baseline requirement in my case) to using 64-bit long double, abandoning the IBM format altogether (little endian will transition to 128-bit IEEE long double once it's ready on your side, as that assumes POWER8 baseline which includes VSX).
> 
> I recommend new ports that cannot jump to IEEE QP float directly to use
> long double == double for the time being, avoiding the extra
> complications that IBM double double would bring.  But you'll still have
> a transition to IEEE 128 if you ever want to go there.
> 
> But if you already use double-double, I don't know if the cost changing
> away from that is worth it now.

The transition cost is relatively low, which is why I'm thinking about this in the first place. For one, relatively few things use long double in the first place. For two, on ppc*, at least the bfd linker (which we use always in Void) always tags ELFs with an FP ABI tag, and things not using long double (or using 64-bit long double) don't receive this tag. It even tags *which* ABI is used. See:

```
$ readelf -a /usr/bin/echo|grep Power_ABI_FP
$ 
$ readelf -a /usr/lib/libc-2.30.so|grep Power_ABI_FP
  Tag_GNU_Power_ABI_FP: hard float, 128-bit IBM long double
```

I went through this once already (I had the 64-bit ldbl transition nearly done) and the number of packages to rebuild in the whole repo was about 200-300 out of ~12000.

> 
> > What would be the best way for me to proceed with that? I actually experimented with this, using the old glibc compat symbols from pre-ibm128 times, and I mostly had it working, except I haven't managed to find a way to switch the default symbols to 64-bit ones, which is problematic as linking everything against nldbl_nonshared is fragile and potentially quirky (breaks dlsym, function pointer equality across libraries, etc).
> 
> Yup.  "Rebuild the world" works :-/  I don't have any  better advice,
> nothing you cannot figure out yourself.

See above.

> 
> > There is also one more thing while we're at this. The 64-bit big endian Void port uses the ELFv2 ABI, even on glibc. This is not officially supported on glibc as far as I can tell, but it does work out of box, without any patching (things in general match little endian then, i.e. ld64.so.2 etc, but they're big endian). Is there any chance of making that support official?
> 
> (I don't talk for glibc).
> 
> The first thing needed is for "us" to have faith in it.  That starts
> with seeing test results for the testsuites!
> 
> (Something similar goes for the GCC port -- there is no official support
> for BE ELFv2, but of course it does work, and if we get test results we
> may keep it that way, hint hint :-) )

Well, FreeBSD defaults to it since 13; OpenBSD's new powerpc64 port (which is supposedly dual-endian) defaults to it; musl defaults to it on LE and BE. FreeBSD and OpenBSD have to, since they primarily target LLVM system toolchain (with GCC in ports) and ld.lld doesn't support ELFv1 (at all). Void's port was new (and any precompiled binaries would generally be enterprisey stuff which doesn't concern us enough - people can just make a chroot/container with say, Debian, if they really need to), so I felt like it didn't make sense to go with the legacy ABI (besides, function descriptors are gross ;)). The situation in the overall userland has been improving too, so the patch burden is actually very low nowadays.

> 
> 
> Segher
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-01 23:45     ` Joseph Myers
  2020-06-01 23:55       ` Joseph Myers
  2020-06-02  0:11       ` Daniel Kolesa
@ 2020-06-02  2:12       ` Segher Boessenkool
  2020-06-02  2:17         ` Daniel Kolesa
  2020-06-02 13:50         ` Joseph Myers
  2 siblings, 2 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-02  2:12 UTC (permalink / raw)
  To: Joseph Myers
  Cc: Daniel Kolesa, libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Mon, Jun 01, 2020 at 11:45:51PM +0000, Joseph Myers wrote:
> On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > Are you sure this would be a new port? Glibc already works in this 
> > combination, as it seems to me it'd be best if it was just a variant of 
> > the existing 32-bit PowerPC port, sharing most conventions besides 
> > endianness with the BE port.
> 
> The supported glibc ABIs are listed at 
> <https://sourceware.org/glibc/wiki/ABIList>.

powerpcle-linux already does work somewhat, and that should also he
worth something, official or not ;-)

(It has worked for very many years already...  That is, I have built it
a lot, I have no idea about running a full distro that way).

> > 128-bit IEEE long double would not work, since that relies on VSX being 
> > present (gcc will explicitly complain if it's not). I'd be all for using 
> 
> The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> understanding was that the suggestion was for 32-bit userspace to run 
> under powerpc64le kernels running on POWER8 or later, meaning that such a 
> 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> available.  Or does VSX not work, at the hardware level, for 32-bit 
> POWER8?  (In which case you could pick another ABI for binary128 argument 
> passing and return.)

The current powerpcle-linux runs on anything 6xx, or maybe older.  It
isn't actually supported of course.

If the CPU is Power8, that does not mean VSX is available to you.

VSX works fine in 32-bit mode (with the standard gotcha that the GPRs
do not preserve the high part in all cases, so e.g. the m[ft]vsr insns
might not work as you want.

Passing IEEE QP float in GPRs would be natural for most ABIs, and it
should work fine indeed.  That isn't currently supported in GCC (needs
some libgcc work), and it might need __int128 to work on 32-bit ports
first.

> > There is also one more thing while we're at this. The 64-bit big endian 
> > Void port uses the ELFv2 ABI, even on glibc. This is not officially 
> > supported on glibc as far as I can tell, but it does work out of box, 
> > without any patching (things in general match little endian then, i.e. 
> > ld64.so.2 etc, but they're big endian). Is there any chance of making 
> > that support official?
> 
> If you want to support ELFv2 for 64-bit big endian in glibc, again that 
> should have a unique dynamic linker name, new symbol versions, only 
> binary128 long double, etc. - avoid all the legacy aspects of the existing 
> ELFv1 port rather than selectively saying that "ELFv1" itself is the only 
> legacy aspect and keeping the others (when it's the others that are 
> actually more problematic in glibc).

You should view it as a variant of the LE ELFv2 port, it has nothing
much to do with the other BE 64-bit PowerPC ports, other than being BE.


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02  2:12       ` Segher Boessenkool
@ 2020-06-02  2:17         ` Daniel Kolesa
  2020-06-02 13:50         ` Joseph Myers
  1 sibling, 0 replies; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-02  2:17 UTC (permalink / raw)
  To: Segher Boessenkool, Joseph Myers
  Cc: libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev



On Tue, Jun 2, 2020, at 04:12, Segher Boessenkool wrote:
> On Mon, Jun 01, 2020 at 11:45:51PM +0000, Joseph Myers wrote:
> > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > > Are you sure this would be a new port? Glibc already works in this 
> > > combination, as it seems to me it'd be best if it was just a variant of 
> > > the existing 32-bit PowerPC port, sharing most conventions besides 
> > > endianness with the BE port.
> > 
> > The supported glibc ABIs are listed at 
> > <https://sourceware.org/glibc/wiki/ABIList>.
> 
> powerpcle-linux already does work somewhat, and that should also he
> worth something, official or not ;-)
> 
> (It has worked for very many years already...  That is, I have built it
> a lot, I have no idea about running a full distro that way).
> 
> > > 128-bit IEEE long double would not work, since that relies on VSX being 
> > > present (gcc will explicitly complain if it's not). I'd be all for using 
> > 
> > The minimum supported architecture for powerpc64le (POWER8) has VSX.  My 
> > understanding was that the suggestion was for 32-bit userspace to run 
> > under powerpc64le kernels running on POWER8 or later, meaning that such a 
> > 32-bit LE port, and any ABI designed for such a port, can assume VSX is 
> > available.  Or does VSX not work, at the hardware level, for 32-bit 
> > POWER8?  (In which case you could pick another ABI for binary128 argument 
> > passing and return.)
> 
> The current powerpcle-linux runs on anything 6xx, or maybe older.  It
> isn't actually supported of course.
> 
> If the CPU is Power8, that does not mean VSX is available to you.
> 
> VSX works fine in 32-bit mode (with the standard gotcha that the GPRs
> do not preserve the high part in all cases, so e.g. the m[ft]vsr insns
> might not work as you want.
> 
> Passing IEEE QP float in GPRs would be natural for most ABIs, and it
> should work fine indeed.  That isn't currently supported in GCC (needs
> some libgcc work), and it might need __int128 to work on 32-bit ports
> first.
> 
> > > There is also one more thing while we're at this. The 64-bit big endian 
> > > Void port uses the ELFv2 ABI, even on glibc. This is not officially 
> > > supported on glibc as far as I can tell, but it does work out of box, 
> > > without any patching (things in general match little endian then, i.e. 
> > > ld64.so.2 etc, but they're big endian). Is there any chance of making 
> > > that support official?
> > 
> > If you want to support ELFv2 for 64-bit big endian in glibc, again that 
> > should have a unique dynamic linker name, new symbol versions, only 
> > binary128 long double, etc. - avoid all the legacy aspects of the existing 
> > ELFv1 port rather than selectively saying that "ELFv1" itself is the only 
> > legacy aspect and keeping the others (when it's the others that are 
> > actually more problematic in glibc).
> 
> You should view it as a variant of the LE ELFv2 port, it has nothing
> much to do with the other BE 64-bit PowerPC ports, other than being BE.

He does have a point with multiarch though - e.g. Debian lets you have all architectures in the same system (with the foreign ones running through binfmt of course) and has individual sysroots for each arch, but dynamic linkers still need to be present in a single location. So, different or not, BE/LE should probably each get its own dynamic linker name (musl solves this nicely by using consistent ld-musl-<archname>so, glibc's naming is fairly arbitrary)

> 
> 
> Segher
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02  2:12       ` Daniel Kolesa
@ 2020-06-02  2:36         ` Segher Boessenkool
  2020-06-02  2:55           ` Daniel Kolesa
  0 siblings, 1 reply; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-02  2:36 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: musl, Joseph Myers, Will Springer, libc-alpha, eery,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 02, 2020 at 04:12:26AM +0200, Daniel Kolesa wrote:
> On Tue, Jun 2, 2020, at 03:58, Segher Boessenkool wrote:
> > I recommend new ports that cannot jump to IEEE QP float directly to use
> > long double == double for the time being, avoiding the extra
> > complications that IBM double double would bring.  But you'll still have
> > a transition to IEEE 128 if you ever want to go there.
> > 
> > But if you already use double-double, I don't know if the cost changing
> > away from that is worth it now.
> 
> The transition cost is relatively low, which is why I'm thinking about this in the first place. For one, relatively few things use long double in the first place.

Then your cost switching to QP float later will be low as well.  I envy
you :-)

> For two, on ppc*, at least the bfd linker (which we use always in Void) always tags ELFs with an FP ABI tag, and things not using long double (or using 64-bit long double) don't receive this tag. It even tags *which* ABI is used. See:

That works for statically linked stuff, sure.  That is the easy case :-/

> I went through this once already (I had the 64-bit ldbl transition nearly done) and the number of packages to rebuild in the whole repo was about 200-300 out of ~12000.

Cool!  Do you perchance have info you can share about which packages?
Offline, if you want.

> > > There is also one more thing while we're at this. The 64-bit big endian Void port uses the ELFv2 ABI, even on glibc. This is not officially supported on glibc as far as I can tell, but it does work out of box, without any patching (things in general match little endian then, i.e. ld64.so.2 etc, but they're big endian). Is there any chance of making that support official?
> > 
> > (I don't talk for glibc).
> > 
> > The first thing needed is for "us" to have faith in it.  That starts
> > with seeing test results for the testsuites!
> > 
> > (Something similar goes for the GCC port -- there is no official support
> > for BE ELFv2, but of course it does work, and if we get test results we
> > may keep it that way, hint hint :-) )
> 
> Well, FreeBSD defaults to it since 13; OpenBSD's new powerpc64 port (which is supposedly dual-endian) defaults to it; musl defaults to it on LE and BE.

... and no one ever has sent us (GCC) test results (nothing I have seen
anyway).  All we "officially" know is that Power7 BE ELFv2 was a
bring-up vehicle for the current powerpc64le-linux.  Everyone tries not
to break things without reason to, of course, and things are a little
bit tested anyway because it is convenient to build ELFv2 stuff on BE
systems as well (if only to figure out effortlessly if some bug is due
to the ABI or due to the endianness), but if we do not know something is
used and we never officially supported it, we might just want to take it
away if it is inconvenient.

> FreeBSD and OpenBSD have to, since they primarily target LLVM system toolchain (with GCC in ports) and ld.lld doesn't support ELFv1 (at all). Void's port was new (and any precompiled binaries would generally be enterprisey stuff which doesn't concern us enough - people can just make a chroot/container with say, Debian, if they really need to),

Yeah, enterprisey enough, then just rebuild :-)

> so I felt like it didn't make sense to go with the legacy ABI (besides, function descriptors are gross ;)).

Descriptors are Great, you just do not understand the True Way!

> The situation in the overall userland has been improving too, so the patch burden is actually very low nowadays.

Is that because long double just isn't used a lot?  Or are there more
reasons?


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02  2:36         ` Segher Boessenkool
@ 2020-06-02  2:55           ` Daniel Kolesa
  0 siblings, 0 replies; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-02  2:55 UTC (permalink / raw)
  To: Segher Boessenkool, musl
  Cc: Joseph Myers, Will Springer, libc-alpha, eery,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 2, 2020, at 04:36, Segher Boessenkool wrote:
> On Tue, Jun 02, 2020 at 04:12:26AM +0200, Daniel Kolesa wrote:
> > On Tue, Jun 2, 2020, at 03:58, Segher Boessenkool wrote:
> > > I recommend new ports that cannot jump to IEEE QP float directly to use
> > > long double == double for the time being, avoiding the extra
> > > complications that IBM double double would bring.  But you'll still have
> > > a transition to IEEE 128 if you ever want to go there.
> > > 
> > > But if you already use double-double, I don't know if the cost changing
> > > away from that is worth it now.
> > 
> > The transition cost is relatively low, which is why I'm thinking about this in the first place. For one, relatively few things use long double in the first place.
> 
> Then your cost switching to QP float later will be low as well.  I envy
> you :-)
> 
> > For two, on ppc*, at least the bfd linker (which we use always in Void) always tags ELFs with an FP ABI tag, and things not using long double (or using 64-bit long double) don't receive this tag. It even tags *which* ABI is used. See:
> 
> That works for statically linked stuff, sure.  That is the easy case :-/

It works for dynamically linked stuff too, since anything either using or exposing any API from anywhere that contains a long double signature will result in the tag being emitted. The difference becomes fuzzier once you switch to -mlong-double-64, as then they effectively become one type, and you can no longer check for it. But when it's still on 128-bit (any format) it's reasonably trivial (mostly just check the whole repo for binaries/libraries, dump the tags everywhere, collect them...)

> 
> > I went through this once already (I had the 64-bit ldbl transition nearly done) and the number of packages to rebuild in the whole repo was about 200-300 out of ~12000.
> 
> Cool!  Do you perchance have info you can share about which packages?
> Offline, if you want.

Sure. Here's the list of stuff I had to bump in our repo when I attempted the transition ~half a year ago: https://gist.github.com/q66/08720863a3aec12a6612356cd4c0110f

Of course, we've since gained packages, so it might be slightly different now, and our repos are smaller than something like Debian's, but otherwise this list is not super difficult to compile. I'll be compiling a new one for the ieee754 binary128 transition on glibc when the time comes :)

> 
> > > > There is also one more thing while we're at this. The 64-bit big endian Void port uses the ELFv2 ABI, even on glibc. This is not officially supported on glibc as far as I can tell, but it does work out of box, without any patching (things in general match little endian then, i.e. ld64.so.2 etc, but they're big endian). Is there any chance of making that support official?
> > > 
> > > (I don't talk for glibc).
> > > 
> > > The first thing needed is for "us" to have faith in it.  That starts
> > > with seeing test results for the testsuites!
> > > 
> > > (Something similar goes for the GCC port -- there is no official support
> > > for BE ELFv2, but of course it does work, and if we get test results we
> > > may keep it that way, hint hint :-) )
> > 
> > Well, FreeBSD defaults to it since 13; OpenBSD's new powerpc64 port (which is supposedly dual-endian) defaults to it; musl defaults to it on LE and BE.
> 
> ... and no one ever has sent us (GCC) test results (nothing I have seen
> anyway).  All we "officially" know is that Power7 BE ELFv2 was a
> bring-up vehicle for the current powerpc64le-linux.  Everyone tries not
> to break things without reason to, of course, and things are a little
> bit tested anyway because it is convenient to build ELFv2 stuff on BE
> systems as well (if only to figure out effortlessly if some bug is due
> to the ABI or due to the endianness), but if we do not know something is
> used and we never officially supported it, we might just want to take it
> away if it is inconvenient.

Void keeps track of recent versions fairly closely, so if something does break, you can be sure you'll hear about it :) If there is a breakage in something common on ppc BE configurations, we're usually the first to come across it these days (the Adélie folks are doing a good job but they're usually on LTS versions, and Gentoo musl folks tend to be helpful, but still)

Unfortunately, the maintainer team is stretched relatively thin, and on ppc-related maintenance in the distro, it's mostly just me when it comes to major work, so deaing with all upstreams can be tricky. Still, I try to make sure we don't lag behind too much.

> 
> > FreeBSD and OpenBSD have to, since they primarily target LLVM system toolchain (with GCC in ports) and ld.lld doesn't support ELFv1 (at all). Void's port was new (and any precompiled binaries would generally be enterprisey stuff which doesn't concern us enough - people can just make a chroot/container with say, Debian, if they really need to),
> 
> Yeah, enterprisey enough, then just rebuild :-)
> 
> > so I felt like it didn't make sense to go with the legacy ABI (besides, function descriptors are gross ;)).
> 
> Descriptors are Great, you just do not understand the True Way!
> 
> > The situation in the overall userland has been improving too, so the patch burden is actually very low nowadays.
> 
> Is that because long double just isn't used a lot?  Or are there more
> reasons?

I meant ELFv2+BE. Recently support was gained in OpenSSL, for example (though it was a fairly trivial patch... but then, it's pretty much always a fairly trivial patch; the most common pitfalls tend to be mixing of __LITTLE_ENDIAN__ and _CALL_ELF == 2, like, in 90% of cases when there's assembly code hidden behind __LITTLE_ENDIAN__ you can be almost sure it's actually just ELFv2, but those cases have been slowly disappearing)

> 
> 
> Segher
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02  0:11       ` Daniel Kolesa
@ 2020-06-02 13:40         ` Joseph Myers
  2020-06-02 14:23           ` Michal Suchánek
  2020-06-02 14:52           ` Daniel Kolesa
  0 siblings, 2 replies; 59+ messages in thread
From: Joseph Myers @ 2020-06-02 13:40 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, 2 Jun 2020, Daniel Kolesa wrote:

> not be limited to being just userspace under ppc64le, but should be 
> runnable on a native kernel as well, which should not be limited to any 
> particular baseline other than just PowerPC.

This is a fairly unusual approach to bringing up a new ABI.  Since new 
ABIs are more likely to be used on new systems rather than switching ABI 
on an existing installation, and since it can take quite some time for all 
the software support for a new ABI to become widely available in 
distributions, people developing new ABIs are likely to think about what 
new systems are going to be relevant in a few years' time when working out 
the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
for powerpc64le fits in with that, for example.)

> either the AIX/ELFv1 nor the ELFv2 ABIs) If we were to introduce new 
> ports, what would those use? ld64.so.3 for BE/v2? ld.so.2 for LE/32-bit? 

Rather than relying on numbers such as "3" or 2" in a particular place 
being unique across all (architecture, ABI) pairs supported by glibc, 
something more obviously specific to a particular architecture and ABI, 
e.g. ld-linux-powerpc64be-elfv2.so.1, would be better.

-- 
Joseph S. Myers
joseph@codesourcery.com

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02  2:12       ` Segher Boessenkool
  2020-06-02  2:17         ` Daniel Kolesa
@ 2020-06-02 13:50         ` Joseph Myers
  2020-06-02 17:47           ` Segher Boessenkool
  1 sibling, 1 reply; 59+ messages in thread
From: Joseph Myers @ 2020-06-02 13:50 UTC (permalink / raw)
  To: Segher Boessenkool
  Cc: libc-alpha, eery, Daniel Kolesa, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Mon, 1 Jun 2020, Segher Boessenkool wrote:

> > The supported glibc ABIs are listed at 
> > <https://sourceware.org/glibc/wiki/ABIList>.
> 
> powerpcle-linux already does work somewhat, and that should also he
> worth something, official or not ;-)
> 
> (It has worked for very many years already...  That is, I have built it
> a lot, I have no idea about running a full distro that way).

Greg McGary's patches as referenced at 
<https://sourceware.org/legacy-ml/libc-hacker/2006-11/msg00013.html> were 
never merged, and I don't know how big the changes to .S files were or if 
they were ever posted.  Many files were subsequently fixed as part of 
bringing up support for powerpc64le, but without actual 32-bit LE testing 
as part of each release cycle there's not much evidence of correctness for 
LE of code not used for powerpc64le.

-- 
Joseph S. Myers
joseph@codesourcery.com

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02 13:40         ` Joseph Myers
@ 2020-06-02 14:23           ` Michal Suchánek
  2020-06-02 15:13             ` Daniel Kolesa
  2020-06-02 14:52           ` Daniel Kolesa
  1 sibling, 1 reply; 59+ messages in thread
From: Michal Suchánek @ 2020-06-02 14:23 UTC (permalink / raw)
  To: Joseph Myers
  Cc: Daniel Kolesa, libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> 
> > not be limited to being just userspace under ppc64le, but should be 
> > runnable on a native kernel as well, which should not be limited to any 
> > particular baseline other than just PowerPC.
> 
> This is a fairly unusual approach to bringing up a new ABI.  Since new 
> ABIs are more likely to be used on new systems rather than switching ABI 
> on an existing installation, and since it can take quite some time for all 
> the software support for a new ABI to become widely available in 
> distributions, people developing new ABIs are likely to think about what 
> new systems are going to be relevant in a few years' time when working out 
> the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> for powerpc64le fits in with that, for example.)
That means that you cannot run ppc64le on FSL embedded CPUs (which lack
the vector instructions in LE mode). Which may be fine with you but
other people may want to support these. Can't really say if that's good
idea or not but I don't foresee them going away in a few years, either.

Thanks

Michal

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02 13:40         ` Joseph Myers
  2020-06-02 14:23           ` Michal Suchánek
@ 2020-06-02 14:52           ` Daniel Kolesa
  1 sibling, 0 replies; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-02 14:52 UTC (permalink / raw)
  To: Joseph Myers
  Cc: libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 2, 2020, at 15:40, Joseph Myers wrote:
> On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> 
> > not be limited to being just userspace under ppc64le, but should be 
> > runnable on a native kernel as well, which should not be limited to any 
> > particular baseline other than just PowerPC.
> 
> This is a fairly unusual approach to bringing up a new ABI.  Since new 
> ABIs are more likely to be used on new systems rather than switching ABI 
> on an existing installation, and since it can take quite some time for all 
> the software support for a new ABI to become widely available in 
> distributions, people developing new ABIs are likely to think about what 
> new systems are going to be relevant in a few years' time when working out 
> the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> for powerpc64le fits in with that, for example.)

In this case the whole point is targeting existing hardware that we already have. It also aims to largely utilize things that are already present in all parts of the toolchain, otherwise it's a lot of effort with questionable benefit and artificially limits the baseline for no good reason.

> 
> > either the AIX/ELFv1 nor the ELFv2 ABIs) If we were to introduce new 
> > ports, what would those use? ld64.so.3 for BE/v2? ld.so.2 for LE/32-bit? 
> 
> Rather than relying on numbers such as "3" or 2" in a particular place 
> being unique across all (architecture, ABI) pairs supported by glibc, 
> something more obviously specific to a particular architecture and ABI, 
> e.g. ld-linux-powerpc64be-elfv2.so.1, would be better.

Yes, agreed on that - probably ld-linux-powerpc64-elfv2.so.1, to match existing conventions (powerpc64 implicitly refers to BE in target triples, etc). It's just inconsistent with the existing ports, but I guess the reason in those is legacy in the first place, so not much point in worrying about that.

> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02 14:23           ` Michal Suchánek
@ 2020-06-02 15:13             ` Daniel Kolesa
  2020-06-02 15:27               ` Michal Suchánek
  2020-06-04 17:12               ` Segher Boessenkool
  0 siblings, 2 replies; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-02 15:13 UTC (permalink / raw)
  To: Michal Suchánek, Joseph Myers
  Cc: libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 2, 2020, at 16:23, Michal Suchánek wrote:
> On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > 
> > > not be limited to being just userspace under ppc64le, but should be 
> > > runnable on a native kernel as well, which should not be limited to any 
> > > particular baseline other than just PowerPC.
> > 
> > This is a fairly unusual approach to bringing up a new ABI.  Since new 
> > ABIs are more likely to be used on new systems rather than switching ABI 
> > on an existing installation, and since it can take quite some time for all 
> > the software support for a new ABI to become widely available in 
> > distributions, people developing new ABIs are likely to think about what 
> > new systems are going to be relevant in a few years' time when working out 
> > the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> > for powerpc64le fits in with that, for example.)
> That means that you cannot run ppc64le on FSL embedded CPUs (which lack
> the vector instructions in LE mode). Which may be fine with you but
> other people may want to support these. Can't really say if that's good
> idea or not but I don't foresee them going away in a few years, either.

well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)

> 
> Thanks
> 
> Michal
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02 15:13             ` Daniel Kolesa
@ 2020-06-02 15:27               ` Michal Suchánek
  2020-06-02 15:40                 ` Daniel Kolesa
  2020-06-04 17:20                 ` Segher Boessenkool
  2020-06-04 17:12               ` Segher Boessenkool
  1 sibling, 2 replies; 59+ messages in thread
From: Michal Suchánek @ 2020-06-02 15:27 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: Joseph Myers, libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> On Tue, Jun 2, 2020, at 16:23, Michal Suchánek wrote:
> > On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> > > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > > 
> > > > not be limited to being just userspace under ppc64le, but should be 
> > > > runnable on a native kernel as well, which should not be limited to any 
> > > > particular baseline other than just PowerPC.
> > > 
> > > This is a fairly unusual approach to bringing up a new ABI.  Since new 
> > > ABIs are more likely to be used on new systems rather than switching ABI 
> > > on an existing installation, and since it can take quite some time for all 
> > > the software support for a new ABI to become widely available in 
> > > distributions, people developing new ABIs are likely to think about what 
> > > new systems are going to be relevant in a few years' time when working out 
> > > the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> > > for powerpc64le fits in with that, for example.)
> > That means that you cannot run ppc64le on FSL embedded CPUs (which lack
> > the vector instructions in LE mode). Which may be fine with you but
> > other people may want to support these. Can't really say if that's good
> > idea or not but I don't foresee them going away in a few years, either.
> 
> well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)

What hardware are you targetting then? I did not notice anything
specific mentioned in the thread.

Naturally on POWER the first cpu that has LE support is POWER8 so you
can count on all other POWER8 features to be present. With other
architecture variants the situation is different.

Thanks

Michal

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02 15:27               ` Michal Suchánek
@ 2020-06-02 15:40                 ` Daniel Kolesa
  2020-06-02 15:56                   ` Michal Suchánek
  2020-06-04 17:20                 ` Segher Boessenkool
  1 sibling, 1 reply; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-02 15:40 UTC (permalink / raw)
  To: Michal Suchánek
  Cc: Joseph Myers, libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev



On Tue, Jun 2, 2020, at 17:27, Michal Suchánek wrote:
> On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > On Tue, Jun 2, 2020, at 16:23, Michal Suchánek wrote:
> > > On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> > > > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > > > 
> > > > > not be limited to being just userspace under ppc64le, but should be 
> > > > > runnable on a native kernel as well, which should not be limited to any 
> > > > > particular baseline other than just PowerPC.
> > > > 
> > > > This is a fairly unusual approach to bringing up a new ABI.  Since new 
> > > > ABIs are more likely to be used on new systems rather than switching ABI 
> > > > on an existing installation, and since it can take quite some time for all 
> > > > the software support for a new ABI to become widely available in 
> > > > distributions, people developing new ABIs are likely to think about what 
> > > > new systems are going to be relevant in a few years' time when working out 
> > > > the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> > > > for powerpc64le fits in with that, for example.)
> > > That means that you cannot run ppc64le on FSL embedded CPUs (which lack
> > > the vector instructions in LE mode). Which may be fine with you but
> > > other people may want to support these. Can't really say if that's good
> > > idea or not but I don't foresee them going away in a few years, either.
> > 
> > well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)
> 
> What hardware are you targetting then? I did not notice anything
> specific mentioned in the thread.
> 
> Naturally on POWER the first cpu that has LE support is POWER8 so you
> can count on all other POWER8 features to be present. With other
> architecture variants the situation is different.

This is not true; nearly every 32-bit PowerPC CPU has LE support (all the way back to 6xx), these would be the native-hardware targets for the port (would need kernel support implemented, but it's technically possible).

As far as 64-bit CPUs go, POWER7 is the first one that could in practice run the current ppc64le configuration, but in glibc it's limited to POWER8 and in gcc the default for powerpc64le is also POWER8 (however, it is perfectly possible to configure gcc for POWER7 and use musl libc with it).

> 
> Thanks
> 
> Michal
>

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02 15:40                 ` Daniel Kolesa
@ 2020-06-02 15:56                   ` Michal Suchánek
  0 siblings, 0 replies; 59+ messages in thread
From: Michal Suchánek @ 2020-06-02 15:56 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: Joseph Myers, libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 02, 2020 at 05:40:39PM +0200, Daniel Kolesa wrote:
> 
> 
> On Tue, Jun 2, 2020, at 17:27, Michal Suchánek wrote:
> > On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > > On Tue, Jun 2, 2020, at 16:23, Michal Suchánek wrote:
> > > > On Tue, Jun 02, 2020 at 01:40:23PM +0000, Joseph Myers wrote:
> > > > > On Tue, 2 Jun 2020, Daniel Kolesa wrote:
> > > > > 
> > > > > > not be limited to being just userspace under ppc64le, but should be 
> > > > > > runnable on a native kernel as well, which should not be limited to any 
> > > > > > particular baseline other than just PowerPC.
> > > > > 
> > > > > This is a fairly unusual approach to bringing up a new ABI.  Since new 
> > > > > ABIs are more likely to be used on new systems rather than switching ABI 
> > > > > on an existing installation, and since it can take quite some time for all 
> > > > > the software support for a new ABI to become widely available in 
> > > > > distributions, people developing new ABIs are likely to think about what 
> > > > > new systems are going to be relevant in a few years' time when working out 
> > > > > the minimum hardware requirements for the new ABI.  (The POWER8 minimum 
> > > > > for powerpc64le fits in with that, for example.)
> > > > That means that you cannot run ppc64le on FSL embedded CPUs (which lack
> > > > the vector instructions in LE mode). Which may be fine with you but
> > > > other people may want to support these. Can't really say if that's good
> > > > idea or not but I don't foresee them going away in a few years, either.
> > > 
> > > well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)
> > 
> > What hardware are you targetting then? I did not notice anything
> > specific mentioned in the thread.
> > 
> > Naturally on POWER the first cpu that has LE support is POWER8 so you
> > can count on all other POWER8 features to be present. With other
> > architecture variants the situation is different.
> 
> This is not true; nearly every 32-bit PowerPC CPU has LE support (all the way back to 6xx), these would be the native-hardware targets for the port (would need kernel support implemented, but it's technically possible).
I find dealing with memory management issues on 32bit architectures a
pain. There is never enough address space.
> 
> As far as 64-bit CPUs go, POWER7 is the first one that could in practice run the current ppc64le configuration, but in glibc it's limited to POWER8 and in gcc the default for powerpc64le is also POWER8 (however, it is perfectly possible to configure gcc for POWER7 and use musl libc with it).
That's interesting. I guess I was tricked but the glibc limitation.

Thanks

Michal

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02 13:50         ` Joseph Myers
@ 2020-06-02 17:47           ` Segher Boessenkool
  0 siblings, 0 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-02 17:47 UTC (permalink / raw)
  To: Joseph Myers
  Cc: libc-alpha, eery, Daniel Kolesa, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Tue, Jun 02, 2020 at 01:50:32PM +0000, Joseph Myers wrote:
> On Mon, 1 Jun 2020, Segher Boessenkool wrote:
> 
> > > The supported glibc ABIs are listed at 
> > > <https://sourceware.org/glibc/wiki/ABIList>.
> > 
> > powerpcle-linux already does work somewhat, and that should also he
> > worth something, official or not ;-)
> > 
> > (It has worked for very many years already...  That is, I have built it
> > a lot, I have no idea about running a full distro that way).
> 
> Greg McGary's patches as referenced at 
> <https://sourceware.org/legacy-ml/libc-hacker/2006-11/msg00013.html> were 
> never merged, and I don't know how big the changes to .S files were or if 
> they were ever posted.  Many files were subsequently fixed as part of 
> bringing up support for powerpc64le, but without actual 32-bit LE testing 
> as part of each release cycle there's not much evidence of correctness for 
> LE of code not used for powerpc64le.

Yeah sorry, I meant I have built the toolchain parts a lot, not libc.
GCC and binutils.  I reckon there is more work to do in libc, certainly
for configuration and similar.


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02 15:13             ` Daniel Kolesa
  2020-06-02 15:27               ` Michal Suchánek
@ 2020-06-04 17:12               ` Segher Boessenkool
  2020-06-04 17:18                 ` Rich Felker
  1 sibling, 1 reply; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-04 17:12 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: Michal Suchánek, Joseph Myers, libc-alpha, eery, musl,
	Will Springer, Palmer Dabbelt via binutils, via libc-dev,
	linuxppc-dev

On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> well, ppc64le already cannot be run on those, as far as I know (I don't think it's possible to build ppc64le userland without VSX in any configuration)

VSX is required by the ELFv2 ABI:

"""
Specifically, to use this ABI and ABI-compliant programs, OpenPOWER-
compliant processors must implement the following categories:

[...]

Vector-Scalar
"""


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 17:12               ` Segher Boessenkool
@ 2020-06-04 17:18                 ` Rich Felker
  2020-06-04 17:33                   ` Segher Boessenkool
  0 siblings, 1 reply; 59+ messages in thread
From: Rich Felker @ 2020-06-04 17:18 UTC (permalink / raw)
  To: Segher Boessenkool
  Cc: Daniel Kolesa, Michal Suchánek, Joseph Myers, libc-alpha,
	eery, musl, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

On Thu, Jun 04, 2020 at 12:12:32PM -0500, Segher Boessenkool wrote:
> On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > well, ppc64le already cannot be run on those, as far as I know (I
> > don't think it's possible to build ppc64le userland without VSX in
> > any configuration)
> 
> VSX is required by the ELFv2 ABI:
> 
> """
> Specifically, to use this ABI and ABI-compliant programs, OpenPOWER-
> compliant processors must implement the following categories:

This is not actually ABI but IBM policy laundered into an ABI
document, which musl does not honor.

Rich

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-02 15:27               ` Michal Suchánek
  2020-06-02 15:40                 ` Daniel Kolesa
@ 2020-06-04 17:20                 ` Segher Boessenkool
  1 sibling, 0 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-04 17:20 UTC (permalink / raw)
  To: Michal Suchánek
  Cc: Daniel Kolesa, libc-alpha, eery, musl, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev,
	Joseph Myers

On Tue, Jun 02, 2020 at 05:27:24PM +0200, Michal Suchánek wrote:
> Naturally on POWER the first cpu that has LE support is POWER8 so you
> can count on all other POWER8 features to be present.

This is not true.

The oldest CPU the ELFv2 ABI (and so, powerpc64le-linux) supports is
POWER8, but most 6xx/7xx CPUs had a working LE mode already.  There are
very old ABIs that support LE as well.


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 17:18                 ` Rich Felker
@ 2020-06-04 17:33                   ` Segher Boessenkool
  2020-06-04 17:46                     ` Rich Felker
  2020-06-04 20:39                     ` Daniel Kolesa
  0 siblings, 2 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-04 17:33 UTC (permalink / raw)
  To: Rich Felker
  Cc: Daniel Kolesa, Michal Suchánek, Joseph Myers, libc-alpha,
	eery, musl, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

On Thu, Jun 04, 2020 at 01:18:44PM -0400, Rich Felker wrote:
> On Thu, Jun 04, 2020 at 12:12:32PM -0500, Segher Boessenkool wrote:
> > On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > > well, ppc64le already cannot be run on those, as far as I know (I
> > > don't think it's possible to build ppc64le userland without VSX in
> > > any configuration)
> > 
> > VSX is required by the ELFv2 ABI:
> > 
> > """
> > Specifically, to use this ABI and ABI-compliant programs, OpenPOWER-
> > compliant processors must implement the following categories:
> 
> This is not actually ABI but IBM policy laundered into an ABI
> document, which musl does not honor.

It is the ABI.  If you think it should be different, make your own ABI,
don't pretend the existing ABI is different than what it is.  Thank you.


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 17:33                   ` Segher Boessenkool
@ 2020-06-04 17:46                     ` Rich Felker
  2020-06-04 19:00                       ` David Edelsohn
  2020-06-04 20:39                     ` Daniel Kolesa
  1 sibling, 1 reply; 59+ messages in thread
From: Rich Felker @ 2020-06-04 17:46 UTC (permalink / raw)
  To: Segher Boessenkool
  Cc: Daniel Kolesa, Michal Suchánek, Joseph Myers, libc-alpha,
	eery, musl, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

On Thu, Jun 04, 2020 at 12:33:12PM -0500, Segher Boessenkool wrote:
> On Thu, Jun 04, 2020 at 01:18:44PM -0400, Rich Felker wrote:
> > On Thu, Jun 04, 2020 at 12:12:32PM -0500, Segher Boessenkool wrote:
> > > On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > > > well, ppc64le already cannot be run on those, as far as I know (I
> > > > don't think it's possible to build ppc64le userland without VSX in
> > > > any configuration)
> > > 
> > > VSX is required by the ELFv2 ABI:
> > > 
> > > """
> > > Specifically, to use this ABI and ABI-compliant programs, OpenPOWER-
> > > compliant processors must implement the following categories:
> > 
> > This is not actually ABI but IBM policy laundered into an ABI
> > document, which musl does not honor.
> 
> It is the ABI.  If you think it should be different, make your own ABI,
> don't pretend the existing ABI is different than what it is.  Thank you.

Our ABI is as specified in the ELFv2 document, but with ld as ld64,
and minus gratuitous requirements on ISA level that are not part of
implementing linkage.

Rich

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 17:46                     ` Rich Felker
@ 2020-06-04 19:00                       ` David Edelsohn
  2020-06-04 19:37                         ` Rich Felker
  0 siblings, 1 reply; 59+ messages in thread
From: David Edelsohn @ 2020-06-04 19:00 UTC (permalink / raw)
  To: musl
  Cc: Segher Boessenkool, Daniel Kolesa, Michal Suchánek,
	Joseph Myers, GNU C Library, eery, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Thu, Jun 4, 2020 at 1:46 PM Rich Felker <dalias@libc.org> wrote:
>
> On Thu, Jun 04, 2020 at 12:33:12PM -0500, Segher Boessenkool wrote:
> > On Thu, Jun 04, 2020 at 01:18:44PM -0400, Rich Felker wrote:
> > > On Thu, Jun 04, 2020 at 12:12:32PM -0500, Segher Boessenkool wrote:
> > > > On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > > > > well, ppc64le already cannot be run on those, as far as I know (I
> > > > > don't think it's possible to build ppc64le userland without VSX in
> > > > > any configuration)
> > > >
> > > > VSX is required by the ELFv2 ABI:
> > > >
> > > > """
> > > > Specifically, to use this ABI and ABI-compliant programs, OpenPOWER-
> > > > compliant processors must implement the following categories:
> > >
> > > This is not actually ABI but IBM policy laundered into an ABI
> > > document, which musl does not honor.
> >
> > It is the ABI.  If you think it should be different, make your own ABI,
> > don't pretend the existing ABI is different than what it is.  Thank you.
>
> Our ABI is as specified in the ELFv2 document, but with ld as ld64,
> and minus gratuitous requirements on ISA level that are not part of
> implementing linkage.

Rich,

If you are changing the Power ELFv2 ABI then it is not the Power ELFv2
ABI.  You can't cherry-pick what you like and claim that it is
compatible.  You are not conforming to the ABI.

Thanks, David

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 19:00                       ` David Edelsohn
@ 2020-06-04 19:37                         ` Rich Felker
  0 siblings, 0 replies; 59+ messages in thread
From: Rich Felker @ 2020-06-04 19:37 UTC (permalink / raw)
  To: David Edelsohn
  Cc: musl, Segher Boessenkool, Daniel Kolesa, Michal Suchánek,
	Joseph Myers, GNU C Library, eery, Will Springer,
	Palmer Dabbelt via binutils, via libc-dev, linuxppc-dev

On Thu, Jun 04, 2020 at 03:00:51PM -0400, David Edelsohn wrote:
> On Thu, Jun 4, 2020 at 1:46 PM Rich Felker <dalias@libc.org> wrote:
> >
> > On Thu, Jun 04, 2020 at 12:33:12PM -0500, Segher Boessenkool wrote:
> > > On Thu, Jun 04, 2020 at 01:18:44PM -0400, Rich Felker wrote:
> > > > On Thu, Jun 04, 2020 at 12:12:32PM -0500, Segher Boessenkool wrote:
> > > > > On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > > > > > well, ppc64le already cannot be run on those, as far as I know (I
> > > > > > don't think it's possible to build ppc64le userland without VSX in
> > > > > > any configuration)
> > > > >
> > > > > VSX is required by the ELFv2 ABI:
> > > > >
> > > > > """
> > > > > Specifically, to use this ABI and ABI-compliant programs, OpenPOWER-
> > > > > compliant processors must implement the following categories:
> > > >
> > > > This is not actually ABI but IBM policy laundered into an ABI
> > > > document, which musl does not honor.
> > >
> > > It is the ABI.  If you think it should be different, make your own ABI,
> > > don't pretend the existing ABI is different than what it is.  Thank you.
> >
> > Our ABI is as specified in the ELFv2 document, but with ld as ld64,
> > and minus gratuitous requirements on ISA level that are not part of
> > implementing linkage.
> 
> Rich,
> 
> If you are changing the Power ELFv2 ABI then it is not the Power ELFv2
> ABI.  You can't cherry-pick what you like and claim that it is
> compatible.  You are not conforming to the ABI.

You are aware of this and you are aware that I don't care about your
opinion on the matter, and that I don't appreciate your ongoing
harassment of users of musl who run on pre-POWER8 hardware that IBM
does not approve them using.

Rich

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 17:33                   ` Segher Boessenkool
  2020-06-04 17:46                     ` Rich Felker
@ 2020-06-04 20:39                     ` Daniel Kolesa
  2020-06-04 21:10                       ` Segher Boessenkool
  1 sibling, 1 reply; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-04 20:39 UTC (permalink / raw)
  To: Segher Boessenkool, Rich Felker
  Cc: Michal Suchánek, Joseph Myers, libc-alpha, eery, musl,
	Will Springer, Palmer Dabbelt via binutils, via libc-dev,
	linuxppc-dev

On Thu, Jun 4, 2020, at 19:33, Segher Boessenkool wrote:
> On Thu, Jun 04, 2020 at 01:18:44PM -0400, Rich Felker wrote:
> > On Thu, Jun 04, 2020 at 12:12:32PM -0500, Segher Boessenkool wrote:
> > > On Tue, Jun 02, 2020 at 05:13:25PM +0200, Daniel Kolesa wrote:
> > > > well, ppc64le already cannot be run on those, as far as I know (I
> > > > don't think it's possible to build ppc64le userland without VSX in
> > > > any configuration)
> > > 
> > > VSX is required by the ELFv2 ABI:
> > > 
> > > """
> > > Specifically, to use this ABI and ABI-compliant programs, OpenPOWER-
> > > compliant processors must implement the following categories:
> > 
> > This is not actually ABI but IBM policy laundered into an ABI
> > document, which musl does not honor.
> 
> It is the ABI.  If you think it should be different, make your own ABI,
> don't pretend the existing ABI is different than what it is.  Thank you.
> 

Well then - in that case, what do you suggest that I do?

Void currently ships an ELFv2 (or apparently not, I guess) 64-bit big endian port that works on 970/G5 up. It is important to me that it stays that way (a large amount of users are running 970s, so introducing a VSX dependency means I might as well abandon the port entirely).

It currently works out of box - there are no changes required in glibc, and nearly the entire userland builds and works (about ~11500 out of ~12000 software packages, those that don't work either don't work on ppc64le either, or have issues related to endianness, or some other unrelated reason).

I'd like to eventually get this into a state where I don't have to worry about glibc arbitrarily breaking it - which means it would be necessary to stabilize it upstream. While I can probably maintain a downstream patchset when it comes to it, I'd much prefer if I didn't have to - but this sounds like an official ELFv2 glibc BE port would be impossible unless the VSX requirement (and thus IEEE 128-bit long double and so on) was in place, which would defeat the point of the port.

Is there *any* way I can take that would make upstreams of all parts of the toolchain happy? I explicitly don't want to go back to ELFv1. While at it, I'd like to transition to ld64 long double format, to match musl and improve software compatibility, which I feel will raise more objections from IBM side.

> 
> Segher
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 20:39                     ` Daniel Kolesa
@ 2020-06-04 21:10                       ` Segher Boessenkool
  2020-06-04 21:43                         ` Daniel Kolesa
  0 siblings, 1 reply; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-04 21:10 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: Rich Felker, Michal Suchánek, Joseph Myers, libc-alpha,
	eery, musl, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

Hi!

On Thu, Jun 04, 2020 at 10:39:30PM +0200, Daniel Kolesa wrote:
> On Thu, Jun 4, 2020, at 19:33, Segher Boessenkool wrote:
> > It is the ABI.  If you think it should be different, make your own ABI,
> > don't pretend the existing ABI is different than what it is.  Thank you.
> 
> Well then - in that case, what do you suggest that I do?
> 
> Void currently ships an ELFv2 (or apparently not, I guess) 64-bit big endian port that works on 970/G5 up. It is important to me that it stays that way (a large amount of users are running 970s, so introducing a VSX dependency means I might as well abandon the port entirely).

You can just clearly document what ABI changes you use, and try to make
sure that everyone who uses your distro / your tools / your ABI variant
knows about it.  Telling your users that it is ELFv2, without telling
them it is not compliant, namely X Y Z are different, is a bit of a
disservice to your users, and worse to everyone else involved.

If you always use -mcpu=970 (or similar), then not very much is
different for you most likely -- except of course there is no promise
to the user that they can use VSX and all instructions in ISA 2.07,
which is a very useful promise to have normally.

> It currently works out of box - there are no changes required in glibc, and nearly the entire userland builds and works (about ~11500 out of ~12000 software packages, those that don't work either don't work on ppc64le either, or have issues related to endianness, or some other unrelated reason).

Very nice!

> I'd like to eventually get this into a state where I don't have to worry about glibc arbitrarily breaking it - which means it would be necessary to stabilize it upstream. While I can probably maintain a downstream patchset when it comes to it, I'd much prefer if I didn't have to - but this sounds like an official ELFv2 glibc BE port would be impossible unless the VSX requirement (and thus IEEE 128-bit long double and so on) was in place, which would defeat the point of the port.
> 
> Is there *any* way I can take that would make upstreams of all parts of the toolchain happy? I explicitly don't want to go back to ELFv1.

Oh absolutely, it sounds like things are in quite good shape already!
It will safe a lot of grief on all sides if you make clear this is not
"plain" ELFv2, and in what ways it differs.

Btw, if you use GCC, *please* send in testresults?  :-)

> While at it, I'd like to transition to ld64 long double format, to match musl and improve software compatibility, which I feel will raise more objections from IBM side.

I have no idea what "ld64 long double" is?  Is that just IEEE DP float?
Aka "long double is the same as double".  That is likely easier for new
ports than "double-double", yes, even if the eventual goal should be
IEEE QP float -- a much smoother transition.

Same goes here: document it!  If your users know that the ELFv2 variant
you give them is not *the* ELFv2, but it differs in some clear ways,
everyone will be happier :-)


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 21:10                       ` Segher Boessenkool
@ 2020-06-04 21:43                         ` Daniel Kolesa
  2020-06-04 22:08                           ` Joseph Myers
  2020-06-04 23:35                           ` Segher Boessenkool
  0 siblings, 2 replies; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-04 21:43 UTC (permalink / raw)
  To: Segher Boessenkool, musl
  Cc: Rich Felker, Michal Suchánek, Joseph Myers, libc-alpha,
	eery, Will Springer, Palmer Dabbelt via binutils, via libc-dev,
	linuxppc-dev

On Thu, Jun 4, 2020, at 23:10, Segher Boessenkool wrote:
> Hi!
> 
> On Thu, Jun 04, 2020 at 10:39:30PM +0200, Daniel Kolesa wrote:
> > On Thu, Jun 4, 2020, at 19:33, Segher Boessenkool wrote:
> > > It is the ABI.  If you think it should be different, make your own ABI,
> > > don't pretend the existing ABI is different than what it is.  Thank you.
> > 
> > Well then - in that case, what do you suggest that I do?
> > 
> > Void currently ships an ELFv2 (or apparently not, I guess) 64-bit big endian port that works on 970/G5 up. It is important to me that it stays that way (a large amount of users are running 970s, so introducing a VSX dependency means I might as well abandon the port entirely).
> 
> You can just clearly document what ABI changes you use, and try to make
> sure that everyone who uses your distro / your tools / your ABI variant
> knows about it.  Telling your users that it is ELFv2, without telling
> them it is not compliant, namely X Y Z are different, is a bit of a
> disservice to your users, and worse to everyone else involved.

The thing is, I've yet to see in which way the ELFv2 ABI *actually* requires VSX - I don't think compiling for 970 introduces any actual differences. There will be omissions, yes - but then the more accurate thing would be to say that a subset of ELFv2 is used, rather than it being a different ABI per se.

The ELFv2 document specifies things like passing of quadruple precision floats. Indeed, VSX is needed there, but that's not a concern if you *don't* use quadruple precision floats.

> 
> If you always use -mcpu=970 (or similar), then not very much is
> different for you most likely -- except of course there is no promise
> to the user that they can use VSX and all instructions in ISA 2.07,
> which is a very useful promise to have normally.

Yes, -mcpu=970 is used for default packages. *However*, it is possible that the user compiles their own packages with -mcpu=power9 or something similar, and then it'll be possible to utilize VSX and all, and it should still work with the existing userland. When speaking of ABI, what matters is... well, the binary interface, which is the same - so I believe this is still ELFv2. A subset is always compliant with the whole.

That's why I'm worried when you speak of introducing a new ABI. As it is, we can benefit from having the compiler being generally the same (-mabi=elfv2 producing correct results even for 970) and retaining interoperability when people compile their own code for modern targets that cover the ELFv2 ABI as a whole. As I said, it's perfectly possible for somebody to run BE Void on their POWER9 machine, then compile their software for POWER9, and still have it work with the system packages built for 970 baseline. Pretty sure glibc will still provide optimized stuff (e.g. memcpy and so on) for the modern targets based on runtime detection, too.

So the "differences" in our case come down to "This is ELFv2, except you can't strictly assume that all features are present. In general that means no quad precision floating point for you if you want things to run on 970, since you don't have VSX regs"

> 
> > It currently works out of box - there are no changes required in glibc, and nearly the entire userland builds and works (about ~11500 out of ~12000 software packages, those that don't work either don't work on ppc64le either, or have issues related to endianness, or some other unrelated reason).
> 
> Very nice!
> 
> > I'd like to eventually get this into a state where I don't have to worry about glibc arbitrarily breaking it - which means it would be necessary to stabilize it upstream. While I can probably maintain a downstream patchset when it comes to it, I'd much prefer if I didn't have to - but this sounds like an official ELFv2 glibc BE port would be impossible unless the VSX requirement (and thus IEEE 128-bit long double and so on) was in place, which would defeat the point of the port.
> > 
> > Is there *any* way I can take that would make upstreams of all parts of the toolchain happy? I explicitly don't want to go back to ELFv1.
> 
> Oh absolutely, it sounds like things are in quite good shape already!
> It will safe a lot of grief on all sides if you make clear this is not
> "plain" ELFv2, and in what ways it differs.

See above.

> 
> Btw, if you use GCC, *please* send in testresults?  :-)

Yes, it's all gcc (we do have clang, but compiling repo packages with clang is generally frowned upon in the project, as we have vast majority of packages cross-compilable, and our cross-compiling infrastructure is gcc-centric, plus we enable certain things by default such as hardening flags that clang does not support). I'll try to remember next time I'm running tests.

> 
> > While at it, I'd like to transition to ld64 long double format, to match musl and improve software compatibility, which I feel will raise more objections from IBM side.
> 
> I have no idea what "ld64 long double" is?  Is that just IEEE DP float?
> Aka "long double is the same as double".  That is likely easier for new
> ports than "double-double", yes, even if the eventual goal should be
> IEEE QP float -- a much smoother transition.

Yes, I mean double == long double (like musl, and glibc before 2.4). I don't think there is any other viable way, since the IBM double-double format is legacy/often broken and real QP is constrained by hardware in this case. We're as much of a source distro as a binary distro, so a potential ppc64 build doesn't have to have any vector requirements at all, even if the default binary packages do.

I have a feeling that glibc would object to such port, since it means it would have to exist in parallel with a potential different ELFv2 port that does have a POWER8 minimal requirement; gcc would need a way to tell them apart, too (based on what would they be told apart? the simplest way would probably be something like, if --with-abi=elfv2 { if --with-cpu < power8 -> use glibc-novsx else use glibc-vsx } ...)

> 
> Same goes here: document it!  If your users know that the ELFv2 variant
> you give them is not *the* ELFv2, but it differs in some clear ways,
> everyone will be happier :-)

See above.

> 
> 
> Segher
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 21:43                         ` Daniel Kolesa
@ 2020-06-04 22:08                           ` Joseph Myers
  2020-06-04 22:26                             ` Daniel Kolesa
  2020-06-04 23:42                             ` Segher Boessenkool
  2020-06-04 23:35                           ` Segher Boessenkool
  1 sibling, 2 replies; 59+ messages in thread
From: Joseph Myers @ 2020-06-04 22:08 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: Segher Boessenkool, musl, Rich Felker, Michal Suchánek,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

On Thu, 4 Jun 2020, Daniel Kolesa wrote:

> The ELFv2 document specifies things like passing of quadruple precision 
> floats. Indeed, VSX is needed there, but that's not a concern if you 
> *don't* use quadruple precision floats.

My understanding is that the registers used for argument passing are all 
ones that exactly correspond to the Vector registers in earlier 
instruction set versions.  In other words, you could *in principle* 
produce an object, or a whole libm shared library, that (a) passes or 
receives _Float128 values in registers, (b) does not use any instructions 
beyond those available with -mcpu=970, (c) would work as intended whether 
executed on a 970 or on POWER8 and (d) when executed on POWER8, would 
fully interoperate with objects receiving or passing _Float128 values and 
compiled for POWER8 to use VSX instructions for that purpose.  GCC may not 
support _Float128 for older processors, but that doesn't prevent you from 
maintaining patches to add such support.  (But if you want to support 
those 64-bit processors that don't have Vector registers at all, you 
indeed can't use binary128 and interoperate with code using VSX for that 
format in POWER8.)

(Cf. how the Arm hard-float ABI variant works even on processors with 
single-precision-only VFP, because such processors still have the 
double-precision loads and stores although not double-precision 
arithmetic.  When working on that ABI support in GCC some years ago, I 
also made sure that GNU vector types corresponding to NEON vector types 
were passed consistently for the hard-float ABI whether or not any vector 
instructions were present - thus, avoiding depending on the machine modes 
for those vector types because GCC could choose a different machine mode 
depending on the instructions available.)

-- 
Joseph S. Myers
joseph@codesourcery.com

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 22:08                           ` Joseph Myers
@ 2020-06-04 22:26                             ` Daniel Kolesa
  2020-06-05  0:02                               ` Segher Boessenkool
  2020-06-04 23:42                             ` Segher Boessenkool
  1 sibling, 1 reply; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-04 22:26 UTC (permalink / raw)
  To: Joseph Myers
  Cc: Segher Boessenkool, musl, Rich Felker, Michal Suchánek,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

On Fri, Jun 5, 2020, at 00:08, Joseph Myers wrote:
> On Thu, 4 Jun 2020, Daniel Kolesa wrote:
> 
> > The ELFv2 document specifies things like passing of quadruple precision 
> > floats. Indeed, VSX is needed there, but that's not a concern if you 
> > *don't* use quadruple precision floats.
> 
> My understanding is that the registers used for argument passing are all 
> ones that exactly correspond to the Vector registers in earlier 
> instruction set versions.  In other words, you could *in principle* 
> produce an object, or a whole libm shared library, that (a) passes or 
> receives _Float128 values in registers, (b) does not use any instructions 
> beyond those available with -mcpu=970, (c) would work as intended whether 
> executed on a 970 or on POWER8 and (d) when executed on POWER8, would 
> fully interoperate with objects receiving or passing _Float128 values and 
> compiled for POWER8 to use VSX instructions for that purpose.  GCC may not 
> support _Float128 for older processors, but that doesn't prevent you from 
> maintaining patches to add such support.  (But if you want to support 
> those 64-bit processors that don't have Vector registers at all, you 
> indeed can't use binary128 and interoperate with code using VSX for that 
> format in POWER8.)

There's a potential userbase with 64-bit BE processors from Freescale/NXP that don't have any AltiVec support, I believe they are still in production - I'd like to retain support for these targets, as well as older IBM processors. The userland generally also supports that, and we've had multiple requests for support of this kind of hardware.

And while implementing it with just VMX may be possible, most hardware running this ABI wouldn't have any support for quad precision FP, and would perform better with using just double-precision.

We're not a commercial project, so we're just trying to support users within the FOSS community; I definitely wouldn't mind having this be just an ABI variant parallel to the others. Using 64-bit long doubles also has the benefit of being the same ABI as musl, which would enable things such as gcompat to work.

Either way I'll think about it some more and possibly prepare an RFC port. I'm definitely willing to put in the work and later maintenance effort if that's what it takes to make it happen.

> 
> (Cf. how the Arm hard-float ABI variant works even on processors with 
> single-precision-only VFP, because such processors still have the 
> double-precision loads and stores although not double-precision 
> arithmetic.  When working on that ABI support in GCC some years ago, I 
> also made sure that GNU vector types corresponding to NEON vector types 
> were passed consistently for the hard-float ABI whether or not any vector 
> instructions were present - thus, avoiding depending on the machine modes 
> for those vector types because GCC could choose a different machine mode 
> depending on the instructions available.)
> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 21:43                         ` Daniel Kolesa
  2020-06-04 22:08                           ` Joseph Myers
@ 2020-06-04 23:35                           ` Segher Boessenkool
  2020-06-05  2:18                             ` Daniel Kolesa
  1 sibling, 1 reply; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-04 23:35 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: musl, Rich Felker, Michal Suchánek, Joseph Myers,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

Hi!

On Thu, Jun 04, 2020 at 11:43:53PM +0200, Daniel Kolesa wrote:
> The thing is, I've yet to see in which way the ELFv2 ABI *actually* requires VSX - I don't think compiling for 970 introduces any actual differences. There will be omissions, yes - but then the more accurate thing would be to say that a subset of ELFv2 is used, rather than it being a different ABI per se.

Two big things are that binaries that someone else made are supposed to
work for you as well -- including binaries using VSX registers, or any
instructions that require ISA 2.07 (or some older ISA after 970).  This
includes DSOs (shared libraries).  So for a distribution this means that
they will not use VSX *anywhere*, or only in very specialised things.
That is a many-years setback, for people/situations where it could be
used.

> The ELFv2 document specifies things like passing of quadruple precision floats. Indeed, VSX is needed there, but that's not a concern if you *don't* use quadruple precision floats.

As Joseph says, QP float is passed in the VRs, so that works just fine
*if* you have AltiVec.  If not, you probably should pass such values in
the GPRs, or however a struct of 16 bytes is passed in whatever ABI you
use (this is a much more general problem than just BE ELFv2 ;-) )  And
then you have some kind of "partial soft float".  Fun!  (Or not...)

> > If you always use -mcpu=970 (or similar), then not very much is
> > different for you most likely -- except of course there is no promise
> > to the user that they can use VSX and all instructions in ISA 2.07,
> > which is a very useful promise to have normally.
> 
> Yes, -mcpu=970 is used for default packages. *However*, it is possible that the user compiles their own packages with -mcpu=power9 or something similar, and then it'll be possible to utilize VSX and all, and it should still work with the existing userland. When speaking of ABI, what matters is... well, the binary interface, which is the same - so I believe this is still ELFv2. A subset is always compliant with the whole.

The same calling convention will probably work, yes.  An ABI is more
than that though.


> > Btw, if you use GCC, *please* send in testresults?  :-)
> 
> Yes, it's all gcc (we do have clang, but compiling repo packages with clang is generally frowned upon in the project, as we have vast majority of packages cross-compilable, and our cross-compiling infrastructure is gcc-centric, plus we enable certain things by default such as hardening flags that clang does not support). I'll try to remember next time I'm running tests.

Thanks in advance!

> I have a feeling that glibc would object to such port, since it means it would have to exist in parallel with a potential different ELFv2 port that does have a POWER8 minimal requirement; gcc would need a way to tell them apart, too (based on what would they be told apart? the simplest way would probably be something like, if --with-abi=elfv2 { if --with-cpu < power8 -> use glibc-novsx else use glibc-vsx } ...)

The target name allows to make such distinctions: this could for example
be  powerpc64-*-linux-void  (maybe I put the distinction in the wrong
part of the name here?  The glibc people will know better, and "void" is
probably not a great name anyway).


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 22:08                           ` Joseph Myers
  2020-06-04 22:26                             ` Daniel Kolesa
@ 2020-06-04 23:42                             ` Segher Boessenkool
  1 sibling, 0 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-04 23:42 UTC (permalink / raw)
  To: Joseph Myers
  Cc: Daniel Kolesa, musl, Rich Felker, Michal Suchánek,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

Hi!

On Thu, Jun 04, 2020 at 10:08:02PM +0000, Joseph Myers wrote:
> > The ELFv2 document specifies things like passing of quadruple precision 
> > floats. Indeed, VSX is needed there, but that's not a concern if you 
> > *don't* use quadruple precision floats.
> 
> My understanding is that the registers used for argument passing are all 
> ones that exactly correspond to the Vector registers in earlier 
> instruction set versions.  In other words, you could *in principle* 
> produce an object, or a whole libm shared library,  [...]

And then there is the VRSAVE register, if your OS still uses that.
Let's hope not :-)

This is similar to what -mno-float128-hardware does (which actually
requires VSX hardware for the emulation library currently).


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 22:26                             ` Daniel Kolesa
@ 2020-06-05  0:02                               ` Segher Boessenkool
  0 siblings, 0 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-05  0:02 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: Joseph Myers, musl, Rich Felker, Michal Suchánek,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

Hi!

On Fri, Jun 05, 2020 at 12:26:22AM +0200, Daniel Kolesa wrote:
> Either way I'll think about it some more and possibly prepare an RFC port. I'm definitely willing to put in the work and later maintenance effort if that's what it takes to make it happen.

Yeah, you'll need to convince all parties involved that it will not be
more work to them then they are willing to put in.  Initial development
and ongoing work, as you say.

For GCC it is probably an easy decision (but we'll see your proposed ABI
amendments): it shouldn't be much work at all, and it even benefits us
directly (it'll fill some holes in our testing matrix).


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-04 23:35                           ` Segher Boessenkool
@ 2020-06-05  2:18                             ` Daniel Kolesa
  2020-06-05 17:27                               ` Segher Boessenkool
  0 siblings, 1 reply; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-05  2:18 UTC (permalink / raw)
  To: Segher Boessenkool
  Cc: musl, Rich Felker, Michal Suchánek, Joseph Myers,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

On Fri, Jun 5, 2020, at 01:35, Segher Boessenkool wrote:
> Hi!
> 
> On Thu, Jun 04, 2020 at 11:43:53PM +0200, Daniel Kolesa wrote:
> > The thing is, I've yet to see in which way the ELFv2 ABI *actually* requires VSX - I don't think compiling for 970 introduces any actual differences. There will be omissions, yes - but then the more accurate thing would be to say that a subset of ELFv2 is used, rather than it being a different ABI per se.
> 
> Two big things are that binaries that someone else made are supposed to
> work for you as well -- including binaries using VSX registers, or any
> instructions that require ISA 2.07 (or some older ISA after 970).  This
> includes DSOs (shared libraries).  So for a distribution this means that
> they will not use VSX *anywhere*, or only in very specialised things.
> That is a many-years setback, for people/situations where it could be
> used.

Third party precompiled stuff doesn't really need to concern us, since none really exists. It's also still an upgrade over ELFv1 regardless (I mean, the same things apply there). I'm also not really all that convinced that vectors make a huge difference in non-specialized code (autovectorization still has a way to go) and code written to use vector instructions should probably check auxval and take those paths at runtime. As for other instructions, fair enough, but from my rough testing, it doesn't make such a massive difference for average case (and where it does, one can always rebuild their thing with CFLAGS=-mcpu=power9)

> 
> > The ELFv2 document specifies things like passing of quadruple precision floats. Indeed, VSX is needed there, but that's not a concern if you *don't* use quadruple precision floats.
> 
> As Joseph says, QP float is passed in the VRs, so that works just fine
> *if* you have AltiVec.  If not, you probably should pass such values in
> the GPRs, or however a struct of 16 bytes is passed in whatever ABI you
> use (this is a much more general problem than just BE ELFv2 ;-) )  And
> then you have some kind of "partial soft float".  Fun!  (Or not...)

As I mentioned previously, I kinda want to be able to cover the same targets as ELFv1 can, which also means non-altivec hardware, so it doesn't matter that much either way... being able to cover existing stable compilers with a manageable patchset would be nice, too.

> 
> > > If you always use -mcpu=970 (or similar), then not very much is
> > > different for you most likely -- except of course there is no promise
> > > to the user that they can use VSX and all instructions in ISA 2.07,
> > > which is a very useful promise to have normally.
> > 
> > Yes, -mcpu=970 is used for default packages. *However*, it is possible that the user compiles their own packages with -mcpu=power9 or something similar, and then it'll be possible to utilize VSX and all, and it should still work with the existing userland. When speaking of ABI, what matters is... well, the binary interface, which is the same - so I believe this is still ELFv2. A subset is always compliant with the whole.
> 
> The same calling convention will probably work, yes.  An ABI is more
> than that though.
> 
> 
> > > Btw, if you use GCC, *please* send in testresults?  :-)
> > 
> > Yes, it's all gcc (we do have clang, but compiling repo packages with clang is generally frowned upon in the project, as we have vast majority of packages cross-compilable, and our cross-compiling infrastructure is gcc-centric, plus we enable certain things by default such as hardening flags that clang does not support). I'll try to remember next time I'm running tests.
> 
> Thanks in advance!
> 
> > I have a feeling that glibc would object to such port, since it means it would have to exist in parallel with a potential different ELFv2 port that does have a POWER8 minimal requirement; gcc would need a way to tell them apart, too (based on what would they be told apart? the simplest way would probably be something like, if --with-abi=elfv2 { if --with-cpu < power8 -> use glibc-novsx else use glibc-vsx } ...)
> 
> The target name allows to make such distinctions: this could for example
> be  powerpc64-*-linux-void  (maybe I put the distinction in the wrong
> part of the name here?  The glibc people will know better, and "void" is
> probably not a great name anyway).

Hm, I'm not a huge fan of putting ABI specifics in the triplet, it feels wrong - there is no precedent for it with POWER (ARM did it with EABI though), the last part should remain 'gnu' as it's still glibc; besides, gcc is compiled for exactly one target triplet, and traditionally with ppc compilers it's always been possible to target everything with just one compiler (endian, 32bit, 64bit, abi...). Detection based on CPU is probably quirky too, actually, since it'd mean different CPU types selecting different dynamic linkers.

The best way would probably be adding a new -mabi, e.g. -mabi=elfv2-novsx (just an example), which would behave exactly like -mabi=elfv2, except it'd emit some extra detection macro - that would be documented in the ABI spec (and glibc could use it to choose the correct port), allow gcc (or clang) to select a correct dynamic linker name, and probably default to a 64-bit long double format. This would also get rid of the patches used for musl which pick the default -mlong-double based on target triples :) Alternatively, since -mabi is additive, it could be a modifier for -mabi=elfv2, which would perhaps make even more sense. Clang would need similar changes. I think this way would respect existing conventions the most and be overall least invasive.

Thoughts?

> 
> 
> Segher
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-05  2:18                             ` Daniel Kolesa
@ 2020-06-05 17:27                               ` Segher Boessenkool
  2020-06-05 17:50                                 ` Rich Felker
  2020-06-05 21:59                                 ` Daniel Kolesa
  0 siblings, 2 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-05 17:27 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: musl, Rich Felker, Michal Suchánek, Joseph Myers,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

On Fri, Jun 05, 2020 at 04:18:18AM +0200, Daniel Kolesa wrote:
> On Fri, Jun 5, 2020, at 01:35, Segher Boessenkool wrote:
> > > The thing is, I've yet to see in which way the ELFv2 ABI *actually* requires VSX - I don't think compiling for 970 introduces any actual differences. There will be omissions, yes - but then the more accurate thing would be to say that a subset of ELFv2 is used, rather than it being a different ABI per se.
> > 
> > Two big things are that binaries that someone else made are supposed to
> > work for you as well -- including binaries using VSX registers, or any
> > instructions that require ISA 2.07 (or some older ISA after 970).  This
> > includes DSOs (shared libraries).  So for a distribution this means that
> > they will not use VSX *anywhere*, or only in very specialised things.
> > That is a many-years setback, for people/situations where it could be
> > used.
> 
> Third party precompiled stuff doesn't really need to concern us, since none really exists.

... Yet.  And if you claim you support ELFv2, not mentioning the ways
your implementation deviates from it, users will be unhappy.

> It's also still an upgrade over ELFv1 regardless (I mean, the same things apply there).

Yeah, in mostly minor ways, but it all adds up for sure.

> I'm also not really all that convinced that vectors make a huge difference in non-specialized code (autovectorization still has a way to go)

They do make a huge difference, depending on the application of course.
But VSX is not just vectors even: it also gives you twice as many
floating point scalars (64 now), and in newer versions of the ISA it can
be beneficially used for integer scalars even.

> and code written to use vector instructions should probably check auxval and take those paths at runtime.

No, that is exactly the point of requiring ISA 2.07.  Anything can use
ISA 2.07 (incl. VSX) without checking first, and without having a
fallback to some other implementation.  Going from ISA 2.01 to 2.07 is
more than a decade of improvements, it is not trivial at all.


> As for other instructions, fair enough, but from my rough testing, it doesn't make such a massive difference for average case

That depends on what you call the average case.  Code that is control
and memory-bound will not benefit much from *anything* :-)

> (and where it does, one can always rebuild their thing with CFLAGS=-mcpu=power9)

Yeah, but it helps quite a bit if your system (shared) libraries get all
improvements they can as well.


I'm not trying to dissuade you from not requiring VSX and 2.07 -- this
sounds like your best option, given the constraints.  I'm just saying
the cost is not trivial (even ignoring the ABI divergence).


> > The target name allows to make such distinctions: this could for example
> > be  powerpc64-*-linux-void  (maybe I put the distinction in the wrong
> > part of the name here?  The glibc people will know better, and "void" is
> > probably not a great name anyway).
> 
> Hm, I'm not a huge fan of putting ABI specifics in the triplet, it feels wrong - there is no precedent for it with POWER (ARM did it with EABI though),

Maybe look at what the various BSDs use?  We do have things like this.

> the last part should remain 'gnu' as it's still glibc; besides, gcc is compiled for exactly one target triplet, and traditionally with ppc compilers it's always been possible to target everything with just one compiler (endian, 32bit, 64bit, abi...).

This isn't completely true.

Yes, the compiler allows you to change word size, endianness, ABI, some
more things.  That does not mean you can actually build working binaries
for all resulting combinations.  As a trivial example, it will still
pick up the same libraries from the same library paths usually, and
those will spectacularly fail to work.

We are biarch for some targets, which means that both powerpc-linux
targets and powerpc64-linux targets can actually handle both of those,
with just -m32 or -m64 needed to switch which configuration is used.
But you cannot magically transparently switch to many other
configurations: for those, you just build a separate toolchain for that
specfic (variant) configuration, in the general case.

> The best way would probably be adding a new -mabi, e.g. -mabi=elfv2-novsx (just an example), which would behave exactly like -mabi=elfv2, except it'd emit some extra detection macro

Yeah, that sounds like a good idea.  Patches welcome :-)

(A separate target name is still needed, but this will make development
simpler for sure).


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-05 17:27                               ` Segher Boessenkool
@ 2020-06-05 17:50                                 ` Rich Felker
  2020-06-05 23:45                                   ` Segher Boessenkool
  2020-06-05 21:59                                 ` Daniel Kolesa
  1 sibling, 1 reply; 59+ messages in thread
From: Rich Felker @ 2020-06-05 17:50 UTC (permalink / raw)
  To: Segher Boessenkool
  Cc: Daniel Kolesa, musl, Michal Suchánek, Joseph Myers,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

On Fri, Jun 05, 2020 at 12:27:02PM -0500, Segher Boessenkool wrote:
> On Fri, Jun 05, 2020 at 04:18:18AM +0200, Daniel Kolesa wrote:
> > On Fri, Jun 5, 2020, at 01:35, Segher Boessenkool wrote:
> > > > The thing is, I've yet to see in which way the ELFv2 ABI *actually* requires VSX - I don't think compiling for 970 introduces any actual differences. There will be omissions, yes - but then the more accurate thing would be to say that a subset of ELFv2 is used, rather than it being a different ABI per se.
> > > 
> > > Two big things are that binaries that someone else made are supposed to
> > > work for you as well -- including binaries using VSX registers, or any
> > > instructions that require ISA 2.07 (or some older ISA after 970).  This
> > > includes DSOs (shared libraries).  So for a distribution this means that
> > > they will not use VSX *anywhere*, or only in very specialised things.
> > > That is a many-years setback, for people/situations where it could be
> > > used.
> > 
> > Third party precompiled stuff doesn't really need to concern us, since none really exists.
> 
> .... Yet.  And if you claim you support ELFv2, not mentioning the ways
> your implementation deviates from it, users will be unhappy.
> 
> > It's also still an upgrade over ELFv1 regardless (I mean, the same things apply there).
> 
> Yeah, in mostly minor ways, but it all adds up for sure.
> 
> > I'm also not really all that convinced that vectors make a huge difference in non-specialized code (autovectorization still has a way to go)
> 
> They do make a huge difference, depending on the application of course.
> But VSX is not just vectors even: it also gives you twice as many
> floating point scalars (64 now), and in newer versions of the ISA it can
> be beneficially used for integer scalars even.

Vectorization is useful for a lot of things, and I'm sure there are
specialized workloads that benefit from 64 scalars, but I've never
encountered a place where having more than 16 registers made a
practical difference.

The fact that there are specialized areas where this stuff matters
does not imply there aren't huge domains where it's completely
irrelevant.

> > and code written to use vector instructions should probably check
> > auxval and take those paths at runtime.
> 
> No, that is exactly the point of requiring ISA 2.07.  Anything can use
> ISA 2.07 (incl. VSX) without checking first, and without having a
> fallback to some other implementation.  Going from ISA 2.01 to 2.07 is
> more than a decade of improvements, it is not trivial at all.

This only affects code that's non-portable and PPC-specific, which a
lot of people have no interest in and don't care about. Any portable
code is going to either only use vectors via the compiler's choice to
vectorize or conditionally on being one of a set of supported targets
with a vector ISA it supports available. Anyone building for a target
that doesn't have them just gets the portable version of the code.

I think a lot of the unnecessary fighting on this topic is arising
from differences of opinion over what an ABI entails. I would call
what you're talking about a "platform" and more of a platform-specific
*API* than an ABI -- it's about guarantees of interfaces available to
the programmer, not implementation details of linkage.

Rich

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-05 17:27                               ` Segher Boessenkool
  2020-06-05 17:50                                 ` Rich Felker
@ 2020-06-05 21:59                                 ` Daniel Kolesa
  2020-06-06  0:12                                   ` Segher Boessenkool
  1 sibling, 1 reply; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-05 21:59 UTC (permalink / raw)
  To: Segher Boessenkool, musl
  Cc: Rich Felker, Michal Suchánek, Joseph Myers, libc-alpha,
	eery, Will Springer, Palmer Dabbelt via binutils, via libc-dev,
	linuxppc-dev

On Fri, Jun 5, 2020, at 19:27, Segher Boessenkool wrote:
> On Fri, Jun 05, 2020 at 04:18:18AM +0200, Daniel Kolesa wrote:
> > On Fri, Jun 5, 2020, at 01:35, Segher Boessenkool wrote:
> > > > The thing is, I've yet to see in which way the ELFv2 ABI *actually* requires VSX - I don't think compiling for 970 introduces any actual differences. There will be omissions, yes - but then the more accurate thing would be to say that a subset of ELFv2 is used, rather than it being a different ABI per se.
> > > 
> > > Two big things are that binaries that someone else made are supposed to
> > > work for you as well -- including binaries using VSX registers, or any
> > > instructions that require ISA 2.07 (or some older ISA after 970).  This
> > > includes DSOs (shared libraries).  So for a distribution this means that
> > > they will not use VSX *anywhere*, or only in very specialised things.
> > > That is a many-years setback, for people/situations where it could be
> > > used.
> > 
> > Third party precompiled stuff doesn't really need to concern us, since none really exists.
> 
> ... Yet.  And if you claim you support ELFv2, not mentioning the ways
> your implementation deviates from it, users will be unhappy.

Will they? The system explicitly mentions that the minimum target in the binary packages is 970. Users can't be expecting features that the hardware we support doesn't support :) Also, we're not the only ones that do this - there's musl of course, but there's also the BSDs. FreeBSD 13 uses ELFv2, and supports all the old hardware including processors without VMX, let alone VSX. OpenBSD is likely to take the same path. But I'm not opposed to making this explicit, if that's what it takes to make other people happy with it.

> 
> > It's also still an upgrade over ELFv1 regardless (I mean, the same things apply there).
> 
> Yeah, in mostly minor ways, but it all adds up for sure.

It's made my life simpler on numerous occasions, and allowed us to bring in software that'd otherwise take significant patching (no software is complete until it has its own assembly implementation of coroutines or something like that :P), or would be outright impossible without a lot of work (e.g. the lld linker).

> 
> > I'm also not really all that convinced that vectors make a huge difference in non-specialized code (autovectorization still has a way to go)
> 
> They do make a huge difference, depending on the application of course.

I've yet to see non-specialized things without explicit VSX impls where the difference is more than 5%, usually it's less.

> But VSX is not just vectors even: it also gives you twice as many
> floating point scalars (64 now), and in newer versions of the ISA it can
> be beneficially used for integer scalars even.
> 
> > and code written to use vector instructions should probably check auxval and take those paths at runtime.
> 
> No, that is exactly the point of requiring ISA 2.07.  Anything can use
> ISA 2.07 (incl. VSX) without checking first, and without having a
> fallback to some other implementation.  Going from ISA 2.01 to 2.07 is
> more than a decade of improvements, it is not trivial at all.

Portable code will need runtime checks anyway if they target big endian systems, especially existing ones, since modern vector instructions can be used even in legacy ABI, where you get no such guarantee. Sure, it's nice to not have to worry about it, but once everything is considered, going with a more modern ABI is still a big net gain for us, even if the guarantees are not all there :)

> 
> 
> > As for other instructions, fair enough, but from my rough testing, it doesn't make such a massive difference for average case
> 
> That depends on what you call the average case.  Code that is control
> and memory-bound will not benefit much from *anything* :-)

Average case is, quite literally, an average case - i.e. the average of all the software packages shipped in a distro :)

> 
> > (and where it does, one can always rebuild their thing with CFLAGS=-mcpu=power9)
> 
> Yeah, but it helps quite a bit if your system (shared) libraries get all
> improvements they can as well.

Well, glibc will still benefit automatically to a degree even if not built for a modern baseline, since it has runtime checks in place already; as for other things... well, at least for Void, I already mentioned before we're as much of a source distro as a binary one - people can easily rebuild things that bottleneck them, with modern CFLAGS, and still have things be interoperable.

> 
> 
> I'm not trying to dissuade you from not requiring VSX and 2.07 -- this
> sounds like your best option, given the constraints.  I'm just saying
> the cost is not trivial (even ignoring the ABI divergence).

Of course the cost is there - it's just not something I can do anything about. I generally recommend that people who can run LE should run LE. We're a bi-endian distribution, so there is a complete, fully functional, ISA-2.07-baseline ppc64le variant (in fact, it's our best-supported port, with greatest repo coverage and testing), as well as the 970-targeting ppc64 variant.

The 970-targeting ppc64 variant is still capable of running on modern hardware, as there are people who wish to run big endian even on their modern machines, and I don't want to take that choice away from them (and since kernel 4.20, you can even have one generic kernel capable of booting POWER4-POWER9, previously separate kernels were needed for <= POWER6 and >= POWER7). Those people can either deal with their computers running a bit slower, or rebuild the packages that specifically bottleneck them with modern CFLAGS.

> 
> 
> > > The target name allows to make such distinctions: this could for example
> > > be  powerpc64-*-linux-void  (maybe I put the distinction in the wrong
> > > part of the name here?  The glibc people will know better, and "void" is
> > > probably not a great name anyway).
> > 
> > Hm, I'm not a huge fan of putting ABI specifics in the triplet, it feels wrong - there is no precedent for it with POWER (ARM did it with EABI though),
> 
> Maybe look at what the various BSDs use?  We do have things like this.

Briefly, the FreeBSD 13 powerpc64 target triple had an -elfv2 suffix, this was done in clang, but I noticed it was reverted in the end and replaced with checks not based on triples. I believe there is no ABI in BSD powerpc triples right now.

> 
> > the last part should remain 'gnu' as it's still glibc; besides, gcc is compiled for exactly one target triplet, and traditionally with ppc compilers it's always been possible to target everything with just one compiler (endian, 32bit, 64bit, abi...).
> 
> This isn't completely true.
> 
> Yes, the compiler allows you to change word size, endianness, ABI, some
> more things.  That does not mean you can actually build working binaries
> for all resulting combinations.  As a trivial example, it will still
> pick up the same libraries from the same library paths usually, and
> those will spectacularly fail to work.

I know, I meant mostly from purely compiler perspective. It's good enough for bare-metal (in that way it matters for stuff like GRUB etc. where you still need to build a 32-bit big-endian ELF, etc)

> 
> We are biarch for some targets, which means that both powerpc-linux
> targets and powerpc64-linux targets can actually handle both of those,
> with just -m32 or -m64 needed to switch which configuration is used.
> But you cannot magically transparently switch to many other
> configurations: for those, you just build a separate toolchain for that
> specfic (variant) configuration, in the general case.

I know about the biarch case as well, and there is also multilib, as an even more elaborate form of that. That's not directly related to what I originally said, though

> 
> > The best way would probably be adding a new -mabi, e.g. -mabi=elfv2-novsx (just an example), which would behave exactly like -mabi=elfv2, except it'd emit some extra detection macro
> 
> Yeah, that sounds like a good idea.  Patches welcome :-)
> 
> (A separate target name is still needed, but this will make development
> simpler for sure).

I don't think a separate target is strictly necessary - it can be done, sure, but with an ABI switch it's more just informative than anything else.

> 
> 
> Segher
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-05 17:50                                 ` Rich Felker
@ 2020-06-05 23:45                                   ` Segher Boessenkool
  0 siblings, 0 replies; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-05 23:45 UTC (permalink / raw)
  To: Rich Felker
  Cc: Daniel Kolesa, musl, Michal Suchánek, Joseph Myers,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

Hi!

On Fri, Jun 05, 2020 at 01:50:46PM -0400, Rich Felker wrote:
> On Fri, Jun 05, 2020 at 12:27:02PM -0500, Segher Boessenkool wrote:
> > > I'm also not really all that convinced that vectors make a huge difference in non-specialized code (autovectorization still has a way to go)
> > 
> > They do make a huge difference, depending on the application of course.
> > But VSX is not just vectors even: it also gives you twice as many
> > floating point scalars (64 now), and in newer versions of the ISA it can
> > be beneficially used for integer scalars even.
> 
> Vectorization is useful for a lot of things, and I'm sure there are
> specialized workloads that benefit from 64 scalars, but I've never
> encountered a place where having more than 16 registers made a
> practical difference.

20 years ago 32 FP registers was already often a limitation, making FFT
and similar kernels almost twice slower than they could otherwise be.
Things are only *worse* with short vectors, not better.  In general with
floating point data you need more registers (because you have more state
to look at concurrently) than with integer data.

*Of course* having 64 floating point registers does not matter if your
whole program only ever uses three floating point values, total, let
alone concurrently.

> The fact that there are specialized areas where this stuff matters
> does not imply there aren't huge domains where it's completely
> irrelevant.

There are very few domains where ISA 2.07 does not have significant
advantages over ISA 2.01.  That is Power8 vs. Power4.

> > No, that is exactly the point of requiring ISA 2.07.  Anything can use
> > ISA 2.07 (incl. VSX) without checking first, and without having a
> > fallback to some other implementation.  Going from ISA 2.01 to 2.07 is
> > more than a decade of improvements, it is not trivial at all.
> 
> This only affects code that's non-portable and PPC-specific, which a

No, it does not.  It is not only about vector registers, either.

> I think a lot of the unnecessary fighting on this topic is arising
> from differences of opinion over what an ABI entails. I would call
> what you're talking about a "platform" and more of a platform-specific
> *API* than an ABI -- it's about guarantees of interfaces available to
> the programmer, not implementation details of linkage.

No, this is very much about the ABI.  The B stands for Binary.  Which
is what this is about.


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-05-30 22:17   ` Will Springer
@ 2020-06-05 23:54     ` Will Springer
  2020-06-12  5:13       ` Christophe Leroy
  0 siblings, 1 reply; 59+ messages in thread
From: Will Springer @ 2020-06-05 23:54 UTC (permalink / raw)
  To: linuxppc-dev, Christophe Leroy
  Cc: libc-alpha, eery, daniel, musl, binutils, libc-dev

On Saturday, May 30, 2020 3:17:24 PM PDT Will Springer wrote:
> On Saturday, May 30, 2020 8:37:43 AM PDT Christophe Leroy wrote:
> > There is a series at
> > https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231
> > to switch powerpc to the Generic C VDSO.
> > 
> > Can you try and see whether it fixes your issue ?
> > 
> > Christophe
> 
> Sure thing, I spotted that after making the initial post. Will report
> back with results.
> 
> Will [she/her]

Sorry for the wait, I just sat down to work on this again yesterday.

Tested this series on top of stable/linux-5.7.y (5.7.0 at the time of 
writing), plus the one-line signal handler patch. Had to rewind to the 
state of powerpc/merge at the time of the mail before the patch would 
apply, then cherry-picked to 5.6 until I realized the patchset used some 
functionality that didn't land until 5.7, so I moved it there.

Good news is that `date` now works correctly with the vdso call in 32-bit 
LE. Bad news is it seems to have broken things on the 64-bit side—in my 
testing, Void kicks off runit but hangs after starting eudev, and in a 
Debian Stretch system, systemd doesn't get to the point of printing 
anything whatsoever. (I had to `init=/bin/sh` to confirm the date worked 
in ppcle, although in ppc64le running `date` also hung the system when it 
made the vdso call...) Not sure how to approach debugging that, so I'd 
appreciate any pointers.

Will [she/her]




^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-05 21:59                                 ` Daniel Kolesa
@ 2020-06-06  0:12                                   ` Segher Boessenkool
  2020-06-06  2:13                                     ` Daniel Kolesa
  0 siblings, 1 reply; 59+ messages in thread
From: Segher Boessenkool @ 2020-06-06  0:12 UTC (permalink / raw)
  To: Daniel Kolesa
  Cc: musl, Rich Felker, Michal Suchánek, Joseph Myers,
	libc-alpha, eery, Will Springer, Palmer Dabbelt via binutils,
	via libc-dev, linuxppc-dev

On Fri, Jun 05, 2020 at 11:59:32PM +0200, Daniel Kolesa wrote:
> On Fri, Jun 5, 2020, at 19:27, Segher Boessenkool wrote:
> > > Third party precompiled stuff doesn't really need to concern us, since none really exists.
> > 
> > ... Yet.  And if you claim you support ELFv2, not mentioning the ways
> > your implementation deviates from it, users will be unhappy.
> 
> Will they?

Yes; not only *your* users, but also users of the "actual" ELFv2 (for
BE), if anything starts using that: the ABI is defined, and at least at
some point it worked, it is not unreasonable to think someone might
want to start using it).

Just don't pretend X and Y are the same if they are not, people do not
like to be misled.  Just be clear upfront what the differences are, and
everyone will be happy.  Confusing people and wasting their time won't
make you very popular though ;-)

> The system explicitly mentions that the minimum target in the binary packages is 970. Users can't be expecting features that the hardware we support doesn't support :)

But you are not the only user of ELFv2.

> Also, we're not the only ones that do this - there's musl of course, but there's also the BSDs. FreeBSD 13 uses ELFv2, and supports all the old hardware including processors without VMX, let alone VSX. OpenBSD is likely to take the same path. But I'm not opposed to making this explicit, if that's what it takes to make other people happy with it.

Yeah, please do :-)

> > > It's also still an upgrade over ELFv1 regardless (I mean, the same things apply there).
> > 
> > Yeah, in mostly minor ways, but it all adds up for sure.
> 
> It's made my life simpler on numerous occasions,

Great to hear that!

> and allowed us to bring in software that'd otherwise take significant patching (no software is complete until it has its own assembly implementation of coroutines or something like that :P),

.. but does it have a mail client?

> > That depends on what you call the average case.  Code that is control
> > and memory-bound will not benefit much from *anything* :-)
> 
> Average case is, quite literally, an average case - i.e. the average of all the software packages shipped in a distro :)

Ah, mostly boring stuff :-)

> > Yeah, but it helps quite a bit if your system (shared) libraries get all
> > improvements they can as well.
> 
> Well, glibc will still benefit automatically to a degree even if not built for a modern baseline, since it has runtime checks in place already; as for other things... well, at least for Void, I already mentioned before we're as much of a source distro as a binary one - people can easily rebuild things that bottleneck them, with modern CFLAGS, and still have things be interoperable.

Yeah, good point there.

> > I'm not trying to dissuade you from not requiring VSX and 2.07 -- this
> > sounds like your best option, given the constraints.  I'm just saying
> > the cost is not trivial (even ignoring the ABI divergence).
> 
> Of course the cost is there - it's just not something I can do anything about. I generally recommend that people who can run LE should run LE. We're a bi-endian distribution, so there is a complete, fully functional, ISA-2.07-baseline ppc64le variant (in fact, it's our best-supported port, with greatest repo coverage and testing), as well as the 970-targeting ppc64 variant.

Ah, so your BE target is mostly for legacy hardware?  That took a while
to sink in, sorry!  :-)

> I know about the biarch case as well, and there is also multilib, as an even more elaborate form of that. That's not directly related to what I originally said, though

Biarch is why -m32 and -m64 can work at all (for building user binaries).
My point is that this does *not* work for most finer ABI (or OS) -related
points -- you really do need a toolchain built specifically for that
config.


Segher

^ permalink raw reply	[flat|nested] 59+ messages in thread

* Re: [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-06  0:12                                   ` Segher Boessenkool
@ 2020-06-06  2:13                                     ` Daniel Kolesa
  0 siblings, 0 replies; 59+ messages in thread
From: Daniel Kolesa @ 2020-06-06  2:13 UTC (permalink / raw)
  To: Segher Boessenkool, musl
  Cc: Rich Felker, Michal Suchánek, Joseph Myers, libc-alpha,
	eery, Will Springer, Palmer Dabbelt via binutils, via libc-dev,
	linuxppc-dev

On Sat, Jun 6, 2020, at 02:12, Segher Boessenkool wrote:
> On Fri, Jun 05, 2020 at 11:59:32PM +0200, Daniel Kolesa wrote:
> > On Fri, Jun 5, 2020, at 19:27, Segher Boessenkool wrote:
> > > > Third party precompiled stuff doesn't really need to concern us, since none really exists.
> > > 
> > > ... Yet.  And if you claim you support ELFv2, not mentioning the ways
> > > your implementation deviates from it, users will be unhappy.
> > 
> > Will they?
> 
> Yes; not only *your* users, but also users of the "actual" ELFv2 (for
> BE), if anything starts using that: the ABI is defined, and at least at
> some point it worked, it is not unreasonable to think someone might
> want to start using it).

I guess you mean like, if third party software starts shipping builds using ELFv2 and people suddenly find these things don't work because $reasons. The ABI itself wouldn't cause issues here as at binary interface level we're still compatible, but problems may actually arise in case glibc has a modern HW targeting ELFv2 port with IEEE quad float at some point. (not to mention that would be 100% incompatible at binary level because symvers, so i want to resolve this until then either way)

> 
> Just don't pretend X and Y are the same if they are not, people do not
> like to be misled.  Just be clear upfront what the differences are, and
> everyone will be happy.  Confusing people and wasting their time won't
> make you very popular though ;-)
> 
> > The system explicitly mentions that the minimum target in the binary packages is 970. Users can't be expecting features that the hardware we support doesn't support :)
> 
> But you are not the only user of ELFv2.

Sure, the point was more like "ELFv2 + 970 minimum", both of which are explicitly mentioned, cannot really mean anything else than what it already does.

> 
> > Also, we're not the only ones that do this - there's musl of course, but there's also the BSDs. FreeBSD 13 uses ELFv2, and supports all the old hardware including processors without VMX, let alone VSX. OpenBSD is likely to take the same path. But I'm not opposed to making this explicit, if that's what it takes to make other people happy with it.
> 
> Yeah, please do :-)
> 
> > > > It's also still an upgrade over ELFv1 regardless (I mean, the same things apply there).
> > > 
> > > Yeah, in mostly minor ways, but it all adds up for sure.
> > 
> > It's made my life simpler on numerous occasions,
> 
> Great to hear that!
> 
> > and allowed us to bring in software that'd otherwise take significant patching (no software is complete until it has its own assembly implementation of coroutines or something like that :P),
> 
> .. but does it have a mail client?

Probably some ad-hoc, informally specified, bug-ridden implementations of half of Common Lisp as well...

> 
> > > That depends on what you call the average case.  Code that is control
> > > and memory-bound will not benefit much from *anything* :-)
> > 
> > Average case is, quite literally, an average case - i.e. the average of all the software packages shipped in a distro :)
> 
> Ah, mostly boring stuff :-)
> 
> > > Yeah, but it helps quite a bit if your system (shared) libraries get all
> > > improvements they can as well.
> > 
> > Well, glibc will still benefit automatically to a degree even if not built for a modern baseline, since it has runtime checks in place already; as for other things... well, at least for Void, I already mentioned before we're as much of a source distro as a binary one - people can easily rebuild things that bottleneck them, with modern CFLAGS, and still have things be interoperable.
> 
> Yeah, good point there.
> 
> > > I'm not trying to dissuade you from not requiring VSX and 2.07 -- this
> > > sounds like your best option, given the constraints.  I'm just saying
> > > the cost is not trivial (even ignoring the ABI divergence).
> > 
> > Of course the cost is there - it's just not something I can do anything about. I generally recommend that people who can run LE should run LE. We're a bi-endian distribution, so there is a complete, fully functional, ISA-2.07-baseline ppc64le variant (in fact, it's our best-supported port, with greatest repo coverage and testing), as well as the 970-targeting ppc64 variant.
> 
> Ah, so your BE target is mostly for legacy hardware?  That took a while
> to sink in, sorry!  :-)

That's what the average user on BE is expected to be. I'm not opinionated enough to force BE on an average modern hardware user, there's just too many drawbacks nowadays, some of them inherent, like limited graphics hardware support (the choice of HW is limited, modern GPUs don't work at all, older GPUs are limited to OpenGL 3.2 even if they support newer API, and even if newer GPUs *were* working, they'd be crippled performance-wise as modern GPUs are always LE these days - as a gamedev person I need my graphics stuff working, and I expect my RX 5700 XT to be more than a paperweight), nevertheless, if someone is opinionated enough to use BE despite everything when they don't have to, they're welcome and so are their patches.

> 
> > I know about the biarch case as well, and there is also multilib, as an even more elaborate form of that. That's not directly related to what I originally said, though
> 
> Biarch is why -m32 and -m64 can work at all (for building user binaries).
> My point is that this does *not* work for most finer ABI (or OS) -related
> points -- you really do need a toolchain built specifically for that
> config.

Indeed. But target triples don't give you the necessary distinctions. This is why I find that the "we're an everything-compiler" approach of Clang that's supposedly meant to simplify things to the user actually complicates them; with gcc, you can configure your toolchain for the target env when building it, with clang you are not given this option, all you can do is patch its source code, as the ideas of targeting everything and of configuring the compiler details for a target are inherently in conflict. And when cross-building the real complexity for the user lies in setting up the sysroot anyway, at least for complex things with actual dependencies.

> 
> 
> Segher
>

Daniel

^ permalink raw reply	[flat|nested] 59+ messages in thread

* [musl] Re: ppc64le and 32-bit LE userland compatibility
  2020-06-05 23:54     ` Will Springer
@ 2020-06-12  5:13       ` Christophe Leroy
  0 siblings, 0 replies; 59+ messages in thread
From: Christophe Leroy @ 2020-06-12  5:13 UTC (permalink / raw)
  To: Will Springer, linuxppc-dev
  Cc: libc-alpha, eery, daniel, musl, binutils, libc-dev



Le 06/06/2020 à 01:54, Will Springer a écrit :
> On Saturday, May 30, 2020 3:17:24 PM PDT Will Springer wrote:
>> On Saturday, May 30, 2020 8:37:43 AM PDT Christophe Leroy wrote:
>>> There is a series at
>>> https://patchwork.ozlabs.org/project/linuxppc-dev/list/?series=173231
>>> to switch powerpc to the Generic C VDSO.
>>>
>>> Can you try and see whether it fixes your issue ?
>>>
>>> Christophe
>>
>> Sure thing, I spotted that after making the initial post. Will report
>> back with results.
>>
>> Will [she/her]
> 
> Sorry for the wait, I just sat down to work on this again yesterday.
> 
> Tested this series on top of stable/linux-5.7.y (5.7.0 at the time of
> writing), plus the one-line signal handler patch. Had to rewind to the
> state of powerpc/merge at the time of the mail before the patch would
> apply, then cherry-picked to 5.6 until I realized the patchset used some
> functionality that didn't land until 5.7, so I moved it there.
> 
> Good news is that `date` now works correctly with the vdso call in 32-bit
> LE. Bad news is it seems to have broken things on the 64-bit side—in my
> testing, Void kicks off runit but hangs after starting eudev, and in a
> Debian Stretch system, systemd doesn't get to the point of printing
> anything whatsoever. (I had to `init=/bin/sh` to confirm the date worked
> in ppcle, although in ppc64le running `date` also hung the system when it
> made the vdso call...) Not sure how to approach debugging that, so I'd
> appreciate any pointers.
> 

Does it breaks only ppc64le vdso or also ppc64 (be) vdso ?

I never had a chance to run any test on ppc64 as I only have a kernel 
cross compiler.

Would you have a chance to build and run vdsotest from 
https://github.com/nathanlynch/vdsotest ?

Thanks
Christophe

^ permalink raw reply	[flat|nested] 59+ messages in thread

end of thread, back to index

Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-29 19:03 [musl] ppc64le and 32-bit LE userland compatibility Will Springer
2020-05-29 19:24 ` Rich Felker
2020-05-30 22:56   ` Will Springer
2020-05-30 15:37 ` [musl] " Christophe Leroy
2020-05-30 22:17   ` Will Springer
2020-06-05 23:54     ` Will Springer
2020-06-12  5:13       ` Christophe Leroy
2020-05-30 19:22 ` Segher Boessenkool
2020-05-31  0:57   ` Will Springer
2020-05-31 20:42     ` Segher Boessenkool
2020-05-31 22:29       ` Daniel Kolesa
2020-06-02  1:36         ` Segher Boessenkool
2020-06-01 21:28 ` Joseph Myers
2020-06-01 21:36   ` Rich Felker
2020-06-01 23:26   ` Daniel Kolesa
2020-06-01 23:45     ` Joseph Myers
2020-06-01 23:55       ` Joseph Myers
2020-06-02  0:13         ` Daniel Kolesa
2020-06-02  0:11       ` Daniel Kolesa
2020-06-02 13:40         ` Joseph Myers
2020-06-02 14:23           ` Michal Suchánek
2020-06-02 15:13             ` Daniel Kolesa
2020-06-02 15:27               ` Michal Suchánek
2020-06-02 15:40                 ` Daniel Kolesa
2020-06-02 15:56                   ` Michal Suchánek
2020-06-04 17:20                 ` Segher Boessenkool
2020-06-04 17:12               ` Segher Boessenkool
2020-06-04 17:18                 ` Rich Felker
2020-06-04 17:33                   ` Segher Boessenkool
2020-06-04 17:46                     ` Rich Felker
2020-06-04 19:00                       ` David Edelsohn
2020-06-04 19:37                         ` Rich Felker
2020-06-04 20:39                     ` Daniel Kolesa
2020-06-04 21:10                       ` Segher Boessenkool
2020-06-04 21:43                         ` Daniel Kolesa
2020-06-04 22:08                           ` Joseph Myers
2020-06-04 22:26                             ` Daniel Kolesa
2020-06-05  0:02                               ` Segher Boessenkool
2020-06-04 23:42                             ` Segher Boessenkool
2020-06-04 23:35                           ` Segher Boessenkool
2020-06-05  2:18                             ` Daniel Kolesa
2020-06-05 17:27                               ` Segher Boessenkool
2020-06-05 17:50                                 ` Rich Felker
2020-06-05 23:45                                   ` Segher Boessenkool
2020-06-05 21:59                                 ` Daniel Kolesa
2020-06-06  0:12                                   ` Segher Boessenkool
2020-06-06  2:13                                     ` Daniel Kolesa
2020-06-02 14:52           ` Daniel Kolesa
2020-06-02  2:12       ` Segher Boessenkool
2020-06-02  2:17         ` Daniel Kolesa
2020-06-02 13:50         ` Joseph Myers
2020-06-02 17:47           ` Segher Boessenkool
2020-06-02  1:58     ` Segher Boessenkool
2020-06-02  2:09       ` Jeffrey Walton
2020-06-02  2:12       ` Daniel Kolesa
2020-06-02  2:36         ` Segher Boessenkool
2020-06-02  2:55           ` Daniel Kolesa
2020-06-02  1:42   ` Segher Boessenkool
2020-06-02  2:03     ` Daniel Kolesa

mailing list of musl libc

Archives are clonable: git clone --mirror http://inbox.vuxu.org/musl

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.archive.musl


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git