[musl] FDPIC on sh4?

[musl] FDPIC on sh4?

[Rob Landley]

Is there any easy way to build FDPIC support for sh4 with-mmu? In theory ARM can
do it, so I tried editing the kconfig BINFMT_ELF_FDPIC dependencies in
fs/Kconfig.binfmt to move "SUPERH" out of the !MMU list and put it next to ARM,
switched on the FDPIC loader, and the build broke with:

fs/binfmt_elf_fdpic.o: in function `load_elf_fdpic_binary':
binfmt_elf_fdpic.c:(.text+0x1b44): undefined reference to
`elf_fdpic_arch_lay_out_mm'

Backstory: I want to test nommu under qemu, and my existing tests are all using
qemu-system-$TARGET builds with musl-libc toolchains built by musl-cross-make.

I have a nommu test environment on real (FPGA) hardware, using an sh2 fdpic
toolchain built from musl-cross-make using gcc 9.4.0, but that's not part of my
normal "cursor up and hit enter" fully automated regression testing that builds
and launches qemu instances with a prepared filesystem that launches tests from
the init script and does "expect" style processing on the serial console
connected to qemu's stdin/stdout.

Testing on the FPGA board requires repeatedly removing and inserting sd cards,
so I don't do it nearly as often, and right now that's my ONLY nommu test
environment.

I want to get a qemu-system-something running nommu. Since musl-libc only
supports fdpic, this limits my choices to the intersection of Linux's FDPIC support:

$ grep FDPIC -m1 -A3 fs/Kconfig.binfmt
config BINFMT_ELF_FDPIC
	bool "Kernel support for FDPIC ELF binaries"
	default y if !BINFMT_ELF
	depends on ARM || ((M68K || RISCV || SUPERH || XTENSA) && !MMU)

And gcc's FDPIC support:

$ dirname $(grep -irl fdpic gcc/config) | sort -u
gcc/config/arm
gcc/config/bfin
gcc/config/frv
gcc/config/sh

Ever since linux threw blackfin and frv overboard to lighten the load, this
means gcc can produce fdpic output for exactly two targets that Linux supports:
arm and superh. (No, riscv is not joining them. I emailed about
https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/ZjYUJswknQ4/m/QnTEfur4BwAJ
yesterday and was told "Sadly that project didn't go beyond the ABI design
phase." so riscv-fdpic is _not_ currently in development.)

In theory arm can use the FDPIC loader on a with-mmu kernel, but the arm
configure doesn't have a way to combine "musl" with the magic
"uclinuxfdpiceabidoodahdoodah" blob because their matches keep looking like:
"arm*-*-uclinuxfdpiceabi" and "*-linux-musl*" which aren't compatible. (If that
first one didn't have the extra dash...) Alas arm-unknown-musl-uclinuxfdpiceabi
makes binutils go "checking target system type... Invalid configuration
`armv7m-linux-musl-uclinuxfdpiceabi': machine `armv7m-linux-musl' not recognized".

So I was thinking "maybe I can build an sh4 kernel with an sh4 compiler, and
build a userspace with the sh2eb compiler" (this is unlikely to work because
every sh4 system I've ever used is little endian and the j-core guys
inexplicably chose big endian, but I can burn that bridge when I come to it...)
But that's how I hit the "enabling FDPIC on with-mmu only works with arm,
nothing else" issue above. And that's incompatible with selecting musl support.

Rob

PS. the sh2 fdpic toolchain in musl-cross-make doesn't reproduce with the newest
"supported" gcc (11.2.0) because:

In file included from libstdc++-v3/libsupc++/eh_call.cc:32:
libstdc++-v3/../libgcc/unwind-pe.h: In function 'const unsigned char*
read_encoded_value_with_base(unsigned char, _Unwind_Ptr, const unsigned char*,
_Unwind_Ptr*)':
libstdc++-v3/../libgcc/unwind-pe.h:270:25: error: '_Unwind_gnu_Find_got' was not
declared in this scope
  270 |               result += _Unwind_gnu_Find_got ((_Unwind_Ptr) u);

Which has been broken ever since musl-cross-make's newest last commit almost 2
years ago.

Re: [musl] FDPIC on sh4?

[Rich Felker]

On Thu, Feb 15, 2024 at 04:31:00AM -0600, Rob Landley wrote:
> Is there any easy way to build FDPIC support for sh4 with-mmu? In theory ARM can

On the userspace toolchain and musl side, yes, I see no reason you
shouldn't be able to build musl for sh4 with fdpic ABI or build a
toolchain for sh4 that defaults to fdpic and has fdpic target-libs. I
just tested passing -mfdpic to sh4-linux-musl-gcc and it seems to
produce correct fdpic code.

On the kernel side, I'm not sure, but the normal ELF loader should be
able to load fdpic binaries on a system with mmu. It will not float
the data segment separately from text, but doesn't need to because it
has an mmu. If this is no longer working it's a kernel regression;
that's how I always tested sh2eb fdpic binaries on qemu-system-sh4eb.

> do it, so I tried editing the kconfig BINFMT_ELF_FDPIC dependencies in
> fs/Kconfig.binfmt to move "SUPERH" out of the !MMU list and put it next to ARM,
> switched on the FDPIC loader, and the build broke with:
> 
> fs/binfmt_elf_fdpic.o: in function `load_elf_fdpic_binary':
> binfmt_elf_fdpic.c:(.text+0x1b44): undefined reference to
> `elf_fdpic_arch_lay_out_mm'

It looks like there's an arch-provided function that's conditional on
!MMU in arch/sh but that, now that fdpic loader is intended to be
supported on mmu-ful systems, should be changed to be conditional on
fdpic support (or maybe even unconditional if fdpic can be loaded as a
module). Just look for where it's defined in arch/sh. If it's in a
nommu-specific file it might need to be moved somewhere more
appropriate, or an mmu-ful variant may need to be written and placed
somewhere more appropriate.

> Backstory: I want to test nommu under qemu, and my existing tests are all using
> qemu-system-$TARGET builds with musl-libc toolchains built by musl-cross-make.
> 
> I have a nommu test environment on real (FPGA) hardware, using an sh2 fdpic
> toolchain built from musl-cross-make using gcc 9.4.0, but that's not part of my
> normal "cursor up and hit enter" fully automated regression testing that builds
> and launches qemu instances with a prepared filesystem that launches tests from
> the init script and does "expect" style processing on the serial console
> connected to qemu's stdin/stdout.
> 
> Testing on the FPGA board requires repeatedly removing and inserting sd cards,
> so I don't do it nearly as often, and right now that's my ONLY nommu test
> environment.
> 
> I want to get a qemu-system-something running nommu. Since musl-libc only
> supports fdpic, this limits my choices to the intersection of Linux's FDPIC support:

musl also supports non-fdpic ELF just fine on nommu, but it has to be
fully position independent (PIE) and of course does not get shareable
text. But this is usable for testing.

> $ grep FDPIC -m1 -A3 fs/Kconfig.binfmt
> config BINFMT_ELF_FDPIC
> 	bool "Kernel support for FDPIC ELF binaries"
> 	default y if !BINFMT_ELF
> 	depends on ARM || ((M68K || RISCV || SUPERH || XTENSA) && !MMU)
> 
> And gcc's FDPIC support:
> 
> $ dirname $(grep -irl fdpic gcc/config) | sort -u
> gcc/config/arm
> gcc/config/bfin
> gcc/config/frv
> gcc/config/sh
> 
> Ever since linux threw blackfin and frv overboard to lighten the load, this
> means gcc can produce fdpic output for exactly two targets that Linux supports:
> arm and superh. (No, riscv is not joining them. I emailed about
> https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/ZjYUJswknQ4/m/QnTEfur4BwAJ
> yesterday and was told "Sadly that project didn't go beyond the ABI design
> phase." so riscv-fdpic is _not_ currently in development.)
> 
> In theory arm can use the FDPIC loader on a with-mmu kernel, but the arm
> configure doesn't have a way to combine "musl" with the magic
> "uclinuxfdpiceabidoodahdoodah" blob because their matches keep looking like:
> "arm*-*-uclinuxfdpiceabi" and "*-linux-musl*" which aren't compatible. (If that
> first one didn't have the extra dash...) Alas arm-unknown-musl-uclinuxfdpiceabi
> makes binutils go "checking target system type... Invalid configuration
> `armv7m-linux-musl-uclinuxfdpiceabi': machine `armv7m-linux-musl' not recognized".
> 
> So I was thinking "maybe I can build an sh4 kernel with an sh4 compiler, and
> build a userspace with the sh2eb compiler" (this is unlikely to work because
> every sh4 system I've ever used is little endian and the j-core guys
> inexplicably chose big endian, but I can burn that bridge when I come to it...)
> But that's how I hit the "enabling FDPIC on with-mmu only works with arm,
> nothing else" issue above. And that's incompatible with selecting musl support.
> 
> Rob
> 
> PS. the sh2 fdpic toolchain in musl-cross-make doesn't reproduce with the newest
> "supported" gcc (11.2.0) because:
> 
> In file included from libstdc++-v3/libsupc++/eh_call.cc:32:
> libstdc++-v3/../libgcc/unwind-pe.h: In function 'const unsigned char*
> read_encoded_value_with_base(unsigned char, _Unwind_Ptr, const unsigned char*,
> _Unwind_Ptr*)':
> libstdc++-v3/../libgcc/unwind-pe.h:270:25: error: '_Unwind_gnu_Find_got' was not
> declared in this scope
>   270 |               result += _Unwind_gnu_Find_got ((_Unwind_Ptr) u);
> 
> Which has been broken ever since musl-cross-make's newest last commit almost 2
> years ago.

Could you file a bug report for this? I think it's actually a GCC
regression but I'm not sure, so there probably needs to be a report on
the GCC tracker.

Rich

Re: [musl] FDPIC on sh4?

[Geert Uytterhoeven]

Hi Rich,

On Thu, Feb 15, 2024 at 2:49 PM Rich Felker <dalias@libc.org> wrote:
> On Thu, Feb 15, 2024 at 04:31:00AM -0600, Rob Landley wrote:
> > Is there any easy way to build FDPIC support for sh4 with-mmu? In theory ARM can
>
> On the userspace toolchain and musl side, yes, I see no reason you
> shouldn't be able to build musl for sh4 with fdpic ABI or build a
> toolchain for sh4 that defaults to fdpic and has fdpic target-libs. I
> just tested passing -mfdpic to sh4-linux-musl-gcc and it seems to
> produce correct fdpic code.
>
> On the kernel side, I'm not sure, but the normal ELF loader should be
> able to load fdpic binaries on a system with mmu. It will not float
> the data segment separately from text, but doesn't need to because it
> has an mmu. If this is no longer working it's a kernel regression;
> that's how I always tested sh2eb fdpic binaries on qemu-system-sh4eb.
>
> > do it, so I tried editing the kconfig BINFMT_ELF_FDPIC dependencies in
> > fs/Kconfig.binfmt to move "SUPERH" out of the !MMU list and put it next to ARM,
> > switched on the FDPIC loader, and the build broke with:
> >
> > fs/binfmt_elf_fdpic.o: in function `load_elf_fdpic_binary':
> > binfmt_elf_fdpic.c:(.text+0x1b44): undefined reference to
> > `elf_fdpic_arch_lay_out_mm'
>
> It looks like there's an arch-provided function that's conditional on
> !MMU in arch/sh but that, now that fdpic loader is intended to be
> supported on mmu-ful systems, should be changed to be conditional on
> fdpic support (or maybe even unconditional if fdpic can be loaded as a
> module). Just look for where it's defined in arch/sh. If it's in a
> nommu-specific file it might need to be moved somewhere more
> appropriate, or an mmu-ful variant may need to be written and placed
> somewhere more appropriate.

ARM is the sole architecture that provides elf_fdpic_arch_lay_out_mm().

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [musl] FDPIC on sh4?

[Rich Felker]

On Thu, Feb 15, 2024 at 03:53:53PM +0100, Geert Uytterhoeven wrote:
> Hi Rich,
> 
> On Thu, Feb 15, 2024 at 2:49 PM Rich Felker <dalias@libc.org> wrote:
> > On Thu, Feb 15, 2024 at 04:31:00AM -0600, Rob Landley wrote:
> > > Is there any easy way to build FDPIC support for sh4 with-mmu? In theory ARM can
> >
> > On the userspace toolchain and musl side, yes, I see no reason you
> > shouldn't be able to build musl for sh4 with fdpic ABI or build a
> > toolchain for sh4 that defaults to fdpic and has fdpic target-libs. I
> > just tested passing -mfdpic to sh4-linux-musl-gcc and it seems to
> > produce correct fdpic code.
> >
> > On the kernel side, I'm not sure, but the normal ELF loader should be
> > able to load fdpic binaries on a system with mmu. It will not float
> > the data segment separately from text, but doesn't need to because it
> > has an mmu. If this is no longer working it's a kernel regression;
> > that's how I always tested sh2eb fdpic binaries on qemu-system-sh4eb.
> >
> > > do it, so I tried editing the kconfig BINFMT_ELF_FDPIC dependencies in
> > > fs/Kconfig.binfmt to move "SUPERH" out of the !MMU list and put it next to ARM,
> > > switched on the FDPIC loader, and the build broke with:
> > >
> > > fs/binfmt_elf_fdpic.o: in function `load_elf_fdpic_binary':
> > > binfmt_elf_fdpic.c:(.text+0x1b44): undefined reference to
> > > `elf_fdpic_arch_lay_out_mm'
> >
> > It looks like there's an arch-provided function that's conditional on
> > !MMU in arch/sh but that, now that fdpic loader is intended to be
> > supported on mmu-ful systems, should be changed to be conditional on
> > fdpic support (or maybe even unconditional if fdpic can be loaded as a
> > module). Just look for where it's defined in arch/sh. If it's in a
> > nommu-specific file it might need to be moved somewhere more
> > appropriate, or an mmu-ful variant may need to be written and placed
> > somewhere more appropriate.
> 
> ARM is the sole architecture that provides elf_fdpic_arch_lay_out_mm().

Ah, I missed that this was a new mmu-ful only function. So I guess
something like the ARM one is needed. I'm not clear why this is
expected to be arch-specific, so it would probably be nice for the
kernel to provide a generic version that archs can use unless they
need specific behaviors here, or just make it conditional whether it's
called at all (only if the arch declares its presence) and use
defaults otherwise.

Rich

Re: [musl] FDPIC on sh4?

[Rob Landley]

On 2/15/24 10:47, Rich Felker wrote:
> On Thu, Feb 15, 2024 at 03:53:53PM +0100, Geert Uytterhoeven wrote:
>> Hi Rich,
>> 
>> On Thu, Feb 15, 2024 at 2:49 PM Rich Felker <dalias@libc.org> wrote:
>> > On Thu, Feb 15, 2024 at 04:31:00AM -0600, Rob Landley wrote:
>> > > Is there any easy way to build FDPIC support for sh4 with-mmu? In theory ARM can
>> >
>> > On the userspace toolchain and musl side, yes, I see no reason you
>> > shouldn't be able to build musl for sh4 with fdpic ABI or build a
>> > toolchain for sh4 that defaults to fdpic and has fdpic target-libs. I
>> > just tested passing -mfdpic to sh4-linux-musl-gcc and it seems to
>> > produce correct fdpic code.
>> >
>> > On the kernel side, I'm not sure, but the normal ELF loader should be
>> > able to load fdpic binaries on a system with mmu. It will not float
>> > the data segment separately from text, but doesn't need to because it
>> > has an mmu. If this is no longer working it's a kernel regression;
>> > that's how I always tested sh2eb fdpic binaries on qemu-system-sh4eb.
>> >
>> > > do it, so I tried editing the kconfig BINFMT_ELF_FDPIC dependencies in
>> > > fs/Kconfig.binfmt to move "SUPERH" out of the !MMU list and put it next to ARM,
>> > > switched on the FDPIC loader, and the build broke with:
>> > >
>> > > fs/binfmt_elf_fdpic.o: in function `load_elf_fdpic_binary':
>> > > binfmt_elf_fdpic.c:(.text+0x1b44): undefined reference to
>> > > `elf_fdpic_arch_lay_out_mm'
>> >
>> > It looks like there's an arch-provided function that's conditional on
>> > !MMU in arch/sh but that, now that fdpic loader is intended to be
>> > supported on mmu-ful systems, should be changed to be conditional on
>> > fdpic support (or maybe even unconditional if fdpic can be loaded as a
>> > module). Just look for where it's defined in arch/sh. If it's in a
>> > nommu-specific file it might need to be moved somewhere more
>> > appropriate, or an mmu-ful variant may need to be written and placed
>> > somewhere more appropriate.
>> 
>> ARM is the sole architecture that provides elf_fdpic_arch_lay_out_mm().
> 
> Ah, I missed that this was a new mmu-ful only function. So I guess
> something like the ARM one is needed. I'm not clear why this is
> expected to be arch-specific, so it would probably be nice for the
> kernel to provide a generic version that archs can use unless they
> need specific behaviors here, or just make it conditional whether it's
> called at all (only if the arch declares its presence) and use
> defaults otherwise.

It's in arch/arm/kernel/elf.c, and pretty short. Doesn't LOOK
architecture-specific, although I'm not an expert. (Why SZ_16M instead of
RLIM_STACK? No idea...)

#if defined(CONFIG_MMU) && defined(CONFIG_BINFMT_ELF_FDPIC)

void elf_fdpic_arch_lay_out_mm(struct elf_fdpic_params *exec_params,
                               struct elf_fdpic_params *interp_params,
                               unsigned long *start_stack,
                               unsigned long *start_brk)
{
        elf_set_personality(&exec_params->hdr);

        exec_params->load_addr = 0x8000;
        interp_params->load_addr = ELF_ET_DYN_BASE;
        *start_stack = TASK_SIZE - SZ_16M;

        if ((exec_params->flags & ELF_FDPIC_FLAG_ARRANGEMENT) ==
ELF_FDPIC_FLAG_INDEPENDENT) {
                exec_params->flags &= ~ELF_FDPIC_FLAG_ARRANGEMENT;
                exec_params->flags |= ELF_FDPIC_FLAG_CONSTDISP;
        }
}

#endif

Oddly, it's NOT in arch/arm64. Does the 64 bit arch pull in bits of the 32 bit
one, or is this only supported for 32 bit arm?

Rob

Re: [musl] FDPIC on sh4?

[Rob Landley]

[-- Attachment #1: Type: text/plain, Size: 3535 bytes --]



On 2/16/24 06:25, Rob Landley wrote:
> On 2/15/24 10:47, Rich Felker wrote:
>> On Thu, Feb 15, 2024 at 03:53:53PM +0100, Geert Uytterhoeven wrote:
>>> Hi Rich,
>>> 
>>> On Thu, Feb 15, 2024 at 2:49 PM Rich Felker <dalias@libc.org> wrote:
>>> > On Thu, Feb 15, 2024 at 04:31:00AM -0600, Rob Landley wrote:
>>> > > Is there any easy way to build FDPIC support for sh4 with-mmu? In theory ARM can
>>> >
>>> > On the userspace toolchain and musl side, yes, I see no reason you
>>> > shouldn't be able to build musl for sh4 with fdpic ABI or build a
>>> > toolchain for sh4 that defaults to fdpic and has fdpic target-libs. I
>>> > just tested passing -mfdpic to sh4-linux-musl-gcc and it seems to
>>> > produce correct fdpic code.
>>> >
>>> > On the kernel side, I'm not sure, but the normal ELF loader should be
>>> > able to load fdpic binaries on a system with mmu. It will not float
>>> > the data segment separately from text, but doesn't need to because it
>>> > has an mmu. If this is no longer working it's a kernel regression;
>>> > that's how I always tested sh2eb fdpic binaries on qemu-system-sh4eb.
>>> >
>>> > > do it, so I tried editing the kconfig BINFMT_ELF_FDPIC dependencies in
>>> > > fs/Kconfig.binfmt to move "SUPERH" out of the !MMU list and put it next to ARM,
>>> > > switched on the FDPIC loader, and the build broke with:
>>> > >
>>> > > fs/binfmt_elf_fdpic.o: in function `load_elf_fdpic_binary':
>>> > > binfmt_elf_fdpic.c:(.text+0x1b44): undefined reference to
>>> > > `elf_fdpic_arch_lay_out_mm'
>>> >
>>> > It looks like there's an arch-provided function that's conditional on
>>> > !MMU in arch/sh but that, now that fdpic loader is intended to be
>>> > supported on mmu-ful systems, should be changed to be conditional on
>>> > fdpic support (or maybe even unconditional if fdpic can be loaded as a
>>> > module). Just look for where it's defined in arch/sh. If it's in a
>>> > nommu-specific file it might need to be moved somewhere more
>>> > appropriate, or an mmu-ful variant may need to be written and placed
>>> > somewhere more appropriate.
>>> 
>>> ARM is the sole architecture that provides elf_fdpic_arch_lay_out_mm().
>> 
>> Ah, I missed that this was a new mmu-ful only function. So I guess
>> something like the ARM one is needed. I'm not clear why this is
>> expected to be arch-specific, so it would probably be nice for the
>> kernel to provide a generic version that archs can use unless they
>> need specific behaviors here, or just make it conditional whether it's
>> called at all (only if the arch declares its presence) and use
>> defaults otherwise.
> 
> It's in arch/arm/kernel/elf.c, and pretty short. Doesn't LOOK
> architecture-specific, although I'm not an expert. (Why SZ_16M instead of
> RLIM_STACK? No idea...)

arch/sh/kernel/elf.c doesn't exist so I cut and pasted the function into setup.c
just to see what would happen...

(Question about that file while I'm there: why are calibrate_delay(),
generic_mode_pins(), and test_mode_pin() not marked __init? And is marking
sh_fdt_init() as __ref equivalent to marking it __init?)

The build broke, I fixed it up to at least compile, and my sh2eb fdpic
filesystem didn't boot because...

Starting init: /bin/sh exists but couldn't execute it (error -8)

Exec format error, but I _think_ that's me trying to run a bit endian executable
on a little endian kernel build. :)

Anyway, to keep you posted, here's the "wrong fix" so far.  (Attached because
thunderbird "upgraded" itself and broke my wordrap disable plugin.)

Rob

[-- Attachment #2: sh-fdpic-mmu.patch --]
[-- Type: text/x-patch, Size: 1102 bytes --]

diff --git a/arch/sh/kernel/setup.c b/arch/sh/kernel/setup.c
index d3175f09b3aa..effda8b21370 100644
--- a/arch/sh/kernel/setup.c
+++ b/arch/sh/kernel/setup.c
@@ -404,3 +404,28 @@ void __init arch_cpu_finalize_init(void)
 #endif
 	*p = '\0';
 }
+
+#if defined(CONFIG_MMU) && defined(CONFIG_BINFMT_ELF_FDPIC)
+
+#include <linux/personality.h>
+#include <linux/elf-fdpic.h>
+
+void elf_fdpic_arch_lay_out_mm(struct elf_fdpic_params *exec_params,
+                               struct elf_fdpic_params *interp_params,
+                               unsigned long *start_stack,
+                               unsigned long *start_brk)
+{
+	set_personality((current->personality & ~PER_MASK) | PER_LINUX);
+
+        exec_params->load_addr = 0x8000;
+        interp_params->load_addr = ELF_ET_DYN_BASE;
+        *start_stack = TASK_SIZE - SZ_16M;
+
+        if ((exec_params->flags & ELF_FDPIC_FLAG_ARRANGEMENT) == ELF_FDPIC_FLAG_INDEPENDENT) {
+                exec_params->flags &= ~ELF_FDPIC_FLAG_ARRANGEMENT;
+                exec_params->flags |= ELF_FDPIC_FLAG_CONSTDISP;
+        }
+}
+
+#endif
+