Re: [musl] Musl on Cortex-M Devices

[Patrick Oppenlander <patrick.oppenlander@gmail.com>]

On Fri, Dec 18, 2020 at 6:17 PM Jesse DeGuire <jesse.a.deguire@gmail.com> wrote:
>
> On Thu, Dec 17, 2020 at 12:10 AM Patrick Oppenlander
> <patrick.oppenlander@gmail.com> wrote:
> >
> > On Thu, Dec 17, 2020 at 3:55 PM Patrick Oppenlander
> > <patrick.oppenlander@gmail.com> wrote:
> > >
> > > On Thu, Dec 17, 2020 at 11:24 AM Rich Felker <dalias@libc.org> wrote:
> > > >
> > > > On Wed, Dec 16, 2020 at 06:43:15PM -0500, Jesse DeGuire wrote:
> > > > > Hey everyone,
> > > > >
> > > > > I'm working on putting together a Clang-based toolchain to use with
> > > > > Microchip PIC32 (MIPS32) and SAM (Cortex-M) microcontrollers as an
> > > > > alternative to their paid XC32 toolchain and I'd like to use Musl as
> > > > > the C library. Currently, I'm trying to get it to build for a few
> > > > > different Cortex-M devices and have found that Musl builds fine for
> > > > > ARMv7-M, but not for ARMv6-M or v8-M Baseline because Musl uses
> > > > > instructions not supported on the Thumb ISA subset used by v6-M and
> > > > > v8-M Baseline devices. I'm using the v1.2.1 tag version of Musl, but
> > > > > can easily switch to HEAD if needed. I am using a Python script to
> > > > > build Musl (and eventually the rest of the toolchain), which you can
> > > > > see on GitHub at the following link. It's a bit of a mess at the
> > > > > moment, but the build_musl() function is what I'm currently using to
> > > > > build Musl.
> > > >
> > > > I had assumed the thumb1[-ish?] subset wouldn't be interesting, but if
> > > > it is, let's have a look.
> > > >
> > > > > https://github.com/jdeguire/buildPic32Clang/blob/master/buildPic32Clang.py
> > > > >
> > > > > Anyway, I have managed to get Musl to build for v6-M, v7-M, and v8-M
> > > > > Baseline and have attached a diff to this email. If you like, I can go
> > > > > into more detail as to why I made the changes I made; however, many
> > > > > changes were merely the result of my attempts to correct errors
> > > > > reported by Clang due to it encountering instruction sequences not
> > > > > supported on ARMv6-M.
> > > >
> > > > Are there places where clang's linker is failing to make substitutions
> > > > that the GNU one can do, that would make this simpler? For example I
> > > > know the GNU one can replace bx rn by mov pc,rn if needed (based on a
> > > > relocation the assembler emits on the insn).
> > > >
> > > > > A number of errors were simply the result of
> > > > > ARMv6-M requiring one to use the "S" variant of an instruction that
> > > > > sets status flags (such as "ADDS" vs "ADD" or "MOVS" vs "MOV"). A few
> > > > > files I had to change from a "lower case s" to a "capital-S" file so
> > > > > that I could use macros to check for either the Thumb1 ISA
> > > > > ("__thumb2__ || !__thumb__") or for an M-Profile device
> > > > > ("!__ARM_ARCH_ISA_ARM").
> > > >
> > > > Is __ARM_ARCH_ISA_ARM universally available (even on old compilers)?
> > > > If not this may need an alternate detection. But I'd like to drop as
> > > > much as possible and just make the code compatible rather than having
> > > > 2 versions of it. I don't think there are any places where the
> > > > performance or size is at all relevant.
> > > >
> > > > > The changes under
> > > > > "src/thread/arm/__set_thread_area.c" are different in that I made
> > > > > those because I don't believe Cortex-M devices could handle what was
> > > > > there (no M-Profile device has Coprocessor 15, for example) and so I
> > > >
> > > > Unless this is an ISA level that can't be executed on a normal (non-M)
> > > > ARM profile, it still needs all the backends that might be needed and
> > > > runtime selection of which to use. This is okay. I believe Linux for
> > > > nommu ARM has a syscall for get_tp, which is rather awful but probably
> > > > needs to be added as a backend. The right way to do this would have
> > > > been with trap-and-emulate (of cp15) I think...
> > >
> > > Linux emulates mrc 15 on old -A cores but they decided not to on -M
> > > for some reason. BTW, the syscall is called get_tls.
> > >
> > > Is there any option other than supporting the get_tls syscall? Even if
> > > someone puts in the effort to wire up the trap-and-emulate backend,
> > > musl linked executables will still only run on new kernels.
> > >
> > > I took the trap-and-emulate approach in Apex RTOS to avoid opening
> > > this can of worms. It's the only missing link for musl on armv7-m.
> > > Everything else works beautifully.
> >
> > Another consideration is qemu-user: Currently it aborts when
> > encountering an mrc 15 instruction while emulating armv7-m. I guess
> > that would probably also be solved by supporting the syscall.
> >
> > Patrick
>
> ARMv6-M and v8-M.base do not support the MRC instruction at all. Could
> that play into why Linux and qemu bail?
>
> Jesse

Sorry, I missed this reply.

qemu-user refuses to translate the instruction because cp15 is not
implemented on armv7-m, exactly the same issue as is being discussed
here. If you run the same executable but tell qemu to emulate an A
profile core instead it happily runs it.

Linux will probably kill the executable with SIGILL or something like
that (I haven't tried, just guessing).

It's related to this discussion as changing musl to use the syscall
will likely result in qemu-user working too.

I would personally prefer to see a solution which doesn't use the
syscall. It's possible to implement the trap-and-emulate much more
efficiently than the syscall as it can quite easily be done without
preserving any more registers than the core pushes on exception entry
anyway. https://github.com/apexrtos/apex/blob/master/sys/arch/arm/v7m/emulate.S
is what I came up with. That implementation could be even tighter as
it can never run from handler mode, so the stack detection at the
beginning is unnecessary. However, I haven't considered v6-m or v8-m.

trap-and-emulate also gracefully degrades when running the same
executable on A vs M cores.

Patrick