How to handle attempts to combine ARM Thumb with frame pointers?

How to handle attempts to combine ARM Thumb with frame pointers?

[Andre McCurdy]

When compiling for ARM Thumb or Thumb2 with frame pointers enabled (ie
-O0 or with -fno-omit-frame-pointer in CFLAGS) the frame pointer is
stored in r7, which leads to build errors ("error: r7 cannot be used
in asm here") whenever a syscall macro is included in a C function.
It's certainly a corner case, but one which I've run into recently.

Would it be worth trying to catch this combination earlier and failing
from the configure script? It's not trivial to do reliably since I
think detecting whether or not frame pointers are going to be used by
examining CFLAGS means determining the effective optimisation level if
multiple -O0, -Os, etc options are given, together with the effective
outcome of potentially multiple -fno-omit-frame-pointer and
-fomit-frame-pointer options.

I can work on a patch for the configure script but first wanted to
check what the philosophy is - should the configure script be trying
to catch every possible misconfiguration?

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Khem Raj]

On Fri, Oct 6, 2017 at 5:53 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
> When compiling for ARM Thumb or Thumb2 with frame pointers enabled (ie
> -O0 or with -fno-omit-frame-pointer in CFLAGS) the frame pointer is
> stored in r7, which leads to build errors ("error: r7 cannot be used
> in asm here") whenever a syscall macro is included in a C function.
> It's certainly a corner case, but one which I've run into recently.
>
> Would it be worth trying to catch this combination earlier and failing
> from the configure script? It's not trivial to do reliably since I
> think detecting whether or not frame pointers are going to be used by
> examining CFLAGS means determining the effective optimisation level if
> multiple -O0, -Os, etc options are given, together with the effective
> outcome of potentially multiple -fno-omit-frame-pointer and
> -fomit-frame-pointer options.
>
> I can work on a patch for the configure script but first wanted to
> check what the philosophy is - should the configure script be trying
> to catch every possible misconfiguration?

I think these files assume that frame pointer is not stored, quick fix is to
compile the specific file with -fomit-frame-pointer something like
CFLAGS-<filenname> = -fomit-frame-pointer

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Andre McCurdy]

On Fri, Oct 6, 2017 at 5:53 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
> When compiling for ARM Thumb or Thumb2 with frame pointers enabled (ie
> -O0 or with -fno-omit-frame-pointer in CFLAGS) the frame pointer is
> stored in r7, which leads to build errors ("error: r7 cannot be used
> in asm here") whenever a syscall macro is included in a C function.
> It's certainly a corner case, but one which I've run into recently.
>
> Would it be worth trying to catch this combination earlier and failing
> from the configure script? It's not trivial to do reliably since I
> think detecting whether or not frame pointers are going to be used by
> examining CFLAGS means determining the effective optimisation level if
> multiple -O0, -Os, etc options are given, together with the effective
> outcome of potentially multiple -fno-omit-frame-pointer and
> -fomit-frame-pointer options.
>
> I can work on a patch for the configure script but first wanted to
> check what the philosophy is - should the configure script be trying
> to catch every possible misconfiguration?

Sorry to reply to my own thread, but it looks like this long standing
clang bug may actually be due to this same issue:

  http://clang-developers.42468.n3.nabble.com/Bug-in-ARM-Thumb-inline-asm-td4043905.html
  https://bugs.llvm.org/show_bug.cgi?id=34165

The only real difference being that where gcc aborts with an error
about reusing r7, clang silently generates broken code :-(

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Rich Felker]

On Fri, Oct 06, 2017 at 05:53:38PM -0700, Andre McCurdy wrote:
> When compiling for ARM Thumb or Thumb2 with frame pointers enabled (ie
> -O0 or with -fno-omit-frame-pointer in CFLAGS) the frame pointer is
> stored in r7, which leads to build errors ("error: r7 cannot be used
> in asm here") whenever a syscall macro is included in a C function.
> It's certainly a corner case, but one which I've run into recently.
> 
> Would it be worth trying to catch this combination earlier and failing
> from the configure script? It's not trivial to do reliably since I
> think detecting whether or not frame pointers are going to be used by
> examining CFLAGS means determining the effective optimisation level if
> multiple -O0, -Os, etc options are given, together with the effective
> outcome of potentially multiple -fno-omit-frame-pointer and
> -fomit-frame-pointer options.
> 
> I can work on a patch for the configure script but first wanted to
> check what the philosophy is - should the configure script be trying
> to catch every possible misconfiguration?

At the core, I think this is a bug in GCC and clang, in the sense that
they shouldn't be enforcing fixed registers in a way that conflicts
with asm constraints. IIRC this was fixed on x86 for ebx and ebp a
while back. But indeed if it's the state of things, that's how it is.

If you do want to test for broken configurations, rather than
hard-coding an assumption that some configuration is broken, you
should test for it. This would look something like, if ARCH is arm,
try compiling a trivial function with inline asm using r7 and see if
it fails. If so, exit with an error or perhaps try adding
-fomit-frame-pointer and retrying. I haven't tried any of this yet so
I don't know how ugly/hackish it would be and whether it would be
appropriate to include but it sounds like it could be.

If clang generates broken code silently, though, I don't know any good
way to test for that.

Rich

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Andre McCurdy]

On Sat, Oct 7, 2017 at 8:21 PM, Rich Felker <dalias@libc.org> wrote:
> On Fri, Oct 06, 2017 at 05:53:38PM -0700, Andre McCurdy wrote:
>> When compiling for ARM Thumb or Thumb2 with frame pointers enabled (ie
>> -O0 or with -fno-omit-frame-pointer in CFLAGS) the frame pointer is
>> stored in r7, which leads to build errors ("error: r7 cannot be used
>> in asm here") whenever a syscall macro is included in a C function.
>> It's certainly a corner case, but one which I've run into recently.
>>
>> Would it be worth trying to catch this combination earlier and failing
>> from the configure script? It's not trivial to do reliably since I
>> think detecting whether or not frame pointers are going to be used by
>> examining CFLAGS means determining the effective optimisation level if
>> multiple -O0, -Os, etc options are given, together with the effective
>> outcome of potentially multiple -fno-omit-frame-pointer and
>> -fomit-frame-pointer options.
>>
>> I can work on a patch for the configure script but first wanted to
>> check what the philosophy is - should the configure script be trying
>> to catch every possible misconfiguration?
>
> At the core, I think this is a bug in GCC and clang, in the sense that
> they shouldn't be enforcing fixed registers in a way that conflicts
> with asm constraints.

I think gcc's current behaviour of aborting with an error is
reasonable. In this particular case, not using a frame pointer is the
only solution since the registers required to make a syscall can't be
changed. But there may be other cases where inline asm is using r7
arbitrarily and changing the asm to use a different register would be
a more appropriate solution. The user probably needs to make that
decision rather gcc.

> IIRC this was fixed on x86 for ebx and ebp a
> while back. But indeed if it's the state of things, that's how it is.
>
> If you do want to test for broken configurations, rather than
> hard-coding an assumption that some configuration is broken, you
> should test for it. This would look something like, if ARCH is arm,
> try compiling a trivial function with inline asm using r7 and see if
> it fails.

Yes, I came to the same conclusion after seeing the clang bug, which
seems to suggest that clang uses a frame pointer even with
optimisation enabled.

> If so, exit with an error or perhaps try adding
> -fomit-frame-pointer and retrying.

If we over-ride the user supplied CFLAGS then there's probably no need
to test the behaviour of the compiler - we can just force
-fomit-frame-pointer unconditionally when compiling for Thumb/Thumb2.

There's a slight complication though that if -fno-omit-frame-pointer
is present in the user supplied CFLAGS then adding
-fomit-frame-pointer to CFLAGS_AUTO won't over-ride it (since CFLAGS
appears on the final compiler command line after CFLAGS_AUTO).

Would it be OK for the configure script to append to CFLAGS? Or should
the configure script perhaps setup a new variable (CFLAGS_FORCE?)
which the Makefile would then add to CFLAGS_ALL after CFLAGS?

> I haven't tried any of this yet so
> I don't know how ugly/hackish it would be and whether it would be
> appropriate to include but it sounds like it could be.
>
> If clang generates broken code silently, though, I don't know any good
> way to test for that.
>
> Rich

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Szabolcs Nagy]

* Andre McCurdy <armccurdy@gmail.com> [2017-10-09 09:48:29 -0700]:
> On Sat, Oct 7, 2017 at 8:21 PM, Rich Felker <dalias@libc.org> wrote:
> > On Fri, Oct 06, 2017 at 05:53:38PM -0700, Andre McCurdy wrote:
> > If you do want to test for broken configurations, rather than
> > hard-coding an assumption that some configuration is broken, you
> > should test for it. This would look something like, if ARCH is arm,
> > try compiling a trivial function with inline asm using r7 and see if
> > it fails.
> 
> Yes, I came to the same conclusion after seeing the clang bug, which
> seems to suggest that clang uses a frame pointer even with
> optimisation enabled.
> 
> > If so, exit with an error or perhaps try adding
> > -fomit-frame-pointer and retrying.
> 
> If we over-ride the user supplied CFLAGS then there's probably no need
> to test the behaviour of the compiler - we can just force
> -fomit-frame-pointer unconditionally when compiling for Thumb/Thumb2.
> 
> There's a slight complication though that if -fno-omit-frame-pointer
> is present in the user supplied CFLAGS then adding
> -fomit-frame-pointer to CFLAGS_AUTO won't over-ride it (since CFLAGS
> appears on the final compiler command line after CFLAGS_AUTO).
> 
> Would it be OK for the configure script to append to CFLAGS? Or should
> the configure script perhaps setup a new variable (CFLAGS_FORCE?)
> which the Makefile would then add to CFLAGS_ALL after CFLAGS?
> 

glibc works this around in thumb mode by extern syscall asm
(of course it cannot guarantee that r7 is a frame pointer at
all times, an interrupt can observe r7 with syscall num in it,
i'm not sure if that's acceptable for users who compile with
frame-pointers, in musl there is some asm code which wont
have fp setup anyway).

http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/arm/sysdep.h;h=6a64351cdd87c2041d639a17efc9f681262d5e3f;hb=HEAD#l335

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Adhemerval Zanella]

On 25/10/2017 19:16, Szabolcs Nagy wrote:
> * Andre McCurdy <armccurdy@gmail.com> [2017-10-09 09:48:29 -0700]:
>> On Sat, Oct 7, 2017 at 8:21 PM, Rich Felker <dalias@libc.org> wrote:
>>> On Fri, Oct 06, 2017 at 05:53:38PM -0700, Andre McCurdy wrote:
>>> If you do want to test for broken configurations, rather than
>>> hard-coding an assumption that some configuration is broken, you
>>> should test for it. This would look something like, if ARCH is arm,
>>> try compiling a trivial function with inline asm using r7 and see if
>>> it fails.
>>
>> Yes, I came to the same conclusion after seeing the clang bug, which
>> seems to suggest that clang uses a frame pointer even with
>> optimisation enabled.
>>
>>> If so, exit with an error or perhaps try adding
>>> -fomit-frame-pointer and retrying.
>>
>> If we over-ride the user supplied CFLAGS then there's probably no need
>> to test the behaviour of the compiler - we can just force
>> -fomit-frame-pointer unconditionally when compiling for Thumb/Thumb2.
>>
>> There's a slight complication though that if -fno-omit-frame-pointer
>> is present in the user supplied CFLAGS then adding
>> -fomit-frame-pointer to CFLAGS_AUTO won't over-ride it (since CFLAGS
>> appears on the final compiler command line after CFLAGS_AUTO).
>>
>> Would it be OK for the configure script to append to CFLAGS? Or should
>> the configure script perhaps setup a new variable (CFLAGS_FORCE?)
>> which the Makefile would then add to CFLAGS_ALL after CFLAGS?
>>
> 
> glibc works this around in thumb mode by extern syscall asm
> (of course it cannot guarantee that r7 is a frame pointer at
> all times, an interrupt can observe r7 with syscall num in it,
> i'm not sure if that's acceptable for users who compile with
> frame-pointers, in musl there is some asm code which wont
> have fp setup anyway).
> 
> http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/arm/sysdep.h;h=6a64351cdd87c2041d639a17efc9f681262d5e3f;hb=HEAD#l335
> 

Why do you mean by glibc strategy might not be acceptable? What
kind of issue are you referring on interrupt case?

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Rich Felker]

On Thu, Oct 26, 2017 at 02:48:11PM -0200, Adhemerval Zanella wrote:
> 
> 
> On 25/10/2017 19:16, Szabolcs Nagy wrote:
> > * Andre McCurdy <armccurdy@gmail.com> [2017-10-09 09:48:29 -0700]:
> >> On Sat, Oct 7, 2017 at 8:21 PM, Rich Felker <dalias@libc.org> wrote:
> >>> On Fri, Oct 06, 2017 at 05:53:38PM -0700, Andre McCurdy wrote:
> >>> If you do want to test for broken configurations, rather than
> >>> hard-coding an assumption that some configuration is broken, you
> >>> should test for it. This would look something like, if ARCH is arm,
> >>> try compiling a trivial function with inline asm using r7 and see if
> >>> it fails.
> >>
> >> Yes, I came to the same conclusion after seeing the clang bug, which
> >> seems to suggest that clang uses a frame pointer even with
> >> optimisation enabled.
> >>
> >>> If so, exit with an error or perhaps try adding
> >>> -fomit-frame-pointer and retrying.
> >>
> >> If we over-ride the user supplied CFLAGS then there's probably no need
> >> to test the behaviour of the compiler - we can just force
> >> -fomit-frame-pointer unconditionally when compiling for Thumb/Thumb2.
> >>
> >> There's a slight complication though that if -fno-omit-frame-pointer
> >> is present in the user supplied CFLAGS then adding
> >> -fomit-frame-pointer to CFLAGS_AUTO won't over-ride it (since CFLAGS
> >> appears on the final compiler command line after CFLAGS_AUTO).
> >>
> >> Would it be OK for the configure script to append to CFLAGS? Or should
> >> the configure script perhaps setup a new variable (CFLAGS_FORCE?)
> >> which the Makefile would then add to CFLAGS_ALL after CFLAGS?
> >>
> > 
> > glibc works this around in thumb mode by extern syscall asm
> > (of course it cannot guarantee that r7 is a frame pointer at
> > all times, an interrupt can observe r7 with syscall num in it,
> > i'm not sure if that's acceptable for users who compile with
> > frame-pointers, in musl there is some asm code which wont
> > have fp setup anyway).
> > 
> > http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/arm/sysdep.h;h=6a64351cdd87c2041d639a17efc9f681262d5e3f;hb=HEAD#l335
> > 
> 
> Why do you mean by glibc strategy might not be acceptable? What
> kind of issue are you referring on interrupt case?

If you're compiling with frame pointers because you want them to be
present (and always valid) for debugging purposes or similar, there's
no way to achieve that while making syscalls -- and the most likely
place for a process to get stopped debugging is usually at a syscall.
Maybe this doesn't matter. It's not something we can change, just an
observation about a problem with the ABI, I think.

I think what we could do to ensure that compiling with frame pointers
otherwise works is add a configure test for use of r7 in inline asm,
and if it fails, -D something that would cause arch/arm/syscall_arch.h
not to define any syscall inlines but instead #define
SYSCALL_NO_INLINE which results in src/internal/syscall.h just using
the external __syscall function (which will of course clobber r7 for
the duration of the syscall).

Rich

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Andre McCurdy]

On Thu, Oct 26, 2017 at 10:00 AM, Rich Felker <dalias@libc.org> wrote:
> On Thu, Oct 26, 2017 at 02:48:11PM -0200, Adhemerval Zanella wrote:
>> On 25/10/2017 19:16, Szabolcs Nagy wrote:
>> > * Andre McCurdy <armccurdy@gmail.com> [2017-10-09 09:48:29 -0700]:
>> >> On Sat, Oct 7, 2017 at 8:21 PM, Rich Felker <dalias@libc.org> wrote:
>> >>> On Fri, Oct 06, 2017 at 05:53:38PM -0700, Andre McCurdy wrote:
>> >>> If you do want to test for broken configurations, rather than
>> >>> hard-coding an assumption that some configuration is broken, you
>> >>> should test for it. This would look something like, if ARCH is arm,
>> >>> try compiling a trivial function with inline asm using r7 and see if
>> >>> it fails.
>> >>
>> >> Yes, I came to the same conclusion after seeing the clang bug, which
>> >> seems to suggest that clang uses a frame pointer even with
>> >> optimisation enabled.
>> >>
>> >>> If so, exit with an error or perhaps try adding
>> >>> -fomit-frame-pointer and retrying.
>> >>
>> >> If we over-ride the user supplied CFLAGS then there's probably no need
>> >> to test the behaviour of the compiler - we can just force
>> >> -fomit-frame-pointer unconditionally when compiling for Thumb/Thumb2.
>> >>
>> >> There's a slight complication though that if -fno-omit-frame-pointer
>> >> is present in the user supplied CFLAGS then adding
>> >> -fomit-frame-pointer to CFLAGS_AUTO won't over-ride it (since CFLAGS
>> >> appears on the final compiler command line after CFLAGS_AUTO).
>> >>
>> >> Would it be OK for the configure script to append to CFLAGS? Or should
>> >> the configure script perhaps setup a new variable (CFLAGS_FORCE?)
>> >> which the Makefile would then add to CFLAGS_ALL after CFLAGS?
>> >
>> > glibc works this around in thumb mode by extern syscall asm
>> > (of course it cannot guarantee that r7 is a frame pointer at
>> > all times, an interrupt can observe r7 with syscall num in it,
>> > i'm not sure if that's acceptable for users who compile with
>> > frame-pointers, in musl there is some asm code which wont
>> > have fp setup anyway).
>> >
>> > http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/arm/sysdep.h;h=6a64351cdd87c2041d639a17efc9f681262d5e3f;hb=HEAD#l335
>>
>> Why do you mean by glibc strategy might not be acceptable? What
>> kind of issue are you referring on interrupt case?
>
> If you're compiling with frame pointers because you want them to be
> present (and always valid) for debugging purposes or similar, there's
> no way to achieve that while making syscalls -- and the most likely
> place for a process to get stopped debugging is usually at a syscall.
> Maybe this doesn't matter. It's not something we can change, just an
> observation about a problem with the ABI, I think.
>
> I think what we could do to ensure that compiling with frame pointers
> otherwise works is add a configure test for use of r7 in inline asm,
> and if it fails

Using r7 in inline asm together with frame pointers fails at build
time with gcc, but not with clang.

But perhaps an alternative way to detect whether the current
combination of compiler + cflags is going to try to use frame pointers
is to compile a trivial function to assembler and parse the output. I
haven't tested clang, but gcc adds a helpful "frame_needed" comment
which is easy to grep for.

  $ arm-linux-gnueabi-gcc -S tst.c -o - -O0 | grep frame_needed
    @ frame_needed = 1, uses_anonymous_args = 0

  $ arm-linux-gnueabi-gcc -S tst.c -o - -O0 -fomit-frame-pointer |
grep frame_needed
    @ frame_needed = 0, uses_anonymous_args = 0

  $ arm-linux-gnueabi-gcc -S tst.c -o - -O2 | grep frame_needed
    @ frame_needed = 0, uses_anonymous_args = 0

  $ arm-linux-gnueabi-gcc -S tst.c -o - -O2 -fno-omit-frame-pointer |
grep frame_needed
    @ frame_needed = 1, uses_anonymous_args = 0


>, -D something that would cause arch/arm/syscall_arch.h
> not to define any syscall inlines but instead #define
> SYSCALL_NO_INLINE which results in src/internal/syscall.h just using
> the external __syscall function (which will of course clobber r7 for
> the duration of the syscall).
>
> Rich

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Rich Felker]

On Thu, Oct 26, 2017 at 10:48:41AM -0700, Andre McCurdy wrote:
> On Thu, Oct 26, 2017 at 10:00 AM, Rich Felker <dalias@libc.org> wrote:
> > On Thu, Oct 26, 2017 at 02:48:11PM -0200, Adhemerval Zanella wrote:
> >> On 25/10/2017 19:16, Szabolcs Nagy wrote:
> >> > * Andre McCurdy <armccurdy@gmail.com> [2017-10-09 09:48:29 -0700]:
> >> >> On Sat, Oct 7, 2017 at 8:21 PM, Rich Felker <dalias@libc.org> wrote:
> >> >>> On Fri, Oct 06, 2017 at 05:53:38PM -0700, Andre McCurdy wrote:
> >> >>> If you do want to test for broken configurations, rather than
> >> >>> hard-coding an assumption that some configuration is broken, you
> >> >>> should test for it. This would look something like, if ARCH is arm,
> >> >>> try compiling a trivial function with inline asm using r7 and see if
> >> >>> it fails.
> >> >>
> >> >> Yes, I came to the same conclusion after seeing the clang bug, which
> >> >> seems to suggest that clang uses a frame pointer even with
> >> >> optimisation enabled.
> >> >>
> >> >>> If so, exit with an error or perhaps try adding
> >> >>> -fomit-frame-pointer and retrying.
> >> >>
> >> >> If we over-ride the user supplied CFLAGS then there's probably no need
> >> >> to test the behaviour of the compiler - we can just force
> >> >> -fomit-frame-pointer unconditionally when compiling for Thumb/Thumb2.
> >> >>
> >> >> There's a slight complication though that if -fno-omit-frame-pointer
> >> >> is present in the user supplied CFLAGS then adding
> >> >> -fomit-frame-pointer to CFLAGS_AUTO won't over-ride it (since CFLAGS
> >> >> appears on the final compiler command line after CFLAGS_AUTO).
> >> >>
> >> >> Would it be OK for the configure script to append to CFLAGS? Or should
> >> >> the configure script perhaps setup a new variable (CFLAGS_FORCE?)
> >> >> which the Makefile would then add to CFLAGS_ALL after CFLAGS?
> >> >
> >> > glibc works this around in thumb mode by extern syscall asm
> >> > (of course it cannot guarantee that r7 is a frame pointer at
> >> > all times, an interrupt can observe r7 with syscall num in it,
> >> > i'm not sure if that's acceptable for users who compile with
> >> > frame-pointers, in musl there is some asm code which wont
> >> > have fp setup anyway).
> >> >
> >> > http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/arm/sysdep.h;h=6a64351cdd87c2041d639a17efc9f681262d5e3f;hb=HEAD#l335
> >>
> >> Why do you mean by glibc strategy might not be acceptable? What
> >> kind of issue are you referring on interrupt case?
> >
> > If you're compiling with frame pointers because you want them to be
> > present (and always valid) for debugging purposes or similar, there's
> > no way to achieve that while making syscalls -- and the most likely
> > place for a process to get stopped debugging is usually at a syscall.
> > Maybe this doesn't matter. It's not something we can change, just an
> > observation about a problem with the ABI, I think.
> >
> > I think what we could do to ensure that compiling with frame pointers
> > otherwise works is add a configure test for use of r7 in inline asm,
> > and if it fails
> 
> Using r7 in inline asm together with frame pointers fails at build
> time with gcc, but not with clang.
> 
> But perhaps an alternative way to detect whether the current
> combination of compiler + cflags is going to try to use frame pointers
> is to compile a trivial function to assembler and parse the output. I
> haven't tested clang, but gcc adds a helpful "frame_needed" comment
> which is easy to grep for.

This is not a good approach. It depends on specific compiler behavior
(text that's not part of the code) and thus has both false negatives
and false positives (it would break on compilers that allow you to use
r7 in asm constraints even when the compiler is using frame pointers).

I had forgotten about the clang issue though --- is it actually
silently generating bad code that doesn't respect the constraint? Or
something else? If so we probably need a separate way to detect it.

Rich

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Andre McCurdy]

On Thu, Oct 26, 2017 at 10:54 AM, Rich Felker <dalias@libc.org> wrote:
> On Thu, Oct 26, 2017 at 10:48:41AM -0700, Andre McCurdy wrote:
>> On Thu, Oct 26, 2017 at 10:00 AM, Rich Felker <dalias@libc.org> wrote:
>> > On Thu, Oct 26, 2017 at 02:48:11PM -0200, Adhemerval Zanella wrote:
>> >> On 25/10/2017 19:16, Szabolcs Nagy wrote:
>> >> > * Andre McCurdy <armccurdy@gmail.com> [2017-10-09 09:48:29 -0700]:
>> >> >> On Sat, Oct 7, 2017 at 8:21 PM, Rich Felker <dalias@libc.org> wrote:
>> >> >>> On Fri, Oct 06, 2017 at 05:53:38PM -0700, Andre McCurdy wrote:
>> >> >>> If you do want to test for broken configurations, rather than
>> >> >>> hard-coding an assumption that some configuration is broken, you
>> >> >>> should test for it. This would look something like, if ARCH is arm,
>> >> >>> try compiling a trivial function with inline asm using r7 and see if
>> >> >>> it fails.
>> >> >>
>> >> >> Yes, I came to the same conclusion after seeing the clang bug, which
>> >> >> seems to suggest that clang uses a frame pointer even with
>> >> >> optimisation enabled.
>> >> >>
>> >> >>> If so, exit with an error or perhaps try adding
>> >> >>> -fomit-frame-pointer and retrying.
>> >> >>
>> >> >> If we over-ride the user supplied CFLAGS then there's probably no need
>> >> >> to test the behaviour of the compiler - we can just force
>> >> >> -fomit-frame-pointer unconditionally when compiling for Thumb/Thumb2.
>> >> >>
>> >> >> There's a slight complication though that if -fno-omit-frame-pointer
>> >> >> is present in the user supplied CFLAGS then adding
>> >> >> -fomit-frame-pointer to CFLAGS_AUTO won't over-ride it (since CFLAGS
>> >> >> appears on the final compiler command line after CFLAGS_AUTO).
>> >> >>
>> >> >> Would it be OK for the configure script to append to CFLAGS? Or should
>> >> >> the configure script perhaps setup a new variable (CFLAGS_FORCE?)
>> >> >> which the Makefile would then add to CFLAGS_ALL after CFLAGS?
>> >> >
>> >> > glibc works this around in thumb mode by extern syscall asm
>> >> > (of course it cannot guarantee that r7 is a frame pointer at
>> >> > all times, an interrupt can observe r7 with syscall num in it,
>> >> > i'm not sure if that's acceptable for users who compile with
>> >> > frame-pointers, in musl there is some asm code which wont
>> >> > have fp setup anyway).
>> >> >
>> >> > http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/arm/sysdep.h;h=6a64351cdd87c2041d639a17efc9f681262d5e3f;hb=HEAD#l335
>> >>
>> >> Why do you mean by glibc strategy might not be acceptable? What
>> >> kind of issue are you referring on interrupt case?
>> >
>> > If you're compiling with frame pointers because you want them to be
>> > present (and always valid) for debugging purposes or similar, there's
>> > no way to achieve that while making syscalls -- and the most likely
>> > place for a process to get stopped debugging is usually at a syscall.
>> > Maybe this doesn't matter. It's not something we can change, just an
>> > observation about a problem with the ABI, I think.
>> >
>> > I think what we could do to ensure that compiling with frame pointers
>> > otherwise works is add a configure test for use of r7 in inline asm,
>> > and if it fails
>>
>> Using r7 in inline asm together with frame pointers fails at build
>> time with gcc, but not with clang.
>>
>> But perhaps an alternative way to detect whether the current
>> combination of compiler + cflags is going to try to use frame pointers
>> is to compile a trivial function to assembler and parse the output. I
>> haven't tested clang, but gcc adds a helpful "frame_needed" comment
>> which is easy to grep for.
>
> This is not a good approach. It depends on specific compiler behavior
> (text that's not part of the code) and thus has both false negatives
> and false positives (it would break on compilers that allow you to use
> r7 in asm constraints even when the compiler is using frame pointers).

Yes, agreed. Just checking for the gcc comment isn't robust. But I
think there are other differences between the two cases which could be
detected reliably with a slightly more elaborate test, e.g. checking
for the use of r7 in the object code (assuming that for a trivial
function which just returns there's no reason that the compiler would
ever use of r7 except for a frame pointer).

  $ arm-linux-gnueabi-gcc -mthumb -O0 -c tst.c
  $ arm-linux-gnueabi-objdump -d -M reg-names-raw tst.o

    tst.o:     file format elf32-littlearm

    Disassembly of section .text:

    00000000 <a>:
       0:    b480          push    {r7}
       2:    af00          add    r7, sp, #0
       4:    bf00          nop
       6:    46bd          mov    r13, r7
       8:    f85d 7b04     ldr.w    r7, [r13], #4
       c:    4770          bx    r14


  $ arm-linux-gnueabi-gcc -mthumb -O1 -c tst.c
  $ arm-linux-gnueabi-objdump -d -M reg-names-raw  tst.o

    tst.o:     file format elf32-littlearm

    Disassembly of section .text:

    00000000 <a>:
       0:    4770          bx    r14


> I had forgotten about the clang issue though --- is it actually
> silently generating bad code that doesn't respect the constraint? Or
> something else? If so we probably need a separate way to detect it.

As far as I understand it, clang doesn't correctly identify that
inline asm is using r7 and assumes that the frame pointer setup at
function entry is valid throughout the function.

  https://bugs.llvm.org/show_bug.cgi?id=34165

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Rich Felker]

On Thu, Oct 26, 2017 at 11:51:17AM -0700, Andre McCurdy wrote:
> >> But perhaps an alternative way to detect whether the current
> >> combination of compiler + cflags is going to try to use frame pointers
> >> is to compile a trivial function to assembler and parse the output. I
> >> haven't tested clang, but gcc adds a helpful "frame_needed" comment
> >> which is easy to grep for.
> >
> > This is not a good approach. It depends on specific compiler behavior
> > (text that's not part of the code) and thus has both false negatives
> > and false positives (it would break on compilers that allow you to use
> > r7 in asm constraints even when the compiler is using frame pointers).
> 
> Yes, agreed. Just checking for the gcc comment isn't robust. But I
> think there are other differences between the two cases which could be
> detected reliably with a slightly more elaborate test, e.g. checking
> for the use of r7 in the object code (assuming that for a trivial
> function which just returns there's no reason that the compiler would
> ever use of r7 except for a frame pointer).

That's not really a reasonable assumption. There are all sorts of
reasons the compiler might use a particular register even in a trivial
function, for instrumentation, sanitizer, etc. type reasons. Or, of
course, as a frame pointer. The use of r7 is not what means we can't
use the inline syscall asm. Rather, the compiler bug whereby it fails
to let you use r7 in inline asm because it's doing the
reservation-for-frame-pointer incorrectly (GCC), or where it silently
generates wrong code (clang), is the reason that a workaround is
needed in some cases. If GCC started handling the r7 constraint fine,
spilling the frame pointer before the asm block and restoring it
afterward so that the constraint could be met, there would not be any
problem.

Rich

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Andre McCurdy]

On Thu, Oct 26, 2017 at 5:33 PM, Rich Felker <dalias@libc.org> wrote:
> On Thu, Oct 26, 2017 at 11:51:17AM -0700, Andre McCurdy wrote:
>> >> But perhaps an alternative way to detect whether the current
>> >> combination of compiler + cflags is going to try to use frame pointers
>> >> is to compile a trivial function to assembler and parse the output. I
>> >> haven't tested clang, but gcc adds a helpful "frame_needed" comment
>> >> which is easy to grep for.
>> >
>> > This is not a good approach. It depends on specific compiler behavior
>> > (text that's not part of the code) and thus has both false negatives
>> > and false positives (it would break on compilers that allow you to use
>> > r7 in asm constraints even when the compiler is using frame pointers).
>>
>> Yes, agreed. Just checking for the gcc comment isn't robust. But I
>> think there are other differences between the two cases which could be
>> detected reliably with a slightly more elaborate test, e.g. checking
>> for the use of r7 in the object code (assuming that for a trivial
>> function which just returns there's no reason that the compiler would
>> ever use of r7 except for a frame pointer).
>
> That's not really a reasonable assumption. There are all sorts of
> reasons the compiler might use a particular register even in a trivial
> function, for instrumentation, sanitizer, etc. type reasons.

True. But these are all false positives. If any of these other reasons
were to cause musl to use out of line syscalls then there's no real
negative impact, apart from a missed optimisation.

If I understand the glibc approach correctly it doesn't do any kind of
test and always falls back to out of line syscalls for Thumb. Even if
musl added a test which could generate a false positive it would be
doing better than glibc.

> Or, of
> course, as a frame pointer. The use of r7 is not what means we can't
> use the inline syscall asm. Rather, the compiler bug whereby it fails
> to let you use r7 in inline asm because it's doing the
> reservation-for-frame-pointer incorrectly (GCC), or where it silently
> generates wrong code (clang), is the reason that a workaround is
> needed in some cases. If GCC started handling the r7 constraint fine,
> spilling the frame pointer before the asm block and restoring it
> afterward so that the constraint could be met, there would not be any
> problem.

Doesn't enabling frame pointers imply any kind of guarantee that the
fp register is valid throughout the function? If the compiler is free
to spill and restore the fp register then I agree, but up to now I've
been assuming the compiler wasn't free to do that.

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Andre McCurdy]

On Thu, Oct 26, 2017 at 7:17 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
> On Thu, Oct 26, 2017 at 5:33 PM, Rich Felker <dalias@libc.org> wrote:
>> On Thu, Oct 26, 2017 at 11:51:17AM -0700, Andre McCurdy wrote:
>>> >> But perhaps an alternative way to detect whether the current
>>> >> combination of compiler + cflags is going to try to use frame pointers
>>> >> is to compile a trivial function to assembler and parse the output. I
>>> >> haven't tested clang, but gcc adds a helpful "frame_needed" comment
>>> >> which is easy to grep for.
>>> >
>>> > This is not a good approach. It depends on specific compiler behavior
>>> > (text that's not part of the code) and thus has both false negatives
>>> > and false positives (it would break on compilers that allow you to use
>>> > r7 in asm constraints even when the compiler is using frame pointers).
>>>
>>> Yes, agreed. Just checking for the gcc comment isn't robust. But I
>>> think there are other differences between the two cases which could be
>>> detected reliably with a slightly more elaborate test, e.g. checking
>>> for the use of r7 in the object code (assuming that for a trivial
>>> function which just returns there's no reason that the compiler would
>>> ever use of r7 except for a frame pointer).
>>
>> That's not really a reasonable assumption. There are all sorts of
>> reasons the compiler might use a particular register even in a trivial
>> function, for instrumentation, sanitizer, etc. type reasons.
>
> True. But these are all false positives. If any of these other reasons
> were to cause musl to use out of line syscalls then there's no real
> negative impact, apart from a missed optimisation.
>
> If I understand the glibc approach correctly it doesn't do any kind of
> test and always falls back to out of line syscalls for Thumb. Even if
> musl added a test which could generate a false positive it would be
> doing better than glibc.

For some additional context, this appears to be the glibc solution for
similar issues on x86:

  https://sourceware.org/bugzilla/show_bug.cgi?id=21029
  https://sourceware.org/git/gitweb.cgi?p=glibc.git;a=commitdiff;h=3b33d6ed6096c1d20d05a650b06026d673f7399a

It seems to be based around a configure test which relies on the gcc
compile time error to determine whether or not it's possible to inline
syscalls.

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Adhemerval Zanella]

On 26/10/2017 15:00, Rich Felker wrote:
> On Thu, Oct 26, 2017 at 02:48:11PM -0200, Adhemerval Zanella wrote:
>>
>>
>> On 25/10/2017 19:16, Szabolcs Nagy wrote:
>>> * Andre McCurdy <armccurdy@gmail.com> [2017-10-09 09:48:29 -0700]:
>>>> On Sat, Oct 7, 2017 at 8:21 PM, Rich Felker <dalias@libc.org> wrote:
>>>>> On Fri, Oct 06, 2017 at 05:53:38PM -0700, Andre McCurdy wrote:
>>>>> If you do want to test for broken configurations, rather than
>>>>> hard-coding an assumption that some configuration is broken, you
>>>>> should test for it. This would look something like, if ARCH is arm,
>>>>> try compiling a trivial function with inline asm using r7 and see if
>>>>> it fails.
>>>>
>>>> Yes, I came to the same conclusion after seeing the clang bug, which
>>>> seems to suggest that clang uses a frame pointer even with
>>>> optimisation enabled.
>>>>
>>>>> If so, exit with an error or perhaps try adding
>>>>> -fomit-frame-pointer and retrying.
>>>>
>>>> If we over-ride the user supplied CFLAGS then there's probably no need
>>>> to test the behaviour of the compiler - we can just force
>>>> -fomit-frame-pointer unconditionally when compiling for Thumb/Thumb2.
>>>>
>>>> There's a slight complication though that if -fno-omit-frame-pointer
>>>> is present in the user supplied CFLAGS then adding
>>>> -fomit-frame-pointer to CFLAGS_AUTO won't over-ride it (since CFLAGS
>>>> appears on the final compiler command line after CFLAGS_AUTO).
>>>>
>>>> Would it be OK for the configure script to append to CFLAGS? Or should
>>>> the configure script perhaps setup a new variable (CFLAGS_FORCE?)
>>>> which the Makefile would then add to CFLAGS_ALL after CFLAGS?
>>>>
>>>
>>> glibc works this around in thumb mode by extern syscall asm
>>> (of course it cannot guarantee that r7 is a frame pointer at
>>> all times, an interrupt can observe r7 with syscall num in it,
>>> i'm not sure if that's acceptable for users who compile with
>>> frame-pointers, in musl there is some asm code which wont
>>> have fp setup anyway).
>>>
>>> http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/arm/sysdep.h;h=6a64351cdd87c2041d639a17efc9f681262d5e3f;hb=HEAD#l335
>>>
>>
>> Why do you mean by glibc strategy might not be acceptable? What
>> kind of issue are you referring on interrupt case?
> 
> If you're compiling with frame pointers because you want them to be
> present (and always valid) for debugging purposes or similar, there's
> no way to achieve that while making syscalls -- and the most likely
> place for a process to get stopped debugging is usually at a syscall.
> Maybe this doesn't matter. It's not something we can change, just an
> observation about a problem with the ABI, I think.

I think this might be a problem for musl where it does not provide
unwind information through CFI.  For debugging with GLIBC, afaik GDB
will these information along libgcc unwind symbols to get correct call
frame and libc-do-syscall.S does seems to have correct CFI annotations.

> 
> I think what we could do to ensure that compiling with frame pointers
> otherwise works is add a configure test for use of r7 in inline asm,
> and if it fails, -D something that would cause arch/arm/syscall_arch.h
> not to define any syscall inlines but instead #define
> SYSCALL_NO_INLINE which results in src/internal/syscall.h just using
> the external __syscall function (which will of course clobber r7 for
> the duration of the syscall).
> 
> Rich
> 

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Rich Felker]

On Thu, Oct 26, 2017 at 04:46:04PM -0200, Adhemerval Zanella wrote:
> >>> http://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/arm/sysdep.h;h=6a64351cdd87c2041d639a17efc9f681262d5e3f;hb=HEAD#l335
> >>>
> >>
> >> Why do you mean by glibc strategy might not be acceptable? What
> >> kind of issue are you referring on interrupt case?
> > 
> > If you're compiling with frame pointers because you want them to be
> > present (and always valid) for debugging purposes or similar, there's
> > no way to achieve that while making syscalls -- and the most likely
> > place for a process to get stopped debugging is usually at a syscall.
> > Maybe this doesn't matter. It's not something we can change, just an
> > observation about a problem with the ABI, I think.
> 
> I think this might be a problem for musl where it does not provide
> unwind information through CFI.  For debugging with GLIBC, afaik GDB
> will these information along libgcc unwind symbols to get correct call
> frame and libc-do-syscall.S does seems to have correct CFI annotations.

In that case, it might make sense to add debug or runtime unwind
information to pure asm functions that clobber the frame pointer,
which would basically just be __syscall I think. But this is a
separate topic from fixing the build issue at hand.

Rich

Re: How to handle attempts to combine ARM Thumb with frame pointers?

[Szabolcs Nagy]

* Adhemerval Zanella <adhemerval.zanella@linaro.org> [2017-10-26 16:46:04 -0200]:
> On 26/10/2017 15:00, Rich Felker wrote:
> > If you're compiling with frame pointers because you want them to be
> > present (and always valid) for debugging purposes or similar, there's
> > no way to achieve that while making syscalls -- and the most likely
> > place for a process to get stopped debugging is usually at a syscall.
> > Maybe this doesn't matter. It's not something we can change, just an
> > observation about a problem with the ABI, I think.
> 
> I think this might be a problem for musl where it does not provide
> unwind information through CFI.  For debugging with GLIBC, afaik GDB
> will these information along libgcc unwind symbols to get correct call
> frame and libc-do-syscall.S does seems to have correct CFI annotations.
> 

if you configure musl with --enable-debug (i.e. -g cflag) then you
get cfi annotations, but not for asm code (except on some targets
we do generate cfi for asm, but not yet on arm)

there are no runtime unwind tables though, so libgcc would not work,
however that does not work at interrupts in glibc either because arm
unwind tables are not asynchronous.

i think some tools prefer to use frame pointers for unwinding
(e.g. perf call graph profiler uses fp by default, not sure how
that works on arm thumb)