Re: [musl] PAC/BTI Support on aarch64

[Rich Felker <dalias@libc.org>]

On Mon, Feb 12, 2024 at 05:18:22PM -0600, William Roberts wrote:
> On Mon, Feb 12, 2024 at 5:05 PM enh <enh@google.com> wrote:
> >
> > On Mon, Feb 12, 2024 at 2:46 PM Rich Felker <dalias@libc.org> wrote:
> > >
> > > On Mon, Feb 12, 2024 at 03:25:48PM -0600, William Roberts wrote:
> > > > On Mon, Feb 12, 2024 at 12:42 PM Rich Felker <dalias@libc.org> wrote:
> > > > >
> > > > > On Mon, Feb 12, 2024 at 10:38:50AM -0600, William Roberts wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I was just wondering if there was any work being done to support PAC
> > > > > > and BTI in aarch64? I could add support but didn't want to duplicate
> > > > > > the work.
> > > > >
> > > > > I'm not aware of any active work on this, but before writing a full
> > > > > implementation, it would be really helpful to start with a basic
> > > > > proposal for the scope of changes needed to make it work to assess
> > > > > whether these are manageable and acceptable cost.
> > > >
> > > > It's a matter of building with -mbranch-protection=standard
> > > >
> > > > Just the ASM labels need the first instruction to be a BTI. They're in
> > > > the NOP space
> > > > so they are backwards compatible, older hardware will just NOP it.
> > >
> > > I think it's a little more elaborate than that. Those asm instructions
> > > need to be added (probably as .instr or .word or something, unless
> > > there's a way to spell this particular nop that existing tooling will
> > > understand).
> >
> > depends on your toolchain version. when we added this to bionic, the
> > toolchain work was still happening. so you'll want to test against
> > whatever your oldest-supported toolchain is.
> >
> 
> You just use the hint <immediate> instructions, they are understood by old
> toolchains. But you can only support a subset of the BTI/PAC instructions
> but it's been enough for most projects that follow the normal ABI conventions
> like OpenSSL/BoringSSL,etc, but not enough for libffi for example.

If hint goes all the way back, that's probably fine and ideal to use.

> > > Or it could be made conditional, but that would require
> > > converting any asm that's not already .S files to .S. Not bad, but not
> 
> as in inline asm? Unless it's a branch target, no need.

No, .S (preprocessed) vs .s (not). But if the hint insn works, I think
just having it there unconditionally is probably the way to go.

> > > I also wondered if [sig]setjmp/longjmp would be affected, but probably
> > > not.
> >
> > bionic does use PAC, but i think glibc has its own "pointer mangling" thing?
> 
> You need it, as the first instruction from a branch (where longjmp returns to)
> needs to be a BTI instruction.

Is that different from a normal function return?

Note that in the case of sigsetjmp, (sig)longjmp returns to a point
inside the sigsetjmp asm, so that point needs the annotation I think.

> > > > It's been done for many projects, glibc and bionic have it. The
> > > > problem with BTI is that when one item in the link
> > > > list doesn't support BTI the loader/linker turns it off. So when it's
> > > > something like a libc that is fundamental in the link chain,
> > > > it turns it off for everything.
> > >
> > > This presumably requires some kind of machinery for how dynamic
> > > linking will work, and possibly turning it off if a library without it
> > > is dlopened?
> > >
> > > My understanding doing some brief searches though was that you can
> > > individually mprotect it off in certain regions. So maybe it's
> > > possible to just enable only for DSOs that support it?
> >
> > correct.

OK, that's good to know. So which direction is it? Do DSOs that
support BTI need it explicitly turned on via mprotect/mmap flags? Or
is there some process-global flag to turn it on, and then ones that
don't support it need it turned off?

I suspect it's possible to first enable BTI for third-party libraries
as a feature of the dynamic linker, and add BTI support for libc
itself as a separate thing. That might be a nice factoring to make
changes minimal and easy for ppl to read.

The changes in dynlink.c should be as arch-agnostic as possible. If
there's a corresponding feature on other archs, it should use the same
basic code, with arch-specific headers (arch/$ARCH/reloc.h) defining
the mechanisms for evaluating if an ELF file is compatible, how to do
the mprotect, etc.

> > > > The initial scope of code changes would be what's reported when
> > > > LDFLAGS=-Wl,-zforce-bti,--fatal-warnings
> > >
> > > Is there a way to disable these warnings so that every asm file does
> > > not need to be cluttered with annotations?
> >
> > well, that's the ELF note stuff i was talking about, and if you don't
> > have it you'll fall foul of the static linker saying "not all this
> > code is BTI-enabled, therefore this .so isn't", and the dynamic linker
> > doing nothing because the static linker effectively tells it not to.
> 
> Yep, well said ENH. It's been since Android since we crossed paths :-).
> 
> It's not that hard to annotate an asm file :-p I forget what project
> (I think it was gnutls, but they just use openssl's code for the asm)
> but I just put it in a header file and by virtue of #include'ing it you get the
> notes added.

Yes, we generally don't do that. There are no "asm headers" in musl;
all asm files are self-contained and readable standalone. So if
there's no way to tell the assembler/linker from the command line that
files are BTI-compatible without generating a huge load of warning
spam, I guess it's a mess of copy-and-paste...

Rich