From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 Received: from second.openwall.net (second.openwall.net [193.110.157.125]) by inbox.vuxu.org (Postfix) with SMTP id D56892396E for ; Thu, 15 Feb 2024 01:22:34 +0100 (CET) Received: (qmail 12106 invoked by uid 550); 15 Feb 2024 00:19:29 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 12068 invoked from network); 15 Feb 2024 00:19:28 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707956540; x=1708561340; darn=lists.openwall.com; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=hgzlQexFCkKZRwcDdAHDMeEKpDvKfRDGAqdyIKWX+Ow=; b=xfgyXD6uHhxmP76YJXuLDxokd+put5Xs9oG7AnDPvMa4DX9CUam7JP6nwbHxw9hwmb 9my7RJpBtECoCURe096tPBFis2Ty2gXJj7OMGrsiGQ38Qaq9lYQvQZbc4WO3ZnOYYRFY YoGHgduLcIxT7ZKnsxVtJMxtO2YcpIsmPYt/Tz7Wjcyw6HRtXlsPtuAEzd3rY3hFaK53 pdCZEA9u2W1quV+jX4qu9V6bklxBAu25V4P/CkUHslGjrbgZyVphRmyRdY732F47E8mU tsQTbIG1PETnPVBH/+twiLlvjrJQYzjc4CxR7AHH9Nrf7N8/KBauZ9Uc4wyAzYAHH3Dy 2HxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707956540; x=1708561340; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hgzlQexFCkKZRwcDdAHDMeEKpDvKfRDGAqdyIKWX+Ow=; b=VYPeuYvhfKmLVYa7AJhj3Pfag45otGFdCFskjidmAqIEn3gpFdXb3i0KQnMnLp3aWi pqUOWBMuCloZ1gcyVELSzr8l9HW3re5GGM5dIRSJprKiGS41kfWZ+beMp5sKU1JD6oeD Cyuqgm86AZiDQFyeDiTOaP9CNYM9rqcAfiOYG5oWkwo0ZRoA8Nq+mwle7q++hdJfllvK oVMLjx0pbJ4saNkaAgGDunvnjem6Coyw3IJ2q4V5eAu84Isj3c+Q9J41tD5IofpY4uTH b5LrF5aC/UfoD0l1BJeD8+H8wrQ+QOG0UAH6acF2n6fArAk99AZsnm29zfOb0nPwNbIc 0hLA== X-Forwarded-Encrypted: i=1; AJvYcCWMAtYUTvHr4TYHlHe0zfYkcMpWSJ4UzcQ6K03gWyg4sP8IlhFrZsV7AOglTjzaqZ5/OMhZ+XIjABk5f60qvY+1PtNpILgmbw== X-Gm-Message-State: AOJu0YwnDG9ePc/DlgSmh+VxxgvDgUPX4nFVhZKYgHFCdzNvLxNnBJ8R OucHJJm/XLDbYmAnvy4NjTvTpqK4bRYbz7kOMvr9jiy0FO0qtk+eTghRgVRAiOHVR6ErXrBTK1Q l1Fnb4m4Nfw9oWvtbSiqhTmj9JDCv/fp8JSXn X-Google-Smtp-Source: AGHT+IGgEv38Fnp4O0aUWHtniFnsXDh0nVcY9BqPiv0yCc5Kg9ndR64QW9J4a3IJzYGZ+ay1Q1+svY1ywNSxXuVMYyg= X-Received: by 2002:a05:6214:c2b:b0:68c:d991:d2f8 with SMTP id a11-20020a0562140c2b00b0068cd991d2f8mr231567qvd.65.1707956540057; Wed, 14 Feb 2024 16:22:20 -0800 (PST) MIME-Version: 1.0 References: <20240212184236.GZ4163@brightrain.aerifal.cx> <20240212224657.GA4163@brightrain.aerifal.cx> <20240213020834.GB4163@brightrain.aerifal.cx> <20240215000353.GA1884416@port70.net> In-Reply-To: <20240215000353.GA1884416@port70.net> From: enh Date: Wed, 14 Feb 2024 16:22:05 -0800 Message-ID: To: Rich Felker , William Roberts , enh , musl@lists.openwall.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [musl] PAC/BTI Support on aarch64 On Wed, Feb 14, 2024 at 4:03=E2=80=AFPM Szabolcs Nagy wrot= e: > > * Rich Felker [2024-02-12 21:08:34 -0500]: > > On Mon, Feb 12, 2024 at 05:18:22PM -0600, William Roberts wrote: > > > On Mon, Feb 12, 2024 at 5:05=E2=80=AFPM enh wrote: > > > > On Mon, Feb 12, 2024 at 2:46=E2=80=AFPM Rich Felker wrote: > > > > > On Mon, Feb 12, 2024 at 03:25:48PM -0600, William Roberts wrote: > > > > > > It's a matter of building with -mbranch-protection=3Dstandard > > > > > > > > > > > > Just the ASM labels need the first instruction to be a BTI. The= y're in > > > > > > the NOP space > > > > > > so they are backwards compatible, older hardware will just NOP = it. > > not quite that simple. sorry long brain dump follows: > > tl;dr: i think the main issues are asm handling (property notes > and cfi for pac-ret), property note handling in ld.so, perf > overhead and possible compat issues of pac-ret (not possible > to disable per process) and testing (ensuring the code works > when pac/bti is not nop). > > - asm code needs manual marking. > > GNU_PROPERTY note in asm is ugly and error prone, see > https://github.com/ARM-software/optimized-routines/blob/master/string/aar= ch64/asmdefs.h#L23 > > i.e. no equivalent to -Wa,--noexecstack we use for GNU_STACK > (this was an oversight, llvm added an option, binutils gas has none). what's the option? (since Android only supports llvm, that might be worth considering as a slight cleanup for us...) > - if asm code does indirect tailcall it should use x16 or x17. > > bti c is compatible with the indirect branch that way. > > - only functions that *maybe called indirectly* need bti c. > > this turned out to be trickier than expected: lld/llvm and bfd.ld/gcc > currently disagree about the definition in case of linker inserted > veneers: gcc assumes ld handles the case if an inserted veneer > indirectly branches to a non-bti location, llvm emits bti in all > functions (including local ones without their address taken), just > in case ld inserts an indirect veneer. > https://github.com/ARM-software/abi-aa/issues/196 > > there is some difference to how PLTs are emitted between the > linkers, but i don't think that causes compat issues (might cause > trouble for tools that try to interpret the PLT). > > - dynamic linker has to figure out when to enable it. > > systemd MDWE (memory deny write exec) feature used to seccomp > filter mprotect(PROT_EXEC) so even if the underlying mapping was > already PROT_EXEC and it just added a PROT_BTI on top, mprotect > would fail. this was fixed by adding an MDWE prctl to linux for > systemd to make it stop using that filter, but there may be other > similar seccomp filters and old kernels without MDWE so glibc > re-mmaps the exec segment (which systemd happily accepts). > > note: the bit that tells if a load segment needs to be mapped as > PROT_BTI is in the load segment (usually program headers are in > the executable segment) so first mmap cannot get it right, unless > a quick read of the prog headers are done before mmap but that has > a lot of failure modes (the size can be unbounded) and does not > gain much compared to just mmap twice. except when the exe is mapped > by the kernel, then we cannot mmap since there is no fd and mprotect > may fail). > MDWE prctl: https://lwn.net/Articles/937315/ > > another detail is that static-exe / ld.so / vdso marking is handled > by the kernel, while other dsos are PROT_BTI marked by ld.so. an > interesting case is dynamic linked exe which was originally handled > by ld.so, but after the MDWE fiasco the kernel started loading it > with PROT_BTI (i.e. now all binaries mapped by the kernel are BTI > protected by the kernel). > > ld.so should also take care to gracefully handle BTI protection > failure (or invalid notes) in dlopen. (although invalid note is > more of an x86 thing.) > > - special functions may need to return indirectly instead of ret > > the only really nasty one is swapcontext which is not supported by > musl so that's good (and it is only a problem if bti is used with > a shadow-stack-like feature). > > for returns_twice functions (like setjmp) the compiler emits bti j > at the call site so the second return can use an indirect branch. > return from unwinder to an exception handling landing pad via > indirect branch is supported too. otherwise return via indirect > branch is not supported (no bti j at call sites). > > - a64fx (hpc core) implemented hint nops in a slow way > > so glibc only adds bti if glibc is configured for bti. > https://sourceware.org/pipermail/libc-alpha/2021-May/125784.html > > > > > > I think it's a little more elaborate than that. Those asm instruc= tions > > > > > need to be added (probably as .instr or .word or something, unles= s > > > > > there's a way to spell this particular nop that existing tooling = will > > > > > understand). > > > > > > You just use the hint instructions, they are understood b= y old > > > toolchains. But you can only support a subset of the BTI/PAC instruct= ions > > yes 'hint ' is armv8.0-a, part of the base isa. > > i think the non-hint pac instructions are not relevant to musl. > > > > but it's been enough for most projects that follow the normal ABI con= ventions > > > like OpenSSL/BoringSSL,etc, but not enough for libffi for example. > > as far as i know openssl is the reason android does not enable pac: > they added the hint instructions incorrectly at first so there are > binaries that fail if the hw enables pac. that was one reason why "android does not enable pac" _by default in the android target triples for app developers_, yes --- though i think we're at the point where we think we should flip that default (not least because the number of users whose devices would actually _benefit_ from the extra instructions is a lot larger now!): https://github.com/android/ndk/issues/1914 > this reveals another issue with pac/bti: since they are nops on most > existing hw, they are not properly tested (e.g. by distro QA) so a > binary can look ok until you move it to a newer machine. (but recent > amazon graviton 4 has pac/bti so we may get more coverage soon.) exactly --- that was one of our key concerns, that app developers would _think_ they've tested with pac/bti but unknowingly used a device without. (or even an x86-64 emulator!) > > > > > I also wondered if [sig]setjmp/longjmp would be affected, but pro= bably > > > > > not. > > > > > > > > bionic does use PAC, but i think glibc has its own "pointer manglin= g" thing? > > > > > > You need it, as the first instruction from a branch (where longjmp re= turns to) > > > needs to be a BTI instruction. > > > > Is that different from a normal function return? > > > > Note that in the case of sigsetjmp, (sig)longjmp returns to a point > > inside the sigsetjmp asm, so that point needs the annotation I think. > > setjmp/sigsetjmp has to decide how to protect the longjmp return. > > with a shadow-stack-like bw-edge-cfi, longjmp cannot return with > ret (first ret from setjmp consumes the return address from the > shadow stack), it must use indirect branch. (there is now gcs > which is aarch64 shadow stack, linux support is in progress). > > since jmpbuf does not expose the return address representation a > libc specific protection can be applied (mangling or pac) and then > longjmp can use ret and remain protected. (but e.g. setcontext > exposes the pc/lr so those cannot be mangled in memory). > > compilers emit bti j at the call site of returns_twice functions > to allow both ret and indirect branch. musl sigsetjmp can decide > what it does. > > > > > > > It's been done for many projects, glibc and bionic have it. The > > > > > > problem with BTI is that when one item in the link > > > > > > list doesn't support BTI the loader/linker turns it off. So whe= n it's > > > > > > something like a libc that is fundamental in the link chain, > > > > > > it turns it off for everything. > > > > > > > > > > This presumably requires some kind of machinery for how dynamic > > > > > linking will work, and possibly turning it off if a library witho= ut it > > > > > is dlopened? > > > > > > > > > > My understanding doing some brief searches though was that you ca= n > > > > > individually mprotect it off in certain regions. So maybe it's > > > > > possible to just enable only for DSOs that support it? > > > > > > > > correct. > > > > OK, that's good to know. So which direction is it? Do DSOs that > > support BTI need it explicitly turned on via mprotect/mmap flags? Or > > is there some process-global flag to turn it on, and then ones that > > don't support it need it turned off? > > per dso marking and explicit PROT_BTI setting. > > > I suspect it's possible to first enable BTI for third-party libraries > > as a feature of the dynamic linker, and add BTI support for libc > > itself as a separate thing. That might be a nice factoring to make > > changes minimal and easy for ppl to read. > > for third-party, it is enough to fix crt*.o and handle markings > in ld.so. > > (but of course for bti to be effective the libc must have it too, > otherwise there are likely enough gadgets to do whatever) > > > The changes in dynlink.c should be as arch-agnostic as possible. If > > there's a corresponding feature on other archs, it should use the same > > basic code, with arch-specific headers (arch/$ARCH/reloc.h) defining > > the mechanisms for evaluating if an ELF file is compatible, how to do > > the mprotect, etc. > > generic code should work. > > but e.g. see above about who handles the marking (kernel vs ld.so), > turns out x86 (and likely aarch64, riscv) shadow stack uses different > rules: always libc handles the marking. so there are caveats. > > > > > > > The initial scope of code changes would be what's reported when > > > > > > LDFLAGS=3D-Wl,-zforce-bti,--fatal-warnings > > > > > > > > > > Is there a way to disable these warnings so that every asm file d= oes > > > > > not need to be cluttered with annotations? > > > > > > > > well, that's the ELF note stuff i was talking about, and if you don= 't > > > > have it you'll fall foul of the static linker saying "not all this > > > > code is BTI-enabled, therefore this .so isn't", and the dynamic lin= ker > > > > doing nothing because the static linker effectively tells it not to= . > > > > > > Yep, well said ENH. It's been since Android since we crossed paths :-= ). > > > > > > It's not that hard to annotate an asm file :-p I forget what project > > > (I think it was gnutls, but they just use openssl's code for the asm) > > > but I just put it in a header file and by virtue of #include'ing it y= ou get the > > > notes added. > > > > Yes, we generally don't do that. There are no "asm headers" in musl; > > all asm files are self-contained and readable standalone. So if > > there's no way to tell the assembler/linker from the command line that > > files are BTI-compatible without generating a huge load of warning > > spam, I guess it's a mess of copy-and-paste... > > currently there is only the ugly asm directives, see above. > > final notes: > > bti is fairly deployable (iirc x86 ibt failed because it is not per > dso, so dlopen does not really work, but aarch64 does not have that > problem), not strong security (the final binary is littered with > bti j/c so plenty opportunity to misdirect an indirect jump/call), > but at least it has minimal impact (minimal compatibility issues and > on a modern core even when bti is enabled it should not be slower). > > for pac every function can independently decide if it uses pac-ret > (aka return address signing), no need for per dso marking. however > it has bigger compat as well as performance impact: > > there are custom unwinders, pac-ret uses a new dwarf cfi (to mark > the code regions where the return address is signed), custom > unwinders may not understand this and that's a runtime crash. > > some code looks at or modifies the return address (various hacks), such > code needs to be updated or not use pac-ret in relevant functions, but > such issues are hard to discover without hw. > > pac is a per-boot system-wide setting, not per-process, so if there is > any issue or bug there is no way to disable it for one broken binary > (nowadays there is a disable prctl, but it is documented to slow the > system down, so not suitable for working around perf issues). > > on simple cores pac can be slow (can add latency to non-leaf functions) > > with 48bit va space, there is only 7bit pac, i.e. limited protection.