[musl] Potential DL_NOMMU_SUPPORT bug.

[musl] Potential DL_NOMMU_SUPPORT bug.

[Andrew Rogers]

[-- Attachment #1.1: Type: text/plain, Size: 951 bytes --]

Hi,

I was trying a DL_NOMMU_SUPPORT build so I could load binaries from the
sdcard on an android device. I managed to succeed but only after making a
mod which I later realised might apply beyond my application.

The mmap_fixed() function would return as if successful even when mmap()
call had failed

Hopefully the link and the patch below help.

Regards
Andrew

https://git.musl-libc.org/cgit/musl/tree/ldso/dynlink.c?id=85e0e3519655220688e757b9d5bfd314923548bd#n584

diff -Naur musl-1.2.2-orig/ldso/dynlink.c musl-1.2.2-new/ldso/dynlink.c
--- musl-1.2.2-orig/ldso/dynlink.c 2021-01-15 02:26:00.000000000 +0000
+++ musl-1.2.2-new/ldso/dynlink.c 2021-01-23 06:26:26.861158169 +0000
@@ -581,7 +581,7 @@
  char *q;
  if (!no_map_fixed) {
  q = mmap(p, n, prot, flags|MAP_FIXED, fd, off);
- if (!DL_NOMMU_SUPPORT || q != MAP_FAILED || errno != EINVAL)
+ if (!DL_NOMMU_SUPPORT && q != MAP_FAILED && errno != EINVAL)
  return q;
  no_map_fixed = 1;
  }

[-- Attachment #1.2: Type: text/html, Size: 1326 bytes --]

[-- Attachment #2: musl-DL_NOMMU_SUPPORT-dynlink.patch --]
[-- Type: text/x-patch, Size: 485 bytes --]

diff -Naur musl-1.2.2-orig/ldso/dynlink.c musl-1.2.2-new/ldso/dynlink.c
--- musl-1.2.2-orig/ldso/dynlink.c	2021-01-15 02:26:00.000000000 +0000
+++ musl-1.2.2-new/ldso/dynlink.c	2021-01-23 06:26:26.861158169 +0000
@@ -581,7 +581,7 @@
 	char *q;
 	if (!no_map_fixed) {
 		q = mmap(p, n, prot, flags|MAP_FIXED, fd, off);
-		if (!DL_NOMMU_SUPPORT || q != MAP_FAILED || errno != EINVAL)
+		if (!DL_NOMMU_SUPPORT && q != MAP_FAILED && errno != EINVAL)
 			return q;
 		no_map_fixed = 1;
 	}

Re: [musl] Potential DL_NOMMU_SUPPORT bug.

[Rich Felker]

On Sat, Jan 23, 2021 at 06:47:00AM +0000, Andrew Rogers wrote:
> Hi,
> 
> I was trying a DL_NOMMU_SUPPORT build so I could load binaries from the
> sdcard on an android device. I managed to succeed but only after making a
> mod which I later realised might apply beyond my application.
> 
> The mmap_fixed() function would return as if successful even when mmap()
> call had failed
> 
> Hopefully the link and the patch below help.
> 
> Regards
> Andrew
> 
> https://git.musl-libc.org/cgit/musl/tree/ldso/dynlink.c?id=85e0e3519655220688e757b9d5bfd314923548bd#n584
> 
> diff -Naur musl-1.2.2-orig/ldso/dynlink.c musl-1.2.2-new/ldso/dynlink.c
> --- musl-1.2.2-orig/ldso/dynlink.c 2021-01-15 02:26:00.000000000 +0000
> +++ musl-1.2.2-new/ldso/dynlink.c 2021-01-23 06:26:26.861158169 +0000
> @@ -581,7 +581,7 @@
>   char *q;
>   if (!no_map_fixed) {
>   q = mmap(p, n, prot, flags|MAP_FIXED, fd, off);
> - if (!DL_NOMMU_SUPPORT || q != MAP_FAILED || errno != EINVAL)
> + if (!DL_NOMMU_SUPPORT && q != MAP_FAILED && errno != EINVAL)
>   return q;
>   no_map_fixed = 1;
>   }

> diff -Naur musl-1.2.2-orig/ldso/dynlink.c musl-1.2.2-new/ldso/dynlink.c
> --- musl-1.2.2-orig/ldso/dynlink.c	2021-01-15 02:26:00.000000000 +0000
> +++ musl-1.2.2-new/ldso/dynlink.c	2021-01-23 06:26:26.861158169 +0000
> @@ -581,7 +581,7 @@
>  	char *q;
>  	if (!no_map_fixed) {
>  		q = mmap(p, n, prot, flags|MAP_FIXED, fd, off);
> -		if (!DL_NOMMU_SUPPORT || q != MAP_FAILED || errno != EINVAL)
> +		if (!DL_NOMMU_SUPPORT && q != MAP_FAILED && errno != EINVAL)
>  			return q;
>  		no_map_fixed = 1;
>  	}

The condition was correct as written. If any of the 3 are true, there
is no fallback to be done:

- If it's not an arch that could be nommu, mmap can be expected to
  work. Any error should be reported rather than attempting to
  emulate.

- If the return value isn't failure, it already succeeded, so of
  course you don't want to emulate on top of that.

- If the call failed but errno is something other than EINVAL (the
  error mmap returns on nommu when it can't do MAP_FIXED) then it's an
  error to report rather than emulating.

Could you clarify what you're trying to do? Android devices are not
nommu and loading the binary from SD card vs elsewhere should not be
relevant to mmap failure here.

Rich

Re: [musl] Potential DL_NOMMU_SUPPORT bug.

[Alexander Monakov]

On Sun, 24 Jan 2021, Rich Felker wrote:

> Could you clarify what you're trying to do? Android devices are not
> nommu and loading the binary from SD card vs elsewhere should not be
> relevant to mmap failure here.

sdcard [pseudo-]partition is usually mounted noexec, so mmap with PROT_EXEC
should fail.

Alexander

Re: [musl] Potential DL_NOMMU_SUPPORT bug.

[Rich Felker]

On Sun, Jan 24, 2021 at 06:58:10PM +0300, Alexander Monakov wrote:
> On Sun, 24 Jan 2021, Rich Felker wrote:
> 
> > Could you clarify what you're trying to do? Android devices are not
> > nommu and loading the binary from SD card vs elsewhere should not be
> > relevant to mmap failure here.
> 
> sdcard [pseudo-]partition is usually mounted noexec, so mmap with PROT_EXEC
> should fail.

Uhg, that makes no sense. Does it enforce that even for MAP_PRIVATE,
which should semantically be equivalent to just making anon memory
with the requested permissions and copying the file contents into it??

Rich

Re: [musl] Potential DL_NOMMU_SUPPORT bug.

[Alexander Monakov]

> > sdcard [pseudo-]partition is usually mounted noexec, so mmap with PROT_EXEC
> > should fail.
> 
> Uhg, that makes no sense. Does it enforce that even for MAP_PRIVATE,
> which should semantically be equivalent to just making anon memory
> with the requested permissions and copying the file contents into it??

I think it makes sense: isn't the entire point of 'noexec' that a user
who has write access only to noexec filesystems will not be able to run
arbitrary binary code (assuming the already-present binaries are not
cooperative, unlike musl ld.so with the above patch would be)? Enforcing
noexec for MAP_PRIVATE ensures the users can not trivially side-step
noexec by invoking ld.so (without extra checks on ld.so side).

Alexander

Re: [musl] Potential DL_NOMMU_SUPPORT bug.

[Rich Felker]

On Sun, Jan 24, 2021 at 09:48:11PM +0300, Alexander Monakov wrote:
> > > sdcard [pseudo-]partition is usually mounted noexec, so mmap with PROT_EXEC
> > > should fail.
> > 
> > Uhg, that makes no sense. Does it enforce that even for MAP_PRIVATE,
> > which should semantically be equivalent to just making anon memory
> > with the requested permissions and copying the file contents into it??
> 
> I think it makes sense: isn't the entire point of 'noexec' that a user
> who has write access only to noexec filesystems will not be able to run
> arbitrary binary code (assuming the already-present binaries are not
> cooperative, unlike musl ld.so with the above patch would be)? Enforcing
> noexec for MAP_PRIVATE ensures the users can not trivially side-step
> noexec by invoking ld.so (without extra checks on ld.so side).

I always viewed noexec (as opposed to something like nosuid) as a
non-security-boundary, a sort of soft block for mounting filesystems
that you don't want to execute programs from, for example a disk image
known to contain malware that you're analyzing or a flash drive not
expected to contain meaningful executable data but where all files
would appear as +x due to FAT limitations. The expectation is that it
can be bypassed. With a "restricted shell" type environment (very
careful selection of what programs are present), it can plausibly be
turned into a (very fragile) security boundary, but I didn't expect
the kernel to be making weird rules to facilitate that.

In any case, it seems that's how it is, and inability to dlopen (or
LD_LIBRARY_PATH+DT_NEEDED or whatnot) from a noexec mount is
annoying...

Rich

Re: [musl] Potential DL_NOMMU_SUPPORT bug.

[Andrew Rogers]

[-- Attachment #1: Type: text/plain, Size: 2641 bytes --]

On Sun, Jan 24, 2021 at 6:55 PM Rich Felker <dalias@libc.org> wrote:

> On Sun, Jan 24, 2021 at 09:48:11PM +0300, Alexander Monakov wrote:
> > > > sdcard [pseudo-]partition is usually mounted noexec, so mmap with
> PROT_EXEC
> > > > should fail.
> > >
> > > Uhg, that makes no sense. Does it enforce that even for MAP_PRIVATE,
> > > which should semantically be equivalent to just making anon memory
> > > with the requested permissions and copying the file contents into it??
> >
> > I think it makes sense: isn't the entire point of 'noexec' that a user
> > who has write access only to noexec filesystems will not be able to run
> > arbitrary binary code (assuming the already-present binaries are not
> > cooperative, unlike musl ld.so with the above patch would be)? Enforcing
> > noexec for MAP_PRIVATE ensures the users can not trivially side-step
> > noexec by invoking ld.so (without extra checks on ld.so side).
>
> I always viewed noexec (as opposed to something like nosuid) as a
> non-security-boundary, a sort of soft block for mounting filesystems
> that you don't want to execute programs from, for example a disk image
> known to contain malware that you're analyzing or a flash drive not
> expected to contain meaningful executable data but where all files
> would appear as +x due to FAT limitations. The expectation is that it
> can be bypassed. With a "restricted shell" type environment (very
> careful selection of what programs are present), it can plausibly be
> turned into a (very fragile) security boundary, but I didn't expect
> the kernel to be making weird rules to facilitate that.
>
> In any case, it seems that's how it is, and inability to dlopen (or
> LD_LIBRARY_PATH+DT_NEEDED or whatnot) from a noexec mount is
> annoying...
>
> Rich
>

Thank you very much for your responses. I am reassured that there is no bug
and that my patch just provides a convenient workaround for my use case.
Albeit by accident rather than design!

I am attempting to load binary executables and shared libraries from the
sdcard on Android. My patch does allow me to execute the busybox binary
from sdcard if I load them using my patched musl. I have not yet tried
loading any shared libraries from the sdcard.

An alternative I am experimenting with at the moment is using LLVM and
storing the bitcode on the sdcard and running it under lli.

Your responses are very informative so I might have another look at
patching musl to see if shared libraries can be loaded from sdcard also.
The dlopen function will probably need to be reworked to use open rather
than mmap but I need to learn some more first!

Many thanks,
Andrew

[-- Attachment #2: Type: text/html, Size: 3283 bytes --]

Re: [musl] Potential DL_NOMMU_SUPPORT bug.

[Rich Felker]

On Sat, Jan 30, 2021 at 05:44:36PM +0000, Andrew Rogers wrote:
> On Sun, Jan 24, 2021 at 6:55 PM Rich Felker <dalias@libc.org> wrote:
> 
> > On Sun, Jan 24, 2021 at 09:48:11PM +0300, Alexander Monakov wrote:
> > > > > sdcard [pseudo-]partition is usually mounted noexec, so mmap with
> > PROT_EXEC
> > > > > should fail.
> > > >
> > > > Uhg, that makes no sense. Does it enforce that even for MAP_PRIVATE,
> > > > which should semantically be equivalent to just making anon memory
> > > > with the requested permissions and copying the file contents into it??
> > >
> > > I think it makes sense: isn't the entire point of 'noexec' that a user
> > > who has write access only to noexec filesystems will not be able to run
> > > arbitrary binary code (assuming the already-present binaries are not
> > > cooperative, unlike musl ld.so with the above patch would be)? Enforcing
> > > noexec for MAP_PRIVATE ensures the users can not trivially side-step
> > > noexec by invoking ld.so (without extra checks on ld.so side).
> >
> > I always viewed noexec (as opposed to something like nosuid) as a
> > non-security-boundary, a sort of soft block for mounting filesystems
> > that you don't want to execute programs from, for example a disk image
> > known to contain malware that you're analyzing or a flash drive not
> > expected to contain meaningful executable data but where all files
> > would appear as +x due to FAT limitations. The expectation is that it
> > can be bypassed. With a "restricted shell" type environment (very
> > careful selection of what programs are present), it can plausibly be
> > turned into a (very fragile) security boundary, but I didn't expect
> > the kernel to be making weird rules to facilitate that.
> >
> > In any case, it seems that's how it is, and inability to dlopen (or
> > LD_LIBRARY_PATH+DT_NEEDED or whatnot) from a noexec mount is
> > annoying...
> 
> Thank you very much for your responses. I am reassured that there is no bug
> and that my patch just provides a convenient workaround for my use case.
> Albeit by accident rather than design!
> 
> I am attempting to load binary executables and shared libraries from the
> sdcard on Android. My patch does allow me to execute the busybox binary
> from sdcard if I load them using my patched musl. I have not yet tried
> loading any shared libraries from the sdcard.
> 
> An alternative I am experimenting with at the moment is using LLVM and
> storing the bitcode on the sdcard and running it under lli.
> 
> Your responses are very informative so I might have another look at
> patching musl to see if shared libraries can be loaded from sdcard also.
> The dlopen function will probably need to be reworked to use open rather
> than mmap but I need to learn some more first!

If the system is setup to gratuitously disallow direct PROT_EXEC
mapping from the filesystem, I would just drop the binaries in a tmpfs
and load them from there.

Rich