a workaround when mremap() is not functional?

a workaround when mremap() is not functional?

[u-uy74]

Is there any variation to the musl malloc which would make realloc()
work when the Linux ABI implementation lacks a usable mremap()?

Otherwse realloc() under Linux ABI in FreeBSD fails due to mremap()
being mostly a stub and returning -1/ENOMEM for any growth.

LinuxABI under FreeBSD apparently behaves differently than the
Linux kernel, which can be seen as bugs when the purpose is "Linux
compatibility".

OTOH it would be nice to reduce the strength of the conformity
requirements by musl, to be able to accomodate the extra platform
(possibly could help on other platforms too).

Regards,
Rune

Re: a workaround when mremap() is not functional?

[Rich Felker]

On Sun, Jun 11, 2017 at 11:51:14AM +0200, u-uy74@aetey.se wrote:
> Is there any variation to the musl malloc which would make realloc()
> work when the Linux ABI implementation lacks a usable mremap()?
> 
> Otherwse realloc() under Linux ABI in FreeBSD fails due to mremap()
> being mostly a stub and returning -1/ENOMEM for any growth.
> 
> LinuxABI under FreeBSD apparently behaves differently than the
> Linux kernel, which can be seen as bugs when the purpose is "Linux
> compatibility".
> 
> OTOH it would be nice to reduce the strength of the conformity
> requirements by musl, to be able to accomodate the extra platform
> (possibly could help on other platforms too).

This was already on my radar because mremap for enlarging anon
mappings is also broken on nommu Linux. I think I just need to add a
"if failed, goto the existing malloc-and-memcpy-and-free code"
one-liner.

Rich

Re: a workaround when mremap() is not functional?

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

On Sun, Jun 11, 2017 at 09:17:13AM -0400, Rich Felker wrote:
> On Sun, Jun 11, 2017 at 11:51:14AM +0200, u-uy74@aetey.se wrote:
> > Is there any variation to the musl malloc which would make realloc()
> > work when the Linux ABI implementation lacks a usable mremap()?
> > 
> > Otherwse realloc() under Linux ABI in FreeBSD fails due to mremap()
> > being mostly a stub and returning -1/ENOMEM for any growth.
> > 
> > LinuxABI under FreeBSD apparently behaves differently than the
> > Linux kernel, which can be seen as bugs when the purpose is "Linux
> > compatibility".
> > 
> > OTOH it would be nice to reduce the strength of the conformity
> > requirements by musl, to be able to accomodate the extra platform
> > (possibly could help on other platforms too).
> 
> This was already on my radar because mremap for enlarging anon
> mappings is also broken on nommu Linux. I think I just need to add a
> "if failed, goto the existing malloc-and-memcpy-and-free code"
> one-liner.

Does the attached work for you? It eliminates the old behavior of
keeping the old mapping on failure when shrinking since that can lead
to massive memory waste (e.g. malloc huge buffer in case you need it,
then shrink to tiny one) if mremap failure-to-shrink is something that
may be common and not just for the VMA-limit-exceeded case.

Rich

[-- Attachment #2: mremap_may_not_work.diff --]
[-- Type: text/plain, Size: 683 bytes --]

diff --git a/src/malloc/malloc.c b/src/malloc/malloc.c
index c38c46f..d5ee428 100644
--- a/src/malloc/malloc.c
+++ b/src/malloc/malloc.c
@@ -406,7 +406,7 @@ void *realloc(void *p, size_t n)
 		if (oldlen == newlen) return p;
 		base = __mremap(base, oldlen, newlen, MREMAP_MAYMOVE);
 		if (base == (void *)-1)
-			return newlen < oldlen ? p : 0;
+			goto copy_realloc;
 		self = (void *)(base + extra);
 		self->csize = newlen - extra;
 		return CHUNK_TO_MEM(self);
@@ -439,6 +439,7 @@ void *realloc(void *p, size_t n)
 		return CHUNK_TO_MEM(self);
 	}
 
+copy_realloc:
 	/* As a last resort, allocate a new chunk and copy to it. */
 	new = malloc(n-OVERHEAD);
 	if (!new) return 0;

Re: a workaround when mremap() is not functional?

[u-uy74]

On Sun, Jun 11, 2017 at 09:38:51AM -0400, Rich Felker wrote:
> > This was already on my radar because mremap for enlarging anon
> > mappings is also broken on nommu Linux. I think I just need to add a
> > "if failed, goto the existing malloc-and-memcpy-and-free code"
> > one-liner.
> 
> Does the attached work for you? It eliminates the old behavior of
> keeping the old mapping on failure when shrinking since that can lead
> to massive memory waste (e.g. malloc huge buffer in case you need it,
> then shrink to tiny one) if mremap failure-to-shrink is something that
> may be common and not just for the VMA-limit-exceeded case.

FreeBSD's linux_mremap() seems to be able to shrink the mapping
but it is not possible to know for all implementations.

> diff --git a/src/malloc/malloc.c b/src/malloc/malloc.c
> index c38c46f..d5ee428 100644
> --- a/src/malloc/malloc.c
> +++ b/src/malloc/malloc.c
> @@ -406,7 +406,7 @@ void *realloc(void *p, size_t n)
>  		if (oldlen == newlen) return p;
>  		base = __mremap(base, oldlen, newlen, MREMAP_MAYMOVE);
>  		if (base == (void *)-1)
> -			return newlen < oldlen ? p : 0;
> +			goto copy_realloc;
>  		self = (void *)(base + extra);
>  		self->csize = newlen - extra;
>  		return CHUNK_TO_MEM(self);
> @@ -439,6 +439,7 @@ void *realloc(void *p, size_t n)
>  		return CHUNK_TO_MEM(self);
>  	}
>  
> +copy_realloc:
>  	/* As a last resort, allocate a new chunk and copy to it. */
>  	new = malloc(n-OVERHEAD);
>  	if (!new) return 0;

This looks working on FreeBSD, as expected!

Thanks a lot for this very timely tweak, what a relief.

Regards,
Rune