[musl] [PATCH] workaround the issue that pthread_getattr_np won't return in FreeBSD Linuxulator

[musl] [PATCH] workaround the issue that pthread_getattr_np won't return in FreeBSD Linuxulator

[MouriNaruto]

Some notes from Kenji Mouri (MouriNaruto) about this workaround:

I tried to use musl-based userspace in FreeBSD's Linuxulator, I found some
Node.js based app will hung at startup. I did some investigations and finally
found that caused by the mremap loop in the musl's pthread_getattr_np. I read
the FreeBSD's Linuxulator source code, it seems mremap in FreeBSD's Linuxulator
in won't support expand the size.

With Edward Tomasz Napierala (https://wiki.freebsd.org/EdwardTomaszNapierala)'s
suggestion, choose to use RLIMIT_STACK _in addition_ to the current measurement
can workaround the issue without doing much modifications.

Consider that getrlimit may failed, so I think we should use __default_stacksize
when getrlimit failed.

I have tested this commit in my FreeBSD environment with Visual Studio Code
Remote SSH server which requires the FreeBSD's Linuxulator.
---
 src/thread/pthread_getattr_np.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/src/thread/pthread_getattr_np.c b/src/thread/pthread_getattr_np.c
index 2881831f..54b1ba34 100644
--- a/src/thread/pthread_getattr_np.c
+++ b/src/thread/pthread_getattr_np.c
@@ -2,6 +2,7 @@
 #include "pthread_impl.h"
 #include "libc.h"
 #include <sys/mman.h>
+#include <sys/resource.h>
 
 int pthread_getattr_np(pthread_t t, pthread_attr_t *a)
 {
@@ -12,11 +13,19 @@ int pthread_getattr_np(pthread_t t, pthread_attr_t *a)
 		a->_a_stackaddr = (uintptr_t)t->stack;
 		a->_a_stacksize = t->stack_size;
 	} else {
+		struct rlimit rl_stack = {0};
+		/* get the current stack limit to workaround the issue */
+		/* that mremap loop won't stop in FreeBSD Linuxulator */
+		if (getrlimit(RLIMIT_STACK, &rl_stack) != 0) {
+			/* use default stack size if getrlimit fails */
+			rl_stack.rlim_cur = __default_stacksize;
+			rl_stack.rlim_max = __default_stacksize;
+		}
 		char *p = (void *)libc.auxv;
 		size_t l = PAGE_SIZE;
 		p += -(uintptr_t)p & PAGE_SIZE-1;
 		a->_a_stackaddr = (uintptr_t)p;
-		while (mremap(p-l-PAGE_SIZE, PAGE_SIZE, 2*PAGE_SIZE, 0)==MAP_FAILED && errno==ENOMEM)
+		while (l <= rl_stack.rlim_cur && mremap(p-l-PAGE_SIZE, PAGE_SIZE, 2*PAGE_SIZE, 0)==MAP_FAILED && errno==ENOMEM)
 			l += PAGE_SIZE;
 		a->_a_stacksize = l;
 	}
-- 
2.47.2

Re: [musl] [PATCH] workaround the issue that pthread_getattr_np won't return in FreeBSD Linuxulator

[Rich Felker]

On Sat, Mar 22, 2025 at 01:04:16AM +0800, MouriNaruto wrote:
> Some notes from Kenji Mouri (MouriNaruto) about this workaround:
> 
> I tried to use musl-based userspace in FreeBSD's Linuxulator, I found some
> Node.js based app will hung at startup. I did some investigations and finally
> found that caused by the mremap loop in the musl's pthread_getattr_np. I read
> the FreeBSD's Linuxulator source code, it seems mremap in FreeBSD's Linuxulator
> in won't support expand the size.
> 
> With Edward Tomasz Napierala (https://wiki.freebsd.org/EdwardTomaszNapierala)'s
> suggestion, choose to use RLIMIT_STACK _in addition_ to the current measurement
> can workaround the issue without doing much modifications.
> 
> Consider that getrlimit may failed, so I think we should use __default_stacksize
> when getrlimit failed.
> 
> I have tested this commit in my FreeBSD environment with Visual Studio Code
> Remote SSH server which requires the FreeBSD's Linuxulator.
> ---
>  src/thread/pthread_getattr_np.c | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/src/thread/pthread_getattr_np.c b/src/thread/pthread_getattr_np.c
> index 2881831f..54b1ba34 100644
> --- a/src/thread/pthread_getattr_np.c
> +++ b/src/thread/pthread_getattr_np.c
> @@ -2,6 +2,7 @@
>  #include "pthread_impl.h"
>  #include "libc.h"
>  #include <sys/mman.h>
> +#include <sys/resource.h>
>  
>  int pthread_getattr_np(pthread_t t, pthread_attr_t *a)
>  {
> @@ -12,11 +13,19 @@ int pthread_getattr_np(pthread_t t, pthread_attr_t *a)
>  		a->_a_stackaddr = (uintptr_t)t->stack;
>  		a->_a_stacksize = t->stack_size;
>  	} else {
> +		struct rlimit rl_stack = {0};
> +		/* get the current stack limit to workaround the issue */
> +		/* that mremap loop won't stop in FreeBSD Linuxulator */
> +		if (getrlimit(RLIMIT_STACK, &rl_stack) != 0) {
> +			/* use default stack size if getrlimit fails */
> +			rl_stack.rlim_cur = __default_stacksize;
> +			rl_stack.rlim_max = __default_stacksize;
> +		}
>  		char *p = (void *)libc.auxv;
>  		size_t l = PAGE_SIZE;
>  		p += -(uintptr_t)p & PAGE_SIZE-1;
>  		a->_a_stackaddr = (uintptr_t)p;
> -		while (mremap(p-l-PAGE_SIZE, PAGE_SIZE, 2*PAGE_SIZE, 0)==MAP_FAILED && errno==ENOMEM)
> +		while (l <= rl_stack.rlim_cur && mremap(p-l-PAGE_SIZE, PAGE_SIZE, 2*PAGE_SIZE, 0)==MAP_FAILED && errno==ENOMEM)
>  			l += PAGE_SIZE;
>  		a->_a_stacksize = l;
>  	}
> -- 
> 2.47.2

This change isn't acceptable upstream - it will necessarily return a
size that can be smaller than the actual stack, if the rlimit has been
lowered, by stopping the scan early. In fact, if you reach the rlimit
without the loop ending, you know either one of two things is true:
either the FreeBSD Linux emulation bug is present, or the stack is
already larger than the current rlimit.

If you have ideas for other ways to scan for the actual limit, I'm
open to hearing them, but I think it would make sense first to see if
FreeBSD could just fix the error code to reflect whether there's a
mapping present. I looked at the old mincore interface but it seems to
only tell if it's currently paged in, not if it's mapped. There's
procfs which could be used if mounted, but that's more of a pain to
work with and not always present, and I'm not sure it would help on
FreeBSD anyway.

Rich

Re: [musl] [PATCH] workaround the issue that pthread_getattr_np won't return in FreeBSD Linuxulator

[Leah Neukirchen]

Rich Felker <dalias@libc.org> writes:

> only tell if it's currently paged in, not if it's mapped. There's
> procfs which could be used if mounted, but that's more of a pain to
> work with and not always present, and I'm not sure it would help on
> FreeBSD anyway.

FreeBSD took precautions to make "linprocfs" work in such a way that
glibc parses it properly:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253337

-- 
Leah Neukirchen  <leah@vuxu.org>  https://leahneukirchen.org/

Re: [musl] [PATCH] workaround the issue that pthread_getattr_np won't return in FreeBSD Linuxulator

[Rich Felker]

On Fri, Mar 21, 2025 at 11:27:53PM +0100, Leah Neukirchen wrote:
> Rich Felker <dalias@libc.org> writes:
> 
> > only tell if it's currently paged in, not if it's mapped. There's
> > procfs which could be used if mounted, but that's more of a pain to
> > work with and not always present, and I'm not sure it would help on
> > FreeBSD anyway.
> 
> FreeBSD took precautions to make "linprocfs" work in such a way that
> glibc parses it properly:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253337

I think this is promising that they might also fix mremap so that musl
can interpret it properly.

As for procfs, does that include /proc/self/maps? That's what we'd
have to parse, and I'm not even sure how rigorously safe it is to
parse... so I'd rather not go there.

Rich

Re: [musl] [PATCH] workaround the issue that pthread_getattr_np won't return in FreeBSD Linuxulator

[Leah Neukirchen]

Rich Felker <dalias@libc.org> writes:

> On Fri, Mar 21, 2025 at 11:27:53PM +0100, Leah Neukirchen wrote:
>> Rich Felker <dalias@libc.org> writes:
>> 
>> > only tell if it's currently paged in, not if it's mapped. There's
>> > procfs which could be used if mounted, but that's more of a pain to
>> > work with and not always present, and I'm not sure it would help on
>> > FreeBSD anyway.
>> 
>> FreeBSD took precautions to make "linprocfs" work in such a way that
>> glibc parses it properly:
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253337
>
> I think this is promising that they might also fix mremap so that musl
> can interpret it properly.
>
> As for procfs, does that include /proc/self/maps? That's what we'd
> have to parse, and I'm not even sure how rigorously safe it is to
> parse... so I'd rather not go there.

Exactly, here's an example on FreeBSD:

# cat /tmp/compat/self/maps
000027a17ce04000-000027a17ce06000 r--p 00000000 00:00 48618     /bin/cat
000027a17ce06000-000027a17ce08000 r-xp 00001000 00:00 48618     /bin/cat
000027a17ce08000-000027a17ce09000 r--p 00000000 00:00 0 
000027a17ce09000-000027a17ce0a000 rw-p 00001000 00:00 0 
000027a99d469000-000027a99d489000 rw-p 00000000 00:00 0 
000027a99d60c000-000027a99d60d000 r-xs 00000000 00:00 0 
000027a99daaa000-000027a99db2f000 r--p 00000000 00:00 39034     /lib/libc.so.7
000027a99db30000-000027a99dc7a000 r-xp 00085000 00:00 39034     /lib/libc.so.7
000027a99dc7a000-000027a99dc84000 r--p 0039c000 00:00 39034     /lib/libc.so.7
000027a99dc84000-000027a99dc8b000 rw-p 003ae000 00:00 39034     /lib/libc.so.7
000027a99dc8b000-000027a99ddad000 rw-p 00000000 00:00 0 
000027a99edad000-000027a99edae000 r--s 00000000 00:00 0 
00003140936df000-00003140936e6000 r--p 00000000 00:00 39556     /libexec/ld-elf.so.1
00003140936e6000-00003140936fc000 r-xp 00006000 00:00 39556     /libexec/ld-elf.so.1
00003140936fc000-00003140936fd000 r--p 00036000 00:00 39556     /libexec/ld-elf.so.1
00003140936fd000-00003140936fe000 r--p 00000000 00:00 0 
00003140936fe000-0000314093700000 rw-p 00001000 00:00 0 
00004acf42400000-00004acf42421000 rw-p 00000000 00:00 0 
00004acf42600000-00004acf43000000 rw-p 00000000 00:00 0 

Notably the [stack] is missing from Linux, but you should infer the
proper mapping anyway.  Else it's just parsing two hex numbers at the
beginning of the line, not terribly difficult.

-- 
Leah Neukirchen  <leah@vuxu.org>  https://leahneukirchen.org

Re: [musl] [PATCH] workaround the issue that pthread_getattr_np won't return in FreeBSD Linuxulator

[Rich Felker]

On Sat, Mar 22, 2025 at 12:08:56AM +0100, Leah Neukirchen wrote:
> Rich Felker <dalias@libc.org> writes:
> 
> > On Fri, Mar 21, 2025 at 11:27:53PM +0100, Leah Neukirchen wrote:
> >> Rich Felker <dalias@libc.org> writes:
> >> 
> >> > only tell if it's currently paged in, not if it's mapped. There's
> >> > procfs which could be used if mounted, but that's more of a pain to
> >> > work with and not always present, and I'm not sure it would help on
> >> > FreeBSD anyway.
> >> 
> >> FreeBSD took precautions to make "linprocfs" work in such a way that
> >> glibc parses it properly:
> >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253337
> >
> > I think this is promising that they might also fix mremap so that musl
> > can interpret it properly.
> >
> > As for procfs, does that include /proc/self/maps? That's what we'd
> > have to parse, and I'm not even sure how rigorously safe it is to
> > parse... so I'd rather not go there.
> 
> Exactly, here's an example on FreeBSD:
> 
> # cat /tmp/compat/self/maps
> 000027a17ce04000-000027a17ce06000 r--p 00000000 00:00 48618     /bin/cat
> 000027a17ce06000-000027a17ce08000 r-xp 00001000 00:00 48618     /bin/cat
> 000027a17ce08000-000027a17ce09000 r--p 00000000 00:00 0 
> 000027a17ce09000-000027a17ce0a000 rw-p 00001000 00:00 0 
> 000027a99d469000-000027a99d489000 rw-p 00000000 00:00 0 
> 000027a99d60c000-000027a99d60d000 r-xs 00000000 00:00 0 
> 000027a99daaa000-000027a99db2f000 r--p 00000000 00:00 39034     /lib/libc.so.7
> 000027a99db30000-000027a99dc7a000 r-xp 00085000 00:00 39034     /lib/libc.so.7
> 000027a99dc7a000-000027a99dc84000 r--p 0039c000 00:00 39034     /lib/libc.so.7
> 000027a99dc84000-000027a99dc8b000 rw-p 003ae000 00:00 39034     /lib/libc.so.7
> 000027a99dc8b000-000027a99ddad000 rw-p 00000000 00:00 0 
> 000027a99edad000-000027a99edae000 r--s 00000000 00:00 0 
> 00003140936df000-00003140936e6000 r--p 00000000 00:00 39556     /libexec/ld-elf.so.1
> 00003140936e6000-00003140936fc000 r-xp 00006000 00:00 39556     /libexec/ld-elf.so.1
> 00003140936fc000-00003140936fd000 r--p 00036000 00:00 39556     /libexec/ld-elf.so.1
> 00003140936fd000-00003140936fe000 r--p 00000000 00:00 0 
> 00003140936fe000-0000314093700000 rw-p 00001000 00:00 0 
> 00004acf42400000-00004acf42421000 rw-p 00000000 00:00 0 
> 00004acf42600000-00004acf43000000 rw-p 00000000 00:00 0 
> 
> Notably the [stack] is missing from Linux, but you should infer the
> proper mapping anyway.  Else it's just parsing two hex numbers at the
> beginning of the line, not terribly difficult.

As long as you can be sure there are no embedded newlines in the last
field. Which I think you can, but I didn't check. And yes you'd just
find the range containing something on main thread's stack.

Rich

RE: [musl] [PATCH] workaround the issue that pthread_getattr_np won't return in FreeBSD Linuxulator

[Mouri Kenji]

>> FreeBSD took precautions to make "linprocfs" work in such a way that
>> glibc parses it properly

> I think this is promising that they might also fix mremap so that musl
> can interpret it properly.

I had talked this way before with Edward Tomasz Napierala like this:

> Actually, I have some ideas for that if we need to make a patch to musl
> mainline.
> 
> ```
> @MoVM-FreeBSD:~ $ uname --help
> BusyBox v1.37.0 (2025-01-17 18:12:01 UTC) multi-call binary.
> 
> Usage: uname [-amnrspvio]
> 
> Print system information
> 
>         -a      Print all
>         -m      Machine (hardware) type
>         -n      Hostname
>         -r      Kernel release
>         -s      Kernel name (default)
>         -p      Processor type
>         -v      Kernel version
>         -i      Hardware platform
>         -o      OS name
> @MoVM-FreeBSD:~ $ uname -v
> FreeBSD 14.2-RELEASE-p1 GENERIC
> @MoVM-FreeBSD:~ $ cat /proc/self/maps
> 0000000001021000-0000000001027000 r--p 00000000 00:00 3366673     /bin/busybox
> 0000000001027000-00000000010c1000 r-xp 00006000 00:00 3366673     /bin/busybox
> 00000000010c1000-00000000010e3000 r--p 000a0000 00:00 3366673     /bin/busybox
> 00000000010e3000-00000000010e7000 r--p 00182000 00:00 3366673     /bin/busybox
> 00000000010e7000-00000000010e8000 rw-p 00000000 00:00 0
> 00000000010e8000-00000000010e9000 ---p 00001000 00:00 0
> 00000000010e9000-00000000010ea000 rw-p 00002000 00:00 0
> 00000008010e3000-00000008010f7000 r--p 00000000 00:00 2166162     /compat/alpine/lib/ld-musl-x86_64.so.1
> 00000008010f7000-000000080114e000 r-xp 00014000 00:00 2166162     /compat/alpine/lib/ld-musl-x86_64.so.1
> 000000080114e000-0000000801184000 r--p 0006b000 00:00 2166162     /compat/alpine/lib/ld-musl-x86_64.so.1
> 0000000801184000-0000000801185000 r--p 00140000 00:00 2166162     /compat/alpine/lib/ld-musl-x86_64.so.1
> 0000000801185000-0000000801189000 rw-p 00000000 00:00 0
> 00007ffffffdd000-00007fffffffd000 rw-p 00000000 00:00 0           [stack]
> 00007fffffffd000-00007fffffffe000 r-xs 00000000 00:00 0
> 00007fffffffe000-0000800000000000 r-xs 00000000 00:00 0           [vdso]
> @MoVM-FreeBSD:~ $
> ```
> 
> According my simple experiments with some commands in FreeBSD Linuxulator.
> I think may be we can:
> 
> (1) Use uname function to check the environment is FreeBSD Linuxulator or not.
>     (check the version string has FreeBSD string or not)
> (2) Get the current stack address.
> (3) Parse the /proc/self/maps to find which item matches the current stack
>     address, then calculate the size.
> 
> It looks dirty but may be can match the musl author's need.

And I got the following replies:

> I wonder if we could just parse /proc/self/maps in all cases?
> Could you tell me how many iterations (calls to mremap(2))
> pthread_getattr_np() makes per call, under Linux, in some typical case?

In my opinion, I only have these questions:

- Does pthread_getattr_np will be called frequently in typical cases?
- If I start to make a patch which parses /proc/self/maps to resolve this issue,
  what should I do that can make musl mainline happy to accept?

Regards,
Kenji Mouri (https://github.com/MouriNaruto)