[musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer

[musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer

[Mouri Kenji]

Hello all,

When I tried to use musl-based userspace in FreeBSD's Linux compatibility layer, I found some app will hung.

I do some debugging with lldb. I have found app hang in mremap loop in the musl's pthread_getattr_np implementation.

I read the FreeBSD's Linux compatibility layer source code, it seems mremap in FreeBSD's Linux compatibility layer in won't support expand the size.

Here is the source code for FreeBSD's Linux compatibility layer mremap implementation: https://github.com/freebsd/freebsd-src/blob/780a4667bbde0daa90db900bb0f93f6337d6208b/sys/compat/linux/linux_misc.c#L302

I only can send a report e-mail to this mail list instead of making a patch because I'm the newbie of Linux's libc internals, and I feel sorry for that.

I have already talked that in musl's IRC but I think I should send a mail to mail list as archive. I know "Some already implemented syscalls are also suboptimal, for example mremap and others." in https://docs.freebsd.org/en/articles/linux-emulation/#future-work, but I hope musl will have the workaround for that. And it can make people can use musl happily in FreeBSD's linuxemu.

Kenji Mouri (https://github.com/MouriNaruto)

Re: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer

[Yao Zi]

On Sat, Mar 08, 2025 at 02:18:04PM +0000, Mouri Kenji wrote:
> Hello all,
> 
> When I tried to use musl-based userspace in FreeBSD's Linux
> compatibility layer, I found some app will hung.
> 
> I do some debugging with lldb. I have found app hang in mremap loop
> in the musl's pthread_getattr_np implementation.
> 
> I read the FreeBSD's Linux compatibility layer source code, it seems
> mremap in FreeBSD's Linux compatibility layer in won't support expand the size.
>
> Here is the source code for FreeBSD's Linux compatibility layer mremap
> implementation: https://github.com/freebsd/freebsd-src/blob/780a4667bbde0daa90db900bb0f93f6337d6208b/sys/compat/linux/linux_misc.c#L302
> 
> I only can send a report e-mail to this mail list instead of making a
> patch because I'm the newbie of Linux's libc internals, and I feel
> sorry for that.

imho the report and the patch should all go into the FreeBSD's mailing
list or similar things. Honestly, why should musl workaround for a
non-officially-supported platform?

> I have already talked that in musl's IRC but I think I should send a
> mail to mail list as archive. I know "Some already implemented
> syscalls are also suboptimal, for example mremap and others." in
> https://docs.freebsd.org/en/articles/linux-emulation/#future-work, but
> I hope musl will have the workaround for that. And it can make people
> can use musl happily in FreeBSD's linuxemu.
> 
> Kenji Mouri (https://github.com/MouriNaruto)
> 

Best regards,
Yao Zi

RE: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer

[Mouri Kenji]

Hi,

I think we can have the hope for that because glibc's pthread_getattr_np won't hung in FreeBSD's Linux compatibility layer.

Kenji Mouri (https://github.com/MouriNaruto)

-----Original Message-----
From: Yao Zi <ziyao@disroot.org>
Sent: Saturday, March 8, 2025 10:31 PM
To: musl@lists.openwall.com
Subject: Re: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer

On Sat, Mar 08, 2025 at 02:18:04PM +0000, Mouri Kenji wrote:
> Hello all,
>
> When I tried to use musl-based userspace in FreeBSD's Linux
> compatibility layer, I found some app will hung.
>
> I do some debugging with lldb. I have found app hang in mremap loop
> in the musl's pthread_getattr_np implementation.
>
> I read the FreeBSD's Linux compatibility layer source code, it seems
> mremap in FreeBSD's Linux compatibility layer in won't support expand the size.
>
> Here is the source code for FreeBSD's Linux compatibility layer mremap
> implementation: https://github.com/freebsd/freebsd-src/blob/780a4667bbde0daa90db900bb0f93f6337d6208b/sys/compat/linux/linux_misc.c#L302
>
> I only can send a report e-mail to this mail list instead of making a
> patch because I'm the newbie of Linux's libc internals, and I feel
> sorry for that.

imho the report and the patch should all go into the FreeBSD's mailing
list or similar things. Honestly, why should musl workaround for a
non-officially-supported platform?

> I have already talked that in musl's IRC but I think I should send a
> mail to mail list as archive. I know "Some already implemented
> syscalls are also suboptimal, for example mremap and others." in
> https://docs.freebsd.org/en/articles/linux-emulation/#future-work, but
> I hope musl will have the workaround for that. And it can make people
> can use musl happily in FreeBSD's linuxemu.
>
> Kenji Mouri (https://github.com/MouriNaruto)
>

Best regards,
Yao Zi

Re: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer

[Rich Felker]

On Sat, Mar 08, 2025 at 02:34:42PM +0000, Mouri Kenji wrote:
> Hi,
> 
> I think we can have the hope for that because glibc's
> pthread_getattr_np won't hung in FreeBSD's Linux compatibility
> layer.

That's because glibc's doesn't actually compute the initial thread's
stack size. Instead it just uses the RLIMIT_STACK as a proxy for it.
This gives the completely wrong result if the rlimit has been changed
since execution started or if the rlimit is infinite. Even when it's
"right", it's measuring a maximum possible size it certainly can't
grow beyond, not a size it actually has or necessarily could attain.

Rich
> -----Original Message-----
> From: Yao Zi <ziyao@disroot.org>
> Sent: Saturday, March 8, 2025 10:31 PM
> To: musl@lists.openwall.com
> Subject: Re: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer
> 
> On Sat, Mar 08, 2025 at 02:18:04PM +0000, Mouri Kenji wrote:
> > Hello all,
> >
> > When I tried to use musl-based userspace in FreeBSD's Linux
> > compatibility layer, I found some app will hung.
> >
> > I do some debugging with lldb. I have found app hang in mremap loop
> > in the musl's pthread_getattr_np implementation.
> >
> > I read the FreeBSD's Linux compatibility layer source code, it seems
> > mremap in FreeBSD's Linux compatibility layer in won't support expand the size.
> >
> > Here is the source code for FreeBSD's Linux compatibility layer mremap
> > implementation: https://github.com/freebsd/freebsd-src/blob/780a4667bbde0daa90db900bb0f93f6337d6208b/sys/compat/linux/linux_misc.c#L302
> >
> > I only can send a report e-mail to this mail list instead of making a
> > patch because I'm the newbie of Linux's libc internals, and I feel
> > sorry for that.
> 
> imho the report and the patch should all go into the FreeBSD's mailing
> list or similar things. Honestly, why should musl workaround for a
> non-officially-supported platform?

If there were an easy fix I'd happily work around it, since support
for FreeBSD's linux compat layer has always been a nice perk and a
good answer to "can I use musl on FreeBSD?"

If there's a way to detect the broken mremap and error out, maybe we
could do that?

But hopefully FreeBSD can fix this.

Rich

RE: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer

[Mouri Kenji]

Hi Rich,

First, thanks for your reply.

I'm curious about why we need to call mremap with size expansion in musl's pthread_getattr_np implementation. It seems get the current stack size for the specific thread. I want to know more details for helping me think how to workaround that. Thanks.

Kenji Mouri (https://github.com/MouriNaruto)

-----Original Message-----
From: Rich Felker <dalias@libc.org>
Sent: Sunday, March 9, 2025 2:09 AM
To: Mouri Kenji <Mouri_Naruto@outlook.com>
Cc: musl@lists.openwall.com
Subject: Re: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer

On Sat, Mar 08, 2025 at 02:34:42PM +0000, Mouri Kenji wrote:
> Hi,
>
> I think we can have the hope for that because glibc's
> pthread_getattr_np won't hung in FreeBSD's Linux compatibility
> layer.

That's because glibc's doesn't actually compute the initial thread's
stack size. Instead it just uses the RLIMIT_STACK as a proxy for it.
This gives the completely wrong result if the rlimit has been changed
since execution started or if the rlimit is infinite. Even when it's
"right", it's measuring a maximum possible size it certainly can't
grow beyond, not a size it actually has or necessarily could attain.

Rich
> -----Original Message-----
> From: Yao Zi <ziyao@disroot.org>
> Sent: Saturday, March 8, 2025 10:31 PM
> To: musl@lists.openwall.com
> Subject: Re: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer
>
> On Sat, Mar 08, 2025 at 02:18:04PM +0000, Mouri Kenji wrote:
> > Hello all,
> >
> > When I tried to use musl-based userspace in FreeBSD's Linux
> > compatibility layer, I found some app will hung.
> >
> > I do some debugging with lldb. I have found app hang in mremap loop
> > in the musl's pthread_getattr_np implementation.
> >
> > I read the FreeBSD's Linux compatibility layer source code, it seems
> > mremap in FreeBSD's Linux compatibility layer in won't support expand the size.
> >
> > Here is the source code for FreeBSD's Linux compatibility layer mremap
> > implementation: https://github.com/freebsd/freebsd-src/blob/780a4667bbde0daa90db900bb0f93f6337d6208b/sys/compat/linux/linux_misc.c#L302
> >
> > I only can send a report e-mail to this mail list instead of making a
> > patch because I'm the newbie of Linux's libc internals, and I feel
> > sorry for that.
>
> imho the report and the patch should all go into the FreeBSD's mailing
> list or similar things. Honestly, why should musl workaround for a
> non-officially-supported platform?

If there were an easy fix I'd happily work around it, since support
for FreeBSD's linux compat layer has always been a nice perk and a
good answer to "can I use musl on FreeBSD?"

If there's a way to detect the broken mremap and error out, maybe we
could do that?

But hopefully FreeBSD can fix this.

Rich

Re: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer

[Rich Felker]

On Sat, Mar 08, 2025 at 07:12:13PM +0000, Mouri Kenji wrote:
> Hi Rich,
> 
> First, thanks for your reply.
> 
> I'm curious about why we need to call mremap with size expansion in
> musl's pthread_getattr_np implementation. It seems get the current
> stack size for the specific thread. I want to know more details for
> helping me think how to workaround that. Thanks.

The stack size isn't directly reported anywhere, so we need a way to
probe for presence of stack pages without actually altering anything,
and mremap does the job. The explanation is with commit
5db951ef80cae8b627f95b995811bf916c069757:

--
in pthread_getattr_np, use mremap rather than madvise to measure stack

the original motivation for this patch was that qemu (and possibly
other syscall emulators) nop out madvise, resulting in an infinite
loop. however, there is another benefit to this change: madvise may
actually undo an explicit madvise the application intended for its
stack, whereas the mremap operation is a true nop. the logic here is
that mremap must fail if it cannot resize the mapping in-place, and
the caller knows that it cannot resize in-place because it knows the
next page of virtual memory is already occupied.
--

It's expected that it will error out with a proper reason (other then
ENOMEM, which is the error for when resizing with the given parameters
is impossible) once we get to the end of the stack.

Rich


> 
> Kenji Mouri (https://github.com/MouriNaruto)
> 
> -----Original Message-----
> From: Rich Felker <dalias@libc.org>
> Sent: Sunday, March 9, 2025 2:09 AM
> To: Mouri Kenji <Mouri_Naruto@outlook.com>
> Cc: musl@lists.openwall.com
> Subject: Re: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer
> 
> On Sat, Mar 08, 2025 at 02:34:42PM +0000, Mouri Kenji wrote:
> > Hi,
> >
> > I think we can have the hope for that because glibc's
> > pthread_getattr_np won't hung in FreeBSD's Linux compatibility
> > layer.
> 
> That's because glibc's doesn't actually compute the initial thread's
> stack size. Instead it just uses the RLIMIT_STACK as a proxy for it.
> This gives the completely wrong result if the rlimit has been changed
> since execution started or if the rlimit is infinite. Even when it's
> "right", it's measuring a maximum possible size it certainly can't
> grow beyond, not a size it actually has or necessarily could attain.
> 
> Rich
> > -----Original Message-----
> > From: Yao Zi <ziyao@disroot.org>
> > Sent: Saturday, March 8, 2025 10:31 PM
> > To: musl@lists.openwall.com
> > Subject: Re: [musl] Calling pthread_getattr_np won't return in the mremap loop when using musl in FreeBSD's Linux compatibility layer
> >
> > On Sat, Mar 08, 2025 at 02:18:04PM +0000, Mouri Kenji wrote:
> > > Hello all,
> > >
> > > When I tried to use musl-based userspace in FreeBSD's Linux
> > > compatibility layer, I found some app will hung.
> > >
> > > I do some debugging with lldb. I have found app hang in mremap loop
> > > in the musl's pthread_getattr_np implementation.
> > >
> > > I read the FreeBSD's Linux compatibility layer source code, it seems
> > > mremap in FreeBSD's Linux compatibility layer in won't support expand the size.
> > >
> > > Here is the source code for FreeBSD's Linux compatibility layer mremap
> > > implementation: https://github.com/freebsd/freebsd-src/blob/780a4667bbde0daa90db900bb0f93f6337d6208b/sys/compat/linux/linux_misc.c#L302
> > >
> > > I only can send a report e-mail to this mail list instead of making a
> > > patch because I'm the newbie of Linux's libc internals, and I feel
> > > sorry for that.
> >
> > imho the report and the patch should all go into the FreeBSD's mailing
> > list or similar things. Honestly, why should musl workaround for a
> > non-officially-supported platform?
> 
> If there were an easy fix I'd happily work around it, since support
> for FreeBSD's linux compat layer has always been a nice perk and a
> good answer to "can I use musl on FreeBSD?"
> 
> If there's a way to detect the broken mremap and error out, maybe we
> could do that?
> 
> But hopefully FreeBSD can fix this.
> 
> Rich