[musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Andreas Dröscher]

Hi

I'm building a new toolchain for a very old hardware with a very old Linux 
Kernel (2.6.20). The CPU is a Alchemy (now AMD) AU1100 (production was 
discontinued).

Obviously the Kernel lacks a lot of the modern system calls. I however expect 
the general system call interface to be consistent. Moreover, musl has fallbacks 
for many system-calls in place, kudos! However, the fallback is never triggered. 
I will present the issue on one example (epoll):

excerpt from src/linux/epoll.c:
int epoll_create1(int flags)
{
int r = __syscall(SYS_epoll_create1, flags);
#ifdef SYS_epoll_create
if (r==-ENOSYS && !flags) r = __syscall(SYS_epoll_create, 1);
#endif
return __syscall_ret(r);
}

If r is -89 (negative ENOSYS) the fallback is triggered else the result is 
returned as it is. However, in my case __syscall returnes 89 (positive ENOSYS).
I've tracked the return into the kernel and there the negative value is 
returned. The Kernel additionally sets r7 to 1.

excerpt from arch/mips/syscall_arch.h:
static inline long __syscall1(long n, long a)
{
register long r4 __asm__("$4") = a;
register long r7 __asm__("$7");
register long r2 __asm__("$2") = n;
__asm__ __volatile__ (
"syscall"
: "+r"(r2), "=r"(r7)
: "r"(r4)
: SYSCALL_CLOBBERLIST, "$8", "$9", "$10");
return r7 ? -r2 : r2;
}

I assume the "bug" is triggered by __syscall1 If r7 is set it will change the 
sign of r2. I can patch that by replacing:
return r7 ? -r2 : r2;
with
return (r7 && r2 > 0) ? -r2 : r2;

However I've no idea if I'm triggering any side effects or if I selected the 
wrong implementation for my architecture.

Any suggestions are appreciated.
~Andreas

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Rich Felker]

On Tue, Mar 10, 2020 at 10:10:15PM +0100, Andreas Dröscher wrote:
> Hi
> 
> I'm building a new toolchain for a very old hardware with a very old
> Linux Kernel (2.6.20). The CPU is a Alchemy (now AMD) AU1100
> (production was discontinued).
> 
> Obviously the Kernel lacks a lot of the modern system calls. I
> however expect the general system call interface to be consistent.
> Moreover, musl has fallbacks for many system-calls in place, kudos!
> However, the fallback is never triggered. I will present the issue
> on one example (epoll):
> 
> excerpt from src/linux/epoll.c:
> int epoll_create1(int flags)
> {
> int r = __syscall(SYS_epoll_create1, flags);
> #ifdef SYS_epoll_create
> if (r==-ENOSYS && !flags) r = __syscall(SYS_epoll_create, 1);
> #endif
> return __syscall_ret(r);
> }
> 
> If r is -89 (negative ENOSYS) the fallback is triggered else the
> result is returned as it is. However, in my case __syscall returnes
> 89 (positive ENOSYS).
> I've tracked the return into the kernel and there the negative value
> is returned. The Kernel additionally sets r7 to 1.
> 
> excerpt from arch/mips/syscall_arch.h:
> static inline long __syscall1(long n, long a)
> {
> register long r4 __asm__("$4") = a;
> register long r7 __asm__("$7");
> register long r2 __asm__("$2") = n;
> __asm__ __volatile__ (
> "syscall"
> : "+r"(r2), "=r"(r7)
> : "r"(r4)
> : SYSCALL_CLOBBERLIST, "$8", "$9", "$10");
> return r7 ? -r2 : r2;
> }
> 
> I assume the "bug" is triggered by __syscall1 If r7 is set it will
> change the sign of r2. I can patch that by replacing:
> return r7 ? -r2 : r2;
> with
> return (r7 && r2 > 0) ? -r2 : r2;
> 
> However I've no idea if I'm triggering any side effects or if I
> selected the wrong implementation for my architecture.

It sounds like what you're saying is that the ENOSYS codepath for
mips, at least on your old kernel, is not setting the error flag in r7
and returning ENOSYS in r2, but is instead returning -ENOSYS already
(and not clear whether it's setting r7 at all or just leaving a stale
value there).

Can anyone else confirm this, or point to kernel history that might
suggest it's a real bug? Your workaround looks like it should at least
be *safe* to do, and probably the right thing if this was/is a real
kernel bug in the official kernel rather than something some vendor
broke in their fork.

Rich

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Andreas Dröscher]

Am 11.03.20 um 01:55 schrieb Rich Felker:
> On Tue, Mar 10, 2020 at 10:10:15PM +0100, Andreas Dröscher wrote:
>> Hi
>>
>> I'm building a new toolchain for a very old hardware with a very old
>> Linux Kernel (2.6.20). The CPU is a Alchemy (now AMD) AU1100
>> (production was discontinued).
>>
>> Obviously the Kernel lacks a lot of the modern system calls. I
>> however expect the general system call interface to be consistent.
>> Moreover, musl has fallbacks for many system-calls in place, kudos!
>> However, the fallback is never triggered. I will present the issue
>> on one example (epoll):
>>
>> excerpt from src/linux/epoll.c:
>> int epoll_create1(int flags)
>> {
>> int r = __syscall(SYS_epoll_create1, flags);
>> #ifdef SYS_epoll_create
>> if (r==-ENOSYS && !flags) r = __syscall(SYS_epoll_create, 1);
>> #endif
>> return __syscall_ret(r);
>> }
>>
>> If r is -89 (negative ENOSYS) the fallback is triggered else the
>> result is returned as it is. However, in my case __syscall returnes
>> 89 (positive ENOSYS).
>> I've tracked the return into the kernel and there the negative value
>> is returned. The Kernel additionally sets r7 to 1.
>>
>> excerpt from arch/mips/syscall_arch.h:
>> static inline long __syscall1(long n, long a)
>> {
>> register long r4 __asm__("$4") = a;
>> register long r7 __asm__("$7");
>> register long r2 __asm__("$2") = n;
>> __asm__ __volatile__ (
>> "syscall"
>> : "+r"(r2), "=r"(r7)
>> : "r"(r4)
>> : SYSCALL_CLOBBERLIST, "$8", "$9", "$10");
>> return r7 ? -r2 : r2;
>> }
>>
>> I assume the "bug" is triggered by __syscall1 If r7 is set it will
>> change the sign of r2. I can patch that by replacing:
>> return r7 ? -r2 : r2;
>> with
>> return (r7 && r2 > 0) ? -r2 : r2;
>>
>> However I've no idea if I'm triggering any side effects or if I
>> selected the wrong implementation for my architecture.
> 
> It sounds like what you're saying is that the ENOSYS codepath for
> mips, at least on your old kernel, is not setting the error flag in r7
> and returning ENOSYS in r2, but is instead returning -ENOSYS already
> (and not clear whether it's setting r7 at all or just leaving a stale
> value there).
> 
> Can anyone else confirm this, or point to kernel history that might
> suggest it's a real bug? Your workaround looks like it should at least
> be *safe* to do, and probably the right thing if this was/is a real
> kernel bug in the official kernel rather than something some vendor
> broke in their fork.
> 
> Rich
> 

Sorry for not including that excerpt in the first place:

illegal_syscall:
	li	v0, -ENOSYS			# error
	sw	v0, PT_R2(sp)
	li	t0, 1				# set error flag
	sw	t0, PT_R7(sp)
	j	o32_syscall_exit
	END(handle_sys)

Source: 
https://github.com/torvalds/linux/blob/62d0cfcb27cf755cebdc93ca95dabc83608007cd/arch/mips/kernel/scall32-o32.S#L186

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Rich Felker]

On Wed, Mar 11, 2020 at 02:19:31AM +0100, Andreas Dröscher wrote:
> Am 11.03.20 um 01:55 schrieb Rich Felker:
> >On Tue, Mar 10, 2020 at 10:10:15PM +0100, Andreas Dröscher wrote:
> >>Hi
> >>
> >>I'm building a new toolchain for a very old hardware with a very old
> >>Linux Kernel (2.6.20). The CPU is a Alchemy (now AMD) AU1100
> >>(production was discontinued).
> >>
> >>Obviously the Kernel lacks a lot of the modern system calls. I
> >>however expect the general system call interface to be consistent.
> >>Moreover, musl has fallbacks for many system-calls in place, kudos!
> >>However, the fallback is never triggered. I will present the issue
> >>on one example (epoll):
> >>
> >>excerpt from src/linux/epoll.c:
> >>int epoll_create1(int flags)
> >>{
> >>int r = __syscall(SYS_epoll_create1, flags);
> >>#ifdef SYS_epoll_create
> >>if (r==-ENOSYS && !flags) r = __syscall(SYS_epoll_create, 1);
> >>#endif
> >>return __syscall_ret(r);
> >>}
> >>
> >>If r is -89 (negative ENOSYS) the fallback is triggered else the
> >>result is returned as it is. However, in my case __syscall returnes
> >>89 (positive ENOSYS).
> >>I've tracked the return into the kernel and there the negative value
> >>is returned. The Kernel additionally sets r7 to 1.
> >>
> >>excerpt from arch/mips/syscall_arch.h:
> >>static inline long __syscall1(long n, long a)
> >>{
> >>register long r4 __asm__("$4") = a;
> >>register long r7 __asm__("$7");
> >>register long r2 __asm__("$2") = n;
> >>__asm__ __volatile__ (
> >>"syscall"
> >>: "+r"(r2), "=r"(r7)
> >>: "r"(r4)
> >>: SYSCALL_CLOBBERLIST, "$8", "$9", "$10");
> >>return r7 ? -r2 : r2;
> >>}
> >>
> >>I assume the "bug" is triggered by __syscall1 If r7 is set it will
> >>change the sign of r2. I can patch that by replacing:
> >>return r7 ? -r2 : r2;
> >>with
> >>return (r7 && r2 > 0) ? -r2 : r2;
> >>
> >>However I've no idea if I'm triggering any side effects or if I
> >>selected the wrong implementation for my architecture.
> >
> >It sounds like what you're saying is that the ENOSYS codepath for
> >mips, at least on your old kernel, is not setting the error flag in r7
> >and returning ENOSYS in r2, but is instead returning -ENOSYS already
> >(and not clear whether it's setting r7 at all or just leaving a stale
> >value there).
> >
> >Can anyone else confirm this, or point to kernel history that might
> >suggest it's a real bug? Your workaround looks like it should at least
> >be *safe* to do, and probably the right thing if this was/is a real
> >kernel bug in the official kernel rather than something some vendor
> >broke in their fork.
> >
> >Rich
> >
> 
> Sorry for not including that excerpt in the first place:
> 
> illegal_syscall:
> 	li	v0, -ENOSYS			# error
> 	sw	v0, PT_R2(sp)
> 	li	t0, 1				# set error flag
> 	sw	t0, PT_R7(sp)
> 	j	o32_syscall_exit
> 	END(handle_sys)
> 
> Source: https://github.com/torvalds/linux/blob/62d0cfcb27cf755cebdc93ca95dabc83608007cd/arch/mips/kernel/scall32-o32.S#L186

OK, this was fixed by commit bda8229bdd087167f463ad5e74299987924f8137
in 2008. But it looks like there's still another path, called
"einval" from before commit fb498e2570eedc6c9c3d165e370624dfc3aed97b,
that returns -ENOSYS. All of this is awful, and I think your fix is
probably the right thing to do.

Rich

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Andreas Dröscher]

Am 11.03.20 um 02:40 schrieb Rich Felker:
> On Wed, Mar 11, 2020 at 02:19:31AM +0100, Andreas Dröscher wrote:
>> Am 11.03.20 um 01:55 schrieb Rich Felker:
>>> On Tue, Mar 10, 2020 at 10:10:15PM +0100, Andreas Dröscher wrote:
>>>> Hi
>>>>
>>>> I'm building a new toolchain for a very old hardware with a very old
>>>> Linux Kernel (2.6.20). The CPU is a Alchemy (now AMD) AU1100
>>>> (production was discontinued).
>>>>
>>>> Obviously the Kernel lacks a lot of the modern system calls. I
>>>> however expect the general system call interface to be consistent.
>>>> Moreover, musl has fallbacks for many system-calls in place, kudos!
>>>> However, the fallback is never triggered. I will present the issue
>>>> on one example (epoll):
>>>>
>>>> excerpt from src/linux/epoll.c:
>>>> int epoll_create1(int flags)
>>>> {
>>>> int r = __syscall(SYS_epoll_create1, flags);
>>>> #ifdef SYS_epoll_create
>>>> if (r==-ENOSYS && !flags) r = __syscall(SYS_epoll_create, 1);
>>>> #endif
>>>> return __syscall_ret(r);
>>>> }
>>>>
>>>> If r is -89 (negative ENOSYS) the fallback is triggered else the
>>>> result is returned as it is. However, in my case __syscall returnes
>>>> 89 (positive ENOSYS).
>>>> I've tracked the return into the kernel and there the negative value
>>>> is returned. The Kernel additionally sets r7 to 1.
>>>>
>>>> excerpt from arch/mips/syscall_arch.h:
>>>> static inline long __syscall1(long n, long a)
>>>> {
>>>> register long r4 __asm__("$4") = a;
>>>> register long r7 __asm__("$7");
>>>> register long r2 __asm__("$2") = n;
>>>> __asm__ __volatile__ (
>>>> "syscall"
>>>> : "+r"(r2), "=r"(r7)
>>>> : "r"(r4)
>>>> : SYSCALL_CLOBBERLIST, "$8", "$9", "$10");
>>>> return r7 ? -r2 : r2;
>>>> }
>>>>
>>>> I assume the "bug" is triggered by __syscall1 If r7 is set it will
>>>> change the sign of r2. I can patch that by replacing:
>>>> return r7 ? -r2 : r2;
>>>> with
>>>> return (r7 && r2 > 0) ? -r2 : r2;
>>>>
>>>> However I've no idea if I'm triggering any side effects or if I
>>>> selected the wrong implementation for my architecture.
>>>
>>> It sounds like what you're saying is that the ENOSYS codepath for
>>> mips, at least on your old kernel, is not setting the error flag in r7
>>> and returning ENOSYS in r2, but is instead returning -ENOSYS already
>>> (and not clear whether it's setting r7 at all or just leaving a stale
>>> value there).
>>>
>>> Can anyone else confirm this, or point to kernel history that might
>>> suggest it's a real bug? Your workaround looks like it should at least
>>> be *safe* to do, and probably the right thing if this was/is a real
>>> kernel bug in the official kernel rather than something some vendor
>>> broke in their fork.
>>>
>>> Rich
>>>
>>
>> Sorry for not including that excerpt in the first place:
>>
>> illegal_syscall:
>> 	li	v0, -ENOSYS			# error
>> 	sw	v0, PT_R2(sp)
>> 	li	t0, 1				# set error flag
>> 	sw	t0, PT_R7(sp)
>> 	j	o32_syscall_exit
>> 	END(handle_sys)
>>
>> Source: https://github.com/torvalds/linux/blob/62d0cfcb27cf755cebdc93ca95dabc83608007cd/arch/mips/kernel/scall32-o32.S#L186
> 
> OK, this was fixed by commit bda8229bdd087167f463ad5e74299987924f8137
> in 2008. But it looks like there's still another path, called
> "einval" from before commit fb498e2570eedc6c9c3d165e370624dfc3aed97b,
> that returns -ENOSYS. All of this is awful, and I think your fix is
> probably the right thing to do.
> 
> Rich
> 

Thank you very much for your review.

Just as a side note. I’ve just figured out that there is a second issue with old 
kernels.

The current implementation of __syscall5, __syscall6 and __syscall7 (those use 
caller saved registers) violate the calling conventions of MIPS32 Linux Kernels 
prior 2.6.35. Those were assuming that the instruction immediately preceding the 
SYSCALL instruction was an instruction for loading the syscall number.

I’ll will try to rearrange the stack pushes to accommodate this requirement and 
report back if I manage to come up with something presentable.

Andreas

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Rich Felker]

On Wed, Mar 11, 2020 at 03:08:22AM +0100, Andreas Dröscher wrote:
> >>Sorry for not including that excerpt in the first place:
> >>
> >>illegal_syscall:
> >>	li	v0, -ENOSYS			# error
> >>	sw	v0, PT_R2(sp)
> >>	li	t0, 1				# set error flag
> >>	sw	t0, PT_R7(sp)
> >>	j	o32_syscall_exit
> >>	END(handle_sys)
> >>
> >>Source: https://github.com/torvalds/linux/blob/62d0cfcb27cf755cebdc93ca95dabc83608007cd/arch/mips/kernel/scall32-o32.S#L186
> >
> >OK, this was fixed by commit bda8229bdd087167f463ad5e74299987924f8137
> >in 2008. But it looks like there's still another path, called
> >"einval" from before commit fb498e2570eedc6c9c3d165e370624dfc3aed97b,
> >that returns -ENOSYS. All of this is awful, and I think your fix is
> >probably the right thing to do.
> 
> Thank you very much for your review.
> 
> Just as a side note. I’ve just figured out that there is a second
> issue with old kernels.
> 
> The current implementation of __syscall5, __syscall6 and __syscall7
> (those use caller saved registers) violate the calling conventions
> of MIPS32 Linux Kernels prior 2.6.35. Those were assuming that the
> instruction immediately preceding the SYSCALL instruction was an
> instruction for loading the syscall number.
> 
> I’ll will try to rearrange the stack pushes to accommodate this
> requirement and report back if I manage to come up with something
> presentable.

Uhg, so commit 604f8d3d8b08ee4f548de193050ef93a7753c2e0 was probably
wrong and there was a reason for the nonsensical code it removed:
making old broken kernels happy. I'm not sure if you can just revert
it or need to make new changes.

Do you know if this "rule" applies to n32/n64 too or just o32?

Rich

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Andreas Dröscher]

Am 11.03.20 um 03:18 schrieb Rich Felker:
> On Wed, Mar 11, 2020 at 03:08:22AM +0100, Andreas Dröscher wrote:
>> The current implementation of __syscall5, __syscall6 and __syscall7
>> (those use caller saved registers) violate the calling conventions
>> of MIPS32 Linux Kernels prior 2.6.35. Those were assuming that the
>> instruction immediately preceding the SYSCALL instruction was an
>> instruction for loading the syscall number.
>>
>> I’ll will try to rearrange the stack pushes to accommodate this
>> requirement and report back if I manage to come up with something
>> presentable.
> 
> Uhg, so commit 604f8d3d8b08ee4f548de193050ef93a7753c2e0 was probably
> wrong and there was a reason for the nonsensical code it removed:
> making old broken kernels happy. I'm not sure if you can just revert
> it or need to make new changes.
> 
> Do you know if this "rule" applies to n32/n64 too or just o32?

I've reverted 604f8d3d8b08ee4f548de193050ef93a7753c2e0 and additionally
replaced all:
return r7 ? -r2 : r2;
with
return (r7 && r2 > 0) ? -r2 : r2;

My software stack (built with OE-Core Zeus) now works almost flawlessly.
Some Daemons have hiccups but those most likely come from source that expects 
syscalls to always succeed and on my system they are simply missing.

Thank you for your helping to sort this out.

You asked about n32/n64. I am not familiar with more modern MIPS Architectures.
Therefore I can't give any informed answer. I found some documentation:
https://www.linux-mips.org/wiki/Syscall but it does not give a definitive 
answer. It just points towards "all 3 mips are effected by the ordering 
requirement".

Andreas

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Rich Felker]

On Wed, Mar 11, 2020 at 10:08:11PM +0100, Andreas Dröscher wrote:
> Am 11.03.20 um 03:18 schrieb Rich Felker:
> >On Wed, Mar 11, 2020 at 03:08:22AM +0100, Andreas Dröscher wrote:
> >>The current implementation of __syscall5, __syscall6 and __syscall7
> >>(those use caller saved registers) violate the calling conventions
> >>of MIPS32 Linux Kernels prior 2.6.35. Those were assuming that the
> >>instruction immediately preceding the SYSCALL instruction was an
> >>instruction for loading the syscall number.
> >>
> >>I’ll will try to rearrange the stack pushes to accommodate this
> >>requirement and report back if I manage to come up with something
> >>presentable.
> >
> >Uhg, so commit 604f8d3d8b08ee4f548de193050ef93a7753c2e0 was probably
> >wrong and there was a reason for the nonsensical code it removed:
> >making old broken kernels happy. I'm not sure if you can just revert
> >it or need to make new changes.
> >
> >Do you know if this "rule" applies to n32/n64 too or just o32?
> 
> I've reverted 604f8d3d8b08ee4f548de193050ef93a7753c2e0 and additionally
> replaced all:
> return r7 ? -r2 : r2;
> with
> return (r7 && r2 > 0) ? -r2 : r2;
> 
> My software stack (built with OE-Core Zeus) now works almost flawlessly.
> Some Daemons have hiccups but those most likely come from source
> that expects syscalls to always succeed and on my system they are
> simply missing.
> 
> Thank you for your helping to sort this out.
> 
> You asked about n32/n64. I am not familiar with more modern MIPS Architectures.
> Therefore I can't give any informed answer. I found some documentation:
> https://www.linux-mips.org/wiki/Syscall but it does not give a
> definitive answer. It just points towards "all 3 mips are effected
> by the ordering requirement".

OK, here's the kernel commit that changed it: 
8f5a00eb422ed86e77bb8f67e08b9fe6d30f679a. This was a lot of work to
find, since the reference to it from a later referred to it by a hash
it had in the linux-mips git repo which has since been deleted and
doesn't turn up at all on searches.

So it looks like all 3 variants were affected. This means I'll need to
evaluate reverting the change for the others too, make sure there was
no other change that needs to be kept, and then applying your patch
afterwards.

I'll try to get this done asap. Let me know if you find anything else
interesting that looks like it's still broken after reverting and
applying your patch.

Also, this regression in the final 1.1.x series might justify a 1.1.25
release at some point.

Rich

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Rich Felker]

On Wed, Mar 11, 2020 at 10:08:11PM +0100, Andreas Dröscher wrote:
> Am 11.03.20 um 03:18 schrieb Rich Felker:
> >On Wed, Mar 11, 2020 at 03:08:22AM +0100, Andreas Dröscher wrote:
> >>The current implementation of __syscall5, __syscall6 and __syscall7
> >>(those use caller saved registers) violate the calling conventions
> >>of MIPS32 Linux Kernels prior 2.6.35. Those were assuming that the
> >>instruction immediately preceding the SYSCALL instruction was an
> >>instruction for loading the syscall number.
> >>
> >>I’ll will try to rearrange the stack pushes to accommodate this
> >>requirement and report back if I manage to come up with something
> >>presentable.
> >
> >Uhg, so commit 604f8d3d8b08ee4f548de193050ef93a7753c2e0 was probably
> >wrong and there was a reason for the nonsensical code it removed:
> >making old broken kernels happy. I'm not sure if you can just revert
> >it or need to make new changes.
> >
> >Do you know if this "rule" applies to n32/n64 too or just o32?
> 
> I've reverted 604f8d3d8b08ee4f548de193050ef93a7753c2e0 and additionally
> replaced all:
> return r7 ? -r2 : r2;
> with
> return (r7 && r2 > 0) ? -r2 : r2;
> 
> My software stack (built with OE-Core Zeus) now works almost flawlessly.
> Some Daemons have hiccups but those most likely come from source
> that expects syscalls to always succeed and on my system they are
> simply missing.
> 
> Thank you for your helping to sort this out.
> 
> You asked about n32/n64. I am not familiar with more modern MIPS Architectures.
> Therefore I can't give any informed answer. I found some documentation:
> https://www.linux-mips.org/wiki/Syscall but it does not give a
> definitive answer. It just points towards "all 3 mips are effected
> by the ordering requirement".

I'm posting a patch series now.

Rich

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Rich Felker]

On Wed, Mar 11, 2020 at 07:08:48PM -0400, Rich Felker wrote:
> On Wed, Mar 11, 2020 at 10:08:11PM +0100, Andreas Dröscher wrote:
> > Am 11.03.20 um 03:18 schrieb Rich Felker:
> > >On Wed, Mar 11, 2020 at 03:08:22AM +0100, Andreas Dröscher wrote:
> > >>The current implementation of __syscall5, __syscall6 and __syscall7
> > >>(those use caller saved registers) violate the calling conventions
> > >>of MIPS32 Linux Kernels prior 2.6.35. Those were assuming that the
> > >>instruction immediately preceding the SYSCALL instruction was an
> > >>instruction for loading the syscall number.
> > >>
> > >>I’ll will try to rearrange the stack pushes to accommodate this
> > >>requirement and report back if I manage to come up with something
> > >>presentable.
> > >
> > >Uhg, so commit 604f8d3d8b08ee4f548de193050ef93a7753c2e0 was probably
> > >wrong and there was a reason for the nonsensical code it removed:
> > >making old broken kernels happy. I'm not sure if you can just revert
> > >it or need to make new changes.
> > >
> > >Do you know if this "rule" applies to n32/n64 too or just o32?
> > 
> > I've reverted 604f8d3d8b08ee4f548de193050ef93a7753c2e0 and additionally
> > replaced all:
> > return r7 ? -r2 : r2;
> > with
> > return (r7 && r2 > 0) ? -r2 : r2;
> > 
> > My software stack (built with OE-Core Zeus) now works almost flawlessly.
> > Some Daemons have hiccups but those most likely come from source
> > that expects syscalls to always succeed and on my system they are
> > simply missing.
> > 
> > Thank you for your helping to sort this out.
> > 
> > You asked about n32/n64. I am not familiar with more modern MIPS Architectures.
> > Therefore I can't give any informed answer. I found some documentation:
> > https://www.linux-mips.org/wiki/Syscall but it does not give a
> > definitive answer. It just points towards "all 3 mips are effected
> > by the ordering requirement".
> 
> I'm posting a patch series now.

I've pushed a version of this upstream now, a long with a lot of other
commits that had backed up in my queue. Please let me know if this
does or doesn't fix the issues with mips on old kernels.

Rich

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Andreas Dröscher]

Am 20.03.20 um 17:34 schrieb Rich Felker:
> On Wed, Mar 11, 2020 at 07:08:48PM -0400, Rich Felker wrote:
>> On Wed, Mar 11, 2020 at 10:08:11PM +0100, Andreas Dröscher wrote:
>>> Am 11.03.20 um 03:18 schrieb Rich Felker:
>>>> On Wed, Mar 11, 2020 at 03:08:22AM +0100, Andreas Dröscher wrote:
>>>>> The current implementation of __syscall5, __syscall6 and __syscall7
>>>>> (those use caller saved registers) violate the calling conventions
>>>>> of MIPS32 Linux Kernels prior 2.6.35. Those were assuming that the
>>>>> instruction immediately preceding the SYSCALL instruction was an
>>>>> instruction for loading the syscall number.
>>>>>
>>>>> I’ll will try to rearrange the stack pushes to accommodate this
>>>>> requirement and report back if I manage to come up with something
>>>>> presentable.
>>>>
>>>> Uhg, so commit 604f8d3d8b08ee4f548de193050ef93a7753c2e0 was probably
>>>> wrong and there was a reason for the nonsensical code it removed:
>>>> making old broken kernels happy. I'm not sure if you can just revert
>>>> it or need to make new changes.
>>>>
>>>> Do you know if this "rule" applies to n32/n64 too or just o32?
>>>
>>> I've reverted 604f8d3d8b08ee4f548de193050ef93a7753c2e0 and additionally
>>> replaced all:
>>> return r7 ? -r2 : r2;
>>> with
>>> return (r7 && r2 > 0) ? -r2 : r2;
>>>
>>> My software stack (built with OE-Core Zeus) now works almost flawlessly.
>>> Some Daemons have hiccups but those most likely come from source
>>> that expects syscalls to always succeed and on my system they are
>>> simply missing.
>>>
>>> Thank you for your helping to sort this out.
>>>
>>> You asked about n32/n64. I am not familiar with more modern MIPS Architectures.
>>> Therefore I can't give any informed answer. I found some documentation:
>>> https://www.linux-mips.org/wiki/Syscall but it does not give a
>>> definitive answer. It just points towards "all 3 mips are effected
>>> by the ordering requirement".
>>
>> I'm posting a patch series now.
> 
> I've pushed a version of this upstream now, a long with a lot of other
> commits that had backed up in my queue. Please let me know if this
> does or doesn't fix the issues with mips on old kernels.
>

Sorry for now sending an update in a timely manner. I've tested your patches and 
they definitely get me to the point I had with my manual changes. However, the 
incompatibilities I reported earlier stil need to be ironed out. Sadly I got 
sidetracked by another project. There was no progress in the past week.

I’m planning to pick up the task on Monday. I was wondering if the inclusion of 
our fixes for a 13-year-old kernel benefits anyone except my niche use case.

Andreas

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Rich Felker]

On Fri, Mar 20, 2020 at 11:18:56PM +0100, Andreas Dröscher wrote:
> Am 20.03.20 um 17:34 schrieb Rich Felker:
> >On Wed, Mar 11, 2020 at 07:08:48PM -0400, Rich Felker wrote:
> >>On Wed, Mar 11, 2020 at 10:08:11PM +0100, Andreas Dröscher wrote:
> >>>Am 11.03.20 um 03:18 schrieb Rich Felker:
> >>>>On Wed, Mar 11, 2020 at 03:08:22AM +0100, Andreas Dröscher wrote:
> >>>>>The current implementation of __syscall5, __syscall6 and __syscall7
> >>>>>(those use caller saved registers) violate the calling conventions
> >>>>>of MIPS32 Linux Kernels prior 2.6.35. Those were assuming that the
> >>>>>instruction immediately preceding the SYSCALL instruction was an
> >>>>>instruction for loading the syscall number.
> >>>>>
> >>>>>I’ll will try to rearrange the stack pushes to accommodate this
> >>>>>requirement and report back if I manage to come up with something
> >>>>>presentable.
> >>>>
> >>>>Uhg, so commit 604f8d3d8b08ee4f548de193050ef93a7753c2e0 was probably
> >>>>wrong and there was a reason for the nonsensical code it removed:
> >>>>making old broken kernels happy. I'm not sure if you can just revert
> >>>>it or need to make new changes.
> >>>>
> >>>>Do you know if this "rule" applies to n32/n64 too or just o32?
> >>>
> >>>I've reverted 604f8d3d8b08ee4f548de193050ef93a7753c2e0 and additionally
> >>>replaced all:
> >>>return r7 ? -r2 : r2;
> >>>with
> >>>return (r7 && r2 > 0) ? -r2 : r2;
> >>>
> >>>My software stack (built with OE-Core Zeus) now works almost flawlessly.
> >>>Some Daemons have hiccups but those most likely come from source
> >>>that expects syscalls to always succeed and on my system they are
> >>>simply missing.
> >>>
> >>>Thank you for your helping to sort this out.
> >>>
> >>>You asked about n32/n64. I am not familiar with more modern MIPS Architectures.
> >>>Therefore I can't give any informed answer. I found some documentation:
> >>>https://www.linux-mips.org/wiki/Syscall but it does not give a
> >>>definitive answer. It just points towards "all 3 mips are effected
> >>>by the ordering requirement".
> >>
> >>I'm posting a patch series now.
> >
> >I've pushed a version of this upstream now, a long with a lot of other
> >commits that had backed up in my queue. Please let me know if this
> >does or doesn't fix the issues with mips on old kernels.
> >
> 
> Sorry for now sending an update in a timely manner. I've tested your
> patches and they definitely get me to the point I had with my manual
> changes. However, the incompatibilities I reported earlier stil need
> to be ironed out. Sadly I got sidetracked by another project. There
> was no progress in the past week.
> 
> I’m planning to pick up the task on Monday. I was wondering if the
> inclusion of our fixes for a 13-year-old kernel benefits anyone
> except my niche use case.

I'm not sure, but critical regression on supported kernel is always a
bug that needs to be fixed. This is actually probably serious enough
to merit a follow-up to the 1.1.x series, though I'll hold off for
anything else that might come up in the next month or two.

Rich

Re: [musl] mips32 little endian -ENOSYS is not -(-ENOSYS)

[Andreas Dröscher]

Am 20.03.20 um 23:33 schrieb Rich Felker:
> On Fri, Mar 20, 2020 at 11:18:56PM +0100, Andreas Dröscher wrote:
>>
>> Sorry for now sending an update in a timely manner. I've tested your
>> patches and they definitely get me to the point I had with my manual
>> changes. However, the incompatibilities I reported earlier stil need
>> to be ironed out. Sadly I got sidetracked by another project. There
>> was no progress in the past week.
>>
>> I’m planning to pick up the task on Monday. I was wondering if the
>> inclusion of our fixes for a 13-year-old kernel benefits anyone
>> except my niche use case.
> 
> I'm not sure, but critical regression on supported kernel is always a
> bug that needs to be fixed. This is actually probably serious enough
> to merit a follow-up to the 1.1.x series, though I'll hold off for
> anything else that might come up in the next month or two.

I still own you an update.

I had to define SYSCALL_USE_SOCKETCALL to get the network stack working:

diff --git a/arch/mips/syscall_arch.h b/arch/mips/syscall_arch.h
index 17e9fa56..96998deb 100644
--- a/arch/mips/syscall_arch.h
+++ b/arch/mips/syscall_arch.h
@@ -147,3 +147,5 @@ static inline long __syscall7(long n, long a, long b, long 
c, long d, long e, lo

#define SO_SNDTIMEO_OLD 0x1005
#define SO_RCVTIMEO_OLD 0x1006
+
+#define SYSCALL_USE_SOCKETCALL

Next to that I'm struggling to get an old OS abstraction layer working (ACE).
The library somehow manages to create pthread mutexes where even single threaded
process gets stuck during a pthread_mutexe_lock. I assume the data structure is
not properly initialized.

I'm additionally battling with ACE-Threads randomly returning from select/epoll 
with EINTR. Both issues are very vague at the moment. Since all native daemons
(e.g. ssh, mosquito, dbus, avahi) are up and running now, I expect the issue to
be caused by the OS abstraction layer.

Andreas