[musl] no 32bit timestamp compatible stat/lstat/fstat?

[musl] no 32bit timestamp compatible stat/lstat/fstat?

[Norbert van Bolhuis]

Hi All,

Why does musl-1.2.x not offer the 32bit timestamp compatible stat/lstat/fstat?

We're using a 3rd-party binary (compiled against glibc) on an arm32v7 platform
which fails to execute ever since we jumped from musl v1.1.x to 1.2.x, see:

/ # wl
Error relocating /usr/sbin/wl: __fxstat: symbol not found

The arm32v7 glibc-2.3x (working with 64bit timestamps) does provide them.

Looking at: https://www.openwall.com/lists/musl/2019/08/01/1
it seems this is a known limitation.

Any plans on fixing this?

I solved my case by preloading a shared library for which the source is:


#include <sys/stat.h>

extern int __fxstat64(int ver, int fd, struct stat32 *buf);

int __fxstat(int ver, int fd, struct stat32 *buf)
{
         return __fxstat64(ver, fd, buf);
}


Regards,
Norbert van Bolhuis


Btw. please CC me on replies (as I'm not subscribed to musl@lists.openwall.com)

Re: [musl] no 32bit timestamp compatible stat/lstat/fstat?

[Rich Felker]

On Mon, Nov 29, 2021 at 02:11:47PM +0100, Norbert van Bolhuis wrote:
> Hi All,
> 
> Why does musl-1.2.x not offer the 32bit timestamp compatible stat/lstat/fstat?
> 
> We're using a 3rd-party binary (compiled against glibc) on an arm32v7 platform
> which fails to execute ever since we jumped from musl v1.1.x to 1.2.x, see:
> 
> / # wl
> Error relocating /usr/sbin/wl: __fxstat: symbol not found

This is not the glibc symbol for 32-bit time_t but for 32-bit off_t, a
very bad legacy ABI that musl has never supported and that shouldn't
be used. Since it was not compiled with -D_FILE_OFFSET_BITS=64, any
calls from this code to functions that take off_t, or structures
containing off_t, will be wrong. If the only such functions are the
stat family, you can probably make shims to convert and make it work.
If it has calls to other functions like lseek, etc., then you may need
to also patch the binary to call shims for those.

Of course the best and easiest solution if possible would be to get
the provider of this library to recompile it properly with
-D_FILE_OFFSET_BITS=64, since without that it cannot handle large
files or filesystems where inode numbers may not fit in 32 bits. This
problem has nothing to do with musl; it's already a problem if you're
using their code on glibc.

> The arm32v7 glibc-2.3x (working with 64bit timestamps) does provide them.
> 
> Looking at: https://www.openwall.com/lists/musl/2019/08/01/1
> it seems this is a known limitation.

No, this was resolved, and it's a different issue: allowing
glibc-built code to call the "lfs64" (_FILE_OFFSET_BITS=64) glibc
names for the 64-bit off_t functions rather than their unadorned names
(which is what musl uses natively since off_t is always 64-bit).

> 
> Any plans on fixing this?
> 
> I solved my case by preloading a shared library for which the source is:
> 
> 
> #include <sys/stat.h>
> 
> extern int __fxstat64(int ver, int fd, struct stat32 *buf);
> 
> int __fxstat(int ver, int fd, struct stat32 *buf)
> {
>         return __fxstat64(ver, fd, buf);
> }

That's insufficient because __fxstat is not expected to write the
version of the structure __fxstat64 would write. It's expected to
write some ancient legacy version of the sturcture (from the 1990s)
where the offset and inode fields are only 32-bit. A working shim
would require looking up (in the glibc headers) what that format is,
and writing the code to convert on success.

Rich

Re: [musl] no 32bit timestamp compatible stat/lstat/fstat?

[Norbert van Bolhuis]

Hi Rich,

Thanks for your answer!

Ah, it's about 32-bit off_t compatibility, but then why
does musl-1.1.24 (and the corresponding libc6-compat package on
Alpine 3.12) provide this (ancient) __fxstat function?

So this can never work properly and the reason why it worked with
Alpine 3.12 (musl-1.1.24) is that the 3rd-party binary never really
used __fxstat (because there is no way the 64-bit structure is
compatible with the 32-bit one), right?

Glibc does provide the ancient 32-bit variant, see:
nm -D --defined-only /lib/arm-linux-gnueabihf/libc.so.6 | grep __fxstat
00093e68 T __fxstat
00093fdc T __fxstat64
000942a4 T __fxstatat
00094320 T __fxstatat64
I would expect musl libc6-compat package to provide it, why it doesn't?

Anyway, I'll certainly ask the provider of the 3rd-party binary to
recompile with -D_FILE_OFFSET_BITS=64.

I may lookup this ancient 32-bit 'struct stat' and write a shim, another
short-term "solution" is to simply return EOVERFLOW. This will tell me
if __fxstat is called anywayr. Right now I have no idea, it sure looks like
we don't need it for our use-case since we never had an issue with
musl-1.1.x.

Regards,
Norbert


On 11/29/21 6:35 PM, Rich Felker wrote:
> On Mon, Nov 29, 2021 at 02:11:47PM +0100, Norbert van Bolhuis wrote:
>> Hi All,
>>
>> Why does musl-1.2.x not offer the 32bit timestamp compatible stat/lstat/fstat?
>>
>> We're using a 3rd-party binary (compiled against glibc) on an arm32v7 platform
>> which fails to execute ever since we jumped from musl v1.1.x to 1.2.x, see:
>>
>> / # wl
>> Error relocating /usr/sbin/wl: __fxstat: symbol not found
> 
> This is not the glibc symbol for 32-bit time_t but for 32-bit off_t, a
> very bad legacy ABI that musl has never supported and that shouldn't
> be used. Since it was not compiled with -D_FILE_OFFSET_BITS=64, any
> calls from this code to functions that take off_t, or structures
> containing off_t, will be wrong. If the only such functions are the
> stat family, you can probably make shims to convert and make it work.
> If it has calls to other functions like lseek, etc., then you may need
> to also patch the binary to call shims for those.
> 
> Of course the best and easiest solution if possible would be to get
> the provider of this library to recompile it properly with
> -D_FILE_OFFSET_BITS=64, since without that it cannot handle large
> files or filesystems where inode numbers may not fit in 32 bits. This
> problem has nothing to do with musl; it's already a problem if you're
> using their code on glibc.
> 
>> The arm32v7 glibc-2.3x (working with 64bit timestamps) does provide them.
>>
>> Looking at: https://www.openwall.com/lists/musl/2019/08/01/1
>> it seems this is a known limitation.
> 
> No, this was resolved, and it's a different issue: allowing
> glibc-built code to call the "lfs64" (_FILE_OFFSET_BITS=64) glibc
> names for the 64-bit off_t functions rather than their unadorned names
> (which is what musl uses natively since off_t is always 64-bit).
> 
>>
>> Any plans on fixing this?
>>
>> I solved my case by preloading a shared library for which the source is:
>>
>>
>> #include <sys/stat.h>
>>
>> extern int __fxstat64(int ver, int fd, struct stat32 *buf);
>>
>> int __fxstat(int ver, int fd, struct stat32 *buf)
>> {
>>          return __fxstat64(ver, fd, buf);
>> }
> 
> That's insufficient because __fxstat is not expected to write the
> version of the structure __fxstat64 would write. It's expected to
> write some ancient legacy version of the sturcture (from the 1990s)
> where the offset and inode fields are only 32-bit. A working shim
> would require looking up (in the glibc headers) what that format is,
> and writing the code to convert on success.
> 
> Rich
> 

Re: [musl] no 32bit timestamp compatible stat/lstat/fstat?

[Rich Felker]

On Tue, Nov 30, 2021 at 12:15:59AM +0100, Norbert van Bolhuis wrote:
> Hi Rich,
> 
> Thanks for your answer!
> 
> Ah, it's about 32-bit off_t compatibility, but then why
> does musl-1.1.24 (and the corresponding libc6-compat package on
> Alpine 3.12) provide this (ancient) __fxstat function?

The symbol there in 1.1.x didn't behave as needed to make 32-bit-off_t
glibc code work. It just happened to be there as an alternate name for
the 64-bit-off_t function __fxstat64 as a consequence of 32-bit archs
not being special-cased.

> So this can never work properly and the reason why it worked with
> Alpine 3.12 (musl-1.1.24) is that the 3rd-party binary never really
> used __fxstat (because there is no way the 64-bit structure is
> compatible with the 32-bit one), right?

Right. Well, it might have 'worked' if it didn't actually inspect the
fields of the structure, but since the 64-bit structure is larger than
the object the caller provided, it was overflowing a buffer and
probably clobbering other data.

> Glibc does provide the ancient 32-bit variant, see:
> nm -D --defined-only /lib/arm-linux-gnueabihf/libc.so.6 | grep __fxstat
> 00093e68 T __fxstat
> 00093fdc T __fxstat64
> 000942a4 T __fxstatat
> 00094320 T __fxstatat64
> I would expect musl libc6-compat package to provide it, why it doesn't?

There's no way to support the 32-bit-off_t ABI with just a library,
because the symbol names clash. It would require musl's dynamic linker
to be specifically aware of foreign glibc libraries and remap their
symbol references. This would be something of a cooperative project
between musl and the folks working on gcompat to come up with a
contract between them for making it all work -- something we've talked
about as a vague future goal.

Rich