[musl] _GNU_SOURCE and _LARGEFILE_SOURCE

[musl] _GNU_SOURCE and _LARGEFILE_SOURCE

[Brian Cain]

In the "WHATSNEW" text, there's an item from 0.9.2 that states "- make 
_GNU_SOURCE imply _LARGEFILE64_SOURCE".  Is that intended to be the case 
generally?  I ask because in 25e6fee2 (remove LFS64 programming 
interfaces (macro-only) from _GNU_SOURCE, 2022-09-27) it stated 
"portable software should be prepared for them not to exist" and "the 
intent is that this be a very short-term measure and that the macros be 
removed entirely in the next release cycle."

This comes up because in the QEMU project, there's a linux multiarch 
test case that uses readdir64 and it does define _GNU_SOURCE but does 
not define _LARGEFILE64_SOURCE and as such the cross compiler complains 
that there's no declaration of readdir64 before the call site.  I 
suppose that the test case would be more portable if it defined 
_LARGEFILE64_SOURCE.  But I'd also be happy to send a patch to musl that 
could have _GNU_SOURCE imply _LARGEFILE64_SOURCE (again?) if that's 
desired.  But - I gather that defining the macros is not what we want.  
Instead of macros I should add declarations for readdir64() and its 
LFS64 friends, but only when _LARGEFILE64_SOURCE is defined?

-Brian

Re: [musl] _GNU_SOURCE and _LARGEFILE_SOURCE

[Markus Wichmann]

Am Tue, Sep 17, 2024 at 07:29:17PM -0500 schrieb Brian Cain:
> In the "WHATSNEW" text, there's an item from 0.9.2 that states "- make
> _GNU_SOURCE imply _LARGEFILE64_SOURCE".  Is that intended to be the case
> generally?  I ask because in 25e6fee2 (remove LFS64 programming interfaces
> (macro-only) from _GNU_SOURCE, 2022-09-27) it stated "portable software
> should be prepared for them not to exist" and "the intent is that this be a
> very short-term measure and that the macros be removed entirely in the next
> release cycle."
>
> This comes up because in the QEMU project, there's a linux multiarch test
> case that uses readdir64 and it does define _GNU_SOURCE but does not define
> _LARGEFILE64_SOURCE and as such the cross compiler complains that there's no
> declaration of readdir64 before the call site.  I suppose that the test case
> would be more portable if it defined _LARGEFILE64_SOURCE.  But I'd also be
> happy to send a patch to musl that could have _GNU_SOURCE imply
> _LARGEFILE64_SOURCE (again?) if that's desired.  But - I gather that
> defining the macros is not what we want.  Instead of macros I should add
> declarations for readdir64() and its LFS64 friends, but only when
> _LARGEFILE64_SOURCE is defined?
>
> -Brian
>

The LFS64 interfaces are not in POSIX, so you cannot assume they exist.
Rather, you can test for their existance with a configure test, and if
it fails fall back to the portable interface. Or just use the portable
interface in the first place.

If you want to compile portable code on glibc, then define
_FILE_OFFSET_BITS to 64 and you get the exact same interface!

Ciao,
Markus

Re: [musl] _GNU_SOURCE and _LARGEFILE_SOURCE

[Brian Cain]

[-- Attachment #1: Type: text/plain, Size: 2931 bytes --]


On 9/17/2024 9:57 PM, Markus Wichmann wrote:
> Am Tue, Sep 17, 2024 at 07:29:17PM -0500 schrieb Brian Cain:
>> In the "WHATSNEW" text, there's an item from 0.9.2 that states "- make
>> _GNU_SOURCE imply _LARGEFILE64_SOURCE".  Is that intended to be the case
>> generally?  I ask because in 25e6fee2 (remove LFS64 programming interfaces
>> (macro-only) from _GNU_SOURCE, 2022-09-27) it stated "portable software
>> should be prepared for them not to exist" and "the intent is that this be a
>> very short-term measure and that the macros be removed entirely in the next
>> release cycle."
>>
>> This comes up because in the QEMU project, there's a linux multiarch test
>> case that uses readdir64 and it does define _GNU_SOURCE but does not define
>> _LARGEFILE64_SOURCE and as such the cross compiler complains that there's no
>> declaration of readdir64 before the call site.  I suppose that the test case
>> would be more portable if it defined _LARGEFILE64_SOURCE.  But I'd also be
>> happy to send a patch to musl that could have _GNU_SOURCE imply
>> _LARGEFILE64_SOURCE (again?) if that's desired.  But - I gather that
>> defining the macros is not what we want.  Instead of macros I should add
>> declarations for readdir64() and its LFS64 friends, but only when
>> _LARGEFILE64_SOURCE is defined?
>>
>> -Brian
>>
> The LFS64 interfaces are not in POSIX, so you cannot assume they exist.
> Rather, you can test for their existance with a configure test, and if
> it fails fall back to the portable interface. Or just use the portable
> interface in the first place.
>
> If you want to compile portable code on glibc, then define
> _FILE_OFFSET_BITS to 64 and you get the exact same interface!

Okay, this is a bit new to me, so let me see if I am following along:

musl defines a readdir symbol and no readdir64 symbol, because readdir64 
is not specified by POSIX only by Single Unix Specification?  But at one 
time readdir64 (et al) mappings were provided for _GNU_SOURCE and those 
now remain but only under _LARGEFILE_SOURCE.  At some future date those 
mappings might no longer be provided? And musl does not want to take 
patches that define a readdir64 symbol because that would be beyond the 
scope of POSIX.  Applications that want to be portable among POSIX 
targets can define _FILE_OFFSET_BITS to 64 and this will have no effect 
on musl type definitions nor function declarations.  But when using that 
define, glibc's type definitions for off_t (et al) will be such that 
they (1) can be used to represent enough values for large files among 
32-bit and 64-bit targets, even when calling readdir-not-readdir64 and 
(2) are compatible / correspond with musl's type definitions?  The 
wording of that last part is not-quite-right, but I am capturing some of 
the essence there? "we'll end up with those same semantics that musl has 
in the absence of _GNU_SOURCE / _LARGEFILE_SOURCE" maybe?


-Brian

[-- Attachment #2: Type: text/html, Size: 3564 bytes --]

Re: [musl] _GNU_SOURCE and _LARGEFILE_SOURCE

[Markus Wichmann]

Am Wed, Sep 18, 2024 at 12:06:44AM -0500 schrieb Brian Cain:
> Okay, this is a bit new to me, so let me see if I am following along:
>
> musl defines a readdir symbol and no readdir64 symbol, because readdir64 is
> not specified by POSIX only by Single Unix Specification?  But at one time
> readdir64 (et al) mappings were provided for _GNU_SOURCE and those now
> remain but only under _LARGEFILE_SOURCE.  At some future date those mappings
> might no longer be provided? And musl does not want to take patches that
> define a readdir64 symbol because that would be beyond the scope of POSIX. 
> Applications that want to be portable among POSIX targets can define
> _FILE_OFFSET_BITS to 64 and this will have no effect on musl type
> definitions nor function declarations.  But when using that define, glibc's
> type definitions for off_t (et al) will be such that they (1) can be used to
> represent enough values for large files among 32-bit and 64-bit targets,
> even when calling readdir-not-readdir64 and (2) are compatible / correspond
> with musl's type definitions?  The wording of that last part is
> not-quite-right, but I am capturing some of the essence there? "we'll end up
> with those same semantics that musl has in the absence of _GNU_SOURCE /
> _LARGEFILE_SOURCE" maybe?
>
>
> -Brian

That is all largely correct. See, the only reason for readdir64 to exist
is because originally, ino_t was defined as the system word size, and
readdir returns inodes. So glibc added a type ino64_t, and a readdir64()
that returns pointers to struct dirent64. Now they've changed to
defining ino_t to either a system word or a 64-bit word depending on the
state of _FILE_OFFSET_BITS. And I don't know what magic they use to
redirect the calls to readdir() accordingly.

All of which is mostly pointless, since the inode as returned by
readdir() can't be used for anything by userspace. If the directory
entry is a mountpoint, then the inode number given is not even the same
as the number stat() would return.

But anyway, this means:

- there has never been a combination of Linux and libc version that had
  anything other than 64-bit inodes (and file offsets) on 64-bit targets
- on 32-bit targets, musl always defines inodes and file offsets, and
  glibc does the same when _FILE_OFFSET_BITS=64 is defined.
- the only reason to use the LFS64 symbols is compatibility with (at
  this point) really old glibc versions on 32-bit targets.

Note that you can just use compile tests to see if you need the LFS64
symbols, and if they are available. If sizeof(ino_t) == 8, you don't
need readdir64() at all.

Ciao,
Markus

Re: [musl] _GNU_SOURCE and _LARGEFILE_SOURCE

[Markus Wichmann]

Am Wed, Sep 18, 2024 at 04:24:11PM +0200 schrieb Markus Wichmann:
> - the only reason to use the LFS64 symbols is compatibility with (at
>   this point) really old glibc versions on 32-bit targets.

Have to correct myself on that one. I said old when I should have said
ancient. A quick check in git showed that support for _FILE_OFFSET_BITS
was added in October of 1997. If you don't know what that means: That
was between Pricess Diana's funeral and the Kyoto Protocol.

So there really is absolutely no reason to use LFS64 functions ever.

Ciao,
Markus