[musl] Re: [PATCH] memmem.3: Added list of known systems where this is available

[musl] Re: [PATCH] memmem.3: Added list of known systems where this is available

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 5547 bytes --]

Hi all!

On 11/23/22 13:52, Stefan Puiu wrote:
> Hi,
> 
> On Wed, Nov 23, 2022 at 1:39 AM Guillem Jover <guillem@hadrons.org> wrote:
>>
>> Hi!
>>
>> On Thu, 2022-11-10 at 12:36:47 +0100, Alejandro Colomar wrote:
>>> On 11/10/22 01:13, Andrew Clayton wrote:
>>>> While looking at which systems provide memmem(3) I have been able to
>>>> discern the following:
>>>>
>>>>     musl libc since v0.9.7
>>>>     bionic since Android 9
>>>>
>>>>     FreeBSD since 6.0
>>>>     OpenBSD since 5.4
>>>>     NetBSD
>>>>     macOS
>>>>     Illumos
>>
>>>> For macOS and Illumos I checked the memmem(3) man page on those systems.
>>>> For the rest there are links below to on-line man pages or commit logs.
>>
>>>> +FreeBSD 6.0, OpenBSD 5.4, NetBSD, macOS & Illumos.
>>
>>> For the commit message, it's interesting to note macOS and Bionic, for
>>> speleology purposes.  However, I'm opposed to adding them to the page itself
>>> because of the following:
>>>
>>> -  macOS is not free software.  I refuse to reference nonfree software on
>>> this project.
>>
>> I understand where you are coming from,

:-)

>> and from a full system PoV,
>> it's very true that macOS is non-free. But the lower layers of its
>> stack are free software (at least according to the FSF and OSI), such
>> as its kernel (Darwin), or its libc (where memmem() is implemented):
>>
>>    https://opensource.apple.com/source/Libc/Libc-1439.40.11/
>>
>> among others.

Hmmm.  Fair enough.

>>
>> Similar with Solaris and Illumos (but with the CDDL, MIT, BSD), AFAIUI.
>>
>>> -  Android is not a real Unix system, in that you can't even program in C in
>>> there, unless you're Google or have hacked your system.  It's not friendly
>>> to us programmers, so we don't need to be friendly to it.  I don't want to
>>> be cluttering the pages with information that is irrelevant to normal users.
>>
>> I'm assuming bionic is being used in some of the Android free
>> alternatives too, but then I'm not sure how usable for programming those
>> are either. And, well musl libc is not a real Unix system you can program
>> against either. :)

The difference is only that bionic is not in use in useful systems (AFAIK).  :)

So we have to do some decission here (and also about newlib, as reported by Brian).

>>
>> In any case I also find it useful to have this kind of portability
>> information when deciding what to use in code.

And I must admit it's also useful to me (this all started because Andrew and I 
had to use memmem(3) at a project where macOS compatibility is relevant --not 
critical, but relevant--).

>> But can understand the
>> reluctance to include such references, at least if thought out as the
>> entire system. Perhaps thinking about these merely at their kernel or
>> libc level would make including information about some of them more
>> palatable, given that standalone they are free software? So perhaps
>> an option is to instead refer to their specific components, say
>> "bionic libc X.Y", "Apple Libc M.N.O" or similar.

Yep, that's more palatable :)

I think I'd accept a patch stating Apple Libc version for memmem(3).

> 
> Not sure if it's the job of Linux man-pages to document when other
> OSes started supporting certain APIs. That info has to be maintained,
> updated etc. People can always read the man pages of other systems,
> right? Maybe it's worth only pointing out when an interface is
> Linux-only, or the Linux implementation diverges significantly.

The good thing is that in most cases, it's either something in POSIX (for which 
I gon't care at all if Apple Libc or another-weirdo-libc decide to not support 
it), or it's a Linux-only function.

So, there are few functions or syscalls that are generally available but are not 
in POSIX, and it might be interesting to document that they're effectively as 
portable as anything POSIX.  Maybe even POSIX editors read this and decide to 
add it.

In those cases, we still need to decide what we care about or not.

Musl libc is definitely a first-class citizen these days in Linux distributions. 
  I would start documenting them in the project at large if someone from musl 
provides patches (I discussed this some time ago, but can't remember with who).
Rich, if you would like to discuss about this, we can have some chat.

> 
> For musl and other libcs, I think the man pages already document some
> instances where their behavior diverges from glibc.

As said, for musl, I'd document them officially, if there's anyone interested 
enough to send patches.

For other libcs, we have to decide if they're important enough, and probably 
decide on a case-by-case basis.

Michael tried to have some decent coverage of non-GNU/Linux systems, both in the 
man-pages and in his TLPI book.  It's a useful thing.  So much that sometimes 
you don't even need to read other systems' pages at all to know how portable is 
a GNU/Linux function.

So, I'd like to get opinions on interest about documenting details about:

- newlib (I never heard about it before; is it a widespread thing? do you think 
it's useful?)
- Apple Libc (I still don't like it, but I must admit that it's relevant, and 
being open source, it's more palatable)
- bionic (does anyone know if that's useful at all for anyone at all?)

Support for those wouldn't go as far as musl or glibc.  For now it would only be 
for memmem(3).


Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [musl] Re: [PATCH] memmem.3: Added list of known systems where this is available

[Jeffrey Walton]

On Wed, Nov 23, 2022 at 9:29 AM Alejandro Colomar
<alx.manpages@gmail.com> wrote:
>  ...
> >> In any case I also find it useful to have this kind of portability
> >> information when deciding what to use in code.
>
> And I must admit it's also useful to me (this all started because Andrew and I
> had to use memmem(3) at a project where macOS compatibility is relevant --not
> critical, but relevant--).

If you are a die-hard free software person using GNU gear, then Gnulib
provides memmem. There's no need to worry about availability or
portability courtesy of Gnulib. See
https://www.gnu.org/software/gnulib/manual/html_node/memmem.html .

Jeff

Re: [musl] Re: [PATCH] memmem.3: Added list of known systems where this is available

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 1778 bytes --]

Hi Jeffrey,

On 11/23/22 15:55, Jeffrey Walton wrote:
> On Wed, Nov 23, 2022 at 9:29 AM Alejandro Colomar
> <alx.manpages@gmail.com> wrote:
>>   ...
>>>> In any case I also find it useful to have this kind of portability
>>>> information when deciding what to use in code.
>>
>> And I must admit it's also useful to me (this all started because Andrew and I
>> had to use memmem(3) at a project where macOS compatibility is relevant --not
>> critical, but relevant--).
> 
> If you are a die-hard free software person using GNU gear, then Gnulib
> provides memmem. There's no need to worry about availability or
> portability courtesy of Gnulib. See
> https://www.gnu.org/software/gnulib/manual/html_node/memmem.html .

Thanks!

However, for this project I was talking about, it's not an option; such a 
dependency would not be accepted.

BTW, personally, I always found very confusing the usage of Gnulib compared to 
normal packaged libraries.  Maybe it's just me; don't know.  It also forces you 
to use GNU autotools, which I don't like at all.  I prefer the approach of 
libbsd, which just provides a couple of pc(5) files to allow using as a library 
or as an overlay over the system libc, and after that you're fine with whatever 
build system you prefer.  I know it has some issues, such as 
<https://gitlab.freedesktop.org/libbsd/libbsd/-/issues/5>, which may be the 
reason Gnulib works that way, I don't know.

Guillem, do you think that issue with libbsd and <queue.h> can be fixed?  Or is 
it an inherent issue of the way the overlay works?  Maybe it would be 
interesting to fix it, as a proof of concept that something like Gnulib could be 
implemented in that way.

Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [musl] [PATCH] memmem.3: Added list of known systems where this is available

[Shiz]

Hi there,

> On 23 Nov Reiwa 4, at 14:16, Alejandro Colomar <alx.manpages@gmail.com> wrote:
> 
>>>> -  Android is not a real Unix system, in that you can't even program in C in
>>>> there, unless you're Google or have hacked your system.  It's not friendly
>>>> to us programmers, so we don't need to be friendly to it.  I don't want to
>>>> be cluttering the pages with information that is irrelevant to normal users.
>>> 
>>> I'm assuming bionic is being used in some of the Android free
>>> alternatives too, but then I'm not sure how usable for programming those
>>> are either. And, well musl libc is not a real Unix system you can program
>>> against either. :)
> 
> The difference is only that bionic is not in use in useful systems (AFAIK).  :)
> 
> So we have to do some decission here (and also about newlib, as reported by Brian).

I’d like to chime in here and note that, valuations of usefulness aside, it *is*
possible to develop in C against an Android device, without needing any hacks,
using the NDK[1]. As someone who has occasionally worked on big codebases written
in C and C++ for Android devices, I would argue there’s value to be had there. :-)

> Cheers,
> 
> Alex

- Shiz

[1]: https://developer.android.com/ndk

Re: [musl] [PATCH] memmem.3: Added list of known systems where this is available

[enh]

[-- Attachment #1: Type: text/plain, Size: 4127 bytes --]

On Wed, Nov 23, 2022 at 7:33 AM Shiz <hi@shiz.me> wrote:

> Hi there,
>
> > On 23 Nov Reiwa 4, at 14:16, Alejandro Colomar <alx.manpages@gmail.com>
> wrote:
> >
> >>>> -  Android is not a real Unix system, in that you can't even program
> in C in
> >>>> there, unless you're Google or have hacked your system.  It's not
> friendly
> >>>> to us programmers, so we don't need to be friendly to it.  I don't
> want to
> >>>> be cluttering the pages with information that is irrelevant to normal
> users.
> >>>
> >>> I'm assuming bionic is being used in some of the Android free
> >>> alternatives too, but then I'm not sure how usable for programming
> those
> >>> are either. And, well musl libc is not a real Unix system you can
> program
> >>> against either. :)
> >
> > The difference is only that bionic is not in use in useful systems
> (AFAIK).  :)
> >
> > So we have to do some decission here (and also about newlib, as reported
> by Brian).
>
> I’d like to chime in here and note that, valuations of usefulness aside,
> it *is*
> possible to develop in C against an Android device, without needing any
> hacks,
> using the NDK[1]. As someone who has occasionally worked on big codebases
> written
> in C and C++ for Android devices, I would argue there’s value to be had
> there. :-)
>

plus i think bionic is the only libc that explicitly links to man7.org as
the canonical source for documentation in its header files :-)

funnily enough, this topic ("should the man pages talk about availability
in libcs other than glibc?") came up before, and specifically for bionic
mkerrisk seemed to think it was a reasonable idea ... but i (bionic
maintainer) was less convinced. although i think that, yes, if someone's
reading that part of the page, realistically they're either interested in
macOS/iOS or Android (sorry, AIX or IRIX fans!), but specifically for those
two systems the header files already contain machine-readable availability
information via __attribute__((__available__)).

fwiw, here's an example from bionic's <unistd.h> illustrating both of those
points:

```
/**
 * [copy_file_range(2)](
https://man7.org/linux/man-pages/man2/copy_file_range.2.html) copies
 * a range of data from one file descriptor to another.
 *
 * Returns the number of bytes copied on success, and returns -1 and sets
`errno`
 * on failure.
 *
 * Available since API level 34.
 */
ssize_t copy_file_range(int __fd_in, off64_t* __off_in, int __fd_out,
off64_t* __off_out, size_t __length, unsigned int __flags)
__INTRODUCED_IN(34);
```

(if there's anything android-specific to note, we also mention that, but
there are relatively few examples of those, and it's mostly just "SELinux
means that only system processes can enable/disable swap" or whatever.)

so it wasn't clear to me whether -- unless you automated it -- there was
much value to duplicating that in the man pages? if you try to compile,
you're going to get a clear compiler error saying "you need to be targeting
OS version $whatever" anyway.

i think the real question is "aye, but will they get that far?". for my
part, i think the most value would come from cases that talk about "GNU
extension" being a bit clearer whether that's "unique to glibc" (which a
lot of people assume is what it means) or "initially in glibc, but a good
enough idea that it's effectively ubiquitous now, even if it's not in POSIX
because Solaris will never add it". and maybe the best way to clarify stuff
like that is just to explicitly state "Available on macOS since 10.4.
Available on Android since API level 23."? dunno.

if anyone _does_ want to go that route, i'd be happy to help with the "raw
data" side. (and certainly if anyone does start adding that, you'll force
me to care about maintaining it :-) )

but my biggest problem right now is that man7.org isn't being updated, and
i can't get hold of mkerrisk to try to offer him money to do it :-( (i do
hope he's okay, wherever he is!)


> > Cheers,
> >
> > Alex
>
> - Shiz
>
> [1]: https://developer.android.com/ndk

[-- Attachment #2: Type: text/html, Size: 5417 bytes --]

Re: [musl] [PATCH] memmem.3: Added list of known systems where this is available

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 5153 bytes --]

Hi!

On 11/30/22 20:39, enh wrote:
> 
> 
> On Wed, Nov 23, 2022 at 7:33 AM Shiz <hi@shiz.me <mailto:hi@shiz.me>> wrote:
> 
>     Hi there,
> 
>      > On 23 Nov Reiwa 4, at 14:16, Alejandro Colomar <alx.manpages@gmail.com
>     <mailto:alx.manpages@gmail.com>> wrote:
>      >
>      >>>> -  Android is not a real Unix system, in that you can't even program
>     in C in
>      >>>> there, unless you're Google or have hacked your system.  It's not friendly
>      >>>> to us programmers, so we don't need to be friendly to it.  I don't want to
>      >>>> be cluttering the pages with information that is irrelevant to normal
>     users.
>      >>>
>      >>> I'm assuming bionic is being used in some of the Android free
>      >>> alternatives too, but then I'm not sure how usable for programming those
>      >>> are either. And, well musl libc is not a real Unix system you can program
>      >>> against either. :)
>      >
>      > The difference is only that bionic is not in use in useful systems
>     (AFAIK).  :)
>      >
>      > So we have to do some decission here (and also about newlib, as reported
>     by Brian).
> 
>     I’d like to chime in here and note that, valuations of usefulness aside, it *is*
>     possible to develop in C against an Android device, without needing any hacks,
>     using the NDK[1]. As someone who has occasionally worked on big codebases
>     written
>     in C and C++ for Android devices, I would argue there’s value to be had
>     there. :-)
> 
> 
> plus i think bionic is the only libc that explicitly links to man7.org 
> <http://man7.org> as the canonical source for documentation in its header files :-)

:-)

> 
> funnily enough, this topic ("should the man pages talk about availability in 
> libcs other than glibc?") came up before, and specifically for bionic mkerrisk 
> seemed to think it was a reasonable idea ...

I agree with Michael.

> but i (bionic maintainer) was less 
> convinced. although i think that, yes, if someone's reading that part of the 
> page, realistically they're either interested in macOS/iOS or Android (sorry, 
> AIX or IRIX fans!),

yep.

> but specifically for those two systems the header files 
> already contain machine-readable availability information via 
> __attribute__((__available__)).
> 
> fwiw, here's an example from bionic's <unistd.h> illustrating both of those points:
> 
> ```
> /**
>   * 
> [copy_file_range(2)](https://man7.org/linux/man-pages/man2/copy_file_range.2.html <https://man7.org/linux/man-pages/man2/copy_file_range.2.html>) copies
>   * a range of data from one file descriptor to another.
>   *
>   * Returns the number of bytes copied on success, and returns -1 and sets `errno`
>   * on failure.
>   *
>   * Available since API level 34.
>   */
> ssize_t copy_file_range(int __fd_in, off64_t* __off_in, int __fd_out, off64_t* 
> __off_out, size_t __length, unsigned int __flags) __INTRODUCED_IN(34);
> ```
> 
> (if there's anything android-specific to note, we also mention that, but there 
> are relatively few examples of those, and it's mostly just "SELinux means that 
> only system processes can enable/disable swap" or whatever.)
> 
> so it wasn't clear to me whether -- unless you automated it -- there was much 
> value to duplicating that in the man pages? if you try to compile, you're going 
> to get a clear compiler error saying "you need to be targeting OS version 
> $whatever" anyway.
> 
> i think the real question is "aye, but will they get that far?". for my part, i 
> think the most value would come from cases that talk about "GNU extension" being 
> a bit clearer whether that's "unique to glibc" (which a lot of people assume is 
> what it means) or "initially in glibc, but a good enough idea that it's 
> effectively ubiquitous now, even if it's not in POSIX because Solaris will never 
> add it". and maybe the best way to clarify stuff like that is just to explicitly 
> state "Available on macOS since 10.4. Available on Android since API level 23."? 
> dunno.
> 
> if anyone _does_ want to go that route, i'd be happy to help with the "raw data" 
> side. (and certainly if anyone does start adding that, you'll force me to care 
> about maintaining it :-) )

I don't "want", as in, doing a global dump of data into the pages.  I have 
better things in which to spend my time.  However, if someone sends a patch for 
one or another page, I'll accept it with the following requirements:  It 
references a free software libc.  So for example, macOS no, Apple libc yes.

> 
> but my biggest problem right now is that man7.org <http://man7.org> isn't being 
> updated, and i can't get hold of mkerrisk to try to offer him money to do it :-( 

I talked to him some months ago.  The last times, he was always very busy.  Good 
luck contacting him!  I still hope he will reappear some day...

If I see him again, I'll mention this offer to him.

> (i do hope he's okay, wherever he is!)

I hope he is!


Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[musl] Re: [PATCH] memmem.3: Added list of known systems where this is available

[Stefan Puiu]

Hi,

On Wed, Nov 23, 2022 at 3:16 PM Alejandro Colomar
<alx.manpages@gmail.com> wrote:
>
> Hi all!
>
[... snipped ...]
> >
> > Not sure if it's the job of Linux man-pages to document when other
> > OSes started supporting certain APIs. That info has to be maintained,
> > updated etc. People can always read the man pages of other systems,
> > right? Maybe it's worth only pointing out when an interface is
> > Linux-only, or the Linux implementation diverges significantly.
>
> The good thing is that in most cases, it's either something in POSIX (for which
> I gon't care at all if Apple Libc or another-weirdo-libc decide to not support
> it), or it's a Linux-only function.
>
> So, there are few functions or syscalls that are generally available but are not
> in POSIX, and it might be interesting to document that they're effectively as
> portable as anything POSIX.  Maybe even POSIX editors read this and decide to
> add it.
>
> In those cases, we still need to decide what we care about or not.

Aah OK, so memmem is non-standard, there is no standard to point to.
Do other OSes provide this kind of info? I don't see anything about
portability in the OpenBSD man page (https://man.openbsd.org/memmem),
only "memmem() is a GNU extension". The FreeBSD man page
(https://www.freebsd.org/cgi/man.cgi?query=memmem&manpath=FreeBSD+13.1-RELEASE+and+Ports)
does mention Linux, but only to mention that memmem was broken in old
versions of Linux libc (nothing about current Linux); it also has the
'GNU extension' mention. Interestingly, I don't see the mention of the
function being a GNU extension in the Linux version.

Have you checked, are there many such functions? Do you plan to add
this info for all of them?

>
> Musl libc is definitely a first-class citizen these days in Linux distributions.
>   I would start documenting them in the project at large if someone from musl
> provides patches (I discussed this some time ago, but can't remember with who).
> Rich, if you would like to discuss about this, we can have some chat.
>
> >
> > For musl and other libcs, I think the man pages already document some
> > instances where their behavior diverges from glibc.
>
> As said, for musl, I'd document them officially, if there's anyone interested
> enough to send patches.
>
> For other libcs, we have to decide if they're important enough, and probably
> decide on a case-by-case basis.
>
> Michael tried to have some decent coverage of non-GNU/Linux systems, both in the
> man-pages and in his TLPI book.  It's a useful thing.  So much that sometimes
> you don't even need to read other systems' pages at all to know how portable is
> a GNU/Linux function.

I know. But not sure how much Linux docs should cover about other
OSes, which are also constantly changing (and have their own fine
docs).

As always, just my 2c,
Stefan.

>
> So, I'd like to get opinions on interest about documenting details about:
>
> - newlib (I never heard about it before; is it a widespread thing? do you think
> it's useful?)
> - Apple Libc (I still don't like it, but I must admit that it's relevant, and
> being open source, it's more palatable)
> - bionic (does anyone know if that's useful at all for anyone at all?)
>
> Support for those wouldn't go as far as musl or glibc.  For now it would only be
> for memmem(3).
>
>
> Cheers,
>
> Alex
>
> --
> <http://www.alejandro-colomar.es/>

[musl] Re: [PATCH] memmem.3: Added list of known systems where this is available

[Alejandro Colomar]

[-- Attachment #1.1: Type: text/plain, Size: 2889 bytes --]

Hi Stefan,

On 11/24/22 19:57, Stefan Puiu wrote:
> Aah OK, so memmem is non-standard, there is no standard to point to.
> Do other OSes provide this kind of info? I don't see anything about
> portability in the OpenBSD man page (https://man.openbsd.org/memmem),
> only "memmem() is a GNU extension". The FreeBSD man page
> (https://www.freebsd.org/cgi/man.cgi?query=memmem&manpath=FreeBSD+13.1-RELEASE+and+Ports)
> does mention Linux, but only to mention that memmem was broken in old
> versions of Linux libc (nothing about current Linux); it also has the
> 'GNU extension' mention. Interestingly, I don't see the mention of the
> function being a GNU extension in the Linux version.

The Linux man-pages have a little bit more of tradition talking about other 
systems in that regard.

> 
> Have you checked, are there many such functions?

No, I didn't.

> Do you plan to add
> this info for all of them?

No.  I don't have the time to add this information for all such functions.
But if someone sends a patch, _and_ it only mentions free software libraries 
(i.e., macOS no, but Apple libc yes), and the function is not yet in POSIX, and 
the function is something very useful that might be ported to other systems if 
many programs use it,
then I will accept such a path.

memmem(3) is an example of such a function, so such patches for it are welcome.

> 
>>
>> Musl libc is definitely a first-class citizen these days in Linux distributions.
>>    I would start documenting them in the project at large if someone from musl
>> provides patches (I discussed this some time ago, but can't remember with who).
>> Rich, if you would like to discuss about this, we can have some chat.
>>
>>>
>>> For musl and other libcs, I think the man pages already document some
>>> instances where their behavior diverges from glibc.
>>
>> As said, for musl, I'd document them officially, if there's anyone interested
>> enough to send patches.
>>
>> For other libcs, we have to decide if they're important enough, and probably
>> decide on a case-by-case basis.
>>
>> Michael tried to have some decent coverage of non-GNU/Linux systems, both in the
>> man-pages and in his TLPI book.  It's a useful thing.  So much that sometimes
>> you don't even need to read other systems' pages at all to know how portable is
>> a GNU/Linux function.
> 
> I know. But not sure how much Linux docs should cover about other
> OSes, which are also constantly changing (and have their own fine
> docs).

Support for a function is something that usually is only added.  I'd be careful 
of only adding such info for functions that I expect to grow in popularity, 
maybe even they're added to a future POSIX (I expect this can happen with memmem(3))

> 
> As always, just my 2c,
> Stefan.

Cheers,

Alex

-- 
<http://www.alejandro-colomar.es/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]