Re: Re: Linux manpages (was Re: [musl] Request for volunteers)

["Michael Kerrisk (man-pages)" <mtk.manpages@gmail.com>]

Rich,

On Tue, Jul 9, 2013 at 4:53 AM, Rich Felker <dalias@aerifal.cx> wrote:
> On Tue, Jul 09, 2013 at 02:18:21AM +0200, Michael Kerrisk wrote:
>> > However my feeling is that this would be a very big project
>> > and I'm not sure if Michael would want to go in that direction. I do
>> > think it would greatly improve the quality of Linux software
>> > development, though.
>> >
>> >> The man(2) section is rather glibc specific and makes the syscall details
>> >> rather subsidiary. I will try to send some patches if these would be
>> >> welcome.
>> >
>> > I think it's an error to have anything glibc-specific in section 2 of
>> > the manual, which should be documenting the kernel, not userspace.
>> > What would be useful in the section 2 man pages is to document where
>>
>> ("useful" to who? Few users care about the naked
>> syscall behavior.)
>
> Admittedly, part of the answer is "to me". However I can think of a
> good number of others:
>
> 1. Anyone doing pure asm programming on Linux. I think this is a
>    rather bad idea, but there are people who do it.
>
> 2. People reading strace output. (For instance, if the kernel returns
>    a bogus error code and userspace has to translate it, that's
>    relevant to someone who sees the strace output and errno value in
>    their program mismatching.)
>
> 3. Implementors of any component that uses or provides the syscall.
>    That includes not only libc, but also qemu app-level emulation, BSD
>    Linux-syscall ABI emulation, Zvi's psxcalls layer (intended to
>    eventually allow using musl as the first-ever conforming Windows
>    libc that's actually deployable, unlike cygwin), ...
>
> 4. Anyone trying to understand what libc (musl, glibc, or otherwise)
>    is doing munging the syscall inputs/results.
>
> 5. Kernel developers who want to know the actual contract their
>    interfaces are supposed to satisfy and preserve.
>
> I suspect there are others, but those are the ones that came to mind
> right off.

That's a good, comprehensive list of valid, important users of course.
What I was meaning to say was that that set of users is a small subset
of the total users of the man pages. But, in any case, the goal is
also to satisfy those users by including notes about the bare syscalls
inside the pages.

>> > the syscall is insufficient to provide POSIX semantics, which are left
>> > to userspace to provide. Such section 2 pages could then have
>> > corresponding section 3 pages that document the library behavior.
>>
>> See https://www.kernel.org/doc/man-pages/todo.html#migrate_to_kernel_source
>> I think it would be a retrograde step to split syscall pages into
>> Sections 2 and 3.
>
> Yes, that's understandable. I somewhat question why we even still have
> a "section 2" in the manual, though...

Well then, you'll be amused to hear that the discussion with the BSD
maintainers was about whether FreeBSD (and others) should simply merge
Sections 2 and 3. I can see arguments in favor of it, but they're not
(to my mind) compelling. See one piece from the thread below.

Cheers,

Michael




---------- Forwarded message ----------
From: Michael Kerrisk <mtk.manpages@gmail.com>
Date: Mon, Jun 10, 2013 at 11:14 AM
Subject: Re: Merging man page sections 2 and 3?
To: Matthew Dempsky <matthew@dempsky.org>
Cc: mckusick@mckusick.com, wiz@netbsd.org, Michael Kerrisk-manpages
<mtk.manpages@gmail.com>, schwarze@openbsd.org, Jason McIntyre
<jmc@openbsd.org>, Philip Guenther <guenther@openbsd.org>


Hello Matthew,

Thanks for including the Linux man-pages project in this discussion. I
appreciate that.

On 06/06/13 23:39, Matthew Dempsky wrote:
> Hello fellow OS man page maintainers!
>
> Within OpenBSD, I started a discussion recently about sections 2 and
> 3, and whether it makes sense to continue keeping them separate.

(Do you have a pointer to the archive of that discussion?)

> The
> biggest concern raised against merging them is being gratuitously
> different from other OSes and how it would affect Xr entries in
> third-party manuals, so I was curious in knowing how other OSes feel
> about this.
>
> My view is the distinction between "system call" and "library call" is
> very blurred nowadays,

I can certainly find some reasons to agree with you on that. I'm
assuming that the BSDs will be similar to Linux, where there are
various fuzzy cases. for example, we have system calls that have very
simple wrappers, "system calls" (e.g., getpid() where glibc actually
caches the result of the call, bypassing the system call itself in
most cases--yes, it is a sad, stupid idea, but so it is), system
calls that have rather thicker wrappers in libc, and  multiplexed
syscalls (e.g., on x86-32, one system call, socketcall() provides the
functionality of all the BSD sockets APIs--this came about because of
the history of how the APIs were added to the kernel, long ago). And,
as you noted in response to Kirk, there are cases where library functions
are more expensive than syscalls.

> and it doesn't make sense to continue putting
> them in different sections.  Certainly there's merit to documenting
> the userland<->kernel interface somehow (e.g., because it's exposed
> via syscall() and strace/ktrace and other tools), but I think that
> could be better achieved other ways.  E.g., Linux man-pages has
> syscalls.2 that lists all of the system calls and when they were
> added, and some of the man pages like stat.2 have implementation notes
> describing how glibc implements it using one of a few different actual
> system calls depending on the kernel.
>
> Do any of your respective projects have clear and unambiguous
> guidelines for which section a function should be documented in?  Do
> you think your users benefit from having function documentation split
> across two sections?

The decision for this is somewhat ad hoc on Linux. If there's
an underyling kernel interface, the documentation tends to end
up in a .2 page. But the line is fuzzy. The POSIX message queue
pages, for example, are in Section 3--even though they in part
document underlying system calls--because there are significant
parts that are in libc. On the other hand, Section 2 pages often
include documentation not just of the raw syscall interface, but
the pieces that glibc API adds on top. For example, the select.2
page includes this text:

   Linux Notes
       The Linux pselect() system call modifies its timeout  argument.
       However,  the  glibc  wrapper  function  hides this behavior by
       using a local variable for the timeout argument that is  passed
       to  the  system  call.  Thus, the glibc pselect() function does
       not modify its timeout argument; this is the behavior  required
       by POSIX.1-2001.

There are many other cases similar to that one. The following FAQ
addresses a point tangentially related to this:
https://www.kernel.org/doc/man-pages/todo.html#migrate_to_kernel_source

> Assuming not, how feasible would it be for your project to move its
> manual pages from section 2 into section 3?  Are there reasons you
> could not or would not make that change?

Even if I thought it was a good idea, the biggest impediment would simply
be the effort (for me) and churn (for third part xrefs) that such a
change would cause. Without looking too closely, it's not clear to
me that the change would be just be some simple scripting.

> Certainly there's a significant initial effort to reorganize things,
> but I expect that to be a one-time cost largely solvable by sed.
> Also, third party manual pages might have inaccurate Xr entries
> initially, but overtime I expect upstream projects to adapt and we'll
> end up with more accurate cross-references everywhere.  (And in the
> interim, it's possible to workaround this by having "man 2 foo" also
> search section 3 or something.)
>
> For what it's worth, I did a quick non-scientific scan of the ~2000
> packages installed on my OpenBSD machine, and found only ~1% of the
> man pages contained cross-references to section 2 (which also included
> hilarious mistakes like "Xr strcat 2").  I'd be very interested if
> others tried to reproduce this experiment on different package sets
> and got differing results.
>
> Anyway, if you haven't had a knee-jerk "but that's how we've always
> done it!" reaction, I'm very interested in your thoughts on the
> matter. :)

So, I don't have a knee-jerk reaction ;-). I agree that the current
situation (at least on Linux, and I guess elsewhere) is imperfect.
Things are not nearly so clear cut as the .2/.3 distinction implies.
That said, I'm inclined to leave things be, for several reasons:

* Users expect the distinction. (This is not a compelling
  argument, I agree.)
* Making the change requires time and effort, and my time is
  sadly limited.
* The syscall / lib function distinction does (as you note) need to
  be made clear somehow. The current set up does that moderately
  (but not perfectly) well on Linux.
* I see no compelling reason to make the change you suggest (you
  didn't really make the case at the start of your mail).

Best regards,

Michael