Initial draft of musl documentation/manual

Initial draft of musl documentation/manual

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

Hi all,

The big agenda item for the next release is drafting the official
documentation for musl. While this task still has a long way to go,
and much of it is still a skeleton, there's also a decent amount of
actual content, and it's helping me come up with ideas for other
content that should be included.

Comments on the draft are welcome, especially on the matter of
additional topics that readers might expect to find which I have
overlooked.. For reference, it's based on the outline in the "New docs
outline" thread on the mailing list, from 2012-11-15.

One more thing: the format of the document is presently Markdown
(http://daringfireball.net/projects/markdown/). I may switch to
something else in the future, but Markdown tends to be my favorite
from a standpoint of being fully readable as plain text and also
producing decent formatted output in HTML (and other formats, via
conversion from HTML). Comments on this bikeshed topic are also
welcome, I suppose, unless it gets out of hand...

Rich

[-- Attachment #2: manual4.txt --]
[-- Type: text/plain, Size: 5576 bytes --]

musl
====

Part I - About musl
-------------------

### Introduction and Scope

musl is an implementation of the userspace portion of the standard
library functionality described in the ISO C and POSIX standards, plus
common extensions. It is both a component for use in Linux-based
operating systems and a tool for building application binaries
deployable on a wide range of Linux-based systems and non-Linux
systems which can provide a compatible syscall API layer.

### Conformance statements

The interfaces in musl are modeled upon and intended to conform to the
requirements of the ISO C99 standard (ISO/IEC 9899-1999), including
Annex F, and POSIX 2008 / Single Unix Standard Version 4, with all
current technical corrigenda applied. However, musl has not been
certified by any standards body, and no guarantee of conformance is
made by the copyright holders or any other party with an interest in
musl.

Moreover, since musl provides only the userspace portion of the
standard system interfaces, conformance to the requirements of POSIX
depends in part on the behavior of the underlying kernel. Linux 2.6.39
or later is believed to be sufficient; earlier versions in the 2.6
series will work, but with varying degrees of non-conformance,
particularly in the area of signal handling behavior and close-on-exec
race conditions.

Likewise, conformance to the requirements of ISO C, and especially
Annex F (IEEE floating point semantics), depends in part on both the
compiler used to build musl and the compiler used when building
applications against musl. At this time there is no known fully
conforming compiler.

### Supported Systems

To be written.

### Build and Installation

To be written.

### Installed Components

#### Dynamic linking runtime

`$(syslibdir)/ld-musl-$(ARCH).so.1` provides the dynamic linker, or
"program interpreter", for dynamically linked ELF programs using musl.
The absolute pathname to this file must be stored in all such
programs. The build and installation system provided with musl sets it
up as a symbolic link to `$(libdir)/libc.so`, but system integrators
may choose to make it available in whichever ways they find suitable.

#### Development environment

Header files for use by the C compiler are installed in
`$(includedir)`. The standard headers are fully self-contained, and do
not make use of kernel-provided or compiler-provided headers or
otherwise require such headers to be present.

The file `libc.a` installed in `$(libdir)` provides the entire
standard library implementation for static linking. The file `libc.so`
provides the linker with access to the standard library's symbols for
use at link-time in producing dynamic-linked binaries. It is not
searched at runtime; the standard library is resolved as part of the
program interpreter at `$(syslibdir)/ld-musl-$(ARCH).so.1`.

Additional files `libm.a`, `librt.a`, `libpthread.a`, `libcrypt.a`,
`libutil.a`, `libxnet.a`, `libresolv.a`, and `libdl.a` are provided in
`$(libdir)` as empty library archives. They contain no code, but are
present to satisfy the POSIX requirement that options of the form
`-lm`, `-lpthread`, etc. be accepted by the `c99` compiler.

Several bare object files are also included in `$(libdir)`: `crt1.o`
and `Scrt1.o` are the normal and position-independent versions,
respectively, of the entry point code linked into every program.
`crti.o` and `crtn.o`, also linked into every program and into shared
libraries, provide support for legacy means by which the compiler can
arrange for global constructors and destructors to be executed. It is
possible to setup a legacy-free compiler toolchain that does not need
the `crti.o` and `crtn.o` files if desired.

#### Compiler wrapper

To be written.

### Filesystem Layout Dependencies

musl aims to avoid imposing filesystem policy; however, the following
minimal set of filesystems dependencies must be met in order for
programs using musl to function correctly:

* `/dev/null` - required by POSIX

* `/tmp` - required by POSIX to exist as a directory, and used by
various temporary file creation functions.

* `/dev/shm` - must be a directory, and should have permissions 01777.
If absent, POSIX shared memory and named semaphore interfaces will
fail; programs not using these features will be unaffected.

* `/proc` - must be a mount point for Linux procfs or a symlink to
such. Several functions such as realpath, fexecve, and a number of the
"at" functions added in POSIX 2008 need access to /proc to function
correctly.

* `/etc/resolv.conf` - needed to provide addresses of nameservers to
be used for DNS lookups, unless a working nameserver is available on
the loopback address.

* `../etc/ld-musl-$(ARCH).path`, taken relative to the location of the
"program interpreter" specified in the program's headers - if present,
this will be processed as a text file containing the shared library
search path, with components delimited by newlines or colons. If
absent, a default path of `"/lib:/usr/local/lib:/usr/lib"` will be
used. Not used by static-linked programs.


Part II - Usage
---------------

To be written. This part of the manual will deal with documenting
implementation-defined behavior and further behaviors that are not
required to be documented but for which musl makes additional
guarantees.



Part III - Implementation
-------------------------

To be written. This part of the manual will document the
implementation of musl, including matters such as source tree layout,
built system, algorithms used, musl-internal APIs, coding style, and
information on porting.

Re: Initial draft of musl documentation/manual

[Luca Barbato]

On 01/09/13 06:41, Rich Felker wrote:
> One more thing: the format of the document is presently Markdown
> (http://daringfireball.net/projects/markdown/). I may switch to
> something else in the future, but Markdown tends to be my favorite
> from a standpoint of being fully readable as plain text and also
> producing decent formatted output in HTML (and other formats, via
> conversion from HTML). Comments on this bikeshed topic are also
> welcome, I suppose, unless it gets out of hand...

Markdown to man converters are the best available (e.g. kramdown-man)
and markdown is usually easy to grok.

lu

Re: Initial draft of musl documentation/manual

[Vasily Kulikov]

On Sun, Sep 01, 2013 at 00:41 -0400, Rich Felker wrote:
> ### Filesystem Layout Dependencies
> 
> musl aims to avoid imposing filesystem policy; however, the following
> minimal set of filesystems dependencies must be met in order for
> programs using musl to function correctly:
> 
> * `/dev/null` - required by POSIX
> 
> * `/tmp` - required by POSIX to exist as a directory, and used by
> various temporary file creation functions.
> 
> * `/dev/shm` - must be a directory, and should have permissions 01777.
> If absent, POSIX shared memory and named semaphore interfaces will
> fail; programs not using these features will be unaffected.

If you name such files which are important in case of specific API usage
and are not needed if this API is not used, why not enumerate other
files like:

/dev/ptmx
/dev/pts/*
/dev/tty

Or, even better, compile two lists.  The first is a list of files which
are needed for sure to be able just to start executable compiled against
musl, and a list of should-have files which are needed for several
non-critical APIs like POSIX shared memory stuff.

Thanks,

-- 
Vasily Kulikov
http://www.openwall.com - bringing security into open computing environments

Re: Initial draft of musl documentation/manual

[Rich Felker]

On Sun, Sep 01, 2013 at 05:45:48PM +0400, Vasily Kulikov wrote:
> On Sun, Sep 01, 2013 at 00:41 -0400, Rich Felker wrote:
> > ### Filesystem Layout Dependencies
> > 
> > musl aims to avoid imposing filesystem policy; however, the following
> > minimal set of filesystems dependencies must be met in order for
> > programs using musl to function correctly:
> > 
> > * `/dev/null` - required by POSIX
> > 
> > * `/tmp` - required by POSIX to exist as a directory, and used by
> > various temporary file creation functions.
> > 
> > * `/dev/shm` - must be a directory, and should have permissions 01777.
> > If absent, POSIX shared memory and named semaphore interfaces will
> > fail; programs not using these features will be unaffected.
> 
> If you name such files which are important in case of specific API usage
> and are not needed if this API is not used, why not enumerate other
> files like:
> 
> /dev/ptmx
> /dev/pts/*
> /dev/tty

Indeed, these are unintentional omissions. I will add them.

> Or, even better, compile two lists.  The first is a list of files which
> are needed for sure to be able just to start executable compiled against
> musl,

That list would be empty except for the dynamic linker in the case of
dynamic-linked programs.

> and a list of should-have files which are needed for several
> non-critical APIs like POSIX shared memory stuff.

The problem is that basically all of the files/devices are only needed
for certain particular features, and different people will have
different ideas of what's important. For non-networked use,
DNS/resolv.conf is unimportant. For many programs, /proc is not
important, but I'd rather not document exactly what it's needed for,
since it may be needed for more things in the future.

Perhaps the best structure is:

Group A: These are considered "required" in the sense that they're
used internally and it's intentionally undocumented where they're
used, as they might be used increasingly in the future.

Group B: These are considered "optional" in the sense that they're
used by specific interfaces and we document which interfaces need
them, so you can omit them if you won't be using the associated
features.

Rich

Re: Initial draft of musl documentation/manual

[Rich Felker]

On Sun, Sep 01, 2013 at 11:43:36AM +0200, Luca Barbato wrote:
> On 01/09/13 06:41, Rich Felker wrote:
> > One more thing: the format of the document is presently Markdown
> > (http://daringfireball.net/projects/markdown/). I may switch to
> > something else in the future, but Markdown tends to be my favorite
> > from a standpoint of being fully readable as plain text and also
> > producing decent formatted output in HTML (and other formats, via
> > conversion from HTML). Comments on this bikeshed topic are also
> > welcome, I suppose, unless it gets out of hand...
> 
> Markdown to man converters are the best available (e.g. kramdown-man)
> and markdown is usually easy to grok.

Excellent. I'm not sure man format would be terribly useful for the
musl documentation I'm working on now, but it's good to know for
future reference, especially if we do want to provide a few man pages
that could be installed for musl.

For the big doc, HTML and PDF are probably the more interesting output
formats. The big thing I don't see how to do easily is produce a nice
table of contents or index.

Rich

Re: Initial draft of musl documentation/manual

[Luca Barbato]

On 01/09/13 18:57, Rich Felker wrote:
> For the big doc, HTML and PDF are probably the more interesting output
> formats. The big thing I don't see how to do easily is produce a nice
> table of contents or index.

http://kramdown.rubyforge.org/converter/html.html#toc

Surprisingly easy =)

lu

Re: Initial draft of musl documentation/manual

[Christian Wiese]

Hi,

On Sun, 1 Sep 2013 00:41:06 -0400, Rich Felker <dalias@aerifal.cx> wrote:
> 
> One more thing: the format of the document is presently Markdown
> (http://daringfireball.net/projects/markdown/). I may switch to
> something else in the future, but Markdown tends to be my favorite
> from a standpoint of being fully readable as plain text and also
> producing decent formatted output in HTML (and other formats, via
> conversion from HTML). Comments on this bikeshed topic are also
> welcome, I suppose, unless it gets out of hand...
> 
in my opinion the "being fully readable as plain text" is a good thing
and should be preserved. I don't know much about markdown and its
capabilities, and how easily or non-easily it is done to generate
high-quality documents in different output formats from it.

Asciidoc [1] for example fulfils this criteria, by being fully readable
as plain text, and providing the capability to generate different kinds
of output formats in a high quality.

"The asciidoc(1) command translates AsciiDoc files to HTML, XHTML and
DocBook markups. DocBook can be post-processed to presentation formats
such as HTML, PDF, EPUB, DVI, LaTeX, roff, and Postscript using readily
available Open Source tools."

You can take a look at the asciidoc source file of the index page the
asciddoc website is rendered from to get a slight overview how it looks
like.


Cheers,
Chris


[1] http://asciidoc.org/
[2] http://asciidoc.org/index.txt

Re: Initial draft of musl documentation/manual

[Ivan Kanakarakis]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

On 2 September 2013 03:08, Luca Barbato <lu_zero@gentoo.org> wrote:

> On 01/09/13 18:57, Rich Felker wrote:
> > For the big doc, HTML and PDF are probably the more interesting output
>


​HTML is the usual output of markdown parsers.
AFAIK, GitHub uses redcarpet(ruby)[0] on top of sundown(C)[1]​



>  > formats. The big thing I don't see how to do easily is produce a nice
> > table of contents or index.
>
> http://kramdown.rubyforge.org/converter/html.html#toc
>
> Surprisingly easy =)
>
> lu
>


Kramdown is a superset of markdown​​. ​You cannot have an
automatically generated toc with the original/standard markdown.
Asciidoc[2] has that by default (among lots of others things).
MultiMarkdown(C)[3] also adds lots of features[4] to markdown,
getting markdown a little closer to Asciidoc's featureset.​


  [0]: https://github.com/vmg/redcarpet
  [1]: https://github.com/vmg/sundown
  [2]: http://asciidoc.org/
  [3]: https://github.com/fletcher/MultiMarkdown-4
​  [4]:
https://rawgithub.com/fletcher/human-markdown-reference/master/index.html

​


-- 
*Ivan c00kiemon5ter Kanakarakis*  >:3

[-- Attachment #2: Type: text/html, Size: 3810 bytes --]

Second draft of musl documentation/manual

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

Here is the updated manual draft, with more (most) of Part I filled
in. Parts II and III remain empty. I'm hoping to have some progress on
the remaining parts before releasing 0.9.14, and to include the
documentation in the release.

Rich

[-- Attachment #2: manual5.txt --]
[-- Type: text/plain, Size: 11186 bytes --]

musl
====

Part I - About musl
-------------------

### Introduction and Scope

musl is an implementation of the userspace portion of the standard
library functionality described in the ISO C and POSIX standards, plus
common extensions. It is both a component for use in Linux-based
operating systems and a tool for building application binaries
deployable on a wide range of Linux-based systems and non-Linux
systems which can provide a compatible syscall API layer.

### Conformance

The interfaces in musl are modeled upon and intended to conform to the
requirements of the ISO C99 standard (ISO/IEC 9899-1999), including
Annex F, and POSIX 2008 / Single Unix Standard Version 4, with all
current technical corrigenda applied. However, musl has not been
certified by any standards body, and no guarantee of conformance is
made by the copyright holders or any other party with an interest in
musl.

Moreover, since musl provides only the userspace portion of the
standard system interfaces, conformance to the requirements of POSIX
depends in part on the behavior of the underlying kernel. Linux 2.6.39
or later is believed to be sufficient; earlier versions in the 2.6
series will work, but with varying degrees of non-conformance,
particularly in the area of signal handling behavior and close-on-exec
race conditions.

Likewise, conformance to the requirements of ISO C, and especially
Annex F (IEEE floating point semantics), depends in part on both the
compiler used to build musl and the compiler used when building
applications against musl. At this time there is no known fully
conforming compiler.


### Supported Targets

* i386
    * Requires support or kernel emulation of `cmpxchg` instruction,
      introduced on the 80486
* x86_64
* ARM
    * EABI, standard or hard-float VFP variant
    * Little-endian default; big-endian variants also supported
    * Compiler toolchains only support armv4t and later
* MIPS
    * ABI is o32
    * Big-endian default; little-endian variants also supported
    * MIPS2 or later, or kernel emulation of ll/sc (standard in Linux)
      is required
    * FPU or kernel float emulation (standard in Linux but disabled on
      some OpenWRT builds) is required
* PowerPC
    * Only 32-bit is supported
    * Compiler toolchain must provide 64-bit long double, not IBM
      double-double or IEEE quad
    * For dynamic linking, compiler toolchain must be configured for
      "secure PLT" variant
* Microblaze
    * Big-endian default; little-endian variants also supported
    * Soft-float


### Build and Installation

The build system for musl uses the well-known `./configure` idiom.
musl's configure script is not based on GNU autoconf, but is intended
to closely match the configure API documented in the GNU Coding
Standards. Alternatively, the provided template for `config.mak` may
be edited by hand in place of running `./configure`.

#### Prerequisites

The only build-time prerequisites for musl are the standard POSIX
shell and utilities, GNU Make (version 3.81 or later) and a
freestanding C99 compiler toolchain targeting the desired instruction
set architecture and ABI, with support for gcc-style inline assembly,
weak aliases, and stand-alone assembly source files.

The system used to build musl does not need to be Linux-based, nor do
the Linux kernel headers need to be available.

If support for dynamic linking is desired, some further requriements
are placed on the compiler and linker. In particular, the linker must
support the `-Bsymbolic-functions` option, and the compiler must not
generate gratuitous GOT relocations where GOT-relative or PC-relative
addressing could be used instead.

#### Build options

Running `./configure --help` from the top-level source directory will
print usage information for configure. In most cases, the only options
which should be needed are:

* `--prefix`, used to control where musl will be installed. The prefix
  for musl defaults to `/usr/local/musl` rather than `/usr/local` to
  avoid breaking an existing non-musl environment on the host. If musl
  will be used as the primary system libc, prefix should usually be
  set to `/usr` or `/`.

* `--syslibdir`, used to specify the location at which the dynamic
  linker should be installed and found at runtime. The default of
  `/lib` should only be overridden when installing in `/lib` is
  impossible, since the pathname of the dynamic linker is stored in
  all dynamic-linked executables, and executables using non-standard
  paths for the dynamic linker may be difficult to deploy on other
  systems.

Both `--prefix` and `--syslibdir` should reflect the final runtime
location where musl will be installed. If musl should be installed to
a different location to prepare a package file or new target system
image, the `DESTDIR` variable can be set when running `make install`.
In this case, `DESTDIR` will be prepended to all installation paths,
but will not be saved anywhere in the files installed.

Other build options of interest are:

* `CC=...`, to choose a non-default compiler.

* `CFLAGS=...`, to pass custom options to the compiler.

* `--disable-shared`, to disable building shared `libc.so` if it will
  not be needed. This cuts the build time in half.

* `--disable-static`, to disable building `libc.a`. Other (empty) `.a`
  files are still built. This also cuts the build time in half.

* `--enable-optimize=`*list*, where *list* is a comma-separated list
  of components (subdirectories of `src`, or glob patterns) which will
  be optimized at `-O3` rather than the default optimization level
  `-Os`. Manually specifying an optimization level in the provided
  `CFLAGS`, or using `--enable-debug` or `--disable-optimize`, will
  turn off default optimizations.

* `--enable-warnings`, to turn on the recommended set of GCC warning
  options with which musl is intended to compile warning-free.

* `--enable-debug`, to turn on debugging. Adding `-g` to `CFLAGS`
  manually also works. In the future, `--enable-debug` may also enable
  additional debugging features at the source level.

See `./configure --help` for additional options.

#### Compiling and Installing

After running configure, run `make` to compile and `make install` to
install. If desired, `make install` can be invoked directly without
first running `make`, but it may be desirable to do these as separate
steps if eleveated privileges are needed to install to the final
destination. musl's makefile is fully declarative and non-recursive,
and may be arbitrarily parallelized with the `-j` option.

#### After Installation

If installing for the first time and using dynamic linking, it may be
necessary to create a path file for the dynamic linker. See
`../etc/ld-musl-$(ARCH).path` in the next section of the manual.




### Installed Components

In the following, `$(syslibdir)`, `$(includedir)`, and `$(libdir)`
refer to the paths chosen at build time (by default, `/lib`,
`$(prefix)/include`, and `$(prefix)/lib`, respectively) and `$(ARCH)`
refers to the *full* name for the target CPU architecture/ABI,
including the "subarch" component.

#### Dynamic linking runtime

`$(syslibdir)/ld-musl-$(ARCH).so.1` provides the dynamic linker, or
"program interpreter", for dynamically linked ELF programs using musl.
The absolute pathname to this file must be stored in all such
programs. The build and installation system provided with musl sets it
up as a symbolic link to `$(libdir)/libc.so`, but system integrators
may choose to make it available in whichever ways they find suitable.

#### Development environment

Header files for use by the C compiler are installed in
`$(includedir)`. The standard headers are fully self-contained, and do
not make use of kernel-provided or compiler-provided headers or
otherwise require such headers to be present.

The file `libc.a` installed in `$(libdir)` provides the entire
standard library implementation for static linking. The file `libc.so`
provides the linker with access to the standard library's symbols for
use at link-time in producing dynamic-linked binaries. It is not
searched at runtime; the standard library is resolved as part of the
program interpreter at `$(syslibdir)/ld-musl-$(ARCH).so.1`.

Additional files `libm.a`, `librt.a`, `libpthread.a`, `libcrypt.a`,
`libutil.a`, `libxnet.a`, `libresolv.a`, and `libdl.a` are provided in
`$(libdir)` as empty library archives. They contain no code, but are
present to satisfy the POSIX requirement that options of the form
`-lm`, `-lpthread`, etc. be accepted by the `c99` compiler.

Several bare object files are also included in `$(libdir)`: `crt1.o`
and `Scrt1.o` are the normal and position-independent versions,
respectively, of the entry point code linked into every program.
`crti.o` and `crtn.o`, also linked into every program and into shared
libraries, provide support for legacy means by which the compiler can
arrange for global constructors and destructors to be executed. It is
possible to setup a legacy-free compiler toolchain that does not need
the `crti.o` and `crtn.o` files if desired.

#### Compiler wrapper

To be written.


### Filesystem Layout Dependencies

musl aims to avoid imposing filesystem policy; however, the following
minimal set of filesystems dependencies must be met in order for
programs using musl to function correctly:

* `/dev/null` - required by POSIX

* `/dev/tty` - required by POSIX

* `/tmp` - required by POSIX to exist as a directory, and used by
various temporary file creation functions.

* `/dev/shm` - must be a directory, and should have permissions 01777.
If absent, POSIX shared memory and named semaphore interfaces will
fail; programs not using these features will be unaffected.

* `/dev/ptmx` - must exist and be accessible for read/write in order
for pseudo-terminal opening to work.

* `/dev/pts` - must be a mounted devpts filesystem in order for
pseudo-terminal opening to work.

* `/proc` - must be a mount point for Linux procfs or a symlink to
such. Several functions such as realpath, fexecve, and a number of the
"at" functions added in POSIX 2008 need access to /proc to function
correctly.

* `/etc/resolv.conf` - needed to provide addresses of nameservers to
be used for DNS lookups, unless a working nameserver is available on
the loopback address.

* `../etc/ld-musl-$(ARCH).path`, taken relative to the location of the
"program interpreter" specified in the program's headers - if present,
this will be processed as a text file containing the shared library
search path, with components delimited by newlines or colons. If
absent, a default path of `"/lib:/usr/local/lib:/usr/lib"` will be
used. Not used by static-linked programs.


Part II - Usage
---------------

To be written. This part of the manual will deal with documenting
implementation-defined behavior and further behaviors that are not
required to be documented but for which musl makes additional
guarantees.



Part III - Implementation
-------------------------

To be written. This part of the manual will document the
implementation of musl, including matters such as source tree layout,
built system, algorithms used, musl-internal APIs, coding style, and
information on porting.

Re: Second draft of musl documentation/manual

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2013-09-05 21:12:52 -0400]:
> #### Prerequisites
> 
> The only build-time prerequisites for musl are the standard POSIX
> shell and utilities, GNU Make (version 3.81 or later) and a
> freestanding C99 compiler toolchain targeting the desired instruction
> set architecture and ABI, with support for gcc-style inline assembly,
> weak aliases, and stand-alone assembly source files.
> 

there are some optional extensions and compiler defined macros
or builtins which may worth listing somewhere
(noreturn, noalias, typeof, 1.0i,...)

> #### Dynamic linking runtime
> 
> `$(syslibdir)/ld-musl-$(ARCH).so.1` provides the dynamic linker, or
> "program interpreter", for dynamically linked ELF programs using musl.
> The absolute pathname to this file must be stored in all such
> programs. The build and installation system provided with musl sets it
> up as a symbolic link to `$(libdir)/libc.so`, but system integrators
> may choose to make it available in whichever ways they find suitable.
> 

in the usage section:
./libc.so pathname args
./ldd pathname

> #### Development environment
> 
> Header files for use by the C compiler are installed in
> `$(includedir)`. The standard headers are fully self-contained, and do
> not make use of kernel-provided or compiler-provided headers or
> otherwise require such headers to be present.

it is also worth noting that header files are not c99 (they are
intended to be c89 except for long long and possibly wchar_t literals
and should be usable with c++ as well)

> #### Compiler wrapper
> 
> To be written.
> 
> 
> ### Filesystem Layout Dependencies
> 
> musl aims to avoid imposing filesystem policy; however, the following
> minimal set of filesystems dependencies must be met in order for
> programs using musl to function correctly:
> 
> * `/dev/null` - required by POSIX
> 
> * `/dev/tty` - required by POSIX
> 
> * `/tmp` - required by POSIX to exist as a directory, and used by
> various temporary file creation functions.
> 
> * `/dev/shm` - must be a directory, and should have permissions 01777.
> If absent, POSIX shared memory and named semaphore interfaces will
> fail; programs not using these features will be unaffected.
> 
> * `/dev/ptmx` - must exist and be accessible for read/write in order
> for pseudo-terminal opening to work.
> 
> * `/dev/pts` - must be a mounted devpts filesystem in order for
> pseudo-terminal opening to work.
> 
> * `/proc` - must be a mount point for Linux procfs or a symlink to
> such. Several functions such as realpath, fexecve, and a number of the
> "at" functions added in POSIX 2008 need access to /proc to function
> correctly.
> 
> * `/etc/resolv.conf` - needed to provide addresses of nameservers to
> be used for DNS lookups, unless a working nameserver is available on
> the loopback address.
> 
> * `../etc/ld-musl-$(ARCH).path`, taken relative to the location of the
> "program interpreter" specified in the program's headers - if present,
> this will be processed as a text file containing the shared library
> search path, with components delimited by newlines or colons. If
> absent, a default path of `"/lib:/usr/local/lib:/usr/lib"` will be
> used. Not used by static-linked programs.
> 

i'd probably repeat the program interpreter here again

and i'd list all paths that may be used

/bin/sh

/etc/shadow or /etc/tcb/...
/etc/passwd
/etc/group

default zoneinfo search paths

/dev/log

/etc/services
/etc/hosts

LD_PRELOAD and environment variable dependencies should be documented
around here as well

and the security policy for setuid binaries

> Part II - Usage
> ---------------
> 
> To be written. This part of the manual will deal with documenting
> implementation-defined behavior and further behaviors that are not
> required to be documented but for which musl makes additional
> guarantees.
> 
> 
> 
> Part III - Implementation
> -------------------------
> 
> To be written. This part of the manual will document the
> implementation of musl, including matters such as source tree layout,
> built system, algorithms used, musl-internal APIs, coding style, and
> information on porting.

the documentation looks good so far

Re: Second draft of musl documentation/manual

[Rich Felker]

On Fri, Sep 06, 2013 at 04:41:21AM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@aerifal.cx> [2013-09-05 21:12:52 -0400]:
> > #### Prerequisites
> > 
> > The only build-time prerequisites for musl are the standard POSIX
> > shell and utilities, GNU Make (version 3.81 or later) and a
> > freestanding C99 compiler toolchain targeting the desired instruction
> > set architecture and ABI, with support for gcc-style inline assembly,
> > weak aliases, and stand-alone assembly source files.
> > 
> 
> there are some optional extensions and compiler defined macros
> or builtins which may worth listing somewhere
> (noreturn, noalias, typeof, 1.0i,...)

Is 1.0i needed for compiling musl? I thought CMPLX() would be used,
but I haven't checked. The others are not needed for compiling musl
(used optionally, dependent on __GNUC__), but may be needed for
compiling applications using certain headers. I'm documenting this in
Part II. So far tgmath.h and complex.h are the only examples I have
where a header _needs_ an extension to work. Do you know others I
should document?

> > #### Dynamic linking runtime
> > 
> > `$(syslibdir)/ld-musl-$(ARCH).so.1` provides the dynamic linker, or
> > "program interpreter", for dynamically linked ELF programs using musl.
> > The absolute pathname to this file must be stored in all such
> > programs. The build and installation system provided with musl sets it
> > up as a symbolic link to `$(libdir)/libc.so`, but system integrators
> > may choose to make it available in whichever ways they find suitable.
> > 
> 
> in the usage section:
> ../libc.so pathname args
> ../ldd pathname

Good idea.

> > #### Development environment
> > 
> > Header files for use by the C compiler are installed in
> > `$(includedir)`. The standard headers are fully self-contained, and do
> > not make use of kernel-provided or compiler-provided headers or
> > otherwise require such headers to be present.
> 
> it is also worth noting that header files are not c99 (they are
> intended to be c89 except for long long and possibly wchar_t literals
> and should be usable with c++ as well)

Yes, I've already written that text in Part II (in progress right
now).

> > #### Compiler wrapper
> > 
> > To be written.
> > 
> > 
> > ### Filesystem Layout Dependencies
> > 
> > musl aims to avoid imposing filesystem policy; however, the following
> > minimal set of filesystems dependencies must be met in order for
> > programs using musl to function correctly:
> > 
> > * `/dev/null` - required by POSIX
> > 
> > * `/dev/tty` - required by POSIX
> > 
> > * `/tmp` - required by POSIX to exist as a directory, and used by
> > various temporary file creation functions.
> > 
> > * `/dev/shm` - must be a directory, and should have permissions 01777.
> > If absent, POSIX shared memory and named semaphore interfaces will
> > fail; programs not using these features will be unaffected.
> > 
> > * `/dev/ptmx` - must exist and be accessible for read/write in order
> > for pseudo-terminal opening to work.
> > 
> > * `/dev/pts` - must be a mounted devpts filesystem in order for
> > pseudo-terminal opening to work.
> > 
> > * `/proc` - must be a mount point for Linux procfs or a symlink to
> > such. Several functions such as realpath, fexecve, and a number of the
> > "at" functions added in POSIX 2008 need access to /proc to function
> > correctly.
> > 
> > * `/etc/resolv.conf` - needed to provide addresses of nameservers to
> > be used for DNS lookups, unless a working nameserver is available on
> > the loopback address.
> > 
> > * `../etc/ld-musl-$(ARCH).path`, taken relative to the location of the
> > "program interpreter" specified in the program's headers - if present,
> > this will be processed as a text file containing the shared library
> > search path, with components delimited by newlines or colons. If
> > absent, a default path of `"/lib:/usr/local/lib:/usr/lib"` will be
> > used. Not used by static-linked programs.
> > 
> 
> i'd probably repeat the program interpreter here again

OK.

> and i'd list all paths that may be used
> 
> /bin/sh
> 
> /etc/shadow or /etc/tcb/...
> /etc/passwd
> /etc/group
> 
> default zoneinfo search paths
> 
> /dev/log
> 
> /etc/services
> /etc/hosts

Indeed. Thanks for finding these.

> LD_PRELOAD and environment variable dependencies should be documented
> around here as well

Perhaps these (and the above filesystem dependencies) should be in
Part II. Also, we might want to add separate parts for run-time and
development usage.

> and the security policy for setuid binaries

Good idea.

> > Part II - Usage
> > ---------------
> > 
> > To be written. This part of the manual will deal with documenting
> > implementation-defined behavior and further behaviors that are not
> > required to be documented but for which musl makes additional
> > guarantees.
> > 
> > 
> > 
> > Part III - Implementation
> > -------------------------
> > 
> > To be written. This part of the manual will document the
> > implementation of musl, including matters such as source tree layout,
> > built system, algorithms used, musl-internal APIs, coding style, and
> > information on porting.
> 
> the documentation looks good so far

Thanks!

Rich

Re: Third draft of musl documentation/manual

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 451 bytes --]

Here is the third version of the docs/manual draft, with a lot more
content filled in, some of it based on previous comments. The biggest
task left is filling in all the implementation-defined behavior. For
plain C functions, J.3.12 gives a nice checklist of things to
document, but POSIX has a lot more things which are specified as
implementation-defined for which I don't have such a nice checklist.
Ideas on how to build one would be great.

Rich

[-- Attachment #2: manual6.txt --]
[-- Type: text/plain, Size: 17927 bytes --]

musl
====

Part I - About musl
-------------------

### Introduction and Scope

musl is an implementation of the userspace portion of the standard
library functionality described in the ISO C and POSIX standards, plus
common extensions. It is both a component for use in Linux-based
operating systems and a tool for building application binaries
deployable on a wide range of Linux-based systems and non-Linux
systems which can provide a compatible syscall API layer.

### Conformance

The interfaces in musl are modeled upon and intended to conform to the
requirements of the ISO C99 standard (ISO/IEC 9899-1999), including
Annex F, and POSIX 2008 / Single Unix Standard Version 4, with all
current technical corrigenda applied. However, musl has not been
certified by any standards body, and no guarantee of conformance is
made by the copyright holders or any other party with an interest in
musl.

Moreover, since musl provides only the userspace portion of the
standard system interfaces, conformance to the requirements of POSIX
depends in part on the behavior of the underlying kernel. Linux 2.6.39
or later is believed to be sufficient; earlier versions in the 2.6
series will work, but with varying degrees of non-conformance,
particularly in the area of signal handling behavior and close-on-exec
race conditions.

Likewise, conformance to the requirements of ISO C, and especially
Annex F (IEEE floating point semantics), depends in part on both the
compiler used to build musl and the compiler used when building
applications against musl. At this time there is no known fully
conforming compiler.


### Supported Targets

* i386
    * Requires support or kernel emulation of `cmpxchg` instruction,
      introduced on the 80486
* x86_64
* ARM
    * EABI, standard or hard-float VFP variant
    * Little-endian default; big-endian variants also supported
    * Compiler toolchains only support armv4t and later
* MIPS
    * ABI is o32
    * Big-endian default; little-endian variants also supported
    * MIPS2 or later, or kernel emulation of ll/sc (standard in Linux)
      is required
    * FPU or kernel float emulation (standard in Linux but disabled on
      some OpenWRT builds) is required
* PowerPC
    * Only 32-bit is supported
    * Compiler toolchain must provide 64-bit long double, not IBM
      double-double or IEEE quad
    * For dynamic linking, compiler toolchain must be configured for
      "secure PLT" variant
* Microblaze
    * Big-endian default; little-endian variants also supported
    * Soft-float


### Build and Installation

The build system for musl uses the well-known `./configure` idiom.
musl's configure script is not based on GNU autoconf, but is intended
to closely match the configure API documented in the GNU Coding
Standards. Alternatively, the provided template for `config.mak` may
be edited by hand in place of running `./configure`.

#### Prerequisites

The only build-time prerequisites for musl are the standard POSIX
shell and utilities, GNU Make (version 3.81 or later) and a
freestanding C99 compiler toolchain targeting the desired instruction
set architecture and ABI, with support for gcc-style inline assembly,
weak aliases, and stand-alone assembly source files.

The system used to build musl does not need to be Linux-based, nor do
the Linux kernel headers need to be available.

If support for dynamic linking is desired, some further requriements
are placed on the compiler and linker. In particular, the linker must
support the `-Bsymbolic-functions` option, and the compiler must not
generate gratuitous GOT relocations where GOT-relative or PC-relative
addressing could be used instead.

#### Build options

Running `./configure --help` from the top-level source directory will
print usage information for configure. In most cases, the only options
which should be needed are:

* `--prefix`, used to control where musl will be installed. The prefix
  for musl defaults to `/usr/local/musl` rather than `/usr/local` to
  avoid breaking an existing non-musl environment on the host. If musl
  will be used as the primary system libc, prefix should usually be
  set to `/usr` or `/`.

* `--syslibdir`, used to specify the location at which the dynamic
  linker should be installed and found at runtime. The default of
  `/lib` should only be overridden when installing in `/lib` is
  impossible, since the pathname of the dynamic linker is stored in
  all dynamic-linked executables, and executables using non-standard
  paths for the dynamic linker may be difficult to deploy on other
  systems.

Both `--prefix` and `--syslibdir` should reflect the final runtime
location where musl will be installed. If musl should be installed to
a different location to prepare a package file or new target system
image, the `DESTDIR` variable can be set when running `make install`.
In this case, `DESTDIR` will be prepended to all installation paths,
but will not be saved anywhere in the files installed.

Other build options of interest are:

* `CC=...`, to choose a non-default compiler.

* `CFLAGS=...`, to pass custom options to the compiler.

* `--disable-shared`, to disable building shared `libc.so` if it will
  not be needed. This cuts the build time in half.

* `--disable-static`, to disable building `libc.a`. Other (empty) `.a`
  files are still built. This also cuts the build time in half.

* `--enable-optimize=`*list*, where *list* is a comma-separated list
  of components (subdirectories of `src`, or glob patterns) which will
  be optimized at `-O3` rather than the default optimization level
  `-Os`. Manually specifying an optimization level in the provided
  `CFLAGS`, or using `--enable-debug` or `--disable-optimize`, will
  turn off default optimizations.

* `--enable-warnings`, to turn on the recommended set of GCC warning
  options with which musl is intended to compile warning-free.

* `--enable-debug`, to turn on debugging. Adding `-g` to `CFLAGS`
  manually also works. In the future, `--enable-debug` may also enable
  additional debugging features at the source level.

See `./configure --help` for additional options.

#### Compiling and Installing

After running configure, run `make` to compile and `make install` to
install. If desired, `make install` can be invoked directly without
first running `make`, but it may be desirable to do these as separate
steps if eleveated privileges are needed to install to the final
destination. musl's makefile is fully declarative and non-recursive,
and may be arbitrarily parallelized with the `-j` option.

#### After Installation

If installing for the first time and using dynamic linking, it may be
necessary to create a path file for the dynamic linker. See
`../etc/ld-musl-$(ARCH).path` in the next section of the manual.




### Installed Components

In the following, `$(syslibdir)`, `$(includedir)`, and `$(libdir)`
refer to the paths chosen at build time (by default, `/lib`,
`$(prefix)/include`, and `$(prefix)/lib`, respectively) and `$(ARCH)`
refers to the *full* name for the target CPU architecture/ABI,
including the "subarch" component.

#### Dynamic linking runtime

`$(syslibdir)/ld-musl-$(ARCH).so.1` provides the dynamic linker, or
"program interpreter", for dynamically linked ELF programs using musl.
The absolute pathname to this file must be stored in all such
programs. The build and installation system provided with musl sets it
up as a symbolic link to `$(libdir)/libc.so`, but system integrators
may choose to make it available in whichever ways they find suitable.

#### Development environment

Header files for use by the C compiler are installed in
`$(includedir)`. The standard headers are fully self-contained, and do
not make use of kernel-provided or compiler-provided headers or
otherwise require such headers to be present.

The file `libc.a` installed in `$(libdir)` provides the entire
standard library implementation for static linking. The file `libc.so`
provides the linker with access to the standard library's symbols for
use at link-time in producing dynamic-linked binaries. It is not
searched at runtime; the standard library is resolved as part of the
program interpreter at `$(syslibdir)/ld-musl-$(ARCH).so.1`.

Additional files `libm.a`, `librt.a`, `libpthread.a`, `libcrypt.a`,
`libutil.a`, `libxnet.a`, `libresolv.a`, and `libdl.a` are provided in
`$(libdir)` as empty library archives. They contain no code, but are
present to satisfy the POSIX requirement that options of the form
`-lm`, `-lpthread`, etc. be accepted by the `c99` compiler.

Several bare object files are also included in `$(libdir)`: `crt1.o`
and `Scrt1.o` are the normal and position-independent versions,
respectively, of the entry point code linked into every program.
`crti.o` and `crtn.o`, also linked into every program and into shared
libraries, provide support for legacy means by which the compiler can
arrange for global constructors and destructors to be executed. It is
possible to setup a legacy-free compiler toolchain that does not need
the `crti.o` and `crtn.o` files if desired.

#### Compiler wrapper

Included with musl is a wrapper script `musl-gcc` which can be used
with an existing GCC compiler toolchain to build programs using musl.
If installed, the script itself is located at `$(bindir)/musl-gcc`,
and a supporting GCC specs file it uses is located at
`$(libdir)/musl-gcc.specs`.



Part II - Runtime Usage
-----------------------

### Filesystem Layout Dependencies

musl aims to avoid imposing filesystem policy; however, the following
minimal set of filesystems dependencies must be met in order for
programs using musl to function correctly:

* `/dev/null` - device node, required by POSIX

* `/dev/tty` - device node, required by POSIX

* `/tmp` - required by POSIX to exist as a directory, and used by
  various temporary file creation functions.

* `/bin/sh` - an executable file providing as POSIX-conforming shell

* `/proc` - must be a mount point for Linux procfs or a symlink to
  such. Several functions such as realpath, fexecve, and a number of
  the "at" functions added in POSIX 2008 need access to /proc to
  function correctly.

While some programs may operate correctly even without some or all of
the above, musl's behavior in their absence is unspecified.

### Additional Pathnames Used

* `/dev/log` - a UNIX domain socket to which the `syslog()` interface
  sends log messages. If absent or inaccessible, log messages will be
  discarded.

* `/dev/shm` - a directory; should have permissions 01777. If absent,
  POSIX shared memory and named semaphore interfaces will fail;
  programs not using these features will be unaffected.

* `/dev/ptmx` and `/dev/pts` - device node and devpts filesystem mount
  point, respectively. If absent or inaccessible, `posix_openpt()` and
  `openpty()` will fail.

* `/etc/passwd` and `/etc/group` - text files containing the user and
  group databases, mappings between names and numeric ids, and group
  membership lists, in the standard traditional format. If absent,
  user and/or group lookups will fail.

* `/etc/shadow` - text file containing shadow password hashes for some
  or all users.

* `/etc/resolv.conf` - text file providing addresses of nameservers to
  be used for DNS lookups. If absent, DNS requests will be sent to the
  loopback address and will fail unless the host has its own
  nameserver.

* `/etc/hosts` - text file mapping hostnames to IP addresses.

* `/etc/services` - text file mapping network service names to port
  numbers.

* `/usr/share/zoneinfo`, `/share/zoneinfo`, and `/etc/zoneinfo` -
  directories searched for time zone files when the `TZ` environment
  variable is set to a relative pathname.

* `../etc/ld-musl-$(ARCH).path`, taken relative to the location of the
  "program interpreter" specified in the program's headers - if
  present, this will be processed as a text file containing the shared
  library search path, with components delimited by newlines or
  colons. If absent, a default path of
  `"/lib:/usr/local/lib:/usr/lib"` will be used. Not used by
  static-linked programs.



Part III - Library Usage for Development
----------------------------------------

### Compiler Support

All public interfaces in musl, at both the header file and library
level, are intended to be mostly compatible with any C99, C11, or C++
compiler targetting the same CPU architecture and ABI musl was built
for. C89 compilers are also supported provided that they accept the
`long long` type as an extension. A few public header files do,
however, require compiler-specific extensions in order to provide the
mandated standard features:

* `complex.h` requires `1.0fi` to be accepted as a constant expression
  suitable for defining `_Complex_I`.

* `tgmath.h` requires the `__typeof__` extension.

* FIXME: is this list complete?


### System Header Files

(Add some intro text on musl's headers.)

#### Introduction to Namespace Issues

Any C program using a library, whether the standard library or a
third-party library, needs to observe a contract with the library
regarding usage of identifiers - in particular, which identifiers are
used as part of the library's public interface or header file
implementation, and which identifiers are used by the application.
Having a clear contract is especially important when the library being
used is not a single fixed implementation, but may have multiple
versions or multiple independent implementations. The canonical
example of such a library is the standard library.

ISO C reserves all identifiers which are not explicitly defined or
reserved by the standard for use by the appliction. POSIX, however,
exposes a number of additional identifiers, and popular extensions
outside of the standards define even more. In order to support
applications which are written with different expectations on which
identifiers may be used for the application's purposes, and which ones
are defined by the system, a mechanism must be provided for choosing
*which contract* will be used.

#### Introduction to Feature Test Macros

To solve this problem, POSIX introduced the concept of *feature test
macros*. These are macros which an application may define *prior to
the inclusion of any system header* (either at the source level, or
via `-D` options passed as arguments to the compiler) in order to
request a particular namespace contract. POSIX 2008 specifies two such
feature test macros:

* `_POSIX_C_SOURCE`, defined to `200809L` to request all interfaces
  defined in the POSIX base standard.

* `_XOPEN_SOURCE`, defined to `700` to request all interfaces defined
  under the XSI option in addition to POSIX base.

No requirements are placed on the namespace when neither of these
macros is defined by the application. If one or both of these macros
is defined by the application, two constraints are placed on the
system headers:

* They must define all macros and declare all functions and objects
  which the standard specifies for that header to provide.

* They must not make use of any identifier not specified or reserved
  for that header.

There is, however, an exception to the second rule: since the standard
does not define behavior when the application has defined macros whose
names are reserved for system use, implementations may specify their
own feature test macros to expose additional identifiers alongside the
standard ones.

This is what musl, and most other implementations of the standard
library, do.

#### Feature Test Macros Supported by musl

If no feature test macros are defined, musl's headers operate in
"default features" mode, exposing the equivalent of the `_BSD_SOURCE`
option below. This corresponds fairly well to what most applications
unaware of feature test macros expect, and also provides a number of
more modern features.

Otherwise, if at least one of the below-listed feature test macros is
defined, they are treated additively, starting from pure ISO C as a
base. Unless otherwise specified, musl ignores the value of the macro
and only checks whether it is defined.

* `__STRICT_ANSI__`

    Adds nothing; only suppresses the default features. This macro is
    defined automatically by GCC and other major compilers in strict
    standards-conformance modes.

* `_POSIX_C_SOURCE` (or `_POSIX_SOURCE`)

    As specified by POSIX 2008; adds POSIX base. If defined to a value
    less than `200809L`, or if the deprecated version `_POSIX_SOURCE`
    is defined at all, interfaces which were removed from the standard
    but which are still in widespread use are also exposed.

* `_XOPEN_SOURCE`

    As specified by POSIX 2008; adds all interfaces in POSIX including
    the XSI option. If defined to a value less than `700`, interfaces
    which were removed from the standard but which are still in
    widespread use are also exposed.

* `_BSD_SOURCE`

    Adds everything above, plus a number of traditional and modern
    interfaces modeled after BSD systems, or supported on current BSD
    systems based on older standards such as SVID.

* `_GNU_SOURCE` (or `_ALL_SOURCE`)

    Adds everything above, plus interfaces modelef after GNU libc
    extensions and interfaces for making use of Linux-specific
    features.

The specific extensions provided by the nonstandard feature test
macros are documented in subsequent sections of this manual.


### Implementation-Defined Behavior

To be written.


### BSD-Compatible Extensions

To be written.


### GNU-Compatible Extensions

To be written.


### Linux Extensions

To be written.


### Other Nonstandard Behaviors and Extensions

To be written.


### Quality of Implementation Guarantees

To be written.



Part IV - Implementation
------------------------

To be written. This part of the manual will document the
implementation of musl, including matters such as source tree layout,
built system, algorithms used, musl-internal APIs, coding style, and
information on porting.

Re: Third draft of musl documentation/manual

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2013-09-06 00:20:43 -0400]:
> document, but POSIX has a lot more things which are specified as
> implementation-defined for which I don't have such a nice checklist.
> Ideas on how to build one would be great.

i would download the htmls from

http://pubs.opengroup.org/onlinepubs/9699919799/download/index.html

and then

grep -r --color=always -C2 -i implementation-defined basedefs/ functions/ |less -R

> ### Supported Targets
> 
> * i386
>     * Requires support or kernel emulation of `cmpxchg` instruction,
>       introduced on the 80486

maybe note that both i387 and sse fpus are supported

> #### Compiler wrapper
> 
> Included with musl is a wrapper script `musl-gcc` which can be used
> with an existing GCC compiler toolchain to build programs using musl.
> If installed, the script itself is located at `$(bindir)/musl-gcc`,
> and a supporting GCC specs file it uses is located at
> `$(libdir)/musl-gcc.specs`.
> 

note that it uses ${REALGCC:-gcc}, not the compiler configured to
build musl

> ### Filesystem Layout Dependencies
> 
> musl aims to avoid imposing filesystem policy; however, the following
> minimal set of filesystems dependencies must be met in order for
> programs using musl to function correctly:
> 
> * `/dev/null` - device node, required by POSIX
> 
> * `/dev/tty` - device node, required by POSIX
> 
> * `/tmp` - required by POSIX to exist as a directory, and used by
>   various temporary file creation functions.
> 
> * `/bin/sh` - an executable file providing as POSIX-conforming shell
> 
> * `/proc` - must be a mount point for Linux procfs or a symlink to
>   such. Several functions such as realpath, fexecve, and a number of
>   the "at" functions added in POSIX 2008 need access to /proc to
>   function correctly.
> 
> While some programs may operate correctly even without some or all of
> the above, musl's behavior in their absence is unspecified.
> 

i would still add a note about the hardcoded program interpreter path
in dynamically linked elf binaries that is required so the kernel can
execute those binaries
(referring back to the dynamic linking section about ld-musl-* and libc.so)
(arguably this is not a musl runtime fs layout dependency, you can
interpret the binary as './libc.so name', but is worth mentioning here)

> ### Additional Pathnames Used
> 
> * `/dev/log` - a UNIX domain socket to which the `syslog()` interface
>   sends log messages. If absent or inaccessible, log messages will be
>   discarded.
> 
> * `/dev/shm` - a directory; should have permissions 01777. If absent,
>   POSIX shared memory and named semaphore interfaces will fail;
>   programs not using these features will be unaffected.
> 
> * `/dev/ptmx` and `/dev/pts` - device node and devpts filesystem mount
>   point, respectively. If absent or inaccessible, `posix_openpt()` and
>   `openpty()` will fail.
> 
> * `/etc/passwd` and `/etc/group` - text files containing the user and
>   group databases, mappings between names and numeric ids, and group
>   membership lists, in the standard traditional format. If absent,
>   user and/or group lookups will fail.
> 
> * `/etc/shadow` - text file containing shadow password hashes for some
>   or all users.
> 

what about /etc/tcb/.. that can be used optionally?

> * `/etc/resolv.conf` - text file providing addresses of nameservers to
>   be used for DNS lookups. If absent, DNS requests will be sent to the
>   loopback address and will fail unless the host has its own
>   nameserver.
> 
> * `/etc/hosts` - text file mapping hostnames to IP addresses.
> 
> * `/etc/services` - text file mapping network service names to port
>   numbers.
> 
> * `/usr/share/zoneinfo`, `/share/zoneinfo`, and `/etc/zoneinfo` -
>   directories searched for time zone files when the `TZ` environment
>   variable is set to a relative pathname.
> 
> * `../etc/ld-musl-$(ARCH).path`, taken relative to the location of the
>   "program interpreter" specified in the program's headers - if
>   present, this will be processed as a text file containing the shared
>   library search path, with components delimited by newlines or
>   colons. If absent, a default path of
>   `"/lib:/usr/local/lib:/usr/lib"` will be used. Not used by
>   static-linked programs.

there is the legacy getusershell api which uses /etc/shells

mntent.h defines MOUNTED as /etc/mtab but that's not hardcoded
into the libc

> Part III - Library Usage for Development
> ----------------------------------------
> 
> ### Compiler Support
> 
> All public interfaces in musl, at both the header file and library
> level, are intended to be mostly compatible with any C99, C11, or C++
> compiler targetting the same CPU architecture and ABI musl was built
> for. C89 compilers are also supported provided that they accept the
> `long long` type as an extension. A few public header files do,

stdint.h uses L' ' wide char literal which is technically
not c89 but c94/amd1

there is another minor c89 issue (at least in math.h):
in c89 struct/union/array members cannot be initialized from a function
argument directly and gcc warns about them with -ansi -pedantic
(it also warns about long long so it's not a big deal, but i
can fix this)

> however, require compiler-specific extensions in order to provide the
> mandated standard features:
> 
> * `complex.h` requires `1.0fi` to be accepted as a constant expression
>   suitable for defining `_Complex_I`.
> 
> * `tgmath.h` requires the `__typeof__` extension.
> 
> * FIXME: is this list complete?

stdarg.h requires __builtin_va_* (+headers that use va_list)

NAN in math.h with fenv access support requires __builtin_nanf("")
(in theory if fenv access is on then 0/0.0f raises the invalid
flag at runtime except when used in static initializers)

for pedants stddef requires __builtin_offsetof

> * `_GNU_SOURCE` (or `_ALL_SOURCE`)
> 
>     Adds everything above, plus interfaces modelef after GNU libc

typo

>     extensions and interfaces for making use of Linux-specific
>     features.
> 
> The specific extensions provided by the nonstandard feature test
> macros are documented in subsequent sections of this manual.
> 

Re: Third draft of musl documentation/manual

[Rich Felker]

On Fri, Sep 06, 2013 at 05:14:56PM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@aerifal.cx> [2013-09-06 00:20:43 -0400]:
> > document, but POSIX has a lot more things which are specified as
> > implementation-defined for which I don't have such a nice checklist.
> > Ideas on how to build one would be great.
> 
> i would download the htmls from
> 
> http://pubs.opengroup.org/onlinepubs/9699919799/download/index.html
> 
> and then
> 
> grep -r --color=always -C2 -i implementation-defined basedefs/ functions/ |less -R

Probably need to make that merely "implementation-" to catch cases
broken by hyphenation, unless the html avoids using such line breaks.

> > ### Supported Targets
> > 
> > * i386
> >     * Requires support or kernel emulation of `cmpxchg` instruction,
> >       introduced on the 80486
> 
> maybe note that both i387 and sse fpus are supported

Yes but it's not part of the ABI.

> > #### Compiler wrapper
> > 
> > Included with musl is a wrapper script `musl-gcc` which can be used
> > with an existing GCC compiler toolchain to build programs using musl.
> > If installed, the script itself is located at `$(bindir)/musl-gcc`,
> > and a supporting GCC specs file it uses is located at
> > `$(libdir)/musl-gcc.specs`.
> 
> note that it uses ${REALGCC:-gcc}, not the compiler configured to
> build musl

I'll probably add an additional usage section for the wrapper.

> > ### Filesystem Layout Dependencies
> > 
> > musl aims to avoid imposing filesystem policy; however, the following
> > minimal set of filesystems dependencies must be met in order for
> > programs using musl to function correctly:
> > 
> > * `/dev/null` - device node, required by POSIX
> > 
> > * `/dev/tty` - device node, required by POSIX
> > 
> > * `/tmp` - required by POSIX to exist as a directory, and used by
> >   various temporary file creation functions.
> > 
> > * `/bin/sh` - an executable file providing as POSIX-conforming shell
> > 
> > * `/proc` - must be a mount point for Linux procfs or a symlink to
> >   such. Several functions such as realpath, fexecve, and a number of
> >   the "at" functions added in POSIX 2008 need access to /proc to
> >   function correctly.
> > 
> > While some programs may operate correctly even without some or all of
> > the above, musl's behavior in their absence is unspecified.
> > 
> 
> i would still add a note about the hardcoded program interpreter path
> in dynamically linked elf binaries that is required so the kernel can
> execute those binaries
> (referring back to the dynamic linking section about ld-musl-* and libc.so)
> (arguably this is not a musl runtime fs layout dependency, you can
> interpret the binary as './libc.so name', but is worth mentioning here)

Yes, I just forgot to add it.

> > ### Additional Pathnames Used
> > 
> > * `/dev/log` - a UNIX domain socket to which the `syslog()` interface
> >   sends log messages. If absent or inaccessible, log messages will be
> >   discarded.
> > 
> > * `/dev/shm` - a directory; should have permissions 01777. If absent,
> >   POSIX shared memory and named semaphore interfaces will fail;
> >   programs not using these features will be unaffected.
> > 
> > * `/dev/ptmx` and `/dev/pts` - device node and devpts filesystem mount
> >   point, respectively. If absent or inaccessible, `posix_openpt()` and
> >   `openpty()` will fail.
> > 
> > * `/etc/passwd` and `/etc/group` - text files containing the user and
> >   group databases, mappings between names and numeric ids, and group
> >   membership lists, in the standard traditional format. If absent,
> >   user and/or group lookups will fail.
> > 
> > * `/etc/shadow` - text file containing shadow password hashes for some
> >   or all users.
> 
> what about /etc/tcb/.. that can be used optionally?

Yes, this should still be added too.

> > * `/etc/resolv.conf` - text file providing addresses of nameservers to
> >   be used for DNS lookups. If absent, DNS requests will be sent to the
> >   loopback address and will fail unless the host has its own
> >   nameserver.
> > 
> > * `/etc/hosts` - text file mapping hostnames to IP addresses.
> > 
> > * `/etc/services` - text file mapping network service names to port
> >   numbers.
> > 
> > * `/usr/share/zoneinfo`, `/share/zoneinfo`, and `/etc/zoneinfo` -
> >   directories searched for time zone files when the `TZ` environment
> >   variable is set to a relative pathname.
> > 
> > * `../etc/ld-musl-$(ARCH).path`, taken relative to the location of the
> >   "program interpreter" specified in the program's headers - if
> >   present, this will be processed as a text file containing the shared
> >   library search path, with components delimited by newlines or
> >   colons. If absent, a default path of
> >   `"/lib:/usr/local/lib:/usr/lib"` will be used. Not used by
> >   static-linked programs.
> 
> there is the legacy getusershell api which uses /etc/shells

This could be added as well.

> mntent.h defines MOUNTED as /etc/mtab but that's not hardcoded
> into the libc

Probably doesn't need to be included. This is an app issue.

> > Part III - Library Usage for Development
> > ----------------------------------------
> > 
> > ### Compiler Support
> > 
> > All public interfaces in musl, at both the header file and library
> > level, are intended to be mostly compatible with any C99, C11, or C++
> > compiler targetting the same CPU architecture and ABI musl was built
> > for. C89 compilers are also supported provided that they accept the
> > `long long` type as an extension. A few public header files do,
> 
> stdint.h uses L' ' wide char literal which is technically
> not c89 but c94/amd1

OK, noted.

> there is another minor c89 issue (at least in math.h):
> in c89 struct/union/array members cannot be initialized from a function
> argument directly and gcc warns about them with -ansi -pedantic
> (it also warns about long long so it's not a big deal, but i
> can fix this)

Do you mean the initializers can't be non-constant-expressions? Or is
the issue more subtle?

> > however, require compiler-specific extensions in order to provide the
> > mandated standard features:
> > 
> > * `complex.h` requires `1.0fi` to be accepted as a constant expression
> >   suitable for defining `_Complex_I`.
> > 
> > * `tgmath.h` requires the `__typeof__` extension.
> > 
> > * FIXME: is this list complete?
> 
> stdarg.h requires __builtin_va_* (+headers that use va_list)

Yes. Thanks.

> NAN in math.h with fenv access support requires __builtin_nanf("")
> (in theory if fenv access is on then 0/0.0f raises the invalid
> flag at runtime except when used in static initializers)

Use of __builtin_nanf("") is conditional on __GNUC__. By default,
0.0f/0.0f is used.

> for pedants stddef requires __builtin_offsetof

This also has a non-__GNUC__ fallback, but it may fail to be a
constant expression, yielding slightly incorrect semantics.

> > * `_GNU_SOURCE` (or `_ALL_SOURCE`)
> > 
> >     Adds everything above, plus interfaces modelef after GNU libc
> 
> typo

You're the third one to report it. :-)

Rich

Re: Third draft of musl documentation/manual

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2013-09-06 11:47:30 -0400]:
> > there is another minor c89 issue (at least in math.h):
> > in c89 struct/union/array members cannot be initialized from a function
> > argument directly and gcc warns about them with -ansi -pedantic
> > (it also warns about long long so it's not a big deal, but i
> > can fix this)
> 
> Do you mean the initializers can't be non-constant-expressions? Or is
> the issue more subtle?

c89 3.5.7:

"All the expressions in an initializer for an object that has
static storage duration or in an initializer list for an object
that has aggregate or union type shall be constant expressions."

only -pedantic warns for this though (which also warns for long long
unless __extension__ is used like in the glibc headers)

> > NAN in math.h with fenv access support requires __builtin_nanf("")
> > (in theory if fenv access is on then 0/0.0f raises the invalid
> > flag at runtime except when used in static initializers)
> 
> Use of __builtin_nanf("") is conditional on __GNUC__. By default,
> 0.0f/0.0f is used.
> 
> > for pedants stddef requires __builtin_offsetof
> 
> This also has a non-__GNUC__ fallback, but it may fail to be a
> constant expression, yielding slightly incorrect semantics.
> 

my point was that the fallbacks are not strictly correct
(but yes they work well enough in practice)


> > >     Adds everything above, plus interfaces modelef after GNU libc
> > 
> > typo
> 
> You're the third one to report it. :-)

good, that means at least 3ppl read it thus far :)

Re: Third draft of musl documentation/manual

[Ivan Kanakarakis]

[-- Attachment #1: Type: text/plain, Size: 666 bytes --]

Maybe there should be a Legal/License section mentioning the release
license etc



On 6 September 2013 07:20, Rich Felker <dalias@aerifal.cx> wrote:

> Here is the third version of the docs/manual draft, with a lot more
> content filled in, some of it based on previous comments. The biggest
> task left is filling in all the implementation-defined behavior. For
> plain C functions, J.3.12 gives a nice checklist of things to
> document, but POSIX has a lot more things which are specified as
> implementation-defined for which I don't have such a nice checklist.
> Ideas on how to build one would be great.
>
> Rich
>



-- 
*Ivan c00kiemon5ter Kanakarakis*  >:3

[-- Attachment #2: Type: text/html, Size: 1305 bytes --]

AW: Third draft of musl documentation/manual

[Bortis Kevin]

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

Added a triple, cpu and ARCH table to the manual. This should help the users to guess the sugested triple and most importantly the config name under /etc/ld-musl-$(ARCH).path and of course the linker name. For architectures other than i386, x86_64 and arm, someone should look over the values.

Regards
Kevin

> -----Ursprüngliche Nachricht-----
> Von: Rich Felker [mailto:dalias@aerifal.cx]
> Gesendet: Freitag, 6. September 2013 06:21
> An: musl@lists.openwall.com
> Betreff: Re: [musl] Third draft of musl documentation/manual
> 
> Here is the third version of the docs/manual draft, with a lot more content
> filled in, some of it based on previous comments. The biggest task left is
> filling in all the implementation-defined behavior. For plain C functions,
> J.3.12 gives a nice checklist of things to document, but POSIX has a lot more
> things which are specified as implementation-defined for which I don't have
> such a nice checklist.
> Ideas on how to build one would be great.
> 
> Rich

[-- Attachment #2: manual6-arch-tables.diff --]
[-- Type: application/octet-stream, Size: 1134 bytes --]

--- manual6.txt	2013-09-09 13:36:09.695361698 +0200
+++ manual6-mod.txt	2013-09-09 13:36:32.186602487 +0200
@@ -65,6 +65,25 @@
     * Big-endian default; little-endian variants also supported
     * Soft-float
 
+#### Sugested Triple Table
+
+<Triple>			musl <inker name>
+
+<CPU>-linux-musl		ld-musl-<arch>.so.1
+arm-linux-musleabi		ld-musl-arm.so.1
+arm-linux-musleabihf		ld-musl-armhf.so.1
+
+#### CPU Table
+
+<CPU>				musl <ARCH>
+
+i386				i386
+x86_64				x86_64
+arm				arm, armhf
+mips				mips
+mipsel				mipsel
+powerpc				power
+microblaze			microblaze,microblazeel
 
 ### Build and Installation
 
@@ -161,9 +180,6 @@
 necessary to create a path file for the dynamic linker. See
 `../etc/ld-musl-$(ARCH).path` in the next section of the manual.
 
-
-
-
 ### Installed Components
 
 In the following, `$(syslibdir)`, `$(includedir)`, and `$(libdir)`
@@ -414,7 +430,7 @@
 
 * `_GNU_SOURCE` (or `_ALL_SOURCE`)
 
-    Adds everything above, plus interfaces modelef after GNU libc
+    Adds everything above, plus interfaces modeled after GNU libc
     extensions and interfaces for making use of Linux-specific
     features.
 

AW: Third draft of musl documentation/manual

[Bortis Kevin]

What happens if /etc/ld-musl-$(ARCH).path is:
  * present, but not readable?
  * present, but corrupt?

Does it always fallback to the default "/lib:/usr/local/lib:/usr/lib"?

The manual text does not describe these cases.

====================================
* `../etc/ld-musl-$(ARCH).path`, taken relative to the location of the
  "program interpreter" specified in the program's headers - if
  present, this will be processed as a text file containing the shared
  library search path, with components delimited by newlines or
  colons. If absent, a default path of
  `"/lib:/usr/local/lib:/usr/lib"` will be used. Not used by
  static-linked programs.
====================================

  Kevin

> -----Ursprüngliche Nachricht-----
> Von: Rich Felker [mailto:dalias@aerifal.cx]
> Gesendet: Freitag, 6. September 2013 06:21
> An: musl@lists.openwall.com
> Betreff: Re: [musl] Third draft of musl documentation/manual
> 
> Here is the third version of the docs/manual draft, with a lot more content
> filled in, some of it based on previous comments. The biggest task left is
> filling in all the implementation-defined behavior. For plain C functions,
> J.3.12 gives a nice checklist of things to document, but POSIX has a lot more
> things which are specified as implementation-defined for which I don't have
> such a nice checklist.
> Ideas on how to build one would be great.
> 
> Rich

Re: Third draft of musl documentation/manual

[Rich Felker]

On Mon, Sep 09, 2013 at 02:09:46PM +0200, Bortis Kevin wrote:
> What happens if /etc/ld-musl-$(ARCH).path is:
>   * present, but not readable?
>   * present, but corrupt?
> 
> Does it always fallback to the default "/lib:/usr/local/lib:/usr/lib"?

The present behavior is that it falls back on any failure to open the
file, but if the file is opened and reading subsequently fails, no
fallback is used.

The intent was to avoid situations where resource-exhaustion attacks
could cause the wrong path to be used. I suspect all fopen failures
except ENOENT (e.g. ENFILE, EMFILE, ...) should also suppress the
fallback.

> The manual text does not describe these cases.

"Present but not readable" should probably be documented and have a
defined behavior. "Present but corrupt" is hard to define. In a sense,
any file content is "valid" except possibly embedded null bytes, but
most will not result in meaningful path searches. I'm doubtful that
it's useful to document or promise any reasonable behavior when the
file contains junk.

Thanks for the comments.

Rich
x