C Annex K safe C functions

C Annex K safe C functions

[Jonny Grant]

[-- Attachment #1: Type: text/plain, Size: 133 bytes --]

Hello
Not on the list, so please cc me in replies.
Any plans to support Annex K?
Those safe functions are great, strncpy_s etc
Jonny

[-- Attachment #2: Type: text/html, Size: 272 bytes --]

Re: C Annex K safe C functions

[Szabolcs Nagy]

* Jonny Grant <jg@jguk.org> [2019-02-27 10:30:52 +0700]:
> Not on the list, so please cc me in replies.
> Any plans to support Annex K?
> Those safe functions are great, strncpy_s etc

i wonder why you think they are great,
if they are advertised anywhere as safe or
useful then that should be fixed.

annex k is so incredibly broken and bad
that there is a wg14 paper about it

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1969.htm

normally it's ok to add nonsense interfaces
for compatibility, but in this case there is
no widespread use and the api depends on global
state that causes implementation issues even
if we wanted to implement it.

Re: C Annex K safe C functions

[Rich Felker]

On Wed, Feb 27, 2019 at 10:30:52AM +0700, Jonny Grant wrote:
> Hello
> Not on the list, so please cc me in replies.
> Any plans to support Annex K?
> Those safe functions are great, strncpy_s etc

No, and there's been considerable amounts of stuff written on this in
the past. I'll try to give you a good summary later. The reply so far
by Szabolcs Nagy is a good start.

Rich

Re: C Annex K safe C functions

[Rich Felker]

On Wed, Feb 27, 2019 at 08:11:53AM -0500, Rich Felker wrote:
> On Wed, Feb 27, 2019 at 10:30:52AM +0700, Jonny Grant wrote:
> > Hello
> > Not on the list, so please cc me in replies.
> > Any plans to support Annex K?
> > Those safe functions are great, strncpy_s etc
> 
> No, and there's been considerable amounts of stuff written on this in
> the past. I'll try to give you a good summary later. The reply so far
> by Szabolcs Nagy is a good start.

Here's what comes to mind about the technical and social reasons
behind not adopting Annex K:

1. The Annex K interfaces are clearly inspired by (even named
identically to) the corresponding *_s interfaces provided by MSVC.
However, the MSVC functions don't actually conform to the
specification in Annex K, due to some subtle breakage in types (IIRC
the rsize_t stuff, possibly other things too). Thus there are actually
two slightly incompatible sets of functions with the same names. This
is generally a strong criterion for exclusion from musl, unless one of
the standards we aim to support mandates the presence of the functions
with this problem (like strerror_r).

2. The handling of "runtime constraint violation" and the ability to
customize the handler for it is highly problematic. Code using the
Annex K functions can neither rely on a particular default handling of
these errors, nor can it set the handling it wants unless it's global
initialization code for the whole application, since the runtime
constraint handler is global state that can't be set in a thread-safe
or library-safe manner. Thus, to use these functions safely, you must
both check for the erroneous conditions before making the call (since,
for one choice of handler, the call could abort your program if it
detects them), and you must check for an error return from the
function (since, for a different choice of handler, it might just
return an error, and if you don't check for that yourself, your
program could continue running in an unsafe state).

3. In order to provide any safety over the standard functions they
"replace", the Annex K functions rely on the caller to provide
accurate information about the sizes of buffers, etc. in the
additional function arguments. There's no good reason to believe that
the programmer will get these arguments right when they consistently
fail to get the arguments to the standard functions right. For
example, there's no reason to believe programmers won't just call
memcpy_s(dest, n, src, n) instead of going back to some better
authoritative source for the size of the dest buffer. In many (most?)
cases, n is already computed just as the size of the dest object, and
nothing is gained. These kinds of problems are solved *much* better by
techniques that allow the compiler or runtime to track the size of
objects and automatically insert checks, which aren't subject to human
error or misuse. The _FORTIFY_SOURCE feature, sanitizers, etc. all
provide much better protection than Annex K functions in this area.

4. The Annex K *printf_s specifications spread FUD about %n being
dangerous, ignoring that this is a poor stand-in for the general
unsafety of *passing a literal string where a format string is
expected* and for use of variable format strings. Without %n, the
impact of such bugs is somewhat reduced, but it's still possible to
get heartbleed-like info leak exploits from them. A real hardened
version of printf-family functions would take a list of argument types
and match the format string against it, and this list would be
generated automatically by the tooling (ala _FORTIFY_SOURCE).

5. Expanding on the topic of FUD/misinformation, both the introduction
of the original *_s functions, and lobbying for their inclusion in the
standard (which eventually reached the compromise of just putting them
in an Annex), was not about improving the C language or making useful
tools for programmers, but about introducing incompatibility and
fragmentation to the language/standard with the goal of undermining
it. The company that introduced it produces a product that is not
compatible with the C language as specified and does not even aim to
be, but aims to give the impression of being a C implementation (it's
mainly a C++ implementation, though likely not conforming to that
standard either). This is part of a long history, including wrong
wchar_t handling, inverting the meaning of %s and %ls with the wide
printf functions, etc. etc. etc. See also point #1 above about
incompatibility of the Annex K functions themselves. It's my position,
and I believe it's shared by many others in the musl community and C
language communities, that parties not interested in implementing or
using the standard should not try to influence its direction, and that
this kind of behavior should not be rewarded by playing along with it,
but that it should be shunned as long as doing so is practical.

Rich

Re: C Annex K safe C functions

[Jonny Grant]

Hi!

On 27/02/2019 17:50, Szabolcs Nagy wrote:
> * Jonny Grant <jg@jguk.org> [2019-02-27 10:30:52 +0700]:
>> Not on the list, so please cc me in replies.
>> Any plans to support Annex K?
>> Those safe functions are great, strncpy_s etc
> 
> i wonder why you think they are great,
> if they are advertised anywhere as safe or
> useful then that should be fixed.
> 
> annex k is so incredibly broken and bad
> that there is a wg14 paper about it
> 
> http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1969.htm
> 
> normally it's ok to add nonsense interfaces
> for compatibility, but in this case there is
> no widespread use and the api depends on global
> state that causes implementation issues even
> if we wanted to implement it.

Thanks for your reply!

Well I wouldn't disagree with experts. I should re-read that review though.

However, I was not aware that these APIs have global state? (memset_s, 
memcpy_s, memmove_s, strcpy_s, strncpy_s, strcat_s, strncat_s, strtok_s, 
memset_s, strerror_s, strerrorlen_s, strnlen_s) - do they?

strncpy_s is great, it avoids the bug in strncpy that could cause the 
buffer to not be terminated. It's better than the strlcpy BSD uses which 
truncates buffers.

BSD/OS X supports memset_s etc, but does not support 
set_constraint_handler_s

https://opensource.apple.com/source/Libc/Libc-1244.1.7/string/NetBSD/memset_s.c.auto.html

FreeBSD seems to support memset_s
https://www.freebsd.org/cgi/man.cgi?query=memset&sektion=3&apropos=0&manpath=freebsd

Oracle Solaris supports Annex K
https://docs.oracle.com/cd/E88353_01/html/E37843/strncpy-s-3c.html#scrolltoc


If issues, I'd support amending Annex K, rather than removing. It's good 
they check for NULL/nullptr, they return errno_t directly instead of the 
errno global kludge. Sticking with old APIs forever is difficult, but no 
one uses creat() anymore either.

Could I ask, does your libc follow POSIX spec to the letter? eg not 
checking pointers for NULL (where spec omits to mention checking 
pointers valid) ? eg this call which crashes glibc?

puts(NULL);

It looks like it will still SIGSEGV...

https://git.musl-libc.org/cgit/musl/tree/src/stdio/puts.c



Thanks
Jonny

Re: C Annex K safe C functions

[A. Wilcox]

[-- Attachment #1.1: Type: text/plain, Size: 873 bytes --]

On 03/03/19 22:24, Jonny Grant wrote:
> Could I ask, does your libc follow POSIX spec to the letter? eg not
> checking pointers for NULL (where spec omits to mention checking
> pointers valid) ?


2.1.1 Use and Implementation of Functions

  1. If an argument to a function has an invalid value (such as a value
outside the domain of the function, or a pointer outside the address
space of the program, or a null pointer), the behavior is undefined.


The spec never omits checking the validity of pointers unless actually
specified.  In this way, musl is definitely conformant.

There are some non-conformant bits of musl, though some of these may be
solved since the last test (which was musl 1.1.19):

https://wiki.adelielinux.org/wiki/POSIX


Best,
--arw


-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: C Annex K safe C functions

[Florian Weimer]

* Rich Felker:

> 5. Expanding on the topic of FUD/misinformation, both the introduction
> of the original *_s functions, and lobbying for their inclusion in the
> standard (which eventually reached the compromise of just putting them
> in an Annex), was not about improving the C language or making useful
> tools for programmers, but about introducing incompatibility and
> fragmentation to the language/standard with the goal of undermining
> it. The company that introduced it produces a product that is not
> compatible with the C language as specified and does not even aim to
> be, but aims to give the impression of being a C implementation (it's
> mainly a C++ implementation, though likely not conforming to that
> standard either).

Does this really reflect history?  I thought that Annex K was submitted
for standardization well after the vendor in question withdrew from the
ISO process.

> It's my position, and I believe it's shared by many others in the musl
> community and C language communities, that parties not interested in
> implementing or using the standard should not try to influence its
> direction, and that this kind of behavior should not be rewarded by
> playing along with it, but that it should be shunned as long as doing
> so is practical.

My impression is that compiler vendors and large-scale users are
generally not well-represented in the ISO process anyway.  If true, your
requirement, while looking completely reasonable, would effectively halt
evolution of the standard.

Thanks,
Florian

Re: C Annex K safe C functions

[Rich Felker]

On Mon, Mar 04, 2019 at 11:24:08AM +0700, Jonny Grant wrote:
> Hi!
> 
> On 27/02/2019 17:50, Szabolcs Nagy wrote:
> >* Jonny Grant <jg@jguk.org> [2019-02-27 10:30:52 +0700]:
> >>Not on the list, so please cc me in replies.
> >>Any plans to support Annex K?
> >>Those safe functions are great, strncpy_s etc
> >
> >i wonder why you think they are great,
> >if they are advertised anywhere as safe or
> >useful then that should be fixed.
> >
> >annex k is so incredibly broken and bad
> >that there is a wg14 paper about it
> >
> >http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1969.htm
> >
> >normally it's ok to add nonsense interfaces
> >for compatibility, but in this case there is
> >no widespread use and the api depends on global
> >state that causes implementation issues even
> >if we wanted to implement it.
> 
> Thanks for your reply!
> 
> Well I wouldn't disagree with experts. I should re-read that review though.
> 
> However, I was not aware that these APIs have global state?
> (memset_s, memcpy_s, memmove_s, strcpy_s, strncpy_s, strcat_s,
> strncat_s, strtok_s, memset_s, strerror_s, strerrorlen_s, strnlen_s)
> - do they?

Yes. The action they take when a "runtime constraint violation"
happens is determined by global state. See K.3.6.1.1 The
set_constraint_handler_s function.

> strncpy_s is great, it avoids the bug in strncpy that could cause
> the buffer to not be terminated. It's better than the strlcpy BSD
> uses which truncates buffers.

This is not a bug in strncpy. It's strncpy's purpose being completely
different from what people are wrongly using it for. The strncpy
function is for working with fixed-size, null-padded data fields,
which are not C strings, and which went out of style in the 80s -- the
sort of things often cited in the Y2K problem fiasco. For the most
part, there is no modern use for strncpy. Naming a function with a
completely different purpose after it (strncpy_s) and implying it's a
"secure version of strncpy" contributes to this misconception that's
the whole reason people are wrongly using strncpy in the first place.

> BSD/OS X supports memset_s etc, but does not support
> set_constraint_handler_s

Most BSDs also support explicit_bzero, and musl does too. Providing a
function with the same name as one that's specified by some standard
we don't intend to support, but without fully compatible semantics, is
something musl generally tries to avoid.

> If issues, I'd support amending Annex K, rather than removing. It's
> good they check for NULL/nullptr, they return errno_t directly
> instead of the errno global kludge. Sticking with old APIs forever
> is difficult, but no one uses creat() anymore either.

Returning an error code is an okay choice when it works, but if you
try to apply it everywhere, it leads to severe bugs. The classic
example is allocator functions which don't return void* but instead
take a void** argument into which to store the result. Invariably
people call them with (void**)&ptr, where ptr does not have type
void*, and this is undefined behavior. In a worst case, on an
implementation where different pointer types have different
representations and sizes, it would be a buffer overflow.

This is kinda tangential to the Annex K issue, but the point is that
trying to push for a uniform "everything returns an error code"
convention introduce concrete harm for the sake of somebody's
preferred style, and as such is a really bad idea. A better convention
if you want to avoid errno is taking an int *errcode argument, but the
arguments against errno are generally pretty weak. Most of the time,
errno is the most efficient, in terms of simplifying readability and
flow of code, solution to the problem.

> Could I ask, does your libc follow POSIX spec to the letter? eg not
> checking pointers for NULL (where spec omits to mention checking
> pointers valid) ? eg this call which crashes glibc?
> 
> puts(NULL);
> 
> It looks like it will still SIGSEGV...
> 
> https://git.musl-libc.org/cgit/musl/tree/src/stdio/puts.c

There is fundamentally no way to check a pointer for validity without
false negatives or false positives unless you have a fully memory-safe
C implementation with a runtime which tracks memory in much more
detail than even tools like ASan or valgrind do. Of course it's
possible to check for a null pointer as one special case, but then the
question is what you want to do with it. Unless you have a function
that's specified to treat null pointers specially somehow (e.g. strtol
doesn't store an end position of endptr is null), the caller has
invoked undefined behavior.

One commonly-requested behavior is to return an error. I've written
many times on why this is a bad choice, and wording based on my views
on it has even been integrated into the glibc wiki on the topic:

https://sourceware.org/glibc/wiki/Style_and_Conventions#Invalid_pointers

    "If you return an error code to a caller which has already proven
    itself buggy, the most likely result is that the caller will
    ignore the error, and bad things will happen much later down the
    line when the original cause of the error has become difficult or
    impossible to track down. Why is it reasonable to assume the
    caller will ignore the error you return? Because the caller
    already ignored the error return of malloc or fopen or some other
    library-specific allocation function which returned NULL to
    indicate an error."

Generally, musl adopts a hardening approach of aiming to crash
(terminate with a fatal signal) as early as possible, and as directly
as possible, when undefined behavior has been detected. In most cases,
just dereferencing the potentially-invalid pointer is sufficient to
achieve this without explicit code for it. In other cases, where UB
can be detected without expensive (slow, complex, or error-prone)
runtime tests, musl has an explicit call to a_crash() for this. Some
examples are pthread_join (attempting to join a detached thread), free
(double free or heap corruption), and asctime (date string doesn't fit
in the buffer).

Rich

Re: C Annex K safe C functions

[Szabolcs Nagy]

* Florian Weimer <fweimer@redhat.com> [2019-03-04 09:09:31 +0100]:
> * Rich Felker:
> > 5. Expanding on the topic of FUD/misinformation, both the introduction
> > of the original *_s functions, and lobbying for their inclusion in the
> > standard (which eventually reached the compromise of just putting them
> > in an Annex), was not about improving the C language or making useful
> > tools for programmers, but about introducing incompatibility and
> > fragmentation to the language/standard with the goal of undermining
> > it. The company that introduced it produces a product that is not
> > compatible with the C language as specified and does not even aim to
> > be, but aims to give the impression of being a C implementation (it's
> > mainly a C++ implementation, though likely not conforming to that
> > standard either).
> 
> Does this really reflect history?  I thought that Annex K was submitted
> for standardization well after the vendor in question withdrew from the
> ISO process.

well microsoft did lobby for the _s functions originally,
the msvc library team is likely responsible for the concept.
it turned into tr 24731 and eventually annex k in c11,
by that time microsoft may not have much to do with it.

one can dig through wg14 to see how it evolved.

initial work:
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1007.pdf
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1031.pdf
a tr 24731 meeting (ms is still involved):
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1129.pdf
some tr 24731 drafts:
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1088.pdf
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1146.pdf
austing group comments:
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1106.txt
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1160.pdf
responses:
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1118.htm
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1174.pdf
tr 24731 rationale:
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1173.pdf
c1x inclusion:
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1350.htm
http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1394.pdf