printf family handling of INT_MAX +1 tested on aarch64

[CM Graff <cm0graff@gmail.com>]

Hello everyone,

The C standard states that:
"The number of characters or wide characters transmitted by a formatted output
function (or written to an array, or that would have been written to an array)
is greater
than INT_MAX" is undefined behavior.

POSIX states that:

"In addition, all forms of fprintf() shall fail if:

[...]
[EOVERFLOW]
    [CX] [Option Start] The value to be returned is greater than {INT_MAX}.
[Option End]
"

Though arguments of over INT_MAX are undefined behavior it seems like some
provisions have been made in musl to handle it, and the method for handling
such appear similar in effect to that of glibc and freebsd's libc. INT_MAX + 2
appears to represent this case, however INT_MAX + 1 produces a segfault on my
aarch64 test box running debian version 9.5.

I do not have a suggested fix other than to either carefully inspect the
EOVERFLOW semantics or to mitigate the need for more complex mathematics by
using a size_t as the primary counter for the stdio family instead of an int.

This segfault was discovered when testing my own small libc
(https://github.com/hlibc/hlibc) against the various robust production grade
libc to understand more about how to properly handle EOVERFLOW and in general
the cases of INT_MAX related undefined behavior for the formatted stdio
functions as per specified in the C standard and POSIX.

I am not sure that handling this is an important case for musl, however I
thought it best to report the scenario as best I could describe it.

Here is a script and a small C program to verify this segfault on aarch64,
I apologize for not testing on other architectures but my time is limited
lately as I'm working toward my degree in mathematics.

#!/bin/sh
git clone git://git.musl-libc.org/musl
cd musl
./configure --prefix=$(pwd)/usr
make -j4 > log 2>&1
make install >> log 2>&1
./usr/bin/musl-gcc -static ../printf_overflow.c
./a.out > log2



#include <stdio.h>
#include <limits.h>
#include <errno.h>
#include <string.h>
#include <stdlib.h>
int main(void)
{
        size_t i = INT_MAX;
        ++i;
        char *s = malloc(i);
        if (!(s))
        {
                fprintf(stderr, "unable to allocate enough memory\n");
                return 1;
        }
        memset(s, 'A', i - 1);
        s[i] = 0;
        /* make sure printf is not changed to puts() by the compiler */
        int len = printf("%s", s, 1);

        if (errno == EOVERFLOW)
                fprintf(stderr, "printf set EOVERFLOW\n");
        else
                fprintf(stderr, "printf did not set EOVERFLOW\n");

        fprintf(stderr, "printf returned %d\n", len);
        return 0;
}

Thank you for your time,

Graff