[musl] [PATCH] printf: Fix int overflow in rounding calculation

[Peter Ammon <corydoras@ridiculousfish.com>]

In printf float formatting, a calculation of digits to round may trigger signed int overflow, if the precision is large and the exponent is large and negative. For example, `printf("%.*g", INT_MAX, 1e-100)` will reproduce it.

The overflow case corresponds to a huge number of digits after the decimal, which means no rounding is required, so I opted to saturate j at INT_MAX. Using a wider type is an alternative.

Note that p is non-negative, so underflow is impossible.

---
src/stdio/vfprintf.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/stdio/vfprintf.c b/src/stdio/vfprintf.c
index 497c5e19..eb4c755a 100644
--- a/src/stdio/vfprintf.c
+++ b/src/stdio/vfprintf.c
@@ -307,7 +307,8 @@ static int fmt_fp(FILE *f, long double y, int w, int p, int fl, int t)
	else e=0;

	/* Perform rounding: j is precision after the radix (possibly neg) */
-	j = p - ((t|32)!='f')*e - ((t|32)=='g' && p);
+	i = ((t|32)!='f')*e + ((t|32)=='g' && p);
+	j = (i < 0 && p > INT_MAX+i) ? INT_MAX : p-i;
	if (j < 9*(z-r-1)) {
		uint32_t x;
		/* We avoid C's broken division of negative numbers */
--
2.39.2