Re: [musl] Re: [BUG] ioctl: overflow in implicit constant conversion

[Jeffrey Walton <noloader@gmail.com>]

On Tue, Feb 21, 2023 at 11:05 AM Markus Wichmann <nullplan@gmx.net> wrote:
> On Mon, Feb 20, 2023 at 09:26:05PM -0800, Ralph Little wrote:
> > I have been picking up some old pending issues related to the SANE project.
> > One of our CI builds is on Alpine and it is generating warnings for ioctl()
> > calls from the musl library:
> >
> > |error: overflow in conversion from 'long unsigned int' to 'int' changes
> > value from '2147577985' to '-2147389311' [-Werror=overflow]
> > |
> > ||ioctl (fd, PPRSTATUS, &status);
> >
> > ||I see that Olaf Meeuwissen raised this issue a couple of years ago and the
> > discussion petered out somewhat and I don't believe that the issue was ever
> > really resolved:
> >
> > https://www.openwall.com/lists/musl/2020/01/20/2
> >
> > Is there any possibility that this could be addressed in the near future?
> > I see that Alpine have closed their issue and are not interested in patching
> > their downstream musl:
> >
> > https://gitlab.alpinelinux.org/alpine/aports/-/issues/7580#note_287168
> [...]
> So, I had a look at it. As far as I can tell, the issue is that musl
> declares ioctl()'s second argument to be an int. Together with the other
> defintions, this means that any _IOC_READ constant will overflow and
> generate those warnings. Also, this is technically undefined behavior,
> as value bits are shifted into the sign bit of a signed integer.
>
> Linux itself defines the ioctl syscall to have a second argument of type
> unsigned int.
>
> So this issue could be resolved by simply making the second argument of
> the ioctl() function unsigned. Does that create ABI issues? To my
> knowledge, all ABIs pass ints and unsigned ints the same way. Even if on
> some 64-bit arch there was a sign extension at the top, only the low
> 32 bits are defined.

In this case, I think the best course of action is to cast a,b,c to
unsigned, then perform the shifts, and finally cast back to int. That
is what the C standard requires. And it should not mess with the ABI.

If the code remains undefined, then it is subject to removal by the
compiler. The casts, while ugly, keep the code in well defined
territory. Also, if anyone ever performs testing with
-fsantize=undefined, then the code will trigger real findings that
could keep the code from passing through a security gate (for those
folks who have to work in that kind of environment).

I've had to work bug reports that were a result of the missing casts
during shifts and rotates. It is not fun. I was able to track all of
them down with -fsantize=undefined .

Jeff