Re: [musl] Re: [BUG] ioctl: overflow in implicit constant conversion

[Rich Felker <dalias@libc.org>]

On Wed, Feb 22, 2023 at 05:23:12AM +0100, Markus Wichmann wrote:
> On Tue, Feb 21, 2023 at 10:17:31PM -0500, Rich Felker wrote:
> > Unless you're seeing something I'm not, there's no UB. The shifts take
> > place on the unsigned type, and the conversion from unsigned to signed
> > is implementation-defined, not undefined. The implementation-defined
> > definition relevant to us is modular reduction.
> >
> 
> Yeah, sorry, I had missed that the directions are defined as unsigned
> constants. That turns any shift that might have been undefined into an
> unsigned shift, where it is defined. As I said in the other mail, the
> compiler is just warning about nothing here, and it should probably not
> warn for system header files, anyway.
> 
> We may be able to silence the compiler by adding an explicit int
> conversion to the _IOC macro. But I'm not sure if we want to set a
> precedent that we will add code just to shut up overeager warnings.

I think it's reasonably arguable that the ioctl command macros should
have the right type matching the type the ioctl function takes. But
it's possible there could be places where sign extension bites you
then. As a stupid example, if someone bypassed the libc function and
tried to make the ioctl syscall directly, having the macros be of
signed type would cause the kernel to receive them with a bunch of
ones in the high bits, and possibly reject them as a result. I'm not
sure if there's any reasonable usage that would be impacted, though.

Rich