Re: [musl] bug: isatty() can return wrong value

[Rich Felker <dalias@libc.org>]

On Tue, Aug 20, 2024 at 11:50:02AM +0200, Gil Pedersen wrote:
> Hi,
> 
> I found a musl related issue in systemd, where it relies on glibc specific behaviour for isatty(), which happens to work out. When using musl, it will fail in another way, causing some essential retry logic to not run.
> 
> The error occurs when the isatty() implementation calls an ioctl() that sets the EIO error. glibc will directly forward any errors, while musl remaps it to an ENOTTY error.
> 
> Neither behaviour are POSIX compliant. glibc sets an undocumented errno, while musl sets ENOTTY for something that is definitely a TTY.
> 
> It seems that the correct fix is to remap the EIO to a success (1).
> 
> See https://github.com/systemd/systemd/pull/34039 and https://sourceware.org/bugzilla/show_bug.cgi?id=32103.
> 
> I'm not on the mailing list, so please CC any replies.

What guarantee do we have that nothing other than a tty in this state
will return EIO for the ioctl? The implementation on the kernel side
suggests that, if they tried to put any other device in such a state
by replacing its ioctl function the same way, it would also give EIO
for these ioctls.

The right way to do this on the kernel side would have been to only
return EIO for supported tty ioctls when in this state, and otherwise
return ENOTTY (inappropriate ioctl for device type).

I'm hopeful there's some good fix here where we don't have to have
either of these bad behaviors, but returning true for isatty() of a
device that is absolutely not a tty is a much worse behavior than
returning false for a device that is/was a tty when it was opened but
that's been replaced by the kernel with a defunct device.

Rich