[musl] bug: isatty() can return wrong value

[musl] bug: isatty() can return wrong value

[Gil Pedersen]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

Hi,

I found a musl related issue in systemd, where it relies on glibc specific behaviour for isatty(), which happens to work out. When using musl, it will fail in another way, causing some essential retry logic to not run.

The error occurs when the isatty() implementation calls an ioctl() that sets the EIO error. glibc will directly forward any errors, while musl remaps it to an ENOTTY error.

Neither behaviour are POSIX compliant. glibc sets an undocumented errno, while musl sets ENOTTY for something that is definitely a TTY.

It seems that the correct fix is to remap the EIO to a success (1).

See https://github.com/systemd/systemd/pull/34039 and https://sourceware.org/bugzilla/show_bug.cgi?id=32103.

I'm not on the mailing list, so please CC any replies.

Regards,
Gil

[-- Attachment #2: Type: text/html, Size: 1305 bytes --]

Re: [musl] bug: isatty() can return wrong value

[Rich Felker]

On Tue, Aug 20, 2024 at 11:50:02AM +0200, Gil Pedersen wrote:
> Hi,
> 
> I found a musl related issue in systemd, where it relies on glibc specific behaviour for isatty(), which happens to work out. When using musl, it will fail in another way, causing some essential retry logic to not run.
> 
> The error occurs when the isatty() implementation calls an ioctl() that sets the EIO error. glibc will directly forward any errors, while musl remaps it to an ENOTTY error.
> 
> Neither behaviour are POSIX compliant. glibc sets an undocumented errno, while musl sets ENOTTY for something that is definitely a TTY.
> 
> It seems that the correct fix is to remap the EIO to a success (1).
> 
> See https://github.com/systemd/systemd/pull/34039 and https://sourceware.org/bugzilla/show_bug.cgi?id=32103.
> 
> I'm not on the mailing list, so please CC any replies.

What guarantee do we have that nothing other than a tty in this state
will return EIO for the ioctl? The implementation on the kernel side
suggests that, if they tried to put any other device in such a state
by replacing its ioctl function the same way, it would also give EIO
for these ioctls.

The right way to do this on the kernel side would have been to only
return EIO for supported tty ioctls when in this state, and otherwise
return ENOTTY (inappropriate ioctl for device type).

I'm hopeful there's some good fix here where we don't have to have
either of these bad behaviors, but returning true for isatty() of a
device that is absolutely not a tty is a much worse behavior than
returning false for a device that is/was a tty when it was opened but
that's been replaced by the kernel with a defunct device.

Rich

Re: [musl] bug: isatty() can return wrong value

[Thorsten Glaser]

Rich Felker dixit:

>returning true for isatty() of a
>device that is absolutely not a tty is a much worse behavior than
>returning false for a device that is/was a tty when it was opened but
>that's been replaced by the kernel with a defunct device.

As isatty consumer (mksh) I agree.

bye,
//mirabilos
-- 
Gestern Nacht ist mein IRC-Netzwerk explodiert. Ich hatte nicht damit
gerechnet, darum bin ich blutverschmiert… wer konnte ahnen, daß SIE so
reagier’n… gestern Nacht ist mein IRC-Netzwerk explodiert~~~
	(as of 2021-06-15 The MirOS Project temporarily reconvenes on OFTC)

Re: [musl] bug: isatty() can return wrong value

[Gil Pedersen]

[-- Attachment #1: Type: text/plain, Size: 1140 bytes --]

> On 20 Aug 2024, at 13.11, Rich Felker <dalias@libc.org> wrote:
> 
> What guarantee do we have that nothing other than a tty in this state
> will return EIO for the ioctl? The implementation on the kernel side
> suggests that, if they tried to put any other device in such a state
> by replacing its ioctl function the same way, it would also give EIO
> for these ioctls.

Yeah, EIO is unconditionally signalled for any (expect TIOCSPGRP) ioctl call when a tty is hung.
While a TIOCGWINSZ ioctl call that returns EIO will very likely be a tty, there is indeed no such guarantee.

> I'm hopeful there's some good fix here where we don't have to have
> either of these bad behaviors, but returning true for isatty() of a
> device that is absolutely not a tty is a much worse behavior than
> returning false for a device that is/was a tty when it was opened but
> that's been replaced by the kernel with a defunct device.

I have proposed a change to the ioctl handling in the kernel:

https://lore.kernel.org/linux-serial/20241121111506.4717-1-gpdev@gpost.dk/

If merged, the issue should be fixed going forward.

/Gil

[-- Attachment #2: Type: text/html, Size: 8562 bytes --]

Re: [musl] bug: isatty() can return wrong value

[Rich Felker]

On Tue, Nov 26, 2024 at 11:03:36AM +0100, Gil Pedersen wrote:
> > On 20 Aug 2024, at 13.11, Rich Felker <dalias@libc.org> wrote:
> > 
> > What guarantee do we have that nothing other than a tty in this state
> > will return EIO for the ioctl? The implementation on the kernel side
> > suggests that, if they tried to put any other device in such a state
> > by replacing its ioctl function the same way, it would also give EIO
> > for these ioctls.
> 
> Yeah, EIO is unconditionally signalled for any (expect TIOCSPGRP) ioctl call when a tty is hung.
> While a TIOCGWINSZ ioctl call that returns EIO will very likely be a tty, there is indeed no such guarantee.
> 
> > I'm hopeful there's some good fix here where we don't have to have
> > either of these bad behaviors, but returning true for isatty() of a
> > device that is absolutely not a tty is a much worse behavior than
> > returning false for a device that is/was a tty when it was opened but
> > that's been replaced by the kernel with a defunct device.
> 
> I have proposed a change to the ioctl handling in the kernel:
> 
> https://lore.kernel.org/linux-serial/20241121111506.4717-1-gpdev@gpost.dk/
> 
> If merged, the issue should be fixed going forward.

Great. AIUI the proposed kernel change would fix this issue without
needing any further action on the libc side, right?

Rich