[musl] Should fdopen() check the passed file descriptor?

[musl] Should fdopen() check the passed file descriptor?

[Sascha Brawer]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

Dear list

Should musl’s fdopen() verify whether the passed file descriptor refers to
an open file? glibc seems to do that, whereas musl doesn’t.

According to the spec, fdopen "may" fail with EBADF if a bad file
descriptor is passed.
https://pubs.opengroup.org/onlinepubs/009696899/functions/fdopen.html

I just ran into this difference while porting some user library (libosmium,
for OpenStreetMap) to Alpine Linux which uses musl. For now I’m patching
the user library so it tests for fcntl(fd, F_GETFD) < 0 before invoking
fdopen(). But I wonder if this check shouldn’t rather be done within musl
itself. What do you think?

Thanks for cc’ing me on your thoughts, I’m not subscribed to the musl
mailing list. Best,

— Sascha Brawer, sascha@brawer.ch

[-- Attachment #2: Type: text/html, Size: 1108 bytes --]

Re: [musl] Should fdopen() check the passed file descriptor?

[Markus Wichmann]

On Thu, Aug 04, 2022 at 02:42:49PM +0200, Sascha Brawer wrote:
> Dear list
>
> Should musl’s fdopen() verify whether the passed file descriptor refers to
> an open file?
>

It should not. Normal musl policy for "may fail" is to just not check
that. musl only implements "shall fail".

In this case, using numbers that are not FDs as such is really really
bad. It means the control flow is utterly br0ken. How did the program
even get into a state in which a variable is supposed to be holding an
FD but doesn't?

It betrays a really big problem with the program if this can happen. If
the program is multi-threaded (or opens files from a signal handler,
which can happen since open(2) is async-signal-safe), the program might
suddenly start using a file descriptor wrong. That would be an
interesting problem to debug.

In short, any program calling fdopen() with a number that is not
guaranteed to be an open file has a logic error that needs to be fixed
post-haste. Returning EBADF is really only plastering over the problem.
It would be like dereferencing a pointer fresh out of malloc() and
catching the SIGSEGV instead of just testing whether the pointer is 0.

Ciao,
Markus

Re: [musl] Should fdopen() check the passed file descriptor?

[Rich Felker]

On Thu, Aug 04, 2022 at 04:23:22PM +0200, Markus Wichmann wrote:
> On Thu, Aug 04, 2022 at 02:42:49PM +0200, Sascha Brawer wrote:
> > Dear list
> >
> > Should musl’s fdopen() verify whether the passed file descriptor refers to
> > an open file?
> >
> 
> It should not. Normal musl policy for "may fail" is to just not check
> that. musl only implements "shall fail".
> 
> In this case, using numbers that are not FDs as such is really really
> bad. It means the control flow is utterly br0ken. How did the program
> even get into a state in which a variable is supposed to be holding an
> FD but doesn't?
> 
> It betrays a really big problem with the program if this can happen. If
> the program is multi-threaded (or opens files from a signal handler,
> which can happen since open(2) is async-signal-safe), the program might
> suddenly start using a file descriptor wrong. That would be an
> interesting problem to debug.
> 
> In short, any program calling fdopen() with a number that is not
> guaranteed to be an open file has a logic error that needs to be fixed
> post-haste. Returning EBADF is really only plastering over the problem.
> It would be like dereferencing a pointer fresh out of malloc() and
> catching the SIGSEGV instead of just testing whether the pointer is 0.

Yes, this is a really good explanation of the reasoning here.

Moreover, in this case checking it would incur high extra cost for
correct programs for the sake of producing a useless (because "may
fail") error that incorrect programs can't even rely on getting. The
high cost is because we'd have to do an extra probing syscall *before*
doing anything with side effects. We can't even move the existing
syscalls that would probe (only present in some code paths) before the
malloc, because then they would have their side effects even if the
malloc failed, and if we moved them afterwards, we'd have to have a
failure exit path after malloc already succeeded, thus forcing free to
be linked in a program that otherwise might not want it.

Rich