Re: [musl] Should fdopen() check the passed file descriptor?

[Rich Felker <dalias@libc.org>]

On Thu, Aug 04, 2022 at 04:23:22PM +0200, Markus Wichmann wrote:
> On Thu, Aug 04, 2022 at 02:42:49PM +0200, Sascha Brawer wrote:
> > Dear list
> >
> > Should musl’s fdopen() verify whether the passed file descriptor refers to
> > an open file?
> >
> 
> It should not. Normal musl policy for "may fail" is to just not check
> that. musl only implements "shall fail".
> 
> In this case, using numbers that are not FDs as such is really really
> bad. It means the control flow is utterly br0ken. How did the program
> even get into a state in which a variable is supposed to be holding an
> FD but doesn't?
> 
> It betrays a really big problem with the program if this can happen. If
> the program is multi-threaded (or opens files from a signal handler,
> which can happen since open(2) is async-signal-safe), the program might
> suddenly start using a file descriptor wrong. That would be an
> interesting problem to debug.
> 
> In short, any program calling fdopen() with a number that is not
> guaranteed to be an open file has a logic error that needs to be fixed
> post-haste. Returning EBADF is really only plastering over the problem.
> It would be like dereferencing a pointer fresh out of malloc() and
> catching the SIGSEGV instead of just testing whether the pointer is 0.

Yes, this is a really good explanation of the reasoning here.

Moreover, in this case checking it would incur high extra cost for
correct programs for the sake of producing a useless (because "may
fail") error that incorrect programs can't even rely on getting. The
high cost is because we'd have to do an extra probing syscall *before*
doing anything with side effects. We can't even move the existing
syscalls that would probe (only present in some code paths) before the
malloc, because then they would have their side effects even if the
malloc failed, and if we moved them afterwards, we'd have to have a
failure exit path after malloc already succeeded, thus forcing free to
be linked in a program that otherwise might not want it.

Rich