Difficulty emulating F_DUPFD_CLOEXEC

[Rich Felker <dalias@aerifal.cx>]

Hi again,

Old kernels lack fcntl F_DUPFD_CLOEXEC, which musl needs internally
and wants to provide to applications. Thus, I'd like to emulate it
like we do for pipe2, dup3, socket SOCK_CLOEXEC, etc.; the emulation
has a race condition that leaks fds, but it's better than nothing.

The problem I'm having is how to detect the case where the kernel
lacks F_DUPFD_CLOEXEC. For other other atomic close-on-exec
operations, we either have ENOSYS (newly added syscall) or an
unambiguous EINVAL. But for fcntl, we could get EINVAL because
F_DUPFD_CLOEXEC is not recognized by the kernel, or because the
argument is larger than OPEN_MAX. So we need a test for the cause
behind EINVAL.

False positives are not an option, because if we wrongly detect that
F_DUPFD_CLOEXEC was not supported, we would emulate it with the code
that has race conditions and fd leaks, even on new kernels which
should not have these problems.

The best idea I have so far is:

1. Try F_DUPFD_CLOEXEC. If it succeeds or fails with any error other
   than EINVAL, we're done.

2. If it fails with EINVAL, retry with an argument of 0. This will
   eliminate the other cause of EINVAL, so now we should get EINVAL
   only on old kernels that lack F_DUPFD_CLOEXEC. If this test
   succeeds, we need to close the new duplicate fd we made (on the
   wrong fd number) and report EINVAL back to the caller.

3. If the test in step 2 failed, F_DUPFD_CLOEXEC is unsupported, and
   we have to use the fallback code with race conditions.

This uglifies fcntl.c a bit more, but I think it works. Does the above
reasoning make sense? Any other ideas?

Rich