I don't think that code is right. It's correct to return with a bit set in rfds for a file descriptor that's ready with only an end of file (zero read).

EBADF should only be given for a file descriptor number that's invalid.

If X11 is reacting to EBADF, I'd expect that would only be because it knows that it will close a file descriptor when a client vanishes (say), but in that case,

it's emulated file descriptor in APE should also be closed, and it's the subsequent attempt to use it (in select or elsewhere) that should produce EBADF.

In other words, I'd fix select, and then change whatever else needs to be changed (if anything does) if X11 then stops working.