Well, since I have no experience with Macs I can only guess, but the basic workflow is the following:

The configure.sh script tries to find the "X11/Xlib.h" file and to compile and link a simple C program using the X headers and libs. If this fails it assumes that X is not installed and does not build the "Graphics" module.

That's the bare-bones version, but as I said, I have no experience with Macs:

That's what a quick search have provided me with: https://stackoverflow.com/questions/14321038/x11-xlib-h-no-such-file-or-directory-on-mac-os-x-mountain-lion