Well, the best book has not been written, since one that says "well hey,
this is how the whole mess developed" doesn't seem to be out there, but

1. IBM PS/2 hardware technical reference manual is quite definitive on VGAs
    but hard to come by!

2. Programmers Guide to PC & PS/2 Video Systems, R. Wilton, Microsoft Press
    is pretty good, if a bit "BIOS"y, but is certainly pure VGA

3. Programmers Guide to the EGA, VGA and Super VGA Cards, Ferraro, Adison Wesley
    was quite good on first generation SVGAs, and has some history

4. The Indispensible PC Hardware Book, Messmer, Adison Wesley, is always useful