Just curious about this: was there any feedback from Jeff Bonwick and/or Bill Moore re the ARC -vs- page cache?
Or would any of the design notes document the reasoning behind the decision?
Surely it must have come up and been justified or got an exception in the Solaris architecture review (SARC "20 Q's", wasn't it called?) Since AFAICS it affected Solaris O/S interface (former-)guarantees. Although those notes are probably lost / inaccessible now...
There's also the monthly OpenZFS leadership meeting, Matt Ahrens et al are in there: I wonder if they would have access to some of the original reasoning; how it was justified / why it was permitted.
Dave, btw: check out the high-level structure of ZFS metadata -- every block is checksummed, and the checksum kept in the parent block (i.e. *not* kept together), applicable for both data and metadata blocks, and at least two copies are kept of metadata (but you can request more depending on your paranoia, see also "ditto" blocks). Compression is optional at the filesystem level (not held at the pool aka volume level; a pool may contain multiple filesystems), when compression is enabled if affects future created files, same if unset or changed to another algorithm; the filesystem handles a mix of files (blocks, even; I forget offhand) existing with various or no compression.
Rgds, Stuart.