I always used the design principle "Write locally, read over NFS".
This was the basic idea of AFS. Originally, the CMU folks did whole file caching, but by AFS 4.0 time, they had a Locus token manager (think DLM) that scaled really well so partial caching was allowed. It actually made a small disk system possible. What tended to happen, on your first boot, of course, you had to fill /bin and lot of heavily used directories. But what happened is that your system quickly had only the files you really needed on the local disk. - the ones you were writing, and the few you used over and over.
FWIW: I know a couple of people that still run it. I ran it until a few years ago when I switched NAS units just for cost reasons.