I offer the IPv4 to IPv6 fiasco as an example of a change because we could (and we thought it would help - hey I did in the early 1990s), but it failed for economic reasons. In the end, any real change has to take into account some level of economics.
The examples of the differences in the shell is actually a different issue -- that was territorial and not economics -- each vendor adding stuff that helped them (and drove IVS/end users of multiple platforms crazy). The reality with SunOS sh vs Ultrix sh vs HP-UX sh vs System V (att sh) was yet another similar but different -- every manufacturer messed with a V7 derivative sh was a little different -- including AT&T, Korn et al. For that matter you (Rob) created a new syntax command with Plan9 [although you did not try to be and never claimed to be V7 compatible -- to your point you break things where you thought it matters and as a researcher I accept that]. But because all the manufacturers were a little different, it was exactly why IEEE said -- wait a minute -- let's define a base syntax which will work everywhere and it is something we can all agree and if we all support it -- great. We did that, and we call that POSIX (and because it was designed by compromise and committee - like a camel it has some humps).
The problem Ted and Larry describes is real ... research vs. production. So it begs the question, at what time does it make it sensible/ (worth it/economically viable) to move on?
Apple famously breaks things and it drives me bonkers because many (most I would suggest) of those changes are hardly worth it -- be it my iPhone or my Mac. I just want to use the darned thing BTW: Last week, the clowns at Telsa just rolled out a new UI for my Model S --- ugh -- because they could (now I'm fumbling trying deal with the climate system or the radio -- it would not do bad if they had rolled out a the new UI on a simulator for my iPad so I could at least get used to it -- but I'm having to learn it live -- what a PITA -- that really makes me grumpy).
What I ask for this august body to consider is that before we start looking at these changes is to ask what we are really getting in return when a new implementation breaks something that worked before. e.g. I did not think systemd bought end users much value able, must like IPv6 in practice, it was thought to solve many problems, but did not buy that much and has caused (continues to cause) many more.
In biolog every so often we have an "ice age" and kill a few things off and get to start over. That rarely happens in technology, except when a real Christianen style disruption takes place -- which is based on economics -- a new market values the new idea and the old market dies off. I believe that from the batch/mainframe 1960s/early 70s world, Unix was just that -- but we got to start over because the economics of 'open systems' and the >>IP<< being 'freely available' [which compared to VMS and other really proprietary systems] did kill them off. I also think that the economics of completely free (Linux) ended up killing the custom Unix diversions.
Frankly, if (at the beginning) Plan9 has been a tad easier/cheaper/more economical for >>everyone<< in the community obtain (unlike original Unix release time, Plan9 was not the same rules because AT&T was under different rules and HW cost rules had changed things), it >>might<< have been the strong strain that killed off the old. If IPv6 has been (in practice) cheaper to use than IPv4 [which is what I personally thought the ISP would do with it - since it had been designed to help them] and not made as a premium feature (i.e they had made it economically to change), it might have killed of IPv4.
Look at 7 decades of Programming Language design, just being 'better' is not good enough. As I have said here and many other places, the reality is that Fortran still pays the salary of people like me in the HPC area [and I don't see Julia or for that matter, my own company's pretty flower - Data Parallel C++ making inroads soon]. It's possible that Rust as a system programming language >>might<< prove economical to replace C. I personally hope Go makes the inroads to replace C++ in user space. But for either to do that, there has to be an economical reason - no brainer style for management.
What got us here was a discussion of the original implementation of directory files, WRT links and how paths are traversed. The basic argument comes from issues with how and when objects are named. Rob, I agree with you, that just because UNIX (or any other system) used a scheme previously does not make the end-all. And I do believe that rethinking some of the choices made 5-6 decades ago is in order. But I ask the analysis of the new verse the old takes into account, how to mitigate the damage done. If its economics prove valuable, the evolution to using it will allow a stronger strain to take over, but just because something new vs. the old, does not make it valuable.
Respectfully ....
Happy new year everyone and hopefully 2022 proves a positive time for all of you.
Clem