It's more than that. Many board vendors will use a secured stage 1 bootloader that assumes U-Boot. It's probably possible to shove in a different second stage loader, but you'll still need to do board initialization as you mentioned. It's not that onerous and there is source out there, but I think it's really a question of motivation. U-Boot is supported by the same vendors and covers of the behavior you will likely want in a boot loader (and then some). Board configs are easy to customize so that you aren't carrying around a massive binary. The binary I use for the Arndale is measured in kilobytes, not megabytes.

Every SoC is going to have a different process - in the end, you'll have something that will probably look quite a bit like U-Boot without any real benefit. I'd rather tilt at other windmills...