I don't know which other ARM board you tried, but I have always found terrible I/O performance of the Pi to be a bigger problem that the ARM speed. The USB2 interface is really slow, and there arn't really many other (documented) alternative options. The Ethernet goes through the same slow USB interface, and there is only so much that you can do bit bashing data with GPIO's. The sdCard interface seems to be the only non-usb filesystem I/O available. And that in turn limits the viability of relieving the RAM contraints with virtual memory. So the ARM processor itself is not usually the problem for me.
In general I find the pi a nice little device for quite a few things - like low power, low bandwidth, low cost servers or displays with plenty of open source compatability.. Or hacking/prototyping where I don't want to have to worry too much about blowing things up. But it not good for high throughput I/O, memory intensive applications, or anything requiring a lot of processing power.
The validity of your conclusion regarding low power ARM in general probably depends on what the other board you tried was..
DigbyT