Re: [9fans] myricom 10 gigabit ethernet

[W B Hacker <wbh@conducive.org>]

erik quanstrom wrote:
>> erik quanstrom wrote:
>>> this is not correct.  please read http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect
>>> even at 33 mhz, pci has 133MB/s of bandwidth.  gbe uses about 112MB/s.
>>>
>>> although there are some lower standards defined, pci-x has a practical base
>>> frequency of 133Mhz * 64 bits which is over 1GB/s.
>>>
>>> pcie is (8/10)*2.5gbit bidirectional per lane.  just 1 lane is required for gbe.
>>>
>> Do you suppose there is nothing else wanting to 'do work' on, or use that bus?
>> Or place 'non-bus' demands on the rest of the system?
> 
> pci-[ex] "busses" aren't shared resources.  they are point-to-point links.  there is 
> no contention on a pci-[ex] link.

Right.  And each has its own CPU, RAM, et al to fill and empty it... ??

I think not..

> 
> also, even modestly modern machines have multiple pci busses. my 997Mhz pIII
> hand-me-down has 3 and my 450Mhz 440gx fileserver has 2.
> 

.. of which one or two are commonly used 'internally' in the ~bridge for 
on-board gadgetry...

According to catalog (if memory serves) a fully-equipped RS-6000 could have 
someting on the order of 53 PCI slots.  Seriously doubt that many were so 
optioned, though thye *did* have 256-bit-wide solid-state crossbar switching to 
allow diffeent CPU-RAM-I/O to have simultaneous paths.

>> 4 x Gig-E is about the most you can keep up with and still do 'useful work', 
>> even with twin dual-core AMD.
>>
>> - unless you have the cash for Many-MP IBM Power5 / Power6 ?
>>
>> Everyone wants to add-up the 'best case' burst rates for every component.
> 
> do you have any data to back this up?  

Well - 'wire' speed' was not the object of this particular exercise, but I'll 
cite it for what is 'between the lines':

http://www.usenix.org/events/usenix05/tech/freenix/full_papers/hensbergen/hensbergen_html/index.html

- which used mere Gig-E and rather modest boxen at both ends.

But even with bespoke benchmarks, one might guess that there is more going on 
than bus or NIC speed alone - say file replication, archiving, DB transactions, 
queue'ing and caching methodology (or NOT), even.

*something* has to fill and empty these critters, and that 'something' usually 
involves computations being performed on the data, verification, checksums, 
encryption, reads and writes to memory and storage media.

All that before we do anything like 'work' with that data - such as computing a 
few hundreds of thousands of itemized telephone bills and spooling them to 
high-speed printers. (BT,DT,GTTS)

All activities which contend - if not for space on the same wire, at least for a 
share of CPU, RAM, storage I/O.  Hence all just as important a part of the 
end-to-end equation as raw communications link bandwidth.

> 
>> Reality is to take the worst-case, then focus on fixing those bottlenecks first.
>>
>> As usual. the main one is *money*...
> 
> isn't that too easy?   i think that if i take that attitude as a software engineer,
> i won't look at what my software's doing and work to make it better.
> 

Much of the work of a software engineer DOES have to do with finding clever ways 
around obstacles - time and money constraints just as much as hardware ones.

> i believe the fundamental bottleneck is ideas.  without a good idea, unlimited
> wherewithall buys nothing.
> 
> - erik
> 

Can't disagree w/r the 'wherewithall'. Though even SAGE didn't have 'unlimited'.

But the same pragmatism applies to software engineering as to management in general:

"..accomplishing the task at hand with the resources you have, not those you 
*wish* you had..'

Is Plan9's future limited by lack of affordable Myricom hardware?

I don't think that even Gig-E is today's major bottleneck, let alone 10-Gig-E 
and beyond.

Gig-E is certainly affordable enough *now*, and 10-Gig will soon be also.

Regardless - the 9P toolset has, among other features (see above citation), an 
efficiency edge in certain applications.

And not just a few. Many. And that should scale as the bandwidth becomes 
affordable. As it has done.

But how much of that 'edge' is lost when 9P must be encapsulated in, for example 
TCP/IP because there are few 'highways' that can route raw 9P across town, let 
alone across a continent?

The Plan9 community hasn't much chance of driving the cost of faster link 
hardware down singlehandedly.  That will move with a far larger market, and we 
should be thankful for gamers and 'pulp' entertainment consumers.

But there *might* be room left for improvement elsewhere.

Encryption.  Hashing generation and comparison. Compression. Queueing. 
Protocols. Error handling.

Dare I say 'caching'? Or even fs redesign?  Is there no room left for 
improvement of Fossil and/or Venti?

The Pick Operating System was fit for its purpose.  Very fit.

But where is it now?

...so stone me for a heretic, then...  but a pragmatic heretic, if you please!

;-)

Bill