[9fans] myricom 10 gigabit ethernet

[9fans] myricom 10 gigabit ethernet

[matt]

> coraid was given the blessing on monday to give the
> the myricom 10 gigabit ethernet *driver* with firmware
> for the cpu and fs kernels to the plan 9 community. 

I can't find one online to buy and no-one's ebaying *anything* by Myricom.
Nor can I find a 10G switch.

I presume these items are all outside of the pocket of hobbyists ?

Re: [9fans] myricom 10 gigabit ethernet

[Robert Sherwood]

[-- Attachment #1: Type: text/plain, Size: 425 bytes --]

>
> I can't find one online to buy and no-one's ebaying *anything* by Myricom.
> Nor can I find a 10G switch.


You must be kidding: http://www.google.com/search?q=myricom


I presume these items are all outside of the pocket of hobbyists ?
>
>
> $700 for a card, and $12K for the switch. I believe you're right about
that.

Does anyone know whether the PCI or PCI-x backplane even has 10 gbps of
throughput?

[-- Attachment #2: Type: text/html, Size: 819 bytes --]

Re: [9fans] myricom 10 gigabit ethernet

[matt]

 > I can't find one online to buy and no-one's ebaying *anything* by 
Myricom.
>
>     Nor can I find a 10G switch.
>
>
> You must be kidding: http://www.google.com/search?q=myricom

I had "pages from the UK" ticked.

Re: [9fans] myricom 10 gigabit ethernet

[Paweł Lasek]

On 4/7/07, Robert Sherwood <robert.sherwood@gmail.com> wrote:

> Does anyone know whether the PCI or PCI-x backplane even has 10 gbps of
> throughput?

Correct me if my info is wrong :-)

IIRC one PCI-Express line equals bi-directional sync. 250 MiB/s. The
fastest setting which is available now is AFAIK 32 lines per device.
I wouldn't be surprised if those cards use PCI-e 8x slots, which give
around 2 GiB/s and are fairly available in high-end machines, your
typical PC excluded ;-) as most "standard" boards include one 16x slot
or (SLI/CrossFire) 2x "physical" 16x slots, which are wired 1x&16x or
8x&8x depending whether you use SLI or not.

On server/workstation boards it's fairly easy to get 8x slots. MacPro,
IIRC, can support two cards at 8x, especially if you cut off bandwidth
to graphic card (It was advertised to have configurable bandwidth
settings, i.e. all slots could accommodate up to 8x or more, but you
had less "lines available"+config program in firmware allowing you to
set bandidth to each slot.)
-- 
Paul Lasek

Re: [9fans] myricom 10 gigabit ethernet

[W B Hacker]

Robert Sherwood wrote:
>>
>> I can't find one online to buy and no-one's ebaying *anything* by 
>> Myricom.
>> Nor can I find a 10G switch.
> 
> 
> You must be kidding: http://www.google.com/search?q=myricom
> 
> 
> I presume these items are all outside of the pocket of hobbyists ?
>>
>>
>> $700 for a card, and $12K for the switch. I believe you're right about
> that.
> 
> Does anyone know whether the PCI or PCI-x backplane even has 10 gbps of
> throughput?
> 

PCI does not. It struggles to keep Gig-E 'well utilized' let alone saturated.

PCI-X & PCI-E *may* have. Depends on a lot more than the spec itself.

In any case, 'benchmark' setups aside, the bigger question is if it 'matters 
here yet'.

Regardless of NIC and switch costs.

Hardware - storage/OS/bus & CPU/RAM transport - on platforms an individual or 
small team can afford all play a major part in presenting other links in the 
chain that may be much weaker w/r either sustained or burst throughput.

The pragmatic item is twin on-board Gig-E (most server MB, some top-end DIY MB, 
plus a few top-end embedded), plus - perhaps - one or two add-in Gig-E cards.

4 X Gig-E is about all you can readily drive with current CPU/RAM/bus, but at 
least are relatively cheap and cheerful, as are their switching hubs and decent 
cables.

Now if somebody has 'deep pockets', it's another story.  Google the data rates 
on DWDM and CWDM fibre-optics.  And their costs..

Not much around that can fill those pipes on a non-shared basis that doesn't 
come with 'big iron' nameplates.

And - from experience selling such backbones to Fortune 400 - CFO's who don't 
want the money spent on them there either...

:-(

Bill

Re: [9fans] myricom 10 gigabit ethernet

[erik quanstrom]

> I presume these items are all outside of the pocket of hobbyists ?
>>
>>
>> $700 for a card, and $12K for the switch. I believe you're right about
> that.

our fileserver and cpu server are connected by an inexpensive edgecore
2-port 10gbe + 24-port gbe switch + 2-port fiber switch.  i believe it's 
in the neighborhood of $4000.

also, 10gbaseCX cards can be hooked back-to-back with no special
requirements.

> Does anyone know whether the PCI or PCI-x backplane even has 10 gbps of
> throughput?

myricom doesn't make a pci or pci-x 10gbe card.  they are 8-lane pcie only.
on my testing machines, i got between 1200 and 2600 MB/s read/write
dma performance between the card and main memory.  10gbe is approx.
1250 MB/s.

- erik

Re: [9fans] myricom 10 gigabit ethernet

[erik quanstrom]

well, this is a good news, bad news situation.  the good news
is that there's some support for bleeding-edge hardware.
the bad news is that bleeding-edge hardware is relatively
expensive.

the marvell 88sx[56]0xx driver is much more affordable.
you can get a 8-port card based on this chipset from newegg
for about $115.  this is pretty nice for hot-swap hardware.

if you're interested in any of this hardware, please 
contact me off list.

- erik

>> coraid was given the blessing on monday to give the
>> the myricom 10 gigabit ethernet *driver* with firmware
>> for the cpu and fs kernels to the plan 9 community. 
> 
> I can't find one online to buy and no-one's ebaying *anything* by Myricom.
> Nor can I find a 10G switch.
> 
> I presume these items are all outside of the pocket of hobbyists ?

Re: [9fans] myricom 10 gigabit ethernet

[erik quanstrom]

> Robert Sherwood wrote:
> PCI does not. It struggles to keep Gig-E 'well utilized' let alone saturated.
> PCI-X & PCI-E *may* have. Depends on a lot more than the spec itself.

this is not correct.  please read http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect
even at 33 mhz, pci has 133MB/s of bandwidth.  gbe uses about 112MB/s.

although there are some lower standards defined, pci-x has a practical base
frequency of 133Mhz * 64 bits which is over 1GB/s.

pcie is (8/10)*2.5gbit bidirectional per lane.  just 1 lane is required for gbe.

> Hardware - storage/OS/bus & CPU/RAM transport - on platforms an individual or 
> small team can afford all play a major part in presenting other links in the 
> chain that may be much weaker w/r either sustained or burst throughput.

this isn't ment to be advertizing, it's just the only numbers that i know.  a coraid sr1521
can stream 210 MB/s from disk over 2xgbe on a raid 5 lblade.  it should be within
the price range of a modest group.

> 4 X Gig-E is about all you can readily drive with current CPU/RAM/bus, but at 
> least are relatively cheap and cheerful, as are their switching hubs and decent 
> cables.

this hasn't been my experience and sometimes latency, not bandwidth, is the driver.
i think a fileserver may be one of those cases.  i'll post some numbers when i get a
chance.

- erik

Re: [9fans] myricom 10 gigabit ethernet

[W B Hacker]

erik quanstrom wrote:
>> Robert Sherwood wrote:
>> PCI does not. It struggles to keep Gig-E 'well utilized' let alone saturated.
>> PCI-X & PCI-E *may* have. Depends on a lot more than the spec itself.
> 
> this is not correct.  please read http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect
> even at 33 mhz, pci has 133MB/s of bandwidth.  gbe uses about 112MB/s.
> 
> although there are some lower standards defined, pci-x has a practical base
> frequency of 133Mhz * 64 bits which is over 1GB/s.
> 
> pcie is (8/10)*2.5gbit bidirectional per lane.  just 1 lane is required for gbe.
> 

Do you suppose there is nothing else wanting to 'do work' on, or use that bus?
Or place 'non-bus' demands on the rest of the system?

And if not, then what is there to be transferred OVER that fast link?
 From whence? and to where?

Unless we are designing the backplane fabric for a 10Gig-E router - and even 
those have to do 'work' besides just streaming bits - raw bit-rate is only a 
small part of the picture.

4 x Gig-E is about the most you can keep up with and still do 'useful work', 
even with twin dual-core AMD.

- unless you have the cash for Many-MP IBM Power5 / Power6 ?

Everyone wants to add-up the 'best case' burst rates for every component.

Reality is to take the worst-case, then focus on fixing those bottlenecks first.

As usual. the main one is *money*...

;-)

Bill

Re: [9fans] myricom 10 gigabit ethernet

[erik quanstrom]

> erik quanstrom wrote:
>> this is not correct.  please read http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect
>> even at 33 mhz, pci has 133MB/s of bandwidth.  gbe uses about 112MB/s.
>> 
>> although there are some lower standards defined, pci-x has a practical base
>> frequency of 133Mhz * 64 bits which is over 1GB/s.
>> 
>> pcie is (8/10)*2.5gbit bidirectional per lane.  just 1 lane is required for gbe.
>> 
> 
> Do you suppose there is nothing else wanting to 'do work' on, or use that bus?
> Or place 'non-bus' demands on the rest of the system?

pci-[ex] "busses" aren't shared resources.  they are point-to-point links.  there is 
no contention on a pci-[ex] link.

also, even modestly modern machines have multiple pci busses. my 997Mhz pIII
hand-me-down has 3 and my 450Mhz 440gx fileserver has 2.

> 
> 4 x Gig-E is about the most you can keep up with and still do 'useful work', 
> even with twin dual-core AMD.
> 
> - unless you have the cash for Many-MP IBM Power5 / Power6 ?
> 
> Everyone wants to add-up the 'best case' burst rates for every component.

do you have any data to back this up?  

> Reality is to take the worst-case, then focus on fixing those bottlenecks first.
> 
> As usual. the main one is *money*...

isn't that too easy?   i think that if i take that attitude as a software engineer,
i won't look at what my software's doing and work to make it better.

i believe the fundamental bottleneck is ideas.  without a good idea, unlimited
wherewithall buys nothing.

- erik

Re: [9fans] myricom 10 gigabit ethernet

[W B Hacker]

erik quanstrom wrote:
>> erik quanstrom wrote:
>>> this is not correct.  please read http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect
>>> even at 33 mhz, pci has 133MB/s of bandwidth.  gbe uses about 112MB/s.
>>>
>>> although there are some lower standards defined, pci-x has a practical base
>>> frequency of 133Mhz * 64 bits which is over 1GB/s.
>>>
>>> pcie is (8/10)*2.5gbit bidirectional per lane.  just 1 lane is required for gbe.
>>>
>> Do you suppose there is nothing else wanting to 'do work' on, or use that bus?
>> Or place 'non-bus' demands on the rest of the system?
> 
> pci-[ex] "busses" aren't shared resources.  they are point-to-point links.  there is 
> no contention on a pci-[ex] link.

Right.  And each has its own CPU, RAM, et al to fill and empty it... ??

I think not..

> 
> also, even modestly modern machines have multiple pci busses. my 997Mhz pIII
> hand-me-down has 3 and my 450Mhz 440gx fileserver has 2.
> 

.. of which one or two are commonly used 'internally' in the ~bridge for 
on-board gadgetry...

According to catalog (if memory serves) a fully-equipped RS-6000 could have 
someting on the order of 53 PCI slots.  Seriously doubt that many were so 
optioned, though thye *did* have 256-bit-wide solid-state crossbar switching to 
allow diffeent CPU-RAM-I/O to have simultaneous paths.

>> 4 x Gig-E is about the most you can keep up with and still do 'useful work', 
>> even with twin dual-core AMD.
>>
>> - unless you have the cash for Many-MP IBM Power5 / Power6 ?
>>
>> Everyone wants to add-up the 'best case' burst rates for every component.
> 
> do you have any data to back this up?  

Well - 'wire' speed' was not the object of this particular exercise, but I'll 
cite it for what is 'between the lines':

http://www.usenix.org/events/usenix05/tech/freenix/full_papers/hensbergen/hensbergen_html/index.html

- which used mere Gig-E and rather modest boxen at both ends.

But even with bespoke benchmarks, one might guess that there is more going on 
than bus or NIC speed alone - say file replication, archiving, DB transactions, 
queue'ing and caching methodology (or NOT), even.

*something* has to fill and empty these critters, and that 'something' usually 
involves computations being performed on the data, verification, checksums, 
encryption, reads and writes to memory and storage media.

All that before we do anything like 'work' with that data - such as computing a 
few hundreds of thousands of itemized telephone bills and spooling them to 
high-speed printers. (BT,DT,GTTS)

All activities which contend - if not for space on the same wire, at least for a 
share of CPU, RAM, storage I/O.  Hence all just as important a part of the 
end-to-end equation as raw communications link bandwidth.

> 
>> Reality is to take the worst-case, then focus on fixing those bottlenecks first.
>>
>> As usual. the main one is *money*...
> 
> isn't that too easy?   i think that if i take that attitude as a software engineer,
> i won't look at what my software's doing and work to make it better.
> 

Much of the work of a software engineer DOES have to do with finding clever ways 
around obstacles - time and money constraints just as much as hardware ones.

> i believe the fundamental bottleneck is ideas.  without a good idea, unlimited
> wherewithall buys nothing.
> 
> - erik
> 

Can't disagree w/r the 'wherewithall'. Though even SAGE didn't have 'unlimited'.

But the same pragmatism applies to software engineering as to management in general:

"..accomplishing the task at hand with the resources you have, not those you 
*wish* you had..'

Is Plan9's future limited by lack of affordable Myricom hardware?

I don't think that even Gig-E is today's major bottleneck, let alone 10-Gig-E 
and beyond.

Gig-E is certainly affordable enough *now*, and 10-Gig will soon be also.

Regardless - the 9P toolset has, among other features (see above citation), an 
efficiency edge in certain applications.

And not just a few. Many. And that should scale as the bandwidth becomes 
affordable. As it has done.

But how much of that 'edge' is lost when 9P must be encapsulated in, for example 
TCP/IP because there are few 'highways' that can route raw 9P across town, let 
alone across a continent?

The Plan9 community hasn't much chance of driving the cost of faster link 
hardware down singlehandedly.  That will move with a far larger market, and we 
should be thankful for gamers and 'pulp' entertainment consumers.

But there *might* be room left for improvement elsewhere.

Encryption.  Hashing generation and comparison. Compression. Queueing. 
Protocols. Error handling.

Dare I say 'caching'? Or even fs redesign?  Is there no room left for 
improvement of Fossil and/or Venti?

The Pick Operating System was fit for its purpose.  Very fit.

But where is it now?

...so stone me for a heretic, then...  but a pragmatic heretic, if you please!

;-)

Bill