* [9fans] myricom 10 gigabit ethernet
@ 2007-04-07 16:03 matt
2007-04-07 16:18 ` Robert Sherwood
2007-04-07 17:58 ` erik quanstrom
0 siblings, 2 replies; 11+ messages in thread
From: matt @ 2007-04-07 16:03 UTC (permalink / raw)
To: 9fans
> coraid was given the blessing on monday to give the
> the myricom 10 gigabit ethernet *driver* with firmware
> for the cpu and fs kernels to the plan 9 community.
I can't find one online to buy and no-one's ebaying *anything* by Myricom.
Nor can I find a 10G switch.
I presume these items are all outside of the pocket of hobbyists ?
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 16:03 [9fans] myricom 10 gigabit ethernet matt @ 2007-04-07 16:18 ` Robert Sherwood 2007-04-07 16:58 ` matt ` (3 more replies) 2007-04-07 17:58 ` erik quanstrom 1 sibling, 4 replies; 11+ messages in thread From: Robert Sherwood @ 2007-04-07 16:18 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs [-- Attachment #1: Type: text/plain, Size: 425 bytes --] > > I can't find one online to buy and no-one's ebaying *anything* by Myricom. > Nor can I find a 10G switch. You must be kidding: http://www.google.com/search?q=myricom I presume these items are all outside of the pocket of hobbyists ? > > > $700 for a card, and $12K for the switch. I believe you're right about that. Does anyone know whether the PCI or PCI-x backplane even has 10 gbps of throughput? [-- Attachment #2: Type: text/html, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 16:18 ` Robert Sherwood @ 2007-04-07 16:58 ` matt 2007-04-07 17:00 ` Paweł Lasek ` (2 subsequent siblings) 3 siblings, 0 replies; 11+ messages in thread From: matt @ 2007-04-07 16:58 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs > I can't find one online to buy and no-one's ebaying *anything* by Myricom. > > Nor can I find a 10G switch. > > > You must be kidding: http://www.google.com/search?q=myricom I had "pages from the UK" ticked. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 16:18 ` Robert Sherwood 2007-04-07 16:58 ` matt @ 2007-04-07 17:00 ` Paweł Lasek 2007-04-07 17:03 ` W B Hacker 2007-04-07 17:52 ` erik quanstrom 3 siblings, 0 replies; 11+ messages in thread From: Paweł Lasek @ 2007-04-07 17:00 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs On 4/7/07, Robert Sherwood <robert.sherwood@gmail.com> wrote: > Does anyone know whether the PCI or PCI-x backplane even has 10 gbps of > throughput? Correct me if my info is wrong :-) IIRC one PCI-Express line equals bi-directional sync. 250 MiB/s. The fastest setting which is available now is AFAIK 32 lines per device. I wouldn't be surprised if those cards use PCI-e 8x slots, which give around 2 GiB/s and are fairly available in high-end machines, your typical PC excluded ;-) as most "standard" boards include one 16x slot or (SLI/CrossFire) 2x "physical" 16x slots, which are wired 1x&16x or 8x&8x depending whether you use SLI or not. On server/workstation boards it's fairly easy to get 8x slots. MacPro, IIRC, can support two cards at 8x, especially if you cut off bandwidth to graphic card (It was advertised to have configurable bandwidth settings, i.e. all slots could accommodate up to 8x or more, but you had less "lines available"+config program in firmware allowing you to set bandidth to each slot.) -- Paul Lasek ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 16:18 ` Robert Sherwood 2007-04-07 16:58 ` matt 2007-04-07 17:00 ` Paweł Lasek @ 2007-04-07 17:03 ` W B Hacker 2007-04-07 18:27 ` erik quanstrom 2007-04-07 17:52 ` erik quanstrom 3 siblings, 1 reply; 11+ messages in thread From: W B Hacker @ 2007-04-07 17:03 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs Robert Sherwood wrote: >> >> I can't find one online to buy and no-one's ebaying *anything* by >> Myricom. >> Nor can I find a 10G switch. > > > You must be kidding: http://www.google.com/search?q=myricom > > > I presume these items are all outside of the pocket of hobbyists ? >> >> >> $700 for a card, and $12K for the switch. I believe you're right about > that. > > Does anyone know whether the PCI or PCI-x backplane even has 10 gbps of > throughput? > PCI does not. It struggles to keep Gig-E 'well utilized' let alone saturated. PCI-X & PCI-E *may* have. Depends on a lot more than the spec itself. In any case, 'benchmark' setups aside, the bigger question is if it 'matters here yet'. Regardless of NIC and switch costs. Hardware - storage/OS/bus & CPU/RAM transport - on platforms an individual or small team can afford all play a major part in presenting other links in the chain that may be much weaker w/r either sustained or burst throughput. The pragmatic item is twin on-board Gig-E (most server MB, some top-end DIY MB, plus a few top-end embedded), plus - perhaps - one or two add-in Gig-E cards. 4 X Gig-E is about all you can readily drive with current CPU/RAM/bus, but at least are relatively cheap and cheerful, as are their switching hubs and decent cables. Now if somebody has 'deep pockets', it's another story. Google the data rates on DWDM and CWDM fibre-optics. And their costs.. Not much around that can fill those pipes on a non-shared basis that doesn't come with 'big iron' nameplates. And - from experience selling such backbones to Fortune 400 - CFO's who don't want the money spent on them there either... :-( Bill ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 17:03 ` W B Hacker @ 2007-04-07 18:27 ` erik quanstrom 2007-04-07 20:27 ` W B Hacker 0 siblings, 1 reply; 11+ messages in thread From: erik quanstrom @ 2007-04-07 18:27 UTC (permalink / raw) To: 9fans > Robert Sherwood wrote: > PCI does not. It struggles to keep Gig-E 'well utilized' let alone saturated. > PCI-X & PCI-E *may* have. Depends on a lot more than the spec itself. this is not correct. please read http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect even at 33 mhz, pci has 133MB/s of bandwidth. gbe uses about 112MB/s. although there are some lower standards defined, pci-x has a practical base frequency of 133Mhz * 64 bits which is over 1GB/s. pcie is (8/10)*2.5gbit bidirectional per lane. just 1 lane is required for gbe. > Hardware - storage/OS/bus & CPU/RAM transport - on platforms an individual or > small team can afford all play a major part in presenting other links in the > chain that may be much weaker w/r either sustained or burst throughput. this isn't ment to be advertizing, it's just the only numbers that i know. a coraid sr1521 can stream 210 MB/s from disk over 2xgbe on a raid 5 lblade. it should be within the price range of a modest group. > 4 X Gig-E is about all you can readily drive with current CPU/RAM/bus, but at > least are relatively cheap and cheerful, as are their switching hubs and decent > cables. this hasn't been my experience and sometimes latency, not bandwidth, is the driver. i think a fileserver may be one of those cases. i'll post some numbers when i get a chance. - erik ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 18:27 ` erik quanstrom @ 2007-04-07 20:27 ` W B Hacker 2007-04-07 21:13 ` erik quanstrom 0 siblings, 1 reply; 11+ messages in thread From: W B Hacker @ 2007-04-07 20:27 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs erik quanstrom wrote: >> Robert Sherwood wrote: >> PCI does not. It struggles to keep Gig-E 'well utilized' let alone saturated. >> PCI-X & PCI-E *may* have. Depends on a lot more than the spec itself. > > this is not correct. please read http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect > even at 33 mhz, pci has 133MB/s of bandwidth. gbe uses about 112MB/s. > > although there are some lower standards defined, pci-x has a practical base > frequency of 133Mhz * 64 bits which is over 1GB/s. > > pcie is (8/10)*2.5gbit bidirectional per lane. just 1 lane is required for gbe. > Do you suppose there is nothing else wanting to 'do work' on, or use that bus? Or place 'non-bus' demands on the rest of the system? And if not, then what is there to be transferred OVER that fast link? From whence? and to where? Unless we are designing the backplane fabric for a 10Gig-E router - and even those have to do 'work' besides just streaming bits - raw bit-rate is only a small part of the picture. 4 x Gig-E is about the most you can keep up with and still do 'useful work', even with twin dual-core AMD. - unless you have the cash for Many-MP IBM Power5 / Power6 ? Everyone wants to add-up the 'best case' burst rates for every component. Reality is to take the worst-case, then focus on fixing those bottlenecks first. As usual. the main one is *money*... ;-) Bill ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 20:27 ` W B Hacker @ 2007-04-07 21:13 ` erik quanstrom 2007-04-07 23:15 ` W B Hacker 0 siblings, 1 reply; 11+ messages in thread From: erik quanstrom @ 2007-04-07 21:13 UTC (permalink / raw) To: 9fans > erik quanstrom wrote: >> this is not correct. please read http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect >> even at 33 mhz, pci has 133MB/s of bandwidth. gbe uses about 112MB/s. >> >> although there are some lower standards defined, pci-x has a practical base >> frequency of 133Mhz * 64 bits which is over 1GB/s. >> >> pcie is (8/10)*2.5gbit bidirectional per lane. just 1 lane is required for gbe. >> > > Do you suppose there is nothing else wanting to 'do work' on, or use that bus? > Or place 'non-bus' demands on the rest of the system? pci-[ex] "busses" aren't shared resources. they are point-to-point links. there is no contention on a pci-[ex] link. also, even modestly modern machines have multiple pci busses. my 997Mhz pIII hand-me-down has 3 and my 450Mhz 440gx fileserver has 2. > > 4 x Gig-E is about the most you can keep up with and still do 'useful work', > even with twin dual-core AMD. > > - unless you have the cash for Many-MP IBM Power5 / Power6 ? > > Everyone wants to add-up the 'best case' burst rates for every component. do you have any data to back this up? > Reality is to take the worst-case, then focus on fixing those bottlenecks first. > > As usual. the main one is *money*... isn't that too easy? i think that if i take that attitude as a software engineer, i won't look at what my software's doing and work to make it better. i believe the fundamental bottleneck is ideas. without a good idea, unlimited wherewithall buys nothing. - erik ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 21:13 ` erik quanstrom @ 2007-04-07 23:15 ` W B Hacker 0 siblings, 0 replies; 11+ messages in thread From: W B Hacker @ 2007-04-07 23:15 UTC (permalink / raw) To: Fans of the OS Plan 9 from Bell Labs erik quanstrom wrote: >> erik quanstrom wrote: >>> this is not correct. please read http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect >>> even at 33 mhz, pci has 133MB/s of bandwidth. gbe uses about 112MB/s. >>> >>> although there are some lower standards defined, pci-x has a practical base >>> frequency of 133Mhz * 64 bits which is over 1GB/s. >>> >>> pcie is (8/10)*2.5gbit bidirectional per lane. just 1 lane is required for gbe. >>> >> Do you suppose there is nothing else wanting to 'do work' on, or use that bus? >> Or place 'non-bus' demands on the rest of the system? > > pci-[ex] "busses" aren't shared resources. they are point-to-point links. there is > no contention on a pci-[ex] link. Right. And each has its own CPU, RAM, et al to fill and empty it... ?? I think not.. > > also, even modestly modern machines have multiple pci busses. my 997Mhz pIII > hand-me-down has 3 and my 450Mhz 440gx fileserver has 2. > .. of which one or two are commonly used 'internally' in the ~bridge for on-board gadgetry... According to catalog (if memory serves) a fully-equipped RS-6000 could have someting on the order of 53 PCI slots. Seriously doubt that many were so optioned, though thye *did* have 256-bit-wide solid-state crossbar switching to allow diffeent CPU-RAM-I/O to have simultaneous paths. >> 4 x Gig-E is about the most you can keep up with and still do 'useful work', >> even with twin dual-core AMD. >> >> - unless you have the cash for Many-MP IBM Power5 / Power6 ? >> >> Everyone wants to add-up the 'best case' burst rates for every component. > > do you have any data to back this up? Well - 'wire' speed' was not the object of this particular exercise, but I'll cite it for what is 'between the lines': http://www.usenix.org/events/usenix05/tech/freenix/full_papers/hensbergen/hensbergen_html/index.html - which used mere Gig-E and rather modest boxen at both ends. But even with bespoke benchmarks, one might guess that there is more going on than bus or NIC speed alone - say file replication, archiving, DB transactions, queue'ing and caching methodology (or NOT), even. *something* has to fill and empty these critters, and that 'something' usually involves computations being performed on the data, verification, checksums, encryption, reads and writes to memory and storage media. All that before we do anything like 'work' with that data - such as computing a few hundreds of thousands of itemized telephone bills and spooling them to high-speed printers. (BT,DT,GTTS) All activities which contend - if not for space on the same wire, at least for a share of CPU, RAM, storage I/O. Hence all just as important a part of the end-to-end equation as raw communications link bandwidth. > >> Reality is to take the worst-case, then focus on fixing those bottlenecks first. >> >> As usual. the main one is *money*... > > isn't that too easy? i think that if i take that attitude as a software engineer, > i won't look at what my software's doing and work to make it better. > Much of the work of a software engineer DOES have to do with finding clever ways around obstacles - time and money constraints just as much as hardware ones. > i believe the fundamental bottleneck is ideas. without a good idea, unlimited > wherewithall buys nothing. > > - erik > Can't disagree w/r the 'wherewithall'. Though even SAGE didn't have 'unlimited'. But the same pragmatism applies to software engineering as to management in general: "..accomplishing the task at hand with the resources you have, not those you *wish* you had..' Is Plan9's future limited by lack of affordable Myricom hardware? I don't think that even Gig-E is today's major bottleneck, let alone 10-Gig-E and beyond. Gig-E is certainly affordable enough *now*, and 10-Gig will soon be also. Regardless - the 9P toolset has, among other features (see above citation), an efficiency edge in certain applications. And not just a few. Many. And that should scale as the bandwidth becomes affordable. As it has done. But how much of that 'edge' is lost when 9P must be encapsulated in, for example TCP/IP because there are few 'highways' that can route raw 9P across town, let alone across a continent? The Plan9 community hasn't much chance of driving the cost of faster link hardware down singlehandedly. That will move with a far larger market, and we should be thankful for gamers and 'pulp' entertainment consumers. But there *might* be room left for improvement elsewhere. Encryption. Hashing generation and comparison. Compression. Queueing. Protocols. Error handling. Dare I say 'caching'? Or even fs redesign? Is there no room left for improvement of Fossil and/or Venti? The Pick Operating System was fit for its purpose. Very fit. But where is it now? ...so stone me for a heretic, then... but a pragmatic heretic, if you please! ;-) Bill ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 16:18 ` Robert Sherwood ` (2 preceding siblings ...) 2007-04-07 17:03 ` W B Hacker @ 2007-04-07 17:52 ` erik quanstrom 3 siblings, 0 replies; 11+ messages in thread From: erik quanstrom @ 2007-04-07 17:52 UTC (permalink / raw) To: 9fans > I presume these items are all outside of the pocket of hobbyists ? >> >> >> $700 for a card, and $12K for the switch. I believe you're right about > that. our fileserver and cpu server are connected by an inexpensive edgecore 2-port 10gbe + 24-port gbe switch + 2-port fiber switch. i believe it's in the neighborhood of $4000. also, 10gbaseCX cards can be hooked back-to-back with no special requirements. > Does anyone know whether the PCI or PCI-x backplane even has 10 gbps of > throughput? myricom doesn't make a pci or pci-x 10gbe card. they are 8-lane pcie only. on my testing machines, i got between 1200 and 2600 MB/s read/write dma performance between the card and main memory. 10gbe is approx. 1250 MB/s. - erik ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [9fans] myricom 10 gigabit ethernet 2007-04-07 16:03 [9fans] myricom 10 gigabit ethernet matt 2007-04-07 16:18 ` Robert Sherwood @ 2007-04-07 17:58 ` erik quanstrom 1 sibling, 0 replies; 11+ messages in thread From: erik quanstrom @ 2007-04-07 17:58 UTC (permalink / raw) To: 9fans well, this is a good news, bad news situation. the good news is that there's some support for bleeding-edge hardware. the bad news is that bleeding-edge hardware is relatively expensive. the marvell 88sx[56]0xx driver is much more affordable. you can get a 8-port card based on this chipset from newegg for about $115. this is pretty nice for hot-swap hardware. if you're interested in any of this hardware, please contact me off list. - erik >> coraid was given the blessing on monday to give the >> the myricom 10 gigabit ethernet *driver* with firmware >> for the cpu and fs kernels to the plan 9 community. > > I can't find one online to buy and no-one's ebaying *anything* by Myricom. > Nor can I find a 10G switch. > > I presume these items are all outside of the pocket of hobbyists ? ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2007-04-07 23:15 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2007-04-07 16:03 [9fans] myricom 10 gigabit ethernet matt 2007-04-07 16:18 ` Robert Sherwood 2007-04-07 16:58 ` matt 2007-04-07 17:00 ` Paweł Lasek 2007-04-07 17:03 ` W B Hacker 2007-04-07 18:27 ` erik quanstrom 2007-04-07 20:27 ` W B Hacker 2007-04-07 21:13 ` erik quanstrom 2007-04-07 23:15 ` W B Hacker 2007-04-07 17:52 ` erik quanstrom 2007-04-07 17:58 ` erik quanstrom
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).