Re: [9fans] Hardware for venti/fossil/cpu server

[Nigel Roles <nigel@9fs.org>]

geoff@collyer.net wrote:

>I've never heard of the built-in Gbe.  You're probably better off
>just buying an Intel PRO/1000 MT and putting it in.
>
The Marvell device is also in the nForce 3 chipset.
There is a reverse engineered driver in Linux which seems fine so
far. I thought the driver name quite amusing - forcedeth.
It's only 2000 lines. I know this sounds a lot, but it looks
straightforward enough, and a lot of the 2000 lines is #defines.

>I've always distrusted hardware built into the motherboard
>and the current round of built-in Gbe controllers is doing
>nothing to change that distrust.  They're either broken
>(the not-quite igbe), Broadcom or Brand X (Marvell).
>I'm not sure how breaking the igbe controllers reduces
>anybody's costs.
>
>  
>
Well, if you want 1Gb out of your 1Gb ethernet chip, then
plugging into the PCI bus is a non starter. PCI is just not very
fast. We have tried a Tyan tomcat with opteron 248 + 2 broadcom
on PCI, and it maxes out at 800mbps and 100% CPU! Some of
that is the Broadcom chip's fault, but the rest is the fact that the
bus is full. PCI is only just over a Gbit when bursting.

You need the Gbe connected either to a better bus (e.g. PCI-X) or
the CPU side of the PCI bridge. This can be achieved either by
embedding in the chipset (e.g. nForce3, KT800PRO), or by having
an alternate ethernet specific bus (Intel CSA).

PCI-X is expensive. The sockets are long, and there's no space
on the motherboards. You only get it on two uniprocessor boards
that I know of (SuperMicro and a very new Gigabyte one), otherwise
it has to be dual processor or more. This doubles the cost of the
motherboard from 100 to 200 quid.

That's how it reduces costs.