[9fans] Hardware for venti/fossil/cpu server

[9fans] Hardware for venti/fossil/cpu server

[Matthias Teege]

Moin,

i'll build my new venti/fossil/cpu/auth all in one plan9 server now
but I'm not sure about the supported hardware. I select a AMD64, an
ASUS A8V DEL S939 VIAKT800PRO with SATA and 4 BARRACUDAs with 120GB.
The specs of the mainboard are at
http://uk.asus.com/prog/spec.asp?m=A8V%20Deluxe&langs=11. Im not sure
about AMD64,SATA and the on board GBit Ethernet. Ethernet isn't a
problem because I can use a simple Intel Etherexpress but ist plan9
supporting SATA and AMD64. Can somebody recommend another working
system with actual hardware?

Many thanks
Matthias

Re: [9fans] Hardware for venti/fossil/cpu server

[geoff]

I've never heard of the built-in Gbe.  You're probably better off
just buying an Intel PRO/1000 MT and putting it in.

SATA is supposed to just work, but I believe there's limited
experience with it so far on Plan 9.

The AMD64 shouldn't be a problem, as long you are content to run
it in 32-bit mode for now.  8[cal] don't yet generate the
instruction prefixes (prefaces?) for 64-bit addresses.

I'd also recommend avoiding built-in i8254[17]* controllers;
they aren't quite compatible with the igbe drivers in Plan 9
and need various bug workarounds in the drivers (no surprise,
the word `compatible' has been greatly watered-down by the
hardware vendors).  I may get around to adding those
workarounds, but the reference Linux driver from Intel is
10,000 lines long, so it may not happen immediately.

I've always distrusted hardware built into the motherboard
and the current round of built-in Gbe controllers is doing
nothing to change that distrust.  They're either broken
(the not-quite igbe), Broadcom or Brand X (Marvell).
I'm not sure how breaking the igbe controllers reduces
anybody's costs.

Re: [9fans] Hardware for venti/fossil/cpu server

[Nigel Roles]

geoff@collyer.net wrote:

>I've never heard of the built-in Gbe.  You're probably better off
>just buying an Intel PRO/1000 MT and putting it in.
>
The Marvell device is also in the nForce 3 chipset.
There is a reverse engineered driver in Linux which seems fine so
far. I thought the driver name quite amusing - forcedeth.
It's only 2000 lines. I know this sounds a lot, but it looks
straightforward enough, and a lot of the 2000 lines is #defines.

>I've always distrusted hardware built into the motherboard
>and the current round of built-in Gbe controllers is doing
>nothing to change that distrust.  They're either broken
>(the not-quite igbe), Broadcom or Brand X (Marvell).
>I'm not sure how breaking the igbe controllers reduces
>anybody's costs.
>
>  
>
Well, if you want 1Gb out of your 1Gb ethernet chip, then
plugging into the PCI bus is a non starter. PCI is just not very
fast. We have tried a Tyan tomcat with opteron 248 + 2 broadcom
on PCI, and it maxes out at 800mbps and 100% CPU! Some of
that is the Broadcom chip's fault, but the rest is the fact that the
bus is full. PCI is only just over a Gbit when bursting.

You need the Gbe connected either to a better bus (e.g. PCI-X) or
the CPU side of the PCI bridge. This can be achieved either by
embedding in the chipset (e.g. nForce3, KT800PRO), or by having
an alternate ethernet specific bus (Intel CSA).

PCI-X is expensive. The sockets are long, and there's no space
on the motherboards. You only get it on two uniprocessor boards
that I know of (SuperMicro and a very new Gigabyte one), otherwise
it has to be dual processor or more. This doubles the cost of the
motherboard from 100 to 200 quid.

That's how it reduces costs.

Re: [9fans] Hardware for venti/fossil/cpu server

[geoff]

Thanks for the explanation, Nigel.

I can see now why the motherboard makers want to include a CSA
version of the igbe, I just don't see why they felt the
need to break the CSA version of the igbe in the process.
I guess it's part of this crazy business of making the
ether drivers micromanage the configuration of the PHYs
and EEPROMs and other irrelevant rubbish that the hardware
ought to be taking care of internally (or at least setting
to sensible default state initially).

Re: [9fans] Hardware for venti/fossil/cpu server

[Nigel Roles]

geoff@collyer.net wrote:

>Thanks for the explanation, Nigel.
>
>I can see now why the motherboard makers want to include a CSA
>version of the igbe, I just don't see why they felt the
>need to break the CSA version of the igbe in the process.
>I guess it's part of this crazy business of making the
>ether drivers micromanage the configuration of the PHYs
>and EEPROMs and other irrelevant rubbish that the hardware
>ought to be taking care of internally (or at least setting
>to sensible default state initially).
>
>  
>
Actually on closer inspection the marvell chip is external on the Asus
motherboard, and still only connected via (!) PCI.

Only the Marvell PHY is used by Nvidia, and the embedded Gbe is their
own.

The linux driver for the Marvell chip (Yukon aka SysKonnect sk98)
is 35000 lines. Sorry, that's not fair: 43000 including headers. See,
I said 2000 lines for the Nvidia driver was nothing.

Thus the Asus board is an example of how not to connect your
Gbe.

I'm not about to try it, but I suspect getting Plan 9 to work on the
nForce 3 chipsets would not be too bad.

Re: [9fans] Hardware for venti/fossil/cpu server

[jmk]

The 2 AMD64 motherboards we have experience with are
the Tyan S2882 dual Opteron (mentioned before) and recently
an Aopen MK89-L (http://usa.aopen.com/products/mb/MK89-L.htm).
We stuck it in an Antec Aria case to see if it made a decent
desktop machine. We've not investigated it fully, e.g. it has
a Realtek 8110S 1000/100/10 controller on-board and we haven't
got round to making that work (we boot it with PXE via
that controller then switch to using an Intel 82557 card in
one of the PCI slots). Realtek were unresponsive when asked
for programming details but looking at existing *nix drivers
it doesn't look hard to fix up the existing ether8139.c.

As Nigel says, this is low end and if you want 64-bit PCI, etc.
then you have to pay a lot more.

--jim

Re: [9fans] Hardware for venti/fossil/cpu server

[geoff]

Plan 9's kernel would need some work to ensure that it would
run using 64-bit addresses (64-bit integers are supported),
and 8[cal] would need to generate the 64-bit instruction
prefixes.  My impression is that it's less work than
a port to a brand-new 64-bit processor but not completely
trivial, as the kernel would need at least an inspection
for places where pointers are assumed to be 32 bits wide.

Re: [9fans] Hardware for venti/fossil/cpu server

[vdharani]

jmk,

> The 2 AMD64 motherboards we have experience with are
> the Tyan S2882 dual Opteron (mentioned before) and recently
> an Aopen MK89-L (http://usa.aopen.com/products/mb/MK89-L.htm).
> We stuck it in an Antec Aria case to see if it made a decent
> desktop machine. We've not investigated it fully, e.g. it has
> a Realtek 8110S 1000/100/10 controller on-board and we haven't
> got round to making that work (we boot it with PXE via
> that controller then switch to using an Intel 82557 card in
> one of the PCI slots). Realtek were unresponsive when asked
> for programming details but looking at existing *nix drivers
> it doesn't look hard to fix up the existing ether8139.c.

is it that Opteron processor is run in 32 bit mode? Or is it that 64 bit
is supported in plan9 tools now?

thanks
dharani

Re: [9fans] Hardware for venti/fossil/cpu server

[Kenji Okamoto]

> There are
> only three to five users. I need the machine for file service, email,
> web service on a slow internet connection and the daily fun with sam
> and rio. 

We are using 1GB, 16GHz Pen4 with ide disks for 6 users here (Venti+Fossil).
Our usages are maily for graphics, which means large data will communicate
between file server and terminal.   We don't see any frustration here.
However, I don't know how CPU/AUth server can load more.

Kenji

Re: [9fans] Hardware for venti/fossil/cpu server

[geoff]

If you just need to be able to connect more devices, you can
add IDE or SCSI PCI cards.  Companies like Promise make compatible
IDE cards, just don't take their claims of doing RAID in hardware
too seriously; Promise at least do it entirely in their proprietary
drivers.

Re: [9fans] Hardware for venti/fossil/cpu server

[Matthias Teege]

On Wed, 22 Sep 2004 11:19:21 -0400, jmk@plan9.bell-labs.com
<jmk@plan9.bell-labs.com> wrote:

> As Nigel says, this is low end and if you want 64-bit PCI, etc.
> then you have to pay a lot more.

I don't need GB Ethernet and highest possible performance. There are
only three to five users. I need the machine for file service, email,
web service on a slow internet connection and the daily fun with sam
and rio. Because the machine will run some years it would be nice if I
can use "new" hardware. I would like to put 4 disks and a CDROM in the
server so I need a bus where I can put 5 units on. Thats why I'm
looking for SATA.

If GB Ethernet the only problem I can deal with it. The Tyan maybe an
option. If there is another/better ...

Thanks for help
Matthias

Re: [9fans] Hardware for venti/fossil/cpu server

[jmk]

The 2 AMD64 motherboards we have experience with are
the Tyan S2882 dual Opteron (mentioned before) and recently
an Aopen MK89-L (http://usa.aopen.com/products/mb/MK89-L.htm).
We stuck it in an Antec Aria case to see if it made a decent
desktop machine. We've not investigated it fully, e.g. it has
a Realtek 8110S 1000/100/10 controller on-board and we haven't
got round to making that work (we boot it with PXE via
that controller then switch to using an Intel 82557 card in
one of the PCI slots). Realtek were unresponsive when asked
for programming details but looking at existing *nix drivers
it doesn't look hard to fix up the existing ether8139.c.

As Nigel says, this is low end and if you want 64-bit PCI, etc.
then you have to pay a lot more.

--jim