[9fans] pineview atom

[9fans] pineview atom

[erik quanstrom]

it seems that the pineview atom can be
a great plan 9 machine.  i've got a
x7spa-h atom d510 motherboard with
2 x 82574 gbe, etc.  it supports 4gb
of memory.  it "just works".

could support amd64, too (dx 0x20000000
indicates 64-bit support)

	; aux/cpuid -n 0x80000001
	00000000 00000000 00000001 20100000

- erik

-----
Plan 9
E820: 00000000 0009d800 memory
E820: 00100000 bf680000 memory
E820: 100000000 140000000 memory
126 holes free
00018000 00088000 458752
002d9000 10000000 265449472
265908224 bytes free
cpu0: 1667MHz GenuineIntel Atom (cpuid: AX 0x106CA DX 0xBFEBFBFF)
pat: 0107040600070406
ELCR: CC80
LAPIC: fee00000 e0000000
cpu1: 1668MHz GenuineIntel Atom (cpuid: AX 0x106CA DX 0xBFEBFBFF)
#l0: i82574: 1000Mbps port 0xFEAE0000 irq 11 tu 1514: [ea]
#l1: i82574: 1000Mbps port 0xFEBE0000 irq 10 tu 1514: [ea]
sdata: blind probe 1f0
sdata: blind probe 170
#S/sdE: ahci ich port 0xe0042000: sss 1 ncs 31 coal 1 mport 5 led 1 clo 1 ems 1
#S/sdE: ich: sata-II with 6 ports
ahci enclosure size = 0002; loc = 0100*4
ehci: handoff: bios owned
ehci: handoff: bios owned
ehci: handoff failed
3062M memory: 256M kernel data, 2806M user, 3431M swap

Re: [9fans] pineview atom

[matt]

>   it supports 4gb of memory.
>
>
of non-ECC memory, so nice terminal, bad server


"the probability of having at least one bit error in 4 gigabyes of
memory at sea level on planet Earth in 72 hours is over 95%."


 http://lambda-diode.com/opinion/ecc-memory

Re: [9fans] pineview atom

[ron minnich]

On Thu, Feb 18, 2010 at 10:26 AM, matt <maht-9fans@maht0x0r.net> wrote:
>
>
>>  it supports 4gb of memory.
>
> of non-ECC memory, so nice terminal, bad server
>
>
> "the probability of having at least one bit error in 4 gigabyes of memory at
> sea level on planet Earth in 72 hours is over 95%."


humorous and true story. The Big Mac system at Virginia Tech could
tell you if it was day or night. There was a dramatic increase in
memory error rates during the day. I saw a very nice graph showing the
effect.

Seems the earth is a pretty good shield for nasty high energy
particles from our local light source :-)

So just run it as a server at night and you may be fine!

ron

Re: [9fans] pineview atom

[Patrick Kelly]

On Feb 18, 2010, at 1:31 PM, ron minnich <rminnich@gmail.com> wrote:

> On Thu, Feb 18, 2010 at 10:26 AM, matt <maht-9fans@maht0x0r.net>
> wrote:
>>
>>
>>>  it supports 4gb of memory.
>>
>> of non-ECC memory, so nice terminal, bad server
>>
>>
>> "the probability of having at least one bit error in 4 gigabyes of
>> memory at
>> sea level on planet Earth in 72 hours is over 95%."
>
>
> humorous and true story. The Big Mac system at Virginia Tech could
> tell you if it was day or night. There was a dramatic increase in
> memory error rates during the day. I saw a very nice graph showing the
> effect.
Do you have a link to that graph? may be an article too?
>
> Seems the earth is a pretty good shield for nasty high energy
> particles from our local light source :-)
>
> So just run it as a server at night and you may be fine!
>
> ron
>

Re: [9fans] pineview atom

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

On Thu, Feb 18, 2010 at 10:31 AM, ron minnich <rminnich@gmail.com> wrote:

> On Thu, Feb 18, 2010 at 10:26 AM, matt <maht-9fans@maht0x0r.net> wrote:
> >
> >
> >>  it supports 4gb of memory.
> >
> > of non-ECC memory, so nice terminal, bad server
> >
> >
> > "the probability of having at least one bit error in 4 gigabyes of memory
> at
> > sea level on planet Earth in 72 hours is over 95%."
>
>
> humorous and true story. The Big Mac system at Virginia Tech could
> tell you if it was day or night. There was a dramatic increase in
> memory error rates during the day. I saw a very nice graph showing the
> effect.
>
> Seems the earth is a pretty good shield for nasty high energy
> particles from our local light source :-)
>
> So just run it as a server at night and you may be fine!
>
> ron
>

I had a little interaction with that setup myself.  Didn't they move to
Xserves ultimately so they could have ECC RAM?  I think it was all PowerMac
G5s to start.

I know of a larger cluster of Xserves too that was in Huntsville AL that I
physically visited and worked with that was all Xserves (2500 of them I
believe at the time).

[-- Attachment #2: Type: text/html, Size: 1629 bytes --]

Re: [9fans] pineview atom

[erik quanstrom]

> of non-ECC memory, so nice terminal, bad server
>
>
> "the probability of having at least one bit error in 4 gigabyes of
> memory at sea level on planet Earth in 72 hours is over 95%."
>
>
>  http://lambda-diode.com/opinion/ecc-memory

while i agree in general that ecc is a good idea, i can't
subscribe to predictions that seem so far from real-world
experience.

i have been running a amd64 machine as a cpu server
for 2 years.  it uses standard ddr2 memory.  i have not
seen any unexplained crashes or reboots during this time.

at work, i have been running 5 un-ecc'd machines for
3 years.  also no unexplained crahses or reboots.

how could this be if i would expect 240 + 1825 bit errors?

- erik

Re: [9fans] pineview atom

[ron minnich]

On Thu, Feb 18, 2010 at 10:39 AM, erik quanstrom
<quanstro@labs.coraid.com> wrote:

> while i agree in general that ecc is a good idea, i can't
> subscribe to predictions that seem so far from real-world
> experience.

in many cases it's all about location. Where I used to live, 7200 feet
up, it was a huge issue. Where you live, i am assuming close to sea
level, and with a small number of machines, the statistics say that
you're unlikely to see it. But I would not want to take several
thousand of your machines to Los Alamos and try to make them run ...

ron

Re: [9fans] pineview atom

[erik quanstrom]

> in many cases it's all about location. Where I used to live, 7200 feet
> up, it was a huge issue. Where you live, i am assuming close to sea
> level, and with a small number of machines, the statistics say that
> you're unlikely to see it. But I would not want to take several
> thousand of your machines to Los Alamos and try to make them run ...

exactly.  and coraid storage appliances use ecc throughout the
datapath.

all i'm saying is that 1 error per 3 days per 4 gb at sea level
(the orginal claim) seems exaggerated to me.   i believe my sample
size is big enough to refute that claim to very high confidence.

- erik

Re: [9fans] pineview atom

[Dave Eckhardt]

> i have been running a amd64 machine as a cpu server
> for 2 years.  it uses standard ddr2 memory.  i have not
> seen any unexplained crashes or reboots during this time.
>
> at work, i have been running 5 un-ecc'd machines for
> 3 years.  also no unexplained crahses or reboots.
>
> how could this be if i would expect 240 + 1825 bit errors?

There is no mechanism which directly translates bit flips
to crashes!  The bad case is actually a corruption which
does *not* cause a crash, but is written to disk.  How
often do you check your venti's SHA-1 hashes?

Also, according to the data cited, machines do differ.

Dave Eckhardt

Re: [9fans] pineview atom

[erik quanstrom]

> There is no mechanism which directly translates bit flips
> to crashes!  The bad case is actually a corruption which
> does *not* cause a crash, but is written to disk.  How

indirection?  executable code being turned into illegal
instructions?  it's not 100% efficiency but it will translate
flipped bits into crashes.

- erik

Re: [9fans] pineview atom

[roger peppe]

On 18 February 2010 22:38, erik quanstrom <quanstro@labs.coraid.com> wrote:
> indirection?  executable code being turned into illegal
> instructions?  it's not 100% efficiency but it will translate
> flipped bits into crashes.

i'm interested - does anyone know what a typical relative
percentage of pointer- vs. non-pointer-valued memory is?

it might be interesting to find out.

Re: [9fans] pineview atom

[Adrian Tritschler]

On 19 February 2010 09:38, erik quanstrom <quanstro@labs.coraid.com> wrote:
>> There is no mechanism which directly translates bit flips
>> to crashes!  The bad case is actually a corruption which
>> does *not* cause a crash, but is written to disk.  How

> indirection?  executable code being turned into illegal
> instructions?  it's not 100% efficiency but it will translate
> flipped bits into crashes.

I believe Dave was implying that there is no mechanism that
_guarantees_ that a bit flip anywhere in memory will result in a code
crash.  Some bit flips just might mean the wrong colour pixel on your
screen, others might mean that someone's pay scale goes from $7.50 an
hour to $4096 + $17.50 an hour, some might just be in an unused chunk
of RAM. Flips in code are more likely to cause crashes, but still not
guaranteed.

> - erik

-- 
Adrian

Re: [9fans] pineview atom

[ron minnich]

On Thu, Feb 18, 2010 at 3:12 PM, Adrian Tritschler <ajft@ajft.org> wrote:
>  Flips in code are more likely to cause crashes, but still not
> guaranteed.

You'd need to look at fraction of total that is data vs. code, then at
fraction of total code that is going to cause hurt if flipped. This
stuff can have numbers attached.

Here's an example from my world. 1 MB of code, 32 MB of kernel, and
2GB minus that of data. This is a lower end ratio as the nodes don't
have much memory.

If the data is flipped, you're not going to know of errors unless you
are looking for numerical instability. People do.

And if you're in the middle of a checkpoint you won't know then.

But, this is all stuff that could be estimated ... and in some cases I
think has been.

ron

Re: [9fans] pineview atom

[Dave Eckhardt]

> You'd need to look at fraction of total that is data vs. code,
> then at fraction of total code that is going to cause hurt if
> flipped.  This stuff can have numbers attached.
>
> Here's an example from my world. 1 MB of code, 32 MB of kernel,
> and 2GB minus that of data.  This is a lower end ratio as the
> nodes don't have much memory.
>
> If the data is flipped, you're not going to know of errors unless
> you are looking for numerical instability.

Also subtract out all of the kernel code which is boot-only:  it
needs to be uncorrupted for just the twinkling of an eye.  Almost
all of every format string (used or not) can be corrupted without
anything dramatic happening.  While you're in the kernel, the
exception-handling label stack could be totally trashed as long
as nobody invokes error() during this system call.  Or maybe a bit
flip rewrites an instruction to use %ebx instead of %eax, but
at a point when they both contain the same value.

There's lots of stuff which doesn't have to be totally right to
"work", and even the stuff that must be 100% right may be fine
if it's wrong at the the right time.

"Back in the old days", a lot of VAX-11/750's running BSD Unix
crashed because of parity errors in their TLB's.  750's running
VMS "didn't have this problem", because VMS would silently work
around it; BSD grew that code--see, for example, <229@astrovax.UUCP>.
Then bits could flip all the time with nobody noticing!

Dave Eckhardt

Re: [9fans] pineview atom

[erik quanstrom]

> "Back in the old days", a lot of VAX-11/750's running BSD Unix
> crashed because of parity errors in their TLB's.  750's running
> VMS "didn't have this problem", because VMS would silently work
> around it; BSD grew that code--see, for example, <229@astrovax.UUCP>.
> Then bits could flip all the time with nobody noticing!

nobody noticed, or the os reloaded the tlb?  a tlb is usually
just a cache, and if it's parity protected obviously one could just
reload it on error.

- erik

Re: [9fans] pineview atom

[Dave Eckhardt]

>> "Back in the old days", a lot of VAX-11/750's running BSD Unix
>> crashed because of parity errors in their TLB's.  750's running
>> VMS "didn't have this problem", because VMS would silently work
>> around it; BSD grew that code--see, for example, <229@astrovax.UUCP>.
>> Then bits could flip all the time with nobody noticing!

> nobody noticed, or the os reloaded the tlb?

This was back in the days when TLB's loaded themselves.  I think
the work-around code as to flush the entry.

I mentioned the example because:

* Bits were flipping pretty often.  I think we got 10-ish events
per day.

* If there hadn't been parity protection, the result would have
been occasional unrepeatable weird crashes and data corruption.
That would have been really painful.  This isn't the same as the
general RAM case, because there aren't quiet backwaters of a
TLB which can go bad with no effect.

* Because VMS silently worked around the error and BSD didn't, for
a while the issue was a perplexing "Unix problem":  BSD ran pretty
well on older 750's and much less well on newer ones; VMS ran fine
on both.  In actuality the problem was quality control inside DEC,
but the combination of parity and restarting the operation enabled
successful computation on somewhat sketchy hardware.

Dave Eckhardt

Re: [9fans] pineview atom

[hugh]

On Feb 21, 2:48 pm, davide...@cs.cmu.edu (Dave Eckhardt) wrote:

> * Bits were flipping pretty often.  I think we got 10-ish events
> per day.

TLB bits are not like DRAM bits.  They were surely static cells, built
for speed and functionality (CAM) not density.  The cells would be
quite large.  It is unlikely that this problem came from external
radiation.  Guess: the problem was a marginal design of the circuitry.

At about that time DRAM cells seemed to be suffering from radiation-
induced bit flips.  It was felt that 16Kbit chips would be the limit
because of this (please realise that my own memory might be slightly
faulty).  It turned out that the radiation was actually coming from
the chip packaging material.  Once that was sorted, RAM density
marched on to where we are now.

As cells shrink, and voltages shrink, I understand that radiation can
have greater effects.  Eventually mainstream systems will have ECC.
But I've been thinking this for as long as there have been personal
computers built out of microprocessors.

Adding ECC to memory seems to me to be an easy no-brainer.  Adding it
"everywhere" in processors does not seem easy.

Actually, even adding it in memory isn't that easy.  In the old days,
a simple Hamming code was good enough because each bit in a word lived
on a different chip.  Now memory chips are wider and so the code has
to account for multi-bit errors (flipping of bits is not independent).

Cray famously said "Parity is for farmers".  It was an obscure joke
(referring to some US agricultural subsidy) but really he meant that
he didn't want to waste circuitry on error checking (as I  understand
it).  This was one of the things that made me averse to his systems.

It is really hard to guess what the conversion rate of bit flips into
observed anomalies on ordinary systems.  I wonder if any research has
been done on this.  In the real world, software bugs take surely most
of the blame.  Users seem to have been trained to accept lower
reliability in computer systems.

Apple seems to be one of the few vendors that might be able to market
the idea of ECC to consumers.

Re: [9fans] pineview atom

[Dave Eckhardt]

> Actually, even adding it in memory isn't that easy.  In the old days,
> a simple Hamming code was good enough because each bit in a word lived
> on a different chip.  Now memory chips are wider and so the code has
> to account for multi-bit errors (flipping of bits is not independent).

Indeed... http://en.wikipedia.org/wiki/Chipkill

Dave Eckhardt

Re: [9fans] pineview atom

[Corey Thomasson]

"Also avoid using your laptop [...] near radioactive areas (Three Mile
Island, Chernobyl, Ground Zero)." - from the follow up article.
>
>
>>   it supports 4gb of memory.
>>
>>
> of non-ECC memory, so nice terminal, bad server
>
>
> "the probability of having at least one bit error in 4 gigabyes of
> memory at sea level on planet Earth in 72 hours is over 95%."
>
>
>  http://lambda-diode.com/opinion/ecc-memory
>
>
>
>


--
Corey Thomasson - Web Development / Programming

Re: [9fans] pineview atom

[Albert Skye]

>>   it supports 4gb of memory.
> of non-ECC memory, so nice terminal, bad server

Any recommendations of similar hardware which does support SECDED ECC?

Re: [9fans] pineview atom

[erik quanstrom]

On Mon Mar  8 13:36:54 EST 2010, mistlail@yahoo.co.uk wrote:
> >>   it supports 4gb of memory.
> > of non-ECC memory, so nice terminal, bad server
>
> Any recommendations of similar hardware which does support SECDED ECC?

the memory controller is integrated into the cpu
package (though not the cpu) of the pineview
atom processors.  the memory controller
doesn't support ecc.  so you're pretty stuck with
pineview.  the older N-series atoms use an
off-package memory controller but i have never
seen one with a northbridge supporting ecc.

- erik

Re: [9fans] pineview atom

[Jonas Amoson]

Maybe Supermicro X8SIL

Don't know how well this card is supported by Plan 9, but
it has a �-ATX form factor and supports DDR3 ECC if you
couple it with a Xeon 34xx in the LGA1156 socket. The Quad
Core (no HT) Xeon L3426 is listed to have a TDP of 45 Watts.

Albert Skye wrote:
>>>   it supports 4gb of memory.
>> of non-ECC memory, so nice terminal, bad server
>
> Any recommendations of similar hardware which does support SECDED ECC?
>
>

Re: [9fans] pineview atom

[erik quanstrom]

> Maybe Supermicro X8SIL

obviously not an atom, but

> Don't know how well this card is supported by Plan 9, but

i have tested it and everything works fine.  of course
(repeating the op) the xeon processor is required to
get ecc, since the memory controller is part of the cpu.
and the core iX versions don't support ecc.

basically all the xeon 55xx up/dp/mp series work fine.
potential trouble spots might be bios or nics.
but many vendors appear to be using i82574s, which
are fine chips.

- erik