[9fans] Kernel panic when compiling Go on native Plan 9

[9fans] Kernel panic when compiling Go on native Plan 9

[Pavel Klinkovsky]

Hi all,

I have troubles to compile Go on my native Plan 9 - Go revision 14739
and later.
I do have a kernel panic when compiling exp/locale/collate.

Does anyone else have the same problem?
Thank you.

Pavel

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Pavel Klinkovsky]

I made another test.
I compared compilation of exp/locale/collate of the Go build 14738 and
14739 (and later).

Compilation of 14738 does not consume RAM a lot.
Compilation of 14739 (and later) consume a huge amount of RAM
(involving swap).

So it seems the Plan 9 has a problem with the virtual memory
management (when using swap), IMO.

Pavel

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Richard Miller]

Are you running a recent (since 24 August) kernel on a multiprocessor?

If so, try booting with *nomp=1 in plan9.ini and see if running on
just one cpu prevents the panic.

I've just updated my kernel, and I'm getting a panic on opening
a particular email message in acme.  Just debugging it now.

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Anthony Martin]

Pavel Klinkovsky <pavel.klinkovsky@gmail.com> once said:
> I made another test.
> I compared compilation of exp/locale/collate of the Go build 14738 and
> 14739 (and later).
>
> Compilation of 14738 does not consume RAM a lot.
> Compilation of 14739 (and later) consume a huge amount of RAM
> (involving swap).
>
> So it seems the Plan 9 has a problem with the virtual memory
> management (when using swap), IMO.

How much memory does your system have?

Changeset 14739 grew the Unicode collation tables
in the exp/locale/collate package by a considerable
amount. The compiler's memory usage now goes above
400 MB when building that package, almost 2.5x the
amount used to compile the second heavyweight and
15x the average.

  Anthony

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[erik quanstrom]

On Fri Nov  2 08:00:29 EDT 2012, 9fans@hamnavoe.com wrote:
> Are you running a recent (since 24 August) kernel on a multiprocessor?
>
> If so, try booting with *nomp=1 in plan9.ini and see if running on
> just one cpu prevents the panic.
>
> I've just updated my kernel, and I'm getting a panic on opening
> a particular email message in acme.  Just debugging it now.

what change do you see as problematic?

i know it's typical for us to run with multi-gigabyte processes.  so
assuming that the op has got a couple of gigs of memory and there is
no dastardly swap configured, it would be either
(a) something in the sources kernel has introduced a panic
(b) or something in go is tickling an existing panic.

but the changes i see all look to be corner cases, or startup
code which would fail on boot.  i'm missing what change could
have introduced this.

i'd guess an old bug.

- erik

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Pavel Klinkovsky]

> Are you running a recent (since 24 August) kernel on a multiprocessor?

I am running the latest kernel (pulled from the bell-labs repository).
I am running the Plan 9 on the single CPU only (old IBM T30).

> If so, try booting with *nomp=1 in plan9.ini and see if running on
> just one cpu prevents the panic.

I have got such option set by default.

> I've just updated my kernel, and I'm getting a panic on opening
> a particular email message in acme.  Just debugging it now.

I saw the problem occured a couple of seconds after the RAM was full,
and swapping was started.

Pavel

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Pavel Klinkovsky]

> How much memory does your system have?

- 512 MB RAM
- 512 MB swap

> Changeset 14739 grew the Unicode collation tables
> in the exp/locale/collate package by a considerable
> amount. The compiler's memory usage now goes above
> 400 MB when building that package, almost 2.5x the
> amount used to compile the second heavyweight and
> 15x the average.

I see.
I can confirm when the kernel panic occured I saw (in stats):
- full RAM
- small portion of swap occupied

Pavel

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Richard Miller]

> I am running the Plan 9 on the single CPU only (old IBM T30).

Sorry, that means we are looking at two different panics.
Mine seems to be a side effect of mpacpi.c trying to enable
the same cpu twice:

cpu0:  1599MHz GenuineIntel Atom (cpuid: AX 0x106C2 DX 0xBFE9FBFF)
...
mpinit: mp table describes 2 cpus
mpinit: scanning acpi madt for extra cpus
...
	mpacpiproc: apic 0xf04705c0
	mpacpiproc: apic 0xf0470648
cpu1:  1600MHz GenuineIntel Atom (cpuid: AX 0x106C2 DX 0xBFE9FBFF)
cpu1:  1599MHz GenuineIntel Atom (cpuid: AX 0x106C2 DX 0xBFE9FBFF)
...

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[erik quanstrom]

On Fri Nov  2 09:52:22 EDT 2012, 9fans@hamnavoe.com wrote:
> > I am running the Plan 9 on the single CPU only (old IBM T30).
>
> Sorry, that means we are looking at two different panics.
> Mine seems to be a side effect of mpacpi.c trying to enable
> the same cpu twice:
>
> cpu0:  1599MHz GenuineIntel Atom (cpuid: AX 0x106C2 DX 0xBFE9FBFF)
> ...
> mpinit: mp table describes 2 cpus
> mpinit: scanning acpi madt for extra cpus
> ...
> 	mpacpiproc: apic 0xf04705c0
> 	mpacpiproc: apic 0xf0470648
> cpu1:  1600MHz GenuineIntel Atom (cpuid: AX 0x106C2 DX 0xBFE9FBFF)
> cpu1:  1599MHz GenuineIntel Atom (cpuid: AX 0x106C2 DX 0xBFE9FBFF)

i found in nix that when mixing acpi and mp i had to defend against
double-entries for i/o apics since the numbering can be different.  but
it will be interesting to see what happened to start 2 cpu1s.

[...]
ioapic: 1 addr fec00000 base 0
ioapic: 3 addr fec8a000 base 24
ioapic: 8 same pa as apic 1
ioapic: 9 same pa as apic 3

- erik

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[erik quanstrom]

On Fri Nov  2 09:44:43 EDT 2012, pavel.klinkovsky@gmail.com wrote:
> > How much memory does your system have?
>
> - 512 MB RAM
> - 512 MB swap
>
> > Changeset 14739 grew the Unicode collation tables
> > in the exp/locale/collate package by a considerable
> > amount. The compiler's memory usage now goes above
> > 400 MB when building that package, almost 2.5x the
> > amount used to compile the second heavyweight and
> > 15x the average.
>
> I see.
> I can confirm when the kernel panic occured I saw (in stats):
> - full RAM
> - small portion of swap occupied

i might give the 9front kernel a go.  i think that cinap spent
some time trying to make swap work a little bit.

i'd wonder though if there were some way to cut down the module
so it doesn't take quite so much memory.  even halving it would
mean you could ditch swap.

- erik

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Pavel Klinkovsky]

> > I am running the Plan 9 on the single CPU only (old IBM T30).
>
> Sorry, that means we are looking at two different panics.

Yes.
My panic occurs inside 'fault' (search for 'faultarm' string) function of /sys/src/9/port/fault.c.
It seems as an unfixable page fault.

I thought I have a problem with the swap disk region.
I tried 'cat /dev/sdC0/swap | wc -c' but it was ok.

Pavel

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Anthony Martin]

erik quanstrom <quanstro@quanstro.net> once said:
> On Fri Nov  2 09:44:43 EDT 2012, pavel.klinkovsky@gmail.com wrote:
> > > How much memory does your system have?
> >
> > - 512 MB RAM
> > - 512 MB swap
> >
> > > Changeset 14739 grew the Unicode collation tables
> > > in the exp/locale/collate package by a considerable
> > > amount. The compiler's memory usage now goes above
> > > 400 MB when building that package, almost 2.5x the
> > > amount used to compile the second heavyweight and
> > > 15x the average.
> >
> > I see.
> > I can confirm when the kernel panic occured I saw (in stats):
> > - full RAM
> > - small portion of swap occupied

If you want to work around this for the time being,
it's safe to remove that package since it's currently
an experiment and no other package depends on it.

Just 'rm -rf' the exp/locale/collate directory and
run you should be good.

> i'd wonder though if there were some way to cut down the module
> so it doesn't take quite so much memory.  even halving it would
> mean you could ditch swap.

There's a note at the top of the generated tables.go file that says
"TODO: implement more compact representation for sparse blocks".

I'm going to investigate what's causing such high memory usage
in the compiler. I imagine those huge array initializations cause
hundreds of thousands of Node allocations, at the very least.

  Anthony

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[erik quanstrom]

> Just 'rm -rf' the exp/locale/collate directory and
> run you should be good.

it would be sad to see go sucked into the locale debacale.

- erik

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Pavel Klinkovsky]

> i might give the 9front kernel a go.  i think that cinap spent
> some time trying to make swap work a little bit.

Well, actually I prefer to follow the main trunk (bell-labs) version.

Pavel

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Richard Miller]

> i found in nix that when mixing acpi and mp i had to defend against
> double-entries for i/o apics since the numbering can be different.  but
> it will be interesting to see what happened to start 2 cpu1s.

Looks like the same thing: MP table says 2 cpus with apic ids 0 and 1,
ACPI table says 2 cpus (the same ones) with ids 0 and 2.

Mixing information from MP and ACPI tables seems to me like just asking
for trouble.

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Pavel Klinkovsky]

> Just 'rm -rf' the exp/locale/collate directory and
> run you should be good.

Thank you for the hint.
I confirm such work-around helps. ;)

Pavel

Re: [9fans] Kernel panic when allocating a huge memory

[Pavel Klinkovsky]

I prepared the very simple program sequentially allocating 1 MB blocks of memory.
When it reached the end of RAM, the kernel panic occured.

It really seems as a problem with swap. :(

Pavel

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

On Fri Nov  2 12:24:44 EDT 2012, pavel.klinkovsky@gmail.com wrote:
> I prepared the very simple program sequentially allocating 1 MB blocks of memory.
> When it reached the end of RAM, the kernel panic occured.
>
> It really seems as a problem with swap. :(

this is well known, and solutions are available
even if you don't care to use them.

- erik

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[cinap_lenrek]

9front kernel extracts and uses all the cpu and apic and interrupt
routing info from acpi tables if you boot it with *acpi=1. theres
no mixing of mptable and acpi tables.

--
cinap

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[steve]

i believe 8l uses its own storage allocator, but coul easily be tweeked to use malloc and free
by hacking the sources a little (global search and replace), if memory serves me well this
trades memory use for speed, and by going back to traditional malloc in go's 8l you could
reduce the ram use but make it quite a bit slower.

-Steve


On 2 Nov 2012, at 14:00, erik quanstrom <quanstro@quanstro.net> wrote:

> On Fri Nov  2 09:44:43 EDT 2012, pavel.klinkovsky@gmail.com wrote:
>>> How much memory does your system have?
>> 
>> - 512 MB RAM
>> - 512 MB swap
>> 
>>> Changeset 14739 grew the Unicode collation tables
>>> in the exp/locale/collate package by a considerable
>>> amount. The compiler's memory usage now goes above
>>> 400 MB when building that package, almost 2.5x the
>>> amount used to compile the second heavyweight and
>>> 15x the average.
>> 
>> I see.
>> I can confirm when the kernel panic occured I saw (in stats):
>> - full RAM
>> - small portion of swap occupied
> 
> i might give the 9front kernel a go.  i think that cinap spent
> some time trying to make swap work a little bit.
> 
> i'd wonder though if there were some way to cut down the module
> so it doesn't take quite so much memory.  even halving it would
> mean you could ditch swap.
> 
> - erik

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Skip Tavakkolian]

Here's another solution ;)

http://www.crucial.com/store/mpartspecs.aspx?mtbpoid=09E2BA19A5CA7304

On Fri, Nov 2, 2012 at 9:03 AM, Pavel Klinkovsky
<pavel.klinkovsky@gmail.com> wrote:
>> Just 'rm -rf' the exp/locale/collate directory and
>> run you should be good.
>
> Thank you for the hint.
> I confirm such work-around helps. ;)
>
> Pavel
>

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[erik quanstrom]

On Fri Nov  2 13:08:27 EDT 2012, skip.tavakkolian@gmail.com wrote:
> Here's another solution ;)
>
> http://www.crucial.com/store/mpartspecs.aspx?mtbpoid=09E2BA19A5CA7304
>

addressing the actual problem FTW!

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[Pavel Klinkovsky]

>> It really seems as a problem with swap. :(
>
> this is well known, and solutions are available
> even if you don't care to use them.

Oh, does it mean the official Plan 9 distribution contains non-working swap? :O
It is clear I missed something...

Sorry for the noise.

Pavel

Re: [9fans] Kernel panic when compiling Go on native Plan 9

[Pavel Klinkovsky]

> > Here's another solution ;)
> > http://www.crucial.com/store/mpartspecs.aspx?mtbpoid=09E2BA19A5CA7304

Yeah, it really can help!
...until golang guys multiply the compilation memory needs again. ;)

Pavel

Re: [9fans] Kernel panic when allocating a huge memory

[John Floren]

On Fri, Nov 2, 2012 at 10:36 AM, Pavel Klinkovsky
<pavel.klinkovsky@gmail.com> wrote:
>>> It really seems as a problem with swap. :(
>>
>> this is well known, and solutions are available
>> even if you don't care to use them.
>
> Oh, does it mean the official Plan 9 distribution contains non-working swap? :O
> It is clear I missed something...
>
> Sorry for the noise.
>
> Pavel
>

Swap has been broken since at least 2005 (my first experiments with
Plan 9). Once I stopped trying to compile ghostscript on a 32 MB
laptop, I never really had problems with the lack... hell, I did my
master's work and most of my personal computing on a laptop with only
1 GB of RAM and no swap for most of 2010, only ran into problems when
aptitude decided to calculate a multi-gig dependency graph.

Swapping is so painful, please consider buying more RAM. It may be
simple to fix the swap code, if you're inclined to do some kernel
hacking, because the kernel in general is pleasant to work with.


john

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

> Swap has been broken since at least 2005 (my first experiments with
> Plan 9). Once I stopped trying to compile ghostscript on a 32 MB
> laptop, I never really had problems with the lack... hell, I did my
> master's work and most of my personal computing on a laptop with only
> 1 GB of RAM and no swap for most of 2010, only ran into problems when
> aptitude decided to calculate a multi-gig dependency graph.
>
> Swapping is so painful, please consider buying more RAM. It may be
> simple to fix the swap code, if you're inclined to do some kernel
> hacking, because the kernel in general is pleasant to work with.

if you're looking to the future and thinking of dealing with multiple
page sizes, the current swap is going to need rewriting from scratch.

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[Skip Tavakkolian]

it's by design:
http://9fans.net/archive/2006/07/229

-Skip

On Fri, Nov 2, 2012 at 10:36 AM, Pavel Klinkovsky
<pavel.klinkovsky@gmail.com> wrote:
>>> It really seems as a problem with swap. :(
>>
>> this is well known, and solutions are available
>> even if you don't care to use them.
>
> Oh, does it mean the official Plan 9 distribution contains non-working swap? :O
> It is clear I missed something...
>
> Sorry for the noise.
>
> Pavel
>

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

On Fri Nov  2 15:10:00 EDT 2012, skip.tavakkolian@gmail.com wrote:
> it's by design:
> http://9fans.net/archive/2006/07/229
>
> -Skip
>
> On Fri, Nov 2, 2012 at 10:36 AM, Pavel Klinkovsky
> <pavel.klinkovsky@gmail.com> wrote:
> >>> It really seems as a problem with swap. :(
> >>
> >> this is well known, and solutions are available
> >> even if you don't care to use them.
> >
> > Oh, does it mean the official Plan 9 distribution contains non-working swap? :O
> > It is clear I missed something...
> >
> > Sorry for the noise.

i don't think that's quite fair to the current situation.  there
was a swapper, and it's broken.  it should either be fixed or
removed.  leaving the thing in in the state it's in (buffalo
buffalo?) doesn't make any sense, and is as seperate from
the question of whether to page (to disk) or not as, ahem, vm is from
paging (to disk).

imo, swap needs to go.

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

There's a non-trivial chance that what now goes wrong with paging
(which did once work, even if it isn't great) is a symptom of a bug
that afflicts the virtual memory code itself. (For instance, a page unlocked
during a critical period, a race, and so on.)



On 2 November 2012 19:18, erik quanstrom <quanstro@quanstro.net> wrote:

> On Fri Nov  2 15:10:00 EDT 2012, skip.tavakkolian@gmail.com wrote:
> > it's by design:
> > http://9fans.net/archive/2006/07/229
> >
> > -Skip
> >
> > On Fri, Nov 2, 2012 at 10:36 AM, Pavel Klinkovsky
> > <pavel.klinkovsky@gmail.com> wrote:
> > >>> It really seems as a problem with swap. :(
> > >>
> > >> this is well known, and solutions are available
> > >> even if you don't care to use them.
> > >
> > > Oh, does it mean the official Plan 9 distribution contains non-working
> swap? :O
> > > It is clear I missed something...
> > >
> > > Sorry for the noise.
>
> i don't think that's quite fair to the current situation.  there
> was a swapper, and it's broken.  it should either be fixed or
> removed.  leaving the thing in in the state it's in (buffalo
> buffalo?) doesn't make any sense, and is as seperate from
> the question of whether to page (to disk) or not as, ahem, vm is from
> paging (to disk).
>
> imo, swap needs to go.
>
> - erik
>
>

[-- Attachment #2: Type: text/html, Size: 2107 bytes --]

Re: [9fans] Kernel panic when allocating a huge memory

[pmarin]

To be clear, is the swap partition completely useless in Plan9?

pmarin.

On Fri, Nov 2, 2012 at 9:28 PM, Charles Forsyth
<charles.forsyth@gmail.com> wrote:
> There's a non-trivial chance that what now goes wrong with paging
> (which did once work, even if it isn't great) is a symptom of a bug
> that afflicts the virtual memory code itself. (For instance, a page unlocked
> during a critical period, a race, and so on.)
>
>
>
> On 2 November 2012 19:18, erik quanstrom <quanstro@quanstro.net> wrote:
>>
>> On Fri Nov  2 15:10:00 EDT 2012, skip.tavakkolian@gmail.com wrote:
>> > it's by design:
>> > http://9fans.net/archive/2006/07/229
>> >
>> > -Skip
>> >
>> > On Fri, Nov 2, 2012 at 10:36 AM, Pavel Klinkovsky
>> > <pavel.klinkovsky@gmail.com> wrote:
>> > >>> It really seems as a problem with swap. :(
>> > >>
>> > >> this is well known, and solutions are available
>> > >> even if you don't care to use them.
>> > >
>> > > Oh, does it mean the official Plan 9 distribution contains non-working
>> > > swap? :O
>> > > It is clear I missed something...
>> > >
>> > > Sorry for the noise.
>>
>> i don't think that's quite fair to the current situation.  there
>> was a swapper, and it's broken.  it should either be fixed or
>> removed.  leaving the thing in in the state it's in (buffalo
>> buffalo?) doesn't make any sense, and is as seperate from
>> the question of whether to page (to disk) or not as, ahem, vm is from
>> paging (to disk).
>>
>> imo, swap needs to go.
>>
>> - erik
>>
>

Re: [9fans] Kernel panic when allocating a huge memory

[steve]

if you use the 9front distribution swap works,
if you use the either 9atom or the labs distribution it does not.

the labs or erik may take the fixes from 9front but there are good reasons
for dropping swap all together - it is very slow, rarely used,
and ram is cheap these days.

i think this is a fair summary.

-Steve

On 3 Nov 2012, at 05:43, pmarin <pmarin.mail@gmail.com> wrote:

> To be clear, is the swap partition completely useless in Plan9?
> 
> pmarin.
> 
> On Fri, Nov 2, 2012 at 9:28 PM, Charles Forsyth
> <charles.forsyth@gmail.com> wrote:
>> There's a non-trivial chance that what now goes wrong with paging
>> (which did once work, even if it isn't great) is a symptom of a bug
>> that afflicts the virtual memory code itself. (For instance, a page unlocked
>> during a critical period, a race, and so on.)
>> 
>> 
>> 
>> On 2 November 2012 19:18, erik quanstrom <quanstro@quanstro.net> wrote:
>>> 
>>> On Fri Nov  2 15:10:00 EDT 2012, skip.tavakkolian@gmail.com wrote:
>>>> it's by design:
>>>> http://9fans.net/archive/2006/07/229
>>>> 
>>>> -Skip
>>>> 
>>>> On Fri, Nov 2, 2012 at 10:36 AM, Pavel Klinkovsky
>>>> <pavel.klinkovsky@gmail.com> wrote:
>>>>>>> It really seems as a problem with swap. :(
>>>>>> 
>>>>>> this is well known, and solutions are available
>>>>>> even if you don't care to use them.
>>>>> 
>>>>> Oh, does it mean the official Plan 9 distribution contains non-working
>>>>> swap? :O
>>>>> It is clear I missed something...
>>>>> 
>>>>> Sorry for the noise.
>>> 
>>> i don't think that's quite fair to the current situation.  there
>>> was a swapper, and it's broken.  it should either be fixed or
>>> removed.  leaving the thing in in the state it's in (buffalo
>>> buffalo?) doesn't make any sense, and is as seperate from
>>> the question of whether to page (to disk) or not as, ahem, vm is from
>>> paging (to disk).
>>> 
>>> imo, swap needs to go.
>>> 
>>> - erik
>> 

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

On Sat Nov  3 02:54:52 EDT 2012, steve@quintile.net wrote:
> if you use the 9front distribution swap works,
> if you use the either 9atom or the labs distribution it does not.
>
> the labs or erik may take the fixes from 9front but there are good reasons
> for dropping swap all together - it is very slow, rarely used,
> and ram is cheap these days.
>
> i think this is a fair summary.

yup.

9atom is dropping swap since the current setup conflicts
with multiple page sizes, and i haven't run swap on any
machine for 10 years.

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 152 bytes --]

And soldered onto motherboards in many ultrabooks.

On 3 November 2012 06:53, steve <steve@quintile.net> wrote:

> and ram is cheap these days.
>

[-- Attachment #2: Type: text/html, Size: 471 bytes --]

Re: [9fans] Kernel panic when allocating a huge memory

[lucio]

> And soldered onto motherboards in many ultrabooks.
>
> On 3 November 2012 06:53, steve <steve@quintile.net> wrote:
>
>> and ram is cheap these days.
>>
Should that "And" not be a "But"?  There is little cheap about
soldered RAM, if you need to increase it.

++L

PS: I'd be curious to see a mathematical explanation of the semantic
differences between "and" and "but".  Any pointers?

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

On Sat Nov  3 05:17:39 EDT 2012, charles.forsyth@gmail.com wrote:

> And soldered onto motherboards in many ultrabooks.
>
> On 3 November 2012 06:53, steve <steve@quintile.net> wrote:
>
> > and ram is cheap these days.

does plan 9 run on any ultrabooks natively?  swapping within a vm?
my head hurts to think of it.

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

We'll see.

I'll reiterate my original remark that it would be worthwhile tracking down
and fixing the virtual memory or paging bug in sources plan 9,
not necessarily to make use of paging, but to ensure that the paging
problem isn't just a symptom of something else that normally gets by.
If 9front doesn't suffer from it, that must help to narrow it down. (I
thought the relevant virtual memory changes from 9front were in
sources, but evidently not. There are so many variants now I've lost track.

On 3 November 2012 15:40, erik quanstrom <quanstro@quanstro.net> wrote:

> does plan 9 run on any ultrabooks natively?

[-- Attachment #2: Type: text/html, Size: 1075 bytes --]

Re: [9fans] Kernel panic when allocating a huge memory

[Kurt H Maier]

On Sat, Nov 03, 2012 at 11:40:05AM -0400, erik quanstrom wrote:
>
> does plan 9 run on any ultrabooks natively?  swapping within a vm?
> my head hurts to think of it.
>

Your head hurts to think that sometimes extra memory is needed?  On the
VMs I host for people, I partition 256 MB of memory.  This is not enough
to compile python, so you turn on swap before mk, then turn it off again
when you're done.  Or not.  I'm not a cop.

Either way, nobody has died yet, or even complained about headaches.

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

On Sat Nov  3 12:32:38 EDT 2012, khm-9@intma.in wrote:
> On Sat, Nov 03, 2012 at 11:40:05AM -0400, erik quanstrom wrote:
> >
> > does plan 9 run on any ultrabooks natively?  swapping within a vm?
> > my head hurts to think of it.
> >
>
> Your head hurts to think that sometimes extra memory is needed?  On the
> VMs I host for people, I partition 256 MB of memory.  This is not enough
> to compile python, so you turn on swap before mk, then turn it off again
> when you're done.  Or not.  I'm not a cop.
>
> Either way, nobody has died yet, or even complained about headaches.

usually the vm does paging itself, and often more complicated things like
memory compression and deduplication.  so why would the hosted os page as well?

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[hiro]

memory deduplication? is that true?

Re: [9fans] Kernel panic when allocating a huge memory

[Kurt H Maier]

On Sat, Nov 03, 2012 at 12:33:52PM -0400, erik quanstrom wrote:
>
> usually the vm does paging itself, and often more complicated things like
> memory compression and deduplication.  so why would the hosted os page as well?
>

Are you deliberately conflating swapping and paging?

Re: [9fans] Kernel panic when allocating a huge memory

[balaji]

On Sat, Nov 3, 2012 at 9:31 AM, Kurt H Maier <khm-9@intma.in> wrote:
> On Sat, Nov 03, 2012 at 11:40:05AM -0400, erik quanstrom wrote:
>>
>> does plan 9 run on any ultrabooks natively?  swapping within a vm?
>> my head hurts to think of it.
>>
>
> Your head hurts to think that sometimes extra memory is needed?  On the
> VMs I host for people, I partition 256 MB of memory.  This is not enough
> to compile python, so you turn on swap before mk, then turn it off again
> when you're done.  Or not.  I'm not a cop.
>
> Either way, nobody has died yet, or even complained about headaches.
>
This is a good case for swap and i +1 charles's suggestion.

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

On Sat Nov  3 12:52:03 EDT 2012, khm-9@intma.in wrote:
> On Sat, Nov 03, 2012 at 12:33:52PM -0400, erik quanstrom wrote:
> >
> > usually the vm does paging itself, and often more complicated things like
> > memory compression and deduplication.  so why would the hosted os page as well?
> >
>
> Are you deliberately conflating swapping and paging?

in modern systems, i believe they mean the same thing.

http://en.wikipedia.org/wiki/Paging#Terminology

> memory deduplication? is that true?

http://lwn.net/Articles/454795/

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[Matthew Veety]

> 
> does plan 9 run on any ultrabooks natively?  swapping within a vm?
> my head hurts to think of it.
> 
> - erik
> 

I have 9front running natively with some help on my MacBook Air. Ive been working on getting rid of the help.

--
Veety

Re: [9fans] Kernel panic when allocating a huge memory

[Kurt H Maier]

On Sat, Nov 03, 2012 at 01:04:15PM -0400, erik quanstrom wrote:
>
> in modern systems, i believe they mean the same thing.
>
> http://en.wikipedia.org/wiki/Paging#Terminology
>

Sorry, I didn't know you were talking about Windows NT.

> > memory deduplication? is that true?
>
> http://lwn.net/Articles/454795/

hiro was asking if plan 9 deduplicates memory.

Re: [9fans] Kernel panic when allocating a huge memory

[Dan Cross]

On Sat, Nov 3, 2012 at 1:13 PM, Kurt H Maier <khm-9@intma.in> wrote:
> On Sat, Nov 03, 2012 at 01:04:15PM -0400, erik quanstrom wrote:
>> in modern systems, i believe they mean the same thing.
>>
>> http://en.wikipedia.org/wiki/Paging#Terminology
>
> Sorry, I didn't know you were talking about Windows NT.

I didn't know you were talking about VAX Unix.

>> > memory deduplication? is that true?
>>
>> http://lwn.net/Articles/454795/
>
> hiro was asking if plan 9 deduplicates memory.

That's odd, because Erik was pretty obviously talking about the host
virtual machine.

But hey; whatever.  It's cool.

        - Dan C.

Re: [9fans] Kernel panic when allocating a huge memory

[Kurt H Maier]

On Sat, Nov 03, 2012 at 01:22:50PM -0400, Dan Cross wrote:
>
> I didn't know you were talking about VAX Unix.
>

Thanks for letting me know.

>
> That's odd, because Erik was pretty obviously talking about the host
> virtual machine.
>

"Host virtual machine," eh?

> But hey; whatever.  It's cool.
>

Relieved you approve.

Re: [9fans] Kernel panic when allocating a huge memory

[hiro]

Actually even in linux having something one could call "memory
deduplication" surprises me.
Is there a timemachine for memory on macos?

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

On Sat Nov  3 13:14:17 EDT 2012, khm-9@intma.in wrote:
> On Sat, Nov 03, 2012 at 01:04:15PM -0400, erik quanstrom wrote:
> >
> > in modern systems, i believe they mean the same thing.
> >
> > http://en.wikipedia.org/wiki/Paging#Terminology
> >
>
> Sorry, I didn't know you were talking about Windows NT.
>
> > > memory deduplication? is that true?
> >
> > http://lwn.net/Articles/454795/
>
> hiro was asking if plan 9 deduplicates memory.

perhaps my comment about double-swap/paging was not clear
enough.  i was considering the hosted os, with some standard
vm such as esxi, vbox, xen or whatever as the host.  in such a case it makes
no sense to me for the hosted os to page/swap as the hypervisor
is perfectly capable of doing this itself.  in fact, i think having
the guest page/swap while the hypervisor is page/swaping is going
to tend to make things more difficult because of i/o contention,
and the fact that doing i/o tends to temporarly increase memory
use.

on the other hand, if you want to press plan 9 into service as the
hypervisor (has anyone done this?), you are going to need fairly
robust swap/pageing capabilities.  and if you want to get real utility
out of the hypervisor, you're going to need snapshotting and
support for network connection redirection as well.

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[Kurt H Maier]

On Sat, Nov 03, 2012 at 01:40:48PM -0400, erik quanstrom wrote:
> perhaps my comment about double-swap/paging was not clear
> enough.  i was considering the hosted os, with some standard
> vm such as esxi, vbox, xen or whatever as the host.  in such a case it makes
> no sense to me for the hosted os to page/swap as the hypervisor
> is perfectly capable of doing this itself.  in fact, i think having
> the guest page/swap while the hypervisor is page/swaping is going
> to tend to make things more difficult because of i/o contention,
> and the fact that doing i/o tends to temporarly increase memory
> use.

This makes more sense.  However, if your hypervisor is swapping, you've
screwed up your planning.  RAM oversubscription is the reason most
dime-store VPS services suck really badly.  I leave swapping to the
guest OS, since that's where malloc is being called.

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

>
> This makes more sense.  However, if your hypervisor is swapping, you've
> screwed up your planning.  RAM oversubscription is the reason most
> dime-store VPS services suck really badly.  I leave swapping to the
> guest OS, since that's where malloc is being called.

i'm not a big vm guy, but i do know that oversubscription is a big problem.

however, i think that queuing theory in general says that one queue with global
sorting beats n smaller queues with local sorting.  i think this is sometimes called the
checkout-line problem.

other factors, like global knowledge of memory use stats and page duplication
should put the vm in an even better position than general queueing theory
would suggest to make decisions on what pages to move to disk wrt. global
(that is total machine) throughput.

do you have a reference that demonstrates or derives that a similarly-loaded
machine can perform better with all the guests swapping indepdently and
the vm not swapping, rather than preventing the guests from swapping and
letting the vm swap?

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[Kurt H Maier]

On Sat, Nov 03, 2012 at 01:57:10PM -0400, erik quanstrom wrote:
>
> however, i think that queuing theory in general says that one queue with global
> sorting beats n smaller queues with local sorting.  i think this is sometimes called the
> checkout-line problem.
>

This would be true, I think, if the "big queue" had sufficient
visibility into the guests.  They almost never do.

> other factors, like global knowledge of memory use stats and page duplication
> should put the vm in an even better position than general queueing theory
> would suggest to make decisions on what pages to move to disk wrt. global
> (that is total machine) throughput.

Page dedup, at least on linux, is not particularly useful for
virtualization yet.  It can be made to work but I've never seen people
benefit from it unless the guest operating systems are also linux
systems.

> do you have a reference that demonstrates or derives that a similarly-loaded
> machine can perform better with all the guests swapping indepdently and
> the vm not swapping, rather than preventing the guests from swapping and
> letting the vm swap?

I do not have any formal data on the subject; only the things I've seen
"in the field" as it were.  9cloud does not oversubscribe or page out
VMs unless the system is in danger of crashing.  vm.overcommit_memory is
set to 2 and swap is there in case a system process spirals out of
control.  The systems are reasonably performant.  I manage the
hypervisor in this style based on years of seeing other options burn me.

Re: [9fans] Kernel panic when allocating a huge memory

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

local paging algorithms can avoid thrashing: "the process pages against
itself".
global paging algorithms typically do not (invariably do not, in my
experience, but most people use essentially the same one, so there might be
some that worked).

Wilkes has a nice discussion of paging algorithms as an application of
control theory
in "The Dynamics of Paging".
http://comjnl.oxfordjournals.org/content/16/1/4.short

"It is notorious that the use of apparently innocuous scheduling and paging
algorithms can give rise to the type of unstable behaviour known as
thrashing."


On 3 November 2012 17:57, erik quanstrom <quanstro@quanstro.net> wrote:

> other factors, like global knowledge of memory use stats and page
> duplication
> should put the vm in an even better position than general queueing theory
> would suggest to make decisions on what pages to move to disk wrt. global
> (that is total machine) throughput.
>

[-- Attachment #2: Type: text/html, Size: 1633 bytes --]

Re: [9fans] Kernel panic when allocating a huge memory

[hiro]

>> Are you deliberately conflating swapping and paging?
>
> in modern systems, i believe they mean the same thing.
>
> http://en.wikipedia.org/wiki/Paging#Terminology
>
>> memory deduplication? is that true?
>
> http://lwn.net/Articles/454795/

most things I learn about on 9fans (and why I'm happily still
subscribed) is about what "technology" to avoid.

"They used Word to write the spec"

Re: [9fans] Kernel panic when allocating a huge memory

[Steve Simon]

        - Dan C.

Glad to see you survived the storm Dan.

-Steve

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

On Sat Nov  3 14:49:39 EDT 2012, charles.forsyth@gmail.com wrote:

> local paging algorithms can avoid thrashing: "the process pages
> against itself".  global paging algorithms typically do not
> (invariably do not, in my experience, but most people use essentially
> the same one, so there might be some that worked).
>
> Wilkes has a nice discussion of paging algorithms as an application of
> control theory in "The Dynamics of Paging".
> http://comjnl.oxfordjournals.org/content/16/1/4.short
>
> "It is notorious that the use of apparently innocuous scheduling and
> paging algorithms can give rise to the type of unstable behaviour
> known as thrashing."

good point.

however, if running 10 copies of the same os install, a common occurance,
and the hypervisor is consolidating identical pages (usually using cas), the
pages an os is likely to free are likely to be duplicated.  unless they all page
out that page, no  memory is saved unless the hypervisor swaps it out.

vmware makes the same point www.vmware.com/files/pdf/mem_mgmt_perf_vsphere5.pdf
p. 13.  but they also make the case that hat the balloon technique can
outpreform host swapping, p. 20, fig. 12.

(vdi (virtual desktop) would be interesting to graph.  couldn't find that.)

p. 27 best practices, bullet 3 basically says, make sure you have have enough
memory because paging sucks.  :-)

- erik

p.s. sharepoint takes 5 vms to run?  really guys?

Re: [9fans] Kernel panic when allocating a huge memory

[hiro]

that paging sucks is not a very new discovery.

Re: [9fans] Kernel panic when allocating a huge memory

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 313 bytes --]

Except that it's only really free when they all DO page it out, because
otherwise it's in someone's working set, and should remain.

On 4 November 2012 14:02, erik quanstrom <quanstro@quanstro.net> wrote:

>  unless they all page
> out that page, no  memory is saved unless the hypervisor swaps it out.
>

[-- Attachment #2: Type: text/html, Size: 619 bytes --]

Re: [9fans] Kernel panic when allocating a huge memory

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

It needn't. Today, with the price of RAM, as many have noted, it's probably
not worth the code and complexity
(though there needn't be much of either), but it can be designed and made
to work well.

Also, despite the low cost of RAM, the systems still run out: my 3 Gbyte
notebook running Chrome under Linux
can run out. What Linux then does with no paging file is not pleasant, so I
give it a small one, and it still thrashes
trying to reclaim enough memory to continue. Pathetic.

My new 4Gb notebook has the memory soldered in (Samsung Series 9).

On 4 November 2012 14:18, hiro <23hiro@gmail.com> wrote:

> that paging sucks is not a very new discovery.

[-- Attachment #2: Type: text/html, Size: 1256 bytes --]

Re: [9fans] Kernel panic when allocating a huge memory

[Martin Harriss]

On 11/03/2012 02:48 PM, Charles Forsyth wrote:
> local paging algorithms can avoid thrashing: "the process pages against
> itself".
> global paging algorithms typically do not (invariably do not, in my
> experience, but most people use essentially the same one, so there might
> be some that worked).
>
> Wilkes has a nice discussion of paging algorithms as an application of
> control theory
> in "The Dynamics of Paging".
> http://comjnl.oxfordjournals.org/content/16/1/4.short
>
> "It is notorious that the use of apparently innocuous scheduling and
> paging algorithms can give rise to the type of unstable behaviour known
> as thrashing."


Charles,

Regarding local paging algorithms, perhaps at this juncture you should
give yourself a pat on the back and post a pointer to the work you did
applying the EMAS-style paging behaviour to Unix.

(The EMAS papers are a joy to read, even 40 years on.  I'd dig up a
reference but here on the east coast of the US I'm in my 6th day without
power and have other things to worry about right now.)

Martin

Re: [9fans] Kernel panic when allocating a huge memory

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 232 bytes --]

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.51.1897

On 4 November 2012 16:14, Martin Harriss <martin@princeton.edu> wrote:

> post a pointer to the work you did applying the EMAS-style paging
> behaviour to Unix.

[-- Attachment #2: Type: text/html, Size: 642 bytes --]

Re: [9fans] Kernel panic when allocating a huge memory

[tlaronde]

On Sat, Nov 03, 2012 at 06:48:47PM +0000, Charles Forsyth wrote:
> Wilkes has a nice discussion of paging algorithms as an application of
> control theory
> in "The Dynamics of Paging".
> http://comjnl.oxfordjournals.org/content/16/1/4.short
>
> "It is notorious that the use of apparently innocuous scheduling and paging
> algorithms can give rise to the type of unstable behaviour known as
> thrashing."

Just for the (historical) record, the original G.R.A.S.S. team, since
the processing i.e. some kind of sorting of huge data typically raster
may need a lot of memory, went as far as implementing a library in
G.R.A.S.S. to do user level paging and swapping (indeed segmentation and
use of disks but actually file system use to store segments of
processing---the segment library).

Has a "paging / swapping" filesystem (non persistent data, processes dependant
timelife "memory" allocation, with storing/reloading to/from disk, and
use of real memory when available) been attempted?

--
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                      http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [9fans] Kernel panic when allocating a huge memory

[Bakul Shah]

[-- Attachment #1: Type: text/plain, Size: 1187 bytes --]

What is really needed is to have some of your hotshot programmers develop on 3+ year old "average" computers. Their code bloat will reduce when their own productivity suffers due to slow machines.

Still there will be cases where a lean program will run out of memory. It should do its own memory management. For instance, don't try to map in multi GB files in their entirety if you are justvstreaming data!

On Nov 4, 2012, at 6:58 AM, Charles Forsyth <charles.forsyth@gmail.com> wrote:

> It needn't. Today, with the price of RAM, as many have noted, it's probably not worth the code and complexity
> (though there needn't be much of either), but it can be designed and made to work well.
> 
> Also, despite the low cost of RAM, the systems still run out: my 3 Gbyte notebook running Chrome under Linux
> can run out. What Linux then does with no paging file is not pleasant, so I give it a small one, and it still thrashes
> trying to reclaim enough memory to continue. Pathetic.
> 
> My new 4Gb notebook has the memory soldered in (Samsung Series 9).
> 
> On 4 November 2012 14:18, hiro <23hiro@gmail.com> wrote:
> that paging sucks is not a very new discovery.
> 

[-- Attachment #2: Type: text/html, Size: 2442 bytes --]

Re: [9fans] Kernel panic when allocating a huge memory

[tlaronde]

On Sun, Nov 04, 2012 at 05:27:58PM +0100, tlaronde@polynum.com wrote:
>
> Has a "paging / swapping" filesystem (non persistent data, processes dependant
> timelife "memory" allocation, with storing/reloading to/from disk, and
> use of real memory when available) been attempted?
>

Just to add what would be "not existing":

- user level
- IPC since these memory blocks would be named "files".

And as far as the G.R.A.S.S (geographical data processing) was involved,
one could really need lots of "memory"; far more than what was available
(end of eighties); and with increasing data size, problem is still here
(well, with geographical data, one sorting is obvious: by location hence
tiling; problem starts when one has to consider how to weave tile
results, if the result of the whole may depend on non local (twice the
size of the tile) interdependencies).
--
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                      http://www.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [9fans] Kernel panic when allocating a huge memory

[Pavel Klinkovsky]

> there are good reasons for dropping swap all together -
> it is very slow, rarely used, and ram is cheap these days.

It seems true.

But in my case, it is easier, faster and cheaper to reinstall Plan 9 with larger (if working) swap partition than increase the RAM size.
However it is just a theory since official Plan 9 does not have a working swap...

Pavel

Re: [9fans] Kernel panic when allocating a huge memory

[erik quanstrom]

> It seems true.
>
> But in my case, it is easier, faster and cheaper to reinstall Plan 9 with larger (if working) swap partition than increase the RAM size.
> However it is just a theory since official Plan 9 does not have a working swap...

it would be far easier for you to just remove the *experimental* go package.

- erik

Re: [9fans] Kernel panic when allocating a huge memory

[cinap_lenrek]

you dont need a dedicated swap partition. just swap
to a /tmp file.

--
cinap

Re: [9fans] Kernel panic when allocating a huge memory

[Pavel Klinkovsky]

> > But in my case, it is easier, faster and cheaper to reinstall Plan 9 with larger (if working) swap partition than increase the RAM size.
> it would be far easier for you to just remove the *experimental* go package.

Already done, and my work continues... ;)

Pavel

Re: [9fans] Kernel panic when allocating a huge memory

[Pavel Klinkovsky]

> you dont need a dedicated swap partition. just swap to a /tmp file.

Oh, yes.
I forgot such a possibility.

Pavel