Re: [9fans] Threads: Sewing badges of honor onto a Kernel

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Keith Nash]

On Tuesday 02 March 2004 04:56, ron minnich wrote:
> ... v9fs ...
>
> It works. I don't know why we're not all using it.
>
> ron

Can you do a tarball release?  The last one was in 2002.  Most of the sf site (except CVS?) is out of date.

Keith.

[9fans] Threads: Sewing badges of honor onto a Kernel

[dbailey27]

A few of us had a discussion this evening on the state of Linux and
bloat in the Operating System development community. This isn't an
event isolated to this evening, however tonight had a bit of spice.

Andrey gleamed this bit of interesting text on the O'Reilly[1] website.
Let's have a look at a snippet, shall we?

Under the "Kernel Space and User Space" heading:

| ...  Another example of this happened in the Plan 9 operating system.
| They had this really cool system call to do a better process fork--a
| simple way for a program to split itself into two and continue processing
| along both forks. This new fork, which Plan 9 called R-Fork (and SGI later
| called S-Proc) essentially creates two separate process spaces that share
| an address space. This is helpful for threading especially.

| Linux does this too with its clone system call, but it was implemented
| properly. However, with the SGI and Plan9 routines they decided that
| programs with two branches can share the same address space but use
| separate stacks. Normally when you use the same address in both threads,
| you get the same memory location. But you have a stack segment that is
| specific, so if you use a stack-based memory address you actually get
| two different memory locations that can share a stack pointer without
| overriding the other stack.

| While this is a clever feat, the downside is that the overhead in maintaining
| the stacks makes this in practice really stupid to do. They found out too
| late that the performance went to hell. Since they had programs which used
| the interface they could not fix it. Instead they had to introduce an additional
| properly-written interface so that they could do what was wise with the stack
|  space. ...

This commentary from Linus Torvalds on rfork[2] proclaims that a single, unified
stack space for multiple threads is not only beneficial but imperative for proper
thread propagation. What we are wondering is ... why?

What scenario would make a unified stack segment beneficial in a multi-threaded
execution environment. Is this something that should be an optional parameter
on-the-fly in a call to rfork() ? Does clone() in Linux allow you to decide, or, does
it enforce the unified stack segment? Doesn't unification create collisions within
scopes of process namespaces through recursion and non-leaf code vectors?

Since we couldn't figure out the logic behind this Lunix tantra, we thought
trolling the public would be a slick idea. Who has an opinion on (or a defense
for ;-)) this statement?

Also, who is the "They" that "found out too late that the performance went to
hell".

Curious minds are interested.

Don (north_)

References:

[1] http://www.oreilly.com/catalog/opensources/books/linus.html
[2] fork(2)

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Scott Schwartz]

Beware that this mailing list doesn't permit postings
with more than a few cc-s.

  To: 9fans@cse.psu.edu
  cc: torvalds@osdl.org, sionide@beefed.org, fa1th@beefed.org

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[andrey mirtchovski]

the url is wrong -- s/books/book/

also, you omitted the conclusion from the chapter, which is (imho) the best:

| In fifteen years, I expect somebody else to come along and say, hey, I can
| do everything that Linux can do but I can be lean and mean about it because
| my system won't have twenty years of baggage holding it back. They'll say
| Linux was designed for the 386 and the new CPUs are doing the really
| interesting things differently. Let's drop this old Linux stuff. This is
| essentially what I did when creating Linux. And in the future, they'll be
| able to look at our code, and use our interfaces, and provide binary
| compatibility, and if all that happens I'll be happy.

andrey "just fanning the flames" mirtchovski

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[dbailey27]

> the url is wrong -- s/books/book/

Ah, yep. Can't paste over VNC. Thanks, Andrey

full URL is:
http://www.oreilly.com/catalog/opensources/book/linus.html

Don (north_)

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Rob Pike]

this is a peculiar distortion of a peculiar episode.  the original
issue linus
objected to, and rather rudely if i remember right, is that if the
stacks are
distinct, there can be confusing bugs if you pass pointers around that
accidentally happen to refer to stack addresses.  in practice, we never
do this because the thread library makes it all but impossible.  however
this issue really rankled linus,

on the other hand, having a separate piece of address space with
different contents is a powerful idea.  it's called something like
per-thread space, and it's really convenient for many implementation
details in threading.  but the feature is (or was; i really don't know
the
situation any more) hard to provide on linux and in fact the clone
interface used to be really nasty because the two processes returned
from the syscall with two processes sharing the same stack.  i
understand
something has been done to make the code after a clone on linux
easier to write, but when linus was dyspeptic about rfork, the linux
dance was really tricky.

so it's just a detail, a design choice with advantages and disadvantages
either way.  win some, lose some. yet somehow our code
is 'stupid' and linux is 'proper', so i guess the snarkiness endures.

-rob

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[dbailey27]

> so it's just a detail, a design choice with advantages and disadvantages
> either way.  win some, lose some. yet somehow our code
> is 'stupid' and linux is 'proper', so i guess the snarkiness endures.

What I am mainly confused about, is... We're talking about user
context threading. In user context threading we've got the abstractions
of a BSS, Data, Text, Stack, (etc in some cases) to maintain a sense
of program-persistant-object-classification. As a coder, we know
and can specify (for the most part) where these "objects" will end
up.

So the question to me is, why would you *want* a unified stack?
The only state in which this might be relevant is threads maintained
in ASM where data is passed back and forth via Stack and manipulated
via Registers. Otherwise, things just don't make much sense. In a
land of higher level languages, this semantic seems irrational.

This is mainly what I love about Plan 9's threading implementation, that
it unifies memory in the heap, and not at the stack. I agree that this
creates a powerful tool not only in efficiency, but also in reliability.

I see Linus' logic as more of a negation of both efficiency and reliability.

I'm not interested in starting a flame war, but I *am* interested in
understanding the logic, because I just can't see it clearly from my
point of view.

Don (north_)

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[ron minnich]

On Fri, 27 Feb 2004 dbailey27@ameritech.net wrote:

> So the question to me is, why would you *want* a unified stack?

well, you don't. It makes programming a nightmare and you need assembly
goo to seperate the stacks after a fork, else you will be stepping on
each others' stack. As the freebsd users of their official version of
rfork() learned, that meant you hit segv walls very quickly. Ouch.

> I see Linus' logic as more of a negation of both efficiency and reliability.

I don't even understand it.

ron

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[boyd, rounin]

> > So the question to me is, why would you *want* a unified stack?
>
> well, you don't. It makes programming a nightmare and you need assembly
> goo to seperate the stacks after a fork, ...

yup

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Rob Pike]

i agree with you. i think the model, although done for reasons of
expediency, has some advantages.  the split stack provides a
unique and powerful memory abstraction: same address, different
value.  (c.f. the up and mp registers in the kernel.)

> I see Linus' logic as more of a negation of both efficiency and
> reliability.

i don't understand his logic.

-rob

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[dbailey27]

> > I see Linus' logic as more of a negation of both efficiency and
> > reliability.

> i don't understand his logic.

Well, I interpreted his logic that each process, itself, should be the
manager of how data is abstracted into each thread. Further, that
the kernel should not impose an interface on such userspace
processes. When I think in terms of future scalability and dynamic
code design, this *seems* to be a good thing.

However, over time, we've seen that the lack of a conformed
interface has prompted the design of such an interface: pthreads,
etc. So, in the end, the necessity for POSIX compliance (or what ever
compliance you prefer) is perceived as paramount.

Though, this could have been avoided and deemed unnecessary if
these kernels had only implemented an interface in the first place.

I believe the shortsightedness of not imposing a strict type created
too many possibilities. This creates a bigger problem when dealing
with libraries imported into a system with no defined interface, and
you end up with the desire for one.

So, in the end, Plan 9 looks like the better solution. The foresight was
there to see a necessity for conformity where all possible permutations
of a given set tended to yield outcomes meeting at the same axis:
thus an interface.

I duno, that's my two cents, anyway. But, that's dependant on me
getting Linus' logic correct, which is why he was CC'd.

Don (north_)

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[boyd, rounin]

> I believe the shortsightedness of not imposing a strict type created
> too many possibilities. This creates a bigger problem when dealing
> with libraries imported into a system with no defined interface, and
> you end up with the desire for one.

limbo does this right.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Kenji Okamoto]

By the way, it took half a day for me to read plenty of mails during
these three days while I was knocked down by flu...  Yeah, it was
interesting to read those.

Kenji

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[dbailey27]

> also, you omitted the conclusion from the chapter, which is (imho) the best:

I didn't want to paste *too* much into the email ;-)

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Rob Pike]

in case that wasn't clear enough, i have no idea what linus is talking
about when he says we had 'overhead' that made it 'really stupid'
'in practice'.

-rob

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Scott Schwartz]

Al Viro once told me that if Plan 9 had shared libraries, rfork wouldn't
work as well.  Apparently they're concerned about page tables when there
are lots of different combinations of sharing going on.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Bruce Ellis]

i had trouble believing that the screeds of banter were from linus.
maybe he has a ghost writer.  when he did the "i don't write
user programs with threads" and thru in some kernel examples,
with obligatory assembly language results - well i thought
what are these guys discussing? gee, i just write in limbo.
threads are fun, linus, when you don't have to do all that crap
to manage them.  i just use "spawn", the only language support
for threads.  the rest is easy - i'm happy.

correct or challenge on any presummption.

brucee
----- Original Message -----
From: "Rob Pike" <rob@mightycheese.com>
To: <9fans@cse.psu.edu>
Sent: Friday, February 27, 2004 4:28 PM
Subject: Re: [9fans] Threads: Sewing badges of honor onto a Kernel


> in case that wasn't clear enough, i have no idea what linus is talking
> about when he says we had 'overhead' that made it 'really stupid'
> 'in practice'.
>
> -rob

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 438 bytes --]

i didn't say this at the time, but now that you mention inferno, i will.
if someone is seriously worried about TLB flushing on context switches,
likes setting up the MMU pretty much once-for-all,
thinks all threads should share all the address space, and absolutely
disdains user mode, he should be writing in limbo for
native inferno, where all of that is the usual state,
and indeed there is no other.  we're all in it together.

[-- Attachment #2: Type: message/rfc822, Size: 3045 bytes --]

From: "Bruce Ellis" <brucee@chunder.com>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] Threads: Sewing badges of honor onto a Kernel
Date: Sat, 28 Feb 2004 20:38:35 +1100
Message-ID: <006201c3fdde$a6339030$8201a8c0@cc77109e>

i had trouble believing that the screeds of banter were from linus.
maybe he has a ghost writer.  when he did the "i don't write
user programs with threads" and thru in some kernel examples,
with obligatory assembly language results - well i thought
what are these guys discussing? gee, i just write in limbo.
threads are fun, linus, when you don't have to do all that crap
to manage them.  i just use "spawn", the only language support
for threads.  the rest is easy - i'm happy.

correct or challenge on any presummption.

brucee
----- Original Message -----
From: "Rob Pike" <rob@mightycheese.com>
To: <9fans@cse.psu.edu>
Sent: Friday, February 27, 2004 4:28 PM
Subject: Re: [9fans] Threads: Sewing badges of honor onto a Kernel


> in case that wasn't clear enough, i have no idea what linus is talking
> about when he says we had 'overhead' that made it 'really stupid'
> 'in practice'.
>
> -rob

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Bruce Ellis]

yeah, it made no sense.  both the froggie and the ps2 ports
have very static page tables.  and the tlb is always right
after a first access.  private stacks?  every thread.

brucee
----- Original Message -----
From: "Charles Forsyth" <forsyth@terzarima.net>
To: <9fans@cse.psu.edu>
Sent: Saturday, February 28, 2004 10:10 AM
Subject: Re: [9fans] Threads: Sewing badges of honor onto a Kernel


> i didn't say this at the time, but now that you mention inferno, i will.
> if someone is seriously worried about TLB flushing on context switches,
> likes setting up the MMU pretty much once-for-all,
> thinks all threads should share all the address space, and absolutely
> disdains user mode, he should be writing in limbo for
> native inferno, where all of that is the usual state,
> and indeed there is no other.  we're all in it together.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[boyd, rounin]

> correct or challenge on any presummption.

any number between 1 and 9 is a pint!!

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[boyd, rounin]

> i had trouble believing that the screeds of banter were from linus.
> maybe he has a ghost writer.

this is the bigest kernel (sic) in the southern hemisphere.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[ron minnich]

I did the original rfork for freebsd, and I wrote it so I split the stacks
too. It was lovely, since you just rforked and all was shared save the
stack -- no assembly required. When they did the committed version, after
fork, all was shared.  This sucked. It was quite a mess to fix up your
stack so you weren't walking on each other's stack -- required assembly
goo to make it go. I hated it and stopped using it.

So I like plan 9 rfork quite a bit. I think Linus/Linux are wrong.

ron

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[George Michaelson]

what happens on MP systems?

-George

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[ron minnich]

On Fri, 27 Feb 2004, George Michaelson wrote:

> what happens on MP systems?
>


not sure what you're driving at.

ron

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[George Michaelson]

On Thu, 26 Feb 2004 22:59:36 -0700 (MST) ron minnich <rminnich@lanl.gov> wrote:

>On Fri, 27 Feb 2004, George Michaelson wrote:
>
>> what happens on MP systems?
>>
>
>
>not sure what you're driving at.

on an MP system, the forked process/thread has chances to be run on a distinct
CPU underneath. I would have thought the complexity of managing a shared stack
was magnified in that circumstance.

process migration -wise?

-George

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[boyd, rounin]

> > what happens on MP systems?
> >
> not sure what you're driving at.

i do.  they have forgotten their lister and the old (now quick):

    s = splfoo();

    /* critical region */

    splx(s);

usually simplifies things, unlike the lunacy you have to code
on linix >= 2.2 -- i've coded it; it made me sick.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Douglas A. Gwyn]

dbailey27@ameritech.net wrote:
> Under the "Kernel Space and User Space" heading:
[very confused description omitted]
> | While this is a clever feat, the downside is that the overhead in maintaining
> | the stacks makes this in practice really stupid to do. They found out too
> | late that the performance went to hell. Since they had programs which used
> | the interface they could not fix it. Instead they had to introduce an additional
> | properly-written interface so that they could do what was wise with the stack
> |  space. ...
I don't think any of that is correct.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Martin C.Atkins]

If I understand right, I could summarise (one of) Linus's arguments,
as follows:

	The cost of sharing some of VM, but not all (in terms of peformance, due
	to less-visible things like TLB flushes, etc), out-weighs having to
	write a little bit of assembler wrapping the clone system call (which
	only has to be got right once, however horrible it might be).

I haven't seen any argument rebutting this in any way. If it is true,
then surely a performance hit on all (forked?) processes, is more
important than having to shim a system call? If it is not true, then it
would be nice to know!

Tell me where I'm wrong, please?

Martin

--
Martin C. Atkins			martin@parvat.com
Parvat Infotech Private Limited		http://www.parvat.com{/,/martin}

RE: [9fans] Threads: Sewing badges of honor onto a Kernel

[Nigel Roles]

9fans-admin@cse.psu.edu wrote:
> If I understand right, I could summarise (one of) Linus's arguments,
> as follows:
>
> 	The cost of sharing some of VM, but not all (in terms of peformance,
> 	due to less-visible things like TLB flushes, etc), out-weighs having
> 	to write a little bit of assembler wrapping the clone system call
> 	(which only has to be got right once, however horrible it might be).
>
> I haven't seen any argument rebutting this in any way. If it is true,
> then surely a performance hit on all (forked?) processes, is more
> important than having to shim a system call? If it is not true, then
> it would be nice to know!
>
> Tell me where I'm wrong, please?
>

The performance argument may well still be regarded by Linus as
stronger, but there are other differences. One is that the stack
used by the clone, being allocated on the heap, is fixed in size,
and unprotected from overflow.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[viro]

On Sat, Feb 28, 2004 at 09:44:24AM -0000, Nigel Roles wrote:

> The performance argument may well still be regarded by Linus as
> stronger, but there are other differences. One is that the stack
> used by the clone, being allocated on the heap, is fixed in size,
> and unprotected from overflow.

clone() uses whatever you pass to it; man 2 mmap for further inspiration...

RE: [9fans] Threads: Sewing badges of honor onto a Kernel

[Nigel Roles]

9fans-admin@cse.psu.edu wrote:
> On Sat, Feb 28, 2004 at 09:44:24AM -0000, Nigel Roles wrote:
>
>> The performance argument may well still be regarded by Linus as
>> stronger, but there are other differences. One is that the stack
>> used by the clone, being allocated on the heap, is fixed in size,
>> and unprotected from overflow.
>
> clone() uses whatever you pass to it; man 2 mmap for further
> inspiration...

To paraphrase Captain Mainwaring, "I wondered if you'd spot that".
Whilst writing the email, it occurred to me that you could
probably pull stunts with mmap/mremap and catching signals,
and get what is wanted.

We are now composing several system calls in some moderately
clever ways to get a behaviour which, whilst not equivalent to
rfork() is as flexible. It's not exactly obvious though is it?
I think I would expect to see a helper function.

Nice to see you on the list again Al.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[boyd, rounin]

> To paraphrase Captain Mainwaring, "I wondered if you'd spot that".

just "don't panic, don't panic"!!

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[viro]

On Sat, Feb 28, 2004 at 01:40:48PM -0000, Nigel Roles wrote:
> Whilst writing the email, it occurred to me that you could
> probably pull stunts with mmap/mremap and catching signals,
> and get what is wanted.

mmap(2)
       MAP_GROWSDOWN
              Used for stacks. Indicates to the kernel VM system that the map-
              ping should extend downwards in memory.

See mm/mmap.c:expand_stack() for details.

So no signals involved and that's precisely what is used for stack anyway -
anonymous VM_GROWSDOWN mapping.

Note: from my reading of the calc_vm_flag_bits() there is a problem on
parisc, what with the stack growing up - no way to get VM_GROWSUP from
mmap() and that's what would be needed to act as stacks there.  For that
matter, parisc do_page_fault() looks fishy in that area, but I'm not
familiar enough with that platform.

RE: [9fans] Threads: Sewing badges of honor onto a Kernel

[Nigel Roles]

9fans-admin@cse.psu.edu wrote:
> Subject: Re: [9fans] Threads: Sewing badges of honor onto a Kernel
>
>
> On Sat, Feb 28, 2004 at 01:40:48PM -0000, Nigel Roles wrote:
>> Whilst writing the email, it occurred to me that you could
>> probably pull stunts with mmap/mremap and catching signals,
>> and get what is wanted.
>
> mmap(2)
>        MAP_GROWSDOWN
>               Used for stacks. Indicates to the kernel VM system that
>               the map- ping should extend downwards in memory.
>
> See mm/mmap.c:expand_stack() for details.
>
> So no signals involved and that's precisely what is used for stack
> anyway - anonymous VM_GROWSDOWN mapping.
>
> Note: from my reading of the calc_vm_flag_bits() there is a problem on
> parisc, what with the stack growing up - no way to get VM_GROWSUP from
> mmap() and that's what would be needed to act as stacks there.  For
> that matter, parisc do_page_fault() looks fishy in that area, but I'm
> not familiar enough with that platform.

Fair point; I had missed the utility of MAP_ANON.

So there it is, Martin. Near enough the same semantics.
It's not pretty, and not obvious, but equivalent. With a helper
function it could be transparent to the user level (i.e. as
easy to deploy as rfork()), and apparently more MMU efficient.
I say apparently because I don't feel qualified to judge.

The question is, does this make Linux 'right' and Plan 9 'wrong',
which was quite a bit of what touched off the thread?

Neither, I would have thought.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[viro]

On Sat, Feb 28, 2004 at 06:16:50PM -0000, Nigel Roles wrote:

> The question is, does this make Linux 'right' and Plan 9 'wrong',
> which was quite a bit of what touched off the thread?
>
> Neither, I would have thought.

<shrug>

There is only one way to figure out and it had been described upthread.
For Linux such experiments had been done and results were very clear -
price of TLB flushes was considerable and that's aside of the complexity
of supporting a lot of mappings with partial VM sharing.  For Plan 9 the
answer might be different, might be the same, but there's no way to
pull that answer out of thin air.  FWIW, I suspect that full-shared
VM would be a win, but it can only be decided by trying and comparing.
End of story, unless somebody has time for that work and is willing to
do it.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Charles Forsyth]

>>For Linux such experiments had been done and results were very clear -
>>price of TLB flushes was considerable and that's aside of the complexity
>>of supporting a lot of mappings with partial VM sharing.  For Plan 9 the

were those real applications or a synthetic test?  i'm curious what it actually did.
sorry, wrong question.  can you please point me to the paper and/or file that might answer them?
i wonder about the claimed `complexity'.  then again, i've seen signal.h
not to mention sys/types.h

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Rob Pike]

>>> For Linux such experiments had been done and results were very clear
>>> -
>>> price of TLB flushes was considerable and that's aside of the
>>> complexity
>>> of supporting a lot of mappings with partial VM sharing.  For Plan 9
>>> the
>
> were those real applications or a synthetic test?  i'm curious what it
> actually did.
> sorry, wrong question.  can you please point me to the paper and/or
> file that might answer them?
> i wonder about the claimed `complexity'.  then again, i've seen
> signal.h
> not to mention sys/types.h

mike burrows and i talked about this yesterday and concluded that there
is certainly a cost for plan 9's model, since you must, depending on the
MMU, do at least one of:
	- share the TLB among >1 task ids
	- fault in some MMU entries from memory after a context switch,
	  since the root of the MMU tree must be different for each process.
however, there are two mitigating factors.  first, the cost of sharing
the
TLB will be very program dependent.  sometimes it will hurt, but often
it will not.  for typical plan 9 programs with most of the non-main
procs
just doing an i/o loop, the cost will be minimal.  for a program that's
a multithreaded memset or its equivalent, it's probably a measurable
factor but not deal-breaking except in unrealistic pathological
situations.  in any case, this issue
matters more on MIPS-like MMUs than on the x86, i believe.

second, the total
overhead to fault in the TLB path to the pages you're using after a
context switch is something like a couple of microseconds in current
processors, perhaps as high as 10 microseconds for a very bad and
unusual case.  if those few microseconds are a performance issue for
your
program, you should be pretty happy.

if performance is your dominant concern, you could turn these points
into
a condemnation of plan 9's model.  i've never been in that position,
though.  for me, performance is but one factor in the figure of merit
of a system and is often offset by other factors.

-rob

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[viro]

On Sat, Feb 28, 2004 at 07:44:31PM +0000, Charles Forsyth wrote:
> >>For Linux such experiments had been done and results were very clear -
> >>price of TLB flushes was considerable and that's aside of the complexity
> >>of supporting a lot of mappings with partial VM sharing.  For Plan 9 the
> 
> were those real applications or a synthetic test?  i'm curious what it actually did.
> sorry, wrong question.  can you please point me to the paper and/or file that might answer them?

Hmm...  Probably Ingo Molnar would have all details.  IIRC, that was on real
applications, but back then I was dealing with filesystem side of things and
not much else, so that's second-hand information.  I can ask him for details
if you want them; I certainly agree that e.g. presense of shared libraries
changes the picture, so those results do not apply directly to Plan 9 and if
somebody really cares they should try and compare for normal Plan 9 workloads.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[dbailey27]

> Hmm...  Probably Ingo Molnar would have all details.  IIRC, that was on real
> applications, but back then I was dealing with filesystem side of things and
> not much else, so that's second-hand information.  I can ask him for details
> if you want them; 

Yes, please.

Don

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[boyd, rounin]

> The performance argument may well still be regarded by Linus as
> stronger, but there are other differences.

get real, we're no longer using 1 MIP VAXes, where TLB flushes
etc were a real problem.  128 users on an 11/780, anyone?

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Douglas A. Gwyn]

boyd, rounin wrote:
> get real, we're no longer using 1 MIP VAXes, where TLB flushes
> etc were a real problem.  128 users on an 11/780, anyone?

There are a large number of platforms where there are
important constraints imposed by the MMU architecture.
Consider what it means when an OS requires such resources
that it cannot feasibly be implemented on a VAX, yet does
almost nothing more than could have readily been done
(with a different design) on a VAX.  It's nearly as bad
as a 1GHz machine getting so bogged down with GUI code
and poor interface design that some common tasks take
longer to accomplish than on a 1MHz PDP-11 running Unix.
(Whose "Law" was it that said that file storage always
expands to fill whatever disk resources are available?
The same thing seems to apply to CPU cycles.)

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Geoff Collyer]

from a fortune file:

	The steady state of disks is full.  --Ken Thompson

I believe it was Mike O'Dell who asked (approximately) `Why is it that
despite processors getting faster, there never seems to be any
additional cycles available to me?'.  His answer was window systems
and networks, among others.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Joel Salomon]

Douglas A. Gwyn said:
> (Whose "Law" was it that said that file storage always
> expands to fill whatever disk resources are available?
> The same thing seems to apply to CPU cycles.)

I've seen it quoted: "Intel giveth, Microsoft taketh away."

Is interesting to look at various open-source projects and see which ones
get larger & slower with newer versions (gcc, KDE, linux gets larger, but
if Linus is to be believed, they *are* getting faster) and those that
*decrease* in size -- any examples, anyone? Gnome, I *think*, but that's
just because they started of with infinite bloat, nowhere to go but
smaller/faster ;-)

--Joel

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[Douglas A. Gwyn]

Nigel Roles wrote:
> The performance argument may well still be regarded by Linus as
> stronger, but there are other differences. One is that the stack
> used by the clone, being allocated on the heap, is fixed in size,
> and unprotected from overflow.

That would be a serious flaw on a system with a small address space.
It's problematic anyway in its inefficient use of PTEs for the
process, since far more table entries are needed (extra stack
for each thread).

However, overflow detection can in principle be achieved by
mapping all pages of each thread's private stack region except
for the last page, which allows the MMU to flag any overflow.

Re: [9fans] Threads: Sewing badges of honor onto a Kernel

[David Presotto]

The overhead on fork/exec of having to copy the stack descriptors
into the forked process is pretty minimal compared to the exec.

If you are just forking without execing then getting a new
stack is what you want.

If you are forking and sharing memory twixt the two forked processes,
we do indeed take a hit every time the processes context switch
especially on the x86 where we lose the previous tlb context as
soon as we putcr3().  If the two processes shared the TLB state
(pid on mips, page table on x86) we'ld be able to avoid that.
Not having any private segments would allow us to do it.

If you are serious about caring, throw another bit into rfork
that says dump the stack segment.  You'll also have to find
someplace in all architectures to hide the pointer to the
thread private memory.  Add to the kernel support for sharing
the TLB state.  Then measure the two and tell us how much you
saved in various programs.  If its non-neglibigle, you'll
have a reason argument instead of this endless whining back
and forth.