[9fans] protection against resource exhaustion

[9fans] protection against resource exhaustion

[arisawa]

Hello 9fans

my mac has a protection below:
-bash$ cat foo
#!/bin/sh
./foo
-bash$ ./foo
./foo: fork: Resource temporarily unavailable
-bash$

on the other hand, Plan 9 does not.
kernel is protected against such programs, however they are not killed.
therefore no new process can be created.

does anyone have idea to fix the problem?

Kenji Arisawa

Re: [9fans] protection against resource exhaustion

[mischief]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

You would need to implement something like ulimits to fix process exhaustion, or some kind of rate limit. There are several forms of resource exhaustion in plan 9. One other that comes to mind is cat /dev/random prevents other clients from making progress reading /dev/random, such as cpu.

On January 24, 2015 10:16:32 PM PST, arisawa <arisawa@ar.aichi-u.ac.jp> wrote:
>
>Hello 9fans
>
>my mac has a protection below:
>-bash$ cat foo
>#!/bin/sh
>./foo
>-bash$ ./foo
>./foo: fork: Resource temporarily unavailable
>-bash$ 
>
>on the other hand, Plan 9 does not.
>kernel is protected against such programs, however they are not killed.
>therefore no new process can be created.
>
>does anyone have idea to fix the problem?
>
>Kenji Arisawa

[-- Attachment #2: Type: text/html, Size: 1080 bytes --]

Re: [9fans] protection against resource exhaustion

[arisawa]

thank you, mischief.

considering after I have posted previous mail, 
I have come to think that the script below should fail with error and stop execution.

maia% cat foo
#!/bin/rc
./foo
maia% 

current rc does not stop execution.
probably not due to protection problem by kernel.
there must be a bug in rc.

Kenji Arisawa

> 2015/01/25 15:16、arisawa <arisawa@ar.aichi-u.ac.JP> のメール：
> 
> 
> Hello 9fans
> 
> my mac has a protection below:
> -bash$ cat foo
> #!/bin/sh
> ./foo
> -bash$ ./foo
> ./foo: fork: Resource temporarily unavailable
> -bash$ 
> 
> on the other hand, Plan 9 does not.
> kernel is protected against such programs, however they are not killed.
> therefore no new process can be created.
> 
> does anyone have idea to fix the problem?
> 
> Kenji Arisawa
> 
> 

Re: [9fans] protection against resource exhaustion

[Bence Fábián]

[-- Attachment #1: Type: text/plain, Size: 1046 bytes --]

>there must be a bug in rc.

Or maybe it does tail call elimination :)


2015-01-25 10:04 GMT+01:00 arisawa <arisawa@ar.aichi-u.ac.jp>:

> thank you, mischief.
>
> considering after I have posted previous mail,
> I have come to think that the script below should fail with error and stop
> execution.
>
> maia% cat foo
> #!/bin/rc
> ./foo
> maia%
>
> current rc does not stop execution.
> probably not due to protection problem by kernel.
> there must be a bug in rc.
>
> Kenji Arisawa
>
> > 2015/01/25 15:16、arisawa <arisawa@ar.aichi-u.ac.JP> のメール：
> >
> >
> > Hello 9fans
> >
> > my mac has a protection below:
> > -bash$ cat foo
> > #!/bin/sh
> > ./foo
> > -bash$ ./foo
> > ./foo: fork: Resource temporarily unavailable
> > -bash$
> >
> > on the other hand, Plan 9 does not.
> > kernel is protected against such programs, however they are not killed.
> > therefore no new process can be created.
> >
> > does anyone have idea to fix the problem?
> >
> > Kenji Arisawa
> >
> >
>
>
>

[-- Attachment #2: Type: text/html, Size: 1601 bytes --]

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

On Sat Jan 24 23:16:07 PST 2015, mischief@9.offblast.org wrote:

> You would need to implement something like ulimits to fix process
> exhaustion, or some kind of rate limit.  There are several forms of
> resource exhaustion in plan 9.  One other that comes to mind is cat
> /dev/random prevents other clients from making progress reading
> /dev/random, such as cpu.

this is the same route as oom killer, which can be depended on to kill
exactly the wrong thing at exactly the wrong time.  fixing things to be
oomkiller-proof tends to be an exercise in wack-a-mole.

about tail-calls mentioned elsewhere.  rc simple executes the file through
a system call and does not inspect its shebang line.  so there is no possibility
of optimization; rc doesn't know it's an rc script.  (and this problem
doesn't depend on the executible recursively executing itself being an rc script.)

i have eliminated the resrcwait() in the process allocation loop in my kernels.
this sounds much worse than it is.  the advantage is it breaks the loop and gets
the machine running again in a bounded amount of time.  the disadvantage
is one loses program state.  i've found it a win over all.  instead of getting paged
in the middle of the night, a few days later someone might mention that the
machine rebooted.  the logs are almost always intact, so debugging the issue
isn't too hard /during the middle of the day/.  :-)

- erik

Re: [9fans] protection against resource exhaustion

[arisawa]

Hello,

I have been playing the following program.
I tried on official plan9, 9front, and 9atom.
none of them showed messages that come from:
		sysfatal("fork: %r”);
		sysfatal("exec: %r");
I suspect that
	fork()
does not return -1 even if it failed in creating new process.

Be ware this program may cause system panic.

#include <u.h>
#include <libc.h>
#define ERRLEN 256

static int
waitfor(int pid, char *msg)
{
	Waitmsg *w;
	while((w = wait()) != nil){
		if(w->pid == pid){
			strncpy(msg, w->msg, ERRMAX);
			free(w);
			return 0;
		}
		free(w);
	}
	return -1;
}

int
run(char *path, char *cmd)
{
	int pid;
	int status;
	int n;
	char *args[32];
	char msg[ERRLEN];

	n = tokenize(cmd, args, 32);
	args[n] = nil;

	switch(pid = fork()) {/* assign = */
	case -1:
		sysfatal("fork: %r");
	case 0:
		close(0);
		exec(path, args);
		sysfatal("exec: %r");
	default:
		break;
	}
	status = waitfor(pid, msg);
	if(status < 0){
		werrstr("waitfor: %r");
		return -1;
	}
	return 0;
}

void main(int argc, char *argv[])
{	char *e;
	int n;
	int m = 100;
	char buf[32];
	ARGBEGIN{
	case 'm':	m = atoi(ARGF());
		break;
	default: sysfatal("usage");
	}ARGEND

	if(argv[0])
		n = atoi(argv[0]);
	else
		n = 0;
	if(n == m)
		sysfatal("stop");

	print("%d\n",n);
	snprint(buf,sizeof(buf),"8.out -m %d %d",m,n + 1);
	run("./8.out", buf);
	exits(nil);
}

Re: [9fans] protection against resource exhaustion

[cinap_lenrek]

yes, look at newproc() in port/proc.c

the resrcwait() call in there is what erik was refering to
that makes rfork(RFPROC) sleep until a process gets available
instead of failing.

--
cinap

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

On Mon Jan 26 03:43:36 PST 2015, arisawa@ar.aichi-u.ac.jp wrote:
> Hello,
> 
> I have been playing the following program.
> I tried on official plan9, 9front, and 9atom.
> none of them showed messages that come from:
> 		sysfatal("fork: %r”);
> 		sysfatal("exec: %r");
> I suspect that
> 	fork()
> does not return -1 even if it failed in creating new process.

exactly.

the reason it never returns -1, is that introduces a new failure case in
every program running, and testing seems like a huge pain.  just
testing for "handles fork returning -1" is not good enough, it has to have
a reasonable strategy for deailing with no procs.  this is hard to get right.
just reboot.  i have never been able to recover a system that hit noprocs.
i ended up wasting people's time trying.

- erik

ps.  how may lines like this have you seen.

	p = malloc(n);
	if(p == nil){
		...
	}

is this really useful in most tools?

Re: [9fans] protection against resource exhaustion

[arisawa]

Hello,

many of resource exhaustion come from careless programming.
I would like such processes to be killed immediately.
throwing up to broken state might be better.
I prefer an option to plan9.ini that enable resrcwait(), because the call breaks
traditional programming style and probably we need more time to have 
right solution.

> 2015/01/26 23:13、erik quanstrom <quanstro@quanstro.net> のメール：
> 
> On Mon Jan 26 03:43:36 PST 2015, arisawa@ar.aichi-u.ac.jp wrote:
>> Hello,
>> 
>> I have been playing the following program.
>> I tried on official plan9, 9front, and 9atom.
>> none of them showed messages that come from:
>> 		sysfatal("fork: %r”);
>> 		sysfatal("exec: %r");
>> I suspect that
>> 	fork()
>> does not return -1 even if it failed in creating new process.
> 
> exactly.
> 
> the reason it never returns -1, is that introduces a new failure case in
> every program running, and testing seems like a huge pain.  just
> testing for "handles fork returning -1" is not good enough, it has to have
> a reasonable strategy for deailing with no procs.  this is hard to get right.
> just reboot.  i have never been able to recover a system that hit noprocs.
> i ended up wasting people's time trying.
> 
> - erik
> 
> ps.  how may lines like this have you seen.
> 
> 	p = malloc(n);
> 	if(p == nil){
> 		...
> 	}
> 
> is this really useful in most tools?
> 

Re: [9fans] protection against resource exhaustion

[Lyndon Nerenberg]

[-- Attachment #1: Type: text/plain, Size: 675 bytes --]


On Jan 26, 2015, at 4:33 PM, arisawa <arisawa@ar.aichi-u.ac.jp> wrote:

> many of resource exhaustion come from careless programming.
> I would like such processes to be killed immediately.
> throwing up to broken state might be better.
> I prefer an option to plan9.ini that enable resrcwait(), because the call breaks
> traditional programming style and probably we need more time to have 
> right solution.

Not plan9.ini, as that's intel specific.  But perhaps a global runtime switch to turn this on and off (something under /dev/cpu*?), and a corresponding switch in the /proc/*/ctl interface that would let individual processes say what's best for them?


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 817 bytes --]

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

> > many of resource exhaustion come from careless programming.
> > I would like such processes to be killed immediately.
> > throwing up to broken state might be better.
> > I prefer an option to plan9.ini that enable resrcwait(), because the call breaks
> > traditional programming style and probably we need more time to have
> > right solution.
>
> Not plan9.ini, as that's intel specific.  But perhaps a global runtime switch to turn this on and off (something under /dev/cpu*?), and a corresponding switch in the /proc/*/ctl interface that would let individual processes say what's best for them?

plan9.ini is used by all my machines that pxe boot, though the name is
changed to /cfg/pxe/$ea.  there are other config mechanisms, but they
all rely on bootstrapping variables into #ec.  we don't need to invent
a new mechanism, even if we do not use plan9.ini.

- erik

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

> many of resource exhaustion come from careless programming.
> I would like such processes to be killed immediately.
> throwing up to broken state might be better.

at the danger of repeating myself, the linux oom killer fills a similar role.  its job
is to try to identify the process that caused the box to run out of memory and kill
it.  it's a dasterdly beast that's wrong alot.  this isn't because it isn't thoughtfully and
well built, it's because the "right" program to kill is not expressed by the system.
the oom killer has to guess.

i think it will go the same way with fork protection.  how do you tell which program
is at fault?  how do you tell a program forking at high frequency, with short lived
children from a fork bomb?  (such as a busy web server.)

> I prefer an option to plan9.ini that enable resrcwait(), because the call breaks
> traditional programming style and probably we need more time to have
> right solution.

i'm not sure i understand what you mean by "traditional programming style" here
as plan 9 exists in part to break unix rules.

- erik

Re: [9fans] protection against resource exhaustion

[arisawa]

Hello,

> i think it will go the same way with fork protection.  how do you tell which program
> is at fault?  how do you tell a program forking at high frequency, with short lived
> children from a fork bomb?  (such as a busy web server.)

only system administrator knows which processes should keep running.
therefore, as Lyndon mentioned, we need a mark “don’t kill by resource exhaustion” to processes.
if automatic determination is desired, the last stage of /rc/bin/cpurc and /rc/bin/termrc may be the right place.

> i'm not sure i understand what you mean by "traditional programming style" here
> as plan 9 exists in part to break unix rules.

as Eric mentioned, we have many many codes such as
	p = malloc(n);
	if(p == nil){
		...
	}
or
	switch(pid = fork()) {/* assign = */
	case -1:
		sysfatal("fork: %r");
	case 0:
		...
	default:
		...
	}
I have beeb writing codes believing those error return is working.

Re: [9fans] protection against resource exhaustion

[Ori Bernstein]

Too complex to use, especially since processes come and go regularly.

IMO, instead of killing processes, it would be better to keep the hanging
behavior in place, but place limits on the resources used by a subsection
of the process tree. Think ulimit, but applying to entire process heirarchies.

On Tue, 27 Jan 2015 16:03:16 +0900
arisawa <arisawa@ar.aichi-u.ac.jp> wrote:

> Hello,
> 
> > i think it will go the same way with fork protection.  how do you tell which program
> > is at fault?  how do you tell a program forking at high frequency, with short lived
> > children from a fork bomb?  (such as a busy web server.)
> 
> only system administrator knows which processes should keep running.
> therefore, as Lyndon mentioned, we need a mark “don’t kill by resource exhaustion” to processes.
> if automatic determination is desired, the last stage of /rc/bin/cpurc and /rc/bin/termrc may be the right place.
> 
> > i'm not sure i understand what you mean by "traditional programming style" here
> > as plan 9 exists in part to break unix rules.
> 
> as Eric mentioned, we have many many codes such as
> 	p = malloc(n);
> 	if(p == nil){
> 		...
> 	}
> or
> 	switch(pid = fork()) {/* assign = */
> 	case -1:
> 		sysfatal("fork: %r");
> 	case 0:
> 		...
> 	default:
> 		...
> 	}
> I have beeb writing codes believing those error return is working.
> 
> 


-- 
Ori Bernstein <ori@eigenstate.org>

Re: [9fans] protection against resource exhaustion

[lucio]

> I have beeb writing codes believing those error return is working.

And so you should.  Just because an error condition does not have a
clean resolution, does not mean it should not be reported.  Or, for
that matter, that one should not be assisted in figuring out whether
an operation was successful ot not.

I don't think Erik is encouraging not reporting error conditions, I
hope he's merely pointing out that an unreasonable amount of code
doesn't.  It's easy to confude cause and effect, in these situations.

Lucio.

Re: [9fans] protection against resource exhaustion

[lucio]

> IMO, instead of killing processes, it would be better to keep the hanging
> behavior in place, but place limits on the resources used by a subsection
> of the process tree. Think ulimit, but applying to entire process heirarchies.

Once you introduce limits, you become responsible for applying them
wisely.  Good luck!

Lucio.

Re: [9fans] protection against resource exhaustion

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 192 bytes --]

On 27 January 2015 at 04:22, erik quanstrom <quanstro@quanstro.net> wrote:

> this isn't because [linux oom killer] isn't thoughtfully and
> well built,
>

it's actually fairly stupid.

[-- Attachment #2: Type: text/html, Size: 587 bytes --]

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

On Tue Jan 27 02:49:44 PST 2015, charles.forsyth@gmail.com wrote:

> On 27 January 2015 at 04:22, erik quanstrom <quanstro@quanstro.net> wrote:
>
> > this isn't because [linux oom killer] isn't thoughtfully and
> > well built,
> >
>
> it's actually fairly stupid.

the proof that a non-stupid one could be written for me would be in the writing.
there are a lot of smart people who care about the matter not doing this.  :-)

- erik

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

> IMO, instead of killing processes, it would be better to keep the hanging
> behavior in place, but place limits on the resources used by a subsection
> of the process tree. Think ulimit, but applying to entire process heirarchies.

doesn't solve the issues of every program needing to deal with failed forks.
wouldn't you just love it if a programming error were to put acme in a wierd
state?  the machine might as well reboot, since you are likely the only user,
and the editor is wack.

- erik

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

> > i think it will go the same way with fork protection.  how do you tell which program
> > is at fault?  how do you tell a program forking at high frequency, with short lived
> > children from a fork bomb?  (such as a busy web server.)
>
> only system administrator knows which processes should keep running.

do you wake him up in the middle of the night if this happens to arbitrate?
this knowledge of what should be preserved may only be post facto knowledge.
"i'll know what to kill off once i see what's running."  which assumes a working
fork, at least for the administrator.

in any event, i'd be interested in code that does do a good job, especially
if it passes tests other than the trivial fork bomb, such as many users contributing
to exhaustion.

> I have beeb writing codes believing those error return is working.

do you have tests?  did you write a test malloc that will fail when called
at every location, and ensure sane behavior?

- erik

Re: [9fans] protection against resource exhaustion

[arisawa]

we don’t have perfect solution.
nevertheless, we must protect system.

if we search ideal (or nearly ideal) solution, we should assign limited resource to each user.
however this is a big job, I believe.

current plan9 system is running under shared resource model.
under this model, it is very hard to protect system from evil-minded users.

keeping this model, we can do something that is, of course, imperfect (but easy to implement, I believe).
for example:
(a) select processes that should keep running. (with resrcwait flag, for example)
(b) kill processe that failed to be allocated resource if it doesn’t has resrcwait flag.

this strategy has following problems:
(1) innocent processes may be killed.
the probability is small if the origin is careless program, but can be large by evil-mined program.
(2) error return from malloc() and fork() are disabled.

> 2015/01/27 23:10、erik quanstrom <quanstro@quanstro.net> のメール：
> 
>>> i think it will go the same way with fork protection.  how do you tell which program
>>> is at fault?  how do you tell a program forking at high frequency, with short lived
>>> children from a fork bomb?  (such as a busy web server.)
>> 
>> only system administrator knows which processes should keep running.
> 
> do you wake him up in the middle of the night if this happens to arbitrate?
> this knowledge of what should be preserved may only be post facto knowledge.
> "i'll know what to kill off once i see what's running."  which assumes a working
> fork, at least for the administrator.
> 
> in any event, i'd be interested in code that does do a good job, especially
> if it passes tests other than the trivial fork bomb, such as many users contributing
> to exhaustion.
> 
>> I have beeb writing codes believing those error return is working.
> 
> do you have tests?  did you write a test malloc that will fail when called
> at every location, and ensure sane behavior?
> 
> - erik
> 

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

On Tue Jan 27 16:06:49 PST 2015, arisawa@ar.aichi-u.ac.jp wrote:
> we don’t have perfect solution.
> nevertheless, we must protect system.

why does limiting forks "protect the system"? why must be "protect the system"?
and what does that phrase mean in this context?

> if we search ideal (or nearly ideal) solution, we should assign limited resource to each user.
> however this is a big job, I believe.
> 
> current plan9 system is running under shared resource model.
> under this model, it is very hard to protect system from evil-minded users.

plan 9 has no hope against malicious users.  they can fill up your disk, or
use all your memory, too.  i believe the quote attributed to presotto is
"we don't have quotas.  ken just yells at anyone who hogs the jukebox."

nonetheless, i have experience running multi-user plan 9 systems, and users
were not usually the issue.

> keeping this model, we can do something that is, of course, imperfect (but easy to implement, I believe).
> for example:
> (a) select processes that should keep running. (with resrcwait flag, for example)
> (b) kill processe that failed to be allocated resource if it doesn’t has resrcwait flag.
> 
> this strategy has following problems:
> (1) innocent processes may be killed.
> the probability is small if the origin is careless program, but can be large by evil-mined program.
> (2) error return from malloc() and fork() are disabled.

i think you've turned a problem with bounded recovery time into a
situation where the recovery code itself will inadvertently dos attack its
users.

- erik

Re: [9fans] protection against resource exhaustion

[arisawa]

Hello,

> nonetheless, i have experience running multi-user plan 9 systems, and users
> were not usually the issue.

Eric’s users are all gentleman, all careful people and all skillful programmers.
If your system is served for university students, you will have different thought.

> i think you've turned a problem with bounded recovery time into a
> situation where the recovery code itself will inadvertently dos attack its
> users.

in case that a process failed in getting resource such as memory or process, 
what it should do is very limited: puts out some message and exits.
this is right behavior.
I have never seen programs that retry malloc() or fork() until succeed.
if all processes retry them, the system will get down.
this is what I have observed in current plan9 kernel.

if any one has cleaner solution, i.e., a solution that never kill innocent process,
I want to see it.

Re: [9fans] protection against resource exhaustion

[lucio]

> if any one has cleaner solution, i.e., a solution that never kill innocent process,
> I want to see it.

I have an almost mathematical certainty that the human factor is by
far the greatest contributor to the problem and solving any technical
issue would be insignificant by comparison.  Like any other form of
security, the trade offs are what count.

Bloating Plan 9 with feeble attempts to shelter the system against
malicious users is, in my opinion, too expensive.

Lucio.

Re: [9fans] protection against resource exhaustion

[Quintile]

I always thought the best solution for a university system, is to implement the fair share scheduler. thus people can use any resource the want on an idle machine, but a saturated machine splits its load based on rules.

I had this experience when I managed a cray (YMP-El only) and it worked perfectly. it did not limit processes per user, but it could.

similarly ram use and network bandwidth - though this only works per machine.

personally I like erik's idea, though I would make it a boot option.

I would also keep malloc() returning nil on fail, but add emalloc() to libc.
and finally have an environment car that makes emalloc() clas sys fatal)) or abort() - for debug.

Steve





> On 28 Jan 2015, at 06:50, arisawa <arisawa@ar.aichi-u.ac.jp> wrote:
> 
> Hello,
> 
>> nonetheless, i have experience running multi-user plan 9 systems, and users
>> were not usually the issue.
> 
> Eric’s users are all gentleman, all careful people and all skillful programmers.
> If your system is served for university students, you will have different thought.
> 
>> i think you've turned a problem with bounded recovery time into a
>> situation where the recovery code itself will inadvertently dos attack its
>> users.
> 
> in case that a process failed in getting resource such as memory or process, 
> what it should do is very limited: puts out some message and exits.
> this is right behavior.
> I have never seen programs that retry malloc() or fork() until succeed.
> if all processes retry them, the system will get down.
> this is what I have observed in current plan9 kernel.
> 
> if any one has cleaner solution, i.e., a solution that never kill innocent process,
> I want to see it.
> 
> 

Re: [9fans] protection against resource exhaustion

[cinap_lenrek]

> in case that a process failed in getting resource such as memory or process,
> what it should do is very limited: puts out some message and exits.
> this is right behavior.

practically, for most simple programs fork error is handled by sysfatal().
but things get more tricky for libthread programs like rio.

from thread(2):
        Errors, notes and resources
          Thread library functions do not return on failure; if errors
          occur, the entire program is aborted.

in your simple forkloop case, the kernel can break the loop by failing the fork there
but if at the same time you draw a window or hit interrupt key while the system
ran out of processes, the proccreate() in rio can render your window system unusable.

and what about stuff like your multithreaded filservers? factotum? cs? dns? plumber?
note that there are interdependencies betweeen these... and there might be
multiple instances running (multiple users on a cpu server).

the point is, you need to take on the effort of hardening *all* the long running and
forking servers against this so they can recover from failing fork, or else you
have nothing gained compared to what the kernel does currently.

could be done, but its hard to get right and test if you got it right.

i'd go for giving the students ther own terminals so they crash ther own machines
instead of taking down cpu servers :)

--
cinap

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

> > in case that a process failed in getting resource such as memory or process,
> > what it should do is very limited: puts out some message and exits.
> > this is right behavior.
>
> practically, for most simple programs fork error is handled by sysfatal().
> but things get more tricky for libthread programs like rio.

i'd forgotten the added difficulty with the 32-bit kernels that plan 9 overcommits
memory.  so in a low-memory situation, a process can be killed by using more
stack!

- erik

Re: [9fans] protection against resource exhaustion

[lucio]

> i'd forgotten the added difficulty with the 32-bit kernels that plan 9 overcommits
> memory.  so in a low-memory situation, a process can be killed by using more
> stack!

I could have sworn I heard otherwise.  But things sometimes are not as I think they are.  Here, I'm very curious to know why I remember differently.  Nemo?

Lucio.

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

On Wed Jan 28 06:05:50 PST 2015, lucio@proxima.alt.za wrote:
> > i'd forgotten the added difficulty with the 32-bit kernels that plan 9 overcommits
> > memory.  so in a low-memory situation, a process can be killed by using more
> > stack!
>
> I could have sworn I heard otherwise.  But things sometimes are not as I think they are.  Here, I'm very curious to know why I remember differently.  Nemo?

the reason is that each process can have up to 16mb of stack, and this is unaccounted.
thus the stack or (seg)?brk can commit to memory that will fault when you touch it.

- erik

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

On Tue Jan 27 22:46:22 PST 2015, arisawa@ar.aichi-u.ac.jp wrote:
> Hello,
> 
> > nonetheless, i have experience running multi-user plan 9 systems, and users
> > were not usually the issue.
> 
> Eric’s users are all gentleman, all careful people and all skillful programmers.
> If your system is served for university students, you will have different thought.

erik has students, too.  i don't nerf the system for them.

- erik

Re: [9fans] protection against resource exhaustion

[lucio]

> the reason is that each process can have up to 16mb of stack, and this is unaccounted.
> thus the stack or (seg)?brk can commit to memory that will fault when you touch it.

It's a vague recollection, but the fact that the stack is being
overcommitted rings a bell.  Thanks, Erik.

Lucio.

Re: [9fans] protection against resource exhaustion

[Skip Tavakkolian]

[-- Attachment #1: Type: text/plain, Size: 793 bytes --]

as someone mentioned, a dedicated real or virtual term (9vx, rpi+9pi) is
the right option. is there a reason this might not work for your
environment?

as for system watchdog, usually an external subsystem is used. i wonder if
a cpu (e.g. 9pi) dedicated to monitoring the main cpu's /proc (and perhaps
/net) for "abnormal activity" (whatever that is) and killing suspicious
procs could work.

On Wed Jan 28 2015 at 6:54:01 AM <lucio@proxima.alt.za> wrote:

> > the reason is that each process can have up to 16mb of stack, and this
> is unaccounted.
> > thus the stack or (seg)?brk can commit to memory that will fault when
> you touch it.
>
> It's a vague recollection, but the fact that the stack is being
> overcommitted rings a bell.  Thanks, Erik.
>
> Lucio.
>
>
>

[-- Attachment #2: Type: text/html, Size: 1037 bytes --]

Re: [9fans] protection against resource exhaustion

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 2032 bytes --]

On 28 January 2015 at 06:50, arisawa <arisawa@ar.aichi-u.ac.jp> wrote:

> Eric’s users are all gentleman, all careful people and all skillful
> programmers.
> If your system is served for university students, you will have different
> thought.
>

You might also be running services on it, and it's reasonable that it
should manage its resources.
It's best to refuse a new load if it won't fit, but that requires a fairly
careful system design and works
best in closed systems. In a capability system you'd have to present a
capability to create a process,
and it's easy to ration them. Inside Plan 9, one could have a nested
resource allocation scheme
for the main things without too much trouble. Memory is a little different
because you either under-use
(by reserving space) or over-commit, as usually happens with file-system
quotas.

With memory, I had some success with a hybrid scheme, still not intended
for multi-user,
but it could be extended to do that. The system accounts in advance for
probable demands of
things like fork/exec and segment growth (including stack growth). If that
preallocation is then not
used (eg, because of copy-on-write fork before exec) it is gradually
written off. When that fails,
the system looks at the most recently created process groups (note groups)
and kills them.
It differs from the OOM because it doesn't assume that the biggest thing is
actually the problem.
Instead it looks back through the most recently-started applications since
that corresponded to my usage.
It's quite common to have long-running components that
soak up lots of memory (eg, a cache process or fossil) and they often
aren't the ones that caused
the problem. Instead I assume something will have started recently that was
ill-advised. It would be better
to track allocation history across process groups and use that instead but
I needed something simple,
and it was reasonably effective.

For resources such as process (slots), network connections, etc I think a

[-- Attachment #2: Type: text/html, Size: 2983 bytes --]

Re: [9fans] protection against resource exhaustion

[cinap_lenrek]

your mail got cut off after "I think a"

--
cinap

Re: [9fans] protection against resource exhaustion

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 218 bytes --]

On 28 January 2015 at 17:39, <cinap_lenrek@felloff.net> wrote:

> your mail got cut off after "I think a"


I'd moved the body of that paragraph into the first paragraph, but that bit
had scrolled out of sight.

[-- Attachment #2: Type: text/html, Size: 521 bytes --]

Re: [9fans] protection against resource exhaustion

[arisawa]

Thanks for all.

I have looked old source code /sys/src/9/port/proc.c, and found the next code exists from very old day.

	lock(&procalloc);
	for(;;) {
		if((p = procalloc.free) != nil)
			break;

		snprint(msg, sizeof msg, "no procs; %s forking",
			up != nil ? up->text: "kernel");
		unlock(&procalloc);
		resrcwait(msg);
		lock(&procalloc);
	}
	procalloc.free = p->qnext;
	unlock(&procalloc);

and rfork(2) says:
	Rfork will sleep, if necessary, until required process resources are available.
which is also exists from old day.

Nevertheless all programs, I believe, in /sys/src are written assuming rfork() may return -l:
	switch(rfork(….)){
	case -1:
	  …
	}

Any merit?

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

> Nevertheless all programs, I believe, in /sys/src are written assuming rfork() may return -l:

* cifs keepalive thread may not start
* ssh2 will malfunction in ssh.c
* htmlroff will malfunction
* imap4d idle will fail with no warning
* upas/scanmail:/^qmail, same issue
* page/gs.c will die wierdly

etc.

so is is incorrect to assume that plan 9 programs even have code to handle
fork errors.

- erik

Re: [9fans] protection against resource exhaustion

[erik quanstrom]

> You might also be running services on it, and it's reasonable that it
> should manage its resources.  It's best to refuse a new load if it
> won't fit, but that requires a fairly careful system design and works
> best in closed systems.  In a capability system you'd have to present
> a capability to create a process, and it's easy to ration them.
> Inside Plan 9, one could have a nested resource allocation scheme for
> the main things without too much trouble.  Memory is a little
> different because you either under-use (by reserving space) or
> over-commit, as usually happens with file-system quotas.
>
> With memory, I had some success with a hybrid scheme, still not
> intended for multi-user, but it could be extended to do that.  The
> system accounts in advance for probable demands of things like
> fork/exec and segment growth (including stack growth).  If that
> preallocation is then not used (eg, because of copy-on-write fork
> before exec) it is gradually written off.  When that fails, the system
> looks at the most recently created process groups (note groups) and
> kills them.  It differs from the OOM because it doesn't assume that
> the biggest thing is actually the problem.  Instead it looks back
> through the most recently-started applications since that corresponded
> to my usage.  It's quite common to have long-running components that
> soak up lots of memory (eg, a cache process or fossil) and they often
> aren't the ones that caused the problem.  Instead I assume something
> will have started recently that was ill-advised.  It would be better
> to track allocation history across process groups and use that instead
> but I needed something simple, and it was reasonably effective.
>
> For resources such as process (slots), network connections, etc I

thanks, charles.

i hope i haven't overplayed my argument.  i am for real solutions to this issue.
i'm not for the current solution, or more complicanted variants.

how does one account/adjust for long-running, important processes, say forking for
a small but important task?

perhaps an edf-like scheme predeclaring the expected usage for "important" processes
might work?

- erik

Re: [9fans] protection against resource exhaustion

[arisawa]

Hi, erik,

Thanks a lot!

> 2015/01/29 15:42、erik quanstrom <quanstro@quanstro.net> のメール：
> 
> i hope i haven't overplayed my argument.  i am for real solutions to this issue.
> i'm not for the current solution, or more complicanted variants.