[9fans] Maximal number of processes

[9fans] Maximal number of processes

[Pavel Klinkovský]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

Hi all,

I would like to know whether there is any hard (based on CPU architecture)
limit of maximal number of processes in Plan9 on Intel or ARM.

I do not think the soft limit like the lack of memory... ;)

Thanks in advance for any hint.

Pavel

[-- Attachment #2: Type: text/html, Size: 368 bytes --]

Re: [9fans] Maximal number of processes

[erik quanstrom]

> I would like to know whether there is any hard (based on CPU architecture)
> limit of maximal number of processes in Plan9 on Intel or ARM.
>
> I do not think the soft limit like the lack of memory... ;)

there is not.

- erik

Re: [9fans] Maximal number of processes

[Steven Stallion]

On Wed, Jan 8, 2014 at 2:28 AM, Pavel Klinkovský
<pavel.klinkovsky@gmail.com> wrote:
> Hi all,
>
> I would like to know whether there is any hard (based on CPU architecture)
> limit of maximal number of processes in Plan9 on Intel or ARM.
>
> I do not think the soft limit like the lack of memory... ;)
>
> Thanks in advance for any hint.

Hi Pavel,

It absolutely does. Depending on the kernel you are booting,
conf.nproc will be set in a variety of different ways. This is
generally set in confinit, called from main. The obvious example is
the pc kernel:

    conf.nproc = 100 + ((conf.npage*BY2PG)/MB)*5;
    if(cpuserver)
            conf.nproc *= 3;
    if(conf.nproc > 2000)
            conf.nproc = 2000;

In general, you will find that 2000 is the highest allowable due to
limits imposed by proc.c. Other architectures can (and will) place
additional restrictions. A non-FCSE ARM implementation could elect to
only support 256 processes to avoid additional switching overhead for
example.

At the end of the day, the only way to be sure is to read the source.
It will give you a better familiarity with the system and probably
avoid some unnecessary perturbation.

Cheers,

Steve

Re: [9fans] Maximal number of processes

[erik quanstrom]

> In general, you will find that 2000 is the highest allowable due to
> limits imposed by proc.c. Other architectures can (and will) place
> additional restrictions. A non-FCSE ARM implementation could elect to
> only support 256 processes to avoid additional switching overhead for
> example.

i had a different interpretation of the op's definition of "hard",
since memory limits were deemed irrelevant.

what limits are you thinking of in proc.c?

the only hard limit in the portable code that i see is in devproc.c,
the number of processes needs to fit in the space reserved in
the qid.  i'd forgotten about that since its 2^23, which is so
large that it might safely be forgotten.

there are plenty of soft limits in the portable code.  linear scans
of the process table are made in a number of places, for example.
and of course process allocation must be static due to the locking
schemes used.  (cf. the recent procalarm discussion.)

- erik

Re: [9fans] Maximal number of processes

[Pavel Klinkovský]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

Hi Steven,


    conf.nproc = 100 + ((conf.npage*BY2PG)/MB)*5;
>     if(cpuserver)
>             conf.nproc *= 3;
>     if(conf.nproc > 2000)
>             conf.nproc = 2000;
>
> In general, you will find that 2000 is the highest allowable due to
> limits imposed by proc.c.


but if I understand it correctly it is just "soft limit", not a "hard" one
inborn in CPU architecture.

By the hard limit I consider something like "maximal capacity of GDT, LDT"
or something similar, if exists.


> At the end of the day, the only way to be sure is to read the source.
>

I did (of course, I am not an expert on Plan9 kernel), and I did not find
any "hard limit" there.
However to be sure I issued my question here.

Pavel

[-- Attachment #2: Type: text/html, Size: 1225 bytes --]

Re: [9fans] Maximal number of processes

[Bruce Ellis]

[-- Attachment #1: Type: text/plain, Size: 1079 bytes --]

suck it and see, the answerers didn't understand the question. add
nproc=XXX to plan9.ini and use the environment, or hard code code it. i'd
like to see your results for nproc=50 and nproc=5000.

brucee


On 9 January 2014 19:08, Pavel Klinkovský <pavel.klinkovsky@gmail.com>wrote:

> Hi Steven,
>
>
>     conf.nproc = 100 + ((conf.npage*BY2PG)/MB)*5;
>>     if(cpuserver)
>>             conf.nproc *= 3;
>>     if(conf.nproc > 2000)
>>             conf.nproc = 2000;
>>
>> In general, you will find that 2000 is the highest allowable due to
>> limits imposed by proc.c.
>
>
> but if I understand it correctly it is just "soft limit", not a "hard" one
> inborn in CPU architecture.
>
> By the hard limit I consider something like "maximal capacity of GDT, LDT"
> or something similar, if exists.
>
>
>> At the end of the day, the only way to be sure is to read the source.
>>
>
> I did (of course, I am not an expert on Plan9 kernel), and I did not find
> any "hard limit" there.
> However to be sure I issued my question here.
>
> Pavel
>

[-- Attachment #2: Type: text/html, Size: 1976 bytes --]

Re: [9fans] Maximal number of processes

[Pavel Klinkovský]

[-- Attachment #1.1: Type: text/plain, Size: 359 bytes --]

Hi Bruce,

suck it and see, the answerers didn't understand the question. add
> nproc=XXX to plan9.ini and use the environment, or hard code code it. i'd
> like to see your results for nproc=50 and nproc=5000.
>

I tried what you proposed.
It really seems there is no hard limit in Plan9 kernel, only soft ones -
nproc or available memory.

Pavel

[-- Attachment #1.2: Type: text/html, Size: 680 bytes --]

[-- Attachment #2: nproc100.png --]
[-- Type: image/png, Size: 112863 bytes --]

[-- Attachment #3: nproc10000.png --]
[-- Type: image/png, Size: 106835 bytes --]

Re: [9fans] Maximal number of processes

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

On 9 January 2014 08:08, Pavel Klinkovský <pavel.klinkovsky@gmail.com>wrote:

> By the hard limit I consider something like "maximal capacity of GDT, LDT"
> or something similar, if exists.


GDT and even Tss are per-processor; ldt isn't used.

As to soft limits, apart from the few linear searches mentioned, which
could be eliminated if necessary,
for instance by tracking process groups, the main constraint is the kernel
virtual address space and physical memory consumed by
the KSTACK for each process. It's only 4k (in 32 bit mode) but it might be
difficult to handle hundreds of thousands of the things
comfortably.

I tried 3000, on a lowly Atom with 2 Gb RAM:

nova% ps|wc -l
   2992

and it seemed fine. Do I hear "more!"?

nova% ps|wc -l
   3991

nova% cat /dev/swap
2135371776 memory
4096 pagesize
48986 kernel
28594/472345 user
0/320000 swap
35004768/142308320 kernel malloc
0/16777216 kernel draw

At that point I decided to quite while I was still ahead. I'd probably need
a bit more DRAM to do anything much with all of them
(they were only running sleep 60), but that's easy to find. Except that
they'll have changed DRAM types again no doubt.

[-- Attachment #2: Type: text/html, Size: 2147 bytes --]

Re: [9fans] Maximal number of processes

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 346 bytes --]

On 10 January 2014 09:11, Charles Forsyth <charles.forsyth@gmail.com> wrote:

> At that point I decided to quite while I was still ahead.


20,000 did not work because it ran out of kernel physical memory. That
preallocation could be adjusted, but at some point the available kernel
virtual address space will limit what it can allocate.

[-- Attachment #2: Type: text/html, Size: 639 bytes --]

Re: [9fans] Maximal number of processes

[Bruce Ellis]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

Good work. As my good friend Boyd once said "Don't give me bullshit
speculation. Measure something!".

brucee


On 10 January 2014 20:15, Charles Forsyth <charles.forsyth@gmail.com> wrote:

>
> On 10 January 2014 09:11, Charles Forsyth <charles.forsyth@gmail.com>wrote:
>
>> At that point I decided to quite while I was still ahead.
>
>
> 20,000 did not work because it ran out of kernel physical memory. That
> preallocation could be adjusted, but at some point the available kernel
> virtual address space will limit what it can allocate.
>

[-- Attachment #2: Type: text/html, Size: 1200 bytes --]

Re: [9fans] Maximal number of processes

[erik quanstrom]

> 20,000 did not work because it ran out of kernel physical memory. That
> preallocation could be adjusted, but at some point the available kernel
> virtual address space will limit what it can allocate.

at the cost of moving KZERO down 256MB on the pae kernel,

	ivey# ps|wc
	  15961  111727  957732

and even more

	ivey; ps|wc
	  30064  210448 1803915

	ivey# cat /dev/swap
	17153769472 memory
	4096 pagesize
	131072 kernel
	271322/4056860 user
	0/0 swap
	0/17877472 kernel malloc
	0/17877472 kernel draw

interestingly, the time to launch 1000 sleeps went up in a non-linear way
here's the time to launch each 1000 processes

1000	0
2000	0
3000	1
4000	0
5000	1
6000	0
7000	1
8000	1
9000	0
10000	1
11000	1
12000	1
13000	1
14000	1
15000	2
16000	1
17000	2
18000	2
19000	2
20000	3
21000	3
22000	4
23000	4
24000	5
25000	5
26000	5
27000	7
28000	7
29000	8
30000	8

launching 32000 processes was not possible.  the kernel got stuck.

here's one thing that's not immediately obvious, even when running the
kernel.  conv.nmach must be less than 0x7fffffff/(1000*1000) = 2147
to prevent reprioritize from doing silly things:

	/*
	 *  fairshare = 1.000 * conf.nproc * 1.000/load,
	 * except the decimal point is moved three places
	 * on both load and fairshare.
	 */
	fairshare = (conf.nmach*1000*1000)/load;

- erik

Re: [9fans] Maximal number of processes

[erik quanstrom]

> launching 32000 processes was not possible.  the kernel got stuck.

sloppy statement.  it's not clear if the kernel was really stuck or just
hit something exponential.

> here's one thing that's not immediately obvious, even when running the
> kernel.  conv.nmach must be less than 0x7fffffff/(1000*1000) = 2147
> to prevent reprioritize from doing silly things:
>
> 	/*
> 	 *  fairshare = 1.000 * conf.nproc * 1.000/load,
> 	 * except the decimal point is moved three places
> 	 * on both load and fairshare.
> 	 */
> 	fairshare = (conf.nmach*1000*1000)/load;

and, look.  the comment's wrong.  never read the comments!

- erik

Re: [9fans] Maximal number of processes

[Pavel Klinkovský]

[-- Attachment #1.1: Type: text/plain, Size: 1021 bytes --]

I discovered another interesting feature in my tests:

If 'fork' exceeds nproc value, kernel panics.
If 'fork' reaches the available memory limit, it is blocked until some
memory is released (e.g. by the finish of some process).

Pavel



2014/1/10 erik quanstrom <quanstro@quanstro.net>

> > launching 32000 processes was not possible.  the kernel got stuck.
>
> sloppy statement.  it's not clear if the kernel was really stuck or just
> hit something exponential.
>
> > here's one thing that's not immediately obvious, even when running the
> > kernel.  conv.nmach must be less than 0x7fffffff/(1000*1000) = 2147
> > to prevent reprioritize from doing silly things:
> >
> >       /*
> >        *  fairshare = 1.000 * conf.nproc * 1.000/load,
> >        * except the decimal point is moved three places
> >        * on both load and fairshare.
> >        */
> >       fairshare = (conf.nmach*1000*1000)/load;
>
> and, look.  the comment's wrong.  never read the comments!
>
> - erik
>
>

[-- Attachment #1.2: Type: text/html, Size: 1592 bytes --]

[-- Attachment #2: processes.PNG --]
[-- Type: image/png, Size: 33921 bytes --]