From mboxrd@z Thu Jan  1 00:00:00 1970
From: erik quanstrom <quanstro@quanstro.net>
Date: Fri, 16 Apr 2010 10:46:37 -0400
To: 9fans@9fans.net
Message-ID: <938095881ad5430643c47765d1a61c31@kw.quanstro.net>
MIME-Version: 1.0
Content-Type: text/plain; charset="US-ASCII"
Content-Transfer-Encoding: 7bit
Subject: [9fans] pc *kernelpercent improvement
Topicbox-Message-UUID: 03ad34b4-ead6-11e9-9d60-3106f5b1d025

with some nic drivers (myricom) requiring a vast amount of
buffer space, and the general pc split between user space
and kernel space on a 4gb machine giving only 71mb to
the kernel (1.6%), i find it necessary to set *kernelpercent
on a few machines that have 3-4 nics.  unfortunately, the
kernel is just too smart to allow me to set *kernelpercent
and applies (imho) too much logic, generally defeating
my settings.

this patch allows one to set *kernelprecent to anything
"reasonable".  reasonable means that kernel memory will
fit in the available virtual memory at or above KZERO.
this, kernel memory is capped at 256MB for the standard
KZERO of 0xf0000000.   but if one had KZERO set to
0x80000000, kernel memory would be capped at 2gb.
naturally, the lowest possible KZERO while maintaining
pci(e) space and user memory would be 0x20000000.  but
i haven't been that brave.  i have only tested 0x30000000.

if *kernelpercent is not set, then the logic is unchanged
from before.

/n/sources/plan9//sys/src/9/pc/main.c:397,405 - main.c:397,408
  		 * page list in imagereclaim just takes too long.
  		 */
  		if(kpages > (64*MB + conf.npage*sizeof(Page))/BY2PG){
- 			kpages = (64*MB + conf.npage*sizeof(Page))/BY2PG;
+ 			if(p == nil){
+ 				kpages = (64*MB + conf.npage*sizeof(Page))/BY2PG;
+ 				kpages += (conf.nproc*KSTACK)/BY2PG;
+ 			}else if(kpages > (ulong)-KZERO)
+ 				kpages = (ulong)-KZERO;
  			conf.nimage = 2000;
- 			kpages += (conf.nproc*KSTACK)/BY2PG;
  		}
  	} else {
  		if(userpcnt < 10) {

one thing that could save quite a bit of buffer space for
most drivers that keep a private pool would be a function
allocbpool(uint cnt, uint size, uint align, void (*free)(Block*)).
if size%align < align you can save cnt*(align - size%align) - align
bytes by allocating the buffers seperately from the Block contents,
in large allocations.  or packing the Block header carefully in the slop.
(in addition to Pool overhead.)

for cnt=512, size=16k and align=4k (myricom needs
a power-of-two buffer and has a 9k mtu) this is
	512*(4k - 0) - 4k = 2mb - 4096b (25%).
for cnt=1024, size=12k and align=4k (82563), this is 4mb (25%).
for 4 or 6 interfaces, this really starts to add up.

- erik