From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 4183 invoked from network); 16 Dec 1999 09:56:22 -0000 Received: from sunsite.auc.dk (130.225.51.30) by ns1.primenet.com.au with SMTP; 16 Dec 1999 09:56:22 -0000 Received: (qmail 20508 invoked by alias); 16 Dec 1999 09:56:13 -0000 Mailing-List: contact zsh-workers-help@sunsite.auc.dk; run by ezmlm Precedence: bulk X-No-Archive: yes X-Seq: 9079 Received: (qmail 20457 invoked from network); 16 Dec 1999 09:56:08 -0000 Date: Thu, 16 Dec 1999 10:56:07 +0100 (MET) Message-Id: <199912160956.KAA13359@beta.informatik.hu-berlin.de> From: Sven Wischnowsky To: zsh-workers@sunsite.auc.dk In-reply-to: "Bart Schaefer"'s message of Mon, 13 Dec 1999 18:24:44 +0000 Subject: Re: zsh and memory [ Moved to zsh-workers ] Bart Schaefer wrote: > On Dec 13, 6:15am, Danny Dulai wrote: > } Subject: zsh and memory > } > } Any hints on how to make zsh not consume as much memory as it does? > } I'm not using the zftp or stat modules. Any thing else I can do? > > It'd be easier to answer the question if we knew what you ARE doing. > "Is there any way I can get rid of these lumps on my head? I'm not > hitting myself with a hammer." > > Various things you should avoid if memory is a problem include: > > setopt ALLEXPORT > use large here-documents > define lots of shell functions without using autoload > use "zed" (old) or the mapfile module (new) to edit files > use the new 3.1.6 shell-function-based completion system > > The last one is a bit unfortunate, but just "compinit" (without even > trying any completions yet) adds half a megabyte to the RSS of zsh on > my system, and it only goes up from there as functions autload and > start caching their results in shell variables. I've had a look at this... Horrors! I tried it with `zsh -f', with `zsh -f' and loading the completion system, and with that and doing `ls ../Com/Cor'. Looking at the output of the mem builtin first revealed that there were almost 400KB of free memory, in relatively few large blocks -- remnents of freed heaps. So I looked who was using lots of heap memory and found the tokstr handling in lex.c. There we always allocate at least 256 bytes and if the buffer needs expanding (in add()) in gets resized to inbufct (or larger). Some more investigation showed that we almost never need a tokstr with more than 32 bytes, so I changed that. I also changed add() to be more careful when expanding the buffer -- I left the old code conditioned out because I don't kno if there was a reason to resize it to inbufct bytes; I at least don't see a reason for that. Then I had another look at the mem output and had to see several *used* blocks starting with `#autoload..# ...'. Yes, indeed, the beginnings of the autoloaded functions. getfpfunc() uses a zalloc()ed buffer for the whole file content and didn't free it (it may be that this was my fault, may have happened in the non-copying execution code patches). So I fixed that, too. Then I added pushheap()s and popheap()s around function loading code and that helped some more. These patches save me (in the tests mentioned above) 200KB and 600KB, respectively (resident size). But I still have about 160KB of free memory (old heaps). I'm really tempted to allocate heaps using mmap() (anonymous) to get them out of the way of the zalloc() allocator. I small test showed that with this I only get 39KB of free memory after the completion test, which is really not too bad. I don't have a patch for that yet, though. Bye Sven diff -ru ../z.old/Src/exec.c Src/exec.c --- ../z.old/Src/exec.c Thu Dec 16 09:46:11 1999 +++ Src/exec.c Thu Dec 16 10:10:07 1999 @@ -2905,6 +2905,8 @@ int noalias = noaliases; List l; + pushheap(); + noaliases = (shf->flags & PM_UNALIASED); l = getfpfunc(shf->nam); noaliases = noalias; @@ -2929,6 +2931,8 @@ } LASTALLOC; shf->flags &= ~PM_UNDEFINED; } + popheap(); + execlist(shf->funcdef, 1, 0); return lastval; } @@ -2942,6 +2946,8 @@ int noalias = noaliases; List l; + pushheap(); + noaliases = (shf->flags & PM_UNALIASED); l = getfpfunc(shf->nam); noaliases = noalias; @@ -2955,6 +2961,8 @@ } LASTALLOC; shf->flags &= ~PM_UNDEFINED; + popheap(); + return 0; } @@ -3118,9 +3126,14 @@ HEAPALLOC { r = parse_string(d, 1); } LASTALLOC; + + zfree(d, len + 1); + return r; } else close(fd); + + zfree(d, len + 1); } else { close(fd); } diff -ru ../z.old/Src/lex.c Src/lex.c --- ../z.old/Src/lex.c Thu Dec 16 09:46:12 1999 +++ Src/lex.c Thu Dec 16 09:57:18 1999 @@ -470,6 +470,7 @@ { *bptr++ = c; if (bsiz == ++len) { +#if 0 int newbsiz; newbsiz = bsiz * 8; @@ -477,6 +478,12 @@ newbsiz *= 2; bptr = len + (tokstr = (char *)hrealloc(tokstr, bsiz, newbsiz)); bsiz = newbsiz; +#endif + + int newbsiz = bsiz + 32; + + bptr = len + (tokstr = (char *)hrealloc(tokstr, bsiz, newbsiz)); + bsiz = newbsiz; } } @@ -556,7 +563,7 @@ /* word includes the last character read and possibly \ before ! */ if (dbparens) { len = 0; - bptr = tokstr = (char *)ncalloc(bsiz = 256); + bptr = tokstr = (char *)ncalloc(bsiz = 32); hungetc(c); cmdpush(CS_MATH); c = dquote_parse(infor ? ';' : ')', 0); @@ -671,7 +678,7 @@ } if (incmdpos) { len = 0; - bptr = tokstr = (char *)ncalloc(bsiz = 256); + bptr = tokstr = (char *)ncalloc(bsiz = 32); return cmd_or_math(CS_MATH) ? DINPAR : INPAR; } } else if (d == ')') @@ -818,7 +825,7 @@ peek = STRING; if (!sub) { len = 0; - bptr = tokstr = (char *)ncalloc(bsiz = 256); + bptr = tokstr = (char *)ncalloc(bsiz = 32); } for (;;) { int act; -- Sven Wischnowsky wischnow@informatik.hu-berlin.de