Re: zsh and memory

Re: zsh and memory

[Sven Wischnowsky]

[ Moved to zsh-workers ]

Bart Schaefer wrote:

> On Dec 13,  6:15am, Danny Dulai wrote:
> } Subject: zsh and memory
> }
> } Any hints on how to make zsh not consume as much memory as it does?
> } I'm not using the zftp or stat modules. Any thing else I can do?
> 
> It'd be easier to answer the question if we knew what you ARE doing.
> "Is there any way I can get rid of these lumps on my head?  I'm not
> hitting myself with a hammer."
> 
> Various things you should avoid if memory is a problem include:
> 
>     setopt ALLEXPORT
>     use large here-documents
>     define lots of shell functions without using autoload
>     use "zed" (old) or the mapfile module (new) to edit files
>     use the new 3.1.6 shell-function-based completion system
> 
> The last one is a bit unfortunate, but just "compinit" (without even
> trying any completions yet) adds half a megabyte to the RSS of zsh on
> my system, and it only goes up from there as functions autload and
> start caching their results in shell variables.

I've had a look at this... Horrors!


I tried it with `zsh -f', with `zsh -f' and loading the completion
system, and with that and doing `ls ../Com/Cor<TAB>'. Looking at the
output of the mem builtin first revealed that there were almost 400KB
of free memory, in relatively few large blocks -- remnents of freed
heaps. So I looked who was using lots of heap memory and found the
tokstr handling in lex.c. There we always allocate at least 256 bytes
and if the buffer needs expanding (in add()) in gets resized to
inbufct (or larger). Some more investigation showed that we almost
never need a tokstr with more than 32 bytes, so I changed that. I also 
changed add() to be more careful when expanding the buffer -- I left
the old code conditioned out because I don't kno if there was a reason 
to resize it to inbufct bytes; I at least don't see a reason for that.

Then I had another look at the mem output and had to see several
*used* blocks starting with `#autoload..# ...'. Yes, indeed, the
beginnings of the autoloaded functions. getfpfunc() uses a zalloc()ed
buffer for the whole file content and didn't free it (it may be that
this was my fault, may have happened in the non-copying execution code 
patches). So I fixed that, too.

Then I added pushheap()s and popheap()s around function loading code
and that helped some more.

These patches save me (in the tests mentioned above) 200KB and 600KB,
respectively (resident size). But I still have about 160KB of free
memory (old heaps). I'm really tempted to allocate heaps using mmap()
(anonymous) to get them out of the way of the zalloc() allocator. I
small test showed that with this I only get 39KB of free memory after
the completion test, which is really not too bad. I don't have a patch 
for that yet, though.

Bye
 Sven

diff -ru ../z.old/Src/exec.c Src/exec.c
--- ../z.old/Src/exec.c	Thu Dec 16 09:46:11 1999
+++ Src/exec.c	Thu Dec 16 10:10:07 1999
@@ -2905,6 +2905,8 @@
     int noalias = noaliases;
     List l;
 
+    pushheap();
+
     noaliases = (shf->flags & PM_UNALIASED);
     l = getfpfunc(shf->nam);
     noaliases = noalias;
@@ -2929,6 +2931,8 @@
 	} LASTALLOC;
 	shf->flags &= ~PM_UNDEFINED;
     }
+    popheap();
+
     execlist(shf->funcdef, 1, 0);
     return lastval;
 }
@@ -2942,6 +2946,8 @@
     int noalias = noaliases;
     List l;
 
+    pushheap();
+
     noaliases = (shf->flags & PM_UNALIASED);
     l = getfpfunc(shf->nam);
     noaliases = noalias;
@@ -2955,6 +2961,8 @@
     } LASTALLOC;
     shf->flags &= ~PM_UNDEFINED;
 
+    popheap();
+
     return 0;
 }
 
@@ -3118,9 +3126,14 @@
 		    HEAPALLOC {
 			r = parse_string(d, 1);
 		    } LASTALLOC;
+
+		    zfree(d, len + 1);
+
 		    return r;
 		} else
 		    close(fd);
+
+		zfree(d, len + 1);
 	    } else {
 		close(fd);
 	    }
diff -ru ../z.old/Src/lex.c Src/lex.c
--- ../z.old/Src/lex.c	Thu Dec 16 09:46:12 1999
+++ Src/lex.c	Thu Dec 16 09:57:18 1999
@@ -470,6 +470,7 @@
 {
     *bptr++ = c;
     if (bsiz == ++len) {
+#if 0
 	int newbsiz;
 
 	newbsiz = bsiz * 8;
@@ -477,6 +478,12 @@
 	    newbsiz *= 2;
 	bptr = len + (tokstr = (char *)hrealloc(tokstr, bsiz, newbsiz));
 	bsiz = newbsiz;
+#endif
+
+	int newbsiz = bsiz + 32;
+
+	bptr = len + (tokstr = (char *)hrealloc(tokstr, bsiz, newbsiz));
+	bsiz = newbsiz;
     }
 }
 
@@ -556,7 +563,7 @@
     /* word includes the last character read and possibly \ before ! */
     if (dbparens) {
 	len = 0;
-	bptr = tokstr = (char *)ncalloc(bsiz = 256);
+	bptr = tokstr = (char *)ncalloc(bsiz = 32);
 	hungetc(c);
 	cmdpush(CS_MATH);
 	c = dquote_parse(infor ? ';' : ')', 0);
@@ -671,7 +678,7 @@
 	    }
 	    if (incmdpos) {
 		len = 0;
-		bptr = tokstr = (char *)ncalloc(bsiz = 256);
+		bptr = tokstr = (char *)ncalloc(bsiz = 32);
 		return cmd_or_math(CS_MATH) ? DINPAR : INPAR;
 	    }
 	} else if (d == ')')
@@ -818,7 +825,7 @@
     peek = STRING;
     if (!sub) {
 	len = 0;
-	bptr = tokstr = (char *)ncalloc(bsiz = 256);
+	bptr = tokstr = (char *)ncalloc(bsiz = 32);
     }
     for (;;) {
 	int act;

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: zsh and memory

[Bart Schaefer]

On Dec 16, 10:56am, Sven Wischnowsky wrote:
} Subject: Re: zsh and memory
}
} [...] I looked who was using lots of heap memory and found the
} tokstr handling in lex.c. There we always allocate at least 256 bytes
} and if the buffer needs expanding (in add()) in gets resized to
} inbufct (or larger). Some more investigation showed that we almost
} never need a tokstr with more than 32 bytes, so I changed that.  I also 
} changed add() to be more careful when expanding the buffer -- I left
} the old code conditioned out because I don't kno if there was a reason 
} to resize it to inbufct bytes; I at least don't see a reason for that.

Probably the idea was to eliminate lots of small allocations in the
event that the "token" is a long quoted string or the like, for speed.
We should watch out for performance problems if those changes are kept.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: zsh and memory

[Sven Wischnowsky]

[ Also noticed by Mike... ]

Timothy Writer wrote:

> Sven Wischnowsky <wischnow@informatik.hu-berlin.de> writes:
> 
> > I wrote:
> > 
> > > I'm really tempted to allocate heaps using mmap()
> > > (anonymous) to get them out of the way of the zalloc() allocator. I
> > > small test showed that with this I only get 39KB of free memory after
> > > the completion test, which is really not too bad. I don't have a patch 
> > > for that yet, though.
> > 
> > ... and now I don't think I'll ever write one: neither Solaris nor
> > Linux seem to have MAP_ANONYMOUS. Sniff.
> 
> Hmmm ... how are you calling mmap exactly?  I've definitely used
> MAP_ANONYMOUS on Linux (as far back as kernel 1.3) and I think I've used it
> on Solaris too.  Linux doesn't have shared anonymous mappings, i.e. you have
> to use MAP_PRIVATE with MAP_ANONYMOUS.

I had only looked at the manual and there MAP_ANONYMOUS doesn't appear 
in the list -- only in a short comment near the end.

For Solaris, however: the 5.6 box I have access to here really doesn't 
have a MAP_ANONYMOUS in the header file.

Anyway, below is the patch to make zsh heaps allocated using
mmap(). At least for me this keeps the amount of unused space on the
real heap nicely small. I don't see any disadvantages with this.


BUT, of course, the real problem is the way we store shell code, which 
eats a lot of space. After the next official release I'd like to try
to write a byte-code machine for zsh; that should also make things
faster... I don't know if I'll find enough free time, though.


Bye
 Sven

diff -ru ../z.old/Src/mem.c Src/mem.c
--- ../z.old/Src/mem.c	Fri Dec 17 11:40:35 1999
+++ Src/mem.c	Fri Dec 17 11:43:04 1999
@@ -64,15 +64,29 @@
 	to duplicate a structure in order to preserve it (i.e. a function
 	definition), call permalloc(), then dupstruct().
 
-	If we use zsh's own allocator we use a simple trick to avoid that
-	the (*real*) heap fills up with empty zsh-heaps: we allocate a
-	large block of memory before allocating a heap pool, this memory
-	is freed again immediately after the pool is allocated. If there
-	are only small blocks on the free list this guarantees that the
-	memory for the pool is at the end of the memory which means that
-	we can give it back to the system when the pool is freed.
+	If possible, the heaps are allocated using mmap() so that the
+	(*real*) heap isn't filled up with empty zsh heaps. If mmap()
+	is not available and zsh's own allocator we use a simple trick
+	to avoid that: we allocate a large block of memory before allocating
+	a heap pool, this memory is freed again immediately after the pool
+	is allocated. If there are only small blocks on the free list this
+	guarantees that the memory for the pool is at the end of the memory
+	which means that we can give it back to the system when the pool is
+	freed.
 */
 
+#if defined(HAVE_SYS_MMAN_H) && defined(HAVE_MMAP) && defined(HAVE_MUNMAP)
+
+#include <sys/mman.h>
+
+#if defined(MAP_ANONYMOUS) && defined(MAP_PRIVATE)
+
+#define USE_MMAP 1
+#define MMAP_FLAGS (MAP_ANONYMOUS | MAP_PRIVATE)
+
+#endif
+#endif
+
 /* != 0 if we are allocating in the heaplist */
  
 /**/
@@ -232,8 +246,13 @@
 	    if (!fheap && h->used < HEAP_ARENA_SIZE)
 		fheap = h;
 	    hl = h;
-	} else
+	} else {
+#ifdef USE_MMAP
+	    munmap((void *) h, h->size);
+#else
 	    zfree(h, HEAPSIZE);
+#endif
+	}
     }
     if (hl)
 	hl->next = NULL;
@@ -268,8 +287,13 @@
 	    zfree(hs, sizeof(*hs));
 
 	    hl = h;
-	} else
+	} else {
+#ifdef USE_MMAP
+	    munmap((void *) h, h->size);
+#else
 	    zfree(h, HEAPSIZE);
+#endif
+	}
     }
     if (hl)
 	hl->next = NULL;
@@ -303,7 +327,7 @@
     {
 	Heap hp;
         /* not found, allocate new heap */
-#ifdef ZSH_MEM
+#if defined(ZSH_MEM) && !defined(USE_MMAP)
 	static int called = 0;
 	void *foo = called ? (void *)malloc(HEAPFREE) : NULL;
             /* tricky, see above */
@@ -313,9 +337,34 @@
 	n = HEAP_ARENA_SIZE > size ? HEAPSIZE : size + sizeof(*h);
 	for (hp = NULL, h = heaps; h; hp = h, h = h->next);
 
+#ifdef USE_MMAP
+	{
+	    static size_t pgsz = 0;
+
+	    if (!pgsz) {
+
+#ifdef _SC_PAGESIZE
+		pgsz = sysconf(_SC_PAGESIZE);     /* SVR4 */
+#else
+# ifdef _SC_PAGE_SIZE
+		pgsz = sysconf(_SC_PAGE_SIZE);    /* HPUX */
+# else
+		pgsz = getpagesize();
+# endif
+#endif
+
+		pgsz--;
+	    }
+	    n = (n + pgsz) & ~pgsz;
+	    h = (Heap) mmap(NULL, n, PROT_READ | PROT_WRITE,
+			    MMAP_FLAGS, -1, 0);
+	    h->size = n;
+	}
+#else
 	h = (Heap) zalloc(n);
+#endif
 
-#ifdef ZSH_MEM
+#if defined(ZSH_MEM) && !defined(USE_MMAP)
 	if (called)
 	    zfree(foo, HEAPFREE);
 	called = 1;
@@ -381,7 +430,11 @@
 	    else
 		heaps = h->next;
 	    fheap = NULL;
+#ifdef USE_MMAP
+	    munmap((void *) h, h->size);
+#else
 	    zfree(h, HEAPSIZE);
+#endif
 	    return NULL;
 	}
 	if (old > HEAP_ARENA_SIZE || new > HEAP_ARENA_SIZE) {
@@ -696,7 +749,9 @@
 {
     struct m_hdr *m, *mp, *mt;
     long n, s, os = 0;
+#ifndef USE_MMAP
     struct heap *h, *hp, *hf = NULL, *hfp = NULL;
+#endif
 
     /* some systems want malloc to return the highest valid address plus one
        if it is called with an argument of zero */
@@ -769,12 +824,14 @@
     } else
 	s = 0;
 
-/* search the free list for an block of at least the requested size */
+    /* search the free list for an block of at least the requested size */
     for (mp = NULL, m = m_free; m && m->len < size; mp = m, m = m->next);
 
- /* if there is an empty zsh heap at a lower address we steal it and take
-    the memory from it, putting the rest on the free list (remember
-    that the blocks on the free list are ordered) */
+#ifndef USE_MMAP
+
+    /* if there is an empty zsh heap at a lower address we steal it and take
+       the memory from it, putting the rest on the free list (remember
+       that the blocks on the free list are ordered) */
 
     for (hp = NULL, h = heaps; h; hp = h, h = h->next)
 	if (!h->used &&
@@ -801,6 +858,7 @@
 
 	for (mp = NULL, m = m_free; m && m->len < size; mp = m, m = m->next);
     }
+#endif
     if (!m) {
 	long nal;
 	/* no matching free block was found, we have to request new
diff -ru ../z.old/Src/zsh.h Src/zsh.h
--- ../z.old/Src/zsh.h	Fri Dec 17 11:40:37 1999
+++ Src/zsh.h	Fri Dec 17 11:43:04 1999
@@ -1556,6 +1556,7 @@
 
 struct heap {
     struct heap *next;		/* next one                                  */
+    size_t size;		/* size of heap                              */
     size_t used;		/* bytes used from the heap                  */
     struct heapstack *sp;	/* used by pushheap() to save the value used */
 #ifdef PAD_64_BIT

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: zsh and memory

[Sven Wischnowsky]

Bart Schaefer wrote:

> On Dec 16, 10:56am, Sven Wischnowsky wrote:
> } Subject: Re: zsh and memory
> }
> } [...] I looked who was using lots of heap memory and found the
> } tokstr handling in lex.c. There we always allocate at least 256 bytes
> } and if the buffer needs expanding (in add()) in gets resized to
> } inbufct (or larger). Some more investigation showed that we almost
> } never need a tokstr with more than 32 bytes, so I changed that.  I also 
> } changed add() to be more careful when expanding the buffer -- I left
> } the old code conditioned out because I don't kno if there was a reason 
> } to resize it to inbufct bytes; I at least don't see a reason for that.
> 
> Probably the idea was to eliminate lots of small allocations in the
> event that the "token" is a long quoted string or the like, for speed.
> We should watch out for performance problems if those changes are kept.

I know, but always enlarging the buffer to inbufct chars means that
if a somewhat longer string in an autoloaded function is encountered,
the buffer is enlarged to the size of the function file. And this will 
only seldom (never?) be the right thing.

The patch below goes on top of the previous one and doubles the size
of the buffer in add(), limiting it to inbufct chars. I hope, this is
a good compromise.

Bye
 Sven

diff -ru ../z.old/Src/lex.c Src/lex.c
--- ../z.old/Src/lex.c	Fri Dec 17 10:04:36 1999
+++ Src/lex.c	Fri Dec 17 11:36:08 1999
@@ -480,7 +480,10 @@
 	bsiz = newbsiz;
 #endif
 
-	int newbsiz = bsiz + 32;
+	int newbsiz = bsiz * 2;
+
+	if (newbsiz > inbufct && inbufct > bsiz)
+	    newbsiz = inbufct;
 
 	bptr = len + (tokstr = (char *)hrealloc(tokstr, bsiz, newbsiz));
 	bsiz = newbsiz;

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: zsh and memory

[Timothy Writer]

Sven Wischnowsky <wischnow@informatik.hu-berlin.de> writes:

> I wrote:
> 
> > I'm really tempted to allocate heaps using mmap()
> > (anonymous) to get them out of the way of the zalloc() allocator. I
> > small test showed that with this I only get 39KB of free memory after
> > the completion test, which is really not too bad. I don't have a patch 
> > for that yet, though.
> 
> ... and now I don't think I'll ever write one: neither Solaris nor
> Linux seem to have MAP_ANONYMOUS. Sniff.

Hmmm ... how are you calling mmap exactly?  I've definitely used
MAP_ANONYMOUS on Linux (as far back as kernel 1.3) and I think I've used it
on Solaris too.  Linux doesn't have shared anonymous mappings, i.e. you have
to use MAP_PRIVATE with MAP_ANONYMOUS.

-- 
tim writer <tim@starnix.com>                                  starnix inc.
tollfree: 1-87-pro-linux                         brampton, ontario, canada
http://www.starnix.com              professional linux services & products

Re: zsh and memory

[Mike Fletcher]

>>>>> "Sven" == Sven Wischnowsky <wischnow@informatik.hu-berlin.de> writes:

    Sven> I wrote:

    >> I'm really tempted to allocate heaps using mmap() (anonymous)
    >> to get them out of the way of the zalloc() allocator. I small
    >> test showed that with this I only get 39KB of free memory after
    >> the completion test, which is really not too bad. I don't have
    >> a patch for that yet, though.

    Sven> ... and now I don't think I'll ever write one: neither
    Sven> Solaris nor Linux seem to have MAP_ANONYMOUS. Sniff.

        My copy seems to, at least according to mmap(2):

[...]
       The  above three flags are described in POSIX.1b (formerly
       POSIX.4).  Linux also knows about MAP_DENYWRITE,  MAP_EXE-
       CUTABLE and MAP_ANON(YMOUS).
[...]

        There's also Ralf Engelschall's mm library that hides shared
memory allocations in a platform independent way:

http://www.engelschall.com/sw/mm/

        But that's more for handling shared segments between multiple
processes.  Of course there was that talk of making all zsh processes
share the same zcompdump . . . :).

-- 
Fletch                | "If you find my answers frightening,       __`'/|
fletch@phydeaux.org   |  Vincent, you should cease askin'          \ o.O'
678 443-6239(w)       |  scary questions." -- Jules                =(___)=
                      |                                               U

Re: zsh and memory

[Sven Wischnowsky]

I wrote:

> I'm really tempted to allocate heaps using mmap()
> (anonymous) to get them out of the way of the zalloc() allocator. I
> small test showed that with this I only get 39KB of free memory after
> the completion test, which is really not too bad. I don't have a patch 
> for that yet, though.

... and now I don't think I'll ever write one: neither Solaris nor
Linux seem to have MAP_ANONYMOUS. Sniff.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de