Re: Bug#245678: zsh: built-in rm -rf fills up the memory

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Clint Adams]

> When I do a rm -rf on a big maildir directory, zsh fills up the
> memory, which makes the whole machine unresponsive (the load of
> the machine becomes very high according to "xload", and the mouse
> pointer no longer moves) and random processes be killed, before
> zsh itself dies. According to "top", the used memory by zsh was
> up to 583 MB. And nothing was actually removed.

I can reproduce the memory consumption with a 7210-file Maildir, except
that my kernel does the right thing and kills the runaway zsh.

If I abort the rm before it OOMs, zsh will continue to take up a large
amount of memory until it exits.

Here's some valgrind output from one such run.

==31822== Memcheck, a memory error detector for x86-linux.
==31822== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward.
==31822== Using valgrind-2.1.1, a program supervision framework for x86-linux.
==31822== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward.
==31822== 
==31822== My PID = 31822, parent PID = 1021.  Prog and args are:
==31822==    zsh
==31822==    -f
==31822== 
==31822== Valgrind library directory: /usr/lib/valgrind
==31822== Command line
==31822==    zsh
==31822==    -f
==31822== Startup, with flags:
==31822==    --tool=memcheck
==31822==    --logfile=/tmp/zshremove
==31822==    -v
==31822==    --leak-check=yes
==31822==    --show-reachable=yes
==31822==    --
==31822== Reading syms from /bin/zsh4 (0x8048000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/ld-2.3.2.so (0x3C000000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/ld-2.3.2.so (0xB0000000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/tls/i686/cmov/libdl-2.3.2.so (0xB0020000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/tls/i686/cmov/libc-2.3.2.so (0xB0023000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /usr/lib/valgrind/vgskin_memcheck.so (0xB035D000)
==31822== Reading syms from /usr/lib/valgrind/stage2 (0xB8000000)
==31822== Reading suppressions file: /usr/lib/valgrind/default.supp
==31822== REDIRECT soname:libc.so.6(__GI___errno_location) to soname:libpthread.so.0(__errno_location)
==31822== REDIRECT soname:libc.so.6(__errno_location) to soname:libpthread.so.0(__errno_location)
==31822== REDIRECT soname:libc.so.6(__GI___h_errno_location) to soname:libpthread.so.0(__h_errno_location)
==31822== REDIRECT soname:libc.so.6(__h_errno_location) to soname:libpthread.so.0(__h_errno_location)
==31822== REDIRECT soname:libc.so.6(__GI___res_state) to soname:libpthread.so.0(__res_state)
==31822== REDIRECT soname:libc.so.6(__res_state) to soname:libpthread.so.0(__res_state)
==31822== REDIRECT soname:libc.so.6(stpcpy) to *vgpreload_memcheck.so*(stpcpy)
==31822== REDIRECT soname:libc.so.6(strnlen) to *vgpreload_memcheck.so*(strnlen)
==31822== REDIRECT soname:ld-linux.so.2(stpcpy) to *vgpreload_memcheck.so*(stpcpy)
==31822== REDIRECT soname:ld-linux.so.2(strchr) to *vgpreload_memcheck.so*(strchr)
==31822== 
==31822== Reading syms from /usr/lib/valgrind/vg_inject.so (0x3C01A000)
==31822== Reading syms from /usr/lib/valgrind/vgpreload_memcheck.so (0x3C01D000)
==31822== Reading syms from /lib/libcap.so.1.10 (0x3C02E000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/tls/i686/cmov/libdl-2.3.2.so (0x41179000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/tls/i686/cmov/libnsl-2.3.2.so (0x3C033000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/libncurses.so.5.4 (0x3C049000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/tls/i686/cmov/libm-2.3.2.so (0x41154000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/tls/i686/cmov/libc-2.3.2.so (0x41019000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/tls/i686/cmov/libnss_compat-2.3.2.so (0x3C2AE000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/tls/i686/cmov/libnss_nis-2.3.2.so (0x3C2B7000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /lib/tls/i686/cmov/libnss_files-2.3.2.so (0x3C2C2000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==  Address 0x4FFFE0D4 is on thread 1's stack
==31822== TRANSLATE: 0x41092130 redirected to 0x3C01EC70
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x0: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1F: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3F: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7F: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== Reading syms from /usr/lib/zsh/4.1.1/zsh/zle.so (0x3C2CD000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /usr/lib/zsh/4.1.1/zsh/complete.so (0x3C2FC000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /usr/lib/zsh/4.1.1/zsh/compctl.so (0x3C31A000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== Reading syms from /usr/lib/zsh/4.1.1/zsh/files.so (0x3C329000)
==31822==    object doesn't have a symbol table
==31822==    object doesn't have any debug info
==31822== 
==31822== Invalid read of size 4
==31822==    at 0x4103003B: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==  Address 0x3C0BD7CC is 8 bytes inside a block of size 60 free'd
==31822==    at 0x3C01F918: free (vg_replace_malloc.c:127)
==31822==    by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== Invalid write of size 4
==31822==    at 0x4103004E: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==  Address 0x3C0BD7CC is 8 bytes inside a block of size 60 free'd
==31822==    at 0x3C01F918: free (vg_replace_malloc.c:127)
==31822==    by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== Invalid read of size 4
==31822==    at 0x41030053: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==  Address 0x3C0BD7E4 is 32 bytes inside a block of size 60 free'd
==31822==    at 0x3C01F918: free (vg_replace_malloc.c:127)
==31822==    by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== Invalid read of size 4
==31822==    at 0x4103005A: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==  Address 0x3C0BD7C4 is 0 bytes inside a block of size 60 free'd
==31822==    at 0x3C01F918: free (vg_replace_malloc.c:127)
==31822==    by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== discard syms at 0x3C2AE000-0x3C2B6000 in /lib/tls/i686/cmov/libnss_compat-2.3.2.so due to munmap()
==31822== discard syms at 0x3C2B7000-0x3C2C1000 in /lib/tls/i686/cmov/libnss_nis-2.3.2.so due to munmap()
==31822== discard syms at 0x3C2C2000-0x3C2CC000 in /lib/tls/i686/cmov/libnss_files-2.3.2.so due to munmap()
==31822== 
==31822== ERROR SUMMARY: 110 errors from 36 contexts (suppressed: 32 from 1)
==31822== 
==31822== 1 errors in context 1 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 2 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 3 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 4 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 5 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 6 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 7 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 8 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 9 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 10 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 11 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 12 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 13 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 14 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 15 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 16 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 17 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 18 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 19 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 20 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7F: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 21 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3F: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 22 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1F: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 23 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 24 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 1 errors in context 25 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 2 errors in context 26 of 36:
==31822== Invalid read of size 4
==31822==    at 0x4103005A: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==  Address 0x3C0BD7C4 is 0 bytes inside a block of size 60 free'd
==31822==    at 0x3C01F918: free (vg_replace_malloc.c:127)
==31822==    by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== 2 errors in context 27 of 36:
==31822== Invalid read of size 4
==31822==    at 0x41030053: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==  Address 0x3C0BD7E4 is 32 bytes inside a block of size 60 free'd
==31822==    at 0x3C01F918: free (vg_replace_malloc.c:127)
==31822==    by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== 2 errors in context 28 of 36:
==31822== Invalid write of size 4
==31822==    at 0x4103004E: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==  Address 0x3C0BD7CC is 8 bytes inside a block of size 60 free'd
==31822==    at 0x3C01F918: free (vg_replace_malloc.c:127)
==31822==    by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== 2 errors in context 29 of 36:
==31822== Invalid read of size 4
==31822==    at 0x4103003B: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==  Address 0x3C0BD7CC is 8 bytes inside a block of size 60 free'd
==31822==    at 0x3C01F918: free (vg_replace_malloc.c:127)
==31822==    by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== 2 errors in context 30 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0xFFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 2 errors in context 31 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x7FFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 2 errors in context 32 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x1FFFFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 2 errors in context 33 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3FFF: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 2 errors in context 34 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x0: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 2 errors in context 35 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==    by 0x3: ???
==31822==  Address 0x4FFFE75C is on thread 1's stack
==31822== 
==31822== 65 errors in context 36 of 36:
==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s)
==31822==    at 0x3C000C02: (within /lib/ld-2.3.2.so)
==31822==  Address 0x4FFFE0D4 is on thread 1's stack
--31822-- 
--31822-- supp:   32 Ugly strchr error in /lib/ld-2.3.2.so
==31822== 
==31822== IN SUMMARY: 110 errors from 36 contexts (suppressed: 32 from 1)
==31822== 
==31822== malloc/free: in use at exit: 51472 bytes in 932 blocks.
==31822== malloc/free: 3294 allocs, 2362 frees, 175556 bytes allocated.
==31822== 
==31822== searching for pointers to 932 not-freed blocks.
==31822== checked 2918772 bytes.
==31822== 
==31822== 
==31822== 6 bytes in 1 blocks are still reachable in loss record 1 of 24
==31822==    at 0x3C01FD01: realloc (vg_replace_malloc.c:162)
==31822==    by 0x3C07086F: _nc_doalloc (in /lib/libncurses.so.5.4)
==31822==    by 0x3C0737AE: (within /lib/libncurses.so.5.4)
==31822==    by 0x3C073401: tparm (in /lib/libncurses.so.5.4)
==31822== 
==31822== 
==31822== 12 bytes in 2 blocks are still reachable in loss record 2 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x41091DAA: strdup (in /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== 
==31822== 24 bytes in 1 blocks are still reachable in loss record 3 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x3C070AB5: _nc_home_terminfo (in /lib/libncurses.so.5.4)
==31822==    by 0x3C077A72: _nc_read_entry (in /lib/libncurses.so.5.4)
==31822==    by 0x3C071FBE: (within /lib/libncurses.so.5.4)
==31822== 
==31822== 
==31822== 28 bytes in 1 blocks are still reachable in loss record 4 of 24
==31822==    at 0x3C01FC03: calloc (vg_replace_malloc.c:141)
==31822==    by 0x3C07709F: (within /lib/libncurses.so.5.4)
==31822==    by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4)
==31822==    by 0x3C0778A1: (within /lib/libncurses.so.5.4)
==31822== 
==31822== 
==31822== 36 bytes in 1 blocks are definitely lost in loss record 5 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x41104618: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== 
==31822== 40 bytes in 2 blocks are still reachable in loss record 6 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x3C01FC88: realloc (vg_replace_malloc.c:154)
==31822==    by 0x3C007D08: (within /lib/ld-2.3.2.so)
==31822==    by 0x3C007E8E: _dl_lookup_symbol (in /lib/ld-2.3.2.so)
==31822== 
==31822== 
==31822== 40 bytes in 5 blocks are still reachable in loss record 7 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x4110418D: __nss_lookup_function (in /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==    by 0x3C2B0A2E: ???
==31822== 
==31822== 
==31822== 44 bytes in 1 blocks are still reachable in loss record 8 of 24
==31822==    at 0x3C01FC03: calloc (vg_replace_malloc.c:141)
==31822==    by 0x3C0770EB: (within /lib/libncurses.so.5.4)
==31822==    by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4)
==31822==    by 0x3C0778A1: (within /lib/libncurses.so.5.4)
==31822== 
==31822== 
==31822== 76 bytes in 1 blocks are still reachable in loss record 9 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x41125906: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== 
==31822== 78 bytes in 1 blocks are still reachable in loss record 10 of 24
==31822==    at 0x3C01FC03: calloc (vg_replace_malloc.c:141)
==31822==    by 0x3C077152: (within /lib/libncurses.so.5.4)
==31822==    by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4)
==31822==    by 0x3C0778A1: (within /lib/libncurses.so.5.4)
==31822== 
==31822== 
==31822== 79 bytes in 1 blocks are still reachable in loss record 11 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x3C0708A7: _nc_doalloc (in /lib/libncurses.so.5.4)
==31822==    by 0x3C0733B5: _nc_tparm_analyze (in /lib/libncurses.so.5.4)
==31822==    by 0x3C073631: (within /lib/libncurses.so.5.4)
==31822== 
==31822== 
==31822== 80 bytes in 5 blocks are still reachable in loss record 12 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x410F1D9B: tsearch (in /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== 
==31822== 131 bytes in 4 blocks are still reachable in loss record 13 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x3C00913E: (within /lib/ld-2.3.2.so)
==31822==    by 0x3C004C8B: (within /lib/ld-2.3.2.so)
==31822==    by 0x3C0064A5: _dl_map_object (in /lib/ld-2.3.2.so)
==31822== 
==31822== 
==31822== 131 bytes in 4 blocks are still reachable in loss record 14 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x3C0064EB: _dl_map_object (in /lib/ld-2.3.2.so)
==31822==    by 0x4112502E: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==    by 0x3C00BF25: _dl_catch_error (in /lib/ld-2.3.2.so)
==31822== 
==31822== 
==31822== 172 bytes in 1 blocks are still reachable in loss record 15 of 24
==31822==    at 0x3C01FC03: calloc (vg_replace_malloc.c:141)
==31822==    by 0x3C072307: setupterm (in /lib/libncurses.so.5.4)
==31822==    by 0x3C0727FE: tgetent (in /lib/libncurses.so.5.4)
==31822==    by 0x8073BD9: init_term (in /bin/zsh4)
==31822== 
==31822== 
==31822== 256 bytes in 1 blocks are still reachable in loss record 16 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x80820C6: zrealloc (in /bin/zsh4)
==31822==    by 0x80939C7: patcompile (in /bin/zsh4)
==31822==    by 0x805F32F: evalcond (in /bin/zsh4)
==31822== 
==31822== 
==31822== 272 bytes in 4 blocks are still reachable in loss record 17 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x3C00AE9A: _dl_map_object_deps (in /lib/ld-2.3.2.so)
==31822==    by 0x411250D5: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==    by 0x3C00BF25: _dl_catch_error (in /lib/ld-2.3.2.so)
==31822== 
==31822== 
==31822== 352 bytes in 4 blocks are still reachable in loss record 18 of 24
==31822==    at 0x3C01FC03: calloc (vg_replace_malloc.c:141)
==31822==    by 0x3C00D232: _dl_check_map_versions (in /lib/ld-2.3.2.so)
==31822==    by 0x411255E8: (within /lib/tls/i686/cmov/libc-2.3.2.so)
==31822==    by 0x3C00BF25: _dl_catch_error (in /lib/ld-2.3.2.so)
==31822== 
==31822== 
==31822== 352 bytes in 1 blocks are still reachable in loss record 19 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x4107AC0A: _IO_fdopen (in /lib/tls/i686/cmov/libc-2.3.2.so)
==31822== 
==31822== 
==31822== 1168 bytes in 1 blocks are still reachable in loss record 20 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x3C076FD8: (within /lib/libncurses.so.5.4)
==31822==    by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4)
==31822==    by 0x3C0778A1: (within /lib/libncurses.so.5.4)
==31822== 
==31822== 
==31822== 1656 bytes in 1 blocks are still reachable in loss record 21 of 24
==31822==    at 0x3C01FC03: calloc (vg_replace_malloc.c:141)
==31822==    by 0x3C0771C9: (within /lib/libncurses.so.5.4)
==31822==    by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4)
==31822==    by 0x3C0778A1: (within /lib/libncurses.so.5.4)
==31822== 
==31822== 
==31822== 2387 bytes in 4 blocks are still reachable in loss record 22 of 24
==31822==    at 0x3C01FC03: calloc (vg_replace_malloc.c:141)
==31822==    by 0x3C008F23: (within /lib/ld-2.3.2.so)
==31822==    by 0x3C004C8B: (within /lib/ld-2.3.2.so)
==31822==    by 0x3C0064A5: _dl_map_object (in /lib/ld-2.3.2.so)
==31822== 
==31822== 
==31822== 19716 bytes in 264 blocks are still reachable in loss record 23 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x8081DBA: zshcalloc (in /bin/zsh4)
==31822==    by 0x8075721: zsh_main (in /bin/zsh4)
==31822==    by 0x805210A: main (in /bin/zsh4)
==31822== 
==31822== 
==31822== 24336 bytes in 621 blocks are still reachable in loss record 24 of 24
==31822==    at 0x3C01F40D: malloc (vg_replace_malloc.c:105)
==31822==    by 0x8081C95: zalloc (in /bin/zsh4)
==31822==    by 0x809A725: ztrdup (in /bin/zsh4)
==31822==    by 0x8073387: parseargs (in /bin/zsh4)
==31822== 
==31822== LEAK SUMMARY:
==31822==    definitely lost: 36 bytes in 1 blocks.
==31822==    possibly lost:   0 bytes in 0 blocks.
==31822==    still reachable: 51436 bytes in 931 blocks.
==31822==         suppressed: 0 bytes in 0 blocks.
--31822--     TT/TC: 0 tc sectors discarded.
--31822--            8091 chainings, 0 unchainings.
--31822-- translate: new     11409 (178624 -> 2283438; ratio 127:10)
--31822--            discard 180 (2199 -> 30012; ratio 136:10).
--31822--  dispatch: 13550000 jumps (bb entries), of which 201341 (1%) were unchained.
--31822--            1497/21775 major/minor sched events.  12377 tt_fast misses.
--31822-- reg-alloc: 1808 t-req-spill, 417567+11458 orig+spill uis, 56546 total-reg-r.
--31822--    sanity: 1419 cheap, 57 expensive checks.
--31822--    ccalls: 39857 C calls, 56% saves+restores avoided (132048 bytes)
--31822--            54348 args, avg 0.86 setup instrs each (14802 bytes)
--31822--            0% clear the stack (119064 bytes)
--31822--            15881 retvals, 31% of reg-reg movs avoided (9546 bytes)

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Bart Schaefer]

On Apr 24,  9:36pm, Clint Adams wrote:
}
} I can reproduce the memory consumption with a 7210-file Maildir, except
} that my kernel does the right thing and kills the runaway zsh.
} 
} If I abort the rm before it OOMs, zsh will continue to take up a large
} amount of memory until it exits.

There's got to be something here you're not telling us.  Either there is
an alias for "rm", or a function named "rm", or the zsh/files module is
loaded; otherwise "rm" and "/bin/rm" should be equivalent (assuming there
is no other "rm" in the path) and in any case shouldn't execute inside
the shell.

Assuming that it's the zsh/files module, the interesting thing there is
that, because it's a depth-first traversal, it allocates zsh-heap memory
to store all the file names in the directory, and it does so by growing
the allocated heap block once for each filename, by the length of that
filename (plus one).  If ZSH_MEM was not defined at compile time, doing
this efficiently is left up to the underlying malloc() implementation [if
I'm reading mem.c correctly] and therefore could be subject to bugs we've
seen before where malloc never reclaims nor consolidates blocks when the
implementation repeatedly realloc()s in slightly larger chunks.

I don't know anything about how zsh is compiled on debian or even why
the zsh/files module would be used (seemingly) by default, but in that's
probably the place to start looking.

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Clint Adams]

> Assuming that it's the zsh/files module, the interesting thing there is
> that, because it's a depth-first traversal, it allocates zsh-heap memory
> to store all the file names in the directory, and it does so by growing
> the allocated heap block once for each filename, by the length of that
> filename (plus one).  If ZSH_MEM was not defined at compile time, doing
> this efficiently is left up to the underlying malloc() implementation [if
> I'm reading mem.c correctly] and therefore could be subject to bugs we've
> seen before where malloc never reclaims nor consolidates blocks when the
> implementation repeatedly realloc()s in slightly larger chunks.
> 
> I don't know anything about how zsh is compiled on debian or even why
> the zsh/files module would be used (seemingly) by default, but in that's
> probably the place to start looking.

Yes, sorry.  We're talking about when zsh/files is loaded explicitly.
ZSH_MEM isn't being defined.

Is there perhaps a mallopt() setting which would help?

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Clint Adams]

> Yes, sorry.  We're talking about when zsh/files is loaded explicitly.
> ZSH_MEM isn't being defined.

Additionally, a recompile with ZSH_MEM shows that it still OOMs (perhaps
much more slowly, but that could be an error in my perception).

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Bart Schaefer]

On Apr 25,  5:12pm, Clint Adams wrote:
}
} Is there perhaps a mallopt() setting which would help?

I'm not all that familiar with mallopt() but a look at the documentation
doesn't suggest anything.

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Clint Adams]

> I'm not all that familiar with mallopt() but a look at the documentation
> doesn't suggest anything.

I think I've found the problem.  Here's an excerpt from an strace of
the zsh/files rm -rf going wild.  There are 55 or 56 such calls to mmap
for every length value.  Thus it appears that zsh is allocating
considerably more memory than it needs.

mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x583d9000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58415000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58451000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5848d000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x584c9000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58505000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58541000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5857d000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x585b9000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x585f5000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58631000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5866d000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x586a9000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x586e5000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58721000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5875d000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58799000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x587d5000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58811000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5884d000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58889000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x588c5000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58901000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5893d000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58979000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x589b5000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x589f1000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58a2d000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58a69000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58aa5000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58ae1000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58b1d000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58b59000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58b95000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58bd1000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58c0d000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58c49000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58c85000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58cc1000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58cfd000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58d39000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58d75000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58db1000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58ded000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58e29000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58e65000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58ea1000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58edd000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58f19000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58f55000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58f91000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58fcd000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x59009000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x59045000
mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x59081000

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Clint Adams]

> I think I've found the problem.  Here's an excerpt from an strace of
> the zsh/files rm -rf going wild.  There are 55 or 56 such calls to mmap
> for every length value.  Thus it appears that zsh is allocating
> considerably more memory than it needs.

More on this:

recursivecmd_dorec() calls hrealloc() for each target file, incrementing
size accordingly.  hrealloc() rounds the new length up to the nearest
page boundary, and mmaps a new Heap for each file.

So, for example, if you have a directory with one hundred files with
9-byte filenames, zsh might allocate one hundred 4096-byte areas.  If
you have several thousand files with 70- to 80-byte filenames, zsh
might allocate over 50 4096-byte areas, over 50 8192-byte areas, and so
on.

Now, should this be fixed by making hrealloc() smarter or
recursivecmd_dorec() more efficient?

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Clint Adams]

> Now, should this be fixed by making hrealloc() smarter or
> recursivecmd_dorec() more efficient?

Here's one non-portable way of preventing the OOM, and making the rm -rf
go quickly.

Should H_ISIZE be changed to the page size when USE_MMAP is defined, or
is there a reason to keep it the sizeof(union mem_align) ?

--- orig/Src/mem.c
+++ mod/Src/mem.c
@@ -379,8 +379,8 @@
 {
     Heap h, ph;
 
-    old = (old + H_ISIZE - 1) & ~(H_ISIZE - 1);
-    new = (new + H_ISIZE - 1) & ~(H_ISIZE - 1);
+    old = (old + 4096 - 1) & ~(4096 - 1);
+    new = (new + 4096 - 1) & ~(4096 - 1);
 
     if (old == new)
 	return p;

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Bart Schaefer]

On Sat, 8 May 2004, Clint Adams wrote:

> So, for example, if you have a directory with one hundred files with
> 9-byte filenames, zsh might allocate one hundred 4096-byte areas.

If that's really what's happening, it's a bug somewhere else, because zsh
should only allocate another heap page if the current heap page doesn't
have room for the requested growth.

That is, 409 9-byte file names should need 4090 bytes (add 1 to each for
the NUL byte), and it should put all 4090 of those bytes in the same
4096-byte heap block.  It shouldn't allocate a new heap block until it
gets to the 410th file, at which point it should allocate another
8192-byte block and copy the old block into it, and return the original
4096-byte block to the heap pool for re-use.

(Where I'm assuming HEAP_ARENA_SIZE is 4096 here.)

The following code should *never* be executed unless hrealloc() has
previously been used to *shrink* a block:

    if (p + old < arena(h) + h->used) {
	if (new > old) {
	    char *ptr = (char *) zhalloc(new);
	    memcpy(ptr, p, old);
#ifdef ZSH_MEM_DEBUG
	    memset(p, 0xff, old);
#endif
	    unqueue_signals();
	    return ptr;
	} else {
	    unqueue_signals();
	    return new ? p : NULL;
	}
    }

Because unless we shrank, (p + old == arena(h) + h->used), as asserted by
the DPUTS() that immediately follows.

So, taking #ifdefs for MMAP into account, this code is what should run:

    if (h->used + (new - old) <= HEAP_ARENA_SIZE) {
	h->used += new - old;
	unqueue_signals();
	return p;
    } else {
	char *t = zhalloc(new);
	memcpy(t, p, old > new ? new : old);
	h->used -= old;
#ifdef ZSH_MEM_DEBUG
	memset(p, 0xff, old);
#endif
	unqueue_signals();
	return t;
    }

And you're saying that (h->used + (new - old) <= HEAP_ARENA_SIZE) is 
always false, so the zhalloc() is always called?

If that's not what you find to be the case, then the bug is elsewhere and
we need to keep looking.  If it is what you find, then there's a problem
with h->used record-keeping, or something.

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Bart Schaefer]

On Sat, 8 May 2004, Clint Adams wrote:

> Should H_ISIZE be changed to the page size when USE_MMAP is defined, or
> is there a reason to keep it the sizeof(union mem_align) ?

Changing H_ISIZE won't fix the underlying cause of the bug, it'll just 
cause it to manifest only once every 4096/sizeof(union mem_align) calls
rather than on every call.

We need to find out what code branch is calling zhalloc() on every small
increment, and why that code branch is being taken.

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Clint Adams]

> And you're saying that (h->used + (new - old) <= HEAP_ARENA_SIZE) is 
> always false, so the zhalloc() is always called?

I don't know that it's always false, but here's a small snapshot of
those comparisons during a zsh/files rm -rf attempt.

[ h->used + (new - old) > HEAP_ARENA_SIZE ]
63432 + (63504 - 63432) > 16360
63504 + (63576 - 63504) > 16360
63576 + (63648 - 63576) > 16360
63648 + (63728 - 63648) > 16360
63728 + (63800 - 63728) > 16360
63800 + (63872 - 63800) > 16360
63872 + (63944 - 63872) > 16360
63944 + (64024 - 63944) > 16360
64024 + (64096 - 64024) > 16360
64096 + (64168 - 64096) > 16360
64168 + (64240 - 64168) > 16360
64240 + (64320 - 64240) > 16360

> If that's not what you find to be the case, then the bug is elsewhere and
> we need to keep looking.  If it is what you find, then there's a problem
> with h->used record-keeping, or something.

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Clint Adams]

> [ h->used + (new - old) > HEAP_ARENA_SIZE ]
> 63432 + (63504 - 63432) > 16360
> 63504 + (63576 - 63504) > 16360

I added in h->size to the printf, to show the heap sizes.

60856 [out of 61440] + (60928 - 60856) > 16360
60928 [out of 61440] + (61008 - 60928) > 16360
61008 [out of 61440] + (61080 - 61008) > 16360
61080 [out of 61440] + (61152 - 61080) > 16360
61152 [out of 61440] + (61224 - 61152) > 16360
61224 [out of 61440] + (61304 - 61224) > 16360
61304 [out of 61440] + (61376 - 61304) > 16360
61376 [out of 61440] + (61448 - 61376) > 16360
61448 [out of 65536] + (61520 - 61448) > 16360
61520 [out of 65536] + (61600 - 61520) > 16360
61600 [out of 65536] + (61672 - 61600) > 16360
61672 [out of 65536] + (61744 - 61672) > 16360
61744 [out of 65536] + (61816 - 61744) > 16360
61816 [out of 65536] + (61896 - 61816) > 16360
61896 [out of 65536] + (61968 - 61896) > 16360
61968 [out of 65536] + (62040 - 61968) > 16360
62040 [out of 65536] + (62112 - 62040) > 16360
62112 [out of 65536] + (62184 - 62112) > 16360
62184 [out of 65536] + (62256 - 62184) > 16360
62256 [out of 65536] + (62328 - 62256) > 16360
62328 [out of 65536] + (62400 - 62328) > 16360
62400 [out of 65536] + (62472 - 62400) > 16360
62472 [out of 65536] + (62552 - 62472) > 16360
62552 [out of 65536] + (62624 - 62552) > 16360
62624 [out of 65536] + (62696 - 62624) > 16360
62696 [out of 65536] + (62768 - 62696) > 16360
62768 [out of 65536] + (62840 - 62768) > 16360
62840 [out of 65536] + (62912 - 62840) > 16360
62912 [out of 65536] + (62984 - 62912) > 16360
62984 [out of 65536] + (63056 - 62984) > 16360
63056 [out of 65536] + (63136 - 63056) > 16360
63136 [out of 65536] + (63208 - 63136) > 16360
63208 [out of 65536] + (63280 - 63208) > 16360
63280 [out of 65536] + (63352 - 63280) > 16360
63352 [out of 65536] + (63432 - 63352) > 16360
63432 [out of 65536] + (63504 - 63432) > 16360
63504 [out of 65536] + (63576 - 63504) > 16360
63576 [out of 65536] + (63648 - 63576) > 16360
63648 [out of 65536] + (63728 - 63648) > 16360
63728 [out of 65536] + (63800 - 63728) > 16360
63800 [out of 65536] + (63872 - 63800) > 16360
63872 [out of 65536] + (63944 - 63872) > 16360
63944 [out of 65536] + (64024 - 63944) > 16360
64024 [out of 65536] + (64096 - 64024) > 16360
64096 [out of 65536] + (64168 - 64096) > 16360
64168 [out of 65536] + (64240 - 64168) > 16360
64240 [out of 65536] + (64320 - 64240) > 16360
64320 [out of 65536] + (64392 - 64320) > 16360
64392 [out of 65536] + (64464 - 64392) > 16360
64464 [out of 65536] + (64536 - 64464) > 16360
64536 [out of 65536] + (64616 - 64536) > 16360
64616 [out of 65536] + (64688 - 64616) > 16360
64688 [out of 65536] + (64760 - 64688) > 16360
64760 [out of 65536] + (64832 - 64760) > 16360
64832 [out of 65536] + (64904 - 64832) > 16360
64904 [out of 65536] + (64984 - 64904) > 16360
64984 [out of 65536] + (65056 - 64984) > 16360
65056 [out of 65536] + (65128 - 65056) > 16360
65128 [out of 65536] + (65200 - 65128) > 16360
65200 [out of 65536] + (65280 - 65200) > 16360
65280 [out of 65536] + (65352 - 65280) > 16360
65352 [out of 65536] + (65424 - 65352) > 16360
65424 [out of 65536] + (65496 - 65424) > 16360
65496 [out of 65536] + (65576 - 65496) > 16360
65576 [out of 69632] + (65648 - 65576) > 16360
65648 [out of 69632] + (65720 - 65648) > 16360
65720 [out of 69632] + (65792 - 65720) > 16360
65792 [out of 69632] + (65872 - 65792) > 16360
65872 [out of 69632] + (65944 - 65872) > 16360
65944 [out of 69632] + (66016 - 65944) > 16360
66016 [out of 69632] + (66088 - 66016) > 16360
66088 [out of 69632] + (66168 - 66088) > 16360
66168 [out of 69632] + (66240 - 66168) > 16360
66240 [out of 69632] + (66312 - 66240) > 16360
66312 [out of 69632] + (66384 - 66312) > 16360
66384 [out of 69632] + (66456 - 66384) > 16360
66456 [out of 69632] + (66536 - 66456) > 16360
66536 [out of 69632] + (66608 - 66536) > 16360
66608 [out of 69632] + (66680 - 66608) > 16360
66680 [out of 69632] + (66752 - 66680) > 16360
66752 [out of 69632] + (66832 - 66752) > 16360
66832 [out of 69632] + (66904 - 66832) > 16360
66904 [out of 69632] + (66976 - 66904) > 16360
66976 [out of 69632] + (67048 - 66976) > 16360
67048 [out of 69632] + (67120 - 67048) > 16360
67120 [out of 69632] + (67200 - 67120) > 16360
67200 [out of 69632] + (67272 - 67200) > 16360
67272 [out of 69632] + (67344 - 67272) > 16360
67344 [out of 69632] + (67416 - 67344) > 16360
67416 [out of 69632] + (67488 - 67416) > 16360
67488 [out of 69632] + (67568 - 67488) > 16360
67568 [out of 69632] + (67640 - 67568) > 16360
67640 [out of 69632] + (67712 - 67640) > 16360
67712 [out of 69632] + (67784 - 67712) > 16360
67784 [out of 69632] + (67856 - 67784) > 16360
67856 [out of 69632] + (67928 - 67856) > 16360
67928 [out of 69632] + (68000 - 67928) > 16360
68000 [out of 69632] + (68072 - 68000) > 16360
68072 [out of 69632] + (68144 - 68072) > 16360
68144 [out of 69632] + (68216 - 68144) > 16360
68216 [out of 69632] + (68288 - 68216) > 16360
68288 [out of 69632] + (68360 - 68288) > 16360
68360 [out of 69632] + (68432 - 68360) > 16360
68432 [out of 69632] + (68504 - 68432) > 16360
68504 [out of 69632] + (68576 - 68504) > 16360
68576 [out of 69632] + (68648 - 68576) > 16360
68648 [out of 69632] + (68720 - 68648) > 16360
68720 [out of 69632] + (68792 - 68720) > 16360
68792 [out of 69632] + (68864 - 68792) > 16360
68864 [out of 69632] + (68936 - 68864) > 16360
68936 [out of 69632] + (69008 - 68936) > 16360
69008 [out of 69632] + (69080 - 69008) > 16360
69080 [out of 69632] + (69152 - 69080) > 16360
69152 [out of 69632] + (69224 - 69152) > 16360
69224 [out of 69632] + (69296 - 69224) > 16360
69296 [out of 69632] + (69368 - 69296) > 16360
69368 [out of 69632] + (69440 - 69368) > 16360
69440 [out of 69632] + (69512 - 69440) > 16360
69512 [out of 69632] + (69584 - 69512) > 16360
69584 [out of 69632] + (69656 - 69584) > 16360
69656 [out of 73728] + (69728 - 69656) > 16360
69728 [out of 73728] + (69800 - 69728) > 16360
69800 [out of 73728] + (69872 - 69800) > 16360
69872 [out of 73728] + (69944 - 69872) > 16360
69944 [out of 73728] + (70016 - 69944) > 16360
70016 [out of 73728] + (70088 - 70016) > 16360
70088 [out of 73728] + (70160 - 70088) > 16360
70160 [out of 73728] + (70232 - 70160) > 16360
70232 [out of 73728] + (70304 - 70232) > 16360
70304 [out of 73728] + (70376 - 70304) > 16360
70376 [out of 73728] + (70448 - 70376) > 16360
70448 [out of 73728] + (70520 - 70448) > 16360
70520 [out of 73728] + (70592 - 70520) > 16360
70592 [out of 73728] + (70664 - 70592) > 16360
70664 [out of 73728] + (70736 - 70664) > 16360
70736 [out of 73728] + (70808 - 70736) > 16360
70808 [out of 73728] + (70880 - 70808) > 16360
70880 [out of 73728] + (70952 - 70880) > 16360
70952 [out of 73728] + (71024 - 70952) > 16360
71024 [out of 73728] + (71096 - 71024) > 16360
71096 [out of 73728] + (71168 - 71096) > 16360
71168 [out of 73728] + (71240 - 71168) > 16360
71240 [out of 73728] + (71312 - 71240) > 16360
71312 [out of 73728] + (71384 - 71312) > 16360
71384 [out of 73728] + (71456 - 71384) > 16360
71456 [out of 73728] + (71528 - 71456) > 16360
71528 [out of 73728] + (71600 - 71528) > 16360
71600 [out of 73728] + (71672 - 71600) > 16360
71672 [out of 73728] + (71744 - 71672) > 16360
71744 [out of 73728] + (71816 - 71744) > 16360
71816 [out of 73728] + (71888 - 71816) > 16360
71888 [out of 73728] + (71960 - 71888) > 16360
71960 [out of 73728] + (72032 - 71960) > 16360
72032 [out of 73728] + (72104 - 72032) > 16360
72104 [out of 73728] + (72176 - 72104) > 16360
72176 [out of 73728] + (72248 - 72176) > 16360
72248 [out of 73728] + (72320 - 72248) > 16360
72320 [out of 73728] + (72392 - 72320) > 16360
72392 [out of 73728] + (72464 - 72392) > 16360
72464 [out of 73728] + (72536 - 72464) > 16360
72536 [out of 73728] + (72608 - 72536) > 16360
72608 [out of 73728] + (72680 - 72608) > 16360
72680 [out of 73728] + (72752 - 72680) > 16360
72752 [out of 73728] + (72824 - 72752) > 16360
72824 [out of 73728] + (72896 - 72824) > 16360
72896 [out of 73728] + (72968 - 72896) > 16360
72968 [out of 73728] + (73040 - 72968) > 16360
73040 [out of 73728] + (73112 - 73040) > 16360
73112 [out of 73728] + (73184 - 73112) > 16360
73184 [out of 73728] + (73256 - 73184) > 16360
73256 [out of 73728] + (73328 - 73256) > 16360
73328 [out of 73728] + (73400 - 73328) > 16360
73400 [out of 73728] + (73472 - 73400) > 16360
73472 [out of 73728] + (73544 - 73472) > 16360
73544 [out of 73728] + (73616 - 73544) > 16360
73616 [out of 73728] + (73688 - 73616) > 16360
73688 [out of 73728] + (73760 - 73688) > 16360
73760 [out of 77824] + (73832 - 73760) > 16360
73832 [out of 77824] + (73904 - 73832) > 16360
73904 [out of 77824] + (73976 - 73904) > 16360
73976 [out of 77824] + (74048 - 73976) > 16360
74048 [out of 77824] + (74120 - 74048) > 16360
74120 [out of 77824] + (74192 - 74120) > 16360
74192 [out of 77824] + (74264 - 74192) > 16360
74264 [out of 77824] + (74336 - 74264) > 16360
74336 [out of 77824] + (74408 - 74336) > 16360
74408 [out of 77824] + (74480 - 74408) > 16360
74480 [out of 77824] + (74552 - 74480) > 16360
74552 [out of 77824] + (74624 - 74552) > 16360
74624 [out of 77824] + (74696 - 74624) > 16360
74696 [out of 77824] + (74768 - 74696) > 16360
74768 [out of 77824] + (74840 - 74768) > 16360
74840 [out of 77824] + (74912 - 74840) > 16360
74912 [out of 77824] + (74984 - 74912) > 16360
74984 [out of 77824] + (75056 - 74984) > 16360
75056 [out of 77824] + (75128 - 75056) > 16360
75128 [out of 77824] + (75200 - 75128) > 16360
75200 [out of 77824] + (75272 - 75200) > 16360
75272 [out of 77824] + (75344 - 75272) > 16360
75344 [out of 77824] + (75416 - 75344) > 16360
75416 [out of 77824] + (75488 - 75416) > 16360
75488 [out of 77824] + (75560 - 75488) > 16360
75560 [out of 77824] + (75632 - 75560) > 16360
75632 [out of 77824] + (75704 - 75632) > 16360
75704 [out of 77824] + (75776 - 75704) > 16360
75776 [out of 77824] + (75848 - 75776) > 16360
75848 [out of 77824] + (75920 - 75848) > 16360
75920 [out of 77824] + (75992 - 75920) > 16360
75992 [out of 77824] + (76064 - 75992) > 16360
76064 [out of 77824] + (76136 - 76064) > 16360
76136 [out of 77824] + (76208 - 76136) > 16360
76208 [out of 77824] + (76280 - 76208) > 16360
76280 [out of 77824] + (76352 - 76280) > 16360
76352 [out of 77824] + (76424 - 76352) > 16360
76424 [out of 77824] + (76496 - 76424) > 16360
76496 [out of 77824] + (76568 - 76496) > 16360
76568 [out of 77824] + (76640 - 76568) > 16360
76640 [out of 77824] + (76712 - 76640) > 16360
76712 [out of 77824] + (76784 - 76712) > 16360
76784 [out of 77824] + (76856 - 76784) > 16360
76856 [out of 77824] + (76928 - 76856) > 16360
76928 [out of 77824] + (77000 - 76928) > 16360
77000 [out of 77824] + (77072 - 77000) > 16360
77072 [out of 77824] + (77144 - 77072) > 16360
77144 [out of 77824] + (77216 - 77144) > 16360
77216 [out of 77824] + (77288 - 77216) > 16360
77288 [out of 77824] + (77360 - 77288) > 16360
77360 [out of 77824] + (77432 - 77360) > 16360
77432 [out of 77824] + (77504 - 77432) > 16360
77504 [out of 77824] + (77576 - 77504) > 16360
77576 [out of 77824] + (77648 - 77576) > 16360
77648 [out of 77824] + (77720 - 77648) > 16360
77720 [out of 77824] + (77792 - 77720) > 16360
77792 [out of 77824] + (77864 - 77792) > 16360
77864 [out of 81920] + (77936 - 77864) > 16360
77936 [out of 81920] + (78008 - 77936) > 16360
78008 [out of 81920] + (78080 - 78008) > 16360
78080 [out of 81920] + (78152 - 78080) > 16360
78152 [out of 81920] + (78224 - 78152) > 16360
78224 [out of 81920] + (78296 - 78224) > 16360
78296 [out of 81920] + (78368 - 78296) > 16360
78368 [out of 81920] + (78440 - 78368) > 16360
78440 [out of 81920] + (78512 - 78440) > 16360
78512 [out of 81920] + (78584 - 78512) > 16360
78584 [out of 81920] + (78656 - 78584) > 16360
78656 [out of 81920] + (78728 - 78656) > 16360
78728 [out of 81920] + (78800 - 78728) > 16360
78800 [out of 81920] + (78872 - 78800) > 16360
78872 [out of 81920] + (78944 - 78872) > 16360
78944 [out of 81920] + (79016 - 78944) > 16360
79016 [out of 81920] + (79088 - 79016) > 16360
79088 [out of 81920] + (79160 - 79088) > 16360
79160 [out of 81920] + (79232 - 79160) > 16360
79232 [out of 81920] + (79304 - 79232) > 16360
79304 [out of 81920] + (79376 - 79304) > 16360
79376 [out of 81920] + (79448 - 79376) > 16360
79448 [out of 81920] + (79520 - 79448) > 16360
79520 [out of 81920] + (79592 - 79520) > 16360
79592 [out of 81920] + (79664 - 79592) > 16360
79664 [out of 81920] + (79736 - 79664) > 16360
79736 [out of 81920] + (79808 - 79736) > 16360
79808 [out of 81920] + (79880 - 79808) > 16360
79880 [out of 81920] + (79952 - 79880) > 16360
79952 [out of 81920] + (80024 - 79952) > 16360
80024 [out of 81920] + (80096 - 80024) > 16360
80096 [out of 81920] + (80168 - 80096) > 16360
80168 [out of 81920] + (80240 - 80168) > 16360
80240 [out of 81920] + (80312 - 80240) > 16360

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Bart Schaefer]

On May 9,  7:30pm, Clint Adams wrote:
}
} > And you're saying that (h->used + (new - old) <= HEAP_ARENA_SIZE) is 
} > always false, so the zhalloc() is always called?
} 
} I don't know that it's always false, but here's a small snapshot of
} those comparisons during a zsh/files rm -rf attempt.
} 
} [ h->used + (new - old) > HEAP_ARENA_SIZE ]
} 63432 + (63504 - 63432) > 16360

Aha!  Note that h->used > HEAP_ARENA_SIZE all by itself!

Comparing to HEAP_ARENA_SIZE is likely wrong, it should compare to the 
maximum of either HEAP_ARENA_SIZE or the previously-mmapped page size.
I'm not entirely sure of that ...

... but that's why it begins re-zhalloc()-ing.  In the non-MMAP case it
falls back on realloc() when a single allocation exceeds HEAP_ARENA_SIZE.

So what probably needs to happen to prevent the out-of-memory condition
is that we need to explicitly munmap the old mmap'd block, somewhere.

And then we need to emulate realloc() behavior somehow, in terms of
knowing whether to actually re-mmap-and-copy or whether the existing
mapped pages are large enough to contain the additional requested space.

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Peter Stephenson]

Bart Schaefer wrote:
> } [ h->used + (new - old) > HEAP_ARENA_SIZE ]
> } 63432 + (63504 - 63432) > 16360
> 
> Aha!  Note that h->used > HEAP_ARENA_SIZE all by itself!
> 
> Comparing to HEAP_ARENA_SIZE is likely wrong, it should compare to the 
> maximum of either HEAP_ARENA_SIZE or the previously-mmapped page size.
> I'm not entirely sure of that ...
> 
> ... but that's why it begins re-zhalloc()-ing.  In the non-MMAP case it
> falls back on realloc() when a single allocation exceeds HEAP_ARENA_SIZE.
> 
> So what probably needs to happen to prevent the out-of-memory condition
> is that we need to explicitly munmap the old mmap'd block, somewhere.
> 
> And then we need to emulate realloc() behavior somehow, in terms of
> knowing whether to actually re-mmap-and-copy or whether the existing
> mapped pages are large enough to contain the additional requested space.

Looks like a rewrite of hrealloc is definitely in order...

Does the bad behaviour go away if USE_MMAP is undefined?

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Clint Adams]

> Looks like a rewrite of hrealloc is definitely in order...
> 
> Does the bad behaviour go away if USE_MMAP is undefined?

It performs the rm -rf quickly, and there is only one instance in which
h->used + (new - old) is greater than HEAP_ARENA_SIZE:
16336 [out of 0] + (15984 - 15912) > 16360

Odd that h->size seems to be 0.

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Bart Schaefer]

On Mon, 10 May 2004, Peter Stephenson wrote:

> Looks like a rewrite of hrealloc is definitely in order...

Just the MMAP part.  Though we might consider increasing HEAP_ARENA_SIZE;
16K isn't much on modern machinery.

> Does the bad behaviour go away if USE_MMAP is undefined?

Examination of the code suggests so, as does Clint's test result.

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Peter Stephenson]

Bart Schaefer wrote:
> } [ h->used + (new - old) > HEAP_ARENA_SIZE ]
> } 63432 + (63504 - 63432) > 16360
> 
> Aha!  Note that h->used > HEAP_ARENA_SIZE all by itself!
> 
> Comparing to HEAP_ARENA_SIZE is likely wrong, it should compare to the 
> maximum of either HEAP_ARENA_SIZE or the previously-mmapped page size.

This is all ghastly even by zsh standards.

I thought what you thought, until... Look at zhalloc(), where either we
allocate n bytes normally, or we mmap an equivalent chunk of memory.
Neither of those explicitly use HEAP_ARENA_SIZE, which was used to get n
in the first place.

We are probably wasting space since the only thing we record for later
use is the original size asked for (`size') as the used size; we don't
record the number we've ended up with (`n'), goodness knows why.

For USE_MMAP, we do record n in h->size, but this is only used for
munmap, not for everything else --- it's obviously an afterthought.
(That's why it's zero when USE_MMAP isn't defined.)  The smart thing to
do would be always record it and use it for subsequent calculation
instead of all the jiggery pokery with rounding things to boundaries in
obscure ways.  This is basically your conclusion, however I think it's
just unfinished business rather than a programming error.

However, I don't think that's actually the source of the problem.
Closer to the point is this:

#ifndef USE_MMAP
	if (old > HEAP_ARENA_SIZE || new > HEAP_ARENA_SIZE) {
	    size_t n = HEAP_ARENA_SIZE > new ? HEAPSIZE : new + sizeof(*h);

	    if (ph)
		ph->next = h = (Heap) realloc(h, n);
	    else
		heaps = h = (Heap) realloc(h, n);
	}
	h->used = new;
	unqueue_signals();
	return arena(h);
#endif

Without USE_MMAP, this takes care of making sure the heap is the right
size when it's larger than normal.  Those tests would be triggering in
the case we're looking at.  As they're not, we're dropping through to
the zhalloc() below every time.

What we should do is the equivalent of realloc as an #else of the code
above --- mmap a larger chunk, unmap the old one, and return the new
one.

However, I don't think even *that* is the basic problem.  I think this:

    old = (old + H_ISIZE - 1) & ~(H_ISIZE - 1);
    new = (new + H_ISIZE - 1) & ~(H_ISIZE - 1);

is wrong --- it should be the heap size here, not H_ISIZE which is
simply sizeof(union mem_align), which is probably only 4 or 8 words.  We
only need to reallocate to heap size boundaries, i.e. 16384 possibly
minus something or other.  We've got away with it before because the
realloc() quoted above isn't too fussed, an hrealloc() isn't used that
often.  So actually Clint's original proposal is close to the mark.

I hope.

I might be tempted by proposed solutions involving the complete removal
of hrealloc.

Please don't use sentences containing `why' and ending with `?' or I'll
start to whimper.  Anyone fixing this could useful add comments.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Bart Schaefer]

On Mon, 10 May 2004, Peter Stephenson wrote:

> This is all ghastly even by zsh standards.

Well, that's what comes of having glued MMAP into the hrealloc() algorithm 
piecemeal.  Probably what should have been done was just to #ifdef the 
entire function body.

> What we should do is the equivalent of realloc as an #else of the code
> above --- mmap a larger chunk, unmap the old one, and return the new
> one.

I think I said that, though I hadn't got so far as to pinpoint where it
needed to happen.

> However, I don't think even *that* is the basic problem.  I think this:
> 
>     old = (old + H_ISIZE - 1) & ~(H_ISIZE - 1);
>     new = (new + H_ISIZE - 1) & ~(H_ISIZE - 1);
> 
> is wrong --- it should be the heap size here, not H_ISIZE which is
> simply sizeof(union mem_align), which is probably only 4 or 8 words.

It took me a while staring at this to decide that it was doing the right 
thing, but I came to the conclusion that it's only actually _wrong_ if
HEAP_ARENA_SIZE is _not_ a multiple of H_ISIZE.  And even then I'm pretty
sure "wrong" just means "inefficient".

> We only need to reallocate to heap size boundaries

That's correct, but we _do_ only reallocate to heap size boundaries in the
non-MMAP case.  I think.  It's only the MMAP'd code, which as I said was
pasted in long after the original algorithm was written, that botches it.

> I might be tempted by proposed solutions involving the complete removal
> of hrealloc.

Which would mean what, exactly?  Never grow heap blocks?

(Note that I didn't begin with "why".)

Re: Bug#245678: zsh: built-in rm -rf fills up the memory

[Peter Stephenson]

Bart Schaefer wrote:
> > However, I don't think even *that* is the basic problem.  I think this:
> > 
> >     old = (old + H_ISIZE - 1) & ~(H_ISIZE - 1);
> >     new = (new + H_ISIZE - 1) & ~(H_ISIZE - 1);
> > 
> > is wrong --- it should be the heap size here, not H_ISIZE which is
> > simply sizeof(union mem_align), which is probably only 4 or 8 words.
> 
> It took me a while staring at this to decide that it was doing the right 
> thing, but I came to the conclusion that it's only actually _wrong_ if
> HEAP_ARENA_SIZE is _not_ a multiple of H_ISIZE.  And even then I'm pretty
> sure "wrong" just means "inefficient".

I'd have to follow through with a debugger, really... But I can't see
any mechanism to prevent it reallocating for every few bytes of increase
once the size goes beyond HEAP_ARENA_SIZE, which can't be the
intention.  The chunk #ifdef'd out for USE_MMAP is the first point where
we see if we have enough space for `new' already (except for special
cases old == new, we allocated the space then shrank, new is NULL).
So for new and old larger than HEAP_ARENA_SIZE (which is bound to
happen after the first attempt to grow), we call realloc() every time
the difference crosses an H_ISIZE boundary.  What have I missed?

> > We only need to reallocate to heap size boundaries
> 
> That's correct, but we _do_ only reallocate to heap size boundaries in the
> non-MMAP case.  I think.  It's only the MMAP'd code, which as I said was
> pasted in long after the original algorithm was written, that botches it.

Yes, but we then call realloc unnecessarily in between actually growing
the heap, unless my logic above is wrong.  It will reallocate even
though the new size of the heap is the same as the old one.

> > I might be tempted by proposed solutions involving the complete removal
> > of hrealloc.
> 
> Which would mean what, exactly?  Never grow heap blocks?

hrealloc is only used as an optimisation.  If necessary we could always
allocate new space.  In the case of the files module we could use
zalloc and zrealloc and let the system worry about the problem.  Then we
lose MMAP altogether, of course.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 692070


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************