* Re: Bug#245678: zsh: built-in rm -rf fills up the memory [not found] <20040424162150.GA4210@ay.vinc17.org> @ 2004-04-25 1:36 ` Clint Adams 2004-04-25 20:45 ` Bart Schaefer 0 siblings, 1 reply; 19+ messages in thread From: Clint Adams @ 2004-04-25 1:36 UTC (permalink / raw) To: zsh-workers; +Cc: Vincent Lefevre, 245678-forwarded > When I do a rm -rf on a big maildir directory, zsh fills up the > memory, which makes the whole machine unresponsive (the load of > the machine becomes very high according to "xload", and the mouse > pointer no longer moves) and random processes be killed, before > zsh itself dies. According to "top", the used memory by zsh was > up to 583 MB. And nothing was actually removed. I can reproduce the memory consumption with a 7210-file Maildir, except that my kernel does the right thing and kills the runaway zsh. If I abort the rm before it OOMs, zsh will continue to take up a large amount of memory until it exits. Here's some valgrind output from one such run. ==31822== Memcheck, a memory error detector for x86-linux. ==31822== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward. ==31822== Using valgrind-2.1.1, a program supervision framework for x86-linux. ==31822== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward. ==31822== ==31822== My PID = 31822, parent PID = 1021. Prog and args are: ==31822== zsh ==31822== -f ==31822== ==31822== Valgrind library directory: /usr/lib/valgrind ==31822== Command line ==31822== zsh ==31822== -f ==31822== Startup, with flags: ==31822== --tool=memcheck ==31822== --logfile=/tmp/zshremove ==31822== -v ==31822== --leak-check=yes ==31822== --show-reachable=yes ==31822== -- ==31822== Reading syms from /bin/zsh4 (0x8048000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/ld-2.3.2.so (0x3C000000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/ld-2.3.2.so (0xB0000000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/tls/i686/cmov/libdl-2.3.2.so (0xB0020000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/tls/i686/cmov/libc-2.3.2.so (0xB0023000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /usr/lib/valgrind/vgskin_memcheck.so (0xB035D000) ==31822== Reading syms from /usr/lib/valgrind/stage2 (0xB8000000) ==31822== Reading suppressions file: /usr/lib/valgrind/default.supp ==31822== REDIRECT soname:libc.so.6(__GI___errno_location) to soname:libpthread.so.0(__errno_location) ==31822== REDIRECT soname:libc.so.6(__errno_location) to soname:libpthread.so.0(__errno_location) ==31822== REDIRECT soname:libc.so.6(__GI___h_errno_location) to soname:libpthread.so.0(__h_errno_location) ==31822== REDIRECT soname:libc.so.6(__h_errno_location) to soname:libpthread.so.0(__h_errno_location) ==31822== REDIRECT soname:libc.so.6(__GI___res_state) to soname:libpthread.so.0(__res_state) ==31822== REDIRECT soname:libc.so.6(__res_state) to soname:libpthread.so.0(__res_state) ==31822== REDIRECT soname:libc.so.6(stpcpy) to *vgpreload_memcheck.so*(stpcpy) ==31822== REDIRECT soname:libc.so.6(strnlen) to *vgpreload_memcheck.so*(strnlen) ==31822== REDIRECT soname:ld-linux.so.2(stpcpy) to *vgpreload_memcheck.so*(stpcpy) ==31822== REDIRECT soname:ld-linux.so.2(strchr) to *vgpreload_memcheck.so*(strchr) ==31822== ==31822== Reading syms from /usr/lib/valgrind/vg_inject.so (0x3C01A000) ==31822== Reading syms from /usr/lib/valgrind/vgpreload_memcheck.so (0x3C01D000) ==31822== Reading syms from /lib/libcap.so.1.10 (0x3C02E000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/tls/i686/cmov/libdl-2.3.2.so (0x41179000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/tls/i686/cmov/libnsl-2.3.2.so (0x3C033000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/libncurses.so.5.4 (0x3C049000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/tls/i686/cmov/libm-2.3.2.so (0x41154000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/tls/i686/cmov/libc-2.3.2.so (0x41019000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/tls/i686/cmov/libnss_compat-2.3.2.so (0x3C2AE000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/tls/i686/cmov/libnss_nis-2.3.2.so (0x3C2B7000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /lib/tls/i686/cmov/libnss_files-2.3.2.so (0x3C2C2000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== Address 0x4FFFE0D4 is on thread 1's stack ==31822== TRANSLATE: 0x41092130 redirected to 0x3C01EC70 ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x0: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1F: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3F: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7F: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== Reading syms from /usr/lib/zsh/4.1.1/zsh/zle.so (0x3C2CD000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /usr/lib/zsh/4.1.1/zsh/complete.so (0x3C2FC000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /usr/lib/zsh/4.1.1/zsh/compctl.so (0x3C31A000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== Reading syms from /usr/lib/zsh/4.1.1/zsh/files.so (0x3C329000) ==31822== object doesn't have a symbol table ==31822== object doesn't have any debug info ==31822== ==31822== Invalid read of size 4 ==31822== at 0x4103003B: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== Address 0x3C0BD7CC is 8 bytes inside a block of size 60 free'd ==31822== at 0x3C01F918: free (vg_replace_malloc.c:127) ==31822== by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== Invalid write of size 4 ==31822== at 0x4103004E: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== Address 0x3C0BD7CC is 8 bytes inside a block of size 60 free'd ==31822== at 0x3C01F918: free (vg_replace_malloc.c:127) ==31822== by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== Invalid read of size 4 ==31822== at 0x41030053: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== Address 0x3C0BD7E4 is 32 bytes inside a block of size 60 free'd ==31822== at 0x3C01F918: free (vg_replace_malloc.c:127) ==31822== by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== Invalid read of size 4 ==31822== at 0x4103005A: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== Address 0x3C0BD7C4 is 0 bytes inside a block of size 60 free'd ==31822== at 0x3C01F918: free (vg_replace_malloc.c:127) ==31822== by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== discard syms at 0x3C2AE000-0x3C2B6000 in /lib/tls/i686/cmov/libnss_compat-2.3.2.so due to munmap() ==31822== discard syms at 0x3C2B7000-0x3C2C1000 in /lib/tls/i686/cmov/libnss_nis-2.3.2.so due to munmap() ==31822== discard syms at 0x3C2C2000-0x3C2CC000 in /lib/tls/i686/cmov/libnss_files-2.3.2.so due to munmap() ==31822== ==31822== ERROR SUMMARY: 110 errors from 36 contexts (suppressed: 32 from 1) ==31822== ==31822== 1 errors in context 1 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 2 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 3 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 4 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 5 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 6 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 7 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 8 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 9 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 10 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 11 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 12 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 13 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 14 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 15 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 16 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 17 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 18 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 19 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 20 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7F: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 21 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3F: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 22 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1F: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 23 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 24 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 1 errors in context 25 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 2 errors in context 26 of 36: ==31822== Invalid read of size 4 ==31822== at 0x4103005A: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== Address 0x3C0BD7C4 is 0 bytes inside a block of size 60 free'd ==31822== at 0x3C01F918: free (vg_replace_malloc.c:127) ==31822== by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== 2 errors in context 27 of 36: ==31822== Invalid read of size 4 ==31822== at 0x41030053: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== Address 0x3C0BD7E4 is 32 bytes inside a block of size 60 free'd ==31822== at 0x3C01F918: free (vg_replace_malloc.c:127) ==31822== by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== 2 errors in context 28 of 36: ==31822== Invalid write of size 4 ==31822== at 0x4103004E: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== Address 0x3C0BD7CC is 8 bytes inside a block of size 60 free'd ==31822== at 0x3C01F918: free (vg_replace_malloc.c:127) ==31822== by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== 2 errors in context 29 of 36: ==31822== Invalid read of size 4 ==31822== at 0x4103003B: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== Address 0x3C0BD7CC is 8 bytes inside a block of size 60 free'd ==31822== at 0x3C01F918: free (vg_replace_malloc.c:127) ==31822== by 0x41030DBB: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== 2 errors in context 30 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0xFFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 2 errors in context 31 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x7FFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 2 errors in context 32 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x1FFFFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 2 errors in context 33 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3FFF: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 2 errors in context 34 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x0: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 2 errors in context 35 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== by 0x3: ??? ==31822== Address 0x4FFFE75C is on thread 1's stack ==31822== ==31822== 65 errors in context 36 of 36: ==31822== Syscall param sigaction(act) contains uninitialised or unaddressable byte(s) ==31822== at 0x3C000C02: (within /lib/ld-2.3.2.so) ==31822== Address 0x4FFFE0D4 is on thread 1's stack --31822-- --31822-- supp: 32 Ugly strchr error in /lib/ld-2.3.2.so ==31822== ==31822== IN SUMMARY: 110 errors from 36 contexts (suppressed: 32 from 1) ==31822== ==31822== malloc/free: in use at exit: 51472 bytes in 932 blocks. ==31822== malloc/free: 3294 allocs, 2362 frees, 175556 bytes allocated. ==31822== ==31822== searching for pointers to 932 not-freed blocks. ==31822== checked 2918772 bytes. ==31822== ==31822== ==31822== 6 bytes in 1 blocks are still reachable in loss record 1 of 24 ==31822== at 0x3C01FD01: realloc (vg_replace_malloc.c:162) ==31822== by 0x3C07086F: _nc_doalloc (in /lib/libncurses.so.5.4) ==31822== by 0x3C0737AE: (within /lib/libncurses.so.5.4) ==31822== by 0x3C073401: tparm (in /lib/libncurses.so.5.4) ==31822== ==31822== ==31822== 12 bytes in 2 blocks are still reachable in loss record 2 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x41091DAA: strdup (in /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== ==31822== 24 bytes in 1 blocks are still reachable in loss record 3 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x3C070AB5: _nc_home_terminfo (in /lib/libncurses.so.5.4) ==31822== by 0x3C077A72: _nc_read_entry (in /lib/libncurses.so.5.4) ==31822== by 0x3C071FBE: (within /lib/libncurses.so.5.4) ==31822== ==31822== ==31822== 28 bytes in 1 blocks are still reachable in loss record 4 of 24 ==31822== at 0x3C01FC03: calloc (vg_replace_malloc.c:141) ==31822== by 0x3C07709F: (within /lib/libncurses.so.5.4) ==31822== by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4) ==31822== by 0x3C0778A1: (within /lib/libncurses.so.5.4) ==31822== ==31822== ==31822== 36 bytes in 1 blocks are definitely lost in loss record 5 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x41104618: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== ==31822== 40 bytes in 2 blocks are still reachable in loss record 6 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x3C01FC88: realloc (vg_replace_malloc.c:154) ==31822== by 0x3C007D08: (within /lib/ld-2.3.2.so) ==31822== by 0x3C007E8E: _dl_lookup_symbol (in /lib/ld-2.3.2.so) ==31822== ==31822== ==31822== 40 bytes in 5 blocks are still reachable in loss record 7 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x4110418D: __nss_lookup_function (in /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== by 0x3C2B0A2E: ??? ==31822== ==31822== ==31822== 44 bytes in 1 blocks are still reachable in loss record 8 of 24 ==31822== at 0x3C01FC03: calloc (vg_replace_malloc.c:141) ==31822== by 0x3C0770EB: (within /lib/libncurses.so.5.4) ==31822== by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4) ==31822== by 0x3C0778A1: (within /lib/libncurses.so.5.4) ==31822== ==31822== ==31822== 76 bytes in 1 blocks are still reachable in loss record 9 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x41125906: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== ==31822== 78 bytes in 1 blocks are still reachable in loss record 10 of 24 ==31822== at 0x3C01FC03: calloc (vg_replace_malloc.c:141) ==31822== by 0x3C077152: (within /lib/libncurses.so.5.4) ==31822== by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4) ==31822== by 0x3C0778A1: (within /lib/libncurses.so.5.4) ==31822== ==31822== ==31822== 79 bytes in 1 blocks are still reachable in loss record 11 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x3C0708A7: _nc_doalloc (in /lib/libncurses.so.5.4) ==31822== by 0x3C0733B5: _nc_tparm_analyze (in /lib/libncurses.so.5.4) ==31822== by 0x3C073631: (within /lib/libncurses.so.5.4) ==31822== ==31822== ==31822== 80 bytes in 5 blocks are still reachable in loss record 12 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x410F1D9B: tsearch (in /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== ==31822== 131 bytes in 4 blocks are still reachable in loss record 13 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x3C00913E: (within /lib/ld-2.3.2.so) ==31822== by 0x3C004C8B: (within /lib/ld-2.3.2.so) ==31822== by 0x3C0064A5: _dl_map_object (in /lib/ld-2.3.2.so) ==31822== ==31822== ==31822== 131 bytes in 4 blocks are still reachable in loss record 14 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x3C0064EB: _dl_map_object (in /lib/ld-2.3.2.so) ==31822== by 0x4112502E: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== by 0x3C00BF25: _dl_catch_error (in /lib/ld-2.3.2.so) ==31822== ==31822== ==31822== 172 bytes in 1 blocks are still reachable in loss record 15 of 24 ==31822== at 0x3C01FC03: calloc (vg_replace_malloc.c:141) ==31822== by 0x3C072307: setupterm (in /lib/libncurses.so.5.4) ==31822== by 0x3C0727FE: tgetent (in /lib/libncurses.so.5.4) ==31822== by 0x8073BD9: init_term (in /bin/zsh4) ==31822== ==31822== ==31822== 256 bytes in 1 blocks are still reachable in loss record 16 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x80820C6: zrealloc (in /bin/zsh4) ==31822== by 0x80939C7: patcompile (in /bin/zsh4) ==31822== by 0x805F32F: evalcond (in /bin/zsh4) ==31822== ==31822== ==31822== 272 bytes in 4 blocks are still reachable in loss record 17 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x3C00AE9A: _dl_map_object_deps (in /lib/ld-2.3.2.so) ==31822== by 0x411250D5: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== by 0x3C00BF25: _dl_catch_error (in /lib/ld-2.3.2.so) ==31822== ==31822== ==31822== 352 bytes in 4 blocks are still reachable in loss record 18 of 24 ==31822== at 0x3C01FC03: calloc (vg_replace_malloc.c:141) ==31822== by 0x3C00D232: _dl_check_map_versions (in /lib/ld-2.3.2.so) ==31822== by 0x411255E8: (within /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== by 0x3C00BF25: _dl_catch_error (in /lib/ld-2.3.2.so) ==31822== ==31822== ==31822== 352 bytes in 1 blocks are still reachable in loss record 19 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x4107AC0A: _IO_fdopen (in /lib/tls/i686/cmov/libc-2.3.2.so) ==31822== ==31822== ==31822== 1168 bytes in 1 blocks are still reachable in loss record 20 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x3C076FD8: (within /lib/libncurses.so.5.4) ==31822== by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4) ==31822== by 0x3C0778A1: (within /lib/libncurses.so.5.4) ==31822== ==31822== ==31822== 1656 bytes in 1 blocks are still reachable in loss record 21 of 24 ==31822== at 0x3C01FC03: calloc (vg_replace_malloc.c:141) ==31822== by 0x3C0771C9: (within /lib/libncurses.so.5.4) ==31822== by 0x3C0777E4: _nc_read_file_entry (in /lib/libncurses.so.5.4) ==31822== by 0x3C0778A1: (within /lib/libncurses.so.5.4) ==31822== ==31822== ==31822== 2387 bytes in 4 blocks are still reachable in loss record 22 of 24 ==31822== at 0x3C01FC03: calloc (vg_replace_malloc.c:141) ==31822== by 0x3C008F23: (within /lib/ld-2.3.2.so) ==31822== by 0x3C004C8B: (within /lib/ld-2.3.2.so) ==31822== by 0x3C0064A5: _dl_map_object (in /lib/ld-2.3.2.so) ==31822== ==31822== ==31822== 19716 bytes in 264 blocks are still reachable in loss record 23 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x8081DBA: zshcalloc (in /bin/zsh4) ==31822== by 0x8075721: zsh_main (in /bin/zsh4) ==31822== by 0x805210A: main (in /bin/zsh4) ==31822== ==31822== ==31822== 24336 bytes in 621 blocks are still reachable in loss record 24 of 24 ==31822== at 0x3C01F40D: malloc (vg_replace_malloc.c:105) ==31822== by 0x8081C95: zalloc (in /bin/zsh4) ==31822== by 0x809A725: ztrdup (in /bin/zsh4) ==31822== by 0x8073387: parseargs (in /bin/zsh4) ==31822== ==31822== LEAK SUMMARY: ==31822== definitely lost: 36 bytes in 1 blocks. ==31822== possibly lost: 0 bytes in 0 blocks. ==31822== still reachable: 51436 bytes in 931 blocks. ==31822== suppressed: 0 bytes in 0 blocks. --31822-- TT/TC: 0 tc sectors discarded. --31822-- 8091 chainings, 0 unchainings. --31822-- translate: new 11409 (178624 -> 2283438; ratio 127:10) --31822-- discard 180 (2199 -> 30012; ratio 136:10). --31822-- dispatch: 13550000 jumps (bb entries), of which 201341 (1%) were unchained. --31822-- 1497/21775 major/minor sched events. 12377 tt_fast misses. --31822-- reg-alloc: 1808 t-req-spill, 417567+11458 orig+spill uis, 56546 total-reg-r. --31822-- sanity: 1419 cheap, 57 expensive checks. --31822-- ccalls: 39857 C calls, 56% saves+restores avoided (132048 bytes) --31822-- 54348 args, avg 0.86 setup instrs each (14802 bytes) --31822-- 0% clear the stack (119064 bytes) --31822-- 15881 retvals, 31% of reg-reg movs avoided (9546 bytes) ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-04-25 1:36 ` Bug#245678: zsh: built-in rm -rf fills up the memory Clint Adams @ 2004-04-25 20:45 ` Bart Schaefer 2004-04-25 21:12 ` Clint Adams 0 siblings, 1 reply; 19+ messages in thread From: Bart Schaefer @ 2004-04-25 20:45 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-forwarded On Apr 24, 9:36pm, Clint Adams wrote: } } I can reproduce the memory consumption with a 7210-file Maildir, except } that my kernel does the right thing and kills the runaway zsh. } } If I abort the rm before it OOMs, zsh will continue to take up a large } amount of memory until it exits. There's got to be something here you're not telling us. Either there is an alias for "rm", or a function named "rm", or the zsh/files module is loaded; otherwise "rm" and "/bin/rm" should be equivalent (assuming there is no other "rm" in the path) and in any case shouldn't execute inside the shell. Assuming that it's the zsh/files module, the interesting thing there is that, because it's a depth-first traversal, it allocates zsh-heap memory to store all the file names in the directory, and it does so by growing the allocated heap block once for each filename, by the length of that filename (plus one). If ZSH_MEM was not defined at compile time, doing this efficiently is left up to the underlying malloc() implementation [if I'm reading mem.c correctly] and therefore could be subject to bugs we've seen before where malloc never reclaims nor consolidates blocks when the implementation repeatedly realloc()s in slightly larger chunks. I don't know anything about how zsh is compiled on debian or even why the zsh/files module would be used (seemingly) by default, but in that's probably the place to start looking. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-04-25 20:45 ` Bart Schaefer @ 2004-04-25 21:12 ` Clint Adams 2004-04-25 21:38 ` Clint Adams 2004-04-26 17:10 ` Bart Schaefer 0 siblings, 2 replies; 19+ messages in thread From: Clint Adams @ 2004-04-25 21:12 UTC (permalink / raw) To: Bart Schaefer; +Cc: zsh-workers, 245678-submitter > Assuming that it's the zsh/files module, the interesting thing there is > that, because it's a depth-first traversal, it allocates zsh-heap memory > to store all the file names in the directory, and it does so by growing > the allocated heap block once for each filename, by the length of that > filename (plus one). If ZSH_MEM was not defined at compile time, doing > this efficiently is left up to the underlying malloc() implementation [if > I'm reading mem.c correctly] and therefore could be subject to bugs we've > seen before where malloc never reclaims nor consolidates blocks when the > implementation repeatedly realloc()s in slightly larger chunks. > > I don't know anything about how zsh is compiled on debian or even why > the zsh/files module would be used (seemingly) by default, but in that's > probably the place to start looking. Yes, sorry. We're talking about when zsh/files is loaded explicitly. ZSH_MEM isn't being defined. Is there perhaps a mallopt() setting which would help? ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-04-25 21:12 ` Clint Adams @ 2004-04-25 21:38 ` Clint Adams 2004-04-26 17:10 ` Bart Schaefer 1 sibling, 0 replies; 19+ messages in thread From: Clint Adams @ 2004-04-25 21:38 UTC (permalink / raw) To: Bart Schaefer; +Cc: zsh-workers, 245678-submitter > Yes, sorry. We're talking about when zsh/files is loaded explicitly. > ZSH_MEM isn't being defined. Additionally, a recompile with ZSH_MEM shows that it still OOMs (perhaps much more slowly, but that could be an error in my perception). ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-04-25 21:12 ` Clint Adams 2004-04-25 21:38 ` Clint Adams @ 2004-04-26 17:10 ` Bart Schaefer 2004-05-08 4:35 ` Clint Adams 1 sibling, 1 reply; 19+ messages in thread From: Bart Schaefer @ 2004-04-26 17:10 UTC (permalink / raw) To: 245678-submitter, zsh-workers On Apr 25, 5:12pm, Clint Adams wrote: } } Is there perhaps a mallopt() setting which would help? I'm not all that familiar with mallopt() but a look at the documentation doesn't suggest anything. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-04-26 17:10 ` Bart Schaefer @ 2004-05-08 4:35 ` Clint Adams 2004-05-08 14:02 ` Clint Adams 0 siblings, 1 reply; 19+ messages in thread From: Clint Adams @ 2004-05-08 4:35 UTC (permalink / raw) To: Bart Schaefer; +Cc: 245678-submitter, zsh-workers > I'm not all that familiar with mallopt() but a look at the documentation > doesn't suggest anything. I think I've found the problem. Here's an excerpt from an strace of the zsh/files rm -rf going wild. There are 55 or 56 such calls to mmap for every length value. Thus it appears that zsh is allocating considerably more memory than it needs. mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x583d9000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58415000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58451000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5848d000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x584c9000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58505000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58541000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5857d000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x585b9000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x585f5000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58631000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5866d000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x586a9000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x586e5000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58721000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5875d000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58799000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x587d5000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58811000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5884d000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58889000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x588c5000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58901000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x5893d000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58979000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x589b5000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x589f1000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58a2d000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58a69000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58aa5000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58ae1000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58b1d000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58b59000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58b95000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58bd1000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58c0d000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58c49000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58c85000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58cc1000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58cfd000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58d39000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58d75000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58db1000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58ded000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58e29000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58e65000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58ea1000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58edd000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58f19000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58f55000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58f91000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x58fcd000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x59009000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x59045000 mmap2(NULL, 245760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x59081000 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-08 4:35 ` Clint Adams @ 2004-05-08 14:02 ` Clint Adams 2004-05-08 14:13 ` Clint Adams 2004-05-09 22:48 ` Bart Schaefer 0 siblings, 2 replies; 19+ messages in thread From: Clint Adams @ 2004-05-08 14:02 UTC (permalink / raw) To: Bart Schaefer; +Cc: 245678-submitter, zsh-workers > I think I've found the problem. Here's an excerpt from an strace of > the zsh/files rm -rf going wild. There are 55 or 56 such calls to mmap > for every length value. Thus it appears that zsh is allocating > considerably more memory than it needs. More on this: recursivecmd_dorec() calls hrealloc() for each target file, incrementing size accordingly. hrealloc() rounds the new length up to the nearest page boundary, and mmaps a new Heap for each file. So, for example, if you have a directory with one hundred files with 9-byte filenames, zsh might allocate one hundred 4096-byte areas. If you have several thousand files with 70- to 80-byte filenames, zsh might allocate over 50 4096-byte areas, over 50 8192-byte areas, and so on. Now, should this be fixed by making hrealloc() smarter or recursivecmd_dorec() more efficient? ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-08 14:02 ` Clint Adams @ 2004-05-08 14:13 ` Clint Adams 2004-05-09 22:54 ` Bart Schaefer 2004-05-09 22:48 ` Bart Schaefer 1 sibling, 1 reply; 19+ messages in thread From: Clint Adams @ 2004-05-08 14:13 UTC (permalink / raw) To: Bart Schaefer; +Cc: 245678-submitter, zsh-workers > Now, should this be fixed by making hrealloc() smarter or > recursivecmd_dorec() more efficient? Here's one non-portable way of preventing the OOM, and making the rm -rf go quickly. Should H_ISIZE be changed to the page size when USE_MMAP is defined, or is there a reason to keep it the sizeof(union mem_align) ? --- orig/Src/mem.c +++ mod/Src/mem.c @@ -379,8 +379,8 @@ { Heap h, ph; - old = (old + H_ISIZE - 1) & ~(H_ISIZE - 1); - new = (new + H_ISIZE - 1) & ~(H_ISIZE - 1); + old = (old + 4096 - 1) & ~(4096 - 1); + new = (new + 4096 - 1) & ~(4096 - 1); if (old == new) return p; ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-08 14:13 ` Clint Adams @ 2004-05-09 22:54 ` Bart Schaefer 0 siblings, 0 replies; 19+ messages in thread From: Bart Schaefer @ 2004-05-09 22:54 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter On Sat, 8 May 2004, Clint Adams wrote: > Should H_ISIZE be changed to the page size when USE_MMAP is defined, or > is there a reason to keep it the sizeof(union mem_align) ? Changing H_ISIZE won't fix the underlying cause of the bug, it'll just cause it to manifest only once every 4096/sizeof(union mem_align) calls rather than on every call. We need to find out what code branch is calling zhalloc() on every small increment, and why that code branch is being taken. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-08 14:02 ` Clint Adams 2004-05-08 14:13 ` Clint Adams @ 2004-05-09 22:48 ` Bart Schaefer 2004-05-09 23:30 ` Clint Adams 1 sibling, 1 reply; 19+ messages in thread From: Bart Schaefer @ 2004-05-09 22:48 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter On Sat, 8 May 2004, Clint Adams wrote: > So, for example, if you have a directory with one hundred files with > 9-byte filenames, zsh might allocate one hundred 4096-byte areas. If that's really what's happening, it's a bug somewhere else, because zsh should only allocate another heap page if the current heap page doesn't have room for the requested growth. That is, 409 9-byte file names should need 4090 bytes (add 1 to each for the NUL byte), and it should put all 4090 of those bytes in the same 4096-byte heap block. It shouldn't allocate a new heap block until it gets to the 410th file, at which point it should allocate another 8192-byte block and copy the old block into it, and return the original 4096-byte block to the heap pool for re-use. (Where I'm assuming HEAP_ARENA_SIZE is 4096 here.) The following code should *never* be executed unless hrealloc() has previously been used to *shrink* a block: if (p + old < arena(h) + h->used) { if (new > old) { char *ptr = (char *) zhalloc(new); memcpy(ptr, p, old); #ifdef ZSH_MEM_DEBUG memset(p, 0xff, old); #endif unqueue_signals(); return ptr; } else { unqueue_signals(); return new ? p : NULL; } } Because unless we shrank, (p + old == arena(h) + h->used), as asserted by the DPUTS() that immediately follows. So, taking #ifdefs for MMAP into account, this code is what should run: if (h->used + (new - old) <= HEAP_ARENA_SIZE) { h->used += new - old; unqueue_signals(); return p; } else { char *t = zhalloc(new); memcpy(t, p, old > new ? new : old); h->used -= old; #ifdef ZSH_MEM_DEBUG memset(p, 0xff, old); #endif unqueue_signals(); return t; } And you're saying that (h->used + (new - old) <= HEAP_ARENA_SIZE) is always false, so the zhalloc() is always called? If that's not what you find to be the case, then the bug is elsewhere and we need to keep looking. If it is what you find, then there's a problem with h->used record-keeping, or something. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-09 22:48 ` Bart Schaefer @ 2004-05-09 23:30 ` Clint Adams 2004-05-09 23:36 ` Clint Adams 0 siblings, 1 reply; 19+ messages in thread From: Clint Adams @ 2004-05-09 23:30 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter > And you're saying that (h->used + (new - old) <= HEAP_ARENA_SIZE) is > always false, so the zhalloc() is always called? I don't know that it's always false, but here's a small snapshot of those comparisons during a zsh/files rm -rf attempt. [ h->used + (new - old) > HEAP_ARENA_SIZE ] 63432 + (63504 - 63432) > 16360 63504 + (63576 - 63504) > 16360 63576 + (63648 - 63576) > 16360 63648 + (63728 - 63648) > 16360 63728 + (63800 - 63728) > 16360 63800 + (63872 - 63800) > 16360 63872 + (63944 - 63872) > 16360 63944 + (64024 - 63944) > 16360 64024 + (64096 - 64024) > 16360 64096 + (64168 - 64096) > 16360 64168 + (64240 - 64168) > 16360 64240 + (64320 - 64240) > 16360 > If that's not what you find to be the case, then the bug is elsewhere and > we need to keep looking. If it is what you find, then there's a problem > with h->used record-keeping, or something. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-09 23:30 ` Clint Adams @ 2004-05-09 23:36 ` Clint Adams 2004-05-09 23:51 ` Bart Schaefer 0 siblings, 1 reply; 19+ messages in thread From: Clint Adams @ 2004-05-09 23:36 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter > [ h->used + (new - old) > HEAP_ARENA_SIZE ] > 63432 + (63504 - 63432) > 16360 > 63504 + (63576 - 63504) > 16360 I added in h->size to the printf, to show the heap sizes. 60856 [out of 61440] + (60928 - 60856) > 16360 60928 [out of 61440] + (61008 - 60928) > 16360 61008 [out of 61440] + (61080 - 61008) > 16360 61080 [out of 61440] + (61152 - 61080) > 16360 61152 [out of 61440] + (61224 - 61152) > 16360 61224 [out of 61440] + (61304 - 61224) > 16360 61304 [out of 61440] + (61376 - 61304) > 16360 61376 [out of 61440] + (61448 - 61376) > 16360 61448 [out of 65536] + (61520 - 61448) > 16360 61520 [out of 65536] + (61600 - 61520) > 16360 61600 [out of 65536] + (61672 - 61600) > 16360 61672 [out of 65536] + (61744 - 61672) > 16360 61744 [out of 65536] + (61816 - 61744) > 16360 61816 [out of 65536] + (61896 - 61816) > 16360 61896 [out of 65536] + (61968 - 61896) > 16360 61968 [out of 65536] + (62040 - 61968) > 16360 62040 [out of 65536] + (62112 - 62040) > 16360 62112 [out of 65536] + (62184 - 62112) > 16360 62184 [out of 65536] + (62256 - 62184) > 16360 62256 [out of 65536] + (62328 - 62256) > 16360 62328 [out of 65536] + (62400 - 62328) > 16360 62400 [out of 65536] + (62472 - 62400) > 16360 62472 [out of 65536] + (62552 - 62472) > 16360 62552 [out of 65536] + (62624 - 62552) > 16360 62624 [out of 65536] + (62696 - 62624) > 16360 62696 [out of 65536] + (62768 - 62696) > 16360 62768 [out of 65536] + (62840 - 62768) > 16360 62840 [out of 65536] + (62912 - 62840) > 16360 62912 [out of 65536] + (62984 - 62912) > 16360 62984 [out of 65536] + (63056 - 62984) > 16360 63056 [out of 65536] + (63136 - 63056) > 16360 63136 [out of 65536] + (63208 - 63136) > 16360 63208 [out of 65536] + (63280 - 63208) > 16360 63280 [out of 65536] + (63352 - 63280) > 16360 63352 [out of 65536] + (63432 - 63352) > 16360 63432 [out of 65536] + (63504 - 63432) > 16360 63504 [out of 65536] + (63576 - 63504) > 16360 63576 [out of 65536] + (63648 - 63576) > 16360 63648 [out of 65536] + (63728 - 63648) > 16360 63728 [out of 65536] + (63800 - 63728) > 16360 63800 [out of 65536] + (63872 - 63800) > 16360 63872 [out of 65536] + (63944 - 63872) > 16360 63944 [out of 65536] + (64024 - 63944) > 16360 64024 [out of 65536] + (64096 - 64024) > 16360 64096 [out of 65536] + (64168 - 64096) > 16360 64168 [out of 65536] + (64240 - 64168) > 16360 64240 [out of 65536] + (64320 - 64240) > 16360 64320 [out of 65536] + (64392 - 64320) > 16360 64392 [out of 65536] + (64464 - 64392) > 16360 64464 [out of 65536] + (64536 - 64464) > 16360 64536 [out of 65536] + (64616 - 64536) > 16360 64616 [out of 65536] + (64688 - 64616) > 16360 64688 [out of 65536] + (64760 - 64688) > 16360 64760 [out of 65536] + (64832 - 64760) > 16360 64832 [out of 65536] + (64904 - 64832) > 16360 64904 [out of 65536] + (64984 - 64904) > 16360 64984 [out of 65536] + (65056 - 64984) > 16360 65056 [out of 65536] + (65128 - 65056) > 16360 65128 [out of 65536] + (65200 - 65128) > 16360 65200 [out of 65536] + (65280 - 65200) > 16360 65280 [out of 65536] + (65352 - 65280) > 16360 65352 [out of 65536] + (65424 - 65352) > 16360 65424 [out of 65536] + (65496 - 65424) > 16360 65496 [out of 65536] + (65576 - 65496) > 16360 65576 [out of 69632] + (65648 - 65576) > 16360 65648 [out of 69632] + (65720 - 65648) > 16360 65720 [out of 69632] + (65792 - 65720) > 16360 65792 [out of 69632] + (65872 - 65792) > 16360 65872 [out of 69632] + (65944 - 65872) > 16360 65944 [out of 69632] + (66016 - 65944) > 16360 66016 [out of 69632] + (66088 - 66016) > 16360 66088 [out of 69632] + (66168 - 66088) > 16360 66168 [out of 69632] + (66240 - 66168) > 16360 66240 [out of 69632] + (66312 - 66240) > 16360 66312 [out of 69632] + (66384 - 66312) > 16360 66384 [out of 69632] + (66456 - 66384) > 16360 66456 [out of 69632] + (66536 - 66456) > 16360 66536 [out of 69632] + (66608 - 66536) > 16360 66608 [out of 69632] + (66680 - 66608) > 16360 66680 [out of 69632] + (66752 - 66680) > 16360 66752 [out of 69632] + (66832 - 66752) > 16360 66832 [out of 69632] + (66904 - 66832) > 16360 66904 [out of 69632] + (66976 - 66904) > 16360 66976 [out of 69632] + (67048 - 66976) > 16360 67048 [out of 69632] + (67120 - 67048) > 16360 67120 [out of 69632] + (67200 - 67120) > 16360 67200 [out of 69632] + (67272 - 67200) > 16360 67272 [out of 69632] + (67344 - 67272) > 16360 67344 [out of 69632] + (67416 - 67344) > 16360 67416 [out of 69632] + (67488 - 67416) > 16360 67488 [out of 69632] + (67568 - 67488) > 16360 67568 [out of 69632] + (67640 - 67568) > 16360 67640 [out of 69632] + (67712 - 67640) > 16360 67712 [out of 69632] + (67784 - 67712) > 16360 67784 [out of 69632] + (67856 - 67784) > 16360 67856 [out of 69632] + (67928 - 67856) > 16360 67928 [out of 69632] + (68000 - 67928) > 16360 68000 [out of 69632] + (68072 - 68000) > 16360 68072 [out of 69632] + (68144 - 68072) > 16360 68144 [out of 69632] + (68216 - 68144) > 16360 68216 [out of 69632] + (68288 - 68216) > 16360 68288 [out of 69632] + (68360 - 68288) > 16360 68360 [out of 69632] + (68432 - 68360) > 16360 68432 [out of 69632] + (68504 - 68432) > 16360 68504 [out of 69632] + (68576 - 68504) > 16360 68576 [out of 69632] + (68648 - 68576) > 16360 68648 [out of 69632] + (68720 - 68648) > 16360 68720 [out of 69632] + (68792 - 68720) > 16360 68792 [out of 69632] + (68864 - 68792) > 16360 68864 [out of 69632] + (68936 - 68864) > 16360 68936 [out of 69632] + (69008 - 68936) > 16360 69008 [out of 69632] + (69080 - 69008) > 16360 69080 [out of 69632] + (69152 - 69080) > 16360 69152 [out of 69632] + (69224 - 69152) > 16360 69224 [out of 69632] + (69296 - 69224) > 16360 69296 [out of 69632] + (69368 - 69296) > 16360 69368 [out of 69632] + (69440 - 69368) > 16360 69440 [out of 69632] + (69512 - 69440) > 16360 69512 [out of 69632] + (69584 - 69512) > 16360 69584 [out of 69632] + (69656 - 69584) > 16360 69656 [out of 73728] + (69728 - 69656) > 16360 69728 [out of 73728] + (69800 - 69728) > 16360 69800 [out of 73728] + (69872 - 69800) > 16360 69872 [out of 73728] + (69944 - 69872) > 16360 69944 [out of 73728] + (70016 - 69944) > 16360 70016 [out of 73728] + (70088 - 70016) > 16360 70088 [out of 73728] + (70160 - 70088) > 16360 70160 [out of 73728] + (70232 - 70160) > 16360 70232 [out of 73728] + (70304 - 70232) > 16360 70304 [out of 73728] + (70376 - 70304) > 16360 70376 [out of 73728] + (70448 - 70376) > 16360 70448 [out of 73728] + (70520 - 70448) > 16360 70520 [out of 73728] + (70592 - 70520) > 16360 70592 [out of 73728] + (70664 - 70592) > 16360 70664 [out of 73728] + (70736 - 70664) > 16360 70736 [out of 73728] + (70808 - 70736) > 16360 70808 [out of 73728] + (70880 - 70808) > 16360 70880 [out of 73728] + (70952 - 70880) > 16360 70952 [out of 73728] + (71024 - 70952) > 16360 71024 [out of 73728] + (71096 - 71024) > 16360 71096 [out of 73728] + (71168 - 71096) > 16360 71168 [out of 73728] + (71240 - 71168) > 16360 71240 [out of 73728] + (71312 - 71240) > 16360 71312 [out of 73728] + (71384 - 71312) > 16360 71384 [out of 73728] + (71456 - 71384) > 16360 71456 [out of 73728] + (71528 - 71456) > 16360 71528 [out of 73728] + (71600 - 71528) > 16360 71600 [out of 73728] + (71672 - 71600) > 16360 71672 [out of 73728] + (71744 - 71672) > 16360 71744 [out of 73728] + (71816 - 71744) > 16360 71816 [out of 73728] + (71888 - 71816) > 16360 71888 [out of 73728] + (71960 - 71888) > 16360 71960 [out of 73728] + (72032 - 71960) > 16360 72032 [out of 73728] + (72104 - 72032) > 16360 72104 [out of 73728] + (72176 - 72104) > 16360 72176 [out of 73728] + (72248 - 72176) > 16360 72248 [out of 73728] + (72320 - 72248) > 16360 72320 [out of 73728] + (72392 - 72320) > 16360 72392 [out of 73728] + (72464 - 72392) > 16360 72464 [out of 73728] + (72536 - 72464) > 16360 72536 [out of 73728] + (72608 - 72536) > 16360 72608 [out of 73728] + (72680 - 72608) > 16360 72680 [out of 73728] + (72752 - 72680) > 16360 72752 [out of 73728] + (72824 - 72752) > 16360 72824 [out of 73728] + (72896 - 72824) > 16360 72896 [out of 73728] + (72968 - 72896) > 16360 72968 [out of 73728] + (73040 - 72968) > 16360 73040 [out of 73728] + (73112 - 73040) > 16360 73112 [out of 73728] + (73184 - 73112) > 16360 73184 [out of 73728] + (73256 - 73184) > 16360 73256 [out of 73728] + (73328 - 73256) > 16360 73328 [out of 73728] + (73400 - 73328) > 16360 73400 [out of 73728] + (73472 - 73400) > 16360 73472 [out of 73728] + (73544 - 73472) > 16360 73544 [out of 73728] + (73616 - 73544) > 16360 73616 [out of 73728] + (73688 - 73616) > 16360 73688 [out of 73728] + (73760 - 73688) > 16360 73760 [out of 77824] + (73832 - 73760) > 16360 73832 [out of 77824] + (73904 - 73832) > 16360 73904 [out of 77824] + (73976 - 73904) > 16360 73976 [out of 77824] + (74048 - 73976) > 16360 74048 [out of 77824] + (74120 - 74048) > 16360 74120 [out of 77824] + (74192 - 74120) > 16360 74192 [out of 77824] + (74264 - 74192) > 16360 74264 [out of 77824] + (74336 - 74264) > 16360 74336 [out of 77824] + (74408 - 74336) > 16360 74408 [out of 77824] + (74480 - 74408) > 16360 74480 [out of 77824] + (74552 - 74480) > 16360 74552 [out of 77824] + (74624 - 74552) > 16360 74624 [out of 77824] + (74696 - 74624) > 16360 74696 [out of 77824] + (74768 - 74696) > 16360 74768 [out of 77824] + (74840 - 74768) > 16360 74840 [out of 77824] + (74912 - 74840) > 16360 74912 [out of 77824] + (74984 - 74912) > 16360 74984 [out of 77824] + (75056 - 74984) > 16360 75056 [out of 77824] + (75128 - 75056) > 16360 75128 [out of 77824] + (75200 - 75128) > 16360 75200 [out of 77824] + (75272 - 75200) > 16360 75272 [out of 77824] + (75344 - 75272) > 16360 75344 [out of 77824] + (75416 - 75344) > 16360 75416 [out of 77824] + (75488 - 75416) > 16360 75488 [out of 77824] + (75560 - 75488) > 16360 75560 [out of 77824] + (75632 - 75560) > 16360 75632 [out of 77824] + (75704 - 75632) > 16360 75704 [out of 77824] + (75776 - 75704) > 16360 75776 [out of 77824] + (75848 - 75776) > 16360 75848 [out of 77824] + (75920 - 75848) > 16360 75920 [out of 77824] + (75992 - 75920) > 16360 75992 [out of 77824] + (76064 - 75992) > 16360 76064 [out of 77824] + (76136 - 76064) > 16360 76136 [out of 77824] + (76208 - 76136) > 16360 76208 [out of 77824] + (76280 - 76208) > 16360 76280 [out of 77824] + (76352 - 76280) > 16360 76352 [out of 77824] + (76424 - 76352) > 16360 76424 [out of 77824] + (76496 - 76424) > 16360 76496 [out of 77824] + (76568 - 76496) > 16360 76568 [out of 77824] + (76640 - 76568) > 16360 76640 [out of 77824] + (76712 - 76640) > 16360 76712 [out of 77824] + (76784 - 76712) > 16360 76784 [out of 77824] + (76856 - 76784) > 16360 76856 [out of 77824] + (76928 - 76856) > 16360 76928 [out of 77824] + (77000 - 76928) > 16360 77000 [out of 77824] + (77072 - 77000) > 16360 77072 [out of 77824] + (77144 - 77072) > 16360 77144 [out of 77824] + (77216 - 77144) > 16360 77216 [out of 77824] + (77288 - 77216) > 16360 77288 [out of 77824] + (77360 - 77288) > 16360 77360 [out of 77824] + (77432 - 77360) > 16360 77432 [out of 77824] + (77504 - 77432) > 16360 77504 [out of 77824] + (77576 - 77504) > 16360 77576 [out of 77824] + (77648 - 77576) > 16360 77648 [out of 77824] + (77720 - 77648) > 16360 77720 [out of 77824] + (77792 - 77720) > 16360 77792 [out of 77824] + (77864 - 77792) > 16360 77864 [out of 81920] + (77936 - 77864) > 16360 77936 [out of 81920] + (78008 - 77936) > 16360 78008 [out of 81920] + (78080 - 78008) > 16360 78080 [out of 81920] + (78152 - 78080) > 16360 78152 [out of 81920] + (78224 - 78152) > 16360 78224 [out of 81920] + (78296 - 78224) > 16360 78296 [out of 81920] + (78368 - 78296) > 16360 78368 [out of 81920] + (78440 - 78368) > 16360 78440 [out of 81920] + (78512 - 78440) > 16360 78512 [out of 81920] + (78584 - 78512) > 16360 78584 [out of 81920] + (78656 - 78584) > 16360 78656 [out of 81920] + (78728 - 78656) > 16360 78728 [out of 81920] + (78800 - 78728) > 16360 78800 [out of 81920] + (78872 - 78800) > 16360 78872 [out of 81920] + (78944 - 78872) > 16360 78944 [out of 81920] + (79016 - 78944) > 16360 79016 [out of 81920] + (79088 - 79016) > 16360 79088 [out of 81920] + (79160 - 79088) > 16360 79160 [out of 81920] + (79232 - 79160) > 16360 79232 [out of 81920] + (79304 - 79232) > 16360 79304 [out of 81920] + (79376 - 79304) > 16360 79376 [out of 81920] + (79448 - 79376) > 16360 79448 [out of 81920] + (79520 - 79448) > 16360 79520 [out of 81920] + (79592 - 79520) > 16360 79592 [out of 81920] + (79664 - 79592) > 16360 79664 [out of 81920] + (79736 - 79664) > 16360 79736 [out of 81920] + (79808 - 79736) > 16360 79808 [out of 81920] + (79880 - 79808) > 16360 79880 [out of 81920] + (79952 - 79880) > 16360 79952 [out of 81920] + (80024 - 79952) > 16360 80024 [out of 81920] + (80096 - 80024) > 16360 80096 [out of 81920] + (80168 - 80096) > 16360 80168 [out of 81920] + (80240 - 80168) > 16360 80240 [out of 81920] + (80312 - 80240) > 16360 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-09 23:36 ` Clint Adams @ 2004-05-09 23:51 ` Bart Schaefer 2004-05-10 10:16 ` Peter Stephenson 2004-05-10 16:19 ` Peter Stephenson 0 siblings, 2 replies; 19+ messages in thread From: Bart Schaefer @ 2004-05-09 23:51 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter On May 9, 7:30pm, Clint Adams wrote: } } > And you're saying that (h->used + (new - old) <= HEAP_ARENA_SIZE) is } > always false, so the zhalloc() is always called? } } I don't know that it's always false, but here's a small snapshot of } those comparisons during a zsh/files rm -rf attempt. } } [ h->used + (new - old) > HEAP_ARENA_SIZE ] } 63432 + (63504 - 63432) > 16360 Aha! Note that h->used > HEAP_ARENA_SIZE all by itself! Comparing to HEAP_ARENA_SIZE is likely wrong, it should compare to the maximum of either HEAP_ARENA_SIZE or the previously-mmapped page size. I'm not entirely sure of that ... ... but that's why it begins re-zhalloc()-ing. In the non-MMAP case it falls back on realloc() when a single allocation exceeds HEAP_ARENA_SIZE. So what probably needs to happen to prevent the out-of-memory condition is that we need to explicitly munmap the old mmap'd block, somewhere. And then we need to emulate realloc() behavior somehow, in terms of knowing whether to actually re-mmap-and-copy or whether the existing mapped pages are large enough to contain the additional requested space. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-09 23:51 ` Bart Schaefer @ 2004-05-10 10:16 ` Peter Stephenson 2004-05-10 14:09 ` Clint Adams 2004-05-10 15:54 ` Bart Schaefer 2004-05-10 16:19 ` Peter Stephenson 1 sibling, 2 replies; 19+ messages in thread From: Peter Stephenson @ 2004-05-10 10:16 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter Bart Schaefer wrote: > } [ h->used + (new - old) > HEAP_ARENA_SIZE ] > } 63432 + (63504 - 63432) > 16360 > > Aha! Note that h->used > HEAP_ARENA_SIZE all by itself! > > Comparing to HEAP_ARENA_SIZE is likely wrong, it should compare to the > maximum of either HEAP_ARENA_SIZE or the previously-mmapped page size. > I'm not entirely sure of that ... > > ... but that's why it begins re-zhalloc()-ing. In the non-MMAP case it > falls back on realloc() when a single allocation exceeds HEAP_ARENA_SIZE. > > So what probably needs to happen to prevent the out-of-memory condition > is that we need to explicitly munmap the old mmap'd block, somewhere. > > And then we need to emulate realloc() behavior somehow, in terms of > knowing whether to actually re-mmap-and-copy or whether the existing > mapped pages are large enough to contain the additional requested space. Looks like a rewrite of hrealloc is definitely in order... Does the bad behaviour go away if USE_MMAP is undefined? -- Peter Stephenson <pws@csr.com> Software Engineer CSR Ltd., Science Park, Milton Road, Cambridge, CB4 0WH, UK Tel: +44 (0)1223 692070 ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ********************************************************************** ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-10 10:16 ` Peter Stephenson @ 2004-05-10 14:09 ` Clint Adams 2004-05-10 15:54 ` Bart Schaefer 1 sibling, 0 replies; 19+ messages in thread From: Clint Adams @ 2004-05-10 14:09 UTC (permalink / raw) To: Peter Stephenson; +Cc: zsh-workers, 245678-submitter > Looks like a rewrite of hrealloc is definitely in order... > > Does the bad behaviour go away if USE_MMAP is undefined? It performs the rm -rf quickly, and there is only one instance in which h->used + (new - old) is greater than HEAP_ARENA_SIZE: 16336 [out of 0] + (15984 - 15912) > 16360 Odd that h->size seems to be 0. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-10 10:16 ` Peter Stephenson 2004-05-10 14:09 ` Clint Adams @ 2004-05-10 15:54 ` Bart Schaefer 1 sibling, 0 replies; 19+ messages in thread From: Bart Schaefer @ 2004-05-10 15:54 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter On Mon, 10 May 2004, Peter Stephenson wrote: > Looks like a rewrite of hrealloc is definitely in order... Just the MMAP part. Though we might consider increasing HEAP_ARENA_SIZE; 16K isn't much on modern machinery. > Does the bad behaviour go away if USE_MMAP is undefined? Examination of the code suggests so, as does Clint's test result. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-09 23:51 ` Bart Schaefer 2004-05-10 10:16 ` Peter Stephenson @ 2004-05-10 16:19 ` Peter Stephenson 2004-05-10 16:51 ` Bart Schaefer 1 sibling, 1 reply; 19+ messages in thread From: Peter Stephenson @ 2004-05-10 16:19 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter Bart Schaefer wrote: > } [ h->used + (new - old) > HEAP_ARENA_SIZE ] > } 63432 + (63504 - 63432) > 16360 > > Aha! Note that h->used > HEAP_ARENA_SIZE all by itself! > > Comparing to HEAP_ARENA_SIZE is likely wrong, it should compare to the > maximum of either HEAP_ARENA_SIZE or the previously-mmapped page size. This is all ghastly even by zsh standards. I thought what you thought, until... Look at zhalloc(), where either we allocate n bytes normally, or we mmap an equivalent chunk of memory. Neither of those explicitly use HEAP_ARENA_SIZE, which was used to get n in the first place. We are probably wasting space since the only thing we record for later use is the original size asked for (`size') as the used size; we don't record the number we've ended up with (`n'), goodness knows why. For USE_MMAP, we do record n in h->size, but this is only used for munmap, not for everything else --- it's obviously an afterthought. (That's why it's zero when USE_MMAP isn't defined.) The smart thing to do would be always record it and use it for subsequent calculation instead of all the jiggery pokery with rounding things to boundaries in obscure ways. This is basically your conclusion, however I think it's just unfinished business rather than a programming error. However, I don't think that's actually the source of the problem. Closer to the point is this: #ifndef USE_MMAP if (old > HEAP_ARENA_SIZE || new > HEAP_ARENA_SIZE) { size_t n = HEAP_ARENA_SIZE > new ? HEAPSIZE : new + sizeof(*h); if (ph) ph->next = h = (Heap) realloc(h, n); else heaps = h = (Heap) realloc(h, n); } h->used = new; unqueue_signals(); return arena(h); #endif Without USE_MMAP, this takes care of making sure the heap is the right size when it's larger than normal. Those tests would be triggering in the case we're looking at. As they're not, we're dropping through to the zhalloc() below every time. What we should do is the equivalent of realloc as an #else of the code above --- mmap a larger chunk, unmap the old one, and return the new one. However, I don't think even *that* is the basic problem. I think this: old = (old + H_ISIZE - 1) & ~(H_ISIZE - 1); new = (new + H_ISIZE - 1) & ~(H_ISIZE - 1); is wrong --- it should be the heap size here, not H_ISIZE which is simply sizeof(union mem_align), which is probably only 4 or 8 words. We only need to reallocate to heap size boundaries, i.e. 16384 possibly minus something or other. We've got away with it before because the realloc() quoted above isn't too fussed, an hrealloc() isn't used that often. So actually Clint's original proposal is close to the mark. I hope. I might be tempted by proposed solutions involving the complete removal of hrealloc. Please don't use sentences containing `why' and ending with `?' or I'll start to whimper. Anyone fixing this could useful add comments. -- Peter Stephenson <pws@csr.com> Software Engineer CSR Ltd., Science Park, Milton Road, Cambridge, CB4 0WH, UK Tel: +44 (0)1223 692070 ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ********************************************************************** ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-10 16:19 ` Peter Stephenson @ 2004-05-10 16:51 ` Bart Schaefer 2004-05-10 17:23 ` Peter Stephenson 0 siblings, 1 reply; 19+ messages in thread From: Bart Schaefer @ 2004-05-10 16:51 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter On Mon, 10 May 2004, Peter Stephenson wrote: > This is all ghastly even by zsh standards. Well, that's what comes of having glued MMAP into the hrealloc() algorithm piecemeal. Probably what should have been done was just to #ifdef the entire function body. > What we should do is the equivalent of realloc as an #else of the code > above --- mmap a larger chunk, unmap the old one, and return the new > one. I think I said that, though I hadn't got so far as to pinpoint where it needed to happen. > However, I don't think even *that* is the basic problem. I think this: > > old = (old + H_ISIZE - 1) & ~(H_ISIZE - 1); > new = (new + H_ISIZE - 1) & ~(H_ISIZE - 1); > > is wrong --- it should be the heap size here, not H_ISIZE which is > simply sizeof(union mem_align), which is probably only 4 or 8 words. It took me a while staring at this to decide that it was doing the right thing, but I came to the conclusion that it's only actually _wrong_ if HEAP_ARENA_SIZE is _not_ a multiple of H_ISIZE. And even then I'm pretty sure "wrong" just means "inefficient". > We only need to reallocate to heap size boundaries That's correct, but we _do_ only reallocate to heap size boundaries in the non-MMAP case. I think. It's only the MMAP'd code, which as I said was pasted in long after the original algorithm was written, that botches it. > I might be tempted by proposed solutions involving the complete removal > of hrealloc. Which would mean what, exactly? Never grow heap blocks? (Note that I didn't begin with "why".) ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Bug#245678: zsh: built-in rm -rf fills up the memory 2004-05-10 16:51 ` Bart Schaefer @ 2004-05-10 17:23 ` Peter Stephenson 0 siblings, 0 replies; 19+ messages in thread From: Peter Stephenson @ 2004-05-10 17:23 UTC (permalink / raw) To: zsh-workers; +Cc: 245678-submitter Bart Schaefer wrote: > > However, I don't think even *that* is the basic problem. I think this: > > > > old = (old + H_ISIZE - 1) & ~(H_ISIZE - 1); > > new = (new + H_ISIZE - 1) & ~(H_ISIZE - 1); > > > > is wrong --- it should be the heap size here, not H_ISIZE which is > > simply sizeof(union mem_align), which is probably only 4 or 8 words. > > It took me a while staring at this to decide that it was doing the right > thing, but I came to the conclusion that it's only actually _wrong_ if > HEAP_ARENA_SIZE is _not_ a multiple of H_ISIZE. And even then I'm pretty > sure "wrong" just means "inefficient". I'd have to follow through with a debugger, really... But I can't see any mechanism to prevent it reallocating for every few bytes of increase once the size goes beyond HEAP_ARENA_SIZE, which can't be the intention. The chunk #ifdef'd out for USE_MMAP is the first point where we see if we have enough space for `new' already (except for special cases old == new, we allocated the space then shrank, new is NULL). So for new and old larger than HEAP_ARENA_SIZE (which is bound to happen after the first attempt to grow), we call realloc() every time the difference crosses an H_ISIZE boundary. What have I missed? > > We only need to reallocate to heap size boundaries > > That's correct, but we _do_ only reallocate to heap size boundaries in the > non-MMAP case. I think. It's only the MMAP'd code, which as I said was > pasted in long after the original algorithm was written, that botches it. Yes, but we then call realloc unnecessarily in between actually growing the heap, unless my logic above is wrong. It will reallocate even though the new size of the heap is the same as the old one. > > I might be tempted by proposed solutions involving the complete removal > > of hrealloc. > > Which would mean what, exactly? Never grow heap blocks? hrealloc is only used as an optimisation. If necessary we could always allocate new space. In the case of the files module we could use zalloc and zrealloc and let the system worry about the problem. Then we lose MMAP altogether, of course. -- Peter Stephenson <pws@csr.com> Software Engineer CSR Ltd., Science Park, Milton Road, Cambridge, CB4 0WH, UK Tel: +44 (0)1223 692070 ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ********************************************************************** ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2004-05-10 17:24 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20040424162150.GA4210@ay.vinc17.org>
2004-04-25 1:36 ` Bug#245678: zsh: built-in rm -rf fills up the memory Clint Adams
2004-04-25 20:45 ` Bart Schaefer
2004-04-25 21:12 ` Clint Adams
2004-04-25 21:38 ` Clint Adams
2004-04-26 17:10 ` Bart Schaefer
2004-05-08 4:35 ` Clint Adams
2004-05-08 14:02 ` Clint Adams
2004-05-08 14:13 ` Clint Adams
2004-05-09 22:54 ` Bart Schaefer
2004-05-09 22:48 ` Bart Schaefer
2004-05-09 23:30 ` Clint Adams
2004-05-09 23:36 ` Clint Adams
2004-05-09 23:51 ` Bart Schaefer
2004-05-10 10:16 ` Peter Stephenson
2004-05-10 14:09 ` Clint Adams
2004-05-10 15:54 ` Bart Schaefer
2004-05-10 16:19 ` Peter Stephenson
2004-05-10 16:51 ` Bart Schaefer
2004-05-10 17:23 ` Peter Stephenson
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).