infinite recursion when handling the "out of memory" state

[Kamil Dudka <kdudka@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

If zsh is compiled with multibyte support, handling of the "out of memory" 
state does not work well in certain cases -- instead of printing the error 
message and exiting, zsh ends up in an infinite recursion and crashes due to 
stack overflow.

The memory allocation functions in mem.c use zerr() to print the fatal error 
messages.  However, zerr() calls zwarning() and transitively mb_niceformat(), 
which allocates heap memory (and may call zerr() on failure).

I see three options how to prevent the stack overflow in such cases:

1. avoid using zerr() to print "out of memory" error messages

2. implement zerr() such that it does not trigger heap memory allocation
(I am afraid this is not really possible)

3. introduce a flag that would prevent zerr() from recurring into itself

I have attached a reproducer script.  It works reliably if the zsh executable 
is linked statically.  Otherwise it may happen that the dynamic linker is more 
hungry on memory than zsh itself, which would hide the bug.

Originally reported at: https://bugzilla.redhat.com/1300958

Kamil

[-- Attachment #2: oom.zsh --]
[-- Type: application/x-shellscript, Size: 699 bytes --]