Re: infinite recursion when handling the "out of memory" state

[Peter Stephenson <p.stephenson@samsung.com>]

On Mon, 25 Jan 2016 16:02:50 +0100
Kamil Dudka <kdudka@redhat.com> wrote:
> If zsh is compiled with multibyte support, handling of the "out of memory" 
> state does not work well in certain cases -- instead of printing the error 
> message and exiting, zsh ends up in an infinite recursion and crashes due to 
> stack overflow.
> 
> The memory allocation functions in mem.c use zerr() to print the fatal error 
> messages.  However, zerr() calls zwarning() and transitively mb_niceformat(), 
> which allocates heap memory (and may call zerr() on failure).
> 
> 3. introduce a flag that would prevent zerr() from recurring into itself

I think this is easy --- we already do this except that the flag is set
after the warning is printed rather than before.  The zwarning()
function that is called in the middle isn't directly sensitive to the
flag, so moving the code up should have the desired effect and is the
sane thing to do with error messages anyway.

diff --git a/Src/utils.c b/Src/utils.c
index fd0bab3..17ebfeb 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -169,12 +169,12 @@ VA_DCL
 	    errflag |= ERRFLAG_ERROR;
 	return;
     }
+    errflag |= ERRFLAG_ERROR;
 
     VA_START(ap, fmt);
     VA_GET_ARG(ap, fmt, const char *);
     zwarning(NULL, fmt, ap);
     va_end(ap);
-    errflag |= ERRFLAG_ERROR;
 }
 
 /**/
@@ -188,13 +188,13 @@ VA_DCL
 
     if (errflag || noerrs)
 	return;
+    errflag |= ERRFLAG_ERROR;
 
     VA_START(ap, fmt);
     VA_GET_ARG(ap, cmd, const char *);
     VA_GET_ARG(ap, fmt, const char *);
     zwarning(cmd, fmt, ap);
     va_end(ap);
-    errflag |= ERRFLAG_ERROR;
 }
 
 /**/