zsh behavior when fork() failed

zsh behavior when fork() failed

[Dipak Gaigole]

Hi,

Recently I have observed that the behavior of zsh differs from other
shells like bash, ksh whenever fork () fails.
In order to simulate and make fork () fail with EAGAIN error I have
used "ulimit -u <no>"

BASH:
bash-2.05b$ ulimit -u 11
bash-2.05b$ echo $$
4699
bash-2.05b$ bash
bash: fork: Resource temporarily unavailable
bash-2.05b$ echo $$
4824
bash-2.05b$ date
bash: fork: Resource temporarily unavailable
bash-2.05b$ echo $?
1
bash-2.05b$


KSH:
bash-2.05b$ ksh
$ date
ksh: cannot fork - try again
$ echo $?
1
$


ZSH:
bash-2.05b$ zsh
/etc/zshrc:21: fork failed: resource temporarily unavailable
[dipak@rhas30]/tmp% date
zsh: fork failed: resource temporarily unavailable
[dipak@rhas30]/tmp% echo $?
0
[dipak@rhas30]/tmp%


As we can see that in zsh whenever fork() fails with EAGAIN, the
return status is incorrect (i.e. $? is 0) and this causes further
failures if this happens in a big script where script's progress
depends on execution of previous command i.e $?

I have tried this with zsh version 4.0.7, 4.3.4 as well as 4.3.15
(latest as of now I suppose) and found the same behavior.

So this looks like a bug to me. Please correct me if I am wrong.


Thanks,
Dipak

Re: zsh behavior when fork() failed

[Bart Schaefer]

On Feb 23,  4:10pm, Dipak Gaigole wrote:
}
} [dipak@rhas30]/tmp% date
} zsh: fork failed: resource temporarily unavailable
} [dipak@rhas30]/tmp% echo $?
} 0

There are several places where zsh forks without obviously propagating
the error state, but I only found one case where $? was zero after the
command.  If the command is part of a pipeline, or is in $(...) or any
process substitution, then $? is properly set to 1.

However, there are other differences from bash.  For example, in zsh,
the "not" operator inverts the fork-failed error state:

torch% date | read -E
zsh: fork failed: resource temporarily unavailable
torch% print $?
1
torch% ! date | read -E
zsh: fork failed: resource temporarily unavailable
torch% print $?
0

In bash, the result is 1 in both cases.

The appended patch seems to fix both situations, but in some of the
instances of fork failure I haven't traced through how lastval and
errflag both get set (except it doesn't happen at the point of the
fork) so it is possible that some form of command is still wrong.

} As we can see that in zsh whenever fork() fails with EAGAIN, the
} return status is incorrect (i.e. $? is 0) and this causes further
} failures if this happens in a big script where script's progress
} depends on execution of previous command i.e $?

Can you provide an example script?  A simple one that I put together
bails out on fork failure even though $? is zero.

Perhaps you mean a case where script A calls script B, then script B
fails but script A proceeds because the error wasn't propagated?

Index: Src/exec.c
===================================================================
--- Src/exec.c	20 Dec 2011 17:13:38 -0000	1.43
+++ Src/exec.c	23 Feb 2012 16:07:50 -0000
@@ -1617,9 +1617,8 @@
 		 (list_pipe || (pline_level && !(jn->stat & STAT_SUBJOB)))))
 		deletejob(jn, 0);
 	    thisjob = pj;
-
 	}
-	if (slflags & WC_SUBLIST_NOT)
+	if ((slflags & WC_SUBLIST_NOT) && !errflag)
 	    lastval = !lastval;
     }
     if (!pline_level)
@@ -2810,6 +2820,7 @@
 	    close(synch[1]);
 	    if (oautocont >= 0)
 		opts[AUTOCONTINUE] = oautocont;
+	    lastval = errflag = 1;
 	    return;
 	}
 	if (pid) {

Re: zsh behavior when fork() failed

[Dipak Gaigole]

On Thu, Feb 23, 2012 at 9:44 PM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
>
> Perhaps you mean a case where script A calls script B, then script B
> fails but script A proceeds because the error wasn't propagated?

Yes, you are right. This is what I mean to say.

>
> Index: Src/exec.c
> ===================================================================
> --- Src/exec.c  20 Dec 2011 17:13:38 -0000      1.43
> +++ Src/exec.c  23 Feb 2012 16:07:50 -0000
> @@ -1617,9 +1617,8 @@
>                 (list_pipe || (pline_level && !(jn->stat & STAT_SUBJOB)))))
>                deletejob(jn, 0);
>            thisjob = pj;
> -
>        }
> -       if (slflags & WC_SUBLIST_NOT)
> +       if ((slflags & WC_SUBLIST_NOT) && !errflag)
>            lastval = !lastval;
>     }
>     if (!pline_level)
> @@ -2810,6 +2820,7 @@
>            close(synch[1]);
>            if (oautocont >= 0)
>                opts[AUTOCONTINUE] = oautocont;
> +           lastval = errflag = 1;
>            return;
>        }
>        if (pid) {
>

I have applied this patch and the newly built zsh returns proper $?
(i.e. 1) whenever fork fails. So this has fixed the behavior of error
propagation, but the script behavior still looks different.

Here is a simple script example:

bash-2.05b$ cat /tmp/test.sh
#!/bin/sh
x="My default value"
x=`date`
echo $?
echo "Current Date is:" "$x"
date
echo $?
bash-2.05b$


BASH:
bash-2.05b$ bash -x  /tmp/test.sh
+ x=My default value
/tmp/test.sh: fork: Resource temporarily unavailable
bash-2.05b$ echo $?
128
bash-2.05b$


KSH:
bash-2.05b$ ksh -x  /tmp/test.sh
+ x=My default value
/tmp/test.sh[3]: cannot fork - try again
bash-2.05b$ echo $?
1
bash-2.05b$


ZSH: (Applied above code patch)
bash-2.05b$ zsh -x  /tmp/test.sh
+/tmp/test.sh:2> x='My default value'
+/tmp/test.sh:3> x=/tmp/test.sh:3: fork failed: resource temporarily unavailable

+/tmp/test.sh:4> echo 1
1
+/tmp/test.sh:5> echo 'Current Date is:' 'My default value'
Current Date is: My default value
/tmp/test.sh:7: fork failed: resource temporarily unavailable
+/tmp/test.sh:8> echo 1
1
bash-2.05b$ echo $?
0
bash-2.05b$


As we can see that zsh continues even if it knows that it has failed
in fork and finally the script return status is 0.
Also checking for $? after each command is not feasible. So doesn't
this zsh behavior looks misleading?

Thanks,
Dipak

Re: zsh behavior when fork() failed

[Bart Schaefer]

On Feb 24,  4:38pm, Dipak Gaigole wrote:
}
} As we can see that zsh continues even if it knows that it has failed
} in fork and finally the script return status is 0.

The script return status is zero because the last command it executed
was successful.  That has nothing to do with the rest of the question.

} Also checking for $? after each command is not feasible. So doesn't
} this zsh behavior looks misleading?

Depends on whether fork failure is supposed to be fatal to the shell.
I looked at:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html

Section 2.8.1 lists command execution failures where a non-interactive
shell "shall exit", but fork failure is not among them.

Re: zsh behavior when fork() failed

[Dipak Gaigole]

On Fri, Feb 24, 2012 at 11:35 PM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Feb 24,  4:38pm, Dipak Gaigole wrote:
> }
> } As we can see that zsh continues even if it knows that it has failed
> } in fork and finally the script return status is 0.
>
> The script return status is zero because the last command it executed
> was successful.  That has nothing to do with the rest of the question.
>
Yes, I understand.

> } Also checking for $? after each command is not feasible. So doesn't
> } this zsh behavior looks misleading?
>
> Depends on whether fork failure is supposed to be fatal to the shell.
> I looked at:
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
>
> Section 2.8.1 lists command execution failures where a non-interactive
> shell "shall exit", but fork failure is not among them.

Got it. But I don't want to add a check of $? after each command in
the scripts specially in cases where scripts are 1000+ lines.

Is there any zsh option to make sure script exit when fork fails? Else
can I patch the zsh code to make script exit on fork failure?

Thanks,
Dipak

Re: zsh behavior when fork() failed

[Bart Schaefer]

On Feb 25, 10:03pm, Dipak Gaigole wrote:
} Subject: Re: zsh behavior when fork() failed
}
} > Section 2.8.1 lists command execution failures where a non-interactive
} > shell "shall exit", but fork failure is not among them.
} 
} Got it.

That doesn't mean I didn't miss something.  (I was kind of in a hurry
on Friday ...)

This also raises the question of whether a non-interactive shell should
exit on a botched "exec" (see zsh-workers/30111 and subsequent thread,
and comment on line 3364 excerpted below).

} But I don't want to add a check of $? after each command in
} the scripts specially in cases where scripts are 1000+ lines.

Would "set -e" (setopt err_exit) do what you need?

} Is there any zsh option to make sure script exit when fork fails? Else
} can I patch the zsh code to make script exit on fork failure?

It's entirely possible that "return;" on lines 2818 and 2824 in exec.c:

2814         if (pipe(synch) < 0) {
2815             zerr("pipe failed: %e", errno);
2816             if (oautocont >= 0)
2817                 opts[AUTOCONTINUE] = oautocont;
2818             return;
2819         } else if ((pid = zfork(&bgtime)) == -1) {
2820             close(synch[0]);
2821             close(synch[1]);
2822             if (oautocont >= 0)
2823                 opts[AUTOCONTINUE] = oautocont;
2824             return;
2825         }

should instead behave the way POSIXBUILTINS does:

3362         /*
3363          * For POSIX-compatible behaviour with special
3364          * builtins (including exec which we don't usually
3365          * classify as a builtin) we treat all errors as fatal.
3366          * The "command" builtin is not special so resets this behaviour.
3367          */
3368         if (redir_err || errflag) {
3369             if (!isset(INTERACTIVE)) {
3370                 if (forked)
3371                     _exit(1);
3372                 else
3373                     exit(1);
3374             }
3375             errflag = 1;
3376         }

There are a couple of other cases (e.g., around line 1680) where's also
possible the error should be considered unrecoverable.

That would make a full patch look something like the following (which
still exits with 1, rather than with 128 like bash, but does exit).
The oautocont stuff goes away because it's handled below the "fatal"
label (outside the visible diff context).  The second hunk still might
not cause a proper exit, it's in execpline2() from which it's less clear
to me how to reach the "fatal" condition but it's pretty obvious that
it shouldn't just be falling through in those error states.


Index: Src/exec.c
--- ../zsh-forge/current/Src/exec.c	2012-02-12 13:31:49.000000000 -0800
+++ Src/exec.c	2012-02-26 11:47:48.000000000 -0800
@@ -1617,9 +1617,8 @@
 		 (list_pipe || (pline_level && !(jn->stat & STAT_SUBJOB)))))
 		deletejob(jn, 0);
 	    thisjob = pj;
-
 	}
-	if (slflags & WC_SUBLIST_NOT)
+	if ((slflags & WC_SUBLIST_NOT) && !errflag)
 	    lastval = !lastval;
     }
     if (!pline_level)
@@ -1679,9 +1678,13 @@
 
 	    if (pipe(synch) < 0) {
 		zerr("pipe failed: %e", errno);
+		lastval = errflag = 1;
+		return;
 	    } else if ((pid = zfork(&bgtime)) == -1) {
 		close(synch[0]);
 		close(synch[1]);
+		lastval = errflag = 1;
+		return;
 	    } else if (pid) {
 		char dummy, *text;
 
@@ -2490,7 +2493,7 @@
 		    if (!firstnode(args)) {
 			zerr("exec requires a command to execute");
 			errflag = lastval = 1;
-			return;
+			goto fatal;
 		    }
 		    uremnode(args, firstnode(args));
 		    if (!strcmp(next, "--"))
@@ -2507,12 +2510,12 @@
 				if (!firstnode(args)) {
 				    zerr("exec requires a command to execute");
 				    errflag = lastval = 1;
-				    return;
+				    goto fatal;
 				}
 				if (!nextnode(firstnode(args))) {
 				    zerr("exec flag -a requires a parameter");
 				    errflag = lastval = 1;
-				    return;
+				    goto fatal;
 				}
 				exec_argv0 = (char *)
 				    getdata(nextnode(firstnode(args)));
@@ -2813,15 +2816,12 @@
 
 	if (pipe(synch) < 0) {
 	    zerr("pipe failed: %e", errno);
-	    if (oautocont >= 0)
-		opts[AUTOCONTINUE] = oautocont;
-	    return;
+	    goto fatal;
 	} else if ((pid = zfork(&bgtime)) == -1) {
 	    close(synch[0]);
 	    close(synch[1]);
-	    if (oautocont >= 0)
-		opts[AUTOCONTINUE] = oautocont;
-	    return;
+	    lastval = errflag = 1;
+	    goto fatal;
 	}
 	if (pid) {
 
@@ -3365,6 +3365,7 @@
 	 * classify as a builtin) we treat all errors as fatal.
 	 * The "command" builtin is not special so resets this behaviour.
 	 */
+    fatal:
 	if (redir_err || errflag) {
 	    if (!isset(INTERACTIVE)) {
 		if (forked)

Re: zsh behavior when fork() failed

[Bart Schaefer]

On Feb 26, 11:52am, Bart Schaefer wrote:
}
} This also raises the question of whether a non-interactive shell should
} exit on a botched "exec" (see zsh-workers/30111 and subsequent thread,
} and comment on line 3364 excerpted below).

Interestingly, bash does not exit on an error in exec's argument list.

Re: zsh behavior when fork() failed

[Dipak Gaigole]

> There are a couple of other cases (e.g., around line 1680) where's also
> possible the error should be considered unrecoverable.
>
> That would make a full patch look something like the following (which
> still exits with 1, rather than with 128 like bash, but does exit).
> The oautocont stuff goes away because it's handled below the "fatal"
> label (outside the visible diff context).  The second hunk still might
> not cause a proper exit, it's in execpline2() from which it's less clear
> to me how to reach the "fatal" condition but it's pretty obvious that
> it shouldn't just be falling through in those error states.
>
>
> Index: Src/exec.c
> --- ../zsh-forge/current/Src/exec.c     2012-02-12 13:31:49.000000000 -0800
> +++ Src/exec.c  2012-02-26 11:47:48.000000000 -0800
> @@ -1617,9 +1617,8 @@
>                 (list_pipe || (pline_level && !(jn->stat & STAT_SUBJOB)))))
>                deletejob(jn, 0);
>            thisjob = pj;
> -
>        }
> -       if (slflags & WC_SUBLIST_NOT)
> +       if ((slflags & WC_SUBLIST_NOT) && !errflag)
>            lastval = !lastval;
>     }
>     if (!pline_level)
> @@ -1679,9 +1678,13 @@
>
>            if (pipe(synch) < 0) {
>                zerr("pipe failed: %e", errno);
> +               lastval = errflag = 1;
> +               return;
>            } else if ((pid = zfork(&bgtime)) == -1) {
>                close(synch[0]);
>                close(synch[1]);
> +               lastval = errflag = 1;
> +               return;
>            } else if (pid) {
>                char dummy, *text;
>
> @@ -2490,7 +2493,7 @@
>                    if (!firstnode(args)) {
>                        zerr("exec requires a command to execute");
>                        errflag = lastval = 1;
> -                       return;
> +                       goto fatal;
>                    }
>                    uremnode(args, firstnode(args));
>                    if (!strcmp(next, "--"))
> @@ -2507,12 +2510,12 @@
>                                if (!firstnode(args)) {
>                                    zerr("exec requires a command to execute");
>                                    errflag = lastval = 1;
> -                                   return;
> +                                   goto fatal;
>                                }
>                                if (!nextnode(firstnode(args))) {
>                                    zerr("exec flag -a requires a parameter");
>                                    errflag = lastval = 1;
> -                                   return;
> +                                   goto fatal;
>                                }
>                                exec_argv0 = (char *)
>                                    getdata(nextnode(firstnode(args)));
> @@ -2813,15 +2816,12 @@
>
>        if (pipe(synch) < 0) {
>            zerr("pipe failed: %e", errno);
> -           if (oautocont >= 0)
> -               opts[AUTOCONTINUE] = oautocont;
> -           return;
> +           goto fatal;
>        } else if ((pid = zfork(&bgtime)) == -1) {
>            close(synch[0]);
>            close(synch[1]);
> -           if (oautocont >= 0)
> -               opts[AUTOCONTINUE] = oautocont;
> -           return;
> +           lastval = errflag = 1;
> +           goto fatal;
>        }
>        if (pid) {
>
> @@ -3365,6 +3365,7 @@
>         * classify as a builtin) we treat all errors as fatal.
>         * The "command" builtin is not special so resets this behaviour.
>         */
> +    fatal:
>        if (redir_err || errflag) {
>            if (!isset(INTERACTIVE)) {
>                if (forked)

I have applied this patch but unfortunately the script still continues
even after the fork failures while executing commands from the script.

bash-2.05b$ cat /tmp/test1.sh
#!/bin/sh
x="My default value"
x=`date`
echo $?
echo "Current Date is:" "$x"
date
echo $?
bash-2.05b$ zsh /tmp/test1.sh
/tmp/test1.sh:3: fork failed: resource temporarily unavailable
1
Current Date is: My default value
/tmp/test1.sh:6: fork failed: resource temporarily unavailable
1
bash-2.05b$

Can you please check this on your end?

Thanks,
Dipak

Re: zsh behavior when fork() failed

[Bart Schaefer]

On Feb 29,  6:27pm, Dipak Gaigole wrote:
}
} I have applied this patch but unfortunately the script still continues
} even after the fork failures while executing commands from the script.
} 
} bash-2.05b$ cat /tmp/test1.sh
} #!/bin/sh
} x="My default value"
} x=`date`
} echo $?
} echo "Current Date is:" "$x"
} date
} echo $?
} bash-2.05b$ zsh /tmp/test1.sh
} /tmp/test1.sh:3: fork failed: resource temporarily unavailable
} 1
} Current Date is: My default value
} /tmp/test1.sh:6: fork failed: resource temporarily unavailable
} 1
} bash-2.05b$
} 
} Can you please check this on your end?

I get (schaefer[N] is my top-level prompt):

schaefer[691] Src/zsh -f /tmp/test1.sh
/tmp/test1.sh:3: fork failed: resource temporarily unavailable
1
Current Date is: My default value
/tmp/test1.sh:6: fork failed: resource temporarily unavailable
schaefer[692] echo $?
1

There's no exit after `date` because that was in a subshell, but I do
see the exit after the command in the parent shell.

Are you sure a path-searched "zsh" is finding the right binary?

Re: zsh behavior when fork() failed

[Dipak Gaigole]

On Thu, Mar 1, 2012 at 12:36 AM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Feb 29,  6:27pm, Dipak Gaigole wrote:
> }
> } I have applied this patch but unfortunately the script still continues
> } even after the fork failures while executing commands from the script.
> }
> } bash-2.05b$ cat /tmp/test1.sh
> } #!/bin/sh
> } x="My default value"
> } x=`date`
> } echo $?
> } echo "Current Date is:" "$x"
> } date
> } echo $?
> } bash-2.05b$ zsh /tmp/test1.sh
> } /tmp/test1.sh:3: fork failed: resource temporarily unavailable
> } 1
> } Current Date is: My default value
> } /tmp/test1.sh:6: fork failed: resource temporarily unavailable
> } 1
> } bash-2.05b$
> }
> } Can you please check this on your end?
>
> I get (schaefer[N] is my top-level prompt):
>
> schaefer[691] Src/zsh -f /tmp/test1.sh
> /tmp/test1.sh:3: fork failed: resource temporarily unavailable
> 1
> Current Date is: My default value
> /tmp/test1.sh:6: fork failed: resource temporarily unavailable
> schaefer[692] echo $?
> 1
>
> There's no exit after `date` because that was in a subshell, but I do
> see the exit after the command in the parent shell.
>
> Are you sure a path-searched "zsh" is finding the right binary?

Sorry my bad :(  I had missed one of the fatal condition.

Thanks for your help Bart !!!

-Dipak