[BUG] 'exec' runs shell functions and builtins

[BUG] 'exec' runs shell functions and builtins

[Martijn Dekker]

In zsh, 'exec' looks up shell functions and builtins before external
commands, and if it finds one it appears to do the equivalent of running
the function or builtin followed by 'exit'. This is different from most
other shells and turns out[1] to be a bug in POSIX terms; 'exec' is
supposed to launch a program that overlays the current shell[2],
implying the program launched by 'exec' is always external to the shell.

This means that
    (exec commandname arguments ...)
is a POSIXly correct way of guaranteeing the execution of an external
command without specifying the path (something the "command" command
doesn't do in POSIX mode). This was confirmed the other day by Geoff
Clare and Chet Ramey on austin-group-l. (I seem to recall Bart asked on
this list quite some time ago if POSIX provides for a way to do this.)

This behaviour also appears to be contrary to the documentation in
zshbuiltins(1) ("Replace the current shell with an external command
rather than forking").

Test script:

testFn() {
        exec :
}
if (PATH=/dev/null exec testFn) 2>/dev/null; then
        echo "this shell execs both functions and builtins"
fi

(The only other POSIX-ish shell that zsh has this in common with is
pdksh/mksh. In pdksh this is definitely a bug as this behaviour differs
from ksh88 which pdksh is supposed to be a clone of. The POSIX spec is
also based on ksh88 behaviour.)

Confirmed down to 4.3.17 (I didn't check older versions).

- M.

[1] https://www.mail-archive.com/austin-group-l@opengroup.org/msg01469.html

[2]
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_20_14

Re: [BUG] 'exec' runs shell functions and builtins

[Peter Stephenson]

On Tue, 25 Jul 2017 22:49:33 +0100
Martijn Dekker <martijn@inlv.org> wrote:
> In zsh, 'exec' looks up shell functions and builtins before external
> commands, and if it finds one it appears to do the equivalent of running
> the function or builtin followed by 'exit'. This is different from most
> other shells and turns out[1] to be a bug in POSIX terms; 'exec' is
> supposed to launch a program that overlays the current shell[2],
> implying the program launched by 'exec' is always external to the shell.

Prior art suggests we can get away with adding this behaviour to the
POSIX_BUILTINS option.  (I'd like to hope people don't actually
set the POSIX options separately anyway, but feel free not to tell me.)

pws

diff --git a/Doc/Zsh/options.yo b/Doc/Zsh/options.yo
index cc6ae2a..edf43d4 100644
--- a/Doc/Zsh/options.yo
+++ b/Doc/Zsh/options.yo
@@ -2140,6 +2140,10 @@ In addition, various error conditions associated with the above builtins
 or tt(exec) cause a non-interactive shell to exit and an interactive
 shell to return to its top-level processing.
 
+Furthermore, functions ahd shell builtins are not executed after
+an tt(exec) prefix; the command to be executed must be an external
+command found in the path.
+
 Furthermore, the tt(getopts) builtin behaves in a POSIX-compatible
 fashion in that the associated variable tt(OPTIND) is not made
 local to functions.
diff --git a/Src/exec.c b/Src/exec.c
index 0a96879..f339dd6 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -2778,6 +2778,12 @@ execcmd_exec(Estate state, Execcmd_params eparams,
 		 * Reserved words take precedence over shell functions.
 		 */
 		checked = 1;
+	    } else if (isset(POSIXBUILTINS) && (cflags & BINF_EXEC)) {
+		/*
+		 * POSIX doesn't allow "exec" to operate on builtins
+		 * or shell functions.
+		 */
+		break;
 	    } else {
 		if (!(cflags & (BINF_BUILTIN | BINF_COMMAND)) &&
 		    (hn = shfunctab->getnode(shfunctab, cmdarg))) {
@@ -3123,10 +3129,14 @@ execcmd_exec(Estate state, Execcmd_params eparams,
 	     *   - we have determined there are options which would
 	     *     require us to use the "command" builtin); or
 	     * - we aren't using POSIX and so BINF_COMMAND indicates a zsh
-	     *   precommand modifier is being used in place of the builtin
+	     *   precommand modifier is being used in place of the
+	     *   builtin
+	     * - we are using POSIX and this is an EXEC, so we can't
+	     *   execute a builtin or function.
 	     */
 	    if (errflag || checked || is_builtin ||
-		(unset(POSIXBUILTINS) && (cflags & BINF_COMMAND)))
+		(isset(POSIXBUILTINS) ?
+		 (cflags & BINF_EXEC) : (cflags & BINF_COMMAND)))
 		break;
 
 	    cmdarg = (char *) peekfirst(args);
diff --git a/Test/E01options.ztst b/Test/E01options.ztst
index dac9430..f01d835 100644
--- a/Test/E01options.ztst
+++ b/Test/E01options.ztst
@@ -831,6 +831,20 @@
 >val2
 >val2
 
+  print "Contents of file" >cat_arg
+  (
+     cat() { print Function with argument $1 }
+     print Without
+     (exec cat cat_arg; print Not reached)
+     print With
+     (setopt posixbuiltins; exec cat cat_arg; print Not reached)
+  )
+0:POSIX_BUILTINS and exec
+>Without
+>Function with argument cat_arg
+>With
+>Contents of file
+
 # PRINTEXITVALUE only works if shell input is coming from standard input.
 # Goodness only knows why.
   $ZTST_testdir/../Src/zsh -f <<<'

Re: [BUG] 'exec' runs shell functions and builtins

[Peter Stephenson]

On Wed, 26 Jul 2017 10:23:53 +0100
Peter Stephenson <p.stephenson@samsung.com> wrote:
> +Furthermore, functions ahd shell builtins are not executed after
                          ^^^

fixed already, don't write in...

pws

Re: [BUG] 'exec' runs shell functions and builtins

[Peter Stephenson]

This clarifies the normal behaviour.

diff --git a/Doc/Zsh/builtins.yo b/Doc/Zsh/builtins.yo
index 333db1c..cbaf67a 100644
--- a/Doc/Zsh/builtins.yo
+++ b/Doc/Zsh/builtins.yo
@@ -675,6 +675,10 @@ executed.
 See ifzman(the section `Precommand Modifiers' in zmanref(zshmisc))\
 ifnzman(noderef(Precommand Modifiers)).
 
+If the option tt(POSIX_BUILTINS) is not set, var(command) may be
+interpreted as a shell builtin command or shell function: this is run as
+normal (as if tt(exec) were not present), then the shell exits.
+
 cindex(redirection, current shell's I/O)
 If var(command) is omitted but any redirections are specified,
 then the redirections will take effect in the current shell.

pws

Re: [BUG] 'exec' runs shell functions and builtins

[Bart Schaefer]

AFAIK zsh has always had this behavior.  It also allows for example
"exec builtin echo" as well as "builtin exec echo" etc.  What would
POSIX say that
    exec command echo foo
should do?  It's relatively easy to stop "exec" from finding functions
and builtin commands, but I think fairly painful to get it to ignore
other "precommand modifier" tokens.  Even after PWS's 41464 the above
example runs /bin/echo whereas I think the POSIX expectation would be
to report "command: command not found".

> This behaviour also appears to be contrary to the documentation in
> zshbuiltins(1) ("Replace the current shell with an external command
> rather than forking").

The documentation under "precommand modifiers" (which is
cross-referenced from zshbuiltins) says:

     The following command together with any arguments is run in place
     of the current process, rather than as a sub-process.  The shell
     does not fork and is replaced.

I think this is a case of the zshbuiltins doc being ancient and
imprecise rather than intentionally indicative of anything, but:

Re: [BUG] 'exec' runs shell functions and builtins

[Bart Schaefer]

On Wed, Jul 26, 2017 at 10:46 AM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> imprecise rather than intentionally indicative of anything, but:

Deleted a partial implementation of the patch from 41464 but forgot to
remove the last word from that phase.

Re: [BUG] 'exec' runs shell functions and builtins

[Peter Stephenson]

On Wed, 26 Jul 2017 10:46:06 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> AFAIK zsh has always had this behavior.  It also allows for example
> "exec builtin echo" as well as "builtin exec echo" etc.  What would
> POSIX say that
>     exec command echo foo
> should do?  It's relatively easy to stop "exec" from finding functions
> and builtin commands, but I think fairly painful to get it to ignore
> other "precommand modifier" tokens.  Even after PWS's 41464 the above
> example runs /bin/echo whereas I think the POSIX expectation would be
> to report "command: command not found".

This isn't what I get:

% (setopt posixbuiltins; exec command echo foo)
zsh: command not found: command
% cat ~/bin/builtin
#!/bin/sh
echo External command named $0
% (setopt posixbuiltins; exec builtin echo)
External command named /export/home/pws/bin/builtin

There's no further processing of the arguments when we find an exec
with posixbuiltins set, so the above is what I'd expect.

pws

diff --git a/Doc/Zsh/builtins.yo b/Doc/Zsh/builtins.yo
index 99b1dd4..0e8580d 100644
--- a/Doc/Zsh/builtins.yo
+++ b/Doc/Zsh/builtins.yo
@@ -679,7 +679,10 @@ See ifzman(the section `Precommand Modifiers' in zmanref(zshmisc))\
 ifnzman(noderef(Precommand Modifiers)).
 
 If the option tt(POSIX_BUILTINS) is set, var(command) is never
-interpreted as a shell builtin command or shell function.
+interpreted as a shell builtin command or shell function.  This
+means further precommand modifiers such as tt(builtin) and
+tt(noglob) are also not interpreted within the shell.  Hence
+var(command) is always found by searching the command path.
 
 cindex(redirection, current shell's I/O)
 If var(command) is omitted but any redirections are specified,

Re: [BUG] 'exec' runs shell functions and builtins

[Bart Schaefer]

On Thu, Jul 27, 2017 at 2:02 AM, Peter Stephenson
<p.stephenson@samsung.com> wrote:
>
> % (setopt posixbuiltins; exec command echo foo)
> zsh: command not found: command

Chet Ramey emailed me:
=====
"However, all of the standard utilities, including the regular built-ins in
the table, but not the special built-ins described in Special Built-In
Utilities, shall be implemented in a manner so that they can be accessed
via the exec family of functions as defined in the System Interfaces volume
of POSIX.1-2008 and can be invoked directly by those standard utilities
that require it (env, find, nice, nohup, time, xargs)."

`command' is explicitly listed in the table in

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html#tag_17_06
=====

So "exec command ..." is actually supposed to work, as is "exec read",
"exec getopts", etc.  Urk.

Re: [BUG] 'exec' runs shell functions and builtins

[Stephane Chazelas]

2017-07-26 13:53:38 +0100, Peter Stephenson:
> This clarifies the normal behaviour.
> 
> diff --git a/Doc/Zsh/builtins.yo b/Doc/Zsh/builtins.yo
> index 333db1c..cbaf67a 100644
> --- a/Doc/Zsh/builtins.yo
> +++ b/Doc/Zsh/builtins.yo
> @@ -675,6 +675,10 @@ executed.
>  See ifzman(the section `Precommand Modifiers' in zmanref(zshmisc))\
>  ifnzman(noderef(Precommand Modifiers)).
>  
> +If the option tt(POSIX_BUILTINS) is not set, var(command) may be
> +interpreted as a shell builtin command or shell function: this is run as
> +normal (as if tt(exec) were not present), then the shell exits.
> +
[...]

(you jumped the gun a bit there. The issue is  only being
discussed on the austin-group mailing list. The standard is
currently unclear on the matter, it might very well change to
allow the zsh/pdksh behaviour. Then again, though the zsh
behaviour makes more sense to me, in sh emulation, what matters
more is compatibility and it seems more of the widely used sh
implementations got with the ksh88 way).

It might be worth adding that after

exec a_builtin

or

exec a_function

the EXIT, DEBUG or ERR traps will not be executed, as the text
above would suggest they may.

Other traps and asynchronous job notifications are still handled
though (as opposed to when one execs an external command).

Not sure it's worth saying what would (not) happen in

! exec something

or

time exec something

-- 
Stephane

Re: [BUG] 'exec' runs shell functions and builtins

[Kamil Dudka]

On Friday, July 28, 2017 10:58:59 Bart Schaefer wrote:
> On Thu, Jul 27, 2017 at 2:02 AM, Peter Stephenson
> 
> <p.stephenson@samsung.com> wrote:
> > % (setopt posixbuiltins; exec command echo foo)
> > zsh: command not found: command
> 
> Chet Ramey emailed me:
> =====
> "However, all of the standard utilities, including the regular built-ins in
> the table, but not the special built-ins described in Special Built-In
> Utilities, shall be implemented in a manner so that they can be accessed
> via the exec family of functions as defined in the System Interfaces volume
> of POSIX.1-2008 and can be invoked directly by those standard utilities
> that require it (env, find, nice, nohup, time, xargs)."
> 
> `command' is explicitly listed in the table in
> 
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap01.html#tag
> _17_06 =====
> 
> So "exec command ..." is actually supposed to work, as is "exec read",
> "exec getopts", etc.  Urk.

Today I discovered that bash in Fedora installs /usr/bin/cd and the like
as external commands to make 'exec' work on those built-ins (primarily
in bash):

https://bugzilla.redhat.com/820192

I am really not happy about this solution because the external commands
only confuse people for no good reason (other than satisfying the POSIX 
requirement).

It is good that zsh does not need this in its native configuration but
how is it supposed to work in the POSIX emulation mode now?

% (exec cd)
% emulate sh
% (exec cd) 
zsh: command not found: cd

If I understand http://www.zsh.org/mla/workers/2017/msg01138.html correctly,
then the above command should work?

Kamil

Re: [BUG] 'exec' runs shell functions and builtins

[Peter Stephenson]

On Fri, 11 Aug 2017 17:10:12 +0200
Kamil Dudka <kdudka@redhat.com> wrote:
> It is good that zsh does not need this in its native configuration but
> how is it supposed to work in the POSIX emulation mode now?
> 
> % (exec cd)
> % emulate sh
> % (exec cd) 
> zsh: command not found: cd
> 
> If I understand http://www.zsh.org/mla/workers/2017/msg01138.html correctly,
> then the above command should work?

I can't actually think of a useful application of the above that would
make "work" meaningful.  What's the "exec" supposed to be doing?  Who
would even try to exec a builtin, even if they discover that for some
bizarre reason the standard says you can?  The one possible use of
"exec" in that case might be to ensure you're *not* using a builtin ---
but apparently you are?  What?

pws

Re: [BUG] 'exec' runs shell functions and builtins

[Kamil Dudka]

On Friday, August 11, 2017 16:25:01 Peter Stephenson wrote:
> On Fri, 11 Aug 2017 17:10:12 +0200
> 
> Kamil Dudka <kdudka@redhat.com> wrote:
> > It is good that zsh does not need this in its native configuration but
> > how is it supposed to work in the POSIX emulation mode now?
> > 
> > % (exec cd)
> > % emulate sh
> > % (exec cd)
> > zsh: command not found: cd
> > 
> > If I understand http://www.zsh.org/mla/workers/2017/msg01138.html
> > correctly, then the above command should work?
> 
> I can't actually think of a useful application of the above that would
> make "work" meaningful.  What's the "exec" supposed to be doing?  Who
> would even try to exec a builtin, even if they discover that for some
> bizarre reason the standard says you can?  The one possible use of
> "exec" in that case might be to ensure you're *not* using a builtin ---
> but apparently you are?  What?
> 
> pws

Thanks for the reply!  It did not make much sense to me either but I am by
no means expert in this area.  In fact I hoped to hear some explanation on 
this list because the topic had been raised here recently.  Given the lack
of information about the reasoning behind the strange POSIX requirement, I 
agree to keep the behavior of zsh as it is now.

Kamil