[PATCH v3 0/1] Run pipeline command in subshell in sh mode

[PATCH v3 0/1] Run pipeline command in subshell in sh mode

[brian m. carlson]

Most sh implementations[0] run each command in a pipeline in a subshell,
although zsh (and AT&T ksh) do not: instead, they run the final command
in the main shell.  This leads to very different behavior when the final
command is a shell function which modifies variables.

zsh is starting to be used in some cases as /bin/sh, such as on macOS
Catalina.  Consequently, it makes sense to emulate the more common sh
behavior as much as possible when emulating sh, since that's the least
surprising behavior, and the behavior that many programs unwittingly
rely on.  This patch does exactly that.

With this patch, using "zsh --emulate sh" passes the Git testsuite.  I
expect that it will also be fully functional as /bin/sh on Debian,
although I have not tested.

There was some discussion about this change and it was agreed that it
was a good patch, but it needed a note in the incompatibilities section,
so I've added one.  I'm happy to hear about additional changes people
would like to see; I hope that I'll get around to them a little faster
this time.

v2 can be seen in the archives at
https://www.zsh.org/mla/workers/2020/msg00794.html.

I'm not subscribed to the list, so please CC me if you have questions or
comments.

Changes from v2:
* Update commit message to remove text about which there was
  disagreement (what POSIX actually says about this issue).
* Add note in incompatibilities section.

Changes from v1:
* Expanded commit message to make the code easier to reason about.

[0] bash, dash, posh, pdksh, mksh, busybox sh, FreeBSD's /bin/sh (ash),
    and OpenBSD's /bin/sh and ksh implementations.  I believe, but am
    not certain, that the original Bourne sh provided this behavior as
    well.

brian m. carlson (1):
  exec: run final pipeline command in a subshell in sh mode

 README               |  4 ++++
 Src/exec.c           | 10 ++++++----
 Test/B07emulate.ztst | 22 ++++++++++++++++++++++
 3 files changed, 32 insertions(+), 4 deletions(-)

[PATCH v3 1/1] exec: run final pipeline command in a subshell in sh mode

[brian m. carlson]

zsh typically runs the final command in a pipeline in the main shell
instead of a subshell.  However, POSIX specifies that all commands in a
pipeline run in a subshell, but permits zsh's behavior as an extension.
The default /bin/sh implementations on various Linux distros and the
BSDs always use a subshell for all components of a pipeline.

Since zsh may be used as /bin/sh in some cases (such as macOS Catalina),
it makes sense to have the common sh behavior when emulating sh, so do
that by checking for being the final item of a multi-item pipeline and
creating a subshell in that case.

From the comment above execpline(), we know the following:

  last1 is a flag that this command is the last command in a shell that
  is about to exit, so we can exec instead of forking.  It gets passed
  all the way down to execcmd() which actually makes the decision.  A 0
  is always passed if the command is not the last in the pipeline. […]
  If last1 is zero but the command is at the end of a pipeline, we pass
  2 down to execcmd().

So there are three cases to consider in this code:

• last1 is 0, which means we are not at the end of a pipeline, in which
  case we should not change behavior.
• last1 is 1, which means we are effectively running in a subshell,
  because nothing that happens due to the exec is going to affect the
  actual shell, since it will have been replaced.  So there is nothing
  to do here.
• last1 is 2, which means our command is at the end of the pipeline, so
  in sh mode we should create a subshell by forking.

input is nonzero if the input to this process is a pipe that we've
opened.  At the end of a multi-stage pipeline, it will necessarily be
nonzero.

Note that several of the tests may appear bizarre, since most developers
do not place useless variable assignments directly at the end of a
pipeline.  However, as the function tests demonstrate, there are cases
where assignments may occur when a shell function is used at the end of
a command.  The remaining assignment tests simply test additional cases,
such as the use of local, that would otherwise be untested.
---
 README               |  4 ++++
 Src/exec.c           | 10 ++++++----
 Test/B07emulate.ztst | 22 ++++++++++++++++++++++
 3 files changed, 32 insertions(+), 4 deletions(-)

diff --git a/README b/README
index 9b1b1605f..3877594ae 100644
--- a/README
+++ b/README
@@ -92,6 +92,10 @@ not set the new, fourth field will continue to work under both 5.8 and 5.9.
 (As it happens, adding a comma after "bold" will make both 5.8 and 5.9 do the
 right thing, but this should be viewed as an unsupported hack.)
 
+emulate sh: When zsh emulates sh, the final command in a pipeline is now run in
+a subshell.  This differs from the behavior in the native (zsh) mode, but is
+consistent with most other sh implementations.
+
 Incompatibilities between 5.7.1 and 5.8
 ---------------------------------------
 
diff --git a/Src/exec.c b/Src/exec.c
index ecad923de..0f6f48b23 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -2882,11 +2882,13 @@ execcmd_exec(Estate state, Execcmd_params eparams,
 	    pushnode(args, dupstring("fg"));
     }
 
-    if ((how & Z_ASYNC) || output) {
+    if ((how & Z_ASYNC) || output ||
+	(last1 == 2 && input && EMULATION(EMULATE_SH))) {
 	/*
-	 * If running in the background, or not the last command in a
-	 * pipeline, we don't need any of the rest of this function to
-	 * affect the state in the main shell, so fork immediately.
+	 * If running in the background, not the last command in a
+	 * pipeline, or the last command in a multi-stage pipeline
+	 * in sh mode, we don't need any of the rest of this function
+	 * to affect the state in the main shell, so fork immediately.
 	 *
 	 * In other cases we may need to process the command line
 	 * a bit further before we make the decision.
diff --git a/Test/B07emulate.ztst b/Test/B07emulate.ztst
index 7b1592fa9..45c39b51d 100644
--- a/Test/B07emulate.ztst
+++ b/Test/B07emulate.ztst
@@ -276,3 +276,25 @@ F:Some reserved tokens are handled in alias expansion
 0:--emulate followed by other options
 >yes
 >no
+
+  emulate sh -c '
+  foo () {
+    VAR=foo &&
+    echo $VAR | bar &&
+    echo "$VAR"
+  }
+  bar () {
+    tr f b &&
+    VAR="$(echo bar | tr r z)" &&
+    echo "$VAR"
+  }
+  foo
+  '
+  emulate sh -c 'func() { echo | local def="abc"; echo $def;}; func'
+  emulate sh -c 'abc="def"; echo | abc="ghi"; echo $abc'
+0:emulate sh uses subshell for last pipe entry
+>boo
+>baz
+>foo
+>
+>def

Re: [PATCH v3 1/1] exec: run final pipeline command in a subshell in sh mode

[Lawrence Velázquez]

On Sun, Jan 3, 2021, at 7:22 PM, brian m. carlson wrote:
> zsh typically runs the final command in a pipeline in the main shell
> instead of a subshell.  However, POSIX specifies that all commands in a
> pipeline run in a subshell, but permits zsh's behavior as an extension.
> The default /bin/sh implementations on various Linux distros and the
> BSDs always use a subshell for all components of a pipeline.
> 
> Since zsh may be used as /bin/sh in some cases (such as macOS Catalina),
> it makes sense to have the common sh behavior when emulating sh, so do
> that by checking for being the final item of a multi-item pipeline and
> creating a subshell in that case.

ping for review

vq

Re: [PATCH v3 1/1] exec: run final pipeline command in a subshell in sh mode

[Lawrence Velázquez]

On Sat, Mar 27, 2021, at 3:34 PM, Lawrence Velázquez wrote:
> On Sun, Jan 3, 2021, at 7:22 PM, brian m. carlson wrote:
> > zsh typically runs the final command in a pipeline in the main shell
> > instead of a subshell.  However, POSIX specifies that all commands in a
> > pipeline run in a subshell, but permits zsh's behavior as an extension.
> > The default /bin/sh implementations on various Linux distros and the
> > BSDs always use a subshell for all components of a pipeline.
> > 
> > Since zsh may be used as /bin/sh in some cases (such as macOS Catalina),
> > it makes sense to have the common sh behavior when emulating sh, so do
> > that by checking for being the final item of a multi-item pipeline and
> > creating a subshell in that case.
> 
> ping for review

Ping: 2.0 You Can (Not) Review

vq

Re: [PATCH v3 1/1] exec: run final pipeline command in a subshell in sh mode

[Lawrence Velázquez]

On Sat, Apr 3, 2021, at 11:49 AM, Lawrence Velázquez wrote:
> On Sat, Mar 27, 2021, at 3:34 PM, Lawrence Velázquez wrote:
> > On Sun, Jan 3, 2021, at 7:22 PM, brian m. carlson wrote:
> > > zsh typically runs the final command in a pipeline in the main shell
> > > instead of a subshell.  However, POSIX specifies that all commands in a
> > > pipeline run in a subshell, but permits zsh's behavior as an extension.
> > > The default /bin/sh implementations on various Linux distros and the
> > > BSDs always use a subshell for all components of a pipeline.
> > > 
> > > Since zsh may be used as /bin/sh in some cases (such as macOS Catalina),
> > > it makes sense to have the common sh behavior when emulating sh, so do
> > > that by checking for being the final item of a multi-item pipeline and
> > > creating a subshell in that case.
> > 
> > ping for review
> 
> Ping: 2.0 You Can (Not) Review

The Ping for Review Part III

vq

Re: [PATCH v3 1/1] exec: run final pipeline command in a subshell in sh mode

[dana]

On 3 Jan 2021, at 18:22, brian m. carlson <sandals@crustytoothpaste.net> wrote:
> Since zsh may be used as /bin/sh in some cases (such as macOS Catalina),
> it makes sense to have the common sh behavior when emulating sh

This seems inoffensive to me. Since nobody objected to the code itself in the
previous discussions i've committed it.

I did want to mention that macOS does *not* use zsh as /bin/sh. That continues
to be provided by bash 3.2; only the default interactive shell was changed to
zsh. I don't think i know of any system that uses zsh for /bin/sh

dana

Re: [PATCH v3 1/1] exec: run final pipeline command in a subshell in sh mode

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

On 2021-04-10 at 23:02:39, dana wrote:
> On 3 Jan 2021, at 18:22, brian m. carlson <sandals@crustytoothpaste.net> wrote:
> > Since zsh may be used as /bin/sh in some cases (such as macOS Catalina),
> > it makes sense to have the common sh behavior when emulating sh
> 
> This seems inoffensive to me. Since nobody objected to the code itself in the
> previous discussions i've committed it.

Great, thanks.

> I did want to mention that macOS does *not* use zsh as /bin/sh. That continues
> to be provided by bash 3.2; only the default interactive shell was changed to
> zsh. I don't think i know of any system that uses zsh for /bin/sh

That's definitely true by default, but I believe users can also choose
to set dash or zsh as /bin/sh by changing /var/select/sh, as of
Catalina.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]