'emulate sh -c' and $0

'emulate sh -c' and $0

[Richard Hansen]

Hi all,

I just encountered what I think is a bug in Zsh 5.0.5.  The following
command:

    zsh -c '
        emulate sh -c "echo \"\$0\""
        bar() { emulate sh -c "echo \"\$0\""; }
        bar
    ' foo arg1

produces the following output:

    foo
    bar

I expected it to produce:

    foo
    foo

This is relevant when sourcing a file containing (POSIX) sh code that
might examine $0 (e.g., for logging or to 'exec "$0" "$@"' after
exporting/unsetting environment variables).

Perhaps Zsh should save the original value of $0 somewhere and restore
it when entering sh emulation mode.

Thanks,
Richard

Re: 'emulate sh -c' and $0

[Bart Schaefer]

On May 29,  7:04pm, Richard Hansen wrote:
}
} I just encountered what I think is a bug in Zsh 5.0.5.

To the extent that it's working exactly as documented, it's not a bug ...

}     zsh -c '
}         emulate sh -c "echo \"\$0\""
}         bar() { emulate sh -c "echo \"\$0\""; }
}         bar
}     ' foo arg1
} 
} I expected it to produce:
} 
}     foo
}     foo

If you throw in "unsetopt functionargzero" any time before calling "bar" 
then it does produce that output.

If you rewrite your example as

    zsh -c '
        emulate sh -c "echo \"\$0\""
        emulate sh -c "bar() { echo \"\$0\"; }"
        bar
    ' foo arg1

then it also produces your expected output; you just need to define the
function in the right scope.

} This is relevant when sourcing a file containing (POSIX) sh code that
} might examine $0 (e.g., for logging or to 'exec "$0" "$@"' after
} exporting/unsetting environment variables).
} 
} Perhaps Zsh should save the original value of $0 somewhere and restore
} it when entering sh emulation mode.

I don't find those examples particularly compelling, but the original
value of $0 is already stashed; what would need to change is that the
*local* value of $0 gets temporarily replaced by the global one.  (The
[un]setting of functionargzero controls whether a local value is ever
created in the first place.)

Re: 'emulate sh -c' and $0

[Richard Hansen]

(please cc me in replies as I am not subscribed to the list)

On 2014-05-29 23:45, Bart Schaefer wrote:
> On May 29,  7:04pm, Richard Hansen wrote:
>>
>> I just encountered what I think is a bug in Zsh 5.0.5.
> 
> To the extent that it's working exactly as documented, it's not a bug ...

Would you mind pointing me to where this specific behavior is
documented?  It was non-obvious to me when I was digging around.

(I am aware of the documentation for the FUNCTION_ARGZERO option.  I'm
more interested in what it really means to be running in sh emulation
mode, as that's where I think the bug is.)

>>     zsh -c '
>>         emulate sh -c "echo \"\$0\""
>>         bar() { emulate sh -c "echo \"\$0\""; }
>>         bar
>>     ' foo arg1
>> 
>> I expected it to produce:
>> 
>>     foo
>>     foo
> 
> If you throw in "unsetopt functionargzero" any time before calling "bar" 
> then it does produce that output.
> 
> If you rewrite your example as
> 
>     zsh -c '
>         emulate sh -c "echo \"\$0\""
>         emulate sh -c "bar() { echo \"\$0\"; }"
>         bar
>     ' foo arg1
> 
> then it also produces your expected output; you just need to define the
> function in the right scope.

It is not always possible to unset that option in a meaningful way --
the code that sources the file with POSIX shell code may itself be in a
file that has been sourced.  By that time, $0 has already been
overridden.  Moving the unsetopt up another level may not be
desirable/feasible.

For example:

cat <<\EOF >foo.zsh
# zsh-specific code goes here
unsetopt FUNCTION_ARGZERO
. ./bar.sh
# zsh-specific code goes here
EOF
cat <<\EOF >bar.sh
printf %s\\n "$0"
EOF
zsh -c '. ./foo.zsh' baz

>> This is relevant when sourcing a file containing (POSIX) sh code that
>> might examine $0 (e.g., for logging or to 'exec "$0" "$@"' after
>> exporting/unsetting environment variables).
>> 
>> Perhaps Zsh should save the original value of $0 somewhere and restore
>> it when entering sh emulation mode.
> 
> I don't find those examples particularly compelling,

Here's the real-world problem that motivated my bug report; perhaps it
is a more compelling example (or perhaps you'll think of a better way to
solve the problem I was addressing):

http://article.gmane.org/gmane.comp.version-control.git/250409

> but the original
> value of $0 is already stashed; what would need to change is that the
> *local* value of $0 gets temporarily replaced by the global one.

That's good news; that should make it easier to write a patch that
temporarily replaces the local value with the global value.

Would you (or anyone else in the community) be opposed to such a patch?
 If not, can you point me to the relevant bits of code to help me get
started?

Thanks,
Richard

Re: 'emulate sh -c' and $0

[Bart Schaefer]

On May 30,  4:49am, Richard Hansen wrote:
}
} On 2014-05-29 23:45, Bart Schaefer wrote:
} > On May 29,  7:04pm, Richard Hansen wrote:
} >>
} >> I just encountered what I think is a bug in Zsh 5.0.5.
} > 
} > To the extent that it's working exactly as documented, it's not a bug ...
} 
} Would you mind pointing me to where this specific behavior is
} documented?  It was non-obvious to me when I was digging around.

I don't pretend there couldn't be more clarity in the zsh documentation,
but under "Parameters" there's first this:

 In the parameter lists that follow, the mark `<S>' indicates that the
 parameter is special.  Special parameters cannot have their type
 changed or their readonly attribute turned off, and if a special
 parameter is unset, then later recreated, the special properties will
 be retained.  `<Z>' indicates that the parameter does not exist when
 the shell initializes in sh or ksh emulation mode.

And then in the "Parameters Set by the Shell" subsection there is:

 0 <S>
     The name used to invoke the current shell.  If the
     FUNCTION_ARGZERO option is set, this is set temporarily within a
     shell function to the name of the function, and within a sourced
     script to the name of the script.

There's no <Z> there, and there's no mention in the "Compatibility"
section about the behavior of $0, only that NO_FUNCTION_ARGZERO is set.
Finally the entry for the emulate builtin says:

     With single argument set up zsh options to emulate the specified
     shell as much as possible.  `csh' will never be fully emulated.
     If the argument is not one of the shells listed above, zsh will be
     used as a default; more precisely, the tests performed on the
     argument are the same as those used to determine the emulation at
     startup based on the shell name, see Compatibility.

} (I am aware of the documentation for the FUNCTION_ARGZERO option.  I'm
} more interested in what it really means to be running in sh emulation
} mode, as that's where I think the bug is.)

In general, emulation is at its most complete if and only if the shell
is actually started as an emulator (e.g., the path name to the shell
binary itself is not zsh, or ARGV0 is set in the environment).  The
"emulate" builtin only changes setopts to the closest possible.

} > I don't find those examples particularly compelling,
} 
} Here's the real-world problem that motivated my bug report; perhaps it
} is a more compelling example (or perhaps you'll think of a better way to
} solve the problem I was addressing):
} 
} http://article.gmane.org/gmane.comp.version-control.git/250409

Instead of "compelling" I perhaps should have said "likely to come up
in common usage."  You have a fairly rare special case there.  In that
example,

    ARGV0=sh exec zsh "$0" "$ <at> "

might do what you want, but I'm not entirely following from the diff
context what's intended.
 
} > but the original
} > value of $0 is already stashed; what would need to change is that the
} > *local* value of $0 gets temporarily replaced by the global one.
} 
} That's good news; that should make it easier to write a patch that
} temporarily replaces the local value with the global value.

Unfortunately the way the local value is implemented is usually to use
C local variable scoping to stash and restore the contents of a C global
pointer, so this would mean at least one additional C global.

} Would you (or anyone else in the community) be opposed to such a patch?

The use cases in both directions seem pretty unusual to me.  Losing the
ability to "localize" $0 for scripts feels almost as likely to create
questions as does your situation.  I suppose if both values were in the
C global state, it would be possible to have the "correct" one appear
at the instant functionargzero changes, instead of being determined by
the setting at the time the function is entered.  OTOH that would be a
larger behavior difference / lack of backward compatibilty.

} If not, can you point me to the relevant bits of code to help me get
} started?

Search Src/*.c for references to "argzero", with particular attention to
builtin.c:bin_emulate.

Re: 'emulate sh -c' and $0

[Richard Hansen]

On 2014-05-30 13:00, Bart Schaefer wrote:
> Finally the entry for the emulate builtin says:
> 
>      With single argument set up zsh options to emulate the specified
>      shell as much as possible.  `csh' will never be fully emulated.
>      If the argument is not one of the shells listed above, zsh will be
>      used as a default; more precisely, the tests performed on the
>      argument are the same as those used to determine the emulation at
>      startup based on the shell name, see Compatibility.

Thanks for the documentation references.  I had read "to emulate the
specified shell as much as possible" without paying enough attention to
the qualifying "set up zsh options".  So you are right:  The
documentation says that emulate only toggles options, and the behavior
of $0 with FUNCTION_ARGZERO is clear, so there's no reason to expect Zsh
to reset $0 to the original $0 when in sh emulation mode.

That being said, I still think there's value in changing Zsh's behavior.

> 
>> (I am aware of the documentation for the FUNCTION_ARGZERO option.  I'm
>> more interested in what it really means to be running in sh emulation
>> mode, as that's where I think the bug is.)
> 
> In general, emulation is at its most complete if and only if the shell
> is actually started as an emulator (e.g., the path name to the shell
> binary itself is not zsh, or ARGV0 is set in the environment).  The
> "emulate" builtin only changes setopts to the closest possible.

Would it add too much complexity to the code or documentation if the
emulate builtin did more than just toggle options (specifically:
temporarily change the binding of $0 to the original value)?

Perhaps the behavior of FUNCTION_ARGZERO could be altered so that $0
expands as follows:

    If option FUNCTION_ARGZERO is enabled and $0 is expanded inside the
    body of a function, $0 expands to the name of the enclosing
    function.

    Otherwise, if option FUNCTION_ARGZERO is enabled and $0 is expanded
    inside a sourced file, $0 expands to the pathname given to the
    'source' or '.' builtin command.

    Otherwise, if the shell was invoked with an argument naming a
    script containing shell commands to be executed, $0 expands to the
    value of that argument.

    Otherwise, if the shell was invoked with the '-c' flag and at least
    one non-option non-flag argument was given, $0 expands to the value
    of the first non-option non-flag argument.

    Otherwise, $0 expands to the value of the first argument passed to
    zsh from its parent (argv[0] in C).

This modification would make it possible to toggle the setting back and
forth to examine the local or original value as desired, even within the
same function.  I wouldn't expect this change to break many scripts, but
maybe any backward incompatibility is unacceptable.

>>> I don't find those examples particularly compelling,
>> 
>> Here's the real-world problem that motivated my bug report; perhaps it
>> is a more compelling example (or perhaps you'll think of a better way to
>> solve the problem I was addressing):
>> 
>> http://article.gmane.org/gmane.comp.version-control.git/250409
> 
> Instead of "compelling" I perhaps should have said "likely to come up
> in common usage."  You have a fairly rare special case there.

Good point.  :)

> In that example,
> 
>     ARGV0=sh exec zsh "$0" "$@"
> 
> might do what you want, but I'm not entirely following from the diff
> context what's intended.

Some more context if you're curious:  The Git distribution comes with
t/test-lib.sh, a file containing POSIX shell code implementing common
test infrastructure (print error messages, declare and run test cases,
etc.).  The test scripts are POSIX shell scripts that source this shared
file, with two exceptions:

  * t/t9903-bash-prompt.sh starts off running under /bin/sh, but it
    does the following early on:

        exec bash "$0" "$@"

    so that it can run and test Bash-specific shell code.  After
    reinvoking itself under Bash, the code sources test-lib.sh in order
    to reuse the shared test infrastructure code.  (The code in
    test-lib.sh is interpreted as Bash code, not POSIX shell code, but
    that doesn't really matter because the code is compatible with both
    shells.)

  * t/t9904-zsh-prompt.sh (new in that linked patch series) is similar
    to t9903, except it restarts itself under Zsh instead of Bash.
    Like t9903, it sources test-lib.sh, but because the code in
    test-lib.sh is incompatible with Zsh, it uses Zsh's sh emulation to
    source test-lib.sh.

The point of these two test scripts is to run Bash and Zsh in their
native modes as much as possible -- emulation is explicitly avoided
except as necessary to run the shared test infrastructure.

So 'ARGV0=sh exec zsh "$0" "$@"' doesn't work for two reasons:

  * at the time that line is executed, the script is being interpreted
    by /bin/sh and not Zsh, so the ARGV0 assignment won't have the
    desired effect

  * we want as little as possible to run in sh emulation mode so that
    we can test Zsh-specific code

>  
>>> but the original
>>> value of $0 is already stashed; what would need to change is that the
>>> *local* value of $0 gets temporarily replaced by the global one.
>> 
>> That's good news; that should make it easier to write a patch that
>> temporarily replaces the local value with the global value.
> 
> Unfortunately the way the local value is implemented is usually to use
> C local variable scoping to stash and restore the contents of a C global
> pointer, so this would mean at least one additional C global.
> 
>> Would you (or anyone else in the community) be opposed to such a patch?
> 
> The use cases in both directions seem pretty unusual to me.  Losing the
> ability to "localize" $0 for scripts feels almost as likely to create
> questions as does your situation.

I'm not sure what you mean by losing the ability to localize $0.

I see a few OK options:

  * Option #1:
    1. Add a new global variable 'orig_argzero' to hold the original
       value of $0.  This variable is never modified once set.
    2. The existing global variable 'argzero' continues to serve its
       current role of holding the "localized" value of $0 (it is
       updated when executing functions or sourcing files if
       FUNCTION_ARGZERO is enabled).
    3. When 'emulate sh' starts, temporarily set argzero to
       orig_argzero.  Restore argzero when 'emulate sh' returns.

This would result in behavior that is identical to the current behavior
except $0 would match the POSIX spec when in sh emulation mode (and only
in sh emulation mode).

  * Option #2:
    1. Add a new global variable 'orig_argzero' to hold the original
       value of $0.  This variable is never modified once set.
    2. The existing global variable 'argzero' continues to serve its
       current role of holding the "localized" value of $0 (it is
       updated when executing functions or sourcing files if
       FUNCTION_ARGZERO is enabled).
    3. Add a new option; let's call it LOCALIZE_ARGZERO for now.  If
       LOCALIZE_ARGZERO is enabled, use argzero to expand $0.  If
       LOCALIZE_ARGZERO is disabled, use orig_argzero to expand $0.
    4. Enable LOCALIZE_ARGZERO by default, but disable it in sh
       emulation mode.
    5. Stop disabling FUNCTION_ARGZERO by default in sh emulation mode.

  * Option #3:
    1. Add a new global variable 'orig_argzero' to hold the original
       value of $0.  This variable is never modified once set.
    2. Whenever a function is called or a file sourced, update the
       global variable holding the "localized" $0 ('argzero'), even if
       FUNCTION_ARGZERO is disabled.
    3. Modify the expansion rules for $0 as follows:  If
       FUNCTION_ARGZERO is enabled, use argzero to expand $0.  If
       FUNCTION_ARGZERO is disabled, use orig_argzero to expand $0.

Pros and cons:

Option #1 is simplest to implement, simple for users, and (mostly)
backward compatible, but less powerful than options #2 and #3 and
'emulate' no longer just sets options.

Option #2 is complex but powerful (scripts can read both the original $0
and the localized $0 in the same chunk of code) and (mostly) backward
compatible.  Note that option #1 can be used as a stepping stone to
option #2.

Option #3 is simple for users but not backward compatible.

I think my preference is to go with option #1 with a possible future
step to option #2 (at which time FUNCTION_ARGZERO can be deprecated in
favor of LOCALIZE_ARGZERO).

> I suppose if both values were in the
> C global state, it would be possible to have the "correct" one appear
> at the instant functionargzero changes, instead of being determined by
> the setting at the time the function is entered.  OTOH that would be a
> larger behavior difference / lack of backward compatibilty.

Oops, I should have thoroughly read your email before proposing the same
thing but with more words.  :)

> 
>> If not, can you point me to the relevant bits of code to help me get
>> started?
> 
> Search Src/*.c for references to "argzero", with particular attention to
> builtin.c:bin_emulate.

Thanks.  No promises that I'll have the time to submit a patch soon (or
even at all), but I plan on taking a crack at it this weekend.

-Richard

Re: 'emulate sh -c' and $0

[Bart Schaefer]

On May 30,  5:14pm, Richard Hansen wrote:
}
} Would it add too much complexity to the code or documentation if the
} emulate builtin did more than just toggle options (specifically:
} temporarily change the binding of $0 to the original value)?

No, probably not.  We could certainly get away with making the change
to $0 be an additional effect of the (relatively new) -c option, i.e.,

    emulate sh

changes only the options, but 

    emulate sh -c 'some command'

changes both options and $0 in the scope of 'some command'.  In fact I
like that idea a lot, now that I've written it down ... but it'd be a
bit complicated to add that to "sticky emulation" so a simpler plan is
probably better.

} > The use cases in both directions seem pretty unusual to me.  Losing the
} > ability to "localize" $0 for scripts feels almost as likely to create
} > questions as does your situation.
} 
} I'm not sure what you mean by losing the ability to localize $0.

You gave two examples in your original message on this thread: the current
behavior and the behavior you expected.  Someone else might be expecting
the current behavior, and I don't see any clear criteria for deciding
which one is "best."  If we make any of the suggested changes here, the
current behavior is lost.

} I see a few OK options:
} 
}   * Option #1:
}     3. When 'emulate sh' starts, temporarily set argzero to
}        orig_argzero.  Restore argzero when 'emulate sh' returns.
} 
}   * Option #2:
}     3. Add a new option; let's call it LOCALIZE_ARGZERO for now.  If
}        LOCALIZE_ARGZERO is enabled, use argzero to expand $0.  If
}        LOCALIZE_ARGZERO is disabled, use orig_argzero to expand $0.
}     4. Enable LOCALIZE_ARGZERO by default, but disable it in sh
}        emulation mode.
}     5. Stop disabling FUNCTION_ARGZERO by default in sh emulation mode.
} 
}   * Option #3:
}     3. Modify the expansion rules for $0 as follows:  If
}        FUNCTION_ARGZERO is enabled, use argzero to expand $0.  If
}        FUNCTION_ARGZERO is disabled, use orig_argzero to expand $0.

I was initially leaning toward #3, because #1 takes away the current
behavior and because I'd rather not add yet another option as in #2 ...
but then I had a different idea ...

It seems to me that the ideal situation would be that

    emulate sh

works exactly as it does now ("With SINGLE ARGUMENT set up zsh options"
says the doc, emphasis mine) but that

    emulate sh -c 'some command'

alters the behavior of $0 as well.  The advantage of this is that the -c
option to emulate is relatively new; any scripts relying on the current
$0 behavior are likely old and won't use -c.  The drawback to this is
"sticky emulation" for functions, which is based entirely on option
settings; but that can be made to work if we add an option.  Hence:

If we leave FUNCTION_ARGZERO as it is (that is, off by default for sh
emulation) and add an option POSIX_ARGZERO which exposes the global
argzero when set (inverting your Option #2) but which is never on by
default, then bin_emulate can set POSIX_ARGZERO in the -c scope when
emulating sh/ksh, and it will be sticky for functions defined there.

We wouldn't even have to build those smarts into bin_emulate; a user
who wanted the POSIX semantics could explicitly do

    emulate sh -o POSIX_ARGZERO -c '...'

The only "magic" necessary is that POSIX_ARGZERO exposes the original
value of $0 in spite of the current FUNCTION_ARGZERO setting.

Here are the bits outside bin_emulate, and not yet with doc.  I suppose
there may be some places where posixzero needs to be saved / changed /
restored, which this hasn't covered.


diff --git a/Src/init.c b/Src/init.c
index fd12412..5e92f59 100644
--- a/Src/init.c
+++ b/Src/init.c
@@ -226,7 +226,7 @@ parseargs(char **argv, char **runscript)
     char **x;
     LinkList paramlist;
 
-    argzero = *argv++;
+    argzero = posixzero = *argv++;
     SHIN = 0;
 
     /* There's a bit of trickery with opts[INTERACTIVE] here.  It starts *
@@ -253,7 +253,7 @@ parseargs(char **argv, char **runscript)
     if (*argv) {
 	if (unset(SHINSTDIN)) {
 	    if (cmd)
-		argzero = *argv;
+		argzero = posixzero = *argv;
 	    else
 		*runscript = *argv;
 	    opts[INTERACTIVE] &= 1;
@@ -275,6 +275,7 @@ parseargs(char **argv, char **runscript)
     while ((*x++ = (char *)getlinknode(paramlist)));
     free(paramlist);
     argzero = ztrdup(argzero);
+    posixzero = ztrdup(posixzero);
 }
 
 /* Insert into list in order of pointer value */
diff --git a/Src/options.c b/Src/options.c
index ce73d99..e83dc58 100644
--- a/Src/options.c
+++ b/Src/options.c
@@ -207,6 +207,7 @@ static struct optname optns[] = {
 {{NULL, "pathscript",	      OPT_EMULATE|OPT_BOURNE},	 PATHSCRIPT},
 {{NULL, "pipefail",           OPT_EMULATE},              PIPEFAIL},
 {{NULL, "posixaliases",       OPT_EMULATE|OPT_BOURNE},	 POSIXALIASES},
+{{NULL, "posixargzero",       OPT_EMULATE},              POSIXARGZERO},
 {{NULL, "posixbuiltins",      OPT_EMULATE|OPT_BOURNE},	 POSIXBUILTINS},
 {{NULL, "posixcd",            OPT_EMULATE|OPT_BOURNE},	 POSIXCD},
 {{NULL, "posixidentifiers",   OPT_EMULATE|OPT_BOURNE},	 POSIXIDENTIFIERS},
diff --git a/Src/params.c b/Src/params.c
index 7901029..0699ead 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -67,6 +67,7 @@ char **path,		/* $path        */
 /**/
 mod_export
 char *argzero,		/* $0           */
+     *posixzero,	/* $0           */
      *home,		/* $HOME        */
      *nullcmd,		/* $NULLCMD     */
      *oldpwd,		/* $OLDPWD      */
@@ -194,6 +195,8 @@ static const struct gsu_integer euid_gsu =
 static const struct gsu_integer ttyidle_gsu =
 { ttyidlegetfn, nullintsetfn, stdunsetfn };
 
+static const struct gsu_scalar argzero_gsu =
+{ argzerogetfn, nullstrsetfn, nullunsetfn };
 static const struct gsu_scalar username_gsu =
 { usernamegetfn, usernamesetfn, stdunsetfn };
 static const struct gsu_scalar dash_gsu =
@@ -285,6 +288,7 @@ IPDEF2("WORDCHARS", wordchars_gsu, 0),
 IPDEF2("IFS", ifs_gsu, PM_DONTIMPORT),
 IPDEF2("_", underscore_gsu, PM_DONTIMPORT),
 IPDEF2("KEYBOARD_HACK", keyboard_hack_gsu, PM_DONTIMPORT),
+IPDEF2("0", argzero_gsu, 0),
 
 #ifdef USE_LOCALE
 # define LCIPDEF(name) IPDEF2(name, lc_blah_gsu, PM_UNSET)
@@ -340,7 +344,6 @@ IPDEF7U("RPROMPT2", &rprompt2),
 IPDEF7("PS3", &prompt3),
 IPDEF7("PS4", &prompt4),
 IPDEF7("SPROMPT", &sprompt),
-IPDEF7("0", &argzero),
 
 #define IPDEF8(A,B,C,D) {{NULL,A,D|PM_SCALAR|PM_SPECIAL},BR((void *)B),GSU(colonarr_gsu),0,0,NULL,C,NULL,0}
 IPDEF8("CDPATH", &cdpath, "cdpath", 0),
@@ -3981,6 +3984,17 @@ lcsetfn(Param pm, char *x)
 }
 #endif /* USE_LOCALE */
 
+/* Function to get value for special parameter `0' */
+
+/**/
+static char *
+argzerogetfn(UNUSED(Param pm))
+{
+    if (isset(POSIXARGZERO))
+	return posixzero;
+    return argzero;
+}
+
 /* Function to get value for special parameter `HISTSIZE' */
 
 /**/
diff --git a/Src/zsh.h b/Src/zsh.h
index 5fbff57..620883b 100644
--- a/Src/zsh.h
+++ b/Src/zsh.h
@@ -2151,6 +2151,7 @@ enum {
     PATHSCRIPT,
     PIPEFAIL,
     POSIXALIASES,
+    POSIXARGZERO,
     POSIXBUILTINS,
     POSIXCD,
     POSIXIDENTIFIERS,

Re: 'emulate sh -c' and $0

[Bart Schaefer]

On May 30, 10:13pm, Bart Schaefer wrote:
} Subject: Re: 'emulate sh -c' and $0
}
}     emulate sh
} 
} changes only the options, but 
} 
}     emulate sh -c 'some command'
} 
} changes both options and $0 in the scope of 'some command'.
} 
[...]
} 
} It seems to me that the ideal situation would be that
} 
}     emulate sh
} 
} works exactly as it does now ("With SINGLE ARGUMENT set up zsh options"
} says the doc, emphasis mine) but that
} 
}     emulate sh -c 'some command'
} 
} alters the behavior of $0 as well.

Wow, I thought I'd deleted that top part before I wrote the later part.
That's what I get for rushing off to a concert in the middle of composing
an email message, I guess.

Re: 'emulate sh -c' and $0

[Richard Hansen]

On 2014-05-31 01:13, Bart Schaefer wrote:
> If we leave FUNCTION_ARGZERO as it is (that is, off by default for sh
> emulation) and add an option POSIX_ARGZERO which exposes the global
> argzero when set (inverting your Option #2) but which is never on by
> default, then bin_emulate can set POSIX_ARGZERO in the -c scope when
> emulating sh/ksh, and it will be sticky for functions defined there.
> 
> We wouldn't even have to build those smarts into bin_emulate; a user
> who wanted the POSIX semantics could explicitly do
> 
>     emulate sh -o POSIX_ARGZERO -c '...'
> 
> The only "magic" necessary is that POSIX_ARGZERO exposes the original
> value of $0 in spite of the current FUNCTION_ARGZERO setting.
> 
> Here are the bits outside bin_emulate, and not yet with doc.  I suppose
> there may be some places where posixzero needs to be saved / changed /
> restored, which this hasn't covered.

Wow, thank you for committing this change to master!  It works well in
my limited testing, except for the documented limitation that
POSIX_ARGZERO stays enabled when calling a non-emulated function from a
function defined in 'emulate <shell> -c'.  I'm not sure how much this
will matter in practice, but if backward compatibility wasn't a concern
it'd be nice if zsh temporarily restored options when invoking a
function outside the 'emulate <shell> -c' boundary.

Although it would be a behavior change, I think it would be best if both
'emulate sh' and 'emulate sh -c' set POSIX_ARGZERO by default:  I
suspect that someone who runs 'emulate sh' cares more about accurate sh
emulation than compatibility with previous sh emulation behavior.  :)

Thanks,
Richard

Re: 'emulate sh -c' and $0

[Peter Stephenson]

On Tue, 03 Jun 2014 16:15:25 -0400
Richard Hansen <rhansen@bbn.com> wrote:
> Although it would be a behavior change, I think it would be best if both
> 'emulate sh' and 'emulate sh -c' set POSIX_ARGZERO by default:  I
> suspect that someone who runs 'emulate sh' cares more about accurate sh
> emulation than compatibility with previous sh emulation behavior.  :)

Yes, that's the policy --- backward compatibility is for native mode, sh
compatibility can be improved without worrying about that.

I'm not sure why we missed this one.  Most of the POSIX options are on
in sh emulations.

pws

Re: 'emulate sh -c' and $0

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]

On Jun 3, 2014 1:27 PM, "Peter Stephenson" <p.w.stephenson@ntlworld.com>
wrote:
>
> On Tue, 03 Jun 2014 16:15:25 -0400
> Richard Hansen <rhansen@bbn.com> wrote:
> > Although it would be a behavior change, I think it would be best if both
> > 'emulate sh' and 'emulate sh -c' set POSIX_ARGZERO by default
>
> Yes, that's the policy --- backward compatibility is for native mode, sh
> compatibility can be improved without worrying about that.

The complexity here is that we're not just dealing with a particular
emulation, we're dealing with switching from one emulation to another (and
possibly back again) in the middle of a running shell session, and the
effect that has on a dynamically scoped variable that crosses the emulation
boundaries.

If I start in zsh mode and change $0, or $PATH, or any other parameter, and
then enter  a different emulation, the values of those parameters don't
normally change.

> I'm not sure why we missed this one.  Most of the POSIX options are on
> in sh emulations.

We didn't really miss it -- FUNCTION_ARGZERO is correctly turned off when
in sh emulation.  The complication is that it affects $0 only upon entry to
the scope, so although $0 doesn't change *again*, it also doesn't revert if
it was changed previously.

Re: 'emulate sh -c' and $0

[Richard Hansen]

On 2014-06-03 17:10, Bart Schaefer wrote:
> On Jun 3, 2014 1:27 PM, "Peter Stephenson" <p.w.stephenson@ntlworld.com>
> wrote:
>> On Tue, 03 Jun 2014 16:15:25 -0400
>> Richard Hansen <rhansen@bbn.com> wrote:
>>> Although it would be a behavior change, I think it would be best if both
>>> 'emulate sh' and 'emulate sh -c' set POSIX_ARGZERO by default
>>
>> Yes, that's the policy --- backward compatibility is for native mode, sh
>> compatibility can be improved without worrying about that.
> 
> The complexity here is that we're not just dealing with a particular
> emulation, we're dealing with switching from one emulation to another (and
> possibly back again) in the middle of a running shell session, and the
> effect that has on a dynamically scoped variable that crosses the emulation
> boundaries.

Isn't this a general problem with how zsh supports mix-and-match shell
code?  Here's a contrived example:

    echo_stuff() { printf %s\\n "$*"; }
    foo() { ${CMD} words here; }
    CMD=echo_stuff
    IFS=_
    printf "outside sh emulation: "; foo
    printf "inside sh emulation:  "; emulate sh -c foo

As stated in detail #2 in the documentation for the emulate builtin,
options aren't restored when calling foo from sh emulation mode.  That
causes the above script to produce the following output:

    outside sh emulation: words_here
    inside sh emulation:  stuff words here

Because foo was defined outside of emulate I would have expected zsh to
treat the body of foo as native zsh code regardless of the emulation
mode of the calling code.  If I hadn't read the detailed emulate rules
and didn't understand how emulate worked with regard to options, I would
have expected the following output:

    outside sh emulation: words_here
    inside sh emulation:  words_here

(same output for the same function in the same script)

To get the behavior I expect, I have to do the following:

    define_functions() {
        echo_stuff() { printf %s\\n "$*"; }
        foo() { ${CMD} words here; }
    }
    emulate zsh -c define_functions
    CMD=echo_stuff
    IFS=_
    printf "outside sh emulation: "; foo
    printf "inside sh emulation:  "; emulate sh -c foo

If zsh restored options when a non-sticky function is called from within
emulation then I wouldn't have to do the above define_functions hack.
(And $0 would act like I expect assuming POSIX_ARGZERO is enabled in sh
emulation mode.)

-Richard

Re: 'emulate sh -c' and $0

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 320 bytes --]

On Jun 3, 2014 4:35 PM, "Richard Hansen" <rhansen@bbn.com> wrote:
>
> Isn't this a general problem with how zsh supports mix-and-match shell
> code?

One man's problem is another man's solution.  The shell has always been a
dynamically scoped language.  If that's not what you want, you should be
using a different one.

Re: 'emulate sh -c' and $0

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

On Jun 3, 2014 1:15 PM, "Richard Hansen" <rhansen@bbn.com> wrote:
>
> Wow, thank you for committing this change to master!  It works well in
> my limited testing, except for the documented limitation that
> POSIX_ARGZERO stays enabled when calling a non-emulated function from a
> function defined in 'emulate <shell> -c'.  I'm not sure how much this
> will matter in practice, but if backward compatibility wasn't a concern
> it'd be nice if zsh temporarily restored options when invoking a
> function outside the 'emulate <shell> -c' boundary.

What this is effectively requesting is that all functions have "sticky"
options, always.  We already rejected that idea for reasons of dynamic
scoping when originally designing "emulate ... -c".  I suppose a special
case could be made of POSIX_ARGZERO such that it is always sticky even when
other options are not, but that seems like an awful lot of effort for
something that doesn't matter most of the time.

Re: 'emulate sh -c' and $0

[Richard Hansen]

On 2014-06-03 17:21, Bart Schaefer wrote:
> On Jun 3, 2014 1:15 PM, "Richard Hansen" <rhansen@bbn.com> wrote:
>>
>> Wow, thank you for committing this change to master!  It works well in
>> my limited testing, except for the documented limitation that
>> POSIX_ARGZERO stays enabled when calling a non-emulated function from a
>> function defined in 'emulate <shell> -c'.  I'm not sure how much this
>> will matter in practice, but if backward compatibility wasn't a concern
>> it'd be nice if zsh temporarily restored options when invoking a
>> function outside the 'emulate <shell> -c' boundary.
> 
> What this is effectively requesting is that all functions have "sticky"
> options, always.

Not necessarily -- I was thinking more along the lines of temporarily
restoring the top-level (non-emulated) option state when calling a
function that was not defined inside of 'emulate <shell> -c'.  (Maybe
there's not a significant implementation difference between what I'm
thinking and assigning sticky options to all functions.)

I'm not familiar with Zsh's implementation, but this is my current
(probably flawed) mental model:

  * there is a structure type (e.g., 'struct option_state') that is
    effectively an array of booleans (one for each option) that is
    intended to record which options are enabled/disabled
  * there is a global variable (e.g., 'global_option_state') of the
    above structure type that stores which options are enabled/disabled
    when *not* running in an emulation mode
  * there is a global variable (e.g., 'current_option_state') of type
    pointer to the above structure type that is used to indicate which
    options are currently active
  * when it's time to call a function, the following happens:

      if the function was defined under an emulation mode and that
      emulation mode differs from the current emulation mode (if
      applicable):
          1. create a new struct option_state
          2. load the sticky options into the new struct option_state
          3. back up the current_option_state pointer
          4. set current_option_state to point to the new struct
             option_state
          5. call the function
          6. restore current_option_state
          7. delete the new struct option_state
      else:
          1. call the function

If my mental model is roughly correct, then the following could be
inserted between the above 'if' case and the 'else' case to temporarily
restore options when invoking a function outside the 'emulate -c' boundary:

      else if the function was not defined under an emulation mode but
      we're currently running under an emulation mode:
          1. back up the current_option_state pointer
          2. set current_option_state to point to global_option_state
          3. call the function
          4. restore current_option_state

> We already rejected that idea for reasons of dynamic
> scoping when originally designing "emulate ... -c".  I suppose a special
> case could be made of POSIX_ARGZERO such that it is always sticky even when
> other options are not, but that seems like an awful lot of effort for
> something that doesn't matter most of the time.

Yeah, that doesn't seem worthwhile.  In general I dislike special cases,
and it seems like the other options would also benefit from being
temporarily restored to their non-emulation state when calling a
function defined outside of 'emulate <shell> -c'.

-Richard

Re: 'emulate sh -c' and $0

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 1207 bytes --]

On Jun 3, 2014 3:54 PM, "Richard Hansen" <rhansen@bbn.com> wrote:
>
> I was thinking more along the lines of temporarily
> restoring the top-level (non-emulated) option state when calling a
> function that was not defined inside of 'emulate <shell> -c'.  (Maybe
> there's not a significant implementation difference between what I'm
> thinking and assigning sticky options to all functions.)

Implementation aside, operationally this still violates dynamic scoping.
It means for example that the completion system can't set extendedglob on
entry and be sure it remains in effect throughout any helper functions it
calls.

The reason sticky emulation works the way it does is because it allows the
function author to explicitly assert that dynamic scoping should not apply
to the options in effect in that function, but the default scope is still
dynamic for all other functions.

It might be possible to introduce another builtin/precommand that
temporarily unwinds one level of option state before entering the next,
sort of a "go play at Grandma's house" wrapper ... but I haven't really
thought about how that could be done.  It would require exporting a bunch
of the local C state of doshfunc(), I bet.

Re: 'emulate sh -c' and $0

[Bart Schaefer]

On Jun 3,  5:03pm, Bart Schaefer wrote:
}
} On Jun 3, 2014 3:54 PM, "Richard Hansen" <rhansen@bbn.com> wrote:
} >
} > I was thinking more along the lines of temporarily
} > restoring the top-level (non-emulated) option state when calling a
} > function that was not defined inside of 'emulate <shell> -c'.  (Maybe
} > there's not a significant implementation difference between what I'm
} > thinking and assigning sticky options to all functions.)
} 
} Implementation aside, operationally this still violates dynamic scoping.
} It means for example that the completion system can't set extendedglob on
} entry and be sure it remains in effect throughout any helper functions it
} calls.

OK, that's not quite true.  What you're suggesting is not that the called
function has made the global options sticky, rather that if the calling
function has sticky options, it automatically reverts them upon making
a nested call.  So that would only mess up completion if somebody did
"emulate zsh -c compinit" to load it.

That still prevents the calling function from intentionally propagating
a particular set of options down to the called function, which I think
is the more common use case (certainly so far it has been the *only*
use case, though that may just be because no other is possible).

And as I suspected "reverts them" currently involves completing the C
fuction scope of doshfunc(), though that could be disentangled with a
bit of effort.

Re: 'emulate sh -c' and $0

[Bart Schaefer]

On Jun 3,  6:54pm, Richard Hansen wrote:
}
}       else if the function was not defined under an emulation mode but
}       we're currently running under an emulation mode:
}           1. back up the current_option_state pointer
}           2. set current_option_state to point to global_option_state
}           3. call the function
}           4. restore current_option_state

The flaw in this model is that (1) there is no global option state object,
only a local state object on the C call stack that represents the global
state at the time the function was entered; and (2) even if there were a
global state, pointing to that would still be wrong; what you want is to
point to (i.e., overwrite the current global state with) the option state
from one frame earlier in the call stack.

Next you have the issue of the LOCAL_OPTIONS option, and what that means
to your emulated state when that is not set in the frame from which you
restored.