bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Glyph]

If we have a shell file like this, designed to be sourced:


#!/bin/zsh

function sample_function () {
    (
        setopt INTERACTIVECOMMENTS;
        # interactive comment test
        local bogus="$(
            # test comment
            echo "sample value";
        )";
        echo "${NOTSET:-${bogus}}";
    )
}

Comments in sample_function are interpreted normally, but comments *inside* the "bogus" expression will be interpreted as executing in an interactive context, and will therefore cause errors if INTERACTIVECOMMENTS is not set, if the overall shell is interactive.

Thanks,

-g

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Bart Schaefer]

On Sun, Aug 29, 2021 at 1:58 PM Glyph <glyph@twistedmatrix.com> wrote:
>
> Comments in sample_function are interpreted normally, but comments *inside* the "bogus" expression will be interpreted as executing in an interactive context

Hm, well, this is quite obviously intentional based on the code that
interprets $(...) expressions.  The only way I can think of to change
it without breaking the behavior of $(...) typed at the actual command
line, would be for "source" and related operations to put
INTERACTIVECOMMENTS into the "sticky" emulation context during parsing
of the file (even if it was off in the parent shell).

That is, if "source" et al. always behaved as if

emulate $(emulate) -o interactivecomments -c 'source ...'

then comments in $(...) and related contexts would be ignored as if
they were parsed during the "source" command instead of during the
execution of the substitution.

But that could have some pretty confusing side-effects for functions
that want to examine $options (for example).

Incidentally, you almost certainly want

setopt LOCALOPTIONS INTERACTIVECOMMENTS

in any functions where you intend to put comments inside $(...),
otherwise you will propagate INTERACTIVECOMMENTS into the parent
shell.

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Stephane Chazelas]

2021-08-30 14:35:56 -0700, Bart Schaefer:
> On Sun, Aug 29, 2021 at 1:58 PM Glyph <glyph@twistedmatrix.com> wrote:
> >
> > Comments in sample_function are interpreted normally, but comments *inside* the "bogus" expression will be interpreted as executing in an interactive context
> 
> Hm, well, this is quite obviously intentional based on the code that
> interprets $(...) expressions.  The only way I can think of to change
> it without breaking the behavior of $(...) typed at the actual command
> line, would be for "source" and related operations to put
> INTERACTIVECOMMENTS into the "sticky" emulation context during parsing
> of the file (even if it was off in the parent shell).
> 
> That is, if "source" et al. always behaved as if
> 
> emulate $(emulate) -o interactivecomments -c 'source ...'
> 
> then comments in $(...) and related contexts would be ignored as if
> they were parsed during the "source" command instead of during the
> execution of the substitution.
> 
> But that could have some pretty confusing side-effects for functions
> that want to examine $options (for example).
> 
> Incidentally, you almost certainly want
> 
> setopt LOCALOPTIONS INTERACTIVECOMMENTS
> 
> in any functions where you intend to put comments inside $(...),
> otherwise you will propagate INTERACTIVECOMMENTS into the parent
> shell.


But that used to work OK, up to 5.4.1 at least, and also applies
to echo $(#comment) outside of functions.

~$ echo 'echo `# c`' > a
~$ . ./a
./a:1: bad pattern: #


I would say it's a bug and a regression or at least a drastic
change of behaviour since the documentation seems to be silent
on that aspect.

It used to work OK even for functions, as comments were always
recognised inside command substitution even in interactive
shells with interactive_comments off:

5.1.1% echo 'f() { echo `# c`; }; f' > a
5.1.1% . ./a

5.1.1% f

5.1.1% which f
f () {
        echo `# c`
}
5.1.1% echo `# c`

5.1.1% echo $(# c
cmdsubst> )

5.1.1% [[ -o interactive_comments ]] || echo no
no

Was mentioned again at https://unix.stackexchange.com/questions/666931/command-not-found-in-sourced-multiline-command-with-comments

Comments are still recognised today (5.8) inside process
substitution in interactive shells:

$ cat <( echo test # comment )
test

See how the ")" is still found as "#" is not considered as a
comment by the outer shell, but the comment is still stripped by
the subshell.

$ set -o interactivecomments
$ cat <( echo test # comment  )
cmdsubst> )
test

-- 
Stephane

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Bart Schaefer]

On Mon, Aug 30, 2021 at 11:03 PM Stephane Chazelas
<stephane@chazelas.org> wrote:
>
> But that used to work OK, up to 5.4.1 at least, and also applies
> to echo $(#comment) outside of functions.

+Changes from 5.4 to 5.4.3
+-------------------------
+
+The effect of the NO_INTERACTIVE_COMMENTS option extends into $(...) and
+`...` command substitutions when used on the command line.  Previously,
+comments were always recognized within command substitutions unless the
+comment character "#" was disabled via reset of $histchars.

Thread started with workers/41656 (September 2017) about writing
aliases that start with "#" and being able to use them inside $(...).

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Stephane Chazelas]

2021-08-31 20:41:57 -0700, Bart Schaefer:
> On Mon, Aug 30, 2021 at 11:03 PM Stephane Chazelas
> <stephane@chazelas.org> wrote:
> >
> > But that used to work OK, up to 5.4.1 at least, and also applies
> > to echo $(#comment) outside of functions.
> 
> +Changes from 5.4 to 5.4.3
> +-------------------------
> +
> +The effect of the NO_INTERACTIVE_COMMENTS option extends into $(...) and
> +`...` command substitutions when used on the command line.  Previously,
> +comments were always recognized within command substitutions unless the
> +comment character "#" was disabled via reset of $histchars.
> 
> Thread started with workers/41656 (September 2017) about writing
> aliases that start with "#" and being able to use them inside $(...).

Thanks.

That still leaves a few problems:

First and more important, comments are meant to be stripped in
scripts, whether they're executed or sourced, whether that's
from an interactive shell with interactivecomments or not.

If not, you wouldn't be able to have comments in any script you
intent to source other than by adding something like  
set -o interactivecomments at the beginning (and restoring upon
return).

So here, I'd say:

~$ cat a
echo foo # comment
echo $(
  echo bar # comment
)
~$ . ./a
foo
./a:3: bad pattern: #

is clearly a bug (regression).

Once that's fixed there's still the problem of delayed (second)
parsing of code inside command substitution. Which would still
cause functions defined in sourced scripts to fail (or behave
differently) when invoked from interactive shells.

It seems to me the interactivecomments option should be honoured
at time of reading code, and comments stripped then when it's
on.

$ set -o interactivecomments
~$ f() {
echo foo # comment
echo $(
echo bar # comment
)
}
~$ f
foo
bar
~$ set +o interactivecomments
~$ f
foo
f:3: bad pattern: #

$ which f
f () {
        echo foo
        echo $(
echo bar # comment
)
}


IMO, that function f should have been stored as:

$ which f
f () {
        echo foo
        echo $(
echo bar
)
}

As already noted, there's also a consistency problem in that
while comments are no longer stripped in command substitutions
with interactivecomments off, they still are in process
substitutions

(tangentially related, Also note the misleading PS2 output in:

$ cat <(
cmdsubst>

)

(expected procsubst> / prcsubst>...)
-- 
Stephane

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Peter Stephenson]

> On 01 September 2021 at 08:55 Stephane Chazelas <stephane@chazelas.org> wrote:
> So here, I'd say:
> 
> ~$ cat a
> echo foo # comment
> echo $(
>   echo bar # comment
> )
> ~$ . ./a
> foo
> ./a:3: bad pattern: #
> 
> is clearly a bug (regression).

Yes, that does seem pretty clear.  Looks easy to fix and test.

Is there anything similar to source that might require attention?
I couldn't think of anything --- source looks like the only case
where we deflect the input from another file without starting a
new shell.

pws

diff --git a/Src/exec.c b/Src/exec.c
index 79d8064b6..1f23a862d 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -4622,7 +4622,7 @@ getoutput(char *cmd, int qt)
     char *s;
 
     int onc = nocomments;
-    nocomments = (interact && unset(INTERACTIVECOMMENTS));
+    nocomments = (interact && !sourcelevel && unset(INTERACTIVECOMMENTS));
     prog = parse_string(cmd, 0);
     nocomments = onc;
 
diff --git a/Test/A01grammar.ztst b/Test/A01grammar.ztst
index 88fc8606e..c114ff103 100644
--- a/Test/A01grammar.ztst
+++ b/Test/A01grammar.ztst
@@ -944,3 +944,12 @@ F:Note that the behaviour of 'exit' inside try-list inside a function is unspeci
  if : ${(e)a}; then echo x; fi
 1:Status on bad substitution in if without else
 ?(eval):2: bad substitution
+
+ echo 'echo foo # comment
+  echo $(
+  echo bar # comment
+ )' >source_comments.zsh
+ $ZTST_testdir/../Src/zsh -f -o extendedglob -is -c '. ./source_comments.zsh'
+0:Comments should be handled in command subst in interactively sourced files
+>foo
+>bar

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Bart Schaefer]

On Wed, Sep 1, 2021 at 8:18 AM Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
>
> Is there anything similar to source that might require attention?

Autoloaded functions?

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Bart Schaefer]

On Wed, Sep 1, 2021 at 8:18 AM Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
>
> Yes, that does seem pretty clear.  Looks easy to fix and test.

I'm not sure it's actually all that clear.  Consider the original case:

% alias '#x'='echo The X Factor'
% source =(<<<'echo $(
#x is not a comment
)')
The X Factor is not a comment

I haven't come up with a way to satisfy both workers/41656 and the
current complaint, without having to turn the tokenizer inside-out to
check for aliases during comment removal.

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Stephane Chazelas]

2021-09-01 11:09:33 -0700, Bart Schaefer:
> On Wed, Sep 1, 2021 at 8:18 AM Peter Stephenson
> <p.w.stephenson@ntlworld.com> wrote:
> >
> > Yes, that does seem pretty clear.  Looks easy to fix and test.
> 
> I'm not sure it's actually all that clear.  Consider the original case:
> 
> % alias '#x'='echo The X Factor'
> % source =(<<<'echo $(
> #x is not a comment
> )')
> The X Factor is not a comment
[...]

That's sourcing a script, same as source =(sed '/end/,$d' script)
so the stripping of comments should take precendence here IMO.

Note that zsh still strips comments in

$ cat =(echo foo # comment)
foo

$ eval 'echo foo # comment'
foo

I wonder what the original intention was for not honouring
comments when run interactively. That doesn't sound useful to
me. Was that to mimic csh?

I find that much annoying when copy-pasting code at the prompt.

It feels wrong to break the shell syntax to accomodate weird use
cases.

-- 
Stephane

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Stephane Chazelas]

2021-09-01 19:23:24 +0100, Stephane Chazelas:
[...]
> I wonder what the original intention was for not honouring
> comments when run interactively. That doesn't sound useful to
> me. Was that to mimic csh?
[...]

FYI, in 1.0 (1990), INTERACTIVE_COMMENTS was a #ifdef (compile
time setting) off by default. Changed to an option (run time
setting) in 2.0 (1991), still off by default.

-- 
Stephane

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Bart Schaefer]

On Wed, Sep 1, 2021 at 11:23 AM Stephane Chazelas <stephane@chazelas.org> wrote:
>
> I wonder what the original intention was for not honouring
> comments when run interactively. That doesn't sound useful to
> me. Was that to mimic csh?

Almost certainly, yes.

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Peter Stephenson]

> On 01 September 2021 at 19:09 Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Wed, Sep 1, 2021 at 8:18 AM Peter Stephenson
> <p.w.stephenson@ntlworld.com> wrote:
> >
> > Yes, that does seem pretty clear.  Looks easy to fix and test.
> 
> I'm not sure it's actually all that clear.  Consider the original case:
> 
> % alias '#x'='echo The X Factor'
> % source =(<<<'echo $(
> #x is not a comment
> )')
> The X Factor is not a comment

Hmm, this is certainly true, but sourcing a file actually created on the
command line is a much more special case than sourcing a file that
already lives somewhere in the file system, even without the devious
alias.

pws

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Bart Schaefer]

On Thu, Sep 2, 2021 at 1:26 PM Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
>
> ... sourcing a file actually created on the
> command line is a much more special case ...

I don't have any objection to pushing workers/49353.

Re: bug report: INTERACTIVECOMMENTS option interacts oddly with shell functions in an interactive context

[Peter Stephenson]

On Wed, 2021-09-08 at 20:50 -0700, Bart Schaefer wrote:
> On Thu, Sep 2, 2021 at 1:26 PM Peter Stephenson
> <p.w.stephenson@ntlworld.com> wrote:
> > 
> > ... sourcing a file actually created on the
> > command line is a much more special case ...
> 
> I don't have any objection to pushing workers/49353.

Now committed.

pws