$(...) and <<

$(...) and <<

[Stephane Chazelas]

$ zsh -c 'echo $(cat << EOF
blah)
EOF
); echo test'
zsh:4: parse error near `)'

Most shells (ash being a notable exception) used to suffer from that bug.

zsh seems to be the only one nowadays (tested ksh93/bash/dash/yash/mksh).

By the way, is there  any plan of having an issue tracker for
zsh somewhere. That would help to keep track of the known
issues.

Cheers
Stephane

Re: $(...) and <<

[Bart Schaefer]

On Jul 31, 10:27am, Stephane Chazelas wrote:
}
} $ zsh -c 'echo $(cat << EOF
} blah)
} EOF
} ); echo test'
} zsh:4: parse error near `)'

Yeah, to fix that we're going to have a write a whole new chunk of parser
(lexer, really) specifically for $(...).  There's an entry point for it
now, but all it does is try to lexically consume a string ending with ")";
it handles balanced paren pairs but not the situation above.  Same with

% echo $(case $foo in
bar) echo test;;
zsh: parse error near `;;'

} By the way, is there  any plan of having an issue tracker for
} zsh somewhere. That would help to keep track of the known
} issues.

An issue tracker needs to be part of a larger plan -- we tried using the
tracker at SourceForge but it didn't work because no one monitors it, so
it became an issue black hole.  At the moment there isn't anyone to take
that responsibility.

Re: $(...) and <<

[Peter Stephenson]

On Thu, 31 Jul 2014 09:20:23 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jul 31, 10:27am, Stephane Chazelas wrote:
> }
> } $ zsh -c 'echo $(cat << EOF
> } blah)
> } EOF
> } ); echo test'
> } zsh:4: parse error near `)'
> 
> Yeah, to fix that we're going to have a write a whole new chunk of parser
> (lexer, really) specifically for $(...).  There's an entry point for it
> now, but all it does is try to lexically consume a string ending with ")";
> it handles balanced paren pairs but not the situation above.

I'm wondering if it's not *quite* that hard, but still far from minor...
one of those things that could really, really, really do with someone
with newly kindled enthusiasm taking an active interest in...

- Check if it looks like $( ... ) or $(( ... ) ... ) but not $(( ... ))
as before, in other words we have to guess if it's $( ... ) by looking
for a stray ")" that might actually not be the end of the command
substitution but probably implies it's not a $(( ... )) substitution.
A bit icky but we've done worse.

- Re-enter the parser at this point within the lexer and keep going
until we encounter an unexpected token, and if it's ")" everything's OK.
This is not substantially different from parsing the $(...) contents as
now, just much earlier, so it might not be such a lot of work.

- Now it gets interesting.  (Unless I'm missing something.)  We've just
parsed something that as far as the top-level (or rather next-level-up)
parser is concerned is just a string that's part of an argument.  Either
we've got to store that in a special way to say "retrieve this
pre-parsed stuff as part of this string when you need to substitute it",
or we simply record what we parsed and store it as a string for later
reparsing.  We'd almost certainly get away with the latter --- it may be
the text is present in the current history line, since we can't rely on
going back and getting from the input once that's been parsed, though
I'm not sure that's always maintained in every circumstance.

> } By the way, is there  any plan of having an issue tracker for
> } zsh somewhere. That would help to keep track of the known
> } issues.
> 
> An issue tracker needs to be part of a larger plan -- we tried using the
> tracker at SourceForge but it didn't work because no one monitors it, so
> it became an issue black hole.  At the moment there isn't anyone to take
> that responsibility.

Yes, exactly.

A full issue tracker allowing public input is a lot of maintenance work.
In principle it would be less work just to have an internal list of
issues that's basically the result of discussion on this list and can be
published as such.  But it's still some work for someone.

pws

Re: $(...) and <<

[Stephane Chazelas]

2014-07-31 17:42:08 +0100, Peter Stephenson:
[...]
> > An issue tracker needs to be part of a larger plan -- we tried using the
> > tracker at SourceForge but it didn't work because no one monitors it, so
> > it became an issue black hole.  At the moment there isn't anyone to take
> > that responsibility.
> 
> Yes, exactly.
> 
> A full issue tracker allowing public input is a lot of maintenance work.
> In principle it would be less work just to have an internal list of
> issues that's basically the result of discussion on this list and can be
> published as such.  But it's still some work for someone.
[...]

What about github?

Would it really be a lot more work than following this ML?

-- 
Stephane

Issue trackers (Re: $(...) and <<)

[Bart Schaefer]

On Jul 31,  6:23pm, Stephane Chazelas wrote:
}
} What about github?
} 
} Would it really be a lot more work than following this ML?

I haven't used github, but the mere fact of having to use a web browser
interface and fill out forms makes the process more tedious than
going through a mail folder -- and it's *in addition to* following
this list rather than instead.

However, the issue is by no means permanently decided.

-- 
Barton E. Schaefer

Re: $(...) and <<

[Bart Schaefer]

On Jul 31,  5:42pm, Peter Stephenson wrote:
} Subject: Re: $(...) and <<
}
} > } $ zsh -c 'echo $(cat << EOF
} > } blah)
} > } EOF
} > } ); echo test'
} > } zsh:4: parse error near `)'
} 
} - Check if it looks like $( ... ) or $(( ... ) ... ) but not $(( ... ))
} as before, in other words we have to guess if it's $( ... ) by looking
} for a stray ")" that might actually not be the end of the command
} substitution but probably implies it's not a $(( ... )) substitution.
} A bit icky but we've done worse.
} 
} - Re-enter the parser at this point within the lexer and keep going

I tried this a couple of different ways but always ended up with one or
more of the "make check" tests failing because I wasn't backtracking
properly after discovering that the first ")" wasn't a "))".  It's also
fairly ugly to discover that the first ")" in "$( ... ) ... )" is NOT
closing the expression unless you are already in the parser.

What it seems realy needs to happen is to always attempt to parse math
at "$((" and if that fails, back up to the second "(" and attempt to
parse shell code.  But that can require arbitrary lookahead ... which
is no worse than attempting to consume everything up to ")" as a string
and then parse it, but (as you mentioned) a bit harder to extract the
result because it's passing through the parser rather than lexer.

Re: $(...) and <<

[Stephane Chazelas]

2014-07-31 10:27:11 +0100, Stephane Chazelas:
> $ zsh -c 'echo $(cat << EOF
> blah)
> EOF
> ); echo test'
> zsh:4: parse error near `)'
> 
> Most shells (ash being a notable exception) used to suffer from that bug.
> 
> zsh seems to be the only one nowadays (tested ksh93/bash/dash/yash/mksh).
[...]

BTW,

zsh -c '
  echo $(
    case a in
      *) echo test
    esac)'

is also a problem.

-- 
Stephane