Is there something in parser that always starts new token?

Is there something in parser that always starts new token?

[Sebastian Gniazdowski]

Hello
a='`something something else'; print -rl -- "${(@z)a}"
output: `something something else

I would like to manually divide the string after "something". Is there
something in (z) flag that could be used to force new token?

Best regards,
Sebastian Gniazdowski

Re: Is there something in parser that always starts new token?

[Peter Stephenson]

On Fri, 02 Sep 2016 18:27:11 +0200
Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
> Hello
> a='`something something else'; print -rl -- "${(@z)a}"
> output: `something something else
> 
> I would like to manually divide the string after "something". Is there
> something in (z) flag that could be used to force new token?

The trouble here, as I see it, is that once you've realised the
expression you've got doesn't work as it is (in this case you've got
what looks like an incomplete backquoted substitution), what you decide
to do with it isn't down to the parser, which has given you its
best shot.  This is compounded by the fact that zsh doesn't understand
incomplete expressions --- there's nothing like parse-partial-sexp ---
so you can't take them apart, and there's nothing that does
token-by-token splitting rather than word-by-word splitting.  What's
more, ${(Q)a} doesn't even understand complete backquote expressions
since it only deals with quotes with no side effect.

You could just split the expression using non-syntactic word splitting,
and take the words after the first as your new input.

If instead what you want to do is remove the backquote, then I think
you're going to have to do that by hand.

pws

Re: Is there something in parser that always starts new token?

[Sebastian Gniazdowski]

On 2 September 2016 at 18:43, Peter Stephenson <p.stephenson@samsung.com> wrote:
> On Fri, 02 Sep 2016 18:27:11 +0200
> Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
>> Hello
>> a='`something something else'; print -rl -- "${(@z)a}"
>> output: `something something else
>>
>> I would like to manually divide the string after "something". Is there
>> something in (z) flag that could be used to force new token?
>
> The trouble here, as I see it, is that once you've realised the
> expression you've got doesn't work as it is (in this case you've got
> what looks like an incomplete backquoted substitution), what you decide
> to do with it isn't down to the parser, which has given you its
> best shot.

I'm using (z) in a special way: I have that vertical and horizontal
selection list, seen here:

https://asciinema.org/a/8k8v5ni5lnnpoc3z5jzl6jgav

All horizontal blocks navigated with the red highlight are obtained
via (z). So what I need is a way to unconditionally force (z) to start
new token, when I detect that a part of line is unbounded via:

exp='`x'                   # the part of line
exp="$exp test"      # an append to test
tokens=( "${(@z)exp}" )
[ "${#tokens}" = "1" ] && echo unbounded

When I get the "unbounded" condition, I can do exp="$exp (some special
char)" so that (z) will just stop at the special value. This way,
despite adding some random special value to the text, I'll at least
have rest of the text correctly splitted.

Best regards,
Sebastian Gniazdowski