Inconsistencies in "-quoting and @-splitting, could someone elaborate?

Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Sebastian Gniazdowski]

Hello.
I'm having problem in grasping, is this behavior:
- "X-flag creates array, unless its result is a single element", and
- "X-flag creates array even when double quoted without use of @"

is this behavior consistent, or per-flag, or general
random/historical.. Because the two behaviors seem to apply for (s::)
and (z) (a='a b c d'; print -rl "${(z)a}" will print 4 lines, despite
"-quoting and lack of @), but they look like a two unrelated,
accidentally-similar, with possible further differences, exceptions,
so I would state there's a major inconsistency/historically-driven
problem in Zsh. Are there other such flags in Zsh?

I would add this information to Zsh Native Scripting Handbook. I will
investigate by experimenting and come up with conclusion myself if no
one will help.

Worth noting: the first behavior – no-array for singular result – can
be fixed into consistent (i.e. "always array") behavior by the updated
(A) flag, patch 40640, first appearing in Zsh version 5.4.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[dana]

On 27 Oct 2018, at 05:33, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
>I'm having problem in grasping, is this behavior:
>- "X-flag creates array, unless its result is a single element", and
>- "X-flag creates array even when double quoted without use of @"

I don't know if it's accurate to say that they 'create an array' — they expand
to a list of zero, one, or multiple words (or elements, or arguments, or
whatever you want to call them), which you might then *put into* or (in certain
parameter-expansion cases) *act on as* an array, but the result of the expansion
is not an array per se.

(Maybe that's arguable when it comes to (A), given how it's implemented, but i
find it useful to think of it that way even in that case.)

On 27 Oct 2018, at 05:33, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
>is this behavior consistent, or per-flag, or general
>random/historical.. Because the two behaviors seem to apply for (s::)
>and (z) (a='a b c d'; print -rl "${(z)a}" will print 4 lines, despite
>"-quoting and lack of @), but they look like a two unrelated,
>accidentally-similar, with possible further differences, exceptions,
>so I would state there's a major inconsistency/historically-driven
>problem in Zsh. Are there other such flags in Zsh?

I don't understand where the inconsistency is...? Are you just speculating that
there might be one?

dana

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Bart Schaefer]

On Sat, Oct 27, 2018 at 5:34 PM dana <dana@dana.is> wrote:
>
> On 27 Oct 2018, at 05:33, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
> >I'm having problem in grasping, is this behavior:
> >- "X-flag creates array, unless its result is a single element", and
> >- "X-flag creates array even when double quoted without use of @"
>
> I don't know if it's accurate to say that they 'create an array' — they expand
> to a list of zero, one, or multiple words (or elements, or arguments, or
> whatever you want to call them), which you might then *put into* or (in certain
> parameter-expansion cases) *act on as* an array, but the result of the expansion
> is not an array per se.

You're not wrong, but the documentation consistently refers to a word
list occuring in parameter expansion context as "an array".

To address Sebastian's question, the behavior of (s::) is historical.
Because zsh passes expansions around by value rather than by
reference, in the original implementation of nested expansions a
single-element array value was indistinguishable from a scalar.  This
didn't change until 2015, when extra effort was made to retain the
properties of a parameter across levels of nested expansion.  However,
the result of a split with (s::) is never a parameter (it's just an
array value), so it kept the old behavior.  This is true of anything
that is considered "simple word splitting" (#11 in the "Rules"
subsection of the parameter expansion docs), and also of the (z::)
operation (rule #17).  However, you'll never see the effect unless the
array value is inside a nested expansion; if there is only a single
level of expansion, double-quotes (rule #5) apply before splitting, so
the result remains an array.

To circumvent this historical behavior, the (@) and (A) flags are used
to propagate array interpretation from an inner nesting to an outer
one.

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Sebastian Gniazdowski]

On Sun, 28 Oct 2018 at 02:34, dana <dana@dana.is> wrote:

> >accidentally-similar, with possible further differences, exceptions,
> >so I would state there's a major inconsistency/historically-driven
> >problem in Zsh. Are there other such flags in Zsh?
>
> I don't understand where the inconsistency is...? Are you just speculating that
> there might be one?

@ is refered to as "@  -- double-quoted splitting of scalars" in
completion. I understand it as: for quoted expression, regenerate
array-form by doing splitting. So "$array" is a single string
containing the whole array contents (if not ksharrays, where it would
have only 1st element), while "${(@)array}" is still an array, because
theoretically, like the completion says, the "-induced string is split
to again become an array.

So one can obtain always-array behavior with @. This flag doesn't help
(s::) and (z) when the result is string. I feel this is inconsistent,
I should be able to "split" the resulting string into an array, like
it would happen here with @ and array:

% array=( "foo" )
% print "${#${(@)array}}"
1

Despite array is single element, @ splits it into array type. (s::)
and (z) behave differently for single-element result, that is one
(maybe subjective) inconsistency, and for the second – @ doesn't help
(ie. (z@) still doesn't return as an array), that's second
inconsistency IMO.

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Sebastian Gniazdowski]

On Sun, 28 Oct 2018 at 12:54, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
> Despite array is single element, @ splits it into array type. (s::)
> and (z) behave differently for single-element result, that is one
> (maybe subjective) inconsistency, and for the second – @ doesn't help
> (ie. (z@) still doesn't return as an array), that's second
> inconsistency IMO.

I've forgot one inconsistency and thought that a more explicit
description of them (of the (subjective) inconsistencies) can be
provided:

1.a) array=( "foo" ); print "${#${(@)array}}" -> 1
1.b) words="singleword"; print "${#${(z)words}}" -> 10
1.description) with @, it is possible to obtain "always-array"
behavior, while (z) and (s::) have exceptions and not always return
arrays (one could think that if a flag takes the responsibility for
splitting in string-context ("-quoted), then that it will mimic the
first tool to do that, the @)

2.a) the same as 1.a
2.b) words="singleword"; print "${#${(z@)words}}" -> 10
2.description) if ${(@)array} makes single-element quoted array back
an array again, then shouldn't @ acting on single-element data (Bart
said it's not array) returned by (z)/(s::) also make it an array
again?

3.a) array=( "foo" "bar" ); print -rl "$array" -> foo bar # both
elements in one line, signaling the expected "scalarization"
3.b) words="two words"; print -rl "${(z)words}" -> foo\nbar # two
lines, with one word each, indicating non-scalar result
3.c) words="two words"; print -rl "${(s: :)words}" -> foo\nbar # --- " ---
3.description) Basically, the general method of Zshell to obtain
scalars with no splits – double-quoting – isn't followed by (z) and
(s::) flags. The manuals often elaborate about some situations and
"-quoting like if this would be a general "scalarization" method
guaranteed by Zsh design. Hard to give examples, but maybe this one:

    ${name:|arrayname}
              ... If the substitution is scalar, either because
              name is a scalar variable >>or the expression is
quoted<<, the elements of arrayname are instead  tested
              against the entire expression.

Above slightly suggests "-quoting as a general method for obtaining scalars.
-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Sebastian Gniazdowski]

PS. I'm not examining this to induce some changes in Zsh – I think
every project can have bunch of its distinct traits, non-symmetrical –
but to gain more knowledge and maybe get information about other flags
like z&s, to provide a possibly clear paragraph about exceptions in
the document "Zsh Native Scripting Handbook". It currently follows "@
is double-quoted splitting" based on completion-description and some
insight/revelations, and it's too optimistic, doesn't describe
exceptions.
On Sun, 28 Oct 2018 at 16:41, Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> On Sun, 28 Oct 2018 at 12:54, Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> > Despite array is single element, @ splits it into array type. (s::)
> > and (z) behave differently for single-element result, that is one
> > (maybe subjective) inconsistency, and for the second – @ doesn't help
> > (ie. (z@) still doesn't return as an array), that's second
> > inconsistency IMO.
>
> I've forgot one inconsistency and thought that a more explicit
> description of them (of the (subjective) inconsistencies) can be
> provided:
>
> 1.a) array=( "foo" ); print "${#${(@)array}}" -> 1
> 1.b) words="singleword"; print "${#${(z)words}}" -> 10
> 1.description) with @, it is possible to obtain "always-array"
> behavior, while (z) and (s::) have exceptions and not always return
> arrays (one could think that if a flag takes the responsibility for
> splitting in string-context ("-quoted), then that it will mimic the
> first tool to do that, the @)
>
> 2.a) the same as 1.a
> 2.b) words="singleword"; print "${#${(z@)words}}" -> 10
> 2.description) if ${(@)array} makes single-element quoted array back
> an array again, then shouldn't @ acting on single-element data (Bart
> said it's not array) returned by (z)/(s::) also make it an array
> again?
>
> 3.a) array=( "foo" "bar" ); print -rl "$array" -> foo bar # both
> elements in one line, signaling the expected "scalarization"
> 3.b) words="two words"; print -rl "${(z)words}" -> foo\nbar # two
> lines, with one word each, indicating non-scalar result
> 3.c) words="two words"; print -rl "${(s: :)words}" -> foo\nbar # --- " ---
> 3.description) Basically, the general method of Zshell to obtain
> scalars with no splits – double-quoting – isn't followed by (z) and
> (s::) flags. The manuals often elaborate about some situations and
> "-quoting like if this would be a general "scalarization" method
> guaranteed by Zsh design. Hard to give examples, but maybe this one:
>
>     ${name:|arrayname}
>               ... If the substitution is scalar, either because
>               name is a scalar variable >>or the expression is
> quoted<<, the elements of arrayname are instead  tested
>               against the entire expression.
>
> Above slightly suggests "-quoting as a general method for obtaining scalars.
> --
> Sebastian Gniazdowski
> News: https://twitter.com/ZdharmaI
> IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
> Blog: http://zdharma.org



-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Bart Schaefer]

On Sun, Oct 28, 2018 at 4:54 AM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> @ is refered to as "@  -- double-quoted splitting of scalars" in
> completion.

Well, that's incorrect, Oliver should not have put it that way.  @ has
nothing to do with splitting of scalars; it would more correctly say
"prevent double-quoted joining of arrays".  (In general you should
assume the manual is correct when it conflicts with a completion
description, because the latter are often radically abbreviated for
short lines.)

> I understand it as: for quoted expression, regenerate
> array-form by doing splitting.

No, it regenerates nothing.  It just preserves the array form that
already existed.  If there was no array form in the first place, it
has no effect.

> So one can obtain always-array behavior with @.

Also wrong, which is why the (A) flag was extended to mean "treat
single-element arrays as arrays instead of as scalars".  The
description in _brace_parameter has not yet been updated for this
(another example of why you should believe the manual rather than the
completion descriptions).

> % array=( "foo" )
> % print "${#${(@)array}}"
> 1
>
> Despite array is single element, @ splits it into array type.

Again, no.  See my previous message.  ${${array}} is special because
$array is a parameter name.  ${${(s: :)scalar}} remains scalar if
there is no space to split on.  If you want the result of (s::) to
always be an array, use (As::).

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Bart Schaefer]

On Sun, Oct 28, 2018 at 8:41 AM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> (one could think that if a flag takes the responsibility for
> splitting in string-context ("-quoted), then that it will mimic the
> first tool to do that, the @)

Just to beat this entirely to death, you are wrong because that is NOT
WHAT @ DOES.

> 2.description) if ${(@)array} makes single-element quoted array back
> an array again, then shouldn't @ acting on single-element data (Bart
> said it's not array) returned by (z)/(s::) also make it an array
> again?

We could have done that instead of using (A), but it would have broken
historical behavior, so we did not.

> 3.description) Basically, the general method of Zshell to obtain
> scalars with no splits – double-quoting – isn't followed by (z) and
> (s::) flags.

Also explained in my first message.  Implicit joining occurs before
explicit splitting.

> The manuals often elaborate about some situations

This is spelled out in painful detail in the "Rules" subsection.

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Sebastian Gniazdowski]

On Sun, 28 Oct 2018 at 22:13, Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> On Sun, Oct 28, 2018 at 4:54 AM Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> >
> > @ is refered to as "@  -- double-quoted splitting of scalars" in
> > completion.
>
> Well, that's incorrect, Oliver should not have put it that way.  @ has
> nothing to do with splitting of scalars; it would more correctly say
> "prevent double-quoted joining of arrays".  (In general you should
...
> > So one can obtain always-array behavior with @.
>
> Also wrong, which is why the (A) flag was extended to mean "treat
> single-element arrays as arrays instead of as scalars".  The

Ok, with those clear pointers a new whole-picture can be constructed:
- @ - keep array form
- A - apply array form

> > I understand it as: for quoted expression, regenerate
> > array-form by doing splitting.
>
> No, it regenerates nothing.  It just preserves the array form that
> already existed.  If there was no array form in the first place, it
> has no effect.

I could see this clear fact from the execution times. @ doesn't cause
much slowdown. That said, people on IRC also directed to the
conclusion that @ is about splitting.

It is hard to create final-truth documents, I hope I will be able to
nicely rephrase it (ZNSHandbook).

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Bart Schaefer]

On Mon, Oct 29, 2018 at 2:11 AM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> Ok, with those clear pointers a new whole-picture can be constructed:
> - @ - keep array form
> - A - apply array form

Yes, much better.  The other effect of (@) is that, in double quotes,
the array elements become individually/separately quoted, similar to
the difference between "$*" and "$@".

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Sebastian Gniazdowski]

On Mon, 29 Oct 2018 at 16:24, Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> On Mon, Oct 29, 2018 at 2:11 AM Sebastian Gniazdowski
> <sgniazdowski@gmail.com> wrote:
> >
> > Ok, with those clear pointers a new whole-picture can be constructed:
> > - @ - keep array form
> > - A - apply array form
>
> Yes, much better.  The other effect of (@) is that, in double quotes,
> the array elements become individually/separately quoted, similar to
> the difference between "$*" and "$@".

I've rephrased the Zsh Native Scripting Handbook, its section about @.
Maybe you could have a brief look, about the correctness of the
statements?

http://zdharma.org/Zsh-100-Commits-Club/Zsh-Native-Scripting-Handbook.html#at-sign-about-keeping-array-form

-- 
Sebastian Gniazdowski
News: https://twitter.com/ZdharmaI
IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin
Blog: http://zdharma.org

Re: Inconsistencies in "-quoting and @-splitting, could someone elaborate?

[Bart Schaefer]

On Mon, Nov 5, 2018 at 11:04 AM Sebastian Gniazdowski
<sgniazdowski@gmail.com> wrote:
>
> I've rephrased the Zsh Native Scripting Handbook, its section about @.
> Maybe you could have a brief look, about the correctness of the
> statements?

It's a bit of a convoluted way to come at it, and the explanation of
(A) is kind of buried in another topic, but I don't see anything
actually wrong.

If I were to suggest edits, it would be to separate the KSH_ARRAYs and
Zsh native cases more fully, rather than mix the exceptions into the
flow as you have.