[doc] "sh_word_split nothing to do with word splitting"?

[doc] "sh_word_split nothing to do with word splitting"?

[Stephane Chazelas]

SH_WORD_SPLIT (-y) <K> <S>
     Causes field splitting to be performed on unquoted parameter
     expansions.  Note that this option has nothing to do with word
     splitting.  (See *note Parameter Expansion::.)

[...]

${=SPEC}
     Perform word splitting using the rules for SH_WORD_SPLIT during the
     evaluation of SPEC, but regardless of whether the parameter appears
     in double quotes; if the '=' is doubled, turn it off.  This forces
     parameter expansions to be split into separate words before
     substitution, using IFS as a delimiter.  This is done by default in
     most other shells.

     Note that splitting is applied to WORD in the assignment forms of
     SPEC _before_ the assignment to NAME is performed.  This affects
     the result of array assignments with the A flag.

It's unclear to me what is meant by "Note that this option has
nothing to do with word splitting" here.

It seems the  SH_WORD_SPLIT description makes a distinction
between "field splitting" and "word splitting", but then again
throughout the manual (including the $=spec description above),
"word splitting" and "field splitting" are used interchangeably
to refer to the same thing.

Was the intended meaning something like: "Note that it does not
affect how the shell delimits tokens" or something like that?

From:
https://unix.stackexchange.com/questions/26661/what-is-word-splitting-why-is-it-important-in-shell-programming

-- 
Stephane

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Bart Schaefer]

On Mar 11,  9:44am, Stephane Chazelas wrote:
} Subject: [doc] "sh_word_split nothing to do with word splitting"?
}
} SH_WORD_SPLIT (-y) <K> <S>
}      Causes field splitting to be performed on unquoted parameter
}      expansions.  Note that this option has nothing to do with word
}      splitting.  (See *note Parameter Expansion::.)

What this means to say:

This option applies to field splitting within an expanded shell word,
not to splitting a command line into shell words during parsing.

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Stephane Chazelas]

2018-03-11 11:24:07 -0700, Bart Schaefer:
> On Mar 11,  9:44am, Stephane Chazelas wrote:
> } Subject: [doc] "sh_word_split nothing to do with word splitting"?
> }
> } SH_WORD_SPLIT (-y) <K> <S>
> }      Causes field splitting to be performed on unquoted parameter
> }      expansions.  Note that this option has nothing to do with word
> }      splitting.  (See *note Parameter Expansion::.)
> 
> What this means to say:
> 
> This option applies to field splitting within an expanded shell word,
> not to splitting a command line into shell words during parsing.

Thanks Bart for confirming that.

What about changing it to something like:


diff --git a/Doc/Zsh/options.yo b/Doc/Zsh/options.yo
index 25b3d57..7677f73 100644
--- a/Doc/Zsh/options.yo
+++ b/Doc/Zsh/options.yo
@@ -2314,9 +2314,17 @@ pindex(NOSHWORDSPLIT)
 cindex(field splitting, sh style)
 cindex(sh, field splitting style)
 item(tt(SH_WORD_SPLIT) (tt(-y)) <K> <S>)(
-Causes field splitting to be performed on unquoted parameter expansions.
-Note that this option has nothing to do with word splitting.
-(See noderef(Parameter Expansion).)
+Causes tt($IFS) field splitting to be performed on unquoted parameter
+expansions in addition to command substitutions. Note that contrary to
+POSIX shells, field splitting is still not performed on unquoted
+arithmetic expansions and contrary to the Bourne shell, not on words
+that are not the result of expansions. Like in other Bourne-like shells,
+field splitting is only performed in contexts where several words may be
+expected, such as in arguments to simple commands or anonymous functions,
+array assignments and for loop word lists. If the tt(MULTIOS) option is
+enabled, it is also performed in the targets of tt(<), tt(>) and tt(>>)
+redirection operators. (See noderef(Parameter Expansion) and
+noderef(Redirection)).
 )
 pindex(TRAPS_ASYNC)
 pindex(NO_TRAPS_ASYNC)

A reference to $IFS may also be useful. Is it possible to add
references to index entries in yodl?

(I only include the >, >>, < redirections, that's missing the
clobber variants and the >& >>&... Not sure how to express it
without making it too wordy).

Not sure it's worth mentioning:

$var() function-definition

as a context where shwordsplit happens.

See also https://unix.stackexchange.com/a/382914

-- 
Stephane

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Martijn Dekker]

Op 11-03-18 om 21:53 schreef Stephane Chazelas:
> What about changing it to something like:
[...]
> +Causes tt($IFS) field splitting to be performed on unquoted parameter
> +expansions in addition to command substitutions. Note that contrary to
> +POSIX shells, field splitting is still not performed on unquoted
> +arithmetic expansions

In terms of sh emulation, that's actually a bug. It may be terrible (who
would ever want to split an arithmetic expansion?), but emulation is
emulation and SH_WORD_SPLIT should cause splitting as in POSIX sh.

>                        and contrary to the Bourne shell, not on words
> +that are not the result of expansions.

Now that even Solaris finally got rid of it, I think the ancient Bourne
shell is definitively obsolete and not worth mentioning in current
documentation. POSIX is the norm now.

>                                         Like in other Bourne-like shells,

I'd say "Like in other POSIX shells,".

[...]
> Not sure it's worth mentioning:
> 
> $var() function-definition
> 
> as a context where shwordsplit happens.

It might be better if the possibility to define functions by names
resulting from expansions were disabled completely if SH_WORD_SPLIT is
active, so the program fails properly instead of producing weird/broken
behaviour.

Alternatively, SH_WORD_SPLIT could be ignored in that context.

- M.

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Stephane Chazelas]

2018-03-12 00:41:03 +0100, Martijn Dekker:
> Op 11-03-18 om 21:53 schreef Stephane Chazelas:
> > What about changing it to something like:
> [...]
> > +Causes tt($IFS) field splitting to be performed on unquoted parameter
> > +expansions in addition to command substitutions. Note that contrary to
> > +POSIX shells, field splitting is still not performed on unquoted
> > +arithmetic expansions
> 
> In terms of sh emulation, that's actually a bug. It may be terrible (who
> would ever want to split an arithmetic expansion?), but emulation is
> emulation and SH_WORD_SPLIT should cause splitting as in POSIX sh.

I'd rather zsh keep it that way as a statement of resistance
against silliness, at least until someone complains that his
POSIX script fails when run on zsh because its arithmetic
expansions are not split as expected.

Not doing so would keep fixing scripts where authors omitted to
quote arithmetic expansions in contexts where $IFS contained
dashes (or digits) on the ground that no shell author in their
right mind would ever want to subject arithmetic expansion to
split+glob (note that no shell does split+glob upon tilde
expansion nor process substitution nor $'...' expansions (though
bash used to do globbing upon tilde expansion) where that
wouldn't make sense *either*).

pdksh was not doing word/field splitting there and posh and
OpenBSD sh still don't.

> 
> >                        and contrary to the Bourne shell, not on words
> > +that are not the result of expansions.
> 
> Now that even Solaris finally got rid of it, I think the ancient Bourne
> shell is definitively obsolete and not worth mentioning in current
> documentation. POSIX is the norm now.

Agreed. I suspect the original text was making that point about
"shwordsplit having nothing to do with word splitting" in
reference to the Bourne shell.

> >                                         Like in other Bourne-like shells,
> 
> I'd say "Like in other POSIX shells,".

Except that zsh (like many other Bourne-like shells) is not
strictly speaking fully compliant (though on several aspects is more
compliant in sh emulation than some certified ones like the
/usr/xpg4/bin/sh of Solaris)

> [...]
> > Not sure it's worth mentioning:
> > 
> > $var() function-definition
> > 
> > as a context where shwordsplit happens.
> 
> It might be better if the possibility to define functions by names
> resulting from expansions were disabled completely if SH_WORD_SPLIT is
> active, so the program fails properly instead of producing weird/broken
> behaviour.
> 
> Alternatively, SH_WORD_SPLIT could be ignored in that context.
[...]

AFAIK, that's a feature which is actually used in some of the
functions shipped with zsh to declare several functions at once
with the same body.

It's not a POSIX conformance issue as the behaviour is
unspecified there for:

$var() { ...; }

As $var is not a valid function name.

In any case, I wouldn't expect anyone to use SH_WORD_SPLIT for
anything but to interpret POSIX/ksh scripts (via "emulate sh/ksh")
which wouldn't have such constructs, so I don't expect it would
cause any problem.

On the other hand, one might still want to do:

$=myfunctions() { something; zle .$WIDGET; }

for instance, that is using the explicit word-split operator to
define several functions at once.

-- 
Stephane

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Stephane Chazelas]

2018-03-12 07:43:29 +0000, Stephane Chazelas:
[...]
> Not doing so would keep fixing scripts where authors omitted to
> quote arithmetic expansions in contexts where $IFS contained
> dashes (or digits) on the ground that no shell author in their
> right mind would ever want to subject arithmetic expansion to
> split+glob (note that no shell does split+glob upon tilde
> expansion nor process substitution nor $'...' expansions (though
> bash used to do globbing upon tilde expansion) where that
> wouldn't make sense *either*).
[...]

Also note that zsh arithmetic expansions may output globs
(though the extendedglob + emulate sh + [#base] is an unlikely
combination):

/dev/fd$ ARGV0=sh zsh -o extendedglob -c 'a=$(([#8]8)); echo $a'
10

(8#10 expanded to 10 as # is an extendedglob operator and
there's a "10" file in /dev/fd).

POSIX also requires the result of arithmetic expansions to be
subject to globbing though of course it doesn't really apply in
POSIX scripts except maybe in things like [0$((-3))]

So if zsh was made POSIX compliance in that regard,
echo $(([#8]8)) would expand to 10 in that context above.

-- 
Stephane

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Stephane Chazelas]

So, how about this new take at it? Renders (in info) as:

SH_WORD_SPLIT (-y) <K> <S>
     Causes $IFS field splitting to be performed on unquoted parameter
     expansions in addition to command substitutions.  Note that
     contrary to POSIX shells, field splitting is still not performed on
     unquoted arithmetic expansions.  Like in other Bourne-like shells,
     field splitting is only performed in contexts where several words
     may be expected, such as in arguments to simple commands or inline
     functions, array assignments and for loop word lists.  If the
     MULTIOS option is enabled, it is also performed in the targets of
     file redirections (except for the read+write ones).  (See *note
     Parameter Expansion:: and *note Redirection::).

diff --git a/Doc/Zsh/options.yo b/Doc/Zsh/options.yo
index 25b3d57..97782de 100644
--- a/Doc/Zsh/options.yo
+++ b/Doc/Zsh/options.yo
@@ -2314,9 +2314,16 @@ pindex(NOSHWORDSPLIT)
 cindex(field splitting, sh style)
 cindex(sh, field splitting style)
 item(tt(SH_WORD_SPLIT) (tt(-y)) <K> <S>)(
-Causes field splitting to be performed on unquoted parameter expansions.
-Note that this option has nothing to do with word splitting.
-(See noderef(Parameter Expansion).)
+Causes tt($IFS) field splitting to be performed on unquoted parameter
+expansions in addition to command substitutions. Note that contrary to
+POSIX shells, field splitting is still not performed on unquoted
+arithmetic expansions. Like in other Bourne-like shells, field splitting
+is only performed in contexts where several words may be expected, such
+as in arguments to simple commands or inline functions, array
+assignments and for loop word lists. If the tt(MULTIOS) option is
+enabled, it is also performed in the targets of file redirections
+(except for the read+write ones). (See noderef(Parameter Expansion) and
+noderef(Redirection)).
 )
 pindex(TRAPS_ASYNC)
 pindex(NO_TRAPS_ASYNC)

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Bart Schaefer]

On Fri, Mar 16, 2018 at 10:26 AM, Stephane Chazelas
<stephane.chazelas@gmail.com> wrote:
> So, how about this new take at it? Renders (in info) as:
>
> SH_WORD_SPLIT (-y) <K> <S>
>      Causes $IFS field splitting to be performed on unquoted parameter
>      expansions in addition to command substitutions.  Note that

This is OK, I just keep thinking there should be a reference back to
the shell grammar; field splitting applies to anything identified in
the grammar by the token WORD provided that the WORD is not quoted.
Arithmetic expressions are treated as if double-quoted.  That's really
all there is to it.

Of course we don't formally spell out the grammer for the basic cases,
instead describing it in prose in the first few paragraphs under
"simple commands" ... and the "general comments on syntax" direct one
to *this* entry, so I suppose the detail is in fact needed here.

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Stephane Chazelas]

2018-03-16 11:28:23 -0700, Bart Schaefer:
> On Fri, Mar 16, 2018 at 10:26 AM, Stephane Chazelas
> <stephane.chazelas@gmail.com> wrote:
> > So, how about this new take at it? Renders (in info) as:
> >
> > SH_WORD_SPLIT (-y) <K> <S>
> >      Causes $IFS field splitting to be performed on unquoted parameter
> >      expansions in addition to command substitutions.  Note that
> 
> This is OK, I just keep thinking there should be a reference back to
> the shell grammar; field splitting applies to anything identified in
> the grammar by the token WORD provided that the WORD is not quoted.
> Arithmetic expressions are treated as if double-quoted.  That's really
> all there is to it.

Note that for $((text)), there's possible confusion. In the
POSIX spec, the reference to "as if double quoted" is about the
interpretation of text, not $((text)).

In POSIX, in $(($1 + 2)), $1 is not subject to split + glob
because $1 + 2 is treated as if quoted, but $(($1 + 2)) is
*required* (!) to be split+globbed.

As in:

IFS=2; echo $((11 * 11))

is *required* to output

1 1

but zsh, even in sh emulation outputs "121" as everyone would
expect.

That's why I mention it in the text to clarify that it's an
intentionaly deviation from the POSIX standard.

(I will try and have POSIX lift that silly requirement but I
have little hope they'll accept; note that a few shells have
already gone out of their way to change their sensible
behaviour to meet POSIX compliance on that front)

-- 
Stephane

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Martijn Dekker]

Op 12-03-18 om 08:43 schreef Stephane Chazelas:
> I'd rather zsh keep it that way as a statement of resistance
> against silliness, at least until someone complains that his
> POSIX script fails when run on zsh because its arithmetic
> expansions are not split as expected.

The thing is, there are very good reasons for that silliness, and for
not resisting it in sh emulation.

In a way, POSIX is primarily descriptive, not prescriptive. It intends
to incorporate the historical practice that real-world shell
implementations have copied into the standard, so that everyone can be
compatible with it.

For the most part, that historical practice was established by AT&T
ksh88. It's mostly obsolete now but very much lives on in all the shells
that copy its behaviour.

ksh88 implemented the original shell arithmetic expansion. It subjects
those expansions to IFS field splitting.

Not emulating this in zsh's sh emulation mode will probably not cause
many scripts to fail on zsh-as-sh. However, someone could use zsh to
develop a POSIX script intended to be compatible with other shells.

And that someone might end up under the mistaken impression that it's
pointless to quote arithmetic expansions. Cue potential mysterious
breakage on non-zsh shells.

That's why I believe it's important that this is fixed in sh emulation.

> pdksh was not doing word/field splitting there and posh and
> OpenBSD sh still don't.

Then pdksh, supposedly a ksh88 clone, failed to clone ksh88 in that
aspect -- among many others.

pdksh, posh, NetBSD ksh, OpenBSD sh/ksh, and nearly all other pdksh
derivatives are all full of bugs and fatally broken when it comes to
implementing POSIX compatibility.

The only properly working pdksh derivative is mksh (and its "legacy"
even-more-POSIX companion lksh).

- M.

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Stephane Chazelas]

2018-03-24 21:17:48 +0100, Martijn Dekker:
> Op 12-03-18 om 08:43 schreef Stephane Chazelas:
> > I'd rather zsh keep it that way as a statement of resistance
> > against silliness, at least until someone complains that his
> > POSIX script fails when run on zsh because its arithmetic
> > expansions are not split as expected.
> 
> The thing is, there are very good reasons for that silliness, and for
> not resisting it in sh emulation.
> 
> In a way, POSIX is primarily descriptive, not prescriptive. It intends
> to incorporate the historical practice that real-world shell
> implementations have copied into the standard, so that everyone can be
> compatible with it.
> 
> For the most part, that historical practice was established by AT&T
> ksh88. It's mostly obsolete now but very much lives on in all the shells
> that copy its behaviour.
[...]

There have been many cases where POSIX have specified accidents
of implementation or design bugs of original implementations and
the POSIX specification has been fixed later on when spotted
even in some cases forcing the original implementation to be
fixed.

IMO, here, it's clearly one of those. The solution here is not to
implement the bug but fix the specification.

> ksh88 implemented the original shell arithmetic expansion. It subjects
> those expansions to IFS field splitting.
> 
> Not emulating this in zsh's sh emulation mode will probably not cause
> many scripts to fail on zsh-as-sh. However, someone could use zsh to
> develop a POSIX script intended to be compatible with other shells.
> 
> And that someone might end up under the mistaken impression that it's
> pointless to quote arithmetic expansions. Cue potential mysterious
> breakage on non-zsh shells.

There are many "unspecified areas" in POSIX where it's the case,
if you rely on unspecified behaviour you can't expect anything
portably.

Here I would expect POSIX, when fixed would at first leave the
behaviour unspecified to allow both the bogus historical
behaviour and the zsh behaviour (which until not so long ago was
the behaviour of most other shells, it's quite sad to see that
many have broken their shell to meet POSIX compliance), maybe
with a "future direction" requiring the zsh behaviour.

In the mean time, we should document the difference so it's no
longer "mysterious" like in my  suggested patch.

> That's why I believe it's important that this is fixed in sh emulation.
> 
> > pdksh was not doing word/field splitting there and posh and
> > OpenBSD sh still don't.
> 
> Then pdksh, supposedly a ksh88 clone, failed to clone ksh88 in that
> aspect -- among many others.
[...]

pdksh has fixed a number of issues in ksh88 (itself broken on
many aspects as you've found out) as well. Another one shared
with zsh: that the ".*" glob does not include "." nor ".." (in
the case of pdksh actually inherited from the Forsyth shell).
That also breaks POSIX compliance, though last time I mentioned
it on the Austin Group mailing list, they would likely be
willing to change the spec to allow it (POSIX already leaves it
undefined if readdir() returns those IIRC).

Another example is brace expansion (also supported by zsh and bash)
which also breaks POSIX compliance which was not in ksh88, added
in pdksh (1992), and then later in ksh93.

-- 
Stephane

Re: [doc] "sh_word_split nothing to do with word splitting"?

[Martijn Dekker]

Op 25-03-18 om 08:42 schreef Stephane Chazelas:
> There have been many cases where POSIX have specified accidents
> of implementation or design bugs of original implementations and
> the POSIX specification has been fixed later on when spotted
> even in some cases forcing the original implementation to be
> fixed.
> 
> IMO, here, it's clearly one of those. The solution here is not to
> implement the bug but fix the specification.

By that token, global field splitting and globbing themselves are
massive design flaws (which they are -- as you know I've been working on
a cross-shell way to make it practical to get rid of them).

Yet, POSIX shells conform, at least in their respective POSIX modes.

At least the current spec is consistent. All POSIX expansions are
subject to global split & glob (unless those are disabled).

Why would $( ... ) be subject to split+glob but not $(( ... ))? That
would be a bit confusing, and would detract from quoting expansions
consistently.

I think consistent and broken is better than inconsistent and broken.

> There are many "unspecified areas" in POSIX where it's the case,
> if you rely on unspecified behaviour you can't expect anything
> portably.

I don't see that as a reason to add yet another unspecified behaviour.
In practice we'd be stuck with it for a decade anyway, maybe two.

To rephrase myself, I think specified and broken is better than
unspecified and broken.

> In the mean time, we should document the difference so it's no
> longer "mysterious" like in my  suggested patch.

Yes, but what wonderful planet do you live on, where people actually
read documentation? :)

>> Then pdksh, supposedly a ksh88 clone, failed to clone ksh88 in that
>> aspect -- among many others.
> [...]
> 
> pdksh has fixed a number of issues in ksh88 (itself broken on
> many aspects as you've found out) as well. Another one shared
> with zsh: that the ".*" glob does not include "." nor ".." (in
> the case of pdksh actually inherited from the Forsyth shell).

I don't really see that as a fix though, just as another mostly
pointless difference causing incompatibility surprises. ksh93 matches
'.' and '..' to this day.

On zsh, it looks like another emulation bug. The SH_GLOB option should
cause ".*" to match "." and "..".

(Actually, modernish ought to detect this as a shell quirk with its own
ID. Thanks for reminding me of it.)

- M.