"${*:offset:length}" and ksh93/bash compatibility

"${*:offset:length}" and ksh93/bash compatibility

[Stephane Chazelas]

Hello,

From workers/28418 I understand the ${param:offset:length} was
introduced only for compatibility with bash (though the feature
was initially from ksh93) as zsh already had $param[first,last]
(long before bash had arrays or ksh93 even existed).

Still, there's a difference with ksh/bash when
${*:offset:length} (same for ${array[*]}) is quoted:

$ zsh --emulate ksh -c 'printf "<%s>\n" ${*:1:2}' zsh foo bar baz
<foo>
<bar>
$ zsh --emulate ksh -c 'printf "<%s>\n" "${*:1:2}"' zsh foo bar baz
<oo>

$ ksh -c 'printf "<%s>\n" ${*:1:2}' ksh foo bar baz
<foo>
<bar>
$ ksh -c 'printf "<%s>\n" "${*:1:2}"' ksh foo bar baz
<foo bar>

$ bash -c 'printf "<%s>\n" ${*:1:2}' ksh foo bar baz
<foo>
<bar>
$ bash -c 'printf "<%s>\n" "${*:1:2}"' ksh foo bar baz
<foo bar>

Should zsh align with bash/ksh there.

-- 
Stephane

Re: "${*:offset:length}" and ksh93/bash compatibility

[Mikael Magnusson]

On 3/10/22, Stephane Chazelas <stephane@chazelas.org> wrote:
> Hello,
>
> From workers/28418 I understand the ${param:offset:length} was
> introduced only for compatibility with bash (though the feature
> was initially from ksh93) as zsh already had $param[first,last]
> (long before bash had arrays or ksh93 even existed).
>
> Still, there's a difference with ksh/bash when
> ${*:offset:length} (same for ${array[*]}) is quoted:
>
> $ zsh --emulate ksh -c 'printf "<%s>\n" ${*:1:2}' zsh foo bar baz
> <foo>
> <bar>
> $ zsh --emulate ksh -c 'printf "<%s>\n" "${*:1:2}"' zsh foo bar baz
> <oo>
>
> $ ksh -c 'printf "<%s>\n" ${*:1:2}' ksh foo bar baz
> <foo>
> <bar>
> $ ksh -c 'printf "<%s>\n" "${*:1:2}"' ksh foo bar baz
> <foo bar>
>
> $ bash -c 'printf "<%s>\n" ${*:1:2}' ksh foo bar baz
> <foo>
> <bar>
> $ bash -c 'printf "<%s>\n" "${*:1:2}"' ksh foo bar baz
> <foo bar>
>
> Should zsh align with bash/ksh there.

If we do, it would probably look something like this (does not handle
all cases), unless someone has a better idea...

diff --git i/Src/subst.c w/Src/subst.c
index 465fe970ff..a34ad5f571 100644
--- i/Src/subst.c
+++ w/Src/subst.c
@@ -2718,7 +2718,7 @@ paramsubst(LinkList l, LinkNode n, char **str,
int qt, int pf_flags,
            }
        }
        /* See if this is a reference to the positional parameters. */
-       if (v && v->pm && v->pm->gsu.a == &vararray_gsu &&
+       if (v && v->pm &&
            (char ***)v->pm->u.data == &pparams)
            horrible_offset_hack = 1;
        else
@@ -3441,6 +3441,17 @@ colonsubscript:
                        }
                        if (offset < 0)
                            offset = 0;
+                   } else if (offset > 0 && horrible_offset_hack) {
+                       offset--;
+                       if (length_set && length > 0) {
+                           char **elem;
+                           int newlen = 0;
+                           for (elem = aval; *elem && length; elem++) {
+                               newlen += strlen(*elem) + 1;
+                               length--;
+                           }
+                           length = newlen - 1;
+                       }
                    }
                    given_offset = offset;
                    MB_METACHARINIT();


-- 
Mikael Magnusson

Re: "${*:offset:length}" and ksh93/bash compatibility

[Peter Stephenson]

On 10 March 2022 at 12:48 Mikael Magnusson <mikachu@gmail.com> wrote:
> On 3/10/22, Stephane Chazelas <stephane@chazelas.org> wrote:
> > Should zsh align with bash/ksh there.
> 
> If we do, it would probably look something like this (does not handle
> all cases), unless someone has a better idea...

I think it should change --- the documentation is already clear this is
here to please users of other shells, not native zsh.  Otherwise we end
up with what's (as we say where I come from) neither nowt nor something.

  This  syntax  gives  effects  similar to parameter subscripting in the form
  $name[start,end], but is compatible with other shells; note that both  off‐
  set  and  length  are interpreted differently from the components of a sub‐
  script.

pws

Re: "${*:offset:length}" and ksh93/bash compatibility

[Mikael Magnusson]

On 3/10/22, Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> On 10 March 2022 at 12:48 Mikael Magnusson <mikachu@gmail.com> wrote:
>> On 3/10/22, Stephane Chazelas <stephane@chazelas.org> wrote:
>> > Should zsh align with bash/ksh there.
>>
>> If we do, it would probably look something like this (does not handle
>> all cases), unless someone has a better idea...
>
> I think it should change --- the documentation is already clear this is
> here to please users of other shells, not native zsh.  Otherwise we end
> up with what's (as we say where I come from) neither nowt nor something.
>
>   This  syntax  gives  effects  similar to parameter subscripting in the
> form
>   $name[start,end], but is compatible with other shells; note that both
> off‐
>   set  and  length  are interpreted differently from the components of a
> sub‐
>   script.

I realized that this is not limited to $*,

zsh% a=(one two three); echo "${a[@]:0:2}"
one two
zsh% a=(one two three); echo "${a[*]:0:2}"
on
zsh% bash -c 'a=(one two three); echo "${a[@]:0:2}"'
one two
zsh% bash -c 'a=(one two three); echo "${a[*]:0:2}"'
one two

So perhaps this should be attacked in a different place/way than my
test patch (though i think the horrible offset hack is still needed
for $*:0 specifically (and i actually didn't even handle that in the
patch anyway.).)

-- 
Mikael Magnusson

Re: "${*:offset:length}" and ksh93/bash compatibility

[Bart Schaefer]

Arriving to this a bit late, have been traveling.

On Sun, Mar 13, 2022 at 8:22 AM Mikael Magnusson <mikachu@gmail.com> wrote:
>
> zsh% a=(one two three); echo "${a[*]:0:2}"
> on
> zsh% bash -c 'a=(one two three); echo "${a[*]:0:2}"'
> one two

The [*] subscript in double quotes has always been treated differently
in zsh.  You need [@] or the (@) modifier to preserve array-ness.  I'm
not sure the presence of the :off:len suffix should change that as a
side-effect, at least not in native zsh mode ... and it might be
difficult to "notice" the suffix early enough in the expansion code to
cause that effect anyway.  Particularly without also having the effect
that the end result is not joined into a single string.

Re: "${*:offset:length}" and ksh93/bash compatibility

[Mikael Magnusson]

On 3/13/22, Bart Schaefer <schaefer@brasslantern.com> wrote:
> Arriving to this a bit late, have been traveling.
>
> On Sun, Mar 13, 2022 at 8:22 AM Mikael Magnusson <mikachu@gmail.com> wrote:
>>
>> zsh% a=(one two three); echo "${a[*]:0:2}"
>> on
>> zsh% bash -c 'a=(one two three); echo "${a[*]:0:2}"'
>> one two
>
> The [*] subscript in double quotes has always been treated differently
> in zsh.  You need [@] or the (@) modifier to preserve array-ness.  I'm
> not sure the presence of the :off:len suffix should change that as a
> side-effect, at least not in native zsh mode ... and it might be
> difficult to "notice" the suffix early enough in the expansion code to
> cause that effect anyway.  Particularly without also having the effect
> that the end result is not joined into a single string.

The way it works in bash is it does become a scalar, but the subscript
works on the array elements before joining (just like [1,3] does in
zsh in the same context). I'm reasonably sure that there would be no
difference in the zsh code for handling $*:X or $a[*]:X, other than
having to add the extra faked element for X=0. I think the correct
answer is to always have :off:len work on the array indices if they
exist, regardless of double-quoted-ness... But I also just took a look
at getindex() and don't want to replicate the appropriate parts of
that logic (which would require understanding what any of the parts
do).

I'm also not sure if it matters too much since it's been some years
before anyone even noticed it, I just (somewhat naively) thought that
maybe it wouldn't be too difficult to add...

-- 
Mikael Magnusson

Re: "${*:offset:length}" and ksh93/bash compatibility

[Bart Schaefer]

On Sun, Mar 13, 2022 at 12:23 PM Mikael Magnusson <mikachu@gmail.com> wrote:
>
> The way it works in bash is it does become a scalar, but the subscript
> works on the array elements before joining (just like [1,3] does in
> zsh in the same context).

Ah, OK. You're thinking of "${a[*][1,3]}" and I'm thinking of
"${${a[*]}[1,3]}".  Empirically, :off:len is (currently) implemented
as the latter rather than as the former.

> I also just took a look
> at getindex() and don't want to replicate the appropriate parts of
> that logic (which would require understanding what any of the parts
> do).

Indeed.

Re: "${*:offset:length}" and ksh93/bash compatibility

[Jun T]

> 2022/03/14 0:22, Mikael Magnusson <mikachu@gmail.com> wrote:
> 
> I realized that this is not limited to $*,
> 
> zsh% a=(one two three); echo "${a[@]:0:2}"
> one two
> zsh% a=(one two three); echo "${a[*]:0:2}"
> on

The following patch seems to work for
 "${*:1:2}"   "${@:1:2}"  "$a[@]:0:2}"  "${a[*]:0:2}"

For example (with or without KSH_ARRAYS):

zsh% printf '<%s>\n' "${a[@]:0:2}"
<one>
<two>

If KSH_ARRAYS is not set, "$a:0:2" behaves like "${a[*]:0:2}".

I don't know if there are better/simpler solutions.
'make check' passes, but please do some tests to see if there
are any bad side effects.



diff --git a/Src/subst.c b/Src/subst.c
index 465fe970f..0f98e6ea3 100644
--- a/Src/subst.c
+++ b/Src/subst.c
@@ -1854,6 +1854,12 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags,
      * nested (P) flags.
      */
     int fetch_needed;
+    /*
+     * If an array parameter is quoted but has :offset:length (as in
+     * "${array:off:len}"), we apply :off:len as array index before
+     * joining the array into a string (for compatibility with ksh/bash).
+     */
+    int quoted_array_with_offset = 0;
 
     *s++ = '\0';
     /*
@@ -3377,7 +3383,16 @@ colonsubscript:
 			    return NULL;
 		    }
 		}
-		if (isarr) {
+		/*
+		 * We've got :OFFSET (and :LENGTH).
+		 * If aval is non-NULL but isarr is 0, PARAM is (probably)
+		 * an array but quoted like "${PARAM:OFFSET}". We apply
+		 * :OFFSET as array index (as if it is not quoted). We will
+		 * join them later (search for quoted_array_with_offset).
+		 */
+		if (aval && !isarr)
+		    quoted_array_with_offset = 1;
+		if (isarr || quoted_array_with_offset) {
 		    int alen, count;
 		    char **srcptr, **dstptr, **newarr;
 
@@ -3622,9 +3637,9 @@ colonsubscript:
      * exception is that ${name:-word} and ${name:+word} will have already
      * done any requested splitting of the word value with quoting preserved.
      */
-    if (ssub || spbreak || spsep || sep) {
+    if (ssub || spbreak || spsep || sep || quoted_array_with_offset) {
 	int force_split = !ssub && (spbreak || spsep);
-	if (isarr) {
+	if (isarr || quoted_array_with_offset) {
 	    /* sep non-null here means F or j flag, force join */
 	    if (nojoin == 0 || sep) {
 		val = sepjoin(aval, sep, 1);

Re: "${*:offset:length}" and ksh93/bash compatibility

[Mikael Magnusson]

On 3/16/22, Jun T <takimoto-j@kba.biglobe.ne.jp> wrote:
>
>> 2022/03/14 0:22, Mikael Magnusson <mikachu@gmail.com> wrote:
>>
>> I realized that this is not limited to $*,
>>
>> zsh% a=(one two three); echo "${a[@]:0:2}"
>> one two
>> zsh% a=(one two three); echo "${a[*]:0:2}"
>> on
>
> The following patch seems to work for
>  "${*:1:2}"   "${@:1:2}"  "$a[@]:0:2}"  "${a[*]:0:2}"
>
> For example (with or without KSH_ARRAYS):
>
> zsh% printf '<%s>\n' "${a[@]:0:2}"
> <one>
> <two>
>
> If KSH_ARRAYS is not set, "$a:0:2" behaves like "${a[*]:0:2}".
>
> I don't know if there are better/simpler solutions.
> 'make check' passes, but please do some tests to see if there
> are any bad side effects.

It's certainly a lot simpler than my idea, and also seems to work in
the cases I could come up with. I guess if KSH_ARRAYS is not set, then
there is no expectation of bash compatibility anyway so whatever we do
there is fine.

-- 
Mikael Magnusson

Re: "${*:offset:length}" and ksh93/bash compatibility

[Jun T]

> 2022/03/16 14:57, I wrote:
> 
> The following patch seems to work for
> "${*:1:2}"   "${@:1:2}"  "$a[@]:0:2}"  "${a[*]:0:2}"

This is a test for the patch.


diff --git a/Test/D04parameter.ztst b/Test/D04parameter.ztst
index b6b1f2e33..20f731dcb 100644
--- a/Test/D04parameter.ztst
+++ b/Test/D04parameter.ztst
@@ -1693,6 +1693,30 @@
 >b
 >c
 
+   () {
+     emulate -L sh
+     local a=( one two three )
+     printf '<%s><%s>\n' ${a[*]:0:2}
+     printf '<%s><%s>\n' "${a[*]:0:2}"
+     printf '<%s><%s>\n' ${a[@]:0:2}
+     printf '<%s><%s>\n' "${a[@]:0:2}"
+     printf '<%s><%s>\n' "${a:0:2}"
+     printf '<%s><%s>\n' ${*:1:2}
+     printf '<%s><%s>\n' "${*:1:2}"
+     printf '<%s><%s>\n' ${@:1:2}
+     printf '<%s><%s>\n' "${@:1:2}"
+   } one two three
+0:Bash-style offsets, quoted array
+><one><two>
+><one two><>
+><one><two>
+><one><two>
+><on><>
+><one><two>
+><one two><>
+><one><two>
+><one><two>
+
    printf "%n" '[0]'
 1:Regression test for identifier test
 ?(eval):1: not an identifier: [0]

Re: "${*:offset:length}" and ksh93/bash compatibility

[Mikael Magnusson]

On 3/23/22, Jun T <takimoto-j@kba.biglobe.ne.jp> wrote:
>
>> 2022/03/16 14:57, I wrote:
>>
>> The following patch seems to work for
>> "${*:1:2}"   "${@:1:2}"  "$a[@]:0:2}"  "${a[*]:0:2}"
>
> This is a test for the patch.
>
>
> diff --git a/Test/D04parameter.ztst b/Test/D04parameter.ztst
> index b6b1f2e33..20f731dcb 100644
> --- a/Test/D04parameter.ztst
> +++ b/Test/D04parameter.ztst
> @@ -1693,6 +1693,30 @@
>  >b
>  >c
>
> +   () {
> +     emulate -L sh
> +     local a=( one two three )
> +     printf '<%s><%s>\n' ${a[*]:0:2}
> +     printf '<%s><%s>\n' "${a[*]:0:2}"
> +     printf '<%s><%s>\n' ${a[@]:0:2}
> +     printf '<%s><%s>\n' "${a[@]:0:2}"
> +     printf '<%s><%s>\n' "${a:0:2}"
> +     printf '<%s><%s>\n' ${*:1:2}
> +     printf '<%s><%s>\n' "${*:1:2}"
> +     printf '<%s><%s>\n' ${@:1:2}
> +     printf '<%s><%s>\n' "${@:1:2}"
> +   } one two three
> +0:Bash-style offsets, quoted array
> +><one><two>
> +><one two><>
> +><one><two>
> +><one><two>
> +><on><>
> +><one><two>
> +><one two><>
> +><one><two>
> +><one><two>
> +
>     printf "%n" '[0]'
>  1:Regression test for identifier test
>  ?(eval):1: not an identifier: [0]

I think the test should include
 +     printf '<%s><%s>\n' ${*:0:2}
 +     printf '<%s><%s>\n' "${*:0:2}"
 +     printf '<%s><%s>\n' ${@:0:2}
 +     printf '<%s><%s>\n' "${@:0:2}"
as well, since 0 is handled specially (the horrible offset hack).

-- 
Mikael Magnusson