[bug] $scalar[(pws[\0])1] or $scalar[(pws[á])1] doesn't work

[bug] $scalar[(pws[\0])1] or $scalar[(pws[á])1] doesn't work

[Stephane Chazelas]

From: https://unix.stackexchange.com/questions/724638/subscripted-variable-expansion-on-nul-delimited-words-in-zsh

Note that we're talking of the s[separator] *array subscript*
flag combined with w and p, not the s[separator] parameter
expansion flag here.

$ a=$'foo\0bar'
$ echo $a[(pws[\0])1]
f

Expected "foo".

It seems that's some missing metafication as confirmed by:

$ echo á | hd
00000000  c3 a1 0a                                          |...|
00000003

(0xa1 needing metafication as well)

$ a=fooábar
$ echo $a[(pws[á])1]
fooábar
$ a=fooébar
$ echo $a[(pws[é])1]
foo

$ n=$'\x83\x20' # a metafied NUL
$ a=$'foo\0bar'
$ eval "echo \$a[(pws[$n])1]"
foo

Also note the doc says about the "w" subscript flag:

> w
>    If the parameter subscripted is a scalar then this flag makes
>    subscripting work on words instead of characters.  The default word
>    separator is whitespace.[...]

I find that's not true. By default it seems to split on $IFS
characters (which by default does include space, tab and
newline but also NUL). Also I find empty elements are always
discarded even if $IFS-splitting with non-whitespace characters:


$ a=::foo:bar
$ echo $a[(ws[:])1]
foo
$ IFS=:
$ echo $a[(w)1]
foo

Also, it would be nice if one could do $a[(pws[$var])1] to align
with ${(ps[$var])var}

-- 
Stephane

Re: [bug] $scalar[(pws[\0])1] or $scalar[(pws[á])1] doesn't work

[Stephane Chazelas]

2022-11-13 08:14:52 +0000, Stephane Chazelas:
[...]
> $ a=$'foo\0bar'
> $ echo $a[(pws[\0])1]
> f
> 
> Expected "foo".
> 
> It seems that's some missing metafication as confirmed by:
[...]

It seems the problems are not limited to the s subscript flag.

$ ltrace -e getpwnam zsh -c "u=$'st\u00e9ph\u00e1ne';  echo *(u:\$u:)"
zsh->getpwnam("st\303\251ph\303\203\201ne")                                                       = 0
zsh:1: unknown username 'stépháne'
+++ exited (status 1) +++

See \241 changed to \203\201 in the call to getpwnam().

The error message has the correct name though.

-- 
Stephane

Re: [bug] $scalar[(pws[\0])1] or $scalar[(pws[á])1] doesn't work

[Jun. T]

> 2022/11/13 17:14, Stephane Chazelas <stephane@chazelas.org> wrote:
> 
> $ a=$'foo\0bar'
> $ echo $a[(pws[\0])1]
> f
> 
> Expected "foo".
> 
> It seems that's some missing metafication

Yes. The following seems to work (simple test is included)


diff --git a/Src/params.c b/Src/params.c
index 27ea82298..f1fe38955 100644
--- a/Src/params.c
+++ b/Src/params.c
@@ -1262,7 +1262,6 @@ getarg(char **str, int *inv, Value v, int a2, zlong *w,
     /* first parse any subscription flags */
     if (v->pm && (*s == '(' || *s == Inpar)) {
 	int escapes = 0;
-	int waste;
 	for (s++; *s != ')' && *s != Outpar && s != *str; s++) {
 	    switch (*s) {
 	    case 'r':
@@ -1339,8 +1338,13 @@ getarg(char **str, int *inv, Value v, int a2, zlong *w,
 		sav = *t;
 		*t = '\0';
 		s += arglen;
-		sep = escapes ? getkeystring(s, &waste, GETKEYS_SEP, NULL)
-		    : dupstring(s);
+		if (escapes) {
+		    int len;
+		    sep = getkeystring(s, &len, GETKEYS_SEP, NULL);
+		    sep = metafy(sep, len, META_HREALLOC);
+		}
+		else
+		    sep = dupstring(s);
 		*t = sav;
 		s = t + arglen - 1;
 		break;
diff --git a/Test/D06subscript.ztst b/Test/D06subscript.ztst
index adbd398c4..21127e641 100644
--- a/Test/D06subscript.ztst
+++ b/Test/D06subscript.ztst
@@ -294,3 +294,8 @@ F:Regression test for workers/42297
   [[ ${a[$i]} = ${a[i]} ]]
 0f:Math evaluation of commas in array subscripts
 F:In math, (($i)) should be the same as ((i)), see workers/47748.
+
+  string=$'foo\0bar'
+  echo ${string[(pws:\0:)1]}
+0:Word splitting by NUL
+>foo

Re: [bug] $scalar[(pws[\0])1] or $scalar[(pws[á])1] doesn't work

[Jun. T]

> 2022/11/18 21:25, Stephane Chazelas <stephane@chazelas.org> wrote:
> 
> It seems the problems are not limited to the s subscript flag.
> 
> $ ltrace -e getpwnam zsh -c "u=$'st\u00e9ph\u00e1ne';  echo *(u:\$u:)"
> zsh->getpwnam("st\303\251ph\303\203\201ne")                                                       = 0
> zsh:1: unknown username 'stépháne'
> +++ exited (status 1) +++
> 
> See \241 changed to \203\201 in the call to getpwnam().
> 
> The error message has the correct name though.

I think this is a different problem.
Could you try the following patch?

diff --git a/Src/glob.c b/Src/glob.c
index 400be12d5..66ef8d53d 100644
--- a/Src/glob.c
+++ b/Src/glob.c
@@ -1481,7 +1481,7 @@ zglob(LinkList list, LinkNode np, int nountok)
 			    sav = *tt;
 			    *tt = '\0';
 
-			    if ((pw = getpwnam(s + arglen)))
+			    if ((pw = getpwnam(unmetafy(s + arglen, NULL))))
 				data = pw->pw_uid;
 			    else {
 				zerr("unknown username '%s'", s + arglen);

Re: [bug] $scalar[(pws[\0])1] or $scalar[(pws[á])1] doesn't work

[Jun T]

> 2022/11/29 0:50, Jun. T <takimoto-j@kba.biglobe.ne.jp> wrote:
> 
> Could you try the following patch?

Sorry, it brakes the error message. Does this work?


diff --git a/Src/glob.c b/Src/glob.c
index 400be12d5..490bafc37 100644
--- a/Src/glob.c
+++ b/Src/glob.c
@@ -1481,7 +1481,7 @@ zglob(LinkList list, LinkNode np, int nountok)
 			    sav = *tt;
 			    *tt = '\0';
 
-			    if ((pw = getpwnam(s + arglen)))
+			    if ((pw = getpwnam(unmeta(s + arglen))))
 				data = pw->pw_uid;
 			    else {
 				zerr("unknown username '%s'", s + arglen);