[BUG] var=$* and var=$@ create array with SHWORDSPLIT and null or unset IFS

[BUG] var=$* and var=$@ create array with SHWORDSPLIT and null or unset IFS

[Martijn Dekker]

In SHWORDSPLIT mode, if IFS is null (no field splitting) or unset
(default fieldsplitting), both var=$* and var=$@ act like var=("$@"), so
turn 'var' into an array.

In native mode, zsh acts correctly (POSIXly), i.e.: all the parameters
get concatenated with no separator, since IFS is null. So this should be
fixed for sh mode.

Note that var="$@" and var="$*" act correctly in any mode. So, to
trigger the bug:
- the quotes must be absent
- IFS must be null (i.e. set and empty) or unset
- SHWORDSPLIT must be on

Here's a little test script.

#! Src/zsh -f
set -- *
for ifs in default null unset; do
	for wordsplit in native sh; do
		print -r -- "--- $ifs IFS, $wordsplit splitting ---"

		case $ifs in
		default) IFS=$' \t\n\00' ;;
		null)    IFS= ;;
		unset)   unset -v IFS ;;
		esac

		case $wordsplit in
		native)  unsetopt shwordsplit ;;
		sh)      setopt shwordsplit ;;
		esac

		for testcmd in 'var=$@' 'var=$*' 'var="$@"' 'var="$*"'; do
			(set -x; eval "$testcmd")
		done
	done
done

You see the bug happen under "--- null IFS, sh splitting ---" and "---
unset IFS, sh splitting ---":

--- null IFS, sh splitting ---
+test.zsh:19> eval 'var=$@'
+(eval):1> var=( Config Doc Etc Makefile Src Test config.cache config.h
config.log config.modules config.modules.sh config.status stamp-h
test.zsh )
+test.zsh:19> eval 'var=$*'
+(eval):1> var=( Config Doc Etc Makefile Src Test config.cache config.h
config.log config.modules config.modules.sh config.status stamp-h
test.zsh )
[...]
--- unset IFS, sh splitting ---
+test.zsh:19> eval 'var=$@'
+(eval):1> var=( Config Doc Etc Makefile Src Test config.cache config.h
config.log config.modules config.modules.sh config.status stamp-h
test.zsh )
+test.zsh:19> eval 'var=$*'
+(eval):1> var=( Config Doc Etc Makefile Src Test config.cache config.h
config.log config.modules config.modules.sh config.status stamp-h
test.zsh )

Thanks,

- M.

Re: [BUG] var=$* and var=$@ create array with SHWORDSPLIT and null or unset IFS

[Martijn Dekker]

Op 17-02-17 om 00:05 schreef Martijn Dekker:
> Here's a little test script.

Figured I might as well save you some work and turn it into a proper zsh
test case instead.

diff --git a/Test/D04parameter.ztst b/Test/D04parameter.ztst
index 9128c3c..ca1dcc2 100644
--- a/Test/D04parameter.ztst
+++ b/Test/D04parameter.ztst
@@ -2056,3 +2056,81 @@
 0:Regression: "-" became Dash in workers/37689, breaking ~- expansion
 *>*
 F:We do not care what $OLDPWD is, as long as it doesn't cause an error
+
+  (
+  set -- one 'two three' four
+  for ifs in default null unset; do
+    for wordsplit in native sh; do
+      print -r -- "--- $ifs IFS, $wordsplit splitting ---"
+      case $ifs in
+      default) IFS=$' \t\n\00' ;;
+      null)    IFS= ;;
+      unset)   unset -v IFS ;;
+      esac
+      case $wordsplit in
+      native)  unsetopt shwordsplit ;;
+      sh)      setopt shwordsplit ;;
+      esac
+      for testcmd in 'var=$@' 'var=$*' 'var="$@"' 'var="$*"'; do
+	print -r -- "> $testcmd"
+        eval "$testcmd"
+	printf '[%s]\n' "${var[@]}"
+      done
+    done
+  done
+  )
+0:Assigning $@, $*, "$@", "$*" to var with various shwordsplit/IFS settings
+>--- default IFS, native splitting ---
+>> var=$@
+>[one two three four]
+>> var=$*
+>[one two three four]
+>> var="$@"
+>[one two three four]
+>> var="$*"
+>[one two three four]
+>--- default IFS, sh splitting ---
+>> var=$@
+>[one two three four]
+>> var=$*
+>[one two three four]
+>> var="$@"
+>[one two three four]
+>> var="$*"
+>[one two three four]
+>--- null IFS, native splitting ---
+>> var=$@
+>[one two three four]
+>> var=$*
+>[onetwo threefour]
+>> var="$@"
+>[one two three four]
+>> var="$*"
+>[onetwo threefour]
+>--- null IFS, sh splitting ---
+>> var=$@
+>[one two three four]
+>> var=$*
+>[onetwo threefour]
+>> var="$@"
+>[one two three four]
+>> var="$*"
+>[onetwo threefour]
+>--- unset IFS, native splitting ---
+>> var=$@
+>[one two three four]
+>> var=$*
+>[one two three four]
+>> var="$@"
+>[one two three four]
+>> var="$*"
+>[one two three four]
+>--- unset IFS, sh splitting ---
+>> var=$@
+>[one two three four]
+>> var=$*
+>[one two three four]
+>> var="$@"
+>[one two three four]
+>> var="$*"
+>[one two three four]

Re: [BUG] var=$* and var=$@ create array with SHWORDSPLIT and null or unset IFS

[Bart Schaefer]

On Feb 17, 12:05am, Martijn Dekker wrote:
}
} In SHWORDSPLIT mode, if IFS is null (no field splitting) or unset
} (default fieldsplitting), both var=$* and var=$@ act like var=("$@"), so
} turn 'var' into an array.

This sounded naggingly familiar, and indeed:

    30299: "$*" was split with SHWORDSPLIT if IFS was empty or unset

Undoing 30299 fixes the bug Martijn is reporting here, but re-introduces
the incorrect splitting behavior.

I haven't gotten any further with it than that.

Re: [BUG] var=$* and var=$@ create array with SHWORDSPLIT and null or unset IFS

[Bart Schaefer]

On Feb 16,  9:14pm, Bart Schaefer wrote:
}
} } In SHWORDSPLIT mode, if IFS is null (no field splitting) or unset
} } (default fieldsplitting), both var=$* and var=$@ act like var=("$@"), so
} } turn 'var' into an array.
} 
} Undoing 30299 fixes the bug Martijn is reporting here, but re-introduces
} the incorrect splitting behavior.

I have a question about one of your test cases.

>--- null IFS, native splitting ---
>> var=$@
>[one two three four]
>> var=$*
>[onetwo threefour]
>> var="$@"
>[one two three four]
>> var="$*"
>[onetwo threefour]

Joining the array for assignment is forced by prefork() passing
PREFORK_SINGLE down to paramsubst().  Joining is supposed to be done on
the first character of $IFS in this case.  When IFS is null (as opposed
to unset), this causes both $@ and $* to join on empty string.

Where is the space supposed to come from when joining $@ here?

As for the reported bug -- in certain circumstances paramsubst() skips
doing joins/splits to avoid other problems with the semantics, which
results in an array being returned to prefork() even though it asked
for a scalar.  Instead of choking on this, prefork() converts to an
array assignment.

So the following fixes this, but still joins on empty string rather
than on space in the case of $@.  There's probably a better way to turn
the LinkList back into an array?  But if this join is done any sooner,
then sorting, rcexpandparam, etc. don't happen properly.  If we can
figure out what to pass to sepjoin() in place of NULL as the second
argument, to answer my question above, all Martijn's tests can pass.

The first bit with ms_flags may be meaningless, I was never sure whether
that needed resetting in any case.  Meat is the last hunk.

diff --git a/Src/subst.c b/Src/subst.c
index 1c2397c..4df53bd 100644
--- a/Src/subst.c
+++ b/Src/subst.c
@@ -3475,7 +3475,6 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags,
 	    if (nojoin == 0 || sep) {
 		val = sepjoin(aval, sep, 1);
 		isarr = 0;
-		ms_flags = 0;
 	    } else if (force_split &&
 		       (spsep || nojoin == 2 || (!ifs && isarr < 0))) {
 		/* Hack to simulate splitting individual elements:
@@ -3485,6 +3484,8 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags,
 		val = sepjoin(aval, (nojoin == 1 ? NULL : spsep), 1);
 		isarr = 0;
 	    }
+	    if (!isarr)
+		ms_flags = 0;
 	}
 	if (force_split && !isarr) {
 	    aval = sepsplit(val, spsep, 0, 1);
@@ -4007,6 +4008,18 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags,
 	    y = dupstring(nulstring);
 	setdata(n, (void *) y);
     }
+    if (isarr && ssub) {
+	/* prefork() wants a scalar, so join no matter what else */
+	LinkNode tn;
+
+	aval = hlinklist2array(l, 0);
+	val = sepjoin(aval, NULL, 1);
+	n = firstnode(l);
+	for (tn = lastnode(l); tn && tn != n; tn = lastnode(l))
+	    uremnode(l, tn);
+	setdata(n, (void *) val);
+	l->list.flags &= ~LF_ARRAY;
+    }
     if (eval)
 	*str = (char *) getdata(n);
 

Re: [BUG] var=$* and var=$@ create array with SHWORDSPLIT and null or unset IFS

[Bart Schaefer]

On Feb 20, 10:58am, Bart Schaefer wrote:
}
} >--- null IFS, native splitting ---
} >> var=$@
} >[one two three four]
} >> var=$*
} >[onetwo threefour]
} >> var="$@"
} >[one two three four]
} >> var="$*"
} >[onetwo threefour]
} 
} Joining the array for assignment is forced by prefork() passing
} PREFORK_SINGLE down to paramsubst().  Joining is supposed to be done on
} the first character of $IFS in this case.  When IFS is null (as opposed
} to unset), this causes both $@ and $* to join on empty string.
} 
} Where is the space supposed to come from when joining $@ here?

Thanks to Chet Ramey for pointing me at
http://austingroupbugs.net/view.php?id=888

Which says (very last block of examples) that var=$@ and var="$@" and
a whole slew of their variations, have unspecified behavior.

Consequently I'm going to take the position that zsh is allowed to join
these with empty string rather than pulling a field separator out of
its proverbial hat, and I will tweak 40565 appropriately and commit.

Re: [BUG] var=$* and var=$@ create array with SHWORDSPLIT and null or unset IFS

[Martijn Dekker]

Op 20-02-17 om 22:20 schreef Bart Schaefer:
> Thanks to Chet Ramey for pointing me at
> http://austingroupbugs.net/view.php?id=888
> 
> Which says (very last block of examples) that var=$@ and var="$@" and
> a whole slew of their variations, have unspecified behavior.

True. I should not have included that in the test case.

It makes sense for var=$@ and var="$@" to be unspecified. In a pure
POSIX context these does not make sense, because POSIX does not have
arrays, and we've already got $* to combine the PPs into a scalar. But
for zsh or bash, which do have arrays, it could actually make sense to
make var=$@ and var="$@" properly equivalent to var=("$@"). It's just
that no shell currently does this, at least not intentionally; zsh's
current behaviour as reported was clearly not consistent.

> Consequently I'm going to take the position that zsh is allowed to join
> these with empty string rather than pulling a field separator out of
> its proverbial hat, and I will tweak 40565 appropriately and commit.

Thanks.

- M.