Libtool/zsh quoting problem

Libtool/zsh quoting problem

[David Gómez]

Hi all ;),

>From some time till now i was having problems compiling certaing packages, in
particular those ones that used macros in the command with spaces in it. I
thought it was a libtool problem, but the culprit seems zsh ;).

I was missing the origin of the problem because in another systems with zsh
everything compiled correctly. The difference was that in those machines /bin/sh
pointed to a bash shell (no problem) and in my system /bin/sh points to zsh.

All libtool scripts start with /bin/sh, so everything will work ok if the 
sh symlink is a bash shell.

A example, with the littlecms source (http://www.littlecms.com):

[huma@fargo] [~/lcms-1.15] % make
Making all in src
make[1]: Entering directory `/home/huma/lcms-1.15/src'
if /bin/sh ../libtool --tag=CC --mode=compile gcc -DPACKAGE_NAME=\"lcms\"
-DPACKAGE_TARNAME=\"lcms\" -DPACKAGE_VERSION=\"1.15\" -DPACKAGE_STRING=\"lcms\
1.15\" -DPACKAGE_BUGREPORT=\"\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1
-DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1
-DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1
-DHAVE_DLFCN_H=1 -DSIZEOF_INT=4 -DSIZEOF_UNSIGNED_INT=4 -DSIZEOF_LONG=4
-DSIZEOF_UNSIGNED_LONG=4 -DSIZEOF_LONG_LONG=8 -DSIZEOF_UNSIGNED_LONG_LONG=8
-DHasJPEG=1 -DHasZLIB=1 -DHasTIFF=1 -DHAVE_TIFFCONF_H=1  -I. -I. -I../include
-I../include    -g -O2 -MT cmscnvrt.lo -MD -MP -MF ".deps/cmscnvrt.Tpo" -c -o
cmscnvrt.lo cmscnvrt.c; \ then mv -f ".deps/cmscnvrt.Tpo" ".deps/cmscnvrt.Plo";
else rm -f ".deps/cmscnvrt.Tpo"; exit 1; fi
 gcc -DPACKAGE_NAME=\"lcms\" -DPACKAGE_TARNAME=\"lcms\"
-DPACKAGE_VERSION=\"1.15\" -DPACKAGE_STRING=\"lcms 1.15\"
-DPACKAGE_BUGREPORT=\"\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1
-DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1
-DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1
-DHAVE_DLFCN_H=1 -DSIZEOF_INT=4 -DSIZEOF_UNSIGNED_INT=4 -DSIZEOF_LONG=4
-DSIZEOF_UNSIGNED_LONG=4 -DSIZEOF_LONG_LONG=8 -DSIZEOF_UNSIGNED_LONG_LONG=8
-DHasJPEG=1 -DHasZLIB=1 -DHasTIFF=1 -DHAVE_TIFFCONF_H=1 -I. -I. -I../include
-I../include -g -O2 -MT cmscnvrt.lo -MD -MP -MF .deps/cmscnvrt.Tpo -c
cmscnvrt.c  -fPIC -DPIC -o .libs/cmscnvrt.o gcc: 1.15": No such file or
directory
make[1]: *** [cmscnvrt.lo] Error 1
make[1]: Leaving directory `/home/huma/lcms-1.15/src'
make: *** [all-recursive] Error 1

Where -DPACKAGE_STRING=\"lcms 1.15\" in the gcc command should be quoted
to avoid the error.

I noticed this problem too compiling other packages: Xorg modular and libIDL.

Is there any option in zsh to avoid this problem?

thanks,

-- 
David Gómez                                      Jabber ID: davidge@jabber.org

Re: Libtool/zsh quoting problem

[DervishD]

    Hi David :)

 * David <david@pleyades.net> dixit:
> From some time till now i was having problems compiling certaing
> packages, in particular those ones that used macros in the command
> with spaces in it. I thought it was a libtool problem, but the
> culprit seems zsh ;).

    It's not exactly a "culprit" ;) In fact zsh is behaving
correctly according to the SUS standard, as far as I know. More
below.
 
> A example, with the littlecms source (http://www.littlecms.com):
[Non relevant parts of the command stripped]
> if /bin/sh ../libtool [...] -DPACKAGE_STRING=\"lcms\ 1.15\" 
>  gcc [...] -DPACKAGE_STRING=\"lcms 1.15\" cmscnvrt.c -o .libs/cmscnvrt.o
>  gcc: 1.15": No such file or directory
[...]
> Where -DPACKAGE_STRING=\"lcms 1.15\" in the gcc command should be
> quoted to avoid the error.

    Exactly. If it is not quoted (I mean, with additional double
quotes or something like that), gcc will do the above. The problem is
that libtool may be using some bash-only construct to do the
additional quoting. That's the standard (SUS/POSIX) behaviour: by
default unquoted strings are splitted in spaces.

    But the problem is a bit more subtle ;) Look at the "libtool"
invocation: there, the space is properly quoted with a backslash
that, after that, is lost. The "gcc" invocation no longer has that
backslash.

    This may be a problem in libtool, which is not correctly
propagating that backslash, or in the makefile, which is passing that
backslash to the shell invoking libtool so it is lost (the shell
strips it after processing the command line, so if the same string is
passed, unquoted, to another command (gcc in this case), it is
interpreted as TWO arguments.
 
> Is there any option in zsh to avoid this problem?

    Not that I know of, because I think that SH_WORDSPLIT doesn't
apply here.

    I can investigate (if you want), where the problem lies and why
Bash behaves differently from zsh to see if the culprit is zsh, bash
or libtool. I recently discovered that bash was the culprit because
it violates the standard and I thought that the problem was in zsh
O:)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: Libtool/zsh quoting problem

[David Gómez]

Hi Raúl ;),

On Feb 10 at 09:46:30, DervishD wrote:
>     But the problem is a bit more subtle ;) Look at the "libtool"
> invocation:

I know. Libtool is quoting positional parameters which has
shell metacharacters in them. But it's no working because the
string is already split in that point. I've been looking at the
libtool script and i think the problem is in a 'for arg in' that
it's failing with zsh in doing the right splitting.

> > Is there any option in zsh to avoid this problem?
> 
>     Not that I know of, because I think that SH_WORDSPLIT doesn't
> apply here.

No, i've already tested that option.

I just wanted to know if there was some zsh option that solved
this problem...

Thanks,

-- 
David Gómez                                      Jabber ID: davidge@jabber.org

Re: Libtool/zsh quoting problem

[DervishD]

    Hi David :)

 * David <david@pleyades.net> dixit:
> On Feb 10 at 09:46:30, DervishD wrote:
> >     But the problem is a bit more subtle ;) Look at the "libtool"
> > invocation:
> 
> I know. Libtool is quoting positional parameters which has shell
> metacharacters in them. But it's no working because the string is
> already split in that point. I've been looking at the libtool
> script and i think the problem is in a 'for arg in' that it's
> failing with zsh in doing the right splitting.

    That's weird, because zsh doesn't split arguments at spaces in
arguments to the for loop, unless SH_WORDSPLIT is set.

    Anyway, not doing that splitting is the right thing to do if I
understand correctly the SUS standard :?

    Obviously, Bash doesn't do that splitting no matter if invoked as
/bin/sh or not. The problem is that libtool shouldn't be assuming
that the splitting is not done.
 
> > > Is there any option in zsh to avoid this problem?
> > 
> >     Not that I know of, because I think that SH_WORDSPLIT doesn't
> > apply here.
> 
> No, i've already tested that option.

    Please note that if you just did:

    setopt SH_WORDSPLIT
    make

    that won't work!. The option will be reset as soon as zsh is
invoked again as /bin/sh because it will be running in "sh" emulation
mode (so to say).

    You can try to set SH_WORDSPLIT in your RC files only when zsh is
invoked as /bin/sh, but anyway I think that the behaviour of libtool
is wrong regarding word splitting assumptions. Maybe a standard
expertise in this list can clarify that, but anyway in SUS you can
read that the list given to the "for" loop is expanded (chapter 2 of
"Shell & Utilities", tag 02_09_04_03). Expansion consists on
parameter expansion, command substitution, arithmetic expansion and
field splitting, so the list MUST be expanded on the value of IFS
unless quoted.

    I think that both bash and zsh doesn't do such splitting
internally, but zsh forces such splitting if SH_WORDSPLIT is set.

    Do you want me to take a look at libtool sources? If yes, which
version are you using?
 
    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: Libtool/zsh quoting problem

[David Gómez]

Hi Raúl ;),

On Feb 10 at 12:34:19, DervishD wrote:
> > I know. Libtool is quoting positional parameters which has shell
> > metacharacters in them. But it's no working because the string is
> > already split in that point. I've been looking at the libtool
> > script and i think the problem is in a 'for arg in' that it's
> > failing with zsh in doing the right splitting.
> 
>     That's weird, because zsh doesn't split arguments at spaces in
> arguments to the for loop, unless SH_WORDSPLIT is set.

It's just a guess, i'm not sure if the problem is there.

>     that won't work!. The option will be reset as soon as zsh is
> invoked again as /bin/sh because it will be running in "sh" emulation
> mode (so to say).

When zsh is invoked as /bin/sh, shwordsplit is disabled, so that's
why i though that something went wrong because libtool was receiving
splitted parameters.

It happens with all zsh versions and libtool versions. I tested with
zsh-2.4.0 and zsh-4.3.0-dev-2 and several libtool 1.5.XX versions, from 1.5.6
to 1.5.22.

Thanks again,

-- 
David Gómez                                      Jabber ID: davidge@jabber.org

Re: Libtool/zsh quoting problem

[DervishD]

    Hi David :)

 * David <david@pleyades.net> dixit:
> On Feb 10 at 12:34:19, DervishD wrote:
> > > I know. Libtool is quoting positional parameters which has shell
> > > metacharacters in them. But it's no working because the string is
> > > already split in that point. I've been looking at the libtool
> > > script and i think the problem is in a 'for arg in' that it's
> > > failing with zsh in doing the right splitting.
> > 
> >     That's weird, because zsh doesn't split arguments at spaces in
> > arguments to the for loop, unless SH_WORDSPLIT is set.
> 
> It's just a guess, i'm not sure if the problem is there.

    But you probably are right, because the for loop is one of the
points where the list can be splitted.
 
> >     that won't work!. The option will be reset as soon as zsh is
> > invoked again as /bin/sh because it will be running in "sh" emulation
> > mode (so to say).
> 
> When zsh is invoked as /bin/sh, shwordsplit is disabled, so that's
> why i though that something went wrong because libtool was receiving
> splitted parameters.

    You're having another problem, David, because when invoked as
/bin/sh, SH_WORD_SPLIT must be enabled (well, set), as you can read
in the Zsh manual (sections 3.2 and 15.2.10). If yours have that
option disabled, then there is a problem in your RCs :((
 
> It happens with all zsh versions and libtool versions. I tested
> with zsh-2.4.0 and zsh-4.3.0-dev-2 and several libtool 1.5.XX
> versions, from 1.5.6 to 1.5.22.

    OK, I'll take a look. Have you prepared any kind of test case
with only libtool? Some simple command to reproduce the problem, if
any, so I can avoid compiling a whole package to isolate the bug.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: Libtool/zsh quoting problem

[David Gómez]

Hi Raúl ;),

On Feb 10 at 03:25:21, DervishD wrote:
> > It happens with all zsh versions and libtool versions. I tested
> > with zsh-2.4.0 and zsh-4.3.0-dev-2 and several libtool 1.5.XX
> > versions, from 1.5.6 to 1.5.22.
> 
>     OK, I'll take a look. Have you prepared any kind of test case
> with only libtool? Some simple command to reproduce the problem, if
> any, so I can avoid compiling a whole package to isolate the bug.

littlecms is a small package with no dependencies, that's why i
used it as an example.

cheers,

-- 
David Gómez                                      Jabber ID: davidge@jabber.org

Re: Libtool/zsh quoting problem

[DervishD]

    Hi David :)

 * David <david@pleyades.net> dixit:
> On Feb 10 at 03:25:21, DervishD wrote:
> > > It happens with all zsh versions and libtool versions. I tested
> > > with zsh-2.4.0 and zsh-4.3.0-dev-2 and several libtool 1.5.XX
> > > versions, from 1.5.6 to 1.5.22.
> > 
> >     OK, I'll take a look. Have you prepared any kind of test case
> > with only libtool? Some simple command to reproduce the problem, if
> > any, so I can avoid compiling a whole package to isolate the bug.
> 
> littlecms is a small package with no dependencies, that's why i
> used it as an example.

    Cool, then I'll use it :)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

[SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[DervishD]

    Hi all

 * David <david@pleyades.net> dixit:
> A example, with the littlecms source (http://www.littlecms.com):
[irrelevant lines stripped...]
> [huma@fargo] [~/lcms-1.15] % make
> ../libtool --tag=CC --mode=compile gcc -DPACKAGE_STRING=\"lcms\ 1.15\"
[...]
>  gcc -DPACKAGE_STRING=\"lcms 1.15\" [...]
[...]
>  gcc: 1.15": No such file or directory
[...]
> Where -DPACKAGE_STRING=\"lcms 1.15\" in the gcc command should be quoted
> to avoid the error.

    The problem is that zsh does word splitting in this construct:

    ${1+"$@"}

    This is easy to reproduce with this shell script:

#! /bin/sh

printf -- "------\n"
for ddmyvar
do
    printf -- "\t%s\n" "$ddmyvar"
done
printf -- "ARG is %s\n" "$arg"
printf -- "------\n"

# This is more or less what libtool does.
set variable "datum" ${1+"$@"}

printf -- "------\n"
for ddmyvar
do
    printf -- "\t%s\n" "$ddmyvar"
done
printf -- "ARG is %s\n" "$arg"
printf -- "------\n"

exit 0

    Invoked like this:

    $./test.sh Abroken\ string anotherarg
    ------
	    Abroken string
    	anotherarg
    ARG is 
    ------
    ------
	    variable
    	datum
	    Abroken
    	string
	    anotherarg
    ARG is 
    ------


    I don't know if this can be considered a zsh bug or not. The SUS
standard says that field splitting is done after parameter expansion,
so I think it should be done on the above expansion. Unfortunately,
Bash doesn't seem to do it.

    Anyway, the above construct doesn't look useful to me. I mean, if
$1 is set, we substitute all available arguments. If it is unset, we
don't substitune anything. So, this is equivalent to just
substituting "$@", and zsh doesn't perform word splitting on that...

    I'm thinking about submitting a patch to the libtool people, with
something like this:

    alias -g '${1+"$@"}'='"$@"'

    I know, that's a dirty hack, but I cannot think about any other
fix (except "fixing" zsh itself, although I'm not sure this is a
bug).

    Any suggestion???

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Andrey Borzenkov]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 11 February 2006 12:36, DervishD wrote:
>
>     The problem is that zsh does word splitting in this construct:
>
>     ${1+"$@"}
>

See this thread: <http://www.zsh.org/mla/workers/2002/msg00542.html>; there 
was an attempt to fix it in 16949 but apparently it was incomplete and never 
went in.

>     I don't know if this can be considered a zsh bug or not. The SUS
> standard says that field splitting is done after parameter expansion,
> so I think it should be done on the above expansion. Unfortunately,
> Bash doesn't seem to do it.
>

bash is right and zsh is not.

>     Anyway, the above construct doesn't look useful to me. I mean, if
> $1 is set, we substitute all available arguments. If it is unset, we
> don't substitune anything. So, this is equivalent to just
> substituting "$@", and zsh doesn't perform word splitting on that...
>
>     I'm thinking about submitting a patch to the libtool people, with
> something like this:
>
>     alias -g '${1+"$@"}'='"$@"'
>

The funny part of it is that autoconf includes exactly the above workaround 
since quoted thread; this is also documented in autoconf info.

>     I know, that's a dirty hack, but I cannot think about any other
> fix (except "fixing" zsh itself, although I'm not sure this is a
> bug).
>
>     Any suggestion???
>

Fix a bug in zsh :)

- -andrey
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFD7bq7R6LMutpd94wRAv9pAKDR4e7JsNgsehnc1ROJMudjQpUHJwCgyj1H
ZXCpGBkuMvDaZYHRFUoXty0=
=VLVF
-----END PGP SIGNATURE-----

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[DervishD]

    Hi Andrey :)

 * Andrey Borzenkov <arvidjaar@newmail.ru> dixit:
> On Saturday 11 February 2006 12:36, DervishD wrote:
> >
> >     The problem is that zsh does word splitting in this construct:
> >
> >     ${1+"$@"}
> >
> 
> See this thread:
> <http://www.zsh.org/mla/workers/2002/msg00542.html>; there was an
> attempt to fix it in 16949 but apparently it was incomplete and
> never went in.

    Thanks for the information, Alexey :)))
 
> >     I don't know if this can be considered a zsh bug or not. The SUS
> > standard says that field splitting is done after parameter expansion,
> > so I think it should be done on the above expansion. Unfortunately,
> > Bash doesn't seem to do it.
> 
> bash is right and zsh is not.

    Could you please point me to where the standard says that the
splitting shouldn't be done after that particular expansion? I've
been searching with no success :((( The only thing I've found is that
when a parameter expansion occurs in double quotes, field splitting
shall not be done in the result, except for "@".

    The problem is that libtool doesn't use "${1+... but just $1+...,
without quotes, so the splitting *should* be done. That's why I think
bash is wrong here and zsh is right.

    Anyway, if the people at autoconf used the "alias -g" solution,
libtool people could, too...

> >     I'm thinking about submitting a patch to the libtool people, with
> > something like this:
> >
> >     alias -g '${1+"$@"}'='"$@"'
> >
> The funny part of it is that autoconf includes exactly the above
> workaround since quoted thread; this is also documented in autoconf
> info.

    Oops O:))) The funny part here is that I've studied autoconf code
(and of course its documentation) and I've completely missed this...
I spent almost three hours trying to catch the bug :((((

> >     I know, that's a dirty hack, but I cannot think about any other
> > fix (except "fixing" zsh itself, although I'm not sure this is a
> > bug).
> >
> >     Any suggestion???
> 
> Fix a bug in zsh :)

    Easy to say, difficult to do XD I'm not familiar with the code
and if I try to fix this I'm sure I'm going to break a lot of things,
unfortunately :((((

    Thanks for your help, Alexey :))))

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[David Gómez]

Hi Raúl ;),

On Feb 11 at 12:06:30, DervishD wrote:
>     The problem is that libtool doesn't use "${1+... but just $1+...,
> without quotes, so the splitting *should* be done. That's why I think
> bash is wrong here and zsh is right.
> 
>     Anyway, if the people at autoconf used the "alias -g" solution,
> libtool people could, too...

Thanks a lot for your help, i'd never find out this problem by myself ;).
In fact i'm still trying to understand it :), because for me that ${1+"$@"}
construct is bit weird...

cheers,

-- 
David Gómez                                      Jabber ID: davidge@jabber.org

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[DervishD]

    Hi David :)

 * David <david@pleyades.net> dixit:
> On Feb 11 at 12:06:30, DervishD wrote:
> >     The problem is that libtool doesn't use "${1+... but just $1+...,
> > without quotes, so the splitting *should* be done. That's why I think
> > bash is wrong here and zsh is right.
> > 
> >     Anyway, if the people at autoconf used the "alias -g" solution,
> > libtool people could, too...
> 
> Thanks a lot for your help, i'd never find out this problem by myself ;).

    You're more than welcome. Thanks for reporting the problem.
Unfortunately libtool is heavily used nowadays, so many people is
going to be bitten by this bug if they use Zsh as /bin/sh (as I do).

> In fact i'm still trying to understand it :), because for me that ${1+"$@"}
> construct is bit weird...

    Some shells (ugly, old shells) doesn't treat "$@" (double quoted)
as "nothing", as the SUS standard says. Instead, they treat the
expression above as "" (and empty pair of double quotes), which is
not exactly "nothing". Well, to work around this, you always can use
this: ${1+"$@"}, what will *correctly* substitute "nothing" if no
parameters are available (that is, when $@ is empty). Unfortunately,
zsh does field splitting on the above, but bash doesn't. The libtool
script does heavy use of the above construct, and I think it can be
avoided most of the time, but I NOT going to hack libtool, the code
is a nightmare.

    If the libtool people accept the patch, well, then there is a
possibility. Otherwise... The patch is not very intrusive, and anyway
they are already doing a couple of things under Zsh, so...

    I've prepared the patch and it works with littlecms. Now I have
to check libtool pristine sources ;)

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Andrey Borzenkov]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

You do not need to Cc me, I am reading list every now and then :)

>
>     Could you please point me to where the standard says that the
> splitting shouldn't be done after that particular expansion? I've
> been searching with no success :((( The only thing I've found is that
> when a parameter expansion occurs in double quotes, field splitting
> shall not be done in the result, except for "@".
>
>     The problem is that libtool doesn't use "${1+... but just $1+...,
> without quotes, so the splitting *should* be done. That's why I think
> bash is wrong here and zsh is right.
>

<http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_05_01> 
combined with semantic of ${var+word}: 
<http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_02>

"$@" never does word splitting; and ${var+word} is replaced with result of 
_expansion_ of word. Word in this case is "$@" ...

- -andrey
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFD7dh5R6LMutpd94wRAqpsAKCStkZZpKrvXXxck9mfahpWJZcbqQCfY1+K
ZXvoo+fWKpWLtKKkHTvk7cQ=
=+Mo4
-----END PGP SIGNATURE-----

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[DervishD]

    Hi Andrey :)

 * Andrey Borzenkov <arvidjaar@newmail.ru> dixit:
> You do not need to Cc me, I am reading list every now and then :)

    In fact I'm Cc'ing the list, not you O:) Well, I just told
"reply" to Mutt, and your address end up in the To and the list
address in the Cc headers.

    I've fixed it with this message :)

> >     Could you please point me to where the standard says that the
> > splitting shouldn't be done after that particular expansion? I've
> > been searching with no success :((( The only thing I've found is that
> > when a parameter expansion occurs in double quotes, field splitting
> > shall not be done in the result, except for "@".
> <http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_05_01> 
> combined with semantic of ${var+word}: 
> <http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html#tag_02_06_02>
> 
> "$@" never does word splitting; and ${var+word} is replaced with result of 
> _expansion_ of word. Word in this case is "$@" ...

    OK, I did read it bad: I missed that "separate field" part...
Thanks for the explanation :)))

    So, effectively, zsh is wrong here. It should do field
separation, as dictated by the standard, but each parameter must go
to a separate field...

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[DervishD]

    Hi Andrey :)

 * Andrey Borzenkov <arvidjaar@newmail.ru> dixit:

    Sorry for calling you Alexey (twice!) in my last message O:))

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Wayne Davison]

On Sat, Feb 11, 2006 at 01:21:43PM +0300, Andrey Borzenkov wrote:
> See this thread: <http://www.zsh.org/mla/workers/2002/msg00542.html>; there 
> was an attempt to fix it in 16949 but apparently it was incomplete and never 
> went in.

I just took a look at that code using Peter's old patch as a start, and
the assertion that the quotes have been stripped out is not quite true:
there are still Dnull and Snull characters remaining which indicate that
there used to be a double- or single-quotes present.  This allows us to
check the string.  An overly simple version is this:

--- subst.c	6 Feb 2006 11:57:06 -0000	1.44
+++ subst.c	11 Feb 2006 18:07:23 -0000
@@ -2012,6 +2012,8 @@ paramsubst(LinkList l, LinkNode n, char 
 		 */
 		multsub(&val, (aspar ? NULL : &aval), &isarr, NULL);
 		copied = 1;
+		if (spbreak && (*s == Dnull || *s == Snull))
+		    spbreak = 0;
 	    }
 	    break;
 	case ':':

This will turn off word-splitting if the string started with a single or
a double quote.  This is incomplete because the arg may be something
more complicated (such as: 'one' 'two' 'three four'), but it seems
better than what we have now.

One question I have is what to do about ${=1:"$@"}?  In the patch above,
I chose to let the double quotes override the '='.  If that is not
desired, change the new "if" to only trigger disallow an spbreak of 2:

+		if (spbreak == 1 && (*s == Dnull || *s == Snull))

..wayne..

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[DervishD]

    Hi Wayne :)

 * Wayne Davison <wayned@users.sourceforge.net> dixit:
> On Sat, Feb 11, 2006 at 01:21:43PM +0300, Andrey Borzenkov wrote:
> > See this thread: <http://www.zsh.org/mla/workers/2002/msg00542.html>; there 
> > was an attempt to fix it in 16949 but apparently it was incomplete and never 
> > went in.
> 
> I just took a look at that code using Peter's old patch as a start, and
> the assertion that the quotes have been stripped out is not quite true:
> there are still Dnull and Snull characters remaining which indicate that
> there used to be a double- or single-quotes present.  This allows us to
> check the string.  An overly simple version is this:

    Thanks for the patch, Wayne :)

> One question I have is what to do about ${=1:"$@"}?  In the patch above,
> I chose to let the double quotes override the '='.

    I think this is correct, because anyway using both "=" and double
quotes doesn't make much sense...

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Wayne Davison]

On Sat, Feb 11, 2006 at 07:22:54PM +0100, DervishD wrote:
> I think this is correct, because anyway using both "=" and double
> quotes doesn't make much sense...

Yeah, it may not make a lot of sense, but I've changed my mind on what
I think should be done:  if the user told us to word-split the result
(e.g. ${=foo+">$foo<"}, I now think we should just word-split it.

..wayne..

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Bart Schaefer]

On Feb 11, 10:14am, Wayne Davison wrote:
}
} +		if (spbreak && (*s == Dnull || *s == Snull))
} +		    spbreak = 0;
} 
} This will turn off word-splitting if the string started with a single or
} a double quote.  This is incomplete because the arg may be something
} more complicated (such as: 'one' 'two' 'three four'), but it seems
} better than what we have now.

I experimented with but rejected a similar patch back around the time
of the previous discussion because of what happens with ${1+word "$@"}.
It seemed better to me for the shell to behave consistently even if not
posixly-correctly, and I'm loathe to mask an obvious bug with a less-
obvious one.

However, I wouldn't object to Wayne's patch if others think it's a good
idea.

} One question I have is what to do about ${=1:"$@"}?  In the patch above,
} I chose to let the double quotes override the '='.  If that is not
} desired, change the new "if" to only trigger disallow an spbreak of 2:

I think the latter would be more consistent with other treatment of
${=...}, but again it's not a big deal.

-- 
Bart Schaefer
http://www.well.com/user/barts

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Wayne Davison]

On Sat, Feb 11, 2006 at 07:42:53PM +0000, Bart Schaefer wrote:
> I experimented with but rejected a similar patch back around the time
> of the previous discussion because of what happens with ${1+word "$@"}.

One kluge idea for bourne-shell emulation mode:  automatically add an
invisible global alias for ${1+"@"} (i.e. don't list it if the user asks
for a list of aliases).  We'd also want to make sure that we allow the
user to turn it into a visible global alias if they manually add one.
This would just serve to improve our bourne-shell emulation without
trying to fix the underlying problem.

..wayne..

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Peter Stephenson]

Wayne Davison wrote:
> One kluge idea for bourne-shell emulation mode:  automatically add an
> invisible global alias for ${1+"@"} (i.e. don't list it if the user asks
> for a list of aliases).  We'd also want to make sure that we allow the
> user to turn it into a visible global alias if they manually add one.
> This would just serve to improve our bourne-shell emulation without
> trying to fix the underlying problem.

I'm sure it would have to be invisible...  an extra global alias no one
is expecting is neither here nor there, as far as I can see, but a
random invisible change in syntax (applying only to a single form of
many possible variants) is.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page still at http://www.pwstephenson.fsnet.co.uk/

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Peter Stephenson]

On Sun, 2006-02-12 at 20:28 +0000, Peter Stephenson wrote:
> Wayne Davison wrote:
> > One kluge idea for bourne-shell emulation mode:  automatically add an
> > invisible global alias for ${1+"@"} (i.e. don't list it if the user asks
> > for a list of aliases).  We'd also want to make sure that we allow the
> > user to turn it into a visible global alias if they manually add one.
> > This would just serve to improve our bourne-shell emulation without
> > trying to fix the underlying problem.
> 
> I'm sure it would have to be invisible...  an extra global alias no one
> is expecting is neither here nor there, as far as I can see, but a
> random invisible change in syntax (applying only to a single form of
> many possible variants) is.

I completely reversed the sense of the first sentence, as the rest
should make clear: I meant I'm not sure it would have to be invisible.

pws




To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Andrey Borzenkov]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 11 February 2006 21:14, Wayne Davison wrote:
> On Sat, Feb 11, 2006 at 01:21:43PM +0300, Andrey Borzenkov wrote:
> > See this thread: <http://www.zsh.org/mla/workers/2002/msg00542.html>;
> > there was an attempt to fix it in 16949 but apparently it was incomplete
> > and never went in.
>
> I just took a look at that code using Peter's old patch as a start, and
> the assertion that the quotes have been stripped out is not quite true:
> there are still Dnull and Snull characters remaining which indicate that
> there used to be a double- or single-quotes present.  This allows us to
> check the string.  An overly simple version is this:
>
> --- subst.c	6 Feb 2006 11:57:06 -0000	1.44
> +++ subst.c	11 Feb 2006 18:07:23 -0000
> @@ -2012,6 +2012,8 @@ paramsubst(LinkList l, LinkNode n, char
>  		 */
>  		multsub(&val, (aspar ? NULL : &aval), &isarr, NULL);
>  		copied = 1;
> +		if (spbreak && (*s == Dnull || *s == Snull))
> +		    spbreak = 0;
>  	    }
>  	    break;
>  	case ':':
>
> This will turn off word-splitting if the string started with a single or
> a double quote.  This is incomplete because the arg may be something
> more complicated (such as: 'one' 'two' 'three four'), but it seems
> better than what we have now.
>

I am not at all sure that the above is valid in POSIX shell. The word in 
${var+word} is a token, and token is delimited by the first blank; so while 
you can do ${var+$(any command with blank)} (because command substitution 
forms token by itself) you apparently cannot do ${var+foo bar baz}.

That's said, bash accepts the above just fine and correctly preserves quoting:

bash-3.1$ args()
> {
> while [[ $# -gt 0 ]]; do
> echo "$1"
> shift
> done
> }
bash-3.1$ args ${var+"foo bar" baz}
foo bar
baz

So assuming my interpretation is correct, we have two options:

- -  turn the above on only is sh (POSIX?) emulation mode on assumption that 
multiple words on the right side are invalid anyway

- - attempt to fix it to be compatible with bash i.e. correctly preserve 
individual quoted words.

Is it possible to recursively call expansion on rhs word? According to SUS: 
"word shall be subjected to tilde expansion, parameter expansion, command 
substitution, and arithmetic expansion." so recursively calling expansion 
looks the right way to go.

> One question I have is what to do about ${=1:"$@"}?  In the patch above,
> I chose to let the double quotes override the '='.  If that is not
> desired, change the new "if" to only trigger disallow an spbreak of 2:
>
> +		if (spbreak == 1 && (*s == Dnull || *s == Snull))
>
As long as it is documented ...

- -andrey
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFD7ue/R6LMutpd94wRAs4KAJ9MWbzrm9d3KLWMUesXus3PMNliAACeM5Zo
zFEwjjuV8xmtoEGbr7agYuc=
=3Tbz
-----END PGP SIGNATURE-----

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Andrey Borzenkov]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 12 February 2006 10:46, Andrey Borzenkov wrote:
> > One question I have is what to do about ${=1:"$@"}?  In the patch above,
> > I chose to let the double quotes override the '='.  If that is not
> > desired, change the new "if" to only trigger disallow an spbreak of 2:
> >
> > +		if (spbreak == 1 && (*s == Dnull || *s == Snull))
>
> As long as it is documented ...
>

And BTW this will be consistent with manual: "Perform word splitting using the 
rules for SH_WORD_SPLIT during the evaluation of SPEC, but regardless of 
whether the parameter appears in double quotes  This forces parameter 
expansions to be split into separate words before substitution, using IFS as 
a delimiter." So it is already documented.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFD7umjR6LMutpd94wRAhUnAJ4nPVOQ+SUsZEvbrpPm451F4O1nPwCfYrnM
pIRowIN/pGHEAHESHEQN9Lg=
=4EYQ
-----END PGP SIGNATURE-----

Re: [SOLVED] Libtool/zsh quoting problem: a zsh... bug?

[Peter Stephenson]

Wayne Davison wrote:
> I just took a look at that code using Peter's old patch as a start, and
> the assertion that the quotes have been stripped out is not quite true:
> there are still Dnull and Snull characters remaining which indicate that
> there used to be a double- or single-quotes present.  This allows us to
> check the string.  An overly simple version is this:
> 
> --- subst.c	6 Feb 2006 11:57:06 -0000	1.44
> +++ subst.c	11 Feb 2006 18:07:23 -0000
> @@ -2012,6 +2012,8 @@ paramsubst(LinkList l, LinkNode n, char 
>  		 */
>  		multsub(&val, (aspar ? NULL : &aval), &isarr, NULL);
>  		copied = 1;
> +		if (spbreak && (*s == Dnull || *s == Snull))
> +		    spbreak = 0;
>  	    }
>  	    break;
>  	case ':':
> 
> This will turn off word-splitting if the string started with a single or
> a double quote.  This is incomplete because the arg may be something
> more complicated (such as: 'one' 'two' 'three four'), but it seems
> better than what we have now.

Frankly, it's such a mess at the moment that I think any practical
improvement is a good thing.  However, I think a comment would be a good
idea.  (I've already covered paramsubst() with ranty comments about the
mess but never worked out what to do about it.)

> One question I have is what to do about ${=1:"$@"}?  In the patch above,
> I chose to let the double quotes override the '='.  If that is not
> desired, change the new "if" to only trigger disallow an spbreak of 2:
> 
> +		if (spbreak == 1 && (*s == Dnull || *s == Snull))

As pointed out elsewhere, the ${1+"$@"} is to work around limitations of
old Bourne shells and this sort of thing isn't that natural in native
zsh, which is the only place we have to worry about the = syntax.  I
would probably expect the splitting to take place, but I'm not that
fussed.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page still at http://www.pwstephenson.fsnet.co.uk/

PATCH: fixing ${1+"$@"} when word-splitting

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

On Sun, Feb 12, 2006 at 08:26:17PM +0000, Peter Stephenson wrote:
> Frankly, it's such a mess at the moment that I think any practical
> improvement is a good thing.

OK.  Here's something even better.  This appears to make us totally
compatible with bash.  I did the following:

I added an extra arg to multsub() that lets the caller request that the
string be split on unquoted whitespace (as defined by IFS) before being
parsed.  The code that handles ${...+...} uses this new flag, and then
disables spbreak to prevent any further splitting.  While I was at it,
I fixed the comment for multsub() (it needed to be updated for when it
returns an array vs a string).

Because the docs say that using ${=foo} splits like SH_WORD_SPLIT, I
made an explicit '=' work the same way (yeah, I'm waffling on this
issue).  For example:

  % set 1 '2 3 4' 5 '6 7'
  % for w in ${=1+"$@" 'extra words' 'and such here'}; echo $w
  1
  2 3 4
  5
  6 7
  extra words
  and such here
  % for w in ${==1+more "$@" 'extra words' 'and such here'}; echo $w
  more 1
  2 3 4
  5
  6 7 extra words and such here

My checking for quoted whitespace currently honors Dnull, Snull, Tick,
Inpar, and Outpar.  Is that everything?

Attached is the patch (which also includes the good changes from my last
email).  Let me know how you like it.

..wayne..

[-- Attachment #2: splitting.patch --]
[-- Type: text/plain, Size: 6363 bytes --]

--- Src/subst.c	6 Feb 2006 11:57:06 -0000	1.44
+++ Src/subst.c	13 Feb 2006 10:39:31 -0000
@@ -295,29 +295,76 @@ singsub(char **s)
     DPUTS(nonempty(&foo), "BUG: singsub() produced more than one word!");
 }
 
-/* Perform substitution on a single word. Unlike with singsub, the      *
- * result can have more than one word. A single word result is stored   *
- * in *s and *isarr is set to zero; otherwise *isarr is set to 1 and    *
- * the result is stored in *a. If `a' is zero a multiple word result is *
- * joined using sep or the IFS parameter if sep is zero and the result  *
- * is returned in *s.  The return value is true iff the expansion       *
- * resulted in an empty list.                                           *
- * The mult_isarr variable is used by paramsubst() to tell if it yields *
- * an array.                                                            */
+/* Perform substitution on a single word, *s. Unlike with singsub(), the
+ * result can be more than one word. If split is non-zero, the string is
+ * first word-split using IFS, but only for non-quoted whitespace (as
+ * indicated by Snull and Dnull chars).
+ *
+ * If "a" was non-NULL, the resulting string(s) are stored in *a (even for a
+ * 1-element array) and *isarr is set to 1.  Otherwise, *isarr is set to 0,
+ * and the result is stored in *s, with multiple elements joined together
+ * using sep (which can be NULL to use IFS).  The return value is true iff
+ * the expansion resulted in an empty list.
+ *
+ * The mult_isarr variable is used by paramsubst() to tell if it yields an
+ * array. */
 
 /**/
 static int mult_isarr;
 
 /**/
 static int
-multsub(char **s, char ***a, int *isarr, UNUSED(char *sep))
+multsub(char **s, int split, char ***a, int *isarr, char *sep)
 {
     int l, omi = mult_isarr;
-    char **r, **p;
+    char **r, **p, *x = *s;
     local_list1(foo);
 
     mult_isarr = 0;
-    init_list1(foo, *s);
+
+    if (split) {
+	for ( ; *x; x += l+1) {
+	    char c = (l = *x == Meta) ? x[1] ^ 32 : *x;
+	    if (!iwsep(c))
+		break;
+	}
+    }
+
+    init_list1(foo, x);
+
+    if (split) {
+	LinkNode n = firstnode(&foo);
+	int inq = 0, inp = 0;
+	for ( ; *x; x += l+1) {
+	    char c = (l = *x == Meta) ? x[1] ^ 32 : *x;
+	    if (!inq && !inp && isep(c)) {
+		*x++ = '\0';
+		for (x += l; *x; x += l+1) {
+		    c = (l = *x == Meta) ? x[1] ^ 32 : *x;
+		    if (!isep(c))
+			break;
+		}
+		if (!*x)
+		    break;
+		insertlinknode(&foo, n, (void *)x), incnode(n);
+	    }
+	    switch (c) {
+	    case Dnull:
+	    case Snull:
+	    case Tick:
+		/* These always occur in unnested pairs. */
+		inq = !inq;
+		break;
+	    case Inpar:
+		inp++;
+		break;
+	    case Outpar:
+		inp--;
+		break;
+	    }
+	}
+    }
+
     prefork(&foo, 0);
     if (errflag) {
 	if (isarr)
@@ -325,7 +372,7 @@ multsub(char **s, char ***a, int *isarr,
 	mult_isarr = omi;
 	return 0;
     }
-    if ((l = countlinknodes(&foo))) {
+    if ((l = countlinknodes(&foo)) > (a ? 0 : 1)) {
 	p = r = hcalloc((l + 1) * sizeof(char*));
 	while (nonempty(&foo))
 	    *p++ = (char *)ugetnode(&foo);
@@ -345,7 +392,7 @@ multsub(char **s, char ***a, int *isarr,
 	    mult_isarr = omi;
 	    return 0;
 	}
-	*s = sepjoin(r, NULL, 1);
+	*s = sepjoin(r, sep, 1);
 	mult_isarr = omi;
 	if (isarr)
 	    *isarr = 0;
@@ -1457,7 +1504,7 @@ paramsubst(LinkList l, LinkNode n, char 
 	 * remove the aspar test and extract a value from an array, if
 	 * necessary, when we handle (P) lower down.
 	 */
-	if (multsub(&val, (aspar ? NULL : &aval), &isarr, NULL) && quoted) {
+	if (multsub(&val, 0, (aspar ? NULL : &aval), &isarr, NULL) && quoted) {
 	    /* Empty quoted string --- treat as null string, not elided */
 	    isarr = -1;
 	    aval = (char **) hcalloc(sizeof(char *));
@@ -1993,25 +2040,16 @@ paramsubst(LinkList l, LinkNode n, char 
 	case '-':
 	    if (vunset) {
 		val = dupstring(s);
-		/*
-		 * This is not good enough for sh emulation!  Sh would
-		 * split unquoted substrings, yet not split quoted ones
-		 * (except according to $@ rules); but this leaves the
-		 * unquoted substrings unsplit, and other code below
-		 * for spbreak splits even within the quoted substrings.
-		 *
-		 * TODO: I think multsub needs to be told enough to
-		 * decide about splitting with spbreak at this point
-		 * (and equally in the `=' handler below).  Then
-		 * we can turn off spbreak to avoid the join & split
-		 * nastiness later.
-		 *
-		 * What we really want to do is make this look as
-		 * if it were the result of an assignment from
-		 * the same value, taking account of quoting.
-		 */
-		multsub(&val, (aspar ? NULL : &aval), &isarr, NULL);
+		/* If word-splitting is enabled, we ask multsub() to split
+		 * the substituted string at unquoted whitespace.  Then, we
+		 * turn off spbreak so that no further splitting occurs.
+		 * This allows a construct such as ${1+"$@"} to correctly
+		 * keep its array splits, and weird constructs such as
+		 * ${str+"one two" "3 2 1" foo "$str"} to only be split
+		 * at the unquoted spaces. */
+		multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
 		copied = 1;
+		spbreak = 0;
 	    }
 	    break;
 	case ':':
@@ -2029,7 +2067,6 @@ paramsubst(LinkList l, LinkNode n, char 
 
 		*idend = '\0';
 		val = dupstring(s);
-		isarr = 0;
 		/*
 		 * TODO: this is one of those places where I don't
 		 * think we want to do the joining until later on.
@@ -2037,9 +2074,9 @@ paramsubst(LinkList l, LinkNode n, char 
 		 * point and unset them.
 		 */
 		if (spsep || spbreak || !arrasg)
-		    multsub(&val, NULL, NULL, sep);
+		    multsub(&val, 0, NULL, &isarr, NULL);
 		else
-		    multsub(&val, &aval, &isarr, NULL);
+		    multsub(&val, 0, &aval, &isarr, NULL);
 		if (arrasg) {
 		    /*
 		     * This is an array assignment in a context
--- Test/D04parameter.ztst	11 Oct 2005 16:48:06 -0000	1.13
+++ Test/D04parameter.ztst	13 Feb 2006 10:23:51 -0000
@@ -547,20 +547,16 @@
 >shell
 >again
 
-  set If "this test fails" maybe "we have finally fixed" the shell
+  set Make sure "this test keeps" on "preserving all" quoted whitespace
   print -l ${=1+"$@"}
 0:Regression test of unfixed ${=1+"$@"} bug
->If
->this
->test
->fails
->maybe
->we
->have
->finally
->fixed
->the
->shell
+>Make
+>sure
+>this test keeps
+>on
+>preserving all
+>quoted
+>whitespace
 
   unset SHLVL
   (( SHLVL++ ))

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Peter Stephenson]

Wayne Davison wrote:
> --LyciRD1jyfeSSjG0
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> 
> On Sun, Feb 12, 2006 at 08:26:17PM +0000, Peter Stephenson wrote:
> > Frankly, it's such a mess at the moment that I think any practical
> > improvement is a good thing.
> 
> OK.  Here's something even better.  This appears to make us totally
> compatible with bash.

This looks extremelly promising (particularly since it still passes all
the tests as modified) but there is one glitch which turned up from the
function is-at-least, which is called liberally from my .zshrc.
It traces down to:

fn() {
  local IFS=.-
  print -l ${=1:-1.2}
}

With no arguments, this used to give

1
2

but now gives

1.2

With 1.2 passed as an argument instead it always gives the first
behaviour.  I suspect this is the wrong behaviour since bash gives me:

$ fn() { local IFS=.-; echo ${1:-1.2}; }
$ fn
1 2

which suggests the argument should be split (as it's unquoted).
(Naively, I would also expect an unquoted 1.2 to be split in the same way
whether it came from the positional parameter or the default.)

Anyway, I expect that's yet another complexity.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Wayne Davison]

On Mon, Feb 13, 2006 at 11:34:38AM +0000, Peter Stephenson wrote:
>   local IFS=.-
>   print -l ${=1:-1.2}
[...]
> but now gives
> 1.2

This is due to the multi_isarr weirdness:  when it is 0, multsub()
returns the resulting array joined with $IFS[0], which is what was
occurring here.  I added this line to fix the problem:

--- Src/subst.c	2006-02-13 09:25:18.000000000 -0800
+++ Src/subst.c	2006-02-13 09:30:33.000000000 -0800
@@ -347,6 +347,7 @@
 		if (!*x)
 		    break;
 		insertlinknode(&foo, n, (void *)x), incnode(n);
+		mult_isarr = 1;
 	    }
 	    switch (c) {
 	    case Dnull:

This makes sure that when we split the string into multiple pieces
before parsing it, it will be returned as an array.

..wayne..

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Peter Stephenson]

Wayne Davison wrote:
> On Mon, Feb 13, 2006 at 11:34:38AM +0000, Peter Stephenson wrote:
> >   local IFS=.-
> >   print -l ${=1:-1.2}
> [...]
> > but now gives
> > 1.2
> 
> This is due to the multi_isarr weirdness:  when it is 0, multsub()
> returns the resulting array joined with $IFS[0], which is what was
> occurring here.  I added this line to fix the problem:

OK, thanks, that example now works, but unfortunately the saga
continues... this still fails:

% fn() { local IFS=.-; print -l ${=1:-$ZSH_VERSION}; }
% fn
4.3.0-dev-3

which should also be split since $ZSH_VERSION isn't quoted.  (This one's
from is-at-least, too:  it seems running "is-at-least 4.0" is a good test
of word splitting.)  I presume this is due to the truly horrible
argument passing back from the lower levels.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

On Mon, Feb 13, 2006 at 06:08:57PM +0000, Peter Stephenson wrote:
> unfortunately the saga continues...

:-)  Yup.  Thanks for the extensive testing!

> % fn() { local IFS=.-; print -l ${=1:-$ZSH_VERSION}; }
> % fn
> 4.3.0-dev-3

The attached patch fixes this by temporarily setting SHWORDSPLIT
during the multsub() call when '=' was specified.  I also tested
that this change works properly with ${=1:-"$ZSH_VERSION"} (i.e.
giving the output above).

..wayne..

[-- Attachment #2: more-splitting.diff --]
[-- Type: text/plain, Size: 696 bytes --]

--- subst.c	2006-02-13 09:53:12.000000000 -0800
+++ subst.c	2006-02-13 10:50:52.000000000 -0800
@@ -2040,6 +2040,7 @@
 	/* Fall Through! */
 	case '-':
 	    if (vunset) {
+		int ws = opts[SHWORDSPLIT];
 		val = dupstring(s);
 		/* If word-splitting is enabled, we ask multsub() to split
 		 * the substituted string at unquoted whitespace.  Then, we
@@ -2048,7 +2049,9 @@
 		 * keep its array splits, and weird constructs such as
 		 * ${str+"one two" "3 2 1" foo "$str"} to only be split
 		 * at the unquoted spaces. */
+		opts[SHWORDSPLIT] = spbreak;
 		multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
+		opts[SHWORDSPLIT] = ws;
 		copied = 1;
 		spbreak = 0;
 	    }

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

Attached is my current version of the patch relative to CVS.  This has
some comment changes that weren't in my prior patches, and I changed one
other line slightly, mainly to make it clearer, but also a little more
optimal -- this one:

+    if ((l = countlinknodes(&foo)) > 1 || (a && mult_isarr)) {

..wayne..

[-- Attachment #2: multsub.diff --]
[-- Type: text/plain, Size: 6137 bytes --]

--- Src/subst.c	6 Feb 2006 11:57:06 -0000	1.44
+++ Src/subst.c	13 Feb 2006 19:20:45 -0000
@@ -295,29 +295,79 @@ singsub(char **s)
     DPUTS(nonempty(&foo), "BUG: singsub() produced more than one word!");
 }
 
-/* Perform substitution on a single word. Unlike with singsub, the      *
- * result can have more than one word. A single word result is stored   *
- * in *s and *isarr is set to zero; otherwise *isarr is set to 1 and    *
- * the result is stored in *a. If `a' is zero a multiple word result is *
- * joined using sep or the IFS parameter if sep is zero and the result  *
- * is returned in *s.  The return value is true iff the expansion       *
- * resulted in an empty list.                                           *
- * The mult_isarr variable is used by paramsubst() to tell if it yields *
- * an array.                                                            */
+/* Perform substitution on a single word, *s. Unlike with singsub(), the
+ * result can be more than one word. If split is non-zero, the string is
+ * first word-split using IFS, but only for non-quoted "whitespace" (as
+ * indicated by Dnull, Snull, Tick, Inpar, and Outpar).
+ *
+ * If arg "a" was non-NULL _and_ the parsing set mult_isarr, the resulting
+ * strings are stored in *a (even for a 1-element array) and *isarr is set
+ * to 1.  Otherwise, *isarr is set to 0, and the result is stored in *s,
+ * with any necessary joining of multiple elements using sep (which can be
+ * NULL to use IFS).  The return value is true iff the expansion resulted
+ * in an empty list.
+ *
+ * The mult_isarr variable is used by paramsubst() to tell us if the
+ * substitutions yielded an array, but we will also set it if we split *s
+ * into multiple items (since that also yields an array). */
 
 /**/
 static int mult_isarr;
 
 /**/
 static int
-multsub(char **s, char ***a, int *isarr, UNUSED(char *sep))
+multsub(char **s, int split, char ***a, int *isarr, char *sep)
 {
     int l, omi = mult_isarr;
-    char **r, **p;
+    char **r, **p, *x = *s;
     local_list1(foo);
 
     mult_isarr = 0;
-    init_list1(foo, *s);
+
+    if (split) {
+	for ( ; *x; x += l+1) {
+	    char c = (l = *x == Meta) ? x[1] ^ 32 : *x;
+	    if (!iwsep(c))
+		break;
+	}
+    }
+
+    init_list1(foo, x);
+
+    if (split) {
+	LinkNode n = firstnode(&foo);
+	int inq = 0, inp = 0;
+	for ( ; *x; x += l+1) {
+	    char c = (l = *x == Meta) ? x[1] ^ 32 : *x;
+	    if (!inq && !inp && isep(c)) {
+		*x++ = '\0';
+		for (x += l; *x; x += l+1) {
+		    c = (l = *x == Meta) ? x[1] ^ 32 : *x;
+		    if (!isep(c))
+			break;
+		}
+		if (!*x)
+		    break;
+		insertlinknode(&foo, n, (void *)x), incnode(n);
+		mult_isarr = 1;
+	    }
+	    switch (c) {
+	    case Dnull:
+	    case Snull:
+	    case Tick:
+		/* These always occur in unnested pairs. */
+		inq = !inq;
+		break;
+	    case Inpar:
+		inp++;
+		break;
+	    case Outpar:
+		inp--;
+		break;
+	    }
+	}
+    }
+
     prefork(&foo, 0);
     if (errflag) {
 	if (isarr)
@@ -325,7 +375,7 @@ multsub(char **s, char ***a, int *isarr,
 	mult_isarr = omi;
 	return 0;
     }
-    if ((l = countlinknodes(&foo))) {
+    if ((l = countlinknodes(&foo)) > 1 || (a && mult_isarr)) {
 	p = r = hcalloc((l + 1) * sizeof(char*));
 	while (nonempty(&foo))
 	    *p++ = (char *)ugetnode(&foo);
@@ -345,7 +395,7 @@ multsub(char **s, char ***a, int *isarr,
 	    mult_isarr = omi;
 	    return 0;
 	}
-	*s = sepjoin(r, NULL, 1);
+	*s = sepjoin(r, sep, 1);
 	mult_isarr = omi;
 	if (isarr)
 	    *isarr = 0;
@@ -1457,7 +1507,7 @@ paramsubst(LinkList l, LinkNode n, char 
 	 * remove the aspar test and extract a value from an array, if
 	 * necessary, when we handle (P) lower down.
 	 */
-	if (multsub(&val, (aspar ? NULL : &aval), &isarr, NULL) && quoted) {
+	if (multsub(&val, 0, (aspar ? NULL : &aval), &isarr, NULL) && quoted) {
 	    /* Empty quoted string --- treat as null string, not elided */
 	    isarr = -1;
 	    aval = (char **) hcalloc(sizeof(char *));
@@ -1992,26 +2042,20 @@ paramsubst(LinkList l, LinkNode n, char 
 	/* Fall Through! */
 	case '-':
 	    if (vunset) {
+		int ws = opts[SHWORDSPLIT];
 		val = dupstring(s);
-		/*
-		 * This is not good enough for sh emulation!  Sh would
-		 * split unquoted substrings, yet not split quoted ones
-		 * (except according to $@ rules); but this leaves the
-		 * unquoted substrings unsplit, and other code below
-		 * for spbreak splits even within the quoted substrings.
-		 *
-		 * TODO: I think multsub needs to be told enough to
-		 * decide about splitting with spbreak at this point
-		 * (and equally in the `=' handler below).  Then
-		 * we can turn off spbreak to avoid the join & split
-		 * nastiness later.
-		 *
-		 * What we really want to do is make this look as
-		 * if it were the result of an assignment from
-		 * the same value, taking account of quoting.
-		 */
-		multsub(&val, (aspar ? NULL : &aval), &isarr, NULL);
+		/* If word-splitting is enabled, we ask multsub() to split
+		 * the substituted string at unquoted whitespace.  Then, we
+		 * turn off spbreak so that no further splitting occurs.
+		 * This allows a construct such as ${1+"$@"} to correctly
+		 * keep its array splits, and weird constructs such as
+		 * ${str+"one two" "3 2 1" foo "$str"} to only be split
+		 * at the unquoted spaces. */
+		opts[SHWORDSPLIT] = spbreak;
+		multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
+		opts[SHWORDSPLIT] = ws;
 		copied = 1;
+		spbreak = 0;
 	    }
 	    break;
 	case ':':
@@ -2029,7 +2073,6 @@ paramsubst(LinkList l, LinkNode n, char 
 
 		*idend = '\0';
 		val = dupstring(s);
-		isarr = 0;
 		/*
 		 * TODO: this is one of those places where I don't
 		 * think we want to do the joining until later on.
@@ -2037,9 +2080,9 @@ paramsubst(LinkList l, LinkNode n, char 
 		 * point and unset them.
 		 */
 		if (spsep || spbreak || !arrasg)
-		    multsub(&val, NULL, NULL, sep);
+		    multsub(&val, 0, NULL, &isarr, NULL);
 		else
-		    multsub(&val, &aval, &isarr, NULL);
+		    multsub(&val, 0, &aval, &isarr, NULL);
 		if (arrasg) {
 		    /*
 		     * This is an array assignment in a context

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Peter Stephenson]

Wayne Davison wrote:
> > % fn() { local IFS=.-; print -l ${=1:-$ZSH_VERSION}; }
> > % fn
> > 4.3.0-dev-3
> 
> The attached patch fixes this by temporarily setting SHWORDSPLIT
> during the multsub() call when '=' was specified.  I also tested
> that this change works properly with ${=1:-"$ZSH_VERSION"} (i.e.
> giving the output above).

is-at-least is now happy.

Here's an add-on test that handles a lot of the possibilities.  I think
the results are all correct.

--- Test/D04parameter.ztst.old	2006-02-13 19:21:32.000000000 +0000
+++ Test/D04parameter.ztst	2006-02-13 19:31:10.000000000 +0000
@@ -549,7 +549,7 @@
 
   set Make sure "this test keeps" on "preserving all" quoted whitespace
   print -l ${=1+"$@"}
-0:Regression test of unfixed ${=1+"$@"} bug
+0:Regression test of ${=1+"$@"} bug
 >Make
 >sure
 >this test keeps
@@ -558,6 +558,103 @@
 >quoted
 >whitespace
 
+  splitfn() {
+    local IFS=.-
+    local foo=1-2.3-4
+    #
+    print "Called with argument '$1'"
+    print "No quotes"
+    print -l ${=1:-1-2.3-4} ${=1:-$foo}
+    print "With quotes on default argument only"
+    print -l ${=1:-"1-2.3-4"} ${=1:-"$foo"}
+  }
+  print 'Using "="'
+  splitfn
+  splitfn 5.6-7.8
+  #
+  splitfn() {
+    emulate -L zsh
+    setopt shwordsplit
+    local IFS=.-
+    local foo=1-2.3-4
+    #
+    print "Called with argument '$1'"
+    print "No quotes"
+    print -l ${1:-1-2.3-4} ${1:-$foo}
+    print "With quotes on default argument only"
+    print -l ${1:-"1-2.3-4"} ${1:-"$foo"}
+  }
+  print Using shwordsplit
+  splitfn
+  splitfn 5.6-7.8
+0:Test of nested word splitting with and without quotes
+>Using "="
+>Called with argument ''
+>No quotes
+>1
+>2
+>3
+>4
+>1
+>2
+>3
+>4
+>With quotes on default argument only
+>1-2.3-4
+>1-2.3-4
+>Called with argument '5.6-7.8'
+>No quotes
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+>With quotes on default argument only
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+>Using shwordsplit
+>Called with argument ''
+>No quotes
+>1
+>2
+>3
+>4
+>1
+>2
+>3
+>4
+>With quotes on default argument only
+>1-2.3-4
+>1-2.3-4
+>Called with argument '5.6-7.8'
+>No quotes
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+>With quotes on default argument only
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+
   unset SHLVL
   (( SHLVL++ ))
   print $SHLVL

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 297 bytes --]

On Mon, Feb 13, 2006 at 07:33:43PM +0000, Peter Stephenson wrote:
> Here's an add-on test that handles a lot of the possibilities.  I think
> the results are all correct.

Looks good to me.  I also added a couple tests.  The combination of both
our changes relative to CVS is attached.

..wayne..

[-- Attachment #2: d04.patch --]
[-- Type: text/plain, Size: 2300 bytes --]

--- Test/D04parameter.ztst	11 Oct 2005 16:48:06 -0000	1.13
+++ Test/D04parameter.ztst	13 Feb 2006 20:04:16 -0000
@@ -547,20 +547,128 @@
 >shell
 >again
 
-  set If "this test fails" maybe "we have finally fixed" the shell
+  local sure_that='sure that' varieties_of='varieties of' one=1 two=2
+  set Make $sure_that "this test keeps" on 'preserving all' "$varieties_of" quoted whitespace
   print -l ${=1+"$@"}
-0:Regression test of unfixed ${=1+"$@"} bug
->If
->this
->test
->fails
->maybe
->we
->have
->finally
->fixed
->the
->shell
+  print -l ${=1+Make $sure_that "this test keeps" on 'preserving all' "$varieties_of" quoted whitespace}
+  print -l ${=1+$one $two}
+0:Regression test of ${=1+"$@"} bug and some related expansions
+>Make
+>sure that
+>this test keeps
+>on
+>preserving all
+>varieties of
+>quoted
+>whitespace
+>Make
+>sure
+>that
+>this test keeps
+>on
+>preserving all
+>varieties of
+>quoted
+>whitespace
+>1
+>2
+
+  splitfn() {
+    local IFS=.-
+    local foo=1-2.3-4
+    #
+    print "Called with argument '$1'"
+    print "No quotes"
+    print -l ${=1:-1-2.3-4} ${=1:-$foo}
+    print "With quotes on default argument only"
+    print -l ${=1:-"1-2.3-4"} ${=1:-"$foo"}
+  }
+  print 'Using "="'
+  splitfn
+  splitfn 5.6-7.8
+  #
+  splitfn() {
+    emulate -L zsh
+    setopt shwordsplit
+    local IFS=.-
+    local foo=1-2.3-4
+    #
+    print "Called with argument '$1'"
+    print "No quotes"
+    print -l ${1:-1-2.3-4} ${1:-$foo}
+    print "With quotes on default argument only"
+    print -l ${1:-"1-2.3-4"} ${1:-"$foo"}
+  }
+  print Using shwordsplit
+  splitfn
+  splitfn 5.6-7.8
+0:Test of nested word splitting with and without quotes
+>Using "="
+>Called with argument ''
+>No quotes
+>1
+>2
+>3
+>4
+>1
+>2
+>3
+>4
+>With quotes on default argument only
+>1-2.3-4
+>1-2.3-4
+>Called with argument '5.6-7.8'
+>No quotes
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+>With quotes on default argument only
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+>Using shwordsplit
+>Called with argument ''
+>No quotes
+>1
+>2
+>3
+>4
+>1
+>2
+>3
+>4
+>With quotes on default argument only
+>1-2.3-4
+>1-2.3-4
+>Called with argument '5.6-7.8'
+>No quotes
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+>With quotes on default argument only
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
 
   unset SHLVL
   (( SHLVL++ ))

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 412 bytes --]

On Mon, Feb 13, 2006 at 09:43:38AM -0800, Wayne Davison wrote:
>  		insertlinknode(&foo, n, (void *)x), incnode(n);
> +		mult_isarr = 1;
>  	    }

This turned out to be inadequate in the face of the following expansion:

    print -l ${=1:-$LINES $COLUMNS}

This is because mult_isarr got reset to 0.  The attached patch fixes
this (which should be applied to whatever patched version you're using).

..wayne..

[-- Attachment #2: better-splitting.diff --]
[-- Type: text/plain, Size: 774 bytes --]

--- Src/subst.c	2006-02-13 11:35:16.000000000 -0800
+++ Src/subst.c	2006-02-13 11:40:48.000000000 -0800
@@ -337,6 +337,7 @@
     if (split) {
 	LinkNode n = firstnode(&foo);
 	int inq = 0, inp = 0;
+	split = 0; /* used to flag if we really split anything */
 	for ( ; *x; x += l+1) {
 	    char c = (l = *x == Meta) ? x[1] ^ 32 : *x;
 	    if (!inq && !inp && isep(c)) {
@@ -349,7 +350,7 @@
 		if (!*x)
 		    break;
 		insertlinknode(&foo, n, (void *)x), incnode(n);
-		mult_isarr = 1;
+		split = 1;
 	    }
 	    switch (c) {
 	    case Dnull:
@@ -375,6 +376,9 @@
 	mult_isarr = omi;
 	return 0;
     }
+    if (split)
+	mult_isarr = 1;
+
     if ((l = countlinknodes(&foo)) > 1 || (a && mult_isarr)) {
 	p = r = hcalloc((l + 1) * sizeof(char*));
 	while (nonempty(&foo))

Re: PATCH: fixing ${1+"$@"} when word-splitting

[DervishD]

    Hi Wayne :)

 * Wayne Davison <wayned@users.sourceforge.net> dixit:
> Attached is the patch (which also includes the good changes from my last
> email).  Let me know how you like it.

    I'm not familiar enough with zsh code to say wether the patch is
good or not, but it *looks* good to me. Thanks anyway for your work
:) If I can afford the time this night I'll try to test the patch.

    Raúl Núñez de Arenas Coronado

-- 
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It's my PC and I'll cry if I want to... RAmen!

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 664 bytes --]

I noticed that my patch wasn't handling backslash-quoted whitespace, so
I fixed that.  I then did some testing of filename-matching, and noticed
an improvement:

  % emulate sh
  % touch '1 2'  '3 4'
  % print -l ${1:-*[\ ]*}
  1 2
  3 4

Unpatched zsh would have output "*[" and "]*" on separate lines, which
is the behavior that happens without the backslash (in bash and both
patched/unpatched zsh).

One inconsistency with bash is the handling of '~' -- we no longer split
it if it contains a space, but bash does.  I think I'll just leave this
alone for now.

I added a few more tests to the D04 file.  Attached is the latest
version of the patch.

..wayne..

[-- Attachment #2: splitting.diff --]
[-- Type: text/plain, Size: 9139 bytes --]

--- Src/subst.c	6 Feb 2006 11:57:06 -0000	1.44
+++ Src/subst.c	14 Feb 2006 06:45:52 -0000
@@ -295,29 +295,86 @@ singsub(char **s)
     DPUTS(nonempty(&foo), "BUG: singsub() produced more than one word!");
 }
 
-/* Perform substitution on a single word. Unlike with singsub, the      *
- * result can have more than one word. A single word result is stored   *
- * in *s and *isarr is set to zero; otherwise *isarr is set to 1 and    *
- * the result is stored in *a. If `a' is zero a multiple word result is *
- * joined using sep or the IFS parameter if sep is zero and the result  *
- * is returned in *s.  The return value is true iff the expansion       *
- * resulted in an empty list.                                           *
- * The mult_isarr variable is used by paramsubst() to tell if it yields *
- * an array.                                                            */
+/* Perform substitution on a single word, *s. Unlike with singsub(), the
+ * result can be more than one word. If split is non-zero, the string is
+ * first word-split using IFS, but only for non-quoted "whitespace" (as
+ * indicated by Dnull, Snull, Tick, Inpar, and Outpar).
+ *
+ * If arg "a" was non-NULL _and_ the parsing set mult_isarr, the resulting
+ * strings are stored in *a (even for a 1-element array) and *isarr is set
+ * to 1.  Otherwise, *isarr is set to 0, and the result is stored in *s,
+ * with any necessary joining of multiple elements using sep (which can be
+ * NULL to use IFS).  The return value is true iff the expansion resulted
+ * in an empty list.
+ *
+ * The mult_isarr variable is used by paramsubst() to tell us if the
+ * substitutions yielded an array, but we will also set it if we split *s
+ * into multiple items (since that also yields an array). */
 
 /**/
 static int mult_isarr;
 
 /**/
 static int
-multsub(char **s, char ***a, int *isarr, UNUSED(char *sep))
+multsub(char **s, int split, char ***a, int *isarr, char *sep)
 {
     int l, omi = mult_isarr;
-    char **r, **p;
+    char **r, **p, *x = *s;
     local_list1(foo);
 
     mult_isarr = 0;
-    init_list1(foo, *s);
+
+    if (split) {
+	for ( ; *x; x += l+1) {
+	    char c = (l = *x == Meta) ? x[1] ^ 32 : *x;
+	    if (!iwsep(c))
+		break;
+	}
+    }
+
+    init_list1(foo, x);
+
+    if (split) {
+	LinkNode n = firstnode(&foo);
+	int inq = 0, inp = 0;
+	split = 0; /* use this to flag if we really split anything */
+	for ( ; *x; x += l+1) {
+	    char c = (l = *x == Meta) ? x[1] ^ 32 : *x;
+	    if (!inq && !inp && isep(c)) {
+		*x = '\0';
+		for (x += l+1; *x; x += l+1) {
+		    c = (l = *x == Meta) ? x[1] ^ 32 : *x;
+		    if (!isep(c))
+			break;
+		}
+		if (!*x)
+		    break;
+		insertlinknode(&foo, n, (void *)x), incnode(n);
+		split = 1;
+	    }
+	    switch (c) {
+	    case Dnull:  /* " */
+	    case Snull:  /* ' */
+	    case Tick:   /* ` (note: no Qtick!) */
+		/* These always occur in unnested pairs. */
+		inq = !inq;
+		break;
+	    case Inpar:  /* ( */
+		inp++;
+		break;
+	    case Outpar: /* ) */
+		inp--;
+		break;
+	    case Bnull:  /* \ */
+	    case Bnullkeep:
+		/* The parser verified the following char's existence. */
+		x += l+1;
+		l = *x == Meta;
+		break;
+	    }
+	}
+    }
+
     prefork(&foo, 0);
     if (errflag) {
 	if (isarr)
@@ -325,7 +382,10 @@ multsub(char **s, char ***a, int *isarr,
 	mult_isarr = omi;
 	return 0;
     }
-    if ((l = countlinknodes(&foo))) {
+    if (split)
+	mult_isarr = 1;
+
+    if ((l = countlinknodes(&foo)) > 1 || (a && mult_isarr)) {
 	p = r = hcalloc((l + 1) * sizeof(char*));
 	while (nonempty(&foo))
 	    *p++ = (char *)ugetnode(&foo);
@@ -345,7 +405,7 @@ multsub(char **s, char ***a, int *isarr,
 	    mult_isarr = omi;
 	    return 0;
 	}
-	*s = sepjoin(r, NULL, 1);
+	*s = sepjoin(r, sep, 1);
 	mult_isarr = omi;
 	if (isarr)
 	    *isarr = 0;
@@ -1457,7 +1517,7 @@ paramsubst(LinkList l, LinkNode n, char 
 	 * remove the aspar test and extract a value from an array, if
 	 * necessary, when we handle (P) lower down.
 	 */
-	if (multsub(&val, (aspar ? NULL : &aval), &isarr, NULL) && quoted) {
+	if (multsub(&val, 0, (aspar ? NULL : &aval), &isarr, NULL) && quoted) {
 	    /* Empty quoted string --- treat as null string, not elided */
 	    isarr = -1;
 	    aval = (char **) hcalloc(sizeof(char *));
@@ -1992,26 +2052,20 @@ paramsubst(LinkList l, LinkNode n, char 
 	/* Fall Through! */
 	case '-':
 	    if (vunset) {
+		int ws = opts[SHWORDSPLIT];
 		val = dupstring(s);
-		/*
-		 * This is not good enough for sh emulation!  Sh would
-		 * split unquoted substrings, yet not split quoted ones
-		 * (except according to $@ rules); but this leaves the
-		 * unquoted substrings unsplit, and other code below
-		 * for spbreak splits even within the quoted substrings.
-		 *
-		 * TODO: I think multsub needs to be told enough to
-		 * decide about splitting with spbreak at this point
-		 * (and equally in the `=' handler below).  Then
-		 * we can turn off spbreak to avoid the join & split
-		 * nastiness later.
-		 *
-		 * What we really want to do is make this look as
-		 * if it were the result of an assignment from
-		 * the same value, taking account of quoting.
-		 */
-		multsub(&val, (aspar ? NULL : &aval), &isarr, NULL);
+		/* If word-splitting is enabled, we ask multsub() to split
+		 * the substituted string at unquoted whitespace.  Then, we
+		 * turn off spbreak so that no further splitting occurs.
+		 * This allows a construct such as ${1+"$@"} to correctly
+		 * keep its array splits, and weird constructs such as
+		 * ${str+"one two" "3 2 1" foo "$str"} to only be split
+		 * at the unquoted spaces. */
+		opts[SHWORDSPLIT] = spbreak;
+		multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
+		opts[SHWORDSPLIT] = ws;
 		copied = 1;
+		spbreak = 0;
 	    }
 	    break;
 	case ':':
@@ -2029,7 +2083,6 @@ paramsubst(LinkList l, LinkNode n, char 
 
 		*idend = '\0';
 		val = dupstring(s);
-		isarr = 0;
 		/*
 		 * TODO: this is one of those places where I don't
 		 * think we want to do the joining until later on.
@@ -2037,9 +2090,9 @@ paramsubst(LinkList l, LinkNode n, char 
 		 * point and unset them.
 		 */
 		if (spsep || spbreak || !arrasg)
-		    multsub(&val, NULL, NULL, sep);
+		    multsub(&val, 0, NULL, &isarr, NULL);
 		else
-		    multsub(&val, &aval, &isarr, NULL);
+		    multsub(&val, 0, &aval, &isarr, NULL);
 		if (arrasg) {
 		    /*
 		     * This is an array assignment in a context
--- Test/D04parameter.ztst	11 Oct 2005 16:48:06 -0000	1.13
+++ Test/D04parameter.ztst	14 Feb 2006 06:45:52 -0000
@@ -547,20 +547,148 @@
 >shell
 >again
 
-  set If "this test fails" maybe "we have finally fixed" the shell
+  local sure_that='sure that' varieties_of='varieties of' one=1 two=2
+  set Make $sure_that "this test keeps" on 'preserving all' "$varieties_of" quoted whitespace
   print -l ${=1+"$@"}
-0:Regression test of unfixed ${=1+"$@"} bug
->If
->this
->test
->fails
->maybe
->we
->have
->finally
->fixed
->the
->shell
+  print -l ${=1+Make $sure_that "this test keeps" on 'preserving all' "$varieties_of" quoted whitespace}
+  print -l ${=1+$one $two}
+0:Regression test of ${=1+"$@"} bug and some related expansions
+>Make
+>sure that
+>this test keeps
+>on
+>preserving all
+>varieties of
+>quoted
+>whitespace
+>Make
+>sure
+>that
+>this test keeps
+>on
+>preserving all
+>varieties of
+>quoted
+>whitespace
+>1
+>2
+
+  splitfn() {
+    emulate -L sh
+    local HOME="/differs from/bash"
+    print -l ${1:-~}
+    touch has\ space
+    print -l ${1:-*[ ]*}
+    print -l ${1:-*[\ ]*}
+    print -l ${1:-*}
+    rm has\ space
+  }
+  splitfn
+0:More bourne-shell-compatible nested word-splitting with wildcards and ~
+>/differs from/bash
+>*[
+>]*
+>has space
+>boringfile
+>evenmoreboringfile
+>has space
+
+  splitfn() {
+    local IFS=.-
+    local foo=1-2.3-4
+    #
+    print "Called with argument '$1'"
+    print "No quotes"
+    print -l ${=1:-1-2.3-4} ${=1:-$foo}
+    print "With quotes on default argument only"
+    print -l ${=1:-"1-2.3-4"} ${=1:-"$foo"}
+  }
+  print 'Using "="'
+  splitfn
+  splitfn 5.6-7.8
+  #
+  splitfn() {
+    emulate -L zsh
+    setopt shwordsplit
+    local IFS=.-
+    local foo=1-2.3-4
+    #
+    print "Called with argument '$1'"
+    print "No quotes"
+    print -l ${1:-1-2.3-4} ${1:-$foo}
+    print "With quotes on default argument only"
+    print -l ${1:-"1-2.3-4"} ${1:-"$foo"}
+  }
+  print Using shwordsplit
+  splitfn
+  splitfn 5.6-7.8
+0:Test of nested word splitting with and without quotes
+>Using "="
+>Called with argument ''
+>No quotes
+>1
+>2
+>3
+>4
+>1
+>2
+>3
+>4
+>With quotes on default argument only
+>1-2.3-4
+>1-2.3-4
+>Called with argument '5.6-7.8'
+>No quotes
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+>With quotes on default argument only
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+>Using shwordsplit
+>Called with argument ''
+>No quotes
+>1
+>2
+>3
+>4
+>1
+>2
+>3
+>4
+>With quotes on default argument only
+>1-2.3-4
+>1-2.3-4
+>Called with argument '5.6-7.8'
+>No quotes
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
+>With quotes on default argument only
+>5
+>6
+>7
+>8
+>5
+>6
+>7
+>8
 
   unset SHLVL
   (( SHLVL++ ))

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Wayne Davison]

One other minor fix that I noticed in my patched code affects a request
for no-splitting when SH_WORD_SPLIT is enabled:

  setopt shwordsplit
  foo='1 2'
  print -l ${==foo+$foo}

That used to split the variable $foo if it was not quoted (an
interesting case where quotes made a difference in the old code).

I've checked in my changes, including a couple new sentences in the
zshexpn man page.

If we decide that the change in splitting is undesirable after all, the
following one-line change will put things back to the way they were
before (except that the bug above will still be fixed):

--- Src/subst.c	2006-02-15 01:58:31 -0800
+++ Src/subst.c	2006-02-15 02:03:41 -0800
@@ -2065,7 +2065,7 @@
 		multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
 		opts[SHWORDSPLIT] = ws;
 		copied = 1;
-		spbreak = 0;
+		/*spbreak = 0;*/
 	    }
 	    break;
 	case ':':

Of course, we'd also need to change some of the D04 tests and the docs.

..wayne..

Re: PATCH: fixing ${1+"$@"} when word-splitting

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 495 bytes --]

I noticed another long-standing bug (I think the final result is wrong):

 % foo=(1 2)
 % bar=3
 % print -l ${foo+$bar$foo}
 31
 2
 % print -l ${foo+$foo$bar}
 1 23

This is a case of the arrayness of the result being lost because the
non-array, $bar, came after the array, $foo.

Attached is a patch that fixes this by having multsub() only honor
mult_isarr as a flag that indicates that a single-item result was
really an array, not that a multi-item result should not be an array.

..wayne..

[-- Attachment #2: arrayness.patch --]
[-- Type: text/plain, Size: 4363 bytes --]

Index: Src/subst.c
--- Src/subst.c	15 Feb 2006 10:13:41 -0000	1.45
+++ Src/subst.c	15 Feb 2006 11:24:32 -0000
@@ -300,16 +300,15 @@ singsub(char **s)
  * first word-split using IFS, but only for non-quoted "whitespace" (as
  * indicated by Dnull, Snull, Tick, Bnull, Inpar, and Outpar).
  *
- * If arg "a" was non-NULL _and_ the parsing set mult_isarr, the resulting
- * strings are stored in *a (even for a 1-element array) and *isarr is set
- * to 1.  Otherwise, *isarr is set to 0, and the result is stored in *s,
+ * If arg "a" was non-NULL and we got an array as a result of the parsing,
+ * the strings are stored in *a (even for a 1-element array) and *isarr is
+ * set to 1.  Otherwise, *isarr is set to 0, and the result is put into *s,
  * with any necessary joining of multiple elements using sep (which can be
  * NULL to use IFS).  The return value is true iff the expansion resulted
  * in an empty list.
  *
- * The mult_isarr variable is used by paramsubst() to tell us if the
- * substitutions yielded an array, but we will also set it if we split *s
- * into multiple items (since that also yields an array). */
+ * The mult_isarr variable is used by paramsubst() to tell us if a single-
+ * item result was an array.  We always restore its value on exit. */
 
 /**/
 static int mult_isarr;
@@ -337,7 +336,6 @@ multsub(char **s, int split, char ***a, 
     if (split) {
 	LinkNode n = firstnode(&foo);
 	int inq = 0, inp = 0;
-	split = 0; /* use this to flag if we really split anything */
 	for ( ; *x; x += l+1) {
 	    char c = (l = *x == Meta) ? x[1] ^ 32 : *x;
 	    if (!inq && !inp && isep(c)) {
@@ -350,7 +348,6 @@ multsub(char **s, int split, char ***a, 
 		if (!*x)
 		    break;
 		insertlinknode(&foo, n, (void *)x), incnode(n);
-		split = 1;
 	    }
 	    switch (c) {
 	    case Dnull:  /* " */
@@ -382,24 +379,19 @@ multsub(char **s, int split, char ***a, 
 	mult_isarr = omi;
 	return 0;
     }
-    if (split)
-	mult_isarr = 1;
 
     if ((l = countlinknodes(&foo)) > 1 || (a && mult_isarr)) {
 	p = r = hcalloc((l + 1) * sizeof(char*));
 	while (nonempty(&foo))
 	    *p++ = (char *)ugetnode(&foo);
 	*p = NULL;
-	/*
-	 * This is the most obscure way of deciding whether a value is
-	 * an array it would be possible to imagine.  It seems to result
-	 * partly because we don't pass down the qt and ssub flags from
-	 * paramsubst() through prefork() properly, partly because we
-	 * don't tidy up to get back the return type from multsub we
-	 * need properly.  The crux of neatening this up is to get rid
-	 * of the following test.
-	 */
-	if (a && mult_isarr) {
+	/* We need a way to figure out if a one-item result was a scalar
+	 * or a single-item array.  The parser will have set mult_isarr
+	 * in the latter case, allowing us to return it as an array to
+	 * our caller (if they provided for that result).  It would be
+	 * better if this information were encoded in the list itself
+	 * (e.g. by adding a flag to the LinkList structure). */
+	if (a && (l > 1 || mult_isarr)) {
 	    *a = r;
 	    *isarr = SCANPM_MATCHMANY;
 	    mult_isarr = omi;
@@ -2397,12 +2389,6 @@ paramsubst(LinkList l, LinkNode n, char 
     }
     /* ssub is true when we are called from singsub (via prefork).
      * It means that we must join arrays and should not split words. */
-    /*
-     * TODO: this is what is screwing up the use of SH_WORD_SPLIT
-     * after `:-' etc.  If we fix multsub(), we might get away
-     * with simply unsetting the appropriate flags when they
-     * get handled.
-     */
     if (ssub || spbreak || spsep || sep) {
 	if (isarr)
 	    val = sepjoin(aval, sep, 1), isarr = 0;
Index: Test/D04parameter.ztst
--- Test/D04parameter.ztst	15 Feb 2006 10:13:43 -0000	1.14
+++ Test/D04parameter.ztst	15 Feb 2006 11:24:32 -0000
@@ -550,10 +550,12 @@
 >again
 
   local sure_that='sure that' varieties_of='varieties of' one=1 two=2
+  extra=(5 4 3)
   set Make $sure_that "this test keeps" on 'preserving all' "$varieties_of" quoted whitespace
   print -l ${=1+"$@"}
   print -l ${=1+Make $sure_that "this test keeps" on 'preserving all' "$varieties_of" quoted whitespace}
   print -l ${=1+$one $two}
+  print -l ${1+$extra$two$one}
 0:Regression test of ${=1+"$@"} bug and some related expansions
 >Make
 >sure that
@@ -574,6 +576,9 @@
 >whitespace
 >1
 >2
+>5
+>4
+>321
 
   splitfn() {
     emulate -L sh