Re: Bug with bash emulation regarding ':'

Re: Bug with bash emulation regarding ':'

[Jilles Tjoelker]

> [zsh performs word splitting on ${...:=$(...)}]

It seems strange to perform word splitting on word in ${var=word}
because a single word is required to assign to var. Indeed, zsh pastes
the resulting words back together to a single word to assign.

Historically, there have been two ways to perform word splitting here:
either by expanding the construct as ${var-word} or by expanding $var
after assignment. POSIX requires the latter. This implies that quoting
characters inside word does not prevent word splitting and pathname
generation. (Quoting the entire expansion does.)

In either case, if assignment takes place, the assigned value is the
expanded value of word with quote removal performed on it. (POSIX lacked
the detail of quote removal, formally requiring the assigned value to
retain the quoting characters; an interpretation has been issued.)

-- 
Jilles Tjoelker

Bug with bash emulation regarding ':'

[Felipe Contreras]

Hi,

Seems like there's a bug in bash emulation:

---
#!sh

_test ()
{
	array="one two three"
	for e in $array
	do
		echo "'$e'"
	done
}

l=
: ${l:=$(_test)}
echo "_${l}_"
---

Bash generates:

_'one'
'two'
'three'_

Zsh with bash emulation generates:

_'one two three'_

-- 
Felipe Contreras

Re: Bug with bash emulation regarding ':'

[Benjamin R. Haskell]

On Sun, 29 Jan 2012, Felipe Contreras wrote:

> Hi,
>
> Seems like there's a bug in bash emulation:
>
> [...]
>
> Zsh with bash emulation generates [...]

When you say "bash emulation" what do you mean?

Zsh has 'sh' and 'ksh' emulation (and incomplete 'csh' emulation).  I assume 
you mean 'sh' emulation, but just wanted to be sure it wasn't something related 
to your bash completion emulation fixes.

-- 
Best,
Ben

Re: Bug with bash emulation regarding ':'

[Felipe Contreras]

On Sun, Jan 29, 2012 at 7:31 AM, Benjamin R. Haskell <zsh@benizi.com> wrote:
> On Sun, 29 Jan 2012, Felipe Contreras wrote:
>> Seems like there's a bug in bash emulation:
>>
>> [...]
>>
>> Zsh with bash emulation generates [...]
>
>
> When you say "bash emulation" what do you mean?

emulate bash

> Zsh has 'sh' and 'ksh' emulation (and incomplete 'csh' emulation).  I assume
> you mean 'sh' emulation, but just wanted to be sure it wasn't something
> related to your bash completion emulation fixes.

Yeah, not related.

Cheers.

-- 
Felipe Contreras

Re: Bug with bash emulation regarding ':'

[Mikael Magnusson]

On 29 January 2012 14:44, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> On Sun, Jan 29, 2012 at 7:31 AM, Benjamin R. Haskell <zsh@benizi.com> wrote:
>> On Sun, 29 Jan 2012, Felipe Contreras wrote:
>>> Seems like there's a bug in bash emulation:
>>>
>>> [...]
>>>
>>> Zsh with bash emulation generates [...]
>>
>>
>> When you say "bash emulation" what do you mean?
>
> emulate bash

There's no such emulation mode.

emulate [ -LR ] [ {zsh|sh|ksh|csh} [ -c arg ] ]
With single argument set up zsh options to emulate the specified shell
as much as possible.  csh will never be fully emulated.  If the
argument is not one of the shells listed above, zsh will be used as a
default

This seems to not actually be true though,
    if (ch == 'r')
	ch = zsh_name[1];

    /* Work out the new emulation mode */
    if (ch == 'c')
	emulation = EMULATE_CSH;
    else if (ch == 'k')
	emulation = EMULATE_KSH;
    else if (ch == 's' || ch == 'b')
	emulation = EMULATE_SH;
    else
	emulation = EMULATE_ZSH;

-- 
Mikael Magnusson

Re: Bug with bash emulation regarding ':'

[Mikael Magnusson]

On 29 January 2012 15:25, Mikael Magnusson <mikachu@gmail.com> wrote:
> On 29 January 2012 14:44, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>> On Sun, Jan 29, 2012 at 7:31 AM, Benjamin R. Haskell <zsh@benizi.com> wrote:
>>> On Sun, 29 Jan 2012, Felipe Contreras wrote:
>>>> Seems like there's a bug in bash emulation:
>>>>
>>>> [...]
>>>>
>>>> Zsh with bash emulation generates [...]
>>>
>>>
>>> When you say "bash emulation" what do you mean?
>>
>> emulate bash
>
> There's no such emulation mode.
>
> emulate [ -LR ] [ {zsh|sh|ksh|csh} [ -c arg ] ]
> With single argument set up zsh options to emulate the specified shell
> as much as possible.  csh will never be fully emulated.  If the
> argument is not one of the shells listed above, zsh will be used as a
> default
>
> This seems to not actually be true though,
>    if (ch == 'r')
>        ch = zsh_name[1];
>
>    /* Work out the new emulation mode */
>    if (ch == 'c')
>        emulation = EMULATE_CSH;
>    else if (ch == 'k')
>        emulation = EMULATE_KSH;
>    else if (ch == 's' || ch == 'b')
>        emulation = EMULATE_SH;
>    else
>        emulation = EMULATE_ZSH;
>

(btw, this code is from 'Initial revision' anno 99)

-- 
Mikael Magnusson

Re: Bug with bash emulation regarding ':'

[Vincent Lefevre]

On 2012-01-29 15:25:13 +0100, Mikael Magnusson wrote:
> On 29 January 2012 14:44, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> > emulate bash
> 
> There's no such emulation mode.

I think the problem is that "emulate" doesn't return an error
if the argument is unsupported.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Bug with bash emulation regarding ':'

[Mikael Magnusson]

On 29 January 2012 19:38, Vincent Lefevre <vincent@vinc17.net> wrote:
> On 2012-01-29 15:25:13 +0100, Mikael Magnusson wrote:
>> On 29 January 2012 14:44, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>> > emulate bash
>>
>> There's no such emulation mode.
>
> I think the problem is that "emulate" doesn't return an error
> if the argument is unsupported.

Well, there are no unsupported arguments, anything that isn't
recognized is documented (although not accurately) to enter zsh mode
:).

-- 
Mikael Magnusson

Re: Bug with bash emulation regarding ':'

[Vincent Lefevre]

On 2012-01-29 19:53:51 +0100, Mikael Magnusson wrote:
> On 29 January 2012 19:38, Vincent Lefevre <vincent@vinc17.net> wrote:
> > On 2012-01-29 15:25:13 +0100, Mikael Magnusson wrote:
> >> On 29 January 2012 14:44, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> >> > emulate bash
> >>
> >> There's no such emulation mode.
> >
> > I think the problem is that "emulate" doesn't return an error
> > if the argument is unsupported.
> 
> Well, there are no unsupported arguments, anything that isn't
> recognized is documented (although not accurately) to enter zsh mode
> :).

Yes, but this is not intuitive. An error would be better, IMHO.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Bug with bash emulation regarding ':'

[Peter Stephenson]

On Sun, 29 Jan 2012 20:01:13 +0100
Vincent Lefevre <vincent@vinc17.net> wrote:
> On 2012-01-29 19:53:51 +0100, Mikael Magnusson wrote:
> > On 29 January 2012 19:38, Vincent Lefevre <vincent@vinc17.net> wrote:
> > > On 2012-01-29 15:25:13 +0100, Mikael Magnusson wrote:
> > >> On 29 January 2012 14:44, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> > >> > emulate bash
> > >>
> > >> There's no such emulation mode.
> > >
> > > I think the problem is that "emulate" doesn't return an error
> > > if the argument is unsupported.
> > 
> > Well, there are no unsupported arguments, anything that isn't
> > recognized is documented (although not accurately) to enter zsh mode
> > :).
> 
> Yes, but this is not intuitive. An error would be better, IMHO.

emulate "bash" is treated as emulate "sh", but isn't so documented.

However, having any random emulation treated as zsh in the emulate
command seems a bit pointless.  I think it only does that because the
code is shared with what happens at shell start-up.

How about the following compromise?

- Behave the same at start-up as always.  There's no point in an
error here if we don't recognise what zsh is supposed to be emulating.

- In the emulate command, keep the behaviour that we only test the
initial letter (and that "b" is like "s"), but document it.

- However, if the initial letter isn't recognised within the emulate
command (i.e. we now require a 'z' for native mode), complain.

This will annoy anyone who deliberately used "emulate native" to enter
zsh mode (for which I have only limited sympathy), but unless I've
missed something it should keep everyone else happy; indeed, the emulate
command only lists the explicit possibilities "zsh", "sh", "ksh", "csh",
but we did document that zsh was used as a default.  Usually Bart spots
something I've missed at this point, however.

I'd argue along with Vincent that even if there is a small amount of
pain it's better than letting people assume "emulate pdksh" or "emulate
dash" did what they presumably thought.

Index: Doc/Zsh/builtins.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/builtins.yo,v
retrieving revision 1.136
diff -p -u -r1.136 builtins.yo
--- Doc/Zsh/builtins.yo	3 Jun 2011 19:54:44 -0000	1.136
+++ Doc/Zsh/builtins.yo	29 Jan 2012 20:21:45 -0000
@@ -354,10 +354,19 @@ Without any argument print current emula
 With single argument set up zsh options to emulate the specified shell
 as much as possible.
 bf(csh) will never be fully emulated.
-If the argument is not one of the shells listed above, tt(zsh)
-will be used as a default; more precisely, the tests performed on the
-argument are the same as those used to determine the emulation at startup
-based on the shell name, see
+
+The test is actually based on the initial letter of the emulation mode,
+unless the first letter is `tt(r)', in which case the second letter
+is used.  (The `tt(r)' is assumed to stand for `restricted', however
+no further account of this is taken by `tt(emulate)'.)  In addition
+to the letters `tt(z)', `tt(s)', `tt(k)' and `tt(c)', the letter `tt(b)'
+is treated the same as `tt(s)'.  (The `tt(b)' stands for `tt(bash)', however
+no distinction is made between this and `tt(sh)'.)  Any other initial
+letter is treated as an error.
+
+The tests performed on the argument are the same as those used to
+determine the emulation at startup based on the shell name, with the
+exception that at startup zsh is silently used as a default.  See
 ifzman(\
 the section `Compatibility' in zmanref(zshmisc)
 )\
Index: Src/builtin.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/builtin.c,v
retrieving revision 1.254
diff -p -u -r1.254 builtin.c
--- Src/builtin.c	29 Aug 2011 17:21:40 -0000	1.254
+++ Src/builtin.c	29 Jan 2012 20:21:45 -0000
@@ -4989,7 +4995,8 @@ bin_emulate(UNUSED(char *nam), char **ar
 
     /* with single argument set current emulation */
     if (!argv[1]) {
-	emulate(*argv, OPT_ISSET(ops,'R'));
+	if (emulate(*argv, OPT_ISSET(ops,'R'), 1))
+	    return 1;
 	if (OPT_ISSET(ops,'L'))
 	    opts[LOCALOPTIONS] = opts[LOCALTRAPS] = 1;
 	return 0;
@@ -5016,11 +5023,12 @@ bin_emulate(UNUSED(char *nam), char **ar
     memcpy(saveopts, opts, sizeof(opts));
     saveemulation = emulation;
     savesticky_emulation = sticky_emulation;
-    emulate(*argv, OPT_ISSET(ops,'R'));
-    sticky_emulation = emulation;
-    ret = eval(argv+2);
-    memcpy(opts, saveopts, sizeof(opts));
-    sticky_emulation = savesticky_emulation;
+    if (!emulate(*argv, OPT_ISSET(ops,'R'), 1)) {
+	sticky_emulation = emulation;
+	ret = eval(argv+2);
+	memcpy(opts, saveopts, sizeof(opts));
+	sticky_emulation = savesticky_emulation;
+    }
     emulation = saveemulation;
     return ret;
 }
Index: Src/init.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/init.c,v
retrieving revision 1.122
diff -p -u -r1.122 init.c
--- Src/init.c	5 Jan 2012 20:01:25 -0000	1.122
+++ Src/init.c	29 Jan 2012 20:21:45 -0000
@@ -1510,7 +1510,7 @@ zsh_main(UNUSED(int argc), char **argv)
     fdtable = zshcalloc(fdtable_size*sizeof(*fdtable));
 
     createoptiontable();
-    emulate(zsh_name, 1);   /* initialises most options */
+    emulate(zsh_name, 1, 0);   /* initialises most options */
     opts[LOGINSHELL] = (**argv == '-');
     opts[PRIVILEGED] = (getuid() != geteuid() || getgid() != getegid());
     opts[USEZLE] = 1;   /* may be unset in init_io() */
Index: Src/options.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/options.c,v
retrieving revision 1.59
diff -p -u -r1.59 options.c
--- Src/options.c	8 Jan 2012 16:02:55 -0000	1.59
+++ Src/options.c	29 Jan 2012 20:21:45 -0000
@@ -502,8 +502,8 @@ installemulation(void)
 }
 
 /**/
-void
-emulate(const char *zsh_name, int fully)
+int
+emulate(const char *zsh_name, int fully, int error_if_not_matched)
 {
     char ch = *zsh_name;
 
@@ -517,12 +517,18 @@ emulate(const char *zsh_name, int fully)
 	emulation = EMULATE_KSH;
     else if (ch == 's' || ch == 'b')
 	emulation = EMULATE_SH;
-    else
+    else if (ch == 'z' || !error_if_not_matched)
 	emulation = EMULATE_ZSH;
+    else {
+	zwarnnam("emulate", "emulation not recognised: %s", zsh_name);
+	return 1;
+    }
 
     if (fully)
 	emulation |= EMULATE_FULLY;
     installemulation();
+
+    return 0;
 }
 
 /* setopt, unsetopt */

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug with bash emulation regarding ':'

[Mikael Magnusson]

On 29 January 2012 21:30, Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> On Sun, 29 Jan 2012 20:01:13 +0100
> Vincent Lefevre <vincent@vinc17.net> wrote:
>> On 2012-01-29 19:53:51 +0100, Mikael Magnusson wrote:
>> > On 29 January 2012 19:38, Vincent Lefevre <vincent@vinc17.net> wrote:
>> > > On 2012-01-29 15:25:13 +0100, Mikael Magnusson wrote:
>> > >> On 29 January 2012 14:44, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>> > >> > emulate bash
>> > >>
>> > >> There's no such emulation mode.
>> > >
>> > > I think the problem is that "emulate" doesn't return an error
>> > > if the argument is unsupported.
>> >
>> > Well, there are no unsupported arguments, anything that isn't
>> > recognized is documented (although not accurately) to enter zsh mode
>> > :).
>>
>> Yes, but this is not intuitive. An error would be better, IMHO.
>
> emulate "bash" is treated as emulate "sh", but isn't so documented.
>
> However, having any random emulation treated as zsh in the emulate
> command seems a bit pointless.  I think it only does that because the
> code is shared with what happens at shell start-up.
>
> How about the following compromise?
>
> - Behave the same at start-up as always.  There's no point in an
> error here if we don't recognise what zsh is supposed to be emulating.
>
> - In the emulate command, keep the behaviour that we only test the
> initial letter (and that "b" is like "s"), but document it.
>
> - However, if the initial letter isn't recognised within the emulate
> command (i.e. we now require a 'z' for native mode), complain.
>
> This will annoy anyone who deliberately used "emulate native" to enter
> zsh mode (for which I have only limited sympathy), but unless I've
> missed something it should keep everyone else happy; indeed, the emulate
> command only lists the explicit possibilities "zsh", "sh", "ksh", "csh",
> but we did document that zsh was used as a default.  Usually Bart spots
> something I've missed at this point, however.

I noticed that the 'Compatibility' section referred to only talks
about ksh and sh emulation, not csh, and it doesn't mention emulating
sh when the first letter is b either.

-- 
Mikael Magnusson

Re: Bug with bash emulation regarding ':'

[Felipe Contreras]

On Sun, Jan 29, 2012 at 4:25 PM, Mikael Magnusson <mikachu@gmail.com> wrote:
> On 29 January 2012 14:44, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>> On Sun, Jan 29, 2012 at 7:31 AM, Benjamin R. Haskell <zsh@benizi.com> wrote:
>>> On Sun, 29 Jan 2012, Felipe Contreras wrote:
>>>> Seems like there's a bug in bash emulation:
>>>>
>>>> [...]
>>>>
>>>> Zsh with bash emulation generates [...]
>>>
>>>
>>> When you say "bash emulation" what do you mean?
>>
>> emulate bash
>
> There's no such emulation mode.
>
> emulate [ -LR ] [ {zsh|sh|ksh|csh} [ -c arg ] ]
> With single argument set up zsh options to emulate the specified shell
> as much as possible.  csh will never be fully emulated.  If the
> argument is not one of the shells listed above, zsh will be used as a
> default
>
> This seems to not actually be true though,
>    if (ch == 'r')
>        ch = zsh_name[1];
>
>    /* Work out the new emulation mode */
>    if (ch == 'c')
>        emulation = EMULATE_CSH;
>    else if (ch == 'k')
>        emulation = EMULATE_KSH;
>    else if (ch == 's' || ch == 'b')
>        emulation = EMULATE_SH;
>    else
>        emulation = EMULATE_ZSH;

And to add more to the confusion:

#define EMULATE_SH   (1<<3) /* Bourne shell */

To me it looks like ksh is closer to bash than sh... perhaps it would
make more sense to map 'b' to ksh. Even better would be to just create
another separate emulation option to avoid the confusion.

Cheers.

-- 
Felipe Contreras

Re: Bug with bash emulation regarding ':'

[Mikael Magnusson]

On 30 January 2012 23:15, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> On Sun, Jan 29, 2012 at 4:25 PM, Mikael Magnusson <mikachu@gmail.com> wrote:
>> On 29 January 2012 14:44, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>>> On Sun, Jan 29, 2012 at 7:31 AM, Benjamin R. Haskell <zsh@benizi.com> wrote:
>>>> On Sun, 29 Jan 2012, Felipe Contreras wrote:
>>>>> Seems like there's a bug in bash emulation:
>>>>>
>>>>> [...]
>>>>>
>>>>> Zsh with bash emulation generates [...]
>>>>
>>>>
>>>> When you say "bash emulation" what do you mean?
>>>
>>> emulate bash
>>
>> There's no such emulation mode.
>>
>> emulate [ -LR ] [ {zsh|sh|ksh|csh} [ -c arg ] ]
>> With single argument set up zsh options to emulate the specified shell
>> as much as possible.  csh will never be fully emulated.  If the
>> argument is not one of the shells listed above, zsh will be used as a
>> default
>>
>> This seems to not actually be true though,
>>    if (ch == 'r')
>>        ch = zsh_name[1];
>>
>>    /* Work out the new emulation mode */
>>    if (ch == 'c')
>>        emulation = EMULATE_CSH;
>>    else if (ch == 'k')
>>        emulation = EMULATE_KSH;
>>    else if (ch == 's' || ch == 'b')
>>        emulation = EMULATE_SH;
>>    else
>>        emulation = EMULATE_ZSH;
>
> And to add more to the confusion:
>
> #define EMULATE_SH   (1<<3) /* Bourne shell */
>
> To me it looks like ksh is closer to bash than sh... perhaps it would
> make more sense to map 'b' to ksh. Even better would be to just create
> another separate emulation option to avoid the confusion.

I'm not exactly sure what the point is here, are you perhaps confusing
the bourne shell (sh) with the bourne again shell (bash)?

-- 
Mikael Magnusson

Re: Bug with bash emulation regarding ':'

[Peter Stephenson]

On Sun, 29 Jan 2012 04:46:35 +0200
Felipe Contreras <felipe.contreras@gmail.com> wrote:
> _test ()
> {
> 	array="one two three"
> 	for e in $array
> 	do
> 		echo "'$e'"
> 	done
> }
> 
> l=
> : ${l:=$(_test)}
> echo "_${l}_"
>...
> _'one two three'_

As this is OK with

l=$(_test)

presumably something is going on with quoting and/or tokenisation in or
close to multsub(), which is some of the hairier code in substitution,
when SH_WORD_SPLIT is in effect.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug with bash emulation regarding ':'

[Bart Schaefer]

On Jan 29,  6:36pm, Peter Stephenson wrote:
}
} > _test ()
} > {
} > 	array="one two three"
} > 	for e in $array
} > 	do
} > 		echo "'$e'"
} > 	done
} > }
} > 
} > l=
} > : ${l:=$(_test)}
} > echo "_${l}_"
} >...
} > _'one two three'_
} 
} presumably something is going on with quoting and/or tokenisation in or
} close to multsub(), which is some of the hairier code in substitution,
} when SH_WORD_SPLIT is in effect.

Given that this sorta kinda worked back in 4.2.x ... I think the problem
is explained by this comment (dates to 4.3.0-dev-4, which means this bug
has gone unnoticed for almost exactly five years):

2619   case '-':
2620       if (vunset) {
2621           int ws = opts[SHWORDSPLIT];
2622           val = dupstring(s);
2623           /* If word-splitting is enabled, we ask multsub() to split
2624            * the substituted string at unquoted whitespace.  Then, we
2625            * turn off spbreak so that no further splitting occurs.
2626            * This allows a construct such as ${1+"$@"} to correctly
2627            * keep its array splits, and weird constructs such as
2628            * ${str+"one two" "3 2 1" foo "$str"} to only be split
2629            * at the unquoted spaces. */
2630           opts[SHWORDSPLIT] = spbreak;
2631           multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
2632           opts[SHWORDSPLIT] = ws;
2633           copied = 1;
2634           spbreak = 0;

The breakage in this specific example is at 2658 where SHWORDSPIT has been
forced off, but there's probably another example where the above breaks too:

2648   case Equals:
2649       if (vunset) {
2650           int ws = opts[SHWORDSPLIT];
2651           char sav = *idend;
2652           int l;
2653 
2654           *idend = '\0';
2655           val = dupstring(s);
2656           if (spsep || !arrasg) {
2657               opts[SHWORDSPLIT] = 0;
2658               multsub(&val, 0, NULL, &isarr, NULL);
2659           } else {
2660               opts[SHWORDSPLIT] = spbreak;
2661               multsub(&val, spbreak, &aval, &isarr, NULL);
2662               spbreak = 0;
2663           }
2664           opts[SHWORDSPLIT] = ws;

The problem is that we want to suppress any further splitting in the
current shell, but when we enter the subshell for $(...) it should
revert to the original setting.  I'm not sure how to pass that down
the call stack appropriately.

Re: Bug with bash emulation regarding ':'

[Bart Schaefer]

On Jan 31,  8:29pm, Bart Schaefer wrote:
}
} 2657               opts[SHWORDSPLIT] = 0;
} 2658               multsub(&val, 0, NULL, &isarr, NULL);
} 
} The problem is that we want to suppress any further splitting in the
} current shell, but when we enter the subshell for $(...) it should
} revert to the original setting.

I can verify this by defining _test with "emulate sh -c ..." so that
the SHWORDSPLIT option becomes sticky in that function.

I'm still not sure what to do about it.  The question goes a bit deeper
in that there are several ways that shell code might get invoked during
stringsubst() -- the (e::) glob flag, dynamic directory names, etc. --
and presumably those should also all use the original splitting style,
whereas simple nested expansions should not.

Re: Bug with bash emulation regarding ':'

[Peter Stephenson]

On Wed, 01 Feb 2012 08:29:29 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jan 31,  8:29pm, Bart Schaefer wrote:
> }
> } 2657               opts[SHWORDSPLIT] = 0;
> } 2658               multsub(&val, 0, NULL, &isarr, NULL);
> } 
> } The problem is that we want to suppress any further splitting in the
> } current shell, but when we enter the subshell for $(...) it should
> } revert to the original setting.
> 
> I can verify this by defining _test with "emulate sh -c ..." so that
> the SHWORDSPLIT option becomes sticky in that function.
> 
> I'm still not sure what to do about it.  The question goes a bit deeper
> in that there are several ways that shell code might get invoked during
> stringsubst() -- the (e::) glob flag, dynamic directory names, etc. --
> and presumably those should also all use the original splitting style,
> whereas simple nested expansions should not.

Realistically, I think we either have to bite the bullet and find
something ugly that works or forget about it entirely.

How about this for the original problem?  It's at least a simple change,
tied to one documented C variable.  It doesn't address the related areas
you mentioned, but I think those are less important, because they don't
affect Bourne/POSIX emulation and though in principle we should
certainly make all combinations of features behave consistently, in
practice I'm not keen on encouraging native zsh programmers to use
arbitrary option settings.  (Actually, I don't really want to think
about what 'a native zsh programmer' might mean...)

There's still a difference from bash, where the output includes newlines
(i.e. the literal function output), whereas zsh turns the newlines into
spaces.  We use IFS to split the value returned by the function, but
bash doesn't.  Defining an IFS without $'\n' gives us what bash does,
but that's not the difference since bash has $'\n' in the default IFS,
too.  So this difference is somewhere in word splitting for parameter
substitution, too.  I haven't looked into the details, but I would guess
that in bash

: ${...:=$(...)}

does what we would do with

: ${...:="$(...)"}

However, KSH_TYPESET doesn't change this (and as noted recently is not
on for bash anyway).  I would guess KSH_TYPESET *should* have that
effect, and am inclining to the view turning it on for SH mode would be
sensible.  So more investigation is needed.  Feel free.

Index: Src/exec.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/exec.c,v
retrieving revision 1.207
diff -p -u -r1.207 exec.c
--- Src/exec.c	16 Jan 2012 22:14:11 -0000	1.207
+++ Src/exec.c	5 Feb 2012 19:39:50 -0000
@@ -3724,6 +3724,19 @@ getoutput(char *cmd, int qt)
     redup(pipes[1], 1);
     entersubsh(ESUB_PGRP|ESUB_NOMONITOR);
     cmdpush(CS_CMDSUBST);
+    /*
+     * Special case if this is a command substitution called
+     * recursively from parameter substitution.  In that
+     * case we've tampered with SHWORDSPLIT in order to fix
+     * up parameter substitution arguments.  However, here
+     * we're back at command level, so we need to restore
+     * it to whatever it was before.  Then we'll pretend
+     * we never tampered with it.
+     */
+    if (real_shwordsplit != -1) {
+	opts[SHWORDSPLIT] = real_shwordsplit;
+	real_shwordsplit = -1;
+    }
     execode(prog, 0, 1, "cmdsubst");
     cmdpop();
     close(1);
Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.130
diff -p -u -r1.130 subst.c
--- Src/subst.c	21 Dec 2011 22:39:28 -0000	1.130
+++ Src/subst.c	5 Feb 2012 19:39:50 -0000
@@ -35,6 +35,22 @@
 /**/
 char nulstring[] = {Nularg, '\0'};
 
+/*
+ * The real value of the option SH_WORD_SPLIT.
+ * We need this because of the following hierarchy:
+ * - paramsubst alters SH_WORD_SPLIT
+ * - multsub() does some substitution with SH_WORD_SPLIT off
+ * - a nested $(...) executes commands.  This should reflect
+ *   the original value of SH_WORD_SPLIT.
+ *
+ * -1 indicates this isn't in use.
+ *
+ * Even so, yuk.  This will be done properly in the object-oriented rewrite
+ * of zsh currently scheduled for the year 3712, weather permitting.
+ */
+/**/
+int real_shwordsplit = -1;
+
 /* Do substitutions before fork. These are:
  *  - Process substitution: <(...), >(...), =(...)
  *  - Parameter substitution
@@ -213,7 +229,16 @@ stringsubst(LinkList list, LinkNode node
 		setdata(node, (void *) str3);
 		continue;
 	    } else {
+		/*
+		 * Save and restore the value of SHWORDSPLIT.
+		 * See notes on real_shwordsplit above.
+		 * The restore so that we don't use real_shwordsplit
+		 * outside code called from paramsubst().
+		 */
+		char old_shwordsplit;
+		old_shwordsplit = real_shwordsplit = opts[SHWORDSPLIT];
 		node = paramsubst(list, node, &str, qt, ssub);
+		real_shwordsplit = old_shwordsplit;
 		if (errflag || !node)
 		    return NULL;
 		str3 = (char *)getdata(node);
Index: Test/D04parameter.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D04parameter.ztst,v
retrieving revision 1.60
diff -p -u -r1.60 D04parameter.ztst
--- Test/D04parameter.ztst	17 Aug 2011 19:00:10 -0000	1.60
+++ Test/D04parameter.ztst	5 Feb 2012 19:39:51 -0000
@@ -255,6 +255,20 @@
 >two
 >words
 
+  (setopt shwordsplit # ensure this doesn't get set in main shell...
+  test_splitting ()
+  {
+    array="one two three"
+    for e in $array; do
+      echo "'$e'"
+    done
+  }
+  test_split_var=
+  : ${test_split_var:=$(test_splitting)}
+  echo "_${test_split_var}_")
+0:SH_WORD_SPLIT inside $(...) inside ${...}
+>_'one' 'two' 'three'_
+
   print -l "${(f)$(print first line\\nsecond line\\nthird line)}"
 0:${(f)$(...)}
 >first line

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug with bash emulation regarding ':'

[Bart Schaefer]

On Feb 5,  8:11pm, Peter Stephenson wrote:
}
} How about this for the original problem?  It's at least a simple change,
} tied to one documented C variable.  It doesn't address the related areas
} you mentioned, but I think those are less important, because they don't
} affect Bourne/POSIX emulation

Seems reasonable to me.  I was hunting for a way to do it without adding
another global, but what you've done is probably the most expedient.

(I wonder if changing the boolean "split" parameter to multsub() into a
set of bitflags would suffice.)

I don't have anything more to add on the rest of this discussion, a this
point ...

Re: Bug with bash emulation regarding ':'

[Peter Stephenson]

On Sun, 05 Feb 2012 13:21:46 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> (I wonder if changing the boolean "split" parameter to multsub() into a
> set of bitflags would suffice.)

It took some thinking about, but I think the following is about the most
minimal code.  Even allowing for the fact that explicit splitting is
separate from sh-wordsplitting (it's a stronger condition), it needed a
third flag to distinguish the cases...

- SHWORDSPLIT is on normally.
- SHWORDSPLIT isn't on, but is required by a recursive substitution.
- SHWORDSPLIT needs to be forced off by a recursive substitution.

At least I think so.

There's now no temporary setting of SHWORDSPLIT itself and the special
flags are only tested on the way through to paramsubst(), so command
substitution doesn't seem them.

I don't know if this helps with the other issues you mentioned.

Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.130
diff -p -u -r1.130 subst.c
--- Src/subst.c	21 Dec 2011 22:39:28 -0000	1.130
+++ Src/subst.c	10 Feb 2012 20:54:23 -0000
@@ -74,7 +74,10 @@ prefork(LinkList list, int flags)
 	     */
 	    setdata(node, cptr);
 	}
-	if (!(node = stringsubst(list, node, flags & PF_SINGLE, asssub))) {
+	if (!(node = stringsubst(list, node,
+				 flags & (PF_SINGLE|PF_SPLIT|
+					  PF_SHWORDSPLIT|PF_NOSHWORDSPLIT),
+				 asssub))) {
 	    unqueue_signals();
 	    return;
 	}
@@ -145,7 +148,7 @@ stringsubstquote(char *strstart, char **
 
 /**/
 static LinkNode
-stringsubst(LinkList list, LinkNode node, int ssub, int asssub)
+stringsubst(LinkList list, LinkNode node, int pf_flags, int asssub)
 {
     int qt;
     char *str3 = (char *)getdata(node);
@@ -213,7 +216,11 @@ stringsubst(LinkList list, LinkNode node
 		setdata(node, (void *) str3);
 		continue;
 	    } else {
-		node = paramsubst(list, node, &str, qt, ssub);
+		if ((isset(SHWORDSPLIT) && !(pf_flags & PF_NOSHWORDSPLIT)) ||
+		    (pf_flags & PF_SPLIT))
+		    pf_flags |= PF_SHWORDSPLIT;
+		node = paramsubst(list, node, &str, qt,
+				  pf_flags & (PF_SINGLE|PF_SHWORDSPLIT));
 		if (errflag || !node)
 		    return NULL;
 		str3 = (char *)getdata(node);
@@ -268,7 +275,7 @@ stringsubst(LinkList list, LinkNode node
 		       (qt && str[1] == '"'))))
 		    *str = ztokens[c - Pound];
 	    str++;
-	    if (!(pl = getoutput(str2 + 1, qt || ssub))) {
+	    if (!(pl = getoutput(str2 + 1, qt || (pf_flags & PF_SINGLE)))) {
 		zerr("parse error in command substitution");
 		return NULL;
 	    }
@@ -278,7 +285,7 @@ stringsubst(LinkList list, LinkNode node
 		str = strcpy(str2, str);
 		continue;
 	    }
-	    if (!qt && ssub && isset(GLOBSUBST))
+	    if (!qt && (pf_flags & PF_SINGLE) && isset(GLOBSUBST))
 		shtokenize(s);
 	    l1 = str2 - str3;
 	    l2 = strlen(s);
@@ -306,7 +313,7 @@ stringsubst(LinkList list, LinkNode node
 	     * We are in a normal argument which looks like an assignment
 	     * and is to be treated like one, with no word splitting.
 	     */
-	    ssub = 1;
+	    pf_flags |= PF_SINGLE;
 	}
 	str++;
     }
@@ -392,13 +399,13 @@ singsub(char **s)
 
 /**/
 static int
-multsub(char **s, int split, char ***a, int *isarr, char *sep)
+multsub(char **s, int pf_flags, char ***a, int *isarr, char *sep)
 {
     int l;
     char **r, **p, *x = *s;
     local_list1(foo);
 
-    if (split) {
+    if (pf_flags & PF_SPLIT) {
 	/*
 	 * This doesn't handle multibyte characters, but we're
 	 * looking for whitespace separators which must be ASCII.
@@ -413,7 +420,7 @@ multsub(char **s, int split, char ***a, 
 
     init_list1(foo, x);
 
-    if (split) {
+    if (pf_flags & PF_SPLIT) {
 	LinkNode n = firstnode(&foo);
 	int inq = 0, inp = 0;
 	MB_METACHARINIT();
@@ -467,7 +474,7 @@ multsub(char **s, int split, char ***a, 
 	}
     }
 
-    prefork(&foo, 0);
+    prefork(&foo, pf_flags);
     if (errflag) {
 	if (isarr)
 	    *isarr = 0;
@@ -1437,7 +1444,7 @@ check_colon_subscript(char *str, char **
 
 /**/
 static LinkNode
-paramsubst(LinkList l, LinkNode n, char **str, int qt, int ssub)
+paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags)
 {
     char *aptr = *str, c, cc;
     char *s = aptr, *fstr, *idbeg, *idend, *ostr = (char *) getdata(n);
@@ -1514,7 +1521,7 @@ paramsubst(LinkList l, LinkNode n, char 
      * where we shouldn't, in particular on the multsubs for
      * handling embedded values for ${...=...} and the like.
      */
-    int spbreak = isset(SHWORDSPLIT) && !ssub && !qt;
+    int spbreak = (pf_flags & PF_SHWORDSPLIT) && !(pf_flags & PF_SINGLE) && !qt;
     /* Scalar and array value, see isarr above */
     char *val = NULL, **aval = NULL;
     /*
@@ -1564,6 +1571,11 @@ paramsubst(LinkList l, LinkNode n, char 
      */
     int shsplit = 0;
     /*
+     * "ssub" is true when we are called from singsub (via prefork):
+     * it means that we must join arrays and should not split words.
+     */
+    int ssub = (pf_flags & PF_SINGLE);
+    /*
      * The separator from (j) and (s) respectively, or (F) and (f)
      * respectively (hardwired to "\n" in that case).  Slightly
      * confusingly also used for ${#pm}, thought that's at least
@@ -1620,7 +1632,7 @@ paramsubst(LinkList l, LinkNode n, char 
      * This is one of the things that decides whether multsub
      * will produce an array, but in an extremely indirect fashion.
      */
-    int nojoin = isset(SHWORDSPLIT) ? !(ifs && *ifs) : 0;
+    int nojoin = (pf_flags & PF_SHWORDSPLIT) ? !(ifs && *ifs) : 0;
     /*
      * != 0 means ${...}, otherwise $...  What works without braces
      * is largely a historical artefact (everything works with braces,
@@ -2618,7 +2630,7 @@ paramsubst(LinkList l, LinkNode n, char 
 	/* Fall Through! */
 	case '-':
 	    if (vunset) {
-		int ws = opts[SHWORDSPLIT];
+		int split_flags;
 		val = dupstring(s);
 		/* If word-splitting is enabled, we ask multsub() to split
 		 * the substituted string at unquoted whitespace.  Then, we
@@ -2627,9 +2639,20 @@ paramsubst(LinkList l, LinkNode n, char 
 		 * keep its array splits, and weird constructs such as
 		 * ${str+"one two" "3 2 1" foo "$str"} to only be split
 		 * at the unquoted spaces. */
-		opts[SHWORDSPLIT] = spbreak;
-		multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
-		opts[SHWORDSPLIT] = ws;
+		if (spbreak) {
+		    split_flags = PF_SHWORDSPLIT;
+		    if (!aspar)
+			split_flags |= PF_SPLIT;
+		} else {
+		    /*
+		     * It's not good enough not passing the flag to use
+		     * SHWORDSPLIT, because when we get to a nested
+		     * paramsubst we need to ignore isset(SHWORDSPLIT).
+		     */
+		    split_flags = PF_NOSHWORDSPLIT;
+		}
+		multsub(&val, split_flags, (aspar ? NULL : &aval),
+			&isarr, NULL);
 		copied = 1;
 		spbreak = 0;
 		/* Leave globsubst on if forced */
@@ -2647,21 +2670,21 @@ paramsubst(LinkList l, LinkNode n, char 
 	case '=':
 	case Equals:
 	    if (vunset) {
-		int ws = opts[SHWORDSPLIT];
 		char sav = *idend;
-		int l;
+		int l, split_flags;
 
 		*idend = '\0';
 		val = dupstring(s);
 		if (spsep || !arrasg) {
-		    opts[SHWORDSPLIT] = 0;
-		    multsub(&val, 0, NULL, &isarr, NULL);
+		    multsub(&val, PF_NOSHWORDSPLIT, NULL, &isarr, NULL);
 		} else {
-		    opts[SHWORDSPLIT] = spbreak;
-		    multsub(&val, spbreak, &aval, &isarr, NULL);
+		    if (spbreak)
+			split_flags = PF_SPLIT|PF_SHWORDSPLIT;
+		    else
+			split_flags = PF_NOSHWORDSPLIT;
+		    multsub(&val, split_flags, &aval, &isarr, NULL);
 		    spbreak = 0;
 		}
-		opts[SHWORDSPLIT] = ws;
 		if (arrasg) {
 		    /* This is an array assignment. */
 		    char *arr[2], **t, **a, **p;
@@ -3118,8 +3141,7 @@ paramsubst(LinkList l, LinkNode n, char 
      * (afterward) may split the joined value (e.g. (s:-:) sets "spsep").  One
      * exception is that ${name:-word} and ${name:+word} will have already
      * done any requested splitting of the word value with quoting preserved.
-     * "ssub" is true when we are called from singsub (via prefork):
-     * it means that we must join arrays and should not split words. */
+     */
     if (ssub || (spbreak && isarr >= 0) || spsep || sep) {
 	if (isarr) {
 	    val = sepjoin(aval, sep, 1);
Index: Src/zsh.h
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/zsh.h,v
retrieving revision 1.179
diff -p -u -r1.179 zsh.h
--- Src/zsh.h	8 Jan 2012 16:02:55 -0000	1.179
+++ Src/zsh.h	10 Feb 2012 20:54:23 -0000
@@ -1648,6 +1648,9 @@ enum {
 #define PF_TYPESET	0x01	/* argument handled like typeset foo=bar */
 #define PF_ASSIGN	0x02	/* argument handled like the RHS of foo=bar */
 #define PF_SINGLE	0x04	/* single word substitution */
+#define PF_SPLIT        0x08    /* explicitly split nested substitution */
+#define PF_SHWORDSPLIT  0x10    /* SHWORDSPLIT in parameter expn */
+#define PF_NOSHWORDSPLIT 0x20   /* SHWORDSPLIT forced off in nested subst */
 
 /*
  * Structure for adding parameters in a module.
Index: Test/D04parameter.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D04parameter.ztst,v
retrieving revision 1.60
diff -p -u -r1.60 D04parameter.ztst
--- Test/D04parameter.ztst	17 Aug 2011 19:00:10 -0000	1.60
+++ Test/D04parameter.ztst	10 Feb 2012 20:54:23 -0000
@@ -255,6 +255,20 @@
 >two
 >words
 
+  (setopt shwordsplit # ensure this doesn't get set in main shell...
+  test_splitting ()
+  {
+    array="one two three"
+    for e in $array; do
+      echo "'$e'"
+    done
+  }
+  test_split_var=
+  : ${test_split_var:=$(test_splitting)}
+  echo "_${test_split_var}_")
+0:SH_WORD_SPLIT inside $(...) inside ${...}
+>_'one' 'two' 'three'_
+
   print -l "${(f)$(print first line\\nsecond line\\nthird line)}"
 0:${(f)$(...)}
 >first line

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug with bash emulation regarding ':'

[Bart Schaefer]

On Feb 10,  9:04pm, Peter Stephenson wrote:
}
} It took some thinking about, but I think the following is about the most
} minimal code.

Nice.

} Even allowing for the fact that explicit splitting is
} separate from sh-wordsplitting (it's a stronger condition), it needed a
} third flag to distinguish the cases...
} 
} - SHWORDSPLIT is on normally.
} - SHWORDSPLIT isn't on, but is required by a recursive substitution.
} - SHWORDSPLIT needs to be forced off by a recursive substitution.

I agree that this covers everything.  (It's a bit convoluted that it's
possible for both PF_SHWORDSPLIT and PF_NOSHWORDSPLIT to be true at
the same time, but <shrug>.)

Incidentally, are we in any danger of conflict with the "PF_" prefix
as used in <socket.h> for protocol families?  PF_UNIX, PF_INET, etc.?
 
} There's now no temporary setting of SHWORDSPLIT itself [...]
} 
} I don't know if this helps with the other issues you mentioned.

I believe it should -- the other issues all stemmed from modifying and
then restoring opts[SHWORDPSLIT], which this eliminates.

Re: Bug with bash emulation regarding ':'

[Peter Stephenson]

On Sat, 11 Feb 2012 13:08:18 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> I agree that this covers everything.  (It's a bit convoluted that it's
> possible for both PF_SHWORDSPLIT and PF_NOSHWORDSPLIT to be true at
> the same time, but <shrug>.)

I'm not sure they can be true at the same time, I think they're always
in different branches and each paramsubst() passes a fresh set of flags
rather than using the ones passed in.
 
> Incidentally, are we in any danger of conflict with the "PF_" prefix
> as used in <socket.h> for protocol families?  PF_UNIX, PF_INET, etc.?

We had quite a lot of PF this and that.  I've made them more readable by
renaming them to PREFORK.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug with bash emulation regarding ':'

[Peter Stephenson]

On Sun, 5 Feb 2012 20:11:33 +0000
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> There's still a difference from bash...

namely that in bash

  test_split_var=
  : ${test_split_var:=$(echo "one
  two
  three")}
  echo "$test_split_var"

outputs the original string, with no splitting on the returned variable
from the command substitution.  zsh does split the command substitution
before the assignment when in sh emulation (the same split array that
later gets substituted onto the command line), so the newlines turn into
spaces when the string is joined up again.

However,

  test_split_var=
  echo ${test_split_var:=$(echo "one
  two
  three")}

does perform splitting, evidently after the assignment has been done
because the value assigned to test_split_var is (of course) as before.
In zsh I'm not sure there's even a corresponding place where we could do
splitting, even if we did something to fix the first difference, since
it would have to be after multsub() returned in order to get the
assignment to work as before, and at that point we don't want to screw
up white space that was originally quoted, about which we no longer have
any information.

I don't know quite what ought to trigger behaviour to change the first
difference, either (in principle I can fix this relatively easily, but
it makes the second problem much more noticeable). It might or might
not be the same condition as what triggers KSH_TYPESET, which in that
case might want to be on in sh emulation even though that's not bash
specific.

At this point I lost the will to live.  This will remain my definitive
position until further notice.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: Bug with bash emulation regarding ':'

[Chet Ramey]

On 2/14/12 12:41 PM, Peter Stephenson wrote:
> On Sun, 5 Feb 2012 20:11:33 +0000
> Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
>> There's still a difference from bash...
> 
> namely that in bash
> 
>   test_split_var=
>   : ${test_split_var:=$(echo "one
>   two
>   three")}
>   echo "$test_split_var"
> 
> outputs the original string, with no splitting on the returned variable
> from the command substitution.  zsh does split the command substitution
> before the assignment when in sh emulation (the same split array that
> later gets substituted onto the command line), so the newlines turn into
> spaces when the string is joined up again.

Bash doesn't split word in ${parameter:OPword} because Posix says not
to:

"word shall be subjected to tilde expansion, parameter expansion, command
substitution, and arithmetic expansion."

A later interpretation adds quote removal to the list.

> 
> However,
> 
>   test_split_var=
>   echo ${test_split_var:=$(echo "one
>   two
>   three")}
> 
> does perform splitting, evidently after the assignment has been done
> because the value assigned to test_split_var is (of course) as before.

The same thing actually happens in both examples; you just look at the
results of the expansion in the second case.  The rhs gets expanded,
but not split, assigned to test_split_var, and that becomes the result
of the parameter expansion.  Those results get split, since the param
expansion doesn't occur within double quotes, and passed to `:' and echo
as arguments.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@case.edu    http://cnswww.cns.cwru.edu/~chet/

Re: Bug with bash emulation regarding ':'

[Peter Stephenson]

On Tue, 14 Feb 2012 18:30:21 -0500
Chet Ramey <chet.ramey@case.edu> wrote:
> Bash doesn't split word in ${parameter:OPword} because Posix says not
> to:
>..
> The same thing actually happens in both examples; you just look at the
> results of the expansion in the second case.  The rhs gets expanded,
> but not split, assigned to test_split_var, and that becomes the result
> of the parameter expansion.  Those results get split, since the param
> expansion doesn't occur within double quotes, and passed to `:' and echo
> as arguments.

Thanks, that's along the lines I supposed but more detailed.

To get this behaviour in zsh we'd have to do the assignment earlier
than we currently do.  The assignments around line 2705 of subst.c would
need to buried inside multsub() in such a way that any splitting owing to
the top-level settings happened later --- which is very difficult since
we rely on the nested substitution to do the splitting we see in the
top-level parameter expansion because it parses it so knows how to split
it.  Whether there are remaining gotchas owing to ordering I don't know.

Yuk.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: Bug with bash emulation regarding ':'

[Bart Schaefer]

On Feb 15, 12:36pm, Peter Stephenson wrote:
}
} To get this behaviour in zsh we'd have to do the assignment earlier
} than we currently do.  The assignments around line 2705 of subst.c would
} need to buried inside multsub() in such a way that any splitting owing to
} the top-level settings happened later --- which is very difficult since
} we rely on the nested substitution to do the splitting we see in the
} top-level parameter expansion because it parses it so knows how to split
} it.  Whether there are remaining gotchas owing to ordering I don't know.

I've been puzzling over this on and off for the last few days and it just
doesn't seem as though it ought to be that hard.

There are some comments in subst.c like this one:

	 * This is where we extract a value (we know now we have
	 * one) into the local parameters for a scalar (val) or
	 * array (aval) value.  TODO: move val and aval into
	 * a structure with a discriminator.  Hope we can make
	 * more things array values at this point and dearrayify later.
	 * v->isarr tells us whether the stuff from down below looks
	 * like an array.

So at first I was thinking that it might be time to do that, so multsub
could for example return both the split and not-split expansions of the
right-side of the assignment and the caller would then use one for the
assignment and the other for the inline expansion.  But then I traced
the call and discovered that the newlines have already been removed and
rejoined as spaces before multsub even returns; the call that handles
this branch doesn't even pass down the aval parameter.

Hence it appears "we rely on the nested substitution to do the splitting
we see in the top-level parameter expansion" is incorrect, at least in
this instance?

Consequently I tried this:

--- ../zsh-forge/current/Src/subst.c	2012-02-12 13:31:49.000000000 -0800
+++ Src/subst.c	2012-02-19 15:19:49.000000000 -0800
@@ -2693,7 +2693,7 @@
 		*idend = '\0';
 		val = dupstring(s);
 		if (spsep || !arrasg) {
-		    multsub(&val, PREFORK_NOSHWORDSPLIT, NULL, &isarr, NULL);
+		    multsub(&val, PREFORK_SINGLE|PREFORK_NOSHWORDSPLIT, NULL,
&isarr, NULL);
 		} else {
 		    if (spbreak)
 			split_flags = PREFORK_SPLIT|PREFORK_SHWORDSPLIT;

This faithfully reproduces the bash behavior in several tests I tried.
In fact it may be that PREFORK_SINGLE is the only thing needed there,
and PREFORK_NOSHWORDSPLIT is extraneous?  (Aside:  My first thought
was to use PREFORK_ASSIGN, but that's only used in filesub.  A more
thorough approach might be to teach multsub about assignment context.)

Have I missed something important?

(Chet Cc'd just for interest, he can probably be dropped from further
discussion on this topic.)

Re: Bug with bash emulation regarding ':'

[Bart Schaefer]

On Feb 19,  3:45pm, Bart Schaefer wrote:
}
} Consequently I tried this:
} 
}  		if (spsep || !arrasg) {
} -		    multsub(&val, PREFORK_NOSHWORDSPLIT, NULL, &isarr, NULL);
} +		    multsub(&val, PREFORK_SINGLE|PREFORK_NOSHWORDSPLIT, NULL,
} &isarr, NULL);
} 
} This faithfully reproduces the bash behavior in several tests I tried.

I should note that this wouldn't have been anywhere near this easy if
PWS hadn't done the background work on passing the bitflags.

Although all tests still pass (except the new one introduced by 30203
which must be tweaked), I should note the above changes the "native"
zsh behavior as well.  I woudn't imagine this is going to cause much
trouble -- I found no uses of ${...=...} in the default $fpath, which
includes the whole completion suite, etc. -- but it does mean we might
want to make it conditional upon something.  For example:

Index: Src/subst.c
--- ../zsh-forge/current/Src/subst.c	2012-02-12 13:31:49.000000000 -0800
+++ Src/subst.c	2012-02-20 00:32:30.000000000 -0800
@@ -2693,7 +2693,12 @@
 		*idend = '\0';
 		val = dupstring(s);
 		if (spsep || !arrasg) {
-		    multsub(&val, PREFORK_NOSHWORDSPLIT, NULL, &isarr, NULL);
+		    /* POSIX requires PREFORK_SINGLE semantics here, but
+		     * traditional zsh used PREFORK_NOSHWORDSPLIT.  Base
+		     * behavior on caller choice of PREFORK_SHWORDSPLIT. */
+		    multsub(&val,
+			    spbreak ? PREFORK_SINGLE : PREFORK_NOSHWORDSPLIT,
+			    NULL, &isarr, NULL);
 		} else {
 		    if (spbreak)
 			split_flags = PREFORK_SPLIT|PREFORK_SHWORDSPLIT;

I'd be just as happy going with PREFORK_SINGLE unconditionally, I think,
but I hesitate to change anything arbitrarily.  Either way, here is the
tweak to the test suite:

Index: Test/D04parameter.ztst
--- ../zsh-forge/current/Test/D04parameter.ztst	2012-02-12 13:31:49.000000000 -0800
+++ Test/D04parameter.ztst	2012-02-19 23:46:19.000000000 -0800
@@ -267,7 +267,9 @@
   : ${test_split_var:=$(test_splitting)}
   echo "_${test_split_var}_")
 0:SH_WORD_SPLIT inside $(...) inside ${...}
->_'one' 'two' 'three'_
+>_'one'
+>'two'
+>'three'_
 
   print -l "${(f)$(print first line\\nsecond line\\nthird line)}"
 0:${(f)$(...)}

Re: Bug with bash emulation regarding ':'

[Peter Stephenson]

On Sun, 19 Feb 2012 15:45:15 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Hence it appears "we rely on the nested substitution to do the splitting
> we see in the top-level parameter expansion" is incorrect, at least in
> this instance?

That's good.  Maybe we're only relying on it to get the assigned value,
after all, so the splitting is done in the same order as bash (just
hitherto with different options applied at each stage).

> In fact it may be that PREFORK_SINGLE is the only thing needed there,
> and PREFORK_NOSHWORDSPLIT is extraneous?

The basic behaviour is certainly given with PREFORK_SINGLE only (spbreak
initialisation around line 1540); there's one oddity where we'll use
SHWORDSPLIT on its own to determine the default for (@) behaviour if not
forced on around line 1652.  That's rather confusing and my previous
change may already have had some impact on this, since we've reduced the
number of cases where SHWORDSPLIT is forced on.

> Although all tests still pass (except the new one introduced by 30203
> which must be tweaked), I should note the above changes the "native"
> zsh behavior as well.  I woudn't imagine this is going to cause much
> trouble -- I found no uses of ${...=...} in the default $fpath, which
> includes the whole completion suite, etc. -- but it does mean we might
> want to make it conditional upon something.  For example:

> +			    spbreak ? PREFORK_SINGLE : PREFORK_NOSHWORDSPLIT,

I suppose that's as good as anything.  Goodness knows how to document
this.  I'm not sure I really understand how the ramifications of one
option cause so much divergence internally, but as far as I can see (and
I can't, really) that should cover the two obvious cases where
SHWORDSPLIT is or is not on at the top level.  Whether it covers
explicit splitting in nested substitutions or (@) processing is another
matter.

However, given we don't tend to rely on ${...=...}, and the shenanigans
with parameter flags don't apply to POSIX, it doesn't seem worth doing
any more research, so I think this is probably as good as we need.
Thanks.

It might be worth folding this in and making 4.3.17 immediately, there's
no real point in releasing only a partial fix.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog

Re: Bug with bash emulation regarding ':'

[Bart Schaefer]

On Feb 20, 10:59am, Peter Stephenson wrote:
} Subject: Re: Bug with bash emulation regarding ':'
}
} On Sun, 19 Feb 2012 15:45:15 -0800
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > In fact it may be that PREFORK_SINGLE is the only thing needed there,
} > and PREFORK_NOSHWORDSPLIT is extraneous?
} 
} The basic behaviour is certainly given with PREFORK_SINGLE only (spbreak
} initialisation around line 1540); there's one oddity where we'll use
} SHWORDSPLIT on its own to determine the default for (@) behaviour if not
} forced on around line 1652. [...]
} 
} > +			    spbreak ? PREFORK_SINGLE : PREFORK_NOSHWORDSPLIT,

Hrm.  That may imply that we should do

    spbreak ? (PREFORK_SINGLE|PREFORK_SHWORDPSLIT) : PREFORK_NOSHWORDSPLIT

As you later pointed out, this would only affects some strange cases
like ${(@)=...=...}, because SINGLE beats SHWORDSPLIT at line 1540; in
those cases the workaround must previously always have been to use
additional nested expansions to turn the flags on in the right places.

So I think we're safe with just SINGLE there.

} I suppose that's as good as anything.  Goodness knows how to document
} this.  I'm not sure I really understand how the ramifications of one
} option cause so much divergence internally

Well, it's because we've overloaded its meaning from the original one
of "split words on IFS" to instead mean "split words on IFS in exactly
the same circumstances and manner the POSIX shell would do so."  Plus
keeping the non-POSIXy behavior of the traditional shell when it's not
set, which is the real divergence.

} It might be worth folding this in and making 4.3.17 immediately, there's
} no real point in releasing only a partial fix.

I'll commit shortly if you haven't already.

Re: Bug with bash emulation regarding ':'

[Felipe Contreras]

On Sun, Jan 29, 2012 at 4:46 AM, Felipe Contreras
<felipe.contreras@gmail.com> wrote:
> Seems like there's a bug in bash emulation:
>
> ---
> #!sh
>
> _test ()
> {
>        array="one two three"
>        for e in $array
>        do
>                echo "'$e'"
>        done
> }
>
> l=
> : ${l:=$(_test)}
> echo "_${l}_"
> ---

Actually, the same with:

 true ${l:=$(_test)}
 blah=${l:=$(_test)}

Or pretty much anything.

-- 
Felipe Contreras