Re: Fix testsuite errors due to shell quoted parameter expansion issue.

Re: Fix testsuite errors due to shell quoted parameter expansion issue.

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2156 bytes --]

[adding autoconf to document some shell bugs]

On 08/03/2010 02:32 PM, Ralf Wildenhues wrote:
> Interesting shell unportability:
> 
> $ bash -c 'f=" val" e=; echo "$e"$f'
> val
> $ ksh -c 'f=" val" e=; echo "$e"$f'
>  val
> 
> ksh93, dash, zsh all do it like ksh.  Is that a bug in bash?

Yes; adding bug-bash accordingly.  According to POSIX:

http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_05

"After parameter expansion ( Parameter Expansion ), command substitution
( Command Substitution ), and arithmetic expansion ( Arithmetic
Expansion  ), the shell shall scan the results of expansions and
substitutions that did not occur in double-quotes for field splitting
and multiple fields can result."

Since $f is not quoted, its expansion must undergo field splitting.  But
since "$e" is quoted, it must not be elided even though empty.  The
result must be _two_ fields, as if you had done "echo '' 'val'".

But it is _also_ a bug in zsh; adding zsh-workers accordingly.

$ zsh -cvx 'f=" val" e=; echo "$e"$f'
+zsh:1> f=' val' e=''
+zsh:1> echo ' val'
 val

Oops - zsh only passed one argument to echo, with a leading space,
instead of passing an empty argument and letting echo supply the space.
 ksh93, pdksh, and dash get it right (although dash doesn't use quotes
in -vx output, hence my use of n() to force things to tell; n() is
another way to expose the bash and zsh bugs).

$ ksh -cvx 'n() { echo $#; }; f=" val" e=; n "$e"$f'
n() { echo $#; }; f=" val" e=; n ""$f+ f=' val'
+ e=''
+ n '' val
+ echo 2
2

And meanwhile, I found a ksh93 parsing bug (don't know where to report
that):

$ ksh -c 'a(){ echo hi; }; a'
ksh: syntax error at line 1: `}' unexpected
$ ksh -c 'a() { echo hi; }; a'
hi
$ bash -c 'a(){ echo hi; }; a'
hi
$ /bin/sh -c 'a(){ echo hi; }; a'
hi

ksh is the only shell that requires a space between the ) and {, even
though the ) is a metacharacter and should not need trailing space (even
Solaris /bin/sh got that right).

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Fix testsuite errors due to shell quoted parameter expansion issue.

[Mikael Magnusson]

On 3 August 2010 22:55, Eric Blake <eblake@redhat.com> wrote:
> [adding autoconf to document some shell bugs]
>
> On 08/03/2010 02:32 PM, Ralf Wildenhues wrote:
>> Interesting shell unportability:
>>
>> $ bash -c 'f=" val" e=; echo "$e"$f'
>> val
>> $ ksh -c 'f=" val" e=; echo "$e"$f'
>>  val
>>
>> ksh93, dash, zsh all do it like ksh.  Is that a bug in bash?
>
> Yes; adding bug-bash accordingly.  According to POSIX:
>
> http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_05
>
> "After parameter expansion ( Parameter Expansion ), command substitution
> ( Command Substitution ), and arithmetic expansion ( Arithmetic
> Expansion  ), the shell shall scan the results of expansions and
> substitutions that did not occur in double-quotes for field splitting
> and multiple fields can result."
>
> Since $f is not quoted, its expansion must undergo field splitting.  But
> since "$e" is quoted, it must not be elided even though empty.  The
> result must be _two_ fields, as if you had done "echo '' 'val'".
>
> But it is _also_ a bug in zsh; adding zsh-workers accordingly.
>
> $ zsh -cvx 'f=" val" e=; echo "$e"$f'
> +zsh:1> f=' val' e=''
> +zsh:1> echo ' val'
>  val
>
> Oops - zsh only passed one argument to echo, with a leading space,
> instead of passing an empty argument and letting echo supply the space.
>  ksh93, pdksh, and dash get it right (although dash doesn't use quotes
> in -vx output, hence my use of n() to force things to tell; n() is
> another way to expose the bash and zsh bugs).

zsh doesn't do word splitting by default, you can enable it with the = modifier:
% zsh -fcvx 'f=" val" e=; echo "$e"$=f'
+zsh:1> f=' val' e=''
+zsh:1> echo '' val
 val

does what you want

Alternatively you can make zsh try to be closer to sh by setting
argv[0] to sh when executing it, or running 'emulate sh' as the first
command (and possibly other ways I don't know about):
% zsh -fcvx 'emulate sh;f=" val" e=; echo "$e"$f'
+zsh:1> emulate sh
+zsh:1> f=' val' e=''
+zsh:1> echo '' val
 val

There's also --shwordsplit for this specific case:
% zsh --shwordsplit -fcvx 'f=" val" e=; echo "$e"$f'
+zsh:1> f=' val' e=''
+zsh:1> echo '' val
 val

(the -f above only avoids loading my .zshenv which would spam my output)

-- 
Mikael Magnusson

Re: Fix testsuite errors due to shell quoted parameter expansion issue.

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

[trimming the cc a bit]

On 08/03/2010 03:21 PM, Mikael Magnusson wrote:
>> But it is _also_ a bug in zsh; adding zsh-workers accordingly.
>>
>> $ zsh -cvx 'f=" val" e=; echo "$e"$f'
>> +zsh:1> f=' val' e=''
>> +zsh:1> echo ' val'
>>  val
>>
>> Oops - zsh only passed one argument to echo, with a leading space,
>> instead of passing an empty argument and letting echo supply the space.
>>  ksh93, pdksh, and dash get it right (although dash doesn't use quotes
>> in -vx output, hence my use of n() to force things to tell; n() is
>> another way to expose the bash and zsh bugs).
> 
> zsh doesn't do word splitting by default, you can enable it with the = modifier:
> % zsh -fcvx 'f=" val" e=; echo "$e"$=f'
> +zsh:1> f=' val' e=''
> +zsh:1> echo '' val
>  val
> 
> does what you want

Thanks for the information.

> 
> Alternatively you can make zsh try to be closer to sh by setting
> argv[0] to sh when executing it, or running 'emulate sh' as the first
> command (and possibly other ways I don't know about):
> % zsh -fcvx 'emulate sh;f=" val" e=; echo "$e"$f'
> +zsh:1> emulate sh
> +zsh:1> f=' val' e=''
> +zsh:1> echo '' val
>  val

In the context of portable shell programming, and autoconf in
particular, we already _always_ force zsh into 'emulate sh' mode, so
that we don't trip up on non-POSIX incompatibilities like this behavior
of word splitting.  So it should not be a problem for autoconf, other
than to remind users that portable programming requires the use of
'emulate sh' in scripts that don't pick up autoconf's automatic shell
sanitization prelude.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: Fix testsuite errors due to shell quoted parameter expansion issue.

[Peter Stephenson]

On Tue, 03 Aug 2010 14:55:36 -0600
Eric Blake <eblake@redhat.com> wrote:
> But it is _also_ a bug in zsh; adding zsh-workers accordingly.
> 
> $ zsh -cvx 'f=" val" e=; echo "$e"$f'
> +zsh:1> f=' val' e=''
> +zsh:1> echo ' val'
>  val
> 
> Oops - zsh only passed one argument to echo, with a leading space,
> instead of passing an empty argument and letting echo supply the space.

That's not a bug, you haven't enabled the SH_WORD_SPLIT option.  Try
in sh compatibility mode.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

RE: Fix testsuite errors due to shell quoted parameter expansion issue.

[John Lumby]

[-- Attachment #1: Type: text/plain, Size: 3681 bytes --]


Re the statement

    since "$e" is quoted, it must not be elided

I don't think that is correct.   In fact "$e" *must* be elided  -  because you juxtaposed it with the next token.    eliding and quoting are orthogonal.

What you appear to be be expecting is that the effect of quoting one token should somehow have some effect on interpretation of a special character (blank) in a subsequent unquoted token.   I don't think that is expected behaviour.      The quoting of $e has the effect of quoting any special characters inside *it*, of which there are none.     your blank is unquoted.    Then word-spilling proceeds.

What I find interesting is this

bash -c 'declare -i length; f=" val" e=;concat="$e"$f; length=${#concat}; echo "length= $length"'
4

John Lumby

> Date: Tue, 3 Aug 2010 23:21:02 +0200
> From: mikachu@gmail.com
> To: eblake@redhat.com
> CC: zsh-workers@zsh.org; Autoconf@gnu.org; libtool-patches@gnu.org
> Subject: Re: Fix testsuite errors due to shell quoted parameter expansion 	issue.
> 
> On 3 August 2010 22:55, Eric Blake <eblake@redhat.com> wrote:
> > [adding autoconf to document some shell bugs]
> >
> > On 08/03/2010 02:32 PM, Ralf Wildenhues wrote:
> >> Interesting shell unportability:
> >>
> >> $ bash -c 'f=" val" e=; echo "$e"$f'
> >> val
> >> $ ksh -c 'f=" val" e=; echo "$e"$f'
> >>  val
> >>
> >> ksh93, dash, zsh all do it like ksh.  Is that a bug in bash?
> >
> > Yes; adding bug-bash accordingly.  According to POSIX:
> >
> > http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_05
> >
> > "After parameter expansion ( Parameter Expansion ), command substitution
> > ( Command Substitution ), and arithmetic expansion ( Arithmetic
> > Expansion  ), the shell shall scan the results of expansions and
> > substitutions that did not occur in double-quotes for field splitting
> > and multiple fields can result."
> >
> > Since $f is not quoted, its expansion must undergo field splitting.  But
> > since "$e" is quoted, it must not be elided even though empty.  The
> > result must be _two_ fields, as if you had done "echo '' 'val'".
> >
> > But it is _also_ a bug in zsh; adding zsh-workers accordingly.
> >
> > $ zsh -cvx 'f=" val" e=; echo "$e"$f'
> > +zsh:1> f=' val' e=''
> > +zsh:1> echo ' val'
> >  val
> >
> > Oops - zsh only passed one argument to echo, with a leading space,
> > instead of passing an empty argument and letting echo supply the space.
> >  ksh93, pdksh, and dash get it right (although dash doesn't use quotes
> > in -vx output, hence my use of n() to force things to tell; n() is
> > another way to expose the bash and zsh bugs).
> 
> zsh doesn't do word splitting by default, you can enable it with the = modifier:
> % zsh -fcvx 'f=" val" e=; echo "$e"$=f'
> +zsh:1> f=' val' e=''
> +zsh:1> echo '' val
>  val
> 
> does what you want
> 
> Alternatively you can make zsh try to be closer to sh by setting
> argv[0] to sh when executing it, or running 'emulate sh' as the first
> command (and possibly other ways I don't know about):
> % zsh -fcvx 'emulate sh;f=" val" e=; echo "$e"$f'
> +zsh:1> emulate sh
> +zsh:1> f=' val' e=''
> +zsh:1> echo '' val
>  val
> 
> There's also --shwordsplit for this specific case:
> % zsh --shwordsplit -fcvx 'f=" val" e=; echo "$e"$f'
> +zsh:1> f=' val' e=''
> +zsh:1> echo '' val
>  val
> 
> (the -f above only avoids loading my .zshenv which would spam my output)
> 
> -- 
> Mikael Magnusson
> 
> _______________________________________________
> Autoconf mailing list
> Autoconf@gnu.org
> http://lists.gnu.org/mailman/listinfo/autoconf
 		 	   		  

Re: Fix testsuite errors due to shell quoted parameter expansion issue.

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2379 bytes --]

[please don't top-post on technical lists]

On 08/04/2010 09:18 AM, John Lumby wrote:
> 
> Re the statement
> 
>     since "$e" is quoted, it must not be elided
> 
> I don't think that is correct.   In fact "$e" *must* be elided  -  because you juxtaposed it with the next token.    eliding and quoting are orthogonal.

Your statement is confusing to me.  Here, in more detail, is exactly
what I meant (and what I though I implied in the first case):

The word in question is '"$e"$f' before either parameter substitution,
or word splitting, or quote elision.

After parameter substitution, you are left with '"" val', where a
portion of the string was originally quoted, but you also had a portion
of the string that was unquoted.  Only the ' val' portion of the result
is subject to word splitting.

Because "$e" is quoted, the resulting empty string must not be elided
during word splitting.

Therefore, the result after word splitting MUST be the two words '""'
and 'val'; then you apply quote elision and are left with two words: ''
and 'val'.

> 
> What you appear to be be expecting is that the effect of quoting one token should somehow have some effect on interpretation of a special character (blank) in a subsequent unquoted token.   I don't think that is expected behaviour.      The quoting of $e has the effect of quoting any special characters inside *it*, of which there are none.     your blank is unquoted.    Then word-spilling proceeds.

No, what I expect is that within a single word, the quoted portion of
the word must not be elided during word splitting, except in the special
case of "$@".

> 
> What I find interesting is this
> 
> bash -c 'declare -i length; f=" val" e=;concat="$e"$f; length=${#concat}; echo "length= $length"'
> 4

Why do you find that interesting?  There is NO word splitting in
assignment contexts, so the effect of
concat="$e"$f
is the same as if you had done
concat=$e$f
or
concat="$e$f"
in all three cases, concat is assigned the value ' val', which is indeed
length 4.

But it is unrelated to the bash bug at hand, which is that when "$e"$f
is in a context subject to word splitting, then it must result in the
two words '' and 'val', not the single word 'val'.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

RE: Fix testsuite errors due to shell quoted parameter expansion issue.

[John Lumby]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]


Eric Blake wrote> On 08/04/2010 09:18 AM, John Lumby wrote:
> 
> > 
> > Re the statement
> > 
> >     since "$e" is quoted, it must not be elided
> > 
> > I don't think that is correct.   In fact "$e" *must* be elided  -
> >  because you juxtaposed it with the next token.    eliding and quoting are orthogonal.
>  
> Your statement is confusing to me.  Here, in more detail, is exactly
> what I meant (and what I though I implied in the first case):
>  
>  ...
>  
> Because "$e" is quoted, the resulting empty string must not be elided
> during word splitting.
>

I see -   thanks.   I was not aware of that effect of quoting.
But I see you are correct.

>  
>  ...
>  
> Why do you find that interesting?  There is NO word splitting in
> assignment contexts, so the effect of

Again I stand corrected,   thanks.  I thought an unquoted blank would terminate an assignment value.
Maybe I was confusing tokenization before substitution with word splitting after substitution.


 		 	   		  

Re: Fix testsuite errors due to shell quoted parameter expansion issue.

[Chet Ramey]

On 8/3/10 4:55 PM, Eric Blake wrote:
> [adding autoconf to document some shell bugs]
> 
> On 08/03/2010 02:32 PM, Ralf Wildenhues wrote:
>> Interesting shell unportability:
>>
>> $ bash -c 'f=" val" e=; echo "$e"$f'
>> val
>> $ ksh -c 'f=" val" e=; echo "$e"$f'
>>  val
>>
>> ksh93, dash, zsh all do it like ksh.  Is that a bug in bash?
> 
> Yes; adding bug-bash accordingly.  According to POSIX:

Thanks; this was an easy fix.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU    chet@case.edu    http://cnswww.cns.cwru.edu/~chet/