Expansion/quoting quirks

Expansion/quoting quirks

[Thorsten Meinecke]

Hello,

albeit Zoltan's claims of improved sh-compatibility are almost true,
there is still a serious bug in his release that prevents the ${1+"$@"}
idiom from doing the right thing, because quoted arguments with embedded
whitespace are split despite their quotes. To demonstrate this bug and a
few other possible problems I threw together a tiny test script, which is
appended below.

Incidentally, the only shell I have access to, which passes all the tests
of my script, is `PD KSH v5.1.3 94/12/22'. Even bash has trouble with null
args. Other contestants were 2.6-beta11-test12 and 2.6-beta11-test9-hzoli11,
each of them invoked as `./sh -f' and `./ksh -f', respectively.

Save as `expbug', set symlinks from sh (and ksh) to zsh, and execute with
`expbug ./sh -f', for example. The testee is running as a coprocess, and
its output will be compared to the expected output, which is on the lines
beginning with `#%'. Sample output (for hzoli) looks like this:


  $ ~/expbug ./ksh -f
  Testing ./ksh -f ...
  1c1
  < \
  ---
  > 
  7c7
  < argc=1, argv=( 'a b' )
  ---
  > argc=2, argv=( 'a' 'b' )
  9a10
  > argc=4, argv=( 'a' 'b' '' 'c' )
  11,12c12
  < argc=3, argv=( 'a b' '' 'c' )
  < argc=3, argv=( 'a b' '' 'c' )
  ---
  > zsh: closing brace expected

  Not OK, output left in file /tmp/zshtest18454


If I'm missing something, I'm sure, you'll tell me. :-)
--Thorsten

==================== cut here ==================== 
#!/bin/zsh -f
# expbug: a few (k)sh-compatibility tests
( 
    TEMP=${TMPPREFIX}test$$
    if (( $# ))
    then
        WHAT=($@)
    else
        WHAT=(zsh -f)
    fi
    echo "Testing $WHAT ..."
    coproc 2>&1 $WHAT
    sed -ne "/^#/!p" -e "s/^#%[ ]*//w $TEMP" >&p
    if <&p cat -v | diff $TEMP -
    then
        echo OK.
        rm -f $TEMP
        exit 0
    else
        echo "\nNot OK, output left in file $TEMP"
        exit 1
    fi
) <<\EOF

# Command substitution eats too much backslashes
#% \
echo `echo \\\\`    # broken in hzoli, and in vanilla zsh if invoked as (k)sh
#% \
echo $(echo \\\\)
#% \
echo "$(echo \\\\)" # sh and ksh seem to differ here (bash would give `\\')

# Single quotes aren't recognized (fixed in hzoli)
#% $foo
foo=10 x=foo
y='$'$x
echo $y

# Backslash ignored between single quotes (fixed in hzoli)
#% \$x
echo $(echo '\$x')


# nargs(): A function which neatly prints the arguments passed to it
nargs () {
  echo -n "argc=$#, argv=( "
  while [ $# -ne 0 ]; do
    echo -n "'$1' "
    shift
  done
  echo ")"
}

#% argc=2, argv=( 'a' 'b' )
nargs ${undef-a b}
#% argc=1, argv=( 'a b' )
nargs ${undef-"a b"}             # vanilla + hzoli: shouldn't split here
#% argc=1, argv=( 'a b' )
nargs "${undef-a b}"

# The following are broken in vanilla

set "a b" "" c

#% argc=3, argv=( 'a' 'b' 'c' )
nargs ${undef-$@}                # This works in hzoli's sh-mode 

#% argc=3, argv=( 'a b' '' 'c' )
nargs ${undef-"$@"}              # hzoli: 'a b' shouldn't split into 'a' 'b'

#% argc=3, argv=( 'a b' '' 'c' )
nargs "${undef-$@}"              # This works also in hzoli's sh-mode

#% argc=3, argv=( 'a b' '' 'c' )
nargs "${undef-"$@"}"            # hzoli: zsh: closing brace expected

exit
EOF
==================== cut here ==================== 

Re: Expansion/quoting quirks

[Barton E. Schaefer]

On Nov 5,  2:00pm, Thorsten Meinecke wrote:
} Subject: Expansion/quoting quirks
}
} its output will be compared to the expected output, which is on the lines
} beginning with `#%'.
} 
} 
} # Backslash ignored between single quotes (fixed in hzoli)
} #% \$x
} echo $(echo '\$x')

This one seems really strange to me.  Why is $(foo) different from `foo`
in ksh?  I think I prefer zsh's behavior, even if it isn't compatible.

    $ echo $(echo '\$x')
    \$x
    $ echo `echo '\$x'`
    $x
    $

Look, ksh isn't even consistent about it!  Why should that $ cause a
magically different behavior?

    $ echo $(echo '\x')
    \x
    $ echo `echo '\x'`
    \x
    $

(This is ksh88, I believe, in case that matters.)

-- 
Bart Schaefer                     Vice President, Technology, Z-Code Software
schaefer@z-code.com                  Division of NCD Software Corporation
http://www.well.com/www/barts

Re: Expansion/quoting quirks

[Zoltan Hidvegi]

> This one seems really strange to me.  Why is $(foo) different from `foo`
> in ksh?  I think I prefer zsh's behavior, even if it isn't compatible.
> 
>     $ echo $(echo '\$x')
>     \$x
>     $ echo `echo '\$x'`
>     $x
>     $
> 
> Look, ksh isn't even consistent about it!  Why should that $ cause a
> magically different behavior?
> 
>     $ echo $(echo '\x')
>     \x
>     $ echo `echo '\x'`
>     \x
>     $

Here is the relevant part from the ksh93 manual:

   Command Substitution.
       The standard output from a command enclosed in parentheses
       preceded  by  a  dollar  sign  (  $() ) or a pair of grave
       accents (``) may be used as part or all of a word;  trail-
       ing new-lines are removed.  In the second (obsolete) form,
       the string between the quotes  is  processed  for  special
       quoting  characters  before  the  command is executed (see
       Quoting below).  The command substitution $(cat file)  can
       be replaced by the equivalent but faster $(<file).

                                                ...  Inside grave
       quote marks (``), \ quotes the characters \, `, and $.  If
       the  grave  quotes occur within double quotes, then \ also
       quotes the character ".

The baseline zsh implement this behaviour.  However when it is invoked as sh
or ksh, it sets the glob_subst option which.  But glob_subst is buggy in the
baseline, since the result of the expansion is not only globbed but also
expanded using parameter expansion.

E.g. foo='$foo' ; echo $foo

causes an infinite loop when glob_subst is set.

Cheers,

   Zoltan

Re: Expansion/quoting quirks

[Zoltan Hidvegi]

Thorsten Meinecke wrote:
> echo `echo \\\\`    # broken in hzoli, and in vanilla zsh if invoked as (k)sh

Yes, that's really broken.  However echo is not good for testing here since
shells are diffen in how they interpret escape sequences in echo.  It's better
to alias echo to 'print -r --'.  With zsh it is enough to set the bsdecho
option.  This bug appeared with the input patches from Peter.  The following
happens:  the \\\\ within `...` is parsed as Bnull\Bnull\.  The lexer is
called again with that and it thinks that the first \ quotes the Bnull.  The
last \ then a parse error.  Below is a patch to input.c to drop tokens from
the input.  These returned tokens caused some other bugs earlier and it can be
dangerous when a script contains some tokens.

> echo "$(echo \\\\)" # sh and ksh seem to differ here (bash would give `\\')

sh should give two slashes.  The difference is probably in the escape handling
of sh.

> nargs ${undef-"a b"}             # vanilla + hzoli: shouldn't split here

That's difficult.  sh_word_split splits the result of a parameter expansion.
Here the result is 'a b' which is split to 'a' 'b'.

> #% argc=3, argv=( 'a b' '' 'c' )
> nargs ${undef-"$@"}              # hzoli: 'a b' shouldn't split into 'a' 'b'

Same as the previous example.

> #% argc=3, argv=( 'a b' '' 'c' )
> nargs "${undef-"$@"}"            # hzoli: zsh: closing brace expected

That's because the second " closes the first.  It would be easy to fix it.

My problem is that I do not know what is the standard behaviour here.  My
library does not have the relevant POSIX papers.

It would be important to know how to parse these things.  It seems that the
lexer should be called on the body of ${...-...}.  I'll try to fix these if
someone tells me what the standards say here.  I have ksh93.  May I assume
that ksh93 behaviour is the standard?

The most difficult part here is ${...##...}.  Here the body should be
interpreted as a pattern.  Here the expanded body shoud be parsed again for
quotes.  E.g. foo='te\s\t' bar='\s\t' ; echo ${foo%%$bar} does not removes the
tail of foo since \ only escapes the s and t.  But foo='te"st"' bar='"??"'
echo ${foo%$bar} does remove the tail.

Bye,

  Zoltan

diff -c Src/input.c~ Src/input.c
*** Src/input.c~	Sat Nov  4 09:47:43 1995
--- Src/input.c	Mon Nov  6 17:50:17 1995
***************
*** 109,115 ****
  	if (inbufleft) {
  	    inbufleft--;
  	    inbufct--;
! 	    return lastc = (unsigned)*inbufptr++;
  	}
  	/*
  	 * No characters in input buffer.
--- 109,118 ----
  	if (inbufleft) {
  	    inbufleft--;
  	    inbufct--;
! 	    lastc = (unsigned)*inbufptr++;
! 	    if (itok(lastc))
! 		continue;
! 	    return lastc;
  	}
  	/*
  	 * No characters in input buffer.