quoting bug

quoting bug

[Raymond Nijssen]

zsh fails to break up strings properly where other shells do the right thing.

This happens with beta21 and all previous versions I have.

Best Regards,
-Raymond

----------------------------------------------------------------------

123[krait:~/tmp/zshbug]% <bug.zsh 

FOO="ls -C"

BAR="`$FOO -F`"

echo $BAR

124[krait:~/tmp/zshbug]% ksh ./bug.zsh
bug.zsh

125[krait:~/tmp/zshbug]% bash ./bug.zsh 
bug.zsh

126[krait:~/tmp/zshbug]% sh ./bug.zsh 
bug.zsh

127[krait:~/tmp/zshbug]% /bin/posix/sh ./bug.zsh 
bug.zsh

128[krait:~/tmp/zshbug]% zsh -f ./bug.zsh 
./bug.zsh: command not found: ls -C [4]

----------------------------------------------------------------------

Re: quoting bug

[Zefram]

>zsh fails to break up strings properly where other shells do the right thing.

setopt SH_WORD_SPLIT

-zefram

Re: quoting bug

[Bruce Stephens]

>>>>> "Zefram" == Zefram  <A.Main@dcs.warwick.ac.uk> writes:

>> zsh fails to break up strings properly where other shells do the
>> right thing.

> setopt SH_WORD_SPLIT

What does POSIX say?

Re: quoting bug

[Zoltan Hidvegi]

> >>>>> "Zefram" == Zefram  <A.Main@dcs.warwick.ac.uk> writes:
> 
> >> zsh fails to break up strings properly where other shells do the
> >> right thing.
> 
> > setopt SH_WORD_SPLIT
> 
> What does POSIX say?

Zsh must be invoked as sh/ksh if you want POSIX.  Even is sh/ksh mode there
are some minor incompatibilities between zsh and POSIX but these rarely
cause problems.  Also the zsh documentation does not claim that zsh
conforms to POSIX.  On my computer /bin/sh is a link to zsh and it works
fine.

Zoltan

Re: quoting bug

[Zefram]

>> setopt SH_WORD_SPLIT
>
>What does POSIX say?

POSIX requires the SH_WORD_SPLIT behaviour.  It also requires field
splitting in normal words, as well as the result of expansions:

% setopt SH_WORD_SPLIT
% IFS=x
% echo fooxbar
fooxbar

$ IFS=x
$ echo fooxbar
foo bar

I think we should implement this behaviour -- when SH_WORD_SPLIT is
set, of course.  It should be very easy.  I also think we should retain
the current default, as it's much nicer behaviour.

-zefram

Re: quoting bug, and comparisons with ksh93

[Bruce Stephens]

>>>>> "Zoltan" == Zoltan Hidvegi <hzoli@cs.elte.hu> writes:

>> >>>>> "Zefram" == Zefram <A.Main@dcs.warwick.ac.uk> writes:
>> 
>> > setopt SH_WORD_SPLIT
>> 
>> What does POSIX say?

> Zsh must be invoked as sh/ksh if you want POSIX.  Even is sh/ksh
> mode there are some minor incompatibilities between zsh and POSIX
> but these rarely cause problems.  Also the zsh documentation does
> not claim that zsh conforms to POSIX.  On my computer /bin/sh is a
> link to zsh and it works fine.

That sounds fine then.  It would be a shame to have SH_WORD_SPLIT be
the default: the current position is more logical, even if it does
sometimes fool me.

Moving to another topic, I was reading about ksh93 in the recent AT&T
book on reusable UNIX software.  They've extended the pattern syntax
so it's equivalent to egrep style regexps (my guess is that zsh's are
pretty close), and added the nice touch that some option of print will
convert from a regexp to a ksh pattern.  There are lots of handy
things for writing scripts, but I tend to use zsh only interactively,
so for all I know some of these may already exist, although some
definitely don't: I can't have a variable called a.fred, for example.

One interesting, and probably not too hard feature (now that TCL and
Perl have done it) is dynamic loading of commands (on systems which
support it).  The idea is that they've written a library of routines
wc(argc, argv), basename(argc, argv) and so on which are all POSIX
implementations of the commands of the same names (so you can wrap a
10 line main around them to get the POSIX commands), and if you want
(and your machine supports it), you can arrange for ksh to dynamically
load this library, making lots of commands builtins.  It would be a
nice addition to zsh, I think: the mechanism that Perl uses is more
complex, because of its different types, but TCL's extensions are just
bunches of routines which take (int argc, char **argv), so presumably
the mechanism it uses to dynamically load things could be borrowed.

Re: quoting bug

[Zoltan Hidvegi]

> >> setopt SH_WORD_SPLIT
> >
> >What does POSIX say?
> 
> POSIX requires the SH_WORD_SPLIT behaviour.  It also requires field
> splitting in normal words, as well as the result of expansions:
> 
> % setopt SH_WORD_SPLIT
> % IFS=x
> % echo fooxbar
> fooxbar

No.  POSIX does not allow field splitting here:

"IFS is used for performing field splitting on the results of parameter
 and command substitution; it is not used for splitting all fields.
 Previous versions of the shell used it for splitting all fields during
 field splitting, but this has severe problems because the shell can no
 longer parse its own script.  There are also important security
 implications caused by this behavior.  All useful applications of IFS use
 it for parsing input of the read utility and for splitting the results of
 parameter and command substitution.  New versions of the shell have fixed
 this bug, and POSIX.2 requires the corrected behavior."

> I think we should implement this behaviour -- when SH_WORD_SPLIT is
> set, of course.  It should be very easy.  I also think we should retain
> the current default, as it's much nicer behaviour.

Yes, it can be implemented easily by adding IBLANK type to IFS elements but
I still think that it is unnecessary.  And there may be hidden problems as
this change affects the parser.

Zoltan

Re: quoting bug, and comparisons with ksh93

[Zoltan Hidvegi]

> Moving to another topic, I was reading about ksh93 in the recent AT&T
> book on reusable UNIX software.  They've extended the pattern syntax
> so it's equivalent to egrep style regexps (my guess is that zsh's are
> pretty close), and added the nice touch that some option of print will
> convert from a regexp to a ksh pattern.  There are lots of handy
> things for writing scripts, but I tend to use zsh only interactively,
> so for all I know some of these may already exist, although some
> definitely don't: I can't have a variable called a.fred, for example.

ksh93 has many powerful features for scripts.  I admit that for scripts
sometime ksh93 may be better than zsh.  But after the zsh-3.0 we can add
the missing features.  I have already made some experiments with dynamic
loading.  It is very easy to implement is on elf systems.  What I do not
know yet is the best interface to modules.  Mudules will certainly appear
after zsh-3.0 is out.

Zoltan

Re: quoting bug

[Hrvoje Niksic]

Zefram (A.Main@dcs.warwick.ac.uk) wrote:
> POSIX requires the SH_WORD_SPLIT behaviour.  It also requires field
> splitting in normal words, as well as the result of expansions:

What do you think of setting shwordsplit when the environmental
variable POSIXLY_CORRECT is set?

-- 
hniksic@srce.hr              |  Student of electrical engineering
hniksic@fly.cc.fer.hr        |  University of Zagreb, Croatia
------------------------------------------------------------------
   Mime is a standard for waving your hands through the air, aimlessly,
while people stand around yawning.

Re: quoting bug

[Chet Ramey]

> POSIX requires the SH_WORD_SPLIT behaviour.  It also requires field
> splitting in normal words, as well as the result of expansions:

I'm usually silent here, but I can't let this one by.  POSIX.2 explicitly
disallows field splitting on words that are not the result of expansions.
See section 3.6.5 of POSIX.2-1992.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer

Chet Ramey, Case Western Reserve University	Internet: chet@po.CWRU.Edu

Re: quoting bug

[Zefram]

>Zefram (A.Main@dcs.warwick.ac.uk) wrote:
>> POSIX requires the SH_WORD_SPLIT behaviour.  It also requires field
>> splitting in normal words, as well as the result of expansions:

Actually it appears that it doesn't.  It's been a while since I read
this area of the standard, but as I had specifically looked up this
issue I thought I had remembered it correctly.  I think it would be
nice to implement this kind of field splitting anyway (predicated on
SH_WORD_SPLIT, and have POSIX_FIELD_SPLIT to get POSIX behaviour).

>What do you think of setting shwordsplit when the environmental
>variable POSIXLY_CORRECT is set?

I think that we should have a POSIX emulation mode ("emulate posix").
It would be nice to use it by default if invoked as sh with
POSIXLY_CORRECT set in the environment.  We are going to need a couple
of options for POSIX conformance that contradict sh and ksh
compatibility.

-zefram

Re: quoting bug

[Raymond Nijssen]

::: "Z" == Zefram  <A.Main@dcs.warwick.ac.uk> writes:

 >> zsh fails to break up strings properly where other shells do the right thing.
 > setopt SH_WORD_SPLIT


Thanks for the swift reply.  Sorry that it appears to be a FAQ.

I can see the point made in the FAQ regarding the usefulness of this feature.

Mainly for compatibility reasons, I can not see why the incompatible behavior
is the default.

Especially because it's biting me :-{ since setopts are not inherited across
subshell invocations.  That is, this doesn't seem to be the case, though I
couldn't find anything in the documentation whether or not that is supposed to
happen.

Also, adding this setopt to my ~/.zshenv doesn't help either since the
subshells start with #!/usr/bin/zsh -f   (which is a very useful option to
prevent destruction of the enviroment constructed in the parent script)


So the only two ways around this that I can see are either adding this setopt
to /etc/zshenv, which is clearly unacceptable on a multi-user system, or
modification of all shell scripts .... :-(

Should invoking zsh via a link to `ksh' affect the zsh -f flag?  AFAIK, `ksh
-f' is not defined.

-Raymond