Is this a bug? Why not?

Is this a bug? Why not?

[Bart Schaefer]

macadamia% print $ZSH_VERSION $ZSH_PATCHLEVEL
4.3.10-dev-1 1.4947
macadamia% print ${var-???}
CVS Doc Etc Src
macadamia% print ${${var}-???}

macadamia% print ${${var}+???}
CVS Doc Etc Src
macadamia% print ${${var}:-???}
CVS Doc Etc Src
macadamia% print ${${var}:+???}

macadamia%

Re: Is this a bug? Why not?

[Phil Pennock]

On 2010-03-30 at 22:46 -0700, Bart Schaefer wrote:
> macadamia% print $ZSH_VERSION $ZSH_PATCHLEVEL
> 4.3.10-dev-1 1.4947
> macadamia% print ${var-???}
> CVS Doc Etc Src
> macadamia% print ${${var}-???}
> 
> macadamia% print ${${var}+???}
> CVS Doc Etc Src
> macadamia% print ${${var}:-???}
> CVS Doc Etc Src
> macadamia% print ${${var}:+???}
> 
> macadamia%

So the string ${var} when treated as a variable-name is empty but
defined?  The four expansions with a nested ${var} are entirely
consistent with the behaviour when expanding a variable which is defined
with a value of length 0.

Seems unusual, but consistent.

In bash, ksh, bad substitution.  In ksh93, it gets weird:
  $ echo ${var-???}
  OSM SCM bin dbg doc etc lib man src tmp www
  $ echo ${${var}-???}
  ksh93: syntax error: `!' unexpected

The ksh reference-name expansion made me think of the somewhat opposite
expansion in zsh, (P)var ...

Hrm, zsh 4.3.10 1.4705:

  % print ${(P)-???}
  zsh: 0239BCJPXZgiklms: ??
  % print ${(P)+???}
  zsh: bad substitution
  % print ${(P):-???}
  OSM SCM bin dbg doc etc lib man src tmp www
  % print ${(P):+???}

  %

Strange that the first entry is somehow tying into expanding $- for the
shell options ...

  % print ${-}
  0239BCJPXZgiklms
  % print ${(P)-}

  % print ${(P)-?}
  zsh: 0239BCJPXZgiklms: parameter not set

That strikes me as dubious.  You?

Re: Is this a bug? Why not?

[Bart Schaefer]

Incidentally, this came up because of a discussion on the POSIX standards
mailing list (austin-group) in which David Korn just asserted that he'd
like to add the syntax ${"var"} which means to expand the value of var
as if it's quoted (what zsh's normal mode does all the time).

This differs from "${var}" because you can write ${"var"-*} and get the
value of $var quoted but the glob pattern unquoted.  Apparently the old
Bourne shell allows the abominable "${var"-*}  (note quote placement
overlapping with brace placement) to accomplish this.

I was hoping to be able to say "Oh, zsh already has syntax XYZ for that"
but in fact we don't -- zsh either always, or never, does it, depending
on the globsubst option; there's no way to flip globsubst on the fly.

On Mar 30, 11:06pm, Phil Pennock wrote:
} Subject: Re: Is this a bug? Why not?
}
} So the string ${var} when treated as a variable-name is empty but
} defined?  The four expansions with a nested ${var} are entirely
} consistent with the behaviour when expanding a variable which is
} defined with a value of length 0.
} 
} Seems unusual, but consistent.

Indeed.  OK, I can live with that, as it's been that way forever.

} In bash, ksh, bad substitution.  In ksh93, it gets weird:
}   $ echo ${var-???}
}   OSM SCM bin dbg doc etc lib man src tmp www
}   $ echo ${${var}-???}
}   ksh93: syntax error: `!' unexpected

That is weird, but AFAIK only zsh of all shells allows wrapping ${...}
in another ${...} as valid syntax, so this may just be an error ksh93
didn't catch.

} The ksh reference-name expansion made me think of the somewhat opposite
} expansion in zsh, (P)var ...
} 
} Hrm, zsh 4.3.10 1.4705:
} 
}   % print ${(P)-???}
}   zsh: 0239BCJPXZgiklms: ??
} 
} Strange that the first entry is somehow tying into expanding $- for the
} shell options ...

Actually that makes perfect sense; only the :- and :+ forms are doc'd
as working with a missing parameter name, so ${(P)-} really does mean
to grab the value of $- and try to use that as a parameter name.  It
otherwise has to behave like ${-?string}.

Re: Is this a bug? Why not?

[Phil Pennock]

On 2010-03-31 at 08:11 -0700, Bart Schaefer wrote:
> Incidentally, this came up because of a discussion on the POSIX standards
> mailing list (austin-group) in which David Korn just asserted that he'd
> like to add the syntax ${"var"} which means to expand the value of var
> as if it's quoted (what zsh's normal mode does all the time).
> 
> This differs from "${var}" because you can write ${"var"-*} and get the
> value of $var quoted but the glob pattern unquoted.  Apparently the old
> Bourne shell allows the abominable "${var"-*}  (note quote placement
> overlapping with brace placement) to accomplish this.
> 
> I was hoping to be able to say "Oh, zsh already has syntax XYZ for that"
> but in fact we don't -- zsh either always, or never, does it, depending
> on the globsubst option; there's no way to flip globsubst on the fly.

No *neat* syntax.

% ls
lib         viewvc.conf
% print -l ${~~foo-${(~):-*}}
lib
viewvc.conf
% foo='???'
$ print -l ${~~foo-${(~):-*}}
???
% print -l $foo
lib
% unsetopt glob_subst
% print -l $foo      
???
%

*cough*

Re: Is this a bug? Why not?

[Bart Schaefer]

On Apr 1,  1:26am, Phil Pennock wrote:
} Subject: Re: Is this a bug? Why not?
}
} On 2010-03-31 at 08:11 -0700, Bart Schaefer wrote:
} > Incidentally, this came up because of a discussion on the POSIX standards
} > mailing list (austin-group) in which David Korn just asserted that he'd
} > like to add the syntax ${"var"} which means to expand the value of var
} > as if it's quoted (what zsh's normal mode does all the time).
} > 
} > I was hoping to be able to say "Oh, zsh already has syntax XYZ for that"
} > but in fact we don't -- zsh either always, or never, does it, depending
} > on the globsubst option; there's no way to flip globsubst on the fly.
} 
} No *neat* syntax.
} 
} % ls
} lib         viewvc.conf
} % print -l ${~~foo-${(~):-*}}

That isn't really what you mean, is it?  The (~) flag only applies to
the (j::) et al. strings.

However, it's not even necessary to do the ${:-*} thing, ${~~foo-*} is
sufficient.

For austin-group purposes, however, I was hoping for something that did
not rely on parameter expansion flags.  E.g., why doesn't the following
cause $foo to be quoted?

schaefer[516] ARGV0=sh Src/zsh
$ foo="???"
$ print ${foo+"$foo"}
Doc Etc Src
$ 

It works in bash.

Re: Is this a bug? Why not?

[Phil Pennock]

On 2010-04-01 at 07:36 -0700, Bart Schaefer wrote:
> On Apr 1,  1:26am, Phil Pennock wrote:
> } Subject: Re: Is this a bug? Why not?
> }
> } On 2010-03-31 at 08:11 -0700, Bart Schaefer wrote:
> } > Incidentally, this came up because of a discussion on the POSIX standards
> } > mailing list (austin-group) in which David Korn just asserted that he'd
> } > like to add the syntax ${"var"} which means to expand the value of var
> } > as if it's quoted (what zsh's normal mode does all the time).
> } > 
> } > I was hoping to be able to say "Oh, zsh already has syntax XYZ for that"
> } > but in fact we don't -- zsh either always, or never, does it, depending
> } > on the globsubst option; there's no way to flip globsubst on the fly.
> } 
> } No *neat* syntax.
> } 
> } % ls
> } lib         viewvc.conf
> } % print -l ${~~foo-${(~):-*}}
> 
> That isn't really what you mean, is it?  The (~) flag only applies to
> the (j::) et al. strings.

Oh, interesting, I missed that glob_subst wasn't affecting the default
at all.

> However, it's not even necessary to do the ${:-*} thing, ${~~foo-*} is
> sufficient.

Okay, *phew*, I did understand the problem statement.

> For austin-group purposes, however, I was hoping for something that did
> not rely on parameter expansion flags.  E.g., why doesn't the following
> cause $foo to be quoted?
> 
> schaefer[516] ARGV0=sh Src/zsh
> $ foo="???"
> $ print ${foo+"$foo"}
> Doc Etc Src
> $ 
> 
> It works in bash.

Defining "works" as seeing ??? emitted: it works in zsh with glob_subst
disabled.  It works in FreeBSD /bin/sh ...

Is this use-case important enough to warrant yet another option?

-Phil

Re: Is this a bug? Why not?

[Bart Schaefer]

On Apr 1,  2:57pm, Phil Pennock wrote:
} Subject: Re: Is this a bug? Why not?
}
} > E.g., why doesn't the following cause $foo to be quoted?
} > 
} > schaefer[516] ARGV0=sh Src/zsh
} > $ foo="???"
} > $ print ${foo+"$foo"}
} > Doc Etc Src
} > $ 
} > 
} > It works in bash.
} 
} Defining "works" as seeing ??? emitted: it works in zsh with glob_subst
} disabled.

Well, note the "ARGV0=sh", so I'm specifically talking about glob_subst.
Just ${foo} "works" by the no_glob_subst definition.

} Is this use-case important enough to warrant yet another option?

Normally I'd be a stickler for not changing existing behavior, but in
this case it looks so obviously like a bug in the way glob_subst is
implemented that I'm not sure a new option is needed.

Re: Is this a bug? Why not?

[Peter Stephenson]

On Thu, 01 Apr 2010 15:51:18 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Apr 1,  2:57pm, Phil Pennock wrote:
> } Subject: Re: Is this a bug? Why not?
> }
> } > E.g., why doesn't the following cause $foo to be quoted?
> } > 
> } > schaefer[516] ARGV0=sh Src/zsh
> } > $ foo="???"
> } > $ print ${foo+"$foo"}
> } > Doc Etc Src
> } > $ 
> } > 
> } > It works in bash.
> 
> } Is this use-case important enough to warrant yet another option?
> 
> Normally I'd be a stickler for not changing existing behavior, but in
> this case it looks so obviously like a bug in the way glob_subst is
> implemented that I'm not sure a new option is needed.

I think that's correct.  It might be as simple as this.

Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.102
diff -p -u -r1.102 subst.c
--- Src/subst.c	27 Mar 2010 19:04:36 -0000	1.102
+++ Src/subst.c	9 Apr 2010 14:34:26 -0000
@@ -2417,7 +2417,7 @@ paramsubst(LinkList l, LinkNode n, char 
 		multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
 		opts[SHWORDSPLIT] = ws;
 		copied = 1;
-		spbreak = 0;
+		spbreak = globsubst = 0;
 	    }
 	    break;
 	case ':':
Index: Test/D04parameter.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D04parameter.ztst,v
retrieving revision 1.40
diff -p -u -r1.40 D04parameter.ztst
--- Test/D04parameter.ztst	15 Mar 2010 10:19:38 -0000	1.40
+++ Test/D04parameter.ztst	9 Apr 2010 14:34:26 -0000
@@ -221,6 +221,16 @@
 >\\foo matched by \\*\\
 >a\\b not matched by \\*\\
 
+  (
+    setopt globsubst
+    foo="boring*"
+    print ${foo+$foo}
+    print ${foo+"$foo"}
+  )
+0:globsubst together with nested quoted expansion
+>boringfile
+>boring*
+
   print -l "${$(print one word)}" "${=$(print two words)}"
 0:splitting of $(...) inside ${...}
 >one word


-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: Is this a bug? Why not?

[Mikael Magnusson]

On 9 April 2010 16:37, Peter Stephenson <pws@csr.com> wrote:
> On Thu, 01 Apr 2010 15:51:18 -0700
> Bart Schaefer <schaefer@brasslantern.com> wrote:
>> On Apr 1,  2:57pm, Phil Pennock wrote:
>> } Subject: Re: Is this a bug? Why not?
>> }
>> } > E.g., why doesn't the following cause $foo to be quoted?
>> } >
>> } > schaefer[516] ARGV0=sh Src/zsh
>> } > $ foo="???"
>> } > $ print ${foo+"$foo"}
>> } > Doc Etc Src
>> } > $
>> } >
>> } > It works in bash.
>>
>> } Is this use-case important enough to warrant yet another option?
>>
>> Normally I'd be a stickler for not changing existing behavior, but in
>> this case it looks so obviously like a bug in the way glob_subst is
>> implemented that I'm not sure a new option is needed.
>
> I think that's correct.  It might be as simple as this.
>
> Index: Src/subst.c
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
> retrieving revision 1.102
> diff -p -u -r1.102 subst.c
> --- Src/subst.c 27 Mar 2010 19:04:36 -0000      1.102
> +++ Src/subst.c 9 Apr 2010 14:34:26 -0000
> @@ -2417,7 +2417,7 @@ paramsubst(LinkList l, LinkNode n, char
>                multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
>                opts[SHWORDSPLIT] = ws;
>                copied = 1;
> -               spbreak = 0;
> +               spbreak = globsubst = 0;
>            }
>            break;
>        case ':':

This patch breaks at least cd ~-/<tab> and anything using _pids for
me. Reverting it on top of current cvs makes everything work again,
found with git bisect. Frank confirmed he has the same problems with a
recent build too.

-- 
Mikael Magnusson

Re: Is this a bug? Why not?

[Peter Stephenson]

On Sat, 17 Apr 2010 15:06:53 +0200
Mikael Magnusson <mikachu@gmail.com> wrote:
> This patch

(resetting globsubst for the argument of a variable substitution when
the pattern matching is already done)

> breaks at least cd ~-/<tab>

    realpath=${~:-\~$linepath}/

doesn't expand the second "~" despite the first "~" because the second
tilde is quoted.  Strictly this new behaviour is probably correct
because the quote should indeed stop the second tilde being active, but
actually the hack below fixes this so I've left it (even though removing
the backslash seemed to work in this case, too).

> and anything using _pids for me.

That's because given

  desc='  PID TTY          TIME CMD' 
  out=( ' 2591 pts/1    00:00:02 zsh'
        ' 2600 pts/1    00:00:01 dropbox'
        ' 3271 pts/1    00:00:00 zsh'
        ' 3272 pts/1    00:00:00 ps' ) 

this expression:

  pids=( "${(@)${(@M)out#${(l.${#desc[1,(r)(#i)[[:blank:]]pid]}..?.)~:-}[^[:blank:]]#}##*[[:blank:]]}" )

sets

  pids=( '' '' '' '' ) 

Er... obviously.  What's happening is

${(l.${#desc[1,(r)(#i)[[:blank:]]pid]}..?.)~:-}

should turn into ????? which are activated for use in patterns, because
of the "~", but it isn't because globsubst is now turned off when we get
to padding because we already handled the value as the right hand side
of ":-" (i.e. the empty string) so there was no value to activate.

One fix is to leave globsubst on when it's forced by the "~" flag.  That
doesn't screw up sh compatibility, which knows nothing about the flag,
but it's not really the right thing to do, which would be to track
through to see where the text came from and conditionally tokenize it,
yeugh.  The difference is in the case where GLOB_SUBST is set but you're
using zsh substitution syntax, which is not typical (and is the sort of
complicated hybrid I'd dearly love to forbid but will never be able to).
As I'm definitely not doing any major surgery on paramsubst() to
preserve different notions of globsubst at different points in the
function myself (feel free to volunteer) it may be all you will get in
any case.  See the test file for a simpler indication of implications.

Ultimately, I think this hack may actually be the right thing to do
because it preserves traditional zsh behaviour in typical cases (where
the option isn't set and ~ is used to force it) as well as any script
adhering to sh syntax.

I'll think about some documentation.

  "Ceterum censeo functio de substitutione parametrum esse rescribenda"
  ("Furthermore, I am of the opinion that paramsubst() ought to be
   rewritten")  ---  Marcus Porcius Cato the Elder

Index: Src/subst.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/subst.c,v
retrieving revision 1.103
diff -p -u -r1.103 subst.c
--- Src/subst.c	9 Apr 2010 15:40:14 -0000	1.103
+++ Src/subst.c	18 Apr 2010 19:03:01 -0000
@@ -1386,7 +1386,8 @@ paramsubst(LinkList l, LinkNode n, char 
     int plan9 = isset(RCEXPANDPARAM);
     /*
      * Likwise, but with ~ and ~~.  Also, we turn it off later
-     * on if qt is passed down.
+     * on if qt is passed down. The value can go to 2 if we
+     * use ~ to force this on.
      */
     int globsubst = isset(GLOBSUBST);
     /*
@@ -1899,12 +1900,12 @@ paramsubst(LinkList l, LinkNode n, char 
 	     * spsep, NULL means $IFS.
 	     */
 	} else if (c == '~' || c == Tilde) {
-	    /* GLOB_SUBST on or off (doubled) */
+	    /* GLOB_SUBST (forced) on or off (doubled) */
 	    if ((c = *++s) == '~' || c == Tilde) {
 		globsubst = 0;
 		s++;
 	    } else
-		globsubst = 1;
+		globsubst = 2;
 	} else if (c == '+') {
 	    /*
 	     * Return whether indicated parameter is set. 
@@ -1935,7 +1936,8 @@ paramsubst(LinkList l, LinkNode n, char 
 	    break;
     }
     /* Don't activate special pattern characters if inside quotes */
-    globsubst = globsubst && !qt;
+    if (qt)
+	globsubst = 0;
 
     /*
      * At this point, we usually expect a parameter name.
@@ -2417,7 +2419,10 @@ paramsubst(LinkList l, LinkNode n, char 
 		multsub(&val, spbreak && !aspar, (aspar ? NULL : &aval), &isarr, NULL);
 		opts[SHWORDSPLIT] = ws;
 		copied = 1;
-		spbreak = globsubst = 0;
+		spbreak = 0;
+		/* Leave globsubst on if forced */
+		if (globsubst != 2)
+		    globsubst = 0;
 	    }
 	    break;
 	case ':':
Index: Test/D04parameter.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D04parameter.ztst,v
retrieving revision 1.41
diff -p -u -r1.41 D04parameter.ztst
--- Test/D04parameter.ztst	9 Apr 2010 15:40:14 -0000	1.41
+++ Test/D04parameter.ztst	18 Apr 2010 19:03:01 -0000
@@ -226,10 +226,12 @@
     foo="boring*"
     print ${foo+$foo}
     print ${foo+"$foo"}
+    print ${~foo+"$foo"}
   )
 0:globsubst together with nested quoted expansion
 >boringfile
 >boring*
+>boringfile
 
   print -l "${$(print one word)}" "${=$(print two words)}"
 0:splitting of $(...) inside ${...}

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/