order of processing in brace expansion

order of processing in brace expansion

[Allan Poindexter]

Consider the following:

pts/11 10:37 208%setopt braceccl                cato:/export/home/cato/apoindex
pts/11 10:38 209%foo=b,c                        cato:/export/home/cato/apoindex
pts/11 10:38 210%echo a{$foo}d                  cato:/export/home/cato/apoindex
a,d abd acd
pts/11 10:38 211%echo a{b,c}d                   cato:/export/home/cato/apoindex
abd acd
pts/11 10:38 212%unsetopt braceccl              cato:/export/home/cato/apoindex
pts/11 10:38 213%echo a{$foo}d                  cato:/export/home/cato/apoindex
a{b,c}d


According to info:

  Expansion is done in the above specified order in five steps.  The
  first is "history expansion" which is only performed in interactive
  shells.  The next step is "alias expansion" which is done right before
  the command line is parsed.  They are followed by "process
  substitution", "parameter expansion", "command substitution",
  "arithmetic expansion", and "brace expansion" which are performed in
  one step in left-to-right fashion.  After these expansions, all
  unquoted occurrences of the characters `\', `'', and `"' are removed
  and the result is subjected to "filename expansion" followed by
  "filename generation".

Since brace expansion has syntactic markers at two points there is some
ambiguity about where it would fall in the left to right processing but it
seems that neither choice should cause the actual behavior seen.


If brace expansion is considered to occur at the opening brace then I would
expect:

%setopt braceccl
%echo a{$foo}d
a$d afd aod
%unsetopt braceccl
%echo a{$foo}d
ab,cd

in the above example since the brace expansion would occur before $foo was
parameter expanded. 


If brace expansion is considered to occur at the closing brace then I would
expect:

%echo a{$foo}d
abd acd

since parameter expansion would occur first.  (The setting of brace_ccl would
be unimportant in this case.)


What actually occurs seems to be that it is decided what kind of brace
expansion is needed at the opening brace but the brace expansion is actually
done (consistent with the decision of type made earlier) at the closing brace.

My personal preference would be for brace expansion to occur entirely at the
closing brace since that seems to me to be the most useful definition.
Perhaps others have a different view.

Can anyone shed any light on what I might be overlooking here?


/-\ |_ |_ /-\ |\|

Re: order of processing in brace expansion

[Peter Stephenson]

Allan Poindexter <apoindex@aoc.nrao.edu> wrote:
> Consider the following:
> 
> pts/11 10:37 208%setopt braceccl
> pts/11 10:38 209%foo=b,c
> pts/11 10:38 210%echo a{$foo}d
> a,d abd acd
> 
> Can anyone shed any light on what I might be overlooking here?

By default, parameters in zsh are expanded without turning the contents
into tokens, so the comma is just treated as a normal character, just as if
you had typed a{'b,c'}d.  This is the behaviour I would expect.

The real bug, I would say, is this:

% echo a{$~foo}d
a,d abd acd

The ~ force $foo to be interpreted pretty much as if the characters had
been put directly on the command line (the option GLOB_SUBST does the same
thing).  This is done by default by a lot of other shells.  In *this* case,
I would expect what you expect: the expression is interpreted the same as
a{b,c}d.  Unfortunately it isn't.  The reason seems to be that the comma
doesn't get turned back into a token.  If we want to fix this, the
following patch will do it.

Index: Src/glob.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/glob.c,v
retrieving revision 1.15
diff -u -r1.15 glob.c
--- Src/glob.c	2001/04/24 05:45:17	1.15
+++ Src/glob.c	2001/04/27 17:57:25
@@ -2373,6 +2373,7 @@
 	case '*':
 	case '?':
 	case '=':
+	case ',':
 	    for (t = ztokens; *t; t++)
 		if (*t == *s) {
 		    if (bslash)

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: order of processing in brace expansion

[Bart Schaefer]

On Apr 27,  7:09pm, Peter Stephenson wrote:
} Subject: Re: order of processing in brace expansion
}
} I would expect what you expect: the expression is interpreted the same as
} a{b,c}d.  Unfortunately it isn't.  The reason seems to be that the comma
} doesn't get turned back into a token.  If we want to fix this, the
} following patch will do it.

The only issue, I think, is that comma is not a token unless it is inside
braces, and we don't know at time of expanding $~foo that the surrounding
expression is eligible for brace expansion.  A tokenized comma does still
match an "ordinary" comma during file globbing, but does that mean that it
always matches an ordinary comma?

If tokenizing when it's not necessary doesn't hurt anything, then I think
we should fix the bug.  ("make check" does pass with the patch applied.)

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: order of processing in brace expansion

[Bart Schaefer]

On Apr 28,  6:05am, Bart Schaefer wrote:
} Subject: Re: order of processing in brace expansion
}
} On Apr 27,  7:09pm, Peter Stephenson wrote:
} } Subject: Re: order of processing in brace expansion
} }
} } I would expect what you expect: the expression is interpreted the same as
} } a{b,c}d.  Unfortunately it isn't.  The reason seems to be that the comma
} } doesn't get turned back into a token.  If we want to fix this, the
} } following patch will do it.
} 
} The only issue, I think, is that comma is not a token unless it is inside
} braces

Perhaps that's not the only issue ... consider:

zsh% x='*,v'
zsh% echo abc{*,$~x}

That is, in all versions of zsh so far, using a parameter expansion is a
way to quote commas against brace expansion while still getting filename
generation after the expansion.

This makes me more reluctant to apply the patch; I just can't decide.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: order of processing in brace expansion

[Peter Stephenson]

> On Apr 27,  7:09pm, Peter Stephenson wrote:
> } Subject: Re: order of processing in brace expansion
> }
> } I would expect what you expect: the expression is interpreted the same as
> } a{b,c}d.  Unfortunately it isn't.  The reason seems to be that the comma
> } doesn't get turned back into a token.  If we want to fix this, the
> } following patch will do it.
> 
> The only issue, I think, is that comma is not a token unless it is inside
> braces, and we don't know at time of expanding $~foo that the surrounding
> expression is eligible for brace expansion.  A tokenized comma does still
> match an "ordinary" comma during file globbing, but does that mean that it
> always matches an ordinary comma?
> 
> If tokenizing when it's not necessary doesn't hurt anything, then I think
> we should fix the bug.  ("make check" does pass with the patch applied.)

I would be fairly confident about this, since there are plenty of other
tokens which might be left hanging around, e.g. #'s or ~'s when
EXTENDEDGLOB isn't set.  These cases are all handled by a generic search
for tokens which need turning back into normal characters.  So I'll commit
this and see.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: order of processing in brace expansion

[Bart Schaefer]

On May 2, 10:50am, Peter Stephenson wrote:
} Subject: Re: order of processing in brace expansion
}
} > If tokenizing when it's not necessary doesn't hurt anything, then I think
} > we should fix the bug.  ("make check" does pass with the patch applied.)
} 
} I would be fairly confident about this, since there are plenty of other
} tokens which might be left hanging around, e.g. #'s or ~'s when
} EXTENDEDGLOB isn't set.

With regard to zsh-workers/14153:

> That is, in all versions of zsh so far, using a parameter expansion is a
> way to quote commas against brace expansion while still getting filename
> generation after the expansion.

I was wondering whether there might be some sort of compromise, such as
only tokenizing commas when BRACE_CCL is set.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: order of processing in brace expansion

[Peter Stephenson]

Bart wrote:
> > That is, in all versions of zsh so far, using a parameter expansion is a
> > way to quote commas against brace expansion while still getting filename
> > generation after the expansion.
> 
> I was wondering whether there might be some sort of compromise, such as
> only tokenizing commas when BRACE_CCL is set.

I don't really like filling the code full of this sort of dependency,
unless it can be proven that someone, somewhere was using this particular
hack which none of us here even suspected existed until a few days ago.

But it has just occurred to me that actually the fix results in some other
illogical results.

% foo='{a,b}'
% print $~foo
{a,b}
% print {$~foo}
{a b}

This is rather hairy.  The obvious fix is to tokenize braces, too.  This
seems to do the trick, but the change of behaviour is now much more
obvious.  We probably ought to make it either all or nothing.  Bash doesn't
expand braces that result from parameters (probably due to the ordering of
expansions).  Any suggestions?  It doesn't seem worth an option.

Index: Src/glob.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/glob.c,v
retrieving revision 1.16
diff -u -r1.16 glob.c
--- Src/glob.c	2001/05/02 09:53:32	1.16
+++ Src/glob.c	2001/05/02 15:22:28
@@ -2374,6 +2374,8 @@
 	case '?':
 	case '=':
 	case ',':
+	case '{':
+	case '}':
 	    for (t = ztokens; *t; t++)
 		if (*t == *s) {
 		    if (bslash)
-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: order of processing in brace expansion

[Bart Schaefer]

On May 2,  4:28pm, Peter Stephenson wrote:
}
} This is rather hairy.  The obvious fix is to tokenize braces, too.  This
} seems to do the trick, but the change of behaviour is now much more
} obvious.  We probably ought to make it either all or nothing.  Bash doesn't
} expand braces that result from parameters (probably due to the ordering of
} expansions).  Any suggestions?

I'd prefer that the code and the doc agree with on another.  Doc:

${~SPEC}
     Turn on the GLOB_SUBST option for the evaluation of SPEC; if the
     `~' is doubled, turn it off.  When this option is set, the string
     resulting from the expansion will be interpreted as a pattern
     anywhere that is possible, such as in filename expansion and
     filename generation and pattern-matching contexts ...

GLOB_SUBST <C> <K> <S>
     Treat any characters resulting from parameter expansion as being
     eligible for file expansion and filename generation, and any
     characters resulting from command substitution as being eligible
     for filename generation.

There's no mention of brace expansion there anywhere, and braces are not
expanded in "a pattern" for most senses of "pattern" elsewhere in the doc;
on the other hand, for command substitutions, it's not even supposed to
tokenize a leading `~' or `=', which it obviously does anyway:

zsh% echo $(echo '~')
~
zsh% (setopt glob_subst; echo $(echo '~'))
/home/schaefer

[That latter is definitely a SH_FILE_EXPANSION interaction with GLOB_SUBST;
the GLOB_SUBST doc is describing the case where SH_FILE_EXPANSION is set:

zsh% (setopt glob_subst sh_file_expansion; echo $(echo '~')) 
~

So we have some doc to fix even if we don't change comma/brace behavior.]

As of this moment, my leaning in the "all or nothing" category is towards
"nothing", particularly because of the possibility of breaking sh/bash
compatibility (note that GLOB_SUBST is set by default for sh/ksh emulation).

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: order of processing in brace expansion

[Peter Stephenson]

I've backed off the patch that tokenizes commas coming from a parameter
expansion with globsubst.  Probably the following is a good idea to make it
clearer.

Index: Doc/Zsh/options.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/options.yo,v
retrieving revision 1.16
diff -u -r1.16 options.yo
--- Doc/Zsh/options.yo	2001/03/30 16:20:03	1.16
+++ Doc/Zsh/options.yo	2001/05/09 09:05:02
@@ -473,7 +473,8 @@
 Treat any characters resulting from parameter expansion as being
 eligible for file expansion and filename generation, and any
 characters resulting from command substitution as being eligible for
-filename generation.
+filename generation.  Braces (and commas in between) do not become eligible
+for expansion.
 )
 pindex(HASH_CMDS)
 cindex(hashing, of commands)

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Re: order of processing in brace expansion

[Allan Poindexter]

 Peter> I've backed off the patch that tokenizes commas coming from a
 Peter> parameter expansion with globsubst.  Probably the following is a good
 Peter> idea to make it clearer.

You may also want to add a clarification to this description in the Expansion
node about brace expansion falling in the left to right order at the closing
brace:

  They are followed by "process substitution", "parameter expansion", "command
  substitution", "arithmetic expansion", and "brace expansion" which are
  performed in one step in left-to-right fashion.

I'm not sure this is absolutely necessary but it may forestall a potential for
confusion.

Re: order of processing in brace expansion

[Allan Poindexter]

 Bart> There's no mention of brace expansion there anywhere, and braces are
 Bart> not expanded in "a pattern" for most senses of "pattern" elsewhere in
 Bart> the doc; on the other hand, for command substitutions, it's not even
 Bart> supposed to tokenize a leading `~' or `=', which it obviously does
 Bart> anyway:

I confess that conforming to the docs is a good thing.  OTOH I don't see any
way to parameterize patterns for brace expansion if this is done.
(E.g. setting a parameter in a script to a pattern that will be used in brace
expansion in several places and which may need to be changed from time to
time.  IMO is a good thing to be able to do).  Am I overlooking something?  Is
there some other way to do this?

Re: order of processing in brace expansion

[Bart Schaefer]

On May 11,  1:22pm, Allan Poindexter wrote:
> 
> I don't see any way to parameterize patterns for brace expansion if
> this is [not] done. (E.g. setting a parameter in a script to a pattern
> that will be used in brace expansion in several places and which may
> need to be changed from time to time. IMO is a good thing to be able
> to do). Am I overlooking something? Is there some other way to do
> this?

Brace expansion is completely equivalent to expanding an array parameter
with rc_expand_param set.  So instead of

	alternates=x,y,z
	echo foo/{$alternates}/bar

you do

	alternates=(x y z)
	echo foo/$^alternates/bar

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net