syntactic question

syntactic question

[Ray Andrews]

All,

Running this script:

    test="one\ttwo"

    echo  1 ${test//'\'/\\}
    echo "2 ${test//'\'/\\}"
    echo  3 ${test//\\/\\}
    echo "4 ${test//\\/\\}"
    echo  5 ${test//'\'/'\'}
    echo "6 ${test//'\'/'\'}"
    echo "7 ${test//'\'/'\\'}"
    echo "8 ${test//'\'/'\\\\'}"
    echo "9 ${test//'\\'/'\\'}"

... I get this output:

    1 one    two
    2 one    two
    3 one    two
    4 one    two
    5 one    two
    6 one'\'ttwo
    7 one'\'ttwo
    8 one'\'ttwo
    9 one    two


Lines one thru five are no puzzle, they're just different ways
of saying the same thing: we're replacing the backslash character
with itself (as an exercise). But at line six the rules change.
Now the outer quotes seem to be meddling inside the substitution,
and the single-quoted backslash output becomes literal: '\'
but the input syntax is unchanged.

I've always thought that all parsing systems work from the 'inside out'
which is to say that (as with math) you 'do' the deepest parenthesis
first, and then work outward. In the same spirit, I'm expecting
the substitution to do what it does following it's own rules and
nevermind any quotations or anything else 'further out'. But if
zsh does permit the outer quotes to change what happens inside
the substitution, then why is it not equal in it's effect on the
input string?

I'm puzzled by the fact that lines six, seven and eight produce the
same output, and there again there is no symmetry between the input
and the output strings, since, in line nine, doubling the input
backslash doesn't work, whereas it does work in the output string (seven).
It seems 'obvious' to me that all of the above, except seven and eight,
are saying 'replace a backslash with itself'. Six should work the same
as one thru five, and if seven works, then so should nine. Why line
eight works, I'm not sure I even want to know ;-)

In practice there's no huge problem tho, since the 'all backslash'
forms always work the same way. Is this understandable for a mere
mortal?

Re: syntactic question

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 987 bytes --]

On Nov 21, 2014 12:25 PM, "Ray Andrews" <rayandrews@eastlink.ca> wrote:
>
> I've always thought that all parsing systems work from the 'inside out'
> which is to say that (as with math) you 'do' the deepest parenthesis
> first, and then work outward.

Quotes aren't parens, there is no left and right so you can't nest them the
way you are thinking of.  With only a small number of exceptions, quoting
in shells works left to right - once a quote is open, you obey only the
rules of that type of quote until it closes again.  Backslashes happen to
have special meaning both inside and outside of quotes, but the details
depend on which type of quote is active.

> In the same spirit, I'm expecting
> the substitution to do what it does following it's own rules and
> nevermind any quotations or anything else 'further out'.

Substitution occurs if there are no quotes or inside double quotes, but not
inside single quotes.  The rules are different when in double quotes, and
documented.

Re: syntactic question

[Ray Andrews]

On 11/21/2014 01:29 PM, Bart Schaefer wrote:
>
>
> On Nov 21, 2014 12:25 PM, "Ray Andrews" <rayandrews@eastlink.ca 
> <mailto:rayandrews@eastlink.ca>> wrote:
> >
> > I've always thought that all parsing systems work from the 'inside out'
> > which is to say that (as with math) you 'do' the deepest parenthesis
> > first, and then work outward.
>
> Quotes aren't parens, there is no left and right so you can't nest 
> them the way you are thinking of.  With only a small number of 
> exceptions, quoting in shells works left to right - once a quote is 
> open, you obey only the rules of that type of quote until it closes 
> again.  Backslashes happen to have special meaning both inside and 
> outside of quotes, but the details depend on which type of quote is 
> active.
>
It's one of those things that's hard to get into one's DNA.
>
> > In the same spirit, I'm expecting
> > the substitution to do what it does following it's own rules and
> > nevermind any quotations or anything else 'further out'.
>
> Substitution occurs if there are no quotes or inside double quotes, 
> but not inside single quotes.  The rules are different when in double 
> quotes, and documented.
>
Well, as long as the system works, and is understandable.  I think my 
intuitions are forgivable, even if not correct.  Bart, is there some 
readable doc that wraps it all up?  In practice I get most all my 
problems solved now, but I despair of really understanding it, I tend to 
use trial and error.

Re: syntactic question

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

On Nov 21, 2014 3:46 PM, "Ray Andrews" <rayandrews@eastlink.ca> wrote:
>
>   Bart, is there some readable doc that wraps it all up?  In practice I
get most all my problems solved now, but I despair of really understanding
it, I tend to use trial and error.

Minimally, section 6.9 Quoting in the zsh info explains the three types of
quotes (backticks look like quotes but are really a substitution).

Then you need 14.3 Parameter Expansion and 14.4 Command Substitution to
explain what happens inside double quotes.  14.3.2 Rules and 14.3.3
Examples may be the most helpful.

Re: syntactic question

[Ray Andrews]

On 11/21/2014 04:00 PM, Bart Schaefer wrote:
> On Nov 21, 2014 3:46 PM, "Ray Andrews" <rayandrews@eastlink.ca> wrote:
>>    Bart, is there some readable doc that wraps it all up?  In practice I
> get most all my problems solved now, but I despair of really understanding
> it, I tend to use trial and error.
>
> Minimally, section 6.9 Quoting in the zsh info explains the three types of
> quotes (backticks look like quotes but are really a substitution).
>
> Then you need 14.3 Parameter Expansion and 14.4 Command Substitution to
> explain what happens inside double quotes.  14.3.2 Rules and 14.3.3
> Examples may be the most helpful.
>
Thanks, it's queued.

Re: syntactic question

[Kurtis Rader]

[-- Attachment #1: Type: text/plain, Size: 3300 bytes --]

Is this understandable for a mere mortal? Yes, but it requires a good grasp
for the rules of how a statement is parsed into tokens. Specifically, how
quotes are handled. It's a pretty good example for why any non-trivial
substitutions (i.e., those involving quotes) is probably better done in a
language which doesn't have such an intimate relationship between between
parsing and evaluating a statement.

What's going on might be a little clearer if you use "print -r" rather than
"echo":

#!/bin/zsh
test="one\ttwo"

print -r  1 ${test//'\'/\\}
print -r "2 ${test//'\'/\\}"
print -r  3 ${test//\\/\\}
print -r "4 ${test//\\/\\}"
print -r  5 ${test//'\'/'\'}
print -r "6 ${test//'\'/'\'}"
print -r "7 ${test//'\'/'\\'}"
print -r "8 ${test//'\'/'\\\\'}"
print -r "9 ${test//'\\'/'\\'}"

print

print -r  1 ${test//'\'/X}
print -r "2 ${test//'\'/X}"
print -r  3 ${test//\\/X}
print -r "4 ${test//\\/X}"
print -r  5 ${test//'\'/'X'}
print -r "6 ${test//'\'/'X'}"
print -r "7 ${test//'\'/'X'}"
print -r "8 ${test//'\'/'X'}"
print -r "9 ${test//'\\'/'X'}"


On Fri, Nov 21, 2014 at 12:23 PM, Ray Andrews <rayandrews@eastlink.ca>
wrote:

> All,
>
> Running this script:
>
>    test="one\ttwo"
>
>    echo  1 ${test//'\'/\\}
>    echo "2 ${test//'\'/\\}"
>    echo  3 ${test//\\/\\}
>    echo "4 ${test//\\/\\}"
>    echo  5 ${test//'\'/'\'}
>    echo "6 ${test//'\'/'\'}"
>    echo "7 ${test//'\'/'\\'}"
>    echo "8 ${test//'\'/'\\\\'}"
>    echo "9 ${test//'\\'/'\\'}"
>
> ... I get this output:
>
>    1 one    two
>    2 one    two
>    3 one    two
>    4 one    two
>    5 one    two
>    6 one'\'ttwo
>    7 one'\'ttwo
>    8 one'\'ttwo
>    9 one    two
>
>
> Lines one thru five are no puzzle, they're just different ways
> of saying the same thing: we're replacing the backslash character
> with itself (as an exercise). But at line six the rules change.
> Now the outer quotes seem to be meddling inside the substitution,
> and the single-quoted backslash output becomes literal: '\'
> but the input syntax is unchanged.
>
> I've always thought that all parsing systems work from the 'inside out'
> which is to say that (as with math) you 'do' the deepest parenthesis
> first, and then work outward. In the same spirit, I'm expecting
> the substitution to do what it does following it's own rules and
> nevermind any quotations or anything else 'further out'. But if
> zsh does permit the outer quotes to change what happens inside
> the substitution, then why is it not equal in it's effect on the
> input string?
>
> I'm puzzled by the fact that lines six, seven and eight produce the
> same output, and there again there is no symmetry between the input
> and the output strings, since, in line nine, doubling the input
> backslash doesn't work, whereas it does work in the output string (seven).
> It seems 'obvious' to me that all of the above, except seven and eight,
> are saying 'replace a backslash with itself'. Six should work the same
> as one thru five, and if seven works, then so should nine. Why line
> eight works, I'm not sure I even want to know ;-)
>
> In practice there's no huge problem tho, since the 'all backslash'
> forms always work the same way. Is this understandable for a mere
> mortal?
>
>


-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

Re: syntactic question

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

On Nov 21, 2014 1:43 PM, "Kurtis Rader" <krader@skepticism.us> wrote:
>
> Is this understandable for a mere mortal? Yes, but it requires a good
grasp
> for the rules of how a statement is parsed into tokens.

Also, as you indirectly pointed out, it requires understanding that some
tokens are handled by the shell on input whereas others are handled by
commands like echo on output.  You can always confuse yourself if you mix
the two.

Re: syntactic question

[Ray Andrews]

On 11/21/2014 03:45 PM, Bart Schaefer wrote:
> On Nov 21, 2014 1:43 PM, "Kurtis Rader" <krader@skepticism.us> wrote:
>> Is this understandable for a mere mortal? Yes, but it requires a good
> grasp
>> for the rules of how a statement is parsed into tokens.
> Also, as you indirectly pointed out, it requires understanding that some
> tokens are handled by the shell on input whereas others are handled by
> commands like echo on output.  You can always confuse yourself if you mix
> the two.
Yeah, that would explain the asymmetry of the input side from the output 
side (which is
really all I'd hope for). Perhaps one day it will all make sense and 
I'll see that it is all for the
best. For now tho, thinking 'left to right' vs. 'inside to outside' is 
enough of a challenge.

Bart, any thoughts on those errors in the debugging binary? 'TRAPDEBUG' 
works perfectly
in the normal build. Interesting how some of the pop out when doing a 
completion.