quoting question

quoting question

[Jim]

[-- Attachment #1.1: Type: text/plain, Size: 808 bytes --]

Hi everyone,

Zsh quoting at times makes me wonder. Do quoting? Don't do quoting?
The following case has me scratching my head.  Can someone explain what is
going on?

datetimetest ()
{
  local     DT
  DT="${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}"
  print $DT
  DT=${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}
  print $DT
  print -- "${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}"
  print -- ${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}
  print -- "Date and Time:  ${(%):-%D{%Y-%m-%d %H:%M:%S %Z}} Test"
}

Output:

2023-09-12 13:21:40 CDT}    <--  why "}" when quoted
2023-09-12 13:21:40 CDT
2023-09-12 13:21:40 CDT}    <--  why "}" when quoted
2023-09-12 13:21:40 CDT
Date and Time:  2023-09-12 13:21:40 CDT} Test    <--  why "}" when quoted

ZSH_PATCHLEVEL:  zsh-5.9-208-gf80ad32

Also included test function in an attachment.

Regards,

Jim Murphy

[-- Attachment #1.2: Type: text/html, Size: 1202 bytes --]

[-- Attachment #2: datetimetest.txt --]
[-- Type: text/plain, Size: 294 bytes --]

datetimetest ()
{
  local     DT
  DT="${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}"
  print $DT
  DT=${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}
  print $DT
  print -- "${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}"
  print -- ${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}
  print -- "Date and Time:  ${(%):-%D{%Y-%m-%d %H:%M:%S %Z}} Test"
}

Re: quoting question

[Bart Schaefer]

On Tue, Sep 12, 2023 at 11:46 AM Jim <linux.tech.guy@gmail.com> wrote:
>
> Zsh quoting at times makes me wonder. Do quoting? Don't do quoting?
> The following case has me scratching my head.  Can someone explain what is
> going on?

This doesn't directly have to do with quoting, rather it has to do
with ${param} expansion and brace expansion.  The short answer is you
should never have unquoted { } inside the ${...}, because the first }
encountered is usually going to be interpreted as a match to the ${
and not to any other { in the string.  Where double-quoting gets
involved is because inside a double-quoted string, none of the pairs
of { } have been tokenized, so the lexer can't distinguish which of
the two }} is the actual end of the expansion.

That's what happens here:

>   DT="${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}"
>   print $DT
> 2023-09-12 13:21:40 CDT}    <--  why "}" when quoted

The string is read as ${(%):-%D{%Y-%m-%d %H:%M:%S %Z} followed by }
because the %D{ is quoted (not tokenized) so does not lexically match
with any closing }, so the first one ends the ${...}.

In this case:

>   DT=${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}
>   print $DT
> 2023-09-12 13:21:40 CDT

The inner { } are tokenized and checked for whether they constitute an
{x,y,z} brace expression.  There's no comma, so they're left alone as
a matching pair, and the final } matches the opening ${.  You can see
this by inserting a comma:

DT=${(%):-%D{%Y-%m-%d, %H:%M:%S %Z}}

Now %H becomes the home directory instead of the hour, etc.

If you setopt ignorebraces you'll see that all five examples work the
same and always leave the extra } at the end.

Re: quoting question

[Peter Stephenson]

On Tue, 2023-09-12 at 13:45 -0500, Jim wrote:
> Hi everyone,
> 
> Zsh quoting at times makes me wonder. Do quoting? Don't do quoting?
> The following case has me scratching my head.  Can someone explain what is
> going on?
> 
> datetimetest ()
> {
>   local     DT
>   DT="${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}"
>   print $DT
>   DT=${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}
>   print $DT
>   print -- "${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}"
>   print -- ${(%):-%D{%Y-%m-%d %H:%M:%S %Z}}
>   print -- "Date and Time:  ${(%):-%D{%Y-%m-%d %H:%M:%S %Z}} Test"
> }
> 
> Output:
> 
> 2023-09-12 13:21:40 CDT}    <--  why "}" when quoted
> 2023-09-12 13:21:40 CDT
> 2023-09-12 13:21:40 CDT}    <--  why "}" when quoted
> 2023-09-12 13:21:40 CDT
> Date and Time:  2023-09-12 13:21:40 CDT} Test    <--  why "}" when quoted

That's a very, very good question and I'm very glad you asked.  Er.

It looks like we're not couting brace pairs within quotes the way we do
outside.  I suppose the reason for that is something along the lines of:
if we encounter a "{" inside quotes itʼs just a reguler character so
doesnʼt have a special syntax associated.  But the inconvenience of
counting braces differently just because youʼve added double quotes surely
outweighs that.

The fix is something like the straightforward tweak at the bottom, though
this causes a failure in Y01completion.ztst that needs tracking down.
The test is

  comptesteval 'comptest-postfunc() {}'
  comptest $': *\t\t\t\t\t\t'
0:_expand shows file types
>line: {: dir1/}{}
>DESCRIPTION:{expansions}
>DI:{dir1}
>DI:{dir2}
>FI:{file1}
>FI:{file2}
>DESCRIPTION:{all expansions}
>NO:{dir1 dir2 file1 file2}
>DESCRIPTION:{original}
>NO:{*}
>line: {: dir2/}{}
>line: {: file1 }{}
>line: {: file2 }{}
>line: {: dir1 dir2 file1 file2 }{}
>line: {: *}{}

and the failure is (even though all braces here are balanced)

--- /tmp/zsh.ztst.38062/ztst.out        2023-09-12 20:52:55.852041189 +0100
+++ /tmp/zsh.ztst.38062/ztst.tout       2023-09-12 20:52:55.988039951 +0100
@@ -1,15 +1,6 @@
-line: {: dir1/}{}
-DESCRIPTION:{expansions}
-DI:{dir1}
-DI:{dir2}
-FI:{file1}
-FI:{file2}
-DESCRIPTION:{all expansions}
-NO:{dir1 dir2 file1 file2}
-DESCRIPTION:{original}
-NO:{*}
-line: {: dir2/}{}
-line: {: file1 }{}
-line: {: file2 }{}
-line: {: dir1 dir2 file1 file2 }{}
+line: {: *}{}
+line: {: *}{}
+line: {: *}{}
+line: {: *}{}
+line: {: *}{}
 line: {: *}{}
Test ./Y01completion.ztst failed: output differs from expected as shown above for:
  comptesteval 'comptest-postfunc() {}'
  comptest $': *\t\t\t\t\t\t'


diff --git a/Src/lex.c b/Src/lex.c
index 2f7937410..30d69a28b 100644
--- a/Src/lex.c
+++ b/Src/lex.c
@@ -1584,6 +1584,13 @@ dquote_parse(char endchar, int sub)
 	    } else
 		err = 1;
 	    break;
+	case '{':
+	    if (intick || !bct)
+		break;
+	    c = Inbrace;
+	    bct++;
+	    cmdpush(CS_BRACEPAR);
+	    break;
 	}
 	if (err || lexstop)
 	    break;


pws

Re: quoting question

[Bart Schaefer]

On Tue, Sep 12, 2023 at 1:08 PM Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
>
> It looks like we're not couting brace pairs within quotes the way we do
> outside.  I suppose the reason for that is something along the lines of:
> if we encounter a "{" inside quotes itʼs just a reguler character so
> doesnʼt have a special syntax associated.  But the inconvenience of
> counting braces differently just because youʼve added double quotes surely
> outweighs that.

I went to incredible lengths with the ${| ... } parsing to get around
this very thing.

Refer workers/51907
(https://www.zsh.org/mla/workers/2023/msg00636.html) for Mikael's
argument of why this should NOT be changed.

In any case this patch conflicts with workers/51987 ... it appears
possible to merge them and all my D10 tests still pass, so not a great
problem.

Re: quoting question

[Bart Schaefer]

On Tue, Sep 12, 2023 at 1:23 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> Refer workers/51907

Incidentally your patch will also change the way the PS2 prompt
reports %_ -- for example, it will say it's in a brace expansion even
when ignorebraces means that brace expansion is disabled.

Re: quoting question

[Mikael Magnusson]

On 9/12/23, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Tue, Sep 12, 2023 at 1:23 PM Bart Schaefer <schaefer@brasslantern.com>
> wrote:
>>
>> Refer workers/51907
>
> Incidentally your patch will also change the way the PS2 prompt
> reports %_ -- for example, it will say it's in a brace expansion even
> when ignorebraces means that brace expansion is disabled.

I think it's completely wrong to do this, it becomes impossible to
write { inside a ${:-here} expansion:
% echo "${:-{{{{}"
<dquote braceparam braceparam braceparam braceparam dquote:~code/zsh>%
% echo "${:-\{\{\{\{}"
<dquote braceparam braceparam braceparam braceparam dquote:~code/zsh>%

And even if it was possible with extra quoting, we can't just break
existing strings that happen to have { in them. { inside "" should be
completely non-special.

-- 
Mikael Magnusson

Re: quoting question

[Jim]

[-- Attachment #1: Type: text/plain, Size: 409 bytes --]

On Tue, Sep 12, 2023 at 3:07 PM Bart Schaefer <schaefer@brasslantern.com>
wrote:

>
> If you setopt ignorebraces you'll see that all five examples work the
> same and always leave the extra } at the end.
>

Unless I misunderstood, adding 'setopt ignorebraces' to the function,
returned the
same results as it did without it. Don't know if that helps, or confuses
things more.

Regards,

Jim

[-- Attachment #2: Type: text/html, Size: 809 bytes --]

Re: quoting question

[Bart Schaefer]

On Tue, Sep 12, 2023 at 7:51 PM Jim <linux.tech.guy@gmail.com> wrote:
>
> Unless I misunderstood, adding 'setopt ignorebraces' to the function, returned the
> same results as it did without it. Don't know if that helps, or confuses things more.

You have to setopt ignorebraces before you define the datetimetest
function.  Otherwise the internals of the function have already been
parsed and the behavior doesn't change.

Re: quoting question

[Peter Stephenson]

> On 13/09/2023 02:56 Mikael Magnusson <mikachu@gmail.com> wrote:
> On 9/12/23, Bart Schaefer <schaefer@brasslantern.com> wrote:
> > On Tue, Sep 12, 2023 at 1:23 PM Bart Schaefer <schaefer@brasslantern.com>
> > wrote:
> >>
> >> Refer workers/51907
> >
> > Incidentally your patch will also change the way the PS2 prompt
> > reports %_ -- for example, it will say it's in a brace expansion even
> > when ignorebraces means that brace expansion is disabled.
> 
> I think it's completely wrong to do this, it becomes impossible to
> write { inside a ${:-here} expansion:
> % echo "${:-{{{{}"
> <dquote braceparam braceparam braceparam braceparam dquote:~code/zsh>%
> % echo "${:-\{\{\{\{}"
> <dquote braceparam braceparam braceparam braceparam dquote:~code/zsh>%
> 
> And even if it was possible with extra quoting, we can't just break
> existing strings that happen to have { in them. { inside "" should be
> completely non-special.

If there's a significant knock-on effect, we'll certainly have to leave it alone.

pws

Re: quoting question

[Jim]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

On Tue, Sep 12, 2023 at 3:07 PM Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Tue, Sep 12, 2023 at 11:46 AM Jim <linux.tech.guy@gmail.com> wrote:
> >
> > Zsh quoting at times makes me wonder. Do quoting? Don't do quoting?
> > The following case has me scratching my head.  Can someone explain what
> is
> > going on?
>
> This doesn't directly have to do with quoting, rather it has to do
> with ${param} expansion and brace expansion.  The short answer is you
> should never have unquoted { } inside the ${...}, because the first }
> encountered is usually going to be interpreted as a match to the ${
> and not to any other { in the string.  Where double-quoting gets
> involved is because inside a double-quoted string, none of the pairs
> of { } have been tokenized, so the lexer can't distinguish which of
> the two }} is the actual end of the expansion.
>
>
For the fifth statement:

 print -- "Date and Time:  ${(%):-%D{%Y-%m-%d %H:%M:%S %Z}} Test"

The following works:

print -- "Date and Time:  ${(%):-"%D{%Y-%m-%d %H:%M:%S %Z}"} Test"

Is this the correct interpretation of your statement, or did I somehow put
a bandaid on it.

I appreciate your input.

Thanks and best regards,

Jim

[-- Attachment #2: Type: text/html, Size: 1896 bytes --]

Debug zsh (was: Re: quoting question)

[Pier Paolo Grassi]

[-- Attachment #1: Type: text/plain, Size: 851 bytes --]

Hello, hope I'm doing this banching of the thread the right way
I would like to understand this:

% echo "${:-{{{{}"
<dquote braceparam braceparam braceparam braceparam dquote:~code/zsh>%

is that output generated automatically or was it human-made?

thanks

Pier Paolo Grassi


Il giorno mer 13 set 2023 alle ore 05:42 Bart Schaefer <
schaefer@brasslantern.com> ha scritto:

> On Tue, Sep 12, 2023 at 7:51 PM Jim <linux.tech.guy@gmail.com> wrote:
> >
> > Unless I misunderstood, adding 'setopt ignorebraces' to the function,
> returned the
> > same results as it did without it. Don't know if that helps, or confuses
> things more.
>
> You have to setopt ignorebraces before you define the datetimetest
> function.  Otherwise the internals of the function have already been
> parsed and the behavior doesn't change.
>
>

[-- Attachment #2: Type: text/html, Size: 1536 bytes --]

Re: Debug zsh (was: Re: quoting question)

[Mikael Magnusson]

On 9/13/23, Pier Paolo Grassi <pierpaolog@gmail.com> wrote:
> Il giorno mer 13 set 2023 alle ore 05:42 Bart Schaefer <
> schaefer@brasslantern.com> ha scritto:
>
>> On Tue, Sep 12, 2023 at 7:51 PM Jim <linux.tech.guy@gmail.com> wrote:
>> >
>> > Unless I misunderstood, adding 'setopt ignorebraces' to the function,
>> returned the
>> > same results as it did without it. Don't know if that helps, or
>> > confuses
>> things more.
>>
>> You have to setopt ignorebraces before you define the datetimetest
>> function.  Otherwise the internals of the function have already been
>> parsed and the behavior doesn't change.
> Hello, hope I'm doing this banching of the thread the right way

other than the top posting, it's probably fine :).

> I would like to understand this:
>
> % echo "${:-{{{{}"
> <dquote braceparam braceparam braceparam braceparam dquote:~code/zsh>%
>
> is that output generated automatically or was it human-made?

That's the PS2 prompt, you can see the same thing if you run the same
command but without the quotes. The output above was meant to
demonstrate that the patch adding the same behavior with quotes would
be very backwards incompatible. (The expected result is to print the
string "{{{{")

-- 
Mikael Magnusson

Re: quoting question

[Bart Schaefer]

On Wed, Sep 13, 2023 at 4:29 AM Jim <linux.tech.guy@gmail.com> wrote:
>
> print -- "Date and Time:  ${(%):-"%D{%Y-%m-%d %H:%M:%S %Z}"} Test"
>
> Is this the correct interpretation of your statement

Yes, that's one possible correct interpretation.