lexing

lexing

[Ray Andrews]

Gentlemen:

The syntax highlighting in my editors makes the predictable mistake when 
it bumps into these:

(#b)
(#c1,2)
(#a3)

because the lexing mostly follows bash rules.  The people who write the 
highlighter are interested in fixing that for zsh, but they want a 
rigorous description of the rule.  As far as I know the only rule there 
is that if the hash is preceded by an open parenthesis then it is not a 
comment.  Is that sufficient or are there further subtleties?

Re: lexing

[ZyX]

29.11.2015, 18:26, "Ray Andrews" <rayandrews@eastlink.ca>:
> Gentlemen:
>
> The syntax highlighting in my editors makes the predictable mistake when
> it bumps into these:
>
> (#b)
> (#c1,2)
> (#a3)
>
> because the lexing mostly follows bash rules. The people who write the
> highlighter are interested in fixing that for zsh, but they want a
> rigorous description of the rule. As far as I know the only rule there
> is that if the hash is preceded by an open parenthesis then it is not a
> comment. Is that sufficient or are there further subtleties?

With `setopt extendedglob` `echo ech#test` errors with `zsh: no match`: AFAIR this is something like `*` in PCRE (and `##` is `+` in PCRE). With `unsetopt extendedglob` `echo ech#test` echoes `ech#test`, `interactivecomments` option is set in both cases. So the rule I think would be that `#` starts a comment if it is preceded by a whitespace.

Also:

1. ${(kv#)var}
2. ${#var}: usual thing, also in POSIX shells
3. $#var
4. ${(s. # .)var}
5. $(( #var )) (usually causes Vim highlighting to fail) (also $[ #var ])
6. ${var// # /test}
7. ${var//# /test} (hash means different thing here)
8. ${var%#}, ${var:-#}, etc (POSIX)
9. '#', "#", \# (POSIX, ways of escaping hash)


Note that `posh -c 'echo foo#bar'` (also bash, dash, busybox and ksh) echoes `foo#bar`. So without `setopt extendedglob` this is not zsh-specific. So if they show a “predictable mistake” in this case as well then they are false regardless of the shell they write highlighting for.

Re: lexing

[Bart Schaefer]

On Nov 29,  7:24am, Ray Andrews wrote:
}
} The people who write the highlighter are interested in fixing that
} for zsh, but they want a rigorous description of the rule. As far as
} I know the only rule there is that if the hash is preceded by an open
} parenthesis then it is not a comment. Is that sufficient or are there
} further subtleties?

I'll leave the following analysis of all the places where a "#" is NOT
a comment, because I went to the trouble of writing it all down, after
which I realized it's a whole lot easier to explain:

  The # character introduces a comment ONLY when it is immediately
  preceded by a command terminator (including start of line) or by
  whitespace that acts as a word separator.

Everywhere else, # is NOT a comment, in contexts such as:

$# is the obvious one, but they probably have that covered.  Also any
"#" that appears inside ${...} should not be treated as a comment,
except that inside $(...) the normal comment rules apply even if
that is also inside ${...}.

(Aside to zsh-workers:  $(...) is apparently NOT "interactive" even if
typed at the command line, for INTERACTIVE_COMMENTS purposes.)

In arithmetic expressions, digits followed by "#" are the base of the
number, e.g., 2#111 is 10#7.  Also open-bracket followed by "#" and
then digits and close-bracket specifies an output base, e.g. $[[#2]7]
substitutes 2#111 -- I'm deliberately using the $[...] alternative to
$(( [#2] 7 )) and squashing spaces).  Oh, and [#10_3] means output
base 10 with underscores between every group of 3 digits, and [##8]
means output in octal with no 8# prefix (works for any base).

Then (still in math context) there's ##Z which outputs the value of
the character Z, which can also be a sequence e.g. $(( ##\e )) is 27,
and (( #VAR )) is the same as ## except on the first character in
the value of $VAR.

The form you're using (#something), is only not-a-comment in pattern
or filename-generation context, so $(# this IS a comment).  And of
course once you have started a (#...) that is not a comment, any #
that is before the closing paren is also not a comment.

Lastly there's # and ## at the end of any string in pattern or name-
generation context, which is never a comment even if extended_glob is
not set.

Final note:  The one thing zsh borrowed from csh that I freely concede
is pretty horrible, is the ability to change the comment character by
changing the third character of $histchars.  Please never do this.

Re: lexing

[Ray Andrews]

On 11/29/2015 10:07 AM, Bart Schaefer wrote:
>    The # character introduces a comment ONLY when it is immediately
>    preceded by a command terminator (including start of line) or by
>    whitespace that acts as a word separator.

Ok, so the comment is actually the exception.  That spares me going 
through ZyX's list and seeing which of them is highlighted wrong.  I 
expect there's a big overlap with bash rules anyway, so we're only 
interested in exceptions from bash.
> Everywhere else, # is NOT a comment, in contexts such as:

Yeah, I know the hash is used all over the place, but I've only noticed 
the " (#...) " error so far, tho I suppose I should check all the other 
candidates.
> The form you're using (#something), is only not-a-comment in pattern
> or filename-generation context, so $(# this IS a comment).

When will I run out of things that astonish me about zsh syntax? What 
sort of need would there be to put a comment inside " $(#...) "  ?  So 
that's a comment but " (#b) " is not a comment ... sometimes, but don't 
bet on it.

> Final note:  The one thing zsh borrowed from csh that I freely concede
> is pretty horrible, is the ability to change the comment character by
> changing the third character of $histchars.  Please never do this.
Pardon the mini-rant,  but that's demented.  So the whole edifice of zsh 
lexing, all the stuff mentioned, and everything else, has to be filtered 
thu some engine that first determines if the comment character has been 
changed?  Changed to what? What character is 'free' for assumption as 
the comment marker?  I don' t know if I'll ever understand.
>

Re: lexing

[Daniel Shahaf]

Ray Andrews wrote on Sun, Nov 29, 2015 at 18:10:30 -0800:
> On 11/29/2015 10:07 AM, Bart Schaefer wrote:
> >The form you're using (#something), is only not-a-comment in pattern
> >or filename-generation context, so $(# this IS a comment).
> 
> When will I run out of things that astonish me about zsh syntax?
> What sort of need would there be to put a comment inside " $(#...) "
> ?  So that's a comment but " (#b) " is not a comment ... sometimes,
> but don't bet on it.

For the same reason as always — to document things for the next reader:

    echo $( # Use a subshell because foo chokes if isatty(1)
            foo
          )

I don't think Bart mentioned the subshell syntax:

    ( # this is a comment
      echo 42 )

Re: lexing

[Ray Andrews]

On 11/29/2015 07:33 PM, Daniel Shahaf wrote:
> For the same reason as always — to document things for the next reader: 

Yup, that's questions and answers are for.

> echo $( # Use a subshell because foo chokes if isatty(1) foo ) I don't 
> think Bart mentioned the subshell syntax: ( # this is a comment echo 42 ) 

As long as his general rule holds.  The white space serves as a word 
separator, so really, following that more or less simple rule, it hasta 
be a comment, which pretty much kills my last mini-rant--it's a comment 
because the rule makes it a comment, and because there's no reason to 
take the trouble to prevent it from being a comment so it's a comment, 
strange as it might look.

Re: lexing

[Ray Andrews]

A fruitful hunt for highlighting errors:

These are not mistaken as comments:

    #!/usr/bin/zsh
    ${(kv#)var}
    ${#var}
    $#var
    ${(s. # .)var}
    ${var// # /test}
    ${var//# /test}
    ${var%#}, ${var:-#}
    2#111
      '#', "#", \#
      `posh -c 'echo foo#bar'`


These are mistaken for comments:

    (#b)
    (#c1,2)
    (#a3)
    $[[#2]7]
    ##Z

... as are these:

    $(( ##\e )) is 27,
    (( #VAR ))
    $(( #var ))


... but, given that there is preceding whitespace, would these three not 
break Bart's rule?  Perhaps the whitespace here is does not qualify as 
'word separating'?  Or is the " $(()) " construct an exception?

... and unfortunately:

     $(# this IS a comment).

is not considered to be a comment.

Re: lexing

[Bart Schaefer]

On Nov 29,  8:47pm, Ray Andrews wrote:
}
}     $(( ##\e )) is 27,
}     (( #VAR ))
}     $(( #var ))
} 
} ... but, given that there is preceding whitespace, would these three not 
} break Bart's rule? Perhaps the whitespace here is does not qualify as 
} 'word separating'?  Or is the " $(()) " construct an exception?

The whitespace is not "word separating" because (( stuff )) is shorthand
for 
    let "stuff"
so the spaces are quoted, at the level at which comments apply.  $(( ))
is just a way to substitute inline the "let".

Re: lexing

[Ray Andrews]

On 11/30/2015 08:23 AM, Bart Schaefer wrote:
> On Nov 29,  8:47pm, Ray Andrews wrote:
> }
> }     $(( ##\e )) is 27,
> }     (( #VAR ))
> }     $(( #var ))
> }
> } ... but, given that there is preceding whitespace, would these three not
> } break Bart's rule? Perhaps the whitespace here is does not qualify as
> } 'word separating'?  Or is the " $(()) " construct an exception?
>
> The whitespace is not "word separating" because (( stuff )) is shorthand
> for
>      let "stuff"
> so the spaces are quoted, at the level at which comments apply.  $(( ))
> is just a way to substitute inline the "let".
>
I thought of that construction as 'math'.  But ... well nuts, so it is, 
and so is 'let' but with the implied quotation. So visually it's a 
gotcha, but not a very deep gotcha.  So it's an exception but it should 
be simple enough to lex. Mysteries evaporate when one sees what's under 
the hood.

master class

[Ray Andrews]

All,

I've run out of things to do in zsh.  If one of you would be so gracious 
as to look over my respin of Sebastian's stuff and point out any errors 
or improvements I'd consider it a great kindness. I'd call it a master 
class except that I'm not a master, but maybe no longer a novice 
either.  Otherwise I'm about done and will more or less leave you guys 
in peace.  Many thanks for the lessons and (from most) the patience.