zsh -n does not detect incorrect associative array declaration

zsh -n does not detect incorrect associative array declaration

[Paul Wayper]

Hi there!

Sourceforge has pointed me to this list as a way to get a bug fixed in zsh.

I've discovered that an incorrect associative array declaration in zsh
isn't detected via 'zsh -n script.zsh', even though it does get flagged
when the script is executed.  For example:

$ cat zsh_array.zsh 
#!/usr/bin/zsh

typeset -A fn
fn=(foo_key foo_val bar_key bar_val)

printf %s\\n ${fn[foo_key]} ${fn[bar_key]}

$ cat zsh_bad_array.zsh 
#!/usr/bin/zsh

typeset -A fn
fn=(foo_key foo_val bar_key)

printf %s\\n $fn[foo_key] $fn[bar_key]

$ zsh -n zsh_array.zsh && echo $?
0
$ zsh zsh_array.zsh
foo_val bar_val

# And yet:
$ zsh -n zsh_bad_array.zsh && echo $?
0
$ zsh zsh_bad_array.zsh
/tmp/zsh_bad_array.zsh:4: bad set of key/value pairs for associative array


This syntax for associative array declaration is the one documented as
correct:

http://zshwiki.org/home/scripting/array

gives the first example of associative array assignment as:

typeset -A buffer
buffer=( key1 val1 key2 val2 )


Ideally what I'd like is for zsh -n to give the same error message on an
incorrect associative array declaration as zsh gives when executing the
script.

Thanks in advance,

Paul

-- 
Paul Wayper -- Senior Software Maintenance Engineer -- RHCE
Red Hat -- Australia -- Canberra

Re: zsh -n does not detect incorrect associative array declaration

[Bart Schaefer]

On Tue, Mar 22, 2016 at 3:14 PM, Paul Wayper <paulway@redhat.com> wrote:
>
> I've discovered that an incorrect associative array declaration in zsh
> isn't detected via 'zsh -n script.zsh', even though it does get flagged
> when the script is executed.

This is not a bug; it's something that it's impossible to check with
the NO_EXEC option.  Consider:

typeset -a array
typeset -A fn
array=( $(some external command) )
fn=( $array )

With NO_EXEC set, the shell is prohibited from executing the $(some
external command) expression, so there is no way to determine how many
elements would be in $array if execution were allowed, and therefore
no way to determine whether an odd number of elements would be present
in the assignment to fn.

Technically, the shell is ALSO prohibited by NO_EXEC from executing
the "typeset" command, and therefore can't possibly know that "fn"
represents an associative array in the first place.

The NO_EXEC option is only useful for the most rudimentary of syntax
checks.  It cannot detect/predict execution-time inaccuracies.

Re: zsh -n does not detect incorrect associative array declaration

[Paul Wayper]

On 23/03/16 11:20, Bart Schaefer wrote:
> On Tue, Mar 22, 2016 at 3:14 PM, Paul Wayper <paulway@redhat.com> wrote:
>> I've discovered that an incorrect associative array declaration in zsh
>> isn't detected via 'zsh -n script.zsh', even though it does get flagged
>> when the script is executed.
> This is not a bug; it's something that it's impossible to check with
> the NO_EXEC option.  Consider:
>
> typeset -a array
> typeset -A fn
> array=( $(some external command) )
> fn=( $array )
>
> With NO_EXEC set, the shell is prohibited from executing the $(some
> external command) expression, so there is no way to determine how many
> elements would be in $array if execution were allowed, and therefore
> no way to determine whether an odd number of elements would be present
> in the assignment to fn.
>
> Technically, the shell is ALSO prohibited by NO_EXEC from executing
> the "typeset" command, and therefore can't possibly know that "fn"
> represents an associative array in the first place.
>
> The NO_EXEC option is only useful for the most rudimentary of syntax
> checks.  It cannot detect/predict execution-time inaccuracies.

I agree that evaluation of external commands and output substitution
into the script is something that can't be checked with NO_EXEC.

Given that situation, should we update the zsh manual to point out that
the -n option cannot check the syntax of commands that are evaluated, so
that this is more explicit?  I'd be happy to write such an update and
push it if you'd prefer that.

However, I don't see why you can't at least check that the syntax is
correct for things that don't use evaluation.  That by far must be the
majority of such cases.

Hope this helps,

Paul

-- 
Paul Wayper -- Senior Software Maintenance Engineer -- RHCE
Red Hat -- Australia -- Canberra

Re: zsh -n does not detect incorrect associative array declaration

[Bart Schaefer]

On Tue, Mar 22, 2016 at 7:28 PM, Paul Wayper <paulway@redhat.com> wrote:
> On 23/03/16 11:20, Bart Schaefer wrote:
>>
>> Technically, the shell is ALSO prohibited by NO_EXEC from executing
>> the "typeset" command, and therefore can't possibly know that "fn"
>> represents an associative array in the first place.
>>
>> The NO_EXEC option is only useful for the most rudimentary of syntax
>> checks.  It cannot detect/predict execution-time inaccuracies.
>
> Given that situation, should we update the zsh manual to point out that
> the -n option cannot check the syntax of commands that are evaluated, so
> that this is more explicit?  I'd be happy to write such an update and
> push it if you'd prefer that.

I don't have a preference here, but I don't think there's any reason
for the zsh manual to be any more explicit than the manual for any
other shell; for example bash:

     -n      Read commands but do not execute them.  This may be used
             to check a shell script  for  syntax  errors.   This  is
             ignored by interactive shells.

> However, I don't see why you can't at least check that the syntax is
> correct for things that don't use evaluation.  That by far must be the
> majority of such cases.

There's nothing *syntactically* wrong with

fn=(foo_key foo_val bar_key)

Even when executing commands normally, the syntax analyzer does not
know that the assignment will fail if there are an odd number of
values in the list.  That's a semantic error discoverable only when
the assignment is performed.  The shell language is interpreted, not
truly compiled, so parameter type information is not used in syntax
analysis.  Plus, you're still ignoring the fact that the shell doesn't
know "fn" is associative because it was not allowed to interpret the
foregoing "typeset" command.

Aside:  In your original zsh_bad_array.zsh example you can tell that
it was a semantic/runtime error rather than a syntax error because
when the script was run WITHOUT "-n" the statements before and after
the bad assignment were still executed.  An actual syntax error would
have aborted the entire script, probably with a "parse error" printed
to stderr.

Contrast this with ksh where associative array assignments look like

fn=( [foo_key]=foo_val [bar_key]= )

There the parser doesn't need to know the type of fn because the
association is explicit in the format of the parenthesized list.  If
we ever get around to implementing that bit of ksh syntax your
assertion would become valid.  (But in that case the assignment
forcibly changes the type of "fn" to become an associative array so
the typeset is irrelevant.)

Re: zsh -n does not detect incorrect associative array declaration

[Peter Stephenson]

On Tue, 22 Mar 2016 23:40:54 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Contrast this with ksh where associative array assignments look like
> 
> fn=( [foo_key]=foo_val [bar_key]= )
> 
> There the parser doesn't need to know the type of fn because the
> association is explicit in the format of the parenthesized list.  If
> we ever get around to implementing that bit of ksh syntax your
> assertion would become valid.  (But in that case the assignment
> forcibly changes the type of "fn" to become an associative array so
> the typeset is irrelevant.)

Off the original topic, but actually it doesn't, at least not in bash.

$ a=(1 2 3)
$ echo ${a[2]}
3
$ a=([one]=1 [two]=2)
$ echo ${a[*]}
2

This has been interpreted as a[one]=1 then a[two]=2 with "a" still an
array, so "one" and "two" are treated as 0.  This version was quite old
(4.1.7).

pws

Re: zsh -n does not detect incorrect associative array declaration

[Bart Schaefer]

On Mar 23,  9:23am, Peter Stephenson wrote:
} Subject: Re: zsh -n does not detect incorrect associative array declaratio
}
} On Tue, 22 Mar 2016 23:40:54 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > Contrast this with ksh where associative array assignments look like
} > 
} > fn=( [foo_key]=foo_val [bar_key]= )
} > 
} > forcibly changes the type of "fn" to become an associative array
} 
} Off the original topic, but actually it doesn't, at least not in bash.

I'd forgotten that bash even supports that syntax, but in any case it
does change the type in ksh.

$ typeset -a x
$ typeset -p x
typeset -a x
$ x=( [a]=1 [b]=2 )
$ typeset -p x
typeset -A x=([a]=1 [b]=2)
$ 

Bash allows you to set numerically-indexed positions in a normal array
with the ( [key]=value ) syntax, and also outputs the arrays that way
with "typeset -p".  You can have sparse arrays in bash.

Re: zsh -n does not detect incorrect associative array declaration

[Peter Stephenson]

On Wed, 23 Mar 2016 19:08:30 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> I'd forgotten that bash even supports that syntax, but in any case it
> does change the type in ksh.
>..
> Bash allows you to set numerically-indexed positions in a normal array
> with the ( [key]=value ) syntax, and also outputs the arrays that way
> with "typeset -p".  You can have sparse arrays in bash.

We probably need to decide which to support at some point.

pws

Re: zsh -n does not detect incorrect associative array declaration

[Bart Schaefer]

On Mar 24,  9:46am, Peter Stephenson wrote:
} Subject: Re: zsh -n does not detect incorrect associative array declaratio
}
} On Wed, 23 Mar 2016 19:08:30 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > Bash allows you to set numerically-indexed positions in a normal array
} > with the ( [key]=value ) syntax, and also outputs the arrays that way
} > with "typeset -p".  You can have sparse arrays in bash.
} 
} We probably need to decide which to support at some point.

It would be a pretty major rewrite to support sparse arrays, we do all
sorts of things based on the idea that a zsh array is basically the
same as an argv in C.  (Bash, I believe, uses linked lists.)

Of course to be "traditional" about it, we'd do it the ksh way when
KSH_ARRAYS and the bash way when ... something else.

Re: zsh -n does not detect incorrect associative array declaration

[Paul Wayper]

On 23/03/16 17:40, Bart Schaefer wrote:
> On Tue, Mar 22, 2016 at 7:28 PM, Paul Wayper <paulway@redhat.com> wrote:
>> On 23/03/16 11:20, Bart Schaefer wrote:
>>> Technically, the shell is ALSO prohibited by NO_EXEC from executing
>>> the "typeset" command, and therefore can't possibly know that "fn"
>>> represents an associative array in the first place.
>>>
>>> The NO_EXEC option is only useful for the most rudimentary of syntax
>>> checks.  It cannot detect/predict execution-time inaccuracies.
>> Given that situation, should we update the zsh manual to point out that
>> the -n option cannot check the syntax of commands that are evaluated, so
>> that this is more explicit?  I'd be happy to write such an update and
>> push it if you'd prefer that.
> I don't have a preference here, but I don't think there's any reason
> for the zsh manual to be any more explicit than the manual for any
> other shell; for example bash:
>
>      -n      Read commands but do not execute them.  This may be used
>              to check a shell script  for  syntax  errors.   This  is
>              ignored by interactive shells.

I guess I would change that to read:

"Read commands but do not execute them.  This may be used to check a
shell script for most syntax errors, but cannot check inside evals and
other invocations."

> There's nothing *syntactically* wrong with
>
> fn=(foo_key foo_val bar_key)
>
> Even when executing commands normally, the syntax analyzer does not
> know that the assignment will fail if there are an odd number of
> values in the list.  That's a semantic error discoverable only when
> the assignment is performed.  The shell language is interpreted, not
> truly compiled, so parameter type information is not used in syntax
> analysis.  Plus, you're still ignoring the fact that the shell doesn't
> know "fn" is associative because it was not allowed to interpret the
> foregoing "typeset" command.

I see what you mean.  I would have expected the syntax check depended on
the typeset command, enough that it allows it to execute or stores the
type of the declared variable.  But as you say this is an interpreted
language so typed variables are somewhat bolted-on.

Anyway, I understand the situation now, so I can solve my problem.

Thanks once again,

Paul

-- 
Paul Wayper -- Senior Software Maintenance Engineer -- RHCE
Red Hat -- Australia -- Canberra

Re: zsh -n does not detect incorrect associative array declaration

[Sebastian Gniazdowski]

Just a small tip: if you want to detect syntax errors in script you
can use zcompile. That's what I'm doing in zplugin, turning some
options on and off and checking if a plugin compiles correctly:

https://asciinema.org/a/9x2y2xvmpflkq5wzxeaegln12

This allows some non-zero degree of syntax correctness verification,
for given set of options.

Best regards,
Sebastian Gniazdowski