* ${(z)} parsing of multiple array assignments @ 2019-12-23 17:31 Daniel Shahaf 2019-12-24 20:34 ` Phil Pennock 2019-12-29 20:55 ` Peter Stephenson 0 siblings, 2 replies; 11+ messages in thread From: Daniel Shahaf @ 2019-12-23 17:31 UTC (permalink / raw) To: zsh-users In the following two cases, why are the assignments to $b parsed differently to the assignments to $a? % pz() { print -rl -- ${(qqqq)${(z)1}} } % pz 'a=() b=()' $'a=(' $')' $'b=' $'()' % pz 'a=(foo) b=(bar)' $'a=(' $'foo' $')' $'b=(bar)' Thanks, Daniel ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-23 17:31 ${(z)} parsing of multiple array assignments Daniel Shahaf @ 2019-12-24 20:34 ` Phil Pennock 2019-12-24 23:16 ` Sebastian Gniazdowski 2019-12-29 20:55 ` Peter Stephenson 1 sibling, 1 reply; 11+ messages in thread From: Phil Pennock @ 2019-12-24 20:34 UTC (permalink / raw) To: Daniel Shahaf; +Cc: zsh-users On 2019-12-23 at 17:31 +0000, Daniel Shahaf wrote: > In the following two cases, why are the assignments to $b > parsed differently to the assignments to $a? > > % pz() { print -rl -- ${(qqqq)${(z)1}} } Looks to be a bug, a lack of state reset. In two places, perhaps: you can see that one example resets when you alternate back away from having empty assignment lists, where the b/c pairing here is reset by `d`, so that the f/g pairing matches. But I can't trigger a reset to the initial parse state used for `a`; unless and until I introduce a newline, where in the second example you can see the same parse used for `e`: % echo $ZSH_VERSION 5.7.1 % x='a=(foo) b=() c=() d=(bar) e=(baz) f=() g=()' % print -rl -- "${(z)x}" a=( foo ) b= () c=( ) d=(bar) e=(baz) f= () g=( ) ## differs in $'...' not '...' and \n between d and e: % x=$'a=(foo) b=() c=() d=(bar)\ne=(baz) f=() g=()' % print -rl -- "${(z)x}" a=( foo ) b= () c=( ) d=(bar) ; e=( baz ) f= () g=( ) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-24 20:34 ` Phil Pennock @ 2019-12-24 23:16 ` Sebastian Gniazdowski 0 siblings, 0 replies; 11+ messages in thread From: Sebastian Gniazdowski @ 2019-12-24 23:16 UTC (permalink / raw) To: Phil Pennock; +Cc: Daniel Shahaf, Zsh Users Looks like an empty assignment does reset the state for a single next one: % x='a=(foo) b=() d=(bar)' % print -rl -- "${(z)x}" a=( foo ) b= () d=( bar ) On Tue, 24 Dec 2019 at 21:35, Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote: > > On 2019-12-23 at 17:31 +0000, Daniel Shahaf wrote: > > In the following two cases, why are the assignments to $b > > parsed differently to the assignments to $a? > > > > % pz() { print -rl -- ${(qqqq)${(z)1}} } > > Looks to be a bug, a lack of state reset. In two places, perhaps: you > can see that one example resets when you alternate back away from having > empty assignment lists, where the b/c pairing here is reset by `d`, so > that the f/g pairing matches. But I can't trigger a reset to the > initial parse state used for `a`; unless and until I introduce a > newline, where in the second example you can see the same parse used for > `e`: > > % echo $ZSH_VERSION > 5.7.1 > > % x='a=(foo) b=() c=() d=(bar) e=(baz) f=() g=()' > % print -rl -- "${(z)x}" > a=( > foo > ) > b= > () > c=( > ) > d=(bar) > e=(baz) > f= > () > g=( > ) > > ## differs in $'...' not '...' and \n between d and e: > > % x=$'a=(foo) b=() c=() d=(bar)\ne=(baz) f=() g=()' > % print -rl -- "${(z)x}" > a=( > foo > ) > b= > () > c=( > ) > d=(bar) > ; > e=( > baz > ) > f= > () > g=( > ) > -- Sebastian Gniazdowski News: https://twitter.com/ZdharmaI IRC: https://kiwiirc.com/client/chat.freenode.net:+6697/#zplugin Blog: http://zdharma.org ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-23 17:31 ${(z)} parsing of multiple array assignments Daniel Shahaf 2019-12-24 20:34 ` Phil Pennock @ 2019-12-29 20:55 ` Peter Stephenson 2019-12-29 23:45 ` Peter Stephenson 1 sibling, 1 reply; 11+ messages in thread From: Peter Stephenson @ 2019-12-29 20:55 UTC (permalink / raw) To: zsh-users On Mon, 2019-12-23 at 17:31 +0000, Daniel Shahaf wrote: > In the following two cases, why are the assignments to $b > parsed differently to the assignments to $a? > > % pz() { print -rl -- ${(qqqq)${(z)1}} } > > % pz 'a=() b=()' > $'a=(' > $')' > $'b=' > $'()' > > % pz 'a=(foo) b=(bar)' > $'a=(' > $'foo' > $')' > $'b=(bar)' Unless I'm missing some trick, bufferwords() is the function where we need to update any parser state --- ctxtlex() is too low level for that, it just handles the next token given the current state. So it's probably something like this. Will need a new test adding. Presumably the commenting out of ENVSTRING in ctxtlex() means we stay in incmdpos in that case --- as this is just a single word assignment that probably handles that case OK. pws diff --git a/Src/hist.c b/Src/hist.c index 74116e82f..37cba4579 100644 --- a/Src/hist.c +++ b/Src/hist.c @@ -3321,6 +3321,7 @@ bufferwords(LinkList list, char *buf, int *index, int flags) int owb = wb, owe = we, oadx = addedx, onc = nocomments; int ona = noaliases, ocs = zlemetacs, oll = zlemetall; int forloop = 0, rcquotes = opts[RCQUOTES]; + int envarray = 0; char *p, *addedspaceptr; if (!list) @@ -3404,6 +3405,14 @@ bufferwords(LinkList list, char *buf, int *index, int flags) ctxtlex(); if (tok == ENDINPUT || tok == LEXERR) break; + /* + * After an array assignment, return to the initial + * start-of-command state. There could be a second ENVARRAY. + */ + if (tok == OUTPAR && envarray) { + incmdpos = 1; + envarray = 0; + } if (tok == FOR) { /* * The way for (( expr1 ; expr2; expr3 )) is parsed is: @@ -3441,6 +3450,7 @@ bufferwords(LinkList list, char *buf, int *index, int flags) switch (tok) { case ENVARRAY: p = dyncat(tokstr, "=("); + envarray = 1; break; case DINPAR: ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-29 20:55 ` Peter Stephenson @ 2019-12-29 23:45 ` Peter Stephenson 2019-12-30 18:13 ` Daniel Shahaf 0 siblings, 1 reply; 11+ messages in thread From: Peter Stephenson @ 2019-12-29 23:45 UTC (permalink / raw) To: zsh-users On Sun, 2019-12-29 at 20:55 +0000, Peter Stephenson wrote: > Unless I'm missing some trick, bufferwords() is the function where we > need to update any parser state --- ctxtlex() is too low level for that, > it just handles the next token given the current state. So it's > probably something like this. Will need a new test adding. We also need to do something similar for completion, otherwise foo=(stuff) bar=(... looks like a glob qualifier. In this case it's already half handled, so there's a variable to hook into. pws diff --git a/Src/Zle/zle_tricky.c b/Src/Zle/zle_tricky.c index 2b25d6b2e..fdd168763 100644 --- a/Src/Zle/zle_tricky.c +++ b/Src/Zle/zle_tricky.c @@ -1236,8 +1236,10 @@ get_comp_string(void) else if (tok == OUTPAR) { if (parct) parct--; - else + else if (linarr) { linarr = 0; + incmdpos = 1; + } } if (inredir && IS_REDIROP(tok)) { rdstr = rdstrbuf; diff --git a/Src/hist.c b/Src/hist.c index 74116e82f..5281e8718 100644 --- a/Src/hist.c +++ b/Src/hist.c @@ -3321,6 +3321,7 @@ bufferwords(LinkList list, char *buf, int *index, int flags) int owb = wb, owe = we, oadx = addedx, onc = nocomments; int ona = noaliases, ocs = zlemetacs, oll = zlemetall; int forloop = 0, rcquotes = opts[RCQUOTES]; + int envarray = 0; char *p, *addedspaceptr; if (!list) @@ -3404,6 +3405,14 @@ bufferwords(LinkList list, char *buf, int *index, int flags) ctxtlex(); if (tok == ENDINPUT || tok == LEXERR) break; + /* + * After an array assignment, return to the initial + * start-of-command state. There could be a second ENVARRAY. + */ + if (tok == OUTPAR && envarray) { + incmdpos = 1; + envarray = 0; + } if (tok == FOR) { /* * The way for (( expr1 ; expr2; expr3 )) is parsed is: @@ -3441,6 +3450,7 @@ bufferwords(LinkList list, char *buf, int *index, int flags) switch (tok) { case ENVARRAY: p = dyncat(tokstr, "=("); + envarray = 1; break; case DINPAR: ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-29 23:45 ` Peter Stephenson @ 2019-12-30 18:13 ` Daniel Shahaf 2019-12-30 18:21 ` Daniel Shahaf 0 siblings, 1 reply; 11+ messages in thread From: Daniel Shahaf @ 2019-12-30 18:13 UTC (permalink / raw) To: zsh-users Peter Stephenson wrote on Sun, Dec 29, 2019 at 23:45:00 +0000: > On Sun, 2019-12-29 at 20:55 +0000, Peter Stephenson wrote: > > Unless I'm missing some trick, bufferwords() is the function where we > > need to update any parser state --- ctxtlex() is too low level for that, > > it just handles the next token given the current state. So it's > > probably something like this. Will need a new test adding. > > We also need to do something similar for completion, otherwise > > foo=(stuff) bar=(... > > looks like a glob qualifier. > > In this case it's already half handled, so there's a variable to hook > into. Thanks for the fix; it didn't occur to me to change bufferwords(). Here's a test. diff --git a/Test/D04parameter.ztst b/Test/D04parameter.ztst index c91af1a9c..899b8f498 100644 --- a/Test/D04parameter.ztst +++ b/Test/D04parameter.ztst @@ -640,6 +640,74 @@ >echo >$(|||) bar + argv=( + $'a=() b=()' + $'a=(foo) b=(bar)' + $'a=(foo) b=() c=() d=(bar) e=(baz) f=() g=()' + $'a=(foo) b=() c=() d=(bar)\ne=(baz) f=() g=()' + $'a=(foo) b=() d=(bar)' + ) + for 1; print -rl -- ${(z)1} && print +0:${(z)} regression test: multiple array assignments +>a=( +>) +>b=( +>) +> +>a=( +>foo +>) +>b=( +>bar +>) +> +>a=( +>foo +>) +>b=( +>) +>c=( +>) +>d=( +>bar +>) +>e=( +>baz +>) +>f=( +>) +>g=( +>) +> +>a=( +>foo +>) +>b=( +>) +>c=( +>) +>d=( +>bar +>) +>; +>e=( +>baz +>) +>f=( +>) +>g=( +>) +> +>a=( +>foo +>) +>b=( +>) +>d=( +>bar +>) +> + foo=$'\x06ZUI\x1f text-field example: \x1azuitfieldtfield1_1\x1a\'\'\x1a\'\'\x1a1\x1aZUI\\[my_tfield1_width\\]\x1aZUI\\[my_tfield1_start\\]\x1aZUI\\[my_tfield1_data\\]\x1c' print "${#${(z@)foo}}" 0:Test real-world data that once seemed to fail Cheers, Daniel ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-30 18:13 ` Daniel Shahaf @ 2019-12-30 18:21 ` Daniel Shahaf 2019-12-30 22:37 ` Peter Stephenson 0 siblings, 1 reply; 11+ messages in thread From: Daniel Shahaf @ 2019-12-30 18:21 UTC (permalink / raw) To: zsh-users Daniel Shahaf wrote on Mon, Dec 30, 2019 at 18:13:39 +0000: > Peter Stephenson wrote on Sun, Dec 29, 2019 at 23:45:00 +0000: > > On Sun, 2019-12-29 at 20:55 +0000, Peter Stephenson wrote: > > > Unless I'm missing some trick, bufferwords() is the function where we > > > need to update any parser state --- ctxtlex() is too low level for that, > > > it just handles the next token given the current state. So it's > > > probably something like this. Will need a new test adding. > > > > We also need to do something similar for completion, otherwise > > > > foo=(stuff) bar=(... > > > > looks like a glob qualifier. > > > > In this case it's already half handled, so there's a variable to hook > > into. > > Thanks for the fix; it didn't occur to me to change bufferwords(). > > Here's a test. One more: diff --git a/Test/Y01completion.ztst b/Test/Y01completion.ztst index 7ef319f7b..51f604bcf 100644 --- a/Test/Y01completion.ztst +++ b/Test/Y01completion.ztst @@ -216,6 +216,15 @@ F:regression test workers/31611 >NO:{3pm} >NO:{10pm} + comptest $'a=() b=(\t' +0:multiple envarrays +>line: {a=() b=(}{} +>DESCRIPTION:{file} +>DI:{dir1} +>DI:{dir2} +>FI:{file1} +>FI:{file2} + %clean zmodload -ui zsh/zpty ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-30 18:21 ` Daniel Shahaf @ 2019-12-30 22:37 ` Peter Stephenson 2019-12-30 23:46 ` Daniel Shahaf 0 siblings, 1 reply; 11+ messages in thread From: Peter Stephenson @ 2019-12-30 22:37 UTC (permalink / raw) To: zsh-users On Mon, 2019-12-30 at 18:21 +0000, Daniel Shahaf wrote: > Daniel Shahaf wrote on Mon, Dec 30, 2019 at 18:13:39 +0000: > > Here's a test. > > One more: Thanks; shall I commit my change? It looks like it ought to be local enough we can be reasonably confident it's well tested. pws ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-30 22:37 ` Peter Stephenson @ 2019-12-30 23:46 ` Daniel Shahaf 2019-12-31 11:58 ` Daniel Shahaf 2019-12-31 18:35 ` Peter Stephenson 0 siblings, 2 replies; 11+ messages in thread From: Daniel Shahaf @ 2019-12-30 23:46 UTC (permalink / raw) To: zsh-users Peter Stephenson wrote on Mon, 30 Dec 2019 22:37 +00:00: > On Mon, 2019-12-30 at 18:21 +0000, Daniel Shahaf wrote: > > Daniel Shahaf wrote on Mon, Dec 30, 2019 at 18:13:39 +0000: > > > Here's a test. > > > > One more: > > Thanks; shall I commit my change? It looks like it ought to be local > enough we can be reasonably confident it's well tested. The changes do seem fairly local, and not likely to break anything that _doesn't_ use array assignments interactively (get_comp_string()) or in ${(z)} (bufferwords()), so small risk. I don't have a gut feeling for how likely they are to break anything, but if you feel they're not going to necessitate a 5.8.1, then sure, go ahead and commit to master. If you aren't sure, then go ahead and commit to the 5.9 branch. :) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-30 23:46 ` Daniel Shahaf @ 2019-12-31 11:58 ` Daniel Shahaf 2019-12-31 18:35 ` Peter Stephenson 1 sibling, 0 replies; 11+ messages in thread From: Daniel Shahaf @ 2019-12-31 11:58 UTC (permalink / raw) To: zsh-users Daniel Shahaf wrote on Mon, 30 Dec 2019 23:46 +00:00: > Peter Stephenson wrote on Mon, 30 Dec 2019 22:37 +00:00: > > On Mon, 2019-12-30 at 18:21 +0000, Daniel Shahaf wrote: > > > Daniel Shahaf wrote on Mon, Dec 30, 2019 at 18:13:39 +0000: > > > > Here's a test. > > > > > > One more: > > > > Thanks; shall I commit my change? It looks like it ought to be local > > enough we can be reasonably confident it's well tested. > > The changes do seem fairly local, and not likely to break anything > that _doesn't_ use array assignments interactively (get_comp_string()) > or in ${(z)} (bufferwords()), so small risk. I don't have a gut > feeling for how likely they are to break anything, but if you feel > they're not going to necessitate a 5.8.1, then sure, go ahead and > commit to master. > > If you aren't sure, then go ahead and commit to the 5.9 branch. :) I suppose that was too teflony. What I was trying to say is that I don't know the two functions in question well enough to have a gut feeling about how destabilizing/risky a particular change is, but in general, if I had doubts I would err on the side of caution and defer to the next release. Cheers, Daniel ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ${(z)} parsing of multiple array assignments 2019-12-30 23:46 ` Daniel Shahaf 2019-12-31 11:58 ` Daniel Shahaf @ 2019-12-31 18:35 ` Peter Stephenson 1 sibling, 0 replies; 11+ messages in thread From: Peter Stephenson @ 2019-12-31 18:35 UTC (permalink / raw) To: zsh-users On Mon, 2019-12-30 at 23:46 +0000, Daniel Shahaf wrote: > Peter Stephenson wrote on Mon, 30 Dec 2019 22:37 +00:00: > > On Mon, 2019-12-30 at 18:21 +0000, Daniel Shahaf wrote: > > > Daniel Shahaf wrote on Mon, Dec 30, 2019 at 18:13:39 +0000: > > > > Here's a test. > > > > > > One more: > > > > Thanks; shall I commit my change? It looks like it ought to be local > > enough we can be reasonably confident it's well tested. > > The changes do seem fairly local, and not likely to break anything that > _doesn't_ use array assignments interactively (get_comp_string()) or in > ${(z)} (bufferwords()), so small risk. I don't have a gut feeling for how likely > they are to break anything, but if you feel they're not going to necessitate > a 5.8.1, then sure, go ahead and commit to master. > > If you aren't sure, then go ahead and commit to the 5.9 branch. :) I don't see any problem myself, but it was best to check. I've committed it. pws ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2019-12-31 18:36 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-12-23 17:31 ${(z)} parsing of multiple array assignments Daniel Shahaf 2019-12-24 20:34 ` Phil Pennock 2019-12-24 23:16 ` Sebastian Gniazdowski 2019-12-29 20:55 ` Peter Stephenson 2019-12-29 23:45 ` Peter Stephenson 2019-12-30 18:13 ` Daniel Shahaf 2019-12-30 18:21 ` Daniel Shahaf 2019-12-30 22:37 ` Peter Stephenson 2019-12-30 23:46 ` Daniel Shahaf 2019-12-31 11:58 ` Daniel Shahaf 2019-12-31 18:35 ` Peter Stephenson
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).