nesting issue

nesting issue

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

Getting more comfortable with nesting things:

#1:
local in=($1,*(N))
#2:
local sorted=$(print -l "${(n)in[@]}")
#3:
sorted=( "${(f)sorted}" )

     ... works fine, now I do an  exact verbatim nesting of line two 
into line three:

local sorted=( "${(f)$(print -l "${(n)in[@]}") }" )

     "bad substitution".  Ok, that could be because we can't nest quotes so:

local sorted=( "${(f)$(print -l "${(n)in[@]}") }" )
local sorted=( "${(f)$(print -l \"${(n)in[@]}\") }" )
local sorted=( "${(f)$(print -l ${(n)in[@]}) }" )
local sorted=( ${(f)$(print -l ${(n)in[@]}) } )

     .... none of those work.  Is this something were a literal 
substitution of one block of characters can't work?  It hardly matters 
in practice, I prefer the three steps above, it's easier to digest, but 
purely as a theoretical question, can I nest line two and three, or even 
all three lines?  I'd like to understand why the parser doesn't like my 
efforts.  I can see that the 'no nested quotes' rule could make some 
literal substitutions logically impossible. But maybe there is a way.  
If so, seeing how will be instructive.



[-- Attachment #2: Type: text/html, Size: 1600 bytes --]

Re: nesting issue

[Bart Schaefer]

On Sat, May 4, 2024 at 12:36 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> local sorted=( "${(f)$(print -l "${(n)in[@]}") }" )
> local sorted=( "${(f)$(print -l \"${(n)in[@]}\") }" )
> local sorted=( "${(f)$(print -l ${(n)in[@]}) }" )
> local sorted=( ${(f)$(print -l ${(n)in[@]}) } )
>
>     .... none of those work.

They give "bad substitution" because you can't have a space before the
closing "}" / after the closing ")" in the parameter expansion.

local sorted=( "${(f)$(print -l "${(n)in[@]}")}" )

I'm not going into whether that produces the result that you intend.

Re: nesting issue

[Lawrence Velázquez]

On Sat, May 4, 2024, at 11:20 PM, Bart Schaefer wrote:
> On Sat, May 4, 2024 at 12:36 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>>
>> local sorted=( "${(f)$(print -l "${(n)in[@]}") }" )
>> local sorted=( "${(f)$(print -l \"${(n)in[@]}\") }" )
>> local sorted=( "${(f)$(print -l ${(n)in[@]}) }" )
>> local sorted=( ${(f)$(print -l ${(n)in[@]}) } )
>>
>>     .... none of those work.
>
> They give "bad substitution" because you can't have a space before the
> closing "}" / after the closing ")" in the parameter expansion.
>
> local sorted=( "${(f)$(print -l "${(n)in[@]}")}" )
>
> I'm not going into whether that produces the result that you intend.

He should probably throw out the "print -l" and splitting fluff.

	local sorted=($1,*(Nn))

-- 
vq

Re: nesting issue

[Roman Perepelitsa]

On Sun, May 5, 2024 at 5:21 AM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> local sorted=( "${(f)$(print -l "${(n)in[@]}")}" )
>
> I'm not going into whether that produces the result that you intend.

My guess is that it almost certainly does not produce the result Ray
intended. This might though:

    local sorted=( $1,*(nN) )

The difference can be observed in the following cases:

- Nothing matches $1,*.
- $1 starts with a dash.
- At least one element of $1,* contains a newline.
- At least one element of $1,* contains a C style escape sequence.

I think it unlikely that in any of these cases the results produced by
the variant with the print are desired.


Roman.

Re: nesting issue

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]



On 2024-05-04 20:20, Bart Schaefer wrote:
> They give "bad substitution" because you can't have a space before the
> closing "}" / after the closing ")" in the parameter expansion.
>
> local sorted=( "${(f)$(print -l "${(n)in[@]}")}" )
>
Nuts, I wouldn't have even considered that.

Even tho it is accepted here?:

     sorted=( "${(f)sorted}" )

I've tended to space things out for legibility, ok, good to know that 
sometimes the parser doesn't like it.  It really is nit picking but 
'that' space seems to be 'outboard' of the actual text substitution, IOW 
it seems as if the swapping of 'sorted' for '$(print -l "${(n)in[@]}")'
is accepted verbatim an *then* the issue with the space comes up after 
the fact.  Not that it's worth worrying about.  I consider the parser to 
be a miracle worker as it is.

One thing tho, since nested quotes end up un-quoting the inner quotation 
(or is this an exception?), I'm thinking that:

  local sorted=( "${(f)$(print -l ${(n)in[@]})}" )

... one set of quotes -- should be fine.  And it seems fine.  Or is this 
one of those things where spaces in filenames is going to gotcha me?


[-- Attachment #2: Type: text/html, Size: 1910 bytes --]

Re: nesting issue

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 1278 bytes --]



On 2024-05-04 22:40, Roman Perepelitsa wrote:
> local sorted=( $1,*(nN) )

Beautiful.  I get a 'version sort' in one go which is just what I want.  
Subsequent to my code snip I run the thing through ' | sort -V ' to get 
the version sort, but  little 'n' is smart enough to handle that in one 
line.  Dunno, it always seemed to me that 'numeric sort' is intuitively 
what the sort command calls 'version sort'.

This is what I dare say everybody wants:

8 /aWorking/Zsh/Source/Wk/Boneyard 1 % local in=(aaa,*(nN)); print -l $in
aaa,1
aaa,2
aaa,2,howdy
aaa,3a,pard
aaa,3,pard
aaa,4,2a howdy
aaa,5c,pard
aaa,5,pard
aaa,6
aaa,6a,howdy
aaa,7,aaa,7
aaa,8
aaa,9
aaa,9a
aaa,10
aaa,11
aaa,11,pard
aaa,12,12a
aaa,12,some_somme

(
One quibble, I see (cut down):

aaa,3a,pard
aaa,3,pard
aaa,6
aaa,6a,howdy

... I'd expect

aaa,3,pard
aaa,3a,pard

... however ' sort -V' does exactly the same as 'n' so I'm not whining.
)

... and this is not:

8 /aWorking/Zsh/Source/Wk/Boneyard 1 % local in=(aaa,*(nN)); print -l 
$in | sort -n
aaa,1
aaa,10
aaa,11
aaa,11,pard
aaa,12,12a
aaa,12,some_somme
aaa,2
aaa,2,howdy
aaa,3a,pard
aaa,3,pard
aaa,4,2a howdy
aaa,5c,pard
aaa,5,pard
aaa,6
aaa,6a,howdy
aaa,7,aaa,7
aaa,8
aaa,9
aaa,9a

(sorry about the garbage filenames, they're stress testers.)


[-- Attachment #2: Type: text/html, Size: 2237 bytes --]

Re: nesting issue

[Lawrence Velázquez]

On Sun, May 5, 2024, at 9:13 AM, Ray Andrews wrote:
> On 2024-05-04 20:20, Bart Schaefer wrote:
>> They give "bad substitution" because you can't have a space before the
>> closing "}" / after the closing ")" in the parameter expansion.
>> 
>> local sorted=( "${(f)$(print -l "${(n)in[@]}")}" )
>> 
> Nuts, I wouldn't have even considered that.   
>
> Even tho it is accepted here?:
>
>     sorted=( "${(f)sorted}" )

You're not comparing the right spaces.  The problem Bart pointed
out is more akin to

        sorted=( "${(f)sorted }" )


> One thing tho, since nested quotes end up un-quoting the inner 
> quotation (or is this an exception?)

Command substitution introduces a new quoting context.  Things like

        "$(cmd "arg with spaces")"

work fine.


> I'm thinking that:
>
>  local sorted=( "${(f)$(print -l ${(n)in[@]})}" )
>
> ... one set of quotes -- should be fine.  And it seems fine.

Unquoted ${(n)in[@]} drops empty elements and is equivalent to
unquoted ${(n)in}.  Whether that's acceptable is dependent on
application requirements, as always.


-- 
vq

Re: nesting issue

[Lawrence Velázquez]

On Sun, May 5, 2024, at 9:52 AM, Ray Andrews wrote:
> On 2024-05-04 22:40, Roman Perepelitsa wrote:
>> local sorted=( $1,*(nN) ) 
> Beautiful.  I get a 'version sort' in one go which is just what I want. 
>  Subsequent to my code snip I run the thing through ' | sort -V ' to 
> get the version sort, but  little 'n' is smart enough to handle that in 
> one line.  Dunno, it always seemed to me that 'numeric sort' is 
> intuitively what the sort command calls 'version sort'.

Those are not the same.

	% : >foo{-2,10,1,+3}
	% print -l -- foo*(n)
	foo+3
	foo-2
	foo1
	foo10
	% printf %s\\n foo* | gsort -V
	foo1
	foo10
	foo+3
	foo-2


> This is what I dare say everybody wants:

Nonsense.  There is no universally desired sort.  It depends entirely
on what the application needs.


> One quibble, I see (cut down):
>
> aaa,3a,pard
> aaa,3,pard

I don't see this behavior.

	% locale
	LANG="en_US.UTF-8"
	LC_COLLATE="en_US.UTF-8"
	LC_CTYPE="en_US.UTF-8"
	LC_MESSAGES="en_US.UTF-8"
	LC_MONETARY="en_US.UTF-8"
	LC_NUMERIC="en_US.UTF-8"
	LC_TIME="en_US.UTF-8"
	LC_ALL=
	% : >aaa,3{,a},pard
	% in=(aaa,*(n)) 
	% print -l $in
	aaa,3,pard
	aaa,3a,pard


-- 
vq

Re: nesting issue

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 338 bytes --]



On 2024-05-05 12:44, Lawrence Velázquez wrote:
> Command substitution introduces a new quoting context. Things like
>          "$(cmd "arg with spaces")"
>
> work fine.
That's good to know and grammatically  satisfying.  Almost 'necessary' 
because the work of quotation in a command substitution is independent 
of what comes next.

[-- Attachment #2: Type: text/html, Size: 783 bytes --]

Re: nesting issue

[Lawrence Velázquez]

On Sun, May 5, 2024, at 4:14 PM, Ray Andrews wrote:
> That's good to know and grammatically  satisfying.  Almost 'necessary' 
> because the work of quotation in a command substitution is independent 
> of what comes next.

Useful and sensible, but not "necessary" by any means.  One of the
reasons the legacy `...` form is discouraged is that in some shells
it requires nested double quotes to be backslash-quoted.

	% ksh -c 'typeset -p .sh.version'
	.sh.version='Version AJM 93u+ 2012-08-01'
	% ksh -c 'echo "`echo "a   b   c"`"'
	ksh: : cannot execute [Is a directory]
	a b c
	% ksh -c 'echo "`echo \"a   b   c\"`"'
	a   b   c

You can make sense of this if you squint, but there's a reason no
one likes it.

-- 
vq

Re: nesting issue

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]



On 2024-05-05 13:14, Lawrence Velázquez wrote:
> Nonsense. There is no universally desired sort. It depends entirely on 
> what the application needs.
Naturally, but I expect that when we are sorting numbers, '2' should 
come before '10' most of the time.  No not 'everybody' but ... sheesh, 
I'm just saying that I'm glad that '(n)' gives me what I'd call version 
sort.  That's all.  Can't I be satisfied?
> I don't see this behavior.
> ...
> 	aaa,3,pard
> 	aaa,3a,pard
>
8 /aWorking/Zsh/Source/Wk/Boneyard 1 % var=( aaa,3*(nN) ); print -l $var
aaa,3a,pard
aaa,3,pard

... I don't know what to say.

8 /aWorking/Zsh/Source/Wk/Boneyard 1 % var=( aaa,6*(nN) ); print -l $var
aaa,6
aaa,6,    # sixth char ',' precedes sixth char 'a'
aaa,6a
aaa,6a,
aaa,6a,howdy # sixth char 'a' precedes sixth char ','
aaa,6,howdy

It's not how I understand sorting but I don't doubt there is some method 
to it.  'sort' does the same.  'sort' has been around a long time so I 
don't doubt that what it does is very well thought out whether it makes 
sense to me or not.   Version sort is not a strict 'dictionary' sort, it 
'interprets', so the above does what it thinks best.  I would have expected:

aaa,6,
aaa,6a
aaa,6,howdy
aaa,6a,howdy

... but that's not the way it is. Not worth worrying about.

vmlinuz-6.1.0-20-amd64

... that's my Linux version, the algorithm for sorting such versions 
must be a bit arbitrary.  I'm not complaining.





[-- Attachment #2: Type: text/html, Size: 2711 bytes --]

Re: nesting issue

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 414 bytes --]



On 2024-05-05 13:45, Lawrence Velázquez wrote:
> Useful and sensible, but not "necessary" by any means.  One of the
Useful and sensible are good enough for me.

> reasons the legacy `...` form is discouraged is that in some shells
> it requires nested double quotes to be backslash-quoted.
>
> 	% ksh -c 'echo "`echo \"a   b   c\"`"'
That's sadistic.  How you guys keep the parser sane is beyond my imagining.

[-- Attachment #2: Type: text/html, Size: 1038 bytes --]

Re: nesting issue

[Lawrence Velázquez]

On Sun, May 5, 2024, at 6:18 PM, Ray Andrews wrote:
> On 2024-05-05 13:14, Lawrence Velázquez wrote:
>> I don't see this behavior.
>> ...
>> 	aaa,3,pard
>> 	aaa,3a,pard
>> 
> 8 /aWorking/Zsh/Source/Wk/Boneyard 1 % var=( aaa,3*(nN) ); print -l $var       
> aaa,3a,pard
> aaa,3,pard
>
> ... I don't know what to say.

Seems like differing locales (which I overlooked earlier).  On
Debian 12:

	% locale | grep -e LC_COLLATE -e LC_ALL
	LC_COLLATE="en_US.UTF-8"
	LC_ALL=
	% : >aaa,3{,a},pard
	% print -lr -- aaa,3*(n)
	aaa,3a,pard
	aaa,3,pard
	% () { local LC_COLLATE=C; print -lr -- aaa,3*(n) }
	aaa,3,pard
	aaa,3a,pard

My earlier result was with "en_US.UTF-8" on macOS.


> 8 /aWorking/Zsh/Source/Wk/Boneyard 1 % var=( aaa,6*(nN) ); print -l $var
> aaa,6
> aaa,6,    # sixth char ',' precedes sixth char 'a'
> aaa,6a
> aaa,6a,
> aaa,6a,howdy    # sixth char 'a' precedes sixth char ','
> aaa,6,howdy

Can't say I understand this result, but as above:

	% : >aaa,6{,a}{,\,,\,howdy}
	% print -lr -- aaa,6*(n)
	aaa,6
	aaa,6,
	aaa,6a
	aaa,6a,
	aaa,6a,howdy
	aaa,6,howdy
	% () { local LC_COLLATE=C; print -lr -- aaa,6*(n) }
	aaa,6
	aaa,6,
	aaa,6,howdy
	aaa,6a
	aaa,6a,
	aaa,6a,howdy


> I would have expected:
>
> aaa,6,    
> aaa,6a
> aaa,6,howdy
> aaa,6a,howdy

Why?


-- 
vq

Re: nesting issue

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 1476 bytes --]



On 2024-05-05 16:26, Lawrence Velázquez wrote:
>> ... I don't know what to say.
> Seems like differing locales (which I overlooked earlier).  On
> Debian 12:
Man ... I dunno, maybe the world needs different sorting logic between 
Canada and the US but to me it just looks like making trouble.  Anyway 
kudos for figuring that out.

I would have expected:
>> aaa,6,
>> aaa,6a
>> aaa,6,howdy
>> aaa,6a,howdy
> Why?
Because at least the dictionary sort proceeds left to right with each 
subsequent character sorted as a sub-sort within the fixed sort arrived 
at by previous characters.  IOW as you move right, nothing ever changes 
to the left.  If 'aaa,6,' precedes 'aaa,6a' then every possible 
expansion of 'aaa,6,' will precede every possible expansion of 'aaa,6a'. 
It's the same as with numbers: every 2,xxx is greater than every 1,xxx.  
I suppose there's something in the version sort that interferes with 
that.  Mind ... (n) is not an explicit version sort, so what's the 
algorithm?  This gets interesting.  And ... Ooops, no, that's not what 
I'm expecting I'm expecting:

> aaa,6,
> aaa,6,howdy
> aaa,6a
> aaa,6a,howdy

... a few more tests ... 'sort -V' is different.  But '(n)' seems the 
same as 'sort -n':

aaa,6
aaa,6,
aaa,6a
aaa,6a,
aaa,6a,howdy
aaa,6,howdy    # I'd just like to know how this ends up last. I'd expect 
it 3d.  Both version sort and '(n)' agree that this is last. There will 
be an algorithm I'd just like to know what it is.


[-- Attachment #2: Type: text/html, Size: 2597 bytes --]

Re: nesting issue

[Bart Schaefer]

On Sun, May 5, 2024 at 6:51 PM Ray Andrews <rayandrews@eastlink.ca> wrote:
>
> Man ... I dunno, maybe the world needs different sorting logic between Canada and the US but to me it just looks like making trouble.

C sort is done in the order of the ASCII values of the characters.
Locale-based sorting is intended to match (hardcopy) dictionary order
for the represented language.  The latter may mean that "A" and "a"
both sort before "B" and "b" and so on, whereas ASCII sorts all
uppercase letters before all lowercase.  Handling of numbers and
punctuation relative to letters and numbers may also vary.

Sorting usually proceeds left to right by character position, which
means e.g. "22" sorts before "4" because ASCII "2" (062) is compared
to "4" (064).  When you ask for "numeric" sorting, each consecutive
series of digits is instead interpreted as an integer that occupies
one "position" -- but the strings are still compared position by
position, so when comparing "a2" to "10b" the character "a" is
compared to the integer "10" and sorts after it.  You only get actual
numeric ordering for "mixed" alphanumerics when the numbers occupy the
same relative positions.

Version sorting is yet another thing entirely, see
Functions/Misc/is-at-least for example.

Re: nesting issue

[Lawrence Velázquez]

On Sun, May 5, 2024, at 9:51 PM, Ray Andrews wrote:
> There will be an algorithm I'd just like to know what it is.

It's not that simple.  You're conflating the sorting method with
the locale's collation rules.  Under Debian's en_US.UTF-8 locale
even straight lexicographic sort can be surprising:

	% foo=(1, 1a,); print -lr -- ${(o)foo}
	1,
	1a,
	% foo=(1,a 1a,a); print -lr -- ${(o)foo}
	1,a
	1a,a
	% foo=(1,b 1a,b); print -lr -- ${(o)foo}
	1a,b
	1,b

I'm afraid I can't explain this; perhaps someone else can.  If it
bothers you, consider temporarily setting LC_ALL=C (or C.UTF-8)
while you sort.

-- 
vq

Re: nesting issue

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]



On 2024-05-05 20:22, Bart Schaefer wrote:
> Version sorting is yet another thing entirely, see
> Functions/Misc/is-at-least for example.
>
A superb summary Bart.  It's why I don't get too worried  about it 
because there are so many possible interpretations.  I don't want to 
waste list time on it much.  I am curious about what zsh (or sort) is 
doing tho:

aaa,6,
aaa,6a
aaa,6a,howdy
aaa,6,howdy

I'm expecting the last line to be second because comma precedes 'a' 
irrespective of what happens in column 7.  But really, don't waste time 
on it.  In practice I'm quite happy with '(n)'.

[-- Attachment #2: Type: text/html, Size: 1151 bytes --]

Re: nesting issue

[Ray Andrews]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]



On 2024-05-05 20:43, Lawrence Velázquez wrote:
> I'm afraid I can't explain this; perhaps someone else can.  If it
> bothers you, consider temporarily setting LC_ALL=C (or C.UTF-8)
> while you sort.
>
Don't waste any time on it.  I've run into this 'locale' stuff in the 
past and as I said I think it's a can of worms.  You might think that 
all English dictionaries would use the same lexicography, but ... In 
every practical way '(n)' gives me just what I want.  Oh, but it was 
good to shake out the idea that 'numeric' or 'version' were the issue.  
As you show, even dictionary sort seems to go strange. Let it be.

[-- Attachment #2: Type: text/html, Size: 1090 bytes --]